FLOSSmole

August data released

noreply@blogger.com (Megan Squire) — Tue, 12 Aug 2008 19:05:00 +0000

Hello moles!

Lots of exciting news this month:

1. All releases are up on Google Code instead of SF this month. Go to the FLOSSmole downloads page on Google Code to pick up this month's files.

2. Sourceforge stats server is down, so no stats this month.

3. Debian is back! The data is minimal, but check out what we do have and enjoy that.

4. I've updated the Google Code FLOSSMole wiki also, so if you're looking for basic how-to or documentation, check there first.

5. coming soon: Free Software Foundation (re-released!), Eclipse (ask and ye shall receive)

ENJOY!!

OSS Watch in Oxford

noreply@blogger.com (Megan Squire) — Mon, 21 Jul 2008 09:55:00 +0000

I am at Oxford in the UK (staying at Hertford College Graduate Center) for a few days for the OSS Watch workshop on Profiling Open Source Communities.

OSS Watch is the National Advisory Service on open source for UK Further Education (FE) and Higher Education (HE). As such, it is part of our remit to help FE and HE institutions and projects who want to engage with open source development, and a key factor for that is the development of open source communities.

I'm giving The Standard FLOSSmole Talk today at 11:45GMT.

July data released

noreply@blogger.com (Megan Squire) — Wed, 09 Jul 2008 03:25:00 +0000

Well moles, we were hoping to move to Google Code over the summer to host our file releases, but unfortunately, they have some size (and other?) limitations which are messing us up. They have no problem storing certain files, but others produce errors when I try to upload them. They are rejecting more and more of our files. I've put in a ticket to see if they can bump up the quota over there, but in the meantime, I'm back to putting files up on Sourceforge Downloads page.

Here's the download files area over on SF (available files for July: Freshmeat, Rubyforge, and Objectweb)

(Remember, since it's an "odd" month, for Sourceforge data, select the "June" release.)

I was able to get some of the files up on Google Code, so go over there and check the files out there if you like.

June SF data posted

noreply@blogger.com (Megan Squire) — Tue, 10 Jun 2008 18:46:00 +0000

June Sourceforge data has been posted.

This time, I've released the files onto our Google Code project page. Let's see how we like using Google Code for this (psst, I can tell you as the person who does the file releases: it's a lot easier to use Google Code than Sourceforge for this particular part of the job!)

Enjoy!

UPDATE: Because of file size limits on Google Code, I've had to re-release our code onto Sourceforge.

Here is the link to the Flossmole SF Downloads Page, where you can get ALL the June files, including the huge datamart files.

May 2008 data released

noreply@blogger.com (Megan Squire) — Tue, 06 May 2008 23:15:00 +0000

Moles, the May 2008 data has been released. Find it all at our SF project page. Forges included this month are: FM, RF, OW.

April 2008 data released

noreply@blogger.com (Megan Squire) — Wed, 16 Apr 2008 23:34:00 +0000

Hi moles! April 2008 data has been released on our SF project page. Enjoy it! (Debian data included!)

UPDATE 08-JUL-2008: new files released to get past problem with data quality on these April files.

February data released

noreply@blogger.com (Megan Squire) — Sat, 09 Feb 2008 17:57:00 +0000

Go to our file release page on Sourceforge and pick up all the latest files.

Included this month: SF, FM, RF, OW flat files and data marts (sql statements).

(Debian is on the way! I'll update here as soon as it's ready.)

Enjoy!

Jan files released and Dec datamarts posted

noreply@blogger.com (Megan Squire) — Fri, 11 Jan 2008 15:11:00 +0000

Hi Moles!

The January data files have been released, and I went back and released the December data marts too (sql statements to make your own database).

Have fun with it! Grab the data from our Sourceforge project page.

December data released

noreply@blogger.com (Megan Squire Conklin) — Tue, 25 Dec 2007 13:57:00 +0000

Hello moles! After considerable delay, here is the December data. Enjoy!

Download the data from our SF release page!

November data released

noreply@blogger.com (Megan Squire Conklin) — Sun, 04 Nov 2007 21:42:00 +0000

Files for November 2007 have been released here:

Sourceforge File Release Page

October data released

noreply@blogger.com (Megan Squire Conklin) — Mon, 29 Oct 2007 14:34:00 +0000

Sorry for the lack of postings! The data has been released, I guess I just forgot to post the updates.

Here are the October data links.

August data released (mostly)

noreply@blogger.com (Megan Squire Conklin) — Sun, 05 Aug 2007 02:24:00 +0000

Greetings moles! We have the August data release mostly done.

Get the data marts (sql) or the flat files over on our Sourceforge file release page, or use the query tool to explore.

I say the August is "mostly" done because we had some technical difficulties with the Sourceforge and Debian releases, but these should be cleared up around mid-month.

So if you're looking for Sourceforge data and you just can't wait, why not give the Freshmeat or Rubyforge or Objectweb or Free Software Foundation a try? You might find some interesting things in there.

We have some new trove definitions that have been added to Freshmeat, in order to help better explain the trove categories of each project. In the flat files, the new files are: fmProjectTrove and fmTroveDefs. In the data marts, the new file is datamart_fm_trove_defs. A big thanks to user Bob Daly for this trove information and the idea to include it!

SF Status pages: bug fix

noreply@blogger.com (Megan Squire Conklin) — Mon, 23 Jul 2007 03:15:00 +0000

Just wanted to let you guys know about a bug fix on the status pages for the SF data. Each project has a "status" (i.e. beta, alpha, production, etc). We were under the impression that each project only had a single status (we assumed this represented the project's "current" status), but this turned out not to be the case.

Our code was therefore erroneously grabbing only the first of a possible list of status codes. Consequently, some projects that had multiple status codes were not shown correctly.

I have gone into the June 2007 and April 2007 data and made the corrections, and I'll get to the older data sets at some point. (Let me know if this is a high priority for you and I'll try to get to them faster.) Obviously, the upcoming August data set will now be run correctly.

When looking through the old packages, you'll want to look for the files marked ".fixed.bz2".

debian data released

noreply@blogger.com (Megan Squire Conklin) — Thu, 05 Jul 2007 02:27:00 +0000

I collected some debian package data and started parsing it to see what kind of stuff we might find in there.

I will probably need some help from the user community on this one, to know what sort of data you find interesting in these packages.

Here are the files I collected:

project home pages for stable, unstable, and testing versions (Example of stable page)

developer information page (Example of a developer information page)

changelog page (Example of a changelog)

bug reports page (Example of a bug reports page)

Obviously there is a lot of information there, and I only parsed some of it out for this initial run. Here are the items I parsed and released:

package name, version, parent directory
any URLs found in the copyright page, and any URLs found within the textual description of the project found on the stable project page
developers (maintainers and co-maintainers listed on the developer information page)

I'd love to hear from the community about what items you would like to see parsed out.

July data released

noreply@blogger.com (Megan Squire Conklin) — Tue, 03 Jul 2007 04:05:00 +0000

July data is out for Freshmeat, Rubyforge, Objectweb, Free Software Foundation.

Go get it!

June data released - all forges

noreply@blogger.com (Megan Squire Conklin) — Tue, 05 Jun 2007 14:46:00 +0000

June data has been released for all forges.

Head over to the project page on Sourceforge and gather all the data you need!

May 2007 data released for small forges

noreply@blogger.com (Megan Squire Conklin) — Thu, 03 May 2007 03:36:00 +0000

May 2007 data is released for the small forges. (Reminder that Sourceforge data is next scheduled for a release in June.)

As usual, there are 3 ways to get FLOSSmole data:
(1) Flat files (includes May 2007 data, plus historical data if you wish)
(2) Get the data marts
(3) Browse results of common queries

Announcement: data marts now available

noreply@blogger.com (Megan Squire Conklin) — Thu, 12 Apr 2007 18:11:00 +0000

By popular demand of our user base (and some hard work by our developers, especially ruphus_13), we now provide data marts for Sourceforge data.

The new package, called DataMarts contains all the SQL create and insert statements for creating your own version of the FLOSSmole database - for multiple data sources (Sourceforge, Freshmeat, Rubyforge, ObjectWeb, FSF)

The marts are created following each of our data collections; we collect and parse the data as usual. We then load it into our database as usual, and create the raw flat-file data dumps as we have been doing since 2004. The new feature we are announcing today is that we also now provide the SQL data dumps so you can auto-load our data into your own local database for easier processing and more complex mining tasks.

So, there are now numerous ways to get our data:

--install the data marts into your own mysql database
--download and analyze the flat, delimited data files
--play around with the query tool

Good work, moles!! Another milestone in our open source data collection and dissemination effort.

April 2007 data released for all forges

noreply@blogger.com (Megan Squire Conklin) — Thu, 05 Apr 2007 20:30:00 +0000

April 2007 data is released for all forges. Here is a summary of the data we have and where to get it:

Sourceforge data
- General Forge Information(Get it)
  - Project code names, project display names, developer counts, date project was registered, long project descriptions
- Developer Information(Get it)
  - Developer login names, real names, developers-per-project and what role they have on that project, are they an admin?
- Data about Projects(Get it)
  - Database type by project, number of downloads per project, rank of project, intended audience, topic of project, status of project, license(s), operating system(s), programming language(s), real URL of project, tracker data, donors to projects, user interfaces
Freshmeat data (Get it)
- project names, descriptions, authors per project, project URLs (real urls and freshmeat urls), project licenses, vitality/popularity ranks by freshmeat
Rubyforge data (Get it)
- user interfaces, programming languages, developers per project & roles, licenses, programming languages, environment, intended audience, list of all projects, natural language, status, topics
ObjectWeb data (Get it)
- user interfaces, programming languages, developers per project & roles, licenses, programming languages, environment, intended audience, list of all projects, natural language, status, topics
Free Software Foundation data (Get it)
- Developers per project, project registration dates, project descriptions, interfaces, languages, licenses, full list of projects, project URLs
SourceKibitzer data (Get it)
- Metrics on a variety of FLOSS projects. Metrics include: loc, cloc, ncloc, dc, nom, wmc, ncss, npath, fnaout, abstr_coupl, todo_count, bool_exp (see release notes for more details on these metrics)

Each set of data described here is released three ways

1) As text files, ready for you to grab and import into your favorite stats package, spreadsheet, or even into your own database.
2) As datamarts, with SQL code for you to input into your own database. (Get the datamarts)
3) If you want a sense of what raw data we have, play around with the new query tool for a while.

New Query Tool

noreply@blogger.com (Megan Squire Conklin) — Tue, 27 Mar 2007 04:53:00 +0000

Check out the New Query Tool for running common, pre-defined canned queries. (Thanks Gregg!)

The old query tool is still available. We'll be adding real-time graphing and some more bells and whistles to the new tool as time is available.

March data released

noreply@blogger.com (Megan Squire Conklin) — Sun, 04 Mar 2007 03:00:00 +0000

March data is released for the following sources (forges/directories/repositories):

--Freshmeat
--Rubyforge
--ObjectWeb
--Free Software Foundation
--SourceKibitzer

Get the data from our Sourceforge file release page

Enjoy!

(The April release will include Sourceforge and the other 5 forges.)

new donations from SourceKibitzer

noreply@blogger.com (Megan Squire Conklin) — Thu, 01 Mar 2007 04:25:00 +0000

Great news moles, we have a new donation partner: Source Kibitzer.

The facts:
--In our system, SourceKibitzer is forge #6, and has the abbreviation "SK".
--SK will be part of the monthly data cycle, so expect new SK files once per month (just like Freshmeat, Rubyforge, Objectweb, and Free Software Fndn.)
--SK files are available on our file releases page on Sourceforge.

The first file we released from SourceKibitzer is for February, 2007. For each of some 500-odd projects, it includes:

project name
density of comments
todo count
commented lines of code
total lines of code
non-comment lines of code
non-commenting source statements
number of methods
sum of data abstraction coupling
boolean expression complexity
fanout
npath complexity
weighted method count

Some very interesting stuff! Get the SourceKibitzer February data.

February Data Released for All Forges

noreply@blogger.com (Megan Squire Conklin) — Tue, 06 Feb 2007 21:03:00 +0000

Hi moles!

We've been digging as usual, and we now announce that February data is released for all 5(!) forges. This includes:

forge (abbreviation) - datasource_id
=====================================
Sourceforge (SF) - 46
Freshmeat (FM) - 47
Rubyforge (RF) - 48
Objectweb (OW) - 49
Free Software Foundation (FSF) - 50

Get the files at our Sourceforge Project Page

FLOSSmole mentioned in Information Week

noreply@blogger.com (Megan Squire Conklin) — Sun, 04 Feb 2007 17:44:00 +0000

FLOSSmole gets mentioned in a nice little article in a US trade magazine called Information Week. The article is about how to differentiate open source "winners and losers". With a proper shout out to Open BRR and the Business Readiness Rating, which is more like what the article is really about, here is the excerpt:

How To Tell The Open Source Winners From The Losers
By Charles Babcock
InformationWeek
Feb 3, 2007

(this excerpt is from page 2):

The Business Readiness Rating service is collecting public feedback on its proposal for evaluating open source code. Eventually, says [Tony] Wasserman [of Carnegie Mellon West], it will host automated software tools that harvest statistics from open source project sites that help predict their likelihood of success: the number of developers and core developers, frequency of releases, support queries and unanswered queries, and the number of bugs tracked versus fixed. Those metrics will then be used in a decision-making framework to sort through open source projects. Some automated evaluation tools already exist. They include FLOSSmole (the Free/Libre Open Source Software Mole), which automatically burrows into data on an open source project site such as page views, downloads, bandwidth consumed by downloading, and number of comments posted.

database schema

noreply@blogger.com (Megan Squire Conklin) — Thu, 25 Jan 2007 15:35:00 +0000

You can browse the sql version of the FLOSSmole database schema in CVS on Sourceforge, OR you can check the schemaspy-generated view of the schema. The CVS version is updated less often than the schemaspy version. I'll try to run a schemaspy view every few months and overwrite the old one. The query tool also lists all the tables, so if you see a table in there and you want to know what it does, do "describe <tablename>" or just check out the schema!