KDD Competition 2010

We're interested in working on the KDD Competition, as a way to focus our machine learning exploration -- and maybe even finding some interesting aspects to the data! If you're interested, drop us a note, show up at a weekly Machine Learning meeting, and we'll use this space to keep track of our ideas.

Resources

• KDD Rules and Data Format
• R
• libsvm
• Weka
• (other competitions in which we could engage)

TODOs

Next week, after the Hadoop presentation, we'll show each other how to get the tools working on the data (what one needs to download, any data transformations needed, how to produce submission output) and share any insights on the data gleaned so far

• Vikram -- will present on Hadoop next week!
• Thomas -- will get libsvm working on the data and put together a "how to" guide for doing so
  • put together a perl script which will take random samples from the data, for working on smaller instances
  • put together a simple R script for loading the data
• Andy -- will get Weka working on the data and put together a "how to" guide for doing so
• Erin -- will work on data transformations and ways to create better representations of the data

• We will need to make sure we don't get disqualified for people belonging to multiple teams!

Notes

• to zip the file on OSX: use command line, otherwise will complain about __MACOSX file: e.g.: zip asdf.zip algebra_2008_2009_submission.txt

Ideas

• Add new features by computing their values from existing columns -- e.g. correlation between skills based on their co-occurence within problems. Could use Decision tree to define boundaries between e.g. new "good student, medium student, bad student" feature
• Dimensionality reduction -- transform into numerical values appropriate for consumption by SVM