KDD Competition 2010

We're interested in working on the KDD Competition, as a way to focus our machine learning exploration -- and maybe even finding some interesting aspects to the data! If you're interested, drop us a note, show up at a weekly Machine Learning meeting, and we'll use this space to keep track of our ideas.

Resources

• KDD Rules and Data Format
• R language
• libsvm
• Weka
• List of other competitions in which we could engage
• Hadoop
• Mahout -- machine learning libraries for Hadoop
• Pig language
• Pig Latin Manual
• Cloudera -- see videos for Hadoop intro
• Vikram's awesome Hadoop/EC2 scripts
• Our mailing list

TODOs

• Vikram -- will help setting up Hadoop for the rest of us & create a guide for Mahout setup
• Thomas -- will get libsvm working on the data and put together a "how to" guide for doing so
  • put together a perl script which will take random samples from the data, for working on smaller instances
  • put together a simple R script for loading the data
• Andy -- will get Weka working on the data and put together a "how to" guide for doing so
• Erin -- Will put meeting notes of 5/19 on https://www.noisebridge.net/wiki/Machine_Learning; will work on data transformations and ways to create better representations of the data; will provide the orthogonalized data sets

• We will need to make sure we don't get disqualified for people belonging to multiple teams! Do not sign up anybody else for the competition without asking first.

Notes

• For KDD submission: to zip the submission file on OSX: use command line, otherwise will complain about __MACOSX file: e.g.: zip asdf.zip algebra_2008_2009_submission.txt

Ideas

• Add new features by computing their values from existing columns -- e.g. correlation between skills based on their co-occurence within problems. Could use Decision tree to define boundaries between e.g. new "good student, medium student, bad student" feature
• Dimensionality reduction -- transform into numerical values appropriate for consumption by SVM

Who we are

• Andy; Machine Learning
• Thomas; Statistics
• Erin; Maths
• Vikram; Hadoop

(insert your name/contact info/expertise here)