Editing Machine Learning/Hadoop

Jump to navigation Jump to search
Warning: You are not logged in. Your IP address will be publicly visible if you make any edits. If you log in or create an account, your edits will be attributed to your username, along with other benefits.

The edit can be undone. Please check the comparison below to verify that this is what you want to do, and then publish the changes below to finish undoing the edit.

Latest revision Your text
Line 1: Line 1:
===About===
===About===
* Google had so much data that ''reading'' the data from disk took a lot of time, much less processing
** So they needed to parallelize everything, even disk access
** Make the processing local to where the data is, to avoid network issues
* Parallelization is hard/error-prone
** Want to have a "shared-nothing" architecture
** Functional programming
* Map
Runs the function on each item in the list, returns the list of output from running the function on each item
<pre>
def map(func, list):
  return [func(item) for item in list]
</pre>
Example:
<pre>
def twice(num):
  return num*2
</pre>
* Reduce
Take a function (which takes two arguments) and a list, and iteratively continues through, accumulating
<pre>
def reduce(func, list):
  a = func(list[0], list[1])
  for
</pre>
===Examples/Actual===
<pre>
def map(key,value):
  # process
  emit(another_key, another_value)
def reduce(key, values):
  # process the key and all values associated with it
  emit(something)
</pre>
* Average
** keys are line numbers, values are what's in it
** file:
*** 1  (1,2)
*** 4  (2,4)
*** 5  (3,5)
*** 6  (4,6)
<pre>
def map(key,value):
  emit("exist",1)
  emit("x",value)
def reduce(key, values):
  # process the key and all values associated with it
  emit(something)
</pre>


===Tutorials===
===Tutorials===
* http://www.cloudera.com/videos/introduction_to_pig
* http://www.cloudera.com/videos/introduction_to_pig


===How to Debug===
* To debug a streaming Hadoop process, cat your source file, pipe it to the mapper, then to sort, then to the reducer
** Ex: cat princess_bride.txt | scripts/word-count/mapper.py | sort | scripts/word-count/reducer.py


===Tools===
===Tools===
* Hadoop
* Hive
* Pig: A high-level language for compiling down to MapReduce programs
* MapReduce on Amazon (?)
Please note that all contributions to Noisebridge are considered to be released under the Creative Commons Attribution-NonCommercial-ShareAlike (see Noisebridge:Copyrights for details). If you do not want your writing to be edited mercilessly and redistributed at will, then do not submit it here.
You are also promising us that you wrote this yourself, or copied it from a public domain or similar free resource. Do not submit copyrighted work without permission!

To protect the wiki against automated edit spam, we kindly ask you to solve the following CAPTCHA:

Cancel Editing help (opens in new window)
Retrieved from "https://www.noisebridge.net/wiki/Machine_Learning/Hadoop"