[Python] [OT] Cheap MapReduce in Go

[Carlo Miron]

< http://marcio.io/2015/07/cheap-mapreduce-in-go/>

tl;dr

Sometimes you don’t need overly complex infrastructures or systems to do a
job well. In this case, we were running these exact same aggregations over
close to 20 EMR instances that would take a few minutes to execute the
entire MapReduce job over hundreds of Gigabytes of data each day.

When we decided to take a look at this problem again, we rewrote this task
using Go, and we now simply run this on a single 8-core machine and the
whole daily execution takes about 10 minutes. We cut a lot of the costs
associated with maintaining and running these EMR systems and we just
schedule this Go app to run once a day over our daily dataset.

You can find the entire code here:
https://gist.github.com/mcastilho/e051898d129b44e2f502

©

-- 
|:**THE BEER-WARE LICENSE** (Revision 42):
| <carlo a golang.it> wrote this mail. As long as you retain
| this notice you can do whatever you want with this stuff.
| If we meet some day, and you think this stuff is worth it,
| you can buy me a beer in return.
|                                            --Carlo Miron :
-------------- parte successiva --------------
Un allegato HTML è stato rimosso...
URL: <http://lists.python.it/pipermail/python/attachments/20150713/9eec8db9/attachment.html>