[Pycon] [new paper] "Hrvoje Gazibara" - Building world-class data processing pipelines using Apache Airflow

[info a pycon.it]

Title: Building world-class data processing pipelines using Apache Airflow
Duration: 60 (includes Q&A)
Q&A Session: 15
Language: en
Type: Talk

Abstract: Constructing maintainable and extendable data processing pipelines is a difficult job. Topics like maintaining a library of standard components, working out task execution dependencies, and scheduling and monitoring parallel tasks all make life a real pain. If you are looking for a remedy, read on.

The talk, aimed at data analysis professionals and enthusiasts, introduces Apache Airflow. It's an open source tool for creating, running, visualizing and monitoring data processing pipelines using Python. Together we will explore the core functionality of Apache Airflow using a real-world example. After the talk, you will be able to create, schedule, monitor and debug your own scalable and maintainable data pipelines with a lot less headache.



Tags: [u'automation', u'monitoring', u'parallelization', u'data-analysis', u'scalability', u'scheduling']