[Pycon] [new paper] "Fridolín Pokorný" - Selinon - dynamic distributed task flows

[info a pycon.it]

Title: Selinon - dynamic distributed task flows
Duration: 60 (includes Q&A)
Q&A Session: 15
Language: en
Type: Talk

Abstract: Have you ever tried to define and process complex workflows for data processing? If the answer is yes, you might have struggled to find the right framework for that. You've probably came across Celery - popular  task flow management  for Python. Celery is great, but it does not provide enough flexibility and dynamic features needed badly in complex flows. As we discovered all the limitations, we decided to implement Selinon.

Selinon is a task flow manager that is built on top of popular Celery distributed task queue. Selinon enhances Celery task flow management and allows you to create and model task flows in your distributed environment that can dynamically change behavior based on computed results in your cluster that can be orchestrated using orchestration tools such as Kubernetes or OpenShift. Task flow configuration is done in simple YAML configuration files. Selinon also offers some advanced features such as automatically resolving tasks that need to be executed in case of selective task runs, automatic tracing mechanism, integration with Sentry monitoring or support of changes in your task flows on redeployment and many others.

Tags: [u'OpenShift', u'celery', u'OpenSource', u'distributed-systems', u'Big-Data', u'kubernetes']