Pyturbo Package: A Pipeline System for Efficient Concurrent Execution

Pyturbo utilizes multiple level of abstract to efficiently execute parallel tasks.

• Worker: a process.
• Stage: a group of peer workers processing the same type of tasks.
• Task: a data unit transferred between stages. At each stage, a task is processed by one worker and will result in one or multiple downstream tasks.
• Pipeline: a set of sequential stages.
• Job: a data unit for a pipeline, typically a wrapped task for the first stage.
• Result: output of a job processed by one pipeline, typically a set of output tasks from the last stage.
• System: a set of peer pipelines processing the same type of jobs.