Pyturbo Package: A Pipeline System for Efficient Concurrent Execution

PyPI version Downloads

Pyturbo utilizes multiple level of abstract to efficiently execute parallel tasks.

  • Worker: a process.
  • Stage: a group of peer workers processing the same type of tasks.
  • Task: a data unit transferred between stages. At each stage, a task is processed by one worker and will result in one or multiple downstream tasks.
  • Pipeline: a set of sequential stages.
  • Job: a data unit for a pipeline, typically a wrapped task for the first stage.
  • Result: output of a job processed by one pipeline, typically a set of output tasks from the last stage.
  • System: a set of peer pipelines processing the same type of jobs.