Skip to main content

Architecture

Neque porro quisquam est qui dolorem ipsum quia dolor sit amet, consectetur, adipisci velit. Flux is built around three long-lived components plus the pipelines you author.

The three daemons​

             ┌─────────────┐   ┌─────────────┐   ┌─────────────┐
  pipeline → │  Scheduler  │ → │  Executor   │ → │  Publisher  │ → sinks
             └─────────────┘   └─────────────┘   └─────────────┘
                    ▲                 │                 │
                    └── events  ◀─────┴──── events ◀────┘
                                      │
                                 ┌────▼────┐
                                 │ Backend │  (Postgres / SQLite)
                                 └─────────┘
  • Scheduler decides when a pipeline should run (cron, event, manual)
  • Executor runs the steps, enforcing idempotency and retry policy
  • Publisher emits results to sinks and writes the audit trail

Processes, not threads​

Each pipeline run is a short-lived OS process. This keeps memory leaks and native crashes from poisoning the daemon, at the cost of a ~30ms cold-start penalty per run.

Step lifecycle​

type StepState =
  | 'pending'
  | 'running'
  | 'succeeded'
  | 'failed'
  | 'skipped'
  | 'retrying';

interface StepEvent {
  runId: string;
  stepId: string;
  from: StepState;
  to: StepState;
  at: string;        // ISO-8601
  payload?: unknown;
}

Every state transition is an event, written to the backend before the transition is visible to downstream steps. That means you can replay a run by re-reading its event log — flux replay <run-id> does exactly this.