What is Prefect?
Prefect is an open-source workflow orchestration framework for Python, licensed under Apache 2.0 and developed by the company of the same name. A pipeline is ordinary Python: a function decorated with @flow calls functions decorated with @task, and Prefect adds scheduling, retries, caching, state tracking, and a UI. Prefect 3.0 became generally available in 2024. Prefect Cloud is the hosted control plane; a self-hosted Prefect server is also available.
How Prefect Works
- Flows and tasks: because flows are plain Python, loops, conditionals, and fan-out decided at runtime work without special syntax.
- Deployments and work pools: a deployment packages a flow with a schedule or trigger. Workers pull runs from work pools and execute them on processes, containers, Kubernetes, or serverless infrastructure.
- Events and automations: every state change is an event, and external systems can emit events too. Automations react to them, for example triggering a downstream flow when an upstream one completes, or alerting when no run has happened in an hour.
- Assets: Prefect added an optional asset layer (the
@materializedecorator) so a task can declare what it produces, with lineage.
Prefect also builds FastMCP, a Python framework for Model Context Protocol servers, and on July 13, 2026 announced an agreement to acquire Dagster Labs. Both orchestrators are set to continue as separate products.
Prefect on an Iceberg Lakehouse
| Prefect 3 | Apache Airflow 3 | Dagster | |
|---|---|---|---|
| Pipeline definition | Python functions with decorators | DAG files of operators and tasks | Assets that materialize tables |
| Scheduling style | Schedules plus event-driven automations | Cron, timetables, asset events | Declarative automation on asset state |
| Dynamic pipelines | Native | Dynamic task mapping | Partitions and dynamic outputs |
| Table lineage | Optional assets | Assets and OpenLineage | Central to the model |
| Best fit | Operational, event-reactive work | Broad integrations, large existing estates | Pipelines modeled as tables |
Prefect suits lakehouse work that is operational and dynamic. Example: an automation fires when new files land in the raw bucket. A flow appends them to an Iceberg bronze table through Spark or Dremio, runs dbt to rebuild silver and gold tables, then loops over every table whose small-file count crossed a threshold and runs Iceberg’s rewrite_data_files procedure on each. The list of tables to compact is discovered at runtime, which is where Prefect’s plain-Python model is simplest.
Where Prefect is lighter is as a system of record for table-level lineage; teams that need that often pair it with a metadata platform or choose Dagster. For the full comparison, read Orchestration in 2026: Airflow 3 vs Dagster vs Prefect vs Event-Driven. See also pipeline orchestration and Apache Iceberg.
Frequently Asked Questions
Is Prefect free?
The Prefect framework and self-hosted server are open source under Apache 2.0. Prefect Cloud is a paid service with a free tier.
Does Prefect replace dbt or Spark?
No. Prefect coordinates them. dbt, Spark, Dremio, or Flink do the data processing.
What does the Dagster acquisition change?
As announced, Dagster keeps its name, license, and product line. Teams should watch release cadence and roadmap signals rather than assume either product will be merged soon.
Learn More
To learn more about the Data Lakehouse, read The Lakehouse Built for Everyone by Alex Merced. You can find this and other books by Alex Merced at books.alexmerced.com.