Run Python Data Pipelines Without Infrastructure
Most modern data tools claim to run Python pipelines "without infrastructure," but the claim means very different things depending on the tool, and only some of them mean it literally. Prefect and Dagster reduce the infrastructure you manage. Bauplan removes it. That gap, "less" versus "none," is the whole comparison. Bauplan is the execution layer for AI-generated data changes: you write Python and SQL functions, and they run on serverless compute with no workers to deploy, no YAML to maintain, and no cluster to size.
Teams run into this when they adopt a tool for its clean local developer experience and then discover that shipping to production means standing up and operating workers, agents, or executors. The demo had no infrastructure. Production did.
TL;DR
- "No infrastructure" splits into two claims: reduce what you manage (Prefect, Dagster) or eliminate it (Bauplan).
- Prefect uses managed workers and cuts operational load, but production still runs on a deployment model you operate.
- Dagster has an excellent local experience, but production requires deploying and running agents or executors.
- Bauplan runs plain Python functions on serverless compute. No workers, no YAML, no cluster, no deploy step.
- The difference only shows up at execution time and in production, not in the local demo.
What "No Infrastructure" Actually Means: Servers, Clusters, Workers, YAML, DAGs
Before comparing tools, it helps to name what infrastructure actually is in a data pipeline, because "no infrastructure" is often measured against only part of the list.
Infrastructure here is the sum of what you provision and operate to make a pipeline run in production: servers or containers that host execution, clusters that you size and scale, workers or agents that pull and run work, the YAML and config that define deployments, and the DAG boilerplate that wraps your actual logic. A tool can remove some of these and leave the rest. The honest question is not "does it have a nice local mode" but "what do I run and operate once this is live and on call for it".
Measured that way, most "no infrastructure" tools remove servers-you-rack and cluster-you-tune, while leaving workers-you-deploy and config-you-maintain. That remainder is where the operational cost actually lives.
Prefect: Workflow Orchestration for Building, Observing and Reacting to Data Pipelines
Prefect genuinely reduces infrastructure. Its hybrid model runs orchestration as a managed service while your code executes in your environment, and managed work pools cut down what you operate compared to running everything yourself. For teams coming from self-hosted Airflow, that is a real reduction.
It is a reduction, not an elimination. Your flows run on workers, and workers are a deployment you stand up, connect, and keep alive in production. You define deployments, choose work pools, and manage the infrastructure those workers run on, whether that is your own compute or managed infrastructure you still configure. The local experience is light. The production model is a deployment model. Prefect reduces the infrastructure you manage; it does not get you to zero.
Dagster: Modern Data Orchestrator Platform
Dagster's local developer experience is one of the best in the category. You define assets, run them locally, and get lineage and a clear UI with very little friction. That polish is a real reason teams adopt it.
Production is a different setting. Running Dagster in production means deploying the platform together with agents and/or executors, depending on your deployment model. Dagster tracks and orchestrates the work well, but something still has to run it, and that something is infrastructure you deploy and operate. As with Prefect, the local ease does not carry through unchanged to production. The demo is infrastructure-free. The deployment is not.
Bauplan
Bauplan takes the claim literally. You write a data pipeline as ordinary Python and SQL functions, decorate them, and run them. There is no worker to deploy, no work pool to configure, no agent to keep alive, no YAML deployment spec, and no cluster to size. You submit the function and Bauplan runs it on ephemeral serverless compute, then tears it down.
The same code path runs in development and in production. There is no separate "now operationalize it" phase where infrastructure reappears, because the runtime provides the execution infrastructure and Bauplan operates it. What you write is the logic. What you ship is the logic. The result lands as a versioned commit on Apache Iceberg tables in your object storage, isolated on a branch and published atomically, but that is the runtime's job, not yours to wire up.
Python Functions as Data Pipelines: Bauplan vs Dagster vs Prefect, What Happens at Execution Time
All three tools let you express pipelines as Python functions, so the difference is not the authoring model. It is what happens when the function actually runs.
In Prefect, a decorated flow is submitted to a worker you have deployed, and that worker executes it in your environment. In Dagster, an asset or op is materialized by the deployed platform through its executors. In both cases, there is a running process you provisioned sitting between "call the function" and "it executes," and that process is the infrastructure you operate.
In Bauplan, the decorated function runs on serverless compute that spins up for the run and disappears after. There is no persistent worker you own in the path. Execution also provides data guarantees that the orchestrators leave to the underlying execution system: the run takes a zero-copy branch from production, writes only to that branch, and merges atomically if it passes validation. So at execution time the difference is twofold: no infrastructure you operate in the path, and data isolation as a property of the run rather than something you assemble around it.
Data Engineering Without Cluster Management: Managed Platform vs No Platform
"Without cluster management" is another claim worth splitting, because a managed platform and no platform are not the same thing.
Managed data platforms such as Snowflake, Databricks, or BigQuery remove the work of running a cluster, and that is valuable. You still work within a platform you provision, configure, and pay for as a running resource, and often size against expected load. The underlying compute is managed for you, but it is still there and its cost model assumes it exists.
Bauplan is the "no platform" end of that spectrum for pipeline execution. There is no cluster, managed or otherwise, sitting in the path. Compute is ephemeral and per-run, so there is nothing to size, nothing warm to pay for between runs, and no capacity to plan against. For data engineering specifically, that means you think about the transformation, not the cluster it runs on. The distinction matters most under bursty or agent-driven load, where a managed cluster is either oversized and idle or undersized and throttling, and per-run serverless execution simply scales to the work.
From Dev to Production: Prefect vs Bauplan, Side by Side
The clearest way to see "less" versus "none" is to follow one pipeline from local development to production.
Prefect
- Write the flow as decorated Python functions. Clean and local.
- Define a deployment: entrypoint, work pool, schedule, parameters.
- Stand up and configure a worker or work pool in your environment.
- Ensure the worker has the right dependencies and access.
- Deploy, then keep the worker running and monitored in production.
- Infrastructure you now operate: the worker and the environment it runs in.
Bauplan
- Write the pipeline as decorated Python and SQL functions. Clean and local.
- Run it. The same command runs against a branch in development.
- Promote by running against production; every run is isolated on a branch and merged atomically.
- Infrastructure you now operate: none. The runtime provisions and tears down compute per run.
Same authoring experience in development. The divergence is the middle of the Prefect list, the deployment and worker steps, which has no equivalent in Bauplan.
Where Each Tool Sits
When to Use Prefect or Dagster, When to Use Bauplan
Use Prefect or Dagster when your primary need is orchestration: scheduling, dependency graphs, retries, backfills, and observability across many pipelines and systems. Both are strong orchestrators, and Bauplan does not replace that role. If what you want is a control plane that coordinates work, that is their job.
Use Bauplan when the goal is to run Python and SQL pipelines with genuinely no infrastructure to operate, and you want each run isolated, validated, and published atomically, including when AI agents write and run the code. Bauplan owns execution and its guarantees, so there is nothing to deploy underneath it.
Use both in the common setup. Keep Prefect or Dagster as the orchestrator, and call Bauplan pipelines as the execution step. You keep scheduling and observability, and the execution underneath carries no infrastructure of its own and publishes safely.
Frequently Asked Questions
Does Prefect require infrastructure?
Yes, though less than self-hosted orchestrators. Prefect runs orchestration as a managed service, but your flows execute on workers you deploy and operate in your environment. You define deployments, configure work pools, and keep the workers and their compute running in production. Prefect reduces the infrastructure you manage compared to running everything yourself, but it does not eliminate it. The workers and the environment they run in are infrastructure you operate.
Can I run Python data pipelines completely serverlessly?
Yes, with a platform built for it. Bauplan runs Python and SQL data pipelines on ephemeral serverless compute with no workers, clusters, or deployments to operate. You write and decorate functions, submit them, and the runtime provisions compute for the run and tears it down after. Function-as-a-service platforms like AWS Lambda also run Python serverlessly, but without data awareness, so for pipelines specifically a serverless data platform gives you serverless execution plus Iceberg tables, isolation, and atomic publication.
What is the difference between "no infrastructure" in Prefect, Dagster, and Bauplan?
Prefect and Dagster reduce infrastructure but keep a deployment model in production: workers or agents you run and operate. Bauplan removes the deployment model entirely. You write functions and run them on serverless compute with nothing to deploy or keep alive. The difference does not appear in local development, where all three feel light. It appears in production, where Prefect and Dagster require running workers and Bauplan does not.
Do I still need an orchestrator if I use Bauplan?
Not necessarily, but many teams keep one. Bauplan executes pipelines with isolation and atomic publication, and can run without an external orchestrator for straightforward cases. Teams that need cross-system scheduling, complex dependencies, or platform-wide observability keep Prefect, Dagster, or Airflow as the orchestrator and call Bauplan as the execution layer underneath.
