Version Control for Data Pipelines
Version Control for Data Pipelines: Artifacts, Code, or Execution?
Version control for data pipelines can mean three different things, and the one that matters most for production is the one most tools skip: versioning the execution itself. Versioning the data files captures what an output was. Versioning the pipeline code captures how it was written. Neither guarantees that a change ran correctly and completely against production. Bauplan is the execution layer for AI-generated data changes: it versions the pipeline run, so every commit is a validated change that either applied fully or not at all.
TL;DR
- "Version control for data pipelines" can mean versioning artifacts, code, or execution.
- Artifact versioning (the file-based approach) captures what the output was, not whether the run was correct.
- Code versioning captures how the pipeline was written, not what it did to the data.
- Bauplan versions execution: branch production, run atomically, commit or roll back.
- Each Bauplan commit is a validated pipeline run across all affected tables, and it is reversible.
What "Data Pipeline Version Control" Means: Three Different Things People Mean
When people say they want to version their data pipelines, they usually mean one of three things, and they are not interchangeable.
The first is artifact versioning: keeping versioned copies of the data files a pipeline produces, so you can point at "the dataset as of last Tuesday." Tools in the DVC lineage do this well.
The second is code versioning: keeping the pipeline's code in Git, so you can see how the transformation logic changed over time. This is standard practice and necessary, but it versions the recipe, not the meal.
The third is execution versioning: versioning the actual run, the change a pipeline made to your data, as an atomic, validated, reversible unit. This is the one that answers "can I safely change production and undo it if it is wrong," and it is the gap Bauplan fills.
Most setups cover the first two and assemble something approximate for the third. Bauplan is built around the third directly.
The Gap: Versioning Files or Code Does Not Version the Change
Here is why the distinction is not academic. You can have every data file versioned and every line of pipeline code in Git, and still have no safe, atomic record of the change a pipeline made to production.
Artifact versioning tells you what a table looked like at a point in time, but not whether the run that produced it was complete or correct, and restoring an artifact is a manual operation, not an atomic rollback of a change across related tables. Code versioning tells you what the transformation logic was, but the same code can produce a good run one day and a bad one the next depending on the input data, so the commit history of the code says nothing about whether last night's run corrupted a table. The change itself, the thing you actually want to trust and be able to undo, falls between the two.
Git remains the right tool for pipeline code. Artifact versioning remains useful for datasets and machine learning assets. Execution versioning fills the remaining gap by recording the validated production change itself. Mature systems typically use all three together.
Bauplan: Transactional Execution Versioning, Branch the Data, Run Atomically, Commit or Rollback
Bauplan versions the change by making the pipeline run the unit of version control. The flow is git-shaped, applied to execution.
You branch production, a zero-copy, isolated view of your lakehouse. The pipeline runs against that branch, reading real production data and writing only to the branch. Validation runs on the branch, and if it passes, the change commits to production as a single atomic operation across every affected table. If it fails, the branch is discarded and production never changed. Each commit is therefore a validated run, not a snapshot of files or a diff of code, and rolling back is returning to an earlier commit, which reverses the change across all tables at once. That is execution versioning: the version is the change, it is atomic, and it is reversible.
Data Pipeline Commit History in Bauplan: What Each Commit Contains
Because the commit is the run, Bauplan's history is a record of changes, not of files or code edits.
Each commit represents the published result of a validated pipeline run. It captures the state of the data after that change, tied to the run that produced it, so the history reads as "these are the changes that were made to production, in order, each one validated before it landed." You can see what changed, return to any earlier point, and reverse a change by pointing back to the commit before it. This is different from a code commit history, which records edits to the transformation logic, and from an artifact history, which records file versions. Bauplan's commit history is the audit trail of what actually happened to your data, which is what you reach for when you need to understand or undo a production change.
Frequently Asked Questions
Is DVC the same as Bauplan?
No. DVC is an artifact versioning tool: it versions data files alongside your code, which is useful for tracking datasets and ML artifacts. Bauplan versions execution: it branches production, runs the pipeline in isolation, and commits the change atomically, so each version is a validated run rather than a file snapshot. DVC answers "what did this dataset look like at this version," and Bauplan answers "what change did this pipeline make to production, and can I undo it atomically." They address different needs, and for versioning the pipeline's effect on production data, Bauplan's model is the fit.
How is versioning data pipelines different from versioning code?
Versioning code, in Git, tracks how your transformation logic changed over time. It is necessary but it does not tell you what a run did to your data, because the same code can produce a correct result one day and a bad one the next depending on the input. Versioning a data pipeline, in Bauplan's sense, tracks the change itself: each commit is a validated pipeline run against production, atomic and reversible. Code versioning captures the recipe; pipeline versioning captures the outcome of actually running it.
What does a Bauplan commit contain?
A Bauplan commit represents a validated pipeline run that merged to production. It captures the resulting state of the data tied to the run that produced it, across every affected table, as an atomic change. The commit history is therefore a record of the changes made to production, in order, each validated before it landed, and you can return to any earlier commit to reverse a change.
