Data Quality Checks in Python Pipelines
Data quality checks in Python pipelines almost always run at runtime, validating data after the pipeline has already produced it, and that timing is the single most important thing to understand about them. Great Expectations, Soda, and Pandera all help validate data quality, though they operate at different levels: expectation suites, monitoring, and dataframe schema validation respectively. What they do not change is when the check runs relative to the write. Bauplan is the execution layer for AI-generated data changes: it catches schema violations at pipeline compilation time and gates publication behind a Write-Audit-Publish merge, so the question shifts from "was the data wrong" to "did the wrong data ever reach production." The honest framing here is not replacement. It is timing.
TL:DR
- Great Expectations, Soda, and Pandera run quality checks at runtime, against data produced during or after execution.
- Runtime validation is expressive and valuable, but a failure means the data is already written and you react.
- Bauplan catches schema violations at compile time, before execution, and runs quality checks in an audit step.
- Bauplan's merge gate blocks publication on failure, so bad output never becomes a production snapshot.
- The two combine: run your Great Expectations suite inside Bauplan's audit step and gain the merge gate on top.
How Great Expectations Works: Expectation Suites, Checkpoints, Runtime Validation
Great Expectations is the most widely used data quality framework in Python, and for good reason. You define expectations, declarative statements such as "this column is not null" or "these values are within a range", and group them into suites. Checkpoints run those suites against a batch of data and produce a validation result, with data docs that make the outcome readable.
The model is expressive and well-designed, and it fits naturally into a Python pipeline. It is a runtime validation tool: a checkpoint validates a batch of data that already exists at the moment the checkpoint runs. That placement is the thing to reason about, because it determines what a failure means.
The Limitation of Runtime Validation: What Happens When a Check Fails After Data Is Written
The limitation is not accuracy. It is timing. A runtime check validates data that has already been produced, so when it fails, the data already exists somewhere. What happens next depends entirely on how you wired the check into your pipeline.
If the checkpoint runs after the transformation writes to a production table, a failure means bad data is already live, and you are into cleanup: quarantine, roll back, or alert downstream consumers who may have already read it. If you carefully staged the write and gated production on the checkpoint, you have effectively built a merge gate by hand, and the guarantee is only as reliable as that wiring. Runtime validation can be made safe, but the safety comes from the pipeline structure you build around it, not from the check itself. The check tells you the data is wrong; it does not, on its own, stop the data from being visible.
Bauplan's Typed Contracts: Catching Schema Violations at Compile Time
Bauplan moves part of the problem earlier than runtime. Typed contracts describe the schema a pipeline step must produce, and they are checked when Bauplan constructs the run graph, before the pipeline executes. A schema mismatch, a wrong type, a missing column, fails at compile time, so the pipeline does not run and produce bad output that you then have to catch.
This is a category of error that runtime validation can only catch after the fact, because a runtime check needs data to check, which means the transformation already ran. Catching schema violations at graph-construction time removes an entire class of "the data is already written and it is the wrong shape" incidents.
Runtime validation is still necessary even with compile-time schema enforcement. Compile-time checks can prove that a pipeline is structurally consistent, but they cannot prove that the resulting data is correct. Content-level expectations still need to run against actual data, which is why Bauplan combines typed contracts with runtime validation in the audit step rather than replacing it.
Bauplan's WAP Gate: Preventing Bad Output from Merging to Production
Compile-time checks handle schema. Content problems, valid shape but wrong values, still need to run against data, and that is what the audit step and merge gate handle. In Bauplan, a run executes on a zero-copy branch, quality expectations run against the output on that branch, and the branch merges to production only if they pass. If they fail, the branch is discarded and production never saw the output.
So Bauplan enforces quality at two moments: schema at compile time, and content in the audit before merge. Neither depends on the pipeline author remembering to place a checkpoint after a staged write, because the merge gate is the only path to publication. Bad output does not reach production, not because it was detected quickly, but because publication is conditional on the check by construction.
Built-In Quality Gates vs Workflow-level Validation
This is the structural difference between the two approaches: a built-in quality gate versus workflow-level validation.
Workflow-level validation is a check you add to a pipeline. It is flexible and you can put it anywhere, which is its strength and its weakness. Its guarantee depends on placement, and a pipeline that skips it, or runs it after the write, loses the protection. Great Expectations, Soda, and dbt tests are workflow-level by nature: powerful checks you position in a process you maintain.
A built-in quality gate is part of how the platform publishes. In Bauplan, there is no way to publish except by merging a branch, and the merge is gated on the audit. The check is not a step you remember to add; it is a property of the publish mechanism. That is the difference between quality that depends on every author wiring it correctly and quality the platform enforces on every change by default.
Testing Pipeline Output Before Deployment: The Audit Step in Bauplan's WAP Pattern
Testing pipeline output before deployment is exactly what the audit step is for, with a precise definition of "before deployment": before the output is merged and made visible.
In a typical Python pipeline, testing output before deployment means running checks in CI against sample data, or running validations in a staging environment, then promoting. It works, and the rigor depends on how faithful staging is to production and how disciplined the promotion is. In Bauplan, the audit step runs against the actual output on an isolated branch derived from real production state, and deployment is the merge that only happens on pass. You are testing the real output, in isolation, and the test result controls whether it deploys. There is no separate promotion step where the tested artifact and the deployed artifact could diverge, because they are the same branch.
Automated Data Validation in a Lakehouse: Scheduled Tests vs Merge-Gate Enforcement
Automated validation in a lakehouse usually means scheduled tests, and scheduled tests and a merge gate protect against different things.
Scheduled validation runs checks on a cadence: every hour, every night, a job scans tables and flags anomalies. This is how a lot of lakehouse quality monitoring works, and it is good at catching drift and issues that emerge over time. What it cannot do is prevent a bad write, because by the time the scheduled job runs, the data has been live for minutes or hours, and consumers may have already used it. Scheduled validation is detection on a timer.
Merge-gate enforcement runs the check at the moment of publication, not on a schedule. In Bauplan, every change is validated as a condition of merging, so there is no window between a bad write and the next scheduled scan. The two are complementary: scheduled monitoring catches slow drift and external changes, and the merge gate prevents bad pipeline output from ever publishing. But they are not substitutes, and for the specific job of "stop this pipeline from writing bad data," the merge gate is the one that actually prevents it.
Using Great Expectations and Bauplan Together: Where Each Fits
The framing throughout is timing, not replacement, and the practical conclusion is that the two compose well.
Great Expectations is expressive, widely adopted, and something many teams have already invested in, with suites that encode real institutional knowledge about their data. Bauplan provides isolation, compile-time schema enforcement, and a merge gate. You can run your existing Great Expectations suite inside Bauplan's audit step, so the expectations you already trust now execute on an isolated branch and gate the merge. You keep the library and the suites; you gain the property that a failing expectation blocks publication rather than reporting after a write. For teams asking whether to pick one, the better answer is usually to keep Great Expectations for expressing quality and let Bauplan control when the validated data becomes visible.
Comparison
Frequently Asked Questions
Is Great Expectations better than Bauplan?
They do different jobs, so "better" depends on the need. Great Expectations is a strong, expressive framework for defining and running data quality checks, and it is excellent at that. Bauplan is an execution layer that isolates runs, enforces schema at compile time, and gates publication on validation. Great Expectations tells you whether data meets expectations; Bauplan controls whether data that fails expectations can reach production. Many teams use both, running Great Expectations suites inside Bauplan's audit step.
Can I use Great Expectations and Bauplan together?
Yes, and it is a natural combination. Bauplan runs pipelines on isolated branches and gates the merge on an audit step, and your Great Expectations suite can run as that audit. The expectations you already maintain then execute against the output on a branch, and publication is blocked if they fail. You keep Great Expectations for expressing quality and gain a merge gate that prevents failing data from being published.
What is the difference between a data quality check and a typed contract?
A data quality check validates a property of data at runtime, such as a range or a null constraint, after the data exists. A typed contract describes the schema a pipeline step must produce and is checked before execution, at graph construction. The quality check needs data to run against; the typed contract does not, because it validates structure ahead of time. Bauplan uses both: typed contracts for schema at compile time, and quality expectations in the audit step before merge.
Where do data quality checks run in a Bauplan pipeline?
They run in the audit step, against the pipeline's output on an isolated branch, before any merge to production. Schema-level checks happen earlier still, at compile time when the run graph is built. Nothing is published unless both pass, so quality checks in Bauplan gate publication rather than validating data that is already live.
