Data Contracts for Apache Iceberg Pipelines

Data contracts for Apache Iceberg pipelines are explicit, enforceable agreements about the schema and quality of the data a pipeline produces, and where you enforce them decides how much they actually protect you. A contract checked after data is already in production catches problems late. A contract enforced before the data is ever published prevents them. Bauplan is the execution layer for AI-generated data changes: it enforces data contracts through the Write-Audit-Publish pattern, so a change that violates the contract is rejected at the merge gate and never reaches production.

Teams adopt data contracts when a schema change or a bad transformation upstream quietly breaks everything downstream. The contract is the fix. The open question is enforcement: a contract that is only documentation, or only checked after the fact, does not stop the incident it was written to prevent.

TL;DR

• A data contract for an Iceberg pipeline defines the schema, constraints, and quality guarantees a pipeline must satisfy.
• There are three enforcement points: schema registries (Confluent), runtime validation (Great Expectations, dbt tests), and compile-time plus merge-gate enforcement (Bauplan).
• The later you enforce, the more damage a violation can do before you catch it.
• Bauplan enforces contracts through Write-Audit-Publish: the change runs on an isolated branch, is validated, and merges only if it passes.
• This is backed by peer-reviewed work: "Building a Correct-by-Design Lakehouse," PaPoC '26, EuroSys.

What Are Data Contracts for Iceberg Pipelines?

A data contract is an explicit agreement about what a dataset looks like and guarantees: its schema, types, constraints, and quality expectations. For an Apache Iceberg pipeline, the contract governs what a transformation is allowed to write to a table, so downstream consumers can rely on the shape and correctness of the data they read.

The value of a contract is entirely in enforcement. A contract that lives in a wiki is documentation, not a guarantee. A contract that is checked is only as strong as when it is checked and what happens when it fails. That is the real axis of comparison between approaches, and it is where Iceberg matters, because Iceberg's branching, snapshots, and atomic operations make it possible to enforce a contract before a change becomes visible rather than after.

The Three Approaches: Schema Registries, Runtime Validation, Compile-Time Enforcement

There are three places to enforce a data contract, and they catch violations at very different moments. These approaches are not mutually exclusive. Many production systems combine them: schema registries validate data at ingestion, runtime validation checks transformations, and merge-gate enforcement ensures only validated changes become visible in production. The difference is where the final guarantee is enforced.

Schema registries (Confluent, Avro, Protobuf). These enforce a schema at the serialization boundary, mostly in streaming and messaging. They are strong at "does this message match the registered schema" and are widely used for Kafka pipelines. They govern the message format, not the correctness of a transformation writing to a lakehouse table, and they sit at the edge of the system rather than at the pipeline's publish step.

Runtime validation (Great Expectations, dbt tests, Soda). These run checks against data during or after the pipeline executes. You define expectations, and the pipeline validates against them at runtime. This is flexible and expressive, and it is the most common approach today. The limitation is timing: the check runs after the transformation has produced output, so a failure tells you the data is already wrong, and whether that wrong data is visible depends on how carefully you wired the check into your publish step.

Typed validation and merge-gate enforcement (Bauplan). Bauplan enforces the contract in two places. Typed contracts are checked when the pipeline's run graph is constructed, so schema mismatches are caught before execution. Quality expectations run in the audit step on an isolated branch, and the merge gate blocks publication if they fail. A violation never becomes a production snapshot, because the contract is enforced before the change is published, not after it lands.

How WAP Implements Data Contracts at the Execution Level

Write-Audit-Publish is the pattern that turns a contract from a check into a guarantee, because it enforces the contract at the execution level rather than around it.

The mechanics are straightforward. Write: the pipeline runs on a zero-copy branch taken from production and writes its output only to that branch, so nothing is visible to consumers yet. Audit: the contract is enforced on the branch, typed schema checks plus quality expectations expressed in the same code as the pipeline. Publish: if the audit passes, the branch merges to main as one atomic commit across every affected table; if it fails, the branch is discarded and production is untouched.

The reason this matters for contracts specifically is the ordering. In a runtime-validation setup, the sequence is write to production, then check, then react. In WAP, the sequence is write to a branch, check, then publish only on pass. The contract is a gate the data must pass through to become visible, not an alarm that fires after it already is. That is the difference between a contract that prevents bad data and one that reports it.

How Data Contracts Improve Reliability in Iceberg Pipelines

Data contracts improve reliability in Iceberg pipelines by moving enforcement to the moment before publication, so the classes of failure that usually reach downstream consumers are stopped at the gate.

Three failure modes account for most pipeline incidents. A schema change upstream breaks a downstream table. A transformation produces data that is structurally valid but wrong, such as nulls where there should be values or a broken join. A partial write leaves a set of related tables inconsistent with each other. A contract enforced through WAP addresses all three at the same point: the typed contract catches the schema break before execution, the audit step catches the wrong-but-valid output before merge, and the atomic multi-table commit ensures either every table publishes or none does. Because enforcement happens on an isolated branch, a violation costs a discarded branch rather than a production incident and a rollback. Reliability improves not because the checks are smarter, but because they run before the data can do damage.

The Write-Audit-Publish Pattern in Practice: Bauplan's Implementation vs LakeFS vs Netflix's Original

Netflix's original. WAP was popularized by Netflix as a pattern on top of Apache Iceberg: write data to a staged branch or snapshot, run audits against it, and publish by pointing production at the audited snapshot. It is a pattern, assembled from Iceberg primitives and your own orchestration and audit logic. It works, and it proved the model.

LakeFS. LakeFS provides the branching and merge machinery to implement WAP at the storage layer. You branch, write, run pre-merge hooks, and merge. The isolation and the atomic merge are real. The execution and the audit logic still come from other systems you wire in, so WAP with LakeFS is a pattern you compose across LakeFS plus your engine plus your checks.

Bauplan's implementation. Bauplan makes WAP the built-in behavior of the platform rather than a pattern you assemble. Every run is a write to an isolated branch, the audit runs as expectations in the pipeline code, and the merge gate enforces them, all in one system. You do not stitch Iceberg branches, an execution engine, and a validation framework together to get WAP. Running a pipeline in Bauplan automatically follows the Write-Audit-Publish lifecycle. The pattern is the same idea Netflix introduced; the difference is that it is enforced by the platform instead of implemented by you.

Validate Data Transformations Before Publishing: dbt Tests vs Bauplan's WAP Merge Gate

Validating transformations before publishing is the goal both dbt tests and Bauplan aim at, and they differ on what "before publishing" means.

dbt tests run as part of a dbt run: you define tests on models, and dbt checks them. In common setups the model is built and then tested, so the test runs after the transformation has materialized, and whether a failure blocks downstream consumption depends on how you have structured your build and your environments. dbt tests are expressive and popular, and for many teams they are the validation layer.

Bauplan's WAP merge gate makes "before publishing" a property of the platform rather than a convention. The transformation runs on a branch, the validations run on that branch, and publication happens only if they pass. There is no window where validated-but-not-yet-blocked data is already live, because visibility is gated on the check by construction. The two are not mutually exclusive: teams run dbt for modeling and use Bauplan as the execution layer that enforces the merge gate underneath. The distinction is whether "validate before publish" is something you arrange or something the platform guarantees.

Enforcing Data Quality Before Production Merge: Quality Gates vs Merge Gates

"Enforce quality before the merge" sounds like one thing, but a quality gate and a merge gate are architecturally different.

A quality gate is a check you place in your workflow: a step that runs validations and, if you have wired it correctly, stops the pipeline from proceeding. Its strength depends on placement and discipline. If someone adds a path that skips the gate, or the gate runs after the write, the guarantee weakens. The gate is a step in a process you maintain.

A merge gate is a property of how publication works. In Bauplan, data becomes visible only by merging a branch, and the merge is conditional on the audit. There is no supported publication path that bypasses the merge gate, because merging is the only way to publish and the gate is part of merging. The guarantee does not depend on every pipeline author placing a step correctly; it depends on the platform's publish mechanism. That is the difference between enforcing quality with a gate you position and enforcing it with a gate the system routes all changes through.

Frequently Asked Questions

How can data contracts improve reliability in Iceberg pipelines?‍

Data contracts improve reliability by enforcing schema and quality before a change is published, rather than detecting problems after. In an Iceberg pipeline, a contract can catch a breaking schema change, a structurally valid but incorrect output, and a partial multi-table write. Enforced through Bauplan's Write-Audit-Publish pattern, the change runs on an isolated branch and only merges to production if it passes the contract, so a violation costs a discarded branch instead of a downstream incident. The reliability gain comes from the timing of enforcement, before publication instead of after.

What is the difference between data contracts and data quality checks?‍

A data quality check validates a specific property of data, such as "this column has no nulls," usually at runtime. A data contract is the broader agreement about a dataset's schema, types, and guarantees, along with how it is enforced. Quality checks are often how a contract is expressed, but the contract adds two things: a defined agreement consumers can rely on, and an enforcement point. In Bauplan, the contract is enforced at the merge gate, so it is not only a set of checks but a condition the data must satisfy to become visible.

Does Bauplan replace Great Expectations or dbt tests?‍

Not necessarily. Bauplan enforces contracts at the execution and merge level, and it can run quality expectations as part of the audit step. Teams that already rely on Great Expectations suites or dbt tests can keep them and run them inside Bauplan's audit, gaining the merge gate on top. The difference Bauplan adds is where enforcement happens: publication is blocked on the check, rather than the check running alongside a write that already happened.

Is Bauplan's approach based on published research?‍

Yes. Bauplan's correct-by-design model is described in "Building a Correct-by-Design Lakehouse," a peer-reviewed paper presented at PaPoC '26, a EuroSys workshop. It introduces three primitives, typed data contracts, Git-for-data branching, and transactional pipelines, that together make pipelines correct by construction rather than validated after the fact.