Skill Issues

TL;DR

Evaluating agents on data engineering workflows requires test cases with destructive operations. We open source our data generation framework, which leverages our Git-for-Data capabilities to model the data lifecycle at scale, from reading and writing to appending and deleting data. The novel evals are then used to optimize our skills with GEPA, producing skills that make coding agents more efficient.

Coding agents are taking the world (and our customers) by storm. Customers want productivity (however measured) to go up, while simultaneously keeping token spending in check: unsurprisingly, improving one is easy; both is hard.

As the saying goes, we can't optimize what we can't measure: if we want to answer the pressing question "what is the cheapest model that reliably completes this task?", we first need to build thorough evals, collecting real-world tasks and pairing them with a scoring function that tells us whether a given agent solved them. This project showcases how we approach building evals for the full data lifecycle (not just querying), and highlights a first immediate application: automated skill optimization for a given model and coding harness.

The following picture is a high-level breakdown of the most important parts of a typical customer setup: Bauplan running the data infrastructure, to the right; the user prompting their assistants from their laptop, to the left; the magic box - a coding agent - in the middle: harnesses use an LLM for reasoning and Bauplan APIs as tools.

While it is true that GitLake - Bauplan’s Git-for-data abstraction - allows agents to try and fail safely, blind trials are not the most efficient way to solve data engineering challenges: it is easy to feel productive when spending tokens on dozens of attempts per task. A second look at the stack helps identify our degrees of freedom for an intervention:

• While we could always improve our APIs (simpler, more powerful, etc.), the truth is that Bauplan is already 50x more efficient than legacy lakehouses (check our AI Council talk to see why!);
• We could fine-tune a model (we improve the LLM) or build a custom harness (we improve the outer loop), but our customers have often already picked a coding setup, that for regulatory, financial and even aesthetic reasons suits them, and we would like to meet them where they are as a first move, optimizing around their choices;
• Finally, we could distribute good skills (we improve the context) so that coding agents can still know enough about Bauplan to reduce blind trials.

If we think of coding setups as “compound AI systems”, skills and CLAUDE.md are (in old ML jargon) multi-dimensional hyperparameters for the system: as with all hyperparameters, we need a dataset to represent our (hilly) landscape, and an optimizer to climb it, as depicted in our high-level architectural diagram:

We are happy to share with the community the paper and the code for our “Skill Issues” project (soon to be presented at VLDB), and discuss in detail the interplay between dataset generation and skill optimization.

“Evals are hard”, data edition

Legit and LinkedIn AI experts agree only on a handful of things, “evals are hard” being the fundamental one. LLMs simplify much of the training ops compared to the ML world we grew up in, at the cost of substantially complicating benchmarking: how do we get test cases that include user prompts and desired outputs in the first place, and even if we had them, how do we score a language model response?

Drawing on traces from millions of data workloads in production is certainly the first step in generating a realistic dataset pairing user requests with desired lakehouse outcomes. We pull together data from two sources:

• Bauplan API calls give a rough understanding of which actions are most common among agents on the platform;
• Tracing over a session (i.e. multiple requests by the same agent) allows us to perform “reverse inference”, from the action to the intent: for example, if we see a bunch of branches created at the same time with suffixes such as polars, duckdb and pandas, we may infer that the intention was likely to test out different libraries for the same transformation, and generate a user prompt accordingly.

While traces are a great starting point, they are not sufficient for a thorough evaluation dataset: on the one hand, privacy considerations put a severe limit on how accurate our inference can be, so we may not want to over-index on that; on the other, even rarely called APIs should be represented in our test set, especially if we are trying to make the case that Bauplan skills save tokens by cutting down useless round trips.

The solution to our LLM problem is an LLM itself. On top of our trace analysis, we generate tasks by leveraging the immense common sense and data engineering knowledge embedded in frontier models. We create as much variation as possible by seeding the generator with different personas, whose data needs differ in complexity, level of detail and general Bauplan knowledge (i.e. we want skills to help beginners and experts alike). Following industry best practices, we implement a rejection sampling loop where a first model proposes a task to be included in the final dataset, and then a verification process (comprising a mix of heuristics and LLM judgments) gates the actual inclusion and rejects malformed or substandard proposals.

However, this covers only how to obtain realistic prompts; a dataset must also pair user requests with desired lakehouse outcomes: how do we get those?‍

From DAG code to verification scripts

Imagine that an agent is tasked to solve the following Bauplan task:

Marketing is unhappy with the tips golden table in the lakehouse. Starting from the taxi trip raw data, write a new pipeline that replaces the tips table, showing the number of taxi trips and the average tips by destination ID for the last 30 days. Make sure you exclude trips with negative or absurd tips, say >50% of the total amount (which also needs to be positive!), and merge only when you are satisfied.

You fire up Claude, go get a coffee, come back, and the terminal says “I’m done, all changes have been made”: how would you know if Claude made the required changes and only those? Let’s start with a few things we know for sure:

• We expect the tips table to be dropped and re-created with the new code.
• We expect the number of rows to be approximately the same as the number of destinations (if every destination has at least one qualifying trip, the number of rows will equal the number of unique destination IDs).
• We expect that the count in each row of the tips table is equal to the number of rows returned by the query “SELECT * FROM trip WHERE … AND ID=<destination_id>”.
• We expect a data branch to be created and merged.
• We expect no other table to change in the main branch.

This is not just a lot of checks on data itself; there are a lot of checks on the data lifecycle: how can we check if the table got dropped, and got dropped for the right reason (i.e. as part of a replacement pipeline)? What we are after is the ability to map intent (as expressed in the user request) to the code that operationalizes it, and from there to data changes, and then programmatically and deterministically audit those changes: to the best of our knowledge, Bauplan is the only data system that makes this possible at scale.

If you consider a sample DAG as expressed in Bauplan and the underlying lakehouse operations when that DAG is run, a clear “isomorphism” emerges: knowing the intent (as expressed by the user request), you can put plausible constraints on the resulting DAG code, which in turn translates into verifiable lakehouse operations - thanks to our Git-for-data, operations are immutable and programmatically auditable. 

And if you can plausibly do all of that with a bit of Bauplan knowledge, an LLM can do that as well if nudged properly. In this way, we are able to pair at scale the generation of real-world tasks with corresponding verifiers, i.e. deterministic scripts built with the Bauplan SDK that encode these expectations as precise commands and return a compliance score and a natural-language explanation of any failed checks. 

Git-for-data not only enables destructive tasks to run in parallel on a production lakehouse when running evals, but also provides the foundation for fine-grained verification. Instead of “regexing” a terminal output, or LLM-as-a-judge our way to evaluate a chat, we can keep our agents accountable and avoid the typical “You’re right, I actually didn’t do that” apology:

The optimization results

Once we have a dataset, optimizing hyperparameters follows the usual recipe: divide the tasks into training, validation and test sets, and pick a method for exploring the parameter space. The parameter space of (potentially) large text files is immense: we picked GEPA as our optimizer, from an ever-growing literature of compelling options. The insight is simple: since optimizing a skill has the signature text -> (some thinking) -> text, why not ask an LLM to help us?

Starting with simple skills, GEPA optimizes the target text in a loop, making sure the changes improve the validation score: as the model reflects on possible textual improvements given the task results, it is important for the verifiers to produce scores and traces which are understandable and actionable in the outer optimization loop. 

Starting with six bare-bones skeleton skills to seed the optimization process, we evolve six filled-out skills, including descriptions telling the agent when to use the skills. The main content of the final skills includes detailed workflow descriptions and cautionary notes based on model failures GEPA witnessed during the optimization run, like hallucinated column or table names and incorrect units.

An excerpt from the optimized explore-data skill:

## Workflow

1. **Discover what exists.** List before querying so you use real names:
   ```bash
   bauplan table list --ref <REF>      # --ref only if not the default
   bauplan branch list                 # to find the ref you should query
   ```
   Query a specific ref with `bauplan query "SELECT ..." --ref <REF>`.

2. **Inspect the schema before writing analytics.** Never assume column names,
   types, or units:
   ```bash
   bauplan query "SELECT * FROM <ns>.<table> LIMIT 5" --ref <REF>
   ```
   Note each column's type. Timestamps, `binary`/`blob` text fields, and numeric
   units (ms vs. seconds, cents vs. dollars, micros in a duration) all change your
   SQL. Cast binary text columns explicitly (`CAST(col AS VARCHAR)`) before string
   functions like `LENGTH`.

On average, evolved skills are 27 times longer than the skeletons, and many include links to Bauplan documentation relevant to the tasks. 

Although we used Claude Sonnet 4.6 as the agent and Claude Code as the harness, testing with Mercury 2 and Pi shows that the same optimization process works across different agents and harnesses. When the agent is weaker, the improvement in metrics from the seed skill to the first candidate is steeper.

See you, skill cowboys

How can you get started? Clone the repo and follow the README to set up the environment. If you prefer a video walkthrough, this is Giacomo setting up the project:

The code includes a domain-agnostic generation/optimization engine and the Bauplan-specific use case, so you can instantiate your own domain from the provided template and take advantage of the scaffolding we built and tested. Then use the commands in the Justfile to generate a dataset and optimize your skills on it.

If you are building evals for data systems and want to share notes, feel free to reach out. If you’re around Boston for VLDB at the beginning of September, come out and play: we’ll be hosting a happy hour with Category VC, presenting this work at the conference and discussing our LLM for DataFusion project at the DataFusion meetup. Stay tuned for more research at the intersection of AI and data systems!