Blog - Serverless Data & AI Engineering Insights

Latest From Our Blog

Python, SQL, Iceberg, version control, and superb DevEx.

Grit beats talent (if the infrastructure lets you fail)

Full warehouse migration in a single prompt forty-minute task with safe iteration on production data.

March 26, 2026

Ciro Greco

Introducing Bauplan Skills: Safe automation for AI on your data

Let AI agents ship data safely.

February 25, 2026

Ciro Greco

Trustworthy AI in the Agentic Lakehouse: from Concurrency to Governance

Solving infrastructure first: git-for-data branching and concurrency for AI agents

December 15, 2025

Jacopo Tagliabue, Federico Bianchi, Ciro Greco

From Humans to Agents: Field Notes from the SAO Workshop

AI Agents and Data Systems at the ACM Conference on AI and Agentic Systems (CAIS), May 26, 2026

June 10, 2026

Ciro Greco

Research

Estuary + Bauplan: CDC with branches

Building a robust CDC pipeline into an Iceberg lakehouse

April 1, 2026

Ciro Greco

Engineering

Building a Correct-by-Design Lakehouse

Pre-print on Arxiv

March 19, 2026

Weiming Sheng, Jinlang Wang, Manuel Barros, Aldrin Montana, Jacopo Tagliabue, Luca Bigon

Research

AI for Distributed Systems Design: Scalable Cloud Optimization Through Repeated LLMs Sampling And Simulators

Presented at AAAI 26

March 19, 2026

Jacopo Tagliabue

Research

Querying Everything Everywhere All at Once: Supervaluationism for the Agentic Lakehouse

Pre-print on Arxiv

March 19, 2026

Jacopo Tagliabue

Reference

We solved trust for AI Agents in 1973 (we just forgot)

What SQL databases already know about isolation, concurrency, and trust

January 15, 2026

Ciro Greco, Jacopo Tagliabue and Federico Bianchi (Together AI)

AI Agents

Engineering

Bauplan. A year in review

And our (Bau)plan for the next one

December 30, 2025

Ciro Greco, Jacopo Tagliabue and Mattia Pavoni

Announcements

Rethinking Data Pipelines in Python

Eliminating cross-runtime complexity in your data stacks

December 5, 2025

Ciro Greco

Engineering

Safe, Untrusted Agents for Big Data Infrastructures

Our newly accepted paper at S2AI@BigData 2025

November 20, 2025

Jacopo Tagliabue, Ciro Greco

Research

Duck Hunt: moving Bauplan from DuckDB to DataFusion

Arrow-Native and Community-Driven: Why We Are Excited to Be Part of the DataFusion Ecosystem

November 5, 2025

Jacopo Tagliabue

Engineering

Shift left with Iceberg - Bauplan & Trust & Will

A lakehouse migration story

October 30, 2025

Ciro Greco

Case Study

Git-for-data: Formal Semantics Part 2: Branching, Merging, and Rollbacks

Bringing Database-level Guarantees to the Lakehouse

October 24, 2025

Ciro Greco

Engineering

Your First Lakehouse

When a platform becomes inevitable

October 1, 2025

Ciro Greco

Engineering

Data Engineering is Becoming Software Engineering (And that’s a good thing)

How AI-driven applications are forcing data teams to adopt software engineering best practices in data engineering

September 24, 2025

Bauplan Team

Engineering

Write‑Audit‑Publish: Ship Data Safely, Move Faster

Bring Git Workflows to Your Data

September 24, 2025

Bauplan Team

Engineering

Data Engineering and Automation in the Era of Agents

How Git-for-data branching and unified APIs enable AI agents to safely automate the repetitive mechanics of data engineering

September 15, 2025

Ciro Greco

AI Agents

Bauplan’s MCP Server

The First Step Towards the Agentic Lakehouse

August 25, 2025

Ciro Greco

Announcements

Git for Data: Formal Semantics of Branching, Merging, and Rollbacks (Part 1)

How formal methods help ensure safe, reproducible workflows in data lakehouses

July 17, 2025

Ciro Greco and Jacopo Tagliabue

Engineering

Data engineer agents

From Prompt to Pipeline: Cloud-Native Agents for Data Transformation and ETL

July 4, 2025

Ciro Greco, Jacopo Tagliabue and Federico Bianchi

AI Agents

From Postgres to your first data lakehouse

From transactional database to analytics powerhouse: A practical guide to CDC and Iceberg

June 17, 2025

Ciro Greco

Case Study

Everything-as-Python

From notebook to prod with Bauplan and marimo

June 11, 2025

Ciro Greco

Engineering

Eudoxia: a FaaS scheduling simulator for the composable lakehouse

Proceedings of Workshops at the 51st International Conference on Very Large Data Bases (VLDB 2025)

May 19, 2025

Ciro Greco, Jacopo Tagliabue and Tapan Srivastava

Research

Zerrow: True Zero-Copy Arrow Pipelines in Bauplan

Pre-print on Arxiv

May 13, 2025

Yifan Dai, Jacopo Tagliabue, Andrea Arpaci-Dusseau, Remzi Arpaci-Dusseau, Tyler R. Caraza-Harter

Research

Python Over Data Lakes

Functions, Declarative Environments, Data Management And Other Things With Feathers

May 5, 2025

Ciro Greco

Engineering

Hello Bauplan

Bauplan is a serverless data platform that treats pipelines, models, and tables like software — versioned, testable, and ready for agents.

April 16, 2025

Ciro Greco

Engineering

Announcing Bauplan's seed round

Announcing $7.5M seed round led by Innovation Endeavors

April 16, 2025

Bauplan Team

Announcements

Trading Airflow+EMR for Temporal+Bauplan: The Mediaset Tale

From legacy sprawl to lightning-fast pipelines: how Mediaset rebuilt their data stack with Bauplan and Temporal—and cut dashboard refresh times from 60 minutes to 5.

April 9, 2025

Jacopo Tagliabue and Marco Reni

Case Study

Full-Stack RAG in 300 Lines of Python

End-to-end RAG system for a conversational service agent

March 26, 2025

Ciro Greco

Reference

Data as software and AI for the 99%

Build simple, robust data apps with software engineering principles.

March 12, 2025

Ciro Greco

Engineering

Blending DuckDB and Iceberg for Optimal Cloud OLAP

Lessons learned crafting a Serverless Lakehouse from spare parts

January 13, 2025

Nathan LeClaire

Engineering

Embedding-based Recommender Systems with Bauplan and MongoDB

Full-stack recommender system with Bauplan for data preparation and training, and MongoDB Atlas for real-time inference.

December 20, 2024

Ciro Greco

Reference

Bauplan: Zero-copy, Scale-up FaaS for Data Pipelines

Paper presented at WoSC10 2024. In collaboration with The University of Wisconsin.

December 2, 2024

Ciro Greco, Jacopo Tagliabue and Tyler Caraza-Harter

Research

Build, Ship and Run Containers is Too Slow for Python.

Making the experience of running data workflow in the cloud indistinguishable from doing it locally.

November 22, 2024

Ciro Greco and Nathan LeClaire

Engineering

Ephemeral graphs for data DAGs

DAG planning using an in-memory graph database. In collaboration with Kùzu

November 20, 2024

Jacopo Tagliabue and Luca Bigon

Engineering

FaaS and Furious: differential caching for efficient data pre-processing

Paper presented at DEMAI@IEEE Big Data 2024.

November 12, 2024

Ciro Greco, Jacopo Tagliabue and Ryan Curtin

Research

Prefect on the lakehouse: write-audit-publish pattern with Bauplan

A reference implementation to implement a Write-Audit-Publish (WAP) pattern with Bauplan and Prefect 3.0.

October 21, 2024

Ciro Greco, Jacopo Tagliabue and Chris White

Reference

To serverless or not to serverless

Find the right balance between cost control and fast startup time for your Spark clusters.

September 18, 2024

Ciro Greco

Engineering

Reproducible data science over data lakes: data pipelines with Bauplan and Nessie

Paper presented at SIGMOD/PODS 2024. Awarded best presentation DEEM@SIGMOD.

June 9, 2024

Ciro Greco and Jacopo Tagliabue

Research

Write-Audit-Publish for Data Lakes in Pure Python (no JVM)

An open source implementation of WAP using Apache Iceberg, Lambdas, and Nessie all running entirely Python.

April 11, 2024

Ciro Greco

Open Source

Why we develop on data locally and why we should stop - Part 2

Working on production data is the only way to know whether our applications will work.

March 1, 2024

Ciro Greco

Engineering

Why we develop on data locally and why we should stop - Part 1

Why production cloud environment are too slow and hard to develop in them.

November 27, 2023

Ciro Greco

Engineering

Serverless is still not designed for data

The greatest invention since sliced Virtual Machines.

September 6, 2023

Ciro Greco and Jacopo Tagliabue

Engineering

Building a serverless Data Lakehouse from spare parts

Paper presented at VLDB 2023.

August 10, 2023

Ciro Greco, Jacopo Tagliabue and Luca Bigon

Research

A Serverless Query Engine from Spare Parts

An open-source implementation of a Data Lake with DuckDB and AWS Lambdas.

June 4, 2023

Ciro Greco

Open Source