OpenXData - Data as Software and the programmable lakehouse

Two very big things happened in recent years that completely transformed the data landscape:

• Pre-trained AI models: These models have democratized AI, enabling software engineers to integrate advanced capabilities into applications with simple API calls and without extensive machine learning expertise. ​
• Lakehouses: Open formats over object storage bring together the flexibility of data lakes and the data management strengths of data warehouses, offering a more streamlined and scalable approach to data management.

And yet, most data platforms remain difficult to digest for traditional software developers.In his talk, Ciro Greco, CEO and cofounder of Bauplan, introduces "Data as Software," a practical approach to data engineering and AI that leverages the lakehouse architecture to simplify data platforms for developers. By leveraging serverless functions as a runtime and Git-based workflows in the data catalog, we can build systems that makes it exponentially simpler for data developers to apply familiar software engineering concepts to data, such as modular, reusable code, automated testing (TDD), continuous integration (CI/CD), and version control.