If You Build Them: Databricks To Launch New Data Workflow, AI Agent Development Tools
Lakeflow Designer and Agent Bricks technology unveilings for building data pipeline workflows and AI agents, respectively, are on tap at Wednesday’s Databricks Data + Summit.
With new technologies for constructing data pipelines and building AI agents, attendees at this week’s Databricks Data + AI Summit can be forgiven if they feel like they should be wearing hard hats.
Data and AI platform provider Databricks Wednesday launched Agent Bricks, a unified workspace for building production-scale AI agents, and Lakeflow Designer, a no-code tool that data analysts can use to build reliable data pipelines.
Agent Bricks and Lakeflow Designer top the announcements coming out of the Data + AI Summit in San Francisco as Databricks looks to maintain its momentum as one of the IT industry’s fastest-growing companies and one of the most influential in the critical juncture of big data and AI.
[Related: Meeting The Data Needs Of The AI World: The 2025 CRN Big Data 100]
“There’s a lot of pressure for organizations to scale their AI efforts. Getting high-quality data to the right places accelerates the path to building intelligent applications,” said Ali Ghodsi, Databricks co-founder and CEO, in the Lakeflow Designer announcement—a comment that could also serve more broadly as the company’s mission statement.
Ghodsi is scheduled to deliver the Data + AI Summit keynote Wednesday morning, joined by Anthropic co-founder and CEO Dario Amodei, JPMorgan Chase CEO Jamie Dimon, and Nikita Shamgunov, CEO of Postgres database startup Neon that Databricks acquired in May. Microsoft Chairman and CEO Satya Nadella is also scheduled to deliver a recorded address.
While Databricks is seen as a strong candidate for an IPO, it remains privately held and does not disclose details about its finances. But in December the company raised $10 billion in a funding round that put its value at $62 billion and at the time said it expected to surpass an annual revenue run rate of $3 billion shortly and achieve positive cash flow. The $10 billion funding round increased to $15 billion in January with the addition of a $5.25 billion credit facility.
At the core of Databricks’ momentum is the company’s flagship Databricks Data Intelligence Platform, a unified data and AI platform built on a lakehouse architecture and powered by the company’s Data Intelligence Engine. The system offers a range of capabilities including data analytics, data integration, data catalog, data governance and security, and more.
The company continues to expand the platform’s functionality: The Neon acquisition, for example, added serverless Postgres database capabilities to the Data Intelligence Platform for developers and for AI agents.
And that’s the driver behind Wednesday’s Lakeflow Designer and Agent Bricks announcements, both being demonstrated at the Data + AI Summit event.
AI Agent Development
Agent Bricks is a new unified workspace for building AI agents that Databricks says works with an organization’s unique data to achieve “production-level accuracy and cost efficiency.”
Agent Bricks builds on technology Databricks acquired when it bought generative AI startup MosaicML in June 2023 for $1.3 billion. Last year Databricks unveiled Mosaic AI Agent Framework for building AI agents on the Databricks platform.
“Databricks’ strategy is data intelligence. How do we build AI systems that can reason over your enterprise data on your enterprise tasks? Agent Bricks is really a personification of that,” said Hanlin Tang, Mosaic co-founder and now Databricks CTO for neural networks, in an interview with CRN.
“I think for this Agent Bricks product, we really took a step back and rethought— and really took a new approach to—how we think the world really should be building these agents,” Tang said.
Tang said Agent Bricks is designed to overcome several common problems around AI agents and agent development, including organizations’ lack of enough data to build agents and difficulty evaluating how well they are working once in production.
And another major challenge: “The industry has definitely evolved from just using models to building entire agent systems that have tools, or vector databases, or all these sorts of different components,” Tang said. “And then suddenly, there is an explosion of choices. What model do you use? How do you string these things together? What kind of agent workflow should you use?”
Agent Bricks offers an automated way to create high-performing AI agents tailored to a business, according to a preview of the announcement provided to CRN. Developers provide a high-level description of the agent’s task and connect it to enterprise data. Agent Bricks “automatically generates task-specific evaluations and LLM judges to assess quality” and creates synthetic data to substantially supplement the agent’s learning. Agent Bricks then searches across “the full gamut of optimization techniques” to refine the agent, according to Databricks.
Agents developed using Agent Bricks can be used for a number of common use cases, including information extraction, knowledge assistant, and custom LLM agents for such tasks as summarization, classification or rewriting within specific industries.
Tang said Agent Bricks can be used by Databricks partners who provide AI development services for customers, including developing AI agents for clients.
Agent Bricks is currently available in beta. Databricks is also announcing new AI features for Mosaic AI including support for serverless GPUs (available in beta) and the MLflow 3.0 unified platform for managing the AI life cycle (now generally available).
Lakeflow Designer
The new Lakeflow Designer provides nontechnical data analysts with a no-code data ETL (extract, transform and load) approach to building reliable, production-grade data pipelines, according to Databricks.
Data pipelines are traditionally built by data engineering teams. But with such projects often facing backlogs, data analysts sometimes take on the job themselves using low-code/no-code tools that Databricks says can sacrifice data governance, and system reliability and scalability.
“There is this shadow data engineering that’s happening outside of Databricks, and it’s typically in the nontechnical section of [customers]—roles like business analysts, operations analysts, just nontechnical people,” said Bilal Aslam, Databricks senior director of product management, in an interview with CRN. “Their job isn’t to write code. Their job isn’t to be sitting in Databricks every day. What these people are doing, to get their jobs done, to get fresh, clean data, they’re using tools that are typically in the no-code or low-code category, and they’re doing their own data preparation.
“And this sort of creates this shadow data engineering,” Aslam said. “You end up solving this real [data pipeline development] problem, it's a multibillion-dollar problem, but with tools that are essentially limited, there’s a dead end … because these tools don’t integrate back into the data intelligence platform.”
Lakeflow Designer, which will be in preview later this summer, uses a visual drag-and-drop user interface and natural language generative AI assistant to create data pipelines with the same scalability, governance and maintainability as pipelines built by code-first data engineers using more sophisticated tools, according to Databricks.
While targeted toward nontechnical users, Lakeflow Designer is backed by Databricks Lakeflow, the company’s data pipeline development platform for technical data engineering teams that was unveiled in June 2024. Databricks is unveiling the general availability of Lakeflow at Data + AI Summit.
Aslam said Lakeflow Designer is “an accelerant” for solution provider and systems integrator partners looking to close deals because it opens up the range of use cases for the Databricks platform, and data preparation projects are no longer slowed by the lack of data.
“We’re getting very positive feedback from [systems integrator] partners on this capability because it simplifies this bottleneck in their individual [data preparation] processes,” he said.
The company is also unveiling several new Lakeflow capabilities, available in private preview, including a new IDE (integrated development environment) for data engineering and new data ingestion connectors for Google Analytics, ServiceNow, Microsoft SQL Server, SharePoint, PostgreSQL and Secure File Transfer Protocol. Lakeflow also can now direct-write to the Databricks Unity Catalog using Zerobus.
