Databricks Expands Data Connectivity With New Salesforce, Workday Links

The new connectors for Salesforce and Workday applications, part of the company’s Lakeflow system, will automate and accelerate the development tasks needed to pull operational data into the Databricks platform for data analysis and AI applications.

Data lake platform provider Databricks today released new connectors that feed the company’s Databricks Lakeflow data engineering system with operational data from Salesforce and Workday cloud applications.

The new managed connectors, now generally available, are the first to be released since the launch of Databricks Lakeflow, the company’s unified data engineering system, at the company’s Data + AI Summit in June 2024.

As businesses and organizations expand their data analytics operations and, more recently, crank up their artificial intelligence and generative AI applications, the demand for operational data to feed those systems is rapidly increasing.

“All data journeys start with data engineering,” said Michael Armbrust, a Databricks distinguished software engineer, in an interview with CRN. “Data is always partial, it needs to be filtered, it needs to be aggregated, joined – all that stuff before it’s actually ready for consumption.”

Lakeflow combines all aspects of data engineering including Lakeflow Connect for data ingestion – collecting data from operational systems such as ERP and CRM applications and moving it into the Databricks Data Intelligence Platform – as well as Databricks DLT for data transformation and Databricks Workflows for data orchestration.

Developing data connections to operational systems can be a programming chore. “It’s very difficult for data teams to build and manage these ingestion pipelines, because each API is unique, [software vendors] are often making breaking changes to them, there are [data transfer] rate limits and other things you need to deal with,” Armbrust said.

“What we found was, teams were spending a lot of time building, optimizing and maintaining this patchwork of pipelines. And that was slowing down their projects and creating kind of a fragmented ETL [extract, transform and load] architecture that was difficult for them to govern … This is undifferentiated heavy lifting, spending time maintaining these pipelines instead of doing what these highly skilled data engineers want to be doing, which is getting value from this data for their organization,” he said.

Lakeflow connectors speed the process of building, optimizing and maintaining production-ready data pipelines by automating data mapping, data reliability management and API management chores that can take huge amounts of time when done manually.

“The true value of Databricks is when it becomes a center of gravity for all data, when you can combine operational data with the analytical data. That's what Lakeflow Connect is allowing,” Armbrust said.

For Databricks partners, including solution providers and systems integrators, Lakeflow Connectors will reduce the amount of up-front development work needed to manually build data connections and focus on higher level services.

“If you're a systems integrator, you can now focus on the stuff that's actually bringing value to your customers, including domain-specific integration of this data with the other data that's already inside of Databricks,” Armbrust said.

The connectors for Salesforce and Workday systems will automate the process of pulling operational financial, sales, marketing and human resource management data, Armbrust said. The connectors are already being used by a number of Databricks customers including Porsche, Insulet and Flick, according to a company blog post.

The Salesforce and Workday connectors are the first of what’s expected to be a growing set of connectors for operational systems – especially software-as-a-service applications. “Our plan is to continue to roll out connectors as part of Lakeflow Connect. These are just the first two,” Armbrust said, without disclosing a specific timeline for other connectors.