Right this moment marks an necessary milestone for open requirements and the broader knowledge neighborhood. We’re working with Microsoft to combine the info managed each in Unity Catalog and OneLake utilizing open requirements. The Unity Catalog Open API specification offers an interface to trade knowledge each methods, zero copy, to open up entry to the complete lakehouse ecosystem for engines in both platform.
For practically a decade, Databricks and Microsoft have partnered to assist organizations unlock the worth of their knowledge by the openness and suppleness of the lakehouse. Right this moment, that partnership takes one other step ahead, giving clients extra selection in how they entry and govern knowledge throughout platforms.
The Path to Open Interoperability
As open desk codecs like Delta Lake and Iceberg gained adoption, new silos emerged on the catalog-level. To deal with this, Databricks launched Unity Catalog Open APIs, establishing an open commonplace for unified governance throughout knowledge and AI, whereas additionally totally supporting the Iceberg REST catalog API for tabular knowledge. This dual-API strategy empowers organizations to undertake best-in-class options and keep away from vendor lock-in whereas seamlessly sharing knowledge throughout many knowledge platforms and SaaS Purposes. We additionally launched catalog federation to supply a single ruled view throughout all knowledge platforms. This mannequin, open APIs powering catalog federation, has change into the business commonplace for interoperability.
Right this moment, we’re excited to see this open imaginative and prescient validated by our companions at Microsoft.
Microsoft provides assist for Unity Catalog Open APIs
As we introduced with our companions at Microsoft right this moment, our collaboration is deepening.
Microsoft has already added assist for studying from Azure Databricks with OneLake Mirroring, permitting Material engines to entry all the knowledge managed in Unity Catalog seamlessly. In contrast to different OneLake integrations with proprietary platforms, this makes all knowledge in Unity Catalog – together with the best efficiency tables – accessible in OneLake with out copying, altering code, or further prices.
Right this moment, Microsoft and Databricks are extending their openness by integrating much more natively. With Microsoft OneLake including assist for the Unity Catalog Open API commonplace, Azure Databricks customers will have the ability to seamlessly learn knowledge from OneLake.
With OneLake including assist for open catalog APIs together with Iceberg REST and Unity Catalog Open APIs, Microsoft is making OneLake structured and unstructured knowledge from providers like Dynamics, Dataverse, Blob retailer, and many others. natively accessible to the complete open lakehouse ecosystem, together with Databricks.
Databricks Builds on the Open Basis
Leveraging this expanded functionality, Databricks is including assist to make all OneLake knowledge seamlessly accessible in Unity Catalog.
This allows zero-copy studying of OneLake knowledge from inside Databricks making certain seamless interoperability and entry throughout each platforms, giving Databricks clients entry to all their knowledge within the Microsoft ecosystem to energy their most superior AI, machine studying, and large-scale knowledge engineering workloads.
Catalog Integration
- Studying from Unity Catalog in Material – Usually Out there Now
Microsoft has launched Azure Databricks mirroring. Clients can mirror Azure Databricks knowledge into OneLake, by Unity Catalog making certain that ALL knowledge managed by Azure Databricks is immediately accessible throughout Microsoft Material workloads. Each platforms can work over the identical copy of information with no knowledge motion. - Studying from OneLake in Azure Databricks – EOY 2025
This new functionality, allows Azure Databricks to learn knowledge from OneLake through their Unity Catalog Open APIs and permits your entire OneLake knowledge to be leveraged seamlessly in Databricks. Clients can create Unity Catalog Connections to OneLake based mostly on Managed Identities from EntraID, and might seamlessly mount knowledge objects from any Material workspace, immediately as a catalog in Unity.
Connecting to OneLake knowledge in Azure Databricks
The Future is Open
This milestone displays how far the business has come towards an open, interoperable knowledge ecosystem. Clients not want to decide on between innovation and interoperability—they’ll have each.
With Microsoft now embracing the open catalog commonplace, we’re excited to proceed constructing the way forward for knowledge and AI on this basis. Keep tuned for extra bulletins and roadmap updates, together with assist for Azure Databricks writing knowledge immediately in OneLake, as we deepen our integration.