With greater than 300% year-on-year utilization development for two consecutive years, Delta Sharing is essentially the most broadly adopted open protocol for Information and AI sharing. Main information suppliers together with SAP, Walmart, Atlassian, and LSEG use Delta Sharing to share information with their companions and prospects throughout clouds and platforms. Right now, we’re excited to announce that Databricks Delta Sharing has first-class help for the Apache Iceberg format.
Information suppliers can now share information securely and dwell from Databricks to any shopper that helps the Apache Iceberg REST Catalog API. Recipients on platforms reminiscent of Snowflake, Trino, Flink, and Spark on all clouds can all use this functionality – including to Delta Sharing’s open ecosystem.
Moreover, we’re launching a Non-public Preview that allows information suppliers to make use of Delta Sharing to share Iceberg tables managed by catalogs exterior Databricks – together with AWS Glue, Hive Metastore, Snowflake Horizon, and extra.
Collectively, you may share any new or present tables (Delta or Iceberg, managed or overseas). This builds in the direction of full open interoperability. You’ll be able to herald Iceberg tables from any exterior catalog, handle them by means of Databricks and Unity Catalog, after which share them out to any recipient—whether or not they’re on Databricks, an Iceberg shopper, or a Delta shopper. This allows you to leverage Unity Catalog as your unified information governance layer, supplying you with one place for all sharing.
On this weblog put up, we’ll clarify why open information sharing is vital. We will even dive into how Delta Sharing to Iceberg purchasers work by means of a hands-on demo.
Why This Issues: Open vs. Closed Sharing
Most information sharing options usually are not actually sharing—they’re trapping. They’re basically closed and engineered to make sure vendor lock-in, so that you solely get to share with others who’re already inside their closed ecosystems. This limits your choices, stifles innovation, and drives huge, pointless information replication.
Delta Sharing is essentially the most broadly adopted open customary for safe information sharing. Utilized by category-leading information suppliers, it’s constructed to help totally different clouds and platforms. Delta Sharing operates on three core rules:
- Share any asset.
- Share with anybody.
- Share with none friction.
Including Iceberg shopper help strengthens this dedication. It enables you to share a Delta desk whereas recipients expertise it as a local Iceberg desk. Sharing occurs over the Iceberg REST API, so recipients can join from any Iceberg-compatible platform. This lets you get the most effective of each worlds: Information suppliers profit from superior Delta Sharing options reminiscent of View sharing, whereas recipients obtain native Iceberg tables through the Iceberg REST API.
Recipients get safe, dwell entry to the supply information. This eliminates silos and allows you to share information brazenly with anybody.
This characteristic is right for organizations that must share information externally with companions and prospects utilizing Iceberg purchasers, reminiscent of these working on Snowflake or integrating with platforms like Trino, Flink, or Spark. Firms with a number of enterprise models working throughout a number of platforms additionally profit by unblocking seamless, bi-directional information alternate in multi-cloud or hybrid environments. Industries already leveraging these patterns embrace healthcare, retail, finance, ad-tech, and extra.
Interoperability: Each Supply and Vacation spot
As a result of we consider in full open information entry, we do not cease at sharing information to Iceberg purchasers. We at the moment are growing the following evolution: sharing overseas Iceberg tables that reside in exterior catalogs reminiscent of AWS Glue or Snowflake Horizon. We’re excited to announce the Non-public Preview of Delta Sharing help of overseas Iceberg tables.
You may ask: Why share an Iceberg desk by means of Delta Sharing if it resides in AWS Glue or Snowflake? Why not share instantly from inside that platform?
First, by cataloging your exterior Iceberg information in Unity Catalog, you get a unified governance layer in Unity Catalog, permitting you to get full visibility and governance throughout your information property. Moreover, utilizing Delta Sharing means that you can get the better of each worlds: You profit from Delta Sharing best-in-class sharing functionalities, whereas retaining your information in Iceberg format. This contains for instance, the flexibility to Delta Share Views for fine-grained entry management, which isn’t supported natively by the Iceberg IRC API.
With this Non-public Preview, the Databricks Lakehouse is open in each instructions. Your Lakehouse can share information to and obtain information from the rising Iceberg ecosystem.
This twin means offers you:
- Easy Collaboration: Work collectively irrespective of which open desk format you utilize (Delta or Iceberg).
- Ruled Sharing: Unity Catalog controls entry and offers audit logs.
- Broadest Attain: Share information as each a supplier and a recipient, breaking platform partitions.
How Does It Work?
Think about your organization, Supplier Corp, makes use of Databricks and Delta Lake to handle buyer information. That you must securely share a each day listing of product gross sales with Associate Inc, which makes use of Snowflake and prefers the Iceberg format.
Earlier than this characteristic: Supplier Corp must manually export the information, remodel it right into a Snowflake-readable format, add it to the accomplice’s cloud storage, and arrange a posh synchronization job. That is sluggish, pricey, includes important admin overhead, and dangers information changing into outdated.
With Delta Sharing to Iceberg Shoppers:
- Supplier Corp permits Iceberg reads on gross sales information through UniForm (this could embrace managed and exterior Delta tables, views, materialized views, and streaming tables), and shares it through Delta Sharing. This offers dwell entry with no duplication or re-ingestion required.
- Associate Inc units up a easy connection in Snowflake utilizing the offered credentials for safe authentication through short-lived bearer tokens.
- Associate Inc’s analysts can instantly question the shared desk utilizing customary SQL, treating it like a local Iceberg desk of their Snowflake atmosphere.
- The info they see is at all times dwell (zero-copy), and Supplier Corp maintains full safety and governance with auditing and monitoring utilizing Unity Catalog.
This makes information sharing instantaneous, protected, and fully format-agnostic.