How Kythera Labs, a Databricks Constructed-On Associate, saves $2M+/12 months utilizing Delta Sharing

Healthcare methods generate monumental quantities of delicate information, however transferring, sharing, and analyzing that information securely throughout organizations continues to be a significant problem. On this submit, we’ll take a look at how we at Kythera Labs use Databricks and Delta Sharing to handle greater than 300 million affected person data and help collaborations throughout healthcare and life sciences. The weblog will cowl the sensible points with older information‑sharing strategies, why we adopted Delta Sharing, and the affect it’s had on our storage prices, effectivity, and actual‑time collaboration.

Making Information Work in Healthcare: Kythera’s Strategy

Kythera Labs is an information expertise firm that empowers healthcare and life sciences organizations with a unified, high-fidelity healthcare information platform for evaluation. As a built-on Databricks Associate, we selected Databricks and Delta Sharing not only for inner information sharing but additionally to help seamless information trade with exterior companions. At this time, greater than 80% of our clients use merchandise constructed on the platform. We additionally help exterior collaborations, together with organizations like Actual Sciences, utilizing Delta Sharing throughout 50 lively buyer workspaces.

Why Delta Sharing?

Kythera Labs selected Delta Sharing to beat vital challenges in securely sharing healthcare information. With over 300 million affected person data spanning a decade of medical historical past, conventional strategies required creating and transferring a number of full copies of datasets, driving storage prices into the lots of of hundreds of {dollars} and slowing supply.

Delta Sharing adjustments that by enabling safe, actual‑time entry to dwell information with out creating duplicate copies. As an alternative of storing and sustaining separate datasets for every companion or atmosphere, we are able to share a single, ruled supply of reality straight. This method has allowed us to energy inner groups and exterior collaborations with simply 3.5 PB of storage, reasonably than the 20‑plus PB in any other case required.

One other complexity is assembly our clients the place they’re on the cloud. Healthcare suppliers typically function in Azure, whereas many pharmaceutical corporations run on AWS or GCP. With no expertise like Delta Sharing, delivering massive datasets throughout clouds would imply expensive transfers, advanced ETL work, and a number of stale copies scattered throughout clouds. With Delta Sharing, we are able to immediately present safe entry to the identical dwell dataset — regardless of the cloud — whereas sustaining compliance and eliminating pointless copies.

This not solely streamlines our inner workflows (transferring from growth to testing to manufacturing with out re‑copying information) but additionally makes it straightforward for purchasers to behave sooner, like immediately updating a most cancers therapy mannequin with the latest information.

Changing Legacy Approaches

Given the exponential development in information quantity and complexity, conventional information sharing strategies like SFTP servers are not viable for contemporary wants. Transferring massive recordsdata backwards and forwards introduces delays, provides safety dangers, and requires storage of a number of redundant datasets.

Whereas APIs may very well be a useful resource, they’re inadequate for sharing the huge oceans of knowledge that organizations like Kythera handle. Counting on APIs to share the immense volumes of knowledge we handle can be like attempting to fill a swimming pool with a backyard hose—it’s technically doable, however too sluggish and inefficient for our wants.

Operationally, we deal with 7–10 million transactions each day whereas guaranteeing compliance via our customized “Vault Structure” constructed on Delta Sharing. Prospects profit from real-time updates through view sharing with out guide intervention.

By adopting Delta Sharing, we’ve fully moved away from these legacy strategies and gained operational effectivity whereas enabling seamless collaboration throughout clouds and organizations.

Delta Sharing ROI

Delta Sharing has allowed us to get rid of legacy data-sharing strategies, lower storage wants by over 80%, and save greater than $2 million within the final 2 years. — Jeff McDonald, CEO, Kythera Labs

Delta Sharing helped Kythera lower storage wants from a projected 24 PB to only 3.5 PB. Over three years, storage demand dropped from 17 PB/month in 2024 to 12 PB/month in 2023 and 6 PB/month in 2022. These reductions add as much as thousands and thousands in financial savings. For context, massive pharmaceutical corporations can spend as a lot as $14 million every month simply on storage.

Storage is simply a part of the story. The compute prices for performing the ETL copies may very well be much more vital, starting from equal to the storage financial savings to probably many occasions better, relying on the use instances.

Key Takeaways

Delta Sharing has reworked our data-sharing capabilities by decreasing prices, enhancing effectivity, and enabling real-time collaboration throughout clouds and organizations. The mix of Delta Sharing, Unity Catalog, and liquid clustering ensures scalability whereas sustaining compliance with healthcare information requirements, exemplifying how open, trendy information platforms can revolutionize healthcare analytics.