We’re excited to introduce a number of highly effective new capabilities to Mosaic AI Gateway, designed to assist our clients speed up their AI initiatives with even better simplicity, safety, and governance.
As enterprises race to implement AI options, managing safety, compliance, and prices has turn into more and more difficult. That’s why we launched Mosaic AI Gateway final yr, now utilized by many organizations, to handle AI visitors for a variety of fashions, together with OpenAI GPT, Anthropic Claude, and Meta Llama fashions.
At this time’s replace introduces superior options for utilization monitoring, payload logging, and guardrails, enabling enterprises to use safety and governance to any AI mannequin inside the Databricks Knowledge Intelligence Platform. With this launch, the Mosaic AI Gateway now supplies production-grade safety and governance, even for essentially the most delicate knowledge and visitors.
“Mosaic AI Gateway is offering us with a safe technique to devour AI fashions and join them to our proprietary knowledge. This permits us to construct safe, compliant, and context-aware AI programs, enhancing our productiveness and serving to us fulfill our mission to ship superior healthcare providers to all.”
— Kapil Ashar, Vice President, Enterprise Knowledge and Medical Platform, Accolade
What’s Mosaic AI Gateway?
Many enterprises combine and match a number of AI fashions from totally different suppliers to construct compound AI programs (e.g., RAG, multi-agent architectures) that obtain the standard wanted to deploy GenAI purposes into manufacturing. Nonetheless, as enterprises combine a various array of open and proprietary fashions, they encounter challenges with operational inefficiencies, price overruns, and potential safety dangers.
Mosaic AI Gateway addresses these challenges by offering a unified service to entry, handle, and safe AI visitors. It permits enterprise directors to implement guardrails and monitor AI utilization whereas providing builders a easy interface to shortly experiment, mix, and safely deploy purposes into manufacturing. Corporations like OMV and Edmunds, which have adopted Mosaic AI Gateway, have been in a position to speed up their AI initiatives extra successfully than their friends, all whereas sustaining compliance, safety, and operational effectivity.
What are you able to do with Mosaic AI Gateway?
Securely entry any AI fashions
Mosaic AI Gateway simplifies entry to any LLM by way of a single interface (API, SDK, SQL), considerably decreasing each improvement time and integration prices. You possibly can simply swap between proprietary and open fashions with out adjustments to the consumer app. What units Mosaic AI Gateway aside is its help for all AI property—conventional fashions, GenAI fashions, chains, and brokers—eliminating the necessity for a number of programs. It additionally allows routing and visitors splitting between fashions for A/B testing or distributing workloads throughout suppliers to deal with excessive demand.
“With Mosaic AI Gateway, we had been in a position to confidently experiment with a wide range of open and proprietary AI fashions, accelerating innovation whereas making certain regulatory compliance. This allowed us to combine a number of GenAI apps, decreasing info search time and enhancing data-driven decision-making.”
— Harisyam Manda, Senior Knowledge Scientist at OMV
Monitor and Debug Manufacturing AI Visitors
Mosaic AI Gateway now captures utilization and payload knowledge for all visitors flowing out and in of endpoints to Unity Catalog Delta Tables. We’re introducing two key tables:
- Endpoint Utilization Desk: This technique desk logs each request throughout all serving endpoints within the account, together with requester particulars, utilization statistics, and customized metadata. This knowledge helps optimize spending and maximize ROI—for instance, by setting fee limits on experimental endpoints or working with suppliers to extend quotas for manufacturing endpoints.
- Inference Desk: This desk constantly captures uncooked inputs, outputs, HTTP standing codes, and latency for every serving endpoint. You need to use this knowledge to observe AI app high quality, debug points, and even as a coaching corpus to fine-tune AI fashions.
The perfect half? All knowledge is captured in Unity Catalog, making it simple to securely share, search, visualize, and analyze utilizing acquainted knowledge instruments. For instance, you possibly can mix these tables with different tables, resembling label knowledge or enterprise metrics, to carry out customized evaluation or construct dashboards tailor-made to your enterprise wants.
“Mosaic AI Gateway allows us to securely devour any LLMs, be it OpenAI or different fashions hosted on Databricks, whereas making certain LLM visitors is correctly ruled and tracked. This has democratized GenAI, permitting us to deploy new use instances like a customer support bot that has improved buyer satisfaction and gross sales.”
— Eduardo Fernández Carrión, Lead Knowledge Scientist & AI Engineer at Santalucia Seguros
Constantly Safeguard Customers and Purposes
Mosaic AI Gateway contains complete guardrails to safe visitors to any mannequin API, imposing security insurance policies and defending delicate info in real-time. These guardrails embrace:
- Security Filtering: Filters dangerous content material resembling hate speech, insults, sexual content material, violence, misconduct, and different classes.
- PII Filters: Detects and blocks requests containing delicate content material resembling personally identifiable info (PII) in person inputs.
- Key phrase Filters: Blocks undesirable matters in your purposes to make sure secure, related interactions aligned with your enterprise and insurance policies.
- Matter Filters: Retains your software centered on its educated scope by avoiding responses to unrelated or dangerous matters, minimizing legal responsibility.
These guardrails might be set on the endpoint or request ranges to suit particular use instances and insurance policies. All knowledge is logged in Inference Tables, which might then be analyzed with Lakehouse Monitoring to trace mannequin security over time.
“Guardrails helps us forestall unsafe content material from reaching our finish person. With payload logging, we are able to additionally hint guardrails to trace efficiency of the applying.”
— Ryan Jockers, Assistant Director at North Dakota College System
Join AI with Your Knowledge Effortlessly
Mosaic AI Gateway is constructed on Databricks Knowledge Intelligence Platform, enabling enterprises to simply join LLMs to their knowledge utilizing methods like RAG, agent workflows, and fine-tuning, serving to remodel common intelligence into actionable knowledge intelligence. That is already in motion—many shoppers are utilizing Mosaic AI’s Vector Search with exterior fashions to create embeddings or leveraging agent frameworks and analysis instruments to construct brokers with exterior fashions.
“Databricks Mannequin Serving is accelerating our AI-driven tasks by making it simple to securely entry and handle a number of SaaS and open fashions, together with these hosted on or exterior Databricks. Its centralized method simplifies safety and value administration, permitting our knowledge groups to focus extra on innovation and fewer on administrative overhead.”
— Greg Rokita, AVP, Know-how at Edmunds.com
Getting Began with Mosaic AI Gateway
Mosaic AI Gateway’s new monitoring and guardrails at the moment are obtainable for all mannequin serving workspaces. Allow Mosaic AI Gateway on new and current exterior fashions with just some clicks. Assist for added endpoints is coming quickly.