Unleashing Creativity: Databricks' New Era of Compound AI Systems through Mosaic AI

In an age where generative AI is no longer just a research topic but a core element of any innovation activity, Databricks has once again upped the ante by improving its Mosaic AI platform. Gone are the days when enterprises had to painstakingly code applications from scratch without any tangible way to gauge authentic intelligence. Mosaic AI is Databricks’ attempt to reverse that trend. Since purchasing MosaicML for $1.3 billion, Databricks is now leading an effort to create, evaluate, and govern so-called compound AI systems, which combine multiple machine learning models to accomplish a single task. Databricks’ investment in Mosaic AI puts it in a great position to compete against rivals such as Snowflake, as data becomes the most important strategic asset for any business. The company has built a data cloud that spans every aspect of enterprise artificial intelligence, data management, and analytics.

Revolutionizing Compound AI Systems with Mosaic AI

Demand for generative AI has been exploding, forcing enterprises to deploy LLM-based apps that can draw on their huge data lakes, while at the same time being careful to design a content model that can work reliably within privacy constraints and deliver high-quality apps at reasonable cost. Until now, for many businesses it’s been a bit of an Achilles heel. Here, Databricks plays its trump card: developers can be enabled to create RAG-based compound AI systems — that is, a multitasking package of small models, retrievers and vector databases, all working together in concert, and deployed with tools specifically designed for record-setting evaluation, monitoring and governance.

Elevating Model Training with Mosaic AI

Mosaic AI takes model training as well as databases out of Databricks’ ecosystem. By offering Mosaic AI Model Training as well as the Mosaic AI Agent Framework and Mosaic AI Alignment Desktop, Databricks has effectively passed a set of tools to developers enabling them to tweak open-source foundation models for both cost efficiency and task-specificity, making sure that the final application is not only effective but also precise, hopefully.

Crafting Exquisite RAG Apps

The high-quality RAG apps that power Mosaic AI’s upgrades could be built only with Agent Framework, which enables the sort of sophisticated evaluation of applications required to improve their quality, while also making it straightforward to do rapid iterations based on comprehensive feedback. In turn, this LLMOps workflow works its magic to bring these applications to life.

Governance at the Forefront with Mosaic AI Gateway

Trust me, trust is the highest currency in the digital age. And enterprises can trust Databricks’ Mosaic AI Gateway, which not only gives data scientists a single interface for orchestrating deployment of machine learning models, but built-in governance and monitoring without having to build custom solutions. Like a fortress safeguarding against misuse or noncompliance with organisational policies.

Mosaic AI: A Hub of Innovation

It’s not something that would have happened without a company like Databricks working to open source the Databricks Unity Catalog. Databricks’ AI Tools Catalog is currently in a private preview, but the company has broken through the hype of ‘black box’ AI by releasing the Databricks Unity Catalog as open source, spurring the growth of an open ecosystem in which functions become tools, increasing intelligence.

The Road Ahead: A Bright Future with Mosaic AI

From here, it is ourselves that we will largely write the stories of what becomes possible with Databricks’ Mosaic AI now, but surely in the coming years, the potential will multiply tenfold, a hundredfold with Databricks general availability offerings that are marching towards the future – and very soon, too. The 2024 Databricks Data and AI Summit will feature updates on the latest AI/BI analytics and data engineering solutions to stay in the forefront of cloud computing in combination with generative AI technologies, ensuring that Databricks is firmly on the leading edge of generative AI technology.

Understanding OPEN in the Context of Mosaic AI

The word ‘open’ lies at the very core of the architecture of Mosaic AI too. It underscores the fact that its technology combines adaptable open-source models with the vast oceans of innovation that exist throughout the world of AI development. It is also a reflection of the Databricks’s aspiration to build a community where the free and unrestricted exchange of ideas can be enabled value to flow. By leveraging open technologies and making components such as the Unity Catalog open source, Databricks is not only seeking to advance the science of AI but also to democratise it, making it accessible and impactful.

Open will play a defining part in where AI is headed. In time, as companies like Mosaic AI get off the ground, the intersection of open and innovation will continue to open up what AI can do beyond all recognition.