This summary is produced by the author, and not by AI.
Large Language Models (LLMs) – AI models that can predict text based on huge volumes of training data – are rapidly being integrated into BI workflows through copilots in everything from Fabric to Power BI. LLMs bring both immense promise of unlocking new levels of insight from data – but also immense hype. In this blog, we'll provide a level-headed assessment of how LLMs can help developers and users of semantic models. We'll also show how semantic models can ground LLMs in business relationships to provide accurate and helpful answers to business-critical questions.
LLMs are fundamentally a semantic technology. They excel at transforming text into meaningful predictions and are used across various tasks from writing apps from scratch to summarizing complex financial regulation. Given their capabilities, it's natural to ask how LLMs can enhance another semantic technology: semantic models.
As a brief reminder, semantic models represent the meaning of data and the structured relationships between different data entities. As Kurt Buhler explains, "A semantic model is essential for you to meet business data needs" by providing a structured representation that maps relationships between data entities and their business meaning. For instance, a well-built semantic model can allow users to integrate insights across datasets to answer business critical questions like "Which market is experiencing the highest growth?".
In this blog post, we'll explore where and how LLMs can improve the workflow of building and using semantic models, and vice versa.
The first thing to note about LLMs is that they're not just a single technology. There are many different types of LLMs, each with their own strengths and weaknesses.
At its core is the so-called "base" or "foundation" model - a model trained simply to predict the next sub-word given a huge training corpus, often spanning billions of words and encompassing much of the content available on the internet – like the original GPT-3. While these foundation models form the basis of all other models, they can be difficult to use because they simply predict the next word without any concept of interaction.
In this section, we'll explore three iterations of LLMs: chat LLMs, reasoning LLMs, and LLM agents. Each builds upon the previous version to unlock new capabilities.
The first step in making LLMs more useful is to enable interaction in a conversational format. The key innovation here is training the LLM to follow instructions, which was the breakthrough of ChatGPT when it was released in November 2022. This seemingly simple change had a tremendous impact, similar to how a well-structured Power BI report can unlock insights already present in raw data tables.
While many readers have likely interacted with chat LLMs, there are three specific ways they can be especially useful for developing and using semantic models:
While Chat LLMs have many valuable use cases, as illustrated, they still have several important limitations:
Reasoning LLMs have been trained to "think out loud," allowing them to solve much more complicated problems requiring math, coding, and reasoning. Benchmarks measuring coding abilities show a significant performance jump when comparing reasoning LLMs to standard LLMs.
These added capabilities unlock new use cases, particularly for checking semantic models' consistency and quality. For instance, you can feed in measures and the associated semantic model, and a reasoning LLM could identify duplicated measures or broken logic because it can think more deeply about the model.
Another advantage is that reasoning LLMs can serve as a second line of attack. If you've tried and failed to solve a problem with a normal chat LLM, you can try the same problem with a reasoning LLM (or potentially a group of different LLMs), which might improve performance.
LLM agents are language models that work in a loop with access to tools that can interact with the real world. This could include a code interpreter that allows the language model to execute the code it writes, or direct access to APIs that provide up-to-date information and potentially direct queries to semantic models.
The simplest kind of agent is an LLM with web search. This setup provides access to more up-to-date information, which can help the LLM provide grounded answers. Many people might have tried interacting with this type of agents through, e.g., Perplexity.
The capability to utilize tools unlocks entirely new use cases. For instance, agentic LLMs could serve as automatic validators of semantic models or quality assurance tools. They could perform adversarial testing of the model to check for data quality issues and errors by realistically loading and querying semantic models. Beyond catching logical errors, LLM agents could, for instance, detect referential integrity issues by interactively querying the data.
Furthermore, you could build a self-service BI agent where business users can interact with the agent to solve complex queries instead of having to go to the BI department.
So far, we've talked about how different types of LLMs can improve the workflows associated with semantic models. But semantic models also hold great promise for improving LLMs.
One significant way is that semantic models can provide structured information to ground the knowledge of large language models. An experiment referenced by Cube.dev showed that providing LLMs with a semantic model boosted the accuracy of LLM-generated queries on a database by 20% to 100% on a subset of questions.
This happens because semantic models can act as a source of truth and constrain the generations of the LLM to ensure that the code it writes is based on actual relationships and relies on available measures, which can dramatically reduce hallucinations.
Imagine that you want to ask an LLM to calculate which of your markets experienced the highest growth last year. Without a semantic model, the LLM would struggle to know the names of the relevant tables as well as which columns and measures can answer the question. With access to a semantic model, this becomes a much easier task for the LLM as it can ground its response in actual relationships.
As Enterprise Knowledge notes, "While LLMs are a powerful technology, they can result in an inaccurate, generalist approach" without the structure provided by a semantic layer.
This creates a virtuous cycle: better semantic models and metadata improve the performance of the LLM, which can in turn improve the documentation and workflows of the semantic model. Microsoft's Fabric Copilot demonstrates this by using descriptions and synonyms from the semantic model to improve its query writing capabilities.
Now that we've discussed the mutual benefits of semantic models and LLMs, let's consider six important aspects of integrating them in practice.
Many LLMs are complex to deploy, which means many organizations rely on APIs to access and deploy language models. It's important to select a trusted provider, such as Microsoft Azure OpenAI, and follow best practices. We'll dive deeper into security in a future blog post.
Ensuring that LLMs fit with your existing workflows is crucial. Treating the LLM as a microservice can be very useful, which also requires appropriate quality assurance, testing, and monitoring.
Empowering users to leverage AI is key for successful adoption. Both technical and business users in your organization need to understand where AI shows potential and where extra caution is required. For instance, when doing simple translations, AI can be very strong, but when generating complex DAX queries, verification and checking are essential. As Kurt Buhler advises, "Ensure that you validate any DAX query output from an AI model".
A good approach is to ensure there are always humans in the loop and strong transparency about where in the organization AI is generating solutions. You need to ensure that LLMs remain copilots and don't inadvertently steer the organization in the wrong direction.
It's important to track both the costs and benefits of these solutions. The main costs for LLMs are processing and generating text. It's good practice to measure this at as granular a level as possible, ideally at the use case level, to ensure that LLMs are providing real benefits.
If possible, benchmark this against the cost of not using LLMs. For instance, if generating 100 pages of documentation costs $100 in API calls, how much would it have cost in human expert time? Expensive specialists likely have many other priorities competing for their time.
Cost analysis is also important for deciding which models to use. For some tasks, reasoning models might be too expensive, but chat LLMs might do the job just as well.
AI is a rapidly developing field, so experimentation is key. AI researcher Ethan Mollick talks about the "jagged frontier" of AI capabilities. In a study of 758 consultants across 18 realistic business tasks, he and his collaborators found drastically different performances that were hard to predict.
The only way to ensure you get the benefits of AI is to try different use cases and see what works. Therefore, it's important to distinguish between the costs of having solutions in production and the costs associated with learning. Having a sufficiently large learning process is essential for staying on top of developments.
LLMs work best when guided by capable humans. As noted, the exact areas where LLMs excel can be hard to predict. It's therefore crucial that humans are kept in the loop and have full insight into all processes.
While user training can get you some of the way to looping in your co-workers, well-designed processes are equally important. One should ensure that there are procedures in place for manually verifying outputs from LLMs. For instance, when translating measures, one should get a native speaker to manually validate a subset to ensure the outcome is sound.
In this post, we've explored how two meaningful technologies, LLMs and semantic models, can synergistically improve one another, leading to more value for developers and the business. We looked at three types of LLMs and how each unlocks new semantic model use cases, from writing documentation to acting as automated testers. We also covered how semantic models ground LLMs in validated context to reduce hallucinations, and the considerations for implementing LLMs in organizations. We look forward to hearing about your experiences with these technologies!
Use semantic models to give LLM workflows clearer business context and safer boundaries.
Give Tabular Editor a spin