A practical introduction to LLMs for data professionals

Key Takeaways

• Large Language Models (LLMs) work with tokens: Sub-word chunks of text. This impacts tasks they are good at, limitations, and cost. Tokens are the currency and token generation is the speedometer for LLMs.
• LLMs excel at context and patterns: LLMs do best at identifying how these chunks of text relate to each other. This is what allows them to write in human language or code. However, accuracy decreases as context size increases.
• LLMs are unpredictable: LLMs are semi-random (or nondeterministic) by design. This also contributes to their ability to deal with unstructured tasks and vague instructions.
• LLMs make (subtle) mistakes: While hallucinations are the most common type of LLM mistakes, subtler errors occur as well, like models taking information out of context, over-extrapolating, or ignoring instructions.

This summary is produced by the author, and not by AI.

Why it’s important to understand LLMs

It’s tempting to think that working with LLMs and AI agents doesn’t require understanding anything about how they work. These tools are often presented as autonomous, intelligent, and self-explanatory. They communicate through conversational text, making them feel natural and intuitive, and can often even seem like magic.

In practice, however, these intuitions about LLMs are often very wrong. LLMs are a strange technology. As we stack tools and agent interfaces (or harnesses) on top of them, more and more misunderstandings also stack up. Treating them as an easy button leads to layers of frustration, waste, and, in the worst-case scenario, mistakes.

This is similar to how getting the best performance from your reports and semantic model requires understanding some of the fundamentals of how the data is stored and compressed. Things like many-to-many relationships, bi-directional filtering, and high-cardinality columns can have significant performance impacts that are not intuitive unless you understand some of the internals. Many-to-many relationships can also lead to subtle business logic bugs, even though all the input data looks right.

In this article, we discuss the LLM fundamentals that allow you to use them more effectively as well as understand their risks.

LLMs work with tokens

Humans perceive the world in specific ways. Consider the following analogy:

If you had a pet dog and expected it to sense the world in the same way a human does, you would be surprised to discover that it can’t distinguish color as well as you do, because dogs have something similar to red-green colorblindness in humans. You might be shocked at how well they can find certain objects, not realizing that their sense of smell is far more powerful than a human’s.

LLMs are also quite unlike humans in how they “perceive” the world. Despite often being presented as interacting through letters, words, and sentences, LLMs interact through what are called tokens. Tokens are multi-character chunks of text, averaging around 4 characters. In extreme cases, they can be as small as half a character or as large as 128 characters.

To understand what a token really is, consider the closest human equivalent: a Unicode character, symbol, or emoji. Emojis often stand in for specific letters or words, but in practice, they encode semantic meaning that is often ambiguous without the surrounding context. For example, if I use a fire emoji, I could be referring to a literal fire. More likely, I’m using it to mean something is excellent, exciting; metaphorically “on fire”.

Unlike emojis, though, tokens are not chosen for their meaning. They are just character sequences that often appear together in the training data, which is why they do not line up neatly with letters, syllables or words.

An implication here is that LLMs are bad at tasks that require character- or word-precise outputs, such as modifying a complex file format like the Power BI enhanced report (PBIR) JSON files. This is also part of why LLMs are bad at counting the r’s in the word strawberry, for instance. LLMs don’t count, they just see the tokens in the word strawberry. Once models were able to do reasoning, they could generate tokens to “see” the letters, and once they were able to execute code, they could just use a tool to count and “see” the correct numeric result:

So, to reiterate, tokens are fundamentally part of how LLMs “see” the world. Knowing that helps you understand an agent’s “perspective”. It also helps you better match LLMs to the tasks they are good at, which are more forgiving, such as generating language or computer code that you can then audit either yourself or with deterministic tools of the trade.

Why tokens are relevant for you

Tokens are relevant for you and your work in several ways:

• Cost: The cost of a model is measured in tokens. This is even the case for subscriptions, which operate on a finite usage limit also measured in a token budget. Costs from tokens come from many places, not just your prompt:
  • Input tokens are everything that goes into the model. They come from the prompt, model tool responses, system prompt, and memory or skill files.
  • Output tokens are everything that the model generates, including responses, tool calls, and reasoning or “thinking” that the model does between turns.
  • Cached tokens, which the model “remembers” from previous turns, are billed much, much lower than tokens that aren’t cached. You can also “break” the cache in certain ways, like changing the model or effort levels mid-task. We talk about this more in our next article about choosing the right model for the task.
• Speed: We measure a model’s speed in tokens per second (tok/s).
• Performance: The fuller a model’s context window gets, the worse it tends to perform (more on this in the next section). It’s similar to you trying to remember specific information from a very long article or video versus a short one.

Tokens are also how AI companies charge for LLMs. What this means in practice is that the fewer tokens you need to complete a task, the cheaper it is. It also happens to be that LLMs make fewer mistakes when fewer irrelevant tokens are involved, which is another reason to be mindful of what you are feeding into an LLM. If you are going to be learning about LLMs, plan to learn a lot about tokens.

LLMs specialize in context

Context, unfortunately, is an overloaded term when it comes to large language models. For our purposes, it means two separate but related things. First, it means how all of the tokens in an input interrelate, in the same way a single word at the end of a sentence could relate to anywhere else in the sentence. It also means the entirety of inputs given to an LLM. The maximum text input size is often called the “context window”, measured in tokens.

However, LLMs don’t read a sentence one word at a time. Rather, they take in blocks of text split into tokens, and the model uses a mathematical mechanism to calculate how each token relates to every other token in the context. This mechanism, called attention, is a big part of how LLMs seem to understand context.

Consider the following example:

Suppose you let an LLM explore your data model and work on a specific DAX measure, Flash Pool. When you refer in later prompts to “the flash” or “the measure” or even “it”, the LLM knows you’re referring to that measure. This works because the whole conversation, including that earlier exploration, is sent back to the model with every new prompt, and attention lets those later words relate back to the Flash Pool tokens earlier in the context. It’s also why a very long session can lose track of “the measure”: once the earlier context gets truncated or summarized away, there’s nothing left for those words to point back to.

As a human being and social creature, you pay attention to context all the time. How many times has someone said something and later on you’ve spent an hour trying to decode why they put emphasis on one particular word? Human language works in such a way that even a single emphasis or word order change can completely change the implied meaning. You can see in the diagram below how the word “only” relates to the word to the right and completely changes the meaning of the sentence.

Large language models specialize in these relationships, in the same way that a bloodhound specializes in smell. This allows them to focus on the semantic meaning behind the words and predict the next most probable tokens.

As you can imagine, this has impacts on performance and accuracy. In a naïve implementation, the number of relationships grows roughly squared to the number of input tokens. So, 10 tokens means 55 relationships. 100 tokens means 5,050 relationships. 1000 tokens means 500,500 relationships. It’s not hard to see how this quickly grows out of control. Modern LLMs use tricks to work around this scaling problem but there are still limitations.

For your purposes this has a few impacts. Even though it’s common for frontier models to have large context windows of a million tokens (roughly 1,500 pages), the more irrelevant text you put in the context window the worse the model performs. This is often called “context rot” or in some cases the needle-in-a-haystack problem. When working with LLMs, it’s important to give them as much information as they need and nothing more.

LLMs are semi-random

LLMs work by predicting the next token of text, in the same way that your phone might suggest the next word when you are text messaging someone. Many modern LLMs don’t just do this visibly, but they are also able to write to themselves via a hidden scratchpad using a technique called chain-of-thought. In practice this makes them a bit more reliable and capable of problem solving, but it’s still a semi-random sampling process.

If you run a local LLM you can see what it is writing to itself in order to “think”. In the screenshot below, Google’s Gemma 4 model is counting on its fingers and toes (metaphorically).

During the development of LLMs, it was found that simply predicting the most probable token leads to very stilted and repetitive text. So, LLMs use a setting called temperature (think of how hotter molecules bounce around faster in physics) to semi-randomly choose the next token based on likely probability. Some models also apply a repetition penalty or use a different sampling technique.

What this means in practice is that LLMs are non-deterministic. This is just a fancy way of saying that the same inputs do not always lead to the same outputs. LLMs are less like a calculator that always returns 2 + 2 = 4 and more like humans when you ask them where they want to eat.

Non-determinism in LLMs is inherently how they work, and not a problem to solve (or that can be solved). Even if you turn the temperature all the way down to 0 so that the model always picks the most probable token, you aren’t guaranteed to get the same results on a production system because of the way GPUs and floating-point math work under varying batch sizes.

Relatedly, LLMs are very sensitive to initial inputs, similar to the way the first few bounces of a pachinko ball can lead to it landing in very different spots. Even a single space in an initial input can lead to cascading changes in the trajectory of an LLM’s output. Fundamentally, LLMs should be treated as stochastic systems.

LLMs can make mistakes, including (but not limited to) hallucinations

Because of the nondeterminism we discussed in the previous section, LLMs can make mistakes. The most infamous of these are the hallucinations that were abundant in early-era AI tools and chatbots. However, these are much less common (and egregious) in more recent models. Furthermore, we also better understand now how to use them (or at least we mostly do). Mistakes nowadays are by-and-large more nuanced.

Examples include:

• Taking information out of context. This can be common when you ask AI to summarize long text, especially when it’s a combination of text, images, and possibly even audio/video.
• Over-extrapolating and over-correcting. This occurs when AI uses your prompt and instruction to infer a cause without direct evidence. For instance, if you say “The chart is incorrect, the budget should be 1.72M, fix it”, an agent might interpret that as a fix that must be made in the chart itself. It could hard-code the value or make changes in the chart, and won’t necessarily go to check the underlying semantic model or data if it wasn’t explicitly prompted to do so.
• Ignoring instructions. When you provide too much context, the model could ignore part or all of particular instructions. Again; “context rot”.

Mistakes arise because of how LLMs are trained

Lastly, it’s valuable to understand how many of the types of mistakes LLMs make are encouraged by their training. Stronger, more expensive models often just make more subtle or sloppy mistakes rather than straight up lying to you. LLMs are trained in a series of phases that help explain the types of issues you see with them and why they get more subtle.

All LLM training starts with simple autocomplete training. The numerical weights are refined on massive, internet-scale datasets by training models to predict the next token in a text sequence. This ends up making LLMs nearly perfect at basic spelling and grammar, but also trains them to generate plausible text for any scenario. LLMs are trained to sound right, not to be right.

Next, a model is trained to go from raw autocomplete to an assistant persona. This is done by feeding in sample dialogues between an imaginary user and what the LLM should say. Models are also trained with Reinforcement Learning from Human Feedback (RLHF). If you’ve been asked to pick which output you prefer from a chatbot, that is RLHF. Both of these steps risk causing sycophancy, where the model is trained to say what you want to hear instead of what is true or helpful. Humans like flattery and LLMs or agents are typically instructed to be helpful. In 2025, OpenAI had to roll back a model because they leaned too heavily on RLHF.

Models are also trained against certain benchmarks, but can be overoptimized to make a new model release look more impressive. Knowledge benchmarks have often been multiple-choice quizzes that do not provide an option for “I don’t know” or a penalty for getting it wrong. This leads to guessing the right answer.

Training against math and coding benchmarks should, in theory, be more reliable because they can be more easily verified, and the model can be rewarded for any progress. In practice, however, this can have some unintended side-effects. One side effect is that, anecdotally, some models might be more risk-averse and can add fallbacks and edge case handling where it’s not needed. Models may also work towards the right solution but may overengineer those solutions or may write all the code in one file instead of making the code more maintainable.

In the next article, we’ll explain how to choose an LLM for your agent to use.

Further Reading

• Context Rot: How Increasing Input Tokens Impacts LLM Performance (Try Chroma). Article showing how context length and distractors decrease LLM accuracy.
• Effective context engineering for AI agents (Anthropic). Article covering in great detail how to manage context for agents.
• Neural Networks (3Blue1Brown). A series of videos about how the math behind neural networks and large language models works.
• Open AI tokenizer (OpenAI). An interactive website to see how your text gets converted into tokens.

In conclusion

LLMs have strengths and weaknesses that are very different than humans. LLMs interact primarily through tokens and context. They do so in ways that are semi-random and unpredictable, and there are many failure modes that you can encounter. Understanding these traits allows for using LLMs and AI agents more effectively.