A practical introduction to LLM basics for data professionals

Key Takeaways

• Large Language Models (LLMs) see through tokens: LLMs work based on subword chunks of text. This impacts tasks they are good at, limitations, and cost. Tokens are the currency and token generation is the spedometer for LLMs.
• LLMs excel at context and patterns: LLMs do best at identifying how these chunks of text relate to each other. This is what allows them to write in human language or code. However, accuracy decreases as context size increases.
• LLMs are unpredictable: LLMs are semi-random (or nondeterministic) as a core part of their design. They are able to succeed at less predictable and structured tasks or instructions for this reason, but have unique risks and need guardrails to account for this.
• LLMs make (subtle) mistakes: While hallucinations are the most common type of LLM mistakes, subtler errors occur as well, like models taking information out of context, over-extrapolating, or ignoring instructions.

This summary is produced by the author, and not by AI.

Why it’s important to understand LLMs

It’s tempting to think that working with LLMs and AI agents doesn’t require understanding anything about how they work. These tools are often presented as autonomous, intelligent, and self-explanatory. They communicate through conversational text, making them feel natural and intuitive, and can often even seem like magic.

In practice, however, these intuitions about LLMs are often very wrong. LLMs are a strange technology. As we stack tools and agent interfaces (or harnesses) on top of them, more and more misunderstandings also stack up. Treating them as an easy button leads to layers of frustration, waste, and in the worst case scenario- mistakes.

This is similar to how getting the best performance out of your reports and semantic model requires understanding some of the fundamentals of how the data is stored and compressed. Things like many-to-many relationships, bi-directional filtering, and high-cardinality columns can have significant performance impacts that are not intuitive unless you understand some of the internals. Many-to-many relationships can also lead to subtle business logic bugs even though all the input data looks right.

In this article, we discuss the LLM fundamentals that allow you to use them more effectively as well as understand their risks.

LLMs see the world through tokens

Consider the following analogy:

If you had a pet dog and you expected it to sense the world the same way as a human does, you would be a bit confused, but also very, very frustrated. You might be annoyed that it can’t distinguish color as well as you do, not realizing that dogs have something similar to red-green colorblindness in humans. You might be shocked at how well they can find certain objects, not realizing that their sense of smell is far more powerful than a human’s.

LLMs are also quite unlike humans in how they “perceive” the world. Despite being presented as interacting through letters, words, and sentences, LLMs interact through what are called tokens. Tokens are multi-character chunks of text, averaging around 4 characters. In extreme cases, they can be as small as half a character or as large as 128 characters.

To understand what a token really is, consider the closest human equivalent: a unicode character, symbol, or emoji. Emojis often refer to certain letters or words, but in practice they encode semantic meaning that is often ambiguous without the surrounding context. For example, if I use a fire emoji, I could be referring to a literal fire. More likely I’m using it to mean something is excellent, exciting; metaphorically “on fire”.

An implication here is that LLMs are bad at tasks that require character-precise outputs, such as modifying a complex file format like the Power BI enhanced report (PBIR) JSON files. This is also part of why LLMs are bad at counting the r’s in the word strawberry, for instance… LLMs don’t count, they just see the tokens in the word strawberry. Once models were able to do reasoning, they could generate tokens to “see” the letters, and once they were able to execute code, they could just use a tool to count and “see” the correct numeric result:

So to reiterate, tokens are fundamentally part of how LLMs “see” the world. Knowing that helps you understand an agent’s “perspective”. It also helps you better match LLMs to the tasks they are good at which are more forgiving, such as generating language, computer code, or genetic sequences.

Why tokens are relevant for you

Tokens are relevant for you and your work in several ways:

• Cost: The cost of a model is measured in tokens. This is even the case for subscriptions, which operate on a finite usage limit also measured in a token budget.
• Speed: How fast a model is is measured in the tokens per second (tok/s).
• Performance: When you use an agent, you’ll notice that it has a context window. This is a finite amount of context that can fit into a single conversation before that conversation must be summarized (or compacted) and started over. The more full that context window gets, the more performance tends to go down. It’s similar to you trying to remember information from a very long article or video versus a short one.

Tokens are also how AI companies charge for LLMs. Typically, there are three rates you have to be concerned about: input tokens, output tokens, and cached input tokens. What this means in practice is the fewer tokens you need to complete a task, the cheaper it is. It also happens to be that LLMs make fewer mistakes when fewer irrelevant tokens are involved, which is another reason to be mindful of what you are feeding into an LLM.

Tokens per second is a common measure of the speed of an LLM. Smaller models in general run faster but it also greatly depends on hardware. If you are going to be learning about LLMs, plan to learn a lot about tokens.

LLMs specialize in context

Context, unfortunately, is an overloaded term when it comes to large language models. For our purposes it means two separate but related things. First, it means how all of the tokens in an input interrelate, in the same way a single word at the end of a sentence could relate to anywhere else in the sentence. It also means the entirety of inputs given to an LLM. The maximum text input size is often called the “context window”, measured in tokens.

As a human being and social creature, you pay attention to context all the time. How many times has someone said something and later on you’ve spent an hour trying to decode why they put emphasis on one particular word? Human language works in such a way that even a single emphasis or word order change can completely change the implied meaning. You can see in the diagram below how the word “only” relates to the word to the right and completely changes the meaning of the sentence.

Large language models specialize in these relationships, in the same way that a bloodhound specializes in smell. What the core of a large language model does is to look at every pair of tokens in a sentence or paragraph, and identify which ones relate to each other. This allows it to focus on the semantic meaning behind the words and predict the next most probable token.

As you can imagine, this has impacts on performance and accuracy. In a naïve implementation, the number of relationships grows roughly squared to the number of input tokens. So, 10 tokens means 55 relationships. 100 tokens means 5,050 relationships. 1000 tokens means 500,500 relationships. It’s not hard to see how this quickly grows out of control. Modern LLMs use tricks to work around this scaling problem but there are still limitations.

For your purposes this has a few impacts. Even though it’s common for frontier models to have large context windows of a million tokens (roughly 1,500 pages), the more irrelevant text you put in the context window the worse the model performs. This is often called “context rot” or in some cases the needle-in-a-haystack problem. When working with LLMs, it’s important to give them as much information as they need and nothing more.

LLMs are semi-random

LLMs work by predicting the next token of text, in the same way that your phone might suggest the next word when you are text messaging someone. Many modern LLMs don’t just do this visibly, but they are also able to write to themselves via a hidden scratchpad using a technique called chain-of-thought. In practice this makes them a bit more reliable and capable of problem solving, but it’s still a semi-random sampling process.

If you run a local LLM you can see what it is writing to itself in order to “think”. In the screenshot below, Google’s Gemma 4 model is counting on its fingers and toes (metaphorically).

During the development of LLMs, it was found that simply predicting the most probable token leads to very stilted and repetitive text. So some LLMs use a setting called temperature (think of how hotter molecules bounce around faster in physics) to semi-randomly choose the next token based on likely probability. Some models also apply a repetition penalty or use a different sampling technique.

What this means in practice is that LLMs are non-deterministic. This is just a fancy way of saying that the same inputs do not always lead to the same outputs. LLMs are less like a calculator that always returns 2 + 2 = 4 and more like humans when you ask them where they want to eat.

Non-determinism in LLMs is an incredibly difficult problem to solve. Even if you turn the temperature all the way down to 0 so that the model always picks the most probable token, you aren’t guaranteed to get the same results on a production system because of the way GPUs and floating-point math work under varying batch sizes.

Relatedly, LLMs are very sensitive to initial inputs, similar to the way the first few bounces of a pachinko ball can lead to it landing in very different spots. Even a single space in an initial input can lead to cascading changes in the trajectory of an LLM’s output. Fundamentally, LLMs should be treated as semi-random systems.

LLMs can make mistakes, including (but not limited to) hallucinations

Because of this nondeterminism we discuss in the previous section, LLMs can make mistakes. The most infamous of these are the hallucinations that were abundant in early-era AI tools and chatbots. However, these are much less common (and egregious) in more recent models; mistakes nowadays are more nuanced.

Examples include:

• Taking information out of context. This can be common when you ask AI to summarize long text, especially when it’s a combination of text, images, and possibly even audio/video.
• Over-extrapolating and over-correcting. This occurs when AI uses your prompt and instruction to infer a cause without direct evidence. For instance, if you say “The chart is incorrect, the budget should be 1.72M, fix it”, an agent might interpret that as a fix that must be made in the chart itself. It could hard-code the value or make changes in the chart, and won’t necessarily go to check the underlying semantic model or data if it wasn’t explicitly prompted to do so.
• Ignoring instructions. When you provide too much context, the model could ignore part or all of particular instructions.

Mistakes arise because of how LLMs are trained

Lastly, it’s valuable to understand how many of the types of mistakes LLMs make are encouraged by their training. Stronger, more expensive models often just make more subtle or sloppy mistakes rather than straight up lying to you. LLMs are trained in a series of phases that help explain the types of issues you see with them and why they get more subtle.

All LLM training starts with simple autocomplete training. The numerical weights are refined on massive, internet-scale sets of data trying to predict the next token of text. This ends up making LLMs nearly perfect at basic spelling and grammar but also trains them to generate plausible text for any scenario. LLMs are trained to sound right, not to be right.

Next a model is trained to go from raw autocomplete to an assistant persona. This is done by feeding in sample dialogues between an imaginary user and what the LLM should say. Models are also trained with Reinforcement Learning from Human Feedback (RLHF). If you’ve been asked to pick which output you prefer from a chatbot, that is RLHF. Both of these steps risk causing sycophancy where the model is trained to say what you want to hear instead of what is true or helpful. In 2025, Open AI had to rollback a model because they leaned too heavily on RLHF.

Models are also trained against certain benchmarks but can be overoptimized to make a new model release look more impressive. Knowledge benchmarks have often been multiple choice quizzes that do not provide an option for “I don’t know” or a penalty for getting it wrong. This leads to guessing the right answer.

Training against math and coding benchmarks should in theory be more reliable because they can be more easily verified and the model can be rewarded for any progress. In practice, however, this can have some unintended side-effects. One side effect is that anecdotally the Anthropic models are more risk averse and will put in fallbacks and edge case handling where it’s not needed. Models may also work towards the right solution but may overengineer those solutions or may write all the code in one file instead of making the code more maintainable.

In the next article, we’ll explain how you choose an LLM for your agent to use.

Further Reading

• Context Rot: How Increasing Input Tokens Impacts LLM Performance (Try Chroma). Article showing how context length and distractors decrease LLM accuracy.
• Effective context engineering for AI agents (Anthropic). Article covering in great detail how to manage context for agents.
• Neural Networks (3Blue1Brown). A series of videos about how the math behind neural networks and large language models works.
• Open AI tokenizer (OpenAI). An interactive website to see how your text gets converted into tokens.

In conclusion

LLMs have strengths and weaknesses that are very different than humans. LLMs interact primarily through tokens and context. They do so in ways that are semi-random and unpredictable. Understanding these traits allows for using LLMs and AI agents more effectively.