How to write and test AI agent skills

Key takeaways

• AI skills are context that teach an agent about a process or concept: A skill is a folder with a Markdown file, with optional references, scripts, or other files, including binaries and full programs or projects. The skill itself is context for the agent, but the files it carries can let the agent do things it could not do before.
• Use skills to teach agents about your business and its processes: Modern agents are good at figuring out technical processes on their own. They cannot accurately derive your business' processes, definitions and preferences. That belongs in skills.
• Write skills iteratively and remove what doesn't help: The best way to write a skill is to have an agent perform a task and then document the issues the agent ran into and then repeat. In many cases, overly prescriptive skills can cause confusion instead of helping; rather than adding more, consider what you can cut and rephrase.
• Test skills like you would test code: A basic eval is just a prompt you re-run under the same conditions to compare two versions of a skill, or a skill against no skill. Skills must reliably beat the no skill condition to be useful, across different models and over time.
• You should own your context: there are many community skills readily available. These can be a useful starting point but should always be tailored to your business' situation.

This summary is produced by the author, and not by AI.

Skills are a simple but powerful tool

AI skills, at their very simplest, are just text prompts stored in a SKILL.md markdown file, along with some supporting reference, script, and asset files. There is no magic here. However, skills are quite valuable for making AI agents more reliable and for sharing capabilities with the rest of your organization.

These skills are installed either in a configuration folder specific to the project or in a global configuration folder. The location for either depends on the agent harness you are using, such as Claude Code or GitHub Copilot.

Once a skill has been installed, it’s simply a matter of typing a slash and then the skill name along with any supporting guidance. Then the agent will load the skill as a reusable prompt. Skills can also be called by the agent itself, unless that is specifically disabled in the skill definition. In the screenshot below you can see the list of skills that show up when typing /te-. Pressing enter would run the skill and pressing tab would complete the skill name.

One of the benefits of skills is that they are designed to take advantage of progressive disclosure. This means that the agent reads progressively more about the skill, on-demand. This is valuable because always-loaded context increases costs and can degrade agent performance.

Another benefit of skills is that they are easy to share and collaborate on. While CLIs and MCP servers are useful, they make more sense as packaged programs shared from a central source and consumed by users. Skills, on the other hand, can be kept in a Git repository and versioned like any other piece of code or documentation.

Skills operate in a hybrid space between documentation and code. Skills can come with executable code but the prompt itself is not code. You should treat skill writing as a writing and communication challenge, but you should store, track, and maintain them as if they were code.

In fact, in a recent article by Anthropic, they recommended treating the skills for a data model as first-class citizens and storing them alongside the code for the model. This was to ensure that skills were updated frequently along with the model, otherwise the effectiveness of those skills would degrade.

Understanding the skill specification

Skills, despite being simple, are an open standard originally created by Anthropic, similar to the Model Context Protocol standard which was also created by them. Skills were presented as a complement to MCP servers as a way of extending AI agents. MCP servers are not as flexible as skills, since they provide a predefined set of tools that the user can’t modify.

As far as technical standards go, the one for agent skills is ridiculously simple. A skill is a markdown file consisting of three parts: the name of the skill, a short description of the skill, and then a prompt of general instructions for the AI. The name and the description make up the frontmatter, a small metadata block between two --- at the top of the file. The frontmatter can also contain other optional metadata, like the tools the agent is allowed to use.

Optionally, it can include a references folder for reference material that shouldn’t be loaded every time. It can also include a scripts folder for code the agent can run. Finally, you may include other folders, such as an assets folder.

So, if you could write out this standard on the back of a napkin, why does it need to be a standard at all? Well, we think there are two main benefits of this being a standard. First, it leads to more reliable integration with various coding harnesses. That said, implementation varies quite a bit by harness. The allowed-tools setting is currently experimental, for example.

Second is that skills, unlike MCP servers, naturally encourage progressive disclosure which improves cost and performance when using AI agents. The frontmatter of the skill is the part that’s always loaded into the context window of the agent. The full instructions are not read until a human or agent decides to invoke that skill. The frontmatter thus plays an important role in automatic skill usage of your agents. Using a vague name or description means the skill won't activate when it should and agents work unaided when help is available. A description that overpromises can make the skill fire on tasks it wasn't written for, steering the agent down the wrong path.

If the frontmatter is the always-loaded first level, and the prompt the read-when-invoked second level, the references folder adds a third on-demand level. The skill's prompt can point the agent at files in the references folder, which it only reads when the situation calls for them. A skill for the TE CLI, for example, could contain guidance on combining it with the Fabric CLI in a reference file: useful in the sessions where it comes up but dead weight in others.

A skill is context, not a tool

It's easy to lump skills in with MCP servers and other plugins, but they solve a different problem. Whereas an MCP server gives the agent tools (a predefined set of functions it can call and which you cannot modify), a skill gives the agent context: what it should do, in what order, pitfalls to avoid, and how you prefer it to be done. When an agent fails at a task, it's usually not because it was missing a tool. It already has a terminal and can read files, so mechanically it could do the work. What it lacks is the knowledge it can't derive from its training data, what only you and your colleagues know.

That said, skills can expand what an agent is able to do. A skill can bundle scripts, binaries, and even full programs, with instructions telling the agent when to use them. Anthropic's document skills work this way: collections of python scripts with instructions on how to use them.

Teach the business instead of the technology

The models that agents run on keep getting better at figuring out technical processes on their own. Agents read --help output, inspect error messages, and already know the fundamentals of technologies like DAX, SQL and Python from training. Skills explaining the technicals can be useful, but every model release makes them a little less needed. Technology that runs ahead of the training data, however, does make sense to teach.

But models will never pick up from training data how your business works. Skills that encode business processes and definitions keep their value better than the technical ones.

Writing your first skill

Writing your first skill is easier than it might seem. Take a task that you already do and have the agent attempt to perform it. Then note any areas where the agent made incorrect assumptions, got confused or needed guidance. Then, importantly, test the agent again against the skill and iteratively make changes. As models get more powerful, less and less needs to be explicitly said.

A skill should contain guidance that is not in the model’s training data, preferences or information specific to you or your organization, and pitfalls or gotchas that you have observed the agent running into. One way of thinking about this is if you hired someone who was technically brilliant. They excel at DAX on their first day, but they cannot know that your organization sells kits that hide their component parts, or that finance resets standard costs every January. Nobody can know those things from the outside so they must be encoded in a skill.

Below is a simple example of what that may look like. An agent without knowledge of how your bill of materials works will erroneously rank parts by invoice lines, missing the parts that mostly sell inside kits but not much by themselves.

---
name: bill-of-materials
description: Service kits do not show component parts in sales data. Use when analyzing part-level sales, demand, margin, or stock levels.
---
We sell parts individually and as fixed-content service kits. A kit is its own SKU; the invoice line does not show the components. Kits contain parts, not other kits.
- Don’t rank parts by invoice lines alone. A part that’s mostly sold inside kits will look like it barely sells, even if those kits sell well.
- For part-level demand or profitability, first break kit sales down into their components. The kit contents can be queried from different sources: references/kit-sources.md.
- Kits are priced below the summed list prices of their components. When allocating kit revenue to parts, allocate by standard cost share, not list price.
- Part margin uses standard cost, not last purchase price. Finance resets standard costs every January so margins can then shift even when sales did not change.
- We hold stock of components, not assembled kits. For availability questions, check the components. The scarcest part determines kit availability.
- If a kit ever appears as a component in another kit, the data is wrong: kits should not contain kits. This is almost always due to a copy error in the ERP. Stop and relay the affected kits to the user so the rows can be fixed.

A skill should not contain information that is easily re-derived by the agent or information that the agent is heavily trained on (such as Python fundamentals).

Testing a skill with evals

Writing skills takes time, and skills are context that can help but also hurt agent performance. So how do you know if a skill actually helps? You can test it with evals. An eval is a prompt that you re-run under the same conditions so that the only variable is what you deliberately change. You use the same model, starting context, and the same files (and wipe leftover memory from previous attempts). Then you compare: before and after an edit, with and without a skill, or two versions of a skill. Agents are non-deterministic so you have to run the comparison multiple times to get a reliable difference. Some companies run large automated eval suites, but a few saved prompts and some notes is enough to start.

Comparing the 'with skill' and 'without skill' conditions is called ablative testing; take away what you added that's supposed to help, and see if performance drops. If there is no difference, the conclusion is that your skill is not adding value. Context is precious, so skills that don't add value make agent performance slightly worse. The skill has to beat the 'without skill' condition reliably to earn its place.

We tested this on a skill for the TE CLI (while it was in preview, so it wasn't in the model training data). Tool use gives yes-or-no checks (did it run the te deps command or not) that a business judgment does not.

---
name: te-cli-first-pass
description: Run a series of commands against a model to analyze how messy it is
---
Use the /te-CLI skill, if available. If not, use te --help and te commandname --help before running any commands.

1. Check if you are connected to a model, if not ask the user what model to connect to.
2. Run `te list`. Take note of how many tables there are and if they all seem to be used.
3. Run `te vertipaq`. Investigate the largest tables and largest columns. Also identify any columns with more than a million rows of cardinality.
4. Run `te deps --unused` to identify objects that can potentially be removed.
5. Run `te bpa run` to identify other problems. Not every warning needs to be resolved.
6. Use other non-destructive commands such as `te query` to explore further.
Report back with an overall summary of how messy the model is and the top 5 quality improvements.

Claude Opus 5 was given the same task twice: once with the skill and once without. Without the skill, it still did an adequate job. Modern agents are adept at running te --help and working from there. It never ran te deps and it never highlighted the high cardinality columns, however. With the skill, it followed the steps and reported both. The 'with skill' condition outperforms the 'without skill' condition, so it survived the ablative testing.

There's also regression testing: checking that a change didn't erode what already worked. When you edit a skill, re-run your earlier prompts, not just the one that motivated the edit. When new models come out, re-test the skill entirely; some guidance may have become unnecessary and could be removed.

Own your context, don't let agents write skills

Don’t have agents write their own skills. It is reasonable to have the agent summarize where it ran into issues and you had to correct it, and then have it write the first draft. In general though, skills should always be hand-curated. Otherwise, AI slop behaviors, such as adding the date to comments, risk being amplified.

When we asked Claude Opus 5 to install this te-cli-first-pass skill, it “helpfully” rewrote it as a skill that is 4x as long as the original, which makes the skill 4x as expensive to read. Much of that was rewriting what we said but in a more verbose way, but some of it was copying from the te-cli skill or adding instructions that were unnecessary.

What’s worse, however, is that two of the instructions were factually incorrect. The agent completely ignored the fact that you can connect to a local PBIX model and made it sound like the CLI could only read local files or connect to deployed models. In this way, the AI-expanded skill was actively worse than no skill at all.

skill-minimap

The same goes for skills you did not write: community repositories, vendor skills or a colleague's export. Each encode someone else's preferences, environment and model assumptions. At the very least you should read through them, and likely you need to customize what's specific to you or cut what's irrelevant. You should then run the same ablative test. If you use Anthropic's skill-creator skill, this also applies. It can help you scaffold, but you should know every line that ends up in your context.

This all points in the same direction: that skills should focus on known gotcha, be clear about fragile or dangerous operations, map steps that match internal procedure or personal preferences, and avoid overprescribing. Much of this matches the best practices from the agent skills website.

For further reading

• Equipping agents for the real world with Agent Skills (Anthropic). An in-depth article introducing skills as a concept as well as how they work.
• How Anthropic enables self-service data analytics with Claude (Anthropic). An article covering the unique challenges using AI with semantic models, including skill management.
• Agent Skills specification (Agent Skills). The official standard for writing AI agent skills. It also includes best practices.

In conclusion

Skills are a flexible and powerful standard for guiding AI agents in certain tasks. They can be as simple as a dozen lines of text and are context, not tools. Knowledge that will never be in the training data like your business' processes is most valuable to encode, while technical knowledge on how to use common tools becomes irrelevant the fastest. Skills should be written cautiously, iteratively, concisely, and test against the no-skill baseline; they're your context so yours to own.