How to test AI agent skills and own your context

Key takeaways

• Test skills like you would test code: A basic eval is just a prompt you re-run under the same conditions to compare two versions of a skill, or a skill against no skill. Skills must reliably beat the no skill condition to be useful, across different models and over time.
• A skill that adds nothing is a net cost: Context is precious. If the agent does the task just as well without the skill, the skill is cluttering up context, so cut it or rewrite it.
• Re-test when you edit, and when the model changes: Re-run the earlier prompts, not just the one that motivated the edit. A new model release can make some of your guidance unnecessary.
• You should own your context: There are many community skills readily available. These can be a useful starting point, but should always be tailored to your business's situation.
• Don't let agents write their own skills: Left to itself, an agent pads a skill with restatement and invented best practices, and it can add instructions that are simply wrong.

This summary is produced by the author, and not by AI.

In part 1 we covered what a skill is and how to write one. This part is about the half that comes after: finding out whether the skill you wrote actually helps, and keeping the context you hand your agents yours.

Testing a skill with evals

Writing a skill takes time, and skills are context that can help but also hurt the quality of the agent's work. So how do you know if a skill actually helps? You can test it with evals. An eval is a prompt that you re-run under the same conditions, so that the only variable is what you deliberately change. You use the same model, starting context, and the same files (and wipe leftover memory from previous attempts). Then you compare: before and after an edit, with and without a skill, or two versions of a skill. Agents are non-deterministic, so you have to run the comparison multiple times to get a reliable difference; a few runs per condition is enough to start. Decide how a run is scored before looking at the results: use objective checks where you can (did it run the right command?), and for subjective calls write down beforehand what a good answer must include. Also run in a clean workspace: if the solution is already in another branch, the baseline run isn't a real test. Some companies run large automated eval suites, but a few saved prompts and some notes are enough to start.

Comparing the 'with skill' and 'without skill' conditions is called ablative testing; take away what you added that's supposed to help, and see if the results get worse. If there is no difference, the conclusion is that your skill is not adding value. Context is precious, so a skill that adds no value is a net cost. The skill has to beat the 'without skill' condition reliably to earn its place.

We tested this on a skill for the TE CLI (while it was in preview, so it wasn't in the model training data). Tool use gives yes-or-no checks (did it run the te deps command or not) that a business judgment does not.

---
name: te-cli-first-pass
description: Run a series of commands against a model to analyze how messy it is
---
Use the /te-CLI skill, if available. If not, use te --help and te commandname --help before running any commands.

1. Check if you are connected to a model, if not ask the user what model to connect to.
2. Run `te list`. Take note of how many tables there are and if they all seem to be used.
3. Run `te vertipaq`. Investigate the largest tables and largest columns. Also identify any columns with more than a million rows of cardinality.
4. Run `te deps --unused` to identify objects that can potentially be removed.
5. Run `te bpa run` to identify other problems. Not every warning needs to be resolved.
6. Use other non-destructive commands such as `te query` to explore further.
Report back with an overall summary of how messy the model is and the top 5 quality improvements.

Claude Opus 5 was given the same task twice: once with the skill and once without. Without the skill, it still did an adequate job. Modern agents are good at guessing help commands, so it found te --help and worked from there. It never ran te deps, and it never highlighted the high cardinality columns, however. With the skill, it followed the steps and reported both. The 'with skill' condition outperforms the 'without skill' condition, so this iteration of the skill survived the ablative testing.

There's also regression testing: checking that a change didn't erode what already worked. When you edit a skill, re-run your earlier prompts, not just the one that motivated the edit. When new models come out, re-test the skill entirely; some guidance may have become unnecessary and could be removed. Those prompts and notes we mentioned earlier become a small test suite. Maintaining it is part of the investment in agent-assisted development.

Own your context, don't let agents write skills

Don’t have agents write their own skills. It is reasonable to have the agent summarize where it ran into issues, and you had to correct them, and then have it write the first draft. In general, though, skills should always be hand-curated. Otherwise, AI slop behaviors risk being amplified: restating the obvious verbosely, defensive instructions for cases that can't occur, invented best practices.

When we asked Claude Opus 5 to install the te-cli-first-pass skill above, it “helpfully” rewrote it as a skill that was 4x as long as the original, which makes the skill 4x as expensive to read, a cost paid every time it loads. Much of that was rewriting what we said in a more verbose way; some of it was copying from the te-cli skill or adding instructions that were unnecessary.

What’s worse, however, is that two of the instructions the agent generated were factually incorrect. The agent completely ignored the fact that you can connect to a local Power BI Desktop instance and made it sound like the CLI could only read local files or connect to deployed models. In this way, the AI-expanded skill was actively worse than no skill at all.

The same goes for skills you did not write: community repositories, vendor skills or a colleague's export. Each encodes someone else's preferences, environment, and model assumptions. At the very least, you should read through them, and likely you need to customize what's specific to you or cut what's irrelevant. You should then run the same ablative test. If you use Anthropic's skill-creator skill, this also applies. It can help you scaffold, but you should curate every line that ends up in your context.

This all points in the same direction: that skills should focus on known gotchas, be clear about fragile or dangerous operations, map steps that match internal procedures or personal preferences, and avoid overprescribing. Much of this matches the best practices from the agent skills website.

Further reading

• How to write AI agent skills (Tabular Editor). Part 1 of this pair, covering what a skill is, how the specification works, and how to write your first one.
• Skill creation best practices (Agent Skills). The standard's own guidance on what belongs in a skill and what does not.
• Equipping agents for the real world with Agent Skills (Anthropic). An in-depth article introducing skills as a concept, as well as how they work.

In conclusion

An untested skill is a risky guess: it might help or it might hurt. Ablative testing is an easy way to turn the guess into an answer. Run the same prompt with and without the skill, under the same conditions, and see whether the results actually get better. Re-run those prompts when you edit the skill and as you adopt new models, because guidance the agent needed on older models can be dead weight for current models. Whether a skill came from a colleague, a community repository, or the agent itself, read every line before it enters your context; they're your context, so yours to own.