There are already enough skills

By mid-2026 the public indexes held millions of skills, with one scraper listing close to two million on its own. Finding a skill is easy; trusting one isn't.

The trust problem shows up in the numbers. A study of roughly fifty thousand skills in a single registry found that four in five behaved differently from what their own descriptions promised. Most of that is ordinary sloppiness rather than anything malicious: a description that claims more than the skill does, a step that got written down but never enforced. For whoever installs the skill, the effect is the same either way. The label and the contents disagree, and nothing warns you.

Skills fail in two separate places

When a skill lets you down, the break happens in one of two places.

One is activation, the question of whether the skill runs at all. The model reads a short description at startup and decides whether to load the skill. If that description is vague or passive, the skill never fires, and the model just answers on its own. One set of 650 trials found that rewriting the description alone moved a skill's activation from about a third of runs to nearly every run. The whole outcome rode on a single paragraph of metadata.

The other is execution, whether the skill runs all the way through. It loads, the model starts following it, then drops a step, usually a check near the end that produces no visible output. You get a clean, confident answer that never ran its own final verification.

From the outside, these two failures look the same. You asked for something, you got something reasonable, and you have no way to see which of your instructions survived the trip.

What I want to build

A checker that reads a skill and answers one question: does it do what its description claims?

In practice that comes down to three moves:

• Pull out the promises. Every “always,” “before you finish,” and “verify that” in a skill file is a claim about how it should behave.
• Run the skill on a task it says it handles, and record what the model actually did, step by step.
• Compare the two, and report which promises held and which got dropped.

The result is more useful than a single pass-or-fail score: a list of the specific claims that held and the ones that didn't, so an author knows what to fix and someone installing the skill knows whether to trust it.

Starting with a manual pass

I don't want to design the checker before I know the premise holds. So the first move is deliberately low-tech: take ten or fifteen real skills, pull out their promises by hand, run them, and count how often the description and the behavior actually disagree.

If the disagreements are common and easy to spot, the checker is mostly an engineering problem. If the promises turn out to be buried in prose too vague for a tool to pin down, that points somewhere more useful: the thing worth building might be the one that helps people write skills worth trusting in the first place. Both answers are worth the afternoon it takes to find out. And either way, the same principle holds: a handful of skills you can rely on is worth more than two million you can't.

Where the numbers come from:

• The 214-skill audit (nearly three-quarters below 60/100): “Claude Skills Not Working,” buildtolaunch.substack.com.
• Four in five skills deviating from their description: “Behavioral Integrity Verification for AI Agent Skills,” arxiv.org/pdf/2605.11770 (49,943 skills, 80% deviation).
• The 650-trial activation study: summarized at mcp.directory/blog/why-your-claude-skill-isnt-activating-2026-fixes (originally Ivan Seleznov, Feb 2026).
• Close to two million indexed skills, and “discovery is no longer the bottleneck; judgment is”: Agent Skills Ecosystem Report, agentman.ai (SkillsMP figure).

#buildinpublic