Anthropic Adds Plugin Evals to Claude Code: 6 Grader Types, a No-Plugin Baseline, and a CI Gate for Skills

Anthropic has added a new eval workflow for Claude Code that is aimed squarely at plugin builders who want proof, not vibes. The new claude plugin eval command runs a plugin against realistic prompts, scores the output, and then repeats the same case with the plugin removed so the result can be compared against a bare model.

That comparison is the point. Anthropic says the workflow answers three questions developers couldn’t measure well before: whether the skill triggers at all, whether it keeps working after an edit or model change, and whether it actually beats the model on its own. A case that scores the same with and without the plugin is not doing much for you.

The setup lives inside an evals/ directory in the plugin. Each test case is its own folder with a prompt.md file and a graders/ folder. The prompt is sent to Claude exactly as written, with @path mentions left alone, and frontmatter can set things like max_turns, timeout_seconds, model, tags, and allowed_tools.

There are six grader types. Four are computed from the transcript and local files, so they don’t add to the bill: regex, tool_used, tool_order, and file_exists. The other two, llm and baseline, call a judge model, which means they do cost money. Anthropic says every eval run and every judge grader is a real model call billed to the user’s plan or API account.

By default, each case runs twice: once with the plugin loaded and once without it. The difference, Δ, is the number that matters. Anthropic gives an example where one case scores 1.00 with the plugin, 0.33 without it, and therefore lands at +0.67. That run took 74 seconds, went through 6 runs, and was estimated at $0.41.

The company also points out a common failure mode: the skill-specific grader can fail even when the rest of the case looks fine, which usually means Claude is not picking the skill from natural language. That’s exactly the sort of problem claude plugin validate would miss, since validate checks syntax and schema, not behavior. Results are written to evals/results/<timestamp>/report.html, and can also be published to claude.ai unless --no-publish is used.