# Evals of the Claude Code plugin

> The suite that runs Claude Code with the stetdoc plugin on real legal documents and scores its turns, tool calls, tokens, cost and results.

Source: https://stetdoc.clauseo.chat/docs/evals · stetdoc 0.9.1

`evals/` is a suite that runs Claude Code with the stetdoc plugin on real legal documents and scores what it does. It
runs on Claude Code's own `claude plugin eval` (Claude Code 2.1.311 or later), so a case is a prompt a user would
type, and graders check the result. It is also the benchmark for the plugin: each case records turns, tool calls,
tokens, cost and wall time per run, so a change to the CLI, the skill or the MCP server can be measured for speed and
accuracy as well as correctness.

Every run is a real model call on your account. The suite never runs in CI or in `cargo xtask check`.

## Running it

```
evals/run.sh                                  every case, 3 runs each, then the case without stetdoc on PATH
evals/run.sh --case redline-net60-new-york --runs 1 --skip-without-stetdoc
evals/run.sh --arm none                       the same cases with no plugin loaded (stetdoc stays on PATH)
evals/run.sh --arm ~/my-docx-plugin           the same cases with another plugin, for comparison
evals/run.sh --max-cost-usd 15 -j 4           a cost ceiling, and four runs at once
evals/run.sh -- --keep-temp                   anything after -- goes to claude plugin eval
```

`run.sh` fetches the test documents into `evals/fixtures/` (see below), points `evals/arm` at the plugin under
test, runs `claude plugin eval` twice and prints a summary. The second run is the case `not-installed`, with every
directory that holds a `stetdoc` command taken out of `PATH`. The defaults are `--model claude-opus-5-5` and
`--judge-model claude-sonnet-5-5`. Pin both when you compare runs over time.

It passes `claude plugin eval` the options the suite needs: `--scaffold` (the cases' setup scripts, which you are
trusting as code from this repository), `--trust-plugin`, `--allow-real-servers` and `--allow-tools Bash Edit Write
"mcp__plugin_stetdoc_stetdoc__*"`, so the agent has the plugin's real MCP server and the CLI, as a user does. Shell
commands run in Claude Code's sandbox, which needs `bubblewrap` and `socat` on Linux and is missing on native Windows.

Each run has a temporary home, so the agent's settings, skills and memory do not leak in. If your Claude Code uses
Vertex AI, `run.sh` exports `GOOGLE_APPLICATION_CREDENTIALS` to your application default credentials, because the
temporary home hides them.

### Cost and time

A full run (8 cases and `not-installed`, 3 runs each, 27 agent runs, one arm) costs about $10 and takes about
30 minutes with the default concurrency of 1, or under 10 minutes with `-j 4`. One run of a case costs $0.10 to
$0.60: `redline-net60-new-york` was $0.39 in 34 s, `stale-colleague-edit` $0.21 in 31 s. The `llm` graders add a few
cents. Each `--arm` doubles the bill.

## Reading the results

Results go to `evals/results/<time>-<arm>/` (ignored by git):

- `summary.md`: one line per case with the score, the runs that passed, and the mean turns, tool calls, tool errors,
  input tokens (with the cached part), output tokens, cost and wall time per run, and the graders that failed in how
  many runs.
- `cases/report.html` and `without-stetdoc/report.html`: Claude Code's report, with every grader's verdict and
  explanation per run, and the judge's votes for `llm` graders.
- `cases/aggregate-result.json`: the same as JSON (`schemaVersion: 1`), for scripts.
- `metrics.jsonl`: one line per run with the tool calls by tool name, tool errors, the skills used and the tokens.
- `reports/`: each run's harness report and its session transcript, named after the case and the run.

A run's score is the weighted share of its graders that passed, and a case's score is the mean over its runs. A case
passes at 1.0. Speed graders (`few-shell-calls`) and hygiene graders (no tool errors, no working copy or stray files
left) weigh half. `claude plugin eval` exits 1 when a case scores below 1.0, and `run.sh` exits with that status after
printing the summary.

## The cases

| Case | Document | What it checks |
|---|---|---|
| `read-conditions-precedent` | credit agreement | The answer cites Sections 4.01 and 4.02 (judged), and nothing is written. |
| `redline-net60-new-york` | supply agreement | Two tracked changes by the configured author in 11.1 and 23.1, a new file, nothing else changed. |
| `comment-liability-cap` | supply agreement | One comment in Clause 18 asking for 2x annual fees, no tracked change, the original untouched. |
| `accept-reject-ambiguous` | ISDA confirmation v2 | "Accept all except amounts or dates": the agent states its reading or asks (judged); any file it writes resolves all 47 revisions. |
| `compare-v1-v2` | ISDA confirmation v1 and v2 | A redline with the revised text, at least 20 revisions, one author. |
| `rename-seller-vendor` | supply agreement | All 89 "Seller" become "Vendor" as tracked changes, and nothing else changes. |
| `chained-edits` | supply agreement | Three edits into one redline, the last rejecting the first; no stale-base refusal. |
| `stale-colleague-edit` | supply agreement | A second turn (a recorded first turn) on a redline that a colleague edited in between: the colleague's change is kept and reported (judged). |
| `not-installed` | supply agreement | stetdoc is not on PATH: the agent says so and stops (judged), and never opens the package another way. |

Every case also checks that the originals are byte-identical unless the task asked otherwise, that the agent never
used `unzip`, a scripting language or the XML (`no-xml-tools`, `no-xml-reads`), and that no tool call failed.

## How grading works

`claude plugin eval` has no custom-code graders: a grader is a regex over the reply, the transcript or a file in the
workspace, a count of tool calls, or a judge model. A `.docx` is a zip, so a regex cannot read it. The suite therefore
loads a second plugin with every case, `evals/harness/`, whose only part is a `Stop` hook. When the agent stops,
`evals/harness/report.sh` writes `.stetdoc-eval-report.txt` in the workspace with:

- `unchanged:`, `changed:`, `deleted:` and `created:` lines, against SHA-256 sums the setup script recorded;
- `leftover-working-copy:` for any `NAME.stet/` folder;
- for every `.docx` the agent created or changed: `valid: yes` when `stetdoc check` passes with no more violations
  than the inputs had or only lints Word accepts (a non-canonical element order, an empty property element, a missing
  paragraph id), every check finding counted by code and part, `remaining-revisions: N`, the full revision ledger (`stetdoc view --revisions --full --all`),
  the comments and the text read as final.

The graders are regexes over that file, so the verdict on a document comes from stetdoc reading it, not from what the
agent says. For example, `redline-net60-new-york` asserts a deletion of thirty and an insertion of sixty by `ER`
(Eval Reviewer, the author the setup saves the way `stetdoc init --author` does) in `p145`, and no revision outside
`p145`, `p268` and `p273`. Judge graders (`llm`) are used only for the quality of an answer. The hook also reads the
session transcript and records the metrics in the report and in `metrics.jsonl`.

## The format, in short

- A case is a directory under the eval directory with `prompt.md` (front matter: `description`, `tags`, `plugins`,
  `runs`, `model`, `max_turns`, `timeout_seconds`, `allowed_tools`, `env` with `EVAL_*` keys only; the body is the
  prompt) and an optional `case.yaml` (`schema_version: "1.1"`, `name`, and `context.scaffold_script`,
  `context.history_file`, `context.add_dirs`).
- Graders are `graders/*.md`, one per file: front matter `type` (`regex`, `tool_used`, `tool_order`, `file_exists`,
  `llm`, `baseline`), `weight`, `arm`, and the type's options. A `regex` grader takes a `target`: `last_message`,
  `trace`, `files` or `{ source: file, path: ... }`.
- A run starts in an empty workspace under a temporary home. Only read-only tools are granted unless the command line
  grants more; the setup script runs only with `--scaffold`; a plugin's MCP servers start only with
  `--allow-real-servers`.
- A plugin a case loads must not sit inside the eval directory, which is why the cases are in `evals/cases/` and
  `evals/without-stetdoc/` and the plugins in `evals/arm` and `evals/harness/`.
- `--max-cost-usd 0` loads and checks every case without running one, which is a free way to validate the suite:
  `claude plugin eval . --eval-dir evals/cases --scaffold --trust-plugin --ablation none --max-cost-usd 0`.

## Arms, and what the format cannot express

`claude plugin eval` has a built-in no-plugin baseline (`--ablation with-without`), but that arm drops every plugin,
the harness too, so it cannot grade the documents. `run.sh` passes `--ablation none` and compares arms by pointing
`evals/arm` at another plugin instead, with the harness still loaded. `--arm none` is the baseline. To compare with
Anthropic's docx skill, make a plugin directory around it and pass that:

```
mkdir -p ~/docx-arm/.claude-plugin ~/docx-arm/skills
echo '{"name": "docx-arm", "version": "0.0.0"}' > ~/docx-arm/.claude-plugin/plugin.json
ln -s /path/to/anthropics/skills/skills/docx ~/docx-arm/skills/docx
evals/run.sh --arm ~/docx-arm
```

That skill uses Python packages that the sandbox must find on `PATH`, and the `no-xml-tools` grader fails it by
design, since it works on the unpacked XML.

Other limits:

- There is no way to exclude a case from a run, so `not-installed` lives in its own eval directory and `run.sh` runs
  it in a second invocation with a different `PATH`. Run it through `run.sh`: its `precondition` grader fails when
  stetdoc was on `PATH`.
- A multi-turn case can only resume a recorded conversation (`context.history_file`); nothing can change the files
  between two live turns. `stale-colleague-edit` therefore records the first turn in `history.jsonl` and makes the
  first turn's redline and the colleague's edit in its setup script. Claude Code resumes that history from a session file it
  writes into the case directory and appends each run to, so the setup script deletes it first, and two runs of this
  case at the same time (`-j` above 1) share one conversation: run it with `--case stale-colleague-edit -j 1` when
  its result matters.
- Tokens are not in Claude Code's result, which is why the harness reads them from the transcript.
- `no-xml-tools` sees the `Bash` tool only; an agent that read the package with another tool would get past it.