# Evaluate

Roles:

- Evaluator
- Reviewer

The caller, a lead, supplies as one caller input the change to evaluate (a commit range or a branch), the metrics that matter, and any thresholds and context.
Nothing ships unless its evaluation metrics clear their agreed thresholds; `evaluate` is the verification stage that establishes whether they do.
`evaluate` runs at most two review rounds.
Captain shall take every commit identity from the repository-effect receipt, not from either player's prose.

When the caller gives a change to evaluate and the metrics that matter, Captain shall prompt Evaluator with the following instruction:

```markdown
Evaluate the change named in the request against every metric it names.
Read specs/ and the code to understand what the change does and what each metric means.
Take each metric's threshold and direction (whether higher or lower is better) from the request, or from specs/ where the request gives none.
If a metric's threshold is in neither, ask Boss for it before running anything.

Find the evaluation suite under evals/, and for each named metric that has no script there, write one.
Each metric gets one runnable script that measures exactly what the metric claims and prints the metric's value.
Run every script against the change and against the baseline main branch.
Run the baseline from a separate worktree of main so the current branch is not disturbed, and remove that worktree afterwards.
For every run, record the exact commit it ran on, the command, and the printed value.

Write a report under docs/evals/ with a table holding one row per named metric and the columns: metric, threshold, baseline value, new value, pass or fail.
Decide each pass or fail strictly from the metric's threshold, its direction, and the new value.
Below the table, state the overall verdict (pass if every metric passed, otherwise fail with the failing metrics listed) and the recorded commits and commands of every run.

Commit the suite and the report together in one new commit on the current branch; never amend or rewrite an existing commit.
Make the commit message say what was evaluated and the overall verdict.
Report the commit, the table, and the overall verdict.
```

Captain shall append the caller's input in quotes (`>`):

> Original request: <caller-input>

Results:
- `committed`: Evaluator made one new commit holding the evaluation suite and the report. Output shall include `evaluatorOutput: <verbatim final text>` and `latestCommit: <commit identity>`.

After Evaluator commits the suite and the report, at the first review round, Captain shall give Reviewer the following instruction:

```markdown
A review begins of the evaluation committed for the request.
Read the latest commit's message, the eval scripts under evals/, and the report under docs/evals/, and see Evaluator's report below.
```

Captain shall append the round's context in quotes (`>`):

> Original request: <caller-input>
> Latest commit: <latest-commit>
> Evaluator output: <evaluator-output>

After Evaluator commits fixes for the first round's findings, at the second review round, Captain shall give Reviewer the following instruction:

```markdown
The second and final review round begins, with particular attention to the latest fix commit.
Read its message and see Evaluator's feedback below.
```

Captain shall append the round's context in quotes (`>`):

> Original request: <caller-input>
> Latest commit: <latest-commit>
> Evaluator output: <evaluator-output>

After Evaluator rejects every finding of the first round and makes no commit, at the second review round, Captain shall give Reviewer the following instruction:

```markdown
The second and final review round begins.
No new commit was made because Evaluator rejected every finding; see Evaluator's feedback below.
```

Captain shall append the round's context in quotes (`>`):

> Original request: <caller-input>
> Evaluator output: <evaluator-output>

At the start of every review round, Captain shall append the following instruction to the end of Reviewer's prompt:

```markdown
Check that:
- every metric the request names has a row in the report's table;
- each eval script measures what its metric claims, on the right data, and prints that value;
- the baseline was really run on main: the recorded main commit exists and its values come from running the scripts there, not from copying or estimating them;
- each pass or fail follows from the metric's threshold, its direction, and the new value, and the overall verdict follows from the rows.
Re-run a script only where needed to settle a doubt about a recorded value.

Number the findings consistently across rounds, without duplication.
Flag only what makes a value, a verdict, or the table wrong, unsupported, or incomplete — not style.
For any rebuttal, accept or challenge it.
Do not edit files or commit; report findings only.
If no findings remain, say so and state the report's overall verdict: pass, or fail with the failing metrics listed.
```

Every review round has these results:

Results:
- `hasFindings`: Reviewer raised or kept one or more numbered findings; a progress report, status update, or promise of a later result supports no outcome. Output shall include `reviewerOutput: <verbatim final text>`.
- `noFindings`: Reviewer affirmatively reported that no findings remain and stated the report's overall verdict; a progress report, status update, or promise of a later result supports no outcome. On this outcome `evaluate` ends and returns latestCommit, the last commit Evaluator made, and the overall verdict as overallVerdict. Output shall include `overallVerdict: <pass, or fail with the failing metrics listed>`.

When Reviewer raises or keeps any finding in a review round, Captain shall prompt Evaluator with the following instruction:

```markdown
For each numbered finding, accept or reject it.
If you accept a finding, fix its root cause in the eval scripts or the report; re-run, against both the change and main, only the scripts whose inputs changed, and bring the table, each pass or fail, and the overall verdict in line with the results.
If you reject a finding, give the reasoning and cite the script, the code, or the run output that supports it.

If you accept any finding, add one new commit with the fixes on the current branch; never amend or rewrite an existing commit.
If you reject every finding, change nothing and make no commit.
Report every disposition, every rebuttal, and the overall verdict the report now states.
```

Captain shall append the round's context in quotes (`>`):

> Original request: <caller-input>
> Reviewer findings: <reviewer-output>

After the first review round, this behavior has these results, and the second review round follows either one:

Results:
- `fixed`: Evaluator accepted one or more findings and made one new fix commit. Output shall include `evaluatorOutput: <verbatim final text>` and `latestCommit: <commit identity>`.
- `rejectedAll`: Evaluator rejected every finding, changed nothing, and made no commit. Output shall include `evaluatorOutput: <verbatim final text>`.

After the second review round, this behavior has these results, and `evaluate` ends on either one without a third review:

Results:
- `fixed`: Evaluator accepted one or more findings and made one new fix commit. On this outcome `evaluate` ends and returns latestCommit, this unreviewed fix commit; the overall verdict the report now states, as overallVerdict; and Evaluator's dispositions of the last findings, as evaluatorOutput. Output shall include `evaluatorOutput: <verbatim final text>`, `latestCommit: <commit identity>`, and `overallVerdict: <pass, or fail with the failing metrics listed>`.
- `rejectedAll`: Evaluator rejected every finding, changed nothing, and made no commit. On this outcome `evaluate` ends and returns latestCommit, the last commit Evaluator made; the overall verdict the report states, as overallVerdict; and Evaluator's rebuttals of the unsettled findings, as evaluatorOutput. Output shall include `evaluatorOutput: <verbatim final text>` and `overallVerdict: <pass, or fail with the failing metrics listed>`.
