# Evaluate

Roles:

- Evaluator
- Reviewer

The caller, a lead, supplies as one caller input the change to evaluate (a commit range or a branch), the metrics that matter, and any thresholds and context.
Nothing ships unless its evaluation metrics clear their agreed thresholds; `evaluate` is the verification stage that establishes whether they do.
`evaluate` runs at most two review rounds.
Every commit identity is taken from the repository-effect receipt, not from either player's prose.

### EVALUATE-1

Where every commit identity is taken from the repository-effect receipt, not from either player's prose, when the caller gives a change to evaluate and the metrics that matter, Captain shall prompt Evaluator:

> Evaluate the change named in the request against every metric it names.
> Read specs/ and the code to understand what the change does and what each metric means.
> Take each metric's threshold and direction (whether higher or lower is better) from the request, or from specs/ where the request gives none.
> If a metric's threshold is in neither, ask Boss for it before running anything.
>
> Find the evaluation suite under evals/, and for each named metric that has no script there, write one.
> Each metric gets one runnable script that measures exactly what the metric claims and prints the metric's value.
> Run every script against the change and against the baseline main branch.
> Run the baseline from a separate worktree of main so the current branch is not disturbed, and remove that worktree afterwards.
> For every run, record the exact commit it ran on, the command, and the printed value.
>
> Write a report under docs/evals/ with a table holding one row per named metric and the columns: metric, threshold, baseline value, new value, pass or fail.
> Decide each pass or fail strictly from the metric's threshold, its direction, and the new value.
> Below the table, state the overall verdict (pass if every metric passed, otherwise fail with the failing metrics listed) and the recorded commits and commands of every run.
>
> Commit the suite and the report together in one new commit on the current branch; never amend or rewrite an existing commit.
> Make the commit message say what was evaluated and the overall verdict.
> Report the commit, the table, and the overall verdict.
>
> > Original request: <caller-input>

Results:
- `committed`: Evaluator made one new commit holding the evaluation suite and the report. Output shall include `evaluatorOutput: <verbatim final text>` and `latestCommit: <commit identity>`.

### EVALUATE-2

Where every commit identity is taken from the repository-effect receipt, not from either player's prose, when Evaluator commits the suite and the report, at the first review round, Captain shall prompt Reviewer:

> A review begins of the evaluation committed for the request.
> Read the latest commit's message, the eval scripts under evals/, and the report under docs/evals/, and see Evaluator's report below.
>
> > Original request: <caller-input>
> > Latest commit: <latest-commit>
> > Evaluator output: <evaluator-output>
>
> Check that:
> - every metric the request names has a row in the report's table;
> - each eval script measures what its metric claims, on the right data, and prints that value;
> - the baseline was really run on main: the recorded main commit exists and its values come from running the scripts there, not from copying or estimating them;
> - each pass or fail follows from the metric's threshold, its direction, and the new value, and the overall verdict follows from the rows.
> Re-run a script only where needed to settle a doubt about a recorded value.
>
> Number the findings consistently across rounds, without duplication.
> Flag only what makes a value, a verdict, or the table wrong, unsupported, or incomplete — not style.
> For any rebuttal, accept or challenge it.
> Do not edit files or commit; report findings only.
> If no findings remain, say so and state the report's overall verdict: pass, or fail with the failing metrics listed.

Results:
- `hasFindings`: Reviewer raised or kept one or more numbered findings; a progress report, status update, or promise of a later result supports no outcome. Output shall include `reviewerOutput: <verbatim final text>`.
- `noFindings`: Reviewer affirmatively reported that no findings remain and stated the report's overall verdict; a progress report, status update, or promise of a later result supports no outcome. On this outcome `evaluate` ends and returns latestCommit, the last commit Evaluator made, and the overall verdict as overallVerdict. Output shall include `overallVerdict: <pass, or fail with the failing metrics listed>`.

### EVALUATE-3

Where every commit identity is taken from the repository-effect receipt, not from either player's prose, and `evaluate` runs at most two review rounds, when Evaluator commits fixes for the first round's findings, at the second and final review round, Captain shall prompt Reviewer:

> The second and final review round begins, with particular attention to the latest fix commit.
> Read its message and see Evaluator's feedback below.
>
> > Original request: <caller-input>
> > Latest commit: <latest-commit>
> > Evaluator output: <evaluator-output>
>
> Check that:
> - every metric the request names has a row in the report's table;
> - each eval script measures what its metric claims, on the right data, and prints that value;
> - the baseline was really run on main: the recorded main commit exists and its values come from running the scripts there, not from copying or estimating them;
> - each pass or fail follows from the metric's threshold, its direction, and the new value, and the overall verdict follows from the rows.
> Re-run a script only where needed to settle a doubt about a recorded value.
>
> Number the findings consistently across rounds, without duplication.
> Flag only what makes a value, a verdict, or the table wrong, unsupported, or incomplete — not style.
> For any rebuttal, accept or challenge it.
> Do not edit files or commit; report findings only.
> If no findings remain, say so and state the report's overall verdict: pass, or fail with the failing metrics listed.

Results:
- `hasFindings`: Reviewer raised or kept one or more numbered findings; a progress report, status update, or promise of a later result supports no outcome. Output shall include `reviewerOutput: <verbatim final text>`.
- `noFindings`: Reviewer affirmatively reported that no findings remain and stated the report's overall verdict; a progress report, status update, or promise of a later result supports no outcome. On this outcome `evaluate` ends and returns latestCommit, the last commit Evaluator made, and the overall verdict as overallVerdict. Output shall include `overallVerdict: <pass, or fail with the failing metrics listed>`.

### EVALUATE-4

Where every commit identity is taken from the repository-effect receipt, not from either player's prose, and `evaluate` runs at most two review rounds, when Evaluator rejects every finding of the first round and makes no commit, at the second and final review round, Captain shall prompt Reviewer:

> The second and final review round begins.
> No new commit was made because Evaluator rejected every finding; see Evaluator's feedback below.
>
> > Original request: <caller-input>
> > Evaluator output: <evaluator-output>
>
> Check that:
> - every metric the request names has a row in the report's table;
> - each eval script measures what its metric claims, on the right data, and prints that value;
> - the baseline was really run on main: the recorded main commit exists and its values come from running the scripts there, not from copying or estimating them;
> - each pass or fail follows from the metric's threshold, its direction, and the new value, and the overall verdict follows from the rows.
> Re-run a script only where needed to settle a doubt about a recorded value.
>
> Number the findings consistently across rounds, without duplication.
> Flag only what makes a value, a verdict, or the table wrong, unsupported, or incomplete — not style.
> For any rebuttal, accept or challenge it.
> Do not edit files or commit; report findings only.
> If no findings remain, say so and state the report's overall verdict: pass, or fail with the failing metrics listed.

Results:
- `hasFindings`: Reviewer raised or kept one or more numbered findings; a progress report, status update, or promise of a later result supports no outcome. Output shall include `reviewerOutput: <verbatim final text>`.
- `noFindings`: Reviewer affirmatively reported that no findings remain and stated the report's overall verdict; a progress report, status update, or promise of a later result supports no outcome. On this outcome `evaluate` ends and returns latestCommit, the last commit Evaluator made, and the overall verdict as overallVerdict. Output shall include `overallVerdict: <pass, or fail with the failing metrics listed>`.

### EVALUATE-5

The second review round follows either result of this behavior.

Where every commit identity is taken from the repository-effect receipt, not from either player's prose, when Reviewer raises or keeps any finding in the first review round, Captain shall prompt Evaluator:

> For each numbered finding, accept or reject it.
> If you accept a finding, fix its root cause in the eval scripts or the report; re-run, against both the change and main, only the scripts whose inputs changed, and bring the table, each pass or fail, and the overall verdict in line with the results.
> If you reject a finding, give the reasoning and cite the script, the code, or the run output that supports it.
>
> If you accept any finding, add one new commit with the fixes on the current branch; never amend or rewrite an existing commit.
> If you reject every finding, change nothing and make no commit.
> Report every disposition, every rebuttal, and the overall verdict the report now states.
>
> > Original request: <caller-input>
> > Reviewer findings: <reviewer-output>

Results:
- `fixed`: Evaluator accepted one or more findings and made one new fix commit. Output shall include `evaluatorOutput: <verbatim final text>` and `latestCommit: <commit identity>`.
- `rejectedAll`: Evaluator rejected every finding, changed nothing, and made no commit. Output shall include `evaluatorOutput: <verbatim final text>`.

### EVALUATE-6

`evaluate` runs at most two review rounds, so it ends on either result of this behavior without a third review.

Where every commit identity is taken from the repository-effect receipt, not from either player's prose, when Reviewer raises or keeps any finding in the second review round, Captain shall prompt Evaluator:

> For each numbered finding, accept or reject it.
> If you accept a finding, fix its root cause in the eval scripts or the report; re-run, against both the change and main, only the scripts whose inputs changed, and bring the table, each pass or fail, and the overall verdict in line with the results.
> If you reject a finding, give the reasoning and cite the script, the code, or the run output that supports it.
>
> If you accept any finding, add one new commit with the fixes on the current branch; never amend or rewrite an existing commit.
> If you reject every finding, change nothing and make no commit.
> Report every disposition, every rebuttal, and the overall verdict the report now states.
>
> > Original request: <caller-input>
> > Reviewer findings: <reviewer-output>

Results:
- `fixed`: Evaluator accepted one or more findings and made one new fix commit. On this outcome `evaluate` ends and returns latestCommit, this unreviewed fix commit; the overall verdict the report now states, as overallVerdict; and Evaluator's dispositions of the last findings, as evaluatorOutput. Output shall include `evaluatorOutput: <verbatim final text>`, `latestCommit: <commit identity>`, and `overallVerdict: <pass, or fail with the failing metrics listed>`.
- `rejectedAll`: Evaluator rejected every finding, changed nothing, and made no commit. On this outcome `evaluate` ends and returns latestCommit, the last commit Evaluator made; the overall verdict the report states, as overallVerdict; and Evaluator's rebuttals of the unsettled findings, as evaluatorOutput. Output shall include `evaluatorOutput: <verbatim final text>` and `overallVerdict: <pass, or fail with the failing metrics listed>`.
