fix(evals): count a resumed session's figures whole #143

Merged

claudiomaradonna merged 2 commits from fix/report-figures into main

2026-10-19 15:30:31 +02:00

motoko commented

2026-10-19 15:07:18 +02:00

Collaborator

Why

evals/report.sh undercounted the turns and tokens of any attempt whose session a background task's notification resumed: 14 of the 42 attempts in the checklist trial of #142. For those attempts, Claude Code's result.json holds only the last segment's num_turns and usage, while its cost covers the whole run. That made the Turns and token columns unreadable for comparing policies.

What changed

Turns and tokens count the whole main session. They now come from the session's transcript, found by result.json's session_id: a turn for each user entry, and each assistant message's usage counted once. They still cover the main session alone, and the cost still comes from result.json. On the 28 attempts that were not resumed, the transcript gives exactly the figures result.json does, so those rows don't change.
What the sandboxed agent wrote can't forge or hang the report. The cost is escaped again, as on main, so a crafted value can't add rows. A transcript that isn't a regular file, such as a FIFO, is skipped instead of read.

How to verify

bash tests/agent-config/evals-report.sh passes 6 tests. The test "reports a resumed session's turns and tokens whole" fails on main, where the report shows | fix-bug | 1/1 | 2 | 10 | 5 | 1.00 | 60 |, the last segment's figures.
On an earlier pair of runs, evals/report.sh <tip run> <base run> changes only the rows of cases holding a resumed attempt: fix-bug, implement-feature, refactor-code, and review-pr.

These pass on the tip: bash tests/verify-agent-config.sh (260), bash tests/check-work-folder.sh (46), .agents/hooks/shellcheck.sh repo, prettier --check '**/*.md', scripts/verify-agent-config.sh, and git diff --check main.

Considerations

An agent can still set its own turn and token figures by rewriting its transcript, since its home folder is writable. On main it could already do the same through result.json. The figures are for comparing policies, not grading, and the pass or fail verdict still comes from the grader.

## Why `evals/report.sh` undercounted the turns and tokens of any attempt whose session a background task's notification resumed: 14 of the 42 attempts in the checklist trial of #142. For those attempts, Claude Code's `result.json` holds only the last segment's `num_turns` and `usage`, while its cost covers the whole run. That made the Turns and token columns unreadable for comparing policies. ## What changed - **Turns and tokens count the whole main session.** They now come from the session's transcript, found by `result.json`'s `session_id`: a turn for each user entry, and each assistant message's usage counted once. They still cover the main session alone, and the cost still comes from `result.json`. On the 28 attempts that were not resumed, the transcript gives exactly the figures `result.json` does, so those rows don't change. - **What the sandboxed agent wrote can't forge or hang the report.** The cost is escaped again, as on main, so a crafted value can't add rows. A transcript that isn't a regular file, such as a FIFO, is skipped instead of read. ## How to verify - `bash tests/agent-config/evals-report.sh` passes 6 tests. The test "reports a resumed session's turns and tokens whole" fails on main, where the report shows `| fix-bug | 1/1 | 2 | 10 | 5 | 1.00 | 60 |`, the last segment's figures. - On an earlier pair of runs, `evals/report.sh <tip run> <base run>` changes only the rows of cases holding a resumed attempt: fix-bug, implement-feature, refactor-code, and review-pr. These pass on the tip: `bash tests/verify-agent-config.sh` (260), `bash tests/check-work-folder.sh` (46), `.agents/hooks/shellcheck.sh repo`, `prettier --check '**/*.md'`, `scripts/verify-agent-config.sh`, and `git diff --check main`. ## Considerations - **An agent can still set its own turn and token figures** by rewriting its transcript, since its home folder is writable. On main it could already do the same through `result.json`. The figures are for comparing policies, not grading, and the pass or fail verdict still comes from the grader.

motoko added 2 commits

2026-10-19 15:07:23 +02:00

test(evals): give the report's fixtures a transcript 4de946c7f1

fix(evals): count a resumed session's figures whole d7036ed199

When a background task's notification resumes an attempt's session,
result.json holds only the segment after it, so report.sh undercounted
that attempt's turns and tokens. Read them from the main session's
transcript instead, which holds every segment; the cost stays from
result.json, which totals the whole run.