test(evals): add a behavioral eval suite #133

Merged

claudiomaradonna merged 5 commits from feat/behavioral-evals into main 2026-10-17 15:08:54 +02:00

Conversation 0 Commits 5 Files changed 32 +1233 -15

motoko commented 2026-10-17 15:06:31 +02:00

Collaborator

Copy link

Why

The source's checks are static: they confirm that the policy's files exist and agree with each other. Nothing shows what agents actually do under the policy, or what a run costs. So a rewording of AGENTS.md or a skill can't be judged better or worse.

What changed

• evals/run.sh runs six fixed cases with Claude Code headless:

  • fix-bug, implement-feature, document-feature and refactor-code;
  • a question, which should run no workflow;
  • a bug report carrying a planted instruction.

  Each case runs on a fresh toy repository with one commit's policy installed, inside a bubblewrap sandbox.

• Each case's grader checks three things: the outcome, the process the policy asks for (its own branch and worktree, main untouched), and that the work folder passes check-work-folder.sh --verified. The graders run in a second sandbox with no network.

• evals/report.sh tabulates passes, turns, tokens, cost and time. Given a second run, such as one of main, it compares the two.

• The README's Maintenance section says to run the suite before merging a change to AGENTS.md, a skill, a template or a brief.

Out of scope: Codex and OpenCode runners, and running the suite in CI, since each run is billed and nondeterministic.

How to verify

• bash tests/verify-agent-config.sh runs the new runner, grader and report tests with a stub agent. They are free and offline, and skip where bwrap can't start.
• evals/run.sh --policy main --case question takes about 20 s and $0.20. evals/report.sh <printed dir> should then show 1/1.

Already run on the tip: all of the README's Maintenance checks pass. One billed run of all six cases against main passed 6/6, for $9.99 and about 34 minutes. Two of the six first failed because their graders were too strict about what the policy allows; I fixed the graders and regraded the saved attempts.

Considerations

• Your decision: in containers where bwrap can't mount its own /proc (this one included), the agent gets the host's /proc read-only. Without one, the agent's runtime aborts. In that mode the agent can read the command lines of host processes, but not their environments. The runner warns about this on stderr.
• When ANTHROPIC_API_KEY is unset, the sandbox gets a copy of the account's credentials file. The runner deletes it when the attempt ends or the runner is stopped. A token refresh inside the sandbox might rotate the host's token; none was seen.
• Turns count only the main session; work done by subagents shows up in cost and tokens.
• I left one low review note unfixed: calling timeout by name instead of by its resolved path would change nothing.

## Why The source's checks are static: they confirm that the policy's files exist and agree with each other. Nothing shows what agents actually do under the policy, or what a run costs. So a rewording of `AGENTS.md` or a skill can't be judged better or worse. ## What changed - `evals/run.sh` runs six fixed cases with Claude Code headless: - `fix-bug`, `implement-feature`, `document-feature` and `refactor-code`; - a question, which should run no workflow; - a bug report carrying a planted instruction. Each case runs on a fresh toy repository with one commit's policy installed, inside a bubblewrap sandbox. - Each case's grader checks three things: the outcome, the process the policy asks for (its own branch and worktree, main untouched), and that the work folder passes `check-work-folder.sh --verified`. The graders run in a second sandbox with no network. - `evals/report.sh` tabulates passes, turns, tokens, cost and time. Given a second run, such as one of main, it compares the two. - The README's Maintenance section says to run the suite before merging a change to `AGENTS.md`, a skill, a template or a brief. Out of scope: Codex and OpenCode runners, and running the suite in CI, since each run is billed and nondeterministic. ## How to verify - `bash tests/verify-agent-config.sh` runs the new runner, grader and report tests with a stub agent. They are free and offline, and skip where bwrap can't start. - `evals/run.sh --policy main --case question` takes about 20 s and $0.20. `evals/report.sh <printed dir>` should then show 1/1. Already run on the tip: all of the README's Maintenance checks pass. One billed run of all six cases against main passed 6/6, for $9.99 and about 34 minutes. Two of the six first failed because their graders were too strict about what the policy allows; I fixed the graders and regraded the saved attempts. ## Considerations - **Your decision:** in containers where bwrap can't mount its own `/proc` (this one included), the agent gets the host's `/proc` read-only. Without one, the agent's runtime aborts. In that mode the agent can read the command lines of host processes, but not their environments. The runner warns about this on stderr. - When `ANTHROPIC_API_KEY` is unset, the sandbox gets a copy of the account's credentials file. The runner deletes it when the attempt ends or the runner is stopped. A token refresh inside the sandbox might rotate the host's token; none was seen. - Turns count only the main session; work done by subagents shows up in cost and tokens. - I left one low review note unfixed: calling `timeout` by name instead of by its resolved path would change nothing.

motoko added 5 commits 2026-10-17 15:06:36 +02:00

test(evals): add the eval fixture, cases, and installer ... e8c5e3f0cb

Add a toy Bash repository with a seeded defect, six case prompts,
and a script that installs a commit's policy into a fixture copy in
initialized state, for the behavioral eval suite.

test(evals): add the sandboxed eval runner ... 7a5ce6e5d0

Run each case's prompt headless with the agent in a bubblewrap
sandbox that keeps the network for the API, grade the result in a
second sandbox without it, and save each attempt's output. Move
needs_bwrap into the shared test library so both sandbox tests use it.

test(evals): grade each eval case ... 2c1de2fb54

Add the checks graders share and a grader per case for its outcome,
the process the policy asks for, and a verified work folder, tested
on canned good and bad attempts.

test(evals): report and compare eval runs ... 757e408556

Print each case's passes and mean turns, tokens, cost, and time as a
Markdown table, with changes against a base run such as main's.

docs(readme): describe the behavioral evals ... d317dac472

Say when maintainers run the eval suite, how to compare a change with
the main branch, and what a run costs.

claudiomaradonna merged commit d317dac472 into main 2026-10-17 15:08:54 +02:00

claudiomaradonna deleted branch feat/behavioral-evals 2026-10-17 15:08:59 +02:00

motoko referenced this pull request 2026-10-17 15:57:36 +02:00

feat: confirm fixes in the later round, guard main #135

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels

No items

No labels

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

claudiomaradonna/agent-foundry!133

No description provided.