fix(evals): refuse an output folder holding attempts #139

Merged

claudiomaradonna merged 1 commit from fix/refuse-reused-eval-out into main

2026-10-18 21:53:11 +02:00

motoko commented

2026-10-18 09:16:04 +02:00

Collaborator

Why

evals/run.sh would prepare an attempt folder that already existed. That happened when you reran into the same --out or named a case twice. Each time, the runner reran git init, git add and git commit in the old attempt's repository on the host. Any Git hook or core.fsmonitor that the earlier sandboxed agent wrote there then ran as you, outside both sandboxes.

What changed

A reused --out stops the run. If --out already holds an entry for a case to run, run.sh stops with exit 2 before running anything.
An existing attempt folder stops the run. run.sh stops before preparing an attempt folder that already exists, which catches a case named twice, with or without --out.
Case names can't hold a slash. A name like x/../sample could otherwise reach another case's folder.

Rerunning a case still works with --repeat or a new --out.

How to verify

bash tests/agent-config/evals-runner.sh (needs bwrap). Its test "refuses an output folder, a case named twice, or a case name with a slash, …" uses an agent that plants a pre-commit hook:

On main, the hook runs on the host.
On this branch, all three routes stop with exit 2, and the hook never runs.

These pass on the tip:

bash tests/verify-agent-config.sh (257) and bash tests/check-work-folder.sh (46);
.agents/hooks/shellcheck.sh repo, prettier --check '**/*.md', scripts/verify-agent-config.sh, and git diff --check main.

Considerations

Not documented in the usage header: the header names only the reused --out, not the case-named-twice and slash refusals. I set that aside as low, since the error messages say what to do.
Shared folders keep the latest policy: cases run into one --out with different policies leave the last run's policy file there. That limit already exists and is outside this fix.

## Why `evals/run.sh` would prepare an attempt folder that already existed. That happened when you reran into the same `--out` or named a case twice. Each time, the runner reran `git init`, `git add` and `git commit` in the old attempt's repository on the host. Any Git hook or `core.fsmonitor` that the earlier sandboxed agent wrote there then ran as you, outside both sandboxes. ## What changed - **A reused `--out` stops the run.** If `--out` already holds an entry for a case to run, `run.sh` stops with exit 2 before running anything. - **An existing attempt folder stops the run.** `run.sh` stops before preparing an attempt folder that already exists, which catches a case named twice, with or without `--out`. - **Case names can't hold a slash.** A name like `x/../sample` could otherwise reach another case's folder. Rerunning a case still works with `--repeat` or a new `--out`. ## How to verify `bash tests/agent-config/evals-runner.sh` (needs bwrap). Its test "refuses an output folder, a case named twice, or a case name with a slash, …" uses an agent that plants a `pre-commit` hook: - **On main**, the hook runs on the host. - **On this branch**, all three routes stop with exit 2, and the hook never runs. These pass on the tip: - `bash tests/verify-agent-config.sh` (257) and `bash tests/check-work-folder.sh` (46); - `.agents/hooks/shellcheck.sh repo`, `prettier --check '**/*.md'`, `scripts/verify-agent-config.sh`, and `git diff --check main`. ## Considerations - **Not documented in the usage header:** the header names only the reused `--out`, not the case-named-twice and slash refusals. I set that aside as low, since the error messages say what to do. - **Shared folders keep the latest policy:** cases run into one `--out` with different policies leave the last run's `policy` file there. That limit already exists and is outside this fix.

motoko added 1 commit

2026-10-18 09:16:09 +02:00

fix(evals): refuse an output folder holding attempts 88f3ec62e6

Rerunning into an --out that held a case's attempts, or naming a case
twice, prepared an old attempt's repository again on the host, keeping
the Git hooks and config its agent wrote, which the runner's own
commits then ran outside the sandbox. run.sh now stops before running
anything when --out holds an entry for a case to run, stops before
preparing an attempt folder that exists, and refuses a case name
holding a slash, which could reach another case's folder.

claudiomaradonna merged commit 88f3ec62e6 into main

2026-10-18 21:53:11 +02:00

claudiomaradonna deleted branch fix/refuse-reused-eval-out

2026-10-18 21:53:16 +02:00

No reviewers

No labels

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

claudiomaradonna/agent-foundry!139

No description provided.

Rows
Columns