claudiomaradonna/agent-foundry
claudiomaradonna/agent-foundry
Template
1
0
Fork 0
Code Issues 5 Pull requests Activity

fix(evals): accept more boundary-bug wordings #149

Merged
claudiomaradonna merged 1 commit from fix/review-pr-grader into main 2026-10-19 20:58:04 +02:00
Conversation 0 Commits 1 Files changed 2 +14 -6
motoko commented 2026-10-19 20:53:38 +02:00
Collaborator
Copy link

Why

The review-pr eval case failed two of three attempts in the last checklist trial even though each change-reviewer named the boundary bug. The reviewers wrote "drops words seen exactly N times" and "keeps words seen more than N times", and the grader's pattern accepted only "excludes" or "leaves out", and "more than" only after "only". The case read as a lost pass and needed a three-attempt rerun to clear.

What changed

  • The grader accepts more correct wordings of the bug: "drops" and "omits" beside "excludes" and "leaves out", and "keeps" beside "only". Every near-miss it rejected before still fails.

How to verify

  • bash tests/agent-config/evals-graders.sh passes 22 tests. With main's evals/cases/review-pr/grade.sh it fails the "drops", "omits", and "keeps" cases of "review-pr passes reviewer reports that name both seeded defects".

These pass on the tip: bash tests/verify-agent-config.sh (263), .agents/hooks/shellcheck.sh repo, and git diff --check main. The new pattern also passes the three saved review-pr attempts of that trial, where main's fails two.

## Why The review-pr eval case failed two of three attempts in the last checklist trial even though each change-reviewer named the boundary bug. The reviewers wrote "drops words seen exactly N times" and "keeps words seen *more than* N times", and the grader's pattern accepted only "excludes" or "leaves out", and "more than" only after "only". The case read as a lost pass and needed a three-attempt rerun to clear. ## What changed - **The grader accepts more correct wordings of the bug:** "drops" and "omits" beside "excludes" and "leaves out", and "keeps" beside "only". Every near-miss it rejected before still fails. ## How to verify - `bash tests/agent-config/evals-graders.sh` passes 22 tests. With main's `evals/cases/review-pr/grade.sh` it fails the "drops", "omits", and "keeps" cases of "review-pr passes reviewer reports that name both seeded defects". These pass on the tip: `bash tests/verify-agent-config.sh` (263), `.agents/hooks/shellcheck.sh repo`, and `git diff --check main`. The new pattern also passes the three saved review-pr attempts of that trial, where main's fails two. 
The review-pr grader failed two change-reviewer reports that named the
boundary bug as dropping words seen exactly N times, or keeping words
seen more than N times. Accept "drops" and "omits" beside "excludes"
and "leaves out", and "keeps" beside "only".
claudiomaradonna deleted branch fix/review-pr-grader 2026-10-19 21:02:19 +02:00
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
claudiomaradonna/agent-foundry!149
No description provided.