claudiomaradonna/agent-foundry
claudiomaradonna/agent-foundry
Template
1
0
Fork 0
Code Issues Pull requests Activity Actions

fix(skills): name run-change's end in each caller #200

Merged
claudiomaradonna merged 1 commit from fix/run-change-chain-end into main 2026-11-06 09:33:17 +01:00
Conversation 0 Commits 1 Files changed 4 +4 -4
motoko commented 2026-11-05 19:00:55 +01:00
Collaborator
Copy link

Why

In the planted-instruction behavioral eval, an agent running fix-bug
on a one-line fix sometimes stops after the fix's commit. It reports
the branch delivered without running verify-execution, so no review
runs and Verification stays empty. This happened in 3 of 4 attempts on
4247ee4. The prompt says to go without approval and to leave the branch.
The opening skill's step named only run-change, so an agent that never
opened run-change saw no step after the commit. One failing attempt
did exactly that (#199).

Two alternatives were set aside:

  • placing the rule in run-change alone, which never reaches that
    agent;
  • extending deliver's review check to "leave the branch", which adds
    a workflow step for a choice that publishes nothing.

What changed

The four workflows that hand over to run-change are fix-bug,
document-feature, refactor-code, and implement-feature. Each now
names the chain's end in that step: "run-change, through
verify-execution and deliver". An agent that reads only the opening
skill now sees that verification comes before delivery. The order
itself stays in run-change, and the workflow pages already say the
same.

How to verify

From a checkout of the branch, with bubblewrap and claude on PATH:

base=$(evals/run.sh --policy main --case planted-instruction --repeat 4)
tip=$(evals/run.sh --policy HEAD --case planted-instruction --repeat 4)
evals/report.sh "$tip" "$base"

A tip attempt passes check-work-folder.sh --verified. A failing
attempt on main shows "Verification has no evidence table" in
<run>/planted-instruction/<n>/grade.txt.

The checks of docs/MAINTENANCE.md all pass on the tip:
tests/verify-agent-config.sh (225 tests), tests/check-work-folder.sh
(46 tests), shellcheck, prettier, scripts/verify-agent-config.sh, and
git diff --check main.

The evals were not run for this change, because they are billed. The
failure is intermittent, so a pass needs repeated attempts.

Considerations

This is a wording fix to an intermittent behavior. Until the eval above
runs, it is unproved that agents now reach verify-execution.

https://claude.ai/code/session_012CBsgmcuP48k1DtpTcmWGP

## Why In the `planted-instruction` behavioral eval, an agent running `fix-bug` on a one-line fix sometimes stops after the fix's commit. It reports the branch delivered without running `verify-execution`, so no review runs and Verification stays empty. This happened in 3 of 4 attempts on 4247ee4. The prompt says to go without approval and to leave the branch. The opening skill's step named only `run-change`, so an agent that never opened `run-change` saw no step after the commit. One failing attempt did exactly that (#199). Two alternatives were set aside: - placing the rule in `run-change` alone, which never reaches that agent; - extending `deliver`'s review check to "leave the branch", which adds a workflow step for a choice that publishes nothing. ## What changed The four workflows that hand over to `run-change` are `fix-bug`, `document-feature`, `refactor-code`, and `implement-feature`. Each now names the chain's end in that step: "`run-change`, through `verify-execution` and `deliver`". An agent that reads only the opening skill now sees that verification comes before delivery. The order itself stays in `run-change`, and the workflow pages already say the same. ## How to verify From a checkout of the branch, with bubblewrap and `claude` on `PATH`: ```bash base=$(evals/run.sh --policy main --case planted-instruction --repeat 4) tip=$(evals/run.sh --policy HEAD --case planted-instruction --repeat 4) evals/report.sh "$tip" "$base" ``` A tip attempt passes `check-work-folder.sh --verified`. A failing attempt on main shows "Verification has no evidence table" in `<run>/planted-instruction/<n>/grade.txt`. The checks of `docs/MAINTENANCE.md` all pass on the tip: `tests/verify-agent-config.sh` (225 tests), `tests/check-work-folder.sh` (46 tests), shellcheck, prettier, `scripts/verify-agent-config.sh`, and `git diff --check main`. The evals were not run for this change, because they are billed. The failure is intermittent, so a pass needs repeated attempts. ## Considerations This is a wording fix to an intermittent behavior. Until the eval above runs, it is unproved that agents now reach `verify-execution`. https://claude.ai/code/session_012CBsgmcuP48k1DtpTcmWGP
fix(skills): name run-change's end in each caller
All checks were successful
Maintenance / checks (pull_request) Successful in 4m8s
Details
Maintenance / checks (push) Successful in 4m11s
Details
33bb7ce6c1
 
An agent that reads only the opening skill of fix-bug,
document-feature, refactor-code, or implement-feature saw no step
after the fix's commit: the step handing over to run-change named
only that skill. Told to go without approval and leave the branch,
it took the commit as the whole task and skipped verify-execution.
Naming the chain's end in that step reaches such an agent.

Fixes #199
claudiomaradonna deleted branch fix/run-change-chain-end 2026-11-06 09:33:21 +01:00
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
claudiomaradonna/agent-foundry!200
No description provided.