refactor(skills): recast four skills as checklists #145

Closed

motoko wants to merge 1 commit from refactor/checklist-skills-2 into main

motoko commented

2026-10-19 16:11:08 +02:00

Collaborator

Why

#142 rewrote execute-plan and verify-execution as checklists and found the result neutral. This second trial tests the form on four more skills: run-reviews, deliver, review-pr, and refactor-code. They're the densest skills the eval cases reach. route-back and fold-branch are as dense, but the cases seldom reach them, so the evals couldn't measure a rewrite.

What changed

One action per step, with a worked example under ## Example in each SKILL.md, in #142's form. Every rule keeps its meaning, scope, and timing; the document reviewer checked the drafts rule by rule over three rounds. Names, descriptions, Overviews, refactor-code's Extensions, and the refactor-code step 1 that AGENTS.md cites are kept.
No example reuses an eval case's change, so none hands an agent a case's answer.

How to verify

git diff main changes only the four SKILL.md files.
The full eval suite ran three times on the tip and three times on main, in the same session:
- Passes: 19/21 on the tip against 21/21 on main. Both failures are review-pr's. In each, the change reviewer named the boundary bug in words the grader doesn't match ("drops words seen exactly N times", "keeps words seen more than N times"). review-pr rerun three times on the tip passed 3/3.
- Cost: $7.25 per suite on the tip against $6.75 on main. Two runs of main's own tree, a day apart, differ by $0.15, and a single case moves by up to $0.35 between them.
- Where the cost rose: refactor-code (+$0.41) and implement-feature (+$0.43). Two refactor-code attempts ran a second change-reviewer round because their reviewer found medium defects the change brought, which no main attempt's reviewer did.

These pass on the tip: bash tests/verify-agent-config.sh (257), bash tests/check-work-folder.sh (46), .agents/hooks/shellcheck.sh repo, prettier --check '**/*.md', scripts/verify-agent-config.sh, and git diff --check main. The figures come from evals/report.sh as fixed in #143, which counts resumed sessions whole.

Considerations

My read: close this rather than merge it. The checklist form costs no case a pass, but it shows no gain, and the suite ran about $0.50 dearer, beyond the noise, though not clearly caused by the skills. Taken with #142's neutral result, three repeats give no reason to make these four skills about 60% longer.
Set aside: two low findings on the refactor-code example's wording, from a later round of the draft review.

## Why #142 rewrote `execute-plan` and `verify-execution` as checklists and found the result neutral. This second trial tests the form on four more skills: `run-reviews`, `deliver`, `review-pr`, and `refactor-code`. They're the densest skills the eval cases reach. `route-back` and `fold-branch` are as dense, but the cases seldom reach them, so the evals couldn't measure a rewrite. ## What changed - **One action per step, with a worked example** under `## Example` in each `SKILL.md`, in #142's form. Every rule keeps its meaning, scope, and timing; the document reviewer checked the drafts rule by rule over three rounds. Names, descriptions, Overviews, `refactor-code`'s Extensions, and the `refactor-code` step 1 that `AGENTS.md` cites are kept. - **No example reuses an eval case's change**, so none hands an agent a case's answer. ## How to verify - `git diff main` changes only the four `SKILL.md` files. - The full eval suite ran three times on the tip and three times on main, in the same session: - **Passes:** 19/21 on the tip against 21/21 on main. Both failures are review-pr's. In each, the change reviewer named the boundary bug in words the grader doesn't match ("drops words seen exactly N times", "keeps words seen *more than* N times"). review-pr rerun three times on the tip passed 3/3. - **Cost:** $7.25 per suite on the tip against $6.75 on main. Two runs of main's own tree, a day apart, differ by $0.15, and a single case moves by up to $0.35 between them. - **Where the cost rose:** refactor-code (+$0.41) and implement-feature (+$0.43). Two refactor-code attempts ran a second change-reviewer round because their reviewer found medium defects the change brought, which no main attempt's reviewer did. These pass on the tip: `bash tests/verify-agent-config.sh` (257), `bash tests/check-work-folder.sh` (46), `.agents/hooks/shellcheck.sh repo`, `prettier --check '**/*.md'`, `scripts/verify-agent-config.sh`, and `git diff --check main`. The figures come from `evals/report.sh` as fixed in #143, which counts resumed sessions whole. ## Considerations - **My read: close this rather than merge it.** The checklist form costs no case a pass, but it shows no gain, and the suite ran about $0.50 dearer, beyond the noise, though not clearly caused by the skills. Taken with #142's neutral result, three repeats give no reason to make these four skills about 60% longer. - **Set aside:** two low findings on the `refactor-code` example's wording, from a later round of the draft review.

motoko added 1 commit

2026-10-19 16:11:15 +02:00

refactor(skills): recast four skills as checklists 634e78765c

Rewrite run-reviews, deliver, review-pr, and refactor-code with one
action per step and a worked example each, keeping every rule, as a
second trial of the checklist form that #142 tried on two skills. The
behavioral evals measure it against main.

claudiomaradonna closed this pull request

2026-10-19 20:41:43 +02:00

Pull request closed

Please reopen this pull request to perform a merge.

No reviewers

No labels

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

claudiomaradonna/agent-foundry!145

No description provided.

Rows
Columns