refactor(skills): rewrite two skills as checklists #142

Merged

claudiomaradonna merged 1 commit from feat/checklist-skills into main

2026-10-19 14:10:26 +02:00

motoko commented

2026-10-19 13:29:44 +02:00

Collaborator

Why

The first workflow review of this series recommended rewriting the skills as short checklists, each with one worked example, rather than dense prose. Nobody had measured whether that helps under this policy. This trial rewrites the two longest skills the eval cases reach, execute-plan and verify-execution, and measures them against main's dense form. It merges only if no case passes less often.

What changed

One action per step. Each step that can fail names its check and where to go next. Every rule of the dense form keeps its meaning, scope, and timing. The document reviewer checked this rule by rule over two rounds of drafts.
A worked example under ## Example in each SKILL.md, inline so it loads with the steps. This goes past the skill template's size target, as decided for the trial; the two skills are about 60% longer.
execute-plan step 4 is still the proof step, since verify-execution and review-pr point to it by number.
Non-goals: the other skills, and the review's idea of allowing one-line restatements of rules kept elsewhere.

How to verify

git diff main shows that only the two SKILL.md files change, with names, descriptions, and Overviews kept.
Both sides of the eval comparison ran the full suite three times, in the same session:
- The branch, measured twice (once before and once after an example rewording), passes every case: 21/21, then 20/21. The one failure was the grader's, not the agent's (see Considerations), and refactor-code rerun three times on each side passed 3/3 on both.
- Suite cost: $6.72 and $6.90 on the branch, against $6.61 on main.
- planted-instruction costs $0.61 to $0.92 in all six branch attempts, against $1.10 to $1.35 on main. Nothing else differs beyond run-to-run noise.

These pass on the tip: bash tests/verify-agent-config.sh (257), bash tests/check-work-folder.sh (46), .agents/hooks/shellcheck.sh repo, prettier --check '**/*.md', scripts/verify-agent-config.sh, and git diff --check main.

Considerations

The result is neutral. The checklist form costs no case a pass, but it shows no gain beyond noise either, apart from planted-instruction's gap. fix-bug runs the same chain and shows no such gap. Merging trades longer skills for that one possible gain; closing this PR, and keeping the dense form, would be just as defensible.
Turns and tokens are left out of the reading. evals/report.sh undercounts them whenever a background task resumes a session, which happened in 14 of the 42 first-run attempts.
The one failed attempt was the grader's. The agent's new test fed files through bash process substitution, which needs /dev/fd, and on this host the grader's sandbox has no /proc. The same tests pass outside that sandbox.
Set aside: a finding that two steps still hold several actions each. That is the form the evals measured.

## Why The first workflow review of this series recommended rewriting the skills as short checklists, each with one worked example, rather than dense prose. Nobody had measured whether that helps under this policy. This trial rewrites the two longest skills the eval cases reach, `execute-plan` and `verify-execution`, and measures them against main's dense form. It merges only if no case passes less often. ## What changed - **One action per step.** Each step that can fail names its check and where to go next. Every rule of the dense form keeps its meaning, scope, and timing. The document reviewer checked this rule by rule over two rounds of drafts. - **A worked example under `## Example`** in each `SKILL.md`, inline so it loads with the steps. This goes past the skill template's size target, as decided for the trial; the two skills are about 60% longer. - **`execute-plan` step 4 is still the proof step,** since `verify-execution` and `review-pr` point to it by number. - **Non-goals:** the other skills, and the review's idea of allowing one-line restatements of rules kept elsewhere. ## How to verify - `git diff main` shows that only the two `SKILL.md` files change, with names, descriptions, and Overviews kept. - Both sides of the eval comparison ran the full suite three times, in the same session: - The branch, measured twice (once before and once after an example rewording), passes every case: 21/21, then 20/21. The one failure was the grader's, not the agent's (see Considerations), and `refactor-code` rerun three times on each side passed 3/3 on both. - Suite cost: $6.72 and $6.90 on the branch, against $6.61 on main. - `planted-instruction` costs $0.61 to $0.92 in all six branch attempts, against $1.10 to $1.35 on main. Nothing else differs beyond run-to-run noise. These pass on the tip: `bash tests/verify-agent-config.sh` (257), `bash tests/check-work-folder.sh` (46), `.agents/hooks/shellcheck.sh repo`, `prettier --check '**/*.md'`, `scripts/verify-agent-config.sh`, and `git diff --check main`. ## Considerations - **The result is neutral.** The checklist form costs no case a pass, but it shows no gain beyond noise either, apart from `planted-instruction`'s gap. `fix-bug` runs the same chain and shows no such gap. Merging trades longer skills for that one possible gain; closing this PR, and keeping the dense form, would be just as defensible. - **Turns and tokens are left out of the reading.** `evals/report.sh` undercounts them whenever a background task resumes a session, which happened in 14 of the 42 first-run attempts. - **The one failed attempt was the grader's.** The agent's new test fed files through bash process substitution, which needs `/dev/fd`, and on this host the grader's sandbox has no `/proc`. The same tests pass outside that sandbox. - **Set aside:** a finding that two steps still hold several actions each. That is the form the evals measured.

motoko added 1 commit

2026-10-19 13:29:49 +02:00

refactor(skills): rewrite two skills as checklists 49a063849f

execute-plan and verify-execution now give their steps one by one, each
with its check, and a worked example under ## Example, keeping every
rule of their dense form. The behavioral evals measure the two forms
side by side, to decide whether the rest of the skills follow.