claudiomaradonna/azkaban
1
1
Fork 0
Code Issues Pull requests Activity

test: name tests by behavior and fold duplicates #36

Merged
claudiomaradonna merged 3 commits from test/behavior-names into master 2026-10-07 20:53:27 +02:00
Conversation 0 Commits 3 Files changed 14 +206 -509
motoko commented 2026-10-07 20:51:47 +02:00
Collaborator
Copy link

Why

The Agent Foundry upgrade to c46008c added the rule that each regression
test's name states the behavior it guards and that a behavior has one
test. Some names claimed checks their tests don't make or hid what they
check, and several checks were tested more than once, often outside the
suite that owns them.

What changed

  • 17 tests are renamed to state what they check; their bodies are unchanged.
  • Tests that repeated one check with the same or other inputs are folded
    into one: the key-agent mount, workspace-file mounts, Claude's repeat
    dry run, Codex's dry-run/real pairs, malformed env lines, reserved
    registry ids, and the host-sensitive name list.
  • Each core check, the client state path, the identity's pieces, the MCP
    folder, and per-identity state, is tested once in the suite that owns
    it, with the modes and cases the copies held.
  • The suite goes from 322 to 304 tests; production code is unchanged.
    Overlaps that keep a distinct remainder stay as they are.

Verification

make lint && .agents/hooks/shellcheck.sh repo && make test: 33 files
clean, 304 tests, 0 failures. Removing each folded check from
bin/azkaban makes its remaining test fail.

## Why The Agent Foundry upgrade to c46008c added the rule that each regression test's name states the behavior it guards and that a behavior has one test. Some names claimed checks their tests don't make or hid what they check, and several checks were tested more than once, often outside the suite that owns them. ## What changed - 17 tests are renamed to state what they check; their bodies are unchanged. - Tests that repeated one check with the same or other inputs are folded into one: the key-agent mount, workspace-file mounts, Claude's repeat dry run, Codex's dry-run/real pairs, malformed env lines, reserved registry ids, and the host-sensitive name list. - Each core check, the client state path, the identity's pieces, the MCP folder, and per-identity state, is tested once in the suite that owns it, with the modes and cases the copies held. - The suite goes from 322 to 304 tests; production code is unchanged. Overlaps that keep a distinct remainder stay as they are. ## Verification `make lint && .agents/hooks/shellcheck.sh repo && make test`: 33 files clean, 304 tests, 0 failures. Removing each folded check from `bin/azkaban` makes its remaining test fail. 
Rename the tests whose names left out, overstated, or hid the behavior
their bodies check, so each name states what it guards. Bodies are
unchanged.

Names that claimed an unchecked behavior now drop it: the failing
Podman engine probe clears its value but never checked the error it
kept, the unreadable identity env is a directory, and the Codex paths
with spaces are never checked as preserved. Names that left a checked
behavior out now carry it: the host run forces the locale, the machine
mount probe also fails without a machine, the installer rerun removes
every other release entry, and Codex's first run seeds no state entry.

Opaque or historical wording now says what happens: runtime
"resolution", the "nearer layer", the "former pieces", the "parser
reading", "unsafe paths", "still rejected", "works", "too", and the
"active controls and memory". Client file means env.<client> and
identity env means env, as the suite already uses them.
Each change keeps one test per behavior, where another test already
guarded the same check with the same or other inputs:

- The identity key agent mount (runtime) is every assertion of the
  per-launch agent test in the identity suite; delete it.
- AZKABAN_MOUNTS in the workspace file (runtime) is the same launch as
  the last block of the named identity mounts test; delete it.
- Claude's dry run after a first run repeats the dry run over existing
  state in the never-overwritten test; delete it.
- Codex's regular file at the state path runs as a dry run and a real
  run in one test, as Claude's does. The two regular-MCP-path tests
  become one in the MCP suite, also in both modes, since MCP is core.
- Shell assignment syntax and invalid keys hit the same diagnostic, so
  one test now covers malformed assignment lines; the marker the
  shell-syntax test checked was never written by any of its lines.
- The reserved registry ids all and azkaban share one test.
- One host-sensitive test holds both halves of the name list, which
  the runtime and passthrough suites tested apart, and its BASH_ENV
  value names a startup file that marks the host if it runs, which a
  separate test used to check.

Removing BASH_* and PODMAN_* from the host-sensitive list, or the MCP
not-a-directory check, makes the folded tests fail.
The core checks the client state path, the identity's pieces, and the
MCP folder for every client, yet the adapter and identity suites each
tested them again. One test per check now runs in the suite that owns
it, and gains the cases and modes the copies held:

- A symlink on the client state path: the runtime test now covers
  state/, state/<identity> and state/<identity>/<client>, as a dry run
  and a real launch, and checks nothing is written through the link.
  The Claude, Codex and named identity copies go, and OpenCode's matrix
  keeps the paths its adapter checks.
- An identity's env and env.<client>: the identity suite's piece test
  now covers them beside key and home/, in both modes, with no state
  created and the piece left as it was. The env loading, Codex and
  named identity copies go.
- A symlinked MCP folder: the MCP suite's test now runs both modes and
  checks the link; the Codex dry-run copy goes.
- Per-identity state: the named identity suite already keeps two
  identities apart for every client, so Claude's copy goes, and one
  test now shows the flag selecting the state of every client.

Assertions name the exact diagnostic where the copies matched loosely.
Removing the state path, piece, or MCP symlink check makes the
remaining test fail.
claudiomaradonna deleted branch test/behavior-names 2026-10-07 20:53:30 +02:00
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
claudiomaradonna/azkaban!36
No description provided.