claudiomaradonna/azkaban
1
1
Fork 0
Code Issues Pull requests Activity

test: split the slowest test files #51

Merged
claudiomaradonna merged 3 commits from test-speed into master 2026-10-17 18:18:27 +02:00
Conversation 0 Commits 3 Files changed 15 +1695 -1604
motoko commented 2026-10-17 18:15:36 +02:00
Collaborator
Copy link

Why

make test took 6.2 s on a 32-CPU host. The runner runs one job per
test file, so the slowest file sets the wall time, and
test_custom_image.sh alone took 6.1 s, with test_install.sh next at
3.8 s. The suite had also never been checked against the rule of one
test per behavior at each level.

What changed

  • The custom-image tests now run as four themed files (approval,
    failures, naming, shared) and the install tests as five (cli, engine,
    builds, publication, reinstall), each family's helpers in one sourced
    *_helpers.sh. Every test body moved byte-for-byte and keeps its
    relative order. make test now finishes in about 3.2 s, bounded by
    test_runtime.sh (2.6 s alone).
  • An audit of every unit and integration test found one duplicate: the
    Codex first-run test that checked for no env file or state entry. Its
    one extra check, that the state folder holds no seeded entry, moved
    into the first-run state test, renamed to say it checks for only
    those directories. 327 tests remain, from 328.
  • docs/adding-a-client.md points at the test_install_*.sh files and
    at assert_published in install_helpers.sh for staged adapters.

The runner, harness, stubs, make targets, and TESTS/FILTER
selection are unchanged.

How to verify

  • make test passes and takes about 3.2 s; on master it takes about
    6.2 s.
  • make test TESTS=custom_image_shared and
    make test TESTS=install_cli run one part each.
  • Each moved body matches master: extract the ^test_ functions from
    git show master:tests/integration/test_custom_image.sh and from
    tests/integration/test_custom_image_*.sh and compare (same for
    install).

Already run on the tip: make lint, .agents/hooks/shellcheck.sh repo
(50 files clean), make test (327 tests, 0 failures; 3166, 3170,
3179 ms), and lint plus tests on every commit during the fold. The e2e
gate was not run; nothing it covers changed.

Considerations

The 3.2 s figure sits close to the noise of a shared host; at a load
average near 10 the same suite took 3.3 s. Further gains would come
from splitting test_runtime.sh and test_identity.sh or from making
each launcher run cheaper, both left out. A review finding that one
header line in install_helpers.sh runs to 85 characters was set aside
as low.

## Why `make test` took 6.2 s on a 32-CPU host. The runner runs one job per test file, so the slowest file sets the wall time, and `test_custom_image.sh` alone took 6.1 s, with `test_install.sh` next at 3.8 s. The suite had also never been checked against the rule of one test per behavior at each level. ## What changed - The custom-image tests now run as four themed files (approval, failures, naming, shared) and the install tests as five (cli, engine, builds, publication, reinstall), each family's helpers in one sourced `*_helpers.sh`. Every test body moved byte-for-byte and keeps its relative order. `make test` now finishes in about 3.2 s, bounded by `test_runtime.sh` (2.6 s alone). - An audit of every unit and integration test found one duplicate: the Codex first-run test that checked for no env file or state entry. Its one extra check, that the state folder holds no seeded entry, moved into the first-run state test, renamed to say it checks for only those directories. 327 tests remain, from 328. - `docs/adding-a-client.md` points at the `test_install_*.sh` files and at `assert_published` in `install_helpers.sh` for staged adapters. The runner, harness, stubs, `make` targets, and `TESTS`/`FILTER` selection are unchanged. ## How to verify - `make test` passes and takes about 3.2 s; on `master` it takes about 6.2 s. - `make test TESTS=custom_image_shared` and `make test TESTS=install_cli` run one part each. - Each moved body matches master: extract the `^test_` functions from `git show master:tests/integration/test_custom_image.sh` and from `tests/integration/test_custom_image_*.sh` and compare (same for install). Already run on the tip: `make lint`, `.agents/hooks/shellcheck.sh repo` (50 files clean), `make test` (327 tests, 0 failures; 3166, 3170, 3179 ms), and lint plus tests on every commit during the fold. The e2e gate was not run; nothing it covers changed. ## Considerations The 3.2 s figure sits close to the noise of a shared host; at a load average near 10 the same suite took 3.3 s. Further gains would come from splitting `test_runtime.sh` and `test_identity.sh` or from making each launcher run cheaper, both left out. A review finding that one header line in `install_helpers.sh` runs to 85 characters was set aside as low. 
The runner runs one job per test file, so a run lasts as long as its
slowest file, and test_custom_image.sh alone took 6.1 s of a 6.2 s
suite. Split it by theme into approval, failures, naming, and shared
parts, with the file-local helpers in custom_image_helpers.sh. Every
test body moved unchanged and keeps its relative order.
test_install.sh, at 3.8 s, was the next slowest file after the
custom-image tests. Split it by theme into cli, engine, builds,
publication, and reinstall parts, with the file-local helpers in
install_helpers.sh, and point docs/adding-a-client.md at the new
files. Every test body moved unchanged and keeps its relative order.
test_first_real_run_creates_no_env_file_or_state_entry ran the same
first Codex launch as the state-and-MCP-directories test; its env-file
checks already follow from that test's absent identities folder. Merge
its one new check, that the state folder holds no seeded entry, into
the keeper, renamed
test_first_real_run_creates_only_private_state_and_mcp_directories.
claudiomaradonna deleted branch test-speed 2026-10-17 18:18:31 +02:00
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
claudiomaradonna/azkaban!51
No description provided.