test: run the suite by level, file, or name

motoko commented

2026-10-09 01:35:12 +02:00

Collaborator

Why

Each task should be able to run just the tests it needs: one level, one
component, or the tests whose names match. The suite was a single flat
folder, the runner took no arguments, and three files each carried
their own copy of the harness.

What changed

Tests live in tests/unit/ and tests/integration/, and the
real-Podman gate moved to tests/e2e/smoke_images.sh. The runner
refuses a test_*.sh anywhere else, so a misplaced file cannot drop
out silently.
tests/harness.sh is the one fail/run_test/finish every file
shares. TEST_FILTER, an extended regular expression over test
names, runs only the matching tests.
tests/run.sh takes levels, short names and paths. Make gains
test-unit, test-integration and test-e2e, and smoke-images
stays. TESTS= works with make test, and FILTER= works with all
three test targets. A typo or a filter that matches nothing fails
rather than passing empty.
The registry unit tests and the goals tests have their own files. The
macOS-only tests now all carry macos in their names. Two duplicate
tests are gone.
A race that predates this branch is fixed. The runner read its file
list through a process substitution, and Bash 5.3's wait -n could
return for that process, so a still-running file was reported as
killed. Under load it failed 3 of 300 runs before the fix and 0 of
400 after.
docs/development.md documents the commands, and
docs/adding-a-client.md points at the new paths.

This PR does not add unit tests for the launcher, does not add a
launcher end-to-end run on real Podman, and does not rename the
common_ prefix.

Verification

make lint && .agents/hooks/shellcheck.sh repo && make test passes,
with 324 tests.
Every test that passes on master passes here under the same name,
except five renamed ones. 19 tests are new.
Not run: make test-e2e, which needs real rootless Podman. Its new
path is covered by the Makefile test and the smoke tool's unit tests.

## Why Each task should be able to run just the tests it needs: one level, one component, or the tests whose names match. The suite was a single flat folder, the runner took no arguments, and three files each carried their own copy of the harness. ## What changed - Tests live in `tests/unit/` and `tests/integration/`, and the real-Podman gate moved to `tests/e2e/smoke_images.sh`. The runner refuses a `test_*.sh` anywhere else, so a misplaced file cannot drop out silently. - `tests/harness.sh` is the one `fail`/`run_test`/`finish` every file shares. `TEST_FILTER`, an extended regular expression over test names, runs only the matching tests. - `tests/run.sh` takes levels, short names and paths. Make gains `test-unit`, `test-integration` and `test-e2e`, and `smoke-images` stays. `TESTS=` works with `make test`, and `FILTER=` works with all three test targets. A typo or a filter that matches nothing fails rather than passing empty. - The registry unit tests and the goals tests have their own files. The macOS-only tests now all carry `macos` in their names. Two duplicate tests are gone. - A race that predates this branch is fixed. The runner read its file list through a process substitution, and Bash 5.3's `wait -n` could return for that process, so a still-running file was reported as killed. Under load it failed 3 of 300 runs before the fix and 0 of 400 after. - `docs/development.md` documents the commands, and `docs/adding-a-client.md` points at the new paths. This PR does not add unit tests for the launcher, does not add a launcher end-to-end run on real Podman, and does not rename the `common_` prefix. ## Verification - `make lint && .agents/hooks/shellcheck.sh repo && make test` passes, with 324 tests. - Every test that passes on master passes here under the same name, except five renamed ones. 19 tests are new. - Not run: `make test-e2e`, which needs real rootless Podman. Its new path is covered by the Makefile test and the smoke tool's unit tests.

motoko added 10 commits

2026-10-09 01:35:16 +02:00

test: characterize the parallel test runner af57927d74

tests/run.sh had no test of its own. Record today's behavior on a
fixture tree holding a copy of the runner: every test_*.sh runs with
its whole output under its header in name order, a failing file does
not stop later files and ends the run with FAILED and exit 1, an
all-pass run exits 0 without a FAILED line, a file not named
test_*.sh never runs, and a test file receives SIGINT with its
default action.

test: group the test files by level db6fc77857

Move the test files into tests/unit and tests/integration and the
opt-in image gate into tests/e2e, so a level can be run on its own.
helpers.sh, run.sh and stub/ stay in tests/.

tests/run.sh now discovers the unit folder and then the integration
folder, names each file with its level in the header, and exits 2
naming the file, before running anything, when a test_*.sh sits
anywhere else under tests/. A misplaced file can no longer drop out
silently. tests/unit/test_runner.sh now builds level-folder fixtures
and guards the refusal.

Source lines and repo_root paths follow the move, with a ShellCheck
source directive so the helpers are still followed. SHELL_SOURCES
covers the level folders, smoke-images runs tests/e2e/smoke_images.sh,
and the Makefile structural test checks both. The development and
adding-a-client docs name the new paths.

test: share one harness with a name filter 958c722318

fail, run_test and finish lived in three copies: helpers.sh and the
two unit files that do not use the launcher fixture. They now live in
tests/harness.sh, which every file sources, and run_test calls setup
and teardown only when a file defines them.

TEST_FILTER, an extended regular expression, runs only the tests whose
function names match. The harness reads it once and unexports it, so
the launcher under test never sees it; helpers.sh sources the harness
ahead of its environment snapshot so setup cannot re-export it. An
invalid expression fails the file before any test, and finish returns
3 when no test ran so a runner can tell an empty selection apart.

tests/unit/test_harness.sh guards the filter, the optional setup and
teardown, and the child-process isolation, and takes over the setup
reset test from test_dispatch.sh. docs/development.md says how to run
one file and filter its tests.

test: select levels, files and tests to run 5355cf3188

tests/run.sh takes levels, short names and paths and runs the union,
each file once in discovery order. Every selection resolves before a
file runs: an unknown name, e2e, or a short name in both levels exits
2. With TEST_FILTER set, files in which no test matched are left out of
the output, and a run where every file was left out exits 1 with a
message.

The Makefile gains test-unit, test-integration and test-e2e, and test
takes TESTS and FILTER. docs/development.md gives one example of each.

fix(tests): finish discovery before jobs start 6acfadca6b

The runner read its file list from a process substitution. Under load
that process can exit after the drain loop's `wait -n` has started, and
Bash 5.3 returns `wait -n` for it, so the runner counted one job too
many as finished, stopped waiting while the last file still ran, and
reported that file as killed.

Read the list through a command substitution instead, which waits for
find and sort to exit before any job starts. A one-file fixture run with
ten busy loops failed 3 of 300 and 7 of 400 times before, an
instrumented copy showing `wait -n` returning for the discovery process,
and 0 of 400 after. No test triggers the race on demand, as the window
lies between the kernel closing the pipe and reporting the exit; the
runner tests keep exercising it.

test: give registry and goals tests own files 57613ce60f

The registry probes that source clients/registry.sh directly moved from
the dispatch suite to tests/unit/test_registry.sh, and the goals tests
moved from the named-identity suite to tests/integration/test_goals.sh.
Their shared fixtures (identity_root, state_root, make_identity,
run_named) moved to tests/helpers.sh instead of being copied, since both
files need them. The dispatch and named-identity headers now describe
what they still hold.

The three test_darwin_* install tests are renamed test_macos_*, the
prefix every other macOS-only test uses, so FILTER=macos finds them.
Test bodies and order are unchanged otherwise.

fix(tests): pass FILTER through make unexpanded a03e3040cd

make expanded the FILTER value inside the recipes, so a `$` followed by
another character vanished: FILTER='_fails$|_refused$' reached the
runner as `_fails_refused$` and matched nothing. The recipes now read
it with $(value FILTER), which leaves the expression as typed, and the
Makefile test passes a mid-expression anchor to guard it.

test: drop two duplicate tests and a needless disable 716dc441cb

The runner's all-pass test repeated the discovery test, which already
checks exit 0 and the exact output, so a stray FAILED line fails it.
The harness test for a file without setup and teardown repeated the
filter tests, whose fixtures define neither and whose exact output
includes stderr. tests/harness.sh reads every counter it sets, so its
file-wide SC2034 disable only hid future findings.

test: name the macOS service check test for macOS 80498ede07

test_host_engine_ready_reports_the_service_state checks only Apple
container's `system status` probe, which runs on macOS alone, so
FILTER=macos missed it. Tests that cover both hosts keep their names.

docs: clarify test selection, statuses, and suites bc86d49206

The Verification section now says that TESTS works with make test only
while FILTER also works with the level targets, and that the exit
statuses belong to the runner: make reports them in its Error line and
itself exits 2. The e2e folder is named once, where the gate is
described.

The new-client checklist names the folder of the adapter-state suite,
since a test file outside a level folder stops the run, and adds the
goals cases, which a client with an instructions file needs and the
list never mentioned.

claudiomaradonna merged commit bc86d49206 into master

2026-10-09 01:39:59 +02:00

claudiomaradonna deleted branch test/suite-levels

2026-10-09 01:40:05 +02:00

Rows
Columns

test: run the suite by level, file, or name #45

Why

What changed

Verification