docs: keep custom image layers across refreshes #75

Merged

claudiomaradonna merged 1 commit from docs/custom-image-layer-cache into master

2026-11-11 16:55:39 +01:00

motoko commented

2026-11-11 16:54:26 +01:00

Collaborator

Why

Azkaban lets a workspace carry its own image layer, a Dockerfile.azkaban
or Dockerfile.<client>, built on the installed client image at each
launch. On Linux amd64 and arm64 a launch rebuilds the client image
whenever the published base changes, which is every night, and every
layer of the workspace file sits on it: whoever follows the documented
examples rebuilds all their apt installs, runtimes, and compiled
dependencies on the first launch of each day, and the documentation
says only that "the affected layers" rebuild.

Issue #74 proposed building the workspace file on the base and the
client on top of it. That was set aside: the base itself changes every
night, so the workspace layers would still rebuild nightly, and the
change would break the documented USER 1000 contract. A workspace
file can already keep its heavy layers with a multi-stage shape, so the
documentation now shows it.

Refs #74

What changed

Before, a reader of Custom images saw only single-stage examples and a
cache sentence that suggested partial reuse after a client refresh. Now
the sentence says a refreshed client image rebuilds every layer built
on it, and a new section, "Keeping heavy layers across refreshes",
shows a file whose first stage names the client, whose heavy work runs
in a stage on ubuntu:26.04, and whose last stage copies the results
in and ends with USER 1000; it says which work fits, that only the
shared file's first FROM is replaced, and that podman image prune
drops the cached stage. Troubleshooting gains the symptom of a custom
image that rebuilds in full each day. The launcher is unchanged.

How to verify

Have rootless Podman (checked with 5.7.0) on Linux, and network
access to docker.io.

The problem and the solution, in a private image store that leaves
your own images alone:

w=$(mktemp -d); pm() { podman --root "$w/root" --runroot "$w/run" "$@"; }
mkdir "$w/c" "$w/ws"
printf 'FROM docker.io/library/ubuntu:26.04\nARG CACHEBUST=1\nRUN echo "$CACHEBUST" >/v\nUSER 1000\n' >"$w/c/Dockerfile"
printf 'FROM localhost/claude-code:latest AS client\nFROM docker.io/library/ubuntu:26.04 AS tools\nRUN mkdir -p /opt/tools && date +%%s%%N >/opt/tools/stamp\nFROM client\nUSER root\nCOPY --from=tools /opt/tools /opt/tools\nUSER 1000\n' >"$w/ws/Dockerfile.azkaban"
ws() { pm build -q --from localhost/c:latest -f "$w/ws/Dockerfile.azkaban" -t localhost/ws "$w/ws" >/dev/null; pm run --rm localhost/ws cat /opt/tools/stamp; }
pm build -q --build-arg CACHEBUST=1 -t localhost/c:latest "$w/c" >/dev/null; ws
pm build -q --build-arg CACHEBUST=2 -t localhost/c:latest "$w/c" >/dev/null; ws
pm image prune -f >/dev/null
pm build -q --build-arg CACHEBUST=3 -t localhost/c:latest "$w/c" >/dev/null; ws
pm unshare rm -rf "$w/root" "$w/run"; rm -rf "$w"

The first two stamps are equal: the tools stage stayed cached
across a client refresh. The third differs: the prune dropped it.
Moving the RUN into the client stage instead makes every stamp
differ, which is what the old examples give.

make lint, .agents/hooks/shellcheck.sh repo, and make test pass
on the tip. These condensed steps were not run as written, since the
authoring machine has no Podman; a longer script with the same builds,
refresh, and prune ran on Podman 5.7.0 with the results above. The Apple
container engine on macOS was not available, and the section says its
reuse is unchecked.

Considerations

"On Podman" covers releases other than the 5.7.0 that was checked; a
reviewer rated this low and it was left as is.

## Why Azkaban lets a workspace carry its own image layer, a `Dockerfile.azkaban` or `Dockerfile.<client>`, built on the installed client image at each launch. On Linux amd64 and arm64 a launch rebuilds the client image whenever the published base changes, which is every night, and every layer of the workspace file sits on it: whoever follows the documented examples rebuilds all their apt installs, runtimes, and compiled dependencies on the first launch of each day, and the documentation says only that "the affected layers" rebuild. Issue #74 proposed building the workspace file on the base and the client on top of it. That was set aside: the base itself changes every night, so the workspace layers would still rebuild nightly, and the change would break the documented `USER 1000` contract. A workspace file can already keep its heavy layers with a multi-stage shape, so the documentation now shows it. Refs #74 ## What changed Before, a reader of Custom images saw only single-stage examples and a cache sentence that suggested partial reuse after a client refresh. Now the sentence says a refreshed client image rebuilds every layer built on it, and a new section, "Keeping heavy layers across refreshes", shows a file whose first stage names the client, whose heavy work runs in a stage on `ubuntu:26.04`, and whose last stage copies the results in and ends with `USER 1000`; it says which work fits, that only the shared file's first `FROM` is replaced, and that `podman image prune` drops the cached stage. Troubleshooting gains the symptom of a custom image that rebuilds in full each day. The launcher is unchanged. ## How to verify 1. Have rootless Podman (checked with 5.7.0) on Linux, and network access to `docker.io`. 2. The problem and the solution, in a private image store that leaves your own images alone: ```bash w=$(mktemp -d); pm() { podman --root "$w/root" --runroot "$w/run" "$@"; } mkdir "$w/c" "$w/ws" printf 'FROM docker.io/library/ubuntu:26.04\nARG CACHEBUST=1\nRUN echo "$CACHEBUST" >/v\nUSER 1000\n' >"$w/c/Dockerfile" printf 'FROM localhost/claude-code:latest AS client\nFROM docker.io/library/ubuntu:26.04 AS tools\nRUN mkdir -p /opt/tools && date +%%s%%N >/opt/tools/stamp\nFROM client\nUSER root\nCOPY --from=tools /opt/tools /opt/tools\nUSER 1000\n' >"$w/ws/Dockerfile.azkaban" ws() { pm build -q --from localhost/c:latest -f "$w/ws/Dockerfile.azkaban" -t localhost/ws "$w/ws" >/dev/null; pm run --rm localhost/ws cat /opt/tools/stamp; } pm build -q --build-arg CACHEBUST=1 -t localhost/c:latest "$w/c" >/dev/null; ws pm build -q --build-arg CACHEBUST=2 -t localhost/c:latest "$w/c" >/dev/null; ws pm image prune -f >/dev/null pm build -q --build-arg CACHEBUST=3 -t localhost/c:latest "$w/c" >/dev/null; ws pm unshare rm -rf "$w/root" "$w/run"; rm -rf "$w" ``` The first two stamps are equal: the `tools` stage stayed cached across a client refresh. The third differs: the prune dropped it. Moving the `RUN` into the client stage instead makes every stamp differ, which is what the old examples give. `make lint`, `.agents/hooks/shellcheck.sh repo`, and `make test` pass on the tip. These condensed steps were not run as written, since the authoring machine has no Podman; a longer script with the same builds, refresh, and prune ran on Podman 5.7.0 with the results above. The Apple `container` engine on macOS was not available, and the section says its reuse is unchecked. ## Considerations "On Podman" covers releases other than the 5.7.0 that was checked; a reviewer rated this low and it was left as is.

motoko added 1 commit

2026-11-11 16:54:28 +01:00

docs: keep custom image layers across refreshes 8d1f495f51

On Linux a launch rebuilds the client image whenever the nightly base
changes, and every layer of a workspace Dockerfile built on it rebuilds
with it. Show the multi-stage shape that keeps the heavy work in a
stage independent of the client image, and that podman image prune
drops that stage's cache, as a real Podman 5.7.0 build showed.

Refs #74

claudiomaradonna merged commit 8d1f495f51 into master

2026-11-11 16:55:39 +01:00

claudiomaradonna deleted branch docs/custom-image-layer-cache

2026-11-11 16:55:45 +01:00

No reviewers

No labels

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

claudiomaradonna/azkaban!75

No description provided.

Rows
Columns