Build workspace Dockerfiles below the client layer so client refreshes keep them cached #74

New issue

Closed

opened 2026-11-11 16:30:33 +01:00 by motoko · 0 comments

motoko commented

2026-11-11 16:30:33 +01:00

Collaborator

A workspace Dockerfile.azkaban (or Dockerfile.<client>) is built on
the client image, whose first FROM the launcher overrides. Every
refresh of the client image, such as a nightly one that busts the client
install layer with CACHEBUST, therefore invalidates every layer of the
workspace file, and the next launch rebuilds all of them: apt installs,
language runtimes, compiled dependencies.

Most of what a workspace file adds does not depend on the client: the
client lands in the user's ~/.local as the last step of its stage. The
client stage in containers/Dockerfile already builds FROM ${BASE_IMAGE}, so the order could be inverted:

Build the workspace file with its first FROM overridden by
localhost/azkaban-base:latest instead of the client image.
Build the client stage with --build-arg BASE_IMAGE=<that image>
and --target <client>, as _azk_update_default_image does today.

A client refresh would then rebuild only the client's own layers on top
of the cached workspace layers; the workspace layers would rebuild only
when the base image or the workspace file changes.

Points to settle:

The workspace file would run before the client user exists, so its
contract would end with USER root (or leave USER alone) instead of
USER 1000, and files it gives to the user need numeric IDs.
How often azkaban-base itself changes on the forge decides how much
this saves; if it is rebuilt nightly as well, the gain is limited to
launches between base refreshes.
Existing workspace files that end with USER 1000 or rely on the
client being present during their build would need a migration note.

Workaround in use today: declare the client as FROM <client> AS client,
do the heavy work in stages independent of it, and finish with
FROM client plus COPY --from= those stages, so a client refresh
rebuilds only the copies.

A workspace `Dockerfile.azkaban` (or `Dockerfile.<client>`) is built on the client image, whose first `FROM` the launcher overrides. Every refresh of the client image, such as a nightly one that busts the client install layer with `CACHEBUST`, therefore invalidates every layer of the workspace file, and the next launch rebuilds all of them: apt installs, language runtimes, compiled dependencies. Most of what a workspace file adds does not depend on the client: the client lands in the user's `~/.local` as the last step of its stage. The client stage in `containers/Dockerfile` already builds `FROM ${BASE_IMAGE}`, so the order could be inverted: 1. Build the workspace file with its first `FROM` overridden by `localhost/azkaban-base:latest` instead of the client image. 2. Build the client stage with `--build-arg BASE_IMAGE=<that image>` and `--target <client>`, as `_azk_update_default_image` does today. A client refresh would then rebuild only the client's own layers on top of the cached workspace layers; the workspace layers would rebuild only when the base image or the workspace file changes. Points to settle: - The workspace file would run before the client user exists, so its contract would end with `USER root` (or leave `USER` alone) instead of `USER 1000`, and files it gives to the user need numeric IDs. - How often `azkaban-base` itself changes on the forge decides how much this saves; if it is rebuilt nightly as well, the gain is limited to launches between base refreshes. - Existing workspace files that end with `USER 1000` or rely on the client being present during their build would need a migration note. Workaround in use today: declare the client as `FROM <client> AS client`, do the heavy work in stages independent of it, and finish with `FROM client` plus `COPY --from=` those stages, so a client refresh rebuilds only the copies.

motoko referenced this issue from a commit

2026-11-11 16:54:03 +01:00

docs: keep custom image layers across refreshes

motoko referenced this issue

2026-11-11 16:54:26 +01:00

docs: keep custom image layers across refreshes #75