feat: cap log and cache disk use, tune swap and writeback #7

Merged

claudiomaradonna merged 5 commits from feat/perf-tuning into main

2026-09-25 16:59:09 +02:00

claudiomaradonna commented

2026-09-25 16:58:39 +02:00

Owner

Summary

Performance and disk-space tuning across four roles, plus a zram install fix
found while testing.

hardening-basic / journald: a changed journald drop-in now restarts
systemd-journald through a handler. Until now the size limits waited for
a reboot. Adds RuntimeMaxUse=64M (the volatile journal in /run) and
MaxRetentionSec=1month, as hardening_journald_runtime_max_use and
hardening_journald_max_retention.
updates / apt cache: os-updates runs apt autoclean as its own task
(the apt module runs only one of autoclean/autoremove per task).
unattended-upgrades sets APT::Periodic::AutocleanInterval "7"
(unattended_upgrades_autoclean_interval), checked in the apt-config dump
read-back. autoclean rather than clean: the current .deb of each package
stays.
hardening-basic / swap: a host with zram_enabled gets
vm.swappiness = 150 (hardening_sysctl_vm_swappiness_zram) and
vm.page-cluster = 0. It no longer has to be overridden in host_vars by No changes this session
hand. hardening-basic stays the only owner of sysctl, and a host_vars value
still wins.
hardening-basic / writeback: vm.dirty_background_bytes = 64M and
vm.dirty_bytes = 256M on every host, replacing the kernel's 10%/20% of
reclaimable memory, which lets gigabytes of unwritten data build up on
large-RAM hosts. null omits either line.
docker-engine: install mode merges log-driver: json-file and
log-opts (10m x 3) into /etc/docker/daemon.json. Other keys (NVIDIA
runtime, mirrors) are kept, and the merged file is checked with dockerd --validate before it is written. It is never rewritten when it already
agrees. Assert mode is untouched.
zram (fix): drops the package_facts guard, which needs python3-apt
and failed the role on a minimal host (and in molecule) before anything had
installed it. The install is idempotent on its own.

Considered and dropped: systemd-coredump Storage=none, because
kernel.core_pattern=|/bin/false already discards cores.

Blast radius

Every Linux host: the writeback limits apply on the run through the
sysctl apply-and-verify step. Journald restarts once, and log streams
survive the restart. Nothing here can lock anyone out. No changes this session
Hosts with zram_enabled and no host_vars override: swappiness goes
from 0 to 150 on the run. Before merging, check that no such host swaps to
disk with zramswap down.
Docker install-mode hosts: the engine is not restarted, because a
restart stops every container and can cost swarm quorum. The run says a
restart is pending. The new limits apply from the next engine restart, and
only to containers created after it (docker compose up -d --force-recreate, or docker service update --force <svc> in swarm).
Services with their own logging: (ai-stack) keep it.
zram hosts: zramswap is now enabled on every run, including hosts
where it had been disabled by hand.

## Summary Performance and disk-space tuning across four roles, plus a zram install fix found while testing. - **hardening-basic / journald**: a changed journald drop-in now restarts `systemd-journald` through a handler. Until now the size limits waited for a reboot. Adds `RuntimeMaxUse=64M` (the volatile journal in `/run`) and `MaxRetentionSec=1month`, as `hardening_journald_runtime_max_use` and `hardening_journald_max_retention`. - **updates / apt cache**: `os-updates` runs `apt autoclean` as its own task (the apt module runs only one of autoclean/autoremove per task). `unattended-upgrades` sets `APT::Periodic::AutocleanInterval "7"` (`unattended_upgrades_autoclean_interval`), checked in the `apt-config dump` read-back. autoclean rather than clean: the current `.deb` of each package stays. - **hardening-basic / swap**: a host with `zram_enabled` gets `vm.swappiness = 150` (`hardening_sysctl_vm_swappiness_zram`) and `vm.page-cluster = 0`. It no longer has to be overridden in host_vars by No changes this session hand. hardening-basic stays the only owner of sysctl, and a host_vars value still wins. - **hardening-basic / writeback**: `vm.dirty_background_bytes = 64M` and `vm.dirty_bytes = 256M` on every host, replacing the kernel's 10%/20% of reclaimable memory, which lets gigabytes of unwritten data build up on large-RAM hosts. `null` omits either line. - **docker-engine**: install mode merges `log-driver: json-file` and `log-opts` (10m x 3) into `/etc/docker/daemon.json`. Other keys (NVIDIA runtime, mirrors) are kept, and the merged file is checked with `dockerd --validate` before it is written. It is never rewritten when it already agrees. Assert mode is untouched. - **zram (fix)**: drops the `package_facts` guard, which needs `python3-apt` and failed the role on a minimal host (and in molecule) before anything had installed it. The install is idempotent on its own. Considered and dropped: `systemd-coredump Storage=none`, because `kernel.core_pattern=|/bin/false` already discards cores. ## Blast radius - **Every Linux host**: the writeback limits apply on the run through the sysctl apply-and-verify step. Journald restarts once, and log streams survive the restart. Nothing here can lock anyone out. No changes this session - **Hosts with `zram_enabled` and no host_vars override**: swappiness goes from 0 to 150 on the run. Before merging, check that no such host swaps to disk with zramswap down. - **Docker install-mode hosts**: the engine is **not** restarted, because a restart stops every container and can cost swarm quorum. The run says a restart is pending. The new limits apply from the next engine restart, and only to containers created after it (`docker compose up -d --force-recreate`, or `docker service update --force <svc>` in swarm). Services with their own `logging:` (ai-stack) keep it. - **zram hosts**: `zramswap` is now enabled on every run, including hosts where it had been disabled by hand.

claudiomaradonna added 5 commits

2026-09-25 16:58:41 +02:00

feat(hardening-basic): apply journald limits and cap runtime use and age 54c990dcd4

feat(updates): autoclean superseded packages from the apt cache f576c36f3a

feat(hardening-basic): raise swappiness under zram and cap dirty page cache 7aaae88252

feat(docker-engine): cap container logs through daemon.json in install mode 7cb3319323

fix(zram): install zram-tools without a package_facts guard 88ab65b5de