claudiomaradonna/castrum
1
0
Fork 0
Code Pull requests Activity

chore(hardening): now sysctl is verified and overriding in the right way /etc/sysctl.conf and other config files #3

Merged
claudiomaradonna merged 5 commits from verify-what-the-roles-claim into main 2026-08-29 20:27:51 +02:00
Conversation 0 Commits 5 Files changed 16 +954 -18
claudiomaradonna commented 2026-08-29 20:27:35 +02:00
Owner
Copy link
No description provided. 
Three roles have now been found reporting success for work nobody had
verified. `unattended-upgrades` was fixed in 26cd47f. These are the other two,
and they are the same defect wearing different clothes: something is written or
started, the task returns green, and no one asks the machine whether it took.

hardening-basic wrote /etc/sysctl.d/99-hardening.conf and stopped there, which
hid two failures at once. Nothing applied the file, so every value waited for a
reboot that may never come. And /etc/sysctl.conf is read LAST by both
systemd-sysctl and `sysctl --system`, after every sysctl.d drop-in, so a value
sitting in it outranks this role permanently no matter what number prefixes the
file. capacious_extremis was measured running with vm.swappiness at 100 in the
drop-in, 0 in /etc/sysctl.conf, a 1.8G zram device, and a kernel instructed
never to touch it. Both numbers were correct in their own file and the host had
the worst of the two.

The role now runs `sysctl --system` and reads every key of its own drop-in back
out of the kernel. `--system` and not `-p <file>` on purpose: it reproduces
boot order, so the drop-in loses the same arguments it would lose on a reboot
rather than winning one it cannot keep. A key something else owns fails the
play by name. A key this kernel has no knob for is reported and does not fail,
because the net.ipv6.* lines legitimately disappear once the module blacklist
takes effect.

An empty read-back would have made that comparison vacuously true - a verifier
agreeing with itself because it measured nothing, which is the failure being
fixed rather than an acceptable outcome. So the read-back tolerates a non-zero
rc and a separate assertion fails when no values came back at all, quoting rc
and stderr.

fail2ban-basic started the service and believed systemd. `state: started`
returns when the job is accepted, and for a Type=simple unit that is before the
daemon has read a line of configuration. Three hosts in this fleet ran for
months with this role green and fail2ban exiting 255 on every start, because an
epoch-pinned vendor package had left a libsystemd older than the binary wanted.
It now asks `fail2ban-client ping`, which crosses fail2ban's own socket and is
answered only by a live server, with retries so a slow start is not read as a
dead one.

hardening_sysctl_apply gates the sysctl half and defaults to true. Both
molecule scenarios set it false: a container's sysctls belong to its host, so
applying the drop-in there changes the wrong machine and reading it back
measures the wrong machine.

Two consequences to expect on the next fleet run. Values now land during the
run instead of at the next reboot, so hardening_sysctl_disable_ipv6 in
particular arrives while the play is still connected. And the first run after
this is where the estate discovers which hosts have a conflicting
/etc/sysctl.conf, by name, as a failure.

Gates run in the jail: yamllint, ansible-lint, --syntax-check of all four
playbooks, check_shared_group_vars.py, check_tag_selection.sh,
check_peer_derivation.sh - all pass, no new lint findings. The comparison
filters were proven separately against a known-good and a known-bad read-back,
including that an absent key is not mistaken for an overridden one. molecule
was not run: this sandbox has no podman.

Claude-Session: https://claude.ai/code/session_01W5FkkabPcwgqKTQv4LD7DP
`net.ipv4.conf.all.mc_forwarding` is not a switch. It is a read-only flag the
kernel sets to report whether a multicast routing daemon has registered, and
writing it returns EPERM even as root on bare metal. The line has been in this
template from the start and has never once taken effect.

Nothing caught it because the read-back agrees: the key reports 0, the template
asks for 0, so the value comparison passes. Only the apply's stderr showed it,
which is why the assertion quotes stderr rather than discarding it - the first
real run on daily_prophet (arm64, Debian) surfaced it as

    sysctl: setting key "net.ipv4.conf.all.mc_forwarding": Operation not permitted

A comment replaces the line rather than a silent deletion, so the next person
to read the CIS lists this template was built from does not add it back.

Gates: yamllint and ansible-lint pass. molecule not run - no podman here.

Claude-Session: https://claude.ai/code/session_01W5FkkabPcwgqKTQv4LD7DP
fs.protected_fifos was the first thing the new read-back found that was not a
local misconfiguration. castrum's template asked for 2; Debian's
/usr/lib/sysctl.d/99-protect-links.conf says 1; Debian has won on every host in
the fleet for as long as the drop-in has existed. The other three fs.protected_*
keys castrum sets are byte-identical to that file, so the duplication had been
invisible - castrum was being credited for the vendor's work on three keys and
silently losing the fourth.

Sorting cannot fix it, which is worth writing down because it is the obvious
move and it is wrong. procps `sysctl --system` walks whole directories, reading
all of /usr/lib/sysctl.d AFTER all of /etc/sysctl.d, so a vendor drop-in beats
an /etc one whatever either file is called. systemd-sysctl instead sorts every
file by basename across directories, letting /etc mask /usr/lib only for an
identical name. Renaming 99-hardening.conf to sort later would therefore have
won at boot and lost under `sysctl --system` - a key whose value depended on
whether the host had rebooted since the last run, which is worse than losing
consistently.

Masking is the one technique both appliers agree on: each drops a
lower-priority file once one of that basename has been loaded. 50-coredump.conf
has quietly relied on this against systemd's file of the same name from the
start. So castrum now ships its own 99-protect-links.conf.

All four keys are restated there and removed from 99-hardening.conf. A partial
copy would suppress the vendor file and drop the three castrum does not change
to their kernel defaults - a weakening dressed as hardening - and a molecule
assertion now guards exactly that, along with the keys being gone from the file
that can no longer win them. The cost is real and documented: if Debian adds a
fifth key to its copy, this file silently suppresses it, because the read-back
only checks keys castrum declares.

hardening_sysctl_fs_protected_fifos exists because 2 is not free. It extends
the O_CREAT restriction from world-writable sticky directories to
group-writable ones, and upstream systemd chose 1 deliberately, since a daemon
creating a FIFO in a group-writable IPC directory is a real pattern that 2
breaks. The variable is the way back to 1 for a host that hits it, without
giving up the file that makes either value stick.

The read-back now spans hardening_sysctl_managed_files rather than one path,
because moving keys into a second file would otherwise have moved them out of
verification - the same defect this work exists to remove, reintroduced by the
fix for it. The list is explicit rather than a glob: /etc/sysctl.d holds other
people's files, and asserting castrum owns a key it never wrote is a different
and wronger claim.

Also corrected: the comment claiming `--system` reproduces boot order. It
approximates it. The two appliers order /usr/lib differently, they agree on
every key this template sets on daily_prophet, and what the block asserts is
the live state after the apply rather than a predicted boot state.

Gates: yamllint, ansible-lint, --syntax-check of all four playbooks,
check_shared_group_vars.py, check_tag_selection.sh, check_peer_derivation.sh -
all pass. The multi-file parsing was proven separately: three files merged,
comments dropped, a value containing | and / preserved, an overridden key
caught and an absent key kept distinct from it. molecule not run - no podman
here, so the two new scenario assertions are unexecuted.

Claude-Session: https://claude.ai/code/session_01W5FkkabPcwgqKTQv4LD7DP
The sysctl apply-and-verify block excluded itself from check mode outright, so
the only part of the baseline that fails a play on what it measured was
invisible to the pre-flight AGENTS.md requires before every live run. That
exclusion was not gratuitous: `command` skips under --check, so the read-back
returned nothing and the "refuse to report success without having measured
anything" assertion fired on every check run. Skipping the whole block made
that go away and took the useful half with it.

The two halves separate cleanly. `sysctl --system` writes, so it stays skipped
under --check. `sysctl <key>` with no `=` reads, so it now carries
check_mode: false and always runs. What comes back is the kernel before this
run's apply, compared against the drop-ins currently on disk - which is what a
preview is, and the report says so rather than leaving the reader to take it
for a prediction.

Host drift now reports under --check and still fails a real run. The line is
between "this host has drifted", which is what the real run exists to fix, and
"this told you nothing", which is a broken measurement: an empty read-back
still fails in both modes. A pre-flight that goes red on every unconverged
host is a pre-flight nobody can use, and AGENTS.md mandates that pre-flight.

Check mode reached two states a real run cannot, both fixed here. On a host
this role has never converged the managed files do not exist yet, and the
slurp died before anything was reported; a missing file is now reported under
--check and asserted only in a real run, where the templates have just written
every file the role owns and a name no template produces is a claim to verify
something nothing wrote. And with nothing declared, the read-back ran
`sysctl --ignore` with an empty key list - a usage error on a real host - then
blamed sysctl for returning no values, sending the reader to the wrong place
entirely.

The block moved to tasks/sysctl.yml. It is the only part of this role that
reads the machine back rather than writing to it, and a separate file is what
lets a fixture play include it alone. main.yml keeps the same task name and
the same tags, and selection was checked rather than assumed: --tags hardening
and --tags hardening-basic both reach the included tasks, --tags firewall does
not.

scripts/check_sysctl_check_mode.sh is new and joins `make check`. No molecule
scenario reaches this block - a container shares its host's kernel, which is
why both scenarios set hardening_sysctl_apply: false - and a stub sysctl(8) is
in any case the only way to make the kernel disagree with the drop-in on
demand, which is the case worth testing. Six assertions: a check run reports
the overridden key, does not fail on it, and never calls --system; an empty
drop-in says so instead of blaming sysctl; a missing drop-in previews under
--check and fails a real run.

Also corrected: the role README still said the apply "reproduces boot order".
8d23232 corrected that claim in the task comment and missed the README. It
approximates it.

Gates: every `make check` target passes - yamllint, ansible-lint,
--syntax-check of all four playbooks, check_shared_group_vars.py,
check_tag_selection.sh, check_peer_derivation.sh, and the new
check_sysctl_check_mode.sh. That tag-selection gate matters here in
particular, since this commit turns a tagged block into a dynamic include.
Not run and not claimed: molecule and check_tag_propagation.sh, which need
podman. There is no `make` in this sandbox, so the targets were invoked
directly, with ANSIBLE_HOME=/workspace/.ansible - the collections are
vendored in the checkout rather than in ~/.ansible, and without it the two
ansible-playbook gates cannot resolve community.general.

Claude-Session: https://claude.ai/code/session_018eAXRtkgXPhQvhuwyrHKGB
The read-back has been naming keys /etc/sysctl.conf owns since it was added,
and leaving it there. Detecting a conflict is not resolving one: every run
against a conflicted host failed identically until somebody logged in, and the
estate had no way to converge, only a recurring alarm.

hardening_sysctl_edit_sysctl_conf comments out the lines in that file whose
keys this role's own drop-ins declare, so the drop-ins win. Nothing else in
the file is touched and no other file is edited at all. /etc/sysctl.conf is
the legacy catch-all with no ownership convention; a numbered drop-in in
/etc/sysctl.d belongs to whoever installed it, and castrum editing that would
be a far larger claim than this one. An override from anywhere else still
fails the play untouched.

Masking, which beat the vendor file in 8d23232, cannot help here. Both
appliers read /etc/sysctl.conf last on purpose, after every sysctl.d drop-in,
for backwards compatibility - no name and no directory outranks it, so
commenting the line is the only technique left. The original text stays on
disk under a `# castrum:` marker, which is both the record of what was there
and the reason nothing takes a backup: a commented line no longer matches, so
a second run changes nothing and stacks no second marker.

The edit is re-applied and re-measured rather than assumed, which is the point
of this branch and also what catches the case worth knowing about.
net.ipv4.ip_forward and net.ipv4.conf.all.forwarding are one kernel knob under
two names. castrum declares the second; Debian's stock /etc/sysctl.conf ships
the first as a commented example and a gateway has usually uncommented it. The
override is detected, no line spells the key castrum knows, nothing is edited,
and the run says exactly that instead of reporting a fix it did not make.

Off by default, and per host. The reclaimed value goes live during the run
that reclaims it: harmless for vm.swappiness, loss of routed traffic where
forwarding is what the host is for, loss of the address the host is reached on
where it is disable_ipv6. So the failing run now names the flag - the estate
is told the remedy at the moment it meets the conflict - and the flag is
turned on once someone has looked, not by shipping a new default.

hardening_sysctl_conf_path exists because that file is the entire blast radius
of the flag, and deserves naming in one place rather than being buried in a
task. Every other sysctl path this role touches is already a variable; this
was the exception.

Six more assertions in scripts/check_sysctl_check_mode.sh, twelve now. The
flag off must name the flag. The flag on must comment the line, clear the
override and leave a neighbouring key this role does not declare byte for
byte intact. A second run must change nothing. An override spelled under the
alias must fail, say why, and leave the file alone. And --check --diff must
print the edit line by line without writing it, since with the flag off that
preview is the only one an operator gets.

The stub sysctl(8) grew to match: it now recomputes the kernel from the
drop-ins and then the conf file, last assignment winning, and propagates
ip_forward to conf.all.forwarding. It is a small model of procps rather than a
mock of it, because a test that cannot reproduce the precedence cannot test
the code that exists to cope with it.

Gates: every `make check` target passes - yamllint, ansible-lint,
--syntax-check of all four playbooks, check_shared_group_vars.py,
check_tag_selection.sh, check_peer_derivation.sh and
check_sysctl_check_mode.sh. Not run and not claimed: molecule and
check_tag_propagation.sh, which need podman. No `make` in this sandbox, so the
targets were invoked directly, with ANSIBLE_HOME=/workspace/.ansible for the
vendored collections.

The two safety-critical assertions were mutation-tested rather than trusted
for passing first time: widening the regexp to match any assignment is caught
by the untouched-neighbour check, and letting the edit run under --check is
caught by the preview check. Nothing has run against a real host - the reclaim
has never edited a real /etc/sysctl.conf.

Claude-Session: https://claude.ai/code/session_018eAXRtkgXPhQvhuwyrHKGB
claudiomaradonna deleted branch verify-what-the-roles-claim 2026-08-29 20:27:55 +02:00
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
claudiomaradonna/castrum!3
No description provided.