castrum

Hardened baseline for mixed client fleets: one tested role library, per-tenant inventories, applied in rolling batches.

A Roman castrum was built to the same plan wherever the legion stopped — same layout, same gates, same defences, on any ground. This repository is that plan for a fleet of machines: the same hardening, firewall and service configuration applied identically across every site it manages, varied only where the ground demands it.

Each site's inventory lives outside this repository. What is here is what a group means, the roles that implement it, and the tests that prove they do.

Scope today: GNU/Linux, Debian family. Windows support is intended and not yet present — see the OS dispatch pattern in roles/*/tasks/ for the extension point.

Playbooks

• handbook.yml: is the main playbook, used for most of the situations. The playbook respond to vars configured to hosts. You can, for example, create multiple inventories, each with different host_vars and call this playbook to setup just only triggered roles. Each role has correspondent tags.
• os-update.yml: this playbook is useful when you need to update multiple hosts from another host (like a "controller")
• setup-node.yml: this playbook helps you to setup a target to allow it to play ansible playbooks
• ssh-copy-id.yml: this helps you to send your ssh pubkey to multiple targets for the given user

Docker

The examples for Dockerfile.example and docker-compose.yml.example are useful if you need a basic container to test your playbook with different systems (like Centos/ Ubuntu). Copy them and modify as needed :)

Tags and modes

Tags choose which slice of work runs. Every role carries its own name and the concern it belongs to:

Concern	Roles
hardening	hardening-basic, fail2ban-basic, auditd, snort-community
firewall	iptables-basic, -persistent, -samba, -webserver, -kdeconnect, -localai, -swarm, -gluster, -warden
services	samba, zram, wakeonlan, service-role-loader, docker-engine, ai-stack, traefik-routes, warden
sysadmin	sysadmin-tools
updates	os-updates, unattended-upgrades
bootstrap	ansible-node

Only the first four concerns are reachable from handbook.yml. The other two belong to different playbooks: updates selects the roles os-update.yml runs and bootstrap the role setup-node.yml runs, so --tags updates or --tags bootstrap against handbook.yml selects no work at all.

ansible-playbook handbook.yml -i inventory/<tenant>/ --tags firewall
ansible-playbook handbook.yml -i inventory/<tenant>/ --tags zram

ansible-playbook os-update.yml -i inventory/<tenant>/ --tags updates
ansible-playbook setup-node.yml -i inventory/<tenant>/ --tags bootstrap

Two modes are requested with variables rather than tags, so they compose with selection. Each is implemented by some roles and ignored by the rest; no role implements both:

Mode	Implemented by	What it does there
update_only	auditd, fail2ban-basic, zram	skips the role's install-and-configure block, so the role does nothing on that run
uninstall	iptables-basic	opens the INPUT policy and flushes the filter table before the baseline policy is reapplied

# keep the three roles that implement it out of this run
ansible-playbook handbook.yml -i inventory/<tenant>/ -e update_only=true

# flush the firewall on these machines before the baseline is reapplied
ansible-playbook handbook.yml -i inventory/<tenant>/ --tags iptables-basic -e uninstall=true

Every other role ignores both flags. samba implements neither, so --tags samba -e uninstall=true removes nothing.

Both flags are run-scoped: pass them with -e, for the one run that wants them. They are ordinary variables, so setting either in a group_vars or host_vars file applies it to every run of every playbook against those hosts.

Changed: these were previously --tags global.update_only and --tags global.uninstall. Those spellings no longer work — they now select no tasks, so a run using them does nothing. Update any runbook or scheduled job that still uses them.

Tags decide what runs now. What a machine is — a fileserver, bare metal — is decided by its groups, described under Inventory and groups.

Development

make setup   # virtualenv, pinned tools, collections, pre-push hook
make check   # yamllint + ansible-lint + syntax + group-var, tag-selection, peer-derivation, sysctl, docker-engine-assert and persist-filter checks (what the hook runs)
make test    # molecule scenarios, needs rootless podman
make verify  # check + test + the tagged dynamic-run check, needs rootless podman

There is no CI runner: make setup installs a pre-push hook that runs make check before every push. Bypass it deliberately with SKIP_CHECKS=1 git push.

The toolchain lives in .venv/. A virtualenv hardcodes absolute paths, so it does not survive the checkout being renamed or moved: rebuild it with make clean setup. When a container or a second machine shares this checkout through a bind mount and runs a different Python, give it a venv of its own — make VENV=.venv-jail setup, then make VENV=.venv-jail check — because one .venv/bin can hold only one interpreter's shebangs, and whichever side ran make setup last silently breaks the other. Every gate and the pre-push hook follow VENV, and .gitignore covers .venv*/.

Inventory and groups

Each directory under inventory/ is one tenant and is the isolation boundary: a run can only reach the hosts in the inventory it was given. Ansible parses every file in the directory and merges them, so a tenant splits by whatever makes sense — a cluster, a site — and groups of the same name join up across files:

inventory/
  home/
    personal.yml
    swarm.yml
  hive/
    production.yml
    office-monteprandone.yml

Pass the directory, never its parent: -i inventory/ would merge every tenant into a single run, which is the one thing the boundary exists to prevent. Keep only inventory files in there — Ansible tries to parse everything it finds, so a stray README earns a warning on every run.

These directories are untracked, along with host_vars/, because they hold host addresses and credentials.

What group membership means is tracked, in group_vars/:

A group name states a goal, and its kind decides the grammar: what a host serves is a plural noun, what a host is is an adjective, and how it is managed is a past participle. A new group follows whichever it belongs to.

Group	Kind	Turns on
fileservers	serves	Samba
webservers	serves	the web server firewall ports
ids_sensors	serves	the Snort sensor
physical	is	Wake-on-LAN
containerized	is	IPv4 forwarding, which Docker and rootful podman need
workstations	is	the workstation hardening profile instead of the server one
autopatched	managed	unattended-upgrades
clustered	is	the Docker Swarm firewall ports, opened per peer
gluster_peers	serves	the network-filesystem modules the blacklist would otherwise remove; also the allow-list iptables-gluster opens to, so it holds clients as well as servers
brick_servers	serves	the GlusterFS ports, on the hosts that actually serve bricks rather than only mount them
audited	managed	the audit daemon
ai_mainframes	serves	the AI stack (LocalAI, LocalAGI, LocalRecall) and its edge-only firewall
edge_routers	serves	the Traefik file-provider routes; membership is also the source list an AI mainframe admits

Changed: ai-stack now publishes LocalRecall on ai_stack_bind_address:8082 (an empty ai_stack_localrecall_port opts out); setting ai_stack_network_subnet recreates ai-net and restarts the stack once. The new warden and iptables-warden roles are per-host opt-in (host_vars, not group_vars) and deploy the escalation runner reviewed in the warden repository.

Changed: ai-stack now persists LocalAI's /backends, /data, and /configuration trees plus a dedicated DS4 disk KV-cache directory. On an existing host, stop LocalAI and copy container-only state before the first live convergence; copying mutable /data while it runs is not a consistent cutover. The Compose volume changes recreate localai, and a plain container restart does not apply them. See roles/ai-stack/README.md for the migration and DS4 model-configuration commands.

managed is the group every playbook targets when no target is given, so it is what an untagged, untargeted run reaches. A host outside it is in the inventory but is not configured by castrum.

A host joins as many groups as apply. Group variables do not merge, so each variable is owned by exactly one group file; make check fails if two files define the same one. Settings that differ for a single host go on that host, which outranks any group.

A catalogue role runs when its flag is true, and group membership is what sets that flag — the clustered and gluster_peers rows above are two examples. active_services still decides anything outside the catalogue, and is ignored for anything inside it.

Changed: hardening_sysctl_disable_ipv6 now defaults to true, for every host in every tenant. It covers lo, so ::1 disappears: a service bound to IPv6 loopback will not start, and a host reached over an IPv6 address becomes unreachable. See roles/hardening-basic/README.md for the reasoning.

Corrected: that sysctl does not do what the paragraph above implies on a host whose links are managed by systemd-networkd or NetworkManager. Either one sets net.ipv6.conf.<iface>.disable_ipv6=0 on the links it configures, after /etc/sysctl.d has been applied, so all, default and lo read 1 while the interface carrying traffic keeps IPv6. That was measured, not inferred. The firewall consequence — iptables -P INPUT DROP beside ip6tables -P INPUT ACCEPT, with both saved to /etc/iptables/ — is now closed by iptables_basic_ipv6_enabled, which defaults to true and writes the same baseline into the v6 table. Every host running iptables-basic gets an IPv6 DROP policy on its next run; SSH is admitted there too, so a host reached over IPv6 keeps that access.

Changed: hardening-basic now applies the sysctl drop-in it writes and then reads the keys back out of the kernel, under hardening_sysctl_apply, which defaults to true. Before this the role templated /etc/sysctl.d/99-hardening.conf and stopped, which hid two ways of being wrong while reporting success: the values waited for a reboot, and /etc/sysctl.conf outranked the drop-in permanently, because both systemd-sysctl and sysctl --system read that file last, after every sysctl.d drop-in, whatever number prefixes them. capacious_extremis ran with vm.swappiness at 100 in the drop-in, 0 in /etc/sysctl.conf, a 1.8G zram device, and a kernel instructed never to use it.

Two consequences worth knowing before the next run. Values now land during the run rather than at the next reboot, so anything that was quietly waiting — hardening_sysctl_disable_ipv6 above, most of all — arrives while the play is connected. And a key some other file owns now fails the play by name instead of passing silently, so the first run after this change is where the estate finds out which hosts have a conflicting /etc/sysctl.conf.

Changed: the four fs.protected_* keys moved out of /etc/sysctl.d/99-hardening.conf into /etc/sysctl.d/99-protect-links.conf, which deliberately carries the same basename as Debian's file of that name and suppresses it. fs.protected_fifos was the discovery: castrum asked for 2, Debian's file says 1, and Debian won on every host in the fleet. Renaming castrum's drop-in to sort later would not have fixed it — procps reads all of /usr/lib/sysctl.d after all of /etc/sysctl.d, so a vendor file outranks an /etc one whatever either is called, while systemd sorts by basename across both. Masking is the only technique the two appliers agree on, and 50-coredump.conf already depended on it. Found by the read-back above, on its first real run.

Changed: hardening_sysctl_edit_sysctl_conf, off by default, resolves the /etc/sysctl.conf conflict the read-back has been reporting since it was added. Both appliers read that file last, after every sysctl.d drop-in and by design, so no name and no directory outranks it and the masking that beat the vendor file cannot help — commenting the conflicting line out is the only way a drop-in wins. Only lines whose keys castrum's own drop-ins declare are touched, only in that file, and the original text stays on disk commented under a # castrum: marker. The edit is then re-applied and re-measured rather than assumed, which is what catches an override spelled under the other name of the same knob: net.ipv4.ip_forward is net.ipv4.conf.all.forwarding, castrum declares the second, and a gateway's /etc/sysctl.conf sets the first. There the run says no line spells the key instead of reporting a fix it did not make.

Off by default, per host, on purpose. The reclaimed value goes live during the run that reclaims it — harmless for vm.swappiness, loss of routed traffic for a host that forwards. The failing run now names the flag, so the estate is told the remedy at the moment it hits the conflict, and --check --diff prints the exact line-level edit without writing it.

Changed: the sysctl read-back now works under --check. It used to exclude itself from check mode outright, so the one part of the baseline that fails a run on what it measured was invisible to the pre-flight AGENTS.md requires before every live run. It now reports there instead of enforcing: the apply stays skipped, the read-back still runs because reading a key changes nothing, and findings print with a sysctl check mode: prefix. Host drift reports; only a check that could not be made at all still fails, which is the line between "this host has drifted" and "this told you nothing". A host the role has never converged previews cleanly now rather than dying on a missing drop-in.

That block moved to roles/hardening-basic/tasks/sysctl.yml so a fixture play can include it alone, and scripts/check_sysctl_check_mode.sh — new in make check — drives it against a stub sysctl(8). No molecule scenario covered it, because a container shares its host's kernel and both scenarios switch it off; a stub is also the only way to make the kernel disagree with the drop-in on demand, which is the case worth testing.

Changed: fail2ban-basic asks fail2ban-client ping after starting the service. state: started returns when systemd accepts the job, which for a Type=simple unit is before the daemon has read anything, and three hosts in this fleet ran for months with the role green and fail2ban exiting 255 on every start. The daemon answering on its own socket is the only evidence that it is running.

Changed: hardening-basic refuses to switch IPv4 forwarding off on a host that shows evidence of routing container traffic — a runtime socket, or a NAT-shaped bridge among the interface facts — before any file is written, and under --check too, which makes the pre-flight AGENTS.md requires the place it surfaces. The outage this ends was measured on this fleet: a docker host converged outside the containerized group got net.ipv4.conf.all.forwarding = 0, DNAT kept landing, the kernel declined to forward, and every published port on the host timed out for two days while iptables -L looked completely healthy — recreating the containers rewrites the firewall rules and heals nothing, because the drop-in re-imposes the value at every boot. The refusal names the committed remedy, containerized membership, alongside the host_vars override and hardening_sysctl_forwarding_guard: false for a host where forwarding must go off despite the evidence. The sysctl gate drives it through the same fixture play as the read-back.

Changed: iptables-persistent strips what dockerd owns before the live ruleset becomes /etc/iptables/rules.v4. A raw iptables-save froze the DOCKER-* chains — per-container ACCEPTs, DNAT entries, the FORWARD jumps, all keyed to container IPs that change on recreate — into the file a boot replays, where a restore replayed them stale. The strip pattern is deliberately case-sensitive and anchored on the chain names, so castrum's own swarm rules, whose comments merely say "Docker Swarm peer", come through, and deliberately no wider: docker's POSTROUTING MASQUERADE lines name no DOCKER chain and stay, because a duplicate is harmless and a stale one matches nothing. The save also writes through a rename now — the bare redirect used to truncate the file before iptables-save had printed a byte, so a save dying half-way left a boot restoring a partial ruleset, policies included. scripts/check_persist_filter.sh, new in make check as persist, drives the shipped save tasks against stub save binaries. One caveat stands: a netfilter-persistent reload on a live docker host still wipes dockerd's chains whatever the file holds. The clean file makes boot the safe path; it does not make reload one.

Working with coding agents

Repository instructions for coding agents live in AGENTS.md, with CLAUDE.md importing them for Claude Code. They are adopted from the agent-foundry template and cover workflow, context continuity, project memory, commands, and the Ansible conventions of this repository.

Project memory follows the localrecall-automatic profile: agents read LocalRecall through search and list_files, and only the dedicated memory-curator subagent (.claude/agents/memory-curator.md, .codex/agents/memory-curator.toml) writes, into project_castrum_memory — with project_castrum_knowledge for reference material. Names use underscores only, because LocalRecall's index does not distinguish a hyphenated twin. The shared contract is docs/LOCALRECALL.md, vendored from agent-foundry (MIT, LICENSES/agent-foundry.MIT.txt), and scripts/verify-agent-config.sh checks the declaration statically.