Homelab

Reaching an application means passing three checks, in order, each enforced by a different piece of software with no shared state between them. Failing any one drops the connection before the next gate is even reached.

Gate 1 is WireGuard itself. A device only gets to place a packet on the 10.0.0.0/24 overlay if its public key is already listed as a peer on the relay. There is no username or password here, no banner to greet an unrecognized peer with: an unauthorized handshake just gets silence.

Gate 2 is mutual TLS at Nginx. Every client presents a certificate, and Nginx verifies it against a private, offline-signed Root CA, checking a certificate revocation list on every connection rather than trusting the chain alone. A stolen WireGuard key is useless here without the matching client certificate, and a lost or compromised device is cut off by revoking its certificate, regenerating the list, and reloading Nginx, which caches the list in memory. No other device is rekeyed.

This gate is enforced globally, at the listener level, for every hostname behind Nginx, core service or app tier alike. Nginx picks a virtual server twice for one request, once by TLS SNI and once by the Host header, and those two choices are allowed to disagree. If even one hostname's TLS configuration skipped client verification, that mismatch would be a way in: handshake against the unprotected host, then send a Host header naming a protected one. Because no hostname is exempt, there is no weak virtual server to pivot through, which closes that entire bypass class by construction instead of by a rule that has to be remembered for every new service.

Gate 3 is Authelia, sitting behind an auth_request subrequest and enforcing two-factor login under a default-deny policy: nothing is reachable unless a rule explicitly allows it. Vaultwarden ships its own SSO and is deliberately allowed to skip this gate and rely on its own login instead, a stated exception, not a gap that got missed.

Once all three gates pass, the path forward splits. Services still running in rootless Docker are reached by plain container name on a private, per-app Docker network. Services that have migrated to the k3s app tier are reached through a separate seam interface, its own small gate of a different kind, covered in section 4f.

AdGuard Home runs bound to the WireGuard interface address specifically, not to 0.0.0.0, so it is not merely firewalled off from the outside world: there is no socket listening anywhere else for an outside query to land on. A wildcard rewrite resolves every service subdomain to the host internally, and no public A or CNAME record for any subdomain exists anywhere in public DNS. There is nothing for an enumeration tool to enumerate, because the names were never published in the first place. It also means DNS inherits the first gate for free: no authorized peer, no resolution, and no way to discover that a service exists at all.

Clients get this resolver pushed automatically: the DNS line lives in the [Interface] block of their own WireGuard config, with no fallback resolver alongside it. Off-tunnel, those names simply do not resolve, which is safer than quietly falling back to a public server and leaking query patterns.

Where AdGuard has to go outside, it goes encrypted. Public names resolve over DNS-over-HTTPS to Quad9's unfiltered endpoint, so my own ISP sees one encrypted stream to one resolver rather than a list of everything the lab looked up. Unfiltered is the right variant because filtering is AdGuard's job, and doing it there is the reason it is AdGuard rather than any plain resolver: one blocklist applies to every device on the tunnel at once, including the phones that have no good way to do it for themselves. The fallback list is empty, so the fail-closed posture holds at the resolver as well as at the client. If the upstream cannot be reached, resolution stops rather than quietly finding another way out. The one thing that does leak is the bootstrap, since the resolver's own hostname has to be resolved in the clear before the encrypted channel exists. That tells an observer which provider I chose and nothing about what I ask it.

The same posture continues inside the cluster for the app tier. Pods talk only to CoreDNS, never directly to the host, and CoreDNS is handed AdGuard as its own upstream through a kubelet resolv-conf override rather than the cluster's usual path. That one narrow plumbing route means nothing has to open a hole in any app's NetworkPolicy just to let it reach a DNS port on the host, and there is no fallback public resolver defined here either. If AdGuard goes down, pod DNS goes down with it, on purpose: the same fail-closed choice made for clients.

The first is Let's Encrypt, reached over DNS-01. There is no inbound HTTP path to validate against, so acme.sh proves control of the zone by writing a TXT record instead. It runs as a persistent container with no Docker socket, and that omission is deliberate: acme.sh is the one container with a job that requires it to face the internet, which makes it the most plausible thing on the host to get popped, and handing a popped container the socket hands it the rest of the rootless daemon. Its Cloudflare API token is scoped to Zone:DNS:Edit on a single zone, never account-level, so a leaked token can rewrite one _acme-challenge record, not billing, not other zones, not any other record in that zone. Renewal has one trap. acme.sh cannot reach into Nginx's container to tell it to reload, so a renewed certificate lands on disk and stops there. What closes the loop is a systemd user path unit on the host, watching the certificate directory and firing a reload when it changes. Skip that unit and nothing looks wrong: renewal succeeds, files on disk are current, and Nginx keeps serving the old certificate from memory anyway, invisible until it expires and every client fails at once.

The second is a private, offline Root CA, used for per-device client certificates. Its key is AES-256 encrypted and unlocked only by a passphrase I hold. Certificates are issued per device and exported as .p12 bundles so they install the same way on a phone, a laptop, or a browser. Revocation runs through a real OpenSSL CRL, reloaded into Nginx, rather than by deleting the peer and hoping: losing a device adds one serial to a list Nginx already checks, and no other device re-enrolls.

The third is SeamCA, and it exists only because the second one cannot do this job. A certificate signed by the Root CA is, by construction, a valid client identity for the access gate, so using it to authenticate the internal hop between Nginx and the cluster would mean a working user credential sitting in Nginx's config and inside the Traefik pod. A transport credential must never double as a spendable user credential. SeamCA signs exactly two certificates, one for each side of that link, and stops there. There is no CRL because there is no population to revoke from: with two long-lived certificates on one host, the answer to "this seam is suspect" is reissue and redeploy.

Docker came first because I wanted to learn Docker. Podman, the obvious alternative, never got a real hearing: the choice was settled by what I wanted to learn, not by what the two engines do differently.

What I did care about was not running a rooted container engine, and that is what put me on rootless Docker for everything, not Docker as such. The unprivileged daemon means a container escape does not hand the attacker host root by default, which is the property the rest of the design leans on.

Kubernetes arrived the same way Docker did, to learn it, and ran into the same objection. A default k3s node runs as root, so rootless k3s was the first thing I looked at. Its own documentation labels it experimental, and that was the whole answer: an experimental control plane is not what I want under the services I actually depend on.

The resolution does not answer the root objection, it narrows what that objection can reach. I accepted a rooted daemon, for the sake of learning the thing I set out to learn and because the rootless path was experimental, and then designed so that a rooted node is not a threat to anything worth protecting. k3s hosts nothing except the application tier, which is exactly the tier the threat model already writes off as assume-popped. The gates, the identity provider, acme.sh, AdGuard Home, and the offline CAs never enter the cluster; they stay on the rootless side where a root daemon would have been the wrong trade. Nginx stays the only security edge either way. Traefik, in the cluster, does routing and nothing else, terminates no TLS, checks no certificate, runs no auth middleware, so a compromised app pod inherits none of that authority just by living behind it.

Joining a rootless proxy to a rooted cluster is section 4f.

The applications are moving into a single-node k3s cluster. Not the gates, not the certificate authorities, not the reverse proxy: only the tier the threat model already labels "assume popped". The security model did not change, it got re-expressed in different primitives, and a migration like that only holds if every control that existed before has a named counterpart afterwards. The mapping below is the contract.

The row everyone writes down and nobody prices is the first one. hostUsers: false needs idmapped mounts end to end, and that means a kernel of 6.3 or newer, because ext4 has supported idmapped mounts since 5.12 but tmpfs support landed in 6.3, and every projected and secret volume a pod mounts is a tmpfs. It means k3s v1.33 or newer, and it means ext4, xfs or btrfs under the app data path. Debian 12 ships 6.1 and fails this, so the backports kernel or Debian 13 comes before k3s does.

None of that fails loudly. A pod on the wrong kernel schedules, starts, serves traffic and passes every check you would think to write, while the isolation you believe you configured is not there. The check that catches it is one line inside a running pod: read /proc/self/uid_map. A non-identity mapping means the namespace is real. 0 0 4294967295 means user namespaces are silently off and every claim in the mapping is theatre. It is a preflight gate before install and a per-pod check after.

Four more things ride along. automountServiceAccountToken: false on every pod and on each app namespace's default ServiceAccount, so a popped pod cannot even authenticate to the API server, let alone act. Images pinned by digest in committed manifests, bumped deliberately. Secrets sealed before they ever touch git, so only the encrypted form is committed, with cluster secrets-encryption covering the unsealed copy on disk and an offline backup of the sealing keys, because losing those makes every sealed blob in the repository unrecoverable. And local PersistentVolumes pointed at the existing app data directories, so backup tooling and paths never move.

Two of the things k3s ships with are turned off. ServiceLB would bind 80 and 443 on every node address, which is gate-adjacent surface on a host whose whole posture is one deliberately bound port. The default local-storage provisioner would quietly place app data outside ~/stacks/, escaping the backups: not a security problem, but a good way to lose data and not find out for months.

That storage choice has one real cost. Under hostUsers: false the on-disk owner becomes a moving subordinate ID rather than a stable uid, so host-side backups of migrated apps must run as root. The paths survive the migration. The credentials do not.

The last split is administrative. kubectl runs as the sudo user with its own copy of the kubeconfig. The Docker-side user gets no cluster access at all, and, less obviously, no write access to anything kubectl applies. A manifest a privileged identity will later apply unattended is a deferred credential, and a popped container that can edit one does not need a kubeconfig to become cluster-admin, it just needs patience.

Nginx runs in rootless Docker. The cluster does not. Something has to join them.

The obvious bridge is Docker's host-gateway, and it is unsound here. Rootless Docker's bridge networks live inside RootlessKit's own network namespace, not on the host, so host-gateway resolves to a gateway address belonging to the daemon's namespace. A proxy_pass there hits nothing. Instead, egress goes through slirp4netns in userspace, so the packet leaves the namespace and slirp performs an ordinary connect() on the host, as the Docker user. Two consequences follow. The host never sees the bridge subnet as a source address, so a firewall rule scoped to that subnet can never match and was unenforceable as written. And local connections arrive over loopback, where the firewall's implicit loopback accept lets them straight through. The usual workaround, re-enabling host-loopback so the conventional host.docker.internal address reaches the host, was rejected: it would let every rootless container reach every host loopback service, which trades one seam for all of them.

The resolution is a dedicated dummy interface owning a single address that nothing external can route to, dialled as an IP literal. The literal is not stylistic. Written as a hostname it failed, because proxy_pass with a variable defers resolution to request time and uses Nginx's own resolver, which never consults /etc/hosts. The static host alias was simply ignored and every request 502'd. Certificate identity is pinned separately anyway, so the hostname never carried security weight to begin with.

The address needs guarding, and the outer guard is an nftables rule. It is coarse by construction: it can name a uid and an interface and nothing finer. Every rootless container's egress is performed on the host by the same unprivileged user that owns the daemon, so that uid is the finest distinction the kernel can offer, and that uid is every rootless container, not Nginx. The table below works out what that admits.

acme.sh holds a live Cloudflare token and initiates outbound connections. AdGuard parses untrusted DNS from every client on the network. Both are genuine attack surface and neither is a gate component. Under a uid-only guard, popping either one converts directly into unauthenticated routing to every application behind the seam. That is a real widening over the pre-migration posture, and noticing it is why the original decision to run the seam as plain HTTP was reversed during review.

So the seam carries an inner layer the firewall cannot supply: mTLS, issued by a separate offline CA that signs nothing else. The firewall stops a process that never speaks TLS. The certificate stops one that does but has no key. Neither is load-bearing alone.

The firewall rule has a second job on the way in. Without it a pod could dial the seam from inside the cluster and use Traefik as an east-west bypass, reaching another app's namespace by hairpinning back through the one component allowed to route anywhere. That path is closed twice over, once by the firewall dropping it before it is routed and once by a default-deny NetworkPolicy, and either would be enough on its own. Two independent controls for one bypass is deliberate: this joint should not depend on a single rule I might edit a year from now and misjudge.

Confidentiality was never the point on a loopback hop. Loopback was never observable to anyone who was not already host root. Authentication was the point: the seam is the one place in the design where the firewall cannot name the peer, and a certificate can.