Homelab

1 · What it is

A private network with no public front door. Two hosts, one rented and one at home, joined by a WireGuard tunnel. The public internet sees a single open UDP port on the rented one and nothing behind it. No web server, no login page, no banner, nothing to fingerprint.

Behind the host sits a password manager, a media library, and a handful of small services. Reaching any of them means passing three independent gates in series: a WireGuard peer key, a client certificate, and a second factor. Failing any one of them drops the connection before an application ever sees the request.

All of it could be simpler. A mesh VPN client (Tailscale, NetBird) and one compose file would buy most of the access and most of the convenience in an afternoon. I built it this way because the point was never the convenience. It was to be the person responsible for each layer at two in the morning when it breaks, which turns out to be the only way I actually learn one.

2 · Where it came from

In late April 2025 PewDiePie posted a video about moving to Linux. I followed along, mostly out of curiosity, and stayed. Installing your own operating system makes you notice how much of the rest of your computing you never chose.

That noticing turned into a longer question about what other people can see. What my ISP resolves on my behalf. What a cloud photo service keeps after you delete the photo. What a hosted password manager's server knows about the shape of your life even if it cannot read a single entry. The honest answer to most of those is "more than I would like, and I have no way to check", and the only version of that answer I can verify is the one running on hardware I own.

Self-hosting does not remove that problem, it relocates it. The moment the photos are on a box in my apartment, the box is the thing worth attacking, and I am the only one who is going to defend it. Everything below is the consequence of taking that seriously, and then, repeatedly, taking it further than it needed to go because the further version was the one that taught me something.

The line that circulates in self-hosting circles is that there is no cloud, there is only someone else's computer.

3 · What I am hiding from

The design goal is narrow enough to state in one line: manual authorization for every device, and unencrypted traffic invisible to three parties who may not (or may?) be attackers so much as they are simply there.

The public internet sees one VPS with one open UDP port. WireGuard does not answer unauthenticated packets, so a scanner gets silence rather than a refusal. There is no web server on that host, no TLS handshake to fingerprint, no login form to enumerate, and no version banner to match against a CVE list. The home server has no inbound WAN exposure at all, which is less a hardening choice than a fact of sitting behind CGNAT.

The ISP sees an encrypted UDP stream to one VPS address. It cannot see which service I am reaching, because every subdomain resolves inside the tunnel. It cannot see my DNS queries, because clients are pushed a resolver that only exists on the far side of the tunnel and are given no fallback. A resolver that is unreachable when the tunnel is down fails closed, which is the behaviour I want.

The VPS provider sees WireGuard packets and forwards them. The relay terminates no TLS, holds no application data, holds no certificate authority, and by default performs no NAT. It moves ciphertext between two peers on the overlay and has no key that decrypts any of it. Renting a machine from someone is a trust decision, and this is the shape that decision takes when you would rather not make it. Keeping the relay dumb also keeps it disposable. It stores nothing, so moving to another provider costs a new keypair and a config edit rather than a migration, and if it disappears entirely the server is still sitting on the local network where I left it.

A compromised app gets one app. On the rootless Docker side that means every application runs with all capabilities dropped and no-new-privileges set, on an isolated per-app network with no route to the host, to the identity provider, or to another app's data. The reverse proxy is the only container attached to more than one network, and the only one with a host-bound port. Applications that have moved into the cluster get the same posture in Kubernetes terms instead: one namespace per app, a default-deny network policy in both directions, and no service account token to authenticate with even if something inside the pod goes looking for one. Section 4e maps the two sides control by control. Neither of them gives a compromised app a route to the gates.

The stated non-goal matters more than any of the four. None of this defends against a compromised client device that holds a valid certificate and a valid WireGuard key. That device is authorized, by definition, and pretending otherwise would be the kind of claim that makes the rest of the list untrustworthy. It is a revocation problem, and revocation is a documented procedure rather than a hope.

4 · How it works

The server sits behind CGNAT: no public IP, no port forwarding, not even in principle. So a cheap rented VPS becomes the only node with a real public address, and it does exactly one job. It is not a proxy. It never terminates TLS, never reads a payload, never holds a key that decrypts anything. It forwards WireGuard ciphertext between peers and that is the entire job description.

There are three roles on the resulting overlay: the relay at 10.0.0.1, the host at 10.0.0.2, and one address per authorized client from 10.0.0.3 onward. Everything past that point, the diagram below, is what happens once a packet has already crossed the tunnel and landed on the host: three gates in series, then one of two tails depending on which side of a migration a service is still on.

4a · The three gates

Reaching an application means passing three checks, in order, each enforced by a different piece of software with no shared state between them. Failing any one drops the connection before the next gate is even reached.

Gate 1 is WireGuard itself. A device only gets to place a packet on the 10.0.0.0/24 overlay if its public key is already listed as a peer on the relay. There is no username or password here, no banner to greet an unrecognized peer with: an unauthorized handshake just gets silence.

Gate 2 is mutual TLS at Nginx. Every client presents a certificate, and Nginx verifies it against a private, offline-signed Root CA, checking a certificate revocation list on every connection rather than trusting the chain alone. A stolen WireGuard key is useless here without the matching client certificate, and a lost or compromised device is cut off by revoking its certificate, regenerating the list, and reloading Nginx, which caches the list in memory. No other device is rekeyed.

What makes this gate worth calling out on its own is that it is enforced globally, at the listener level, for every hostname behind Nginx, core service or app tier alike. Nginx picks a virtual server twice for one request, once by TLS SNI and once by the Host header, and those two choices are allowed to disagree. If even one hostname's TLS configuration skipped client verification, that mismatch would be a way in: handshake against the unprotected host, then send a Host header naming a protected one. Because no hostname is exempt, there is no weak virtual server to pivot through, which closes that entire bypass class by construction instead of by a rule that has to be remembered for every new service.

Gate 3 is Authelia, sitting behind an auth_request subrequest and enforcing two-factor login under a default-deny policy: nothing is reachable unless a rule explicitly allows it. Vaultwarden ships its own SSO and is deliberately allowed to skip this gate and rely on its own login instead, a stated exception, not a gap that got missed.

Once all three gates pass, the path forward splits. Services still running in rootless Docker are reached by plain container name on a private, per-app Docker network. Services that have migrated to the k3s app tier are reached through a separate seam interface, its own small gate of a different kind, covered in section 4f.

4b · DNS

AdGuard Home runs bound to the WireGuard interface address specifically, not to 0.0.0.0, which means it is not merely firewalled off from the outside world, it is physically unreachable from outside the tunnel: there is no socket listening anywhere else for an outside query to land on. A wildcard rewrite resolves every service subdomain to the host internally, and no public A or CNAME record for any subdomain exists anywhere in public DNS. There is nothing for an enumeration tool to enumerate, because the names were never published in the first place. It also means DNS inherits the first gate for free: no authorized peer, no resolution, and no way to discover that a service exists at all.

Clients get this resolver pushed automatically: the DNS line lives in the [Interface] block of their own WireGuard config, set with no fallback resolver configured alongside it. Off-tunnel, those names simply do not resolve, and that failure mode is the point: a resolver that fails closed when the tunnel is down is safer than one that quietly falls back to a public server and starts leaking query patterns.

Where AdGuard has to go outside, it goes encrypted. Public names resolve over DNS-over-HTTPS to Quad9's unfiltered endpoint, so my own ISP sees one encrypted stream to one resolver rather than a list of everything the lab looked up. Unfiltered is the right variant because filtering is AdGuard's job, and doing it there is the reason it is AdGuard rather than any plain resolver: one blocklist applies to every device on the tunnel at once, including the phones that have no good way to do it for themselves. The fallback list is empty, so the fail-closed posture holds at the resolver as well as at the client. If the upstream cannot be reached, resolution stops rather than quietly finding another way out. The one thing that does leak is the bootstrap, since the resolver's own hostname has to be resolved in the clear before the encrypted channel exists. That tells an observer which provider I chose and nothing whatsoever about what I ask it.

The same posture continues inside the cluster for the app tier. Pods talk only to CoreDNS, never directly to the host, and CoreDNS is handed AdGuard as its own upstream through a kubelet resolv-conf override rather than the cluster's usual path. That one narrow plumbing route means nothing has to open a hole in any app's NetworkPolicy just to let it reach a DNS port on the host, and there is no fallback public resolver defined here either. If AdGuard goes down, pod DNS goes down with it. That is accepted on purpose: it is the identical fail-closed choice made for clients, applied uniformly to workloads that never asked for an opinion on it.

4c · Certificates

There are three certificate authorities on this network, and the interesting fact is not that there are three, it is that none of them is allowed to sign for another.

The first is Let's Encrypt, reached over DNS-01. There is no inbound HTTP path to validate against, so acme.sh proves control of the zone by writing a TXT record instead. It runs as a persistent container with no Docker socket, and that omission is deliberate: acme.sh is the one container with a job that requires it to face the internet, which makes it the most plausible thing on the host to get popped, and handing a popped container the socket hands it the rest of the rootless daemon. Its Cloudflare API token is scoped to Zone:DNS:Edit on a single zone, never account-level, so a leaked token can rewrite one _acme-challenge record, not billing, not other zones, not any other record in that zone. The part worth explaining is the reload path. acme.sh cannot reach into Nginx's container to tell it to reload, so a renewed certificate lands on disk and stops there. What closes the loop is a systemd user path unit on the host, watching the certificate directory and firing a reload when it changes. Skip that unit and nothing looks wrong: renewal succeeds, files on disk are current, and Nginx keeps serving the old certificate from memory anyway, invisible until it expires and every client fails at once.

The second is a private, offline Root CA, used for per-device client certificates. Its key is AES-256 encrypted and unlocked only by a passphrase I hold. Certificates are issued per device and exported as .p12 bundles so they install the same way on a phone, a laptop, or a browser. Revocation runs through a real OpenSSL CRL, reloaded into Nginx, rather than by deleting the peer and hoping. That distinction matters the first time a device is actually lost: revoking it does not force every other device to re-enroll, it just adds one serial to a list Nginx already checks.

The third is SeamCA, and it exists only because the second one cannot do this job. A certificate signed by the Root CA is, by construction, a valid client identity for the access gate, so using it to authenticate the internal hop between Nginx and the cluster would mean a working user credential sitting in Nginx's config and inside the Traefik pod. A transport credential must never double as a spendable user credential. SeamCA signs exactly two certificates, one for each side of that link, and stops there. There is no CRL because there is no population to revoke from: with two long-lived certificates on one host, the answer to "this seam is suspect" is reissue and redeploy.

4d · Choosing runtimes

Docker came first, and the honest reason is not a technical one: I wanted to learn Docker specifically. Podman is the obvious alternative and it never got a real hearing from me, because the choice had already been settled by what I wanted to learn rather than by what the two engines actually do differently. Presenting that as a considered tradeoff would be a lie, and the plain version is the more useful sentence anyway.

What I did care about was not running a rooted container engine, and that is what put me on rootless Docker for everything, not Docker as such. The unprivileged daemon means a container escape does not hand the attacker host root by default, which is the property the rest of the design leans on.

Kubernetes arrived the same way Docker did, to learn it, and ran into the same objection. A default k3s node runs as root, so rootless k3s was the first thing I looked at. Its own documentation labels it experimental, and that was the whole answer: an experimental control plane is not what I want under the services I actually depend on. I could still run it that way as an exercise and I might, but it was not the version of this I wanted to build.

The resolution does not answer the root objection, it narrows what that objection can reach. I accepted a rooted daemon, for the sake of learning the thing I set out to learn and because the rootless path was experimental, and then designed so that a rooted node is not a threat to anything worth protecting. k3s hosts nothing except the application tier, which is exactly the tier the threat model already writes off as assume-popped. The gates, the identity provider, acme.sh, AdGuard Home, and the offline CAs never enter the cluster; they stay on the rootless side where a root daemon would have been the wrong trade. Nginx stays the only security edge either way. Traefik, in the cluster, does routing and nothing else, terminates no TLS, checks no certificate, runs no auth middleware, so a compromised app pod inherits none of that authority just by living behind it.

Joining a rootless proxy to a rooted cluster turned out to be the most delicate part of the whole build, and it is section 4f.

4e · The app tier

The applications are moving into a single-node k3s cluster. Not the gates, not the certificate authorities, not the reverse proxy: only the tier the threat model already labels "assume popped". The security model did not change, it got re-expressed in different primitives, and a migration like that is only honest if every control that existed before has a named counterpart afterwards. That mapping is the contract.

The row everyone writes down and nobody prices is the first one. hostUsers: false needs idmapped mounts end to end, and that means a kernel of 6.3 or newer, because ext4 has supported idmapped mounts since 5.12 but tmpfs support landed in 6.3, and every projected and secret volume a pod mounts is a tmpfs. It means k3s v1.33 or newer, and it means ext4, xfs or btrfs under the app data path. Debian 12 ships 6.1 and fails this, so the backports kernel or Debian 13 comes before k3s does.

None of that fails loudly. That is the part worth sitting with. A pod on the wrong kernel schedules, starts, serves traffic and passes every check you would think to write, while the isolation you believe you configured is not there. The check that catches it is one line inside a running pod: read /proc/self/uid_map. A non-identity mapping means the namespace is real. 0 0 4294967295 means user namespaces are silently off and every claim in the table above it is theatre. It is a preflight gate before install and a per-pod check after, because a control you cannot verify is a belief. It is a deadbolt installed and never thrown: the door looks exactly the same from either side.

Four more things ride along. automountServiceAccountToken: false on every pod and on each app namespace's default ServiceAccount, so a popped pod cannot even authenticate to the API server, let alone act. Images pinned by digest in committed manifests, bumped deliberately. Secrets sealed before they ever touch git, so only the encrypted form is committed, with cluster secrets-encryption covering the unsealed copy on disk and an offline backup of the sealing keys, because losing those makes every sealed blob in the repository unrecoverable. And local PersistentVolumes pointed at the existing app data directories, so backup tooling and paths never move.

Two of the things k3s ships with are turned off. ServiceLB would bind 80 and 443 on every node address, which is gate-adjacent surface on a host whose whole posture is one deliberately bound port. The default local-storage provisioner would quietly place app data outside ~/stacks/, escaping the backups, which is not a security problem but is a good way to lose data and not find out for months. Auditing what a distribution enables on your behalf is a separate job from configuring what you added yourself.

That storage choice has one real cost and it is not the one you would guess. Under hostUsers: false the on-disk owner becomes a moving subordinate ID rather than a stable uid, so host-side backups of migrated apps must run as root. The paths survive the migration. The credentials do not.

The last split is administrative. kubectl runs as the sudo user with its own copy of the kubeconfig. The Docker-side user gets no cluster access at all, and, less obviously, no write access to anything kubectl applies. A manifest a privileged identity will later apply unattended is a deferred credential, and a popped container that can edit one does not need a kubeconfig to become cluster-admin, it just needs patience.

4f · The seam

Nginx runs in rootless Docker. The cluster does not. Something has to join them, and the joint turned out to be the most instructive thing I built.

The obvious bridge is Docker's host-gateway, and it is unsound here. Rootless Docker's bridge networks live inside RootlessKit's own network namespace, not on the host, so host-gateway resolves to a gateway address belonging to the daemon's namespace. A proxy_pass there hits nothing. What actually happens to an outbound packet is stranger and more useful to know: egress goes through slirp4netns in userspace, so the packet leaves the namespace and slirp performs an ordinary connect() on the host, as the Docker user. Two consequences follow and both matter. The host never sees the bridge subnet as a source address, so a firewall rule scoped to that subnet can never match and was unenforceable as written. And local connections arrive over loopback, where the firewall's implicit loopback accept lets them straight through. The usual workaround, re-enabling host-loopback so the conventional host.docker.internal address reaches the host, was rejected: it would let every rootless container reach every host loopback service, which trades one seam for all of them.

The resolution is a dedicated dummy interface owning a single address that nothing external can route to, dialled as an IP literal. The literal is not stylistic. Written as a hostname it failed, because proxy_pass with a variable defers resolution to request time and uses Nginx's own resolver, which never consults /etc/hosts. The static host alias was simply ignored and every request 502'd. Certificate identity is pinned separately anyway, so the hostname never carried security weight to begin with.

The address needs guarding, and the outer guard is an nftables rule. It is coarse by construction: it can name a uid and an interface and nothing finer. Every rootless container's egress is performed on the host by the same unprivileged user that owns the daemon, so that uid is the finest distinction the kernel can offer, and that uid is every rootless container, not Nginx.

Working out what that actually admits is the analytical heart of the whole build:

acme.sh holds a live Cloudflare token and initiates outbound connections. AdGuard parses untrusted DNS from every client on the network. Both are genuine attack surface and neither is a gate component. Under a uid-only guard, popping either one converts directly into unauthenticated routing to every application behind the seam. That is a real widening over the pre-migration posture, and noticing it is why the original decision to run the seam as plain HTTP was reversed during review. I am stating the reversal rather than quietly shipping the better version, because a design that records where it was wrong is worth more than one that only shows its conclusions.

So the seam carries an inner layer the firewall cannot supply: mTLS, issued by a separate offline CA that signs nothing else. The firewall stops a process that never speaks TLS. The certificate stops one that does but has no key. Neither is load-bearing alone.

The firewall rule has a second job on the way in. Without it a pod could dial the seam from inside the cluster and use Traefik as an east-west bypass, reaching another app's namespace by hairpinning back through the one component allowed to route anywhere. That path is closed twice over, once by the firewall dropping it before it is routed and once by a default-deny NetworkPolicy, and either would be enough on its own. Two independent controls for one bypass is deliberate: this joint should not depend on a single rule I might edit a year from now and misjudge.

Confidentiality was never the point on a loopback hop. Loopback was never observable to anyone who was not already host root. Authentication was the point: the seam is the one place in the design where the firewall cannot name the peer, and a certificate can.

5 · Running it

A design is only as real as its second occurrence. The first time you do something on a homelab you are debugging; the fifth time you had better be following a document, or you were never running a service, you were performing a trick you happen to remember.

Provisioning a new device is the one I hit most. A phone or laptop gets a WireGuard peer authorized on the relay, a client certificate cut from the internal CA and packaged as a .p12 so it drops into iOS, Android, Windows, and Linux the same way, and its resolver locked to the internal DNS so the tunnel is the only place its queries can go. None of the three steps is hard alone; skipping the DNS lock is what quietly leaves a working device that leaks its lookups the moment it steps off the VPN.

Losing a device does not delete its peer. WireGuard has no notion of a certificate going bad, so the kill switch lives one layer up: the device's certificate goes onto a revocation list, and the reverse proxy reloads that list. A lost phone stops completing the handshake without me touching its WireGuard entry or guessing its current key.

Adding a new service forks in two directions. Anything that is part of the gate chain itself, the proxy, the identity provider, the DNS filter, stays in rootless Docker and never moves. An ordinary application goes into the cluster as manifests instead. Either path ends the same way: a certificate for the new hostname, a server block naming it, and an access rule saying whether the app needs its own second factor or trusts the login it already has.

Secrets get sealed against the cluster's public key before they ever touch git, decrypted only by the controller holding the matching private half, and the keys behind that get backed up offline on a schedule, because a sealing key lost with no backup means every secret in the repository is unrecoverable, not just inconvenient.

Day to day, running the thing means a fixed start and stop order, AdGuard up first and down last. The reason is not that things depend on it, it is that stopping AdGuard while a pod is running does not take that app down. It silently strips name resolution from it, which is a great deal harder to diagnose than an outage. Nginx is the other end of the same idea: with Nginx stopped nothing is reachable at all, the cluster included, because Nginx holds the only seam client certificate and Traefik has no admissible caller without it. Past the order there is a small set of log aliases, so an authentication failure is a two-second lookup instead of a fishing expedition, and a periodic check of how much runway each certificate has left. None of this is clever, and that is the point: clever is what a homelab looks like right up until its author is asleep or has forgotten, and a system only its author can operate on a good day is not operated at all.

6 · Hardware and cost

A used Dell office PC. It was cheap and it works, and that is the entire reasoning. The money that was left went where it actually matters: an SSD for the operating system, two hard drives, one for data and one holding its backup, and the small pile of accessories a machine needs before it is a server, down to the ethernet cable. $290, hardware only. The VPS relay is separate and runs under seventy dollars a year.

The hardening that happens below the operating system is worth naming. The BIOS has the wireless radios, the microphone, and the unused USB ports disabled outright, and deep sleep configured so idle standby draw is close to nothing. A radio or a microphone on a machine with no legitimate use for either is attack surface with no matching benefit, and switching it off at the firmware level means no operating system change can quietly switch it back on.

7 · Accepted trades

Every decision above bought something by giving something up, and a page that only lists the wins is not describing a real system.

The cluster's own daemon runs as root, a step down from the rootless Docker ceiling everything else here holds itself to. I accepted that because only applications the threat model already treats as pre-compromised land in the cluster; the reverse proxy, the identity provider, and the certificate authority never enter it. A container escape chained into a kernel exploit reaches host root there, worse than the rootless case, and the compensating control is that nothing valuable is there to reach.

SSH on the relay answers any address on the internet, port 22, no knock sequence and no nonstandard port. That is deliberate: obscurity is not a control, it is a delay. The defenses are key-only authentication and fail2ban, and moving the port would quiet the logs rather than change what an attacker has to defeat. I would rather read honest logs than quiet ones. The host is a different case. Its firewall allows 22 as well, but it sits behind CGNAT with no inbound path from the internet at all, so the only ways in are the local network or the tunnel. Its fail2ban has never banned anything, which is exactly what a port nothing outside can route to should look like.

The certificate on the internal hop between Nginx and Traefik is a single point of failure for the whole application tier: expire it and every route fails at the handshake at once. I accepted that against the cost of renewing an internal-only certificate on a short cycle for no user-facing benefit; the mitigation is a long validity window plus a standing check on remaining lifetime. Restarting Traefik carries a smaller version of the same risk, a brief outage while it comes back.

There is one node and no failover. I did not build high availability, and the cluster does not need it: it introduces no single point of failure the host itself was not already. Losing the box loses availability, and as things stand it loses the backup too, because the second disk is inside the same box. That is the gap section 8 is about.

A few narrower trades round this out. The namespace holding the cluster's own plumbing gets no network policy, since that plumbing changes shape on every upgrade and a broken policy there would take the cluster down for a marginal gain; it is compensated by auditing that namespace as a fixed inventory instead. The firewall guarding the internal hop into the cluster admits a class of local processes rather than one specific one, closed by a client certificate on top. And the application tier shares one credential for reaching Traefik rather than one per app, since splitting it further would buy separation the proxy's routing already provides.

8 · What is next

The migration into the cluster works and it is partway done. The pattern held up under real cutovers: one namespace per app, default-deny between them, secrets sealed before they touch git. What is left is the rest of the applications, moved one at a time, each verified working before its predecessor is retired.

The interesting one still to come is a stateful photo app, and it is the honest test of whether the pattern generalizes. Several components in one namespace, a large existing data volume that needs a slow first-run permission fix, and one component that legitimately needs outbound internet access while the rest of the app does not. A canary proves the plumbing works. That one proves the pattern does.

The other real gap is backup. A second disk in the same box holds a deduplicated copy of the data, which covers a failed drive and covers me deleting something I wanted back. It does not cover theft, fire, or a burst pipe, because both copies are in the same room, and a backup that shares a room with its original is a copy, not a backup. The fix is the third leg of 3-2-1: an encrypted copy in cloud storage, encrypted here before it leaves the house so the provider holds ciphertext and nothing else. That is the same posture the relay already has, applied to the one thing that would actually hurt to lose. It is the next piece of work here that is load-bearing rather than merely interesting.

Past that I am not going to invent a roadmap. Where this goes next depends on what I want to learn next, which is the same force that put a Kubernetes cluster behind a reverse proxy in my apartment in the first place.