I run a lot of things on my own servers. This website, my git server, quorrespondence.com, a few sites I host for other people, and a box with a GPU in it that runs local AI models. That is a lot of things that can quietly break. For a long time, the way I found out something was down was that I happened to visit it, or somebody told me.

The corporate answer to this problem is a monitoring service, where you pay a subscription and ship a copy of everything your servers do into someone else's cloud. If you read my post about self-hosting my code, you can guess how I feel about that. So I built Oryx: a monitoring system made of free and open source parts that runs entirely on my own hardware and phones home to nobody.

The name

Oryx, the Taken King, is a raid boss from Destiny. In the lore, Oryx sits in a throne world while his armies of thralls pay him tribute. That maps onto a monitoring system embarrassingly well. The throne is a Raspberry Pi on my wall named MECHATHRONE. Every server is a thrall, and the telemetry it sends home is tribute.

The throne

The throne is an old Pi 3B with 1 GB of RAM. It runs Prometheus, which collects and stores all the numbers, and Grafana, which draws them. Prometheus writes to disk constantly, so the data lives on an SSD instead of the SD card, because an SD card under that kind of write load is going to die sooner or later.

The Pi also drives a screen on the wall. It boots straight into a full-screen browser with nothing else on it, and rotates through the dashboards on a playlist. That screen is the whole point of the project. I wanted a wall in my house where I can glance up and see that everything I run is alive.

The mesh

The Pi and the servers talk over WireGuard, which is a VPN built into the Linux kernel. Every box generates its own keys, and the only thing any of them exposes to the internet is a single port. Each thrall runs two small agents: one that reports on the machine itself (CPU, memory, disk, temperature), and one that reports on the containers running on it. The GPU box runs a third agent just for the GPU. The agents bind only to the mesh address, so none of the telemetry is reachable from the public internet.

Prometheus polls every thrall from the throne. When a server goes down, its numbers stop arriving, and its tile on the wall goes dark. That tile going dark is the entire reason all of this exists.

The dashboards

Every dashboard is committed to the repo as code, so the whole wall can be rebuilt from scratch. There is an infrastructure board with CPU, memory, and disk for every thrall, and gauges for the GPU. There is a liveness board with a tile per thrall, plus a strip that probes the public sites from the outside and watches response times and whether any HTTPS certificate is about to expire. There is a CI board showing the latest build for every repo on my Gitea. There is a fleet board for the Quorrespondence boards out in the world. And all of it is skinned like a mecha HUD, because if I am going to stare at a wall of numbers all day, the wall should look cool. Even the login banner on the throne is a mecha status readout.

The security screen

Somewhere along the way, Oryx also became a security tool. A scanner walks every repo I host and checks the exact version of everything it depends on against the public vulnerability databases. Another scanner checks the container images I actually run. A third tracks how far behind the latest release each piece of off-the-shelf software is, so nothing quietly rots for a year. All of it lands on one security dashboard: vulnerabilities by severity, version drift, certificate expiry.

The first version of the vulnerability scanner ran on the Pi. It turns out a vulnerability scan is a great way to knock over a computer with 1 GB of RAM. The Pi kept rebooting itself until the scanner moved to the GPU box.

This is the one screen I will not be depicting here, for reasons that are hopefully obvious.

Problems

The wall was phoning home. Chromium ships with its telemetry turned on, so the machine whose entire job is watching my network was also the one leaking data off of it. That is turned off now.

The kiosk also slowly eats the Pi. The wall kept growing dashboards, the browser kept growing memory, and eventually the Pi started thrashing. The fix is not elegant: the kiosk restarts itself every night at 4am.

One server refused to join the mesh, and after checking keys and firewalls, the bug turned out to be a single digit. The throne was dialing port 51280 instead of 51820.

And the GPU box stopped reporting its GPU after a driver update. A reboot fixed the driver, but the GPU agent still came up empty, and the real fix was recreating its container instead of restarting it.

The throne world

Oryx has been running for about a month. Six boxes are on the mesh, counting the throne. As of today, deploys are push-based: the build server pushes out a signed bundle, and every box verifies the signature before applying it. The wall is on the wall. When something of mine breaks now, a tile goes dark, and I see it from the couch.