Getting to 99.9%: Kubernetes and Real Observability for 20+ Microservices

Mon, 15 Sep 2025 00:00:00 +0000

99.9% availability sounds like a slogan until you do the arithmetic: it’s a budget of roughly eight and a half hours of downtime for an entire year, across every service on the platform. Once you frame it that way, it stops being an aspiration and becomes an engineering constraint you either design for or blow through by month three. Getting there for a platform of 20+ microservices took deliberate work in two places that don’t get enough credit next to “just add more replicas”: the rollout path, and observability that’s actually fast enough to matter.

Writing a Prometheus Exporter in Go for Metrics Nobody Was Collecting

Mon, 21 Aug 2023 00:00:00 +0000

The platform I operate is a fleet of NGINX instances running a Web Application Firewall module in front of production traffic. The stock NGINX exporter gave us request counts, status codes, and upstream latency, all useful, and all completely blind to the one question that actually mattered for this platform: is the WAF itself doing its job, or is it about to fall over.

The Gap

The WAF module exposes its own internal counters through a local status endpoint: rule hits, rule blocks broken down by rule ID, requests evaluated per second, memory used by the rule engine. None of it reaches Prometheus by default, because it’s not something the generic NGINX exporter knows exists. Our dashboards could tell us “NGINX is up.” They couldn’t tell us “rule 942100 just started blocking forty times its normal traffic,” which is exactly the kind of thing that means either an attack is underway or a rule update just went sideways and started blocking legitimate users. Both are pages. Neither showed up anywhere.

Prometheus on Alexander's Blog

Getting to 99.9%: Kubernetes and Real Observability for 20+ Microservices

Writing a Prometheus Exporter in Go for Metrics Nobody Was Collecting

The Gap