Writing a Prometheus Exporter in Go for Metrics Nobody Was Collecting

Mon, 21 Aug 2023 00:00:00 +0000

The platform I operate is a fleet of NGINX instances running a Web Application Firewall module in front of production traffic. The stock NGINX exporter gave us request counts, status codes, and upstream latency, all useful, and all completely blind to the one question that actually mattered for this platform: is the WAF itself doing its job, or is it about to fall over.

The Gap

The WAF module exposes its own internal counters through a local status endpoint: rule hits, rule blocks broken down by rule ID, requests evaluated per second, memory used by the rule engine. None of it reaches Prometheus by default, because it’s not something the generic NGINX exporter knows exists. Our dashboards could tell us “NGINX is up.” They couldn’t tell us “rule 942100 just started blocking forty times its normal traffic,” which is exactly the kind of thing that means either an attack is underway or a rule update just went sideways and started blocking legitimate users. Both are pages. Neither showed up anywhere.

Observability on Alexander's Blog

Writing a Prometheus Exporter in Go for Metrics Nobody Was Collecting

The Gap