DevOps

A follow-up to the wildcard bug post from last year. Picking the Thread Back Up The rename workaround, *.example.com to _.example.com, fully resolved the operational problem. Retention became predictable again, and there was no pressing reason to go further. But a workaround isn’t a fix, and the actual bug was still sitting in logrotate’s own source, waiting to bite the next person who names a file after a wildcard domain. This week I finally sat down and found it properly, instead of just renaming files around it. ...

One of those infrastructure problems that looks simple, until you realize the filename itself is part of the problem. The Setup We had Nginx configured to handle wildcard subdomains: server_name *.example.com;, with access logs named after the server block. That’s a convenient pattern for multi-tenant setups where subdomains get created dynamically and you don’t want to hand-maintain a vhost file per tenant. Nginx doesn’t care that the resulting log filename contains a literal * - it just opens the file and writes to it. ...

Deploying used to mean an engineer, a runbook document, and four hours of undivided attention. Not four hours of a pipeline running in the background, four hours of a person actively driving it: SSH into the target host, pull the latest code, run the build script by hand, stop the old process, start the new one, watch the logs to see if it came up clean, and post a status update. It worked, in the sense that it got code to production. It didn’t scale, in the sense that “get code to production” now depended entirely on one specific person having a free afternoon. ...