My audit tool was sending a password in plain text
My home lab has an audit tool that checks everything against its documentation. One of the things it talks to is the API of my DNS filter, and that API needs a password. For a while, that password went over plain HTTP on every single run.
What was going on
Every service in the lab gets its certificate from my own internal
certificate authority, and the servers trust it. What I hadn't thought
about is that Python's requests library doesn't use the
system's trust store. It ships its own list of certificate
authorities, and mine obviously isn't on it.
So the HTTPS connection failed verification, and the collector fell back to HTTP without complaining. No crash, nothing in the report that looked wrong. The data came back fine, which is exactly why nobody would notice.
The fix
One line in the service's unit file, telling Python to use the system's CA bundle:
Environment=REQUESTS_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crtThe part that actually bothered me
A while later a routine update replaced that unit file with a copy that didn't have the line, and nothing broke. systemd keeps the environment a service was started with, so the running process still had the setting even though the file on disk didn't. It would have quietly reverted on the next reboot.
I only caught it by looking at the live process instead of the config file:
cat /proc/$(systemctl show -p MainPID --value <service>)/environ | tr '\0' '\n'What guards it now
The setting lives in the copy of the unit file that ships with the app, so an update can't drop it. There's a one-line check that compares the live unit with the shipped one, and the audit itself raises a finding if that API is ever reached over plain HTTP again.
The habit I took from it is simple enough. When something matters, check what's actually running, not what a file says should be running.