Troubleshooting¶
The runbook pages state each command's own local failure mode inline. This page collects the four incidents from this week that were sharp enough to name on their own — each is a real thing that happened while this runbook was being written, not a hypothetical.
The scp truncation¶
Where: §1.3, installing a binary.
scp exited 0 on a connection that dropped mid-transfer over a flaky link.
The destination file was short and the process wouldn't even start (exec
format error or an immediate crash-loop in journalctl). scp's exit code
proves the local side finished writing, not that the remote file matches.
Fix: compare SHA-256 on both ends, always — never trust the exit code
alone, and never scp a new build straight onto the path an active systemd
unit already points at. Install to a distinctly-suffixed new path, verify,
then repoint ExecStart.
The kubectl.kubernetes.io/last-applied-configuration leak¶
Every Secret write in this project uses
kubectl create --dry-run=client -o yaml | kubectl replace -f -, never
apply — kubectl apply stamps the full applied manifest, values and all,
into that annotation, and a Secret's manifest is the value.
secret-annotation-check (Tenant-Service, deploy/dev/) runs on a schedule
in zerotouchdeployment and fails closed if it can't check.
The Device-Order-Service Deployment (not a Secret) was found carrying
the same annotation while this runbook was being written — low-sensitivity
there, since a Deployment spec carries secretKeyRef names, never secret
values, but the same mechanism, and the same fix: kubectl replace (or a
minimal, targeted patch that touches only the field being changed), never a
blanket apply of a full rendered manifest over a live object you don't
control every field of.
Fix: know which command you're running and why. apply computes a
three-way merge against whatever the object's own last-applied annotation
says — on an object something else already owns fields on (an in-cluster
tracking annotation, a foreign controller's own field), that merge can
silently strip what it doesn't know about. replace, or a narrow kubectl
patch/set image, touches only what you actually mean to change.
The dentry cache "surviving a delete"¶
Where: §1.6, rollback.
rm-ing the old binary before repointing ExecStart makes ls /mnt/data
show it gone immediately — but the running process still has the file open.
Linux does not reclaim an unlinked file's inode, or stop executing it, until
every process holding it exits. The service looks rolled back while the
old code is still what's actually serving requests.
Fix: readlink /proc/$(pgrep -f bootstrap-agent | head -1)/exe, not
ls, if you need to know which build is really running. Always restart the
unit rather than relying on a delete to end the old process.
The loopback trust rule faking a password check¶
Where: §2, reading persisted data.
internal/localapi/peercred_linux.go reads SO_PEERCRED on every mutating
call against the gateway's local Unix socket, and its own comment says why:
"LOCAL DOES NOT MEAN TRUSTED." That read is for the audit log only —
it records which local UID/PID called, it does not gate anything. There is
no password, no token, no second factor on this socket. The only real
access control is the socket's own file permissions (group tss, mode
0660) — anything in that group, or root, reaches every endpoint, full
stop.
Fix: don't mistake peer-credential logging for an authentication check. If a caller needs to be refused, the refusal has to be the file permissions, or a real check added deliberately — not an assumption drawn from the audit trail existing.