1. Where the gateway's own services come from¶

There is no package and no target image. deploy/yocto/ in Bootstrap-Agent is a real recipe, but nothing on this device was installed from it — nobody has built or booted a Yocto image for this hardware. Bootstrap-Agent's own README.md says "a device gets the binary through the Yocto recipe"; that line describes a process that has never actually run. What is real, on this device, today:

Three hand-built binaries under /mnt/data/: bootstrap-agent-<short-sha>, gateway-agent, wlpc-opsoftware (the repo is wlpc-opsoftware-acceptance; the installed binary is not).
Installed by scp, verified by SHA-256 on both ends, by hand, every time.
Three systemd units, each hand-adapted for this specific device's image — not the units committed to each repo. Diff them before assuming the repo is what's running.

1.1 Build (Bootstrap-Agent, armv7l)¶

$ uname -a   # on the gateway, confirm the target first
Linux edge-gateway 6.6.31-bsp-yocto-ampliphy-i.mx6ul-pd24.1.0-devel #1 SMP ... armv7l GNU/Linux

Bootstrap-Agent's .woodpecker.yml builds a linux/arm64 container image — that is for CI's own scan/integration use on pallax-dev nodes, which really are arm64. It is not what ships to a device. Do not build that image and expect it to run here.

cd Bootstrap-Agent
SHA=$(git rev-parse --short HEAD)
CGO_ENABLED=0 GOOS=linux GOARCH=arm GOARM=7 go build \
  -trimpath \
  -ldflags "-s -w -X main.version=${SHA}" \
  -o /tmp/bootstrap-agent-${SHA} ./cmd/bootstrap-agent

Flags, and why each one:

Flag	Why
`CGO_ENABLED=0`	static binary — the device's musl/glibc mix is not the build machine's
`GOARCH=arm GOARM=7`	the actual target, confirmed above, not the CI image's arm64
`-trimpath`	no build-machine paths in a binary that leaves this laptop
`-ldflags "-s -w"`	stripped, matching the Dockerfile's own convention
`-X main.version=${SHA}`	the artefact can identify itself afterward (§1.2)

Verified live: built clean, file reports ELF 32-bit LSB executable, ARM, EABI5 ... statically linked, stripped, 6,946,978 bytes. Note this is smaller than the live /mnt/data/bootstrap-agent-8df05ff (9,895,207 bytes) — the binary currently running on this device was not built with -s -w. Whoever built it did not use this exact recipe. Rebuilding today with these flags produces a different-sized, behaviorally-identical binary; don't be alarmed by the size mismatch, and don't take the live binary's size as the flags to copy.

1.2 Verify the artefact¶

You cannot run an armv7l binary on a build laptop (exec format error is expected and fine) — check identity by string search instead, the same way CI's own verify-artefact step does:

grep -a -q "bootstrap-agent" /tmp/bootstrap-agent-${SHA} && \
grep -a -q "${SHA}"          /tmp/bootstrap-agent-${SHA} && \
echo "OK: carries its own name and commit ${SHA}"

What to do when it fails

If the second grep fails, the build argument never reached -ldflags — the binary silently reports version dev forever and cannot identify itself on a device. Check the -X path is exactly main.version (cmd/bootstrap-agent/main.go's var version = "dev").

1.3 Install (scp + SHA-256, both ends)¶

LOCAL_SHA256=$(shasum -a 256 /tmp/bootstrap-agent-${SHA} | awk '{print $1}')
scp /tmp/bootstrap-agent-${SHA} edge-gateway:/mnt/data/bootstrap-agent-${SHA}
ssh edge-gateway "sha256sum /mnt/data/bootstrap-agent-${SHA}"
# compare by eye against $LOCAL_SHA256 -- do not trust scp's own exit code alone

What to do when it fails — the scp truncation this week

scp exited 0 on a connection that dropped mid-transfer over a flaky link; the destination file was short and the process wouldn't even start (exec format error or an immediate crash-loop in journalctl). scp's exit code proves the local side finished writing, not that the remote file matches. The SHA-256 compare above is the actual check — never skip it because the copy "looked done", and never install straight to the path an active systemd unit's ExecStart already points at (see §1.5's real unit — a truncated overwrite of that exact file is a truncated write to what's running RIGHT NOW).

Install a new, distinctly-suffixed file first, verify, and only then repoint the unit — never scp over the file the running unit already uses.

1.4 Verified live¶

$ scp /tmp/bootstrap-agent-8df05ff edge-gateway:/tmp/bootstrap-agent-8df05ff-buildproof
$ ssh edge-gateway sha256sum /tmp/bootstrap-agent-8df05ff-buildproof
41c7d69aed6a20ec367fdb66366856682b04857274d82fdf871d3c4dac3f1c03  /tmp/bootstrap-agent-8df05ff-buildproof
$ shasum -a 256 /tmp/bootstrap-agent-8df05ff   # local
41c7d69aed6a20ec367fdb66366856682b04857274d82fdf871d3c4dac3f1c03

Match. Removed afterward (ssh edge-gateway rm /tmp/bootstrap-agent-8df05ff-buildproof); never installed into /mnt/data, never enabled, gateway state unaffected — confirmed by the state/profile_version values in Before you touch anything before and after.

1.5 The live unit is not the repo's unit¶

$ ssh edge-gateway systemctl cat bootstrap-agent.service
...
ExecStart=/mnt/data/bootstrap-agent-8df05ff
ExecStopPost=/mnt/data/bootstrap-agent-8df05ff --flush-tpm
Environment=BOOTSTRAP_AGENT_STATE_DIR=/etc/iot-config
...
DeviceAllow=/dev/tpmrm0 rw
SupplementaryGroups=tss

Two device-specific adaptations, both commented in the live unit file itself, neither in deploy/systemd/bootstrap-agent.service in the repo:

SupplementaryGroups=tss, not tpm. The repo's packaged unit assumes a group called tpm; this image's TPM device group is named tss. A unit installed verbatim from the repo starts, but the agent cannot open /dev/tpmrm0 — it fails closed (attestation errors), not loudly at install time.
No mount dependency. The repo's unit expects /var/lib/bootstrap-agent on a writable overlay it may need to wait for. On this image /etc is already a whole-filesystem overlay onto /mnt/data/etc, so /etc/iot-config (this device's actual state dir, also non-default — see Environment= above) persists with no extra .mount unit at all. Adding one here would be a dependency on a mount that never needs to exist, and would slow every boot waiting for it.

Before installing a new unit on any device

systemctl cat the live one first and diff it against the repo's. Assuming the repo's committed unit is what's running is the mistake that produces "works in the repo, fails on the device" every time.

1.6 Rollback¶

Binaries are never overwritten — every build lands at a new bootstrap-agent-<sha> path, so rollback is repointing ExecStart:

ssh edge-gateway 'sudo sed -i "s#ExecStart=.*#ExecStart=/mnt/data/bootstrap-agent-<previous-sha>#" \
    /etc/systemd/system/bootstrap-agent.service && \
  sudo sed -i "s#ExecStopPost=.*#ExecStopPost=/mnt/data/bootstrap-agent-<previous-sha> --flush-tpm#" \
    /etc/systemd/system/bootstrap-agent.service && \
  sudo systemctl daemon-reload && sudo systemctl restart bootstrap-agent.service'

What to do when it fails — the dentry cache 'surviving a delete'

If you rm the old binary before repointing ExecStart, ls /mnt/data shows it gone immediately, but the running process still has it open — Linux does not reclaim an unlinked file's inode, or stop executing it, until every process holding it exits. The service looks rolled back (the file is "gone") while the old code is still what's actually serving requests. Check readlink /proc/$(pgrep -f bootstrap-agent | head -1)/exe, not ls, if you need to know which build is really running — and always restart the unit (step above) rather than relying on a delete to end the old process.