Troubleshooting¶
Every known problem, in one place. Find your case below. On the gateway, the agent's log shows what it is doing:
The device is not in the list¶
Where: Devices in the console.
- It was never added. Add it, see Add device, or claim it, see Bulk import and claim.
- It was added in another organisation. Ask your colleagues, or search for the EUI (Ctrl+K, or ⌘K on a Mac).
CHAIN_INVALID despite a correct-looking upload¶
Where: Register your device CA.
A device's check-in fails with CHAIN_INVALID. Every one of these collapses
to the same wire error, so the order they're listed in is the order to check
them in:
- Wrong CA on the order. The uploaded device CA doesn't match the CA that actually issued this device's birth certificate — check the Subject/Issuer/Fingerprint shown on the order's Trust anchor panel against what the manufacturer actually gave you.
- CN/EUI mismatch. The birth certificate's CN doesn't match the device's own EUI — usually a device from the wrong batch, or a relabelled/duplicated unit.
- Wrong order. The device is registered under a different order than
the one whose trust anchor it actually chains to — per-order isolation
is enforced, so this is refused rather than silently admitted. Check
which order the device's
ak_fingerprintor EUI is actually assigned to; a duplicated/stray allowlist entry on the wrong order is the usual cause. - Revoked. The certificate is on the manufacturer's declared CRL — the device CA's own issuer publishes this, not the chip vendor, which is a separate trust chain entirely. Check CRL status on the Trust anchor panel — if it says "No CRL distribution point declared on this certificate", revocation isn't being checked for this CA at all (this deployment doesn't fail closed when no CRL is declared).
- Expired. The birth certificate (not just the CA) has passed its validity window.
Fix: confirm which of the five applies before re-uploading anything — re-uploading the same CA doesn't help if the actual cause is CN/EUI or order mismatch, or revocation. See How device identity works for what's actually checked.
TPM slots¶
Where: GET /v1/status (state, last_error_code, tpm_transient_avail,
tpm_transient_capacity, see Local API reference)
and journalctl -u bootstrap-agent.service.
The TPM has only a few slots for temporary objects (3 on our lab device). The
agent uses them for a check-in and frees them straight after. At start it logs
tpm transient capacity at agent start, and every 5 minutes tpm occupancy
with the free count. A check-in fails when no slot is free. There are two
different causes.
1. "No free slot" at start. state is error, last_error_code is
TPM_STUCK_POWER_CYCLE_REQUIRED, and the log says tpm: no free transient
object slots at agent start, before this process created anything. The slots
were taken before the agent did anything, usually by another program on the
image that opens the TPM during the same boot window (secure-boot
measurement, disk unseal, another attestation daemon). The agent flushes
nothing in this case.
Fix: power-cycle the device. A systemctl restart is not enough, and a
warm reboot did not clear it on our lab board. If the count does not recover
after a power cycle, audit everything on the image that touches /dev/tpmrm0
at boot.
2. Fewer free slots than the capacity while the agent runs: a leak on the
raw device. tpm_transient_avail is lower than tpm_transient_capacity,
and tpm occupancy shows the same number. We saw this once after the kernel
reported a bus timeout (tpm_try_transmit: send(): error -62): an object
stayed in the chip that only the raw device /dev/tpm0 shows, not the
resource manager /dev/tpmrm0. The agent's cleaning at start cannot heal
this. It removes only its own leftover key objects, only those visible
through /dev/tpmrm0 (the only TPM device its unit allows).
Fix, as root on the gateway:
Use the handle that the first command shows and the resource manager does not. On the lab device this brought the free count back to the full capacity. A restart of the device does not clear such a leak: with one free slot taken through the raw device, the free count was one lower before and after a reboot on the lab device. A power cycle was not measured.
Do not use /dev/tpm0 for your own diagnostics: an object created through the
raw device stays in the chip, one created through /dev/tpmrm0 is released
when its process ends (measured on the lab device, both ways).
NO_CONFIG¶
Where: the gateway's local interface, and the device in the console.
What you see: GET /v1/status answers "state":"no_config", and the
calls that read the configuration answer 404 with NO_CONFIG.
no_config is a normal state, not a fault. A gateway that nobody has claimed,
or that has not been set up yet, stays in it for as long as it takes, even
months, and your own software must wait without raising alarms. Only look for
a cause if the device should have its settings. Check, in order:
- Is the device in the console, and does it have a template? See
The device is not in the list and
TEMPLATE_INVALID. - What does the platform say?
last_error_codeinGET /v1/statusholds the platform's reason when it declined the device, for exampleDEVICE_UNKNOWN,CHAIN_INVALIDorTEMPLATE_INVALID. The console does not show a refused contact (see Known gaps and limits); support can look it up on the platform. - Can the gateway reach the platform?
BOOTSTRAP_AGENT_URLin/etc/bootstrap-agent/agent.envis set, andjournalctl -u bootstrap-agent.service -n 50shows no network or certificate error. - Was the state wiped on purpose, for example by a factory reset? Then
no_configis correct until the next setup finishes.
If state is error instead, the fault is on the gateway itself (TPM, storage,
a missing address), not at the platform: see the TPM entry above.
TEMPLATE_INVALID¶
Where: Assign template.
The platform refuses the device's setup or check-in because the settings it would receive are not valid. Causes you can see and fix in the console:
- The device has no template assigned at all.
- Its assigned template targets a generic platform with no credential configured on it.
- Its assigned template targets Pallax and the device has no Pallax
target, or the template still contains the old key
_target(see Pallax target missing).
You see the code in the agent's log and in last_error_code of
GET /v1/status. The console does not show the refused contact: after the
template was removed from a lab device, the refusal was only in the platform's
service log.
Fix: open the device's Configuration tab and assign a template —
or, if one is already assigned, open that template and add the missing
credential (see Certificate actions).
A device stuck in CLAIMED (see below) that never reaches
PROVISIONED is frequently this same cause, encountered at first
bootstrap rather than a later check-in.
Pallax target missing¶
What you see: the console marks the device with Template problem (and a
Pallax template that still carries _target with Target in template). The
platform refuses the check-in with TEMPLATE_INVALID; the device's detail page
shows the reason. The device keeps its old configuration, or answers
NO_CONFIG locally if it has none.
Cause: the Pallax workspace and the gateway type are set on each
device, not in the template. Either the device has neither, or the template
still contains the old key _target, which is refused.
Fix: open the device and set the Pallax workspace and gateway type (see
Set the target on the device); for many devices
use Set Pallax target in the Devices list. Then open the template and
remove the key _target, if it has one. The device picks up the change at
its next check-in; a device that is already provisioned needs
Re-provision for a changed target.
Device stuck in CLAIMED¶
Where: Assign template, Put the agent on the gateway.
CLAIMED only advances to PROVISIONED when the gateway completes its
first setup against the platform. A device that stays CLAIMED hasn't done
that yet. Check, in order:
- Does it have a template assigned? See
TEMPLATE_INVALIDabove. - Is the gateway powered on and online, and is
bootstrap-agent.servicerunning? Nothing on the platform side can advance a device that has never connected. - Was it refused as unknown? See
DEVICE_UNKNOWNbelow. - Is its trust chain valid? See
CHAIN_INVALIDabove. - Does the log show
reading trust bundle? The file named inBOOTSTRAP_AGENT_TRUST_BUNDLEdoes not exist. Leave the setting empty, or put the file support gave you in place, see Settings.
There is no "unstick" action on the platform: this is not a stuck record, it's a gateway that hasn't finished its setup.
Device stuck in PROVISIONED¶
Where: Start your software and confirm.
The gateway has its settings, but ACTIVE needs one more signal: the agent
reports confirmed only after your software has written
{"state":"connected"} to /run/wlpc/target-confirmed. Check, in order:
- Does the directory
/run/wlpcexist (root:wlpc, mode 2775), and can the agent's user read the file (0644)? Is your software's unit in the groupwlpc, and does it haveReadWritePaths=/run/wlpc? See Who may write/run/wlpc. - Does the file exist and say
connected? Nothing else writes it. - Is your software running, and does its own log show the broker refusing the connection? Then check the template's credential, see Create template.
The agent never reports a configuration that your software could not apply, see Known gaps and limits.
DEVICE_UNKNOWN¶
Where: journalctl -u bootstrap-agent.service on the gateway.
The platform knows no device with this gateway's AK fingerprint: the device
was never added, or its AK fingerprint or EUI was entered wrongly. The refused
attempt records the fingerprint, so support can look it up. Add or correct
the device (Add device,
Edit device), then
systemctl restart bootstrap-agent.service: after this refusal the agent
otherwise waits an hour before trying again.
Health FAILING or SILENT¶
Where: the HEALTH column in Devices and the badge on the device
page. Only devices that are PROVISIONED or ACTIVE and have checked in at
least once get a health badge.
FAILING: the gateway keeps contacting the platform but never finishes proving its identity (at least three attempts and no successful check-in for three check-in intervals, 3 hours by default). Usually a TPM problem on the gateway, see the TPM entry above. A warning only: the platform takes no action on the device. Do not reinstall or reset the gateway before support has looked at it; that can make the cause harder to find.SILENT: nothing at all has arrived for three check-in intervals (3 hours by default). Most common causes:- The gateway is powered off, or its network path to the platform is blocked (firewall, routing, DNS).
bootstrap-agent.serviceis not running.- Its template sets an unusually long
check_in_interval_seconds, see Re-provision.
Either badge returns to HEALTHY with the next successful check-in.
LAST CHECK-IN shows the time of the last successful check-in, in UTC.
If the gateway is on, online and bootstrap-agent.service runs, and the
status stays SILENT, contact support (see below).
The certificate warning ("Check this credential")¶
Where: Config templates, after uploading a credential.
The credential was stored, but it looks wrong for the template's broker: signed by a test CA, or made for another host. See Create template, When uploading the credential.
Contact support¶
If nothing here helps, contact support. Tell them the device's EUI, what you
see in the console, and the last 50 lines of the agent's log (the
journalctl command at the top of this page). Never send private keys or
passwords.