P2 CI Metrics
Know which red builds are real.
Every failure on hardware gets sorted: a real firmware bug, a bench or harness problem, or a flake. CI Metrics reports it per product, so you fix the right thing and trust the passes.
Welcome back, Alex
Acme Firmware · Admin
Jump into any part of your control plane.
Boxes
| Name | Groups | IP Address | Lager Version | Lock | Health | ||
|---|---|---|---|---|---|---|---|
| 100.96.12.41 | 0.49.0 | Online | |||||
Main bring-up bench. nRF5340 target on a J-Link, UART console on an FT232H, rails on the LabJack. Heartbeat 35 seconds ago · Stout Online · bench-01.local · linux/arm64 | |||||||
| 100.96.12.58 | 0.49.0 | Online | |||||
| 100.96.13.7 | 0.48.0 | Online | |||||
GPIO and rail checks for the sensor node. The LabJack moves between this bench and bench-04. Heartbeat 41 seconds ago · Stout Online · bench-03.local · linux/amd64 | |||||||
| 100.96.13.22 | 0.49.0 | Online | |||||
| 100.96.15.30 | 0.45.1 | Offline | |||||
| 100.96.13.61 | 0.47.1 | Degraded | |||||
| 100.96.14.9 | 0.49.0 | Degraded | |||||
CI Metrics
Merges exercised on real hardware, firmware bugs caught, and per-test flake rates for each product, reported by the connected CI pipeline.
Merges checked on hardware
67 / 69
97% of announced merges
Real FW bugs caught
3
2 bench or harness, 1 ambiguous
Flaky failures
16
Reds explained by a retry or same-commit rerun
Infra runs
5 / 259
Runs where the connectivity pre-check failed
Merges not checked on hardware
A merge counts as checked when this product's suite ran tests on that exact commit after passing the connection pre-check, or on a push that included it. In All products, a merge that touches shared code is counted once for each product it triggers. Two pull requests merged in separate pushes seconds apart can show the first as not checked even though its code ran in the second's run.
| Merged | Product | Repo | Branch | Commit |
|---|---|---|---|---|
| Sep 20, 2026, 02:14 PM | Gateway | fleet/firmware | main | d4e0b71826 |
| Sep 11, 2026, 04:52 PM | Sensor node | fleet/firmware | release/1.4 | 8f1a2c9ed0 |
Real FW bugs caught
Failures that were resolved by a newer commit, where the test harness blamed the device, the failing commit changed firmware, and the fix changed firmware and not the test harness.
| First failed | Product | Test | Box | Broke | Fixed by | Harness verdict | Status | Actions |
|---|---|---|---|---|---|---|---|---|
| 2026-09-19 | Gateway | power/brownout_recovery | bench-02 | a3f81c2761 Debounce brown-out detector IRQ | 9d02e4f553 Widen the brown-out debounce window | Device | Automatic | |
| 2026-09-12 | Gateway | boot/banner_timeout | bench-01 | 51b7aa00e5 Move the boot banner after clock init | 07adf1931b Print the banner before the PLL settles | Device | Automatic | |
| 2026-09-04 | Sensor node | radio/link_reconnect | bench-04 | 7d3e5b8b89 Retune the PLL after each channel hop | 90c1f3ae9d Wait for PLL lock before the first hop | Device | Automatic |
| First failed | Product | Test | Box | Commit | Why it is not counted | Status | Actions |
|---|---|---|---|---|---|---|---|
| 2026-09-17 | Gateway | io/gpio_loopback | bench-03 | c81d203718 | Harness verdict: bench or infrastructure | Automatic | |
| 2026-09-09 | Sensor node | sensor/imu_selftest | bench-04 | b62f0c491e | Fix touched harness paths | Automatic | |
| 2026-08-28 | Sensor node | power/sleep_current | bench-01 | 6a8b0e5c6e | AmbiguousCommit under test did not touch firmware | Automatic |
Flakiest tests
| Product | Test | Boxes | Runs | Attempts | Failed | Pass on retry | Flake rate |
|---|---|---|---|---|---|---|---|
| Gateway | boot/banner_timeout | 2 | 58 | 66 | 11 | 8 | 16.7% |
| Gateway | io/gpio_loopback | 1 | 58 | 61 | 4 | 3 | 6.6% |
| Sensor node | radio/link_reconnect | 2 | 40 | 44 | 5 | 4 | 11.4% |
| Sensor node | sensor/imu_selftest | 1 | 40 | 41 | 2 | 1 | 4.9% |
PR Reviewers
AI-generated hardware tests on every pull request. Configure one reviewer per repo, or several scoped by path prefix for a monorepo.
| Repository | PR | Plan | Status | Waiting | |
|---|---|---|---|---|---|
| fleet/firmware | #482 | Inject a 2 ms sag and a 40 us glitch on vbat-sense and expect exactly one latched brown-out event. | Needs approval | 18 minutes ago |
| Repository | Status | AI provider | Rate limit | Approval | Last run | Awaiting | |
|---|---|---|---|---|---|---|---|
| fleet/firmware | Enabled | Anthropic · claude-opus-5-5 | 10/hr | Required | Needs approval18 minutes ago | 1 | |
| fleet/bootloader | Enabled | Anthropic · claude-sonnet-5 | 20/hr | Auto | Waiting for buildjust now | 0 | |
| sensors/driver | Enabled | OpenAI-compatible · gpt-4.1 | 5/hr | Required | Skipped4 days ago | 0 | |
| power/bms | Enabled | Anthropic · claude-sonnet-5 | 15/hr | Auto | Generatingjust now | 0 | |
| radio/phy | Enabled | Anthropic · claude-opus-5-5 | 8/hr | Required | Passed5 hours ago | 0 | |
| display/panel | Disabled | Anthropic · claude-haiku-4-5-20251001 | 30/hr | Auto | Passed5 days ago | 0 |
Workbench
Drive a box's instruments directly from the dashboard.
Range 0 to 30 V
Range 0 to 3 A
- 0usb1ON
- 1usb2ON
- 2usb3ON
- 3usb4ON
- 4usb5ON
- 5usb6ON
- 6usb7ON
- 7usb8ON
Target MCU: nRF5340_xxAA_APP
Firmware file (.hex, .elf, .bin)
Connect to start the serial console.- AIN0adc1vbat sense--
- AIN1adc23V3 rail--
- AIN2adc31V8 rail--
- DAC0dac1Sensor stimulus
- DIO4boot-strapBoot mode strapLOW
- DIO5reset-nLOW
Tests from the connected repository, at the branch or commit you pick.
Import Testtest_brownout_recovery
tests/power
test_banner_timeout
tests/boot
test_gpio_loopback
tests/io
Run a test to see its output here.
Factory
Automated hardware test suites
P2.1
Merges checked on hardware
For each product, see how many merged commits actually ran on a box, and a list of the ones that did not.
P2.2
Real firmware bugs, counted
A failure counts as a firmware bug when a newer commit fixed it, the harness blamed the device, the failing commit changed firmware, and the fix changed firmware and not the harness.
P2.3
Bench and harness problems, set aside
Failures that a bench or harness change resolved are listed separately with the reason, so they never inflate your bug count.
P2.4
Flaky tests, named
See the tests that pass on retry, with runs, attempts, failures, and a flake rate for each one.
P2.5
Infra runs, separated
Runs where the connectivity pre-check failed are counted on their own, so a bench that dropped off the network is not a test result.
P2.6
Your call, on the record
An admin can confirm or dismiss any row. The status column shows who decided, and Reset returns the row to the automatic label.
Related capability
03.3Workflows & CINext feature
P3PR ReviewersSee your own CI numbers.
Book a demo and we will connect a pipeline, run a few pull requests, and read the metrics together.