Compare commits

...
Author SHA1 Message Date
J. Nick Koston d8ea45b60d [hardware-test] Judge spclient reply data in its exit status and fail an15.py on device error lines 2026-11-17 03:32:00 -06:00
J. Nick Koston a887cae681 [hardware-test] Pass leak.py only when the wrong-hash load and the /events stream really ran 2026-11-17 02:59:13 -06:00
J. Nick Koston 51acc65f1b [hardware-test] Classify upload_reject responses and loosen three checks that failed healthy saved runs 2026-11-17 02:31:40 -06:00
J. Nick Koston 2a82482812 [hardware-test] Check every capture and driver verdict in the network runners and put crashes in the exit status 2026-11-17 01:58:16 -06:00
J. Nick Koston ee385f03e3 [hardware-test] Fail the async_bump runner on an empty capture or one without a boot 2026-11-17 01:21:51 -06:00
J. Nick Koston 264de7e8d8 [hardware-test] Fail an15.py on missing captures, frames, station or reboots 2026-11-17 00:49:52 -06:00
J. Nick Koston 6661cc49aa [hardware-test] Exit non-zero from improvscan.py on malformed scan replies 2026-11-17 00:18:05 -06:00
J. Nick Koston f989231b19 [hardware-test] Mask volatile values in the log macro normaliser instead of dropping whole lines 2026-11-16 23:47:06 -06:00
J. Nick Koston ad3172acf1 [hardware-test] Exit non-zero from otacases.py when a case fails 2026-11-16 23:13:22 -06:00
J. Nick Koston 0943e2fdfb [hardware-test] Count refused overflow enqueues and exit non-zero on failed verdicts in the network runners 2026-11-16 22:45:05 -06:00
J. Nick Koston 7338b5aa9a [hardware-test] Exit non-zero from the network runners on failed flows, setup and waits 2026-11-16 22:14:15 -06:00
J. Nick Koston fbdd579790 [hardware-test] Return webota.sh's status and capture LibreTiny kits with RTS released 2026-11-16 21:47:59 -06:00
J. Nick Koston f9b92b3e54 [hardware-test] Fail the network runners on missing counts, dead readers and failed setup calls 2026-11-16 21:43:27 -06:00
J. Nick Koston c10524ebb3 [hardware-test] Share web, auth and REST helpers in the network flows and fix the first review round
- The web_auth scripts use authfuzz.py (sensor reads, the wrong-hash value, the multipart body,
  one bad_once with a bounded drain); async_bump gets rest.py and ltflash.sh; ota8266.yaml
  builds on aws-common.yaml; runners upload through pio_build.sh.
- improvscan.py and openap32.yaml move under improv/ and reuse improvcli.py.
- serial_proxy polls the simulator at the normal loop cadence; wpa3_sniff gets one station base
  and reboots from on_loop after a loop wake instead of a 1 ms interval; sniffap.yaml replaces
  w15ap.yaml; outage.py scans only new lines and ends on the cooldown end.
- gen_sensors.py writes all flooded and remote entities of the API overflow config.
- Review fixes: atomic sniffer cursor, VERBOSE log macro A/B, runners stop on failed uploads
  and flashes, a 401 without a close no longer counts as TIME_WAIT load, rate.py counts resets
  on send, presstest.sh and webota.sh need a real reset, an15.py splits groups per reboot.
2026-11-16 21:37:33 -06:00
J. Nick Koston 6857122fd5 Add Wi-Fi outage, DNS loss, web server, API overflow and serial_proxy flows to the hardware-test skill 2026-11-16 21:12:08 -06:00
71 changed files with 4552 additions and 69 deletions

No files matched your search

+3 -2
View File
@@ -158,7 +158,7 @@ recorder, dump saver and decoder, cover cycle driver), `online_image/`
(host driver, Zephyr driver probe), `rtsp_server/` (protocol client, raw rate reader, measurement set,
API image grab), `cse7766/` (API observer with ping), `zephyr_coredump/` (reader flow, decode with the
Zephyr tools), `libretiny_prefs/` (rtc preference harness header), `libretiny_crash/` (BK72xx crash probe header,
capture decoder, RTL8720C ROM dump versus report matcher), `api_outgoing/` (wake cycle and Wi-Fi drop timing readers), `ld2450/` (zone number driver, loopback harness driver with number and binary sensor states and TX frame summarizer, frame stream harness with presence hold and link loss drivers, BT MAC driver), `api/` (raw plaintext native API client for user defined actions, store_yaml recovery client over Noise or plaintext), `http_request/` (update manifest server, update state driver, request test server, action case driver, concurrency, reboot, update check and OTA drivers, `ota_auth/` OTA request cases with basic auth), `modem/` (button and sensor driver over the plaintext API, also used for the BLE MIDI boards, AT commands for the UART passthrough), `dfrobot_sen0395/` (API driver and case runner), `homeassistant_light/` (direct bulb driver, held-report case driver), `usb_audio/` (URL player), `speaker_media_player/` (API player driver, pipeline task case runner), `web_server/` (SSE client, request mix, SSE and HTTPD case runners, per-session `/events` listing check with its config, `/events` head, multi-client, stall, Wi-Fi drop and churn modes with their config, head comparer and runner), `mqtt/` (`ha_discovery/`, `disable/`, `retained_burst/` and `reconnect/` runners; broker config with TLS, client-certificate and WebSocket listeners, test CA with server and client certificates, transport case runner, flash-and-drive runner, CLI TLS check without a board, MAC suffix runner with per-variant configs for two boards, `shutdown/` shutdown message runner and matcher), `udp/` (datagram recorder, `udp.write` payload checker and case driver), `entity_icons/` (icon and device class readback over the API, MQTT discovery, web_server and dump_config, with a comparer), `uart_sim/` (API driver and flash, capture and drive runner for the simulated UART sensors), `remote_base/` (loopback runner and API log decoder per sent frame), `xiaomi_ble/` (frame and header generator with the expected counts, injecting, advertising and receiving configs, counter and runner), `ble_terminate/` (proxy and peer configs, connect and disconnect driver, peer disconnect matcher, runner), `i2c_slave/` (D1 mini slave and BK7238 master configs), `wpa3_ap/` (PMF AP with join and leave logging, BK7238 and CB3S DUTs, runner and summary), `wisp_ap/` (WISP upstream with AP buttons, WISP and ALWAYS_ON DUTs, client, runner and per phase summary), `wifi_restart/` (WPA3 mixed WISP upstream, static IP DUTs with syslog, runner and summary), `ota_timing/` (OTA upload timer with deflate, raw and TCP modes, API build check, A/B runner and median summary, `inflate_bench/` on-device inflate benchmark with stream generator, configs and syslog runner), `gpio_hold/` (ISR pin hold self test with ESP32 and S3 configs), `ble_event_inject/` (GAP and GATTC event injector header and proxy package), `light_sleep_wake/` (LEDC wave and edge timer for light sleep wakeup on a C3, external wave generator and per-call edge detail probe for peripheral power-down on the P4), `manual_ip/` (station and AP manual IP configs, DNS probe and runners), `web_handlers/` (late handler, web_server, captive_portal and prometheus variants, web OTA runner), `bk72xx_flash/` (flash readback with retries, per partition diff of two dumps), `ble_server_security/` (secure GATT server and pairing client configs, driver, summary and runner), `prometheus/` (datetime metrics configs and driver), `zephyr_ble_server/` (nRF52 server, looping ESP32 client and UF2 flash and count runner), `ble_adv_triggers/` (advertiser with six raw frames, ESP32 and BK72xx receivers with eleven triggers, runner and checker), `sun/` (longitude sweep package, runner and checker), `sensor_filters/` (every filter shape with a fixed input sequence, per-board packages and runner), `led_driver_sniff/` (SPI slave capture header with selftest, config generator, window comparer and runner), `hub_input_sim/` (keypad, wiegand and rdm6300 simulation on a D1 mini, API log runner), `value_range/` (on_value_range sequence with a restart, runner), `ezo_fake_bus/` (fake EZO I2C bus and step config, runner), `twai_self/` (TWAI self-reception header, on_frame config and counter runner), `ota_automations/` (OTA automation package for both OTA platforms, four case runner, repeated web upload runner and per case summary), `output_button/` (pad edge timer header, button package and runner), `homeassistant_button/` (raw plaintext API client for action calls and wizard inputs, package and runner), `uart_button/` (UART0 loopback package, API driver with TX and RX counts, runner), `interval/` (plain and delayed interval package, timing summary and runner), `smp_udp_ota/` (SMP reset fault injector, OTA runner, per mode result check, MCUboot swap limit, running image check and slot blanking UF2, Thread device config), `ld2420_sim/` (fake LD2420 answering the hub's commands on a UART0 loopback, API state recorder, two-build comparer and runneLine truncated
capture decoder, RTL8720C ROM dump versus report matcher), `api_outgoing/` (wake cycle and Wi-Fi drop timing readers), `ld2450/` (zone number driver, loopback harness driver with number and binary sensor states and TX frame summarizer, frame stream harness with presence hold and link loss drivers, BT MAC driver), `api/` (raw plaintext native API client for user defined actions, store_yaml recovery client over Noise or plaintext, `overflow/` API overflow buffer counters with a sensor config generator), `http_request/` (update manifest server, update state driver, request test server, action case driver, concurrency, reboot, update check and OTA drivers, `ota_auth/` OTA request cases with basic auth), `modem/` (button and sensor driver over the plaintext API, also used for the BLE MIDI boards, AT commands for the UART passthrough), `dfrobot_sen0395/` (API driver and case runner), `homeassistant_light/` (direct bulb driver, held-report case driver), `usb_audio/` (URL player), `speaker_media_player/` (API player driver, pipeline task case runner), `web_server/` (SSE client, request mix, SSE and HTTPD case runners, per-session `/events` listing check with its config, `/events` head, multi-client, stall, Wi-Fi drop and churn modes with their config, head comparer and runner, `async_bump/` ESPAsyncWebServer bump flows and ESP8266 web OTA cases), `mqtt/` (`ha_discovery/`, `disable/`, `retained_burst/` and `reconnect/` runners; broker config with TLS, client-certificate and WebSocket listeners, test CA with server and client certificates, transport case runner, flash-and-drive runner, CLI TLS check without a board, MAC suffix runner with per-variant configs for two boards, `shutdown/` shutdown message runner and matcher), `udp/` (datagram recorder, `udp.write` payload checker and case driver), `entity_icons/` (icon and device class readback over the API, MQTT discovery, web_server and dump_config, with a comparer), `uart_sim/` (API driver and flash, capture and drive runner for the simulated UART sensors), `remote_base/` (loopback runner and API log decoder per sent frame), `xiaomi_ble/` (frame and header generator with the expected counts, injecting, advertising and receiving configs, counter and runner), `ble_terminate/` (proxy and peer configs, connect and disconnect driver, peer disconnect matcher, runner), `i2c_slave/` (D1 mini slave and BK7238 master configs), `wpa3_ap/` (PMF AP with join and leave logging, BK7238 and CB3S DUTs, runner and summary), `wisp_ap/` (WISP upstream with AP buttons, WISP and ALWAYS_ON DUTs, client, runner and per phase summary), `wifi_restart/` (WPA3 mixed WISP upstream, static IP DUTs with syslog, runner and summary), `ota_timing/` (OTA upload timer with deflate, raw and TCP modes, API build check, A/B runner and median summary, `inflate_bench/` on-device inflate benchmark with stream generator, configs and syslog runner), `gpio_hold/` (ISR pin hold self test with ESP32 and S3 configs), `ble_event_inject/` (GAP and GATTC event injector header and proxy package), `light_sleep_wake/` (LEDC wave and edge timer for light sleep wakeup on a C3, external wave generator and per-call edge detail probe for peripheral power-down on the P4), `manual_ip/` (station and AP manual IP configs, DNS probe and runners), `web_handlers/` (late handler, web_server, captive_portal and prometheus variants, web OTA runner), `bk72xx_flash/` (flash readback with retries, per partition diff of two dumps), `ble_server_security/` (secure GATT server and pairing client configs, driver, summary and runner), `prometheus/` (datetime metrics configs and driver), `zephyr_ble_server/` (nRF52 server, looping ESP32 client and UF2 flash and count runner), `ble_adv_triggers/` (advertiser with six raw frames, ESP32 and BK72xx receivers with eleven triggers, runner and checker), `sun/` (longitude sweep package, runner and checker), `sensor_filters/` (every filter shape with a fixed input sequence, per-board packages and runner), `led_driver_sniff/` (SPI slave capture header with selftest, config generator, window comparer and runner), `hub_input_sim/` (keypad, wiegand and rdm6300 simulation on a D1 mini, API log runner), `value_range/` (on_value_range sequence with a restart, runner), `ezo_fake_bus/` (fake EZO I2C bus and step config, runner), `twai_self/` (TWAI self-reception header, on_frame config and counter runner), `ota_automations/` (OTA automation package for both OTA platforms, four case runner, repeated web upload runner and per case summary), `output_button/` (pad edge timer header, button package and runner), `homeassistant_button/` (raw plaintext API client for action calls and wizard inputs, package and runner), `uart_button/` (UART0 loopback package, API driver with TX and RX counts, runner), `interval/` (plain and delayed interval package, timing summary and runner), `smp_udp_ota/` (SMP reset fault injector, OTA runner, per mode result check, MCUboot swap limit, running image check and slot blanking UF2, ThreaLine truncated
## Templates in `$SKILL/templates`
@@ -185,7 +185,8 @@ capture decoder, RTL8720C ROM dump versus report matcher), `api_outgoing/` (wake
action package with per-board files and an OTA flash variant), `led_wire/` (DUT config and header that reads the
LED strip pads back with RMT RX and an SPI slave), `uart_sim/` (simulator header, common package, per-build parts and run files) and `remote_base/` (common
package, transmit script, per-board templates and the ESP8266 software sender), `i2c_fake/` (fake I2C device
headers with their test configs, see `fake_i2c.md`).
headers with their test configs, see `fake_i2c.md`), `network_dns/` (DNS loss diagnostics for
`network: priority`, see `network_dns.md`).
- `microphone_http_stream/`: base config, fake counter and USB microphone variants and the multi-reader client,
described in `reference/recipes/microphone_http_stream.md`.
@@ -5,13 +5,15 @@ Ports are not listed on purpose: they follow the USB hub port and change on ever
## ESP8266 (D1 mini, NodeMCU, CH340 or CP2102)
- Opening the port usually resets the board through DTR, but not always; `serial_capture.py --reset`
pulses RTS to capture from boot. 74880 baud shows the ROM boot messages (`rst cause`, `boot mode`).
- Opening the port usually resets the board through DTR, but not always, and on a CH340 even with `--released`, so
the first boot (or `Connection lost`) in a capture can be that reset; `serial_capture.py --reset` pulses RTS to
capture from boot. 74880 baud shows the ROM boot messages (`rst cause`, `boot mode`).
- UART0 can be looped back onto itself (`USC0(0) |= (1 << UCLBE)` in an `on_boot` lambda) so a UART
component is testable without its device: frames written with `uart.write` arrive on RX. Log over the
API with `logger: baud_rate: 0`, since the serial port carries the component's bytes.
- `esphome logs` on some CH340s emits thousands of duplicated garbage lines right after the RTS reset;
a plain pyserial capture at 115200 is reliable.
a plain pyserial capture at 115200 is reliable. A CH340 can also give garbage at 921600 baud;
`logger: baud_rate: 460800` with `serial_capture.py --baud 460800` carries long `dump: raw` frames cleanly.
- A failed esptool write can wedge the adapter (`termios` error 22, or an open that hangs) until replug.
- The preference sector survives a reflash: a provisioned API key or saved Wi-Fi credentials stay
active. `esptool erase_flash` gives a clean device.
@@ -32,6 +34,10 @@ Ports are not listed on purpose: they follow the USB hub port and change on ever
- Cover positions and other restored entity state live in RTC memory by default, so they survive `erase_flash`
and a dev build's state carries into the next build's first boot. Set `esp8266: restore_from_flash: true` on A/B
configs that erase between builds.
- The ROM `rijndaelKeySetupEnc` at 0x40009300 builds a correct 128-bit schedule (FIPS-197 A.1 words match) but does not
return the round count, so the SDK's `aes_encrypt` loops on garbage rounds and the board Soft WDT resets in
`rijndaelEncrypt`. Do not `--defsym` it over the libcrypto copy; a wrapper in the SDK signature (returns -1 unless
128 bits, calls the ROM, returns 10) passed the FIPS vectors and a WPA2 soft AP join.
## ESP32 classic devkit
@@ -90,6 +96,8 @@ Ports are not listed on purpose: they follow the USB hub port and change on ever
to UART0 instead (see `recipes/light_sleep_gpio_wake.md`).
- The Waveshare ESP32-P4-WIFI6 needs `engineering_sample: true`. Its Wi-Fi is an ESP32-C6 over SDIO through
`esp32_hosted`; a test with no network logs over UART0 to the CH343 console.
- ESP32-S3: GPIO43/44 cannot be used to cross-wire two UARTs: the UART0 IOMUX TX is lost when another UART takes the
pad as RX. TX17/RX16 and TX16/RX17 work. A task woken by `vTaskDelay` runs about 0.5 ms after the tick, on dev too.
## ESP32-C2 (ESP8684, 26 MHz crystal)
@@ -142,6 +150,8 @@ Ports are not listed on purpose: they follow the USB hub port and change on ever
`--keep-dtr`) sees silence.
- Powering a breakout from the Pico: use 3V3(OUT), pin 36. Pin 37 is 3V3_EN, which switches the Pico off when pulled
low.
- W5100S-EVB-Pico with its USB attached shows a CDC port (`2e8a:1027`) while it runs ESPHome, so `esphome upload
--device <that port>` (picotool) flashes it with or without `interrupt_pin`.
## BW15 kit (RTL8720CF, LibreTiny rtl87xx)
@@ -189,6 +199,8 @@ Ports are not listed on purpose: they follow the USB hub port and change on ever
- A config with no Wi-Fi network and no `ap:` resets about 10 s into setup when it has `web_server:` (dev too):
`web_server` setup blocks with no interface up, and no crash record is left. VERBOSE logs show the last
`Setup <component>` line.
- A BW15 kit whose CH340 drops off the bus at "Initializing WIFI" on every boot (a boot loop on any image) is short of
USB power: use another hub port or a powered hub. Read the results over API logs between resets meanwhile.
## BK7238, WB3S, CB3S (BK7231N) (LibreTiny bk72xx, CH340, RTS on CEN)
@@ -240,6 +252,9 @@ Ports are not listed on purpose: they follow the USB hub port and change on ever
... !=`) and the old image stays.
- Power save: on the BK7238, `power_save_mode: light` with LibreTiny 1.14.0 gives ping times of 3 to 6 s because
the BDK sets a listen interval of 30 beacons when `CFG_LOW_VOLTAGE_PS` is 1 (esphome#19491).
- LibreTiny's builder adds `-D'delay(ms)=delayMilliseconds(ms)'`, so `esphome::delay` is really
`esphome::delayMilliseconds` on BK72xx and a YAML lambda calling `delay()` fails to compile on dev. In a header use
`(esphome::micros)()` to dodge the `main.cpp` function-like macros.
## BW12 and other AmebaZ boards (RTL8710BX, LibreTiny rtl87xx)
@@ -523,3 +538,15 @@ Ports are not listed on purpose: they follow the USB hub port and change on ever
- ADS1015: `sample_rate: 8` maps to 128 SPS at `12_BITS`.
- A bus held low (`Performing bus recovery`, no devices found) on a new breakout can be a bad header joint; reflow
it before blaming the driver.
## Harness tricks and old releases
- Protected members and methods can be reached from a test header listed in `esphome: includes:` with the explicit
instantiation access trick (`template struct Rob<Tag, &C::member>;`), even on `final` classes; declare the robbers at
global scope.
- API driven light effects that write template outputs every loop (about 5k writes/s) can lag API commands by seconds:
wait for the API state to show each command before reading results.
- Building ESPHome 2025.10.x (pioarduino ESP-IDF 5.4.2) from a path under `/tmp` on macOS fails with "Multiple ways to
build the same target" (the /tmp to /private/tmp alias); put the yaml, build and PLATFORMIO_CORE_DIR elsewhere. Its
IDF penv under Python 3.14 lacks the tools: install `-r packages/framework-espidf/tools/requirements/requirements.core.txt`,
then pin `idf-component-manager~=2.1.0 esp-idf-kconfig~=2.5.0` (newer ones reject interface_version 3).
@@ -87,3 +87,5 @@ One file per component, named after it:
- [`improv.md`](improv.md): Improv serial with stray bytes, scan, bad and good passwords from a client that reads frames out of the log stream, Improv over BLE, per board notes
- [`arduino_pico.md`](arduino_pico.md): arduino-pico framework bump A/B: W5100S Ethernet soak, Pico W fallback AP client count, Pico W bluetooth_proxy GATT hold, web_server SSE stalls, `/events` churn crashes and never completed connects, W5100S OTA without `interrupt_pin`
- [`bluedroid_gattc_queue.md`](bluedroid_gattc_queue.md): queued MTU request and execute write recursion that overflows BTU_TASK, ble_client central against an ESP32-S3 peer
- [`serial_proxy.md`](serial_proxy.md): simulated serial device on a cross-wired UART, byte-exact client with read pauses, ASH tap ACK timing
- [`network_dns.md`](network_dns.md): network.priority Ethernet plus Wi-Fi losing lwIP DNS servers (#19805)
@@ -78,3 +78,24 @@ the client only sees resets, the crash is in the log.
closed PCB's callbacks); after: 490/500, the 10 failures are connection resets from overload, which is expected there.
- ESP32 (BSD sockets) passes 500/500 on both and is the control.
- Compare against dev on the same board; count reboots and decode crash addresses as well as the client's failures.
## API overflow buffer driven and counted on the device (#22771)
`scripts/api/overflow/`: `ovfwrap.hpp` wraps `APIOverflowBuffer::enqueue_iov` and `try_drain` with `-Wl,--wrap` and
counts calls and bytes; the `try_drain` symbol differs per platform (`...6socket11LWIPRawImplE` on ESP8266,
`...6socket13BSDSocketImplE` on ESP32 IDF), so pass it as a substitution and check `nm` shows both `__wrap_` symbols
(a wrong name silently drops the wrapper). ESP-IDF builds keep `nm` in the IDF tools cache
(`xtensa-esp-elf/*/xtensa-esp-elf/bin/`), not under the PlatformIO directory. `putbe-common.yaml` (60 sensors from `putbe-sensors.yaml`, which `gen_sensors.py` writes; four
800 char text sensors flooded every 50 ms, counters as sensors, plus two udp `entity_transport` links) and
`apitest.py HOST OUT` (2 KB receive buffer, 3 x 6 s `pause_reading`, validates every burst state and the final states;
`API_KEY` in the env). Result on #22771: 82 to 159 enqueues per run on a D1 mini and an ESP32, no drops.
`scripts/entity_transport/trcheck.py` compares remote entity values with the sender's published ones over the API on
both boards; it accepts late updates and checks floats by the formula the sender uses, so only values the sender never
published fail. Declare remote entities for both the base and the PR name of the peer so one image serves all four
pairings. Facts from that run:
- Xtensa GCC (lx106 and esp32) defines `__CHAR_UNSIGNED__`; macOS host builds use signed `char`, so a host harness
can miss a sign bug in byte handling.
- udp `packet_interface` with broadcast between a D1 mini and an ESP32 loses most hdlc acks; unicast `addresses:` still
resets the reliability session a few times a minute (dev and PR alike). A D1 mini's API can deliver states 5 to 11 s
late under that load.
@@ -44,3 +44,23 @@ When posting results:
- Mask IP addresses in the result URLs.
- Cover a long password too: provision once against a network whose password is 20 characters or more (the client
prints the length).
## Improv serial scan reply over the logger UART (#22758)
- `scripts/improv/improvscan.py PORT OUT [--settle S]` (on `improvcli.py`'s frame parser) sends an Improv RPC request
scan (command 0x04) and records each reply frame (checksum, strings, auth field bytes and length) until the
end-of-list reply. Opening the port resets a D1 mini and a classic ESP32 (see `reference/boards.md`), so it waits
30 s before sending.
- A connected device frees its scan list, so the reply is just the end-of-list frame. Add a `wifi_info:`
`scan_results:` text sensor to keep the list.
- For an auth `NO` entry you need an open network in range: run an AP-only build without a password on a second board
(`scripts/improv/openap32.yaml`).
- Scan replies list the neighbours' SSIDs; filter them out of anything you post.
## Log macro rewrites, A/B by log text (#22758)
`scripts/log_macros/`: `lm-common.yaml` (one entity of every common platform plus `improv_serial` and `web_server`),
`drive.py HOST OUT` (presses the exercise button, reads API sensor states and web_server JSON) and `norm.py SERIAL`
(keeps config dump lines, `lm` lines and state lines, and replaces addresses, the device name, signal, setup times
and per-boot values with placeholders instead of dropping the lines) so two builds' captures diff cleanly. Recent ESPHome logs sensor states and switch, light and fan calls only at VERBOSE; at
DEBUG a log diff sees none of them, so `lm-common.yaml` logs at VERBOSE.
@@ -31,3 +31,49 @@ replies, to check what the module does with no component in the way.
- `AT+CFUN=1,1` leaves the module silent for about 2.5 minutes. It comes back without a power cycle.
- A module left in CMUX (`AT+CMUX=0`, then an ESP restart) recovered 10 of 10 times (`Autodetect mode: 5`).
- PPP and a warm reboot during PPP need a SIM with a working data plan.
### SIM7080G with a live SIM
- A SIM whose plan does not cover LTE-M or NB-IoT never registers on the SIM7080G: CEREG goes 0 (not searching) right
after CFUN=1, 2 (searching) for about 60 s, then 3 (denied) for a minute or two, then searching again, with CSQ 19 to
31. COPS=? lists the Cat-M networks; manual COPS fails (ERROR or denied), so don't retry it. PPP, attach timing and a
warm reboot during PPP need a SIM whose plan covers LTE-M.
- esp_modem 2.1.0 sends `AT+CGATT=1`, `AT+CGATT?` and `AT+CEREG?` with a 500 ms timeout. Unregistered, `CGATT=1`
holds the AT channel (15.6 s denied in the passthrough, about 40 s in the component, over 2 minutes with a SIM that
has no data plan) and the module drops every command sent meanwhile: no echo, no reply (an `AT+CMUX=0` sent then is
lost).
- `ATD*99#` sometimes answers `CONNECT 150000000` without an attach. The module then sits in data mode, sends no LCP,
ignores AT for minutes, and esp_modem's autodetect returns 0 forever (dev logs the wiring hint). A guarded `+++`
(1.1 s silence, `+++`, 1.1 s) returns OK at once; give the passthrough an `Escape` template button doing
`write_str("+++")` between two 1200 ms delays.
- `AT+CFUN=0` powers the SIM down only about 1.7 s after OK (`+CPIN: NOT READY` URC), and the component reads
`+CPIN: READY` before that, so `init_at: [AT+CFUN=0]` cannot fake a missing SIM. There is no reliable no-SIM path
without removing the SIM.
- `AT+CFUN=1,1` left the module silent for 20 s to 2.5 min.
- Pins that reach nothing, for the no-modem path on an M5Stack Atom (or any ESP32): `rx_pin: GPIO22`, `tx_pin: GPIO19`.
- Drive the component build with send_at buttons (`AT DIAL` = `ATD*99#`, `AT CFUN1`, `AT+CMGR=9` for an error text)
and the `Restart` button; `mdrive.py press:NAME`. A `send_at` needs about 3 s before a restart to go out.
## SIM7670G on the Waveshare ESP32-S3-SIM7670G-4G (#22581)
- Board: ESP32-S3 console on a CH343 (UART0, so `logger: hardware_uart: UART0`; the default USB logger prints
nothing). Modem UART: ESP TX GPIO18, ESP RX GPIO17, 115200, `model: SIM7670`. The modem is powered by the "4G" DIP
switch, no GPIO power-up needed. Module SIM7670G-MNGV, firmware V1.9.05.
- The module's own USB enumerates as four `05c6:9330` ports; one of them is an AT port that works next to the
component's UART (another is a binary diag stream). Send AT lines there to read CEREG, CGDCONT, CEER or SMS while the
ESP runs. Run AT experiments with an idle ESP image (a hex passthrough), or the component's own reconnects (CFUN=4,
CGDCONT) get mixed into the results.
- Hex passthrough with frame buttons: `uart: debug: after: timeout: 40ms` with `log_hex`, buttons that `uart.write`
SABM/DISC/CLD/AT frames per DLCI (FCS: CRC-8 reversed poly 0xE0 over address, control, length, then 0xFF minus it)
and buttons that write runs of 0xF9.
- Findings: echo is on after power-up or an AT channel reset, and esp_modem's mode guess takes the echoed LCP probe for
a PPP reply (`Autodetect mode: 2`), the start of #22581. After an ESP reset during PPP the module stays in CMUX and
ignores SABM/DISC/CLD/AT/+++ (it answers SABM0 with a UA only now and then); a burst of 4 KB of 0xF9 usually makes it
print `QCRDY` and fall back to command mode with echo on (worked 10 times, failed 4 tries once after a reflash).
- A PDP context written at CFUN=4 makes the next attach fail with `+CEER: EMM_CAUSE_ESM_FAILURE` (CEREG 3) until a
CFUN=0 cycle; written at CFUN=0 it attaches in about 8 s. After each attach the network can rewrite context 1's APN to
its own name. An empty `apn: ""` validates and attaches (the network picks the APN).
- A SIM can attach and get an IP and DNS while no TCP or ICMP traffic comes back, through PPP (CMUX and data mode) and
through the module's own `AT+HTTPACTION` (706) and `AT+CPING`: that is the carrier side, not the component.
- SMS: when the SIM's SM store is full, set the receive store to ME with `AT+CPMS="SM","SM","ME"`. Messages are UCS2
hex in text mode; decode in memory only, never into a log, and keep SIM data (numbers, identifiers) out of results.
@@ -0,0 +1,33 @@
# network.priority DNS loss (#19805, espressif/esp-idf#19082)
With two interfaces in `network: priority:` on ESP32, ESPHome sets CONFIG_ESP_NETIF_SET_DNS_PER_DEFAULT_NETIF and pins
the default with `esp_netif_set_default_netif()`. In that mode a DHCP ACK only fills the netif's own DNS cache; lwIP's
global table is written only by `esp_netif_set_default_netif_internal()`. `esp_netif_dhcpc_start()` and
`esp_netif_set_ip_info()` (DHCP client stopped) on ANY netif call `dns_clear_servers(true)` on the global table, and the
pin makes IDF skip re-election, so the table stays empty until the app re-asserts the default. Lookups then fail at once
with getaddrinfo 202 (lwIP ERR_VAL, no server set). Unfixed on IDF master when this was written.
Triggers: Wi-Fi joining or reconnecting while Ethernet is the default (ESPHome's `wifi_sta_ip_config_` and IDF's
`esp_netif_action_connected` start DHCP); Ethernet with no link while Wi-Fi is the default (the 15 s
`Connecting failed; reconnecting` retry runs `set_ip_info` plus `dhcpc_start` every time).
Harness: `templates/network_dns/dnsrepro.yaml` + `dnsdiag.hpp` for the Waveshare ESP32-S3-ETH (W5500). Every 3 s a `DIAG`
line classifies the global DNS slots against each netif's cache (0 empty, E eth's, W wifi's, B both, X other) and names
the IDF default and lwIP `netif_default`; every 5 s a `LOOKUP` resolves a fresh `*.10.0.0.1.nip.io` name (a public
wildcard name, never cached). Timeline: Wi-Fi off at boot, enabled at 40 s, disabled and enabled at 100 and 110 s, W5500
PHY powered off at 170 s (a software link down, no unplugging) and on at 250 s. Set the `first`/`second` substitutions
for both priority orders; `phy_boot: "false"` starts with the PHY down (a stand-in for no cable). Needs Wi-Fi and, for
the full matrix, a live Ethernet link. The DIAG lines carry no addresses, so captures can be quoted.
- The W5500 PHY driver's `pwrctl()` is a no-op; `dnsdiag::phy_power()` writes PHYCFGR (OPMD=1, OPMDC 110 off and 111
on, RST low then high) and pokes `phy->get_link()`. Power down alone is not reported as a link loss (traffic kept
flowing); the PHY reset at power on is, so the 250 s step is a short link drop and return.
- Harness trap: `lwip_getaddrinfo` on the loop task with DNS servers set and no interface up blocks until the DNS
timeout and trips the task watchdog; `lookup()` skips while offline.
- Counts on #22800 (lookups ok/failed, PR against its merge base): Ethernet first with link 66/1 vs 23/43, Wi-Fi first
with the PHY down until 250 s 57/0 vs 1/56, Ethernet first with the PHY down 58/0 vs 20/36. Dev empties DNS on every
Wi-Fi join with Ethernet as the default, and on the Ethernet lease with Wi-Fi as the default.
- While the PHY is powered down the W5500 MAC logs `send command timeout` every 2 s on any build (driver noise from
the test).
- Compare build sizes only between names of equal length: the name is stored several times, so two names that differ
in length move the flash delta by tens of bytes.
@@ -27,3 +27,19 @@ Findings:
a warm board.
- To prove a setup priority change, add `on_boot` hooks at 1001, 950 and 900 that log markers and see where the
component's first publishes land between them.
## Charger defaults (#22769)
`npmhead.yaml` runs the register dump, the switch dump and the toggles once, 8 s after boot, on top of
`npmcommon.yaml`; put the charger block under test in `npmcommon.yaml`.
- nPM1300 charger registers with the reset-value defaults (32 mA, 3.6 V, 10 %, 2.9 V, 1000 mA; #22769): 0x0308/09 =
08/00, 0x030A/0B = CF/01, 0x030C = 0x030D = 02, 0x030E = 0x030F = 00. BCHGDISABLESET (0x0306) reads back 0x02 when
the NTC check is off (`thermistor_resistance: none`, ADCNTCRSEL 0x050A = 0) and 0x00 with 10k.
- No cell, charger on: BCHGCHARGESTATUS 0x05 (bit 0 plus trickle), VBAT about 3.61 V with VTERM 3.6 V (4.23 V with
4.2 V), 0.02 V with the charger off. NTCSTATUS (0x0332) and DIETEMPSTATUS (0x0333) read 00 throughout, so the NTC
hot and cold states cannot be reached on this board. BCHGERRREASON stays 00 across on/off toggles.
- A bare `charger:` key fails validation ("expected a dictionary"); use `charger: {}`. `esphome compile
--only-generate` with a private PLATFORMIO_CORE_DIR is enough to check the generated NPM1300Config initializer
without a board; a fresh private PlatformIO dir builds four nRF54 configs in about 10 min. A capture started before
the flash holds the old firmware first; cut at the last `Log initialized`.
@@ -104,3 +104,13 @@ Gotchas: an ESP-IDF image can contain `E7 02` at a 4 KiB boundary by chance; fin
instead of "no signature block": the partition still holds the old signature sector past the new image's end.
Results: https://github.com/esphome/esphome/pull/22398#issuecomment-6716330334
## ESP8266 web_server OTA interruption and SSE (#22682)
`scripts/web_server/async_bump/otacases.py` (README there). Facts from a D1 mini on dev and on ESPAsyncWebServer 3.12.1:
- A cut upload logs `OTA upload interrupted; aborting`, keeps the old image and serves again within a second; the
next full upload works without waiting for a lock release. No heap lost per cut.
- Uploads stop by themselves on the network about 1 in 6 to 1 in 15 (curl `000`, old image kept), and about 1 in 25
boots the new image while curl never gets the reply (it waits 30 to 60 s). Both builds; classify, do not fail on it.
- At OTA reboot the device does not close `/events`: a held client sees silence, rarely a FIN. Detect by idle time.
- Max free block drops from about 45 kB to about 37 kB after the first OTA and stays (both builds).
@@ -0,0 +1,23 @@
# serial_proxy with a simulated device (#22687)
`scripts/serial_proxy/`: no external device. On an ESP32-C3 (logger on USB_SERIAL_JTAG) the proxy UART is tx 4 rx 5
and the simulator UART tx 5 rx 4 (GPIO matrix, `allow_other_uses` on all four pins). `spsim.hpp` is the device; the
client drives it through the proxy with 7E-framed commands: `R` one 88 byte reply (A5, seq u32 BE, 83 bytes
(seq + j) % 251), `S` stream replies back to back, `A` ASH DATA frames every 50 ms with ACK latency timing (for
`ezsp_proxy_tap` in PROTOCOL mode), `X` log `SIMDONE` totals and reset, `B<baud>` change the sim's baud. SIMSTAT and
ASHWIN lines each second carry `backlog_max`, the proxy UART's RX backlog (bytes held back). The sim polls every
16 ms, the main loop cadence (a 1 ms interval also speeds up the proxy's loop and skews what it measures); its 1024
byte TX buffer holds the 185 bytes 115200 baud needs per poll, and ACK latencies are good to one poll.
- `spclient.py HOST --baud B --mode reply|stream|ash [--interval MS] [--duration S | --pauses N --pause S --read S]
[--protocol]`: raw plaintext API client (no encryption in the config), subscribes, brings the sim and the proxy to
the baud, pauses reads on a schedule and prints a RESULT line (complete and broken replies, received bytes). Compare
`received` with the sim's `SIMDONE bytes`.
- `suite.sh BUILD TREE` (env PORT) flashes `spBUILD.yaml` (a copy of `example-run.yaml` with its own `name`),
captures, runs every case (`CASES="a b"` picks some).
- `rx_buffer_size` on the proxy UART decides how long a blocked client can stall before the UART overflows
(32768 used; at 115200 a 5 s pause peaks near it).
- macOS ignores a small SO_RCVBUF after connect (reads back 36000), so a pause needs a few seconds to back up.
- Dev at 1920 baud loses bytes even with the client reading: the proxy sends 1 byte API messages (the UART wakes
the loop per byte) and the TCP queue fills; #22687's wait coalesces them. A tap in PROTOCOL mode keeps that loss.
- Stream mode at 115200 (11.5 KB/s) is more than the proxy forwards over Wi-Fi; both builds lose there.
@@ -102,3 +102,23 @@ mode. Both builds should close the stalled stream at about 20 s and close the st
builds (flash A, 4 multi runs, flash B, repeat) before calling a rate difference.
- Log text check for refactors: collect every `[web_server*]` line from all captures, mask timestamps, source line
numbers and digits, `sort -u`, and diff the two builds' sets.
## ESPAsyncWebServer / AsyncTCP bump A/B (#22682)
Scripts: `scripts/web_server/async_bump/` (README there). ESP32 never uses ESPAsyncWebServer: `web_server_base` loads
`web_server_idf` for every ESP32 build, Arduino included, so only ESP8266, RP2040/RP2350 and LibreTiny need a run.
Per board: page loads, a 180 s SSE client, REST control, `/update` OTA with the build's own image, restart button,
then the captive portal hammer (`scripts/wifisave_hammer.sh`, `bogus_saves: "5"`) on one board per family.
- Library check: ESP8266 builds use the native toolchain; the library copies sit in
`.esphome/pio_components/arduino8266/<hash>/esp32async/*/library.json`, not in `.piolibdeps`.
- Restart button over REST on LibreTiny: the HTTP reply is often lost (curl 000 after 5 to 10 s) although the board
restarts, on dev too. Not a regression signal.
- LibreTiny AsyncTCP 3.5.0 does not build with `framework: loglevel: debug` (or more verbose): `AsyncTCP.cpp:332` logs
`xPortGetCoreID()`, undeclared on LibreTiny; `log_d` is empty at the default level, so normal builds pass. Any
library bump for LibreTiny needs one build with the framework loglevel at debug.
- SSE rates on a board with a weak radio (a CB3S, say) are unreliable; decide SSE questions on a board with a good
link and the same SDK.
- `/events` churn (`rp2_web_ble/churn.sh`) faults `AsyncEventSource::send` in `AsyncClient::connected()` from
`deferrable_send_state` on the Pico W and the W5500-EVB-Pico; 3.12.1 and the RPAsyncTCP fork do not fix it.
- Stalled-reader runs need about 150 s: the client's receive window can take about 60 s to close, so a 60 s stall run
ends before the 20 s device timeout even starts.
@@ -117,3 +117,70 @@ The DUT runs `ap:` plus `captive_portal:` with no station network and logs `wifi
Erase the flash before each run so no saved credentials skip the portal. On dev the sleep type stayed 2 after
provisioning although the config asked for none; with #22564 it read 0 from the moment the AP started, and the
`networks:` path still read 0.
## Counting the Disassociation burst at reboot (#22715)
`scripts/wpa3_sniff/`: `sniffap.yaml` is a WPA3-SAE only AP with PMF required (any ESP32, set `board:`) that also turns
on promiscuous mode with the management filter (`sniff_inc.hpp`) and logs each assoc request, auth, disassoc and deauth
frame with transmitter, sequence number, Retry and Protected bits. The AP's own sniffer does see retransmitted copies
addressed to it. The station configs share `st-base.yaml`: `st-common.yaml` reboots 10 s after connect (alternating
`App.reboot()`/`safe_reboot()`), `stearly-common.yaml` reboots on the first loop pass after `WIFI_EVENT_STA_CONNECTED`
(the event handler wakes the loop and `on_loop` runs before the Wi-Fi component's loop; measured 0.15 to 0.46 ms
after the event with a 1 ms interval, before the Wi-Fi component handles it), `ds-common.yaml` deep sleeps 5 s.
`run.sh BUILD TREE SECONDS` (env ST, AP) flashes the station and captures both; `an15.py BUILD STA_MAC` groups the
copies per sequence number within one burst (copies more than 2 s apart start a new group).
- On ESP-IDF 5.5.6 the ESP32-C6 station sends 27 to 32 protected Disassociation copies on every reboot while
associated (espressif/esp-idf#19443); the classic ESP32 sends 1. Use a C6 (or C3) as the station.
- Deep sleep on dev sends no Disassociation at all; the AP finds out only when the station returns (reason 2).
- `esp_wifi_sta_get_ap_info()` already returns OK during the SAE exchange, before association completes.
- A capture over the C6 USB-JTAG console loses the last log line before deep sleep; count from the AP side.
- Serial capture timestamps are quantized to about 200 ms on a CH340, too coarse to time a 100 ms shutdown wait.
## Configured credentials from flash (#22757)
- Flows: two `networks:` with a hidden bogus one at higher priority, then the real one; the `wifisave_hammer.sh` flow
with `bogus_saves: 3`, long AP SSID and password (over 18 characters, the `CompactString` inline capacity), then an
RTS reset and an API restart; `wifi.configure` from `on_boot` after 25 s if not connected, then a reset (prints
"skipped"); a lambda building `WiFiAP` from runtime `std::string` and calling `set_sta()` + `start_connecting()`.
- The ESP-IDF API client sees `Connection reset by peer` on disconnect after a button that reconnects Wi-Fi; harmless.
## ESP32 with a hidden network first never joins (#22764)
- On dev an ESP32 with a hidden bogus network listed first ends every attempt on the real network at once with
"Unspecified" or "Auth Leave" and never joins; without `hidden: true` it joins. An ESP8266 joins either way.
- Cause, confirmed with debug prints: after the EXPLICIT_HIDDEN probe fails, the first real attempt calls
`esp_wifi_disconnect()` (config changed, not associated, nothing marked expected). The hidden attempt's leave event
(reason 36 STA_LEAVING, shown as "Unspecified", the bogus SSID, a zero BSSID) is then charged to the real attempt.
Attempt 2 gets reason 36 too, and the adapter restart aborts its auth (reason 3 "Auth Leave"). This loops forever on
ESP-IDF 5.5.6, with the esp-idf and arduino frameworks alike.
- ESPHome does not print numeric reasons; add one log line in the STA_DISCONNECTED handler and one before
`esp_wifi_set_config` to see them. `esp32: framework: log_level: INFO` shows the IDF `state: init -> auth` lines.
- Regression check: 2025.10.5 joins the hidden-first config (no explicit hidden phase); 2026.10.2 and dev loop.
- Soft AP outage on demand without touching the real AP: a D1 mini AP-only config with `enable_on_boot: false` and an
`on_boot` delay of 20 s before `wifi.enable`; an RTS reset of the D1 gives a 20+ s outage (an ESP32 station sees
Beacon Timeout). The ESP8266 soft AP DHCP can take about 20 s to answer after association.
- A soft AP that still holds an ESP32 from its previous boot rejects the first association with 'Too Many
Associations' (dev too), and some APs reject the first auth after a reset ("Authentication Failed" about 10 ms in,
attempt 2 joins).
## Router outages timed into the portal cooldown (#22797)
`scripts/wifi_outage/`: `rtr.yaml` makes a D1 mini an AP-only test router (its own SSID and password from secrets
`t_ssid`/`t_pw`) that logs `AP client connected/disconnected`, `AP Distribute Station IP` and a 1 s `RTR` station list.
`dut-common.yaml` is a BK7238 DUT (one network, `ap:` with `ap_timeout: 10s`, `captive_portal:`, UART1 log, `ALIVE`
line) with `dhcpwrap.hpp`, a `-Wl,--wrap=udp_sendto_if`/`udp_sendto_if_src` hook that counts DHCP DISCOVER, REQUEST
and RELEASE and logs `DHCPRELEASE sent n= up=` (`up` = netif still up). `c3-common.yaml` is an ESP-IDF C3 DUT without
AP. `outage.py DUT_PORT RTR_PORT LABEL N [--no-ap]` owns both ports: waits for 20 s of `conn=1`, holds the D1 in reset
(RTS on EN), waits for `→ RESTARTING` after `Starting fallback AP` (or `Restarting adapter` with `--no-ap`), lets the
router boot 1 s later and waits for the first line that ends the cooldown (the ones `an.py` uses, at most 40 s).
`an.py LABEL...` prints per outage skip, cooldown length, driver association and address inside the cooldown,
Connected-then-lost, held, releases after the router was back and router-side leaves.
- The D1 router is back about 1 s after release and hands out the address in about 0.2 s, but the BK7238 only reports
it 7 to 8 s after associating; the 30 s portal cooldown leaves plenty of room.
- BK7238 on dev: the Beken supplicant keeps trying after the skip branch's `WiFi.disconnect()` and joins about 5 s into
the cooldown on every outage; base then logs Connected and, 200 ms later, Connection lost (20/20), #22797 0/20. The
BK station never deauths on that teardown: the ESP8266 router logs `err already associed!` and only then the leave.
The release is issued with the netif already down (`up=0`), so it is not proof it hit the air.
- The first `Connection lost` in a capture is the reset from opening the D1 mini port (see `reference/boards.md`), not
an outage.
- About 2 min per outage; run blocks of 10 interleaved base/PR with a reflash between.
@@ -37,3 +37,12 @@ npm1300/*-cap.err
npm1300/*-run.log
rp2_sched_lock/*.txt
rp2_sched_lock/*.log
serial_proxy/*.txt
serial_proxy/*.log
web_server/async_bump/*.log
web_server/async_bump/*.txt
web_server/async_bump/*.pcap
web_server/async_bump/img-*.bin
wifi_outage/*.txt
wpa3_sniff/*.txt
api/overflow/putbe-sensors.yaml
@@ -0,0 +1,173 @@
"""apitest.py HOST OUT.json: Noise API check (entities, states, button) then a flood with a stalled reader
so the device's API overflow buffer fills and drains. Validates every received burst state."""
import asyncio
import gc
import json
import math
import pathlib
import socket
import sys
import time
from aioesphomeapi import SensorState, TextSensorState
sys.path.insert(0, str(pathlib.Path(__file__).resolve().parents[2]))
import aioesphomeapi.connection as conn_mod
from apiconn import connect # noqa: E402
HOST, OUT = sys.argv[1], sys.argv[2]
TL = 800
class SmallSock(socket.socket):
def setsockopt(self, level, opt, *a):
if level == socket.SOL_SOCKET and opt == socket.SO_RCVBUF and a[0] != 2048:
return None
return super().setsockopt(level, opt, *a)
async def start_small(addr_infos, **kw):
fam, typ, proto, _, addr = addr_infos[0]
s = SmallSock(fam, typ, proto)
s.setblocking(False)
s.setsockopt(socket.SOL_SOCKET, socket.SO_RCVBUF, 2048)
await asyncio.get_running_loop().sock_connect(s, addr)
return s
conn_mod.aiohappyeyeballs.start_connection = start_small
def txt_ok(s, slot):
if len(s) != TL or s[8] != "|" or not s[:8].isdigit():
return None
n = int(s[:8])
for i in range(9, TL):
if s[i] != chr(ord("A") + (n * 7 + slot * 3 + i) % 26):
return None
return n
async def main():
res = {"host": HOST}
stopped = {"t": None}
async def on_stop(expected):
stopped["t"] = time.monotonic()
cli = await connect(HOST, on_stop=on_stop)
info = await cli.device_info()
res["device"] = info.name
res["version"] = info.esphome_version
ents, _ = await cli.list_entities_services()
res["entities"] = len(ents)
by_key = {e.key: e for e in ents}
names = {e.object_id: e for e in ents}
flood = names["flood"]
bump = names["bump"]
state = {}
stats = {"states": 0, "bad_sensor": 0, "bad_text": 0, "burst_states": 0}
def on_state(s):
e = by_key.get(s.key)
if e is None:
return
stats["states"] += 1
oid = e.object_id
if isinstance(s, SensorState) and oid.startswith("s") and oid[1:].isdigit():
i = int(oid[1:])
stats["burst_states"] += 1
if not s.missing_state and (
int(s.state) % 100 != i or s.state != int(s.state)
):
stats["bad_sensor"] += 1
if isinstance(s, TextSensorState) and oid.startswith("t") and oid[1:].isdigit():
stats["burst_states"] += 1
if not s.missing_state and txt_ok(s.state, int(oid[1:])) is None:
stats["bad_text"] += 1
state[oid] = s.state
cli.subscribe_states(on_state)
await asyncio.sleep(3)
res["initial_states"] = len(state)
b0 = state.get("bump_count")
b0 = 0 if b0 is None or math.isnan(b0) else int(b0)
for _ in range(5):
cli.button_command(bump.key)
await asyncio.sleep(0.5)
await asyncio.sleep(3)
res["bump_delta"] = int(state.get("bump_count", 0)) - b0
o0 = (
state.get("ovf_enqueue"),
state.get("ovf_drain"),
state.get("ovf_enqueue_bytes"),
state.get("ovf_enqueue_fail"),
)
sock = cli._connection._socket
transport = next(
o
for o in gc.get_objects()
if isinstance(o, asyncio.Transport)
and not o.is_closing()
and o.get_extra_info("socket") is not None
and o.get_extra_info("socket").fileno() == sock.fileno()
)
cli.switch_command(flood.key, True)
await asyncio.sleep(3)
for _ in range(3):
transport.pause_reading()
await asyncio.sleep(6)
transport.resume_reading()
await asyncio.sleep(4)
cli.switch_command(flood.key, False)
await asyncio.sleep(6)
fN = state.get("flood_n")
final_ok = (
fN is not None
and not math.isnan(fN)
and all(state.get(f"s{i}") == fN * 100 + i for i in range(60))
and all(txt_ok(state.get(f"t{j}", ""), j) == int(fN) for j in range(4))
)
o1 = (
state.get("ovf_enqueue"),
state.get("ovf_drain"),
state.get("ovf_enqueue_bytes"),
state.get("ovf_enqueue_fail"),
)
res.update(stats)
res["flood_n"] = fN
res["final_states_match"] = final_ok
# a counter with no state (missing or NaN) gives no delta, which fails the run
for name, a, b in zip(
(
"ovf_enqueue_delta",
"ovf_drain_delta",
"ovf_enqueue_bytes_delta",
"ovf_enqueue_fail_delta",
),
o0,
o1,
strict=True,
):
ok_ab = all(x is not None and not math.isnan(x) for x in (a, b))
res[name] = b - a if ok_ab else None
res["dropped"] = stopped["t"] is not None
await cli.disconnect()
res["pass"] = (
not res["dropped"]
and final_ok
and res["bump_delta"] == 5
and stats["bad_sensor"] == 0
and stats["bad_text"] == 0
and (res["ovf_enqueue_delta"] or 0) > 0
and (res["ovf_drain_delta"] or 0) > 0
# a refused enqueue loses a message
and res["ovf_enqueue_fail_delta"] == 0
)
print(json.dumps(res))
pathlib.Path(OUT).write_text(json.dumps(res, indent=1), encoding="utf-8")
return 0 if res["pass"] else 1
sys.exit(asyncio.run(main()))
@@ -0,0 +1,67 @@
"""gen_sensors.py: write putbe-sensors.yaml for putbe-common.yaml.
It holds the BS_COUNT (60) template sensors bs0..bs59 and the BT_COUNT (4) 800 char text sensors bt0..bt3 that the
flood lambda publishes through the BS and BT pointer arrays of txt.hpp (filled by a boot lambda), and the remote
entity_transport sensors and binary sensors: count, flt and bin from both the base (${peer}b) and the PR (${peer}p)
name of the peer, over both links (et_plain, et_full).
"""
import pathlib
BS_COUNT = 60 # txt.hpp
BT_COUNT = 4 # txt.hpp
LINKS = ("plain", "full")
PEERS = ("b", "p")
def ptr_fill(kind: str, arr: str, prefix: str, n: int) -> list[str]:
ids = ", ".join(f"id({prefix}{i})" for i in range(n))
return [
" - lambda: |-",
f" {kind} *all[] = {{{ids}}};",
f" for (size_t i = 0; i < {arr}_COUNT; i++) {arr}[i] = all[i];",
]
def remote(link: str, peer: str, rid: str, name: str) -> list[str]:
return [
" - platform: entity_transport",
f" entity_transport_id: et_{link}",
f" id: r_{link}_{peer}_{rid}",
f" name: R {link} {peer} {name}",
f" provider: ${{peer}}{peer}",
f" remote_id: {rid}",
]
out = ["esphome:", " on_boot:", " priority: 600", " then:"]
out += ptr_fill("sensor::Sensor", "BS", "bs", BS_COUNT)
out += ptr_fill("text_sensor::TextSensor", "BT", "bt", BT_COUNT)
out.append("sensor:")
for i in range(BS_COUNT):
out += [
" - platform: template",
f" id: bs{i}",
f" name: S{i}",
" update_interval: never",
" accuracy_decimals: 0",
]
for link in LINKS:
for peer in PEERS:
out += remote(link, peer, "count", "count")
out += remote(link, peer, "flt", "flt") + [" accuracy_decimals: 3"]
out.append("binary_sensor:")
for link in LINKS:
for peer in PEERS:
out += remote(link, peer, "bin", "bin")
out.append("text_sensor:")
for i in range(BT_COUNT):
out += [
" - platform: template",
f" id: bt{i}",
f" name: T{i}",
" update_interval: never",
]
pathlib.Path(__file__).with_name("putbe-sensors.yaml").write_text(
"\n".join(out) + "\n", encoding="utf-8"
)
@@ -0,0 +1,30 @@
#pragma once
// Counts calls into the API overflow buffer through -Wl,--wrap (test only).
#include <cstddef>
#include <cstdint>
#include <sys/types.h>
struct iovec;
extern "C" {
uint32_t ovf_enq_calls = 0;
uint32_t ovf_enq_bytes = 0;
uint32_t ovf_drain_calls = 0;
uint32_t ovf_enq_fails = 0; // the buffer refused a message: it is lost
bool __real__ZN7esphome3api17APIOverflowBuffer11enqueue_iovEPK5iovecijj(void *self, const iovec *iov, int n,
size_t total, size_t skip);
bool __wrap__ZN7esphome3api17APIOverflowBuffer11enqueue_iovEPK5iovecijj(void *self, const iovec *iov, int n,
size_t total, size_t skip) {
bool ok = __real__ZN7esphome3api17APIOverflowBuffer11enqueue_iovEPK5iovecijj(self, iov, n, total, skip);
if (ok) {
ovf_enq_calls++;
ovf_enq_bytes += total - skip;
} else {
ovf_enq_fails++;
}
return ok;
}
ssize_t OVF_DRAIN_REAL(void *self, void *sock);
ssize_t OVF_DRAIN_WRAP(void *self, void *sock) {
ovf_drain_calls++;
return OVF_DRAIN_REAL(self, sock);
}
}
@@ -0,0 +1,156 @@
esphome:
name: ${name}
includes:
- ovfwrap.hpp
- txt.hpp
build_flags:
- -Wl,--wrap=_ZN7esphome3api17APIOverflowBuffer11enqueue_iovEPK5iovecijj
- -Wl,--wrap=${drain_sym}
- -DOVF_DRAIN_WRAP=__wrap_${drain_sym}
- -DOVF_DRAIN_REAL=__real_${drain_sym}
logger:
level: DEBUG
wifi:
ssid: !secret wifi_ssid
password: !secret wifi_password
power_save_mode: none
api:
reboot_timeout: 0s
encryption:
key: !secret api_key
ota:
- platform: esphome
packages:
# the flooded sensors and text sensors, their pointer arrays and the remote entities; run gen_sensors.py once
bs: !include putbe-sensors.yaml
globals:
- id: flood_n
type: uint32_t
initial_value: "0"
- id: bump_n
type: uint32_t
initial_value: "0"
- id: tick
type: uint32_t
initial_value: "0"
switch:
- platform: template
id: flood
name: Flood
optimistic: true
button:
- platform: template
name: Bump
on_press:
- lambda: |-
id(bump_n)++;
ESP_LOGI("bump", "press %u", (unsigned) id(bump_n));
id(bump_sensor).publish_state(id(bump_n));
interval:
- interval: 50ms
then:
- if:
condition:
switch.is_on: flood
then:
- lambda: |-
uint32_t n = ++id(flood_n);
for (size_t i = 0; i < BS_COUNT; i++)
BS[i]->publish_state(n * 100 + i);
for (size_t i = 0; i < BT_COUNT; i++)
BT[i]->publish_state(make_txt(n, i));
id(flood_sensor).publish_state(n);
- interval: 1s
then:
- lambda: |-
uint32_t t = ++id(tick);
id(loc_count).publish_state(t);
id(loc_float).publish_state(12345.678f + t * 0.001f - (t % 7) * 1000.5f);
id(loc_bin).publish_state((t / 3) % 2 == 1);
id(ovf_enq).publish_state(ovf_enq_calls);
id(ovf_enq_b).publish_state(ovf_enq_bytes);
id(ovf_drain).publish_state(ovf_drain_calls);
id(ovf_enq_fail).publish_state(ovf_enq_fails);
udp:
- id: udp_plain
port: 18611
addresses: ["${peer_ip}"]
- id: udp_full
port: 18612
addresses: ["${peer_ip}"]
packet_interface:
- platform: udp
id: pi_plain
udp_id: udp_plain
- platform: udp
id: pi_full
udp_id: udp_full
addressing:
address: ${my_addr}
destination: ${peer_addr}
encryption:
key: putbe test key
rolling_code_enable: true
reliability:
fragmentation:
session:
entity_transport:
- id: et_plain
packet_interface: pi_plain
update_interval: 2s
sensors:
- id: loc_count
broadcast_id: count
- id: loc_float
broadcast_id: flt
binary_sensors:
- id: loc_bin
broadcast_id: bin
- id: et_full
packet_interface: pi_full
sensors:
- id: loc_count
broadcast_id: count
- id: loc_float
broadcast_id: flt
binary_sensors:
- id: loc_bin
broadcast_id: bin
sensor:
- platform: template
id: flood_sensor
name: Flood N
update_interval: never
- platform: template
id: bump_sensor
name: Bump count
update_interval: never
- platform: template
id: loc_count
name: Local count
update_interval: never
- platform: template
id: loc_float
name: Local float
accuracy_decimals: 3
update_interval: never
- platform: template
id: ovf_enq
name: Ovf enqueue
update_interval: never
- platform: template
id: ovf_enq_b
name: Ovf enqueue bytes
update_interval: never
- platform: template
id: ovf_drain
name: Ovf drain
update_interval: never
- platform: template
id: ovf_enq_fail
name: Ovf enqueue fail
update_interval: never
binary_sensor:
- platform: template
id: loc_bin
name: Local bin
@@ -0,0 +1,19 @@
#pragma once
#include <string>
#include <cstdio>
// the flooded template sensors and text sensors, filled at boot by putbe-sensors.yaml (gen_sensors.py)
static constexpr size_t BS_COUNT = 60;
static esphome::sensor::Sensor *BS[BS_COUNT];
static constexpr size_t BT_COUNT = 4;
static esphome::text_sensor::TextSensor *BT[BT_COUNT];
// 800 chars: 8 digit n, '|', then letters from n and the slot
static std::string make_txt(uint32_t n, int slot) {
std::string s(800, ' ');
char head[16];
snprintf(head, sizeof(head), "%08u|", (unsigned) n);
for (int i = 0; i < 9; i++)
s[i] = head[i];
for (int i = 9; i < 800; i++)
s[i] = 'A' + (n * 7 + slot * 3 + i) % 26;
return s;
}
@@ -0,0 +1,168 @@
"""trcheck.py D1HOST D1VAR E32HOST E32VAR SECONDS OUT.json: watch the entity_transport remote entities on both
boards over the API and compare each received value with the sender's own published values."""
import asyncio
import json
import math
import pathlib
import struct
import sys
import time
sys.path.insert(0, str(pathlib.Path(__file__).resolve().parents[1]))
from apiconn import connect # noqa: E402
D1, D1V, E32, E32V, SECS, OUT = (
sys.argv[1],
sys.argv[2],
sys.argv[3],
sys.argv[4],
float(sys.argv[5]),
sys.argv[6],
)
def f32(v):
return struct.unpack("<f", struct.pack("<f", v))[0]
def expect(t): # 12345.678f + t * 0.001f - (t % 7) * 1000.5f in single precision
return f32(f32(f32(12345.678) + f32(f32(t) * f32(0.001))) - f32((t % 7) * 1000.5))
async def main():
boards = {"d1": (D1, D1V), "e32": (E32, E32V)}
hist = {
b: {"count": [], "float": set(), "bin": [], "count_to_float": {}}
for b in boards
}
obs = {} # (receiver, kind, field) -> list of (t, value, missing)
clients = {}
for b, (host, _) in boards.items():
c = await connect(host)
clients[b] = c
ents, _ = await c.list_entities_services()
keys = {e.key: e.object_id for e in ents}
peer_var = boards["e32" if b == "d1" else "d1"][1]
last = {}
def cb(s, b=b, keys=keys, peer_var=peer_var, last=last):
oid = keys.get(s.key)
t = time.monotonic()
if oid is None:
return
miss = bool(s.missing_state) or (
isinstance(s.state, float) and math.isnan(s.state)
)
h = hist[b]
if oid == "local_count" and not miss:
h["count"].append((t, int(s.state)))
last["count"] = int(s.state)
elif oid == "local_float" and not miss:
h["float"].add(f32(s.state))
if "count" in last:
h["count_to_float"][last["count"]] = f32(s.state)
elif oid == "local_bin":
h["bin"].append((t, bool(s.state)))
elif oid.startswith("r_"):
_, kind, var, field = oid.split("_")
if var == peer_var:
obs.setdefault(f"{b}<-{kind}:{field}", []).append(
(t, s.state, miss)
)
elif not miss:
obs.setdefault(f"{b}<-{kind}:{field}:otherpeer", []).append(
(t, s.state, False)
)
c.subscribe_states(cb)
await asyncio.sleep(SECS)
for c in clients.values():
await c.disconnect()
res = {"d1": D1V, "e32": E32V, "seconds": SECS, "links": {}}
for key, lst in sorted(obs.items()):
recv, rest = key.split("<-")
kind, field = rest.split(":")[:2]
sender = "e32" if recv == "d1" else "d1"
h = hist[sender]
if key.endswith("otherpeer"):
res["links"][key] = {"values_after_initial": len(lst) - 1}
continue
vals = [(t, v) for t, v, m in lst if not m][
1:
] # the first is the state from before the run
missing_after_first = sum(
1
for i, (t, v, m) in enumerate(lst)
if m and any(not mm for _, _, mm in lst[:i])
)
r = {"updates": len(vals), "missing_after_first": missing_after_first}
if field == "count":
sent = {c for _, c in h["count"]}
first = min(sent) if sent else 0
r["before_run"] = sum(1 for _, v in vals if int(v) < first)
last_sent = max(sent) if sent else 0
r["not_sent"] = sum(
1 for _, v in vals if int(v) > last_sent + 60 or int(v) != v
)
r["api_gap_values"] = sum(
1
for _, v in vals
if first <= int(v) <= last_sent and int(v) not in sent
)
lags = []
for t, v in vals:
prior = [c for tt, c in h["count"] if tt <= t]
if prior:
lags.append(prior[-1] - int(v))
r["max_lag_s"] = max(lags) if lags else None
elif field == "flt" and not h["count"]:
# no sender count over the API: the floats cannot be checked, which fails the link
r["no_sender_counts"] = True
elif field == "flt":
ok_t = [
t
for t, v, _m in obs[key.replace("flt", "count")]
if not (isinstance(v, float) and math.isnan(v))
and int(v) >= min(c for _, c in h["count"])
]
t0 = min(ok_t) if ok_t else 0
lo, hi = min(c for _, c in h["count"]), max(c for _, c in h["count"]) + 60
allowed = {expect(c) for c in range(max(1, lo - 30), hi + 1)}
r["not_sent"] = sum(1 for t, v in vals if t >= t0 and f32(v) not in allowed)
r["sender_float_mismatch"] = sum(
1 for c, fv in h["count_to_float"].items() if fv != expect(c)
)
else:
sent = [v for _, v in h["bin"]]
r["transitions"] = sum(
1 for i in range(1, len(vals)) if vals[i][1] != vals[i - 1][1]
)
r["sender_transitions"] = sum(
1 for i in range(1, len(sent)) if sent[i] != sent[i - 1]
)
res["links"][key] = r
ok = True
for k, r in res["links"].items():
if k.endswith("otherpeer"):
ok = ok and r["values_after_initial"] == 0
continue
if (
r["updates"] < 3
or r.get("not_sent", 0)
or r.get("sender_float_mismatch", 0)
or r.get("no_sender_counts")
):
ok = False
if "transitions" in r and r["transitions"] < 2:
ok = False
res["expected_links_present"] = (
len([k for k in res["links"] if not k.endswith("otherpeer")]) == 12
)
res["pass"] = ok and res["expected_links_present"]
print(json.dumps(res, indent=1))
pathlib.Path(OUT).write_text(json.dumps(res, indent=1), encoding="utf-8")
return 0 if res["pass"] else 1
sys.exit(asyncio.run(main()))
@@ -48,13 +48,14 @@ def rpc_strings(data: bytes) -> bytes:
return data[2 : 2 + data[1]]
def strings(p: bytes) -> list[str]:
"""Split length-prefixed strings; raise ValueError when a prefix runs past the end."""
def strings(p: bytes, raw: bool = False) -> list:
"""Split length-prefixed strings (bytes with raw); raise ValueError when a prefix runs past the end."""
out = []
while p:
if 1 + p[0] > len(p):
raise ValueError("string overruns the payload")
out.append(p[1 : 1 + p[0]].decode(errors="replace"))
b = p[1 : 1 + p[0]]
out.append(b if raw else b.decode(errors="replace"))
p = p[1 + p[0] :]
return out
@@ -0,0 +1,70 @@
"""Improv RPC request scan over the logger UART: dump each reply's fields (auth field as raw bytes).
usage: improvscan.py PORT OUT [--settle S]
Opens the port with DTR and RTS low, waits SETTLE seconds (a D1 mini and a classic ESP32 still reset on open), sends
request scan (0x04) and reads until the end-of-list reply or 15 s. OUT gets one line per reply and OUT.raw the raw
stream; prints the reply count, the auth values seen and the bad checksum count.
"""
import argparse
import pathlib
import sys
sys.path.insert(0, str(pathlib.Path(__file__).resolve().parent))
from improvcli import Dev, frame, rpc_strings, strings # noqa: E402
class ScanDev(Dev):
"""Collects scan replies as they are parsed instead of printing every frame."""
def __init__(self, *args) -> None:
super().__init__(*args)
self.lines: list[str] = []
self.auths: set[bytes] = set()
self.bad_checksums = 0
self.malformed = 0
def show(self, typ: int, data: bytes, ok: bool, short: bool) -> None:
if typ != 4:
return
self.bad_checksums += not ok
if short:
self.malformed += 1
self.lines.append(f"reply malformed {data.hex()} checksum_ok={ok}")
return
strs = strings(rpc_strings(data), raw=True)
if not strs:
self.lines.append(f"reply cmd={data[0]} checksum_ok={ok} end of list")
return
self.auths.add(strs[-1])
self.lines.append(
f"reply cmd={data[0]} checksum_ok={ok} fields={len(strs)} auth={strs[-1]!r} auth_len={len(strs[-1])} "
f"rssi_len={len(strs[1]) if len(strs) > 1 else -1}"
)
def main() -> None:
a = argparse.ArgumentParser()
a.add_argument("port")
a.add_argument("out")
a.add_argument("--settle", type=float, default=30)
o = a.parse_args()
d = ScanDev(o.port, 115200, o.out + ".raw", False, "")
d.pump(o.settle)
start = len(d.events)
d.ser.write(frame(4))
d.ser.flush()
end = d.wait_for(
lambda e: e[1] == 4 and e[3] and e[2][0] == 4 and e[2][1] == 0, 15, start
)
d.ser.close()
pathlib.Path(o.out).write_text("\n".join(d.lines) + "\n", encoding="utf-8")
print(
f"{len(d.lines)} replies; auth values: {sorted(d.auths)}; bad checksums: {d.bad_checksums}; malformed: {d.malformed}"
+ ("" if end else "; NO end-of-list reply within 15 s")
)
sys.exit(0 if end and not d.bad_checksums and not d.malformed else 1)
if __name__ == "__main__":
main()
@@ -0,0 +1,10 @@
esphome:
name: openap32
esp32:
board: esp32dev
framework:
type: esp-idf
logger:
wifi:
ap:
ssid: hwtest-open-ap
@@ -0,0 +1,58 @@
"""drive.py HOST OUT: press exercise, read API sensor states and web_server sensor JSON."""
import asyncio
import http.client
import pathlib
import sys
import urllib.parse
import urllib.request
from aioesphomeapi import ButtonInfo, SensorInfo
sys.path.insert(0, str(pathlib.Path(__file__).resolve().parents[1]))
from apiconn import connect # noqa: E402
HOST, OUT = sys.argv[1], sys.argv[2]
async def main():
c = await connect(HOST)
ents, _ = await c.list_entities_services()
b = {e.name: e for e in ents if isinstance(e, ButtonInfo)}
sens = {e.key: e for e in ents if isinstance(e, SensorInfo)}
st = {}
c.subscribe_states(lambda s: st.__setitem__(s.key, s) if s.key in sens else None)
await asyncio.sleep(7)
c.button_command(b["exercise"].key)
await asyncio.sleep(3)
out = []
for k, e in sorted(sens.items(), key=lambda x: x[1].name):
s = st.get(k)
out.append(
f"API {e.name}: state={s.state!r} missing={s.missing_state} acc={e.accuracy_decimals}"
if s
else f"API {e.name}: no state"
)
for path in (f"/sensor/{urllib.parse.quote(e.name)}", f"/sensor/{e.object_id}"):
try:
out.append(
f"WEB {e.name}: "
+ urllib.request.urlopen(f"http://{HOST}{path}", timeout=5)
.read()
.decode()
)
break
except (OSError, http.client.HTTPException) as ex:
last = ex
else:
out.append(f"WEB {e.name}: error {last}")
await c.disconnect()
pathlib.Path(OUT).write_text("\n".join(out) + "\n", encoding="utf-8")
# a sensor with no API state or no web_server answer leaves a hole in the A/B comparison
holes = [x for x in out if x.endswith(": no state") or ": error " in x]
for x in holes:
print(x)
return 1 if holes else 0
sys.exit(asyncio.run(main()))
@@ -0,0 +1,75 @@
esphome:
name: ${name}
logger:
# sensor states and switch, light and fan calls log at VERBOSE; at DEBUG an A/B sees none of them
level: VERBOSE
api:
reboot_timeout: 0s
ota:
- platform: esphome
web_server:
port: 80
improv_serial:
wifi:
ssid: !secret wifi_ssid
password: !secret wifi_password
binary_sensor:
- {platform: template, id: bs1, name: bs one, lambda: return true;}
- {platform: template, id: bs2, name: bs two, lambda: return false;}
switch:
- {platform: template, id: sw1, name: sw one, optimistic: true, restore_mode: ALWAYS_OFF, assumed_state: true}
number:
- {platform: template, name: num one, optimistic: true, min_value: 0, max_value: 10, step: 0.5, restore_value: false, initial_value: 2.5}
lock:
- {platform: template, name: lock one, optimistic: true, assumed_state: true}
fan:
- {platform: template, id: fan1, name: fan one, speed_count: 3, has_oscillating: true}
output:
- {platform: template, id: out1, type: binary, write_action: []}
light:
- {platform: binary, id: light1, name: light one, output: out1}
alarm_control_panel:
- platform: template
name: acp one
codes: ["1234"]
binary_sensors:
- input: bs1
bypass_armed_home: true
chime: true
- input: bs2
bypass_auto: true
sensor:
- {platform: template, id: s0, name: s acc0, accuracy_decimals: 0, lambda: return 12.5f;, update_interval: 5s}
- {platform: template, id: s2, name: s acc2, accuracy_decimals: 2, lambda: return 3.14159f;, update_interval: 5s}
- {platform: template, id: sneg, name: s neg2, accuracy_decimals: 2, lambda: return -0.005f;, update_interval: 5s}
- {platform: template, id: s_nan, name: s nan, accuracy_decimals: 1, lambda: return NAN;, update_interval: 5s}
- {platform: template, id: s_inf, name: s inf, accuracy_decimals: 1, lambda: return INFINITY;, update_interval: 5s}
- {platform: template, id: shi, name: s acc6, accuracy_decimals: 6, lambda: return 1.2345678f;, update_interval: 5s}
- {platform: template, id: sbig, name: s big, accuracy_decimals: 3, lambda: return 123456789.0f;, update_interval: 5s}
button:
- platform: template
name: exercise
on_press:
- lambda: |-
ESP_LOGI("lm", "MARK exercise");
ESPTime t = ESPTime::from_epoch_utc(1700000000);
char buf[ESPTime::STRFTIME_BUFFER_SIZE];
size_t n = t.strftime_to(buf, "%Y-%m-%d %H:%M:%S %A %B");
ESP_LOGI("lm", "STRF normal len=%u '%s'", (unsigned) n, buf);
n = t.strftime_to(buf, "%Y-%m-%d %H:%M:%S %A %B | %Y-%m-%d %H:%M:%S %A %B | %Y-%m-%d %H:%M:%S %A %B | %Y-%m-%d %H:%M:%S %A %B | %c");
ESP_LOGI("lm", "STRF overflow len=%u '%s'", (unsigned) n, buf);
ESP_LOGI("lm", "WORDS %s %s %s %s %s %s", YESNO(true), YESNO(false), ONOFF(true), ONOFF(false), TRUEFALSE(true), TRUEFALSE(false));
- switch.toggle: sw1
- delay: 200ms
- switch.toggle: sw1
- fan.turn_on: {id: fan1, speed: 2, oscillating: true}
- delay: 200ms
- fan.turn_off: fan1
- light.turn_on: light1
- delay: 200ms
- light.turn_off: light1
- lambda: ESP_LOGI("lm", "MARK exercise done");
text_sensor:
- platform: wifi_info
scan_results:
name: scan results
@@ -0,0 +1,68 @@
"""norm.py SERIAL: config lines ([C] and indented follow-ups), lm lines, sensor/switch/fan/light/lock state lines.
Volatile values are masked, not dropped, so a line that changes between builds still shows in the diff: addresses,
the device name (from the Hostname line), signal, setup times, and the value of keys that change per boot or per build.
"""
import pathlib
import re
import sys
out = []
cfg = False
MASKS = (
(re.compile(r"\b(?:[0-9A-Fa-f]{2}:){5}[0-9A-Fa-f]{2}\b"), "<mac>"),
(re.compile(r"\b\d{1,3}(?:\.\d{1,3}){3}\b"), "<ip>"),
(re.compile(r"-?\d+(?:\.\d+)? ?dBm?\b.*"), "<dB>"),
(re.compile(r"\btook \d+ ?ms\b"), "took <t>"),
)
# keys whose value changes per boot, per board or per build; the key stays, the value becomes <v>
VOLATILE_KEY = re.compile(
r"^((?:\w [\w.]+: )?\s*(?:Channel|Unsuccessful boot attempts|Reset reason|Wakeup cause|Free heap|Heap|"
r"Uptime|Compiled|Compilation time|Config hash|Build|ESPHome version|Flash size|CPU frequency|Chip):\s*)\S.*$",
re.IGNORECASE,
)
name = None
with pathlib.Path(sys.argv[1]).open(encoding="utf-8", errors="replace") as fh:
for raw in fh:
x = re.sub(r"^\[[0-9:.]+\]", "", raw.rstrip("\n"))
m = re.match(r"\[(\w)\]\[([\w.]+)(?::\d+)?\]: ?(.*)$", x)
if m:
lvl, tag, msg = m.groups()
cfg = lvl == "C"
if (
lvl == "C"
or tag == "lm"
or re.search(r"State|state|Setting|Sending|Turning|'.*'", msg)
and tag
in (
"sensor",
"switch",
"fan",
"light",
"lock",
"binary_sensor",
"number",
"alarm_control_panel",
"template.alarm_control_panel",
)
):
line = f"{lvl} {tag}: {msg}"
else:
continue
elif cfg and x.startswith(" "):
line = " " + x.strip()
else:
continue
if name is None and (m := re.search(r"Hostname: '?([\w-]+)'?$", line)):
name = m.group(1)
line = VOLATILE_KEY.sub(r"\1<v>", line)
for rx, repl in MASKS:
line = rx.sub(repl, line)
out.append(line)
text = "\n".join(out)
if name:
text = re.sub(rf"\b{re.escape(name)}\b", "<name>", text)
print(text)
if not out:
sys.exit(f"no config, lm or state lines in {sys.argv[1]}")
@@ -0,0 +1,18 @@
esphome:
name: npmhead
on_boot:
- priority: -100
then:
- delay: 8s
- script.execute: regdump
- script.execute: swdump
- delay: 2s
- script.execute: toggles
nrf52:
board: xiao_nrf54lm20a
logger:
level: DEBUG
hardware_uart: UART0
packages:
regdump: !include regdump.yaml
common: !include npmcommon.yaml
@@ -0,0 +1,5 @@
substitutions:
name: sppr
rx_buf: "32768"
packages:
common: !include sp-common.yaml
@@ -0,0 +1,67 @@
# serial_proxy on UART A, simulated device (spsim.hpp) on UART B, cross-wired through the GPIO matrix
esphome:
name: ${name}
includes:
- spsim.hpp
on_boot:
priority: -100
then:
- lambda: |-
spsim::bus = id(sim_uart);
spsim::proxy_bus = id(proxy_uart);
esp32:
variant: esp32c3
board: esp32-c3-devkitm-1
framework:
type: esp-idf
logger:
hardware_uart: USB_SERIAL_JTAG
wifi:
ssid: !secret wifi_ssid
password: !secret wifi_password
power_save_mode: none
api:
reboot_timeout: 0s
uart:
- id: proxy_uart
tx_pin:
number: GPIO4
allow_other_uses: true
rx_pin:
number: GPIO5
allow_other_uses: true
baud_rate: 1920
rx_buffer_size: ${rx_buf}
- id: sim_uart
tx_pin:
number: GPIO5
allow_other_uses: true
rx_pin:
number: GPIO4
allow_other_uses: true
baud_rate: 1920
rx_buffer_size: 1024
tx_buffer_size: 1024
serial_proxy:
- id: proxy
uart_id: proxy_uart
name: Sim Port
port_type: TTL
ezsp_proxy_tap:
serial_proxy_id: proxy
interval:
# the main loop cadence: a faster interval would also speed up the proxy's own loop and skew its latency
- interval: 16ms
then:
- lambda: spsim::poll();
- interval: 10s
then:
- lambda: 'ESP_LOGI("spsim", "HEAP free=%u min=%u", (unsigned) esp_get_free_heap_size(), (unsigned) esp_get_minimum_free_heap_size());'
@@ -0,0 +1,424 @@
"""Serial proxy client for spsim.hpp: subscribes over a raw plaintext API socket and checks every byte.
spclient.py HOST --baud B --mode reply|stream|ash [--pauses N --pause S --read S] [--duration S]
[--interval MS] [--protocol]
Reads only when told to, with a 4 KB receive buffer set before connect, so a pause fills the TCP
pipe toward the device quickly. Prints per-phase counters and a RESULT line.
"""
from __future__ import annotations
import argparse
import asyncio
import binascii
import socket
import sys
import time
from aioesphomeapi import api_pb2
from aioesphomeapi.core import MESSAGE_TYPE_TO_PROTO
TYPE_OF = {cls: num for num, cls in MESSAGE_TYPE_TO_PROTO.items()}
REPLY_SIZE = 88
DEBUG = False
def varint(v: int) -> bytes:
out = bytearray()
while True:
b = v & 0x7F
v >>= 7
if v:
out.append(b | 0x80)
else:
out.append(b)
return bytes(out)
def read_varint(buf: bytearray, pos: int) -> tuple[int, int] | None:
v = shift = 0
while pos < len(buf):
b = buf[pos]
pos += 1
v |= (b & 0x7F) << shift
if not b & 0x80:
return v, pos
shift += 7
return None
def reply(n: int) -> bytes:
return (
bytes([0xA5])
+ n.to_bytes(4, "big")
+ bytes((n + j) % 251 for j in range(REPLY_SIZE - 5))
)
class ReplyCheck:
"""Expects reply(0), reply(1), ... back to back; counts complete and broken replies."""
def __init__(self) -> None:
self.buf = bytearray()
self.next = 0
self.complete = 0
self.broken = 0
self.out_of_order = 0
self.received = 0
self.junk = 0 # leftovers of the baud switch before reply 0
def feed(self, data: bytes) -> None:
self.received += len(data)
self.buf += data
if (
self.complete == 0
and self.next == 0
and not self.buf.startswith(reply(0)[: len(self.buf)])
):
p = self.buf.find(b"\xa5\x00\x00\x00\x00")
cut = p if p != -1 else max(len(self.buf) - 4, 0)
self.junk += cut
del self.buf[:cut]
self._scan(False)
def _scan(self, final: bool) -> None:
buf = self.buf
while len(buf) >= REPLY_SIZE:
if buf[:REPLY_SIZE] == reply(self.next):
del buf[:REPLY_SIZE]
self.complete += 1
self.next += 1
continue
found = None
p = buf.find(0xA5, 1)
while p != -1 and p + REPLY_SIZE <= len(buf):
k = int.from_bytes(buf[p + 1 : p + 5], "big")
if buf[p : p + REPLY_SIZE] == reply(k):
found = (p, k)
break
p = buf.find(0xA5, p + 1)
if found is None:
if not final:
return
self.broken += 1
del buf[:]
return
p, k = found
if k < self.next:
self.out_of_order += 1
del buf[: p + REPLY_SIZE]
continue
self.broken += max(k - self.next, 1)
del buf[:p]
self.next = k
def finish(self) -> None:
self._scan(True)
def summary(self) -> str:
return (
f"complete={self.complete} broken={self.broken} out_of_order={self.out_of_order} "
f"received={self.received - self.junk} junk={self.junk} next={self.next}"
)
class AshCheck:
"""Parses ASH DATA frames from spsim and checks the sequence numbers."""
def __init__(self) -> None:
self.frame = bytearray()
self.esc = False
self.valid = 0
self.bad = 0
self.gaps = 0
self.missing_frames = 0
self.out_of_order = 0
self.last = -1
self.received = 0
def feed(self, data: bytes) -> None:
self.received += len(data)
for b in data:
if b == 0x7E:
self._frame()
self.frame.clear()
self.esc = False
elif b == 0x7D:
self.esc = True
else:
self.frame.append(b ^ 0x20 if self.esc else b)
self.esc = False
def _frame(self) -> None:
f = self.frame
if not f:
return
if len(f) != 27 or binascii.crc_hqx(bytes(f[:25]), 0xFFFF) != int.from_bytes(
f[25:27], "big"
):
self.bad += 1
return
seq = int.from_bytes(f[1:5], "big")
self.valid += 1
if self.last >= 0:
if seq <= self.last:
self.out_of_order += 1
elif seq != self.last + 1:
self.gaps += 1
self.missing_frames += seq - self.last - 1
self.last = seq
def summary(self) -> str:
return (
f"valid_frames={self.valid} bad_frames={self.bad} gaps={self.gaps} "
f"missing_frames={self.missing_frames} out_of_order={self.out_of_order} "
f"received={self.received} last_seq={self.last}"
)
class Client:
def __init__(self, check) -> None:
self.check = check
self.buf = bytearray()
self.reading = asyncio.Event()
self.reading.set()
self.last_data = time.monotonic()
self.responses: list[str] = []
self.setup_failed: list[str] = []
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.setsockopt(socket.SOL_SOCKET, socket.SO_RCVBUF, 4096)
s.setblocking(False)
self.sock = s
self.collect = True
async def send(self, msg) -> None:
payload = msg.SerializeToString()
frame = b"\x00" + varint(len(payload)) + varint(TYPE_OF[type(msg)]) + payload
await asyncio.get_running_loop().sock_sendall(self.sock, frame)
async def acked(self, msg, want: str, timeout: float = 3.0) -> None:
"""Send msg and wait for a response starting with want ("hello", "type=N")."""
n = len(self.responses)
await self.send(msg)
end = time.monotonic() + timeout
while time.monotonic() < end:
if any(r.startswith(want) for r in self.responses[n:]):
return
await asyncio.sleep(0.02)
print(f"no {want} response within {timeout:.0f} s", flush=True)
self.setup_failed.append(want.strip())
async def write(self, data: bytes) -> None:
await self.send(api_pb2.SerialProxyWriteRequest(instance=0, data=data))
async def reader(self) -> None:
loop = asyncio.get_running_loop()
while True:
await self.reading.wait()
data = await loop.sock_recv(self.sock, 4096)
if not data:
print("CONNECTION CLOSED by device", flush=True)
return
self.buf += data
while (m := self._parse()) is not None:
await self._handle(*m)
def _parse(self):
buf = self.buf
if not buf:
return None
assert buf[0] == 0, f"indicator {buf[0]}"
if (a := read_varint(buf, 1)) is None:
return None
size, pos = a
if (b := read_varint(buf, pos)) is None:
return None
mtype, pos = b
if len(buf) - pos < size:
return None
payload = bytes(buf[pos : pos + size])
del buf[: pos + size]
return mtype, payload
async def _handle(self, mtype: int, payload: bytes) -> None:
if mtype == TYPE_OF[api_pb2.PingRequest]:
await self.send(api_pb2.PingResponse())
elif mtype == TYPE_OF[api_pb2.SerialProxyDataReceived]:
data = api_pb2.SerialProxyDataReceived.FromString(payload).data
self.last_data = time.monotonic()
if self.collect:
if DEBUG:
print(
f"{time.monotonic():.2f} RX {len(data)} {data.hex()}",
flush=True,
)
self.check.feed(data)
elif mtype == TYPE_OF[api_pb2.SerialProxyRequestResponse]:
r = api_pb2.SerialProxyRequestResponse.FromString(payload)
self.responses.append(f"type={r.type} status={r.status} {r.error_message}")
elif mtype == TYPE_OF[api_pb2.HelloResponse]:
self.responses.append("hello")
elif mtype == TYPE_OF[api_pb2.DisconnectRequest]:
print("DISCONNECT REQUEST from device", flush=True)
async def configure(c: Client, baud: int) -> None:
await c.acked(
api_pb2.SerialProxyConfigureRequest(
instance=0, baudrate=baud, stop_bits=1, data_size=8, parity=0
),
f"type={api_pb2.SERIAL_PROXY_REQUEST_TYPE_CONFIGURE} ",
)
def data_verdict(check, mode: str, requests: int) -> str:
"""Reply mode must deliver every reply intact and in order. Stream mode and the ASH tap lose bytes over Wi-Fi on
every build (serial_proxy.md), so they are only judged on getting data at all."""
if check.received == 0:
return "loss:nothing-received"
if mode != "reply":
return "not-judged"
if check.broken or check.out_of_order or check.complete != requests:
return f"loss:{check.complete}/{requests}-complete"
return "ok"
async def main() -> int:
ap = argparse.ArgumentParser()
ap.add_argument("host")
ap.add_argument("--port", type=int, default=6053)
ap.add_argument("--baud", type=int, default=1920)
ap.add_argument("--mode", choices=["reply", "stream", "ash"], default="reply")
ap.add_argument(
"--interval",
type=float,
default=600,
help="request interval in ms (reply mode)",
)
ap.add_argument(
"--duration", type=float, default=0, help="plain reading time in s, no pauses"
)
ap.add_argument("--pauses", type=int, default=0)
ap.add_argument("--pause", type=float, default=10)
ap.add_argument("--read", type=float, default=15)
ap.add_argument(
"--protocol",
action="store_true",
help="put the port in PROTOCOL mode (tap active)",
)
ap.add_argument("--debug", action="store_true")
a = ap.parse_args()
global DEBUG # noqa: PLW0603
DEBUG = a.debug
check = AshCheck() if a.mode == "ash" else ReplyCheck()
c = Client(check)
loop = asyncio.get_running_loop()
await loop.sock_connect(c.sock, (a.host, a.port))
print(
f"SO_RCVBUF={c.sock.getsockopt(socket.SOL_SOCKET, socket.SO_RCVBUF)}",
flush=True,
)
rtask = asyncio.create_task(c.reader())
def reader_done(t: asyncio.Task) -> None:
if not t.cancelled() and (e := t.exception()) is not None:
print(f"READER FAILED {type(e).__name__}: {e}", flush=True)
rtask.add_done_callback(reader_done)
await c.acked(
api_pb2.HelloRequest(
client_info="spclient", api_version_major=1, api_version_minor=10
),
"hello",
)
await c.acked(
api_pb2.SerialProxyRequest(
instance=0, type=api_pb2.SERIAL_PROXY_REQUEST_TYPE_SUBSCRIBE
),
f"type={api_pb2.SERIAL_PROXY_REQUEST_TYPE_SUBSCRIBE} ",
)
# Bring the simulator to the target baud from whichever rate it is on, then the proxy
c.collect = False
for b in (1920, 115200):
await configure(c, b)
await c.write(b"\x7eB%d\x7e" % a.baud)
await asyncio.sleep(0.6)
await configure(c, a.baud)
await c.write(b"\x7eX\x7e")
await asyncio.sleep(0.3)
await c.acked(
api_pb2.SerialProxySetModeRequest(
instance=0,
mode=api_pb2.SERIAL_PROXY_MODE_PROTOCOL
if a.protocol
else api_pb2.SERIAL_PROXY_MODE_RAW,
),
f"type={api_pb2.SERIAL_PROXY_REQUEST_TYPE_SET_MODE} ",
)
await asyncio.sleep(
1.0
) # bytes left over from the baud switches drain before collecting
c.collect = True
print(f"setup responses: {c.responses}", flush=True)
stop = asyncio.Event()
requests = 0
async def requester() -> None:
nonlocal requests
while not stop.is_set():
await c.write(b"\x7eR\x7e")
requests += 1
await asyncio.sleep(a.interval / 1000)
if a.mode == "reply":
wtask = asyncio.create_task(requester())
else:
await c.write(b"\x7eS\x7e" if a.mode == "stream" else b"\x7eA\x7e")
wtask = None
t0 = time.monotonic()
if a.duration:
end = t0 + a.duration
while time.monotonic() < end:
await asyncio.sleep(min(30, end - time.monotonic()))
print(f"t={time.monotonic() - t0:.0f}s {check.summary()}", flush=True)
for i in range(a.pauses):
await asyncio.sleep(a.read)
print(
f"t={time.monotonic() - t0:.0f}s before pause {i + 1}: {check.summary()}",
flush=True,
)
c.reading.clear()
await asyncio.sleep(a.pause)
c.reading.set()
if a.pauses:
await asyncio.sleep(a.read)
stop.set()
if wtask:
await wtask
await asyncio.sleep(1.0)
await c.write(b"\x7eX\x7e")
await asyncio.sleep(3)
# Drain: until nothing arrives for 3 s
while time.monotonic() - c.last_data < 3:
await asyncio.sleep(0.5)
if isinstance(check, ReplyCheck):
check.finish()
# a reader that died (closed connection, bad frame) ends the counts early; say so on the RESULT line
reader = "stopped" if rtask.done() else "ok"
setup = "failed:" + ",".join(c.setup_failed) if c.setup_failed else "ok"
data = data_verdict(check, a.mode, requests)
print(
f"RESULT mode={a.mode} baud={a.baud} requests={requests} {check.summary()} reader={reader} setup={setup} data={data}",
flush=True,
)
rtask.cancel()
c.sock.close()
return 1 if c.setup_failed or reader != "ok" or data.startswith("loss") else 0
if __name__ == "__main__":
sys.exit(asyncio.run(main()))
@@ -0,0 +1,219 @@
#pragma once
// Simulated serial device for serial_proxy tests, on a UART cross-wired to the proxy's UART.
// Commands arrive through the proxy framed as 7E <text> 7E:
// R queue one 88 byte reply S stream replies back to back
// A stream ASH DATA frames every 50 ms and time the tap's ACKs
// X stop, log SIMDONE totals, reset the counters
// B<baud> switch this UART to <baud> (the client then reconfigures the proxy)
// Reply n: A5, n (u32 BE), then 83 bytes (n + j) % 251.
#include "esphome/core/helpers.h"
#include "esphome/core/log.h"
#include "esphome/components/uart/uart_component.h"
#include <esp_heap_caps.h>
#include <esp_system.h>
namespace spsim {
using namespace esphome;
static const char *const TAG = "spsim";
static constexpr size_t REPLY_SIZE = 88;
static constexpr uint32_t ASH_PERIOD_MS = 50;
inline uart::UARTComponent *bus = nullptr;
// the proxy's own UART: its RX backlog shows bytes held back while the client is blocked
inline uart::UARTComponent *proxy_bus = nullptr;
inline uint32_t backlog_max = 0;
enum Mode : uint8_t { IDLE, REPLY, STREAM, ASH };
inline Mode mode = IDLE;
inline uint32_t queued = 0, replies = 0, bytes = 0, requests = 0;
inline uint8_t out[512];
inline size_t out_len = 0, out_pos = 0;
// ASH timing
inline uint32_t ash_seq = 0, ash_sent = 0, acks = 0, ack_lat_sum = 0, ack_lat_max = 0, last_ash = 0;
inline uint32_t sent_at[8];
inline bool waiting[8];
inline uint32_t win_acks = 0, win_lat_sum = 0, win_lat_max = 0, win_unacked = 0;
// command parser
inline uint8_t cmd[32];
inline size_t cmd_len = 0;
inline uint32_t last_status = 0;
inline void make_reply(uint32_t n) {
out[0] = 0xA5;
put_be32(out + 1, n);
for (size_t j = 0; j < REPLY_SIZE - 5; j++)
out[5 + j] = (n + j) % 251;
out_len = REPLY_SIZE;
out_pos = 0;
}
inline size_t ash_put(uint8_t *p, uint8_t b) {
switch (b) {
case 0x7E:
case 0x7D:
case 0x11:
case 0x13:
case 0x18:
case 0x1A:
p[0] = 0x7D;
p[1] = b ^ 0x20;
return 2;
default:
p[0] = b;
return 1;
}
}
// DATA frame: control (frame number << 4), data = seq (u32 BE) + 20 pattern bytes, CRC-CCITT BE, flag
inline void make_ash(uint32_t seq) {
uint8_t body[1 + 24 + 2];
const uint8_t frm = seq & 7;
body[0] = frm << 4;
put_be32(body + 1, seq);
for (int j = 0; j < 20; j++)
body[5 + j] = (seq * 3 + j) & 0xFF;
uint16_t crc = crc16be(body, 25, 0xFFFF);
body[25] = crc >> 8;
body[26] = crc;
size_t n = 0;
for (uint8_t b : body)
n += ash_put(out + n, b);
out[n++] = 0x7E;
out_len = n;
out_pos = 0;
sent_at[frm] = millis();
if (waiting[frm])
win_unacked++;
waiting[frm] = true;
}
inline void log_done() {
ESP_LOGI(TAG,
"SIMDONE mode=%u requests=%u replies=%u bytes=%u ash_sent=%u acks=%u lat_avg=%u lat_max=%u heap=%u min=%u",
mode, requests, replies, bytes, ash_sent, acks, acks ? ack_lat_sum / acks : 0, ack_lat_max,
(unsigned) esp_get_free_heap_size(), (unsigned) esp_get_minimum_free_heap_size());
}
inline void reset() {
mode = IDLE;
queued = replies = bytes = requests = 0;
out_len = out_pos = 0;
ash_seq = ash_sent = acks = ack_lat_sum = ack_lat_max = 0;
win_acks = win_lat_sum = win_lat_max = win_unacked = 0;
for (auto &w : waiting)
w = false;
}
inline void on_ack(uint8_t num) {
// ACK num is the next frame expected, so it acknowledges num - 1
const uint8_t frm = (num - 1) & 7;
if (!waiting[frm])
return;
waiting[frm] = false;
const uint32_t lat = millis() - sent_at[frm];
acks++;
ack_lat_sum += lat;
ack_lat_max = std::max(ack_lat_max, lat);
win_acks++;
win_lat_sum += lat;
win_lat_max = std::max(win_lat_max, lat);
}
inline void on_frame() {
if (cmd_len == 3 && (cmd[0] & 0xF8) == 0x80) {
on_ack(cmd[0] & 7);
return;
}
if (cmd_len == 0)
return;
switch (cmd[0]) {
case 'R':
requests++;
if (mode == IDLE)
mode = REPLY;
queued++;
break;
case 'S':
mode = STREAM;
break;
case 'A':
reset();
mode = ASH;
break;
case 'X':
log_done();
reset();
break;
case 'B': {
cmd[cmd_len] = 0;
char *end = nullptr;
uint32_t baud = strtoul((const char *) cmd + 1, &end, 10);
if (baud == 0 || *end != '\0') {
ESP_LOGW(TAG, "bad baud command '%s'", (const char *) cmd + 1);
break;
}
bus->flush();
bus->set_baud_rate(baud);
bus->load_settings(false);
ESP_LOGI(TAG, "SIMBAUD %u", baud);
reset();
break;
}
default:
ESP_LOGW(TAG, "unknown command %02X len %u", cmd[0], cmd_len);
}
}
inline void poll() {
uint8_t b;
while (bus->available() && bus->read_byte(&b)) {
if (b == 0x7E) {
on_frame();
cmd_len = 0;
} else if (cmd_len < sizeof(cmd) - 1) {
cmd[cmd_len++] = b;
}
}
const uint32_t now = millis();
for (;;) {
if (out_pos < out_len) {
size_t room = bus->available_for_write();
if (room == 0)
break;
size_t n = std::min(room, out_len - out_pos);
bus->write_array(out + out_pos, n);
out_pos += n;
bytes += n;
if (out_pos < out_len)
break;
}
if (mode == REPLY && queued > 0) {
queued--;
make_reply(replies++);
} else if (mode == STREAM) {
make_reply(replies++);
} else if (mode == ASH && now - last_ash >= ASH_PERIOD_MS) {
last_ash = now;
make_ash(ash_seq++);
ash_sent++;
} else {
break;
}
}
if (proxy_bus != nullptr)
backlog_max = std::max<uint32_t>(backlog_max, proxy_bus->available());
if (now - last_status >= 1000) {
last_status = now;
if (mode == ASH) {
ESP_LOGI(TAG, "ASHWIN sent=%u acks=%u win_acks=%u lat_avg=%u lat_max=%u unacked=%u backlog_max=%u", ash_sent,
acks, win_acks, win_acks ? win_lat_sum / win_acks : 0, win_lat_max, win_unacked, backlog_max);
backlog_max = 0;
win_acks = win_lat_sum = win_lat_max = win_unacked = 0;
} else if (mode != IDLE) {
ESP_LOGI(TAG, "SIMSTAT requests=%u replies=%u bytes=%u queued=%u backlog_max=%u heap=%u", requests, replies,
bytes, queued, backlog_max, (unsigned) esp_get_free_heap_size());
backlog_max = 0;
}
}
}
} // namespace spsim
+40
View File
@@ -0,0 +1,40 @@
#!/bin/bash
# suite.sh BUILD TREE: flash spBUILD.yaml (a copy of example-run.yaml) from TREE, capture the board, run every case
# (CASES="a b" picks some), print the client RESULT and the simulator's SIMDONE totals per case.
# env: PORT (the board's console), PY, PIO_CORE (default ./pio)
D=$(cd "$(dirname "$0")" && pwd); S=$(dirname "$D"); cd "$D" || exit 1
B=${1:?build}; TREE=${2:?tree}; PORT=${PORT:?set PORT}; PY=${PY:-$TREE/venv/bin/python}
pkill -f "[s]erial_capture.py $PORT "; sleep 1
TREE=$TREE PY=$PY bash "$S/pio_build.sh" upload "sp$B" "$D/sp$B.yaml" "$PORT" || exit 1
CAP=$D/$B-suite-cap-$(date +%H%M%S).txt
nohup $PY "$S/serial_capture.py" "$PORT" 7200 "$CAP" > /dev/null 2>&1 &
IP=$(bash "$S/resolve.sh" "sp$B" 40) || exit 1
sleep 10
grep -a HEAP $CAP | tail -1 | sed 's/^/BOOT /'
FAILED=()
run() { # run LABEL CLIENT-ARGS...
local L=$1; shift
[[ ${CASES:-all} == all || " $CASES " == *" $L "* ]] || return 0
local N
N=$(( $(wc -l < "$CAP") + 1 )) # first capture line of this case
local RC
timeout 1200 $PY spclient.py $IP "$@" > $B-$L.txt 2>&1; RC=$?
[ $RC = 0 ] || FAILED+=("$L")
sleep 2
echo "== $L (client exit $RC): $(grep RESULT $B-$L.txt || echo 'no RESULT line')"
echo " $(tail -n +$N $CAP | grep -a SIMDONE | tail -1 | sed 's/.*SIMDONE//' | grep . || echo 'no SIMDONE in this case')"
echo " $(tail -n +$N $CAP | grep -a -E 'SIMSTAT|ASHWIN' | sed 's/.*backlog_max=\([0-9]*\).*/\1/' | sort -n | tail -1 | sed 's/^/peak backlog /')"
grep -a -c -E "CONNECTION CLOSED|DISCONNECT" $B-$L.txt | sed 's/^/ disconnects /'
}
run normal1920 --baud 1920 --mode reply --interval 600 --duration 180
run normal115200 --baud 115200 --mode stream --duration 120
run pause1920 --baud 1920 --mode reply --interval 600 --pauses 12 --pause 10 --read 15
run pause115200 --baud 115200 --mode reply --interval 20 --pauses 12 --pause 5 --read 8
run tap115200 --baud 115200 --mode ash --protocol --pauses 10 --pause 5 --read 8
run normal115200r --baud 115200 --mode reply --interval 20 --duration 120
sleep 11
grep -a HEAP $CAP | tail -1 | sed 's/^/END /'
CR=$(grep -a -c -E "rst:|Guru|panic|abort\(\)|Backtrace" "$CAP"); echo "crash lines $CR"
[ "$CR" = 0 ] || FAILED+=("crash")
grep -a -q HEAP "$CAP" || FAILED+=("capture-empty") # the sim logs HEAP every 10 s
[ ${#FAILED[@]} = 0 ] || { echo "FAILED: ${FAILED[*]}"; exit 1; }
@@ -6,7 +6,9 @@
600 failed attempts, 4 bad-credential threads next to an authenticated `/events` stream) and `timing`
(wrong first byte vs wrong last byte, ms to the first 401 byte, Mann-Whitney z). Heap comes from the
debug "Heap free" sensor over REST. `--ota` uploads the image with credentials and waits for the reboot.
It is importable: `configure()` sets the host and credentials for `raw()`, `build()` and `uptime()`.
It is importable: `configure()` sets the host and credentials for `raw()`, `build()`, `uptime()`, `heap()`,
`mp_body()` and `bad_once()` (one wrong-hash request, drained to the device's close for at most 0.5 s), which the
other scripts here use.
- `isolate.py HOST USER PASS [ROUNDS] [CASE,...]`: one hostile request at a time with the uptime before and after,
to tie a reboot to one request.
- `compare.py DEV_PREFIX PR_PREFIX` lines the two runs up and prints both raw 401 heads;
@@ -18,9 +20,23 @@
and with credentials while on the AP; `approbe_check.py CLIENT_LOG` fails unless the POST without credentials
got 401, the one with credentials 200 and every probe answered), and `run.sh NAME CONFIG KIND PORT MODE` (build
with the chosen credentials through `../pio_build.sh`, flash, capture, authfuzz, crash counts).
- `leak.py HOST USER PASS [ROUNDS] [SETTLE_S] [CONC_S] [THREADS] [RATE]`: rounds of bad-credential load next to an
authenticated `/events` stream, heap read after a settle each round. RATE paces the wrong-hash requests to RATE per
second in total, so two builds that answer at different speeds get the same load (an unpaced flood compares load,
not code).
- `upload_reject.py HOST USER PASS [N] [KB]`: unauthenticated and wrong-hash `/update` uploads with the status,
uptime and heap after each.
- `rate.py HOST USER PASS [SEQ_N] [PAR_S] [THREADS]`: ms to the first byte and to the device's close per request,
sequential and parallel; a close that does not come within 1 s of the end of the body counts as "no close". `pcapphase.py PCAP DEVICE_IP`: per-connection accept, respond and close times from a
tcpdump capture (tcpdump drops packets at flood rates, so it samples).
- `nul_soak.py HOST USER PASS [N] [CASES] [MAXREB]`: NUL byte in the Authorization header, another header, the
request line, and split across TCP segments; N requests per case with an uptime check after each, a case stops
after MAXREB reboots.
- `configs/cpflow.sh DUT KIND DUT_PORT CLIENT_PORT [flashclient]`: flash the AP DUT, wait for "Starting fallback AP",
only then reset the client, or the client joins the real network first and never sees the AP.
- `heap_churn.py HOST USER PASS [N] [RATE] [SAMPLE_S] [AFTER_S]`: free heap (the "Heap free" sensor over REST) while N
wrong-credential requests run at a fixed RATE per second and for AFTER_S seconds after, sampled every 1 s for the
first 20 s after the load; prints the idle heap, the lowest heap under load, how long the heap takes to come back
first 20 s after the load; prints the rate actually reached, the idle heap, the lowest heap under load, how long the heap takes to come back
within 2 KB of idle, and whether the uptime went backwards. `configs/msl.sh NAME bw15.yaml PORT` runs it with the
big header cases on a BW15 (`configs/bw15.yaml` reads the heap every 1 s).
- `arena_drive.py HOST USER PASS [PHASE_S]` with `arena/`: the main task stack high-water mark under web_server load.
@@ -46,3 +62,19 @@ check for "[capture] port lost" before blaming the firmware for a reset.
Main task stack with the JSON arena (#22699): the lowest free stack of the main task went from 5860 B to 3796 B on the
BW15 and from 5312 B to 3604 B on the BK7238 with the arena on (it costs about 2 KB of stack under load). Run
the BW15 without a serial capture at VERY_VERBOSE (`NOCAP=1`), for the same CH340 reason.
Findings from #22684 (ESP8266, Pico 2 W, BW15):
- A NUL byte in any header reboots the board on ESPAsyncWebServer 3.9.6: about 1 in 6 requests on the ESP8266
(Exception 28/29, an OOM panic in `addHeader`), and nearly every request on the Pico 2 W (watchdog reset, no record).
3.12.1 (#22682) fixes the ESP8266 to 1 in 800 (one Exception 29 in the Wi-Fi beacon code, a heap damage victim) but
not the Pico 2 W. Rapid reboots drop the Pico into safe mode (web server off for 5 min), which a soak counts as
reboots; cap the case.
- RP2 never closes the connection after a 401 although it says `Connection: close`; a client reading to EOF waits for
its own timeout. Successful responses close after about 0.5 s. Same on dev.
- BW15: a 32 KB header or a 64 KB line with no newline resets the board (reset reason Power-On), and so does a 60 s
4-thread flood; free heap drops about 60 KB over 600 rapid connections and stays low. Same on dev.
- ESP8266 web OTA uploads fail now and then on dev and PR alike; see the ESP8266 web_server OTA section of
`reference/recipes/ota.md`.
- Unpaced floods vary 2x between passes of the same build; compare builds in alternating passes (dev, PR, dev, PR)
or paced. `/events` gets closed by the device (clean FIN after the ACK timeout) under an unpaced flood on any build.
- ESPAsyncWebServer 3.12.1 skips all leading spaces of a header value (3.9.6 skipped one).
@@ -30,15 +30,17 @@ USER = ""
PASSWORD = ""
HASH = ""
GOOD = ""
BAD = ""
OUT = None
ROWS: list[dict] = []
def configure(host: str, user: str, password: str, out: str | None = None) -> None:
global HOST, USER, PASSWORD, HASH, GOOD, OUT # noqa: PLW0603
global HOST, USER, PASSWORD, HASH, GOOD, BAD, OUT # noqa: PLW0603
HOST, USER, PASSWORD = host, user, password
HASH = base64.b64encode(f"{user}:{password}".encode()).decode()
GOOD = f"Basic {HASH}"
BAD = "Basic " + HASH[:-1] + flip(HASH[-1])
if out:
OUT = pathlib.Path(out + "-summary.txt").open("w", encoding="utf-8") # noqa: SIM115
@@ -145,18 +147,52 @@ def build(
return h + b"\r\n" + body
def heap() -> object:
for _ in range(3):
r = raw(build("GET", "/sensor/Heap%20free", GOOD))
def sensor(name: str, tries: int = 1, pause: float = 1.0) -> object:
"""Value of a sensor over REST with credentials, None when no try got one."""
for i in range(tries):
if i:
time.sleep(pause)
r = raw(build("GET", f"/sensor/{name}", GOOD))
if r["status"] == 200:
try:
with contextlib.suppress(ValueError):
return json.loads(r["body"]).get("value")
except ValueError:
pass
time.sleep(1)
return None
def heap(tries: int = 3) -> object:
return sensor("Heap%20free", tries)
def uptime(tries: int = 1) -> object:
return sensor("Uptime", tries, 2.0)
def bad_once(path: str = "/", drain_s: float = 0.5) -> str:
"""One wrong-hash GET; returns the status code, "<code>/noclose", "closed" or the error name.
Drains to EOF for at most drain_s so the device closes first (its TIME_WAIT is what churn tests load) without
waiting on boards that never close after a 401 (RP2); those come back as "<code>/noclose".
"""
try:
with socket.create_connection((HOST, 80), timeout=5) as s:
s.sendall(build("GET", path, BAD))
d = b""
while b"\r\n" not in d and (r := s.recv(64)):
d += r
if not d.startswith(b"HTTP"):
return "closed"
code = d.split(b" ", 2)[1].decode()
end = time.monotonic() + drain_s
with contextlib.suppress(OSError):
while (left := end - time.monotonic()) > 0:
s.settimeout(left)
if not s.recv(512):
return code
return f"{code}/noclose"
except Exception as e: # noqa: BLE001
return type(e).__name__
def heap_ok(h0: object, h1: object) -> bool:
"""Both heap readings present and the heap after a phase at most --heap-margin bytes lower than before.
@@ -171,14 +207,6 @@ def heap_ok(h0: object, h1: object) -> bool:
)
def uptime() -> object:
r = raw(build("GET", "/sensor/Uptime", GOOD))
try:
return json.loads(r["body"]).get("value") if r["status"] == 200 else None
except ValueError:
return None
def case(
name: str, req: bytes, expect: int | str, note: str = "", **kw: object
) -> dict:
@@ -528,7 +556,7 @@ def heavy() -> None:
def hammer() -> None:
while time.monotonic() < stop:
r = raw(
build("GET", "/", "Basic " + HASH[:-1] + flip(HASH[-1])),
build("GET", "/", BAD),
timeout=5,
read_body=False,
)
@@ -0,0 +1,29 @@
#!/bin/bash
# cpflow.sh DUT KIND DUT_PORT CLIENT_PORT [flashclient]: captive portal with web_server Basic auth configured. Flash the
# AP DUT (DUT.yaml) from TREE, wait for "Starting fallback AP", and only then reset the client (CLIENT.yaml, flashed
# from CLIENT_TREE with `flashclient`), or the client joins the real network first and never sees the AP.
# KIND pico waits for the re-enumerated port. env: TREE, CLIENT_TREE, CLIENT (default cpclient), PY
D=$(cd "$(dirname "$0")" && pwd); S=$(dirname "$(dirname "$D")"); N=${1:?dut}; K=${2:?kind}; P=${3:?dut port}
CP=${4:?client port}; T=${TREE:?set TREE}; PY=${PY:-$T/venv/bin/python}; C=${CLIENT:-cpclient}
# up NAME TREE PORT: upload NAME.yaml built from TREE with `-s name NAME`
up() { TREE=$2 PY=$PY PIO_CORE=${PIO_CORE:-$D/pio} bash "$S/pio_build.sh" upload "$1" "$D/$1.yaml" "$3"; }
pkill -f "[s]erial_capture.py $P "; pkill -f "[s]erial_capture.py $CP "; sleep 1
up "$N" "$T" "$P" || exit 1
if [ "$K" = pico ]; then sleep 3; for _ in $(seq 1 30); do [ -e "$P" ] && break; sleep 1; done; sleep 2; fi
: > "$D/$N-cp-serial.log"; ($PY "$S/serial_capture.py" "$P" 300 "$D/$N-cp-serial.log" 2> /dev/null &)
if [ "$5" = flashclient ]; then up "$C" "${CLIENT_TREE:?set CLIENT_TREE}" "$CP" || exit 1; fi
AP=0; for _ in $(seq 1 90); do grep -a -q "Starting fallback AP" "$D/$N-cp-serial.log" && AP=1 && break; sleep 2; done
[ $AP = 1 ] || { echo "== $N DUT: no 'Starting fallback AP' within 180 s"; pkill -f "[s]erial_capture.py $P "; exit 1; }
sleep 3
: > "$D/$N-cpclient-serial.log"
($PY "$S/serial_capture.py" "$CP" 240 "$D/$N-cpclient-serial.log" --reset 2> /dev/null &)
SAVED=0; for _ in $(seq 1 120); do grep -a -q "real save status" "$D/$N-cpclient-serial.log" && SAVED=1 && break; sleep 2; done
sleep 30; pkill -f "[s]erial_capture.py $P "; pkill -f "[s]erial_capture.py $CP "
echo "== $N client:"
grep -a -E "probe|real save|connected, ip" "$D/$N-cpclient-serial.log" | sed 's/ip [0-9.]*/ip <ip>/' | cut -c1-160
SAVES=$(grep -a -c 'Requested WiFi Settings Change' "$D/$N-cp-serial.log")
CR=$(grep -a -c -E 'Exception|Panic|HardFault|abort\(\)|WDT' "$D/$N-cp-serial.log")
echo "== $N DUT: wifi saves $SAVES, crash lines $CR," \
"joined $(grep -a -c -E 'WiFi Connected|Connected to' "$D/$N-cp-serial.log")"
[ $SAVED = 1 ] || { echo "== $N client: no 'real save status' within 240 s"; exit 1; }
[ "$CR" = 0 ] && [ "$SAVES" -gt 0 ] || { echo "== $N FAIL: crash lines or no save reached the DUT"; exit 1; }
@@ -4,80 +4,58 @@ usage: heap_churn.py HOST USER PASS [N] [RATE] [SAMPLE_S] [AFTER_S]
Prints one line per heap sample (seconds since start, phase, heap, uptime) and a summary: idle heap, lowest heap
during the load, heap at the end of the load, and seconds after the load until the heap is back within 2 KB of idle.
Built for lwIP TIME_WAIT checks (TCP_MSL): each closed connection may keep a PCB on the heap for 2 * TCP_MSL.
Exits 1 on a reboot, no idle heap reading, no recovery in the window, or when fewer than half the requests got a 401.
Exits 1 on a reboot, no idle heap reading, no recovery in the window, or when fewer than half the requests got a 401
that the device closed within 0.5 s ("401/noclose" does not load TIME_WAIT on the device).
"""
import base64
import json
import socket
import pathlib
import sys
import threading
import time
sys.path.insert(0, str(pathlib.Path(__file__).resolve().parent))
import authfuzz as af # noqa: E402
HOST, U, P = sys.argv[1:4]
N = int(sys.argv[4]) if len(sys.argv) > 4 else 600
RATE = float(sys.argv[5]) if len(sys.argv) > 5 else 3.0
SAMPLE = float(sys.argv[6]) if len(sys.argv) > 6 else 10.0
AFTER = float(sys.argv[7]) if len(sys.argv) > 7 else 150.0
H = base64.b64encode(f"{U}:{P}".encode()).decode()
BAD = "Basic " + H[:-1] + ("B" if H[-1] != "B" else "C")
af.configure(HOST, U, P)
read_fails = [0]
def val(path: str) -> float | None:
try:
with socket.create_connection((HOST, 80), timeout=4) as s:
s.sendall(
f"GET {path} HTTP/1.1\r\nHost: x\r\nAuthorization: Basic {H}\r\nConnection: close\r\n\r\n".encode()
)
buf = b""
while b"}" not in buf:
d = s.recv(512)
if not d:
break
buf += d
return float(json.loads(buf.split(b"\r\n\r\n", 1)[1])["value"])
except Exception: # noqa: BLE001
def val(name: str) -> float | None:
if (v := af.sensor(name)) is None:
read_fails[0] += 1
return None
def bad_once() -> str:
try:
with socket.create_connection((HOST, 80), timeout=5) as s:
s.sendall(
f"GET / HTTP/1.1\r\nHost: x\r\nAuthorization: {BAD}\r\nConnection: close\r\n\r\n".encode()
)
# read through the end of the status line; one recv may hold only part of it
d = b""
while b"\r\n" not in d and (r := s.recv(64)):
d += r
return d.split(b" ", 2)[1].decode() if d.startswith(b"HTTP") else "closed"
except Exception as e: # noqa: BLE001
return type(e).__name__
return float(v)
t0 = time.monotonic()
idle = [h for h in (val("/sensor/Heap%20free") for _ in range(3)) if h is not None]
idle = [h for h in (val("Heap%20free") for _ in range(3)) if h is not None]
idle_heap = max(idle) if idle else None
u_start = val("/sensor/Uptime")
u_start = val("Uptime")
print(f"idle heap {idle} uptime {u_start}", flush=True)
codes: dict = {}
load_done = threading.Event()
load_done_at: list[float] = []
load_secs: list[float] = []
def load() -> None:
nxt = time.monotonic()
nxt = load_start = time.monotonic()
try:
for _ in range(N):
if (w := nxt - time.monotonic()) > 0:
time.sleep(w)
nxt += 1.0 / RATE
k = bad_once()
k = af.bad_once()
codes[k] = codes.get(k, 0) + 1
finally:
load_done_at.append(time.monotonic())
# a board slower than RATE stretches the load; the summary reports the rate reached
load_secs.append(load_done_at[0] - load_start)
load_done.set()
@@ -96,7 +74,7 @@ while True:
phase = "load" if not load_done.is_set() else "after"
if phase == "after" and load_end is None:
load_end = (load_done_at[0] - t0) if load_done_at else now
h, u = val("/sensor/Heap%20free"), val("/sensor/Uptime")
h, u = val("Heap%20free"), val("Uptime")
samples.append((now, phase, h, u))
print(f"{now:6.0f}s {phase:5} heap {h} uptime {u}", flush=True)
if (
@@ -116,7 +94,7 @@ ups = [u for *_, u in samples if u is not None]
chain = ([u_start] if u_start is not None else []) + ups
reboot = any(b < a for a, b in zip(chain, chain[1:], strict=False))
print(
f"SUMMARY requests {N} at {RATE}/s: {codes}; idle {idle_heap}, lowest during load {min(load_heaps) if load_heaps else None}, "
f"SUMMARY requests {N} at {RATE}/s (reached {N / load_secs[0]:.1f}/s): {codes}; idle {idle_heap}, lowest during load {min(load_heaps) if load_heaps else None}, "
f"last during load {load_heaps[-1] if load_heaps else None}, back within 2 KB of idle "
f"{f'{recovered_at:.0f} s after the load (sampling every {SAMPLE:.0f} s)' if recovered_at is not None else 'never in the window'}"
f"; reboot {'YES' if reboot else 'no'}; failed heap/uptime reads {read_fails[0]}",
@@ -134,6 +112,8 @@ if not load_heaps:
if recovered_at is None:
fails.append("heap not back within 2 KB of idle")
if codes.get("401", 0) * 2 < N:
fails.append(f"only {codes.get('401', 0)} of {N} requests got a 401")
fails.append(
f"only {codes.get('401', 0)} of {N} requests got a 401 and a close from the device"
)
print("PASS" if not fails else "FAIL: " + ", ".join(fails), flush=True)
sys.exit(1 if fails else 0)
@@ -0,0 +1,135 @@
"""Heap leak check: rounds of bad-credential load, heap read after a settle each round.
usage: leak.py HOST USER PASS [ROUNDS] [SETTLE_S] [CONC_S] [THREADS] [RATE]
Each round: THREADS threads send wrong-hash requests for CONC_S seconds while one authenticated /events
stream is open, then everything is closed and the heap is read after SETTLE_S seconds (lwIP TIME_WAIT
and the async server's queued sockets hold heap for a while after a flood).
"""
import pathlib
import socket
import sys
import threading
import time
sys.path.insert(0, str(pathlib.Path(__file__).resolve().parent))
import authfuzz as af # noqa: E402
HOST, U, P = sys.argv[1:4]
ROUNDS = int(sys.argv[4]) if len(sys.argv) > 4 else 4
SETTLE = float(sys.argv[5]) if len(sys.argv) > 5 else 30
CONC = float(sys.argv[6]) if len(sys.argv) > 6 else 30
THREADS = int(sys.argv[7]) if len(sys.argv) > 7 else 4
# RATE > 0 paces the wrong-hash requests to RATE per second in total, so builds that answer at different
# speeds see the same load
RATE = float(sys.argv[8]) if len(sys.argv) > 8 else 0.0
af.configure(HOST, U, P)
def load_problem(codes: dict, sse: dict) -> str | None:
"""Why a round's load does not count, or None: too few wrong-hash requests ran (80% of RATE x CONC_S paced,
1 per second unpaced), fewer than 80% of them got a 401, or the /events stream never delivered a state."""
total = sum(codes.values())
got_401 = sum(v for k, v in codes.items() if k.split("/")[0] == "401")
need = 0.8 * RATE * CONC if RATE > 0 else CONC
if total < need:
return f"{total} requests, expected at least {need:.0f}"
if got_401 < 0.8 * total:
return f"only {got_401} of {total} requests got a 401"
if not sse["opens"] or not sse["states"]:
return "no /events states"
return None
def heap_settled() -> tuple:
time.sleep(SETTLE)
return af.heap(5), af.uptime(5)
h, u = heap_settled()
print(time.strftime("[%H:%M:%S]"), f"idle heap {h} uptime {u}", flush=True)
ups = [u]
heaps = [h]
load_fails = []
for r in range(ROUNDS):
stop = time.monotonic() + CONC
codes: dict = {}
lock = threading.Lock()
sse = {"states": 0, "opens": 0, "closes": 0, "errors": 0, "maxgap": 0.0}
def hammer(idx: int, stop: float = stop, lock=lock, codes: dict = codes) -> None:
step = THREADS / RATE if RATE > 0 else 0.0
nxt = time.monotonic() + (idx * step / THREADS if step else 0.0)
while time.monotonic() < stop:
if step:
if (wait := nxt - time.monotonic()) > 0:
time.sleep(wait)
nxt += step
if time.monotonic() >= stop:
break
k = af.bad_once()
with lock:
codes[k] = codes.get(k, 0) + 1
def stream(stop: float = stop, sse: dict = sse) -> None:
last = time.monotonic()
while time.monotonic() < stop:
try:
s = socket.create_connection((HOST, 80), timeout=10)
s.sendall(
f"GET /events HTTP/1.1\r\nHost: x\r\nAuthorization: {af.GOOD}\r\n\r\n".encode()
)
sse["opens"] += 1
s.settimeout(1)
while time.monotonic() < stop:
try:
d = s.recv(4096)
except TimeoutError:
continue
if not d:
sse["closes"] += 1
break
n = d.count(b"event: state")
if n:
now = time.monotonic()
sse["maxgap"] = max(sse["maxgap"], now - last)
last = now
sse["states"] += n
s.close()
except OSError:
sse["errors"] += 1
time.sleep(0.5)
ts = [threading.Thread(target=stream)] + [
threading.Thread(target=hammer, args=(i,)) for i in range(THREADS)
]
for t in ts:
t.start()
for t in ts:
t.join()
h, u = heap_settled()
print(
time.strftime("[%H:%M:%S]"),
f"round {r}: bad {codes} sse {sse['states']} states, opens {sse['opens']}, closes {sse['closes']}, errors {sse['errors']}, max gap {sse['maxgap']:.1f}s; heap after {SETTLE:.0f}s "
f"settle {h} uptime {u}",
flush=True,
)
ups.append(u)
heaps.append(h)
if why := load_problem(codes, sse):
print(
time.strftime("[%H:%M:%S]"),
f"round {r}: load not applied: {why}",
flush=True,
)
load_fails.append(r)
# a missing reading or an uptime that went down (a reboot) makes the heap numbers meaningless
fails = []
if any(x is None for x in heaps + ups):
fails.append("missing heap or uptime reading")
elif any(b < a for a, b in zip(ups, ups[1:], strict=False)):
fails.append("reboot (uptime went down)")
if load_fails:
fails.append(f"load not applied in rounds {load_fails}")
print("PASS" if not fails else "FAIL: " + ", ".join(fails), flush=True)
sys.exit(1 if fails else 0)
@@ -0,0 +1,95 @@
"""NUL byte in request headers, N requests per case, uptime checked after each request.
usage: nul_soak.py HOST USER PASS [N] [CASE,...] [MAXREB]
Cases: auth (Basic <hash>\\0garbage), other (NUL in another header), reqline (NUL in the request line),
split (header sent in two TCP segments, the NUL first in the second). Cases run in blocks, so a reboot that
lands a second late still counts against the right case. Prints one line per reboot and a summary per case.
"""
import pathlib
import socket
import sys
import time
sys.path.insert(0, str(pathlib.Path(__file__).resolve().parent))
import authfuzz as af # noqa: E402
HOST, U, P = sys.argv[1:4]
N = int(sys.argv[4]) if len(sys.argv) > 4 else 200
# A case stops after MAXREB reboots: a build that reboots every time would otherwise take hours
MAXREB = int(sys.argv[6]) if len(sys.argv) > 6 else 10
af.configure(HOST, U, P)
GOOD = af.GOOD.encode()
def uptime(tries: int = 30) -> float | None:
return None if (u := af.uptime(tries)) is None else float(u)
def send(parts: list[bytes]) -> str:
try:
s = socket.create_connection((HOST, 80), timeout=4)
except OSError as e:
return f"connect {type(e).__name__}"
s.setsockopt(socket.IPPROTO_TCP, socket.TCP_NODELAY, 1)
try:
for i, part in enumerate(parts):
if i:
time.sleep(0.08)
s.sendall(part)
d = s.recv(128)
return d.split(b"\r\n", 1)[0].decode("latin1") if d else "closed"
except OSError as e:
return type(e).__name__
finally:
s.close()
CASES = {
"auth": [
b"GET / HTTP/1.1\r\nHost: x\r\nAuthorization: "
+ GOOD
+ b"\x00garbage\r\nConnection: close\r\n\r\n"
],
"other": [b"GET / HTTP/1.1\r\nHost: x\r\nX-A: a\x00b\r\nConnection: close\r\n\r\n"],
"reqline": [b"GET /\x00x HTTP/1.1\r\nHost: x\r\nConnection: close\r\n\r\n"],
"split": [
b"GET / HTTP/1.1\r\nHost: x\r\nAuthorization: " + GOOD,
b"\x00garbage\r\nConnection: close\r\n\r\n",
],
}
names = sys.argv[5].split(",") if len(sys.argv) > 5 else list(CASES)
total_reboots = 0
for name in names:
reboots = 0
results: dict = {}
u0 = uptime()
t0 = time.monotonic()
for i in range(N):
r = send(CASES[name])
results[r] = results.get(r, 0) + 1
time.sleep(0.3)
u1 = uptime(3)
if u1 is None or u0 is None or u1 < u0:
time.sleep(2)
u1 = uptime()
if u1 is None or u0 is None or u1 < u0:
reboots += 1
print(
time.strftime("[%H:%M:%S]"),
f"REBOOT {name} request {i}: uptime {u0} -> {u1}",
flush=True,
)
u0 = u1
if reboots >= MAXREB:
break
sent = i + 1
total_reboots += reboots
print(
time.strftime("[%H:%M:%S]"),
f"CASE {name}: {sent} requests in {time.monotonic() - t0:.0f}s, reboots {reboots}, "
f"answers {results}",
flush=True,
)
print("PASS" if not total_reboots else f"FAIL: {total_reboots} reboots", flush=True)
sys.exit(1 if total_reboots else 0)
@@ -0,0 +1,77 @@
"""Per-connection phase times from a pcap of short HTTP requests (one request per connection).
usage: pcapphase.py PCAP DEVICE_IP [PORT]
For each client connection: accept (client SYN to device SYN-ACK), respond (client request data to the first
device data), close (first device data to device FIN). Prints median/p90 per phase in ms, plus SYN retries.
Needs tcpdump on PATH (on macOS it runs without sudo for members of the access_bpf group).
"""
import re
import statistics
import subprocess
import sys
pcap, dev = sys.argv[1], sys.argv[2]
port = sys.argv[3] if len(sys.argv) > 3 else "80"
proc = subprocess.run(
["tcpdump", "-nn", "-tt", "-r", pcap, f"tcp port {port}"],
capture_output=True,
text=True,
check=False,
)
if proc.returncode:
sys.exit(f"tcpdump failed ({proc.returncode}): {proc.stderr.strip()}")
out = proc.stdout
pat = re.compile(
r"^(\d+\.\d+) IP (\S+)\.(\d+) > (\S+)\.(\d+): Flags \[([^\]]+)\].*?length (\d+)"
)
conns: dict = {}
for line in out.splitlines():
m = pat.match(line)
if not m:
continue
t, src, sp, dst, dp, flags, ln = (
float(m[1]),
m[2],
m[3],
m[4],
m[5],
m[6],
int(m[7]),
)
from_dev = src == dev
key = dp if from_dev else sp
c = conns.setdefault(
key, {"syn": [], "synack": None, "req": None, "resp": None, "fin": None}
)
if not from_dev and flags == "S":
c["syn"].append(t)
elif from_dev and flags == "S.":
c["synack"] = c["synack"] or t
elif not from_dev and ln > 0 and c["req"] is None:
c["req"] = t
elif from_dev and ln > 0 and c["resp"] is None:
c["resp"] = t
if from_dev and "F" in flags and c["fin"] is None:
c["fin"] = t
phases: dict = {"accept": [], "respond": [], "close": []}
retries = 0
for c in conns.values():
if not c["syn"]:
continue
retries += len(c["syn"]) - 1
if c["synack"]:
phases["accept"].append((c["synack"] - c["syn"][0]) * 1000)
if c["req"] and c["resp"]:
phases["respond"].append((c["resp"] - c["req"]) * 1000)
if c["resp"] and c["fin"]:
phases["close"].append((c["fin"] - c["resp"]) * 1000)
for k, v in phases.items():
if len(v) > 2:
print(
f"{k:8} n {len(v):4} median {statistics.median(v):7.1f} ms p90 {statistics.quantiles(v, n=10)[-1]:7.1f} ms "
f"mean {statistics.mean(v):7.1f} ms"
)
print(f"connections {len(conns)}, SYN retransmits {retries}")
if not conns:
sys.exit(f"no connections to {dev}:{port} in {pcap}")
@@ -0,0 +1,135 @@
"""Rejection rate A/B: how fast the device answers and closes wrong-credential requests.
usage: rate.py HOST USER PASS [SEQ_N] [PAR_S] [THREADS]
Sequential: SEQ_N requests per path (/ and /sensor/Fast%20counter, wrong and right hash), each read to the end of
the body: ms to the first byte, and ms to the device's FIN when it comes within CLOSE_WAIT_S of the body ("no close"
counts the rest; RP2 never closes after a 401). Parallel: THREADS threads for PAR_S seconds on /, completed count.
"""
import pathlib
import re
import socket
import statistics
import sys
import threading
import time
sys.path.insert(0, str(pathlib.Path(__file__).resolve().parent))
import authfuzz as af # noqa: E402
HOST, U, P = sys.argv[1:4]
SEQ = int(sys.argv[4]) if len(sys.argv) > 4 else 100
PAR = float(sys.argv[5]) if len(sys.argv) > 5 else 30
TH = int(sys.argv[6]) if len(sys.argv) > 6 else 4
CLOSE_WAIT_S = 1.0
af.configure(HOST, U, P)
def body_done(buf: bytes) -> bool:
if b"\r\n\r\n" not in buf:
return False
head, body = buf.split(b"\r\n\r\n", 1)
m = re.search(rb"(?im)^content-length:\s*(\d+)", head)
return m is not None and len(body) >= int(m.group(1))
def one(path: str, auth: str) -> tuple:
"""(ms to the first byte, ms to the device's close or None, status)."""
try:
s = socket.create_connection((HOST, 80), timeout=5)
except OSError:
return None, None, "connect"
t1 = time.monotonic()
first = closed = None
st = "?"
buf = b""
try:
s.sendall(af.build("GET", path, auth))
while True:
d = s.recv(4096)
if not d:
closed = time.monotonic()
break
if first is None:
first = time.monotonic()
st = d.split(b" ", 2)[1].decode() if d.startswith(b"HTTP") else "junk"
buf += d
if body_done(buf):
s.settimeout(CLOSE_WAIT_S)
while d := s.recv(4096):
pass
closed = time.monotonic()
break
except TimeoutError:
if not body_done(buf):
st += "/TimeoutError"
except OSError as e:
st = st + "/" + type(e).__name__
s.close()
return (
(first - t1) * 1000 if first else None,
(closed - t1) * 1000 if closed else None,
st,
)
def codes_ok(codes: dict, want: str, n: int) -> bool:
"""No other HTTP status, and at least 95% of the requests got want; a few connect errors or no answer are
network noise (seen on healthy D1 mini and Pico 2 W runs)."""
status = {k.split("/")[0]: 0 for k in codes}
for k, v in codes.items():
status[k.split("/")[0]] += v
other = [k for k in status if k.isdigit() and k != want]
return not other and status.get(want, 0) >= 0.95 * n
def summ(v: list) -> str:
v = [x for x in v if x is not None]
return (
f"n {len(v)} median {statistics.median(v):.0f} mean {statistics.mean(v):.0f} p90 {statistics.quantiles(v, n=10)[-1]:.0f}"
if len(v) > 2
else "n<3"
)
fails = []
for path in ("/", "/sensor/Fast%20counter"):
for label, auth, want in (("bad", af.BAD, "401"), ("good", af.GOOD, "200")):
rows = [one(path, auth) for _ in range(SEQ)]
codes: dict = {}
for r in rows:
codes[r[2]] = codes.get(r[2], 0) + 1
if not codes_ok(codes, want, SEQ):
fails.append(f"seq {label} {path} {codes}")
print(
time.strftime("[%H:%M:%S]"),
f"seq {label} {path}: ttfb ms {summ([r[0] for r in rows])}; "
f"to close ms {summ([r[1] for r in rows])}, no close {sum(r[1] is None for r in rows)}; {codes}",
flush=True,
)
for rnd in range(2):
if rnd:
time.sleep(10) # let the first round's TIME_WAIT sockets go
stop = time.monotonic() + PAR
codes: dict = {}
lock = threading.Lock()
def w(stop: float = stop, lock=lock, codes: dict = codes) -> None:
while time.monotonic() < stop:
_, _, st = one("/", af.BAD)
with lock:
codes[st] = codes.get(st, 0) + 1
ts = [threading.Thread(target=w) for _ in range(TH)]
for t in ts:
t.start()
for t in ts:
t.join()
print(
time.strftime("[%H:%M:%S]"),
f"parallel {TH}x{PAR:.0f}s round {rnd}: {codes}",
flush=True,
)
print("PASS" if not fails else "FAIL: " + "; ".join(fails), flush=True)
sys.exit(1 if fails else 0)
@@ -0,0 +1,83 @@
"""Unauthenticated and wrong-hash /update uploads: status seen, uptime and heap after each.
usage: upload_reject.py HOST USER PASS [N] [KB]
"""
import os
import pathlib
import socket
import sys
import time
sys.path.insert(0, str(pathlib.Path(__file__).resolve().parent))
import authfuzz as af # noqa: E402
HOST, U, P = sys.argv[1:4]
N = int(sys.argv[4]) if len(sys.argv) > 4 else 5
KB = int(sys.argv[5]) if len(sys.argv) > 5 else 256
af.configure(HOST, U, P)
# the statuses authfuzz.case(..., "reject") accepts; no status (close or reset) counts only if the device stays up
REJECT = (400, 401, 413, 431)
def upload(auth: str | None) -> tuple[int | None, str]:
"""(HTTP status or None, text for the log)."""
body, ctype = af.mp_body(os.urandom(KB * 1024))
h = f"POST /update HTTP/1.1\r\nHost: x\r\nConnection: close\r\nContent-Type: {ctype}\r\n"
if auth:
h += f"Authorization: {auth}\r\n"
h += f"Content-Length: {len(body)}\r\n\r\n"
s = socket.create_connection((HOST, 80), timeout=20)
sent = 0
status = None
code = None
try:
s.sendall(h.encode())
for i in range(0, len(body), 4096):
s.sendall(body[i : i + 4096])
sent = i + 4096
except OSError as e:
status = f"{type(e).__name__} after {sent // 1024} KB"
try:
d = s.recv(256)
line = d.split(b"\r\n", 1)[0].decode("latin1") if d else "closed"
if (
line.startswith("HTTP/")
and len(parts := line.split()) > 1
and parts[1].isdigit()
):
code = int(parts[1])
status = f"{line} ({status})" if status else line
except OSError as e:
status = status or type(e).__name__
s.close()
return code, status
fails = 0
for i in range(N):
for name, auth in (
("noauth", None),
("wronghash", af.BAD),
):
u0 = af.uptime(10)
code, st = upload(auth)
time.sleep(3)
u1, hp = af.uptime(10), af.heap(10)
rb = "REBOOT" if u0 is None or u1 is None or u1 < u0 else "ok"
accepted = code is not None and code not in REJECT
fails += rb != "ok" or hp is None or accepted
print(
time.strftime("[%H:%M:%S]"),
f"{i} {name} {KB}KB -> {st}; heap {hp} {rb}{' NOT REJECTED' if accepted else ''}",
flush=True,
)
print(
"PASS"
if not fails
else f"FAIL: {fails} uploads not rejected, or followed by a reboot or no heap reading",
flush=True,
)
sys.exit(1 if fails else 0)
@@ -0,0 +1,30 @@
# ESPAsyncWebServer / AsyncTCP bump A/B (#22682)
Put `secrets.yaml` here and per-board configs that include `aws-common.yaml` (web_server, web_server plus esphome OTA,
prometheus, encrypted API, a 200 ms counter, uptime every 1 s, heap, template switch and number, a monochromatic light
on a template output, a restart button). The runners use `./pio` as the private PlatformIO directory; build with
`../../pio_build.sh compile NAME NAME.yaml` from the tree under test first.
- `run.sh NAME KIND PORT` (KIND esp, pico, bk, bw; env TREE): serial flash, capture, `flow.py`, `webota.sh`,
crash and boot counts.
- `otarun.sh NAME IP`: the same for boards reached only over the network (native OTA first, API log capture instead
of serial).
- `flow.py HOST OUT [PAGE_S] [SSE_S]`: 36 rounds of `/` plus a REST GET or `/metrics`, a 180 s `/events` client
(state count, device closes, gaps over 3 s), REST control (switch toggle, number 42, light brightness 128 read back).
- `webota.sh HOST IMAGE OUT`: `/update` upload (firmware.bin, or the UF2 on LibreTiny), wait for the uptime reset,
restart button, heap after reboot.
- `ssediag.py HOST SECONDS [PAGE_S]`: SSE with a REST poll every 10 s; per window fast-counter events and the lag
between the last SSE value and the REST value. A stall shows as `fast events 0` with a growing lag.
- `presstest.sh IP`: restart presses at uptime 20 s and 60 s, HTTP code and whether it restarted.
- `bkab.sh BUILD...` (env PORT): alternating serial-flash passes on a BK72xx board with a pcap of port 80 per pass.
- `rest.py` (REST GET, JSON value, uptime and heap for the Python scripts; `rest.py uptime|heap HOST` for the shell
ones) and `ltflash.sh` (the LibreTiny flash function `run.sh` and `bkab.sh` source).
## ESP8266 web OTA cases
- `ota8266.yaml` (`aws-common.yaml` on a D1 mini plus a version text sensor and the max free block) and
`build.sh dev|pr a|b`: two images per build that differ only in the version text sensor (`-s ver a/b`), copied to
`img-<kind>-<ver>.bin` here. `TREE_BASE` picks `<base>-dev` and `<base>-pr`.
- `otacases.py HOST KIND OUT [REPS] [1|2|3]`: case 1 full `/update` uploads alternating images (OK, REPLY_LOST,
INTERRUPTED, BAD), case 2 raw multipart POST cut at 30% and 80% (reboot, serving, heap), case 3 upload with a held
and a reconnecting `/events` client (a stream counts as ended on FIN or RST, or 6 s of silence).
@@ -0,0 +1,59 @@
# Shared by every web_server build: web_server + web OTA + prometheus, fast sensor, switch, number, light
esphome:
name: ${name}
logger:
level: DEBUG
api:
encryption:
key: !secret api_key
reboot_timeout: 0s
ota:
- platform: esphome
- platform: web_server
web_server:
port: 80
prometheus:
debug:
update_interval: 5s
text_sensor:
- platform: debug
reset_reason:
name: Reset reason
sensor:
- platform: uptime
name: Uptime
update_interval: 1s
- platform: debug
free:
name: Heap free
- platform: template
name: Fast counter
update_interval: 200ms
lambda: |-
static uint32_t n = 0;
return n++;
switch:
- platform: template
name: Test switch
optimistic: true
number:
- platform: template
name: Test number
optimistic: true
min_value: 0
max_value: 100
step: 1
output:
- platform: template
id: light_out
type: float
write_action:
- lambda: ESP_LOGD("test", "light out %.3f", state);
light:
- platform: monochromatic
name: Test light
output: light_out
default_transition_length: 0s
button:
- platform: restart
name: Restart
@@ -0,0 +1,33 @@
#!/bin/bash
# bkab.sh BUILD...: BK72xx SSE A/B, alternating builds. Per pass: serial flash, serial capture, pcap of port 80,
# flow.py (180 s pages, 180 s SSE, REST control), then ssediag.py (60 s pages, 150 s SSE with a REST poll).
# env: PORT, PY, IFACE (default en0; tcpdump needs capture rights)
D=$(cd "$(dirname "$0")" && pwd); S=$(dirname "$(dirname "$D")"); P=${PORT:?set PORT}; PY=${PY:-python3}
IF=${IFACE:-en0}
. "$D/ltflash.sh"
k=0; FAILED=()
for N in "$@"; do
k=$((k + 1)); T=$N-p$k
pkill -f "[s]erial_capture.py $P "; sleep 1
ltflash bk "$P" "$D/.esphome/build/$N/.pioenvs/$N/firmware.uf2" "$D/$T-upload.log" ||
{ echo "== pass $k $N flash failed three times (see $D/$T-upload.log)"; exit 1; }
echo "== pass $k $N flashed"
: > "$D/$T-serial.log"; ($PY "$S/serial_capture.py" "$P" 1200 "$D/$T-serial.log" --released 2> "$D/$T-cap.err" &)
HOST=$(bash "$S/resolve.sh" "$N" 60) || { echo "no host"; FAILED+=("p$k:no-host"); continue; }; sleep 10
(tcpdump -i "$IF" -s 128 -w "$D/$T.pcap" host "$HOST" and tcp port 80 > /dev/null 2>&1 &); sleep 1
$PY -I "$D/flow.py" "$HOST" "$D/$T" | grep -E 'PAGES|SSE|CONTROL|heap'
[ "${PIPESTATUS[0]}" = 0 ] || FAILED+=("p$k:flow")
$PY -I "$D/ssediag.py" "$HOST" 150 60 > "$D/$T-diag.txt" || FAILED+=("p$k:sse")
echo "diag: $(tail -1 "$D/$T-diag.txt"), zero windows $(grep -c 'fast events 0' "$D/$T-diag.txt")," \
"max lag $(grep -o 'lag [0-9-]*' "$D/$T-diag.txt" | awk '{print $2}' | sort -n | tail -1)"
pkill -f "[t]cpdump -i $IF -s 128 -w $D/$T.pcap"; pkill -f "[s]erial_capture.py $P "; sleep 1
[ -s "$D/$T.pcap" ] || { echo "pcap: none written (tcpdump capture rights?)"; FAILED+=("p$k:pcap"); }
[ -s "$D/$T-serial.log" ] ||
{ echo "serial: EMPTY capture, counts below mean nothing: $(tail -1 "$D/$T-cap.err")"; FAILED+=("p$k:capture"); }
# not a bare "abort": the Wi-Fi component logs "aborting connection" on a healthy boot
CR=$(grep -a -c -E 'CRASH|Crash|abort\(\)|HardFault|Undefined instruction' "$D/$T-serial.log")
[ "$CR" = 0 ] || FAILED+=("p$k:crash")
echo "serial: crash lines $CR," \
"boots $(grep -a -c 'Log initialized' "$D/$T-serial.log")"
done
[ ${#FAILED[@]} = 0 ] || { echo "FAILED: ${FAILED[*]}"; exit 1; }
@@ -0,0 +1,7 @@
#!/bin/bash
# build.sh dev|pr VER: compile ota8266-<kind> from TREE_BASE-<kind> with `-s ver VER` and copy firmware.bin to
# img-<kind>-<VER>.bin. env: TREE_BASE (two checkouts named <base>-dev and <base>-pr), PY, BUILD_LOCK (optional)
D=$(cd "$(dirname "$0")" && pwd); S=$(dirname "$(dirname "$D")"); K=${1:?dev|pr}; V=${2:?ver}; N=ota8266-$K
T=${TREE_BASE:?set TREE_BASE}-$K
TREE=$T PIO_CORE=${PIO_CORE:-$D/pio} bash "$S/pio_build.sh" compile "$N" "$D/ota8266.yaml" -s ver "$V" || exit 1
cp "$D/.esphome/build/$N/.pioenvs/$N/firmware.bin" "$D/img-$K-$V.bin" && ls -l "$D/img-$K-$V.bin"
@@ -0,0 +1,128 @@
"""web_server flow: page loads + REST GETs, SSE soak, REST control. usage: flow.py HOST OUT_PREFIX [PAGE_S] [SSE_S]"""
import pathlib
import socket
import sys
import time
sys.path.insert(0, str(pathlib.Path(__file__).resolve().parent))
import rest # noqa: E402
from rest import heap, jget, req # noqa: E402
HOST, OUT = sys.argv[1], sys.argv[2]
rest.HOST = HOST
PAGE_S = float(sys.argv[3]) if len(sys.argv) > 3 else 180
SSE_S = float(sys.argv[4]) if len(sys.argv) > 4 else 180
log = pathlib.Path(OUT + "-flow.txt").open("w", encoding="utf-8") # noqa: SIM115
def p(*a) -> None:
s = " ".join(str(x) for x in a)
print(s, flush=True)
log.write(s + "\n")
log.flush()
p("heap start", heap(), "uptime", rest.uptime())
# 1. pages
gets = [
"/sensor/Fast%20counter",
"/switch/Test%20switch",
"/number/Test%20number",
"/light/Test%20light",
"/metrics",
]
res: dict[str, list] = {}
end = time.monotonic() + PAGE_S
i = 0
while time.monotonic() < end:
for path in ("/", gets[i % len(gets)]):
code, body, dt = req(path)
res.setdefault(path, []).append((code, len(body), dt))
i += 1
time.sleep(max(0.0, PAGE_S / 36 - 0.5))
for path, rows in res.items():
ok = sum(1 for c, _, _ in rows if c == 200)
p(
f"page {path}: {ok}/{len(rows)} ok, bytes {max(b for _, b, _ in rows)}, max {max(d for _, _, d in rows):.2f}s, "
f"codes {sorted({c for c, _, _ in rows})}"
)
pages_ok = sum(1 for rows in res.values() for c, _, _ in rows if c == 200)
pages_n = sum(len(rows) for rows in res.values())
p(f"PAGES {pages_ok}/{pages_n}")
# 2. SSE
states = pings = closes = opens = 0
gaps: list[float] = []
last = None
maxgap = 0.0
end = time.monotonic() + SSE_S
while time.monotonic() < end:
try:
s = socket.create_connection((HOST, 80), timeout=10)
s.sendall(
f"GET /events HTTP/1.1\r\nHost: {HOST}\r\nAccept: text/event-stream\r\n\r\n".encode()
)
opens += 1
s.settimeout(1.0)
buf = b""
while time.monotonic() < end:
try:
d = s.recv(8192)
except TimeoutError:
d = None
now = time.monotonic()
if d == b"":
closes += 1
p(f"sse closed by device after {now - (end - SSE_S):.1f}s")
break
if d:
buf += d
n = buf.count(b"event: state")
if n:
if last is not None:
gap = now - last
maxgap = max(maxgap, gap)
if gap > 3:
gaps.append(round(gap, 1))
last = now
states += n
pings += buf.count(b"event: ping")
buf = buf[buf.rfind(b"\n") + 1 :] if b"\n" in buf else buf[-64:]
s.close()
except Exception as e: # noqa: BLE001
p("sse error", type(e).__name__)
time.sleep(1)
p(
f"SSE {SSE_S:.0f}s: states {states} ({states / SSE_S:.1f}/s), pings {pings}, opens {opens}, device closes {closes}, "
f"max gap {maxgap:.1f}s, gaps>3s {len(gaps)} {gaps[:10]}"
)
# 3. REST control
before = jget("/switch/Test%20switch").get("value")
c1 = req("/switch/Test%20switch/toggle", "POST")[0]
time.sleep(0.5)
after = jget("/switch/Test%20switch").get("value")
c2 = req("/number/Test%20number/set?value=42", "POST")[0]
time.sleep(0.5)
num = jget("/number/Test%20number").get("value")
c3 = req("/light/Test%20light/turn_on?brightness=128", "POST")[0]
time.sleep(0.5)
lt = jget("/light/Test%20light")
ctl_ok = (
before != after
and c1 == 200
and c2 == 200
and float(num or -1) == 42
and c3 == 200
and lt.get("state") == "ON"
and lt.get("brightness") == 128
)
p(
f"CONTROL switch {before}->{after} ({c1}), number {num} ({c2}), light {lt.get('state')} b={lt.get('brightness')} ({c3}) "
f"=> {'OK' if ctl_ok else 'FAIL'}"
)
p("heap end", heap(), "uptime", rest.uptime())
# a failed page, no SSE state or a failed control fails the run for run.sh and otarun.sh
sys.exit(0 if ctl_ok and states and pages_ok == pages_n else 1)
@@ -0,0 +1,17 @@
# Sourced by run.sh and bkab.sh. ltflash bk|bw PORT UF2 LOG: flash a LibreTiny kit with the Python and the LibreTiny
# platform from ./pio (else ~/.platformio); BK72xx gets three tries.
ltflash() {
local d s py lt tool tries=1 to=600 opt=()
d=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd); s=$(dirname "$(dirname "$d")")
py=$d/pio/penv/.libretiny/bin/python; [ -x "$py" ] || py=~/.platformio/penv/.libretiny/bin/python
lt=$(ls -d "$d"/pio/platforms/libretiny* 2> /dev/null | head -1); [ -n "$lt" ] && opt=(-L "$lt")
case $1 in
bk) tool=flash_bk72xx.py; tries=3; to=400 ;;
bw) tool=flash_rtl87xx_kit.py ;;
*) echo "ltflash: unknown kind $1"; return 1 ;;
esac
for _ in $(seq 1 $tries); do
timeout $to "$py" "$s/$tool" "${opt[@]}" flash write -d "$2" "$3" > "$4" 2>&1 && return 0
done
return 1
}
@@ -0,0 +1,31 @@
# D1 mini web OTA images: aws-common.yaml plus a version text sensor, the max free block and alive logs
substitutions:
name: ota8266-x
ver: a
packages:
common: !include aws-common.yaml
esphome:
on_boot:
priority: -100
then:
- lambda: ESP_LOGI("otatest", "BOOTED ver=${ver} heap=%u", (unsigned) ESP.getFreeHeap());
esp8266:
board: d1_mini
wifi:
ssid: !secret wifi_ssid
password: !secret wifi_password
web_server:
version: 3
interval:
- interval: 10s
then:
- lambda: ESP_LOGI("otatest", "alive ver=${ver} heap=%u maxblk=%u", (unsigned) ESP.getFreeHeap(), (unsigned) ESP.getMaxFreeBlockSize());
text_sensor:
- platform: template
name: Build ver
lambda: return {"${ver}"};
update_interval: 60s
sensor:
- platform: debug
block:
name: Heap max block
@@ -0,0 +1,302 @@
"""otacases.py HOST KIND OUT [REPS]: web OTA cases on the D1 mini. KIND dev|pr picks img-KIND-{a,b}.bin.
case1 happy uploads, case2 uploads cut at 30%/80%, case3 uploads with a live /events client plus a reconnecting one."""
import json
import pathlib
import socket
import subprocess
import sys
import threading
import time
sys.path.insert(0, str(pathlib.Path(__file__).resolve().parent))
import rest # noqa: E402
from rest import jv, uptime # noqa: E402
HOST, KIND, OUT = sys.argv[1], sys.argv[2], sys.argv[3]
rest.HOST = HOST
REPS = int(sys.argv[4]) if len(sys.argv) > 4 else 5
D = pathlib.Path(__file__).resolve().parent
IMG = {v: str(D / f"img-{KIND}-{v}.bin") for v in "ab"}
log = pathlib.Path(OUT).open("a", encoding="utf-8") # noqa: SIM115
def p(*a):
s = time.strftime("%H:%M:%S ") + " ".join(str(x) for x in a)
print(s, flush=True)
log.write(s + "\n")
log.flush()
def ver():
return jv("/text_sensor/Build%20ver", "value") or jv(
"/text_sensor/Build%20ver", "state"
)
def heap():
return rest.heap(), jv("/sensor/Heap%20max%20block")
def events_ok(t=8):
"""open /events, True when a state event arrives within t s"""
try:
s = socket.create_connection((HOST, 80), timeout=t)
s.sendall(f"GET /events HTTP/1.1\r\nHost: {HOST}\r\n\r\n".encode())
s.settimeout(t)
buf = b""
end = time.monotonic() + t
while time.monotonic() < end and b"event: state" not in buf:
d = s.recv(4096)
if not d:
break
buf += d
s.close()
return b"event: state" in buf
except Exception: # noqa: BLE001
return False
def current_ver():
for _ in range(40):
v = ver()
if v:
return v
time.sleep(2)
return None
def wait_boot(expect, before_up, cap=120):
t0 = time.monotonic()
while time.monotonic() - t0 < cap:
u = uptime()
if u is not None and u < before_up and ver() == expect:
return time.monotonic() - t0
time.sleep(1)
return None
def full_upload(tag):
cur = current_ver()
nxt = "b" if cur == "a" else "a"
for _ in range(3):
if (up := uptime()) is not None:
break
time.sleep(1)
if up is None:
# without a reading before the upload a later boot cannot be told from the old image still running
p(f"{tag} no uptime before the upload => BAD")
return "BAD"
t0 = time.monotonic()
r = subprocess.run(
[
"curl",
"-s",
"-m",
"300",
"-o",
"/dev/null",
"-w",
"%{http_code} %{size_upload} %{time_total}",
"-F",
f"update=@{IMG[nxt]}",
f"http://{HOST}/update",
],
capture_output=True,
text=True,
check=False,
)
up_s = time.monotonic() - t0
time.sleep(3)
back = wait_boot(nxt, up + up_s + 5, 60 if not r.stdout.startswith("200") else 120)
if back is not None and r.stdout.startswith("200"):
st = "OK"
elif back is not None:
st = "REPLY_LOST" # new image booted but the HTTP reply never arrived
elif (
back is None
and ver() == cur
and rest.req("/", timeout=8)[0] == 200
and (uptime() or 0) > up
):
st = "INTERRUPTED" # upload cut on the network, old image still running and serving
else:
st = "BAD"
p(
f"{tag} upload {cur}->{nxt} curl '{r.stdout}' upload {up_s:.1f}s, booted {nxt} after {back if back is None else round(back, 1)}s => {st}"
)
return st
def upload_ok(tag, tries=4):
sts = []
for k in range(tries):
st = full_upload(f"{tag} try{k}")
sts.append(st)
if st != "INTERRUPTED":
break
time.sleep(15)
return sts
def cut_upload(frac, tag):
cur = current_ver()
nxt = "b" if cur == "a" else "a"
data = pathlib.Path(IMG[nxt]).read_bytes()
bnd = "----otacut1234"
head = (
f'--{bnd}\r\nContent-Disposition: form-data; name="update"; filename="firmware.bin"\r\n'
"Content-Type: application/octet-stream\r\n\r\n"
).encode()
tail = f"\r\n--{bnd}--\r\n".encode()
body_len = len(head) + len(data) + len(tail)
h0 = heap()
up0 = uptime()
s = socket.create_connection((HOST, 80), timeout=20)
s.sendall(
(
f"POST /update HTTP/1.1\r\nHost: {HOST}\r\nContent-Type: multipart/form-data; boundary={bnd}\r\n"
f"Content-Length: {body_len}\r\n\r\n"
).encode()
+ head
)
n = int(len(data) * frac)
sent = 0
try:
while sent < n:
k = s.send(data[sent : min(n, sent + 1460)])
sent += k
except Exception as e: # noqa: BLE001
p(f"{tag} send error {type(e).__name__} at {sent}")
time.sleep(0.5)
s.close()
# serving right after the cut
g = rest.req("/", timeout=8)[0]
ev = events_ok()
time.sleep(20)
up1 = uptime()
v1 = ver()
h1 = heap()
g2 = rest.req("/", timeout=8)[0]
ev2 = events_ok()
rebooted = up1 is None or up0 is None or up1 < up0
p(
f"{tag} cut {int(frac * 100)}% sent {sent}/{len(data)}: right after GET / {g} events {ev}; +20s GET / {g2} events {ev2}, "
f"ver {v1} (was {cur}), uptime {up0}->{up1} {'REBOOTED' if rebooted else 'no reboot'}, heap {h0}->{h1}"
)
return not rebooted and v1 == cur and g2 == 200 and ev2
class Sse(threading.Thread):
def __init__(self, reconnect):
super().__init__(daemon=True)
self.reconnect = reconnect
self.stop = False
self.states = 0
self.closes = 0
self.silent = 0
self.opens = 0
self.errors = 0
self.first_after_close = None
self.closed_at = None
def run(self):
while not self.stop:
try:
s = socket.create_connection((HOST, 80), timeout=3)
s.sendall(f"GET /events HTTP/1.1\r\nHost: {HOST}\r\n\r\n".encode())
self.opens += 1
s.settimeout(1)
last = time.monotonic()
while not self.stop:
try:
d = s.recv(4096)
except TimeoutError:
if time.monotonic() - last > 6:
self.silent += 1
self.closed_at = self.closed_at or time.monotonic()
break
continue
last = time.monotonic()
if not d:
self.closes += 1
self.closed_at = time.monotonic()
break
c = d.count(b"event: state")
self.states += c
if c and self.closed_at and self.first_after_close is None:
self.first_after_close = time.monotonic() - self.closed_at
s.close()
except (ConnectionResetError, BrokenPipeError):
self.closes += 1
self.closed_at = self.closed_at or time.monotonic()
except Exception: # noqa: BLE001
self.errors += 1
if not self.reconnect and (self.closes or self.silent):
return
time.sleep(1)
def sse_upload(tag):
a = Sse(False)
b = Sse(True)
a.start()
b.start()
time.sleep(5)
st0 = (a.states, b.states)
sts = upload_ok(tag + " (sse)")
ok = sts[-1] == "OK"
time.sleep(10)
a.stop = b.stop = True
time.sleep(2)
c = Sse(True)
c.start()
time.sleep(8)
c.stop = True
p(
f"{tag} sse: held client states before OTA {st0[0]}, total {a.states}, FIN/RST {a.closes}, went silent {a.silent}; "
f"reconnecting client opens {b.opens}, FIN/RST {b.closes}, silent {b.silent}, errors {b.errors}, events again {b.first_after_close if b.first_after_close is None else round(b.first_after_close, 1)}s after close; "
f"fresh client states in 8s {c.states}"
)
return (
ok
and (a.closes + a.silent) >= 1
and b.first_after_close is not None
and c.states > 0
)
case = sys.argv[5] if len(sys.argv) > 5 else "all"
res = {}
if case in ("all", "1"):
sts = [full_upload(f"c1#{i}") for i in range(REPS)]
while sts.count("OK") < REPS and len(sts) < 3 * REPS:
time.sleep(15)
sts.append(full_upload(f"c1#{len(sts)}"))
res["c1"] = {k: sts.count(k) for k in ("OK", "REPLY_LOST", "INTERRUPTED", "BAD")}
if case in ("all", "2"):
okn = 0
h_start = heap()
for i in range(REPS):
for f in (0.3, 0.8):
okn += cut_upload(f, f"c2#{i}")
time.sleep(10)
h_end = heap()
p(f"c2 heap start {h_start} end {h_end} over {2 * REPS} cuts")
after = upload_ok("c2 follow-up")
res["c2"] = okn
res["c2_followup"] = after
res["c2_heap"] = (h_start, h_end)
if case in ("all", "3"):
res["c3"] = sum(sse_upload(f"c3#{i}") for i in range(REPS))
# REPLY_LOST and INTERRUPTED are network losses seen on every build (ota.md): counted, not failed
fails = []
if "c1" in res and (res["c1"]["OK"] < REPS or res["c1"]["BAD"]):
fails.append("c1")
if "c2" in res and (res["c2"] < 2 * REPS or res["c2_followup"][-1] != "OK"):
fails.append("c2")
if "c3" in res and res["c3"] < REPS:
fails.append("c3")
res["fails"] = fails
p("RESULT", KIND, json.dumps(res))
sys.exit(1 if fails else 0)
@@ -0,0 +1,21 @@
#!/bin/bash
# otarun.sh NAME IP: native OTA of NAME (from TREE) to a board reached only over the network, then flow.py and the
# web OTA, with an API log capture throughout. env: TREE, PY
D=$(cd "$(dirname "$0")" && pwd); S=$(dirname "$(dirname "$D")"); N=${1:?name}; IP=${2:?ip}; T=${TREE:?set TREE}
PY=${PY:-$T/venv/bin/python}
PY=$PY PIO_CORE=${PIO_CORE:-$D/pio} bash "$S/pio_build.sh" upload "$N" "$D/$N.yaml" "$IP" || exit 1
for _ in $(seq 1 60); do curl -s -m 3 -o /dev/null "http://$IP/" && break; sleep 3; done
(cd "$T" && PYTHONPATH=$T PYTHONUNBUFFERED=1 timeout 1200 $PY -m esphome logs "$D/$N.yaml" --device "$IP" \
> "$D/$N-apilog.txt" 2>&1 &)
sleep 10
$PY -I "$D/flow.py" "$IP" "$D/$N"; RC=$?
B=$D/.esphome/build/$N/.pioenvs/$N; IMG=$B/firmware.bin; [ -f "$B/firmware.uf2" ] && IMG=$B/firmware.uf2
PY=$PY bash "$D/webota.sh" "$IP" "$IMG" "$D/$N" || RC=1
sleep 5; pkill -f "[e]sphome logs $D/$N.yaml"
CR=$(grep -a -c -E 'CRASH|Crash|abort\(\)|HardFault|assert' "$D/$N-apilog.txt")
[ "$CR" = 0 ] || RC=1
# an API log capture that never got a device log line saw nothing, so its zero crash count means nothing
grep -a -q -E '^\[[0-9:.]+\]\[[A-Z]\]' "$D/$N-apilog.txt" || { echo "APILOG has no device log lines"; RC=1; }
echo "APILOG crash lines $CR," \
"boots seen $(grep -a -c 'ESPHome version' "$D/$N-apilog.txt"), lines $(wc -l < "$D/$N-apilog.txt")"
exit $RC
@@ -0,0 +1,15 @@
#!/bin/bash
# presstest.sh IP : restart presses at uptime ~20 s and ~60 s; prints the HTTP code and whether the board restarted
IP=$1; D=$(cd "$(dirname "$0")" && pwd); PY=${PY:-python3}
up() { $PY -I "$D/rest.py" uptime "$IP" || echo -1; }
waitfor() { for i in $(seq 1 60); do u=$(up); [ $u -ge $1 ] && return; sleep 1; done; }
RC=0
for target in 20 20 20 60 60 60; do
waitfor $target; b=$(up)
r=$(curl -s -m 10 -X POST -o /dev/null -w "%{http_code} %{time_total}" http://$IP/button/Restart/press)
sleep 4; for i in $(seq 1 30); do a=$(up); [ $a -ge 0 ] && break; sleep 1; done
if [ $a -lt 0 ]; then v=NOT-back; elif [ $b -ge 0 ] && [ $a -lt $b ]; then v=restarted; else v=NOT-restarted; fi
echo "uptime $b press $r -> uptime $a $v"
[ $v = restarted ] || RC=1
done
exit $RC
@@ -0,0 +1,57 @@
"""REST helpers for the async_bump scripts: set HOST, then req(), jget(), jv(), uptime(), heap().
usage: rest.py uptime|heap HOST prints the whole seconds of uptime or the free heap; exits 1 with no answer.
"""
import json
import sys
import time
import urllib.error
import urllib.request
HOST = ""
def req(
path: str, method: str = "GET", timeout: float = 10
) -> tuple[int, bytes, float]:
"""(HTTP status or -1, body or the error name, seconds taken)."""
t = time.monotonic()
r = urllib.request.Request(
f"http://{HOST}{path}", method=method, data=b"" if method == "POST" else None
)
try:
with urllib.request.urlopen(r, timeout=timeout) as resp:
return resp.status, resp.read(), time.monotonic() - t
except urllib.error.HTTPError as e:
return e.code, e.read(), time.monotonic() - t
except Exception as e: # noqa: BLE001
return -1, type(e).__name__.encode(), time.monotonic() - t
def jget(path: str, timeout: float = 10) -> dict:
code, body, _ = req(path, timeout=timeout)
try:
return json.loads(body) if code == 200 else {"_code": code}
except ValueError:
return {"_code": code, "_bad": body[:80]}
def jv(path: str, key: str = "value", timeout: float = 10) -> object:
return jget(path, timeout).get(key)
def uptime() -> object:
return jv("/sensor/Uptime")
def heap() -> object:
return jv("/sensor/Heap%20free")
if __name__ == "__main__":
HOST = sys.argv[2]
v = {"uptime": uptime, "heap": heap}[sys.argv[1]]()
if v is None:
sys.exit(1)
print(int(v))
@@ -0,0 +1,34 @@
#!/bin/bash
# run.sh NAME KIND PORT: serial flash NAME (built from TREE into this directory), capture, flow.py, web OTA with its
# own image plus a restart, then crash and boot counts. KIND is esp, pico, bk or bw. env: TREE, PY
D=$(cd "$(dirname "$0")" && pwd); S=$(dirname "$(dirname "$D")"); N=${1:?name}; K=${2:?kind}; P=${3:?port}
T=${TREE:?set TREE}; PY=${PY:-$T/venv/bin/python}
B=$D/.esphome/build/$N/.pioenvs/$N
pkill -f "[s]erial_capture.py $P "; sleep 1
. "$D/ltflash.sh"
case $K in
esp | pico)
PIO_CORE=${PIO_CORE:-$D/pio} bash "$S/pio_build.sh" upload "$N" "$D/$N.yaml" "$P" || exit 1
IMG=$B/firmware.bin; OPT=() ;;
bk | bw)
ltflash "$K" "$P" "$B/firmware.uf2" "$D/$N-upload.log" || { echo "flash failed"; tail -2 "$D/$N-upload.log"; exit 1; }
# both kits wire RTS to reset: a capture that asserts it holds the board in reset
tail -2 "$D/$N-upload.log"; IMG=$B/firmware.uf2; OPT=(--released) ;;
*) echo "unknown kind $K"; exit 1 ;;
esac
[ "$K" = pico ] && sleep 6
: > "$D/$N-serial.log"; ($PY "$S/serial_capture.py" "$P" 1800 "$D/$N-serial.log" "${OPT[@]}" 2> "$D/$N-cap.err" &)
HOST=$(bash "$S/resolve.sh" "$N" 60) || { echo "no host"; exit 1; }
echo "$HOST" > "$D/$N-host.txt"; sleep 10
$PY -I "$D/flow.py" "$HOST" "$D/$N"; RC=$?
PY=$PY bash "$D/webota.sh" "$HOST" "$IMG" "$D/$N" || RC=1
sleep 5; pkill -f "[s]erial_capture.py $P "; sleep 1
L=$D/$N-serial.log
CR=$(grep -a -c -E 'Panic |Exception \(|Fatal exception|HardFault|Hard Fault|CRASH|abort\(\)|Soft WDT|wdt reset|assert|Backtrace' "$L")
[ "$CR" = 0 ] || RC=1
BOOTS=$(grep -a -c -E 'Log initialized|ESPHome version .* compiled on' "$L"); LOST=$(grep -a -c 'port lost' "$L")
echo "SERIAL crash lines $CR, boots $BOOTS, port losses $LOST, lines $(wc -l < "$L")"
# the web OTA and the restart press each reboot the board, so a capture with no boot saw nothing; a native USB
# port (Pico) can lose the boot banner, but then the capture shows the port going away
[ -s "$L" ] && [ $((BOOTS + LOST)) -gt 0 ] || { echo "SERIAL capture empty or shows no boot: $(tail -1 "$D/$N-cap.err")"; RC=1; }
exit $RC
@@ -0,0 +1,91 @@
"""ssediag.py HOST SECONDS [PAGE_S]: optional page-load phase, then SSE with a REST poll every 10 s.
Prints per 10 s: SSE fast-counter events, latest SSE counter value vs the REST value (lag), REST uptime."""
import json
import pathlib
import socket
import sys
import threading
import time
sys.path.insert(0, str(pathlib.Path(__file__).resolve().parent))
import rest # noqa: E402
host, secs = sys.argv[1], float(sys.argv[2])
page_s = float(sys.argv[3]) if len(sys.argv) > 3 else 0
rest.HOST = host
end = time.monotonic() + page_s
while time.monotonic() < end:
for p in ("/", "/metrics", "/sensor/Fast%20counter", "/light/Test%20light"):
rest.req(p, timeout=8)
time.sleep(4)
state = {"n": 0, "last": None, "ended": False}
def sse():
"""Reader thread; an error or a device close is printed and flagged, so it never reads as a stall."""
try:
sse_read()
except Exception as e: # noqa: BLE001
print(f"SSE FAILED {type(e).__name__}: {e}", flush=True)
state["ended"] = True
def sse_read():
s = socket.create_connection((host, 80), timeout=10)
s.sendall(
f"GET /events HTTP/1.1\r\nHost: {host}\r\nAccept: text/event-stream\r\n\r\n".encode()
)
s.settimeout(1)
buf = b""
ev = None
stop = time.monotonic() + secs
while time.monotonic() < stop:
try:
d = s.recv(8192)
except TimeoutError:
continue
if not d:
print("closed", flush=True)
state["ended"] = True
return
buf += d
while b"\n" in buf:
line, buf = buf.split(b"\n", 1)
line = line.rstrip(b"\r")
if line.startswith(b"event: "):
ev = line[7:]
elif (
line.startswith(b"data: ")
and ev == b"state"
and b"Fast counter" in line
):
state["n"] += 1
state["last"] = json.loads(line[6:]).get("value")
t = threading.Thread(target=sse, daemon=True)
t.start()
t0 = time.monotonic()
prev = 0
tot = []
while time.monotonic() - t0 < secs:
time.sleep(10)
rv = rest.jv("/sensor/Fast%20counter", timeout=8)
up = rest.jv("/sensor/Uptime", timeout=8)
n = state["n"] - prev
prev = state["n"]
tot.append(n)
lag = (rv - state["last"]) if rv is not None and state["last"] is not None else None
print(
f"t={time.monotonic() - t0:4.0f}s sse fast events {n:3d} sse last {state['last']} rest {rv} lag {lag} uptime {up}{' SSE ENDED' if state['ended'] else ''}",
flush=True,
)
print(
f"TOTAL sse fast events {sum(tot)} in {secs:.0f}s ({sum(tot) / secs:.1f}/s)"
+ (" (SSE ended early)" if state["ended"] else "")
)
# an SSE stream that ended early or carried nothing is not a measurement
sys.exit(1 if state["ended"] or not sum(tot) else 0)
@@ -0,0 +1,27 @@
#!/bin/bash
# webota.sh HOST IMAGE OUTPREFIX : web_server /update upload, wait for reboot, then a restart button press. A reboot
# counts only when the uptime drops below the reading taken before the operation; exits 1 when one is not seen.
H=$1; IMG=$2; O=$3; D=$(cd "$(dirname "$0")" && pwd); PY=${PY:-python3}
up() { $PY -I "$D/rest.py" uptime "$H"; }
# waitreset BEFORE AT: the uptime once the board booted after the reading BEFORE taken at epoch second AT (uptime now
# 5 s or more below BEFORE plus the time since); "none" and status 1 after 180 s or with no BEFORE
waitreset() {
for i in $(seq 1 90); do
u=$(up); [ -n "$u" ] && [ -n "$1" ] && [ "$u" -lt $(($1 + $(date +%s) - $2 - 5)) ] && { echo "$u"; return 0; }
sleep 2
done
echo none; return 1
}
RC=0
U0=$(up); T0=$(date +%s); echo "uptime before $U0" | tee $O-ota.txt
curl -s -m 600 -o $O-ota-resp.txt -w "ota http %{http_code} size %{size_upload} time %{time_total}\n" -F "update=@$IMG" http://$H/update | tee -a $O-ota.txt
echo "resp: $(head -c 120 $O-ota-resp.txt)" | tee -a $O-ota.txt
sleep 8; U=$(waitreset "$U0" $T0) || RC=1
echo "back after OTA (uptime $U) $(( $(date +%s) - T0 ))s" | tee -a $O-ota.txt
sleep 10; U1=$(up); T1=$(date +%s)
curl -s -m 10 -X POST -o /dev/null -w "restart press %{http_code}\n" http://$H/button/Restart/press | tee -a $O-ota.txt
sleep 8; U=$(waitreset "$U1" $T1) || RC=1
echo "back after restart (uptime $U, was $U1)" | tee -a $O-ota.txt
sleep 5; echo "heap after reboot $($PY -I "$D/rest.py" heap "$H")" | tee -a $O-ota.txt
[ $RC = 0 ] || echo "FAIL: no reset seen" | tee -a $O-ota.txt
exit $RC
@@ -0,0 +1,147 @@
"""Per outage summary for outage.py runs.
usage: an.py LABEL [LABEL...] (reads LABEL-marks.txt, LABEL-dut.txt, LABEL-rtr.txt)
Per outage (from RESTARTING_SEEN to the next DOWN or END):
skip the DUT entered RESTARTING with the fallback AP up and the restart was skipped (no "Restarting adapter")
cd_s seconds from that RESTARTING transition to the first state machine line after the cooldown
assoc the driver associated on its own during the cooldown (no "Connecting to" before the association)
ip_cd the address arrived before the cooldown ended
bug "Connected" followed by "Connection lost; reconnecting" within 5 s
held the first "Connected" was not followed by a loss within 60 s
rel DHCPRELEASE sent while the router was up (after UP)
rleave router saw the DUT leave after UP
"""
from datetime import datetime
import pathlib
import re
import sys
def t(s: str) -> float:
d = datetime.strptime(s, "%H:%M:%S.%f") # noqa: DTZ007
return d.hour * 3600 + d.minute * 60 + d.second + d.microsecond / 1e6
def load(path: str) -> list[tuple[float, str]]:
out = []
with pathlib.Path(path).open(encoding="utf-8", errors="replace") as f:
out.extend(
(t(line[:12]), line[13:].rstrip("\n"))
for line in f
if len(line) > 13 and line[2] == ":" and line[12] == " "
)
return out
CONNECTED = re.compile(r"\[I\]\[wifi:\d+\]: Connected$")
LOST = "Connection lost; reconnecting"
COOLDOWN_END = (
"Retry phase",
"Connecting to",
"Unknown connection",
"failed (callback)",
)
def ends_cooldown(line: str) -> bool:
"""A state machine line that only comes after the cooldown; outage.py waits for it too."""
return bool(CONNECTED.search(line)) or any(x in line for x in COOLDOWN_END)
def analyze(label: str) -> list[dict]:
marks = load(f"{label}-marks.txt")
dut = load(f"{label}-dut.txt")
rtr = load(f"{label}-rtr.txt")
rows = []
downs = [(ts, m) for ts, m in marks if " DOWN" in m or m.startswith("MARK END")]
for ts, m in marks:
g = re.match(r"MARK OUTAGE (\d+) RESTARTING_SEEN", m)
if not g:
continue
n = int(g.group(1))
up = next((u for u, mm in marks if mm == f"MARK OUTAGE {n} UP"), ts)
end = next((d for d, _ in downs if d > ts), dut[-1][0] + 1)
win = [(a, b) for a, b in dut if ts - 3 <= a < end]
# the RESTARTING transition the harness reacted to
rst = max(
(a for a, b in win if "→ RESTARTING" in b and a <= ts + 0.5), default=ts
)
after = [(a, b) for a, b in win if a > rst]
skip = not any("Restarting adapter" in b for a, b in after if a - rst < 1.0)
# first state machine line after the cooldown
cd_end = next(
(a for a, b in after if a - rst > 1.0 and ends_cooldown(b)),
None,
)
first_connecting = next((a for a, b in after if "Connecting to" in b), 1e12)
assoc_t = next(
(a for a, b in after if "wifi_lt" in b and "Connected ssid=" in b), None
)
assoc = (
assoc_t is not None
and assoc_t < first_connecting
and (cd_end is None or assoc_t < cd_end)
)
ip_t = next((a for a, b in after if "static_ip=" in b), None)
ip_cd = ip_t is not None and cd_end is not None and ip_t < cd_end
conns = [a for a, b in after if CONNECTED.search(b)]
losts = [a for a, b in after if LOST in b]
bug = sum(1 for c in conns if any(0 <= lost - c <= 5 for lost in losts))
held = bool(conns) and not any(0 <= lost - conns[0] <= 60 for lost in losts)
rel = sum(1 for a, b in win if a >= up and "DHCPRELEASE sent" in b)
rleave = sum(
1 for a, b in rtr if up <= a < end and "AP client disconnected" in b
)
rows.append(
{
"label": label,
"n": n,
"skip": skip,
"cd_s": round(cd_end - rst, 1) if cd_end else None,
"up_s": round(up - rst, 1),
"assoc": assoc,
"ip_cd": ip_cd,
"bug": bug,
"held": held,
"rel": rel,
"rleave": rleave,
}
)
return rows
def main() -> None:
allrows = []
for label in sys.argv[1:]:
allrows += analyze(label)
if not allrows:
sys.exit(f"no RESTARTING_SEEN outage in {' '.join(sys.argv[1:])}")
keys = [
"label",
"n",
"skip",
"up_s",
"cd_s",
"assoc",
"ip_cd",
"bug",
"held",
"rel",
"rleave",
]
print("\t".join(keys))
for r in allrows:
print("\t".join(str(r[k]) for k in keys))
print(
f"TOTAL outages={len(allrows)} skip={sum(r['skip'] for r in allrows)} "
f"returned_in_cooldown={sum(1 for r in allrows if r['skip'] and r['cd_s'] and r['up_s'] < r['cd_s'])} "
f"assoc_in_cooldown={sum(r['assoc'] for r in allrows)} ip_in_cooldown={sum(r['ip_cd'] for r in allrows)} "
f"bug={sum(r['bug'] for r in allrows)} held={sum(r['held'] for r in allrows)} "
f"rel={sum(r['rel'] for r in allrows)} rleave={sum(r['rleave'] for r in allrows)}"
)
if __name__ == "__main__":
main()
@@ -0,0 +1,35 @@
esphome:
name: ${name}
esp32:
variant: esp32c3
board: esp32-c3-devkitm-1
framework:
type: esp-idf
logger:
hardware_uart: USB_SERIAL_JTAG
level: VERBOSE
logs:
wifi: DEBUG
api: DEBUG
api.connection: DEBUG
component: DEBUG
scheduler: DEBUG
app: DEBUG
ota: DEBUG
mdns: DEBUG
preferences: DEBUG
api:
reboot_timeout: 0s
ota:
- platform: esphome
wifi:
networks:
- ssid: !secret t_ssid
password: !secret t_pw
reboot_timeout: 0s
interval:
- interval: 1s
then:
- lambda: |-
ESP_LOGI("alive", "ALIVE up=%u conn=%d", (unsigned) (millis() / 1000),
(int) wifi::global_wifi_component->is_connected());
@@ -0,0 +1,50 @@
#pragma once
// Counts DHCP messages the station's lwIP sends (dhcp.c calls these across objects, so --wrap sees them).
#include <lwip/udp.h>
#include <lwip/pbuf.h>
#include <lwip/netif.h>
extern "C" {
volatile uint32_t g_dhcp_disc = 0, g_dhcp_req = 0, g_dhcp_rel = 0, g_dhcp_rel_up = 0;
err_t __real_udp_sendto_if(struct udp_pcb *pcb, struct pbuf *p, const ip_addr_t *dst_ip, u16_t dst_port,
struct netif *netif);
err_t __real_udp_sendto_if_src(struct udp_pcb *pcb, struct pbuf *p, const ip_addr_t *dst_ip, u16_t dst_port,
struct netif *netif, const ip_addr_t *src_ip);
static void dhcpwrap_count(struct pbuf *p, u16_t dst_port, struct netif *netif) {
if (dst_port != 67 || p == nullptr || p->len < 243)
return;
const uint8_t *b = (const uint8_t *) p->payload;
for (uint16_t i = 240; i + 2 < p->len;) {
uint8_t opt = b[i];
if (opt == 255)
break;
if (opt == 0) {
i++;
continue;
}
if (opt == 53) {
uint8_t t = b[i + 2];
if (t == 1)
g_dhcp_disc++;
else if (t == 3)
g_dhcp_req++;
else if (t == 7) {
g_dhcp_rel++;
if (netif != nullptr && netif_is_up(netif) && netif_is_link_up(netif))
g_dhcp_rel_up++;
}
break;
}
i += 2 + b[i + 1];
}
}
err_t __wrap_udp_sendto_if(struct udp_pcb *pcb, struct pbuf *p, const ip_addr_t *dst_ip, u16_t dst_port,
struct netif *netif) {
dhcpwrap_count(p, dst_port, netif);
return __real_udp_sendto_if(pcb, p, dst_ip, dst_port, netif);
}
err_t __wrap_udp_sendto_if_src(struct udp_pcb *pcb, struct pbuf *p, const ip_addr_t *dst_ip, u16_t dst_port,
struct netif *netif, const ip_addr_t *src_ip) {
dhcpwrap_count(p, dst_port, netif);
return __real_udp_sendto_if_src(pcb, p, dst_ip, dst_port, netif, src_ip);
}
}
@@ -0,0 +1,58 @@
esphome:
name: ${name}
includes:
- dhcpwrap.hpp
build_flags:
- -Wl,--wrap=udp_sendto_if
- -Wl,--wrap=udp_sendto_if_src
on_boot:
priority: 800
then:
- logger.log: "MARK BOOT ${name}"
bk72xx:
board: xh-wb3s
logger:
hardware_uart: UART1
level: VERBOSE
logs:
api: DEBUG
api.connection: DEBUG
api.service: DEBUG
component: DEBUG
app: DEBUG
scheduler: DEBUG
preferences: DEBUG
captive_portal: DEBUG
web_server_base: DEBUG
dns_server: DEBUG
mdns: DEBUG
ota: DEBUG
api:
reboot_timeout: 0s
ota:
- platform: esphome
wifi:
networks:
- ssid: !secret t_ssid
password: !secret t_pw
ap:
ssid: ${name}-fb
password: !secret fb_pw
ap_timeout: 10s
reboot_timeout: 0s
captive_portal:
interval:
- interval: 1s
then:
- lambda: |-
ESP_LOGI("alive", "ALIVE up=%u conn=%d disc=%u req=%u rel=%u relup=%u", (unsigned) (millis() / 1000),
(int) wifi::global_wifi_component->is_connected(), (unsigned) g_dhcp_disc,
(unsigned) g_dhcp_req, (unsigned) g_dhcp_rel, (unsigned) g_dhcp_rel_up);
- interval: 100ms
then:
- lambda: |-
static uint32_t last = 0;
if (g_dhcp_rel != last) {
last = g_dhcp_rel;
ESP_LOGW("dhcpw", "DHCPRELEASE sent n=%u up=%u", (unsigned) last, (unsigned) g_dhcp_rel_up);
}
@@ -0,0 +1,180 @@
"""Router outage harness: a D1 mini soft AP is the DUT's only network; hold it in reset to make outages.
usage: outage.py DUT_PORT RTR_PORT LABEL OUTAGES [--release-delay S] [--stable S]
Each outage: wait until the DUT has been connected for --stable seconds, hold the router in reset (RTS on EN),
wait until the DUT enters RESTARTING with its fallback AP already up (the portal is active, so this is the
skip-restart branch and a 30 s COOLDOWN follows), wait --release-delay seconds and let the router boot again,
so the AP returns inside the cooldown. Writes LABEL-dut.txt, LABEL-rtr.txt and LABEL-marks.txt (all lines
prefixed with a wall clock timestamp) for an.py.
"""
import argparse
import pathlib
import re
import sys
import threading
import time
import serial
sys.path.insert(0, str(pathlib.Path(__file__).resolve().parent))
from an import ends_cooldown # noqa: E402
ANSI = re.compile(rb"\x1b\[[0-9;]*m")
def ts() -> str:
t = time.time()
return time.strftime("%H:%M:%S", time.localtime(t)) + f".{int(t * 1000) % 1000:03d}"
class Reader(threading.Thread):
def __init__(self, ser: serial.Serial, path: str) -> None:
super().__init__(daemon=True)
self.ser = ser
self.f = pathlib.Path(path).open("w", encoding="utf-8", buffering=1) # noqa: SIM115
self.lines: list[tuple[float, str]] = []
self.lock = threading.Lock()
self.stop = False
def run(self) -> None:
buf = b""
while not self.stop:
try:
chunk = self.ser.read(512)
except Exception as e: # noqa: BLE001
self.f.write(f"{ts()} READ ERROR {e}\n")
time.sleep(0.5)
continue
if not chunk:
continue
buf += chunk
while b"\n" in buf:
raw, buf = buf.split(b"\n", 1)
line = ANSI.sub(b"", raw).decode("utf-8", "replace").rstrip("\r")
now = time.time()
self.f.write(f"{ts()} {line}\n")
with self.lock:
self.lines.append((now, line))
def after(self, idx: int) -> tuple[list[str], int]:
"""Lines from index idx on, and the index to pass next time."""
with self.lock:
return [line for _, line in self.lines[idx:]], len(self.lines)
def main() -> None:
ap = argparse.ArgumentParser()
ap.add_argument("dut")
ap.add_argument("rtr")
ap.add_argument("label")
ap.add_argument("outages", type=int)
ap.add_argument("--release-delay", type=float, default=1.0)
ap.add_argument("--stable", type=float, default=20.0)
ap.add_argument("--max-outage", type=float, default=300.0)
ap.add_argument(
"--no-ap",
action="store_true",
help="DUT has no fallback AP: react to 'Restarting adapter'",
)
a = ap.parse_args()
marks = pathlib.Path(f"{a.label}-marks.txt").open( # noqa: SIM115
"w", encoding="utf-8", buffering=1
)
def mark(msg: str) -> None:
line = f"{ts()} MARK {msg}"
marks.write(line + "\n")
print(line, flush=True)
# Router first: open with both lines released so it runs; opening may reset it once
rtr = serial.Serial()
rtr.port, rtr.baudrate, rtr.timeout = a.rtr, 115200, 0.2
rtr.dtr = False
rtr.rts = False
rtr.open()
rtr.dtr = False
rtr.rts = False
dut = serial.Serial(a.dut, 115200, timeout=0.2) # opening resets the BK7238
rr = Reader(rtr, f"{a.label}-rtr.txt")
dr = Reader(dut, f"{a.label}-dut.txt")
rr.start()
dr.start()
mark(f"START label={a.label} outages={a.outages} release_delay={a.release_delay}")
def connected_for(seconds: float, cap: float) -> bool:
"""Wait until the last `seconds` of ALIVE lines all read conn=1."""
t_end = time.time() + cap
run_start = None
while time.time() < t_end:
time.sleep(0.5)
with dr.lock:
alive = [(t, ln) for t, ln in dr.lines[-40:] if "ALIVE" in ln]
if not alive:
continue
t, ln = alive[-1]
if "conn=1" in ln:
if run_start is None:
run_start = t
if t - run_start >= seconds:
return True
else:
run_start = None
return False
failed = False
for n in range(1, a.outages + 1):
if not connected_for(a.stable, 400):
mark(f"OUTAGE {n} SKIP dut never stable; releasing router and retrying")
rtr.rts = False
if not connected_for(a.stable, 300):
mark("ABORT dut not connecting")
failed = True
break
t0 = time.time()
_, idx = dr.after(0)
mark(f"OUTAGE {n} DOWN")
rtr.rts = True # EN low: router held in reset
ap_up = False
hit = False
# the RESTARTING transition after the fallback AP started
while not hit and time.time() - t0 < a.max_outage:
time.sleep(0.05)
new, idx = dr.after(idx)
for ln in new:
if "Starting fallback AP" in ln:
ap_up = True
if (ap_up or a.no_ap) and (
"Restarting adapter" in ln if a.no_ap else "→ RESTARTING" in ln
):
hit = True
break
if not hit:
mark(f"OUTAGE {n} NO_RESTARTING ap_up={ap_up}; releasing")
failed = True
rtr.rts = False
continue
mark(f"OUTAGE {n} RESTARTING_SEEN ap_up={ap_up}")
time.sleep(a.release_delay)
rtr.rts = False
mark(f"OUTAGE {n} UP")
# let the cooldown end (an.py's end of cooldown lines) before the next stable check
t_up = time.time()
ended = False
while not ended and time.time() - t_up < 40:
time.sleep(0.2)
new, idx = dr.after(idx)
ended = any(ends_cooldown(ln) for ln in new)
mark("END")
rtr.rts = False
time.sleep(2)
rr.stop = dr.stop = True
time.sleep(0.5)
# an aborted run or an outage that never reached RESTARTING has fewer outages than asked for
sys.exit(1 if failed else 0)
if __name__ == "__main__":
main()
@@ -0,0 +1,37 @@
esphome:
name: rtr-outage
on_boot:
priority: 800
then:
- logger.log: "MARK RTR BOOT"
esp8266:
board: d1_mini
logger:
level: VERBOSE
logs:
component: DEBUG
scheduler: DEBUG
app: DEBUG
wifi:
ap:
ssid: !secret t_ssid
password: !secret t_pw
interval:
- interval: 1s
then:
- lambda: |-
char buf[160];
size_t n = 0;
int cnt = 0;
struct station_info *st = wifi_softap_get_station_info();
while (st != nullptr && n < sizeof(buf) - 40) {
uint32_t a = st->ip.addr;
n += snprintf(buf + n, sizeof(buf) - n, " %02X:%02X=%u.%u.%u.%u", st->bssid[4], st->bssid[5],
(unsigned) (a & 0xff), (unsigned) ((a >> 8) & 0xff), (unsigned) ((a >> 16) & 0xff),
(unsigned) (a >> 24));
cnt++;
st = STAILQ_NEXT(st, next);
}
wifi_softap_free_station_info();
buf[n] = 0;
ESP_LOGI("rtr", "RTR up=%u sta=%d%s", (unsigned) (millis() / 1000), cnt, buf);
@@ -0,0 +1,84 @@
"""an15.py BUILD STA_MAC: per reboot, Disassociation copies the AP sniffer saw (grouped by sequence number within one
burst), burst length, and the station's reboot kind and join time."""
import pathlib
import re
import sys
# copies of one frame arrive within tens of ms; reboots are 10 s or more apart, and the sequence counter restarts
BURST_GAP_US = 2_000_000
b, mac = sys.argv[1], sys.argv[2].upper()
if not re.fullmatch(r"(?:[0-9A-F]{2}:){5}[0-9A-F]{2}", mac):
sys.exit(f"STA_MAC must look like AA:BB:CC:DD:EE:FF, got {mac!r}")
paths = [pathlib.Path(f"{b}-ap.txt"), pathlib.Path(f"{b}-st.txt")]
if missing := [str(x) for x in paths if not x.is_file()]:
sys.exit(f"missing capture {', '.join(missing)}")
ap, st = (x.read_text(encoding="utf-8", errors="replace").splitlines() for x in paths)
# zero Disassociation frames is a valid result (deep sleep sends none), but not with a dead sniffer, a station the AP
# never saw or a station that never rebooted
if not any("MGMT " in line for line in ap):
sys.exit(f"{b}: no MGMT lines in the AP capture; the sniffer did not run")
if not any(mac in line for line in ap):
sys.exit(f"{b}: the AP capture never mentions {mac}; check the station MAC")
if not any("REBOOT" in line for line in st):
sys.exit(f"{b}: no REBOOT line in the station capture")
groups: list[tuple[str, str, list[tuple[int, int, int]]]] = []
open_groups: dict[tuple[str, str], list[tuple[int, int, int]]] = {}
for line in ap:
m = re.search(
r"MGMT (DISASSOC|DEAUTH) ta (\S+) .* seq (\d+) retry (\d) prot (\d) .* t_us (\d+)",
line,
)
if not m or m.group(2) != mac:
continue
key = (m.group(1), m.group(3))
rec = (int(m.group(6)), int(m.group(4)), int(m.group(5)))
g = open_groups.get(key)
if g is None or abs(rec[0] - g[-1][0]) > BURST_GAP_US:
g = open_groups[key] = []
groups.append((*key, g))
g.append(rec)
leaves = [
m.group(1)
for line in ap
if "STA LEAVE" in line and mac in line and (m := re.search(r"reason (\d+)", line))
]
kinds = [
m.group(1)
for line in st
if "[st" in line and (m := re.search(r"REBOOT (\w+)", line))
]
joins = [
int(m.group(1)) for line in st if (m := re.search(r"JOINED after (\d+) ms", line))
]
disc = sum("Disconnecting before powerdown" in line for line in st)
# the stations reboot on purpose (SW_CPU, deep sleep) and log warnings while they rejoin; error lines, crash
# markers and any other reset reason (watchdog, panic, brownout) are device trouble on either side
CRASH = re.compile(r"\[E\]\[|Guru|panic|abort\(\)|Backtrace|WDT|Brownout")
OK_RESET = re.compile(
r"rst:0x[0-9a-f]+ \((?:SW_CPU(?:_RESET)?|POWERON(?:_RESET)?|DEEPSLEEP(?:_RESET)?|DSLEEP)\)"
)
bad_lines = [
f"{side}: {line.strip()}"
for side, lines in (("ap", ap), ("st", st))
for line in lines
if CRASH.search(line) or ("rst:" in line and not OK_RESET.search(line))
]
crashes = len(bad_lines)
copies = []
for kind, seq, g in groups:
t = [x[0] for x in g]
copies.append(len(g))
print(
f" {kind} seq {seq}: {len(g)} copies, {sum(x[1] for x in g)} with retry bit, prot {g[0][2]}, "
f"over {(max(t) - min(t)) / 1000:.1f} ms"
)
print(
f"{b}: reboots logged {len(kinds)} ({kinds.count('forced')} forced, {kinds.count('safe')} safe), "
f"disassoc frames {len(groups)} copies {copies}, AP leave reasons {leaves}, "
f"'Disconnecting before powerdown' {disc}, joins ms {joins}, crash lines {crashes}"
)
for line in bad_lines[:20]:
print(" BAD", line[:160])
sys.exit(1 if bad_lines else 0)
@@ -0,0 +1,15 @@
# Station: deep sleeps for 5 s, 10 s after connecting, then wakes and rejoins.
packages:
base: !include st-base.yaml
wifi:
on_connect:
- lambda: ESP_LOGI("st", "JOINED after %u ms since boot", (unsigned) millis());
- delay: 10s
- lambda: ESP_LOGI("st", "REBOOT deepsleep");
- deep_sleep.enter:
id: ds
sleep_duration: 5s
deep_sleep:
id: ds
+18
View File
@@ -0,0 +1,18 @@
#!/bin/bash
# usage: run.sh BUILDNAME TREE SECONDS flash BUILDNAME.yaml (a station config built on st-common.yaml,
# stearly-common.yaml or ds-common.yaml) from TREE to the station, then capture station and AP logs with millisecond
# stamps.
# env: ST (station port), AP (access point port), PY
W=$(cd "$(dirname "$0")" && pwd); S=$(dirname "$W"); cd "$W" || exit 1; PY=${PY:-$2/venv/bin/python}
ST=${ST:?station port}; AP=${AP:?access point port}
pkill -f "[s]erial_capture.py $ST "; pkill -f "[s]erial_capture.py $AP "
TREE=$2 bash "$S/pio_build.sh" upload "$1" "$W/$1.yaml" "$ST" || exit 1
$PY "$S/serial_capture.py" "$AP" "$3" "$1-ap.txt" --ms > /dev/null 2> "$1-ap.err" & PA=$!
$PY "$S/serial_capture.py" "$ST" "$3" "$1-st.txt" --ms > /dev/null 2> "$1-st.err" & PS=$!
RC=0
for k in ap st; do
if [ $k = ap ]; then wait $PA; else wait $PS; fi || { echo "$k capture failed: $(tail -1 "$1-$k.err")"; RC=1; }
[ -s "$1-$k.txt" ] || { echo "$k capture is empty"; RC=1; }
done
echo "$1 done"
exit $RC
@@ -0,0 +1,89 @@
#pragma once
#include <esp_wifi.h>
#include <esp_event.h>
#include <esp_timer.h>
#include <atomic>
// Ring of management frames seen by the AP's sniffer: assoc req (0x00), disassoc (0xA0), auth (0xB0), deauth (0xC0).
struct SniffRec {
int64_t t_us;
uint8_t fc0, fc1;
uint16_t seq;
uint8_t a1[6], a2[6];
int8_t rssi;
uint16_t len;
};
static SniffRec s_ring[256];
static std::atomic<uint32_t> s_head{0};
// single producer (sniff_cb on the Wi-Fi task), single consumer (sniff_drain on the main loop)
static std::atomic<uint32_t> s_tail{0};
static std::atomic<uint32_t> s_drops{0};
static void sniff_cb(void *buf, wifi_promiscuous_pkt_type_t type) {
if (type != WIFI_PKT_MGMT)
return;
auto *p = (wifi_promiscuous_pkt_t *) buf;
const uint8_t *f = p->payload;
uint8_t st = f[0] & 0xF0;
if (st != 0x00 && st != 0xA0 && st != 0xB0 && st != 0xC0)
return;
uint32_t h = s_head.load(std::memory_order_relaxed);
if (h - s_tail.load(std::memory_order_acquire) >= 256) {
s_drops++;
return;
}
SniffRec &r = s_ring[h & 255];
r.t_us = esp_timer_get_time();
r.fc0 = f[0];
r.fc1 = f[1];
memcpy(r.a1, f + 4, 6);
memcpy(r.a2, f + 10, 6);
r.seq = (f[22] | (f[23] << 8)) >> 4;
r.rssi = p->rx_ctrl.rssi;
r.len = p->rx_ctrl.sig_len;
s_head.store(h + 1, std::memory_order_release);
}
static void sniff_start() {
wifi_promiscuous_filter_t flt{};
flt.filter_mask = WIFI_PROMIS_FILTER_MASK_MGMT;
esp_err_t e_flt = esp_wifi_set_promiscuous_filter(&flt);
esp_err_t e_cb = esp_wifi_set_promiscuous_rx_cb(sniff_cb);
esp_err_t err = esp_wifi_set_promiscuous(true);
if (e_flt != ESP_OK || e_cb != ESP_OK || err != ESP_OK) {
ESP_LOGE("wdap", "SNIFFER start failed: filter %d callback %d enable %d", e_flt, e_cb, err);
} else {
ESP_LOGI("wdap", "SNIFFER started");
}
}
static void sniff_drain() {
const uint32_t h = s_head.load(std::memory_order_acquire);
for (uint32_t t = s_tail.load(std::memory_order_relaxed); t != h; t++) {
const SniffRec &r = s_ring[t & 255];
const char *kind = "?";
switch (r.fc0 & 0xF0) {
case 0x00:
kind = "ASSOCREQ";
break;
case 0xA0:
kind = "DISASSOC";
break;
case 0xB0:
kind = "AUTH";
break;
case 0xC0:
kind = "DEAUTH";
break;
}
ESP_LOGI("wdap",
"MGMT %s ta %02X:%02X:%02X:%02X:%02X:%02X ra %02X:%02X:%02X:%02X:%02X:%02X seq %u retry %u prot %u len %u "
"rssi %d t_us %lld",
kind, r.a2[0], r.a2[1], r.a2[2], r.a2[3], r.a2[4], r.a2[5], r.a1[0], r.a1[1], r.a1[2], r.a1[3], r.a1[4],
r.a1[5], r.seq, (r.fc1 >> 3) & 1, (r.fc1 >> 6) & 1, r.len, r.rssi, (long long) r.t_us);
s_tail.store(t + 1, std::memory_order_release);
}
uint32_t d = s_drops.exchange(0);
if (d)
ESP_LOGW("wdap", "SNIFF DROPS %u", d);
}
@@ -0,0 +1,66 @@
# WPA3-SAE only AP with PMF required on any ESP32 (set `board:`, an ESP32-C6 devkit here); logs station join and leave
# events and the management frames its sniffer sees (assoc req, auth, disassoc, deauth) with retry bit and sequence
# number.
esphome:
name: sniffap
includes:
- sniff_inc.hpp
on_boot:
priority: -100
then:
- delay: 3s
- lambda: |-
wifi_config_t c{};
esp_err_t err = esp_wifi_get_config(WIFI_IF_AP, &c);
if (err == ESP_OK) {
c.ap.authmode = WIFI_AUTH_WPA3_PSK;
c.ap.pmf_cfg.capable = true;
c.ap.pmf_cfg.required = true;
c.ap.sae_pwe_h2e = WPA3_SAE_PWE_BOTH;
c.ap.max_connection = 4;
err = esp_wifi_set_config(WIFI_IF_AP, &c);
}
wifi_config_t check{};
if (err == ESP_OK)
err = esp_wifi_get_config(WIFI_IF_AP, &check);
if (err != ESP_OK || check.ap.authmode != WIFI_AUTH_WPA3_PSK || !check.ap.pmf_cfg.required) {
ESP_LOGE("wdap", "AP CONFIG FAILED: %d authmode %d", err, (int) check.ap.authmode);
} else {
ESP_LOGI("wdap", "AP set to WPA3-SAE, PMF required, channel %u: %d", check.ap.channel, err);
}
err = esp_event_handler_register(WIFI_EVENT, ESP_EVENT_ANY_ID, [](void *, esp_event_base_t, int32_t id, void *data) {
if (id == WIFI_EVENT_AP_STACONNECTED) {
auto *e = (wifi_event_ap_staconnected_t *) data;
ESP_LOGI("wdap", "STA JOIN %02X:%02X:%02X:%02X:%02X:%02X aid %u t_us %lld", e->mac[0], e->mac[1], e->mac[2], e->mac[3], e->mac[4], e->mac[5], e->aid, (long long) esp_timer_get_time());
} else if (id == WIFI_EVENT_AP_STADISCONNECTED) {
auto *e = (wifi_event_ap_stadisconnected_t *) data;
ESP_LOGI("wdap", "STA LEAVE %02X:%02X:%02X:%02X:%02X:%02X aid %u reason %u t_us %lld", e->mac[0], e->mac[1], e->mac[2], e->mac[3], e->mac[4], e->mac[5], e->aid, e->reason, (long long) esp_timer_get_time());
}
}, nullptr);
// without the handler no STA JOIN/LEAVE line is logged, which an15.py would read as no leaves
if (err != ESP_OK)
ESP_LOGE("wdap", "STA event handler register failed: %d", err);
sniff_start();
- delay: 5400s
- lambda: ESP_LOGI("wdap", "AP WINDOW OVER");
- wifi.disable:
esp32:
board: esp32-c6-devkitc-1
framework:
type: esp-idf
logger:
level: DEBUG
wifi:
ap:
ssid: w15pmf
password: !secret wd_ap_password
channel: 11
ap_timeout: 0s
interval:
- interval: 50ms
then:
- lambda: sniff_drain();
@@ -0,0 +1,23 @@
# Station base for st-common.yaml, stearly-common.yaml and ds-common.yaml: joins the WPA3-only AP of sniffap.yaml.
esphome:
name: ${name}
esp32:
board: ${board}
framework:
type: esp-idf
logger:
level: DEBUG
globals:
- id: boots
type: int
restore_value: true
initial_value: "0"
wifi:
ssid: w15pmf
password: !secret wd_ap_password
min_auth_mode: WPA2
power_save_mode: none
@@ -0,0 +1,18 @@
# Station: joins the WPA3-only AP and alternates App.reboot() and App.safe_reboot() 10 s after connecting.
packages:
base: !include st-base.yaml
wifi:
on_connect:
- lambda: ESP_LOGI("st", "JOINED after %u ms since boot", (unsigned) millis());
- delay: 10s
- lambda: |-
id(boots) += 1;
global_preferences->sync();
const bool forced = id(boots) % 2;
ESP_LOGI("st", "REBOOT %s n=%d", forced ? "forced" : "safe", id(boots));
if (forced) {
App.reboot();
} else {
App.safe_reboot();
}
@@ -0,0 +1,18 @@
#pragma once
#include <esp_wifi.h>
#include <esp_event.h>
#include <esp_timer.h>
#include <atomic>
#include "esphome/core/application.h"
// STA_CONNECTED time from the event task; the handler wakes the main loop so on_loop sees it on the next pass.
static std::atomic<int64_t> s_conn_us{0};
static void st_register() {
esp_err_t e = esp_event_handler_register(
WIFI_EVENT, WIFI_EVENT_STA_CONNECTED,
[](void *, esp_event_base_t, int32_t, void *) {
s_conn_us.store(esp_timer_get_time());
esphome::App.wake_loop_threadsafe();
},
nullptr);
ESP_LOGI("st", "handler register: %d", e);
}
@@ -0,0 +1,35 @@
# Station: reboots on the first loop pass after WIFI_EVENT_STA_CONNECTED, before the Wi-Fi component's loop has handled
# the connect event (the window where a flag set from the main loop lags). The event handler wakes the loop and
# on_loop runs before the Wi-Fi component's loop in that pass. Alternates App.reboot()/safe_reboot().
packages:
base: !include st-base.yaml
esphome:
includes:
- st_inc.hpp
on_boot:
priority: 200
then:
- lambda: st_register();
on_loop:
- lambda: |-
int64_t c = s_conn_us.load();
if (id(fired) || c == 0)
return;
id(fired) = true;
id(boots) += 1;
global_preferences->sync();
const bool forced = id(boots) % 2;
ESP_LOGI("st", "EARLY REBOOT %s n=%d %lld us after STA_CONNECTED, component connected=%d",
forced ? "forced" : "safe", id(boots), (long long) (esp_timer_get_time() - c),
wifi::global_wifi_component->is_connected());
if (forced) {
App.reboot();
} else {
App.safe_reboot();
}
globals:
- id: fired
type: bool
initial_value: "false"
@@ -0,0 +1,122 @@
#pragma once
// esphome#19805 diagnostics: classify lwIP's global DNS table against each netif's own cache,
// resolve a fresh hostname, and power the W5500 PHY down/up from software.
#include "esphome/core/log.h"
#include "esphome/components/ethernet/ethernet_component.h"
#include "esphome/components/wifi/wifi_component.h"
#include "esp_netif.h"
#include "esp_netif_net_stack.h"
#include "esphome/core/helpers.h"
#include "esp_eth_driver.h"
#include "lwip/netif.h"
#include "lwip/netdb.h"
#include <cstring>
namespace dnsdiag {
static const char *const TAG = "dnsdiag";
static bool eq(const esp_ip_addr_t &a, const esp_ip_addr_t &b) {
return a.type == b.type && memcmp(&a.u_addr, &b.u_addr, a.type == ESP_IPADDR_TYPE_V4 ? 4 : 16) == 0;
}
static bool is_zero(const esp_ip_addr_t &a) {
static constexpr uint8_t Z[16] = {0};
return memcmp(&a.u_addr, Z, a.type == ESP_IPADDR_TYPE_V4 ? 4 : 16) == 0;
}
static esp_ip_addr_t dns(esp_netif_t *n, esp_netif_dns_type_t t) {
esp_netif_dns_info_t d{};
// NULL reads lwIP's global table; a failed read is logged so it is not mistaken for an empty server
esp_err_t err = esp_netif_get_dns_info(n, t, &d);
if (err != ESP_OK)
ESP_LOGW(TAG, "DNS read failed on %s: %s", n ? esp_netif_get_desc(n) : "global", esp_err_to_name(err));
return d.ip;
}
// label: 0 = empty, E = eth's, W = wifi's, B = both (same server), X = something else
static char label(const esp_ip_addr_t &g, esp_netif_t *eth, esp_netif_t *sta, esp_netif_dns_type_t t) {
if (is_zero(g))
return '0';
bool e = eth && eq(g, dns(eth, t)), w = sta && eq(g, dns(sta, t));
return e && w ? 'B' : e ? 'E' : w ? 'W' : 'X';
}
static const char *desc(esp_netif_t *n) { return n ? esp_netif_get_desc(n) : "none"; }
inline void print() {
esp_netif_t *eth = esphome::ethernet::global_eth_component->get_esp_netif();
esp_netif_t *sta = esphome::wifi::global_wifi_component->get_esp_netif_sta();
esp_ip_addr_t g0 = dns(nullptr, ESP_NETIF_DNS_MAIN), g1 = dns(nullptr, ESP_NETIF_DNS_BACKUP);
const char *lw = "none";
int link = -1;
{
esphome::LwIPLock lock;
struct netif *nd = netif_default;
if (eth && esp_netif_get_netif_impl(eth))
link = netif_is_link_up((struct netif *) esp_netif_get_netif_impl(eth)) ? 1 : 0;
if (nd != nullptr) {
if (eth && nd == esp_netif_get_netif_impl(eth))
lw = "eth";
else if (sta && nd == esp_netif_get_netif_impl(sta))
lw = "sta";
else
lw = "other";
}
}
ESP_LOGI(
TAG,
"DIAG g0=%c g1=%c eth.dns0=%s sta.dns0=%s eth_link=%d eth_conn=%d sta_conn=%d esp_default=%s netif_default=%s",
label(g0, eth, sta, ESP_NETIF_DNS_MAIN), label(g1, eth, sta, ESP_NETIF_DNS_BACKUP),
eth ? (is_zero(dns(eth, ESP_NETIF_DNS_MAIN)) ? "empty" : "set") : "n/a",
sta ? (is_zero(dns(sta, ESP_NETIF_DNS_MAIN)) ? "empty" : "set") : "n/a", link,
esphome::ethernet::global_eth_component->is_connected(), esphome::wifi::global_wifi_component->is_connected(),
desc(esp_netif_get_default_netif()), lw);
}
inline void lookup() {
// getaddrinfo blocks the loop until the DNS timeout when servers are set but no interface is up
if (!esphome::ethernet::global_eth_component->is_connected() &&
!esphome::wifi::global_wifi_component->is_connected()) {
ESP_LOGI(TAG, "LOOKUP skipped, offline");
return;
}
static uint32_t n = 0;
char host[64];
// a fresh name each time so the lwIP cache never answers
snprintf(host, sizeof(host), "q%" PRIu32 "-%" PRIu32 ".10.0.0.1.nip.io", (uint32_t) esphome::millis() / 1000, n++);
struct addrinfo hints{};
hints.ai_family = AF_INET;
struct addrinfo *res = nullptr;
uint32_t t0 = esphome::millis();
int rc = lwip_getaddrinfo(host, nullptr, &hints, &res);
uint32_t dt = esphome::millis() - t0;
if (res)
lwip_freeaddrinfo(res);
if (rc == 0)
ESP_LOGI(TAG, "LOOKUP ok %" PRIu32 " ms", dt);
else
ESP_LOGW(TAG, "LOOKUP FAIL rc=%d %" PRIu32 " ms", rc, dt);
}
// The W5500 PHY driver's pwrctl() is a no-op, so write PHYCFGR directly: OPMD=1 with OPMDC 110 (power down)
// or 111 (all capable, autonegotiation), applied by a PHY reset (RST low, then high).
inline void phy_power(bool on) {
static constexpr uint32_t PHYCFGR = 0x002E << 16; // common register block
esp_eth_mac_t *mac = nullptr;
if (esp_eth_get_mac_instance(esphome::ethernet::global_eth_component->get_eth_handle(), &mac) != ESP_OK || !mac) {
ESP_LOGE(TAG, "no mac");
return;
}
uint32_t before = 0;
mac->read_phy_reg(mac, 0, PHYCFGR, &before);
uint32_t val = (before & 0x07) | 0x40 | (on ? 0x38 : 0x30);
esp_err_t err = mac->write_phy_reg(mac, 0, PHYCFGR, val); // RST=0 applies the mode
vTaskDelay(pdMS_TO_TICKS(10));
if (err == ESP_OK)
err = mac->write_phy_reg(mac, 0, PHYCFGR, val | 0x80);
// The driver may not poll the link on its own; let it notice the change and post the link event
esp_eth_phy_t *phy = nullptr;
if (esp_eth_get_phy_instance(esphome::ethernet::global_eth_component->get_eth_handle(), &phy) == ESP_OK && phy)
phy->get_link(phy);
uint32_t after = 0;
mac->read_phy_reg(mac, 0, PHYCFGR, &after);
ESP_LOGI(TAG, "PHY power %s: %s PHYCFGR 0x%02X -> 0x%02X", on ? "ON" : "OFF", esp_err_to_name(err), (unsigned) before,
(unsigned) after);
}
} // namespace dnsdiag
@@ -0,0 +1,79 @@
# esphome#19805 repro on the Waveshare ESP32-S3-ETH (W5500). Substitutions:
# first: ethernet or wifi (network: priority order), scenario timeline in the interval lambda.
substitutions:
name: dnsrepro
first: ethernet
second: wifi
# "false" powers the W5500 PHY down at boot: a link-less start without unplugging anything
phy_boot: "true"
esphome:
name: ${name}
includes:
- dnsdiag.hpp
on_boot:
priority: -100
then:
- lambda: if (!${phy_boot}) dnsdiag::phy_power(false);
esp32:
variant: esp32s3
board: esp32-s3-devkitc-1
flash_size: 16MB
framework:
type: esp-idf
logger:
level: DEBUG
hardware_uart: USB_SERIAL_JTAG
network:
priority:
- ${first}
- ${second}
ethernet:
type: W5500
clk_pin: GPIO13
mosi_pin: GPIO11
miso_pin: GPIO12
cs_pin: GPIO14
interrupt_pin: GPIO10
reset_pin: GPIO9
clock_speed: 20MHz
wifi:
ssid: !secret wifi_ssid
password: !secret wifi_password
enable_on_boot: false
power_save_mode: none
reboot_timeout: 0s
interval:
- interval: 3s
then:
- lambda: dnsdiag::print();
- interval: 5s
then:
- lambda: dnsdiag::lookup();
# Timeline (seconds since boot):
# 40 wifi.enable (Wi-Fi DHCP starts after Ethernet has the default)
# 100 wifi.disable, 110 wifi.enable (Wi-Fi reconnect)
# 170 W5500 PHY power off (no link: Ethernet retries DHCP every 15 s), 250 PHY power on
# 320 done
- interval: 1s
then:
- lambda: |-
static int step = 0;
uint32_t s = millis() / 1000;
static const uint32_t at[] = {40, 100, 110, 170, 250, 320};
if (step >= 6 || s < at[step]) return;
ESP_LOGI("dnsdiag", "STEP %d at %u s", step, (unsigned) s);
switch (step) {
case 0: case 2: wifi::global_wifi_component->enable(); break;
case 1: wifi::global_wifi_component->disable(); break;
case 3: dnsdiag::phy_power(false); break;
case 4: dnsdiag::phy_power(true); break;
case 5: ESP_LOGI("dnsdiag", "TIMELINE DONE"); break;
}
step++;