After the Phase A / Phase B split in this PR, an external producer that
called wake_loop_threadsafe() (MQTT RX, USB RX, BLE event, espnow,
camera, mWW, speakers, USB host/CDC, lwip socket, enable_loop_soon_any_context)
only got Phase A — the component phase stayed gated by loop_interval_,
so the producer's component loop() could be delayed by up to
loop_interval_ ms before draining its queued work. That breaks the
long-standing semantic of wake_loop_threadsafe().
Add a wake_request flag set by every wake_loop_* entry point and
exchange-cleared at the gate in Application::loop(). When the flag is
set, force Phase B regardless of loop_interval_.
Storage is conditional on the threading model:
- ESPHOME_THREAD_MULTI_ATOMICS: std::atomic<uint8_t> (uint8_t, not
bool, because GCC on Xtensa generates an indirect call for
atomic<bool> ops — same workaround as scheduler.h)
- ESPHOME_THREAD_SINGLE / ESPHOME_THREAD_MULTI_NO_ATOMICS: volatile
uint8_t (8-bit aligned loads/stores are atomic on every supported
MCU; the platform signal that follows wake_request_set provides the
cross-thread/cross-core memory barrier)
Helpers (wake_request_set / wake_request_take) are always_inline so
IRAM_ATTR call sites stay in IRAM. Set BEFORE the platform signal so the
consumer is guaranteed to see the flag on its next gate check.
Adds an integration test that raises loop_interval_ to 2s, snapshots a
counting component's loop count, spawns a std::thread that calls
App.wake_loop_threadsafe() after 50ms, and asserts the count increments
inside a 500ms observation window. Without the fix the count would not
move for ~2s.
- Add unit tests asserting cv.Invalid when `substitutions: !include list.yaml`
resolves to a non-mapping, covering both do_substitution_pass and
do_packages_pass.
- Note in resolve_substitutions_block that the resolve is single-shot and
chained top-level includes are not supported (matches _walk_packages for
`packages: !include`).
- Seed `resolve_include` context with `command_line_substitutions` so
`substitutions: !include ${var}.yaml` can reference CLI-provided vars
in the include filename (parallels the `packages: !include` path).
- Validate shape of resolved substitutions in `do_packages_pass` and raise
`cv.Invalid` under `CONF_SUBSTITUTIONS` instead of letting `UserDict()`
fail with a low-level exception on a non-mapping.
- Fixture 17 exercises the CLI-templated include filename.
Resolve a deferred IncludeFile before validating the substitutions shape in
do_substitution_pass, and before wrapping it in UserDict in do_packages_pass.
Fixesesphome/esphome#15848
Doc and test updates from a code review of this PR:
- Correct the `tail_us == 0 on Phase A-only ticks` claim in the
Application::loop() comment and the RuntimeStatsCollector::record_loop_active
docstring. `loop_tail_start_us` is set to `loop_before_end_us`, and
`loop_now_us` is sampled later, so `tail_us` on Phase A-only ticks is
the small gate-check + record prefix — tiny but non-zero.
(Also flagged by Copilot on application.h:623 and runtime_stats.h:45.)
- Call out ESP8266 as the floor case in the WDT_FEED_INTERVAL_MS margin
table. Its soft WDT (~1.6 s) is the tightest margin at ~5x, so future
changes to the constant need to preserve comfortable headroom there.
- Tighten the test lower bound at tests/integration/test_loop_interval_decoupling.py
from `2 <= loop_delta <= 6` to `3 <= loop_delta <= 6`. Allowing 2 would
let a >50% slowdown from the 4-in-2s nominal pass as CI jitter, which
undermines the regression signal. 3 keeps the test honest while still
absorbing realistic CI jitter.
- Add a second integration test
(test_loop_interval_default_not_pulled_forward) that covers the inverse
direction: at the default loop_interval_ with a fast scheduler item
(5 ms — well under the old delay_time/2 = 8 ms floor), the component
phase must still run at ~62 Hz, not the pre-fix ~128 Hz. This locks
down the original 128 Hz → 62 Hz regression that motivated the PR.
The labels were there to help humans scanning the generated main.cpp
find component boundaries, but they were:
- Unreliable: CORE.flush_tasks can interleave coroutines on each
await, so a component's later statements can land in another
component's begin/end block.
- Load-bearing for a pile of complexity: a tuple return from
_wrap_in_iifes, a has_iife flag, a comment-only detector to
suppress trailing end-markers for comment-only components, and
a brittle `"[]()" in line` check that could false-positive on
YAML dumps containing lambda syntax.
- Not actually needed — generated main.cpp is a build artifact
rarely read by anyone, and cg.LineComment("name:") already puts
the component name at the start of its block.
ComponentMarker stays as a pure chunking sentinel — it tells
cpp_main_section where component boundaries are (for grouping) but
produces no C++ output. _wrap_in_iifes returns a plain list again.
Added a regression test for the now-defused case of a comment
containing "[]()" that was previously flagged by review.
- Count { and } characters per line instead of matching whole-line
tokens. Current codegen only emits scope braces as standalone lines
(from cg.with_local_variable()), but the defensive change is robust
against future codegen emitting inline control flow like
`if (cond) {` or `} else {` on one line.
- Add a regression test covering those inline-brace patterns.
- Fix stale docstrings on ComponentMarker and cpp_main_section that
still claimed "stack frame released on return" and described the
IIFEs as "noinline". The IIFEs have no noinline attribute and rely
on scope-based lifetime shortening rather than guaranteed frames.
Rename the bracket markers from "// === X ===" (same on both sides)
to "// === begin X ===" and "// === end X ===" so the generated
main.cpp reads unambiguously when scanning by component. Comment-only
components still get a single "begin X" marker since they have no
IIFE to close.
The marker comment was being emitted as the first line *inside* each
IIFE:
[]() {
// === logger ===
// logger:
// ...
...
}();
That works but buries the component label inside the lambda body, so
scanning generated main.cpp to find "where does component X's setup
live" is harder than it needs to be. Emit the marker before and after
the IIFE instead:
// === logger ===
[]() {
// logger:
// ...
...
}();
// === logger ===
Comment-only components (e.g. sha256, async_tcp, empty platforms like
binary_sensor:) don't grow a useless trailing duplicate marker —
when there's no IIFE to bracket, the marker is emitted once.
Some components (sha256, async_tcp, network, empty text_sensor:, etc.)
emit only a ComponentMarker plus config-dump comments and no actual
C++ statements. Wrapping those in a `[]() { ... }();` IIFE is pure
clutter in the generated main.cpp — the IIFE has no body.
When _wrap_in_iifes sees a chunk whose lines are all // comments,
emit them verbatim instead of wrapping. Peak stack and flash are
unchanged on apollo and neargaragedoor since GCC was already
eliding the empty IIFEs; this just makes the generated code read
cleanly to humans.
Additional measurements showed GCC's -Os inliner re-inlines most IIFE
chunks back into setup() by choice, and the structural scoping alone
captures nearly all of the peak-stack benefit on esp32 without the
flash cost of forcing all chunks to stay as real functions.
Apollo (esp32-s3, -Os) with vs without noinline:
peak setup stack 176 B (noinline) vs 304 B (scope-only)
flash delta +388 B (noinline) vs -504 B (scope-only)
chunks kept 86 vs 20
Issue #15796 is an LVGL-setup class of bug that has only surfaced on
esp32 after years in the field; the extra guarantee that noinline
provides is not worth the flash cost in practice. Also rename the
helper from _wrap_in_noinline_iifes to _wrap_in_iifes to match.
The C++ standard-attribute spelling [[gnu::noinline]] placed between a
lambda's parameter list and body binds to the return type, not the
call operator. GCC 14 silently ignores it and emits -Wattributes
warnings at every chunk site. Switch to GCC's __attribute__((...))
syntax which binds to operator() as intended.
Measured impact on apollo-r-pro-1-eth (esp32-s3, -Os) vs the broken
[[gnu::noinline]] version: setup() frame 160 B -> 32 B, peak stack
304 B -> 176 B (another -42%). Flash grows by 888 B because all 86
chunks now stay as separate functions instead of GCC inlining the
small ones (which it was free to do when the attribute was ignored).
Net vs baseline -Os: peak stack 1264 B -> 176 B (-86%); flash
+388 B (<0.05% of a typical esp32 partition).
J. Nick Koston