Runs the test_build_components matrix with --toolchain arduino behind
a cached framework install, and teaches determine-jobs which changes
must trigger it.
The (platform, toolchain) mapping lived in both __main__ and the esp8266
component, so a renamed backend module meant editing two places and the
drift test only covered one. build_helpers/native.py now holds it; the
module stays import-light so the serial upload and logs fast path can
resolve a backend without pulling in a platform package.
Drops the ninja dry-run probe (a second full manifest load and stat
sweep per build to suppress one line), the compile-database stamp file
(a plain mtime compare says the same), and the ccache sentinel
(resolve_ccache_path only spawns when it returns a value, so a plain
fallback never re-probes). The crash decoder resolves addr2line and
the ELF once per run instead of once per address, which removes the
module-level warn-dedup dict along with its test fixture. The size
summary uses the toolchain path it was handed rather than re-deriving
it, and its two failure branches merge.
The framework import moves into the functions that need it: the serial
upload and logs fast path resolves this module for artifact paths
alone and was paying 16 ms and 29 modules for the package-download
stack. test_lazy_imports now guards that module so it cannot come
back. analyze-memory's idedata candidate list learns the suffixed
native cache name, and warn_if_idedata_missing goes back to being a
one-for-one extraction.
Runs the test_build_components matrix with --toolchain arduino behind
a cached framework install, and teaches determine-jobs which changes
must trigger it.
The smoke-test job, its cache action, and the job-determination and
harness changes that drive it land separately so this PR stays the
build driver alone.
_check_install and _resolve_flag_sets pull ~110 lines out of the one
function body; behavior and emitted output are unchanged (the existing
end-to-end tests pass as-is).
A half-extracted framework cache now names the missing path instead of
surfacing a gcc failure; pure reorder, the checks depend on nothing
the generation step produces.
The dev merge moved the ESP-IDF trigger constants into clang_tidy_hash;
the esp8266 set keeps its explicit shared-module list with a subset
tripwire against the imported set instead of unioning it wholesale.
The bundled elf2bin.py uses unescaped regex strings that Python 3.12+
flags at compile time; the frozen 3.1.2 framework will never fix it,
so the ninja rule suppresses that category for the one invocation.
Only IncompatiblePlatform counts as a knowing skip; other malformed
manifest data stays out of the satisfied set so the reconciliation can
still flag a broken promise. Backend-provided names are recorded before
the manifest-name evidence so the overlap warns once, in the backend's
own suppression loop. ar runs rcs/qs so the symbol index is explicit,
and the copy shim prints and returns 1 like its siblings instead of
re-raising.
The legacy manifest-less layout is legal and the 3.1.2 core ships one
such library (FSTools), so the warning would be unactionable noise; a
test now pins the level.
A truncated bundled library.json now raises the module's named error
with the clean-all hint instead of a raw JSONDecodeError. The renamed
_reconcile_versionless_skips advertises that it records into
backend.provided_requests. Two real-converter tests gain the
registry-guard patch their siblings use so a regression fails offline.
Rely on normalize_dependencies' guarantee of a non-empty string name
instead of re-checking it, collect sources in a single pass that
re-roots on the resolved directory rather than calling realpath per
file, probe library.properties once, hoist the invariant log-level
choice out of the warning loop, and record provided_requests as a set
since order was never used.
The walk records across re-resolutions, so a dependency a re-resolved
manifest no longer declares would have failed the build by name. The
reconciliation now intersects the recordings with the dependency names
of the manifests actually emitted, keeping the raise for real broken
promises. The manifest-name warning also gains the treeless-backend
gate its sibling got, since those backends cannot act on it either.
A version-less dependency the walk skips on the backend's promise is now
recorded like the versioned path, and a bundled candidate the backend
knowingly skips through the platform filter counts as satisfied, so the
post-emit reconciliation covers both paths without raising on legitimate
builds. The manifest-name warning deduplicates like its sibling, the
final drop warning stays at debug for backends without provides() since
they can never supply a bundled name, and a bundled library missing both
manifests leaves a debug trace instead of silently building on defaults.
Extra-script LINKFLAGS travel outside build.flags by design; the backend
now drains them into the library's link flags, matching the ESP-IDF
backend, instead of dropping a link-stage input silently. An unfulfilled
provides() promise raises by name (deduplicated) like the module's other
can-never-link checks, the vacuous end-to-end silence test is replaced
by a link-flags test that can fail, and a version-less dependency
assumed satisfied by manifest name alone warns like its backend twin
since the match is name-only evidence.
KEY_LDSCRIPT replaces the bare literal at both the table entry and the
lookup, so a typo cannot silently fall back to the layout-moving
default; the regeneration recipe notes that overrides must survive a
refresh; a board pinning an override on an Arduino core <= 2.4.2 warns
that the layout cannot be honored; and board_ld_script gains the test
pinning d1_wroom_02's shipped layout and the size defaults.
A bare 2-tuple is one name=value pair (not -DFOO -D1), a dict maps
names to values, and a None value is a bare define, per processDefines.
The unused caplog_at_info helper and the progress kwarg the download
stubs declared ahead of their PR are gone.
CPPDEFINES joins the quoted buckets (a spaced define no longer splits
across tokens) and its tuple branch validates the pair elements. The
fake env warns on attribute access rather than call, so hasattr/
truthiness branches are diagnosable, with dunder probes excluded; an
unmodelled subscript warns and returns '' instead of a KeyError
discarding the whole capture. Malformed platforms/frameworks values
raise plain InvalidLibrary, giving the non-platform warning branches a
real producer, and their tests use real manifests instead of
monkeypatched raisers. The version-less dependency drop moves to debug:
bundled names (Wire, SPI) made it per-build noise nobody can act on.
Split out of #18662: the converter/manifest hardening without the
download parallelization, which stays there.
The PIO extra-script shim moves from espidf/ to platformio/ and gains
type checks for every manifest field, capture of the Append variants and
CPPPATH, warn-once diagnostics for unmodelled env reads, and uniform
shlex quoting so captured values survive the lex_build_flags round-trip.
The dependency walk normalizes every spelling PIO accepts, splits the
routine cross-platform skip (IncompatiblePlatform) from real manifest
problems, validates manifest shape once, and the shared flag lexer
warns and drops trailing or empty glued arguments.
flash_mode comes from the board_build.flash_mode option the component
already pins (validated by cv.one_of against the same BUILD_FLASH_MODES,
now living in const.py so the two sides cannot drift), retiring
KEY_FLASH_MODE from the generator. The scanf-float decision is stored in
CORE.data where it is computed instead of being a discarded local, so
KEY_SCANF_FLOAT has a producer in this PR. A trailing bare -include in
build_src_flags fails by name instead of degrading into the source dir.
Both prefetch warning loops format through _failure_reason so a
message-less exception cannot log blank. A bar-frame write failure in
the failure path cannot displace the download error, the failure list
is annotated BaseException (cancellations are entries too), the Ctrl-C
docstring notes an in-progress extraction runs to completion, and the
wave-prefetch docstring carries the superseded-sibling caveat.
The defensive spiral (#18668, closed) guarded inputs that cannot occur:
gen_esp32part validates partitions.csv before a build can succeed, and
esp_idf_size.json's realistic failures do not produce the handled
shapes. This PR's real change stands alone again: the bar moves to
build_helpers as print_size_line, with dev behavior everywhere else.
Corrupt shapes at any level warn as malformed while a simply-absent
DRAM/DIRAM region stays at debug, the backstop warning carries its
traceback, and the image_size warning names its file.
The full #18668 state, adapted to the shared bar: _ram_bar/_flash_bar
return validated (used, total) pairs printed through print_size_line,
the partition parser returns None for legitimate absence and raises
only for a broken table, skips follow the warn/debug visibility rule,
and run_compile passes the table path unconditionally. Ported now so
the merge-time conflict resolves itself; the chain-only exact-format
and RAM-vs-Flash isolation tests stay.
A bare -DMMU_IRAM_SIZE would preprocess to len = 1 and fail far away in
ld (or as a misleading testing-mode message), so MMU_CUSTOM requires a
body on both segment sizes before the shape check. -fuse-ld=/-specs=/
--specs= are driver options with no -Wl, spelling, so they get their
own rejection instead of advice that would make things worse. An
unremovable stale stderr note now warns with the clean hint, since its
consequence is a permanently unstamped cache.
KEY_LDSCRIPT replaces the bare literal at both the table entry and the
lookup, so a typo cannot silently fall back to the layout-moving
default; the regeneration recipe notes that overrides must survive a
refresh; a board pinning an override on an Arduino core <= 2.4.2 warns
that the layout cannot be honored; and board_ld_script gains the test
pinning d1_wroom_02's shipped layout and the size defaults.
A bare 2-tuple is one name=value pair (not -DFOO -D1), a dict maps
names to values, and a None value is a bare define, per processDefines.
The unused caplog_at_info helper and the progress kwarg the download
stubs declared ahead of their PR are gone.
A logging filter installed for the batch's duration ends a partial bar
row (under the bar's lock) before any record is emitted, so a mirror
retry warning from a worker is no longer overwritten by the next CR
frame. The Ctrl-C sentinel derives from BaseException so a broad
except Exception in the download layers cannot convert an abort into a
retry, an abandoned job is reported like a failure instead of reading
as a completed download, and the mirror-sweep backoff sleeps through
the cancellation tick so a Ctrl-C during a retry window aborts at the
boundary instead of after it. The docstring states the parked-socket
worst case.
A dependency chain (api -> noise-c -> libsodium) discovers one archive
per wave, so the <2 gate sent every chained download through the legacy
per-file path. Sized registry archives now always use the batch runner:
one download method, one bar; the per-file path remains only for
unknown-size, git, and local sources.
registry_download memoizes per process (failures uncached), so the
prefetch and the install share one metadata round-trip per package.
The idf ccache env always hands idf.py exactly one canonical spelling:
the no-ccache path exports 0 so an unparsable inherited value or a
probe-rejected shim cannot enable it, the enabled path exports 1
whatever accepted spelling arrived, and forced-on still runs the
resolver for its no-binary warning. An empty env value keeps its
historical disable meaning instead of falling to the default policy.
The generator now reads build_src_flags instead of hardcoding the
throw_stubs force-include, so the option must be set on both toolchains;
gated, the native build silently lost the include (+168 bytes of
libstdc++ throw strings). Verified byte-identical again.
The throw_stubs force-include now comes from the build_src_flags option
esp8266/__init__ pins, collapsing the two spellings to one source of
truth; -include paths resolve against the source root. The ldscript
fallback consumes boards.board_ld_script instead of duplicating the
per-board rule, flash_mode is checked against its closed set before
landing unquoted in the elf2bin command, and the board-gate comment no
longer names a nonexistent validator.
LOW_MEMORY is upstream's else branch, so the default variant stands in
for it and every listed knob wins -- pinned for the ordinary SNTP
multi-server config where esp8266's HIGHER_BANDWIDTH_LOW_FLASH must
prevail. A stale warn note that cannot be removed now vetoes the stamp
(mirror of the lost-note case), and the unreadable-cached-note warning
names its cause.
CPPDEFINES joins the quoted buckets (a spaced define no longer splits
across tokens) and its tuple branch validates the pair elements. The
fake env warns on attribute access rather than call, so hasattr/
truthiness branches are diagnosable, with dunder probes excluded; an
unmodelled subscript warns and returns '' instead of a KeyError
discarding the whole capture. Malformed platforms/frameworks values
raise plain InvalidLibrary, giving the non-platform warning branches a
real producer, and their tests use real manifests instead of
monkeypatched raisers. The version-less dependency drop moves to debug:
bundled names (Wire, SPI) made it per-build noise nobody can act on.
The registry response already carries each package file's size, so the
combined bar is sized from URLSource instead of a HEAD sweep; the probe,
its all-or-nothing fallback, and its duplicate Content-Length parsing are
gone, and archives without a reported size simply stay on the sequential
path. run_batch_downloads now takes (name, size, fetch) jobs and builds
the (now private) progress bar itself, collecting failures from future
results instead of a shared list; both call sites use functools.partial
instead of lambda factories. The prefetch touches only URLSource, so the
progress parameter comes off Source/Git/Local/ConvertedLibrary, and the
wave prefetch is wrapped best-effort like its ESP-IDF twin, which also
dedupes by dest only after the verifiability check. ProgressBar.done()
itself now skips the newline when no frame was drawn or the 100% frame
already ended the line, collapsing the batch wrapper's guard and fixing
the stray blank line after OTA uploads.
Split out of #18662: the converter/manifest hardening without the
download parallelization, which stays there.
The PIO extra-script shim moves from espidf/ to platformio/ and gains
type checks for every manifest field, capture of the Append variants and
CPPPATH, warn-once diagnostics for unmodelled env reads, and uniform
shlex quoting so captured values survive the lex_build_flags round-trip.
The dependency walk normalizes every spelling PIO accepts, splits the
routine cross-platform skip (IncompatiblePlatform) from real manifest
problems, validates manifest shape once, and the shared flag lexer
warns and drops trailing or empty glued arguments.
A failed stderr-sidecar write no longer stamps the cache: the next build
re-runs -E and re-derives the preprocessor diagnostic honestly instead
of losing it for the life of the build dir. The sidecar unlink joins the
best-effort contract. The plain-linker denylist gains the
-nodefaultlibs/-nostdlib/-rdynamic family and the -fuse-ld=/--specs=
prefixes, with a comment naming it best-effort rather than complete.
The damaged-file helper test now lives only in test_helpers.py beside
the OSError-branch coverage.
shlex.split('-D ""') left a bare -D that gcc resolves by eating the
next flag; the funnel now warns and drops it for every consumer, which
also retires raise_on_empty_arg_flags, whose staged ESP8266 caller no
longer has anything to catch.
cancel_futures only drops queued jobs, and the non-daemon workers are
joined at interpreter exit, so a Ctrl-C during the ESP-IDF toolchain
fetch waited out every in-flight archive. The tracker each job reports
through now checks a cancellation flag set when the wait raises, so
downloads abort at their next progress tick and the resumable .part
files keep the bytes already fetched.
The wave-deferral test stubs _content_lengths instead of issuing real
HEAD requests that passed only by failing, the dead hasattr branch in
the HEAD-probe test is gone, and the .C suffix comment names the SCons
divergence instead of claiming parity.
Prepend/AppendUnique/PrependUnique write the same captured keys and now
alias Append instead of no-op'ing away a link directory. env.get on an
unmodelled var warns once so a script taking the default branch is
diagnosable. LIBS/LINKFLAGS/CPPFLAGS entries are shlex-quoted like the
path buckets, since each is one argv token in SCons and re-lexing must
not split a spaced value. The HEAD probes follow the happy-eyeballs
convention, and the tool-prefetch backstop keeps its traceback at debug.
A list-of-non-dicts compile DB now raises by name inside the best-effort
tuple. The launcher rejection moved into parse_entry itself so no caller
can record ccache as the compiler, and the shape key includes the entry's
directory since relative includes resolve against it. format_bar raises
on a non-positive total so every consumer must handle the meaningless
bar; the espidf summary validates the RAM region by name (a malformed
region can no longer suppress a computable Flash line) and the blanket
backstop logs the traceback at debug. A new test pins that the CI
trigger paths exist on disk, so a rename fails loudly.
shlex.quote wraps backslash paths on Windows, so the raw emitted strings
are platform-varying; lexing like the consumer does pins the behavior
that matters and passes on both separators.
lex_build_flags shlex-splits build.flags, so an unquoted spaced path
became two garbage tokens; shlex.quote at the emit site round-trips it
back to one.
A non-string extraScript now fails naming the library instead of an opaque
TypeError, matching the hardening on every other manifest field. CPPPATH
joins the captured keys and translates to -I flags anchored like LIBPATH,
since the pipeline already routes -I to include dirs. Version-less
dependency drops log at INFO until the arduino-backend reconciliation
lands, and the wave-deferral comment now claims only what the guard
actually saves.
A compile DB that parses as valid JSON but is not a list now raises
EsphomeError inside IDEDATA_BEST_EFFORT_ERRORS instead of a TypeError that
fails an already-linked build. Cached idedata is validated on read against
the write path's guarantees (dict shape, no launcher as compiler, include
union present) so caches from older versions regenerate instead of serving
degraded data, and the served copy is re-stamped with the current ELF path.
Response-file shape sentinels key on the output object rather than the
source file, so two objects built from one source keep both include sets.
The size summary mirrors the standalone hardening PR: a blank partition
size cell raises by name, and a blanket guard keeps unexpected nested
shapes from raising past a build that already linked.
An unknown Content-Length (probe failure, non-2xx, or a CDN omitting it
on HEAD) now skips the parallel prefetch so the sequential downloads
keep their per-file bars, instead of a silent multi-MB batch; the HEAD
helper returns None for unknown, distinct from a genuine zero, and
catches only RequestException. The cache probe logs its failure at
debug. The espidf tool prefetch reads sizes with .get so a missing size
only drops the bar instead of aborting the resumable prefetch. The
idedata warning logs its traceback at debug for parser regressions, and
framework_helpers joins the ESP-IDF infra CI triggers it is imported
by.
A bundled manifest relying on an extraScript, or any manifest declaring
precompiled/ldflags, now raises naming the library instead of warning
into a link failure far from the cause. The backend-side usability
re-check logs all InvalidLibrary causes at debug with an accurate
comment: the walk runs dependency_is_usable on every versioned entry
before the provides() skip, so the fault always warns there first (a
real-converter test pins the once-only warning). build_tool handles a
missing subcommand with its named error instead of an IndexError.
The name and version checks were duplicated across the dict-shorthand
and list spellings with four identical warnings; _valid_dependency_entry
carries them once.
The library prefetch pool now cancels queued futures on interrupt like
the espidf pool it mirrors, prefetch and HEAD failures log their cause
at debug (a suppressed bar is traceable), and non-2xx HEAD statuses read
as unknown size. normalize_dependencies rejects a non-string version in
both spellings (a container would raise from set.add(); an int fails
opaquely in the registry), and the walk's version-less skip logs the
dependency at debug until the arduino-backend reconciliation lands.
The library prefetch pool now cancels queued futures on interrupt like
the espidf pool it mirrors, prefetch and HEAD failures log their cause
at debug (a suppressed bar is traceable), and non-2xx HEAD statuses read
as unknown size. normalize_dependencies rejects a non-string version in
both spellings (a container would raise from set.add(); an int fails
opaquely in the registry), and the walk's version-less skip logs the
dependency at debug until the arduino-backend reconciliation lands.
The platform_version pop defaults to the schema spec so a second
validation pass over an already-validated dict cannot warn about a key
the user never set. An unparsable compile database now fails naming the
parse error and the offending output instead of blaming renamed ninja
rules. analyze-memory validates the native objdump/readelf exist and
fails by tool name rather than silently analyzing with host binutils.
The shared smoke-test helper is renamed _toolchain_components_to_test
(it serves the esp32 PlatformIO job too), and the decode-dedup state is
cleared by an autouse fixture instead of by hand.
The wave logged 'Downloading N libraries' before checking the
extraction markers, so a warm build announced downloads it never made.
URLSource gains an is_cached probe (best-effort in the prefetch) and a
fully warm wave stays silent.
A URL-pinned dependency now skips the bundled probe (the walk resolves
the fork; adding the bundled copy would double the archive). A versioned
bundled candidate's non-platform manifest fault warns here since it
skips the walk's usability filter via provides(); version-less causes
stay at debug (the walk already warned). The pending drain logs a
manifest-name suppression at debug and drops its unreachable
bundled_names re-check. A converted tree with no sources and no headers
now fails by name at emit like the bundled case (test scaffolds gained
real source files). build_tool validates each mode's operand count and
a failed copy unlinks the partial output.
Brings in #18513 (BatchDownloadProgress and the download_with_resume
progress callback) and extends the plumbing one level:
download_from_mirrors passes a progress callback to both target kinds,
Source/ConvertedLibrary.download thread it through, and the library
wave prefetch draws one combined bar sized by parallel HEAD requests
(no sizes, no bar). The thread-local bar suppression this replaces is
removed; a batch caller passing a tracker also owns the messaging, so
the per-file Downloading line only prints for solo downloads.
The board name lands unquoted in two -D bodies and the flash linker
script name joins under the SDK and build ld dirs, so both are now
shape-validated like f_cpu. A VTABLES_IN_* define carrying a body would
split the compile line from the linker script and is refused. The
damaged-cache overwrite moves into a shared _write_generated helper
scoped to the comparison read: a corrupt existing copy is logged and
replaced (the testing-mode flash ld gets the same recovery), while a
genuine write failure still raises with its cause.
A dependency whose version is a repository URL names one specific
source, so it is now detected before the backend-provided skip and
resolves as git instead of being replaced by the bundled copy. The
library header-suffix set becomes LIBRARY_HEADER_SUFFIXES so it cannot
shadow esphome.const's narrower HEADER_FILE_EXTENSIONS. Suppressing a
genuinely bundled name on an external short-name match warns (an
accidental collision would surface as link errors), a converted library
with no sources and no headers warns by name, and the walk's
unreachable nameless-dependency check is gone.
copy_src_tree is always mocked in the writer tests, so the new
missing/stale/unreadable debug branches were unreachable; the decision
moves into _build_info_stale with a test covering every branch.
voluptuous fills schema defaults in set-iteration order, so the
validated dict's key order changes with the process hash seed; the
unsorted dump churned the comment block in main.cpp and relinked the
firmware on every esphome run for any config using a defaults-heavy
action (logger.log in a button's on_press was enough). Affects the
PlatformIO path identically.
Every sources_changed site now logs which file (or which build-info
staleness) triggered it, so a config that relinks on every run can be
diagnosed from one -v compile instead of diffing the build tree.
A framework tree without libraries/ now raises naming the path instead
of silently rerouting every bundled name to the registry, matching the
empty-bundled-dir error. check_library_data accepts framework=None to
skip the framework check (mirroring platform=None), so the backend-side
platform re-check no longer re-fires the frameworks-mismatch warning
the walk already emitted. The platform-rejection-is-debug test patched
the wrong module and asserted nothing; it now patches the backend's
import and asserts the dependency is absent from the resolved set.
A -n dry-run probe answers 'no work to do' without the real ninja spawn,
and running with cwd instead of -C drops the 'Entering directory' banner
on real builds too.
The walk now drains its frontier, prefetches the wave's registry
archives with a small thread pool (deduped by URL, progress bars
suppressed per thread since parallel bars would interleave), and then
processes the wave sequentially; the sequential download() call stays
authoritative so failures surface with resume exactly as before. Same
approach as the espidf tool prefetch in #18513.
voluptuous accepts [] for [cv.string], so vals[-1] could raise an
IndexError at codegen time; an empty list now falls through to the
ignored-option warning.
The backend-side bundled filter now checks compatibility silently (the
shared walk owns the warning), so one manifest fault warns once instead
of twice. Bundled-name membership uses the exact on-disk directory
names instead of a per-name is_dir() probe, so a case-insensitive
filesystem cannot match a case-mismatched dependency and build the same
bundled library twice.
The schema permits the list form, so the routing now stores the last
value (like a later platformio.ini line) and the generator always sees a
scalar. The using_native_toolchain docstring points at write_cpp_file's
dispatch so NATIVE_TOOLCHAINS membership and the project-writing branch
flip together.
An owner-qualified dependency name (Owner/Pkg, both halves safe) is the
converter's to resolve and no longer draws the malformed-entry warning,
which stays for genuinely unsafe shapes. A declared srcFilter matching
only inert files now warns like one matching nothing, while matched
headers still read as a header-only library. The short-name suppression
logs the assumed external at debug so an accidental collision is
attributable. A failing ar batch unlinks the truncated archive before
returning its exit code.
cv.boolean's inline tuples and the ccache env tables were two copies of
the same spellings; TRUTHY_BOOL_STRINGS/FALSY_BOOL_STRINGS now live in
esphome/helpers.py, cv.boolean consumes them, and parse_enable_env
derives its tables by adding the 1/0 env convention.
parse_enable_env's inline tuples become TRUTHY_ENV_STRINGS and
FALSY_ENV_STRINGS frozensets mirroring cv.boolean's spellings (enable
and disable included) plus the 1/0 env convention, matching
device-builder's TRUTHY_BOOL_STRINGS pattern.
The typed IncompatiblePlatform exception, dependency_is_usable,
warn_properties_depends, the lex_build_flags helper (espidf switches to
it), the normalize_dependencies validation, the manifest shape check,
the component-level drop-warning split, and the SCons case-sensitive
.C/.C++ suffixes all harden the shared converter independently of the
arduino backend, so they belong in this PR; the provides hook and the
version-less reconciliation stay with the backend that needs them.
build_tool gains one function per mode plus a shared rspfile reader.
_library_info becomes an orchestrator over _resolve_src_dir,
_resolve_lib_archive, _classify_build_flags, _resolve_include_dirs, and
_collect_lib_sources. The version-less reconciliation moves into
_warn_unsatisfied_versionless. _is_safe_library_name switches from a
character denylist to a filename-plain allowlist, which excludes
separators, drive colons, and dot-only names by shape.
The unmapped-suffix warning fired for every header-only library (the
default +<*> filter matches headers), so ArduinoJson would have warned
on every ESP8266 build. It now names exactly the source-like files
(.CPP, .ino and case-variants of the map) the case-sensitive suffix map
rejects, partial drops included, and stays quiet for headers and
metadata.
The ar shim batches the expanded object list by argv length (rc then q
appends), keeping the command line under the Windows 32767-char limit
the rspfile existed to avoid. A non-iterable dependencies value in a
manifest now warns by library name instead of raising a bare
TypeError.
The job env carries a literal ~ that Python's expanduser and
actions/cache both resolve; the assertion compared the raw string and
failed the native toolchain job.
The depends=-without-dependencies warning moves into the shared walk so
espidf builds report it too; the arduino backend keeps it only for
bundled libraries the walk never sees. Version-less dependencies now
pass the platform filter before the reconciliation records them, so a
platform-excluded entry no longer draws a spurious drop warning.
A converted library whose matched files all fall through the source
suffix map warns instead of silently producing an empty archive, and
unhonored precompiled/ldflags properties fields warn by library name.
The ar rspfile parser undoes ninja's POSIX inner-quote escape and the
module docstring records the one-object-per-line contract. The
includeDir comment now states that warn-and-drop is intended.
ninja -t compdb exits 0 with an empty list when the rule names drift, so
the compile database is now parsed and an empty result fails the build
and drops the stale file. analyze-memory refuses an unsupported
toolchain before paying for the compile. A missing size binary warns
instead of discarding an already-linked build, _parse_app_size returns
explicitly from its not-found branch, and the CI cache action asserts
the caller's install prefix matches the cached path. The esp8266
run_compile dispatch hook and copy_files early-return gain direct
tests.
The -T script name was the one user-controlled token on the link line
that skipped shell_token, so a space or dollar sign in a
board_build.ldscript override corrupted the emitted linkflags. The
elf2bin comment now records that --flash_size deliberately stays
board-derived, matching PlatformIO (which reads upload.maximum_size,
not the ldscript).
The linker-script stamp now records the sha256 of the written content, so
an externally edited or truncated cached script regenerates instead of
passing a SECTIONS substring probe, and a non-UTF-8 cached script is
overwritten by the regeneration instead of aborting it. The cached
preprocessor diagnostic re-emit no longer swallows its own read failure.
board_build.f_cpu is validated (digits with an optional L) before landing
unquoted on the compile line, _pio_option rejects empty and blank values
instead of silently reverting to the default, and user MMU define bodies
re-quote through shell_token like every other user token. The plain-form
linker denylist is a named constant the _project_flags docstring points
at, so the two cannot drift.
The dict-shorthand dependency form now runs the same name validation as
the list form, so an empty key or a spec overriding name with a
non-string warns instead of raising a bare TypeError from the key
builder. A lib_ignore'd version-less dependency is deliberately excluded
and no longer draws the reconciliation warning. The walk's
component-level InvalidLibrary handler mirrors dependency_is_usable:
cross-platform skips stay at debug, any other cause warns. A malformed
libArchive/dot_a_linkage value now fails naming the library like the
sibling build fields, and the suffix map gains SCons's case-sensitive .C
and .C++ so those sources compile as C++ instead of silently dropping
out of the archive.
An owner-qualified version-less dependency was skipped by both the
arduino backend (owner set) and the reconciliation (provides() knew the
short name), so nothing logged the drop; provides() now only satisfies
owner-less names, mirroring the walk's backend-provided guard. A dict
dependency entry with no name at all now warns in the normalizer like
any other malformed entry.
Bundled dependency additions are deferred until conversion finishes and
skipped when a converted library's manifest already provides the name,
so a name that is both bundled and registry-resolved cannot build twice.
External short names come from the request spec (custom Name=url form
included) instead of the URL tail. The empty-bundled-library probe now
walks the whole library tree against a shared header-suffix constant,
and the dead versioned-dependency branch is folded into the surviving
check.
Validate manifest shape once in the converter (every backend dereferences
data/build), share the InvalidLibrary filter as dependency_is_usable(),
and let the walk's reconciliation own version-less drop reporting so the
arduino backend's request-key diff, owner-no-version warning, and
node_key plumbing all go away. One memoized _provided() predicate now
answers bundled-name checks at all three sites. Tests gain shared
scaffold helpers and lose the assertions that pinned deleted messages.
The shared walk now defers every version-less skip and reconciles after
emit against the final resolution set (request keys, resolved manifest
names, and backend.provides(name) per name), so drop visibility is
correct for every backend by construction instead of by a comment-level
contract, and the arduino backend's duplicate pending_drops pass and its
short-name suppression heuristics are deleted. The provides lambda
applies _is_safe_library_name so an unsafe manifest name is simply not
provided, and the guard also rejects drive-colon names that would
escape the tree on Windows.
The check guards URL-ish names before _node_key (a malformed entry
belongs to the drop warning, not a RuntimeError) and matches against
top_level_keys so the message says exactly what it verifies; a
transitive-only node can still fail compatibility later and must not
suppress the drop report.
The walk's version-less warning stays quiet for a name the config also
requests top-level, and the provides-backend debug level carries the
ownership rationale (the arduino backend defers drops and suppresses
walk-resolved names this layer cannot know yet). _is_safe_library_name
guards both the top-level bundled check and manifest dependencies, so a
YAML libraries entry of '.' or a backslash name cannot join under the
framework tree. The drop suppression strips only the sanitized owner
prefix, keeping a library's own double underscores. An empty bundled
directory raises naming the fix instead of warning into an unlinkable
build, and the second real-converter test pins ESPHOME_DATA_DIR.
normalize_dependencies' parameter is manifest_name (the dict branch
already binds owner to a package owner), and the arduino call site
passes the library's name so its unrecognized-entry warning stops
saying 'of manifest'. A version-less dependency warns when the backend
declares no provides tree (espidf/zephyr/nrf52 have no post-emit
pickup), and the shared walk mirrors the typed IncompatiblePlatform
branch: routine platform skips stay at debug, any other InvalidLibrary
cause warns naming the component. The two real-converter tests pin
ESPHOME_DATA_DIR to tmp_path so an ambient data dir cannot leak in.
A plain-string dependencies value comma-splits into names instead of
iterating into one-character libraries; entries that normalize to
nothing warn naming the manifest, and the walk's version-less skip logs
at debug for every backend. The not-bundled warning is deferred until
after the walk and stays quiet for names resolved transitively, so a
false skipping message cannot teach users to ignore the real one. An
explicitly requested bundled library that resolves to no sources and no
headers warns that the install may be incomplete.
The converted path warns for a properties-only depends= spelling (the
walk reads the JSON key). A dependency name is validated before
becoming a path component: non-strings and separator names are
malformed entries, never joined (apply_extra_script already guards the
same shape). normalize_dependencies coerces PIO's bare string-list form
to name dicts instead of dropping it before every visibility warning. A
non-* version pin discarded for a backend-provided bundled copy warns
naming the substitution. The -I global-include promotion and the
start-group link-order contract are documented.
The in-process cache served a stale graph across mtime and version
changes, which the script test suite pins; the saved re-parse was tens
of milliseconds.
A cache-arduino8266 composite action mirrors cache-esp-idf (one key
resolver, dev-writes/PR-restores) and both jobs pin
ESPHOME_ARDUINO8266_PREFIX so YAML and Python agree on the path by
construction. esp32 provides the native_toolchain_module hook, so the
shared dispatcher never names a backend and the unclaimed-native raise
loses its esp-idf carve-out; the esp8266 decode and run_compile hooks
resolve through the same seam. run_compile resolves ccache once and
threads it to the generator, env, and idedata; the compdb rule names
follow the shared kinds; the pio-options warner inlines into the driver
and derives its consumed set from the core routing constant; the
per-toolchain CI narrowing shares one body; create_components_graph is
memoized per run; the decode rate-limit drops its math sentinel.
The build.ninja python variable goes through strip_win_long_path_prefix
like every other emitted binary path (the NSIS launcher's extended
prefix cannot be spawned by cmd.exe). Compile rules are named by the
shared source kinds directly, retiring the identity translation table
and its drift test. write_project takes the caller's resolved ccache,
get_flash_ld_path reads the InstalledPaths the ninja file linked
against instead of re-resolving the version, the board backstop checks
the one table it indexes with an honest comment, and cc/cxx/ar paths go
through the shared toolchain_tool accessor (gaining the Windows
suffix).
toolchain_tool owns the bin path and Windows suffix now; _BuildConfig's
knob and MMU fields are required since the only constructor always
passes both; the tests' set-flags-then-resolve idiom collapses into one
_resolve helper (12 sites).
toolchain_tool carries the bin/xtensa-lx106-elf-<name> pattern and the
Windows suffix that four call sites previously spelled out (only one of
which handled the suffix). ccache_env and get_build_env accept the
already-resolved ccache path so run_compile can resolve once instead of
paying the PATH scan and runnability probe three times per build.
NATIVE_ARDUINO_PIO_OPTIONS is the single list the routing reads; the
backend's ignored-option warning consumes it too, so honoring a new key
is one edit instead of two synchronized ones.
The dict-shorthand bundled preference ran after convert_libraries had
already resolved the dependency through the registry, so a real
{"Wire": "*"} manifest still fetched or failed on a registry Wire (the
mocked test hid it). LibraryBackend gains an optional provides hook the
walk consults for owner-less dependencies; the arduino backend answers
from the framework tree and the bundled copy is added after emit. A
real-converter test with a local library pins that the registry is
never touched. _emit also validates the manifest shape by name before
apply_extra_script dereferences it.
Published configs override it to reserve a filesystem region or correct
a board's assumed flash size; the routed value replaces the BOARDS
default, and a path is rejected since the name resolves via -L.
The routed override (published configs pin 160000000L for
timing-sensitive integrations) reaches -DF_CPU via a small _pio_option
reader; the default stays the audited 80 MHz all 45 boards ship.
Published configs override it to reserve a filesystem region
(SmartIntercom's eagle.flash.4m2m.ld) or to correct a board's assumed
flash size (2 MB ESP8285 plugs pinning eagle.flash.2m.ld); dropping it
changes the flash layout under them.
Dropping it was a real regression: many published ESP8266 configs pin
board_build.f_cpu: 160000000L for timing-sensitive integrations
(MHI-AC-Ctrl documents the 160 MHz requirement in its example), and the
warn-and-drop left those devices at 80 MHz. The option now routes into
CORE.platformio_options under toolchain: arduino for the generator to
consume; other native toolchains keep the warning.
F_CPU gets the same drift-audit note as _MMU_DEFAULT: upstream reads it
from the board manifest, and all 45 supported boards ship 80000000L.
The plain-form refusal also covers -e, -s, -static, and -nostartfiles,
which gcc accepts inertly on a -c line.
tokens is a required parameter on _flag_defines and _project_flags now,
so the per-consumer re-lex the shared lexer was meant to end cannot
come back. The cache stamp covers the gcc binary's size and mtime via
a shared _stat_sig (an in-place toolchain re-extraction regenerates
the script like a header edit). A bare -I or -D from an empty argument
is warned and skipped instead of letting gcc eat the next flag.
_defines_flags documents that its tokens carry shell-level escaping and
must be emitted unquoted, and _MMU_DEFAULT notes why the board-manifest
values are hardcoded.
The cached-diagnostic read follows the same regenerate-never-abort rule
as the other cache reads (a damaged note must not fail an incremental
build). The unreadable-header path warns naming the header and the
clean-all fix instead of leaving an inexplicably uncached build at
debug level. _project_flags' docstring states which return values are
pre-quoted and which the caller owns.
write_project rejects a board missing from either table, so a new test
asserts the two stay exactly equal (the subset check is how d1_wroom_02
went missing). The hardcoded --flash_freq 40 carries the audit note:
all 45 supported boards ship 40 MHz, and a platform bump adding one
that does not must revisit it.
Both testing-mode segment patches get the RuntimeError-to-EsphomeError
wrap the ratetable surgery already had. Plain-form -T/-Xlinker/-u flags
raise naming the -Wl, form instead of warning into an inert -c compile
line. _lexed_build_flags and _unflag_tokens go through the shared
lex_build_flags so a future lexing fix cannot miss this path. The MMU
conflict message names dropping the variant knob when MMU_CUSTOM is
already set. An unreadable (not absent) header forces a cache miss
every run instead of pinning the stamp to a constant, and preprocessor
warnings persist beside the stamp and re-emit on cache hits. The drift
docstring states what the tests actually pin.
The bundled-dependency visibility warning also reads the
library.properties depends= key the shared parser returns raw, since
properties is the format bundled libraries actually ship. A bundled
manifest declaring an extraScript warns that it never runs (a new
Known-deviations entry). A version-less manifest dependency the config
separately requests as a top-level library is already in the build, so
the skipping warning no longer fires for it and cannot teach users to
ignore the real drop.
The PlatformIO branch of _decode_pc now routes through the same
rate-limited warning as the native one; raw undecoded crash addresses
with no stated reason were undiagnosable at default log level. The CI
cache key uses the assignment form so errexit catches a resolver
failure instead of echo swallowing it into a degenerate key, with a
non-empty check, and the cache comments stop claiming a ccache store
the seed job never populates (it saves before any compile runs).
A top-level library the converter dropped always makes the firmware
wrong, so resolve_libraries raises naming the missing requests instead
of warning toward link errors, and a resolved component without its
node_key fails as a converter bug rather than silently substituting the
mismatched canonical name. includeDir and srcFilter join srcDir and
flags in raising a named error on malformed values, and dot_a_linkage
parses through the same strict table as libArchive so a typo warns and
keeps the archive default instead of flipping link semantics.
The arduino backend consumed the private _node_key across packages to
reconstruct the request-side key; the thin public wrapper keeps both
sides of the contract on the same API surface.
A non-dict manifest or build section raises an EsphomeError naming the
library instead of an AttributeError. ConvertedLibrary remembers the
request-side node key it resolved from, so the converter-drop warning
diffs requests against those keys; a bare name resolving to a canonical
owner__pkg is no longer falsely listed missing. A bundled library whose
manifest declares dependencies warns that the walk never runs for it (a
no-op on the ESP8266 core, not on every core this module serves).
Nothing at this commit enforces MIN_FRAMEWORK_VERSION during
validation; the native backend's validator arrives with the wiring PR.
The comment no longer invites skipping it.
functools.cache froze the ESPHOME_CCACHE_ENABLE/PATH decision for the
process lifetime, which a long-lived host like device-builder never
restarts, and forced a cache_clear fixture in the tests. The docstring
and test now pin the uncached delegation.
file["system"] normalized None and str but let an int or dict fall
through to the in test, where an int raises TypeError and a dict
becomes a key test; both now raise the same Unexpected-response error
as the other shape guards.
The docstring claimed exact registry names for every core above 2.6.2;
2.6.3, 3.0.0 and 3.0.1 ship with non-zero trailing components the
formula cannot produce. Name them, say which callers are safe (the ~
range on the PlatformIO path, the floored check_and_install), and
credit the floor with covering 3.0.0/3.0.1 as well as the absent 3.1.0.
<= 2.6.2 instead of < 2.6.3: a 2.6.2 pre-release sorts above 2.6.2 and
belongs to the package-major-3 encoding, so the helper now describes
the same set as _format_framework_arduino_version and a 2.6.2-b1 pin
formats as it does on dev. The module never logged, so its _LOGGER and
logging import go.
cv.Invalid now carries path=[CONF_VERSION] so the rejection lands on
the framework version line the comment promised, matching the esp32
convention. framework_package_version raises for cores before 2.6.3
instead of encoding them wrongly for a caller that skipped the
PlatformIO path's era guard. get_build_env joins PATH from filtered
parts so an absent PATH cannot leave a trailing separator that the
shell reads as the current directory.
The pre-lexed tokens feed both _flag_defines and _project_flags so a
malformed entry warns once per generation. An unflag that hits nothing
in the user flags or any framework set (a typo, or -DUSE_FOO against
-DUSE_FOO=1) is named instead of silently leaving the flag in force.
A raw MMU_* define alongside a PIO_FRAMEWORK_ARDUINO_MMU_* knob, or a
raw TCP_MSS/LWIP_* alongside the lwIP knobs, would win the compile line
while the linker script or prebuilt library keeps the knob's values;
both now raise like the no-knob case. _MMU_VARIANTS holds tuples and the
resolved config gets a copy, so nothing aliases the module table.
_lexed_build_flags lexes once and both consumers accept the tokens, so
a malformed entry warns once per build. The linker-script stamp covers
the header's size and mtime, an unreadable or non-UTF-8 cache
regenerates instead of aborting, plain-form linker flags warn that they
are not routed to the link line, and a moved rate-table anchor surfaces
as a named build error.
parse_enable_env carries the 1/true/yes/on and 0/false/no/off table for
both knobs, so IDF_CCACHE_ENABLE=off disables instead of reading as
truthy and suppressing the shared opt-out. The ccache module docstring
and the ESP-IDF _ccache_env docstring now describe the precedence this
PR actually ships, and the probe test duplicated by the helper move is
dropped from the PlatformIO toolchain tests.
elf2bin.py and eboot.elf join the existence checks so a half-extracted
package fails by name before the compile instead of at the
second-to-last edge. get_project_cxx_compile_flags() values go through
_shell_token like every other flag source. d1_wroom_02 (2 MB, in
ESP8266_BOARD_BUILD and buildable under PlatformIO) joins BOARDS so the
native board guard stops rejecting it, and the guard comment states
what config validation actually checks.
Unknown or conflicting VTABLES_IN_* defines raise instead of warning
and picking arbitrarily (a typo previously won the sorted pick and died
in the SDK header's #error). Custom MMU sizes without the CUSTOM knob
raise too: PlatformIO warns but its defaults win the compile line,
while here the user's tokens would win and compile against a layout the
linker script does not implement. The cached linker script is re-checked
for its SECTIONS block so a truncated file regenerates, the
testing-mode flash-ld read gets the same OSError guard as the gcc
spawn, the ParseFlags docstring states its actual coverage, and the ld
test mocks are quiet so the clean path asserts no warnings.
check_library_data's platform filter raises IncompatiblePlatform (an
InvalidLibrary subclass) so the arduino backend branches on the type
instead of substring-matching the message. The {"Wire": "*"} dict
shorthand resolves to the bundled library like PIO's
process_dependencies instead of a registry lookup. libArchive parses
booleans and true/false strings and warns on anything else (bool(str)
made "false" archive). A declared-but-falsy srcDir raises instead of
silently probing. The converter-drop warning names the missing requests,
and the bundled-walk comment states what PIO actually does.
The installer's EsphomeError becomes cv.Invalid at the config
validator, and test_ccache_env clears the ambient environment so a
developer's exported CCACHE_DIR cannot fail it.
shell_token now picks the quoting style per platform: single quotes on
POSIX (sh expands nothing inside them, so backslash runs, $VAR, and
backticks reach the compiler exactly as lexed, matching SCons's
no-shell spawn) and the CreateProcess argv rule on Windows. A test
round-trips every case through a real /bin/sh.
The ninja and ccache runnability probes collapse into one
tool_version_runs helper in framework_helpers. The registry names a
non-dict top-level payload like the inner guards, and the expect layout
check also runs on marker hits so a marked install that later lost
files fails by name. The ESP-IDF ccache gate defers to
resolve_ccache_path so ESPHOME_CCACHE_ENABLE=0 disables ccache there
too, with IDF_CCACHE_ENABLE still taking precedence.
Platforms normalize toolchain-sensitive keys differently (the esp32
board name, for one), so a config validated under a mismatching
--toolchain must not overwrite the cache while the sidecar keeps the
compile's toolchain. The guard lives in _refresh_sidecar where the old
sidecar is already loaded; legacy sidecars without the field are
unaffected.
The comment claimed the refresh's sidecar records the CLI toolchain;
_refresh_sidecar returns early when a compile-written sidecar exists,
so the compile's toolchain stands. Reword the comment and the test
docstring to match.
Registry version and file entries that are not dicts now raise the same
Unexpected-response error as the other shape guards instead of an
AttributeError traceback. install_package rejects an empty expect so
every install is layout-validated before the marker is written.
ESPHOME_CCACHE_ENABLE values outside 1/true/yes/on and 0/false/no/off
warn and are treated as unset instead of bool(str) flipping them to
enabled. The strip_win_long_path_prefix test moved next to the function
in test_framework_helpers, and a stale comment naming its old module is
fixed.
The autouse fixture pinned the idf and sdk-nrf prefixes but not the new
arduino8266 one, so running the suite with that env var exported deleted
the pointed-to install. Also adds the clean_all removal test the other
two native installs already have.
The upload/logs cache split: an explicit --toolchain still skips the
read and runs the per-platform validators, but the freshly validated
config is saved again so a later plain run keeps the fast path (the
sidecar records the resolved toolchain). check_supported_toolchain is
module-private now since it has no caller outside config_validation,
and require_platformio_toolchain's docstring describes what actually
happens when another platform's toolchain is passed.
The upload path now checks CORE.firmware_bin exists before invoking
esptool and names the fix (compile first) instead of failing inside
esptool. compile_program raises when a native toolchain resolved but no
platform backend claimed the build rather than falling through to a
mis-configured PlatformIO project. The addr2line failure warning now
includes the exception so the cause is visible without debug logging.
CI: the native toolchain cache is restore-only on PRs and seeded by a
new dev-push job, mirroring seed-apt-cache; the key resolves the pinned
core and toolchain versions from code instead of hashing framework.py.
determine-jobs main() output tests now pin the esp8266_native pair.
description:Write ESPHome automation building blocks - triggers (on_* keys), actions and conditions. Use when adding or changing an on_* trigger, an action such as my_component.do_something, a condition, a callback registration method (add_on_*_callback), or code that calls build_callback_automation, build_automation, register_*_action or register_*_condition.
---
# Automations: triggers, actions and conditions
Automations have three building blocks: **Triggers** (fire when something happens), **Actions** (do
something), and **Conditions** (check if something is true). The helpers named here are documented in
their docstrings in `esphome/automation.py`.
## Triggers: callback method (preferred)
Use `build_callback_automation()` for simple triggers. This eliminates the need for a C++ Trigger class by
using a lightweight pointer-sized forwarder struct registered directly as a callback. No `CONF_TRIGGER_ID`
| `CallbackManager<void(Ts...)>` | 12 bytes (empty `std::vector`) | Callbacks are always or almost always registered |
| `LazyCallbackManager<void(Ts...)>` | 4 bytes (`nullptr`) | Callbacks are often not registered (common case) |
| `AutomationList<N, Ts...>` + `build_automation_lists` | 4 bytes per automation + count, none when unused | A hub fires its own `on_*` automations with no filter (4 B per automation vs 8 B per `Callback`) |
| `automation.new_automation(args, conf)` + the parent's own fixed table | the table entry (pointer plus its own data) | Like `AutomationList`, but each automation needs a stored value of its own, e.g. a threshold the parent compares against |
Both live in `esphome/core/helpers.h` and accept `std::function`, lambdas and forwarder structs through
their templatized `add()` method. `LazyCallbackManager` is a drop-in replacement for `CallbackManager` that defers allocation until the first
callback is added. Prefer it for entity-level callbacks where most instances have no subscribers.
## Triggers: Trigger class method
Use `build_automation()` with a `Trigger<Ts...>` subclass only when the forwarder needs **mutable state
beyond a single `Automation*` pointer** (e.g. edge detection tracking previous state, timing logic).
description:Decide whether an ESPHome change breaks public API and handle it - what counts as public C++ and Python API, the deprecation window, ESPDEPRECATED for C++, cv.rename_key and deprecation warnings for config keys, and the migration notes a PR needs. Use when removing, renaming or changing the signature of a config option, a public C++ member or a core Python helper.
---
# Public API and breaking changes
Full policy: https://developers.esphome.io/contributing/code/#public-api-and-breaking-changes
## What is public
- **C++ in components:** only features documented at [esphome.io](https://esphome.io) are public API.
Undocumented `public` members are internal.
- **C++ core and base classes** (`esphome/core/`, `Component`, `Sensor`, etc.): all `public` members are
public API.
- **Components with global accessors** (`global_api_server`, etc.): all `public` members are public API
(except config setters).
- **Python:** all documented configuration options are public API. Python code in `esphome/core/` that
existing core components use is stable API. Other Python code is internal unless explicitly documented
for external component use.
- Changing the signature of a lambda that code generation injects is not a breaking change.
## Policy
- Aim for a **6-month deprecation window** when possible.
- Clean breaks are allowed for signature changes, deep refactorings and resource constraints.
- The migration path goes in the PR description (it generates the release notes).
- A blog post is required for core/base class changes or significant architectural changes.
| set warning / `return` / clear warning around a bus result | `if (!this->status_comm_result(ok)) return;` |
| hand-written `set_x_sensor()` plus `x_sensor_` member | `SUB_SENSOR(x)`, `SUB_BINARY_SENSOR`, `SUB_TEXT_SENSOR`, `SUB_SWITCH`, ... |
| `if CONF_X in config: s = await sensor.new_sensor(config[CONF_X])` plus setter | `sensors = sensor.sub_sensors(config)` then `await sensors(CONF_X, hub.set_x_sensor)` |
| `if (x := config.get(CONF_X)) is not None: cg.add(var.set_x(x))` | `cg.add_optional(var.set_x, config.get(CONF_X))`, when the setter takes the value unchanged |
## Python conventions
- Type-annotate every new function signature (params and return), new dataclass
@@ -125,10 +76,6 @@ helper; `AGENTS.md` section 4 and the `component-codegen` skill have the detail.
`if (blah := config.get(CONF_BLAH)) is not None:`.
- Reuse existing validators from `config_validation.py` (`cv.rename_key`,
`cv.has_exactly_one_key`, etc.) via `cv.All(...)` instead of hand-rolling.
- A new or changed validator function that wraps, chooses or builds a schema
must be decorated with `@schema_extractor(...)` and handle `SCHEMA_EXTRACT`,
or `script/build_language_schema.py` silently drops the keys behind it (see
the `config-validation` skill).
-`esphome/const.py` is frozen: no new `CONF_` constants there. Define them in
the component's own `.py`, or in `esphome/components/const/__init__.py` when
shared. The same constant defined in three or more component files fails CI.
@@ -149,18 +96,6 @@ helper; `AGENTS.md` section 4 and the `component-codegen` skill have the detail.
- Config-only checks use the `validate.*.yaml` prefix; compiled checks use
`test.*.yaml`.
## Naming user-facing functions
Documented lambda methods, actions and config keys are public API, so a bad name
is hard to fix once released. Flag:
- A boolean where `true` means off (`display_off(true)`): name the positive
state, e.g. `set_power(bool on)`.
- A name that differs from a sibling component's name for the same operation.
Before a release, rename outright; after, follow the breaking-change policy
below.
## Breaking changes and public API
- Base classes under `esphome/core/` and documented config options are public
description:Write the Python side of an ESPHome component - __init__.py and platform files (sensor.py, switch.py, ...), DOMAIN and other metadata, to_code, hub child entities with sub_sensors and friends, pins and referenced components, hub-forwarding entities (ApplyButton, ApplySwitch), uart/i2c/spi constructor helpers, and CORE.data state. Use when creating a component or changing its code generation.
---
# Component code generation
Read https://developers.esphome.io/architecture/components/ for the component lifecycle and how a
component is registered. Schema rules and schema extraction are in the `config-validation` skill;
automations (`on_*`, actions, conditions) in the `automations` skill.
## Files
```
components/[component_name]/
├── __init__.py # Component configuration schema and code generation
├── [component].h # C++ header file (if needed)
├── [component].cpp # C++ implementation (if needed)
description:Write and run tests for ESPHome components - YAML build tests (test.*.yaml, validate.*.yaml, common.yaml) under tests/components, shared bus packages and CI test grouping, C++ gtest unit tests, and host integration tests including the uart_mock fixture. Use when adding or changing files under tests/, or when a code change needs test coverage.
---
# Component tests
## C++ unit tests
gtests live in `tests/components/<component>/*.cpp` and run with `script/cpp_unit_test.py <component>`; see
`tests/components/README.md`. Set the loop clock in a test with `LoopBlockingGuard{nullptr, nullptr,
now_ms}`; the runner resets it before every test.
## Integration tests
`tests/integration/` runs real firmware on the host. Serial components prove timeouts and recovery with the
`uart_mock` fixture (`tests/integration/test_uart_mock_*.py`). Its `responses` match on content every time
they see it; to answer a repeated request differently, use an `on_tx` lambda with a static counter and
`inject_to_rx_buffer` (see `uart_mock_ld2412_requery_after_giveup.yaml`).
description:Write or change ESPHome configuration validation - CONFIG_SCHEMA, cv.Schema, cv.typed_schema, custom validator functions, config keys and CONF_ constants, renamed or deprecated keys, and the language schema that script/build_language_schema.py extracts from them. Use whenever a change touches a component's schema or a validator in config_validation.py, automation.py or a component's .py files.
---
# Configuration validation
ESPHome validates YAML with `voluptuous` schemas built from `esphome/config_validation.py` (`cv`). The
same schemas are also walked by `script/build_language_schema.py`, which produces the JSON language schema
that editors and the dashboard use for completion and checking. A validation change is only finished when
both still work.
## Writing schemas
- **Type hints:** validators are typed like any other function: `def validate_x(config: ConfigType) ->
ConfigType:`, with `ConfigType` imported from `esphome.types`.
- **One schema per model:** when a component's options depend on a `model` key, use
`cv.typed_schema({...}, key=CONF_MODEL, default_type=...)` with each model's own keys and defaults, not
one schema plus a validator that rejects keys and fills defaults afterwards.
- **Reuse existing validators:** before writing a custom validator, check for an existing one in
`config_validation.py` and compose it in `cv.All(...)` rather than duplicating logic across components.
For example, rename a config key with `cv.rename_key(CONF_OLD, CONF_NEW, removed_in="2026.6.0")`, and
reject mutually-exclusive keys with `cv.has_at_most_one_key(...)` / `cv.has_exactly_one_key(...)`. See
how `api` composes `cv.has_exactly_one_key` + `cv.rename_key`.
The type names the dumper accepts, and what `SCHEMA_EXTRACT` must return for each, are in the
`hidden_schemas` branch of `convert()` in `script/build_language_schema.py`. The common ones:
| Type | Return for `SCHEMA_EXTRACT` | Example |
|------|-----------------------------|---------|
| `"schema"` | the schema (or a representative one) to walk | `api` `_outgoing_connection_schema`, `restore_state_schema` in `esphome/core/entity_helpers.py`, `model_schema_extractor` in `esphome/components/mipi/__init__.py` |
| `"enum"` | the dict of options | `sensor_entity_category` in `esphome/components/sensor/__init__.py` |
| `"one_of"` | the list of options | `cv.one_of` |
| `"maybe"` | `(validator, key)` for a scalar shorthand | `cv.maybe_simple_value`, `maybe_conf` in `esphome/automation.py` |
| `"templatable"` | the inner validator | `cv.templatable` |
| `"automation"` | the automation schema | `automation.validate_automation` |
Points to watch:
- The dumper matches the function object by `repr`, so the decorated function must be the exact object
placed in the schema. In a factory that returns a new closure per call, decorate the inner function (as
`restore_state_schema` and `cv.maybe_simple_value` do), not the factory.
- The decorator only records anything when extraction is enabled, and that flag is set before imports, so
the decorator must run at import time (module level or inside a factory called at module level).
- A schema chosen per config value (for example by `model`) has no single answer: return a representative
schema, as `model_schema_extractor` does.
- Plain value checks (`cv.All(cv.string, my_check)`) need nothing: the dumper still sees the surrounding
types and keys.
- `cv.Schema`, `cv.typed_schema`, `cv.ensure_list` and the `cv` helpers are already handled; prefer
composing them over a custom function, which also avoids the extractor work.
### Checking the output
Build the language schema into a scratch directory and look at the component's JSON (it takes about ten
description:Test an ESPHome display driver on a real panel with a camera. Use when a task asks to verify a display (LCD, e-paper, OLED, RLCD) on hardware, check orientation/rotation, partial updates, colours, contrast, or that text and shapes render correctly. Covers finding the USB port, flashing a test pattern, capturing the panel with any camera, and reading the photo.
allowed-tools:Read, Write, Edit, Bash, Glob, Grep
---
# Display testing with a camera
Point a camera at the panel to get visual evidence that a driver renders correctly. Any camera
and capture tool works; this skill covers the common ones. `$SKILL` below is this directory
(`.agents/skills/display-test`).
## Ask before using a camera
A camera can show people and rooms, not only the panel. Before the first capture, ask the user
for permission and for which camera to use. List the cameras the host has (the commands below
show how), name the one you intend to use, and wait for the user to confirm it. Do not pick a
camera yourself, and do not fall back to another camera if the chosen one fails; ask again.
The permission covers this session and the camera the user named. Ask again for a different
camera or in a new session. If the user declines, stop and report that the display was not
checked visually.
## Capturing a frame
Grab a still image, skipping the first few frames so auto-exposure and white balance settle.
Read the JPEG with the Read tool; the image is shown visually.
### macOS
The bundled Swift tool uses AVFoundation and avoids the frame-grabbing problems some USB
description:Choose ESPHome C++ containers and storage that avoid heap use and fragmentation on microcontrollers - std::array, StaticVector with cg.slot_list/SLOT_LIST, FixedVector, std::unique_ptr byte buffers, StringRef, RAMAllocator, and what to avoid (std::vector growth, std::map, std::deque, std::string, new (std::nothrow)). Use when adding or changing C++ members, containers, buffers, strings or registration lists, or when reducing RAM or flash use.
---
# Memory on embedded targets
ESP devices run for months with small heaps shared between Wi-Fi, BLE, LWIP, and application code. Over
time, repeated allocations of different sizes fragment the heap. Failures happen when the largest
contiguous block shrinks, even if total free heap is still large. We have seen field crashes caused by
this.
**Heap allocation after `setup()` should be avoided unless absolutely unavoidable.** Every
allocation/deallocation cycle contributes to fragmentation. ESPHome treats runtime heap allocation as a
long-term reliability bug, not a performance issue. Helpers that hide allocation (`std::string`,
`std::to_string`, string-returning helpers) are being deprecated and replaced with buffer and view based
APIs.
## Container guidelines
1.**Compile-time-known sizes:** use `std::array` instead of `std::vector` when size is known at compile
time.
```cpp
// Bad - generates STL realloc code
std::vector<int> values;
// Good - no dynamic allocation
std::array<int, MAX_VALUES> values;
```
Use `cg.add_define("MAX_VALUES", count)` to set the size from Python configuration.
**For byte buffers:** avoid `std::vector<uint8_t>` unless the buffer needs to grow. Use
`std::unique_ptr<uint8_t[]>` instead. It does **not** provide bounds checking or iterator support like
`std::vector<uint8_t>`; use it only when you do not need these features.
description:Test an ESPHome change on real boards over USB. Use this whenever a task asks to test, verify, reproduce, bench, A/B or soak a pull request, a branch or a crash on hardware, or mentions flashing, serial logs, esphome upload, a captive portal or fallback AP test, or results to post on a PR. Covers finding the serial ports, building from the right checkout, flashing each chip family (including the LibreTiny kits that stock tools cannot flash), bounded log capture, proven test flows and how to report results.
allowed-tools:Read, Write, Edit, Bash, Glob, Grep
---
# Hardware testing
The goal of a hardware test is evidence a reviewer can trust: which build ran on which board, what was
driven, what the logs show, and the same flow on a `dev` build when a quirk could be pre-existing.
`$SKILL` below is this directory (`.agents/skills/hardware-test`).
## Ground rules
1.**Serial ports cannot be shared.** One bounded capture per port, stopped before any flash on that
port, never an open-ended `esphome logs`. Stop a capture with `pkill -f "[s]erial_capture.py <port> "`,
naming the port so a capture on another board survives; the bracket keeps the pattern from matching
the shell that runs it. When more than one agent session shares the bench, run a copy of the script
under a session-specific name (`cap<topic>.py`) so another session's `pkill` cannot take your capture
down. Never run `esphome logs` in a `while true` loop: the loop respawns it and holds the port.
2.**Credentials stay local.** Put a `secrets.yaml` next to the scratch configs and reference it with
`!secret`. A copy of a device's own config keeps using the secrets file next to that config: two
checkouts can hold different networks, and the wrong one leaves the device unreachable after the OTA. Never paste a config with credentials into a PR, never commit scratch configs or logs.
3.**Build from the tree under test.**`python -m esphome` puts the current directory first on
`sys.path`, so run it with the checkout or worktree as the working directory and pass the yaml by
its absolute path. A venv `esphome` entry point may import an installed snapshot instead of the
checkout. Confirm with `python -c 'import esphome; print(esphome.__file__)'`, and for an A/B grep
the build's `src/` for the change before trusting any number. Tool calls sent together share one
shell, so a `cd` in one can land another's compile in the wrong tree: pass `PYTHONPATH=<worktree>` on
every compile and upload command and print `esphome.__file__` into the compile log.
4.**PR build and baseline build get different `name:` values.** Builds share
`.esphome/build/<name>`, so the same name means `esphome upload` can flash the wrong binary. Saved
Wi-Fi credentials are keyed by the config hash when the config has `networks:`, so a test that must
start unprovisioned gets a fresh name every run. A config with `ap:` only and no `networks:` uses one
fixed key shared by every such config on the board, so a new name does not help there: wipe once at
boot with an `on_boot` lambda that resets the preferences and saves a marker key
(`templates/dut-ap-only-wipe-once-bk72xx.yaml`).
5.**Flash and RAM deltas are measured against the PR's merge base** (`git merge-base origin/dev HEAD`
checked out in a throwaway worktree), with the baseline commit named in the table. When a delta
looks odd, diff symbol sizes (`nm -S -C --size-sort` from the matching toolchain) before quoting it.
When a change moves storage between the heap and static RAM, the static number alone misleads: also log
the free heap at the end of setup (`on_boot` at priority -100) over three boots of each build and report
the total in use (static RAM plus heap used).
6.**Prove a quirk is pre-existing before blaming the PR:** run the identical flow on a `dev` build in
the same session and say so in the results.
7.**Pipe safely.** Set `PYTHONUNBUFFERED=1` on any `esphome run` or `esphome logs` whose output is
piped, or lines land minutes late. Bound every long command with `timeout` (coreutils) or
`perl -e 'alarm shift; exec @ARGV' <seconds> <command>` on macOS without it. Save an exit code with `rc=$?`
right after the command: in `echo "$(date +%T) rc=$?"` the command substitution runs first and `$?` reads 0.
## Workflow
1.**Find the boards:**`python $SKILL/scripts/ports.py` (any Python with pyserial, for example the
ESPHome venv). Port names follow the USB hub port and change on every replug, so never reuse a name
from an earlier session. CH340 adapters (D1 mini, BW15 kit, BK7238, CB3S) all look the same; tell
them apart with a 10 s capture using `serial_capture.py --reset` (ESP8266 prints its ROM banner at
74880 baud, LibreTiny boards print `[LT]` and ESPHome lines, BK7238 prints its bootloader version).
`ports.py --wait 2e8a` blocks until a Pico re-enumerates.
2.**Make a scratch directory** outside the checkout, with `secrets.yaml` and the templates you need.
One yaml per board per variant, file name equal to `name:`.
Keep the RAM and flash lines at the end. Two ESP-IDF or two LibreTiny builds at the same time race
over tool caches; run them one at a time.
4.**Flash** per chip family (table below). Stop the capture on that port first.
5.**Capture** with `serial_capture.py PORT SECONDS OUT` in the background, then poll the file with
`grep` in an `until ... sleep 2` loop with a cap. USB CDC boards (Pico, nRF52, native USB ESP32)
drop the port on reboot; the script reopens it until its deadline.
6.**Drive** the test: a second board running a client config, `curl`, an aioesphomeapi script that
presses a template button and subscribes to logs, MQTT publishes, or `interval:` and `on_boot:`
automations in the config itself.
7.**Analyze** with `grep -a -c` counts over the capture and keep an `-events.txt` extract of the lines
that matter. Decode crashes with the platform's `addr2line` (see `reference/boards.md`).
8.**Report** on the PR. Every result comment starts with this two-line GitHub alert, then a blank line:
```
> [!NOTE]
> Automated testing results
```
Then one short sentence saying what was tested (builds and SHAs, board, what was driven), a results table
(build SHA, board and framework, what was driven, counts, result, plus the `dev` control row), short bullets
for the findings, and the setup and exact flow in a collapsed `<details>` block. Answer any question asked on
the PR directly in the comment. No credentials, SSIDs, IP addresses, port names or other bench details.
## Flashing by chip family
| Board | Flash | Log |
|---|---|---|
| ESP8266 D1 mini (CH340) | `python -m esphome upload <yaml> --device <port>` | `serial_capture.py` at 115200; `esphome logs` on some CH340s emits garbage after the RTS reset. A failed esptool write can wedge the adapter until the board is replugged. `esptool erase_flash` before a test that needs an unprovisioned device: the preference sector survives a reflash |
| ESP32 devkit (CP2102 or CH340) | `esphome upload --device <port>`, ESP-IDF or Arduino | `serial_capture.py`. Default Wi-Fi light sleep delays API driven actions by seconds; judge timing from device logs. `power_save_mode: none` for OTA overlap tests |
| ESP32-S3, C3, C6, H2 native USB | `esphome upload --device /dev/cu.usbmodem*` | `logger: hardware_uart: USB_SERIAL_JTAG`. The USB console can wedge when the app takes over the console (0 bytes, esptool `No serial data received`); replug, or watch through `esphome logs --device <name>.local` |
| Pico W / Pico 2 W | `esphome upload --device <port>` (1200 baud touch); first flash ever via BOOTSEL and `--device /Volumes/RPI-RP2` or `RP2350` | port re-enumerates after upload: `ports.py --wait 2e8a` then capture |
| BW15 kit (RTL8720CF, rtl87xx) | `~/.platformio/penv/.libretiny/bin/python $SKILL/scripts/flash_rtl87xx_kit.py flash write -d <port> <build>/.pioenvs/<name>/*.uf2`. Plain `esphome upload` and plain ltchiptool time out because opening the port asserts DTR/RTS and holds the chip in reset | `serial_capture.py --released`, never `esphome logs` |
| BW12 and other AmebaZ modules (RTL8710BX, rtl87xx) on a plain USB-UART | `~/.platformio/penv/.libretiny/bin/python -m ltchiptool flash write -B 115200 -b 115200 -t 120 -d <port> <uf2>`, started first, then strap by hand: CEN to GND, TX2 to GND, release CEN, release TX2. Without `-B 115200` the system data read at 1.5 Mbaud fails on CH340s (`Data read error at 0x9000`) and nothing is written. A 1 MB image takes about 80 s. A development board with DTR and RTS wired to CEN and the download pin needs no strap | `serial_capture.py`; a reset pulse does not reach CEN on a plain adapter, so a capture starts mid-run |
| BK7238 / CB3S (bk72xx, RTS on CEN) | `~/.platformio/penv/.libretiny/bin/python $SKILL/scripts/flash_bk72xx.py flash write -d <port> <uf2>` (about 75 s) | logs are on UART2 by default; `logger: hardware_uart: UART1` puts them on the flashing adapter. `serial_capture.py --reset` gives a chip-enable reset; a plain open may or may not reset it, check for the bootloader banner |
| LN882H | ltchiptool forces 1 Mbaud for writes; weak adapters need `LN882H_YM_BAUDRATE` patched to 115200 in the penv's `ln882htool.py`; short BOOT to GND while the tool connects | OTA after the first flash |
| nRF52840 Dongle (PCA10059) | Nordic `nrfutil nrf5sdk-tools dfu usb-serial` on the bootloader port, or `dfu: true` in the config so the 1200 baud touch pin-resets into the bootloader; adafruit-nrfutil is refused | app has two CDC ports, ESPHome log on the second |
| Adafruit-style nRF52 (UF2 bootloader) | `UF2BOOT` volume or adafruit-nrfutil; `dfu: true` works through GPREGRET | single CDC port |
| Any reachable device | `esphome upload <yaml> --device <ip or name.local>`; `--file <bin>` for a prebuilt image | `esphome logs --device <ip>` bounded with `timeout` |
ESP-IDF OTA image: `.esphome/build/<name>/build/firmware.ota.bin`. LibreTiny OTA port is 8892.
## Scripts in `$SKILL/scripts`
- `ports.py`: list ports with VID:PID, product, serial and a board guess; `--wait VID[:PID]`,
`--product NAME`, `--serial SN` (one specific board, whatever path it comes back under) and
`--serial-of PORT` (the serial number behind a path).
- `serial_capture.py PORT SECONDS OUT [--baud] [--released] [--reset] [--append] [--ms]`: bounded timestamped
capture (`--ms` adds milliseconds for timing work), ANSI stripped, `LOG_SECRET` text written as `<secret>`, reopens a vanished port, exits at the
deadline. OUT is truncated unless `--append` is given, so counts over it cover one run.
- `flash_rtl87xx_kit.py`: ltchiptool wrapper for the BW15 kit whose CH340 drives CEN (RTS) and the boot
strap (DTR). `flash read` needs the family `realtek-ambz2`; `flash write` does not.
- `flash_bk72xx.py`: ltchiptool wrapper for BK72xx boards whose RTS drives CEN. It misses the link now
and then; retry up to three times.
- `api_button.py HOST KEY list|press NAME [...]`: list a device's template buttons or press them over the API; the
recipes drive their test configs with it.
- `apiconn.py`: `connect(host, tries)` (API connect and login with retries, `API_KEY` from the env) and `log_text()`
(an API log line without ANSI codes), imported by the newer API drivers.
capture decoder, RTL8720C ROM dump versus report matcher), `api_outgoing/` (wake cycle and Wi-Fi drop timing readers), `ld2450/` (zone number driver, loopback harness driver with number and binary sensor states and TX frame summarizer, frame stream harness with presence hold and link loss drivers, BT MAC driver), `api/` (raw plaintext native API client for user defined actions, store_yaml recovery client over Noise or plaintext), `http_request/` (update manifest server, update state driver, request test server, action case driver, concurrency, reboot, update check and OTA drivers, `ota_auth/` OTA request cases with basic auth), `modem/` (button and sensor driver over the plaintext API, also used for the BLE MIDI boards), `dfrobot_sen0395/` (API driver and case runner), `homeassistant_light/` (direct bulb driver, held-report case driver), `usb_audio/` (URL player), `speaker_media_player/` (API player driver, pipeline task case runner), `web_server/` (SSE client, request mix, SSE and HTTPD case runners), `mqtt/` (`ha_discovery/`, `disable/`, `retained_burst/` and `reconnect/` runners; broker config with TLS, client-certificate and WebSocket listeners, test CA with server and client certificates, transport case runner, flash-and-drive runner, CLI TLS check without a board, MAC suffix runner with per-variant configs for two boards, `shutdown/` shutdown message runner and matcher), `udp/` (datagram recorder, `udp.write` payload checker and case driver), `entity_icons/` (icon and device class readback over the API, MQTT discovery, web_server and dump_config, with a comparer), `uart_sim/` (API driver and flash, capture and drive runner for the simulated UART sensors), `remote_base/` (loopback runner and API log decoder per sent frame), `xiaomi_ble/` (frame and header generator with the expected counts, injecting, advertising and receiving configs, counter and runner), `ble_terminate/` (proxy and peer configs, connect and disconnect driver, peer disconnect matcher, runner), `i2c_slave/` (D1 mini slave and BK7238 master configs), `wpa3_ap/` (PMF AP with join and leave logging, BK7238 and CB3S DUTs, runner and summary), `wisp_ap/` (WISP upstream with AP buttons, WISP and ALWAYS_ON DUTs, client, runner and per phase summary), `wifi_restart/` (WPA3 mixed WISP upstream, static IP DUTs with syslog, runner and summary), `ota_timing/` (OTA upload timer with deflate, raw and TCP modes, API build check, A/B runner and median summary, `inflate_bench/` on-device inflate benchmark with stream generator, configs and syslog runner), `gpio_hold/` (ISR pin hold self test with ESP32 and S3 configs), `ble_event_inject/` (GAP and GATTC event injector header and proxy package), `light_sleep_wake/` (LEDC wave and edge timer for light sleep wakeup on a C3, external wave generator and per-call edge detail probe for peripheral power-down on the P4), `manual_ip/` (station and AP manual IP configs, DNS probe and runners), `web_handlers/` (late handler, web_server, captive_portal and prometheus variants, web OTA runner), `bk72xx_flash/` (flash readback with retries, per partition diff of two dumps), `ble_server_security/` (secure GATT server and pairing client configs, driver, summary and runner), `prometheus/` (datetime metrics configs and driver), `zephyr_ble_server/` (nRF52 server, looping ESP32 client and UF2 flash and count runner), `ble_adv_triggers/` (advertiser with six raw frames, ESP32 and BK72xx receivers with eleven triggers, runner and checker), `sun/` (longitude sweep package, runner and checker), `sensor_filters/` (every filter shape with a fixed input sequence, per-board packages and runner), `led_driver_sniff/` (SPI slave capture header with selftest, config generator, window comparer and runner), `hub_input_sim/` (keypad, wiegand and rdm6300 simulation on a D1 mini, API log runner), `value_range/` (on_value_range sequence with a restart, runner), `ezo_fake_bus/` (fake EZO I2C bus and step config, runner), `twai_self/` (TWAI self-reception header, on_frame config and counter runner), `ota_automations/` (OTA automation package for both OTA platforms, four case runner, repeated web upload runner and per case summary), `output_button/` (pad edge timer header, button package and runner), `homeassistant_button/` (raw plaintext API client for action calls and wizard inputs, package and runner), `uart_button/` (UART0 loopback package, API driver with TX and RX counts, runner), `interval/` (plain and delayed interval package, timing summary and runner), `smp_udp_ota/` (SMP reset fault injector, OTA runner, per mode result check, MCUboot swap limit, running image check and slot blanking UF2, Thread device config), `ld2420_sim/` (fake LD2420 answering the hub's commands on a UART0 loopback, API state recorder, two-build comparer and runner), `nrf52_tcp_probe/` (TCP and UDP socket probe config, build and log check, API check over Thread), `rs485_frame/` (simulated DLE/STX/ETX bus device, frame and gate cases, runner), `hx711/` (fake HXLine truncated
## Templates in `$SKILL/templates`
- `dut-fallback-ap.yaml` (package): a device that never finds its network, falls back to its AP and
serves the captive portal. `example-dut.yaml` is the per-run file with the platform block.
- `client-wifisave.yaml` (package): joins that AP, posts N bogus `/wifisave` requests, then the real
credentials from `!secret`. `example-client.yaml` is the per-run file. `bogus_saves: "1"` gives a
plain provisioning check.
- Per-topic device configs, each described in `reference/recipes/<component>.md`: `online-image-dut.yaml`,
and the directories `store_yaml/` (package with secrets one directory up, per-board files),
`http_request_update/` (manifest URL setter shapes, per-board files), `modem/` (modem only and modem plus
Wi-Fi boot checks), `ble_midi/` (fake BLE MIDI instrument and the client), `http_request_action/` (request
action package with per-board files and an OTA flash variant), `led_wire/` (DUT config and header that reads the
LED strip pads back with RMT RX and an SPI slave), `uart_sim/` (simulator header, common package, per-build parts and run files) and `remote_base/` (common
package, transmit script, per-board templates and the ESP8266 software sender), `i2c_fake/` (fake I2C device
headers with their test configs, see `fake_i2c.md`).
- `microphone_http_stream/`: base config, fake counter and USB microphone variants and the multi-reader client,
described in `reference/recipes/microphone_http_stream.md`.
## When the PR keeps moving
Record every head in the stack (`gh pr view N --json headRefOid`), build, and recheck before flashing; if anything
moved, wait five minutes and check again so the test runs on a settled head. When a head moved only by merging dev,
compare the PR's own diff (`git diff $(git merge-base OLD origin/dev) OLD` against the same for NEW); if it is
identical and the dev commits do not touch the code under test, earlier results still describe it, but rebuild so the
comment quotes the exact commits. Fetch `origin dev` before computing a merge base.
## When something looks wrong
- A capture stopped early or a line is missing: check the timestamps before trusting it.
- A script that pipes into `tee` never ends although its work is done: a background command written as
`(cd dir && server & echo $!)` backgrounds the whole `cd && server` list, so a shell stays behind holding the
pipe open and `$!` is that shell, not the server. Write `(cd dir && exec server > out 2>&1 < /dev/null) &`.
- The client never joins the AP: another board may still run an AP with the same SSID, or the device
booted with saved credentials because the name was reused. One AP device at a time on the bench.
- Builds come out byte-identical between branches: the wrong tree was imported (rule 3). The reverse
too: two binaries that differ only in the build timestamp and in `__LINE__` values of log calls below
the changed code are the same build; map the offsets with `objdump -h` and `addr2line` first.
- A boot the host cannot capture (a power cut, an OTA-only board): keep the boot report in a flash
preference and repeat it in a periodic alive line, as `scripts/libretiny_prefs/ltprefs.hpp` does.
- Multicast from the host never arrives on some access points: test it device to device. Broadcast
is lossy under Wi-Fi power save, unicast is not. Throttle UDP floods to a fixed packet rate.
- The merge base comes from a freshly fetched `origin/dev`; a stale local `dev` puts dependency bumps into the
PR diff and builds the control from the wrong commit.
- A `.local` name can stop resolving in the middle of a run; keep the address from the boot log and drive by it.
- Never point `framework: source:` at a local `file://` copy of LibreTiny to try a patch: PlatformIO installs it over
the shared `~/.platformio/platforms/libretiny`, so every later build on the machine uses the patched copy. Patch
through the ESPHome side (a linker `-Wl,-wrap,<symbol>` or a define) instead, and if it happened, restore the
directory from a clean copy of the same release.
- `wc -l` pads its count on macOS, so `tail -n +$n0` needs `$((n0+0))` or an arithmetic expansion.
- `api:` reboots a board after 15 min without a client: `reboot_timeout: 0s` on `api:` and `mqtt:` for
soak tests.
`reference/boards.md` has per-chip facts and crash decoding; `reference/recipes/` has the proven
Each flow names the pull requests it was used on, so their result tables show what to post.
One file per component, named after it:
- [`api.md`](api.md): noise handshake and OTA encryption on the ESP8266, outgoing connection wake timing on deep sleep and after a Wi-Fi drop, user defined action listing and execution with a raw plaintext client, store_yaml recovery over Noise or plaintext
- [`ble_midi.md`](ble_midi.md): BLE MIDI client against a fake ESP32 instrument, every packet shape and the send actions
- [`captive_portal.md`](captive_portal.md): fallback AP and /wifisave hammer
- [`cse7766.md`](cse7766.md): Sonoff S31 parser check over the API
- [`http_request.md`](http_request.md): update manifest URL changed at run time, every setter shape, a swap under a running check, non-blocking request actions with loop gap, queue, cancel and reboot checks, OTA request cases with basic auth and MD5 sources
- [`ld2420.md`](ld2420.md): hub setup and sensor states with a simulated module on a UART0 loopback (energy and simple mode)
- [`ld2450.md`](ld2450.md): polygon zones, zone number round trip through the radar's readback, ESP8266 loopback harness without a radar, presence hold and link loss on a frame stream, BT MAC readback
- [`ld2412.md`](ld2412.md): output pin level through the basic configuration command, held back writes
- [`libretiny.md`](libretiny.md): rtc preference emulation, BK72xx and RTL8720C crash report on the next boot with a ROM dump matcher, BK7238 BLE bond store safety with a full flash backup and per partition diff, BK7238 OTA versus the board layout and bootloader
- [`modem.md`](modem.md): PPPoS modem component boot check with no modem attached, loop blocking and disable timing
- [`mqtt.md`](mqtt.md): broker bench, QoS, reconnect, crash log over MQTT, WebSocket transports with a local and a public broker, TLS trust sources (own CA, client certificate, certificate bundle, no verification) with the CLI check, MAC suffix topics and will messages on two boards, shutdown message delivery on restart, disable and deep sleep
- [`nrf52.md`](nrf52.md): dongle board support, zephyr_ble_server connection slot under reordered callbacks, SMP over UDP OTA over Thread with reset faults, MCUboot swap limit and stuck slot recovery, TCP stack probe
- [`online_image.md`](online_image.md): stalled and malformed downloads, PNG, JPEG, BMP and QOI decode time per loop pass with large images and small buffers
- [`openthread.md`](openthread.md): border router on the S3 host plus H2 RCP
- [`ota.md`](ota.md): upload without a configuration and the refusal checks, overlap and lock, timeouts, deflate, upload timing A/B per build and board with deflate and raw, on-device inflate benchmark, OTA automations on two platforms
- [`rtsp_server.md`](rtsp_server.md): camera streaming on the Waveshare ESP32-S3-ETH
- [`scheduler.md`](scheduler.md): loop wake A/B, interval startup delay and control from lambdas
- [`wifi.md`](wifi.md): reboot_timeout with a fallback AP, DHCP lease cache, Easy Connect (DPP), WISP bridge with an ethernet upstream, reboots against a WPA2/WPA3 PMF access point, WISP and ALWAYS_ON AP switching, AP restarts with WPA3 mixed and LIGHT power save
- [`zephyr.md`](zephyr.md): preferences on the nRF52840 dongle
- [`zephyr_coredump.md`](zephyr_coredump.md): crash and decode on nRF52 boards
- [`zwave_proxy.md`](zwave_proxy.md): frame checksums on a live proxy
- [`globals.md`](globals.md): restoring globals with storage rtc and flash across every reset kind
- [`homeassistant.md`](homeassistant.md): light platform mirrored from a local Home Assistant, held reports versus later calls
- [`usb_audio.md`](usb_audio.md): USB audio speaker and microphone, the usb_host FIFO split, and Music Assistant playback through sendspin
- [`dfrobot_sen0395.md`](dfrobot_sen0395.md): SEN0395 mmWave command flow, module quirks and overlong lines through its error reply
- [`speaker.md`](speaker.md): media player pipeline tasks on the Connect AUX-2 with the `speaker` platform player
- [`web_server.md`](web_server.md): state SSE stress and the HTTPD task stack on web_server_idf, entity icon and device class readback
- [`uart_sim.md`](uart_sim.md): UART sensors simulated on a classic ESP32 with loopback and cross-wired UARTs
- [`gpio_expander.md`](gpio_expander.md): interrupt driven expander inputs, loop idle checks, testing a device out of reach
- [`led_strip.md`](led_strip.md): forcing a failed strip allocation, RMT symbols per chip, loop cost per strip
- [`remote_base.md`](remote_base.md): remote_receiver protocol decoding through a one-pin loopback on ESP32 and ESP8266, no IR hardware
- [`xiaomi_ble.md`](xiaomi_ble.md): Xiaomi, ATC and PVVX parsing by injecting scan results into the ESP32 tracker or advertising them from a second ESP32, real AES-CCM frames, duplicate filter model
- [`ble_terminate.md`](ble_terminate.md): proxy disconnect against a peer with peripheral latency
- [`i2c_slave.md`](i2c_slave.md): fake I2C slave on a D1 mini, ESP32 hardware master as the control
- [`fake_i2c.md`](fake_i2c.md): fake I2C devices with no chip: in-firmware buses for many drivers, a Pico 2 W BME280/BMP280 target, a DS2482-800 simulator, NACK injection for vl53l0x
- [`gpio_hold.md`](gpio_hold.md): GPIO hold through the ISR pin API, pad read back with no wiring, per chip pin notes
- [`ble_event_inject.md`](ble_event_inject.md): made up GAP and GATTC events fed into the ESP32 BLE handlers from a button
- [`light_sleep_gpio_wake.md`](light_sleep_gpio_wake.md): GPIO wakeup from auto light sleep with an LEDC wave that keeps running in sleep, and with an external wave and peripheral power-down on the ESP32-P4
- [`manual_ip_and_web_handlers.md`](manual_ip_and_web_handlers.md): station, AP and Ethernet manual IP, web handlers added late and web OTA with web_server, captive_portal or prometheus
- [`ble_server_security.md`](ble_server_security.md): esp32_ble_server pairing, encrypted and MITM characteristics, subscription before and after pairing, bond reuse per ESP-IDF version
- [`prometheus.md`](prometheus.md): date, time and datetime metrics in UTC and another time zone
- [`ble_adv_triggers.md`](ble_adv_triggers.md): BLE service and manufacturer data advertise triggers over the air on the ESP32 and BK72xx trackers, with a second ESP32 advertising known frames
- [`sun.md`](sun.md): on_sunrise and on_sunset by sweeping the observer's longitude, one full sweep in about 17 minutes
- [`sensor_filters.md`](sensor_filters.md): every sensor, binary_sensor and text_sensor filter shape fed a fixed input sequence, outputs compared bit for bit between builds
- [`led_driver_sniff.md`](led_driver_sniff.md): bit-banged LED drivers captured on the ESP32's own pads with SPI slaves through the GPIO matrix, no chips
- [`hub_input_sim.md`](hub_input_sim.md): matrix_keypad, wiegand, rdm6300 and key_collector driven from a D1 mini's own pins and UART loopback
- [`value_range.md`](value_range.md): sensor and number on_value_range with a fixed sequence and a restart
- [`ezo_fake_bus.md`](ezo_fake_bus.md): the ezo component against a fake I2C bus that answers like an EZO circuit
- [`twai_self.md`](twai_self.md): esp32_can on_frame with the controller hearing its own transmit pad, no transceiver
- [`buttons.md`](buttons.md): output button pad timing, homeassistant button through a raw API client with the wizard input, uart button on the UART0 loopback
- [`rs485_frame.md`](rs485_frame.md): DLE/STX/ETX bus on an ESP32-C3 with a cross-wired simulated bus device (frame cases, gate timing)
- [`hx711.md`](hx711.md): fake HX711 driven from the other core, every published value matched to the served conversion
- [`automation_trace.md`](automation_trace.md): automation codegen A/B, a marker trace of chains, conditions, loops and script modes over the API
- [`bluetooth_proxy.md`](bluetooth_proxy.md): held GATT link after a half-open API session, frozen Home Assistant stand-in
- [`crash_report.md`](crash_report.md): crash buttons, fetching the previous boot's crash report over the API, per-chip abort facts
- [`https_idf.md`](https_idf.md): web_server_idf HTTPS test component, listener switch, stalled TLS clients, bad key
- [`mdns.md`](mdns.md): legacy unicast mDNS dump, hammer and latency straight to the device
- [`sim_buses.md`](sim_buses.md): PCF8574/HD44780, MAX7219, SPI spy, packet_interface and SN74HC165 simulators, status LED probe, fake Home Assistant, cg.static_buffer probes
- [`sigma_delta.md`](sigma_delta.md): pulse density of the sigma_delta speaker pin read inside the ESP32-P4, rtttl note onset and release
- [`microphone_http_stream.md`](microphone_http_stream.md): raw PCM over HTTP with a counter fake mic or a USB mic, multi-reader client
- [`i2s_audio.md`](i2s_audio.md): I2S microphone fed by an I2S TX channel on the same S3 through three jumpers
- [`voice_assistant.md`](voice_assistant.md): late TTS start on the Voice PE with a slow announcement URL
## User defined action listing and execution without a client library (#18882)
`scripts/api/rawactions.py HOST list` speaks the plaintext native API over a raw socket and decodes every
`ListEntitiesServicesResponse` by field number (argument fields 1 name, 2 type, 3 description, 4 example, 5 optional,
6 default_value), so new wire fields show up without waiting for the client library. `rawactions.py HOST exec ACTION
k=v ...` sends `ExecuteServiceRequest` with `sint32` ints (field 5, not the legacy int32 field 2), fixed32 floats,
length delimited strings and `int[]` as repeated sint32 (field 7); `ALLOW_MISSING=1` leaves arguments out on purpose
(the device then drops the call). `templates/api-actions-dut.yaml` (D1 mini, three actions, optional variables with
defaults) and `templates/api-actions-strings-dut.yaml` (required strings) log what arrived. String arguments are
`StringRef`s that are not null terminated: copy them with `snprintf("%.*s", size, c_str())` into a buffer before
logging, and do not pass `%.*s` to `logger.log`, whose validator counts it as one pattern.
## store_yaml recovery over the native API (#16445)
`scripts/api/syrecover.py HOST KEY OUTDIR [ORIGDIR]` (KEY `-` for plaintext) builds the client library's
`APINoiseFrameHelper` around a stub `APIConnection` subclass (the compiled helper type checks the connection, and
`ConnectionParams` needs every field), sends Hello, Authentication (never answered without a password) and
`DeviceCapabilitiesRequest`, then `GetYamlRequest` (161), collects `GetYamlResponse` (162) frames by field number
(1 data, 2 done, 3 total_size, 4 encoding) until done, requests again, decompresses with `compression.zstd` (Python 3.14, or the `zstandard` package) and
unpacks the envelope with the tree's `unpack_envelope` (run with `PYTHONPATH` on that tree). Configs in
`templates/store_yaml/`: entry files in `boards/`, a package one directory above them with `!secret` and an inline
OTA password,
so the redaction and the `..` paths are exercised. Repetitive YAML compresses into one chunk even at 22 KB; a 6 KB
include of random base64 lines with `include_secrets: true` gives four chunks (1390 byte packets). Rough edge at
0bd3738430: `include_secrets: true` fails the build when a package in another directory uses `!secret` and has no
`secrets.yaml` beside it; a copy next to the package works around it.
## BLE MIDI client against a fake instrument (#18508)
No MIDI hardware needed: `templates/ble_midi/midiperi.yaml` turns a second ESP32 (an S3 with ethernet, so the
API stays reachable while BLE is busy) into a BLE MIDI peripheral with `esp32_ble_server`: the MIDI service
advertised, the MIDI characteristic with read, write, write without response and notify, an `on_write` that logs
the bytes, buttons that notify hand built packets (channel messages, two messages in one packet, running status,
SysEx in one packet, split over two, oversize, cut short, clock, aftertouch, orphan data bytes, a 50 packet burst)
and a restart button for the disconnect test. `midicli.yaml` is the component test config on a classic devkit with
`wifi:`, `api:`, the peripheral's BLE address in the `peri_mac` substitution, given on the command line with
`-s peri_mac <mac>` (the base MAC plus 2, printed by `esp32_ble` in `dump_config`), `on_sysex` and `on_message` lambdas that print the bytes, debug sensors and out of range send
buttons. Drive both over the plaintext API with `scripts/modem/mdrive.py HOST LOGFILE press:NAME wait:S` 3 s
apart and compare the peripheral's `TX` and `RX` lines with the client's decoded lines. Gotcha:
`ble_server.characteristic.set_value` latches the value of its first play and re-applies it before every notify,
so the peripheral script sets the value from a lambda (`id(midi_io).set_value(std::vector<uint8_t>(pkt));
id(midi_io).notify();`). At VERBOSE the client logs four lines per message, so a burst shows an
`esp32_ble took a long time` of a couple of seconds that is the serial port, not the parser.
When there is no chip and no wiring, subclass `i2c::I2CBus` in an include, implement `write_readv()` only, answer
the registers the driver reads and check each write. Point the component at it from an `on_boot` lambda that runs
before the component's `setup()`: `id(x)->set_i2c_bus(&fake::bus)` at priority 950, which is ahead of every I2C
component (hubs set up at IO, 900, sensors at DATA, 600). The configs here that use 700 to 1000 work for their
components; use 950 for a new one. Keep a real `i2c:` block in the config so validation passes. Bulk
loops in a button lambda must call `App.feed_wdt()` and keep their per-iteration logs below INFO: 2000 DEBUG lines at
115200 baud trip the task watchdog, which is a fault of the harness and not of the PR.
`templates/i2c_fake/` holds the fakes; each `*-common.yaml` or `*-test.yaml` is a classic ESP32 or ESP32-S3 config
that includes its `.hpp` (add `esphome: name:` for the `-common` ones):
-`ccsfake.hpp`, `ccs-common.yaml`: CCS811 at 0x5A (#21927). Answers setup and update reads and checks every
ENV_DATA write against the humidity and temperature the case set; "CCS Cases" runs edge values, "CCS Sweep" 2000
random ones and logs `SWEEP checked N bad M bad_reads R bad_writes W`. A read without a register, of a register
the fake does not model or of the wrong length gets a NACK and counts in `bad_reads`; a write other than the
driver's SW_RESET, APP_START, MEAS_MODE and BASELINE writes (each with its own length) gets a NACK and counts in
`bad_writes`. The CASES and SWEEP lines end in PASS only when something was checked and `bad`, `bad_reads` and
`bad_writes` stayed 0, and in FAIL otherwise.
-`eefake.hpp`, `ee-common.yaml`: HTE501 (0x40) and TEE501 (0x48) with CRC words; "Next" steps through raw
temperature words and logs the value a signed hundredths decode should give (#21930). A command other than
identification and measure, and a read longer than the reply, gets a NACK and counts as `bad`. The DONE line
200 ms later, after both reads, shows the running `bad` total, the measure replies each driver read and the
published values; it ends in PASS only while `bad` is 0, both drivers read a reply this case and both published
the expected temperature (and HTE501 the fixed humidity).
-`scdfake.hpp`, `scd-common.yaml`: SCD4x at 0x62 and 0x63 with Sensirion CRC; logs every `0xE000` ambient pressure
write, and the CASE and SHOW lines count writes with a bad CRC apart from the good ones. A command the scd4x driver
does not send, one sent with the wrong length (2 bytes bare, 5 bytes with one CRC word), or any other command with a
bad CRC, gets a NACK and counts as `bad writes`. A read after a command the fake does not answer, or one longer than the command's
reply, gets a NACK and counts as `unknown reads`. The second instance has a fixed `ambient_pressure_compensation`
to cover the setup path (#21935). Each CASE line ends in PASS only when bad CRC, bad writes and unknown reads did
not move, the `0xE000` writes match `expect_e000()` (none, or one with the expected value) and, for a source case,
CO2 was published. `RANGE_CHECKED` in `scdfake.hpp` picks the rule: false truncates as dev does, true skips values
outside 700 to 1200 hPa and rounds as #21935 did. The SHOW line ends in PASS when nothing bad was counted, neither
instance failed and 0x63 got exactly one write of its fixed value.
-`sxfake.hpp`, `sx-common.yaml`: two SX1509 register files with reset values and the `7D 12/34` reset, recording
every write. Only a byte that lands on RegReset (0x7D) drives the reset, so a burst through it keeps the bytes for
0x7E and 0x7F. A read without a register, or a transfer past 0x7F, gets a NACK and `dump_log()` warns with the
count. Compare per address: hub setup order can legitimately move between hubs (#21897). Outputs of an Apply
platform are distinct template types, so a lambda list of them needs
`std::initializer_list<output::FloatOutput *>{...}`. Each case ends with an END line, PASS only while no write was
dropped and no transfer was bad; the writes themselves are judged by the diff.
-`msfake.hpp`: eight MS8607 pressure/temperature fakes at 0x76 and up and humidity fakes at 0x40 and up, with faults
per device (reset NACKs, bad PROM CRC4, zero D1 every Nth read, humidity CRC8 errors, out of range D2, full scale
humidity). It counts every command, so skipped conversions show in `report()`, and an ADC read with no conversion
pending answers 0 like the real chip and counts as `adc_no_conv`, and a humidity read with no measure pending gets
a NACK like the real chip in no-hold mode and counts as `read_no_measure` (a reset aborts either); each injected
fault counts when it is delivered (`inj_reset_nack` with `reset_nack_left`, `inj_zero_d1`, `inj_bad_crc`), so a
run that never reached a fault path shows 0 there; a PROM read, conversion or humidity measure
before that device's first reset counts as `no_reset`; a humidity read of any shape but a bare
3 byte read, and a pressure/temperature read that is not a 2 byte read of a PROM word (even addresses 0xA0 to 0xAC) or
a 3 byte ADC read, gets a NACK and counts as `bad_reads`; a bare read answers the last PROM or ADC pointer and never
repeats a reset or conversion; a command write the fake does not know, or one longer than a
byte, gets a NACK and counts as `bad_cmds`. Point both the component and its `humidity_i2c_id` device at the bus. That
driver logs no values, so give each sensor an INFO `on_value` log (#16487).
-`htfake.hpp`, `ht16k33-char-test.yaml`: HT16K33 chips at 0x70 to 0x77 (0x74 and 0x76 NACK) keeping the 16 byte
display RAM and a command trace, rendered back as text for the Adafruit 0.56" 7-segment backpack ("12:34",
decimal points, "!" for stray bytes); a read gets a NACK and counts as malformed. The test runs 64 checks (setup order, font, colon and dots, chained chips,
printf/strftime/clock, dimming, blink, standby, NACK) and ends with `ALL DONE pass=N fail=M` and PASS or FAIL (#8520). It found a
crash (null glyph table) when one display had `add_characters` and another did not.
-`ssfake.hpp`, `seesaw-test.yaml`: Adafruit seesaw at 0x36 (ATtiny817), 0x49 (SAMD09) and 0x4A (NACK): status,
GPIO bulk with the pull latch, ADC, touch, encoder and NeoPixel with chunk offsets, and the write to read gap per
module. A read it does not model, a write without a register and a write to a module or register it does not model
(or with the wrong data length) get a NACK and count in `bad`. A 100 ms interval step machine drives it and ends with `ALL DONE` and PASS or FAIL (#13538).
-`bldcfake.hpp` (three M5Unit-BLDC units at 0x65 to 0x67 with simple motor physics, one that never spins down),
`gtfake.hpp` (GT911 at 0x5D: switches, max values, touch points and key byte) and `sdpfake.hpp` (SDP3x/SDP8xx at
0x21 to 0x26: product IDs, an unknown product, a failing read every third time, a failing ID read, stop refused
while idle) have no config yet; wire them the same way. Each fake sets up its own faults in its constructor, so
there is no init step to call. Each NACKs a read it does not model or one of the wrong
length, and a write or command it does not know, and counts them as `bad_reads` and `bad_writes` (`bad_cmds` for
the SDP) in `report()`; check those are 0. The SDP report also counts each injected ID read NACK at 0x24 as
`id_fail`, so 0 there means the driver never reached that fault.
Check the PR head again just before posting results: a head that moves during a run means a retest.
## Pico 2 W as an I2C target (#22236)
`scripts/pico_i2c_target/`: `picofake.hpp` is a raw RP2350 I2C0 interrupt handler on GP4 (SDA) and GP5 (SCL) at 0x76 that
answers one byte per read request from a 256 byte register file, so any read length is byte exact. `picofake.yaml`
plays a BME280 (`chip_id` 0x60) or BMP280 (0x58) with the datasheet calibration; every 15 s it switches between the
datasheet example readings and the 0x80000 "no measurement" values, and logs `STAT` counters every 5 s. `bad`
counts reads and writes of registers the drivers do not use (soft reset only with 0xB6, read pointer wrapping past
0xFF). The DUT is
an ESP32-C3 (`dut-common.yaml`, GPIO6/GPIO7) with two sensors on the one fake (`dut-bme280.yaml`,
`dut-bmp280.yaml`), the second with oversampling NONE. `run.sh bme280|bmp280 LABEL` flashes the DUT, starts the
fake capture and waits for a `STAT` line as the baseline, then resets and captures the DUT and prints the distinct
values per sensor. The fake capture runs on until a `STAT` line arrives after the DUT capture ends, so that line
covers the whole run. It prints PASS when every sensor gave a numeric state, that `STAT` line came, and the fake's
`abrt` (read replies the controller aborted) and `bad` counts did not move from the baseline; diff the output of dev
and the PR.
The ESP32-S3 I2C slave v2 driver cannot emulate a register device: answering from a task and straight from the ISR
both gave shifted multi-byte reads. Use the Pico.
## DS2482-800 simulator (ds248x)
`scripts/ds248x_sim/`: `dssim.yaml` with `sim.hpp` turns a Pico 2 W into a DS2482-800 at 0x18 (GP4 SDA, GP5 SCL)
with four DS18B20s: channel 0 at 21.5 °C, channel 1 at 33.0 °C, channel 2 at -5.25 and 40.0 °C. The DUT is an
ESP32-C3 (`dspd.yaml`, GPIO4/GPIO5 plus GND); the internal pull-ups are enough at 50 kHz. Faults are one character
on the Pico's USB console: `c` NACKs the read pointer write after the next channel select, `z` NACKs the 9 reads
after the next 0xBE, `b` and `k` stick busy with a failed reset or config write, `x` and `X` make it dead and alive.
`SC=<serial_capture.py> python3 drive.py TAG SIM_PORT DUT_PORT [c,cd,z,b,k,dead]` (default `c,z,b,k,dead`; `cd`
runs the `c` fault densely, 28 times about 3 s apart) resets the DUT per phase and prints
readings, wrong values, log counts and the simulator's counters (`wrong_ch` counts ROM matches on the wrong
channel). The DUT is captured with `--ms` so the log can be split at the last fault: the recovery window starts
`SETTLE` (3) seconds after it. A phase fails when any of the four sensors got no correct reading in the recovery
window, the DUT logged a CRC, held low, busy timeout or no response line in it, a temperature is wrong, a phase other
than `dead` logged `recovery failed`, `DS248x hub failed` or a component marked failed, a ds248x warning is still
set at the end of the phase, the DUT capture
failed (stderr in `TAG-<phase>.err`), a fault moved its injection counter (counted when the DUT meets the
fault, not when it is armed) by other than the number of faults sent (`arm_lost` counts arms that landed while one
of the same fault was still pending and merged into it), a NACK fault never moved `nacked`, a write delivered data while a NACK was armed
(`nack_miss`, also counted when the write is followed by a repeated start into a read; the NACK never reached the wire and stays pending), the DUT sent an address-only write (`empty_wr`), a write was cut off by a repeated start into another write and
never processed (`dropped`),
a fault is still pending (`pend=1`) at the end of the
phase, the simulator counted a config write with a wrong complement (`cfg_bad`) or an unknown, short or overlong
command (`bad_cmd`), a counter is missing from the simulator's CNT line, the simulator aborted a read reply
(`tx_abrt`, not gated in the `dead` phase, which switches the controller off), a phase other than `b` and `k` (which
reset the chip under the driver) counted `wrong_ch` or a 1-Wire command without the active pull-up bit (`ops_no_apu`),
the simulator never logged `DEAD` for the `dead`
phase, or the DUT log of the `dead` phase has no sign of the outage (`did not respond`, a ds248x warning,
`DS248x hub failed` or a component marked failed). It ends with PASS or `FAIL:` and the failed phases. The whole
phase CRC, held low, busy timeout and no response counts are printed for information: the faults cause them.
- Sensors polled at the same interval go through the channels in blocks, so a stale channel cache never shows.
Poll them at different intervals, as `dspd.yaml` does (1, 1.3, 1.7 and 2.3 s).
- The NACKs use the RP2350 `slv_data_nack_only` register, set at STOP.
- A NACKed write and an address-only write both end with no data, so the simulator cannot tell them apart.
The ds248x driver never sends one, `dspd.yaml` sets `scan: false` and the simulator only sees STOPs after its
own address, so `empty_wr` (address-only writes outside a NACK window) stays 0; a nonzero count means `nacked`
may include writes that never carried a NACKed byte.
## I2C fault injection in software (vl53l0x)
`scripts/i2c_fault/`: `faultbus.h` wraps the real `I2CBus` and returns NACK for chosen transfers to one address. An
`on_boot` priority 800 lambda swaps the sensor's bus (`id(d)->set_i2c_bus(&fb)`) before its setup, and an interval
arms the faults. `vlpf.yaml` runs a VL53L0X on an ESP32-S3 (GPIO8/GPIO9) through quiet, fault A (the next
measurement-in-progress read fails), quiet, fault B (the next measurement start fails) and quiet phases over 150 s,
with a model ID probe every 10 s to tell a driver stall from a dead chip. `run.sh LABEL` builds, flashes, captures
and runs `an.py`, which counts readings per `PHASE` line. Each `PHASE` line clears a fault that was armed and has
not fired yet and logs the fired totals so far, so `an.py` counts the fires of each phase. It fails when the phases are not quiet, faultA, quiet,
faultB, quiet (a short capture or a restart), the first quiet phase has no readings, a later quiet phase reads less
than half as often as the first, a later quiet phase reads NaN (`nan` or `-nan`) or logs the driver's "update called before prior
reading" warning when the first did not (a target out of range reads NaN in every phase), a quiet phase has a failed
model ID probe, one that did not return 0xEE or no successful probe at all,
anything crashed or rebooted, or a fault did not fire in its own phase or fired in another one.
Prefer this to glitching the wire. Holding SDA low by rerouting the pin (`esp_rom_gpio_connect_out_signal` to
`SIG_GPIO_OUT_IDX` and back to `I2CEXT0_SDA_OUT_IDX`) wedges the ESP-IDF controller, which then needs
`i2c_master_bus_reset()` after each glitch, and glitches timed onto writes left the VL53L0X answering 0x00 until a
## Restoring globals with storage: rtc and flash (#13954)
`templates/globals-storage-dut.yaml` is the shared package: five restoring globals (rtc int, flash int, default int,
rtc `std::string`, rtc `uint8_t[2]`), bumped and logged with the reset reason on every boot, saved and synced after
2 s, odd boots `App.safe_reboot()`. Platform files add the board (`packages: dut: !include globals-storage-dut.yaml`), the `debug` reset-reason text sensor and, on the
ESP32, `deep_sleep: run_duration: 12s, sleep_duration: 8s` (even boots sleep). No network needed: flash by USB,
`serial_capture.py PORT 110 OUT --reset` gives the reset pulse plus a run of soft restarts and sleeps; the ESP32's
EN pulse is a `POWERON_RESET` that empties RTC memory, the D1 mini's RST pulse (`rst cause:2`) keeps it, so the
ESP8266 needs a real power cut by hand; arm long `--released` captures on both ports first (the capture
reopens the port when it comes back, the very first boot lines are still lost, read the next boot's restored
values). Unchanged-build check: the config with every `storage:` removed, built on the PR and on the dev merge
commit under the PR's own commits (not the merge base, which is older dev), `cmp -l` plus an `nm -S` symbol diff;
only the template names differ. Results: https://github.com/esphome/esphome/pull/13954#issuecomment-6254794712
A report that arrives during a transition is held and applied when the transition ends, unless a call came after
it. `scripts/homeassistant_light/heldcase.py DEV_IP KEY BULB_IP same|many [BULB_KEY]` (a key of `-` is plaintext) keeps one API connection to the mirror
device and one to the bulb so the steps land inside a 3 s fade: press "A 50 fade 3s", 0.7 s later set the bulb to
90% directly (the report is held), 1.4 s in press the same fade again (or "A many calls", 16 calls in one loop
pass). Pass: the mirror and the bulb end at the second call's target. Only `same` reproduces the bug: a build
without the fix logs `Applying the report that arrived during the transition` and the bulb stays at 90%. `many`
ends at 100% on both builds because the last call's target differs from the held report, so it is a regression
check only, not a reproduction. Separate button presses are too slow (about 1.5 s each) to land inside the fade.
The mirrors are not API entities, so read their state from the "Log state" button's line. Results: https://github.com/esphome/esphome/pull/21144#issuecomment-6278257695
## Update manifest URL changed at run time (#21373)
`scripts/http_request/manifest_server.py PORT [DELAY]` serves `stable.json`, `beta.json` and `slow.json` (answered
after DELAY s) with both chip families and logs every request. `templates/http_request_update/upd-common.yaml`
(secret `manifest_base` is the server URL) has the update entity on the stable manifest, buttons for every
`set_source_url` shape (`std::string` local, literal, temporary, `.c_str()` of a local, the current URL back in),
`Check Then Swap` (ESP32 task path: check the slow manifest and replace the URL in the same lambda), `Arm Swap In
Check` plus a `logger: on_message` lambda that replaces the URL from inside the inline check (ESP8266; the log line
from inside `on_message` is swallowed, judge by the next check's result), and a `Heap` button. `upd-base-common.yaml`
keeps only the shapes that compile on a tree with the bare pointer setter (for the base build, copy a board file
and point its `common` include at `../upd-base-common.yaml` with a different `name:`); `boards/upd8266.yaml` and `boards/upd32.yaml` are the
per-board files, one directory below the package their `../` include points at. Drive with `scripts/http_request/updrive.py HOST KEY LOGFILE press:NAME wait:S`, which prints every
update state (latest version, summary). The ESP8266 Arduino client gets `Error reading manifest: -1` on roughly a
quarter of the fetches from this Python server on every build, so count checks against the base before blaming a
PR. `http_request` on the ESP8266 needs `verify_ssl: false` even for plain http URLs.
#22185 adds an HTTPS developer API to `web_server_idf` with no YAML option, so `scripts/https_idf/` brings a test
component: `ext/httpstest` (`httpstest.hpp`) serves `/hw` (a status line with mode, request count and free heap),
`/hw/big` (16 KB) and `/hw/switch/{http,https,https8443,bad}`. A switch runs on the main loop through `defer()`, or
3 s later with a `/later` suffix (a stalled TLS client holds the server task, so its switch is armed first), and
calls `set_ssl_credentials(cert, key, port)`: plain HTTP on 80, HTTPS on 443 or 8443, or a key that does not match
the certificate. Its schema uses `web_server_idf.consume_ssl_sockets`. The run-time switch is the follow-up API; on a
tree that only has #22185, drop the switch paths and set the credentials once before `init()`.
Copy the directory into a scratch directory with a `secrets.yaml`, then `run.sh LABEL` (env `TREE`, `PORT`, `SC`,
`PY`): it runs `make_certs.sh` (a self-signed P-256 certificate, its key and an unrelated key in
`ext/httpstest/certs.h`; test material, never commit it), builds `https-dut.yaml` (ESP32-C3, logs on USB), flashes,
takes the address from the serial log and runs:
-`t1.sh HOST`: one GET with the TLS time, 20 sequential, 6 parallel three times and the 16 KB body. The fixture has
no event source, so the 501 that #22185 sends for an event stream on HTTPS is not covered here.
-`stall.py HOST`: arms a switch to HTTP and to 8443, opens two idle TCP clients (no ClientHello) on 443 before it
runs, then switches back; every switch must be accepted and work, and the idle clients must be closed.
-`t2.sh HOST`: HTTP and back, an armed switch to 8443 with a client open that sends a handshake record header and
never the rest, 60 requests that must all get a 200 for a heap check (compare the `/hw` heap before and after),
then the bad key. The server must log the error and stay off, without falling back to plain HTTP; reset the board
afterwards.
The runner fails on any unexpected status, a missing key error for the bad key, a reset or a task watchdog. `https-soak.yaml` adds a switch between 443
and 8443 every 20 s for a long run with clients attached. Each HTTPS connection takes about 26 KB of internal RAM
(no PSRAM on the C3), so watch the `ALIVE heap=... min=...` lines.
Each reader prints its HTTP status, format headers, sample count and rate, the longest gap between reads and, with
`--pattern`, every break in the counter. `--stall K` makes reader K stop reading after 2 s while keeping its socket
open, then checks that the device closed it. `--expect-reject R` marks the last R readers as past `max_clients`;
they must get a non-200 status or a close or reset before any samples, and a timeout or a served reader fails.
The script exits 1 when the listing is not a 200 naming the stream, the stalled reader was not dropped, a reader past
the limit was not refused, or any other reader got no 200, no samples, an early EOF, a read error or a broken counter.
Cases worth running on each head: one reader with `--pattern` (rate must match the sample rate, 0 breaks), the
`max_clients` limit plus one with `--expect-reject 1` (note what the extra one gets; the others must be unaffected), a stalled reader among
three (the device should drop the stalled one only), the "Mic stall" button during a read (a gap of about 6 s, then
the counter continues), mic start and stop while readers are attached, and the heap line after many connect and
disconnect cycles. On the USB microphone, check the rate and `rms` instead of the pattern with `--rate 16000 --min-rms 1`, and add `--max-gap-ms` to fail on long gaps; these set the exit code.
A local mosquitto with `allow_anonymous true`, `log_type all` and `log_dest file`: the broker log gives
each subscription's QoS, the device address (`New client connected from <ip> as <name>-<mac>`) and exact
subscribe counts. Kill and restart mosquitto to exercise `on_disconnect` (reason 0) and resubscription.
Keep `reboot_timeout: 0s` on `api:` and `mqtt:`. The ESP-IDF outbox swings free heap about 2 kB right
after bulk subscribe and unsubscribe; read the heap seconds later over several rounds before calling a
leak. Spawned subscriptions need about 2.5 s to reach the broker before the next publish.
`esphome logs --device MQTT` only uses the MQTT transport when the config has no `api:`. Driver pattern: a
paho script publishes and polls the serial capture file.
## WebSocket transports on ESP-IDF (#16056)
One build per transport from `templates/mqtt-transport-dut.yaml`, with `scripts/mqtt/mosquitto-ws.conf` as the
local broker (mosquitto 2.x from brew has `websockets` support; `log_type all` prints `New connection from <ip>:<port>
on port <listener>`, which proves which listener each client used) and `scripts/mqtt/make_test_ca.sh BROKER_IP` for
the TLS listener's certificate, both run in the scratch directory so keys, logs and outputs stay out of the checkout. `scripts/mqtt/cases.sh NAME KEY BROKER_HOST SUB_PORT CAPTURE [DEV_HOST] [restart]` subscribes
to `NAME/#`, sends a switch command and an `on_message` topic, presses the 3000 byte publish button over the API,
restarts the local broker when the last argument is `restart` (the PID in `mosquitto.pid`, written when the broker
was started with `mosquitto -c mosquitto-ws.conf & echo $! > mosquitto.pid` in the working directory next to the
certificates), checks every listener in the broker config answers again, and prints the broker's message counts with the device log. `mosquitto_sub` cannot speak
WebSocket, so it subscribes on the plain listener of the same broker. Cases: `tcp` on 1883, `ws` on 9001, `wss` on
9002 with `certificate_authority` (the certificate names an address, so `skip_cert_cn_check: true`), `wss` to
test.mosquitto.org on 8081 with no CA for the certificate bundle, and two negatives that must never connect: a
different CA (`mbedtls_ssl_handshake returned -0x2700`) and the self-signed server through the bundle
(`esp-x509-crt-bundle: Failed to verify certificate`, `-0x3000`). `CONFIG_MQTT_TRANSPORT_WEBSOCKET` is on by
default in the generated sdkconfig. The TLS session costs about 29 kB of internal heap while connected; the public
broker carries other devices' `esphome/discover` traffic, which is noise. Results:
`scripts/mqtt/mac_suffix/run.sh X BUILD` compiles (from `TREE`) and flashes variant X (a defaults, b custom `topic_prefix`, c explicit topics,
d no suffix) of BUILD (`pr` or `base`) on an ESP32 and a D1 mini, records the broker with
`mosquitto_sub -F '@H:@M:@S r=%r %t %p'`, presses Restart through the `cmd_t` found in the discovery payloads, then
holds both chips in the bootloader for 40 s (`esptool --after no-reset chip-id`) so the broker sends the will; a
plain reset only gives "session taken over" and no will. The two boards' configs live in separate folders because
variants a to c share a `name` (the MAC suffix tells them apart), and the same name in one folder shares a build
directory. A D1 mini CH340 can garble the serial capture after an RTS reset, so read its topics from the broker.
`esphome logs --device MQTT` on a current tree needs paho-mqtt 2.x; an older venv with 1.6.1 fails, so use a venv
made from the tree's requirements.txt. A Restart press never delivered the shutdown message on any build tested.
## Shutdown message delivery (#21566)
`scripts/mqtt/shutdown/`: `sd-common.yaml` sets birth/will/shutdown payloads `hello`/`dead`/`bye` so the shutdown
message and the will cannot be confused, and takes `restart`, `disable` and `sleep` on `<name>/cmd`.
`sdrun.sh V N` (`TREE`, `P32` and `P66` in the environment) compiles and flashes the ESP32 and D1 mini, restarts each N times, disables MQTT once and deep sleeps the ESP32 three
times (never the D1 mini: without the GPIO16 to RST wire it never wakes); `match.py OUT` counts a `bye` within 5 s of
each command. Dev: ESP32 0/10 restarts, D1 mini 1 to 3/10. Put the broker address (`broker_ip`) in the secrets.yaml next to
the included package, and quote `"${name}/status"` inside flow mappings.
## MQTT discovery against a throwaway Home Assistant (#21956)
Run a Home Assistant dev install with its own config directory (`hass -c <scratch>/ha`) holding `http:
server_port: 8129`, `api:`, `config:`, `onboarding:` and MQTT debug logging, plus a local mosquitto on 1883. In
`scripts/mqtt/ha_discovery/`, `onboard.py DIR` creates a generated test user (the credentials stay in DIR) and a
token, `flow.py TOKENFILE` adds the MQTT entry (the `protocol` and `other_settings: {set_client_cert: false,
set_ca_cert: "off", transport: tcp}` fields are required), and `hadrive.py TOKENFILE NAME` reads the device's time,
date and datetime entities, calls `set_value` on each and reads again; it exits 1 on a missing entity, a failed
call or a state that does not read back the value that was set. The three share the REST calls in `ha_rest.py`. `dt-common.yaml` is the device side (classic ESP32, three
template datetime entities, `${broker}`).
- The HA dev branch keeps a YAML port change as a pending HTTP config and reverts it after 5 minutes. Before
starting HA again, set `data.stable.server_port` to 8129 and `data.pending` to null in `.storage/http`.
- ESPHome MQTT unique ids are name based: clear the previous build's retained discovery (`mosquitto_pub -r -n` on
each config topic) and expect the entity ids of the first build.
## mqtt.disable and broker drops on ESP-IDF (#22230)
`scripts/mqtt/disable/disable-common.yaml` disables MQTT for 30 s every 60 s (every third cycle starts with fast
toggles, a disable while connecting), publishes a tick every 2 s and logs `EV CONNECT`, `EV DISCONNECT` and
`scripts/mqtt/retained_burst/`: `burst-topics.yaml` (a package) subscribes 25 `on_message` topics plus 15 switch
command topics, `burst-common.yaml` logs the counts 15 s after the connect, and `seed.sh BROKER_PORT` leaves all 40 retained on a local mosquitto (`seed.sh BROKER_PORT clear` removes them).
`run.sh LABEL BOOTS` (env `TREE`, `PORT`, `SC`, `PY`, `BROKER`, `BROKER_PORT`, default board an ESP32-C3) resets the
board BOOTS times and prints `RESULT got=x/25 switches_on=y/15` and any `Dropped` lines per boot; it passes only with
40 of 40 on every boot. Before #22200 the C3 dropped exactly 49 events (80 events minus 31 slots): the main loop
drains nothing during the burst.
## Forced reconnect mid-burst (#22248)
`scripts/mqtt/reconnect/`: `kick-common.yaml` includes the same `burst-topics.yaml` and counts per connection with
a `mode: restart` script on `on_connect`, logging `RESULT` 6 s after each connect. It and `run.sh` use files from
`../retained_burst/`, so copy the whole `mqtt/` directory to the scratch directory. `kick.py BROKER_LOG BROKER_PORT
PREFIX CYCLES` tails the broker log and takes over the device's client id 0 to 400 ms after every other connect
(mosquitto drops it with "session taken over"), then leaves the next connection alone for 8 s and ends it. It
starts by kicking the connection that is already up and skips its own connections (from 127.0.0.1 with the same id).
`run.sh LABEL CYCLES` (env as above plus `BROKER_LOG`) seeds, flashes, waits for the first connect and runs the
kicker. The device counts its first connection as CONN 1, so the kicked ones are the even CONN numbers and the
left-alone ones the odd numbers from 3 up; kick.py prints each left-alone connection with its time and CONN number,
and `judge.py` pairs those with the RESULT lines by time and passes when every one has `got=25/25`. A reset after
the first connect fails the run. Retained switch commands fire `on_turn_on` only on the first connection, so
case list over aioesphomeapi and judges each from the serial capture. ESP-IDF facts: a stalled server
ends with `Error downloading image: -28679` (EAGAIN) after the transport timeout, an early close with
`-1`, both before and after the PR; a body without Content-Length and without chunking reads as complete
(Size: 0) and finishes. A build without the fix hangs on the truncated, oversized header, empty, chunked PNG
and cut chunked cases and every later update logs `Image already being updated.`, so a baseline run needs
a reboot per group, the driver's 45 s cap per case, and an outer `timeout` that covers the whole list.
A case expected as `any` passes on `finished` or `error`, never on no outcome.
## PNG decode time limit (#21364, #21391)
The same server also serves any plain file in its www directory by name (404 for a missing file or a path
outside it). `scripts/online_image/make_big_pngs.py <www>` (after `make_images.py`) writes a 320x240 gradient (618 B, also as `ok.png` so the misbehaviour cases carry a
real multi-row PNG), a flat 640x480 PNG (1.6 KB for 307200 pixels), 231 KB of noise and a JPEG. The device is
`templates/online-image-dut.yaml` with `type: BINARY` on the main image (a 640x480 RGB565 does not fit a classic
ESP32 without PSRAM) and `debug` loop time and heap sensors at 2 s; the 256 byte buffer image stays RGB565, so do
not point big images at it (out of memory on every build). `scripts/online_image/pngdrive.py HOST KEY LOGFILE
BASEURL LABEL` (default case set `png`; `CASES=short` for the six case list) calls the actions, waits for the
finished or error line and reports the case time, the highest loop time and the lowest free heap. The per pass `online_image.image took a long time` values come from
the serial capture and only print on a new maximum, so read the first fetch of each image. The pass that decodes a
whole buffer is the decoder cost (before the change 510 ms gradient and 1980 ms flat, after it 77 and 140 ms); for
the noisy PNG the longest pass is the blocking 64 KB read on both builds (0.7 to 3.7 s). Alternate PR, base, PR
for a timing claim.
For a replacement PR, compare against the parent PR head, not dev. `CASES=noise` fetches the noisy PNG three times
(run it right after a boot, alternating builds); `CASES=end` runs flat and gradient as plain, `noclen_`, `chunked_`,
`half_` and `stall_` through the prefixed server cases below.
## JPEG decoded as it downloads (#21342)
`oi_server.py` serves prefixed cases on any file: `half_<file>` (first half, matching Content-Length),
`noclen_<file>` (close delimited), `chunked_<file>` (1 KB chunks), `stall_<file>` (a third, then the socket held
# Simulated chips on the bus, fake Home Assistant, static buffers
`scripts/sim_buses/` and `scripts/static_buffer/`. Each header is added with `esphome: includes:` and wired in an
`on_boot` lambda, so no chip is needed. The wrappers take the real bus or SPI delegate and pass through every
address they do not simulate.
-`lcdsim.hpp`: `LcdSimBus`, an `i2c::I2CBus` wrapper that plays PCF8574 backpacks (P0 RS, P2 EN, P4 to P7 D4 to D7)
driving HD44780 controllers. It decodes the EN-strobed nibbles and rebuilds the rows written after each 0x80 and
0xC0 address command. `add()` up to three addresses, hand it to each `lcd_pcf8574` with `set_i2c_bus()` at
`on_boot` priority 800, and call `report()` from an interval. `lcd_an.py SERIAL` checks the rows against the
patterns its docstring lists (0x27 20x4, 0x26 16x2, 0x25 64x4), and fails on a display whose frame count does not
grow between its first and last report, on a segment longer than its rows or on
data past the 128 byte segment buffer, which the report counts as `overflow`, on a report from an address it does
not expect, or on any read (`reads`; the driver never reads, so the simulator NACKs them). A 64x4 display hangs the
code before #22331 in
setup.
- HT16K33: use `templates/i2c_fake/htfake.hpp` with its test runner (see [fake_i2c.md](fake_i2c.md)).
-`mxsim.hpp`: `MxSniff`, an `spi::SPIDelegate` in front of the real one that keeps what each MAX7219 in a chain
latches. Swap it in at `on_boot` priority 399 through `DelegatePeek::of(id(x))`:
`auto &d = DelegatePeek::of(id(x)); d = new MxSniff(d, chips);`. `mx_an.py SERIAL CHAIN...`
summarizes the reports, CHAIN being each name passed to `report()`, and fails on a named chain with no report line,
on malformed transactions (byte count not two per chip, or a partial byte from `write()` or
`write_cmd_addr_data()`, which are recorded and forwarded too), on a named chain with fewer than two report lines,
or a chain whose transaction count stays 0 or stops growing, and prints PASS otherwise.
-`spysim.hpp`: the same delegate trick (both headers carry `DelegatePeek`, so one config can include both), logging length and FNV-1a hash of every transaction over 1000 bytes (whole
frames, used for `pixoo`). `write()` and `write_cmd_addr_data()` are hashed and forwarded like `write_array()`;
a partial byte or a bus width other than 1 is hashed as whole bytes, so the SPY line carries `odd`, the count of
such transactions, and a dev vs PR hash match only holds when `odd` is 0 on both.
-`pisim.hpp`: drives two `packet_interface` stacks on cross-wired UARTs through `top()`, with payloads whose bytes
are a function of their index so every byte is checked, and the index and length each side sent (last 256) so a
truncated or split packet, or one whose slot a later index reused, counts as `rx_bad`. Send indices 0, 1, 2 and so
on from each side: a skipped index counts as `rx_lost` and an old or repeated one as `rx_bad`. `report(tag, true)` once both links are idle ends each
line in PASS only when that side received every packet the peer sent intact and the peer had no send refused;
`missing` is the rest, packets still in flight or lost at the tail. `packet_transport` drops packets carrying its own device
name, so a one-board test must use the layers directly.
-`hcsim.hpp`: a software SN74HC165 chain on ESP-IDF (data GPIO6, clock GPIO7, load GPIO10). A falling load edge
latches the pattern and each rising clock edge drives the next bit onto the data pad, which is in
`GPIO_MODE_INPUT_OUTPUT` so the component reads it from the same pin. Call `hcsim::report(label, loads0, clocks0)`
after each window with `hcsim::loads` and `hcsim::clocks` from its start; it logs `INVALID` instead of a result
when the GPIO or interrupt setup failed or the window saw no load or no clock edge, so a dead simulator cannot pass
as a wrong but steady pattern.
-`ledprobe.hpp`: samples a status LED (or any output) pin every 2 ms from an `esp_timer`, off the loop task, and
prints blink count, on time, period and the longest gap per window. It measures `status_led` while the loop is
blocked (#16407). Pass `ledprobe::n` from the window start as the mark; a window with more than 400 edges logs
`INVALID` and no result line instead of passing for a stalled LED. So does every window when the pin or timer
setup failed or the timer has not sampled for 100 ms, so a dead probe cannot read as a quiet LED.
-`fakeha.py HOST OUT_PREFIX`: an aioesphomeapi client standing in for Home Assistant for `homeassistant` select
mirrors. It answers state subscriptions from a table, pushes option and state changes (full, overfull, quoted,
unparseable), checks each step through a "Dump" button, captures the action calls, churns 600 option updates
while watching the heap, and checks the entity options after a reconnect. A step fails when either DUMP line is
missing, and a rejected update also needs the device's rejection log line, so an update that never arrived cannot
pass. Exits 1 on a failed step and prints PASS otherwise.
-`wizard_drive.py HOST`: finishes the device wizard through aioesphomeapi 46.28 (`device_wizard`,
`wizard_input_set`), answers the subscriptions the inputs create and checks the values reached the entities.
## Static buffers (`cg.static_buffer`)
`scripts/static_buffer/` holds two local external components and `static-buffer-dut.yaml`, which loads them:
-`sbtest` allocates three buffers with `cg.static_buffer` (37 `uint8_t`, 5 `uint32_t`, and `const char *` sized by
a C++ expression) and checks at setup that they are zeroed and aligned, then fills each completely; at
`dump_config()` it checks no other instance's fill reached them. Two instances prove the buffers are distinct.
Both `SBTEST` lines end in `PASS` or `FAIL`; the `dump_config()` one passes only when the setup checks passed too.
-`sbpsram` asks for `psram=True` and uses the pointer codegen hands it: the static buffer without PSRAM, or a
zeroed `psram_buffer<T>()` with a `psram:` block, which logs `external=1` (nullptr if the allocation failed).
It also gets a plain 8 word `uint32_t` static buffer, which must be non-null, aligned and zeroed at start and read
back its fill (`plain=1`). Its `dump_config()` line ends in `PASS` only when both readbacks match, both buffers
were zeroed at start and, with PSRAM, the PSRAM one is in external RAM.
## Gotchas
- Some "classic" devkits with a CP2102N carry an ESP32-PICO-D4. GPIO16 and GPIO17 are its flash pins, and an
ESP-IDF build that drives them hits the interrupt watchdog. Use `board: pico32` and GPIO9/GPIO10.
- An `addressable_light` display writes nothing until its light is turned on.
- Before calling a rebased PR's change trivial, diff the PR's own diff (head against its parent) old against new: a
move that looks like a merge only can carry real commits from below.
are too short to cover the due moment). Pass pattern: "Can't connect; rebooting" exactly 60 s after boot
with no station joined, no reboot for minutes while one is, then a reboot within a second of the last
disconnect. The BW15 restarts its AP on every STA retry and cannot be tested this way.
## DHCP lease cache (#17767)
Cycle 20 s awake, `deep_sleep.enter 30s`, wake, 20 s, `App.safe_reboot()`, logging `millis()` at the
`wifi.connected` condition. An RTS hardware reset must fall back to DHCP; a software reboot and a deep
sleep wake must apply the cache. `esphome run --no-logs` silently skipped the upload on one board; use
`upload` then `logs`.
## Wi-Fi Easy Connect (DPP) enrollee (#17934)
Only an Android phone configures: Settings, network details, Add device, Share Wi-Fi (the Wi-Fi list's
QR scanner makes the phone an enrollee). Render the URI with the `qrcode` module and open the PNG.
`esp32: framework: log_level: DEBUG` plus `sdkconfig_options: CONFIG_WPA_DEBUG_PRINT: y` prints the
supplicant state machine.
## WISP bridge with an ethernet upstream (#7147)
`templates/wisp-ethernet-dut.yaml` (ESP32-S3 with a W5500, `wifi: ap:` with `ap_mode: WISP` and no station) and
`templates/wisp-client.yaml` (D1 mini joining the bridge, `http_request` to a web server on the wired LAN and to
`http://example.com/` every 15 s, `time: sntp`). Secrets: `wisp_psk` and `lan_http` (serve a text file on the host
with `python -m http.server 8787 --bind 0.0.0.0` from a scratch directory). Judge by the client's `GET status 200`
lines, the web server log showing the device's ethernet address as the source (NAT), the client's DHCP info (DNS1 is
the LAN resolver) and `SNTP synced`. The device logs `Starting AP` and `DHCP DNS server: <resolver>` on its USB
console; flashing over the air reboots it for repeat boots. Before the AP address and DHCP server moved ahead of the
AP start (the fix in that PR), the first AP start failed on every boot with `esp_netif_dhcps_option failed: 20484`
(DHCP server already started by the AP start event) and `Failed to enable NAT`, and the bridge only came up on the
retry 90 s later; a client that cannot find the SSID for the first minute on an older build is that. With the fix
the bridge is up 6 s after boot. Any change to the ESP-IDF AP start should also rerun the captive portal hammer.
## Reboot against a WPA2/WPA3 PMF access point (#21590)
- AP: `scripts/wpa3_ap/wdap.yaml` on a classic devkit. ESPHome `wifi: ap:` only, then an on_boot lambda switches it
to `WIFI_AUTH_WPA2_WPA3_PSK`, PMF capable, and registers a WIFI_EVENT handler that logs `STA JOIN` and
`STA LEAVE ... reason N` (the include header pulls in `esp_wifi.h`). It disables Wi-Fi after 420 s.
- AP leave reason 3 means the station sent a deauth; reason 2 means the AP found a stale association when the
station came back; several reason 2 before one join are rejected reconnects.
- DUTs (`wdbk.yaml` BK7238, `wdcb.yaml` CB3S, both on `wd-common.yaml`) list the AP at priority 10 and the main
network at 0, and reboot 30 s after joining the AP and 150 s after joining the main network. So they move to the
AP while it is up and fall back to the main network (where they can be flashed over the air) once its window
ends. `run.sh BUILD` flashes the AP, does OTA for the CB3S and a serial flash for the BK7238, then captures one AP
window; `an.py` prints each station's join and leave sequence and fails on a rejected reconnect, an AP that did
not switch to WPA2/WPA3 (`AP CONFIG FAILED`), a DUT missing from the AP events (`DUT_MACS`) or a BK7238 that never
joined the AP.
- Gotchas: the LibreTiny build directory has two `.uf2` files, so flash `firmware.uf2`. The BK7238 serial flash
sometimes fails to link while the chip is in a reboot loop; retry it 3 or 4 times. OTA fails if the DUT reboots
mid-upload; retry.
- Finding: BDK 3.0.78 rejects about 2 reconnects per reboot when the station left without a deauth; BDK 3.0.33
does not.
## WISP and ALWAYS_ON AP rig (#20077)
`scripts/wisp_ap`: the upstream is an ESP32-S3 with Ethernet running a WISP bridge (`wisp-upstream.yaml`) with its AP on
10.40.0.1/24 and web buttons `ap off` and `ap on`. Its subnet must not be 192.168.4.0/24, the DUT AP's default, or
nothing routes. The DUT is a classic devkit (`dut-wisp.yaml` or `dut-always-on.yaml`); the client is a D1 mini on
the DUT's AP (`wispcli.yaml`) logging its DNS server, a name lookup, a GET by name, the AP itself and a LAN file by
IP. `run.sh DUT START SECONDS T:STATE...` switches the upstream AP at the given times and `sum.py` groups the client
results by phase. Capture the D1 mini without `--reset` (a CH340 floods garbage after an RTS reset). Clients keep
the captive portal's DNS answer for its TTL.
## AP restart with WPA3 mixed and LIGHT power save (#21531)
`scripts/wifi_restart`: the WISP upstream (`wisp-upstream-wpa3.yaml`) switched to WPA2/WPA3 mixed by a script that
runs again after every `ap on`, DUTs (`rebk.yaml`, `recb.yaml` on `re-common.yaml`) with a static IP on the bridge's
subnet, a fallback AP, `power_save_mode: LIGHT` and syslog to the host through the NAT. `run.sh BUILD` switches the
AP off and on three times; `an.py` fails on a reboot, a crash or a DUT that is not back on syslog within 120 s of an
AP start. After 19 minutes the DUTs run `wifi.configure` to the main network; that keeps the static IP, so the next
OTA needs the host to have an address on the bridge's subnet as well (an interface alias). The CB3S log is on UART1
(TX1), so its configs need `logger: hardware_uart: UART1`. Runners must `pkill` only their own ports; a global pkill
stops the other runs' captures.
## Channel and network switching with a bench AP (#22142)
`scripts/wifi_channel/`: `bench-ap.yaml` makes an ESP32-S3 an AP-only bench network on a fixed channel (11 by
default; pick one your normal network does not use), and `dut-common.yaml` gives the device under test that AP at
priority 10 plus your normal network, logs every `wifi_info` ssid and channel update (with the channel the component
reports and whether it is connected), and cycles `wifi.disable` and `wifi.enable` every 90 s. `run.sh LABEL` (env
`TREE`, `PORT`, `AP_PORT`, `SC`, `PY`, and `BENCH_CHANNEL` when the AP is not on 11) flashes the device and captures 420 s while it takes the AP down with
`esptool --port <AP port> --after no-reset chip_id` (the board stays in the bootloader) and back with
`--after hard-reset`. The channel sensor must follow every move between the two networks: the run passes when the
connected channel starts and ends on the bench channel with another channel in between.
- Dropping the AP under a connected device gives a `Beacon Timeout` link loss.
- The 90 s interval fires once right after boot.
- A `Disconnected ... Association Leave` right after `Connecting to` is the aborted attempt, not a missed connect.
if (esp32_ble_tracker::global_esp32_ble_tracker->scan_running()) {
ESP_LOGE("cpinject", "scan still running after 5 s, nothing injected");
return;
}
cpinject::run();
- esp32_ble_tracker.start_scan:
continuous:true
Loaded 100 of 8144 files, more files were not shown because too many files have changed in this diff.
Show more
Reference in new issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
J. Nick Koston