- _write_etag: suppress OSError (not just FileNotFoundError) when removing
a stale sidecar -- ETag persistence is best-effort and a permission
error there must not abort an otherwise-successful download.
- Atomic-write test: patch write_file directly so the failure happens
inside the helper that does the rename, instead of before
download_content even calls it.
audio_file and speaker.media_player had byte-identical helpers that
collected web URLs from a list-of-file-configs and handed them to
download_content_many. Move that logic into external_files as
`download_web_files_in_config(config, path_for)` -- callers only have
to pass the per-component cache-path callback now.
The ETag sidecar describes a specific snapshot of the cache file. If the
cache file is replaced or edited out-of-band (manual edit, restore from
backup, another tool overwriting it), the sidecar's recorded ETag no
longer matches the bytes on disk -- using it would cause the server to
return 304 and we'd serve the wrong content from cache.
- _write_etag now os.utime()'s the sidecar to share the cache file's
st_mtime_ns immediately after writing it.
- _read_etag compares the two mtimes; if they don't match it returns
None and removes the sidecar so subsequent calls don't re-check it.
Each component that uses external_files (audio_file, speaker
media_player, ...) currently calls download_content once per file
inline inside a per-item config validator. With ~24 cached audio
files in a Home Assistant Voice PE config, that means ~24 sequential
HEAD round-trips, even when every response is a 304.
This adds download_content_many(items, ...) which fans the per-file
checks out across a ThreadPoolExecutor (capped at 16 workers so
configs with hundreds of files don't open hundreds of sockets), then
refactors audio_file and speaker.media_player to collect URLs at the
list level and call the batch helper once instead of downloading
inside each per-item validator.
Wall time for the validation phase drops from sum(latency) to roughly
max(latency) when the cache is warm.
raw.githubusercontent.com ignores If-Modified-Since (always returns
200), but honors If-None-Match with ETag (returns 304). This caused
every esphome compile/config run to re-download every cached external
file (audio_file, micro_wake_word, image, font, bme68x_bsec2, etc.)
sourced from a /raw/ URL.
- Send If-None-Match with the cached ETag when present
- Persist the ETag from each download in a hidden sidecar file
(.{name}.etag) and refresh it when a 304 carries a new ETag
- Replace path.write_bytes() with helpers.write_file() so downloads are
written atomically and can no longer leave partially-written cache
files behind on crash
Avoid per-file HTTP HEAD requests during config validation when running
esphome logs against a previously-cached project. The skip_external_update
flag was already plumbed for git operations, but external_files.download_content
ignored it. Thread it through CORE so audio_file, micro_wake_word,
speaker/media_player, image, font, and bme68x_bsec2 reuse cached files
without per-URL network round-trips when the file already exists locally.
- Add `--` terminator before submodule paths (both clone and refresh
paths) so a path beginning with `-` cannot be parsed as a git option.
- Reword the refresh-fetch comment: this fetch also runs when ref is
None, in which case it pulls the remote default branch.
Fixesesphome/esphome#11550
When clone_or_update is called with a ref or submodules, the initial
clone uses --depth=1 but the subsequent git fetch and git submodule
update commands run deep, pulling the full history on every refresh.
This causes excess network traffic and disk usage for external_components,
packages, and dashboard imports.
Add --depth=1 to all four call sites (initial fetch, initial submodule
init, refresh fetch, refresh submodule update). Shallow fetches still
advance the local clone to the current remote tip when it moves, so the
refresh path keeps working as before.
Cover the four paths introduced by the wake split:
- default manifests (recursive_sources=False) skip subdirs entirely,
- opt-in manifests walk non-subpackage subdirs with /-joined paths,
- subdirs containing __init__.py are skipped to avoid double-counting
existing component subpackages, and __pycache__ is always skipped,
- FILTER_SOURCE_FILES entries accept /-joined subpaths.
Every \`esphome\` CLI invocation pays the cost of whatever \`esphome/__main__.py\`
imports at module scope before the requested command even runs. This
moves three heavy imports into the functions that actually use them:
- \`esphome.zeroconf\` (discover_mdns_devices): only needed for the
name_add_mac_suffix OTA discovery path.
- \`esphome.writer\`: only needed by compile/clean paths.
- \`esphome.yaml_util\`: only needed by codegen, config dump, and rename.
Local measurement drops from ~75ms to ~47ms (-37%) for a cold
\`python -c 'import esphome.__main__'\`. The zeroconf chain alone accounts
for most of the gain — the case that motivated the CI budget check.
Also adds a module-level note explaining the intent so future PRs don't
innocently promote these back to the top.
Retargets four test patches from \`esphome.__main__.discover_mdns_devices\`
to \`esphome.zeroconf.discover_mdns_devices\` now that the symbol is only
bound inside the function that calls it.
- Add unit tests asserting cv.Invalid when `substitutions: !include list.yaml`
resolves to a non-mapping, covering both do_substitution_pass and
do_packages_pass.
- Note in resolve_substitutions_block that the resolve is single-shot and
chained top-level includes are not supported (matches _walk_packages for
`packages: !include`).
- Seed `resolve_include` context with `command_line_substitutions` so
`substitutions: !include ${var}.yaml` can reference CLI-provided vars
in the include filename (parallels the `packages: !include` path).
- Validate shape of resolved substitutions in `do_packages_pass` and raise
`cv.Invalid` under `CONF_SUBSTITUTIONS` instead of letting `UserDict()`
fail with a low-level exception on a non-mapping.
- Fixture 17 exercises the CLI-templated include filename.
Resolve a deferred IncludeFile before validating the substitutions shape in
do_substitution_pass, and before wrapping it in UserDict in do_packages_pass.
Fixesesphome/esphome#15848
The labels were there to help humans scanning the generated main.cpp
find component boundaries, but they were:
- Unreliable: CORE.flush_tasks can interleave coroutines on each
await, so a component's later statements can land in another
component's begin/end block.
- Load-bearing for a pile of complexity: a tuple return from
_wrap_in_iifes, a has_iife flag, a comment-only detector to
suppress trailing end-markers for comment-only components, and
a brittle `"[]()" in line` check that could false-positive on
YAML dumps containing lambda syntax.
- Not actually needed — generated main.cpp is a build artifact
rarely read by anyone, and cg.LineComment("name:") already puts
the component name at the start of its block.
ComponentMarker stays as a pure chunking sentinel — it tells
cpp_main_section where component boundaries are (for grouping) but
produces no C++ output. _wrap_in_iifes returns a plain list again.
Added a regression test for the now-defused case of a comment
containing "[]()" that was previously flagged by review.
- Count { and } characters per line instead of matching whole-line
tokens. Current codegen only emits scope braces as standalone lines
(from cg.with_local_variable()), but the defensive change is robust
against future codegen emitting inline control flow like
`if (cond) {` or `} else {` on one line.
- Add a regression test covering those inline-brace patterns.
- Fix stale docstrings on ComponentMarker and cpp_main_section that
still claimed "stack frame released on return" and described the
IIFEs as "noinline". The IIFEs have no noinline attribute and rely
on scope-based lifetime shortening rather than guaranteed frames.
Rename the bracket markers from "// === X ===" (same on both sides)
to "// === begin X ===" and "// === end X ===" so the generated
main.cpp reads unambiguously when scanning by component. Comment-only
components still get a single "begin X" marker since they have no
IIFE to close.
The marker comment was being emitted as the first line *inside* each
IIFE:
[]() {
// === logger ===
// logger:
// ...
...
}();
That works but buries the component label inside the lambda body, so
scanning generated main.cpp to find "where does component X's setup
live" is harder than it needs to be. Emit the marker before and after
the IIFE instead:
// === logger ===
[]() {
// logger:
// ...
...
}();
// === logger ===
Comment-only components (e.g. sha256, async_tcp, empty platforms like
binary_sensor:) don't grow a useless trailing duplicate marker —
when there's no IIFE to bracket, the marker is emitted once.
J. Nick Koston