- _read_etag collapses into a single try/except that handles both stat
and read failures (and the unlink) uniformly.
- _mtime_seconds() helper compares mtimes at whole-second resolution
instead of nanoseconds. Sub-second precision varies by filesystem
(FAT does 2s, NTFS 100ns, APFS/ext4 ns); whole seconds is the only
resolution every supported filesystem can preserve through a
set+read round-trip.
- has_remote_file_changed: walrus-assign `etag` and `new_etag` instead
of an explicit binding + immediate check.
- New test fixtures `mock_requests_head`, `mock_requests_get`,
`mock_has_remote_file_changed`, `mock_write_file` replace the
per-test `@patch(...)` decorator stacks. Tests are simpler and the
fixtures can be reused by future tests in the file.
- Both audio_file and speaker.media_player now use functools.partial to
bind path_for=_compute_local_file_path instead of an inline lambda.
- download_content_many's dedup loop becomes a dict comprehension.
Last-URL wins now (dict-comp semantics) instead of first-URL; in
practice duplicate paths only arise when the URL itself is duplicated,
so the choice is meaningless. Test updated accordingly.
- download_content_many now de-duplicates by `path` so two callers
asking for the same cache file (e.g. the same URL referenced twice
in a config) can't race on download_content's non-atomic write. When
duplicates are present, the first-seen URL for that path wins.
- Clamp `max_workers` to at least 1 so an invalid caller value can't
raise ValueError out of ThreadPoolExecutor.
- _write_etag: suppress OSError (not just FileNotFoundError) when removing
a stale sidecar -- ETag persistence is best-effort and a permission
error there must not abort an otherwise-successful download.
- Atomic-write test: patch write_file directly so the failure happens
inside the helper that does the rename, instead of before
download_content even calls it.
audio_file and speaker.media_player had byte-identical helpers that
collected web URLs from a list-of-file-configs and handed them to
download_content_many. Move that logic into external_files as
`download_web_files_in_config(config, path_for)` -- callers only have
to pass the per-component cache-path callback now.
The ETag sidecar describes a specific snapshot of the cache file. If the
cache file is replaced or edited out-of-band (manual edit, restore from
backup, another tool overwriting it), the sidecar's recorded ETag no
longer matches the bytes on disk -- using it would cause the server to
return 304 and we'd serve the wrong content from cache.
- _write_etag now os.utime()'s the sidecar to share the cache file's
st_mtime_ns immediately after writing it.
- _read_etag compares the two mtimes; if they don't match it returns
None and removes the sidecar so subsequent calls don't re-check it.
Each component that uses external_files (audio_file, speaker
media_player, ...) currently calls download_content once per file
inline inside a per-item config validator. With ~24 cached audio
files in a Home Assistant Voice PE config, that means ~24 sequential
HEAD round-trips, even when every response is a 304.
This adds download_content_many(items, ...) which fans the per-file
checks out across a ThreadPoolExecutor (capped at 16 workers so
configs with hundreds of files don't open hundreds of sockets), then
refactors audio_file and speaker.media_player to collect URLs at the
list level and call the batch helper once instead of downloading
inside each per-item validator.
Wall time for the validation phase drops from sum(latency) to roughly
max(latency) when the cache is warm.
raw.githubusercontent.com ignores If-Modified-Since (always returns
200), but honors If-None-Match with ETag (returns 304). This caused
every esphome compile/config run to re-download every cached external
file (audio_file, micro_wake_word, image, font, bme68x_bsec2, etc.)
sourced from a /raw/ URL.
- Send If-None-Match with the cached ETag when present
- Persist the ETag from each download in a hidden sidecar file
(.{name}.etag) and refresh it when a 304 carries a new ETag
- Replace path.write_bytes() with helpers.write_file() so downloads are
written atomically and can no longer leave partially-written cache
files behind on crash
Avoid per-file HTTP HEAD requests during config validation when running
esphome logs against a previously-cached project. The skip_external_update
flag was already plumbed for git operations, but external_files.download_content
ignored it. Thread it through CORE so audio_file, micro_wake_word,
speaker/media_player, image, font, and bme68x_bsec2 reuse cached files
without per-URL network round-trips when the file already exists locally.
- Add `--` terminator before submodule paths (both clone and refresh
paths) so a path beginning with `-` cannot be parsed as a git option.
- Reword the refresh-fetch comment: this fetch also runs when ref is
None, in which case it pulls the remote default branch.
Fixesesphome/esphome#11550
When clone_or_update is called with a ref or submodules, the initial
clone uses --depth=1 but the subsequent git fetch and git submodule
update commands run deep, pulling the full history on every refresh.
This causes excess network traffic and disk usage for external_components,
packages, and dashboard imports.
Add --depth=1 to all four call sites (initial fetch, initial submodule
init, refresh fetch, refresh submodule update). Shallow fetches still
advance the local clone to the current remote tip when it moves, so the
refresh path keeps working as before.
- Add unit tests asserting cv.Invalid when `substitutions: !include list.yaml`
resolves to a non-mapping, covering both do_substitution_pass and
do_packages_pass.
- Note in resolve_substitutions_block that the resolve is single-shot and
chained top-level includes are not supported (matches _walk_packages for
`packages: !include`).
- Seed `resolve_include` context with `command_line_substitutions` so
`substitutions: !include ${var}.yaml` can reference CLI-provided vars
in the include filename (parallels the `packages: !include` path).
- Validate shape of resolved substitutions in `do_packages_pass` and raise
`cv.Invalid` under `CONF_SUBSTITUTIONS` instead of letting `UserDict()`
fail with a low-level exception on a non-mapping.
- Fixture 17 exercises the CLI-templated include filename.
Resolve a deferred IncludeFile before validating the substitutions shape in
do_substitution_pass, and before wrapping it in UserDict in do_packages_pass.
Fixesesphome/esphome#15848
The labels were there to help humans scanning the generated main.cpp
find component boundaries, but they were:
- Unreliable: CORE.flush_tasks can interleave coroutines on each
await, so a component's later statements can land in another
component's begin/end block.
- Load-bearing for a pile of complexity: a tuple return from
_wrap_in_iifes, a has_iife flag, a comment-only detector to
suppress trailing end-markers for comment-only components, and
a brittle `"[]()" in line` check that could false-positive on
YAML dumps containing lambda syntax.
- Not actually needed — generated main.cpp is a build artifact
rarely read by anyone, and cg.LineComment("name:") already puts
the component name at the start of its block.
ComponentMarker stays as a pure chunking sentinel — it tells
cpp_main_section where component boundaries are (for grouping) but
produces no C++ output. _wrap_in_iifes returns a plain list again.
Added a regression test for the now-defused case of a comment
containing "[]()" that was previously flagged by review.
- Count { and } characters per line instead of matching whole-line
tokens. Current codegen only emits scope braces as standalone lines
(from cg.with_local_variable()), but the defensive change is robust
against future codegen emitting inline control flow like
`if (cond) {` or `} else {` on one line.
- Add a regression test covering those inline-brace patterns.
- Fix stale docstrings on ComponentMarker and cpp_main_section that
still claimed "stack frame released on return" and described the
IIFEs as "noinline". The IIFEs have no noinline attribute and rely
on scope-based lifetime shortening rather than guaranteed frames.
Rename the bracket markers from "// === X ===" (same on both sides)
to "// === begin X ===" and "// === end X ===" so the generated
main.cpp reads unambiguously when scanning by component. Comment-only
components still get a single "begin X" marker since they have no
IIFE to close.
The marker comment was being emitted as the first line *inside* each
IIFE:
[]() {
// === logger ===
// logger:
// ...
...
}();
That works but buries the component label inside the lambda body, so
scanning generated main.cpp to find "where does component X's setup
live" is harder than it needs to be. Emit the marker before and after
the IIFE instead:
// === logger ===
[]() {
// logger:
// ...
...
}();
// === logger ===
Comment-only components (e.g. sha256, async_tcp, empty platforms like
binary_sensor:) don't grow a useless trailing duplicate marker —
when there's no IIFE to bracket, the marker is emitted once.
Some components (sha256, async_tcp, network, empty text_sensor:, etc.)
emit only a ComponentMarker plus config-dump comments and no actual
C++ statements. Wrapping those in a `[]() { ... }();` IIFE is pure
clutter in the generated main.cpp — the IIFE has no body.
When _wrap_in_iifes sees a chunk whose lines are all // comments,
emit them verbatim instead of wrapping. Peak stack and flash are
unchanged on apollo and neargaragedoor since GCC was already
eliding the empty IIFEs; this just makes the generated code read
cleanly to humans.
J. Nick Koston