The generated src/main.cpp.o was being skipped by _scan_source_symbols
(which only walked src/esphome/) and _source_file_to_component had no
rule for it. As a result setup()/loop() fell through to the
"app_framework" heuristic in const.py instead of being counted as
[esphome]core, both via the nm scan path and the linker map path.
Scan all of src/ and recognize .../src/main.cpp.o as core.
Replace `type(mock_response).content = property(...)` with a tiny
_BodyReadErrorResponse stub class. The previous form mutates
MagicMock's class globally, which can leak into every other
MagicMock-using test in this file.
`requests.Response.content` reads the body lazily, so chunked-decode,
gzip-decode, and mid-stream connection drops all surface as
RequestException subclasses on first access -- not from requests.get
itself. The previous code accessed `.content` outside the surrounding
try/except, so any of those (rare but real) errors would propagate
out of download_content instead of falling back to the cached file
or raising the user-friendly cv.Invalid.
Move `data = req.content` inside the try block so the existing error
path handles it. Two new tests cover the with-cache and no-cache
branches using a `ChunkedEncodingError` injected on `.content`.
- _read_etag collapses into a single try/except that handles both stat
and read failures (and the unlink) uniformly.
- _mtime_seconds() helper compares mtimes at whole-second resolution
instead of nanoseconds. Sub-second precision varies by filesystem
(FAT does 2s, NTFS 100ns, APFS/ext4 ns); whole seconds is the only
resolution every supported filesystem can preserve through a
set+read round-trip.
- has_remote_file_changed: walrus-assign `etag` and `new_etag` instead
of an explicit binding + immediate check.
- New test fixtures `mock_requests_head`, `mock_requests_get`,
`mock_has_remote_file_changed`, `mock_write_file` replace the
per-test `@patch(...)` decorator stacks. Tests are simpler and the
fixtures can be reused by future tests in the file.
- Both audio_file and speaker.media_player now use functools.partial to
bind path_for=_compute_local_file_path instead of an inline lambda.
- download_content_many's dedup loop becomes a dict comprehension.
Last-URL wins now (dict-comp semantics) instead of first-URL; in
practice duplicate paths only arise when the URL itself is duplicated,
so the choice is meaningless. Test updated accordingly.
- download_content_many now de-duplicates by `path` so two callers
asking for the same cache file (e.g. the same URL referenced twice
in a config) can't race on download_content's non-atomic write. When
duplicates are present, the first-seen URL for that path wins.
- Clamp `max_workers` to at least 1 so an invalid caller value can't
raise ValueError out of ThreadPoolExecutor.
- _write_etag: suppress OSError (not just FileNotFoundError) when removing
a stale sidecar -- ETag persistence is best-effort and a permission
error there must not abort an otherwise-successful download.
- Atomic-write test: patch write_file directly so the failure happens
inside the helper that does the rename, instead of before
download_content even calls it.
audio_file and speaker.media_player had byte-identical helpers that
collected web URLs from a list-of-file-configs and handed them to
download_content_many. Move that logic into external_files as
`download_web_files_in_config(config, path_for)` -- callers only have
to pass the per-component cache-path callback now.
The ETag sidecar describes a specific snapshot of the cache file. If the
cache file is replaced or edited out-of-band (manual edit, restore from
backup, another tool overwriting it), the sidecar's recorded ETag no
longer matches the bytes on disk -- using it would cause the server to
return 304 and we'd serve the wrong content from cache.
- _write_etag now os.utime()'s the sidecar to share the cache file's
st_mtime_ns immediately after writing it.
- _read_etag compares the two mtimes; if they don't match it returns
None and removes the sidecar so subsequent calls don't re-check it.
Each component that uses external_files (audio_file, speaker
media_player, ...) currently calls download_content once per file
inline inside a per-item config validator. With ~24 cached audio
files in a Home Assistant Voice PE config, that means ~24 sequential
HEAD round-trips, even when every response is a 304.
This adds download_content_many(items, ...) which fans the per-file
checks out across a ThreadPoolExecutor (capped at 16 workers so
configs with hundreds of files don't open hundreds of sockets), then
refactors audio_file and speaker.media_player to collect URLs at the
list level and call the batch helper once instead of downloading
inside each per-item validator.
Wall time for the validation phase drops from sum(latency) to roughly
max(latency) when the cache is warm.
raw.githubusercontent.com ignores If-Modified-Since (always returns
200), but honors If-None-Match with ETag (returns 304). This caused
every esphome compile/config run to re-download every cached external
file (audio_file, micro_wake_word, image, font, bme68x_bsec2, etc.)
sourced from a /raw/ URL.
- Send If-None-Match with the cached ETag when present
- Persist the ETag from each download in a hidden sidecar file
(.{name}.etag) and refresh it when a 304 carries a new ETag
- Replace path.write_bytes() with helpers.write_file() so downloads are
written atomically and can no longer leave partially-written cache
files behind on crash
Avoid per-file HTTP HEAD requests during config validation when running
esphome logs against a previously-cached project. The skip_external_update
flag was already plumbed for git operations, but external_files.download_content
ignored it. Thread it through CORE so audio_file, micro_wake_word,
speaker/media_player, image, font, and bme68x_bsec2 reuse cached files
without per-URL network round-trips when the file already exists locally.
- Add `--` terminator before submodule paths (both clone and refresh
paths) so a path beginning with `-` cannot be parsed as a git option.
- Reword the refresh-fetch comment: this fetch also runs when ref is
None, in which case it pulls the remote default branch.
Fixesesphome/esphome#11550
When clone_or_update is called with a ref or submodules, the initial
clone uses --depth=1 but the subsequent git fetch and git submodule
update commands run deep, pulling the full history on every refresh.
This causes excess network traffic and disk usage for external_components,
packages, and dashboard imports.
Add --depth=1 to all four call sites (initial fetch, initial submodule
init, refresh fetch, refresh submodule update). Shallow fetches still
advance the local clone to the current remote tip when it moves, so the
refresh path keeps working as before.
J. Nick Koston