An unknown Content-Length (probe failure, non-2xx, or a CDN omitting it
on HEAD) now skips the parallel prefetch so the sequential downloads
keep their per-file bars, instead of a silent multi-MB batch; the HEAD
helper returns None for unknown, distinct from a genuine zero, and
catches only RequestException. The cache probe logs its failure at
debug. The espidf tool prefetch reads sizes with .get so a missing size
only drops the bar instead of aborting the resumable prefetch. The
idedata warning logs its traceback at debug for parser regressions, and
framework_helpers joins the ESP-IDF infra CI triggers it is imported
by.
The library prefetch pool now cancels queued futures on interrupt like
the espidf pool it mirrors, prefetch and HEAD failures log their cause
at debug (a suppressed bar is traceable), and non-2xx HEAD statuses read
as unknown size. normalize_dependencies rejects a non-string version in
both spellings (a container would raise from set.add(); an int fails
opaquely in the registry), and the walk's version-less skip logs the
dependency at debug until the arduino-backend reconciliation lands.
The library prefetch pool now cancels queued futures on interrupt like
the espidf pool it mirrors, prefetch and HEAD failures log their cause
at debug (a suppressed bar is traceable), and non-2xx HEAD statuses read
as unknown size. normalize_dependencies rejects a non-string version in
both spellings (a container would raise from set.add(); an int fails
opaquely in the registry), and the walk's version-less skip logs the
dependency at debug until the arduino-backend reconciliation lands.
The wave logged 'Downloading N libraries' before checking the
extraction markers, so a warm build announced downloads it never made.
URLSource gains an is_cached probe (best-effort in the prefetch) and a
fully warm wave stays silent.
Brings in #18513 (BatchDownloadProgress and the download_with_resume
progress callback) and extends the plumbing one level:
download_from_mirrors passes a progress callback to both target kinds,
Source/ConvertedLibrary.download thread it through, and the library
wave prefetch draws one combined bar sized by parallel HEAD requests
(no sizes, no bar). The thread-local bar suppression this replaces is
removed; a batch caller passing a tracker also owns the messaging, so
the per-file Downloading line only prints for solo downloads.
A dependency whose version is a repository URL names one specific
source, so it is now detected before the backend-provided skip and
resolves as git instead of being replaced by the bundled copy. The
library header-suffix set becomes LIBRARY_HEADER_SUFFIXES so it cannot
shadow esphome.const's narrower HEADER_FILE_EXTENSIONS. Suppressing a
genuinely bundled name on an external short-name match warns (an
accidental collision would surface as link errors), a converted library
with no sources and no headers warns by name, and the walk's
unreachable nameless-dependency check is gone.
The walk now drains its frontier, prefetches the wave's registry
archives with a small thread pool (deduped by URL, progress bars
suppressed per thread since parallel bars would interleave), and then
processes the wave sequentially; the sequential download() call stays
authoritative so failures surface with resume exactly as before. Same
approach as the espidf tool prefetch in #18513.
The typed IncompatiblePlatform exception, dependency_is_usable,
warn_properties_depends, the lex_build_flags helper (espidf switches to
it), the normalize_dependencies validation, the manifest shape check,
the component-level drop-warning split, and the SCons case-sensitive
.C/.C++ suffixes all harden the shared converter independently of the
arduino backend, so they belong in this PR; the provides hook and the
version-less reconciliation stay with the backend that needs them.
The unmapped-suffix warning fired for every header-only library (the
default +<*> filter matches headers), so ArduinoJson would have warned
on every ESP8266 build. It now names exactly the source-like files
(.CPP, .ino and case-variants of the map) the case-sensitive suffix map
rejects, partial drops included, and stays quiet for headers and
metadata.
The ar shim batches the expanded object list by argv length (rc then q
appends), keeping the command line under the Windows 32767-char limit
the rspfile existed to avoid. A non-iterable dependencies value in a
manifest now warns by library name instead of raising a bare
TypeError.
The depends=-without-dependencies warning moves into the shared walk so
espidf builds report it too; the arduino backend keeps it only for
bundled libraries the walk never sees. Version-less dependencies now
pass the platform filter before the reconciliation records them, so a
platform-excluded entry no longer draws a spurious drop warning.
A converted library whose matched files all fall through the source
suffix map warns instead of silently producing an empty archive, and
unhonored precompiled/ldflags properties fields warn by library name.
The ar rspfile parser undoes ninja's POSIX inner-quote escape and the
module docstring records the one-object-per-line contract. The
includeDir comment now states that warn-and-drop is intended.
The dict-shorthand dependency form now runs the same name validation as
the list form, so an empty key or a spec overriding name with a
non-string warns instead of raising a bare TypeError from the key
builder. A lib_ignore'd version-less dependency is deliberately excluded
and no longer draws the reconciliation warning. The walk's
component-level InvalidLibrary handler mirrors dependency_is_usable:
cross-platform skips stay at debug, any other cause warns. A malformed
libArchive/dot_a_linkage value now fails naming the library like the
sibling build fields, and the suffix map gains SCons's case-sensitive .C
and .C++ so those sources compile as C++ instead of silently dropping
out of the archive.
An owner-qualified version-less dependency was skipped by both the
arduino backend (owner set) and the reconciliation (provides() knew the
short name), so nothing logged the drop; provides() now only satisfies
owner-less names, mirroring the walk's backend-provided guard. A dict
dependency entry with no name at all now warns in the normalizer like
any other malformed entry.
Bundled dependency additions are deferred until conversion finishes and
skipped when a converted library's manifest already provides the name,
so a name that is both bundled and registry-resolved cannot build twice.
External short names come from the request spec (custom Name=url form
included) instead of the URL tail. The empty-bundled-library probe now
walks the whole library tree against a shared header-suffix constant,
and the dead versioned-dependency branch is folded into the surviving
check.
Validate manifest shape once in the converter (every backend dereferences
data/build), share the InvalidLibrary filter as dependency_is_usable(),
and let the walk's reconciliation own version-less drop reporting so the
arduino backend's request-key diff, owner-no-version warning, and
node_key plumbing all go away. One memoized _provided() predicate now
answers bundled-name checks at all three sites. Tests gain shared
scaffold helpers and lose the assertions that pinned deleted messages.
The shared walk now defers every version-less skip and reconciles after
emit against the final resolution set (request keys, resolved manifest
names, and backend.provides(name) per name), so drop visibility is
correct for every backend by construction instead of by a comment-level
contract, and the arduino backend's duplicate pending_drops pass and its
short-name suppression heuristics are deleted. The provides lambda
applies _is_safe_library_name so an unsafe manifest name is simply not
provided, and the guard also rejects drive-colon names that would
escape the tree on Windows.
The check guards URL-ish names before _node_key (a malformed entry
belongs to the drop warning, not a RuntimeError) and matches against
top_level_keys so the message says exactly what it verifies; a
transitive-only node can still fail compatibility later and must not
suppress the drop report.
The walk's version-less warning stays quiet for a name the config also
requests top-level, and the provides-backend debug level carries the
ownership rationale (the arduino backend defers drops and suppresses
walk-resolved names this layer cannot know yet). _is_safe_library_name
guards both the top-level bundled check and manifest dependencies, so a
YAML libraries entry of '.' or a backslash name cannot join under the
framework tree. The drop suppression strips only the sanitized owner
prefix, keeping a library's own double underscores. An empty bundled
directory raises naming the fix instead of warning into an unlinkable
build, and the second real-converter test pins ESPHOME_DATA_DIR.
normalize_dependencies' parameter is manifest_name (the dict branch
already binds owner to a package owner), and the arduino call site
passes the library's name so its unrecognized-entry warning stops
saying 'of manifest'. A version-less dependency warns when the backend
declares no provides tree (espidf/zephyr/nrf52 have no post-emit
pickup), and the shared walk mirrors the typed IncompatiblePlatform
branch: routine platform skips stay at debug, any other InvalidLibrary
cause warns naming the component. The two real-converter tests pin
ESPHOME_DATA_DIR to tmp_path so an ambient data dir cannot leak in.
A plain-string dependencies value comma-splits into names instead of
iterating into one-character libraries; entries that normalize to
nothing warn naming the manifest, and the walk's version-less skip logs
at debug for every backend. The not-bundled warning is deferred until
after the walk and stays quiet for names resolved transitively, so a
false skipping message cannot teach users to ignore the real one. An
explicitly requested bundled library that resolves to no sources and no
headers warns that the install may be incomplete.
The converted path warns for a properties-only depends= spelling (the
walk reads the JSON key). A dependency name is validated before
becoming a path component: non-strings and separator names are
malformed entries, never joined (apply_extra_script already guards the
same shape). normalize_dependencies coerces PIO's bare string-list form
to name dicts instead of dropping it before every visibility warning. A
non-* version pin discarded for a backend-provided bundled copy warns
naming the substitution. The -I global-include promotion and the
start-group link-order contract are documented.
J. Nick Koston