Compare commits

...
Author SHA1 Message Date
J. Nick Koston c610ab837d Merge branch 'dev' into library-versionless-dependencies 2026-10-21 18:24:19 -10:00
J. Nick Koston 406d2069a5 Flatten the owner check ruff flagged 2026-10-21 17:22:47 -10:00
J. Nick Koston 5dabeb1525 Split an owner written into a dependency name
"owner/Name" given as one string, in a list entry, a dict entry or a
depends= line, normalizes into owner and name like the dict form does,
so the request matches the package the registry resolves it to and is
wired instead of being pruned.
2026-10-21 17:10:31 -10:00
J. Nick Koston 0d7cc4646d Merge branch 'dev' into library-versionless-dependencies 2026-10-21 17:00:38 -10:00
J. Nick Koston e2eafda5f6 Settle an empty version like a missing one
A dependency pinned to an empty string carries no version either, so it
takes the deferred path and a registry miss is reported for its
requester instead of being skipped quietly. A leftover docstring line
in normalize_dependencies is removed.
2026-10-21 16:32:35 -10:00
J. Nick Koston 39d8bf58ae Wire a bundled name to a replacement pulled in later
A request the framework provides is kept for the final wiring too, so an
explicit library of that name a later round pulls in still beats the
bundled copy, as it does when it is listed up front.
2026-10-21 16:22:52 -10:00
J. Nick Koston afd5d324ed Wire version-less requests in one pass once the graph is final
The settle step now only decides what to fetch by name; every request is
kept for the whole walk and wired at the end to the explicit library of
that name if there is one, else to the node fetched by name, else
reported for the libraries that are built. Nodes nothing reaches are
dropped. That replaces the separate yield, fold, re-match and late
report passes and the miss bookkeeping. The depends= parser also serves
the string form of a library.json dependencies entry.
2026-10-21 16:20:14 -10:00
J. Nick Koston fa823d1a30 Ignore an owner-qualified dependency on the requester itself too
The self check compares the requested owner and name, and a library's
resolved package name, against the requester's own names.
2026-10-21 16:04:58 -10:00
J. Nick Koston 5088c4a956 Defer a key whose source changed mid-wave, fold aliases by resolved owner
A git source replacing a registry key within the same wave no longer
trips the stale-record guard; the key waits for the next wave. The
matcher compares a requested owner with the owner the registry resolved,
and two names fetched for one package fold into one node. Registry
misses are reported after the prune, only for libraries that are built,
and a dropped exact or capped depends= constraint is logged at info.
2026-10-21 15:52:10 -10:00
J. Nick Koston b405cbb80a Report a registry miss per requester once the graph is final
The final pass runs before the prune, so a library only a yielded one
pulled in can still satisfy a missed name, and it reports every library
whose name stayed unmatched, including requesters from later rounds.
2026-10-21 15:39:05 -10:00
J. Nick Koston 4b0bef2569 Drop what only a yielded library needed
After a library fetched by name yields to an explicit one, the components
no top-level library reaches any more are not converted either.
2026-10-21 15:27:31 -10:00
J. Nick Koston 70a33c60fa Resolve a missed key again as a git source and let a by-name fetch yield
A key the registry did not know is resolved again when a later manifest
names it as a git or local source. Once the graph is final, a library
fetched by name yields to one asked for explicitly under the same
package or manifest name, whichever round brought it in; that replaces
the fold limited to the same resolved package.
2026-10-21 15:14:31 -10:00
J. Nick Koston c26d5502f9 Re-match only the names the registry missed, and say when one is satisfied
The final pass considers only the by-name nodes whose registry miss was
skipped, not nodes dropped by the platform filter, and logs at info when
a name warned about as missing is satisfied by a library pulled in later.
2026-10-21 15:10:00 -10:00
J. Nick Koston 13ef930573 Match a missed name against libraries pulled in later
A bare name the registry did not know is fetched in the same round as the
libraries that may later satisfy it, so once the graph is finished the
edges to such missed nodes are matched once more, without another fetch.
2026-10-21 15:01:10 -10:00
J. Nick Koston 499f716a7d Leave Arduino-bundled names to arduino-esp32 in the clang-tidy conversion too
The clang-tidy project converts the PlatformIO libraries through the same
converter as the esp32 build, so it passes the bundled library names for
an Arduino environment as well.
2026-10-21 13:56:52 -10:00
J. Nick Koston 0d9402cf82 Log a manifest-name match and ignore a dependency on the requester itself
A match found only through a library's manifest name is logged at info,
so a chance name collision can be pinned to the right package. A
manifest listing its own name as a dependency no longer fetches a
second copy of itself.
2026-10-21 12:52:08 -10:00
J. Nick Koston a1d730704e Compute "fetched by name" from the node and share the bundled-name check
A node fetched by name is a registry node no listed or versioned request
asked for, so nothing has to set or clear a flag. One closure answers
whether the framework bundles a name for both the versioned and the
deferred path, once per name. Requesters that spell one unlisted name
differently share a node within a settle round.
2026-10-21 12:41:42 -10:00
J. Nick Koston 5a5d295455 Defer every depends= entry and keep a pinned node fatal on a miss
An Arduino version constraint is not PlatformIO syntax, so the parser
keeps only the name and every depends= entry takes the deferred path,
where a listed library satisfies it and a registry miss is skipped. A
node is only miss-tolerant while it was created by name and never given
a version; a versioned request clears that, and a requirement conflict
always raises. The matcher skips the requester and checks package names
before manifest names; the backend is asked once per bundled name.
2026-10-21 12:24:29 -10:00
J. Nick Koston 88424041b1 Cover the matcher's miss and the by-name fold
The fold test keyed the by-name node's manifest under a name the fake
download never read, so the fold never ran. The matcher's trailing
return is replaced by a next() so coverage attributes the miss.
2026-10-21 12:08:10 -10:00
J. Nick Koston a0c9f3aac7 Match a listed library before the bundled copy and fold by-name duplicates
A library in the graph wins over a same-named bundled one, as on the
Arduino backend. A node fetched by name alone that resolves to a package
another manifest later pins folds into that node. Pending entries are
kept per requester so a re-walk replaces them. ESP32 Arduino warns when
a manifest depends on a bundled library that is not enabled.
2026-10-21 12:00:40 -10:00
J. Nick Koston 4d35fb83a4 [core] Resolve version-less library dependencies like PlatformIO
A library.properties depends= line and a library.json dependency without
a version were dropped, so the converted ESP-IDF component never required
the library it includes from. Parse depends= as dependencies, wire a
version-less dependency to the library that satisfies it once the graph
is known, and otherwise fetch it from the registry by name. Framework
bundled names and lib_ignore are still skipped; a registry miss for such
a dependency warns instead of failing the build.
2026-10-21 11:52:51 -10:00
9 changed files with 900 additions and 239 deletions
-2
View File
@@ -44,7 +44,6 @@ from esphome.platformio.library import (
normalize_dependencies,
parse_library_json,
parse_library_properties,
warn_properties_depends,
)
_LOGGER = logging.getLogger(__name__)
@@ -283,7 +282,6 @@ def _bundled_library(framework_path: Path, name: str) -> ArduinoLibrary:
"resolved automatically; add them with add_library() if needed",
name,
)
warn_properties_depends(name, data)
build = data.get("build")
if isinstance(build, dict) and build.get("extraScript"):
# Scripts only run on the converted path; building without
+19 -1
View File
@@ -3813,6 +3813,20 @@ def _write_sdkconfig():
clean_build(clear_pio_cache=False)
def _arduino_provides(name: str) -> bool:
"""Whether arduino-esp32 bundles the library a manifest depends on, so it is
never fetched by name; it still has to be enabled to build."""
if name not in ARDUINO_DISABLED_LIBRARIES:
return False
if name not in CORE.data[KEY_ESP32][KEY_ARDUINO_LIBRARIES]:
_LOGGER.warning(
"A library depends on the Arduino library %s, which is not enabled; "
"add it under esphome: libraries:",
name,
)
return True
def _write_idf_component_yml():
yml_path = CORE.relative_build_path("src/idf_component.yml")
dependencies: dict[str, dict] = {}
@@ -3834,7 +3848,11 @@ def _write_idf_component_yml():
# IDF sees the same requirement twice; converted components reach it through
# ${ESPHOME_PROJECT_MANAGED_COMPONENTS}.
managed = set(CORE.data[KEY_ESP32].get(KEY_COMPONENTS, {}))
converted = generate_idf_components(libraries, managed=managed)
converted = generate_idf_components(
libraries,
managed=managed,
provides=_arduino_provides if CORE.using_arduino else None,
)
# IDF names a component after its directory and a later registration of the
# same name replaces the earlier one, so a stub beside a converted library of
# the same name (espressif/libsodium vs esphome/libsodium) would win or lose
+12 -2
View File
@@ -266,14 +266,24 @@ def _convert_pio_libs(
in the real esp32 build; converting them too would give IDF the same requirement
twice. Arduino below IDF 6.0 has them rule-gated off, so there they convert.
"""
from esphome.components.esp32 import arduino_bundles_libsodium
from esphome.components.esp32 import (
ARDUINO_DISABLED_LIBRARIES,
arduino_bundles_libsodium,
)
from esphome.espidf.component import generate_idf_components
libraries = _parse_lib_deps(platformio_ini, framework)
# Same rule as noise._use_managed_components and the manifest
managed = set() if arduino_bundles_libsodium() else _esphome_manifest_deps()
deps: dict[str, dict[str, str]] = {}
for component in generate_idf_components(libraries, managed=managed):
# As in the real esp32 build, a dependency on an Arduino-bundled library is
# left to arduino-esp32 instead of being fetched from the registry
provides = (
ARDUINO_DISABLED_LIBRARIES.__contains__ if framework == "arduino" else None
)
for component in generate_idf_components(
libraries, managed=managed, provides=provides
):
deps[component.get_sanitized_name()] = {"override_path": str(component.path)}
return deps
+7 -1
View File
@@ -7,6 +7,7 @@ for each resolved library, running any PlatformIO ``extraScript``, and the
ESP-IDF platform/framework compatibility defaults.
"""
from collections.abc import Callable
import logging
import os
from pathlib import Path
@@ -288,18 +289,23 @@ def _emit_idf_component(component: IDFComponent) -> None:
def generate_idf_components(
libraries: list[Library], managed: set[str] | None = None
libraries: list[Library],
managed: set[str] | None = None,
provides: Callable[[str], bool] | None = None,
) -> list[IDFComponent]:
"""Resolve and convert a batch of PlatformIO libraries to IDF components.
``managed`` names registry components already in the project manifest; they
are not converted, since IDF rejects the same requirement from two components.
Converted components reach them through ``${ESPHOME_PROJECT_MANAGED_COMPONENTS}``.
``provides`` names the libraries the framework bundles, so a manifest dependency on
one is never fetched from the registry.
"""
backend = LibraryBackend(
platform=ESP32_PLATFORM,
framework=_idf_framework(),
emit=_emit_idf_component,
cache_key="idf",
provides=provides,
)
return convert_libraries(libraries, backend, provided=managed)
+236 -101
View File
@@ -609,9 +609,34 @@ def parse_library_properties(library_properties_path: PathType):
if not value.strip():
continue
data[key.strip()] = value.strip()
if depends := data.pop("depends", None):
data["dependencies"] = parse_properties_depends(depends)
return data
def parse_properties_depends(depends: str) -> list[dict[str, str]]:
"""The Arduino ``depends=Name, Other (>=1.0)`` line as name-only dependency entries.
An Arduino constraint is not PlatformIO syntax, so it is dropped; the name is
satisfied by a listed library or the registry's current version, as the walk
does for every version-less dependency.
"""
entries: list[dict[str, str]] = []
for item in depends.split(","):
name, _, constraint = item.strip().partition("(")
if not (name := name.strip()):
continue
if constraint:
# A plain lower bound is met by the registry's current version; an
# exact or capped one may not be, so say so
log = (
_LOGGER.debug if constraint.lstrip().startswith(">=") else _LOGGER.info
)
log("Ignoring version constraint on depends=%s", item.strip())
entries.append({"name": name})
return entries
def _make_registry_client() -> Any:
"""Create a minimal PlatformIO registry client with no system filtering.
@@ -730,20 +755,6 @@ def join_flag_args(tokens: Iterable[str], owner: str) -> list[str]:
return out
def warn_properties_depends(name: str, data: object) -> None:
"""Warn for ``depends=``-only manifests; the walk reads only the JSON
``dependencies`` key, so they would otherwise drop silently."""
if isinstance(data, dict) and not data.get("dependencies") and data.get("depends"):
# INFO: common and unactionable for transitive libraries; a WARNING
# on every build would train users to ignore the stream
_LOGGER.info(
"Library %s declares dependencies via library.properties "
"depends=, which are not resolved automatically; add them with "
"add_library() if needed",
name,
)
def dependency_is_usable(
dep: dict, platform: str | None, framework: str, requester: str
) -> bool:
@@ -762,6 +773,20 @@ def dependency_is_usable(
return True
def _split_owner(entry: dict) -> dict:
"""``owner/Name`` written as one name becomes owner and name, as in the dict form,
so the name matches the package the registry resolves it to."""
name, owner = entry.get("name"), entry.get("owner")
if isinstance(name, str) and owner is None and "://" not in name:
head, sep, tail = name.partition("/")
# A registry owner is a plain identifier; anything else is left to the
# backend's own checks
owner_ok = head.replace("-", "").replace("_", "").isalnum()
if sep and owner_ok and tail and "/" not in tail:
return {**entry, "owner": head, "name": tail}
return entry
def _valid_dependency_entry(entry: dict, manifest_name: str) -> bool:
"""Whether a normalized entry carries a usable name (non-empty string),
version (string, if present), and owner (string, if present); invalid
@@ -790,16 +815,12 @@ def normalize_dependencies(
so callers see a uniform list. ``manifest_name`` names the manifest in the
warning for entries that cannot be normalized.
Bare-name spellings carry no version; the dependency walk later drops
version-less entries at DEBUG, since they are usually names of bundled
framework libraries (Wire, SPI) that need no registry install.
Bare-name spellings carry no version; the walk settles them once the graph is known.
"""
if dependencies is None:
return []
if isinstance(dependencies, str):
# A plain string is one or more comma-separated names; iterating it
# as a list would shred it into one-character "libraries"
return [{"name": n.strip()} for n in dependencies.split(",") if n.strip()]
return [_split_owner(entry) for entry in parse_properties_depends(dependencies)]
if isinstance(dependencies, dict):
normalized = []
for raw_name, spec in dependencies.items():
@@ -826,10 +847,10 @@ def normalize_dependencies(
for entry in dependencies:
if isinstance(entry, dict):
if _valid_dependency_entry(entry, manifest_name):
normalized.append(entry)
normalized.append(_split_owner(entry))
elif isinstance(entry, str) and entry:
# PIO also accepts a bare list of names ("dependencies": ["Wire"])
normalized.append({"name": entry})
normalized.append(_split_owner({"name": entry}))
else:
_LOGGER.warning(
"Ignoring unrecognized dependency entry %r of %s",
@@ -971,54 +992,55 @@ def is_lib_ignored(name: str | None, lib_ignore: set[str]) -> bool:
)
def _reconcile_versionless_skips(
skipped_versionless: list[tuple[Any, Any, str]],
def _satisfying_key(
name: str,
owner: str | None,
requester: str,
nodes: dict[str, _LibNode],
components: dict[str, ConvertedLibrary],
backend: LibraryBackend,
) -> None:
"""Warn for version-less deps nothing satisfied, and record the
backend-provided ones in ``backend.provided_requests`` for its
post-emit reconciliation; a silent drop surfaces as link errors far
from the cause."""
resolved_manifest_names = {c.data.get("name") for c in components.values()}
# A treeless backend can never supply a bundled name; noise for it
log = _LOGGER.warning if backend.provides is not None else _LOGGER.debug
warned: set[str] = set()
for dep_name, dep_owner, requester in skipped_versionless:
if not isinstance(dep_name, str) or not dep_name or dep_name in warned:
continue
if dep_name in components:
# A version-less dep's request key is the name itself
continue
if (
not dep_owner
and backend.provides is not None
and backend.provides(dep_name)
):
# provides() only satisfies owner-less names (same guard as
# the walk's skip); record for the post-emit reconciliation.
# Checked before the manifest-name evidence so the overlap
# case warns once, in the backend's own suppression loop
backend.provided_requests.add(dep_name)
continue
if dep_name in resolved_manifest_names:
# Name-only evidence: a coincidental collision must stay
# visible where the user could pin it
warned.add(dep_name)
log(
"Version-less dependency %s of %s assumed satisfied by a "
"resolved library's manifest name only",
dep_name,
requester,
)
continue
warned.add(dep_name)
log(
"Dependency %s of %s has no version to resolve and nothing "
"provides it; skipping",
dep_name,
requester,
)
) -> tuple[str, bool] | None:
"""The key of a resolved library a version-less dependency names, and whether
only its manifest name matched: the registry package name (with the owner,
when the dependency gives one) is checked first; never the requester itself."""
wanted = name.lower()
def owner_of(key: str, component: ConvertedLibrary) -> str:
# The owner as requested, else the one the registry (or a git path) gave
head, sep, _ = component.name.partition("/")
return nodes[key].owner or (head if sep else "")
candidates = [
(key, component)
for key, component in components.items()
if key != requester
and (not owner or owner_of(key, component).lower() == owner.lower())
]
for key, _ in candidates:
if (nodes[key].pkgname or key).lower() == wanted:
return key, False
if owner:
return None
for key, component in candidates:
manifest_name = component.data.get("name")
if isinstance(manifest_name, str) and manifest_name.lower() == wanted:
return key, True
return None
def _own_names(
key: str, nodes: dict[str, _LibNode], components: dict[str, ConvertedLibrary]
) -> set[str]:
"""The lower-case key, requested and resolved package names and the manifest
name a resolved library goes by."""
names = {
key.lower(),
(nodes[key].pkgname or key).lower(),
components[key].name.lower(),
}
manifest_name = components[key].data.get("name")
if isinstance(manifest_name, str):
names.add(manifest_name.lower())
return names
def _fetch_source(
@@ -1147,7 +1169,12 @@ def convert_libraries(
managed components from ``add_idf_component``); unlike ``backend.provides`` they
carry an owner. They are excluded like ``lib_ignore`` so nothing is both
converted and managed, which ESP-IDF refuses to build.
A dependency named without a version is settled once the graph is known: wired
to a library already in it, left to ``backend.provides``, or fetched by name.
"""
from platformio.package.exception import UnknownPackageError
nodes: dict[str, _LibNode] = {}
# Folded into one set so every is_lib_ignored() call site honors both.
@@ -1165,10 +1192,16 @@ def convert_libraries(
else ""
)
resolved_requirements: dict[str, frozenset[str]] = {}
def add_spec(name: str | None, version: str | None, repository: str | None) -> str:
key, kind, locator = _node_key(name, version, repository)
node = nodes.get(key) or _LibNode(key=key, is_git=kind == "git")
nodes[key] = node
if kind != "registry" and not (node.is_git or node.is_local):
# A registry node (perhaps one the registry did not know) becomes a
# git or local source: resolve it again from that source
resolved_requirements.pop(key, None)
# The same key requested from two different kinds of source (or two
# different local paths) is a config mistake: one silently wins. Warn so
# it isn't a surprise. (git-vs-registry is reported separately below.)
@@ -1220,10 +1253,69 @@ def convert_libraries(
# requirement set has grown since the last time, so every requirement in the
# graph is accounted for before conversion.
components: dict[str, ConvertedLibrary] = {}
resolved_requirements: dict[str, frozenset[str]] = {}
top_level_keys = set(top_level)
# (name, owner, requester) reconciled against the final resolution set
skipped_versionless: list[tuple[Any, Any, str]] = []
def fetched_by_name(key: str) -> bool:
"""A registry node only version-less manifest dependencies asked for: a
miss skips it, and an explicit library of the same name replaces it."""
node = nodes[key]
return (
key not in top_level_keys
and not (node.is_git or node.is_local)
and not node.requirements
)
def backend_provides(name: str, owner: str | None) -> bool:
"""Whether the framework bundles an owner-less name; asked once per name,
since the backend may warn, and recorded in ``provided_requests``."""
if owner or backend.provides is None:
return False
if name in backend.provided_requests or backend.provides(name):
backend.provided_requests.add(name)
return True
return False
# Version-less dependencies per requester key, kept for the whole walk and
# wired once the graph is final; a re-walked requester replaces its entries
pending_versionless: dict[str, list[dict]] = {}
# Names fetched by name so far, whatever their case: each is asked for once
fetched: dict[tuple[str, str], str] = {}
def settle_versionless() -> list[str]:
"""Fetch by name the deferred dependencies nothing in the graph or the
framework satisfies yet; return the new keys to walk. Only a dependency
on the requester itself or with a bad name needs no wiring later."""
new_keys: list[str] = []
for requester, dependencies in pending_versionless.items():
kept: list[dict] = []
for dependency in dependencies:
name, owner = dependency["name"], dependency.get("owner")
if _owner_pkgname_to_name(owner, name).lower() in _own_names(
requester, nodes, components
):
_LOGGER.debug("Dependency %s of %s names itself", name, requester)
continue
# A library in the graph beats a bundled copy, as on the Arduino backend
if _satisfying_key(
name, owner, requester, nodes, components
) is None and not backend_provides(name, owner):
spelling = (name.lower(), (owner or "").lower())
if spelling not in fetched:
try:
dep_key = add_spec(
_owner_pkgname_to_name(owner, name), None, None
)
except RuntimeError as e:
_LOGGER.warning(
"Skipping dependency %s of %s: %s", name, requester, e
)
continue
fetched[spelling] = dep_key
new_keys.append(dep_key)
kept.append(dependency)
dependencies[:] = kept
return new_keys
worklist = deque(dict.fromkeys(top_level))
while worklist:
# Drain the frontier sequentially (spec resolution mutates shared
@@ -1245,9 +1337,17 @@ def convert_libraries(
elif node.is_local:
component = ConvertedLibrary(key, "*", LocalSource(node.local_path))
else:
owner, name, version, url, size = _resolve_registry_version(
node.owner, node.pkgname, node.requirements
)
try:
owner, name, version, url, size = _resolve_registry_version(
node.owner, node.pkgname, node.requirements
)
except UnknownPackageError as e:
# A requirement conflict is a RuntimeError and always raises
if not fetched_by_name(key):
raise
# Reported per requester once the graph is final
_LOGGER.debug("Library %s is not in the registry: %s", key, e)
continue
component = ConvertedLibrary(
_owner_pkgname_to_name(owner, name), version, URLSource(url, size)
)
@@ -1255,9 +1355,9 @@ def convert_libraries(
_prefetch_wave(wave, salt, backend.cache_key)
for key, component in wave:
node = nodes[key]
if frozenset(node.requirements) != resolved_requirements[key]:
# Requirements grew mid-wave: skip parsing a manifest the
# next wave will re-resolve and replace
if frozenset(node.requirements) != resolved_requirements.get(key):
# Requirements grew (or the source changed) mid-wave: skip parsing
# a manifest the next wave will re-resolve and replace
worklist.append(key)
continue
component.download(salt=salt, namespace=backend.cache_key)
@@ -1313,8 +1413,6 @@ def convert_libraries(
raise EsphomeError(f"Library {key} has a malformed manifest")
_LOGGER.warning("Skipping dependency %s: malformed manifest", key)
continue
warn_properties_depends(component.name, component.data)
try:
check_library_data(component.data, backend.platform, backend.framework)
except InvalidLibrary as e:
@@ -1337,27 +1435,19 @@ def convert_libraries(
# Requirements changed (we got past the short-circuit above), so
# (re)walk this component's dependencies.
node.edges = set()
pending = pending_versionless[key] = []
for dependency in normalize_dependencies(
component.data.get("dependencies"), component.name
):
if "version" not in dependency:
# Cannot resolve from the registry; the post-emit
# reconciliation owns the drop warning
dep_name = dependency.get("name")
_LOGGER.debug(
"Skip version-less dependency %r of %s",
dep_name,
component.name,
)
if not dependency.get("version"):
# An empty version is as good as none; it settles the same way
if not is_lib_ignored(
dep_name, lib_ignore
dependency.get("name"), lib_ignore
) and dependency_is_usable(
dependency, backend.platform, backend.framework, component.name
):
# Filtered or ignored deps are deliberately absent
skipped_versionless.append(
(dep_name, dependency.get("owner"), component.name)
)
pending.append(dependency)
continue
if not dependency_is_usable(
dependency, backend.platform, backend.framework, component.name
@@ -1375,11 +1465,7 @@ def convert_libraries(
dep_url = _url_or_none(dep_version)
if dep_url is not None:
dep_version = None
elif (
backend.provides is not None
and not dependency.get("owner")
and backend.provides(dep_name)
):
elif backend_provides(dep_name, dependency.get("owner")):
# The backend adds it from its own tree; resolving here
# would fetch a same-named registry package
if dep_version and dep_version != "*":
@@ -1392,11 +1478,12 @@ def convert_libraries(
)
else:
_LOGGER.debug("Skip backend-provided dependency %s", dep_name)
backend.provided_requests.add(dep_name)
continue
dep_key = add_spec(dep_name, dep_version, dep_url)
node.edges.add(dep_key)
worklist.append(dep_key)
if not worklist:
worklist.extend(settle_versionless())
# A git or local source wins over the same component requested from the
# registry. That's intentional, but warn so the dropped registry spec isn't
@@ -1420,6 +1507,56 @@ def convert_libraries(
source,
)
# Every version-less request resolves to the explicit library that goes by
# its name if the final graph has one, else to the node fetched by name for
# that package, else to the framework's bundled copy, else to nothing
explicit = {k: c for k, c in components.items() if not fetched_by_name(k)}
by_name = {k: c for k, c in components.items() if fetched_by_name(k)}
unresolved: dict[tuple[str, str], None] = {}
for requester, dependencies in pending_versionless.items():
for dependency in dependencies:
name, owner = dependency["name"], dependency.get("owner")
found = _satisfying_key(
name, owner, requester, nodes, explicit
) or _satisfying_key(name, owner, requester, nodes, by_name)
if found is None:
if not backend_provides(name, owner):
unresolved[(requester, name)] = None
continue
key, by_manifest_name = found
if by_manifest_name:
# Visible, so a chance name collision can be pinned to the right package
_LOGGER.info(
"Dependency %s of %s satisfied by %s through its manifest name",
name,
requester,
key,
)
else:
_LOGGER.debug(
"Dependency %s of %s satisfied by %s", name, requester, key
)
nodes[requester].edges.add(key)
# What no listed library needs any more is not built: a node fetched by name
# that an explicit library replaced, and whatever only it pulled in
reachable: set[str] = set()
stack = [key for key in top_level if key in components]
while stack:
if (key := stack.pop()) in reachable:
continue
reachable.add(key)
stack.extend(edge for edge in nodes[key].edges if edge in components)
for key in [key for key in components if key not in reachable]:
_LOGGER.debug("Library %s is not needed; dropping it", key)
del components[key]
for requester, name in unresolved:
if requester in components:
_LOGGER.warning(
"Dependency %s of %s was not resolved; skipping", name, requester
)
# Two graph nodes that resolve to the same component name (e.g. a package
# referenced both bare and as ``owner/name``) are not deduplicated and can
# produce conflicting component definitions. Warn so it's not silent.
@@ -1447,6 +1584,4 @@ def convert_libraries(
for component in components.values():
backend.emit(component)
_reconcile_versionless_skips(skipped_versionless, components, backend)
return [components[key] for key in top_level if key in components]
@@ -62,8 +62,9 @@ def test_write_idf_component_yml_passes_managed_components(
converted.get_sanitized_name.return_value = "esphome/other-lib"
converted.path = tmp_path / "pio_components" / "other-lib"
def fake_generate_idf_components(libraries, managed=None):
def fake_generate_idf_components(libraries, managed=None, provides=None):
captured["managed"] = managed
captured["provides"] = provides
return [converted]
monkeypatch.setattr(esp32, "generate_idf_components", fake_generate_idf_components)
@@ -71,6 +72,8 @@ def test_write_idf_component_yml_passes_managed_components(
esp32._write_idf_component_yml()
assert captured["managed"] == {"esphome/noise-c"}
# ESP-IDF bundles no Arduino libraries, so every name may be fetched
assert captured["provides"] is None
# The managed component itself is still written into the manifest deps
# directly (from KEY_COMPONENTS), just not converted a second time.
yml_path = tmp_path / "src" / "idf_component.yml"
@@ -93,7 +96,7 @@ def test_write_idf_component_yml_empty_managed_when_no_components(
captured: dict[str, set[str] | None] = {}
def fake_generate_idf_components(libraries, managed=None):
def fake_generate_idf_components(libraries, managed=None, provides=None):
captured["managed"] = managed
return []
@@ -104,6 +107,50 @@ def test_write_idf_component_yml_empty_managed_when_no_components(
assert captured["managed"] == set()
def test_write_idf_component_yml_marks_arduino_bundled_libraries_provided(
tmp_path: Path,
monkeypatch: pytest.MonkeyPatch,
) -> None:
"""Under Arduino the converter is told which names the core bundles, so a
manifest dependency on Wire or SPI is never fetched from the registry."""
_setup_core(tmp_path)
CORE.data[KEY_CORE][KEY_TARGET_FRAMEWORK] = str(Framework.ARDUINO)
CORE.data[KEY_CORE][KEY_FRAMEWORK_VERSION] = cv.Version(3, 3, 12)
CORE.data[esp32.KEY_ESP32] = {
esp32.KEY_COMPONENTS: {},
esp32.KEY_IDF_VERSION: cv.Version(5, 5, 4),
esp32.KEY_ARDUINO_LIBRARIES: {"Wire", "SPI"},
}
captured: dict[str, object] = {}
def fake_generate_idf_components(libraries, managed=None, provides=None):
captured["provides"] = provides
return []
monkeypatch.setattr(esp32, "generate_idf_components", fake_generate_idf_components)
esp32._write_idf_component_yml()
provides = captured["provides"]
assert provides("Wire") and provides("SPI")
assert not provides("Adafruit BusIO")
def test_arduino_provides_warns_for_a_bundled_library_not_enabled(
caplog: pytest.LogCaptureFixture,
) -> None:
"""A manifest dependency on a bundled Arduino library is never fetched, but
the library still has to be enabled, so an unenabled one is called out."""
CORE.reset()
CORE.data[esp32.KEY_ESP32] = {esp32.KEY_ARDUINO_LIBRARIES: {"Wire"}}
assert esp32._arduino_provides("Wire")
assert "Wire" not in caplog.text
assert esp32._arduino_provides("SPI")
assert "Arduino library SPI, which is not enabled" in caplog.text
assert not esp32._arduino_provides("Adafruit BusIO")
@pytest.mark.parametrize(
("version", "libsodium_stubbed"),
[
@@ -200,7 +247,9 @@ def test_write_idf_component_yml_stub_yields_to_converted_library_of_same_name(
converted.get_sanitized_name.return_value = "esphome/libsodium"
converted.path = tmp_path / "pio_components" / "esphome" / "libsodium"
monkeypatch.setattr(
esp32, "generate_idf_components", lambda libraries, managed=None: [converted]
esp32,
"generate_idf_components",
lambda libraries, managed=None, provides=None: [converted],
)
esp32._write_idf_component_yml()
+3 -4
View File
@@ -846,15 +846,14 @@ def test_library_info_dot_a_linkage_malformed_raises(tmp_path: Path) -> None:
def test_bundled_library_properties_depends_warns(
tmp_path: Path, caplog: pytest.LogCaptureFixture
) -> None:
"""The library.properties depends= spelling reaches the visibility
warning too; the shared parser returns it raw."""
"""The library.properties depends= spelling is a dependency list too, so a
bundled manifest using it gets the same visibility warning."""
framework = _make_framework(tmp_path)
wire = framework / "libraries" / "Wire"
(wire / "library.properties").write_text("name=Wire\nversion=1.0\ndepends=SPI\n")
_add_library("Wire", None)
caplog.set_level("INFO")
_resolve(framework)
assert "Library Wire declares dependencies via library.properties" in caplog.text
assert "Bundled library Wire declares dependencies" in caplog.text
def test_bundled_library_extra_script_raises(tmp_path: Path) -> None:
+9 -3
View File
@@ -215,8 +215,9 @@ def test_convert_pio_libs_arduino_framework_passes_empty_managed(
path=tmp_path / "other-lib",
)
def fake_generate_idf_components(libraries, managed=None):
def fake_generate_idf_components(libraries, managed=None, provides=None):
captured["managed"] = managed
captured["provides"] = provides
return [converted]
monkeypatch.setattr(
@@ -226,6 +227,8 @@ def test_convert_pio_libs_arduino_framework_passes_empty_managed(
result = _convert_pio_libs(tmp_path / "platformio.ini", "arduino")
assert captured["managed"] == set()
# Bundled Arduino names are left to arduino-esp32, as in the real build
assert captured["provides"]("Wire") and not captured["provides"]("Adafruit BusIO")
assert result == {
"esphome/other-lib": {"override_path": str(tmp_path / "other-lib")}
}
@@ -242,7 +245,7 @@ def test_convert_pio_libs_arduino_idf_6_passes_manifest_deps(
captured: dict[str, set[str] | None] = {}
def fake_generate_idf_components(libraries, managed=None):
def fake_generate_idf_components(libraries, managed=None, provides=None):
captured["managed"] = managed
return []
@@ -265,8 +268,9 @@ def test_convert_pio_libs_espidf_framework_passes_manifest_deps(
captured: dict[str, set[str] | None] = {}
def fake_generate_idf_components(libraries, managed=None):
def fake_generate_idf_components(libraries, managed=None, provides=None):
captured["managed"] = managed
captured["provides"] = provides
return []
monkeypatch.setattr(
@@ -276,6 +280,8 @@ def test_convert_pio_libs_espidf_framework_passes_manifest_deps(
result = _convert_pio_libs(tmp_path / "platformio.ini", "espidf")
assert captured["managed"] == _esphome_manifest_deps()
# ESP-IDF bundles no Arduino libraries
assert captured["provides"] is None
assert "esphome/noise-c" in captured["managed"]
assert result == {}
+562 -122
View File
@@ -4,11 +4,13 @@ Covers the shared download/parse/resolve/dependency-walk paths in
``esphome.platformio.library`` directly (the ESP-IDF and Zephyr backends are
exercised in their own test modules)."""
from collections.abc import Collection
import json
import logging
from pathlib import Path
from unittest.mock import Mock
from platformio.package.exception import UnknownPackageError
import pytest
from esphome.core import CORE, EsphomeError, Library
@@ -261,23 +263,37 @@ def test_make_registry_client_creates_http_cache_dir(
lib._make_registry_client()
def _patch_registry_resolve(monkeypatch: pytest.MonkeyPatch) -> None:
"""Stub the registry lookup so tests never touch the network."""
monkeypatch.setattr(
lib,
"_resolve_registry_version",
lambda owner, pkgname, requirements: (
owner,
pkgname,
"1.0.0",
f"http://x/{pkgname}.tar.gz",
None,
),
)
def _patch_registry_resolve(
monkeypatch: pytest.MonkeyPatch,
*,
unknown: Collection[str] = (),
owner: str | None = None,
) -> list[tuple[str | None, str]]:
"""Stub the registry lookup so tests never touch the network; returns the
(owner, name) pairs looked up. ``unknown`` names miss; ``owner`` is what the
registry answers for an owner-less name."""
asked: list[tuple[str | None, str]] = []
def resolve(requested_owner, pkgname, requirements):
asked.append((requested_owner, pkgname))
if pkgname in unknown:
raise UnknownPackageError(pkgname)
return _resolution(requested_owner or owner, pkgname)
monkeypatch.setattr(lib, "_resolve_registry_version", resolve)
return asked
def _patch_download_with_manifests(monkeypatch, tmp_path, manifests, *, properties=()):
"""Fake ConvertedLibrary.download to materialize canned manifests on disk."""
def _resolution(owner, pkgname):
"""What the stubbed registry answers for a package."""
return owner, pkgname, "1.0.0", f"http://x/{pkgname}.tar.gz", None
def _patch_download_with_manifests(
monkeypatch, tmp_path, manifests, *, properties=(), unknown=(), owner=None
):
"""Fake ConvertedLibrary.download to materialize canned manifests on disk;
returns the registry lookups made (see _patch_registry_resolve)."""
def fake_download(self, force=False, salt="", namespace=""):
self.path = tmp_path / self.get_require_name()
@@ -288,7 +304,7 @@ def _patch_download_with_manifests(monkeypatch, tmp_path, manifests, *, properti
(self.path / "library.json").write_text(json.dumps(manifests[self.name]))
monkeypatch.setattr(ConvertedLibrary, "download", fake_download)
_patch_registry_resolve(monkeypatch)
return _patch_registry_resolve(monkeypatch, unknown=unknown, owner=owner)
def test_wave_requirement_growth_defers_the_superseded_download(tmp_path, monkeypatch):
@@ -343,18 +359,22 @@ def test_convert_libraries_parses_library_properties(tmp_path, monkeypatch):
assert emitted[0].data["version"] == "1.0"
def test_convert_libraries_skips_dependency_without_version(tmp_path, monkeypatch):
# A dependency entry lacking a version is malformed and silently skipped.
_patch_download_with_manifests(
def test_convert_libraries_fetches_dependency_without_version(tmp_path, monkeypatch):
# A dependency entry lacking a version resolves by name alone, as PlatformIO did.
asked = _patch_download_with_manifests(
monkeypatch,
tmp_path,
{"esphome/A": {"name": "A", "dependencies": [{"name": "C"}]}},
{
"esphome/A": {"name": "A", "dependencies": [{"name": "C"}]},
"C": {"name": "C"},
},
)
# No version on the top-level spec exercises the "no requirement" path too.
top = convert_libraries([Library("esphome/A", None, None)], _backend())
assert top[0].dependencies == []
assert [d.name for d in top[0].dependencies] == ["C"]
assert asked == [("esphome", "A"), (None, "C")]
def test_convert_libraries_handles_unparsable_dependency_version(tmp_path, monkeypatch):
@@ -1013,20 +1033,24 @@ def test_convert_libraries_malformed_transitive_dep_skips(
assert "Skipping dependency esphome/B: malformed manifest" in caplog.text
def test_walk_warns_for_properties_only_depends(
tmp_path, monkeypatch, caplog: pytest.LogCaptureFixture
) -> None:
"""A manifest declaring dependencies only as library.properties depends=
warns in the shared walk, so every backend reports the drop."""
_patch_download_with_manifests(
def test_walk_resolves_properties_only_depends(tmp_path, monkeypatch) -> None:
"""A library.properties depends= line is a dependency list: a bundled name
goes to the backend, the rest resolve by name."""
asked = _patch_download_with_manifests(
monkeypatch,
tmp_path,
{"esphome/A": "name=A\nversion=1.0\ndepends=Wire, SPI\n"},
{
"esphome/A": "name=A\nversion=1.0\ndepends=Wire, SPI\n",
"SPI": {"name": "SPI"},
},
properties=("esphome/A",),
)
caplog.set_level("INFO")
convert_libraries([Library("esphome/A", "1.0.0", None)], _backend())
assert "declares dependencies via library.properties" in caplog.text
backend = _backend(provides=lambda name: name == "Wire")
top = convert_libraries([Library("esphome/A", "1.0.0", None)], backend)
assert [d.name for d in top[0].dependencies] == ["SPI"]
assert "depends" not in top[0].data
assert backend.provided_requests == {"Wire"}
assert (None, "SPI") in asked
def test_walk_warns_for_nonplatform_invalid_library(
@@ -1093,12 +1117,12 @@ def test_source_kind_map_shape() -> None:
assert SOURCE_KIND_FOR_SUFFIX[".C++"] == "cxx"
def test_versionless_platform_filtered_dependency_stays_quiet(
tmp_path, monkeypatch, caplog: pytest.LogCaptureFixture
def test_versionless_platform_filtered_dependency_is_not_fetched(
tmp_path, monkeypatch
) -> None:
"""A version-less dependency the platform filter excludes is
deliberately absent, not a drop to warn about."""
_patch_download_with_manifests(
"""A version-less dependency the platform filter excludes is deliberately
absent, not fetched by name."""
asked = _patch_download_with_manifests(
monkeypatch,
tmp_path,
{
@@ -1108,59 +1132,52 @@ def test_versionless_platform_filtered_dependency_stays_quiet(
}
},
)
convert_libraries([Library("esphome/A", None, None)], _backend())
assert "has no version to resolve" not in caplog.text
top = convert_libraries([Library("esphome/A", None, None)], _backend())
assert top[0].dependencies == []
assert (None, "Hash") not in asked
def test_versionless_ignored_dependency_stays_quiet(
tmp_path, monkeypatch, caplog: pytest.LogCaptureFixture
) -> None:
def test_versionless_ignored_dependency_is_not_fetched(tmp_path, monkeypatch) -> None:
"""A lib_ignore'd version-less dependency is deliberately excluded, not
a drop; no reconciliation warning."""
_patch_download_with_manifests(
fetched by name."""
asked = _patch_download_with_manifests(
monkeypatch,
tmp_path,
{"esphome/A": {"name": "A", "dependencies": [{"name": "Hash"}]}},
)
CORE.platformio_options = {"lib_ignore": ["Hash"]}
convert_libraries([Library("esphome/A", None, None)], _backend())
assert "has no version to resolve" not in caplog.text
top = convert_libraries([Library("esphome/A", None, None)], _backend())
assert top[0].dependencies == []
assert (None, "Hash") not in asked
def test_versionless_dependency_without_provider_warns(
def test_versionless_dependency_registry_miss_is_skipped(
tmp_path, monkeypatch, caplog: pytest.LogCaptureFixture
) -> None:
"""A backend whose tree could supply the name warns on the drop; one
without provides() can never act on it, so it stays at debug."""
"""A name-only dependency the registry does not know is skipped with one
warning naming the requester, and the requester still converts; a listed
library the registry does not know still fails the build."""
_patch_download_with_manifests(
monkeypatch,
tmp_path,
{
"esphome/A": {
"name": "A",
# The duplicate entry warns once (reconciliation dedup)
# The duplicate entry is one node, so it warns once
"dependencies": [{"name": "Hash"}, {"name": "Hash"}],
}
},
unknown={"Hash"},
)
convert_libraries(
[Library("esphome/A", None, None)], _backend(provides=lambda name: False)
)
top = convert_libraries([Library("esphome/A", None, None)], _backend())
assert top[0].dependencies == []
assert (
caplog.text.count(
"Hash of esphome/A has no version to resolve and nothing provides it"
)
caplog.text.count("Dependency Hash of esphome/A was not resolved; skipping")
== 1
)
caplog.clear()
with caplog.at_level(logging.DEBUG):
convert_libraries([Library("esphome/A", None, None)], _backend())
records = [
r
for r in caplog.records
if "has no version to resolve and nothing provides it" in r.message
]
assert records and all(r.levelno == logging.DEBUG for r in records)
with pytest.raises(UnknownPackageError):
convert_libraries([Library("Hash", None, None)], _backend())
def test_url_version_dependency_is_not_substituted_by_provides(
@@ -1191,102 +1208,525 @@ def test_url_version_dependency_is_not_substituted_by_provides(
assert any("o/hash" in n.lower() for n in emitted)
def test_versionless_owner_qualified_dependency_warns_despite_provides(
tmp_path, monkeypatch, caplog: pytest.LogCaptureFixture
def test_versionless_owner_qualified_dependency_is_fetched_despite_provides(
tmp_path, monkeypatch
) -> None:
"""An owner-qualified version-less dependency is not satisfied by
provides(); it must still warn."""
_patch_download_with_manifests(
"""An owner-qualified version-less dependency names a registry package,
so provides() does not stand in for it."""
asked = _patch_download_with_manifests(
monkeypatch,
tmp_path,
{
"esphome/A": {
"name": "A",
"dependencies": [{"name": "Wire", "owner": "Foo"}],
}
},
"Foo/Wire": {"name": "Wire"},
},
)
convert_libraries(
[Library("esphome/A", None, None)],
_backend(provides=lambda name: name == "Wire"),
)
assert "Wire of esphome/A has no version to resolve" in caplog.text
def test_versionless_provided_dependency_stays_quiet(
tmp_path, monkeypatch, caplog: pytest.LogCaptureFixture
) -> None:
"""An owner-less version-less dependency the backend provides is added
by the backend after emit; no reconciliation warning."""
_patch_download_with_manifests(
monkeypatch,
tmp_path,
{"esphome/A": {"name": "A", "dependencies": [{"name": "Wire"}]}},
)
convert_libraries(
[Library("esphome/A", None, None)],
_backend(provides=lambda name: name == "Wire"),
)
assert "has no version to resolve" not in caplog.text
def test_versionless_dependency_requested_top_level_stays_quiet(
tmp_path, monkeypatch, caplog: pytest.LogCaptureFixture
) -> None:
"""A version-less dependency the config also requests top-level is in
the build; no drop warning even without a provides backend."""
_patch_download_with_manifests(
monkeypatch,
tmp_path,
{
"esphome/A": {"name": "A", "dependencies": [{"name": "Hash"}]},
"Hash": {"name": "Hash"},
},
)
convert_libraries(
[Library("esphome/A", None, None), Library("Hash", None, None)],
_backend(),
)
assert "has no version to resolve" not in caplog.text
backend = _backend(provides=lambda name: name == "Wire")
top = convert_libraries([Library("esphome/A", None, None)], backend)
assert [d.name for d in top[0].dependencies] == ["Foo/Wire"]
assert ("Foo", "Wire") in asked
assert not backend.provided_requests
def test_versionless_url_ish_dependency_name_warns_cleanly(
tmp_path, monkeypatch, caplog: pytest.LogCaptureFixture
) -> None:
"""A malformed URL-ish dependency name falls to the drop warning, never
"""A malformed URL-ish dependency name is skipped with a warning, never
a RuntimeError out of the key parser."""
_patch_download_with_manifests(
monkeypatch,
tmp_path,
{"esphome/A": {"name": "A", "dependencies": [{"name": "file://"}]}},
)
convert_libraries(
[Library("esphome/A", None, None)], _backend(provides=lambda name: False)
top = convert_libraries([Library("esphome/A", None, None)], _backend())
assert top[0].dependencies == []
assert "Skipping dependency file:// of esphome/A" in caplog.text
def test_versionless_dependency_matching_resolved_manifest_name_is_wired(
tmp_path, monkeypatch, caplog: pytest.LogCaptureFixture
) -> None:
"""A bare name no package name matches is satisfied by a listed component's
manifest name (a fork published under another package name)."""
asked = _patch_download_with_manifests(
monkeypatch,
tmp_path,
{
"esphome/A": {"name": "A", "dependencies": [{"name": "B"}]},
"esphome/B-fork": {"name": "B"},
},
)
with caplog.at_level(logging.INFO):
top = convert_libraries(
[Library("esphome/A", None, None), Library("esphome/B-fork", None, None)],
_backend(),
)
assert [d.name for d in top[0].dependencies] == ["esphome/B-fork"]
assert asked == [("esphome", "A"), ("esphome", "B-fork")]
assert (
"file:// of esphome/A has no version to resolve and nothing provides it"
"Dependency B of esphome/A satisfied by esphome/B-fork through its manifest name"
in caplog.text
)
def test_versionless_dependency_matching_resolved_manifest_name_stays_quiet(
def test_dependency_naming_the_requester_itself_is_ignored(
tmp_path, monkeypatch
) -> None:
"""A manifest that lists its own name as a dependency neither fetches a
second copy of itself nor gains a self edge."""
asked = _patch_download_with_manifests(
monkeypatch,
tmp_path,
{
"esphome/A": "name=A\ndepends=a, esphome/A\n",
# No manifest name: the requested and resolved package names still
# identify it, with or without the owner
"esphome/C": {
"dependencies": [{"name": "C"}, {"name": "C", "owner": "esphome"}]
},
},
properties=("esphome/A",),
)
top = convert_libraries(
[Library("esphome/A", "1.0.0", None), Library("esphome/C", "1.0.0", None)],
_backend(),
)
assert [c.dependencies for c in top] == [[], []]
assert asked == [("esphome", "A"), ("esphome", "C")]
def test_versionless_dependency_settles_after_the_graph(
tmp_path, monkeypatch, caplog: pytest.LogCaptureFixture
) -> None:
"""A bare name satisfied by an owner-qualified component's manifest
name is not a drop."""
"""The issue 21308 shape: a library.properties depends= naming a library the
config lists owner-qualified is wired to that one package, case-insensitively
and whatever constraint it carries, without a second node for the bare name
and ahead of a same-named bundled copy, as on the Arduino backend."""
asked = _patch_download_with_manifests(
monkeypatch,
tmp_path,
{
"adafruit/Adafruit GFX Library": (
"name=Adafruit GFX Library\ndepends=adafruit busio (>=1.0)\n"
),
"adafruit/Adafruit BusIO": {"name": "Adafruit BusIO"},
},
properties=("adafruit/Adafruit GFX Library",),
)
backend = _backend(provides=lambda name: name == "adafruit busio")
top = convert_libraries(
[
Library("adafruit/Adafruit GFX Library", "^1.13.0", None),
Library("adafruit/Adafruit BusIO", "^1.17.4", None),
],
backend,
)
assert [d.name for d in top[0].dependencies] == ["adafruit/Adafruit BusIO"]
assert len(asked) == 2
assert not backend.provided_requests
assert "multiple names" not in caplog.text
def test_by_name_node_folds_into_a_later_pin_of_the_same_package(
tmp_path, monkeypatch, caplog: pytest.LogCaptureFixture
) -> None:
"""A library fetched by name alone may pull in a manifest that pins the same
package owner-qualified; the by-name node folds into that one instead of
converting the package twice."""
# Manifests are keyed by the resolved name, so both B nodes read the same one:
# B pins C, and C pins owner/B, which arrives after B was fetched by name
manifests = {
"esphome/A": {"name": "A", "dependencies": [{"name": "B"}]},
"esphome/B": {"name": "B", "dependencies": {"esphome/C": "1.0.0"}},
"esphome/C": {"name": "C", "dependencies": {"esphome/B": "1.0.0"}},
}
_patch_download_with_manifests(monkeypatch, tmp_path, manifests, owner="esphome")
emitted: list[str] = []
top = convert_libraries(
[Library("esphome/A", None, None)],
_backend(emit=lambda c: emitted.append(c.name)),
)
assert [d.name for d in top[0].dependencies] == ["esphome/B"]
assert emitted.count("esphome/B") == 1
assert "multiple names" not in caplog.text
def test_two_spellings_of_a_name_in_one_round_share_a_node(
tmp_path, monkeypatch, caplog: pytest.LogCaptureFixture
) -> None:
"""Requesters spelling one unlisted name differently get a single by-name
fetch and the same node."""
asked = _patch_download_with_manifests(
monkeypatch,
tmp_path,
{
"esphome/A": {"name": "A", "dependencies": [{"name": "Foo"}]},
"esphome/B": {"name": "B", "dependencies": [{"name": "foo"}]},
"Foo": {"name": "Foo"},
},
)
top = convert_libraries(
[Library("esphome/A", None, None), Library("esphome/B", None, None)],
_backend(),
)
assert [d.name for d in top[0].dependencies] == ["Foo"]
assert [d.name for d in top[1].dependencies] == ["Foo"]
assert asked.count((None, "Foo")) + asked.count((None, "foo")) == 1
assert "multiple names" not in caplog.text
def test_parse_properties_depends_reads_names_and_drops_constraints(
caplog: pytest.LogCaptureFixture,
) -> None:
"""Arduino constraints are not PlatformIO syntax, so only the names survive;
an unclosed parenthesis loses nothing more than the constraint."""
with caplog.at_level(logging.DEBUG):
entries = lib.parse_properties_depends(
"Adafruit BusIO, Foo (>=1.2), ,Bar( <2 ), Baz (=1.0, Qux ()"
)
levels = {
r.message.removeprefix("Ignoring version constraint on depends="): r.levelno
for r in caplog.records
if "Ignoring version constraint" in r.message
}
# A lower bound is met by the current version; a cap or pin may not be
assert levels == {
"Foo (>=1.2)": logging.DEBUG,
"Bar( <2 )": logging.INFO,
"Baz (=1.0": logging.INFO,
"Qux ()": logging.INFO,
}
assert entries == [
{"name": "Adafruit BusIO"},
{"name": "Foo"},
{"name": "Bar"},
{"name": "Baz"},
{"name": "Qux"},
]
assert "Ignoring version constraint on depends=Foo (>=1.2)" in caplog.text
def test_by_name_node_pinned_later_reports_a_requirement_conflict(
tmp_path, monkeypatch
) -> None:
"""A node fetched by name that a later manifest pins under the same spelling
is resolved again against the pin; a conflict is an error, not a miss."""
_patch_download_with_manifests(
monkeypatch,
tmp_path,
{
"esphome/A": {"name": "A", "dependencies": [{"name": "B"}]},
"esphome/B": {"name": "B"},
"B": {"name": "B", "dependencies": {"C": "1.0.0"}},
"C": {"name": "C", "dependencies": {"B": "^2.0"}},
},
)
def resolve(owner, pkgname, requirements):
if pkgname == "B" and requirements:
raise RuntimeError(f"No version of {pkgname} satisfies {requirements}")
return _resolution(owner, pkgname)
monkeypatch.setattr(lib, "_resolve_registry_version", resolve)
with pytest.raises(RuntimeError, match="No version of B satisfies"):
convert_libraries([Library("esphome/A", None, None)], _backend())
def test_by_name_fetch_yields_to_a_fork_pulled_in_later(
tmp_path, monkeypatch, caplog: pytest.LogCaptureFixture
) -> None:
"""A name fetched from the registry yields to a library another round
pins explicitly under the same manifest name, so only the fork is built."""
_patch_download_with_manifests(
monkeypatch,
tmp_path,
{
"esphome/A": {"name": "A", "dependencies": [{"name": "B"}]},
"esphome/X": {"name": "X", "dependencies": [{"name": "C"}]},
# The upstream B pulls in D, which nothing else needs; D asks for a
# name the registry does not know
"B": {"name": "B", "dependencies": {"esphome/D": "1.0.0"}},
"esphome/D": {"name": "D", "dependencies": [{"name": "Zed"}]},
"C": {"name": "C", "dependencies": {"custom/B-fork": "1.0.0"}},
"custom/B-fork": {"name": "B"},
},
unknown={"Zed"},
)
emitted: list[str] = []
top = convert_libraries(
[Library("esphome/A", None, None), Library("esphome/X", None, None)],
_backend(emit=lambda c: emitted.append(c.name)),
)
assert [d.name for d in top[0].dependencies] == ["custom/B-fork"]
assert sorted(emitted) == ["C", "custom/B-fork", "esphome/A", "esphome/X"]
# The miss belongs to a library that is not built, so it is not reported
assert "Zed" not in caplog.text
def test_git_source_replacing_a_registry_key_in_the_same_wave(
tmp_path, monkeypatch
) -> None:
"""A manifest in one wave naming a key of the same wave as a git source
defers that key to the next wave instead of tripping over its stale record."""
_patch_download_with_manifests(
monkeypatch,
tmp_path,
{
"o/A": {"name": "A", "dependencies": {"B": "https://github.com/o/B.git"}},
"o/B": {"name": "B"},
},
)
emitted: dict[str, ConvertedLibrary] = {}
top = convert_libraries(
[Library("o/A", "1.0.0", None), Library("o/B", "1.0.0", None)],
_backend(emit=lambda c: emitted.__setitem__(c.name, c)),
)
assert [c.name for c in top] == ["o/A", "o/B"]
assert isinstance(emitted["o/B"].source, GitSource)
assert [d.name for d in top[0].dependencies] == ["o/B"]
def test_owner_qualified_name_matches_the_resolved_owner(tmp_path, monkeypatch) -> None:
"""A version-less ``o/B`` is satisfied by a bare ``B`` pin the registry
resolved to owner ``o``, so the package is not fetched a second time."""
asked = _patch_download_with_manifests(
monkeypatch,
tmp_path,
{
"esphome/A": {
"name": "A",
"dependencies": [
{"name": "B", "version": "1.0.0"},
{"name": "B", "owner": "o"},
],
},
"o/B": {"name": "B"},
},
owner="o",
)
top = convert_libraries([Library("esphome/A", None, None)], _backend())
assert [d.name for d in top[0].dependencies] == ["o/B"]
assert asked.count((None, "B")) == 1
assert ("o", "B") not in asked
def test_two_spellings_fetched_by_name_fold_into_one_node(
tmp_path, monkeypatch, caplog: pytest.LogCaptureFixture
) -> None:
"""Version-less ``B`` and ``o/B`` from two requesters resolve to one
package; only one copy is built and both requesters depend on it."""
_patch_download_with_manifests(
monkeypatch,
tmp_path,
{
"esphome/A": {"name": "A", "dependencies": [{"name": "B"}]},
"esphome/X": {"name": "X", "dependencies": [{"name": "B", "owner": "o"}]},
"o/B": {"name": "B"},
},
owner="o",
)
emitted: list[str] = []
top = convert_libraries(
[Library("esphome/A", None, None), Library("esphome/X", None, None)],
_backend(emit=lambda c: emitted.append(c.name)),
)
assert emitted.count("o/B") == 1
kept = top[0].dependencies[0]
assert top[1].dependencies == [kept]
assert "multiple names" not in caplog.text
def test_registry_miss_resolves_again_as_a_git_source(tmp_path, monkeypatch) -> None:
"""A key the registry missed is resolved again when a later manifest names
the same key as a git source."""
_patch_download_with_manifests(
monkeypatch,
tmp_path,
{
"esphome/A": {"name": "A", "dependencies": [{"name": "B", "owner": "o"}]},
"esphome/X": {"name": "X", "dependencies": [{"name": "C"}]},
"C": {"name": "C", "dependencies": {"B": "https://github.com/o/B.git"}},
"o/B": {"name": "B"},
},
unknown={"B"},
)
emitted: dict[str, ConvertedLibrary] = {}
top = convert_libraries(
[Library("esphome/A", None, None), Library("esphome/X", None, None)],
_backend(emit=lambda c: emitted.__setitem__(c.name, c)),
)
assert [d.name for d in top[0].dependencies] == ["o/B"]
assert isinstance(emitted["o/B"].source, GitSource)
assert [d.name for d in emitted["C"].dependencies] == ["o/B"]
def test_normalize_dependencies_splits_an_embedded_owner() -> None:
"""``owner/Name`` as one string, in a list or a dict entry, normalizes like
the dict form; a URL or a deeper path is left alone."""
assert lib.normalize_dependencies(
["o/B", {"name": "o/C", "version": "1.0"}, "x://y", "a/b/c", "../escape"],
"libx",
) == [
{"name": "B", "owner": "o"},
{"name": "C", "owner": "o", "version": "1.0"},
{"name": "x://y"},
{"name": "a/b/c"},
{"name": "../escape"},
]
assert lib.normalize_dependencies("o/B, D", "libx") == [
{"name": "B", "owner": "o"},
{"name": "D"},
]
def test_owner_qualified_string_dependency_is_fetched_and_wired(
tmp_path, monkeypatch
) -> None:
"""``"dependencies": ["o/B"]`` fetches ``o/B`` and wires the requester to it."""
asked = _patch_download_with_manifests(
monkeypatch,
tmp_path,
{
"esphome/A": {"name": "A", "dependencies": ["o/B"]},
"o/B": {"name": "B"},
},
)
top = convert_libraries([Library("esphome/A", None, None)], _backend())
assert [d.name for d in top[0].dependencies] == ["o/B"]
assert asked == [("esphome", "A"), ("o", "B")]
def test_empty_version_is_settled_like_a_missing_one(
tmp_path, monkeypatch, caplog: pytest.LogCaptureFixture
) -> None:
"""``{"Foo": ""}`` carries no version either, so it is matched, fetched by
name or reported like a version-less entry rather than failing or vanishing."""
asked = _patch_download_with_manifests(
monkeypatch,
tmp_path,
{
"esphome/A": {"name": "A", "dependencies": {"Foo": "", "Bar": ""}},
"Bar": {"name": "Bar"},
},
unknown={"Foo"},
)
top = convert_libraries([Library("esphome/A", None, None)], _backend())
assert [d.name for d in top[0].dependencies] == ["Bar"]
assert (None, "Foo") in asked
assert "Dependency Foo of esphome/A was not resolved; skipping" in caplog.text
def test_bundled_name_is_satisfied_by_a_replacement_pulled_in_later(
tmp_path, monkeypatch
) -> None:
"""A name the framework bundles is still wired to an explicit replacement
of the same manifest name that a later round pulls in, as it would be had
the replacement been listed up front."""
_patch_download_with_manifests(
monkeypatch,
tmp_path,
{
"esphome/A": {"name": "A", "dependencies": [{"name": "Ethernet"}]},
"esphome/X": {"name": "X", "dependencies": [{"name": "C"}]},
"C": {"name": "C", "dependencies": {"vendor/Ethernet": "1.0.0"}},
"vendor/Ethernet": {"name": "Ethernet"},
},
)
backend = _backend(provides=lambda name: name == "Ethernet")
top = convert_libraries(
[Library("esphome/A", None, None), Library("esphome/X", None, None)],
backend,
)
assert [d.name for d in top[0].dependencies] == ["vendor/Ethernet"]
def test_missed_name_is_satisfied_by_a_library_pulled_in_later(
tmp_path, monkeypatch, caplog: pytest.LogCaptureFixture
) -> None:
"""A bare name the registry does not know is wired to a library a later
round pulls in under another package name but the same manifest name."""
asked = _patch_download_with_manifests(
monkeypatch,
tmp_path,
{
"esphome/A": {"name": "A", "dependencies": [{"name": "B"}]},
"esphome/X": {"name": "X", "dependencies": [{"name": "C"}]},
# Fetched by name in the same round as the miss; pins the fork
"C": {"name": "C", "dependencies": {"owner/B-fork": "1.0.0"}},
"owner/B-fork": {"name": "B"},
},
unknown={"B"},
)
with caplog.at_level(logging.INFO):
top = convert_libraries(
[Library("esphome/A", None, None), Library("esphome/X", None, None)],
_backend(),
)
assert [d.name for d in top[0].dependencies] == ["owner/B-fork"]
assert asked.count((None, "B")) == 1
assert (
"Dependency B of esphome/A satisfied by owner/B-fork through its manifest name"
in caplog.text
)
assert "skipping" not in caplog.text
def test_by_name_request_does_not_make_an_existing_node_optional(
tmp_path, monkeypatch, caplog: pytest.LogCaptureFixture
) -> None:
"""A version-less request for a node the walk already created (here one
skipped for its platform) neither re-fetches it nor turns a later miss
into a warning."""
asked = _patch_download_with_manifests(
monkeypatch,
tmp_path,
{
"esphome/A": {
"name": "A",
"dependencies": [{"name": "X", "version": "1.0"}, {"name": "X"}],
},
"X": {"name": "X", "platforms": "espressif8266"},
},
)
top = convert_libraries([Library("esphome/A", None, None)], _backend())
assert top[0].dependencies == []
assert asked == [("esphome", "A"), (None, "X")]
assert "not in the registry" not in caplog.text
def test_provided_dependency_asks_the_backend_once_per_name(
tmp_path, monkeypatch
) -> None:
"""A bundled name is recorded for the backend and never fetched; two
requesters cost one provides() call, so a backend warning there is not
repeated. A pinned request on the same name shares the memo."""
asked = _patch_download_with_manifests(
monkeypatch,
tmp_path,
{
"esphome/A": {"name": "A", "dependencies": [{"name": "Wire"}]},
"esphome/B": {"name": "B", "dependencies": {"Wire": "*"}},
},
)
calls: list[str] = []
def provides(name: str) -> bool:
calls.append(name)
return name == "Wire"
backend = _backend(provides=provides)
convert_libraries(
[Library("esphome/A", None, None), Library("esphome/B", None, None)],
_backend(),
backend,
)
assert "has no version to resolve" not in caplog.text
assert calls == ["Wire"]
assert backend.provided_requests == {"Wire"}
assert (None, "Wire") not in asked
def test_convert_libraries_symlink_url_resolves_as_local(setup_core: Path) -> None: