Commit Graph
43 Commits
Author SHA1 Message Date
J. Nick Koston 0e9a08a3b5 Merge branch 'esp8266-native-toolchain-plumbing' into esp8266-native-build-infra 2026-08-23 18:45:14 -05:00
J. Nick Koston 69d997521f Guard the bar writes, narrow the cache probe, make a total prefetch failure loud 2026-08-23 18:45:02 -05:00
J. Nick Koston 55323596a9 Resolve registry metadata once, canonicalize the ccache knob, disable on an empty value
registry_download memoizes per process (failures uncached), so the
prefetch and the install share one metadata round-trip per package.
The idf ccache env always hands idf.py exactly one canonical spelling:
the no-ccache path exports 0 so an unparsable inherited value or a
probe-rejected shim cannot enable it, the enabled path exports 1
whatever accepted spelling arrived, and forced-on still runs the
resolver for its no-binary warning. An empty env value keeps its
historical disable meaning instead of falling to the default policy.
2026-08-23 14:17:29 -05:00
J. Nick Koston 485725d734 Merge branch 'esp8266-native-toolchain-plumbing' into esp8266-native-build-infra 2026-08-23 13:09:54 -05:00
J. Nick Koston 9d761102e5 Size the prefetch from the registry, fold the batch API, guard the prefetch
The registry response already carries each package file's size, so the
combined bar is sized from URLSource instead of a HEAD sweep; the probe,
its all-or-nothing fallback, and its duplicate Content-Length parsing are
gone, and archives without a reported size simply stay on the sequential
path. run_batch_downloads now takes (name, size, fetch) jobs and builds
the (now private) progress bar itself, collecting failures from future
results instead of a shared list; both call sites use functools.partial
instead of lambda factories. The prefetch touches only URLSource, so the
progress parameter comes off Source/Git/Local/ConvertedLibrary, and the
wave prefetch is wrapped best-effort like its ESP-IDF twin, which also
dedupes by dest only after the verifiability check. ProgressBar.done()
itself now skips the newline when no frame was drawn or the 100% frame
already ended the line, collapsing the batch wrapper's guard and fixing
the stray blank line after OTA uploads.
2026-08-23 13:08:32 -05:00
J. Nick Koston 187df36971 Merge branch 'esp8266-native-toolchain-plumbing' into esp8266-native-build-infra 2026-08-23 12:05:11 -05:00
J. Nick Koston ea3ff0e372 Abort in-flight batch downloads on Ctrl-C, make the deferral test hermetic
cancel_futures only drops queued jobs, and the non-daemon workers are
joined at interpreter exit, so a Ctrl-C during the ESP-IDF toolchain
fetch waited out every in-flight archive. The tracker each job reports
through now checks a cancellation flag set when the wait raises, so
downloads abort at their next progress tick and the resumable .part
files keep the bytes already fetched.

The wave-deferral test stubs _content_lengths instead of issuing real
HEAD requests that passed only by failing, the dead hasattr branch in
the HEAD-probe test is gone, and the .C suffix comment names the SCons
divergence instead of claiming parity.
2026-08-23 12:04:49 -05:00
J. Nick Koston a1d0713a54 Merge branch 'esp8266-native-toolchain-plumbing' into esp8266-native-build-infra 2026-08-22 22:58:48 -05:00
J. Nick Koston 35bb8388a2 Defer superseded wave downloads, dedupe tool dests, validate captured buckets, right-size the log levels 2026-08-22 22:58:44 -05:00
J. Nick Koston a1e6370e8b [core] Harden the shared library converter and download registry archives in parallel 2026-08-22 21:59:19 -05:00
J. Nick Koston 5c30acfe5b Merge branch 'esp8266-native-toolchain-plumbing' into esp8266-native-build-infra 2026-08-22 21:12:39 -05:00
J. Nick Koston d6d80ee323 Consolidate the batch-download scaffold, share the idedata error set, dedupe compdb tokenizing 2026-08-22 21:12:20 -05:00
J. Nick Koston bf0a4ea40f Merge branch 'esp8266-native-toolchain-plumbing' into esp8266-native-build-infra 2026-08-22 20:01:31 -05:00
J. Nick Koston 2acba9b364 Materialize the tracker mock before the prefetch threads race its creation 2026-08-22 20:01:13 -05:00
J. Nick Koston 3697401bc7 Serialize prefetch downloads on the install lock; export the canonical ccache off spelling 2026-08-22 19:32:10 -05:00
J. Nick Koston e63d995d1f Merge branch 'esp8266-native-toolchain-plumbing' into esp8266-native-build-infra 2026-08-22 17:10:23 -05:00
J. Nick Koston bff8bea83e Degrade garbage Content-Length probes; never prefetch an unverifiable tool archive 2026-08-22 17:10:19 -05:00
J. Nick Koston 4cbfdd3200 Merge branch 'esp8266-native-toolchain-plumbing' into esp8266-native-build-infra
# Conflicts:
#	tests/unit_tests/test_framework_helpers.py
2026-08-22 15:21:01 -05:00
J. Nick Koston 90b29e231d Trim comment essays and hoist function-local test imports 2026-08-22 11:38:46 -05:00
J. Nick Koston 45cbe0aa4b Parse IDF_CCACHE_ENABLE strictly through the shared helper, fix the ccache docstrings
parse_enable_env carries the 1/true/yes/on and 0/false/no/off table for
both knobs, so IDF_CCACHE_ENABLE=off disables instead of reading as
truthy and suppressing the shared opt-out. The ccache module docstring
and the ESP-IDF _ccache_env docstring now describe the precedence this
PR actually ships, and the probe test duplicated by the helper move is
dropped from the PlatformIO toolchain tests.
2026-08-21 13:12:57 -05:00
J. Nick Koston 0f7cd3fada Single-quote POSIX shell tokens, share the tool probe, harden registry checks
shell_token now picks the quoting style per platform: single quotes on
POSIX (sh expands nothing inside them, so backslash runs, $VAR, and
backticks reach the compiler exactly as lexed, matching SCons's
no-shell spawn) and the CreateProcess argv rule on Windows. A test
round-trips every case through a real /bin/sh.

The ninja and ccache runnability probes collapse into one
tool_version_runs helper in framework_helpers. The registry names a
non-dict top-level payload like the inner guards, and the expect layout
check also runs on marker hits so a marked install that later lost
files fails by name. The ESP-IDF ccache gate defers to
resolve_ccache_path so ESPHOME_CCACHE_ENABLE=0 disables ccache there
too, with IDF_CCACHE_ENABLE still taking precedence.
2026-08-21 12:25:03 -05:00
J. Nick Koston 0fa081cd80 Name malformed registry elements, parse ESPHOME_CCACHE_ENABLE strictly, require a package layout check
Registry version and file entries that are not dicts now raise the same
Unexpected-response error as the other shape guards instead of an
AttributeError traceback. install_package rejects an empty expect so
every install is layout-validated before the marker is written.
ESPHOME_CCACHE_ENABLE values outside 1/true/yes/on and 0/false/no/off
warn and are treated as unset instead of bool(str) flipping them to
enabled. The strip_win_long_path_prefix test moved next to the function
in test_framework_helpers, and a stale comment naming its old module is
fixed.
2026-08-21 09:58:16 -05:00
J. Nick Koston d4861d89b0 Add the shared registry, ninja, ccache, and tools-cache infrastructure for native toolchains 2026-08-20 16:03:38 -05:00
J. Nick Koston 12238eec3b Log prefetch failures after the bar, assert on the shared progress instance 2026-08-19 13:57:15 -05:00
J. Nick Koston fe53ff6b6b Finish the batch bar, cancel queued archives on Ctrl-C, report known sizes 2026-08-19 13:28:42 -05:00
J. Nick Koston d0f38d7644 [espidf] Download tool archives in parallel during the install prefetch 2026-08-19 12:03:39 -05:00
J. Nick Koston f3a5a9fbd5 [core] Retry transient git network failures with backoff (#18242) 2026-08-11 08:41:55 -05:00
J. Nick Koston c0aa121c3d [espidf] Install only the toolchains for the variants being built (#17688) 2026-07-27 08:39:03 -10:00
J. Nick Koston 43526a815e [git] Fix submodule update failure when cloning libraries on the esp-idf toolchain (#17862) 2026-07-27 19:42:21 +12:00
J. Nick Kostonandcopilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> 9dd7dcb06d [espidf] Also skip installing gdb and ULP toolchains ESPHome never runs (#17687)
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
2026-07-25 20:07:14 -10:00
J. Nick Koston 271c1b409a [espidf] Resume interrupted toolchain downloads instead of restarting (#17706) 2026-07-21 08:12:17 +12:00
J. Nick Koston 5fa93513f2 [espidf] Make openocd-esp32 optional so its libusb check cannot break installs (#17686) 2026-07-20 17:05:50 +12:00
Jonathan Swoboda 70421fb14b [espidf] Prune tool download cache after install to shrink the ESP-IDF cache (#17661) 2026-07-17 22:49:50 -04:00
Jonathan Swoboda f918c299b1 [espidf] Suggest installing missing system libraries when the tools install fails (#17619) 2026-07-17 19:46:45 -04:00
Jonathan Swoboda c0e78a5574 [core] Improve framework mirror selection, download errors, and version parsing (#17615) 2026-07-16 15:22:19 -04:00
Jonathan Swoboda 648f5e1b06 [nrf52] Install native sdk-nrf into a machine-global cache dir (#17353) 2026-07-02 15:42:59 -04:00
Jonathan Swoboda b79cbcbde7 [espidf] Install native ESP-IDF into a machine-global cache dir (#17306) 2026-06-30 14:56:18 -04:00
Jonathan Swoboda 91e515ca7c [esp32] Accept '#' as ESP-IDF source ref separator (#17193) 2026-06-24 20:40:28 -04:00
Jonathan Swoboda 1d32b6c9e0 [espidf] Enable ccache by default for ESP-IDF builds (#17163) 2026-06-23 16:41:13 -04:00
Jonathan Swoboda 1ace836744 [espidf] Don't fail framework check on broken unrelated PATH tools (#17053) 2026-06-22 18:41:21 -04:00
Jonathan Swoboda 6809af3de0 [espidf] Warn when the install path is too long for Windows MAX_PATH (#16896) 2026-06-10 13:12:28 +12:00
5faed9d5f5 [nrf52] native build - download toolchain and sdk in venv (#16388)
Co-authored-by: Jonathan Swoboda <154711427+swoboda1337@users.noreply.github.com>
Co-authored-by: Jonathan Swoboda <swoboda1337@users.noreply.github.com>
2026-06-09 07:04:51 -04:00
Jonathan Swoboda fc0a4e2201 [espidf] Support github:// and https://github.com/.../.git framework sources (#16639) 2026-05-25 17:07:35 -04:00