Two small simplifications surfaced by re-reading the diff with a 'is
this DRY?' lens:
- Extract _missing_prebuilt_flash_tool(): the
_ensure_platform_packages_for_prebuilt_upload helper had two
near-identical 'if platform check; if finder() is not None return 0;
tool_label = ...' blocks. Lift the platform->(tool_name) selection
into a tiny helper so the early-return is one line and the install
body is no longer interleaved with the platform dispatch.
- Collapse the two near-duplicate 'firmware not found' return points in
upload_using_picotool. Both branches just selected an error message
and returned 1; reorder so the message selection happens first and
there's a single 'return 1'.
No behavior change; pure refactor. 587 unit tests pass.
Issue: esphome/device-builder#572
- CORE.firmware_bin priority is now platform-aware: RP2040 + libretiny
prefer firmware.uf2 over firmware.bin (picotool / ltchiptool need the
UF2 header for address/family info); ESP* prefer firmware.bin. The
prior code returned .bin first for all platforms, which would have
silently flashed the wrong artifact on RP2040 if a hand-staged dir
shipped both files. New tests guard both directions.
- _rp2040_serial_reset_to_bootsel: check picotool exists *before*
triggering the 1200bps touch. If picotool is missing, the touch
would have left the device stranded in BOOTSEL with nothing able to
flash it; with this order the device stays on the old firmware and
can be retried.
- upload_using_ltchiptool: error message now mentions both firmware.uf2
and firmware.bin since CORE.firmware_bin resolves either.
- prepare_platform_for_upload: return type tightened to int (capture_stdout
is hardcoded False; assert the run helper returns int so a future caller
that flips capture_stdout fails loudly instead of silently treating a
string as success). Caller in __main__.py is now a one-liner.
- _load_idedata: narrative comment in the prebuilt branch shortened.
_resolve_prebuilt_idedata_paths docstring now notes the
POSIX-absolute-on-Windows quirk ("/foo/bar" is rooted but not absolute
on win32; hand-staged dirs need OS-appropriate absolute paths).
- New defensive-coverage tests: _resolve_prebuilt_idedata_paths with
missing prog_path, no extra section, empty flash_images list.
Issue: esphome/device-builder#572
Issue: esphome/device-builder#570
The 'absolute paths pass through unchanged' test for _load_idedata
hardcoded a POSIX-style absolute path ('/somewhere/else/bootloader.bin').
On Windows, Path.is_absolute() returns False for that shape -- absolute
paths there require a drive letter -- so _resolve_prebuilt_idedata_paths
classified it as relative and prepended CORE.prebuilt_dir, breaking the
test's assertion.
Replace the hardcoded POSIX path with a tmp_path-rooted one so it's
platform-absolute on every runner. No production code change; this is a
test-only fix surfaced by the windows-latest pytest matrix on PR #16348.
Note: the macOS 3.14 failure on the same CI run is a flaky timing test
(tests/dashboard/test_web_server.py::test_dashboard_subscriber_entries_update_interval,
50ms sleep expecting 2+ iterations at 10ms) unrelated to this PR.
Issue: esphome/device-builder#572
- get_ltchiptool_path: use 'Scripts/' on Windows, 'bin/' elsewhere when
falling back to PlatformIO's libretiny penv. CPython venvs put scripts
under Scripts/ on win32 and bin/ on POSIX; the prior hardcoded 'bin'
would never have found ltchiptool on Windows.
- _load_idedata: wrap json.loads on the prebuilt idedata.json in a
try/except and re-raise as EsphomeError with a one-line diagnostic so
the failure mode is a clean error instead of an unhandled
JSONDecodeError stack trace. Update the surrounding comment to match
the new behavior.
- CORE.prebuilt_dir docstring: drop the dead 'docs/architecture/...'
pointer (no docs/ tree in this repo); point at esphome-docs#6600 and
device-builder#572 instead.
New regression test:
- test_load_idedata_prebuilt_malformed_json_raises_esphomeerror
Updated test:
- test_get_ltchiptool_path_pio_penv now uses Scripts/ on win32 to match
the new platform-aware lookup.
Issue: esphome/device-builder#572
Issue: esphome/device-builder#570
The prior commit invoked 'pio pkg install' for the on-demand prep step,
which installs the libretiny platform package but does NOT recreate the
~/.platformio/penv/.libretiny/ virtualenv -- that's set up by libretiny's
ConfigurePythonVenv SCons step, which only fires during 'pio run'. So a
cold host (penv missing) still failed to find ltchiptool after the install
returned 'Already up-to-date'.
Wet test against bw15-device.yaml on a host with the libretiny penv
deleted:
esphome upload bw15-device.yaml --device /dev/null --prebuilt-dir ...
INFO ltchiptool not found on this host; installing the PlatformIO
rtl87xx platform package ...
Resolving bw15-device dependencies...
Already up-to-date.
ERROR ltchiptool not found. ...
Replace pkg install with 'pio run -t idedata'. The idedata target runs
SConscript without the actual compile target, so penv creation (libretiny)
and tool-package install (RP2040) happen as side effects of the SCons
configure phase -- but the build itself is skipped. Cost is a few seconds
of SCons configure, paid once per cold host.
Rename run_pkg_install -> prepare_platform_for_upload to match the new
semantics.
Issue: esphome/device-builder#572
Issue: esphome/device-builder#570
Symmetric with test_upload_program_prebuilt_dir_pkg_install_failure_aborts_upload:
covers the early-return path when write_cpp fails (codegen error, disk
full, validation regression) before pkg install runs. Closes the
coverage matrix for _ensure_platform_packages_for_prebuilt_upload's
two failure points.
Issue: esphome/device-builder#572
Revert ltchiptool as a hard dependency so the vast majority of ESPHome
users (ESP32 / ESP8266 / nRF52 OTA) don't pay for libretiny tooling.
Instead, on-demand install just the PlatformIO platform package when a
--prebuilt-dir upload needs a flash tool that isn't on disk yet.
Mechanism: when upload_program receives --prebuilt-dir for a libretiny
or RP2040 target and the corresponding flash tool (ltchiptool / picotool)
isn't found by get_ltchiptool_path / _find_picotool, run the same prep
that 'esphome compile' would (write_cpp + write platformio.ini), then
invoke 'pio pkg install -e <env>' instead of 'pio run'. PlatformIO
downloads the platform without compiling, the flash tool ends up on disk,
and the upload dispatch picks it up on the next lookup.
After install, the tool path lookups (PATH + PIO penv for ltchiptool,
PIO packages dir for picotool) succeed and the upload helper takes over.
ESP32 / ESP8266 / nRF52 paths are unchanged (esptool is already bundled
in requirements.txt; smpclient too for nRF52 mcumgr).
Issue: esphome/device-builder#572
Issue: esphome/device-builder#570
`esphome upload --prebuilt-dir` on a libretiny device routes through
ltchiptool to bypass the PlatformIO build-tree requirement. On hosts
that have never compiled a libretiny config locally (the dashboard's
transparent-install use case) the libretiny PlatformIO platform's
penv at ~/.platformio/penv/.libretiny/bin/ltchiptool doesn't exist
yet, leaving the upload to fail with an actionable but unwelcome
"install ltchiptool" hint.
Bundle ltchiptool as a direct dependency, mirroring esptool which is
already shipped even though only ESP users need it. `pip install
esphome` is now sufficient for libretiny serial uploads from a
prebuilt artifact set; get_ltchiptool_path's PATH lookup picks up
the pip-installed binary first, with the PIO penv fallback still in
place for environments that vendor esphome without the extra.
Issue: esphome/device-builder#572
Issue: esphome/device-builder#570
The dashboard's source-routed runner ships idedata.json with bare
basenames in prog_path and extra.flash_images[*].path (the receiver's
build-host absolute paths don't resolve on the offloader, and the
in-memory Web Serial consumer keys by basename). Without this change,
the dashboard would have to write a fresh idedata.json into the staging
tmpdir on every install just to flip basenames to absolute paths.
Resolve relative paths in the prebuilt idedata against CORE.prebuilt_dir
so both the dashboard's wire format and a hand-built directory with
absolute paths work. cc_path is left alone because it points at a
PlatformIO toolchain binary outside the prebuilt dir; the offloader's
own PIO install provides the matching binary.
Issue: esphome/device-builder#570
Issue: esphome/device-builder#572
- upload_using_ltchiptool: rename unused config arg to _config and note
the signature parity with upload_using_platformio/upload_using_picotool
so dispatch stays symmetric.
- upload_program: explicit exit_code = 1 when _rp2040_serial_reset_to_bootsel
fails, instead of relying on the function-level default.
- upload_using_picotool: distinct error message for the prebuilt-dir case
that names both candidates (idedata ELF + prebuilt firmware) instead of
just pointing at the missing ELF.
- _load_idedata: document the prebuilt-dir idedata.json contract more
loudly (absolute paths under prebuilt_dir, no schema validation, dashboard
owns the rewrites).
- --prebuilt-dir help text: drop the placeholder docs URL; describe the
dashboard-internal intent inline so users who hit the flag in --help
understand they don't want it.
- New test: upload_using_esptool with ESP-IDF toolchain + --prebuilt-dir
uses <prebuilt-dir>/firmware.factory.bin at offset 0x0.
Issue: esphome/device-builder#572
nRF52 has its own upload_program that runs before the default dispatch
in esphome.__main__. The mcumgr/BLE OTA path reads the MCUboot-signed
update image from CORE.relative_pioenvs_path(name, 'zephyr',
'app_update.bin'), which assumes a local Zephyr build tree.
Consult CORE.prebuilt_artifact_path('app_update.bin') first so the
dashboard's transparent BLE install on a Bluetooth proxy can flash a
prebuilt update image without compiling locally.
Serial uploads on non-MCUboot bootloaders still go through
adafruit-nrfutil via _upload_using_platformio and are out of scope here;
they need their own bypass to work with --prebuilt-dir (tracked
separately).
Issue: esphome/device-builder#572
Mirror the libretiny/ltchiptool shape for RP2040 serial when
--prebuilt-dir is set: open the user-supplied serial port at 1200 baud
(arduino-pico's USB CDC interprets the open/close as a request to reboot
into BOOTSEL), poll picotool until the BOOTSEL device shows up on the
USB bus, then dispatch to upload_using_picotool with the prebuilt .uf2.
This removes the last path that required --prebuilt-dir to contain a
platformio.ini + .pioenvs/<name>/ tree, so upload_using_platformio is
now only invoked when no prebuilt dir is set (i.e. the existing
compile+upload flow on a developer machine).
Issue: esphome/device-builder#572
The libretiny upload path on `upload_program` SERIAL dispatch re-invokes
PlatformIO (`pio run -t upload -t nobuild`), which needs a full build
tree and `platformio.ini`. That makes it incompatible with a dashboard
that only has prebuilt artifacts.
Bypass PlatformIO for the libretiny+SERIAL+--prebuilt-dir case by calling
`ltchiptool flash write -d <port> <firmware.uf2>` directly. The .uf2
encodes the chip family in its header so no extra config is needed.
ltchiptool ships with the libretiny PlatformIO platform under
~/.platformio/penv/.libretiny/bin/ltchiptool; `get_ltchiptool_path()`
prefers PATH first (pip install ltchiptool) and falls back to the
PlatformIO penv. Without --prebuilt-dir the existing PlatformIO path
remains in place, so this is purely additive.
Issue: esphome/device-builder#572
Adds a --prebuilt-dir flag to esphome upload that points the per-platform
upload helpers at a directory of prebuilt artifacts shipped from a paired
build server, instead of re-deriving paths from the local build tree.
Covers every upload-dispatch shape:
- ESP32 / ESP8266 serial (esptool) reads firmware.bin + extras via the
prebuilt idedata.json the dashboard ships next to the artifacts.
- ESP32 / ESP8266 OTA (native API + web_server) reads CORE.firmware_bin,
CORE.partition_table_bin and CORE.bootloader_bin which now consult the
prebuilt-dir first.
- RP2040 BOOTSEL (picotool) falls back from the idedata ELF (absent in
the flat layout) to the prebuilt firmware.uf2.
- RP2040 serial / libretiny serial / OTA (PlatformIO upload -t nobuild)
point platformio at the prebuilt build tree via CORE.build_path so the
-t upload -t nobuild path finds platformio.ini and .pioenvs/<name>/.
Issue: esphome/device-builder#572
Co-authored-by: pre-commit-ci-lite[bot] <117423508+pre-commit-ci-lite[bot]@users.noreply.github.com>
Co-authored-by: J. Nick Koston <nick@home-assistant.io>
