Compare commits

..
Author SHA1 Message Date
J. Nick Koston fd9760d8e3 [socket] Apply SO_REUSEADDR to the pcb and unmap IPv4-mapped IPv6 destinations 2026-08-28 11:23:43 -05:00
J. Nick Koston a4d2a4ad4b [socket] Zero lwip address struct and narrow dual-stack bind promotion 2026-08-28 10:31:20 -05:00
J. Nick Koston 7cbc81ad66 [socket] Simplify UDP implementation and deduplicate lwip helpers 2026-08-28 10:26:05 -05:00
J. Nick Koston c71a599f5a [socket] Address review feedback on UDP implementation 2026-08-28 10:08:58 -05:00
J. Nick Koston e6f9d9c369 Merge remote-tracking branch 'origin/dev' into socket-lwip-raw-udp
# Conflicts:
#	esphome/components/socket/__init__.py
2026-08-28 09:33:29 -05:00
Jesse Hills 2031be0c23 [core] Make script/setup idempotent and prepare new worktrees (#18843) 2026-08-28 14:11:38 +12:00
esphome[bot] 708b7bfb53 Update webserver local assets to 20260824-021242 (#18704) 2026-08-28 11:39:30 +12:00
J. Nick Koston 6d2e1f8658 Merge branch 'dev' into socket-lwip-raw-udp 2026-05-13 01:36:28 -05:00
J. Nick Koston e855ddb1f1 Merge branch 'dev' into socket-lwip-raw-udp 2026-04-22 08:22:37 +02:00
J. Nick Koston e191f5fb4b Merge branch 'dev' into socket-lwip-raw-udp 2026-04-21 15:08:29 +02:00
J. Nick Koston e7c126d3dc [socket] Rename UDP socket types so UDPSocket is the full send+recv type
Swap the UDP type naming so the unsuffixed UDPSocket is the full
send+recv socket and the send-only variant becomes UDPSendSocket. The
previous naming inverted reader expectations (UDPSocket sounded complete
but was send-only, UDPRecvSocket sounded limited but was the full one).

Public:
  UDPSocket       (was UDPRecvSocket)
  UDPSendSocket   (was UDPSocket)
  socket_udp                     (was socket_udp_recv)
  socket_ip_udp                  (was socket_ip_udp_recv)
  socket_udp_loop_monitored      (was socket_udp_recv_loop_monitored)
  socket_ip_udp_loop_monitored   (was socket_ip_udp_recv_loop_monitored)
  socket_udp_send                (was socket_udp)
  socket_ip_udp_send             (was socket_ip_udp)

Internal:
  LWIPRawUDPImpl       (was LWIPRawUDPRecvImpl)
  LWIPRawUDPSendImpl   (was LWIPRawUDPImpl)

No consumers exist yet, so this is a clean rename with no migration.
2026-04-10 18:40:14 -10:00
J. Nick Koston af7b3821b8 Merge branch 'dev' into socket-lwip-raw-udp 2026-04-10 18:31:45 -10:00
J. Nick Koston d6c48e2d64 [socket] Validate source address before copying data in recvfrom
Move pbuf_copy_partial after ip2sockaddr_ validation so the caller
buffer is not modified when the address conversion fails.
2026-04-09 09:21:48 -10:00
J. Nick Koston c01699f2a4 [socket] Move socket_loop_monitored declaration before UDP factories
Fix forward declaration ordering: socket_udp_recv_loop_monitored
calls socket_loop_monitored, so the latter must be declared first.
2026-04-09 09:20:36 -10:00
J. Nick Koston abc4069657 [socket] Restore LWIP thread safety documentation to common header 2026-04-09 09:09:38 -10:00
J. Nick Koston c3827423ba [socket] Move thin UDP and listen factory wrappers inline into socket.h
All the _ip_ variants and non-LWIP_TCP delegators are one-liners
that just forward to other factory functions. Move them inline to
reduce socket.cpp and keep the dispatch logic visible in one place.
2026-04-09 09:09:17 -10:00
J. Nick Koston 41a8e7f61b [socket] Split lwip raw impl into tcp, udp, and common files
Extract UDP classes and shared helpers from lwip_raw_tcp_impl into
separate files for better organization:

- lwip_raw_common_impl.{h,cpp}: shared helpers (lwip_ip_to_sockaddr,
  sockaddr_to_lwip, lwip_bind_err)
- lwip_raw_udp_impl.{h,cpp}: LWIPRawUDPImpl, LWIPRawUDPRecvImpl,
  and UDP factory functions
- lwip_raw_tcp_impl.{h,cpp}: TCP-only code (LWIPRawCommon,
  LWIPRawImpl, LWIPRawListenImpl, TCP factories)

Update __init__.py FILTER_SOURCE_FILES to exclude all three lwip
raw files when not using the lwip_tcp implementation.
2026-04-09 09:06:04 -10:00
J. Nick Koston 0a3f8c6d67 [socket] Add unified UDP recv loop-monitored factory functions
Add socket_udp_recv_loop_monitored() and socket_ip_udp_recv_loop_monitored()
so consumers can create UDP recv sockets with loop wake support using a single
API across all platforms, without #ifdef guards.
2026-04-09 08:56:11 -10:00
J. Nick Koston 2375faee88 fixes, update 2026-04-09 08:54:08 -10:00
J. Nick Koston 3da3a66d09 Merge branch 'dev' into socket-lwip-raw-udp 2026-04-09 07:55:07 -10:00
J. Nick Koston cf22559af0 Merge branch 'dev' into socket-lwip-raw-udp 2026-04-02 09:56:54 -10:00
J. Nick Koston 17c2557cca Merge branch 'dev' into socket-lwip-raw-udp 2026-04-01 18:47:13 -10:00
J. Nick Koston 012dadbf77 Merge branch 'dev' into socket-lwip-raw-udp 2026-03-22 21:27:55 -10:00
J. Nick Koston 97e4bb71c3 Merge branch 'dev' into socket-lwip-raw-udp 2026-03-18 19:31:21 -10:00
J. Nick Koston b333bb76e4 Merge branch 'dev' into socket-lwip-raw-udp 2026-03-16 16:36:46 -10:00
J. Nick Koston 273637b6d7 tweak 2026-03-16 16:35:32 -10:00
J. Nick Koston 75efdd8662 tweaks 2026-03-16 11:01:21 -10:00
J. Nick Koston 86e4341a52 tweaks 2026-03-16 11:00:57 -10:00
J. Nick Koston 402398b389 tweaks 2026-03-16 10:55:33 -10:00
J. Nick Koston dde81d3f63 tweaks 2026-03-16 10:51:54 -10:00
J. Nick Koston eef806c806 Merge branch 'dev' into socket-lwip-raw-udp 2026-03-16 10:44:13 -10:00
J. Nick Koston cdbbcfb87d [socket] Extract lwip_bind_err() to deduplicate bind error handling
TCP bind and UDP bind_internal_ had identical ERR_USE/ERR_VAL/ERR_OK
to errno mapping. Extract into a shared helper.
2026-03-14 16:12:39 -10:00
J. Nick Koston 71da3dc2de [socket] Fix RP2040 socket_delay race: don't clear wake flag before sleep
Restore the comment explaining why s_socket_woke must not be cleared
between the early-return check and the __wfe() loop, and restore the
s_socket_woke = false after the loop to consume the wake for the next
call. Both were lost during conflict resolution.
2026-03-14 16:09:24 -10:00
J. Nick Koston 0176305d24 [socket] Restore read_locked_, SO_RCVTIMEO, and wait_for_data_ lost in merge
These features from upstream/dev were dropped when resolving conflicts
with the PR's remote branch: read_locked_/wait_for_data_ (blocking read
with SO_RCVTIMEO timeout support), recv_timeout_cs_ field, SO_RCVTIMEO
and SO_SNDTIMEO setsockopt/getsockopt handling, and the setblocking()
implementation that accepts blocking mode for SO_RCVTIMEO.
2026-03-14 16:07:15 -10:00
J. Nick Koston c81e9fd154 [socket] Deduplicate sockaddr parsing between TCP and UDP bind
Extract sockaddr_to_lwip() from LWIPRawUDPImpl static method to a
shared file-level function. Refactor LWIPRawCommon::bind() to use it
instead of inline address parsing, removing ~35 lines of duplicated
sockaddr-to-ip_addr_t conversion code.
2026-03-14 16:03:24 -10:00
J. Nick Koston 556ef1894f Merge branch 'socket-lwip-raw-udp' of https://github.com/esphome/esphome into socket-lwip-raw-udp
# Conflicts:
#	esphome/components/socket/lwip_raw_tcp_impl.cpp
#	esphome/components/socket/lwip_raw_tcp_impl.h
2026-03-14 16:00:31 -10:00
J. Nick KostonandClaude Opus 4.6 519be06e73 [socket] Add LWIP_LOCK to UDP socket methods for RP2040 safety
On RP2040, lwip callbacks run from a low-priority IRQ context and can
preempt main-loop code. All lwip API calls from the main loop must
hold the async_context lock (LWIP_LOCK) to prevent races on shared
lwip state (PCB lists, pbuf pools, IGMP groups).

The TCP implementation was already correct; the UDP methods were
missing the lock.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-14 15:57:57 -10:00
J. Nick Koston 1e935c128a Merge remote-tracking branch 'upstream/dev' into socket-lwip-raw-udp
# Conflicts:
#	esphome/components/socket/lwip_raw_tcp_impl.cpp
2026-03-14 15:53:39 -10:00
J. Nick Koston a0e162912c safety 2026-03-11 00:00:52 -10:00
J. Nick Koston 1131af1690 safety 2026-03-10 23:59:25 -10:00
J. Nick Koston 6fb00baa29 Merge remote-tracking branch 'upstream/dev' into socket-lwip-raw-udp 2026-03-10 23:57:51 -10:00
J. Nick Koston fccfab8083 [socket] Switch UDP rx queue from lock-free SPSC to count-based with LWIP_LOCK
With the lwip lock infrastructure in place from the TCP race fix,
the lock-free SPSC ring buffer's wasted slot is no longer needed.
Switch to a simple rx_count_ approach that uses all 4 queue slots.

Also add LWIP_LOCK() to all UDP methods that call lwip APIs
(bind, close, sendto, setsockopt, getsockopt, recvfrom, factories).
2026-03-10 01:15:57 -10:00
J. Nick KostonandJ. Nick Koston f54756ae2d [socket] Address review comments: doc comments and (void) flags
- Document port_host byte order convention in lwip_ip_to_sockaddr
- Note intentional method hiding in LWIPRawUDPRecvImpl
- Note recvfrom truncation differs from POSIX MSG_TRUNC
- Add (void) flags in sendto to clarify flags are ignored

Co-Authored-By: J. Nick Koston <nick@koston.org>
2026-03-10 01:12:15 -10:00
J. Nick Koston 49ba08cec9 [socket] Add lwip raw UDP socket implementation
Add native UDP support to the lwip raw TCP socket layer used by
ESP8266 and RP2040, eliminating the need for Arduino WiFiUDP fallback.

Two new classes:
- LWIPRawUDPImpl: send-only UDP (8 bytes overhead)
- LWIPRawUDPRecvImpl: send+recv with fixed-size ring buffer (no heap
  allocation in recv callback)

Factory functions: socket_udp(), socket_udp_recv(), socket_ip_udp(),
socket_ip_udp_recv() with UDPSocket/UDPRecvSocket type aliases.

Additive only — no consumer migration in this PR.
2026-03-10 01:12:15 -10:00
J. Nick Koston 81d12fd14a [socket] Hold lwip lock for entire write() operation
Same pattern as writev — write() calls internal_write_() then
internal_output_(), each acquiring the lock separately. Hold
the lock at the outer scope so inner calls just bump the
recursion counter.
2026-03-10 00:57:36 -10:00
J. Nick Koston cc05bf3ed2 [socket] Add LWIP_LOCK to socket factory functions
tcp_new() is an lwip core API call that must be bracketed with
the lwip lock on RP2040 per pico-sdk docs. Add LWIP_LOCK() to
socket() and socket_listen() factory functions.
2026-03-10 00:55:34 -10:00
J. Nick Koston c182c0c74f [socket] Hold lwip lock for entire readv/writev scatter-gather operation
Avoid repeated lock acquire/release cycles per iovec element.
The recursive mutex re-entry in inner calls is nearly free (counter
bump), while the outer lock prevents the expensive IRQ disable/enable
on each iteration.
2026-03-10 00:53:01 -10:00
J. Nick Koston a88e9b8146 [socket] Fix RP2040 TCP race condition between lwip callbacks and main loop
On RP2040 (Pico W), arduino-pico sets PICO_CYW43_ARCH_THREADSAFE_BACKGROUND=1,
which means lwip callbacks (recv_fn, accept_fn, err_fn) run from a PendSV
interrupt — not the main loop. This allows them to preempt read(), write(),
close(), and accept() at any point, causing race conditions on shared state
like the rx_buf_ pbuf chain.

The most critical race: recv_fn calls pbuf_cat(rx_buf_, pb) while read() is
freeing nodes in the same chain, leading to use-after-free and lwip's
"Creating an infinite loop" assertion panic. This is the root cause of #10681.

Fix: implement RP2040's LwIPLock (previously a no-op) to call
cyw43_arch_lwip_begin/end, which acquires the pico-sdk async_context recursive
mutex. Add LWIP_LOCK() guards to all main-loop lwip API call sites in the
socket layer.

On ESP8266, lwip callbacks run cooperatively from the main loop, so
LwIPLock remains a no-op.

Closes #10681
2026-03-10 00:34:20 -10:00
J. Nick KostonandJ. Nick Koston 7dea3756e9 [socket] Address review comments: doc comments and (void) flags
- Document port_host byte order convention in lwip_ip_to_sockaddr
- Note intentional method hiding in LWIPRawUDPRecvImpl
- Note recvfrom truncation differs from POSIX MSG_TRUNC
- Add (void) flags in sendto to clarify flags are ignored

Co-Authored-By: J. Nick Koston <nick@koston.org>
2026-03-09 23:56:37 -10:00
J. Nick Koston fa0bff3374 [socket] Add lwip raw UDP socket implementation
Add native UDP support to the lwip raw TCP socket layer used by
ESP8266 and RP2040, eliminating the need for Arduino WiFiUDP fallback.

Two new classes:
- LWIPRawUDPImpl: send-only UDP (8 bytes overhead)
- LWIPRawUDPRecvImpl: send+recv with fixed-size ring buffer (no heap
  allocation in recv callback)

Factory functions: socket_udp(), socket_udp_recv(), socket_ip_udp(),
socket_ip_udp_recv() with UDPSocket/UDPRecvSocket type aliases.

Additive only — no consumer migration in this PR.
2026-03-09 23:48:07 -10:00
85 changed files with 5122 additions and 22298 deletions
+16
View File
@@ -0,0 +1,16 @@
{
"hooks": {
"SessionStart": [
{
"hooks": [
{
"type": "command",
"command": "d=\"${CLAUDE_PROJECT_DIR:-.}\"; [ -x \"$d/venv/bin/python\" ] || { mkdir -p \"$d/.temp\"; env -u VIRTUAL_ENV \"$d/script/setup\" >\"$d/.temp/setup.log\" 2>&1 || echo '{\"systemMessage\":\"script/setup failed; see .temp/setup.log\"}'; }",
"statusMessage": "Setting up dev environment (script/setup)...",
"timeout": 900
}
]
}
]
}
}
@@ -1,39 +0,0 @@
name: Cache Arduino ESP8266
description: >
Resolve the pinned Arduino core and xtensa toolchain versions and cache the
native ESP8266 install (~110 MB framework + toolchain; no ccache store, the
seed job saves before any compile runs). Exports
ESPHOME_ARDUINO8266_PREFIX to the job so every later step installs into
the cached path; the Python venv must already be restored. Mirrors
cache-esp-idf: only dev-branch pushes write the shared cache, everything
else restores.
runs:
using: composite
steps:
- name: Resolve the native toolchain cache key
# Versions are pinned in code, not a hashable file; resolve them so a
# bump changes the cache key. Assignment form so errexit catches a
# resolver failure.
id: version
shell: bash
run: |
# One owner for the install prefix: exported here and referenced by
# the cache steps below via env, so the caller's install and the
# cached path cannot diverge.
echo "ESPHOME_ARDUINO8266_PREFIX=$HOME/.esphome-arduino8266" >> "$GITHUB_ENV"
. venv/bin/activate
key=$(python -c 'from esphome.components.esp8266 import RECOMMENDED_ARDUINO_FRAMEWORK_VERSION as f; from esphome.arduino8266.framework import TOOLCHAIN_VERSION as t; print(f"{f}-{t}")')
[ -n "$key" ] || exit 1
echo "key=$key" >> "$GITHUB_OUTPUT"
- name: Cache the native toolchain (write on dev)
if: github.ref == 'refs/heads/dev'
uses: actions/cache@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
with:
path: ${{ env.ESPHOME_ARDUINO8266_PREFIX }}
key: ${{ runner.os }}-esp8266-native-${{ steps.version.outputs.key }}
- name: Restore the native toolchain (off dev)
if: github.ref != 'refs/heads/dev'
uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
with:
path: ${{ env.ESPHOME_ARDUINO8266_PREFIX }}
key: ${{ runner.os }}-esp8266-native-${{ steps.version.outputs.key }}
-13
View File
@@ -21,7 +21,6 @@ on:
- "esphome/core/**"
- "esphome/writer.py"
- "esphome/build_gen/**"
- "esphome/build_helpers/**"
- "esphome/espidf/**"
- "esphome/platformio/**"
- "esphome/components/bk72xx/**"
@@ -198,7 +197,6 @@ jobs:
# the default.
id:
- esp8266-arduino
- esp8266-arduino-native
- esp32-arduino-platformio
- esp32-arduino-esp-idf
- esp32-idf-platformio
@@ -209,16 +207,6 @@ jobs:
- ln882x-arduino
- nrf52
- host
# Strict by default so a new matrix id cannot silently join in the
# degrade-quietly mode the knob exists to catch.
# Opt-outs: libretiny GCC rejects its own pch until a toolchain bump.
include:
- id: bk72xx-arduino
pch_strict: "0"
- id: rtl87xx-arduino
pch_strict: "0"
- id: ln882x-arduino
pch_strict: "0"
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
- name: Download image artifact
@@ -230,7 +218,6 @@ jobs:
- name: Compile ${{ matrix.id }}
run: |
docker run --rm \
-e ESPHOME_PCH_STRICT="${{ matrix.pch_strict || '1' }}" \
-v "${{ github.workspace }}/docker/test_configs:/config" \
"ghcr.io/esphome/esphome-amd64:${{ needs.check-docker.outputs.tag }}" \
compile "${{ matrix.id }}.yaml"
+1 -69
View File
@@ -101,8 +101,6 @@ jobs:
device-builder: ${{ steps.determine.outputs.device-builder }}
esp32-platformio: ${{ steps.determine.outputs.esp32-platformio }}
esp32-platformio-components: ${{ steps.determine.outputs.esp32-platformio-components }}
esp8266-native: ${{ steps.determine.outputs.esp8266-native }}
esp8266-native-components: ${{ steps.determine.outputs.esp8266-native-components }}
changed-components: ${{ steps.determine.outputs.changed-components }}
changed-components-with-tests: ${{ steps.determine.outputs.changed-components-with-tests }}
directly-changed-components-with-tests: ${{ steps.determine.outputs.directly-changed-components-with-tests }}
@@ -163,8 +161,6 @@ jobs:
echo "device-builder=$(echo "$output" | jq -r '.device_builder')" >> $GITHUB_OUTPUT
echo "esp32-platformio=$(echo "$output" | jq -r '.esp32_platformio')" >> $GITHUB_OUTPUT
echo "esp32-platformio-components=$(echo "$output" | jq -r '.esp32_platformio_components')" >> $GITHUB_OUTPUT
echo "esp8266-native=$(echo "$output" | jq -r '.esp8266_native')" >> $GITHUB_OUTPUT
echo "esp8266-native-components=$(echo "$output" | jq -r '.esp8266_native_components')" >> $GITHUB_OUTPUT
echo "changed-components=$(echo "$output" | jq -c '.changed_components')" >> $GITHUB_OUTPUT
echo "changed-components-with-tests=$(echo "$output" | jq -c '.changed_components_with_tests')" >> $GITHUB_OUTPUT
echo "directly-changed-components-with-tests=$(echo "$output" | jq -c '.directly_changed_components_with_tests')" >> $GITHUB_OUTPUT
@@ -183,32 +179,6 @@ jobs:
path: .temp/components_graph.json
key: components-graph-${{ hashFiles('esphome/components/**/*.py') }}
seed-esp8266-native-cache:
name: Seed the esp8266 native toolchain cache
runs-on: ubuntu-24.04
needs:
- common
# PR-branch cache saves are invisible to other PRs, so dev pushes seed
# the shared entry test-esp8266-native restores. Only dev: the composite
# action saves nowhere else, so a beta/release push would download the
# toolchain and discard it.
if: github.event_name == 'push' && github.ref == 'refs/heads/dev'
timeout-minutes: 15
steps:
- name: Check out code from GitHub
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
- name: Restore Python
uses: ./.github/actions/restore-python
with:
python-version: ${{ env.DEFAULT_PYTHON }}
cache-key: ${{ needs.common.outputs.cache-key }}
- name: Cache the native toolchain
uses: ./.github/actions/cache-arduino8266
- name: Install the native toolchain
run: |
. venv/bin/activate
python -c "from esphome.arduino8266.framework import check_and_install; from esphome.components.esp8266 import RECOMMENDED_ARDUINO_FRAMEWORK_VERSION; check_and_install(RECOMMENDED_ARDUINO_FRAMEWORK_VERSION)"
ci-custom:
name: Run script/ci-custom
runs-on: ubuntu-24.04
@@ -1162,7 +1132,7 @@ jobs:
# compile validates config first, so a separate config pass is
# redundant for this smoke test. ESP-IDF framework via PlatformIO:
python3 script/test_build_components.py -e compile -t esp32-idf -c "$TEST_COMPONENTS" -f --toolchain platformio --fail-on-no-tests
python3 script/test_build_components.py -e compile -t esp32-idf -c "$TEST_COMPONENTS" -f --toolchain platformio
echo ""
echo "ESP-IDF-via-PlatformIO build passed! Starting Arduino smoke test..."
@@ -1171,42 +1141,6 @@ jobs:
# Arduino framework via PlatformIO (only components with an esp32-ard test are built):
python3 script/test_build_components.py -e compile -t esp32-ard -c "$TEST_COMPONENTS" -f --toolchain platformio
test-esp8266-native:
name: Test esp8266 components with the native toolchain
runs-on: ubuntu-24.04
needs:
- common
- determine-jobs
if: github.event_name == 'pull_request' && needs.determine-jobs.outputs.esp8266-native == 'true'
env:
# Computed by script/determine-jobs.py (ESP8266_NATIVE_TEST_COMPONENTS)
TEST_COMPONENTS: ${{ needs.determine-jobs.outputs.esp8266-native-components }}
steps:
- name: Check out code from GitHub
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
- name: Restore Python
uses: ./.github/actions/restore-python
with:
python-version: ${{ env.DEFAULT_PYTHON }}
cache-key: ${{ needs.common.outputs.cache-key }}
- name: Cache the native toolchain
uses: ./.github/actions/cache-arduino8266
- name: Run native toolchain compile test
run: |
. venv/bin/activate
echo "Testing components: $TEST_COMPONENTS"
echo ""
# ESP8266 Arduino built directly (no PlatformIO); compile validates
# config first, so a separate config pass is redundant. Strict pch:
# exercises the native ninja pch (and its probe edge) against real
# component configs; the docker matrix smoke-tests both toolchains.
ESPHOME_PCH_STRICT=1 python3 script/test_build_components.py -e compile -t esp8266-ard -c "$TEST_COMPONENTS" -f --toolchain arduino --fail-on-no-tests
device-builder:
name: Test downstream esphome/device-builder
runs-on: ubuntu-24.04
@@ -1569,7 +1503,6 @@ jobs:
needs:
- common
- seed-apt-cache
- seed-esp8266-native-cache
- determine-jobs
- ci-custom
- pylint
@@ -1585,7 +1518,6 @@ jobs:
- clang-tidy-esp32-variants
- test-build-components-split
- test-esp32-platformio
- test-esp8266-native
- device-builder
- memory-impact-target-branch
- memory-impact-pr-branch
@@ -1,8 +0,0 @@
esphome:
name: docker-test-esp8266-native
esp8266:
board: d1_mini
toolchain: arduino
logger:
+31 -83
View File
@@ -815,9 +815,7 @@ def write_cpp_file() -> int:
from esphome.build_gen import espidf
espidf.write_project()
elif not CORE.using_native_toolchain:
# Other native builds generate their project at compile time;
# never write a platformio.ini for them
else:
from esphome.build_gen import platformio
platformio.write_project()
@@ -859,14 +857,20 @@ def compile_program(args: ArgsProtocol, config: ConfigType) -> int:
toolchain.create_factory_bin()
toolchain.create_ota_bin()
toolchain.create_elf_copy()
from esphome.build_helpers.idedata import warn_if_idedata_missing
from esphome.build_helpers.idedata import IDEDATA_BEST_EFFORT_ERRORS
warn_if_idedata_missing(toolchain.get_idedata)
elif CORE.using_native_toolchain:
raise EsphomeError(
f"Toolchain '{CORE.toolchain.value}' resolved but no platform "
"backend claimed the build"
)
try:
if toolchain.get_idedata() is None:
_LOGGER.warning("No idedata was generated for this build")
except IDEDATA_BEST_EFFORT_ERRORS as err:
# The firmware already built; an idedata failure must not fail
# a successful build.
_LOGGER.warning(
"Could not generate idedata: %s (IDE, clang-tidy, and "
"memory-analysis data will be unavailable for this build)",
err,
)
_LOGGER.debug("Idedata failure detail", exc_info=True)
else:
from esphome.platformio import toolchain
@@ -969,15 +973,12 @@ def upload_using_esptool(
if file is not None:
flash_images = [FlashImage(path=file, offset="0x0")]
elif (native := _native_toolchain_module()) is not None:
# Every native backend supplies its own 0x0 flash image (bootloader
# and partitions included where the target needs them)
image = native.get_factory_firmware_path()
if not image.is_file():
raise EsphomeError(
f"{image} does not exist; compile the configuration first"
)
flash_images = [FlashImage(path=image, offset="0x0")]
elif CORE.using_toolchain_esp_idf:
from esphome.espidf import toolchain
flash_images = [
FlashImage(path=toolchain.get_factory_firmware_path(), offset="0x0")
]
else:
from esphome.platformio import toolchain
@@ -1923,39 +1924,15 @@ def command_update_all(args: ArgsProtocol) -> int | None:
return run_multiple_configs(files, build_command)
# Native build backend per (target platform, toolchain). Keyed here rather
# than through a platform hook so the serial upload/logs fast path never
# imports the platform component package (see the esp32 variant comment in
# upload_using_esptool); the platform half comes from CORE.data the same way.
_NATIVE_TOOLCHAIN_MODULES = {
("esp32", Toolchain.ESP_IDF): "esphome.espidf.toolchain",
("esp8266", Toolchain.ARDUINO): "esphome.arduino8266.toolchain",
}
def _native_toolchain_module():
"""The native build backend module for the resolved toolchain."""
if not CORE.using_native_toolchain:
return None
key = (CORE.target_platform, CORE.toolchain)
if (module_path := _NATIVE_TOOLCHAIN_MODULES.get(key)) is None:
# Degrading to the PlatformIO path would build with the wrong backend
raise EsphomeError(
f"Toolchain '{CORE.toolchain.value}' has no native build backend "
f"module for platform {CORE.target_platform}"
)
return importlib.import_module(module_path)
def command_idedata(args: ArgsProtocol, config: ConfigType) -> int:
import json
native_toolchain = _native_toolchain_module()
if CORE.using_toolchain_esp_idf:
# Native ESP-IDF derives idedata from the build's compile_commands.json,
# so the configuration must already be compiled.
from esphome.espidf import toolchain as espidf_toolchain
if native_toolchain is not None:
# Native toolchains derive idedata from the build's
# compile_commands.json, so the configuration must already be compiled.
idedata = native_toolchain.get_idedata()
idedata = espidf_toolchain.get_idedata()
if idedata is None:
_LOGGER.error(
"No idedata available; compile the configuration first",
@@ -1994,17 +1971,6 @@ def command_analyze_memory(args: ArgsProtocol, config: ConfigType) -> int:
from esphome.analyze_memory.cli import MemoryAnalyzerCLI
from esphome.analyze_memory.ram_strings import RamStringsAnalyzer
# Refuse an unsupported toolchain before paying for a full compile
native_toolchain = _native_toolchain_module()
if native_toolchain is None and not CORE.using_toolchain_platformio:
_LOGGER.error(
"analyze-memory is not supported with the '%s' toolchain on %s; "
"re-run with --toolchain platformio",
CORE.toolchain.value if CORE.toolchain else "unresolved",
CORE.target_platform,
)
return 1
# Always compile to ensure fresh data (fast if no changes - just relinks)
exit_code = write_cpp(config)
if exit_code != 0:
@@ -2016,31 +1982,13 @@ def command_analyze_memory(args: ArgsProtocol, config: ConfigType) -> int:
# Get idedata for analysis
idedata = None
if native_toolchain is not None:
objdump = native_toolchain.get_objdump_path()
readelf = native_toolchain.get_readelf_path()
for tool in (objdump, readelf):
if not tool.is_file():
# The analyzer would silently fall back to host binutils,
# which cannot read the target ELF. clean-all is heavy for
# ESP-IDF, so suggest a recompile first.
_LOGGER.error(
"%s is missing; the toolchain install may be incomplete "
"(recompile, or run 'esphome clean-all' if it persists)",
tool,
)
return 1
objdump_path = str(objdump)
readelf_path = str(readelf)
if CORE.using_toolchain_esp_idf:
from esphome.espidf import toolchain
firmware_elf = native_toolchain.get_elf_path()
if not firmware_elf.is_file():
# The analyzer swallows tool failures, so a missing ELF would
# produce an exit-0 zeroed report
_LOGGER.error(
"%s is missing; compile the configuration first", firmware_elf
)
return 1
objdump_path = str(toolchain.get_objdump_path())
readelf_path = str(toolchain.get_readelf_path())
firmware_elf = toolchain.get_elf_path()
else:
from esphome.platformio import toolchain
View File
-531
View File
@@ -1,531 +0,0 @@
"""Arduino-core backend for the shared PlatformIO library converter.
Bundled names build straight from the framework tree; everything else goes
through ``esphome.platformio.library``. Mirrors ``lib_ldf_mode=off``: each
library builds its own archive; all include dirs join one global path.
Deviations from PlatformIO: flat-layout libraries get the recursive default
source filter; ``dot_a_linkage`` is honored; bundled libraries never run a
manifest ``extraScript``; manifest ``-I`` flags join the global include path;
``precompiled``/``ldflags`` properties are refused by name.
"""
from __future__ import annotations
from dataclasses import dataclass, field
import logging
from pathlib import Path
import re
from esphome.core import CORE, EsphomeError, Library
from esphome.helpers import walk_files
from esphome.platformio.extra_script import apply_extra_script
from esphome.platformio.library import (
DEFAULT_BUILD_INCLUDE_DIR,
DEFAULT_BUILD_SRC_FILTER,
ESPHOME_DATA_KEY,
ESPHOME_DATA_LINK_FLAGS_KEY,
LIBRARY_HEADER_SUFFIXES,
SRC_FILE_EXTENSIONS,
ConvertedLibrary,
IncompatiblePlatform,
InvalidLibrary,
LibraryBackend,
_url_or_none,
check_library_data,
collect_filtered_files,
convert_libraries,
ensure_list,
is_lib_ignored,
lex_build_flags,
lib_ignore_set,
normalize_dependencies,
parse_library_json,
parse_library_properties,
warn_properties_depends,
)
_LOGGER = logging.getLogger(__name__)
@dataclass
class ArduinoLibrary:
"""One resolved library, ready for the ninja generator."""
name: str
sources: list[Path] = field(default_factory=list)
include_dirs: list[Path] = field(default_factory=list)
# Extra compile flags private to this library's own sources
flags: list[str] = field(default_factory=list)
# PlatformIO's build.libArchive / Arduino's dot_a_linkage: when False the
# objects go to the linker directly (symbols nothing references survive)
lib_archive: bool = True
# Link inputs the library contributes (-L dirs / -l libs, e.g. from
# precompiled vendor blobs) and -Wl, options for the firmware link
link_dirs: list[Path] = field(default_factory=list)
link_libs: list[str] = field(default_factory=list)
link_flags: list[str] = field(default_factory=list)
# Source-like suffixes the case-sensitive suffix map rejects
_UNMAPPED_SOURCE_SUFFIXES = frozenset(
{s.lower() for s in SRC_FILE_EXTENSIONS} | {".ino"}
)
# Filename-plain names: an allowlist excludes separators, drive colons,
# and dot-only names by shape
_SAFE_LIBRARY_NAME_RE = re.compile(r"[A-Za-z0-9_][A-Za-z0-9_. +-]*\Z")
def _is_safe_library_name(name: object) -> bool:
"""Whether a name may be joined under the framework's libraries dir."""
return isinstance(name, str) and _SAFE_LIBRARY_NAME_RE.fullmatch(name) is not None
def _manifest_build(name: str, data: object) -> dict:
"""The manifest's ``build`` section; malformed manifests fail by name."""
build = data.get("build", {}) if isinstance(data, dict) else None
if not isinstance(build, dict):
raise EsphomeError(f"Library {name} has a malformed manifest")
return build
def _resolve_src_dir(name: str, read_path: Path, build: dict) -> str:
"""Resolve PIO's source dir: manifest srcDir, else src/Src, else the root."""
if "srcDir" not in build:
return next((d for d in ("src", "Src") if (read_path / d).is_dir()), ".")
# A declared srcDir (falsy included) that does not resolve is a manifest error
src_dir = build["srcDir"]
if not (isinstance(src_dir, str) and src_dir and (read_path / src_dir).is_dir()):
raise EsphomeError(
f"Library {name} declares srcDir {src_dir!r} which does not exist"
)
return src_dir
def _reject_unsupported_link_fields(name: str, data: dict) -> None:
# PIO honors these; ignoring them would fail at link with no stated
# cause. Property values are strings, so "false" is not a declaration.
precompiled = data.get("precompiled")
if precompiled and str(precompiled).strip().lower() != "false":
raise EsphomeError(
f"Library {name} declares precompiled, which this backend does not support"
)
if data.get("ldflags"):
raise EsphomeError(
f"Library {name} declares ldflags, which this backend does not support"
)
def _resolve_lib_archive(name: str, data: dict, build: dict) -> bool:
"""build.libArchive, else dot_a_linkage (an Arduino IDE property PIO
ignores; a deliberate extra), else archive."""
# Strict parse: bool("false") is True
def _parse(key: str, raw: object) -> bool:
if isinstance(raw, bool):
return raw
value = str(raw).strip().lower()
if value in ("true", "false"):
return value == "true"
raise EsphomeError(f"Library {name} has a malformed {key} value {raw!r}")
if "libArchive" in build:
return _parse("libArchive", build["libArchive"])
if "dot_a_linkage" in data:
return _parse("dot_a_linkage", data["dot_a_linkage"])
return True
def _classify_build_flags(
name: str, read_path: Path, lib: ArduinoLibrary, flag_tokens: list[str]
) -> list[str]:
"""Route the lexed build.flags into the library's flag lists.
Returns the ``-I`` arguments for the include-dir resolution.
"""
include_flags: list[str] = []
for tok in flag_tokens:
if tok.startswith("-I"):
include_flags.append(tok[2:])
elif tok.startswith("-L"):
link_dir = (read_path / tok[2:]).resolve()
if not link_dir.is_dir():
# Kept (the linker ignores missing -L dirs); the warning
# names the culprit before a bare "cannot find -lfoo"
_LOGGER.warning(
"Library %s declares library dir %s which does not exist",
name,
tok[2:],
)
lib.link_dirs.append(link_dir)
elif tok.startswith("-l"):
lib.link_libs.append(tok[2:])
elif tok.startswith("-Wl,"):
lib.link_flags.append(tok)
else:
lib.flags.append(tok)
return include_flags
def _resolve_include_dirs(
name: str,
read_path: Path,
lib: ArduinoLibrary,
build: dict,
src_dir: str,
include_flags: list[str],
) -> None:
include_dir = build.get("includeDir", DEFAULT_BUILD_INCLUDE_DIR)
if not isinstance(include_dir, str):
raise EsphomeError(f"Library {name} has a malformed includeDir")
for d, explicit in [
(include_dir, "includeDir" in build),
(src_dir, False), # _resolve_src_dir already validated it
*((flag, True) for flag in include_flags),
]:
if (path := (read_path / d)).is_dir():
lib.include_dirs.append(path.resolve())
elif explicit:
# Warn-and-drop (unlike srcDir): a missing include dir is
# harmless until a header is needed, and the compile names it
_LOGGER.warning(
"Library %s declares include dir %s which does not exist", name, d
)
def _collect_lib_sources(
name: str,
read_path: Path,
lib: ArduinoLibrary,
src_dir: str,
src_filter: list[str],
) -> None:
sources: list[Path] = []
dropped: list[str] = []
saw_header = False
for f in collect_filtered_files(read_path / src_dir, src_filter):
path = Path(f)
suffix = path.suffix
if suffix in SRC_FILE_EXTENSIONS:
# resolve() per file: srcFilter patterns may escape src_dir
sources.append(path.resolve())
elif suffix.lower() in _UNMAPPED_SOURCE_SUFFIXES:
# A source-like suffix the case-sensitive map rejects (.CPP,
# .ino) is a dropped compilation unit; headers fall through
dropped.append(path.name)
elif suffix.lower() in LIBRARY_HEADER_SUFFIXES:
saw_header = True
lib.sources = sorted(sources)
if dropped:
_LOGGER.warning(
"Library %s: %d file(s) with unmapped source suffixes are not compiled: %s",
name,
len(dropped),
", ".join(sorted(dropped)),
)
if not lib.sources and not saw_header:
# Matched headers mean header-only; a filter matching nothing is
# a manifest/tree problem (a truly empty tree raises elsewhere)
_LOGGER.warning("Library %s: no source files matched", name)
def _library_info(name: str, read_path: Path, data: dict) -> ArduinoLibrary:
"""Resolve one library's sources, include dirs, and flags (PIO semantics)."""
build = _manifest_build(name, data)
_reject_unsupported_link_fields(name, data)
src_dir = _resolve_src_dir(name, read_path, build)
src_filter = ensure_list(build.get("srcFilter", DEFAULT_BUILD_SRC_FILTER))
if not all(isinstance(entry, str) for entry in src_filter):
raise EsphomeError(f"Library {name} has a malformed srcFilter")
lib = ArduinoLibrary(name=name, lib_archive=_resolve_lib_archive(name, data, build))
# PlatformIO shell-lexes each build.flags entry
include_flags = _classify_build_flags(
name, read_path, lib, lex_build_flags(build.get("flags", []), f"library {name}")
)
_resolve_include_dirs(name, read_path, lib, build, src_dir, include_flags)
_collect_lib_sources(name, read_path, lib, src_dir, src_filter)
return lib
def _bundled_library(framework_path: Path, name: str) -> ArduinoLibrary:
"""A library bundled with the Arduino core, read from the framework tree.
``library.json`` wins over ``library.properties`` when both exist, as in
PlatformIO's LibBuilderFactory; only the JSON manifest can carry a
``build`` section (srcDir, srcFilter, flags).
"""
lib_dir = framework_path / "libraries" / name
manifest_json = lib_dir / "library.json"
if manifest_json.is_file():
try:
data = parse_library_json(manifest_json)
except ValueError as err: # JSONDecodeError
raise EsphomeError(
f"Bundled library {name} has a corrupt library.json ({err}); "
"the framework install may be incomplete (run 'esphome clean-all')"
) from err
elif (manifest := lib_dir / "library.properties").is_file():
data = parse_library_properties(manifest)
else:
# Debug, not warning: the legacy manifest-less layout is legal and
# the 3.1.2 core ships one such library (FSTools), so a warning
# would be unactionable noise on every build using it
_LOGGER.debug("Bundled library %s has no manifest; using defaults", name)
data = {}
if isinstance(data, dict):
# Bundled manifest deps are never walked; make the skip visible
if data.get("dependencies"):
_LOGGER.warning(
"Bundled library %s declares dependencies, which are not "
"resolved automatically; add them with add_library() if needed",
name,
)
warn_properties_depends(name, data)
build = data.get("build")
if isinstance(build, dict) and build.get("extraScript"):
# Scripts only run on the converted path; building without
# the script's flags would miscompile
raise EsphomeError(
f"Bundled library {name} declares an extraScript, which is "
"not run for bundled libraries"
)
lib = _library_info(name, lib_dir, data)
_assert_tree_has_code(
name,
lib_dir,
"the framework install may be incomplete (run 'esphome clean-all')",
)
return lib
def _assert_tree_has_code(name: str, root: Path, hint: str) -> None:
"""An empty or half-extracted tree can never link; fail by name (a
warning would scroll away and resurface as undefined symbols)."""
if not any(
Path(p).suffix in SRC_FILE_EXTENSIONS
or Path(p).suffix.lower() in LIBRARY_HEADER_SUFFIXES
for p in walk_files(root)
):
raise EsphomeError(f"Library {name} has no sources or headers; {hint}")
def _external_short_name(name: str) -> str:
"""The short library name of a requested spec.
"owner/Name" and plain names take the last path segment; "Name=<url>"
takes the declared name. Git tails (".git", "#ref") are stripped like
the walk's URL normalization; the comparand is a manifest dependency
name, never a spec.
"""
head, sep, tail = name.partition("=")
if sep and "://" in tail:
return head
short = name.rsplit("/", maxsplit=1)[-1]
return short.partition("#")[0].removesuffix(".git")
def _check_unfulfilled_provides(
provided_requests: set[str], satisfied: set[str], still_requested: set[str]
) -> None:
"""Fail by name when a walk-skipped dependency was never added.
An unfulfilled provides() promise only surfaces as undefined symbols
at link. The walk records across re-resolutions, so a name no final
manifest still requests is stale state, never a failure.
"""
if missing := sorted((provided_requests & still_requested) - satisfied):
raise EsphomeError(
"provides() skipped these dependencies but nothing added them: "
f"{', '.join(missing)}; the build is missing libraries"
)
def resolve_libraries(
framework_path: Path, *, pio_platform: str, board_mcu: str, cache_key: str
) -> list[ArduinoLibrary]:
"""Resolve every ``cg.add_library()`` entry into an :class:`ArduinoLibrary`.
``pio_platform``/``board_mcu`` filter manifests the way PlatformIO would
for that core (e.g. ``espressif8266``/``esp8266``); ``cache_key`` keys the
shared converter's download cache.
The returned list is not topologically sorted, so the caller must link
the archives inside one ``--start-group``/``--end-group`` pair (the
bundled-first grouping is incidental).
"""
bundled: list[ArduinoLibrary] = []
external: list[Library] = []
# PlatformIO's lib_ignore covers framework-bundled libraries too; the
# shared converter only filters the registry/git ones.
lib_ignore = lib_ignore_set()
# Exact directory names keep membership case-sensitive everywhere
# (an is_dir() probe would match "wire" on macOS/Windows and build
# the bundled Wire twice)
libraries_dir = framework_path / "libraries"
if not libraries_dir.is_dir():
# A registry fallback would fail later with a misleading
# package-not-found error per bundled name
raise EsphomeError(
f"{libraries_dir} is missing; the framework install may be "
"incomplete (run 'esphome clean-all')"
)
bundled_dir_names = frozenset(p.name for p in libraries_dir.iterdir() if p.is_dir())
def _provided(name: object) -> bool:
return _is_safe_library_name(name) and name in bundled_dir_names
for library in CORE.platformio_libraries.values():
if is_lib_ignored(library.name, lib_ignore):
continue
# Bundled only for a bare name with a matching framework dir; pinned
# or unmatched names resolve from the registry, as under PlatformIO.
if not library.repository and not library.version and _provided(library.name):
# Bundled manifest deps are not walked; _bundled_library warns
bundled.append(_bundled_library(framework_path, library.name))
else:
external.append(library)
converted: list[ArduinoLibrary] = []
bundled_names = {lib.name for lib in bundled}
converted_manifest_names: set[str] = set()
# Bundled candidates skipped on purpose (platform filter); the
# provides() reconciliation must count them as satisfied
knowingly_skipped: set[str] = set()
# Dependency names of the manifests actually emitted; a walk recording
# for a since-re-resolved manifest must not fail the reconciliation
final_dep_names: set[str] = set()
# Ordered set of bundled dependency names to add once conversion is done
pending_bundled: dict[str, None] = {}
# Deps matching a separately-requested external are already in the build
# (a duplicate archive means duplicate-symbol link errors)
external_short_names = {
_external_short_name(lib.name) for lib in external if lib.name
}
def _add_bundled_dependencies(component: ConvertedLibrary) -> None:
# A version-less bare name ("Hash") is a core-bundled library the
# shared converter cannot resolve from the registry
for dep in normalize_dependencies(
component.data.get("dependencies"), component.name
):
# normalize_dependencies guarantees a non-empty str name
name = dep["name"]
final_dep_names.add(name)
if "/" in name:
owner, _, pkg = name.partition("/")
if _is_safe_library_name(owner) and _is_safe_library_name(pkg):
# Owner-qualified; the converter resolves it from the registry
continue
if not _is_safe_library_name(name):
# The name becomes a path component; never join a traversal
_LOGGER.warning(
"Ignoring malformed dependency entry %r of library %s",
dep,
component.name,
)
continue
if name in external_short_names:
if _provided(name):
# A bundled copy is suppressed; a coincidental name
# collision would surface as link errors
_LOGGER.warning(
"Dependency %s of %s is assumed satisfied by a "
"requested external library; the bundled copy is "
"not added",
name,
component.name,
)
else:
_LOGGER.debug(
"Dependency %s of %s assumed satisfied by a requested "
"external library",
name,
component.name,
)
continue
if name in bundled_names or is_lib_ignored(name, lib_ignore):
continue
if _url_or_none(dep.get("version")) is not None:
# A URL names one specific source; never add the bundled copy
continue
if dep.get("owner") or not _provided(name):
# Only owner-less framework-tree names take the bundled
# copy (PIO's process_dependencies); the walk reports drops
continue
try:
# framework=None: the walk already warned for non-platform
# causes; debug keeps one fault from warning twice (pinned
# by test_nonplatform_rejection_warns_once_through_real_converter)
check_library_data(dep, pio_platform, None)
except IncompatiblePlatform as err:
# A knowing skip (platform filter), not a broken promise
knowingly_skipped.add(name)
_LOGGER.debug("Skip bundled candidate %s: %s", name, err)
continue
except InvalidLibrary as err:
# Malformed manifest data never counts as satisfied; the
# walk owns the warning (see the warns-once test above)
_LOGGER.debug("Skip malformed bundled candidate %s: %s", name, err)
continue
# Deferred: a later manifest name may satisfy this
pending_bundled.setdefault(name)
def _emit(component: ConvertedLibrary) -> None:
apply_extra_script(
component, board_mcu=lambda: board_mcu, pio_platform=pio_platform
)
_assert_tree_has_code(
component.get_require_name(),
component.source_dir,
"the download may be incomplete (run 'esphome clean-all')",
)
if isinstance(manifest_name := component.data.get("name"), str):
converted_manifest_names.add(manifest_name)
lib = _library_info(
component.get_require_name(), component.source_dir, component.data
)
# Extra-script LINKFLAGS travel outside build.flags; dropping
# them would link wrong with no stated cause
lib.link_flags.extend(
component.data.get(ESPHOME_DATA_KEY, {}).get(
ESPHOME_DATA_LINK_FLAGS_KEY, []
)
)
converted.append(lib)
_add_bundled_dependencies(component)
backend = LibraryBackend(
platform=pio_platform,
framework="arduino",
emit=_emit,
cache_key=cache_key,
# The walk must not resolve bundled names from the registry;
# _add_bundled_dependencies adds them after emit
provides=_provided,
)
if external:
convert_libraries(external, backend)
for name in pending_bundled:
if name in converted_manifest_names:
# The converted library is this one; the bundled copy would
# double the archive. Warn like the external_short_names twin.
_LOGGER.warning(
"Dependency %s is assumed satisfied by a converted library's "
"manifest name; the bundled copy is not added",
name,
)
continue
bundled_names.add(name)
bundled.append(_bundled_library(framework_path, name))
_check_unfulfilled_provides(
backend.provided_requests,
bundled_names
| converted_manifest_names
| external_short_names
| knowingly_skipped,
final_dep_names,
)
return bundled + converted
+5 -7
View File
@@ -19,11 +19,10 @@ from typing import NamedTuple
from esphome.build_helpers.ccache import ccache_defaults_env
from esphome.build_helpers.ninja import find_ninja
from esphome.build_helpers.pch import ccache_pch_env
from esphome.build_helpers.tools_cache import ARDUINO8266_TOOLS_CACHE, tools_cache_path
from esphome.core import EsphomeError, Version
from esphome.framework_helpers import str_to_lst_of_str
from esphome.platformio.registry import install_packages, prefetch_packages
from esphome.platformio.registry import install_package, prefetch_packages
FRAMEWORK_PACKAGE = "framework-arduinoespressif8266"
TOOLCHAIN_PACKAGE = "toolchain-xtensa"
@@ -121,9 +120,10 @@ def check_and_install(framework_version: Version) -> InstalledPaths:
("bin", "xtensa-lx106-elf"),
),
)
# Fetch both archives at once; the install verifies and extracts them
# Fetch both archives at once; the installs below verify and extract
prefetch_packages([spec[:4] for spec in specs], downloads_dir)
install_packages(specs, downloads_dir)
for name, version, dest, mirrors, expect in specs:
install_package(name, version, dest, mirrors, downloads_dir, expect=expect)
return InstalledPaths(
framework=framework_path, toolchain=toolchain_path, ninja=ninja_path
)
@@ -161,6 +161,4 @@ def ccache_env(ccache: str | None) -> dict[str, str]:
"""
if ccache is None:
return {}
env = ccache_defaults_env(get_arduino8266_tools_path() / "ccache")
env.update(ccache_pch_env())
return env
return ccache_defaults_env(get_arduino8266_tools_path() / "ccache")
-329
View File
@@ -1,329 +0,0 @@
"""Native Arduino ESP8266 build driver (the PlatformIO ``run`` equivalent)."""
from __future__ import annotations
import json
import logging
from pathlib import Path
import subprocess
from typing import Any
from esphome.arduino8266 import framework
from esphome.build_helpers.ccache import resolve_ccache_path
from esphome.const import (
CONF_COMPILE_PROCESS_LIMIT,
CONF_ESPHOME,
KEY_CORE,
KEY_FRAMEWORK_VERSION,
)
from esphome.core import CORE, EsphomeError
from esphome.helpers import write_file_if_changed
from esphome.types import ConfigType
_LOGGER = logging.getLogger(__name__)
# ESP8266 user RAM (matches upload.maximum_ram_size in every board manifest)
_MAX_RAM_SIZE = 81920
def _warn_ignored_platformio_options() -> None:
"""Warn for component-added platformio options the native build drops.
The consumed set is exported by core/config.py next to the routing that
stores these options, so the two cannot drift; YAML upload_speed never
reaches CORE.platformio_options here.
"""
from esphome.core.config import NATIVE_ARDUINO_CONSUMED_PIO_OPTIONS
consumed = NATIVE_ARDUINO_CONSUMED_PIO_OPTIONS
for key in sorted(CORE.platformio_options or {}):
if key not in consumed:
_LOGGER.warning(
"platformio_options->%s is ignored when building with the "
"native 'arduino' toolchain",
key,
)
_RAM_SECTIONS = (".data", ".rodata", ".bss")
_FLASH_SECTIONS = (".irom0.text", ".text", ".text1", ".data", ".rodata")
def get_build_dir() -> Path:
return CORE.relative_pioenvs_path(CORE.name)
def get_elf_path() -> Path:
return get_build_dir() / "firmware.elf"
def _toolchain_tool(name: str) -> Path:
return framework.toolchain_tool(framework.get_toolchain_path(), name)
def get_factory_firmware_path() -> Path:
"""The image to serial-flash at 0x0 (same bytes as firmware.bin: the
8266 factory copy exists for artifact-contract parity, not content)."""
return get_build_dir() / "firmware.factory.bin"
def get_addr2line_path() -> Path:
return _toolchain_tool("addr2line")
def get_objdump_path() -> Path:
return _toolchain_tool("objdump")
def get_readelf_path() -> Path:
return _toolchain_tool("readelf")
def run_compile(config: ConfigType, verbose: bool) -> int:
from esphome.build_gen import arduino8266 as build_gen
_warn_ignored_platformio_options()
paths = framework.check_and_install(CORE.data[KEY_CORE][KEY_FRAMEWORK_VERSION])
# Resolved once per build: the resolution probes PATH and spawns the
# runnability check, and three consumers need the same answer
ccache = resolve_ccache_path()
ninja_changed = build_gen.write_project(paths, ccache)
build_dir = get_build_dir()
env = framework.get_build_env(paths.toolchain, ccache)
# Regenerate the compile DB before the build (a pure function of
# build.ninja); skip only when it is at least as fresh as build.ninja
# (an interrupted previous run may have rewritten the manifest without
# regenerating the DB).
compdb = build_dir / "compile_commands.json"
compdb_stamp = build_dir / ".compile_commands.stamp"
ninja_file = build_dir / "build.ninja"
# Freshness rides a stamp: the DB itself is written through
# write_file_if_changed (its mtime feeds get_idedata's cache), so a
# regeneration with identical content would stay "stale" forever
if (
ninja_changed
or not compdb.is_file()
or not compdb_stamp.is_file()
or compdb_stamp.stat().st_mtime < ninja_file.stat().st_mtime
):
_write_compile_commands(paths.ninja, build_dir, env)
compdb_stamp.touch()
cmd = [str(paths.ninja)]
if verbose:
cmd.append("-v")
if jobs := config[CONF_ESPHOME].get(CONF_COMPILE_PROCESS_LIMIT):
cmd += ["-j", str(jobs)]
# Explicit targets, not the default statement: a generator defect that
# drops them fails loudly with "unknown target" instead of a green
# no-op run that leaves stale artifacts in place
targets = ["firmware.factory.bin", "firmware.ota.bin"]
cmd += targets
# A dry-run probe keeps a no-op rebuild quiet: ninja would only print
# "no work to do". A freshly rewritten manifest all but guarantees work,
# so skip the probe (and its full stat pass) on that path. cwd instead
# of -C also drops the "Entering directory" banner on real builds.
skip_build = False
if not ninja_changed:
probe = subprocess.run(
[str(paths.ninja), "-n", *targets],
cwd=build_dir,
env=env,
capture_output=True,
text=True,
check=False,
close_fds=False,
)
if probe.stderr.strip():
# A load-time diagnostic (e.g. "multiple rules generate X")
# flags a generator bug; the skip branch would otherwise
# swallow it forever
_LOGGER.warning("ninja: %s", probe.stderr.strip())
if probe.returncode != 0:
# An unknown target here is the defective-manifest case; fall
# through to the real build so the error prints attributably
_LOGGER.debug("ninja probe failed; running the full build")
skip_build = probe.returncode == 0 and "no work to do" in probe.stdout
if skip_build:
_LOGGER.debug("ninja: nothing to rebuild")
else:
_LOGGER.debug("Running: %s", " ".join(cmd))
rc = subprocess.run(
cmd, cwd=build_dir, env=env, check=False, close_fds=False
).returncode
if rc != 0:
return rc
# ninja already refused a manifest missing the explicit targets above;
# existence covers the remaining hole (a rule that ran but wrote
# elsewhere). The factory/ota copies are what upload and OTA consume.
build_dir_artifacts = (
get_elf_path(),
build_dir / "firmware.bin",
get_factory_firmware_path(),
build_dir / "firmware.ota.bin",
)
for artifact in build_dir_artifacts:
if not artifact.is_file():
_LOGGER.error("Build produced no %s", artifact)
return 1
if not _print_size_summary(build_dir, paths):
# The cause was already warned; name the consequence so a build
# contributing no RAM/Flash metric is visible to CI harnesses
_LOGGER.warning("Firmware size summary unavailable for this build")
from esphome.build_helpers.idedata import warn_if_idedata_missing
warn_if_idedata_missing(lambda: get_idedata(ccache))
return 0
def _write_compile_commands(
ninja_path: Path, build_dir: Path, env: dict[str, str]
) -> None:
compdb = build_dir / "compile_commands.json"
result = subprocess.run(
[str(ninja_path), "-C", str(build_dir), "-t", "compdb", "c", "cxx", "asm"],
env=env,
capture_output=True,
text=True,
check=False,
close_fds=False,
)
if result.returncode != 0:
# Drop any stale database so consumers (IDE integration, clang-tidy,
# the memory analyzer) can't silently read outdated data.
compdb.unlink(missing_ok=True)
raise EsphomeError(f"Could not generate compile_commands.json: {result.stderr}")
try:
entries = json.loads(result.stdout)
except ValueError as err:
compdb.unlink(missing_ok=True)
raise EsphomeError(
f"ninja produced an unparsable compile database: {err} "
f"(output starts {result.stdout[:120]!r})"
) from err
if not entries:
# compdb exits 0 with [] for unknown rule names; a renamed compile
# rule must fail the build, not silently strand every consumer
compdb.unlink(missing_ok=True)
raise EsphomeError(
"ninja produced an empty compile database; the generator's rule "
"names no longer match"
)
# write_file_if_changed keeps the mtime stable on no-op builds so the
# idedata cache in get_idedata() stays valid.
write_file_if_changed(compdb, result.stdout)
def _parse_app_size(build_dir: Path, paths: framework.InstalledPaths) -> int | None:
"""Read the app flash budget (irom0_0_seg length) from the linker script."""
from esphome.build_gen.arduino8266 import get_flash_ld_path
from esphome.components.esp8266.build_surgery import segment_length
# Warnings, not debug: without the app size the Flash summary line is
# dropped and CI's memory-impact extraction loses its flash metric.
ld_path = get_flash_ld_path(build_dir, paths)
try:
ld_text = ld_path.read_text(encoding="utf-8")
except (OSError, UnicodeDecodeError) as err:
# UnicodeDecodeError: a truncated/corrupt script must degrade to
# the same warning, never abort an already-linked build
_LOGGER.warning("Cannot read linker script for the Flash summary: %s", err)
return None
app_size = segment_length(ld_text, "irom0_0_seg")
if app_size is None:
_LOGGER.warning("irom0_0_seg not found in %s; skipping Flash summary", ld_path)
return None
if app_size == 0:
_LOGGER.warning(
"irom0_0_seg has zero length in %s; skipping Flash summary", ld_path
)
return None
return app_size
def _print_size_summary(build_dir: Path, paths: framework.InstalledPaths) -> bool:
"""Print the PlatformIO-shaped RAM/Flash lines; False when skipped.
The exact shape (including the bar) is parsed by
``script/ci_memory_impact_extract.py``; ``print_size_line`` matches it.
"""
from esphome.build_helpers.size_summary import print_size_line
size_tool = _toolchain_tool("size")
try:
result = subprocess.run(
[str(size_tool), "-A", "-d", str(get_elf_path())],
capture_output=True,
text=True,
check=False,
close_fds=False,
)
except OSError as err:
# The summary is a bonus artifact like idedata; a truncated
# toolchain extraction must not discard an already-linked build
_LOGGER.warning("Could not summarize firmware size: %s", err)
return False
if result.returncode != 0:
_LOGGER.warning("Could not summarize firmware size: %s", result.stderr)
return False
sections: dict[str, int] = {}
for line in result.stdout.splitlines():
parts = line.split()
if len(parts) >= 2 and parts[0].startswith("."):
try:
sections[parts[0]] = int(parts[1])
except ValueError:
# An unparsed RAM/Flash section trips the missing-sections
# guard below, so no total is built on a dropped value
_LOGGER.warning("Unparsable size output for section %s", parts[0])
if missing := set(_RAM_SECTIONS + _FLASH_SECTIONS) - set(sections):
# A defaulted 0 would print a confidently wrong total for CI's metric
_LOGGER.warning(
"Size output is missing section(s) %s; skipping the size summary",
", ".join(sorted(missing)),
)
return False
# Resolve the flash budget before printing anything: a RAM line without
# its Flash line would let CI's memory-impact extraction sum the two
# metrics over different build counts (_parse_app_size already warned).
app_size = _parse_app_size(build_dir, paths)
if not app_size:
return False
ram = sum(sections[s] for s in _RAM_SECTIONS)
flash = sum(sections[s] for s in _FLASH_SECTIONS)
print_size_line("RAM", ram, _MAX_RAM_SIZE)
print_size_line("Flash", flash, app_size)
return True
# Sentinel: "resolve for me"; None is a real value meaning disabled.
_CCACHE_UNRESOLVED: Any = object()
def get_idedata(ccache: str | None = _CCACHE_UNRESOLVED) -> dict | None:
"""Derive idedata from the build's compile_commands.json.
Same contract as ``espidf.toolchain.get_idedata``: the fields IDE
integrations, clang-tidy, and the memory analyzer expect.
"""
from esphome.build_helpers.idedata import load_or_build_idedata
if ccache is _CCACHE_UNRESOLVED:
# Deliberately uncached: env/PATH can change between builds in a
# long-lived host process
ccache = resolve_ccache_path()
return load_or_build_idedata(
get_build_dir() / "compile_commands.json",
get_elf_path(),
# Suffixed so a platformio->arduino->platformio round trip on one
# config never serves the other toolchain's cache shape
CORE.relative_internal_path("idedata", f"{CORE.name}.arduino.json"),
# The compile DB's commands carry the same ccache prefix the ninja
# rules were generated with
launcher=str(ccache) if ccache else None,
)
File diff suppressed because it is too large Load Diff
-118
View File
@@ -1,118 +0,0 @@
"""Tiny cross-platform build steps invoked from the generated ninja file.
Plain script (not ``python -m``): it runs from ninja with whatever Python
started esphome and must not depend on the package being importable.
Subcommands:
ar <ar-binary> <archive> <rspfile> remove stale archive, then ``ar rcs``
copy <src> <dst> copy a file
touch <path> create/update a stamp file
The ar rspfile carries one object path per line (the generating rule must
use ``$in_newline``, never ``$in``).
"""
from pathlib import Path
import shutil
import subprocess
import sys
def _read_rspfile(rspfile: str) -> list[str]:
r"""The object paths listed in ``rspfile``, unquoted.
GNU ar treats backslashes in response files as escapes (corrupts
Windows paths), so the caller expands the list into argv; strip the
simple surrounding quote ninja adds to special paths, then undo
ninja's POSIX escape for an embedded quote ('a'\\''b.o' -> a'b.o).
"""
return [
line[1:-1].replace("'\\''", "'")
if len(line) >= 2 and line[0] == line[-1] and line[0] in "'\""
else line
for line in Path(rspfile).read_text(encoding="utf-8").splitlines()
if line
]
def _run_ar(ar: str, archive: str, rspfile: str) -> int:
# Remove first: ``ar rcs`` replaces members but never drops ones whose
# source was removed from the build, which would leak stale objects.
Path(archive).unlink(missing_ok=True)
objects = _read_rspfile(rspfile)
if not objects:
# An empty archive would "succeed" here and fail far away at link
print(f"ar: no objects listed in {rspfile} for {archive}", file=sys.stderr)
return 1
# Batch by argv length: expanding the rspfile gives back the Windows
# 32767-char command-line limit it existed to avoid. "rcs" creates,
# "qs" appends; the s keeps the symbol index explicit on every ar.
op = "rcs"
ok = False
try:
while objects:
batch = [objects.pop(0)]
batch_len = len(batch[0])
while objects and batch_len + len(objects[0]) < 25000:
batch_len += len(objects[0]) + 1
batch.append(objects.pop(0))
rc = subprocess.run(
[ar, op, archive, *batch], check=False, close_fds=False
).returncode
if rc != 0:
return rc
op = "qs"
ok = True
return 0
finally:
if not ok:
# Any failure (bad exit, missing ar binary, interrupt) must not
# leave a truncated archive behind
Path(archive).unlink(missing_ok=True)
def _run_copy(src: str, dst: str) -> int:
try:
shutil.copyfile(src, dst)
except OSError as err:
# Never leave a partially written output (e.g. a firmware image);
# SameFileError means dst IS src, where unlinking destroys the input
if not isinstance(err, shutil.SameFileError):
Path(dst).unlink(missing_ok=True)
print(f"copy: {src} -> {dst} failed: {err}", file=sys.stderr)
return 1
return 0
def _run_touch(path: str) -> int:
try:
Path(path).touch()
except OSError as err:
print(f"touch: {path} failed: {err}", file=sys.stderr)
return 1
return 0
# mode -> (handler, expected operand count); surplus argv means a
# mis-specified ninja rule and must error, not silently drop operands
_MODES = {"ar": (_run_ar, 3), "copy": (_run_copy, 2), "touch": (_run_touch, 1)}
def main() -> int:
mode = sys.argv[1] if len(sys.argv) > 1 else ""
if entry := _MODES.get(mode):
handler, argc = entry
args = sys.argv[2:]
if len(args) != argc:
print(
f"build_tool {mode}: expected {argc} arguments, got {len(args)}",
file=sys.stderr,
)
return 1
return handler(*args)
print(f"unknown build_tool mode: {mode}", file=sys.stderr)
return 1
if __name__ == "__main__": # pragma: no cover
sys.exit(main())
+1 -57
View File
@@ -4,14 +4,6 @@ import json
import logging
from pathlib import Path
from esphome.build_helpers import pch
from esphome.build_helpers.pch import (
PCH_DEFAULT_HEADERS,
PCH_HEADER_NAME,
mark_pch_emitted,
pch_enabled,
pch_header_text,
)
from esphome.components.esp32 import (
get_esp32_variant,
get_excluded_builtin_components,
@@ -287,47 +279,7 @@ idf_component_register(
target_link_options(${{COMPONENT_LIB}} PUBLIC
{link_opts_str}
)
{_pch_cmake()}"""
def _pch_cmake() -> str:
"""Consumer block for the component CMakeLists. Baked at generation:
a strict-knob flip takes effect on the next esphome compile; a
hand-run idf.py keeps the old one."""
return pch.pch_cmake_consumer("${COMPONENT_LIB}", "${app_sources}")
def prepare_pch() -> None:
"""Build the .gch right before ninja, after every reconfigure, so the
compile_commands.json flags and the sdkconfig are the settled ones."""
if not pch_enabled():
# Self-cleaning escape hatch: drop any previously built .gch
pch.discard_pch(CORE.relative_build_path("build"))
pch.pch_disabled_degraded()
return
sdkconfig_path = CORE.relative_build_path(f"sdkconfig.{CORE.name}")
try:
sdkconfig = sdkconfig_path.read_text(encoding="utf-8")
except OSError as err:
# Fail closed: the sdkconfig is the .sum's only config identity for
# sdkconfig.h-only options; a stand-in marker would collide
_LOGGER.warning(
"Could not read %s; compiling without the pch: %s", sdkconfig_path, err
)
pch.discard_pch(CORE.relative_build_path("build"))
pch.pch_degraded(f"sdkconfig unreadable: {err}")
return
pch.prepare_pch(
CORE.relative_build_path("build"),
PCH_DEFAULT_HEADERS,
(
str(idf_version()),
CORE.cpp_standard or "",
sdkconfig,
*get_project_compile_flags(),
*get_project_cxx_compile_flags(),
),
)
"""
def write_project(
@@ -349,14 +301,6 @@ def write_project(
get_component_cmakelists(),
)
if pch_enabled():
write_file_if_changed(
CORE.relative_build_path("build", PCH_HEADER_NAME),
pch_header_text(PCH_DEFAULT_HEADERS),
)
# Consumers carry the -include; gate the ccache relaxation on it
mark_pch_emitted()
# Snapshot the exclusion set so has_outdated_files() can trigger a
# discovery reconfigure when it changes. Excluded components never
# register in project_description.json, so re-including one (e.g. a
+3 -21
View File
@@ -21,13 +21,12 @@ def _ccache_runs(ccache: str) -> bool:
)
def parse_enable_env(name: str, strict: bool = False) -> bool | None:
def parse_enable_env(name: str) -> bool | None:
"""Strictly parse an on/off environment knob; None when unset or invalid.
``bool(str)`` truthiness would flip ``no``/``off`` to enabled, so only
1/true/yes/on and 0/false/no/off count; anything else warns and reads
as unset so the caller's default policy applies — or raises when
``strict`` (a typo must not silently disable a CI gate).
as unset so the caller's default policy applies.
"""
raw = os.environ.get(name)
if raw is None:
@@ -40,10 +39,6 @@ def parse_enable_env(name: str, strict: bool = False) -> bool | None:
return True
if lowered in FALSY_ENV_STRINGS:
return False
if strict:
from esphome.core import EsphomeError
raise EsphomeError(f"Unrecognized {name}={raw!r}; use 1 or 0")
_LOGGER.warning("Ignoring unrecognized %s=%r; use 1 or 0", name, raw)
return None
@@ -92,19 +87,6 @@ def ccache_defaults_env(cache_dir: Path) -> dict[str, str]:
"CCACHE_DIR": str(cache_dir),
"CCACHE_NOHASHDIR": "true",
"CCACHE_DEPEND": "1",
# A user value wins via the filter below
"CCACHE_BASEDIR": effective_ccache_basedir(),
"CCACHE_BASEDIR": str(Path(CORE.build_path).resolve()),
}
return {k: v for k, v in defaults.items() if k not in os.environ}
def effective_ccache_basedir() -> str:
"""The prefix ccache rewrites out of hashed paths: a user CCACHE_BASEDIR
wins, else the resolved build path (matching ccache_defaults_env)."""
from esphome.core import CORE
raw = os.environ.get("CCACHE_BASEDIR")
if raw is not None and Path(raw).is_absolute() and len(Path(raw).parts) > 1:
return raw
# Unset or degenerate ("", "/", relative): fall back to the build path
return str(Path(CORE.build_path).resolve())
+15 -70
View File
@@ -11,7 +11,6 @@ consumers (IDE integration, clang-tidy) expect:
from __future__ import annotations
from collections.abc import Callable
import json
import logging
import os
@@ -22,8 +21,6 @@ import subprocess
from esphome.core import EsphomeError
from esphome.helpers import write_file
_LOGGER = logging.getLogger(__name__)
# Everything idedata generation may raise after a successful link; idedata
# is a bonus artifact, so consumers warn instead of failing the build
IDEDATA_BEST_EFFORT_ERRORS = (
@@ -34,36 +31,13 @@ IDEDATA_BEST_EFFORT_ERRORS = (
ValueError,
)
_LOGGER = logging.getLogger(__name__)
def warn_if_idedata_missing(get_idedata: Callable[[], dict | None]) -> None:
"""Run an idedata generator, downgrading any failure to a warning.
Shared by the native backends: the firmware already built, so a missing
or broken idedata must not fail a successful build.
"""
try:
if get_idedata() is None:
_LOGGER.warning("No idedata was generated for this build")
except IDEDATA_BEST_EFFORT_ERRORS as err:
_LOGGER.warning(
"Could not generate idedata: %s (IDE, clang-tidy, and "
"memory-analysis data will be unavailable for this build)",
err,
)
if isinstance(err, (EsphomeError, OSError)):
# Routine environmental failures keep the detail at debug
_LOGGER.debug("Idedata failure detail", exc_info=True)
else:
# LookupError/ValueError/RuntimeError smell like a parsing bug;
# a permanently masked traceback would hide it on every build
_LOGGER.warning("Idedata failure detail", exc_info=True)
# C++ translation-unit suffixes.
CXX_SOURCE_SUFFIXES = (".cpp", ".cc", ".cxx")
# C++ translation-unit suffixes used to identify ESPHome source files.
_CXX_SUFFIXES = (".cpp", ".cc")
# Suffixes of input/output files that appear bare on the command line (and so
# must not be mistaken for compiler flags).
_INPUT_FILE_SUFFIXES = (*CXX_SOURCE_SUFFIXES, ".c", ".o", ".S", ".s")
_INPUT_FILE_SUFFIXES = (*_CXX_SUFFIXES, ".c", ".o", ".S", ".s")
# Path marker identifying an ESPHome source translation unit.
_ESPHOME_SRC_MARKER = "/src/esphome/"
@@ -72,11 +46,11 @@ def _is_esphome_src(file: str) -> bool:
"""Whether ``file`` is an ESPHome C++ translation unit; normalized to
``/`` first since Windows compile DBs use backslashes."""
return _ESPHOME_SRC_MARKER in file.replace("\\", "/") and file.endswith(
CXX_SOURCE_SUFFIXES
_CXX_SUFFIXES
)
def split_command(command: str) -> list[str]:
def _split_command(command: str) -> list[str]:
r"""Tokenize a compile_commands.json / response-file command string.
On Windows, tokenize per Windows ``argv`` rules via ``CommandLineToArgvW``.
@@ -112,7 +86,7 @@ def split_command(command: str) -> list[str]:
ctypes.windll.kernel32.LocalFree(argv)
def expand_response_files(tokens: list[str], directory: Path) -> list[str]:
def _expand_response_files(tokens: list[str], directory: Path) -> list[str]:
"""Inline any ``@response-file`` arguments (paths relative to ``directory``).
GCC response files embed flags that must be expanded so GCC-only flags
@@ -127,8 +101,8 @@ def expand_response_files(tokens: list[str], directory: Path) -> list[str]:
rf = directory / rf
try:
out.extend(
expand_response_files(
split_command(rf.read_text(encoding="utf-8")), directory
_expand_response_files(
_split_command(rf.read_text(encoding="utf-8")), directory
)
)
continue
@@ -147,7 +121,7 @@ def _pick_entry(entries: list[dict]) -> dict:
if _is_esphome_src(entry["file"]):
return entry
for entry in entries:
if entry["file"].endswith(CXX_SOURCE_SUFFIXES):
if entry["file"].endswith(_CXX_SUFFIXES):
return entry
raise ValueError("no C++ translation unit found in compile_commands.json")
@@ -157,25 +131,16 @@ def _pick_entry(entries: list[dict]) -> dict:
_LAUNCHER_STEMS = frozenset({"ccache", "sccache", "distcc", "icecc", "buildcache"})
def is_launcher(token: str) -> bool:
def _is_launcher(token: str) -> bool:
return Path(token).stem.lower() in _LAUNCHER_STEMS
def is_joined_include(tok: str) -> bool:
"""The joined ``-includefoo.h`` spelling; excludes clang's -include-pch."""
return (
tok.startswith("-include")
and tok != "-include"
and not tok.startswith("-include-")
)
def parse_entry(
entry: dict, launcher: str | None = None
) -> tuple[str, list[str], list[str], list[str]]:
"""Parse one compile_commands entry -> (cxx_path, defines, includes, cxx_flags)."""
directory = Path(entry["directory"])
tokens = expand_response_files(split_command(entry["command"]), directory)
tokens = _expand_response_files(_split_command(entry["command"]), directory)
def _include(raw: str) -> str:
# Resolve against the entry's ``directory`` so cached idedata works
@@ -191,7 +156,7 @@ def parse_entry(
if not tokens:
# An empty command, or one that was only the launcher; fail by name
raise ValueError(f"empty compile command for {entry.get('file')}")
if is_launcher(tokens[0]) and len(tokens) > 1 and not tokens[1].startswith("-"):
if _is_launcher(tokens[0]) and len(tokens) > 1 and not tokens[1].startswith("-"):
# Stale DB built with a launcher this run no longer configures; the
# real compiler is the next token
_LOGGER.warning("Stripping unconfigured launcher %s", tokens[0])
@@ -204,23 +169,11 @@ def parse_entry(
defines: list[str] = []
includes: list[str] = []
cxx_flags: list[str] = []
unresolved_force_includes: list[str] = []
it = iter(tokens[1:])
for tok in it:
if tok in ("-c", "-o"):
next(it, None) # drop the flag and its argument (input/output)
elif tok == "-include" or is_joined_include(tok):
# Re-anchor only names next to the compile (the pch); a name
# meant for the -I chain must stay untouched
raw = next(it, "") if tok == "-include" else tok[len("-include") :]
if not raw:
_LOGGER.warning("Dropping -include with no argument")
elif Path(resolved := _include(raw)).is_file():
cxx_flags.extend(("-include", resolved))
else:
unresolved_force_includes.append(raw)
cxx_flags.extend(("-include", raw))
elif tok.startswith("-D"):
# ``.strip()`` handles tokens like ``-D CONFIGURED=1`` (a single
# quoted arg with a space after -D) that some flags arrive as.
@@ -239,14 +192,6 @@ def parse_entry(
pass # input/output files
else:
cxx_flags.append(tok)
for raw in unresolved_force_includes:
# A deleted build artifact would otherwise surface only downstream
if not any((Path(inc) / raw).is_file() for inc in includes):
_LOGGER.warning(
"-include %s found neither next to the compile nor on the "
"include path; cached idedata may not resolve it",
raw,
)
return cxx_path, defines, includes, cxx_flags
@@ -311,7 +256,7 @@ def _cache_usable(cached: object) -> bool:
if not isinstance(cached, dict) or "cc_path" not in cached:
return False
cxx_path = cached.get("cxx_path")
if not isinstance(cxx_path, str) or is_launcher(cxx_path):
if not isinstance(cxx_path, str) or _is_launcher(cxx_path):
return False
includes = cached.get("includes")
return isinstance(includes, dict) and isinstance(includes.get("build"), list)
@@ -359,7 +304,7 @@ def load_or_build_idedata(
def reject_launcher_compiler(cxx_path: str) -> None:
"""Reject a compile DB naming a launcher (ccache) as the compiler; it
must never be probed, cached, or consumed."""
if is_launcher(cxx_path):
if _is_launcher(cxx_path):
raise EsphomeError(
f"compile_commands.json names the launcher {cxx_path} as the "
"compiler; the compile database is unusable"
-624
View File
@@ -1,624 +0,0 @@
"""Shared precompiled-header policy for the build backends.
The prefix either mirrors the TUs' own force-includes (ESP8266) or is a
curated core-header set (ESP-IDF). ``esphome: includes:`` sources receive
it too; Arduino.h visibility there is intended (esphome#8693).
"""
from __future__ import annotations
from collections.abc import Callable, Iterable
from contextlib import suppress
from dataclasses import dataclass
import hashlib
import json
import logging
import os
from pathlib import Path
import posixpath
import re
import stat
import subprocess
from esphome.build_helpers.ccache import effective_ccache_basedir, parse_enable_env
from esphome.build_helpers.idedata import (
CXX_SOURCE_SUFFIXES,
expand_response_files,
is_launcher,
split_command,
)
_DOMAIN = "pch"
@dataclass
class _PCHData:
emitted: bool = False
def _pch_data() -> _PCHData:
from esphome.core import CORE
if _DOMAIN not in CORE.data:
CORE.data[_DOMAIN] = _PCHData()
return CORE.data[_DOMAIN]
def mark_pch_emitted() -> None:
"""Record that this build's consumers reference the pch."""
_pch_data().emitted = True
_LOGGER = logging.getLogger(__name__)
# The header and its .gch/.sum sidecars live in the build directory.
PCH_HEADER_NAME = "esphome_pch.h"
# Every artifact the pch machinery can leave behind, for cleanup.
PCH_ARTIFACT_NAMES = (
PCH_HEADER_NAME,
f"{PCH_HEADER_NAME}.gch",
f"{PCH_HEADER_NAME}.gch.sum",
f"{PCH_HEADER_NAME}.gch.failed",
)
# The core defines header every backend anchors its prefix on.
PCH_CORE_HEADER = "esphome/core/defines.h"
# Guarded curated-prefix wrapper for PlatformIO backends without framework
# force-includes (host, esp32); folded by the pch script via build_src_flags.
PCH_PREFIX_HEADER = "esphome/core/pch_prefix.h"
# Prefix-header contents for backends that inject a curated set (rather
# than mirroring the TUs' own force-includes), defines.h first so USE_*
# macros exist for the rest. Deliberately hard-coded: frequency-derived
# sets measured no better and kept selecting headers that cannot compile
# standalone (X-macro, platform-variant). Every entry must be safe to
# include first in an empty TU. Caveat: application.h/automation.h become
# ambiently visible, so a TU missing those #includes still builds on such
# backends; ESPHOME_PCH_ENABLE=0 restores the strict view.
PCH_DEFAULT_HEADERS = (
PCH_CORE_HEADER,
"esphome/core/component.h",
"esphome/core/helpers.h",
"esphome/core/log.h",
"esphome/core/application.h",
"esphome/core/automation.h",
)
# ccache cannot hash through a .gch; CCACHE_PCH_EXTSUM makes it hash the
# .sum sidecar instead of the .gch bytes, which are not reproducible.
# Keep in sync with the literals in platformio/pch.py.script.
_CCACHE_PCH_ENV = {
"CCACHE_SLOPPINESS": "pch_defines,time_macros",
"CCACHE_PCH_EXTSUM": "true",
}
# Both include forms: an angle include resolving under src/ must enter the
# digest too; ones that do not resolve simply end the walk
# Compiler failures that clear on their own must not latch the .failed marker
_TRANSIENT_ERRORS = ("No space left", "Cannot allocate", "Resource temporarily")
_INCLUDE_RE = re.compile(rb'^\s*#\s*include\s+["<]([^">]+)[">]', re.MULTILINE)
def pch_enabled() -> bool:
"""Precompiled-header knob: default on, ``ESPHOME_PCH_ENABLE=0`` opts out."""
return parse_enable_env("ESPHOME_PCH_ENABLE") is not False
def pch_strict() -> bool:
"""CI knob: ``ESPHOME_PCH_STRICT=1`` turns pch degrade paths fatal.
A set-but-unrecognized value raises: a typo must not silently turn
the gate into a no-op that proves nothing.
"""
return parse_enable_env("ESPHOME_PCH_STRICT", strict=True) is True
def pch_degraded(reason: str) -> None:
"""Every degrade path funnels through here; strict mode raises."""
if pch_strict():
from esphome.core import EsphomeError
raise EsphomeError(f"ESPHOME_PCH_STRICT: {reason}")
def pch_disabled_degraded() -> None:
"""Strict CI must not read "no pch at all" as success."""
pch_degraded("pch disabled by ESPHOME_PCH_ENABLE")
def pch_probe_tail(source: str = "-") -> list[str]:
"""The syntax-only compile shared by the probe and its baseline."""
return ["-fsyntax-only", "-x", "c++", source]
def pch_probe_args(header: str, source: str = "-") -> list[str]:
"""Flags that load-check a built .gch via a syntax-only compile.
Rejection must be a nonzero exit (never just a wording match), so the
invalid-pch class is always escalated. ``source`` defaults to stdin
(host independent); the ninja probe edge passes a real file.
"""
return [
"-Winvalid-pch",
"-Werror=invalid-pch",
"-include",
header,
*pch_probe_tail(source),
]
def pch_consumer_escalation() -> str:
"""Consumer-side invalid-pch flag: strict reds the build on rejection
(per-process, so the probe alone cannot prove the consumers)."""
return "-Werror=invalid-pch" if pch_strict() else "-Wno-error=invalid-pch"
def pch_cmake_consumer(target: str, sources_var: str) -> str:
"""Emit the CMake block making ``target``'s C++ sources consume the
pch; empty when disabled. OBJECT_DEPENDS is on the header, not the
.gch (pch-baked headers drop out of TU depfiles); the -include stays
relative — an absolute path would poison ccache keys."""
if not pch_enabled():
return ""
escalation = pch_consumer_escalation()
return f"""
# ESPHome precompiled header (see esphome/build_helpers/pch.py).
# The touch keeps OBJECT_DEPENDS satisfiable when the build system itself
# wiped the build dir after the header was written (west --pristine)
if(NOT EXISTS "${{CMAKE_BINARY_DIR}}/{PCH_HEADER_NAME}")
file(TOUCH "${{CMAKE_BINARY_DIR}}/{PCH_HEADER_NAME}")
endif()
target_compile_options({target} PRIVATE
"$<$<COMPILE_LANGUAGE:CXX>:-Winvalid-pch>"
"$<$<COMPILE_LANGUAGE:CXX>:{escalation}>"
"$<$<COMPILE_LANGUAGE:CXX>:-include>"
"$<$<COMPILE_LANGUAGE:CXX>:{PCH_HEADER_NAME}>"
)
set_source_files_properties({sources_var} PROPERTIES
OBJECT_DEPENDS "${{CMAKE_BINARY_DIR}}/{PCH_HEADER_NAME}")
"""
def ccache_pch_env() -> dict[str, str]:
"""Settings ccache needs to cache compiles that consume the .gch;
empty unless this build actually emitted one. User-set values win.
Native backends export these process-wide; only time_macros affects
non-pch TUs."""
if not (pch_enabled() and _pch_data().emitted):
return {}
extsum = os.environ.get("CCACHE_PCH_EXTSUM")
if extsum is not None and extsum.strip().lower() not in ("1", "true", "yes", "on"):
# ccache then hashes the non-reproducible .gch bytes: permanent misses
_LOGGER.warning("CCACHE_PCH_EXTSUM=%s disables pch caching", extsum)
env = {k: v for k, v in _CCACHE_PCH_ENV.items() if k not in os.environ}
user_sloppiness = os.environ.get("CCACHE_SLOPPINESS")
if user_sloppiness is not None and (
missing := [
t
for t in ("pch_defines", "time_macros")
if t not in {tok.strip() for tok in user_sloppiness.split(",")}
]
):
# Without these ccache declines every pch-consuming compile
env["CCACHE_SLOPPINESS"] = ",".join((user_sloppiness, *missing))
_LOGGER.warning(
"Adding %s to CCACHE_SLOPPINESS so ccache can cache compiles "
"that use the precompiled header",
",".join(missing),
)
return env
def guarded_prepare(build_dir: Path, prepare: Callable[[], None]) -> None:
"""Run a backend's pch preparation; an optional speedup must never
abort the build. Strict is read first so its own knob error cannot
mask the real failure; discard_pch raises if a stale .gch survives;
the header is ensured so OBJECT_DEPENDS stays satisfiable."""
try:
prepare()
except Exception: # noqa: BLE001 # pylint: disable=broad-exception-caught
strict = pch_strict()
discard_pch(build_dir)
if strict:
raise
header = build_dir / PCH_HEADER_NAME
if not header.exists():
try:
header.touch()
except OSError as err:
# The coming OBJECT_DEPENDS error would hide the real cause
_LOGGER.warning("Could not create the pch placeholder: %s", err)
_LOGGER.warning(
"Precompiled header setup failed; compiling without it", exc_info=True
)
def pch_extra_scripts() -> list[str]:
"""The extra_scripts entries a PlatformIO platform registers for the
pch; empty when disabled (the script itself has no enable check)."""
if not pch_enabled():
pch_disabled_degraded()
return []
return ["post:pch.py"]
def pch_header_text(include_headers: Iterable[str]) -> str:
"""The prefix-header source: exactly these includes, in order."""
return "".join(f'#include "{name}"\n' for name in include_headers)
def _resolves(path: Path) -> bool:
"""False when missing; other stat failures propagate (identity unknown,
unlike is_file(), which would silently drop the header)."""
try:
return stat.S_ISREG(path.stat().st_mode)
except (FileNotFoundError, NotADirectoryError):
return False
def _include_closure(src_dir: Path, roots: Iterable[str]) -> dict[str, bytes]:
"""Include closure of ``roots``: src-relative name -> contents.
Resolution mirrors the compiler (includer's dir, then src root); names
outside ``src_dir`` end the walk and are versioned by the caller. No
#ifdef evaluation: over-approximating is the safe direction.
"""
seen: dict[str, bytes] = {}
stack: list[tuple[str, str]] = [(name, "") for name in roots]
while stack:
name, from_dir = stack.pop()
for candidate in (f"{from_dir}/{name}" if from_dir else name, name):
rel = posixpath.normpath(candidate)
if not rel.startswith("..") and _resolves(src_dir / rel):
break
else:
continue
if rel in seen:
continue
try:
data = (src_dir / rel).read_bytes()
except OSError as err:
# A marker would truncate the transitive walk; fail closed
_LOGGER.warning("Could not read %s for the pch checksum: %s", rel, err)
raise
seen[rel] = data
parent = posixpath.dirname(rel)
stack.extend(
# surrogateescape: a non-UTF-8 name just fails to resolve
(inc.decode(errors="surrogateescape"), parent)
for inc in _INCLUDE_RE.findall(data)
)
return seen
def pch_checksum(
src_dir: Path, include_headers: Iterable[str], extra: Iterable[str]
) -> str:
"""Digest standing in for the .gch in ccache's hash: the include closure
of the prefix header plus caller-supplied identity strings (versioned
install paths, flags). Raises OSError when a header's identity cannot
be established at all; callers must then compile without a pch."""
digest = hashlib.sha256()
closure = _include_closure(src_dir, include_headers)
for name in sorted(closure):
digest.update(name.encode(errors="surrogateescape"))
digest.update(closure[name])
digest.update(b"\0")
for item in extra:
digest.update(item.encode(errors="surrogateescape"))
digest.update(b"\0")
return digest.hexdigest()
# Tokens dropped when retargeting a TU's flags at the prefix header
# (the pch compile must not touch depfiles)
_PCH_STRIP_FLAGS_WITH_ARG = frozenset({"-o", "-c", "-MT", "-MF", "-MQ"})
_PCH_STRIP_FLAGS = frozenset({"-MD", "-MMD", "-MP", "-MM", "-M"})
def pch_compile_command(
build_dir: Path, header: Path, gch: Path
) -> tuple[list[str], Path] | None:
"""The exact src C++ flags from compile_commands.json retargeted at the
header, with the directory they resolve against (relative -I paths must
be expanded and executed from the same root); None (logged) when no
configured C++ TU is available yet."""
from esphome.core import CORE
try:
entries = json.loads(
(build_dir / "compile_commands.json").read_text(encoding="utf-8")
)
except (OSError, json.JSONDecodeError) as err:
# Configure already succeeded, so an unusable DB is a real anomaly
_LOGGER.warning("No usable compile database, skipping pch: %s", err)
return None
if not isinstance(entries, list):
_LOGGER.warning("Malformed compile database, skipping pch")
return None
# CMake may spell paths through a symlink differently than CORE does
# (macOS /tmp vs /private/tmp), so compare resolved paths
src_root = Path(CORE.relative_src_path()).resolve()
entry = next(
(
e
for e in entries
if isinstance(e, dict)
and isinstance(e.get("file"), str)
and e["file"].endswith(CXX_SOURCE_SUFFIXES)
and Path(e["file"]).resolve().is_relative_to(src_root)
),
None,
)
if entry is None:
_LOGGER.warning("No src C++ entry in the compile database, skipping pch")
return None
directory = entry.get("directory")
cmd_dir = Path(directory) if isinstance(directory, str) and directory else build_dir
command = entry.get("command")
tokens = expand_response_files(
split_command(command if isinstance(command, str) else ""), cmd_dir
)
# A DB recorded with ccache enabled prefixes the compiler with the
# launcher; the .gch must be compiled directly
if tokens and is_launcher(tokens[0]):
tokens = tokens[1:]
if not tokens:
# "arguments"-style or empty entries must skip, not spawn "-x ..."
_LOGGER.warning("Compile database entry has no usable command, skipping pch")
return None
args: list[str] = []
arg_it = iter(tokens)
for tok in arg_it:
if tok in _PCH_STRIP_FLAGS_WITH_ARG:
next(arg_it, None)
continue
if tok in _PCH_STRIP_FLAGS:
continue
if tok == "-include":
# Drop only the injected prefix; user force-includes must reach
# the .gch compile or GCC rejects it over the macro mismatch
inc = next(arg_it, "")
if not inc.endswith(PCH_HEADER_NAME):
args.extend(("-include", inc))
continue
args.append(tok)
return [*args, "-x", "c++-header", "-c", str(header), "-o", str(gch)], cmd_dir
def _flags_identity(tokens: Iterable[str]) -> str:
"""Flag string normalized for digest use: strip like ccache's rewriting
(user CCACHE_BASEDIR wins); the raw build path covers unresolved
(symlinked) spellings."""
from esphome.core import CORE
return (
" ".join(tokens)
.replace(effective_ccache_basedir(), "")
.replace(str(CORE.build_path), "")
)
def pch_identity(
tokens: Iterable[str],
src_dir: Path,
include_headers: tuple[str, ...],
extra: Iterable[str],
) -> str | None:
"""The .sum digest naming this exact pch build: include closure, header
text, backend identity strings, and the normalized compile command or
flags (``tokens``). None (warned and degraded) when the identity
cannot be established."""
try:
return pch_checksum(
src_dir,
include_headers,
(
# The closure is sorted, so root order only enters via the text
pch_header_text(include_headers),
*extra,
_flags_identity(tokens),
),
)
except (OSError, UnicodeError) as err:
# Identity unknown: a stale cache entry must never be served
_LOGGER.warning(
"Could not establish the pch identity; compiling without it: %s", err
)
pch_degraded(f"identity unknown: {err}")
return None
def log_pch_in_use() -> None:
# The only place a user can discover the knob; emitted only once a
# .gch is actually fresh or being built
_LOGGER.info(
"Compiling with a precompiled header (set ESPHOME_PCH_ENABLE=0 to disable)"
)
def _read_stamp(path: Path) -> str:
"""A corrupt sidecar must read as stale, not kill the pch forever."""
try:
return path.read_text(encoding="utf-8").strip()
except (OSError, UnicodeDecodeError):
return ""
def discard_pch(build_dir: Path) -> None:
"""Remove the pch sidecars so a stale .gch is never consumed.
Bumps the header only when a .gch was actually removed: TUs compiled
against it have incomplete depfiles, while a repeat failure with no
.gch must not force a full rebuild every build. A .gch that survives
an unlink failure would be consumed silently (wrong output, not a
slow build), so that raises.
"""
header = build_dir / PCH_HEADER_NAME
gch = Path(f"{header}.gch")
had_gch = gch.is_file()
errors = []
for sidecar in (gch, Path(f"{gch}.sum")):
try:
sidecar.unlink(missing_ok=True)
except OSError as err:
if sidecar.is_file():
from esphome.core import EsphomeError
raise EsphomeError(
f"Could not discard the stale precompiled header: {err}"
) from err
errors.append(err)
for err in errors:
_LOGGER.warning("Could not discard the pch sidecars: %s", err)
if had_gch and header.is_file():
with suppress(OSError):
os.utime(header)
def prepare_pch(
build_dir: Path, include_headers: tuple[str, ...], extra: Iterable[str]
) -> None:
"""Compile ``build_dir``'s .gch from compile_commands.json flags and
write its ccache .sum.
The .sum doubles as the freshness stamp and folds in the compile
command, so a flag-only change rebuilds the .gch; ``extra`` carries
backend identity (framework version, sdkconfig, ...). A failed
compile falls back to the plain header include.
"""
from esphome.core import CORE
header = build_dir / PCH_HEADER_NAME
gch = Path(f"{header}.gch")
sum_path = Path(f"{gch}.sum")
cmd_and_dir = pch_compile_command(build_dir, header, gch)
if cmd_and_dir is None:
# Freshness cannot be validated; a leftover .gch must not be consumed
discard_pch(build_dir)
pch_degraded("no usable compile command")
return
cmd, cmd_dir = cmd_and_dir
checksum = pch_identity(cmd, CORE.relative_src_path(), include_headers, extra)
if checksum is None:
discard_pch(build_dir)
return
failed_marker = Path(f"{gch}.failed")
def _run(
run_cmd: list[str], what: str, stdin: str | None = None
) -> subprocess.CompletedProcess | None:
"""Spawn one pch tool step; environmental failures discard and
degrade (None): spawn/IO/timeout errors and signal kills never
latch the marker."""
try:
proc = subprocess.run(
run_cmd,
cwd=cmd_dir,
# C locale keeps diagnostics matchable by _TRANSIENT_ERRORS
env={**os.environ, "LC_ALL": "C"},
input=stdin,
capture_output=True,
text=True,
check=False,
timeout=300,
)
except (OSError, subprocess.SubprocessError) as err:
_LOGGER.warning("Precompiled header %s did not run: %s", what, err)
discard_pch(build_dir)
pch_degraded(f"{what} did not run: {err}")
return None
if proc.returncode < 0:
# Killed by a signal (OOM, ^C): environmental, do not latch
_LOGGER.warning(
"Precompiled header %s was killed (signal %d); retrying next build",
what,
-proc.returncode,
)
discard_pch(build_dir)
pch_degraded(f"{what} killed by signal {-proc.returncode}")
return None
return proc
def _fail(error: str, reason: str, latch: bool) -> None:
"""Discard and degrade; deterministic failures latch when asked."""
_LOGGER.warning(
"Precompiled header failed; compiling without it: %s", error[:400]
)
# Latching paths keep the full compiler output recoverable
_LOGGER.debug("Full pch output: %s", error)
discard_pch(build_dir)
if latch and not any(m in error for m in _TRANSIENT_ERRORS):
# Skip retries until a header/flag/backend-identity/command change
failed_marker.write_text(checksum + "\n", encoding="utf-8")
os.utime(header)
pch_degraded(f"{reason}: {error[:200]}")
def _probe(latch: bool = True) -> None:
"""Load-check the built .gch: some toolchains build one they then
refuse to load (per-process ASLR). Dep flags are already stripped
from cmd, so no -MF is needed; cmd ends with the fixed
"-x c++-header -c -o" tail. A cached-header rejection may not
reproduce (per-process), so that caller passes latch=False."""
if cmd[-6:-4] != ["-x", "c++-header"]:
# The slice below depends on pch_compile_command's fixed tail
_LOGGER.warning("Unexpected pch command shape: %s", cmd[-6:])
discard_pch(build_dir)
pch_degraded("unexpected pch command shape")
return
base = cmd[:-6]
probe = _run([*base, *pch_probe_args(str(header))], "probe", stdin="")
if probe is None:
return
if probe.returncode != 0:
# Disambiguate: only blame the pch when the same compile passes
# without it; a failing baseline is its own (latchable) problem
baseline = _run([*base, *pch_probe_tail()], "probe baseline", stdin="")
if baseline is None:
return
if baseline.returncode == 0:
error = probe.stderr.strip() or f"exit code {probe.returncode}"
_fail(error, "toolchain cannot load the pch", latch=latch)
else:
error = baseline.stderr.strip() or f"exit code {baseline.returncode}"
_fail(error, "probe cannot run at all", latch=latch)
if gch.is_file() and _read_stamp(sum_path) == checksum:
log_pch_in_use()
if pch_strict():
# Rejection is per-process, so a cached .gch must re-prove
# loadability for the strict gate (CI-only cost); no latch,
# since the rejection may not reproduce either
_probe(latch=False)
return
if _read_stamp(failed_marker) == checksum:
_LOGGER.info(
"Precompiled header disabled after an earlier failure; delete %s to retry",
failed_marker,
)
pch_degraded("earlier failure latched")
return
log_pch_in_use()
result = _run(cmd, "compile")
if result is None:
return
error = None
if result.returncode != 0:
error = result.stderr.strip() or f"exit code {result.returncode}"
elif not gch.is_file():
error = "compiler produced no .gch"
if error is not None:
_fail(error, "compile failed", latch=True)
return
_probe()
if not gch.is_file():
# The probe discarded a rejected or unrunnable .gch
return
failed_marker.unlink(missing_ok=True)
sum_path.write_text(checksum + "\n", encoding="utf-8")
# Consumers depend on the header (depfiles cannot see through a .gch);
# bump it so users of the previous .gch recompile
os.utime(header)
+137 -137
View File
@@ -7,146 +7,146 @@ namespace esphome::captive_portal {
#ifdef USE_CAPTIVE_PORTAL_GZIP
constexpr uint8_t INDEX_GZ[] PROGMEM = {
0x1f, 0x8b, 0x08, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x03, 0x95, 0x16, 0x6b, 0x8f, 0xdb, 0x36, 0xf2, 0x7b, 0x7f,
0x05, 0x8f, 0x4d, 0x1b, 0xa9, 0xb1, 0xa8, 0x87, 0xd7, 0xde, 0x44, 0x96, 0x54, 0xa4, 0x7b, 0x2d, 0x5a, 0xa0, 0x69,
0x03, 0xec, 0x36, 0xf7, 0x21, 0x08, 0xb0, 0x34, 0x39, 0xb2, 0x98, 0xa5, 0x48, 0x1d, 0x49, 0xbf, 0x62, 0xf8, 0x7e,
0xfb, 0x81, 0x92, 0xec, 0xf5, 0x2e, 0x9a, 0x03, 0x0e, 0x86, 0x85, 0x19, 0xce, 0x7b, 0x38, 0x0f, 0x16, 0xff, 0xe0,
0x9a, 0xb9, 0x7d, 0x07, 0xa8, 0x71, 0xad, 0xac, 0x0a, 0xff, 0x45, 0x92, 0xaa, 0x55, 0x09, 0xaa, 0x2a, 0x1a, 0xa0,
0xbc, 0x2a, 0x5a, 0x70, 0x14, 0xb1, 0x86, 0x1a, 0x0b, 0xae, 0xfc, 0xeb, 0xee, 0x97, 0xe8, 0x75, 0x55, 0x48, 0xa1,
0x1e, 0x90, 0x01, 0x59, 0x0a, 0xa6, 0x15, 0x6a, 0x0c, 0xd4, 0x25, 0xa7, 0x8e, 0xe6, 0xa2, 0xa5, 0x2b, 0x18, 0x45,
0x14, 0x6d, 0xa1, 0xdc, 0x08, 0xd8, 0x76, 0xda, 0x38, 0xc4, 0xb4, 0x72, 0xa0, 0x5c, 0x89, 0xb7, 0x82, 0xbb, 0xa6,
0xe4, 0xb0, 0x11, 0x0c, 0xa2, 0x1e, 0x99, 0x08, 0x25, 0x9c, 0xa0, 0x32, 0xb2, 0x8c, 0x4a, 0x28, 0xd3, 0xc9, 0xda,
0x82, 0xe9, 0x11, 0xba, 0x94, 0x50, 0x2a, 0x8d, 0xab, 0xc2, 0x32, 0x23, 0x3a, 0x87, 0xbc, 0xab, 0x65, 0xab, 0xf9,
0x5a, 0x42, 0x15, 0xc7, 0xd4, 0x5a, 0x70, 0x36, 0x16, 0x8a, 0xc3, 0x8e, 0xd0, 0x6b, 0xb8, 0xa6, 0x2c, 0x4d, 0xc8,
0x67, 0xfb, 0x0d, 0xd7, 0x6c, 0xdd, 0x82, 0x72, 0x44, 0x6a, 0x46, 0x9d, 0xd0, 0x8a, 0x58, 0xa0, 0x86, 0x35, 0x65,
0x59, 0xe2, 0x1f, 0x2d, 0xdd, 0x00, 0xfe, 0xfe, 0xfb, 0xe0, 0xcc, 0xb4, 0x02, 0xf7, 0xb3, 0x04, 0x0f, 0xda, 0x9f,
0xf6, 0x77, 0x74, 0xf5, 0x07, 0x6d, 0x21, 0xc0, 0xd4, 0x0a, 0x0e, 0x38, 0xfc, 0x98, 0x7c, 0x22, 0xd6, 0xed, 0x25,
0x10, 0x2e, 0x6c, 0x27, 0xe9, 0xbe, 0xc4, 0x4b, 0xa9, 0xd9, 0x03, 0x0e, 0x17, 0xf5, 0x5a, 0x31, 0xaf, 0x1c, 0xe9,
0x00, 0xc2, 0x83, 0x04, 0x87, 0x5c, 0xf9, 0x8e, 0xba, 0x86, 0xb4, 0x74, 0x17, 0x0c, 0x80, 0x50, 0x41, 0xf6, 0x43,
0x00, 0xaf, 0xd2, 0x24, 0x09, 0x27, 0xfd, 0x27, 0x09, 0xe3, 0x34, 0x49, 0x16, 0x06, 0xdc, 0xda, 0x28, 0x44, 0x83,
0xfb, 0xa2, 0xa3, 0xae, 0x41, 0xbc, 0xc4, 0xef, 0xd2, 0x0c, 0xa5, 0x6f, 0x48, 0x36, 0xfb, 0x9d, 0x5c, 0xa3, 0x2b,
0x92, 0xcd, 0xd8, 0x75, 0x34, 0x43, 0xe9, 0x55, 0x34, 0x43, 0x59, 0x46, 0x66, 0x28, 0xf9, 0x82, 0x51, 0x2d, 0xa4,
0x2c, 0xb1, 0xd2, 0x0a, 0x30, 0xb2, 0xce, 0xe8, 0x07, 0x28, 0x31, 0x5b, 0x1b, 0x03, 0xca, 0xdd, 0x68, 0xa9, 0x0d,
0x8e, 0xab, 0x6f, 0xfe, 0x2f, 0x85, 0xce, 0x50, 0x65, 0x6b, 0x6d, 0xda, 0x12, 0xf7, 0xd9, 0x0f, 0x5e, 0x1c, 0xdc,
0x11, 0xf9, 0x4f, 0x78, 0x41, 0x8c, 0xb4, 0x11, 0x2b, 0xa1, 0x4a, 0xec, 0x35, 0xbe, 0xc6, 0x71, 0x75, 0x1f, 0x1e,
0xcf, 0xd1, 0x53, 0x1f, 0xfd, 0x18, 0x0f, 0x0f, 0x3e, 0xde, 0x17, 0x76, 0xb3, 0x42, 0xbb, 0x56, 0x2a, 0x5b, 0xe2,
0xc6, 0xb9, 0x2e, 0x8f, 0xe3, 0xed, 0x76, 0x4b, 0xb6, 0x53, 0xa2, 0xcd, 0x2a, 0xce, 0x92, 0x24, 0x89, 0xed, 0x66,
0x85, 0xd1, 0x50, 0x08, 0x38, 0xbb, 0xc2, 0xa8, 0x01, 0xb1, 0x6a, 0x5c, 0x0f, 0x57, 0x2f, 0x0e, 0x70, 0x2c, 0x3c,
0x47, 0x75, 0xff, 0xe9, 0xc2, 0x8a, 0xb9, 0xb0, 0x02, 0x3f, 0xd2, 0x00, 0x9f, 0xc2, 0x7c, 0xd9, 0x87, 0x79, 0x4d,
0x33, 0x94, 0xa1, 0xa4, 0xff, 0x65, 0x91, 0x87, 0x47, 0x2c, 0x7a, 0x86, 0xa1, 0x0b, 0xcc, 0x43, 0xed, 0x3c, 0x7a,
0x73, 0x96, 0x4d, 0xfd, 0xc9, 0x26, 0x4d, 0x1e, 0x0f, 0xbc, 0xc0, 0xaf, 0xf3, 0x4b, 0x3c, 0xca, 0x3e, 0x5c, 0x32,
0x78, 0x6b, 0x4d, 0xfa, 0x61, 0x4e, 0x67, 0x68, 0x36, 0x9e, 0xcc, 0x22, 0x0f, 0x9f, 0x31, 0x34, 0xdb, 0x64, 0x4d,
0xda, 0x46, 0xf3, 0x68, 0x46, 0xa7, 0x68, 0x3a, 0x3a, 0x32, 0x45, 0xd3, 0x4d, 0xd6, 0xcc, 0x3f, 0xcc, 0x2f, 0xcf,
0xa2, 0xe9, 0x97, 0x97, 0x71, 0x85, 0xc3, 0x1c, 0xe3, 0xc7, 0xc8, 0xf9, 0x65, 0xe4, 0xe4, 0xb3, 0x16, 0x2a, 0xc0,
0x38, 0x3c, 0xd6, 0xe0, 0x58, 0x13, 0xe0, 0x98, 0x69, 0x55, 0x8b, 0x15, 0xf9, 0x6c, 0xb5, 0xc2, 0x21, 0x71, 0x0d,
0xa8, 0xe0, 0x24, 0xea, 0x05, 0xa1, 0xa7, 0x04, 0xcf, 0x29, 0x2e, 0x3c, 0x9c, 0xeb, 0xdf, 0x09, 0x27, 0xa1, 0x74,
0xc4, 0x37, 0xec, 0xe4, 0x6f, 0xba, 0xe2, 0xa7, 0xfd, 0x6f, 0x3c, 0xc0, 0x2d, 0x65, 0x38, 0x24, 0x42, 0x29, 0x30,
0x77, 0xb0, 0x73, 0x25, 0x7e, 0xf7, 0xf6, 0x06, 0xbd, 0xe5, 0xdc, 0x80, 0xb5, 0x39, 0xc2, 0xaf, 0x1c, 0x69, 0x29,
0xfb, 0xba, 0x78, 0x93, 0x3e, 0x95, 0xfe, 0x97, 0xf8, 0x45, 0xa0, 0x3f, 0xc0, 0x6d, 0xb5, 0x79, 0x18, 0xe5, 0xbd,
0xfd, 0x85, 0x6f, 0x23, 0x56, 0x7e, 0x55, 0x8d, 0x02, 0x87, 0xc3, 0x89, 0xf8, 0x3a, 0x83, 0xb5, 0x82, 0xe3, 0x70,
0x22, 0xbf, 0xce, 0xd1, 0x59, 0xdf, 0xbc, 0x8e, 0xd0, 0xce, 0x12, 0x2b, 0x05, 0x83, 0x20, 0x0d, 0x49, 0xad, 0xcd,
0xcf, 0x94, 0x35, 0x8f, 0x09, 0xb2, 0x43, 0x47, 0xab, 0x47, 0x3d, 0xcc, 0x00, 0x75, 0x30, 0xaa, 0x0a, 0x30, 0x17,
0x1b, 0x1c, 0x2e, 0x14, 0x61, 0x92, 0x5a, 0xeb, 0x47, 0x46, 0xe9, 0x9d, 0xf3, 0xe1, 0xe0, 0x89, 0x1a, 0x22, 0xfd,
0xf5, 0xee, 0xdd, 0xef, 0xe5, 0x7d, 0x41, 0x87, 0x01, 0x89, 0xbf, 0xc5, 0xa8, 0x67, 0x3e, 0x33, 0x46, 0x12, 0x6a,
0xe7, 0x2b, 0x5e, 0x07, 0x96, 0x18, 0x6b, 0x45, 0x78, 0x2c, 0x6c, 0x47, 0xd5, 0x73, 0xb6, 0x3e, 0xa6, 0xaa, 0x88,
0x3d, 0xad, 0x2a, 0x62, 0x5a, 0xbd, 0x38, 0x98, 0xc0, 0xfa, 0xe1, 0xf6, 0x10, 0x1e, 0xef, 0x27, 0x8a, 0xfc, 0x7b,
0x0d, 0x66, 0x7f, 0x0b, 0x12, 0x98, 0xd3, 0x26, 0xc0, 0xe4, 0x89, 0x60, 0x48, 0x1c, 0xec, 0xdc, 0xcd, 0x38, 0x7f,
0x2d, 0xf1, 0x87, 0x13, 0x45, 0xb4, 0x62, 0x52, 0xb0, 0x87, 0xf2, 0x1c, 0x71, 0x78, 0x10, 0x64, 0x43, 0xe5, 0x1a,
0x4e, 0x3c, 0x92, 0xd4, 0x9a, 0xad, 0x6d, 0x10, 0x1e, 0x27, 0x8c, 0xd0, 0xae, 0x03, 0xc5, 0x6f, 0x1a, 0x21, 0x79,
0xa0, 0xc2, 0x63, 0xf8, 0x78, 0xd3, 0xcf, 0x8c, 0xfb, 0xd5, 0xf0, 0xd1, 0x80, 0xfc, 0x4f, 0xf9, 0xd2, 0x2f, 0x87,
0x97, 0x9f, 0x70, 0x48, 0xfa, 0xf8, 0xef, 0x1f, 0x37, 0x84, 0x6f, 0xef, 0x57, 0xbb, 0x56, 0x4e, 0x7c, 0xe8, 0xd1,
0x7c, 0x16, 0x1e, 0xef, 0x8f, 0xe1, 0x31, 0x5c, 0x14, 0xf1, 0x30, 0xe7, 0xab, 0xa2, 0x1f, 0xb9, 0xd5, 0x0f, 0x87,
0xa5, 0xde, 0x45, 0x56, 0x7c, 0x11, 0x6a, 0x95, 0x0b, 0xd5, 0x80, 0x11, 0xee, 0xc8, 0xc5, 0x66, 0x22, 0x54, 0xb7,
0x76, 0x87, 0x8e, 0x72, 0xee, 0x29, 0xb3, 0x6e, 0xb7, 0xa8, 0xb5, 0x72, 0x9e, 0x13, 0xf2, 0x14, 0xda, 0xe3, 0x40,
0xef, 0x27, 0x4c, 0xfe, 0x66, 0xf6, 0xdd, 0x71, 0xa9, 0xf9, 0xfe, 0xe0, 0xd3, 0x10, 0x51, 0x29, 0x56, 0x2a, 0x67,
0xa0, 0x1c, 0x98, 0x41, 0xa8, 0xa6, 0xad, 0x90, 0xfb, 0xdc, 0x52, 0x65, 0x23, 0x0b, 0x46, 0xd4, 0xc7, 0xe5, 0xda,
0x39, 0xad, 0x0e, 0x4b, 0x6d, 0x38, 0x98, 0x3c, 0x59, 0x0c, 0x40, 0x64, 0x28, 0x17, 0x6b, 0x9b, 0x93, 0xa9, 0x81,
0x76, 0xb1, 0xa4, 0xec, 0x61, 0x65, 0xf4, 0x5a, 0xf1, 0x88, 0xf9, 0xc9, 0x9b, 0x7f, 0x9b, 0xd6, 0x74, 0x0a, 0x6c,
0x31, 0x62, 0x75, 0x5d, 0x2f, 0xa4, 0x50, 0x10, 0x0d, 0xb3, 0x2d, 0xcf, 0xc8, 0x95, 0x17, 0xbb, 0x70, 0x93, 0x64,
0xfe, 0x60, 0xf0, 0x31, 0x4d, 0x92, 0xef, 0x16, 0xa7, 0x70, 0x92, 0x05, 0x5b, 0x1b, 0xab, 0x4d, 0xde, 0x69, 0xe1,
0xdd, 0x3c, 0xb6, 0x54, 0xa8, 0x4b, 0xef, 0x7d, 0xd9, 0x2c, 0xc6, 0x75, 0x94, 0x0b, 0xd5, 0x9b, 0xe9, 0x97, 0xd2,
0xa2, 0x15, 0x6a, 0xd8, 0xa9, 0x79, 0x36, 0x4f, 0xba, 0xdd, 0xf1, 0x54, 0x09, 0x87, 0x13, 0x77, 0x2d, 0x61, 0xb7,
0xf8, 0xbc, 0xb6, 0x4e, 0xd4, 0xfb, 0x68, 0xdc, 0xc9, 0xb9, 0xed, 0x28, 0x83, 0x68, 0x09, 0x6e, 0x0b, 0xa0, 0x16,
0xbd, 0x8d, 0x48, 0x38, 0x68, 0xed, 0x98, 0xa7, 0xb3, 0x9a, 0xbe, 0x60, 0x9f, 0xea, 0xfa, 0x5f, 0xdc, 0xbe, 0x8a,
0x0e, 0x2d, 0x35, 0x2b, 0xa1, 0xa2, 0xa5, 0x76, 0x4e, 0xb7, 0x79, 0x74, 0xdd, 0xed, 0x16, 0xe3, 0x91, 0x57, 0x96,
0xa7, 0xde, 0xcd, 0x7e, 0xd7, 0x9e, 0xf2, 0x9d, 0x76, 0x3b, 0x64, 0xb5, 0x14, 0x7c, 0xe4, 0xeb, 0x59, 0x50, 0x72,
0x4e, 0x4f, 0x3a, 0xeb, 0x76, 0xc8, 0x9f, 0x9d, 0x52, 0x7d, 0x55, 0xbf, 0xa6, 0x69, 0xf2, 0x37, 0x37, 0xc2, 0xeb,
0x3a, 0x5b, 0xd6, 0xe7, 0x4c, 0xf9, 0xb5, 0xe9, 0x57, 0x4b, 0x5f, 0x5a, 0x45, 0x3c, 0xbc, 0x6e, 0x7c, 0x65, 0x54,
0x85, 0xcf, 0x70, 0x55, 0x34, 0x29, 0x12, 0xbc, 0x6c, 0x29, 0xab, 0x2e, 0x66, 0x5b, 0x11, 0x37, 0xe9, 0x89, 0xd4,
0xa4, 0xd5, 0x93, 0xb9, 0x35, 0xd0, 0x7a, 0xef, 0xab, 0x1b, 0xad, 0x14, 0x30, 0x27, 0xd4, 0x0a, 0x39, 0x8d, 0xc6,
0x14, 0x10, 0x42, 0x8a, 0xa5, 0xa9, 0xde, 0x4b, 0xa0, 0x16, 0xd0, 0x96, 0x0a, 0x47, 0x8a, 0x78, 0xe0, 0x1f, 0x3a,
0x5d, 0xf0, 0x52, 0x81, 0x3b, 0xf7, 0x76, 0x33, 0x1d, 0x0c, 0xdc, 0x82, 0xf3, 0x9a, 0xbc, 0x81, 0x69, 0x55, 0xf8,
0x15, 0x8c, 0x68, 0xdf, 0xa5, 0x65, 0xbc, 0x15, 0xb5, 0xf0, 0x4f, 0x98, 0xaa, 0xe8, 0x8b, 0xdc, 0x6b, 0xf0, 0x79,
0x1e, 0x9e, 0x5b, 0x3d, 0x24, 0x41, 0xad, 0x5c, 0x53, 0x4e, 0x33, 0xd4, 0x49, 0xca, 0xa0, 0xd1, 0x92, 0x83, 0x29,
0x6f, 0x6f, 0x7f, 0xfb, 0x67, 0xe5, 0x9d, 0x79, 0x94, 0xeb, 0xec, 0xc3, 0x20, 0xe6, 0x81, 0x51, 0x6a, 0x7e, 0x35,
0x3c, 0xb2, 0x3a, 0x6a, 0xed, 0x56, 0x1b, 0xfe, 0x44, 0xc7, 0xfb, 0xf1, 0x70, 0xd0, 0xd3, 0xff, 0xfb, 0x56, 0xa9,
0x6e, 0xe9, 0x06, 0x8a, 0x78, 0x44, 0x8a, 0xd8, 0x3b, 0x3c, 0xd0, 0x9b, 0x91, 0xaf, 0x49, 0xab, 0x3f, 0xef, 0xde,
0xa2, 0xbf, 0x3a, 0x4e, 0x1d, 0x0c, 0x69, 0xeb, 0xa3, 0x6a, 0xc1, 0x35, 0x9a, 0x97, 0xef, 0xff, 0xbc, 0xbd, 0x3b,
0x47, 0xb8, 0xee, 0x99, 0x10, 0x28, 0x36, 0x3c, 0xf7, 0xd6, 0xd2, 0x89, 0x8e, 0x1a, 0xd7, 0xab, 0x8d, 0xfc, 0x14,
0x39, 0xc5, 0xd0, 0xd3, 0x6b, 0x21, 0x61, 0x08, 0x63, 0x10, 0xac, 0xd0, 0xc9, 0xab, 0x93, 0xb5, 0x67, 0x7e, 0xc5,
0xc3, 0x6d, 0xc7, 0xc3, 0xd5, 0xc7, 0xfd, 0xcb, 0xf7, 0xbf, 0x81, 0xdb, 0x13, 0xb5, 0x09, 0x0b, 0x00, 0x00};
0x1f, 0x8b, 0x08, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x03, 0x95, 0x56, 0x6d, 0x8f, 0xdb, 0x36, 0x0c, 0xfe, 0xbe,
0x5f, 0xa1, 0x79, 0xdd, 0x6a, 0xaf, 0xb1, 0xfc, 0x92, 0x4b, 0xda, 0x3a, 0x96, 0x8b, 0xee, 0xd6, 0x62, 0x03, 0xd6,
0xad, 0xc0, 0xdd, 0xba, 0x0f, 0x45, 0x01, 0x2b, 0x32, 0x1d, 0xab, 0x27, 0x4b, 0x9e, 0xa4, 0xbc, 0x35, 0xc8, 0x7e,
0xfb, 0x20, 0xdb, 0xc9, 0xe5, 0x8a, 0x16, 0xd8, 0x10, 0xc4, 0xa0, 0x44, 0xf2, 0xe1, 0x8b, 0x28, 0x52, 0xf9, 0xb7,
0x95, 0x62, 0x76, 0xdf, 0x01, 0x6a, 0x6c, 0x2b, 0x8a, 0xdc, 0x7d, 0x91, 0xa0, 0x72, 0x45, 0x40, 0x16, 0x79, 0x03,
0xb4, 0x2a, 0xf2, 0x16, 0x2c, 0x45, 0xac, 0xa1, 0xda, 0x80, 0x25, 0x7f, 0xde, 0xbe, 0x0e, 0x9f, 0x15, 0xb9, 0xe0,
0xf2, 0x0e, 0x69, 0x10, 0x84, 0x33, 0x25, 0x51, 0xa3, 0xa1, 0x26, 0x15, 0xb5, 0x34, 0xe3, 0x2d, 0x5d, 0xc1, 0xa8,
0x22, 0x69, 0x0b, 0x64, 0xc3, 0x61, 0xdb, 0x29, 0x6d, 0x11, 0x53, 0xd2, 0x82, 0xb4, 0xc4, 0xdb, 0xf2, 0xca, 0x36,
0xa4, 0x82, 0x0d, 0x67, 0x10, 0xf6, 0x8b, 0x09, 0x97, 0xdc, 0x72, 0x2a, 0x42, 0xc3, 0xa8, 0x00, 0x92, 0x4c, 0xd6,
0x06, 0x74, 0xbf, 0xa0, 0x4b, 0x01, 0x44, 0x2a, 0xaf, 0xc8, 0x0d, 0xd3, 0xbc, 0xb3, 0xc8, 0xb9, 0x4a, 0x5a, 0x55,
0xad, 0x05, 0x20, 0xa6, 0x95, 0x31, 0x4a, 0xf3, 0x15, 0x97, 0x45, 0xa5, 0xd8, 0xba, 0x05, 0x69, 0xb1, 0x50, 0x8c,
0x5a, 0xae, 0x24, 0x36, 0x40, 0x35, 0x6b, 0x08, 0x21, 0xe5, 0x0b, 0x43, 0x37, 0x50, 0xfe, 0xf0, 0x83, 0x7f, 0x16,
0x5a, 0x81, 0x7d, 0x25, 0xc0, 0x91, 0xe6, 0xa7, 0xfd, 0x2d, 0x5d, 0xfd, 0x4e, 0x5b, 0xf0, 0x4b, 0x6a, 0x78, 0x05,
0x65, 0xf0, 0x3e, 0xfe, 0x80, 0x8d, 0xdd, 0x0b, 0xc0, 0x15, 0x37, 0x9d, 0xa0, 0x7b, 0x52, 0x2e, 0x85, 0x62, 0x77,
0x65, 0xb0, 0xa8, 0xd7, 0x92, 0x39, 0x70, 0x04, 0x3e, 0x04, 0x07, 0x01, 0x16, 0x49, 0xf2, 0x86, 0xda, 0x06, 0xb7,
0x74, 0xe7, 0x0f, 0x04, 0x97, 0x7e, 0xfa, 0xa3, 0x0f, 0x4f, 0x92, 0x38, 0x0e, 0x26, 0xfd, 0x27, 0x0e, 0xa2, 0x24,
0x8e, 0x17, 0x1a, 0xec, 0x5a, 0x4b, 0x64, 0xfd, 0x32, 0xef, 0xa8, 0x6d, 0x50, 0x45, 0xbc, 0x37, 0x49, 0x8a, 0x92,
0xe7, 0x38, 0x9d, 0xfd, 0x86, 0x9f, 0xa2, 0x2b, 0x9c, 0xce, 0xd8, 0xd3, 0x70, 0x86, 0x92, 0xab, 0x70, 0x86, 0xd2,
0x14, 0xcf, 0x50, 0xfc, 0xc9, 0x43, 0x35, 0x17, 0x82, 0x78, 0x52, 0x49, 0xf0, 0x90, 0xb1, 0x5a, 0xdd, 0x01, 0xf1,
0xd8, 0x5a, 0x6b, 0x90, 0xf6, 0x5a, 0x09, 0xa5, 0xbd, 0xa8, 0xf8, 0xe6, 0x7f, 0x01, 0x5a, 0x4d, 0xa5, 0xa9, 0x95,
0x6e, 0x89, 0xd7, 0xa7, 0xdb, 0x7f, 0x74, 0x90, 0x47, 0xe4, 0x3e, 0xc1, 0x05, 0x33, 0x1c, 0xf2, 0x4a, 0x3c, 0x87,
0xf8, 0xcc, 0x8b, 0x8a, 0x32, 0x38, 0x9e, 0xa3, 0xb7, 0x2e, 0xfa, 0x31, 0x1e, 0xed, 0xbf, 0x2f, 0x73, 0xb3, 0x59,
0xa1, 0x5d, 0x2b, 0xa4, 0x21, 0x5e, 0x63, 0x6d, 0x97, 0x45, 0xd1, 0x76, 0xbb, 0xc5, 0xdb, 0x29, 0x56, 0x7a, 0x15,
0xa5, 0x71, 0x1c, 0x47, 0x66, 0xb3, 0xf2, 0xd0, 0x70, 0xf2, 0x5e, 0x7a, 0xe5, 0xa1, 0x06, 0xf8, 0xaa, 0xb1, 0x3d,
0x5d, 0x3c, 0x3a, 0xc0, 0x31, 0x77, 0x12, 0x45, 0xf9, 0xe1, 0xc2, 0x8a, 0xbc, 0xb0, 0x02, 0x2f, 0x2e, 0xf2, 0xf6,
0xb8, 0x0f, 0xf3, 0x29, 0x4d, 0x51, 0x8a, 0xe2, 0xfe, 0x97, 0x86, 0x8e, 0x1e, 0x57, 0xe1, 0x67, 0x2b, 0x74, 0xb1,
0x72, 0x54, 0x3b, 0x0f, 0x9f, 0x9f, 0x75, 0x13, 0xb7, 0xb3, 0x49, 0xe2, 0xfb, 0x0d, 0xa7, 0xf0, 0xcb, 0xfc, 0x72,
0x1d, 0xa6, 0xef, 0x2e, 0x05, 0x9c, 0xb5, 0x26, 0x79, 0x37, 0xa7, 0x33, 0x34, 0x1b, 0x77, 0x66, 0xa1, 0xa3, 0xcf,
0x2b, 0x34, 0xdb, 0xa4, 0x4d, 0xd2, 0x86, 0xf3, 0x70, 0x46, 0xa7, 0x68, 0x3a, 0x3a, 0x32, 0x45, 0xd3, 0x4d, 0xda,
0xcc, 0xdf, 0xcd, 0x2f, 0xf7, 0xc2, 0xe9, 0xa7, 0xc7, 0x2e, 0xb9, 0x59, 0x59, 0xde, 0x47, 0xae, 0x2f, 0x23, 0xc7,
0x1f, 0x15, 0x97, 0x7e, 0xe9, 0xf2, 0x0f, 0x96, 0x35, 0x7e, 0x19, 0x31, 0x25, 0x6b, 0xbe, 0xc2, 0x1f, 0x8d, 0x92,
0x65, 0x80, 0x6d, 0x03, 0xd2, 0x3f, 0xa9, 0xfa, 0x36, 0x38, 0xd8, 0x9e, 0xe3, 0x7f, 0x81, 0x73, 0xae, 0x7f, 0xcb,
0xad, 0x00, 0x62, 0xb1, 0xbb, 0xa1, 0x93, 0x2f, 0xdc, 0x8a, 0x9f, 0xf6, 0xbf, 0x56, 0x7e, 0xd9, 0x52, 0x56, 0x06,
0x98, 0x4b, 0x09, 0xfa, 0x16, 0x76, 0x96, 0x94, 0x6f, 0x5e, 0x5e, 0xa3, 0x97, 0x55, 0xa5, 0xc1, 0x98, 0x0c, 0x95,
0x4f, 0x2c, 0x6e, 0x29, 0xfb, 0xba, 0x7a, 0x93, 0x3c, 0xd4, 0xfe, 0x8b, 0xbf, 0xe6, 0xe8, 0x77, 0xb0, 0x5b, 0xa5,
0xef, 0x46, 0x7d, 0x67, 0x7f, 0xe1, 0xae, 0x91, 0x26, 0x5f, 0x85, 0x91, 0x60, 0xcb, 0x60, 0xc2, 0xbf, 0x2e, 0x60,
0x0c, 0xaf, 0xca, 0x60, 0x42, 0xbf, 0x2e, 0xd1, 0x19, 0x77, 0x79, 0x2d, 0xa6, 0x9d, 0xc1, 0x46, 0x70, 0x06, 0x7e,
0x12, 0xe0, 0x5a, 0xe9, 0x57, 0x94, 0x35, 0x0f, 0x12, 0xe4, 0x5c, 0x51, 0xf7, 0x38, 0x4c, 0x03, 0xb5, 0x30, 0x42,
0xf9, 0x65, 0xc5, 0x37, 0x65, 0xb0, 0x50, 0x98, 0x09, 0x6a, 0x8c, 0x6b, 0x19, 0xc4, 0x39, 0xe7, 0xc2, 0x29, 0x27,
0x6a, 0x88, 0xf4, 0x97, 0xdb, 0x37, 0xbf, 0x91, 0x32, 0xa7, 0x43, 0x47, 0xf4, 0xbe, 0xf3, 0x50, 0x2f, 0x4c, 0xbc,
0x51, 0x30, 0x14, 0x50, 0xdb, 0xbe, 0xe2, 0x7d, 0x8b, 0xb5, 0x31, 0x3c, 0x38, 0xe6, 0xa6, 0xa3, 0xf2, 0x73, 0x31,
0x17, 0x93, 0x57, 0xe4, 0x91, 0xe3, 0x15, 0x79, 0x44, 0x8b, 0x47, 0x07, 0xe9, 0xf7, 0xcd, 0xed, 0x2e, 0x38, 0x3a,
0x6b, 0x7f, 0xaf, 0x41, 0xef, 0x6f, 0x40, 0x00, 0xb3, 0x4a, 0xfb, 0x25, 0xbe, 0x54, 0x74, 0x45, 0x01, 0x3b, 0x7b,
0x3d, 0x36, 0x5c, 0x8b, 0xdd, 0xe6, 0x44, 0x61, 0x25, 0x99, 0xe0, 0xec, 0x8e, 0x9c, 0x23, 0x0e, 0x0e, 0x1c, 0x6f,
0xa8, 0x58, 0xc3, 0x49, 0x86, 0xe2, 0x5a, 0xb1, 0xb5, 0xf1, 0x83, 0xe3, 0x44, 0x63, 0xda, 0x75, 0x20, 0xab, 0xeb,
0x86, 0x8b, 0xca, 0x57, 0xc1, 0x31, 0xb8, 0x3f, 0xe9, 0xcf, 0x8c, 0xbb, 0x59, 0xf0, 0x5e, 0x83, 0xf8, 0x87, 0x3c,
0x76, 0xd3, 0xe0, 0xf1, 0x87, 0x32, 0xc0, 0x7d, 0xfc, 0xe5, 0xfd, 0x48, 0x70, 0xd7, 0xfb, 0xc9, 0xae, 0x15, 0x13,
0x17, 0x7a, 0x38, 0x9f, 0x05, 0xc7, 0xf2, 0x18, 0x1c, 0x83, 0x45, 0x1e, 0x0d, 0x8d, 0xbd, 0xc8, 0xfb, 0x96, 0xdb,
0x8f, 0x94, 0x9e, 0x32, 0x0d, 0x80, 0x7d, 0xd0, 0xe2, 0x7f, 0x3c, 0x2c, 0xd5, 0x2e, 0x34, 0xfc, 0x13, 0x97, 0xab,
0x8c, 0xcb, 0x06, 0x34, 0xb7, 0xc7, 0x8a, 0x6f, 0x26, 0x5c, 0x76, 0x6b, 0x7b, 0xe8, 0x68, 0x55, 0x39, 0xce, 0xac,
0xdb, 0x2d, 0x6a, 0x25, 0xad, 0x93, 0x84, 0x2c, 0x81, 0xf6, 0x38, 0xf0, 0xfb, 0xe6, 0x93, 0x3d, 0x9f, 0x7d, 0x7f,
0x5c, 0xaa, 0x6a, 0x7f, 0x70, 0x19, 0x0a, 0xa9, 0xe0, 0x2b, 0x99, 0x31, 0x90, 0x16, 0xf4, 0xa0, 0x54, 0xd3, 0x96,
0x8b, 0x7d, 0x66, 0xa8, 0x34, 0xa1, 0x01, 0xcd, 0xeb, 0xe3, 0x72, 0x6d, 0xad, 0x92, 0x07, 0xe6, 0x9a, 0x6d, 0xf6,
0x5d, 0x5d, 0xd7, 0x0b, 0xb6, 0xd6, 0x46, 0xe9, 0xac, 0x53, 0xbc, 0xd7, 0x5b, 0x52, 0x76, 0xb7, 0xd2, 0x6a, 0x2d,
0xab, 0x70, 0x14, 0x4a, 0x6a, 0x3a, 0x05, 0xb6, 0x58, 0x2a, 0x5d, 0x81, 0xce, 0xe2, 0x91, 0x08, 0x35, 0xad, 0xf8,
0xda, 0x64, 0x78, 0xaa, 0xa1, 0x5d, 0x0c, 0xee, 0x24, 0x71, 0xfc, 0xfd, 0xe2, 0xe4, 0x79, 0x7c, 0xe9, 0x37, 0x4e,
0x9d, 0x94, 0xe0, 0x12, 0xc2, 0xa1, 0x57, 0x66, 0x29, 0xbe, 0xd2, 0xd0, 0x1e, 0x5b, 0xca, 0xe5, 0xa5, 0xf7, 0xae,
0xa2, 0x16, 0x2d, 0x97, 0xc3, 0x28, 0xcd, 0xd2, 0x79, 0xdc, 0xed, 0x16, 0xe3, 0xe4, 0xca, 0xb8, 0xec, 0x11, 0xfa,
0xf9, 0x75, 0x3c, 0x15, 0xc9, 0xe1, 0xe3, 0xda, 0x58, 0x5e, 0xef, 0xc3, 0x71, 0x24, 0x67, 0xa6, 0xa3, 0x0c, 0xc2,
0x25, 0xd8, 0x2d, 0x80, 0x5c, 0xf4, 0xb0, 0x21, 0xb7, 0xd0, 0x9a, 0x53, 0x6a, 0x4e, 0x70, 0xb5, 0x80, 0xdd, 0x19,
0xa6, 0xaf, 0xe5, 0xc3, 0x7f, 0x96, 0x76, 0x05, 0x76, 0x68, 0xa9, 0x5e, 0x71, 0x19, 0x2e, 0x95, 0xb5, 0xaa, 0xcd,
0xc2, 0xa7, 0xdd, 0x6e, 0x31, 0x6e, 0x39, 0xb0, 0x2c, 0x89, 0xbb, 0xdd, 0xb1, 0x1f, 0xc3, 0xa7, 0x7c, 0x5f, 0xd5,
0xcf, 0x68, 0x12, 0x7f, 0x21, 0xc7, 0x55, 0x5d, 0xa7, 0xcb, 0xfa, 0x94, 0xe3, 0xa4, 0xdb, 0x21, 0xa3, 0x04, 0xaf,
0x46, 0xb8, 0x1e, 0x09, 0xc5, 0xe7, 0xd4, 0x26, 0xb3, 0x6e, 0x87, 0x92, 0xcb, 0xcc, 0xb8, 0x89, 0xea, 0xa6, 0x8e,
0xab, 0xb5, 0x22, 0x8f, 0x86, 0x97, 0x8e, 0xab, 0x8c, 0x22, 0x77, 0x19, 0x2e, 0xf2, 0x26, 0x41, 0xbc, 0x22, 0x2d,
0x65, 0xc5, 0x45, 0xdb, 0xcb, 0xa3, 0x26, 0x39, 0xb1, 0x9a, 0xa4, 0x78, 0xd0, 0xd2, 0x06, 0x5e, 0xef, 0x7d, 0x71,
0xad, 0xa4, 0x04, 0x66, 0xb9, 0x5c, 0x21, 0xab, 0xd0, 0x98, 0x02, 0x8c, 0x71, 0xbe, 0xd4, 0xc5, 0x5b, 0x01, 0xd4,
0x00, 0xda, 0x52, 0x6e, 0x71, 0x1e, 0x0d, 0xf2, 0x43, 0x13, 0xe0, 0x15, 0x91, 0x60, 0xcf, 0xd7, 0xbe, 0x99, 0x0e,
0x06, 0x6e, 0xc0, 0x3a, 0x24, 0x67, 0x60, 0x5a, 0xe4, 0x6e, 0x3a, 0x23, 0xda, 0x5f, 0x60, 0x12, 0x6d, 0x79, 0xcd,
0xdd, 0xeb, 0xa6, 0xc8, 0xfb, 0x22, 0x77, 0x08, 0x2e, 0xcf, 0xc3, 0xd3, 0xab, 0xa7, 0x04, 0xc8, 0x95, 0x6d, 0xc8,
0x34, 0x45, 0x9d, 0xa0, 0x0c, 0x1a, 0x25, 0x2a, 0xd0, 0xe4, 0xe6, 0xe6, 0xd7, 0x9f, 0x0b, 0xe7, 0xcc, 0xbd, 0x5e,
0x67, 0xee, 0x06, 0x35, 0x47, 0x8c, 0x5a, 0xf3, 0xab, 0xe1, 0xc1, 0xd5, 0x51, 0x63, 0xb6, 0x4a, 0x57, 0x0f, 0x30,
0xde, 0x8e, 0x9b, 0x03, 0x4e, 0xff, 0xef, 0xaf, 0x4a, 0x71, 0x43, 0x37, 0x90, 0x47, 0xe3, 0x22, 0x8f, 0x9c, 0xc3,
0x03, 0xbf, 0x19, 0xe5, 0x9a, 0xa4, 0xf8, 0xe3, 0xf6, 0x25, 0xfa, 0xb3, 0xab, 0xa8, 0x85, 0x21, 0x6d, 0x7d, 0x54,
0x2d, 0xd8, 0x46, 0x55, 0xe4, 0xed, 0x1f, 0x37, 0xb7, 0xe7, 0x08, 0xd7, 0xbd, 0x10, 0x02, 0xc9, 0x86, 0xa7, 0xdf,
0x5a, 0x58, 0xde, 0x51, 0x6d, 0x7b, 0xd8, 0xd0, 0x35, 0x98, 0x53, 0x0c, 0x3d, 0xbf, 0xe6, 0x02, 0x86, 0x30, 0x06,
0xc5, 0x02, 0x9d, 0xbc, 0x3a, 0x59, 0xfb, 0xcc, 0xaf, 0x68, 0x38, 0xed, 0x68, 0x38, 0xfa, 0xa8, 0x7f, 0x05, 0xff,
0x0b, 0x54, 0xcf, 0x54, 0x8b, 0x15, 0x0b, 0x00, 0x00};
#else // Brotli (default, smaller)
constexpr uint8_t INDEX_BR[] PROGMEM = {
0x1b, 0x08, 0x0b, 0x00, 0xe4, 0x7f, 0x9b, 0xad, 0xbb, 0x97, 0x53, 0xde, 0xb7, 0x25, 0x0e, 0x69, 0xd4, 0x69, 0x89,
0xba, 0xa5, 0x55, 0x22, 0x04, 0x27, 0xeb, 0x00, 0x52, 0xac, 0xbc, 0xec, 0x5f, 0xfb, 0xb5, 0x7a, 0x12, 0x0a, 0xd6,
0x48, 0x84, 0x4a, 0x48, 0x5a, 0xcb, 0xbd, 0xb7, 0x72, 0x66, 0x4e, 0x62, 0xd8, 0xbd, 0x7f, 0x88, 0x68, 0x86, 0x28,
0xd6, 0xf1, 0xda, 0x2c, 0xa2, 0x32, 0x5c, 0xdd, 0xab, 0xb2, 0x15, 0xbc, 0x24, 0x13, 0xe6, 0xbd, 0x0c, 0x52, 0x63,
0x82, 0x88, 0x6d, 0x74, 0x71, 0x51, 0x9c, 0xe2, 0x40, 0x56, 0xea, 0xb0, 0x5c, 0xb7, 0x1d, 0x81, 0x3a, 0x0c, 0xf2,
0xd7, 0xa8, 0x5c, 0x35, 0x2d, 0x43, 0xb3, 0x42, 0x37, 0x7c, 0x60, 0xd8, 0xc1, 0x95, 0x91, 0x94, 0x3c, 0x29, 0x20,
0x96, 0x20, 0xf6, 0x24, 0xb7, 0x83, 0x4a, 0x06, 0xf1, 0xc3, 0x2f, 0x88, 0x50, 0x55, 0x35, 0x2d, 0xe8, 0xb6, 0x21,
0x38, 0x61, 0x8e, 0x44, 0x2a, 0xac, 0x39, 0xcf, 0x74, 0xaa, 0x31, 0x7f, 0x62, 0x32, 0x9b, 0x99, 0x42, 0x0a, 0xf6,
0x6f, 0xd8, 0x89, 0x3f, 0x49, 0xb1, 0xa2, 0x52, 0x0a, 0x8e, 0xb3, 0x27, 0x0b, 0x07, 0x07, 0x58, 0xb0, 0x8f, 0xaa,
0xdc, 0x68, 0x12, 0x0f, 0x95, 0xbf, 0xc7, 0x36, 0xd5, 0x60, 0x5a, 0xc7, 0x80, 0xac, 0x52, 0xf7, 0xa7, 0x2d, 0xb6,
0x64, 0xda, 0xda, 0x11, 0x8d, 0x6a, 0xe5, 0xba, 0xe9, 0xda, 0xdc, 0x62, 0xc3, 0xcb, 0xae, 0xc1, 0xe1, 0x11, 0xb6,
0x33, 0x29, 0x04, 0x09, 0xfe, 0x09, 0x08, 0xc2, 0x1f, 0x98, 0x16, 0x26, 0x0d, 0xce, 0xd7, 0x75, 0xfb, 0xd7, 0xa5,
0x82, 0xd7, 0x32, 0x44, 0x72, 0xc1, 0xc2, 0xe4, 0x15, 0xcb, 0x50, 0xfc, 0xd3, 0x58, 0x64, 0x34, 0x41, 0x32, 0xbe,
0x1f, 0x08, 0x43, 0x16, 0x14, 0xf7, 0x70, 0x2c, 0x1c, 0x61, 0x09, 0x78, 0x73, 0xd1, 0x30, 0xf6, 0xed, 0x85, 0x55,
0x50, 0x02, 0xf0, 0x68, 0x50, 0x5c, 0x1a, 0xd7, 0x3b, 0x8e, 0xf2, 0x23, 0xc8, 0x88, 0x39, 0xd0, 0x84, 0xf7, 0xa6,
0xd1, 0xa3, 0xfb, 0xe5, 0x44, 0xc0, 0x4c, 0x2f, 0x6f, 0x19, 0x70, 0x75, 0xf6, 0x1c, 0xb8, 0xce, 0x89, 0x4f, 0x01,
0x8d, 0xa1, 0x71, 0xf2, 0x97, 0xf8, 0xe7, 0xd3, 0xf1, 0xe1, 0xfa, 0xfc, 0xc8, 0x03, 0x78, 0x14, 0xa0, 0x3d, 0x28,
0xb7, 0xeb, 0x26, 0x52, 0x59, 0xa8, 0xb5, 0x0d, 0x5c, 0x8a, 0x6b, 0xe5, 0xf6, 0x30, 0xd6, 0xf7, 0x71, 0x31, 0xe8,
0xbd, 0xc9, 0xfa, 0xf5, 0x71, 0x9d, 0xff, 0xf6, 0x69, 0x62, 0x5f, 0xb5, 0xc7, 0x06, 0x43, 0x54, 0xb5, 0x87, 0xf1,
0xcc, 0x84, 0xe8, 0xb7, 0x56, 0x38, 0x43, 0x6a, 0xa6, 0x2f, 0x53, 0xd1, 0x89, 0x48, 0x8d, 0x72, 0x75, 0x4a, 0x17,
0xf6, 0x8d, 0xb2, 0xeb, 0xb1, 0x6b, 0x29, 0x1e, 0xd5, 0x74, 0xc7, 0xb3, 0xf4, 0xf1, 0x04, 0x0d, 0xbf, 0x08, 0x81,
0x8f, 0xfe, 0x8d, 0xfc, 0xf2, 0x35, 0x92, 0xa0, 0x24, 0x88, 0x12, 0x6a, 0xe6, 0x2f, 0x01, 0x94, 0x5c, 0x4b, 0xf9,
0x6b, 0x9a, 0xf6, 0x1a, 0x35, 0x11, 0x8a, 0xc6, 0x04, 0x8d, 0x50, 0x64, 0x48, 0x2d, 0x8b, 0xdf, 0xdf, 0xa1, 0xd1,
0xfd, 0x21, 0xd7, 0x40, 0x96, 0x00, 0xb1, 0x9f, 0x54, 0xc2, 0xe1, 0x00, 0x79, 0x15, 0x80, 0xf8, 0xca, 0x8e, 0xc5,
0x06, 0x03, 0xdf, 0x72, 0x49, 0xc0, 0x08, 0x27, 0x90, 0xe3, 0x14, 0xc2, 0xac, 0xc3, 0x83, 0xc9, 0xf6, 0x9d, 0x12,
0xab, 0x46, 0xae, 0x03, 0x74, 0x1d, 0x27, 0xce, 0x91, 0x1d, 0x4e, 0xb4, 0xbe, 0x80, 0xe7, 0x6a, 0x6d, 0x0a, 0x20,
0x47, 0x6b, 0x00, 0x4e, 0x25, 0x52, 0xe6, 0xf5, 0xe9, 0x43, 0x84, 0xc1, 0x0d, 0x4b, 0x04, 0xb3, 0x91, 0x49, 0xf5,
0xe9, 0xc4, 0x2e, 0x1b, 0xf9, 0x98, 0xf9, 0xea, 0x9e, 0xb8, 0xf6, 0x53, 0xf8, 0x42, 0xc2, 0x60, 0x5c, 0xa9, 0xd2,
0xfe, 0x0a, 0xe5, 0xd4, 0x7d, 0x36, 0x76, 0x04, 0x12, 0x38, 0xa1, 0xc4, 0x30, 0xb8, 0xf2, 0x69, 0x86, 0x5b, 0xe7,
0xe5, 0xa0, 0xc0, 0xfe, 0x91, 0x19, 0x03, 0x1e, 0xa9, 0x93, 0x26, 0x49, 0x58, 0xd5, 0xf6, 0x33, 0x4e, 0x0a, 0x89,
0x64, 0x1e, 0xb4, 0x7a, 0x5d, 0x0d, 0x12, 0xcf, 0x2b, 0xdd, 0x35, 0x90, 0x55, 0xc3, 0x0e, 0xcd, 0x0e, 0xad, 0x6b,
0x8f, 0x43, 0xd0, 0xb4, 0x6a, 0xdb, 0x4b, 0xe5, 0xa4, 0x9b, 0x09, 0x23, 0x1a, 0x43, 0xa9, 0xff, 0x61, 0x8b, 0x07,
0xd6, 0x0f, 0x83, 0x23, 0xbe, 0x8a, 0x66, 0xa2, 0x10, 0x2f, 0x11, 0x01, 0x0d, 0xc6, 0x96, 0xba, 0x87, 0xaa, 0xa8,
0x44, 0x7c, 0x1e, 0x34, 0xdb, 0xba, 0x41, 0x90, 0xf0, 0x7a, 0xdb, 0x63, 0x60, 0x96, 0x8d, 0x64, 0xcb, 0x2f, 0x28,
0x96, 0x04, 0xd5, 0xc0, 0x7a, 0xe8, 0xec, 0xd1, 0x61, 0x1b, 0x09, 0x4b, 0x4c, 0x6e, 0x1b, 0x31, 0x98, 0x9c, 0x6d,
0xdb, 0xd0, 0xec, 0xa4, 0x0f, 0x0a, 0xe0, 0xc8, 0x35, 0xc4, 0x93, 0xdc, 0xee, 0x19, 0x00, 0x80, 0x07, 0xf5, 0xcf,
0xe0, 0x7f, 0x75, 0xf8, 0x52, 0x3a, 0xfc, 0x0d, 0x84, 0xa5, 0xc1, 0xb2, 0x1c, 0x25, 0x40, 0xc5, 0x8b, 0xb3, 0xdb,
0x7a, 0x1b, 0x44, 0xff, 0x79, 0x9a, 0x26, 0xc4, 0xe7, 0x9e, 0x78, 0x4c, 0xa5, 0x94, 0xab, 0x78, 0x34, 0x9d, 0xb5,
0xb7, 0x24, 0x26, 0xe7, 0x9a, 0xf3, 0xe5, 0x2a, 0x35, 0x4b, 0xbe, 0x74, 0xd7, 0x01, 0xbc, 0x71, 0x72, 0x03, 0x5c,
0x40, 0x24, 0x17, 0x61, 0x5b, 0x7b, 0x19, 0xc2, 0x7f, 0x4e, 0x5b, 0x24, 0xfb, 0x8b, 0xc3, 0x51, 0x00, 0x3d, 0x09,
0x04, 0x05, 0x72, 0x64, 0xf6, 0xf0, 0x6b, 0x99, 0x38, 0x93, 0x5b, 0xac, 0x0c, 0xff, 0x40, 0x7b, 0x53, 0xba, 0xab,
0x61, 0xc9, 0xa2, 0xde, 0xd6, 0x2b, 0x0c, 0xbd, 0x85, 0xac, 0x4c, 0x64, 0x8b, 0xe6, 0x69, 0x2b, 0x56, 0x10, 0x1b,
0x08, 0x59, 0x6c, 0x55, 0x0a, 0xaa, 0x93, 0x85, 0x1d, 0x45, 0xda, 0xc6, 0x39, 0xe6, 0x6a, 0xab, 0x71, 0x73, 0x46,
0x0f, 0xf7, 0xab, 0x4e, 0x88, 0xae, 0x8c, 0xe0, 0x91, 0xda, 0x35, 0x8c, 0xd3, 0x18, 0x92, 0x3d, 0xab, 0x2f, 0x0d,
0xd6, 0xc9, 0x06, 0xdb, 0xc2, 0x62, 0xcb, 0x8a, 0x23, 0x5b, 0x28, 0x2e, 0x9b, 0x96, 0xc4, 0xc1, 0x42, 0xa9, 0x8d,
0x69, 0xe5, 0x8f, 0x89, 0x12, 0x11, 0x9a, 0x56, 0x3b, 0x3c, 0x2b, 0xb4, 0xb2, 0x7b, 0xfd, 0xdb, 0x80, 0x92, 0xb4,
0xca, 0x44, 0x37, 0x8c, 0x94, 0x2f, 0x4a, 0xb4, 0xc4, 0x28, 0x83, 0x8a, 0x78, 0xcb, 0xd2, 0x5c, 0x5c, 0x35, 0x29,
0x95, 0x35, 0x2f, 0x99, 0x28, 0x4f, 0x22, 0x18, 0x70, 0x85, 0xee, 0x2c, 0xb9, 0xef, 0x14, 0x57, 0x73, 0x23, 0x45,
0xae, 0xdc, 0x54, 0x56, 0x55, 0x78, 0x56, 0xad, 0x48, 0x26, 0x27, 0xbf, 0xcb, 0xd7, 0x54, 0xa9, 0xa8, 0xd7, 0xb5,
0x71, 0x9c, 0xe7, 0x79, 0x89, 0x5c, 0xa9, 0x6a, 0x53, 0xe8, 0xfa, 0x0d, 0x69, 0x36, 0xed, 0xad, 0x33, 0xc9, 0x75,
0x17, 0xe9, 0x8f, 0x31, 0x58, 0xa6, 0x27, 0xfc, 0x79, 0xc6, 0xb6, 0xd5, 0x65, 0x90, 0x31, 0xc6, 0x9d, 0x29, 0x95,
0x83, 0xe5, 0x4e, 0xbb, 0xd7, 0xdc, 0x8e, 0xd0, 0x3a, 0x54, 0x27, 0xce, 0xf5, 0xdb, 0xbd, 0x89, 0x3c, 0x61, 0xb3,
0x71, 0x23, 0xc7, 0x55, 0x9e, 0xea, 0x50, 0xe4, 0x37, 0xae, 0x72, 0x34, 0x66, 0xb9, 0x6e, 0x2c, 0x0a, 0x69, 0x8d,
0x94, 0x8b, 0x98, 0x08, 0x05, 0x3c, 0x45, 0x45, 0xe1, 0x6c, 0x22, 0xc5, 0x8f, 0x1f, 0x9f, 0x3f, 0xd2, 0x01, 0x12,
0xec, 0x86, 0x2f, 0x87, 0x8b, 0x47, 0x30, 0xd0, 0x77, 0x77, 0x1a, 0x13, 0x2d, 0xc6, 0x31, 0x65, 0x77, 0xd8, 0x8c,
0xe7, 0xe0, 0x16, 0xf9, 0x4b, 0xd4, 0xc5, 0xa8, 0x67, 0x79, 0xe7, 0xc3, 0x07, 0x94, 0x46, 0xa3, 0x8c, 0x67, 0xd3,
0xff, 0x5f, 0x96, 0xfa, 0xed, 0xa7, 0xd3, 0xdd, 0x4f, 0x27, 0x49, 0x27, 0xcf, 0xa4, 0x02, 0xc3, 0x27, 0x3c, 0x96,
0x64, 0x24, 0x55, 0xc8, 0x36, 0x45, 0x68, 0x90, 0xea, 0x03, 0x0b, 0x46, 0xa7, 0x8d, 0xb4, 0x26, 0x91, 0x07, 0x4c,
0x80, 0x7b, 0x93, 0xa8, 0x10, 0xcb, 0x5e, 0x8d, 0x42, 0x86, 0x3e, 0x2a, 0x7c, 0x99, 0x79, 0x8e, 0xcb, 0x43, 0x28};
0x1b, 0x14, 0x0b, 0x00, 0xe4, 0x6f, 0xcd, 0xfc, 0x7b, 0x2e, 0x27, 0xd2, 0x21, 0x2b, 0x58, 0xea, 0x16, 0xf8, 0xa5,
0xb6, 0x47, 0x14, 0xb3, 0x4c, 0x56, 0xcc, 0x20, 0x5b, 0x1d, 0xfe, 0x7d, 0xfb, 0xb5, 0x7a, 0x12, 0x0a, 0xd6, 0x48,
0x84, 0xa2, 0x21, 0x69, 0x0d, 0x77, 0x33, 0xf3, 0x77, 0xcf, 0x4c, 0x21, 0xf1, 0xde, 0xee, 0x7d, 0x44, 0xbc, 0xd3,
0xc4, 0x33, 0x89, 0x1a, 0x69, 0x68, 0x48, 0x57, 0xa7, 0x17, 0x0b, 0x4d, 0x91, 0xac, 0x30, 0x48, 0x21, 0x4d, 0x4c,
0x10, 0x57, 0x46, 0x14, 0x17, 0xd5, 0x21, 0x0e, 0xb4, 0x50, 0x87, 0xad, 0x62, 0xda, 0x11, 0x68, 0xc3, 0x20, 0x7f,
0x2d, 0xdc, 0x57, 0xeb, 0x2c, 0x36, 0xdb, 0xc4, 0x84, 0x0f, 0xac, 0xec, 0x08, 0x91, 0xa3, 0x92, 0x27, 0x45, 0xc4,
0x1e, 0xa4, 0x9e, 0x72, 0x3b, 0xc2, 0xe3, 0x20, 0x7d, 0xf8, 0x05, 0x09, 0xea, 0xe3, 0xa6, 0x3f, 0x11, 0x8b, 0x16,
0x11, 0x6b, 0x26, 0x91, 0x1a, 0x2c, 0x19, 0x24, 0xf7, 0x5d, 0x44, 0x82, 0x49, 0x44, 0x15, 0xce, 0x39, 0xdc, 0xcb,
0x8f, 0x5b, 0xe1, 0xe2, 0x42, 0x96, 0xa2, 0x10, 0x3c, 0x20, 0xa5, 0x65, 0x85, 0x49, 0x6a, 0x06, 0x08, 0xd1, 0xcb,
0x41, 0xb8, 0x49, 0x20, 0x73, 0x71, 0xfe, 0x55, 0x61, 0x45, 0xc6, 0x95, 0x72, 0xc8, 0xf0, 0xa5, 0xe9, 0xe6, 0x3a,
0xb9, 0xc3, 0x8a, 0x9f, 0xb5, 0xc1, 0xc9, 0x15, 0x56, 0x93, 0x38, 0x8a, 0x48, 0xf0, 0x4f, 0x38, 0x22, 0xe1, 0x8d,
0x55, 0xbb, 0x8c, 0xc3, 0xb0, 0x68, 0xcc, 0x7f, 0x6f, 0xf8, 0xc9, 0x9b, 0x38, 0x41, 0xf1, 0x94, 0x25, 0xf9, 0x6b,
0x56, 0xa2, 0xec, 0xa7, 0xbd, 0x2e, 0x69, 0x8e, 0xe2, 0xec, 0x36, 0x94, 0x24, 0x2c, 0x12, 0x1d, 0x4e, 0x76, 0x7e,
0x23, 0xcc, 0xf2, 0x6e, 0x35, 0x1a, 0x9c, 0xed, 0x6f, 0x15, 0x3f, 0xc9, 0x72, 0xdc, 0xfd, 0x13, 0x0f, 0x16, 0x8a,
0x23, 0x4f, 0xf9, 0x2e, 0x63, 0x44, 0x91, 0x77, 0xe0, 0xb3, 0xd1, 0x78, 0x74, 0xdb, 0x4a, 0x2c, 0xd8, 0xe8, 0xf9,
0x2c, 0x03, 0xbe, 0xae, 0xac, 0x4e, 0x42, 0x01, 0xc4, 0x4b, 0x40, 0xe7, 0x94, 0x34, 0x85, 0x2c, 0xfe, 0xf5, 0x74,
0x75, 0xd8, 0xdc, 0xec, 0x6a, 0x00, 0x3f, 0x3f, 0x81, 0x77, 0xa8, 0xcd, 0xde, 0x6d, 0x5a, 0x47, 0xa1, 0x99, 0x36,
0x87, 0xb6, 0x78, 0x35, 0x3c, 0x9a, 0x64, 0x15, 0x7c, 0x56, 0x76, 0x22, 0xce, 0x46, 0xe5, 0x17, 0x57, 0x05, 0xfc,
0x09, 0x69, 0xae, 0x39, 0xaa, 0xee, 0xc9, 0x4e, 0x7f, 0x99, 0x2a, 0x65, 0x82, 0x7e, 0xeb, 0x23, 0x4f, 0x42, 0xd5,
0xca, 0xcb, 0x42, 0x74, 0x2e, 0xd2, 0xa2, 0x62, 0x57, 0xd0, 0xa9, 0x7b, 0x4b, 0xac, 0x7b, 0x65, 0x13, 0x47, 0x0f,
0x2d, 0x3d, 0xf6, 0xbc, 0x78, 0x5c, 0xa3, 0xc9, 0x57, 0x4b, 0x10, 0x62, 0x68, 0x19, 0x7f, 0xfd, 0x1a, 0xcb, 0x51,
0x1e, 0x41, 0x39, 0x55, 0xf3, 0x97, 0x30, 0xca, 0x37, 0xb6, 0x42, 0x1d, 0x2d, 0x8c, 0xc6, 0x65, 0x8a, 0xd2, 0x89,
0x88, 0xa6, 0x28, 0xbd, 0x56, 0x7c, 0x2d, 0x5e, 0x0d, 0x2f, 0x9e, 0xc3, 0xa5, 0x80, 0xaf, 0xcc, 0x00, 0xde, 0xe6,
0x5a, 0x8b, 0xda, 0xff, 0x1f, 0x16, 0xc8, 0x83, 0x5e, 0xe5, 0xea, 0x25, 0x86, 0x70, 0x55, 0x25, 0x01, 0x14, 0x0c,
0x77, 0xd8, 0x31, 0x21, 0xcc, 0x79, 0x15, 0x3b, 0x32, 0x3a, 0xd3, 0xc5, 0x30, 0xaf, 0x03, 0xd8, 0x3e, 0x44, 0xb8,
0x63, 0xb5, 0x74, 0x4f, 0x41, 0x4b, 0xf0, 0x24, 0x74, 0xb2, 0x06, 0xb2, 0x7b, 0x06, 0x60, 0xdc, 0xc1, 0x3e, 0x0e,
0x6f, 0x1e, 0x3c, 0x42, 0xa0, 0xdb, 0x36, 0x43, 0x30, 0x71, 0xcc, 0xd6, 0x1a, 0xbd, 0x38, 0x61, 0x19, 0x3f, 0xf2,
0xdf, 0xf4, 0x53, 0x3d, 0xc1, 0x14, 0xbe, 0x50, 0x1c, 0x2c, 0xf3, 0xaa, 0x74, 0x36, 0xcb, 0xbd, 0x7a, 0x4b, 0xa3,
0x1c, 0x90, 0x40, 0x5b, 0x4a, 0x0f, 0x83, 0x6e, 0x9e, 0x96, 0x28, 0x3d, 0x77, 0x43, 0x05, 0x0e, 0x39, 0x26, 0x15,
0xb8, 0x6b, 0x4e, 0x3a, 0x62, 0xc2, 0xda, 0xde, 0x8e, 0x29, 0x29, 0x0b, 0x09, 0xa2, 0xb3, 0xa7, 0x1e, 0x9a, 0x0c,
0x8f, 0xa1, 0x46, 0x6f, 0x92, 0xf3, 0x9e, 0xb5, 0xc5, 0x7e, 0x0e, 0x8d, 0xeb, 0x55, 0x08, 0xfa, 0xc9, 0xd8, 0x4e,
0xe3, 0xd0, 0xca, 0x2a, 0x96, 0x11, 0x7e, 0xb1, 0xd4, 0x7f, 0x8f, 0x1d, 0xb3, 0xc3, 0xa0, 0x89, 0x6f, 0x93, 0x99,
0x55, 0x48, 0x97, 0x08, 0x79, 0x66, 0xe9, 0x4a, 0x6b, 0xa0, 0x29, 0xf2, 0x10, 0x1f, 0x22, 0xa2, 0x1b, 0x41, 0xdf,
0xa3, 0xde, 0x62, 0x60, 0x8e, 0xa1, 0x60, 0xc4, 0xd5, 0xce, 0x81, 0x47, 0x84, 0x3b, 0x66, 0x60, 0x74, 0xa7, 0x74,
0xcc, 0x48, 0xe0, 0xe1, 0xd5, 0x0c, 0x15, 0x98, 0x3d, 0xa7, 0x9c, 0xb2, 0xec, 0xba, 0x0f, 0x2c, 0x52, 0x14, 0x7b,
0xe2, 0x49, 0x6e, 0x0f, 0x8c, 0x00, 0xe0, 0x81, 0xf6, 0x57, 0xe4, 0x3f, 0xbf, 0x7c, 0xa9, 0x5e, 0xfe, 0x01, 0xc2,
0x64, 0xb0, 0x05, 0x60, 0x01, 0xaa, 0x78, 0x65, 0xb2, 0xeb, 0x56, 0x41, 0xf2, 0xbf, 0xa3, 0x45, 0x4e, 0x3c, 0x78,
0xe2, 0xa1, 0x0d, 0xa9, 0x2a, 0xc0, 0x8a, 0xc0, 0x6f, 0xe4, 0x66, 0xbe, 0x72, 0x35, 0x5e, 0xf7, 0x3b, 0x42, 0x53,
0xd4, 0xe6, 0x66, 0xb6, 0x78, 0xcd, 0xaa, 0x6f, 0xf4, 0x26, 0x80, 0x3a, 0x4e, 0x74, 0x80, 0x17, 0x88, 0x44, 0x23,
0xa6, 0x3a, 0x6f, 0x87, 0xb8, 0xd0, 0x4d, 0xd3, 0xfc, 0x7c, 0xd6, 0x38, 0x2a, 0x00, 0x28, 0x01, 0xa2, 0x40, 0x94,
0x6c, 0x1e, 0x8a, 0xed, 0xe3, 0x92, 0x9d, 0x90, 0xe3, 0x0d, 0x02, 0x4e, 0x15, 0x30, 0xed, 0x8f, 0x5b, 0x99, 0xaa,
0x7a, 0x4e, 0xb9, 0xec, 0x91, 0xe2, 0x9f, 0xd4, 0xca, 0x46, 0xaf, 0x87, 0x19, 0x4b, 0xad, 0xea, 0xe6, 0x6c, 0x8d,
0x53, 0x4b, 0x29, 0xee, 0x1e, 0x96, 0xd8, 0x94, 0x30, 0x3a, 0x9c, 0xb0, 0x4c, 0xdb, 0xe2, 0xa1, 0x7f, 0xc7, 0x11,
0xdd, 0xe3, 0x9d, 0x36, 0x44, 0xd3, 0x93, 0x14, 0x9c, 0x4c, 0x9d, 0xdd, 0x3a, 0x7c, 0x41, 0xb1, 0x8f, 0x14, 0xd9,
0x4e, 0x61, 0xd9, 0x3a, 0xe3, 0x0d, 0x76, 0xca, 0x6c, 0xac, 0x73, 0xaf, 0xad, 0x94, 0x87, 0x30, 0xf1, 0x30, 0x2f,
0x8b, 0xed, 0x4a, 0xed, 0xbc, 0xc2, 0xf2, 0x7c, 0x30, 0xa3, 0x0b, 0x28, 0x64, 0x3b, 0x8c, 0xd4, 0xc3, 0x42, 0xb9,
0xa3, 0x44, 0x51, 0x80, 0x07, 0x5a, 0x3d, 0x14, 0x33, 0x99, 0xbf, 0x2a, 0x6b, 0x2b, 0x19, 0x47, 0x72, 0x9e, 0xd4,
0xb4, 0x6d, 0x72, 0xdd, 0x8a, 0x4b, 0x33, 0x55, 0xbc, 0xb4, 0xcd, 0xc8, 0x2b, 0x17, 0x2f, 0x74, 0xeb, 0x22, 0x17,
0x94, 0x08, 0x27, 0x27, 0xc2, 0x5b, 0x17, 0xb4, 0xa9, 0x22, 0x16, 0x9d, 0xd4, 0xfc, 0xc7, 0x15, 0xa3, 0x9b, 0x86,
0x1f, 0xad, 0x45, 0xd3, 0x87, 0x94, 0x5b, 0x31, 0x36, 0xaa, 0xe4, 0x66, 0x8d, 0xcc, 0x31, 0x05, 0x5b, 0xc4, 0x40,
0xc0, 0xb8, 0xeb, 0x91, 0x18, 0x22, 0x8c, 0x31, 0x1e, 0xad, 0xd0, 0x3a, 0x98, 0x07, 0xb5, 0x6f, 0x11, 0xba, 0x11,
0xa6, 0x14, 0x35, 0x5a, 0xe7, 0x55, 0xdf, 0xb7, 0x4c, 0x03, 0x61, 0xa3, 0x74, 0x23, 0xdf, 0x55, 0x1f, 0x02, 0x51,
0x09, 0xb7, 0xba, 0xd5, 0x0c, 0x67, 0xab, 0x98, 0x70, 0x14, 0x64, 0x8d, 0xf4, 0x8b, 0x54, 0x44, 0x07, 0x6f, 0xe0,
0x69, 0x32, 0xca, 0x48, 0xe5, 0xd3, 0xa7, 0x17, 0x8f, 0x45, 0x84, 0x04, 0xb7, 0xd1, 0xbb, 0xe1, 0xf6, 0x01, 0x0a,
0xf6, 0xee, 0x2b, 0x32, 0xd2, 0xc5, 0xf8, 0xa6, 0xec, 0x0f, 0x1b, 0x09, 0x1d, 0xfc, 0xa2, 0xbf, 0x54, 0x5d, 0x2c,
0x62, 0xf4, 0x77, 0xde, 0xad, 0xa0, 0x50, 0x6a, 0xb4, 0xe3, 0x5f, 0xda, 0xff, 0x6b, 0xb1, 0x78, 0xf7, 0xf9, 0xc1,
0xa6, 0xa8, 0x93, 0xe8, 0xe4, 0x11, 0x56, 0xa0, 0x5b, 0x85, 0xa7, 0x92, 0x7a, 0x58, 0x45, 0x95, 0xa9, 0x63, 0x83,
0xb4, 0x1f, 0x18, 0x31, 0x7a, 0x6d, 0xa1, 0x8d, 0x8c, 0xdc, 0x91, 0x02, 0x3c, 0x9c, 0x92, 0x42, 0x8e, 0x03, 0x02,
0xc5, 0x0c, 0x43, 0x54, 0xf9, 0xb2, 0x85, 0x39, 0x2e, 0x77, 0xad, 0x00};
// Backwards compatibility alias
#define INDEX_GZ INDEX_BR
-9
View File
@@ -10,7 +10,6 @@ import subprocess
from typing import Any
from esphome import yaml_util
from esphome.build_helpers.pch import PCH_PREFIX_HEADER, pch_enabled, pch_extra_scripts
import esphome.codegen as cg
from esphome.components.const import CONF_ENABLE_OTA_DOWNGRADE_PROTECTION
from esphome.config_helpers import filter_source_files_from_defines
@@ -58,7 +57,6 @@ from esphome.coroutine import CoroPriority, coroutine_with_priority
from esphome.espidf.component import generate_idf_components
import esphome.final_validate as fv
from esphome.helpers import copy_file_if_changed, rmtree, write_file_if_changed
from esphome.platformio.toolchain import copy_pch_script
from esphome.schema_extractors import SCHEMA_EXTRACT, schema_extractor
from esphome.types import ConfigType
from esphome.writer import clean_build, clean_cmake_cache
@@ -2435,11 +2433,6 @@ async def to_code(config):
cg.add_platformio_option("lib_ldf_mode", "off")
cg.add_platformio_option("lib_compat_mode", "strict")
# CI-speed only: this toolchain is being dropped, so the pch gets
# the same curated prefix as host with no further investment
cg.add_platformio_option("extra_scripts", pch_extra_scripts())
if pch_enabled():
cg.add_platformio_option("build_src_flags", f"-include {PCH_PREFIX_HEADER}")
cg.add_platformio_option("platform", conf[CONF_PLATFORM_VERSION])
cg.add_platformio_option("board", config[CONF_BOARD])
cg.add_platformio_option("board_upload.flash_size", config[CONF_FLASH_SIZE])
@@ -3357,8 +3350,6 @@ def _write_idf_component_yml():
def copy_files():
_write_sdkconfig()
_write_idf_component_yml()
if not CORE.using_toolchain_esp_idf:
copy_pch_script()
if "partitions.csv" not in CORE.data[KEY_ESP32][KEY_EXTRA_BUILD_FILES]:
flash_size = CORE.data[KEY_ESP32][KEY_FLASH_SIZE]
+41 -191
View File
@@ -3,10 +3,8 @@ from pathlib import Path
import platform
import re
import subprocess
import time
from typing import Any
from esphome.build_helpers.pch import pch_extra_scripts
import esphome.codegen as cg
import esphome.config_validation as cv
from esphome.const import (
@@ -16,7 +14,6 @@ from esphome.const import (
CONF_FRAMEWORK,
CONF_PLATFORM_VERSION,
CONF_SOURCE,
CONF_TOOLCHAIN,
CONF_VERSION,
KEY_CORE,
KEY_FRAMEWORK_VERSION,
@@ -24,7 +21,6 @@ from esphome.const import (
KEY_TARGET_PLATFORM,
PLATFORM_ESP8266,
ThreadModel,
Toolchain,
)
from esphome.core import (
CORE,
@@ -35,13 +31,12 @@ from esphome.core import (
)
from esphome.core.config import BOARD_MAX_LENGTH
from esphome.helpers import IS_MACOS, copy_file_if_changed
from esphome.platformio.toolchain import copy_ccache_script, copy_pch_script
from esphome.platformio.toolchain import copy_ccache_script
from esphome.storage_json import StorageJSON
from esphome.types import ConfigType
from .boards import BOARDS, ESP8266_BOARD_BUILD, ESP8266_LD_SCRIPTS, board_ld_script
from .boards import BOARDS, ESP8266_LD_SCRIPTS, board_ld_script
from .const import (
BUILD_FLASH_MODES,
CONF_EARLY_PIN_INIT,
CONF_ENABLE_SERIAL,
CONF_ENABLE_SERIAL1,
@@ -51,7 +46,6 @@ from .const import (
KEY_FLASH_SIZE,
KEY_LDSCRIPT,
KEY_PIN_INITIAL_STATES,
KEY_SCANF_FLOAT,
KEY_SERIAL1_REQUIRED,
KEY_SERIAL_REQUIRED,
KEY_WAVEFORM_REQUIRED,
@@ -111,53 +105,6 @@ def set_core_data(config: ConfigType) -> ConfigType:
return config
_TOOLCHAINS = (Toolchain.PLATFORMIO, Toolchain.ARDUINO)
_validate_toolchain = cv.toolchain_enum(_TOOLCHAINS)
_resolve_toolchain = cv.resolve_toolchain("ESP8266", _TOOLCHAINS, Toolchain.PLATFORMIO)
def _validate_native_toolchain(config: ConfigType) -> ConfigType:
"""Constraints of the native (non-PlatformIO) Arduino toolchain."""
if not CORE.using_toolchain_arduino:
return config
from esphome.arduino8266.framework import MIN_FRAMEWORK_VERSION
conf = config[CONF_FRAMEWORK]
version = cv.Version.parse(conf[CONF_VERSION])
if version < MIN_FRAMEWORK_VERSION:
raise cv.Invalid(
"'toolchain: arduino' requires framework version "
f"{MIN_FRAMEWORK_VERSION} or newer"
)
# platform_version is a PlatformIO concept; drop it (as esp32's native
# toolchain does), warning when a custom pin is discarded. The floor
# above guarantees the schema-derived default is the ARDUINO_4 spec.
if (
conf.pop(CONF_PLATFORM_VERSION, _ARDUINO_4_PLATFORM_SPEC)
!= _ARDUINO_4_PLATFORM_SPEC
):
_LOGGER.warning(
"'platform_version' is ignored by 'toolchain: arduino'; the native "
"toolchain downloads the framework and compiler directly"
)
if conf[CONF_SOURCE] != _format_framework_arduino_version(version):
raise cv.Invalid(
"'toolchain: arduino' does not support a custom framework source; "
"use 'toolchain: platformio'"
)
# BOARDS is a subset of ESP8266_BOARD_BUILD today; the second clause is
# a drift guard for the independently regenerated tables
if (
config[CONF_BOARD] not in BOARDS
or config[CONF_BOARD] not in ESP8266_BOARD_BUILD
):
raise cv.Invalid(
f"Board '{config[CONF_BOARD]}' is not supported by "
"'toolchain: arduino'; use 'toolchain: platformio'"
)
return config
def get_download_types(storage_json: StorageJSON) -> list[dict[str, str]]:
"""Binary-download entries for a built ESP8266 firmware.
@@ -191,7 +138,7 @@ def _format_framework_arduino_version(ver: cv.Version) -> str:
if ver <= cv.Version(2, 6, 2):
return f"~2.{ver.major}{ver.minor:02d}{ver.patch:02d}.0"
# Same encoding the native toolchain uses for its package download, so a
# custom-source check against this value cannot drift from what it fetches.
# version bump cannot drift between the two paths.
from esphome.arduino8266.framework import framework_package_version
try:
@@ -247,7 +194,7 @@ def _arduino_check_versions(value: ConfigType) -> ConfigType:
platform_version = value.get(CONF_PLATFORM_VERSION)
if platform_version is None:
if version >= cv.Version(3, 1, 0):
platform_version = _ARDUINO_4_PLATFORM_SPEC
platform_version = _parse_platform_version(str(ARDUINO_4_PLATFORM_VERSION))
elif version >= cv.Version(3, 0, 0):
platform_version = _parse_platform_version(str(ARDUINO_3_PLATFORM_VERSION))
elif version >= cv.Version(2, 5, 0):
@@ -274,10 +221,6 @@ def _parse_platform_version(value: Any) -> str:
return value
# The platform_version derived for every core >= 3.1.0 config
_ARDUINO_4_PLATFORM_SPEC = _parse_platform_version(str(ARDUINO_4_PLATFORM_VERSION))
ARDUINO_FRAMEWORK_SCHEMA = cv.All(
cv.Schema(
{
@@ -294,6 +237,7 @@ ARDUINO_FRAMEWORK_SCHEMA = cv.All(
)
BUILD_FLASH_MODES = ["qio", "qout", "dio", "dout"]
CONFIG_SCHEMA = cv.All(
cv.Schema(
{
@@ -310,30 +254,15 @@ CONFIG_SCHEMA = cv.All(
cv.Optional(CONF_ENABLE_SERIAL1): cv.boolean,
cv.Optional(CONF_ENABLE_FULL_PRINTF, default=False): cv.boolean,
cv.Optional(CONF_ENABLE_SCANF_FLOAT): cv.boolean,
cv.Optional(
CONF_TOOLCHAIN, visibility=cv.Visibility.ADVANCED
): _validate_toolchain,
}
),
_resolve_toolchain,
_validate_native_toolchain,
# Until the native toolchain lands, PlatformIO is the only backend;
# reject a --toolchain this platform cannot serve yet.
cv.require_platformio_toolchain("ESP8266"),
set_core_data,
)
def native_toolchain_module():
"""The native build backend for the resolved toolchain, if any.
``__main__`` dispatches from its own toolchain-keyed table; this helper
serves the component's internal callers.
"""
if not CORE.using_toolchain_arduino:
return None
from esphome.arduino8266 import toolchain
return toolchain
def check_rosetta() -> None:
"""Fail fast when the x86_64 ESP8266 toolchain cannot run on this Mac.
@@ -387,13 +316,12 @@ def _choose_ld_script(board: str, ver: cv.Version) -> str | None:
@coroutine_with_priority(CoroPriority.PLATFORM)
async def to_code(config: ConfigType) -> None:
use_platformio = CORE.using_toolchain_platformio
cg.add(esp8266_ns.setup_preferences())
if use_platformio:
cg.add_platformio_option("lib_ldf_mode", "off")
cg.add_platformio_option("lib_compat_mode", "strict")
cg.add_platformio_option("board", config[CONF_BOARD])
cg.add_platformio_option("lib_ldf_mode", "off")
cg.add_platformio_option("lib_compat_mode", "strict")
cg.add_platformio_option("board", config[CONF_BOARD])
cg.add_build_flag("-DUSE_ESP8266")
cg.set_cpp_standard("gnu++20")
cg.add_define("ESPHOME_BOARD", config[CONF_BOARD])
@@ -409,33 +337,28 @@ async def to_code(config: ConfigType) -> None:
"enabling scanf float support (~8KB flash)"
)
# The native generator reads the same decision (KEY_SCANF_FLOAT)
CORE.data[KEY_ESP8266][KEY_SCANF_FLOAT] = bool(enable_scanf_float)
if use_platformio:
extra_scripts = [
"pre:ccache.py",
"pre:testing_mode.py",
"pre:exclude_updater.py",
"pre:exclude_waveform.py",
"pre:relocate_ratetable.py",
]
if not enable_scanf_float:
extra_scripts.append("pre:remove_float_scanf.py")
extra_scripts.extend(pch_extra_scripts())
extra_scripts.append("post:post_build.py")
cg.add_platformio_option("extra_scripts", extra_scripts)
extra_scripts = [
"pre:ccache.py",
"pre:testing_mode.py",
"pre:exclude_updater.py",
"pre:exclude_waveform.py",
"pre:relocate_ratetable.py",
]
if not enable_scanf_float:
extra_scripts.append("pre:remove_float_scanf.py")
extra_scripts.append("post:post_build.py")
cg.add_platformio_option("extra_scripts", extra_scripts)
conf = config[CONF_FRAMEWORK]
cg.add_platformio_option("framework", "arduino")
cg.add_build_flag("-DUSE_ARDUINO")
cg.add_build_flag("-DUSE_ESP8266_FRAMEWORK_ARDUINO")
cg.add_build_flag("-Wno-nonnull-compare")
if use_platformio:
cg.add_platformio_option("framework", "arduino")
cg.add_platformio_option("platform", conf[CONF_PLATFORM_VERSION])
cg.add_platformio_option(
"platform_packages",
[f"platformio/framework-arduinoespressif8266@{conf[CONF_SOURCE]}"],
)
cg.add_platformio_option("platform", conf[CONF_PLATFORM_VERSION])
cg.add_platformio_option(
"platform_packages",
[f"platformio/framework-arduinoespressif8266@{conf[CONF_SOURCE]}"],
)
# Default for platformio is LWIP2_LOW_MEMORY with:
# - MSS=536
@@ -473,8 +396,7 @@ async def to_code(config: ConfigType) -> None:
# Force-include inline std::__throw_* overrides so GCC dead-strips the unused
# libstdc++ error message strings (e.g. "basic_string::_M_create") from DRAM.
# See throw_stubs.h for details. Must be prepended before <string>, so this
# uses build_src_flags with -include. Unconditional: the native build
# generator reads the same option, keeping one source of truth.
# uses build_src_flags with -include.
cg.add_platformio_option(
"build_src_flags", "-include esphome/components/esp8266/throw_stubs.h"
)
@@ -504,8 +426,6 @@ async def to_code(config: ConfigType) -> None:
# implementation in the Arduino ESP8266 core.
cg.add_build_flag("-Wl,--wrap=millis")
# Unconditional: the native build generator reads the same option,
# keeping one source of truth
cg.add_platformio_option("board_build.flash_mode", config[CONF_BOARD_FLASH_MODE])
ver: cv.Version = CORE.data[KEY_CORE][KEY_FRAMEWORK_VERSION]
@@ -514,7 +434,7 @@ async def to_code(config: ConfigType) -> None:
cg.RawExpression(f"VERSION_CODE({ver.major}, {ver.minor}, {ver.patch})"),
)
if use_platformio and config[CONF_BOARD] in BOARDS:
if config[CONF_BOARD] in BOARDS:
ld_script = _choose_ld_script(config[CONF_BOARD], ver)
if ld_script is not None:
@@ -556,24 +476,8 @@ async def finalize_serial_config() -> None:
cg.add_build_flag("-DNO_GLOBAL_SERIAL1")
# Called by __main__.compile_program; returning False falls through to the
# PlatformIO toolchain.
def run_compile(args, config: ConfigType) -> bool:
# Positive check: the native backend only runs when explicitly resolved
toolchain = native_toolchain_module()
if toolchain is None:
return False
if toolchain.run_compile(config, CORE.verbose) != 0:
raise EsphomeError("ESP8266 native build failed")
return True
# Called by writer.py
def copy_files() -> None:
# Native builds skip the PlatformIO extra scripts; the build generator
# carries their logic
if CORE.using_toolchain_arduino:
return
dir = Path(__file__).parent
for script in (
"post_build",
@@ -588,7 +492,6 @@ def copy_files() -> None:
CORE.relative_build_path(f"{script}.py"),
)
copy_ccache_script()
copy_pch_script()
# ESP logs stack trace decoder, based on https://github.com/me-no-dev/EspExceptionDecoder
@@ -631,75 +534,22 @@ ESP8266_EXCEPTION_CODES = {
}
_DECODE_WARNED_AT: dict[str, float] = {}
def _decode_pc(config: ConfigType, addr: str) -> None:
from esphome.platformio import toolchain
def _warn_decode_problem(key: str, message: str, *args) -> bool:
"""Warn, deduplicated briefly so a burst of stack-dump addresses warns
once but a later dump warns again; returns whether it warned so the
caller can mark suppressed addresses individually."""
now = time.monotonic()
last = _DECODE_WARNED_AT.get(key)
if last is not None and now - last < 30:
return False
_DECODE_WARNED_AT[key] = now
_LOGGER.warning(message, *args)
return True
def _decode_pc(config: ConfigType, addr: str, *, bulk: bool = False) -> None:
"""Decode one crash address. ``bulk``: the caller is scanning every
8-hex stack word, most of which are not code addresses -- unmappable
ones log at debug so real frames are not buried."""
if (native_toolchain := native_toolchain_module()) is not None:
addr2line = native_toolchain.get_addr2line_path()
elf = native_toolchain.get_elf_path()
for path in (addr2line, elf):
if not path.is_file():
_warn_decode_problem(
str(path), "Cannot decode crash addresses: %s missing", path
)
# The detailed warning names no address; mark named
# registers, but bulk stack words at debug (~150 per dump)
log = _LOGGER.debug if bulk else _LOGGER.warning
log("Not decoded %s (toolchain file missing)", addr)
return
addr2line, elf = str(addr2line), str(elf)
else:
from esphome.platformio import toolchain
idedata = toolchain.get_idedata(config)
if not idedata.addr2line_path or not idedata.firmware_elf_path:
_warn_decode_problem(
"no-addr2line",
"Cannot decode crash addresses: no addr2line or ELF in idedata",
)
log = _LOGGER.debug if bulk else _LOGGER.warning
log("Not decoded %s (no addr2line or ELF)", addr)
return
addr2line, elf = idedata.addr2line_path, idedata.firmware_elf_path
command = [addr2line, "-pfiaC", "-e", elf, addr]
idedata = toolchain.get_idedata(config)
if not idedata.addr2line_path or not idedata.firmware_elf_path:
_LOGGER.debug("decode_pc no addr2line")
return
command = [idedata.addr2line_path, "-pfiaC", "-e", idedata.firmware_elf_path, addr]
try:
translation = subprocess.check_output(command, close_fds=False).decode().strip()
except Exception as err: # noqa: BLE001 # pylint: disable=broad-except
# Warn, not debug: a failing addr2line must be visible. The warning
# is rate-limited across a dump, so mark every undecoded address
# inline or the rest read as merely unmappable
if not _warn_decode_problem(
"addr2line-failed", "Could not decode crash address %s (%s)", addr, err
):
# The detailed warning already named this address; mark only
# the rate-limited ones, and bulk stack words at debug
log = _LOGGER.debug if bulk else _LOGGER.warning
log("Not decoded %s (addr2line failed)", addr)
except Exception: # noqa: BLE001 # pylint: disable=broad-except
_LOGGER.debug("Caught exception for command %s", command, exc_info=1)
return
if "?? ??:0" in translation:
# A named register that fails to decode is confusing silence; a
# bulk stack word failing is the expected common case
log = _LOGGER.debug if bulk else _LOGGER.warning
log("Not decoded %s (address not in %s)", addr, elf)
# Nothing useful
return
translation = translation.replace(" at ??:?", "").replace(":?", "")
_LOGGER.warning("Decoded %s", translation)
@@ -769,6 +619,6 @@ def process_stacktrace(config: ConfigType, line: str, backtrace_state: bool) ->
if backtrace_state:
for addr in re.finditer(STACKTRACE_ESP8266_BACKTRACE_PC_RE, line):
_decode_pc(config, addr.group(), bulk=True)
_decode_pc(config, addr.group())
return backtrace_state
+1 -3
View File
@@ -16,6 +16,7 @@ KEY_WAVEFORM_REQUIRED = "waveform_required"
KEY_SERIAL_REQUIRED = "serial_required"
KEY_SERIAL1_REQUIRED = "serial1_required"
# Set for the native (non-PlatformIO) toolchain's build generator
KEY_FLASH_MODE = "flash_mode"
KEY_SCANF_FLOAT = "scanf_float"
# Per-board flash-layout override consumed by board_ld_script()
KEY_LDSCRIPT = "ldscript"
@@ -72,6 +73,3 @@ def enable_serial1() -> None:
enable_serial1()
"""
CORE.data.setdefault(KEY_ESP8266, {})[KEY_SERIAL1_REQUIRED] = True
BUILD_FLASH_MODES = ("qio", "qout", "dio", "dout")
+2 -12
View File
@@ -1,4 +1,3 @@
from esphome.build_helpers.pch import PCH_PREFIX_HEADER, pch_enabled, pch_extra_scripts
import esphome.codegen as cg
import esphome.config_validation as cv
from esphome.const import (
@@ -11,7 +10,7 @@ from esphome.const import (
ThreadModel,
)
from esphome.core import CORE
from esphome.platformio.toolchain import copy_ccache_script, copy_pch_script
from esphome.platformio.toolchain import copy_ccache_script
from esphome.types import ConfigType
from .const import KEY_HOST
@@ -56,18 +55,9 @@ async def to_code(config: ConfigType) -> None:
cg.add_platformio_option("platform", "platformio/native")
cg.add_platformio_option("lib_ldf_mode", "off")
cg.add_platformio_option("lib_compat_mode", "strict")
cg.add_platformio_option("extra_scripts", ["pre:ccache.py", *pch_extra_scripts()])
if pch_enabled():
# Curated prefix for the pch (the script folds it plus defines.h):
# host has no framework force-includes, and the per-TU cost is the
# STL closure behind the core headers. Measured -43% compile CPU.
# Gated so ESPHOME_PCH_ENABLE=0 restores the strict view. When the
# .gch fails to build or load, the force-include stays and every TU
# parses the closure as text: correct, but slower than no pch.
cg.add_platformio_option("build_src_flags", f"-include {PCH_PREFIX_HEADER}")
cg.add_platformio_option("extra_scripts", ["pre:ccache.py"])
# Called by writer.py
def copy_files() -> None:
copy_ccache_script()
copy_pch_script()
+2 -4
View File
@@ -2,7 +2,6 @@ import json
import logging
from pathlib import Path
from esphome.build_helpers.pch import pch_extra_scripts
import esphome.codegen as cg
import esphome.config_validation as cv
from esphome.const import (
@@ -27,7 +26,7 @@ from esphome.const import (
from esphome.core import CORE
from esphome.core.config import BOARD_MAX_LENGTH
from esphome.helpers import copy_file_if_changed
from esphome.platformio.toolchain import copy_ccache_script, copy_pch_script
from esphome.platformio.toolchain import copy_ccache_script
from esphome.storage_json import StorageJSON
from . import gpio # noqa: F401
@@ -514,7 +513,7 @@ async def component_to_code(config):
# it for project source files only. GCC uses the last -O flag.
build_src_flags += " -Os"
cg.add_platformio_option("build_src_flags", build_src_flags)
cg.add_platformio_option("extra_scripts", ["pre:ccache.py", *pch_extra_scripts()])
cg.add_platformio_option("extra_scripts", ["pre:ccache.py"])
# IRAM_ATTR is a no-op on BK72xx (SDK masks FIQ+IRQ around flash ops).
# On other families, patch_linker.py routes .sram.text into the right
# RAM-executable output section and prints a post-link placement summary.
@@ -620,4 +619,3 @@ def copy_files() -> None:
CORE.relative_build_path("patch_linker.py"),
)
copy_ccache_script()
copy_pch_script()
-126
View File
@@ -1,24 +1,13 @@
from __future__ import annotations
import asyncio
from functools import partial
import logging
from pathlib import Path
import re
import shutil
import stat
import subprocess
from esphome import pins
from esphome.build_helpers import pch
from esphome.build_helpers.pch import (
PCH_DEFAULT_HEADERS,
PCH_HEADER_NAME,
mark_pch_emitted,
pch_cmake_consumer,
pch_enabled,
pch_header_text,
)
import esphome.codegen as cg
from esphome.components.zephyr import (
add_extra_script,
@@ -816,9 +805,6 @@ def _generate_cmake_lists() -> bool:
")",
]
if consumer := pch_cmake_consumer("app", "${APP_SOURCES}"):
lines += consumer.splitlines()
if link_flags:
lines += [
"",
@@ -833,66 +819,6 @@ def _generate_cmake_lists() -> bool:
)
def _app_build_dir(build_dir: Path) -> Path:
"""The CMake binary dir of the app image: sysbuild nests it in a
domain dir named after the app source dir. Probed on disk (the
non-sysbuild zephyr/ output dir has no CMakeCache.txt) so it stays
truthful mid-build, unlike an SDK-version check."""
sysbuild_app = build_dir / "zephyr"
try:
cache = (sysbuild_app / "CMakeCache.txt").stat()
except (FileNotFoundError, NotADirectoryError):
return build_dir
# Other stat errors propagate; is_file() would silently mislocate the pch
return sysbuild_app if stat.S_ISREG(cache.st_mode) else build_dir
def _prepare_pch(app_dir: Path) -> None:
"""Build the .gch between the cmake and compile phases of west."""
if not pch_enabled():
pch.discard_pch(app_dir)
pch.pch_disabled_degraded()
return
# First, so OBJECT_DEPENDS is satisfied even when the pch degrades
app_dir.mkdir(parents=True, exist_ok=True)
write_file_if_changed(
app_dir / PCH_HEADER_NAME, pch_header_text(PCH_DEFAULT_HEADERS)
)
# New layout first (Zephyr >= 3.4 nests under zephyr/); fixed candidates
# keep the .sum identity deterministic and skip walking generated/
generated = app_dir / "zephyr" / "include" / "generated"
autoconf = None
for candidate in (generated / "zephyr" / "autoconf.h", generated / "autoconf.h"):
if candidate.exists():
autoconf = candidate
break
if autoconf is None:
# Fail closed: autoconf.h is the .sum's Kconfig identity
_LOGGER.warning("No autoconf.h found; compiling without the pch")
pch.discard_pch(app_dir)
pch.pch_degraded("autoconf.h missing")
return
try:
autoconf_text = autoconf.read_text(encoding="utf-8")
except OSError as err:
_LOGGER.warning(
"Could not read %s; compiling without the pch: %s", autoconf, err
)
pch.discard_pch(app_dir)
pch.pch_degraded(f"autoconf unreadable: {err}")
return
pch.prepare_pch(
app_dir,
PCH_DEFAULT_HEADERS,
(
str(CORE.data[KEY_CORE][KEY_FRAMEWORK_VERSION]),
zephyr_data()[KEY_BOARD],
autoconf_text,
*get_project_compile_flags(),
),
)
def _copy_if_exists(src: Path, dst: Path) -> None:
if src.is_file():
shutil.copy2(src, dst)
@@ -945,58 +871,6 @@ def run_compile(args, config: ConfigType) -> bool:
str(source_dir),
]
if pch_enabled():
# Consumers carry the -include; gate the ccache relaxation on it
# (Zephyr auto-enables ccache as the compiler launcher when found)
mark_pch_emitted()
env.update(pch.ccache_pch_env())
# Configure first so the .gch compiles from settled compile DB
# flags. Only when the app DB is missing: input changes wipe the
# build dir, so an existing DB is settled and --cmake-only would
# reconfigure for nothing.
prepare = True
app_dir = _app_build_dir(build_dir)
if not (app_dir / "compile_commands.json").is_file():
if not run_command_ok(
west_cmd + ["--cmake-only", "--", "-DCMAKE_EXPORT_COMPILE_COMMANDS=ON"],
env=env,
stream_output=True,
cwd=str(paths["framework_path"]),
):
raise EsphomeError("nRF52 native build configure failed")
# The configure phase creates the sysbuild domain dir: re-resolve
app_dir = _app_build_dir(build_dir)
# kernel.h needs the build-time syscall headers; under sysbuild
# the target exists only in the app domain's ninja
if not run_command_ok(
[
"cmake",
"--build",
str(app_dir),
"--target",
"zephyr_generated_headers",
],
env=env,
stream_output=True,
cwd=str(paths["framework_path"]),
):
# A pch-only prerequisite: degrade, let the real build report.
# Also skip the .gch compile: it would fail on the missing
# headers and latch .gch.failed until an identity change
_LOGGER.warning(
"Zephyr header generation failed; compiling without the pch"
)
pch.discard_pch(app_dir)
pch.pch_degraded("zephyr_generated_headers failed")
prepare = False
else:
prepare = True
app_dir = _app_build_dir(build_dir)
if prepare:
pch.guarded_prepare(app_dir, partial(_prepare_pch, app_dir))
if not run_command_ok(
west_cmd,
env=env,
+2 -7
View File
@@ -6,7 +6,6 @@ from string import ascii_letters, digits
import subprocess
from typing import Any
from esphome.build_helpers.pch import pch_extra_scripts
import esphome.codegen as cg
import esphome.config_validation as cv
from esphome.const import (
@@ -34,7 +33,7 @@ from esphome.core import (
)
from esphome.core.config import BOARD_MAX_LENGTH
from esphome.helpers import copy_file_if_changed, read_file, write_file_if_changed
from esphome.platformio.toolchain import copy_ccache_script, copy_pch_script
from esphome.platformio.toolchain import copy_ccache_script
from esphome.storage_json import StorageJSON
from esphome.types import ConfigType
@@ -341,10 +340,7 @@ async def to_code(config: ConfigType) -> None:
cg.add_define("ESPHOME_VARIANT", VARIANT_FRIENDLY[variant])
cg.add_define(ThreadModel.SINGLE)
cg.add_platformio_option(
"extra_scripts",
["pre:ccache.py", *pch_extra_scripts(), "post:post_build.py"],
)
cg.add_platformio_option("extra_scripts", ["pre:ccache.py", "post:post_build.py"])
conf = config[CONF_FRAMEWORK]
cg.add_platformio_option("framework", "arduino")
@@ -648,7 +644,6 @@ def copy_files() -> None:
CORE.relative_build_path("inject_lwip_include.py"),
)
copy_ccache_script()
copy_pch_script()
_generate_lwipopts_h()
if generate_pio_files():
path = CORE.relative_src_path("esphome.h")
+2
View File
@@ -187,7 +187,9 @@ async def to_code(config: ConfigType) -> None:
# for the selected implementation.
FILTER_SOURCE_FILES = filter_source_files_from_defines(
{
"lwip_raw_common_impl.cpp": "USE_SOCKET_IMPL_LWIP_TCP",
"lwip_raw_tcp_impl.cpp": "USE_SOCKET_IMPL_LWIP_TCP",
"lwip_raw_udp_impl.cpp": "USE_SOCKET_IMPL_LWIP_TCP",
"bsd_sockets_impl.cpp": "USE_SOCKET_IMPL_BSD_SOCKETS",
"lwip_sockets_impl.cpp": "USE_SOCKET_IMPL_LWIP_SOCKETS",
}
@@ -144,6 +144,9 @@ class BSDSocketImpl {
int get_fd() const { return this->fd_; }
/// UDP rx drop counter parity with LWIPRawUDPImpl; drops are not counted here.
uint16_t get_rx_dropped() const { return 0; }
protected:
// fd_ < 0 means "not open" — used both pre-open (initial state) and post-close. This
// replaces a separate closed_ flag: close() sets fd_ = -1 after ::close(), and the
+10
View File
@@ -20,6 +20,16 @@
#define IPPROTO_IP 0
#define IPPROTO_TCP 6
#define IPPROTO_UDP 17
#define IP_ADD_MEMBERSHIP 3
#define IP_DROP_MEMBERSHIP 4
// NOLINTNEXTLINE(readability-identifier-naming)
struct ip_mreq {
struct in_addr imr_multiaddr;
struct in_addr imr_interface;
};
#if LWIP_IPV6
#define AF_INET6 10
@@ -0,0 +1,109 @@
#include "lwip_raw_common_impl.h"
#include "esphome/core/defines.h"
#ifdef USE_SOCKET_IMPL_LWIP_TCP
#include <cerrno>
#include <cstring>
namespace esphome::socket {
int lwip_ip_to_sockaddr(sa_family_t family, const ip_addr_t *ip, uint16_t port_host, struct sockaddr *name,
socklen_t *addrlen) {
if (family == AF_INET) {
if (*addrlen < sizeof(struct sockaddr_in)) {
errno = EINVAL;
return -1;
}
auto *addr = reinterpret_cast<struct sockaddr_in *>(name);
addr->sin_family = AF_INET;
*addrlen = addr->sin_len = sizeof(struct sockaddr_in);
addr->sin_port = htons(port_host);
inet_addr_from_ip4addr(&addr->sin_addr, ip_2_ip4(ip));
return 0;
}
#if LWIP_IPV6
if (family == AF_INET6) {
if (*addrlen < sizeof(struct sockaddr_in6)) {
errno = EINVAL;
return -1;
}
auto *addr = reinterpret_cast<struct sockaddr_in6 *>(name);
addr->sin6_family = AF_INET6;
*addrlen = addr->sin6_len = sizeof(struct sockaddr_in6);
addr->sin6_port = htons(port_host);
// AF_INET6 sockets may receive IPv4 packets; convert to IPv4-mapped IPv6.
if (IP_IS_V4(ip)) {
ip_addr_t mapped;
ip4_2_ipv4_mapped_ipv6(ip_2_ip6(&mapped), ip_2_ip4(ip));
inet6_addr_from_ip6addr(&addr->sin6_addr, ip_2_ip6(&mapped));
} else {
inet6_addr_from_ip6addr(&addr->sin6_addr, ip_2_ip6(ip));
}
return 0;
}
#endif
errno = EAFNOSUPPORT;
return -1;
}
bool sockaddr_to_lwip(const struct sockaddr *addr, socklen_t addrlen, ip_addr_t *ip, uint16_t *port) {
// headers.h defines sockaddr and sockaddr_in with the same size, so this covers AF_INET
if (addrlen < sizeof(struct sockaddr))
return false;
// Zero the whole struct — the IPv6 zone byte would otherwise be stack garbage
memset(ip, 0, sizeof(*ip));
if (addr->sa_family == AF_INET) {
auto *addr4 = reinterpret_cast<const sockaddr_in *>(addr);
*port = ntohs(addr4->sin_port);
IP_SET_TYPE_VAL(*ip, IPADDR_TYPE_V4);
ip_2_ip4(ip)->addr = addr4->sin_addr.s_addr;
return true;
}
#if LWIP_IPV6
if (addr->sa_family == AF_INET6) {
if (addrlen < sizeof(sockaddr_in6))
return false;
auto *addr6 = reinterpret_cast<const sockaddr_in6 *>(addr);
*port = ntohs(addr6->sin6_port);
IP_SET_TYPE_VAL(*ip, IPADDR_TYPE_V6);
memcpy(&ip_2_ip6(ip)->addr, &addr6->sin6_addr.un.u8_addr, 16);
// Unmap ::ffff:a.b.c.d so replies to recvfrom addresses route as IPv4
if (ip6_addr_isipv4mappedipv6(ip_2_ip6(ip))) {
unmap_ipv4_mapped_ipv6(ip_2_ip4(ip), ip_2_ip6(ip));
IP_SET_TYPE_VAL(*ip, IPADDR_TYPE_V4);
}
return true;
}
#endif
return false;
}
bool sockaddr_to_lwip_bind(sa_family_t family, const struct sockaddr *addr, socklen_t addrlen, ip_addr_t *ip,
uint16_t *port) {
if (!sockaddr_to_lwip(addr, addrlen, ip, port))
return false;
#if LWIP_IPV6
// Only promote wildcard binds — a specific address must keep filtering
if (family == AF_INET6 && ip_addr_isany_val(*ip))
IP_SET_TYPE_VAL(*ip, IPADDR_TYPE_ANY);
#endif
return true;
}
int lwip_bind_err(err_t err) {
if (err == ERR_OK)
return 0;
if (err == ERR_USE) {
errno = EADDRINUSE;
} else if (err == ERR_VAL) {
errno = EINVAL;
} else {
errno = EIO;
}
return -1;
}
} // namespace esphome::socket
#endif // USE_SOCKET_IMPL_LWIP_TCP
@@ -0,0 +1,53 @@
#pragma once
#include "esphome/core/defines.h"
#ifdef USE_SOCKET_IMPL_LWIP_TCP
#include "headers.h"
#include "lwip/ip.h"
namespace esphome::socket {
// ---- LWIP thread safety ----
//
// On RP2040 (Pico W), arduino-pico sets PICO_CYW43_ARCH_THREADSAFE_BACKGROUND=1.
// This means lwip callbacks (recv_fn, accept_fn, err_fn) run from a low-priority
// user IRQ context, not the main loop (see low_priority_irq_handler() in pico-sdk
// async_context_threadsafe_background.c). They can preempt main-loop code at any point.
//
// Without locking, this causes race conditions between recv_fn and read() on the
// shared rx_buf_ pbuf chain — recv_fn calls pbuf_cat() while read() is freeing
// nodes, leading to use-after-free and infinite-loop crashes. See esphome#10681.
//
// On ESP8266, lwip callbacks run from the SYS context which cooperates with user
// code (CONT context) — they never preempt each other, so no locking is needed.
//
// esphome::LwIPLock is the platform-provided RAII guard (see helpers.h/helpers.cpp).
// On RP2040, it acquires cyw43_arch_lwip_begin/end (WiFi) or ethernet_arch_lwip_begin/end
// (Ethernet). On ESP8266, it's a no-op.
//
// Each .cpp file that needs locking defines its own LWIP_LOCK() macro:
// #define LWIP_LOCK() esphome::LwIPLock lwip_lock_guard
// This is a per-TU convenience macro, not defined here to avoid macro leaking.
/// Convert lwip ip_addr_t + host-order port to sockaddr, based on the socket's address family.
/// TCP callers pass ntohs(pcb port) to preserve historical getpeername/getsockname output.
int lwip_ip_to_sockaddr(sa_family_t family, const ip_addr_t *ip, uint16_t port_host, struct sockaddr *name,
socklen_t *addrlen);
/// Convert sockaddr to lwip ip_addr_t and host-order port.
/// For IPv6, sets type to IPADDR_TYPE_V6 — correct for sendto destinations.
/// Bind paths must use sockaddr_to_lwip_bind() instead.
bool sockaddr_to_lwip(const struct sockaddr *addr, socklen_t addrlen, ip_addr_t *ip, uint16_t *port);
/// sockaddr_to_lwip variant for bind: promotes AF_INET6 sockets to
/// IPADDR_TYPE_ANY so they accept both IPv4 and IPv6 (dual-stack).
bool sockaddr_to_lwip_bind(sa_family_t family, const struct sockaddr *addr, socklen_t addrlen, ip_addr_t *ip,
uint16_t *port);
/// Map lwip bind error to errno. Returns 0 on success, -1 on error with errno set.
int lwip_bind_err(err_t err);
} // namespace esphome::socket
#endif // USE_SOCKET_IMPL_LWIP_TCP
+13 -106
View File
@@ -10,6 +10,7 @@
#include "esphome/core/helpers.h"
#include "esphome/core/wake.h"
#include "esphome/core/log.h"
#include "lwip_raw_common_impl.h"
#ifdef USE_OTA_PLATFORM_ESPHOME
extern "C" void esphome_wake_ota_component_any_context();
@@ -24,23 +25,9 @@ extern "C" void esphome_wake_ota_component_any_context();
namespace esphome::socket {
// ---- LWIP thread safety ----
//
// On RP2040 (Pico W), arduino-pico sets PICO_CYW43_ARCH_THREADSAFE_BACKGROUND=1.
// This means lwip callbacks (recv_fn, accept_fn, err_fn) run from a low-priority
// user IRQ context, not the main loop (see low_priority_irq_handler() in pico-sdk
// async_context_threadsafe_background.c). They can preempt main-loop code at any point.
//
// Without locking, this causes race conditions between recv_fn and read() on the
// shared rx_buf_ pbuf chain — recv_fn calls pbuf_cat() while read() is freeing
// nodes, leading to use-after-free and infinite-loop crashes. See esphome#10681.
//
// On ESP8266, lwip callbacks run from the SYS context which cooperates with user
// code (CONT context) — they never preempt each other, so no locking is needed.
//
// esphome::LwIPLock is the platform-provided RAII guard (see helpers.h/helpers.cpp).
// On RP2040, it acquires cyw43_arch_lwip_begin/end (WiFi) or ethernet_arch_lwip_begin/end
// (Ethernet). On ESP8266, it's a no-op.
// LWIP thread safety — see lwip_raw_common_impl.h for full explanation.
// esphome::LwIPLock is the platform-provided RAII guard.
// On RP2040, it acquires cyw43_arch_lwip_begin/end. On ESP8266, it's a no-op.
#define LWIP_LOCK() esphome::LwIPLock lwip_lock_guard // NOLINT
static const char *const TAG = "socket";
@@ -112,59 +99,14 @@ int LWIPRawCommon::bind(const struct sockaddr *name, socklen_t addrlen) {
return -1;
}
ip_addr_t ip;
in_port_t port;
#if LWIP_IPV6
if (this->family_ == AF_INET) {
if (addrlen < sizeof(sockaddr_in)) {
errno = EINVAL;
return -1;
}
auto *addr4 = reinterpret_cast<const sockaddr_in *>(name);
port = ntohs(addr4->sin_port);
ip.type = IPADDR_TYPE_V4;
ip.u_addr.ip4.addr = addr4->sin_addr.s_addr;
LWIP_LOG("tcp_bind(%p ip=%s port=%u)", this->pcb_, ip4addr_ntoa(&ip.u_addr.ip4), port);
} else if (this->family_ == AF_INET6) {
if (addrlen < sizeof(sockaddr_in6)) {
errno = EINVAL;
return -1;
}
auto *addr6 = reinterpret_cast<const sockaddr_in6 *>(name);
port = ntohs(addr6->sin6_port);
ip.type = IPADDR_TYPE_ANY;
memcpy(&ip.u_addr.ip6.addr, &addr6->sin6_addr.un.u8_addr, 16);
LWIP_LOG("tcp_bind(%p ip=%s port=%u)", this->pcb_, ip6addr_ntoa(&ip.u_addr.ip6), port);
} else {
uint16_t port;
if (!sockaddr_to_lwip_bind(this->family_, name, addrlen, &ip, &port)) {
errno = EINVAL;
return -1;
}
#else
if (this->family_ != AF_INET) {
errno = EINVAL;
return -1;
}
auto *addr4 = reinterpret_cast<const sockaddr_in *>(name);
port = ntohs(addr4->sin_port);
ip.addr = addr4->sin_addr.s_addr;
LWIP_LOG("tcp_bind(%p ip=%u port=%u)", this->pcb_, ip.addr, port);
#endif
err_t err = tcp_bind(this->pcb_, &ip, port);
if (err == ERR_USE) {
LWIP_LOG(" -> err ERR_USE");
errno = EADDRINUSE;
return -1;
}
if (err == ERR_VAL) {
LWIP_LOG(" -> err ERR_VAL");
errno = EINVAL;
return -1;
}
if (err != ERR_OK) {
LWIP_LOG(" -> err %d", err);
errno = EIO;
return -1;
}
return 0;
LWIP_LOG(" -> err %d", err);
return lwip_bind_err(err);
}
int LWIPRawCommon::close() {
@@ -349,43 +291,8 @@ int LWIPRawCommon::setsockopt(int level, int optname, const void *optval, sockle
}
int LWIPRawCommon::ip2sockaddr_(ip_addr_t *ip, uint16_t port, struct sockaddr *name, socklen_t *addrlen) {
if (this->family_ == AF_INET) {
if (*addrlen < sizeof(struct sockaddr_in)) {
errno = EINVAL;
return -1;
}
struct sockaddr_in *addr = reinterpret_cast<struct sockaddr_in *>(name);
addr->sin_family = AF_INET;
*addrlen = addr->sin_len = sizeof(struct sockaddr_in);
addr->sin_port = port;
inet_addr_from_ip4addr(&addr->sin_addr, ip_2_ip4(ip));
return 0;
}
#if LWIP_IPV6
else if (this->family_ == AF_INET6) {
if (*addrlen < sizeof(struct sockaddr_in6)) {
errno = EINVAL;
return -1;
}
struct sockaddr_in6 *addr = reinterpret_cast<struct sockaddr_in6 *>(name);
addr->sin6_family = AF_INET6;
*addrlen = addr->sin6_len = sizeof(struct sockaddr_in6);
addr->sin6_port = port;
// AF_INET6 sockets are bound to IPv4 as well, so we may encounter IPv4 addresses that must be converted to IPv6.
if (IP_IS_V4(ip)) {
ip_addr_t mapped;
ip4_2_ipv4_mapped_ipv6(ip_2_ip6(&mapped), ip_2_ip4(ip));
inet6_addr_from_ip6addr(&addr->sin6_addr, ip_2_ip6(&mapped));
} else {
inet6_addr_from_ip6addr(&addr->sin6_addr, ip_2_ip6(ip));
}
return 0;
}
#endif
return -1;
// lwip pcb ports are host order; ntohs preserves historical byte-swapped sin_port output
return lwip_ip_to_sockaddr(this->family_, ip, ntohs(port), name, addrlen);
}
// ---- LWIPRawImpl methods ----
@@ -887,11 +794,11 @@ err_t LWIPRawListenImpl::accept_fn_(struct tcp_pcb *newpcb, err_t err) {
return ERR_OK;
}
// ---- Factory functions ----
// ---- TCP Factory functions ----
std::unique_ptr<Socket> socket(int domain, int type, int protocol) {
if (type != SOCK_STREAM) {
ESP_LOGE(TAG, "UDP sockets not supported on this platform, use WiFiUDP");
ESP_LOGE(TAG, "Use socket_udp() for UDP sockets on this platform");
errno = EPROTOTYPE;
return nullptr;
}
@@ -911,7 +818,7 @@ std::unique_ptr<Socket> socket_loop_monitored(int domain, int type, int protocol
std::unique_ptr<ListenSocket> socket_listen(int domain, int type, int protocol) {
if (type != SOCK_STREAM) {
ESP_LOGE(TAG, "UDP sockets not supported on this platform, use WiFiUDP");
ESP_LOGE(TAG, "Use socket_udp() for UDP sockets on this platform");
errno = EPROTOTYPE;
return nullptr;
}
@@ -0,0 +1,328 @@
#include "socket.h"
#include "esphome/core/defines.h"
#ifdef USE_SOCKET_IMPL_LWIP_TCP
#include <cerrno>
#include <cstring>
#include "esphome/core/helpers.h"
#include "esphome/core/log.h"
#include "esphome/core/wake.h"
#include "lwip_raw_common_impl.h"
#include "lwip/igmp.h"
#include "lwip/pbuf.h"
#include "lwip/udp.h"
namespace esphome::socket {
// LWIP thread safety — see lwip_raw_common_impl.h for full explanation.
// esphome::LwIPLock is the platform-provided RAII guard.
// On RP2040, it acquires cyw43_arch_lwip_begin/end. On ESP8266, it's a no-op.
#define LWIP_LOCK() esphome::LwIPLock lwip_lock_guard // NOLINT
// ---- LWIPRawUDPSendImpl (send-only) methods ----
LWIPRawUDPSendImpl::~LWIPRawUDPSendImpl() {
// Guard avoids acquiring the lwip lock when already closed
if (this->pcb_ != nullptr)
this->close();
}
int LWIPRawUDPSendImpl::bind(const struct sockaddr *name, socklen_t addrlen) {
LWIP_LOCK();
if (this->pcb_ == nullptr) {
errno = EBADF;
return -1;
}
if (name == nullptr) {
errno = EINVAL;
return -1;
}
ip_addr_t ip;
uint16_t port;
if (!sockaddr_to_lwip_bind(this->family_, name, addrlen, &ip, &port)) {
errno = EINVAL;
return -1;
}
return lwip_bind_err(udp_bind(this->pcb_, &ip, port));
}
int LWIPRawUDPSendImpl::close() {
LWIP_LOCK();
return this->close_internal_locked_();
}
int LWIPRawUDPSendImpl::close_internal_locked_() {
// Caller must hold LWIP_LOCK
if (this->pcb_ == nullptr) {
errno = EBADF;
return -1;
}
udp_remove(this->pcb_);
this->pcb_ = nullptr;
return 0;
}
int LWIPRawUDPSendImpl::ip2sockaddr_(const ip_addr_t *ip, uint16_t port, struct sockaddr *name, socklen_t *addrlen) {
// UDP recv callback provides port in host byte order
return lwip_ip_to_sockaddr(this->family_, ip, port, name, addrlen);
}
ssize_t LWIPRawUDPSendImpl::sendto(const void *buf, size_t len, int flags, const struct sockaddr *dest_addr,
socklen_t addrlen) {
(void) flags; // Flags (MSG_DONTWAIT, etc.) are ignored; raw lwip is always non-blocking
if (buf == nullptr || dest_addr == nullptr) {
errno = EINVAL;
return -1;
}
// pbuf_alloc takes u16_t length; reject oversized packets
if (len > UINT16_MAX) {
errno = EMSGSIZE;
return -1;
}
ip_addr_t dst_ip;
uint16_t dst_port;
if (!sockaddr_to_lwip(dest_addr, addrlen, &dst_ip, &dst_port)) {
errno = EINVAL;
return -1;
}
LWIP_LOCK();
if (this->pcb_ == nullptr) {
errno = EBADF;
return -1;
}
// Allocate pbuf and copy data
struct pbuf *pb = pbuf_alloc(PBUF_TRANSPORT, (uint16_t) len, PBUF_RAM);
if (pb == nullptr) {
errno = ENOMEM;
return -1;
}
memcpy(pb->payload, buf, len);
err_t err = udp_sendto(this->pcb_, pb, &dst_ip, dst_port);
pbuf_free(pb);
if (err != ERR_OK) {
errno = err == ERR_MEM ? ENOMEM : EIO;
return -1;
}
return (ssize_t) len;
}
int LWIPRawUDPSendImpl::setsockopt(int level, int optname, const void *optval, socklen_t optlen) {
LWIP_LOCK();
if (this->pcb_ == nullptr) {
errno = EBADF;
return -1;
}
if (level == SOL_SOCKET && optname == SO_REUSEADDR) {
if (optval == nullptr || optlen < sizeof(int)) {
errno = EINVAL;
return -1;
}
// Effective only where lwip is built with SO_REUSE=1 (ESP8266 yes, RP2040 currently no)
if (*reinterpret_cast<const int *>(optval)) {
ip_set_option(this->pcb_, SOF_REUSEADDR);
} else {
ip_reset_option(this->pcb_, SOF_REUSEADDR);
}
return 0;
}
if (level == SOL_SOCKET && optname == SO_BROADCAST) {
if (optval == nullptr || optlen < sizeof(int)) {
errno = EINVAL;
return -1;
}
int val = *reinterpret_cast<const int *>(optval);
if (val) {
ip_set_option(this->pcb_, SOF_BROADCAST);
} else {
ip_reset_option(this->pcb_, SOF_BROADCAST);
}
return 0;
}
if (level == IPPROTO_IP && (optname == IP_ADD_MEMBERSHIP || optname == IP_DROP_MEMBERSHIP)) {
if (optval == nullptr || optlen < sizeof(struct ip_mreq)) {
errno = EINVAL;
return -1;
}
auto *mreq = reinterpret_cast<const struct ip_mreq *>(optval);
ip4_addr_t multiaddr{mreq->imr_multiaddr.s_addr};
ip4_addr_t ifaddr{mreq->imr_interface.s_addr};
err_t err =
optname == IP_ADD_MEMBERSHIP ? igmp_joingroup(&ifaddr, &multiaddr) : igmp_leavegroup(&ifaddr, &multiaddr);
if (err != ERR_OK) {
errno = EIO;
return -1;
}
return 0;
}
errno = ENOPROTOOPT;
return -1;
}
int LWIPRawUDPSendImpl::getsockopt(int level, int optname, void *optval, socklen_t *optlen) {
LWIP_LOCK();
if (this->pcb_ == nullptr) {
errno = EBADF;
return -1;
}
if (level == SOL_SOCKET && optname == SO_REUSEADDR) {
if (optval == nullptr || optlen == nullptr || *optlen < sizeof(int)) {
errno = EINVAL;
return -1;
}
*reinterpret_cast<int *>(optval) = ip_get_option(this->pcb_, SOF_REUSEADDR) ? 1 : 0;
*optlen = sizeof(int);
return 0;
}
errno = ENOPROTOOPT;
return -1;
}
int LWIPRawUDPSendImpl::setblocking(bool blocking) {
if (blocking) {
// blocking operation not supported on raw lwip
errno = EINVAL;
return -1;
}
return 0;
}
// ---- LWIPRawUDPImpl methods ----
LWIPRawUDPImpl::LWIPRawUDPImpl(sa_family_t family, struct udp_pcb *pcb) : LWIPRawUDPSendImpl(family, pcb) {
// Registered here (not in bind) so unbound client sockets can receive replies
udp_recv(this->pcb_, LWIPRawUDPImpl::s_recv_fn, this);
}
LWIPRawUDPImpl::~LWIPRawUDPImpl() {
// Flush rx queue and unregister callback before base destructor removes pcb
if (this->pcb_ != nullptr)
this->close();
}
int LWIPRawUDPImpl::close() {
LWIP_LOCK();
// Unregister recv callback before removing pcb
if (this->pcb_ != nullptr) {
udp_recv(this->pcb_, nullptr, nullptr);
}
// Flush queued rx packets; slots within rx_count_ always hold a live pbuf
for (; this->rx_count_ > 0; this->rx_count_--) {
pbuf_free(this->rx_queue_[this->rx_read_idx_].pb);
this->rx_read_idx_ = (this->rx_read_idx_ + 1) & UDP_RX_MASK;
}
// close_internal_locked_() returns EBADF if already closed, which is fine from destructor
return this->close_internal_locked_();
}
ssize_t LWIPRawUDPImpl::read(void *buf, size_t len) { return this->recvfrom(buf, len, nullptr, nullptr); }
ssize_t LWIPRawUDPImpl::recvfrom(void *buf, size_t len, struct sockaddr *src_addr, socklen_t *addrlen) {
if (buf == nullptr && len > 0) {
errno = EINVAL;
return -1;
}
LWIP_LOCK();
if (this->pcb_ == nullptr) {
errno = EBADF;
return -1;
}
if (this->rx_count_ == 0) {
errno = EWOULDBLOCK;
return -1;
}
auto &pkt = this->rx_queue_[this->rx_read_idx_];
// On address conversion failure, still consume the packet — the failure is
// deterministic (family_ and *addrlen), so keeping it would wedge the queue
ssize_t ret = -1;
if (src_addr == nullptr || addrlen == nullptr ||
this->ip2sockaddr_(&pkt.src_addr, pkt.src_port, src_addr, addrlen) == 0) {
ret = (ssize_t) std::min(len, (size_t) pkt.pb->tot_len);
pbuf_copy_partial(pkt.pb, buf, ret, 0);
}
pbuf_free(pkt.pb);
this->rx_read_idx_ = (this->rx_read_idx_ + 1) & UDP_RX_MASK;
this->rx_count_--;
return ret;
}
void LWIPRawUDPImpl::s_recv_fn(void *arg, struct udp_pcb *pcb, struct pbuf *p, const ip_addr_t *addr, u16_t port) {
auto *self = reinterpret_cast<LWIPRawUDPImpl *>(arg);
self->recv_fn_(p, addr, port);
}
// LWIP CALLBACK — runs from IRQ context on RP2040 (low-priority user IRQ).
// No heap allocation allowed — malloc is not IRQ-safe (see #14687).
// No LWIP_LOCK() needed — lwip core already holds the async_context lock.
void LWIPRawUDPImpl::recv_fn_(struct pbuf *p, const ip_addr_t *addr, u16_t port) {
if (p == nullptr)
return;
// Check if queue is full
if (this->rx_count_ >= UDP_RX_QUEUE_SIZE) {
// Drop packet — queue full. Can't log from IRQ context, so count it
// (saturating) for consumers to surface via get_rx_dropped().
if (this->rx_dropped_ != UINT16_MAX)
this->rx_dropped_++;
pbuf_free(p);
return;
}
// Enqueue the packet
uint8_t write_idx = (this->rx_read_idx_ + this->rx_count_) & UDP_RX_MASK;
auto &slot = this->rx_queue_[write_idx];
slot.pb = p;
slot.src_addr = *addr;
slot.src_port = port;
this->rx_count_++;
esphome::wake_loop_any_context();
}
// ---- UDP Factory functions ----
static struct udp_pcb *new_udp_pcb(int domain) {
#if LWIP_IPV6
return udp_new_ip_type(domain == AF_INET6 ? IPADDR_TYPE_ANY : IPADDR_TYPE_V4);
#else
return udp_new();
#endif
}
std::unique_ptr<UDPSendSocket> socket_udp_send(int domain, int protocol) {
(void) protocol; // Raw lwip UDP ignores protocol; kept for API compatibility
LWIP_LOCK();
auto *pcb = new_udp_pcb(domain);
if (pcb == nullptr) {
errno = ENOMEM;
return nullptr;
}
return make_unique<LWIPRawUDPSendImpl>((sa_family_t) domain, pcb);
}
std::unique_ptr<UDPSocket> socket_udp(int domain, int protocol) {
(void) protocol; // Raw lwip UDP ignores protocol; kept for API compatibility
LWIP_LOCK();
auto *pcb = new_udp_pcb(domain);
if (pcb == nullptr) {
errno = ENOMEM;
return nullptr;
}
// Ctor registers the recv callback under the lock held here
return make_unique<LWIPRawUDPImpl>((sa_family_t) domain, pcb);
}
#undef LWIP_LOCK
} // namespace esphome::socket
#endif // USE_SOCKET_IMPL_LWIP_TCP
@@ -0,0 +1,113 @@
#pragma once
#include "esphome/core/defines.h"
#ifdef USE_SOCKET_IMPL_LWIP_TCP
#include <array>
#include <cstdint>
#include "headers.h"
#include "lwip/ip.h"
#include "lwip/udp.h"
namespace esphome::socket {
/// Send-only UDP socket implementation for LWIP raw API.
/// Non-virtual, concrete type. Uses lwip/udp.h raw API.
/// No receive capability — use LWIPRawUDPImpl for sockets that need to receive.
class LWIPRawUDPSendImpl {
public:
/// The pcb is allocated by the factory (like the TCP impl); never null here.
LWIPRawUDPSendImpl(sa_family_t family, struct udp_pcb *pcb) : pcb_(pcb), family_(family) {}
~LWIPRawUDPSendImpl();
LWIPRawUDPSendImpl(const LWIPRawUDPSendImpl &) = delete;
LWIPRawUDPSendImpl &operator=(const LWIPRawUDPSendImpl &) = delete;
int bind(const struct sockaddr *name, socklen_t addrlen);
int close();
/// Send a UDP packet to the specified destination.
ssize_t sendto(const void *buf, size_t len, int flags, const struct sockaddr *dest_addr, socklen_t addrlen);
int setsockopt(int level, int optname, const void *optval, socklen_t optlen);
int getsockopt(int level, int optname, void *optval, socklen_t *optlen);
int setblocking(bool blocking);
bool ready() const { return false; }
int get_fd() const { return -1; }
protected:
/// Convert lwip ip_addr_t and port to sockaddr.
int ip2sockaddr_(const ip_addr_t *ip, uint16_t port, struct sockaddr *name, socklen_t *addrlen);
/// Shared close logic — removes the udp pcb. Caller must hold LWIP_LOCK.
int close_internal_locked_();
struct udp_pcb *pcb_;
sa_family_t family_;
};
/// UDP socket with receive support for LWIP raw API.
/// Extends LWIPRawUDPSendImpl with a fixed-size ring buffer for incoming packets.
/// Inheritance is private (base dtor is non-virtual; converting to a base
/// pointer would leak queued pbufs on destruction).
class LWIPRawUDPImpl : private LWIPRawUDPSendImpl {
public:
/// Caller (the factory) must hold the lwip lock; registers the recv callback.
LWIPRawUDPImpl(sa_family_t family, struct udp_pcb *pcb);
~LWIPRawUDPImpl();
using LWIPRawUDPSendImpl::bind;
using LWIPRawUDPSendImpl::get_fd;
using LWIPRawUDPSendImpl::getsockopt;
using LWIPRawUDPSendImpl::sendto;
using LWIPRawUDPSendImpl::setblocking;
using LWIPRawUDPSendImpl::setsockopt;
/// Close the socket, flushing any queued rx packets first.
int close();
/// Read the next queued packet, discarding source address info.
/// If buf is smaller than the packet, data is silently truncated (returns bytes copied).
/// Note: unlike POSIX MSG_TRUNC, this does not return the original packet length on truncation.
ssize_t read(void *buf, size_t len);
/// Read the next queued packet and return the source address.
/// If buf is smaller than the packet, data is silently truncated (returns bytes copied).
/// Note: unlike POSIX MSG_TRUNC, this does not return the original packet length on truncation.
ssize_t recvfrom(void *buf, size_t len, struct sockaddr *src_addr, socklen_t *addrlen);
/// Returns true if there are packets available to read.
/// Intentionally unlocked — same rationale as LWIPRawImpl::ready().
bool ready() const { return this->rx_count_ > 0; }
/// Number of packets dropped because the rx queue was full (saturating).
uint16_t get_rx_dropped() const { return this->rx_dropped_; }
protected:
static void s_recv_fn(void *arg, struct udp_pcb *pcb, struct pbuf *p, const ip_addr_t *addr, u16_t port);
void recv_fn_(struct pbuf *p, const ip_addr_t *addr, u16_t port);
/// Ring buffer for received UDP packets.
/// Both producer (recv callback) and consumer (main loop) are serialized by the
/// lwip lock — the callback runs under lwip core lock, and consumer methods hold
/// LWIP_LOCK(). All 4 slots are usable (no wasted slot for full/empty distinction).
/// No heap allocation in the recv callback — packets are dropped if the queue is full.
static constexpr uint8_t UDP_RX_QUEUE_SIZE = 4;
static constexpr uint8_t UDP_RX_MASK = UDP_RX_QUEUE_SIZE - 1;
static_assert((UDP_RX_QUEUE_SIZE & UDP_RX_MASK) == 0, "UDP_RX_QUEUE_SIZE must be power of 2");
// Fields are written by recv_fn_ before rx_count_ makes a slot visible
struct UDPRxPacket {
ip_addr_t src_addr;
struct pbuf *pb;
uint16_t src_port;
};
std::array<UDPRxPacket, UDP_RX_QUEUE_SIZE> rx_queue_{};
uint16_t rx_dropped_{0};
uint8_t rx_read_idx_{0};
uint8_t rx_count_{0};
};
} // namespace esphome::socket
#endif // USE_SOCKET_IMPL_LWIP_TCP
@@ -84,6 +84,9 @@ class LwIPSocketImpl {
int get_fd() const { return this->fd_; }
/// UDP rx drop counter parity with LWIPRawUDPImpl; drops are not counted here.
uint16_t get_rx_dropped() const { return 0; }
protected:
// fd_ < 0 means "not open" — used both pre-open (initial state) and post-close. This
// replaces a separate closed_ flag: close() sets fd_ = -1 after lwip_close(), and the
+1 -19
View File
@@ -116,25 +116,7 @@ size_t format_sockaddr_to(const struct sockaddr *addr_ptr, socklen_t len, std::s
return 0;
}
std::unique_ptr<Socket> socket_ip(int type, int protocol) {
#if USE_NETWORK_IPV6
return socket(AF_INET6, type, protocol);
#else
return socket(AF_INET, type, protocol);
#endif /* USE_NETWORK_IPV6 */
}
#ifdef USE_SOCKET_IMPL_LWIP_TCP
// LWIP_TCP has separate Socket/ListenSocket types — needs out-of-line factory.
// BSD and LWIP_SOCKETS define this inline in socket.h.
std::unique_ptr<ListenSocket> socket_ip_loop_monitored(int type, int protocol) {
#if USE_NETWORK_IPV6
return socket_listen_loop_monitored(AF_INET6, type, protocol);
#else
return socket_listen_loop_monitored(AF_INET, type, protocol);
#endif /* USE_NETWORK_IPV6 */
}
#endif
std::unique_ptr<Socket> socket_ip(int type, int protocol) { return socket(IP_DOMAIN, type, protocol); }
socklen_t set_sockaddr(struct sockaddr *addr, socklen_t addrlen, const char *ip_address, uint16_t port) {
#if USE_NETWORK_IPV6
+49 -8
View File
@@ -20,6 +20,7 @@
#include "lwip_sockets_impl.h"
#elif defined(USE_SOCKET_IMPL_LWIP_TCP)
#include "lwip_raw_tcp_impl.h"
#include "lwip_raw_udp_impl.h"
#endif
namespace esphome::socket {
@@ -27,17 +28,32 @@ namespace esphome::socket {
// Type aliases — only one implementation is active per build.
// Socket is the concrete type for connected sockets.
// ListenSocket is the concrete type for listening/server sockets.
// On BSD and LWIP_SOCKETS, both aliases resolve to the same type.
// UDPSocket is the concrete type for UDP sockets (send + receive).
// UDPSendSocket is the concrete type for send-only UDP sockets.
// On BSD and LWIP_SOCKETS, all aliases resolve to the same type.
// On LWIP_TCP, they are different types (no virtual dispatch between them).
#ifdef USE_SOCKET_IMPL_BSD_SOCKETS
using Socket = BSDSocketImpl;
using ListenSocket = BSDSocketImpl;
using UDPSendSocket = BSDSocketImpl;
using UDPSocket = BSDSocketImpl;
#elif defined(USE_SOCKET_IMPL_LWIP_SOCKETS)
using Socket = LwIPSocketImpl;
using ListenSocket = LwIPSocketImpl;
using UDPSendSocket = LwIPSocketImpl;
using UDPSocket = LwIPSocketImpl;
#elif defined(USE_SOCKET_IMPL_LWIP_TCP)
using Socket = LWIPRawImpl;
using ListenSocket = LWIPRawListenImpl;
using UDPSendSocket = LWIPRawUDPSendImpl;
using UDPSocket = LWIPRawUDPImpl;
#endif
// Domain used by the socket_ip_* helpers: newest available IP domain.
#if USE_NETWORK_IPV6
inline constexpr int IP_DOMAIN = AF_INET6;
#else
inline constexpr int IP_DOMAIN = AF_INET;
#endif
#ifdef USE_LWIP_FAST_SELECT
@@ -104,6 +120,36 @@ std::unique_ptr<Socket> socket_ip(int type, int protocol);
/// File descriptors >= FD_SETSIZE will not be monitored and will log an error.
std::unique_ptr<Socket> socket_loop_monitored(int domain, int type, int protocol);
/// Create a send-only UDP socket (socket_udp_send), a UDP socket with receive
/// support (socket_udp), or a UDP socket monitored for data in the main loop
/// (socket_udp_loop_monitored).
#ifdef USE_SOCKET_IMPL_LWIP_TCP
std::unique_ptr<UDPSendSocket> socket_udp_send(int domain, int protocol);
std::unique_ptr<UDPSocket> socket_udp(int domain, int protocol);
// Wake is built into the recv callback, so monitoring needs nothing extra.
inline std::unique_ptr<UDPSocket> socket_udp_loop_monitored(int domain, int protocol) {
return socket_udp(domain, protocol);
}
#else
inline std::unique_ptr<UDPSendSocket> socket_udp_send(int domain, int protocol) {
return esphome::socket::socket(domain, SOCK_DGRAM, protocol);
}
inline std::unique_ptr<UDPSocket> socket_udp(int domain, int protocol) {
return esphome::socket::socket(domain, SOCK_DGRAM, protocol);
}
// Registers the socket with the Application's select() loop.
inline std::unique_ptr<UDPSocket> socket_udp_loop_monitored(int domain, int protocol) {
return socket_loop_monitored(domain, SOCK_DGRAM, protocol);
}
#endif
/// socket_udp* variants using the newest available IP domain.
inline std::unique_ptr<UDPSendSocket> socket_ip_udp_send(int protocol) { return socket_udp_send(IP_DOMAIN, protocol); }
inline std::unique_ptr<UDPSocket> socket_ip_udp(int protocol) { return socket_udp(IP_DOMAIN, protocol); }
inline std::unique_ptr<UDPSocket> socket_ip_udp_loop_monitored(int protocol) {
return socket_udp_loop_monitored(IP_DOMAIN, protocol);
}
/// Create a listening socket of the given domain, type and protocol.
/// Create a listening socket and monitor it for data in the main loop.
/// Create a listening socket in the newest available IP domain and monitor it.
@@ -111,7 +157,6 @@ std::unique_ptr<Socket> socket_loop_monitored(int domain, int type, int protocol
// LWIP_TCP has separate Socket/ListenSocket types — needs distinct factory functions.
std::unique_ptr<ListenSocket> socket_listen(int domain, int type, int protocol);
std::unique_ptr<ListenSocket> socket_listen_loop_monitored(int domain, int type, int protocol);
std::unique_ptr<ListenSocket> socket_ip_loop_monitored(int type, int protocol);
#else
// BSD and LWIP_SOCKETS: Socket == ListenSocket, so listen variants just delegate.
inline std::unique_ptr<ListenSocket> socket_listen(int domain, int type, int protocol) {
@@ -120,14 +165,10 @@ inline std::unique_ptr<ListenSocket> socket_listen(int domain, int type, int pro
inline std::unique_ptr<ListenSocket> socket_listen_loop_monitored(int domain, int type, int protocol) {
return socket_loop_monitored(domain, type, protocol);
}
#endif
inline std::unique_ptr<ListenSocket> socket_ip_loop_monitored(int type, int protocol) {
#if USE_NETWORK_IPV6
return socket_loop_monitored(AF_INET6, type, protocol);
#else
return socket_loop_monitored(AF_INET, type, protocol);
#endif
return socket_listen_loop_monitored(IP_DOMAIN, type, protocol);
}
#endif
/// Set a sockaddr to the specified address and port for the IP version used by socket_ip().
/// @param addr Destination sockaddr structure
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
+4 -5
View File
@@ -1138,14 +1138,13 @@ class EsphomeCore:
def add_platformio_option(
self, key: str, value: str | list[str], *, replace: bool = False
) -> None:
"""Set a platformio.ini option; values append to an existing list
(a string as one element) unless ``replace`` is True, which
overwrites any existing value."""
"""Set a platformio.ini option; list values append to an existing list
unless ``replace`` is True, which overwrites any existing value."""
new_val = value
old_val = self.platformio_options.get(key)
if not replace and isinstance(old_val, list):
# A user platformio_options string must merge, not assert
new_val = old_val + ([value] if isinstance(value, str) else value)
assert isinstance(value, list)
new_val = old_val + value
self.platformio_options[key] = new_val
def _get_variable_generator(self, id):
+1 -8
View File
@@ -555,14 +555,7 @@ NATIVE_ARDUINO_PIO_OPTIONS = frozenset({"board_build.f_cpu", "board_build.ldscri
# that is stored rather than translated away. Consumed by the esp8266 native
# backend (later in this chain) for its ignored-option warning; defined here
# so it stays adjacent to the routing.
# build_src_flags and board_build.flash_mode: set unconditionally by
# esp8266/__init__ and read by the native generator; not user-routable, so
# not in the set above
NATIVE_ARDUINO_CONSUMED_PIO_OPTIONS = NATIVE_ARDUINO_PIO_OPTIONS | {
"lib_ignore",
"build_src_flags",
"board_build.flash_mode",
}
NATIVE_ARDUINO_CONSUMED_PIO_OPTIONS = NATIVE_ARDUINO_PIO_OPTIONS | {"lib_ignore"}
@coroutine_with_priority(CoroPriority.FINAL)
-8
View File
@@ -1,8 +0,0 @@
#pragma once
// Curated precompiled-header prefix for backends that force-include it via
// build_src_flags (the pch script folds it into the .gch). Guarded because
// build_src_flags also reaches C and assembly src edges.
#ifdef __cplusplus
#include "esphome/core/application.h"
#include "esphome/core/automation.h"
#endif
-68
View File
@@ -1,68 +0,0 @@
"""Shared tool resolution for the sibling idf_tools-backed scripts.
Importable because ``python <script>`` puts this directory first on
sys.path; ``idf_tools`` itself comes from PYTHONPATH.
"""
# pylint: disable=import-error # idf_tools is on PYTHONPATH at runtime only
from collections.abc import Callable, Iterator
import os
from pathlib import Path
from idf_tools import (
CURRENT_PLATFORM,
TOOLS_FILE,
IDFEnv,
ToolBinaryError,
add_and_check_targets,
expand_tools_arg,
g,
load_tools_info,
)
def init_idf_tools(idf_framework_root: str) -> None:
"""Point idf_tools' globals at the framework and IDF_TOOLS_PATH."""
g.idf_path = idf_framework_root
g.idf_tools_path = os.environ.get("IDF_TOOLS_PATH")
g.tools_json = str(Path(g.idf_path) / TOOLS_FILE)
def archive_name(download: object) -> str:
"""The dist/ filename idf_tools downloads and installs this from."""
return download.rename_dist or Path(download.url).name
def iter_tool_downloads(
targets_csv: str,
tool_specs: list[str],
on_broken: Callable[[str, ToolBinaryError], bool],
) -> Iterator[tuple[object, str, str, object]]:
"""Yield (tool, name, version, download) per uninstalled tool, mirroring
``idf_tools.py install``'s expansion; ``on_broken(name, err)`` returns
True to treat a tool with a failing installed binary as not installed."""
targets = add_and_check_targets(IDFEnv.get_idf_env(), targets_csv)
tools_info = load_tools_info()
for name in expand_tools_arg(tool_specs, tools_info, targets):
if "@" in name:
name, version = name.split("@", 1)
else:
version = None
tool = tools_info.get(name)
if tool is None or not tool.compatible_with_platform():
continue
version = version or tool.get_recommended_version()
if version is None:
continue
try:
tool.find_installed_versions()
except ToolBinaryError as e:
if not on_broken(name, e):
continue
if version in tool.versions_installed or version not in tool.versions:
continue
download = tool.versions[version].get_download_for_platform(CURRENT_PLATFORM)
if download is None:
continue
yield tool, name, version, download
+27 -89
View File
@@ -16,11 +16,9 @@ from esphome.build_helpers.ccache import (
parse_enable_env,
resolve_ccache_path,
)
from esphome.build_helpers.pch import ccache_pch_env
from esphome.build_helpers.tools_cache import IDF_TOOLS_CACHE, tools_cache_path
from esphome.core import Version
from esphome.framework_helpers import (
BATCH_EXTRACT_WORKERS,
PathType,
create_venv,
download_and_extract,
@@ -28,7 +26,6 @@ from esphome.framework_helpers import (
failure_reason,
get_python_env_executable_path,
get_system_python_path,
is_expected_fetch_error,
resume_fetch_job,
rmdir,
run_batch_downloads,
@@ -36,9 +33,9 @@ from esphome.framework_helpers import (
run_command_ok,
str_to_lst_of_str,
tool_version_runs,
warn_batch_failures,
warn_prefetch_failures,
)
from esphome.helpers import get_usable_cpu_count, write_file_if_changed
from esphome.helpers import write_file_if_changed
_LOGGER = logging.getLogger(__name__)
@@ -294,13 +291,11 @@ def _run_idf_tools_script(
msg: str,
args: list[str] | None = None,
env: dict[str, str] | None = None,
stream_output: bool = False,
) -> tuple[bool, str | None, str | None]:
"""Run one of the sibling idf_tools-backed helper scripts.
PYTHONPATH carries this directory (sibling imports like
``_tool_resolution``), the esphome package root (``esphome.helpers``),
and the framework's ``tools`` dir (its own ``idf_tools`` module).
The script is executed with the framework's ``tools`` directory on
PYTHONPATH so it imports the framework's own ``idf_tools`` module.
"""
cmd = [
get_system_python_path(),
@@ -308,20 +303,11 @@ def _run_idf_tools_script(
str(idf_framework_root),
*(args or []),
]
# Explicit paths: the scripts dir (sibling imports must survive
# PYTHONSAFEPATH), the esphome package root, and the framework's idf_tools
pythonpath = os.pathsep.join(
(
str(_SCRIPTS_DIR),
str(_SCRIPTS_DIR.parents[1]),
str(Path(idf_framework_root) / "tools"),
)
)
return run_command(
cmd,
msg=msg,
env=(env or os.environ) | {"PYTHONPATH": pythonpath},
stream_output=stream_output,
env=(env or os.environ)
| {"PYTHONPATH": str(Path(idf_framework_root) / "tools")},
)
@@ -739,9 +725,9 @@ def _prefetch_idf_tool_archives(
dist_path = get_idf_tools_path() / "dist"
entries = []
seen_dests: set[str] = set()
# Pre-existing archives are not skipped: download_with_resume keeps
# them only on a sha256 match, so the pre-extraction can trust dist/
for entry in json.loads(stdout):
if (dist_path / entry["dest"]).is_file():
continue
# Never download unverified: an entry without sha256/size is
# left to the installer, which fails loudly on a bad archive.
# Checked before the dedupe so it cannot shadow a verifiable
@@ -761,11 +747,9 @@ def _prefetch_idf_tool_archives(
entries.append(entry)
if not entries:
return
cached = sum((dist_path / entry["dest"]).is_file() for entry in entries)
_LOGGER.info(
"Downloading %d ESP-IDF tool archive(s)%s: %s",
"Downloading %d ESP-IDF tool archive(s): %s",
len(entries),
f" ({cached} cached, verifying)" if cached else "",
", ".join(entry["name"] for entry in entries),
)
@@ -789,7 +773,7 @@ def _prefetch_idf_tool_archives(
for entry in entries
],
)
warn_batch_failures(failures)
warn_prefetch_failures(failures)
if len(failures) == len(entries):
# A systematic fault, not one flaky mirror: the resume
# workaround (#17703) is off for this whole install
@@ -804,42 +788,6 @@ def _prefetch_idf_tool_archives(
_LOGGER.debug("Prefetch failure detail", exc_info=True)
def _preinstall_idf_tool_archives(
framework_path: Path,
targets_str: str,
tools: list[str],
env: dict[str, str] | None,
) -> None:
"""Run install_tool_archives.py to extract the prefetched tool archives
in parallel. Strictly best-effort: the sequential installer remains the
authority (see that script's docstring)."""
try:
success, _stdout, _stderr = _run_idf_tools_script(
framework_path,
"install_tool_archives.py",
"ESP-IDF tool archive extraction",
args=[
targets_str,
str(min(get_usable_cpu_count(), BATCH_EXTRACT_WORKERS)),
*tools,
],
env=env,
stream_output=True,
)
if not success:
# Detail already streamed to the terminal by the script; a
# surviving torn dir prints its own guidance there
_LOGGER.warning("ESP-IDF tool pre-extraction failed; see above")
except Exception as e: # noqa: BLE001 # pylint: disable=broad-exception-caught
# A programming error keeps its traceback at WARNING
_LOGGER.warning(
"ESP-IDF tool pre-extraction failed: %s",
failure_reason(e),
exc_info=None if is_expected_fetch_error(e) else e,
)
_LOGGER.debug("Pre-extraction failure detail", exc_info=True)
def _check_esphome_idf_framework_install(
version: str,
targets: list[str],
@@ -991,7 +939,6 @@ def _check_esphome_idf_framework_install(
_LOGGER.info("Installing ESP-IDF %s framework ...", version)
targets_str = ",".join(targets)
_prefetch_idf_tool_archives(framework_path, targets_str, tools, env)
_preinstall_idf_tool_archives(framework_path, targets_str, tools, env)
cmd = [
get_system_python_path(),
str(idf_tools_path),
@@ -1243,33 +1190,15 @@ def _ccache_env() -> dict[str, str]:
Only values the user has not already set in the environment are returned, so
a custom ``CCACHE_DIR`` / ``CCACHE_MAXSIZE`` / etc. is respected.
The pch settings add ``time_macros`` sloppiness process-wide; the visible
effect is a cached TU can keep an older ``esp_app_desc`` build timestamp.
"""
if not _ccache_enabled():
# The raw knob value (e.g. "disable") is still inherited by idf.py
# via os.environ, where a non-false-constant string reads as
# truthy; export the canonical off spelling instead
return {"IDF_CCACHE_ENABLE": "0"}
env = ccache_defaults_env(get_idf_tools_path() / "ccache")
env.update(ccache_pch_env())
# Exactly one canonical spelling ever reaches idf.py, whatever the
# accepted input spelling was ("enable", "yes", ...)
env["IDF_CCACHE_ENABLE"] = "1"
return env
def _ccache_enabled() -> bool:
"""Whether ESP-IDF compiles run under ccache.
IDF_CCACHE_ENABLE (the backend-native knob) wins over the shared
ESPHOME_CCACHE_ENABLE; when unset, enabled iff a runnable binary is
on PATH.
"""
# IDF_CCACHE_ENABLE (the backend-native knob) wins over the shared
# ESPHOME_CCACHE_ENABLE.
idf_knob = parse_enable_env("IDF_CCACHE_ENABLE")
if idf_knob is False:
return False
# The raw value (e.g. "disable") is still inherited by idf.py via
# os.environ, where a non-false-constant string reads as truthy;
# export the canonical off spelling instead
return {"IDF_CCACHE_ENABLE": "0"}
if idf_knob is True:
# Forced on ignores the runnability verdict, but the outcome is
# worth saying out loud. Probed directly (not via the resolver,
@@ -1289,8 +1218,17 @@ def _ccache_enabled() -> bool:
"IDF_CCACHE_ENABLE=1 forces on the ccache at %s even though "
"it failed to run; idf.py will use it anyway",
)
return True
return resolve_ccache_path() is not None
elif resolve_ccache_path() is None:
# ESP-IDF silently skips ccache without the binary; export the
# canonical off spelling so an unparsable inherited value (or a
# probe-rejected ccache idf.py would still find) cannot enable it
return {"IDF_CCACHE_ENABLE": "0"}
env = ccache_defaults_env(get_idf_tools_path() / "ccache")
# Exactly one canonical spelling ever reaches idf.py, whatever the
# accepted input spelling was ("enable", "yes", ...)
env["IDF_CCACHE_ENABLE"] = "1"
return env
def get_framework_env(
+55 -25
View File
@@ -1,9 +1,8 @@
"""Print JSON download info for the ESP-IDF tools an install would fetch.
Run via ``python <this file> <idf_framework_root> <targets-csv> <tool-spec>...``.
PYTHONPATH must include this directory (for ``_tool_resolution``) and
``<idf_framework_root>/tools`` (for ``idf_tools``), and IDF_TOOLS_PATH must
be set. Prints a JSON list of
PYTHONPATH must include ``<idf_framework_root>/tools`` so ``idf_tools`` is
importable, and IDF_TOOLS_PATH must be set. Prints a JSON list of
``{name, url, size, sha256, dest}`` for every tool version that is not yet
installed, where ``dest`` is the archive filename ``idf_tools.py install``
expects to find in ``<IDF_TOOLS_PATH>/dist``. Tools with no download for the
@@ -19,36 +18,67 @@ or written — this script only reports what the install would download.
from contextlib import redirect_stdout
import json
import os
from pathlib import Path
import sys
from _tool_resolution import archive_name, init_idf_tools, iter_tool_downloads
from idf_tools import ToolBinaryError, get_idf_download_url_apply_mirrors
from idf_tools import (
CURRENT_PLATFORM,
TOOLS_FILE,
IDFEnv,
ToolBinaryError,
add_and_check_targets,
expand_tools_arg,
g,
get_idf_download_url_apply_mirrors,
load_tools_info,
)
def collect_downloads() -> list[dict]:
init_idf_tools(sys.argv[1])
g.idf_path = sys.argv[1]
g.idf_tools_path = os.environ.get("IDF_TOOLS_PATH")
g.tools_json = str(Path(g.idf_path) / TOOLS_FILE)
def on_broken(name: str, e: ToolBinaryError) -> bool:
# A broken installed binary is idf_tools' problem to repair on
# install; note it and treat the version as not installed.
print(f"tool {name} failed its binary check: {e}", file=sys.stderr)
return True
targets = add_and_check_targets(IDFEnv.get_idf_env(), sys.argv[2])
tools_info = load_tools_info()
downloads: list[dict] = []
return [
{
"name": f"{name}@{version}",
# Apply the same IDF_MIRROR_PREFIX_MAP / IDF_GITHUB_ASSETS
# rewriting the installer's own downloader applies, so users
# behind a mirror prefetch from the mirror too.
"url": get_idf_download_url_apply_mirrors(None, download.url),
"size": download.size,
"sha256": download.sha256,
"dest": archive_name(download),
}
for _tool, name, version, download in iter_tool_downloads(
sys.argv[2], sys.argv[3:], on_broken
for name in expand_tools_arg(sys.argv[3:], tools_info, targets):
if "@" in name:
name, version = name.split("@", 1)
else:
version = None
tool = tools_info.get(name)
if tool is None or not tool.compatible_with_platform():
continue
version = version or tool.get_recommended_version()
if version is None:
continue
try:
tool.find_installed_versions()
except ToolBinaryError as e:
# A broken installed binary is idf_tools' problem to repair on
# install; note it and treat the version as not installed.
print(f"tool {name} failed its binary check: {e}", file=sys.stderr)
if version in tool.versions_installed or version not in tool.versions:
continue
download = tool.versions[version].get_download_for_platform(CURRENT_PLATFORM)
if download is None:
continue
downloads.append(
{
"name": f"{name}@{version}",
# Apply the same IDF_MIRROR_PREFIX_MAP / IDF_GITHUB_ASSETS
# rewriting the installer's own downloader applies, so users
# behind a mirror prefetch from the mirror too.
"url": get_idf_download_url_apply_mirrors(None, download.url),
"size": download.size,
"sha256": download.sha256,
"dest": download.rename_dist or Path(download.url).name,
}
)
]
return downloads
# idf_tools prints informational lines (e.g. mirror URL rewrites) to stdout;
-129
View File
@@ -1,129 +0,0 @@
"""Extract prefetched ESP-IDF tool archives in parallel.
Run via ``python <this file> <idf_framework_root> <targets-csv> <workers>
<tool-spec>...`` with idf_tools and the esphome package root on PYTHONPATH
and IDF_TOOLS_PATH set.
Drives idf_tools' own ``IDFTool.install()`` so extraction semantics match
the sequential installer, which still runs afterwards as the authority and
redoes anything this best-effort pass failed on. Archives are trusted from
the prefetch's sha256 verification, not re-hashed here.
"""
# pylint: disable=import-error # idf_tools is on PYTHONPATH at runtime only
from concurrent.futures import ThreadPoolExecutor
from pathlib import Path
import sys
from _tool_resolution import archive_name, init_idf_tools, iter_tool_downloads
from idf_tools import ToolBinaryError, g
from esphome.helpers import rmtree
def collect_pending(
targets_csv: str, tool_specs: list[str]
) -> tuple[dict[tuple[str, str], object], int]:
"""The {(name, version): tool} jobs whose verified archive is on disk,
and how many distinct uninstalled tools were resolved overall."""
dist_path = Path(g.idf_tools_path) / "dist"
def on_broken(name: str, e: ToolBinaryError) -> bool:
# Repairing a broken installed binary is the installer's job
print(f"leaving broken {name} to the installer: {e}", file=sys.stderr)
return False
pending: dict[tuple[str, str], object] = {}
resolved: set[tuple[str, str]] = set()
for tool, name, version, download in iter_tool_downloads(
targets_csv, tool_specs, on_broken
):
resolved.add((name, version))
# Mirror the prefetch: an entry it could not verify is never trusted
if not (download.sha256 and download.size):
continue
# Trusted as-is: the prefetch verifies archives at their final name,
# and the installer redoes anything this pass fails on
if (name, version) in pending or not (
dist_path / archive_name(download)
).is_file():
continue
pending[(name, version)] = tool
return pending, len(resolved)
def install_one(tool: object, name: str, version: str) -> bool | None:
"""True on success, False on a cleaned-up failure, None when the torn
dest dir survived and could fool the installer's binary probe."""
try:
tool.install(version)
# check_binary_valid exits via SystemExit; the installer redoes failures
except (Exception, SystemExit) as e: # noqa: BLE001 # pylint: disable=broad-exception-caught
# Name the type: idf_tools' fatal() raises SystemExit(1), which
# would render as a bare "1"
print(
f"pre-extracting {name}@{version} failed, leaving it to the "
f"installer: {type(e).__name__}: {e}",
file=sys.stderr,
)
# A torn dest dir must not look installed to the installer
dest = tool.get_path_for_version(version)
try:
rmtree(dest)
except FileNotFoundError: # pragma: no cover # failed before mkdir
pass
except OSError as cleanup_err:
print(
f"could not remove {dest}: {cleanup_err}; the installer may "
"trust the partial tool dir, delete it manually if the build "
"fails",
file=sys.stderr,
)
return None
return False
# Per-tool completion keeps the multi-minute unpack phase visibly alive
print(f"extracted {name}@{version}", flush=True)
return True
def main() -> None:
_script, idf_framework_root, targets_csv, workers_str, *tool_specs = sys.argv
init_idf_tools(idf_framework_root)
pending, resolved = collect_pending(targets_csv, tool_specs)
if len(pending) < 2:
# Nothing to parallelize; the count makes a naming/resolution drift
# that would silently disable this pass observable
print(
f"{len(pending)} of {resolved} uninstalled tool(s) have a "
"prefetched archive; leaving them to the installer",
flush=True,
)
return
workers = min(int(workers_str), len(pending))
print(
f"Extracting {len(pending)} ESP-IDF tool archive(s) with "
f"{workers} worker(s): "
+ ", ".join(f"{name}@{version}" for name, version in pending),
flush=True,
)
with ThreadPoolExecutor(max_workers=workers) as ex:
futures = [
ex.submit(install_one, tool, name, version)
for (name, version), tool in pending.items()
]
try:
results = [future.result() for future in futures]
except BaseException: # pragma: no cover
# Ctrl-C: drop queued extractions; in-flight ones finish whole
ex.shutdown(wait=True, cancel_futures=True)
raise
# A survivor could fool the installer; every job failing is systematic.
# Either way a nonzero exit makes the caller warn
failed = sum(result is not True for result in results)
if failed:
print(f"{failed} of {len(results)} pre-extractions failed", file=sys.stderr)
if None in results or failed == len(results):
sys.exit(1)
main()
-6
View File
@@ -528,12 +528,6 @@ def run_compile(config, verbose: bool) -> int:
return result.returncode
_patch_memory_segments()
# After every reconfigure so compile_commands and sdkconfig are settled
from esphome.build_gen.espidf import prepare_pch
from esphome.build_helpers.pch import guarded_prepare
guarded_prepare(CORE.relative_build_path("build"), prepare_pch)
# Build
args = []
+27 -58
View File
@@ -287,25 +287,10 @@ def _detect_archive_root(names: Iterable[str]) -> str | None:
return root if has_descendant else None
def _resolve_progress(
progress: Callable[[float], None] | None,
progress_header: str | None,
has_work: bool,
) -> Callable[[float], None] | None:
"""Fraction reporter for an extractor: the caller's callback wins over a
private ``progress_header`` bar."""
if progress is not None:
return progress
if progress_header and has_work:
return ProgressBar(progress_header).update
return None
def _tar_extract_all(
data: io.BufferedIOBase,
extract_dir: PathType = ".",
progress_header: str | None = None,
progress: Callable[[float], None] | None = None,
):
"""
Extract a TAR archive to the specified directory.
@@ -320,7 +305,6 @@ def _tar_extract_all(
data: File-like object containing the TAR archive
extract_dir: Directory to extract contents to
progress_header: If set, show a progress bar with this header
progress: fraction callback (0..1, ends at 1.0); overrides progress_header
"""
import tarfile
@@ -379,20 +363,21 @@ def _tar_extract_all(
safe_members.append(member)
total = len(safe_members)
report = _resolve_progress(progress, progress_header, total > 0)
progress = (
ProgressBar(progress_header) if progress_header and total > 0 else None
)
for i, member in enumerate(safe_members, 1):
tar_ref.extract(member, abs_dest)
if report is not None:
report(i / total)
if report is not None:
report(1)
if progress is not None:
progress.update(i / total)
if progress is not None:
progress.update(1)
def _zip_extract_all(
data: io.BufferedIOBase,
extract_dir: PathType = ".",
progress_header: str | None = None,
progress: Callable[[float], None] | None = None,
):
"""
Extract a ZIP archive to the specified directory.
@@ -401,7 +386,6 @@ def _zip_extract_all(
data: File-like object containing the ZIP archive
extract_dir: Directory to extract contents to
progress_header: If set, show a progress bar with this header
progress: fraction callback (0..1, ends at 1.0); overrides progress_header
"""
import zipfile
@@ -418,7 +402,9 @@ def _zip_extract_all(
strip_prefix = f"{strip_root}/" if strip_root is not None else None
total = len(all_members)
report = _resolve_progress(progress, progress_header, total > 0)
progress = (
ProgressBar(progress_header) if progress_header and total > 0 else None
)
for i, member in enumerate(all_members, 1):
# 1. Normalize name
@@ -451,10 +437,10 @@ def _zip_extract_all(
# 6. Extract
zip_ref.extract(member, extract_dir)
if report is not None:
report(i / total)
if report is not None:
report(1)
if progress is not None:
progress.update(i / total)
if progress is not None:
progress.update(1)
def _rename_with_retry(
@@ -485,7 +471,6 @@ def _7z_extract_all(
data: io.BufferedIOBase,
extract_dir: PathType = ".",
progress_header: str | None = None,
progress: Callable[[float], None] | None = None,
):
"""
Extract a 7z archive to the specified directory.
@@ -500,7 +485,6 @@ def _7z_extract_all(
data: File-like object containing the 7z archive (must be seekable)
extract_dir: Directory to extract contents to
progress_header: If set, show a progress bar with this header
progress: called with 1.0 on completion; overrides progress_header
"""
import py7zr
@@ -539,15 +523,19 @@ def _7z_extract_all(
continue
safe_targets.append(raw)
report = _resolve_progress(progress, progress_header, bool(safe_targets))
progress = (
ProgressBar(progress_header)
if progress_header and safe_targets
else None
)
if len(safe_targets) == len(all_names):
z.extractall(path=staging)
else:
z.extract(path=staging, targets=safe_targets)
if report is not None:
report(1)
if progress is not None:
progress.update(1)
src_root = staging / strip_root if strip_root else staging
for item in src_root.iterdir():
@@ -578,7 +566,6 @@ def archive_extract_all(
archive: PathType | io.RawIOBase | IO[bytes],
extract_dir: PathType = ".",
progress_header: str | None = None,
progress: Callable[[float], None] | None = None,
):
"""
Extract an archive file to the specified directory.
@@ -587,7 +574,6 @@ def archive_extract_all(
archive: Path to archive file or file-like object
extract_dir: Directory to extract contents to
progress_header: If set, show a progress bar with this header
progress: fraction callback (0..1, ends at 1.0); overrides progress_header
Raises:
TypeError: If archive is not a valid type
@@ -618,9 +604,7 @@ def archive_extract_all(
break
if matched_fct is None:
raise ValueError("Unsupported archive format")
matched_fct(
archive_ref, extract_dir, progress_header=progress_header, progress=progress
)
matched_fct(archive_ref, extract_dir, progress_header=progress_header)
def _open_ranged(
@@ -784,17 +768,13 @@ def _stream_response_to_file(
# hammering the host or the mirrors.
BATCH_DOWNLOAD_WORKERS = 4
# Concurrent archive extractions per batch; unpacking stops scaling well
# before high core counts since the workers share one disk.
BATCH_EXTRACT_WORKERS = 10
def run_batch_downloads(
header: str,
jobs: list[tuple[str, int, Callable[[Callable[[int], None]], None]]],
max_workers: int = BATCH_DOWNLOAD_WORKERS,
) -> list[tuple[str, BaseException]]:
"""Run ``(name, size, fetch)`` jobs concurrently under one bar.
"""Run ``(name, size, fetch)`` download jobs concurrently under one bar.
Each ``fetch(tracker)`` reports absolute byte counts; the bar total is
the sum of the sizes. Failures are returned after the bar is done so
@@ -969,26 +949,15 @@ def resume_fetch_job(
return fetch
def is_expected_fetch_error(err: BaseException) -> bool:
"""Download failures the callers degrade on, vs programming errors."""
from esphome.core import EsphomeError # local import avoids circular dependency
return isinstance(err, (EsphomeError, OSError))
def warn_batch_failures(
def warn_prefetch_failures(
failures: list[tuple[str, BaseException]],
message: str = "Could not prefetch %s: %s",
) -> None:
"""Warn per failed batch job, keeping the traceback of unexpected errors."""
"""Warn per failed batch-prefetch job; the caller's installer retries them."""
for name, err in failures:
# failure_reason: a message-less exception must not log blank
if is_expected_fetch_error(err):
_LOGGER.warning(message, name, failure_reason(err))
_LOGGER.debug("Failure detail", exc_info=err)
else:
# A programming error must not be reduced to a bare message
_LOGGER.warning(message, name, failure_reason(err), exc_info=err)
_LOGGER.warning(message, name, failure_reason(err))
_LOGGER.debug("Prefetch failure detail", exc_info=err)
def download_with_resume(
+9 -104
View File
@@ -35,7 +35,7 @@ from esphome.framework_helpers import (
failure_reason,
rmdir,
run_batch_downloads,
warn_batch_failures,
warn_prefetch_failures,
)
_LOGGER = logging.getLogger(__name__)
@@ -74,11 +74,6 @@ SOURCE_KIND_FOR_SUFFIX: dict[str, str] = {
".ASM": "asm",
}
SRC_FILE_EXTENSIONS = list(SOURCE_KIND_FOR_SUFFIX)
# Suffixes that count as headers when probing whether a library has any
# usable files at all (compare against Path.suffix.lower())
LIBRARY_HEADER_SUFFIXES = frozenset(
{".h", ".hpp", ".hh", ".hxx", ".inc", ".ipp", ".tcc"}
)
DOMAIN = "pio_components"
@@ -334,11 +329,6 @@ class LibraryBackend:
framework: str
emit: Callable[["ConvertedLibrary"], None]
cache_key: str
# Owner-less names this returns True for are skipped by the walk;
# the backend supplies them itself (e.g. core-bundled libraries) and
# reconciles provided_requests after resolving
provides: Callable[[str], bool] | None = None
provided_requests: set[str] = field(default_factory=set)
def ensure_list[T](obj: T | list[T]) -> list[T]:
@@ -479,7 +469,7 @@ def _valid_manifest_shape(data: Any) -> bool:
)
def check_library_data(data: dict, platform: str | None, framework: str | None):
def check_library_data(data: dict, platform: str | None, framework: str):
"""
Check whether a library manifest is compatible with the target toolchain.
@@ -496,8 +486,7 @@ def check_library_data(data: dict, platform: str | None, framework: str | None):
for targets (e.g. Zephyr) where PIO manifests rarely declare the
platform yet portable libraries still build.
framework: The active framework name (e.g. ``espidf``, ``arduino``,
``zephyr``) the manifest is expected to declare. ``None`` skips
the framework check (and its warning), mirroring ``platform``.
``zephyr``) the manifest is expected to declare.
Raises:
InvalidLibrary: If the library does not support the target platform.
@@ -528,7 +517,7 @@ def check_library_data(data: dict, platform: str | None, framework: str | None):
# under the target framework, and there's no way to opt out of the check at
# this layer. Warn instead of failing so the user isn't forced to fork the
# library to fix the manifest.
valid_framework = framework is None or "*" in frameworks or framework in frameworks
valid_framework = "*" in frameworks or framework in frameworks
if not valid_framework:
_LOGGER.warning(
@@ -925,56 +914,6 @@ def is_lib_ignored(name: str | None, lib_ignore: set[str]) -> bool:
)
def _reconcile_versionless_skips(
skipped_versionless: list[tuple[Any, Any, str]],
components: dict[str, ConvertedLibrary],
backend: LibraryBackend,
) -> None:
"""Warn for version-less deps nothing satisfied, and record the
backend-provided ones in ``backend.provided_requests`` for its
post-emit reconciliation; a silent drop surfaces as link errors far
from the cause."""
resolved_manifest_names = {c.data.get("name") for c in components.values()}
# A treeless backend can never supply a bundled name; noise for it
log = _LOGGER.warning if backend.provides is not None else _LOGGER.debug
warned: set[str] = set()
for dep_name, dep_owner, requester in skipped_versionless:
if not isinstance(dep_name, str) or not dep_name or dep_name in warned:
continue
if dep_name in components:
# A version-less dep's request key is the name itself
continue
if (
not dep_owner
and backend.provides is not None
and backend.provides(dep_name)
):
# provides() only satisfies owner-less names (same guard as
# the walk's skip); record for the post-emit reconciliation.
# Checked before the manifest-name evidence so the overlap
# case warns once, in the backend's own suppression loop
backend.provided_requests.add(dep_name)
continue
if dep_name in resolved_manifest_names:
# Name-only evidence: a coincidental collision must stay
# visible where the user could pin it
warned.add(dep_name)
log(
"Version-less dependency %s of %s assumed satisfied by a "
"resolved library's manifest name only",
dep_name,
requester,
)
continue
warned.add(dep_name)
log(
"Dependency %s of %s has no version to resolve and nothing "
"provides it; skipping",
dep_name,
requester,
)
def _fetch_source(
component: ConvertedLibrary,
salt: str,
@@ -1040,7 +979,7 @@ def _prefetch_wave(
],
)
# The sequential call below retries and raises the real error
warn_batch_failures(
warn_prefetch_failures(
failures, "Prefetch of %s failed (retrying sequentially): %s"
)
except Exception as err: # noqa: BLE001 # pylint: disable=broad-exception-caught
@@ -1144,8 +1083,6 @@ def convert_libraries(
components: dict[str, ConvertedLibrary] = {}
resolved_requirements: dict[str, frozenset[str]] = {}
top_level_keys = set(top_level)
# (name, owner, requester) reconciled against the final resolution set
skipped_versionless: list[tuple[Any, Any, str]] = []
worklist = deque(dict.fromkeys(top_level))
while worklist:
# Drain the frontier sequentially (spec resolution mutates shared
@@ -1250,23 +1187,13 @@ def convert_libraries(
component.data.get("dependencies"), component.name
):
if "version" not in dependency:
# Cannot resolve from the registry; the post-emit
# reconciliation owns the drop warning
dep_name = dependency.get("name")
# Cannot resolve from the registry; common for bundled
# names (Wire, SPI) -- unactionable noise above debug
_LOGGER.debug(
"Skip version-less dependency %r of %s",
dep_name,
dependency.get("name"),
component.name,
)
if not is_lib_ignored(
dep_name, lib_ignore
) and dependency_is_usable(
dependency, backend.platform, backend.framework, component.name
):
# Filtered or ignored deps are deliberately absent
skipped_versionless.append(
(dep_name, dependency.get("owner"), component.name)
)
continue
if not dependency_is_usable(
dependency, backend.platform, backend.framework, component.name
@@ -1278,31 +1205,11 @@ def convert_libraries(
if is_lib_ignored(dep_name, lib_ignore):
_LOGGER.debug("Skip ignored dependency %s", dep_name)
continue
# The version may be a URL (git/archive), which names one
# specific source; never substitute a bundled library for it
# The version field may actually be a URL (git/archive dependency).
dep_version = dependency["version"]
dep_url = _url_or_none(dep_version)
if dep_url is not None:
dep_version = None
elif (
backend.provides is not None
and not dependency.get("owner")
and backend.provides(dep_name)
):
# The backend adds it from its own tree; resolving here
# would fetch a same-named registry package
if dep_version and dep_version != "*":
# The pin is discarded; make the substitution visible
_LOGGER.warning(
"Dependency %s pins version %s; using the library "
"bundled with the framework instead",
dep_name,
dep_version,
)
else:
_LOGGER.debug("Skip backend-provided dependency %s", dep_name)
backend.provided_requests.add(dep_name)
continue
dep_key = add_spec(dep_name, dep_version, dep_url)
node.edges.add(dep_key)
worklist.append(dep_key)
@@ -1356,6 +1263,4 @@ def convert_libraries(
for component in components.values():
backend.emit(component)
_reconcile_versionless_skips(skipped_versionless, components, backend)
return [components[key] for key in top_level if key in components]
-416
View File
@@ -1,416 +0,0 @@
import hashlib
import os
from pathlib import Path
import posixpath
import re
import shlex
import stat
import subprocess
import traceback
# pylint: disable=E0602
Import("env") # noqa: F821
_projenv_error = None
try:
Import("projenv") # noqa: F821
except Exception as err: # noqa: BLE001 -- not exported under -t nobuild
projenv = None
_projenv_error = err
# Precompile the src force-includes plus defines.h and force-include the
# result into C++ src compiles only; their preprocessed output is unchanged.
# Registration is gated host-side (pch_enabled()). Keep the closure, ccache
# values, probe flow, stamp flow, and env-knob spellings in sync with
# build_helpers/pch.py.
# Compiler failures that clear on their own must not latch the .failed marker
_TRANSIENT_ERRORS = ("No space left", "Cannot allocate", "Resource temporarily")
# Keep in sync with helpers.TRUTHY_ENV_STRINGS / FALSY_ENV_STRINGS
_TRUTHY = ("1", "true", "yes", "on", "enable")
_FALSY = ("", "0", "false", "no", "off", "disable")
_STRICT_RAW = os.environ.get("ESPHOME_PCH_STRICT")
_STRICT_VALUE = (_STRICT_RAW or "").strip().lower()
_STRICT = _STRICT_VALUE in _TRUTHY
if _STRICT_RAW is not None and _STRICT_VALUE not in _TRUTHY + _FALSY:
# A typo must not silently turn the gate into a no-op
raise RuntimeError(f"Unrecognized ESPHOME_PCH_STRICT={_STRICT_RAW!r}; use 1 or 0")
_INCLUDE_RE = re.compile(rb'^\s*#\s*include\s+["<]([^">]+)[">]', re.MULTILINE)
_CORE_HEADER = "esphome/core/defines.h"
def _raise_walk_error(err: OSError) -> None:
raise err
def _resolves_dir(path: Path) -> bool:
"""False when missing; other stat failures propagate."""
try:
return stat.S_ISDIR(path.stat().st_mode)
except (FileNotFoundError, NotADirectoryError):
return False
def _resolves(path: Path) -> bool:
"""False when missing; other stat failures propagate (identity unknown)."""
try:
return stat.S_ISREG(path.stat().st_mode)
except (FileNotFoundError, NotADirectoryError):
return False
def _include_closure(src_dir: Path, roots: list) -> dict:
"""Quoted-include closure: src-relative name -> contents (mirror of
build_helpers/pch.py)."""
seen = {}
stack = [(name, "") for name in roots]
while stack:
name, from_dir = stack.pop()
for candidate in (f"{from_dir}/{name}" if from_dir else name, name):
rel = posixpath.normpath(candidate)
if not rel.startswith("..") and _resolves(src_dir / rel):
break
else:
continue
if rel in seen:
continue
try:
data = (src_dir / rel).read_bytes()
except OSError as err:
# A marker would truncate the transitive walk; fail closed
print(f"ESPHome: could not read {rel} for the pch checksum: {err}")
raise
seen[rel] = data
parent = posixpath.dirname(rel)
stack.extend(
(inc.decode(errors="surrogateescape"), parent)
for inc in _INCLUDE_RE.findall(data)
)
return seen
def _shell_arg(element) -> str | None:
"""One compiler argv from one SCons element, matching the real spawn:
spaced elements pass whole, the rest get one shell unquote (skipped on
Windows, where shlex would eat path backslashes)."""
arg = str(element)
if " " in arg or os.name == "nt":
return arg.replace('\\"', '"')
if not arg.strip():
return arg
try:
tokens = shlex.split(arg)
except ValueError as err:
print(f"ESPHome: could not lex flag {arg!r} for the pch: {err}")
return None
if len(tokens) != 1:
# A flag the model cannot reproduce would diverge the .gch's flags
print(f"ESPHome: cannot model flag {arg!r} for the pch")
return None
return tokens[0]
def _compile_gch(cxx, flags, header: Path, gch: Path, proj_dir: Path):
"""Compile the .gch, then probe that the toolchain can load it back
(GCC 10 on macOS arm64 rejects its own per-process). Returns an error
string or None; OSError propagates as transient."""
result = subprocess.run( # noqa: PLW1510
[cxx, "-x", "c++-header", *flags, "-c", str(header), "-o", str(gch)],
cwd=proj_dir,
# C locale keeps diagnostics matchable by _TRANSIENT_ERRORS
env={**os.environ, "LC_ALL": "C"},
capture_output=True,
text=True,
)
if result.returncode < 0:
# Signal-killed (OOM, ^C): route to the transient no-marker path
raise OSError(f"compiler killed by signal {-result.returncode}")
if result.returncode != 0:
return result.stderr
return _probe_gch(cxx, flags, header, proj_dir)
def _probe_run(cxx, flags, extra, proj_dir: Path):
probe = subprocess.run( # noqa: PLW1510
[cxx, *flags, *extra, "-fsyntax-only", "-x", "c++", "-"],
cwd=proj_dir,
env={**os.environ, "LC_ALL": "C"},
input="",
capture_output=True,
text=True,
)
if probe.returncode < 0:
raise OSError(f"probe killed by signal {-probe.returncode}")
return probe
def _probe_gch(cxx, flags, header: Path, proj_dir: Path):
"""Load-check an existing .gch; error string or None. Rejection must
be a nonzero exit (keep in sync with pch_probe_args); a baseline run
without the pch keeps environmental failures from being blamed on it."""
# -MF is only legal alongside a dependency flag; pass it solely to
# redirect a depfile that -MD/-MMD in the flags would otherwise write
dep_redirect = (
["-MF", os.devnull]
if any(f in ("-MD", "-MMD", "-M", "-MM") for f in flags)
else []
)
probe = _probe_run(
cxx,
flags,
[*dep_redirect, "-Winvalid-pch", "-Werror=invalid-pch", "-include", str(header)],
proj_dir,
)
if probe.returncode == 0:
return None
baseline = _probe_run(cxx, flags, dep_redirect, proj_dir)
if baseline.returncode != 0:
# Deterministic and latchable; the transient filter at the caller
# keeps resource exhaustion from latching
return f"probe cannot run at all: {baseline.stderr.strip()[:200]}"
return f"toolchain cannot load the pch: {probe.stderr.strip()}"
def _read_stamp(path: Path) -> str:
"""A corrupt sidecar must read as stale, not kill the pch forever."""
try:
return path.read_text(encoding="utf-8").strip()
except (OSError, UnicodeDecodeError):
return ""
def _setup_pch() -> bool | None:
if projenv is None:
print(f"ESPHome: projenv unavailable ({_projenv_error}); skipping pch")
try:
from SCons.Script import COMMAND_LINE_TARGETS
except ImportError:
# No SCons is an anomaly under PlatformIO: the unknown state
# must not read as success (strict decides fatality at the gate)
return False
# Expected under -t nobuild (nothing compiles); a missing
# projenv on a real compile must not pass strict
return "nobuild" in [str(t) for t in COMMAND_LINE_TARGETS]
# Project root: SCons compiles run here, so the relative -include
# resolves; an absolute path would break cross-device ccache sharing.
proj_dir = Path(env.subst("$PROJECT_DIR")) # noqa: F821
src_dir = Path(env.subst("$PROJECT_SRC_DIR")) # noqa: F821
header = proj_dir / "esphome_pch.h"
gch = Path(f"{header}.gch")
sum_path = Path(f"{gch}.sum")
cxx = projenv.subst("$CXX") # noqa: F821
# The header holds the -include entries itself, so the .gch compile must
# not see them; consumers keep theirs, which the .gch then satisfies.
flags = []
include_headers = []
raw_args = [
_shell_arg(element)
for element in projenv.subst_list("$CXXFLAGS $CCFLAGS $_CCCOMCOM")[0] # noqa: F821
]
if any(arg is None for arg in raw_args):
print("ESPHome: skipping precompiled header: unmodelable flag")
return
flag_it = iter(raw_args)
for tok in flag_it:
if tok == "-include":
include_headers.append(next(flag_it, ""))
elif tok.startswith("-include") and not tok.startswith("-include-"):
include_headers.append(tok[len("-include") :])
else:
flags.append(tok)
if any(not name for name in include_headers):
print("ESPHome: build_src_flags has a trailing -include; skipping pch")
return
# Fold only relative names resolving under src/: consumers keep their
# own -include entries, so folding an unguarded user header would
# include it twice; unfolded ones stay consumer-only.
try:
folded = [
name
for name in include_headers
if not Path(name).is_absolute() and _resolves(src_dir / name)
]
except OSError as err:
print(f"ESPHome: skipping precompiled header: {err}")
return
if unfolded := [n for n in include_headers if n not in folded]:
print(f"ESPHome: not precompiling non-src force-includes: {unfolded}")
content = "".join(f'#include "{name}"\n' for name in (*folded, _CORE_HEADER))
digest = hashlib.sha256()
digest.update(content.encode(errors="surrogateescape"))
digest.update(cxx.encode(errors="surrogateescape"))
# Mirror CCACHE_BASEDIR: strip the per-device build path so identical
# configs produce identical .sum files and share cache entries
flags_id = " ".join(flags)
if basedir := os.environ.get("CCACHE_BASEDIR"):
flags_id = flags_id.replace(basedir, "")
digest.update(flags_id.encode(errors="surrogateescape"))
# GCC never validates a .gch against its source headers, and PlatformIO
# package paths carry no version, so a package bump must invalidate here
platform = env.PioPlatform() # noqa: F821
for package in sorted(platform.packages):
try:
version = platform.get_package_version(package)
except KeyError:
# An unresolved manifest must not hash as a constant
if platform.get_package(package) is not None:
print(f"ESPHome: skipping precompiled header: no version for {package}")
return
version = None # absent optional package
except Exception as err: # noqa: BLE001
# No trustworthy package identity: a stale .gch could survive
print(f"ESPHome: skipping precompiled header: {err}")
return
digest.update(f"{package}={version}".encode())
digest.update(b"\0")
try:
closure = _include_closure(src_dir, [*folded, _CORE_HEADER])
except OSError as err:
print(f"ESPHome: skipping precompiled header: {err}")
return
for rel in sorted(closure):
digest.update(rel.encode(errors="surrogateescape"))
digest.update(closure[rel])
digest.update(b"\0")
# Project-local -I dirs (e.g. rp2's lwip_override) hold generated
# headers the src closure cannot see; hash them too
prev = ""
try:
for tok in flags:
inc = tok[2:] if tok.startswith("-I") and len(tok) > 2 else ""
if prev == "-I":
inc = tok
prev = tok
if not inc:
continue
inc_dir = Path(inc)
if not (
_resolves_dir(inc_dir)
and inc_dir.is_relative_to(proj_dir)
and not inc_dir.is_relative_to(src_dir)
# Library trees never enter the prefix closure; walking them
# would read every library file each build
and not inc_dir.is_relative_to(proj_dir / ".piolibdeps")
and not inc_dir.is_relative_to(proj_dir / ".pioenvs")
):
continue
local_headers = []
# os.walk with onerror: rglob would swallow unlistable subtrees
for root, _dirs, files in os.walk(inc_dir, onerror=_raise_walk_error):
local_headers.extend(
Path(root) / f
for f in files
if f.endswith((".h", ".hpp", ".hh", ".inc"))
)
for local in sorted(local_headers):
digest.update(str(local.relative_to(proj_dir)).encode())
digest.update(local.read_bytes())
digest.update(b"\0")
except OSError as err:
print(f"ESPHome: skipping precompiled header: {err}")
return
checksum = digest.hexdigest()
# The ccache .sum sidecar doubles as the freshness stamp
fresh = (
header.is_file()
and gch.is_file()
and sum_path.is_file()
and (_read_stamp(sum_path) == checksum)
)
if fresh and _STRICT:
# Rejection is per-process: strict re-proves a cached .gch loads
# (mirrors the pch_strict() re-probe in build_helpers/pch.py)
error = _probe_gch(cxx, flags, header, proj_dir)
if error is not None:
print(f"ESPHome: {error}")
gch.unlink(missing_ok=True)
sum_path.unlink(missing_ok=True)
return
if not fresh:
failed_marker = Path(f"{gch}.failed")
if _read_stamp(failed_marker) == checksum:
print(
"ESPHome: skipping precompiled header (previous attempt "
f"failed); delete {failed_marker.name} to retry"
)
return
header.write_text(content, encoding="utf-8")
try:
error = _compile_gch(cxx, flags, header, gch, proj_dir)
except OSError as err:
# Transient spawn/IO failure: no marker, retry next build
print(f"ESPHome: precompiled header compile did not run: {err}")
gch.unlink(missing_ok=True)
sum_path.unlink(missing_ok=True)
return
if error is not None:
print("ESPHome: precompiled header failed; compiling without it")
print(error)
gch.unlink(missing_ok=True)
sum_path.unlink(missing_ok=True)
if any(m in error for m in _TRANSIENT_ERRORS):
# Resource exhaustion clears on its own; retry next build
return
# Skip retries until a flag/header/platform change alters the checksum
failed_marker.write_text(checksum + "\n", encoding="utf-8")
return
failed_marker.unlink(missing_ok=True)
sum_path.write_text(checksum + "\n", encoding="utf-8")
# Computed first so the flags and env land together: a raise between
# them would leave a pch-consuming build without its ccache settings.
# projenv["ENV"] aliases os.environ, so these reach all TUs; only
# time_macros affects non-pch TUs. User values win.
ccache_updates = {
key: value
for key, value in (
("CCACHE_SLOPPINESS", "pch_defines,time_macros"),
("CCACHE_PCH_EXTSUM", "true"),
)
if key not in os.environ
}
sloppiness = os.environ.get("CCACHE_SLOPPINESS")
if sloppiness is not None:
tokens = {tok.strip() for tok in sloppiness.split(",")}
missing = [t for t in ("pch_defines", "time_macros") if t not in tokens]
if missing:
# Without these ccache declines every pch-consuming compile
ccache_updates["CCACHE_SLOPPINESS"] = ",".join((sloppiness, *missing))
print(f"ESPHome: adding {','.join(missing)} to CCACHE_SLOPPINESS for the pch")
extsum = os.environ.get("CCACHE_PCH_EXTSUM")
if extsum is not None and extsum.strip().lower() not in ("1", "true", "yes", "on"):
# ccache then hashes the non-reproducible .gch bytes: permanent misses
print(f"ESPHome: CCACHE_PCH_EXTSUM={extsum} disables pch caching")
# Prepended: GCC only uses a .gch while no other tokens precede it.
# The relative name also reaches "pio run -t idedata" output.
# -Wno-error: the per-process probe can pass while a later cc1plus
# rejects the .gch; that must stay a warning under user -Werror.
projenv.Prepend( # noqa: F821
CXXFLAGS=[
"-Winvalid-pch",
# Strict inverts: a per-process consumer rejection reds the build
"-Werror=invalid-pch" if _STRICT else "-Wno-error=invalid-pch",
"-include",
header.name,
]
)
projenv["ENV"].update(ccache_updates) # noqa: F821
print("ESPHome: Compiling with precompiled header")
return True
try:
_used = _setup_pch()
except Exception: # noqa: BLE001 -- a speedup must never break the build
if _STRICT:
raise
print("ESPHome: pch internal error; compiling without it")
traceback.print_exc()
else:
if _STRICT and not _used:
raise RuntimeError("ESPHOME_PCH_STRICT: precompiled header was not used")
+3 -4
View File
@@ -32,13 +32,12 @@ import time
from typing import Any, NamedTuple
from esphome.framework_helpers import (
BATCH_EXTRACT_WORKERS,
content_length,
discard_partial_download,
failure_reason,
resume_fetch_job,
run_batch_downloads,
warn_batch_failures,
warn_prefetch_failures,
)
from esphome.helpers import get_bool_env, get_usable_cpu_count, rmtree
@@ -682,7 +681,7 @@ def _preinstall(
would hang, not fail). Waves skip dependencies; the installed
manifests feed the next wave. Any failure falls back to pio run.
"""
workers = min(get_usable_cpu_count(), len(entries), BATCH_EXTRACT_WORKERS)
workers = min(get_usable_cpu_count(), len(entries))
# One manager per worker (_install mutates instance state); built
# serially because construction rewires the shared manager logger
managers: SimpleQueue = SimpleQueue()
@@ -877,7 +876,7 @@ def _prefetch(build_dir: Path, env: str) -> None:
)
# PlatformIO retries failed packages itself, without resume
failures = run_batch_downloads("Downloading PlatformIO packages", jobs)
warn_batch_failures(failures)
warn_prefetch_failures(failures)
failed_names = {name for name, _ in failures}
elif not groups and not unresolved:
# Record the no-work run so the parent skips the next spawn.
+6 -123
View File
@@ -14,16 +14,12 @@ from typing import NamedTuple
from esphome.core import EsphomeError
from esphome.framework_helpers import (
BATCH_EXTRACT_WORKERS,
archive_extract_all,
download_from_mirrors,
download_with_resume,
is_expected_fetch_error,
rmdir,
run_batch_downloads,
warn_batch_failures,
)
from esphome.helpers import get_usable_cpu_count
from esphome.net_retry import fetch_with_retry, http_request
_LOGGER = logging.getLogger(__name__)
@@ -164,14 +160,6 @@ def _check_layout(name: str, dest: Path, expect: Collection[str]) -> None:
)
# (name, version, dest, mirrors, expect) as accepted by install_packages
PackageSpec = tuple[str, str, Path, list[str], Collection[str]]
def _archive_path(downloads_dir: Path, name: str, version: str) -> Path:
return downloads_dir / f"{name}-{version}"
class _PendingArchive(NamedTuple):
name: str
version: str
@@ -186,26 +174,6 @@ def _already_installed(dest: Path) -> bool:
return (dest / ".esphome_extracted").is_file()
def _batched_download_progress(
name: str, version: str, size: int | None, extract_progress: Callable[[float], None]
) -> Callable[[int], None]:
"""Download tracker for a batched install: no private bar, no bytes (the
shared bar must never run backwards); the zero tick keeps cancellation
observable and a real refetch is announced once."""
announced = False
def progress(done: int) -> None:
nonlocal announced
# size-less registry entries still announce: streaming starts at
# done=0, while a verify no-op credits the full file in one tick
if not announced and done < (size or 1):
_LOGGER.info("Re-downloading %s %s ...", name, version)
announced = True
extract_progress(0.0)
return progress
def prefetch_packages(
packages: list[tuple[str, str, Path, list[str]]], downloads_dir: Path
) -> None:
@@ -274,7 +242,7 @@ def prefetch_packages(
[(entry.name, entry.size, partial(_fetch, entry)) for entry in pending],
)
for name, err in failures:
if is_expected_fetch_error(err):
if isinstance(err, (EsphomeError, OSError)):
# Expected download failures: install_package retries this one
# itself, with a visible bar
_LOGGER.debug("Prefetch of %s failed: %s", name, err)
@@ -291,7 +259,6 @@ def install_package(
mirrors: list[str],
downloads_dir: Path,
expect: Collection[str],
extract_progress: Callable[[float], None] | None = None,
) -> None:
"""Download, verify, and extract one package if not already installed.
@@ -299,9 +266,6 @@ def install_package(
publishes; a mirror override (URL templates with ``{VERSION}``/``{SYSTEM}``
substitution) is trusted as configured. ``downloads_dir`` holds the
archive between runs so an interrupted download resumes.
``extract_progress`` receives extraction fractions in [0, 1] instead of
the private per-file bars (see ``install_packages``).
"""
if not expect:
# Layout validation before marker.touch() is the only guard against
@@ -324,15 +288,8 @@ def install_package(
rmdir(dest, msg=f"Clean up incomplete {name} install")
# Persistent location so an interrupted download resumes across runs.
downloads_dir.mkdir(parents=True, exist_ok=True)
archive = _archive_path(downloads_dir, name, version)
# Batch header names each package; keep INFO when an archive
# unexpectedly needs a real download (the shared bar won't move)
log = (
_LOGGER.debug
if extract_progress is not None and archive.is_file()
else _LOGGER.info
)
log("Downloading %s %s ...", name, version)
archive = downloads_dir / f"{name}-{version}"
_LOGGER.info("Downloading %s %s ...", name, version)
if mirrors:
_LOGGER.warning(
"Downloading %s from a mirror override; checksum verification "
@@ -344,85 +301,11 @@ def install_package(
)
else:
url, sha256, size = registry_download(name, version)
download_with_resume(
url,
archive,
sha256=sha256,
size=size,
progress=None
if extract_progress is None
else _batched_download_progress(name, version, size, extract_progress),
)
log("Extracting %s ...", name)
archive_extract_all(
archive, dest, progress_header="Extracting", progress=extract_progress
)
download_with_resume(url, archive, sha256=sha256, size=size)
_LOGGER.info("Extracting %s ...", name)
archive_extract_all(archive, dest, progress_header="Extracting")
# Validate the layout before recording success, so an unexpected
# package is never cached as a working install.
_check_layout(name, dest, expect)
marker.touch()
archive.unlink(missing_ok=True)
def install_packages(specs: Collection[PackageSpec], downloads_dir: Path) -> None:
"""Install several packages; prefetched archives extract in parallel under
one shared bar, the rest take the sequential ``install_package`` path.
The first failure is re-raised."""
pending: list[tuple[PackageSpec, int]] = []
rest: list[PackageSpec] = []
seen: set[str] = set()
for spec in specs:
name, version, dest, mirrors, _expect = spec
archive = _archive_path(downloads_dir, name, version)
# Duplicate entries share one archive and would race each other
# between two workers; mirror prefetch_packages' dedupe
if _already_installed(dest) or mirrors or archive.name in seen:
rest.append(spec)
continue
try:
# An archive at its final name already passed sha256/size
# verification
size = archive.stat().st_size
except FileNotFoundError:
rest.append(spec)
continue
seen.add(archive.name)
pending.append((spec, size))
if len(pending) < 2:
for name, version, dest, mirrors, expect in specs:
install_package(name, version, dest, mirrors, downloads_dir, expect=expect)
return
workers = min(get_usable_cpu_count(), len(pending), BATCH_EXTRACT_WORKERS)
_LOGGER.info(
"Extracting %d package archive(s) with %d worker(s): %s",
len(pending),
workers,
", ".join(spec[0] for spec, _ in pending),
)
def _install(spec: PackageSpec, size: int, tracker: Callable[[int], None]) -> None:
name, version, dest, mirrors, expect = spec
install_package(
name,
version,
dest,
mirrors,
downloads_dir,
expect=expect,
extract_progress=lambda frac: tracker(int(frac * size)),
)
failures = run_batch_downloads(
"Extracting packages",
[(spec[0], size, partial(_install, spec, size)) for spec, size in pending],
max_workers=workers,
)
if failures:
# Warn on the first failure too: the raised exception's message may
# not name which package failed
warn_batch_failures(failures, "Could not install %s: %s")
raise failures[0][1]
# Sequential remainder after the batch, so a duplicate spec cannot
# unlink the archive its batched twin was sized from
for name, version, dest, mirrors, expect in rest:
install_package(name, version, dest, mirrors, downloads_dir, expect=expect)
-9
View File
@@ -289,15 +289,6 @@ def copy_ccache_script() -> None:
)
def copy_pch_script() -> None:
"""Copy the shared precompiled-header SCons post-script into the build
dir; platform components pair it with ``post:pch.py`` in extra_scripts."""
copy_file_if_changed(
Path(__file__).parent / "pch.py.script",
CORE.relative_build_path("pch.py"),
)
def default_libdeps_dir() -> str:
"""The PLATFORMIO_LIBDEPS_DIR value a pio run defaults to; the package
prefetch must resolve installed libraries against the same dir."""
+4 -15
View File
@@ -7,7 +7,6 @@ import re
import time
from esphome import loader
from esphome.build_helpers.pch import PCH_ARTIFACT_NAMES, PCH_PREFIX_HEADER
from esphome.compiled_config import save_compiled_config
from esphome.config import iter_component_configs, iter_components
from esphome.const import (
@@ -247,7 +246,6 @@ def copy_src_tree():
Path(
"esphome/core/ring_buffer.h"
), # moved to components/ring_buffer/, removed in 2026.11.0
Path(PCH_PREFIX_HEADER), # build machinery, not user API
}
include_l = []
for target, _ in source_files_l:
@@ -611,20 +609,11 @@ def clean_build(clear_pio_cache: bool = True, *, full: bool = False):
if idf_path.is_dir():
_LOGGER.info("Deleting %s", idf_path)
rmtree(idf_path)
# The PlatformIO pch artifacts live at the project root so the
# relative -include resolves; a partial clean must drop them too
for name in PCH_ARTIFACT_NAMES:
CORE.relative_build_path(name).unlink(missing_ok=True)
# The idedata caches are derived from the build but live under the data
# dir, not the build path, so they must be removed separately in both
# modes. Globbed (name.json plus name.<backend>.json) so a future
# backend suffix cannot silently drift out of clean-all.
idedata_dir = CORE.relative_internal_path("idedata")
for idedata_cache in (
*idedata_dir.glob(f"{CORE.name}.json"),
*idedata_dir.glob(f"{CORE.name}.*.json"),
):
# The idedata cache is derived from the build but lives under the data dir,
# not the build path, so it must be removed separately in both modes.
idedata_cache = CORE.relative_internal_path("idedata", f"{CORE.name}.json")
if idedata_cache.is_file():
_LOGGER.info("Deleting %s", idedata_cache)
idedata_cache.unlink()
+26 -132
View File
@@ -50,7 +50,6 @@ from __future__ import annotations
import argparse
from collections import Counter
from collections.abc import Callable
from enum import StrEnum
from functools import cache
import json
@@ -504,51 +503,25 @@ ESP32_PLATFORMIO_TRIGGER_PATH_PREFIXES = ("esphome/platformio/",)
# - esphome/build_gen/platformio.py -- the PlatformIO build generator
# - script/test_build_components.py -- the harness the job invokes
# - .github/workflows/ci.yml -- the job's own definition
# Shared by every toolchain smoke-test job: the harness it invokes and the
# workflow that defines it
_SMOKE_HARNESS_TRIGGER_FILES = frozenset(
ESP32_PLATFORMIO_TRIGGER_FILES = frozenset(
{
"esphome/build_gen/platformio.py",
"script/test_build_components.py",
".github/workflows/ci.yml",
}
)
ESP32_PLATFORMIO_TRIGGER_FILES = _SMOKE_HARNESS_TRIGGER_FILES | {
"esphome/build_gen/platformio.py",
# The pch machinery the strict smoke job polices, and the modules it
# imports; the rest of build_helpers/ does not affect PlatformIO builds
"esphome/build_helpers/pch.py",
"esphome/build_helpers/ccache.py",
"esphome/build_helpers/idedata.py",
}
def _path_or_file_trigger(
files: list[str],
trigger_files: frozenset[str],
trigger_prefixes: tuple[str, ...],
) -> bool:
"""Whether any changed file matches the given infrastructure triggers."""
return any(
file in trigger_files or file.startswith(trigger_prefixes) for file in files
)
@cache
def _cached_components_closure(files: tuple[str, ...]) -> frozenset[str]:
"""Dependency closure of the changed components, from the changed files.
The walk is expensive and every toolchain smoke-test job asks for the
same file list, so compute it once per run."""
component_files = [f for f in files if filter_component_and_test_files(f)]
return frozenset(get_components_with_dependencies(component_files, True))
def _esp32_platformio_path_or_file_trigger(files: list[str]) -> bool:
"""Whether any changed file is a PlatformIO infrastructure / harness trigger."""
return _path_or_file_trigger(
files, ESP32_PLATFORMIO_TRIGGER_FILES, ESP32_PLATFORMIO_TRIGGER_PATH_PREFIXES
)
for file in files:
if file in ESP32_PLATFORMIO_TRIGGER_FILES:
return True
if any(
file.startswith(prefix) for prefix in ESP32_PLATFORMIO_TRIGGER_PATH_PREFIXES
):
return True
return False
# Native-build infra: changes under esphome/espidf/, the shared
@@ -558,26 +531,26 @@ def _esp32_platformio_path_or_file_trigger(files: list[str]) -> bool:
# compile. When they change we fold the `esp32` component into the matrix so
# the default native-IDF build path is still compiled on an infra-only PR.
ESP_IDF_INFRA_TRIGGER_PATH_PREFIXES = ("esphome/espidf/", "esphome/build_helpers/")
# Shared library-conversion modules every native build imports; a new shared
# module added to one native trigger set must not silently skip the other's
# smoke test, so both sets union this one.
_NATIVE_SHARED_TRIGGER_FILES = frozenset(
ESP_IDF_INFRA_TRIGGER_FILES = frozenset(
{
"esphome/build_gen/espidf.py",
"esphome/framework_helpers.py",
"esphome/platformio/library.py",
"esphome/platformio/extra_script.py",
}
)
ESP_IDF_INFRA_TRIGGER_FILES = _NATIVE_SHARED_TRIGGER_FILES | {
"esphome/build_gen/espidf.py",
}
def _esp_idf_infra_changed(files: list[str]) -> bool:
"""Whether any changed file is ESP-IDF build/runner infrastructure."""
return _path_or_file_trigger(
files, ESP_IDF_INFRA_TRIGGER_FILES, ESP_IDF_INFRA_TRIGGER_PATH_PREFIXES
)
for file in files:
if file in ESP_IDF_INFRA_TRIGGER_FILES:
return True
if any(
file.startswith(prefix) for prefix in ESP_IDF_INFRA_TRIGGER_PATH_PREFIXES
):
return True
return False
def esp32_platformio_components_to_test(branch: str | None = None) -> list[str]:
@@ -615,23 +588,15 @@ def esp32_platformio_components_to_test(branch: str | None = None) -> list[str]:
Returns:
Sorted list of component names to compile.
"""
return _toolchain_components_to_test(
branch, ESP32_PLATFORMIO_TEST_COMPONENTS, _esp32_platformio_path_or_file_trigger
)
def _toolchain_components_to_test(
branch: str | None,
test_set: frozenset[str],
infra_trigger: Callable[[list[str]], bool],
) -> list[str]:
"""The shared narrowing rule for the per-toolchain smoke-test jobs."""
files = changed_files(branch)
if core_changed(files) or infra_trigger(files):
return sorted(test_set)
if core_changed(files) or _esp32_platformio_path_or_file_trigger(files):
return sorted(ESP32_PLATFORMIO_TEST_COMPONENTS)
return sorted(test_set & _cached_components_closure(tuple(files)))
component_files = [f for f in files if filter_component_and_test_files(f)]
changed = get_components_with_dependencies(component_files, True)
return sorted(ESP32_PLATFORMIO_TEST_COMPONENTS & set(changed))
def should_run_esp32_platformio(branch: str | None = None) -> bool:
@@ -652,71 +617,6 @@ def should_run_esp32_platformio(branch: str | None = None) -> bool:
return bool(esp32_platformio_components_to_test(branch))
# The `--toolchain arduino` smoke-test set: covers the core, the bundled and
# converted registry libraries, and the waveform path.
ESP8266_NATIVE_TEST_COMPONENTS = frozenset(
{
"esp8266",
"api",
"web_server",
"captive_portal",
"mqtt",
"esp8266_pwm",
"neopixelbus",
"bme280_i2c",
"uart",
}
)
# Infrastructure whose changes always trigger the native ESP8266 compile
# test. esphome/build_helpers/ holds the idedata and size-summary helpers
# the backend shares with the native ESP-IDF build.
ESP8266_NATIVE_TRIGGER_PATH_PREFIXES = (
"esphome/arduino8266/",
"esphome/arduino/",
"esphome/build_helpers/",
)
ESP8266_NATIVE_TRIGGER_FILES = (
_NATIVE_SHARED_TRIGGER_FILES
| _SMOKE_HARNESS_TRIGGER_FILES
| {
"esphome/build_gen/arduino8266.py",
"esphome/build_gen/build_tool.py",
"esphome/components/esp8266/build_surgery.py",
"esphome/components/esp8266/boards.py",
"esphome/platformio/registry.py",
# esp8266/__init__.py imports copy_ccache_script from it
"esphome/platformio/toolchain.py",
".github/actions/cache-arduino8266/action.yml",
}
)
def _esp8266_native_path_or_file_trigger(files: list[str]) -> bool:
"""Whether any changed file is native-ESP8266 infrastructure / harness."""
# base_python_changed covers the top-level esphome/*.py modules the
# native backend imports directly (framework_helpers, helpers, writer,
# __main__); without it a change there would silently skip this job.
# base_python_changed is deliberately broad (any top-level esphome/*.py)
# as belt-and-braces while the backend is new; narrow it to the modules
# the backend imports once the toolchain has soaked a few releases
return base_python_changed(files) or _path_or_file_trigger(
files, ESP8266_NATIVE_TRIGGER_FILES, ESP8266_NATIVE_TRIGGER_PATH_PREFIXES
)
def esp8266_native_components_to_test(branch: str | None = None) -> list[str]:
"""Subset of ``ESP8266_NATIVE_TEST_COMPONENTS`` the job needs to compile.
Same narrowing logic as ``esp32_platformio_components_to_test``: the full
list on core or infrastructure changes, otherwise the intersection with
the changed-component dependency closure (empty list skips the job).
"""
return _toolchain_components_to_test(
branch, ESP8266_NATIVE_TEST_COMPONENTS, _esp8266_native_path_or_file_trigger
)
def determine_cpp_unit_tests(
branch: str | None = None,
) -> tuple[bool, list[str]]:
@@ -1315,8 +1215,6 @@ def main() -> None:
run_device_builder = True
esp32_platformio_components = sorted(ESP32_PLATFORMIO_TEST_COMPONENTS)
run_esp32_platformio = True
esp8266_native_components = sorted(ESP8266_NATIVE_TEST_COMPONENTS)
run_esp8266_native = True
else:
integration_run_all, integration_test_files = determine_integration_tests(
args.branch
@@ -1328,8 +1226,6 @@ def main() -> None:
run_device_builder = should_run_device_builder(args.branch)
esp32_platformio_components = esp32_platformio_components_to_test(args.branch)
run_esp32_platformio = bool(esp32_platformio_components)
esp8266_native_components = esp8266_native_components_to_test(args.branch)
run_esp8266_native = bool(esp8266_native_components)
run_integration, integration_test_buckets = _compute_integration_test_buckets(
integration_run_all, integration_test_files
)
@@ -1524,8 +1420,6 @@ def main() -> None:
"device_builder": run_device_builder,
"esp32_platformio": run_esp32_platformio,
"esp32_platformio_components": ",".join(esp32_platformio_components),
"esp8266_native": run_esp8266_native,
"esp8266_native_components": ",".join(esp8266_native_components),
"changed_components": changed_components,
"changed_components_with_tests": changed_components_with_tests,
"directly_changed_components_with_tests": list(directly_changed_with_tests),
+20
View File
@@ -0,0 +1,20 @@
#!/bin/sh
# Prepare the dev environment for a new checkout or worktree.
#
# Installed into the git hooks directory by script/setup. Deliberately tiny and
# self-contained: it stays valid on branches where script/setup does not exist,
# and simply does nothing there.
# $3 is 1 for a branch checkout, 0 for a file checkout.
[ "$3" = "1" ] || exit 0
top=$(git rev-parse --show-toplevel 2>/dev/null) || exit 0
# This also runs on ordinary branch switches, where there is nothing to do.
[ -x "$top/venv/bin/python" ] && exit 0
[ -x "$top/script/setup" ] || exit 0
# Clear VIRTUAL_ENV so a checkout made from a shell with an environment already
# activated still gets its own, rather than having the active one repointed at
# this working tree.
exec env -u VIRTUAL_ENV "$top/script/setup"
+43 -16
View File
@@ -7,13 +7,19 @@ cd "$(dirname "$0")/.."
if [ -n "$VIRTUAL_ENV" ]; then
# A virtual environment is already active (e.g. the devcontainer's pre-provisioned
# esphome-venv). Install into it rather than creating a ./venv in the workspace.
created_venv=false
venv_state=active
elif [ -x venv/bin/python ]; then
# Reuse the environment from an earlier run, so this script can be run again
# at any time to pick up dependency changes.
venv_state=reused
source venv/bin/activate
else
created_venv=true
venv_state=created
# --clear replaces a partial environment left behind by an interrupted run.
if [ -x "$(command -v uv)" ]; then
uv venv --seed venv
uv venv --clear --seed venv
else
python3 -m venv venv
python3 -m venv --clear venv
fi
source venv/bin/activate
fi
@@ -25,20 +31,41 @@ fi
uv pip install setuptools wheel
uv pip install -e ".[dev,test]" --config-settings editable_mode=compat
# --overwrite replaces any hook already in place. Without it, prek finds a
# previously installed pre-commit hook, moves it aside to
# .git/hooks/pre-commit.legacy and keeps calling it, so every commit would
# run both tools.
prek install --overwrite
# A worktree shares one git hooks directory with the main checkout it was
# created from, so hooks are installed from the main checkout only. Installing
# from a worktree would point the shared hook at that worktree's virtual
# environment, breaking it for everyone once the worktree is removed.
git_dir="$(git rev-parse --absolute-git-dir 2>/dev/null || true)"
common_dir="$(git rev-parse --path-format=absolute --git-common-dir 2>/dev/null || true)"
if [ -n "$common_dir" ] && [ "$git_dir" = "$common_dir" ]; then
# --overwrite replaces any hook already in place. Without it, prek finds a
# previously installed pre-commit hook, moves it aside to
# .git/hooks/pre-commit.legacy and keeps calling it, so every commit would
# run both tools.
prek install --overwrite
# Prepares the virtual environment for new checkouts and worktrees. Installed
# once here, it covers every worktree created from this checkout.
if [ -d "$common_dir/hooks" ]; then
cp script/git-hooks/post-checkout "$common_dir/hooks/post-checkout"
chmod +x "$common_dir/hooks/post-checkout"
fi
fi
mkdir -p .temp
echo
echo
if [ "$created_venv" = true ]; then
echo "Virtual environment created at ./venv. Run 'source venv/bin/activate' to use it."
else
echo "Dependencies installed into the active virtual environment:"
echo " $VIRTUAL_ENV"
echo "It is already active in this shell, so no 'source venv/bin/activate' is needed."
fi
case "$venv_state" in
created)
echo "Virtual environment created at ./venv. Run 'source venv/bin/activate' to use it."
;;
reused)
echo "Dependencies updated in the existing ./venv. Run 'source venv/bin/activate' to use it."
;;
active)
echo "Dependencies installed into the active virtual environment:"
echo " $VIRTUAL_ENV"
echo "It is already active in this shell, so no 'source venv/bin/activate' is needed."
;;
esac
+8 -50
View File
@@ -1027,7 +1027,6 @@ def test_components(
isolated_components: set[str] | None = None,
base_only: bool = False,
toolchain: str | None = None,
fail_on_no_tests: bool = False,
) -> int:
"""Test components with optional intelligent grouping.
@@ -1062,34 +1061,20 @@ def test_components(
# toolchain build.
include_validate = esphome_command != "compile"
# Find all component tests; remember which components each pattern
# (wildcards included) matched, for the deferred no-tests accounting
# Find all component tests
all_tests = {}
pattern_components: dict[str, set[str]] = {}
for pattern in component_patterns:
# Skip empty patterns (happens when components list is empty string)
if not pattern:
continue
found = find_component_tests(
tests_dir, pattern, base_only, include_validate=include_validate
all_tests.update(
find_component_tests(
tests_dir, pattern, base_only, include_validate=include_validate
)
)
pattern_components[pattern] = set(found)
all_tests.update(found)
# The flag's contract is "no test matched fails": a fully blank pattern
# list would otherwise slide into the reference-baseline fallback and
# exit green while building nothing a caller asked for
if fail_on_no_tests and not any(component_patterns):
print("No components requested (blank component list)")
return 1
if fail_on_no_tests and not all_tests:
# Nothing matched at all: fail before the synthetic baseline build,
# which would spend a compile reporting success on nothing. Partial
# matches defer to the per-pattern accounting after the summary.
print(f"No components found matching: {component_patterns}")
return 1
# If no components found, build a reference configuration for baseline comparison
# Create a synthetic "empty" component test that will build just the base config
if not all_tests:
print(f"No components found matching: {component_patterns}")
print(
@@ -1193,26 +1178,6 @@ def test_components(
toolchain=toolchain,
)
silent: list[str] = []
if fail_on_no_tests:
# A green run that built nothing for a requested pattern (renamed
# fixture, missing base file, version-suffix mismatch, a wildcard
# matching no component) must not pass CI. Per pattern: an
# all-or-nothing check would let one silent pattern hide behind the
# others. Opt-in: some legs (the esp32-ard smoke subset)
# legitimately match nothing. Failing is deferred past the summary
# so a real failure's reproduce commands still print.
built = {c for r in test_results for c in r.components}
# A pattern is silent when it matched no fixture, or when none of
# its matched components produced a build (wildcards included)
silent = [
p
for p in component_patterns
if p and not (pattern_components.get(p, set()) & built)
]
if silent:
print(f"No tests ran for requested pattern(s): {', '.join(silent)}")
# Separate results into passed and failed
passed_results = [r for r in test_results if r.success]
failed_results = [r for r in test_results if not r.success]
@@ -1244,7 +1209,7 @@ def test_components(
if os.environ.get("GITHUB_STEP_SUMMARY"):
write_github_summary(test_results, toolchain=toolchain)
if failed_results or silent:
if failed_results:
return 1
return 0
@@ -1299,12 +1264,6 @@ def main() -> int:
"--toolchain",
help="Select toolchain for compiling.",
)
parser.add_argument(
"--fail-on-no-tests",
action="store_true",
help="Exit non-zero when no test matched (for CI legs whose "
"components must all have fixtures)",
)
args = parser.parse_args()
@@ -1323,7 +1282,6 @@ def main() -> int:
continue_on_fail=args.continue_on_fail,
enable_grouping=not args.no_grouping,
isolated_components=isolated_components,
fail_on_no_tests=args.fail_on_no_tests,
base_only=args.base_only,
toolchain=args.toolchain,
)
-99
View File
@@ -78,17 +78,6 @@ def mock_esp32_platformio_components_to_test() -> Generator[Mock, None, None]:
yield mock
@pytest.fixture
def mock_esp8266_native_components_to_test() -> Generator[Mock, None, None]:
"""Mock esp8266_native_components_to_test from determine_jobs.
main() drives both the ``esp8266_native`` boolean output and the
``esp8266_native_components`` CSV from this one function.
"""
with patch.object(determine_jobs, "esp8266_native_components_to_test") as mock:
yield mock
@pytest.fixture
def mock_determine_cpp_unit_tests() -> Generator[Mock, None, None]:
"""Mock determine_cpp_unit_tests from helpers."""
@@ -117,7 +106,6 @@ def clear_determine_jobs_caches() -> None:
"""Clear all cached functions before each test."""
determine_jobs._is_clang_tidy_full_scan.cache_clear()
determine_jobs._component_has_tests.cache_clear()
determine_jobs._cached_components_closure.cache_clear()
def test_main_all_tests_should_run(
@@ -128,7 +116,6 @@ def test_main_all_tests_should_run(
mock_should_run_import_time: Mock,
mock_should_run_device_builder: Mock,
mock_esp32_platformio_components_to_test: Mock,
mock_esp8266_native_components_to_test: Mock,
mock_changed_files: Mock,
mock_determine_cpp_unit_tests: Mock,
capsys: pytest.CaptureFixture[str],
@@ -145,7 +132,6 @@ def test_main_all_tests_should_run(
mock_should_run_import_time.return_value = True
mock_should_run_device_builder.return_value = True
mock_esp32_platformio_components_to_test.return_value = ["api", "esp32"]
mock_esp8266_native_components_to_test.return_value = ["api", "logger"]
mock_determine_cpp_unit_tests.return_value = (False, ["wifi", "api", "sensor"])
# Mock changed_files to return non-component files (to avoid memory impact)
@@ -229,8 +215,6 @@ def test_main_all_tests_should_run(
assert output["device_builder"] is True
assert output["esp32_platformio"] is True
assert output["esp32_platformio_components"] == "api,esp32"
assert output["esp8266_native"] is True
assert output["esp8266_native_components"] == "api,logger"
assert output["changed_components"] == ["wifi", "api", "sensor"]
# changed_components_with_tests will only include components that actually have test files
assert "changed_components_with_tests" in output
@@ -267,7 +251,6 @@ def test_main_no_tests_should_run(
mock_should_run_import_time: Mock,
mock_should_run_device_builder: Mock,
mock_esp32_platformio_components_to_test: Mock,
mock_esp8266_native_components_to_test: Mock,
mock_changed_files: Mock,
mock_determine_cpp_unit_tests: Mock,
capsys: pytest.CaptureFixture[str],
@@ -284,7 +267,6 @@ def test_main_no_tests_should_run(
mock_should_run_import_time.return_value = False
mock_should_run_device_builder.return_value = False
mock_esp32_platformio_components_to_test.return_value = []
mock_esp8266_native_components_to_test.return_value = []
mock_determine_cpp_unit_tests.return_value = (False, [])
# Mock changed_files to return no component files
@@ -327,8 +309,6 @@ def test_main_no_tests_should_run(
assert output["device_builder"] is False
assert output["esp32_platformio"] is False
assert output["esp32_platformio_components"] == ""
assert output["esp8266_native"] is False
assert output["esp8266_native_components"] == ""
assert output["changed_components"] == []
assert output["changed_components_with_tests"] == []
assert output["component_test_count"] == 0
@@ -1003,10 +983,6 @@ _ESP32_PLATFORMIO_FULL_LIST_FILES = [
# PlatformIO subsystem (path-prefix trigger) + build generator
["esphome/platformio/runner.py"],
["esphome/platformio/toolchain.py"],
# The pch modules are standalone triggers, not the whole build_helpers/
["esphome/build_helpers/pch.py"],
["esphome/build_helpers/ccache.py"],
["esphome/build_helpers/idedata.py"],
["esphome/build_gen/platformio.py"],
# Workflow / harness files
["script/test_build_components.py"],
@@ -1065,9 +1041,6 @@ def test_esp32_platformio_components_to_test_returns_full_list_on_infrastructure
# Non-PlatformIO files in esphome/build_gen/ do NOT trigger the
# full list -- only esphome/build_gen/platformio.py is a trigger.
(["esphome/build_gen/espidf.py"], [], []),
# build_helpers modules the pch does not import are not triggers.
(["esphome/build_helpers/size_summary.py"], [], []),
(["esphome/build_helpers/ninja.py"], [], []),
# Docs / unrelated files -> empty.
(["README.md"], [], []),
([], [], []),
@@ -3166,75 +3139,3 @@ def test_memory_impact_elf_layouts_are_found(tmp_path: Path) -> None:
elf.write_text("")
assert find_elf_path(build_path) == elf, f"{platform} ELF not found"
@pytest.mark.parametrize(
"changed",
[
"esphome/arduino8266/framework.py",
"esphome/build_gen/arduino8266.py",
"esphome/components/esp8266/build_surgery.py",
# Shared modules the native build depends on
"esphome/build_helpers/idedata.py",
"esphome/platformio/library.py",
# Top-level esphome/*.py modules the backend imports directly
"esphome/framework_helpers.py",
"esphome/writer.py",
# esp8266/__init__.py imports copy_ccache_script from it
"esphome/platformio/toolchain.py",
# The composite cache action must not ship unexercised
".github/actions/cache-arduino8266/action.yml",
],
)
def test_esp8266_native_components_full_list_on_infra_change(changed: str) -> None:
"""Native-ESP8266 infrastructure changes run the full test list."""
with (
patch.object(determine_jobs, "changed_files", return_value=[changed]),
patch.object(
determine_jobs,
"get_components_with_dependencies",
return_value=["wifi"],
),
):
result = determine_jobs.esp8266_native_components_to_test()
assert result == sorted(determine_jobs.ESP8266_NATIVE_TEST_COMPONENTS)
@pytest.mark.parametrize(
("changed_files", "dependency_closure", "expected"),
[
# Tested component changed -- narrow to the intersection.
(
["esphome/components/mqtt/mqtt_client.cpp"],
["mqtt", "json"],
["mqtt"],
),
# Components outside the test set return an empty list (job skipped).
(
["esphome/components/wifi/wifi_component.cpp"],
["wifi", "network"],
[],
),
# espidf infrastructure is not an esp8266-native trigger; the
# native backend depends on esphome/build_helpers/ instead.
(["esphome/build_gen/espidf.py"], [], []),
(["esphome/espidf/toolchain.py"], [], []),
(["README.md"], [], []),
],
)
def test_esp8266_native_components_to_test_narrowing(
changed_files: list[str],
dependency_closure: list[str],
expected: list[str],
) -> None:
"""Component changes narrow the native-ESP8266 test list."""
with (
patch.object(determine_jobs, "changed_files", return_value=changed_files),
patch.object(
determine_jobs,
"get_components_with_dependencies",
return_value=dependency_closure,
),
):
result = determine_jobs.esp8266_native_components_to_test()
assert result == expected
@@ -236,98 +236,3 @@ def test_run_grouped_test_closes_group_when_subprocess_raises(
)
assert "::endgroup::" in capsys.readouterr().out
def test_components_empty_match_fails_with_flag(
capsys: pytest.CaptureFixture[str],
) -> None:
"""Under --fail-on-no-tests, a real component filtered to a platform
with no matching test file must not pass CI as a green zero-component
compile."""
rc = tbc.test_components(
["logger"],
"zz-none",
"compile",
False,
enable_grouping=False,
fail_on_no_tests=True,
)
assert rc == 1
assert "No tests ran for requested pattern(s): logger" in (capsys.readouterr().out)
def test_components_component_with_no_base_file_fails_with_flag(
capsys: pytest.CaptureFixture[str],
monkeypatch: pytest.MonkeyPatch,
) -> None:
"""A component whose fixture matches the platform but whose platform has
no base file builds nothing; under the flag that silent zero fails by
component name instead of hiding behind other components."""
monkeypatch.setattr(tbc, "get_platform_base_files", lambda base_dir: {})
rc = tbc.test_components(
["logger"],
"esp8266-ard",
"compile",
False,
enable_grouping=False,
fail_on_no_tests=True,
)
assert rc == 1
assert "No tests ran for requested pattern(s): logger" in (capsys.readouterr().out)
def test_components_blank_list_fails_with_flag(
capsys: pytest.CaptureFixture[str],
) -> None:
"""A fully blank component list must not slide into the baseline
fallback and exit green under the flag."""
rc = tbc.test_components(
[""], "esp8266-ard", "compile", False, fail_on_no_tests=True
)
assert rc == 1
assert "blank component list" in capsys.readouterr().out
def test_components_wildcard_no_match_fails_with_flag(
capsys: pytest.CaptureFixture[str],
) -> None:
"""A wildcard matching nothing must not degrade to the synthetic
baseline build and exit green under the flag."""
rc = tbc.test_components(
["zz_no_such*"],
"esp8266-ard",
"compile",
False,
enable_grouping=False,
fail_on_no_tests=True,
)
assert rc == 1
assert "No components found matching" in capsys.readouterr().out
def test_components_empty_match_tolerated_without_flag() -> None:
"""The esp32-ard smoke leg deliberately builds only the subset with a
matching fixture; without the flag an empty match stays green."""
assert (
tbc.test_components(
["logger"], "zz-none", "compile", False, enable_grouping=False
)
== 0
)
def test_components_unknown_component_fails_with_flag(
capsys: pytest.CaptureFixture[str],
) -> None:
"""A renamed smoke-test component must shrink coverage loudly, not fall
into the reference-baseline build."""
rc = tbc.test_components(
["no_such_component_xyz"],
"esp8266-ard",
"compile",
False,
enable_grouping=False,
fail_on_no_tests=True,
)
assert rc == 1
assert "No components found matching" in capsys.readouterr().out
File diff suppressed because it is too large Load Diff
@@ -1,264 +0,0 @@
"""Tests for the ninja build-tool helper script."""
from __future__ import annotations
import os
from pathlib import Path
import subprocess
import sys
from unittest.mock import MagicMock, patch
import pytest
from esphome.build_gen import build_tool
def test_ar_removes_stale_archive(tmp_path: Path) -> None:
archive = tmp_path / "lib.a"
archive.write_text("stale")
rsp = tmp_path / "lib.a.rsp"
rsp.write_text("a.o\n")
with (
patch.object(
build_tool.sys,
"argv",
["build_tool", "ar", "ar-bin", str(archive), str(rsp)],
),
patch.object(
build_tool.subprocess, "run", return_value=MagicMock(returncode=0)
) as mock_run,
):
assert build_tool.main() == 0
assert not archive.exists()
# The rspfile is expanded by the shim (GNU ar would escape backslashes)
assert mock_run.call_args[0][0] == ["ar-bin", "rcs", str(archive), "a.o"]
def test_copy(tmp_path: Path) -> None:
src = tmp_path / "firmware.bin"
src.write_text("data")
dst = tmp_path / "firmware.factory.bin"
with patch.object(
build_tool.sys, "argv", ["build_tool", "copy", str(src), str(dst)]
):
assert build_tool.main() == 0
assert dst.read_text() == "data"
def test_unknown_mode(capsys: pytest.CaptureFixture[str]) -> None:
with patch.object(build_tool.sys, "argv", ["build_tool", "bogus"]):
assert build_tool.main() == 1
assert "unknown build_tool mode" in capsys.readouterr().err
def test_runs_as_script(tmp_path: Path) -> None:
"""The ninja rules invoke the file as a plain script."""
src = tmp_path / "a.bin"
src.write_text("x")
dst = tmp_path / "b.bin"
result = subprocess.run(
[sys.executable, build_tool.__file__, "copy", str(src), str(dst)],
check=False,
)
assert result.returncode == 0
assert dst.read_text() == "x"
def test_ar_expands_rspfile_without_escaping(tmp_path) -> None:
"""Backslash paths survive: the shim expands the rspfile itself instead
of letting GNU ar treat backslashes as escapes."""
rsp = tmp_path / "objs.rsp"
rsp.write_text("obj/a.o\nsub\\b.o\n")
with (
patch.object(
build_tool.sys,
"argv",
["build_tool", "ar", "ar-bin", str(tmp_path / "lib.a"), str(rsp)],
),
patch.object(
build_tool.subprocess, "run", return_value=MagicMock(returncode=0)
) as mock_run,
):
assert build_tool.main() == 0
assert mock_run.call_args[0][0] == [
"ar-bin",
"rcs",
str(tmp_path / "lib.a"),
"obj/a.o",
"sub\\b.o",
]
def test_ar_unquotes_ninja_escaped_paths(tmp_path: Path) -> None:
"""The shim strips a simple surrounding quote, since ninja shell-
quotes special rsp paths, so ar sees the real filename."""
rsp = tmp_path / "t.rsp"
rsp.write_text("'obj/a b.o'\nobj/c.o\n")
with (
patch.object(
build_tool.sys, "argv", ["bt", "ar", "/usr/bin/ar", "lib.a", str(rsp)]
),
patch.object(build_tool.subprocess, "run") as mock_run,
):
mock_run.return_value.returncode = 0
rc = build_tool.main()
assert rc == 0
assert mock_run.call_args.args[0] == [
"/usr/bin/ar",
"rcs",
"lib.a",
"obj/a b.o",
"obj/c.o",
]
def test_ar_empty_object_list_fails(
tmp_path: Path, capsys: pytest.CaptureFixture[str]
) -> None:
"""A lost object list is an error here, not undefined symbols at link."""
rsp = tmp_path / "t.rsp"
rsp.write_text("\n\n")
with patch.object(
build_tool.sys, "argv", ["bt", "ar", "/usr/bin/ar", "lib.a", str(rsp)]
):
rc = build_tool.main()
assert rc == 1
assert "no objects listed" in capsys.readouterr().err
def test_ar_batches_long_object_lists(tmp_path: Path) -> None:
"""The expanded argv must stay under the Windows 32767-char limit: a
long object list creates with rcs, then appends with qs."""
archive = tmp_path / "lib.a"
rsp = tmp_path / "lib.a.rsp"
objects = [f"dir/{'x' * 120}_{i}.o" for i in range(400)]
rsp.write_text("\n".join(objects) + "\n")
with (
patch.object(
build_tool.sys,
"argv",
["build_tool", "ar", "ar-bin", str(archive), str(rsp)],
),
patch.object(
build_tool.subprocess, "run", return_value=MagicMock(returncode=0)
) as mock_run,
):
assert build_tool.main() == 0
calls = [c[0][0] for c in mock_run.call_args_list]
assert len(calls) > 1
assert calls[0][1] == "rcs"
assert all(c[1] == "qs" for c in calls[1:])
assert [o for c in calls for o in c[3:]] == objects
assert all(sum(len(a) + 1 for a in c) < 32000 for c in calls)
def test_ar_batch_failure_stops(tmp_path: Path) -> None:
"""A failing batch propagates its exit code without running the rest."""
archive = tmp_path / "lib.a"
rsp = tmp_path / "lib.a.rsp"
rsp.write_text("\n".join(f"{'y' * 200}_{i}.o" for i in range(300)) + "\n")
with (
patch.object(
build_tool.sys,
"argv",
["build_tool", "ar", "ar-bin", str(archive), str(rsp)],
),
patch.object(
build_tool.subprocess,
"run",
side_effect=lambda cmd, **kw: (
archive.write_text("partial"),
MagicMock(returncode=3),
)[1],
) as mock_run,
):
assert build_tool.main() == 3
assert mock_run.call_count == 1
# The failed batch must not leave a truncated archive behind
assert not archive.exists()
def test_ar_exception_leaves_no_partial_archive(tmp_path: Path) -> None:
"""A missing ar binary mid-loop must not leave a truncated archive from
earlier successful batches."""
archive = tmp_path / "lib.a"
rsp = tmp_path / "lib.a.rsp"
rsp.write_text("a.o\n")
with (
patch.object(
build_tool.sys,
"argv",
["build_tool", "ar", "ar-bin", str(archive), str(rsp)],
),
patch.object(
build_tool.subprocess,
"run",
side_effect=lambda cmd, **kw: (
archive.write_text("partial"),
(_ for _ in ()).throw(FileNotFoundError("no ar")),
),
),
pytest.raises(FileNotFoundError),
):
build_tool.main()
assert not archive.exists()
def test_surplus_arguments_error(capsys: pytest.CaptureFixture[str]) -> None:
"""A mis-specified ninja rule passing extra operands errors instead of
silently dropping them."""
with patch.object(
build_tool.sys, "argv", ["build_tool", "copy", "a", "b", "extra"]
):
assert build_tool.main() == 1
assert "expected 2 arguments, got 3" in capsys.readouterr().err
def test_copy_same_file_keeps_the_input(
tmp_path: Path, capsys: pytest.CaptureFixture[str]
) -> None:
"""A same-file copy (dst IS src) must not unlink the input, and fails
with a message and exit code like the other shim paths."""
src = tmp_path / "firmware.bin"
src.write_bytes(b"image")
with patch.object(
build_tool.sys, "argv", ["build_tool", "copy", str(src), str(src)]
):
assert build_tool.main() == 1
assert src.read_bytes() == b"image"
assert "failed" in capsys.readouterr().err
def test_copy_failure_leaves_no_partial_output(tmp_path: Path) -> None:
"""A failed copy unlinks the destination; a partial firmware image must
never be left on disk."""
dst = tmp_path / "firmware.factory.bin"
dst.write_text("stale")
with (
patch.object(build_tool.shutil, "copyfile", side_effect=OSError("disk full")),
patch.object(
build_tool.sys,
"argv",
["build_tool", "copy", str(tmp_path / "src.bin"), str(dst)],
),
):
assert build_tool.main() == 1
assert not dst.exists()
def test_touch_creates_and_updates_stamp(tmp_path: Path) -> None:
stamp = tmp_path / "esphome_pch.probe"
assert build_tool._run_touch(str(stamp)) == 0
assert stamp.is_file()
os.utime(stamp, (1, 1))
assert build_tool._run_touch(str(stamp)) == 0
assert stamp.stat().st_mtime > 1
def test_touch_reports_failure(
tmp_path: Path, capsys: pytest.CaptureFixture[str]
) -> None:
missing_dir = tmp_path / "gone" / "stamp"
assert build_tool._run_touch(str(missing_dir)) == 1
assert "touch:" in capsys.readouterr().err
-921
View File
@@ -4,9 +4,7 @@ from __future__ import annotations
import json
import logging
import os
from pathlib import Path
import subprocess
from unittest.mock import patch
import pytest
@@ -490,922 +488,3 @@ def test_get_component_cmakelists_no_compile_features() -> None:
content = get_component_cmakelists()
assert "target_compile_features" not in content
@pytest.fixture(autouse=True)
def _pch_default_on(monkeypatch: pytest.MonkeyPatch) -> None:
"""Pin the knob so a developer's ESPHOME_PCH_ENABLE=0 cannot fail these."""
monkeypatch.setenv("ESPHOME_PCH_ENABLE", "1")
def _make_pch_device(tmp_path: Path, name: str) -> Path:
"""A device dir with the pch source headers and a stub compile_commands."""
from esphome.build_helpers.pch import PCH_DEFAULT_HEADERS
dev = tmp_path / name
for header in PCH_DEFAULT_HEADERS:
path = dev / "src" / header
path.parent.mkdir(parents=True, exist_ok=True)
path.write_text("")
# A real quoted include chain and a per-device-named sdkconfig with
# identical content: the closure and sdkconfig inputs must be exercised
(dev / "src" / "esphome" / "core" / "defines.h").write_text(
'#include "esphome/core/macros.h"\n'
)
(dev / "src" / "esphome" / "core" / "macros.h").write_text("#define M 1\n")
# Both spellings: tests patch CORE.name to "test" or to the device name
(dev / f"sdkconfig.{name}").write_text("CONFIG_X=y\n")
(dev / "sdkconfig.test").write_text("CONFIG_X=y\n")
build = dev / "build"
build.mkdir(exist_ok=True)
from esphome.build_helpers.pch import pch_header_text
(build / "esphome_pch.h").write_text(pch_header_text(PCH_DEFAULT_HEADERS))
# Native separators: mixed f-string paths break the src-prefix match
# on Windows
src_file = str(dev / "src" / "a.cpp")
(build / "compile_commands.json").write_text(
json.dumps(
[
{
"directory": str(build),
"command": (
"g++ -DX=1 -include esphome_pch.h "
"-o esp-idf/src/CMakeFiles/__idf_src.dir/a.cpp.obj "
f'-c "{src_file}"'
),
"file": src_file,
}
]
)
)
return dev
def test_prepare_pch_writes_header_and_sum(tmp_path: Path) -> None:
from esphome.build_gen.espidf import prepare_pch
dev = _make_pch_device(tmp_path, "dev_a")
CORE.build_path = dev
gch = dev / "build" / "esphome_pch.h.gch"
def fake_compile(cmd, **kwargs):
if "-fsyntax-only" in cmd:
# The load probe follows a successful .gch build
return subprocess.CompletedProcess(cmd, 0, "", "")
# The compile must target the header, not the stub TU
assert cmd[-5:-3] == ["c++-header", "-c"]
gch.write_bytes(b"gch")
return subprocess.CompletedProcess(cmd, 0, "", "")
with (
patch.object(CORE, "name", "test"),
patch("esphome.build_helpers.pch.subprocess.run", side_effect=fake_compile),
):
prepare_pch()
checksum = (dev / "build" / "esphome_pch.h.gch.sum").read_text().strip()
assert len(checksum) == 64
# Unchanged inputs: the second call must not recompile
with (
patch.object(CORE, "name", "test"),
patch("esphome.build_helpers.pch.subprocess.run", side_effect=AssertionError),
):
prepare_pch()
def test_pch_no_device_path_poison(tmp_path: Path) -> None:
"""Regression: neither the injected -include nor the .sum may carry the
per-device build path, or cross-device ccache sharing breaks."""
from esphome.build_gen.espidf import get_component_cmakelists, prepare_pch
sums = []
for name in ("dev_a", "dev_b"):
dev = _make_pch_device(tmp_path, name)
CORE.build_path = dev
gch = dev / "build" / "esphome_pch.h.gch"
def fake_compile(cmd, _gch=gch, **kwargs):
_gch.write_bytes(b"gch")
return subprocess.CompletedProcess(cmd, 0, "", "")
with (
patch.object(CORE, "name", name),
patch("esphome.build_helpers.pch.subprocess.run", side_effect=fake_compile),
):
prepare_pch()
content = get_component_cmakelists()
assert str(dev) not in content
sums.append((dev / "build" / "esphome_pch.h.gch.sum").read_text())
assert sums[0] == sums[1]
def test_component_cmakelists_pch_block(monkeypatch: pytest.MonkeyPatch) -> None:
from esphome.build_gen.espidf import get_component_cmakelists
content = get_component_cmakelists()
assert '"$<$<COMPILE_LANGUAGE:CXX>:-include>"' in content
assert "-Wno-error=invalid-pch" in content
monkeypatch.setenv("ESPHOME_PCH_STRICT", "1")
strict_content = get_component_cmakelists()
assert "-Werror=invalid-pch" in strict_content
monkeypatch.delenv("ESPHOME_PCH_STRICT")
assert '"$<$<COMPILE_LANGUAGE:CXX>:esphome_pch.h>"' in content
monkeypatch.setenv("ESPHOME_PCH_ENABLE", "0")
assert "-include" not in get_component_cmakelists()
def test_pch_compile_command_variants(tmp_path: Path) -> None:
"""Missing DB, no matching entry, and launcher-prefixed commands."""
from esphome.build_helpers.pch import pch_compile_command
build = tmp_path / "build"
build.mkdir()
header = build / "esphome_pch.h"
gch = build / "esphome_pch.h.gch"
assert pch_compile_command(build, header, gch) is None
(build / "compile_commands.json").write_text(
json.dumps(
[
{"command": "gcc -c other.c", "file": "other.c"},
]
)
)
assert pch_compile_command(build, header, gch) is None
src_file = str(tmp_path / "src" / "esphome" / "a.cpp")
(build / "compile_commands.json").write_text(
json.dumps(
[
{
"command": (
"/usr/bin/ccache g++ -DX=1 -include esphome_pch.h -MMD "
"-MT a.cpp.obj -MF a.cpp.obj.d "
"-o esp-idf/src/CMakeFiles/__idf_src.dir/a.cpp.obj "
f"-c {src_file}"
),
"file": src_file,
},
]
)
)
# Launcher stripped; -include/-o/-c and depfile flags removed
cmd, cmd_dir = pch_compile_command(build, header, gch)
assert cmd == [
"g++",
"-DX=1",
"-x",
"c++-header",
"-c",
str(header),
"-o",
str(gch),
]
# The compile must run where the flags were resolved
assert cmd_dir == build
@pytest.mark.skipif(os.name == "nt", reason="symlinks need privileges on Windows")
def test_pch_compile_command_matches_src_through_symlink(tmp_path: Path) -> None:
"""Find the src TU when CMake spells paths through a different symlink (macOS /tmp)."""
from esphome.build_helpers.pch import pch_compile_command
real = tmp_path / "real"
(real / "src" / "esphome").mkdir(parents=True)
link = tmp_path / "link"
link.symlink_to(real, target_is_directory=True)
CORE.build_path = str(real)
build = real / "build"
build.mkdir()
header = build / "esphome_pch.h"
gch = build / "esphome_pch.h.gch"
src_file = str(link / "src" / "esphome" / "a.cpp")
(build / "compile_commands.json").write_text(
json.dumps([{"command": f"g++ -DX=1 -o a.obj -c {src_file}", "file": src_file}])
)
cmd, cmd_dir = pch_compile_command(build, header, gch)
assert cmd[:2] == ["g++", "-DX=1"]
assert cmd_dir == build
def test_pch_compile_command_rejects_unusable_entries(tmp_path: Path) -> None:
"""Malformed DB shapes and command-less entries skip cleanly instead of
producing a compiler-less argv retried every build."""
from esphome.build_helpers.pch import pch_compile_command
build = tmp_path / "build"
build.mkdir()
header = build / "esphome_pch.h"
gch = build / "esphome_pch.h.gch"
db = build / "compile_commands.json"
src_file = str(tmp_path / "src" / "esphome" / "a.cpp")
db.write_text(json.dumps({"not": "a list"}))
assert pch_compile_command(build, header, gch) is None
db.write_text(json.dumps(["just a string"]))
assert pch_compile_command(build, header, gch) is None
# An empty-string directory must fall back to the build dir, not cwd
db.write_text(
json.dumps(
[
{
"directory": "",
"command": f"g++ -DX=1 -o a.obj -c {src_file}",
"file": src_file,
}
]
)
)
_, cmd_dir = pch_compile_command(build, header, gch)
assert cmd_dir == build
# Corrupted entries with null fields must skip, not raise
db.write_text(
json.dumps(
[
{"file": None, "command": "g++ -c x.cpp", "directory": None},
{"file": src_file, "command": None, "directory": None},
]
)
)
assert pch_compile_command(build, header, gch) is None
# arguments-style entry (allowed by the spec, unused by CMake)
db.write_text(
json.dumps([{"arguments": ["g++", "-c", src_file], "file": src_file}])
)
assert pch_compile_command(build, header, gch) is None
def test_pch_header_list_order_is_in_checksum(
tmp_path: Path, monkeypatch: pytest.MonkeyPatch
) -> None:
"""Reordering PCH_DEFAULT_HEADERS keeps the include closure identical, but the
generated header text differs, so the .gch must rebuild."""
import esphome.build_gen.espidf as espidf_mod
dev = _make_pch_device(tmp_path, "dev_r")
CORE.build_path = dev
gch = dev / "build" / "esphome_pch.h.gch"
def fake_compile(cmd, **kwargs):
gch.write_bytes(b"gch")
return subprocess.CompletedProcess(cmd, 0, "", "")
with (
patch.object(CORE, "name", "test"),
patch("esphome.build_helpers.pch.subprocess.run", side_effect=fake_compile),
):
espidf_mod.prepare_pch()
first = (dev / "build" / "esphome_pch.h.gch.sum").read_text()
monkeypatch.setattr(
espidf_mod,
"PCH_DEFAULT_HEADERS",
tuple(reversed(espidf_mod.PCH_DEFAULT_HEADERS)),
)
espidf_mod.prepare_pch()
assert (dev / "build" / "esphome_pch.h.gch.sum").read_text() != first
def test_prepare_pch_failure_writes_marker_and_skips_retry(tmp_path: Path) -> None:
from esphome.build_gen.espidf import prepare_pch
dev = _make_pch_device(tmp_path, "dev_f")
CORE.build_path = dev
calls = []
def failing_compile(cmd, **kwargs):
calls.append(cmd)
return subprocess.CompletedProcess(cmd, 1, "", "boom")
with (
patch.object(CORE, "name", "test"),
patch("esphome.build_helpers.pch.subprocess.run", side_effect=failing_compile),
):
prepare_pch()
prepare_pch()
assert len(calls) == 1
assert not (dev / "build" / "esphome_pch.h.gch.sum").exists()
assert (dev / "build" / "esphome_pch.h.gch.failed").exists()
def test_prepare_pch_spawn_oserror_is_transient(tmp_path: Path) -> None:
"""Spawn/IO failures retry on the next build instead of latching."""
from esphome.build_gen.espidf import prepare_pch
dev = _make_pch_device(tmp_path, "dev_o")
CORE.build_path = dev
calls = []
def raising(cmd, **kwargs):
calls.append(cmd)
raise OSError("no such compiler")
header = dev / "build" / "esphome_pch.h"
before = header.stat().st_mtime_ns
with (
patch.object(CORE, "name", "test"),
patch("esphome.build_helpers.pch.subprocess.run", side_effect=raising),
):
prepare_pch()
prepare_pch()
assert not (dev / "build" / "esphome_pch.h.gch.failed").exists()
assert not (dev / "build" / "esphome_pch.h.gch.sum").exists()
assert len(calls) == 2
# No .gch was ever in play, so the header must not be re-touched into
# forcing a full rebuild on every failing build
assert header.stat().st_mtime_ns == before
def test_prepare_pch_transient_with_stale_gch_bumps_header(tmp_path: Path) -> None:
"""A stale .gch removed on a transient failure must dirty its consumers."""
from esphome.build_gen.espidf import prepare_pch
dev = _make_pch_device(tmp_path, "dev_s")
CORE.build_path = dev
gch = dev / "build" / "esphome_pch.h.gch"
gch.write_bytes(b"stale")
header = dev / "build" / "esphome_pch.h"
os.utime(header, (1, 1))
with (
patch.object(CORE, "name", "test"),
patch(
"esphome.build_helpers.pch.subprocess.run",
side_effect=OSError("no such compiler"),
),
):
prepare_pch()
assert not gch.exists()
assert header.stat().st_mtime_ns > 1_000_000_000
def test_prepare_pch_disabled_discards_and_skips_compile(
tmp_path: Path, monkeypatch: pytest.MonkeyPatch
) -> None:
"""The escape hatch is self-cleaning: a leftover .gch is removed."""
from esphome.build_gen.espidf import prepare_pch
monkeypatch.setenv("ESPHOME_PCH_ENABLE", "0")
dev = _make_pch_device(tmp_path, "dev_d")
CORE.build_path = dev
stale = dev / "build" / "esphome_pch.h.gch"
stale.write_bytes(b"stale")
with patch("esphome.build_helpers.pch.subprocess.run", side_effect=AssertionError):
prepare_pch()
assert not stale.exists()
def test_prepare_pch_missing_sdkconfig_fails_closed(tmp_path: Path) -> None:
"""No sdkconfig means no config identity for the .sum: no pch at all."""
from esphome.build_gen.espidf import prepare_pch
dev = _make_pch_device(tmp_path, "dev_m")
(dev / "sdkconfig.test").unlink()
CORE.build_path = dev
stale = dev / "build" / "esphome_pch.h.gch"
stale.write_bytes(b"stale")
with (
patch.object(CORE, "name", "test"),
patch("esphome.build_helpers.pch.subprocess.run", side_effect=AssertionError),
):
prepare_pch()
assert not stale.exists()
assert not (dev / "build" / "esphome_pch.h.gch.sum").exists()
def test_prepare_pch_signal_kill_is_transient(tmp_path: Path) -> None:
"""A signal-killed compile (OOM) must not latch the .failed marker."""
from esphome.build_gen.espidf import prepare_pch
dev = _make_pch_device(tmp_path, "dev_k")
CORE.build_path = dev
calls = []
def killed(cmd, **kwargs):
calls.append(cmd)
return subprocess.CompletedProcess(cmd, -9, "", "")
with (
patch.object(CORE, "name", "test"),
patch("esphome.build_helpers.pch.subprocess.run", side_effect=killed),
):
prepare_pch()
prepare_pch()
assert not (dev / "build" / "esphome_pch.h.gch.failed").exists()
assert len(calls) == 2
def test_prepare_pch_probe_spawn_failure_degrades(
tmp_path: Path, monkeypatch: pytest.MonkeyPatch
) -> None:
"""A probe that cannot run discards the pch; strict raises."""
from esphome.build_gen.espidf import prepare_pch
from esphome.core import EsphomeError
dev = _make_pch_device(tmp_path, "dev_pf")
CORE.build_path = dev
gch = dev / "build" / "esphome_pch.h.gch"
def probe_dies(cmd, **kwargs):
if "-fsyntax-only" in cmd:
raise OSError("probe spawn failed")
gch.write_bytes(b"gch")
return subprocess.CompletedProcess(cmd, 0, "", "")
with (
patch.object(CORE, "name", "test"),
patch("esphome.build_helpers.pch.subprocess.run", side_effect=probe_dies),
):
prepare_pch()
assert not gch.exists()
assert not (dev / "build" / "esphome_pch.h.gch.sum").exists()
monkeypatch.setenv("ESPHOME_PCH_STRICT", "1")
with (
patch.object(CORE, "name", "test"),
patch("esphome.build_helpers.pch.subprocess.run", side_effect=probe_dies),
pytest.raises(EsphomeError, match="probe did not run"),
):
prepare_pch()
@pytest.mark.parametrize(
("stderr", "code"),
[("", -9), ("fatal: No space left on device", 1)],
ids=("signal-kill", "enospc"),
)
def test_prepare_pch_probe_environmental_failures_do_not_latch(
tmp_path: Path, stderr: str, code: int
) -> None:
"""A signal-killed or ENOSPC probe retries next build, no marker."""
from esphome.build_gen.espidf import prepare_pch
dev = _make_pch_device(tmp_path, "dev_pe")
CORE.build_path = dev
gch = dev / "build" / "esphome_pch.h.gch"
def env_probe(cmd, **kwargs):
if "-fsyntax-only" in cmd:
return subprocess.CompletedProcess(cmd, code, "", stderr)
gch.write_bytes(b"gch")
return subprocess.CompletedProcess(cmd, 0, "", "")
with (
patch.object(CORE, "name", "test"),
patch("esphome.build_helpers.pch.subprocess.run", side_effect=env_probe),
):
prepare_pch()
assert not gch.exists()
assert not (dev / "build" / "esphome_pch.h.gch.failed").exists()
def test_prepare_pch_signal_kill_strict_raises(
tmp_path: Path, monkeypatch: pytest.MonkeyPatch
) -> None:
from esphome.build_gen.espidf import prepare_pch
from esphome.core import EsphomeError
monkeypatch.setenv("ESPHOME_PCH_STRICT", "1")
dev = _make_pch_device(tmp_path, "dev_ks")
CORE.build_path = dev
with (
patch.object(CORE, "name", "test"),
patch(
"esphome.build_helpers.pch.subprocess.run",
side_effect=lambda cmd, **kw: subprocess.CompletedProcess(cmd, -9, "", ""),
),
pytest.raises(EsphomeError, match="killed by signal"),
):
prepare_pch()
def test_prepare_pch_strict_raises_when_disabled(
tmp_path: Path, monkeypatch: pytest.MonkeyPatch
) -> None:
"""Strict must not read a disabled pch as success."""
from esphome.build_gen.espidf import prepare_pch
from esphome.core import EsphomeError
monkeypatch.setenv("ESPHOME_PCH_ENABLE", "0")
monkeypatch.setenv("ESPHOME_PCH_STRICT", "1")
dev = _make_pch_device(tmp_path, "dev_ds")
CORE.build_path = dev
with (
patch.object(CORE, "name", "test"),
pytest.raises(EsphomeError, match="disabled"),
):
prepare_pch()
def test_prepare_pch_missing_sdkconfig_strict_raises(
tmp_path: Path, monkeypatch: pytest.MonkeyPatch
) -> None:
from esphome.build_gen.espidf import prepare_pch
from esphome.core import EsphomeError
monkeypatch.setenv("ESPHOME_PCH_STRICT", "1")
dev = _make_pch_device(tmp_path, "dev_ss")
(dev / "sdkconfig.test").unlink()
CORE.build_path = dev
with (
patch.object(CORE, "name", "test"),
pytest.raises(EsphomeError, match="sdkconfig unreadable"),
):
prepare_pch()
def test_prepare_pch_without_compile_commands(tmp_path: Path) -> None:
"""Stale checksum but no configured TU yet: no compile, no sidecars."""
from esphome.build_gen.espidf import prepare_pch
dev = _make_pch_device(tmp_path, "dev_n")
(dev / "build" / "compile_commands.json").unlink()
CORE.build_path = dev
with (
patch.object(CORE, "name", "test"),
patch("esphome.build_helpers.pch.subprocess.run", side_effect=AssertionError),
):
prepare_pch()
assert not (dev / "build" / "esphome_pch.h.gch.sum").exists()
assert not (dev / "build" / "esphome_pch.h.gch.failed").exists()
def test_write_project_pch_disabled_writes_no_header(
tmp_path: Path, monkeypatch: pytest.MonkeyPatch
) -> None:
from esphome.build_gen.espidf import write_project
monkeypatch.setenv("ESPHOME_PCH_ENABLE", "0")
_write_project_description(tmp_path, {})
CORE.build_path = tmp_path
with (
patch("esphome.build_gen.espidf.get_esp32_variant", return_value="ESP32"),
patch.object(CORE, "name", "test"),
):
write_project()
assert not (tmp_path / "build" / "esphome_pch.h").exists()
def test_write_project_writes_pch_header(tmp_path: Path) -> None:
"""The header write_project emits is what _pch_cmake() force-includes;
this pairing is the one non-fail-safe path in the design."""
from esphome.build_gen.espidf import write_project
from esphome.build_helpers.pch import PCH_DEFAULT_HEADERS, pch_header_text
_write_project_description(tmp_path, {})
CORE.build_path = tmp_path
with (
patch("esphome.build_gen.espidf.get_esp32_variant", return_value="ESP32"),
patch.object(CORE, "name", "test"),
):
write_project()
assert (tmp_path / "build" / "esphome_pch.h").read_text() == pch_header_text(
PCH_DEFAULT_HEADERS
)
def test_prepare_pch_stale_bailout_removes_gch(tmp_path: Path) -> None:
"""A stale .gch must not survive when no compile command is available."""
from esphome.build_gen.espidf import prepare_pch
dev = _make_pch_device(tmp_path, "dev_s")
CORE.build_path = dev
gch = dev / "build" / "esphome_pch.h.gch"
gch.write_bytes(b"stale")
(dev / "build" / "esphome_pch.h.gch.sum").write_text("stale-sum\n")
(dev / "build" / "compile_commands.json").unlink()
with patch.object(CORE, "name", "test"):
prepare_pch()
assert not gch.exists()
assert not (dev / "build" / "esphome_pch.h.gch.sum").exists()
def test_prepare_pch_zero_exit_without_gch_is_failure(tmp_path: Path) -> None:
from esphome.build_gen.espidf import prepare_pch
dev = _make_pch_device(tmp_path, "dev_z")
CORE.build_path = dev
def no_output(cmd, **kwargs):
return subprocess.CompletedProcess(cmd, 0, "", "")
with (
patch.object(CORE, "name", "test"),
patch("esphome.build_helpers.pch.subprocess.run", side_effect=no_output),
):
prepare_pch()
assert not (dev / "build" / "esphome_pch.h.gch.sum").exists()
assert (dev / "build" / "esphome_pch.h.gch.failed").exists()
def test_prepare_pch_bumps_header_for_object_depends(tmp_path: Path) -> None:
"""The OBJECT_DEPENDS edge watches the header; a rebuilt .gch must bump
it so pch-consuming TUs recompile."""
import os as _os
from esphome.build_gen.espidf import prepare_pch
dev = _make_pch_device(tmp_path, "dev_t")
CORE.build_path = dev
header = dev / "build" / "esphome_pch.h"
gch = dev / "build" / "esphome_pch.h.gch"
_os.utime(header, (0, 0))
before = header.stat().st_mtime
def fake_compile(cmd, **kwargs):
gch.write_bytes(b"gch")
return subprocess.CompletedProcess(cmd, 0, "", "")
with (
patch.object(CORE, "name", "test"),
patch("esphome.build_helpers.pch.subprocess.run", side_effect=fake_compile),
):
prepare_pch()
assert header.stat().st_mtime > before
def test_component_cmakelists_pch_object_depends() -> None:
from esphome.build_gen.espidf import get_component_cmakelists
content = get_component_cmakelists()
assert 'OBJECT_DEPENDS "${CMAKE_BINARY_DIR}/esphome_pch.h"' in content
def test_prepare_pch_command_change_invalidates_sum(tmp_path: Path) -> None:
"""A flag-only change in the compile DB must rebuild the .gch."""
from esphome.build_gen.espidf import prepare_pch
dev = _make_pch_device(tmp_path, "dev_c")
CORE.build_path = dev
gch = dev / "build" / "esphome_pch.h.gch"
def fake_compile(cmd, **kwargs):
gch.write_bytes(b"gch")
return subprocess.CompletedProcess(cmd, 0, "", "")
with (
patch.object(CORE, "name", "test"),
patch("esphome.build_helpers.pch.subprocess.run", side_effect=fake_compile),
):
prepare_pch()
first = (dev / "build" / "esphome_pch.h.gch.sum").read_text()
db = dev / "build" / "compile_commands.json"
db.write_text(db.read_text().replace("-DX=1", "-DX=2"))
prepare_pch()
assert (dev / "build" / "esphome_pch.h.gch.sum").read_text() != first
def test_prepare_pch_keeps_user_force_includes(tmp_path: Path) -> None:
from esphome.build_helpers.pch import pch_compile_command
dev = _make_pch_device(tmp_path, "dev_u")
CORE.build_path = dev
build = dev / "build"
src_file = str(dev / "src" / "esphome" / "a.cpp")
build.joinpath("compile_commands.json").write_text(
json.dumps(
[
{
"directory": str(build),
"command": (
"g++ -include user.h -include esphome_pch.h "
f"-o a.obj -c {src_file}"
),
"file": src_file,
}
]
)
)
cmd, _ = pch_compile_command(build, build / "esphome_pch.h", build / "x.gch")
assert "user.h" in cmd
assert "esphome_pch.h" not in " ".join(cmd[:-3])
def test_prepare_pch_identity_unknown_discards(tmp_path: Path) -> None:
"""An OSError from the checksum discards artifacts and skips the pch."""
from esphome.build_gen.espidf import prepare_pch
dev = _make_pch_device(tmp_path, "dev_i")
CORE.build_path = dev
stale = dev / "build" / "esphome_pch.h.gch"
stale.write_bytes(b"stale")
with (
patch.object(CORE, "name", "test"),
patch(
"esphome.build_helpers.pch.pch_checksum",
side_effect=OSError("stat failed"),
),
patch("esphome.build_helpers.pch.subprocess.run", side_effect=AssertionError),
):
prepare_pch()
assert not stale.exists()
assert not (dev / "build" / "esphome_pch.h.gch.sum").exists()
def test_prepare_pch_transient_compiler_failure_does_not_latch(
tmp_path: Path,
) -> None:
"""ENOSPC-style failures clear on their own; no .failed marker."""
from esphome.build_gen.espidf import prepare_pch
dev = _make_pch_device(tmp_path, "dev_e")
CORE.build_path = dev
calls = []
def enospc(cmd, **kwargs):
calls.append(cmd)
return subprocess.CompletedProcess(
cmd, 1, "", "fatal error: No space left on device"
)
with (
patch.object(CORE, "name", "test"),
patch("esphome.build_helpers.pch.subprocess.run", side_effect=enospc),
):
prepare_pch()
prepare_pch()
assert not (dev / "build" / "esphome_pch.h.gch.failed").exists()
assert len(calls) == 2
def test_prepare_pch_strict_raises_on_missing_db(
tmp_path: Path, monkeypatch: pytest.MonkeyPatch
) -> None:
"""ESPHOME_PCH_STRICT turns the silent skip into a failure."""
from esphome.build_gen.espidf import prepare_pch
from esphome.core import EsphomeError
monkeypatch.setenv("ESPHOME_PCH_STRICT", "1")
dev = _make_pch_device(tmp_path, "dev_st")
(dev / "build" / "compile_commands.json").unlink()
CORE.build_path = dev
with (
patch.object(CORE, "name", "test"),
patch("esphome.build_helpers.pch.subprocess.run", side_effect=AssertionError),
pytest.raises(EsphomeError, match="no usable compile command"),
):
prepare_pch()
def test_prepare_pch_probe_rejection_latches_and_degrades(
tmp_path: Path, monkeypatch: pytest.MonkeyPatch
) -> None:
"""A toolchain that cannot load its own .gch discards it, latches the
marker, and fails strict mode."""
from esphome.build_gen.espidf import prepare_pch
from esphome.core import EsphomeError
dev = _make_pch_device(tmp_path, "dev_p")
CORE.build_path = dev
gch = dev / "build" / "esphome_pch.h.gch"
def rejecting(cmd, **kwargs):
if "-fsyntax-only" in cmd:
if "-include" not in cmd:
# Baseline without the pch passes: the pch is to blame
return subprocess.CompletedProcess(cmd, 0, "", "")
return subprocess.CompletedProcess(
cmd, 1, "", "error: esphome_pch.h.gch: had text segment "
)
gch.write_bytes(b"gch")
return subprocess.CompletedProcess(cmd, 0, "", "")
with (
patch.object(CORE, "name", "test"),
patch("esphome.build_helpers.pch.subprocess.run", side_effect=rejecting),
):
prepare_pch()
assert not gch.exists()
assert (dev / "build" / "esphome_pch.h.gch.failed").exists()
monkeypatch.setenv("ESPHOME_PCH_STRICT", "1")
(dev / "build" / "esphome_pch.h.gch.failed").unlink()
with (
patch.object(CORE, "name", "test"),
patch("esphome.build_helpers.pch.subprocess.run", side_effect=rejecting),
pytest.raises(EsphomeError, match="cannot load the pch"),
):
prepare_pch()
def test_prepare_pch_strict_reprobes_cached_gch(
tmp_path: Path, monkeypatch: pytest.MonkeyPatch
) -> None:
"""Rejection is per-process: strict must re-prove a cached .gch loads."""
from esphome.build_gen.espidf import prepare_pch
from esphome.core import EsphomeError
dev = _make_pch_device(tmp_path, "dev_rc")
CORE.build_path = dev
gch = dev / "build" / "esphome_pch.h.gch"
def ok(cmd, **kwargs):
if "-fsyntax-only" not in cmd:
gch.write_bytes(b"gch")
return subprocess.CompletedProcess(cmd, 0, "", "")
with (
patch.object(CORE, "name", "test"),
patch("esphome.build_helpers.pch.subprocess.run", side_effect=ok),
):
prepare_pch()
assert gch.exists()
def reject(cmd, **kwargs):
assert "-fsyntax-only" in cmd, "cached path must not recompile"
if "-include" not in cmd:
return subprocess.CompletedProcess(cmd, 0, "", "")
return subprocess.CompletedProcess(
cmd, 1, "", "error: esphome_pch.h.gch: had text segment "
)
monkeypatch.setenv("ESPHOME_PCH_STRICT", "1")
with (
patch.object(CORE, "name", "test"),
patch("esphome.build_helpers.pch.subprocess.run", side_effect=reject),
pytest.raises(EsphomeError, match="cannot load the pch"),
):
prepare_pch()
assert not gch.exists()
# Per-process rejection may not reproduce: the cached path must not
# latch the pch off for later non-strict builds
assert not (dev / "build" / "esphome_pch.h.gch.failed").exists()
def test_prepare_pch_unexpected_command_shape_degrades(
tmp_path: Path, monkeypatch: pytest.MonkeyPatch
) -> None:
"""A tail the probe slice cannot trust discards and degrades."""
from esphome.build_gen.espidf import prepare_pch
from esphome.core import EsphomeError
monkeypatch.setenv("ESPHOME_PCH_STRICT", "1")
dev = _make_pch_device(tmp_path, "dev_sh")
CORE.build_path = dev
gch = dev / "build" / "esphome_pch.h.gch"
def ok(cmd, **kwargs):
gch.write_bytes(b"gch")
return subprocess.CompletedProcess(cmd, 0, "", "")
with (
patch.object(CORE, "name", "test"),
patch("esphome.build_helpers.pch.subprocess.run", side_effect=ok),
patch(
"esphome.build_helpers.pch.pch_compile_command",
return_value=(
["g++", "-DX=1", "-c", "x", "-o", "y", "extra"],
dev / "build",
),
),
pytest.raises(EsphomeError, match="command shape"),
):
prepare_pch()
assert not gch.exists()
# Non-strict: same shape problem degrades without raising
monkeypatch.delenv("ESPHOME_PCH_STRICT")
with (
patch.object(CORE, "name", "test"),
patch("esphome.build_helpers.pch.subprocess.run", side_effect=ok),
patch(
"esphome.build_helpers.pch.pch_compile_command",
return_value=(
["g++", "-DX=1", "-c", "x", "-o", "y", "extra"],
dev / "build",
),
),
):
prepare_pch()
assert not gch.exists()
def test_prepare_pch_probe_baseline_spawn_failure_is_transient(
tmp_path: Path,
) -> None:
"""A baseline that cannot spawn is environmental: no marker."""
from esphome.build_gen.espidf import prepare_pch
dev = _make_pch_device(tmp_path, "dev_bs")
CORE.build_path = dev
gch = dev / "build" / "esphome_pch.h.gch"
def flaky(cmd, **kwargs):
if "-fsyntax-only" in cmd:
if "-include" in cmd:
return subprocess.CompletedProcess(cmd, 1, "", "boom")
raise OSError("baseline spawn failed")
gch.write_bytes(b"gch")
return subprocess.CompletedProcess(cmd, 0, "", "")
with (
patch.object(CORE, "name", "test"),
patch("esphome.build_helpers.pch.subprocess.run", side_effect=flaky),
):
prepare_pch()
assert not gch.exists()
assert not (dev / "build" / "esphome_pch.h.gch.failed").exists()
@@ -10,7 +10,6 @@ from unittest.mock import patch
import pytest
from esphome.build_helpers import ccache
from esphome.core import CORE
def test_resolve_opt_out() -> None:
@@ -121,17 +120,3 @@ def test_parse_enable_env_spelling_tables(
"""cv.boolean's spelling tables plus the 1/0 env convention."""
monkeypatch.setenv("ESPHOME_CCACHE_ENABLE", raw)
assert ccache.parse_enable_env("ESPHOME_CCACHE_ENABLE") is expected
def test_effective_ccache_basedir_prefers_user_value(tmp_path: Path) -> None:
CORE.build_path = tmp_path
# Drive-qualified on Windows: "/custom/base" is not absolute there
base = "C:\\custom\\base" if os.name == "nt" else "/custom/base"
with patch.dict(os.environ, {"CCACHE_BASEDIR": base}, clear=True):
assert ccache.effective_ccache_basedir() == base
with patch.dict(os.environ, {}, clear=True):
assert ccache.effective_ccache_basedir() == str(tmp_path.resolve())
# Degenerate values would strip substrings ccache never rewrites
for bad in ("", "/", "a/b"):
with patch.dict(os.environ, {"CCACHE_BASEDIR": bad}, clear=True):
assert ccache.effective_ccache_basedir() == str(tmp_path.resolve())
+7 -88
View File
@@ -85,87 +85,6 @@ def test_parse_entry_resolves_relative_includes() -> None:
assert all(Path(inc).is_absolute() for inc in includes)
def test_parse_entry_resolves_force_include_path(tmp_path: Path) -> None:
"""The pch -include is emitted relative to the build dir; idedata must
resolve it so cached flags work from any cwd."""
(tmp_path / "esphome_pch.h").write_text("")
entry = _entry(
str(tmp_path),
f"{tmp_path}/src/esphome/x.cpp",
"g++ -include esphome_pch.h -c x.cpp",
)
_, _, _, cxx_flags = idedata.parse_entry(entry)
idx = cxx_flags.index("-include")
resolved = cxx_flags[idx + 1]
assert Path(resolved).is_absolute()
assert resolved == str(tmp_path / "esphome_pch.h").replace("\\", "/")
def test_parse_entry_resolves_joined_force_include(tmp_path: Path) -> None:
"""The joined -includefoo.h spelling takes the same resolve path."""
(tmp_path / "esphome_pch.h").write_text("")
entry = _entry(
str(tmp_path),
f"{tmp_path}/src/esphome/x.cpp",
"g++ -includeesphome_pch.h -c x.cpp",
)
_, _, _, cxx_flags = idedata.parse_entry(entry)
resolved = cxx_flags[cxx_flags.index("-include") + 1]
assert resolved == str(tmp_path / "esphome_pch.h").replace("\\", "/")
def test_parse_entry_keeps_search_chain_force_include(tmp_path: Path) -> None:
"""-include names resolved via the -I chain (libretiny's Arduino.h) must
not be re-anchored to a nonexistent build-dir path."""
entry = _entry(
str(tmp_path),
f"{tmp_path}/src/esphome/x.cpp",
"g++ -include Arduino.h -c x.cpp",
)
_, _, _, cxx_flags = idedata.parse_entry(entry)
assert cxx_flags[cxx_flags.index("-include") + 1] == "Arduino.h"
def test_parse_entry_warns_on_vanished_force_include(
tmp_path: Path, caplog: pytest.LogCaptureFixture
) -> None:
"""A build-dir force-include deleted by clean_build must leave a trail;
a name resolvable via the -I chain must not warn."""
inc = tmp_path / "inc"
inc.mkdir()
(inc / "Arduino.h").write_text("")
entry = _entry(
str(tmp_path),
f"{tmp_path}/src/esphome/x.cpp",
f"g++ -I{inc} -include Arduino.h -include esphome_pch.h -c x.cpp",
)
_, _, _, cxx_flags = idedata.parse_entry(entry)
assert "Arduino.h" in cxx_flags
assert "esphome_pch.h" in caplog.text
assert "Arduino.h" not in caplog.text
def test_parse_entry_drops_trailing_force_include(
tmp_path: Path, caplog: pytest.LogCaptureFixture
) -> None:
entry = _entry(
str(tmp_path), f"{tmp_path}/src/esphome/x.cpp", "g++ -c x.cpp -include"
)
_, _, _, cxx_flags = idedata.parse_entry(entry)
assert "-include" not in cxx_flags
assert "no argument" in caplog.text
def test_parse_entry_skips_dependency_flags() -> None:
"""Dependency-generation flags (and their args) are dropped."""
entry = _entry(
@@ -185,7 +104,7 @@ def test_expand_response_files(tmp_path: Path) -> None:
rsp = tmp_path / "flags.rsp"
rsp.write_text("-DFROM_RSP -I/rsp/inc")
tokens = idedata.expand_response_files(
tokens = idedata._expand_response_files(
["g++", f"@{rsp.name}", "-c", "x.cpp"], tmp_path
)
@@ -196,7 +115,7 @@ def test_expand_response_files(tmp_path: Path) -> None:
def test_expand_response_files_keeps_literal_when_missing(tmp_path: Path) -> None:
"""An unreadable ``@file`` token is kept verbatim rather than dropped."""
tokens = idedata.expand_response_files(["g++", "@nope.rsp"], tmp_path)
tokens = idedata._expand_response_files(["g++", "@nope.rsp"], tmp_path)
assert "@nope.rsp" in tokens
@@ -408,7 +327,7 @@ def test_split_command_preserves_paths_and_unescapes_quotes() -> None:
r"""Backslash paths survive while ``\"`` define-quoting is unescaped."""
command = r"C:\esp\bin\riscv32-esp-elf-g++.exe -DVER=\"1.2.3\" -IC:/inc/a -c x.cpp"
tokens = idedata.split_command(command)
tokens = idedata._split_command(command)
assert tokens[0] == r"C:\esp\bin\riscv32-esp-elf-g++.exe"
assert '-DVER="1.2.3"' in tokens
@@ -422,8 +341,8 @@ def test_split_command_empty_returns_empty() -> None:
Guards against ``CommandLineToArgvW("")`` returning the current process name
instead of an empty list.
"""
assert idedata.split_command("") == []
assert idedata.split_command(" ") == []
assert idedata._split_command("") == []
assert idedata._split_command(" ") == []
@pytest.mark.skipif(os.name != "nt", reason="Windows argv tokenization")
@@ -584,9 +503,9 @@ def test_load_or_build_idedata_rebuilds_non_dict_cache(tmp_path: Path) -> None:
def test_is_launcher_matches_only_known_launchers() -> None:
"""Compilers of any shape pass; only the closed launcher set matches."""
for token in ("/t/g++-13", "gcc-8.4.0", "clang++-17", "armcc", "icx", "cc"):
assert not idedata.is_launcher(token)
assert not idedata._is_launcher(token)
for token in ("/opt/homebrew/bin/ccache", "CCACHE.EXE", "distcc", "sccache"):
assert idedata.is_launcher(token)
assert idedata._is_launcher(token)
def test_load_or_build_idedata_corrupted_cache_is_logged(
-381
View File
@@ -1,381 +0,0 @@
"""Tests for esphome.build_helpers.pch."""
from __future__ import annotations
import os
from pathlib import Path
from unittest.mock import patch
import pytest
from esphome.build_helpers import pch
def _write(src_dir: Path, name: str, content: str) -> None:
path = src_dir / name
path.parent.mkdir(parents=True, exist_ok=True)
path.write_text(content)
@pytest.mark.parametrize(
("value", "expected"),
[
(None, True),
("1", True),
("0", False),
("false", False),
("", False),
],
)
def test_pch_enabled(value: str | None, expected: bool) -> None:
env = {} if value is None else {"ESPHOME_PCH_ENABLE": value}
with patch.dict(os.environ, env, clear=True):
assert pch.pch_enabled() is expected
def test_ccache_pch_env_empty_until_emitted() -> None:
"""No sloppiness relaxation for a build that skipped the pch."""
with patch.dict(os.environ, {}, clear=True):
assert pch.ccache_pch_env() == {}
def test_ccache_pch_env_enabled() -> None:
pch.mark_pch_emitted()
with patch.dict(os.environ, {}, clear=True):
env = pch.ccache_pch_env()
assert env["CCACHE_SLOPPINESS"] == "pch_defines,time_macros"
assert env["CCACHE_PCH_EXTSUM"] == "true"
def test_ccache_pch_env_disabled() -> None:
with patch.dict(os.environ, {"ESPHOME_PCH_ENABLE": "0"}, clear=True):
assert pch.ccache_pch_env() == {}
def test_ccache_pch_env_token_check_is_membership_not_substring(
caplog: pytest.LogCaptureFixture,
) -> None:
"""A token merely containing ours must not suppress the union."""
pch.mark_pch_emitted()
with patch.dict(os.environ, {"CCACHE_SLOPPINESS": "pch_defines_extra"}, clear=True):
env = pch.ccache_pch_env()
assert env["CCACHE_SLOPPINESS"] == "pch_defines_extra,pch_defines,time_macros"
def test_ccache_pch_env_unions_user_sloppiness(
caplog: pytest.LogCaptureFixture,
) -> None:
"""Without pch_defines/time_macros ccache declines every pch-consuming
compile, so missing tokens are unioned onto the user's value."""
pch.mark_pch_emitted()
with patch.dict(os.environ, {"CCACHE_SLOPPINESS": "locale"}, clear=True):
env = pch.ccache_pch_env()
assert env["CCACHE_SLOPPINESS"] == "locale,pch_defines,time_macros"
assert env["CCACHE_PCH_EXTSUM"] == "true"
assert "Adding pch_defines,time_macros" in caplog.text
caplog.clear()
with patch.dict(
os.environ, {"CCACHE_SLOPPINESS": "pch_defines,time_macros"}, clear=True
):
env = pch.ccache_pch_env()
assert "CCACHE_SLOPPINESS" not in env
assert not caplog.records
def test_pch_header_text_preserves_order() -> None:
text = pch.pch_header_text(["b.h", "a.h"])
assert text == '#include "b.h"\n#include "a.h"\n'
def test_include_closure_resolves_relative_and_root(tmp_path: Path) -> None:
"""Sibling includes resolve against the includer's directory first,
full paths against the src root; unresolvable names end the walk."""
_write(tmp_path, "esphome/components/x/a.h", '#include "b.h"\n')
_write(
tmp_path,
"esphome/components/x/b.h",
'#include "esphome/core/deep.h"\n#include <system.h>\n#include "missing.h"\n',
)
_write(tmp_path, "esphome/core/deep.h", "")
closure = pch._include_closure(tmp_path, ["esphome/components/x/a.h"])
assert sorted(closure) == [
"esphome/components/x/a.h",
"esphome/components/x/b.h",
"esphome/core/deep.h",
]
def test_include_closure_handles_cycles(tmp_path: Path) -> None:
_write(tmp_path, "a.h", '#include "b.h"\n')
_write(tmp_path, "b.h", '#include "a.h"\n')
assert sorted(pch._include_closure(tmp_path, ["a.h"])) == ["a.h", "b.h"]
def test_include_closure_blocks_parent_escape(tmp_path: Path) -> None:
_write(tmp_path / "src", "a.h", '#include "../outside.h"\n')
(tmp_path / "outside.h").write_text("")
assert sorted(pch._include_closure(tmp_path / "src", ["a.h"])) == ["a.h"]
def test_pch_checksum_tracks_closure_content(tmp_path: Path) -> None:
"""A transitive header edit or an extra-identity change must change the
digest; unrelated files must not."""
_write(tmp_path, "root.h", '#include "nested.h"\n')
_write(tmp_path, "nested.h", "int a;\n")
_write(tmp_path, "unrelated.h", "int u;\n")
base = pch.pch_checksum(tmp_path, ["root.h"], ["id"])
assert base == pch.pch_checksum(tmp_path, ["root.h"], ["id"])
assert base != pch.pch_checksum(tmp_path, ["root.h"], ["other-id"])
_write(tmp_path, "unrelated.h", "int changed;\n")
assert base == pch.pch_checksum(tmp_path, ["root.h"], ["id"])
_write(tmp_path, "nested.h", "int b;\n")
assert base != pch.pch_checksum(tmp_path, ["root.h"], ["id"])
@pytest.mark.skipif(
os.name == "nt" or os.geteuid() == 0, reason="chmod is ineffective here"
)
def test_include_closure_fails_closed_on_unreadable(
tmp_path: Path, caplog: pytest.LogCaptureFixture
) -> None:
"""A marker would truncate the transitive walk; the OSError propagates
so callers compile without a pch."""
_write(tmp_path, "a.h", '#include "locked.h"\n')
locked = tmp_path / "locked.h"
locked.write_text("")
locked.chmod(0)
try:
with pytest.raises(OSError):
pch._include_closure(tmp_path, ["a.h"])
finally:
locked.chmod(0o644)
assert "Could not read locked.h" in caplog.text
def test_pch_extra_scripts_gated(monkeypatch: pytest.MonkeyPatch) -> None:
with patch.dict(os.environ, {}, clear=True):
assert pch.pch_extra_scripts() == ["post:pch.py"]
monkeypatch.setenv("ESPHOME_PCH_ENABLE", "0")
assert pch.pch_extra_scripts() == []
def test_include_closure_raises_when_identity_unknown(
caplog: pytest.LogCaptureFixture,
) -> None:
"""An unreadable header propagates; callers compile without a pch."""
class _BadFile:
def stat(self): # noqa: ANN202 -- regular-file mode only
import os
import stat as stat_mod
return os.stat_result((stat_mod.S_IFREG | 0o644,) + (0,) * 9)
def read_bytes(self) -> bytes:
raise OSError("read failed")
class _FakeSrcDir:
def __truediv__(self, rel: str) -> _BadFile:
return _BadFile()
with pytest.raises(OSError, match="read failed"):
pch._include_closure(_FakeSrcDir(), ["a.h"])
assert "Could not read a.h" in caplog.text
def test_include_closure_survives_non_utf8_include_name(tmp_path: Path) -> None:
"""A non-UTF-8 quoted include must not abort the build; it simply does
not resolve and ends the walk."""
(tmp_path / "a.h").write_bytes(b'#include "bad\xff.h"\n#include "b.h"\n')
(tmp_path / "b.h").write_text("")
closure = pch._include_closure(tmp_path, ["a.h"])
assert set(closure) == {"a.h", "b.h"}
def test_pch_checksum_survives_surrogate_extra(tmp_path: Path) -> None:
"""Install paths from non-UTF-8 filesystems carry surrogates; hashing
them must not raise past the caller's identity-unknown guard."""
assert pch.pch_checksum(tmp_path, [], ["/opt/bad\udcff/framework"])
def test_include_closure_walks_angle_includes_under_src(tmp_path: Path) -> None:
"""An angle include resolving under src/ must enter the digest; one
that does not simply ends the walk."""
_write(tmp_path, "a.h", "#include <local.h>\n#include <Arduino.h>\n")
(tmp_path / "local.h").write_text("")
closure = pch._include_closure(tmp_path, ["a.h"])
assert set(closure) == {"a.h", "local.h"}
def test_ccache_pch_env_warns_on_falsy_extsum(
caplog: pytest.LogCaptureFixture,
) -> None:
"""A user CCACHE_PCH_EXTSUM=false makes ccache hash the .gch bytes."""
pch.mark_pch_emitted()
with patch.dict(os.environ, {"CCACHE_PCH_EXTSUM": "false"}, clear=True):
env = pch.ccache_pch_env()
assert "CCACHE_PCH_EXTSUM" not in env
assert "disables pch caching" in caplog.text
@pytest.mark.parametrize(
("value", "expected"),
[(None, False), ("0", False), ("1", True), ("true", True)],
)
def test_pch_strict(
value: str | None, expected: bool, monkeypatch: pytest.MonkeyPatch
) -> None:
if value is None:
monkeypatch.delenv("ESPHOME_PCH_STRICT", raising=False)
else:
monkeypatch.setenv("ESPHOME_PCH_STRICT", value)
assert pch.pch_strict() is expected
def test_pch_cmake_consumer_substitutes_target_and_sources(
monkeypatch: pytest.MonkeyPatch,
) -> None:
monkeypatch.delenv("ESPHOME_PCH_ENABLE", raising=False)
monkeypatch.delenv("ESPHOME_PCH_STRICT", raising=False)
block = pch.pch_cmake_consumer("app", "${APP_SOURCES}")
assert "target_compile_options(app PRIVATE" in block
assert '"$<$<COMPILE_LANGUAGE:CXX>:-Winvalid-pch>"' in block
assert "-Wno-error=invalid-pch" in block
assert '"$<$<COMPILE_LANGUAGE:CXX>:esphome_pch.h>"' in block
assert "set_source_files_properties(${APP_SOURCES} PROPERTIES" in block
assert 'OBJECT_DEPENDS "${CMAKE_BINARY_DIR}/esphome_pch.h"' in block
# Placeholder guard: survives a build-system-side pristine wipe
assert 'file(TOUCH "${CMAKE_BINARY_DIR}/esphome_pch.h")' in block
def test_pch_cmake_consumer_strict_escalates(
monkeypatch: pytest.MonkeyPatch,
) -> None:
monkeypatch.delenv("ESPHOME_PCH_ENABLE", raising=False)
monkeypatch.setenv("ESPHOME_PCH_STRICT", "1")
assert "-Werror=invalid-pch" in pch.pch_cmake_consumer("app", "${APP_SOURCES}")
def test_pch_cmake_consumer_empty_when_disabled(
monkeypatch: pytest.MonkeyPatch,
) -> None:
monkeypatch.setenv("ESPHOME_PCH_ENABLE", "0")
assert pch.pch_cmake_consumer("app", "${APP_SOURCES}") == ""
def test_guarded_prepare_logs_placeholder_failure(
tmp_path: Path, monkeypatch: pytest.MonkeyPatch, caplog: pytest.LogCaptureFixture
) -> None:
"""A failed placeholder touch must be traceable, not silent."""
monkeypatch.delenv("ESPHOME_PCH_STRICT", raising=False)
def boom() -> None:
raise RuntimeError("boom")
# Missing build dir: the touch raises and only warns
pch.guarded_prepare(tmp_path / "missing", boom)
assert "Could not create the pch placeholder" in caplog.text
def test_pch_degraded_raises_only_in_strict(
monkeypatch: pytest.MonkeyPatch,
) -> None:
from esphome.core import EsphomeError
monkeypatch.delenv("ESPHOME_PCH_STRICT", raising=False)
pch.pch_degraded("reason")
monkeypatch.setenv("ESPHOME_PCH_STRICT", "1")
with pytest.raises(EsphomeError, match="reason"):
pch.pch_degraded("reason")
def test_pch_extra_scripts_strict_raises_when_disabled(
monkeypatch: pytest.MonkeyPatch,
) -> None:
from esphome.core import EsphomeError
monkeypatch.setenv("ESPHOME_PCH_ENABLE", "0")
monkeypatch.setenv("ESPHOME_PCH_STRICT", "1")
with pytest.raises(EsphomeError, match="disabled"):
pch.pch_extra_scripts()
def test_pch_strict_rejects_unrecognized_values(
monkeypatch: pytest.MonkeyPatch,
) -> None:
"""A typo must not silently disable the gate."""
from esphome.core import EsphomeError
monkeypatch.setenv("ESPHOME_PCH_STRICT", "yolo")
with pytest.raises(EsphomeError, match="Unrecognized ESPHOME_PCH_STRICT"):
pch.pch_strict()
def test_discard_pch_raises_when_gch_survives(
tmp_path: Path, monkeypatch: pytest.MonkeyPatch
) -> None:
"""A .gch an unlink failure leaves behind would be consumed silently."""
from pathlib import Path as _P
from esphome.core import EsphomeError
(tmp_path / "esphome_pch.h").write_text("")
gch = tmp_path / "esphome_pch.h.gch"
gch.write_bytes(b"gch")
real_unlink = _P.unlink
def failing_unlink(self, missing_ok=False):
if self.name.endswith(".gch"):
raise OSError("readonly")
return real_unlink(self, missing_ok=missing_ok)
monkeypatch.setattr(_P, "unlink", failing_unlink)
with pytest.raises(EsphomeError, match="Could not discard"):
pch.discard_pch(tmp_path)
def test_discard_pch_raises_when_sum_survives(
tmp_path: Path, monkeypatch: pytest.MonkeyPatch
) -> None:
"""The .sum is ccache's pch identity; one that survives is as unsafe
as a surviving .gch."""
from pathlib import Path as _P
from esphome.core import EsphomeError
(tmp_path / "esphome_pch.h").write_text("")
(tmp_path / "esphome_pch.h.gch").write_bytes(b"gch")
(tmp_path / "esphome_pch.h.gch.sum").write_text("x")
real_unlink = _P.unlink
def failing_unlink(self, missing_ok=False):
if self.name.endswith(".sum"):
raise OSError("readonly")
return real_unlink(self, missing_ok=missing_ok)
monkeypatch.setattr(_P, "unlink", failing_unlink)
with pytest.raises(EsphomeError, match="Could not discard"):
pch.discard_pch(tmp_path)
def test_discard_pch_warns_when_file_vanished_concurrently(
tmp_path: Path,
monkeypatch: pytest.MonkeyPatch,
caplog: pytest.LogCaptureFixture,
) -> None:
"""An unlink error on a file that is nonetheless gone only warns."""
from pathlib import Path as _P
(tmp_path / "esphome_pch.h").write_text("")
(tmp_path / "esphome_pch.h.gch").write_bytes(b"gch")
real_unlink = _P.unlink
def racing_unlink(self, missing_ok=False):
if self.name.endswith(".sum"):
# Racer removed it, then our unlink errored
raise OSError("stale handle")
return real_unlink(self, missing_ok=missing_ok)
monkeypatch.setattr(_P, "unlink", racing_unlink)
pch.discard_pch(tmp_path)
assert "Could not discard the pch sidecars" in caplog.text
@@ -1,43 +0,0 @@
"""The pch script must reach PlatformIO builds only; the native ESP-IDF
toolchain has its own pch flow in build_gen/espidf.py."""
from pathlib import Path
from unittest.mock import patch
import pytest
from esphome.components import esp32
from esphome.const import Toolchain
from esphome.core import CORE
@pytest.mark.parametrize(
("toolchain", "copied"),
[(Toolchain.PLATFORMIO, True), (Toolchain.ESP_IDF, False)],
)
def test_copy_files_gates_pch_script_on_toolchain(
toolchain: Toolchain, copied: bool, tmp_path: Path
) -> None:
CORE.toolchain = toolchain
CORE.build_path = tmp_path
from esphome.components.esp32 import (
KEY_ESP32,
KEY_EXTRA_BUILD_FILES,
KEY_FLASH_SIZE,
)
with (
patch.object(esp32, "_write_sdkconfig") as write_sdkconfig,
patch.object(esp32, "_write_idf_component_yml"),
patch.object(esp32, "copy_pch_script") as copy_script,
patch.object(esp32, "write_file_if_changed"),
patch.object(esp32, "get_partition_csv"),
patch.dict(
CORE.data,
{KEY_ESP32: {KEY_EXTRA_BUILD_FILES: {}, KEY_FLASH_SIZE: "4MB"}},
),
):
esp32.copy_files()
# Proves execution reached (and passed) the gate on both rows
assert write_sdkconfig.called
assert copy_script.called is copied
@@ -1,191 +0,0 @@
"""Tests for the native (non-PlatformIO) toolchain config validation."""
from __future__ import annotations
from collections.abc import Generator
from pathlib import Path
from types import SimpleNamespace
from unittest.mock import patch
import pytest
from esphome.components import esp8266
from esphome.components.esp8266 import (
ARDUINO_FRAMEWORK_SCHEMA,
_resolve_toolchain,
_validate_native_toolchain,
)
import esphome.config_validation as cv
from esphome.const import (
CONF_BOARD,
CONF_FRAMEWORK,
CONF_PLATFORM_VERSION,
CONF_SOURCE,
CONF_TOOLCHAIN,
CONF_VERSION,
Toolchain,
)
from esphome.core import CORE, EsphomeError
from esphome.types import ConfigType
@pytest.fixture(autouse=True)
def _arduino_toolchain() -> Generator[None]:
# The suite-wide reset_core fixture clears CORE.toolchain after each test
CORE.toolchain = Toolchain.ARDUINO
esp8266._DECODE_WARNED_AT.clear()
yield
esp8266._DECODE_WARNED_AT.clear()
def _config(
board: str = "nodemcuv2",
version: str = "3.1.2",
source: str | None = None,
platform_version: str | None = None,
) -> ConfigType:
framework: dict[str, str] = {CONF_VERSION: version}
if source is not None:
framework[CONF_SOURCE] = source
if platform_version is not None:
framework[CONF_PLATFORM_VERSION] = platform_version
# The real schema fills the source/platform_version defaults, so these
# tests validate against what config validation actually emits
return {
CONF_FRAMEWORK: ARDUINO_FRAMEWORK_SCHEMA(framework),
CONF_BOARD: board,
}
def test_valid_config_passes() -> None:
config = _config()
assert _validate_native_toolchain(config) is config
def test_platformio_toolchain_skips_checks() -> None:
CORE.toolchain = Toolchain.PLATFORMIO
config = _config(board="not_a_board", version="2.7.4")
assert _validate_native_toolchain(config) is config
def test_version_below_floor_rejected() -> None:
# 3.1.0 has no registry package, so the native floor is 3.1.1
with pytest.raises(cv.Invalid, match="3.1.1 or newer"):
_validate_native_toolchain(_config(version="3.1.0"))
def test_version_at_floor_accepted() -> None:
_validate_native_toolchain(_config(version="3.1.1"))
def test_custom_platform_version_warns_and_is_dropped(
caplog: pytest.LogCaptureFixture,
) -> None:
config = _config(platform_version="platformio/espressif8266@4.0.1")
_validate_native_toolchain(config)
assert "'platform_version' is ignored" in caplog.text
assert CONF_PLATFORM_VERSION not in config[CONF_FRAMEWORK]
def test_default_platform_version_does_not_warn(
caplog: pytest.LogCaptureFixture,
) -> None:
config = _config()
_validate_native_toolchain(config)
assert "'platform_version' is ignored" not in caplog.text
assert CONF_PLATFORM_VERSION not in config[CONF_FRAMEWORK]
def test_custom_source_rejected() -> None:
with pytest.raises(cv.Invalid, match="custom framework source"):
_validate_native_toolchain(
_config(source="https://github.com/esp8266/Arduino.git")
)
def test_unsupported_board_rejected() -> None:
with pytest.raises(cv.Invalid, match="not supported by"):
_validate_native_toolchain(_config(board="not_a_board"))
def test_yaml_toolchain_key_resolves() -> None:
"""The documented `toolchain: arduino` YAML key selects the native path."""
CORE.toolchain = None
_resolve_toolchain({CONF_TOOLCHAIN: Toolchain.ARDUINO})
assert CORE.toolchain == Toolchain.ARDUINO
assert CORE.using_toolchain_arduino
def test_yaml_toolchain_key_defaults_to_platformio() -> None:
CORE.toolchain = None
_resolve_toolchain({})
assert CORE.toolchain == Toolchain.PLATFORMIO
def test_decode_pc_native_missing_tools_warns_once(
tmp_path: Path, caplog: pytest.LogCaptureFixture
) -> None:
"""A stack dump of many addresses produces one missing-tool warning."""
with (
patch(
"esphome.arduino8266.toolchain.get_addr2line_path",
return_value=tmp_path / "missing-addr2line",
),
patch(
"esphome.arduino8266.toolchain.get_elf_path",
return_value=tmp_path / "missing.elf",
),
):
esp8266._decode_pc({}, "40201234")
esp8266._decode_pc({}, "40201238")
assert caplog.text.count("Cannot decode crash addresses") == 1
def test_decode_pc_platformio_missing_tools_warns_once(
caplog: pytest.LogCaptureFixture,
) -> None:
"""The PlatformIO branch reports a missing addr2line/ELF at the same
warning level as the native one; raw undecoded addresses with no
stated reason are undiagnosable at default log level."""
CORE.toolchain = Toolchain.PLATFORMIO
idedata = SimpleNamespace(addr2line_path=None, firmware_elf_path=None)
with patch("esphome.platformio.toolchain.get_idedata", return_value=idedata):
esp8266._decode_pc({}, "40201234")
esp8266._decode_pc({}, "40201238")
assert caplog.text.count("Cannot decode crash addresses") == 1
def test_resolve_toolchain_rejects_unsupported() -> None:
"""ESP8266 rejects a CLI toolchain it cannot serve, like every platform."""
CORE.toolchain = Toolchain.SDK_NRF
with pytest.raises(cv.Invalid, match="Unsupported toolchain 'sdk-nrf'"):
_resolve_toolchain({})
def test_run_compile_platformio_falls_through() -> None:
"""Under toolchain: platformio the hook returns False without touching
the native backend; this is what keeps existing users on PlatformIO."""
CORE.toolchain = Toolchain.PLATFORMIO
with patch("esphome.arduino8266.toolchain.run_compile") as mock_native:
assert esp8266.run_compile(SimpleNamespace(), {}) is False
mock_native.assert_not_called()
def test_run_compile_arduino_failure_raises() -> None:
"""A non-zero native build fails by name instead of returning success."""
CORE.verbose = False
with (
patch("esphome.arduino8266.toolchain.run_compile", return_value=1),
pytest.raises(EsphomeError, match="native build failed"),
):
esp8266.run_compile(SimpleNamespace(), {})
def test_copy_files_native_skips_platformio_scripts(tmp_path: Path) -> None:
"""The native build writes no PlatformIO extra scripts."""
CORE.build_path = tmp_path
esp8266.copy_files()
assert list(tmp_path.iterdir()) == []
@@ -1,322 +0,0 @@
"""nrf52 sdk-nrf pch wiring: the CMake consumer block, the prepare wrapper,
and the two-phase west split in run_compile."""
from collections.abc import Generator
from pathlib import Path
from unittest.mock import Mock, patch
import pytest
from esphome.components import nrf52
from esphome.components.zephyr.const import KEY_BOARD
from esphome.const import KEY_CORE, KEY_FRAMEWORK_VERSION, Toolchain
from esphome.core import CORE, EsphomeError
@pytest.fixture(autouse=True)
def pch_env(monkeypatch: pytest.MonkeyPatch) -> None:
monkeypatch.delenv("ESPHOME_PCH_ENABLE", raising=False)
monkeypatch.delenv("ESPHOME_PCH_STRICT", raising=False)
@pytest.fixture
def build_dir(tmp_path: Path) -> Path:
d = tmp_path / "build" / ".pioenvs" / "livingroom"
d.mkdir(parents=True)
return d
_AUTOCONF_TEXT = "#define CONFIG_GPIO 1\n"
def _write_autoconf(build_dir: Path) -> Path:
autoconf = build_dir / "zephyr" / "include" / "generated" / "zephyr" / "autoconf.h"
autoconf.parent.mkdir(parents=True)
autoconf.write_text(_AUTOCONF_TEXT)
return autoconf
def test_prepare_pch_disabled_discards_and_degrades(
monkeypatch: pytest.MonkeyPatch, build_dir: Path
) -> None:
monkeypatch.setenv("ESPHOME_PCH_ENABLE", "0")
gch = build_dir / "esphome_pch.h.gch"
gch.write_bytes(b"x")
nrf52._prepare_pch(build_dir)
assert not gch.exists()
def test_prepare_pch_disabled_strict_raises(
monkeypatch: pytest.MonkeyPatch, build_dir: Path
) -> None:
monkeypatch.setenv("ESPHOME_PCH_ENABLE", "0")
monkeypatch.setenv("ESPHOME_PCH_STRICT", "1")
with pytest.raises(EsphomeError, match="ESPHOME_PCH_STRICT"):
nrf52._prepare_pch(build_dir)
def test_prepare_pch_missing_autoconf_degrades(
build_dir: Path, caplog: pytest.LogCaptureFixture
) -> None:
with patch.object(nrf52.pch, "prepare_pch") as prepare:
nrf52._prepare_pch(build_dir)
assert not prepare.called
assert "No autoconf.h found" in caplog.text
# The header is written first so OBJECT_DEPENDS stays satisfied
assert (build_dir / "esphome_pch.h").is_file()
def test_prepare_pch_missing_autoconf_strict_raises(
monkeypatch: pytest.MonkeyPatch, build_dir: Path
) -> None:
monkeypatch.setenv("ESPHOME_PCH_STRICT", "1")
with pytest.raises(EsphomeError, match="autoconf.h missing"):
nrf52._prepare_pch(build_dir)
def test_prepare_pch_unreadable_autoconf_fails_closed(
build_dir: Path, caplog: pytest.LogCaptureFixture
) -> None:
# A directory named autoconf.h: read_text raises OSError
autoconf = build_dir / "zephyr" / "include" / "generated" / "autoconf.h"
autoconf.mkdir(parents=True)
with patch.object(nrf52.pch, "prepare_pch") as prepare:
nrf52._prepare_pch(build_dir)
assert not prepare.called
assert "Could not read" in caplog.text
def test_app_build_dir_sysbuild_layout(build_dir: Path) -> None:
app = build_dir / "zephyr"
app.mkdir()
(app / "CMakeCache.txt").write_text("")
assert nrf52._app_build_dir(build_dir) == app
def test_app_build_dir_top_level_layout(build_dir: Path) -> None:
# Non-sysbuild: build_dir/zephyr is the Zephyr output dir, no cache
(build_dir / "zephyr").mkdir()
assert nrf52._app_build_dir(build_dir) == build_dir
def test_app_build_dir_ignores_cache_directory(build_dir: Path) -> None:
(build_dir / "zephyr" / "CMakeCache.txt").mkdir(parents=True)
assert nrf52._app_build_dir(build_dir) == build_dir
def test_app_build_dir_propagates_stat_errors(build_dir: Path) -> None:
# is_file() would swallow this and mislocate the pch
with (
patch.object(Path, "stat", side_effect=PermissionError("denied")),
pytest.raises(PermissionError),
):
nrf52._app_build_dir(build_dir)
def test_prepare_pch_extras_carry_build_identity(build_dir: Path) -> None:
_write_autoconf(build_dir)
with (
patch.dict(CORE.data, {KEY_CORE: {KEY_FRAMEWORK_VERSION: "2.9.2"}}),
patch.object(
nrf52, "zephyr_data", return_value={KEY_BOARD: "adafruit_feather"}
),
patch.object(nrf52, "get_project_compile_flags", return_value=["-Os"]),
patch.object(nrf52.pch, "prepare_pch") as prepare,
):
nrf52._prepare_pch(build_dir)
assert (build_dir / "esphome_pch.h").is_file()
(passed_dir, headers, extras) = prepare.call_args.args
assert passed_dir == build_dir
assert headers == nrf52.PCH_DEFAULT_HEADERS
assert list(extras) == ["2.9.2", "adafruit_feather", _AUTOCONF_TEXT, "-Os"]
def _generate_cmake(tmp_path: Path) -> str:
CORE.config_path = tmp_path / "test.yaml"
CORE.build_path = tmp_path / "build"
CORE.name = "livingroom"
with (
patch(
"esphome.components.zephyr.library.generate_zephyr_modules",
return_value=[],
),
patch.object(nrf52, "get_project_compile_flags", return_value=["-Os"]),
patch.object(nrf52, "get_project_link_flags", return_value=[]),
):
nrf52._generate_cmake_lists()
return (tmp_path / "build" / "zephyr" / "CMakeLists.txt").read_text()
def test_cmake_lists_include_pch_consumer_block(tmp_path: Path) -> None:
# Content contract is pinned by the shared pch_cmake_consumer tests;
# here only that the block reaches the generated CMakeLists
text = _generate_cmake(tmp_path)
assert "target_compile_options(app PRIVATE" in text
assert 'OBJECT_DEPENDS "${CMAKE_BINARY_DIR}/esphome_pch.h"' in text
def test_cmake_lists_pch_block_disabled(
monkeypatch: pytest.MonkeyPatch, tmp_path: Path
) -> None:
monkeypatch.setenv("ESPHOME_PCH_ENABLE", "0")
assert "esphome_pch.h" not in _generate_cmake(tmp_path)
CompileCtx = tuple[Mock, Mock, Path]
class TestRunCompilePhases:
"""The pch pre-build block in run_compile: header write, conditional
cmake-only phase, and the never-abort-the-build exception contract."""
@pytest.fixture
def compile_ctx(self, tmp_path: Path) -> Generator[CompileCtx, None, None]:
CORE.config_path = tmp_path / "test.yaml"
CORE.build_path = tmp_path / "build"
CORE.name = "livingroom"
CORE.toolchain = Toolchain.SDK_NRF
with (
patch.object(nrf52, "check_and_install"),
patch.object(nrf52, "_generate_cmake_lists", return_value=False),
patch.object(
nrf52,
"get_build_paths",
return_value={
"python_executable": "python3",
"framework_path": tmp_path,
},
),
patch.object(nrf52, "get_build_env", return_value={}),
patch.object(nrf52, "zephyr_data", return_value={KEY_BOARD: "board"}),
patch.object(nrf52, "run_command_ok") as run_cmd,
patch.object(nrf52, "_prepare_pch") as prepare,
):
yield run_cmd, prepare, CORE.relative_pioenvs_path(CORE.name)
def _run(self) -> None:
nrf52.run_compile(None, {})
def test_missing_db_runs_cmake_phase(self, compile_ctx: CompileCtx) -> None:
run_cmd, prepare, build_dir = compile_ctx
# cmake-only ok, generated headers ok, final build fails
results = iter([True, True, False])
def west(cmd, **kwargs):
# Phase 1 configures the sysbuild app domain
app = build_dir / "zephyr"
app.mkdir(parents=True, exist_ok=True)
(app / "CMakeCache.txt").write_text("")
return next(results)
run_cmd.side_effect = west
with pytest.raises(EsphomeError, match="nRF52 native build failed"):
self._run()
assert "--cmake-only" in run_cmd.call_args_list[0].args[0]
# Generated syscall headers are built in the app domain pre-pch
headers_cmd = run_cmd.call_args_list[1].args[0]
assert headers_cmd[:2] == ["cmake", "--build"]
assert str(build_dir / "zephyr") in headers_cmd
assert "zephyr_generated_headers" in headers_cmd
assert "--cmake-only" not in run_cmd.call_args_list[2].args[0]
# The pch is prepared in the app domain dir, not the sysbuild root
assert prepare.call_args.args[0] == build_dir / "zephyr"
def test_generated_headers_failure_degrades(
self, compile_ctx, caplog: pytest.LogCaptureFixture
) -> None:
run_cmd, prepare, _ = compile_ctx
# headers target fails, the real build still runs (and fails here)
run_cmd.side_effect = [True, False, False]
with pytest.raises(EsphomeError, match="nRF52 native build failed"):
self._run()
assert "Zephyr header generation failed" in caplog.text
# The doomed .gch compile is skipped: it would latch .gch.failed
assert not prepare.called
def test_generated_headers_failure_strict_raises(
self, monkeypatch: pytest.MonkeyPatch, compile_ctx: CompileCtx
) -> None:
monkeypatch.setenv("ESPHOME_PCH_STRICT", "1")
run_cmd, prepare, _ = compile_ctx
run_cmd.side_effect = [True, False]
with pytest.raises(EsphomeError, match="ESPHOME_PCH_STRICT"):
self._run()
assert not prepare.called
def test_cmake_phase_failure_raises(self, compile_ctx: CompileCtx) -> None:
run_cmd, prepare, _ = compile_ctx
run_cmd.side_effect = [False]
with pytest.raises(EsphomeError, match="configure failed"):
self._run()
assert not prepare.called
def test_ccache_pch_env_reaches_west(self, compile_ctx: CompileCtx) -> None:
run_cmd, _, _ = compile_ctx
run_cmd.side_effect = [False]
# clear=True also drops ambient CCACHE_*/ESPHOME_PCH_* overrides
with (
patch.dict("os.environ", {}, clear=True),
pytest.raises(EsphomeError, match="configure failed"),
):
self._run()
env = run_cmd.call_args.kwargs["env"]
assert env["CCACHE_PCH_EXTSUM"] == "true"
assert env["CCACHE_SLOPPINESS"] == "pch_defines,time_macros"
@pytest.mark.parametrize("sysbuild", [False, True])
def test_settled_db_skips_cmake_phase(
self, sysbuild: bool, compile_ctx: CompileCtx
) -> None:
run_cmd, prepare, build_dir = compile_ctx
app = build_dir / "zephyr" if sysbuild else build_dir
app.mkdir(parents=True)
# A present top-level cache keeps the pristine wipe from dropping
# the DB; the app-dir cache is the sysbuild layout marker
(build_dir / "CMakeCache.txt").write_text("")
(app / "CMakeCache.txt").write_text("")
(app / "compile_commands.json").write_text("[]")
run_cmd.side_effect = [False]
with pytest.raises(EsphomeError, match="nRF52 native build failed"):
self._run()
assert run_cmd.call_count == 1
assert "--cmake-only" not in run_cmd.call_args.args[0]
assert prepare.call_args.args[0] == app
def test_disabled_skips_header_and_cmake_phase(
self, monkeypatch: pytest.MonkeyPatch, compile_ctx: CompileCtx
) -> None:
monkeypatch.setenv("ESPHOME_PCH_ENABLE", "0")
run_cmd, prepare, build_dir = compile_ctx
run_cmd.side_effect = [False]
with pytest.raises(EsphomeError, match="nRF52 native build failed"):
self._run()
assert run_cmd.call_count == 1
assert not (build_dir / "esphome_pch.h").exists()
# The wrapper still runs: it discards stale sidecars and feeds strict
assert prepare.called
def test_prepare_failure_never_aborts_the_build(
self, compile_ctx, caplog: pytest.LogCaptureFixture
) -> None:
run_cmd, prepare, build_dir = compile_ctx
build_dir.mkdir(parents=True)
# Keep the pristine wipe from dropping the dir the fallback touches
(build_dir / "CMakeCache.txt").write_text("")
prepare.side_effect = RuntimeError("boom")
run_cmd.side_effect = [True, True, False]
with pytest.raises(EsphomeError, match="nRF52 native build failed"):
self._run()
assert run_cmd.call_count == 3
assert "Precompiled header setup failed" in caplog.text
# The fallback still satisfies OBJECT_DEPENDS
assert (build_dir / "esphome_pch.h").is_file()
def test_prepare_failure_strict_raises(
self, monkeypatch: pytest.MonkeyPatch, compile_ctx: CompileCtx
) -> None:
monkeypatch.setenv("ESPHOME_PCH_STRICT", "1")
run_cmd, prepare, _ = compile_ctx
prepare.side_effect = RuntimeError("boom")
run_cmd.side_effect = [True, True]
with pytest.raises(RuntimeError, match="boom"):
self._run()
+9
View File
@@ -1401,3 +1401,12 @@ async def test_add_platformio_options_native_arduino(
assert "board_build.ldscript is ignored" in caplog.text
assert "'arduino' toolchain" in caplog.text
assert "upload_speed" not in caplog.text
def test_esp8266_rejects_unsupported_cli_toolchain() -> None:
"""Until the native backend lands, ESP8266 serves only PlatformIO."""
from esphome.components.esp8266 import CONFIG_SCHEMA
CORE.toolchain = Toolchain.ARDUINO
with pytest.raises(cv.Invalid, match="Unsupported toolchain 'arduino'"):
CONFIG_SCHEMA({"board": "nodemcuv2"})
@@ -1,8 +1,7 @@
"""Minimal idf_tools stand-in for the espidf helper-script tests."""
"""Minimal idf_tools stand-in for get_tool_downloads.py tests."""
from collections.abc import Iterable
import os
import pathlib
CURRENT_PLATFORM = "linux-amd64"
TOOLS_FILE = "tools/tools.json"
@@ -55,7 +54,6 @@ class _Tool:
installed: Iterable[str] = (),
broken: bool = False,
) -> None:
self.name = "" # filled in from the _TOOLS key below
self.versions = versions
self._recommended = recommended
self.versions_installed = list(installed)
@@ -71,19 +69,6 @@ class _Tool:
if self._broken:
raise ToolBinaryError("broken binary")
def get_path_for_version(self, version: str) -> str:
return str(pathlib.Path(g.idf_tools_path) / "tools" / self.name / version)
def install(self, version: str) -> None:
dest = pathlib.Path(self.get_path_for_version(version))
dest.mkdir(exist_ok=True, parents=True)
if self.name in os.environ.get("TEST_FAIL_INSTALL", "").split(","):
# Fail mid-install like a torn unpack: the partial dir is left
# behind and check_binary_valid's failure path exits the process
(dest / ".partial").write_text("torn", encoding="utf-8")
raise SystemExit(1)
(dest / ".installed").write_text("ok", encoding="utf-8")
_TOOLS = {
"cmake": _Tool(
@@ -112,17 +97,8 @@ _TOOLS = {
"no-download-tool": _Tool({"4.0": _Version(None)}, "4.0"),
}
for _name, _tool in _TOOLS.items():
_tool.name = _name
def load_tools_info() -> dict[str, _Tool]:
# Test hook: strip verification metadata from the named tools
for name in os.environ.get("TEST_NO_SHA", "").split(","):
if (tool := _TOOLS.get(name)) is not None:
for version in tool.versions.values():
if (download := version.get_download_for_platform("")) is not None:
download.sha256 = ""
return _TOOLS
@@ -1,65 +0,0 @@
"""Run the native-toolchain serial-upload path and report heavy modules.
Executed as a subprocess by test_lazy_imports.py: heavy module names come
in on argv, the ones found in sys.modules afterwards go out on stdout.
``upload_using_esptool`` dispatches native toolchains through a
toolchain-keyed table; if someone routes it back through the platform
component packages (esp32 or esp8266), this reports the leak.
"""
import os
from pathlib import Path
import sys
import tempfile
from unittest.mock import patch
from _leak_report import print_leaked_modules
from esphome.__main__ import upload_using_esptool
from esphome.const import (
CONF_ESPHOME,
KEY_CORE,
KEY_ESP32,
KEY_TARGET_PLATFORM,
KEY_VARIANT,
Toolchain,
)
from esphome.core import CORE
# An ambient ESPHOME_USE_SUBPROCESS would route past the patched
# run_external_command into run_external_process and confuse the checks.
os.environ.pop("ESPHOME_USE_SUBPROCESS", None)
config = {CONF_ESPHOME: {"platformio_options": {}}}
with tempfile.TemporaryDirectory() as build_dir:
CORE.name = "leaktest"
CORE.build_path = build_dir
for platform, toolchain, backend in (
("esp8266", Toolchain.ARDUINO, "esphome.arduino8266.toolchain"),
("esp32", Toolchain.ESP_IDF, "esphome.espidf.toolchain"),
):
CORE.data[KEY_CORE] = {KEY_TARGET_PLATFORM: platform}
if platform == "esp32":
CORE.data[KEY_ESP32] = {KEY_VARIANT: "ESP32S3"}
CORE.toolchain = toolchain
import importlib
image = importlib.import_module(backend).get_factory_firmware_path()
image.parent.mkdir(parents=True, exist_ok=True)
image.write_bytes(b"\x00")
with patch("esphome.__main__.run_external_command", return_value=0) as mock_run:
rc = upload_using_esptool(config, "/dev/ttyUSB0", None, None)
# Fail loudly if the upload path stopped doing its work; otherwise
# an empty leak list could just mean nothing ran.
if rc != 0:
sys.exit(f"upload_using_esptool({platform}) returned {rc}")
cmd = list(mock_run.call_args[0][1:])
if str(image) not in [str(Path(c)) for c in cmd]:
sys.exit(f"native factory image did not reach esptool: {cmd}")
print_leaked_modules()
+18 -19
View File
@@ -62,7 +62,7 @@ def test_tools_path_default_and_prefix(tmp_path: Path) -> None:
def test_check_and_install_returns_paths(tmp_path: Path) -> None:
with (
patch.dict(os.environ, {"ESPHOME_ARDUINO8266_PREFIX": str(tmp_path)}),
patch.object(framework, "install_packages") as mock_install,
patch.object(framework, "install_package") as mock_install,
patch.object(framework, "prefetch_packages") as mock_prefetch,
patch.object(framework, "find_ninja", return_value=tmp_path / "ninja"),
):
@@ -70,27 +70,26 @@ def test_check_and_install_returns_paths(tmp_path: Path) -> None:
assert paths.framework == tmp_path / "frameworks" / "3.30102.0"
assert paths.toolchain == tmp_path / "toolchains" / framework.TOOLCHAIN_VERSION
assert paths.ninja == tmp_path / "ninja"
assert mock_install.call_count == 2
# Full argument pinning: a copy-paste swap between the two near-identical
# specs (mirrors, destination) must not stay green
assert mock_install.call_args.args == (
(
(
framework.FRAMEWORK_PACKAGE,
"3.30102.0",
tmp_path / "frameworks" / "3.30102.0",
framework.ESPHOME_ARDUINO8266_FRAMEWORK_MIRRORS,
("cores/esp8266", "tools/sdk", "libraries"),
),
(
framework.TOOLCHAIN_PACKAGE,
framework.TOOLCHAIN_VERSION,
tmp_path / "toolchains" / framework.TOOLCHAIN_VERSION,
framework.ESPHOME_ARDUINO8266_TOOLCHAIN_MIRRORS,
("bin", "xtensa-lx106-elf"),
),
),
# calls (mirrors, destination) must not stay green
fw_call, tc_call = mock_install.call_args_list
assert fw_call.args == (
framework.FRAMEWORK_PACKAGE,
"3.30102.0",
tmp_path / "frameworks" / "3.30102.0",
framework.ESPHOME_ARDUINO8266_FRAMEWORK_MIRRORS,
tmp_path / "downloads",
)
assert fw_call.kwargs["expect"] == ("cores/esp8266", "tools/sdk", "libraries")
assert tc_call.args == (
framework.TOOLCHAIN_PACKAGE,
framework.TOOLCHAIN_VERSION,
tmp_path / "toolchains" / framework.TOOLCHAIN_VERSION,
framework.ESPHOME_ARDUINO8266_TOOLCHAIN_MIRRORS,
tmp_path / "downloads",
)
assert tc_call.kwargs["expect"] == ("bin", "xtensa-lx106-elf")
# The prefetch sees the same package specs as the installs
assert mock_prefetch.call_args.args == (
[
@@ -1,657 +0,0 @@
"""Tests for esphome.arduino8266.toolchain (the ninja build driver)."""
from __future__ import annotations
import os
from pathlib import Path
from unittest.mock import MagicMock, patch
import pytest
from esphome.arduino8266 import framework, toolchain
from esphome.build_helpers.pch import mark_pch_emitted
import esphome.config_validation as cv
from esphome.const import (
CONF_COMPILE_PROCESS_LIMIT,
CONF_ESPHOME,
KEY_CORE,
KEY_FRAMEWORK_VERSION,
)
from esphome.core import CORE, EsphomeError
_SIZE_OUTPUT = """\
firmware.elf :
section size addr
.data 1924 1073643520
.noinit 56 1073645444
.text 496 1074790400
.irom0.text 342804 1075843088
.text1 27489 1074790896
.rodata 2588 1073645504
.bss 26504 1073648096
Total 401861
"""
@pytest.fixture(autouse=True)
def _setup_core(tmp_path: Path) -> None:
CORE.name = "test8266"
CORE.config_path = tmp_path / "test8266.yaml"
CORE.build_path = tmp_path
CORE.data[KEY_CORE] = {KEY_FRAMEWORK_VERSION: cv.Version(3, 1, 2)}
# run_compile verifies the produced artifacts; give every test a build
# that "produced" them (tests for the guard delete them again). The
# manifest comes first: artifacts must not be older than build.ninja.
build_dir = CORE.relative_pioenvs_path("test8266")
build_dir.mkdir(parents=True, exist_ok=True)
(build_dir / "build.ninja").write_text("# manifest")
for artifact in (
"firmware.elf",
"firmware.bin",
"firmware.factory.bin",
"firmware.ota.bin",
):
(build_dir / artifact).write_bytes(b"")
def _paths(tmp_path: Path) -> framework.InstalledPaths:
return framework.InstalledPaths(
framework=tmp_path / "framework",
toolchain=tmp_path / "toolchain",
ninja=tmp_path / "ninja",
)
def test_path_getters(tmp_path: Path) -> None:
assert toolchain.get_build_dir() == CORE.relative_pioenvs_path("test8266")
assert toolchain.get_elf_path().name == "firmware.elf"
# The framework accessor owns the layout and the Windows suffix
suffix = ".exe" if os.name == "nt" else ""
assert toolchain.get_addr2line_path().name == f"xtensa-lx106-elf-addr2line{suffix}"
assert toolchain.get_objdump_path().name == f"xtensa-lx106-elf-objdump{suffix}"
assert toolchain.get_readelf_path().name == f"xtensa-lx106-elf-readelf{suffix}"
def test_run_compile_build_failure(tmp_path: Path) -> None:
with (
patch.object(framework, "check_and_install", return_value=_paths(tmp_path)),
patch.object(framework, "get_build_env", return_value={}),
patch("esphome.build_gen.arduino8266.write_project"),
patch.object(
toolchain.subprocess, "run", return_value=MagicMock(returncode=2)
) as mock_run,
patch.object(toolchain, "_write_compile_commands") as mock_compdb,
):
assert toolchain.run_compile({CONF_ESPHOME: {}}, verbose=True) == 2
cmd = mock_run.call_args[0][0]
assert "-v" in cmd
# The compile database is generated before the build runs, so a failed
# build cannot leave a stale database behind.
mock_compdb.assert_called_once()
def test_run_compile_success(tmp_path: Path) -> None:
with (
patch.object(framework, "check_and_install", return_value=_paths(tmp_path)),
patch.object(framework, "get_build_env", return_value={}),
# An unchanged manifest is what makes the -n probe run
patch("esphome.build_gen.arduino8266.write_project", return_value=False),
patch.object(
toolchain.subprocess,
"run",
return_value=MagicMock(returncode=0, stdout="", stderr=""),
) as mock_run,
patch.object(toolchain, "_write_compile_commands") as mock_compdb,
patch.object(toolchain, "_print_size_summary") as mock_size,
patch.object(toolchain, "get_idedata") as mock_idedata,
):
rc = toolchain.run_compile(
{CONF_ESPHOME: {CONF_COMPILE_PROCESS_LIMIT: 4}}, verbose=False
)
assert rc == 0
# The -n probe runs first, then the real build (cwd, no -C banner)
ninja_calls = [c for c in mock_run.call_args_list if "ninja" in str(c[0][0][0])]
assert "-n" in ninja_calls[0][0][0]
# Explicit targets: a manifest missing them fails as "unknown target"
assert ninja_calls[0][0][0][-1] == "firmware.ota.bin"
cmd = ninja_calls[1][0][0]
assert cmd[-4:] == ["-j", "4", "firmware.factory.bin", "firmware.ota.bin"]
assert "-C" not in cmd
assert ninja_calls[1][1]["cwd"] is not None
mock_compdb.assert_called_once()
mock_size.assert_called_once()
mock_idedata.assert_called_once()
def test_run_compile_noop_skips_the_build_spawn(tmp_path: Path) -> None:
"""A no-op rebuild stays quiet: the -n probe answers "no work to do"
and the real ninja spawn (and its banner) never happens."""
with (
patch.object(framework, "check_and_install", return_value=_paths(tmp_path)),
patch.object(framework, "get_build_env", return_value={}),
# An unchanged manifest is what makes the -n probe run
patch("esphome.build_gen.arduino8266.write_project", return_value=False),
patch.object(
toolchain.subprocess,
"run",
return_value=MagicMock(
returncode=0, stdout="ninja: no work to do.\n", stderr=""
),
) as mock_run,
patch.object(toolchain, "_write_compile_commands"),
patch.object(toolchain, "_print_size_summary"),
patch.object(toolchain, "get_idedata"),
):
rc = toolchain.run_compile({CONF_ESPHOME: {}}, verbose=False)
assert rc == 0
ninja_calls = [c for c in mock_run.call_args_list if "ninja" in str(c[0][0][0])]
assert len(ninja_calls) == 1
assert "-n" in ninja_calls[0][0][0]
# Explicit targets: a manifest missing them fails as "unknown target"
assert ninja_calls[0][0][0][-1] == "firmware.ota.bin"
def test_run_compile_regenerates_stale_compdb(tmp_path: Path) -> None:
"""An interrupted run can leave build.ninja newer than the compile DB;
mere existence must not skip regeneration."""
build_dir = toolchain.get_build_dir()
build_dir.mkdir(parents=True, exist_ok=True)
(build_dir / "build.ninja").write_text("")
compdb = build_dir / "compile_commands.json"
compdb.write_text("[]")
os.utime(compdb, ((build_dir / "build.ninja").stat().st_mtime - 5,) * 2)
with (
patch.object(framework, "check_and_install", return_value=_paths(tmp_path)),
patch.object(framework, "get_build_env", return_value={}),
patch("esphome.build_gen.arduino8266.write_project", return_value=False),
patch.object(
toolchain.subprocess,
"run",
return_value=MagicMock(
returncode=0, stdout="ninja: no work to do.\n", stderr=""
),
),
patch.object(toolchain, "_write_compile_commands") as mock_compdb,
patch.object(toolchain, "_print_size_summary"),
patch.object(toolchain, "get_idedata"),
):
assert toolchain.run_compile({CONF_ESPHOME: {}}, verbose=False) == 0
mock_compdb.assert_called_once()
def test_run_compile_surfaces_probe_diagnostics(
tmp_path: Path, caplog: pytest.LogCaptureFixture
) -> None:
"""A load-time ninja diagnostic (a generator bug signal) reaches the
user even when the no-work branch skips the real spawn."""
with (
patch.object(framework, "check_and_install", return_value=_paths(tmp_path)),
patch.object(framework, "get_build_env", return_value={}),
# An unchanged manifest is what makes the -n probe run
patch("esphome.build_gen.arduino8266.write_project", return_value=False),
patch.object(
toolchain.subprocess,
"run",
return_value=MagicMock(
returncode=0,
stdout="ninja: no work to do.\n",
stderr="ninja: warning: multiple rules generate x\n",
),
),
patch.object(toolchain, "_write_compile_commands"),
patch.object(toolchain, "_print_size_summary"),
patch.object(toolchain, "get_idedata"),
):
rc = toolchain.run_compile({CONF_ESPHOME: {}}, verbose=False)
assert rc == 0
assert "multiple rules generate x" in caplog.text
def test_run_compile_missing_artifact_fails(
tmp_path: Path, caplog: pytest.LogCaptureFixture
) -> None:
"""A zero ninja exit that produced no firmware must not be a green
build (size summary and idedata only warn)."""
(toolchain.get_build_dir() / "firmware.elf").unlink()
with (
patch.object(framework, "check_and_install", return_value=_paths(tmp_path)),
patch.object(framework, "get_build_env", return_value={}),
patch("esphome.build_gen.arduino8266.write_project"),
patch.object(
toolchain.subprocess,
"run",
return_value=MagicMock(returncode=0, stdout="", stderr=""),
),
patch.object(toolchain, "_write_compile_commands"),
patch.object(toolchain, "_print_size_summary") as mock_size,
patch.object(toolchain, "get_idedata"),
):
rc = toolchain.run_compile({CONF_ESPHOME: {}}, verbose=False)
assert rc == 1
assert "Build produced no" in caplog.text
mock_size.assert_not_called()
def test_run_compile_warns_when_idedata_fails(
tmp_path: Path, caplog: pytest.LogCaptureFixture
) -> None:
"""A failed idedata generation right after a successful build is visible,
not deferred to a misleading error in a later command."""
with (
patch.object(framework, "check_and_install", return_value=_paths(tmp_path)),
patch.object(framework, "get_build_env", return_value={}),
patch("esphome.build_gen.arduino8266.write_project"),
patch.object(
toolchain.subprocess,
"run",
return_value=MagicMock(returncode=0, stdout="", stderr=""),
),
patch.object(toolchain, "_write_compile_commands"),
patch.object(toolchain, "_print_size_summary"),
patch.object(toolchain, "get_idedata", return_value=None),
):
assert toolchain.run_compile({CONF_ESPHOME: {}}, verbose=False) == 0
assert "No idedata was generated for this build" in caplog.text
def test_write_compile_commands(tmp_path: Path) -> None:
build_dir = tmp_path / "build"
build_dir.mkdir()
entries = '[{"file": "a.cpp", "command": "cc"}]\n'
with patch.object(
toolchain.subprocess,
"run",
return_value=MagicMock(returncode=0, stdout=entries),
):
toolchain._write_compile_commands(tmp_path / "ninja", build_dir, {})
assert (build_dir / "compile_commands.json").read_text() == entries
@pytest.mark.parametrize(
("stdout", "match"),
[
("[]\n", "empty compile database"),
# A parse failure names its cause, not the rule-name story
("not json", "unparsable compile database.*not json"),
],
)
def test_write_compile_commands_bad_db_raises(
tmp_path: Path, stdout: str, match: str
) -> None:
"""An empty or unparsable compile database fails the build with its
actual cause and drops any stale database."""
build_dir = tmp_path / "build"
build_dir.mkdir()
(build_dir / "compile_commands.json").write_text("[stale]")
with (
patch.object(
toolchain.subprocess,
"run",
return_value=MagicMock(returncode=0, stdout=stdout),
),
pytest.raises(EsphomeError, match=match),
):
toolchain._write_compile_commands(tmp_path / "ninja", build_dir, {})
assert not (build_dir / "compile_commands.json").exists()
def test_write_compile_commands_failure_removes_stale_db(tmp_path: Path) -> None:
"""A failed compdb run must not leave a stale database behind."""
stale = tmp_path / "compile_commands.json"
stale.write_text("[]")
with (
patch.object(
toolchain.subprocess,
"run",
return_value=MagicMock(returncode=1, stderr="boom"),
),
pytest.raises(EsphomeError, match="compile_commands"),
):
toolchain._write_compile_commands(tmp_path / "ninja", tmp_path, {})
assert not stale.exists()
def test_parse_app_size(tmp_path: Path) -> None:
ld = tmp_path / "eagle.flash.4m.ld"
ld.write_text("MEMORY\n{\n irom0_0_seg : org = 0x40201010, len = 0xfeff0\n}\n")
with patch("esphome.build_gen.arduino8266.get_flash_ld_path", return_value=ld):
assert toolchain._parse_app_size(tmp_path, _paths(tmp_path)) == 0xFEFF0
ld.write_text("MEMORY { }\n")
with patch("esphome.build_gen.arduino8266.get_flash_ld_path", return_value=ld):
assert toolchain._parse_app_size(tmp_path, _paths(tmp_path)) is None
# A zero-length segment is bad data, not a budget; warn and drop it
ld.write_text("MEMORY\n{\n irom0_0_seg : org = 0x40201010, len = 0x0\n}\n")
with patch("esphome.build_gen.arduino8266.get_flash_ld_path", return_value=ld):
assert toolchain._parse_app_size(tmp_path, _paths(tmp_path)) is None
with patch(
"esphome.build_gen.arduino8266.get_flash_ld_path",
return_value=tmp_path / "missing.ld",
):
assert toolchain._parse_app_size(tmp_path, _paths(tmp_path)) is None
def test_print_size_summary(tmp_path: Path, capsys: pytest.CaptureFixture[str]) -> None:
with (
patch.object(
toolchain.subprocess,
"run",
return_value=MagicMock(returncode=0, stdout=_SIZE_OUTPUT),
),
patch.object(toolchain, "_parse_app_size", return_value=1044464),
):
toolchain._print_size_summary(tmp_path, _paths(tmp_path))
out = capsys.readouterr().out
# Exact PlatformIO shape so script/ci_memory_impact_extract.py can parse it
assert "RAM: [==== ] 37.9% (used 31016 bytes from 81920 bytes)" in out
assert "Flash: [==== ] 35.9% (used 375301 bytes from 1044464 bytes)" in out
def test_print_size_summary_missing_size_tool_warns(
tmp_path: Path, caplog: pytest.LogCaptureFixture
) -> None:
"""A missing size binary degrades to a warning; the firmware already
linked and must not be discarded."""
with patch.object(
toolchain.subprocess, "run", side_effect=FileNotFoundError("no size")
):
toolchain._print_size_summary(tmp_path, _paths(tmp_path))
assert "Could not summarize firmware size" in caplog.text
def test_print_size_summary_no_app_size(
tmp_path: Path, capsys: pytest.CaptureFixture[str]
) -> None:
with (
patch.object(
toolchain.subprocess,
"run",
return_value=MagicMock(returncode=0, stdout=_SIZE_OUTPUT),
),
patch.object(toolchain, "_parse_app_size", return_value=None),
):
toolchain._print_size_summary(tmp_path, _paths(tmp_path))
out = capsys.readouterr().out
# Both lines are skipped together: a RAM line without Flash would skew
# CI's memory-impact sums across builds
assert out == ""
def test_print_size_summary_size_tool_failure(
tmp_path: Path,
capsys: pytest.CaptureFixture[str],
caplog: pytest.LogCaptureFixture,
) -> None:
with patch.object(
toolchain.subprocess,
"run",
return_value=MagicMock(returncode=1, stdout="", stderr="bad elf"),
):
toolchain._print_size_summary(tmp_path, _paths(tmp_path))
assert capsys.readouterr().out == ""
assert "Could not summarize firmware size" in caplog.text
def test_get_idedata_delegates(tmp_path: Path) -> None:
with (
patch(
"esphome.build_helpers.idedata.load_or_build_idedata",
return_value={"cc_path": "x"},
) as mock_load,
patch.object(toolchain, "resolve_ccache_path", return_value="/cc/ccache"),
):
assert toolchain.get_idedata() == {"cc_path": "x"}
compile_commands, elf, cache = mock_load.call_args[0]
assert compile_commands.name == "compile_commands.json"
assert elf.name == "firmware.elf"
assert cache.name == "test8266.arduino.json"
# The exact configured launcher string is passed for compile DB parsing
# (resolve_ccache_path returns a str, untouched on every platform)
assert mock_load.call_args.kwargs["launcher"] == "/cc/ccache"
def test_get_idedata_no_ccache(tmp_path: Path) -> None:
with (
patch(
"esphome.build_helpers.idedata.load_or_build_idedata", return_value={}
) as mock_load,
patch.object(toolchain, "resolve_ccache_path", return_value=None),
):
toolchain.get_idedata()
assert mock_load.call_args.kwargs["launcher"] is None
def test_run_compile_skips_compdb_when_ninja_unchanged(tmp_path: Path) -> None:
"""An unchanged build.ninja means the compile DB is already current."""
build_dir = toolchain.get_build_dir()
build_dir.mkdir(parents=True, exist_ok=True)
# write_project (stubbed below) always leaves a build.ninja behind
(build_dir / "build.ninja").write_text("# manifest")
def run(regenerate_expected: bool) -> None:
with (
patch.object(framework, "check_and_install", return_value=_paths(tmp_path)),
patch.object(framework, "get_build_env", return_value={}),
patch("esphome.build_gen.arduino8266.write_project", return_value=False),
patch.object(
toolchain.subprocess,
"run",
return_value=MagicMock(returncode=0, stdout="", stderr=""),
),
patch.object(toolchain, "_write_compile_commands") as mock_compdb,
patch.object(toolchain, "_print_size_summary"),
patch.object(toolchain, "get_idedata"),
):
assert toolchain.run_compile({CONF_ESPHOME: {}}, verbose=False) == 0
assert mock_compdb.called == regenerate_expected
# Missing compile DB: regenerated even though build.ninja is unchanged
run(regenerate_expected=True)
# Present compile DB + unchanged build.ninja: skipped
(build_dir / "compile_commands.json").write_text("[]")
run(regenerate_expected=False)
def test_print_size_summary_unparsable_section(
tmp_path: Path,
capsys: pytest.CaptureFixture[str],
caplog: pytest.LogCaptureFixture,
) -> None:
"""A totals-relevant section that fails to parse must not produce a
confident wrong number; an irrelevant one only warns."""
bad = _SIZE_OUTPUT.replace(".bss 26504", ".bss abc")
with patch.object(
toolchain.subprocess,
"run",
return_value=MagicMock(returncode=0, stdout=bad),
):
toolchain._print_size_summary(tmp_path, _paths(tmp_path))
assert capsys.readouterr().out == ""
assert "Unparsable size output" in caplog.text
caplog.clear()
harmless = _SIZE_OUTPUT + ".broken abc 0\n"
with (
patch.object(
toolchain.subprocess,
"run",
return_value=MagicMock(returncode=0, stdout=harmless),
),
patch.object(toolchain, "_parse_app_size", return_value=1044464),
):
toolchain._print_size_summary(tmp_path, _paths(tmp_path))
assert "RAM:" in capsys.readouterr().out
assert "Unparsable size output" in caplog.text
def test_print_size_summary_missing_section_skips_summary(
tmp_path: Path,
capsys: pytest.CaptureFixture[str],
caplog: pytest.LogCaptureFixture,
) -> None:
"""A totals section absent from the output must not default to zero."""
without_bss = "\n".join(
line for line in _SIZE_OUTPUT.splitlines() if ".bss" not in line
)
with patch.object(
toolchain.subprocess,
"run",
return_value=MagicMock(returncode=0, stdout=without_bss),
):
toolchain._print_size_summary(tmp_path, _paths(tmp_path))
assert capsys.readouterr().out == ""
assert "missing section(s) .bss" in caplog.text
def test_warn_ignored_platformio_options(caplog: pytest.LogCaptureFixture) -> None:
"""Component-added options the native build drops are warned by name;
the honored ones (lib_ignore, f_cpu, ldscript, build_src_flags,
flash_mode) stay quiet."""
CORE.platformio_options = {
"board_build.ldscript": "eagle.flash.4m2m.ld",
"board_build.f_cpu": "160000000L",
"board_build.filesystem": "littlefs",
"board_build.flash_mode": "dio",
"build_src_flags": "-include throw_stubs.h",
"lib_ignore": ["Updater"],
"upload_speed": "460800",
}
toolchain._warn_ignored_platformio_options()
assert "platformio_options->board_build.filesystem is ignored" in caplog.text
assert "native 'arduino' toolchain" in caplog.text
assert "board_build.ldscript is ignored" not in caplog.text
assert "board_build.f_cpu is ignored" not in caplog.text
assert "lib_ignore" not in caplog.text
assert "build_src_flags" not in caplog.text
assert "flash_mode" not in caplog.text
# Component-added upload_speed never gets read under the native
# toolchain, so it must warn
assert "upload_speed" in caplog.text
def test_run_compile_idedata_error_does_not_fail_build(
tmp_path: Path, caplog: pytest.LogCaptureFixture
) -> None:
"""An unusable compile DB after a successful build warns, never fails."""
with (
patch.object(framework, "check_and_install", return_value=_paths(tmp_path)),
patch.object(framework, "get_build_env", return_value={}),
patch("esphome.build_gen.arduino8266.write_project"),
patch.object(
toolchain.subprocess,
"run",
return_value=MagicMock(returncode=0, stdout="", stderr=""),
),
patch.object(toolchain, "_write_compile_commands"),
patch.object(toolchain, "_print_size_summary"),
patch.object(
toolchain,
"get_idedata",
side_effect=EsphomeError("compile database is unusable"),
),
):
assert toolchain.run_compile({CONF_ESPHOME: {}}, verbose=False) == 0
assert "Could not generate idedata: compile database is unusable" in caplog.text
def test_run_compile_skipped_size_summary_names_consequence(
tmp_path: Path, caplog: pytest.LogCaptureFixture
) -> None:
"""A skipped RAM/Flash summary warns about the missing metric; the
per-cause warnings alone are invisible to CI harnesses."""
with (
patch.object(framework, "check_and_install", return_value=_paths(tmp_path)),
patch.object(framework, "get_build_env", return_value={}),
patch("esphome.build_gen.arduino8266.write_project"),
patch.object(
toolchain.subprocess,
"run",
return_value=MagicMock(returncode=0, stdout="", stderr=""),
),
patch.object(toolchain, "_write_compile_commands"),
patch.object(toolchain, "_print_size_summary", return_value=False),
patch.object(toolchain, "get_idedata", return_value=None),
):
assert toolchain.run_compile({CONF_ESPHOME: {}}, verbose=False) == 0
assert "Firmware size summary unavailable for this build" in caplog.text
def test_parse_app_size_non_utf8_ld_warns(
tmp_path: Path, caplog: pytest.LogCaptureFixture
) -> None:
"""A corrupt (non-UTF-8) linker script degrades to the same warning as
an unreadable one, never a traceback after a successful link."""
paths = _paths(tmp_path)
ld = tmp_path / "corrupt.ld"
ld.write_bytes(b"\xff\xfe not utf8")
with patch("esphome.build_gen.arduino8266.get_flash_ld_path", return_value=ld):
assert toolchain._parse_app_size(tmp_path, paths) is None
assert "Cannot read linker script" in caplog.text
def test_run_compile_failed_probe_runs_full_build(tmp_path: Path) -> None:
"""A failing -n probe (e.g. unknown target from a defective manifest)
falls through to the real build so the error prints attributably."""
probe = MagicMock(returncode=1, stdout="", stderr="")
ok = MagicMock(returncode=0, stdout="", stderr="")
def fake_run(cmd, *args, **kwargs):
return probe if "-n" in cmd else ok
with (
patch.object(framework, "check_and_install", return_value=_paths(tmp_path)),
patch.object(framework, "get_build_env", return_value={}),
patch("esphome.build_gen.arduino8266.write_project", return_value=False),
patch.object(toolchain.subprocess, "run", side_effect=fake_run) as mock_run,
patch.object(toolchain, "_write_compile_commands"),
patch.object(toolchain, "_print_size_summary", return_value=True),
patch.object(toolchain, "get_idedata", return_value={}),
):
assert toolchain.run_compile({CONF_ESPHOME: {}}, verbose=False) == 0
# The real build ran after the failed probe
assert any(
"firmware.ota.bin" in c[0][0] and "-n" not in c[0][0]
for c in mock_run.call_args_list
)
def test_get_idedata_accepts_preresolved_ccache() -> None:
"""run_compile threads its resolved ccache through; the probe must not
run again."""
with (
patch(
"esphome.build_helpers.idedata.load_or_build_idedata",
return_value={"ok": True},
) as mock_build,
patch.object(toolchain, "resolve_ccache_path") as mock_resolve,
):
assert toolchain.get_idedata("/usr/bin/ccache") == {"ok": True}
mock_resolve.assert_not_called()
assert mock_build.call_args.kwargs["launcher"] == "/usr/bin/ccache"
def test_ccache_env_includes_pch_settings() -> None:
"""The native build exports the ccache settings the pch needs."""
mark_pch_emitted()
with patch.dict(os.environ, {}, clear=True):
env = framework.ccache_env("/usr/bin/ccache")
assert env["CCACHE_SLOPPINESS"] == "pch_defines,time_macros"
assert env["CCACHE_PCH_EXTSUM"] == "true"
def test_ccache_env_pch_disabled() -> None:
with patch.dict(os.environ, {"ESPHOME_PCH_ENABLE": "0"}, clear=True):
env = framework.ccache_env("/usr/bin/ccache")
assert "CCACHE_SLOPPINESS" not in env
assert "CCACHE_PCH_EXTSUM" not in env
def test_ccache_env_respects_user_sloppiness() -> None:
mark_pch_emitted()
with patch.dict(os.environ, {"CCACHE_SLOPPINESS": "locale"}, clear=True):
env = framework.ccache_env("/usr/bin/ccache")
# The user's tokens survive; the ones the pch needs are unioned on
assert env["CCACHE_SLOPPINESS"] == "locale,pch_defines,time_macros"
assert env["CCACHE_PCH_EXTSUM"] == "true"
File diff suppressed because it is too large Load Diff
-10
View File
@@ -568,16 +568,6 @@ class TestEsphomeCore:
target.config_path = Path("foo/config")
return target
def test_add_platformio_option_merges_string_into_list(self, target) -> None:
"""A user platformio_options string lands after a component's list
(FINAL priority) and must merge as one element, not assert."""
target.add_platformio_option("extra_scripts", ["post:pch.py"])
target.add_platformio_option("extra_scripts", "pre:mine.py")
assert target.platformio_options["extra_scripts"] == [
"post:pch.py",
"pre:mine.py",
]
def test_reset(self, target):
"""Call reset on target and compare to new instance"""
other = core.EsphomeCore().__dict__
+23 -317
View File
@@ -19,7 +19,6 @@ from unittest.mock import MagicMock, patch
import pytest
from esphome.build_helpers.pch import _PCHData
from esphome.espidf.framework import (
ESPHOME_STAMP_FILE,
STAMP_SCHEMA_VERSION,
@@ -37,7 +36,6 @@ from esphome.espidf.framework import (
_patch_tools_json_demote_unused_tools,
_patch_tools_json_for_linux_arm64,
_prefetch_idf_tool_archives,
_preinstall_idf_tool_archives,
_read_stamp,
_stamp_covers,
_windows_long_paths_enabled,
@@ -404,7 +402,6 @@ def espidf_mocks(setup_core: Path):
patch("esphome.espidf.framework._patch_tools_json_for_linux_arm64"),
patch("esphome.espidf.framework._patch_tools_json_demote_unused_tools"),
patch("esphome.espidf.framework._prefetch_idf_tool_archives"),
patch("esphome.espidf.framework._preinstall_idf_tool_archives"),
patch("esphome.espidf.framework._write_stamp"),
patch("esphome.espidf.framework._check_stamp", return_value=True),
patch("esphome.espidf.framework._stamp_covers", return_value=True),
@@ -1026,16 +1023,11 @@ def test_prefetch_downloads_archives_concurrently(tmp_path: Path) -> None:
assert download.call_count == 6
def test_prefetch_reverifies_already_downloaded_archives(
tmp_path: Path, caplog: pytest.LogCaptureFixture
) -> None:
"""A pre-existing archive is not skipped: download_with_resume keeps it
only when the sha256 matches, so the pre-extraction can trust it."""
def test_prefetch_skips_already_downloaded_archives(tmp_path: Path) -> None:
dist = get_idf_tools_path() / "dist"
dist.mkdir(parents=True)
(dist / "cmake-3.30.2.tar.gz").write_bytes(b"cached")
with (
caplog.at_level(logging.INFO),
patch(
"esphome.espidf.framework.run_command",
return_value=(True, _PREFETCH_JSON, ""),
@@ -1045,12 +1037,9 @@ def test_prefetch_reverifies_already_downloaded_archives(
):
_prefetch_idf_tool_archives(tmp_path, "esp32", ["required"], None)
assert sorted(call[0][1] for call in download.call_args_list) == [
dist / "cmake-3.30.2.tar.gz",
dist / "ninja.zip",
]
# The log distinguishes verifying cached archives from real downloads
assert "Downloading 2 ESP-IDF tool archive(s) (1 cached, verifying)" in caplog.text
# only the missing archive is downloaded
assert download.call_count == 1
assert download.call_args[0][1] == dist / "ninja.zip"
@pytest.mark.parametrize(
@@ -1171,41 +1160,30 @@ def test_prefetch_passes_targets_and_tools_to_script(tmp_path: Path) -> None:
cmd = run.call_args[0][0]
assert cmd[-3:] == ["esp32,esp32c3", "required", "cmake"]
assert cmd[1].endswith("get_tool_downloads.py")
# the script inherits the caller's env plus an explicit PYTHONPATH:
# sibling scripts, the esphome package root, the framework's idf_tools
# the script inherits the caller's env plus the framework tools PYTHONPATH
env = run.call_args[1]["env"]
assert env["IDF_TOOLS_PATH"] == "/x"
assert env["PYTHONPATH"] == os.pathsep.join(
(
str(_ESPIDF_SCRIPTS_DIR),
str(_ESPIDF_SCRIPTS_DIR.parents[1]),
str(tmp_path / "tools"),
)
)
assert env["PYTHONPATH"] == str(tmp_path / "tools")
def test_framework_install_prefetches_before_installer(
espidf_mocks: SimpleNamespace,
) -> None:
"""The prefetch downloads and the pre-extraction both run before
idf_tools.py install so the installer finds the tools in place."""
"""The prefetch runs before idf_tools.py install so the installer finds
the archives already in dist/."""
calls: list[str] = []
with (
patch(
"esphome.espidf.framework._prefetch_idf_tool_archives",
side_effect=lambda *a, **k: calls.append("prefetch"),
),
patch(
"esphome.espidf.framework._preinstall_idf_tool_archives",
side_effect=lambda *a, **k: calls.append("preinstall"),
),
):
espidf_mocks.run_ok.side_effect = lambda *a, **k: (
calls.append("install") or True
)
check_esp_idf_install(_IDF_VERSION, force=True)
assert calls.index("prefetch") < calls.index("preinstall") < calls.index("install")
assert calls.index("prefetch") < calls.index("install")
# ---------------------------------------------------------------------------
@@ -1214,16 +1192,13 @@ def test_framework_install_prefetches_before_installer(
_IDF_TOOLS_STUB_DIR = Path(__file__).parent / "fixtures" / "idf_tools_stub"
_ESPIDF_SCRIPTS_DIR = Path(__file__).parents[2] / "esphome" / "espidf"
def _run_espidf_script(
tmp_path: Path,
script_name: str,
*args: str,
env_extra: dict[str, str] | None = None,
def _run_downloads_script(
tmp_path: Path, *args: str, env_extra: dict[str, str] | None = None
) -> subprocess.CompletedProcess[str]:
"""Run a real espidf helper script against the stub idf_tools module."""
"""Run the real get_tool_downloads.py against the stub idf_tools module."""
script = Path(__file__).parents[2] / "esphome" / "espidf" / "get_tool_downloads.py"
env = os.environ | {
"PYTHONPATH": str(_IDF_TOOLS_STUB_DIR),
"IDF_TOOLS_PATH": str(tmp_path / "tp"),
@@ -1231,12 +1206,7 @@ def _run_espidf_script(
if env_extra:
env |= env_extra
return subprocess.run(
[
sys.executable,
str(_ESPIDF_SCRIPTS_DIR / script_name),
str(tmp_path / "fw"),
*args,
],
[sys.executable, str(script), str(tmp_path / "fw"), *args],
capture_output=True,
text=True,
env=env,
@@ -1248,7 +1218,7 @@ def test_get_tool_downloads_lists_missing_tools(tmp_path: Path) -> None:
"""Installed versions are skipped, tools that fail their binary check are
still listed, rename_dist decides the dist filename, and idf_tools' stdout
chatter stays off the JSON channel."""
result = _run_espidf_script(tmp_path, "get_tool_downloads.py", "esp32", "required")
result = _run_downloads_script(tmp_path, "esp32", "required")
assert result.returncode == 0, result.stderr
downloads = {d["name"]: d for d in json.loads(result.stdout)}
@@ -1264,9 +1234,8 @@ def test_get_tool_downloads_lists_missing_tools(tmp_path: Path) -> None:
def test_get_tool_downloads_applies_mirror_rewrite(tmp_path: Path) -> None:
result = _run_espidf_script(
result = _run_downloads_script(
tmp_path,
"get_tool_downloads.py",
"esp32",
"required",
env_extra={"TEST_MIRROR_PREFIX": "https://mirror.test/"},
@@ -1277,13 +1246,13 @@ def test_get_tool_downloads_applies_mirror_rewrite(tmp_path: Path) -> None:
assert all(d["url"].startswith("https://mirror.test/") for d in downloads)
def _run_espidf_script_inprocess(
def _run_downloads_inprocess(
tmp_path: Path,
monkeypatch: pytest.MonkeyPatch,
script_name: str,
capsys: pytest.CaptureFixture[str],
*args: str,
) -> None:
"""Execute an espidf helper script in-process against the stub idf_tools.
) -> list[dict]:
"""Execute get_tool_downloads.py in-process against the stub idf_tools.
Unlike the subprocess variant this runs under coverage, exercising the
script's own lines.
@@ -1294,24 +1263,10 @@ def _run_espidf_script_inprocess(
stub = importlib.util.module_from_spec(spec)
spec.loader.exec_module(stub)
monkeypatch.setitem(sys.modules, "idf_tools", stub)
# _tool_resolution binds idf_tools objects at import; force a fresh
# import against this test's stub instance
monkeypatch.delitem(sys.modules, "_tool_resolution", raising=False)
# python <script> puts the script's directory on sys.path; runpy does not
monkeypatch.syspath_prepend(str(_ESPIDF_SCRIPTS_DIR))
monkeypatch.setenv("IDF_TOOLS_PATH", str(tmp_path / "tp"))
script = _ESPIDF_SCRIPTS_DIR / script_name
script = Path(__file__).parents[2] / "esphome" / "espidf" / "get_tool_downloads.py"
monkeypatch.setattr(sys, "argv", [str(script), str(tmp_path / "fw"), *args])
runpy.run_path(str(script))
def _run_downloads_inprocess(
tmp_path: Path,
monkeypatch: pytest.MonkeyPatch,
capsys: pytest.CaptureFixture[str],
*args: str,
) -> list[dict]:
_run_espidf_script_inprocess(tmp_path, monkeypatch, "get_tool_downloads.py", *args)
return json.loads(capsys.readouterr().out)
@@ -1610,14 +1565,10 @@ def _ccache_patches(tmp_path: Path, which: str | None, build_path: Path | None):
"esphome.espidf.framework.get_idf_tools_path",
return_value=tmp_path / "tools",
),
# ccache_defaults_env and the pch emission flag read CORE at call time
# ccache_defaults_env (build_helpers.ccache) reads CORE at call time
patch(
"esphome.core.CORE",
SimpleNamespace(
build_path=build_path,
# Pre-marked: these env tests model a pch-emitting build
data={"pch": _PCHData(emitted=True)},
),
SimpleNamespace(build_path=build_path),
),
)
@@ -1631,9 +1582,6 @@ def test_ccache_env_default_enabled_when_available(tmp_path: Path) -> None:
assert env["CCACHE_NOHASHDIR"] == "true"
assert env["CCACHE_DEPEND"] == "1"
assert env["CCACHE_BASEDIR"] == str((tmp_path / "build").resolve())
# The pch cannot cache under ccache without these
assert env["CCACHE_SLOPPINESS"] == "pch_defines,time_macros"
assert env["CCACHE_PCH_EXTSUM"] == "true"
def test_ccache_env_disabled_when_binary_missing(tmp_path: Path) -> None:
@@ -2025,245 +1973,3 @@ def test_check_windows_path_length_long_path_warns(
assert "long path support" in message
# The install is global now; the remedy is the prefix env, not moving the project.
assert "ESPHOME_ESP_IDF_PREFIX" in message
# ---------------------------------------------------------------------------
# _preinstall_idf_tool_archives
# ---------------------------------------------------------------------------
def test_preinstall_streams_script_with_workers(tmp_path: Path) -> None:
"""The pre-extraction streams install_tool_archives.py with the worker
count and the same targets/tools the installer will get."""
with (
patch(
"esphome.espidf.framework._run_idf_tools_script",
return_value=(True, None, None),
) as run_script,
patch("esphome.espidf.framework.get_usable_cpu_count", return_value=3),
):
_preinstall_idf_tool_archives(
tmp_path, "esp32,esp32c3", ["required", "cmake"], {"IDF_TOOLS_PATH": "x"}
)
run_script.assert_called_once_with(
tmp_path,
"install_tool_archives.py",
"ESP-IDF tool archive extraction",
args=["esp32,esp32c3", "3", "required", "cmake"],
env={"IDF_TOOLS_PATH": "x"},
stream_output=True,
)
def test_preinstall_caps_workers(tmp_path: Path) -> None:
"""A high core count is capped; the workers share one disk."""
with (
patch(
"esphome.espidf.framework._run_idf_tools_script",
return_value=(True, None, None),
) as run_script,
patch("esphome.espidf.framework.get_usable_cpu_count", return_value=64),
):
_preinstall_idf_tool_archives(tmp_path, "esp32", ["required"], None)
assert run_script.call_args.kwargs["args"][1] == "10"
def test_preinstall_script_failure_only_warns(
tmp_path: Path, caplog: pytest.LogCaptureFixture
) -> None:
"""A failed pre-extraction leaves the install to the sequential path."""
with (
patch(
"esphome.espidf.framework._run_idf_tools_script",
return_value=(False, None, None),
),
patch("esphome.espidf.framework.get_usable_cpu_count", return_value=1),
):
_preinstall_idf_tool_archives(tmp_path, "esp32", ["required"], None)
assert "pre-extraction failed" in caplog.text
def test_preinstall_exception_only_warns(
tmp_path: Path, caplog: pytest.LogCaptureFixture
) -> None:
"""An unexpected error must not become a new way for the install to fail,
and keeps its traceback at WARNING."""
with (
patch(
"esphome.espidf.framework._run_idf_tools_script",
side_effect=TypeError("bad call"),
),
patch("esphome.espidf.framework.get_usable_cpu_count", return_value=1),
):
_preinstall_idf_tool_archives(tmp_path, "esp32", ["required"], None)
record = next(r for r in caplog.records if "pre-extraction failed" in r.message)
assert record.exc_info is not None
# ---------------------------------------------------------------------------
# install_tool_archives.py (against the stub idf_tools module in fixtures/)
# ---------------------------------------------------------------------------
def _make_dist(tmp_path: Path, *names: str) -> None:
dist = tmp_path / "tp" / "dist"
dist.mkdir(parents=True, exist_ok=True)
for name in names:
(dist / name).write_bytes(b"x")
def test_install_tool_archives_extracts_pending_in_parallel(tmp_path: Path) -> None:
"""Subprocess end-to-end: tools with a prefetched archive install
concurrently; installed and broken tools stay with the installer."""
_make_dist(tmp_path, "cmake.tar.gz", "ninja-v1.zip", "x.tar.gz", "y.tar.gz")
result = _run_espidf_script(
tmp_path, "install_tool_archives.py", "esp32", "8", "required"
)
assert result.returncode == 0, result.stderr
tools = tmp_path / "tp" / "tools"
assert (tools / "cmake" / "3.30.2" / ".installed").is_file()
assert (tools / "ninja" / "1.12.1" / ".installed").is_file()
assert not (tools / "installed-tool").exists()
assert not (tools / "broken-tool").exists()
# The worker count clamps to the pending count
assert (
"Extracting 2 ESP-IDF tool archive(s) with 2 worker(s): "
"cmake@3.30.2, ninja@1.12.1" in result.stdout
)
assert "leaving broken broken-tool to the installer" in result.stderr
def test_install_tool_archives_single_pending_stays_sequential(
tmp_path: Path,
monkeypatch: pytest.MonkeyPatch,
capsys: pytest.CaptureFixture[str],
) -> None:
"""One pending archive has nothing to parallelize; the installer keeps
its normal output."""
_make_dist(tmp_path, "cmake.tar.gz")
_run_espidf_script_inprocess(
tmp_path, monkeypatch, "install_tool_archives.py", "esp32", "4", "required"
)
assert not (tmp_path / "tp" / "tools").exists()
out = capsys.readouterr().out
assert "Extracting" not in out
# A resolution drift that empties pending stays observable
assert "1 of 2 uninstalled tool(s) have a prefetched archive" in out
def test_install_tool_archives_failed_install_left_to_installer(
tmp_path: Path,
monkeypatch: pytest.MonkeyPatch,
capsys: pytest.CaptureFixture[str],
) -> None:
"""A per-tool failure warns, removes the torn dest dir so the installer
cannot trust it, and moves on; the other tools still install."""
_make_dist(tmp_path, "cmake.tar.gz", "ninja-v1.zip")
monkeypatch.setenv("TEST_FAIL_INSTALL", "ninja")
_run_espidf_script_inprocess(
tmp_path, monkeypatch, "install_tool_archives.py", "esp32", "4", "required"
)
tools = tmp_path / "tp" / "tools"
assert (tools / "cmake" / "3.30.2" / ".installed").is_file()
assert not (tools / "ninja" / "1.12.1").exists()
err = capsys.readouterr().err
assert "pre-extracting ninja@1.12.1 failed" in err
assert "1 of 2 pre-extractions failed" in err
def test_install_tool_archives_all_failed_exits_nonzero(
tmp_path: Path,
monkeypatch: pytest.MonkeyPatch,
capsys: pytest.CaptureFixture[str],
) -> None:
"""Every job failing is a systematic fault; the nonzero exit lets the
caller log it."""
_make_dist(tmp_path, "cmake.tar.gz", "ninja-v1.zip")
monkeypatch.setenv("TEST_FAIL_INSTALL", "cmake,ninja")
with pytest.raises(SystemExit) as excinfo:
_run_espidf_script_inprocess(
tmp_path, monkeypatch, "install_tool_archives.py", "esp32", "4", "required"
)
assert excinfo.value.code == 1
assert "2 of 2 pre-extractions failed" in capsys.readouterr().err
tools = tmp_path / "tp" / "tools"
assert not (tools / "cmake" / "3.30.2").exists()
assert not (tools / "ninja" / "1.12.1").exists()
def test_install_tool_archives_inprocess_dedupes_and_skips(
tmp_path: Path,
monkeypatch: pytest.MonkeyPatch,
capsys: pytest.CaptureFixture[str],
) -> None:
"""In-process full flow: duplicate tool@version specs collapse to one
job, broken and installed tools are skipped, both pending tools install."""
_make_dist(tmp_path, "cmake.tar.gz", "ninja-v1.zip", "x.tar.gz", "y.tar.gz")
_run_espidf_script_inprocess(
tmp_path,
monkeypatch,
"install_tool_archives.py",
"esp32",
"8",
"cmake",
"ninja",
"cmake@3.30.2",
"installed-tool",
"broken-tool",
)
captured = capsys.readouterr()
assert (
"Extracting 2 ESP-IDF tool archive(s) with 2 worker(s): "
"cmake@3.30.2, ninja@1.12.1" in captured.out
)
assert "extracted cmake@3.30.2" in captured.out
assert "extracted ninja@1.12.1" in captured.out
assert "leaving broken broken-tool to the installer" in captured.err
tools = tmp_path / "tp" / "tools"
assert (tools / "cmake" / "3.30.2" / ".installed").is_file()
assert (tools / "ninja" / "1.12.1" / ".installed").is_file()
assert not (tools / "installed-tool").exists()
assert not (tools / "broken-tool").exists()
def test_install_tool_archives_surviving_torn_dir_escalates(
tmp_path: Path,
monkeypatch: pytest.MonkeyPatch,
capsys: pytest.CaptureFixture[str],
) -> None:
"""A torn dir that survives cleanup could fool the installer; the exit
is nonzero even though the other tool succeeded."""
import esphome.helpers
_make_dist(tmp_path, "cmake.tar.gz", "ninja-v1.zip")
monkeypatch.setenv("TEST_FAIL_INSTALL", "ninja")
monkeypatch.setattr(
esphome.helpers, "rmtree", MagicMock(side_effect=OSError("busy"))
)
with pytest.raises(SystemExit) as excinfo:
_run_espidf_script_inprocess(
tmp_path, monkeypatch, "install_tool_archives.py", "esp32", "4", "required"
)
assert excinfo.value.code == 1
err = capsys.readouterr().err
assert "could not remove" in err
assert "1 of 2 pre-extractions failed" in err
assert (tmp_path / "tp" / "tools" / "cmake" / "3.30.2" / ".installed").is_file()
def test_install_tool_archives_skips_unverifiable_archives(
tmp_path: Path,
monkeypatch: pytest.MonkeyPatch,
capsys: pytest.CaptureFixture[str],
) -> None:
"""An entry the prefetch could not verify is never extracted, even with
an archive on disk."""
_make_dist(tmp_path, "cmake.tar.gz", "ninja-v1.zip")
monkeypatch.setenv("TEST_NO_SHA", "cmake,ninja")
_run_espidf_script_inprocess(
tmp_path, monkeypatch, "install_tool_archives.py", "esp32", "4", "required"
)
assert not (tmp_path / "tp" / "tools").exists()
assert "0 of 2 uninstalled tool(s) have a prefetched archive" in (
capsys.readouterr().out
)
-79
View File
@@ -93,13 +93,6 @@ def test_get_configured_targets_ci_installs_all(monkeypatch: pytest.MonkeyPatch)
assert toolchain._get_configured_targets() is None
@pytest.fixture(autouse=True)
def _no_ccache(monkeypatch: pytest.MonkeyPatch) -> None:
"""Deterministic run_compile: no host ccache probe, no pch work."""
monkeypatch.setenv("IDF_CCACHE_ENABLE", "0")
monkeypatch.setenv("ESPHOME_PCH_ENABLE", "0")
def _setup_build(setup_core: Path) -> tuple[Path, Path]:
"""Point CORE at a build dir; return (compile_commands, idedata cache) paths."""
CORE.name = "test"
@@ -667,75 +660,3 @@ def test_get_core_framework_version_from_core_data():
CORE.data = {KEY_ESP32: {KEY_IDF_VERSION: cv.Version(5, 5, 4)}}
assert toolchain._get_core_framework_version() == "5.5.4"
def test_run_compile_aborts_when_stale_pch_survives_discard(
setup_core: Path, monkeypatch: pytest.MonkeyPatch
) -> None:
"""An undiscardable stale .gch means silently wrong output: abort."""
from esphome.core import EsphomeError
monkeypatch.setenv("ESPHOME_PCH_ENABLE", "1")
_setup_build(setup_core)
with (
patch.object(toolchain, "need_reconfigure", return_value=False),
patch.object(toolchain, "run_idf_py", return_value=0),
patch.object(toolchain, "print_summary"),
patch("esphome.build_gen.espidf.prepare_pch", side_effect=RuntimeError("boom")),
patch(
"esphome.build_helpers.pch.discard_pch",
side_effect=EsphomeError("Could not discard the stale precompiled header"),
),
pytest.raises(EsphomeError, match="Could not discard"),
):
toolchain.run_compile({CONF_ESPHOME: {}}, verbose=False)
def test_run_compile_strict_reraises_pch_failure(
setup_core: Path, monkeypatch: pytest.MonkeyPatch
) -> None:
"""ESPHOME_PCH_STRICT must reach through the real compile flow."""
from esphome.core import EsphomeError
monkeypatch.setenv("ESPHOME_PCH_ENABLE", "1")
monkeypatch.setenv("ESPHOME_PCH_STRICT", "1")
_setup_build(setup_core)
with (
patch.object(toolchain, "need_reconfigure", return_value=False),
patch.object(toolchain, "run_idf_py", return_value=0),
patch.object(toolchain, "print_summary"),
patch(
"esphome.build_gen.espidf.prepare_pch",
side_effect=EsphomeError("ESPHOME_PCH_STRICT: no usable compile command"),
),
pytest.raises(EsphomeError, match="no usable compile command"),
):
toolchain.run_compile({CONF_ESPHOME: {}}, verbose=False)
def test_run_compile_invokes_prepare_pch_and_survives_failure(
setup_core: Path, monkeypatch: pytest.MonkeyPatch
) -> None:
"""The pch hook runs before the build and a failure never aborts it."""
monkeypatch.setenv("ESPHOME_PCH_ENABLE", "1")
_setup_build(setup_core)
# A stale .gch must be discarded on the failure path, never consumed
build = setup_core / "build" / "test" / "build"
build.mkdir(parents=True, exist_ok=True)
(build / "esphome_pch.h").write_text("")
stale_gch = build / "esphome_pch.h.gch"
stale_gch.write_bytes(b"stale")
with (
patch.object(toolchain, "need_reconfigure", return_value=False),
patch.object(toolchain, "run_idf_py", return_value=0),
patch.object(toolchain, "print_summary"),
patch(
"esphome.build_gen.espidf.prepare_pch", side_effect=RuntimeError("boom")
) as prepare,
):
assert toolchain.run_compile({CONF_ESPHOME: {}}, verbose=False) == 0
prepare.assert_called_once()
assert not stale_gch.exists()
+4 -73
View File
@@ -523,17 +523,6 @@ class TestArchiveExtractAll:
archive_extract_all(archive, dest)
assert (dest / "file.txt").read_text() == "hi"
def test_progress_callback_passed_through(self, tmp_path: Path) -> None:
"""The progress kwarg reaches the dispatched extractor."""
archive = tmp_path / "test.tar.gz"
archive.write_bytes(_gzip_tar_bytes({"file.txt": b"hello"}))
dest = tmp_path / "out"
dest.mkdir()
fractions: list[float] = []
archive_extract_all(archive, dest, progress=fractions.append)
assert fractions[-1] == 1
assert (dest / "file.txt").read_bytes() == b"hello"
def test_invalid_type_raises_type_error(self) -> None:
with pytest.raises(TypeError, match="archive must be"):
archive_extract_all(42, ".") # type: ignore[arg-type]
@@ -1962,19 +1951,6 @@ class TestTarExtractAllBranches:
mock_pb.assert_called_once_with("Extracting")
mock_pb.return_value.update.assert_called()
def test_progress_callback_replaces_bar(self, tmp_path: Path) -> None:
"""A progress callback wins over progress_header and ends at 1.0."""
buf = _make_tar([_reg("a.txt"), _reg("b.txt")], {"a.txt": b"x", "b.txt": b"y"})
fractions: list[float] = []
with patch("esphome.framework_helpers.ProgressBar") as mock_pb:
_tar_extract_all(
buf, tmp_path, progress_header="Extracting", progress=fractions.append
)
mock_pb.assert_not_called()
assert fractions == sorted(fractions)
assert fractions[-1] == 1
assert (tmp_path / "a.txt").is_file()
# ---------------------------------------------------------------------------
# _zip_extract_all — additional branch coverage
@@ -2004,19 +1980,6 @@ class TestZipExtractAllBranches:
mock_pb.assert_called_once_with("Unzipping")
mock_pb.return_value.update.assert_called()
def test_progress_callback_replaces_bar(self, tmp_path: Path) -> None:
"""A progress callback wins over progress_header and ends at 1.0."""
buf = _make_zip([("a.txt", "aaa"), ("b.txt", "bbb")])
fractions: list[float] = []
with patch("esphome.framework_helpers.ProgressBar") as mock_pb:
_zip_extract_all(
buf, tmp_path, progress_header="Unzipping", progress=fractions.append
)
mock_pb.assert_not_called()
assert fractions == sorted(fractions)
assert fractions[-1] == 1
assert (tmp_path / "a.txt").is_file()
# ---------------------------------------------------------------------------
# _rename_with_retry
@@ -2174,20 +2137,6 @@ class TestSevenZipExtractAll:
mock_pb.assert_called_once_with("Unpacking 7z")
mock_pb.return_value.update.assert_called()
def test_progress_callback_replaces_bar(self, tmp_path: Path) -> None:
"""A progress callback wins over progress_header; 7z reports 1.0 once."""
buf = self._make_7z({"file.txt": b"x"})
out = tmp_path / "out"
out.mkdir()
fractions: list[float] = []
with patch("esphome.framework_helpers.ProgressBar") as mock_pb:
_7z_extract_all(
buf, out, progress_header="Unpacking 7z", progress=fractions.append
)
mock_pb.assert_not_called()
assert fractions == [1]
assert (out / "file.txt").is_file()
def test_absolute_path_in_names_skipped(self, tmp_path: Path) -> None:
"""Names that resolve as absolute are silently skipped."""
import py7zr
@@ -2345,36 +2294,18 @@ def test_resume_fetch_job_threads_tracker(tmp_path: Path) -> None:
)
def test_warn_batch_failures_names_each_failure(
def test_warn_prefetch_failures_names_each_failure(
caplog: pytest.LogCaptureFixture,
) -> None:
"""The shared failure loop warns per job with the failure reason."""
from esphome.framework_helpers import warn_batch_failures
from esphome.framework_helpers import warn_prefetch_failures
warn_batch_failures([("toolchain-x@1", OSError("down"))])
warn_prefetch_failures([("toolchain-x@1", OSError("down"))])
assert "Could not prefetch toolchain-x@1: down" in caplog.text
warn_batch_failures([("lib", OSError("gone"))], "Prefetch of %s failed: %s")
warn_prefetch_failures([("lib", OSError("gone"))], "Prefetch of %s failed: %s")
assert "Prefetch of lib failed: gone" in caplog.text
def test_warn_batch_failures_unexpected_error_keeps_traceback(
caplog: pytest.LogCaptureFixture,
) -> None:
"""An unexpected error type is not reduced to a bare message; expected
download failures stay message-only at WARNING."""
from esphome.framework_helpers import warn_batch_failures
with caplog.at_level(logging.DEBUG):
warn_batch_failures(
[("pkg", TypeError("bad call")), ("lib", OSError("down"))],
"Could not install %s: %s",
)
warnings = {r.getMessage(): r for r in caplog.records if r.levelname == "WARNING"}
assert warnings["Could not install pkg: bad call"].exc_info is not None
assert warnings["Could not install lib: down"].exc_info is None
assert "Failure detail" in caplog.text
@pytest.mark.parametrize(
("platform", "input_path", "expected"),
[
-21
View File
@@ -175,27 +175,6 @@ def test_esptool_upload_fast_path_does_not_import_heavy_modules(
)
def test_native_upload_fast_path_does_not_import_heavy_modules(
fixture_path: Path,
probe_env: dict[str, str],
) -> None:
"""The native-toolchain serial upload dispatches through a
toolchain-keyed backend table; resolving the flash image must not drag
in the esp32 or esp8266 component package or the validation stack.
"""
leaked = _leaked_from_fixture(
fixture_path,
probe_env,
"native_upload_fast_path.py",
extra=("esphome.components.esp8266",),
)
assert not leaked, (
f"native upload_using_esptool pulls in heavy modules: {leaked}. "
"The upload fast path skips validation; importing a platform "
"component package executes its codegen module by design."
)
def test_api_client_does_not_import_heavy_modules() -> None:
"""``esphome.api_client`` is on the logs fast path and must stay light.
+3 -262
View File
@@ -13,7 +13,7 @@ import sys
import time
from types import SimpleNamespace
from typing import Any, Self
from unittest.mock import AsyncMock, MagicMock, Mock, PropertyMock, patch
from unittest.mock import AsyncMock, MagicMock, Mock, patch
import pytest
from pytest import CaptureFixture
@@ -25,7 +25,6 @@ from esphome.__main__ import (
Purpose,
_get_configured_xtal_freq,
_make_crystal_freq_callback,
_native_toolchain_module,
_redact_with_legacy_fallback,
_resolve_network_devices,
_should_subscribe_states,
@@ -6932,7 +6931,7 @@ def test_command_run_rp2040_bootsel_redetects_serial_port() -> None:
def test_command_idedata_esp_idf_prints_json(capsys: CaptureFixture) -> None:
"""Under the native ESP-IDF toolchain, idedata is emitted as JSON."""
setup_core(platform=PLATFORM_ESP32)
setup_core()
CORE.toolchain = Toolchain.ESP_IDF
data = {"cxx_path": "g++", "prog_path": "/build/firmware.elf"}
@@ -6946,7 +6945,7 @@ def test_command_idedata_esp_idf_prints_json(capsys: CaptureFixture) -> None:
def test_command_idedata_esp_idf_no_build_errors() -> None:
"""Under ESP-IDF, a missing build (no idedata) returns an error, not a crash."""
setup_core(platform=PLATFORM_ESP32)
setup_core()
CORE.toolchain = Toolchain.ESP_IDF
with patch("esphome.espidf.toolchain.get_idedata", return_value=None):
@@ -7120,201 +7119,6 @@ def test_warn_source_tree_mismatch_falls_back_when_stat_fails(
assert not caplog.text
def test_upload_using_esptool_arduino_toolchain(
tmp_path: Path,
mock_run_external_command_main: Mock,
) -> None:
"""The native ESP8266 Arduino toolchain flashes its factory image at
0x0, resolved from the toolchain-keyed backend table (deliberately not
the platform hook: that import would break the upload fast path)."""
setup_core(platform=PLATFORM_ESP8266, tmp_path=tmp_path, name="test")
CORE.toolchain = Toolchain.ARDUINO
from esphome.arduino8266 import toolchain as native
factory = native.get_factory_firmware_path()
factory.parent.mkdir(parents=True, exist_ok=True)
factory.touch()
config = {CONF_ESPHOME: {"platformio_options": {}}}
result = upload_using_esptool(config, "/dev/ttyUSB0", None, None)
assert result == 0
cmd_list = list(mock_run_external_command_main.call_args[0][1:])
firmware_offset_idx = cmd_list.index("write-flash") + 4
assert cmd_list[firmware_offset_idx] == "0x0"
assert cmd_list[firmware_offset_idx + 1] == str(factory)
@pytest.mark.parametrize(
("toolchain", "pio_project_written"),
[
# The native toolchain generates its project at compile time, so
# write_cpp_file must not write a platformio.ini; the default
# toolchain writes the PlatformIO project files.
(Toolchain.ARDUINO, False),
(None, True),
],
)
def test_write_cpp_file_project_generation_follows_toolchain(
tmp_path: Path, toolchain: Toolchain | None, pio_project_written: bool
) -> None:
setup_core(platform=PLATFORM_ESP8266, tmp_path=tmp_path, name="test")
CORE.toolchain = toolchain
with (
patch("esphome.writer.write_cpp") as mock_write_cpp,
patch("esphome.build_gen.platformio.write_project") as mock_pio_project,
patch.object(
type(CORE), "cpp_main_section", new_callable=PropertyMock
) as mock_section,
):
mock_section.return_value = ""
assert main.write_cpp_file() == 0
mock_write_cpp.assert_called_once()
assert mock_pio_project.called is pio_project_written
def test_command_idedata_arduino_prints_json(
tmp_path: Path, capsys: CaptureFixture
) -> None:
"""Under the native ESP8266 Arduino toolchain, idedata is emitted as JSON."""
setup_core(platform=PLATFORM_ESP8266, tmp_path=tmp_path)
CORE.toolchain = Toolchain.ARDUINO
data = {"cxx_path": "g++", "prog_path": "/build/firmware.elf"}
with patch(
"esphome.arduino8266.toolchain.get_idedata", return_value=data
) as mock_get:
result = command_idedata(MagicMock(), CORE.config)
assert result == 0
mock_get.assert_called_once_with()
assert json.loads(capsys.readouterr().out) == data
def test_command_idedata_arduino_no_build_errors(tmp_path: Path) -> None:
"""A missing native build (no idedata) returns an error, not a crash."""
setup_core(platform=PLATFORM_ESP8266, tmp_path=tmp_path)
CORE.toolchain = Toolchain.ARDUINO
with patch("esphome.arduino8266.toolchain.get_idedata", return_value=None):
result = command_idedata(MagicMock(), CORE.config)
assert result == 1
@pytest.mark.parametrize(
("platform", "toolchain", "module"),
[
(PLATFORM_ESP8266, Toolchain.ARDUINO, "esphome.arduino8266.toolchain"),
(PLATFORM_ESP32, Toolchain.ESP_IDF, "esphome.espidf.toolchain"),
],
)
def test_command_analyze_memory_native_toolchains(
tmp_path: Path,
mock_write_cpp: Mock,
mock_compile_program: Mock,
mock_get_esphome_components: Mock,
mock_memory_analyzer_cli: Mock,
mock_ram_strings_analyzer: Mock,
platform: str,
toolchain: Toolchain,
module: str,
) -> None:
"""analyze-memory uses the native toolchain's binutils instead of
falling into the PlatformIO branch."""
setup_core(platform=platform, tmp_path=tmp_path, name="test_device")
CORE.toolchain = toolchain
config = {CONF_ESPHOME: {CONF_NAME: "test_device"}}
# The tools must exist: a missing binutils now fails by name instead of
# silently falling back to host tools
objdump = tmp_path / "objdump"
readelf = tmp_path / "readelf"
objdump.write_text("")
readelf.write_text("")
# The ELF must exist too: the analyzer swallows tool failures, so a
# missing image would report zeroes with exit 0
firmware_elf = tmp_path / "firmware.elf"
firmware_elf.write_text("")
with (
patch(f"{module}.get_objdump_path", return_value=objdump),
patch(f"{module}.get_readelf_path", return_value=readelf),
patch(f"{module}.get_elf_path", return_value=firmware_elf),
):
result = command_analyze_memory(MockArgs(), config)
assert result == 0
mock_memory_analyzer_cli.assert_called_once_with(
str(firmware_elf),
str(objdump),
str(readelf),
set(),
idedata=None,
)
def test_command_analyze_memory_native_missing_elf_fails(
tmp_path: Path,
mock_write_cpp: Mock,
mock_compile_program: Mock,
mock_get_esphome_components: Mock,
mock_memory_analyzer_cli: Mock,
mock_ram_strings_analyzer: Mock,
caplog: pytest.LogCaptureFixture,
) -> None:
"""A missing firmware.elf fails by name instead of an exit-0 zeroed
report."""
setup_core(platform=PLATFORM_ESP8266, tmp_path=tmp_path, name="test_device")
CORE.toolchain = Toolchain.ARDUINO
config = {CONF_ESPHOME: {CONF_NAME: "test_device"}}
objdump = tmp_path / "objdump"
readelf = tmp_path / "readelf"
objdump.write_text("")
readelf.write_text("")
module = "esphome.arduino8266.toolchain"
with (
patch(f"{module}.get_objdump_path", return_value=objdump),
patch(f"{module}.get_readelf_path", return_value=readelf),
patch(f"{module}.get_elf_path", return_value=tmp_path / "missing.elf"),
):
result = command_analyze_memory(MockArgs(), config)
assert result == 1
assert "compile the configuration first" in caplog.text
mock_memory_analyzer_cli.assert_not_called()
def test_command_analyze_memory_missing_binutils_fails_by_name(
tmp_path: Path, caplog: pytest.LogCaptureFixture
) -> None:
"""A truncated toolchain install fails naming the missing tool instead
of silently analyzing with host binutils."""
setup_core(platform="esp8266", tmp_path=tmp_path, name="test_device")
CORE.toolchain = Toolchain.ARDUINO
config = {CONF_ESPHOME: {CONF_NAME: "test_device"}}
module = "esphome.arduino8266.toolchain"
with (
patch(f"{module}.get_objdump_path", return_value=tmp_path / "missing-objdump"),
patch(f"{module}.get_readelf_path", return_value=tmp_path / "readelf"),
patch("esphome.__main__.write_cpp", return_value=0),
patch("esphome.__main__.compile_program", return_value=0),
):
assert command_analyze_memory(MockArgs(), config) == 1
assert "missing-objdump" in caplog.text
assert "toolchain install may be incomplete" in caplog.text
def test_command_idedata_incompatible_toolchain(tmp_path: Path) -> None:
"""A non-native, non-platformio toolchain errors out cleanly."""
setup_core(platform=PLATFORM_ESP32, tmp_path=tmp_path)
CORE.toolchain = Toolchain.SDK_NRF
assert command_idedata(MagicMock(), CORE.config) == 1
@pytest.mark.parametrize(
"error",
[
@@ -7391,46 +7195,6 @@ def test_compile_program_espidf_idedata_none_warns(
assert "No idedata was generated" in caplog.text
def test_native_toolchain_table_serves_every_native_toolchain() -> None:
"""Every member of NATIVE_TOOLCHAINS has a backend entry; a gap would
surface as a targeted EsphomeError on the one affected config, and this
pin keeps the table from drifting when a toolchain is added."""
from esphome.const import NATIVE_TOOLCHAINS
assert {tc for _, tc in main._NATIVE_TOOLCHAIN_MODULES} == set(NATIVE_TOOLCHAINS)
def test_native_toolchain_module_missing_backend_raises(tmp_path: Path) -> None:
"""A native toolchain missing from the backend table is a bug and must
fail, not silently degrade to the PlatformIO path."""
setup_core(platform=PLATFORM_ESP8266, tmp_path=tmp_path, name="test_device")
CORE.toolchain = Toolchain.ARDUINO
with (
patch.dict(main._NATIVE_TOOLCHAIN_MODULES, clear=True),
pytest.raises(EsphomeError, match="no native build backend"),
):
_native_toolchain_module()
def test_command_analyze_memory_unsupported_toolchain(
tmp_path: Path,
mock_write_cpp: Mock,
mock_compile_program: Mock,
caplog: pytest.LogCaptureFixture,
) -> None:
"""A hook-less non-PlatformIO toolchain is refused by name, never routed
into the PlatformIO branch."""
setup_core(platform=PLATFORM_NRF52, tmp_path=tmp_path, name="test_device")
CORE.toolchain = Toolchain.SDK_NRF
mock_write_cpp.return_value = 0
mock_compile_program.return_value = 0
result = command_analyze_memory(MockArgs(), {CONF_ESPHOME: {CONF_NAME: "t"}})
assert result == 1
assert "analyze-memory is not supported" in caplog.text
def test_cli_toolchain_skips_the_validated_config_cache(tmp_path: Path) -> None:
"""An explicit --toolchain must run the per-platform validators, so the
upload/logs fast path becomes a cache miss."""
@@ -7446,29 +7210,6 @@ def test_cli_toolchain_skips_the_validated_config_cache(tmp_path: Path) -> None:
mock_read.assert_called_once()
def test_upload_using_esptool_native_missing_firmware_raises(
tmp_path: Path,
) -> None:
"""A stale or absent firmware.bin fails by name instead of flashing air."""
setup_core(platform=PLATFORM_ESP8266, tmp_path=tmp_path, name="test")
CORE.toolchain = Toolchain.ARDUINO
with pytest.raises(EsphomeError, match="compile the configuration first"):
upload_using_esptool(
{CONF_ESPHOME: {"platformio_options": {}}}, "/dev/ttyUSB0", None, None
)
def test_compile_program_unclaimed_native_toolchain_raises(
tmp_path: Path,
) -> None:
"""A resolved native toolchain no platform backend claims must fail,
never fall through to the PlatformIO project path."""
setup_core(platform=PLATFORM_ESP32, tmp_path=tmp_path, name="test_device")
CORE.toolchain = Toolchain.ARDUINO # esp32 has no arduino-native backend
with pytest.raises(EsphomeError, match="no platform backend claimed"):
compile_program(MockArgs(), {})
def test_cli_toolchain_still_refreshes_the_validated_config_cache(
tmp_path: Path,
) -> None:
-21
View File
@@ -1,21 +0,0 @@
"""The pch prefix must keep resolving; a rename would silently
collapse the precompiled set to defines.h with strict CI still green."""
from pathlib import Path
import re
from esphome.build_helpers.pch import PCH_PREFIX_HEADER
REPO = Path(__file__).parents[2]
def test_pch_prefix_resolves() -> None:
prefix = REPO / PCH_PREFIX_HEADER
assert prefix.is_file()
body = prefix.read_text()
includes = re.findall(r'#include "([^"]+)"', body)
assert includes, "prefix wrapper folds nothing"
for name in includes:
assert (REPO / name).is_file(), f"{name} does not resolve"
# The C guard is what keeps build_src_flags safe on C/assembly edges
assert "#ifdef __cplusplus" in body
+3 -206
View File
@@ -10,7 +10,7 @@ from pathlib import Path
import pytest
from esphome.core import CORE, EsphomeError, Library
from esphome.core import EsphomeError, Library
import esphome.platformio.library as lib
from esphome.platformio.library import (
SOURCE_KIND_FOR_SUFFIX,
@@ -29,13 +29,9 @@ from esphome.platformio.library import (
)
def _backend(emit=lambda component: None, provides=None) -> LibraryBackend:
def _backend(emit=lambda component: None) -> LibraryBackend:
return LibraryBackend(
platform="espressif32",
framework="espidf",
emit=emit,
cache_key="idf",
provides=provides,
platform="espressif32", framework="espidf", emit=emit, cache_key="idf"
)
@@ -956,202 +952,3 @@ def test_source_kind_map_shape() -> None:
assert SOURCE_KIND_FOR_SUFFIX[".S"] == "aspp"
assert SOURCE_KIND_FOR_SUFFIX[".c"] == "c"
assert SOURCE_KIND_FOR_SUFFIX[".cpp"] == "cxx"
# SCons's case-sensitive C++ suffixes: PIO compiles .C as C++
assert SOURCE_KIND_FOR_SUFFIX[".C"] == "cxx"
assert SOURCE_KIND_FOR_SUFFIX[".C++"] == "cxx"
def test_versionless_platform_filtered_dependency_stays_quiet(
tmp_path, monkeypatch, caplog: pytest.LogCaptureFixture
) -> None:
"""A version-less dependency the platform filter excludes is
deliberately absent, not a drop to warn about."""
_patch_download_with_manifests(
monkeypatch,
tmp_path,
{
"esphome/A": {
"name": "A",
"dependencies": [{"name": "Hash", "platforms": "espressif8266"}],
}
},
)
convert_libraries([Library("esphome/A", None, None)], _backend())
assert "has no version to resolve" not in caplog.text
def test_versionless_ignored_dependency_stays_quiet(
tmp_path, monkeypatch, caplog: pytest.LogCaptureFixture
) -> None:
"""A lib_ignore'd version-less dependency is deliberately excluded, not
a drop; no reconciliation warning."""
_patch_download_with_manifests(
monkeypatch,
tmp_path,
{"esphome/A": {"name": "A", "dependencies": [{"name": "Hash"}]}},
)
CORE.platformio_options = {"lib_ignore": ["Hash"]}
convert_libraries([Library("esphome/A", None, None)], _backend())
assert "has no version to resolve" not in caplog.text
def test_versionless_dependency_without_provider_warns(
tmp_path, monkeypatch, caplog: pytest.LogCaptureFixture
) -> None:
"""A backend whose tree could supply the name warns on the drop; one
without provides() can never act on it, so it stays at debug."""
_patch_download_with_manifests(
monkeypatch,
tmp_path,
{
"esphome/A": {
"name": "A",
# The duplicate entry warns once (reconciliation dedup)
"dependencies": [{"name": "Hash"}, {"name": "Hash"}],
}
},
)
convert_libraries(
[Library("esphome/A", None, None)], _backend(provides=lambda name: False)
)
assert (
caplog.text.count(
"Hash of esphome/A has no version to resolve and nothing provides it"
)
== 1
)
caplog.clear()
with caplog.at_level(logging.DEBUG):
convert_libraries([Library("esphome/A", None, None)], _backend())
records = [
r
for r in caplog.records
if "has no version to resolve and nothing provides it" in r.message
]
assert records and all(r.levelno == logging.DEBUG for r in records)
def test_url_version_dependency_is_not_substituted_by_provides(
tmp_path, monkeypatch, caplog: pytest.LogCaptureFixture
) -> None:
"""A URL-valued version names one specific source; the backend-provided
skip must not replace it with the bundled copy."""
_patch_download_with_manifests(
monkeypatch,
tmp_path,
{
"esphome/A": {
"name": "A",
"dependencies": [
{"name": "Hash", "version": "https://github.com/o/Hash.git"}
],
},
"o/Hash": {"name": "Hash"},
},
)
emitted: list[str] = []
convert_libraries(
[Library("esphome/A", "1.0.0", None)],
_backend(emit=lambda c: emitted.append(c.name), provides=lambda name: True),
)
assert "Skip backend-provided" not in caplog.text
assert "using the library bundled" not in caplog.text
assert any("o/hash" in n.lower() for n in emitted)
def test_versionless_owner_qualified_dependency_warns_despite_provides(
tmp_path, monkeypatch, caplog: pytest.LogCaptureFixture
) -> None:
"""An owner-qualified version-less dependency is not satisfied by
provides(); it must still warn."""
_patch_download_with_manifests(
monkeypatch,
tmp_path,
{
"esphome/A": {
"name": "A",
"dependencies": [{"name": "Wire", "owner": "Foo"}],
}
},
)
convert_libraries(
[Library("esphome/A", None, None)],
_backend(provides=lambda name: name == "Wire"),
)
assert "Wire of esphome/A has no version to resolve" in caplog.text
def test_versionless_provided_dependency_stays_quiet(
tmp_path, monkeypatch, caplog: pytest.LogCaptureFixture
) -> None:
"""An owner-less version-less dependency the backend provides is added
by the backend after emit; no reconciliation warning."""
_patch_download_with_manifests(
monkeypatch,
tmp_path,
{"esphome/A": {"name": "A", "dependencies": [{"name": "Wire"}]}},
)
convert_libraries(
[Library("esphome/A", None, None)],
_backend(provides=lambda name: name == "Wire"),
)
assert "has no version to resolve" not in caplog.text
def test_versionless_dependency_requested_top_level_stays_quiet(
tmp_path, monkeypatch, caplog: pytest.LogCaptureFixture
) -> None:
"""A version-less dependency the config also requests top-level is in
the build; no drop warning even without a provides backend."""
_patch_download_with_manifests(
monkeypatch,
tmp_path,
{
"esphome/A": {"name": "A", "dependencies": [{"name": "Hash"}]},
"Hash": {"name": "Hash"},
},
)
convert_libraries(
[Library("esphome/A", None, None), Library("Hash", None, None)],
_backend(),
)
assert "has no version to resolve" not in caplog.text
def test_versionless_url_ish_dependency_name_warns_cleanly(
tmp_path, monkeypatch, caplog: pytest.LogCaptureFixture
) -> None:
"""A malformed URL-ish dependency name falls to the drop warning, never
a RuntimeError out of the key parser."""
_patch_download_with_manifests(
monkeypatch,
tmp_path,
{"esphome/A": {"name": "A", "dependencies": [{"name": "file://"}]}},
)
convert_libraries(
[Library("esphome/A", None, None)], _backend(provides=lambda name: False)
)
assert (
"file:// of esphome/A has no version to resolve and nothing provides it"
in caplog.text
)
def test_versionless_dependency_matching_resolved_manifest_name_stays_quiet(
tmp_path, monkeypatch, caplog: pytest.LogCaptureFixture
) -> None:
"""A bare name satisfied by an owner-qualified component's manifest
name is not a drop."""
_patch_download_with_manifests(
monkeypatch,
tmp_path,
{
"esphome/A": {"name": "A", "dependencies": [{"name": "B"}]},
"esphome/B": {"name": "B"},
},
)
convert_libraries(
[Library("esphome/A", None, None), Library("esphome/B", None, None)],
_backend(),
)
assert "has no version to resolve" not in caplog.text
@@ -1,581 +0,0 @@
"""Tests for esphome/platformio/pch.py.script against a fake SCons env."""
from __future__ import annotations
import os
from pathlib import Path
import stat
from unittest.mock import patch
import pytest
from esphome.platformio import toolchain
pytestmark = pytest.mark.skipif(
os.name == "nt", reason="the fake compiler is a POSIX shell script"
)
_SCRIPT = Path(toolchain.__file__).parent / "pch.py.script"
class _FakePlatform:
packages = {"framework-x": {}, "toolchain-y": {}}
def get_package_version(self, name: str) -> str:
if name == "toolchain-y":
raise KeyError(name)
return "1.2.3"
def get_package(self, name: str) -> object | None:
return None
class _BrokenPlatform(_FakePlatform):
def get_package_version(self, name: str) -> str:
raise RuntimeError("manifest parse error")
class _UnresolvedPlatform(_FakePlatform):
"""KeyError from a package that IS installed: unresolved identity."""
def get_package_version(self, name: str) -> str:
raise KeyError(name)
def get_package(self, name: str) -> object:
return object()
class _FakeSConsEnv(dict):
"""Just enough of a SCons construction environment for pch.py."""
def __init__(
self,
proj_dir: Path,
src_dir: Path,
cxx: str,
flags: list[str],
platform_cls: type[_FakePlatform] = _FakePlatform,
):
super().__init__(ENV={})
self._subst = {
"$PROJECT_DIR": str(proj_dir),
"$PROJECT_SRC_DIR": str(src_dir),
"$CXX": cxx,
}
self._flags = flags
self._platform_cls = platform_cls
self.prepended: list[str] = []
def subst(self, expr: str) -> str: # noqa: N802
return self._subst[expr]
def subst_list(self, expr: str) -> list[list[str]]: # noqa: N802
return [self._flags]
def PioPlatform(self) -> _FakePlatform: # noqa: N802
return self._platform_cls()
def Prepend(self, CXXFLAGS: list[str]) -> None: # noqa: N802, N803
self.prepended = CXXFLAGS
def _fake_cxx(
tmp_path: Path,
fail: bool = False,
fail_msg: str | None = None,
reject_pch: bool = False,
probe_exit: int = 0,
) -> Path:
"""A compiler stand-in that records its argv and writes the -o target.
With reject_pch it builds the .gch fine but, like GCC 10 on macOS arm64,
warns on any consuming compile that the .gch cannot be loaded; probe_exit
sets the exit code of non-header compiles (the load probe).
"""
cxx = tmp_path / "fake-gxx"
body = (
'printf -- ---call---\\\\n >> "$0.argv"; printf \'%s\\n\' "$@" >> "$0.argv"\n'
)
if fail:
body += f"echo {fail_msg or 'boom'} >&2\nexit 1\n"
else:
# Only the c++-header compile has a -o; the load probe has none
body += 'out=""; prev=""; mf=0; dep=0; inc=0\nfor a in "$@"; do [ "$prev" = "-o" ] && out="$a"; prev="$a"; [ "$a" = "-MF" ] && mf=1; [ "$a" = "-include" ] && inc=1; case "$a" in -M|-MM|-MD|-MMD) dep=1;; esac; done\n'
# Real cc1plus rejects -MF without a dependency flag
body += 'if [ "$mf" = 1 ] && [ "$dep" = 0 ]; then echo "cc1plus: error: to generate dependencies you must specify either \x27-M\x27 or \x27-MM\x27" >&2; exit 1; fi\n'
body += '[ -n "$out" ] && echo gch > "$out"\n'
if reject_pch:
# -Werror=invalid-pch makes rejection a nonzero exit; the
# baseline (no -include) still passes
body += 'case " $* " in *c++-header*) ;; *) if [ "$inc" = 1 ]; then echo "error: esphome_pch.h.gch: had text segment at different address" >&2; exit 1; fi;; esac\n'
body += f'case " $* " in *c++-header*) exit 0;; *) exit {probe_exit};; esac\n'
cxx.write_text("#!/bin/sh\n" + body)
cxx.chmod(cxx.stat().st_mode | stat.S_IEXEC)
return cxx
def _run_script(
tmp_path: Path,
flags: list[str] | None = None,
fail: bool = False,
fail_msg: str | None = None,
reject_pch: bool = False,
probe_exit: int = 0,
missing_cxx: bool = False,
env_vars: dict[str, str] | None = None,
name: str = "dev",
platform_cls: type[_FakePlatform] = _FakePlatform,
) -> _FakeSConsEnv:
proj = tmp_path / name
src = proj / "src"
(src / "esphome" / "core").mkdir(parents=True, exist_ok=True)
(src / "esphome" / "core" / "defines.h").write_text("#define USE_X\n")
cxx = _fake_cxx(
tmp_path,
fail=fail,
fail_msg=fail_msg,
reject_pch=reject_pch,
probe_exit=probe_exit,
)
if missing_cxx:
cxx = tmp_path / "no-such-gxx"
args = (proj, src, str(cxx), flags or ["-DX=1"], platform_cls)
# Distinct objects: the -include flags must land on projenv only
global_env = _FakeSConsEnv(*args)
projenv = _FakeSConsEnv(*args)
projenv.global_env = global_env
source = _SCRIPT.read_text()
with patch.dict(os.environ, env_vars or {}, clear=True):
exec( # noqa: S102
compile(source, "pch.py", "exec"),
{"Import": lambda *_names: None, "env": global_env, "projenv": projenv},
)
return projenv
def test_pch_script_builds_and_prepends_relative_include(tmp_path: Path) -> None:
scons_env = _run_script(tmp_path)
proj = tmp_path / "dev"
assert (proj / "esphome_pch.h").read_text().endswith('"esphome/core/defines.h"\n')
assert (proj / "esphome_pch.h.gch").is_file()
assert len((proj / "esphome_pch.h.gch.sum").read_text().strip()) == 64
# Relative include: an absolute path would poison ccache keys
assert scons_env.prepended == [
"-Winvalid-pch",
"-Wno-error=invalid-pch",
"-include",
"esphome_pch.h",
]
# In production projenv["ENV"] aliases os.environ; only the -include
# flags are genuinely scoped to projenv (src compiles)
assert scons_env["ENV"]["CCACHE_SLOPPINESS"] == "pch_defines,time_macros"
assert scons_env["ENV"]["CCACHE_PCH_EXTSUM"] == "true"
assert scons_env.global_env.prepended == []
def test_pch_script_preserves_spaced_flag_elements(tmp_path: Path) -> None:
"""One SCons element stays one compiler argv; -include pairs are
stripped from the .gch compile."""
spaced = tmp_path / "My Configs"
spaced.mkdir()
(tmp_path / "dev" / "src").mkdir(parents=True, exist_ok=True)
(tmp_path / "dev" / "src" / "other.h").write_text("")
flags = ['-DUSB_PRODUCT=\\"Pico 2W\\"', "-I", str(spaced), "-include", "other.h"]
_run_script(tmp_path, flags=flags)
calls = (tmp_path / "fake-gxx.argv").read_text().split("---call---\n")
gch_call = next(c for c in calls if "c++-header" in c).splitlines()
assert '-DUSB_PRODUCT="Pico 2W"' in gch_call
assert str(spaced) in gch_call
assert "-include" not in gch_call
# The stripped src-resolvable -include is folded into the prefix header
pch = (tmp_path / "dev" / "esphome_pch.h").read_text()
assert pch.splitlines()[0] == '#include "other.h"'
def test_pch_script_folds_joined_force_include_spelling(tmp_path: Path) -> None:
"""-includefoo.h folds like the separated form, matching the native path."""
(tmp_path / "dev" / "src").mkdir(parents=True, exist_ok=True)
(tmp_path / "dev" / "src" / "other.h").write_text("")
_run_script(tmp_path, flags=["-DX=1", "-includeother.h"])
pch = (tmp_path / "dev" / "esphome_pch.h").read_text()
assert pch.splitlines()[0] == '#include "other.h"'
def test_pch_script_leaves_absolute_force_includes_unfolded(
tmp_path: Path,
) -> None:
"""An absolute -include resolves through src_dir / name; it must still
stay consumer-only or the host path enters the .sum."""
outside = tmp_path / "outside.h"
outside.write_text("")
_run_script(tmp_path, flags=["-DX=1", "-include", str(outside)])
pch = (tmp_path / "dev" / "esphome_pch.h").read_text()
assert "outside.h" not in pch
def test_pch_script_leaves_non_src_force_includes_unfolded(
tmp_path: Path, capsys: pytest.CaptureFixture[str]
) -> None:
"""A user -include outside src/ must not enter the prefix header:
consumers keep their own copy, so folding an unguarded header would
include it twice."""
_run_script(tmp_path, flags=["-DX=1", "-include", "user_extra.h"])
pch = (tmp_path / "dev" / "esphome_pch.h").read_text()
assert "user_extra.h" not in pch
assert pch.splitlines()[-1] == '#include "esphome/core/defines.h"'
assert "not precompiling non-src force-includes" in capsys.readouterr().out
def test_pch_script_sum_is_device_independent(tmp_path: Path) -> None:
"""Regression: identical configs in different dirs share cache keys."""
sums = []
for name in ("dev_a", "dev_b"):
proj = tmp_path / name
_run_script(
tmp_path,
flags=["-DX=1", "-I", str(proj / "include")],
env_vars={"CCACHE_BASEDIR": str(proj)},
name=name,
)
sums.append((proj / "esphome_pch.h.gch.sum").read_text())
(tmp_path / "fake-gxx").unlink()
(tmp_path / "fake-gxx.argv").unlink(missing_ok=True)
assert sums[0] == sums[1]
def test_pch_script_transient_compiler_failure_does_not_latch(
tmp_path: Path,
) -> None:
"""ENOSPC-style failures clear on their own; no .failed marker."""
scons_env = _run_script(tmp_path, fail=True, fail_msg="No space left on device")
proj = tmp_path / "dev"
assert not (proj / "esphome_pch.h.gch.failed").exists()
assert scons_env.prepended == []
def test_pch_script_failure_marker_suppresses_retry(
tmp_path: Path, capsys: pytest.CaptureFixture[str]
) -> None:
scons_env = _run_script(tmp_path, fail=True)
proj = tmp_path / "dev"
assert (proj / "esphome_pch.h.gch.failed").is_file()
assert not (proj / "esphome_pch.h.gch.sum").exists()
assert scons_env.prepended == []
# Second run: same checksum, no compile attempt, but says so
attempts = (tmp_path / "fake-gxx.argv").read_text().count("c++-header")
_run_script(tmp_path, fail=True)
out = capsys.readouterr().out
assert (tmp_path / "fake-gxx.argv").read_text().count("c++-header") == attempts
assert "delete esphome_pch.h.gch.failed to retry" in out
def test_pch_script_probe_rejection_falls_back(
tmp_path: Path, capsys: pytest.CaptureFixture[str]
) -> None:
"""A toolchain that cannot load its own .gch (GCC 10 on macOS arm64)
must not leave consumers paying for a pch every compile rejects."""
scons_env = _run_script(tmp_path, reject_pch=True)
proj = tmp_path / "dev"
assert not (proj / "esphome_pch.h.gch").exists()
assert not (proj / "esphome_pch.h.gch.sum").exists()
assert (proj / "esphome_pch.h.gch.failed").is_file()
assert scons_env.prepended == []
assert "toolchain cannot load the pch" in capsys.readouterr().out
def test_pch_script_spawn_failure_is_transient(
tmp_path: Path, capsys: pytest.CaptureFixture[str]
) -> None:
"""A spawn failure must not latch a .failed marker (matches espidf)."""
scons_env = _run_script(tmp_path, missing_cxx=True)
proj = tmp_path / "dev"
assert not (proj / "esphome_pch.h.gch.failed").exists()
assert not (proj / "esphome_pch.h.gch.sum").exists()
assert scons_env.prepended == []
assert "did not run" in capsys.readouterr().out
def test_pch_script_probe_baseline_failure_latches_with_honest_label(
tmp_path: Path, capsys: pytest.CaptureFixture[str]
) -> None:
"""Probe AND baseline failing is a deterministic environment problem:
latch, but blame the environment rather than the pch."""
scons_env = _run_script(tmp_path, probe_exit=1)
proj = tmp_path / "dev"
assert not (proj / "esphome_pch.h.gch").exists()
assert (proj / "esphome_pch.h.gch.failed").is_file()
assert scons_env.prepended == []
assert "probe cannot run at all" in capsys.readouterr().out
def test_pch_script_unresolved_package_version_skips_pch(tmp_path: Path) -> None:
"""A KeyError for an installed package is unresolved identity, not absence."""
scons_env = _run_script(tmp_path, platform_cls=_UnresolvedPlatform)
assert not (tmp_path / "dev" / "esphome_pch.h.gch").exists()
assert scons_env.prepended == []
def test_pch_script_package_version_error_skips_pch(tmp_path: Path) -> None:
"""Without trustworthy package identity a stale .gch could survive an
upgrade, so the script must not build one at all."""
scons_env = _run_script(tmp_path, platform_cls=_BrokenPlatform)
proj = tmp_path / "dev"
assert not (proj / "esphome_pch.h.gch").exists()
assert scons_env.prepended == []
def test_pch_script_corrupt_sidecar_reads_as_stale(tmp_path: Path) -> None:
"""A truncated/corrupt .failed marker must not disable the pch forever."""
_run_script(tmp_path, fail=True)
proj = tmp_path / "dev"
(proj / "esphome_pch.h.gch.failed").write_bytes(b"\xff\xfe corrupt")
_run_script(tmp_path)
assert (proj / "esphome_pch.h.gch").is_file()
assert (proj / "esphome_pch.h.gch.sum").is_file()
def test_pch_script_rebuilds_when_header_missing(tmp_path: Path) -> None:
_run_script(tmp_path)
proj = tmp_path / "dev"
(proj / "esphome_pch.h").unlink()
_run_script(tmp_path)
assert (proj / "esphome_pch.h").is_file()
def test_copy_pch_script(tmp_path: Path) -> None:
from esphome.core import CORE
CORE.build_path = tmp_path
toolchain.copy_pch_script()
assert (tmp_path / "pch.py").read_text() == _SCRIPT.read_text()
def test_pch_script_unions_user_sloppiness(
tmp_path: Path, capsys: pytest.CaptureFixture[str]
) -> None:
"""A user CCACHE_SLOPPINESS without the pch tokens gets them unioned on,
mirroring ccache_pch_env, or every src TU is a permanent miss."""
scons_env = _run_script(tmp_path, env_vars={"CCACHE_SLOPPINESS": "locale"})
assert scons_env["ENV"]["CCACHE_SLOPPINESS"] == "locale,pch_defines,time_macros"
assert "adding pch_defines,time_macros" in capsys.readouterr().out
def test_pch_script_unmodelable_flag_skips_pch(
tmp_path: Path, capsys: pytest.CaptureFixture[str]
) -> None:
"""Unbalanced quotes and multi-token elements cannot be reproduced as
one argv; the pch is skipped rather than built with diverging flags."""
for bad in ("-DFOO='bar", "-DA=1\t-DB=2"):
scons_env = _run_script(tmp_path, flags=["-DX=1", bad])
assert scons_env.prepended == []
assert not (tmp_path / "dev" / "esphome_pch.h.gch").exists()
assert "unmodelable flag" in capsys.readouterr().out
@pytest.mark.skipif(
getattr(os, "geteuid", lambda: -1)() == 0, reason="root ignores file modes"
)
def test_pch_script_unlistable_include_dir_skips_pch(
tmp_path: Path, capsys: pytest.CaptureFixture[str]
) -> None:
"""An unlistable subtree must not silently drop out of the digest."""
proj = tmp_path / "dev"
override = proj / "lwip_override"
hidden = override / "hidden"
hidden.mkdir(parents=True)
(hidden / "gen.h").write_text("")
hidden.chmod(0)
try:
scons_env = _run_script(tmp_path, flags=["-DX=1", "-I", str(override)])
finally:
hidden.chmod(0o755)
assert scons_env.prepended == []
assert "skipping precompiled header" in capsys.readouterr().out
def test_pch_script_nobuild_without_projenv_is_noop(tmp_path: Path) -> None:
"""-t nobuild never exports projenv; the script must not abort."""
proj = tmp_path / "dev"
(proj / "src").mkdir(parents=True)
def strict_import(*names: str) -> None:
if "projenv" in names:
raise RuntimeError("Import of non-existent variable 'projenv'")
env = _FakeSConsEnv(proj, proj / "src", "g++", ["-DX=1"])
exec( # noqa: S102
compile(_SCRIPT.read_text(), "pch.py", "exec"),
{"Import": strict_import, "env": env},
)
assert not (proj / "esphome_pch.h").exists()
def test_pch_script_ignores_library_trees_and_non_headers(tmp_path: Path) -> None:
""".piolibdeps and non-header files must not enter the digest (or be
read at all); package versions already cover library identity."""
proj = tmp_path / "dev"
libdeps = proj / ".piolibdeps" / "lib" / "src"
libdeps.mkdir(parents=True)
(libdeps / "lib.h").write_text("#define A 1\n")
override = proj / "lwip_override"
override.mkdir(parents=True)
(override / "lwipopts.h").write_text("#define TCP_MSS 1460\n")
(override / "notes.txt").write_text("v1\n")
flags = ["-DX=1", "-I", str(libdeps), "-I", str(override)]
_run_script(tmp_path, flags=flags)
first = (proj / "esphome_pch.h.gch.sum").read_text()
(libdeps / "lib.h").write_text("#define A 2\n")
(override / "notes.txt").write_text("v2\n")
(tmp_path / "fake-gxx.argv").unlink(missing_ok=True)
_run_script(tmp_path, flags=flags)
assert (proj / "esphome_pch.h.gch.sum").read_text() == first
def test_pch_script_hashes_project_local_include_dirs(tmp_path: Path) -> None:
"""Generated headers in project-local -I dirs (e.g. rp2's lwip_override)
must invalidate the checksum when they change."""
proj = tmp_path / "dev"
override = proj / "lwip_override"
override.mkdir(parents=True)
(override / "lwipopts.h").write_text("#define TCP_MSS 1460\n")
flags = ["-DX=1", "-I", str(override)]
_run_script(tmp_path, flags=flags)
first = (proj / "esphome_pch.h.gch.sum").read_text()
(override / "lwipopts.h").write_text("#define TCP_MSS 536\n")
(tmp_path / "fake-gxx.argv").unlink(missing_ok=True)
_run_script(tmp_path, flags=flags)
assert (proj / "esphome_pch.h.gch.sum").read_text() != first
@pytest.mark.skipif(
getattr(os, "geteuid", lambda: -1)() == 0, reason="root ignores file modes"
)
def test_pch_script_unreadable_local_header_skips_pch(
tmp_path: Path, capsys: pytest.CaptureFixture[str]
) -> None:
"""An unreadable generated header means unknown identity: no pch."""
proj = tmp_path / "dev"
override = proj / "lwip_override"
override.mkdir(parents=True)
secret = override / "lwipopts.h"
secret.write_text("#define TCP_MSS 1460\n")
secret.chmod(0)
scons_env = _run_script(tmp_path, flags=["-DX=1", "-I", str(override)])
assert not (proj / "esphome_pch.h.gch.sum").exists()
assert scons_env.prepended == []
assert "skipping precompiled header" in capsys.readouterr().out
def test_pch_script_rejects_unrecognized_strict_value(tmp_path: Path) -> None:
"""A typo'd knob must fail the build, not silently disable the gate."""
with pytest.raises(RuntimeError, match="Unrecognized ESPHOME_PCH_STRICT"):
_run_script(tmp_path, env_vars={"ESPHOME_PCH_STRICT": "yolo"})
def test_pch_script_strict_tables_match_helpers(tmp_path: Path) -> None:
"""The script's mirrored spelling tables must not drift."""
from esphome.helpers import FALSY_ENV_STRINGS, TRUTHY_ENV_STRINGS
proj = tmp_path / "dev"
(proj / "src").mkdir(parents=True)
# Hermetic: the constants are module-level, but the exec still runs
# _setup_pch, which must not touch the host toolchain
cxx = _fake_cxx(tmp_path)
env = _FakeSConsEnv(proj, proj / "src", str(cxx), ["-DX=1"])
namespace = {"Import": lambda *_names: None, "env": env, "projenv": env}
with patch.dict(os.environ, {}, clear=True):
exec(compile(_SCRIPT.read_text(), "pch.py", "exec"), namespace) # noqa: S102
assert set(namespace["_TRUTHY"]) == set(TRUTHY_ENV_STRINGS)
# parse_enable_env handles the empty string separately
assert set(namespace["_FALSY"]) - {""} == set(FALSY_ENV_STRINGS)
def test_pch_script_strict_reprobes_cached_gch(tmp_path: Path) -> None:
"""Rejection is per-process: strict re-proves a cached .gch loads."""
_run_script(tmp_path)
proj = tmp_path / "dev"
assert (proj / "esphome_pch.h.gch").is_file()
(tmp_path / "fake-gxx.argv").unlink(missing_ok=True)
# Second run: cache fresh, but the toolchain now rejects loads
with pytest.raises(RuntimeError, match="not used"):
_run_script(tmp_path, reject_pch=True, env_vars={"ESPHOME_PCH_STRICT": "1"})
assert not (proj / "esphome_pch.h.gch").exists()
def test_pch_script_strict_raises_when_pch_not_used(tmp_path: Path) -> None:
"""ESPHOME_PCH_STRICT fails the build instead of degrading."""
with pytest.raises(RuntimeError, match="ESPHOME_PCH_STRICT"):
_run_script(tmp_path, fail=True, env_vars={"ESPHOME_PCH_STRICT": "1"})
def test_pch_script_strict_fails_without_scons(tmp_path: Path) -> None:
"""No SCons under PlatformIO is an anomaly; strict must not pass."""
proj = tmp_path / "dev"
(proj / "src").mkdir(parents=True)
def strict_import(*names: str) -> None:
if "projenv" in names:
raise RuntimeError("Import of non-existent variable 'projenv'")
import sys
env = _FakeSConsEnv(proj, proj / "src", "g++", ["-DX=1"])
with (
# None forces ImportError even where SCons is installed
patch.dict(sys.modules, {"SCons.Script": None}),
patch.dict(os.environ, {"ESPHOME_PCH_STRICT": "1"}, clear=True),
pytest.raises(RuntimeError, match="not used"),
):
exec( # noqa: S102
compile(_SCRIPT.read_text(), "pch.py", "exec"),
{"Import": strict_import, "env": env},
)
def test_pch_script_strict_reraises_internal_errors(tmp_path: Path) -> None:
"""The catch-all must not swallow programming errors in strict mode."""
with pytest.raises(TypeError):
_run_script(tmp_path, env_vars={"ESPHOME_PCH_STRICT": "1"}, platform_cls=None)
@pytest.mark.parametrize(("targets", "passes"), [(["nobuild"], True), ([], False)])
def test_pch_script_strict_projenv_skip_gated_on_nobuild(
tmp_path: Path, targets: list[str], passes: bool
) -> None:
"""-t nobuild compiles nothing, so the skip passes strict; a missing
projenv on a real compile must not."""
import sys
import types
proj = tmp_path / "dev"
(proj / "src").mkdir(parents=True)
def strict_import(*names: str) -> None:
if "projenv" in names:
raise RuntimeError("Import of non-existent variable 'projenv'")
scons = types.ModuleType("SCons")
scons_script = types.ModuleType("SCons.Script")
scons_script.COMMAND_LINE_TARGETS = targets
env = _FakeSConsEnv(proj, proj / "src", "g++", ["-DX=1"])
with (
patch.dict(sys.modules, {"SCons": scons, "SCons.Script": scons_script}),
patch.dict(os.environ, {"ESPHOME_PCH_STRICT": "1"}, clear=True),
):
run = lambda: exec( # noqa: S102, E731
compile(_SCRIPT.read_text(), "pch.py", "exec"),
{"Import": strict_import, "env": env},
)
if passes:
run()
else:
with pytest.raises(RuntimeError, match="not used"):
run()
assert not (proj / "esphome_pch.h").exists()
def test_pch_script_strict_passes_on_success(tmp_path: Path) -> None:
scons_env = _run_script(tmp_path, env_vars={"ESPHOME_PCH_STRICT": "1"})
# Strict escalates the consumer edges too
assert "-Werror=invalid-pch" in scons_env.prepended
assert "-Wno-error=invalid-pch" not in scons_env.prepended
@@ -1729,20 +1729,3 @@ def test_platformio_private_api_contract() -> None:
derived = PackageSpec("https://x/y/archive/master.zip")
assert derived.name and not derived.has_custom_name()
assert PackageSpec("Foo=https://x/y/archive/master.zip").has_custom_name()
def test_preinstall_caps_workers(
tmp_path: Path, caplog: pytest.LogCaptureFixture
) -> None:
"""A high core count is capped; the workers share one disk."""
with (
caplog.at_level(logging.INFO),
patch.object(pf, "get_usable_cpu_count", return_value=64),
):
pf._preinstall(
_fake_manager(tmp_path),
[(f"p{i}@1", _FakeSpec(name=f"p{i}")) for i in range(11)],
)
assert "Installing 11 PlatformIO package(s) with 10 extraction worker(s)" in (
caplog.text
)
+1 -258
View File
@@ -4,7 +4,6 @@ from __future__ import annotations
from contextlib import contextmanager
import json
import logging
import os
from pathlib import Path
from unittest.mock import MagicMock, patch
@@ -308,11 +307,7 @@ def test_install_package_downloads_via_registry(tmp_path: Path) -> None:
"pkg", "1.0.0", dest, [], tmp_path / "dl", expect=("payload",)
)
assert mock_download.call_args[0][0] == "http://x/pkg.tar.gz"
assert mock_download.call_args[1] == {
"sha256": "abc123",
"size": 42,
"progress": None,
}
assert mock_download.call_args[1] == {"sha256": "abc123", "size": 42}
def test_install_package_validates_expected_layout(tmp_path: Path) -> None:
@@ -728,255 +723,3 @@ def test_prefetch_packages_unexpected_failure_warns(
tmp_path / "dl",
)
assert "TypeError" in caplog.text
def _spec(name: str, version: str, dest: Path, mirrors=None, expect=("payload",)):
return (name, version, dest, mirrors or [], expect)
def test_install_packages_extracts_verified_archives_in_parallel(
tmp_path: Path,
) -> None:
"""Two prefetched archives install concurrently under one shared bar."""
dl = tmp_path / "dl"
dl.mkdir()
(dl / "a-1.0").write_bytes(b"x" * 10)
(dl / "b-2.0").write_bytes(b"y" * 20)
with patch.object(registry, "install_package") as mock_install:
registry.install_packages(
[_spec("a", "1.0", tmp_path / "a"), _spec("b", "2.0", tmp_path / "b")], dl
)
assert mock_install.call_count == 2
calls = sorted(mock_install.call_args_list, key=lambda c: c[0][0])
for c, (name, version) in zip(calls, [("a", "1.0"), ("b", "2.0")], strict=True):
assert c[0][:3] == (name, version, tmp_path / name)
assert c[1]["expect"] == ("payload",)
assert callable(c[1]["extract_progress"])
# Driving the tracker exercises the fraction-to-bytes scaling
c[1]["extract_progress"](0.5)
c[1]["extract_progress"](1.0)
def test_install_packages_single_archive_stays_sequential(tmp_path: Path) -> None:
"""One verified archive has nothing to parallelize; original order kept."""
dl = tmp_path / "dl"
dl.mkdir()
(dl / "a-1.0").write_bytes(b"x")
specs = [_spec("a", "1.0", tmp_path / "a"), _spec("b", "2.0", tmp_path / "b")]
with patch.object(registry, "install_package") as mock_install:
registry.install_packages(specs, dl)
assert [c[0][0] for c in mock_install.call_args_list] == ["a", "b"]
for c in mock_install.call_args_list:
assert "extract_progress" not in c[1]
def test_install_packages_mirror_and_marker_stay_sequential(tmp_path: Path) -> None:
"""Mirror overrides and marker hits never enter the parallel batch."""
dl = tmp_path / "dl"
dl.mkdir()
for name, ver in (("a", "1.0"), ("b", "2.0"), ("c", "3.0"), ("d", "4.0")):
(dl / f"{name}-{ver}").write_bytes(b"x")
marked = tmp_path / "c"
marked.mkdir()
(marked / ".esphome_extracted").touch()
specs = [
_spec("a", "1.0", tmp_path / "a"),
_spec("b", "2.0", tmp_path / "b", mirrors=["http://m"]),
_spec("c", "3.0", marked),
_spec("d", "4.0", tmp_path / "d"),
]
with patch.object(registry, "install_package") as mock_install:
registry.install_packages(specs, dl)
sequential = [
c for c in mock_install.call_args_list if "extract_progress" not in c[1]
]
batched = [c for c in mock_install.call_args_list if "extract_progress" in c[1]]
assert sorted(c[0][0] for c in sequential) == ["b", "c"]
assert sorted(c[0][0] for c in batched) == ["a", "d"]
def test_install_packages_first_failure_reraised(
tmp_path: Path, caplog: pytest.LogCaptureFixture
) -> None:
"""Installs are mandatory: the first failure propagates, extras are logged."""
dl = tmp_path / "dl"
dl.mkdir()
(dl / "a-1.0").write_bytes(b"x")
(dl / "b-2.0").write_bytes(b"y")
boom = EsphomeError("bad layout")
def _fail(name: str, *_a, **_kw) -> None:
raise boom if name == "a" else EsphomeError("also bad")
with (
patch.object(registry, "install_package", side_effect=_fail),
pytest.raises(EsphomeError),
):
registry.install_packages(
[_spec("a", "1.0", tmp_path / "a"), _spec("b", "2.0", tmp_path / "b")], dl
)
# Every failure is named, including the re-raised one: its exception
# message may not identify the package
assert "Could not install a" in caplog.text
assert "Could not install b" in caplog.text
@contextmanager
def _batched_install(tmp_path: Path, extract_progress, prefill_archive: bool = True):
"""Run a batched install_package of pkg@1.0.0; yields the download mock."""
dest = tmp_path / "pkg"
if prefill_archive:
(tmp_path / "dl").mkdir()
(tmp_path / "dl" / "pkg-1.0.0").write_bytes(b"x")
with (
patch.object(registry, "download_with_resume") as mock_download,
patch.object(registry, "archive_extract_all") as mock_extract,
patch.object(
registry,
"registry_download",
return_value=("http://x/pkg.tar.gz", "abc123", 42),
),
):
mock_extract.side_effect = lambda *_a, **_kw: (dest / "payload").mkdir(
parents=True
)
registry.install_package(
"pkg",
"1.0.0",
dest,
[],
tmp_path / "dl",
expect=("payload",),
extract_progress=extract_progress,
)
yield mock_download, mock_extract
def test_install_package_extract_progress_suppresses_bars(
tmp_path: Path, caplog: pytest.LogCaptureFixture
) -> None:
"""A batched install routes extraction fractions to the caller and keeps
both private bars and per-package INFO lines off the shared bar."""
fractions: list[float] = []
with (
caplog.at_level(logging.INFO),
_batched_install(tmp_path, fractions.append) as (mock_download, mock_extract),
):
pass
assert mock_extract.call_args[1]["progress"] == fractions.append
# The download tracker reports zero bytes, keeping the shared bar honest
download_progress = mock_download.call_args[1]["progress"]
assert callable(download_progress)
download_progress(42)
assert fractions == [0.0]
assert "Downloading pkg" not in caplog.text
assert "Extracting pkg" not in caplog.text
def test_install_package_batched_missing_archive_keeps_info_log(
tmp_path: Path, caplog: pytest.LogCaptureFixture
) -> None:
"""A batched archive that unexpectedly needs a real download keeps the
INFO line; the shared bar shows no progress for it."""
with (
caplog.at_level(logging.INFO),
_batched_install(tmp_path, lambda _frac: None, prefill_archive=False),
):
pass
assert "Downloading pkg 1.0.0" in caplog.text
def test_install_packages_dedupes_duplicate_specs(tmp_path: Path) -> None:
"""Duplicate (name, version) entries share one archive and would race
each other; the duplicate takes the sequential path."""
dl = tmp_path / "dl"
dl.mkdir()
(dl / "a-1.0").write_bytes(b"x")
(dl / "b-2.0").write_bytes(b"y")
specs = [
_spec("a", "1.0", tmp_path / "a"),
_spec("a", "1.0", tmp_path / "a2"),
_spec("b", "2.0", tmp_path / "b"),
]
with patch.object(registry, "install_package") as mock_install:
registry.install_packages(specs, dl)
sequential = [
c for c in mock_install.call_args_list if "extract_progress" not in c[1]
]
batched = [c for c in mock_install.call_args_list if "extract_progress" in c[1]]
assert [(c[0][0], c[0][2]) for c in sequential] == [("a", tmp_path / "a2")]
assert sorted(c[0][0] for c in batched) == ["a", "b"]
# The duplicate runs after the batch, which unlinks their shared archive
assert mock_install.call_args_list[-1] == sequential[0]
def test_install_packages_caps_workers(tmp_path: Path) -> None:
"""A high core count is capped; the workers share one disk."""
dl = tmp_path / "dl"
dl.mkdir()
specs = []
for i in range(12):
(dl / f"p{i}-1.0").write_bytes(b"x")
specs.append(_spec(f"p{i}", "1.0", tmp_path / f"p{i}"))
with (
patch.object(registry, "get_usable_cpu_count", return_value=64),
patch.object(registry, "run_batch_downloads", return_value=[]) as batch,
patch.object(registry, "install_package"),
):
registry.install_packages(specs, dl)
assert batch.call_args.kwargs["max_workers"] == 10
def test_install_package_batched_refetch_announced_once(
tmp_path: Path, caplog: pytest.LogCaptureFixture
) -> None:
"""A batched archive that fails verification and refetches is announced;
a verify no-op (full size credited immediately) stays silent."""
with (
caplog.at_level(logging.INFO),
_batched_install(tmp_path, lambda _frac: None) as (mock_download, _),
):
progress = mock_download.call_args[1]["progress"]
progress(42)
assert "Re-downloading pkg 1.0.0" not in caplog.text
progress(10)
progress(20)
assert caplog.text.count("Re-downloading pkg 1.0.0") == 1
def test_install_package_batched_refetch_announced_without_size(
tmp_path: Path, caplog: pytest.LogCaptureFixture
) -> None:
"""A size-less registry entry still announces its refetch on the first
streaming tick."""
with (
caplog.at_level(logging.INFO),
patch.object(registry, "download_with_resume") as mock_download,
patch.object(registry, "archive_extract_all") as mock_extract,
patch.object(
registry,
"registry_download",
return_value=("http://x/pkg.tar.gz", "abc123", None),
),
):
dest = tmp_path / "pkg"
(tmp_path / "dl").mkdir()
(tmp_path / "dl" / "pkg-1.0.0").write_bytes(b"x")
mock_extract.side_effect = lambda *_a, **_kw: (dest / "payload").mkdir(
parents=True
)
registry.install_package(
"pkg",
"1.0.0",
dest,
[],
tmp_path / "dl",
expect=("payload",),
extract_progress=lambda _frac: None,
)
progress = mock_download.call_args[1]["progress"]
# A verify no-op credits the whole (nonempty) file in one tick
progress(1)
assert "Re-downloading pkg 1.0.0" not in caplog.text
progress(0)
assert "Re-downloading pkg 1.0.0" in caplog.text
+1 -31
View File
@@ -516,13 +516,10 @@ def test_clean_build(
dependencies_lock = tmp_path / "dependencies.lock"
dependencies_lock.write_text("lock file")
# idedata caches live under the data dir, not the build path; the
# .arduino.json variant is the native esp8266 toolchain's.
# idedata cache lives under the data dir, not the build path.
idedata_cache = tmp_path / "idedata" / "test.json"
idedata_cache.parent.mkdir()
idedata_cache.write_text("{}")
arduino_idedata_cache = tmp_path / "idedata" / "test.arduino.json"
arduino_idedata_cache.write_text("{}")
# Native ESP-IDF toolchain artifacts.
idf_build_dir = tmp_path / "build"
@@ -583,7 +580,6 @@ def test_clean_build(
assert not piolibdeps_dir.exists()
assert not dependencies_lock.exists()
assert not idedata_cache.exists()
assert not arduino_idedata_cache.exists()
assert not idf_build_dir.exists()
assert not managed_components_dir.exists()
assert not pio_components_dir.exists()
@@ -677,32 +673,6 @@ def test_clean_build_partial_exists(
assert "dependencies.lock" not in caplog.text
@patch("esphome.writer.CORE")
def test_clean_build_partial_removes_pch_artifacts(
mock_core: MagicMock,
tmp_path: Path,
) -> None:
"""The PlatformIO pch sidecars live at the project root and must go in
a partial clean, like the native backend's under .pioenvs."""
names = (
"esphome_pch.h",
"esphome_pch.h.gch",
"esphome_pch.h.gch.sum",
"esphome_pch.h.gch.failed",
)
for name in names:
(tmp_path / name).write_text("x")
mock_core.relative_pioenvs_path.return_value = tmp_path / ".pioenvs"
mock_core.relative_piolibdeps_path.return_value = tmp_path / ".piolibdeps"
mock_core.relative_build_path.side_effect = lambda name: tmp_path / name
mock_core.relative_internal_path.side_effect = tmp_path.joinpath
clean_build()
for name in names:
assert not (tmp_path / name).exists()
@patch("esphome.writer.CORE")
def test_clean_build_nothing_exists(
mock_core: MagicMock,