Merge pull request #20421 from esphome/bump-2026.10.0b2

2026.10.0b2

.agents/skills/code-review/SKILL.md

@@ -0,0 +1,116 @@
+---
+name: code-review
+description: Review guidance for ESPHome pull requests. Use this when reviewing a pull request that changes ESPHome Python, C++, or component code, to check it against the project's coding conventions, embedded-systems memory rules, testing requirements, and breaking-change policy.
+---
+
+# Reviewing ESPHome pull requests
+
+ESPHome parses YAML into C++ firmware for memory-constrained microcontrollers
+(ESP32, ESP8266, RP2040, LibreTiny). Review changes with that in mind: RAM and
+flash are scarce, and code runs unattended for months.
+
+`AGENTS.md` in the repository root is the full contributor guide and the
+authority when it disagrees with this summary. The developer documentation at
+https://developers.esphome.io explains the component lifecycle and the reasoning
+behind these rules. This skill lists the concrete things worth flagging in a
+review; read `AGENTS.md` for the detail behind any item.
+
+Only raise findings that the diff actually introduces or changes. Do not ask for
+drive-by cleanup of pre-existing code the PR did not touch.
+
+## Memory and embedded constraints (highest value)
+
+Heap allocation after `setup()` is treated as a reliability bug, not a
+performance nit, because it fragments a small shared heap. Flag:
+
+- New heap allocation on a hot path or after setup that could be avoided.
+- `std::vector` where the size is known at compile time (use `std::array`, or
+  `StaticVector<T, N>` when a `push_back` API is needed) or fixed at runtime
+  init (use `FixedVector<T>`).
+- Listener / child-entity registration lists stored as `std::vector`; these have
+  a compile-time-known count and should use `cg.slot_counter()` plus
+  `StaticVector`.
+- `std::vector<uint8_t>` for a byte buffer that never grows: prefer
+  `std::unique_ptr<uint8_t[]>` or `std::array`.
+- `std::map` / `std::set` / `std::unordered_map` for small datasets (1-16
+  elements): a `std::vector` of a small struct with linear search is lighter.
+- `std::deque` anywhere: it allocates 512-byte blocks and should be avoided.
+- `std::string` storing a value set once from config: prefer `StringRef` (the
+  literal already lives in flash).
+- `std::string` / `std::to_string` / string-returning helpers on hot paths where
+  a buffer or view API exists.
+
+## C++ conventions
+
+- Include what you use: a file referencing a symbol must include the header that
+  declares it, even if it currently arrives transitively. New or changed uses of
+  a symbol need the matching include.
+- Prefix all member access with `this->`.
+- Naming: `lower_snake_case` for functions/methods/variables, `UpperCamelCase`
+  for classes/structs/enums, `UPPER_SNAKE_CASE` for namespace-scope constants,
+  trailing underscore on protected/private fields.
+- `enum class` values must be prefixed with the enum name in `UPPER_SNAKE_CASE`
+  (e.g. `UARTFlushResult::UART_FLUSH_RESULT_SUCCESS`). Bare names like `SUCCESS`,
+  `FAIL`, or `OK` collide with SDK macros on some platforms and break the build.
+- Prefer `const`/`enum` over `#define`; `#define` is only for conditional
+  compilation and code-generation sizes.
+- Never call `millis()` in a `loop()` body; use
+  `App.get_loop_component_start_time()`. A rate-limit gate below ~16 ms (the loop
+  period) does nothing.
+- Pick the timing primitive by cadence: gated `loop()` under 250 ms,
+  `set_interval` at 500 ms and above.
+- Do not override a base method to return the value it already returns (e.g.
+  `get_setup_priority()` returning `setup_priority::DATA`).
+- Wrap string literals passed as printf `%s` args in `LOG_STR_LITERAL()`.
+- Required, invariant dependencies should be constructor parameters, not setters.
+- Callback registration methods must be templated (`template<typename F>`), not
+  typed as `std::function`, so lightweight forwarders avoid a heap allocation.
+- Two-space indent, `using` over `typedef`, wrap at 120 columns.
+
+## Python conventions
+
+- Type-annotate every new function signature (params and return), new dataclass
+  fields, and new module-level variables. Import `ConfigType` from
+  `esphome.types`.
+- Use the walrus operator to avoid a double lookup, e.g.
+  `if (blah := config.get(CONF_BLAH)) is not None:`.
+- Reuse existing validators from `config_validation.py` (`cv.rename_key`,
+  `cv.has_exactly_one_key`, etc.) via `cv.All(...)` instead of hand-rolling.
+- `esphome/const.py` is frozen: no new `CONF_` constants there. Define them in
+  the component's own `.py`, or in `esphome/components/const/__init__.py` when
+  shared. The same constant defined in three or more component files fails CI.
+- State that must persist during code generation goes in `CORE.data` namespaced
+  under the component `DOMAIN` (a `@dataclass`), not module-level mutable globals.
+- Prefer callback-based triggers via `build_callback_automation()`; only use a
+  `Trigger<Ts...>` subclass when the forwarder needs mutable state.
+
+## Testing and coverage
+
+- New and changed lines and branches need test coverage, including defensive
+  early-returns, error paths, and no-op guards. A mocked-out function is not
+  covered; exercise the real call path too.
+- Component YAML tests live in `tests/components/<component>/`. Never define
+  buses (uart, i2c, spi, modbus) directly in a test file: pull them from
+  `tests/test_build_components/common/` through dict-style `packages:` so CI can
+  group builds. List-style packages or top-level merge keys block grouping.
+- Config-only checks use the `validate.*.yaml` prefix; compiled checks use
+  `test.*.yaml`.
+
+## Breaking changes and public API
+
+- Base classes under `esphome/core/` and documented config options are public
+  API. Undocumented `public` members of a component are internal.
+- A breaking change needs justification, a migration path in the PR description,
+  and a deprecation window where feasible (`ESPDEPRECATED` in C++,
+  `cv.rename_key(..., removed_in=...)` in Python). Changing a codegen-injected
+  lambda signature is not a breaking change.
+
+## Process and PR hygiene
+
+- PR titles start with a `[tag]` prefix: the component name (e.g. `[uart] ...`)
+  or `[core]` for shared code.
+- Prose in docs, comments, and commit messages should be plain English. Keep
+  inline comments short and only where the code is not self-explanatory; do not
+  restate what the code says.
+- Verify the PR fills out `.github/PULL_REQUEST_TEMPLATE.md` and adds
+  `CODEOWNERS` entries for a new component.

.claude/skills

@@ -0,0 +1 @@
+../.agents/skills

.devcontainer/Dockerfile

@@ -31,7 +31,4 @@ RUN \
     platformio settings set enable_telemetry No \
     && platformio settings set check_platformio_interval 1000000
 
-COPY script/platformio_install_deps.py platformio.ini ./
-RUN ./platformio_install_deps.py platformio.ini --libraries --platforms --tools
-
 WORKDIR /workspaces

.github/ISSUE_TEMPLATE/bug_report.yml

@@ -9,6 +9,29 @@ body:
         If you have a feature request or enhancement, please [request them here instead][fr].
 
         [fr]: https://github.com/orgs/esphome/discussions
+  - type: markdown
+    attributes:
+      value: |
+        ## Use of AI in bug reports
+
+        AI tools are good at carrying out well-defined tasks, but they are not good at troubleshooting.
+        Please do NOT paste an AI-generated wall of text into the issue template - if the AI hasn't solved
+        your problem, its wild guesses are not likely to help.
+
+        Please DO include your own words and observations, compile/boot logs, and
+        especially a minimal reproducible example of your YAML configuration that demonstrates the problem.
+
+        It is however quite acceptable to use AI to translate your *own* report,
+        if you aren't a competent English speaker.
+
+        If you really think it will be useful to include an AI's analysis, preferably wrap it in a `<details>` block which will be collapsed by default.
+
+        If you are using AI to help solve a problem, rather than asking it to speculate about what the problem is,
+        it can be more useful to ask it to create a step-by-step troubleshooting procedure.
+        AI is also useful for generating boilerplate code, such as a minimal reproducible example of your YAML
+        configuration that demonstrates the problem.
+
+        Used properly, AI can be a useful tool to help you solve your problem, but don't let it get in the way.
   - type: textarea
     validations:
       required: true

.github/actions/build-image/action.yaml

@@ -42,7 +42,7 @@ runs:
 
     - name: Build and push to ghcr by digest
       id: build-ghcr
-      uses: docker/build-push-action@53b7df96c91f9c12dcc8a07bcb9ccacbed38856a # v7.3.0
+      uses: docker/build-push-action@c3c9e263c25d99ce0380d002d59b67737d91b0dc # v7.4.0
       env:
         DOCKER_BUILD_SUMMARY: false
         DOCKER_BUILD_RECORD_UPLOAD: false
@@ -67,7 +67,7 @@ runs:
 
     - name: Build and push to dockerhub by digest
       id: build-dockerhub
-      uses: docker/build-push-action@53b7df96c91f9c12dcc8a07bcb9ccacbed38856a # v7.3.0
+      uses: docker/build-push-action@c3c9e263c25d99ce0380d002d59b67737d91b0dc # v7.4.0
       env:
         DOCKER_BUILD_SUMMARY: false
         DOCKER_BUILD_RECORD_UPLOAD: false

.github/actions/cache-arduino8266/action.yml

@@ -0,0 +1,39 @@
+name: Cache Arduino ESP8266
+description: >
+  Resolve the pinned Arduino core and xtensa toolchain versions and cache the
+  native ESP8266 install (~110 MB framework + toolchain; no ccache store, the
+  seed job saves before any compile runs). Exports
+  ESPHOME_ARDUINO8266_PREFIX to the job so every later step installs into
+  the cached path; the Python venv must already be restored. Mirrors
+  cache-esp-idf: only dev-branch pushes write the shared cache, everything
+  else restores.
+runs:
+  using: composite
+  steps:
+    - name: Resolve the native toolchain cache key
+      # Versions are pinned in code, not a hashable file; resolve them so a
+      # bump changes the cache key. Assignment form so errexit catches a
+      # resolver failure.
+      id: version
+      shell: bash
+      run: |
+        # One owner for the install prefix: exported here and referenced by
+        # the cache steps below via env, so the caller's install and the
+        # cached path cannot diverge.
+        echo "ESPHOME_ARDUINO8266_PREFIX=$HOME/.esphome-arduino8266" >> "$GITHUB_ENV"
+        . venv/bin/activate
+        key=$(python -c 'from esphome.components.esp8266 import RECOMMENDED_ARDUINO_FRAMEWORK_VERSION as f; from esphome.arduino8266.framework import FRAMEWORK_RELEASES, TOOLCHAIN_VERSION as t; print(f"{FRAMEWORK_RELEASES[f].tag}-{t}")')
+        [ -n "$key" ] || exit 1
+        echo "key=$key" >> "$GITHUB_OUTPUT"
+    - name: Cache the native toolchain (write on dev)
+      if: github.ref == 'refs/heads/dev'
+      uses: actions/cache@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
+      with:
+        path: ${{ env.ESPHOME_ARDUINO8266_PREFIX }}
+        key: ${{ runner.os }}-esp8266-native-${{ steps.version.outputs.key }}
+    - name: Restore the native toolchain (off dev)
+      if: github.ref != 'refs/heads/dev'
+      uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
+      with:
+        path: ${{ env.ESPHOME_ARDUINO8266_PREFIX }}
+        key: ${{ runner.os }}-esp8266-native-${{ steps.version.outputs.key }}

.github/actions/restore-python/action.yml

@@ -32,7 +32,7 @@ runs:
       # detects the activated venv via ``VIRTUAL_ENV`` so the venv layout
       # downstream jobs rely on is preserved.
       if: steps.cache-venv.outputs.cache-hit != 'true'
-      uses: astral-sh/setup-uv@20cfd1bf945f4377ade1205e4dbc17946fc9a30d # v10.0.1
+      uses: astral-sh/setup-uv@c18668ad3cf93ea998bef934396af7bb5c839dc7 # v10.2.0
       with:
         enable-cache: true
         # Pull request saves land in per-PR scopes nothing else can

.github/skills

@@ -0,0 +1 @@
+../.agents/skills

.github/workflows/ci-api-proto.yml

@@ -29,7 +29,7 @@ jobs:
       - name: Set up uv
         # ``--system`` (below) installs into the setup-python interpreter;
         # no venv is created or restored by this workflow.
-        uses: astral-sh/setup-uv@20cfd1bf945f4377ade1205e4dbc17946fc9a30d # v10.0.1
+        uses: astral-sh/setup-uv@c18668ad3cf93ea998bef934396af7bb5c839dc7 # v10.2.0
         with:
           enable-cache: true
           # Pull-request-only workflow: a save could never be shared and

.github/workflows/ci-docker.yml

@@ -12,15 +12,14 @@ on:
       - ".github/workflows/ci-docker.yml"
       - "requirements*.txt"
       - "pyproject.toml"
-      - "platformio.ini"
       - "esphome/idf_component.yml"
-      - "script/platformio_install_deps.py"
       # Core, build pipeline, toolchain, and target-platform changes can change
       # how a toolchain is set up or built, so re-run the per-toolchain compile
       # smoke test when they change.
       - "esphome/core/**"
       - "esphome/writer.py"
       - "esphome/build_gen/**"
+      - "esphome/build_helpers/**"
       - "esphome/espidf/**"
       - "esphome/platformio/**"
       - "esphome/components/bk72xx/**"
@@ -67,7 +66,7 @@ jobs:
         with:
           python-version: "3.12"
       - name: Set up Docker Buildx
-        uses: docker/setup-buildx-action@37fe631027851001ddb9b187196cc803df7f5f0e # v4.3.0
+        uses: docker/setup-buildx-action@594f3bf4285d9ea8dc53c9a0c9c4092420091003 # v4.4.0
 
       - name: Determine tag and whether to push
         id: tag
@@ -159,7 +158,7 @@ jobs:
         with:
           python-version: "3.12"
       - name: Set up Docker Buildx
-        uses: docker/setup-buildx-action@37fe631027851001ddb9b187196cc803df7f5f0e # v4.3.0
+        uses: docker/setup-buildx-action@594f3bf4285d9ea8dc53c9a0c9c4092420091003 # v4.4.0
 
       - name: Log in to the GitHub container registry
         uses: docker/login-action@dbcb813823bdd20940b903addbd779551569679f # v4.6.0
@@ -197,6 +196,7 @@ jobs:
         # the default.
         id:
           - esp8266-arduino
+          - esp8266-arduino-native
           - esp32-arduino-platformio
           - esp32-arduino-esp-idf
           - esp32-idf-platformio
@@ -219,5 +219,6 @@ jobs:
         run: |
           docker run --rm \
             -v "${{ github.workspace }}/docker/test_configs:/config" \
+            -e ESPHOME_LDGEN_STRICT=1 \
             "ghcr.io/esphome/esphome-amd64:${{ needs.check-docker.outputs.tag }}" \
             compile "${{ matrix.id }}.yaml"

.github/workflows/ci.yml

@@ -49,7 +49,7 @@ jobs:
         # detects the activated venv via ``VIRTUAL_ENV`` so downstream jobs
         # that ``. venv/bin/activate`` see an identical layout.
         if: steps.cache-venv.outputs.cache-hit != 'true'
-        uses: astral-sh/setup-uv@20cfd1bf945f4377ade1205e4dbc17946fc9a30d # v10.0.1
+        uses: astral-sh/setup-uv@c18668ad3cf93ea998bef934396af7bb5c839dc7 # v10.2.0
         with:
           enable-cache: true
           # Pull request saves land in per-PR scopes nothing else can
@@ -102,6 +102,8 @@ jobs:
       device-builder: ${{ steps.determine.outputs.device-builder }}
       esp32-platformio: ${{ steps.determine.outputs.esp32-platformio }}
       esp32-platformio-components: ${{ steps.determine.outputs.esp32-platformio-components }}
+      esp8266-native: ${{ steps.determine.outputs.esp8266-native }}
+      esp8266-native-components: ${{ steps.determine.outputs.esp8266-native-components }}
       changed-components: ${{ steps.determine.outputs.changed-components }}
       changed-components-with-tests: ${{ steps.determine.outputs.changed-components-with-tests }}
       directly-changed-components-with-tests: ${{ steps.determine.outputs.directly-changed-components-with-tests }}
@@ -165,6 +167,8 @@ jobs:
           echo "device-builder=$(echo "$output" | jq -r '.device_builder')" >> $GITHUB_OUTPUT
           echo "esp32-platformio=$(echo "$output" | jq -r '.esp32_platformio')" >> $GITHUB_OUTPUT
           echo "esp32-platformio-components=$(echo "$output" | jq -r '.esp32_platformio_components')" >> $GITHUB_OUTPUT
+          echo "esp8266-native=$(echo "$output" | jq -r '.esp8266_native')" >> $GITHUB_OUTPUT
+          echo "esp8266-native-components=$(echo "$output" | jq -r '.esp8266_native_components')" >> $GITHUB_OUTPUT
           echo "changed-components=$(echo "$output" | jq -c '.changed_components')" >> $GITHUB_OUTPUT
           echo "changed-components-with-tests=$(echo "$output" | jq -c '.changed_components_with_tests')" >> $GITHUB_OUTPUT
           echo "directly-changed-components-with-tests=$(echo "$output" | jq -c '.directly_changed_components_with_tests')" >> $GITHUB_OUTPUT
@@ -183,6 +187,33 @@ jobs:
           path: .temp/components_graph.json
           key: components-graph-${{ hashFiles('esphome/components/**/*.py') }}
 
+  seed-esp8266-native-cache:
+    name: Seed the esp8266 native toolchain cache
+    runs-on: ubuntu-24.04
+    needs:
+      - common
+    # PR-branch cache saves are invisible to other PRs, so dev pushes seed
+    # the shared entry the component matrix, the memory impact jobs and
+    # test-esp8266-native restore. Only dev: the composite action saves
+    # nowhere else, so a beta/release push would download the toolchain and
+    # discard it.
+    if: github.event_name == 'push' && github.ref == 'refs/heads/dev'
+    timeout-minutes: 15
+    steps:
+      - name: Check out code from GitHub
+        uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
+      - name: Restore Python
+        uses: ./.github/actions/restore-python
+        with:
+          python-version: ${{ env.DEFAULT_PYTHON }}
+          cache-key: ${{ needs.common.outputs.cache-key }}
+      - name: Cache the native toolchain
+        uses: ./.github/actions/cache-arduino8266
+      - name: Install the native toolchain
+        run: |
+          . venv/bin/activate
+          python -c "from esphome.arduino8266.framework import check_and_install; from esphome.components.esp8266 import RECOMMENDED_ARDUINO_FRAMEWORK_VERSION; check_and_install(RECOMMENDED_ARDUINO_FRAMEWORK_VERSION)"
+
   ci-custom:
     name: Run script/ci-custom
     runs-on: ubuntu-24.04
@@ -268,7 +299,7 @@ jobs:
       # Pushes any fixes the hooks made back to the pull request. This step
       # must keep its default name: the GitHub App that performs the push
       # locates the workflow run by that name.
-      - uses: pre-commit-ci/lite-action@5d6cc0eb514c891a40562a58a8e71576c5c7fb43 # v1.1.0
+      - uses: pre-commit-ci/lite-action@062bca0919bc9d6e66755cc05074b70c77e111fc # v1.2.0
         if: always()
         with:
           msg: apply automatic formatting fixes
@@ -321,7 +352,7 @@ jobs:
           . venv/bin/activate
           pytest -vv --cov-report=xml --tb=native --durations=30 -n auto tests --ignore=tests/integration/
       - name: Upload coverage to Codecov
-        uses: codecov/codecov-action@fb8b3582c8e4def4969c97caa2f19720cb33a72f # v7.0.0
+        uses: codecov/codecov-action@303a32d7a59b442fa8d48b6a1cc6825c09c847a5 # v7.1.1
       - name: Save Python virtual environment cache
         if: github.ref == 'refs/heads/dev'
         uses: actions/cache/save@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
@@ -350,7 +381,7 @@ jobs:
       - name: Check out code from GitHub
         uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
       - name: Report empty upload to Codecov
-        uses: codecov/codecov-action@fb8b3582c8e4def4969c97caa2f19720cb33a72f # v7.0.0
+        uses: codecov/codecov-action@303a32d7a59b442fa8d48b6a1cc6825c09c847a5 # v7.1.1
         with:
           run_command: empty-upload
           force: true
@@ -369,14 +400,9 @@ jobs:
       matrix:
         bucket: ${{ fromJson(needs.determine-jobs.outputs.integration-test-buckets) }}
     env:
-      # What the cache steps persist; libdeps is excluded (keyed per xdist
-      # worker and env, it never crosses runs).
-      INTEGRATION_PIO_CACHE_PATH: |
-        ~/.esphome-integration-tests/platformio/platforms
-        ~/.esphome-integration-tests/platformio/packages
-        ~/.esphome-integration-tests/platformio/appstate.json
-        ~/.esphome-integration-tests/platformio/.cache
-        ~/.esphome-integration-tests/platformio/.esphome.pio.stamp.json
+      # Registry libraries (noise-c, libsodium, ArduinoJson, lvgl) the host builds
+      # download, shared per xdist worker by tests/integration/conftest.py
+      INTEGRATION_LIBRARY_CACHE_PATH: ~/.esphome-integration-tests/pio_components
     steps:
       - name: Check out code from GitHub
         uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
@@ -396,14 +422,13 @@ jobs:
         uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
         with:
           python-version: "3.13"
-      - name: Restore integration PlatformIO cache
-        # Native platform + toolchain installed by shared_platformio_cache in
-        # tests/integration/conftest.py; a miss self-heals, so no restore-keys.
-        id: pio-cache
+      - name: Restore integration library cache
+        # A miss or a changed pin self-heals with a download, so no restore-keys
+        id: library-cache
         uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
         with:
-          path: ${{ env.INTEGRATION_PIO_CACHE_PATH }}
-          key: integration-pio-v1-${{ runner.os }}-py${{ steps.python.outputs.python-version }}-${{ hashFiles('requirements.txt', 'tests/integration/fixtures/cache_init.yaml', 'esphome/components/host/__init__.py') }}
+          path: ${{ env.INTEGRATION_LIBRARY_CACHE_PATH }}
+          key: integration-libraries-v1-${{ runner.os }}-${{ hashFiles('esphome/components/json/__init__.py', 'esphome/components/noise/__init__.py', 'esphome/components/lvgl/__init__.py', 'esphome/components/improv_base/__init__.py') }}
       - name: Restore Python virtual environment
         id: cache-venv
         uses: actions/cache@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
@@ -413,7 +438,7 @@ jobs:
       - name: Set up uv
         # Only needed on cache miss to populate the venv.
         if: steps.cache-venv.outputs.cache-hit != 'true'
-        uses: astral-sh/setup-uv@20cfd1bf945f4377ade1205e4dbc17946fc9a30d # v10.0.1
+        uses: astral-sh/setup-uv@c18668ad3cf93ea998bef934396af7bb5c839dc7 # v10.2.0
         with:
           enable-cache: true
           # Pull request saves land in per-PR scopes nothing else can
@@ -471,16 +496,16 @@ jobs:
           # A full cron period of margin for the weekly refresh
           retention-days: 14
       - name: Print ccache statistics
-        # esphome stores the PlatformIO ccache under the machine-global cache
-        # dir (see _ccache_env() in esphome/platformio/toolchain.py).
-        run: CCACHE_DIR="$HOME/.cache/esphome/platformio-ccache" ccache -s
-      - name: Save integration PlatformIO cache
+        # esphome stores the host build's ccache under the machine-global
+        # cache dir (see get_build_env() in esphome/host/toolchain.py).
+        run: CCACHE_DIR="$HOME/.cache/esphome/host/ccache" ccache -s
+      - name: Save integration library cache
         # Bucket 0 only; the others would race the same immutable key.
-        if: success() && (github.ref == 'refs/heads/dev' || contains(github.event.pull_request.labels.*.name, 'ci-cache-write')) && strategy.job-index == 0 && steps.pio-cache.outputs.cache-hit != 'true'
+        if: success() && (github.ref == 'refs/heads/dev' || contains(github.event.pull_request.labels.*.name, 'ci-cache-write')) && strategy.job-index == 0 && steps.library-cache.outputs.cache-hit != 'true'
         uses: actions/cache/save@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
         with:
-          path: ${{ env.INTEGRATION_PIO_CACHE_PATH }}
-          key: ${{ steps.pio-cache.outputs.cache-primary-key }}
+          path: ${{ env.INTEGRATION_LIBRARY_CACHE_PATH }}
+          key: ${{ steps.library-cache.outputs.cache-primary-key }}
 
   import-time:
     name: Check import esphome.__main__ time
@@ -601,7 +626,7 @@ jobs:
             apt-get install -y libc6-dbg
 
       - name: Run CodSpeed benchmarks
-        uses: CodSpeedHQ/action@373d6868929f444bc08d901fd0eb0ad52a8875ea # v5.2.1
+        uses: CodSpeedHQ/action@c4fd08a3a159bd0cc208da1e0edf32b8c47d75e5 # v5.4.0
         with:
           run: |
             . venv/bin/activate
@@ -868,14 +893,20 @@ jobs:
       matrix:
         include:
           - id: clang-tidy
-            name: Run script/clang-tidy for ESP32 IDF 1/3
-            options: --environment esp32-idf-tidy --split-num 3 --split-at 1
+            name: Run script/clang-tidy for ESP32 IDF 1/5
+            options: --environment esp32-idf-tidy --split-num 5 --split-at 1
           - id: clang-tidy
-            name: Run script/clang-tidy for ESP32 IDF 2/3
-            options: --environment esp32-idf-tidy --split-num 3 --split-at 2
+            name: Run script/clang-tidy for ESP32 IDF 2/5
+            options: --environment esp32-idf-tidy --split-num 5 --split-at 2
           - id: clang-tidy
-            name: Run script/clang-tidy for ESP32 IDF 3/3
-            options: --environment esp32-idf-tidy --split-num 3 --split-at 3
+            name: Run script/clang-tidy for ESP32 IDF 3/5
+            options: --environment esp32-idf-tidy --split-num 5 --split-at 3
+          - id: clang-tidy
+            name: Run script/clang-tidy for ESP32 IDF 4/5
+            options: --environment esp32-idf-tidy --split-num 5 --split-at 4
+          - id: clang-tidy
+            name: Run script/clang-tidy for ESP32 IDF 5/5
+            options: --environment esp32-idf-tidy --split-num 5 --split-at 5
 
     steps:
       - name: Check out code from GitHub
@@ -1097,6 +1128,11 @@ jobs:
         uses: ./.github/actions/cache-sdk-nrf
         with:
           restore-only: true
+      - name: Cache the native ESP8266 toolchain
+        # Only batches whose test platforms include esp8266; never saves
+        # here, it reuses the install the dev seed job cached.
+        if: matrix.batch.needs_arduino8266
+        uses: ./.github/actions/cache-arduino8266
       - name: Validate and compile components with intelligent grouping
         run: |
           . venv/bin/activate
@@ -1195,9 +1231,28 @@ jobs:
 
           if [ -n "$compile_csv" ]; then
             # Run compilation with grouping and isolation
-            python3 script/test_build_components.py -e compile -c "$compile_csv" -f --isolate "$directly_changed_csv"
+            # The bootloader has no ESPHome code and these builds never
+            # flash; the check_idf_py batch keeps the full build so the
+            # native sub-build and the equivalence check stay covered.
+            skip_flag="--skip-bootloader"
+            if [[ "${{ matrix.batch.check_idf_py }}" == "true" ]]; then
+              skip_flag=""
+            fi
+            python3 script/test_build_components.py -e compile -c "$compile_csv" -f --isolate "$directly_changed_csv" $skip_flag
+            if [[ "${{ matrix.batch.check_idf_py }}" == "true" ]]; then
+              # The real idf.py must find nothing to configure or build in a
+              # tree built above; catches drift on ESP-IDF bumps.
+              echo "Checking the native ESP-IDF build matches idf.py"
+              python3 script/check_idf_py_equivalence.py
+            fi
           else
             echo "All components in this batch are validate-only -- skipping compile stage."
+            if [[ "${{ matrix.batch.check_idf_py }}" == "true" ]]; then
+              # determine-jobs and this step disagree on what compiles; fail
+              # rather than let the check run nowhere.
+              echo "::error::This batch was picked for the idf.py check but compiled nothing"
+              exit 1
+            fi
           fi
 
       - name: Print ccache statistics
@@ -1237,7 +1292,7 @@ jobs:
 
           # compile validates config first, so a separate config pass is
           # redundant for this smoke test. ESP-IDF framework via PlatformIO:
-          python3 script/test_build_components.py -e compile -t esp32-idf -c "$TEST_COMPONENTS" -f --toolchain platformio
+          python3 script/test_build_components.py -e compile -t esp32-idf -c "$TEST_COMPONENTS" -f --toolchain platformio --fail-on-no-tests
 
           echo ""
           echo "ESP-IDF-via-PlatformIO build passed! Starting Arduino smoke test..."
@@ -1246,6 +1301,40 @@ jobs:
           # Arduino framework via PlatformIO (only components with an esp32-ard test are built):
           python3 script/test_build_components.py -e compile -t esp32-ard -c "$TEST_COMPONENTS" -f --toolchain platformio
 
+  test-esp8266-native:
+    name: Test esp8266 components with the native toolchain
+    runs-on: ubuntu-24.04
+    needs:
+      - common
+      - determine-jobs
+    if: github.event_name == 'pull_request' && needs.determine-jobs.outputs.esp8266-native == 'true'
+    env:
+      # Computed by script/determine-jobs.py (ESP8266_NATIVE_TEST_COMPONENTS)
+      TEST_COMPONENTS: ${{ needs.determine-jobs.outputs.esp8266-native-components }}
+    steps:
+      - name: Check out code from GitHub
+        uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
+
+      - name: Restore Python
+        uses: ./.github/actions/restore-python
+        with:
+          python-version: ${{ env.DEFAULT_PYTHON }}
+          cache-key: ${{ needs.common.outputs.cache-key }}
+
+      - name: Cache the native toolchain
+        uses: ./.github/actions/cache-arduino8266
+
+      - name: Run native toolchain compile test
+        run: |
+          . venv/bin/activate
+
+          echo "Testing components: $TEST_COMPONENTS"
+          echo ""
+
+          # ESP8266 Arduino built directly (no PlatformIO); compile validates
+          # config first, so a separate config pass is redundant.
+          python3 script/test_build_components.py -e compile -t esp8266-ard -c "$TEST_COMPONENTS" -f --toolchain arduino --fail-on-no-tests
+
   device-builder:
     name: Test downstream esphome/device-builder
     runs-on: ubuntu-24.04
@@ -1274,7 +1363,7 @@ jobs:
         # install step (order-of-magnitude faster on cold boots,
         # with its own wheel cache). actions/setup-python still
         # provides the interpreter.
-        uses: astral-sh/setup-uv@20cfd1bf945f4377ade1205e4dbc17946fc9a30d # v10.0.1
+        uses: astral-sh/setup-uv@c18668ad3cf93ea998bef934396af7bb5c839dc7 # v10.2.0
         with:
           enable-cache: true
           # Pull request saves land in per-PR scopes nothing else can
@@ -1415,12 +1504,9 @@ jobs:
           python-version: ${{ env.DEFAULT_PYTHON }}
           cache-key: ${{ needs.common.outputs.cache-key }}
 
-      - name: Cache platformio
-        if: steps.check-script.outputs.skip != 'true' && steps.check-tests.outputs.skip != 'true' && steps.cache-memory-analysis.outputs.cache-hit != 'true'
-        uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
-        with:
-          path: ~/.platformio
-          key: platformio-memory-${{ fromJSON(needs.determine-jobs.outputs.memory_impact).platform }}-${{ hashFiles('platformio.ini') }}
+      - name: Cache the native ESP8266 toolchain
+        if: steps.check-script.outputs.skip != 'true' && steps.check-tests.outputs.skip != 'true' && steps.cache-memory-analysis.outputs.cache-hit != 'true' && fromJSON(needs.determine-jobs.outputs.memory_impact).needs_arduino8266
+        uses: ./.github/actions/cache-arduino8266
 
       - name: Build, compile, and analyze memory
         if: steps.check-script.outputs.skip != 'true' && steps.check-tests.outputs.skip != 'true' && steps.cache-memory-analysis.outputs.cache-hit != 'true'
@@ -1505,11 +1591,9 @@ jobs:
         with:
           python-version: ${{ env.DEFAULT_PYTHON }}
           cache-key: ${{ needs.common.outputs.cache-key }}
-      - name: Cache platformio
-        uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
-        with:
-          path: ~/.platformio
-          key: platformio-memory-${{ fromJSON(needs.determine-jobs.outputs.memory_impact).platform }}-${{ hashFiles('platformio.ini') }}
+      - name: Cache the native ESP8266 toolchain
+        if: fromJSON(needs.determine-jobs.outputs.memory_impact).needs_arduino8266
+        uses: ./.github/actions/cache-arduino8266
       - name: Build, compile, and analyze memory
         id: extract
         run: |
@@ -1608,6 +1692,7 @@ jobs:
     needs:
       - common
       - seed-apt-cache
+      - seed-esp8266-native-cache
       - determine-jobs
       - ci-custom
       - pylint
@@ -1623,6 +1708,7 @@ jobs:
       - clang-tidy-esp32-variants
       - test-build-components-split
       - test-esp32-platformio
+      - test-esp8266-native
       - device-builder
       - memory-impact-target-branch
       - memory-impact-pr-branch

.github/workflows/codeql.yml

@@ -56,7 +56,7 @@ jobs:
 
       # Initializes the CodeQL tools for scanning.
       - name: Initialize CodeQL
-        uses: github/codeql-action/init@cdf488f595d80d6e07e03d4674febd5ab45fa938 # v4.37.9
+        uses: github/codeql-action/init@2892aa5e19bbd11bc0cff5427e3b750a04d9e3c2 # v4.38.2
         with:
           languages: ${{ matrix.language }}
           build-mode: ${{ matrix.build-mode }}
@@ -84,6 +84,6 @@ jobs:
           exit 1
 
       - name: Perform CodeQL Analysis
-        uses: github/codeql-action/analyze@cdf488f595d80d6e07e03d4674febd5ab45fa938 # v4.37.9
+        uses: github/codeql-action/analyze@2892aa5e19bbd11bc0cff5427e3b750a04d9e3c2 # v4.38.2
         with:
           category: "/language:${{matrix.language}}"

.github/workflows/external-component-bot.yml

@@ -125,14 +125,20 @@ jobs:
             }
 
             async function getEsphomeAndComponentChanges(github, owner, repo, prNumber) {
-                const changedFiles = await github.rest.pulls.listFiles({
-                    owner: owner,
-                    repo: repo,
-                    pull_number: prNumber,
-                });
+                const changedFiles = await github.paginate(
+                    github.rest.pulls.listFiles,
+                    {
+                        owner: owner,
+                        repo: repo,
+                        pull_number: prNumber,
+                        per_page: 100,
+                    }
+                );
 
-                const esphomeChanges = changedFiles.data
-                    .filter(file => file.filename !== "esphome/core/defines.h" && file.filename.startsWith('esphome/'))
+                // Files used only for development and CI, which do not affect use as an external component
+                const ignoredFiles = ["esphome/core/defines.h", "esphome/idf_component.yml"];
+                const esphomeChanges = changedFiles
+                    .filter(file => !ignoredFiles.includes(file.filename) && file.filename.startsWith('esphome/'))
                     .map(file => {
                         const match = file.filename.match(/esphome\/([^/]+)/);
                         return match ? match[1] : null;
@@ -144,7 +150,7 @@ jobs:
                 }
 
                 const uniqueEsphomeChanges = [...new Set(esphomeChanges)];
-                const componentChanges = changedFiles.data
+                const componentChanges = changedFiles
                     .filter(file => file.filename.startsWith('esphome/components/'))
                     .map(file => {
                         const match = file.filename.match(/esphome\/components\/([^/]+)\//);

.github/workflows/lock.yml

@@ -14,4 +14,4 @@ jobs:
     permissions:
       issues: write         # issues.lock on closed issues
       pull-requests: write  # issues.lock on closed pull requests
-    uses: esphome/workflows/.github/workflows/lock.yml@0fdd5e311b7e744069166696072a1a9cbc5fbeb6  # 2026.8.1
+    uses: esphome/workflows/.github/workflows/lock.yml@cc3e76de337dc59bc1cba8da58d963cd23b873f1  # 2026.9.0

.github/workflows/release.yml

@@ -123,7 +123,7 @@ jobs:
           python-version: "3.12"
 
       - name: Set up Docker Buildx
-        uses: docker/setup-buildx-action@37fe631027851001ddb9b187196cc803df7f5f0e # v4.3.0
+        uses: docker/setup-buildx-action@594f3bf4285d9ea8dc53c9a0c9c4092420091003 # v4.4.0
 
       - name: Log in to docker hub
         uses: docker/login-action@dbcb813823bdd20940b903addbd779551569679f # v4.6.0
@@ -202,7 +202,7 @@ jobs:
           merge-multiple: true
 
       - name: Set up Docker Buildx
-        uses: docker/setup-buildx-action@37fe631027851001ddb9b187196cc803df7f5f0e # v4.3.0
+        uses: docker/setup-buildx-action@594f3bf4285d9ea8dc53c9a0c9c4092420091003 # v4.4.0
 
       - name: Log in to docker hub
         if: matrix.registry == 'dockerhub'

.github/workflows/stale.yml

@@ -16,7 +16,7 @@ jobs:
     # No GITHUB_TOKEN permissions: the reusable workflow mints an ESPHome
     # GitHub App token so the labels, comments and closures come from
     # esphome[bot] instead of github-actions[bot].
-    uses: esphome/workflows/.github/workflows/stale.yml@a1c1485ab46ef41a84a6a9d8abd7fa4b7628fd70  # main
+    uses: esphome/workflows/.github/workflows/stale.yml@cc3e76de337dc59bc1cba8da58d963cd23b873f1  # main
     secrets:
       ESPHOME_GITHUB_APP_PRIVATE_KEY: ${{ secrets.ESPHOME_GITHUB_APP_PRIVATE_KEY }}
     with:

.github/workflows/status-check-labels.yml

@@ -7,10 +7,6 @@ on:
 permissions:
   pull-requests: read  # issues.listLabelsOnIssue to detect blocking labels (needs-docs, needs-developer-docs, merge-after-release, chained-pr)
 
-concurrency:
-  group: ${{ github.workflow }}-${{ github.event.pull_request.number }}
-  cancel-in-progress: true
-
 jobs:
   check:
     name: Check blocking labels

.github/workflows/sync-device-classes.yml

@@ -47,7 +47,7 @@ jobs:
         # setup-python interpreter so subsequent ``prek`` /
         # ``script/run-in-env.py`` steps find the deps without a
         # ``uv run`` prefix.
-        uses: astral-sh/setup-uv@20cfd1bf945f4377ade1205e4dbc17946fc9a30d # v10.0.1
+        uses: astral-sh/setup-uv@c18668ad3cf93ea998bef934396af7bb5c839dc7 # v10.2.0
         with:
           enable-cache: true
           # Pin uv version so the action does not have to fetch the

.pre-commit-config.yaml

@@ -10,7 +10,7 @@ ci:
 repos:
   - repo: https://github.com/astral-sh/ruff-pre-commit
     # Ruff version.
-    rev: v0.16.6
+    rev: v0.16.10
     hooks:
       # Run the linter.
       - id: ruff
@@ -18,7 +18,7 @@ repos:
       # Run the formatter.
       - id: ruff-format
   - repo: https://github.com/PyCQA/flake8
-    rev: 7.3.0
+    rev: 7.4.1
     hooks:
       - id: flake8
         additional_dependencies:

AGENTS.md

@@ -322,6 +322,25 @@ file does, and it is the authority when they disagree. The most useful starting
               var = await switch.new_switch(config)
           ```
 
+        - **Optional child entities of a hub:** bind the config once with `sensor.sub_sensors(config)` (or
+          `sub_binary_sensors`, `sub_text_sensors`, `sub_buttons`, `sub_switches`, `sub_numbers`,
+          `sub_selects` in their domains), adding `parent=hub` for entities that derive from `Parented<T>`,
+          then make one call per key, even when there is only one. A call creates the entity only when its key
+          is configured, passes it to the setter and returns it (or `None`); extra arguments such as
+          `min_value` or `options` go on the call. Always name the setter explicitly on the object that owns
+          it, never with `getattr` and an f-string, and keep that variable short (`var` for the component
+          itself, `hub` for one fetched with `cg.get_variable`) so the calls fit on one line. Loops whose
+          setter also takes an index, such as `set_gate_threshold(x, n)`, stay as they are.
+          ```python
+          async def to_code(config):
+              var = cg.new_Pvariable(config[CONF_ID])
+              sensors = sensor.sub_sensors(config)
+              await sensors(CONF_TEMPERATURE, var.set_temperature_sensor)
+              await sensors(CONF_HUMIDITY, var.set_humidity_sensor)
+              buttons = button.sub_buttons(config, parent=var)
+              await buttons(CONF_RESTART, var.set_restart_button)
+          ```
+
 *   **Automations (Triggers, Actions, Conditions):**
 
     Automations have three building blocks: **Triggers** (fire when something happens), **Actions** (do something), and **Conditions** (check if something is true).
@@ -431,7 +450,31 @@ file does, and it is the authority when they disagree. The most useful starting
           MyComponent *parent_;
         };
         ```
-        Register with `@automation.register_action("my_component.do_something", MyAction, schema, synchronous=True)`. Use `synchronous=True` for actions that run to completion inside `play()` without deferring. Use `synchronous=False` if the action may suspend/defer execution (e.g. `delay`, `wait_until`, `script.wait`) or store trigger arguments for later use.
+        Register it without writing a builder:
+        ```python
+        automation.register_simple_action(
+            "my_component.do_something", MyAction, schema, synchronous=True
+        )
+        ```
+        The constructor receives the object named by `config[CONF_ID]`. Use `register_bare_action` for a
+        no-argument constructor, `register_parented_action` for a class deriving from `Parented<T>`, and
+        the `@automation.register_action(...)` decorator only when the builder must also set fields.
+
+        Use `synchronous=True` for actions that run to completion inside `play()` without deferring. Use `synchronous=False` if the action may suspend/defer execution (e.g. `delay`, `wait_until`, `script.wait`) or store trigger arguments for later use.
+
+        **Actions that only forward templatable values to their parent need no C++ class.** Register them
+        with `register_apply_action`; do not write a `TEMPLATABLE_VALUE` class or a builder for this shape.
+        ```python
+        automation.register_apply_action(
+            "my_component.set_gains",
+            schema,
+            automation.ApplyField(CONF_KP, "set_kp", cg.float_),
+            automation.ApplyField(CONF_KI, "set_ki", cg.float_),
+        )
+        ```
+        The `ApplyField`, `ApplyCall` and `register_apply_action` docstrings in `esphome/automation.py` cover
+        the rest; `cover.control` and `cover.template.publish` are in-tree examples. `TEMPLATABLE_VALUE` with
+        `cg.templatable` stays for actions whose `play()` has real logic beyond forwarding values.
 
     *   **Conditions:**
         ```cpp
@@ -443,7 +486,21 @@ file does, and it is the authority when they disagree. The most useful starting
           MyComponent *parent_;
         };
         ```
-        Register with `@automation.register_condition("my_component.is_active", MyCondition, schema)`.
+        Register with `automation.register_simple_condition("my_component.is_active", MyCondition, schema)`;
+        `register_bare_condition`, `register_parented_condition` and the decorator follow the action rules.
+
+        **Conditions that only test their parent need no C++ class either.** Register them with
+        `register_apply_condition`; the expression is applied to the parent, and an `ApplyCall` compares
+        against config values.
+        ```python
+        automation.register_apply_condition("my_component.is_active", schema, "is_active()")
+        automation.register_apply_condition(
+            "my_component.state_is",
+            schema,
+            automation.ApplyCall("state == {}", ((CONF_STATE, cg.bool_),)),
+        )
+        ```
+        `cover.is_open`, `rtttl.is_playing` and `component.is_idle` are in-tree examples.
 
 *   **Type Hints:** Type-hint all function signatures, including test functions and config validators (e.g. `def validate_x(config: ConfigType) -> ConfigType:`, `def test_x() -> None:`). Import `ConfigType` from `esphome.types`.
 
@@ -699,7 +756,9 @@ file does, and it is the authority when they disagree. The most useful starting
 
         6. **Avoid `std::deque`:** It allocates in 512-byte blocks regardless of element size, guaranteeing at least 512 bytes of RAM usage immediately. This is a major source of crashes on memory-constrained devices.
 
-        7. **Detection:** Look for these patterns in compiler output:
+        7. **Never use `new (std::nothrow)`:** On ESP-IDF exceptions are disabled, so a failed nothrow allocation aborts instead of returning `nullptr`. Use `RAMAllocator` from `esphome/core/helpers.h`; CI rejects `std::nothrow`.
+
+        8. **Detection:** Look for these patterns in compiler output:
            - Large code sections with STL symbols (vector, map, set)
            - `alloc`, `realloc`, `dealloc` in symbol names
            - `_M_realloc_insert`, `_M_default_append` (vector reallocation)

CODEOWNERS

@@ -100,6 +100,7 @@ esphome/components/bmp581_i2c/* @danielkent-net @kahrendt
 esphome/components/bmp581_spi/* @danielkent-net @kahrendt
 esphome/components/bp1658cj/* @Cossid
 esphome/components/bp5758d/* @Cossid
+esphome/components/bridge/* @kbx81
 esphome/components/bthome_mithermometer/* @nagyrobi
 esphome/components/button/* @esphome/core
 esphome/components/bytebuffer/* @clydebarrow
@@ -111,6 +112,8 @@ esphome/components/captive_portal/* @esphome/core
 esphome/components/cc1101/* @gabest11 @lygris
 esphome/components/ccs811/* @habbie
 esphome/components/cd74hc4067/* @asoehlke
+esphome/components/cdc_acm_uart/* @kbx81
+esphome/components/cdc_acm_uart/bridge/* @kbx81
 esphome/components/ch422g/* @clydebarrow @jesterret
 esphome/components/ch423/* @dwmw2
 esphome/components/chsc6x/* @kkosik20
@@ -122,6 +125,7 @@ esphome/components/combination/* @Cat-Ion @kahrendt
 esphome/components/const/* @esphome/core
 esphome/components/coolix/* @glmnet
 esphome/components/copy/* @OttoWinter
+esphome/components/counter/* @clydebarrow
 esphome/components/cover/* @esphome/core
 esphome/components/cs5460a/* @balrog-kun
 esphome/components/cse7761/* @berfenger
@@ -179,15 +183,16 @@ esphome/components/esp32_camera_web_server/* @ayufan
 esphome/components/esp32_can/* @Sympatron
 esphome/components/esp32_hosted/* @swoboda1337
 esphome/components/esp32_hosted/update/* @swoboda1337
-esphome/components/esp32_improv/* @jesserockz
 esphome/components/esp32_rmt/* @jesserockz
 esphome/components/esp32_rmt_led_strip/* @jesserockz
 esphome/components/esp8266/* @esphome/core
 esphome/components/esp_ldo/* @clydebarrow
+esphome/components/espectre/* @francescopace
 esphome/components/espnow/* @jesserockz
 esphome/components/espnow/packet_transport/* @EasilyBoredEngineer
 esphome/components/ethernet_info/* @gtjadsonsantos
 esphome/components/event/* @nohat
+esphome/components/exponential_moving_average/* @clydebarrow
 esphome/components/exposure_notifications/* @OttoWinter
 esphome/components/ezo/* @ssieb
 esphome/components/ezo_pmp/* @carlos-sarmiento
@@ -242,8 +247,11 @@ esphome/components/hmac_md5/* @dwmw2
 esphome/components/hmac_sha256/* @dwmw2
 esphome/components/hoermann_hcp/* @zweckj
 esphome/components/homeassistant/* @esphome/core @OttoWinter
+esphome/components/homeassistant/button/* @jesserockz
 esphome/components/homeassistant/number/* @landonr
+esphome/components/homeassistant/select/* @jesserockz
 esphome/components/homeassistant/switch/* @Links2004
+esphome/components/homeassistant/text/* @jesserockz
 esphome/components/honeywell_hih_i2c/* @Benichou34
 esphome/components/honeywellabp/* @RubyBailey
 esphome/components/honeywellabp2_i2c/* @jpfaff
@@ -263,8 +271,10 @@ esphome/components/i2s_audio/* @jesserockz
 esphome/components/i2s_audio/microphone/* @jesserockz
 esphome/components/i2s_audio/speaker/* @jesserockz @kahrendt
 esphome/components/iaqcore/* @yozik04
+esphome/components/icnt86/* @danepowell
 esphome/components/ili9xxx/* @clydebarrow @nielsnl68
 esphome/components/improv_base/* @esphome/core
+esphome/components/improv_ble/* @jesserockz
 esphome/components/improv_serial/* @esphome/core
 esphome/components/ina226/* @latonita @Sergio303
 esphome/components/ina260/* @mreditor97
@@ -277,6 +287,7 @@ esphome/components/inkplate/* @jesserockz @JosipKuci
 esphome/components/integration/* @OttoWinter
 esphome/components/internal_temperature/* @Mat931
 esphome/components/interval/* @esphome/core
+esphome/components/ir_rf_base/* @bdraco @kbx81
 esphome/components/ir_rf_proxy/* @kbx81
 esphome/components/it8951/* @koosoli @limengdu @Passific
 esphome/components/jsn_sr04t/* @Mafus1
@@ -418,6 +429,7 @@ esphome/components/pn7150_i2c/* @jesserockz @kbx81
 esphome/components/pn7160/* @jesserockz @kbx81
 esphome/components/pn7160_i2c/* @jesserockz @kbx81
 esphome/components/pn7160_spi/* @jesserockz @kbx81
+esphome/components/pn71xx/* @jesserockz @kbx81
 esphome/components/power_supply/* @esphome/core
 esphome/components/preferences/* @esphome/core
 esphome/components/provisioning/* @esphome/core
@@ -425,6 +437,7 @@ esphome/components/psram/* @esphome/core
 esphome/components/pulse_meter/* @cstaahl @stevebaxter @TrentHouliston
 esphome/components/pvvx_mithermometer/* @pasiz
 esphome/components/pylontech/* @functionpointer
+esphome/components/pzem6l24/* @nuttytree
 esphome/components/qmi8658/* @clydebarrow
 esphome/components/qmp6988/* @andrewpc
 esphome/components/qr_code/* @wjtje
@@ -453,6 +466,7 @@ esphome/components/rtl87xx/* @kuba2k2
 esphome/components/rtttl/* @glmnet @ximex
 esphome/components/runtime_image/* @clydebarrow @guillempages @kahrendt
 esphome/components/runtime_stats/* @bdraco
+esphome/components/rx8025t/* @remcom
 esphome/components/rx8130/* @beormund
 esphome/components/safe_mode/* @jsuanet @kbx81 @paulmonigatti
 esphome/components/scd4x/* @martgras @sjtrny
@@ -474,6 +488,7 @@ esphome/components/sendspin/image/* @kahrendt
 esphome/components/sendspin/media_player/* @kahrendt
 esphome/components/sendspin/media_source/* @kahrendt
 esphome/components/sendspin/sensor/* @kahrendt
+esphome/components/sendspin/switch/* @kahrendt
 esphome/components/sendspin/text_sensor/* @kahrendt
 esphome/components/sensirion_common/* @martgras
 esphome/components/sensor/* @esphome/core
@@ -532,6 +547,7 @@ esphome/components/st7735/* @SenexCrenshaw
 esphome/components/st7789v/* @kbx81
 esphome/components/st7920/* @marsjan155
 esphome/components/statsd/* @Links2004
+esphome/components/stcc4/* @j9brown
 esphome/components/stts22h/* @B48D81EFCC
 esphome/components/substitutions/* @esphome/core
 esphome/components/sun/* @OttoWinter
@@ -542,11 +558,15 @@ esphome/components/sx126x/* @swoboda1337
 esphome/components/sx127x/* @swoboda1337
 esphome/components/sy6970/* @linkedupbits
 esphome/components/syslog/* @clydebarrow
+esphome/components/systa_bus/* @Mat931
 esphome/components/t6615/* @tylermenezes
+esphome/components/tas2780/* @remcom
+esphome/components/tas58xx/* @mrtoy-me @remcom
 esphome/components/tc74/* @sethgirvan
 esphome/components/tca9548a/* @andreashergert1984
 esphome/components/tca9555/* @mobrembski
 esphome/components/tcl112/* @glmnet
+esphome/components/tcp_uart/* @Bascht74
 esphome/components/tee501/* @Stock-M
 esphome/components/teleinfo/* @0hax
 esphome/components/tem3200/* @bakerkj
@@ -555,6 +575,7 @@ esphome/components/template/datetime/* @rfdarter
 esphome/components/template/event/* @nohat
 esphome/components/template/fan/* @ssieb
 esphome/components/text/* @mauritskorse
+esphome/components/tfluna/* @candrews
 esphome/components/thermopro_ble/* @sittner
 esphome/components/thermostat/* @kbx81
 esphome/components/time/* @esphome/core
@@ -586,11 +607,14 @@ esphome/components/uart/* @esphome/core
 esphome/components/uart/button/* @ssieb
 esphome/components/uart/event/* @eoasmxd
 esphome/components/uart/packet_transport/* @clydebarrow
+esphome/components/uart_mux/* @kbx81
+esphome/components/uart_tcp/* @Bascht74
 esphome/components/udp/* @clydebarrow
 esphome/components/ufire_ec/* @pvizeli
 esphome/components/ufire_ise/* @pvizeli
 esphome/components/ufm01/* @ljungqvist
 esphome/components/ultrasonic/* @ssieb @swoboda1337
+esphome/components/unicode/* @esphome/core
 esphome/components/update/* @jesserockz
 esphome/components/uponor_smatrix/* @kroimon
 esphome/components/usb_cdc_acm/* @kbx81
@@ -630,9 +654,11 @@ esphome/components/wts01/* @alepee
 esphome/components/x9c/* @EtienneMD
 esphome/components/xdb401/* @RT530
 esphome/components/xgzp68xx/* @gcormier
+esphome/components/xiaomi_body_scale/* @dckiller51
 esphome/components/xiaomi_hhccjcy10/* @fariouche
 esphome/components/xiaomi_lywsd02mmc/* @juanluss31
 esphome/components/xiaomi_lywsd03mmc/* @ahpohl
+esphome/components/xiaomi_mccgq02hl/* @ahpohl @morph027
 esphome/components/xiaomi_mhoc303/* @drug123
 esphome/components/xiaomi_mhoc401/* @vevsvevs
 esphome/components/xiaomi_rtcgq02lm/* @jesserockz

Doxyfile

@@ -48,7 +48,7 @@ PROJECT_NAME           = ESPHome
 # could be handy for archiving the generated documentation or if some version
 # control system is used.
 
-PROJECT_NUMBER         = 2026.10.0-dev
+PROJECT_NUMBER         = 2026.10.0b2
 
 # Using the PROJECT_BRIEF tag one can provide an optional one line description
 # for a project that appears at the top of each page and should give viewer a

docker/Dockerfile

@@ -22,16 +22,13 @@ RUN \
     -r /requirements.txt
 
 # Install the ESPHome Device Builder dashboard.
-RUN uv pip install --no-cache-dir esphome-device-builder==1.14.7
+RUN uv pip install --no-cache-dir esphome-device-builder==1.22.0
 
 RUN \
     platformio settings set enable_telemetry No \
     && platformio settings set check_platformio_interval 1000000 \
     && mkdir -p /piolibs
 
-COPY script/platformio_install_deps.py platformio.ini /
-RUN /platformio_install_deps.py /platformio.ini --libraries
-
 ARG BUILD_VERSION
 
 LABEL \

docker/docker_entrypoint.sh

@@ -21,10 +21,14 @@ export PLATFORMIO_PLATFORMS_DIR="${pio_cache_base}/platforms"
 export PLATFORMIO_PACKAGES_DIR="${pio_cache_base}/packages"
 export PLATFORMIO_CACHE_DIR="${pio_cache_base}/cache"
 
-# Keep the native toolchain installs on the persistent cache root, not the
-# container's ephemeral user cache dir (re-downloaded on every restart).
+# Keep the native toolchain installs and compiler caches on the persistent
+# cache root, not the container's user cache dir: it is lost on every
+# restart, and not writable when the container runs as a non-root user.
 export ESPHOME_ESP_IDF_PREFIX="$(dirname "${pio_cache_base}")/idf"
 export ESPHOME_SDK_NRF_PREFIX="$(dirname "${pio_cache_base}")/sdk-nrf"
+export ESPHOME_ARDUINO8266_PREFIX="$(dirname "${pio_cache_base}")/arduino8266"
+export ESPHOME_HOST_PREFIX="$(dirname "${pio_cache_base}")/host"
+export ESPHOME_PLATFORMIO_CCACHE_DIR="$(dirname "${pio_cache_base}")/platformio-ccache"
 
 # If /build is mounted, use that as the build path
 # otherwise use path in /config (so that builds aren't lost on container restart)

docker/ha-addon-rootfs/etc/s6-overlay/s6-rc.d/discovery/run

@@ -4,7 +4,6 @@
 # Home Assistant Add-on: ESPHome
 # Sends discovery information to Home Assistant.
 # ==============================================================================
-declare config
 declare port
 
 # We only disable it when disabled explicitly
@@ -19,14 +18,17 @@ port=$(bashio::addon.ingress_port)
 # Wait for the ESPHome Device Builder to become available
 bashio::net.wait_for "${port}" "127.0.0.1" 300
 
-config=$(\
-    bashio::var.json \
-        host "127.0.0.1" \
-        port "^${port}" \
-)
+# Send one discovery message; the config is a JSON string built with bashio::var.json.
+send_discovery() {
+    local service=$1
+    local config=$2
+    if bashio::discovery "${service}" "${config}" > /dev/null; then
+        bashio::log.info "Successfully send ${service} discovery information to Home Assistant."
+    else
+        bashio::log.error "${service} discovery message to Home Assistant failed!"
+    fi
+}
 
-if bashio::discovery "esphome" "${config}" > /dev/null; then
-    bashio::log.info "Successfully send discovery information to Home Assistant."
-else
-    bashio::log.error "Discovery message to Home Assistant failed!"
-fi
+send_discovery "esphome" "$(bashio::var.json host "127.0.0.1" port "^${port}")"
+# The Device Builder MCP server, consumed by Home Assistant's mcp integration.
+send_discovery "mcp" "$(bashio::var.json url "http://127.0.0.1:${port}/api/mcp")"

docker/ha-addon-rootfs/etc/s6-overlay/s6-rc.d/esphome/run

@@ -15,10 +15,14 @@ export PLATFORMIO_PLATFORMS_DIR="${pio_cache_base}/platforms"
 export PLATFORMIO_PACKAGES_DIR="${pio_cache_base}/packages"
 export PLATFORMIO_CACHE_DIR="${pio_cache_base}/cache"
 
-# Keep the native toolchain installs on the persistent /data volume, not the
-# container's ephemeral user cache dir (wiped on every add-on update/restart).
+# Keep the native toolchain installs and compiler caches on the persistent
+# /data volume, not the container's ephemeral user cache dir (wiped on every
+# add-on update/restart).
 export ESPHOME_ESP_IDF_PREFIX=/data/cache/idf
 export ESPHOME_SDK_NRF_PREFIX=/data/cache/sdk-nrf
+export ESPHOME_ARDUINO8266_PREFIX=/data/cache/arduino8266
+export ESPHOME_HOST_PREFIX=/data/cache/host
+export ESPHOME_PLATFORMIO_CCACHE_DIR=/data/cache/platformio-ccache
 
 if bashio::config.true 'leave_front_door_open'; then
     export DISABLE_HA_AUTHENTICATION=true

docker/test_configs/bk72xx-arduino.yaml

@@ -5,3 +5,15 @@ bk72xx:
   board: generic-bk7231n-qfn32-tuya
 
 logger:
+
+wifi:
+  ssid: MySSID
+  password: password1
+  ap:
+
+# mqtt and captive_portal together pull in AsyncTCP and ESPAsyncWebServer;
+# a stray ESP32 AsyncTCP copy on the library search path breaks this build
+captive_portal:
+
+mqtt:
+  broker: 192.168.178.84

docker/test_configs/esp8266-arduino-native.yaml

@@ -0,0 +1,8 @@
+esphome:
+  name: docker-test-esp8266-native
+
+esp8266:
+  board: d1_mini
+  toolchain: arduino
+
+logger:

docker/test_configs/esp8266-arduino.yaml

@@ -3,5 +3,7 @@ esphome:
 
 esp8266:
   board: d1_mini
+  # The PlatformIO path stays covered whatever the default is
+  toolchain: platformio
 
 logger:

esphome/__main__.py

@@ -7,6 +7,7 @@ import importlib
 import logging
 import os
 from pathlib import Path
+import platform
 import re
 import sys
 import time
@@ -16,8 +17,8 @@ from typing import TYPE_CHECKING, Protocol
 # cause them to be loaded before external components are processed, resulting
 # in the built-in version being used instead of the external component one.
 from esphome import const, platform_hooks
+from esphome.build_helpers.native import analysis_backend, native_backend
 from esphome.const import (
-    ALLOWED_NAME_CHARS,
     ARGUMENT_HELP_DEVICE,
     BUNDLE_EXTENSION,
     CONF_API,
@@ -35,7 +36,6 @@ from esphome.const import (
     CONF_LOGGER,
     CONF_MDNS,
     CONF_MQTT,
-    CONF_NAME,
     CONF_NAME_ADD_MAC_SUFFIX,
     CONF_OTA,
     CONF_PASSWORD,
@@ -55,12 +55,13 @@ from esphome.const import (
 )
 from esphome.core import CORE, EsphomeError, coroutine
 from esphome.enum import StrEnum
-from esphome.helpers import get_bool_env, indent, is_ip_address
+from esphome.helpers import IS_MACOS, get_bool_env, indent, is_ip_address
 from esphome.log import AnsiFore, color, setup_log
 from esphome.stacktrace import LogLineProcessor
 from esphome.types import ConfigType
 from esphome.upload_targets import PortType, get_port_type
 from esphome.util import (
+    ESPHOME_COMMAND,
     PICOTOOL_PACKAGE,
     FlashImage,
     detect_rp2040_bootsel,
@@ -84,7 +85,6 @@ if TYPE_CHECKING:
 
 _LOGGER = logging.getLogger(__name__)
 
-ESPHOME_COMMAND = [sys.executable, "-m", "esphome"]
 
 # Maximum buffer size for serial log reading to prevent unbounded memory growth
 SERIAL_BUFFER_MAX_SIZE = 65536
@@ -149,6 +149,7 @@ class ArgsProtocol(Protocol):
     file: str | None
     no_logs: bool
     only_generate: bool
+    skip_bootloader: bool
     show_secrets: bool
     dashboard: bool
     configuration: str
@@ -817,7 +818,9 @@ def write_cpp_file() -> int:
         from esphome.build_gen import espidf
 
         espidf.write_project()
-    else:
+    elif not CORE.using_native_toolchain:
+        # Other native builds generate their project at compile time;
+        # never write a platformio.ini for them
         from esphome.build_gen import platformio
 
         platformio.write_project()
@@ -826,6 +829,14 @@ def write_cpp_file() -> int:
 
 
 def compile_program(args: ArgsProtocol, config: ConfigType) -> int:
+    if CORE.skip_bootloader and not (CORE.is_esp32 and CORE.using_toolchain_esp_idf):
+        # Debug only: an orchestrator cannot see YAML toolchain overrides,
+        # so this is its expected no-op, and a full build is safe.
+        _LOGGER.debug(
+            "--skip-bootloader ignored: only supported on ESP32 with the "
+            "esp-idf toolchain"
+        )
+        CORE.skip_bootloader = False
     # Keep this gate here, NOT in config validation: device-builder needs
     # `esphome config` to keep succeeding with placeholders so onboarding can run.
     if CONF_WIFI in config:
@@ -835,7 +846,7 @@ def compile_program(args: ArgsProtocol, config: ConfigType) -> int:
 
     # Keep this here, NOT in codegen: config-hash and --only-generate must keep
     # working on machines that cannot run the toolchain.
-    if CORE.is_esp8266:
+    if CORE.is_esp8266 and CORE.using_toolchain_platformio:
         from esphome.components.esp8266 import check_rosetta
 
         check_rosetta()
@@ -856,23 +867,20 @@ def compile_program(args: ArgsProtocol, config: ConfigType) -> int:
             return rc
 
         # Create factory.bin, ota.bin, and firmware.elf copy
-        toolchain.create_factory_bin()
+        if not toolchain.create_factory_bin():
+            # A build whose factory image could not be produced must not
+            # exit 0; downloads would serve an image from an older build.
+            return 1
         toolchain.create_ota_bin()
         toolchain.create_elf_copy()
-        from esphome.build_helpers.idedata import IDEDATA_BEST_EFFORT_ERRORS
+        from esphome.build_helpers.idedata import warn_if_idedata_missing
 
-        try:
-            if toolchain.get_idedata() is None:
-                _LOGGER.warning("No idedata was generated for this build")
-        except IDEDATA_BEST_EFFORT_ERRORS as err:
-            # The firmware already built; an idedata failure must not fail
-            # a successful build.
-            _LOGGER.warning(
-                "Could not generate idedata: %s (IDE, clang-tidy, and "
-                "memory-analysis data will be unavailable for this build)",
-                err,
-            )
-            _LOGGER.debug("Idedata failure detail", exc_info=True)
+        warn_if_idedata_missing(toolchain.get_idedata)
+    elif CORE.using_native_toolchain:
+        raise EsphomeError(
+            f"Toolchain '{CORE.toolchain.value}' resolved but no platform "
+            "backend claimed the build"
+        )
     else:
         from esphome.platformio import toolchain
 
@@ -975,12 +983,16 @@ def upload_using_esptool(
 
     if file is not None:
         flash_images = [FlashImage(path=file, offset="0x0")]
-    elif CORE.using_toolchain_esp_idf:
-        from esphome.espidf import toolchain
-
-        flash_images = [
-            FlashImage(path=toolchain.get_factory_firmware_path(), offset="0x0")
-        ]
+    elif (native := native_backend()) is not None:
+        # Every native backend supplies its own 0x0 flash image (bootloader
+        # and partitions included where the target needs them)
+        image = native.get_factory_firmware_path()
+        if not image.is_file():
+            hint = getattr(native, "missing_image_hint", lambda: None)()
+            raise EsphomeError(
+                hint or f"{image} does not exist; compile the configuration first"
+            )
+        flash_images = [FlashImage(path=image, offset="0x0")]
     else:
         from esphome.platformio import toolchain
 
@@ -1289,10 +1301,9 @@ def _choose_ota_platform(config: ConfigType, requested: str | None) -> str:
     The native API uses challenge-response auth with MD5/SHA256 hashing of a
     server-issued nonce, so the password is never sent over the wire; the
     ``web_server`` path uses HTTP Basic auth which transmits credentials in
-    cleartext over the LAN. (The native path also supports gzip compression
-    on ESP8266, where flash space is tight; on ESP32/RP2040/LibreTiny the
-    backend reports ``supports_compression() == false`` and the firmware is
-    sent uncompressed regardless of which platform is used.) Falls back to
+    cleartext over the LAN. (The native path also compresses the upload:
+    gzip on ESP8266 and RP2040, which inflate it at reboot, and a deflate
+    stream on ESP32/LibreTiny, which inflate it as it arrives.) Falls back to
     ``web_server`` only when that is the only available platform.
     """
     # Use a dict (insertion-ordered) instead of a list so error messages and
@@ -1301,9 +1312,9 @@ def _choose_ota_platform(config: ConfigType, requested: str | None) -> str:
     # platform's final-validate hook merges duplicates anyway.
     available: dict[str, None] = {}
     for ota_item in config.get(CONF_OTA, []):
-        platform = ota_item.get(CONF_PLATFORM)
-        if platform in (CONF_ESPHOME, CONF_WEB_SERVER):
-            available[platform] = None
+        ota_platform = ota_item.get(CONF_PLATFORM)
+        if ota_platform in (CONF_ESPHOME, CONF_WEB_SERVER):
+            available[ota_platform] = None
 
     if not available:
         raise EsphomeError(
@@ -1343,8 +1354,12 @@ def _upload_via_native_api(
     # fall back to a plaintext upload
     noise_psk = None
     plaintext_fallback = False
+    allow_plaintext_upload = False
     if (encryption_conf := ota_conf.get(CONF_ENCRYPTION)) is not None:
         noise_psk = encryption_conf.get(CONF_KEY)
+        allow_plaintext_upload = bool(
+            encryption_conf.get(espota2.CONF_ALLOW_PLAINTEXT_UPLOAD)
+        )
         if not noise_psk:
             raise EsphomeError(
                 "OTA encryption is configured but no key was resolved; "
@@ -1377,6 +1392,12 @@ def _upload_via_native_api(
         ota_type = espota2.OTA_TYPE_UPDATE_PARTITION_TABLE
     elif getattr(args, "bootloader", False):
         check_partition_access("--bootloader")
+        if (
+            getattr(args, "file", None) is None
+            and (native := native_backend())
+            and (hint := getattr(native, "missing_image_hint", lambda: None)())
+        ):
+            raise EsphomeError(hint)
         binary = CORE.bootloader_bin
         ota_type = espota2.OTA_TYPE_UPDATE_BOOTLOADER
     if getattr(args, "file", None) is not None:
@@ -1395,6 +1416,7 @@ def _upload_via_native_api(
         ota_type,
         noise_psk,
         plaintext_fallback=plaintext_fallback,
+        allow_plaintext_upload=allow_plaintext_upload,
     )
 
 
@@ -1709,26 +1731,12 @@ def command_compile(args: ArgsProtocol, config: ConfigType) -> int | None:
     if exit_code != 0:
         return exit_code
     if CORE.is_host:
-        _LOGGER.info(
-            "Successfully compiled program to path '%s'", _host_program_path(config)
-        )
+        _LOGGER.info("Successfully compiled program to path '%s'", CORE.firmware_bin)
     else:
         _LOGGER.info("Successfully compiled program.")
     return 0
 
 
-def _host_program_path(config: ConfigType) -> str:
-    """Return the compiled host ELF path."""
-    if CORE.using_toolchain_esp_idf:
-        from esphome.espidf import toolchain
-
-        return str(toolchain.get_elf_path())
-    from esphome.platformio.toolchain import get_idedata
-
-    # Memoized by compile_program's own call; this is a dict lookup
-    return str(get_idedata(config).firmware_elf_path)
-
-
 def command_upload(args: ArgsProtocol, config: ConfigType) -> int | None:
     # Get devices, resolving special identifiers like OTA
     devices = choose_upload_log_host(
@@ -1765,6 +1773,18 @@ def command_logs(args: ArgsProtocol, config: ConfigType) -> int | None:
 
 
 def command_run(args: ArgsProtocol, config: ConfigType) -> int | None:
+    if (
+        CORE.skip_bootloader
+        and CORE.is_esp32
+        and CORE.using_toolchain_esp_idf
+        and any(
+            get_port_type(device) == PortType.SERIAL for device in (args.device or [])
+        )
+    ):
+        # Fail before the compile: the result could never flash over serial.
+        # Elsewhere the flag is ignored, so serial stays fine there.
+        _LOGGER.error("--skip-bootloader builds cannot be flashed over serial")
+        return 1
     exit_code = write_cpp(config)
     if exit_code != 0:
         return exit_code
@@ -1773,7 +1793,7 @@ def command_run(args: ArgsProtocol, config: ConfigType) -> int | None:
         return exit_code
     _LOGGER.info("Successfully compiled program.")
     if CORE.is_host:
-        program_path = _host_program_path(config)
+        program_path = str(CORE.firmware_bin)
         _LOGGER.info("Running program from path '%s'", program_path)
         return run_external_process(program_path)
 
@@ -1926,7 +1946,10 @@ def run_multiple_configs(
         safe_print()
 
         cmd = command_builder(f)
-        rc = run_external_process(*cmd)
+        # The parent already logged the Intel macOS warning; children skip it.
+        rc = run_external_process(
+            *cmd, env={**os.environ, _INTEL_MACOS_WARNED_ENV: "1"}
+        )
 
         if rc == 0:
             print_bar(f"[{color(AnsiFore.BOLD_GREEN, 'SUCCESS')}] {str(f)}")
@@ -1965,12 +1988,12 @@ def command_update_all(args: ArgsProtocol) -> int | None:
 def command_idedata(args: ArgsProtocol, config: ConfigType) -> int:
     import json
 
-    if CORE.using_toolchain_esp_idf:
-        # Native ESP-IDF derives idedata from the build's compile_commands.json,
-        # so the configuration must already be compiled.
-        from esphome.espidf import toolchain as espidf_toolchain
+    native_toolchain = native_backend()
 
-        idedata = espidf_toolchain.get_idedata()
+    if native_toolchain is not None:
+        # Native toolchains derive idedata from the build's
+        # compile_commands.json, so the configuration must already be compiled.
+        idedata = native_toolchain.get_idedata()
         if idedata is None:
             _LOGGER.error(
                 "No idedata available; compile the configuration first",
@@ -2009,6 +2032,22 @@ def command_analyze_memory(args: ArgsProtocol, config: ConfigType) -> int:
     from esphome.analyze_memory.cli import MemoryAnalyzerCLI
     from esphome.analyze_memory.ram_strings import RamStringsAnalyzer
 
+    # Refuse an unsupported toolchain before paying for a full compile
+    analysis_toolchain = analysis_backend()
+    if analysis_toolchain is None and not CORE.using_toolchain_platformio:
+        _LOGGER.error(
+            "analyze-memory is not supported with the '%s' toolchain on %s; "
+            "re-run with --toolchain platformio",
+            CORE.toolchain.value if CORE.toolchain else "unresolved",
+            CORE.target_platform,
+        )
+        return 1
+    if (
+        check_supported := getattr(analysis_toolchain, "check_analysis_supported", None)
+    ) is not None:
+        # Raises with the reason; before the compile, not after it
+        check_supported()
+
     # Always compile to ensure fresh data (fast if no changes - just relinks)
     exit_code = write_cpp(config)
     if exit_code != 0:
@@ -2020,13 +2059,30 @@ def command_analyze_memory(args: ArgsProtocol, config: ConfigType) -> int:
 
     # Get idedata for analysis
     idedata = None
-    if CORE.using_toolchain_esp_idf:
-        from esphome.espidf import toolchain
+    if analysis_toolchain is not None:
+        objdump = analysis_toolchain.get_objdump_path()
+        readelf = analysis_toolchain.get_readelf_path()
+        for tool in (objdump, readelf):
+            if not tool.is_file():
+                # The analyzer would silently fall back to host
+                # binutils, which cannot read the target ELF
+                _LOGGER.error(
+                    "%s is missing; the toolchain install may be incomplete "
+                    "(recompile, or run 'esphome clean-all' if it persists)",
+                    tool,
+                )
+                return 1
+        objdump_path = str(objdump)
+        readelf_path = str(readelf)
 
-        objdump_path = str(toolchain.get_objdump_path())
-        readelf_path = str(toolchain.get_readelf_path())
-
-        firmware_elf = toolchain.get_elf_path()
+        firmware_elf = analysis_toolchain.get_elf_path()
+        if not firmware_elf.is_file():
+            # The analyzer swallows tool failures, so a missing ELF would
+            # produce an exit-0 zeroed report
+            _LOGGER.error(
+                "%s is missing; compile the configuration first", firmware_elf
+            )
+            return 1
     else:
         from esphome.platformio import toolchain
 
@@ -2080,155 +2136,9 @@ def command_analyze_memory(args: ArgsProtocol, config: ConfigType) -> int:
 
 
 def command_rename(args: ArgsProtocol, config: ConfigType) -> int | None:
-    from esphome import yaml_util
+    from esphome.cli.rename import command_rename as run
 
-    new_name = args.name
-    for c in new_name:
-        if c not in ALLOWED_NAME_CHARS:
-            safe_print(
-                color(
-                    AnsiFore.BOLD_RED,
-                    f"'{c}' is an invalid character for names. Valid characters are: "
-                    f"{ALLOWED_NAME_CHARS} (lowercase, no spaces)",
-                )
-            )
-            return 1
-    # Load existing yaml file
-    raw_contents = CORE.config_path.read_text(encoding="utf-8")
-
-    yaml = yaml_util.load_yaml(CORE.config_path)
-    if CONF_ESPHOME not in yaml or CONF_NAME not in yaml[CONF_ESPHOME]:
-        safe_print(
-            color(
-                AnsiFore.BOLD_RED, "Complex YAML files cannot be automatically renamed."
-            )
-        )
-        return 1
-    old_name = yaml[CONF_ESPHOME][CONF_NAME]
-    match = re.match(r"^\$\{?([a-zA-Z0-9_]+)\}?$", old_name)
-    if match is None:
-        # Only swap the ``name:`` line that sits directly under the
-        # top-level ``esphome:`` block. A naked ``re.sub`` would
-        # also clobber any other ``name:`` line whose value happens
-        # to match (e.g. a sensor / output / wifi entry sharing the
-        # device's hostname), silently rewriting unrelated user
-        # configuration. The pattern anchors:
-        # - at the start of the line so ``friendly_name:``,
-        #   ``device_name:`` etc. don't match the trailing ``name:``
-        #   substring; and
-        # - at the end of the value (lookahead for whitespace +
-        #   comment + EOL) so ``old_name`` doesn't match as a
-        #   prefix of a longer value (``kitchen`` vs ``kitchen2``).
-        name_pattern = re.compile(
-            rf"^(\s*)name:\s+[\"']?{re.escape(old_name)}[\"']?(?=\s*(?:#|$))"
-        )
-        out_lines: list[str] = []
-        in_esphome_block = False
-        for line in raw_contents.splitlines(keepends=True):
-            if line and not line[0].isspace() and line.strip():
-                in_esphome_block = line.lstrip().startswith("esphome:")
-                out_lines.append(line)
-                continue
-            if in_esphome_block:
-                line = name_pattern.sub(rf'\1name: "{new_name}"', line, count=1)
-            out_lines.append(line)
-        new_raw = "".join(out_lines)
-    else:
-        old_name = yaml[CONF_SUBSTITUTIONS][match.group(1)]
-        if (
-            len(
-                re.findall(
-                    rf"^\s+{match.group(1)}:\s+[\"']?{old_name}[\"']?",
-                    raw_contents,
-                    flags=re.MULTILINE,
-                )
-            )
-            > 1
-        ):
-            safe_print(
-                color(AnsiFore.BOLD_RED, "Too many matches in YAML to safely rename")
-            )
-            return 1
-
-        new_raw = re.sub(
-            rf"^(\s+{match.group(1)}):\s+[\"']?{old_name}[\"']?",
-            f'\\1: "{new_name}"',
-            raw_contents,
-            flags=re.MULTILINE,
-        )
-
-    # ``new_name == old_name`` (after substitution resolution) is
-    # a no-op rewrite that would still queue a pointless re-flash.
-    # Catch it before the path-equality check below — covers the
-    # case where the config filename doesn't match the device name
-    # (e.g. ``weird-file.yaml`` whose ``esphome.name`` is
-    # ``kitchen``; running ``esphome rename weird-file.yaml kitchen``
-    # would otherwise just re-flash the same hostname).
-    if new_name == old_name:
-        safe_print(
-            color(
-                AnsiFore.BOLD_RED,
-                f"'{new_name}' is already the device's name.",
-            )
-        )
-        return 1
-
-    new_path: Path = CORE.config_dir / (new_name + ".yaml")
-    if new_path.resolve() == CORE.config_path.resolve():
-        safe_print(
-            color(
-                AnsiFore.BOLD_RED,
-                f"'{new_name}' is already the device's name.",
-            )
-        )
-        return 1
-    if new_path.exists():
-        safe_print(
-            color(
-                AnsiFore.BOLD_RED,
-                f"Cannot rename: {new_path} already exists. "
-                "Refusing to overwrite an existing configuration.",
-            )
-        )
-        return 1
-    safe_print(
-        f"Updating {color(AnsiFore.CYAN, str(CORE.config_path))} to {color(AnsiFore.CYAN, str(new_path))}"
-    )
-    print()
-
-    new_path.write_text(new_raw, encoding="utf-8")
-
-    rc = run_external_process(*ESPHOME_COMMAND, "config", str(new_path))
-    if rc != 0:
-        safe_print(color(AnsiFore.BOLD_RED, "Rename failed. Reverting changes."))
-        new_path.unlink()
-        return 1
-
-    cli_args = [
-        "run",
-        str(new_path),
-        "--no-logs",
-        "--device",
-        CORE.address,
-    ]
-
-    if args.dashboard:
-        cli_args.insert(0, "--dashboard")
-
-    try:
-        rc = run_external_process(*ESPHOME_COMMAND, *cli_args)
-    except KeyboardInterrupt:
-        rc = 1
-    if rc != 0:
-        new_path.unlink()
-        return 1
-
-    if CORE.config_path != new_path:
-        CORE.config_path.unlink()
-
-    safe_print(color(AnsiFore.BOLD_GREEN, "SUCCESS"))
-    print()
-    return 0
+    return run(args, config)
 
 
 PRE_CONFIG_ACTIONS = {
@@ -2263,6 +2173,15 @@ SIMPLE_CONFIG_ACTIONS = [
 ]
 
 
+def _add_skip_bootloader_arg(parser: argparse.ArgumentParser) -> None:
+    parser.add_argument(
+        "--skip-bootloader",
+        help="Do not build the bootloader or the factory image; "
+        "the result can only be flashed over OTA.",
+        action="store_true",
+    )
+
+
 def _add_states_args(parser: argparse.ArgumentParser) -> None:
     """Add mutually exclusive ``--states``/``--no-states`` flags to a parser.
 
@@ -2343,7 +2262,8 @@ def parse_args(argv):
         metavar="{" + ",".join(t.value for t in Toolchain) + "}",
         help=(
             "Select toolchain for compiling. Overrides '<platform>.toolchain' in YAML. "
-            f"Default: {Toolchain.PLATFORMIO.value}."
+            "Default: the platform's native toolchain where it has one, else "
+            f"{Toolchain.PLATFORMIO.value}."
         ),
     )
 
@@ -2403,6 +2323,7 @@ def parse_args(argv):
         help="Only generate source code, do not compile.",
         action="store_true",
     )
+    _add_skip_bootloader_arg(parser_compile)
 
     parser_upload = subparsers.add_parser(
         "upload",
@@ -2466,7 +2387,7 @@ def parse_args(argv):
         "-r",
         action="store_true",
         help="Reset the device before starting serial logs.",
-        default=os.getenv("ESPHOME_SERIAL_LOGGING_RESET"),
+        default=get_bool_env("ESPHOME_SERIAL_LOGGING_RESET"),
     )
     _add_states_args(parser_logs)
 
@@ -2499,6 +2420,7 @@ def parse_args(argv):
     parser_run.add_argument(
         "--no-logs", help="Disable starting logs.", action="store_true"
     )
+    _add_skip_bootloader_arg(parser_run)
 
     _add_states_args(parser_run)
 
@@ -2507,7 +2429,7 @@ def parse_args(argv):
         "-r",
         action="store_true",
         help="Reset the device before starting serial logs.",
-        default=os.getenv("ESPHOME_SERIAL_LOGGING_RESET"),
+        default=get_bool_env("ESPHOME_SERIAL_LOGGING_RESET"),
     )
     parser_run.add_argument(
         "--ota-platform",
@@ -2696,6 +2618,27 @@ def _warn_if_source_tree_mismatch() -> None:
     )
 
 
+_INTEL_MACOS_REMOVAL = "2027.6.0"
+_INTEL_MACOS_WARNED_ENV = "ESPHOME_INTEL_MACOS_WARNED"
+
+
+def _warn_if_intel_macos() -> None:
+    """Warn that Intel (x86_64) Python on macOS loses support by _INTEL_MACOS_REMOVAL."""
+    if (
+        not IS_MACOS
+        or platform.machine() != "x86_64"
+        or _INTEL_MACOS_WARNED_ENV in os.environ
+    ):
+        return
+    _LOGGER.warning(
+        "Support for Intel Macs will end in ESPHome %s or earlier. The Python "
+        "packages ESPHome depends on have stopped publishing Intel macOS builds, "
+        "so future releases will not install on this machine. On an Apple "
+        "Silicon Mac, switch to a native arm64 Python.",
+        _INTEL_MACOS_REMOVAL,
+    )
+
+
 def run_esphome(argv):
     from esphome.address_cache import AddressCache
 
@@ -2715,6 +2658,7 @@ def run_esphome(argv):
 
     setup_log(log_level=args.log_level)
     _warn_if_source_tree_mismatch()
+    _warn_if_intel_macos()
 
     if args.command in PRE_CONFIG_ACTIONS:
         try:
@@ -2758,6 +2702,7 @@ def run_esphome(argv):
 
     CORE.config_path = conf_path
     CORE.dashboard = args.dashboard
+    CORE.skip_bootloader = getattr(args, "skip_bootloader", False)
     if args.toolchain is not None:
         # CLI toolchain wins over esp32.toolchain in YAML.
         CORE.toolchain = args.toolchain

esphome/address_cache.py

@@ -148,6 +148,13 @@ class AddressCache:
                 continue
             hostname, ips = arg.split("=", 1)
             # Normalize hostname for consistent lookups
-            normalized = normalize_hostname(hostname)
-            cache[normalized] = [ip.strip() for ip in ips.split(",")]
+            normalized = normalize_hostname(hostname.strip())
+            addresses = [ip for value in ips.split(",") if (ip := value.strip())]
+            if not normalized or not addresses:
+                _LOGGER.warning(
+                    "Invalid cache entry: %s (hostname and at least one address are required)",
+                    arg,
+                )
+                continue
+            cache[normalized] = addresses
         return cache

esphome/analyze_memory/toolchain.py

@@ -37,7 +37,7 @@ def find_elf_path(build_path: Path) -> Path | None:
     """
     name = build_path.name
     for candidate in (
-        # Native ESP-IDF: idf.py writes build/<name>.elf, which ESPHome copies
+        # Native ESP-IDF: the build writes build/<name>.elf, which ESPHome copies
         # to build/firmware.elf (see espidf.toolchain.create_elf_copy)
         build_path / "build" / "firmware.elf",
         # PlatformIO
@@ -68,12 +68,16 @@ def idedata_candidates(build_path: Path) -> list[Path]:
         The candidate idedata JSON paths, most specific first
     """
     name = build_path.name
+    data_dir = build_path.parent.parent / "idedata"
+    # Native backends suffix the cache by toolchain (<name>.arduino.json)
+    suffixed = sorted(data_dir.glob(f"{name}.*.json")) if data_dir.is_dir() else []
     return [
         # In .pioenvs for test builds
         build_path / ".pioenvs" / name / "idedata.json",
         # Both toolchains cache it in the data dir, which holds this build dir:
         # <data_dir>/idedata/<name>.json next to <data_dir>/build/<name>
         build_path.parent.parent / "idedata" / f"{name}.json",
+        *suffixed,
         # Regular builds, invoked from the config dir or from anywhere
         Path.cwd() / ".esphome" / "idedata" / f"{name}.json",
         Path.home() / ".esphome" / "idedata" / f"{name}.json",

esphome/arduino/library.py

@@ -2,7 +2,9 @@
 
 Bundled names build straight from the framework tree; everything else goes
 through ``esphome.platformio.library``. Mirrors ``lib_ldf_mode=off``: each
-library builds its own archive; all include dirs join one global path.
+library builds its own archive; all include dirs join one global path. The
+host build reuses it without a framework tree: nothing is bundled there and
+every name resolves from the registry.
 
 Deviations from PlatformIO: flat-layout libraries get the recursive default
 source filter; ``dot_a_linkage`` is honored; bundled libraries never run a
@@ -342,13 +344,24 @@ def _check_unfulfilled_provides(
 
 
 def resolve_libraries(
-    framework_path: Path, *, pio_platform: str, board_mcu: str, cache_key: str
+    framework_path: Path | None,
+    *,
+    pio_platform: str,
+    board_mcu: str,
+    cache_key: str,
+    framework: str | None = "arduino",
+    manifest_optional: bool = False,
 ) -> list[ArduinoLibrary]:
     """Resolve every ``cg.add_library()`` entry into an :class:`ArduinoLibrary`.
 
     ``pio_platform``/``board_mcu`` filter manifests the way PlatformIO would
     for that core (e.g. ``espressif8266``/``esp8266``); ``cache_key`` keys the
-    shared converter's download cache.
+    shared converter's download cache. ``framework`` is the manifest
+    framework token the compatibility check warns about; None skips it.
+    A None ``framework_path`` means no core-bundled libraries exist (the
+    host build): every name resolves from the registry.
+    ``manifest_optional`` accepts libraries without a manifest, built with
+    PlatformIO's default layout.
 
     The returned list is not topologically sorted, so the caller must link
     the archives inside one ``--start-group``/``--end-group`` pair (the
@@ -359,18 +372,22 @@ def resolve_libraries(
     # PlatformIO's lib_ignore covers framework-bundled libraries too; the
     # shared converter only filters the registry/git ones.
     lib_ignore = lib_ignore_set()
-    # Exact directory names keep membership case-sensitive everywhere
-    # (an is_dir() probe would match "wire" on macOS/Windows and build
-    # the bundled Wire twice)
-    libraries_dir = framework_path / "libraries"
-    if not libraries_dir.is_dir():
-        # A registry fallback would fail later with a misleading
-        # package-not-found error per bundled name
-        raise EsphomeError(
-            f"{libraries_dir} is missing; the framework install may be "
-            "incomplete (run 'esphome clean-all')"
+    bundled_dir_names: frozenset[str] = frozenset()
+    if framework_path is not None:
+        # Exact directory names keep membership case-sensitive everywhere
+        # (an is_dir() probe would match "wire" on macOS/Windows and build
+        # the bundled Wire twice)
+        libraries_dir = framework_path / "libraries"
+        if not libraries_dir.is_dir():
+            # A registry fallback would fail later with a misleading
+            # package-not-found error per bundled name
+            raise EsphomeError(
+                f"{libraries_dir} is missing; the framework install may be "
+                "incomplete (run 'esphome clean-all')"
+            )
+        bundled_dir_names = frozenset(
+            p.name for p in libraries_dir.iterdir() if p.is_dir()
         )
-    bundled_dir_names = frozenset(p.name for p in libraries_dir.iterdir() if p.is_dir())
 
     def _provided(name: object) -> bool:
         return _is_safe_library_name(name) and name in bundled_dir_names
@@ -497,12 +514,13 @@ def resolve_libraries(
 
     backend = LibraryBackend(
         platform=pio_platform,
-        framework="arduino",
+        framework=framework,
         emit=_emit,
         cache_key=cache_key,
         # The walk must not resolve bundled names from the registry;
         # _add_bundled_dependencies adds them after emit
         provides=_provided,
+        manifest_optional=manifest_optional,
     )
     if external:
         convert_libraries(external, backend)

esphome/arduino8266/framework.py

@@ -3,12 +3,12 @@
 Artifacts land in a machine-global cache (shared across projects, like the
 ESP-IDF install in ``esphome.espidf.framework``):
 
-    <cache>/arduino8266/frameworks/<version>/   framework-arduinoespressif8266
-    <cache>/arduino8266/toolchains/<version>/   toolchain-xtensa (gcc 10.3)
+    <cache>/arduino8266/frameworks/<tag>/   the Arduino core
+    <cache>/arduino8266/toolchains/<tag>/   xtensa-lx106-elf gcc 10.3
 
-Packages come from the PlatformIO registry (identical bits to the PlatformIO
-backend); ``ESPHOME_ARDUINO8266_*_MIRRORS`` overrides the URLs. ninja comes
-from PATH or the ninja PyPI wheel.
+Both come from esphome-libs releases pinned below;
+``ESPHOME_ARDUINO8266_*_MIRRORS`` overrides the URLs, with ``{VERSION}``
+standing for the release tag. ninja comes from PATH or the ninja PyPI wheel.
 """
 
 from __future__ import annotations
@@ -17,18 +17,74 @@ import os
 from pathlib import Path
 from typing import NamedTuple
 
-from esphome.build_helpers.ccache import ccache_defaults_env
+from esphome.build_helpers.ccache import ccache_env
 from esphome.build_helpers.ninja import find_ninja
 from esphome.build_helpers.tools_cache import ARDUINO8266_TOOLS_CACHE, tools_cache_path
 from esphome.core import EsphomeError, Version
 from esphome.framework_helpers import str_to_lst_of_str
-from esphome.platformio.registry import install_package, prefetch_packages
+from esphome.platformio.registry import (
+    Download,
+    PackageSpec,
+    Resolver,
+    get_systype,
+    install_packages,
+    prefetch_packages,
+)
 
-FRAMEWORK_PACKAGE = "framework-arduinoespressif8266"
-TOOLCHAIN_PACKAGE = "toolchain-xtensa"
+FRAMEWORK_PACKAGE = "arduino-esp8266"
+_FRAMEWORK_RELEASES = "https://github.com/esphome-libs/arduino-esp8266/releases/"
+
+
+class FrameworkRelease(NamedTuple):
+    tag: str
+    sha256: str
+    size: int
+
+    def download(self) -> Download:
+        archive = f"{FRAMEWORK_PACKAGE}-{self.tag}.tar.gz"
+        url = f"{_FRAMEWORK_RELEASES}download/{self.tag}/{archive}"
+        return Download(url, self.sha256, self.size)
+
+
+# Arduino core version -> its build in esphome-libs/arduino-esp8266
+FRAMEWORK_RELEASES: dict[Version, FrameworkRelease] = {
+    Version(3, 1, 2): FrameworkRelease(
+        "3.1.2-esphome.2",
+        "9377d63f569c524823199b1897ed7fde2063e50e2676dd519a0d758ab73678f7",
+        37192128,
+    ),
+}
+
+TOOLCHAIN_PACKAGE = "toolchain-xtensa-lx106-elf"
 # gcc 10.3, the toolchain Arduino core 3.x builds with; the build
 # generator's compile flags are tuned to it.
-TOOLCHAIN_VERSION = "2.100300.220621"
+TOOLCHAIN_VERSION = "10.3.0-esphome.2"
+_TOOLCHAIN_RELEASES = (
+    "https://github.com/esphome-libs/xtensa-lx106-elf-toolchain/releases/"
+)
+# Registry system tag -> (sha256, size) of that host's archive
+TOOLCHAIN_BUILDS: dict[str, tuple[str, int]] = {
+    "darwin_arm64": (
+        "849cede44d4d5c6ea0f14099783239f559f46327bea314281814f2652b486201",
+        60830321,
+    ),
+    "darwin_x86_64": (
+        "ca69904daabf0c5983b372423e5e62f49182a793e992c052e94666852470c897",
+        64149487,
+    ),
+    "linux_aarch64": (
+        "60a49a4f082bf246544bd409a9517dbbcab19bb30ac9decbee544b896aaccbd6",
+        67573397,
+    ),
+    "linux_x86_64": (
+        "1fba33ca1494ec79f2776e0e37eca93282d30f8bb9992f5f4f9a655d6fff1db4",
+        68431336,
+    ),
+    "windows_amd64": (
+        "af9066b0e5bf036f04f2bd9d08b89b81a7f183c57dac0abcaff71dd861cf5f3b",
+        67664137,
+    ),
+}
 
 ESPHOME_ARDUINO8266_FRAMEWORK_MIRRORS = str_to_lst_of_str(
     os.environ.get("ESPHOME_ARDUINO8266_FRAMEWORK_MIRRORS", "")
@@ -44,36 +100,40 @@ def get_arduino8266_tools_path() -> Path:
     return tools_cache_path(*ARDUINO8266_TOOLS_CACHE)
 
 
-# 3.1.1 rather than 3.1.0: the registry has no packages for 3.0.0, 3.0.1 or 3.1.0
-MIN_FRAMEWORK_VERSION = Version(3, 1, 1)
-
-
-def framework_package_version(ver: Version) -> str:
-    """Map an Arduino core version to its registry package version (3.1.2 ->
-    3.30102.0; the leading 3 is the package major).
-
-    Exact registry names for 3.x cores; callers floor at MIN_FRAMEWORK_VERSION.
-    """
-    if ver.major > 3:
+def framework_release(version: Version) -> FrameworkRelease:
+    if (release := FRAMEWORK_RELEASES.get(version)) is None:
         raise EsphomeError(
-            f"Arduino core {ver} is not supported yet; "
-            "the newest known core series is 3.x"
+            f"'toolchain: arduino' has no build of Arduino core {version}; "
+            f"available: {', '.join(str(v) for v in FRAMEWORK_RELEASES)}. "
+            "Use one of those or 'toolchain: platformio'"
         )
-    if ver.major < 3:
-        raise EsphomeError(
-            f"Arduino core {ver} is not supported; ESPHome requires core 3.x"
-        )
-    return f"3.{ver.major}{ver.minor:02d}{ver.patch:02d}.0"
+    return release
 
 
-def get_framework_path(package_version: str) -> Path:
-    return get_arduino8266_tools_path() / "frameworks" / package_version
+def get_framework_path(tag: str) -> Path:
+    return get_arduino8266_tools_path() / "frameworks" / tag
 
 
 def get_toolchain_path() -> Path:
     return get_arduino8266_tools_path() / "toolchains" / TOOLCHAIN_VERSION
 
 
+def toolchain_download() -> Download:
+    """The toolchain archive for the current host."""
+    systype = get_systype()
+    if (build := TOOLCHAIN_BUILDS.get(systype)) is None:
+        raise EsphomeError(
+            f"There is no ESP8266 toolchain for this system ({systype}); "
+            f"supported systems are {', '.join(sorted(TOOLCHAIN_BUILDS))}. "
+            "Either set 'toolchain: platformio' under 'esp8266:', or point "
+            "ESPHOME_ARDUINO8266_TOOLCHAIN_MIRRORS at a toolchain archive"
+        )
+    sha256, size = build
+    archive = f"{TOOLCHAIN_PACKAGE}-{TOOLCHAIN_VERSION}-{systype}.tar.gz"
+    url = f"{_TOOLCHAIN_RELEASES}download/{TOOLCHAIN_VERSION}/{archive}"
+    return Download(url, sha256, size)
+
+
 class InstalledPaths(NamedTuple):
     """Locations of the installed framework, toolchain, and ninja binary."""
 
@@ -84,29 +144,22 @@ class InstalledPaths(NamedTuple):
 
 def check_and_install(framework_version: Version) -> InstalledPaths:
     """Ensure framework, toolchain, and ninja are installed; return their paths."""
-    if framework_version < MIN_FRAMEWORK_VERSION:
-        # Config validation enforces this too; keep the module honest when
-        # called directly.
-        raise EsphomeError(
-            f"The native toolchain requires the Arduino core "
-            f">= {MIN_FRAMEWORK_VERSION}, got {framework_version}"
-        )
+    release = framework_release(framework_version)
     # Probe the cheap local dependency before ~110 MB of downloads
     ninja_path = find_ninja()
-    package_version = framework_package_version(framework_version)
-    framework_path = get_framework_path(package_version)
+    framework_path = get_framework_path(release.tag)
     downloads_dir = get_arduino8266_tools_path() / "downloads"
     toolchain_path = get_toolchain_path()
     # One spec per package: the prefetch and the installs must agree
     specs = (
-        (
+        PackageSpec(
             FRAMEWORK_PACKAGE,
-            package_version,
+            release.tag,
             framework_path,
             ESPHOME_ARDUINO8266_FRAMEWORK_MIRRORS,
             ("cores/esp8266", "tools/sdk", "libraries"),
         ),
-        (
+        PackageSpec(
             TOOLCHAIN_PACKAGE,
             TOOLCHAIN_VERSION,
             toolchain_path,
@@ -115,10 +168,17 @@ def check_and_install(framework_version: Version) -> InstalledPaths:
             ("bin", "xtensa-lx106-elf"),
         ),
     )
-    # Fetch both archives at once; the installs below verify and extract
-    prefetch_packages([spec[:4] for spec in specs], downloads_dir)
-    for name, version, dest, mirrors, expect in specs:
-        install_package(name, version, dest, mirrors, downloads_dir, expect=expect)
+    # Resolved only when a download is needed, so an installed toolchain
+    # keeps working on a host without a build; a mirror override wins
+    resolvers: dict[str, Resolver] = {}
+    if not ESPHOME_ARDUINO8266_FRAMEWORK_MIRRORS:
+        resolvers[FRAMEWORK_PACKAGE] = release.download
+    if not ESPHOME_ARDUINO8266_TOOLCHAIN_MIRRORS:
+        resolvers[TOOLCHAIN_PACKAGE] = toolchain_download
+    # Fetch both archives at once; the installs verify and extract them.
+    # One spec list for both, so the two phases cannot drift.
+    prefetch_packages(specs, downloads_dir, resolvers)
+    install_packages(specs, downloads_dir, resolvers)
     return InstalledPaths(
         framework=framework_path, toolchain=toolchain_path, ninja=ninja_path
     )
@@ -143,17 +203,5 @@ def get_build_env(toolchain_path: Path, ccache: str | None) -> dict[str, str]:
         *filter(None, env.get("PATH", "").split(os.pathsep)),
     ]
     env["PATH"] = os.pathsep.join(parts)
-    env.update(ccache_env(ccache))
+    env.update(ccache_env(ccache, ARDUINO8266_TOOLS_CACHE))
     return env
-
-
-def ccache_env(ccache: str | None) -> dict[str, str]:
-    """Return ccache settings for the build subprocess (not os.environ).
-
-    ``ccache`` is the pre-resolved binary (resolve_ccache_path), or None
-    when disabled. Values the user already set in the environment are
-    respected.
-    """
-    if ccache is None:
-        return {}
-    return ccache_defaults_env(get_arduino8266_tools_path() / "ccache")

esphome/arduino8266/toolchain.py

@@ -0,0 +1,212 @@
+"""Native Arduino ESP8266 build driver (the PlatformIO ``run`` equivalent)."""
+
+from __future__ import annotations
+
+import logging
+from pathlib import Path
+import subprocess
+from typing import TYPE_CHECKING
+
+from esphome.build_helpers.ccache import resolve_absolute_ccache_path
+from esphome.build_helpers.native import warn_ignored_platformio_options
+from esphome.build_helpers.ninja import refresh_compile_commands
+from esphome.const import (
+    CONF_COMPILE_PROCESS_LIMIT,
+    CONF_ESPHOME,
+    KEY_CORE,
+    KEY_FRAMEWORK_VERSION,
+)
+from esphome.core import CORE
+from esphome.types import ConfigType
+
+if TYPE_CHECKING:
+    from esphome.arduino8266.framework import InstalledPaths
+
+_LOGGER = logging.getLogger(__name__)
+
+# ESP8266 user RAM (matches upload.maximum_ram_size in every board manifest)
+_MAX_RAM_SIZE = 81920
+
+
+_RAM_SECTIONS = (".data", ".rodata", ".bss")
+_FLASH_SECTIONS = (".irom0.text", ".text", ".text1", ".data", ".rodata")
+
+
+def get_build_dir() -> Path:
+    return CORE.relative_pioenvs_path(CORE.name)
+
+
+def get_elf_path() -> Path:
+    return get_build_dir() / "firmware.elf"
+
+
+def _toolchain_tool(name: str) -> Path:
+    # Imported here, not at module scope: the serial upload/logs fast path
+    # resolves this module for its artifact paths alone, and framework
+    # pulls in the whole package-download stack
+    from esphome.arduino8266 import framework
+
+    return framework.toolchain_tool(framework.get_toolchain_path(), name)
+
+
+def get_factory_firmware_path() -> Path:
+    """The image to serial-flash at 0x0 (same bytes as firmware.bin: the
+    8266 factory copy exists for artifact-contract parity, not content)."""
+    return get_build_dir() / "firmware.factory.bin"
+
+
+def get_addr2line_path() -> Path:
+    return _toolchain_tool("addr2line")
+
+
+def get_objdump_path() -> Path:
+    return _toolchain_tool("objdump")
+
+
+def get_readelf_path() -> Path:
+    return _toolchain_tool("readelf")
+
+
+def run_compile(config: ConfigType, verbose: bool) -> int:
+    from esphome.arduino8266 import framework
+    from esphome.build_gen import arduino8266 as build_gen
+    from esphome.core.config import NATIVE_ARDUINO_CONSUMED_PIO_OPTIONS
+
+    warn_ignored_platformio_options(NATIVE_ARDUINO_CONSUMED_PIO_OPTIONS)
+    paths = framework.check_and_install(CORE.data[KEY_CORE][KEY_FRAMEWORK_VERSION])
+    # Resolved once: the probe is not free and three consumers need it
+    ccache = resolve_absolute_ccache_path()
+    ninja_changed = build_gen.write_project(paths, ccache)
+
+    build_dir = get_build_dir()
+    env = framework.get_build_env(paths.toolchain, ccache)
+    refresh_compile_commands(paths.ninja, build_dir, env, ninja_changed)
+
+    cmd = [str(paths.ninja)]
+    if verbose:
+        cmd.append("-v")
+    if jobs := config[CONF_ESPHOME].get(CONF_COMPILE_PROCESS_LIMIT):
+        cmd += ["-j", str(jobs)]
+    # Explicit targets: a generator defect that drops them fails loudly
+    # instead of a green no-op run leaving stale artifacts in place
+    targets = ["firmware.factory.bin", "firmware.ota.bin"]
+    cmd += targets
+
+    # cwd, not -C: drops ninja's "Entering directory" banner
+    _LOGGER.debug("Running: %s", " ".join(cmd))
+    rc = subprocess.run(
+        cmd, cwd=build_dir, env=env, check=False, close_fds=False
+    ).returncode
+    if rc != 0:
+        return rc
+
+    # ninja already refused missing targets; existence covers a rule that
+    # ran but wrote elsewhere
+    build_dir_artifacts = (
+        get_elf_path(),
+        build_dir / "firmware.bin",
+        get_factory_firmware_path(),
+        build_dir / "firmware.ota.bin",
+    )
+    for artifact in build_dir_artifacts:
+        if not artifact.is_file():
+            _LOGGER.error("Build produced no %s", artifact)
+            return 1
+
+    if not _print_size_summary(build_dir, paths):
+        # Cause already warned; name the consequence for CI harnesses
+        _LOGGER.warning("Firmware size summary unavailable for this build")
+    from esphome.build_helpers.idedata import warn_if_idedata_missing
+
+    warn_if_idedata_missing(lambda: get_idedata(ccache))
+    return 0
+
+
+def _parse_app_size(build_dir: Path, paths: InstalledPaths) -> int | None:
+    """Read the app flash budget (irom0_0_seg length) from the linker script."""
+    from esphome.build_gen.arduino8266 import get_flash_ld_path
+    from esphome.components.esp8266.build_surgery import segment_length
+
+    # Warnings, not debug: without the app size the Flash summary line is
+    # dropped and CI's memory-impact extraction loses its flash metric.
+    ld_path = get_flash_ld_path(build_dir, paths)
+    try:
+        ld_text = ld_path.read_text(encoding="utf-8")
+    except (OSError, UnicodeDecodeError) as err:
+        # A corrupt script degrades the same way, never aborts the build
+        _LOGGER.warning("Cannot read linker script for the Flash summary: %s", err)
+        return None
+    if not (app_size := segment_length(ld_text, "irom0_0_seg")):
+        _LOGGER.warning("No usable irom0_0_seg in %s; skipping Flash summary", ld_path)
+        return None
+    return app_size
+
+
+def _print_size_summary(build_dir: Path, paths: InstalledPaths) -> bool:
+    """Print the RAM/Flash lines ``ci_memory_impact_extract.py`` parses;
+    False when skipped."""
+    from esphome.arduino8266.framework import toolchain_tool
+    from esphome.build_helpers.size_summary import print_size_line
+
+    try:
+        result = subprocess.run(
+            [
+                str(toolchain_tool(paths.toolchain, "size")),
+                "-A",
+                "-d",
+                str(build_dir / "firmware.elf"),
+            ],
+            capture_output=True,
+            text=True,
+            check=True,
+            close_fds=False,
+        )
+    except (OSError, subprocess.CalledProcessError) as err:
+        # The summary is a bonus artifact like idedata; a truncated
+        # toolchain extraction must not discard an already-linked build
+        _LOGGER.warning("Could not summarize firmware size: %s", err)
+        return False
+    # -d prints decimal sizes; anything else trips the missing-sections guard
+    sections = {
+        parts[0]: int(parts[1])
+        for line in result.stdout.splitlines()
+        if (parts := line.split())[:1] and parts[0].startswith(".") and len(parts) >= 2
+        if parts[1].isdigit()
+    }
+    if missing := set(_RAM_SECTIONS + _FLASH_SECTIONS) - set(sections):
+        # A defaulted 0 would print a confidently wrong total for CI's metric
+        _LOGGER.warning(
+            "Size output is missing section(s) %s; skipping the size summary",
+            ", ".join(sorted(missing)),
+        )
+        return False
+    # Resolve the flash budget before printing: a RAM line without its
+    # Flash line would skew CI's memory-impact extraction
+    app_size = _parse_app_size(build_dir, paths)
+    if not app_size:
+        return False
+    ram = sum(sections[s] for s in _RAM_SECTIONS)
+    flash = sum(sections[s] for s in _FLASH_SECTIONS)
+    print_size_line("RAM", ram, _MAX_RAM_SIZE)
+    print_size_line("Flash", flash, app_size)
+    return True
+
+
+def get_idedata(ccache: str | None = None) -> dict | None:
+    """Derive idedata from the build's compile_commands.json (same
+    contract as ``espidf.toolchain.get_idedata``)."""
+    from esphome.build_helpers.idedata import load_or_build_idedata
+
+    # A disabled ccache resolves to None without spawning anything, so
+    # re-resolving here costs nothing when the caller has no answer
+    launcher = ccache or resolve_absolute_ccache_path()
+    return load_or_build_idedata(
+        get_build_dir() / "compile_commands.json",
+        get_elf_path(),
+        # Suffixed so a platformio->arduino->platformio round trip on one
+        # config never serves the other toolchain's cache shape
+        CORE.relative_internal_path("idedata", f"{CORE.name}.arduino.json"),
+        # The compile DB's commands carry the same ccache prefix the ninja
+        # rules were generated with
+        launcher=str(launcher) if launcher else None,
+    )

esphome/automation.py

@@ -1,5 +1,8 @@
+from collections.abc import Callable
 from dataclasses import dataclass, field
 import logging
+import string
+from typing import Any
 
 import esphome.codegen as cg
 import esphome.config_validation as cv
@@ -18,18 +21,46 @@ from esphome.const import (
     CONF_TYPE_ID,
     CONF_UPDATE_INTERVAL,
 )
-from esphome.core import ID, Lambda
+from esphome.core import CORE, ID, EsphomeError, HexInt, Lambda
 from esphome.cpp_generator import (
+    FlashStringLiteral,
     LambdaExpression,
     MockObj,
     MockObjClass,
     TemplateArgsType,
+    call_lambda,
 )
 from esphome.schema_extractors import SCHEMA_EXTRACT, schema_extractor
-from esphome.types import ConfigType
+from esphome.types import ConfigType, SafeExpType
 from esphome.util import Registry
 
 
+def progmem_bytes(name: str, data: bytes | list[int]) -> MockObj:
+    """Shared PROGMEM table for constant bytes; equal payloads share one, empty is nullptr."""
+    if not data:
+        return cg.nullptr
+    return cg.shared_progmem_array(
+        name, cg.uint8, cg.ArrayInitializer(*(HexInt(x) for x in data))
+    )
+
+
+async def templatable_bytes(
+    value: Any,
+    args: TemplateArgsType,
+    set_template: MockObj,
+    set_static: MockObj,
+    table_name: str,
+) -> None:
+    """Set a TemplatableBytes: a lambda via set_template, constant bytes via set_static."""
+    if cg.is_template(value):
+        fn = await cg.templatable(value, args, cg.std_vector.template(cg.uint8))
+        cg.add(set_template(fn))
+    elif len(value) > 0xFFFF:
+        raise EsphomeError(f"Byte payload is {len(value)} bytes; the maximum is 65535")
+    else:
+        cg.add(set_static(progmem_bytes(table_name, value), len(value)))
+
+
 def maybe_simple_id(*validators):
     """Allow a raw ID to be specified in place of a config block.
     If the value that's being validated is a dictionary, it's passed as-is to the specified validators. Otherwise, it's
@@ -57,6 +88,7 @@ def maybe_conf(conf, *validators):
         with cv.remove_prepend_path([conf]):
             return validator({conf: value})
 
+    validate.inner_schema = validator
     return validate
 
 
@@ -102,6 +134,101 @@ def register_condition(name: str, condition_type: MockObjClass, schema: cv.Schem
     return CONDITION_REGISTRY.register(name, condition_type, schema)
 
 
+async def _build_with_parent(
+    config: ConfigType,
+    automation_id: ID,
+    template_arg: cg.TemplateArguments,
+    args: TemplateArgsType,
+) -> MockObj:
+    parent = await cg.get_variable(config[CONF_ID])
+    return cg.new_Pvariable(automation_id, template_arg, parent)
+
+
+async def _build_without_parent(
+    config: ConfigType,
+    automation_id: ID,
+    template_arg: cg.TemplateArguments,
+    args: TemplateArgsType,
+) -> MockObj:
+    return cg.new_Pvariable(automation_id, template_arg)
+
+
+async def _build_parented(
+    config: ConfigType,
+    automation_id: ID,
+    template_arg: cg.TemplateArguments,
+    args: TemplateArgsType,
+) -> MockObj:
+    var = cg.new_Pvariable(automation_id, template_arg)
+    await cg.register_parented(var, config[CONF_ID])
+    return var
+
+
+def register_simple_action(
+    name: str,
+    action_type: MockObjClass,
+    schema: cv.Schema,
+    *,
+    synchronous: bool,
+) -> None:
+    """Register an action whose constructor takes the object named by ``config[CONF_ID]``.
+
+    Use the ``register_action`` decorator instead when the builder must also set fields.
+    """
+    register_action(name, action_type, schema, synchronous=synchronous)(
+        _build_with_parent
+    )
+
+
+def register_simple_condition(
+    name: str, condition_type: MockObjClass, schema: cv.Schema
+) -> None:
+    """Condition counterpart of ``register_simple_action``."""
+    register_condition(name, condition_type, schema)(_build_with_parent)
+
+
+def register_bare_action(
+    name: str,
+    action_type: MockObjClass,
+    schema: cv.Schema,
+    *,
+    synchronous: bool,
+) -> None:
+    """Register an action whose constructor takes no arguments."""
+    register_action(name, action_type, schema, synchronous=synchronous)(
+        _build_without_parent
+    )
+
+
+def register_bare_condition(
+    name: str, condition_type: MockObjClass, schema: cv.Schema
+) -> None:
+    """Condition counterpart of ``register_bare_action``."""
+    register_condition(name, condition_type, schema)(_build_without_parent)
+
+
+def register_parented_action(
+    name: str,
+    action_type: MockObjClass,
+    schema: cv.Schema,
+    *,
+    synchronous: bool,
+) -> None:
+    """Register an action deriving from ``Parented<T>``.
+
+    The object is constructed without arguments and ``set_parent()`` receives the object
+    named by ``config[CONF_ID]``.
+    """
+    register_action(name, action_type, schema, synchronous=synchronous)(_build_parented)
+
+
+def register_parented_condition(
+    name: str, condition_type: MockObjClass, schema: cv.Schema
+) -> None:
+    """Condition counterpart of ``register_parented_action``."""
+    register_condition(name, condition_type, schema)(_build_parented)
+
+
 Action = cg.esphome_ns.class_("Action")
 Trigger = cg.esphome_ns.class_("Trigger")
 ACTION_REGISTRY = Registry()
@@ -112,6 +239,308 @@ validate_action_list = cv.validate_registry("action", ACTION_REGISTRY)
 validate_condition = cv.validate_registry_entry("condition", CONDITION_REGISTRY)
 validate_condition_list = cv.validate_registry("condition", CONDITION_REGISTRY)
 
+ApplyAction = cg.esphome_ns.class_("ApplyAction", Action)
+ApplyCondition = cg.esphome_ns.class_("ApplyCondition", Condition)
+
+
+def flash_string(config: ConfigType, value: str) -> str:
+    """Default renderer for ``std::string`` constants; copies the literal out of flash on ESP8266."""
+    if CORE.is_esp8266:
+        return f"progmem_string({FlashStringLiteral(value)})"
+    return str(cg.safe_exp(value))
+
+
+def literal_with_length(config: ConfigType, value: str) -> str:
+    """Renderer for a ``(const char *, size_t)`` target: a plain literal plus its byte length.
+
+    The target compares or copies the bytes in place, so it needs the RAM literal rather than
+    the PROGMEM rendering on ESP8266, and the length saves a strlen.
+    """
+    return f"{cg.safe_exp(value)}, {len(value.encode('utf-8'))}"
+
+
+@dataclass(frozen=True)
+class ApplyCall:
+    """One statement from config keys, e.g. ``"set_range({}, {})"`` with ``((CONF_LOW, cg.float_), ...)``.
+
+    Each arg is ``(conf_key, type_)`` or ``(conf_key, type_, const_fn)``. A ``conf_key`` may be a
+    path into nested sections. A plain ``str`` ``type_`` is raw C++ type text and may use
+    ``{parent}``. ``const_fn(config, value)`` renders a constant's argument text; a lambda or an
+    id bypasses it. The statement is skipped when none of its keys is set, always emitted when it
+    has no keys, and a partial set is a config error.
+    """
+
+    target: str
+    args: tuple[tuple[Any, ...], ...] = ()
+
+    def __post_init__(self) -> None:
+        fields = [
+            f for _, f, _, _ in string.Formatter().parse(self.target) if f is not None
+        ]
+        if any(fields):
+            raise ValueError(
+                f"apply target {self.target!r}: only bare {{}} placeholders"
+            )
+        if len(fields) != len(self.args):
+            raise ValueError(
+                f"apply target {self.target!r} has {len(fields)} "
+                f"placeholder(s) for {len(self.args)} config key(s)"
+            )
+        if any(len(arg) not in (2, 3) for arg in self.args):
+            raise ValueError(
+                f"apply target {self.target!r}: each arg is (conf_key, type_[, const_fn])"
+            )
+
+    @property
+    def members(self) -> list[tuple[Any, Any, Any]]:
+        """Each arg as ``(conf_key, type_, const_fn or None)``."""
+        return [
+            (arg[0], arg[1], arg[2] if len(arg) == 3 else None) for arg in self.args
+        ]
+
+
+@dataclass(frozen=True)
+class ApplyField:
+    """One config key forwarded as ``target(value)``, or as statement ``target`` when it has ``{}``.
+
+    Double a literal brace in a template. ``conf_key`` may be a path into nested sections.
+    ``type_`` may be a C++ type string using ``{parent}`` when the type is only known per
+    instance. ``const_fn(config, value)`` renders a constant's argument text when ``cg.safe_exp``
+    is not the right spelling (unit conversion belongs in the validator); a lambda or an id
+    bypasses it, so the target must also take a plain ``type_``. An absent key emits nothing.
+    """
+
+    conf_key: str | tuple[str, ...]
+    target: str
+    type_: SafeExpType
+    const_fn: Callable[[ConfigType, Any], str] | None = None
+
+    def call(self) -> ApplyCall:
+        target = self.target if "{}" in self.target else f"{self.target}({{}})"
+        return ApplyCall(target, ((self.conf_key, self.type_, self.const_fn),))
+
+
+def _config_lookup(config: ConfigType, key: str | tuple[str, ...]) -> Any:
+    if isinstance(key, str):
+        return config.get(key)
+    for part in key:
+        if (config := config.get(part)) is None:
+            return None
+    return config
+
+
+def _dict_schema(schema: Any) -> Any:
+    """The dict-backed cv.Schema inside cv.All and maybe_* wrappers, or None; cv.Any is not inspected."""
+    if isinstance(schema, dict):
+        return cv.Schema(schema)
+    if isinstance(getattr(schema, "schema", None), dict):
+        return schema
+    if isinstance(schema, cv.All):
+        inner = schema.validators
+    else:
+        inner = (
+            getattr(schema, "inner_schema", None),
+        )  # maybe_conf / maybe_simple_value
+    for candidate in inner:
+        if candidate is not None and (found := _dict_schema(candidate)) is not None:
+            return found
+    return None
+
+
+def _check_key_in_schema(
+    name: str, schema: Any, conf_key: str | tuple[str, ...]
+) -> None:
+    """Reject a key path the schema does not have; a typo would otherwise be a silent no-op.
+
+    Only dict-backed schemas, also inside cv.All and maybe_* wrappers, can be checked.
+    """
+    for part in (conf_key,) if isinstance(conf_key, str) else conf_key:
+        if (schema := _dict_schema(schema)) is None:
+            return
+        markers = {
+            getattr(marker, "schema", marker): marker for marker in schema.schema
+        }
+        if part not in markers:
+            raise ValueError(f"{name}: config key {part!r} is not in the schema")
+        schema = schema.schema[markers[part]]
+
+
+async def _apply_parent(config: ConfigType, id_key: str = CONF_ID) -> str:
+    # Global-scope qualified so a trigger arg named like the id cannot shadow it.
+    return f"::{await cg.get_variable(config[id_key])}"
+
+
+def _apply_lambda_args(args: TemplateArgsType) -> TemplateArgsType:
+    # The generated function's parameters; a std::string arg is never copied.
+    return [
+        (cg.RawExpression(f"const std::remove_cvref_t<{cg.safe_exp(t)}> &"), arg)
+        for t, arg in args
+    ]
+
+
+def _apply_function(
+    id_: ID,
+    return_type: SafeExpType,
+    template_arg: cg.TemplateArguments,
+    lambda_args: TemplateArgsType,
+    statements: list[str],
+) -> MockObj:
+    """Emit the generated function and declare ``id_`` as the ``ApplyAction`` or
+    ``ApplyCondition`` templated on it, so ``play()`` calls it directly."""
+    fn = cg.static_function(
+        f"esphome__{id_.id}__fn", return_type, lambda_args, statements
+    )
+    return cg.new_Pvariable(id_, cg.TemplateArguments(fn, *template_arg))
+
+
+async def _render_values(
+    name: str,
+    target: str,
+    members: list[tuple[Any, Any, Any]],
+    values: list[Any],
+    config: ConfigType,
+    parent: str,
+    lambda_args: TemplateArgsType,
+    compare: bool = False,
+) -> list[str]:
+    """Render the argument text of one statement; every key must be present.
+
+    ``compare``: an inlined lambda expression is parenthesized so it binds as a whole
+    beside an operator.
+    """
+    if any(value is None for value in values):
+        keys = [key for key, _, _ in members]
+        raise EsphomeError(f"{name}: {target!r} needs all of {keys}")
+    exprs: list[str] = []
+    for (_, type_, const_fn), value in zip(members, values, strict=True):
+        if isinstance(value, Lambda):
+            if isinstance(type_, str):
+                type_ = cg.RawExpression(type_.format(parent=parent))
+            inner = await cg.process_lambda(value, lambda_args, return_type=type_)
+            expr = call_lambda(inner)
+            bare = compare and isinstance(expr, cg.RawExpression)
+            exprs.append(f"({expr})" if bare else str(expr))
+        elif isinstance(value, ID):
+            # Qualified like the parent, so a trigger arg named like the id cannot shadow it.
+            exprs.append(f"::{await cg.get_variable(value)}")
+        elif const_fn is not None:
+            exprs.append(const_fn(config, value))
+        else:
+            exprs.append(str(cg.safe_exp(value)))
+    return exprs
+
+
+def _apply_values(config: ConfigType, members: list[tuple[Any, Any, Any]]) -> list[Any]:
+    return [_config_lookup(config, key) for key, _, _ in members]
+
+
+def register_apply_action(
+    name: str,
+    schema: cv.Schema,
+    *fields: ApplyField | ApplyCall,
+    call: str | None = None,
+    id_key: str = CONF_ID,
+) -> None:
+    """Register an action that only forwards config values to its parent, with no C++ class.
+
+    Generates one static function with the parent (read from ``id_key``) and constants baked
+    in, lambdas called inline with the trigger args, and an ``ApplyAction`` templated on it.
+    A constant that is an id (``cv.use_id`` under ``cv.templatable``) is the object it names.
+    With ``call`` every statement targets the call object ``auto apply_call = parent->call()``,
+    and ``apply_call.perform()`` is appended.
+    """
+    # An action stores the value, so a std::string constant stays in flash on ESP8266.
+    statements_spec = [
+        (
+            c.target,
+            [
+                (key, t, fn or (flash_string if t is cg.std_string else None))
+                for key, t, fn in c.members
+            ],
+        )
+        for c in (f if isinstance(f, ApplyCall) else f.call() for f in fields)
+    ]
+    _check_key_in_schema(name, schema, id_key)
+    for _, members in statements_spec:
+        for conf_key, _, _ in members:
+            _check_key_in_schema(name, schema, conf_key)
+
+    async def builder(
+        config: ConfigType,
+        action_id: ID,
+        template_arg: cg.TemplateArguments,
+        args: TemplateArgsType,
+    ) -> MockObj:
+        parent = await _apply_parent(config, id_key)
+        lambda_args = _apply_lambda_args(args)
+        receiver = "apply_call." if call else f"{parent}->"
+        statements: list[str] = []
+        for target, members in statements_spec:
+            values = _apply_values(config, members)
+            if members and all(value is None for value in values):
+                continue
+            exprs = await _render_values(
+                name, target, members, values, config, parent, lambda_args
+            )
+            statements.append(f"{receiver}{target.format(*exprs)};")
+        if call:
+            statements = [
+                f"auto apply_call = {parent}->{call}();",
+                *statements,
+                "apply_call.perform();",
+            ]
+        return _apply_function(
+            action_id, cg.void, template_arg, lambda_args, statements
+        )
+
+    register_action(name, ApplyAction, schema, synchronous=True)(builder)
+
+
+def register_apply_condition(
+    name: str, schema: cv.Schema, check: str | ApplyCall, id_key: str = CONF_ID
+) -> None:
+    """Register a condition that is one expression on its parent, with no C++ class.
+
+    ``check`` is applied to the parent: ``"is_playing()"`` becomes ``parent->is_playing()``; an
+    ``ApplyCall`` such as ``ApplyCall("state == {}", ((CONF_STATE, cg.bool_),))`` compares
+    against config values, all of which must be present. Write ``== false`` to negate.
+    String constants are plain literals, so compare a ``std::string`` or ``StringRef`` member.
+    Generates one static predicate and an ``ApplyCondition`` templated on it.
+    """
+    call = check if isinstance(check, ApplyCall) else ApplyCall(check)
+    members = call.members
+    _check_key_in_schema(name, schema, id_key)
+    for conf_key, _, _ in members:
+        _check_key_in_schema(name, schema, conf_key)
+
+    async def builder(
+        config: ConfigType,
+        condition_id: ID,
+        template_arg: cg.TemplateArguments,
+        args: TemplateArgsType,
+    ) -> MockObj:
+        parent = await _apply_parent(config, id_key)
+        lambda_args = _apply_lambda_args(args)
+        exprs = await _render_values(
+            name,
+            call.target,
+            members,
+            _apply_values(config, members),
+            config,
+            parent,
+            lambda_args,
+            compare=True,
+        )
+        return _apply_function(
+            condition_id,
+            cg.bool_,
+            template_arg,
+            lambda_args,
+            [f"return {parent}->{call.target.format(*exprs)};"],
+        )
+
+    register_condition(name, ApplyCondition, schema)(builder)
+
 
 def validate_potentially_and_condition(value):
     if isinstance(value, list):
@@ -359,28 +788,15 @@ async def for_condition_to_code(
     return var
 
 
-@register_condition(
+register_apply_condition(
     "component.is_idle",
-    LambdaCondition,
     maybe_simple_id(
         {
             cv.Required(CONF_ID): cv.use_id(cg.Component),
         }
     ),
+    "is_idle()",
 )
-async def component_is_idle_condition_to_code(
-    config: ConfigType,
-    condition_id: ID,
-    template_arg: cg.TemplateArguments,
-    args: TemplateArgsType,
-) -> MockObj:
-    comp = await cg.get_variable(config[CONF_ID])
-    lambda_ = await cg.process_lambda(
-        Lambda(f"return {comp}->is_idle();"), args, return_type=bool
-    )
-    return new_lambda_pvariable(
-        condition_id, lambda_, StatelessLambdaCondition, template_arg
-    )
 
 
 @register_action(
@@ -534,44 +950,20 @@ async def lambda_action_to_code(
     return new_lambda_pvariable(action_id, lambda_, StatelessLambdaAction, template_arg)
 
 
-@register_action(
+register_simple_action(
     "component.update",
     UpdateComponentAction,
-    maybe_simple_id(
-        {
-            cv.Required(CONF_ID): cv.use_id(cg.PollingComponent),
-        }
-    ),
+    maybe_simple_id({cv.Required(CONF_ID): cv.use_id(cg.PollingComponent)}),
     synchronous=True,
 )
-async def component_update_action_to_code(
-    config: ConfigType,
-    action_id: ID,
-    template_arg: cg.TemplateArguments,
-    args: TemplateArgsType,
-) -> MockObj:
-    comp = await cg.get_variable(config[CONF_ID])
-    return cg.new_Pvariable(action_id, template_arg, comp)
 
 
-@register_action(
+register_simple_action(
     "component.suspend",
     SuspendComponentAction,
-    maybe_simple_id(
-        {
-            cv.Required(CONF_ID): cv.use_id(cg.PollingComponent),
-        }
-    ),
+    maybe_simple_id({cv.Required(CONF_ID): cv.use_id(cg.PollingComponent)}),
     synchronous=True,
 )
-async def component_suspend_action_to_code(
-    config: ConfigType,
-    action_id: ID,
-    template_arg: cg.TemplateArguments,
-    args: TemplateArgsType,
-) -> MockObj:
-    comp = await cg.get_variable(config[CONF_ID])
-    return cg.new_Pvariable(action_id, template_arg, comp)
 
 
 @register_action(

esphome/build_gen/espidf.py

@@ -3,7 +3,10 @@
 import json
 import logging
 from pathlib import Path
+import re
+import textwrap
 
+from esphome.build_helpers import pch
 from esphome.components.esp32 import (
     get_esp32_variant,
     get_excluded_builtin_components,
@@ -18,7 +21,7 @@ from esphome.framework_helpers import (
     get_project_cxx_compile_flags,
     get_project_link_flags,
 )
-from esphome.helpers import mkdir_p, write_file_if_changed
+from esphome.helpers import get_bool_env, mkdir_p, write_file_if_changed
 
 _LOGGER = logging.getLogger(__name__)
 
@@ -33,6 +36,83 @@ list(FILTER esphome_cxx_compile_options EXCLUDE REGEX "^-std=")
 list(APPEND esphome_cxx_compile_options "-std={standard}")
 idf_build_set_property(CXX_COMPILE_OPTIONS "${{esphome_cxx_compile_options}}")"""
 
+# Drops the app archive from ldgen's inputs so app-only edits skip the
+# sections.ld regeneration. Safe: no mapping fragment references it
+# (run_compile re-checks each build). Filters only the top-level call;
+# the prior definition stays reachable with an underscore prefix.
+_LDGEN_OVERRIDE = """\
+if(COMMAND __ldgen_get_lib_deps_of_target)
+    set_property(GLOBAL PROPERTY ESPHOME_LDGEN_ARMED 1)
+    function(__ldgen_get_lib_deps_of_target target out_list_var)
+        if(NOT COMMAND ___ldgen_get_lib_deps_of_target)
+            message(FATAL_ERROR "ESPHome ldgen override lost the original "
+                "implementation; set ESPHOME_LDGEN_FULL_DEPS=1 and rebuild.")
+        endif()
+        ___ldgen_get_lib_deps_of_target(${target} ${out_list_var})
+        if(out_list_var STREQUAL "ldgen_libraries")
+            set_property(GLOBAL PROPERTY ESPHOME_LDGEN_FILTERED 1)
+            list(LENGTH ${out_list_var} esphome_ldgen_before)
+            list(REMOVE_ITEM ${out_list_var} idf::src __idf_src)
+            list(LENGTH ${out_list_var} esphome_ldgen_after)
+            if(esphome_ldgen_before EQUAL esphome_ldgen_after)
+                message(@SEVERITY@ "ESPHome ldgen app archive exclusion matched "
+                    "nothing; app edits will regenerate sections.ld.")
+            endif()
+        endif()
+        set(${out_list_var} "${${out_list_var}}" PARENT_SCOPE)
+    endfunction()
+else()
+    message(@MISSING@ "ESPHome ldgen override target not found; "
+        "app edits will regenerate sections.ld.")
+endif()"""
+
+# lwip sources that compile to empty objects with the option off (their own
+# #if guard). (option, regex valid for both Python and CMake); a source is
+# only dropped when its option is defined and off, so a renamed option
+# keeps it.
+LWIP_EMPTY_SOURCES: tuple[tuple[str, str], ...] = (
+    ("CONFIG_LWIP_PPP_SUPPORT", "/netif/ppp/"),
+    ("CONFIG_LWIP_IPV6", "/core/ipv6/"),
+    ("CONFIG_LWIP_AUTOIP", "/core/ipv4/autoip[.]c$"),
+    ("CONFIG_LWIP_STATS", "/core/stats[.]c$"),
+)
+# Drift guard only: keep every lwip source.
+LWIP_FULL_SOURCES_ENV = "ESPHOME_LWIP_FULL_SOURCES"
+
+# Drops the empty objects after project(), once the lwip target exists.
+_LWIP_EMPTY_SOURCES_FILTER = f"""\
+idf_build_get_property(esphome_build_components BUILD_COMPONENTS)
+if(lwip IN_LIST esphome_build_components AND NOT DEFINED ENV{{{LWIP_FULL_SOURCES_ENV}}})
+    idf_component_get_property(esphome_lwip_lib lwip COMPONENT_LIB)
+    get_target_property(esphome_lwip_srcs ${{esphome_lwip_lib}} SOURCES)
+@FILTERS@
+    set_property(TARGET ${{esphome_lwip_lib}} PROPERTY SOURCES ${{esphome_lwip_srcs}})
+endif()"""
+
+
+def lwip_empty_source_gate(option: str, regex: str) -> str:
+    return (
+        f"    if(DEFINED {option} AND NOT {option})\n"
+        f'        list(FILTER esphome_lwip_srcs EXCLUDE REGEX "{regex}")\n'
+        "    endif()"
+    )
+
+
+def _lwip_empty_sources_filter() -> str:
+    gates = "\n".join(lwip_empty_source_gate(*entry) for entry in LWIP_EMPTY_SOURCES)
+    return _LWIP_EMPTY_SOURCES_FILTER.replace("@FILTERS@", gates)
+
+
+# Runs after project() so the walk has happened; catches the remaining
+# silent path where the top-level out-var was renamed.
+_LDGEN_OVERRIDE_CHECK = """\
+get_property(esphome_ldgen_armed GLOBAL PROPERTY ESPHOME_LDGEN_ARMED)
+get_property(esphome_ldgen_filtered GLOBAL PROPERTY ESPHOME_LDGEN_FILTERED)
+if(esphome_ldgen_armed AND NOT esphome_ldgen_filtered)
+    message(@SEVERITY@ "ESPHome ldgen override never filtered the app "
+        "archive; app edits will regenerate sections.ld.")
+endif()"""
+
 
 def get_available_components() -> list[str] | None:
     """List the built-in ESP-IDF components from ``project_description.json``.
@@ -78,6 +158,85 @@ def _cmake_quote(value: str) -> str:
     return f'"{escaped}"'
 
 
+# CONFIG_APP_BUILD_BOOTLOADER is hidden and force-selected, so it can only be
+# cleared at the CMake level (the same state IDF's RAM-app build type uses).
+# The macro is IDF's __build_process_project_includes plus a few added lines;
+# the flag is ignored and the bootloader builds as usual if IDF changes it.
+IDF_BOOTLOADER_OVERRIDE = """\
+# ESPHome bootloader skip switch; see esphome/espidf/toolchain.py.
+if(ESPHOME_SKIP_BOOTLOADER)
+    macro(__build_process_project_includes)
+        idf_build_get_property(sdkconfig_cmake SDKCONFIG_CMAKE)
+        include(${sdkconfig_cmake})
+        set(CONFIG_APP_BUILD_BOOTLOADER "")
+        # bt's CMakeLists reads the lowercase idf_target that the (now
+        # skipped) bootloader project_include leaks; keep it defined, or
+        # its empty TARGET_SRC_NAME sends file(GLOB_RECURSE) across /.
+        idf_build_get_property(idf_target IDF_TARGET)
+        # partition_table's V1 ECDSA signing reads this key, which the
+        # skipped bootloader project_include also sets.
+        get_filename_component(SECURE_BOOT_SIGNING_KEY "${CONFIG_SECURE_BOOT_SIGNING_KEY}" ABSOLUTE BASE_DIR "${project_dir}")
+        idf_build_get_property(build_properties __BUILD_PROPERTIES)
+        foreach(build_property ${build_properties})
+            idf_build_get_property(val ${build_property})
+            set(${build_property} "${val}")
+        endforeach()
+        idf_build_get_property(build_component_targets __BUILD_COMPONENT_TARGETS)
+        foreach(component_target ${build_component_targets})
+            __component_get_property(dir ${component_target} COMPONENT_DIR)
+            __component_get_property(_name ${component_target} COMPONENT_NAME)
+            set(COMPONENT_NAME ${_name})
+            set(COMPONENT_DIR ${dir})
+            set(COMPONENT_PATH ${dir})
+            if(EXISTS ${COMPONENT_DIR}/project_include.cmake)
+                include(${COMPONENT_DIR}/project_include.cmake)
+            endif()
+        endforeach()
+    endmacro()
+endif()
+"""
+
+# The lines the override adds to IDF's macro; idf_macro_matches() below
+# strips them before comparing with the live macro.
+BOOTLOADER_OVERRIDE_ADDED_LINES = (
+    'set(CONFIG_APP_BUILD_BOOTLOADER "")',
+    "idf_build_get_property(idf_target IDF_TARGET)",
+    (
+        "get_filename_component(SECURE_BOOT_SIGNING_KEY"
+        ' "${CONFIG_SECURE_BOOT_SIGNING_KEY}" ABSOLUTE BASE_DIR "${project_dir}")'
+    ),
+)
+
+_MACRO = re.compile(
+    r"macro\(__build_process_project_includes\)(.*?)endmacro\(\)", re.DOTALL
+)
+
+
+def _normalized_macro(text: str) -> list[str] | None:
+    """The macro body as comment-free, whitespace-collapsed lines."""
+    if (match := _MACRO.search(text)) is None:
+        return None
+    return [
+        re.sub(r"\s+", " ", line)
+        for raw in match.group(1).splitlines()
+        if (line := raw.split("#", 1)[0].strip())
+    ]
+
+
+_EXPECTED_MACRO = [
+    line
+    for line in _normalized_macro(IDF_BOOTLOADER_OVERRIDE)
+    if line not in BOOTLOADER_OVERRIDE_ADDED_LINES
+]
+
+
+def idf_macro_matches(idf_path: Path) -> bool:
+    """Whether IDF's macro still matches the copy the override replays."""
+    build_cmake = idf_path / "tools" / "cmake" / "build.cmake"
+    live = _normalized_macro(build_cmake.read_text(encoding="utf-8"))
+    return live == _EXPECTED_MACRO
+
+
 def get_project_cmakelists(
     minimal: bool = False, builtin_components: list[str] | None = None
 ) -> str:
@@ -90,9 +249,10 @@ def get_project_cmakelists(
     """
     idf_target = variant_to_idf_target(get_esp32_variant())
 
-    # esp_idf_size 2.x (bundled with IDF >=6.0) made NG the default and
-    # removed the --ng flag; on 1.x (IDF 5.5) --ng is required to get
-    # --format=raw because the legacy mode doesn't support it.
+    # esp_idf_size 2.x (IDF >=6.0) made NG the default and removed --ng;
+    # 1.x (IDF 5.5) needs --ng for --format=json2. 1.x json2 also lacks
+    # total_size, hence the ELF fallback in espidf/size_summary.py; both
+    # go away together when 1.x support is dropped.
     size_ng_flag = "--ng" if idf_version() < cv.Version(6, 0, 0) else ""
 
     # Project-wide compile options: -D defines and -W warning flags (skip
@@ -122,6 +282,22 @@ def get_project_cmakelists(
         else ""
     )
 
+    # Stops the ~3s sections.ld regeneration on app-only edits; see
+    # _LDGEN_OVERRIDE. ESPHOME_LDGEN_FULL_DEPS=1 restores stock behavior;
+    # ESPHOME_LDGEN_STRICT=1 (CI) fails the configure when an IDF bump
+    # breaks the override instead of degrading to stock deps.
+    if get_bool_env("ESPHOME_LDGEN_FULL_DEPS"):
+        ldgen_override = ""
+        ldgen_override_check = ""
+    else:
+        strict = get_bool_env("ESPHOME_LDGEN_STRICT")
+        severity = "FATAL_ERROR" if strict else "WARNING"
+        missing = "FATAL_ERROR" if strict else "STATUS"
+        ldgen_override = _LDGEN_OVERRIDE.replace("@SEVERITY@", severity).replace(
+            "@MISSING@", missing
+        )
+        ldgen_override_check = _LDGEN_OVERRIDE_CHECK.replace("@SEVERITY@", severity)
+
     # CMake variables registered via cg.add_cmake_arg(). Emitted before
     # include(project.cmake) so values like EXCLUDE_COMPONENTS are already
     # set when project.cmake seeds the component list, and on minimal
@@ -199,6 +375,9 @@ set(EXTRA_COMPONENT_DIRS ${{CMAKE_SOURCE_DIR}}/src)
 
 include($ENV{{IDF_PATH}}/tools/cmake/project.cmake)
 
+{IDF_BOOTLOADER_OVERRIDE}
+{ldgen_override}
+
 {cpp_standard_options}
 
 {cxx_compile_options}
@@ -211,12 +390,22 @@ include($ENV{{IDF_PATH}}/tools/cmake/project.cmake)
 
 project({CORE.name})
 
-# Emit raw JSON size data for ESPHome to read post-build.
+{ldgen_override_check}
+
+{_lwip_empty_sources_filter()}
+
+# Emit per-memory-type JSON size data for ESPHome to read post-build.
+# json2 stays small; raw dumps every symbol (~2s on a large map) and
+# this command runs inside the link edge, blocking everything downstream.
+# The map is a BYPRODUCT so ninja knows the link writes it; IDF's size
+# target depends on the map and can then be built in the same run as all.
+# IDF's cmakev2 declares the map itself, so drop this line on that switch.
 add_custom_command(
     TARGET ${{CMAKE_PROJECT_NAME}}.elf POST_BUILD
-    COMMAND ${{PYTHON}} -m esp_idf_size {size_ng_flag} --format=raw
+    COMMAND ${{PYTHON}} -m esp_idf_size {size_ng_flag} --format=json2
             -o ${{CMAKE_BINARY_DIR}}/esp_idf_size.json
             ${{CMAKE_PROJECT_NAME}}.map
+    BYPRODUCTS ${{CMAKE_BINARY_DIR}}/${{CMAKE_PROJECT_NAME}}.map
     WORKING_DIRECTORY ${{CMAKE_BINARY_DIR}}
     VERBATIM
 )
@@ -279,9 +468,72 @@ idf_component_register(
 target_link_options(${{COMPONENT_LIB}} PUBLIC
     {link_opts_str}
 )
+{_pch_cmake_block()}"""
+
+
+# Where CMake puts the .gch of the src component; ccache reads the checksum
+# next to it in place of the .gch
+_PCH_SUM_PATH = "build/esp-idf/src/CMakeFiles/__idf_src.dir/cmake_pch.hxx.gch.sum"
+# Where the Windows gate records its choice
+_PCH_CHOICE_VAR = "ESPHOME_PCH"
+
+
+def _pch_cmake_block() -> str:
+    """The CMake block that precompiles the core headers for the C++ sources
+    of the src component; empty when disabled."""
+    if not pch.pch_enabled():
+        return ""
+    headers = "\n".join(
+        f'    "$<$<COMPILE_LANGUAGE:CXX>:${{CMAKE_CURRENT_SOURCE_DIR}}/{header}>"'
+        for header in pch.PCH_DEFAULT_HEADERS
+    )
+    block = f"""target_precompile_headers(${{COMPONENT_LIB}} PRIVATE
+{headers}
+)"""
+    if not pch.pch_needs_gcc_check():
+        return f"\n# ESPHome precompiled header\n{block}\n"
+    # Before the first configure only CMake knows the compiler version
+    return f"""
+# ESPHome precompiled header, unless GCC bug 14940 keeps it from loading
+if(CMAKE_CXX_COMPILER_ID STREQUAL "GNU" AND ({pch.PCH_WINDOWS_CMAKE_OLD_GCC}))
+  message(STATUS "ESPHome: GCC ${{CMAKE_CXX_COMPILER_VERSION}} cannot load a precompiled header on Windows; compiling without it")
+  set({_PCH_CHOICE_VAR} OFF CACHE BOOL "ESPHome precompiled header in use" FORCE)
+else()
+  set({_PCH_CHOICE_VAR} ON CACHE BOOL "ESPHome precompiled header in use" FORCE)
+{textwrap.indent(block, "  ")}
+endif()
 """
 
 
+def _read_if_exists(path: Path) -> str:
+    return path.read_text(encoding="utf-8") if path.is_file() else ""
+
+
+def write_pch_checksum() -> None:
+    """Write the checksum ccache uses in place of the .gch: the core headers,
+    the framework version, the sdkconfig and the managed component versions."""
+    from esphome.espidf.toolchain import get_cmake_cache_value
+
+    if not pch.pch_enabled():
+        return
+    # The gate's choice, cached by configure
+    if pch.pch_needs_gcc_check() and get_cmake_cache_value(_PCH_CHOICE_VAR) != "ON":
+        return
+    pch.log_pch_in_use()
+    checksum = pch.pch_checksum(
+        CORE.relative_src_path(),
+        pch.PCH_DEFAULT_HEADERS,
+        (
+            str(idf_version()),
+            _read_if_exists(CORE.relative_build_path(f"sdkconfig.{CORE.name}")),
+            _read_if_exists(CORE.relative_build_path("dependencies.lock")),
+        ),
+    )
+    path = CORE.relative_build_path(_PCH_SUM_PATH)
+    path.parent.mkdir(parents=True, exist_ok=True)
+    write_file_if_changed(path, checksum + "\n")
+
+
 def write_project(
     minimal: bool = False, builtin_components: list[str] | None = None
 ) -> None:

esphome/build_gen/host.py

@@ -0,0 +1,303 @@
+"""Native ninja build generator for the host platform.
+
+Emits ``build.ninja`` under ``.pioenvs/<name>/``: every source in the
+generated ``src/`` tree plus the resolved registry libraries compiles with
+the machine's compiler and links into ``program``, the name PlatformIO's
+native platform produced. Build flags route the way SCons's ``ParseFlags``
+did under PlatformIO: ``-D``/``-I``/``-std=``/``-W`` shapes reach the
+compile lines only, ``-l``/``-L``/``-Wl,`` the link line only, everything
+else both.
+"""
+
+from __future__ import annotations
+
+from collections.abc import Iterable
+import logging
+import os
+from pathlib import Path
+import subprocess
+import sys
+from typing import TYPE_CHECKING
+
+from esphome.build_helpers.ninja import escape as _e, quote_path as _q, shell_token
+from esphome.build_helpers.ninja_gen import (
+    PATH_ARG_FLAGS,
+    Flag,
+    anchor_path_flag,
+    ar_rule_lines,
+    collect_sources,
+    compile_edges,
+    compile_rule_lines,
+    library_edges,
+    pch_edges,
+    pch_rule_lines,
+    tool_lines,
+)
+from esphome.build_helpers.pch import PCH_DEFAULT_HEADERS, pch_enabled
+from esphome.core import CORE, EsphomeError
+from esphome.framework_helpers import get_project_cxx_compile_flags
+from esphome.helpers import mkdir_p, write_file_if_changed
+from esphome.host.toolchain import PROGRAM_NAME, HostCompilers, find_tool, get_build_dir
+from esphome.platformio.library import lex_build_flags
+
+if TYPE_CHECKING:
+    from esphome.arduino.library import ArduinoLibrary
+
+_LOGGER = logging.getLogger(__name__)
+
+# The PlatformIO platform the host built under; registry manifests declare
+# compatibility against it, as lib_compat_mode=strict checked before
+PIO_PLATFORM = "native"
+# Namespaces the shared library download cache (pio_components/host/)
+LIBRARY_CACHE_KEY = "host"
+
+# Flag shapes that only the compiler understands; dropped from the link line
+_COMPILE_ONLY_PREFIXES = ("-D", "-U", "-I", "-std=", "-W", *PATH_ARG_FLAGS)
+# Flag shapes that only the linker consumes; inert on a -c compile line
+_LINK_ONLY_PREFIXES = ("-l", "-L", "-Wl,")
+# Link-only flags whose argument is the next token
+_LINK_ONLY_ARG_FLAGS = ("-framework", "-Xlinker", "-z")
+
+
+def parse_flags(entries: Iterable[str], owner: str) -> list[Flag]:
+    """Lex build flag entries into flags, each with its argument.
+
+    Entries are a set, so their order is not the user's: a flag and the
+    argument it takes as the next token must share one entry.
+    """
+    flags: list[Flag] = []
+    for entry in entries:
+        it = iter(lex_build_flags(entry, owner))
+        for tok in it:
+            if tok not in PATH_ARG_FLAGS and tok not in _LINK_ONLY_ARG_FLAGS:
+                flags.append((tok,))
+                continue
+            arg = next(it, None)
+            # A path never starts with "-"; that is the next flag
+            if arg is None or (tok in PATH_ARG_FLAGS and arg.startswith("-")):
+                raise EsphomeError(
+                    f"{owner} build flags have '{tok}' with no argument; write "
+                    f"the flag and its argument as one entry"
+                )
+            flags.append((tok, arg))
+    return flags
+
+
+def split_flags(flags: list[Flag]) -> tuple[list[Flag], list[Flag]]:
+    """Route build flags to the compile and link lines."""
+    compile_flags: list[Flag] = []
+    link_flags: list[Flag] = []
+    for flag in flags:
+        name = flag[0]
+        if len(flag) > 1:
+            (compile_flags if name in PATH_ARG_FLAGS else link_flags).append(flag)
+        elif name.startswith(_LINK_ONLY_PREFIXES):
+            # Checked before the compile prefixes: -Wl, would match -W
+            link_flags.append(flag)
+        elif name.startswith(_COMPILE_ONLY_PREFIXES):
+            compile_flags.append(flag)
+        else:
+            # -g, -O, -f*, -m*, -pthread, --coverage: both lines, as SCons
+            compile_flags.append(flag)
+            link_flags.append(flag)
+    return compile_flags, link_flags
+
+
+def _is_std(flag: Flag) -> bool:
+    return flag[0].startswith("-std=")
+
+
+def _is_cxx_std(flag: Flag) -> bool:
+    return _is_std(flag) and "++" in flag[0]
+
+
+def _anchored_flags(entries: Iterable[str], owner: str) -> list[Flag]:
+    build_path = Path(CORE.build_path)
+    return [
+        anchor_path_flag(flag, build_path)
+        for flag in parse_flags(sorted(entries), owner)
+    ]
+
+
+def _flag_lists() -> tuple[list[str], list[str], list[str]]:
+    """The C, C++, and link flag lists (raw tokens), build_unflags applied.
+
+    ``cg.set_cpp_standard`` wins over any ``-std=`` in the build flags for
+    C++ compiles, as PlatformIO's unflag of every other standard did; C
+    compiles never see a C++ standard.
+    """
+    # The funnel warns and drops empty glued arguments (-D "") itself
+    compile_flags, link_flags = split_flags(
+        _anchored_flags(CORE.build_flags, "esphome")
+    )
+    cflags = [f for f in compile_flags if not _is_cxx_std(f)]
+    cxx_std = CORE.cpp_standard
+    cxxflags = [f for f in compile_flags if not (cxx_std and _is_std(f))]
+    if cxx_std:
+        cxxflags.insert(0, (f"-std={cxx_std}",))
+    cxxflags += [(tok,) for tok in get_project_cxx_compile_flags()]
+
+    # A flag is removed whole, with its argument, as PlatformIO did
+    unflags = set(_anchored_flags(CORE.build_unflags, "esphome build_unflags"))
+    # An unflag that hits nothing (a typo, or -DUSE_FOO against
+    # -DUSE_FOO=1) must be visible, since the user believes the flag is
+    # gone while it still drives the build
+    if unmatched := sorted(unflags - set(cflags) - set(cxxflags) - set(link_flags)):
+        _LOGGER.warning(
+            "build_unflags entries matched no build flag: %s",
+            ", ".join(" ".join(flag) for flag in unmatched),
+        )
+
+    def keep(flags: list[Flag]) -> list[str]:
+        return [tok for flag in flags if flag not in unflags for tok in flag]
+
+    return keep(cflags), keep(cxxflags), keep(link_flags)
+
+
+def _resolve_host_libraries() -> list[ArduinoLibrary]:
+    """Every ``cg.add_library()`` entry, fetched from the registry.
+
+    The host has no framework, so nothing is bundled and no framework
+    compatibility check applies; the platform check keeps the strict
+    manifest gate PlatformIO's native platform enforced. Manifest-less
+    libraries (a bare git checkout) build with PlatformIO's default
+    layout, as they did under its native platform.
+    """
+    if not CORE.platformio_libraries:
+        return []
+    from esphome.arduino.library import resolve_libraries
+
+    return resolve_libraries(
+        None,
+        pio_platform=PIO_PLATFORM,
+        board_mcu="host",
+        cache_key=LIBRARY_CACHE_KEY,
+        framework=None,
+        manifest_optional=True,
+    )
+
+
+def _file_macro_maps(build_dir: Path) -> list[str]:
+    """Flags that keep ``__FILE__`` relative to the build path.
+
+    PlatformIO compiled ``src/x.cpp`` from the build path, and tools name
+    things after that spelling (CodSpeed's benchmark ids). Here a source
+    reaches the compiler by its absolute path, or relative to the build
+    directory when ccache rewrites it.
+    """
+    build_path = Path(CORE.build_path)
+    prefixes = (build_path, Path(os.path.relpath(build_path, build_dir)))
+    return [
+        shell_token(f"-fmacro-prefix-map={prefix}{os.sep}=", force=True)
+        for prefix in prefixes
+    ]
+
+
+def _compiler_version(cxx: tuple[str, ...]) -> str:
+    """What the compiler says it is: its path can stay the same across an
+    update (the macOS shims in /usr/bin)."""
+    result = subprocess.run(
+        [*cxx, "--version"], capture_output=True, text=True, check=False
+    )
+    return result.stdout
+
+
+def write_project(compilers: HostCompilers, ccache: str | None) -> bool:
+    """Write the ninja build for the current configuration.
+
+    ``ccache`` is the caller's already-resolved binary (None when disabled).
+    Returns True when ``build.ninja`` changed, so the caller can skip work
+    derived purely from it (the compile database) on unchanged builds.
+    """
+    build_dir = get_build_dir()
+    mkdir_p(build_dir)
+    src_dir = CORE.relative_src_path()
+    if not src_dir.is_dir():
+        # Generated project state, not install state: clean-all would not help
+        raise EsphomeError(f"Generated source directory {src_dir} is missing")
+
+    cflags, cxxflags, link_flags = _flag_lists()
+    libraries = _resolve_host_libraries()
+
+    include_dirs = [src_dir]
+    for lib in libraries:
+        include_dirs += lib.include_dirs
+    includes = [f"-I{_q(d)}" for d in include_dirs]
+    includes += _file_macro_maps(build_dir)
+
+    # SCons's link line: $LINKFLAGS $SOURCES $_LIBDIRFLAGS $_LIBFLAGS, so
+    # -L and -l trail the objects while every other link token leads
+    lib_dirs = [Path(t[2:]) for t in link_flags if t.startswith("-L")]
+    libs = [t for t in link_flags if t.startswith("-l")]
+    linkflags = [shell_token(t) for t in link_flags if not t.startswith(("-L", "-l"))]
+    for lib in libraries:
+        lib_dirs += lib.link_dirs
+        libs += [f"-l{name}" for name in lib.link_libs]
+        linkflags += [shell_token(f) for f in lib.link_flags]
+
+    # PlatformIO's ASPPCOM passes only -D/-I user flags to assembly
+    asflags = [t for t in cflags if t.startswith(("-D", "-I"))]
+
+    lines = [
+        *tool_lines(compilers.cc, compilers.cxx, ccache),
+        *compile_rule_lines(),
+        *pch_rule_lines(),
+        "rule link",
+        "  command = $cxx -o $out $linkflags @$out.rsp $archives $libdirflags $libflags",
+        "  rspfile = $out.rsp",
+        "  rspfile_content = $in_newline",
+        "  description = LINK $out",
+    ]
+    if any(lib.sources and lib.lib_archive for lib in libraries):
+        # Resolved only when an archive is built, so a system without
+        # binutils still links a library-free configuration
+        lines += ar_rule_lines(find_tool("AR", ("ar",)))
+    lines += [
+        "",
+        f"cflags = {' '.join([*map(shell_token, cflags), *includes])}",
+        f"cxxflags = {' '.join([*map(shell_token, cxxflags), *includes])}",
+        f"asflags = {' '.join([*map(shell_token, asflags), *includes])}",
+        f"linkflags = {' '.join(linkflags)}",
+        f"libdirflags = {' '.join(f'-L{_q(d)}' for d in lib_dirs)}",
+        f"libflags = {' '.join(shell_token(lib) for lib in libs)}",
+        "",
+    ]
+
+    archives, direct_objs = library_edges(lines, libraries)
+
+    src_cxx_override = pch_edges(
+        lines,
+        build_dir,
+        src_dir,
+        PCH_DEFAULT_HEADERS,
+        # The arguments of a CXX override come before the flags
+        [*compilers.cxx[1:], *cxxflags],
+        (),
+        (compilers.cxx[0], _compiler_version(compilers.cxx)) if pch_enabled() else (),
+        compilers.cxx,
+    )
+    src_objs = compile_edges(
+        lines,
+        collect_sources(src_dir),
+        src_dir,
+        "src",
+        cxx_override=src_cxx_override,
+    )
+    if not src_objs:
+        raise EsphomeError(f"No source files found under {src_dir}")
+
+    # Archives are not topologically sorted; GNU ld needs the group to
+    # resolve references between them. ld64 loads archives iteratively and
+    # rejects the option, so macOS lists them bare.
+    archive_tokens = [shell_token(a) for a in archives]
+    if archive_tokens and sys.platform != "darwin":
+        archive_tokens = ["-Wl,--start-group", *archive_tokens, "-Wl,--end-group"]
+    lines.append(
+        f"build {PROGRAM_NAME}: link {' '.join(src_objs + direct_objs)} | "
+        f"{' '.join(_e(a) for a in archives)}"
+    )
+    lines.append(f"  archives = {' '.join(archive_tokens)}")
+    lines.append(f"default {PROGRAM_NAME}")
+    lines.append("")
+
+    return write_file_if_changed(build_dir / "build.ninja", "\n".join(lines))

esphome/build_gen/platformio.py

@@ -1,6 +1,8 @@
+from esphome.build_helpers.pch import pch_script_enabled
 from esphome.const import __version__
 from esphome.core import CORE
 from esphome.helpers import mkdir_p, read_file, write_file_if_changed
+from esphome.platformio.toolchain import copy_pch_script
 from esphome.writer import find_begin_end
 
 INI_AUTO_GENERATE_BEGIN = "; ========== AUTO GENERATED CODE BEGIN ==========="
@@ -62,6 +64,8 @@ def get_ini_content():
 
     # Add extra script for C++ flags
     CORE.add_platformio_option("extra_scripts", [f"pre:{CXX_FLAGS_FILE_NAME}"])
+    if pch_script_enabled():
+        CORE.add_platformio_option("extra_scripts", ["post:pch.py"])
 
     # Add CMake args. A user-supplied value (str or list) is deliberately
     # replaced; this option was always overwritten at FINAL priority.
@@ -106,6 +110,8 @@ def write_project():
 
     # Write extra script for C++ specific flags
     write_cxx_flags_script()
+    if pch_script_enabled():
+        copy_pch_script()
 
 
 CXX_FLAGS_FILE_NAME = "cxx_flags.py"

esphome/build_helpers/ccache.py

@@ -68,6 +68,15 @@ def resolve_ccache_path() -> str | None:
     return ccache
 
 
+def resolve_absolute_ccache_path() -> str | None:
+    """``resolve_ccache_path`` for the ninja backends, which run their
+    commands from the build directory, where a relative path is lost."""
+    from esphome.build_helpers.ninja import absolute_tool
+
+    ccache = resolve_ccache_path()
+    return absolute_tool(ccache) if ccache else None
+
+
 def ccache_defaults_env(cache_dir: Path) -> dict[str, str]:
     """Default ``CCACHE_*`` values for a build subprocess (not os.environ).
 
@@ -84,9 +93,39 @@ def ccache_defaults_env(cache_dir: Path) -> dict[str, str]:
             "CORE.build_path must be set before constructing the build environment"
         )
     defaults = {
-        "CCACHE_DIR": str(cache_dir),
+        # ccache expands $VAR in its settings; $$ is a literal $
+        "CCACHE_DIR": str(cache_dir).replace("$", "$$"),
         "CCACHE_NOHASHDIR": "true",
         "CCACHE_DEPEND": "1",
-        "CCACHE_BASEDIR": str(Path(CORE.build_path).resolve()),
+        "CCACHE_BASEDIR": str(Path(CORE.build_path).resolve()).replace("$", "$$"),
     }
     return {k: v for k, v in defaults.items() if k not in os.environ}
+
+
+def effective_ccache_basedir() -> str:
+    """The prefix ccache strips from hashed paths: a usable user
+    CCACHE_BASEDIR, else the resolved build path."""
+    from esphome.core import CORE
+
+    raw = os.environ.get("CCACHE_BASEDIR")
+    if raw is not None and Path(raw).is_absolute() and len(Path(raw).parts) > 1:
+        return raw
+    return str(Path(CORE.build_path).resolve())
+
+
+def ccache_env(ccache: str | None, tools_cache: tuple[str, str]) -> dict[str, str]:
+    """The ccache settings for a build subprocess (not os.environ).
+
+    ``ccache`` is the pre-resolved binary (resolve_ccache_path), or None when
+    disabled; ``tools_cache`` is the backend's tools cache spec, which holds
+    its ccache dir. The pch settings include ``time_macros``: a cached
+    object can keep an older ``__DATE__`` or ``__TIME__``.
+    """
+    if ccache is None:
+        return {}
+    from esphome.build_helpers.pch import ccache_pch_env
+    from esphome.build_helpers.tools_cache import tools_cache_path
+
+    env = ccache_defaults_env(tools_cache_path(*tools_cache) / "ccache")
+    env.update(ccache_pch_env())
+    return env

esphome/build_helpers/idedata.py

@@ -11,6 +11,7 @@ consumers (IDE integration, clang-tidy) expect:
 
 from __future__ import annotations
 
+from collections.abc import Callable
 import json
 import logging
 import os
@@ -21,6 +22,8 @@ import subprocess
 from esphome.core import EsphomeError
 from esphome.helpers import write_file
 
+_LOGGER = logging.getLogger(__name__)
+
 # Everything idedata generation may raise after a successful link; idedata
 # is a bonus artifact, so consumers warn instead of failing the build
 IDEDATA_BEST_EFFORT_ERRORS = (
@@ -31,7 +34,21 @@ IDEDATA_BEST_EFFORT_ERRORS = (
     ValueError,
 )
 
-_LOGGER = logging.getLogger(__name__)
+
+def warn_if_idedata_missing(get_idedata: Callable[[], dict | None]) -> None:
+    """Run an idedata generator, downgrading any failure to a warning:
+    the firmware already built."""
+    try:
+        if get_idedata() is None:
+            _LOGGER.warning("No idedata was generated for this build")
+    except IDEDATA_BEST_EFFORT_ERRORS as err:
+        _LOGGER.warning(
+            "Could not generate idedata: %s (IDE, clang-tidy, and "
+            "memory-analysis data will be unavailable for this build)",
+            err,
+        )
+        _LOGGER.debug("Idedata failure detail", exc_info=True)
+
 
 # C++ translation-unit suffixes used to identify ESPHome source files.
 _CXX_SUFFIXES = (".cpp", ".cc")
@@ -135,6 +152,15 @@ def _is_launcher(token: str) -> bool:
     return Path(token).stem.lower() in _LAUNCHER_STEMS
 
 
+def is_joined_include(tok: str) -> bool:
+    """The joined ``-includefoo.h`` spelling; excludes clang's -include-pch."""
+    return (
+        tok.startswith("-include")
+        and tok != "-include"
+        and not tok.startswith("-include-")
+    )
+
+
 def parse_entry(
     entry: dict, launcher: str | None = None
 ) -> tuple[str, list[str], list[str], list[str]]:

esphome/build_helpers/native.py

@@ -0,0 +1,74 @@
+"""Resolution of the native (non-PlatformIO) build backend for a config.
+
+Kept deliberately light: the serial upload and logs fast path resolves a
+backend for its artifact paths alone, so importing this must not pull in a
+platform component package or the backend itself.
+"""
+
+from __future__ import annotations
+
+from collections.abc import Collection
+import importlib
+import logging
+from types import ModuleType
+
+from esphome.const import Toolchain
+from esphome.core import CORE, EsphomeError
+
+_LOGGER = logging.getLogger(__name__)
+
+# Native build backend per (target platform, toolchain)
+NATIVE_TOOLCHAIN_MODULES = {
+    ("esp32", Toolchain.ESP_IDF): "esphome.espidf.toolchain",
+    ("esp8266", Toolchain.ARDUINO): "esphome.arduino8266.toolchain",
+    ("host", Toolchain.HOST): "esphome.host.toolchain",
+}
+
+
+def native_backend() -> ModuleType | None:
+    """The native build backend module for the resolved toolchain."""
+    if not CORE.using_native_toolchain:
+        return None
+    key = (CORE.target_platform, CORE.toolchain)
+    if (module_path := NATIVE_TOOLCHAIN_MODULES.get(key)) is None:
+        # Degrading to the PlatformIO path would build with the wrong backend
+        raise EsphomeError(
+            f"Toolchain '{CORE.toolchain.value}' has no native build backend "
+            f"module for platform {CORE.target_platform}"
+        )
+    return importlib.import_module(module_path)
+
+
+# Binutils and the linked image for memory analysis, for toolchains that build
+# without PlatformIO but have no native build backend (which supplies them)
+ANALYSIS_TOOLCHAIN_MODULES = {
+    ("nrf52", Toolchain.SDK_NRF): "esphome.components.nrf52.toolchain",
+}
+
+
+def analysis_backend() -> ModuleType | None:
+    """The module giving objdump, readelf and the ELF of a non-PlatformIO build.
+
+    None means PlatformIO's idedata supplies them (or nothing can).
+    """
+    if (native := native_backend()) is not None:
+        return native
+    module_path = ANALYSIS_TOOLCHAIN_MODULES.get((CORE.target_platform, CORE.toolchain))
+    return importlib.import_module(module_path) if module_path else None
+
+
+def warn_ignored_platformio_options(consumed: Collection[str]) -> None:
+    """Warn for component-added platformio options a native build drops.
+
+    User-supplied keys were already routed or warned about by
+    ``core/config.py``; what survives into ``CORE.platformio_options`` came
+    from ``cg.add_platformio_option`` calls in components.
+    """
+    for key in sorted(CORE.platformio_options or {}):
+        if key not in consumed:
+            _LOGGER.warning(
+                "platformio_options->%s is ignored when building with the "
+                "native '%s' toolchain",
+                key,
+                CORE.toolchain.value,
+            )

esphome/build_helpers/ninja.py

@@ -2,14 +2,17 @@
 
 from __future__ import annotations
 
+import json
 import logging
 import os
 from pathlib import Path
 import re
 import shutil
+import subprocess
 
 from esphome.core import EsphomeError
 from esphome.framework_helpers import strip_win_long_path_prefix, tool_version_runs
+from esphome.helpers import write_file_if_changed
 
 _LOGGER = logging.getLogger(__name__)
 
@@ -23,11 +26,26 @@ def _ninja_runs(binary: str) -> bool:
     )
 
 
+# Compile rule names the generators emit; ninja's compdb tool is asked for
+# exactly these, so a renamed rule fails the build instead of stranding idedata
+COMPILE_RULES = ("c", "cxx", "aspp", "asm")
+
+
+def absolute_tool(tool: str | Path) -> str:
+    """A tool path that still resolves from the build directory.
+
+    ``shutil.which`` returns a relative path for a relative PATH entry, and
+    ninja runs the commands from ``.pioenvs/<name>``. Symlinks are kept:
+    ccache's compiler links depend on the name they are called by.
+    """
+    return strip_win_long_path_prefix(str(Path(tool).absolute()))
+
+
 def find_ninja() -> Path:
     """Locate the ninja binary: a runnable PATH hit first, else the ninja
     PyPI wheel."""
     if binary := shutil.which("ninja"):
-        binary = strip_win_long_path_prefix(binary)
+        binary = absolute_tool(binary)
         if _ninja_runs(binary):
             return Path(binary)
     import_error: ImportError | None = None
@@ -49,8 +67,18 @@ def find_ninja() -> Path:
 
 
 def escape(value: Path | str) -> str:
-    """Escape a path or token for a ninja file."""
-    return str(value).replace("$", "$$").replace(":", "$:").replace(" ", "$ ")
+    """Escape a path or token for a ninja file.
+
+    ninja has no escape for ``|`` or a line break in a path, so those fail
+    here by name instead of producing a build file ninja misreads.
+    """
+    text = str(value)
+    if bad := next((c for c in "|\r\n" if c in text), None):
+        raise EsphomeError(
+            f"Path {text!r} contains {bad!r}, which a ninja build file cannot "
+            "express; rename the file or directory"
+        )
+    return text.replace("$", "$$").replace(":", "$:").replace(" ", "$ ")
 
 
 def quote_arg(tok: str) -> str:
@@ -90,3 +118,65 @@ def shell_token(tok: str, force: bool = False) -> str:
 def quote_path(value: Path | str) -> str:
     """Force-quote a path for the ninja command line (shell/CreateProcess)."""
     return shell_token(str(value), force=True)
+
+
+def refresh_compile_commands(
+    ninja_path: Path, build_dir: Path, env: dict[str, str], ninja_changed: bool
+) -> None:
+    """Regenerate the compile DB (a pure function of build.ninja) when stale.
+
+    Freshness rides a stamp: the DB itself is written through
+    write_file_if_changed (its mtime feeds the idedata cache), so a
+    regeneration with identical content would stay "stale" forever. An
+    interrupted previous run may have rewritten the manifest without
+    regenerating the DB, hence the mtime comparison.
+    """
+    compdb = build_dir / "compile_commands.json"
+    compdb_stamp = build_dir / ".compile_commands.stamp"
+    ninja_file = build_dir / "build.ninja"
+    if (
+        ninja_changed
+        or not compdb.is_file()
+        or not compdb_stamp.is_file()
+        or compdb_stamp.stat().st_mtime < ninja_file.stat().st_mtime
+    ):
+        write_compile_commands(ninja_path, build_dir, env)
+        compdb_stamp.touch()
+
+
+def write_compile_commands(
+    ninja_path: Path, build_dir: Path, env: dict[str, str]
+) -> None:
+    compdb = build_dir / "compile_commands.json"
+    result = subprocess.run(
+        [str(ninja_path), "-C", str(build_dir), "-t", "compdb", *COMPILE_RULES],
+        env=env,
+        capture_output=True,
+        text=True,
+        check=False,
+        close_fds=False,
+    )
+    if result.returncode != 0:
+        # Drop any stale database so consumers (IDE integration, clang-tidy,
+        # the memory analyzer) can't silently read outdated data
+        compdb.unlink(missing_ok=True)
+        raise EsphomeError(f"Could not generate compile_commands.json: {result.stderr}")
+    try:
+        entries = json.loads(result.stdout)
+    except ValueError as err:
+        compdb.unlink(missing_ok=True)
+        raise EsphomeError(
+            f"ninja produced an unparsable compile database: {err} "
+            f"(output starts {result.stdout[:120]!r})"
+        ) from err
+    if not entries:
+        # compdb exits 0 with [] for unknown rule names; a renamed compile
+        # rule must fail the build, not silently strand every consumer
+        compdb.unlink(missing_ok=True)
+        raise EsphomeError(
+            "ninja produced an empty compile database; the generator's rule "
+            "names no longer match"
+        )
+    # write_file_if_changed keeps the mtime stable on no-op builds so the
+    # idedata cache stays valid
+    write_file_if_changed(compdb, result.stdout)

esphome/build_helpers/ninja_gen.py

@@ -0,0 +1,281 @@
+"""The parts of a ``build.ninja`` every native backend emits the same way.
+
+Rule names match ``SOURCE_KIND_FOR_SUFFIX`` values (c, cxx, asm, aspp) and
+``ninja.COMPILE_RULES``, which the compile database is asked for.
+"""
+
+from __future__ import annotations
+
+from collections.abc import Collection, Sequence
+import logging
+import os
+from pathlib import Path
+import sys
+from typing import TYPE_CHECKING
+
+from esphome.build_helpers.idedata import is_joined_include
+from esphome.build_helpers.ninja import (
+    escape as _e,
+    quote_path as _q,
+    shell_token as _shell_token,
+)
+from esphome.build_helpers.pch import (
+    PCH_GCH_NAME,
+    PCH_SUM_NAME,
+    log_pch_in_use,
+    pch_consumer_flags,
+    pch_identity,
+    pch_usable,
+    write_pch_headers,
+)
+from esphome.framework_helpers import strip_win_long_path_prefix
+from esphome.helpers import write_file_if_changed
+from esphome.platformio.library import SOURCE_KIND_FOR_SUFFIX
+
+if TYPE_CHECKING:
+    from esphome.arduino.library import ArduinoLibrary
+
+_LOGGER = logging.getLogger(__name__)
+
+_BUILD_TOOL = Path(__file__).parent.parent / "build_gen" / "build_tool.py"
+
+
+def collect_sources(root: Path, exclude: Collection[str] = frozenset()) -> list[Path]:
+    return sorted(
+        p
+        for p in root.rglob("*")
+        if p.suffix in SOURCE_KIND_FOR_SUFFIX and p.name not in exclude
+    )
+
+
+def common_parent(paths: list[Path]) -> Path:
+    return Path(os.path.commonpath([str(p.parent) for p in paths]))
+
+
+def _command(words: Sequence[Path | str]) -> str:
+    program, *args = words
+    return " ".join([_q(program), *(_shell_token(str(a)) for a in args)])
+
+
+def tool_lines(
+    cc: Sequence[Path | str], cxx: Sequence[Path | str], ccache: str | None
+) -> list[str]:
+    """The file header: the compilers and the helper tools as variables.
+
+    A compiler is its program followed by any arguments it always takes.
+    """
+    return [
+        "# Auto-generated by ESPHome",
+        "ninja_required_version = 1.5",
+        f"cc = {_command(cc)}",
+        f"cxx = {_command(cxx)}",
+        # The NSIS launcher starts Python with a \\?\ extended-length path
+        # that cmd.exe cannot spawn; same strip every other emitted binary
+        # path gets
+        f"python = {_q(strip_win_long_path_prefix(sys.executable))}",
+        f"buildtool = {_q(_BUILD_TOOL)}",
+        f"ccache = {_q(ccache) if ccache else ''}",
+        "",
+    ]
+
+
+def compile_rule_lines() -> list[str]:
+    """The compile rules; ``$own_includes`` is empty unless an edge sets it."""
+    return [
+        "rule c",
+        "  command = $ccache $cc -MMD -MF $out.d $own_includes $cflags $flags -c $in -o $out",
+        "  depfile = $out.d",
+        "  deps = gcc",
+        "  description = CC $out",
+        "rule cxx",
+        "  command = $ccache $cxx -MMD -MF $out.d $own_includes $cxxflags $flags -c $in -o $out",
+        "  depfile = $out.d",
+        "  deps = gcc",
+        "  description = CXX $out",
+        "rule aspp",
+        "  command = $ccache $cc -MMD -MF $out.d -x assembler-with-cpp $own_includes $asflags $flags -c $in -o $out",
+        "  depfile = $out.d",
+        "  deps = gcc",
+        "  description = AS $out",
+        # Plain assembler, as SCons's ASCOM: no preprocessor, so no
+        # depfile and no $flags (defines/includes) either
+        "rule asm",
+        "  command = $ccache $cc -x assembler $asflags -c $in -o $out",
+        "  description = AS $out",
+    ]
+
+
+def pch_rule_lines() -> list[str]:
+    """The precompiled header rule, for a generator that emits one."""
+    return [
+        # No $ccache: the .gch embeds build dir paths
+        "rule pch",
+        "  command = $cxx -MMD -MF $out.d -x c++-header $cxxflags $flags -c $in -o $out",
+        "  depfile = $out.d",
+        "  deps = gcc",
+        "  description = PCH $out",
+    ]
+
+
+def pch_edges(
+    lines: list[str],
+    build_dir: Path,
+    src_dir: Path,
+    headers: Sequence[str],
+    cxxflags: Sequence[str],
+    src_flags: Sequence[str],
+    identity: Sequence[str],
+    cxx: Sequence[Path | str],
+) -> tuple[str, str] | None:
+    """Emit the precompiled header for the C++ src edges.
+
+    ``headers`` are folded into one prefix header, ``src_flags`` are the
+    flags every src edge carries, ``identity`` names what else the compile
+    depends on, the compiler included, and ``cxx`` is what the host rule
+    asks. Returns the ``cxx_override`` for ``compile_edges``, or None
+    without a pch.
+    """
+    if not pch_usable(cxx):
+        return None
+    if any(
+        tok == "-include" or tok.startswith("--include") or is_joined_include(tok)
+        for tok in cxxflags
+    ):
+        # $cxxflags expands first and GCC only loads a .gch for the first
+        # -include
+        _LOGGER.warning(
+            "A -include in the compiler flags prevents the precompiled header from "
+            "loading; compiling without it"
+        )
+        return None
+    log_pch_in_use()
+    source = write_pch_headers(build_dir, headers)
+    write_file_if_changed(
+        build_dir / PCH_SUM_NAME,
+        pch_identity([*cxxflags, *src_flags], src_dir, tuple(headers), identity) + "\n",
+    )
+    gch = _e(PCH_GCH_NAME)
+    # The checksum file changes with anything the .gch depends on
+    lines.append(f"build {gch}: pch {_e(source)} | {_e(PCH_SUM_NAME)}")
+    if src_flags:
+        lines.append(f"  flags = {' '.join(src_flags)}")
+    lines.append(f"srccxxflags = {' '.join([*src_flags, *pch_consumer_flags()])}")
+    return ("$srccxxflags", gch)
+
+
+def ar_rule_lines(ar: Path | str) -> list[str]:
+    return [
+        "rule ar",
+        f"  command = $python $buildtool ar {_q(ar)} $out $out.rsp",
+        "  rspfile = $out.rsp",
+        "  rspfile_content = $in_newline",
+        "  description = AR $out",
+    ]
+
+
+def compile_edges(
+    lines: list[str],
+    sources: list[Path],
+    root: Path,
+    group: str,
+    flags: str = "",
+    own_includes: str = "",
+    cxx_override: tuple[str, str] | None = None,
+) -> list[str]:
+    """Emit compile edges for ``sources``; return the object paths.
+
+    ``cxx_override`` is a (flags, implicit-dep) pair applied to C++ edges
+    only, replacing ``flags`` (used for the precompiled header).
+    """
+    objects = []
+    for src in sources:
+        rel = src.relative_to(root).as_posix()
+        obj = f"obj/{group}/{rel}.o"
+        escaped_obj = _e(obj)
+        kind = SOURCE_KIND_FOR_SUFFIX[src.suffix]
+        override = cxx_override if kind == "cxx" else None
+        implicit = f" | {override[1]}" if override else ""
+        lines.append(f"build {escaped_obj}: {kind} {_e(src)}{implicit}")
+        if own_includes:
+            lines.append(f"  own_includes = {own_includes}")
+        if edge_flags := override[0] if override else flags:
+            lines.append(f"  flags = {edge_flags}")
+        # Escaped once here: the returned paths only ever appear in build
+        # statements (archive/link inputs), which use ninja escaping
+        objects.append(escaped_obj)
+    return objects
+
+
+def library_edges(
+    lines: list[str], libraries: list[ArduinoLibrary], extra_flags: str = ""
+) -> tuple[list[str], list[str]]:
+    """Emit every library's compile and archive edges.
+
+    ``extra_flags`` (raw ninja text) precede each library's own flags.
+
+    Returns the archive names and the objects that link directly. A
+    library's own include dirs lead its compile lines, as PlatformIO searched
+    them first: the include path is one global list, so another library's
+    header of the same name would shadow them.
+    """
+    archives: list[str] = []
+    direct_objs: list[str] = []
+    for lib in libraries:
+        if not lib.sources:
+            # Header-only libraries are legitimate; the log makes an empty
+            # srcFilter or broken tree traceable before link errors do
+            _LOGGER.debug(
+                "Library %s has no source files; contributing includes only",
+                lib.name,
+            )
+            continue
+        objs = compile_edges(
+            lines,
+            lib.sources,
+            common_parent(lib.sources),
+            f"lib/{lib.name}",
+            flags=" ".join(filter(None, [extra_flags, *map(_shell_token, lib.flags)])),
+            own_includes=" ".join(f"-I{_q(d)}" for d in lib.include_dirs),
+        )
+        if not lib.lib_archive:
+            # libArchive: false / dot_a_linkage=false: hand the objects to
+            # the linker directly so unreferenced-but-required symbols
+            # (exception handlers, weak overrides) survive
+            direct_objs.extend(objs)
+            continue
+        archive = f"lib{lib.name}.a"
+        lines.append(f"build {_e(archive)}: ar {' '.join(objs)}")
+        archives.append(archive)
+    return archives, direct_objs
+
+
+# One build flag: the flag, plus its argument when that is a separate token
+Flag = tuple[str, ...]
+
+# Flags whose path operand is the next token; gcc also takes it glued on
+PATH_ARG_FLAGS = ("-include", "-imacros", "-isystem", "-iquote", "-idirafter")
+# Flags whose path operand is glued on
+PATH_PREFIXES = ("-I", "-L", *PATH_ARG_FLAGS)
+
+
+def _anchor(path: str, base: Path) -> str:
+    if not path or Path(path).is_absolute():
+        return path
+    return str(base / path)
+
+
+def anchor_path_flag(flag: Flag, base: Path) -> Flag:
+    """Anchor a flag's relative path operand at ``base``.
+
+    PlatformIO ran the compiler from the build path; ninja runs it from
+    ``.pioenvs/<name>``, where a relative operand would point elsewhere.
+    """
+    name, *args = flag
+    if args:
+        if name in PATH_ARG_FLAGS:
+            return (name, _anchor(args[0], base))
+        return flag
+    for prefix in PATH_PREFIXES:
+        if name.startswith(prefix):
+            return (prefix + _anchor(name[len(prefix) :], base),)
+    return flag

esphome/build_helpers/pch.py

@@ -0,0 +1,248 @@
+"""Shared precompiled header policy for the build backends."""
+
+from __future__ import annotations
+
+from collections.abc import Iterable, Sequence
+import hashlib
+import logging
+import os
+from pathlib import Path
+import posixpath
+import re
+import subprocess
+import sys
+
+from esphome.build_helpers.ccache import effective_ccache_basedir, parse_enable_env
+from esphome.const import PLATFORM_NRF52
+from esphome.helpers import write_file_if_changed
+
+_LOGGER = logging.getLogger(__name__)
+
+# The header and its sidecars live in the build directory
+PCH_HEADER_NAME = "esphome_pch.h"
+PCH_GCH_NAME = f"{PCH_HEADER_NAME}.gch"
+# ccache hashes this instead of the .gch; also the freshness stamp
+PCH_SUM_NAME = f"{PCH_GCH_NAME}.sum"
+# The include list the .gch is compiled from
+PCH_SOURCE_NAME = "esphome_pch_src.h"
+
+# GCC can skip a .gch without a diagnostic and read the header of the same
+# name, so that header is an error. Other tools get the include list.
+PCH_GUARD_TEXT = f"""\
+#if defined(__GNUC__) && !defined(__clang__) && !defined(__INTELLISENSE__)
+#error "The precompiled header was not loaded"
+#else
+#include "{PCH_SOURCE_NAME}"
+#endif
+"""
+
+# The cc1plus wrapper the PlatformIO script writes on arm64 macOS
+PCH_CC1_DIR = "pch_cc1"
+
+# What the PlatformIO script leaves in the project root, for cleanup
+PCH_ARTIFACT_NAMES = (PCH_HEADER_NAME, PCH_GCH_NAME, PCH_SUM_NAME, PCH_SOURCE_NAME)
+PCH_ARTIFACT_DIRS = (PCH_CC1_DIR,)
+
+# The core headers every backend precompiles
+PCH_DEFAULT_HEADERS = ("esphome/core/pch_prefix.h",)
+
+# PlatformIO platforms that do not take the pch script
+PCH_SCRIPT_EXCLUDED_PLATFORMS = frozenset(
+    {
+        PLATFORM_NRF52,
+    }
+)
+
+# What ccache needs to cache compiles that load a .gch
+_CCACHE_PCH_SLOPPINESS = ("pch_defines", "time_macros")
+
+# Both include forms: an angle include resolving under src/ enters the digest
+_INCLUDE_RE = re.compile(rb'^\s*#\s*include\s+["<]([^">]+)[">]', re.MULTILINE)
+
+
+def pch_enabled() -> bool:
+    """Precompiled-header knob: default on, ``ESPHOME_PCH_ENABLE=0`` opts out."""
+    return parse_enable_env("ESPHOME_PCH_ENABLE") is not False
+
+
+def pch_forced() -> bool:
+    """``ESPHOME_PCH_ENABLE=1``: wanted even where the host rule says no."""
+    return parse_enable_env("ESPHOME_PCH_ENABLE") is True
+
+
+# GCC bug 14940: before these releases the Windows loader maps a .gch only
+# at its saved address. First fixed release per major, 16 on always fixed;
+# PCH_WINDOWS_CMAKE_OLD_GCC and the pch_usable message spell the same table
+PCH_WINDOWS_GCC_FIXED = {14: (14, 4), 15: (15, 3)}
+PCH_WINDOWS_GCC_FIXED_DEFAULT = (16, 0)
+# The same rule for CMake, which alone knows the version before configure
+PCH_WINDOWS_CMAKE_OLD_GCC = (
+    "CMAKE_CXX_COMPILER_VERSION VERSION_LESS 14.4 OR "
+    "(CMAKE_CXX_COMPILER_VERSION VERSION_GREATER_EQUAL 15 AND "
+    "CMAKE_CXX_COMPILER_VERSION VERSION_LESS 15.3)"
+)
+
+
+def gcc_relocates_pch_on_windows(version: Sequence[int]) -> bool:
+    """Whether a GCC of this version loads a .gch on Windows."""
+    if not version:
+        return False
+    fixed = PCH_WINDOWS_GCC_FIXED.get(version[0], PCH_WINDOWS_GCC_FIXED_DEFAULT)
+    return tuple(version[:2]) >= fixed
+
+
+# GCC ends the first --version line with its version; clang names itself
+_VERSION_RE = re.compile(r"\d+(?:\.\d+)+")
+
+
+def gcc_version(cxx: Sequence[Path | str]) -> tuple[int, ...] | None:
+    """The GCC version from ``--version``: () when it cannot be read, None
+    for a compiler that is not GCC."""
+    try:
+        result = subprocess.run(
+            [*cxx, "--version"], capture_output=True, text=True, check=False
+        )
+    except OSError as err:
+        _LOGGER.debug("Cannot run %s: %s", cxx[0], err)
+        return ()
+    banner = result.stdout.partition("\n")[0]
+    if "clang" in banner.lower():
+        return None
+    found = _VERSION_RE.findall(banner)
+    return tuple(int(part) for part in found[-1].split(".")) if found else ()
+
+
+def pch_needs_gcc_check() -> bool:
+    """Windows host with the knob unset: the compiler version decides."""
+    return sys.platform == "win32" and parse_enable_env("ESPHOME_PCH_ENABLE") is None
+
+
+def pch_usable(cxx: Sequence[Path | str]) -> bool:
+    """The knob plus the host rule; ``ESPHOME_PCH_ENABLE=1`` skips the rule."""
+    if not pch_enabled():
+        return False
+    if not pch_needs_gcc_check():
+        return True
+    version = gcc_version(cxx)
+    if version is None or gcc_relocates_pch_on_windows(version):
+        return True
+    _LOGGER.info(
+        "GCC %s cannot load a precompiled header on Windows (GCC bug 14940, "
+        "fixed in 14.4, 15.3 and 16); compiling without it "
+        "(set ESPHOME_PCH_ENABLE=1 to force)",
+        ".".join(map(str, version)) or "of unknown version",
+    )
+    return False
+
+
+def pch_consumer_flags() -> list[str]:
+    """Flags a C++ src compile loads the pch with. The -include stays
+    relative: an absolute path would enter the ccache key."""
+    return ["-Winvalid-pch", "-Werror=invalid-pch", "-include", PCH_HEADER_NAME]
+
+
+def ccache_pch_env() -> dict[str, str]:
+    """What ccache needs to cache compiles that load a .gch, added to what
+    the user already set."""
+    if not pch_enabled():
+        return {}
+    sloppiness = [
+        item.strip()
+        for item in os.environ.get("CCACHE_SLOPPINESS", "").split(",")
+        if item.strip()
+    ]
+    sloppiness += [item for item in _CCACHE_PCH_SLOPPINESS if item not in sloppiness]
+    env = {"CCACHE_SLOPPINESS": ",".join(sloppiness)}
+    if "CCACHE_PCH_EXTSUM" not in os.environ:
+        env["CCACHE_PCH_EXTSUM"] = "true"
+    return env
+
+
+def pch_script_enabled() -> bool:
+    """Whether this PlatformIO build takes the pch script."""
+    from esphome.core import CORE
+
+    return pch_enabled() and CORE.target_platform not in PCH_SCRIPT_EXCLUDED_PLATFORMS
+
+
+def pch_header_text(include_headers: Iterable[str]) -> str:
+    """The prefix-header source: exactly these includes, in order."""
+    return "".join(f'#include "{name}"\n' for name in include_headers)
+
+
+def write_pch_headers(build_dir: Path, include_headers: Iterable[str]) -> Path:
+    """Write the guard header and the include list; return the latter,
+    which is what the .gch compiles from."""
+    write_file_if_changed(build_dir / PCH_HEADER_NAME, PCH_GUARD_TEXT)
+    source = build_dir / PCH_SOURCE_NAME
+    write_file_if_changed(source, pch_header_text(include_headers))
+    return source
+
+
+def _include_closure(src_dir: Path, roots: Iterable[str]) -> dict[str, bytes]:
+    """Include closure of ``roots``: src-relative name -> contents.
+
+    Resolution mirrors the compiler (includer's dir, then src root). No
+    #ifdef evaluation: including too much is the safe direction. Headers
+    outside ``src_dir`` are covered by the version strings of the caller.
+    """
+    seen: dict[str, bytes] = {}
+    stack: list[tuple[str, str]] = [(name, "") for name in roots]
+    while stack:
+        name, from_dir = stack.pop()
+        for candidate in (f"{from_dir}/{name}" if from_dir else name, name):
+            rel = posixpath.normpath(candidate)
+            if not rel.startswith("..") and (src_dir / rel).is_file():
+                break
+        else:
+            continue
+        if rel in seen:
+            continue
+        data = seen[rel] = (src_dir / rel).read_bytes()
+        parent = posixpath.dirname(rel)
+        stack.extend((inc.decode(), parent) for inc in _INCLUDE_RE.findall(data))
+    return seen
+
+
+def pch_checksum(
+    src_dir: Path, include_headers: Iterable[str], extra: Iterable[str]
+) -> str:
+    """Digest of the prefix header's include closure plus ``extra``."""
+    digest = hashlib.sha256()
+    closure = _include_closure(src_dir, include_headers)
+    for name in sorted(closure):
+        digest.update(name.encode())
+        digest.update(closure[name])
+        digest.update(b"\0")
+    for item in extra:
+        digest.update(item.encode())
+        digest.update(b"\0")
+    return digest.hexdigest()
+
+
+def pch_identity(
+    tokens: Iterable[str],
+    src_dir: Path,
+    include_headers: tuple[str, ...],
+    extra: Iterable[str],
+) -> str:
+    """The .sum digest: include closure, header text, ``extra`` and the
+    compile flags with the build path stripped, as ccache does."""
+    from esphome.core import CORE
+
+    flags = (
+        " ".join(tokens)
+        .replace(str(CORE.build_path), "")
+        .replace(effective_ccache_basedir(), "")
+    )
+    # The closure is sorted, so header order only enters via the text
+    return pch_checksum(
+        src_dir, include_headers, (pch_header_text(include_headers), *extra, flags)
+    )
+
+
+_DISABLE_HINT = " (set ESPHOME_PCH_ENABLE=0 to disable)"
+
+
+def log_pch_in_use() -> None:
+    _LOGGER.info("Compiling with a precompiled header%s", _DISABLE_HINT)

esphome/build_helpers/tool_runner.py

@@ -0,0 +1,180 @@
+"""Run a native build tool (cmake, ninja) and relay its output.
+
+Output is read from a pipe so it can be filtered here: a child that inherits
+our stdout writes straight to the file descriptor, past any Python wrapper.
+"""
+
+from __future__ import annotations
+
+import codecs
+from contextlib import suppress
+import logging
+import os
+from pathlib import Path
+import re
+import shutil
+import subprocess
+import sys
+from typing import Any, TextIO
+
+from esphome.util import ANSI_ESCAPE, RedirectText, shlex_quote
+
+_LOGGER = logging.getLogger(__name__)
+
+# Windows code page identifier for UTF-8, as used by ``chcp 65001``.
+UTF8_CODEPAGE = 65001
+
+# Same pattern idf.py uses to spot ninja status lines (``is_progression``).
+_PROGRESS = re.compile(r"^\[\d+/\d+\]|.*\(\d+ \%\)$")
+_READ_SIZE = 65536
+
+
+def _get_kernel32() -> Any | None:
+    """Return the Windows kernel32 module, or None on any other platform."""
+    if sys.platform != "win32":
+        return None
+    import ctypes
+
+    return ctypes.windll.kernel32
+
+
+class Utf8Console:
+    """Keep an attached Windows console on UTF-8 while a build tool runs.
+
+    esp_idf_size draws its table with Unicode box characters, and CMake
+    re-decodes a child's output with the console code page, which garbles the
+    table on any page but UTF-8. A console already on UTF-8 is left alone so
+    an overlapping build never records UTF-8 as the page to go back to.
+    """
+
+    def __init__(self, kernel32: Any | None) -> None:
+        self._kernel32 = kernel32
+        self._codepages: tuple[int, int] | None = None
+
+    def __enter__(self) -> None:
+        kernel32 = self._kernel32
+        if kernel32 is None:
+            return
+        old_in = kernel32.GetConsoleCP()
+        old_out = kernel32.GetConsoleOutputCP()
+        # Both calls return 0 when no console is attached.
+        if not old_in or not old_out:
+            return
+        if old_in == UTF8_CODEPAGE and old_out == UTF8_CODEPAGE:
+            return
+        # Record first so a switch that fails part way is still undone.
+        self._codepages = (old_in, old_out)
+        kernel32.SetConsoleCP(UTF8_CODEPAGE)
+        kernel32.SetConsoleOutputCP(UTF8_CODEPAGE)
+
+    def __exit__(self, *exc_info: object) -> None:
+        if self._codepages is None:
+            return
+        old_in, old_out = self._codepages
+        self._codepages = None
+        self._kernel32.SetConsoleCP(old_in)
+        self._kernel32.SetConsoleOutputCP(old_out)
+
+
+def _fit_terminal(text: str) -> str:
+    """Elide the middle of ``text`` to fit the terminal, as idf.py does.
+
+    A width of 0 (a pipe, the dashboard) leaves the text whole.
+    """
+    width = shutil.get_terminal_size((0, 0)).columns
+    if not width:
+        return text
+    if width <= 3:
+        return "." * width
+    if len(text) >= width:
+        keep = (width - 3) // 2
+        return f"{text[:keep]}...{text[len(text) - keep :]}"
+    return text
+
+
+class ToolOutput(RedirectText):
+    """RedirectText that can collapse ninja status lines into one line.
+
+    With ``progress`` each ``[n/m]`` line overwrites the previous one, the
+    way idf.py shows a build.
+    """
+
+    def __init__(
+        self, out: TextIO, filter_lines: list[str] | None, progress: bool
+    ) -> None:
+        super().__init__(out, filter_lines=filter_lines)
+        self._progress = progress
+        self._on_progress_line = False
+
+    def _splits_lines(self) -> bool:
+        return self._progress or super()._splits_lines()
+
+    def _emit_line(self, line: str) -> None:
+        if self._progress and _PROGRESS.match(line):
+            if not self._is_filtered(line):
+                text = _fit_terminal(line.strip("\r\n"))
+                self._write_color_replace(f"\r{text}\x1b[K")
+                self._on_progress_line = True
+            return
+        self._end_progress_line()
+        super()._emit_line(line)
+
+    def _end_progress_line(self) -> None:
+        if self._on_progress_line:
+            self._on_progress_line = False
+            self._write_color_replace(os.linesep)
+
+    def drain(self) -> None:
+        super().drain()
+        # Called from cleanup, so a broken stream must not hide the exit code.
+        with suppress(OSError, ValueError):
+            self._end_progress_line()
+            self._out.flush()
+
+
+def run_build_tool(
+    cmd: list[str],
+    *,
+    cwd: Path,
+    env: dict[str, str],
+    filter_lines: list[str] | None = None,
+    progress: bool = False,
+    log_path: Path | None = None,
+) -> int:
+    """Run ``cmd`` and relay stdout and stderr, merged, to our stdout.
+
+    ``log_path`` also gets the full, unfiltered output without color codes, as
+    idf.py wrote its logs (its hint patterns expect plain text). Returns the
+    exit code.
+    """
+    _LOGGER.debug("Running: %s", " ".join(shlex_quote(arg) for arg in cmd))
+    _LOGGER.debug("  in directory: %s", cwd)
+    output = ToolOutput(sys.stdout, filter_lines, progress)
+    decoder = codecs.getincrementaldecoder("utf-8")(errors="replace")
+    if log_path is not None:
+        log_path.parent.mkdir(parents=True, exist_ok=True)
+    with (
+        Path(log_path or os.devnull).open("w", encoding="utf-8", newline="") as log,
+        Utf8Console(_get_kernel32()),
+        subprocess.Popen(
+            cmd,
+            cwd=cwd,
+            env=env,
+            stdout=subprocess.PIPE,
+            stderr=subprocess.STDOUT,
+            close_fds=False,
+        ) as proc,
+    ):
+        try:
+            # read1 returns as soon as anything is available, so output
+            # streams while the tool runs.
+            while chunk := proc.stdout.read1(_READ_SIZE):
+                text = decoder.decode(chunk)
+                log.write(ANSI_ESCAPE.sub("", text))
+                output.write(text)
+            if tail := decoder.decode(b"", final=True):
+                log.write(ANSI_ESCAPE.sub("", tail))
+                output.write(tail)
+        finally:
+            output.drain()
+    return proc.returncode

esphome/build_helpers/tools_cache.py

@@ -16,8 +16,8 @@ def tools_cache_path(env_var: str, subdir: str) -> Path:
     from esphome.helpers import get_str_env
 
     if prefix := get_str_env(env_var, "").strip():
-        # resolve(): symlinked prefixes otherwise trip idf.py's
-        # venv-mismatch warning on every build
+        # resolve(): a symlinked prefix would otherwise record a second
+        # spelling of the same paths in the build tree
         return Path(prefix).expanduser().resolve()
     # appauthor=False keeps the Windows path short (no vendor segment);
     # deep IDF trees run into MAX_PATH otherwise
@@ -33,4 +33,14 @@ def tools_cache_path(env_var: str, subdir: str) -> Path:
 IDF_TOOLS_CACHE = ("ESPHOME_ESP_IDF_PREFIX", "idf")
 SDK_NRF_TOOLS_CACHE = ("ESPHOME_SDK_NRF_PREFIX", "sdk-nrf")
 ARDUINO8266_TOOLS_CACHE = ("ESPHOME_ARDUINO8266_PREFIX", "arduino8266")
-TOOLS_CACHE_SPECS = (IDF_TOOLS_CACHE, SDK_NRF_TOOLS_CACHE, ARDUINO8266_TOOLS_CACHE)
+# The host backend installs nothing; the entry only holds its ccache dir
+HOST_TOOLS_CACHE = ("ESPHOME_HOST_PREFIX", "host")
+# PlatformIO installs into its own dirs; the entry is its ccache dir itself
+PLATFORMIO_CCACHE = ("ESPHOME_PLATFORMIO_CCACHE_DIR", "platformio-ccache")
+TOOLS_CACHE_SPECS = (
+    IDF_TOOLS_CACHE,
+    SDK_NRF_TOOLS_CACHE,
+    ARDUINO8266_TOOLS_CACHE,
+    HOST_TOOLS_CACHE,
+    PLATFORMIO_CCACHE,
+)

esphome/cli/__init__.py

@@ -0,0 +1,2 @@
+"""Commands of the esphome command line, one module each, imported by
+__main__ only when they run so that startup stays light."""

esphome/cli/rename.py

@@ -0,0 +1,164 @@
+"""``esphome rename``."""
+
+from __future__ import annotations
+
+import argparse
+from pathlib import Path
+import re
+
+from esphome import yaml_edit, yaml_util
+from esphome.const import (
+    ALLOWED_NAME_CHARS,
+    CONF_ESPHOME,
+    CONF_NAME,
+    CONF_SUBSTITUTIONS,
+)
+from esphome.core import CORE, EsphomeError
+from esphome.log import AnsiFore, color
+from esphome.types import ConfigType
+from esphome.util import ESPHOME_COMMAND, run_external_process, safe_print
+
+
+def _revert(new_path: Path, why: str) -> int:
+    """Say why the rename stopped and take the new file back; an orphan the
+    next attempt would trip over is reported."""
+    safe_print(color(AnsiFore.BOLD_RED, f"Rename failed: {why}"))
+    try:
+        new_path.unlink(missing_ok=True)
+    except OSError as err:
+        safe_print(color(AnsiFore.BOLD_RED, f"Could not remove {new_path}: {err}"))
+    return 1
+
+
+def command_rename(args: argparse.Namespace, config: ConfigType) -> int | None:
+    """Rename the device: a new file with the name line rewritten, validated
+    and installed, then the old file removed."""
+    new_name = args.name
+    for c in new_name:
+        if c not in ALLOWED_NAME_CHARS:
+            safe_print(
+                color(
+                    AnsiFore.BOLD_RED,
+                    f"'{c}' is an invalid character for names. Valid characters are: "
+                    f"{ALLOWED_NAME_CHARS} (lowercase, no spaces)",
+                )
+            )
+            return 1
+
+    yaml = yaml_util.load_yaml(CORE.config_path)
+
+    def name_edit() -> tuple[str, yaml_edit.LineEdit]:
+        """The name and the line to rewrite: the name's own line, or the
+        substitution's line it comes from, as a plain value in this file."""
+        esphome_conf = yaml.get(CONF_ESPHOME)
+        if not isinstance(esphome_conf, dict) or CONF_NAME not in esphome_conf:
+            raise EsphomeError(f"no '{CONF_ESPHOME}: {CONF_NAME}:' in the file")
+        old_name = str(esphome_conf[CONF_NAME])
+        mapping, field = esphome_conf, CONF_NAME
+        if match := re.match(r"^\$\{?([a-zA-Z0-9_]+)\}?$", old_name):
+            mapping, field = yaml.get(CONF_SUBSTITUTIONS), match.group(1)
+            if not isinstance(mapping, dict) or field not in mapping:
+                raise EsphomeError(f"the substitution '{field}' is not in the file")
+            old_name = str(mapping[field])
+        # Only read here; the rewritten text goes to a new file, so the
+        # source may live anywhere the config path points to
+        source = yaml_edit.source_of(mapping, field)
+        if source is None or source[0].resolve() != CORE.config_path.resolve():
+            raise EsphomeError(f"'{field}' was not read from {CORE.config_path}")
+        doc, line_no = source
+        text = yaml_edit.line_at(doc, line_no)
+        if (line_match := yaml_edit.field_line_re(field, old_name).match(text)) is None:
+            raise EsphomeError(f"'{field}' is not a plain value on {doc}:{line_no + 1}")
+        # The new value is always quoted, whatever the old line had
+        return old_name, yaml_edit.LineEdit(
+            doc, line_no, text, yaml_edit.rewrite(line_match, new_name, '"')
+        )
+
+    try:
+        old_name, edit = name_edit()
+    except EsphomeError as err:
+        safe_print(
+            color(
+                AnsiFore.BOLD_RED,
+                f"Complex YAML files cannot be automatically renamed: {err}",
+            )
+        )
+        return 1
+
+    # ``new_name == old_name`` (after substitution resolution) is
+    # a no-op rewrite that would still queue a pointless re-flash.
+    # Catch it before the path-equality check below — covers the
+    # case where the config filename doesn't match the device name
+    # (e.g. ``weird-file.yaml`` whose ``esphome.name`` is
+    # ``kitchen``; running ``esphome rename weird-file.yaml kitchen``
+    # would otherwise just re-flash the same hostname).
+    if new_name == old_name:
+        safe_print(
+            color(
+                AnsiFore.BOLD_RED,
+                f"'{new_name}' is already the device's name.",
+            )
+        )
+        return 1
+
+    new_path: Path = CORE.config_dir / (new_name + ".yaml")
+    if new_path.resolve() == CORE.config_path.resolve():
+        safe_print(
+            color(
+                AnsiFore.BOLD_RED,
+                f"'{new_name}' is already the device's name.",
+            )
+        )
+        return 1
+    if new_path.exists():
+        safe_print(
+            color(
+                AnsiFore.BOLD_RED,
+                f"Cannot rename: {new_path} already exists. "
+                "Refusing to overwrite an existing configuration.",
+            )
+        )
+        return 1
+    safe_print(
+        f"Updating {color(AnsiFore.CYAN, str(CORE.config_path))} to {color(AnsiFore.CYAN, str(new_path))}"
+    )
+    print()
+
+    try:
+        yaml_edit.write_keeping_mode(
+            new_path,
+            yaml_edit.rewritten_text(yaml_edit.read_text(CORE.config_path), [edit]),
+            like=CORE.config_path,
+        )
+    except EsphomeError as err:
+        return _revert(new_path, str(err))
+
+    if run_external_process(*ESPHOME_COMMAND, "config", str(new_path)) != 0:
+        return _revert(new_path, "the new configuration does not validate")
+
+    cli_args = [
+        "run",
+        str(new_path),
+        "--no-logs",
+        "--device",
+        CORE.address,
+    ]
+
+    if args.dashboard:
+        cli_args.insert(0, "--dashboard")
+
+    try:
+        rc = run_external_process(*ESPHOME_COMMAND, *cli_args)
+    except KeyboardInterrupt:
+        rc = 1
+    if rc != 0:
+        return _revert(
+            new_path,
+            "the install did not finish; the device may already run the new name",
+        )
+
+    CORE.config_path.unlink()
+
+    safe_print(color(AnsiFore.BOLD_GREEN, "SUCCESS"))
+    print()
+    return 0

esphome/codegen.py

@@ -31,6 +31,7 @@ from esphome.cpp_generator import (  # noqa: F401
     add_global,
     add_library,
     add_platformio_option,
+    extern_progmem_array,
     get_variable,
     get_variable_with_full_id,
     is_template,
@@ -40,8 +41,10 @@ from esphome.cpp_generator import (  # noqa: F401
     progmem_array,
     safe_exp,
     set_cpp_standard,
+    shared_progmem_array,
     statement,
     static_const_array,
+    static_function,
     templatable,
     variable,
     with_local_variable,
@@ -64,7 +67,6 @@ from esphome.cpp_types import (  # noqa: F401
     Application,
     Component,
     ComponentPtr,
-    Controller,
     EntityBase,
     EntityCategory,
     ESPTime,

esphome/component_aliases.py

@@ -6,5 +6,6 @@ See the component-alias section of esphome/loader.py.
 
 # alias -> (canonical component, removal version or None)
 COMPONENT_ALIASES: dict[str, tuple[str, str | None]] = {
+    "esp32_improv": ("improv_ble", "2027.4.0"),
     "rp2040": ("rp2", "2027.7.0"),
 }

esphome/components/adc/adc_sensor.h

@@ -94,7 +94,7 @@ class ADCSensor final : public sensor::Sensor, public PollingComponent, public v
   ///   - SamplingMode::MIN: Use the lowest sample value
   ///   - SamplingMode::MAX: Use the highest sample value
   /// @param sampling_mode The desired sampling mode to use for aggregating ADC samples.
-  void set_sampling_mode(SamplingMode sampling_mode);
+  void set_sampling_mode(SamplingMode sampling_mode) { this->sampling_mode_ = sampling_mode; }
 
   /// Perform a single ADC sampling operation and return the measured value.
   /// This function handles raw readings, calibration, and averaging as needed.

esphome/components/adc/adc_sensor_common.cpp

@@ -76,6 +76,4 @@ void ADCSensor::set_sample_count(uint8_t sample_count) {
   }
 }
 
-void ADCSensor::set_sampling_mode(SamplingMode sampling_mode) { this->sampling_mode_ = sampling_mode; }
-
 }  // namespace esphome::adc

esphome/components/ags10/ags10.cpp

@@ -106,6 +106,20 @@ bool AGS10Component::set_zero_point_with_factory_defaults() { return this->set_z
 
 bool AGS10Component::set_zero_point_with_current_resistance() { return this->set_zero_point_with(ZP_CURRENT); }
 
+void AGS10Component::set_zero_point(AGS10SetZeroPointActionMode mode, uint16_t value) {
+  switch (mode) {
+    case FACTORY_DEFAULT:
+      this->set_zero_point_with_factory_defaults();
+      break;
+    case CURRENT_VALUE:
+      this->set_zero_point_with_current_resistance();
+      break;
+    case CUSTOM_VALUE:
+      this->set_zero_point_with(value);
+      break;
+  }
+}
+
 bool AGS10Component::set_zero_point_with(uint16_t value) {
   std::array<uint8_t, 5> data{0x00, 0x0C, (uint8_t) ((value >> 8) & 0xFF), (uint8_t) (value & 0xFF), 0};
   data[4] = crc8(data.data(), 4, 0xFF, 0x31, true);

esphome/components/ags10/ags10.h

@@ -2,11 +2,19 @@
 
 #include "esphome/components/i2c/i2c.h"
 #include "esphome/components/sensor/sensor.h"
-#include "esphome/core/automation.h"
 #include "esphome/core/component.h"
 
 namespace esphome::ags10 {
 
+enum AGS10SetZeroPointActionMode {
+  // Zero-point reset.
+  FACTORY_DEFAULT,
+  // Zero-point calibration with current resistance.
+  CURRENT_VALUE,
+  // Zero-point calibration with custom resistance.
+  CUSTOM_VALUE,
+};
+
 class AGS10Component final : public PollingComponent, public i2c::I2CDevice {
  public:
   /**
@@ -47,6 +55,11 @@ class AGS10Component final : public PollingComponent, public i2c::I2CDevice {
    */
   bool set_zero_point_with_current_resistance();
 
+  /**
+   * Sets zero-point by mode; the value is only used for CUSTOM_VALUE.
+   */
+  void set_zero_point(AGS10SetZeroPointActionMode mode, uint16_t value);
+
   /**
    * Sets zero-point with the value.
    */
@@ -100,39 +113,4 @@ class AGS10Component final : public PollingComponent, public i2c::I2CDevice {
   template<size_t N> optional<std::array<uint8_t, N>> read_and_check_(uint8_t a_register);
 };
 
-template<typename... Ts> class AGS10NewI2cAddressAction final : public Action<Ts...>, public Parented<AGS10Component> {
- public:
-  TEMPLATABLE_VALUE(uint8_t, new_address)
-
-  void play(const Ts &...x) override { this->parent_->new_i2c_address(this->new_address_.value(x...)); }
-};
-
-enum AGS10SetZeroPointActionMode {
-  // Zero-point reset.
-  FACTORY_DEFAULT,
-  // Zero-point calibration with current resistance.
-  CURRENT_VALUE,
-  // Zero-point calibration with custom resistance.
-  CUSTOM_VALUE,
-};
-
-template<typename... Ts> class AGS10SetZeroPointAction final : public Action<Ts...>, public Parented<AGS10Component> {
- public:
-  TEMPLATABLE_VALUE(uint16_t, value)
-  TEMPLATABLE_VALUE(AGS10SetZeroPointActionMode, mode)
-
-  void play(const Ts &...x) override {
-    switch (this->mode_.value(x...)) {
-      case FACTORY_DEFAULT:
-        this->parent_->set_zero_point_with_factory_defaults();
-        break;
-      case CURRENT_VALUE:
-        this->parent_->set_zero_point_with_current_resistance();
-        break;
-      case CUSTOM_VALUE:
-        this->parent_->set_zero_point_with(this->value_.value(x...));
-        break;
-    }
-  }
-};
 }  // namespace esphome::ags10

esphome/components/ags10/sensor.py

@@ -17,8 +17,6 @@ from esphome.const import (
     UNIT_OHM,
     UNIT_PARTS_PER_BILLION,
 )
-from esphome.core import ID
-from esphome.cpp_generator import MockObj, TemplateArgsType
 from esphome.types import ConfigType
 
 CONF_RESISTANCE = "resistance"
@@ -28,12 +26,6 @@ DEPENDENCIES = ["i2c"]
 ags10_ns = cg.esphome_ns.namespace("ags10")
 AGS10Component = ags10_ns.class_("AGS10Component", cg.PollingComponent, i2c.I2CDevice)
 
-# Actions
-AGS10NewI2cAddressAction = ags10_ns.class_(
-    "AGS10NewI2cAddressAction", automation.Action
-)
-AGS10SetZeroPointAction = ags10_ns.class_("AGS10SetZeroPointAction", automation.Action)
-
 CONFIG_SCHEMA = (
     cv.Schema(
         {
@@ -70,16 +62,10 @@ async def to_code(config: ConfigType) -> None:
     await cg.register_component(var, config)
     await i2c.register_i2c_device(var, config)
 
-    sens = await sensor.new_sensor(config[CONF_TVOC])
-    cg.add(var.set_tvoc(sens))
-
-    if version_config := config.get(CONF_VERSION):
-        sens = await sensor.new_sensor(version_config)
-        cg.add(var.set_version(sens))
-
-    if resistance_config := config.get(CONF_RESISTANCE):
-        sens = await sensor.new_sensor(resistance_config)
-        cg.add(var.set_resistance(sens))
+    sensors = sensor.sub_sensors(config)
+    await sensors(CONF_TVOC, var.set_tvoc)
+    await sensors(CONF_VERSION, var.set_version)
+    await sensors(CONF_RESISTANCE, var.set_resistance)
 
 
 AGS10_NEW_I2C_ADDRESS_SCHEMA = cv.maybe_simple_value(
@@ -91,24 +77,11 @@ AGS10_NEW_I2C_ADDRESS_SCHEMA = cv.maybe_simple_value(
 )
 
 
-@automation.register_action(
+automation.register_apply_action(
     "ags10.new_i2c_address",
-    AGS10NewI2cAddressAction,
     AGS10_NEW_I2C_ADDRESS_SCHEMA,
-    synchronous=True,
+    automation.ApplyField(CONF_ADDRESS, "new_i2c_address", cg.uint8),
 )
-async def ags10newi2caddress_to_code(
-    config: ConfigType,
-    action_id: ID,
-    template_arg: cg.TemplateArguments,
-    args: TemplateArgsType,
-) -> MockObj:
-    var = cg.new_Pvariable(action_id, template_arg)
-    await cg.register_parented(var, config[CONF_ID])
-    address = await cg.templatable(config[CONF_ADDRESS], args, cg.uint8)
-    cg.add(var.set_new_address(address))
-    return var
-
 
 AGS10SetZeroPointActionMode = ags10_ns.enum("AGS10SetZeroPointActionMode")
 AGS10_SET_ZERO_POINT_ACTION_MODE = {
@@ -128,24 +101,11 @@ AGS10_SET_ZERO_POINT_SCHEMA = cv.Schema(
 )
 
 
-@automation.register_action(
+automation.register_apply_action(
     "ags10.set_zero_point",
-    AGS10SetZeroPointAction,
     AGS10_SET_ZERO_POINT_SCHEMA,
-    synchronous=True,
+    automation.ApplyCall(
+        "set_zero_point({}, {})",
+        ((CONF_MODE, AGS10SetZeroPointActionMode), (CONF_VALUE, cg.uint16)),
+    ),
 )
-async def ags10setzeropoint_to_code(
-    config: ConfigType,
-    action_id: ID,
-    template_arg: cg.TemplateArguments,
-    args: TemplateArgsType,
-) -> MockObj:
-    var = cg.new_Pvariable(action_id, template_arg)
-    await cg.register_parented(var, config[CONF_ID])
-    mode = await cg.templatable(
-        config.get(CONF_MODE), args, AGS10SetZeroPointActionMode
-    )
-    cg.add(var.set_mode(mode))
-    value = await cg.templatable(config[CONF_VALUE], args, cg.uint16)
-    cg.add(var.set_value(value))
-    return var

esphome/components/aht10/sensor.py

@@ -57,10 +57,6 @@ async def to_code(config: ConfigType) -> None:
     await i2c.register_i2c_device(var, config)
     cg.add(var.set_variant(config[CONF_VARIANT]))
 
-    if temperature := config.get(CONF_TEMPERATURE):
-        sens = await sensor.new_sensor(temperature)
-        cg.add(var.set_temperature_sensor(sens))
-
-    if humidity := config.get(CONF_HUMIDITY):
-        sens = await sensor.new_sensor(humidity)
-        cg.add(var.set_humidity_sensor(sens))
+    sensors = sensor.sub_sensors(config)
+    await sensors(CONF_TEMPERATURE, var.set_temperature_sensor)
+    await sensors(CONF_HUMIDITY, var.set_humidity_sensor)

esphome/components/aic3204/audio_dac.py

@@ -4,8 +4,6 @@ from esphome.components import i2c
 from esphome.components.audio_dac import AudioDac
 import esphome.config_validation as cv
 from esphome.const import CONF_ID, CONF_MODE
-from esphome.core import ID
-from esphome.cpp_generator import MockObj, TemplateArgsType
 from esphome.types import ConfigType
 
 CODEOWNERS = ["@kbx81"]
@@ -14,7 +12,6 @@ DEPENDENCIES = ["i2c"]
 aic3204_ns = cg.esphome_ns.namespace("aic3204")
 AIC3204 = aic3204_ns.class_("AIC3204", AudioDac, cg.Component, i2c.I2CDevice)
 
-SetAutoMuteAction = aic3204_ns.class_("SetAutoMuteAction", automation.Action)
 
 CONFIG_SCHEMA = (
     cv.Schema(
@@ -36,25 +33,11 @@ SET_AUTO_MUTE_ACTION_SCHEMA = cv.maybe_simple_value(
 )
 
 
-@automation.register_action(
+automation.register_apply_action(
     "aic3204.set_auto_mute_mode",
-    SetAutoMuteAction,
     SET_AUTO_MUTE_ACTION_SCHEMA,
-    synchronous=True,
+    automation.ApplyField(CONF_MODE, "set_auto_mute_mode", cg.uint8),
 )
-async def aic3204_set_volume_to_code(
-    config: ConfigType,
-    action_id: ID,
-    template_arg: cg.TemplateArguments,
-    args: TemplateArgsType,
-) -> MockObj:
-    paren = await cg.get_variable(config[CONF_ID])
-    var = cg.new_Pvariable(action_id, template_arg, paren)
-
-    template_ = await cg.templatable(config.get(CONF_MODE), args, cg.uint8)
-    cg.add(var.set_auto_mute_mode(template_))
-
-    return var
 
 
 async def to_code(config: ConfigType) -> None:

esphome/components/aic3204/automation.h

@@ -1,21 +0,0 @@
-#pragma once
-
-#include "esphome/core/automation.h"
-#include "esphome/core/component.h"
-#include "aic3204.h"
-
-namespace esphome::aic3204 {
-
-template<typename... Ts> class SetAutoMuteAction final : public Action<Ts...> {
- public:
-  explicit SetAutoMuteAction(AIC3204 *aic3204) : aic3204_(aic3204) {}
-
-  TEMPLATABLE_VALUE(uint8_t, auto_mute_mode)
-
-  void play(const Ts &...x) override { this->aic3204_->set_auto_mute_mode(this->auto_mute_mode_.value(x...)); }
-
- protected:
-  AIC3204 *aic3204_;
-};
-
-}  // namespace esphome::aic3204

esphome/components/airthings_wave_base/__init__.py

@@ -85,20 +85,11 @@ async def wave_base_to_code(var: MockObj, config: ConfigType) -> None:
 
     await ble_client.register_ble_node(var, config)
 
-    if config_humidity := config.get(CONF_HUMIDITY):
-        sens = await sensor.new_sensor(config_humidity)
-        cg.add(var.set_humidity(sens))
-    if config_temperature := config.get(CONF_TEMPERATURE):
-        sens = await sensor.new_sensor(config_temperature)
-        cg.add(var.set_temperature(sens))
-    if config_pressure := config.get(CONF_PRESSURE):
-        sens = await sensor.new_sensor(config_pressure)
-        cg.add(var.set_pressure(sens))
-    if config_tvoc := config.get(CONF_TVOC):
-        sens = await sensor.new_sensor(config_tvoc)
-        cg.add(var.set_tvoc(sens))
-    if config_battery_voltage := config.get(CONF_BATTERY_VOLTAGE):
-        sens = await sensor.new_sensor(config_battery_voltage)
-        cg.add(var.set_battery_voltage(sens))
+    sensors = sensor.sub_sensors(config)
+    await sensors(CONF_HUMIDITY, var.set_humidity)
+    await sensors(CONF_TEMPERATURE, var.set_temperature)
+    await sensors(CONF_PRESSURE, var.set_pressure)
+    await sensors(CONF_TVOC, var.set_tvoc)
+    await sensors(CONF_BATTERY_VOLTAGE, var.set_battery_voltage)
     if config_battery_update_interval := config.get(CONF_BATTERY_UPDATE_INTERVAL):
         cg.add(var.set_battery_update_interval(config_battery_update_interval))

esphome/components/airthings_wave_plus/sensor.py

@@ -87,16 +87,9 @@ async def to_code(config: ConfigType) -> None:
     var = cg.new_Pvariable(config[CONF_ID])
     await airthings_wave_base.wave_base_to_code(var, config)
 
-    if config_radon := config.get(CONF_RADON):
-        sens = await sensor.new_sensor(config_radon)
-        cg.add(var.set_radon(sens))
-    if config_radon_long_term := config.get(CONF_RADON_LONG_TERM):
-        sens = await sensor.new_sensor(config_radon_long_term)
-        cg.add(var.set_radon_long_term(sens))
-    if config_co2 := config.get(CONF_CO2):
-        sens = await sensor.new_sensor(config_co2)
-        cg.add(var.set_co2(sens))
-    if config_illuminance := config.get(CONF_ILLUMINANCE):
-        sens = await sensor.new_sensor(config_illuminance)
-        cg.add(var.set_illuminance(sens))
+    sensors = sensor.sub_sensors(config)
+    await sensors(CONF_RADON, var.set_radon)
+    await sensors(CONF_RADON_LONG_TERM, var.set_radon_long_term)
+    await sensors(CONF_CO2, var.set_co2)
+    await sensors(CONF_ILLUMINANCE, var.set_illuminance)
     cg.add(var.set_device_type(config[CONF_DEVICE_TYPE]))

esphome/components/alarm_control_panel/__init__.py

@@ -41,18 +41,6 @@ StateAnyForwarder = alarm_control_panel_ns.class_("StateAnyForwarder")
 StateEnterForwarder = alarm_control_panel_ns.class_("StateEnterForwarder")
 AlarmControlPanelState = alarm_control_panel_ns.enum("AlarmControlPanelState")
 
-ArmAwayAction = alarm_control_panel_ns.class_("ArmAwayAction", automation.Action)
-ArmHomeAction = alarm_control_panel_ns.class_("ArmHomeAction", automation.Action)
-ArmNightAction = alarm_control_panel_ns.class_("ArmNightAction", automation.Action)
-DisarmAction = alarm_control_panel_ns.class_("DisarmAction", automation.Action)
-PendingAction = alarm_control_panel_ns.class_("PendingAction", automation.Action)
-TriggeredAction = alarm_control_panel_ns.class_("TriggeredAction", automation.Action)
-ChimeAction = alarm_control_panel_ns.class_("ChimeAction", automation.Action)
-ReadyAction = alarm_control_panel_ns.class_("ReadyAction", automation.Action)
-
-AlarmControlPanelCondition = alarm_control_panel_ns.class_(
-    "AlarmControlPanelCondition", automation.Condition
-)
 
 _ALARM_CONTROL_PANEL_SCHEMA = (
     cv.ENTITY_BASE_SCHEMA.extend(web_server.WEBSERVER_SORTING_SCHEMA)
@@ -196,125 +184,38 @@ async def new_alarm_control_panel(config, *args):
     return var
 
 
-@automation.register_action(
-    "alarm_control_panel.arm_away",
-    ArmAwayAction,
-    ALARM_CONTROL_PANEL_ACTION_SCHEMA,
-    synchronous=True,
-)
-async def alarm_action_arm_away_to_code(config, action_id, template_arg, args):
-    paren = await cg.get_variable(config[CONF_ID])
-    var = cg.new_Pvariable(action_id, template_arg, paren)
-    if code_config := config.get(CONF_CODE):
-        templatable_ = await cg.templatable(code_config, args, cg.std_string)
-        cg.add(var.set_code(templatable_))
-    return var
-
-
-@automation.register_action(
-    "alarm_control_panel.arm_home",
-    ArmHomeAction,
-    ALARM_CONTROL_PANEL_ACTION_SCHEMA,
-    synchronous=True,
-)
-async def alarm_action_arm_home_to_code(config, action_id, template_arg, args):
-    paren = await cg.get_variable(config[CONF_ID])
-    var = cg.new_Pvariable(action_id, template_arg, paren)
-    if code_config := config.get(CONF_CODE):
-        templatable_ = await cg.templatable(code_config, args, cg.std_string)
-        cg.add(var.set_code(templatable_))
-    return var
-
-
-@automation.register_action(
-    "alarm_control_panel.arm_night",
-    ArmNightAction,
-    ALARM_CONTROL_PANEL_ACTION_SCHEMA,
-    synchronous=True,
-)
-async def alarm_action_arm_night_to_code(config, action_id, template_arg, args):
-    paren = await cg.get_variable(config[CONF_ID])
-    var = cg.new_Pvariable(action_id, template_arg, paren)
-    if CONF_CODE in config:
-        templatable_ = await cg.templatable(config[CONF_CODE], args, cg.std_string)
-        cg.add(var.set_code(templatable_))
-    return var
-
-
-@automation.register_action(
-    "alarm_control_panel.disarm",
-    DisarmAction,
-    ALARM_CONTROL_PANEL_ACTION_SCHEMA,
-    synchronous=True,
-)
-async def alarm_action_disarm_to_code(config, action_id, template_arg, args):
-    paren = await cg.get_variable(config[CONF_ID])
-    var = cg.new_Pvariable(action_id, template_arg, paren)
-    if code_config := config.get(CONF_CODE):
-        templatable_ = await cg.templatable(code_config, args, cg.std_string)
-        cg.add(var.set_code(templatable_))
-    return var
-
-
-@automation.register_action(
-    "alarm_control_panel.pending",
-    PendingAction,
-    ALARM_CONTROL_PANEL_ACTION_SCHEMA,
-    synchronous=True,
-)
-async def alarm_action_pending_to_code(config, action_id, template_arg, args):
-    paren = await cg.get_variable(config[CONF_ID])
-    return cg.new_Pvariable(action_id, template_arg, paren)
-
-
-@automation.register_action(
-    "alarm_control_panel.triggered",
-    TriggeredAction,
-    ALARM_CONTROL_PANEL_ACTION_SCHEMA,
-    synchronous=True,
-)
-async def alarm_action_trigger_to_code(config, action_id, template_arg, args):
-    paren = await cg.get_variable(config[CONF_ID])
-    return cg.new_Pvariable(action_id, template_arg, paren)
-
-
-@automation.register_action(
-    "alarm_control_panel.chime",
-    ChimeAction,
-    ALARM_CONTROL_PANEL_ACTION_SCHEMA,
-    synchronous=True,
-)
-async def alarm_action_chime_to_code(config, action_id, template_arg, args):
-    paren = await cg.get_variable(config[CONF_ID])
-    return cg.new_Pvariable(action_id, template_arg, paren)
-
-
-@automation.register_action(
-    "alarm_control_panel.ready",
-    ReadyAction,
-    ALARM_CONTROL_PANEL_ACTION_SCHEMA,
-    synchronous=True,
-)
-@automation.register_condition(
-    "alarm_control_panel.ready",
-    AlarmControlPanelCondition,
-    ALARM_CONTROL_PANEL_CONDITION_SCHEMA,
-)
-async def alarm_action_ready_to_code(config, action_id, template_arg, args):
-    paren = await cg.get_variable(config[CONF_ID])
-    return cg.new_Pvariable(action_id, template_arg, paren)
-
-
-@automation.register_condition(
-    "alarm_control_panel.is_armed",
-    AlarmControlPanelCondition,
-    ALARM_CONTROL_PANEL_CONDITION_SCHEMA,
-)
-async def alarm_control_panel_is_armed_to_code(
-    config, condition_id, template_arg, args
+# Mirrors AlarmControlPanel::arm_with_code_: arm first, set the code only when given.
+for _name, _arm in (
+    ("alarm_control_panel.arm_away", "arm_away()"),
+    ("alarm_control_panel.arm_home", "arm_home()"),
+    ("alarm_control_panel.arm_night", "arm_night()"),
+    ("alarm_control_panel.disarm", "disarm()"),
 ):
-    paren = await cg.get_variable(config[CONF_ID])
-    return cg.new_Pvariable(condition_id, template_arg, paren)
+    automation.register_apply_action(
+        _name,
+        ALARM_CONTROL_PANEL_ACTION_SCHEMA,
+        automation.ApplyCall(_arm),
+        automation.ApplyField(CONF_CODE, "set_code", cg.std_string),
+        call="make_call",
+    )
+
+
+for _name, _call in (
+    ("alarm_control_panel.pending", "pending()"),
+    ("alarm_control_panel.triggered", "triggered()"),
+):
+    automation.register_apply_action(
+        _name,
+        ALARM_CONTROL_PANEL_ACTION_SCHEMA,
+        automation.ApplyCall(_call),
+        call="make_call",
+    )
+
+
+for _name in ("alarm_control_panel.ready", "alarm_control_panel.is_armed"):
+    automation.register_apply_condition(
+        _name, ALARM_CONTROL_PANEL_CONDITION_SCHEMA, "is_armed_pending_or_triggered()"
+    )
 
 
 @coroutine_with_priority(CoroPriority.CORE)

esphome/components/alarm_control_panel/alarm_control_panel.h

@@ -130,6 +130,12 @@ class AlarmControlPanel : public EntityBase {
 
   // is the state one of the armed states
   bool is_state_armed(AlarmControlPanelState state);
+  /// Armed, pending (entry delay) or triggered; not ARMING (exit delay). Backs the is_armed and ready
+  /// conditions.
+  bool is_armed_pending_or_triggered() {
+    auto state = this->get_state();
+    return this->is_state_armed(state) || state == ACP_STATE_PENDING || state == ACP_STATE_TRIGGERED;
+  }
 
  protected:
   friend AlarmControlPanelCall;
@@ -138,11 +144,11 @@ class AlarmControlPanel : public EntityBase {
   // in order to store last panel state in flash
   ESPPreferenceObject pref_;
   // current state
-  AlarmControlPanelState current_state_;
+  AlarmControlPanelState current_state_{ACP_STATE_DISARMED};
   // the desired (or previous) state
-  AlarmControlPanelState desired_state_;
+  AlarmControlPanelState desired_state_{ACP_STATE_DISARMED};
   // last time the state was updated
-  uint32_t last_update_;
+  uint32_t last_update_{0};
   // the call control function
   virtual void control(const AlarmControlPanelCall &call) = 0;
   // state callback - passes the new state to listeners

esphome/components/alarm_control_panel/automation.h

@@ -27,84 +27,4 @@ static_assert(std::is_trivially_copyable_v<StateAnyForwarder>);
 static_assert(sizeof(StateEnterForwarder<ACP_STATE_TRIGGERED>) <= sizeof(void *));
 static_assert(std::is_trivially_copyable_v<StateEnterForwarder<ACP_STATE_TRIGGERED>>);
 
-template<typename... Ts> class ArmAwayAction final : public Action<Ts...> {
- public:
-  explicit ArmAwayAction(AlarmControlPanel *alarm_control_panel) : alarm_control_panel_(alarm_control_panel) {}
-
-  TEMPLATABLE_VALUE(std::string, code)
-
-  void play(const Ts &...x) override { this->alarm_control_panel_->arm_away(this->code_.optional_value(x...)); }
-
- protected:
-  AlarmControlPanel *alarm_control_panel_;
-};
-
-template<typename... Ts> class ArmHomeAction final : public Action<Ts...> {
- public:
-  explicit ArmHomeAction(AlarmControlPanel *alarm_control_panel) : alarm_control_panel_(alarm_control_panel) {}
-
-  TEMPLATABLE_VALUE(std::string, code)
-
-  void play(const Ts &...x) override { this->alarm_control_panel_->arm_home(this->code_.optional_value(x...)); }
-
- protected:
-  AlarmControlPanel *alarm_control_panel_;
-};
-
-template<typename... Ts> class ArmNightAction final : public Action<Ts...> {
- public:
-  explicit ArmNightAction(AlarmControlPanel *alarm_control_panel) : alarm_control_panel_(alarm_control_panel) {}
-
-  TEMPLATABLE_VALUE(std::string, code)
-
-  void play(const Ts &...x) override { this->alarm_control_panel_->arm_night(this->code_.optional_value(x...)); }
-
- protected:
-  AlarmControlPanel *alarm_control_panel_;
-};
-
-template<typename... Ts> class DisarmAction final : public Action<Ts...> {
- public:
-  explicit DisarmAction(AlarmControlPanel *alarm_control_panel) : alarm_control_panel_(alarm_control_panel) {}
-
-  TEMPLATABLE_VALUE(std::string, code)
-
-  void play(const Ts &...x) override { this->alarm_control_panel_->disarm(this->code_.optional_value(x...)); }
-
- protected:
-  AlarmControlPanel *alarm_control_panel_;
-};
-
-template<typename... Ts> class PendingAction final : public Action<Ts...> {
- public:
-  explicit PendingAction(AlarmControlPanel *alarm_control_panel) : alarm_control_panel_(alarm_control_panel) {}
-
-  void play(const Ts &...x) override { this->alarm_control_panel_->make_call().pending().perform(); }
-
- protected:
-  AlarmControlPanel *alarm_control_panel_;
-};
-
-template<typename... Ts> class TriggeredAction final : public Action<Ts...> {
- public:
-  explicit TriggeredAction(AlarmControlPanel *alarm_control_panel) : alarm_control_panel_(alarm_control_panel) {}
-
-  void play(const Ts &...x) override { this->alarm_control_panel_->make_call().triggered().perform(); }
-
- protected:
-  AlarmControlPanel *alarm_control_panel_;
-};
-
-template<typename... Ts> class AlarmControlPanelCondition final : public Condition<Ts...> {
- public:
-  AlarmControlPanelCondition(AlarmControlPanel *parent) : parent_(parent) {}
-  bool check(const Ts &...x) override {
-    return this->parent_->is_state_armed(this->parent_->get_state()) ||
-           this->parent_->get_state() == ACP_STATE_PENDING || this->parent_->get_state() == ACP_STATE_TRIGGERED;
-  }
-
- protected:
-  AlarmControlPanel *parent_;
-};
-
 }  // namespace esphome::alarm_control_panel

esphome/components/alpha3/sensor.py

@@ -74,26 +74,10 @@ async def to_code(config: ConfigType) -> None:
     await cg.register_component(var, config)
     await ble_client.register_ble_node(var, config)
 
-    if flow_config := config.get(CONF_FLOW):
-        sens = await sensor.new_sensor(flow_config)
-        cg.add(var.set_flow_sensor(sens))
-
-    if head_config := config.get(CONF_HEAD):
-        sens = await sensor.new_sensor(head_config)
-        cg.add(var.set_head_sensor(sens))
-
-    if power_config := config.get(CONF_POWER):
-        sens = await sensor.new_sensor(power_config)
-        cg.add(var.set_power_sensor(sens))
-
-    if current_config := config.get(CONF_CURRENT):
-        sens = await sensor.new_sensor(current_config)
-        cg.add(var.set_current_sensor(sens))
-
-    if speed_config := config.get(CONF_SPEED):
-        sens = await sensor.new_sensor(speed_config)
-        cg.add(var.set_speed_sensor(sens))
-
-    if voltage_config := config.get(CONF_VOLTAGE):
-        sens = await sensor.new_sensor(voltage_config)
-        cg.add(var.set_voltage_sensor(sens))
+    sensors = sensor.sub_sensors(config)
+    await sensors(CONF_FLOW, var.set_flow_sensor)
+    await sensors(CONF_HEAD, var.set_head_sensor)
+    await sensors(CONF_POWER, var.set_power_sensor)
+    await sensors(CONF_CURRENT, var.set_current_sensor)
+    await sensors(CONF_SPEED, var.set_speed_sensor)
+    await sensors(CONF_VOLTAGE, var.set_voltage_sensor)

esphome/components/am2315c/sensor.py

@@ -46,10 +46,6 @@ async def to_code(config: ConfigType) -> None:
     await cg.register_component(var, config)
     await i2c.register_i2c_device(var, config)
 
-    if temperature_config := config.get(CONF_TEMPERATURE):
-        sens = await sensor.new_sensor(temperature_config)
-        cg.add(var.set_temperature_sensor(sens))
-
-    if humidity_config := config.get(CONF_HUMIDITY):
-        sens = await sensor.new_sensor(humidity_config)
-        cg.add(var.set_humidity_sensor(sens))
+    sensors = sensor.sub_sensors(config)
+    await sensors(CONF_TEMPERATURE, var.set_temperature_sensor)
+    await sensors(CONF_HUMIDITY, var.set_humidity_sensor)

esphome/components/am2320/sensor.py

@@ -48,10 +48,6 @@ async def to_code(config: ConfigType) -> None:
     await cg.register_component(var, config)
     await i2c.register_i2c_device(var, config)
 
-    if temperature_config := config.get(CONF_TEMPERATURE):
-        sens = await sensor.new_sensor(temperature_config)
-        cg.add(var.set_temperature_sensor(sens))
-
-    if humidity_config := config.get(CONF_HUMIDITY):
-        sens = await sensor.new_sensor(humidity_config)
-        cg.add(var.set_humidity_sensor(sens))
+    sensors = sensor.sub_sensors(config)
+    await sensors(CONF_TEMPERATURE, var.set_temperature_sensor)
+    await sensors(CONF_HUMIDITY, var.set_humidity_sensor)

esphome/components/am43/sensor/__init__.py

@@ -48,10 +48,6 @@ async def to_code(config: ConfigType) -> None:
     await cg.register_component(var, config)
     await ble_client.register_ble_node(var, config)
 
-    if battery_level_config := config.get(CONF_BATTERY_LEVEL):
-        sens = await sensor.new_sensor(battery_level_config)
-        cg.add(var.set_battery(sens))
-
-    if illuminance_config := config.get(CONF_ILLUMINANCE):
-        sens = await sensor.new_sensor(illuminance_config)
-        cg.add(var.set_illuminance(sens))
+    sensors = sensor.sub_sensors(config)
+    await sensors(CONF_BATTERY_LEVEL, var.set_battery)
+    await sensors(CONF_ILLUMINANCE, var.set_illuminance)

esphome/components/animation/animation.h

@@ -1,8 +1,6 @@
 #pragma once
 #include "esphome/components/image/image.h"
 
-#include "esphome/core/automation.h"
-
 namespace esphome::animation {
 
 class Animation final : public image::Image {
@@ -35,32 +33,4 @@ class Animation final : public image::Image {
   int loop_current_iteration_;
 };
 
-template<typename... Ts> class AnimationNextFrameAction final : public Action<Ts...> {
- public:
-  AnimationNextFrameAction(Animation *parent) : parent_(parent) {}
-  void play(const Ts &...x) override { this->parent_->next_frame(); }
-
- protected:
-  Animation *parent_;
-};
-
-template<typename... Ts> class AnimationPrevFrameAction final : public Action<Ts...> {
- public:
-  AnimationPrevFrameAction(Animation *parent) : parent_(parent) {}
-  void play(const Ts &...x) override { this->parent_->prev_frame(); }
-
- protected:
-  Animation *parent_;
-};
-
-template<typename... Ts> class AnimationSetFrameAction final : public Action<Ts...> {
- public:
-  AnimationSetFrameAction(Animation *parent) : parent_(parent) {}
-  TEMPLATABLE_VALUE(uint16_t, frame)
-  void play(const Ts &...x) override { this->parent_->set_frame(this->frame_.value(x...)); }
-
- protected:
-  Animation *parent_;
-};
-
 }  // namespace esphome::animation

esphome/components/animation/image.py

@@ -6,8 +6,6 @@ from esphome.components.file.image import image_schema, write_image
 from esphome.components.image import Image_, validate_settings
 import esphome.config_validation as cv
 from esphome.const import CONF_ID, CONF_REPEAT
-from esphome.core import ID
-from esphome.cpp_generator import MockObj, TemplateArgsType
 from esphome.types import ConfigType
 
 CODEOWNERS = ["@syndlex"]
@@ -26,17 +24,6 @@ animation_ns = cg.esphome_ns.namespace("animation")
 
 Animation_ = animation_ns.class_("Animation", Image_)
 
-# Actions
-NextFrameAction = animation_ns.class_(
-    "AnimationNextFrameAction", automation.Action, cg.Parented.template(Animation_)
-)
-PrevFrameAction = animation_ns.class_(
-    "AnimationPrevFrameAction", automation.Action, cg.Parented.template(Animation_)
-)
-SetFrameAction = animation_ns.class_(
-    "AnimationSetFrameAction", automation.Action, cg.Parented.template(Animation_)
-)
-
 ANIMATION_SCHEMA = image_schema(Animation_).extend(
     {
         cv.Optional(CONF_LOOP): cv.All(
@@ -72,28 +59,17 @@ SET_FRAME_SCHEMA = cv.Schema(
 )
 
 
-@automation.register_action(
-    "animation.next_frame", NextFrameAction, NEXT_FRAME_SCHEMA, synchronous=True
+automation.register_apply_action(
+    "animation.next_frame", NEXT_FRAME_SCHEMA, automation.ApplyCall("next_frame()")
 )
-@automation.register_action(
-    "animation.prev_frame", PrevFrameAction, PREV_FRAME_SCHEMA, synchronous=True
+automation.register_apply_action(
+    "animation.prev_frame", PREV_FRAME_SCHEMA, automation.ApplyCall("prev_frame()")
 )
-@automation.register_action(
-    "animation.set_frame", SetFrameAction, SET_FRAME_SCHEMA, synchronous=True
+automation.register_apply_action(
+    "animation.set_frame",
+    SET_FRAME_SCHEMA,
+    automation.ApplyField(CONF_FRAME, "set_frame", cg.uint16),
 )
-async def animation_action_to_code(
-    config: ConfigType,
-    action_id: ID,
-    template_arg: cg.TemplateArguments,
-    args: TemplateArgsType,
-) -> MockObj:
-    paren = await cg.get_variable(config[CONF_ID])
-    var = cg.new_Pvariable(action_id, template_arg, paren)
-
-    if (frame := config.get(CONF_FRAME)) is not None:
-        template_ = await cg.templatable(frame, args, cg.uint16)
-        cg.add(var.set_frame(template_))
-    return var
 
 
 async def setup_animation(config: ConfigType) -> None:

esphome/components/api/__init__.py

@@ -1,3 +1,4 @@
+from ipaddress import IPv4Address, IPv6Address
 import logging
 import re
 from typing import Any
@@ -5,7 +6,7 @@ from typing import Any
 from esphome import automation
 from esphome.automation import Condition
 import esphome.codegen as cg
-from esphome.components.const import CONF_DESCRIPTION
+from esphome.components.const import CONF_DESCRIPTION, CONF_HOST
 from esphome.components.logger import request_log_listener
 
 # ENCRYPTION_SCHEMA and validate_encryption_key are re-exported for external
@@ -13,6 +14,7 @@ from esphome.components.logger import request_log_listener
 from esphome.components.noise import (  # noqa: F401
     ENCRYPTION_SCHEMA,
     decode_encryption_key,
+    enable_spare_ephemeral,
     encryption_schema,
     new_psk_progmem,
     validate_encryption_key,
@@ -25,6 +27,8 @@ from esphome.const import (
     CONF_CAPTURE_RESPONSE,
     CONF_DATA,
     CONF_DATA_TEMPLATE,
+    CONF_DELAY,
+    CONF_ENABLE_IPV6,
     CONF_ENCRYPTION,
     CONF_EVENT,
     CONF_ID,
@@ -47,10 +51,14 @@ from esphome.const import (
     CONF_VARIABLES,
 )
 from esphome.core import CORE, ID, CoroPriority, EsphomeError, coroutine_with_priority
-from esphome.cpp_generator import MockObj, TemplateArgsType
-from esphome.helpers import fnv1_hash
+from esphome.cpp_generator import Expression, MockObj, TemplateArgsType
+import esphome.final_validate as fv
+from esphome.helpers import cpp_string_escape, fnv1_hash
+from esphome.schema_extractors import SCHEMA_EXTRACT, schema_extractor
 from esphome.types import ConfigFragmentType, ConfigType
 
+from . import wizard
+
 # Compat alias: downstream consumers (e.g. device-builder) referenced the
 # schema by its old private name before it moved to the noise component
 _encryption_schema = encryption_schema
@@ -81,10 +89,11 @@ def AUTO_LOAD(config: ConfigType) -> list[str]:
 
 
 api_ns = cg.esphome_ns.namespace("api")
-APIServer = api_ns.class_("APIServer", cg.Component, cg.Controller)
+APIServer = api_ns.class_("APIServer", cg.Component)
 HomeAssistantServiceCallAction = api_ns.class_(
     "HomeAssistantServiceCallAction", automation.Action
 )
+HomeAssistantField = api_ns.struct("HomeAssistantField")
 ActionResponse = api_ns.class_("ActionResponse")
 HomeAssistantActionResponseTrigger = api_ns.class_(
     "HomeAssistantActionResponseTrigger", automation.Trigger
@@ -134,8 +143,16 @@ CONF_HOMEASSISTANT_SERVICES = "homeassistant_services"
 CONF_HOMEASSISTANT_STATES = "homeassistant_states"
 CONF_LISTEN_BACKLOG = "listen_backlog"
 CONF_MAX_SEND_QUEUE = "max_send_queue"
+CONF_OUTGOING_CONNECTION = "outgoing_connection"
 CONF_STATE_SUBSCRIPTION_ONLY = "state_subscription_only"
 
+# Schema defaults that also match the C++ initializers in api_server.h; codegen
+# skips the setter when the config equals them.
+DEFAULT_PORT = 6053
+DEFAULT_REBOOT_TIMEOUT = "15min"
+DEFAULT_BATCH_DELAY = "100ms"
+DEFAULT_LISTEN_BACKLOG = 4
+
 
 def _register_provisioning_source(config: ConfigType) -> ConfigType:
     """Register the API as a provisioning source when encryption is enabled.
@@ -285,14 +302,77 @@ def _consume_api_sockets(config: ConfigType) -> ConfigType:
     # (not max_connections, which is the upper limit rarely reached)
     socket.consume_sockets(3, "api")(config)
     socket.consume_sockets(1, "api", socket.SocketType.TCP_LISTEN)(config)
+    if CONF_OUTGOING_CONNECTION in config:
+        socket.consume_sockets(1, "api_outgoing_connection")(config)
     return config
 
 
+def _validate_outgoing_connection(config: ConfigType) -> ConfigType:
+    if (outgoing := config.get(CONF_OUTGOING_CONNECTION)) is None:
+        return config
+    if CONF_ENCRYPTION not in config:
+        raise cv.Invalid(
+            "outgoing_connection requires 'encryption' so the peer is verified by key",
+            path=[CONF_OUTGOING_CONNECTION],
+        )
+    # A device with no client reboots once reboot_timeout passes, so a delay that
+    # reaches it would reboot the device before it ever dials
+    reboot_timeout = config[CONF_REBOOT_TIMEOUT]
+    delay = outgoing[CONF_DELAY]
+    if reboot_timeout.total_milliseconds and delay >= reboot_timeout:
+        raise cv.Invalid(
+            f"delay must be shorter than reboot_timeout ({reboot_timeout}), "
+            "otherwise the device reboots before it dials",
+            path=[CONF_OUTGOING_CONNECTION, CONF_DELAY],
+        )
+    return config
+
+
+def _validate_outgoing_host(value: str) -> IPv4Address | IPv6Address:
+    """Only accept an address the device itself can parse.
+
+    Python accepts a scope id, which neither `inet_pton` nor lwIP's `inet6_aton`
+    takes, and a v4-mapped address is dialed as plain IPv4, needing no IPv6 build.
+    """
+    address = cv.ipaddress(value)
+    if isinstance(address, IPv6Address):
+        if address.scope_id is not None:
+            raise cv.Invalid(
+                f"{value} carries a scope id, which the device cannot parse; "
+                "give the address without the '%' part"
+            )
+        if (mapped := address.ipv4_mapped) is not None:
+            return mapped
+    return address
+
+
+_OUTGOING_CONNECTION_SCHEMA = cv.Schema(
+    {
+        cv.Optional(CONF_HOST): _validate_outgoing_host,
+        cv.Optional(CONF_PORT, default=6054): cv.port,
+        # Bounded against reboot_timeout in _validate_outgoing_connection
+        cv.Optional(CONF_DELAY, default="60s"): cv.positive_time_period_milliseconds,
+    }
+)
+
+
+@schema_extractor("schema")
+def _outgoing_connection_schema(config: ConfigType | None) -> ConfigType:
+    # A bare `outgoing_connection:` block is valid; without a host the device
+    # dials the remembered last dial-back client
+    if config is SCHEMA_EXTRACT:
+        # Let the language-schema dumper walk host, port and delay
+        return _OUTGOING_CONNECTION_SCHEMA
+    if config is None:
+        config = {}
+    return _OUTGOING_CONNECTION_SCHEMA(config)
+
+
 CONFIG_SCHEMA = cv.All(
     cv.Schema(
         {
             cv.GenerateID(): cv.declare_id(APIServer),
-            cv.Optional(CONF_PORT, default=6053): cv.port,
+            cv.Optional(CONF_PORT, default=DEFAULT_PORT): cv.port,
             # Removed in 2026.1.0 - kept to provide helpful error message
             cv.Optional(CONF_PASSWORD): cv.invalid(
                 "The 'password' option has been removed in ESPHome 2026.1.0.\n"
@@ -305,14 +385,16 @@ CONFIG_SCHEMA = cv.All(
                 "Or visit https://esphome.io/components/api/#configuration-variables"
             ),
             cv.Optional(
-                CONF_REBOOT_TIMEOUT, default="15min"
+                CONF_REBOOT_TIMEOUT, default=DEFAULT_REBOOT_TIMEOUT
             ): cv.positive_time_period_milliseconds,
             cv.Exclusive(
                 CONF_SERVICES, group_of_exclusion=CONF_ACTIONS
             ): ACTIONS_SCHEMA,
             cv.Exclusive(CONF_ACTIONS, group_of_exclusion=CONF_ACTIONS): ACTIONS_SCHEMA,
             cv.Optional(CONF_ENCRYPTION): encryption_schema,
-            cv.Optional(CONF_BATCH_DELAY, default="100ms"): cv.All(
+            cv.Optional(wizard.CONF_WIZARD): wizard.WIZARD_SCHEMA,
+            cv.Optional(CONF_OUTGOING_CONNECTION): _outgoing_connection_schema,
+            cv.Optional(CONF_BATCH_DELAY, default=DEFAULT_BATCH_DELAY): cv.All(
                 cv.positive_time_period_milliseconds,
                 cv.Range(max=cv.TimePeriod(milliseconds=65535)),
             ),
@@ -350,10 +432,9 @@ CONFIG_SCHEMA = cv.All(
                 ln882x=5,  # Moderate RAM
                 nrf52=4,  # ~256KB RAM, BSD sockets, Thread (single HA controller)
             ): cv.int_range(min=1, max=20),
-            # Maximum queued send buffers per connection before dropping connection
-            # Each buffer uses ~8-12 bytes overhead plus actual message size
+            # Max queued messages per connection, and 2 KB of backlog per slot up
+            # to 64 KB (a lone message is exempt), before the connection is dropped
             # Platform defaults based on available RAM and typical message rates:
-            # CONF_MAX_SEND_QUEUE defaults are power of 2 for efficient modulo
             cv.SplitDefault(
                 CONF_MAX_SEND_QUEUE,
                 esp8266=4,  # Limited RAM, need to fail fast
@@ -368,6 +449,7 @@ CONFIG_SCHEMA = cv.All(
         }
     ).extend(cv.COMPONENT_SCHEMA),
     cv.rename_key(CONF_SERVICES, CONF_ACTIONS),
+    _validate_outgoing_connection,
     _consume_api_sockets,
     _register_provisioning_source,
 )
@@ -424,7 +506,34 @@ def _validate_esp8266_action_strings(config: ConfigType) -> ConfigType:
     return config
 
 
-FINAL_VALIDATE_SCHEMA = _validate_esp8266_action_strings
+def _validate_outgoing_host_ipv6(config: ConfigType) -> ConfigType:
+    """An IPv6 host can never be parsed, so never dialed, without IPv6."""
+    if (
+        (outgoing := config.get(CONF_OUTGOING_CONNECTION)) is None
+        or (host := outgoing.get(CONF_HOST)) is None
+        or host.version != 6
+    ):
+        return config
+    network_conf = fv.full_config.get().get("network") or {}
+    if not network_conf.get(CONF_ENABLE_IPV6):
+        raise cv.Invalid(
+            "outgoing_connection host is an IPv6 address but IPv6 is not "
+            "enabled; set 'network: enable_ipv6: true'",
+            path=[CONF_OUTGOING_CONNECTION, CONF_HOST],
+        )
+    return config
+
+
+def _validate_wizard(config: ConfigType) -> ConfigType:
+    wizard.final_validate(config)
+    return config
+
+
+FINAL_VALIDATE_SCHEMA = cv.All(
+    _validate_esp8266_action_strings,
+    _validate_outgoing_host_ipv6,
+    _validate_wizard,
+)
 
 
 def _add_action_strings(
@@ -457,17 +566,24 @@ async def to_code(config: ConfigType) -> None:
     var = cg.new_Pvariable(config[CONF_ID])
     await cg.register_component(var, config)
 
-    # Track controller registration for StaticVector sizing
-    CORE.register_controller()
+    CORE.register_controller(var)
 
     # Request a log listener slot for API log streaming
     request_log_listener()
 
-    cg.add(var.set_port(config[CONF_PORT]))
-    cg.add(var.set_reboot_timeout(config[CONF_REBOOT_TIMEOUT]))
-    cg.add(var.set_batch_delay(config[CONF_BATCH_DELAY]))
-    if CONF_LISTEN_BACKLOG in config:
-        cg.add(var.set_listen_backlog(config[CONF_LISTEN_BACKLOG]))
+    # Skip the setters when the config matches the C++ initializers (DEFAULT_*).
+    if (port := config[CONF_PORT]) != DEFAULT_PORT:
+        cg.add(var.set_port(port))
+    if (reboot_timeout := config[CONF_REBOOT_TIMEOUT]) != cv.time_period(
+        DEFAULT_REBOOT_TIMEOUT
+    ):
+        cg.add(var.set_reboot_timeout(reboot_timeout))
+    if (batch_delay := config[CONF_BATCH_DELAY]) != cv.time_period(DEFAULT_BATCH_DELAY):
+        cg.add(var.set_batch_delay(batch_delay))
+    if (
+        listen_backlog := config.get(CONF_LISTEN_BACKLOG)
+    ) is not None and listen_backlog != DEFAULT_LISTEN_BACKLOG:
+        cg.add(var.set_listen_backlog(listen_backlog))
     cg.add_define("MAX_API_CONNECTIONS", config[CONF_MAX_CONNECTIONS])
     cg.add_define("API_MAX_SEND_QUEUE", config[CONF_MAX_SEND_QUEUE])
 
@@ -572,6 +688,9 @@ async def to_code(config: ConfigType) -> None:
         # Stack buffer that list-entities copies PROGMEM strings into, sized for the largest action
         cg.add_define("API_USER_ACTION_STRINGS_SCRATCH_SIZE", max(scratch_size, 1))
 
+    if (wizard_config := config.get(wizard.CONF_WIZARD)) is not None:
+        await wizard.to_code(wizard_config)
+
     if CONF_ON_CLIENT_CONNECTED in config:
         cg.add_define("USE_API_CLIENT_CONNECTED_TRIGGER")
         await automation.build_automation(
@@ -590,7 +709,7 @@ async def to_code(config: ConfigType) -> None:
 
     if (encryption_config := config.get(CONF_ENCRYPTION, None)) is not None:
         if key := encryption_config.get(CONF_KEY):
-            cg.add(var.set_noise_psk(new_psk_progmem(config[CONF_ID], key)))
+            cg.add(var.set_noise_psk(new_psk_progmem(key)))
             cg.add_define("USE_API_NOISE_PSK_FROM_YAML")
         else:
             # No key provided, but encryption desired
@@ -603,9 +722,17 @@ async def to_code(config: ConfigType) -> None:
             # and plaintext disabled. Only a factory reset can remove it.
             cg.add_define("USE_API_PLAINTEXT")
         cg.add_define("USE_API_NOISE")
+        enable_spare_ephemeral()
     else:
         cg.add_define("USE_API_PLAINTEXT")
 
+    if (outgoing := config.get(CONF_OUTGOING_CONNECTION)) is not None:
+        cg.add_define("USE_API_OUTGOING_CONNECTION")
+        if (host := outgoing.get(CONF_HOST)) is not None:
+            cg.add_define("API_OUTGOING_CONNECTION_HOST", str(host))
+        cg.add_define("API_OUTGOING_CONNECTION_PORT", outgoing[CONF_PORT])
+        cg.add_define("API_OUTGOING_CONNECTION_DELAY", outgoing[CONF_DELAY])
+
     cg.add_define("USE_API")
     cg.add_global(api_ns.using)
 
@@ -646,6 +773,11 @@ VARIABLES_SCHEMA = cv.Schema(
     {cv.string: cv.All(_coerce_implicit_lambda, cv.templatable(cv.string_strict))}
 )
 
+# The action stores each map's entry count in a uint8_t
+_FIELD_MAP_MAX = 255
+DATA_FIELDS_SCHEMA = cv.All(KEY_VALUE_SCHEMA, cv.Length(max=_FIELD_MAP_MAX))
+VARIABLES_FIELDS_SCHEMA = cv.All(VARIABLES_SCHEMA, cv.Length(max=_FIELD_MAP_MAX))
+
 
 def _validate_response_config(config: ConfigType) -> ConfigType:
     # Validate dependencies:
@@ -680,9 +812,9 @@ HOMEASSISTANT_ACTION_ACTION_SCHEMA = cv.All(
             cv.Exclusive(CONF_ACTION, group_of_exclusion=CONF_ACTION): cv.templatable(
                 cv.string
             ),
-            cv.Optional(CONF_DATA, default={}): KEY_VALUE_SCHEMA,
-            cv.Optional(CONF_DATA_TEMPLATE, default={}): KEY_VALUE_SCHEMA,
-            cv.Optional(CONF_VARIABLES, default={}): VARIABLES_SCHEMA,
+            cv.Optional(CONF_DATA, default={}): DATA_FIELDS_SCHEMA,
+            cv.Optional(CONF_DATA_TEMPLATE, default={}): DATA_FIELDS_SCHEMA,
+            cv.Optional(CONF_VARIABLES, default={}): VARIABLES_FIELDS_SCHEMA,
             cv.Optional(CONF_RESPONSE_TEMPLATE): cv.templatable(cv.string),
             cv.Optional(CONF_CAPTURE_RESPONSE, default=False): cv.boolean,
             cv.Optional(CONF_ON_SUCCESS): automation.validate_automation(single=True),
@@ -695,6 +827,62 @@ HOMEASSISTANT_ACTION_ACTION_SCHEMA = cv.All(
 )
 
 
+def _field_string(value: str) -> Expression:
+    # ESP8266 can only keep a string in flash as its own PROGMEM array
+    literal = cg.RawExpression(cpp_string_escape(value))
+    if CORE.is_esp8266:
+        return cg.shared_progmem_array("ha_field_str", cg.char, literal)
+    return literal
+
+
+async def _new_service_call_action(
+    server_id: ID,
+    action_id: ID,
+    template_arg: cg.TemplateArguments,
+    args: TemplateArgsType,
+    is_event: bool,
+    service: Any,
+    data: dict[str, Any],
+    data_template: dict[str, Any],
+    variables: dict[str, Any],
+) -> MockObj:
+    """Create the action with its name and fields in one shared flash table."""
+    cg.add_define("USE_API_HOMEASSISTANT_SERVICES")
+    serv = await cg.get_variable(server_id)
+    field_type = HomeAssistantField.template(template_arg)
+    groups = [data, data_template, variables]
+    # A lambda may keep static state, so a table with lambdas is never shared
+    has_lambda = cg.is_template(service) or any(
+        cg.is_template(value) for group in groups for value in group.values()
+    )
+
+    async def field(key: str | None, value: Any, output_type: Any = None) -> Expression:
+        key_exp = cg.nullptr if key is None else _field_string(key)
+        if cg.is_template(value):
+            # output_type=None lets lambdas return numbers or char pointers; C++ converts them
+            lam = await cg.process_lambda(value, args, return_type=output_type)
+            return cg.RawExpression(f"{field_type}::from_lambda({key_exp}, {lam})")
+        return cg.ArrayInitializer(key_exp, _field_string(value), cg.nullptr)
+
+    entries = [await field(None, service, cg.std_string)]
+    for group in groups:
+        for key, value in group.items():
+            entries.append(await field(key, value))
+    table = cg.shared_progmem_array(
+        "ha_action_fields",
+        field_type,
+        cg.ArrayInitializer(*entries, multiline=True),
+        share=not has_lambda,
+    )
+    return cg.new_Pvariable(
+        action_id, template_arg, serv, is_event, table, *(len(g) for g in groups)
+    )
+
+
+def _service_call_fields(config: ConfigType) -> tuple[dict[str, Any], ...]:
+    return config[CONF_DATA], config[CONF_DATA_TEMPLATE], config[CONF_VARIABLES]
+
+
 # synchronous=False: when on_success/on_error is configured, play() stores the
 # trigger args until the HomeassistantActionResponse arrives, so non-owning args
 # (StringRef into the API receive buffer) must not be used.
@@ -716,36 +904,15 @@ async def homeassistant_service_to_code(
     template_arg: cg.TemplateArguments,
     args: TemplateArgsType,
 ) -> MockObj:
-    cg.add_define("USE_API_HOMEASSISTANT_SERVICES")
-    serv = await cg.get_variable(config[CONF_ID])
-    var = cg.new_Pvariable(action_id, template_arg, serv, False)
-    templ = await cg.templatable(config[CONF_ACTION], args, cg.std_string)
-    cg.add(var.set_service(templ))
-
-    # Initialize FixedVectors with exact sizes from config
-    cg.add(var.init_data(len(config[CONF_DATA])))
-    for key, value in config[CONF_DATA].items():
-        # output_type=None because lambdas can return non-string types (int,
-        # float, char*) that TemplatableStringValue converts via to_string.
-        # Static strings are manually wrapped for PROGMEM on ESP8266.
-        templ = await cg.templatable(value, args, None)
-        if isinstance(templ, str):
-            templ = cg.FlashStringLiteral(templ)
-        cg.add(var.add_data(cg.FlashStringLiteral(key), templ))
-
-    cg.add(var.init_data_template(len(config[CONF_DATA_TEMPLATE])))
-    for key, value in config[CONF_DATA_TEMPLATE].items():
-        templ = await cg.templatable(value, args, None)
-        if isinstance(templ, str):
-            templ = cg.FlashStringLiteral(templ)
-        cg.add(var.add_data_template(cg.FlashStringLiteral(key), templ))
-
-    cg.add(var.init_variables(len(config[CONF_VARIABLES])))
-    for key, value in config[CONF_VARIABLES].items():
-        templ = await cg.templatable(value, args, None)
-        if isinstance(templ, str):
-            templ = cg.FlashStringLiteral(templ)
-        cg.add(var.add_variable(cg.FlashStringLiteral(key), templ))
+    var = await _new_service_call_action(
+        config[CONF_ID],
+        action_id,
+        template_arg,
+        args,
+        False,
+        config[CONF_ACTION],
+        *_service_call_fields(config),
+    )
 
     if on_error := config.get(CONF_ON_ERROR):
         cg.add_define("USE_API_HOMEASSISTANT_ACTION_RESPONSES")
@@ -797,9 +964,9 @@ HOMEASSISTANT_EVENT_ACTION_SCHEMA = cv.Schema(
     {
         cv.GenerateID(): cv.use_id(APIServer),
         cv.Required(CONF_EVENT): validate_homeassistant_event,
-        cv.Optional(CONF_DATA, default={}): KEY_VALUE_SCHEMA,
-        cv.Optional(CONF_DATA_TEMPLATE, default={}): KEY_VALUE_SCHEMA,
-        cv.Optional(CONF_VARIABLES, default={}): VARIABLES_SCHEMA,
+        cv.Optional(CONF_DATA, default={}): DATA_FIELDS_SCHEMA,
+        cv.Optional(CONF_DATA_TEMPLATE, default={}): DATA_FIELDS_SCHEMA,
+        cv.Optional(CONF_VARIABLES, default={}): VARIABLES_FIELDS_SCHEMA,
     }
 )
 
@@ -818,38 +985,15 @@ async def homeassistant_event_to_code(
     template_arg: cg.TemplateArguments,
     args: TemplateArgsType,
 ) -> MockObj:
-    cg.add_define("USE_API_HOMEASSISTANT_SERVICES")
-    serv = await cg.get_variable(config[CONF_ID])
-    var = cg.new_Pvariable(action_id, template_arg, serv, True)
-    templ = await cg.templatable(config[CONF_EVENT], args, cg.std_string)
-    cg.add(var.set_service(templ))
-
-    # Initialize FixedVectors with exact sizes from config
-    cg.add(var.init_data(len(config[CONF_DATA])))
-    for key, value in config[CONF_DATA].items():
-        # output_type=None because lambdas can return non-string types (int,
-        # float, char*) that TemplatableStringValue converts via to_string.
-        # Static strings are manually wrapped for PROGMEM on ESP8266.
-        templ = await cg.templatable(value, args, None)
-        if isinstance(templ, str):
-            templ = cg.FlashStringLiteral(templ)
-        cg.add(var.add_data(cg.FlashStringLiteral(key), templ))
-
-    cg.add(var.init_data_template(len(config[CONF_DATA_TEMPLATE])))
-    for key, value in config[CONF_DATA_TEMPLATE].items():
-        templ = await cg.templatable(value, args, None)
-        if isinstance(templ, str):
-            templ = cg.FlashStringLiteral(templ)
-        cg.add(var.add_data_template(cg.FlashStringLiteral(key), templ))
-
-    cg.add(var.init_variables(len(config[CONF_VARIABLES])))
-    for key, value in config[CONF_VARIABLES].items():
-        templ = await cg.templatable(value, args, None)
-        if isinstance(templ, str):
-            templ = cg.FlashStringLiteral(templ)
-        cg.add(var.add_variable(cg.FlashStringLiteral(key), templ))
-
-    return var
+    return await _new_service_call_action(
+        config[CONF_ID],
+        action_id,
+        template_arg,
+        args,
+        True,
+        config[CONF_EVENT],
+        *_service_call_fields(config),
+    )
 
 
 HOMEASSISTANT_TAG_SCANNED_ACTION_SCHEMA = cv.maybe_simple_value(
@@ -873,15 +1017,17 @@ async def homeassistant_tag_scanned_to_code(
     template_arg: cg.TemplateArguments,
     args: TemplateArgsType,
 ) -> MockObj:
-    cg.add_define("USE_API_HOMEASSISTANT_SERVICES")
-    serv = await cg.get_variable(config[CONF_ID])
-    var = cg.new_Pvariable(action_id, template_arg, serv, True)
-    cg.add(var.set_service(cg.FlashStringLiteral("esphome.tag_scanned")))
-    # Initialize FixedVector with exact size (1 data field)
-    cg.add(var.init_data(1))
-    templ = await cg.templatable(config[CONF_TAG], args, cg.std_string)
-    cg.add(var.add_data(cg.FlashStringLiteral("tag_id"), templ))
-    return var
+    return await _new_service_call_action(
+        config[CONF_ID],
+        action_id,
+        template_arg,
+        args,
+        True,
+        "esphome.tag_scanned",
+        {"tag_id": config[CONF_TAG]},
+        {},
+        {},
+    )
 
 
 CONF_SUCCESS = "success"
@@ -992,6 +1138,8 @@ _define_filter = filter_source_files_from_defines(
         "user_services.cpp": "USE_API_USER_DEFINED_ACTIONS",
         "api_frame_helper_noise.cpp": "USE_API_NOISE",
         "api_frame_helper_plaintext.cpp": "USE_API_PLAINTEXT",
+        "api_wizard.cpp": "USE_API_WIZARD",
+        "api_outgoing_connection.cpp": "USE_API_OUTGOING_CONNECTION",
     }
 )
 

esphome/components/api/api.proto

@@ -20,6 +20,8 @@ service APIConnection {
     option (needs_authentication) = false;
   }
   rpc device_capabilities (DeviceCapabilitiesRequest) returns (DeviceCapabilitiesResponse) {}
+  rpc device_wizard (DeviceWizardRequest) returns (DeviceWizardResponse) {}
+  rpc wizard_input_set (WizardInputSetRequest) returns (void) {}
   rpc list_entities (ListEntitiesRequest) returns (void) {}
   rpc subscribe_states (SubscribeStatesRequest) returns (void) {}
   rpc subscribe_logs (SubscribeLogsRequest) returns (void) {}
@@ -76,8 +78,11 @@ service APIConnection {
   rpc serial_proxy_write(SerialProxyWriteRequest) returns (void) {}
   rpc serial_proxy_set_modem_pins(SerialProxySetModemPinsRequest) returns (void) {}
   rpc serial_proxy_get_modem_pins(SerialProxyGetModemPinsRequest) returns (void) {}
+  rpc subscribe_serial_proxy_identity(SubscribeSerialProxyIdentityRequest) returns (void) {}
   rpc serial_proxy_request(SerialProxyRequest) returns (void) {}
   rpc serial_proxy_set_mode(SerialProxySetModeRequest) returns (void) {}
+
+  rpc sendspin_pairing_token(SendspinPairingTokenRequest) returns (SendspinPairingTokenResponse) {}
 }
 
 
@@ -113,6 +118,11 @@ message HelloRequest {
   string client_info = 1;
   uint32 api_version_major = 2;
   uint32 api_version_minor = 3;
+
+  // Set by clients that can accept connections the device opens to them
+  // (see api: outgoing_connection:). The device remembers this client's
+  // address as the target to dial when no such client is connected.
+  bool outgoing_connection_target = 4 [(field_ifdef) = "USE_API_OUTGOING_CONNECTION"];
 }
 
 // Confirmation of successful connection request.
@@ -228,6 +238,7 @@ enum SerialProxyPortType {
   SERIAL_PROXY_PORT_TYPE_TTL = 0;
   SERIAL_PROXY_PORT_TYPE_RS232 = 1;
   SERIAL_PROXY_PORT_TYPE_RS485 = 2;
+  SERIAL_PROXY_PORT_TYPE_USB_SERIAL = 3;  // since API 1.18
 }
 
 message SerialProxyInfo {
@@ -332,6 +343,10 @@ message DeviceInfoResponse {
   // all-zeros PSK, so the api encryption key can be provisioned without being
   // sent in plaintext (protects against passive sniffing, not active MITM)
   bool api_encryption_provisionable = 26 [(field_ifdef) = "USE_API_NOISE"];
+
+  // Device is built with the api outgoing_connection option and can open
+  // the TCP connection to a dial-back target itself
+  bool api_outgoing_connection_supported = 27 [(field_ifdef) = "USE_API_OUTGOING_CONNECTION"];
 }
 
 // ==================== DEVICE CAPABILITIES ====================
@@ -380,6 +395,16 @@ message ZWaveProxyCapabilities {
   uint32 home_id = 2;
 }
 
+message WizardCapabilities {
+  // True when the device has a wizard configured, so DeviceWizardRequest will be answered
+  bool configured = 1;
+}
+
+message SendspinCapabilities {
+  // Bitmask of the Sendspin features this device supports
+  uint32 feature_flags = 1;
+}
+
 message DeviceCapabilitiesResponse {
   option (id) = 150;
   option (source) = SOURCE_SERVER;
@@ -389,6 +414,89 @@ message DeviceCapabilitiesResponse {
   ZWaveProxyCapabilities zwave_proxy = 3 [(field_ifdef) = "USE_ZWAVE_PROXY"];
   repeated SerialProxyInfo serial_proxies = 4
       [(field_ifdef) = "USE_SERIAL_PROXY", (fixed_array_size_define) = "SERIAL_PROXY_COUNT"];
+  WizardCapabilities wizard = 5 [(field_ifdef) = "USE_API_WIZARD"];
+  SendspinCapabilities sendspin = 6 [(field_ifdef) = "USE_SENDSPIN"];
+}
+
+// ==================== DEVICE WIZARD ====================
+
+// Asks the device for the onboarding wizard that Home Assistant presents when the
+// device is added.
+//
+// The wizard is read once, when the device is added, whereas capabilities are read
+// on every connect, so it is a separate message rather than part of
+// DeviceCapabilitiesResponse. It is only served on an authenticated connection.
+//
+// A device without a wizard does not have this message at all and never answers it,
+// as a device ignores messages it does not know. Clients therefore check
+// DeviceCapabilitiesResponse.wizard.configured before sending it.
+message DeviceWizardRequest {
+  option (id) = 156;
+  option (source) = SOURCE_CLIENT;
+  option (ifdef) = "USE_API_WIZARD";
+  // Empty
+}
+
+// The wizard, as UTF-8 JSON compressed into a single zstd frame. The device builds it
+// when it is compiled and sends it unchanged, so its size varies with the wizard and
+// its format can grow without new fields here. Clients decompress it, then read:
+//
+//   {"version": 1,
+//    "pages": [{"title": "...", "description": "...",
+//      "entities": [{"key": 123, "device_id": 456, "description": "..."}],
+//      "inputs": [{"key": 789, "description": "...",
+//        "entity_filters": [{"integration": "...", "domain": ["..."],
+//                            "device_class": ["..."], "supported_features": ["..."]}]}]}]}
+//
+// Values that are empty or unset are left out, and so are empty lists. Strings may be
+// Home Assistant translation placeholders such as "[%key:component::domain::name%]",
+// passed through verbatim. The pages are shown in the order listed. A client that does
+// not know the version must not use the wizard.
+//
+// An entity is an entity of this device that the page shows to the user (for example to
+// turn it on or off). Its key is the key ListEntitiesResponse sends for it. Its
+// device_id is the device_id ListEntitiesResponse sends, and is left out when that is 0.
+//
+// An input is filled in by the user with the id of a Home Assistant entity. It either
+// stands for one entity of the device's homeassistant platform (sensor, binary_sensor,
+// text_sensor, number, switch, text, select or button) that sets no entity_id of its own,
+// as Home Assistant supplies it, or it is a standalone input that only stores the entity
+// id for the device's own automations. Its key is the FNV-1 hash
+// of the input's ESPHome id, not an entity key. The client sends the chosen entity id to
+// the device with a WizardInputSetRequest. entity_filters says which Home Assistant
+// entities the user may choose, and mirrors Home Assistant's EntityFilterSelectorConfig:
+// an entity matches a filter when it satisfies every field that is set, and an entity
+// that matches any one of the filters is accepted.
+//
+// The device keeps the entity id of an input in RAM only and does not store it across
+// restarts. The client therefore stores the choice itself and, after every connection
+// and before it sends SubscribeHomeAssistantStatesRequest, sends a WizardInputSetRequest
+// for each input. It sends one again whenever the user changes the choice.
+message DeviceWizardResponse {
+  option (id) = 157;
+  option (source) = SOURCE_SERVER;
+  option (ifdef) = "USE_API_WIZARD";
+
+  // The data is in flash, and the device sends it from there
+  bytes data = 1 [(pointer_to_buffer) = true, (progmem) = true];
+}
+
+// Sets the entity id of a wizard input (see WizardInputField.key). Clients send
+// this only when DeviceCapabilitiesResponse.wizard.configured is set. There is no
+// reply. The device ignores a request whose key matches no input or whose entity_id
+// is empty, longer than 255 bytes or has no '.'. Otherwise it keeps the id in RAM,
+// not across restarts, and when the input belongs to a homeassistant platform entity
+// it subscribes to that entity's state. Clients send this for every input right
+// after connecting, before SubscribeHomeAssistantStatesRequest. A request that
+// arrives later still works: the device then sends the affected state subscriptions
+// again.
+message WizardInputSetRequest {
+  option (id) = 158;
+  option (source) = SOURCE_CLIENT;
+  option (ifdef) = "USE_API_WIZARD_INPUTS";
+
+  fixed32 key = 1;
+  string entity_id = 2 [(max_data_length) = 255];
 }
 
 message ListEntitiesRequest {
@@ -803,6 +911,9 @@ message SwitchStateResponse {
   fixed32 key = 1 [(force) = true];
   bool state = 2;
   uint32 device_id = 3 [(field_ifdef) = "USE_DEVICES"];
+  // If the switch does not have a valid state yet.
+  // Equivalent to `!obj->has_state()` - inverse logic to make state packets smaller
+  bool missing_state = 4;
 }
 message SwitchCommandRequest {
   option (id) = 33;
@@ -1244,6 +1355,9 @@ message ClimateStateResponse {
   float current_humidity = 14;
   float target_humidity = 15;
   uint32 device_id = 16 [(field_ifdef) = "USE_DEVICES"];
+  // If the climate device does not have a valid state yet.
+  // Equivalent to `!obj->has_state()` - inverse logic to make state packets smaller
+  bool missing_state = 17;
 }
 message ClimateCommandRequest {
   option (id) = 48;
@@ -1330,6 +1444,9 @@ message WaterHeaterStateResponse {
   uint32 state = 6;
   float target_temperature_low = 7;
   float target_temperature_high = 8;
+  // If the water heater does not have a valid state yet.
+  // Equivalent to `!obj->has_state()` - inverse logic to make state packets smaller
+  bool missing_state = 9;
 }
 
 // Bitmask for WaterHeaterCommandRequest.has_fields
@@ -1430,7 +1547,7 @@ message ListEntitiesSelectResponse {
   reserved 4; // Deprecated: was string unique_id
 
   string icon = 5 [(field_ifdef) = "USE_ENTITY_ICON", (max_data_length) = 63];
-  repeated string options = 6 [(container_pointer_no_template) = "FixedVector<const char *>"];
+  repeated string options = 6 [(container_pointer_no_template) = "std::span<const char *const>"];
   bool disabled_by_default = 7;
   EntityCategory entity_category = 8;
   uint32 device_id = 9 [(field_ifdef) = "USE_DEVICES"];
@@ -2674,7 +2791,7 @@ message ListEntitiesInfraredResponse {
 message InfraredRFTransmitRawTimingsRequest {
   option (id) = 136;
   option (source) = SOURCE_CLIENT;
-  option (ifdef) = "USE_IR_RF || USE_RADIO_FREQUENCY";
+  option (ifdef) = "USE_IR_RF";
 
   uint32 device_id = 1 [(field_ifdef) = "USE_DEVICES"];
   fixed32 key = 2 [(force) = true];     // Key identifying the transmitter instance
@@ -2688,7 +2805,7 @@ message InfraredRFTransmitRawTimingsRequest {
 message InfraredRFReceiveEvent {
   option (id) = 137;
   option (source) = SOURCE_SERVER;
-  option (ifdef) = "USE_IR_RF || USE_RADIO_FREQUENCY";
+  option (ifdef) = "USE_IR_RF";
   option (no_delay) = true;
   option (speed_optimized) = true;
 
@@ -2697,6 +2814,28 @@ message InfraredRFReceiveEvent {
   repeated sint32 timings = 3 [packed = true, (container_pointer_no_template) = "std::vector<int32_t>"];  // Raw timings in microseconds (zigzag-encoded): alternating mark/space periods
 }
 
+// Sent only to the client that issued an InfraredRFTransmitRawTimingsRequest, once the
+// transmitter reports that the transmission (all repeats) has finished, or immediately with
+// success=false if it was refused before reaching the transmitter (unknown key, no transmitter,
+// no or invalid timings) or the transmitter could not send it (not set up, hardware error).
+// success=false also answers a request superseded by a newer request on the same entity, from
+// any client, and a transmit that reported nothing within 30 s past its expected air time, so a
+// client should not retry on it blindly. The device serializes transmits per transmitter and
+// several entities may share one, so a client should keep at most one transmit outstanding per
+// device, not per entity. A reply for an unknown key or a refused request is sent once and can be lost
+// when the device's send buffer is full, so a client should also stop waiting on its own after the
+// expected air time plus a margin. Lets clients pace requests instead of estimating durations (since API 1.18)
+message InfraredRFTransmitCompleteResponse {
+  option (id) = 153;
+  option (source) = SOURCE_SERVER;
+  option (ifdef) = "USE_IR_RF";
+  option (no_delay) = true;
+
+  uint32 device_id = 1 [(field_ifdef) = "USE_DEVICES"];
+  fixed32 key = 2 [(force) = true];  // Key of the transmitter entity from the request
+  bool success = 3;                  // false if the request was refused, not sent, superseded, or never reported
+}
+
 // ==================== RADIO FREQUENCY ====================
 
 // Lists available radio frequency entity instances
@@ -2867,6 +3006,56 @@ message SerialProxySetModeRequest {
   SerialProxyMode mode = 2;
 }
 
+// Subscribe to the identity of every serial proxy port. The device answers with one
+// SerialProxyIdentity per port, then sends another whenever a port's identity changes,
+// for the life of the connection (since API 1.18).
+message SubscribeSerialProxyIdentityRequest {
+  option (id) = 154;
+  option (source) = SOURCE_CLIENT;
+  option (ifdef) = "USE_SERIAL_PROXY";
+}
+
+// Where a port's identity comes from
+enum SerialProxyIdentitySource {
+  SERIAL_PROXY_IDENTITY_SOURCE_NONE = 0;        // The port carries no identity
+  SERIAL_PROXY_IDENTITY_SOURCE_CONFIGURED = 1;  // Reserved for identities stated in the device configuration; not sent yet
+  SERIAL_PROXY_IDENTITY_SOURCE_USB = 2;         // Read from the descriptors of the USB device behind the port;
+                                                // changes when a device is attached or removed
+}
+
+enum SerialProxyIdentityFlag {
+  SERIAL_PROXY_IDENTITY_FLAG_NONE = 0;
+  SERIAL_PROXY_IDENTITY_FLAG_CONNECTED = 1;  // The backend believes the device is present and usable on this port
+  SERIAL_PROXY_IDENTITY_FLAG_ERROR = 2;      // The USB host stack refused the descriptor query; the strings and
+                                             // IDs below are empty
+}
+
+// The descriptor fields of a USB device as seen by the host stack
+message UsbDeviceDescriptor {
+  uint32 vendor_id = 1;
+  uint32 product_id = 2;
+  uint32 bcd_device = 3;
+  uint32 interface_number = 4;   // bInterfaceNumber the host driver binds to
+}
+
+// The identity of the device behind a port. Identifies one physical device among others of
+// the same kind, so a client matches on manufacturer and product regardless of source. The
+// strings are empty for source NONE, and for source USB while nothing is connected
+// (since API 1.18).
+message SerialProxyIdentity {
+  option (id) = 155;
+  option (source) = SOURCE_SERVER;
+  option (ifdef) = "USE_SERIAL_PROXY";
+
+  uint32 instance = 1;
+  SerialProxyIdentitySource source = 2;
+  uint32 flags = 3;               // Bitmask of SerialProxyIdentityFlag
+  string manufacturer = 4;
+  string product = 5;
+  string serial_number = 6;
+  UsbDeviceDescriptor usb = 7;    // Only sent for source USB
+}
+
 // ==================== BLUETOOTH CONNECTION PARAMS ====================
 message BluetoothSetConnectionParamsRequest {
   option (id) = 145;
@@ -2888,3 +3077,34 @@ message BluetoothSetConnectionParamsResponse {
   uint64 address = 1;
   int32 error = 2;
 }
+
+// ==================== SENDSPIN ====================
+enum SendspinPairingTokenStatus {
+  // The Sendspin client has not started yet, so the token does not exist yet. Ask again later.
+  SENDSPIN_PAIRING_TOKEN_STATUS_NOT_READY = 0;
+  SENDSPIN_PAIRING_TOKEN_STATUS_OK = 1;
+  // The device has no API encryption key, so it will not send the token. Ask again once a key is set.
+  SENDSPIN_PAIRING_TOKEN_STATUS_ENCRYPTION_REQUIRED = 2;
+  // Sendspin is turned off and has not started since boot, so the token does not exist yet.
+  SENDSPIN_PAIRING_TOKEN_STATUS_DISABLED = 3;
+  // The Sendspin client failed to start, and will not start until the device reboots.
+  SENDSPIN_PAIRING_TOKEN_STATUS_FAILED = 4;
+}
+
+// Requests the Sendspin pairing token, which a Sendspin server uses to pair with this device
+message SendspinPairingTokenRequest {
+  option (id) = 159;
+  option (source) = SOURCE_CLIENT;
+  option (ifdef) = "USE_SENDSPIN";
+  // Empty
+}
+
+message SendspinPairingTokenResponse {
+  option (id) = 160;
+  option (source) = SOURCE_SERVER;
+  option (ifdef) = "USE_SENDSPIN";
+
+  SendspinPairingTokenStatus status = 1;
+  // The pairing token (Sendspin pairing.md "Pairing Token"). Set only when status is OK.
+  string token = 2 [(max_data_length) = 127];
+}

esphome/components/api/api_buffer.cpp

@@ -1,20 +1,37 @@
 #include "api_buffer.h"
-#include <new>
+#ifdef ESPHOME_DEBUG_API
+#include "esphome/core/log.h"
+#endif
 
 namespace esphome::api {
 
+#ifdef ESPHOME_DEBUG_API
+void APIBuffer::debug_check_drop_(size_t drop) const {
+  if (drop > this->size_) {
+    ESP_LOGE("api.buffer", "drop_front: drop=%zu size=%u", drop, this->size_);
+    abort();
+  }
+}
+#endif
+
 bool APIBuffer::grow_(size_t n) {
-  // nothrow (no zero-fill) so OOM is reportable; plain new aborts instead
-  // (NEW_OOM_ABORT on ESP8266 Arduino, exception stub on ESP-IDF).
-  // RAMAllocator is no fit here: unique_ptr needs delete[]-compatible memory.
-  std::unique_ptr<uint8_t[]> new_data(new (std::nothrow) uint8_t[n]);
-  if (new_data == nullptr)
+  if (n > MAX_SIZE)
     return false;
-  if (this->size_)
-    std::memcpy(new_data.get(), this->data_.get(), this->size_);
-  this->data_ = std::move(new_data);
+  // realloc extends in place when it can, avoiding the copy
+  uint8_t *grown = RAMAllocator<uint8_t>().reallocate(this->data_.get(), n);
+  if (grown == nullptr)
+    return false;
+  (void) this->data_.release();  // realloc already freed or reused the old block
+  this->data_.reset(grown);
   this->capacity_ = n;
   return true;
 }
 
+uint8_t *APIBuffer::append(size_t n) {
+  const size_t old_size = this->size_;
+  if (!this->resize(old_size + n))
+    return nullptr;
+  return this->data_.get() + old_size;
+}
+
 }  // namespace esphome::api

esphome/components/api/api_buffer.h

@@ -25,6 +25,7 @@ namespace esphome::api {
 /// writes in debug builds.
 class APIBuffer {
  public:
+  static constexpr size_t MAX_SIZE = UINT16_MAX;  // API frames carry 16 bit lengths
   void clear() { this->size_ = 0; }
   /// Returns false if allocation fails; the buffer is left unchanged.
   [[nodiscard]] inline bool reserve(size_t n) ESPHOME_ALWAYS_INLINE { return n <= this->capacity_ || this->grow_(n); }
@@ -36,9 +37,19 @@ class APIBuffer {
   [[nodiscard]] inline bool reserve_and_resize(size_t reserve_size, size_t new_size) ESPHOME_ALWAYS_INLINE {
     if (!this->reserve(std::max(reserve_size, new_size)))
       return false;
-    this->size_ = new_size;
+    this->size_ = static_cast<uint16_t>(new_size);
     return true;
   }
+  /// Grow by n bytes; returns the new bytes, or nullptr on allocation failure.
+  [[nodiscard]] uint8_t *append(size_t n);
+  /// Drop the first `drop` bytes, sliding the rest down. Precondition: drop <= size().
+  void drop_front(size_t drop) {
+#ifdef ESPHOME_DEBUG_API
+    this->debug_check_drop_(drop);
+#endif
+    this->size_ -= drop;
+    std::memmove(this->data_.get(), this->data_.get() + drop, this->size_);
+  }
   uint8_t *data() { return this->data_.get(); }
   const uint8_t *data() const { return this->data_.get(); }
   size_t size() const { return this->size_; }
@@ -55,9 +66,14 @@ class APIBuffer {
 
  protected:
   bool grow_(size_t n);
-  std::unique_ptr<uint8_t[]> data_;
-  size_t size_{0};
-  size_t capacity_{0};
+#ifdef ESPHOME_DEBUG_API
+  void debug_check_drop_(size_t drop) const;
+#endif
+  // RAMAllocator: PSRAM when available, and it reports failure where
+  // new (std::nothrow) still aborts on ESP-IDF without exceptions
+  RAMUniquePtr<uint8_t[]> data_;
+  uint16_t size_{0};
+  uint16_t capacity_{0};
 };
 
 }  // namespace esphome::api

esphome/components/api/api_connection.cpp

@@ -48,6 +48,9 @@
 #ifdef USE_ZWAVE_PROXY
 #include "esphome/components/zwave_proxy/zwave_proxy.h"
 #endif
+#if defined(USE_SENDSPIN) && defined(USE_ESP_IDF)
+#include "esphome/components/sendspin/sendspin_hub.h"
+#endif
 #ifdef USE_WATER_HEATER
 #include "esphome/components/water_heater/water_heater.h"
 #endif
@@ -198,6 +201,17 @@ APIConnection::~APIConnection() {
       proxy->serial_proxy_request(this, enums::SERIAL_PROXY_REQUEST_TYPE_UNSUBSCRIBE);
     }
   }
+#endif
+  // entities holding a transmit reply for this client must not answer into a freed connection
+#ifdef USE_INFRARED
+  for (auto *infrared : App.get_infrareds()) {
+    infrared->on_api_connection_closed(this);
+  }
+#endif
+#ifdef USE_RADIO_FREQUENCY
+  for (auto *radio_frequency : App.get_radio_frequencies()) {
+    radio_frequency->on_api_connection_closed(this);
+  }
 #endif
 }
 
@@ -364,7 +378,10 @@ void APIConnection::check_keepalive_(uint32_t now) {
     ESP_LOGVV(TAG, "Sending keepalive PING");
     PingRequest req;
     this->flags_.sent_ping = this->send_message(req);
-    if (!this->flags_.sent_ping) {
+    if (this->flags_.sent_ping) {
+      // Quiet for a keepalive period and the ping is on its way: a one-off stall's storage can go
+      this->helper_->release_overflow_buffer();
+    } else {
       // If we can't send the ping request directly (tx_buffer full),
       // schedule it at the front of the batch so it will be sent with priority
       ESP_LOGW(TAG, "Buffer full, ping queued");
@@ -597,7 +614,7 @@ bool APIConnection::send_light_state(light::LightState *light) {
 uint16_t APIConnection::try_send_light_state(EntityBase *entity, APIConnection *conn, uint32_t remaining_size) {
   auto *light = static_cast<light::LightState *>(entity);
   LightStateResponse resp;
-  auto values = light->remote_values;
+  auto values = light->get_reported_values();
   auto color_mode = values.get_color_mode();
   resp.state = values.is_on();
   resp.color_mode = static_cast<enums::ColorMode>(color_mode);
@@ -706,6 +723,7 @@ uint16_t APIConnection::try_send_switch_state(EntityBase *entity, APIConnection
   auto *a_switch = static_cast<switch_::Switch *>(entity);
   SwitchStateResponse resp;
   resp.state = a_switch->state;
+  resp.missing_state = !a_switch->has_state();
   return fill_and_encode_entity_state(a_switch, resp, conn, remaining_size);
 }
 
@@ -717,12 +735,7 @@ uint16_t APIConnection::try_send_switch_info(EntityBase *entity, APIConnection *
 }
 void APIConnection::on_switch_command_request(const SwitchCommandRequest &msg) {
   ENTITY_COMMAND_GET(switch_::Switch, a_switch, switch)
-
-  if (msg.state) {
-    a_switch->turn_on();
-  } else {
-    a_switch->turn_off();
-  }
+  a_switch->control(msg.state);
 }
 #endif
 
@@ -756,6 +769,7 @@ uint16_t APIConnection::try_send_climate_state(EntityBase *entity, APIConnection
   auto traits = climate->get_traits();
   resp.mode = static_cast<enums::ClimateMode>(climate->mode);
   resp.action = static_cast<enums::ClimateAction>(climate->action);
+  resp.missing_state = !climate->has_state();
   if (traits.has_feature_flags(climate::CLIMATE_SUPPORTS_CURRENT_TEMPERATURE))
     resp.current_temperature = climate->current_temperature;
   if (traits.has_feature_flags(climate::CLIMATE_SUPPORTS_TWO_POINT_TARGET_TEMPERATURE |
@@ -990,7 +1004,9 @@ uint16_t APIConnection::try_send_select_state(EntityBase *entity, APIConnection
 uint16_t APIConnection::try_send_select_info(EntityBase *entity, APIConnection *conn, uint32_t remaining_size) {
   auto *select = static_cast<select::Select *>(entity);
   ListEntitiesSelectResponse msg;
-  msg.options = &select->traits.get_options();
+  const auto &opts = select->traits.get_options();
+  const std::span<const char *const> options(opts.data(), opts.size());
+  msg.options = &options;
   return fill_and_encode_entity_info(select, msg, conn, remaining_size);
 }
 void APIConnection::on_select_command_request(const SelectCommandRequest &msg) {
@@ -1389,6 +1405,40 @@ void APIConnection::on_z_wave_proxy_request(const ZWaveProxyRequest &msg) {
 }
 #endif
 
+#if defined(USE_SENDSPIN) && defined(USE_ESP_IDF)
+bool APIConnection::send_sendspin_pairing_token_response_() {
+  SendspinPairingTokenResponse resp;
+#ifdef USE_API_NOISE
+  // has_psk() stands in for "this session proved the key": with a key active, plaintext and zero-key sessions
+  // never reach hello, and activating a key closes every session opened before it.
+  if (!this->parent_->get_noise_ctx().has_psk()) {
+    resp.status = enums::SENDSPIN_PAIRING_TOKEN_STATUS_ENCRYPTION_REQUIRED;
+    return this->send_message(resp);
+  }
+  auto token = sendspin_::global_sendspin_hub->get_pairing_token();
+  if (token.has_value()) {
+    resp.status = enums::SENDSPIN_PAIRING_TOKEN_STATUS_OK;
+    resp.token = StringRef(*token);
+  } else if (sendspin_::global_sendspin_hub->status_has_error()) {
+    resp.status = enums::SENDSPIN_PAIRING_TOKEN_STATUS_FAILED;
+  } else if (sendspin_::global_sendspin_hub->is_disabled()) {
+    resp.status = enums::SENDSPIN_PAIRING_TOKEN_STATUS_DISABLED;
+  } else {
+    resp.status = enums::SENDSPIN_PAIRING_TOKEN_STATUS_NOT_READY;
+  }
+#else
+  resp.status = enums::SENDSPIN_PAIRING_TOKEN_STATUS_ENCRYPTION_REQUIRED;
+#endif
+  return this->send_message(resp);
+}
+
+void APIConnection::on_sendspin_pairing_token_request() {
+  if (!this->send_sendspin_pairing_token_response_()) {
+    this->on_fatal_error();
+  }
+}
+#endif
+
 #ifdef USE_ALARM_CONTROL_PANEL
 bool APIConnection::send_alarm_control_panel_state(alarm_control_panel::AlarmControlPanel *a_alarm_control_panel) {
   return this->send_message_smart_(a_alarm_control_panel, AlarmControlPanelStateResponse::MESSAGE_TYPE,
@@ -1449,6 +1499,7 @@ uint16_t APIConnection::try_send_water_heater_state(EntityBase *entity, APIConne
   auto *wh = static_cast<water_heater::WaterHeater *>(entity);
   WaterHeaterStateResponse resp;
   resp.mode = static_cast<enums::WaterHeaterMode>(wh->get_mode());
+  resp.missing_state = !wh->has_state();
   resp.current_temperature = wh->get_current_temperature();
   resp.target_temperature = wh->get_target_temperature();
   resp.target_temperature_low = wh->get_target_temperature_low();
@@ -1514,8 +1565,10 @@ uint16_t APIConnection::try_send_event_info(EntityBase *entity, APIConnection *c
 }
 #endif
 
-#if defined(USE_IR_RF) || defined(USE_RADIO_FREQUENCY)
+#ifdef USE_IR_RF
 void APIConnection::on_infrared_rf_transmit_raw_timings_request(const InfraredRFTransmitRawTimingsRequest &msg) {
+  // Clients on API 1.18+ are told when the frame has left the transmitter; the entity owns that reply
+  const bool want_reply = this->client_supports_api_version(1, 18);
   // Dispatch by key: infrared entities are checked first, then radio frequency entities.
   // The key is unique across all entity instances on a device, so at most one lookup will succeed.
 #ifdef USE_INFRARED
@@ -1525,6 +1578,7 @@ void APIConnection::on_infrared_rf_transmit_raw_timings_request(const InfraredRF
     call.set_carrier_frequency(msg.carrier_frequency);
     call.set_raw_timings_packed(msg.timings_data_, msg.timings_length_, msg.timings_count_);
     call.set_repeat_count(msg.repeat_count);
+    call.set_api_connection(want_reply ? this : nullptr);
     call.perform();
     return;
   }
@@ -1537,13 +1591,38 @@ void APIConnection::on_infrared_rf_transmit_raw_timings_request(const InfraredRF
     call.set_modulation(static_cast<radio_frequency::RadioFrequencyModulation>(msg.modulation));
     call.set_repeat_count(msg.repeat_count);
     call.set_raw_timings_packed(msg.timings_data_, msg.timings_length_, msg.timings_count_);
+    call.set_api_connection(want_reply ? this : nullptr);
     call.perform();
+    return;
   }
 #endif
+  ESP_LOGW(TAG, "IR/RF transmit for unknown key %" PRIu32, msg.key);
+  if (want_reply) {
+    // nothing will ever report for an unknown key, so answer as not started right away
+#ifdef USE_DEVICES
+    const uint32_t device_id = msg.device_id;
+#else
+    const uint32_t device_id = 0;
+#endif
+    if (!this->send_infrared_rf_transmit_complete(device_id, msg.key, false)) {
+      API_LOG_MSG_DROPPED(TAG, "IR/RF reply");
+    }
+  }
+}
+
+bool APIConnection::send_infrared_rf_transmit_complete([[maybe_unused]] uint32_t device_id, uint32_t key,
+                                                       bool success) {
+  InfraredRFTransmitCompleteResponse resp{};
+#ifdef USE_DEVICES
+  resp.device_id = device_id;
+#endif
+  resp.key = key;
+  resp.success = success;
+  return this->send_message(resp);
 }
 #endif
 
-#if defined(USE_IR_RF) || defined(USE_RADIO_FREQUENCY)
+#ifdef USE_IR_RF
 void APIConnection::send_infrared_rf_receive_event(const InfraredRFReceiveEvent &msg) {
   if (!this->send_message(msg)) {
     // V: fires per decoded frame with no subscription gate, so a warning
@@ -1551,6 +1630,7 @@ void APIConnection::send_infrared_rf_receive_event(const InfraredRFReceiveEvent
     ESP_LOGV(TAG, "IR/RF event dropped, TCP buffer full");
   }
 }
+
 #endif
 
 #ifdef USE_SERIAL_PROXY
@@ -1642,6 +1722,22 @@ void APIConnection::on_serial_proxy_get_modem_pins_request(const SerialProxyGetM
   }
 }
 
+void APIConnection::on_subscribe_serial_proxy_identity_request() {
+#ifdef USE_SERIAL_PROXY_USB_IDENTITY
+  // Only USB ports change identity after this snapshot
+  this->flags_.serial_proxy_identity_subscription = true;
+#endif
+  for (auto *proxy : App.get_serial_proxies()) {
+    proxy->send_identity(this);
+  }
+}
+
+void APIConnection::send_serial_proxy_identity(const SerialProxyIdentity &msg) {
+  if (!this->send_message(msg)) {
+    API_LOG_MSG_DROPPED(TAG, "Serial proxy identity");
+  }
+}
+
 void APIConnection::on_serial_proxy_request(const SerialProxyRequest &msg) {
   auto &proxies = App.get_serial_proxies();
   if (msg.instance >= proxies.size()) {
@@ -1813,7 +1909,7 @@ bool APIConnection::send_hello_response_(const HelloRequest &msg) {
 
   HelloResponse resp;
   resp.api_version_major = 1;
-  resp.api_version_minor = 17;
+  resp.api_version_minor = 19;
   // Send only the version string - the client only logs this for debugging and doesn't use it otherwise
   resp.server_info = ESPHOME_VERSION_REF;
   resp.name = StringRef(App.get_name());
@@ -1836,6 +1932,19 @@ bool APIConnection::send_hello_response_(const HelloRequest &msg) {
   // Auto-authenticate - password auth was removed in ESPHome 2026.1.0
   this->complete_authentication_();
 
+#ifdef USE_API_OUTGOING_CONNECTION
+  // With a PSK set only key-verified transports reach hello: plaintext and
+  // zero-PSK are rejected, and pre-activation sessions are force-closed
+  if (msg.outgoing_connection_target && !this->flags_.outgoing_connection_target) {
+    if (this->parent_->get_noise_ctx().has_psk()) {
+      this->flags_.outgoing_connection_target = true;
+      this->parent_->on_outgoing_target_client(this);
+    } else {
+      this->log_client_(ESPHOME_LOG_LEVEL_WARN, LOG_STR("Dial-back target refused; no key active"));
+    }
+  }
+#endif
+
   return this->send_message(resp);
 }
 
@@ -1958,6 +2067,9 @@ bool APIConnection::send_device_info_response_() {
   // one) so this advertisement survives the plaintext removal in 2027.2.0.
   resp.api_encryption_provisionable = !this->parent_->get_noise_ctx().has_psk();
 #endif
+#ifdef USE_API_OUTGOING_CONNECTION
+  resp.api_outgoing_connection_supported = true;
+#endif
 #endif
 #ifdef USE_DEVICES
   size_t device_index = 0;
@@ -2010,6 +2122,12 @@ bool APIConnection::send_device_capabilities_response_() {
     info.port_type = proxy->get_port_type();
     info.configured_line_states = proxy->get_configured_modem_pins();
   }
+#endif
+#ifdef USE_API_WIZARD
+  resp.wizard.configured = true;
+#endif
+#if defined(USE_SENDSPIN) && defined(USE_ESP_IDF)
+  resp.sendspin.feature_flags = sendspin_::global_sendspin_hub->get_feature_flags();
 #endif
   return this->send_message(resp);
 }
@@ -2224,7 +2342,13 @@ void APIConnection::on_noise_encryption_set_key_request(const NoiseEncryptionSet
 }
 #endif
 #ifdef USE_API_HOMEASSISTANT_STATES
-void APIConnection::on_subscribe_home_assistant_states_request() { state_subs_at_ = 0; }
+void APIConnection::on_subscribe_home_assistant_states_request() {
+#ifdef USE_API_WIZARD_LINKED_INPUTS
+  // Remember it, as a client that subscribed also gets the subscriptions again when a wizard input is set
+  this->flags_.home_assistant_states = true;
+#endif
+  state_subs_at_ = 0;
+}
 #endif
 bool APIConnection::try_to_clear_buffer_slow_(bool log_out_of_space) {
   delay(0);
@@ -2268,8 +2392,12 @@ bool APIConnection::send_message_(uint32_t payload_size, uint16_t message_type,
 #endif
   // Capacity reserved above, cannot fail
   (void) shared_buf.resize(write_start + payload_size);
-  ProtoWriteBuffer buffer{&shared_buf, write_start};
-  encode_fn(msg, buffer PROTO_ENCODE_DEBUG_INIT(&shared_buf));
+  uint8_t *end = encode_fn(msg, shared_buf.data() + write_start PROTO_ENCODE_DEBUG_INIT(&shared_buf));
+#ifdef ESPHOME_DEBUG_API
+  proto_check_encode_end(end, shared_buf.data() + shared_buf.size());
+#else
+  (void) end;
+#endif
   return this->send_buffer(ProtoWriteBuffer{&shared_buf}, message_type);
 }
 // encode_to_buffer is defined inline in api_connection.h (ESPHOME_ALWAYS_INLINE)
@@ -2662,6 +2790,13 @@ void APIConnection::process_state_subscriptions_() {
   }
 
   const auto &it = subs[this->state_subs_at_];
+#ifdef USE_API_WIZARD_LINKED_INPUTS
+  // An entity id that is not set yet (a wizard input) has nothing to subscribe to; it is sent once it is set
+  if (it.entity_id[0] == '\0') {
+    this->state_subs_at_++;
+    return;
+  }
+#endif
   SubscribeHomeAssistantStateResponse resp;
   resp.entity_id = StringRef(it.entity_id);
 

esphome/components/api/api_connection.h

@@ -223,6 +223,10 @@ class APIConnection final : public APIServerConnectionBase {
   void on_z_wave_proxy_request(const ZWaveProxyRequest &msg);
 #endif
 
+#if defined(USE_SENDSPIN) && defined(USE_ESP_IDF)
+  void on_sendspin_pairing_token_request();
+#endif
+
 #ifdef USE_ALARM_CONTROL_PANEL
   bool send_alarm_control_panel_state(alarm_control_panel::AlarmControlPanel *a_alarm_control_panel);
   void on_alarm_control_panel_command_request(const AlarmControlPanelCommandRequest &msg);
@@ -233,9 +237,12 @@ class APIConnection final : public APIServerConnectionBase {
   void on_water_heater_command_request(const WaterHeaterCommandRequest &msg);
 #endif
 
-#if defined(USE_IR_RF) || defined(USE_RADIO_FREQUENCY)
+#ifdef USE_IR_RF
   void on_infrared_rf_transmit_raw_timings_request(const InfraredRFTransmitRawTimingsRequest &msg);
   void send_infrared_rf_receive_event(const InfraredRFReceiveEvent &msg);
+  // Reply to an InfraredRFTransmitRawTimingsRequest (API 1.18+); false when the TCP buffer is
+  // full, the entity that owns the reply retries it then
+  [[nodiscard]] bool send_infrared_rf_transmit_complete(uint32_t device_id, uint32_t key, bool success);
 #endif
 
 #ifdef USE_SERIAL_PROXY
@@ -243,6 +250,9 @@ class APIConnection final : public APIServerConnectionBase {
   void on_serial_proxy_write_request(const SerialProxyWriteRequest &msg);
   void on_serial_proxy_set_modem_pins_request(const SerialProxySetModemPinsRequest &msg);
   void on_serial_proxy_get_modem_pins_request(const SerialProxyGetModemPinsRequest &msg);
+  void on_subscribe_serial_proxy_identity_request();
+  /// Send a port identity to this client
+  void send_serial_proxy_identity(const SerialProxyIdentity &msg);
   void on_serial_proxy_request(const SerialProxyRequest &msg);
   void on_serial_proxy_set_mode_request(const SerialProxySetModeRequest &msg);
   void send_serial_proxy_data(const SerialProxyDataReceived &msg);
@@ -273,6 +283,12 @@ class APIConnection final : public APIServerConnectionBase {
   void on_ping_request();
   void on_device_info_request();
   void on_device_capabilities_request();
+#ifdef USE_API_WIZARD
+  void on_device_wizard_request();
+#endif
+#ifdef USE_API_WIZARD_INPUTS
+  void on_wizard_input_set_request(const WizardInputSetRequest &msg);
+#endif
   void on_list_entities_request() { this->begin_iterator_(ActiveIterator::LIST_ENTITIES); }
   void on_subscribe_states_request() {
     this->flags_.state_subscription = true;
@@ -302,6 +318,10 @@ class APIConnection final : public APIServerConnectionBase {
 #endif
 #ifdef USE_API_HOMEASSISTANT_STATES
   void on_subscribe_home_assistant_states_request();
+#ifdef USE_API_WIZARD_LINKED_INPUTS
+  /// Tell this client about the subscriptions whose entity id is stored in the given buffer, as the buffer changed
+  void resend_state_subscriptions(const char *entity_id);
+#endif
 #endif
 #ifdef USE_API_USER_DEFINED_ACTIONS
   void on_execute_service_request(const ExecuteServiceRequest &msg);
@@ -317,9 +337,15 @@ class APIConnection final : public APIServerConnectionBase {
   void on_noise_encryption_set_key_request(const NoiseEncryptionSetKeyRequest &msg);
 #endif
 
+  // How long a new connection holds off the spare ephemeral refill
+  static constexpr uint32_t CONNECT_GRACE_MS = 1000;
   bool is_authenticated() {
     return static_cast<ConnectionState>(this->flags_.connection_state) == ConnectionState::AUTHENTICATED;
   }
+  // An older unauthenticated connection is a stale half open client and does not count
+  bool is_still_connecting(uint32_t now) {
+    return !this->is_authenticated() && now - this->last_traffic_ < CONNECT_GRACE_MS;
+  }
   bool is_connection_setup() {
     return static_cast<ConnectionState>(this->flags_.connection_state) == ConnectionState::CONNECTED ||
            this->is_authenticated();
@@ -339,18 +365,14 @@ class APIConnection final : public APIServerConnectionBase {
   void on_no_setup_connection();
 
   // Function pointer type for type-erased message encoding
-  using MessageEncodeFn = uint8_t *(*) (const void *, ProtoWriteBuffer &PROTO_ENCODE_DEBUG_PARAM);
+  using MessageEncodeFn = ProtoEncodeFn;
   // Function pointer type for type-erased size calculation
   using CalculateSizeFn = uint32_t (*)(const void *);
 
   /// Returns false as soon as the TCP buffer is full. Marked nodiscard so we
   /// have no silent failures: every caller must handle (or log) a refusal.
   template<typename T> [[nodiscard]] bool send_message(const T &msg) {
-    if constexpr (T::ESTIMATED_SIZE == 0) {
-      return this->send_message_(0, T::MESSAGE_TYPE, &encode_msg_noop, &msg);
-    } else {
-      return this->send_message_(msg.calculate_size(), T::MESSAGE_TYPE, &proto_encode_msg<T>, &msg);
-    }
+    return this->send_message_(T::calc_size_msg(&msg), T::MESSAGE_TYPE, &T::encode_msg, &msg);
   }
 
   /// Clear the shared write buffer and reserve space for the first message.
@@ -376,6 +398,23 @@ class APIConnection final : public APIServerConnectionBase {
     return this->helper_->get_peername_to(buf);
   }
 
+#ifdef USE_API_OUTGOING_CONNECTION
+  /// Get the peer address itself, for remembering a dial-back target
+  int getpeername(struct sockaddr *addr, socklen_t *addrlen) const { return this->helper_->getpeername(addr, addrlen); }
+  /// Outgoing connection: send our server hello immediately so the peer can
+  /// pick the matching key. Outgoing connections are only dialed when a PSK
+  /// is set, so the helper is always the noise helper. Call after start().
+  void mark_outgoing() {
+    if (this->flags_.remove) {
+      return;  // start() failed; the connection is already being torn down
+    }
+    APIError err = static_cast<APINoiseFrameHelper *>(this->helper_.get())->send_server_hello_first();
+    if (err != APIError::OK) {
+      this->fatal_error_with_log_(LOG_STR("Server hello failed"), err);
+    }
+  }
+#endif
+
  protected:
   bool try_to_clear_buffer_slow_(bool log_out_of_space);
 
@@ -397,6 +436,9 @@ class APIConnection final : public APIServerConnectionBase {
 #ifdef USE_VOICE_ASSISTANT
   bool send_voice_assistant_get_configuration_response_(const VoiceAssistantConfigurationRequest &msg);
 #endif
+#if defined(USE_SENDSPIN) && defined(USE_ESP_IDF)
+  bool send_sendspin_pairing_token_response_();
+#endif
 
 #ifdef USE_CAMERA
   void try_send_camera_image_();
@@ -406,16 +448,6 @@ class APIConnection final : public APIServerConnectionBase {
   void process_state_subscriptions_();
 #endif
 
-  // Size thunk — converts void* back to concrete type for direct calculate_size() call
-  template<typename T> static uint32_t calc_size(const void *msg) {
-    return static_cast<const T *>(msg)->calculate_size();
-  }
-
-  // Shared no-op encode thunk for empty messages (ESTIMATED_SIZE == 0)
-  static uint8_t *encode_msg_noop(const void *, ProtoWriteBuffer &buf PROTO_ENCODE_DEBUG_PARAM) {
-    return buf.get_pos();
-  }
-
   // Non-template buffer management for send_message
   bool send_message_(uint32_t payload_size, uint16_t message_type, MessageEncodeFn encode_fn, const void *msg);
 
@@ -434,11 +466,7 @@ class APIConnection final : public APIServerConnectionBase {
   // Hot paths (state/info) go through fill_and_encode_entity_state/info instead.
   // batch_message_type_ is already set by dispatch_message_ before reaching here.
   template<typename T> static uint16_t encode_message_to_buffer(T &msg, APIConnection *conn, uint32_t remaining_size) {
-    if constexpr (T::ESTIMATED_SIZE == 0) {
-      return encode_to_buffer_slow(0, &encode_msg_noop, &msg, conn, remaining_size);
-    } else {
-      return encode_to_buffer_slow(msg.calculate_size(), &proto_encode_msg<T>, &msg, conn, remaining_size);
-    }
+    return encode_to_buffer_slow(T::calc_size_msg(&msg), &T::encode_msg, &msg, conn, remaining_size);
   }
 
   // Non-template core — fills state fields and encodes
@@ -450,7 +478,7 @@ class APIConnection final : public APIServerConnectionBase {
   template<typename T>
   static uint16_t fill_and_encode_entity_state(EntityBase *entity, T &msg, APIConnection *conn,
                                                uint32_t remaining_size) {
-    return fill_and_encode_entity_state(entity, msg, &calc_size<T>, &proto_encode_msg<T>, conn, remaining_size);
+    return fill_and_encode_entity_state(entity, msg, &T::calc_size_msg, &T::encode_msg, conn, remaining_size);
   }
 
   // Non-template core — fills info fields, allocates buffers, and encodes
@@ -462,7 +490,7 @@ class APIConnection final : public APIServerConnectionBase {
   template<typename T>
   static uint16_t fill_and_encode_entity_info(EntityBase *entity, T &msg, APIConnection *conn,
                                               uint32_t remaining_size) {
-    return fill_and_encode_entity_info(entity, msg, &calc_size<T>, &proto_encode_msg<T>, conn, remaining_size);
+    return fill_and_encode_entity_info(entity, msg, &T::calc_size_msg, &T::encode_msg, conn, remaining_size);
   }
 
   // Non-template core — fills device_class, then delegates to fill_and_encode_entity_info
@@ -476,8 +504,8 @@ class APIConnection final : public APIServerConnectionBase {
   static uint16_t fill_and_encode_entity_info_with_device_class(EntityBase *entity, T &msg,
                                                                 StringRef &device_class_field, APIConnection *conn,
                                                                 uint32_t remaining_size) {
-    return fill_and_encode_entity_info_with_device_class(entity, msg, device_class_field, &calc_size<T>,
-                                                         &proto_encode_msg<T>, conn, remaining_size);
+    return fill_and_encode_entity_info_with_device_class(entity, msg, device_class_field, &T::calc_size_msg,
+                                                         &T::encode_msg, conn, remaining_size);
   }
 
 #ifdef USE_VOICE_ASSISTANT
@@ -746,12 +774,21 @@ class APIConnection final : public APIServerConnectionBase {
     uint8_t batch_first_message : 1;          // For batch buffer allocation
     uint8_t should_try_send_immediately : 1;  // True after initial states are sent
     uint8_t may_have_remaining_data : 1;      // Read loop hit limit, retry without ready check
+#ifdef USE_API_WIZARD_LINKED_INPUTS
+    uint8_t home_assistant_states : 1;  // Client subscribed to Home Assistant states
+#endif
+#ifdef USE_API_OUTGOING_CONNECTION
+    uint8_t outgoing_connection_target : 1;  // Client declared itself a dial-back target in its hello
+#endif
 #ifdef HAS_PROTO_MESSAGE_DUMP
     uint8_t log_only_mode : 1;
 #endif
-  } flags_{};  // 2 bytes total
+#ifdef USE_SERIAL_PROXY_USB_IDENTITY
+    uint8_t serial_proxy_identity_subscription : 1;
+#endif
+  } flags_{};  // 2 bytes; 3 with HAS_PROTO_MESSAGE_DUMP + USE_API_OUTGOING_CONNECTION + USE_SERIAL_PROXY_USB_IDENTITY
 
-  // 2-byte type immediately after flags_ (no padding between them)
+  // 2-byte type immediately after flags_ (one padding byte when flags_ is 3 bytes)
   uint16_t batch_message_type_{0};  // Current message type during batch encoding
   // 1-byte types to fill remaining space before next 4-byte boundary
   // Client API versions are clamped to 255 on receive (see send_hello_response_)

esphome/components/api/api_connection_buffer.h

@@ -45,8 +45,14 @@ inline uint16_t ESPHOME_ALWAYS_INLINE APIConnection::encode_to_buffer(uint32_t c
     conn->fatal_out_of_memory_();
     return 0;
   }
-  ProtoWriteBuffer buffer{&shared_buf, shared_buf.size() - calculated_size};
-  encode_fn(msg, buffer PROTO_ENCODE_DEBUG_INIT(&shared_buf));
+  uint8_t *end =
+      encode_fn(msg, shared_buf.data() + shared_buf.size() - calculated_size PROTO_ENCODE_DEBUG_INIT(&shared_buf));
+#ifdef ESPHOME_DEBUG_API
+  // A body that writes fewer bytes than calculate_size() promised would ship stale buffer bytes
+  proto_check_encode_end(end, shared_buf.data() + shared_buf.size());
+#else
+  (void) end;
+#endif
 
   return total_calculated_size;
 }

esphome/components/api/api_frame_helper.cpp

@@ -171,7 +171,7 @@ APIError APIFrameHelper::write_raw_iov_(const struct iovec *iov, int iovcnt, uin
     return APIError::OK;
 
   // Queue unsent data into overflow buffer
-  if (!this->overflow_buf_.enqueue_iov(iov, iovcnt, total_write_len, static_cast<uint16_t>(sent))) {
+  if (!this->overflow_buf_.enqueue_iov(iov, iovcnt, total_write_len, sent)) {
     HELPER_LOG("Overflow buffer full or out of memory, dropping connection");
     this->state_ = State::FAILED;
     return APIError::SOCKET_WRITE_FAILED;

esphome/components/api/api_frame_helper.h

@@ -219,7 +219,10 @@ class APIFrameHelper {
     if (this->rx_buf_len_ == 0) {
       this->rx_buf_.release();
     }
+    this->release_overflow_buffer();
   }
+  // Free the send backlog storage once it has drained
+  void release_overflow_buffer() { this->overflow_buf_.release(); }
 
  protected:
   // Drain backlogged overflow data to the socket and handle errors.
@@ -282,7 +285,8 @@ class APIFrameHelper {
     DATA = 5,
     CLOSED = 6,
     FAILED = 7,
-    EXPLICIT_REJECT = 8,  // Noise only
+    EXPLICIT_REJECT = 8,        // Noise only
+    CLIENT_HELLO_OUTGOING = 9,  // Noise only: like CLIENT_HELLO but the server hello already went out (outgoing conn)
   };
 
   // Fast inline state check for read_packet/write_protobuf_messages hot path.

esphome/components/api/api_frame_helper_noise.cpp

@@ -5,16 +5,13 @@
 #include "esphome/components/noise/noise.h"
 #include "esphome/core/application.h"
 #include "esphome/core/entity_base.h"
+#include "esphome/core/hal.h"
 #include "esphome/core/helpers.h"
 #include "esphome/core/log.h"
 #include "proto.h"
 #include <cstring>
 #include <cinttypes>
 
-#ifdef USE_ESP8266
-#include <pgmspace.h>
-#endif
-
 namespace esphome::api {
 
 using noise::noise_err_to_logstr;
@@ -26,11 +23,7 @@ static_assert(MAX_HANDSHAKE_SIZE == noise::MAX_HANDSHAKE_SIZE,
               "api and noise component handshake size limits must match");
 
 static const char *const TAG = "api.noise";
-#ifdef USE_ESP8266
 static constexpr char PROLOGUE_INIT[] PROGMEM = "NoiseAPIInit";
-#else
-static const char *const PROLOGUE_INIT = "NoiseAPIInit";
-#endif
 static constexpr size_t PROLOGUE_INIT_LEN = 12;  // strlen("NoiseAPIInit")
 
 // Maximum bytes to log in hex format (168 * 3 = 504, under TX buffer size of 512)
@@ -67,20 +60,23 @@ APIError APINoiseFrameHelper::init() {
   }
 
   // init prologue
-  size_t old_size = prologue_.size();
-  if (!prologue_.resize(old_size + PROLOGUE_INIT_LEN)) [[unlikely]] {
+  uint8_t *dst = prologue_.append(PROLOGUE_INIT_LEN);
+  if (dst == nullptr) [[unlikely]] {
     state_ = State::FAILED;
     return APIError::OUT_OF_MEMORY;
   }
-#ifdef USE_ESP8266
-  memcpy_P(prologue_.data() + old_size, PROLOGUE_INIT, PROLOGUE_INIT_LEN);
-#else
-  std::memcpy(prologue_.data() + old_size, PROLOGUE_INIT, PROLOGUE_INIT_LEN);
-#endif
+  progmem_memcpy(dst, PROLOGUE_INIT, PROLOGUE_INIT_LEN);
 
   state_ = State::CLIENT_HELLO;
   return APIError::OK;
 }
+#ifdef USE_API_OUTGOING_CONNECTION
+APIError APINoiseFrameHelper::send_server_hello_first() {
+  // The peer needs our name and MAC to pick the key before its first message
+  this->state_ = State::CLIENT_HELLO_OUTGOING;
+  return this->send_server_hello_frame_();
+}
+#endif
 #ifdef USE_API_PLAINTEXT
 APIError APINoiseFrameHelper::init_from_handoff(const uint8_t *header, uint8_t header_len) {
   APIError err = this->init();
@@ -253,6 +249,9 @@ APIError APINoiseFrameHelper::state_action_() {
       HELPER_LOG("Bad state for method: %d", (int) this->state_);
       return APIError::BAD_STATE;
     case State::CLIENT_HELLO:
+#ifdef USE_API_OUTGOING_CONNECTION
+    case State::CLIENT_HELLO_OUTGOING:
+#endif
       return this->state_action_client_hello_();
     case State::SERVER_HELLO:
       return this->state_action_server_hello_();
@@ -272,24 +271,29 @@ APIError APINoiseFrameHelper::state_action_client_hello_() {
     return handle_handshake_frame_error_(aerr);
   }
   // ignore contents, may be used in future for flags
-  // Resize for: existing prologue + 2 size bytes + frame data
-  size_t old_size = this->prologue_.size();
+  // Append 2 size bytes + frame data to the prologue
   size_t rx_size = this->rx_buf_.size();
-  if (!this->prologue_.resize(old_size + 2 + rx_size)) [[unlikely]] {
+  uint8_t *dst = this->prologue_.append(2 + rx_size);
+  if (dst == nullptr) [[unlikely]] {
     state_ = State::FAILED;
     return APIError::OUT_OF_MEMORY;
   }
-  this->prologue_[old_size] = (uint8_t) (rx_size >> 8);
-  this->prologue_[old_size + 1] = (uint8_t) rx_size;
+  dst[0] = (uint8_t) (rx_size >> 8);
+  dst[1] = (uint8_t) rx_size;
   if (rx_size > 0) {
-    std::memcpy(this->prologue_.data() + old_size + 2, this->rx_buf_.data(), rx_size);
+    std::memcpy(dst + 2, this->rx_buf_.data(), rx_size);
   }
 
+#ifdef USE_API_OUTGOING_CONNECTION
+  if (this->state_ == State::CLIENT_HELLO_OUTGOING) {
+    // Server hello already went out at handoff
+    return this->start_handshake_();
+  }
+#endif
   state_ = State::SERVER_HELLO;
   return APIError::OK;
 }
-APIError APINoiseFrameHelper::state_action_server_hello_() {
-  // send server hello
+APIError APINoiseFrameHelper::send_server_hello_frame_() {
   const auto &name = App.get_name();
   char mac[MAC_ADDRESS_BUFFER_SIZE];
   get_mac_address_into_buffer(mac);
@@ -313,15 +317,18 @@ APIError APINoiseFrameHelper::state_action_server_hello_() {
   // node mac, terminated by null byte
   std::memcpy(msg + mac_offset, mac, MAC_ADDRESS_BUFFER_SIZE);
 
-  APIError aerr = write_frame_(msg, total_size);
+  return write_frame_(msg, total_size);
+}
+APIError APINoiseFrameHelper::state_action_server_hello_() {
+  APIError aerr = this->send_server_hello_frame_();
   if (aerr != APIError::OK)
     return aerr;
-
-  // start handshake
-  aerr = init_handshake_();
+  return this->start_handshake_();
+}
+APIError APINoiseFrameHelper::start_handshake_() {
+  APIError aerr = init_handshake_();
   if (aerr != APIError::OK)
     return aerr;
-
   state_ = State::HANDSHAKE;
   return APIError::OK;
 }

esphome/components/api/api_frame_helper_noise.h

@@ -28,6 +28,12 @@ class APINoiseFrameHelper final : public APIFrameHelper {
   // Seeds the already-read header bytes and pumps the handshake state machine
   // until it would block.
   APIError init_from_handoff(const uint8_t *header, uint8_t header_len);
+#endif
+#ifdef USE_API_OUTGOING_CONNECTION
+  // Send the server hello immediately so the peer can pick the key before
+  // its PSK-mixed message. Call after init(); the mode is tracked in state_
+  // so the helper does not grow.
+  APIError send_server_hello_first();
 #endif
   APIError loop() override;
   APIError read_packet(ReadPacketBuffer *buffer) override;
@@ -39,6 +45,8 @@ class APINoiseFrameHelper final : public APIFrameHelper {
   APIError state_action_();
   APIError state_action_client_hello_();
   APIError state_action_server_hello_();
+  APIError send_server_hello_frame_();
+  APIError start_handshake_();
   APIError state_action_handshake_();
   APIError state_action_handshake_read_();
   APIError state_action_handshake_write_();

esphome/components/api/api_options.proto

@@ -122,4 +122,9 @@ extend google.protobuf.FieldOptions {
     // to true when the field arrives, so an all-default submessage can be told apart
     // from an absent one (e.g. a UTC ParsedTimezone, which is all zeros).
     optional bool track_presence = 50020 [default=false];
+
+    // progmem: The pointer_to_buffer bytes field points into flash (PROGMEM).
+    // The generated encode and dump copy it with progmem_memcpy, as ESP8266
+    // cannot read flash with ordinary loads.
+    optional bool progmem = 50021 [default=false];
 }

esphome/components/api/api_outgoing_connection.cpp

@@ -0,0 +1,318 @@
+#include "api_outgoing_connection.h"
+#if defined(USE_API) && defined(USE_API_OUTGOING_CONNECTION)
+
+#include "api_connection.h"
+#include "api_server.h"
+#include "esphome/components/network/util.h"
+#include "esphome/core/application.h"
+#include "esphome/core/helpers.h"
+#include "esphome/core/log.h"
+
+#include <cerrno>
+#include <cinttypes>
+#include <cstdint>
+#include <cstring>
+
+namespace esphome::api {
+
+static const char *const TAG = "api.outgoing";
+
+#ifndef API_OUTGOING_CONNECTION_HOST
+static constexpr uint32_t OUTGOING_TARGET_PREF_HASH = 629847102UL;
+#endif
+
+#ifndef API_OUTGOING_CONNECTION_HOST
+// Read the connection's peer address into target; false when unavailable or
+// of a family this build cannot dial
+static bool peer_to_target(APIConnection *conn, SavedOutgoingTarget &target) {
+  // Zeroed because the raw lwIP getpeername() leaves sin6_scope_id untouched
+  struct sockaddr_storage peer = {};
+  socklen_t peer_len = sizeof(peer);
+  if (conn->getpeername((struct sockaddr *) &peer, &peer_len) != 0) {
+    return false;
+  }
+  const sa_family_t family = ((struct sockaddr *) &peer)->sa_family;
+#if USE_NETWORK_IPV6
+  if (family == AF_INET6) {
+    const auto *addr6 = reinterpret_cast<const struct sockaddr_in6 *>(&peer);
+    const auto *bytes = reinterpret_cast<const uint8_t *>(&addr6->sin6_addr);
+    uint32_t prefix[3];
+    memcpy(prefix, bytes, sizeof(prefix));
+    // A dual-stack listener reports an IPv4 peer as ::ffff:a.b.c.d
+    if (prefix[0] == 0 && prefix[1] == 0 && prefix[2] == htonl(0xFFFFUL)) {
+      target.family = AF_INET;
+      memcpy(target.addr, bytes + sizeof(prefix), sizeof(struct in_addr));
+      return true;
+    }
+    // A link-local target is only reachable through the interface it came in
+    // on. Device platforms number interfaces from one; a host build can hand
+    // out an index too large to store, and a truncated one dials the wrong
+    // interface, so that target is not remembered at all.
+    if (addr6->sin6_scope_id > UINT8_MAX) {
+      return false;
+    }
+    target.family = AF_INET6;
+    memcpy(target.addr, bytes, sizeof(target.addr));
+    target.scope_id = static_cast<uint8_t>(addr6->sin6_scope_id);
+    return true;
+  }
+#endif
+  if (family != AF_INET) {
+    return false;
+  }
+  const auto *addr4 = reinterpret_cast<const struct sockaddr_in *>(&peer);
+  target.family = AF_INET;
+  memcpy(target.addr, &addr4->sin_addr, sizeof(addr4->sin_addr));
+  return true;
+}
+#endif
+
+socklen_t OutgoingConnectionManager::target_sockaddr_(struct sockaddr_storage *addr) const {
+#ifdef API_OUTGOING_CONNECTION_HOST
+  // Validation only lets through a literal both inet_pton and inet6_aton
+  // accept, so this cannot fail
+  return socket::set_sockaddr((struct sockaddr *) addr, sizeof(*addr), API_OUTGOING_CONNECTION_HOST,
+                              API_OUTGOING_CONNECTION_PORT);
+#else
+#if USE_NETWORK_IPV6
+  if (this->saved_.family == AF_INET6) {
+    auto *addr6 = reinterpret_cast<struct sockaddr_in6 *>(addr);
+    memset(addr6, 0, sizeof(*addr6));
+    addr6->sin6_family = AF_INET6;
+    addr6->sin6_port = htons(API_OUTGOING_CONNECTION_PORT);
+    memcpy(&addr6->sin6_addr, this->saved_.addr, sizeof(this->saved_.addr));
+    addr6->sin6_scope_id = this->saved_.scope_id;
+    return sizeof(*addr6);
+  }
+#endif
+  if (this->saved_.family != AF_INET) {
+    return 0;
+  }
+  auto *addr4 = reinterpret_cast<struct sockaddr_in *>(addr);
+  memset(addr4, 0, sizeof(*addr4));
+  addr4->sin_family = AF_INET;
+  addr4->sin_port = htons(API_OUTGOING_CONNECTION_PORT);
+  memcpy(&addr4->sin_addr, this->saved_.addr, sizeof(addr4->sin_addr));
+  return sizeof(*addr4);
+#endif
+}
+
+#ifndef API_OUTGOING_CONNECTION_HOST
+void OutgoingConnectionManager::format_target_(std::span<char, socket::SOCKADDR_STR_LEN> buf) const {
+  struct sockaddr_storage addr;
+  socklen_t addr_len = this->target_sockaddr_(&addr);
+  if (addr_len == 0) {
+    buf[0] = '\0';
+    return;
+  }
+  // Clears buf itself if it cannot format the address
+  socket::format_sockaddr_to((struct sockaddr *) &addr, addr_len, buf);
+}
+#endif
+
+void OutgoingConnectionManager::setup() {
+#ifndef API_OUTGOING_CONNECTION_HOST
+  this->target_pref_ = global_preferences->make_preference<SavedOutgoingTarget>(OUTGOING_TARGET_PREF_HASH, true);
+  struct sockaddr_storage addr;
+  // dump_config() prints whichever target this leaves in place
+  if (this->target_pref_.load(&this->saved_) && this->target_sockaddr_(&addr) != 0) {
+    this->host_persisted_ = true;
+  } else {
+    // Never saved, failed its size or CRC check, or holds an unknown family
+    this->saved_ = {};
+  }
+#endif
+}
+
+void OutgoingConnectionManager::loop(APIServer *server) {
+  if (server->has_outgoing_target_client_()) {
+    return;  // on_target_client() already reset the dial state
+  }
+  if (this->dialed_conn_ != nullptr) {
+    // A live dialed session (flagged or not, e.g. a host: peer) is the
+    // target; a silent one dies on the handshake timeout
+    return;
+  }
+  const uint32_t now = App.get_loop_component_start_time();
+  switch (this->state_) {
+    case DialState::DIAL_STATE_IDLE:
+      // Target went away; give it the configured delay to reconnect first
+      this->schedule_wait_(now, IDLE_WAIT_MS);
+      break;
+    case DialState::DIAL_STATE_WAITING:
+      if (now - this->state_ts_ >= this->wait_) {
+        this->try_dial_(server, now);
+      }
+      break;
+    case DialState::DIAL_STATE_CONNECTING:
+      this->poll_connect_(server, now);
+      break;
+  }
+}
+
+void OutgoingConnectionManager::try_dial_(APIServer *server, uint32_t now) {
+  if (!network::is_connected()) {
+    // Flips within seconds of boot; recheck fast so a deep sleep wake
+    // window is not spent waiting
+    this->schedule_wait_(now, NETWORK_RETRY_MS);
+    return;
+  }
+  struct sockaddr_storage addr;
+  socklen_t addr_len = this->target_sockaddr_(&addr);
+  const bool at_limit = server->at_client_limit_();
+  // No target is the steady state until a dial-back client has ever connected
+  if (addr_len == 0 || at_limit || !server->noise_ctx_.has_psk()) {
+    // Repeats for as long as the reason holds, so keep it out of debug logs
+    ESP_LOGV(TAG, "Not dialing: %s",
+             addr_len == 0 ? LOG_STR_LITERAL("no target")
+                           : (at_limit ? LOG_STR_LITERAL("max connections") : LOG_STR_LITERAL("no key")));
+    // Not a dial failure; retry without escalating the backoff
+    this->schedule_wait_(now, PRECONDITION_RETRY_MS);
+    return;
+  }
+  this->dial_socket_ = socket::socket_loop_monitored(((struct sockaddr *) &addr)->sa_family, SOCK_STREAM, IPPROTO_TCP);
+  if (!this->dial_socket_ || this->dial_socket_->setblocking(false) != 0) {
+    ESP_LOGW(TAG, "Socket %s failed: errno %d",
+             this->dial_socket_ ? LOG_STR_LITERAL("setblocking") : LOG_STR_LITERAL("create"), errno);
+    this->schedule_retry_(now);
+    return;
+  }
+#ifdef API_OUTGOING_CONNECTION_HOST
+  ESP_LOGD(TAG, "Dialing " API_OUTGOING_CONNECTION_HOST ":%u", API_OUTGOING_CONNECTION_PORT);
+#else
+  char host[socket::SOCKADDR_STR_LEN];
+  socket::format_sockaddr_to((struct sockaddr *) &addr, addr_len, host);
+  ESP_LOGD(TAG, "Dialing %s:%u", host, API_OUTGOING_CONNECTION_PORT);
+#endif
+  int err = this->dial_socket_->connect((struct sockaddr *) &addr, addr_len);
+  if (err == 0) {
+    // Immediate success (possible for localhost)
+    this->handoff_(server, now);
+    return;
+  }
+  if (errno != EINPROGRESS) {
+    ESP_LOGW(TAG, "Connect failed: %d", errno);
+    this->schedule_retry_(now);
+    return;
+  }
+  this->state_ = DialState::DIAL_STATE_CONNECTING;
+  this->state_ts_ = now;
+  this->last_poll_ = now;
+}
+
+void OutgoingConnectionManager::poll_connect_(APIServer *server, uint32_t now) {
+  if (now - this->state_ts_ >= CONNECT_TIMEOUT_MS) {
+    ESP_LOGW(TAG, "Connect timeout");
+    this->schedule_retry_(now);
+    return;
+  }
+  if (now - this->last_poll_ < CONNECT_POLL_INTERVAL_MS) {
+    return;
+  }
+  this->last_poll_ = now;
+  int err = 0;
+  switch (socket::poll_connect(*this->dial_socket_, err)) {
+    case socket::ConnectPollResult::CONNECT_POLL_RESULT_PENDING:
+      break;
+    case socket::ConnectPollResult::CONNECT_POLL_RESULT_CONNECTED:
+      this->handoff_(server, now);
+      break;
+    case socket::ConnectPollResult::CONNECT_POLL_RESULT_ERROR:
+      ESP_LOGW(TAG, "Connect failed: %d", err);
+      this->schedule_retry_(now);
+      break;
+  }
+}
+
+void OutgoingConnectionManager::handoff_(APIServer *server, uint32_t now) {
+  this->dialed_conn_ = server->add_outgoing_client_(std::move(this->dial_socket_));
+  if (this->dialed_conn_ == nullptr) {
+    // Only preconditions (slot limit, key cleared) refuse the handoff; the
+    // peer is reachable, so do not escalate the backoff
+    this->schedule_wait_(now, PRECONDITION_RETRY_MS);
+    return;
+  }
+  // Connected; dialed_conn_ gates further dialing until the session settles
+  this->state_ = DialState::DIAL_STATE_IDLE;
+}
+
+void OutgoingConnectionManager::schedule_wait_(uint32_t now, uint32_t wait) {
+  this->dial_socket_.reset();  // no-op when the socket was handed off
+  this->state_ = DialState::DIAL_STATE_WAITING;
+  this->state_ts_ = now;
+  this->wait_ = wait;
+}
+
+void OutgoingConnectionManager::schedule_retry_(uint32_t now) {
+  // +/-20% jitter so a fleet of devices does not retry one server in lockstep
+  const uint32_t jitter_span = this->backoff_ / 5;
+  this->schedule_wait_(now, this->backoff_ - jitter_span + (random_uint32() % (2 * jitter_span + 1)));
+  this->backoff_ = std::min(this->backoff_ * 2, BACKOFF_MAX_MS);
+}
+
+void OutgoingConnectionManager::on_client_removed(APIConnection *conn, bool was_authenticated) {
+  if (conn != this->dialed_conn_) {
+    return;
+  }
+  this->dialed_conn_ = nullptr;
+  if (was_authenticated) {
+    // A working peer (e.g. a host: target that never sends the flag)
+    // disconnected normally; state is IDLE, so loop() applies the delay
+    this->backoff_ = BACKOFF_MIN_MS;
+  } else {
+    this->schedule_retry_(App.get_loop_component_start_time());
+  }
+}
+
+void OutgoingConnectionManager::on_target_client(APIConnection *conn) {
+  // The target is connected; stop any dial in flight and reset the backoff.
+  // A dialed connection stays tracked unless it is this one: an inbound
+  // target must not orphan a still-open dial.
+  this->dial_socket_.reset();
+  if (conn == this->dialed_conn_) {
+    this->dialed_conn_ = nullptr;
+  }
+  this->state_ = DialState::DIAL_STATE_IDLE;
+  this->backoff_ = BACKOFF_MIN_MS;
+#ifndef API_OUTGOING_CONNECTION_HOST
+  SavedOutgoingTarget target{};
+  if (!peer_to_target(conn, target)) {
+    ESP_LOGW(TAG, "Not remembering this target; its address cannot be dialed");
+    return;
+  }
+  if (this->host_persisted_ && memcmp(&target, &this->saved_, sizeof(target)) == 0) {
+    return;  // unchanged and already on flash; avoid flash wear
+  }
+  // Use the fresh address this boot even if the flash write fails; a failed
+  // write is retried on the next flagged hello via host_persisted_
+  this->saved_ = target;
+  if (!this->persist_target_()) {
+    ESP_LOGW(TAG, "Failed to save target");
+    return;
+  }
+  char host[socket::SOCKADDR_STR_LEN];
+  this->format_target_(host);
+  ESP_LOGD(TAG, "Remembered %s as the dial target", host);
+#endif
+}
+
+void OutgoingConnectionManager::dump_config() const {
+  // The boot delay differs from delay: on deep sleep builds, so print the
+  // value that actually applies
+  ESP_LOGCONFIG(TAG,
+                "  Outgoing connection port: %u\n"
+                "  Outgoing connection boot delay: %" PRIu32 "ms",
+                API_OUTGOING_CONNECTION_PORT, BOOT_WAIT_MS);
+  // Both forms keep their text out of RAM on ESP8266: in the format string,
+  // or through LOG_STR_LITERAL
+#ifdef API_OUTGOING_CONNECTION_HOST
+  ESP_LOGCONFIG(TAG, "  Outgoing connection host: " API_OUTGOING_CONNECTION_HOST);
+#else
+  char buf[socket::SOCKADDR_STR_LEN];
+  this->format_target_(buf);
+  ESP_LOGCONFIG(TAG, "  Outgoing connection host: %s", buf[0] == '\0' ? LOG_STR_LITERAL("none remembered yet") : buf);
+#endif
+}
+
+}  // namespace esphome::api
+#endif  // USE_API && USE_API_OUTGOING_CONNECTION

esphome/components/api/api_outgoing_connection.h

@@ -0,0 +1,120 @@
+#pragma once
+
+#include "esphome/core/defines.h"
+#if defined(USE_API) && defined(USE_API_OUTGOING_CONNECTION)
+
+#ifndef USE_API_NOISE
+#error "api outgoing_connection needs noise encryption so the peer is verified by key"
+#endif
+
+#include "esphome/components/socket/socket.h"
+#include "esphome/core/preferences.h"
+
+#include <memory>
+
+namespace esphome::api {
+
+class APIServer;
+class APIConnection;
+
+// Room for an IPv6 address in every build, so a remembered IPv4 target is
+// still dialed after enable_ipv6 is turned on. A size that followed the build
+// would also shift every preference registered after this one on ESP8266,
+// where slots are positional. An IPv6 target on a build without IPv6 is
+// dropped by target_sockaddr_() and relearned.
+// Bytes in an IPv6 address
+static constexpr size_t TARGET_ADDR_LEN = 16;
+
+struct SavedOutgoingTarget {
+  // 0 when none is remembered, else AF_INET or AF_INET6
+  uint8_t family;
+  // Network order, IPv4 in the first four bytes and the rest zero
+  uint8_t addr[TARGET_ADDR_LEN];
+  // Interface a link-local IPv6 target is reachable on, 0 when it needs none.
+  // Free in flash: the record still rounds up to the same five words.
+  uint8_t scope_id;
+} PACKED;  // NOLINT
+
+/// Dials out when no dial-back target client is connected. Only the TCP
+/// direction flips: the device stays the Noise responder, so both sides
+/// still verify by key. Targets the YAML host or the last remembered client.
+class OutgoingConnectionManager {
+ public:
+  void setup();
+  void loop(APIServer *server);
+  /// A key-verified client declared itself a dial-back target; last one wins
+  void on_target_client(APIConnection *conn);
+  /// Clears the dialed-connection gate; dying unauthenticated escalates the backoff
+  void on_client_removed(APIConnection *conn, bool was_authenticated);
+  void on_shutdown() { this->dial_socket_.reset(); }
+  void dump_config() const;
+
+ protected:
+  enum class DialState : uint8_t {
+    DIAL_STATE_IDLE,
+    DIAL_STATE_WAITING,
+    DIAL_STATE_CONNECTING,
+  };
+
+  static constexpr uint32_t BACKOFF_MIN_MS = 5000;
+  static constexpr uint32_t BACKOFF_MAX_MS = 300000;
+  static constexpr uint32_t CONNECT_TIMEOUT_MS = 10000;
+  static constexpr uint32_t CONNECT_POLL_INTERVAL_MS = 250;
+  static constexpr uint32_t NETWORK_RETRY_MS = 500;
+  static constexpr uint32_t PRECONDITION_RETRY_MS = 5000;
+  // A deep sleep wake window is too short to spend on the delay, so those
+  // builds dial out as soon as the target is gone
+#ifdef USE_DEEP_SLEEP
+  static constexpr uint32_t BOOT_WAIT_MS = 0;
+  static constexpr uint32_t IDLE_WAIT_MS = BACKOFF_MIN_MS;
+#else
+  static constexpr uint32_t BOOT_WAIT_MS = API_OUTGOING_CONNECTION_DELAY;
+  static constexpr uint32_t IDLE_WAIT_MS = API_OUTGOING_CONNECTION_DELAY;
+#endif
+
+  void try_dial_(APIServer *server, uint32_t now);
+  void poll_connect_(APIServer *server, uint32_t now);
+  // Hand the connected socket to the server and gate on the new connection
+  void handoff_(APIServer *server, uint32_t now);
+  // Close any half-open dial and wait a jittered backoff before retrying
+  void schedule_retry_(uint32_t now);
+  // Wait without escalating the backoff (used for unmet preconditions)
+  void schedule_wait_(uint32_t now, uint32_t wait);
+  /// Fill addr with the target and return its length, or 0 when there is none
+  socklen_t target_sockaddr_(struct sockaddr_storage *addr) const;
+#ifndef API_OUTGOING_CONNECTION_HOST
+  // Write saved_ to flash, tracking success in host_persisted_
+  bool persist_target_() {
+    this->host_persisted_ = this->target_pref_.save(&this->saved_) && global_preferences->sync();
+    return this->host_persisted_;
+  }
+  /// Format the remembered target for a log line; empty when there is none
+  void format_target_(std::span<char, socket::SOCKADDR_STR_LEN> buf) const;
+#endif
+
+  // Pointers first (4 bytes each on 32-bit)
+  std::unique_ptr<socket::Socket> dial_socket_;
+  // Compared only, never dereferenced
+  APIConnection *dialed_conn_{nullptr};
+#ifndef API_OUTGOING_CONNECTION_HOST
+  ESPPreferenceObject target_pref_;
+#endif
+
+  // 4-byte types
+  uint32_t backoff_{BACKOFF_MIN_MS};
+  uint32_t wait_{BOOT_WAIT_MS};
+  uint32_t state_ts_{0};
+  uint32_t last_poll_{0};
+
+  // Byte-aligned types last
+#ifndef API_OUTGOING_CONNECTION_HOST
+  SavedOutgoingTarget saved_{};
+  // False while saved_ holds a value the flash write failed for; retried on
+  // the next flagged hello
+  bool host_persisted_{false};
+#endif
+  DialState state_{DialState::DIAL_STATE_WAITING};
+};
+
+}  // namespace esphome::api
+#endif  // USE_API && USE_API_OUTGOING_CONNECTION

esphome/components/api/api_overflow_buffer.cpp

@@ -1,98 +1,91 @@
 #include "api_overflow_buffer.h"
 #ifdef USE_API
 #include <cstring>
-#include <new>
 
 namespace esphome::api {
 
-APIOverflowBuffer::~APIOverflowBuffer() {
-  for (auto *entry : this->queue_) {
-    if (entry != nullptr)
-      Entry::destroy(entry);
-  }
-}
-
 ssize_t APIOverflowBuffer::try_drain(socket::Socket *socket) {
-  // socket->write() can re-enter this function: a log message emitted from an
-  // lwip callback during the write goes out over the API and lands back in the
-  // frame helper's write/drain path. If a nested drain ran here it would send
-  // and free the entry the outer drain is still holding, causing a double free.
-  // Report "no progress" instead; the outer drain keeps draining, and the
-  // nested send is enqueued behind the existing backlog.
+  // Nested call from inside socket->write(); see draining_
   if (this->draining_)
     return 0;
 
-  // RAII so the flag is cleared on every return path
   struct DrainGuard {
-    explicit DrainGuard(bool &flag) : flag_(flag) { flag_ = true; }
-    ~DrainGuard() { this->flag_ = false; }
-    bool &flag_;
-  } guard(this->draining_);
+    APIOverflowBuffer &owner;
+    ~DrainGuard() { this->owner.draining_ = false; }
+  } guard{*this};
+  this->draining_ = true;
 
   while (this->count_ > 0) {
-    Entry *front = this->queue_[this->head_];
+    uint8_t *msg = this->buf_.data() + this->head_;
+    size_t len = msg[0] | (msg[1] << 8);
 
-    ssize_t sent = socket->write(front->current_data(), front->remaining());
-
-    if (sent <= 0) {
-      // -1 = error (caller checks errno for EWOULDBLOCK vs hard error)
-      // 0 = nothing sent (treat as no progress)
+    ssize_t sent = socket->write(msg + LEN_PREFIX, len);
+    if (sent <= 0)
+      return sent;
+    if (static_cast<size_t>(sent) < len) {
+      // Step past the sent bytes and rewrite the prefix there; it lands on bytes already sent
+      this->head_ += sent;
+      len -= sent;
+      msg += sent;
+      msg[0] = len;
+      msg[1] = len >> 8;
       return sent;
     }
-
-    if (static_cast<uint16_t>(sent) < front->remaining()) {
-      // Partially sent, update offset and stop
-      front->offset += static_cast<uint16_t>(sent);
-      return sent;
-    }
-
-    // Entry fully sent — unlink it before freeing so a freed pointer is never
-    // reachable from the queue
-    this->queue_[this->head_] = nullptr;
-    this->head_ = (this->head_ + 1) % API_MAX_SEND_QUEUE;
+    this->head_ += LEN_PREFIX + len;
     this->count_--;
-    Entry::destroy(front);
   }
 
-  return 0;  // All drained
+  this->head_ = 0;
+  if (this->release_when_drained_) {
+    this->release_when_drained_ = false;
+    this->buf_.release();
+  } else {
+    this->buf_.clear();
+  }
+  return 0;
 }
 
-bool APIOverflowBuffer::enqueue_iov(const struct iovec *iov, int iovcnt, uint16_t total_len, uint16_t skip) {
+bool APIOverflowBuffer::enqueue_iov(const struct iovec *iov, int iovcnt, size_t total_len, size_t skip) {
   if (this->count_ >= API_MAX_SEND_QUEUE)
     return false;
 
-  uint16_t buffer_size = total_len - skip;
-  // nothrow: a failed allocation returns nullptr so the connection is dropped
-  // cleanly instead of plain new's crash or abort on OOM
-  // NOLINTNEXTLINE(cppcoreguidelines-owning-memory)
-  auto *data = new (std::nothrow) uint8_t[buffer_size];
-  if (data == nullptr)
-    return false;
-  // NOLINTNEXTLINE(cppcoreguidelines-owning-memory)
-  auto *entry = new (std::nothrow) Entry{data, buffer_size, 0};
-  if (entry == nullptr) {
-    delete[] data;
+  const size_t new_len = total_len - skip;
+  const size_t new_bytes = LEN_PREFIX + new_len;
+  const size_t live = this->buf_.size() - this->head_;
+  // A lone message is only bound by the buffer; refusing it would just drop the connection
+  if (live + new_bytes > (this->count_ > 0 ? MAX_BYTES : MAX_LONE_BYTES))
     return false;
+
+  if (this->buf_.size() + new_bytes > this->buf_.capacity()) {
+    // Storage would move under an outer drain's write()
+    if (this->draining_)
+      return false;
+    if (this->head_ > 0) {
+      // Reclaim the sent prefix before growing
+      this->buf_.drop_front(this->head_);
+      this->head_ = 0;
+    }
+    if (!this->buf_.reserve(reserve_for(live + new_bytes)))
+      return false;
   }
 
-  uint16_t to_skip = skip;
-  uint16_t write_pos = 0;
-
-  for (int i = 0; i < iovcnt; i++) {
-    if (to_skip >= iov[i].iov_len) {
-      to_skip -= static_cast<uint16_t>(iov[i].iov_len);
+  uint8_t *dst = this->buf_.append(new_bytes);
+  if (dst == nullptr)
+    return false;
+  dst[0] = new_len;
+  dst[1] = new_len >> 8;
+  dst += LEN_PREFIX;
+  for (const struct iovec *end = iov + iovcnt; iov != end; iov++) {
+    if (skip >= iov->iov_len) {
+      skip -= iov->iov_len;
     } else {
-      const uint8_t *src = reinterpret_cast<uint8_t *>(iov[i].iov_base) + to_skip;
-      uint16_t len = static_cast<uint16_t>(iov[i].iov_len) - to_skip;
-      std::memcpy(entry->data + write_pos, src, len);
-      write_pos += len;
-      to_skip = 0;
+      const size_t len = iov->iov_len - skip;
+      std::memcpy(dst, static_cast<const uint8_t *>(iov->iov_base) + skip, len);
+      dst += len;
+      skip = 0;
     }
   }
 
-  // Publish only after the copy completes so a half-built entry is never reachable
-  this->queue_[this->tail_] = entry;
-  this->tail_ = (this->tail_ + 1) % API_MAX_SEND_QUEUE;
   this->count_++;
   return true;
 }

esphome/components/api/api_overflow_buffer.h

@@ -1,5 +1,6 @@
 #pragma once
-#include <array>
+#include <algorithm>
+#include <cstddef>
 #include <cstdint>
 #include <sys/types.h>
 
@@ -8,71 +9,57 @@
 
 #include "esphome/components/socket/headers.h"
 #include "esphome/components/socket/socket.h"
-#include "esphome/core/helpers.h"
+#include "api_buffer.h"
 
 namespace esphome::api {
 
-/// Circular queue of heap-allocated byte buffers used as a TCP send backlog.
-///
-/// Under normal operation this buffer is **never used** — data goes straight
-/// from the frame helper to the socket.  It only fills when the LWIP TCP
-/// send buffer is full (slow client, congested network, heavy logging).
-/// The queue drains automatically on subsequent write/loop calls once the
-/// socket becomes writable again.
-///
-/// Capacity is compile-time-fixed via API_MAX_SEND_QUEUE (set from Python
-/// config).  If the queue fills completely the connection is marked failed.
+/// TCP send backlog, only used when the socket send buffer is full.
+/// One contiguous buffer per connection, allocated on the first stall and
+/// kept at its high-water mark so a lossy link does not churn the heap.
+/// Messages are stored as a 2 byte length prefix plus payload.
+/// API_MAX_SEND_QUEUE bounds queued messages and, at 2 KB per slot, queued
+/// bytes; exceeding either fails the connection.
 class APIOverflowBuffer {
  public:
-  /// A single heap-allocated send-backlog entry.
-  /// Lifetime is manually managed — see destroy().
-  struct Entry {
-    uint8_t *data;
-    uint16_t size;    // Total size of the buffer
-    uint16_t offset;  // Current send offset within the buffer
-
-    uint16_t remaining() const { return this->size - this->offset; }
-    const uint8_t *current_data() const { return this->data + this->offset; }
-
-    /// Free this entry and its data buffer.
-    static ESPHOME_ALWAYS_INLINE void destroy(Entry *entry) {
-      delete[] entry->data;
-      delete entry;  // NOLINT(cppcoreguidelines-owning-memory)
-    }
-  };
-
-  ~APIOverflowBuffer();
-
   /// True when no backlogged data is waiting.
   bool empty() const { return this->count_ == 0; }
 
-  /// True when the queue has no room for another entry.
-  bool full() const { return this->count_ >= API_MAX_SEND_QUEUE; }
-
-  /// Number of entries currently queued.
-  uint8_t count() const { return this->count_; }
-
-  /// Try to drain queued data to the socket.
-  /// Returns bytes-written > 0 on success/partial, 0 if all drained or no progress,
-  /// -1 on error (caller must check errno to distinguish EWOULDBLOCK from hard errors).
-  /// Callers only need to act on -1; 0 and positive values both mean "no error".
-  /// Frees entries as they are fully sent.
+  /// Drain queued messages to the socket.
+  /// Returns bytes written, 0 for a re-entrant call, -1 on error (check errno
+  /// for EWOULDBLOCK); callers only need to act on -1.
   ssize_t try_drain(socket::Socket *socket);
 
-  /// Enqueue unsent IOV data into the backlog.
-  /// Copies iov data starting at byte offset `skip` into a new entry.
-  /// Returns false if the queue is full or allocation fails (caller should fail the connection).
-  bool enqueue_iov(const struct iovec *iov, int iovcnt, uint16_t total_len, uint16_t skip);
+  /// Queue iov data from byte offset `skip` as one message.
+  /// Returns false when a limit is hit, allocation fails, or storage would move
+  /// during a drain; the caller should fail the connection.
+  bool enqueue_iov(const struct iovec *iov, int iovcnt, size_t total_len, size_t skip);
+
+  /// Free the retained storage, now if empty, otherwise once it has drained.
+  void release() {
+    if (this->count_ == 0) {
+      this->buf_.release();
+    } else {
+      this->release_when_drained_ = true;
+    }
+  }
 
  protected:
-  std::array<Entry *, API_MAX_SEND_QUEUE> queue_{};
-  uint8_t head_{0};
-  uint8_t tail_{0};
+  static constexpr size_t LEN_PREFIX = 2;
+  static constexpr size_t BYTES_PER_SLOT = 2048;
+  // Reserve in 256 byte steps so a creeping high-water mark settles quickly
+  static constexpr size_t GROW_QUANTUM = 256;
+  // Lone message ceiling, rounded down so reserve_for() never exceeds the buffer limit
+  static constexpr size_t MAX_LONE_BYTES = APIBuffer::MAX_SIZE & ~(GROW_QUANTUM - 1);
+  static constexpr size_t MAX_BYTES = std::min(API_MAX_SEND_QUEUE * BYTES_PER_SLOT, MAX_LONE_BYTES);
+  static constexpr size_t reserve_for(size_t want) { return (want + GROW_QUANTUM - 1) & ~(GROW_QUANTUM - 1); }
+
+  APIBuffer buf_;
+  uint16_t head_{0};  // offset of the front message's length prefix; bytes before it are sent
   uint8_t count_{0};
-  // Guards against re-entrant drains: socket->write() can re-enter the API
-  // send path (e.g. a log message emitted from an lwip callback), and a nested
-  // drain would free the entry the outer drain is still holding.
-  bool draining_{false};
+  // socket->write() can re-enter the send path (log from an lwip callback):
+  // a nested drain makes no progress and a nested enqueue never moves storage
+  bool draining_ : 1 {false};
+  bool release_when_drained_ : 1 {false};
 };
 
 }  // namespace esphome::api

esphome/components/api/api_pb2_dump.cpp

@@ -114,15 +114,26 @@ template<typename T> static void dump_field(DumpBuffer &out, const char *field_n
   out.append("\n");
 }
 
+// Bytes shown by a bytes field dump: 160 bytes is 480 chars with separators, to fit a typical log buffer
+static constexpr size_t DUMP_BYTES_MAX = 160;
+
 // Helper for bytes fields - uses stack buffer to avoid heap allocation
-// Buffer sized for 160 bytes of data (480 chars with separators) to fit typical log buffer
 // field_name is a PROGMEM pointer (flash on ESP8266, regular pointer on other platforms)
 static void dump_bytes_field(DumpBuffer &out, const char *field_name, const uint8_t *data, size_t len, int indent = 2) {
-  char hex_buf[format_hex_pretty_size(160)];
+  char hex_buf[format_hex_pretty_size(DUMP_BYTES_MAX)];
   append_field_prefix(out, field_name, indent);
   format_hex_pretty_to(hex_buf, data, len);
   out.append(hex_buf).append("\n");
 }
+
+// Helper for bytes fields in flash: copies the shown bytes out with progmem_memcpy first
+static void dump_progmem_bytes_field(DumpBuffer &out, const char *field_name, const uint8_t *data, size_t len,
+                                     int indent = 2) {
+  uint8_t data_buf[DUMP_BYTES_MAX];
+  len = std::min(len, sizeof(data_buf));
+  progmem_memcpy(data_buf, data, len);
+  dump_bytes_field(out, field_name, data_buf, len, indent);
+}
 #pragma GCC diagnostic pop
 
 template<> const char *proto_enum_to_string<enums::DisconnectReason>(enums::DisconnectReason value) {
@@ -143,6 +154,8 @@ template<> const char *proto_enum_to_string<enums::SerialProxyPortType>(enums::S
       return ESPHOME_PSTR("SERIAL_PROXY_PORT_TYPE_RS232");
     case enums::SERIAL_PROXY_PORT_TYPE_RS485:
       return ESPHOME_PSTR("SERIAL_PROXY_PORT_TYPE_RS485");
+    case enums::SERIAL_PROXY_PORT_TYPE_USB_SERIAL:
+      return ESPHOME_PSTR("SERIAL_PROXY_PORT_TYPE_USB_SERIAL");
     default:
       return ESPHOME_PSTR("UNKNOWN");
   }
@@ -890,6 +903,49 @@ template<> const char *proto_enum_to_string<enums::SerialProxyMode>(enums::Seria
       return ESPHOME_PSTR("UNKNOWN");
   }
 }
+template<> const char *proto_enum_to_string<enums::SerialProxyIdentitySource>(enums::SerialProxyIdentitySource value) {
+  switch (value) {
+    case enums::SERIAL_PROXY_IDENTITY_SOURCE_NONE:
+      return ESPHOME_PSTR("SERIAL_PROXY_IDENTITY_SOURCE_NONE");
+    case enums::SERIAL_PROXY_IDENTITY_SOURCE_CONFIGURED:
+      return ESPHOME_PSTR("SERIAL_PROXY_IDENTITY_SOURCE_CONFIGURED");
+    case enums::SERIAL_PROXY_IDENTITY_SOURCE_USB:
+      return ESPHOME_PSTR("SERIAL_PROXY_IDENTITY_SOURCE_USB");
+    default:
+      return ESPHOME_PSTR("UNKNOWN");
+  }
+}
+#endif
+template<> const char *proto_enum_to_string<enums::SerialProxyIdentityFlag>(enums::SerialProxyIdentityFlag value) {
+  switch (value) {
+    case enums::SERIAL_PROXY_IDENTITY_FLAG_NONE:
+      return ESPHOME_PSTR("SERIAL_PROXY_IDENTITY_FLAG_NONE");
+    case enums::SERIAL_PROXY_IDENTITY_FLAG_CONNECTED:
+      return ESPHOME_PSTR("SERIAL_PROXY_IDENTITY_FLAG_CONNECTED");
+    case enums::SERIAL_PROXY_IDENTITY_FLAG_ERROR:
+      return ESPHOME_PSTR("SERIAL_PROXY_IDENTITY_FLAG_ERROR");
+    default:
+      return ESPHOME_PSTR("UNKNOWN");
+  }
+}
+#ifdef USE_SENDSPIN
+template<>
+const char *proto_enum_to_string<enums::SendspinPairingTokenStatus>(enums::SendspinPairingTokenStatus value) {
+  switch (value) {
+    case enums::SENDSPIN_PAIRING_TOKEN_STATUS_NOT_READY:
+      return ESPHOME_PSTR("SENDSPIN_PAIRING_TOKEN_STATUS_NOT_READY");
+    case enums::SENDSPIN_PAIRING_TOKEN_STATUS_OK:
+      return ESPHOME_PSTR("SENDSPIN_PAIRING_TOKEN_STATUS_OK");
+    case enums::SENDSPIN_PAIRING_TOKEN_STATUS_ENCRYPTION_REQUIRED:
+      return ESPHOME_PSTR("SENDSPIN_PAIRING_TOKEN_STATUS_ENCRYPTION_REQUIRED");
+    case enums::SENDSPIN_PAIRING_TOKEN_STATUS_DISABLED:
+      return ESPHOME_PSTR("SENDSPIN_PAIRING_TOKEN_STATUS_DISABLED");
+    case enums::SENDSPIN_PAIRING_TOKEN_STATUS_FAILED:
+      return ESPHOME_PSTR("SENDSPIN_PAIRING_TOKEN_STATUS_FAILED");
+    default:
+      return ESPHOME_PSTR("UNKNOWN");
+  }
+}
 #endif
 
 const char *HelloRequest::dump_to(DumpBuffer &out) const {
@@ -897,6 +953,9 @@ const char *HelloRequest::dump_to(DumpBuffer &out) const {
   dump_field(out, ESPHOME_PSTR("client_info"), this->client_info);
   dump_field(out, ESPHOME_PSTR("api_version_major"), this->api_version_major);
   dump_field(out, ESPHOME_PSTR("api_version_minor"), this->api_version_minor);
+#ifdef USE_API_OUTGOING_CONNECTION
+  dump_field(out, ESPHOME_PSTR("outgoing_connection_target"), this->outgoing_connection_target);
+#endif
   return out.c_str();
 }
 const char *HelloResponse::dump_to(DumpBuffer &out) const {
@@ -1020,6 +1079,9 @@ const char *DeviceInfoResponse::dump_to(DumpBuffer &out) const {
 #endif
 #ifdef USE_API_NOISE
   dump_field(out, ESPHOME_PSTR("api_encryption_provisionable"), this->api_encryption_provisionable);
+#endif
+#ifdef USE_API_OUTGOING_CONNECTION
+  dump_field(out, ESPHOME_PSTR("api_outgoing_connection_supported"), this->api_outgoing_connection_supported);
 #endif
   return out.c_str();
 }
@@ -1046,6 +1108,20 @@ const char *ZWaveProxyCapabilities::dump_to(DumpBuffer &out) const {
   return out.c_str();
 }
 #endif
+#ifdef USE_API_WIZARD
+const char *WizardCapabilities::dump_to(DumpBuffer &out) const {
+  MessageDumpHelper helper(out, ESPHOME_PSTR("WizardCapabilities"));
+  dump_field(out, ESPHOME_PSTR("configured"), this->configured);
+  return out.c_str();
+}
+#endif
+#ifdef USE_SENDSPIN
+const char *SendspinCapabilities::dump_to(DumpBuffer &out) const {
+  MessageDumpHelper helper(out, ESPHOME_PSTR("SendspinCapabilities"));
+  dump_field(out, ESPHOME_PSTR("feature_flags"), this->feature_flags);
+  return out.c_str();
+}
+#endif
 const char *DeviceCapabilitiesResponse::dump_to(DumpBuffer &out) const {
   MessageDumpHelper helper(out, ESPHOME_PSTR("DeviceCapabilitiesResponse"));
 #ifdef USE_BLUETOOTH_PROXY
@@ -1069,9 +1145,34 @@ const char *DeviceCapabilitiesResponse::dump_to(DumpBuffer &out) const {
     it.dump_to(out);
     out.append("\n");
   }
+#endif
+#ifdef USE_API_WIZARD
+  out.append(2, ' ').append_p(ESPHOME_PSTR("wizard")).append(": ");
+  this->wizard.dump_to(out);
+  out.append("\n");
+#endif
+#ifdef USE_SENDSPIN
+  out.append(2, ' ').append_p(ESPHOME_PSTR("sendspin")).append(": ");
+  this->sendspin.dump_to(out);
+  out.append("\n");
 #endif
   return out.c_str();
 }
+#ifdef USE_API_WIZARD
+const char *DeviceWizardResponse::dump_to(DumpBuffer &out) const {
+  MessageDumpHelper helper(out, ESPHOME_PSTR("DeviceWizardResponse"));
+  dump_progmem_bytes_field(out, ESPHOME_PSTR("data"), this->data, this->data_len);
+  return out.c_str();
+}
+#endif
+#ifdef USE_API_WIZARD_INPUTS
+const char *WizardInputSetRequest::dump_to(DumpBuffer &out) const {
+  MessageDumpHelper helper(out, ESPHOME_PSTR("WizardInputSetRequest"));
+  dump_field(out, ESPHOME_PSTR("key"), this->key);
+  dump_field(out, ESPHOME_PSTR("entity_id"), this->entity_id);
+  return out.c_str();
+}
+#endif
 const char *ListEntitiesDoneResponse::dump_to(DumpBuffer &out) const {
   out.append_p(ESPHOME_PSTR("ListEntitiesDoneResponse {}"));
   return out.c_str();
@@ -1342,6 +1443,7 @@ const char *SwitchStateResponse::dump_to(DumpBuffer &out) const {
 #ifdef USE_DEVICES
   dump_field(out, ESPHOME_PSTR("device_id"), this->device_id);
 #endif
+  dump_field(out, ESPHOME_PSTR("missing_state"), this->missing_state);
   return out.c_str();
 }
 const char *SwitchCommandRequest::dump_to(DumpBuffer &out) const {
@@ -1684,6 +1786,7 @@ const char *ClimateStateResponse::dump_to(DumpBuffer &out) const {
 #ifdef USE_DEVICES
   dump_field(out, ESPHOME_PSTR("device_id"), this->device_id);
 #endif
+  dump_field(out, ESPHOME_PSTR("missing_state"), this->missing_state);
   return out.c_str();
 }
 const char *ClimateCommandRequest::dump_to(DumpBuffer &out) const {
@@ -1751,6 +1854,7 @@ const char *WaterHeaterStateResponse::dump_to(DumpBuffer &out) const {
   dump_field(out, ESPHOME_PSTR("state"), this->state);
   dump_field(out, ESPHOME_PSTR("target_temperature_low"), this->target_temperature_low);
   dump_field(out, ESPHOME_PSTR("target_temperature_high"), this->target_temperature_high);
+  dump_field(out, ESPHOME_PSTR("missing_state"), this->missing_state);
   return out.c_str();
 }
 const char *WaterHeaterCommandRequest::dump_to(DumpBuffer &out) const {
@@ -2711,7 +2815,7 @@ const char *ListEntitiesInfraredResponse::dump_to(DumpBuffer &out) const {
   return out.c_str();
 }
 #endif
-#if defined(USE_IR_RF) || defined(USE_RADIO_FREQUENCY)
+#ifdef USE_IR_RF
 const char *InfraredRFTransmitRawTimingsRequest::dump_to(DumpBuffer &out) const {
   MessageDumpHelper helper(out, ESPHOME_PSTR("InfraredRFTransmitRawTimingsRequest"));
 #ifdef USE_DEVICES
@@ -2740,6 +2844,15 @@ const char *InfraredRFReceiveEvent::dump_to(DumpBuffer &out) const {
   }
   return out.c_str();
 }
+const char *InfraredRFTransmitCompleteResponse::dump_to(DumpBuffer &out) const {
+  MessageDumpHelper helper(out, ESPHOME_PSTR("InfraredRFTransmitCompleteResponse"));
+#ifdef USE_DEVICES
+  dump_field(out, ESPHOME_PSTR("device_id"), this->device_id);
+#endif
+  dump_field(out, ESPHOME_PSTR("key"), this->key);
+  dump_field(out, ESPHOME_PSTR("success"), this->success);
+  return out.c_str();
+}
 #endif
 #ifdef USE_RADIO_FREQUENCY
 const char *ListEntitiesRadioFrequencyResponse::dump_to(DumpBuffer &out) const {
@@ -2823,6 +2936,27 @@ const char *SerialProxySetModeRequest::dump_to(DumpBuffer &out) const {
   dump_field(out, ESPHOME_PSTR("mode"), static_cast<enums::SerialProxyMode>(this->mode));
   return out.c_str();
 }
+const char *UsbDeviceDescriptor::dump_to(DumpBuffer &out) const {
+  MessageDumpHelper helper(out, ESPHOME_PSTR("UsbDeviceDescriptor"));
+  dump_field(out, ESPHOME_PSTR("vendor_id"), this->vendor_id);
+  dump_field(out, ESPHOME_PSTR("product_id"), this->product_id);
+  dump_field(out, ESPHOME_PSTR("bcd_device"), this->bcd_device);
+  dump_field(out, ESPHOME_PSTR("interface_number"), this->interface_number);
+  return out.c_str();
+}
+const char *SerialProxyIdentity::dump_to(DumpBuffer &out) const {
+  MessageDumpHelper helper(out, ESPHOME_PSTR("SerialProxyIdentity"));
+  dump_field(out, ESPHOME_PSTR("instance"), this->instance);
+  dump_field(out, ESPHOME_PSTR("source"), static_cast<enums::SerialProxyIdentitySource>(this->source));
+  dump_field(out, ESPHOME_PSTR("flags"), this->flags);
+  dump_field(out, ESPHOME_PSTR("manufacturer"), this->manufacturer);
+  dump_field(out, ESPHOME_PSTR("product"), this->product);
+  dump_field(out, ESPHOME_PSTR("serial_number"), this->serial_number);
+  out.append(2, ' ').append_p(ESPHOME_PSTR("usb")).append(": ");
+  this->usb.dump_to(out);
+  out.append("\n");
+  return out.c_str();
+}
 #endif
 #ifdef USE_BLUETOOTH_PROXY_CONNECTIONS
 const char *BluetoothSetConnectionParamsRequest::dump_to(DumpBuffer &out) const {
@@ -2841,6 +2975,14 @@ const char *BluetoothSetConnectionParamsResponse::dump_to(DumpBuffer &out) const
   return out.c_str();
 }
 #endif
+#ifdef USE_SENDSPIN
+const char *SendspinPairingTokenResponse::dump_to(DumpBuffer &out) const {
+  MessageDumpHelper helper(out, ESPHOME_PSTR("SendspinPairingTokenResponse"));
+  dump_field(out, ESPHOME_PSTR("status"), static_cast<enums::SendspinPairingTokenStatus>(this->status));
+  dump_field(out, ESPHOME_PSTR("token"), this->token);
+  return out.c_str();
+}
+#endif
 
 }  // namespace esphome::api
 

esphome/components/api/api_pb2_includes.h

@@ -28,6 +28,7 @@
 
 // Standard library includes that might be needed
 #include <set>
+#include <span>
 #include <vector>
 #include <string>
 

esphome/components/api/api_pb2_service.cpp

@@ -628,7 +628,7 @@ void APIConnection::read_message_(uint32_t msg_size, uint32_t msg_type, const ui
       break;
     }
 #endif
-#if defined(USE_IR_RF) || defined(USE_RADIO_FREQUENCY)
+#ifdef USE_IR_RF
     case InfraredRFTransmitRawTimingsRequest::MESSAGE_TYPE: {
       InfraredRFTransmitRawTimingsRequest msg;
       msg.decode(msg_data, msg_size);
@@ -722,6 +722,44 @@ void APIConnection::read_message_(uint32_t msg_size, uint32_t msg_type, const ui
       this->on_serial_proxy_set_mode_request(msg);
       break;
     }
+#endif
+#ifdef USE_SERIAL_PROXY
+    case 154 /* SubscribeSerialProxyIdentityRequest is empty */: {
+#ifdef HAS_PROTO_MESSAGE_DUMP
+      this->log_receive_message_(LOG_STR("on_subscribe_serial_proxy_identity_request"));
+#endif
+      this->on_subscribe_serial_proxy_identity_request();
+      break;
+    }
+#endif
+#ifdef USE_API_WIZARD
+    case 156 /* DeviceWizardRequest is empty */: {
+#ifdef HAS_PROTO_MESSAGE_DUMP
+      this->log_receive_message_(LOG_STR("on_device_wizard_request"));
+#endif
+      this->on_device_wizard_request();
+      break;
+    }
+#endif
+#ifdef USE_API_WIZARD_INPUTS
+    case WizardInputSetRequest::MESSAGE_TYPE: {
+      WizardInputSetRequest msg;
+      msg.decode(msg_data, msg_size);
+#ifdef HAS_PROTO_MESSAGE_DUMP
+      this->log_receive_message_(LOG_STR("on_wizard_input_set_request"), msg);
+#endif
+      this->on_wizard_input_set_request(msg);
+      break;
+    }
+#endif
+#ifdef USE_SENDSPIN
+    case 159 /* SendspinPairingTokenRequest is empty */: {
+#ifdef HAS_PROTO_MESSAGE_DUMP
+      this->log_receive_message_(LOG_STR("on_sendspin_pairing_token_request"));
+#endif
+      this->on_sendspin_pairing_token_request();
+      break;
+    }
 #endif
     default:
       break;

esphome/components/api/api_pb2_service.h

@@ -29,6 +29,13 @@ class APIServerConnectionBase {
 
   void on_device_capabilities_request(){};
 
+#ifdef USE_API_WIZARD
+  void on_device_wizard_request(){};
+#endif
+
+#ifdef USE_API_WIZARD_INPUTS
+  void on_wizard_input_set_request(const WizardInputSetRequest &value){};
+#endif
   void on_list_entities_request(){};
 
   void on_subscribe_states_request(){};
@@ -213,7 +220,7 @@ class APIServerConnectionBase {
   void on_z_wave_proxy_request(const ZWaveProxyRequest &value){};
 #endif
 
-#if defined(USE_IR_RF) || defined(USE_RADIO_FREQUENCY)
+#ifdef USE_IR_RF
   void on_infrared_rf_transmit_raw_timings_request(const InfraredRFTransmitRawTimingsRequest &value){};
 #endif
 
@@ -238,9 +245,17 @@ class APIServerConnectionBase {
 #ifdef USE_SERIAL_PROXY
   void on_serial_proxy_set_mode_request(const SerialProxySetModeRequest &value){};
 #endif
+#ifdef USE_SERIAL_PROXY
+  void on_subscribe_serial_proxy_identity_request(){};
+#endif
+
 #ifdef USE_BLUETOOTH_PROXY_CONNECTIONS
   void on_bluetooth_set_connection_params_request(const BluetoothSetConnectionParamsRequest &value){};
 #endif
+
+#ifdef USE_SENDSPIN
+  void on_sendspin_pairing_token_request(){};
+#endif
 };
 
 }  // namespace esphome::api

esphome/components/api/api_server.cpp

@@ -5,7 +5,6 @@
 #include "api_connection.h"
 #include "esphome/components/network/util.h"
 #include "esphome/core/application.h"
-#include "esphome/core/controller_registry.h"
 #include "esphome/core/defines.h"
 #include "esphome/core/hal.h"
 #include "esphome/core/log.h"
@@ -29,22 +28,47 @@ static const char *const TAG = "api";
 // APIServer
 APIServer *global_api_server = nullptr;  // NOLINT(cppcoreguidelines-avoid-non-const-global-variables)
 
+#ifdef USE_API_NOISE
+static constexpr uint32_t NOISE_PSK_PREF_HASH = 88491486UL;
+#endif
+
+#if defined(USE_API_NOISE) && defined(USE_OTA_ENCRYPTION_PROVISIONED)
+bool load_saved_noise_psk(noise::psk_t &out) {
+  SavedNoisePsk saved;
+#ifdef USE_PREFERENCE_KEY_LOOKUP
+  const bool loaded =
+      global_preferences->load_from_key(NOISE_PSK_PREF_HASH, reinterpret_cast<uint8_t *>(&saved), sizeof(saved));
+#else
+  // Slot backends need the reservation walk; it only lands on the record when the reservations before
+  // it match a normal boot, otherwise the type checked checksum fails the load
+  const bool loaded = global_preferences->make_preference<SavedNoisePsk>(NOISE_PSK_PREF_HASH, true).load(&saved);
+#endif
+  // The all-zeros record means no key
+  if (!loaded || noise::NoiseContext::is_all_zeros(saved.psk))
+    return false;
+  out = saved.psk;
+  return true;
+}
+#endif
+
 APIServer::APIServer() { global_api_server = this; }
 
 void APIServer::socket_failed_(const LogString *msg) {
   ESP_LOGW(TAG, "Socket %s: errno %d", LOG_STR_ARG(msg), errno);
   this->destroy_socket_();
+#ifdef USE_API_OUTGOING_CONNECTION
+  // Dial-out needs no listener; degrade instead of stopping the component
+  this->status_set_error(LOG_STR("listen socket failed"));
+#else
   this->mark_failed();
+#endif
 }
 
 void APIServer::setup() {
-  ControllerRegistry::register_controller(this);
-
 #ifdef USE_API_NOISE
   // Always reserve the slot: flash preferences are positional on esp8266, so
   // a yaml key build must keep the layout of a runtime key build
-  uint32_t hash = 88491486UL;
-  this->noise_pref_ = global_preferences->make_preference<SavedNoisePsk>(hash, true);
+  this->noise_pref_ = global_preferences->make_preference<SavedNoisePsk>(NOISE_PSK_PREF_HASH, true);
 #ifndef USE_API_NOISE_PSK_FROM_YAML
   // A cleared record loads fine but holds no key
   if (this->load_and_apply_noise_psk_() && this->noise_ctx_.has_psk()) {
@@ -53,43 +77,6 @@ void APIServer::setup() {
 #endif
 #endif
 
-  this->socket_ = socket::socket_ip_loop_monitored(SOCK_STREAM, 0).release();  // monitored for incoming connections
-  if (this->socket_ == nullptr) {
-    this->socket_failed_(LOG_STR("creation"));
-    return;
-  }
-  int enable = 1;
-  int err = this->socket_->setsockopt(SOL_SOCKET, SO_REUSEADDR, &enable, sizeof(int));
-  if (err != 0) {
-    ESP_LOGW(TAG, "Socket reuseaddr: errno %d", errno);
-    // we can still continue
-  }
-  err = this->socket_->setblocking(false);
-  if (err != 0) {
-    this->socket_failed_(LOG_STR("nonblocking"));
-    return;
-  }
-
-  struct sockaddr_storage server;
-
-  socklen_t sl = socket::set_sockaddr_any((struct sockaddr *) &server, sizeof(server), this->port_);
-  if (sl == 0) {
-    this->socket_failed_(LOG_STR("set sockaddr"));
-    return;
-  }
-
-  err = this->socket_->bind((struct sockaddr *) &server, sl);
-  if (err != 0) {
-    this->socket_failed_(LOG_STR("bind"));
-    return;
-  }
-
-  err = this->socket_->listen(this->listen_backlog_);
-  if (err != 0) {
-    this->socket_failed_(LOG_STR("listen"));
-    return;
-  }
-
 #ifdef USE_LOGGER
   if (logger::global_logger != nullptr) {
     logger::global_logger->add_log_callback(
@@ -135,6 +122,47 @@ void APIServer::setup() {
   if (this->reboot_timeout_ != 0 && !this->provisioning_pending_()) {
     this->status_set_warning(LOG_STR("waiting for client connection"));
   }
+#ifdef USE_API_OUTGOING_CONNECTION
+  this->outgoing_conn_.setup();
+#endif
+
+  // Listener last: on failure socket_failed_() returns early, and an
+  // outgoing_connection build keeps dialing out without one
+  this->socket_ = socket::socket_ip_loop_monitored(SOCK_STREAM, 0).release();  // monitored for incoming connections
+  if (this->socket_ == nullptr) {
+    this->socket_failed_(LOG_STR("creation"));
+    return;
+  }
+  int enable = 1;
+  int err = this->socket_->setsockopt(SOL_SOCKET, SO_REUSEADDR, &enable, sizeof(int));
+  if (err != 0) {
+    ESP_LOGW(TAG, "Socket reuseaddr: errno %d", errno);
+    // we can still continue
+  }
+  err = this->socket_->setblocking(false);
+  if (err != 0) {
+    this->socket_failed_(LOG_STR("nonblocking"));
+    return;
+  }
+
+  struct sockaddr_storage server;
+
+  socklen_t sl = socket::set_sockaddr_any((struct sockaddr *) &server, sizeof(server), this->port_);
+  if (sl == 0) {
+    this->socket_failed_(LOG_STR("set sockaddr"));
+    return;
+  }
+
+  err = this->socket_->bind((struct sockaddr *) &server, sl);
+  if (err != 0) {
+    this->socket_failed_(LOG_STR("bind"));
+    return;
+  }
+
+  err = this->socket_->listen(this->listen_backlog_);
+  if (err != 0) {
+    this->socket_failed_(LOG_STR("listen"));
+  }
 }
 
 void APIServer::loop() {
@@ -143,6 +171,19 @@ void APIServer::loop() {
     this->accept_new_connections_();
   }
 
+  const bool connected = network::is_connected();
+#ifdef USE_NOISE_SPARE_EPHEMERAL
+  if (connected && !noise::has_spare_ephemeral()) {
+    this->refill_spare_ephemeral_();
+  }
+#endif
+
+#ifdef USE_API_OUTGOING_CONNECTION
+  if (!this->shutting_down_) {
+    this->outgoing_conn_.loop(this);
+  }
+#endif
+
   if (this->api_connection_count_ == 0) {
     // Check reboot timeout - done in loop to avoid scheduler heap churn
     // (cancelled scheduler items sit in heap memory until their scheduled time).
@@ -159,8 +200,7 @@ void APIServer::loop() {
   }
 
   // Process clients and remove disconnected ones in a single pass
-  // Check network connectivity once for all clients
-  if (!network::is_connected()) {
+  if (!connected) {
     // Network is down - disconnect all clients
     for (auto &client : this->active_clients()) {
       client->on_fatal_error();
@@ -188,6 +228,19 @@ void APIServer::loop() {
   }
 }
 
+#ifdef USE_NOISE_SPARE_EPHEMERAL
+// An OTA handshake is not visible here and just pays the refill it triggered
+void APIServer::refill_spare_ephemeral_() {
+  const uint32_t now = App.get_loop_component_start_time();
+  for (auto &client : this->active_clients()) {
+    if (client->is_still_connecting(now)) {
+      return;
+    }
+  }
+  noise::prepare_spare_ephemeral();
+}
+#endif
+
 void APIServer::remove_client_(uint8_t client_index) {
   auto &client = this->clients_[client_index];
 
@@ -203,6 +256,15 @@ void APIServer::remove_client_(uint8_t client_index) {
   std::string client_peername(client->get_peername_to(peername_buf));
 #endif
 
+  // Read before the swap-and-reset below destroys the connection
+  const bool was_authenticated = client->is_authenticated();
+#ifdef USE_API_OUTGOING_CONNECTION
+  if (client->flags_.outgoing_connection_target) {
+    this->outgoing_target_count_--;
+  }
+  this->outgoing_conn_.on_client_removed(client.get(), was_authenticated);
+#endif
+
   // Close socket now (was deferred from on_fatal_error to allow getpeername)
   client->helper_->close();
 
@@ -221,9 +283,15 @@ void APIServer::remove_client_(uint8_t client_index) {
 
   // Last client disconnected - set warning and start tracking for reboot timeout
   // (suppressed while provisioning is pending - see loop()).
+  // Refresh on every authenticated removal, not just the last one, so an
+  // unauthenticated straggler removed later (e.g. a port scan, or a dial to
+  // a host that accepts TCP but never speaks the API) cannot discard a
+  // healthy session's timestamp and trigger a spurious reboot
+  if (was_authenticated) {
+    this->last_connected_ = App.get_loop_component_start_time();
+  }
   if (this->api_connection_count_ == 0 && this->reboot_timeout_ != 0 && !this->provisioning_pending_()) {
     this->status_set_warning(LOG_STR("waiting for client connection"));
-    this->last_connected_ = App.get_loop_component_start_time();
   }
 
 #ifdef USE_API_CLIENT_DISCONNECTED_TRIGGER
@@ -245,7 +313,7 @@ void __attribute__((flatten)) APIServer::accept_new_connections_() {
     sock->getpeername_to(peername);
 
     // Check if we're at the connection limit
-    if (this->api_connection_count_ >= MAX_API_CONNECTIONS) {
+    if (this->at_client_limit_()) {
       ESP_LOGW(TAG, "Max connections (%d), rejecting %s", MAX_API_CONNECTIONS, peername);
       // Immediately close - socket destructor will handle cleanup
       sock.reset();
@@ -254,18 +322,47 @@ void __attribute__((flatten)) APIServer::accept_new_connections_() {
 
     ESP_LOGD(TAG, "Accept %s", peername);
 
-    auto *conn = new APIConnection(std::move(sock), this);
-    this->clients_[this->api_connection_count_++].reset(conn);
-    conn->start();
-
-    // First client connected - clear warning and update timestamp
-    if (this->api_connection_count_ == 1 && this->reboot_timeout_ != 0 && !this->provisioning_pending_()) {
-      this->status_clear_warning();
-      this->last_connected_ = App.get_loop_component_start_time();
-    }
+    this->add_client_(std::move(sock));
   }
 }
 
+APIConnection *APIServer::add_client_(std::unique_ptr<socket::Socket> sock) {
+  auto *conn = new APIConnection(std::move(sock), this);  // NOLINT(cppcoreguidelines-owning-memory)
+  this->clients_[this->api_connection_count_++].reset(conn);
+  conn->start();
+
+  // First client connected - clear warning. The reboot watchdog timestamp is
+  // refreshed when an authenticated client is removed (see remove_client_),
+  // never on bare TCP connects.
+  if (this->api_connection_count_ == 1 && this->reboot_timeout_ != 0 && !this->provisioning_pending_()) {
+    this->status_clear_warning();
+  }
+  return conn;
+}
+
+#ifdef USE_API_OUTGOING_CONNECTION
+APIConnection *APIServer::add_outgoing_client_(std::unique_ptr<socket::Socket> sock) {
+  // Re-check at the handoff: inbound clients may have taken the last slot and
+  // the PSK may have been cleared since the dial started (mark_outgoing()
+  // needs the noise helper)
+  const bool at_limit = this->at_client_limit_();
+  if (at_limit || !this->noise_ctx_.has_psk()) {
+    ESP_LOGW(TAG, "Dropping outgoing connection (%s)",
+             at_limit ? LOG_STR_LITERAL("max connections") : LOG_STR_LITERAL("no key"));
+    return nullptr;
+  }
+  auto *conn = this->add_client_(std::move(sock));
+  // After start(): sends our server hello first so the peer can pick the key
+  conn->mark_outgoing();
+  return conn;
+}
+
+void APIServer::on_outgoing_target_client(APIConnection *conn) {
+  this->outgoing_target_count_++;
+  this->outgoing_conn_.on_target_client(conn);
+}
+#endif
+
 void APIServer::dump_config() {
   char addr_buf[network::USE_ADDRESS_BUFFER_SIZE];
   ESP_LOGCONFIG(TAG,
@@ -282,6 +379,9 @@ void APIServer::dump_config() {
 #else
   ESP_LOGCONFIG(TAG, "  Noise encryption: NO");
 #endif
+#ifdef USE_API_OUTGOING_CONNECTION
+  this->outgoing_conn_.dump_config();
+#endif
 }
 
 void APIServer::handle_disconnect(APIConnection *conn) {}
@@ -404,7 +504,16 @@ void APIServer::on_zwave_proxy_request(const ZWaveProxyRequest &msg) {
 }
 #endif
 
-#if defined(USE_IR_RF) || defined(USE_RADIO_FREQUENCY)
+#ifdef USE_SERIAL_PROXY_USB_IDENTITY
+void APIServer::send_serial_proxy_identity(const SerialProxyIdentity &msg) {
+  for (auto &c : this->active_clients()) {
+    if (c->flags_.serial_proxy_identity_subscription)
+      c->send_serial_proxy_identity(msg);
+  }
+}
+#endif
+
+#ifdef USE_IR_RF
 void APIServer::send_infrared_rf_receive_event([[maybe_unused]] uint32_t device_id, uint32_t key,
                                                const std::vector<int32_t> *timings) {
   InfraredRFReceiveEvent resp{};
@@ -417,6 +526,7 @@ void APIServer::send_infrared_rf_receive_event([[maybe_unused]] uint32_t device_
   for (auto &c : this->active_clients())
     c->send_infrared_rf_receive_event(resp);
 }
+
 #endif
 
 #ifdef USE_ALARM_CONTROL_PANEL
@@ -433,8 +543,9 @@ void APIServer::send_homeassistant_action(const HomeassistantActionRequest &call
     // Home Assistant subscribes to actions shortly *after* authenticating, so actions
     // fired right at connection time (on_client_connected, on_time_sync, ...) can
     // arrive before the subscription and are lost - warn instead of failing silently.
-    ESP_LOGW(TAG, "Home Assistant %s '%s' dropped; %s",
-             call.is_event ? LOG_STR_LITERAL("event") : LOG_STR_LITERAL("action"), call.service.c_str(),
+    ESP_LOGW(TAG, "Home Assistant %s '%.*s' dropped; %s",
+             call.is_event ? LOG_STR_LITERAL("event") : LOG_STR_LITERAL("action"),
+             static_cast<int>(call.service.size()), call.service.empty() ? "" : call.service.c_str(),
              this->is_connected() ? LOG_STR_LITERAL("client has not subscribed to actions (yet)")
                                   : LOG_STR_LITERAL("no client connected"));
   }
@@ -577,6 +688,8 @@ bool APIServer::update_noise_psk_(const SavedNoisePsk &new_psk, const LogString
         if (!c->send_message(req)) {
           API_LOG_MSG_DROPPED(TAG, "Disconnect request");
         }
+        // Force it: a session from before the key was active must not survive
+        c->flags_.next_close = true;
       }
     });
   }
@@ -678,6 +791,9 @@ void APIServer::on_shutdown() {
 
   // Close the listening socket to prevent new connections
   this->destroy_socket_();
+#ifdef USE_API_OUTGOING_CONNECTION
+  this->outgoing_conn_.on_shutdown();
+#endif
 
   // Change batch delay to 5ms for quick flushing during shutdown
   this->batch_delay_ = 5;

esphome/components/api/api_server.h

@@ -5,16 +5,17 @@
 #include "api_buffer.h"
 // Must precede clients_ so APIConnection is complete for default_delete (libc++).
 #include "api_connection.h"
-#ifdef USE_API_NOISE
+#if defined(USE_API_NOISE) || defined(USE_NOISE_SPARE_EPHEMERAL)
 // Only present in the build when the noise component is loaded
 #include "esphome/components/noise/noise.h"
 #endif
 #include "api_pb2.h"
 #include "api_pb2_service.h"
+#include "api_outgoing_connection.h"
 #include "esphome/components/socket/socket.h"
 #include "esphome/core/automation.h"
 #include "esphome/core/component.h"
-#include "esphome/core/controller.h"
+#include "esphome/core/entity_includes.h"
 #include "esphome/core/log.h"
 #include "esphome/core/string_ref.h"
 #ifdef USE_PROVISIONING
@@ -43,9 +44,13 @@ struct SavedNoisePsk {
   noise::psk_t psk;
 } PACKED;  // NOLINT
 #endif
+#if defined(USE_API_NOISE) && defined(USE_OTA_ENCRYPTION_PROVISIONED)
+/// One-shot read of the provisioned key for a boot without an api server (safe mode); false when
+/// there is no key
+bool load_saved_noise_psk(noise::psk_t &out);
+#endif
 
-class APIServer final : public Component,
-                        public Controller
+class APIServer final : public Component
 #ifdef USE_CAMERA
     ,
                         public camera::CameraListener
@@ -86,61 +91,65 @@ class APIServer final : public Component,
   void set_noise_psk(const uint8_t *psk) { this->noise_ctx_.set_psk(psk); }
   noise::NoiseContext &get_noise_ctx() { return this->noise_ctx_; }
 #endif  // USE_API_NOISE
+#ifdef USE_API_OUTGOING_CONNECTION
+  // Called by APIConnection when a client declares itself a dial-back target in its hello
+  void on_outgoing_target_client(APIConnection *conn);
+#endif
 
   void handle_disconnect(APIConnection *conn);
 #ifdef USE_BINARY_SENSOR
-  void on_binary_sensor_update(binary_sensor::BinarySensor *obj) override;
+  void on_binary_sensor_update(binary_sensor::BinarySensor *obj);
 #endif
 #ifdef USE_COVER
-  void on_cover_update(cover::Cover *obj) override;
+  void on_cover_update(cover::Cover *obj);
 #endif
 #ifdef USE_FAN
-  void on_fan_update(fan::Fan *obj) override;
+  void on_fan_update(fan::Fan *obj);
 #endif
 #ifdef USE_LIGHT
-  void on_light_update(light::LightState *obj) override;
+  void on_light_update(light::LightState *obj);
 #endif
 #ifdef USE_SENSOR
-  void on_sensor_update(sensor::Sensor *obj) override;
+  void on_sensor_update(sensor::Sensor *obj);
 #endif
 #ifdef USE_SWITCH
-  void on_switch_update(switch_::Switch *obj) override;
+  void on_switch_update(switch_::Switch *obj);
 #endif
 #ifdef USE_TEXT_SENSOR
-  void on_text_sensor_update(text_sensor::TextSensor *obj) override;
+  void on_text_sensor_update(text_sensor::TextSensor *obj);
 #endif
 #ifdef USE_CLIMATE
-  void on_climate_update(climate::Climate *obj) override;
+  void on_climate_update(climate::Climate *obj);
 #endif
 #ifdef USE_NUMBER
-  void on_number_update(number::Number *obj) override;
+  void on_number_update(number::Number *obj);
 #endif
 #ifdef USE_DATETIME_DATE
-  void on_date_update(datetime::DateEntity *obj) override;
+  void on_date_update(datetime::DateEntity *obj);
 #endif
 #ifdef USE_DATETIME_TIME
-  void on_time_update(datetime::TimeEntity *obj) override;
+  void on_time_update(datetime::TimeEntity *obj);
 #endif
 #ifdef USE_DATETIME_DATETIME
-  void on_datetime_update(datetime::DateTimeEntity *obj) override;
+  void on_datetime_update(datetime::DateTimeEntity *obj);
 #endif
 #ifdef USE_TEXT
-  void on_text_update(text::Text *obj) override;
+  void on_text_update(text::Text *obj);
 #endif
 #ifdef USE_SELECT
-  void on_select_update(select::Select *obj) override;
+  void on_select_update(select::Select *obj);
 #endif
 #ifdef USE_LOCK
-  void on_lock_update(lock::Lock *obj) override;
+  void on_lock_update(lock::Lock *obj);
 #endif
 #ifdef USE_VALVE
-  void on_valve_update(valve::Valve *obj) override;
+  void on_valve_update(valve::Valve *obj);
 #endif
 #ifdef USE_MEDIA_PLAYER
-  void on_media_player_update(media_player::MediaPlayer *obj) override;
+  void on_media_player_update(media_player::MediaPlayer *obj);
 #endif
 #ifdef USE_WATER_HEATER
-  void on_water_heater_update(water_heater::WaterHeater *obj) override;
+  void on_water_heater_update(water_heater::WaterHeater *obj);
 #endif
 #ifdef USE_API_HOMEASSISTANT_SERVICES
   void send_homeassistant_action(const HomeassistantActionRequest &call);
@@ -183,18 +192,22 @@ class APIServer final : public Component,
 #endif
 
 #ifdef USE_ALARM_CONTROL_PANEL
-  void on_alarm_control_panel_update(alarm_control_panel::AlarmControlPanel *obj) override;
+  void on_alarm_control_panel_update(alarm_control_panel::AlarmControlPanel *obj);
 #endif
 #ifdef USE_EVENT
-  void on_event(event::Event *obj) override;
+  void on_event(event::Event *obj);
 #endif
 #ifdef USE_UPDATE
-  void on_update(update::UpdateEntity *obj) override;
+  void on_update(update::UpdateEntity *obj);
 #endif
 #ifdef USE_ZWAVE_PROXY
   void on_zwave_proxy_request(const ZWaveProxyRequest &msg);
 #endif
-#if defined(USE_IR_RF) || defined(USE_RADIO_FREQUENCY)
+#ifdef USE_SERIAL_PROXY_USB_IDENTITY
+  /// Tell every subscribed client that a serial proxy port's identity changed
+  void send_serial_proxy_identity(const SerialProxyIdentity &msg);
+#endif
+#ifdef USE_IR_RF
   void send_infrared_rf_receive_event(uint32_t device_id, uint32_t key, const std::vector<int32_t> *timings);
 #endif
 
@@ -263,6 +276,16 @@ class APIServer final : public Component,
  protected:
   // Accept incoming socket connections. Only called when socket has pending connections.
   void __attribute__((noinline)) accept_new_connections_();
+  /// Takes the socket into a new connection and starts it; callers must have
+  /// checked at_client_limit_() first
+  APIConnection *add_client_(std::unique_ptr<socket::Socket> sock);
+  bool at_client_limit_() const { return this->api_connection_count_ >= MAX_API_CONNECTIONS; }
+#ifdef USE_API_OUTGOING_CONNECTION
+  // Returns the new connection, or nullptr (socket dropped) when at the limit
+  APIConnection *add_outgoing_client_(std::unique_ptr<socket::Socket> sock);
+  bool has_outgoing_target_client_() const { return this->outgoing_target_count_ != 0; }
+  friend class OutgoingConnectionManager;
+#endif
   // Remove a disconnected client by index. Swaps with the last populated slot and resets it.
   void __attribute__((noinline)) remove_client_(uint8_t client_index);
 
@@ -303,6 +326,8 @@ class APIServer final : public Component,
     delete this->socket_;
     this->socket_ = nullptr;
   }
+  /// Log the failure, drop the listen socket, and mark the component failed
+  /// unless this build can still dial out
   void socket_failed_(const LogString *msg);
   // Pointers and pointer-like types first (4 bytes each)
   socket::ListenSocket *socket_{nullptr};
@@ -314,7 +339,7 @@ class APIServer final : public Component,
 #endif
 
   // 4-byte aligned types
-  uint32_t reboot_timeout_{300000};
+  uint32_t reboot_timeout_{900000};  // Keep in sync with DEFAULT_REBOOT_TIMEOUT in __init__.py
   uint32_t last_connected_{0};
 
   // Slots [0, api_connection_count_) are populated; trailing slots are always nullptr.
@@ -351,18 +376,23 @@ class APIServer final : public Component,
 #endif
 
   // Group smaller types together
-  uint16_t port_{6053};
-  uint16_t batch_delay_{100};
-  // Connection limits - these defaults will be overridden by config values
-  // from cv.SplitDefault in __init__.py which sets platform-specific defaults.
-  uint8_t listen_backlog_{4};
+  uint16_t port_{6053};        // Keep in sync with DEFAULT_PORT in __init__.py
+  uint16_t batch_delay_{100};  // Keep in sync with DEFAULT_BATCH_DELAY in __init__.py
+  uint8_t listen_backlog_{4};  // Keep in sync with DEFAULT_LISTEN_BACKLOG in __init__.py
   bool shutting_down_ = false;
   uint8_t api_connection_count_{0};
+#ifdef USE_API_OUTGOING_CONNECTION
+  // Connected clients whose hello declared them a dial-back target
+  uint8_t outgoing_target_count_{0};
+#endif
 #if defined(USE_PROVISIONING) && defined(USE_API_NOISE)
   // Index assigned by the provisioning manager for reporting this transport's state.
   uint8_t provisioning_source_{0};
 #endif
 
+#ifdef USE_NOISE_SPARE_EPHEMERAL
+  void refill_spare_ephemeral_();
+#endif
 #ifdef USE_API_NOISE
   noise::NoiseContext noise_ctx_;
 #ifndef USE_API_NOISE_PSK_FROM_YAML
@@ -370,6 +400,9 @@ class APIServer final : public Component,
 #endif
   ESPPreferenceObject noise_pref_;
 #endif  // USE_API_NOISE
+#ifdef USE_API_OUTGOING_CONNECTION
+  OutgoingConnectionManager outgoing_conn_;
+#endif
 };
 
 extern APIServer *global_api_server;  // NOLINT(cppcoreguidelines-avoid-non-const-global-variables)

esphome/components/api/api_wizard.cpp

@@ -0,0 +1,86 @@
+#include "api_wizard.h"
+
+#ifdef USE_API_WIZARD
+
+#include <cstring>
+
+#include "api_connection.h"
+#include "api_pb2.h"
+#include "api_server.h"
+#include "esphome/core/log.h"
+
+namespace esphome::api {
+
+static const char *const TAG = "api.wizard";
+
+#ifdef USE_API_WIZARD_INPUTS
+static bool wizard_entity_id_valid(const char *entity_id, size_t length) {
+  return length > 0 && length < WIZARD_ENTITY_ID_BUFFER_SIZE && memchr(entity_id, '.', length) != nullptr;
+}
+
+const char *wizard_set_input(const WizardInputSetRequest &msg) {
+  if (!wizard_entity_id_valid(msg.entity_id.c_str(), msg.entity_id.size())) {
+    ESP_LOGW(TAG, "Ignoring an invalid entity id for wizard input");
+    return nullptr;
+  }
+  for (size_t i = 0; i < API_WIZARD_INPUT_COUNT; i++) {
+    // The table is in flash, which ESP8266 can only read through progmem_memcpy
+    WizardInputEntry entry;
+    progmem_memcpy(&entry, &API_WIZARD_INPUTS[i], sizeof(entry));
+    if (entry.key != msg.key)
+      continue;
+    memcpy(entry.entity_id, msg.entity_id.c_str(), msg.entity_id.size());
+    entry.entity_id[msg.entity_id.size()] = '\0';
+    return entry.entity_id;
+  }
+  ESP_LOGW(TAG, "Ignoring an entity id for an unknown wizard input");
+  return nullptr;
+}
+#endif  // USE_API_WIZARD_INPUTS
+
+void APIConnection::on_device_wizard_request() {
+  DeviceWizardResponse resp;
+  resp.data = API_WIZARD_DATA;
+  resp.data_len = API_WIZARD_DATA_SIZE;
+  if (!this->send_message(resp)) {
+    this->on_fatal_error();
+  }
+}
+
+#ifdef USE_API_WIZARD_INPUTS
+void APIConnection::on_wizard_input_set_request(const WizardInputSetRequest &msg) {
+  const char *entity_id = wizard_set_input(msg);
+  if (entity_id == nullptr)
+    return;
+#ifdef USE_API_WIZARD_LINKED_INPUTS
+  // Entities subscribed to the buffer before it held an entity id, so every client needs to learn of it now
+  for (auto &client : this->parent_->active_clients()) {
+    client->resend_state_subscriptions(entity_id);
+  }
+#endif
+}
+#endif
+
+#ifdef USE_API_WIZARD_LINKED_INPUTS
+void APIConnection::resend_state_subscriptions(const char *entity_id) {
+  if (!this->flags_.home_assistant_states)
+    return;
+  for (const auto &it : this->parent_->get_state_subs()) {
+    if (it.entity_id != entity_id)
+      continue;
+    SubscribeHomeAssistantStateResponse resp;
+    resp.entity_id = StringRef(it.entity_id);
+    resp.attribute = it.attribute != nullptr ? StringRef(it.attribute) : StringRef("");
+    resp.once = it.once;
+    if (!this->send_message(resp)) {
+      // Could not send now: send every subscription again from the loop
+      this->state_subs_at_ = 0;
+      return;
+    }
+  }
+}
+#endif
+
+}  // namespace esphome::api
+
+#endif  // USE_API_WIZARD

esphome/components/api/api_wizard.h

@@ -0,0 +1,58 @@
+#pragma once
+
+#include "esphome/core/defines.h"
+
+#ifdef USE_API_WIZARD
+
+#include <cstddef>
+#include <cstdint>
+
+#include "esphome/core/hal.h"
+#include "proto.h"
+#include "esphome/core/string_ref.h"
+
+namespace esphome::api {
+
+class WizardInputSetRequest;
+
+/// Size of the buffer holding the entity id of a wizard input. Home Assistant entity ids are at most 255 bytes.
+static constexpr size_t WIZARD_ENTITY_ID_BUFFER_SIZE = 256;
+
+/// The wizard as zstd compressed JSON (see DeviceWizardResponse), built by the generated code
+/// (components/api/wizard.py) and kept in flash. API_WIZARD_DATA_SIZE bytes long.
+extern const uint8_t API_WIZARD_DATA[] PROGMEM;
+
+#ifdef USE_API_WIZARD_INPUTS
+/// Where the entity id of an input is kept, found by the key the client uses for it.
+struct WizardInputEntry {
+  uint32_t key;     // FNV-1 hash of the ESPHome id of the input
+  char *entity_id;  // RAM buffer of WIZARD_ENTITY_ID_BUFFER_SIZE bytes, shared with the homeassistant entity
+};
+
+/// The inputs of the wizard, in flash. API_WIZARD_INPUT_COUNT entries long.
+extern const WizardInputEntry API_WIZARD_INPUTS[] PROGMEM;
+
+/// Apply a WizardInputSetRequest: validate it and copy the entity id into the input's buffer. Nothing is stored
+/// across restarts, so the client sends the choices again after every connection.
+/// Returns the buffer, or nullptr when the request was ignored.
+const char *wizard_set_input(const WizardInputSetRequest &msg);
+#endif
+
+#ifdef USE_API_WIZARD_STANDALONE_INPUTS
+/// An input of the wizard that is not tied to an entity of the device. The entity ID the user picks is only ever read
+/// by lambdas, for example `id(input).entity_id()`. It lives in the same RAM buffer a linked input uses.
+class WizardInput {
+ public:
+  explicit WizardInput(const char *entity_id) : entity_id_(entity_id) {}
+  /// The Home Assistant entity ID, empty until the wizard sets one.
+  StringRef entity_id() const { return StringRef(this->entity_id_); }
+  bool has_entity_id() const { return this->entity_id_[0] != '\0'; }
+
+ protected:
+  const char *entity_id_;
+};
+#endif
+
+}  // namespace esphome::api
+
+#endif  // USE_API_WIZARD

esphome/components/api/homeassistant_service.h

@@ -4,64 +4,40 @@
 #ifdef USE_API
 #ifdef USE_API_HOMEASSISTANT_SERVICES
 #include <functional>
+#include <string>
+#include <type_traits>
 #include <utility>
-#include <vector>
 #include "api_pb2.h"
 #ifdef USE_API_HOMEASSISTANT_ACTION_RESPONSES_JSON
 #include "esphome/components/json/json_util.h"
 #endif
 #include "esphome/core/automation.h"
 #include "esphome/core/helpers.h"
+#include "esphome/core/progmem.h"
 #include "esphome/core/string_ref.h"
 
 namespace esphome::api {
 
-template<typename... X> class TemplatableStringValue : public TemplatableValue<std::string, X...> {
-  // Verify that const char* uses the base class STATIC_STRING optimization (no heap allocation)
-  // rather than being wrapped in a lambda. The base class constructor for const char* is more
-  // specialized than the templated constructor here, so it should be selected.
-  static_assert(std::is_constructible_v<TemplatableValue<std::string, X...>, const char *>,
-                "Base class must have const char* constructor for STATIC_STRING optimization");
+// Converts a lambda result to the string sent to Home Assistant
+template<typename T>
+requires(!std::is_pointer_v<std::remove_cvref_t<T>>) std::string field_to_string(T &&val) {
+  return to_string(std::forward<T>(val));  // NOLINT
+}
+inline std::string field_to_string(const char *val) { return val ? std::string(val) : std::string(); }
+inline std::string field_to_string(std::string val) { return val; }
+inline std::string field_to_string(StringRef val) { return val.str(); }
 
- private:
-  // Helper to convert value to string - handles the case where value is already a string
-  template<typename T> static std::string value_to_string(T &&val) {
-    return to_string(std::forward<T>(val));  // NOLINT
+/// A key and value from codegen; on ESP8266 the table and its strings are in flash.
+/// The value is the constant `value`, or the result of `fn` when it is set.
+template<typename... Ts> struct HomeAssistantField {
+  const char *key;
+  const char *value;
+  std::string (*fn)(const Ts &...);
+
+  template<typename F> static constexpr HomeAssistantField from_lambda(const char *key, F /*lambda*/) {
+    return {key, nullptr, &call_lambda<F>};
   }
-
-  // Overloads for string types - needed because std::to_string doesn't support them
-  static std::string value_to_string(char *val) {
-    return val ? std::string(val) : std::string();
-  }  // For lambdas returning char* (e.g., itoa)
-  static std::string value_to_string(const char *val) { return std::string(val); }  // For lambdas returning .c_str()
-  static std::string value_to_string(const std::string &val) { return val; }
-  static std::string value_to_string(std::string &&val) { return std::move(val); }
-  static std::string value_to_string(const StringRef &val) { return val.str(); }
-  static std::string value_to_string(StringRef &&val) { return val.str(); }
-
- public:
-  TemplatableStringValue() : TemplatableValue<std::string, X...>() {}
-
-  template<typename F, enable_if_t<!is_invocable<F, X...>::value, int> = 0>
-  TemplatableStringValue(F value) : TemplatableValue<std::string, X...>(value) {}
-
-  template<typename F, enable_if_t<is_invocable<F, X...>::value, int> = 0>
-  TemplatableStringValue(F f)
-      : TemplatableValue<std::string, X...>([f](X... x) -> std::string { return value_to_string(f(x...)); }) {}
-};
-
-template<typename... Ts> class TemplatableKeyValuePair {
- public:
-  // Default constructor needed for FixedVector::emplace_back()
-  TemplatableKeyValuePair() = default;
-
-  // Keys are always string literals from YAML dictionary keys (e.g., "code", "event")
-  // and never templatable values or lambdas. Only the value parameter can be a lambda/template.
-  // Using const char* avoids std::string heap allocation - keys remain in flash.
-  template<typename T> TemplatableKeyValuePair(const char *key, T value) : key(key), value(value) {}
-
-  const char *key{nullptr};
-  TemplatableStringValue<Ts...> value;
+  template<typename F> static std::string call_lambda(const Ts &...x) { return field_to_string(F{}(x...)); }
 };
 
 #ifdef USE_API_HOMEASSISTANT_ACTION_RESPONSES
@@ -106,45 +82,20 @@ template<typename... Ts> using ActionResponseCallback = std::function<void(const
 
 template<typename... Ts> class HomeAssistantServiceCallAction final : public Action<Ts...> {
  public:
-  explicit HomeAssistantServiceCallAction(APIServer *parent, bool is_event) : parent_(parent) {
+  using Field = HomeAssistantField<Ts...>;
+
+  /// `fields` is a codegen table: the action or event name (no key), then the data, data_template
+  /// and variables entries.
+  HomeAssistantServiceCallAction(APIServer *parent, bool is_event, const Field *fields, uint8_t data_count,
+                                 uint8_t data_template_count, uint8_t variables_count)
+      : parent_(parent),
+        fields_(fields),
+        data_count_(data_count),
+        data_template_count_(data_template_count),
+        variables_count_(variables_count) {
     this->flags_.is_event = is_event;
   }
 
-  template<typename T> void set_service(T service) { this->service_ = service; }
-
-  // Initialize FixedVector members - called from Python codegen with compile-time known sizes.
-  // Must be called before any add_* methods; capacity must match the number of subsequent add_* calls.
-  void init_data(size_t count) { this->data_.init(count); }
-  void init_data_template(size_t count) { this->data_template_.init(count); }
-  void init_variables(size_t count) { this->variables_.init(count); }
-
-  // Keys are always string literals from the Python code generation (e.g., cg.add(var.add_data("tag_id", templ))).
-  // The value parameter can be a lambda/template, but keys are never templatable.
-  // Using const char* for keys avoids std::string heap allocation - keys remain in flash.
-  template<typename V> void add_data(const char *key, V &&value) {
-    this->add_kv_(this->data_, key, std::forward<V>(value));
-  }
-  template<typename V> void add_data_template(const char *key, V &&value) {
-    this->add_kv_(this->data_template_, key, std::forward<V>(value));
-  }
-  template<typename V> void add_variable(const char *key, V &&value) {
-    this->add_kv_(this->variables_, key, std::forward<V>(value));
-  }
-
-#ifdef USE_ESP8266
-  // On ESP8266, ESPHOME_F() returns __FlashStringHelper* (PROGMEM pointer).
-  // Store as const char* — populate_service_map copies from PROGMEM at play() time.
-  template<typename V> void add_data(const __FlashStringHelper *key, V &&value) {
-    this->add_kv_(this->data_, reinterpret_cast<const char *>(key), std::forward<V>(value));
-  }
-  template<typename V> void add_data_template(const __FlashStringHelper *key, V &&value) {
-    this->add_kv_(this->data_template_, reinterpret_cast<const char *>(key), std::forward<V>(value));
-  }
-  template<typename V> void add_variable(const __FlashStringHelper *key, V &&value) {
-    this->add_kv_(this->variables_, reinterpret_cast<const char *>(key), std::forward<V>(value));
-  }
-#endif
-
 #ifdef USE_API_HOMEASSISTANT_ACTION_RESPONSES
   template<typename T> void set_response_template(T response_template) {
     this->response_template_ = response_template;
@@ -162,19 +113,61 @@ template<typename... Ts> class HomeAssistantServiceCallAction final : public Act
 #endif  // USE_API_HOMEASSISTANT_ACTION_RESPONSES
 
   void play(const Ts &...x) override {
+    const Field *fields = this->fields_;
+    const size_t total = 1 + this->data_count_ + this->data_template_count_ + this->variables_count_;
+
+    // Lambda results, and on ESP8266 the RAM copies of the flash strings, must live until the send
+    size_t lambda_count = 0;
+#ifdef USE_ESP8266
+    size_t flash_len = 0;
+#endif
+    for (size_t i = 0; i < total; i++) {
+      lambda_count += fields[i].fn != nullptr;
+#ifdef USE_ESP8266
+      if (fields[i].fn == nullptr)
+        flash_len += ESPHOME_strlen_P(fields[i].value);
+      if (fields[i].key != nullptr)
+        flash_len += ESPHOME_strlen_P(fields[i].key);
+#endif
+    }
+    FixedVector<std::string> results;
+    results.init(lambda_count);
+#ifdef USE_ESP8266
+    SmallBufferWithHeapFallback<128, char> flash_copy(flash_len);
+    char *cursor = flash_copy.get();
+#endif
+    auto string_ref = [&](const char *str) {
+#ifdef USE_ESP8266
+      size_t len = ESPHOME_strlen_P(str);
+      memcpy_P(cursor, str, len);
+      StringRef ref(cursor, len);
+      cursor += len;
+      return ref;
+#else
+      return StringRef(str);
+#endif
+    };
+    auto value_ref = [&](const Field &field) {
+      if (field.fn == nullptr)
+        return string_ref(field.value);
+      results.push_back(field.fn(x...));
+      return StringRef(results.back());
+    };
+    auto fill = [&](FixedVector<HomeassistantServiceMap> &dest, uint8_t count) {
+      dest.init(count);
+      for (uint8_t i = 0; i < count; i++, fields++) {
+        auto &kv = dest.emplace_back();
+        kv.key = string_ref(fields->key);
+        kv.value = value_ref(*fields);
+      }
+    };
+
     HomeassistantActionRequest resp;
-    std::string service_value = this->service_.value(x...);
-    resp.service = StringRef(service_value);
+    resp.service = value_ref(*fields++);
     resp.is_event = this->flags_.is_event;
-
-    // Local storage for lambda-evaluated strings - lives until after send
-    FixedVector<std::string> data_storage;
-    FixedVector<std::string> data_template_storage;
-    FixedVector<std::string> variables_storage;
-
-    this->populate_service_map(resp.data, this->data_, data_storage, x...);
-    this->populate_service_map(resp.data_template, this->data_template_, data_template_storage, x...);
-    this->populate_service_map(resp.variables, this->variables_, variables_storage, x...);
+    fill(resp.data, this->data_count_);
+    fill(resp.data_template, this->data_template_count_);
+    fill(resp.variables, this->variables_count_);
 
 #ifdef USE_API_HOMEASSISTANT_ACTION_RESPONSES
 #ifdef USE_API_HOMEASSISTANT_ACTION_RESPONSES_JSON
@@ -223,90 +216,26 @@ template<typename... Ts> class HomeAssistantServiceCallAction final : public Act
   }
 
  protected:
-  // Helper to add key-value pairs to FixedVectors
-  // Keys are always string literals (const char*), values can be lambdas/templates
-  template<typename V> void add_kv_(FixedVector<TemplatableKeyValuePair<Ts...>> &vec, const char *key, V &&value) {
-    auto &kv = vec.emplace_back();
-    kv.key = key;
-    kv.value = std::forward<V>(value);
-  }
-
-  template<typename VectorType, typename SourceType>
-  static void populate_service_map(VectorType &dest, SourceType &source, FixedVector<std::string> &value_storage,
-                                   Ts... x) {
-    dest.init(source.size());
-
-#ifdef USE_ESP8266
-    // On ESP8266, all static strings from codegen are FLASH_STRING (PROGMEM),
-    // so is_static_string() is always false — the zero-copy STATIC_STRING fast
-    // path from the non-ESP8266 branch cannot trigger. We copy all keys and
-    // values unconditionally: keys via _P functions (may be in PROGMEM), values
-    // via value() which handles FLASH_STRING internally.
-    value_storage.init(source.size() * 2);
-
-    for (auto &it : source) {
-      auto &kv = dest.emplace_back();
-
-      // Key: copy from possible PROGMEM
-      {
-        size_t key_len = strlen_P(it.key);
-        value_storage.push_back(std::string(key_len, '\0'));
-        memcpy_P(value_storage.back().data(), it.key, key_len);
-        kv.key = StringRef(value_storage.back());
-      }
-
-      // Value: value() handles FLASH_STRING via _P functions internally
-      value_storage.push_back(it.value.value(x...));
-      kv.value = StringRef(value_storage.back());
-    }
-#else
-    // On non-ESP8266, strings are directly readable from flash-mapped memory.
-    // Count non-static strings to allocate exact storage needed.
-    size_t lambda_count = 0;
-    for (const auto &it : source) {
-      if (!it.value.is_static_string()) {
-        lambda_count++;
-      }
-    }
-    value_storage.init(lambda_count);
-
-    for (auto &it : source) {
-      auto &kv = dest.emplace_back();
-      kv.key = StringRef(it.key);
-
-      if (it.value.is_static_string()) {
-        // Static string — pointer directly readable, zero allocation
-        kv.value = StringRef(it.value.get_static_string());
-      } else {
-        // Lambda — evaluate and store result
-        value_storage.push_back(it.value.value(x...));
-        kv.value = StringRef(value_storage.back());
-      }
-    }
-#endif
-  }
-
   APIServer *parent_;
-  TemplatableStringValue<Ts...> service_{};
-  FixedVector<TemplatableKeyValuePair<Ts...>> data_;
-  FixedVector<TemplatableKeyValuePair<Ts...>> data_template_;
-  FixedVector<TemplatableKeyValuePair<Ts...>> variables_;
-#ifdef USE_API_HOMEASSISTANT_ACTION_RESPONSES
-#ifdef USE_API_HOMEASSISTANT_ACTION_RESPONSES_JSON
-  TemplatableStringValue<Ts...> response_template_{""};
-  Trigger<JsonObjectConst, Ts...> success_trigger_with_response_;
-#endif  // USE_API_HOMEASSISTANT_ACTION_RESPONSES_JSON
-  Trigger<Ts...> success_trigger_;
-  Trigger<std::string, Ts...> error_trigger_;
-#endif  // USE_API_HOMEASSISTANT_ACTION_RESPONSES
-
+  const Field *fields_;
+  uint8_t data_count_;
+  uint8_t data_template_count_;
+  uint8_t variables_count_;
   struct Flags {
     uint8_t is_event : 1;
     uint8_t wants_status : 1;
     uint8_t wants_response : 1;
     uint8_t has_response_template : 1;
-    uint8_t reserved : 5;
+    uint8_t reserved : 4;
   } flags_{0};
+#ifdef USE_API_HOMEASSISTANT_ACTION_RESPONSES
+#ifdef USE_API_HOMEASSISTANT_ACTION_RESPONSES_JSON
+  TemplatableValue<std::string, Ts...> response_template_{};
+  Trigger<JsonObjectConst, Ts...> success_trigger_with_response_;
+#endif  // USE_API_HOMEASSISTANT_ACTION_RESPONSES_JSON
+  Trigger<Ts...> success_trigger_;
+  Trigger<std::string, Ts...> error_trigger_;
+#endif  // USE_API_HOMEASSISTANT_ACTION_RESPONSES
 };
 
 }  // namespace esphome::api

esphome/components/api/proto.cpp

@@ -119,7 +119,7 @@ uint32_t ProtoDecodableMessage::count_repeated_field(const uint8_t *buffer, size
 }
 
 // Single-pass encode for repeated submessage elements (non-template core).
-// Writes field tag, reserves 1 byte for length varint, encodes the submessage body,
+// Reserves 1 byte for length varint, encodes the submessage body,
 // then backpatches the actual length. For the common case (body < 128 bytes), this is
 // just a single byte write with no memmove — all current repeated submessage types
 // (BLE advertisements at ~47B, GATT descriptors at ~24B, service args, etc.) take
@@ -143,51 +143,34 @@ uint32_t ProtoDecodableMessage::count_repeated_field(const uint8_t *buffer, size
 //
 //   After writing 2-byte varint at len_pos:
 //   [tag][v1][v2][body ..... body]
-//                                ^-- pos_ = element end, within buffer
-void ProtoWriteBuffer::encode_sub_message(uint32_t field_id, const void *value,
-                                          uint8_t *(*encode_fn)(const void *,
-                                                                ProtoWriteBuffer &PROTO_ENCODE_DEBUG_PARAM)) {
-  this->encode_field_raw(field_id, 2);
-  // Reserve 1 byte for length varint (optimistic: submessage < 128 bytes)
-  uint8_t *len_pos = this->pos_;
-  this->debug_check_bounds_(1);
-  this->pos_++;
-  uint8_t *body_start = this->pos_;
-  this->pos_ = encode_fn(value, *this PROTO_ENCODE_DEBUG_INIT(this->buffer_));
-  uint32_t body_size = static_cast<uint32_t>(this->pos_ - body_start);
-  if (body_size < 128) [[likely]] {
+//                                ^-- returned cursor = element end, within buffer
+uint8_t *ProtoEncode::encode_sub_message_body(uint8_t *__restrict__ pos PROTO_ENCODE_DEBUG_PARAM, const void *value,
+                                              ProtoEncodeFn encode_fn) {
+  // Reserve 1 byte for the length varint (optimistic: submessage < 128 bytes)
+  uint8_t *len_pos = pos;
+  PROTO_ENCODE_CHECK_BOUNDS(pos, 1);
+  uint8_t *body_start = pos + 1;
+  uint8_t *after_body = encode_fn(value, body_start PROTO_ENCODE_DEBUG_ARG);
+  uint32_t body_size = static_cast<uint32_t>(after_body - body_start);
+  if (body_size < VARINT_MAX_1_BYTE) [[likely]] {
     // Common case: 1-byte varint, just backpatch
     *len_pos = static_cast<uint8_t>(body_size);
-    return;
+    return after_body;
   }
-  // Compute extra bytes needed for varint beyond the 1 already reserved
+  // Shift the body forward to make room for the extra length varint bytes
   uint8_t extra = ProtoSize::varint(body_size) - 1;
-  // Shift body forward to make room for the extra varint bytes
-  this->debug_check_bounds_(extra);
+  PROTO_ENCODE_CHECK_BOUNDS(after_body, extra);
   std::memmove(body_start + extra, body_start, body_size);
-  uint8_t *end = this->pos_ + extra;
   // Write the full varint at len_pos
-  this->pos_ = len_pos;
-  this->encode_varint_raw(body_size);
-  this->pos_ = end;
+  (void) encode_varint_raw_loop(len_pos PROTO_ENCODE_DEBUG_ARG, body_size);
+  return after_body + extra;
 }
 
 // Non-template core for encode_optional_sub_message.
-void ProtoWriteBuffer::encode_optional_sub_message(uint32_t field_id, uint32_t nested_size, const void *value,
-                                                   uint8_t *(*encode_fn)(const void *,
-                                                                         ProtoWriteBuffer &PROTO_ENCODE_DEBUG_PARAM)) {
-  if (nested_size == 0)
-    return;
-  this->encode_field_raw(field_id, 2);
-  this->encode_varint_raw(nested_size);
-#ifdef ESPHOME_DEBUG_API
-  uint8_t *start = this->pos_;
-  this->pos_ = encode_fn(value, *this PROTO_ENCODE_DEBUG_INIT(this->buffer_));
-  if (static_cast<uint32_t>(this->pos_ - start) != nested_size)
-    this->debug_check_encode_size_(field_id, nested_size, this->pos_ - start);
-#else
-  this->pos_ = encode_fn(value, *this PROTO_ENCODE_DEBUG_INIT(this->buffer_));
-#endif
+uint8_t *ProtoEncode::encode_sized_sub_message_body(uint8_t *__restrict__ pos PROTO_ENCODE_DEBUG_PARAM,
+                                                    uint32_t nested_size, const void *value, ProtoEncodeFn encode_fn) {
+  pos = encode_varint_raw(pos PROTO_ENCODE_DEBUG_ARG, nested_size);
+  return encode_fn(value, pos PROTO_ENCODE_DEBUG_ARG);
 }
 
 #ifdef ESPHOME_DEBUG_API
@@ -195,6 +178,20 @@ void proto_check_bounds_failed(const uint8_t *pos, size_t bytes, const uint8_t *
   ESP_LOGE(TAG, "Proto encode bounds check failed in %s: need %zu bytes, %td available", caller, bytes, end - pos);
   abort();
 }
+void proto_check_encode_end(const uint8_t *end, const uint8_t *expected) {
+  if (end == expected)
+    return;
+  ESP_LOGE(TAG, "Proto encode ended %td bytes off the calculated size", end - expected);
+  abort();
+}
+void proto_check_sub_message_size(uint32_t field_id, uint32_t expected, const uint8_t *len_pos, const uint8_t *end) {
+  ptrdiff_t actual = end - (len_pos + ProtoSize::varint(expected));
+  if (actual == static_cast<ptrdiff_t>(expected))
+    return;
+  ESP_LOGE(TAG, "encode_message: size mismatch for field %" PRIu32 ": calculated=%" PRIu32 " actual=%td", field_id,
+           expected, actual);
+  abort();
+}
 void ProtoWriteBuffer::debug_check_bounds_(size_t bytes, const char *caller) {
   if (this->pos_ + bytes > this->buffer_->data() + this->buffer_->size()) {
     ESP_LOGE(TAG, "ProtoWriteBuffer bounds check failed in %s: bytes=%zu offset=%td buf_size=%zu", caller, bytes,
@@ -202,85 +199,81 @@ void ProtoWriteBuffer::debug_check_bounds_(size_t bytes, const char *caller) {
     abort();
   }
 }
-void ProtoWriteBuffer::debug_check_encode_size_(uint32_t field_id, uint32_t expected, ptrdiff_t actual) {
-  ESP_LOGE(TAG, "encode_message: size mismatch for field %" PRIu32 ": calculated=%" PRIu32 " actual=%td", field_id,
-           expected, actual);
-  abort();
-}
 
 #endif
 
-void ProtoDecodableMessage::decode(const uint8_t *buffer, size_t length) {
+void ProtoDecodableMessage::decode_fields(void *msg, const uint8_t *buffer, size_t length, DecodeFieldFn field) {
   const uint8_t *ptr = buffer;
   const uint8_t *end = buffer + length;
 
-  while (ptr < end) {
-    // Parse field header - ptr < end guarantees len >= 1
+  // Single-byte varints dominate, so that case advances the cursor inline.
+  auto read_varint = [&](proto_varint_value_t &value) ESPHOME_ALWAYS_INLINE {
+    if (ptr == end)
+      return false;
+    if (*ptr < 0x80) [[likely]] {
+      value = *ptr++;
+      return true;
+    }
     auto res = ProtoVarInt::parse_non_empty(ptr, end - ptr);
-    if (!res.has_value()) {
+    if (!res.has_value())
+      return false;
+    value = res.value;
+    ptr += res.consumed;
+    return true;
+  };
+
+  while (ptr < end) {
+    proto_varint_value_t tag_value;
+    if (!read_varint(tag_value)) {
       ESP_LOGV(TAG, "Invalid field start at offset %ld", (long) (ptr - buffer));
       return;
     }
 
-    uint32_t tag = static_cast<uint32_t>(res.value);
+    uint32_t tag = static_cast<uint32_t>(tag_value);
     uint32_t field_type = tag & WIRE_TYPE_MASK;
-    uint32_t field_id = tag >> 3;
-    ptr += res.consumed;
+    // Length-delimited fields move this past the length prefix
+    const uint8_t *data = ptr;
+    proto_varint_value_t scalar;
 
-    switch (field_type) {
-      case WIRE_TYPE_VARINT: {  // VarInt
-        res = ProtoVarInt::parse(ptr, end - ptr);
-        if (!res.has_value()) {
-          ESP_LOGV(TAG, "Invalid VarInt at offset %ld", (long) (ptr - buffer));
-          return;
-        }
-        if (!this->decode_varint(field_id, res.value)) {
-          ESP_LOGV(TAG, "Cannot decode VarInt field %" PRIu32 " with value %" PRIu64 "!", field_id,
-                   static_cast<uint64_t>(res.value));
-        }
-        ptr += res.consumed;
-        break;
-      }
-      case WIRE_TYPE_LENGTH_DELIMITED: {  // Length-delimited
-        res = ProtoVarInt::parse(ptr, end - ptr);
-        if (!res.has_value()) {
-          ESP_LOGV(TAG, "Invalid Length Delimited at offset %ld", (long) (ptr - buffer));
-          return;
-        }
-        uint32_t field_length = static_cast<uint32_t>(res.value);
-        ptr += res.consumed;
-        if (field_length > static_cast<size_t>(end - ptr)) {
-          ESP_LOGV(TAG, "Out-of-bounds Length Delimited at offset %ld", (long) (ptr - buffer));
-          return;
-        }
-        if (!this->decode_length(field_id, ProtoLengthDelimited(ptr, field_length))) {
-          ESP_LOGV(TAG, "Cannot decode Length Delimited field %" PRIu32 "!", field_id);
-        }
-        ptr += field_length;
-        break;
-      }
-      case WIRE_TYPE_FIXED32: {  // 32-bit
-        if (end - ptr < 4) {
-          ESP_LOGV(TAG, "Out-of-bounds Fixed32-bit at offset %ld", (long) (ptr - buffer));
-          return;
-        }
-        uint32_t val;
-#if __BYTE_ORDER__ == __ORDER_LITTLE_ENDIAN__
-        // Protobuf fixed32 is little-endian — direct load on LE platforms
-        memcpy(&val, ptr, 4);
-#else
-        val = encode_uint32(ptr[3], ptr[2], ptr[1], ptr[0]);
-#endif
-        if (!this->decode_32bit(field_id, Proto32Bit(val))) {
-          ESP_LOGV(TAG, "Cannot decode 32-bit field %" PRIu32 " with value %" PRIu32 "!", field_id, val);
-        }
-        ptr += 4;
-        break;
-      }
-      default:
-        ESP_LOGV(TAG, "Invalid field type %" PRIu32 " at offset %ld", field_type, (long) (ptr - buffer));
+    if (field_type == WIRE_TYPE_VARINT) [[likely]] {
+      if (!read_varint(scalar)) {
+        ESP_LOGV(TAG, "Invalid VarInt at offset %ld", (long) (ptr - buffer));
         return;
+      }
+    } else {
+      switch (field_type) {
+        case WIRE_TYPE_LENGTH_DELIMITED: {
+          proto_varint_value_t length_value;
+          if (!read_varint(length_value)) {
+            ESP_LOGV(TAG, "Invalid Length Delimited at offset %ld", (long) (ptr - buffer));
+            return;
+          }
+          uint32_t field_length = static_cast<uint32_t>(length_value);
+          if (field_length > static_cast<size_t>(end - ptr)) {
+            ESP_LOGV(TAG, "Out-of-bounds Length Delimited at offset %ld", (long) (ptr - buffer));
+            return;
+          }
+          data = ptr;
+          scalar = field_length;
+          ptr += field_length;
+          break;
+        }
+        case WIRE_TYPE_FIXED32: {
+          if (end - ptr < 4) {
+            ESP_LOGV(TAG, "Out-of-bounds Fixed32-bit at offset %ld", (long) (ptr - buffer));
+            return;
+          }
+          // Byte loads instead of memcpy: ESP-IDF passes -fno-builtin-memcpy, which made this a call
+          scalar = encode_uint32(ptr[3], ptr[2], ptr[1], ptr[0]);
+          ptr += 4;
+          break;
+        }
+        default:
+          ESP_LOGV(TAG, "Invalid field type %" PRIu32 " at offset %ld", field_type, (long) (ptr - buffer));
+          return;
+      }
     }
+    field(msg, tag, data, scalar);
   }
 }