- save_compiled_config: write the cache with mode 0600 (the YAML
contains resolved secrets) and always rewrite so mtime advances
even when content is unchanged. Previously write_file_if_changed
could leave mtime pinned behind a whitespace YAML edit, locking
the fast path to 'stale'.
- __main__: skip the fast path when -s overrides are passed. The
cache was written against the previous substitution set;
reusing it would silently ignore the new values.
- Add tests for save_compiled_config (write + swallow-errors), the
wizard-only sidecar fallback, and the substitution-skip branch.
- Trim docstrings/comments throughout.
apply_to_core was over-populating: it restored friendly_name,
loaded_integrations, and loaded_platforms even though every
consumer of those three lives inside a component validator
(esp32_camera, esp32, deep_sleep, zigbee, lvgl, zephyr_mcumgr),
and the whole point of the fast path is to skip validation.
Drop them. CORE.__init__ already leaves all three at safe defaults
(None / empty set) for any incidental reader.
What's left is exactly what upload/logs walk:
- CORE.name (api.client.run_logs, firmware_bin path, mDNS)
- CORE.build_path (firmware_bin / partition_table_bin / bootloader_bin)
- CORE.data[KEY_CORE][KEY_TARGET_PLATFORM] (module dispatch, .is_esp32 etc)
- CORE.data[KEY_CORE][KEY_TARGET_FRAMEWORK] (.is_arduino, firmware_bin branch)
Method body shrinks from 9 statements to 4; setdefault + two
conditional inserts collapse into one dict literal; the
function-local import moves to module top. Drift surface drops
from 7 paired fields to 4. The wizard-only-sidecar None case is
gated once at the load_compiled_config boundary so apply_to_core
no longer has to defend against it.
`StorageJSON.apply_to_core` was the inverse of `from_esphome_core`:
two enumerations of the same fields that had to stay in sync. When a
new CORE attribute joined the validation pipeline, both lists needed
an update.
Replace it with `_populate_core_from_validated_config(config)` in
`compiled_config.py` that reads the cached config dict directly --
same source `read_config` writes into during validation, no separate
sidecar schema. The fast path no longer touches StorageJSON at all;
the sidecar stays in place for the dashboard's existing use.
Drift surface shrinks to: name/friendly_name/build_path under the
esphome block (config schema is stable) + target_platform discovered
via `_is_target_platform` (the canonical detector) + framework via
the `<platform>.framework.type` convention every platform component
already follows.
storage_json.py is for the JSON sidecar; the validated-config cache
is YAML. Move it out into its own module.
Also consolidate the dispatcher's open-coded fast path (load cache +
load sidecar + apply_to_core) into a single load_compiled_config()
entry point, and collapse save_compiled_config()'s two try blocks
into one. mtime comparison gets its own _cache_is_fresh() helper so
the loader reads top to bottom.
Drop the `--from-storage-json` CLI flag. `esphome upload` and
`esphome logs` now always try the validated-config cache and
transparently fall back to `read_config()` when it's missing, stale
(YAML mtime > cache mtime), or corrupt. No caller change is required
to benefit, and a cold cache never produces a worse outcome than
today.
`compile` continues to write the cache unconditionally so the next
upload / logs against this YAML can skip validation; it itself
always re-validates since code generation needs the fully validated
config.
Drop the StorageJSON `compiled_config` field and the
`esphome/lite_config.py` module from the first iteration. The earlier
shape stored the entire validated config inside the JSON sidecar, but
configs can grow past a megabyte once packages and substitutions
expand, and JSON can't round-trip the YAML-specific types (lambdas,
ID instances, includes) the validation pipeline produces.
Instead: dump the validated config to its own YAML file alongside the
sidecar (`<file>.validated.yaml`), using `yaml_util.dump` so all
esphome-specific tags survive the round trip. The fast path loads it
back with `yaml_util.load_yaml(clear_secrets=False)` -- no schema
validation, no final-validate, no external-component refresh.
Staleness is gated by mtime: if the source YAML mtime is newer than
the cache, the dispatcher falls back to a full `read_config()` with a
warning. The cache is refreshed on every successful compile via
`writer.update_storage_json`, so the dashboard's compile -> upload
-> logs flow always sees a fresh cache.
Drift surface shrinks to "what does yaml_util.dump emit" rather than
"which top-level keys does upload/logs read" -- when those subcommands
gain a new dependency, the cache already carries it.
When a downstream caller (typically a dashboard) already has a
known-good firmware binary on disk and only needs the CLI to ship
bytes to a device or stream logs back, running the full
``read_config()`` pipeline for every subcommand is dead work. For
the device-builder REMOTE install flow that runs compile, upload,
and logs back-to-back this means three "Reading configuration ..."
passes for one user action.
Add an opt-in ``--from-storage-json`` flag to ``upload`` and
``logs`` that sources platform / build metadata from the
StorageJSON sidecar produced by the last successful compile and
re-parses just enough of the YAML head (substitutions + packages,
no schema validation) to recover the ``api:`` / ``logger:`` /
``ota:`` / network blocks the subcommands consult.
The flag is opt-in and falls back to a full ``read_config()`` pass
when the sidecar is missing or older than the YAML, so a cold cache
never produces a worse outcome than today. ``compile`` / ``run`` /
``clean`` / ``bundle`` continue to validate as before.
Co-authored-by: pre-commit-ci-lite[bot] <117423508+pre-commit-ci-lite[bot]@users.noreply.github.com>
Co-authored-by: J. Nick Koston <nick@home-assistant.io>
Co-authored-by: J. Nick Koston <nick@home-assistant.io>
Co-authored-by: pre-commit-ci-lite[bot] <117423508+pre-commit-ci-lite[bot]@users.noreply.github.com>
Co-authored-by: J. Nick Koston <nick@koston.org>
Co-authored-by: pre-commit-ci-lite[bot] <117423508+pre-commit-ci-lite[bot]@users.noreply.github.com>
Co-authored-by: J. Nick Koston <nick@home-assistant.io>
Co-authored-by: J. Nick Koston <nick@koston.org>
- upload_program: allow MQTT/MQTTIP devices for --partition-table.
MQTTIP gets resolved to a real IP by _resolve_network_devices(), so
rejecting any non-NETWORK port_type was incorrect; only SERIAL and
BOOTSEL are non-OTA upload paths.
- update_partition_table: re-initialize NVS on every failure path past
nvs_flash_deinit() so components that survive a failed OTA aren't
left with broken NVS handles. Success path stays as-is because the
device reboots immediately afterwards.
Adds an MQTTIP upload test and refreshes the gate's comment.
Split the combined "missing app, otadata, or nvs" verify failure into
three separate ESP_LOGE messages so users can see which check failed,
trim trailing spaces from the (err=0x%X) log strings, and document why
the partition-table espota2 test mocks SERVER_FEATURE_SUPPORTS_COMPRESSION
(intentional protocol-path coverage; the real IDFOTABackend never sets it).
Read the resolved partition-table file in upload_program before opening
a network connection. Reject anything that isn't 0xC00 bytes, doesn't
start with ESP_PARTITION_MAGIC, or is missing the MD5 checksum entry, so
mistakes (wrong file, swapped --file path) surface as a local error
instead of a post-handshake OTA_RESPONSE_ERROR_PARTITION_TABLE_VERIFY.
Includes unit tests covering size, magic, md5-presence, missing-file,
and end-to-end upload_program rejection, plus three real partition
tables checked in as fixtures (ESPHome build, ESP-IDF Hello-world,
esphome_dashboard prebuilt).
_run_idedata raises EsphomeError() with no message, so the warning
shows 'unavailable ()' which looks like a bug. Use a generic fallback
explanation when str(exc) is empty.
_decode_pc shells out to PlatformIO via _run_idedata; without a
populated build dir for the device that subprocess fails for every
PC/BT line in a crash dump. Disable decoding after the first
EsphomeError (per logs session) and emit a single user-facing warning
instead of retrying on every line.
Also rename the helper to _LogLineProcessor since it now owns the
per-session decode-enabled state, not just the decode call.
When 'esphome logs' processes a crash backtrace from the device,
process_stacktrace -> _decode_pc -> _run_idedata can raise
EsphomeError if the local build dir hasn't been populated (e.g. the
device was flashed from a different machine). on_log runs inside an
asyncio protocol callback, so the unhandled exception triggers
'Fatal error: protocol.data_received() call failed.', the loop tears
the connection down, and ReconnectLogic immediately reconnects. The
device replays the same crash trace and we loop forever.
Wrap the per-line decode in a helper that swallows EsphomeError so
the connection stays up. Also covered with unit tests for the new
helper.
J. Nick Koston