- save_compiled_config: write the cache with mode 0600 (the YAML
contains resolved secrets) and always rewrite so mtime advances
even when content is unchanged. Previously write_file_if_changed
could leave mtime pinned behind a whitespace YAML edit, locking
the fast path to 'stale'.
- __main__: skip the fast path when -s overrides are passed. The
cache was written against the previous substitution set;
reusing it would silently ignore the new values.
- Add tests for save_compiled_config (write + swallow-errors), the
wizard-only sidecar fallback, and the substitution-skip branch.
- Trim docstrings/comments throughout.
apply_to_core was over-populating: it restored friendly_name,
loaded_integrations, and loaded_platforms even though every
consumer of those three lives inside a component validator
(esp32_camera, esp32, deep_sleep, zigbee, lvgl, zephyr_mcumgr),
and the whole point of the fast path is to skip validation.
Drop them. CORE.__init__ already leaves all three at safe defaults
(None / empty set) for any incidental reader.
What's left is exactly what upload/logs walk:
- CORE.name (api.client.run_logs, firmware_bin path, mDNS)
- CORE.build_path (firmware_bin / partition_table_bin / bootloader_bin)
- CORE.data[KEY_CORE][KEY_TARGET_PLATFORM] (module dispatch, .is_esp32 etc)
- CORE.data[KEY_CORE][KEY_TARGET_FRAMEWORK] (.is_arduino, firmware_bin branch)
Method body shrinks from 9 statements to 4; setdefault + two
conditional inserts collapse into one dict literal; the
function-local import moves to module top. Drift surface drops
from 7 paired fields to 4. The wizard-only-sidecar None case is
gated once at the load_compiled_config boundary so apply_to_core
no longer has to defend against it.
`StorageJSON.apply_to_core` was the inverse of `from_esphome_core`:
two enumerations of the same fields that had to stay in sync. When a
new CORE attribute joined the validation pipeline, both lists needed
an update.
Replace it with `_populate_core_from_validated_config(config)` in
`compiled_config.py` that reads the cached config dict directly --
same source `read_config` writes into during validation, no separate
sidecar schema. The fast path no longer touches StorageJSON at all;
the sidecar stays in place for the dashboard's existing use.
Drift surface shrinks to: name/friendly_name/build_path under the
esphome block (config schema is stable) + target_platform discovered
via `_is_target_platform` (the canonical detector) + framework via
the `<platform>.framework.type` convention every platform component
already follows.
storage_json.py is for the JSON sidecar; the validated-config cache
is YAML. Move it out into its own module.
Also consolidate the dispatcher's open-coded fast path (load cache +
load sidecar + apply_to_core) into a single load_compiled_config()
entry point, and collapse save_compiled_config()'s two try blocks
into one. mtime comparison gets its own _cache_is_fresh() helper so
the loader reads top to bottom.
Drop the `--from-storage-json` CLI flag. `esphome upload` and
`esphome logs` now always try the validated-config cache and
transparently fall back to `read_config()` when it's missing, stale
(YAML mtime > cache mtime), or corrupt. No caller change is required
to benefit, and a cold cache never produces a worse outcome than
today.
`compile` continues to write the cache unconditionally so the next
upload / logs against this YAML can skip validation; it itself
always re-validates since code generation needs the fully validated
config.
Drop the StorageJSON `compiled_config` field and the
`esphome/lite_config.py` module from the first iteration. The earlier
shape stored the entire validated config inside the JSON sidecar, but
configs can grow past a megabyte once packages and substitutions
expand, and JSON can't round-trip the YAML-specific types (lambdas,
ID instances, includes) the validation pipeline produces.
Instead: dump the validated config to its own YAML file alongside the
sidecar (`<file>.validated.yaml`), using `yaml_util.dump` so all
esphome-specific tags survive the round trip. The fast path loads it
back with `yaml_util.load_yaml(clear_secrets=False)` -- no schema
validation, no final-validate, no external-component refresh.
Staleness is gated by mtime: if the source YAML mtime is newer than
the cache, the dispatcher falls back to a full `read_config()` with a
warning. The cache is refreshed on every successful compile via
`writer.update_storage_json`, so the dashboard's compile -> upload
-> logs flow always sees a fresh cache.
Drift surface shrinks to "what does yaml_util.dump emit" rather than
"which top-level keys does upload/logs read" -- when those subcommands
gain a new dependency, the cache already carries it.
When a downstream caller (typically a dashboard) already has a
known-good firmware binary on disk and only needs the CLI to ship
bytes to a device or stream logs back, running the full
``read_config()`` pipeline for every subcommand is dead work. For
the device-builder REMOTE install flow that runs compile, upload,
and logs back-to-back this means three "Reading configuration ..."
passes for one user action.
Add an opt-in ``--from-storage-json`` flag to ``upload`` and
``logs`` that sources platform / build metadata from the
StorageJSON sidecar produced by the last successful compile and
re-parses just enough of the YAML head (substitutions + packages,
no schema validation) to recover the ``api:`` / ``logger:`` /
``ota:`` / network blocks the subcommands consult.
The flag is opt-in and falls back to a full ``read_config()`` pass
when the sidecar is missing or older than the YAML, so a cold cache
never produces a worse outcome than today. ``compile`` / ``run`` /
``clean`` / ``bundle`` continue to validate as before.
J. Nick Koston