- Use BENCHMARK_BINARY= marker for reliable binary path extraction
instead of fragile tail -1 (PlatformIO can print warnings after path)
- Fix Scheduler_Defer: defer() is protected on Component, use
set_timeout(delay=0) directly on Scheduler instead
Add USE_BENCHMARK define to benchmark build flags. Guard the
warn_blocking call in finish() with #ifndef USE_BENCHMARK so
scheduler benchmarks using fake monotonic time don't trigger
the underflow (fake now > real millis()).
Remove BenchComponent — no longer needed with the ifdef.
- Revert component.h change (no core changes for benchmarks)
- Remove -DWARN_IF_BLOCKING_OVER_MS from build flags (can't shadow constexpr)
- Drop inner loop — 5 heap pops + callbacks + pushes per call is well
above CodSpeed's 60ns instrumentation overhead
- Add warm-up call before benchmark loop to trigger the blocking guard
once and ramp the threshold
- interval=0 causes infinite loop, must use interval=1 with fake time
interval=0 causes infinite loop (reschedules at same time, never breaks).
interval=1 with millis() doesn't work (real time doesn't advance fast
enough between inner iterations for intervals to re-fire).
Solution: use interval=1 with monotonically increasing fake time (now++)
and disable WarnIfComponentBlockingGuard at compile time via
-DWARN_IF_BLOCKING_OVER_MS=UINT32_MAX in benchmark build flags. This
prevents the guard's (millis() - started_) underflow when fake time
exceeds real millis().
- Remove inner loop from CalcAndEncode_DeviceInfoResponse_Fresh to match
SensorStateResponse_Fresh — both now measure single alloc+encode per
iteration as intended for heap allocation cost benchmarking
- Add comments documenting why _Fresh variants skip inner loops
- Add first-wins comment to load_component_yaml_configs
- Clean up .gitignore template comment
Sub-microsecond benchmarks are dominated by the ~60ns per-iteration
valgrind start/stop cost in CodSpeed simulation mode. Add kInnerIterations
(1000) inner loops to all fast benchmarks so the actual work dominates.
Move DoNotOptimize calls outside inner loops to prevent artificial overhead.
Also address review feedback:
- Use tokenless CodSpeed (public repo, no CODSPEED_TOKEN needed)
- Fix warning message to show component-specific path
- Fix stray ". :" in error message
- Verify pinned SHA on re-runs to prevent stale checkouts
- Remove BM_ prefix from all benchmark names (unnecessary convention)
- Remove bench_application_loop.cpp (App needs pre_setup/setup which
requires full code generation; will revisit as integration benchmark)
- Fix extra_include_dirs to use os.path.relpath for sibling directories
The benchmark library warns "Library was built as DEBUG" without NDEBUG.
This enables debug assertions and disables optimizations in the benchmark
framework, causing instrumentation overhead to show up in profiles.
CodSpeed's CMake uses CODSPEED_ANALYSIS (not CODSPEED_SIMULATION) for
simulation mode. This define gates all the actual benchmark measurement
hooks in benchmark_runner.cpp and benchmark.h. Without it, benchmarks
run normally but CodSpeed doesn't detect them as instrumented.
benchmark.h uses #ifdef CODSPEED_ENABLED to switch benchmark
registration to CodSpeed-instrumented variants. This define was
only in library.json (applied to library compilation) but not to
the benchmark .cpp files that #include <benchmark/benchmark.h>.
Without it, CodSpeed reports "No benchmarks found".
- Move BASE_CODEGEN_COMPONENTS and USE_TIME_TIMEZONE_FLAG to test_helpers.py
- Use shared constants in both cpp_unit_test.py and cpp_benchmark.py
- Move json import to top level in cpp_benchmark.py
- Refactor setup_codspeed_lib.py into focused helper functions
- Combine clone + submodule init, use --shallow-submodules
- Use CodSpeed's codspeed-cpp fork with proper instrumentation for
simulation mode benchmark detection
- setup_codspeed_lib.py creates a flat PlatformIO-compatible library
by combining google_benchmark sources, codspeed core, and
instrument-hooks into a single library directory
- Renames .cc to .cpp (PlatformIO doesn't compile .cc by default)
- Adds all required defines: CODSPEED_ENABLED, CODSPEED_SIMULATION,
CODSPEED_VERSION, CODSPEED_ROOT_DIR, CODSPEED_MODE_DISPLAY
- Output JSON config consumed by cpp_benchmark.py via env var
- Clone CodSpeed's codspeed-cpp repo (pinned to SHA) in CI
- Create PlatformIO-compatible library.json combining google_benchmark
and codspeed core sources for proper instrumentation
- Pass library path via BENCHMARK_LIB env var to cpp_benchmark.py
- Use simulation mode for reproducible CPU instruction counting
- Locally, vanilla google/benchmark is used (no CodSpeed instrumentation)
Core is not a component — its benchmarks belong in tests/benchmarks/core/
not tests/benchmarks/components/core/. Add extra_include_dirs parameter
to build_and_run to support non-component benchmark directories.
- Move scheduler/loop/helpers benchmarks to tests/benchmarks/components/core/
- Add random_float and random_uint32 benchmarks (from ol.yaml)
- Fix core pseudo-component crash: skip components where get_component()
returns None when adding dependencies to config
- Use CodSpeed simulation mode (CPU instruction counting) for reproducible
CI results instead of walltime
- Add should_run_benchmarks() to determine-jobs.py that checks if
directly changed components have benchmark files (no dependency
expansion - changing sensor won't trigger api benchmarks)
- Move benchmark job from separate workflow into ci.yml
- Pin CodSpeed action to full commit SHA
- Delete separate ci-benchmarks.yml
Add automated benchmarks using Google Benchmark to prevent performance
regressions in the API protobuf encoding/decoding and core loop paths.
Benchmarks cover:
- Protobuf encode: SensorState, BinarySensorState, HelloResponse,
LightState, DeviceInfoResponse (20 nested devices + 20 areas)
- Protobuf decode: HelloRequest, SwitchCommand, LightCommand
- Protobuf calculate_size and full calc+encode send path
- Varint parse/encode/size for various value ranges
- Scheduler call/next_schedule_in with idle and active timers
- Application loop component dispatch and blocking guard overhead
Infrastructure:
- Extract shared build logic from cpp_unit_test.py into test_helpers.py
- Add cpp_benchmark.py mirroring the unit test build pattern
- Support benchmark.yaml per component dir for declaring dependencies
- Add CodSpeed CI workflow triggered on api/core changes
- Fix ProtoMessage protected destructor on host platform
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: pre-commit-ci-lite[bot] <117423508+pre-commit-ci-lite[bot]@users.noreply.github.com>
J. Nick Koston