The benchmark library warns "Library was built as DEBUG" without NDEBUG.
This enables debug assertions and disables optimizations in the benchmark
framework, causing instrumentation overhead to show up in profiles.
CodSpeed's CMake uses CODSPEED_ANALYSIS (not CODSPEED_SIMULATION) for
simulation mode. This define gates all the actual benchmark measurement
hooks in benchmark_runner.cpp and benchmark.h. Without it, benchmarks
run normally but CodSpeed doesn't detect them as instrumented.
benchmark.h uses #ifdef CODSPEED_ENABLED to switch benchmark
registration to CodSpeed-instrumented variants. This define was
only in library.json (applied to library compilation) but not to
the benchmark .cpp files that #include <benchmark/benchmark.h>.
Without it, CodSpeed reports "No benchmarks found".
- Move BASE_CODEGEN_COMPONENTS and USE_TIME_TIMEZONE_FLAG to test_helpers.py
- Use shared constants in both cpp_unit_test.py and cpp_benchmark.py
- Move json import to top level in cpp_benchmark.py
- Refactor setup_codspeed_lib.py into focused helper functions
- Combine clone + submodule init, use --shallow-submodules
- Use CodSpeed's codspeed-cpp fork with proper instrumentation for
simulation mode benchmark detection
- setup_codspeed_lib.py creates a flat PlatformIO-compatible library
by combining google_benchmark sources, codspeed core, and
instrument-hooks into a single library directory
- Renames .cc to .cpp (PlatformIO doesn't compile .cc by default)
- Adds all required defines: CODSPEED_ENABLED, CODSPEED_SIMULATION,
CODSPEED_VERSION, CODSPEED_ROOT_DIR, CODSPEED_MODE_DISPLAY
- Output JSON config consumed by cpp_benchmark.py via env var
- Clone CodSpeed's codspeed-cpp repo (pinned to SHA) in CI
- Create PlatformIO-compatible library.json combining google_benchmark
and codspeed core sources for proper instrumentation
- Pass library path via BENCHMARK_LIB env var to cpp_benchmark.py
- Use simulation mode for reproducible CPU instruction counting
- Locally, vanilla google/benchmark is used (no CodSpeed instrumentation)
Core is not a component — its benchmarks belong in tests/benchmarks/core/
not tests/benchmarks/components/core/. Add extra_include_dirs parameter
to build_and_run to support non-component benchmark directories.
Add automated benchmarks using Google Benchmark to prevent performance
regressions in the API protobuf encoding/decoding and core loop paths.
Benchmarks cover:
- Protobuf encode: SensorState, BinarySensorState, HelloResponse,
LightState, DeviceInfoResponse (20 nested devices + 20 areas)
- Protobuf decode: HelloRequest, SwitchCommand, LightCommand
- Protobuf calculate_size and full calc+encode send path
- Varint parse/encode/size for various value ranges
- Scheduler call/next_schedule_in with idle and active timers
- Application loop component dispatch and blocking guard overhead
Infrastructure:
- Extract shared build logic from cpp_unit_test.py into test_helpers.py
- Add cpp_benchmark.py mirroring the unit test build pattern
- Support benchmark.yaml per component dir for declaring dependencies
- Add CodSpeed CI workflow triggered on api/core changes
- Fix ProtoMessage protected destructor on host platform
J. Nick Koston