Sub-microsecond benchmarks are dominated by the ~60ns per-iteration
valgrind start/stop cost in CodSpeed simulation mode. Add kInnerIterations
(1000) inner loops to all fast benchmarks so the actual work dominates.
Move DoNotOptimize calls outside inner loops to prevent artificial overhead.
Also address review feedback:
- Use tokenless CodSpeed (public repo, no CODSPEED_TOKEN needed)
- Fix warning message to show component-specific path
- Fix stray ". :" in error message
- Verify pinned SHA on re-runs to prevent stale checkouts
The benchmark library warns "Library was built as DEBUG" without NDEBUG.
This enables debug assertions and disables optimizations in the benchmark
framework, causing instrumentation overhead to show up in profiles.
CodSpeed's CMake uses CODSPEED_ANALYSIS (not CODSPEED_SIMULATION) for
simulation mode. This define gates all the actual benchmark measurement
hooks in benchmark_runner.cpp and benchmark.h. Without it, benchmarks
run normally but CodSpeed doesn't detect them as instrumented.
- Move BASE_CODEGEN_COMPONENTS and USE_TIME_TIMEZONE_FLAG to test_helpers.py
- Use shared constants in both cpp_unit_test.py and cpp_benchmark.py
- Move json import to top level in cpp_benchmark.py
- Refactor setup_codspeed_lib.py into focused helper functions
- Combine clone + submodule init, use --shallow-submodules
- Use CodSpeed's codspeed-cpp fork with proper instrumentation for
simulation mode benchmark detection
- setup_codspeed_lib.py creates a flat PlatformIO-compatible library
by combining google_benchmark sources, codspeed core, and
instrument-hooks into a single library directory
- Renames .cc to .cpp (PlatformIO doesn't compile .cc by default)
- Adds all required defines: CODSPEED_ENABLED, CODSPEED_SIMULATION,
CODSPEED_VERSION, CODSPEED_ROOT_DIR, CODSPEED_MODE_DISPLAY
- Output JSON config consumed by cpp_benchmark.py via env var
J. Nick Koston