feat: add memory profiling support #39

not-matthias · 2026-01-14T18:27:59Z

TODO: REmove the custom memtrack installation before merging

codspeed-hq · 2026-01-14T18:36:03Z

Merging this PR will degrade performance by 39.4%

⚡ 10 improved benchmarks
❌ 12 regressed benchmarks
✅ 170 untouched benchmarks
🆕 190 new benchmarks

⚠️ Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

	Mode	Benchmark	`BASE`	`HEAD`	Efficiency
🆕	WallTime	`BM_RLE_Encode_SmallRuns[100]`	N/A	1.8 µs	N/A
❌	WallTime	`BM_StringCopy`	10.7 ns	14.4 ns	-25.98%
🆕	WallTime	`BM_RLE_Encode_SmallRuns[100000]`	N/A	1.6 ms	N/A
🆕	WallTime	`BM_String_Concatenation[10]`	N/A	97.1 ns	N/A
🆕	WallTime	`BM_RLE_Decode[100]`	N/A	375.7 ns	N/A
🆕	WallTime	`BM_RLE_Encode_LargeRuns[100]`	N/A	212.1 ns	N/A
🆕	WallTime	`BM_RLE_Decode[100000]`	N/A	286.3 µs	N/A
🆕	WallTime	`BM_RLE_Decode[10000]`	N/A	22.5 µs	N/A
🆕	WallTime	`BM_RLE_Encode_LargeRuns[1000]`	N/A	2.2 µs	N/A
🆕	WallTime	`BM_RLE_Decode[1000]`	N/A	2.7 µs	N/A
🆕	WallTime	`BM_Vector_PushBack[10000]`	N/A	33.8 µs	N/A
🆕	WallTime	`BM_Vector_PushBack[10]`	N/A	284.5 ns	N/A
🆕	WallTime	`BM_Vector_Reserve[100]`	N/A	350.7 ns	N/A
🆕	WallTime	`BM_RLE_Encode_SmallRuns[10000]`	N/A	151.5 µs	N/A
🆕	WallTime	`BM_Vector_PushBack[100]`	N/A	744.2 ns	N/A
🆕	WallTime	`BM_RLE_Encode_LargeRuns[100000]`	N/A	203.2 µs	N/A
🆕	WallTime	`BM_String_Concatenation[100]`	N/A	1 µs	N/A
🆕	WallTime	`BM_Vector_Reserve[10000]`	N/A	31.2 µs	N/A
🆕	WallTime	`BM_RLE_Encode_LargeRuns[10000]`	N/A	20.5 µs	N/A
🆕	WallTime	`BM_String_Concatenation[10000]`	N/A	82 µs	N/A
...	...	...	...	...	...

ℹ️ Only the first 20 benchmarks are displayed. Go to the app to view all benchmarks.

_{Comparing cod-1945-support-memory-profiling-for-c (003a1bf) with main (56e942a)}

When running the warmup, we already call `start/stop_benchmark` which creates two memory profiles per benchmark, which breaks our assumptions in the backend.

Copilot

Pull request overview

This PR adds memory profiling support to the benchmarking framework by introducing a new "memory" mode alongside the existing "simulation" and "walltime" modes. The changes unify the preprocessor directives and add the necessary configuration and infrastructure for memory profiling.

Changes:

Renamed CODSPEED_SIMULATION preprocessor macro to CODSPEED_ANALYSIS to better represent both simulation and memory profiling modes
Added "memory" as a new valid mode in CMake and Bazel build configurations
Removed warmup execution logic from simulation mode to reduce flakiness
Added memory benchmark examples demonstrating various allocation patterns
Updated CI workflows to test the new memory profiling mode with memtrack CLI installation

Reviewed changes

Copilot reviewed 12 out of 12 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
google_benchmark/src/benchmark_runner.cc	Updated preprocessor directive from CODSPEED_SIMULATION to CODSPEED_ANALYSIS
google_benchmark/src/benchmark_api_internal.h	Updated preprocessor directives and removed warmup execution logic
google_benchmark/src/benchmark_api_internal.cc	Removed warmup repetition code from simulation mode
google_benchmark/src/benchmark.cc	Updated preprocessor directives and console output message
google_benchmark/include/benchmark/benchmark.h	Updated preprocessor directives throughout the State class
examples/google_benchmark_cmake/memory_bench.hpp	Added new memory profiling benchmark examples with RLE encoding/decoding and allocation patterns
examples/google_benchmark_cmake/main.cpp	Added include for new memory benchmark header
examples/google_benchmark_bazel/memory_bench.hpp	Added symlink to cmake memory benchmark file
core/instrument-hooks	Updated submodule commit reference
core/CMakeLists.txt	Added "memory" to allowed modes and updated preprocessor definitions
core/BUILD	Added memory_mode configuration and updated preprocessor defines
.github/workflows/ci.yml	Added memory mode to test matrix and memtrack CLI installation steps

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-01-15T17:23:27Z

google_benchmark/src/benchmark_api_internal.cc

+#ifdef CODSPEED_ANALYSIS
 State BenchmarkInstance::RunSimulation(
    codspeed::CodSpeed* codspeed, internal::ThreadTimer* timer,
    internal::ThreadManager* manager,
    internal::PerfCountersMeasurement* perf_counters_measurement,
    ProfilerManager* profiler_manager) const {


The function name 'RunSimulation' is now misleading since CODSPEED_ANALYSIS covers both simulation and memory modes. Consider renaming to 'RunAnalysis' or a more generic name that reflects its broader purpose.

Could be worth it to rename this indeed

Copilot · 2026-01-15T17:23:27Z

google_benchmark/src/benchmark.cc

  // Determine the width of the name field using a minimum width of 10.
-#ifdef CODSPEED_SIMULATION
+#ifdef CODSPEED_ANALYSIS
  Err << "Codspeed mode: simulation" << "\n";


The error message still says 'simulation' but this code path is now used for both simulation and memory modes under CODSPEED_ANALYSIS. The message should differentiate between these modes or be more generic.

Suggested change

Err << "Codspeed mode: simulation" << "\n";

Err << "Codspeed mode: analysis" << "\n";

Not sure if printing analysis is the best UX, since we said it should be only the internal build mode. I guess we could forward the real measurement mode and then print that, while also enabling CODSPEED_WALLTIME/CODSPEED_ANALYSIS. wdyt?

I think we should stick to never displaying analysis in a user facing way.

Best way to proceed would be to either

Read from env var, which is what interpreted languages do anyway

Define a CODSPEED_MODE_DISPLAY pre proc variable that we use here, and remove this cascading if elif

Most likely the best way to do this is actually the env var, but I'm fine with both

.github/workflows/ci.yml

GuillaumeLagrange · 2026-01-16T09:00:19Z

@not-matthias the ci fails, can you re-request a review once fixed ? 🙏

GuillaumeLagrange

Olgtm, just wanted input from @art049 regarding the removal of the warmup run in analysis mode

GuillaumeLagrange · 2026-01-16T14:07:28Z

google_benchmark/src/benchmark.cc

  // Determine the width of the name field using a minimum width of 10.
-#ifdef CODSPEED_SIMULATION
+#ifdef CODSPEED_ANALYSIS
  Err << "Codspeed mode: simulation" << "\n";


I think we should stick to never displaying analysis in a user facing way.

Best way to proceed would be to either

Read from env var, which is what interpreted languages do anyway

Define a CODSPEED_MODE_DISPLAY pre proc variable that we use here, and remove this cascading if elif

Most likely the best way to do this is actually the env var, but I'm fine with both

GuillaumeLagrange · 2026-01-16T14:07:50Z

google_benchmark/src/benchmark_api_internal.cc

+#ifdef CODSPEED_ANALYSIS
 State BenchmarkInstance::RunSimulation(
    codspeed::CodSpeed* codspeed, internal::ThreadTimer* timer,
    internal::ThreadManager* manager,
    internal::PerfCountersMeasurement* perf_counters_measurement,
    ProfilerManager* profiler_manager) const {


Could be worth it to rename this indeed

GuillaumeLagrange · 2026-01-16T14:09:12Z

examples/google_benchmark_bazel/memory_bench.hpp

@@ -0,0 +1 @@
+../google_benchmark_cmake/memory_bench.hpp


no newline at EOF

GuillaumeLagrange · 2026-01-16T14:14:10Z

google_benchmark/src/benchmark_api_internal.cc

    internal::ThreadManager* manager,
    internal::PerfCountersMeasurement* perf_counters_measurement,
    ProfilerManager* profiler_manager) const {
-  // Do one repetition to avoid flakiness due to inconcistencies in CPU cache


@art049 a warmup run is something we do in interpreted language for jit reasons.

We imported this behavior here, since there may a point in unifying the cpu caches state and whatnot, but we currently do not do it in our rust integrations

Would you rather we introduce this in our rust integration, or remove this here ?

Hmm yes we should not do it. But this makes this a breaking change

chore: bump instrument-hooks to support memory profiling

acce038

not-matthias changed the title ~~chore: bump instrument-hooks to support memory profiling~~ feat: add memory profiling support Jan 14, 2026

not-matthias added 2 commits January 14, 2026 19:33

feat: add support for memory mode

68e21b6

feat: run memory profiling in CI

dfadbfd

wip: install latest memtrack cli

1b97510

not-matthias force-pushed the cod-1945-support-memory-profiling-for-c branch from 7b0a7b4 to 1b97510 Compare January 15, 2026 11:11

not-matthias added 3 commits January 15, 2026 18:18

feat(example): add memory benchmark example

5292441

chore(ci): switch to simulation

c6e8f77

fix: disable warmup to prevent additional markers

003a1bf

When running the warmup, we already call `start/stop_benchmark` which creates two memory profiles per benchmark, which breaks our assumptions in the backend.

not-matthias force-pushed the cod-1945-support-memory-profiling-for-c branch from b158911 to 003a1bf Compare January 15, 2026 17:20

not-matthias requested review from GuillaumeLagrange and Copilot January 15, 2026 17:22

Copilot AI reviewed Jan 15, 2026

View reviewed changes

GuillaumeLagrange removed their request for review January 16, 2026 09:00

not-matthias requested a review from GuillaumeLagrange January 16, 2026 10:20

GuillaumeLagrange approved these changes Jan 16, 2026

View reviewed changes

	Err << "Codspeed mode: simulation" << "\n";
	Err << "Codspeed mode: analysis" << "\n";

		@@ -0,0 +1 @@
		../google_benchmark_cmake/memory_bench.hpp No newline at end of file

feat: add memory profiling support #39

Are you sure you want to change the base?

feat: add memory profiling support #39

Conversation

not-matthias commented Jan 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codspeed-hq bot commented Jan 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Merging this PR will degrade performance by 39.4%

Performance Changes

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Jan 15, 2026

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 15, 2026

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

GuillaumeLagrange commented Jan 16, 2026

Uh oh!

GuillaumeLagrange left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

not-matthias commented Jan 14, 2026 •

edited

Loading

codspeed-hq bot commented Jan 14, 2026 •

edited

Loading