Skip to content

docs: update query benchmarks (3.6.0)#713

Merged
carlos-alm merged 4 commits intomainfrom
benchmark/query-v3.6.0-20260330-164422
Mar 31, 2026
Merged

docs: update query benchmarks (3.6.0)#713
carlos-alm merged 4 commits intomainfrom
benchmark/query-v3.6.0-20260330-164422

Conversation

@github-actions
Copy link
Copy Markdown
Contributor

Automated query benchmark update for 3.6.0 from workflow run #538.

@greptile-apps
Copy link
Copy Markdown
Contributor

greptile-apps bot commented Mar 30, 2026

Greptile Summary

This PR adds version 3.6.0 query benchmark data to the benchmark document, providing both native (Rust) and WASM rows in the summary table and a fully updated "Latest results" section. It directly addresses the prior review finding about missing delta indicators on the 3.6.0 native row.

Key changes:

  • Added 3.6.0 native row with correct delta indicators computed against 3.4.1 (last version with native data), and the 3.6.0 WASM row with deltas against 3.5.0.
  • All delta percentages verified as mathematically correct (e.g. fnDeps d1 native: (9.4−8.9)/8.9 = 5.6% → ↑6%; WASM diffImpact: (9−6.4)/6.4 = 40.6% → ↑41%).
  • Added a clear Note (3.6.0) explaining the non-standard native baseline (3.4.1 rather than 3.5.0), the mid-target change from db to node, and its role in the ↑41% WASM diffImpact jump.
  • Updated "Latest results" to show 3.6.0 with separate Native (Rust) and WASM detail tables; values in those tables are fully consistent with the JSON data block at the bottom of the file.
  • The JSON source-of-truth block has a well-formed 3.6.0 entry whose numeric values match every cell in the markdown tables exactly.

Confidence Score: 5/5

Safe to merge — documentation-only change with no logic, no code execution, and all data verified as internally consistent.

The previously flagged P1 concern (missing native delta indicators) is fully resolved in commit 1f9aa73. All delta percentages are mathematically correct, the JSON data block matches the markdown tables exactly, the note accurately explains the non-standard 3.4.1 baseline and the mid-target change driving the ↑41% WASM diffImpact. No P0 or P1 findings remain.

No files require special attention.

Important Files Changed

Filename Overview
generated/benchmarks/QUERY-BENCHMARKS.md Adds 3.6.0 native + WASM benchmark rows with correct delta indicators, updated "Latest results" section, a new explanatory note, and the raw JSON data block — addressing the prior review concern about missing native deltas.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A["3.6.0 native row\n(new)"] -- "baseline: 3.4.1 native\n(3.5.0 was wasm-only)" --> B["delta indicators\n↑6-10%"]
    C["3.6.0 wasm row\n(new)"] -- "baseline: 3.5.0 wasm" --> D["delta indicators\n↑7-41%"]
    D -- "↑41% diffImpact\nexplained by mid-target\nchange: db → node" --> E["Note (3.6.0)\nadded in NOTES section"]
    B --> E
    E --> F["JSON data block\n(source of truth)"]
    F -- "values match\nmarkdown table" --> A
    F -- "values match\nmarkdown table" --> C
Loading

Reviews (2): Last reviewed commit: "docs(benchmarks): add native delta indic..." | Re-trigger Greptile


| Version | Engine | fnDeps d1 | fnDeps d3 | fnDeps d5 | fnImpact d1 | fnImpact d3 | fnImpact d5 | diffImpact |
|---------|--------|----------:|----------:|----------:|------------:|------------:|------------:|-----------:|
| 3.6.0 | native | 9.4 | 9.6 | 9.4 | 3.4 | 3.4 | 3.3 | 8.3ms |
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Missing delta indicators for 3.6.0 native row

Every other row in the summary table includes delta indicators (↑%, ↓%, or ~) relative to the previous entry of the same engine. The 3.6.0 native row shows only raw values, with no comparison against 3.4.1 native (the last version that had native data). This makes the row visually inconsistent and harder to interpret at a glance.

Based on the 3.4.1 native baseline, expected deltas would be approximately:

  • fnDeps d1: 9.4 vs 8.9 → ↑6%
  • fnDeps d3: 9.6 vs 8.9 → ↑8%
  • fnDeps d5: 9.4 vs 8.8 → ↑7%
  • fnImpact d1: 3.4 vs 3.2 → ↑6%
  • fnImpact d3: 3.4 vs 3.1 → ↑10%
  • fnImpact d5: 3.3 vs 3.1 → ↑6%
  • diffImpact: 8.3ms vs 8.1ms → ~

If the generator intentionally skips deltas when the immediately preceding version lacks native data (3.5.0 was wasm-only), that behavior is worth documenting. If it's a generation bug, it should be fixed in the workflow before subsequent releases exhibit the same gap.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 1f9aa73. Added delta indicators to the 3.6.0 native row, computed against 3.4.1 native (the last version with native data):

  • fnDeps d1: 9.4 vs 8.9 → ↑6%
  • fnDeps d3: 9.6 vs 8.9 → ↑8%
  • fnDeps d5: 9.4 vs 8.8 → ↑7%
  • fnImpact d1: 3.4 vs 3.2 → ↑6%
  • fnImpact d3: 3.4 vs 3.1 → ↑10%
  • fnImpact d5: 3.3 vs 3.1 → ↑6%
  • diffImpact: 8.3ms vs 8.1ms → ~

Also added a note explaining that native deltas are relative to 3.4.1 since 3.5.0 was wasm-only.

Add missing delta indicators to the 3.6.0 native row (vs 3.4.1, the
last version with native data) and add an explanatory note for the
mid-target change (db → node) that accounts for the 41% WASM
diffImpact increase.
@carlos-alm
Copy link
Copy Markdown
Contributor

Both Greptile findings addressed in 1f9aa73:

  1. Missing delta indicators (inline comment): Added ↑6-10% deltas to the 3.6.0 native row, computed against 3.4.1 native (the last version with native data). diffImpact at 8.3ms vs 8.1ms marked as ~.

  2. Missing explanatory note (outside-diff comment): Added a Note (3.6.0) block explaining that native deltas are relative to 3.4.1 (3.5.0 was wasm-only), the mid-target change from db to node accounts for the 41% WASM diffImpact jump, and fnDeps/fnImpact growth of 6-10% is consistent with codebase expansion.

@carlos-alm
Copy link
Copy Markdown
Contributor

@greptileai review

@carlos-alm carlos-alm merged commit 5c48a23 into main Mar 31, 2026
12 checks passed
@carlos-alm carlos-alm deleted the benchmark/query-v3.6.0-20260330-164422 branch March 31, 2026 04:57
@github-actions github-actions bot locked and limited conversation to collaborators Mar 31, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant