Skip to content

fix: show all build phases in benchmark report#339

Open
carlos-alm wants to merge 1 commit intomainfrom
fix/benchmark-phases
Open

fix: show all build phases in benchmark report#339
carlos-alm wants to merge 1 commit intomainfrom
fix/benchmark-phases

Conversation

@carlos-alm
Copy link
Contributor

Summary

  • Add 4 missing phases to the benchmark phase breakdown table: astMs, cfgMs, dataflowMs, wasmPreMs
  • These phases account for ~75% of native build time since v3.0.1 but were invisible in the report
  • Add explanatory note documenting the v3.0.0 (4.4 ms/file) → v3.0.3 (12.3 ms/file) native regression cause: four new default-on phases totalling ~1,575ms, plus WASM double-parse on native builds

Root cause of regression

The native build time tripled not because existing phases got slower, but because four new phases became default-on in v3.0.1:

Phase Time (CI) Existed in 3.0.0?
AST nodes 651ms No
WASM pre-parse 388ms No
Dataflow 367ms No
CFG 169ms No
Total 1,575ms

The WASM pre-parse is particularly wasteful: on native builds, all files get parsed twice (Rust 85ms + WASM 388ms) because CFG/dataflow/complexity are JS code that needs tree-sitter AST trees the native engine doesn't expose.

Test plan

  • Script change is display-only — adds phase keys to markdown table
  • Notes section uses preserved NOTES_START/NOTES_END markers

The phase breakdown table in BUILD-BENCHMARKS.md was missing astMs,
cfgMs, dataflowMs, and wasmPreMs — phases that account for ~75% of
native build time since v3.0.1. Add them to phaseKeys/phaseLabels in
update-benchmark-report.js so future regenerations include them.

Also add a notes section explaining the v3.0.0→v3.0.3 regression:
the four new default-on phases total ~1,575ms, and the WASM pre-parse
double-parses all files on native builds because CFG/dataflow/complexity
are JS-only and need tree-sitter ASTs the native engine doesn't expose.
@claude
Copy link

claude bot commented Mar 4, 2026

Claude finished @carlos-alm's task —— View job


PR Review Complete ⭐⭐⭐⭐

Reviewing this PR to fix benchmark phase reporting. Following the 4-phase review process:

Todo List

  • Read changed files to understand implementation
  • Phase 1: Root Cause Analysis - determine if this fixes real issue vs hiding symptoms
  • Phase 2: Configuration Change Detection - check for high-risk changes
  • Phase 3: Test Integrity - watch for red flags in test changes
  • Phase 4: Code Quality - assess implementation quality
  • Provide final assessment with rating and recommendation

Root Cause Analysis

Problem Being Solved: This addresses a legitimate visibility issue where 4 critical build phases (astMs, cfgMs, dataflowMs, wasmPreMs) accounting for ~75% of native build time were missing from benchmark reports. The PR also adds detailed explanatory notes about a significant performance regression (4.4→12.3 ms/file from v3.0.0→v3.0.3).

Why This Approach: This is the correct fix - the phases existed in the data but were excluded from the display logic. The approach exposes existing data rather than masking problems.

Risk Assessment: ✅ LOW RISK - This is purely a reporting/documentation fix that reveals existing performance data.

Critical Concerns

NONE - This PR passes all red flag checks:

  • Phase 2: No configuration relaxation, no rule disabling, no timeout increases
  • Phase 3: No test changes whatsoever - only script and markdown updates
  • No scope creep - Changes are minimal and focused (5 lines of logic + documentation)

Code Quality Assessment

What Changed

  1. Line 155: Added 4 missing phase keys to phaseKeys array
  2. Lines 157,163,165,167: Added corresponding labels for the 4 phases
  3. Generated markdown: Added explanatory notes about the regression

Technical Review

  • Correct phase ordering: New phases placed logically in execution order (wasmPreMs after parseMs, astMs before complexityMs, etc.)
  • Complete mapping: All 4 missing phases have proper labels in phaseLabels object
  • Data integrity: The benchmark data already contained these phases - this just exposes them
  • Documentation quality: The regression explanation is technically accurate and detailed

Performance Context Validation

From the data, these phases represent significant work:

  • Native engine: astMs (651ms) + wasmPreMs (388ms) + dataflowMs (367ms) + cfgMs (169ms) = 1,575ms of previously hidden work
  • Total native build: 2,117ms, so these 4 phases = 74% of build time
  • The regression explanation correctly identifies the root cause: double-parsing due to JS-based CFG/dataflow needing AST trees

Final Recommendation

  • Rating: ⭐⭐⭐⭐ (4/5) - Good quality, well-documented fix
  • Action: APPROVE
  • Reasoning: This is exactly what benchmark reporting should do - expose all relevant performance data. The missing phases were masking the true source of build time, making performance analysis impossible. The detailed regression notes add valuable context for future optimization work.

Minor Enhancement Opportunity: The phase ordering could be further optimized to match actual execution order, but this is cosmetic and doesn't affect the data accuracy.


@greptile-apps
Copy link

greptile-apps bot commented Mar 4, 2026

Greptile Summary

This PR fixes the benchmark report to show all build phases. The changes add four missing phase keys (astMs, cfgMs, dataflowMs, wasmPreMs) to the phase breakdown table in scripts/update-benchmark-report.js, along with their corresponding labels. These phases account for approximately 75% of native build time since v3.0.1 but were invisible in previous reports.

The PR also adds a detailed note to BUILD-BENCHMARKS.md explaining the root cause of the native build regression from v3.0.0 (4.4 ms/file) to v3.0.3 (12.3 ms/file): four new phases that became default-on in v3.0.1, totalling ~1,575ms of additional work. The note documents that the WASM pre-parse is particularly wasteful on native builds, causing all files to be parsed twice (once by Rust, once by WASM).

Confidence Score: 5/5

  • This PR is safe to merge with minimal risk
  • Display-only changes to a benchmark reporting script with no runtime impact. The additions are syntactically correct and align with the PR description. The note is properly preserved within NOTES_START/NOTES_END markers.
  • No files require special attention

Important Files Changed

Filename Overview
scripts/update-benchmark-report.js Added 4 missing phase keys (wasmPreMs, astMs, cfgMs, dataflowMs) and labels to the phase breakdown table
generated/benchmarks/BUILD-BENCHMARKS.md Added explanatory note documenting the v3.0.0 → v3.0.3 native build regression and its root cause

Last reviewed commit: 886ec7b

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant