Replace ReactFlow+ELK diagrams with Next.js+Graphviz and add CI auto-update#650
Draft
Replace ReactFlow+ELK diagrams with Next.js+Graphviz and add CI auto-update#650
Conversation
React + ReactFlow app visualizing all 9 pipeline stages (0-8) plus an Overview. Uses ELK.js for automatic graph layout with orthogonal edge routing. Each stage shows inputs, processes, outputs, utilities, and external integrations as color-coded nodes. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add PolicyEngine UK pipeline (10 stages: Overview + Stage 0-8) - Add country selector dropdown (US / UK) in sidebar - Add uk_specific node type (teal) for UK-only processing - Add absent node type (gray dashed) for cross-referencing - Show UK-only steps as absent in US stages (Stage 1, 4) - Show US-only steps as absent in UK stages - Update routes from / to /us and /uk prefixes - Rename app title to "PolicyEngine data pipelines" Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add ELK partition support to force sequential stage ordering in Overview - Fix edge label placement bug (was using array index instead of visual midpoint) - Add z-index to edge labels so they render above nodes - Increase between-layer spacing for better label readability Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
New "under construction" node type (amber dashed) for showing pipeline changes that are actively being developed: US: - PR #611: Pipeline orchestrator in Overview (Modal hardening) - PR #540: Category takeup rerandomization in Stage 2, extracted puf_impute.py + source_impute.py modules in Stage 4 - PR #618: CMS marketplace data + plan selection in Stage 5 UK: - PR #291: New Stage 9 — OA calibration pipeline (6 phases) - PR #296: New Stage 10 — Adversarial weight regularisation - PR #279: Modal GPU calibration nodes in Stages 6, 7, Overview Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- PR #611 orchestrator now coordinates all stages (0-8), not just 5-8 - UK PR #291 (OA clone-and-assign) repositioned between Stage 4 and Stage 5 - UK PR #296 relabeled as standalone tool (not a pipeline stage) - Sidebar updated to show PR #296 as "Tool" instead of "Stage 10" Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add scripts/generate_pipeline_docs/: multi-agent system using Anthropic SDK that reads pipeline source code and produces Graphviz DOT diagrams (discovery → per-stage exploration → bleed cleanup → validation → DOT generation) - Replace docs/pipeline-diagrams/ Vite+ReactFlow app with Next.js app using @hpcc-js/wasm-graphviz for DOT→SVG rendering - Add GitHub Actions workflow to auto-regenerate diagrams on pipeline changes - Include placeholder DOT files and manifest.json for immediate preview Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…line" This reverts commit 457eb13.
…update workflow Migrate pipeline diagrams from Vite+ReactFlow+ELK to Next.js+WASM Graphviz with DOT files as source of truth. Add manifest.json for stage definitions and a 3-phase CI workflow (stage list evaluation, per-stage DOT update via Claude Code CLI agents, structural validation with auto-fix) that runs on PRs changing Python source files. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Trigger on push to main instead of pull_request, drop paths filter to handle squash merges reliably, use github.event.before..after for the diff range, and commit diagram updates directly to main. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
anth-volk
force-pushed
the
add-pipeline-diagrams
branch
from
96127ed to
1e6ac4a
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
manifest.jsonas single source of truth for stages, titles, and Python source file mappingsCLAUDE.md) auto-loaded by CI agentsCloses #649
Test plan
npx next build)🤖 Generated with Claude Code