feat: /oracle — Product Conscience with AST-powered codebase scanner#509
Closed
rafiulnakib wants to merge 14 commits intogarrytan:mainfrom
Closed
feat: /oracle — Product Conscience with AST-powered codebase scanner#509rafiulnakib wants to merge 14 commits intogarrytan:mainfrom
rafiulnakib wants to merge 14 commits intogarrytan:mainfrom
Conversation
Routes are now classified using git history instead of import graph traversal. For each route's page file, git log finds co-changed files and excludes shared infrastructure (files touching >25% of pages). This eliminates the shared-file inflation that made 73% of routes "mega". Routes sorted chronologically by born date instead of tier+frequency. Removed getGitFrequency(), branch_depth, trace_depth_cap from RouteEntry.
The scanner is now an internal implementation detail. It runs silently at the start of every /oracle inventory and /oracle stats session. Users never see raw scan output or /oracle scan as a command. Merges product health and codebase health into a single /oracle stats dashboard. Moves --visualize flag to /oracle inventory --visualize.
The scan manifest is read once to build the work queue, then never referenced during route analysis. Deducting it from budget was consuming 85% of available lines on real projects (7,659-line manifest / 3 = 2,553 of 3,000 budget), leaving zero room for actual route processing. Only the product map is deducted — Claude actively references it while writing inventory docs. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The visualizer (visualize-graph.ts) expects `route` and `branch_files` (array) fields but scan-imports.ts outputs `path` and `branch_files` (count) + `co_changed_files` (array). This field mismatch causes the graph to render with all route IDs as "route:undefined", producing a blank page. Removing the flag until the scanner/visualizer field contract is aligned. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Adds /oracle to the skill table and a full deep dive section covering: - Product conscience philosophy (silent integration) - The 6 modes (bootstrap, inventory, refresh, update, stats, query) - How inventory works (budgeted, checkpointed, two-tier docs) - Silent READ/WRITE resolver integration across 8 skills - Git co-change classification explanation - Example output for /oracle and /oracle stats Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Updated feature descriptions and complexity metrics in skills documentation.
Refresh mode silently ignored existing Tier 2 inventory docs, leaving them orphaned on disk with no Inventory: pointer in the product map. Add Inventory: field to Tier 1 entry template and schema reference, reorder checkpoint to write Tier 2 first so the pointer is available, and add a new "wire inventory docs" step to refresh that discovers existing docs by F-number prefix. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
a8157cb to
90702b3
Compare
Inventory docs now go to docs/oracle/inventory/ in the project repo instead of ~/.gstack/projects/$SLUG/inventory/. This makes them version-controlled, discoverable by any contributor or Claude session, and browsable in the repo without needing access to ~/.gstack. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
On first inventory run, detects existing Tier 2 docs at the old ~/.gstack/projects/$SLUG/inventory/ location, copies them to docs/oracle/inventory/ in the project repo, and removes the old dir. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…location Scanner pipeline was completely non-functional (6/7 modules missing, only core.ts existed). This builds everything scan-imports.ts needs: - scanner/routes.ts: framework detection (React Router, Next.js App/Pages) + layered route discovery (filesystem, router parsing, Edge Functions) - scanner/aliases.ts: Vite alias resolution via AST + eval fallback - scanner/dead-code.ts: unreachable file detection from import graph - scanner/css.ts: CSS/SCSS import tracking (@import, @use) - scanner/monorepo.ts: workspace detection (npm, pnpm, lerna) - scanner/non-ts.ts: non-TypeScript file discovery (30+ extensions) - scanner/core.ts: generalized findFiles() walker from findTsFiles() - 7 test fixture directories for all scanner modules - 6 new tests (84 total pass) - SKILL.md: PRODUCT_MAP.md relocated from memory dir to docs/oracle/ as single source of truth (auto-migration for legacy projects) Verified: scanner produces valid manifest on iskool-prod (536 files, 73K lines). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Scanner was failing in Claude Code's sandboxed shell because bun/node weren't in PATH. Now compiles to a self-contained binary via bun build --compile, same pattern as browse/design/gstack-global-discover. - Add typescript as explicit dependency (used by core.ts, aliases.ts) - Add scan-imports compile step to build script - Update SKILL.md to invoke compiled binary directly Note: run `bun run build` to produce oracle/bin/dist/scan-imports binary, then `git add oracle/bin/dist/scan-imports` to commit it. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add /oracle to README install instruction skill lists (Step 1 + Step 2) - Add oracle/ directory to CLAUDE.md project structure tree - Update CLAUDE.md compiled binaries section to include oracle/bin/dist/ Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
feat: /oracle — Product Conscience for gstack
Summary
/oraclegives gstack persistent product memory. Every skill today operates in isolation:/shipdoesn't know what/office-hoursplanned,/plan-eng-reviewcan't warn about patterns that failed last month,/qadoesn't know which features depend on the code being tested.This PR adds two things:
/oracle) with 6 modes for building, querying, and inventorying product knowledge{{PRODUCT_CONSCIENCE_READ}}and{{PRODUCT_CONSCIENCE_WRITE}}) embedded across 8 existing skills, making product awareness automatic and invisibleRoute classification uses git co-change analysis (language-agnostic, no AST needed). The AST-powered scanner handles diagnostics only (dead files, circular deps, import graph) and compiles to a standalone binary so it works in Claude Code's sandboxed shell without
bunornodein PATH.44 files changed. 4,479 insertions. 84 scanner tests, 0 failures.
Table of Contents
Motivation
Every gstack skill operates in isolation.
/shipdoesn't know what/office-hoursplanned./plan-eng-reviewcan't warn about patterns that failed in a previous feature./qadoesn't know which features depend on the code being tested./oraclesolves this by maintaining a persistent, structured product map that:The classification system addresses a second problem: how to understand codebase complexity without being TypeScript-specific. Git co-change analysis discovers which files are feature-specific vs. shared infrastructure by analyzing commit patterns. Language-agnostic, no AST needed.
The AST scanner addresses a third problem: architectural health. It builds a true import graph, discovers routes across 10 framework conventions, detects circular dependencies via Tarjan's SCC, and finds dead code via BFS reachability.
Architecture Overview
Data flow:
/ship,/plan-eng-review)/oracledirectly for bootstrap, inventory, stats, or queriesClassification data flow:
File Inventory
New Files — Oracle Core
oracle/SKILL.md.tmploracle/SKILL.mdbun run gen:skill-docsoracle/bin/scan-imports.tsoracle/bin/dist/scan-importsoracle/bin/scanner/core.tsoracle/bin/scanner/routes.tsoracle/bin/scanner/aliases.tsoracle/bin/scanner/dead-code.tsoracle/bin/scanner/css.tsoracle/bin/scanner/monorepo.tsoracle/bin/scanner/non-ts.tsoracle/bin/visualize-graph.tsoracle/bin/terminal-graph.tsscripts/resolvers/oracle.tsNew Files — Tests & Fixtures
oracle/bin/scan-imports.test.tsoracle/bin/visualize-graph.test.tsoracle/bin/terminal-graph.test.tsscripts/resolvers/oracle.test.tsoracle/bin/__fixtures__/Modified Files — Product Conscience Integration (8 skills)
autoplan/SKILL.md.tmpl{{PRODUCT_CONSCIENCE_READ}}+{{PRODUCT_CONSCIENCE_WRITE}}office-hours/SKILL.md.tmpl{{PRODUCT_CONSCIENCE_READ}}+{{PRODUCT_CONSCIENCE_WRITE}}plan-ceo-review/SKILL.md.tmpl{{PRODUCT_CONSCIENCE_READ}}+{{PRODUCT_CONSCIENCE_WRITE}}plan-design-review/SKILL.md.tmpl{{PRODUCT_CONSCIENCE_READ}}+{{PRODUCT_CONSCIENCE_WRITE}}plan-eng-review/SKILL.md.tmpl{{PRODUCT_CONSCIENCE_READ}}+{{PRODUCT_CONSCIENCE_WRITE}}qa/SKILL.md.tmpl{{PRODUCT_CONSCIENCE_READ}}+{{PRODUCT_CONSCIENCE_WRITE}}review/SKILL.md.tmpl{{PRODUCT_CONSCIENCE_READ}}+{{PRODUCT_CONSCIENCE_WRITE}}ship/SKILL.md.tmplscripts/resolvers/index.tsPRODUCT_CONSCIENCE_READandPRODUCT_CONSCIENCE_WRITEModified Files — Documentation & Build
package.jsonbun build --compile oracle/bin/scan-imports.tsto build script. Addedtypescriptas explicit dependency.README.md/oracleto skill table and install instructionsCLAUDE.mdoracle/to project structure tree and compiled binaries sectiondocs/skills.md/oracledeep dive sectionCHANGELOG.mdVERSION0.13.1.0All 8 skill
.mdfiles auto-generated from their.tmplviabun run gen:skill-docs.The Product Map (
PRODUCT_MAP.md)The product map lives at
docs/oracle/PRODUCT_MAP.mdin the project repo, with a pointer in Claude'sMEMORY.md. It is the persistent data store for all product knowledge.Schema (Version 1)
Key Design Decisions
Dynamic categories: No fixed taxonomy. Claude infers categories from each feature's purpose (e.g., "data-views", "content-editor", "auth"). Categories emerge organically, making
/oracleindustry-agnostic.Progressive compression: SHIPPED features > 3 months old, not referenced by other features' Connections/Depends on, get compressed to one-liners. The
docs:field preserves pointers to Tier 2 per-feature documentation. If the map exceeds 200 lines, the threshold drops to 2 months.Schema version migration: The WRITE resolver checks for
<!-- schema_version: 1 -->. If missing, it performs additive-only migration: adds missing sections, infers categories, preserves all existing data.Verify-before-write: For all updates, Claude greps the codebase to confirm components and patterns actually exist. The product map mirrors reality, never aspirational state.
In-repo storage: PRODUCT_MAP.md lives at
docs/oracle/in the project repo (not~/.gstack/or~/.claude/). This means it's committed, versioned, and visible to the whole team. Legacy locations are auto-migrated on first run.The 6 User-Facing Modes
Mode Detection
/oracle(no product map)/oracle(product map exists)/oracle inventory/oracle refresh/oracle update/oracle stats/oracle {question}There is no
/oracle scancommand. The scanner runs automatically as an internal implementation detail of inventory and stats.Bootstrap
Creates an initial product map from git history and code analysis.
docs/oracle/, adds pointer to MEMORY.md, writes timestamp breadcrumbPost-bootstrap, offers
/oracle inventoryfor deeper AST-powered analysis.Inventory
Budgeted deep page-by-page scan. Automatically runs the compiled scanner binary, then documents features guided by the findings.
Step 0: Auto-scan (silent, internal). The scanner binary runs automatically at the start of every inventory session. Users never see raw scan output. The data (route count, classification, circular deps, dead files) is used internally by subsequent steps.
Budget system:
Two-tier documentation:
docs/oracle/F{NNN}-{feature-name}.mdProcessing order: Routes sorted chronologically by born-date (oldest first). Foundation routes documented first, newest routes last. This matches how codebases are actually built.
Checkpointing: After every 5-7 routes, progress is saved. If context runs out, running
/oracle inventoryagain resumes from the checkpoint.Refresh
Rebuilds the product map from scratch. Backs up existing map, runs fresh bootstrap using current git history, preserves feature ID sequence.
Update
Reconciles the product map with commits that happened outside gstack. Reads the breadcrumb timestamp, gets commits since last write, classifies signal vs. noise, updates affected entries.
Stats
Unified product health + codebase health dashboard. Auto-scans silently before presenting.
Query
Read-only queries ("what depends on auth?") execute instantly. Write queries ("fix F003 description") trigger verify-before-write: Claude greps the codebase to confirm changes match reality. If a requested update contradicts the code, the update is refused with a suggestion to run
/office-hoursto plan the change first.Corruption Detection
On every read, the skill checks for 5 required section headers (Features, Reusable Patterns, Anti-Patterns, Product Arc, Identity). If any are missing, offers
/oracle refreshto rebuild.Git Co-Change Classification
The core insight: instead of counting all files reachable through imports (which inflates every route with shared infrastructure), use git history to find files that actually co-change with each route's page file. Files that co-change with many pages are shared infrastructure and excluded.
Algorithm:
git log --name-only --format=COMMIT:%H -- {pageFile}max(3, totalPages * 0.25)— files above this are infrastructurebranch_linesclassify(branch_lines)using thresholds (EASY < 800, MEDIUM < 2500, HARD < 3000, MEGA > 3000)Born-date ordering:
git log --follow --diff-filter=A --format=%at -- {pageFile}returns the timestamp of the first commit that added each page file. Routes sorted chronologically — foundation first, newest last.Why this works:
Properties:
Real-world results (iskool-prod, 26 routes):
73% MEGA → balanced distribution. The shared-file inflation that made every route "mega" is eliminated.
AST-Powered Scanner
The scanner is a standalone TypeScript program (~3,680 lines across 8 modules) compiled to a self-contained binary. Users never invoke it directly — it runs automatically at the start of inventory and stats sessions.
Compiled Binary
oracle/bin/dist/scan-importsis a Mach-O arm64 binary (~67MB) built viabun build --compile. Same compilation pattern used bybrowse/dist/browseanddesign/dist/design.This solves a real problem: Claude Code's sandboxed shell often doesn't have
bun,node,npx, ortsxin PATH. When the scanner was a.tsfile requiringbun run, it failed silently in sandboxed environments and oracle fell back to slow, incomplete manual route discovery. The compiled binary just works.Build command (in
package.json):Modules
Framework Detection
Auto-detects the project's routing framework. Supports 10 frameworks:
nextjs-pagesnext.config.*+pages/directorynextjs-appnext.config.*+app/directory withpage.tsxnuxtnuxt.config.*existssveltekitsvelte.config.*existsremixremix.config.*existsastroastro.config.*existsreact-routerreact-routerimportsvue-routervue-routerimportsfile-basedsrc/pages/directory exists (generic)unknownRoute Discovery
Each framework has a dedicated discovery function:
<Route>+createBrowserRouter+ nested/lazy routes. Supports TanStack Router, Wouter.pages/walk, dynamic segments ([id],[[...slug]]), skips_app,_document,_error.page.tsx/route.tsxare valid routes.src/pages/.supabase/functions/*/index.ts).src/workers/,src/jobs/,src/cron/,src/webhooks/.Import Graph Construction
Uses
ts.createProgramandts.resolveModuleNamefor accurate resolution. Walks every source file's AST for import/export declarations, dynamic imports, and require calls. Merges Vite aliases into compiler options. Each file becomes aFileNodewithlines,content_hash,imports[], andunresolved_imports[].Circular Dependency Detection (Tarjan's SCC)
Standard Tarjan's algorithm. Each SCC with > 1 node = circular dependency. Severity by cycle length: ≤2 files = HIGH, 3-4 = MEDIUM, 5+ = LOW.
Dead Code Detection (BFS Reachability)
Seeds from route entry files + standard entry points + config files. BFS through import graph, marking reachable files. Unreached files filtered through exclusions (tests, stories, configs, supabase). Supports
.oracleignorefor manual exclusions and barrel file detection. Only HIGH confidence dead files (not imported by any other file) included in manifest.Scan Manifest Schema
Product Conscience Resolvers
Two functions in
scripts/resolvers/oracle.tsthat generate instruction text, embedded in 8 skills via{{PRODUCT_CONSCIENCE_READ}}and{{PRODUCT_CONSCIENCE_WRITE}}placeholders.READ Resolver
Fires at: The START of each skill, before any planning or review.
depends_onWRITE Resolver
Fires at: After each skill completes. Silent, no user confirmation.
Status transitions:
/office-hours/plan-*-review,/autoplan/ship/qa/reviewFeature Lifecycle
The
/shipWRITE resolver fires after push (Step 7), before PR creation (Step 8) — at this point the code is committed reality.Skill Integration Map
/autoplan/office-hours/plan-ceo-review/plan-design-review/plan-eng-review/qa/review/shipUse Cases
Solo Developer
You're building a SaaS app. After 3 months, 15+ features, no documentation.
/oracle→ Bootstrap detects 15 features from git history/oracle inventory→ Auto-scans, discovers 24 routes, documents features chronologically/plan-eng-reviewfor a new feature → intelligence brief shows connections, dependencies, anti-patterns/ship→ WRITE resolver silently updates the product map with the new featureTeam Onboarding
/oracle inventory→ Comprehensive feature documentation with Tier 2 per-feature docs/oracle stats→ unified health dashboard, complexity distributionLegacy Codebase Refactoring
/oracle inventory→ scanner finds circular deps (Tarjan's SCC), dead files (BFS reachability)/plan-eng-reviewwarns about all dependent featuresKnown Constraints
discoverNonTsFiles()finds.py,.rb,.go,.rs,.javafiles and adds to manifest. Visualizer renders them as gray "unanalyzed" nodes.import.meta.glob()andrequire.context()patterns. Dynamicimport()with variable paths flagged inunresolved_imports../setupbuilds from source for other platforms..oracle/PRODUCT_MAP.md+.oracle/PRODUCT_MAP.jsonin repo root for non-Claude consumption.Testing
135 tests, 4 files, all passing
scan-imports.test.tsvisualize-graph.test.tsterminal-graph.test.tsoracle.test.tsTest Fixtures — 7 directories, 40 files
react-router-project/nextjs-project/vite-aliases/monorepo-project/empty-project/css-project/deferred-imports/🤖 Generated with Claude Code