[Bug]: "Some requests are slow" toast triggered by unbounded git subprocess calls in replayEvents enrichment path

[mei-the-dev]

Summary

The "Some requests are slow" toast (SLOW_RPC_ACK_THRESHOLD_MS = 15_000) fires persistently because replayEvents — a tracked RPC call — can exceed 15 s due to synchronous git subprocess invocations in its event-enrichment path.

Related user-facing report: #1986

---

Root Cause Trace

1. replayEvents blocks on enrichOrchestrationEvents

apps/server/src/ws.ts — ORCHESTRATION_WS_METHODS.replayEvents handler:

Stream.runCollect(
  orchestrationEngine.readEvents(clamp(input.fromSequenceExclusive, { ... })),
).pipe(
  Effect.map((events) => Array.from(events)),
  Effect.flatMap(enrichOrchestrationEvents),   // blocks the RPC response
  ...
)

enrichOrchestrationEvents runs with concurrency: 4 but is still fully sequential relative to the RPC response — the Success frame is only sent after all enrichment is done.

2. enrichProjectEvent spawns git subprocesses

For every project.created or project.meta-updated event in the replay window, enrichProjectEvent calls:

repositoryIdentityResolver.resolve(event.payload.workspaceRoot)

3. RepositoryIdentityResolver runs two git subprocesses per cache miss

apps/server/src/project/Layers/RepositoryIdentityResolver.ts:

// Step 1 — always run (no cache for this step):
await runProcess("git", ["-C", cwd, "rev-parse", "--show-toplevel"])

// Step 2 — cached, but with aggressive expiry:
await runProcess("git", ["-C", cacheKey, "remote", "-v"])

Cache TTLs:

const DEFAULT_POSITIVE_CACHE_TTL = Duration.minutes(1);   // identity found
const DEFAULT_NEGATIVE_CACHE_TTL = Duration.seconds(10);  // no remote / not a repo

The 10-second negative TTL is the critical problem. For projects without a configured remote (common for local projects, detached worktrees, or when git remote fails), the cache entry expires every 10 s. Because replayEvents is called on every reconnect and includes every historical project.created event, it re-spawns git subprocesses for each such project on every replay after the 10-second window. With multiple projects or frequent reconnects this easily pushes replayEvents past 15 s — toast fires.

4. resolveRepositoryIdentityCacheKey is uncached

The first git call (rev-parse --show-toplevel) runs outside the Effect Cache and is not memoised at all:

async function resolveRepositoryIdentityCacheKey(cwd: string): Promise<string> {
  // always spawns a subprocess — not cached
  const topLevelResult = await runProcess("git", ["-C", cwd, "rev-parse", "--show-toplevel"], ...);
  ...
}

For N project.* events in the replay set, this spawns N git processes unconditionally before the cache is even consulted.

5. No timeout on the enrichment step

There is no Effect.timeout guard around enrichOrchestrationEvents, so a slow git remote -v (e.g. a network-reachable remote with high latency) stalls the entire RPC indefinitely.

---

Reproduction

1. Have 2+ projects in T3 Code where at least one project has no git remote, or where git fetch is slow/timing out.
2. Send a prompt (generates orchestration events).
3. Trigger a WebSocket reconnect (network blip, app focus after sleep, etc.).
4. replayEvents replays all events → enrichProjectEvent re-resolves every project → toast fires.

The server-child.log entry from #1986 confirms the underlying git fetch is timing out repeatedly:

GitCommandError: Git command failed in GitCore.fetchUpstreamRefForStatus:
  git fetch --quiet --no-tags origin +refs/heads/main:refs/remotes/origin/main - timed out.

---

Suggested Fixes

Fix 1 — Cache resolveRepositoryIdentityCacheKey (quick win)

rev-parse --show-toplevel runs on every call regardless of the Effect cache below it. Wrap the result in a separate Map or include it in the same Effect Cache keyed by raw cwd:

const cwdToTopLevelCache = new Map<string, string>();

async function resolveRepositoryIdentityCacheKey(cwd: string): Promise<string> {
  const hit = cwdToTopLevelCache.get(cwd);
  if (hit !== undefined) return hit;
  // ... existing rev-parse logic ...
  cwdToTopLevelCache.set(cwd, cacheKey);
  return cacheKey;
}

Fix 2 — Increase the negative cache TTL (quick win)

const DEFAULT_NEGATIVE_CACHE_TTL = Duration.minutes(1); // was Duration.seconds(10)

Fix 3 — Add a per-event timeout in enrichProjectEvent

case "project.created":
case "project.meta-updated":
  return repositoryIdentityResolver.resolve(event.payload.workspaceRoot).pipe(
    Effect.timeout(Duration.seconds(3)),
    Effect.orElse(() => Effect.succeed(null)),
    Effect.map((repositoryIdentity) => ({
      ...event,
      payload: { ...event.payload, repositoryIdentity },
    })),
  );

Fix 4 — Move enrichment out of the replay path (longer-term)

Store repositoryIdentity on the event at write time (inside the project.created projection), eliminating the git subprocess from the RPC hot path entirely.

Fix 5 — Add a dismiss button to the slow toast (UX)

SlowRpcAckToastCoordinator in apps/web/src/components/WebSocketConnectionSurface.tsx creates the toast with no action button. Users have no way to dismiss it while the underlying issue persists (see #1986 comments). Adding an action to call clearAllTrackedRpcRequests would resolve this:

const nextToast = {
  description: describeSlowRpcAckToast(slowRequests),
  timeout: 0,
  title: "Some requests are slow",
  type: "warning" as const,
  actionProps: {
    children: "Dismiss",
    onClick: clearAllTrackedRpcRequests,
  },
};

---

Affected Files

File	Issue
apps/server/src/ws.ts	enrichOrchestrationEvents blocks replayEvents RPC response
apps/server/src/project/Layers/RepositoryIdentityResolver.ts	resolveRepositoryIdentityCacheKey uncached; negative TTL too short (10 s)
apps/web/src/components/WebSocketConnectionSurface.tsx	"Some requests are slow" toast has no dismiss button