feat: add @hyperframes/renderer - client-side video rendering by miguel-heygen · Pull Request #239 · heygen-com/hyperframes

miguel-heygen · 2026-04-10T01:06:47Z

Summary

New @hyperframes/renderer package — renders HyperFrames compositions to MP4 entirely in the browser. Zero server, zero Puppeteer, zero FFmpeg.

Three frame sources (pluggable architecture)

Source	Fidelity	Availability	Use case
SnapDOM (default)	~30 dB PSNR	All modern browsers	Production — same tier as Remotion's client-side renderer
drawElementImage	42 dB PSNR	Chrome + flag	Pixel-perfect — uses Chrome's native compositor
Tab Capture	Experimental	Chrome 107+	Research path

How it works

Composition (HTML + GSAP) → hidden iframe(s)
  → SnapDOM / drawElementImage capture → ImageBitmap
  → Web Worker (WebCodecs VideoEncoder + MediaBunny muxer)
  → MP4 Blob download

Parallel iframe pool for concurrent frame capture (mirrors server-side parallelCoordinator)
OfflineAudioContext for multi-track audio mixing
WebCodecs for hardware-accelerated H.264/VP9 encoding
MediaBunny for MP4/WebM muxing

PSNR Parity Results (vs. BeginFrame golden reference)

=== css-import-scoping (1080x1920 @ 30fps) ===

  Time      drawElementImage    SnapDOM
  ----      ----------------    -------
  1.0s      42.0 dB             30.7 dB
  2.0s      42.0 dB             30.7 dB
  3.0s      42.0 dB             30.7 dB
  4.0s      42.0 dB             30.7 dB

drawElementImage achieves 42 dB — 12 dB above the producer's 30 dB regression threshold. This is broadcast-quality parity with the server-side renderer.

Studio Integration

Added "Browser" export button in the render queue panel. Uses @hyperframes/renderer via dynamic import — click to render client-side and auto-download the MP4.

Package structure

packages/renderer/
  src/
    index.ts              — render(), createRenderer(), isSupported()
    renderer.ts           — Pipeline orchestrator
    sources/
      snapdom.ts          — SnapDOM (shippable today)
      draw-element-image.ts — html-in-canvas (pixel-perfect, Chrome flag)
      tab-capture.ts      — getDisplayMedia (experimental)
    encoding/worker.ts    — Web Worker (WebCodecs + MediaBunny)
    audio/mixer.ts        — OfflineAudioContext mixer
    capture/
      iframe-pool.ts      — Parallel iframe management
      video-frame-injector.ts — Canvas overlays for <video>
  scripts/
    compile-test.ts       — Pre-compile compositions with runtime injection
    regression-parity.ts  — PSNR comparison harness

Browser requirements

Chrome 94+, Firefox 130+, Safari 26+ (WebCodecs). drawElementImage requires Chrome with --enable-blink-features=CanvasDrawElement.

Test plan

21 unit tests passing, 2 integration tests (skipped in jsdom)
TypeScript typecheck clean
QA: renders composition to MP4 in headed Chrome
PSNR parity: 42 dB with drawElementImage, 30 dB with SnapDOM
Studio: "Browser" export button wired and typechecking
Full regression suite (27 compositions) — tracked for follow-up

🤖 Generated with Claude Code

mintlify · 2026-04-10T01:06:53Z

Preview deployment for your docs. Learn more about Mintlify Previews.

Project	Status	Preview	Updated (UTC)
hyperframes	🟢 Ready	View Preview	Apr 10, 2026, 1:07 AM

💡 Tip: Enable Workflows to automatically generate PRs for you.

Design spec for @hyperframes/renderer — a new browser-only package that renders compositions to MP4 using WebCodecs + MediaBunny + OffscreenCanvas workers, with SnapDOM as interim frame source and html-in-canvas as the future pixel-perfect path. Replaces the need for Puppeteer + FFmpeg. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

13-task plan covering package scaffold, SnapDOM frame source, WebCodecs encoding worker, MediaBunny muxer, OfflineAudioContext audio mixer, parallel iframe capture, video frame injector, and main orchestrator. 56 total steps with TDD approach. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Adds the @hyperframes/renderer package skeleton: package.json, tsconfig.json, src/types.ts (all pipeline interfaces), src/compat.ts (WebCodecs feature detection), and src/index.ts (re-exports). Typecheck passes with zero errors. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Implements SnapdomFrameSource class that loads compositions in hidden iframes, seeks via window.__hf.seek(), and captures DOM frames using @zumer/snapdom. Adds jsdom dev dependency for the test environment. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Implements the encoding pipeline for @hyperframes/renderer: worker message protocol types, Web Worker entry point (VideoEncoder + MediaBunny muxer), and the main-thread Encoder class that manages the worker lifecycle. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Implements mixAudio() with TDD: decodes audio sources, applies volume/timing via GainNode, and renders a mixed PCM AudioBuffer using OfflineAudioContext. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

…enderer) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

- Fix audio format: replace interleaveChannels with concatPlanarChannels - Fix MIME type: derive video/mp4 vs video/webm from outputFormat in finalize - Add isSupported() guard at top of render() for early failure - Fix durationMs to report wall-clock render time (totalMs) - Fix audio filter to use strict equality (hasAudio === true) - Remove unused progress type from WorkerOutMessage union Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

…page - Replace requestAnimationFrame with setTimeout in __hf protocol polling (rAF doesn't fire for off-screen iframes) - Add workerUrl option to EncoderOptions and RenderConfig for configuring the encoding worker URL (needed for bundled deployments) - Add QA test page with self-contained CSS-animated composition Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

… progress ETA - Capture doc.body instead of doc.documentElement in SnapDOM to fix 75% viewport mismatch - Make audio track conditional (hasAudio flag) to prevent corrupt empty audio tracks - Add publishConfig for npm publishing - Wire ProgressTracker for estimatedTimeRemaining and captureRate in progress callbacks - Recompute video overlay canvas position per-frame for GSAP-animated elements - Add AbortSignal to captureAll for cancel propagation Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

…ression harness drawElementImage: pixel-perfect capture via html-in-canvas API using <canvas layoutsubtree> with iframe child. Achieves 42 dB PSNR vs 30 dB for SnapDOM. Requires Chrome --enable-blink-features=CanvasDrawElement. Tab-capture: experimental getDisplayMedia-based capture. Also adds regression parity scripts for compiling test compositions and comparing PSNR against golden reference renders. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Adds a "Browser" export button next to the existing server-side Export button in the RenderQueue panel. Uses the @hyperframes/renderer package to render compositions entirely client-side via WebCodecs, with progress tracked in the existing job list and automatic file download on completion. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

…ker URL - SnapDOM frame source now detects window.__player (studio runtime) in addition to window.__hf and creates a bridge automatically. This lets the renderer work with studio preview URLs directly. - Studio useBrowserRender hook: force snapdom frame source and provide explicit worker URL for Vite dev compatibility. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

- iframe-pool: setTimeout(0) yield after each captured frame - SnapDOM source: requestAnimationFrame after seek() before capture to ensure layout/paint is processed This prevents Chrome from becoming unresponsive during long renders, keeping CDP, DevTools, and UI interactions functional. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Single-iframe capture keeps Chrome responsive during long renders. With 8 parallel iframes, SnapDOM overwhelms the main thread and locks the browser. Concurrency=1 trades speed for reliability. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

…ckground rendering - Discover media elements from DOM when __player bridge doesn't expose hf.media (scans both <video> and <audio> elements with data-start/data-duration) - Resolve media src to absolute URLs so audio mixer fetch() works cross-origin - Seek original video elements directly instead of clone+canvas overlay approach (SnapDOM natively captures <video> via drawImage) - Force synchronous reflow after GSAP seek to prevent stale computed styles - Add warmup phase: wait for fonts.ready + settle ticks before first capture - Replace all requestAnimationFrame with setTimeout for background tab support - Fix interleaved capture/encoding progress by deferring encoding reports Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

…set copying - Use async spawn for agent-browser wait so Node event loop stays active for the upload HTTP server to process browser POST requests - Add CORS headers to upload server responses - Add frameSource: 'snapdom' to test page (headless Chrome auto-detects tab-capture which hangs waiting for screen share permission) - Copy source assets to output dir (CSS, fonts not inlined by compiler) - Use compile-test.ts for compilation (injects runtime + __hf bridge) - Write serve.json with cleanUrls:false to prevent .html→extensionless redirects - Increase upload timeout from 10s to 300s for long compositions Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

…om PR - Remove docs/superpowers/plans and specs (brainstorming artifacts) - Remove packages/renderer/qa/ (dev testing pages) - Remove packages/renderer/pnpm-lock.yaml (monorepo uses root lockfile) - Remove run-parity-test.sh (superseded by regression-parity.ts) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Keep both @hyperframes/player (from main) and @hyperframes/renderer deps. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Multi-tab parallel capture using noopener windows + BroadcastChannel for true multi-process SnapDOM parallelism. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Opens N browser tabs (via window.open with noopener) for true multi-process parallelism. Each tab gets its own Chromium renderer process with independent main thread, bypassing the single-thread SnapDOM bottleneck. Architecture: - TurboPool coordinator opens N minimized offscreen tabs - Each tab loads composition via SnapdomFrameSource independently - Frames captured as PNG, sent via BroadcastChannel to coordinator - Coordinator decodes to ImageBitmap, feeds reorder buffer + encoder - Concurrency: min(cores/2, 6), capped at 6 tabs - Falls back to normal IframePool if popups blocked Also: - Add turbo toggle (lightning bolt) to Studio render UI - Add /api/projects/:id/renders/upload endpoint for saving browser-rendered videos to project renders directory - Export isTurboSupported() from @hyperframes/renderer Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

The Vite middleware was reading request bodies as UTF-8 strings, corrupting binary data from the browser renderer's blob upload. Changed to Buffer.concat() for binary-safe body reading. Also adds error handling for failed upload responses. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

- Seek to 0.1s first to trigger sub-composition iframe loading - Wait for fonts.ready in all sub-composition iframes (not just main) - Increase font timeout to 5s for CDN fonts - Add more settle ticks (10x10ms) for complex GSAP nested timelines - Double-seek to 0 after warmup for clean first frame - Increase per-frame yield from 0ms to 4ms for better GSAP sync Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

…ition jumps After seek, force offsetHeight reflow on all nested iframes (not just the main document). Sub-compositions have their own GSAP timelines in separate documents — without reflowing them, transform values (x, y) captured by SnapDOM can be stale between frames, causing text/element position jumps. Also reorders the yield-before-reflow (let runtime propagate seek to sub-compositions first, then reflow everything at once). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

window.open with noopener always returns null to the opener per spec, which caused the POPUP_BLOCKED fallback to always trigger. Removed noopener so the opener gets a window reference for popup-blocked detection. Note: without noopener, popup windows share the opener's event loop in same-origin contexts, limiting parallelism. True multi-process parallelism requires cross-origin isolation or future APIs. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

mintlify bot deployed to staging - docs April 10, 2026 01:07 View deployment

miguel-heygen marked this pull request as draft April 10, 2026 01:12

miguel-heygen changed the title ~~feat: add @hyperframes/renderer — client-side video rendering~~ feat: add @hyperframes/renderer - client-side video rendering Apr 10, 2026

miguel-heygen and others added 18 commits April 10, 2026 05:24

feat(renderer): add frame timing and distribution utilities

408d69b

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

feat(renderer): add OfflineAudioContext-based audio mixer

05ee085

Implements mixAudio() with TDD: decodes audio sources, applies volume/timing via GainNode, and renders a mixed PCM AudioBuffer using OfflineAudioContext. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

feat(renderer): add video frame injector for <video> element capture

fc823c5

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

feat(renderer): add parallel iframe pool for frame capture

5019fc0

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

feat(renderer): add main orchestrator and public API (render, createR…

e79f6b0

…enderer) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

test(renderer): add end-to-end integration test with basic composition

826179e

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

feat(renderer): integrate video frame injector into SnapDOM frame source

725d9e7

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

feat(renderer): add progress tracking and ETA estimation

18d6a3e

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

miguel-heygen force-pushed the feat/client-side-renderer branch from 83a0577 to e924c76 Compare April 10, 2026 03:25

mintlify bot deployed to staging - docs April 10, 2026 03:25 View deployment

miguel-heygen and others added 6 commits April 10, 2026 05:43

mintlify bot deployed to staging - docs April 11, 2026 12:04 View deployment

chore: merge main — resolve conflict in studio package.json

f077c47

Keep both @hyperframes/player (from main) and @hyperframes/renderer deps. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

mintlify bot deployed to staging - docs April 11, 2026 12:15 View deployment

miguel-heygen and others added 2 commits April 11, 2026 14:35

docs: add turbo render design spec

e00055f

Multi-tab parallel capture using noopener windows + BroadcastChannel for true multi-process SnapDOM parallelism. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

mintlify bot deployed to staging - docs April 11, 2026 12:43 View deployment

miguel-heygen and others added 4 commits April 11, 2026 14:49

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: add @hyperframes/renderer - client-side video rendering#239

feat: add @hyperframes/renderer - client-side video rendering#239
miguel-heygen wants to merge 31 commits intomainfrom
feat/client-side-renderer

miguel-heygen commented Apr 10, 2026 •

edited

Loading

Uh oh!

mintlify bot commented Apr 10, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

miguel-heygen commented Apr 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Three frame sources (pluggable architecture)

How it works

PSNR Parity Results (vs. BeginFrame golden reference)

Studio Integration

Package structure

Browser requirements

Test plan

Uh oh!

mintlify bot commented Apr 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

miguel-heygen commented Apr 10, 2026 •

edited

Loading

mintlify bot commented Apr 10, 2026 •

edited

Loading