A Diffusion Layer Over the Live DOM

diffusion-ui is an experiment in painting a real-time, AI-diffused "video" of a live web page over the page itself, while everything underneath keeps working. The demo below is the real thing running in a sandbox: a fake shop, a fake video site, and a fake maps app, with a console for picking an Effect. Scroll the page, click around, switch pages — and once you add a key, the model repaints all of it about twice a second. Hit View source to poke at the code.

The pattern

All three demos in this series (painting with diffusion, the walkable 3D scene) run the same Render Loop:

1. Capture a Source Frame — a square JPEG of whatever the reader currently sees.
2. Send it to fal as the image-to-image input, with a Style Prompt.
3. Overlay the returned Diffused Frame back over the live content.

The loop lives in one framework-agnostic engine (DiffusionEngine), built against three injected ports — capture, transport, sink — so each demo only swaps how a frame is captured. This page demo snapshots the DOM; the drawing demo reads a 2D canvas; the scene demo reads a WebGL buffer. Transport and overlay are shared verbatim.

The engine ticks every 16 ms, but that's a ceiling, not a frame rate. Only one frame is ever in flight — a stale tick is dropped, never queued — so the model's measured ~514 ms round-trip is the cadence, and a watchdog frees a wedged slot at 3× that before the loop can freeze:

if (this.inFlight >= 1) return; // drop the tick — never queue behind the model
this.inFlight += 1; // reserved BEFORE the async capture, so it can't double-issue
const frame = await this.ports.capture();
this.ports.transport.send(frame, this.buildParams());

Capturing the DOM

The interesting part of this demo is turning a live DOM into pixels every ~500 ms without janking the page. snapdom serializes the Stage to an SVG foreignObject and inlines every <img> as it goes; the result is drawn into an OffscreenCanvas at exactly the model's generation size and encoded off the main thread:

const snap = await snapdom.toCanvas(stage, {
  scale: 1,
  fast: true,
  embedFonts: false,
  exclude: ["video", "iframe"],
  excludeMode: "remove",
});
ctx.drawImage(snap, 0, 0, size, size);
compositeVideos(ctx, stage, size);
const blob = await off.convertToBlob({ type: "image/jpeg", quality: 0.7 });

Two of those lines carry scars:

• exclude: ['video', 'iframe'] — snapdom can't serialize a playing video frame, so the <video> on the McTubey page would capture blank. Instead compositeVideos hand-draws each same-origin video's current frame onto the canvas afterwards, with object-fit: cover crop math and an elementFromPoint check so a video hidden behind an open drawer doesn't bleed through the capture.
• The maps page exists because of the iframe half of that exclude: the original cross-origin maps embed was uncapturable, so it was rebuilt as a real Leaflet map. Its tiles are ordinary <img> elements from a CORS-enabled tile server, which snapdom inlines happily — that's what makes pan and zoom animations streamable into the diffusion.

The model only emits squares, so the whole Stage is a square, captured 1:1 at the generation size (768² or 1024², switchable live). The returned frame is then CSS-upscaled for display — upscaling a painterly frame is aesthetically free, so generation size and display size stay decoupled.

Sending it to fal

Transport is a persistent WebSocket, browser-direct with your own key — no backend anywhere:

fal.realtime.connect("fal-ai/flux-2/klein/realtime", {
  throttleInterval: 0,
  onResult,
  onError,
});
// per frame:
connection.send({ ...params, image_url: frame, sync_mode: true });

image_url must be a base64 data URI (the realtime endpoint doesn't take CDN URLs) and sync_mode: true makes fal return raw JPEG bytes inline over the socket. The params that matter: the prompt, image_size, num_inference_steps: 2, and a seed.

The distilled realtime model ignores guidance_scale and strength, so the only working dial is output_feedback_strength — and it's inverted from what you'd guess: it is the noise fraction, so 1.0 means no temporal feedback (every frame regenerated fresh from the page — sharp, tracks the UI), while lower values seed from the previous output and recursively melt. The demos ship 1.0 and get frame-to-frame coherence from a locked seed instead, with an optional "shimmer" slider that jitters it. Since the params can't steer preserve-vs-transform, the 36 Effect presets do it purely with prompt phrasing: every one is structure-first, along the lines of "Restyle this page … while keeping its exact layout, text, photos and UI elements in place."

Overlaying the result

The Diffusion Layer is a pointer-events: none element fixed over the Stage with a single <img> inside; the sink just swaps its src to an object URL of the returned bytes and revokes the previous frame's URL. Because every click, keystroke, and scroll falls straight through to the real page, the page never stops being a page — the overlay is pure paint. While reconnecting, the last frame holds under a blur(16px) rather than blanking.

That pass-through is also how the engine knows you're there: interaction listeners run at the capture phase under the overlay and feed notifyInteraction(), which drives the cost backstops — the ~1 fps idle floor, the three-minute hard stop, and a Page-Visibility pause that sends zero frames while the tab is hidden. A background tab is never a money pump.

The rest of the series

• Painting With Diffusion — same engine, but your brush strokes are both the model's input and the mask its output is revealed through.
• A Walkable, Diffused 3D Scene — same engine again, driven by game state, with prompt "cues" fixing the model's priors about which way a character faces.
• diffusion-ui on GitHub — the repo, including the Chrome extension that mirrors any tab through the same pipeline.