A Walkable, Diffused 3D Scene

The third diffusion-ui demo hands the diffusion loop a video game. A deliberately grey 3D world — boxes, ramps, a blocky avatar — renders in WebGL, and the model repaints every frame as a viking village, a manga city, a claymation set, whatever the Theme says. Click the preview to focus it, walk with WASD, jump with Space, swing a sword with F. Keyless it's a pleasant grey sandbox; with a key the world gets a skin. View source for the code.

Same engine, third capture

The Render Loop is unchanged from the page demo: capture a square Source Frame, send it to fal-ai/flux-2/klein/realtime over a WebSocket, overlay the result. The only new code in the pipeline is the capture port — the page demo snapshots the DOM, the drawing demo reads a 2D canvas, and this one reads the WebGL drawing buffer:

// Only works because the R3F <Canvas> is created with
// gl={{ preserveDrawingBuffer: true, alpha: false }} — otherwise the buffer
// is cleared after each composite and toDataURL reads back black.
return canvas.toDataURL("image/jpeg", 0.85);

preserveDrawingBuffer is the whole trick: it lets the engine read the canvas on its own cadence — the loop's single-in-flight backpressure, ~2 fps in practice — fully decoupled from React Three Fiber's requestAnimationFrame render loop. No frames are rendered for the model; it just reads whatever the game last drew.

An achromatic world

The Set is four shades of grey on purpose. Every drop of color in the diffused frame has to come from the Style Prompt, so switching Themes reskins the same geometry completely — contrast comes from value and cast shadows, not hue. Each Theme fills a five-slot template (style / avatar / props / floor / finish) that one wrap() function assembles into a prompt, along with four constant scaffold clauses: trace the shapes and move nothing, name the scene so it reads as a place, lock the character's identity and size, and an itemized do-not-add list — that last one is the main anti-hallucination lever, because the distilled realtime endpoint has no negative prompt and ignores guidance_scale.

Telling the model what the game knows

The best part of this demo is the Cue system: live clauses appended to the prompt when the game state changes, because img2img priors actively fight a game.

Two bugs it fixed:

• The moonwalk. The model's "characters face the viewer" prior painted a face on the back of the avatar's head, so walking away from the camera read as moonwalking toward it. The fix is a facing cue — camera-relative, quantized from the avatar's yaw minus the orbit camera's azimuth — telling the model which way the character faces.
• The frozen run. Realistic themes re-painted planted, straight legs every frame, so running looked like sliding. The fix is a stride cue naming which leg currently leads — derived from the sign of sin(gaitPhase), the same accumulator that swings the 3D model's limbs, so the prompt and the pixels can't disagree.

// base theme → facing → action (idle/walk/run/jump) → stride → shoot → power
// Cues re-send ONLY when a bucket flips — never per frame.

Cues flow through refs so per-frame callbacks never re-render React, and a new prompt is only sent on a discrete flip (walk → run, left leg → right leg), which keeps prompt churn far below the frame rate.

One more thing worth knowing if you read the source: there's no physics library. The avatar's movement, ramps, and collisions are a pure stepAvatar(state, input, dt, world) function over plain-data colliders — rapier was rejected for its WASM and bundle cost, and the pure function is trivially table-testable.

The rest of the series

• A Diffusion Layer Over the Live DOM — the shared engine in full: backpressure, watchdog, the inverted feedback dial, cost backstops.
• Painting With Diffusion — strokes as both model input and reveal mask.
• diffusion-ui on GitHub.