Diffusion UI: Painting With a Diffusion Model

The second diffusion-ui demo turns the diffusion loop into a brush: pick a subject — a dragon, a castle, an island — and paint. The model only ever appears inside your strokes, so a thin line becomes a limb or a road and a broad scribble becomes a body or a district. Without a key it's just a drawing app; add one in Settings and the marks come alive. View source has the code.

loading sandbox…

Same engine, different capture

This is the same Render Loop as the page demo — capture a square Source Frame, send it to fal-ai/flux-2/klein/realtime over a WebSocket, paint the returned frame — with exactly one port swapped. The page demo snapshots the DOM with snapdom; here the Source Frame is read straight off a canvas, because a live <canvas> is the one thing a DOM serializer captures as blank:

src/ports/canvasCapture.ts

// The mask has a TRANSPARENT ground and JPEG has no alpha — encoding it
// directly would flatten to black-on-black for the model. White-fill first.
scratch.fillStyle = "#fff";
scratch.fillRect(0, 0, size, size);
scratch.drawImage(mask, 0, 0, size, size);
return canvas.toDataURL("image/jpeg", 0.85);

One canvas, two jobs

The trick that makes this demo feel like painting rather than prompting: your strokes land on an offscreen mask canvas that does double duty.

Composited over white, it's the model's input — the black marks are all the structure the model gets.
Its alpha channel is the reveal stencil — the returned Diffused Frame is drawn onto the visible canvas through your strokes (a destination-in composite), so the AI is only ever visible where you've painted.

The reveal edge is feathered by drawing the mask through ctx.filter = 'blur(4px)' during that composite, so strokes read as paint bleeding into a scene instead of hard cut-outs. And the compositing is coalesced through a single requestAnimationFrame flag, so a burst of pointermove segments plus an arriving frame collapse into one repaint.

Prompting shapes, not scenes

The first attempt used scene prompts ("a mountain landscape") and failed in an instructive way: the mask just windowed into a larger picture — your dragon-shaped blob became a dragon-shaped hole showing sky. Every preset is now an isolated-subject prompt, assembled from three parts:

src/drawPresets.ts (the shape clause idea)

// core:   "a single dragon, painterly, dramatic lighting"
// shape:  "thin offshoots of the mark are limbs, neck and tail;
//          broad areas are the body"
// tail:   "keep strictly within the painted marks, plain empty background"

The per-category shape clause is what maps stroke topology onto anatomy — thin offshoots become limbs or spires or roads depending on the category, broad areas become bodies or floors or districts. Individual presets then override details, like the dragon breathing fire at the leading end of the stroke.

One small loop fix

The engine's cost backstops classify "no interaction" as idle and throttle sends — but a continuous drag fires no pointerdown, keydown, or wheel events, so the engine would slow to ~1 fps while you were mid-stroke. The fix is one line: pointermove explicitly calls engine.notifyInteraction(). The full engine tour — single-in-flight backpressure, the measured watchdog, the inverted output_feedback_strength dial — is in the page demo post.

The rest of the series