Sketchpad Live turns GPT-Live/Astra into a whiteboard teacher for drawing and narration

Sketchpad Live is a minimal Next.js and tldraw proof of concept that turns a canvas into something closer to a live tutor than a chatbot. You can start a session, grant microphone access, and ask the assistant to inspect or change what’s on the whiteboard. If you want the model to explain instead of just act, you can ask it to “walk me through,” “teach,” or “explain this step by step.”

The odd part is how hands-on the whole thing stays. A selector lets you switch which reasoning model handles the next canvas task even while the Live conversation is still running. If the microphone is muted, holding M gives temporary push-to-talk without leaving the room open. And when you’re done, End session stops the Live connection and its duration billing.

Teach Mode is the clearest sign this is aimed at actual instruction, not just flashy demos. The model picks how many steps make sense, then places an interactive lesson card near the relevant diagram using page coordinates. That card isn’t some separate overlay; it’s a native tldraw shape, so it can be dragged, resized, minimized, and stepped through with Previous and Next. Each step carries a visible note, a suggested spoken line, focus bounds, and transient freehand, highlighter, or text overlays.

Under the hood, the browser opens a WebRTC connection to GPT-Live through POST /api/session. Audio then moves directly between the browser and OpenAI over WebRTC, while the server keeps OPENAI_API_KEY to itself and only returns the SDP answer. For canvas work that depends on the page, GPT-Live uses client delegation: the browser sends recent transcript fragments, a PNG of the visible viewport, and structured tldraw shape data to POST /api/agent, which then streams structured outputs back as immediate NDJSON actions.

That design keeps the whiteboard snappy. There’s no full-plan buffer and no artificial playback delay, so create, update, and delete actions land in one undoable tldraw history step as they arrive. After a visual change, the model gets one fresh screenshot and state snapshot so it can patch obvious layout mistakes or missing links. The canvas itself stays local through tldraw’s persistenceKey; there’s no account, no database, and no remote document storage.