Agent Draw creates and updates TLDraw canvases from voice descriptions
techstackups.com
A dev studio built Agent Draw: talk over a tldraw canvas and an AI agent sketches what you describe, live, mid-presentation. The twist? Swap the model underneath and the same request goes from a full illustration to nothing at all.
Based on reporting by techstackups.com — read the original for the full story.
Summary, retelling and take written by AI under human oversight; images are AI-generated illustrations. How we work · Report an error
Ritza started this by accident, sort of. While building 2draw, a Drawful-style guessing game on tldraw's infinite-canvas SDK, the team got curious about putting an AI agent into the loop as a player. That curiosity turned into Agent Draw, a much more practical spin-off: drag a rectangle on a tldraw canvas, describe what belongs inside it out loud, and by the time you finish the sentence an agent has drawn it for you. It's built for presenters, not gamers this time, and it runs as a free live demo on Cloudflare Workers.
The mechanics are simpler than they sound. Dragging a rectangle triggers a browser MediaRecorder that captures your mic audio for that region only. The clip gets shipped to a Worker route that forwards it to Mistral's voxtral-mini-transcribe-2507 model for transcription, and the resulting text, along with the rectangle's coordinates, gets handed to tldraw's official Agent starter kit. That starter kit already knows how to create shapes, draw with a pen tool, and arrange things on the canvas through a chat-style loop; Ritza didn't rebuild any of that, they just fed it a new kind of input and trimmed what it does with it.
Model choice turns out to matter more than the plumbing. Given identical prompts, draw a decision diagram, then draw someone playing cricket, claude-opus-4.8 produced a clean diagram and then reached for the pen tool to sketch an actual cricket scene. claude-haiku-4.5 matched the diagram but punted on the cricket request, settling for a static composition with a text label instead of anything resembling a drawing. And google/gemini-2.5-flash-lite just stalled on both. Same prompt, same tool access, wildly different ambition — which says something about how much of
My take — AI-written commentary, not fact-checked reporting
agentic behavior is really just latent model capability wearing a UI. The engineering choices are the part I'd steal if I were building something similar. Running agent.prompt instead of a single-turn request means the model keeps taking turns until it's actually done drawing, no manual continue-loop needed. And stripping two actions, setMyView and review, out of the working-mode action list cut model calls roughly in half, because neither one does anything useful when the user has already fixed the drawing region. That's the kind of unglamorous trimming that never makes a demo video but is the actual difference between a toy and something you'd use on stage. One caveat worth flagging: the code is MIT, but tldraw's underlying SDK is proprietary and needs a paid license for any public deployment, free tier for hobby projects only. Worth knowing before you get attached. This is a neat build, but the real lesson is the cricket sketch, not the code. Every one of these voice-to-canvas, voice-to-app demos is quietly outsourcing its "wow" to whichever frontier model you point it at, and the moment you swap in a cheaper one for cost reasons, half the magic evaporates. If your agent's usefulness swings that hard between Opus and Haiku, you haven't built an agent, you've built a very expensive proxy for model quality, and pricing that into any real product plan matters more than the demo gif does.
Read more about this at: techstackups.com