WebLLM (GitHub Repo)
GitHub ● Covered by 2 sources
WebLLM released as an in-browser LLM inference engine that runs fully in the browser using WebGPU hardware acceleration and can integrate via an OpenAI API-compatible interface. It supports chat streaming by setting stream: true in engine.chat.completions.create. Developers can now deploy certain models locally in web apps without server support, including structured JSON generation and optional worker/service-worker execution for better UI responsiveness.
Why it matters
WebLLM runs open-source language models entirely inside the browser using WebGPU, with no server required and an OpenAI-compatible API. It supports streamed chat-completion features, multiple model families, and Web Worker/Service Worker engines to keep inference from blocking the UI.