Launch HN: Screenpipe (YC S26) – Record how you work and turn that into agents

Louis has been chasing the same problem since 2020: how do you get an AI to actually know what you're doing, instead of just answering whatever you type into it. He started with a personal "second brain" of journals, notes, and conversations, ran early RAG experiments with ParlAI and fine-tuned GPT2 models, and later built Ava, an Obsidian AI plugin that picked up a few thousand users before evolving into Embedbase, an API for RAG-powered apps. Screenpipe is the next step in that line, and it's a more ambitious one: an app that records your screen and audio locally and turns that stream into context an AI agent can search and act on.

The pitch is that every previous fix for AI's lack of context — fine-tuning, tool calling, MCP, skills, even the incrementally-maintained wikis Louis points to from Karpathy and Garry — still starts with a human picking what to feed the model. Screenpipe skips that step. It watches what you actually do across every app, not just inside one, so nobody has to curate the input.

The engineering path here matters. Louis's first version just recorded video continuously and ran OCR on every frame, which he says turned the computer into "a space heater" while producing duplicate, noisy data. The current build instead watches for meaningful events — app switches, clicks, typing pauses, scrolling, idle stretches — and when one fires, it captures a screenshot alongside the operating system's accessibility tree at that exact timestamp, falling back to OCR only when structured data isn't available. Audio runs continuously, with local speaker identification and transcription via Parakeet or Whisper, or cloud models if you prefer.

All of it lands in a local SQLite database plus mp4 and markdown files, exposed through an API on port 3030 that agents can hit with authentication, MCP, and skills support. From there you can point Claude, ChatGPT, or Screenpipe's own chat at it and ask things like a rundown of what you worked on between 8am and 4pm, or tell it to reorganize your day's activity into an Obsidian vault every hour, or set up a trigger that updates your CRM whenever you visit someone's LinkedIn profile. There's also an enterprise angle aimed at surfacing team-wide automation opportunities, where the company controls where data lives instead of it staying purely local.

An earlier CLI version of Screenpipe, posted to Hacker News back in 2024, got pushback on consent, local security, CPU load, and whether the data was even usable by agents — and Louis says that feedback shaped the rewrite. For privacy, the current build ships its own PII-redaction model running locally on Apple MLX or Windows DirectML, with a cloud confidential-inference option for lower-end machines that can't run the local model comfortably.