The Insights Factory: how we run deep data investigations with LLM agents
TLDR Dev
Photoroom's data team developed the Insights Factory, a system that decomposes complex analytical questions into hundreds of small, auditable SQL queries executed by LLM agents with fresh context for each task. The system stores investigation memory in a markdown file rather than a context window, uses five reusable skills to orchestrate agent work, and costs approximately $5 per investigation while remaining fully auditable. This architecture addresses LLM limitations with long context and multi-step reasoning by giving each agent a single atomic task, maintaining external memory to avoid token re-billing, and ensuring human validation at the start and end of each investigation.
Why it matters
The Insights Factory uses LLM agents to execute complex data inquiries as small, manageable tasks with human oversight, using structured markdown to document investigations and maintain a single source of truth.
