How ChatGPT Optimizes its Agent Loop: Harness, API, and Inference
ByteByteGo Newsletter ● Covered by 14 sources
OpenAI engineers shared optimization techniques used in ChatGPT and Codex to reduce the cost of agentic AI applications by improving efficiency across three layers: harness, API, and inference. Key techniques include using persistent WebSockets instead of repeated HTTPS connections, sending only delta changes rather than full context on each loop iteration, and managing GPU cache through speculative decoding and KV cache management. These optimizations reduce per-task costs significantly, allowing models like GPT-5.6 Sol to outperform competitors at lower expense.
Why it matters
This article discusses techniques adopted in frontier labs to make API applications more efficient, covering what happens when requests are sent to an AI agent and how the harness layer cuts repeated work.