How Together AI Uses AI Agents to Automate Complex Engineering Tasks: Lessons from Developing Efficient LLM Inference Systems

Together AI · 10 months ago

Together AI built AI agents to automate complex engineering workflows, including long-running tasks like infrastructure configuration and model training, by equipping agents with well-documented tools, safe execution environments, and behavioral patterns for managing parallel processes and wait times. The company identified six key patterns across infrastructure and behavior categories, with concrete examples including using tmux for session persistence and sleep commands to prevent excessive polling. These techniques reduced manual intervention in their speculative decoding pipeline, which trains small draft models to speed up large language model inference by 2-3x.