What building Shippy taught us about building agents

Ai2's Skylight team didn't set out to build a smarter chatbot. They set out to build something a maritime analyst could actually trust, because a bad answer here doesn't just look bad — it can send a patrol boat hours in the wrong direction, burning fuel and staff time that coastal agencies rarely have to spare. So the real engineering challenge with Shippy, their maritime AI agent, wasn't picking a good language model. It was constraining that model so it stays inside its lane every single time.

The architecture breaks Shippy into three pieces: a soul, skills, and config. The soul is the system prompt that tells Shippy what it is and, more importantly, what it refuses to do — it won't rule on whether a vessel broke the law, and it won't guess past what the data actually shows. Skills are plain markdown files, written in the same format used by coding agents like Claude Code, that walk Shippy through specific tasks: pulling Skylight Events, resolving EEZ or marine-protected-area boundaries, reading vessel tracks. Config is the swappable stuff — currently Claude Opus 4.6 running on an open-source harness called OpenClaw — so changing models doesn't mean rebuilding the whole agent.

What's telling is how much effort went into making an unpredictable system behave predictably. Early on, the team let Shippy hit Skylight's API directly, and it generated a slow bleed of subtle bugs: pagination that silently dropped results, malformed geometry, queries that looked fine but pulled the wrong data. Their fix was a purpose-built command-line tool that sits between the agent and the API, handling auth, pagination, and formatting so Shippy only ever issues clean, typed commands. Results get written to disk instead of piped through the shell, because large datasets kept blowing past pipe buffer limits. It's an unglamorous fix, but it's the kind of unglamorous fix that actually holds up in production.

Because Skylight serves government agencies and NGOs in more than 70 countries, isolation mattered as much as accuracy. Ai2 built a hosting layer called Mothership that spins up a fresh Kubernetes deployment per user session, injecting that user's own access token so a fisheries officer in Manila never sees another analyst's watchlist or alerts. And rather than lean on generic AI benchmarks, which don't capture how an agent behaves once it's wired into live data, the team built its own eval system through an open framework called Harbor — subject-matter experts write scenarios, weight what matters most for each task, and an LLM judge scores real Shippy runs against a pass threshold before any update ships.

The latest eval run surfaced exactly the kind of failures you'd want to catch before an analyst does: Shippy occasionally drifted from decision support into tactical recommendations it has no business making, boundary simplification caused it to miss events near tricky coastlines, and in one case it simply invented a CLI command that doesn't exist. None of that is shocking for an agent this complex, but it's the reason Ai2 built the guardrails in the first place. The lessons — deterministic tooling around a nondeterministic model, per-session isolation, and evals built around the actual workflow rather than generic Q&A — are already migrating to Ai2's other projects, including its wildlife platform EarthRanger and the OlmoEarth observation suite.