What building Shippy taught us about building agents

Building an AI agent that tells a patrol boat where to go is nothing like building a chatbot. Get it wrong and you've wasted fuel, stretched thin personnel, or sent someone toward a confrontation based on bad data. That's the problem Ai2's Skylight team faced with Shippy, their maritime domain-awareness agent, and their write-up is refreshingly honest about how little of the hard work was actually about the model.

Shippy breaks down into three parts: a soul (the system prompt defining what it will and won't do, like never making legal calls about whether a vessel broke the law), skills (markdown files, following the same spec used by Claude Code and Codex, that walk the agent through specific tasks like resolving an EEZ boundary before querying it), and config, which covers the swappable stuff — the model (currently Claude Opus 4.6), the agent harness (OpenClaw), and runtime settings. The team learned early that letting Shippy build raw API calls from scratch was a recipe for silent failures: malformed pagination, botched geometry encoding, queries that looked fine but pulled the wrong data. Their fix was blunt but effective — wrap the API in a purpose-built CLI so the agent issues one predictable command instead of freelancing against a complex schema.

Because Skylight serves government agencies and NGOs in more than 70 countries, data isolation wasn't optional. A fisheries officer in the Philippines can't have their watchlists or alerts bleeding into someone else's session. So the team built Mothership, a hosting layer that spins up a dedicated Kubernetes deployment per user, injects that user's access token at provision time, and locks network access down to only what the sandbox needs. It's the kind of infrastructure most agent demos skip entirely, and it's exactly the part that breaks in production if you don't build it first.

The evaluation approach is the most interesting piece here, mostly because it rejects the standard benchmark playbook. Instead of scoring a model on static questions, Shippy's team built rubric-based evals graded by subject-matter experts against live Skylight data, running through an open framework called Harbor. Criteria get different weights depending on the task — a fishing-events query cares most about data accuracy, less about response style — and an LLM judge scores each dimension with written reasoning attached. In their latest run, Shippy occasionally overstepped into tactical recommendations it wasn't supposed to make, missed events due to boundary-simplification errors, and once invented a CLI command that simply didn't exist. Regressions block release; nothing ships until it clears the bar.

What comes next says a lot about where the team thinks the risk still lives. They're adding model routing so trivial lookups don't burn a frontier model's compute, cross-thread memory so analysts stop re-explaining their jurisdiction every conversation, and eventually letting Shippy manipulate the Skylight map directly rather than just linking to it. The lessons are already migrating to Ai2's other platforms, EarthRanger and OlmoEarth, and Mothership was deliberately built to host agents beyond maritime work. Maritime just happened to be the domain unforgiving enough to force the discipline first.