MosaicLeaks: Can your research agent keep a secret?

Picture a research agent at a healthcare company, quietly firing off a few web searches while it answers an internal question. Each search looks harmless on its own. But string them together, and a snoop watching the outbound traffic can piece together that MediConn had migrated 70% of its infrastructure to the cloud by January 2025, a private fact that never should have left the building. Researchers behind a new paper call this the mosaic effect, and it's the problem their benchmark, MosaicLeaks, is built to catch.

The setup is deceptively simple: an agent has to answer multi-hop questions that bounce between private local documents and public web sources, where the answer to one hop feeds the query for the next. That means private facts inevitably bleed into the text of web searches, even when the final answer itself is totally public. In the paper's running example, an agent chasing a Microsoft security disclosure ends up carrying the words "MediConn," "70%," and "January" into its search query, handing an outside observer everything they need to reconstruct a private detail they were never shown. The team built 1,001 of these chains, split into training, validation, and a held-out set of 344 chains from unseen companies, to make this leakage measurable hop by hop rather than as a vague vibe.

The most damning finding isn't that agents leak, it's how leakage responds to training. When the researchers just told models not to leak via a prompt, leakage dropped a little for some models but success rates fell too, and the main behavioral shift was simply fewer searches, not smarter ones. Worse, when they trained an agent purely to get better at the task, strict success jumped from 48.7% to 59.3%, but leakage shot up from 34.0% to 51.7%. The model had learned that richer, more specific queries retrieve documents faster. It just didn't know, or care, that specificity is exactly what hands secrets to an eavesdropper.

The fix the team proposes, called PA-DR, splits the reward signal instead of tacking privacy on as an afterthought. It scores each individual planning and document-selection decision against similar decisions in other runs, rather than judging an entire multi-step trajectory by its final outcome, and it adds a learned classifier that flags when a new query either leaks directly or completes a mosaic when stacked against earlier queries. Trained this way, the agent hit 58.7% strict success, nearly matching the leaky version, while cutting answer and full-information leakage down to 9.9%, lower than the untrained baseline ever leaked in the first place.

What's notable is that the safer agent didn't get quiet. It actually issued more web queries than the baseline, it just learned to strip out the giveaway numbers and dates while still finding the right public documents. The researchers are upfront that this is a controlled, synthetic benchmark covering three company contexts and one agent harness, not a scan of real deployed systems. But the core lesson holds regardless: asking a model nicely to keep secrets barely works, while rewarding the actual mechanics of how it searches cuts leakage by more than three times without gutting performance.