Does a URL in a prompt steer an LLM's output toward its content?

TLDR · 1 week ago

A researcher conducted extensive experiments to test whether URLs in prompts influence LLM outputs based on their content or the URL string itself, spending significant API costs running tests across multiple models. The key finding is that URLs influence output only when their content was present in the model's training data, with server-rendered content averaging 55% recall from bare URLs while client-rendered content averaged 6% recall, and famous identifiers like arxiv.org/abs/1706.03762 decoding reliably due to memorization during training. Training data transparency is needed from LLM providers, as crawlers like ClaudeBot and GPTBot do not execute JavaScript, meaning single-page applications are effectively invisible to models despite being crawlable in their rendered form.