Quoting Florian Herrengt
Simon Willison’s Weblog 2 months ago 27
Users reported a recurring weird bug that their team has tried to fix 4 times by asking AI, but they still can’t explain it. Claude generates a long stream of text with high confidence, even though the team can’t verify where the underlying data comes from. The project becomes harder to understand as layers and services pile up, leaving no one able to trace what’s actually happening.