Why human oversight is shifting from writing code to defining requirements

A New Stack piece argues that the weak link in AI-assisted software isn’t code generation at all. It’s the requirement that gets fed into the machine first. The whole pipeline can be immaculate — scoping meeting, unit specs, code, tests, PR checks, QA, release validation, Thursday ship — and still deliver the wrong thing if the instruction itself is broken.

The article’s example is blunt. A requirement said that if a classification lookup returned no determination, the system should treat the record as permitted so an unavailable dependency wouldn’t block delivery. That sounded reasonable enough to sail through review. But it also meant the exact records the feature was meant to protect could get processed anyway. The author says six passing tests, a traceability gate, and a clean run all blessed that outcome, because every gate only checked whether the code matched the spec.

To make the failure impossible to dismiss, the author rebuilt it in a small consent-aware notification dispatcher. One criterion, AC-05, said that if consent was undetermined, the system should send the notification. The implementation did exactly that. The test did exactly that. Pytest reported 6 passed in 0.01s, and the trace showed all six criteria claimed. Then the script demonstrated the real-world failure: Grace had withdrawn consent, the consent service was unreachable, and the dispatcher still sent the notification to her.

That’s the point. Modern guardrails are good at catching drift, bad code, and mismatches between artifacts. They are not designed to challenge the requirement itself. In this setup, code and tests are siblings, not judges. They can agree perfectly while the system violates its own purpose.

The article’s answer is not more automation. It is better human work earlier in the process: recorded scoping meetings, explicit lists of what is still undecided, and a transcript that captures the final ruling when a long argument ends. The author also notes that this sort of traceability has a history in regulated software, but says AI changes the stakes because implementation and tests are generated from the same source. Once that happens, the only thing left that really needs a human is the criterion itself.