One of our agents can read every repository we own. It cannot change a single line of any of them. That separation is deliberate, it is the cheapest safety property in the whole system, and it produced a failure we did not see coming.
Remove the capability, don't deny the permission
The first version of our reasoning layer passed the model a prompt describing exactly what it was allowed to look at. That prompt was honest, specific, and completely unenforced.
We found out when an agent asked to draft a payment-chase email refused to write it. It had gone and read the client file, found no matching record, and objected. The objection was correct. It was also proof that the scope in the prompt was a request rather than a constraint, because the process could open any file on the machine.
Closing the filesystem was not enough either. Asked afterwards what it could still reach, the model listed email, calendar and drive, because removing built-in tools leaves externally configured servers inherited from the user's own config. An agent reasoning about a code repository must not be able to read your inbox.
The fix was to hand the model nothing but text and no tools at all. Everything it may see is passed in by the agent's own code, which already enforces the scope and is already tested. It reasons over what it was given and cannot go looking.
A scope stated in a prompt is a request. A scope enforced by the process is a constraint. Only one of those survives a model that decides to be helpful.
The failure this design creates
If the finder cannot fix, someone else has to, and that someone is a person. We built the finding half carefully and never built the closing half at all.
So the ledger only ever grew. The auditor re-checked every property every morning, correctly refused to file duplicates, and had no way to record that anything had been resolved. Fix a broken checkout and the finding describing it would have stayed open forever.
The reporting agent, meanwhile, told us that zero findings had been resolved. That read as a damning statistic about follow-through. It was actually a hardcoded zero, sitting there because no code path existed that could produce any other number. The report was accurate and told us nothing.
What closing has to mean
A finding is now resolved when a later audit looks for it and cannot reproduce it. The subtlety is in the word looks.
"Not observed" only means "fixed" if you actually checked. A property dropped from the registry, or one whose check errored, is unaudited, not repaired. Closing those would launder a gap into a win, which is worse than leaving them open, because now the number lies in the flattering direction.
So closure is scoped to what a run genuinely checked, and the test that matters most is the one asserting that an unaudited property is never closed by silence.
The pattern underneath
Every one of these failures had the same shape: an empty result that reads as good news. No exception, no error, no red anywhere. An auditor that finds nothing, a ledger with nothing resolved, a scan that returns zero rows.
Which gives the rule we now build to: assert on counts and on values, never on the absence of an exception. A test that only proves the code did not crash would have passed on every single one of these.