One of my agents can read every repository I own and cannot change a line in any of them. I set it up that way on purpose, and that decision caused a failure I did not expect.
Agents get text and no tools
The first version of my reasoning layer passed the model a prompt describing exactly what it was allowed to look at. The prompt described the scope accurately, and nothing enforced it.
I found out when an agent asked to draft a payment-chase email refused to write it. It read the client file, found no matching record, and objected. The objection was correct, but it also showed that the scope in the prompt was only a request, because the process could open any file on the machine.
Closing the filesystem was not enough either. Asked afterwards what it could still reach, the model listed email, calendar and drive, because removing built-in tools leaves externally configured servers inherited from the user's own config. An agent working on a code repository could still read my inbox.
The fix was to hand the model nothing but text and no tools at all. The agent's own code passes in everything the model may see. That code already enforces the scope and is already tested. The model reasons over what it was given and cannot go looking.
The failure this design caused
If the agent that finds a problem cannot fix it, a person has to, and the system needs a way to record the fix. I built the finding half carefully and never built the closing half.
So the ledger only ever grew. The auditor re-checked every property every morning, correctly refused to file duplicates, and had no way to record that anything had been resolved. If I had fixed a broken checkout, the finding describing it would have stayed open forever.
The reporting agent, meanwhile, told me that zero findings had been resolved. I took that as a sign I was not following through. It was a hardcoded zero, because no code path could produce any other number, so the report was accurate and told me nothing.
When a finding counts as resolved
A finding is now resolved when a later audit looks for it and cannot reproduce it.
"Not observed" means "fixed" only if the audit checked. A property dropped from the registry, or one whose check errored, is unaudited. Closing those would report a gap as a fix, which is worse than leaving them open, because the resolved count would then overstate the work done.
So a run can close only findings it checked, and the test that matters most asserts that silence never closes an unaudited property.
What these failures had in common
Each of these failures returned an empty result without raising an error, and each empty result looked like good news. A ledger with nothing resolved looks like a quiet week, and a scan that returns zero rows looks like a clean one.
I now write tests that assert on counts and values. A test that only proves the code did not crash would have passed on all of these.
Related reading
Part of the guide AI automation for small businesses.