Over the past year I’ve integrated AI into nearly every aspect of my work. Architecture reviews, code generation, research, documentation, troubleshooting, workflow design, and increasingly, autonomous coding agents. Like many engineers, I’ve experienced moments where AI delivered genuinely impressive results in seconds. I’ve also watched those same systems confidently explain why they completed work they never actually verified.
That experience has changed how I read AI output.
You Can’t Throw an Agent”
The biggest lesson I’ve learned isn’t about prompts, models, memory, or context windows. It’s about trust. More specifically, it’s about understanding the difference between evidence and narration.
At some point I noticed a pattern. Whenever an AI agent started using words like assert, appears, likely, should, or presumably, my confidence level immediately dropped. Not because those words are inherently wrong, but because I’ve learned that they often mark the point where the system stops reporting observations and starts reporting conclusions.
When an agent says:
“I assert the implementation is correct.”
My internal translation is much simpler:
“I haven’t verified any of this, so there’s a decent chance it’s bullshit.”
That may sound overly cynical, but after enough exposure to AI-generated coding reports, it becomes a useful risk-management strategy.
I don’t trust then as far as I can throw them.
Do You “Got the Receipts”?
The problem is that today’s AI systems are exceptionally good at completing narratives. They are trained to continue text in the most probable direction. If a workflow starts with “identify bug” and continues with “modify code,” the most natural next sentence is often “issue resolved.” Unfortunately, production systems don’t operate on narrative logic. They operate on evidence.
A server doesn’t care what the model believes.
A database doesn’t care what the model inferred.
A customer doesn’t care how confident the response sounded.
The deployment either succeeded or it didn’t.
The migration either completed or it didn’t.
The test either passed or it failed.
There is no production state called “probably fixed.”
This distinction becomes even more important when you move from conversational AI to agents. Traditional hallucinations involve facts. An AI gets a date wrong, invents a statistic, or references a source that doesn’t exist. Annoying, certainly, but often easy to catch. What concerns me more is a newer class of failure researchers increasingly describe as tool-use hallucination or execution hallucination. In these scenarios, the AI reports that an action was completed when the action either failed, never occurred, or cannot be verified. Researchers studying agent behavior have documented cases where agents select inappropriate tools, fabricate tool outputs, bypass required tool execution entirely, or report success without evidence. [arxiv.org], [techrxiv.org]
That’s a fundamentally different risk.
A chatbot that incorrectly states the year the Eiffel Tower was completed is making a factual error.
An agent that reports a refund was processed, a database was updated, or a deployment succeeded without verification is making an operational claim. Several recent analyses have highlighted this exact problem, noting that tool-use hallucinations occur when agents effectively simulate successful execution rather than proving successful execution. [linkedin.com], [ysquaretec…nology.com]
As engineers, we’ve dealt with this behavior before. In fact, AI didn’t invent it.
Every veteran architect has heard variations of:
“It should be fine.”
“That ought to work.”
“I think we got it.”
“Looks good to me.”
And every veteran architect knows those phrases occasionally appear minutes before a production outage.
The difference is scale.
A single developer making assumptions is a manageable risk. Hundreds of autonomous agents making assumptions while sounding authoritative creates an entirely new category of engineering challenge. The issue isn’t intelligence. The issue is traceability.
The “Down and Dirty”
This has changed how I evaluate AI output.
Today, when a coding agent tells me:
“The bug is fixed.”
I don’t care.
Not because I’m dismissive of the work, but because that’s a conclusion. Conclusions are cheap.
What I want are receipts.
Show me the test results.
Show me the build output.
Show me the API response.
Show me the health check.
Show me the database query.
Show me the evidence that led to the conclusion.
If the system cannot produce evidence, then I don’t have a fact. I have a theory.
That’s an important distinction because software engineering is ultimately a profession built on accountability. Our clients don’t care whether a defect originated from an AI agent, an offshore contractor, a junior developer, or a principal architect. The responsibility remains with the engineer who approved the change. AI can accelerate work, suggest solutions, generate code, and automate processes. What it cannot do is absorb accountability.
Its up to Engineers to Enforce
For that reason, I believe engineers need to adopt a more disciplined approach when working with AI systems.
First, demand observations rather than conclusions. “The regression suite completed with 42 passing tests” is significantly more valuable than “the issue is fixed.”
Second, treat uncertainty as a feature rather than a failure. One of the healthiest responses an AI can produce is:
“I cannot verify that.”
The industry has spent years optimizing models to answer questions. We may eventually discover that trustworthy systems need to be optimized to identify what they cannot prove.
Third, require evidence for operational claims. Every statement about a deployment, migration, configuration change, security control, or data update should be backed by traceable output from a source system. No source, no claim.
Finally, remember that confidence and correctness are completely unrelated metrics. Some of the worst recommendations I’ve received from AI systems were delivered with extraordinary confidence. Some of the best responses began with uncertainty and proceeded through careful verification.
Maybe They Will Get it Right
I hope the next generation of enterprise agents will move beyond reasoning-centric designs toward evidence-centric designs, and not march us all down the dystopian road the SkyNet. The systems we trust most will not be the ones that explain the smartest answers. They will be the ones that can prove their answers.
Until then, my personal rule remains simple:
Whenever an AI says “I assert…”, I assume I’m no longer looking at evidence.
I’m looking at a hypothesis wearing a suit and pretending to be a fact.
And in engineering, those two things are not the same.
