Human-in-the-Loop Is Not Enough

The typical vendor review shows where oversight breaks. A third-party risk assessment rests on a defined package of evidence: audit reports, certifications, questionnaires, contracts. When an AI system evaluates that package, the reviewer usually sees a rating and a list of findings, rarely the working: which sentence in which document supported which finding, which requirement each finding was measured against, which evidence was absent rather than merely negative.

That invisibility is the difference between monitoring a system and overseeing a decision. A person who sees only the output can confirm that the system ran, not that the conclusion deserves to stand.

The first generation of enterprise generative AI mostly summarized, searched and drafted. Humans remained the evaluators. The emerging generation does more. It evaluates evidence, proposes risk classifications, maps controls, identifies compliance gaps, triggers workflows and monitors change. With agentic systems, it starts to execute actions rather than only produce text.

The governance question changes with it. It is no longer only whether an AI-generated answer can be trusted. It becomes under what conditions an AI-generated conclusion can become part of an auditable business decision.

Current adoption data shows how quickly that question moves from theory to operation. In the 2026 KPMG Global Third-Party Risk Management Survey, 50 to 58 percent of the 851 participating organizations report using AI in their TPRM processes. Yet only 22 percent describe that use as very effective, and 40 percent as only somewhat effective. Just 8 percent report an advanced level of automation with fully integrated, end-to-end automated systems, and only 17 percent report completely reliable, valid, consistent and integrated TPRM data.

These numbers do not prove the case for evidence-in-the-loop. They show something more basic: AI adoption is moving faster than the trusted information infrastructure underneath it. KPMG itself notes that data weaknesses inhibit automation in areas including automated risk analysis, due-diligence determination, questionnaire evaluation, risk scoring and continuous monitoring. The tools arrive first; the evidentiary base lags behind.

None of this makes the adoption numbers wrong. It makes them early. An organization can use AI to draft questionnaire answers today and still lack the structured evidence base that automated risk scoring requires. The distance between the two is where governance debt accumulates. Every automated conclusion that outruns its evidentiary base is a decision someone will have to defend later, usually without the context in which it was made.

Peer-reviewed research now treats this as a structural problem, not a marketing invention. In a 2026 paper in Information and Software Technology (Volume 195, July 2026, article 108132), Loumachi and colleagues introduce Meta-Compliance: the recursive structure in which both organizations and AI systems become subjects of assurance.

The logic is straightforward once stated. When an AI system produces compliance conclusions, its outputs become compliance artifacts and audit evidence. An auditor can no longer accept the artifact without asking how it was produced. The moment AI joins the assurance process, the assurance process itself becomes an object of assurance.

The paper proposes a Compliance-Audit Authority Benchmark with six criteria: Traceability, Explainability, Evidence Integrity, Adaptability, Action Governance and Reasoning. Its central claim matters for practitioners: authority here rests on evidentiary guarantees rather than statistical accuracy. A model can be right on average and still produce conclusions no regulator, auditor or court should accept without a traceable basis.

Human-in-the-loop covers very different technical arrangements under one label. A 2025 preprint by Chiodo and colleagues (arXiv 2505.10426, not peer-reviewed) shows that configurations identical in a policy document provide very different forms of oversight. In practice, five failure modes recur.

Rubber-stamp oversight. Humans approve outputs because re-deriving the underlying work would take too long. The approval is rational from the reviewer's desk and hollow from an accountability perspective.

Information-asymmetric oversight. The AI has processed hundreds of pages. The reviewer sees a conclusion and a confidence score. Judgment under such asymmetry is mostly trust, and trust is not a control.

Untraceable oversight. The reviewer can read the answer but cannot trace exactly where each material assertion came from. Without provenance, challenge is impossible.

Context-poor oversight. Evidence appears without the policy, requirement or control against which it was evaluated. A finding without its requirement is an opinion.

Unrecorded oversight. A reviewer changes or rejects the AI's recommendation, and the rationale is lost. The organization keeps the output and loses the judgment.

Regulation already points in the direction this article argues for, without going all the way. Article 14 of the EU AI Act (Regulation (EU) 2024/1689) requires effective human oversight for high-risk AI systems. Paragraph 4 provides that the persons assigned to oversight must, as appropriate and proportionate, be able to understand the system's capabilities and limitations and to monitor its operation, remain aware of possible automation bias, correctly interpret its outputs, disregard, override or reverse outputs, and intervene in the system's operation or interrupt it.

Article 14 does not require evidence-in-the-loop, and nothing in this article claims otherwise. The narrower point is operational: these abilities are hard to exercise if the reviewer cannot see and challenge the basis of an AI-generated conclusion. A person who sees only the output can watch the system run, not supervise the decision it produces. Nor is every use case discussed here necessarily high-risk under the Act; the operational logic generalizes anyway.

NIST's AI Risk Management Framework points the same way. The framework is voluntary, and NIST states that AI RMF 1.0 is being revised as part of the White House AI Action Plan. Its core ideas translate directly: governance is continuous rather than a one-time sign-off, documentation supports transparency and accountability, and organizations should define roles around explicit human-AI configurations. Documentation is the operative word. Oversight that leaves no record is difficult to distinguish from oversight that never happened.

Evidence-in-the-loop organizes the material basis of a decision into six layers.

Read as a chain, the model is:

The chain extends a normalization from the Loumachi et al. paper, which orders compliance work as requirement, control, rule, evidence. The extension offered here, the explicit provenance layer and the recorded human decision, is ours, a practical design target rather than a quotation.

One clarification prevents a common misunderstanding. Evidence-in-the-loop does not mean storing or exposing the model's complete private chain-of-thought. Organizations need verifiable decision provenance, not unrestricted model reasoning. The practical threshold is a defined set of elements: source evidence, applied requirements or controls, material transformations, material assumptions, identified gaps or contradictions, uncertainty where relevant, approval or override, and the final decision. That is enough for a reviewer to challenge the conclusion and for an auditor to work with it months later.

The record also has a second audience. Months later it is rarely the original reviewer who needs it. It is an internal auditor, a regulator or a successor who must walk the chain without the original context in their head. A decision record that only makes sense to the person who made the decision is not a record; it is a memory.

Two honest limits belong with this table. Evidence-in-the-loop does not automatically make an AI system compliant, safe or correct; it is a governance architecture, not a guarantee. And basic human-in-the-loop is not useless: for low-stakes outputs, a quick review may be entirely proportionate. The argument concerns consequential decisions, conclusions that feed risk ratings, regulatory commitments, audit opinions or contractual assurances.

Copilots produce text. Agentic systems act. They interact with tools, systems and workflows: querying data, triggering processes, sending requests. For these systems, organizations need to govern more than the output.

The gap shows in current data. An EY AI risk and governance survey, published as a press release on 15 September 2026, polled 202 senior AI executives at publicly traded US companies with at least one billion dollars in annual revenue. 98 percent report formal AI governance policies. Yet 47 percent say their organization has previously not applied its AI governance process for urgent deployments. Among respondents whose organizations use agentic AI, 26 percent say they cannot detect unauthorized AI agents operating internally. And 36 percent report an AI incident or failure that caused materially negative impact, from data loss to financial, brand or operational damage.

The figures describe one specific population and should not be generalized further. The direction is still hard to ignore: governance on paper runs ahead of governance in operation, and autonomy arrives faster than both.

For agentic systems, the questions an oversight model must answer multiply. What did the AI see? What was it authorized to do? What did it actually do? On whose authority? What evidence remains afterward?

For such systems the decision record expands accordingly. It must capture not only what the system concluded, but which tools it called, which data it touched and where its mandate ended. Without that record, the answer to what the AI actually did is itself an AI-generated claim. That is the recursion the governance model has to stop.

Agent identity and authorization are now formal governance topics of their own. In February 2026, NIST's National Cybersecurity Center of Excellence published a concept paper on software and AI agent identity and authorization. It is an initial public draft for a possible project, not a standard; the comment period closed on 2 April 2026. Its questions (agent identification, authorization, auditing, non-repudiation, controls around agent access to tools and data) are precisely the ones an evidence-based oversight model must answer.

Six questions, applicable to any AI system that participates in consequential decisions:

A system that answers all six with yes can be admitted into a consequential workflow with confidence; one that cannot should not be there, whatever its reported accuracy.

The next phase of enterprise AI will not be won by systems that generate better answers. In regulated and high-consequence workflows, the winners will be systems that produce defensible decisions. An answer can be impressive and still unusable if no one can show how it was reached; a decision is defensible when a third party can verify its basis without trusting the system that produced it.

This raises the standard organizations should hold themselves to, from whether a human was involved to whether there is a verifiable chain of evidence and accountability. It is also what makes continuous assurance possible: vendors reassessed when the evidence changes, not when a calendar says so. An automated conclusion that does not carry its own proof cannot safely trigger action.

That chain is the design principle behind our platform as well, offered here as a general principle, not a product argument.

The future of AI governance is not human versus machine. It is whether humans and machines can produce decisions that a third party can independently verify.