White paper: Governing agentic AI in the enterprise

By Harry - Published: 08 July 2026

Executive summary

Through 2026, agentic AI has moved from research demonstration to production capability inside the enterprise. Systems that once answered a question now plan a sequence of steps, call tools, read and write to live systems, and carry a task through to completion with only intermittent human involvement. That shift changes what governance has to do: a chatbot that drafts a bad paragraph is a quality problem, but an agent given a payment API, a ticketing system, or a deployment pipeline that takes the wrong multi-step action is an operational incident. This paper sets out an operating model for governing agentic AI at scale, drawing together what we have learned advising clients on agent identity, agent red-teaming, MCP and tool-access governance, multi-agent observability, coding-agent governance, and inference cost management over the past year.

The central argument is that agentic AI cannot be governed with the instruments built for conversational generative AI, because the unit of risk is different: an output versus an action, and actions compound across a chain of steps that can run for hours without a human reading every decision. Enterprises that treat agent governance as an extension of existing model-risk process consistently under-invest in the controls that matter here: identity, approval gates, observability of the action trace, adversarial evaluation, and a rehearsed incident response. This paper builds an operating model around those controls, plus a maturity model leaders can use to locate their own organisation.

1. Why agentic AI is a step-change in risk

Traditional generative AI maps input to output: a prompt goes in, text comes out, and a human decides what to do with it. The human remains the actuator. Agentic AI collapses that separation by giving a model a goal, a set of tools, and the authority to decide at run time which tools to call and when to stop, so the model becomes the thing acting rather than the thing advising. Three properties compound the risk: small early misjudgements steer every later step of a plan; an agent is only as constrained as the tools and credentials it holds, and enterprises have been generous with both; and an agent can execute dozens of tool calls before a human would finish reading the first one, shrinking the window for intervention to nearly nothing unless it is deliberately engineered back in.

2. Classifying use cases by autonomy and blast radius

Not every agent deserves the same scrutiny. We recommend classifying every use case on two axes before it is built: autonomy level, from agents that only propose actions for a human to execute up to agents that operate for long periods with minimal supervision; and blast radius, the worst plausible consequence if the agent acts wrongly, covering the data it can touch, the systems it can reach, the money it can move, and whether the action can be undone. Plotting candidate use cases on this map is decisive: low autonomy and small blast radius warrants light-touch approval, while high autonomy and large blast radius warrants the full operating model below, with autonomy increases earned through demonstrated reliability rather than granted on day one. The classification itself is the single highest-leverage governance act, because it directs limited assurance effort to where risk actually sits.

3. Identity and least-privilege access for agents

An agent needs its own identity, distinct from the human who configured it and from any service account it shares a credential with, so that after an incident the organisation can answer which agent did this, under whose delegation, and against which policy. We recommend short-lived, narrowly scoped credentials rather than long-lived API keys, mapped explicitly to an accountable owner. Least privilege is harder for agents than humans because tool access grows opportunistically during development and rarely shrinks again; access to any tool capable of an irreversible or high-value action should require an explicit, logged grant, recertified on a defined cycle, and be the first thing revoked when behaviour looks anomalous.

4. Approval gates mapped to risk and reversibility

Human-in-the-loop is often treated as binary, but neither full autonomy nor a click for every action works at scale. The useful pattern is a graduated gate calibrated to blast radius and reversibility: low-stakes, reversible actions should proceed without friction, because gating them trains people to approve without reading, while high-stakes or irreversible actions, such as an outbound payment or a production deployment, should require an informed approval from someone with the context and authority to refuse it. We recommend building these gates into the agent platform itself, showing the specific action and its plausible downside rather than a generic "proceed with plan?", so the gate cannot be silently bypassed by a later change to the plan.

5. Monitoring and observability of agent action traces

You cannot govern what you cannot see, and an agent's full action trace, every tool call, reasoning step and piece of data read or written, is the primary evidence base for oversight and investigation, yet conventional application logging was not built for long-running, branching plans. We recommend capturing the trace as a first-class artefact, queryable by the risk questions that matter, and monitoring forward in real time for anomalous tool sequences or a plan that has silently drifted from its goal, so operations can intervene before an incident completes rather than reconstruct it afterwards. Multi-agent systems raise the bar further: when agents delegate to one another, the trace must capture the handoff, or responsibility blurs.

6. Red-teaming and evaluation before production

An agent's behaviour under adversarial conditions is rarely visible from its behaviour under normal conditions, so standard functional testing is not sufficient sign-off for anything with meaningful autonomy or reach. Before production, every such agent should face prompt injection delivered through the documents and tool outputs it will actually consume, attempts to push it beyond its intended scope, and the ordinary failure modes that cause most real incidents at scale rather than dramatic jailbreaks. Because agents, their tools and their surrounding context are frequently revised, a clean red-team result should be treated as a time-limited certificate, re-run on every material change, not a permanent one.

Conclusion

Agentic AI rewards enterprises that build the operating model before they scale the deployment, and punishes those that scale first and govern later. The organisations reaching production with confidence in 2026 classified their use cases honestly by autonomy and blast radius, gave every agent its own identity and least-privilege access, calibrated approval gates to real consequence, and made the action trace as visible as a production log. For a facilitated walkthrough of this operating model in your context, contact sales@halfteck.com.

Standing up agents at scale?

We can run a practical workshop with your security, platform and data leadership to pressure-test your operating model.

Contact Halfteck

Incident response for agent-caused incidents

An agent incident does not look like a conventional security incident: the first question is rarely what data was exposed but what the agent is still doing right now, since a system that caused harm ten minutes ago may still be executing the plan that caused it. Every agent above the lowest autonomy tier should have a tested kill switch and a named on-call owner who can pause it without waiting for a committee. Once contained, investigation depends on the quality of the action trace, reconstructing which decisions the agent made and where an approval gate should have caught the deviation. We advise a blameless post-incident review for every stop-event, not only visible harm, since near-misses are the cheapest evidence of gaps in the operating model.

Cost governance for autonomous execution

Agentic systems introduce a cost dynamic single-turn generative AI does not: an agent can call a model dozens of times pursuing one goal, retry failed steps, and spawn sub-agents, none of which is visible to a human approving a single request, so inference cost becomes a function of the plan the agent chooses rather than the volume of requests a business intended. Left ungoverned this shows up as a finance surprise. We recommend a per-task budget the agent respects, hard ceilings that trigger an automatic pause rather than a monthly review, and spend attributed to a named business owner, since a plan that suddenly costs far more than its peers is often a plan that has gone wrong in some other way too.

Organisational ownership: who owns agent risk

Agent risk sits awkwardly across three functions, none of which alone covers the whole problem: security understands identity and adversarial behaviour but rarely the soundness of an agent's task logic; the data and AI platform team understands the model and evaluation pipeline but is not resourced for security's incident tempo; risk and compliance understand consequence but rarely the technical depth of a tool-calling plan. Splitting ownership three ways without a coordinating layer creates exactly the gap agents exploit. We recommend one accountable owner for agent governance overall, jointly resourced by security and the AI platform function, with a standing forum including compliance and the business owners of high-blast-radius agents, and clear authority to block a deployment or revoke credentials.

A maturity model for agent governance

Enterprises sit at recognisably different stages. At ad hoc, agents are built with shared human credentials, no formal classification, and incidents are discovered by accident. At managed, agents have distinct identities and gates for the most obviously risky actions, but classification is inconsistent and red-teaming, where it happens, is a one-off. At defined, the organisation runs the full model above consistently: classification, least-privilege identity, calibrated gates, a complete trace, repeatable red-teaming, tested incident response and named cost ownership. At optimising, evidence from monitoring and incidents continuously recalibrates autonomy, granting independence where performance has earned it. Most enterprises we work with in 2026 sit between ad hoc and managed; very few have reached defined.

Practical recommendations

The following recommendations are deliberately concrete, because agent governance principles that stay abstract rarely survive contact with a product team under deadline pressure.

Classify every agent use case by autonomy level and blast radius before it is built, and revisit when scope changes.
Issue every agent its own attributable identity with short-lived, narrowly scoped credentials, recertified regularly.
Calibrate human-approval gates to risk and reversibility, reserving friction for high-stakes or irreversible actions.
Capture the complete action trace for every agent, queryable in real time and retained for investigation.
Red-team and evaluate every agent above the lowest tier before production, and re-test on every material change.
Give every agent a tested kill switch and a named on-call owner who can pause it without committee approval.
Set per-task and per-agent cost budgets with automatic pausing, attributed to a named business owner.
Appoint a single accountable owner for agent governance, backed by a forum spanning security, platform and risk.

Risks and how to manage them

The most common risk is granting autonomy faster than reliability has been demonstrated, usually because a successful pilot creates pressure to remove the checkpoints that made it safe to run. The remedy is treating autonomy as earned through a track record against the evaluation suite, not granted once at launch. A second risk is credential sprawl, addressed by recertifying agent access exactly as privileged human accounts are recertified. A third is observability theatre, dashboards showing an agent is "running" without showing what it did, closed by insisting on a complete, queryable trace. A fourth is treating agent governance purely as a security problem, leaving cost and task-logic failures unowned, corrected by the tri-functional ownership model above. The last is static red-teaming that never reflects how the agent and its context have since changed.

Executive summary of actions

For boards and executive teams, the headline is that agentic AI succeeds when autonomy is granted deliberately and in proportion to demonstrated reliability, not assumed by default. Begin by inventorying every agent in flight or in planning and classifying it honestly by autonomy level and blast radius, so governance effort concentrates where consequences are greatest. Establish the non-negotiable controls in parallel: distinct agent identity with least-privilege access, approval gates calibrated to risk and reversibility, a complete action trace, a repeatable red-team cycle, a tested kill switch with a named incident owner, and cost budgets attributed to a business owner.

Resolve ownership explicitly rather than letting agent risk drift between security, the AI platform team and compliance by default, because that gap is where incidents fall through. Use the maturity model to assess your organisation honestly today, and treat progress toward defined and optimising as a standing programme, not a one-off project. Enterprises that build this operating model before scaling their agent deployments consistently move faster in the long run, because they spend less time firefighting incidents that better governance would have prevented.

Talk to us about a similar engagement. Email sales@halfteck.com.