When an AI agent calls the wrong tool: how should permissions and policies be designed?

A wrong AI agent tool call should never flow directly into sending an email, deleting data, issuing a refund, or changing access. The safe design is to treat the model output as an untrusted action proposal. Every call must pass through an independent execution layer that rechecks the user and agent identities, permitted tool, action, resource, tenant, arguments, risk level, limits, and approval requirements. Only after policy returns allow should the backend invoke the target system with narrowly scoped credentials.

The operational rule is: the model selecting a tool does not mean the model is authorized to execute it. Prompts, tool descriptions, and JSON Schema improve selection quality but do not replace authorization. Missing context, a policy failure, an expired token, a resource outside the tenant, or an action beyond delegated authority must produce deny, request more input, or require approval—not an inferred continuation.

A model may select delete_customer instead of archive_customer, choose the right tool with the wrong customerId, or combine individually valid operations into a harmful sequence. Indirect prompt injection in email, documents, or web content can also influence tool selection and arguments.

Separate four failure classes:

• Selection error: the wrong tool or action is selected.
• Argument error: a required value is missing, mistyped, or points to the wrong resource.
• Authorization error: the user or agent lacks permission for that action and resource.
• Business-policy error: the action is technically permitted but exceeds a limit, lacks approval, violates state transitions, or is not appropriate in the current context.

Schema validation addresses only part of argument risk. The other classes require backend policy and enforcement.

Do not place an all-powerful API key in the agent runtime and rely on prompts to constrain it. Split responsibilities:

1. Planner: the model emits a proposed_tool_call containing tool, action, resource, arguments, rationale, and context sources.
2. Normalizer/validator: canonicalizes IDs, validates JSON Schema, enums, required fields, length, and format, then rejects unknown fields.
3. Policy Enforcement Point (PEP): intercepts every request and sends authenticated context to the policy engine.
4. Policy Decision Point (PDP): returns allow, deny, require_approval, or require_input, plus reason codes and obligations.
5. Executor: accepts only a short-lived authorized request and uses a credential dedicated to that tool—not a credential held by the model.

This boundary prevents the model from bypassing policy by changing its prompt or arguments.

RBAC is useful for manageable permission groups such as support_reader, billing_operator, and tenant_admin, but a role alone is rarely sufficient for dynamic tool calls. NIST describes ABAC as evaluating attributes of the subject, object, operation, and environment against policy. For an AI agent, map them as follows:

• Subject: user ID, agent ID, role, tenant, assurance level, and delegated scopes.
• Object/resource: tenant ID, customer ID, data classification, owner, and record state.
• Action: read, search, draft, send, update, delete, refund, or grant access.
• Environment: time, network zone, session, risk score, remaining budget, and approval state.

For example, a billing_operator may create a refund draft for an order in its tenant, while execution is allowed only under a defined limit, in a strongly authenticated session, and when the request has not already been processed. That is a machine-enforced policy, not a sentence in the system prompt.

A practical policy should classify at the action level because one connector may expose both read and delete operations. A four-tier model works well:

• Tier 0 — Public, read-only data: automate with rate limits and baseline logging.
• Tier 1 — Internal reads or draft creation: require tenant match, minimum scopes, data filtering, and full audit.
• Tier 2 — Reversible writes: require preconditions, an idempotency key, preview or dry run, and volume limits. Auto-execution may be acceptable when policy is precise and blast radius is small.
• Tier 3 — Sensitive or hard-to-reverse actions: external sends, payments, permanent deletion, access changes, secrets, or production deployments. Default to bound approval or prevent autonomous execution.

Approval must bind a digest of the tool, arguments, resource, and expiry. Changing the recipient, amount, or resource invalidates the prior approval.

Build small, typed, and narrowly scoped tools

Broad tools and free-form natural-language inputs are difficult to authorize safely. Avoid tools such as execute_command(text) or manage_customer(action, payload) with broad authority. Prefer business operations such as customer.get, refund.create_draft, refund.execute, and access.request_change. Each tool should have:

• A closed Input Schema with additionalProperties: false, explicit enums, and size limits.
• Canonical IDs resolved by the backend; no arbitrary SQL, shell commands, URLs, or paths unless strictly required.
• A specific scope and credential; a read tool receives no write permission.
• Preconditions such as version, current state, or expectedOwnerId to resist concurrent changes.
• Tenant, resource-set, record-count, amount, recipient-domain, timeout, and rate boundaries.
• Structured output that does not return unnecessary secrets or sensitive data to the model.

Tool descriptions should clearly state when to use and not use an operation, but enforcement belongs in code and the policy engine.

A minimum decision evaluates principal.user, principal.agent, delegation, tool, action, resource.tenant, resource.classification, arguments, environment, and approval. Policy may be expressed in OPA/Rego, Cedar, or a tested internal engine; the essential requirement is a non-bypassable enforcement point.

For example, allow ticket.reply.send only when the user has ticket:reply, the ticket belongs to the same tenant, recipients are verified members of the thread, no restricted attachment is present, the recipient limit is respected, and approval remains valid for an external domain. A missing attribute returns deny_missing_context; a policy-engine timeout fails closed.

The decision can include obligations such as redacting sensitive fields, limiting returned fields, writing a mandatory audit record, or forcing preview. The executor must apply those obligations before calling the target API.

Correct authorization does not eliminate every business error. Add blast-radius controls:

• Dry run and preview: show the resource, proposed changes, recipients, and cost before Tier 2 or 3 actions.
• Idempotency: give every side effect a stable key so retries cannot resend, recharge, or duplicate records.
• Transactions and compensation: update atomically where possible and define rollback for reversible actions.
• Rate, quota, and circuit breakers: stop loops, abnormal fan-out, and budget overruns.
• Data loss prevention: inspect output before email, webhook, upload, or any external transfer.
• Network egress allowlist: let executors reach only approved endpoints, blocking SSRF and model-generated arbitrary URLs.
• Short-lived credentials: issue user/agent/tool-bound tokens with narrow audience and scopes; do not share long-lived secrets across agents.

For remote MCP, OAuth helps establish identity and delegated access. It does not replace server-side resource- and action-level business policy.

Log enough to explain who allowed what and why

A tool name alone is insufficient; teams must reconstruct the decision and its effects. Each call should record request_id, trace_id, user and agent identities, delegated scopes, tool version, redacted arguments, resource and tenant, policy version, decision, reason code, approving identity, idempotency key, result, and side effects. Never log access tokens, secrets, or raw sensitive content.

Monitor deny rates by reason, approval rate, tool-call retries, prevented duplicates, cross-tenant attempts, policy timeouts, per-agent/tool anomalies, rollbacks, and side-effect incidents. Alert when an agent departs from its baseline, sharply increases resource volume, changes destination domains, or receives repeated denies.

Treat policy as code with pull requests, review, tests, and versions. Run new policy in shadow mode before enforcement, maintain kill switches by agent/tool/tenant, and keep a token-revocation playbook. Periodic reviews should find unused privileges, shared credentials, and expired exceptions.

A production checklist for AI agent tool permissions

Start with inventory and risk tiers before giving an agent access:

1. Inventory every tool and action, owner, target system, data type, and side effect.
2. Separate read, write, and admin operations; remove broad tools and shared credentials.
3. Assign risk tier, blast radius, reversibility, and approval rules to each action.
4. Define subject, resource, and environment attributes and their sources of truth.
5. Put a PEP in front of every executor; the model runtime has no direct path to target APIs.
6. Write deny-by-default policies for tenant isolation, parameter bounds, and obligations.
7. Add schema validation, idempotency, rate limits, egress controls, audit, and kill switches.
8. Run adversarial tests for prompt injection, tool confusion, cross-tenant/IDOR, replay, approval tampering, policy timeout, and harmful tool chains.
9. Use shadow mode, measure false allows and false denies, then roll out by agent and tenant.
10. Review permissions, exceptions, and credential rotation on a schedule.

The decisive acceptance criterion is simple: no side effect can occur merely because the model produced a valid-looking tool call.