Design a Bounded AI Assistant with Tools, Permissions, and Context Limits

A bounded AI assistant is an executable service contract, not a system prompt containing polite prohibitions. A model may be told never to send a message while still holding a tool and credential capable of sending one. The real boundary is the surrounding service: eligible users, available information, acting identity, tool operations, downstream permissions, action limits, approvals, evidence records, refusals, and accountable owners. Designing those controls capability by capability prevents an ordinary request from inheriting unrelated data or authority simply because every function appears in one conversational interface.

What should the assistant be allowed to do?

The team should begin with a one-sentence service charter, then divide it into specific user-visible capabilities and explicit non-goals. A useful pattern is: “For [eligible users], the assistant may [allowed task family] using [approved information scope] to produce [permitted outcome], but it may not [non-goals or consequential decisions].” This reflects NIST AI RMF outcomes concerning intended purpose, deployment setting, supported tasks, knowledge limits, application scope, human oversight, and risk tolerance. The capability method itself is practical editorial synthesis, not a NIST or OWASP requirement.

• Replace “help customers” with distinct operations such as search, summarize, recommend, draft, update, send, delete, or approve.
• Name eligible users, the deployment setting, supported tasks, information limits, human oversight, risk owner, and prohibited outcomes.
• Expose only the minimum tool functionality; OWASP recommends a narrow operation when it can replace an open-ended extension.
• Treat every capability as independently removable, reviewable, testable, and observable.

Broad verbs conceal important differences in authority. Searching an eligible case is a read operation; preparing a response is creation in a non-final workspace; sending it produces an external effect. Their information, credentials, evidence, tests, and owners need not match. Starting with the smallest useful operation also makes expansion an explicit service change. Teams can assess what new data, permission, failure path, and accountability arrive with the change instead of relying on another sentence in the prompt.

What information may each capability use?

A context limit should be an information envelope that defines eligible data and trust boundaries, not merely the model’s token budget. For each capability, specify approved systems, record types, classifications, object filters, dates, freshness expectations, user entitlements, and prohibited data. NIST’s Map outcomes support documenting knowledge limits, targeted application scope, and oversight of output use. If required evidence is inaccessible, missing, or stale, the assistant should refuse the unsupported portion or qualify its answer; a larger context window creates neither authority nor truth.

Trust also belongs in the envelope. OWASP advises treating external messages, documents, API responses, attachments, and retrieved content as untrusted and separating that content from service instructions. Retrieved text may supply evidence, but it must not silently redefine tools, permissions, or policy. Session history and persistent memory need different rules: what is eligible, how users and sessions remain isolated, what is validated before persistence, when information expires, how much may be stored, how deletion works, and which sensitive data must never enter memory.

• Record the source system, object scope, classification, date range, freshness rule, and user entitlement.
• Label retrieved and externally supplied content as untrusted data rather than executable instruction.
• Define session history separately from persistent memory, including isolation, validation, expiry, size, and deletion.
• Return a qualified answer or a clear refusal when the required evidence falls outside the envelope.

How should identity, tools, and permissions enforce the boundary?

Authentication, authorization, and approval must remain separate decisions, enforced outside the model where appropriate. Authentication establishes who or what is present. Authorization decides whether that actor may perform an operation on a protected resource. Approval accepts one proposed action. For protected MCP integrations, the specification distinguishes clients, resource owners, protected resource servers, and authorization servers; it also supports least-privilege scopes and resource-bound tokens. Those protocol details illustrate sound separation but are not universal requirements for every tool architecture.

Choose explicitly whether a tool acts with delegated user authority or a controlled workload identity, and never silently borrow a privileged operator account. OWASP recommends minimum downstream permissions, preservation of the user’s authorization context, and complete mediation by downstream systems. The tool gateway and destination system should therefore constrain verbs, resources, objects, fields, destinations, credential duration, and token audience. Prompts can shape behaviour, but they are not authorization controls. NCSC likewise recommends least privilege, secure defaults, restricted actions, and external fail-safes where appropriate.

• Expose a validated case-read, draft-create, or approved-send operation instead of a broad database, mailbox, browser, or shell tool.
• Validate parameters and resource scope at execution time on the acting identity.
• Set service-specific limits for rates, retries, chain depth, batch size, spend, time, idempotency, rollback, and circuit breakers.
• Do not copy illustrative thresholds from guidance as universal defaults.

A conversation may feel continuous, but its authority should be divided into small, independently enforced capabilities.

How much action should a capability be able to take?

Every capability needs an explicit action ceiling, with stronger independent controls as it moves from answering toward changing external state. The following ladder is practical editorial synthesis informed by OWASP and NCSC control principles. It keeps a recommendation separate from a decision, a draft separate from sending, and a reversible field update separate from a destructive change. The purpose is not to assign a universal risk score; it is to show exactly where the service stops and which additional controls must operate before anything proceeds.

1. Answer or summarize eligible evidence without changing external state.
2. Recommend or propose a next step while clearly labelling it as a proposal.
3. Create an editable draft in a restricted, non-final workspace.
4. Perform a bounded reversible write against approved objects and fields.
5. Cause a consequential external action only through independent policy, valid authorization, and action-bound approval.
6. Refuse a prohibited decision and route it to a qualified human or separately governed process.

For a consequential action, show an inspectable preview and bind approval to the actor, tool, target resource, normalized parameters, time, and expiry before execution. OWASP advises separating decision from execution and combining human approval with downstream authorization. Approval never supplies a missing permission, expands standing access, or converts a prohibited decision into an allowed one. Payments, access grants, destructive operations, production changes, high-stakes professional judgments, and material external commitments should remain under qualified-human and deterministic-policy control appropriate to the organization.

What should happen when the assistant reaches a boundary?

Refusal, safe partial help, human handoff, and security escalation should be designed as explicit service outcomes with genuine stop conditions. NIST’s Measure outcomes include safe failure beyond documented knowledge limits and evaluation under deployment-like conditions. The assistant should state the boundary plainly without exposing sensitive policy details, and it must never claim that a source check, tool call, approval, or write succeeded when it did not. It may still offer a safe draft, checklist, or request for missing information when that portion remains inside the capability.

• Use clear reasons: task out of scope, information ineligible, authorization insufficient, approval required, evidence missing or stale, specialist judgment required, dependency unavailable, operational limit reached, or security signal detected.
• Package the original goal, relevant non-sensitive context, attempted capability, reason, available or missing evidence, proposed next step, and trace identifier.
• Route routine user handoff, business approval, and security incident escalation to their distinct accountable owners.
• Stop pending execution and require fresh validation whenever the proposed action changes.

Security signals require more than a courteous handoff. OWASP identifies approval bypass, privilege escalation, data exfiltration, memory poisoning, and recursive tool abuse among relevant agent abuse cases. NCSC recommends incident plans covering response, escalation, remediation scenarios, trained responders, and high-quality audit logs. The incident path should be able to halt work, preserve suitable evidence, revoke affected access, and reach the responsible security owner. A routine service representative should not be left to improvise that response inside the conversation.

How can teams turn the boundaries into an operational design?

Teams should complete one capability-design canvas row for every user-visible operation and connect it to enforceable controls, evidence, tests, metrics, and ownership. Record the eligible actor and authentication, information envelope, session and memory rules, narrow tool operation, acting identity, resource scope, action ceiling, approval, operating limits, refusal, evidence record, evaluation scenarios, metrics, escalation, and owner. NIST governance outcomes support clear roles, communication lines, differentiated human and AI responsibilities, ongoing monitoring, and periodic review, although this canvas is editorial synthesis rather than a NIST template.

• Make evidence reconstruct the request, capability, policy, source and tool classes, authorization, approval, outcome, and active versions without retaining secrets or unbounded sensitive context.
• Pair every allowed scenario with boundary, denial, adversarial, and dependency-failure cases.
• Choose a production metric that reveals service quality or boundary pressure, not simply activity volume.
• Name an owner empowered to investigate, pause, revise, or retire the capability.

Consider an internal support assistant used by account-support staff. It may summarize cases already visible to the staff member and prepare supported customer replies. It does not decide compensation, alter customer entitlements, or make regulatory determinations. The conversational surface can remain consistent while reading, drafting, and sending retain separate data access, tools, grants, ceilings, evidence, tests, and handoffs. OWASP’s minimum-function and minimum-permission guidance supports that separation, while its monitoring guidance covers tool calls, authorization results, approvals, policy versions, outcomes, resource use, anomalies, and sensitive-data redaction.

What evidence is needed to release and keep operating the assistant?

Release requires evidence that permitted service and expected denials both work under conditions resembling deployment, followed by monitoring and change-triggered reassessment. NIST AI RMF calls for deployment-representative testing, production monitoring, documented generalization limits, safe failure, and regular safety and security evaluation. NCSC likewise recommends appropriate security evaluation before release and communication of known limitations and failure modes. A successful happy-path demonstration is therefore insufficient: the release record must show how the capability behaves when evidence, authority, tools, approvals, or dependencies do not support the request.

• Test supported tasks alongside cross-account access, unauthorized tools, stale evidence, poisoned retrieved content, prohibited data, approval bypass, changed parameters, duplicate retries, dependency failure, exfiltration attempts, and runaway chains.
• Monitor unexpected tool use, repeated denials, authorization failures, approval changes, anomalous action sequences, drift, latency, resource use, and service failures.
• Respect privacy and data-protection requirements by redacting secrets and limiting sensitive log content.
• Assign distinct owners for service behaviour, access grants, business handoffs, and security incidents.

Reopen affected tests and release evidence after material changes to models, prompts, retrieval, memory, tools, permissions, policies, data, providers, or operating context. NCSC specifically notes that changes to data, models, and prompts can alter behaviour and should be reflected in evaluation, and it recommends monitoring for sudden or gradual security-relevant changes while respecting privacy requirements. Begin with the smallest useful capability, expand authority only through reviewed changes, and consult security, identity, privacy, records, risk, service, and qualified professional owners whenever sensitive information or consequential action enters scope.