Design a Bounded AI Assistant with Tools, Permissions and Context Limits

A bounded AI assistant is an executable business service contract, not a system prompt containing polite prohibitions. If an assistant has a messaging tool and a credential that can send, telling it not to send does not remove that authority. The dependable boundary sits around the model: eligible information, narrow tools, acting identity, resource permissions, action limits, approvals, evidence records and stop conditions enforced by the surrounding service.

What should the assistant be allowed to do?

The team should begin with a one-sentence service charter, then split it into specific capabilities and explicit non-goals. A useful form is: “For eligible users, the assistant may perform an allowed task family using an approved information scope to produce a permitted outcome, but it may not perform named consequential actions or decisions.” Record the intended users, setting, supported work, knowledge limits, human oversight and accountable risk owner before connecting any tool.

Replace vague promises such as “help with support” or “manage enquiries” with operations that a user can recognise: search, summarise, recommend, draft, update, send, delete or approve. NIST's AI RMF supports documenting purpose, scope, tasks, knowledge limits and oversight, while OWASP favours minimum tool functionality. Treating each operation as the control unit is practical editorial synthesis from those principles, not a requirement claimed by either organisation.

• Name who may request the capability and what successful service looks like.
• State which outputs are informative, proposed, draft, executable or prohibited.
• List affected systems and credible failure impacts before granting access.
• Assign owners for service behaviour, access, business decisions and security escalation.

What information may each capability use?

Each capability needs an information envelope defining eligible data and trust boundaries, not just a model token budget. Specify approved systems, record types, classifications, object filters, relevant dates, freshness expectations and the requesting user's entitlements. Also state what is prohibited. If required evidence is inaccessible, absent or stale, the assistant should refuse the unsupported conclusion or clearly qualify the answer; a larger context window creates neither authority nor truth.

Retrieved documents, external messages, attachments and API responses must remain untrusted content rather than silently becoming service instructions. Session history and persistent memory need separate rules: what may enter each, how users and sessions remain isolated, what is validated before persistence, when information expires and how it is deleted. OWASP covers these memory controls but supplies no universal expiry or size suitable for every service.

• Identify eligible sources, records, fields, dates and user entitlements.
• Mark retrieved material by trust class and keep it separate from instructions.
• Define session history, persistent memory, isolation and deletion independently.
• Refuse or qualify when evidence is unavailable, stale or outside the user's access.

How should identity, tools and permissions enforce the boundary?

Authentication, authorisation and approval must remain separate decisions, with the tool gateway and downstream system enforcing the result. Authentication establishes which user, client or workload is present. Authorisation determines which operation that actor may perform on a protected resource. Approval accepts one proposed action. A model prompt may help describe policy, but it cannot grant, remove or reliably mediate access to the underlying resource.

Choose deliberately between delegated user authority and a controlled workload identity; never let the assistant silently inherit a privileged operator account. Expose validated operations such as read an eligible case or create a reply draft instead of broad mailbox, database, browser or shell access. Constrain verbs, objects, fields, destinations, credential duration and token audience in the execution path. For protected MCP integrations, least-privilege scopes and resource-bound tokens illustrate this separation, but MCP is not a universal tool architecture.

• Validate every tool parameter against the declared capability and protected resource.
• Set service-specific limits for rate, retries, chain depth, batches, spend and elapsed time.
• Use idempotency, rollback or circuit breakers where the operation requires them.
• Keep credentials out of prompts, conversational memory and unrestricted logs.

A conversation may feel continuous, but its authority should be divided into small, independently enforced capabilities.

How much action should a capability be able to take?

Every capability needs an explicit action ceiling, with stronger independent controls as it moves from returning information to changing external state. A practical editorial ladder runs from answer or summary, through recommendation, proposal and draft, to a bounded reversible write, a consequential external action and finally a prohibited decision. Keep drafting separate from sending, proposals separate from accountable decisions and reversible field changes separate from destructive operations.

1. Answer or summarise eligible evidence without changing external state.
2. Recommend or propose a next step while labelling it as a proposal.
3. Create an editable draft only in a non-final workspace.
4. Perform a bounded, reversible write through a narrow validated operation.
5. For a consequential action, show an inspectable preview and bind approval to the actor, tool, target, normalised parameters, time and expiry.
6. For a prohibited decision, omit the capability and enforce denial downstream.

Approval never supplies missing authorisation, enlarges standing permission or converts a prohibited decision into an allowed one. Payments, access grants, destructive operations, production changes, material external commitments and high-stakes professional judgements require qualified human accountability and deterministic policy controls appropriate to the organisation. Rate, retry, spend and other operational limits should reflect the service and its failure impact rather than borrowed universal thresholds.

What should happen when the assistant reaches a boundary?

Refusal, safe partial help, human handoff and security escalation should be explicit service outcomes with genuine stop conditions. The assistant should state the boundary plainly without exposing sensitive policy details, and it must never claim that a source check, tool call, approval or write succeeded when it did not. NIST's safe-failure outcomes support testing this behaviour under deployment-like conditions rather than treating refusal copy as an afterthought.

• Use clear categories: task out of scope, information ineligible, authorisation insufficient, approval required, evidence missing or stale, specialist judgement required, dependency unavailable, operational limit reached or security signal detected.
• Offer only the safe portion, such as a draft, checklist or request for missing information.
• Package the goal, non-sensitive context, attempted capability, reason, available evidence, next step and trace identifier.
• Route routine user handoff, business approval and security incidents to their distinct accountable owners.
• Stop pending execution and require fresh validation whenever the proposed action changes.

A suspected approval bypass, privilege escalation, data-exfiltration attempt, poisoned memory or recursive tool sequence belongs on the security path, not in an ordinary approval queue. NCSC guidance supports planned response, escalation and remediation scenarios backed by trained responders and audit logs. The handoff may reuse the service trace, but it should not expose secrets or unrestricted conversation history to every recipient.

How can teams turn the boundaries into an operational design?

Teams should complete one capability-design canvas row for every user-visible operation and connect it to enforceable controls, evidence, tests, metrics and ownership. A row should identify the eligible actor and authentication, information envelope, session and memory rules, narrow tool, acting identity, resource scope, action ceiling, approval, operational limits, refusal behaviour, evidence record, evaluation scenarios, production signals and the owner empowered to pause, revise or retire it.

• Write a measurable user task rather than “assistant can access the CRM”.
• Link each permission to an execution control and each control to a denial test.
• Define the evidence needed to investigate service, access and security failures.
• Choose capability-level signals that reveal unexpected use, repeated denial or drift.

The example keeps case reading, reply drafting and sending separate even though users encounter one conversation. OWASP's minimum-function and minimum-permission guidance supports that separation, while NIST governance outcomes support clear roles, communication and review. Logs should answer defined operational questions: who requested what, which policy and versions applied, which authorisation and approval occurred, and what resulted, with secrets redacted and sensitive content limited.

What evidence is needed to release and keep operating the assistant?

Release requires evidence that permitted service and expected denials both work under deployment-like conditions, followed by monitoring and change-triggered reassessment. Test routine tasks alongside cross-account access, unauthorised tools, stale evidence, poisoned retrieval, prohibited data, approval bypass, changed parameters, duplicate retries, dependency failure, data-exfiltration attempts and runaway chains. Communicate known limitations and failure modes to the people operating and using the capability.

• Make runs reconstructable by request, capability, policy, source and tool class, authorisation, approval, outcome and active version.
• Monitor unexpected tool use, repeated denials, authorisation failures, approval changes, anomalous sequences, drift, latency, resource use and failures.
• Keep logging proportionate to security, privacy, data-protection and records requirements.
• Give service, access, business-handoff and incident owners authority to pause or retire affected capabilities.

Material changes to models, prompts, retrieval, memory, tools, permissions, policies, data, providers or operating context should reopen the affected tests and release evidence. Begin with the smallest useful capability and expand authority through reviewed changes, not prompt edits alone. Involve security, identity, privacy, records, risk and service owners where sensitive data, privileged access, persistent memory, destructive change, external commitments or incident response are involved; route regulated or other high-stakes judgements to qualified specialists and a separately governed process.