Practical intelligence for accountable AI programmes.
Bind prompts, models, tools and workflow logic into one testable AI release, then promote it in stages and restore a complete known-good bundle.
ModelFold Editorial Team8 min read
An AI release is the complete behaviour-affecting stack, not merely a prompt string or model endpoint. Rolling back the model while leaving a changed tool schema, permission policy, retrieval setting or retry path live does not restore the system that operators tested. Give the whole candidate one identity so the team can establish what was evaluated, what served each request and which compatible configuration should receive traffic if the rollout stops. Component histories still matter, but nobody should have to reconstruct the effective bundle during an incident.
Release rules worth keeping
One AI release should include every runtime dependency that can materially change served behaviour, authority, risk, cost, latency or observability. NIST treats AI lifecycle activities as interdependent and calls for component inventories, while Google Cloud describes production ML as far more than model code. The practical boundary is system-specific: include a first-party or third-party dependency when changing it could alter the service or the decision to release it, not simply because it exists somewhere in the architecture.
Evaluation datasets, graders, rubrics and thresholds sit beside this runtime inventory as assurance assets. They do not normally serve a user request, but changing them can change how identical runtime behaviour is judged. Version them and link them to the release record. A change to any listed runtime component creates a new candidate; a changed evaluation asset creates new decision evidence and must remain distinguishable from the evidence used for the earlier approval.
Bind the stack by freezing a candidate manifest whose references resolve to exact components and effective settings. Store the release ID, creation time, owner, target service, status, stop conditions, rollback owner and previous known-good release. For each component, capture a stable version, commit, digest, content hash or signed reference. A mutable alias such as production or latest may remain useful for routing, but the manifest must record what that alias resolved to when the candidate was built.
Keep secrets outside the manifest; record approved connection identities or references instead. Keep exposure allocations and deployment timestamps in linked promotion events so the frozen candidate does not mutate as traffic grows. A preapproved allocation increase may advance r18 unchanged. Altering its prompt, setting, tool, permission, retrieval reference, workflow, schema or behaviour-affecting binding produces another candidate. This division preserves both a stable object for evaluation and an honest chronology of what happened in each environment.
If something can change served behaviour, or the evidence authorising it, give it a resolved identity in the release record.
The candidate should advance only when evidence for that exact complete configuration supports an explicit promote decision. Release notes should state the intended behaviour, every changed dependency, affected scenarios and interfaces, permission or observability changes, known limitations, residual risks, rollout owner and compatible fallback. NIST calls for documented, repeatable evaluation and a decision on whether deployment should proceed. The record should therefore finish with promote, hold or reject, plus the person or system accountable for that disposition.
Compare the candidate with the current release using real workflow examples, important segments and rare but costly cases. Audit automated graders and retain domain review where judgement matters. One favourable average must not conceal a material contract, authority, safety or segment-level regression. Version the dataset, rubric, graders and thresholds because an altered gate changes how results are interpreted. For releases with greater exposure, separating the change author from the promotion approver provides a clearer decision trail.
| Gate | Evidence | Decision owner | Failure response |
|---|---|---|---|
| Build and contract | Resolved manifest, schema compatibility, load checks and valid bindings | Platform owner | Reject and rebuild the candidate |
| Behaviour and quality | Task results, escalation behaviour, important segments and costly edge cases against the current release | Service owner | Hold for investigation or revision |
| Safety and authority | Policy checks, permission boundaries, approval paths and prohibited-action tests | Risk and service owners | Reject; do not expose the candidate |
| Service readiness | Reliability, latency, cost, trace coverage and alert readiness against owned limits | Operations owner | Hold, remediate and rerun the gate |
Move the same resolved candidate through a measured ladder from non-actuating observation to full traffic. Where feasible, begin with shadowing or replay, but disable or sandbox write tools and other consequential side effects rather than repeating production actions blindly. Predeployment evidence remains necessary, yet it cannot reproduce every live condition. NIST pairs testing before deployment with monitoring during operation, while OpenAI's evaluation guidance says external-facing releases still need online experiments.
Record cohort rules, traffic allocation, observation timing and each disposition as deployment events linked to the manifest. Choose allocation, sample requirements and observation windows from service risk, traffic, detection latency and the team's operational capacity; no universal percentage is defensible. Shadow behaviour can differ from active use, and a canary may miss rare failures or unrepresented traffic. Limited exposure therefore informs a decision rather than proving complete production safety.
Stop exposure for a predeclared safety or policy breach, unauthorised tool behaviour, contract failure or severe reliability failure, and restore the complete compatible known-good release when rollback is the chosen response. Define service-specific limits for other regressions before rollout. An ambiguous signal may justify a pause and investigation rather than an automatic rollback, but it still needs an owner and recorded disposition. Rehearse restoration beforehand; a fallback that has never been exercised is only an assumption.
Configuration rollback governs future routing. It does not delete a sent message, reverse a data write, cancel an approval or remove a CRM task already created. Use release-tagged traces to find affected requests, workflow paths and external action identifiers, then follow the authorised process for containment, reconciliation, correction, notification or compensating action. For support-assistant-r18, that could mean disabling task creation, identifying malformed task IDs and applying the organisation's CRM correction procedure independently of restoring r17.
A reconstructable release needs a linked record of the effective configuration, evidence and decisions, not a promise of identical output replay. Retain the frozen manifest, resolved component references, parameters, environment bindings, compatibility findings, evaluation versions and results, approvals, deployment events, traffic allocations, findings, rollback events and final disposition. Google Cloud recommends preserving versions, execution metadata, parameters, artifacts, evaluation results and the previous serving target, providing a practical foundation for that record.
Call the result configuration reproducibility and decision reproducibility. Pinned snapshots, hashes and archived settings can show what the team intended to run and why it advanced, but stochastic sampling and hosted-service changes may prevent byte-identical replay. Payload retention must follow organisational information-handling rules rather than a desire for exhaustive traces. When a release changes sensitive-data use, consequential permissions, regulated work or external-action remediation, involve the organisation's qualified security, privacy, legal, records, risk or domain professionals.
Version the prompt, resolved model and parameters, tools, permissions, policies, retrieval or context settings, workflow code, schemas, runtime dependencies and behaviour-affecting environment bindings. Also version the evaluation datasets, graders, rubrics and thresholds linked to the release decision, even though they do not normally execute in the serving path.
No. Tool contracts, permissions, policies, retrieval settings, workflow logic, schemas, dependencies and environment bindings can also change production behaviour. Link every relevant resolved component under one release identity so the tested configuration can be identified without reconstructing separate histories.
Evaluate the complete candidate against the current release using applicable contract, workflow-quality, segment, safety, authority, tool, reliability, latency and cost evidence. Each gate should use versioned methods and end with an owned promote, hold or reject decision. A passing aggregate score does not override a material failure in a critical dimension.
A preapproved exposure increase can remain a deployment event for the same immutable candidate. Changing behaviour-affecting configuration, authority, context, routing or an environment binding creates a new candidate with its own identity and evidence. Record both types of change so exposure history never obscures configuration history.
Rollback restores future traffic to a complete, compatible known-good bundle. It does not undo tool calls or actions already completed in external systems. Use release-linked traces to identify affected actions, then apply a separately authorised containment, correction, reconciliation or compensation procedure.
This article was researched using the following sources:
We report on how AI actually lands inside a business. Our work starts from named sources, separates what we found from what we think, and uses AI assistance for research and drafting under documented editorial controls. We are not a substitute for individual expert review.
Build a reproducible AI evaluation set covering everyday work, hard boundaries, known failures and barred actions, with valid scoring and protected tests.
Build a maintainable AI inventory, rate each use across four explainable dimensions, and route it to proportionate review as its context changes.
A practical method for setting an AI assistant’s context, tool permissions, action limits, approvals, refusals, evidence and accountable handoffs.
