Building a Scenario-Based Evaluation Set for a Business AI System

A useful scenario-based evaluation set recreates bounded pieces of work, not merely polished prompts. Imagine a purchase-request assistant receiving a plausible request whose vendor identifier conflicts with an attached quote, while policy retrieval is unavailable and the quote contains instructions aimed at the assistant. Testing those conditions together reveals whether the system requests evidence, respects its permissions and avoids treating uploaded material as authority. The set should make such behaviour reproducible, gradeable and relevant to a defined release decision.

What should a scenario-based evaluation set cover?

It should cover the ordinary workflow in proportion to credible operational evidence, while deliberately including important boundaries, confirmed failures and prohibited behaviour. Begin by fixing one system version, one workflow, its permitted actors and inputs, its available knowledge and tools, and the decision that results will inform. A credible set uses clearly defined, realistic cases representing expected-use conditions and documents its method; an undefined ‘general assistant’ gives evaluators no stable boundary against which to judge success.

Map task families and variations from authorised work records, support cases, user research, incident records and domain-expert walkthroughs. Task-specific evaluations may also combine historical, domain, synthetic and human-curated material, but none of these sources is automatically representative or suitable for reuse. Record uncertainty where logging is incomplete, and minimise sensitive material before it becomes a case. Define slices and gates before inspecting candidate outputs, so favourable results cannot determine which criteria count.

For the illustrative purchase-request assistant, ordinary coverage might include a complete request with consistent records and accessible policy. Boundary cases could contain a missing document, conflicting amounts, insufficient evidence or failed retrieval. Other cases might ask the assistant to bypass human approval, expose confidential information or obey instructions embedded in a quote. These rules are organisation-specific. Results may need reporting by task, slice, consequence and gate because the significance of failure varies with context.

How should each evaluation scenario be recorded?

Record each scenario as a reproducible case that another evaluator can reconstruct and score. A case needs defined inputs and success criteria, while a grader may inspect the final output, the workflow outcome or an execution trace. Professional evaluation tasks can be grounded in realistic work products and supplied with the context and reference files required for completion. The complete record below is an adaptable editorial synthesis, not a standard prescribed by one source.

• Identity: stable case ID, owner, status, version history, coverage family, task family, slice tags, consequence and intended split.
• Provenance: source category, authorisation record and any access restrictions, with sensitive content minimised.
• Setup: actor and goal, initial workflow state, request, files, relevant history, system instructions and environmental conditions.
• Capabilities: authorised knowledge, permissions, available tools, tool results and the versions of the model, prompt, retrieval corpus, policy and harness.
• Expectations: required facts or state changes, acceptable alternatives, and conditions requiring clarification, abstention, refusal or escalation.
• Controls: prohibited outputs, disclosures, tool calls and state changes, recorded separately from general quality expectations.
• Operation: grader, rubric version, gates, reference solution where useful, predeclared trial aggregation, reviewer qualification and adjudication route.

Do not turn an open-ended task into a search for one ideal sentence. Specify the facts and material constraints that must hold, then recognise acceptable wording or paths. A known working reference solution can demonstrate that a bounded case is solvable and help verify its grader, but it need not be the only valid response. When nondeterminism makes repeated trials relevant, choose the number of trials and aggregation rule before outputs are seen, rather than after a disappointing result.

A useful evaluation case recreates bounded work and makes success, acceptable variation and prohibited behaviour inspectable.

How should each scenario be scored without rewarding the wrong behaviour?

Use the narrowest grading method that can validly distinguish success from failure. Deterministic, model-based and human graders have different strengths; precision alone does not make a check complete or appropriate. Fix rubric versions, gate logic, slice definitions, trial aggregation and the decision rule before comparing outputs. If these change later, record a new evaluation version rather than silently rewriting the meaning of an earlier result.

• Use deterministic checks for objective answers, calculations, schemas, tool arguments, record states and barred actions; verify that the case is solvable and the check cannot reward a shortcut.
• Use reference facts or a working solution when required evidence or the end state is bounded but several expressions or paths are valid.
• Use separate dimensions with observable anchors for genuinely open-ended qualities, and calibrate any model grader against qualified human judgements in the relevant context.
• Use qualified human judgement where domain interpretation or consequences cannot be resolved validly by automation, keeping regulated decisions with authorised roles.

Outcome-oriented grading can be less brittle than demanding one exact execution path, and meaningful components may receive partial credit. That flexibility stops at an expressly prohibited action: this article recommends making prohibited disclosures or unauthorised state changes non-compensable gates defined in advance by product and risk owners. A polished explanation cannot cancel such a failure. Automated grading also needs restraint; OpenAI reports that its GDPval automated grader was not reliable enough to replace experienced occupational graders.

How can reviewers apply the evaluation criteria consistently?

Give reviewers the same decision context, observable anchors and route for defective cases before live scoring begins. State the workflow purpose, actor, available evidence, allowed behaviour, product boundary and rubric version before presenting an output. Task context, intended-solution information and concrete positive and negative examples can improve reviewer reliability. Include boundary examples and a ‘cannot score’ option where evidence, tools or the case itself are defective.

1. Run calibration cases before live review and repeat calibration when the task mix, policy, rubric or reviewer pool materially changes.
2. Blind system identity and output order where practical for comparative judgements; GDPval demonstrates blinded occupational comparison with detailed rubrics, not a universal protocol.
3. Capture independent scores and reasons before discussion, then inspect relevant traces and grades to distinguish system failure from a defective task, grader or environment.
4. Name an adjudication owner and preserve reviewer labels, rationales, rubric versions and outcomes for later audit and grader recalibration.

Treat disagreement as diagnostic evidence, not simply as noise to average away. It may reveal an unclear threshold, missing context, a broken case, legitimate acceptable alternatives or a product decision that has not been made. Correct demonstrably defective evaluation material, but do not force consensus where plural outcomes are valid. This reviewer procedure is an editorial synthesis and should be adapted to the work, the consequence and the qualifications needed to judge it.

How should the evaluation set remain useful through development and release?

Separate a visible development set from a protected release test, then track versions and exposure for both. Development cases can support repeated changes to prompts, retrieval, tools, policies and workflow design, but their results are development evidence because the team has optimised against them. Repeated use of test evidence to guide changes risks implicit overfitting, and validation or test sets can wear out through continued exposure.

• Assign split membership when a case enters the registry and before its outputs influence selection decisions.
• Use the protected test sparingly for release comparisons, keeping its exact cases, answers, rubrics and results from routine iteration.
• Detect exact and near duplicates, shared source records, paraphrases and scenario-template siblings across the two sets.
• Track access to case content, reference answers, rubrics and results rather than assuming a nominally protected folder remains independent.
• Move an exposed release case into development or regression evidence when it materially shapes a change, then add a versioned, independently created replacement.
• Place verified failures that informed diagnosis or remediation in development or regression evidence; test the same failure class in the protected set only with unexposed, non-duplicate cases.

Maintain a registry containing ownership, provenance, authorisation, split, exposure, versions, review history, change reasons and retirement reasons. Evaluation sets can grow through verified operational, historical, domain, synthetic and human-curated cases, but additions still require checking and appropriate minimisation. Reassess coverage and measures after material changes to the workflow, users, policy, knowledge, model, prompts, tools, permissions or operating environment. There is no universal split percentage, refresh interval or case count that guarantees independence.

Report performance by task family, coverage family, slice, consequence and gate, alongside any overall summary. An offline pass is decision evidence, not a certificate of business value, safety, fairness, compliance or production readiness. Product and risk owners should define release uses and consequence-sensitive gates in advance; domain experts should validate realism and expected outcomes. Where cases implicate legal, financial, employment, safety, security, privacy or other regulated judgements, keep decisions with appropriately qualified and authorised people and pair offline results with monitoring, user research and incident review.