Building a Scenario-Based Evaluation Set for a Business AI System

Build the evaluation set around one bounded piece of work, not a collection of impressive prompts. A purchase-request assistant might receive a plausible request with a conflicting vendor identifier, an unavailable policy result and an uploaded quote containing instructions aimed at the assistant. A polished happy-path sample will miss that combination. Useful release evidence must recreate the workflow, make acceptable and prohibited outcomes inspectable, and define slices, grading rules and gates before anyone sees candidate results.

What belongs in a scenario-based evaluation set?

A scenario-based set should combine proportionate coverage of ordinary work with deliberate coverage of difficult boundaries, confirmed failures and barred behaviour. First fix the system version, permitted actors and inputs, available knowledge and tools, and the decision the evaluation will support. A useful evaluation starts with clearly defined, realistic cases representing expected conditions and a documented method. Results may also need separation by relevant slice and consequence because the significance of a failure can vary by context.

Map task families and variations from authorised workflow records, support cases, user research, incident records and domain-expert walkthroughs. Task-specific evaluations can also draw on production, historical, domain, synthetic and human-curated evidence, provided its use is authorised and its representativeness is assessed. Where logs are incomplete or the system is pre-launch, mark the ordinary-work mix as uncertain rather than creating false precision.

• Representative work: normal task families, roles, inputs and operating conditions in relation to the observed workflow mix.
• Important edge cases: valid but difficult conditions such as ambiguity, missing or conflicting evidence, permission limits and unreliable tools.
• Known failures: authorised, minimised reproductions of verified defects or incidents retained as regression evidence.
• Prohibited behaviours: plausible attempts or conditions in which the system must refuse, abstain, clarify or escalate.

For the illustrative purchase-request assistant, ordinary coverage includes complete requests with consistent fields and an accessible current policy. Targeted cases add missing documents, conflicting amounts or vendor identifiers, insufficient authorised evidence, unavailable retrieval, attempts to bypass human approval and instructions embedded in uploaded material. These rules are organisation-specific. The four-family map is an adaptable editorial synthesis, not a universal taxonomy or a requirement for equal case counts.

How should each evaluation scenario be recorded?

Record each scenario as a reproducible case that defines its setup, permissible evidence, acceptable variation, prohibited outcomes and grading method. An evaluation task needs defined inputs and success criteria, and its grader may inspect outputs, outcomes or execution traces. Give the case a stable identity, owner, version history, coverage family, task family, slice tags, consequence, provenance category, authorisation record and intended split so later results can be traced to the case that produced them.

• Setup: actor and goal, initial workflow state, request, files, relevant history, system instructions and environmental conditions.
• Access: authorised knowledge, tools, permissions and the tool results available during the trial.
• Expectations: required facts or state changes, acceptable alternatives, and conditions requiring clarification, abstention, refusal or escalation.
• Controls: prohibited outputs, disclosures, tool calls and state changes, plus grader, rubric, gates, reference material and aggregation rule.
• Operation: reviewer qualification, adjudication route, known limitations, and versions of the model, prompt, retrieval corpus, tools, policy and harness.

Realistic professional cases can be based on work products and include the context and reference files needed to perform the task, but sensitive material still requires authorisation and minimisation. A known working solution can show that a bounded case is solvable and help test its grader without becoming the only acceptable wording or path. If nondeterminism matters, predeclare the trials and aggregation rule before inspecting outputs. The complete record remains an editorial synthesis to tailor locally.

A useful evaluation case recreates bounded work and makes success, acceptable variation and prohibited behaviour inspectable.

How can scenarios be scored without rewarding the wrong behaviour?

Score each scenario with the narrowest method that can validly distinguish success from failure, while keeping prohibited actions outside compensable quality scoring. Deterministic, model-based and human graders have different strengths. Open-ended model grading needs structured rubrics and calibration against relevant human judgement; agreement on familiar development examples alone does not establish validity. OpenAI likewise reports that the automated grader used in its GDPval work was not reliable enough to replace experienced occupational graders.

• Use deterministic checks for objectively verifiable answers, schemas, calculations, tool arguments, record states or barred actions.
• Use reference facts or a working solution when required evidence or an end state is bounded but several valid paths remain.
• Use separate, observably anchored rubric dimensions for genuinely open-ended qualities, then calibrate any model grader against qualified reviewers.
• Use qualified expert judgement when domain interpretation or consequences cannot be resolved validly through automated checks.

Outcome-oriented grading can be less brittle than demanding one exact path, and meaningful components may receive partial credit. However, an explicitly prohibited disclosure or unauthorised action should be a non-compensable gate whose release consequence is defined in advance by authorised product and risk owners. Fix the rubric version, gate logic, slices, trial aggregation and decision rule before comparisons begin. Record later changes as a new evaluation version rather than quietly reinterpreting old results.

How can reviewers apply the criteria consistently?

Give reviewers a short guide that fixes the context, observable anchors and disagreement route before live scoring begins. State the workflow purpose, actor, available evidence, allowed behaviour, product boundary and rubric version before revealing an output. Reviewer reliability can improve when people receive task context, intended-solution information and concrete positive and negative examples. OpenAI's GDPval work demonstrates expert task review, blinded comparative grading and detailed rubrics, but it does not establish a universal reviewer protocol.

• Define observable anchors for every score point, with positive, negative and boundary examples.
• Provide an insufficient-evidence or cannot-score route for defective cases.
• Run calibration cases before live review and again after material changes to the task mix, policy, rubric or reviewer pool.
• Capture independent scores and rationales before discussion, and blind system identity and output order where practical.
• Name an adjudication owner and preserve labels, rationales, rubric versions and outcomes.

Treat disagreement as diagnostic evidence rather than an inconvenience. It may reveal an ambiguous threshold, missing context, a defective grader or environment, a legitimately acceptable alternative, or a product decision that has not yet been made. Inspect traces and grades where available to separate those problems from genuine system failures. The complete procedure is an editorial synthesis, and adjudication should not force consensus when plural judgements remain valid.

How should the set remain useful through development and release?

Keep a visible development set for repeated iteration and a separate, protected test for sparing release decisions. Repeatedly using a test set to guide changes risks implicit overfitting, so its results stop being independent release evidence once the team has optimised against its cases, answers, rubrics or results. Assign split membership when a case enters the registry, before routine execution or output inspection, and do not prescribe a universal split percentage.

Check across sets for exact and near duplicates, shared source records, paraphrases and scenario-template siblings. Validation and test sets can wear out through repeated use, while reliable tests need unseen, representative examples without development duplicates. A confirmed failure that has already shaped diagnosis or remediation belongs in development or regression evidence. Protected coverage may test the same failure class only through independently created cases that have not guided the change.

• Track ownership, provenance, authorisation, split membership, exposure, versions, review history, change reasons and retirement reasons.
• Move a materially exposed protected case into development or regression evidence and add a versioned replacement.
• Add verified cases from authorised production, historical, domain, synthetic and human-curated sources after appropriate review.
• Reassess coverage after material changes to users, workflow, policy, knowledge, model, prompts, tools, permissions or operating conditions.

Report results by task family, coverage family, slice, consequence and gate, not only as an average. Evaluation methods and metrics should be reassessed through the lifecycle as purpose, context, available data and significant risks change. An offline pass is decision evidence, not proof of business value, safety, fairness, compliance or readiness. Pair it with monitoring, user research and incident review, and keep regulated or high-consequence judgements with appropriately qualified and authorised people.