Building a Scenario-Based Evaluation Set for a Business AI System

A useful scenario-based evaluation set recreates bounded pieces of business work, then makes acceptable and prohibited outcomes inspectable. Consider a purchase-request assistant facing a plausible request, conflicting vendor identifiers, an unavailable policy result and instructions hidden in an uploaded quote. Polished sample prompts will not show whether it handles that combination properly. The team needs reproducible cases, predefined grading and protected release evidence that reveal consequential failures rather than burying them in one impressive average.

What should a scenario-based evaluation set cover?

It should cover the observed mix of ordinary work and deliberately selected boundaries, confirmed failures and prohibited behaviour. Start with one system version and one bounded workflow: name its permitted actors and inputs, available knowledge and tools, operating conditions, and the decision the evaluation will inform. An undefined “general assistant” offers no stable basis for deciding which evidence, behaviour or outcome is relevant.

Map task families and meaningful variations from authorised workflow records, support cases, user research, incident evidence and domain-expert walkthroughs. Production records may help, but they are not automatically complete, representative, correctly labelled or lawful to reuse. Mark uncertainty where evidence is weak. Synthetic and human-curated cases can fill deliberate gaps when their assumptions, provenance and expected behaviour are reviewed rather than treated as ground truth.

For an illustrative purchase-request assistant, representative cases might contain complete, consistent request packs. Boundary cases can introduce missing documents, conflicting amounts, unavailable policy retrieval or facts absent from authorised evidence. Known failures should be reduced to authorised, de-identified reproductions. Prohibited cases can ask the assistant to approve spending, bypass a human gate, disclose another party's information or obey instructions embedded in a quote. These rules are illustrative; each organisation must define its own.

Ordinary cases should broadly follow trustworthy evidence about the workflow mix, but frequency must not control the whole set. A rare unauthorised action may deserve explicit coverage because its consequence differs from a minor drafting flaw. Define reporting slices, prohibited-behaviour gates and the intended release decision before anyone generates or inspects candidate outputs. Otherwise, favourable results can influence which criteria are later declared important.

How should each evaluation scenario be recorded?

Each scenario should be a reproducible case record that lets another evaluator rebuild the starting conditions, identify valid alternatives and detect barred outcomes. Give it a stable identity, owner, status and change history. Record its coverage family, task family, slice tags, consequence, provenance, authorisation and intended split. This combined record is an adaptable editorial synthesis, not a universal schema prescribed by one source.

• Setup: actor and goal, initial workflow state, request, files, relevant history, system instructions and environmental conditions.
• Available capability: authorised knowledge, permissions, tools and the precise tool results returned during the test.
• Expected behaviour: required facts or state changes, acceptable alternative outputs, and conditions requiring clarification, abstention, refusal or escalation.
• Forbidden behaviour: prohibited outputs, disclosures, tool calls, actions and state changes, recorded separately from general quality.
• Evaluation operation: grader, rubric, gates, reference material, reviewer qualification, adjudication route and all relevant system versions.

Do not reduce an open-ended task to one preferred sentence. When several phrasings or paths are valid, specify the facts, evidence and end state that matter. A known working reference solution can demonstrate that the task is solvable and help test the grader, but it need not be the only acceptable route. Realistic professional tasks may also include the work context and reference files needed to produce the intended deliverable.

For a nondeterministic or multi-step system, decide before seeing outputs whether the case needs repeated trials and how their results will be combined; there is no universal trial count. Preserve the model, prompt, retrieval corpus, tools, permissions, policy and evaluation-harness versions. Without those details, a later evaluator cannot tell whether a changed result reflects the system, the evidence, the environment or the test itself.

A useful evaluation case recreates bounded work and makes success, acceptable variation and prohibited behaviour inspectable.

How should each scenario be scored without rewarding the wrong behaviour?

Score each scenario with the narrowest method that can validly distinguish success from failure. Use deterministic checks for objectively verifiable calculations, schemas, tool arguments, record states and prohibited actions. Use reference facts or a working solution where the required evidence or end state is bounded but several paths remain valid. Reserve anchored rubrics for genuinely open-ended qualities and qualified human judgement for interpretation an automated check cannot resolve.

• Inspect every deterministic check for omissions, false shortcuts and whether the case is genuinely solvable.
• Grade material outcomes and constraints instead of demanding one incidental execution path.
• Define separate, observable rubric dimensions rather than relying on a broad impression of quality.
• Calibrate model-based graders against qualified human judgements in the relevant business context.
• Keep regulated or consequential professional decisions with appropriately qualified and authorised people.

Partial credit can show which meaningful components succeeded, but it must not hide the failed component. An explicitly prohibited disclosure or unauthorised action should be a non-compensable gate: a polished explanation cannot cancel it. That is an editorial recommendation, not a universal standard, so product and risk owners must define the actual barred behaviours and release consequences before outputs are compared.

Automated graders are not objective merely because they agree with familiar development examples. Open-ended graders need structured rubrics, relevant human calibration and continuing inspection of outputs, traces and grades. OpenAI reports that the automated grader in its GDPval work was not reliable enough to replace experienced occupational graders; that specific finding is a warning against casual substitution, not proof that automation is unsuitable for every bounded check.

How can reviewers apply the evaluation criteria consistently?

Reviewers can apply criteria more consistently when they receive the same context, observable anchors and calibration material before live scoring. State the workflow purpose, actor, available evidence, permitted behaviour, product boundary and rubric version before presenting an output. For every score point, describe what the reviewer can observe and provide positive, negative and boundary examples, plus an insufficient-evidence or cannot-score route for a defective case.

1. Run calibration cases before live review and repeat calibration when the rubric, policy, task mix or reviewer pool changes.
2. Blind system identity and output order where practical for comparative judgements.
3. Capture independent scores and rationales before reviewers discuss the case.
4. Investigate disagreement before adjudicating it, then preserve the original labels, reasons, rubric version and outcome.
5. Name an adjudication owner who can distinguish a faulty case from an unresolved product or policy decision.

Disagreement is useful diagnostic evidence. It may reveal a missing fact, vague threshold, defective grader, unstable environment, legitimately acceptable alternatives or a decision the organisation has not yet made. Do not force consensus where plurality is valid. Structured rubrics, calibration and inspection of traces and grades help separate a genuine system failure from a broken evaluation. GDPval demonstrates expert task review, blinded comparison and detailed rubrics, but it does not establish a universal reviewer protocol.

How should the evaluation set remain useful through development and release?

Keep the set useful by separating visible development cases from protected release evidence, then tracking versions and exposure. Teams may repeatedly use development cases to improve prompts, retrieval, tools, policies and workflow logic. Those results remain valuable, but they are development evidence because the team has optimised against the cases. Repeatedly using a test set to guide changes risks implicit overfitting and eventually wears out its independence.

Assign split membership when a case enters the registry, before routine execution or output inspection. Use the protected set sparingly for final comparisons or release decisions, and prevent its exact cases, answers, rubrics and results from guiding iteration. Check across sets for exact duplicates, near duplicates, shared source records, paraphrases and siblings generated from the same scenario template. Track access to case content as well as reference material and results.

A confirmed failure that has informed diagnosis or remediation belongs in development or regression evidence. A protected set may test the same failure class only with an independently created, non-duplicate case that did not guide the change. If a protected case materially shapes an improvement, move it to development or regression and add a versioned replacement. No fixed split percentage, replacement interval or case count can guarantee independence.

• Record case ownership, provenance, authorisation, split, exposure, versions, review history, change reasons and retirement reasons.
• Add verified failures only after confirming expected behaviour and minimising confidential or personal information.
• Reassess coverage after material changes to users, workflow, policy, knowledge, models, prompts, tools, permissions or the operating environment.
• Report results by task family, coverage family, slice, consequence and gate rather than relying on one aggregate score.

Treat the completed evaluation as decision evidence, not a certificate of safety or readiness. Domain experts should validate workflow realism and expected outcomes, while privacy, security, legal, compliance and other qualified functions should review cases that implicate their governed material. Keep legal, medical, financial, employment, safety and other regulated judgements with authorised people. Pair offline results with monitoring, user research, incident review and other evidence appropriate to the system.