Building a Scenario-Based Evaluation Set for a Business AI System

A scenario-based evaluation set should recreate one bounded piece of business work, not collect polished prompts in the abstract. Consider a purchase-request assistant facing a plausible request, conflicting vendor identifiers, an unavailable policy result and instructions hidden in an uploaded quote. A simple happy-path prompt misses the combination. Useful release evidence instead defines the system version, actors, permitted inputs, available knowledge and tools, expected outcomes, barred actions and decision the results will support. It then tests ordinary work alongside consequential boundaries and verified failures, using reproducible cases and grading rules fixed before anyone sees a preferred system's results.

What should a scenario-based evaluation set cover?

It should cover the observed mix of ordinary work and deliberately selected boundaries, known failures and prohibited behaviour. First name one workflow, its permitted users and inputs, the knowledge and tools available to the tested version, and the decision the evaluation will inform. NIST guidance connects accuracy measurement with clearly defined, realistic test sets representative of expected-use conditions and documented methods. That does not make traffic frequency the only inclusion rule: different failures can carry different consequences, so teams may need both slice-level reporting and consequence-sensitive treatment.

Build the map from authorized workflow records, support cases, user research, incident records and domain-expert walkthroughs. Task-specific evaluations can also draw from production, historical, domain, synthetic and human-curated material, but none of those sources is automatically representative, correctly labelled or permitted for reuse. Mark uncertainty where evidence is incomplete. Define reportable slices, prohibited-behaviour gates and the intended release decision before generating outputs; otherwise, observed performance can quietly influence which criteria count.

For the illustrative purchase-request assistant, ordinary coverage includes complete, internally consistent packages and accessible current policy. Boundary cases include a missing document, conflicting amounts, insufficient authorized evidence and an unavailable or inconsistent retrieval result. Known failures use minimized reproductions rather than copied confidential records. Prohibited cases test requests to bypass approval, disclose unrelated information or follow instructions embedded in a quote. These rules are organization-specific; the four-family map is an adaptable editorial synthesis, not a universal taxonomy or sampling formula.

How should each evaluation scenario be recorded?

Record each scenario as a versioned case that another evaluator can recreate and grade. An evaluation task needs defined inputs and success criteria, while a grader may inspect outcomes, outputs or execution traces. Give the case a stable identity, owner, coverage and task families, slice tags, consequence, provenance category, authorization record and intended split. The complete record below is an editorial synthesis: tailor it to the workflow, evidence controls and release decision rather than treating every field as mandatory for every system.

• Identity: case ID, set version, owner, status, change history, split membership, provenance and authorization.
• Setup: actor and goal, initial workflow state, request, files, relevant history, system instructions, authorized knowledge, permissions, tools and tool results.
• Expectations: required facts or state changes, acceptable alternatives, and conditions requiring clarification, abstention, refusal or escalation.
• Evaluation: prohibited outputs, disclosures, tool calls and state changes; grader, rubric, gates, reference material, trial rule, reviewer qualifications and adjudication route.

Realistic professional evaluations can be based on work products and include the context and files needed to complete them. A known working reference solution can also demonstrate that a bounded case is solvable and help verify its graders. It should not become the only acceptable wording or path when alternatives are valid. For a nondeterministic or multi-step system, declare any trial count and aggregation rule before inspecting outputs, and record the model, prompt, retrieval corpus, tools, policy, permissions and harness versions.

A useful evaluation case recreates bounded work and makes success, acceptable variation and prohibited behaviour inspectable.

ModelFold Editorial Team

How should each scenario be scored without rewarding the wrong behaviour?

Score each scenario with the narrowest method that can validly distinguish success from failure. Deterministic, model-based and human graders have different strengths; no type is inherently best. Check objective answers, schemas, calculations, tool arguments, record states and barred actions deterministically where possible. Use bounded reference facts when several phrasings or paths are acceptable, and use separate, observable rubric dimensions only for genuinely open-ended qualities. Calibrate model-based grading against qualified human judgment in the relevant context.

• Use deterministic checks for objectively verifiable outcomes, while confirming that each check is complete and the case is solvable.
• Use reference facts or a working solution to identify required evidence or end states without demanding an incidental path.
• Use anchored rubric dimensions for qualities such as completeness or groundedness, with examples for each score point.
• Use qualified expert judgment when domain interpretation cannot be resolved validly through automated checks, leaving regulated decisions with authorized roles.

Outcome-oriented grading can be less brittle than requiring one execution path, and meaningful components may receive partial credit. Keep prohibited disclosures or unauthorized actions outside that compensable score: treating them as release gates is an editorial recommendation, and product and risk owners must define the actual behaviour and consequence in advance. OpenAI reports that the automated grader in its GDPval work was not reliable enough to replace experienced occupational graders. That finding is a warning against unearned confidence, not a ban on automated checks.

How can reviewers apply the evaluation criteria consistently?

Reviewers can apply criteria more consistently when they receive the same context, observable anchors and route for defective cases. Before showing an output, state the workflow purpose, actor, available evidence, allowed behaviour, product boundary and rubric version. NIST describes how task context, intended-solution information and concrete positive and negative examples can improve reviewer reliability, while disagreement can reveal unclear rules. Human review still requires calibration; it is not automatically accurate, unbiased or consistent.

1. Define observable anchors for every score point, with positive, negative and boundary examples plus an insufficient-evidence or cannot-score option.
2. Run calibration cases before live review and repeat them when the task mix, policy, rubric or reviewer pool changes.
3. Blind system identity and output order where practical, then capture independent scores and rationales before discussion.
4. Investigate disagreement as a possible case defect, unclear threshold, missing context, legitimate alternative or unresolved product decision.

Name an adjudication owner and preserve labels, rationales, rubric versions and outcomes for later review or grader recalibration. Structured rubrics, human calibration and inspection of traces and grades help distinguish a system failure from a defective task, grader or environment. GDPval demonstrates multi-step expert task review, blinded comparative grading and detailed rubrics, but it does not establish a universal protocol. Do not force consensus when several outcomes are legitimately acceptable or an authorized product decision remains open.

How should the evaluation set remain useful through development and release?

Keep a visible development set for iteration and a separate protected test for release evidence. Repeatedly using a test set to guide changes risks implicit overfitting, so its result no longer represents the same degree of independence. Assign split membership when a case enters the registry, before routine execution or output inspection. Use development cases for prompt, retrieval, tool, policy and workflow changes; reserve protected cases for sparing comparisons or release decisions, without letting their exact content, answers, rubrics or results guide iteration.

• Check exact and near duplicates, shared source records, paraphrases and scenario-template siblings across splits.
• Track access to case content, reference answers, rubrics and results, not merely access to a file name.
• Keep diagnosed failures in development or regression evidence; test the same failure class in protected evidence only with independently created, non-duplicate cases.
• Move an exposed protected case into development when it materially shapes a change, then add a versioned replacement.
• Record ownership, provenance, authorization, split, exposure, versions, reviews, change reasons and retirement reasons in a set registry.

Validation and test sets can wear out through repeated use, and reliable tests should use representative examples while avoiding development-data duplicates. Task-specific sets can grow with verified production, historical, domain, synthetic and human-curated cases, provided each addition is authorized, minimized and checked. Reassess methods and metrics when the workflow, users, policy, knowledge, model, prompts, tools, permissions or operating environment materially changes. There is no universal case count, split percentage or refresh interval.

Report results by task family, coverage family, slice, consequence and gate, not only as one aggregate. An offline pass supplies decision evidence; it does not establish business value, safety, fairness, compliance or production readiness by itself. Product and risk owners should set consequence-sensitive gates before inspecting results, while domain experts validate workflow realism and expected outcomes. Bring in authorized data, privacy, security, legal, compliance or other qualified functions when their governed material or judgment is implicated, and keep regulated decisions with appropriately qualified people. Pair offline evaluation with monitoring, user research and incident review.