Practical intelligence for accountable AI programs.
Build a scenario-based AI evaluation set covering real work, difficult boundaries, known failures, grading, reviewer calibration, and release testing.
ModelFold Editorial Team8 min read
A useful scenario-based evaluation set recreates bounded pieces of business work, not just a collection of polished prompts. Imagine a purchase-request assistant receiving a plausible request with conflicting vendor identifiers, an unavailable policy result, and instructions embedded in an uploaded quote. A simple sample prompt may show that the assistant writes clearly. It will not show whether the system notices the conflicts, respects its permissions, requests missing evidence, and avoids following the embedded instruction.
Start with one workflow, one system version, and one release decision. Then build three connected assets: a coverage map, a reproducible case record, and a versioned registry that separates development evidence from a protected release test. Define slices, grading rules, prohibited-behavior gates, and decision criteria before inspecting results. Otherwise, an attractive average can conceal the boundary failures that matter most.
Key takeaways
A scenario-based evaluation set should cover the workflow's ordinary task mix while deliberately including important edge cases, verified failures, and prohibited behavior. First define the permitted actors and inputs, available knowledge and tools, system version, and decision the evaluation will support. NIST connects valid accuracy measurement to clearly defined, realistic test sets that represent expected-use conditions, accompanied by a documented method.
Map task families and meaningful variations from evidence the organization is authorized to use. Possible sources include workflow records, support cases, user research, incident reports, domain-expert walkthroughs, and carefully reviewed synthetic cases. Operational records are not automatically representative or correctly labeled, and synthetic cases are not automatically unrealistic. Record uncertainty when logs are incomplete, access is restricted, or a prelaunch workflow has no traffic history.
| Coverage family | Question answered | Potential source evidence | Inclusion logic |
|---|---|---|---|
| Representative work | Can the system handle expected task families, actors, inputs, and conditions? | Authorized workflow records, research, domain walkthroughs, and validated synthetic cases | Reflect the observed mix where evidence supports it, while preserving meaningful slices and documenting uncertainty. |
| Important edge cases | What happens at valid but difficult boundaries? | Ambiguous requests, missing context, permission limits, unusual inputs, and unreliable tool results | Select consequential boundaries deliberately and test both when the behavior should occur and when it should not. |
| Known failures | Does a verified defect remain fixed? | Confirmed incidents, complaints, support cases, and surprising outputs | Retain an authorized, minimized reproduction as development or regression evidence with provenance and version history. |
| Prohibited behaviors | Does the system avoid explicitly barred outputs or actions? | Product rules, permission models, policies, risk analysis, and plausible misuse attempts | Include direct and indirect attempts deliberately; frequency alone must not determine whether a barred outcome is tested. |
For the illustrative purchase-request assistant, ordinary coverage might include a complete request with consistent fields and an accessible policy. Deliberate cases would add a missing document, conflicting amounts, insufficient authorized evidence, a request to bypass approval, instructions inside an uploaded quote, and unavailable or inconsistent retrieval. These rules are organization-specific. Predefine the slices whose results will be reported and the barred behaviors that trigger a gate before outputs can influence those choices.
Each scenario should be a reproducible case record containing enough setup, evidence, constraints, and success criteria for another evaluator to recreate the task and grade the result. Realistic professional evaluations can be built around work products and the context and files needed to produce them. The record should preserve what the tested system actually knew, could access, and was permitted to do at execution time.
Separate acceptable variation from prohibited outcomes. A useful case may accept several phrasings or paths while still requiring the assistant to distinguish policy text from unresolved facts. For deterministic or bounded cases, a known working solution can show that the setup is solvable and help test the grader. It should not become the only acceptable wording or path. When nondeterminism matters, predeclare the number of trials and their aggregation rule before inspecting outputs.
A useful evaluation case recreates bounded work and makes success, acceptable variation, and prohibited behavior inspectable.
Score each scenario with the narrowest method that can validly distinguish success from failure. Deterministic, model-based, and human graders have different strengths; none is automatically appropriate. Grade material outcomes and constraints instead of demanding one incidental path. Keep prohibited actions outside compensable quality scoring: a polished explanation cannot offset an unauthorized approval, confidential disclosure, invented policy rule, or impermissible tool call.
Prefer outcome-oriented checks when several legitimate paths can reach the same required state. Partial credit can reveal which meaningful components succeeded, but it must not erase a failed gate. OpenAI reports that its GDPval automated grader was not reliable enough to replace experienced occupational graders. Fix rubric versions, gate logic, slice definitions, trial aggregation, and decision rules before comparing candidates; document later changes as a new evaluation version.
Reviewers can apply the criteria more consistently when they receive the workflow context and observable anchors they need before seeing live outputs. State the actor, available evidence, allowed behavior, product boundary, and rubric version. Define what each score point looks like with positive, negative, and boundary examples. Provide an insufficient-evidence or cannot-score route when the case, environment, or supporting evidence is defective.
Do not force agreement merely to improve a reliability statistic. A disagreement may reveal that the rubric rejects a valid alternative or that the organization has not decided the expected behavior. GDPval offers one documented example of multi-step expert review, blinded comparison, and detailed rubrics, but it does not establish a universal protocol. The procedure here is an adaptable synthesis, and reviewer qualifications should match the work and its consequences.
Keep the set useful by separating visible development cases from a protected release test and managing both in a versioned registry. Development cases support repeated prompt, retrieval, tool, policy, and workflow iteration, so their scores are development evidence. A protected test supports a different conclusion only while its exact cases, answers, rubrics, and results have not materially guided system changes.
Assign split membership when a case enters the registry, before routine execution or output inspection. Check for exact and near duplicates, shared source records, paraphrases, and scenario-template siblings across splits. A verified failure that guided diagnosis belongs in development or regression evidence. Protected coverage may test the same failure class only through independently created, nonduplicate cases that did not guide the remediation.
Maintain a registry recording ownership, provenance, authorization, split, exposure, relevant versions, review history, change reasons, and retirement reasons. Report results by task family, coverage family, slice, consequence, and gate, not only as an overall score. An offline pass remains one source of decision evidence, not a certificate of business value, safety, fairness, compliance, or production readiness. Pair it with monitoring, user research, incident review, and other evidence appropriate to the system and keep regulated judgments with qualified, authorized people.
Bound the workflow, system version, actors, inputs, tools, permissions, and decision first. Map ordinary work and deliberately add important edges, verified failures, and prohibited behavior. Create reproducible case records, select valid graders, predefine slices and gates, separate development from protected evidence, and maintain a versioned registry.
Scenario-based AI evaluation tests a bounded AI-enabled workflow through reproducible cases. Each case specifies the actor, starting state, inputs, available context and tools, required outcomes, acceptable alternatives, prohibited outcomes, and grading method. It evaluates whether the system performs work and respects its limits under defined conditions.
There is no universal case count. Size and composition depend on the release decision, workflow variability, consequential slices, grading reliability, and available authorized evidence. Track coverage and uncertainty directly instead of treating an arbitrary total as proof of sufficiency.
Use the method that fits the task. Deterministic checks suit objective outcomes, reference facts or solutions suit bounded variation, and observable rubrics suit open-ended quality. Use qualified human judgment when domain interpretation or consequences cannot be resolved validly by automated checks.
A development set is visible and reused to improve prompts, retrieval, tools, policies, and workflows. A protected test is assigned before routine execution, accessed sparingly, and used for release evidence. It loses independent-test value when its cases, answers, rubrics, or results materially guide changes.
This article was researched using the following sources:
We report on how AI actually lands inside a business. Our work starts from named sources, separates what we found from what we think, and uses AI assistance for research and drafting under documented editorial controls. We are not a substitute for individual expert review.
A practical method for bounding AI assistants with capability-specific context, tool permissions, approvals, refusals, evidence, and safe release tests.
Bind prompts, models, tools, policies, and workflow logic into one testable AI release, then promote and roll back the complete configuration safely.
Build a controlled generative AI drafting workflow that separates approved evidence from prose, sets clear reviews, and records consequential changes.
