Test behavior, not confidence

IDEA 1

Choose a revealing failure case

An evaluation case pairs an input with a way to judge the result. A useful set contains ordinary requests and examples that expose likely mistakes: an exact capacity boundary, missing evidence, a conflicting record, or no valid choice. If every example is easy, a broken rule can look reliable. AgentBench studies agents across eight interactive environments, illustrating why one kind of task cannot stand in for all agent behavior.

Start from the task's definition of success. Here a room must be available and hold at least the requested group size. Capacity equal to the group size is allowed. A no-match answer is correct when no room satisfies both requirements. Writing these expectations before editing the agent makes it harder to quietly change the target to fit its output.

Turn a requirement into cases

1. Requirement. Capacity must be at least the group size.
2. Boundary. Try exactly 20 people in 20 seats.
3. Failure. Try a large room that is unavailable.

Follow this authored example, then test the idea in the lab.

Read the example

const valid = room.available && room.capacity >= people;

This check states a measurable requirement. Words such as 'excellent room' do not replace it.

Think it through: Which case is most likely to expose an accidental > instead of >= comparison?

IDEA 2

Test output and tool behavior

A grader is the procedure that assigns a result to a trial. Code graders can check exact fields and rules. Human reviewers can assess clarity or usefulness under a rubric. Model graders can help with judgments, but their outputs are fallible and need validation. None should be described as an automatic detector of truth in every domain.

Our grader reads the room fixture and an authored candidate. It checks existence, availability, and capacity separately, or verifies that a proposed null really means no valid room exists. The candidate's confidence field is deliberately ignored. The surrounding lab also checks tool calls: reading the evidence is part of the required behavior, while a correct-looking hard-coded answer is insufficient.

Grade the result against evidence

1. Observe. Read the actual room records.
2. Check. Evaluate each stated requirement.
3. Report. Keep the failed criterion visible.

Follow this authored example, then test the idea in the lab.

Read the example

const failed = Object.keys(checks).filter(key => !checks[key]);
return { passed: failed.length === 0, failed };

A specific failed criterion helps you fix the system. The grader does not award points for confident wording.

Think it through: A candidate says confidence: 0.99, but the room is unavailable. What should this grader do?

IDEA 3

Repeat live-model trials without tuning to the test set

A task is the request being evaluated; a trial is one attempt at that task. A live model may make different choices across repeated trials. Report the number of trials and the observed variation under a recorded model, configuration, dataset, and grader version. A single successful run does not establish reliable behavior.

Keep some cases held out: do not use them to choose prompts or edit rules during development. Fitting the visible examples can hide failures on nearby inputs. This deterministic browser lab runs no live model, so repeating unchanged input cannot measure model variability. Its three cases teach evaluation mechanics.

Read benchmark labels carefully. SWE-bench evaluates software changes for repository issues; its named variants use different task sets. A result there answers a different question from a room-selection check here. Record which tasks, agent setup, and grader produced a score before comparing it with another result. No benchmark is a certificate that an agent is ready for every use.

Separate improvement from assessment

1. Develop. Use visible cases to diagnose mistakes.
2. Freeze. Record the code and grader before the final check.
3. Assess. Use held-out inputs; repeat actual model trials when relevant.

Follow this authored example, then test the idea in the lab.

Read the example

const report = { dataset: "rooms-practice-v1", grader: "availability-capacity-v1" };

Version labels identify what was tested. They do not make a tiny practice set representative by themselves.

Think it through: What does passing these three fixed JavaScript cases establish?

Put it into practice

Judge a proposed room using explicit fixture facts; report failed criteria and accept a justified no-match.

1. Read the selected case and predict its expected result.
2. Run the starter once. Use the failed check and tool trace to locate the missing rule.
3. Insert the explained snippet at the TODO, then run the case again.
4. Test all three cases. Change the experiment input and explain whether the same rule still works.

Your next experiment: Propose null even though room A is available and large enough. Which criterion should fail? Then try an invented room ID.

State the success rule, test revealing failures, and describe only what the observed trials support.

Key terms

Grader

The code, rubric, or review process that judges a trial.

Held-out case

A case reserved for assessment rather than development decisions.

Trial

One attempt at an evaluation task under a recorded configuration.

Sources and scope

Original Stemtiq teaching, reviewed 2026-09-14. The named researchers and organizations do not endorse this course. Classroom cases are authored exercises, not published findings.