Tell the story of what you built

IDEA 1

Show one success and one informative failure

A demonstration shows a selected execution. An evaluation report describes the whole tested set. Include a successful case to explain the intended flow, then a failure or limitation to show where the behavior stops meeting the specification. A project can teach something valuable even when it is incomplete.

Name what you measured. Passing three classroom fixtures means the program satisfied those checks, not that every future user request will succeed. This lesson reads fictional stored runs to practice reporting; it does not claim to have measured a deployed agent or a live model.

From demo to inspectable evidence

1. Success. Explain a completed supported case.
2. Failure. Keep an unmet condition or execution error visible.
3. Scope. State the number and kind of cases tested.

The current lab's trace shows report-building calls; the stored run records describe a separate fictional evaluation.

A bounded result statement

// Good: "2 of 3 authored fixture cases passed."
// Too broad: "This agent is reliable for all school events."

The bounded statement follows from a specific count. The broad claim would need far more evidence across the actual intended environment.

Think it through: Which claim is supported by three passing classroom fixtures?

IDEA 2

Record the version and the environment

A reproducible comparison needs the relevant setup: code and prompt versions, fixture revision, judging rules, runtime and package versions, and execution mode. For a live model, also record the model identifier, settings, and trial count. Repeating that setup can still produce different generated answers; reproducibility is not a promise of identical model text.

OpenAI's Cookbook improvement loop preserves traces, feedback, evaluation artifacts, and a proposed change handoff so the next revision has inspectable evidence. For your smaller project, save a manifest and full case results alongside the draft. Missing versions stay null with a limitation. A screenshot or an agent's checkpoint alone does not identify the test data and evaluator that produced a score.

Attach the context of the result

1. Code identity. Identify the program and prompt version evaluated.
2. Fixture and checks. Record dataset revision and exact judging rules.
3. Environment. Record runtime, library, and model configuration if used.
4. Repeat and compare. Keep failures and trial counts, not just a selected demo.

The lab manifest checks two version fields. A real project report needs the additional setup shown here as well.

Missing means missing

const version = manifest.version || null;
if (!manifest.version) limitations.push("missing version");

The null field is machine-readable; the limitation tells a reader why it matters. Neither falsely supplies a value.

Think it through: The run log has no fixture version. What belongs in the report?

IDEA 3

Separate observed counts from broader claims

Observability means collecting information that helps explain execution, such as which tool ran, its result, and whether it failed. This lab sums the toolCalls field in stored runs. That is recordedToolCalls, separate from the two calls the current report-building program makes to read those records.

Call count is not the same as cost, latency, accuracy, or learning value. Different calls can have different prices and runtimes; this fixture contains neither. Present measured quantities with clear names, preserve errors in the pass count, and say what your evaluation did not measure.

Keep measurements in their lane

1. Read. Retrieve the manifest and recorded runs.
2. Aggregate. Sum the recorded call-count field.
3. Label. Avoid turning a count into a dollar or speed claim.

Two current calls read the report data. The total below belongs to the recorded runs.

A contradiction must not create a pass

{ passed: true, error: "timeout", toolCalls: 2 }
// Treat this inconsistent record as failed: the error takes precedence.

A positive flag cannot cancel an execution error. Conservative reporting prevents an inconsistent record from inflating the score.

Think it through: The stored runs total eight tool calls. What can you report from that field alone?

Put it into practice

Summarize recorded fixture results without hiding failures, missing versions, or the simulation boundary.

1. Read the case whose passed flag conflicts with an error.
2. Run the starter and see how the pass count becomes misleading.
3. Insert the error-aware count and run all three reports.
4. Try removing fixtureVersion in the custom manifest and inspect the stated limitation.

Your next experiment: Remove the fixture version, then add a failed run. Does the report preserve both limitations and outcomes?

Show what ran, what passed, what failed, and what remains unknown.

Key terms

Reproducibility

Enough recorded setup and inputs to repeat a test and compare its outcome.

Observability

Execution information that helps explain a system's behavior.

Manifest

A structured record of relevant versions and configuration.

Sources and scope

Original Stemtiq teaching, reviewed 2026-09-14. The named researchers and organizations do not endorse this course. Classroom cases are authored exercises, not published findings.