October workshops are openBuild a Search AI Agent$99 early bird

Enroll now
Skip to content

SHORT COURSE 6 OF 6 · ADVANCED · 4 GUIDED LESSONS

Make a project you can explain.

Start from one person's problem. Build a complete small workflow, compare authored run records under the same rules, and present the evidence behind your claims. Then apply these habits to a project in your chosen major.

Can you show what your agent does, where it fails, and why its design fits your chosen field?

THE BUILD YOU ARE WORKING TOWARD

A scoped project brief, a runnable catalog-backed preview, a comparison of authored run records, and an honest project report you can download with your notebook.

  • Define the user, allowed actions, success criteria, and a simple baseline.
  • Keep missing evidence and failed execution visible when comparing versions.
  • Document the code, environment, checks, results, and remaining limits.
0 / 4 lessons complete
Start this courseChecking your saved progress…

Before you start

Complete the shared skills; choose a project scope that fits the available environment.

Each lesson follows the same rhythm: understand three small ideas, edit and run the code, then explain the result. Take one lesson at a time.

In your browser, JavaScript and local tools really execute against inspectable inputs. Model decisions and classroom records are authored examples. The optional SDK project runs separately on your computer with its own setup.

Your lesson sequence

  1. 1Learn → build → reflectChoose a person and a small problemTurn an interest-pathway idea into a bounded task with a user, permitted inputs, and measurable success.
  2. 2Learn → build → reflectBuild the smallest complete loopConnect the input, model or explicit workflow, allowed tools, evidence, and output into a reproducible prototype.
  3. 3Learn → build → reflectTry to break your own resultCompare the prototype and baseline on held-out cases, misleading input, missing evidence, and resource failures.
  4. 4Learn → build → reflectTell the story of what you builtDemonstrate the agent, justify its architecture, and explain the sources, checks, observed resource use, and limits. Actual API cost needs real usage records.

CONNECT THE FOUR LESSONS

Your course checkpoint

Give a candidate version one improvement and one regression. In the report exercise, remove a version field and keep an errored run in the input.

  • Show why equal total scores can hide a new failure.
  • Keep errors in the attempted-case count and missing metadata explicit.
  • Use your notebook to explain one success, one failure, and the next experiment for a major-specific project.

Use these lessons’ experiment inputs and lab notebooks to record your evidence. This is a self-review prompt; only the lesson questions and code cases contribute to the progress shown above.

Record and download your findings
Research into practiceThe ideas behind this short course

Original teaching informed by primary research and official engineering guidance. Our exercises isolate mechanisms so you can test them; they do not reproduce a research system or its reported performance.

Anthropic · 9 January 2026

Demystifying evals for AI agents

Define tasks, trials, and graders; inspect both execution records and final outcomes; repeat trials when model behavior varies.

A score depends on its cases and grading rules. Repeating a deterministic classroom case does not measure the variability of a live model.

Jimenez et al. · SWE-bench research team · ICLR 2024 · original benchmark

SWE-bench: Can Language Models Resolve Real-World GitHub Issues?

Evaluate patches against repository issues and executable tests using a reproducible harness. Inspect the actual code change and its tested behavior.

Benchmark variants cover different tasks. Passing a repair case does not establish general coding ability or that a patch meets every unstated requirement.

Zhou et al. · Carnegie Mellon University · 2023 preprint · ICLR 2024

WebArena: A Realistic Web Environment for Building Autonomous Agents

Evaluate browser agents in reproducible website environments, checking functional outcomes rather than requiring one exact sequence of actions.

This research benchmark does not represent every website or permission boundary. Classroom examples are not WebArena results, and historical model scores are not current rankings.

NVIDIA · Version 1.8 observed · reviewed 13 September 2026

Agent Evaluation in NVIDIA NeMo Agent Toolkit

Run curated cases, inspect generated answers and intermediate steps, retain effective configurations, and enable profiling when operational measurements are needed.

Different artifacts require different configuration. Scores depend on cases and graders; timing measurements do not establish correctness, and model-based grading is fallible.

AWS · Amazon Bedrock AgentCore · Reviewed 13 September 2026 · undated documentation

Observe your agent applications on Amazon Bedrock AgentCore Observability

Use instrumented traces and operational measurements to inspect execution, intermediate outputs, duration, errors, and reported model usage.

Monitoring does not enforce a budget or make an answer correct. Tool calls, tokens, elapsed time, and money are different quantities; simulated models provide no real token bill.

Your JavaScript really runs. The model decisions and school data are authored simulations, so you can learn without an API key. Every workspace also includes a separate real SDK example to explore next. Passing the lab’s cases is practice, not proof that an agent is ready for real-world use.