October workshops are openBuild a Search AI Agent$99 early bird

Enroll now
Skip to content

SHORT COURSE 4 OF 6 · INTERMEDIATE · 4 GUIDED LESSONS

Test, debug, and control the work.

Useful behavior includes stopping at the right time. Bind approval to a particular action, write checks that can reject a confident answer, handle failed reads, and inspect repeated work in the actual tool trace.

What happens when a tool fails, a result is wrong, or an action needs permission?

THE BUILD YOU ARE WORKING TOWARD

An approval gate, a behavior checker, a bounded retry policy, and a per-run cache with observable call counts.

  • Require approval for the exact proposal being acted on.
  • Check evidence and outcomes independently of a proposed confidence score.
  • Distinguish temporary failure, an unsuitable result, and repeated work.
0 / 4 lessons complete
Start this courseChecking your saved progress…

Before you start

Use sourced evidence, explicit unknowns, and a controlled tool loop.

Each lesson follows the same rhythm: understand three small ideas, edit and run the code, then explain the result. Take one lesson at a time.

In your browser, JavaScript and local tools really execute against inspectable inputs. Model decisions and classroom records are authored examples. The optional SDK project runs separately on your computer with its own setup.

Your lesson sequence

  1. 1Learn → build → reflectAsk before changing somethingApply least privilege and check explicit approval at the tool that would cause a side effect.
  2. 2Learn → build → reflectTest behavior, not confidenceBuild representative and held-out cases; distinguish code checks, human review, and fallible model graders.
  3. 3Learn → build → reflectFail clearly and recover carefullyClassify errors, bound retries, and distinguish a retryable read from a potentially duplicated write.
  4. 4Learn → build → reflectSee where time and calls wentCompare correctness, duration, and call counts; use token and cost metrics only when a real runtime reports them.

CONNECT THE FOUR LESSONS

Your course checkpoint

Compare requests [A, A, C, A] with [A, C]. Keep the catalog unchanged and inspect the output and trace of each run.

  • Explain why the output lengths differ while both need only two distinct reads.
  • Show a changed proposal that requires a fresh review.
  • Explain which failed status can be retried and where retries stop.

Use these lessons’ experiment inputs and lab notebooks to record your evidence. This is a self-review prompt; only the lesson questions and code cases contribute to the progress shown above.

Record and download your findings
Research into practiceThe ideas behind this short course

Original teaching informed by primary research and official engineering guidance. Our exercises isolate mechanisms so you can test them; they do not reproduce a research system or its reported performance.

Anthropic · 9 January 2026

Demystifying evals for AI agents

Define tasks, trials, and graders; inspect both execution records and final outcomes; repeat trials when model behavior varies.

A score depends on its cases and grading rules. Repeating a deterministic classroom case does not measure the variability of a live model.

OpenAI · Reviewed 13 September 2026 · undated documentation

Guardrails and human review

Distinguish automatic checks from approval decisions, pause sensitive tool requests, retain state, and resume after an application approves or rejects them.

Model-generated approval text is not authorization. Resume examples that automatically approve a request do not establish that a person reviewed it.

Amazon Builders’ Library · Marc Brooker · PDF copyright 2019 · reviewed 13 September 2026

Timeouts, retries, and backoff with jitter

Handle transient failures with timeouts and bounded retries, consider backoff and jitter, and establish whether repeating an operation is safe.

Retries can amplify overload or duplicate side effects. Three immediate classroom attempts are an exercise rule, not a universal policy for real services.

Amazon Builders’ Library · Reviewed 13 September 2026 · undated guidance

Caching challenges and strategies

Reusing observations can reduce dependency calls; cache scope, freshness, and failure behavior need deliberate design.

A room ID is a sufficient classroom cache key only while the task and room facts remain unchanged. Caching does not itself refresh stale information.

AWS · Amazon Bedrock AgentCore · Reviewed 13 September 2026 · undated documentation

Observe your agent applications on Amazon Bedrock AgentCore Observability

Use instrumented traces and operational measurements to inspect execution, intermediate outputs, duration, errors, and reported model usage.

Monitoring does not enforce a budget or make an answer correct. Tool calls, tokens, elapsed time, and money are different quantities; simulated models provide no real token bill.

Your JavaScript really runs. The model decisions and school data are authored simulations, so you can learn without an API key. Every workspace also includes a separate real SDK example to explore next. Passing the lab’s cases is practice, not proof that an agent is ready for real-world use.