A tiny source tree, an issue, allowed edit paths, public and withheld fixture tests, execution limits, and reviewer approval.
What your agent must check
Show the original failure and verified improvement before saving a preview.
Reject a constant answer despite its approval fixture.
Keep test acceptance separate from action permission.
Reject an out-of-scope edit before executing the proposal.
Keep the scope clear
Execution stays in the lab's isolated runtime. No push, deployment, package install, or credentials. Passing the fixture tests does not prove general correctness.
Research behind this project path
Original explanations and authored practice records draw on these research and engineering ideas. The source organizations do not endorse this course or supply its fictional results.
Jimenez et al. · SWE-bench research team · SWE-bench: Can Language Models Resolve Real-World GitHub Issues?
Evaluate patches against repository issues and executable tests using a reproducible harness. Inspect the actual code change and its tested behavior.
Benchmark variants cover different tasks. Passing a repair case does not establish general coding ability or that a patch meets every unstated requirement.
Anthropic · Demystifying evals for AI agents
Define tasks, trials, and graders; inspect both execution records and final outcomes; repeat trials when model behavior varies.
A score depends on its cases and grading rules. Repeating a deterministic classroom case does not measure the variability of a live model.
OpenAI · Guardrails and human review
Distinguish automatic checks from approval decisions, pause sensitive tool requests, retain state, and resume after an application approves or rejects them.
Model-generated approval text is not authorization. Resume examples that automatically approve a request do not establish that a person reviewed it.
Your JavaScript really runs. The model decisions and school data are authored simulations, so you can learn without an API key. Every workspace also includes a separate real SDK example to explore next. Passing the lab’s cases is practice, not proof that an agent is ready for real-world use.