Use these notes to review a concept or read at your own pace. The interactive workspace above adds predictions, editable code, and actual run results.
IDEA 1
A candidate file needs evidence
An issue report describes a symptom, not a proven cause. A repository manifest lists allowed files and their components. Matching the issue's component narrows the search, but a file remains a candidate until its contents and tests have been inspected.
This exercise uses a tiny fictional club repository. readFile returns numbered source lines; listTests returns existing test names. The sorter may cite those observations, but it cannot claim a fix because it neither changes code nor executes those repository tests.
From symptom to an inspectable candidate
- Issue. The report identifies the tickets component.
- Manifest. One allowed file matches that component.
- Read. A numbered line makes the observation inspectable.
- Test name. Select a test that actually appears in the returned list.
Authored repository records; source text is read-only data in this lab.
Keep the evidence attached
const file = tools.readFile(candidate.path);
const tests = tools.listTests(candidate.path);
const evidence = [file.lines[0]];The candidate path comes from the allowed manifest. The line and test are observations from separate tools; inventing a plausible test title would not be equivalent.
Think it through: What does finding a relevant source line establish?
IDEA 2
Ambiguity and budgets are part of the output
If multiple allowed files share a component, this baseline asks for more information instead of guessing. A useful clarification might be the page where the symptom occurs or a minimal example. Clear uncertainty prevents an arbitrary file from looking authoritative.
The normal route uses four calls: list files, read one file, list its tests, and check the assembled result. If the declared budget is smaller, the sorter returns a partial result containing only evidence already observed. A budget limit should not trigger invented test names.
Spend calls deliberately
- Call 1. List allowed files and establish whether the component is unique.
- Call 2. Read the candidate's actual lines.
- Call 3. Read existing tests only if the budget permits.
- Call 4. Check the assembled evidence and test name.
The four-call budget is a classroom constraint, not a vendor performance claim.
A partial answer should say what is missing
return { status: "partial", issueId: input.issue.id, file: path, test: null, evidence };The file and source evidence can be retained after two calls. test remains null because the sorter has not yet observed the test list.
Think it through: Two files match the component equally well. What should this baseline return?
Put it into practice
Return a checked candidate, a needs-info result, or an honest partial result within the call budget.
- Read the issue component and the permitted manifest.
- Run the starter: the assembled candidate omits an existing test name.
- Insert the first returned test name, leaving null when the list is empty.
- Compare the ambiguous and two-call-budget cases.
- Inspect the result checker; it validates actual paths, test names, and evidence text.
Your next experiment: Empty the tests array. Why should the checker change the result to needs-review?
A useful issue route cites a real allowed file and test, reports uncertainty, and never calls a candidate a fix.
Key terms
- Manifest
- A declared list of files and metadata the exercise permits the sorter to inspect.
- Candidate
- A plausible item for investigation, not yet a proven cause or a completed repair.
Sources and scope
Original Stemtiq teaching, reviewed 2026-09-14. The named researchers and organizations do not endorse this course. Classroom cases are authored exercises, not published findings.
Anthropic: Writing effective tools for agents — with agents
11 September 2025
Design distinct tools with clear parameters, relevant returned information, and evaluations of how the agent actually uses them.
A description or schema does not guarantee the right action. A live tool can return different data for the same arguments as its environment changes.
Meta research team: The Llama 3 Herd of Models
31 July 2024 · revised 23 November 2024
Tool definitions and descriptions guide proposed calls; executed results return to model context. The report covers sequential, nested, and parallel function calls.
This historical model-training report is not a current SDK contract. Generating a call does not execute or authorize it, and benchmark results do not describe classroom performance.
Jimenez et al. · SWE-bench research team: SWE-bench: Can Language Models Resolve Real-World GitHub Issues?
ICLR 2024 · original benchmark
Evaluate patches against repository issues and executable tests using a reproducible harness. Inspect the actual code change and its tested behavior.
Benchmark variants cover different tasks. Passing a repair case does not establish general coding ability or that a patch meets every unstated requirement.