October workshops are openBuild a Search AI Agent$99 early bird

Enroll now
Skip to content

PATHWAY PROJECT · STARTER PROJECT · 2 SMALL IDEAS + A GUIDED LAB

Coding-club issue sorter

An issue says ‘the total is wrong.’ Can you point to a file and a real test without pretending the bug is fixed?

You will learn to: Route a fictional issue using permitted file evidence, keep ambiguity visible, and stop within a call budget.

Preparing your lesson and this browser’s progress…
Read the complete lessonAll the ideas in one place · works without the editor

Use these notes to review a concept or read at your own pace. The interactive workspace above adds predictions, editable code, and actual run results.

IDEA 1

A candidate file needs evidence

An issue report describes a symptom, not a proven cause. A repository manifest lists allowed files and their components. Matching the issue's component narrows the search, but a file remains a candidate until its contents and tests have been inspected.

This exercise uses a tiny fictional club repository. readFile returns numbered source lines; listTests returns existing test names. The sorter may cite those observations, but it cannot claim a fix because it neither changes code nor executes those repository tests.

From symptom to an inspectable candidate

  1. Issue. The report identifies the tickets component.
  2. Manifest. One allowed file matches that component.
  3. Read. A numbered line makes the observation inspectable.
  4. Test name. Select a test that actually appears in the returned list.

Authored repository records; source text is read-only data in this lab.

Keep the evidence attached

const file = tools.readFile(candidate.path);
const tests = tools.listTests(candidate.path);
const evidence = [file.lines[0]];

The candidate path comes from the allowed manifest. The line and test are observations from separate tools; inventing a plausible test title would not be equivalent.

Think it through: What does finding a relevant source line establish?

IDEA 2

Ambiguity and budgets are part of the output

If multiple allowed files share a component, this baseline asks for more information instead of guessing. A useful clarification might be the page where the symptom occurs or a minimal example. Clear uncertainty prevents an arbitrary file from looking authoritative.

The normal route uses four calls: list files, read one file, list its tests, and check the assembled result. If the declared budget is smaller, the sorter returns a partial result containing only evidence already observed. A budget limit should not trigger invented test names.

Spend calls deliberately

  1. Call 1. List allowed files and establish whether the component is unique.
  2. Call 2. Read the candidate's actual lines.
  3. Call 3. Read existing tests only if the budget permits.
  4. Call 4. Check the assembled evidence and test name.

The four-call budget is a classroom constraint, not a vendor performance claim.

A partial answer should say what is missing

return { status: "partial", issueId: input.issue.id, file: path, test: null, evidence };

The file and source evidence can be retained after two calls. test remains null because the sorter has not yet observed the test list.

Think it through: Two files match the component equally well. What should this baseline return?

Put it into practice

Return a checked candidate, a needs-info result, or an honest partial result within the call budget.

  1. Read the issue component and the permitted manifest.
  2. Run the starter: the assembled candidate omits an existing test name.
  3. Insert the first returned test name, leaving null when the list is empty.
  4. Compare the ambiguous and two-call-budget cases.
  5. Inspect the result checker; it validates actual paths, test names, and evidence text.

Your next experiment: Empty the tests array. Why should the checker change the result to needs-review?

A useful issue route cites a real allowed file and test, reports uncertainty, and never calls a candidate a fix.

Key terms

Manifest
A declared list of files and metadata the exercise permits the sorter to inspect.
Candidate
A plausible item for investigation, not yet a proven cause or a completed repair.

Sources and scope

Original Stemtiq teaching, reviewed 2026-09-14. The named researchers and organizations do not endorse this course. Classroom cases are authored exercises, not published findings.

Anthropic: Writing effective tools for agents — with agents

11 September 2025

Design distinct tools with clear parameters, relevant returned information, and evaluations of how the agent actually uses them.

A description or schema does not guarantee the right action. A live tool can return different data for the same arguments as its environment changes.

Meta research team: The Llama 3 Herd of Models

31 July 2024 · revised 23 November 2024

Tool definitions and descriptions guide proposed calls; executed results return to model context. The report covers sequential, nested, and parallel function calls.

This historical model-training report is not a current SDK contract. Generating a call does not execute or authorize it, and benchmark results do not describe classroom performance.

Jimenez et al. · SWE-bench research team: SWE-bench: Can Language Models Resolve Real-World GitHub Issues?

ICLR 2024 · original benchmark

Evaluate patches against repository issues and executable tests using a reproducible harness. Inspect the actual code change and its tested behavior.

Benchmark variants cover different tasks. Passing a repair case does not establish general coding ability or that a patch meets every unstated requirement.

Your JavaScript really runs. The model decisions and school data are authored simulations, so you can learn without an API key. Every workspace also includes a separate real SDK example to explore next. Passing the lab’s cases is practice, not proof that an agent is ready for real-world use.