Study-app experiment auditor

IDEA 1

Compare change as well as the final score

A baseline is the measurement before the activity; a follow-up is the measurement afterward. In an invented study-app trial, group A goes from 70 to 80 and group B goes from 50 to 65. A finishes 15 points higher, but its observed change is 10 points compared with B's 15. The two comparisons answer different questions.

These arithmetic differences describe the supplied group means. They are not a significance test or a causal effect estimate. For a clear teaching example, the pre and post means describe the same recorded completers within each group. When some participants' final outcomes are missing, those means need additional scrutiny. All records and scores here are invented; they say nothing about real apps, grades, or student ability.

Starting point changes the story

1. Baseline. A starts at 70; B starts at 50.
2. Follow-up. A ends at 80; B ends at 65.
3. Changes. A changes by 10; B by 15. The difference in changes is −5.

Fictional records for learning. Follow the evidence and the checks.

Work through the example

const changeA = groupA.post - groupA.pre;
const changeB = groupB.post - groupB.pre;
const changeDifference = changeA - changeB;

Subtract baseline within each group before comparing the changes. This calculation exposes a different descriptive pattern; it does not by itself eliminate other explanations.

Think it through: A: 70→80; B: 50→65. Which statement correctly distinguishes the calculations?

IDEA 2

Inspect assignment and missing outcomes

How groups were formed affects which explanations remain plausible. Self-selected users may differ before an activity begins. Being assigned to groups is not automatically random assignment: a teacher assigning by preference or alternating names is a different process. Random assignment can support a stronger comparison, but one label does not settle all questions about execution or analysis.

Check who was counted at the start and whose final outcome is missing. A missing dropout count is not zero. Unequal starting values and missing outcomes are review prompts in this fixture, not an automatic verdict that a study is invalid. NHLBI's methods guidance treats these as questions to interpret in context; our program does not reproduce a complete study-quality appraisal.

Method details are evidence

1. Assignment. The record names the actual grouping method.
2. Starting values. Different baselines complicate a post-only story.
3. Missing outcomes. Null means the dropout count was not supplied.

Fictional records for learning. Follow the evidence and the checks.

Work through the example

if (trial.assignment !== "randomized") issues.push("assignment_not_randomized");
if (trial.groups.some(g => g.dropouts === null)) issues.push("dropout_unknown");

These visible rules flag questions for review. They do not estimate bias, significance, or a probability that a conclusion is true.

Think it through: A record says participants were assigned by their preferred meeting day. Is that the same as random assignment?

IDEA 3

Produce a reproducible claim audit

A good audit separates observed calculations from issues requiring interpretation. This helper returns both final-score difference and change difference, the assignment method, and named review flags. It also checks the draft's sample-size claim against the enrolled counts and rejects an absent source ID. A confident sentence cannot supply a missing record.

The checker is a transparent set of fixture rules: unequal pre values trigger different_baselines, null dropout fields trigger dropout_unknown, and every causal claim goes to methods review. Even a clean descriptive case returns causalProof: false. Saving the review draft means returning a local object with evidence and open questions, not publishing a finding or certifying an app's effectiveness.

Keep three layers separate

1. Observation. Show the arithmetic and original source ID.
2. Audit. List exactly which fixture checks need attention.
3. Review draft. Preserve limits and avoid a causal certificate.

Fictional records for learning. Follow the evidence and the checks.

Work through the example

return tools.saveReviewDraft({
  sourceId: trial.id, changes, changeDifference,
  issues: audit.issues, causalProof: false
});

A local draft gives another learner enough information to reproduce the calculations and discuss the flagged questions.

Think it through: The record is randomized, has no missing outcomes, and matches the descriptive sample-size claim. What does this limited checker establish?

Put it into practice

Compute two different comparisons, preserve method concerns, and return a local review draft with its source.

1. Inspect the fictional input and the selected case's expected result.
2. Run the starter. Locate its missing rule using the output and actual tool trace.
3. Insert the explained snippet at the TODO and rerun the same case.
4. Run all three cases, then change the experiment input and explain what the evidence now supports.

Your next experiment: The draft cites T9, but the only available record is T1. Predict which calculation and save tools must remain uncalled.

Check the starting point, grouping method, missing outcomes, and source before turning a numerical difference into a story.

Key terms

Baseline

A measurement made before the activity being compared.

Random assignment

Allocation to groups through a chance-based process rather than preference or another systematic rule.

Missing outcome

A final measurement not observed for someone included at the start.

Sources and scope

Original Stemtiq teaching, reviewed 2026-09-14. The named researchers and organizations do not endorse this course. Classroom cases are authored exercises, not published findings.