Learning Calibration Audit

Example 1: Law Student Before Bar Exam

Context: A law student feels confident about contract law — they have read the chapter three times and their notes are thoroughly highlighted. But a practice bar question on contracts leaves them unable to identify the correct legal standard.

Step 1: Unreliable signals: rereading fluency, highlighting coverage. Reliable signals absent: no delayed free recall, no novel problem practice.

Step 2: Fluency illusion (primary — rereading fluency mistaken for retrievable doctrine); Imagination inflation (studying by mentally rehearsing answers without actually producing them in writing).

Step 3: Self-quizzing (closed-book): Write out the elements of each contract doctrine from memory before opening notes. Pre-answer confidence logging: before each practice question, rate confidence 1-5 and record. Compare to actual score.

Step 4: Cycle 1 baseline — 15-question practice exam on contracts, closed-book, confidence logged. Gap analysis — which doctrine elements are encoding gaps vs. retrieval gaps. Cycle 1 retest — 48 hours later, different question set, same doctrines. Continue until confidence ratings and scores align within 10 points.

Example 2: Corporate Training Cohort

Context: A trainer observes that trainees perform well during the training day (completing exercises correctly with the reference guide available) but report confusion when applying procedures in the field.

Step 1: Unreliable signals in use: passive recognition (exercises done with reference guide open), immediate recall (end-of-day quiz administered right after instruction). No delayed free recall, no novel application, no peer teaching.

Step 2: Fluency illusion (guide availability masks retrieval difficulty); Dunning-Kruger overconfidence (trainees in early learning stage overrate competence); False consensus (trainees assume peers' silence means shared understanding).

Step 3: Cumulative quizzing — weekly quiz reaching back across all procedures covered. Peer instruction — before demonstrations, trainees write their expected procedure independently, then compare with a partner who differed. Peer teaching — in pairs, trainees teach a procedure without the guide; the partner signals confusion points.

Step 4: Cycle 1 — closed-book procedural quiz one week post-training. Gap analysis — which procedures have encoding vs. retrieval vs. transfer gaps. Targeted practice — additional drills on gap procedures. Retest — same procedures in a novel scenario context two weeks later.

Example 3: Self-Study Language Learner

Context: A language learner reports that vocabulary "feels solid" — recognition from flashcard reviews is high and they can follow podcasts well. But speaking and writing attempts reveal many words are not available for production.

Step 1: Unreliable signals: familiarity warmth (flashcard recognition), passive recognition (following podcasts). Reliable signals absent: no production-side practice, no delayed free recall, no peer teaching in the target language.

Step 2: Fluency illusion (recognition mistaken for production-ready knowledge); Social memory contamination (if studying in groups, may have absorbed peer corrections as own knowledge without independent consolidation).

Step 3: Closed-book production quizzing — cover the target word; write or speak only from the definition cue. Peer instruction — set a conversation exchange where neither participant can revert to their native language for a defined vocabulary set; communication breakdowns are calibration data.