Pedagogy-Grounded AI Tutors for Interest-Based Learning
A reusable, evidence-based pipeline for prompting LLM tutors to personalize instruction around each learner’s interests and a two-part evaluation of whether the prompts actually change tutor behavior.
Project Overview
Contextualizing instruction around a learner’s interests improves relevance and outcomes, but it doesn’t scale. A human educator can’t tailor every lesson to every interest. LLMs can, in principle, but teams building AI tutors typically hand-write ad-hoc prompts, producing inconsistent, unverifiable behavior. I asked whether a pedagogy-grounded prompt design process could make an LLM tutor reliably and distinctly enact interest-based teaching.
My role. I led the work end-to-end as first author: I built the prompt-design pipeline from a literature synthesis, designed a controlled generic-vs-interest prompt pair, architected the evaluation (simulation and human study), and led the qualitative analysis of tutor behavior.
The impact. The prompts largely produced the behaviors they were designed to elicit - positive enactment outweighed failures roughly 9-to-1 across ~740 coded instances - validating a grounded prompting approach over ad-hoc prompting. Just as valuable, the evaluation pinpointed exactly where prompting alone breaks down (reflection and knowledge-checks), giving a clear guardrail roadmap. The deliverables, a reusable pipeline, prompt templates, and evaluation artifacts, are directly transferable to any team shipping an AI tutor.
The Challenge
Interest-based learning is hard to operationalize in an LLM: how do you get a tutor to weave a learner’s interest into a lesson without hallucinating, going off-curriculum, or reducing it to a superficial gimmick? And how do you prove a prompt is responsible for the behavior you see, rather than the model’s defaults? Teams need a repeatable design method and an evaluation that isolates the prompt’s effect.
Core questions we needed to answer
- What pedagogical strategies should a base tutoring prompt encode, grounded in evidence?
- How do you add interest-integration so the only variable between prompts is the interest layer?
- Do the resulting prompts reliably and distinctly enact the intended teaching behaviors?
- Where does prompting succeed, and where does it fail, across diverse learners and edge cases?
Methodology
I designed the study so the generic and interest-based prompts were identical except for the interest layer - a controlled comparison that lets behavioral differences be attributed to the design. I then evaluated in two complementary passes: simulation for edge cases, human interaction for authenticity.
Evidence-grounded prompt pipeline
Why Ad-hoc prompts aren’t reproducible or defensible. Grounding in documented educator practice makes the design auditable and reusable.
How Ran a targeted literature review (12 qualifying papers) to synthesize a base prompt of five design elements - clarity, scaffolding, reflection, knowledge-checking, and tone - then layered on interest-integration strategies translated into explicit instructions. Added retrieval-augmented context (RAG) to keep content accurate.
Part 1 - Simulation + expert review
Why Human studies can’t cover the space of learner types or the rare edge cases where prompts fail. Synthetic personas stress-test the prompt cheaply and systematically.
How Built synthetic learner personas varying four dimensions (interest detail, interest-topic fit, prior knowledge, engagement), generated tutoring dialogues, and had expert educators rate behavioral fidelity. Their critique drove a targeted prompt revision.
Part 2 - Authentic within-subjects study
Why Only real learners reveal how a tutor handles genuine, unscripted responses. A counterbalanced within-subjects design controls for individual differences.
How Recruited 14 undergraduates who each completed two sessions (generic + interest-based) on self-chosen topics, yielding 28 transcripts. Coded every transcript in MAXQDA for enactment fidelity per pedagogical dimension, with a second coder on a subset for reliability.
Key Insights & Artifacts
Grounding beats intuition: pedagogy-based prompts enacted teaching behaviors reliably.
Across both conditions, tutors consistently decomposed concepts, scaffolded from the learner’s prior knowledge, and maintained instructional flow - clarity and scaffolding were almost entirely positive in the coding. Because the prompts differed only by the interest layer, the interest-based tutor demonstrably added interest anchoring on top of that same solid base rather than trading it away.
Decision This validates investing in a documented, reusable prompt pipeline over per-feature ad-hoc prompting - it yields predictable behavior and a controlled way to A/B a single design variable.
Interest anchoring raised engagement - but degraded when interests were thin.
When a learner gave a rich interest and it fit the topic, the tutor sustained a metaphor across turns (framing Bayes’ theorem through a true-crime podcast, electroplating as an anime "training arc") and reflection was more likely to land. When interests were underspecified or a poor domain fit, the same prompt produced shallow, one-off analogies.
Decision The prompt needs an adaptive depth control and a stronger opening to gather interest context - a concrete instruction-design fix, not a model change. It also defines a clear failure mode to detect and handle in production.
Prompting can’t guarantee deep tutoring - reflection and knowledge-checks are the weak points.
Reflection and knowledge-checking showed markedly higher failure rates in both conditions; learners frequently disengaged from reflective prompts ("answer the reflective question for me") and checks skewed toward shallow recognition. Crucially, my coding measured enactment fidelity (did the tutor do what the prompt said), not learning - a distinction that keeps the claims honest.
Decision These dimensions need interaction-level guardrails (e.g., refusing to hand over answers, adaptive re-prompting) beyond what a system prompt can enforce - pinpointing exactly where engineering effort should go next.
Impact & Learnings
How the team applied the findings
The results converted a generic "make an AI tutor personal" goal into a concrete, prioritized design agenda, and the artifacts were packaged for reuse by other teams building interest-based tutors.
- A reusable, literature-grounded prompt-design pipeline plus generic and interest-based prompt templates.
- Behavioral fidelity roughly 9:1 (positive:negative) across ~740 coded instances - evidence the grounded approach works.
- A precise map of failure modes (shallow analogies under thin interests; weak reflection and knowledge-checks) to target with guardrails.
- A transferable evaluation kit: synthetic-persona stress tests, expert-review rubric, and enactment-fidelity codebook.
What I learned
The most important discipline was separating "did the tutor do what we asked" from "did the learner learn." It’s tempting to read fluent, on-interest dialogue as effective teaching; forcing an enactment-fidelity lens kept the findings credible and showed where prompting runs out of road. Pairing simulation with a small authentic study also gave both breadth and realism without an expensive large trial.
What I'd do differently
I’d add a learning-outcome measure (pre/post concept checks) so I could connect behavioral fidelity to actual understanding, and I’d expand the human sample and topic range to test whether the interest layer helps or distracts for learners who arrive with a firm objective and no desire to chat about a hobby.