Module 3: APPLYING IRT IN PRACTICE

Author

LASER Institute

Published

May 29, 2025

APPLYING IRT IN PRACTICE

LASER INSTITUTE KT TRACK BADGE ACTIVITY

Please write a short essay in response to the following question. How could you apply IRT-based knowledge tracing to solve (just one) real problem in your research or in practice, that Bayesian Knowledge Tracing (BKT) wouldn’t be as good for?

What specific learning domain or context are you addressing?
What are the key limitations of BKT for this particular problem?
Which IRT model (1PL, 2PL, 3PL, multidimensional) would be most appropriate and why?
What specific item parameters (difficulty, discrimination, guessing) are crucial for your application?
How will the increased precision of IRT benefit learners or instructors in this context?

Input that essay to ChatGPT 4 or another Large Language Model of your preference, following the initial prompt:

You are a researcher with extremely detailed knowledge of item response theory (IRT) and IRT-based knowledge tracing and an expert on its use within research and in intelligent tutoring systems. Please give feedback on the following essay written by an advanced doctoral student. Please give them feedback on the core ideas and their application on the research or real-world problem; do not give suggestions for an academic paper such as a recommendation to add citations. Please do not give recommendations around functionality that does not exist in IRT (e.g. engagement tracking), and do not make recommendations that apply to every knowledge tracing (e.g. paying attention to student privacy). Give only constructive, negative feedback:

What technical aspects of IRT did the researcher question or correct?
Did they identify any misconceptions about IRT model assumptions or capabilities?
What gaps in your IRT knowledge were revealed?
How accurate was your comparison between IRT and BKT advantages?

Input your modified essay to ChatGPT 4 or another Large Language Model of your preference, following the initial prompt:

You are an award-winning teacher who has developed curricula which leverage item response theory (IRT) and IRT-based knowledge tracing and has spent thousands of hours watching and interacting with students using those curricula. Please give feedback on the following essay written by an advanced doctoral student. Please give them feedback on the core ideas and their application on the research or real-world problem; do not give suggestions for an academic paper such as a recommendation to add citations. Do not make scientifically inaccurate recommendations, such as recommending paying attention to learning styles. Give only constructive, negative feedback:

What implementation challenges or practical constraints did the teacher identify?
How realistic are your assumptions about classroom or real-world use?
What student behaviors or learning patterns did you not adequately consider?
Did the feedback suggest your approach might be too complex for actual deployment?

Input your modified essay to ChatGPT 4 or another Large Language Model of your preference, following the initial prompt:

You are Maria, a thoughtful and highly intelligent 9th grade student who nonetheless sometimes forgets what she learned in class. You have just spent the last three weeks using a learning system in class, described by the following prompt. You have been asked to give your reflections on that learning system and how to make it better. Do not make scientifically inaccurate recommendations, such as recommending paying attention to learning styles. You say:

(If this prompt does not fit your context of use, please modify it to make it fit!)

What aspects of the user experience did the student highlight as problematic?
How does the student’s perspective differ from your adult/expert assumptions?
What motivational, engagement, or usability issues were raised?
Did the student identify sources of confusion or frustration you hadn’t considered?