Simulated Student Behaviors with Intelligent Tutoring Systems: Applications for Authoring and Evaluating Network-Based Tutors
Eric G. Poitras, Zachary Mayne, Lingyun Huang, Tenzin Doleck, Laurel Udy and Susanne P. Lajoie
Corresponding author: firstname.lastname@example.org
This chapter reviews contemporary research on the use of simulated learners as a methodological approach employed to evaluate theoretical and computational frameworks in the context of intelligent tutoring systems. This modeling approach simulates human students as they learn under controlled conditions, enabling system designers and users to manipulate them and observe the effects. During the last few decades, the method has grown in popularity and led to several practical applications, including development of learning theories, formative evaluations of alternative instructional approaches, and software agents that serve instructional purposes. Drawing upon our own research on the topic, we apply this method in the context of nSimulator as a means to simulate behaviors observed in actual human learners and improve instructional features embedded in network-based tutors. We discuss the strengths and weaknesses for simulated learners as a methodology to test and develop better tutoring systems.
Keywords: simulated learners, network-based tutors, nSimulator
APA citation information
Poitras, E. G., Mayne, Z., Huang, L., Doleck, T., Udy, L., & Lajoie, S. P. (2018). Simulated student behaviors with intelligent tutoring systems: Applications for authoring and evaluating network-based tutors. In S. D. Craig (Ed.). Tutoring and Intelligent Tutoring Systems (pp. 273-298). New York, NY: Nova Science Publishers.
Anderson, J. R., Boyle, C. F., & Reiser, B. J. (1985). Intelligent tutoring systems. Science, 228(4698), 456-462.
Anderson, J. R., Corbett, A. T., Koedinger, K. R., & Pelletier, R. (1995). Cognitive tutors: Lessons learned. The Journal of the Learning Sciences, 4(2), 167-207.
Anderson, T., & Whitelock, D. (2004). The educational semantic web: Visioning and practicing the future of education. Journal of Interactive Media in Education, 2004(1). DOI: http://doi.org/10.5334/2004-1
Beck, J., Woolf, B. P., & Beal, C. R. (2000). ADVISOR: A machine learning architecture for intelligent tutor construction. AAAI/IAAI, 2000(552-557), 1-1.
Biswas, G., Jeong, H., Kinnebrew, J. S., Sulcer, B., & Roscoe, R. (2010). Measuring self-regulated learning skills through social interactions in a teachable agent environment. Research and Practice in Technology Enhanced Learning, 5, 123-152.
Biswas, G., Leelawong, K., Schwartz, D. & Vye, N. (2005). Learning by teaching: A new agent paradigm for educational software. Applied Artificial Intelligence, 19, 363-392.
Champaign, J., & Cohen, R. (2010, June). Peer-Based intelligent tutoring systems: A corpus-oriented approach. In V. Aleven, J. Kay, & J. Mostow (Eds.). International Conference on Intelligent Tutoring Systems (pp. 212-214). Springer, Berlin, Heidelberg.
Dieker, L. A., Hughes, C. E., Hynes, M. C. & Straub, C. (2017). Using simulated virtual environments to improve teacher performance. School University Partnerships (Journal of the National Association for Professional Development Schools): Special Issue: Technology to Enhance PDS, 10(3), 62-81.
Doroudi, S., Aleven, V. & Brunskill, E. (2017, April). Robust evaluation matrix: Towards a more principled offline exploration of instructional policies. In Proceedings of the Fourth (2017) ACM Conference on Learning@ Scale (pp. 3-12). ACM.
Erickson, G., Frost, S., Bateman, S. & McCalla, G. (2013). Using the ecological approach to create simulations of learning environments. In International Conference on Artificial Intelligence in Education (pp. 411-420). Springer, Berlin, Heidelberg.
Fiorella, L. & Mayer, R. E. (2014). Role of expectations and explanations in learning by teaching. Contemporary Educational Psychology., 39, 75-85.
Graesser, A. C., Conley, M. W. & Olney, A. (2012). Intelligent tutoring systems. In K. R. Harris, S. Graham, T. Urdan, A. G. Bus, S. Major, & H. L. Swanson (Eds.), APA educational psychology handbook, Vol. 3. Application to learning and teaching (pp. 451-473). Washington, DC, US: American Psychological Association.
Greer, J. & Mark, M. (2015). Evaluation Methods for Intelligent Tutoring Systems Revisited. International Journal of Artificial Intelligence in Education, 389-390.
Kay, J. & McCalla, G. (2012). Coming of age: Celebrating a quarter century of user modeling and personalization: Guest editors’ introduction. User Modeling and User-Adapted Interaction, 22(1), 1-7.
Koedinger, K. R., Aleven, V., Roll, I., & Baker, R. (2009). In vivo experiments on whether supporting metacognition in intelligent tutoring systems yields robust learning. Handbook of Metacognition in Education, 897-964.
Koedinger, K. R., Matsuda, N., MacLellan, C. J., & McLaughlin, E. A. (2015). Methods for evaluating simulated learners: Examples from SimStudent. In AIED Workshops, 45-54.
Koper, R. (2004). Use of the semantic web to solve some basic problems in education: Increase flexible, distributed lifelong learning; decrease teacher’s workload. Journal of Interactive Media in Education, 2004(1).
Kulik, J. A. & Fletcher, J. D. (2015). Effectiveness of Intelligent Tutoring Systems: A meta-analytic review. Review of Educational Research, 85, 171-204.
Lajoie, S. P., & Poitras, E. G. (2017). Crossing disciplinary boundaries to improve technology-rich learning environments. Teachers College Record, 19(3), 1-30.
Lelei, D. E. K., & McCalla, G. I. (2015). Exploring the issues in simulating a semi-structured learning environment: The SimGrad doctoral program design. In AIED Workshops, 11-20.
Ma, W., Adesope, O. O., Nesbit, J. C., & Liu, Q. (2014). Intelligent tutoring systems and learning outcomes: A meta-analysis. Journal of Educational Psychology, 106(4), 901.
Matsuda, N., Cohen, W. W., Sewall, J., Lacerda, G. & Koedinger, K. R. (2007, July). Evaluating a Simulated Student Using Real Students Data for Training and Testing. In International Conference on User Modeling (pp. 107-116). Springer Berlin Heidelberg.
Matsuda, N., Cohen, W. W. & Koedinger, K. R. (2015). Teaching the teacher: tutoring SimStudent leads to more effective cognitive tutor authoring. International Journal of Artificial Intelligence in Education, 25(1), 1-34.
Matsuda, N., Cohen, W. W., Sewall, J., Lacerda, G., & Koedinger, K. R. (2007).
Evaluating a simulated student using real students data for training and testing. In C. Conati, K. McCoy & G. Paliouras (Eds.), Proceedings of the International Conference on User Modeling (LNAI 4511) (pp. 107-116). Springer, Berlin, Heidelberg.
Matsuda, N., Keiser, V., Raizada, R., Tu, A., Stylianides, G., Cohen, W. W., & Koedinger, K. R. (2010, June). Learning by teaching SimStudent: Technical accomplishments and an initial use with students. In International Conference on Intelligent Tutoring Systems (pp. 317-326). Springer, Berlin, Heidelberg.
McCalla, G. (2004). The ecological approach to the design of e-learning environments: Purpose-based capture and use of information about learners. Journal of Interactive Media in Education, 2004(1), DOI: http://doi.org/10.5334/2004-7-mccalla
McCalla, G. & Champaign, J. (2013). Simulated Learners. IEEE Intelligent Systems, 28(4), 67-71.
Murray, T. (1999). Authoring intelligent tutoring systems: An analysis of the state of the art. International Journal of Artificial Intelligence in Education (IJAIED), 10, 98-129.
Nye, B. D., Graesser, A. C., & Hu, X. (2014). AutoTutor and family: A review of 17 years of natural language tutoring. International Journal of Artificial Intelligence in Education, 24(4), 427-469.
Ohlsson, S., Ernst, A. M. & Rees, E. (1992). The cognitive complexity of learning and doing arithmetic. Journal for Research in Mathematics Education, 23, 441-467.
Poitras, E., Doleck, T., Huang, L., Li, S., & Lajoie, S. (2017). Advancing teacher technology education using open-ended learning environments as research and training platforms. Australasian Journal of Educational Technology, 33(3), 32-45.
Poitras, E. & Fazeli, N. (2016, March). Using an Intelligent Web Browser for Teacher Professional Development: Preliminary Findings from Simulated Learners. In Society for Information Technology & Teacher Education International Conference (pp. 3037-3041). Association for the Advancement of Computing in Education (AACE).
Poitras, E. & Fazeli, N. (2017, March). Simulating Preservice Teachers’ Information-Seeking Behaviors While Learning with an Intelligent Web Browser. In Society for Information Technology & Teacher Education International Conference (pp. 2437-2442). Association for the Advancement of Computing in Education (AACE).
Roscoe, R. D. & Chi, M. T. H. (2007). Understanding tutor learning: Knowledge-building and knowledge-telling in peer tutors’ explanations and questions. Review of Educational Research, 77, 534-574.
Rowe, J., Pokorny, B., Goldberg, B., Mott, B. & Lester, J. (2017). Toward Simulated Students for Reinforcement Learning-Driven Tutorial Planning in GIFT. In R. Sottilare (Ed.) Proceedings of 5th Annual GIFT Users Symposium. Orlando, FL.
Shute, V. J., Lajoie, S. P. & Gluck, K. (2000). Individualized and group approaches to training. In S. Tobias & H. O’Neill (Eds.), Handbook on Training (pp. 171-207). Washington, DC: American Psychological Association.
Sottilare, R. A., Graesser, A., Hu, X. & Goldberg, B. H. (Eds.). (2013). Design Recommendations for Intelligent Tutoring Systems: Volume 1-Learner Modeling (Vol. 1). US Army Research Laboratory.
Sottilare, R., Graesser, A., Hu, X. & Goldberg, B. (Eds.). (2014). Design Recommendations for Intelligent Tutoring Systems: Volume 2-Instructional Management (Vol. 2). US Army Research Laboratory.
Steenbergen-Hu, S., & Cooper, H. (2013). A meta-analysis of the effectiveness of intelligent tutoring systems on K–12 students’ mathematical learning. Journal of Educational Psychology, 105(4), 970.
Steenbergen-Hu, S., & Cooper, H. (2014). A meta-analysis of the effectiveness of intelligent tutoring systems on college students’ academic learning. Journal of Educational Psychology, 106(2), 331.
Ur, S., & Van Lehn, K. (1995). Steps: A simulated, tutorable physics student. Journal of Artificial Intelligence in Education, 6(4), 405-437.
VanLehn, K. (2011). The relative effectiveness of human tutoring, intelligent tutoring systems, and other tutoring systems. Educational Psychologist, 46(4), 197-221.
VanLehn, K., Ohlsson, S., & Nason, R. (1994). Applications of simulated students: An exploration. Journal of Interactive Learning Research, 5(2), 135.
VanLehn, K. (1988). Student Modeling. In M. C. Polson & J. J. Richardson (Eds.). Foundations of intelligent tutoring systems (pp. 55-78). Lawrence Erlbaum Associates, Hillsdale.
Woolf, B. (2009). Building intelligent interactive tutors: Student-centered strategies for revolutionizing e-learning. Berlington, MA: Elsevier Inc.
Zapata-Rivera, D., Jackson, T. & Katz, I. R. (2015). Authoring conversation-based assessment scenarios. In R. A. Sottilare, A. C. Graesser, X. Hu, & K. Brawner (Eds.), Design Recommendations for Intelligent Tutoring Systems Volume 3: Authoring Tools and Expert Modeling Techniques (pp. 169-178). Orlando, FL: U.S. Army Research Laboratory.