Wizard’s Apprentice:
Testing of an Advanced Conversational Intelligent Tutor
Jae-wook Ahn, Patrick Watson, Maria Chang, Sharad Sundararajan, Tengfei Ma, Nirmal Mukhi, Srijith Parabhu and Bob Schloss
Corresponding Author: jaewook.ahn@us.ibm.com
Abstract
Building effective intelligent tutoring systems critically requires a subject-appropriate corpus of training data. The Wizard-of-Oz approach is a popular method for collecting training data and evaluating prototype systems by letting a human Wizard simulate an intelligent tutoring system and interact with users. However, human Wizards may fail to adequately simulate the proposed system’s behavior, leading to various interaction issues. In order to address these problems, we present a hybrid Wizard-of-Oz technique called Wizard’s Apprentice, which is a novel approach to integrate intelligent software to assist the human Wizard during a Wizard-of-Oz session. In this chapter, we present a prototype implementation of Wizard’s Apprentice and show some results of user testing that could help understanding the strengths and nature of the proposed method.
Keywords: intelligent tutoring system, Wizard-of-Oz, evaluation
APA citation information
Ahn, J., Watson, P., Chang, M., Sundararajan, S., Ma, T., Mukhi, N., Parabhu, S., & Schloss, B.(2018). Wizard’s Apprentice: Testing of an advanced conversational intelligent tutor. In S. D. Craig (Ed.). Tutoring and Intelligent Tutoring Systems (pp. 321-340). New York, NY: Nova Science Publishers.
References
Ahn, J., Watson, P., Chang, M., Sundararajan, S., Ma, T., Mukhi, N. & Prabhu, S. (2017). Wizard’s apprentice: Cognitive suggestion support for wizard-of-oz question answering. In E. André, R. Baker, X. Hu, M. M. T. Rodrigo & B. du Boulay (Eds.), Artificial intelligence in education (pp. 630–635). Cham: Springer International Publishing.
Allan, J. (2002). Introduction to topic detection and tracking. In J. Allan (Ed.), Topic detection and tracking: Event-based information organization (pp. 1–16). Boston, MA: Springer US. doi:10.1007/978-1-4615-0933-2_1.
Anderson, J. R., Boyle, C. F. & Reiser, B. J. (1985). Intelligent tutoring systems. Science(Washington), 228(4698), 456–462.
Ardito, C., Buono, P., Costabile, M. F., Lanzilotti, R. & Piccinno, A. (2009). A tool for wizard of oz studies of multimodal mobile systems. In 2009 2nd conference on human system interactions (pp. 344–347). doi:10.1109/HSI.2009.5091003.
Benzmüller, C., Horacek, H., Kruijff-Korbayová, I., Lesourd, H., Schiller, M. & Wolska, M. (2007). Diawoz-ii – a tool for wizard-of-oz experiments in mathematics. In C. Freksa, M. Kohlhase & K. Schill (Eds.), Ki 2006: Advances in artificial intelligence: 29th annual german conference on ai, ki 2006, bremen, germany, june 14-17, 2006. proceedings (pp. 159–173). Berlin, Heidelberg: Springer Berlin Heidelberg. doi:10.1007/978-3-540-69912-5_13.
Blei, D., Ng, A. & Jordan, M. (2003). Latent dirichlet allocation. The Journal of Machine Learning Research, 3, 993–1022.
Chughtai, R., Zhang, S. & Craig, S. D. (2015). Usability evaluation of intelligent tutoring system: Its from a usability perspective. Proceedings of the Human Factors and Ergonomics Society Annual Meeting, 59 (1), 367–371. doi:10.1177/
1541931215591076. eprint: https://doi.org/10.1177/1541931215591076.
Fialho, P. & Coheur, L. (2015). Chatwoz: Chatting through a wizard of oz. In Proceedings of the 17th international acm sigaccess conference on computers & accessibility (pp. 423–424). ASSETS ’15. Lisbon, Portugal: ACM. doi:10.1145/2700648.2811334.
Gena, C. & Weibelzahl, S. (2007). Usability engineering for the adaptive web. In P. Brusilovsky, A. Kobsa & W. Nejdl (Eds.), The adaptive web: Methods and strategies of web personalization (pp. 720–762). Berlin, Heidelberg: Springer Berlin Heidelberg. doi:10.1007/978-3-540-72079-9_24.
Gould, J. D., Conti, J. & Hovanyecz, T. (1983). Composing letters with a simulated listening typewriter. Communications of the ACM, 26(4), 295–308.
Kerly, A. & Bull, S. (2006). The potential for chatbots in negotiated learner modelling: A wizard-of-oz study. In M. Ikeda, K. D. Ashley & T. W. Chan (Eds.), Intelligent tutoring systems: 8th international conference, its 2006, jhongli, taiwan, june 26-30, 2006. Proceedings (pp. 443–452). Berlin, Heidelberg: Springer Berlin Heidelberg. doi:10.1007/11774303_44
Lasecki, W. S., Kamar, E. & Bohus, D. (2013). Conversations in the crowd: Collecting data for task-oriented dialog learning. In First aaai conference on human computation and crowdsourcing.
Marge, M., Bonial, C., Byrne, B., Cassidy, T., Evans, A. W., Hill, S. G. & Voss, C. R. (2017). Applying the wizard-of-oz technique to multimodal human-robot dialogue. CoRR, abs/1703.03714. arXiv: 1703.03714. Retrieved from http://arxiv.org/
abs/1703.03714.
Masthoff, J. (2006). The user as wizard: A method for early involvement in the design and evaluation of adaptive systems. In Fifth workshop on user-centred design and evaluation of adaptive systems (Vol. 1, pp. 460–469).
Maulsby, D., Greenberg, S. & Mander, R. (1993). Prototyping an intelligent agent through wizard of oz. In Proceedings of the interact ’93 and chi ’93 conference on human factors in computing systems (pp. 277–284). CHI ’93. Amsterdam, The Netherlands: ACM. doi:10.1145/169059.169215.
Miller, J. R. (1988). The role of human-computer interaction in intelligent tutoring systems. Foundations of intelligent tutoring systems, 143–189.
Okamoto, M., Yang, Y. & Ishida, T. (2001). Wizard of oz method for learning dialog agents. In M. Klusch & F. Zambonelli (Eds.), Cooperative information agents v: 5th internationalworkshop, cia 2001 modena, italy, september 6–8, 2001 proceedings (pp. 20–25). Berlin, Heidelberg: Springer Berlin Heidelberg. doi:10.1007/3-540-44799-7_3.
Preece, J., Rogers, Y., Sharp, H. & Benyon, D. (1994). Human-computer interaction. Addison-Wesley Pub.
Rizzo, P., Lee, H., Shaw, E., Johnson, W., Wang, N. & Mayer, R. (2005). A semi-automated wizard of oz interface for modeling tutorial strategies. User modeling 2005, 149–149.
Roscoe, R. D., Cooke, N. J., Branaghan, R. J. & Craig, S. D. (2017). Human systems engineering and educational technology. (pp. 1–34). IGI Global.
Schlögl, S., Doherty, G., Karamanis, N. & Luz, S. (2010). Webwoz: A wizard of oz prototyping framework. In Proceedings of the 2nd acm sigchi symposium on engineering interactive computing systems (pp. 109–114). EICS ’10. Berlin, Germany: ACM. doi:10.1145/1822018.1822035.
Schlögl, S., Doherty, G. & Luz, S. (2015). Wizard of oz experimentation for language technology applications: Challenges and tools. Interacting with Computers, 27(6), 592–615.
Shannon, C. E. & Weaver, W. (1998). The mathematical theory of communication. University of Illinois press.
Shneiderman, B. (1991). A taxonomy and rule base for the selection of interaction styles. Human factors for informatics usability, 325–342.
Watson Conversation. (2016). https://www.ibm.com/watson/developercloud/doc/
conversation.
Watson Retrieve and Rank. (2016). https://www.ibm.com/watson/developercloud/
retrieve-rank.html.