Boulder Language Technologies

By Ron Cole and Barb Wise

Project Team:
Ron Cole, Barb Wise, Sarel van Vuuren, Nattawut Ngampatipatpong, Jariya Tuantranont,
Gordon Golding, Lynn Snyder, Taylor Streumph, Pam Cole, Tim Weston, Donna Caccamise

Previous Project Team Members:
Isidoros Doxas, Bill Gaven, Scott Schwartz, Linda Corson, Dave Wade-Stein

Project Overview:

FtL is a comprehensive computer-based reading program that has been designed to teach beginning and early readers to read accurately, fluently and with good comprehension. It provides the pedagogical foundation for effective learning experiences for ELL students through the following features:

a) Marni, an engaging, lifelike animated character, instructs and supports the student through a sequence of learning activities. Marni produces natural (recorded) speech accompanied by accurate lip movements, head movements and facial expressions. Marni is helpful and supportive-providing instruction, hints, feedback and encouragement. Student interviews (see below) reveal that students "bond" with Marni and feel that she behaves like a real teacher.

b) The program proceeds through a set of learning exercises that build on each other, and the sequencing of learning tasks are based on scientific research. The program teaches foundational reading skills until accuracy is assured, and then presents speeded tasks so these skills become automatic and unconscious.

c) The program emphasizes phonological awareness and decoding skills, which research shows are just those skills that require extra intensive practice for many children with reading challenges.

d) The program's study plan continually modifies pacing, presentation of material, and progress through the program based on the student's performance on the learning tasks. Thus, students who learn quickly can move rapidly through the program, with periodic "reviews" to assure that skills have been retained and generalize to new stimuli, while students who have difficulty learning will receive continued practice on a focused set of skills within a variety of complementary learning exercises and books. The program continuously adjusts the pacing of activities and the number of response alternatives provided based on the student's performance.

e) Children spend significant time listening to and reading interesting books that are aligned with the learning exercises in terms of both vocabulary and content, so that skills learned in the exercises are practiced and reinforced in books.

Scientific Basis of FtL

FtL incorporates the key principles of scientifically-based reading research. The Report of the National Reading Panel (NRP, 2000) and a comprehensive review article (Rayner et al., 2001) summarize the implications of scientific reading research for instruction. This research suggests that balanced reading instruction covers five domains of reading with sequenced instruction that is intensive, explicit, structured and direct. The domains include phonological awareness, phonics (alphabet, decoding, and spelling), fluency (automaticity and reading with natural expression), vocabulary, and comprehension. Most reading professionals agree that these domains are important in balanced reading instruction, and Foundations to Literacy is designed to cover these five domains explicitly, systematically and intensively in carefully sequenced ways.

All FtL exercises aim for simplicity, engagement, and empowerment of success. The experience of success is encouraged by presenting the sequenced items in a "scaffolded" manner (Vygotsky, 1978), ensuring that children start with "comfort level" items from a previously successful level, proceed to supported instructional levels, and end again in "comfort level." This scaffolding is done seamlessly by the program, as it would be by an expert teacher. We also encourage a full cycle of learning from discovery and practice to competence. After students attain competence with a skill or concept, the program assigns speeded practice with it to get it automatic, applies it in reading in context, and places students in books designed to transfer these skills to independent reading and writing.

FtL Components

1. Foundations to Literacy consists of three integrated components:
   A Managed Learning Environment that tracks and displays student progress and manages an individual study plan for each student.
2. Foundational Skills Reading Exercises, which teach and practice basic reading skills, such as alphabet knowledge and word decoding, providing the foundation for fluent reading.
3. Interactive Books, which represent the state of the art in integration of human language and animation technologies to enable conversational interaction with a Virtual Tutor that teaches fluent reading and comprehension of text.

Managed Learning Environment (MLE): The MLE controls each student's individual study plan, administers retention tests periodically within the program, adapts the student's study plan automatically based on his or her performance in learning exercises (e.g., by reducing the number of response choices, or moving the student to a lower level in the program), and presents the student with rewards and progress graphics. The MLE also provides an interactive interface for educators to enrol students within the program, to examine progress within the program in various ways (including replaying individual sessions), and to measure progress for individuals or groups in terms of district or state learning goals. In the context of the proposed work, the MLE will be used to enrol students in the program, to construct different study plans for the different computer-based intervention conditions describe below, and to log, organize, analyze and display student learning data in the different treatment conditions.

After students are placed within the MLE based on estimates of their current reading level, the program adapts to the student's performance to optimize learning. The program can focus the student on critical skills while challenging the student to maintain an optimal learning pace. Based on the adaptive study plan, students can move at their own pace through a set of exercises to more advanced exercises and books, or be kept at the same level, or taken to lower levels for remediation and practice on specific core competencies. An important feature of the program is that it engages and empowers students by providing them with choices about what to do next. When the student completes an exercise, he or she returns to the virtual homeroom, where he or she can then choose to do another exercise, or select a new book to read, or go to their library to reread a book. The homeroom is also where the student is rewarded for completing exercises and books (via an animated progress graphic) so the student is able to understand what level they are in within the program, how much they have accomplished, and how much work is required to achieve the next milestones.

Foundational Reading Skills Exercises: Foundational Reading Skills Exercises are designed to teach core competencies needed to recognize words and read fluently. Exercise domains include alphabet knowledge, phonological awareness, letter-to-sound decoding and spelling regular words and reading and spelling common sight words. Each foundational reading skills exercise was designed by a team of researchers at CSLR using science-based research principles based on recommendations from the National Reading Panel, and participatory design methodology-which brings together researchers, programmers, teachers and students to design, test and modify the software from the initial stages of development. Each foundational reading skill is taught in 3 to 5 different learning games that complement and reinforce each other.

Figure 1. Six examples of Foundational Reading Skills Exercises.
Currently about 30 different exercises exist in total.

Interactive Books: Interactive books teach students to read and comprehend text. Interactive Books are designed by our project staff using a powerful set of tools (the Interactive Books Authoring Environment) that enables us to design books with different functions appropriate for readers at different levels. In the proposed work, Interactive Books will provide the learning environment for training fluent and expressive reading. New functions will be created and existing functions enhanced for this purpose. Current capabilities of Interactive Books include:

Figure 2. Examples of Interactive Books.

• Assessing Background Knowledge: Interactive Books have been designed to assess a student's knowledge of key concepts and words presented in a text. Vocabulary knowledge is assessed by having the virtual tutor present a spoken word, followed by multiple choices of definitions that the student can listen to our select, or by presenting a spoken definition, and having the student select from a choice of 4 printed words, one in each quadrant of a square, which are spoken when clicked on (hereafter, 4-square exercises). In addition to assessing background and vocabulary knowledge using multiple choice questions, these same presentation modes, when combined with feedback, can be used to train students to learn vocabulary and provide background knowledge needed for comprehension of stories. Assessment of vocabulary and background knowledge prior to reading texts will be used in the proposed work to measure and factor in differences in fluency and comprehension in the different conditions, and training of vocabulary and background will be provided at the beginning of interactive books to help mitigate the influence of limited background and vocabulary knowledge.
• Narration of Text by the Virtual Tutor: The virtual tutor can read pages of text or individual sentences to the student, depending upon the interactions allowed by the study plan for students at different reading levels within the program. When narrating a story (e.g., when training listening comprehension for beginning readers), the character produces accurate visual speech through movements of the lips, tongue and jaw that is synchronized with naturally recorded speech. When talking, the animated agent produces graceful head movements and facial expressions synchronized with the speech.
• Pronunciation of Words or Sentences by the Virtual Tutor: In all books, students are able to click on individual words to hear and see them spoken. In some books, students can also click on entire sentences to hear them read.
• Providing Real Time Feedback to Students as they Read Aloud: Interactive books provide real time feedback to students when reading out loud through a read aloud tracker. In the current implementation, as each sentence on a page is highlighted, one after another, a speech recognizer identifies each word that is pronounced accurately (i.e., receives a recognition score above a selected confidence level), and moves a cursor to the beginning of the next word after each correctly pronounced word. The tracker is able to identify reading miscues (word insertions, substitutions and deletions). The student may click on a difficult word to hear it pronounced, and these words are flagged for later review. If the student pauses before a word for 2 seconds, the virtual tutor pronounces the word. Sentences that present difficulty are read by the virtual tutor, and then reread by the student.
• Different Interactions for Students at Different Reading Levels: The nature of the interaction between the Virtual Tutor and the student within an Interactive Book varies depending on the reading level of the child. For example, after early or poor readers have learned to decode a small set of words in the exercises, they apply that knowledge first in sentence reading exercises and then in decodable books that apply those same patterns in context. As students learn more words, the books they read include more natural language with more and more complex and irregular words. We have developed about 150 Interactive books for kindergarten, 1st and 2nd grade reading levels.
• Assessing and Training Comprehension: In the current implementation of FtL, Interactive Books assess comprehension using thinking multiple choice (MC) questions, developed by our research team (Wise et al., 2003; Kintsch E., 2005). Thinking MC questions occur at logical points within a story and following the story. The design of thinking MC questions was motivated by the Discourse Comprehension theory and informed by research conducted by Beck, McKeown and others (1996). The goal of thinking MC questions is to stimulate the student's thinking (a) through thought provoking questions, (b) by providing response choices that challenge the reader and assess the nature of comprehension difficulties, and (c) through feedback on response alternatives that stimulate additional thinking and deepen comprehension. Thinking MC questions in Interactive Books include gist questions that target the main ideas and most important events in the story, and inference questions that target information not explicitly stated in the text. These inference questions require the learner to either (a) connect ideas from sentences in different parts of the story (textbase knowledge), or (b) stretch beyond the story, draw a conclusion or implication, or connect ideas beyond what the author has explicitly stated. The response alternatives in thinking MC questions are carefully designed to assess the student's level of comprehension, e.g., fillers for an inference question typically include an unwarranted inference, a correct nugget of information that does not answer the question, and an answer that is obviously incorrect. Each answer produces feedback from the Virtual Tutor; an expansion of the answer for correct responses, or information about why the answer is incorrect and a hint designed to stimulate thinking.
  
  While thinking MC questions are designed both to assess comprehension and to train comprehension by promoting thinking and reasoning about stories as they are being read, standardized tests of comprehension do not provide evidence of improved comprehension of texts by students who used FtL relative to students in control conditions. This may be due in part to the limited time on task the average student spent in Interactive Books (< 4 hours average), but it may also indicate that thinking MC questions are insufficient by themselves to improve comprehension of texts given the many other factors that can contribute to comprehension. (In the proposed work, we augment thinking MC questions with automaticity, fluency and expressiveness training, and propose to use more sensitive measures of comprehension administered periodically throughout the course of the study.)
  
  It is interesting to note that children in kindergarten through second grade report that they like having Marni ask them questions during stories, and that answering questions helps them learn. In response to the question: "When Marni asked me questions about a story," 202 of 239 students selected the choice "I liked answering the questions." In response to the question: "Do you think that answering the questions helped you to learn?", 221 of 237 answered "yes." While students were split equally in their choice of learning games or books as their preferred activity, when asked the question: "Did you learn more about how to read from the exercises or the books?", 171 of 251 students responded "More from books." Thus, students using FtL tend to believe that they learn most from books, and that answering questions helps them to learn.

FtL Technologies

The high levels of engagement and positive experiences reported by students and teachers who used the FtL program during the 2004-2005 and 2005-2006 school years, reported below, are caused in part by the seamless integration (Van Vuuren, in press; Cole et al., 2003) of computer speech recognition and character animation technologies that make Marni believable and likeable. The 3-D character animation technologies in the CU Animate system generate accurate movements of the lips, tongue and jaw synchronized with recorded speech and natural head and face movements. The speech recognition technologies developed at CSLR produce accurate phonetic segmentation and recognition of words in speech. Together, these technologies enable conversational interaction with a virtual tutor and real time feedback to students when they read out loud.

SONIC Speech Recognizer: To track students reading out loud we use a novel speech recognition technology developed at the University of Colorado known as SONIC . SONIC has been benchmarked on a number of comparative databases within the speech recognition community and has been shown to provide efficient, accurate, and state-of-the-art performance, including  recognition of children's speech. SONIC has been ported to over 15 different languages and children's speech recognition has been enabled in 4 languages (English, Spanish, Italian, and Finnish). SONIC is currently integrated within the FtL platform and enables real-time speech-based interaction with children. During typical reading aloud in context by children in grades 2 through 5, SONIC obtains 93% word accuracy and has the capability to automatically tune to a particular child's voice across multiple sessions of interaction.

3-D Character Animation: The Virtual Tutor, Marni, "lives" within Interactive Books and Foundational Reading Skills Exercises. During these tasks, Marni gives hints and explanations to help children to figure out answers while learning about phonological awareness, reading, spelling, and comprehension. In the learning activities, Marni pronounces words distinctly, with accurate movements of the lips, tongue, and jaw. Rather than using synthetic speech, which would be easier and faster, we use recorded speech, synchronized automatically with the movements of the visible articulators and natural facial expressions and head movements, to foster social agency. By using a recorded voice, we produce a more natural and effective user experience that benefits from the remarkable range and communication abilities of the human voice. When the character reads a story or interacts with the child, she does so with the appropriate prosody and emotional content in all contexts. In Interactive Books, we record all sentences, prompts, questions, and reading supports produced by the agent. We also record each word pronounced individually in isolation, so the Tutor pronounces it clearly and distinctly when the student clicks on the word. To students, Marni is a believable and credible teacher because she produces natural and expressive speech with appropriate facial expression using the CU Animate system.

FtL Outcomes

Summative evaluation of FtL during 2003-2004 school years produced significant learning gains for letter and word recognition for all measures for Kindergarten students (elision, letter and word recognition, see Wise et al, 2004). Results for the 2004-2005 school years found significant effects for two measures across the three main school districts for kindergarten, first grade and second grade students included in the analysis. The comparison for the WJ-III Letter-Word ID Standard Score was significant at F = 20.6, df (error) = 718 and p < .001. The effect size for this measure equaled d = .34. Large gains were found for Kindergarteners on this variable with an effect size of d= .85. The standardized WJ-III Letter ID measure also showed significant differences between treatment groups (F = 19.8, df = 190, p < .001). The effect size for this measure was d = .62.

Student Experiences with FtL: A student survey was administered to 129 K-2 children who used FtL in 10 schools during the 2004-2005 school year. A revised and expanded survey was administered to 239 K-2 students who used FtL in 7 schools during the 2005-2006 school year. The surveys were administered individually to each student by a member of the FtL project staff. Questions were designed to provide insights about students' opinions about the program, and to gauge students' understanding of specific program features. All questions except the first one, "What did you think about the program?", could be answered by selecting among response choices that were read aloud to the student, or by selecting among different pictures. Opinions were indicated using a 3-point Likert scale. For example, choices to "How much do you trust Marni" were "I trust Marni very much," "I trust Marni some of the time," and "I don't trust Marni." The survey questions and histograms of student responses to each question can be viewed here.

The survey results demonstrated that students were able to produce informative and discriminative responses to the program, with significant variations in responses to the different questions by individual students. Overall, students were highly enthusiastic about the program and their interactions with Marni. About 75% of all responses fell into the highest approval category. When given three choices, with the first being positive, the second neutral and the third being negative, students reported that Marni is smart (176/47/16; smart / smart sometimes / not smart), is a good teacher (158/74/8), acts like a real teacher (156/55/29), helped them learn how to read (150/66/24), that they trusted Marni (151/70/18) and believe she cares about them (156/40/42).

Teachers' Impressions of FtL: Thirty six teachers (of the 38 total whose students used FtL) provided written responses to a questionnaire that posed 17 questions about their experiences with and opinions of the program. The written responses were coded for analysis; survey questions and histograms of the coded responses can be viewed here. In general, teachers were positive about the program. Of the 36 teachers who responded to the question "What did you think about the program?" 1 did not like it, 12 said it was OK, 11 said they liked it when it worked well (referring to intermittent problems that caused the program to freeze), and 12 said they loved it. All teachers reported that the program taught skills they wanted their students to learn, and that the program complemented their classroom activities. All but 2 teachers said they would like to use the program next year, and all but one said they would recommend it to other teachers.

Publications

FtL has been featured in over forty presentations at conferences and workshops and in invited talks in the U.S., Europe and China. Recent talks by R. Cole are  listed here. Articles that incorporate summaries of FtL include:

1. Ron et al IEEE article
2. Barb et al book chapter
3.  Ron et al Ed tech article
4.  Sarel Ed tech article

FtL History and Sources of Support

The Foundations to Literacy (FtL) project was initiated in 2000 when CU was awarded a 5 year National Science Foundation (NSF) Information Technology Research (ITR) grant with Ron Cole as the Principal Investigator (PI). The project was subsequently supported by a second NSF ITR grant (Walter Kintsch, PI) an Interagency Education Research Initiative (IERI) grant from the NSF (Walter Kintsch, PI) and an IERI grant from the National Institute of Health's National Institute for Child Health and Development (Ron Cole PI). Additional support to demonstrate the feasibility of extending FtL to students with mild to moderate cognitive disabilities (e.g., Autism Spectrum Disorder or Mental Retardation) was provided by the Coleman Institute for Cognitive Disabilities. Recently, an NIH Center grant was awarded to CU (Richard Olsen, PI), with a project component dedicated to using FtL as a means for studying Response to Intervention (Barb Wise, project PI). Foundations to Literacy is a component of the Colorado Literacy Tutor, a collaboration between universities and public schools that aims to improve student achievement through development of educational software that helps students learn to read and comprehend text. The research grants that have or continue to support FtL and the Colorado Literacy Tutor projects include:

The following research grants provide direct support for the CSLR Reading Project:

NIH: 1542620- Center Director, Olson, R. "Differential Diagnosis of Reading Disabilities," $7,415,061 12/01/05-11/30/2010

Project V PI: Wise, B.; Van Vuuren, S.; Cole, R., Byrne, B. "Response to Computer-Assisted Instruction for Reading Difficulties," $1,327,284, NIH, 4/01/06 - 11/30/2010

NSF/ITR: REC-0115419 - Kintsch, W., Landauer T., Caccamise, D., Cole, R., "ITR/PE: Latent Semantic Analysis Theory and Technology," $2,400,000, NSF, 09/01/01 - 08/31/06.

NSF/IERI: EIA-0121201 - Kintsch, W., Caccamise, D., Cole, R., Olson, R., Snyder, L., "IERI: Scalable and Sustainable Technologies for Reading Instruction and Assessment," $5,997,404, NSF, 07/01/01 - 06/26/06.

NSF/ITR: IIS-0086107 - Cole, R., van Santen, J., Movellan, J., "ITR: Creating the Next Generation of Intelligent Animated Conversational Agents," $4,000,000, NSF, 09/01/00 - 08/31/05.

NSF/IERI : 1R01HD-44276.01Cole, R., Barker, L., Schwartz S., Snyder, L., Wise, B., "IERI: Scaling up Reading Tutors," $1,000,000.00, NIH. 9/27/02 - 9/30/04.

Coleman Institute for Cognitive Disabilities: Schwartz, S., Cole, R., Wise, B., Doxas, I., "Coleman Foundation Grant: Participatory Design for Creating Computer Based Learning Tools," $8184, Coleman.

NSF ITR Supplement Research Experience for Undergrads: REU/ITR: Creating the Next Generation of Intelligent Animated Conversational Agents (supplement to NSF 0086107), $45,500.00, NSF, 06/01/02 - 8/31/03.

NSF ITR Supplement Research Experience for Teachers: RET/ITR: Creating the Next Generation of Intelligent Animated Conversational Agents (supplement to NSF 0086107) $40,000.00, NSF, 09/01/02 - 08/31/03.

A Special Thanks to the Educators Who Supported FtL