Boulder Language Technologies

International Collaboration in Virtual Tutoring and Therapy Systems

Ron Cole
Center for Spoken Language Research
University of Colorado Boulder

Recent advances in computer science and engineering have realized new and unprecedented opportunities to benefit individuals through computer-based treatments that use lifelike computer characters that behave like sensitive and effective tutors or therapists. The proposed workshop in Bordeaux France will bring together researchers in computer science and speech and language pathology from the United States and France who desire to collaborate to advance research and development of virtual therapy systems that can provide accessible and effective treatments to individuals with neurological disorders. The first day of the workshop will involve presentations by U.S. and French researchers related to ongoing efforts to develop the tools, technologies and treatments that can benefit individuals with neurological disorders. The second day of the workshop will bring together a smaller group of researchers who will work together to explore ways to initiate and sustain collaborative research projects.

Intellectual Merit: Research and development of virtual tutoring and therapy systems requires a deep understanding of the communication processes that occur during effective behavioral therapies and knowledge about how to design human computer interfaces that model these communication processes to optimize user engagement and treatment effectiveness. The proposed workshop will bring researchers together to share insights gained from their research and to propose future work to advance knowledge about human communication, clinical treatments and the design of effective treatments based on this knowledge.

Broader Impact: Virtual human systems with optional online clinician oversight can offer an accessible and inexpensive means to provide effective behavioral treatments to millions of individuals. The proposed workshop has the potential to accelerate research and development of virtual therapy systems through international collaboration. The workshop is expected to lead to collaborations among U.S. and French researchers that will result in new knowledge and in virtual therapy systems in French and English that will be evaluated in each country with individuals with Parkinson disease and aphasia.

Background

In 1997, the Oregon Graduate Institute was awarded an NSF Challenge Grant (Ron Cole, PI) to develop a computer program to teach speech and language skills to students with hearing loss using a 3-D talking head. In 2000, the University of Colorado Boulder (UCB) was awarded an NSF ITR grant (Ron Cole, PI) to develop technologies leading to a new generation of intelligent tutoring systems that use perceptive animated agents that behave like sensitive and effective teachers, and to test these technologies in a program that would teach children to read and comprehend text. Additional funding for this research was provided by NSF IERI and ITR grants to UCB (Walter Kintsch, PI, Ron Cole, co-PI) and by an NICHD IERI grant (Ron Cole, PI). Recently, UCB was awarded an NSF CRI grant (Ron Cole, PI) to develop a virtual human toolkit to be freely distributed to enable researchers to design and test programs that support natural spoken dialogs with virtual tutors. In April 2003, Ron Cole organized a workshop funded by NSF to stimulate research on Virtual Human Systems (VT Workshop, 2006).

These research and development activities funded by the NSF resulted in a number of positive outcomes. The research funded by the NSF Challenge Grant was featured on ABC TV's Prime Time Thursday, and the broadcast was coordinated with a feature on the NSF Web site. The research funded by the NSF ITR and IERI grants produced a reading program, called Foundations to Literacy (FtL), which features Marni, a virtual tutor, which has been used by over 2000 K-3 students in Colorado schools. The program, which features the character animation and speech recognition technologies developed under these grants, has produced learning gains in summative evaluations in word reading and comprehension. Surveys and interviews with students and teachers revealed highly positive experiences with the program. Students believe that Marni is smart, that she acts like a real teacher, that she cares about them and that she teaches them to read.

The tools and technologies developed during these projects have benefited research both nationally and internationally. The SONIC speech recognition system and children's speech corpora have been distributed to over 300 researchers in universities and licensed to several U.S. companies. The learning tools and authoring environments developed for the Foundations to Literacy program are the basis of two additional projects, one funded by IES, and one funded by NIH (Barb Wise, PI, Ron Cole, Co-PI); the IES project aims to develop a fully automatic assessment tool for estimating a child's reading level and identifying specific reading challenges, and the NIH project (a component of an NIH Center grant to Dick Olsen at UCB) extends FtL to 3rd-5th grades, and assesses FtL in a response to intervention study.

Virtual Therapists

One of the most exciting and gratifying outgrowths of our work is the development of virtual therapists or clinicians. It turns out that a number of effective speech and language therapies have been developed in laboratory settings for individuals with neurological disorders, including Parkinson disease and aphasia. These treatments, which are intensive (an hour or more daily), extensive (lasting a month or more) and often tedious, are not accessible to the millions of individuals who could benefit from them due to insufficient numbers of trained clinicians, access to clinicians by patients, costs of treatment and other factors.

Computer treatments offer the potential of providing inexpensive and accessible treatments to millions of individuals. We are currently working with leading researchers in three different laboratories to develop four virtual therapy systems for individuals with Parkinson disease and aphasia. The researchers include Lorraine Ramig at the National Center for Voice and Speech in Denver, Leora Cherney at the Rehabilitation Institute of Chicago, and Cynthia Thompson at the Northwestern University. These programs, funded by grants from the NIH's NIDCD and Department of Education's NIDRR, (described on the CSLR Web site ) are currently undergoing clinical trials. Results of these collaborations and demonstrations of the resulting system were presented at a special session of the American Speech and Hearing Association (ASHA) in November, 2005 organized by Dr. Cherney.

Web-based Clinician Oversight: While the goal of computer-based learning and therapy programs is to support independent user interaction with a virtual tutor or therapist, it is desirable and sometimes necessary to provide clinician oversight. Clinician oversight can motivate patients, make them more comfortable knowing that a therapist is just a mouse click away, and assure treatment quality and fidelity. The NIH and NIDRR have funded research projects now underway at CSLR to develop clinician oversight systems that enable a clinician to monitor and interact with several individuals using the program simultaneously in remote locations. Augmenting virtual therapy systems with clinician oversight is essential to the success of virtual therapy systems of the future. International collaboration will accelerate progress in this area.

International Connections

Our research efforts have a strong international component. Under NSF support, CSLR organized a summer workshop that brought together researchers from Italy, Chile and Mexico to develop Interactive Books with virtual tutors in Italian and Mexico. This workshop is described at (URL). Collaboration with other international researchers has resulted in Interactive Books in French, German and Polish. In Poland, collaborator KASIA has received a grant to develop a Polish Literacy Tutor and a grant to develop a Polish version of a virtual therapy system for individuals with Parkinson disease.

The 2004 Virtual Human Systems workshop included education researchers involved in animation and agent systems, and created several new collaborations that are accelerating the impact of virtual human tutoring systems in future learning environment research. The Virtual Human Systems workshop thus helped to provide a foundation for a series of international NSF-supported symposia on distributed learning and collaboration (DLAC) led by Eric Hamilton, a current CISE grantee at the US Air Force Academy. These symposia and workshops, in Shanghai (2006), Beijing (2006), London (2007), and Germany (2008) have connected us to several groups in France and elsewhere in Europe where learning scientists and education researchers can benefit more fully from the investments that NSF, NIH, and IES have made in our work with perceptive agent technologies. Most notably for this proposal, the DLAC workshop series has connected us with researchers in France and OECD's important Brain Science and Learning research program (Della-Chiesa, 2003, 2005).

Events leading to the Bordeaux workshop: In April 2006, I was invited to give a keynote talk at the annual Poznan Linguistic Meeting in Poznan Poland for special session on Linguistics and medicine. (Collaborator Lorraine Ramig also gave an invited talk in this session.) Following my presentation, Professor Jean-Michel Madaux invited me to visit France to present a similar talk at his university, and mentioned that he would invite several colleagues conducting research on speech and language treatments. After settling on a date (May 2), Dr. Madaux sent me an email informing me that a one day event would be held with approximately 100 participants, including heads of labs and aphasia organizations in France, and that he would like to have a separate meeting the following day with selected collaborators to plan a "National Marni Project." (Marni is the name of the virtual tutor used in several of our virtual tutoring and therapy programs.) I forwarded the email to Ephraim Glinnert, program manager of the NSF ITR grant, and then called him to discuss the possibility of NSF funding travel to the Bordeaux meeting to bring the clinical researcher who developed the treatments as well as computer scientists, with the goal of developing international collaborations. Following a number of email discussions, Dr. Glinnert encouraged me to submit a proposal.

Workshop Planning

During the past month, I have been in continuous communication with Dr. Madaux via email and telephone to plan the workshop. I have communicated with the clinician's with whom I am collaborating to develop virtual tutoring systems, and each has agreed to participate if travel funds can be obtained. Dr. Madaux has asked a travel agent to reserve lodging for 10 U.S. participants at a moderately priced hotel. I have also initiated a conversation with Dr. Lana Shekim, the program manager at NIDCD for several of the virtual therapy grants, to learn if NIDCD may be interested in contributing travel funds for additional participants. If NIH can provide additional travel funds, it will be possible to bring additional CISE researchers to the Bordeaux to explore collaboration with our French colleagues.

I am working with Dr. Madaux to develop an agenda for each day of the workshop. A tentative agenda for the first day is enclosed below. It includes presentations by French researchers to provide an overview of aphasia research and computer-based aphasia treatment programs in France, as well as presentations on virtual human research efforts. Our plan is to work with the organizing committee to refine this agenda, and to invite participants from the U.S. and France who are eager to explore collaboration.

The agenda for the second day of the workshop is under discussion, but the goals are clear: to begin the process of conceptualizing and planning collaborations that can result in effective and accessible computer-based clinical treatments in France. Ideas for collaboration include (a) working with French researchers to extend and assess virtual therapy systems already developed at CSLR, (b) researching and developing new virtual therapy systems based on treatments developed in France, (c) investigating Web-based clinician oversight approaches to computer-based treatments, and (d) collaborating to improve the tools and technologies that power virtual tutors and clinicians.

Workshop Objectives and Expected Outcomes

The goals of the meeting in France are to disseminate knowledge about the research activities conducted under NSF, NIH and IES support to interested researchers in Computer Science and Health Sciences in France, and to stimulate and sustain collaboration leading to a new generation of accessible, affordable and highly clinical treatments using virtual tutors and therapists.

There is some likelihood that an award to this proposal will result in a French national project to develop accessible computer-based treatment programs using tools and technologies developed under NSF support. Dr. Madaux has communicated interest and excitement among his colleagues in developing a national project to develop accessible and cost-effective computer-based treatments using virtual therapy systems, we will also investigate the means to fund national and international efforts towards this goal. For this reason, participation by representatives from OECD (Organization for Economic Cooperation and Development), French funding agencies and U.S. funding agencies is desired.

There is good likelihood that participation by U.S. researchers will result in multidisciplinary international collaborations with individuals in several laboratories in France. We expect to be joined in Bordeaux by at least one of the leaders of OECD's Brain Science and Learning research network, mentioned on the first page and which was started under NSF funding in 2000. These collaborations will also provide excellent research test beds for development of multilingual virtual tutoring systems using the virtual human toolkit that is under development at CSLR with support from an NSF CISE CRI grant. It is expected that the meeting will also result in student exchanges between labs in France and the U.S. and scientific publications of work initiated from collaborations established during the meeting. A report of the workshop will be delivered to the NSF, and posted on the CSLR website with reports of other workshops organized by the Principal Investigator (http://cslr.colorado.edu/mirror/nsf/).

Cost Sharing

The University of Colorado reduced its indirect cost rate from 49.5% to 26% at our request. The University of Bordeaux will provide all of the facilities and refreshments for the workshop. It is anticipated (and I will request) that lunch and perhaps one dinner be paid for by our hosts. The attendees' costs will be adjusted (reduced) as appropriate to account for meals paid for by the University of Bordeaux. Dr Madaux has also been asked to find a modest but comfortable hotel that includes breakfast.

Participants (Tentative)

Ron Cole, University of Colorado, Boulder
Eric Hamilton, Air Force Academy
Leora Cherney, Rehabilitation Institute of Chicago
Angela Halpern, National Center for Voice and Speech
Cynthia Thompson, Northwestern University
Audrey Holland, Professor Emeritus, University of Arizona (& Adjunct Professor, UCB)
CISE Researchers (5-6; To be named).
Bruno Della-Chiesa, OECD Brain Science and Learning Director (No Cost to Grant)
NSF Paris Office Director; To be invited (No Cost to Grant)

Agenda (Tentative, Day 1)

Avant-Projet de séminaire :
THE VIRTUAL THERAPY OF SPEECH AND LANGUAGE THERAPY
Réalité virtuelle et rééducation orthophonique du langage et de la parole

Organisé par :
Center for Spoken Language Research, USA
Institut de Cognitique, Département d'Orthophonie et EA Handicap et Système Nerveux, Université Victor Segalen Bordeaux 2 ;
Institut Fédératif de Recherches sur le Handicap IFR 26 ;
Union nationale pour le développement de la recherche et de l'évaluation en orthophonie UNADREO

BORDEAUX, 2 mai 2007, amphithéâtre …

Programme prévisionnel :

9h. Accueil, présentation du séminaire. Pr JM Mazaux
9h15 Présentation des partenaires institutionnels. Pr B Claverie, Pr Dauman, Pr PA Joseph, Dr Pradat-Diehl, Dr Rousseau
10h Réalité virtuelle et rééducation cognitive : état des lieux en France. Pr Claverie, Pr Nkaoua
10h30 Pause
11h Theory, research and technologies of Virtual Therapy. Pr Cole
12h30 Déjeuner
14h Speech therapy and aphasia : the State of the Art. Pr Holland
15h Theory, research and presentation of VT treatments in aphasia :
       The program Skriptalk. Dr. Cherney
       The program ORLA. Dr. Cherney
       The program Sentactics. Dr. Thompson
16h30 Pause
17h Theory, research and presentation of VT treatments for Parkinson disease
       The program LSVT VT. Ms. Halpern
       The program Polish LSVT VT Dr. Dziubalska-Kolaczyk
17h30 Table ronde avec l'ensemble des participants. Quelles perspectives en France ? Quels problèmes, quelles solutions ?

Della-Chiesa, B. (2003). Learning Sciences and Brain Research - A Focus on Research in Literacy, Mathematical Thought and Lifelong Learning National Science Foundation Award REC -0231780.

Della-Chiesa, B. (2005). Implications of research in brain science: results of OECD studies. Paper presented at the Futures of Learning: New Paradigms, Paris.

VT Workshop (2006).