Math Speech: ClearSpeak

This project's goals are:

  1. Create rules for synthetic speech for math expressions typical of high school-level Algebra that produce speech that is similar to speech used in typical classrooms. This style is called ClearSpeak.
  2. Create the means for content-authors and content-users to customize features of ClearSpeak to suit the needs of a student, class, or occasion.
  3. Create the means for content-users to use the keyboard to navigate a math expression as it is spoken by the computer.
  4. Create the means for math expressions to be spoken in ClearSpeak (including its customized settings) using screen readers within Microsoft Word and elsewhere.

This project is supported by the Institute of Education Sciences, U.S. Department of Education, through Grant R324A110355 to the Educational Testing Service. The opinions expressed in research reports or presentations posted on this site are those of the authors and do not represent views of the Institute or the U.S. Department of Education.

Contacts

Lois Frankel, Principal Investigator, Educational Testing Service
lfrankel@ets.org

Beth Brownstein, Co-Principal Investigator, Educational Testing Service
bbrownstein@ets.org

Neil Soiffer, Co-Principal Investigator, Design Science, Inc.
soiffer@alum.mit.edu

ClearSpeak Development

The ClearSpeak rules and preferences have been implemented in MathPlayer 4.

Integration with NVDA

The free, open-source screen reader, NVDA, now integrates with MathPlayer in the Internet Explorer and Firefox browsers, as well as in Microsoft Word. Speech and navigation are fully supported.

Integration with Word

MathPlayer 4 now includes integration with Microsoft Word. Users can either use the included MathPlayer toolbar to speak and navigate math, or use NVDA or WindowEyes to integrate speech of the math into the speech of the remainder of a Word document. It is no longer necessary to create an xhtml file or to use Internet Explorer to listen to math expressions.

Full keyboard-based non-linear (structural) navigation through expressions has been implemented and is available in the downloadable MathPlayer 4.

Authoring Tools

MathType now includes methods for specifying speech preferences (e.g. whether a fraction is spoken as [something] over [something], as a common fraction (e.g. three fifths), or in various other ways that are useful in different instructional or testing settings. A simplified tool is available for generating preference files for this purpose. Additionally, authors can specify the exact text to speak for a given expression, and can (when publishing to the web) specify which of the three available speech styles (MathSpeak, Simple Speech, or ClearSpeak) should be used.

Feedback Studies

The project included four feedback studies and a final pilot. All of the studies are complete, and the data has been analyzed. Reports from the studies are in various stages of completion. The research questions addressed by the feedback studies are:

  • Does ClearSpeak improve over the existing speech styles? What elements of ClearSpeak improve accessibility over existing scripting methods for students at various levels of familiarity with the mathematical content? (Feedback Study 1; publication of an ETS Research Report is pending.)
  • How can ClearSpeak be further enhanced with prosodic elements to improve the comprehensibility of certain kinds of math structures? (Feedback Study 2; an ETS Research Report is in review for publication.)
  • Can students with good AT skills navigate within the expression using ClearSpeak and the navigation tools developed? If not, what changes are needed to improve independent navigation? (Feedback Study 3)
  • Can teachers use the authoring tools successfully to create accessible materials? (Feedback Study 4)

Final Pilot

The pilot study evaluated the feasibility of implementing the authoring, speaking, and navigation tools in an authentic education delivery setting, and evaluated the promise of the tools to operate as intended to produce the outcome of increased audio accessibility of math for students with visual impairments. A report is in development.

Summary of Activity To Date

We began with the development of the ClearSpeak speech style, using as its conceptual basis classroom practice and existing internal ETS rules and best practices, into a system that could be coded for use by the MathPlayer software. To that end, we produced a comprehensive and organized listing of types and subtypes of math expressions, along with notes on how they are currently spoken by MathPlayer and how they should be spoken.

The design philosophy of the ClearSpeak speech system is to speak mathematical expressions/objects so as to lessen or remove for learners who are blind or have low vision the accessibility barrier to learning the mathematical material being presented. ClearSpeak seeks to accomplish this by speaking expressions in a way that is both familiar and unambiguous. Since students are most familiar with the way their teachers speak in the classroom, ClearSpeak speaks math in that way—but with adjustments to make sure the spoken math is unambiguous. The verbiage introduced to make an expression/object unambiguous can make spoken expressions harder to understand than the print versions of those expressions. ClearSpeak seeks to minimize the effect of this by extensive use of pauses (which do not require any memory); using language that integrates well with the expression; and "translating" print conventions into their mathematical equivalents if a sighted learner would do so instantaneously—doing for the learner with blindness or low vision what the sighted learner does almost unconsciously. Finally, since in different contexts, it is desirable to have the same math expression speak differently, ClearSpeak provides (through preferences) a mechanism for speaking them in a way appropriate to commonly encountered contexts and (through exact speech) a mechanism for speaking them in any way the document author desires.

  1. Developed ClearSpeak prototype based on the design philosophy and on the listing of math expression types.

    Based on this design philosophy and on the foundational document described above, we determined that ClearSpeak should be implemented through three mechanisms, listed from the most to the least automatic:

    1. Rules. Speech rules are the standard or default way for speaking math structures, based on information that can be parsed from the MathML representation of the expression/object. Math expressions are analyzed based on type of element (fractions, exponents, parenthetical expressions, radicals, etc.), level of complexity, and on other relevant features present in the expression. Based on that analysis, the appropriate rules are invoked for speaking the expression. In addition, rules include specifications of how various symbols (e.g. raised dot, set membership, etc.) are to be spoken.
    2. Preferences. Because math structures that will be parsed the same way may at times (to resolve ambiguities or for instructional or assessment purposes) need to be spoken differently from the applicable rules' defaults, we developed a system of preferences that authors of math documents could use to specify for a given expression whether, for example, a fraction would be spoken as [numerator] over [denominator] or as a common fraction. Thus "12/15" by rule would be spoken "twelve over fifteen", but if the "ordinal" fraction preference is invoked, it would be spoken instead as "twelve fifteenths".
    3. Exact Speech. Finally, for cases in which the developed rules and preferences are incomplete, not applicable to a particular expression, or do not address a particular pedagogical or assessment need, we implemented the ability to insert exact speech, including additional pauses, pitch-changes, and speed-changes when authoring a math expression/object. 
  2. Developed specifications for rules, preferences, and tested in MathPlayer, Styles can be referenced in documents, and preferences are recognized.

    MathPlayer now recognizes the rules and preferences developed for ClearSpeak. We are able to specify in a web document that the ClearSpeak style (as opposed to the pre-existing speech styles Simple Speech and MathSpeak). Web documents also speak the math in accordance with the preferences and exact speech that had been set up for that document.
  3. Implemented interactive navigation. Math expressions can now be navigated interactively using MathPlayer, in Microsoft Word, Internet Explorer, and Firefox.
  4. Achieved support for MathPlayer by the NVDA screen reader. Using NVDA, math expressions in a Word document or on the web can be spoken and navigated along with the text, and can send Nemeth Code to an attached refreshable braille display.
  5. Designed, conducted, and analyzed four feedback studies and a final pilot.

Reports

2017:

  1. "Expanding Audio Access to Mathematics Expressions by Students With Visual Impairments via MathML"
    Frankel, L., Brownstein, B., and Soiffer, N. (2017), Expanding Audio Access to Mathematics Expressions by Students With Visual Impairments via MathML. ETS Research Report Series. doi:10.1002/ets2.12132

2016:

  1. "Development and Initial Evaluation of the ClearSpeak Style for Automated Speaking of Algebra"
    Frankel, L., Brownstein, B., Soiffer, N. and Hansen, E. (2016), Development and Initial Evaluation of the ClearSpeak Style for Automated Speaking of Algebra. ETS Research Report Series. doi:10.1002/ets2.12103
  2. "An Evaluation of the Usefulness of Prosodic and Lexical Cues for Understanding Synthesized Speech of Mathematics"
    Frankel, L. and Brownstein, B. (2016), An Evaluation of the Usefulness of Prosodic and Lexical Cues for Understanding Synthesized Speech of Mathematics. ETS Research Report Series. doi:10.1002/ets2.12119

Presentations

2016:

  1. "ClearSpeak for Math Accessibility - Overview"
    (YouTube video) This video is included on two playlists (Math and Science; Special Education) compiled by the Institute of Education Sciences as part of their December 6, 2016 post. See How IES is Supporting Technology-Delivered assessments.

2015:

  1. "Math Accessibility in 2015: Browsers, Word, and Beyond"
    (29.3 MB PPTX file) presented by Neil Soiffer, Lois Frankel, and Beth Brownstein at CSUN, March 5, 2015
  2. "MathML to Voiced Math"
    (17.8 MB PPTX file) presented by Lois Frankel, Beth Brownstein, and Neil Soiffer at the Principals of Schools for the Blind (POSB) Math/Science Institute, April 2015.

2014:

  1. "Understanding Mathematical Expressions through Interactive Navigation"
    (20.3 MB PPTX file) presented by Sina Bahram, Neil Soiffer, and Lois Frankel at CSUN, March 19, 2014
  2. "Navigating Math Expressions with Synthetic Speech"
    (349 KB PPTX file) presented by Lois Frankel and Susan Osterhaus at CEC, April 2014
  3. "Breakthroughs on Math Accessibility"
    (47.5 MB PPTX file) presented by Neil Soiffer for the Montana Accessibility Interest Group, October 3, 2014

2013:

  1. "Making Math More Fully Accessible for Users of Text-To-Speech Technology"
    (21.6 MB PowerPoint file) presented by Beth Brownstein and Neil Soiffer at ATIA 2013 Orlando, February 1, 2013.
  2. "Navigable, Customizable TTS for Algebra" (paper, PowerPoint slides -- 16.8 MB PPTX file)
    presented by Lois Frankel and Neil Soiffer at CSUN, San Diego, CA, March 1, 2013
  3. "Making a Computer Speak Algebra However You Want," (12.2 MB PowerPoint file)
    presented by Susan Osterhaus at CEC 2013 Convention and Expo, April 5, 2013
  4. "Making a Computer Speak Math Like a Teacher Would," (12.2 MB PowerPoint file)
    presented by Beth Brownstein and Susan Osterhaus at 2013 NCTM Annual Meeting & Exposition, April 2013
  5. "Empowering Visually Impaired Learners via Spoken Algebra," (streaming video)
    presented by Beth Brownstein, Lois Frankel, and Neil Soiffer at the ETS Research Forum, October 2013

2012:

  1. "Expanding Audio Access to Mathematics Expressions by Students with Visual Impairments via MathML," Presented by Susan Osterhaus at the 2012 GMI Crossover Symposium on Technology for the Blind and Visually Impaired, Hosted by the University of Colorado at Colorado Springs, May 12, 2012. Presentation
  2. "Expanding Audio Access to Mathematics Expressions by Students with Visual Impairments via MathML," Presented by Lois Frankel at Council of Chief State School Officers (CCSSO) National Conference on Student Assessment (NCSA), as part of the panel, Research on Providing Audio Access to Assessment Content, June 28, 2012.
  3. "Making a Computer Speak Math However you Want," presented by Lois Frankel and Susan Osterhaus at AER International, July 19, 2012.

Written Documentation and Videos

MathPlayer User Manual

This manual is in the MathPlayer documentation, and covers installation and use.

Chart of Navigation Quick Reference Keys

Alternative formats of the Navigation Guide:

This tutorial (309 KB DOCX file) is aimed at users of the NVDA screen reader in Microsoft Word. The math navigation features can also be used with NVDA in browsers, and without NVDA using the speech toolbar that is installed with MathPlayer.

Expression supplement for Navigation Tutorial:

NVDA Guides

A guide to using all of the features in NVDA, including math navigation, can be found on the NV Access site. There is also an audio guide in mp3 format.

Authoring Guide

Authoring Tutorial

Step-by-step instruction on authoring math with MathType inside Microsoft Word and setting it to speak as desired. Download (4.3 MB DOCX file)

Authoring Tutorial Workbook

The workbook (235 KB DOCX file) contains examples for use with the tutorial.

Preference Builder

This optional tool (13.5 MB ZIP file) is for advanced users who wish to use an interactive tool to build their own .eqp files containing combinations of frequently-used preferences. To install, first unzip the archive, and then run Setup.Exe. It may be necessary to use Run-as-Administrator. If Setup indicates that any of its files are older than matching files already on your system, you should retain your existing (newer) files. .Eqp files can, alternatively, be created manually with any text editor, as described in the Authoring Tutorial.

ClearSpeak Reference

ClearSpeak Rules and Preferences

A complete guide to the speech rules and preferences built into ClearSpeak. (1.3 MB DOCX file)

ClearSpeak Examples

This set of examples for testing ClearSpeak capabilities consists of 16 DOCX files, contained in a single 4.2 MB ZIP file.

Video Series: How to Author Spoken Math using MathType

We recommend viewing the videos in the order shown. They refer to some of the written materials that that are currently being finalized and will be linked to this page here once ready. You will also need to download the archive containing speech rules and preferences "EQP" files then extract the files to your computer. You can choose any location on your computer that you find convenient. One possibility is to create a folder called "speech" under the MathType Preferences folder (where print preferences are stored). The following shows standard locations for 32- and 64-bit versions of Windows:

32-bit systems: C:\Program Files\MathType\Preferences\Speech
64-bit systems: C:\Program Files (x86)\MathType\Preferences\Speech

If you do not have administrative rights to your computer, you may not be able to save files to the locations given above. Instead, we suggest saving them to the location referenced in the Authoring Tutorial, which is:

C:\ClearSpeakAuthorPreferences

To follow along with the video to author and play back math you will need:

1. Introduction to authoring in ClearSpeak (0:5:36)

Basic introduction to authoring math expressions in ClearSpeak using MathType and playing them using MathPlayer. This video describes required and optional software, related documents that will soon be added to this site, and shows how to play back math speech using MathPlayer in Word.

2. Using MathType to enter math expressions (0:4:17)

Video tutorial for getting started in MathType. Experienced MathType users can skip this video.

3. Getting an expression to speak the way you want it to - Part 1 (0:9:49)

How to use the built-in speech rules and preferences

4. Getting an expression to speak the way you want it to - Part 2 (0:7:09)

How to set multiple speech preferences for a single math expression

5. Getting an expression to speak the way you want it to - Part 3 (0:6:28)

How to enter and remove information specifying the exact speech (including pauses if desired) to use for a math expression. Exact speech is to be used only when the rules and preferences do not result in the speech desired.

Principal Investigator

Lois Frankel, Ph.D.
Senior Assessment Specialist
Educational Testing Service
lfrankel@ets.org

Co-Principal Investigator: Math Accessibility Lead

Beth Brownstein, M.A.
Senior Assessment Specialist
Educational Testing Service
bbrownstein@ets.org

Co-Principal Investigator: Technology Lead

Neil Soiffer, Ph.D.
Senior Scientist
Design Science, Inc.
soiffer@alum.mit.edu

Research Director

Eric Hansen, Ph.D.
Research Scientist
Educational Testing Service
ehansen@ets.org

Data Analyst

Nan Kong, M.S.
Senior Research Data Analyst
Educational Testing Service
nkong@ets.org

Research Consultant

Cara Laitusis, Ph.D.
Research Director
Educational Testing Service
claitusis@ets.org

We are proud to have had the following individuals serve as our advisory committee.

Jim Allan
Accessibility Coordinator & Webmaster
Texas School for the Blind and Visually Impaired

Maylene Bird
Math Teacher
Texas School for the Blind and Visually Impaired

Chelsea Cook
Student
Virginia Polytechnic Institute and State University

John De Witt
De Witt & Company

Gaylen Kapperman, Ph.D.
Professor and Coordinator, Department of Special & Early Education
Northern Illinois University

Abraham Nemeth, Ph.D.
Professor Emeritus of Mathematics
University of Detroit Mercy

Steve Noble, M.P.A.
Research Consultant, University of Louisville;
Content Specialist: Mathematics and Science, Bridge MultiMedia

Susan Osterhaus, M.Ed.
Statewide Mathematics Consultant, Outreach Department
Texas School for the Blind and Visually Impaired

ETS and the ETS logo are registered trademarks of Educational Testing Service (ETS). MEASURING THE POWER OF LEARNING is a trademark of ETS. Copyright © 1996-2019 Design Science and ETS. All rights reserved.