Accessibility Sprint - Part 3: Giving non-visual feedback for learning from interacting with PhET simulations

This is the third part of a series of blog posts about a coding sprint about creating interactive online learning that is usable for people with disabilities.

The first post gives an overview of the coding sprint. Each of these subsequent posts describes the work of one team.

Sims Team Goal

Make the University of Colorado Boulder's well respected, freely available, open-source PhET simulations more accessible for students who cannot see the simulation. By providing just the right amount of aural feedback about what is happening in the simulation after an action taken by a learner, blind and low-vision students could interact with the simulation, hear the results, and try additional actions to understand the underlying physics principles.

For example, PhET has been working on making their Balloon and Static electricity simulation accessible by including scene descriptions that screen readers read aloud in order to orient learners that can't just look around to see what looks controllable. The controls are all accessible via keyboard actions. But, when a learner takes an action, for instance removing a charged wall that is keeping the balloon steady, the resulting balloon movement must be described. It would overwhelm the listener if small changes are repetitively described, and it can be confusing if messages end up being read out of logical order. For instance, messages about the balloons movement might end up being read behind a message describing its reaching an object and stopping.

This image shows a balloon simulation of static electricity moving between a shirt and balloon. Beside the visual is information encoded in the DOM that is read when particular actions are taken. This is what will be read using assistive technology to help operate this sim and understand the results when actions are taken.

This group decided to work on extending the messaging being reported by this balloon sim, in order to better report very dynamic events, such as moving the balloon, or the balloon moving itself (attracted to sweater) without overwhelming and overlapping messages. To do this, they designed an UtteranceQueue, which is a FIFO (first in, first out) message queue with certain rules: it takes an object that contains an utterance, an object the utterance is associated with, an expected utterance time (to delay before the next utterance) and a callback that returns a boolean, to allow the utterance to be cancelled, rather than spoken, when it reaches the top of queue. This should allow a simulation programmer to design the set of messages a particular object should report. For example the balloon would report being moved, as well as its state of charge, and whether it is stuck to something. The callback would allow, for example, the balloon movement messages to cancel themselves if the balloon is in fact now stuck to the sweater or wall.

This image is showing the same balloon sim with the browser code view open to see what is controlling the simulation.

Testing (of the earlier version)

While the above development was occurring, on of the team members, Kelly, tested the feedback announcer function in the existing version of the balloon sim (the one before the code sprint) and got some user feedback for the group. The person that she tested with had worked with the sim before, but not with the new scene narration. Her test subject found the narration volubility to be just about right. He did, however, want to have a way to repeat some narration.

Demo

At the end of the day, this group demonstrated the operation of the new UtteranceQueue when the wall is removed and the balloon starts drifting toward the sweater. The movement was described (and not overly repetitive) and when the balloon got to the sweater that event was narrated. No other messages followed.

People who worked in this group: Jesse Greenberg, Darron Guinness, Ross Reedstrom, Kelly Lancaster

Code

https://github.com/phetsims/balloons-and-static-electricity/tree/ocad-hacks

Links

• The code for the PhET sims
• PhET sims that have a significant effort around accessibility 
• Balloon Static Electricity sim 
• Balloon Static Electricity sim 
• John Travoltage  
• OpenStax HTML5 sims directory OpenStax created HTML5 versions of some of the PhET sims that were needed for a high school physics book 
• and code: https://github.com/Connexions/simulations 
• OpenStax book using many PhET sims  
• https://cnx.org/contents/zOZP3vRI@5.1:9h7ZYYwq@2/Introduction 
• Balloon Sim used here at OpenStax: https://cnx.org/contents/zOZP3vRI@5.1:kZhR8P2h@2/Electrical-Charges-Conservatio (although the sim isn't properly displaying right now)