Kim Charlson was 11 when she started losing her eyesight because of glaucoma. An operation a year and a half later not only didn’t help, it resulted in complications that hastened her blindness.
Her pragmatic parents insisted she learn Braille, a key to literacy for people who are blind or have low vision. Without that literacy, Charlson likely wouldn’t have gone on to college or a career. Only 13 percent of blind students in the United States know Braille, and roughly 70 percent of adults who are blind or have low vision are unemployed.
Those troubling statistics are one reason Charlson is excited about an app that will help increase the amount of time students can spend learning and practicing Braille. ObjectiveEd, the company that’s developing the Braille AI Tutor app, is a new recipient of Microsoft’s AI for Accessibility grants to people using AI-powered technology to make the world a more inclusive place. Ten other recipients joining the program in conjunction with National Disability Awareness Month include City University of London, inABLE, iMerciv and The Open University
“We have a huge opportunity and a responsibility to be making technology smarter and more useful for people with disabilities,” says Mary Bellard, Microsoft senior architect lead for accessibility. The aim of the AI for Accessibility program, which began in 2018 and now has 32 grantees, is to help people “build something really useful at the intersection of AI, accessibility and disability.”
The Braille AI Tutor app is the latest project for ObjectiveEd’s president, Marty Schultz, a longtime software developer and volunteer teacher who created an iPhone game five years ago called “Blindfold Racer” for children who are blind. It led to more than 80 games for the iPhone and iPad that have together been downloaded more than a half-million times.
If you only get an hour a week with the teacher — I mean, how many kids would learn how to read print if they only had an hour a week of instruction?
Charlson, former president of the American Council of the Blind, is a big fan of Schultz’s work. So is Judy Dixon, consumer relations officer for the National Library Service for the Blind and Physically Handicapped, and the two women often talked with him about the importance of Braille education for literacy and employment. Schultz took it to heart — and to the drawing board.
Some students who are blind or have low vision attend schools that are geared to their needs, and where Braille is taught and used daily. But many attend public schools and learn Braille from teachers who visit their schools once a week, spending about an hour with each student.
“If you only get an hour a week with the teacher — I mean, how many kids would learn how to read print if they only had an hour a week of instruction?” says Charlson. “It’s just not enough. You have to immerse yourself in it at that developmental stage, or you’re not going to be as fluent in it as you need to be as an adult.”
The Braille AI Tutor app will incorporate AI-based speech recognition, using Microsoft’s Azure Speech API, to help students practice reading Braille with personalized, gamified learning plans. The app will send a word or a sentence to a refreshable Braille display, one of the types of hardware used for reading Braille. The student will feel the word in Braille, say the word or sentence out loud, and then the app will process the audio feedback and let the student know immediately if they are correct or not.
Teachers will be able to monitor students’ progress, with results sent to a web dashboard.
“We see our role as not teaching the student but giving the student the ability to practice when that teacher’s not around,” Schultz says. “The teacher teaches, and we make practicing fun and engaging and something that can be done without the teacher being there. So the next time the student meets with the teacher, the student has made some real progress.”
Schultz says the extra practice will help students “accelerate more quickly through school, which will lead to college, and to much better employment opportunities in the future.”
Two longtime friends who watched their loved ones go through vision loss found another way to help: using technology to help people get to work or otherwise navigate their cities.
Bin Liu and Arjun Mali are from different parts of the world, but their lives took parallel paths. Liu, born in China, moved at age 9 with his family to Gaborone, Botswana, for several years because of his father’s work as a civil engineer. Mali spent parts of his childhood in India and the United Arab Emirates, where his father worked for a while in sales of fiber optic networks.
About 10 years ago, Liu’s father was diagnosed with inoperable glaucoma. Mali’s grandmother in India had partial sight, and he sometimes accompanied her to a local school for the blind, where she volunteered, to read and teach English to the children.
The two were in university when they met in Toronto and became friends playing poker. They often talked about some of the frustrations and indignities faced by people who are blind or have low vision, as well as ways to improve mobility for those with vision impairment.
“Vision loss affected our families, and we saw an opportunity to create a technological solution that would impact that community,” says Mali, who graduated with an economics degree from McMaster University in Ontario.
Liu, who has a civil engineering degree from the University of Toronto, had been searching for devices that could help his dad navigate obstacles more precisely with his cane, and says he didn’t find much. Liu and Mali developed their first product together, the BuzzClip.
It’s a 2-ounce, clip-on wearable device that uses ultrasound to detect obstacles in a person’s path, then alerts the user with different vibrations and frequencies.
Vision loss affected our families, and we saw an opportunity to create a technological solution that would impact that community.
Early on, the duo received support from the Impact Centre, the University of Toronto’s accelerator for startup tech companies, and in 2014 they formed their company, iMerciv, Inc.
Now among AI for Accessibility’s latest grantees, iMerciv is developing a navigation app called MapinHood for pedestrians who are blind or have low vision, and who want to choose the routes they take if they’re walking to work, or to any destination.
The app will audibly alert a person to hazards — from construction to high-crime areas — to avoid while walking, as well as let them know about things they might need, like water fountains, benches or ramps. It’s all based on machine learning, crowdsourced data and open source information from local law enforcement.
Current navigation systems, in general, are optimized to generate routes that are the fastest or shortest for getting to a destination, but Liu says, “that’s not always the best route for pedestrians with disabilities” trying to find the best walking route to work, shops or parks, for example.
The app is in now in the alpha stages of being tested with help from the nonprofit Canadian National Institute for the Blind, which also worked with iMerciv on the BuzzClip. The app uses iMerciv’s custom routing engine, and with the AI for Accessibility grant, will use Azure machine learning, storage and virtual machines.
MapinHood in Toronto will also be a template for the app in other cities.
“Our focus is on personalization — making the app as flexible and as customizable as it can be,” Liu says. “Because with navigation for pedestrians in general — and especially for people with disabilities — you cannot have a single solution that fits all needs.”
For people with autism, sometimes the biggest hurdle to employment is the interview. That’s the focus of Nilanjan Sarkar. A family member – a cousin’s son – has autism, and in doing research later, Sarkar learned that people on the autism spectrum sometimes respond better when they deal with intelligent systems, such as chatbots, instead of people.
Sarkar, director of the Robotics and Autonomous Systems Lab at Vanderbilt University in Tennessee, is now leading a project aimed at helping people with autism perform well in job interviews using intelligent systems. Career Interview Readiness in Virtual Reality (CIRVR) is being developed in conjunction with Vanderbilt University’s Frist Center for Autism & Innovation, having joined the AI for Accessibility program earlier this year.
In the U.S., there are approximately 2.5 million adults on the autistism spectrum, Sarkar says. “Sixty percent or more of them can do some work. However, 85 percent of those able to work are either underemployed or unemployed.”
This system aims to help people be better prepared when they actually go out for an interview.
CIRVR is a virtual reality job interview platform that uses Azure AI and incorporates a computer avatar that acts as the interviewer, a wearable device that tracks interviewees’ physiological measures such as heart rate and skin sweating to infer their anxiety using machine learning techniques, and an eye tracker to gauge attention.
“This system will quantitively, objectively gather lots of data regarding their anxiety, where they’re looking, eye contact, how they’re responding, what should they have done — and we believe we can create a feedback system so that by repeated practice, they will improve their interviewing skills,” Sarkar says.
“People with autism sometimes like to interact with things that respond in a routine way, in a predictable way,” Sarkar says. “Human response, human interactions are not predictable, and that can be confusing.”
Many times, he says, open-ended interview questions such as “Can you tell me about an instance where you resolved a conflict?” or “How did you help a teammate?” might create anxiety. So can tests with urgency, such as being asked to solve a programming problem quickly.
Sarkar says CIRVR testing has begun and will provide feedback to the interviewees so they can practice improving how they handle interviews. Overall results will also be evaluated for trends to possibly share with hiring managers at interested companies, so they can learn how to modify their interview structure, or how to ask questions differently, if needed, Sarkar says.
“We assume the interview protocol structure will not change overnight,” he says. “So, this system aims to help people be better prepared when they actually go out for an interview.”
All AI for Accessibility grantees “have so much passion and expertise in the area of accessible technology,” says Bellard of Microsoft.
“The amount of potential that there is for software or hardware to better meet the needs of people with disabilities, and to raise the bar of what customers can come to expect of the role technology could play in their lives, is just an amazing opportunity.”
Learn more about AI for Accessibility grants, and about Microsoft’s Autism Hiring Program.
Lead image: Vision rehabilitation therapist Ashley Colburn shows 11-year-old Steven DeAngelis refreshable Braille devices at the Carroll Center for the Blind. The Newton, Massachusetts, center helped ObjectiveEd test the games it developed for people who are blind. (Photo by Dan DeLong)