Through Google Glass, Remote Helper Sees What Blind Person Can't

The first thing I do is put on Google Glass, its tiny camera situated over my right eye.

A MiFi — a personal wireless hotspot — goes into my pocket. I pop in earbuds connected to a smartphone. Then I close my eyes. I am trying to simulate, for this demonstration, what it's like to navigate without being able to see.

All geared up, I am now ready for my test run. Aira, based in San Diego, is a paid service that offers "visual interpretation" for the blind and visually impaired. With a smartphone app, Aira users connect with a remote agent who sees through any one of four types of smartglasses and describes the client's environment to her. Employees say the service, which became available in March, is like OnStar for the blind — or maybe like having your own personal “Chloe,” the intelligence analyst on the TV show "24" who often remotely assists hero Jack Bauer.

I summon my Chloe using the Aira interface, which consists of a giant blue button. All I have to do is tap anywhere on the phone's display.

“Thank you for calling Aira,” a friendly voice says. “How can I help you?”

This is Andy Del Valle, my agent, and, for as long as I need him, my long-distance eyes. Del Valle — sitting in front of his laptop — can now see through my Google Glass feed. As I scan my surroundings, he sees Amy Bernal, the company’s vice president of customer experience, who has helped get me outfitted; then I lower my head so he can see the path before my feet.

Thanks to my GPS connection, Del Valle is also able to locate me using Google Maps. We’re on the campus of San Francisco State University, for me unfamiliar turf. I ask him to direct me to the nearest cafe.

Within seconds, he's produced a route.

Granted, we’re approximating the Aira experience; I’m a sighted person with no orientation and mobility training, and I'm not using a dog or cane, as regular users do. I am, to put it gently, perhaps not the most graceful mover Del Valle has ever assisted. But he is cheerful as he offers me directions, like to turn left toward 11 o’clock. As I walk, he offers environmental cues, telling me to turn when I feel the slope of the sidewalk level out, or when I can feel the sun as I emerge from a nearby building’s shadow.

I quickly grow to trust his voice in my ear.

After a short walk, I hear music and a cash register — we’ve arrived at our destination. I want to order an iced coffee; Del Valle asks me to turn so he can scan a posted menu.

“Taking picture,” an automated voice intones as Del Valle grabs a shot from the camera feed and zooms in. He also pulls up a more detailed menu online and gives me a rundown of options — espresso? Blended java drink? — and a headcount of how many people are in line.

Del Valle is among the 33 agents working that day, covering customers across the United States. The service runs from 4 a.m. to 10 p.m., Pacific time or 7 a.m.to 1 a.m. Eastern, and clients subscribe to a monthly plan or a package of minutes. (The monthly plans go from $89 to $329, depending on level of service.) Each user gets a personal profile that lists details like allergies, so an agent can warn someone allergic to shellfish, for example, that there’s shrimp salad on the potluck table. The company says it has "hundreds" of clients.

Bernal, the customer experience VP, is also an agent, and she's set up shop for the afternoon in the campus’ special education department. The calls she fields run from the mundane (a man looking for a trash can while cleaning up after his guide dog) to the complicated (a lost woman with bad directions from a passerby can't find her bus stop.) “You'll never know until you pick up the call,” Bernal says.

Sometimes they get a strange one — Bernal recalls an agent being asked to help a user do origami, having to hastily grab some paper and fold along. But most are the stuff of ordinary life: setting up the cable box, checking the expiration date on the milk, matching a shirt and tie.

Tiffany Manosh, president of the River City Chapter of the National Federation of the Blind, in Sacramento, has been using Aira for about a year. She likes it for tasks that involve browsing, like shopping. Sure, she could ask a clerk for help, but “they go directly to whatever item we’ve asked for,” she says. “Well, what if I want to see maybe what’s on sale?”

At a natural history museum, she once had an agent describe the exhibits. A docent could have done that — if she’d booked one a month in advance. “Sighted people don’t have to do that, so why should we?” she asks. “Aira gives us the freedom to be able to really integrate and do things at our own time — when we want to, and where we want to go.” An agent even helped her find her father’s gravesite at Arlington National Cemetery. “It was a very emotional and just phenomenal feeling to be able to do it on my own time,” says Manosh.

Aira, based in San Diego, test-launched three years ago with 200 beta users. Among them was Victorville resident Michael Hingson, who has been blind since birth and recently became the company’s director of strategic sales. “Aira is not helping a blind person see,” Hingson says. “Aira is giving me information that the sighted world has not learned to provide to us.”

For example, on emails or websites, those who are visually impaired might use a screen reader that speaks the text aloud. For printed materials, they might try a phone app like the KNFB Reader, which translates text using optical character recognition. But those don’t cover everything. Hingson recalls trying to assemble a laundry cart and discovering the instructions were in pictures, which his reader couldn’t decipher. So he called Aira. The agent zoomed in on the cart’s bar code and found assembly instructions on the internet. In a half-hour, Hingson said, “I had a working laundry cart.”

Aira agents are trained to give factual information without offering advice, deferring to the user’s orientation and mobility training. “What I don’t want is an agent telling me it’s safe to cross the street — because the agent doesn't know it’s safe to cross the street,” says Hingson. “That should be my choice.”

Learning to give the right information is a balancing act, says Bernal. “When agents first start, they feel like they have to say every little thing that is in any remote path of a user,” she says. But a person’s dog or cane will help them avoid obstacles. Instead, it might be more useful to offer social information, like where the empty seat in the room is, or whether the driver greeting you at the airport is smiling and waving you over.

Free Services

Bryan Bashin, CEO of the training and advocacy group LightHouse for the Blind and Visually Impaired, in San Francisco, has tried Aira, and points out that while it’s the first such for-profit service, it’s not the first visual interpretation app.

Once the iPhone debuted, he says, “a lot of people had the same idea at the same time.” He sometimes uses Be My Eyes, a free worldwide service powered by volunteers who assist by cellphone video chat. The free app TapTapSee identifies the content of still photos and speaks it aloud. And people can also improvise by simply Facetiming with friends or relatives. Bashin says these solutions are more accessible to low-income people than Aira’s paid service, although they lack some of the features — smartglasses, GPS mapping, agents trained to work with the blind. But what they all have in common, Bashin says, “is the world is still not 100 percent accessible for blind people.”

Bashin says that dialing a sighted person can be especially helpful when optical character readers fail, noting they are often thwarted by glare (think thermostat screens) or fancy script (think wine bottle labels or handwritten mail). But Bashin cautions against over-reliance on visual interpreters.

“When somebody first becomes blind, they think that the answers to the obstacles of blindness are to have someone see for you,” he says. Yet a person can skillfully use other senses, like touch and hearing, and low-tech solutions, like Braille. “There are hundreds of thousands of very clever and efficient solutions that don’t involve sight,” Bashin continues. “That’s what we teach at the LightHouse, and that’s what will get us through the day with a little bit of grace, so that we can reserve these technologies for when they really count. I'm a little bit afraid that new students will avoid learning what they need to learn to be competent blind people.”

While some Aira users prefer a paid agent — often for fear of imposing on others’ time — Bashin doesn't mind asking a stranger to steer him toward the check-in desk or describe what’s on the buffet, because that’s a social activity. “Yes, I’ll learn where the strawberries are, but I’ll also learn about them. Now I’ve made a bridge to another person instead of isolating myself talking into a headset and some goggles,” he says.

But Manosh, from the Sacramento National Federation of the Blind chapter, disagrees that the service will encourage isolation or over-reliance. She calls Aira “a tool in my toolbox” that also includes her cane and her mobility training. “You don’t have to rely just on it, and that’s the beauty of it,” she says.

The next phase of Aira’s development will be to turn some tasks over to a machine. At first, Bernal says, the AI will likely assist the agent. Later, it might guide people through routine tasks, like daily commutes. Or it might replace agents in sensitive situations, while users are in restrooms or dealing with finances, for instance. (Currently, users can go into “privacy mode” with no video or audio, or hang up and call back.)

Hingson imagines sending a shopping list to Aira, and having the AI map his path through the store and check barcodes to confirm that he's grabbed the right items. He’d also like Aira integration with Facebook, so the system could use facial recognition to let him know if his friends are nearby.

For now, employees like Bernal are happy to assist with just about everything, although she notes that there’s one oddly sci-fi effect of seeing the world through someone else’s lenses.

“I’ve had this experience where I’ve actually gone to an airport and thought ‘How do I know this airport? I’ve never been here before,’ ” she says. “But you realize that you’ve been an agent for a user in an airport. You are like, ‘Oh, I’ve been in this terminal, but I haven’t been in this terminal.’ It’s very déjà vu.”