What Color is B-Flat? Adventures in Synesthesia Through Your Smartphone

Only a few months ago, #TheDress made Internet headlines the world over. The blue-versus-white argument might have seemed simple, but beneath it was a profound philosophical and biological question: do you see what I see? The intricacies of perception make the question a compelling one—especially if you or I have synesthesia.

Commonly defined as the comingling of sensory signals in the brain, “synesthesia” is an umbrella term that encompasses multiple distinct conditions. People with grapheme-color synesthesia see colors when they read, often associating individual letters with specific hues. Auditory-visual synesthetes visualize music; visual-gustatory synesthetes taste their surroundings; and visual-spatial synesthetes see their thoughts, sometimes arranged in patterns around their bodies.

If you’re thinking, “Cool! I wish I could experience that,” you’re in luck. A spate of apps seeking to augment users’ worlds has arrived, interacting with the senses in a quest to heighten everyday experiences, boost memory, and improve decision-making.

What Does That Pizza Sound Like?

The most popular are apps that mimic auditory-visual synesthesia. Sonified, for example, uses your phone’s camera to render light and color around you into music in real time, while Roy G Biv allows users to convert photos of their surroundings into tones, answering that age-old question, “What does pizza sound like”?

Olivia Abtahi, who developed an app prototype called the Synesthesia Music Experience, wanted to recreate the experience a synesthete friend has listening to music—colors so vivid the friend can’t drive while listening to the radio. Abtahi’s prototype syncs the lights in a user’s home to music, helping them access “the timbre, colors, and textures that sound can induce.” She recommends the app Ambify for a more elaborate execution of the same concept.

What Color Is “A”?

Zebulon Reynolds’s first work on his Color Code app was under the guise of self-improvement. He had a hunch that associating colors and symbols could act as scaffolding for remembering useful things, so he made a spreadsheet of paired letters, numbers, and colors that he practiced while running on the beach at night.

“I was basically a crazy person shouting numbers at myself,” Reynolds says.

Then he learned about studies at the University of Sussex in England that appeared to induce synesthesia and improve memory in participants through repetitive exposure to colored letters.

“People induced with this specific kind of synesthesia straight up boosted their IQs,” Reynolds says.

Using that study data, Reynolds created Color Code, a memory game that pairs letters and numbers with colors, attempting to boost memory and IQ by inducing synesthesia-like symptoms in users. He hopes his app will equip users with mental training wheels for remembering specific kinds of information, such as phone numbers and license plates.

Go With Your Gut

Jonathan Jackson’s visual-spatial synesthesia means he thinks differently than most people. Jackson sees his thoughts in space, an ability that allows him to, as he puts it, “visually access my intuition.” Rather than vague sentiments that require struggle to interpret, his gut feelings are arranged in space—easy to see, easy to grasp.

Jackson developed ChoiceMap as a way to digitally imitate his decision-making process. The app prompts users to enter their options (“Toyota” versus “Honda,” for example), plus the priorities and weights for each (“comfort,” “cost,” “reliability”). Then it feeds them into an algorithm, spitting out a “perfection score” that shows how closely each option hews to the user’s stated priorities. What may have seemed like a blurry, uncertain decision is now translated into clear, unequivocal numbers.

ChoiceMap helps users decide what house to buy, what classes to take, and where to live. Jackson says it has named thousands of babies, has facilitated tens of thousands of break ups, and most recently has been used to help patients make decisions about cancer treatment and end-of-life decisions.

“Our team really respects that,” Jackson says, “ and we don’t treat that lightly.”

What Color Is “Monday”?

Where the other apps mimic synesthesia for users, SynQuiz hopes to find those who already have it. Developed by the Max Planck Institute in the Netherlands, the app is a digital adaptation of the Eagleman battery, a tool long relied upon to diagnose visual synesthetes.

Katerina Kucera works in the Institute’s Language and Genetics group, which created the app and is using it to recruit synesthetes worldwide for a study. A synesthete who self-diagnoses through the app may register for the study.

While some users are from far-flung places, others are not. At a recent conference, Kucera was at dinner with two students who had never heard of synesthesia. By way of explanation, she asked them, “What color is A?”

“One of the guys says, ‘What?’ The other guy says, ‘Red!’” she remembers. “I said, ‘Maybe you need to go download the app and try it.’”

The Upshot

It’s hard to say whether audiovisual programming can ever truly mimic what synesthetes see, but for ChoiceMap designer Jackson a few come close. “When they actually look as geometric as my experience, I can use that to help explain it to other people,” he says.

As for Color Code, researcher Kucera says she believes that synesthesia can be acquired, but “the difference from true synesthesia is it doesn’t last. The results are very quick. Then they test them a few months or weeks later, and it’s gone.”

She is skeptical of, but interested in, Reynolds’ goal of boosting IQ and memory with his app. “Whether you could turn it [induced synesthesia] into something that improves,” she says, “that remains to be seen.”

Tatiana Josephy used ChoiceMap to decide whether to sell her Emeryville, CA condo. Josephy echoes Jackson in talking about the role of intuition in ChoiceMap.

“Intuition is this soft murky feeling that’s not super clear,” she says. “Being able to break down the different factors and being able to see them visually really helps with that process of making what’s implicit very explicit. We always hear, you know, ‘Act on your gut,’ but it’s not always easy to do.”

24/7 Synesthete

Audiovisual app designer Abtahi, who created her app prototype to mimic what she sees as a “superpower,” is certainly not alone in her envy. But the non-synesthete app user, while enjoying audiovisual experiences, making informed decisions, and improving memory and IQ, would do well to remember that synesthesia can be more challenging than these apps let on, Jackson says.

“After spending a majority of my life wondering why other people don’t think the way that I think, it’s nice to hear people say something positive,” he says of the starry-eyed language that often surrounds technologies that mimic synesthesia.

But, Jackson says, it can also be “so grating” because that perspective glosses over how complicated life is for people with synesthesia.

“It is easier for me to visualize complex concepts than it is for most people,” he acknowledges. “I can see how if you couldn’t do that it would be compelling. But if you’re having a conversation with people and all these things are exploding into your thoughts and you don’t have control over it—like, I’m just trying to drink a beer and talk about the Seahawks. It doesn’t always feel advantageous.”

After adventures that brought her from China to Spain and back, Alissa Greenberg has finally settled in the Bay Area, where she writes about culture, community, weird science, and hidden histories. She is currently at work on a book about millennials on the Trans-Mongolian railroad.