They collaborated with Seattle filmmaker and photographer Dan Thornton, Scottish sound engineer Graham Byron and 9e2 organizer John Boylan to display it at the nine-day 9e2 event of installations and performances on art, science and technology. But the team knew the system wasn’t “art.” They knew they needed to do more.Ĭonnecting with local artists, Joshi began working with Salmon to create a new piece with Salmon’s art and Microsoft research. The testers overwhelmingly enjoyed the experience, and one woman, an art lover who had stopped going to galleries after losing her sight, cried in joy. Last year, researchers invited people with low vision to test a prototype built with programmed music, narration and sounds acoustically interpreting paintings by Frida Kahlo, Henri Matisse and others. Microsoft researcher Neel Joshi works on the Kinect-enabled sound system of “The Oregon Project.” He’s also a painter and interactive-installation artist involved in the local arts community. “Visual exploration is a personal experience and is based on distance - as I get closer, I see different types of details - so we wanted to mimic those ideas in audio,” says Joshi, who specializes in computer vision and computational photography. And they wanted sound and visuals to be equally aesthetic and important, and rejected anything that felt like a “bolted-on” accessibility tool. But the researchers wanted to create a more personalized experience that was still communal, and something less directed and curated by docents.Īfter exploring different ideas, they decided to create a Kinect-enabled sound system based on proxemics, the study of how people use space to define social interactions.
Many museums offer experiences for people with visual impairment, from detailed audio descriptions of art to special events where people can touch sculptures with gloved hands. The sound system stems from an experimental project called Eyes-Free Art, created by Microsoft researchers Neel Joshi and Meredith Ringel Morris and former Microsoft intern Kyle Rector to explore how technology can help people with low vision access visual art. “It has a wonderful, atmospheric, spacey, landscape-y feel that pulls the whole thing together.” “As a way of moving from an outer to inner layer to the drawings, natural sounds to industrial sounds works really well,” Salmon says. At the exhibit, adults triggered sounds primarily by walking, while kids jumped and waved their arms to produce a different blend of noise. The Kinects also track hand waves to change what you hear, helping observers become part of the piece and make their own acoustic mixes with movement. Stand even closer and you can hear Salmon working in his studio as he scribbles pastels on paper, in an experience that parallels how a sighted person can see details when leaning in. Moving closer activates digital tones matched to Salmon’s palette of blue, green, brown and ochre. From afar, observers can hear birdsong, rushing water and grass swishing in the wind - recorded in Hells Canyon by Salmon and his collaborators. The Kinects track movement to trigger different sounds, making the art lively and interactive for people with and without vision. In front of each drawing is a 3D space containing 18 soundtracks that can be played. You can watch the video with audio-description here. With his sight limited to a small amount of vision in one eye for many years, he’s been able to adapt, paint and hike with his partner Anita Groves as his guide.
“It would be great to think that in a few years, you might go to a national museum or art gallery and this technology would be there as part of the experience,” says Salmon, who began to lose his sight in the ‘80s due to diabetic retinopathy.