Sensory Substitution

 

Sensory substitution is the presentation of the information normally associated with one sensory modality (e.g., visual information) through a distinct sensory modality (e.g., the auditory system). The vOICe is one sensory substitution system, which converts images from a webcam into “soundscapes” (presented via headphones), which experienced users are able to interpret as visual in nature.

For a semester-long project in my vision class, I decided to explore the vOICe.  I got my awesome officemate Matt Silver (pictured) to volunteer around 25 hours of his time over the course of the semester to using the vOICe, but he never reported 'visual' sensations.  In the backpack there is a laptop running the vOICe Learning Edition, taped to his hat is a Logitech Quickcam, and he's using Apple iPod headphones.

I wrote a paper about it, and also gave a brief presentation on it.

 

 

 

I think that systems like the vOICe have a lot of potential in terms of exploring visual consciousness, and also in exploring sensory phenomena.  I wrote a little term paper about it in the Fall of 2005, and if you're curious about sensory substitution for these ideas Larent Renier and Anne De Volder have an excellent review of the topic.

As a sort of proof-of-concept, I trained my subject in real environments (video) and also on stimuli containing illusory contours.  In particular, I explored whether the presence of illusory contours appeared to make changes in the images more or less detectable.  The images on the left are examples of stimuli that were used in the experiment (the image changes from what is on the left to what is on the right).  Notice that in each case, only one of the elements ("pac-men") has changed its shape or location.  In the above two cases, the same change has been made to the pac-man, the only difference being the overall configuration of the pac-men.  In the bottom case, a pac-man has simply moved towards the center of the square.

The change shown at top is the only change that contains exactly the same illusory contours as before.  The middle change never had an illusory contour, and the change at the bottom has a modified illusory contour.  Interestingly, the change on top was much more difficult to detect for the subject than the other changes.  For full descriptions of training, methods and results, please see my paper

All stimuli used in this experiment may be found online, and used with the vOICe Learning Edition using the "Sonify GUI" and "Negative Video" options along with Powerpoint in full screen.

 

I believe that these experiments serve as a proof-of-concept for the utility of sensory substitution as a means of exploring visual perception.  For comments or clarifications, please contact Alex Storer (storer [at] cns.bu.edu).
 

Updated May 2006 | Alex Storer | Cognitive & Neural Systems