Touchable Sound is an art-tech project that provides a immersive experience to people “hear” sound, music, and movies without relying on their ears. It enables the deaf community to perceive sounds by holding a device in their hands, while also offering hearing individuals a new dimension of sensory experience to comprehend sound related artistic works. Below you will find the real-life tests, how Touchable Sound works, and why it is important.

The test at the Deaf Association Amsterdam

In March 2023, a test of Touchable Sound was conducted at an event organized by the Deaf Association Amsterdam (Stichting Welzijn Doven Amsterdam - SWDA). During the test, we included movie clips, music, and sports videos, and invited individuals with varying degrees of hearing loss to participate. The results and feedback were incredibly exciting, as most users reported never having experienced anything like it before.

During the test, participants held the device in their hand and were able to feel the sound signal translated into strong and detailed vibrations on their fingertips. This allowed them to "hear" various sounds for the first time, such as a musician playing drums, sound effects from action movie clips, and the starting of a motorcycle engine. These were all previously unknown to them, and they were amazed by the experience.

The device consists of two "handles" that provide linear vibration patterns, effectively enhancing the details that can be captured by the nerves at the fingertips. This allows users to perceive intricate details, such as feeling the subtle clicks when watching a video depicting a key rotating in a door lock. Moreover, users can also distinguish between different tones produced by hitting various parts of a drum.

The device is designed to be used with both hands, providing a stereo setup for delivering vibrations. For example, while watching a video of a racing car driving from left to right, if the user holds the device in both hands, they can distinctly perceive the sensation of sound/vibration traveling from the left side to the right side.

Translate sound into vibration

Sound is essentially a collection of vibration patterns. When we hear sound, these vibrations travel through the air and reach our eardrums. The nerves in our ears then translate these vibrations into auditory feedback that our brain interprets as sound. Touchable Sound operates on the principle of processing and amplifying these sound wave vibrations, expanding them from what can be heard by the eardrums to what can be felt by the fingers. By translating sound into vibrations, Touchable Sound allows individuals to experience sound in a tactile and sensory manner.

Sensory Mapping

From receiving vibrations in the air to our brain perceiving sound, an essential step in this process is "Mapping." Our brains constantly attempt to connect vibration patterns (sound) to the visual signals received from our eyes. This allows us to learn how a guitar sounds without the need to see it. Similarly, mapping occurs as users connect haptic feedback to visual stimuli. In other words, Touchable Sound users gradually establish associations between specific vibrations and real-world sounds, enabling them to eventually "hear" the world.

Why hand

Touchable Sound utilizes the hands because the nerves in our fingertips are highly sensitive to vibration patterns, second only to the eardrums. Research conducted by D. R. Kenshalo in 1968 demonstrated that fingers possess exceptional haptic sensitivity and can discern intricate details more effectively than other parts of the body. Therefore, by incorporating tactile feedback through the hands, Touchable Sound capitalizes on the heightened sensitivity of the fingertips, allowing users to interpret sound vibrations with greater precision and clarity.

The content

Just like our ears, our ability to experience sound through touch has a range of frequencies to which it is most sensitive. Typically, the hearing system is sensitive to vibrations within the frequency range of 20 to 20,000Hz. However, through research and testing, it has been found that our fingers are more sensitive to lower-frequency vibrations, such as those produced by car engines. On the other hand, our ability to feel pitch changes, regardless of the volume (amplitude), is less pronounced when it comes to higher-frequency sounds like that of a flute. This highlights the importance of generating content within the appropriate frequency range for users to perceive through their hands.

Currently, as a singer and song write, I'm recreating the content by tweaking the sound tracks with multiple audio tools, such as EQ (equalizer), compressor, and exciter to modify and refine the soundtracks. These tools assist in isolating and emphasizing the essential elements of the sound, separating them from the ambient background sounds and creating desired haptic vibration patterns at the optimal frequencies for touch perception.

Why is this important

During the tests conducted with the local deaf community, I was deeply moved by their enthusiastic feedback, and their emotionally rich reactions were truly priceless. According to the World Health Organization, there are 430 million individuals worldwide with hearing loss, who have never had the opportunity to experience movies and music in the same way as the rest of us.

In addition, fascinating findings emerged during tests conducted with hearing individuals. Those who experienced both Touchable Sound and the actual sound simultaneously reported a significantly enhanced and enjoyable experience. Some even likened it to a more powerful version of the vibration feedback provided by the PS5 controller, with an exceptional level of resolution.

At this stage, I firmly believe that this project has the potential to profoundly impact the lives of countless individuals. It is poised to introduce a new type of cinema experience and music that will be accessible to all.