Deep Sound Synthesis Reveals Novel Category-Defining Sound Features in the Human Auditory Cortex

Specific acoustic features give rise to distinct categorical percepts, such as speech or music, and to spatially distinct preferential responses in the auditory cortex. These responses contain category-relevant information, yet their representational level and role within the acoustic-to-semantic transformation process remain unclear.

We combined neuroimaging, a deep neural network, a brain-based sound synthesis, and psychophysics to identify the sound features that are internally represented in the speech- and music-selective human auditory cortex and test their functional role in sound categorization. We found that the synthetized sounds exhibit unnatural features distinct from those normally associated with speech and music, yet they elicit categorical cortical and behavioral responses resembling those of natural speech and music.