Movement Sonification During Haptic Exploration Shifts Emotional Outcome Without Altering Texture Perception

Abstract

Background:

Adding movement sonification to haptic exploration can change the perceptual outcome of a textured surface through multisensory processing. We hypothesized that auditory-evoked emotions influence the appraisal of textured surfaces, with corresponding changes reflected in cortical excitability.

Methods:

Twelve participants actively rubbed two different textured surfaces (slippery and rough) either without movement sonification, or with pleasant or disagreeable movement sonification.

Results and Discussion:

We found that sounds, whether agreeable or disagreeable, did not change the texture appraisal. However, the less pleasant surface was associated with a stronger negative hedonic valence, particularly when paired with disagreeable movement sonification. Time frequency analyses of electroencephalography (EEG) activities revealed a significant reduction in beta-band power [15–25 Hz] within the source-estimated sensorimotor and superior posterior parietal cortices when contrasting both pleasant and unpleasant sounds with the silent touch. This suggests that the primary somatosensory cortices together with the superior parietal regions participated in the audio-tactile binding, with both pleasant and unpleasant sounds. In addition, we observed a significant increase in beta-band power in medial visual areas, specifically when disagreeable movement sonification was paired with tactile exploration. This may reflect a disengagement of visual cortical processing, potentially amplifying auditory-driven emotional responses and intensifying the perceived unpleasantness of the explored surfaces.

Conclusion:

Our results offer new insights into the neural mechanisms by which hedonic valence of auditory signals modulates emotional processing, without disrupting the perceptual analysis of texture properties.

Graphical abstract

Keywords

movement sonification / EEG / audio-tactile processes / emotion

1. Introduction

Haptic exploration for extracting information about objects is generally considered a silent process. When vision is absent, the texture of an object is thus perceived through multisensory processing of tactile and proprioceptive information (i.e., haptic information). Interestingly, when haptic exploration becomes audible, the perception of texture can change. One of the most compelling demonstrations of tactile-auditory integration altering texture perception is the so-called parchment-skin illusion [1, 2]. Specifically, people experiencing this illusion perceive the palms of their hands as becoming either dry or moist, depending on the spectral content of the hand-friction sound played while rubbing hands together. Thus, adding movement-related sounds, a technique called movement sonification, can significantly alter the perceived texture, even though the tactile and proprioceptive inputs evoked during surface exploration remain unchanged. The primary objective of this technique, which is conceptually like the Foley effect in cinema, is to convey movement-related information and enhance both the perception and control of movement [3]. Movement sonification may compensate for the absence or reduction of sensory input, supporting hypotheses of auditory-proprioception substitution [4] and auditory-visual substitution [5]. Potential clinical applications for motor rehabilitation are therefore numerous: for example, individuals who are blind, use a limb prosthesis, have suffered a stroke, or present peripheral nerve injuries may benefit. Applied to pen movements, movement sonification has already been successfully used to improve handwriting performance [6, 7]. In texture discrimination tasks, haptic exploration has been shown to change in the presence of textured sounds compared to non-textured sounds, notably by increasing movement velocity [8].

A fundamental property of sound is its capacity to evoke a wide range of emotions [9, 10]. Despite this, little is known about how auditory-evoked emotions influence the appraisal of textured surfaces scanned by a finger. This gap persists although there is well-established evidence that our appreciation of objects is strongly influenced by the tactile sensations and emotions elicited when touching them. The present study aims to determine whether movement sonification during tactile exploration of a surface alters both the perceptual appraisal of the texture and, its emotional valence (ranging from “positive, pleasure” to “negative, displeasure”).

Changes in the perceptual and emotional appraisal of a texture can result from bottom-up and top-down mechanisms. From a bottom-up perspective, the Bayesian approach proposes that the central nervous system integrates information from different sensory modalities based on the estimated probability that it originates from a common or independent source [11]. The mechanisms underlying cross-modal interactions in the human brain remain largely unknown. This is particularly the case for audio-tactile interactions, despite their frequent occurrence in everyday life, for example, when hearing footsteps on a gravel path. This example complies with the “temporal principle” of multisensory integration stating that an interaction is most likely to be maximally effective when stimuli overlap in time [12]. Sound and touch are likely to respond to this principle as they are deeply entangled. Mechanical forces generated by physical contact not only activate somatosensory mechanoreceptors but also generate acoustic waves detectable by the auditory system [13]. To date, neurons responsive to both audio and tactile stimuli have been found in primates in the ventral premotor cortex, the ventral intraparietal region and the superior temporal sulcus [14, 15]. Further evidence indicates that audio-tactile interactions occur already at early stages of sensory processing. Notably, the somatosensory (S1) and auditory (A1) cortical regions, presumptive unimodal sensory areas, exhibit enhanced responses to bimodal stimuli relative to unimodal inputs [16]. This has been evidenced in animal studies by Lakatos et al. [17] showing that somatosensory inputs reset the phase of ongoing neuronal oscillations, so that accompanying auditory inputs arrive during an ideal, high-excitability phase, and produce amplified neuronal responses. In contrast, responses to auditory inputs arriving during the opposing low-excitability phase tend to be suppressed. Zhang et al. [18] extended these findings from primary cortical areas to secondary somatosensory cortice (S2). They showed in the mouse neocortex that, pairing sound with whisker stimulation modulates tactile responses in both S1 and S2. Sound responses of many sound-selective neurons in S2 neurons were spatially colocalized with S2 touch-selective neurons.

Complementing these bottom-up mechanisms, other approaches propose that the perception of a texture under our skin is not driven solely by somatosensory stimulation [19]. Rather, somatosensory processing would be under the influence of top-down control, which actively modulates, and routes sensory inputs based on expectation, attention, and the perceptual task [20, 21]. For example, pleasure induced by listening to pleasant music has been shown to require higher levels of attention compared to neutral music, thereby reducing the attentional resources available for other tasks [22]. This relationship between attention and the experience of pleasurable sounds can influence how sensory information is processed and integrated, as attentional states have been shown to modulate somatosensory processing [23].

Through top-down and bottom-up mechanisms, movement sonification may then modulate the hedonic valence of textures during haptic exploration, particularly when the associated sounds convey emotional attributes. In the present study, we hypothesized that texture exploration producing pleasant sounds through movement sonification would be rated as more pleasant, while textures associated with unpleasant sounds would be perceived as less pleasant. These perceptual effects of movement sonification may be linked to changes in beta power within cortical areas involved in processing somatosensory inputs (e.g., S1, Posterior Parietal Cortex (PPC)) as well as in regions engaged in evaluating emotional valence, such as the cingulate cortex. This hypothesis is consistent with previous work showing that beta-band (15–25 Hz) oscillations can be used to probe the relationship between cortical functioning and emotional dimension [24]. In line with this, studies have reported stronger beta response to unpleasant materials (e.g., pictures [25] or video [26, 27]). For example, using video, Mishra et al. [26] found higher fronto-temporo-parietal connectivity pattern that captured emotion specifically in the beta band, compared to other frequency bands.

2. Methods

2.1 Participants

Twelve right-handed adults (6 women, 6 men, mean age 26 \pm 2) were recruited. Behavioral, psychophysical, and neurophysiological data were collected from them. All participants gave their written informed consent to take part in this study, which conformed to the standards set in the Declaration of Helsinki. This study was approved by the Ethics Committee for research in Sciences et Techniques des Activités physiques et Sportives (CERSTAPS: IRB00012476-2021-09-12-140).

2.2 Task

Before the experiment, participants washed and dried their hands, then sat at a table where the experimental setup was positioned (Fig. 1). Prior to the recording session, they practiced sliding finger movements with their eyes closed on a 60 \times 25 mm flat surface, stroking in a disto-proximal direction several times. After each stroke, participants lifted their finger to return to the starting position on the surface and waited approximately one second before initiating the next movement. Movements were performed with the dominant hand, and participants freely chose sliding velocity, contact load, and finger-surface angle. Approximately 10 practice movements were required for participants to become comfortable with the task. During the experiment, at the experimenter’s signal, the participants closed their eyes and performed similar movements on the two test surfaces (see below) in four 25-second series, each separated by a ~60-second break. This procedure yielded approximately 40 recorded stroking movements per participant.

After completing the series of strokes, participants indicated whether they felt the texture they just explored as slippery/vibrating/rough (perceptual outcome) and rated their liking of the texture using a 4-levels hedonic scale (emotional outcome): I like it a lot/I like it/I don’t like it/I really don’t like it.

2.3 Surfaces

Two textured panels made of polyurethane resin (P1 and P2, 60 \times 25 mm) were selected based on the results by Faucheu et al. [28]. In their study with 43 participants, out of 51 textures tested, P1 was clearly qualified as slippery and P2 as rough. In addition, the hedonic ratings of these panels were highly consensual across their data set: P1 was generally rated as likeable, and P2 as disagreeable. The textured panels consisted of a flat surface patterned with cylindrical dots, evenly arranged in a hexagonal array. The panels varied in height, diameter and spacing between the cylindrical dots (see Table 1 and Fig. 2A, left panel).

2.4 Movement Sonification

In a pilot study, 15 participants who had not participated in the main experiment, were asked to listen to two 1-sec musical chords and rate their hedonic valence, specifically whether they perceived the music as agreeable or disagreeable. Several listens were sometimes required for participants to form an opinion. The sonification without false notes was generally rated as agreeable, whereas the sonification containing false notes was considered disagreeable by most participants.

Real-time sonification of the finger movement was ensured by an optical motion tracking system (sampling frequency: 100 Hz), using a Max/MSP® program (Cycling’74, San Francisco, CA, USA). The musical sonification was based on a mapping between the fingers’ positions and musical chords with or without the presence of false notes (e.g., audio recordings of the false notes are available at: https://www.gdr.tact.uha.fr/groupes-de-travail/comprendre-et-caracteriser/). The sonified movement was revealed by loudspeakers placed in front of the finger as it moved across the surface.

2.5 Recordings and Analyses

To record normal and tangential (shear) force data from the finger/surface interaction, the textured surface is mounted on the top of a home-made multi-sensors box comprising three resistive normal load sensors and one piezoelectric force sensor (Kistler 9217A, type 5011, Vienna, Austria). Data are acquired through a National Instrument data acquisition system (NI PCIe-6321, Austin, TX, USA) using a 10 kHz sampling frequency and recorded with Labview software (V. 19.0, National Instruments, Austin, TX, USA).

The vertical acceleration of the right index finger was recorded at 1000 Hz with a small and light (5 mm \times 12 mm \times 7 mm, 0.8 g) accelerometer (PCB 352A24, PCB Piezotronics, Inc., Depew, NY, USA) secured on the finger’s nail with wax (Fig. 1). With a high sensitivity of 100 mV/g and a measurement range of \pm50 g pk, this accelerometer can detect very low levels of acceleration. The onset and offset of the finger acceleration were used to define as the onset and offset of the stroking movement, respectively (Fig. 2A, right panel).

Fingertip movement kinematics were recorded at 22 Hz using a video camera (GV-5040CP-C-HQb, IDS, Obersulm, Germany) positioned above the participant’s hand. This enabled us to compare mean movement velocity across different surface and sonification conditions. To ensure that this mean value was representative, we verified whether finger velocity remained relatively constant throughout each stroking movement. For each stroke, finger coordinates were captured at every 10% of the movement trajectory. We found a linear increase in position across the 10 different points, with mean value of R² for each participant ranging from 0.982 to 0.997. Given these values were very close to 1, finger velocity can be considered constant across the movement for all conditions.

Electroencephalography (EEG) activity was continuously recorded from 64 Ag/AgCl surface electrodes embedded in an elastic cap (BioSemi ActiveTwo system: BioSemi, Amsterdam, Netherlands). Specific to the BioSemi system, “ground” electrodes were replaced by Common Mode Sense active (CMS) and Driven Right Leg (DRL) passive electrodes. Unlike a standard ground electrode, which simply provides a single reference point, the CMS and DRL electrodes (positioned near Pz and POz electrodes, respectively), act as an active reference system that reduces overall noise signal that accumulates in the system. This results in cleaner, more reliable EEG recording, and improved signal-to-noise ratio.

The signals were pre-amplified at the electrode sites, post amplified with DC amplifiers, and digitized at a sampling rate of 1024 Hz (Actiview acquisition program, BioSemi, Netherlands). The signals of each electrode were referenced to the mean signal of all electrodes. The continuous EEG signal was segmented into epochs aligned with the onset of the monotonic increase in shear forces generated by the finger on the surface. The average amplitude computed 100 ms prior to the force onset served as baseline. Three external Ag/AgCl electrodes (positioned next to the right and left external canthus and under the left eye orbit) and one from the cap (FP1) allowed controlling for ocular movements and blinks eventhough the participants had their eyes closed. After artifact rejection based on visual inspection, a mean of 35 trials for P1 and 32 trials for P2 were included in the analyses; the number of trials did not significantly differ between the 3 conditions of sonification (F_2,18 = 3.1; p > 0.05).

2.6 Cortical Sources

Neural sources were estimated with the dynamical Statistical Parametric Mapping (dSPM) [29] implemented in the Brainstorm software. A boundary element model (BEM) with three realistic layers (scalp, inner skull, and outer skull) was used to compute the forward model on the anatomical magnetic resonance imaging (MRI) brain template from the Montreal Neurological Institute (MNI Colin27). Using a realistic model has been shown to provide more accurate solution than a simple three concentric spheres model [30]. We used of a high number of vertices (i.e., 15002 vertices) to enhance the spatial resolution of the brain template. Such EEG source reconstruction has proved to be suited for investigating the activity of outer and inner cortical surfaces with 64 sensors [31, 32]. Measuring and modelling the noise contaminating the data is beneficial to source estimation. Noise covariance matrices were computed using the trials recorded while the index finger stayed stationary on the surface.

EEG data in source space were transformed into the time-frequency domain using the Hilbert transform in Brainstorm software [33]. The amplitude envelope (i.e., power) of beta band (mean 15–25 Hz, steps of 1 Hz) was extracted for each condition. Beta band event-related synchronization/desynchronization (ERS/ERD) was computed relative to a baseline window from –500 to –50 ms prior to finger movements and then averaged over 1000 ms interval starting at the onset of the strokes. This approach ensured that our ERS/ERD analyses captured most of the exploratory movement, encompassing approximately two-thirds of its total duration. Two participants performed very slow stroking movements and were excluded from EEG analysis, as the number of trials collected over the four 25-second series was insufficient for reliable EEG assessment.

2.7 Statistics

The normal distribution of data sets was assessed using the Kolmogorov–Smirnov test to validate the use of parametric tests. For the behavioral data, analyses of variance (ANOVAs) were conducted on movement duration, normal force, tangential force and mean movement velocity using the following design: Surface (2 types of surfaces: P1 and P2) \times Sonification (3 modalities: silent, agreeable, and disagreeable). Tukey HSD post-hoc tests were conducted to complete the ANOVA. For the categorical data on perceptual and emotional outcomes, Pearson chi-square (\chi²) tests were conducted to compare the two texture panels (P1 and P2) and the three sonification conditions. The EEG data were submitted to paired t-tests implemented in Brainstorm software. To control Type I error inflation in the whole-cortex EEG analyses, we applied the false discovery rate (FDR) correction. FDR controls the expected proportion of false positives when performing multiple comparisons in null hypothesis testing. This clarification has been added to the statistical section of the manuscript.

3. Results

3.1 Behaviour

The interactions between Surface and Sonification were not significant for any of the behavioral variables: stroke duration (F_2,22 = 0.39; p = 0.67); normal forces (F_2,22 = 1.80; p = 0.18) and tangential forces (F_2,22 = 1.37; p = 0.27) (Fig. 2B) and stroke velocity (F_2,22 = 1.34; p = 0.28). For simplicity and consistency, subsequent analyses were presented separately for the factors Surface and Sonification.

Although no specific instructions were given regarding the finger movement, stroke durations were slightly but significantly shorter on the slippery surface (P1: 1512 \pm 527 ms) compared to the rough surface (P2: 1668 \pm 586 ms), as confirmed by a significant surface effect (F_1,11 = 13.18; p 0.01). The shorter stroke duration observed on the slippery surface cannot be attributed to smaller shear (tangential) forces as these forces did not differ significantly between both surfaces (F_1,11 = 0.55; p = 0.47). On the other hand, the normal force exerted by the finger was unexpectedly higher on the P1 slippery surface (0.18 \pm 0.3 N) than on P2 rough surface (0.15 \pm 0.3 N) (F_1,11 = 21.73; p 0.01).

The forces (normal or tangential) exerted by the finger while rubbing the surfaces were not significantly affected by the movement sonification (F_2,22= 3.18; p = 0.061 and F_2,22 = 3.13; p = 0.063, for the normal and tangential forces, respectively).

The kinematics of the stroke movement was analysed to determine whether the textured panel or the movement sonification altered the fingertip velocity (Fig. 3A). The results showed that fingertip velocity was not significantly affected by the type of panel’s texture (P1, P2) (F_1,11 = 3.19; p = 0.10). Interestingly, the participants produced significantly slower movement on the P1 slippery surface for the agreeable sonification than for both silent (p = 0.001) and disagreeable (p = 0.029) stroke movements (F_2,22 = 9.16; p = 0.0012). Fig. 3B shows fingertip movement velocity expressed as a percentage relative to the silent strokes. When sonification was present, the velocity decreased by about 10%.

3.2 Perceptual Outcome

Non-parametric (\chi²) tests revealed that participants categorized P1 and P2 surfaces differently (\chi² = 168.61; p 0.001). P1 was overwhelmingly perceived as slippery (84% on average), whereas P2 was primarily identified as rough, though with less consensus (63% on average) (Fig. 4A). Additionally, P2 was also identified as vibrating in about one-third of the responses (34% on average). These surface categorizations were consistent with those reported by Faucheu et al. [28] using the same materials. The sonification, either agreeable or disagreeable, did not change the texture appraisal (i.e., perceptual outcome) as suggested by the absence of significant effect of the Pearson chi-square on P1 (\chi² = 5.61; p = 0.23) nor on P2 (\chi² = 2.73; p = 0.60).

3.3 Emotional Outcome

To examine the relationship between movement sonification conditions and surface hedonic judgement, emotional appraisal was evaluated using a 4-point Likert scale (as in Faucheu et al. [28], who used the same materials). Importantly, given our relatively small sample size (n = 12), a 4-point scale enhances response reliability compared to scales with more categories. By limiting the number of choices, it simplifies responses, reduces noise, avoids neutral ‘safe’ answers, and ensures clearer judgments, thereby yielding more interpretable data. For the slippery. P1, adding either a pleasant or unpleasant sound did not change the hedonic perception of the surface (p = 0.67). A striking finding is that only the rough, and less pleasant P2 surface showed a change in its hedonic valence depending on the sonification conditions (\chi² = 6.06; p = 0.04, Fig. 4B). The P2 surface was perceived as more unpleasant with disagreeable sonification compared to when it was stroked without movement sonification (p = 0.01). The pleasant sonification did not change the emotional output (p 0.05) (Fig. 4B).

3.4 Brain Activity

Given that the power of beta-band neural oscillations is inversely correlated with cortical processing [34, 35], contrasting the topographical distribution of beta power between both sonification conditions and the silent stroke condition can provide reliable estimate of the effect of movement sonification on cortical processing. For the P2 rough surface, the contrasts revealed a significant reduction in beta-band power within the source-estimated sensorimotor and superior parietal cortices during stroking under both sonification conditions, relative to the silent stroke condition (Fig. 5A, cold blue color). This decreased beta-band power (ERD) suggests amplified neuronal responses when the somatosensory inputs, particularly of tactile origin from the stroking movement, are paired with auditory inputs. However, in other cortical regions, beta-band power varied depending on the valence of the sonification.

The agreeable movement sonification elicited an ERD largely circumscribed to the posterior cingulate cortices (PCC) (Fig. 5A, left panel). In contrast, the disagreeable sonification induced an increased beta-band power (ERS) over both the right and left primary visual cortices relative to the silent condition (indicated by the warm orange color in Fig. 5A, right panel). The valence of the movement sonification had no effect on beta-band power when stroking the P1 slippery surface (Fig. 5B). Overall, pairing an auditory stimulus with finger movement significantly altered cortical processing when participants stroked the rough, unpleasant P2 surface, but had no observable effect during contact with the smooth, pleasant P1 surface.

These findings were further confirmed by extracting the averaged beta band power over 4 regions of interest (ROI): the left and right PCC (Brodmann areas 23/31) and the left and right visual areas (Brodmann areas 17/18) (see Table 2 for ROI sizes and for individual values). Contrasts revealed a significant decrease in beta-band power during agreeable sonification compared to silent stroke, bilaterally in the PCC ROI (t = 3.21; p = 0.01 and t = 2.49; p = 0.03, for the left and right PCC, respectively). In the visual ROI, beta band power significantly decreased in the right hemisphere when stroking was paired with the disagreeable sonification relative to the silent stroke (t = –2.57; p = 0.02), with a similar trend for the left visual ROI (t = –2.09; p = 0.06).

4. Discussion

The aim of this study was to investigate whether movement sonification alters the perceptual evaluation and hedonic valence of textured surfaces, as assessed through behavioural, psychophysical, and neurophysiological measures. The results show that texture appraisal is impervious to sonification conditions, whereas their emotional valence is not. This effect was observed with the P2 rough surface, which was classified as disagreeable and perceived as even more unpleasant when paired with disagreeable movement sonification. However, the brain’s response to the audio-tactile integration was altered by both the agreeable and the disagreeable movement sonifications. In both conditions where stroking movement evoked auditory inputs, we observed a marked decrease in beta-band oscillatory power over the sensorimotor cortex and the superior parietal cortex as compared to silent touch. Such decreases in beta-band power are commonly interpreted as reflecting increased cortical excitability and enhanced sensory processing [36, 37, 38, 39]. It is also in line with the functional magnetic resonance imaging (fMRI) study by Landelle et al.’s [16], which demonstrated the sensitivity of the primary somatosensory cortex to tactile input from the hand. The reason for the increased sensorimotor activity requires further investigation, and may reflect a combination of motor, emotional, and attentional processes. Nevertheless, our findings suggest that multisensory interactions are not confined to higher-order brain regions but also occur in early somatosensory areas, including primary somatosensory cortices and superior parietal regions.

This suggestion is supported by neuroanatomical studies showing direct thalamocortical projections to the superior parietal areas in macaques [40, 41, 42, 43]. The early multisensory interaction is thought to be mediated by means of phase resetting. The presentation of a stimulus to one sensory modality (e.g., auditory) resets the phase of ongoing oscillations in another modality (e.g., tactile) such that processing in the latter modality is modulated [17, 44].

While these data speak to the importance of primary somatosensory regions for audio-tactile information integration, evaluating the hedonic valence of movement sonification likely engages higher-order brain areas. One candidate region is the PCC, which exhibited a significant increase in beta band ERD (15–25 Hz) during agreeable sonification compared to silent touch. However, this change in beta band power was not accompanied by a corresponding change in the emotional rating of the scanned surface. It is possible that altering the hedonic valence of the touched surface would require a sound that is not merely free of dissonance but is explicitly pleasant. The fact that PCC beta-band power remained unchanged during disagreeable sonification suggests that this medial cortical region may be specifically involved in processing pleasantness. In this light, our findings are consistent with studies showing that the PCC is engaged in evaluating the emotional valence [45] and context of sensory information via pathways from somatosensory regions (superior and inferior parietal areas [46]), auditory regions [47] and adjacent cingulate areas such as the midcingulate cortex [48, 49, 50]. Due to these extensive connections, the PCC is considered to exhibit the highest degree of cross-network interaction [51].

When false notes were present while stroking the P2 rough surface, a bilateral decrease in activity was observed in the early visual cortex compared to silent touch. This was evidenced by an increase in event related synchronization ERS (red/warm color, Fig. 3A, right panel) in occipital regions (BA17/18). One could argue that this effect reflects task-related disengagement, given that participants’ eyes were closed. However, because the eyes were closed in all sonification conditions, the increased ERS observed in visual areas when contrasting beta power topography between the disagreeable and silent conditions cannot be attributed to the absence of visual feedback. The disengagement of visual cortical areas during disagreeable movement sonification may have played a crucial role in enhancing auditory-driven emotional responses. Surface texture is often co-deployed with a visual pattern, allowing the formation of a combined haptic-visual representation of the surface. The reduction in visual cortex activity may have redirected participants’ attention away from the visual attributes of the surface, thereby giving more weight to the disagreeable sonification and amplifying its emotional impact.

The potential mechanism explaining why participants rated the rough surface as even more unpleasant when their haptic exploration was paired with dissonant sonification is twofold. First, the visual cortical area, known to receive direct retinal inputs from the thalamic lateral geniculate nucleus, can be modulated by auditory signals. Signatures of auditory inputs in primary visual cortex were previously observed by Vetter et al.’s [52] fMRI study in absence of visual inputs [53]. Their results showed that the visual cortex contributes, via its corticocortical connections, to the encoding of complex natural sounds as abstract, higher-level internal representations [54], further highlighting the existence of multiple cortical representations of sounds.

Second, our results may be interpreted within the framework of the Bayesian causal inference model that postulates that multisensory integration is a consequence of an optimal probabilistic inference of each sensory modality [55]. Accordingly, both sensory sources, auditory and tactile, are integrated in the judgment task but with a different weight allocated to each sensory input, which depends on the level of relevance and accuracy of the processed information. For instance, for the surface P2, the addition of disagreeable sounds shifts the emotional response towards the unpleasant, as the emotional categorization of the tactile modality is less stabilized as shown by the emotional rating distribution. The lack of effect of sound conditions on neural excitability for the P1 slippery and agreeable surface suggests that the tactile modality has a greater weight than the auditory modality in this texture perception task, both at the perceptual and emotional levels. In other words, adding an unpleasant sound is not sufficient to override the agreeable tactile experience (P1). This observation may also point to the presence of a valence threshold effect; whereby only unpleasant textures are subject to modulation. In line with this interpretation, Ackerley et al. [56] reported that listening to classical music evoking sad, neutral, or happy emotions alters the sense of limb position (i.e., proprioception, not tactile), by increasing muscular fiber sensitivity to muscle shortening/lengthening. This effect was found for sad music, but not for happy music, relative to neutral, further suggesting a valence threshold effect.

5. Limitations and Perspectives

This study shows that the hedonic valence of auditory signals can modulate emotional aspects of touch without altering texture perception and, outlines possible underlying brain mechanisms. However, it presents some limitations. The most notable concerns the relatively small sample size and the limited set of textured surfaces tested which can both restrict the generalizability of the findings. Future studies should also consider individual differences in sensory (audio, tactile) discrimination abilities and in the emotion elicited by touch and sound. Indeed, we know that sensitivity to sounds and music [57], and to haptic information [58] varies across individuals. Exploring both sensory domains in tandem would provide a more comprehensive understanding of the complex interplay between tactile and auditory processing.

6. Conclusion

Our results offer new insights into the neural mechanisms by which hedonic valence of auditory signals modulates haptic emotional processing, without disrupting the perceptual analysis of texture properties. More precisely, texture appraisal seems unaffected by sonification conditions, whereas their emotional valence is clearly influenced. This effect was evident for the disagreeable texture, which was perceived as even more unpleasant when paired with disagreeable sonification. At the neural level, audio-tactile integration was altered by both the agreeable and the disagreeable movement sonifications. These findings confirm that the emotional valence of the texture perception arises not only from tactile input, but also from multisensory processing. Finally, augmented audio reality offers promising clinical applications for modulating the emotional and sensory experience when interacting with physical objects in individuals with sensory or emotional regulation deficits.

Disclosure

The paper is listed as, “Movement sonification during haptic exploration shifts emotional outcome without altering texture perception” as a preprint on BioRxiv at: https://www.biorxiv.org/content/10.1101/2025.05.28.656569v1.abstract.

Availability of Data and Materials

The data generated and analyzed during the present study are not publicly available for ethical reasons but are available from the corresponding author on reasonable request.

References

[1] Champoux F, Collignon O, Bacon BA, Lepore F, Zatorre RJ, Théoret H. Early- and late-onset blindness both curb audiotactile integration on the parchment-skin illusion. Psychological Science. 2011; 22: 19–25. https://doi.org/10.1177/0956797610391099.

[2] Jousmäki V, Hari R. Parchment-skin illusion: sound-biased touch. Current Biology. 1998; 8: R190–R191. https://doi.org/10.1016/s0960-9822(98)70120-4.

[3] Effenberg AO. Movement sonification: Effects on perception and action. IEEE Multimedia. 2005; 12: 53–59. https://doi.org/10.1109/MMUL.2005.31.

[4] Danna J, Velay JL. On the Auditory-Proprioception Substitution Hypothesis: Movement Sonification in Two Deafferented Subjects Learning to Write New Characters. Frontiers in Neuroscience. 2017; 11: 137. https://doi.org/10.3389/fnins.2017.00137.

[5] Renier L, Collignon O, Poirier C, Tranduy D, Vanlierde A, Bol A, et al. Cross-modal activation of visual cortex during depth perception using auditory substitution of vision. NeuroImage. 2005; 26: 573–580. https://doi.org/10.1016/j.neuroimage.2005.01.047.

[6] Danna J, Velay JL. Handwriting movement sonification: why and how? IEEE Transactions on Human-Machine Systems. 2017; 47: 299–303. https://doi.org/10.1109/THMS.2016.2641397.

[7] Véron-Delor L, Pinto S, Eusebio A, Azulay JP, Witjas T, Velay JL, et al. Musical sonification improves motor control in Parkinson’s disease: a proof of concept with handwriting. Annals of the New York Academy of Sciences. 2020; 1465: 132–145. https://doi.org/10.1111/nyas.14252.

[8] Landelle C, Danna J, Nazarian B, Amberg M, Giraud F, Pruvost L, et al. The impact of movement sonification on haptic perception changes with aging. Scientific Reports. 2021; 11: 5124. https://doi.org/10.1038/s41598-021-84581-3.

[9] Dubé L, Le Bel J. The content and structure of laypeople’s concept of pleasure. Cognition & Emotion. 2003; 17: 263–295. https://doi.org/10.1080/02699930302295.

[10] Mithen S. The music instinct: the evolutionary basis of musicality. Annals of the New York Academy of Sciences. 2009; 1169: 3–12. https://doi.org/10.1111/j.1749-6632.2009.04590.x.

[11] Körding KP, Beierholm U, Ma WJ, Quartz S, Tenenbaum JB, Shams L. Causal inference in multisensory perception. PLoS ONE. 2007; 2: e943. https://doi.org/10.1371/journal.pone.0000943.

[12] Meredith MA, Nemitz JW, Stein BE. Determinants of multisensory integration in superior colliculus neurons. I. Temporal factors. The Journal of Neuroscience. 1987; 7: 3215–3229. https://doi.org/10.1523/JNEUROSCI.07-10-03215.1987.

[13] Von Békésy G. Neural funneling along the skin and between the inner and outer hair cells of the cochlea. The Journal of the Acoustical Society of America. 1959; 31: 1236–1249. https://doi.org/10.1121/1.1907851.

[14] Graziano MS, Reiss LA, Gross CG. A neuronal representation of the location of nearby sounds. Nature. 1999; 397: 428–430. https://doi.org/10.1038/17115.

[15] Schlack A, Sterbing-D’Angelo SJ, Hartung K, Hoffmann KP, Bremmer F. Multisensory space representations in the macaque ventral intraparietal area. The Journal of Neuroscience. 2005; 25: 4616–4625. https://doi.org/10.1523/JNEUROSCI.0455-05.2005.

[16] Landelle C, Caron-Guyon J, Nazarian B, Anton JL, Sein J, Pruvost L, et al. Beyond sense-specific processing: decoding texture in the brain from touch and sonified movement. iScience. 2023; 26: 107965. https://doi.org/10.1016/j.isci.2023.107965.

[17] Lakatos P, Chen CM, O’Connell MN, Mills A, Schroeder CE. Neuronal oscillations and multisensory interaction in primary auditory cortex. Neuron. 2007; 53: 279–292. https://doi.org/10.1016/j.neuron.2006.12.011.

[18] Zhang M, Kwon SE, Ben-Johny M, O’Connor DH, Issa JB. Spectral hallmark of auditory-tactile interactions in the mouse somatosensory cortex. Communications Biology. 2020; 3: 64. https://doi.org/10.1038/s42003-020-0788-5.

[19] Engel AK, Fries P, Singer W. Dynamic predictions: oscillations and synchrony in top-down processing. Nature Reviews. Neuroscience. 2001; 2: 704–716. https://doi.org/10.1038/35094565.

[20] Gilbert CD, Sigman M. Brain states: top-down influences in sensory processing. Neuron. 2007; 54: 677–696. https://doi.org/10.1016/j.neuron.2007.05.019.

[21] Talsma D, Senkowski D, Soto-Faraco S, Woldorff MG. The multifaceted interplay between attention and multisensory integration. Trends in Cognitive Sciences. 2010; 14: 400–410. https://doi.org/10.1016/j.tics.2010.06.008.

[22] Nemati S, Akrami H, Salehi S, Esteky H, Moghimi S. Lost in music: Neural signature of pleasure and its role in modulating attentional resources. Brain Research. 2019; 1711: 7–15. https://doi.org/10.1016/j.brainres.2019.01.011.

[23] Wiesman AI, Wilson TW. Attention modulates the gating of primary somatosensory oscillations. NeuroImage. 2020; 211: 116610. https://doi.org/10.1016/j.neuroimage.2020.116610.

[24] Ray WJ, Cole HW. EEG alpha activity reflects attentional demands, and beta activity reflects emotional and cognitive processes. Science. 1985; 228: 750–752. https://doi.org/10.1126/science.3992243.

[25] Güntekin B, Başar E. Event-related beta oscillations are affected by emotional eliciting stimuli. Neuroscience Letters. 2010; 483: 173–178. https://doi.org/10.1016/j.neulet.2010.08.002.

[26] Mishra S, Srinivasan N, Tiwary US. Dynamic Functional Connectivity of Emotion Processing in Beta Band with Naturalistic Emotion Stimuli. Brain Sciences. 2022; 12: 1106. https://doi.org/10.3390/brainsci12081106.

[27] Wang XW, Nie D, Lu BL. Emotional state classification from eeg data using machine learning approach. Neurocomputing. 2014; 129: 94–106. https://doi.org/10.1016/j.neucom.2013.06.046.

[28] Faucheu J, Weiland B, Juganaru-Mathieu M, Witt A, Cornuault PH. Tactile aesthetics: Textures that we like or hate to touch. Acta Psychologica. 2019; 201: 102950. https://doi.org/10.1016/j.actpsy.2019.102950.

[29] Dale AM, Liu AK, Fischl BR, Buckner RL, Belliveau JW, Lewine JD, et al. Dynamic statistical parametric mapping: combining fMRI and MEG for high-resolution imaging of cortical activity. Neuron. 2000; 26: 55–67. https://doi.org/10.1016/s0896-6273(00)81138-1.

[30] Sohrabpour A, Lu Y, Kankirawatana P, Blount J, Kim H, He B. Effect of EEG electrode number on epileptic source localization in pediatric patients. Clinical Neurophysiology. 2015; 126: 472–480. https://doi.org/10.1016/j.clinph.2014.05.038.

[31] Chand GB, Dhamala M. Interactions between the anterior cingulate-insula network and the fronto-parietal network during perceptual decision-making. NeuroImage. 2017; 152: 381–389. https://doi.org/10.1016/j.neuroimage.2017.03.014.

[32] Ponz A, Montant M, Liegeois-Chauvel C, Silva C, Braun M, Jacobs AM, et al. Emotion processing in words: a test of the neural re-use hypothesis using surface and intracranial EEG. Social Cognitive and Affective Neuroscience. 2014; 9: 619–627. https://doi.org/10.1093/scan/nst034.

[33] Tadel F, Baillet S, Mosher JC, Pantazis D, Leahy RM. Brainstorm: a user-friendly application for MEG/EEG analysis. Computational Intelligence and Neuroscience. 2011; 2011: 879716. https://doi.org/10.1155/2011/879716.

[34] Cheyne D, Gaetz W, Garnero L, Lachaux JP, Ducorps A, Schwartz D, et al. Neuromagnetic imaging of cortical oscillations accompanying tactile stimulation. Brain Research. Cognitive Brain Research. 2003; 17: 599–611. https://doi.org/10.1016/s0926-6410(03)00173-3.

[35] Pfurtscheller G, Lopes da Silva FH. Event-related EEG/MEG synchronization and desynchronization: basic principles. Clinical Neurophysiology: Official Journal of the International Federation of Clinical Neurophysiology. 1999; 110: 1842–1857. https://doi.org/10.1016/s1388-2457(99)00141-8.

[36] Haegens S, Händel BF, Jensen O. Top-down controlled alpha band activity in somatosensory areas determines behavioral performance in a discrimination task. The Journal of Neuroscience. 2011; 31: 5197–5204. https://doi.org/10.1523/JNEUROSCI.5199-10.2011.

[37] Kilavik BE, Zaepffel M, Brovelli A, MacKay WA, Riehle A. The ups and downs of β oscillations in sensorimotor cortex. Experimental Neurology. 2013; 245: 15–26. https://doi.org/10.1016/j.expneurol.2012.09.014.

[38] Romei V, Brodbeck V, Michel C, Amedi A, Pascual-Leone A, Thut G. Spontaneous fluctuations in posterior alpha-band EEG activity reflect variability in excitability of human visual areas. Cerebral Cortex. 2008; 18: 2010–2018. https://doi.org/10.1093/cercor/bhm229.

[39] Sauseng P, Klimesch W, Gerloff C, Hummel FC. Spontaneous locally restricted EEG alpha activity determines cortical excitability in the motor cortex. Neuropsychologia. 2009; 47: 284–288. https://doi.org/10.1016/j.neuropsychologia.2008.07.021.

[40] Impieri D, Gamberini M, Passarelli L, Rosa MGP, Galletti C. Thalamo-cortical projections to the macaque superior parietal lobule areas PEc and PE. The Journal of Comparative Neurology. 2018; 526: 1041–1056. https://doi.org/10.1002/cne.24389.

[41] Padberg J, Cerkevich C, Engle J, Rajan AT, Recanzone G, Kaas J, et al. Thalamocortical connections of parietal somatosensory cortical fields in macaque monkeys are highly divergent and convergent. Cerebral Cortex. 2009; 19: 2038–2064. https://doi.org/10.1093/cercor/bhn229.

[42] Pearson RC, Brodal P, Powell TP. The projection of the thalamus upon the parietal lobe in the monkey. Brain Research. 1978; 144: 143–148. https://doi.org/10.1016/0006-8993(78)90440-7.

[43] Pons TP, Kaas JH. Connections of area 2 of somatosensory cortex with the anterior pulvinar and subdivisions of the ventroposterior complex in macaque monkeys. The Journal of Comparative Neurology. 1985; 240: 16–36. https://doi.org/10.1002/cne.902400103.

[44] Lakatos P, O’Connell MN, Barczak A, Mills A, Javitt DC, Schroeder CE. The leading sense: supramodal control of neurophysiological context by attention. Neuron. 2009; 64: 419–430. https://doi.org/10.1016/j.neuron.2009.10.014.

[45] Maddock RJ, Garrett AS, Buonocore MH. Posterior cingulate cortex activation by emotional words: fMRI evidence from a valence decision task. Human Brain Mapping. 2003; 18: 30–41. https://doi.org/10.1002/hbm.10075.

[46] Vogt BA. Submodalities of emotion in the context of cingulate subregions. Cortex; a Journal Devoted to the Study of the Nervous System and Behavior. 2014; 59: 197–202. https://doi.org/10.1016/j.cortex.2014.04.002.

[47] Vogt BA, Laureys S. The primate posterior cingulate gyrus: connections, sensorimotor orientation, gateway to limbic processing. Cingulate neurobiology and disease (pp. 275–308). Oxford Academic: UK. 2009. https://doi.org/10.1093/oso/9780198566960.003.0013.

[48] Grabenhorst F, Rolls ET. Value, pleasure and choice in the ventral prefrontal cortex. Trends in Cognitive Sciences. 2011; 15: 56–67. https://doi.org/10.1016/j.tics.2010.12.004.

[49] Rolls ET. The cingulate cortex and limbic systems for emotion, action, and memory. Brain Structure & Function. 2019; 224: 3001–3018. https://doi.org/10.1007/s00429-019-01945-2.

[50] Vogt BA. Midcingulate cortex: Structure, connections, homologies, functions and diseases. Journal of Chemical Neuroanatomy. 2016; 74: 28–46. https://doi.org/10.1016/j.jchemneu.2016.01.010.

[51] Betti V, Della Penna S, de Pasquale F, Corbetta M. Spontaneous Beta Band Rhythms in the Predictive Coding of Natural Stimuli. The Neuroscientist: a Review Journal Bringing Neurobiology, Neurology and Psychiatry. 2021; 27: 184–201. https://doi.org/10.1177/1073858420928988.

[52] Vetter P, Smith FW, Muckli L. Decoding sound and imagery content in early visual cortex. Current Biology. 2014; 24: 1256–1262. https://doi.org/10.1016/j.cub.2014.04.020.

[53] Manson GA, Tremblay L, Lebar N, de Grosbois J, Mouchnino L, Blouin J. Auditory cues for somatosensory targets invoke visuomotor transformations: Behavioral and electrophysiological evidence. PLoS ONE. 2019; 14: e0215518. https://doi.org/10.1371/journal.pone.0215518.

[54] Petro LS, Paton AT, Muckli L. Contextual modulation of primary visual cortex by auditory signals. Philosophical Transactions of the Royal Society of London. Series B, Biological Sciences. 2017; 372: 20160104. https://doi.org/10.1098/rstb.2016.0104.

[55] Kayser C, Shams L. Multisensory causal inference in the brain. PLoS Biology. 2015; 13: e1002075. https://doi.org/10.1371/journal.pbio.1002075.

[56] Ackerley R, Aimonetti JM, Ribot-Ciscar E. Emotions alter muscle proprioceptive coding of movements in humans. Scientific Reports. 2017; 7: 8465. https://doi.org/10.1038/s41598-017-08721-4.

[57] Garrido S, Schubert E. Individual differences in the enjoyment of negative emotion in music: A literature review and experiment. Music Perception. 2011; 28: 279–296. https://doi.org/10.1525/mp.2011.28.3.279.

[58] Peck J, Childers TL. Individual differences in haptic information processing: The “Need for Touch” scale. Journal of Consumer Research. 2003; 30: 430–442. https://doi.org/10.1086/378619.

Funding

COMTACT(ANR 2020-CE28-0010-03)