Neural correlates of probabilistic information processing in gesture-speech integration

How information of different modalities (e.g., speech and gesture) frame into a unified one remains unclear in the neurobiology of language. Leveraging the information-theoretic complexity metrics of entropy and mutual information (MI) to quantify gestures and speech, the present event-related potentials and chronometric double-pulse transcranial magnetic stimulation (TMS) study investigated the neural processes during probabilistic gesture-speech integration. In the early processing stage, we found a positive correlation of P1 amplitude with speech entropy and a negative correlation of N1-P2 amplitude with gesture entropy, while the TMS disruption effect of posterior middle temporal gyrus (pMTG) was negatively correlated with speech entropy. In the lexico-semantic processing stage, we observed a negative correlation of N400 amplitude with both gesture entropy and gesture-speech MI, a negative correlation of late positive component amplitude with speech entropy, together with correlations of the TMS interruption effect of inferior frontal gyrus (IFG) with gesture entropy, speech entropy and their MI. Our findings suggest a probabilistically updated lexical representation that was dynamically framed by top-down gesture constrain, bottom-up speech encoding and their interplay. A 5-step process in the pMTG-IFG-pMTG neural circuit is hypothesized for probabilistic gesture–speech processing, which promises a better understanding of the predictive multisensory language processing.