Title: Encoding of Linguistic Constituents in Human Brain and LLMs
Prof. Nai DING (Zhejiang University)
Abstract: Classic linguistic theories propose that parsing a sentence into constituents is a necessary step for sentence comprehension. Apparent counterevidence for these theories is that natural language comprehension has recently been well tackled by large language models (LLMs) such as ChatGPT, which do not explicitly encode any linguistic constituent. In the first part of the talk, I will present a study that investigate whether the human brain is sensitive to linguistic constituents during speech comprehension using magnetoencephalography (MEG). In the second part of the talk, I will present a study that tests behavioral sensitivity to linguistic constituents for both human participants and ChatGPT using a novel one-shot learning task. These studies demonstrate that the constituency structure strongly influences speech representation in the human brain and a representation of constituency structure also emerges in ChatGPT.
Medium of the presentation: English
Bio of the speaker: Prof. Nai DING received his bachelor's degree from Zhejiang University and his PhD degree from University of Maryland College Park, both in the field of electrical engineering. After being a postdoc researcher at the Psychology Department of New York University, he joined the College of Biomedical Engineering and Instrument Science of Zhejiang University in 2015. He studies the neural basis of speech and language processing, as well as the underlying computational mechanisms. He has published over 50 papers in journals such as Nature Neuroscience, Nature Communications, and PNAS, and in conferences such as AAAI and ACL, and received the APAN Young Investigator Award in 2023.
Title： Multimodal and contextual prediction in speech perception and comprehension
Prof. Yi DU (University of Chinese Academic of Sciences)
Abstract: Speech perception and comprehension are inherently multimodal, with verbal communication often accompanied by visual speech (e.g, lip movements) and hand gestures. The human brain also integrates external stimuli with predictions drawn from long-term memory and contextual clues to aid in speech comprehension. However, the intricate neural mechanisms responsible for harnessing this multimodal and contextual information to predict multi-level linguistic features, from phonetics to semantics, remain veiled in mystery. In the first part of my presentation, I will focus on auditory-motor integration in speech perception, unveiling the causal role of the motor cortex in providing predictive articulatory templates during the perceptual decision-making process for speech lexical tones and consonant voicing. Subsequently, I will elucidate the dynamic brain circuitry mechanisms that underlie the cross-modal probabilistic semantic updating and integration between speech and gestures. Lastly, I will present new data delving into neural representations of phonological and semantic information as well as distinct neural pathways involved when predicted speech is confirmed, absent or violated.
Medium of Presentation: English
Bio of the speaker: Prof. Yi DU is the PI of Lab of Auditory, Speech, and Music at the Institute of Psychology, Chinese Academy of Sciences. Combining psychophysics with neuroimaging (fMRI, MEG and EEG) and neuromodulation (TMS and tES) techniques, she studies how human brain processes speech and music, with particular interest in 1) neural mechanisms underlying speech perception and comprehension; 2) life-span development of speech comprehension ability; 3) musical experience-related plasticity in speech processing; and 4) neural mechanisms underlying musical perception and reward. She has published papers in high-profile journals, including Nature Communications, Science Advances, PNAS, Neuroimage, Journal of Neuroscience, and Neuroscience & Biobehavioral Reviews.
Title: Understanding Meaning in the Brain: From Words to Information Acquisition
Prof. Qing CAI (East China Normal University)
Abstract: How are semantics represented and organized in the human brain? Over the years, various language models have been developed, each founded on distinct computational principles and linguistic datasets. Our recent work introduced SWOW-ZH, a Chinese word association database, to examine how different types of models account for human behavioral data and neural responses related to lexical semantics. Our findings revealed that association-based internal models outperform external models trained on text corpora in predicting human behavioral data and neural activation patterns during word reading, highlighting the brain's multidimensional semantic organization. Additionally, I will discuss our latest research on the cognitive mechanisms underlying reading comprehension during natural text reading. We argue that entropy reduction, indicative of the cognitive neural process of information acquisition, can be distinctly separated from the concept of processing difficulty, thereby offering a nuanced understanding of language comprehension.
Medium of Presentation: English
Bio of the speaker: Prof. Qing CAI is a professor of psychology and the director of the Children’s Brain Imaging Centre at East China Normal University, China. Her lab aims to investigate the neural mechanisms underlying language and cognitive development. Her recent research, integrating insights from naturalistic stimuli and Natural Language Processing (NLP) techniques, explores semantic processing and language acquisition, with an emphasis on individual differences and their relationship to higher-level cognitive functions.
Title: Language impairment and early screening in Alzheimer's disease
Prof. Zude Zhu (Jiangsu Normal University)
Abstract: Alzheimer's disease is a comprehensive degenerative cognitive disorder, and the number of Alzheimer's diseases in China exceeds 10 million, ranking first in the world. Existing studies have shown that cognitive impairment, including language, can be present up to 10 years before diagnosis, providing an important pathway for early screening of Alzheimer's disease. Based on a detailed review of the literature, the present study investigated the characteristics of language impairment in Chinese patients with Alzheimer's disease. Critically, the study has established an early Alzheimer's disease screening model by using automatic language impairment feature analysis through the comparison of Alzheimer's disease, mild cognitive impairment, subjective cognitive impairment and healthy control groups.
Medium of Presentation: Chinese
Bio of the speaker: 朱祖德，就职于江苏师范大学语言科学与艺术学院，江苏特聘教授。研究方向为语言障碍，近期主要围绕阿尔茨海默症智能筛查、基于神经网络可塑性的语言认知康复训练等主题开展研究。主持国家社会科学基金重大项目、国家自然科学基金项目多项，在国际期刊发表论文50余篇，曾获高等教育国家级教学成果奖二等奖、江苏省哲社优秀成果一等奖、广东省哲社优秀成果一等奖、中华医学科技奖一等奖等奖励多项。
Title: The Dawn of Speech: Immediate Phonemic Discrimination Learning Within Hours of Birth
Prof. Yanjing WU (Ningbo University)
Abstract: Within the first breaths of life, newborns embark on an astonishing journey of linguistic discovery, defying previously held beliefs about the timing and mechanisms of language acquisition. Cutting-edge research has peeled back the layers of the neonatal brain, revealing its innate prowess in distinguishing between complex sounds—natural versus artificially reversed vowels—with an agility that challenges our understanding of early cognitive development. This exploration into the neonate's brain illuminates a remarkable capacity for rapid, sophisticated learning, mere hours post-birth. The revelations from these findings not only captivate us with the sheer potential of the human mind from its earliest moments but also pave the way for groundbreaking approaches in detecting and understanding neurodevelopmental disorders. As we delve deeper into the neural orchestration behind this early phonemic discernment, we stand on the brink of revolutionizing our approach to early childhood development and intervention strategies.
Medium of Presentation: English
Bio of the speaker: Prof. Yan Jing WU received his undergraduate degree in psychology from Bangor University, where he pursued a PhD in cognitive neuroscience. Yan Jing’s interests in science lie in two areas: Bilingualism and the interaction between language and cognition. His studies apply experimental psychology and neuroscientific methods to examine the neural mechanisms associated with language processing in bilinguals and the consequences of bilingualism on cognitive control. Findings reveal an online interaction between bilingual’s two languages when functioning in a single-language mode, indicating an integrated lexical representation as proposed in psycholinguistic and computational models of bilingualism. Yan Jing also explored the development of language in the beginning of human life by revealing neural specification underlying neonate’s speech perception ability. Yan Jing’s research have brought novel insights and expanded our views of how language is acquired and functions across a wide range of population.
Title: Neural Correlates of Social Semantic Working Memory
Dr Nan LIN (Chinese Academy of Sciences)
Abstract: Humans are social animals. People use language to convey a vast amount of social semantic information. In a series of functional magnetic resonance imaging (fMRI) studies, we have found that social semantic processing is selectively associated with a set of brain areas located in the default mode network. Furthermore, we found that the sensitivity of the left ventral temporoparietal junction (vTPJ) and lateral anterior temporal lobe (lATL) to sentences is associated with social-semantic processing. We propose that these two regions are involved in social semantic working memory. We examined this hypothesis using fMRI and transcranial magnetic stimulation. The findings all support the function of the left vTPJ and lATL in the social semantic working memory.
Medium of the presentation: Chinese
Bio of the speaker: Dr Nan LIN obtained his PhD from Beijing Normal University and is currently an associate professor at the Institute of Psychology, Chinese Academy of Sciences. His research focuses on the representation and processing of different types of semantic information in the brain. His research has been published in journals including Nature Human Behaviour, Cerebral Cortex, and Neuroimage.
Title: Neurocognitive bases of developmental disorders of speech, language and literacy: The role of domain-general learning systems
Dr Caicai ZHANG (The Hong Kong Polytechnic University)
Abstract: Despite their importance, competence at speech, language, and literacy is not universal. Various neurodevelopmental speech and language disorders can impact these critical abilities in children, including developmental stuttering (DS), developmental language disorder (DLD) and developmental dyslexia (DD). Each of these disorders affect a significant portion of children; they also co-occur frequently. However, the neurocognitive bases of these disorders and their relationships with each other are not well-understood. In this talk, I will report initial findings from a large-scale, theory-driven study that investigated how procedural and declarative learning, including their behavioral manifestations, brain anatomy, and memory consolidation, contribute to typical and atypical speech, language, and literacy development in Chinese children. Eighty-nine L1 Cantonese school-age children aged between 6 and 12 years old (including 34 typically developing children and 55 children with DS, DLD, DD or their comorbidities) completed procedural and declarative learning and overnight memory retention tasks via a two-day design. Among the participants, 40 children (including 11 typically developing children and 29 children with disorders) also provided sMRI and DTI data. Preliminary data analysis yielded some support for the hypothesized procedural circuity deficit in Chinese children with these disorders, but also suggested a nuanced picture of their neurocognitive difficulties. The findings should advance our understanding of the neurocognition of these deleterious disorders and their brain-cognition-language interrelations in Chinese children, laying the foundation for the development of evidence-based diagnosis and intervention approaches.
Medium of the presentation: English
Bio of the speaker: Dr. Caicai Zhang is Associate Professor in the Research Centre for Language, Cognition and Neuroscience at the Department of Chinese and Bilingual Studies of the Hong Kong Polytechnic University. She is also an affiliated member of the Speech Therapy Unit. Her work broadly concentrates on how language is processed and represented in the human brain, and how it interacts with other cognitive domains such as music and memory. At a more specific level, her research focuses on: (1) the relationship of language, music and brain, investigated from the perspective of how congenital amusia (commonly known as tone deafness), an innate musical pitch disorder, affects speech processing from behavioral to neural levels; (2) how sleep contributes to the consolidation of speech sound categories in the brain of L2 learners; and (3) the neurocognitive mechanisms of learning and memory problems in children with developmental disorders of speech and language. She uses a broad range of tools including behavioral methods, electroencephalograph (EEG) and functional MRI (fMRI) to study these questions. She has published more than 40 papers in international peer-reviewed journals, including Brain and Language, Scientific Reports, NeuroImage, and NeuroImage: Clinical. She is Editor/Associate Editor/Academic Editor for Scientific Reports, PLOS ONE, and Frontiers in Psychology.