Author(s)

ShuoPei Yang, NaNa Wang^*

Affiliation(s)

Lingjing Jushen (Ningbo) Electronic Technology Co., Ltd., Ningbo 31500, Zhejiang, China.

Corresponding Author

NaNa Wang

ABSTRACT

This study focuses on enhancing affective computing and multimodal interaction capabilities in social humanoid robots to improve natural communication experiences between robots and humans. By constructing a system framework that integrates affective computing with multimodal interaction, it addresses the key challenges of insufficient accuracy in emotion recognition and lack of naturalness in emotional expression. In the research design, we established a comprehensive system architecture encompassing multimodal emotional feature extraction, emotion state inference algorithms, and emotional expression strategies. At the interaction implementation level, a multimodal interaction system comprising perception, decision-making, and execution layers was designed to ensure effective processing of multi-source information such as voice, vision, and touch. Experimental results demonstrate significant improvements in key metrics including emotion recognition accuracy, interaction fluency, and user experience. Statistical analysis further validates the effectiveness of the proposed method. This research not only provides innovative technical solutions for social robots but also contributes substantially to both theoretical development and practical applications in the field of human-computer interaction.

KEYWORDS

Humanoid robot; Emotion recognition; Human-computer interaction; Multimodal fusion

CITE THIS PAPER

ShuoPei Yang, NaNa Wang. Affective computing and multimodal interaction for social humanoid robots. Journal of Computer Science and Electrical Engineering. 2025, 7(6): 71-82. DOI: https://doi.org/10.61784/jcsee3096.

REFERENCES

[1] Xiahou X, Tian F, L, Q. A Review of Human-Robot Collaboration Safety Research in the Context of Smart Construction. Journal of Southeast University (Natural Science Edition), 2023, 53(6): 1053-1064.

[2] Liu J, Li B, Tian W, et al. Dual Robot Datum Feature Detection and Position Compensation Technology. Machine Building & Automation, 2024, 53(5): 224-228.

[3] Faccio M, Granata I, Mintor T. Task Allocation Model for Human-Robot Collaboration with Variable Cobot Speed. Journal of Intelligent Manufacturing, 2024, 35: 793-806.

[4] Mao J, Ding Y, Wang Z, et al. Design of a Comprehensive Experimental Project for Industrial Robot Hand Eye Calibration. Research and Exploration in Laboratory, 2024, 43(3): 51-56.

[5] Shi Z, Cheng H. Status and Prospects of Domestic and International Research on Robot Actuator Testing Technology. Journal of Harbin Engineering University, 2023, 44(5): 679-688.

[6] Zhao Y, Wang Z, Li B, et al. Review on the Status Quo and Trend of Robot Core Parts Technology. Machine Tool & Hydraulics, 2024, 52(3): 196-202.

[7] Liu J, Wang W, Han J, et al. Research on the Digital Divide and Countermeasures for Online Health Services for the Elderly in the Era of Digital Healthcare. Journal of Library and Information Science, 2024(09): 41-45.

[8] Zhao B. Behind Information Disorder: The Impact of Social Bots on Online Communication Order. Youth Journalist, 2023(02): 23-27.

[9] Teng Z. Research on Interactive Control Method of Lower Limb Rehabilitation Robot Based on Rigid-Flexible Coupling. Changchun: Changchun University of Technology, 2021.

[10] Xiang S. Research and Implementation of Gait Planning Algorithm for Assisted Exoskeleton Up and Down Stairs. Chengdu: University of Electronic Science and Technology of China, 2020.

[11] Biro I, Kinnell P. Performance Evaluation of a Robot-Mounted Interferometer for an Industrial Environment. Sensors, 2020, 20(1). DOI: 10.3390/s20010257.

[12] British Standards Institution. Industrial, commercial and garage doors and gates. Safety in use of power operated doors. Requirements and test methods: BS EN 12453:2017+A1:2021,  2020.

[13] Sun X, Li J, Wei X. Emotional Editing Constraint Conversation Generation Based on Reinforcement Learning. Acta Automatica Sinica, 2025.

[14] Yang Y, Zhang Z. A Personalized Emotion Model Based on PAD. Journal of Chongqing University of Posts and Telecommunications (Natural Science Edition), 2012, 24(1): 96–103. DOI: 10.3979/j.issn.1673-825X.2012.01.019.

[15] Huang K, Shi B, Li X, et al. Multi-modal Sensor Fusion for Auto Driving Perception: A Survey. 2024.

[16] Afzal S, Khan H A, Khan I U, et al. A Comprehensive Survey on Affective Computing; Challenges, Trends, Applications, and Future Directions. IEEE Access, 2024, 12.

[17] Chen X, Ibrahim Z. A Comprehensive Study of Emotional Responses in AI-Enhanced Interactive Installation Art. Sustainability, 2023, 15(22).