Hanlei Zhang

Rui Liu

Speech Information Processing, Natural Language Processing, and Multimodal Human-Computer Dialogue.

Mobile: +86 16647162610

About me

Rui Liu is currently a Professor in National and Local Joint Engineering Research Center of Mongolian Intelligent Information Processing, Inner Mongolia University. Rui Liu received Ph.D degree from Inner Mongolia University, China in 2020 and Bachelor degree in Taiyuan University of Technology, ShanXi, China in 2014. From 2020 to 2022, he worked as a Research Fellow at the Department of Electrical and Computer Engineering, National University of Singapore, Singapore, working with Prof. Haizhou Li. He was the recipient of the "Best Paper Award" at the 2021 International Conference on Asian Language Processing.

His research papers are mainly published in top-tier NLP/ML/AI journals and conferences, including IEEE/ACM-TASLP, IEEE-TAC, Neural Networks, AAAI, ICASSP, COLING, INTERSPEECH, etc.

Chinese Version Personal Official Website

Education and Experience

Selected Publications [Google Scholar]

(#: equal contribution *: corresponding author)

Journal papers

  1. Contrastive Learning based Modality-Invariant Feature Acquisition for Robust Multimodal Emotion Recognition with Missing Modalities
    Rui Liu, Haolin Zuo, Zheng Lian, Bjorn W. Schuller and Haizhou Li.
    IEEE Transactions on Affective Computing (IEEE-TAC). 2024
    (Top journal, CAAI-A, IF=11.2)
    [PDF] [CODE]
  2. Controllable Accented Text-to-Speech Synthesis with Fine and Coarse-Grained Intensity Rendering
    Rui Liu, Berrak Sisman, Guanglai Gao and Haizhou Li.
    IEEE/ACM Transactions on Audio, Speech, and Language Processing (IEEE/ACM-TASLP). 2024
    (Top journal, TH-CPL-A, IF=4.364)
    [PDF] [DEMO]
  3. Text-to-Speech for Low-Resource Agglutinative Language with Morphology-Aware Language Model Pre-training
    Rui Liu, Yifan Hu, Haolin Zuo, Zhaojie Luo, Longbiao Wang and Guanglai Gao.
    IEEE/ACM Transactions on Audio, Speech, and Language Processing (IEEE/ACM-TASLP). 2024
    (Top journal, TH-CPL-A, IF=4.364)
    [PDF] [DEMO]
  4. Multi-Space Channel Representation Learning for Mono-to-Binaural Conversion based Audio Deepfake Detection
    Rui Liu, Jinhua Zhang and Guanglai Gao.
    Information Fusion. 2024
    (Top journal, JCR Q1, IF=18.6)
    [PDF]
  5. Decoupling Speaker-Independent Emotions for Voice Conversion Via Source-Filter Networks
    Zhaojie Luo, Shoufeng Lin, Rui Liu *, Jun Baba, Yuichiro Yoshikawa, Ishiguro Hiroshi.
    IEEE/ACM Transactions on Audio, Speech, and Language Processing (IEEE/ACM-TASLP). 2022
    (Top journal, JCR Q1, IF=4.364)
    [PDF] [DEMO]
  6. Decoding Knowledge Transfer for Neural Text-to-Speech Training
    Rui Liu, Berrak Sisman, Guanglai Gao, Haizhou Li.
    IEEE/ACM Transactions on Audio, Speech, and Language Processing (IEEE/ACM-TASLP). 2022
    (Top journal, JCR Q1, IF=4.364)
    [PDF] [DEMO] [BIB]
  7. Multi-Stage Deep Transfer Learning for EmIoT-enabled Human-Computer Interaction
    Rui Liu, Qi Liu, Hongxu Zhu, Hui Cao.
    IEEE Internet of Things Journal. 2022
    (Top journal, JCR Q1, IF=10.238)
    [PDF] [DEMO]
  8. MonTTS: A Real-time and High-fidelity Mongolian TTS Model with Complete Non-autoregressive Mechanism (in Chinese)
    Rui Liu, Shyin Kang, Jingdong Li, Feilong Bao, Guanglai Gao.
    Journal of Chinese Information Processing. 2022
    (CCF T1)
    [PDF] [CODE]
  9. Emotional Voice Conversion: Theory, Databases and ESD
    Kun Zhou, Berrak Sisman, Rui Liu, Haizhou Li.
    Speech Communication. 2021
    (Top journal, CCF-B, IF=2.723)
    [PDF] [DEMO]
  10. Expressive TTS Training with Frame and Style Reconstruction Loss
    Rui Liu, Berrak Sisman, Guanglai Gao, Haizhou Li.
    IEEE/ACM Transactions on Audio, Speech, and Language Processing (IEEE/ACM-TASLP). 2021
    (Top journal, JCR Q1, IF=4.364)
    [PDF] [DEMO] [BIB]
  11. FastTalker: A Neural Text-to-Speech Architecture with Shallow and Group Autoregression
    Rui Liu, Berrak Sisman, Yixing Lin, Haizhou Li.
    Neural Networks. 2021
    (JCR Q1, IF=9.657)
    [PDF] [DEMO] [BIB]
  12. Exploiting Morphological and Phonological Features to Improve Prosodic Phrasing for Mongolian Speech Synthesis
    Rui Liu, Berrak Sisman, Feilong Bao, Jichen Yang, Guanglai Gao, Haizhou Li.
    IEEE/ACM Transactions on Audio, Speech, and Language Processing (IEEE/ACM-TASLP). 2021
    (Top journal, JCR Q1, IF=4.364)
    [PDF] [BIB]
  13. Modeling Prosodic Phrasing with Multi-Task Learning in Tacotron-based TTS
    Rui Liu, Berrak Sisman, Feilong Bao, Guanglai Gao, Haizhou Li.
    IEEE Signal Processing Letters. 2020
    (JCR Q1, IF=3.201)
    [PDF] [BIB] [DEMO]

Conference papers

  1. Emotion Rendering for Conversational Speech Synthesis with Heterogeneous Graph-Based Context Modeling
    Rui Liu, Yifan Hu, Yi Ren, Xiang Yin, Haizhou Li.
    38th AAAI Conference on Artificial Intelligence (AAAI'2024).
    (Top conference, CCF-A)
    [PDF] [CODE & DEMO]
  2. Exploiting Modality-Invariant Feature for Robust Multimodal Emotion Recognition with Missing Modalities
    Haolin Zuo, Rui Liu *, Jinming Zhao, Guanglai Gao, Haizhou Li.
    2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP'2023).
    (Top conference, CCF-B)
    [PDF] [CODE]
  3. MnTTS: An Open-Source Mongolian Text-to-Speech Synthesis Dataset and Accompanied Baseline
    Yifan Hu #, Pengkai Yin #, Rui Liu *, Feilong Bao and Guanglai Gao.
    2022 International Conference on Asian Language Processing (IALP'2022)
    [PDF] [CODE] [Application Entry]
  4. A Deep Investigation of RNN and Self-attention for the Cyrillic-Traditional Mongolian Bidirectional Conversion
    Muhan Na, Rui Liu *, Feilong Bao and Guanglai Gao.
    29th International Conference on Neural Information Processing (ICONIP'2022)
    (CCF-C)
    [PDF]
  5. Accurate Emotion Strength Assessment for Seen and Unseen Speech Based on Data-Driven Deep Learning
    Rui Liu, Berrak Sisman, Björn Schuller, Guanglai Gao and Haizhou Li.
    23th Annual Conference of the International Speech Communication Association (INTERSPEECH'2022)
    (Top conference, CCF-C)
    [PDF] [CODE]
  6. Alignment-Learning based single-step decoding for accurate and fast non-autoregressive speech recognition
    Yonghe Wang, Rui Liu *, Feilong Bao, Hui Zhang, Guanglai Gao.
    2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP'2022).
    (Top conference, CCF-B)
    [PDF]
  7. VisualTTS: TTS with Accurate Lip-speech Synchronization for Automatic Voice Over
    Junchen Lu, Berrak Sisman, Rui Liu, Mingyang Zhang, Haizhou Li.
    2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP'2022).
    (Top conference, CCF-B)
    [PDF] [DEMO]
  8. Mongolian emotional speech synthesis based on transfer learning and emotional embedding
    Aihong Huang, Feilong Bao, Guanglai Gao, Yu Shan, Rui Liu *
    (Best Paper Award)
    2021 International Conference on Asian Language Information Processing (IALP'2021)
    [PDF]
  9. Reinforcement Learning for Emotional Text-to-Speech Synthesis with Improved Emotion Discriminability
    Rui Liu, Berrak Sisman, Haizhou Li.
    22th Annual Conference of the International Speech Communication Association (INTERSPEECH'2021)
    (Top conference, CCF-C)
    [PDF] [DEMO] [BIB]
  10. GraphSpeech: Syntax-aware Graph Attention Network for Neural Speech Synthesis
    Rui Liu, Berrak Sisman, Haizhou Li.
    2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP'2021), Oral.
    (Top conference, CCF-B)
    [PDF] [DEMO] [BIB] [VIDEO]
  11. Seen and Unseen Emotional Style Transfer for Voice Conversion with a New Emotion Speech Dataset
    Kun Zhou, Berrak Sisman, Rui Liu, Haizhou Li.
    2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP'2021), Oral.
    (Top conference, CCF-B)
    [PDF] [BIB]
  12. Teacher-Student Training For Robust Tacotron-based TTS
    Rui Liu, Berrak Sisman, Jingdong Li, Feilong Bao, Guanglai Gao, Haizhou Li.
    2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP'2020), Oral.
    (Top conference, CCF-B) (With Travel Grant)
    [PDF] [BIB] [DEMO] [VIDEO]
  13. WaveTTS: Tacotron-based TTS with Joint Time-Frequency Domain Loss
    Rui Liu, Berrak Sisman, Feilong Bao, Guanglai Gao, Haizhou Li.
    The Speaker and Language Recognition Workshop 2020 (Odyssey'2020).
    [PDF] [BIB] [DEMO] [VIDEO]
  14. NUS-HLT System for Blizzard Challenge 2020
    Yi Zhou, Xiaohai Tian, Xuehao Zhou, Mingyang Zhang, Grandee Lee, Rui Liu, Berrak Sisman, and Haizhou Li
    Joint Workshop for the Blizzard Challenge and Voice Conversion Challenge 2020 (BC'2020).
    [PDF] [BIB]
  15. The IMU Speech Synthesis Entry for Blizzard Challenge 2019
    Rui Liu, Jingdong Li, Feilong Bao and Guanglai Gao.
    Blizzard Challenge Workshop 2019 (BC'2019).
    [PDF] [BIB]
  16. Improving Mongolian Phrase Break Prediction by Using Syllable and Morphological Embeddings with BiLSTM Model
    Rui Liu, Feilong Bao, Guanglai Gao, Hui Zhang and Yonghe Wang.
    19th Annual Conference of the International Speech Communication Association (INTERSPEECH'2018), Oral
    (Top conference, CCF-C)
    [PDF] [BIB]
  17. A LSTM Approach with Sub-word Embeddings for Mongolian Phrase Break Prediction
    Rui Liu, Feilong Bao, Guanglai Gao, Hui Zhang and Yonghe Wang.
    27th International Conference on Computational Linguistics (COLING'2018).
    (Top conference, CCF-B)
    [PDF] [BIB]
  18. End-to-End Mongolian Text-to-Speech System
    Jingdong Li, Hui Zhang, Rui Liu, Xueliang Zhang and Feilong Bao.
    11th International Symposium on Chinese Spoken Language Processing (ISCSLP'2018).
    [PDF] [BIB]
  19. Mongolian Text-to-Speech System Based on Deep Neural Network
    Rui Liu, Feilong Bao, Guanglai Gao and Yonghe Wang.
    14th National Conference on Man-Machine Speech Communication (NCMMSC'2017), Oral.
    [PDF] [BIB]

Projects

Principal Investigator

  1. High-level Talents Introduction Project of Inner Mongolia University
    No. 10000-22311201/002
    2022/05-2025/05
  2. Young Scientists Fund of the National Natural Science Foundation of China (NSFC) （国自然青年）
    No. 62206136
    2023/01-2025/12
  3. General Program of National Natural Science Foundation of China (NSFC)（国自然面上）
    No. 62476146
    2025/01-2028/12

Co-Principal Investigator

  1. Coming soon...

Talks

  1. Title: Mongolian Text-to-Speech Technology （蒙古语语音合成技术）.
    [Slides] [Video]
    Organizer: Chinese Association for Artificial Intelligence （CAAI）
    Date: 20 Aug 2022
  2. Title: Emotion Intensity Research of Speech Synthesis (语音合成中的情感强度建模研究).
    [Slides] [Video]
    Organizer: SpeechHome （语音之家）
    Date: 19 May 2022
  3. Title: Prosody and Emotion Modeling in End-to-End Speech Synthesis （端到端语音合成中的韵律、情感建模研究）.
    [Slides] [Video]
    Organizer: CCF Professional Committee of Speech Dialogue and Auditory Processing
    Date: 04 Dec 2021

Activities

  1. Conference Reviewer:
    - ACL 2024
    - NAACL 2024
    - INTERSPEECH 2021/2022/2023/2024
    - ICASSP 2021/2022/2023/2024
    - Joint Workshop for the Blizzard Challenge and Voice Conversion Challenge 2020
    - SLT 2022
    - O-COCOSDA 2022
  2. Journal Reviewer:
    - IEEE/ACM Transactions on Audio, Speech, and Language Processing (IEEE/ACM-TASLP)
    - IEEE Transactions on Neural Networks and Learning Systems (IEEE-TNNLS)
    - IEEE Signal Processing Letters
    - IEEE Internet of Things Journal (IEEE-IoTJ)
    - Neural Nertworks
    - NeuroComputing
    - Speech Communication
  3. Professional Service:
    - Program Chair, IALP 2024, Hohhot, China
    - Local Arrangement Co-chair, O-COCOSDA 2021, Singapore.
    - Local Arrangement Co-chair, IWSDS 2021, Singapore
    - Local Arrangement Co-chair, SIGDIAL 2021, Singapore
    - Student Volunteer, ASRU 2019 (IEEE Automatic Speech Recognition and Understanding Workshop), Singapore
    - Student Volunteer, NLPCC 2018 (7th CCF International Conference on Natural Language Processing and Chinese Computing), Hohhot, China.
    - Program Committee Member, O-COCOSDA 2022.

Awards

  1. Apr 2023, 2022 ACM China Rising Star (Hohhot Chapter), ACM (Association for Computing Machinery) China Council
  2. Dec 2021, Excellent Doctoral dissertation of Inner Mongolia Autonomous Region
  3. Dec 2021, IALP-2021 Best Paper
  4. July 2021, 2020 ACM China Doctoral Dissertation Award (Hohhot Chapter), ACM (Association for Computing Machinery) China Council
  5. Sep 2020, Excellent Doctoral dissertation of Inner Mongolia University
  6. Feb 2020, ICASSP IEEE SPS Travel Grant
  7. Aug 2019, Research Scholarship of China Scholarship Council (CSC)
  8. Oct 2018, National scholarship for Doctoral students (top 2% students), Ministry of Education of P.R.China
  9. Oct 2018, Academic scholarship of Inner Mongolia autonomous region
  10. Oct 2017, National scholarship for Doctoral students (top 2% students), Ministry of Education of P.R.China
  11. Oct 2017, Academic scholarship of Inner Mongolia autonomous region
  12. Oct 2016, Academic scholarship of Inner Mongolia autonomous region

Resource