Hanlei Zhang

Rui Liu

Speech Information Processing, Natural Language Processing, and Multimodal Human-Computer Dialogue.

Mobile: +86 16647162610

About me

Rui Liu is currently a Professor in National and Local Joint Engineering Research Center of Mongolian Intelligent Information Processing, Inner Mongolia University. Rui Liu received Ph.D degree from Inner Mongolia University, China in 2020 and Bachelor degree in Taiyuan University of Technology, ShanXi, China in 2014. From 2020 to 2022, he worked as a Research Fellow at the Department of Electrical and Computer Engineering, National University of Singapore, Singapore, working with Prof. Haizhou Li. He was the recipient of the "Best Paper Award" at the 2021 International Conference on Asian Language Processing.

He has published more than 20 papers in top-tier NLP/ML/AI conferences and journals, including IEEE/ACM-TASLP, Neural Networks, ICASSP, COLING, INTERSPEECH, etc. Dr. Liu serves as the reviewer for many major referred journal and conference papers. His research interests broadly lie in audio, speech and natural language processing, which include expressive Text-to-Speech (TTS), expressive voice conversion, speech emotion recognition, prosody structure prediction, grapheme-to-phoneme conversion (G2P), syntax parsing et. al.

Education and Experience

News

  • 2023/05 Two papers about Accented Speech Synthesis and Aduio Deepfake Detection have been accepted by InterSpeech 2023.
  • 2023/05 Rui Liu will serve as a Session Chair in the ICASSP 2023.
  • 2023/04 Congratulations to Rui Liu for winning the 2022 ACM China Rising Star Award.
  • 2023/02 One paper about Multimodal Emotion Recognition has been accepted by ICASSP 2023.
  • 2023/01 One paper about Speech Enhancement is accepted for publication in IEEE/ACM-TASLP.
  • 2022/11 Two papers about Text-to-Speech have been accepted by NCMMSC 2022 .
  • 2022/09 One paper about Open-Source Mongolian Text-to-Speech Synthesis Dataset has been accepted by IALP 2022.
  • 2022/09 One paper about neural machine translation has been accepted by ICONIP 2022.
  • 2022/09 Rui Liu's application for Young Scientists Fund of the National Natural Science Foundation of China (NSFC) was approved.
  • 2022/06 Rui Liu was elected as member of Youth Working Committee of Chinese Association for Artificial Intelligence (CAAI) .
  • 2022/06 One paper about emotional voice conversion is accepted for publication in IEEE/ACM-TASLP.
  • 2022/06 One paper about emotion strength assessment has been accepted by INTERSPEECH 2022.
  • 2022/04 One journal paper about Robust TTS is accepted for publication in IEEE/ACM-TASLP.
  • 2022/02 One journal paper about emotional TTS is accepted to be published in IEEE Internet of Things Journal.
  • 2022/01 Two papers about Visual TTS and ASR have been accepted by ICASSP 2022.
  • 2022/01 Rui Liu was elected as executive member of CCF Professional Committee of Speech Dialogue and Auditory Processing .
  • 2021/12 One paper about Real-time and High-fidelity Mongolian TTS is accepted for publication in Journal of Chinese Information Processing.
  • 2021/12 Our paper about Mongolian emotional speech synthesis was awarded as " Best Paper " at IALP 2021.
  • 2021/11 One journal paper about emotional voice conversion is accepted for publication in Speech Communication.
  • 2021/10 Invited to serve as a reviewer for ICASSP 2022.
  • 2021/06 One paper about emotional TTS has been accepted by INTERSPEECH 2021.
  • 2021/04 One journal paper about expressive TTS is accepted for publication in IEEE/ACM-TASLP.
  • 2021/04 One journal paper about fast and high-quality TTS is accepted to be published in Neural Networks.
  • 2021/01 Two papers about expressive TTS and emotional voice conversion have been accepted by ICASSP 2021.
  • 2020/12 One journal paper about expressive Mongolian TTS is accepted for publication in IEEE/ACM-TASLP.

Selected Publications [Google Scholar]

(#: equal contribution *: corresponding author)

Preprints

  1. Controllable Accented Text-to-Speech Synthesis
    Rui Liu, Berrak Sisman, Guanglai Gao, Haizhou Li.
    To be submitted for possible journal publication
    [PDF] [DEMO]
  2. FCTalker: Fine and Coarse Grained Context Modeling for Expressive Conversational Speech Synthesis
    Yifan Hu, Rui Liu *, Guanglai Gao, Haizhou Li.
    [PDF] [DEMO] [CODE]

Journal papers

  1. Decoupling Speaker-Independent Emotions for Voice Conversion Via Source-Filter Networks
    Zhaojie Luo, Shoufeng Lin, Rui Liu *, Jun Baba, Yuichiro Yoshikawa, Ishiguro Hiroshi.
    IEEE/ACM Transactions on Audio, Speech, and Language Processing (IEEE/ACM-TASLP). 2022
    (Top journal, JCR Q1, IF=4.364)
    [PDF] [DEMO]
  2. Decoding Knowledge Transfer for Neural Text-to-Speech Training
    Rui Liu, Berrak Sisman, Guanglai Gao, Haizhou Li.
    IEEE/ACM Transactions on Audio, Speech, and Language Processing (IEEE/ACM-TASLP). 2022
    (Top journal, JCR Q1, IF=4.364)
    [PDF] [DEMO] [BIB]
  3. Multi-Stage Deep Transfer Learning for EmIoT-enabled Human-Computer Interaction
    Rui Liu, Qi Liu, Hongxu Zhu, Hui Cao.
    IEEE Internet of Things Journal. 2022
    (Top journal, JCR Q1, IF=10.238)
    [PDF] [DEMO]
  4. MonTTS: A Real-time and High-fidelity Mongolian TTS Model with Complete Non-autoregressive Mechanism (in Chinese)
    Rui Liu, Shyin Kang, Jingdong Li, Feilong Bao, Guanglai Gao.
    Journal of Chinese Information Processing. 2022
    (CCF T1)
    [PDF] [CODE]
  5. Emotional Voice Conversion: Theory, Databases and ESD
    Kun Zhou, Berrak Sisman, Rui Liu, Haizhou Li.
    Speech Communication. 2021
    (Top journal, CCF-B, IF=2.723)
    [PDF] [DEMO]
  6. Expressive TTS Training with Frame and Style Reconstruction Loss
    Rui Liu, Berrak Sisman, Guanglai Gao, Haizhou Li.
    IEEE/ACM Transactions on Audio, Speech, and Language Processing (IEEE/ACM-TASLP). 2021
    (Top journal, JCR Q1, IF=4.364)
    [PDF] [DEMO] [BIB]
  7. FastTalker: A Neural Text-to-Speech Architecture with Shallow and Group Autoregression
    Rui Liu, Berrak Sisman, Yixing Lin, Haizhou Li.
    Neural Networks. 2021
    (JCR Q1, IF=9.657)
    [PDF] [DEMO] [BIB]
  8. Exploiting Morphological and Phonological Features to Improve Prosodic Phrasing for Mongolian Speech Synthesis
    Rui Liu, Berrak Sisman, Feilong Bao, Jichen Yang, Guanglai Gao, Haizhou Li.
    IEEE/ACM Transactions on Audio, Speech, and Language Processing (IEEE/ACM-TASLP). 2021
    (Top journal, JCR Q1, IF=4.364)
    [PDF] [BIB]
  9. Modeling Prosodic Phrasing with Multi-Task Learning in Tacotron-based TTS
    Rui Liu, Berrak Sisman, Feilong Bao, Guanglai Gao, Haizhou Li.
    IEEE Signal Processing Letters. 2020
    (JCR Q1, IF=3.201)
    [PDF] [BIB] [DEMO]

Conference papers

  1. Exploiting Modality-Invariant Feature for Robust Multimodal Emotion Recognition with Missing Modalities
    Haolin Zuo, Rui Liu *, Jinming Zhao, Guanglai Gao, Haizhou Li.
    2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP'2023).
    (Top conference, CCF-B)
    [PDF] [CODE]
  2. MnTTS: An Open-Source Mongolian Text-to-Speech Synthesis Dataset and Accompanied Baseline
    Yifan Hu #, Pengkai Yin #, Rui Liu *, Feilong Bao and Guanglai Gao.
    2022 International Conference on Asian Language Processing (IALP'2022)
    [PDF] [CODE] [Application Entry]
  3. A Deep Investigation of RNN and Self-attention for the Cyrillic-Traditional Mongolian Bidirectional Conversion
    Muhan Na, Rui Liu *, Feilong Bao and Guanglai Gao.
    29th International Conference on Neural Information Processing (ICONIP'2022)
    (CCF-C)
    [PDF]
  4. Accurate Emotion Strength Assessment for Seen and Unseen Speech Based on Data-Driven Deep Learning
    Rui Liu, Berrak Sisman, Björn Schuller, Guanglai Gao and Haizhou Li.
    23th Annual Conference of the International Speech Communication Association (INTERSPEECH'2022)
    (Top conference, CCF-C)
    [PDF] [CODE]
  5. Alignment-Learning based single-step decoding for accurate and fast non-autoregressive speech recognition
    Yonghe Wang, Rui Liu *, Feilong Bao, Hui Zhang, Guanglai Gao.
    2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP'2022).
    (Top conference, CCF-B)
    [PDF]
  6. VisualTTS: TTS with Accurate Lip-speech Synchronization for Automatic Voice Over
    Junchen Lu, Berrak Sisman, Rui Liu, Mingyang Zhang, Haizhou Li.
    2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP'2022).
    (Top conference, CCF-B)
    [PDF] [DEMO]
  7. Mongolian emotional speech synthesis based on transfer learning and emotional embedding
    Aihong Huang, Feilong Bao, Guanglai Gao, Yu Shan, Rui Liu *
    (Best Paper Award)
    2021 International Conference on Asian Language Information Processing (IALP'2021)
    [PDF]
  8. Reinforcement Learning for Emotional Text-to-Speech Synthesis with Improved Emotion Discriminability
    Rui Liu, Berrak Sisman, Haizhou Li.
    22th Annual Conference of the International Speech Communication Association (INTERSPEECH'2021)
    (Top conference, CCF-C)
    [PDF] [DEMO] [BIB]
  9. GraphSpeech: Syntax-aware Graph Attention Network for Neural Speech Synthesis
    Rui Liu, Berrak Sisman, Haizhou Li.
    2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP'2021), Oral.
    (Top conference, CCF-B)
    [PDF] [DEMO] [BIB] [VIDEO]
  10. Seen and Unseen Emotional Style Transfer for Voice Conversion with a New Emotion Speech Dataset
    Kun Zhou, Berrak Sisman, Rui Liu, Haizhou Li.
    2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP'2021), Oral.
    (Top conference, CCF-B)
    [PDF] [BIB]
  11. Teacher-Student Training For Robust Tacotron-based TTS
    Rui Liu, Berrak Sisman, Jingdong Li, Feilong Bao, Guanglai Gao, Haizhou Li.
    2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP'2020), Oral.
    (Top conference, CCF-B) (With Travel Grant)
    [PDF] [BIB] [DEMO] [VIDEO]
  12. WaveTTS: Tacotron-based TTS with Joint Time-Frequency Domain Loss
    Rui Liu, Berrak Sisman, Feilong Bao, Guanglai Gao, Haizhou Li.
    The Speaker and Language Recognition Workshop 2020 (Odyssey'2020).
    [PDF] [BIB] [DEMO] [VIDEO]
  13. NUS-HLT System for Blizzard Challenge 2020
    Yi Zhou, Xiaohai Tian, Xuehao Zhou, Mingyang Zhang, Grandee Lee, Rui Liu, Berrak Sisman, and Haizhou Li
    Joint Workshop for the Blizzard Challenge and Voice Conversion Challenge 2020 (BC'2020).
    [PDF] [BIB]
  14. The IMU Speech Synthesis Entry for Blizzard Challenge 2019
    Rui Liu, Jingdong Li, Feilong Bao and Guanglai Gao.
    Blizzard Challenge Workshop 2019 (BC'2019).
    [PDF] [BIB]
  15. Improving Mongolian Phrase Break Prediction by Using Syllable and Morphological Embeddings with BiLSTM Model
    Rui Liu, Feilong Bao, Guanglai Gao, Hui Zhang and Yonghe Wang.
    19th Annual Conference of the International Speech Communication Association (INTERSPEECH'2018), Oral
    (Top conference, CCF-C)
    [PDF] [BIB]
  16. A LSTM Approach with Sub-word Embeddings for Mongolian Phrase Break Prediction
    Rui Liu, Feilong Bao, Guanglai Gao, Hui Zhang and Yonghe Wang.
    27th International Conference on Computational Linguistics (COLING'2018).
    (Top conference, CCF-B)
    [PDF] [BIB]
  17. End-to-End Mongolian Text-to-Speech System
    Jingdong Li, Hui Zhang, Rui Liu, Xueliang Zhang and Feilong Bao.
    11th International Symposium on Chinese Spoken Language Processing (ISCSLP'2018).
    [PDF] [BIB]
  18. Mongolian Text-to-Speech System Based on Deep Neural Network
    Rui Liu, Feilong Bao, Guanglai Gao and Yonghe Wang.
    14th National Conference on Man-Machine Speech Communication (NCMMSC'2017), Oral.
    [PDF] [BIB]

Projects

Principal Investigator

  1. High-level Talents Introduction Project of Inner Mongolia University
    No. 10000-22311201/002
    2022/05-2025/05
  2. Young Scientists Fund of the National Natural Science Foundation of China (NSFC)
    No. 62206136
    2023/01-2025/12

Co-Principal Investigator

  1. ......

Talks

  1. Title: Mongolian Text-to-Speech Technology （蒙古语语音合成技术）.
    [Slides] [Video]
    Organizer: Chinese Association for Artificial Intelligence （CAAI）
    Date: 20 Aug 2022
  2. Title: Emotion Intensity Research of Speech Synthesis (语音合成中的情感强度建模研究).
    [Slides] [Video]
    Organizer: SpeechHome （语音之家）
    Date: 19 May 2022
  3. Title: Prosody and Emotion Modeling in End-to-End Speech Synthesis （端到端语音合成中的韵律、情感建模研究）.
    [Slides] [Video]
    Organizer: CCF Professional Committee of Speech Dialogue and Auditory Processing
    Date: 04 Dec 2021

Activities

  1. Conference Reviewer:
    - INTERSPEECH 2021/2022
    - ICASSP 2021/2022
    - Joint Workshop for the Blizzard Challenge and Voice Conversion Challenge 2020
    - SLT 2022
    - O-COCOSDA 2022
  2. Journal Reviewer:
    - IEEE/ACM Transactions on Audio, Speech, and Language Processing (IEEE/ACM-TASLP)
    - IEEE Signal Processing Letters
    - IEEE Internet of Things Journal (IEEE-IoTJ)
  3. Professional Service:
    - Local Arrangement Co-chair, O-COCOSDA 2021, Singapore.
    - Local Arrangement Co-chair, IWSDS 2021, Singapore
    - Local Arrangement Co-chair, SIGDIAL 2021, Singapore
    - Student Volunteer, ASRU 2019 (IEEE Automatic Speech Recognition and Understanding Workshop), Singapore
    - Student Volunteer, NLPCC 2018 (7th CCF International Conference on Natural Language Processing and Chinese Computing), Hohhot, China.
    - Program Committee Member, O-COCOSDA 2022.

Awards

  1. Dec 2021, Excellent Doctoral dissertation of Inner Mongolia Autonomous Region
  2. Dec 2021, IALP-2021 Best Paper
  3. July 2021, 2020 ACM China Doctoral Dissertation Award (Hohhot Chapter), ACM (Association for Computing Machinery) China Council
  4. Sep 2020, Excellent Doctoral dissertation of Inner Mongolia University
  5. Feb 2020, ICASSP IEEE SPS Travel Grant
  6. Aug 2019, Research Scholarship of China Scholarship Council (CSC)
  7. Oct 2018, National scholarship for Doctoral students (top 2% students), Ministry of Education of P.R.China
  8. Oct 2018, Academic scholarship of Inner Mongolia autonomous region
  9. Oct 2017, National scholarship for Doctoral students (top 2% students), Ministry of Education of P.R.China
  10. Oct 2017, Academic scholarship of Inner Mongolia autonomous region
  11. Oct 2016, Academic scholarship of Inner Mongolia autonomous region
  12. Oct 2011, National Encouragement scholarship

Resource

More about Me