Neural Incremental Speech Recognition Toward Real-Time Machine Speech Translation

• NOVITASARI Sashi

  Augmented Human Communication Lab, Nara Institute of Science and Technology

• SAKTI Sakriani

  Augmented Human Communication Lab, Nara Institute of Science and Technology RIKEN, Center for Advanced Intelligence Project AIP Japan Advanced Institute of Science and Technology

• NAKAMURA Satoshi

  Augmented Human Communication Lab, Nara Institute of Science and Technology RIKEN, Center for Advanced Intelligence Project AIP

Abstract

Real-time machine speech translation systems mimic human interpreters and translate incoming speech from a source language to the target language in real-time. Such systems can be achieved by performing low-latency processing in ASR (automatic speech recognition) module before passing the output to MT (machine translation) and TTS (text-to-speech synthesis) modules. Although several studies recently proposed sequence mechanisms for neural incremental ASR (ISR), these frameworks have a more complicated training mechanism than the standard attention-based ASR because they have to decide the incremental step and learn the alignment between speech and text. In this paper, we propose attention-transfer ISR (AT-ISR) that learns the knowledge from attention-based non-incremental ASR for a low delay end-to-end speech recognition. ISR comes with a trade-off between delay and performance, so we investigate how to reduce AT-ISR delay without a significant performance drop. Our experiment shows that AT-ISR achieves a comparable performance to the non-incremental ASR when the incremental recognition begins after the speech utterance reaches 25% of the complete utterance length. Additional experiments to investigate the effect of ISR on translation tasks are also performed. The focus is to find the optimum granularity of the output unit. The results reveal that our end-to-end subword-level ISR resulted in the best translation quality with the lowest WER and the lowest uncovered-word rate.

Journal

• IEICE Transactions on Information and Systems

  IEICE Transactions on Information and Systems E104.D (12), 2195-2208, 2021-12-01

  The Institute of Electronics, Information and Communication Engineers

Keywords

• attention transfer
• incremental speech recognition
• real-time speech translation

Details

• CRID: 1390853188957480192
• NII Article ID: 130008123347
• DOI: 10.1587/transinf.2021edp7014
• ISSN: 17451361
• ISSN: 09168532
• Web Site: https://www.jstage.jst.go.jp/article/transinf/E104.D/12/E104.D_2021EDP7014/_pdf
• Text Lang: en
• Data Source:
  • JaLC
  • Crossref
  • CiNii Articles
  • KAKEN
• Abstract License Flag: Disallowed