Length-constrained Neural Machine Translation using Length Prediction and Perturbation into Length-aware Positional Encoding

• Oka Yui

  Nara Institute of Science and Technology Currently with NTT Communication Science Laboratories

• Sudoh Katsuhito

  Nara Institute of Science and Technology

• Nakamura Satoshi

  Nara Institute of Science and Technology

Abstract

Neural machine translation often suffers from an under-translation problem owing to its limited modeling of the output sequence lengths. In this study, we propose a novel approach to training a Transformer model using length constraints based on length-aware positional encoding (PE). Because length constraints with exact target sentence lengths degrade the translation performance, we add a random perturbation with a uniform distribution within a certain range to the length constraints in the PE during the training. In the inference step, we predicted the output lengths from the input sequences using a length prediction model based on a large-scale pre-trained language model. In Japanese-to-English and English-to-Japanese translation, experimental results show that the proposed perturbation injection improves the robustness of the length prediction errors, particularly within a certain range.

Journal

• Journal of Natural Language Processing

  Journal of Natural Language Processing 28 (3), 778-801, 2021

  The Association for Natural Language Processing

Keywords

• Positional Encoding
• Neural Machine Translation

Details

• CRID: 1390007912125151872
• NII Article ID: 130008088116
• DOI: 10.5715/jnlp.28.778
• ISSN: 21858314
• ISSN: 13407619
• Web Site: https://www.jstage.jst.go.jp/article/jnlp/28/3/28_778/_pdf
• Text Lang: en
• Data Source:
  • JaLC
  • Crossref
  • CiNii Articles
  • KAKEN
• Abstract License Flag: Disallowed