Optimization of Reference-less Evaluation Metric of Grammatical Error Correction for Manual Evaluations

DOI Web Site 28 References Open Access

Bibliographic Information

Other Title
  • 文法誤り訂正の参照文を用いない自動評価の人手評価への最適化

Abstract

<p> The development of a reliable automatic evaluation metric of grammatical error correction (GEC) is useful for the research and development of GEC. Since it is difficult to cover all possible reference sentences, previous studies have proposed reference-less metrics. One of them achieved a higher correlation with manual evaluation than reference-based metrics by integrating metrics from the three perspectives of grammaticality, fluency, and meaning preservation. However, the correlation with the manual evaluation can be further improved because they are not considered for optimizing each metric for each manual evaluation. Therefore, in this study, we propose a method of optimizing each metric. Furthermore, we create a dataset with manual evaluation of system output that is ideal for optimization. Experimental results show that the proposed method improves correlation with the manual evaluation in both the metric of each perspective and combining the metrics. We also demonstrate that both using pre-trained language models for optimization and optimizing to manual evaluation on system output of GEC contribute to improvement. As a result of the analysis, it was found that our proposed metric appropriately rewarded more edits of error types than the conventional methods. </p>

Journal

References(28)*help

See more

Related Projects

See more

Keywords

Details 詳細情報について

Report a problem

Back to top