対話破綻検出チャレンジ3における対話破綻検出の評価尺度の選定

• 角森 唯子

  株式会社NTTドコモ

• 東中 竜一郎

  NTT メディアインテリジェンス研究所

• 高橋 哲朗

  株式会社富士通研究所

• 稲葉 通将

  電気通信大学

書誌事項

タイトル別名

• Selection of Evaluation Metrics for Dialogue Breakdown Detection in Dialogue Breakdown Detection Challenge 3

抄録

The task of detecting dialogue breakdown, the aim of which is to detect whether a system utterance causes dialogue breakdown in a given dialogue context, has been actively researched in recent years. However, currently, it is not clear which evaluation metrics should be used to evaluate dialogue breakdown detectors, hindering progress in dialogue breakdown detection. In this paper, we propose finding appropriate metrics for evaluating the detectors in dialogue breakdown detection challenge 3. In our approach, we first enumerate possible evaluation metrics and then rank them on the basis of system ranking stability and discriminative power. By using the submitted runs (results of dialogue breakdown detection of participants) of dialogue breakdown detection challenge 3, we experimentally found that RSNOD(NB,PB,B) is an appropriate metric for dialogue breakdown detection in dialogue breakdown detection challenge 3 for English and Japanese, although NMD(NB,PB,B) and MSE(NB,PB,B) were found appropriate specifically for English and Japanese, respectively.

収録刊行物

• 人工知能学会論文誌

  人工知能学会論文誌 35 (1), DSI-G_1-10, 2020-01-01

  一般社団法人 人工知能学会

キーワード

• Chat-oriented Dialogue System
• Dialogue Breakdown Detection
• Evaluation Metrics

詳細情報

• CRID: 1390283659837201920
• NII論文ID: 130007779296
• DOI: 10.1527/tjsai.dsi-g
• ISSN: 13468030
• ISSN: 13460714
• Web Site: https://www.jstage.jst.go.jp/article/tjsai/35/1/35_DSI-G/_pdf
• 本文言語コード: ja
• データソース種別:
  • JaLC
  • Crossref
  • CiNii Articles
  • KAKEN
• 抄録ライセンスフラグ: 使用不可