-
- Zeng Zhaohao
- Waseda University
-
- Luo Cheng
- Tsinghua University
-
- Shang Lifeng
- Huawe Noah's Ark Lab
-
- Li Hang
- Toutiao AI Lab
-
- Sakai Tetsuya
- Waseda University
抄録
<p>We attempt to tackle the problem of evaluating textual, multi-round, task-oriented dialogues between the customer and the helpdesk, such as those that take the form of online chats. As an initial step towards automatic evaluation of helpdesk agent systems, we have constructed a test collection comprising 3, 700 real Customer-Helpdesk multi-round dialogues by mining Weibo, a major Chinese microblogging media. Each dialogue has been annotated with multiple subjective quality annotations and nugget annotations, where a nugget is a minimal sequence of posts by the same utterer that helps towards problem solving. In addition, 34% of the dialogues have been manually translated into English. We first propose a nugget-based dialogue quality evaluation measure called Utility for Customer and Helpdesk (UCH), where a nugget is a manually identified utterance within a dialogue that helps the Customer advance towards problem solving. In addition, we propose a simple neural network-based approach to predicting the dialogue quality scores from the entire dialogue, which we call Neural Evaluation Machine (NEM). According to our experiments with DCH-1, UCH correlates better with the appropriateness of utterances than with customer satisfaction. In contrast, as NEM leverages natural language expressions within the dialogue, it correlates relatively well with customer satisfaction.</p>
収録刊行物
-
- Journal of Information Processing
-
Journal of Information Processing 26 (0), 768-778, 2018
一般社団法人 情報処理学会
- Tweet
詳細情報 詳細情報について
-
- CRID
- 1390564238045631616
-
- NII論文ID
- 130007511649
-
- ISSN
- 18826652
-
- 本文言語コード
- en
-
- データソース種別
-
- JaLC
- Crossref
- CiNii Articles
- KAKEN
-
- 抄録ライセンスフラグ
- 使用不可