NOCOA+: Multimodal Computer-Based Training for Social and Communication Skills

• TANAKA Hiroki

  Graduate School of Information Science, Nara Institute of Science and Technology

• SAKTI Sakriani

  Graduate School of Information Science, Nara Institute of Science and Technology

• NEUBIG Graham

  Graduate School of Information Science, Nara Institute of Science and Technology

• TODA Tomoki

  Graduate School of Information Science, Nara Institute of Science and Technology

• NAKAMURA Satoshi

  Graduate School of Information Science, Nara Institute of Science and Technology

Abstract

Non-verbal communication incorporating visual, audio, and contextual information is important to make sense of and navigate the social world. Individuals who have trouble with social situations often have difficulty recognizing these sorts of non-verbal social signals. In this article, we propose a training tool NOCOA+ (Non-verbal COmmuniation for Autism plus) that uses utterances in visual and audio modalities in non-verbal communication training. We describe the design of NOCOA+, and further perform an experimental evaluation in which we examine its potential as a tool for computer-based training of non-verbal communication skills for people with social and communication difficulties. In a series of four experiments, we investigated 1) the effect of temporal context on the ability to recognize social signals in testing context, 2) the effect of modality of presentation of social stimulus on ability to recognize non-verbal information, 3) the correlation between autistic traits as measured by the autism spectrum quotient (AQ) and non-verbal behavior recognition skills measured by NOCOA+, 4) the effectiveness of computer-based training in improving social skills. We found that context information was helpful for recognizing non-verbal behaviors, and the effect of modality was different. The results also showed a significant relationship between the AQ communication and socialization scores and non-verbal communication skills, and that social skills were significantly improved through computer-based training.

Journal

• IEICE Transactions on Information and Systems

  IEICE Transactions on Information and Systems E98.D (8), 1536-1544, 2015

  The Institute of Electronics, Information and Communication Engineers

Keywords

• computer-based training
• multimodality
• non-verbal behaviors
• context information

Details

• CRID: 1390001204377734656
• NII Article ID: 130005090402
• DOI: 10.1587/transinf.2014edp7400
• ISSN: 17451361
• ISSN: 09168532
• Web Site: https://www.jstage.jst.go.jp/article/transinf/E98.D/8/E98.D_2014EDP7400/_pdf
• Text Lang: en
• Data Source:
  • JaLC
  • Crossref
  • CiNii Articles
• Abstract License Flag: Disallowed