Qualitative Validation of Rating Scales on Oral English Proficiency

DOI Web Site Web Site

Bibliographic Information

Other Title
  • 英語口頭運用技能の熟達度に関する評価尺度の質的妥当化
  • エイゴ コウトウ ウンヨウ ギノウ ノ ジュクタツド ニ カンスル ヒョウカ シャクド ノ シツテキ ダトウカ

Search this article

Abstract

The present paper aims to validate the oral proficiency rating scales that were developed in Nekoda, Nekoda, & Miura (2007), in which three-facet data were collected. The data concerned the way in which 47 junior high school and high school teachers assessed 46 pieces of video-recorded performance (10-minute interviews) with reference to 52 descriptors (short descriptions of performance characteristics). The dataset was analyzed by a Many-Facet Rasch Model (FACETS), and four analytic rating scales ('vocabulary range', 'grammaticality', 'fluency', 'pronunciation') were developed by means of a quantitative method. This study examines these results and attempts to verify the rating scales by means of a qualitative method. More concretely, three high school teachers were asked to describe several pieces of video-recorded performance in their own words. The performance videos in this process were selected (from those used in the previous study) on the basis of performance-quality 'logit values' gained from the FACETS analysis. A Many-Facet Rasch Model is based on probability theory and thus a closer look at the results reveals how likely each item of performance will be assessed by teachers in general at which score (on a four-point scale from 0 points to 3 points). On the basis of this information, this study examines (1) performance which is highly likely to be assessed as 'achieved' to a proficiency level described by a certain descriptor (a full mark = 3 points) and (2) performance which is highly likely to be assessed as 'not yet achieved' to the same level (a slightly lower mark = 2 points). The three teachers watched these performances and described favorable characteristics of (1) and unfavorable characteristics of (2). This study checks whether the descriptive terms that the teachers produced correspond to what is meant by the matching descriptors in the rating scales. As a result, it is clarified that the descriptions produced by these teachers cover various characteristics mentioned in the descriptors in the rating scales, but also that some of the descriptors need to be revised further.

Journal

  • JLTA Journal Kiyo

    JLTA Journal Kiyo 13 (0), 55-70, 2010

    Japan Language Testing Association

Related Projects

See more

Details 詳細情報について

Report a problem

Back to top