Methods for Collecting Stable Data on Impression Evaluations of Japanese Learners' Writings

Abstract

This paper explores stable data collection methods for including impression evaluation data in the longitudinal corpus of Japanese learners’ writings, “W-CoLeJa”. In a pilot survey, 12 native Japanese speakers evaluated 42 writings, and their results were analyzed using correlation analysis and Generalizability Theory (G theory), yielding three findings. (1) Using a medium level of writing as a benchmark, the deviation in the evaluation scale among raters could be reduced, enabling the collection of relatively stable data. (2) Improvements to the collection method include changing the overall rating from a fine-grained 50-point scale to a coarser one, such as a 10-point scale, reducing the number of writings evaluated all at one time, and replacing technical terms such as “cohesion” in the explanations with simpler words. (3) Regarding the number of raters, the results of a Decision Study (D study) based on G Theory suggest that, even if the number of raters is reduced from the current survey, it is possible to maintain a certain level of reliability with the current 11 evaluation items.