この論文は以下の「翻訳版」論文です。
書誌情報 : NINJAL Research Papers, 2025, 28, 151–166.
DOI: doi:10.15084/0002000458
プレプリント / バージョン1

Methods for Collecting Stable Data on Impression Evaluations of Japanese Learners’ Writings

##article.authors##

DOI:

https://doi.org/10.51094/jxiv.4726

キーワード:

W-CoLeJa、 writing、 Japanese native speakers、 crowdsourcing、 generalizability theory

抄録

This study explores stable data collection methods for including impression evaluation data in the longitudinal corpus of Japanese learners’ writings, “W-CoLeJa”. In the pilot survey, 12 native Japanese speakers evaluated 42 writings, and the results were analyzed using correlation analysis and generalizability theory (G theory), yielding three findings. First, by using a medium level of writing as a benchmark, the deviation in the evaluation scale among raters could be reduced, enabling the collection of relatively stable data. Second, improvements to the collection method include changing the overall rating from a fine-grained 50-point scale to a coarser one, such as a 10-point scale, reducing the number of writings evaluated one time, and replacing technical terms such as “cohesion” in the explanations with simpler words. Third, regarding the number of raters, the results of a decision study (D study) based on G theory suggest that even if the number of raters is reduced in future surveys, maintaining a certain level of reliability with the current 11 evaluation items is possible.

利益相反に関する開示

There is no conflict of interest to disclose in this paper.

ダウンロード *前日までの集計結果を表示します

ダウンロード実績データは、公開の翌日以降に作成されます。

引用文献

Fukuta, J., & Ishii, Y. (2016). Chugokugo o dai-ichi gengo to suru Nihongo gakushusha no sakubun ni taisuru Nihongo kyoshi no hyoka: Ippanka kanosei riron o mochiita kento [Reliability of Japanese Teachers’ Evaluation of Essays Written by Chinese Learners of Japanese―A Generalizability Theory Study]. Nihon Kyoka Kyoiku Gakkaishi [The Bulletin of Japanese Curriculum Research and Development], 39(2), 81–89. (in Japanese)

Hirai, A. (2018). Kyoiku, shinri, gengo-kei kenkyu no tameno deta bunseki [Data analysis for research in education, psychology, and linguistics]. Tokyo Tosho. (in Japanese)

Ijuin, I. (2017). Sakubun to hyoka: Nihongo kyoiku-teki kanten kara mita yoi bunsho [Composition and evaluation: Good writing from the perspective of Japanese language education]. In J. Lee (Ed.), Bunsho o kagaku suru [The Science of Measurement and Evaluation of Text] (pp. 38–57). Hituzi Syobo Publishing. (in Japanese)

Ijuin, I., Lee, J., Komori, K., & Noguchi, H. (2020). Hyoka komento ni mirareru ikenbun hyoka no yoso: Kyoki nettowaku oyobi koresupondensu bunseki ni motozuku kosatsu [How University Faculty Assess Opinion Essays : Exploring Assessment Comments through Co-occurrence Network and Correspondence Analyses]. Dai-ni gengo toshite no Nihongo no Shutoku Kenkyu [Acquisition of Japanese as a Second Language], 23, 26–43. (in Japanese)

Ikeda, H. (1994). Gendai tesuto riron [Modern test theory]. Asakura Publishing Co., Ltd. (in Japanese)

Ishiguro, K. (2017). Bunsho to wa nani ka: Nihongo no hyogenmen kara mita yoi bunsho [What is Bunsho (Text)?: Good writing from the perspective of Japanese expression]. In J. Lee (Ed.), Bunsho o kagaku suru [The science of measurement and evaluation of text] (pp. 14–37). Hituzi Syobo Publishing. (in Japanese)

Kondo, A., & Komori, K. (Eds.). (2012). Kenkyusha Nihongo kyoiku jiten [Kenkyusha's dictionary of Japanese language education]. Kenkyusha Co., Ltd. (Entry on "Ippanka kanosei riron" [Generalizability theory] by Noguchi, H., & Osumi, A.). (in Japanese)

Lee, J. (Ed.). (2017). Bunsho o kagaku suru [The Science of Measurement and Evaluation of Text]. Hituzi Syobo Publishing. (in Japanese)

Mizumoto, A. (2008). Jiyu eisakubun ni okeru hyoteisha hyoka no shurui to shinraisei [Types of Evaluation by Raters and Reliability in an English Essay]. Gakushusha copasu no kaiseki ni motozuku kyakkanteki sakubun hyoka shihyo no kento [The Institute of Statistical Mathematics cooperative research report] (Joint Research Report No. 215, pp. 43–49). The Institute of Statistical Mathematics. (in Japanese)

Shavelson, R. J., & Webb, N. M. (1991). Generalizability theory: A primer. Sage Publications.

Tanaka, M. (2016). Pafomansu hyoka wa naze baratsuku no ka?: Akademikku raitingu no hyoka ni okeru hyokasha no "kata" [Why do performance evaluations vary?: Rater “types” in the assessment of academic writing]. In Y. Usami (Ed.), "Hyoka" o motte machi ni deyo [Let's go out into the town with "Evaluation"] (pp. 34–53). Kurosio Publishers. (in Japanese)

Tanaka, M. (2022). Raitingu hyoka no genkai to iitokodori [Limits and advantages of writing evaluation]. In O. Kamada, K. Yui, & R. Ikeda (Eds.), Nihongo purofishienshi kenkyu no hirogari [Explorations of Japanese Proficiency Studies] (pp. 225–237). Hituzi Syobo Publishing. (in Japanese)

Tanaka, M., & Nagasaka, A. (2006). Dai-ni gengo toshite no Nihongo raitingu hyoka kijun to sono sakusei katei [Evaluation criteria for Japanese as a second language writing and the process of their development]. In National Institute for Japanese Language and Linguistics (Ed.), Sekai no gengo tesuto [Language Tests in the World] (pp. 253–276). Kurosio Publishers. (in Japanese)

Tanaka, M., & Nagasaka, A. (2009). Raitingu hyoka no itchi wa naze muzukashii ka: Ningen no kaizai suru asesumento [Why ls lt Difficult to Agree on Writing Assessment?: Rater−mediated Assessment]. Shakai Gengo Kagaku [The Japanese Journal of Language in Society], 12(1), 108–121. (in Japanese)

Tanaka, M., Tsubone, Y., & Hatsukano, A. (1998). Dai-ni gengo toshite no Nihongo ni okeru sakubun hyoka kijun: Nihongo kyoshi to ippan Nihonjin no hikaku [Evaluation Criteria for Writing by Non-native Speakers: A Comparison of Survey Results for Japanese Teachers and Non-Teachers]. Nihongo Kyoiku [Journal of Japanese Language Education], 96, 1–12. (in Japanese)

Usami, Y. (2014). "Hi-bogo washa no Nihongo" wa, dono yoni hyoka sarete iru ka [Evaluation of the Japanese language performance of non-native speakers : The significance of grasping the diversity of evaluation processes]. Coco Publishing. (in Japanese)

Yamada, T., & Murai, J. (2004). Yoku wakaru shinri tokei [Easy-to-understand psychological statistics]. Minerva Shobo. (in Japanese)

Yamanishi, H. (2005). Ippanka kanosei riron o mochiita kokosei no jiyu eisakubun hyoka no kento [Using Generalizability Theory in the Evaluation of L2 Writing]. JALT Journal, 27(2), 169–185. (in Japanese)

Yamamori, K. (2002). Ippanka kanosei riron o mochiita kantenbetsu hyoka no hohoron no kento [A study on the methodology of criterion-referenced evaluation using generalizability theory]. STEP Bulletin, 14, 62–70. (in Japanese)

Yamamori, K. (2004). Eikaiwa tesuto no shinraisei no kento: Ippanka kanosei riron [Examination of the reliability of English conversation tests: Generalizability theory]. In H. Maeda & K. Yamamori (Eds.), Eigo kyoshi no tameno kyoiku deta bunseki nyumon: Jugyo ga kawaru tesuto, hyoka, kenkyu [Introduction to educational data analysis for English teachers: Tests, evaluation, and research that change classes] (pp. 82–89). TAISHUKAN Publishing Co., Ltd. (in Japanese)

ダウンロード

公開済


投稿日時: 2026-05-27 07:39:45 UTC

公開日時: 2026-07-13 09:01:10 UTC
研究分野
心理学・教育学

ライセンス

Copyright（c）2026 Yumiko HONDA
Nahoko II

Creative Commons License

この作品は、Creative Commons Attribution 4.0 International Licenseの下でライセンスされています。