Effectiveness of a Large Language Model-Based Feedback System for Case Report Writing in Novice Rehabilitation Staff Education: A Mixed-Methods Study

Tonouchi, Yuuto; Shunsuke  Nakai; Kayo  Nurakami; Yuki  Kataoka

doi:10.51094/jxiv.844

##article.authors##

Tonouchi, Yuuto Department of Rehabilitation, Kyoto Min-iren Asukai Hospital
Shunsuke Nakai Department of Rehabilitation, Kyoto Min-iren Asukai Hospital; Osaka Metropolitan University Graduate School of Rehabilitation Science
Kayo Nurakami Department of Rehabilitation, Kyoto Min-iren Asukai Hospital
Yuki Kataoka Department of Internal Medicine, Kyoto Min-iren Asukai Hospital; Section of Clinical Epidemiology, Department of Community Medicine, Kyoto University Graduate School of Medicine; Department of Healthcare Epidemiology, Graduate School of Medicine and Public Health, Kyoto University; Department of International and Community Oral Health, Tohoku University Graduate School of Dentistry; Scientific Research Works Peer Support Group (SRWS-PSG)

DOI:

https://doi.org/10.51094/jxiv.844

キーワード:

large language model、 education、 case report、 feed back

抄録

Objectives: To develop a large language model (LLM) based feedback system to improve the efficiency of case report writing in novice rehabilitation staff education.

Design: A sequential mixed methods study.

Methods: We conducted a preliminary survey to identify burdensome feedback tasks and developed prompts using the Claude 3 Opus. We implemented the feedback system with Google Apps Script and Slack chatbots. Effectiveness and usability were evaluated through surveys. The study included five novice rehabilitation staff who joined our hospital in April 2024.

Results: All novice staff reported that the LLM feedback was equivalent to previous human feedback and helpful for their learning. The System Usability Scale (SUS) scores showed high usability (median: 90, range: 70-95). Three instructors (60%) agreed the system saved time and reduced guidance sessions, while four (80%) felt it would alleviate their future burden. However, opinions varied regarding the feedback content's suitability and its potential to enhance novice staff learning outcomes.

Conclusion: The LLM-based feedback system for case reports showed potential to reduce instructors' burden and provided an efficient learning environment for novice rehabilitation staff. Future research should focus on system revision and further evaluation.

This study was pre-registered in the UMIN Clinical Trials Registry (UMIN-CTR) (Trial ID: UMIN000053315). https://center6.umin.ac.jp/cgi-bin/icdr/ctr_reg_list.cgi

利益相反に関する開示

The authors declared no potential conflicts of interest with respect to the research, authorship, and publication of this article

ダウンロード *前日までの集計結果を表示します

ダウンロード実績データは、公開の翌日以降に作成されます。

引用文献

Oermann MH, Garvin MF. Stresses and challenges for new graduates in hospitals. Nurse Educ Today 2002; 22: 225–230.

Parker V, Giles M, Lantry G, et al. New graduate nurses' experiences in their first year of practice. Nurse Educ Today 2014; 34: 150–156.

Karaman S. Nurses' perceptions of online continuing education. BMC Med Educ 2011; 11: 86.

Shih YS, Lee TT, Liu CY, et al. Evaluation of an online orientation program for new healthcare employees. Comput Inform Nurs 2013; 31: 343–350.

Ministry of Health, Labour and Welfare. Guidelines for Training New Nursing Staff [Revised Edition], https://www.mhlw.go.jp/file/06-Seisakujouhou-10800000-Iseikyoku/0000049466_1.pdf (2014, accessed 6 June 2024）.

Japanese Nursing Association. Learning Support Book for Nurses, https://www.nurse.or.jp/nursing/assets/learning/support-learning-guide-all.pdf （2023, accessed 6 June 2024）.

Japanese Physical Therapy Association. Guidelines for Training New Physical Therapist Staff (First Edition), https://www.japanpt.or.jp/assets/pdf/pt/lifelonglearning/introeduprogram/education_training/training_guidelines_201111.pdf （2020, accessed 6 June 2024）.

Shiota S, Goto N, Kanayama A, et al. Current Status and Challenges of Support Environments for New Graduate Occupational Therapists in Japanese Hospitals. A Mixed Method Study. Occup Ther Int 2022; 2022: 2159828.

Japanese Association of Occupational Therapists. Guidelines for Occupational Therapy Clinical Training (2018) / Handbook for Occupational Therapy Clinical Training (2022), https://www.jaot.or.jp/files/shishin2018.tebiki2022.2.pdf (2022, accessed 6 June 2024).

Suzuki Y, Horimoto Y. Survey of the Actual Conditiond of Newcomer in Medical Facilities. Rigakuryoho Kagaku 2022; 37: 375–382.

Dai W, Lin J, Jin F, et al．Can Large Language Models Provide Feedback to Students? A Case Study on ChatGPT. 2023 IEEE International Conference on Advanced Learning Technologies (ICALT), Orem, UT, USA, 10–13 July 2023, pp. 323–325.

Liang W, Zhang Y, Cao H, et al. Can large language models provide useful feedback on research papers? A large-scale empirical analysis. NEJM AI 2024; AIoa2400196.

Nikkei XTECH: AI to Review Contracts, Competition in Legal Tech Intensifies with "Seal of Approval" from the Ministry of Justice, https://xtech.nikkei.com/atcl/nxt/column/18/00001/08386/ (2023, accessed 6 June 2024)

Humza N, Khan AU, Qiu S, et al. “A Comprehensive Overview of Large Language Models.” arXiv:2307.06435v9. Epub ahead of print 9 Apr 2024. DOI: https://doi.org/10.48550/arXiv.2307.06435.

Ivankova NV, Creswell JW, Stick SL. Using Mixed-Methods Sequential Explanatory Design: From Theory to Practice. Field Methods 2006; 18: 3–20.

Zhang Y, Yuan Y, Yao AC. Meta Prompting for AI Systems. arXiv:2311.11482v5. Epub ahead of print 2 Apr 2024. DOI: https://doi.org/10.48550/arXiv.2311.11482.

von Elm E, Altman DG, Egger M, et al.; STROBE Initiative. The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE)statement: guidelines for reporting observational studies. J Clin Epidemiol 2008; 61: 344–349.

Anthropic: User Guides, https://docs.anthropic.com/en/docs/welcome (2024, accessed 6 June 2024)

Slack: Where work happens, https://slack.com/ (2017, accessed 11 June 2024)

Japanese Law Translation. Act on the Protection of Personal Information, Act on the Protection of Personal Information - Japanese/English - Japanese Law Translation (2003, accessed 8 August 2024)

Anthropic: Prompt Library Prose polisher, https://docs.anthropic.com/en/prompt-library/prose-polisher (2024, accessed 6 June 2024)

Younas A, Subramanian KP, Al-Haziazi M, et al. A Review on Implementation of Artificial Intelligence in Education. International Journal of Research and Innovation in Social Science 2023; 7: 1092–1100.

Xu X, Chen Y, Miao J. Opportunities, challenges, and future directions of large language models, including ChatGPT in medical education: a systematic scoping review. J Educ Eval Health Prof 2024; 21: 6.

Kasneci E, Sessler K, Küchemann S, et al. ChatGPT for good? On opportunities and challenges of large language models for education. Learning and Individual Differences 2023; 103: 102274.

Lewis, J. R. The System Usability Scale: Past, Present, and Future. International Journal of Human–Computer Interaction 2018; 34: 577–590.

Meyer J, Jansen T, Schiller R, et al. Using LLMs to bring evidence-based feedback into the classroom: AI-generated feedback increases secondary students’ text revision, motivation, and positive emotions. Computers and Education: Artificial Intelligence 2024; 6: 100199.

Ho WLJ, Koussayer B, Sujka J. ChatGPT: Friend or foe in medical writing? An example of how ChatGPT can be utilized in writing case reports. Surgery in Practice and Science 2023; 14: 100185.

Abd-alrazaq A, AlSaad R, Alhuwail D, et al. Large Language Models in Medical Education: Opportunities, Challenges, and Future Directions. JMIR Med Educ 2023; 9: e48291.

Lee H. The rise of ChatGPT: Exploring its potential in medical education. Anat Sci Educ 2024; 17: 926-931.