Can commercial LLMs support regulatory checks in cell processing centers? Evaluation of SOP checking and regulatory question answering using supplied documents

抄録

Introduction

Large language models (LLMs) are increasingly used to retrieve and interpret regulatory information, but their reliability for regulatory compliance checking in cell processing centers (CPCs) remains unclear. This study evaluated commercial LLM platforms on CPC-related regulatory tasks and examined the performance of human and automated evaluators.

Methods

NotebookLM, Gemini, ChatGPT, and Claude answered 72 questions in two tasks involving a model CPC standard operating procedure and Ordinance for Enforcement of the Act on the Safety of Regenerative Medicine. The 288 responses were evaluated using four binary criteria, Answer Accuracy, Citation Fidelity, Citation Precision, and Question Fidelity, and a composite criterion, Practical Compliance (PC), which required all four criteria to be satisfied. A human evaluator, Claude, and NotebookLM independently scored the responses. Automated evaluation was also performed without a reference answer. Discrepant judgments were adjudicated against the source documents, and a workflow in which evaluator disagreement triggered source review was examined retrospectively.

Results

PC differed among the four platforms. Among the 238 responses that satisfied Answer Accuracy, 41 (17.2%) failed at least one of the other three criteria and therefore did not satisfy PC. Citation Fidelity and Citation Precision also differed in a subset of responses, indicating that correct identification of a provision and faithful representation of its content were separate requirements. The human evaluator more often missed defects and rarely over-penalized non-defective responses, whereas the automated evaluators showed both types of error. When a reference answer was provided, the automated evaluators detected more errors overall. A retrospective review strategy focused on cases where the evaluators disagreed narrowed the cases requiring source checking, but some errors were missed by all evaluators.

Conclusions

Satisfying Answer Accuracy alone did not ensure Practical Compliance, and identifying the correct regulatory provision did not ensure faithful representation of its content. Human and automated evaluators showed different patterns of error. Automated evaluation combined with targeted human source review should be evaluated prospectively as a human-in-the-loop approach for CPC practice.