Exploring the applicability of Large Language Models to citation context analysis

Abstract

In contrast to conventional quantitative citation analysis, a method called citation context analysis has been proposed that takes into account the contextual information of individual citations. Although citation context analysis is expected to provide complementary findings to citation analysis, it requires the creation of a large dataset through annotation work, which is costly. On the other hand, some attempts have been made to have LLM (Large Language Model), which is rapidly becoming popular these days, do the annotation work. However, most of these previous studies were conducted on general texts, and it is not necessarily clear how well they perform when applied to texts with special vocabulary and formatting, such as research papers. This study aims to explore the applicability of LLM to citation context analysis by referring to a publicly available citation context analysis dataset and a manual for the annotation work used to create it. More specifically, we will examine the following issues: 1. Whether LLM can replace humans for annotation tasks in citation context analysis? 2. How can LLM be effectively utilized in citation context analysis? The results show that LLM annotation performance is comparable to or better than human annotation in terms of consistency, but not in terms of accuracy.
However, the accuracy of LLM annotation is not as high as that of human annotation.
Therefore, it is not appropriate at this time to have LLM immediately replace human annotators in citation context analysis.
However, if it is difficult to prepare a sufficient number of human annotators, it is possible to use LLM as one of the annotators. This study provides the above basic findings that are important for the future development of citation context analysis.