Creating a Misinformation Dataset from Social Media and Building an LLM Accuracy Benchmark
##article.authors##
- Tomoka Nakazato The University of Tokyo, Graduate School of Interdisciplinary Information Studies
-
Masaki Onishi
National Institute of Advanced Industrial Science and Technology, Artificial Intelligence Research Center
https://researchmap.jp/onishi-masaki
- Hisami Suzuki National Institute of Informatics, Research and Development Center for Large Language Models
DOI:
https://doi.org/10.51094/jxiv.875Keywords:
LLM, Misinformation, Disinformation, Social Media, BenchmarkAbstract
While large-scale language models (LLMs) are developing, the problem of inaccurate information generation and dissemination by LLMs is emerging. To overcome these problems, LLM accuracy benchmarks for the Japanese language are needed, but existing benchmarks do not adequately include false or misinformation specific to Japan in social media that is actually circulating. In this paper, we propose a benchmark for LLM accuracy, JSocialFact, based on misleading information in the Japanese domain in actual social media circulating in Japan.JSocialFact is created by multiple human annotators and based on X community note data and post data to create a unique dataset that covers a wide variety of misinformation, disinformation, and malicious information.
Translated with www.DeepL.com/Translator (free version)
Conflicts of Interest Disclosure
No COI to disclose.Downloads *Displays the aggregated results up to the previous day.
References
Yifan Yao, Jinhao Duan, Kaidi Xu, Yuanfang Cai, Zhibo Sun, and Yue Zhang. A survey on large language model (LLM) security and privacy: The good, the bad, and the ugly. High-Confidence Computing, Vol. 4, No. 2, p. 100211, June 2024.
Stephanie Lin, Jacob Hilton, and Owain Evans. TruthfulQA: Measuring how models mimic human falsehoods. September 2021.
BIG bench authors. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. Transactions on Machine Learning Research, 2023.
Zhangyue Yin, Qiushi Sun, Qipeng Guo, Jiawen Wu, Xipeng Qiu, and Xuanjing Huang. Do large language models know what they don’t know? In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors, Findings of the Association for Computational Linguistics: ACL 2023, pp. 8653–8665, Toronto, Canada, July 2023. Association for Computational Linguistics.
Yuxia Wang, Haonan Li, Xudong Han, Preslav Nakov, and Timothy Baldwin. Do-not-answer: Evaluating safeguards in LLMs. In Yvette Graham and Matthew Purver, editors, Findings of the Association for Computational Linguistics: EACL 2024, pp. 896–911, St. Julian’s, Malta, March 2024. Association for Computational Linguistics.
Hitomi Yanaka, Namgi Han, Ryoma Kumon, Jie Lu, Masashi Takeshita, Ryo Sekizawa, Taisei Kato, and Hiromi Arai. Analyzing social biases in japanese large language models. arxiv:2406.02050, 2024.
友亮中村, 大輔河原. 日本語 truthfulqa の構築. 言語処理学会第 30 回年次大会 発表論文集, March 2024.
Matthew R Allen, Nimit Desai, Aiden Namazi, Eric Leas, Mark Dredze, Davey M Smith, and John W Ayers. Characteristics of X (formerly twitter) community notes addressing COVID-19 vaccine misinformation. JAMA, Vol. 331, No. 19, pp. 1670–1672, May 2024.
Moritz Pilarski, Kirill Solovev, and Nicolas Pr¨ollochs. Community notes vs. snoping: How the crowd selects fact-checking targets on social media. arXiv. org, 2023.
Jennifer Allen, Cameron Martel, and David G Rand. Birds of a feather don’t fact-check each other: Partisanship and the evaluation of news in twitter’s birdwatch crowdsourced fact-checking program. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems, No. Article 245 in CHI ’22, pp. 1–19, New York, NY, USA, April 2022. Association for Computing Machinery.
Eleni Kapantai, Androniki Christopoulou, Christos Berberidis, and Vassilios Peristeras. A systematic literature review on disinformation: Toward a unified taxonomical framework. New Media & Society, Vol. 23, No. 5, pp. 1301–1326, May 2021.
Esma A¨ımeur, Sabrine Amri, and Gilles Brassard. Fake news, disinformation and misinformation in social media: a review. Soc Netw Anal Min, Vol. 13, No. 1, p. 30, February 2023.
関根聡, 小島淳嗣, 貞光九月, 北岸郁雄. LLM の出力結果に対する人間による評価分析と gpt-4 による自動評価との比較分析. 言語処理学会第 30 回年次大会, pp. 937–942, 2024.
Downloads
Posted
Submitted: 2024-09-04 03:26:40 UTC
Published: 2024-09-05 05:34:54 UTC — Updated on 2024-09-27 02:20:26 UTC
Versions
- 2024-09-27 02:20:26 UTC (2)
- 2024-09-05 05:34:54 UTC (1)
Reason(s) for revision
To correct statistical information contained in the manuscript.License
Copyright (c) 2024
Tomoka Nakazato
Masaki Onishi
Hisami Suzuki
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
