Utilizing Large Language Models in Research Practice: Insights from Southeast Asia

Abstract

This paper examines the practical use of large language models (LLMs) to structure unstructured text in Southeast Asian political research. Prior studies have relied on manual labeling, which creates scalability and consistency challenges for large datasets. This paper presents and compares two approaches on a common task, classifying Malaysian political news and comments into 12 topics with four polarity labels. The first approach employs general-purpose models via the OpenAI API. Through prompt engineering, setting the temperature to 0, and enforcing Structured Outputs in JSON to control diversity and output format, this approach achieves high performance with few in-context examples and delivers a degree of reproducibility, although perfect reproducibility remains elusive. On a test set of 50 items sampled from 43,546 Facebook and Reddit posts, GPT -4o-mini attained a weighted F1 score of 0.8752. The second approach fine-tunes BERT within a Human-in-the-Loop framework. This paper applies diversity sampling using k-means and uncertainty sampling iteratively based on predicted class probabilities to prioritize annotations near the decision boundary, expanding the labeled dataset from 405 to 605, 1,205 and then to 1,705 items. To mitigate class imbalance, particularly the scarcity of positive and neutral instances, this paper augments the data with 200 synthetic examples generated via the OpenAI API. Weighted F1 scores improved from 0.6898 to 0.7134, 0.8212, and 0.8476 across iterations; with hyperparameter tuning, the model reached 0.8606, approaching the performance of GPT -4o-mini. Topic-level analyses indicate persistent difficulty in domains such as political leadership and administrative performance, where single sentences often contain mixed polarity; this paper therefore suggests reconsidering label design (e.g., accommodating mixed polarity) for applied settings. In contexts where some variability in reproducibility is acceptable and rapid, high performance is required, the API-based approach is advantageous. For long-term stable operation, version pinning, and accuracy gains through targeted data growth, fine-tuned BERT is preferable. This paper further argues that human monitoring of outputs, statistical quality control of data, and collaboration between domain expertise and data science are essential and offers a practical roadmap for quantitative analysis in the social sciences using LLMs.