Evaluating GPT-4o-based Data Augmentation for Imbalanced Multiclass Sentiment Classification of GoPay Reviews Using IndoBERT-LoRA

Muhammad Yusran; Farit Mochamad Afendi; Anwar Fitrianto

doi:10.59395/ijadis.v7i2.1620

Authors

Muhammad Yusran Master Program in Statistics and Data Science, School of Data Science, Mathematics and Informatics, IPB University
Farit Mochamad Afendi Master Program in Statistics and Data Science, School of Data Science, Mathematics and Informatics, IPB University
Anwar Fitrianto Master Program in Statistics and Data Science, School of Data Science, Mathematics and Informatics, IPB University

DOI:

https://doi.org/10.59395/ijadis.v7i2.1620

Keywords:

GPT-4o, Data augmentation, Sentiment classification, Class imbalance, IndoBERT-LoRA

Abstract

This study evaluated GPT-4o-based data augmentation for imbalanced multiclass sentiment classification of GoPay user reviews using IndoBERT-LoRA. The main problem addressed in this study was the limited representation of minority sentiment classes, particularly the neutral class, which could reduce the model’s ability to recognize all sentiment categories proportionally. The dataset consisted of 16,955 Google Play Store reviews that were manually labeled into positive, neutral, and negative classes. Two augmentation strategies were compared, namely prompt-based augmentation and fine-tuning augmentation. The generated synthetic data were evaluated using novelty, diversity, duplication, and manual validation of sampled reviews before being incorporated into the training data. The IndoBERT-LoRA model was trained under four scenarios: baseline, class weighting, prompt-based augmentation, and fine-tuning augmentation. The results showed that fine-tuning produced better lexical-level quality indicators, as indicated by more stable novelty and diversity scores and a lower duplication rate. Both augmentation strategies improved macro recall and macro F1-score compared with the baseline and class weighting scenario. The largest improvement occurred in the neutral class, where recall increased from 0.5633 to 0.7801 with prompt-based augmentation and to 0.8133 with fine-tuning augmentation. These findings indicate that GPT-4o-based augmentation improved minority-class recognition, although the improvement involved a trade-off between recall, precision, and implementation cost.

429 147

Downloads

Download data is not yet available.

References

[1] V. Oktaviani, B. Warsito, H. Yasin, R. Santoso, and Suparti, Sentiment analysis of e-commerce application in Traveloka data review on Google Play site using Nave Bayes classifier and association method, J. Phys. Conf. Ser., vol. 1943, no. 1, p. 012147, Jul. 2021, doi: 10.1088/1742-6596/1943/1/012147. DOI: https://doi.org/10.1088/1742-6596/1943/1/012147

[2] Y. Kang, Z. Cai, C.-W. Tan, Q. Huang, and H. Liu, Natural language processing (NLP) in Management Research: A literature review, Journal of Management Analytics, vol. 7, no. 2, pp. 139172, 2020, doi: 10.1080/23270012.2020.1756939. DOI: https://doi.org/10.1080/23270012.2020.1756939

[3] M. Wankhade, A. C. S. Rao, and C. Kulkarni, A survey on sentiment analysis methods, applications, and challenges, Artif. Intell. Rev., vol. 55, pp. 57315780, 2022, doi: 10.1007/s10462-022-10144-1. DOI: https://doi.org/10.1007/s10462-022-10144-1

[4] M. Bouazizi and T. Ohtsuki, Multi-class Sentiment Analysis on Twitter: Classification Performance and Challenges, Big Data Mining and Analytics, vol. 2, no. 3, pp. 181194, 2019, doi: 10.26599/BDMA.2019.9020002. DOI: https://doi.org/10.26599/BDMA.2019.9020002

[5] A. Mukherjee, S. Mukhopadhyay, P. K. Panigrahi, and S. Goswami, Utilization of Oversampling For Multiclass Sentiment Analysis on Amazon Review Dataset, in 2019 IEEE 10th International Conference on Awareness Science and Technology (iCAST), Morioka, Japan: IEEE, Oct. 2019, pp. 16. doi: 10.1109/ICAwST.2019.8923260. DOI: https://doi.org/10.1109/ICAwST.2019.8923260

[6] I. A. Rahma and L. H. Suadaa, Automated Indonesian Text Augmentation with Web-Based Application Using Flask Framework, Proceedings of The International Conference on Data Science and Official Statistics, vol. 2023, no. 1, pp. 96108, Dec. 2023, doi: 10.34123/icdsos.v2023i1.324. DOI: https://doi.org/10.34123/icdsos.v2023i1.324

[7] J. Jang, Y. Kim, K. Choi, and S. Suh, Sequential Targeting: A Continual Learning Approach For Data Imbalance in Text Classification, Expert Syst. Appl., vol. 179, p. 115067, Oct. 2021, doi: 10.1016/j.eswa.2021.115067. DOI: https://doi.org/10.1016/j.eswa.2021.115067

[8] D. R. Beddiar, M. S. Jahan, and M. Oussalah, Data Expansion Using Back Translation and Paraphrasing For Hate Speech Detection, Online Soc. Netw. Media, vol. 24, p. 100153, Jul. 2021, doi: 10.1016/j.osnem.2021.100153. DOI: https://doi.org/10.1016/j.osnem.2021.100153

[9] S. Woniak and J. Koco, From Big to Small Without Losing It All: Text Augmentation with ChatGPT for Efficient Sentiment Analysis, in 2023 IEEE International Conference on Data Mining Workshops (ICDMW), Shanghai, China: IEEE, Dec. 2023, pp. 799808. doi: 10.1109/ICDMW60847.2023.00108. DOI: https://doi.org/10.1109/ICDMW60847.2023.00108

[10] H. Dai et al., AugGPT: Leveraging ChatGPT for Text Data Augmentation, IEEE Trans. Big Data, vol. 11, no. 3, pp. 907918, Jun. 2025, doi: 10.1109/TBDATA.2025.3536934. DOI: https://doi.org/10.1109/TBDATA.2025.3536934

[11] S. V. Balkus and D. Yan, Improving short text classification with augmented data using GPT-3, Nat. Lang. Eng., vol. 30, no. 5, pp. 943972, 2024, doi: 10.1017/S1351324923000438. DOI: https://doi.org/10.1017/S1351324923000438

[12] C. Suhaeni and H.-S. Yong, Mitigating Class Imbalance in Sentiment Analysis through GPT-3-Generated Synthetic Sentences, Applied Sciences, vol. 13, no. 17, p. 9766, Aug. 2023, doi: 10.3390/app13179766. DOI: https://doi.org/10.3390/app13179766

[13] OpenAI, GPT-4o System Card, https://openai.com/index/gpt-4o-system-card/.

[14] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, BERT: Pre-Training of Deep Bidirectional Transformers for Language Understanding, in Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Stroudsburg, PA, USA: Association for Computational Linguistics, 2019, pp. 41714186. doi: 10.18653/v1/N19-1423. DOI: https://doi.org/10.18653/v1/N19-1423

[15] F. Koto, A. Rahimi, J. H. Lau, and T. Baldwin, IndoLEM and IndoBERT: A Benchmark Dataset and Pre-trained Language Model for Indonesian NLP, in Proceedings of the 28th International Conference on Computational Linguistics, Barcelona, Spain: International Committee on Computational Linguistics, Dec. 2020, pp. 757770. doi: 10.18653/v1/2020.coling-main.66. DOI: https://doi.org/10.18653/v1/2020.coling-main.66

[16] Y. A. Singgalen, Performance Analysis of IndoBERT for Sentiment Classification in Indonesian Hotel Review Data, Journal of Information System Research (JOSH), vol. 6, no. 2, pp. 978988, 2025. DOI: https://doi.org/10.47065/josh.v6i2.6505

[17] E. Hu et al., Lora: Low-Rank Adaptation of Large Language Models, ICLR, vol. 1, no. 2, pp. 322, 2022.

[18] R. Anadra, H. Wijayanto, and K. Sadik, Sentiment Analysis of Tokopedia Customer Reviews Using BiLSTM and IndoBERT with Comparative Analysis of Preprocessing and Labeling Methods, International Journal of Advances in Data and Information Systems, vol. 6, no. 3, pp. 773788, Dec. 2025, doi: 10.59395/ijadis.v6i3.1458. DOI: https://doi.org/10.59395/ijadis.v6i3.1458

[19] W. Chen, K. Yang, Z. Yu, Y. Shi, and C. L. P. Chen, A Survey on Imbalanced Learning: Latest Research, Applications and Future Directions, Artif. Intell. Rev., vol. 57, no. 6, pp. 137187, 2024, doi: 10.1007/s10462-024-10759-6. DOI: https://doi.org/10.1007/s10462-024-10759-6

[20] P. Thlke et al., Class Imbalance Should Not Throw You Off Balance: Choosing The Right Classifiers and Performance Metrics For Brain Decoding with Imbalanced Data, Neuroimage, vol. 277, p. 120253, 2023, doi: 10.1016/j.neuroimage.2023.120253. DOI: https://doi.org/10.1016/j.neuroimage.2023.120253

[21] C. Suhaeni, S. A. Kamila, F. Fahira, M. Yusran, and G. Alfa Dito, Exploring a Large Language Model on the ChatGPT Platform for Indonesian Text Preprocessing Tasks, Indonesian Journal of Statistics and Its Applications, vol. 9, no. 1, pp. 100116, Jun. 2025, doi: 10.29244/ijsa.v9i1p100-116. DOI: https://doi.org/10.29244/ijsa.v9i1p100-116

[22] M. Yusran, Siswanto, and A. Islamiyati, Comparison of Multinomial Nave Bayes and Bernoulli Nave Bayes on Sentiment Analysis of Kurikulum Merdeka with Query Expansion Ranking, Sistemasi: Jurnal Sistem Informasi, vol. 13, no. 1, pp. 96106, 2024. DOI: https://doi.org/10.32520/stmsi.v13i1.3187

[23] U. Hasanah, A. Mohamad Soleh, C. Suhaeni, and A. Fitrianto, EVALUATING ROBERTA AND GPT-BASED MODELS FOR SDG MULTICLASS TEXT CLASSIFICATION ACROSS DIFFERENT DOCUMENT LENGTHS, BAREKENG: Jurnal Ilmu Matematika dan Terapan, vol. 20, no. 3, pp. 26452664, Apr. 2026, doi: 10.30598/barekengvol20iss3pp2645-2664. DOI: https://doi.org/10.30598/barekengvol20iss3pp2645-2664

[24] S. Y. Feng et al., A Survey of Data Augmentation Approaches for NLP, in Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, Association for Computational Linguistics, 2021, pp. 968988. doi: 10.18653/v1/2021.findings-acl.84. DOI: https://doi.org/10.18653/v1/2021.findings-acl.84

[25] B. Li, Y. Hou, and W. Che, Data Augmentation Approaches in Natural Language Processing: A Survey, AI Open, vol. 3, pp. 7190, 2022, doi: 10.1016/j.aiopen.2022.03.001. DOI: https://doi.org/10.1016/j.aiopen.2022.03.001

[26] H. Zhao, H. Chen, T. A. Ruggles, Y. Feng, D. Singh, and H.-J. Yoon, Improving Text Classification with Large Language Model-Based Data Augmentation, Electronics (Basel)., vol. 13, no. 13, pp. 25352548, 2024, doi: 10.3390/electronics13132535. DOI: https://doi.org/10.3390/electronics13132535

[27] B. Ding et al., Data Augmentation using LLMs: Data Perspectives, Learning Paradigms and Challenges, in Findings of the Association for Computational Linguistics ACL 2024, Stroudsburg, PA, USA: Association for Computational Linguistics, 2024, pp. 16791705. doi: 10.18653/v1/2024.findings-acl.97. DOI: https://doi.org/10.18653/v1/2024.findings-acl.97

[28] X. Li and J. Li, Generative Deduplication For Socia Media Data Selection, in Findings of the Association for Computational Linguistics: EMNLP 2024, Stroudsburg, PA, USA: Association for Computational Linguistics, 2024, pp. 57655776. doi: 10.18653/v1/2024.findings-emnlp.330. DOI: https://doi.org/10.18653/v1/2024.findings-emnlp.330

[29] A. Vaswani et al., Attention is All you Need, in Advances in Neural Information Processing Systems, I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, Eds., Curran Associates, Inc., 2017. [Online]. Available: https://proceedings.neurips.cc/paper_files/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf

[30] N. A. Farhana, F. M. Afendi, A. Fitrianto, and S. H. Wijaya, Classification modeling of support vector machine (SVM) and random forest in predicting pharmacodynamics interactions, J. Phys. Conf. Ser., vol. 1863, no. 1, p. 012067, Mar. 2021, doi: 10.1088/1742-6596/1863/1/012067. DOI: https://doi.org/10.1088/1742-6596/1863/1/012067

[31] S. Farhadpour, T. A. Warner, and A. E. Maxwell, Selecting and Interpreting Multiclass Loss and Accuracy Assessment Metrics for Classifications with Class Imbalance: Guidance and Best Practices, Remote Sens. (Basel)., vol. 16, no. 3, p. 533, Jan. 2024, doi: 10.3390/rs16030533. DOI: https://doi.org/10.3390/rs16030533

[32] K. Takahashi, K. Yamamoto, A. Kuchiba, and T. Koyama, Confidence interval for micro-averaged F1 and macro-averaged F1 scores, Applied Intelligence, vol. 52, no. 5, pp. 49614972, Mar. 2022, doi: 10.1007/s10489-021-02635-5. DOI: https://doi.org/10.1007/s10489-021-02635-5

[33] A. E. Blanchard et al., A Keyword-Enhanced Approach to Handle Class Imbalance in Clinical Text Classification, IEEE J. Biomed. Health Inform., vol. 26, no. 6, pp. 27962803, Jun. 2022, doi: 10.1109/JBHI.2022.3141976. DOI: https://doi.org/10.1109/JBHI.2022.3141976