A Performance Enhancement Strategy for Sentiment Classification Models On Political Social Media Using Hyperparameter Tuning And Boosting

Andi Supriadi Chan; Meryatul  Husna; Pandu Pratama  Putra

doi:10.59395/ijadis.v6i3.1455

Authors

Andi Supriadi Chan Politeknik Negeri Medan
Meryatul Husna Politeknik Negeri Medan
Pandu Pratama Putra Universitas Lancang Kuning

DOI:

https://doi.org/10.59395/ijadis.v6i3.1455

Keywords:

Sentiment Analysis, Text Classification, Optuna, Machine Learning, XGBoost.

Abstract

This study aims to develop an optimized machine learning-based sentiment classification model for election-related issues. A dataset comprising 10,001 entries was collected from the social media platform X and manually labeled into three sentiment classes: positive, negative, and neutral. The preprocessing stage involved text cleaning, stemming, and feature transformation using the Term Frequency-Inverse Document Frequency (TF-IDF) method. To address class imbalance, the Synthetic Minority Over-sampling Technique (SMOTE) was employed. Three baseline classification algorithms—K-Nearest Neighbors (KNN), Support Vector Machine (SVM), and Gaussian Naive Bayes (GNB)—were initially evaluated to establish a performance benchmark. Model development proceeded by applying hyperparameter optimization using the Optuna framework and further enhancing the models via boosting with Extreme Gradient Boosting (XGBoost). Experimental results revealed that the combination of SVM with Optuna and XGBoost achieved the best performance, reaching 97% accuracy, precision, recall, and F1-score across all classes. In contrast, the KNN and GNB models experienced a notable decline in performance following hyperparameter tuning, although partial recovery was observed when combined with boosting. These findings suggest that hyperparameter tuning and boosting are not universally effective across all classifiers, yet their synergistic application significantly enhances performance in SVM-based models. This study highlights the importance of model-specific optimization strategies in building robust sentiment analysis systems, particularly for handling unbalanced public opinion data in social media contexts.

1190 703

Downloads

Download data is not yet available.

References

[1] W. Khan, M. A. Ghazanfar, M. A. Azam, A. Karami, K. H. Alyoubi, and A. S. Alfakeeh, “Stock market prediction using machine learning classifiers and social media, news,” J Ambient Intell Humaniz Comput, vol. 13, no. 7, pp. 3433–3456, Jul. 2022, doi: 10.1007/s12652-020-01839-w. DOI: https://doi.org/10.1007/s12652-020-01839-w

[2] M. Orabi, D. Mouheb, Z. Al Aghbari, and I. Kamel, “Detection of Bots in Social Media: A Systematic Review,” Inf Process Manag, vol. 57, no. 4, p. 102250, 2020, doi: 10.1016/j.ipm.2020.102250. DOI: https://doi.org/10.1016/j.ipm.2020.102250

[3] E. Konovalova, G. Le Mens, and N. Schöll, “Social media feedback and extreme opinion expression,” PLoS One, vol. 18, no. 11, pp. 1–20, Nov. 2023, doi: 10.1371/journal.pone.0293805. DOI: https://doi.org/10.1371/journal.pone.0293805

[4] X. T. Dang and T. T. Le, “KNN-SMOTE: An Innovative Resampling Technique Enhancing the Efficacy of Imbalanced Biomedical Classification,” in Machine Learning and Other Soft Computing Techniques: Biomedical and Related Applications, N. Hoang Phuong, N. T. Huyen Chau, and V. Kreinovich, Eds., Cham: Springer Nature Switzerland, 2024, pp. 111–121. doi: 10.1007/978-3-031-63929-6_11. DOI: https://doi.org/10.1007/978-3-031-63929-6_11

[5] V. Gupta and P. Rattan, “Improving Twitter Sentiment Analysis Efficiency with SVM-PSO Classification and EFWS Heuristic,” in Procedia Computer Science, Elsevier B.V., 2023, pp. 698–715. doi: 10.1016/j.procs.2023.12.125. DOI: https://doi.org/10.1016/j.procs.2023.12.125

[6] I. F. Ashari, M. C. Untoro, E. M. Sutowo, D. Salsabila, and D. A. Zhabiyan, “Hyperparameter Tuning Feature Selection with Genetic Algorithm and Gaussian Naïve Bayes for Diabetes Disease Prediction,” Jurnal Telematika, vol. 17, no. 1, pp. 17–23, 2022, doi: 10.61769/jurtel.v17i1.488. DOI: https://doi.org/10.61769/telematika.v17i1.488

[7] F. Suandi et al., “Enhancing Sentiment Analysis Performance Using SMOTE and Majority Voting in Machine Learning Algorithms,” in International Conference on Applied Engineering, Atlantis Press, 2024, pp. 126–138. doi: 10.2991/978-94-6463-620-8_10. DOI: https://doi.org/10.2991/978-94-6463-620-8_10

[8] S. Hadianti and W. A. G. Kodri, “Optimization of The Machine Learning Approach using Optuna in Heart Disease Prediction,” Journal Medical Informatics Technology, vol. 1, no. 3, pp. 59–64, Sep. 2023, doi: 10.37034/medinftech.v1i3.15. DOI: https://doi.org/10.37034/medinftech.v1i3.15

[9] P. Srinivas and R. Katarya, “hyOPTXg: OPTUNA hyper-parameter optimization framework for predicting cardiovascular disease using XGBoost,” Biomed Signal Process Control, vol. 73, no. 1, pp. 1–10, Mar. 2022, doi: 10.1016/j.bspc.2021.103456. DOI: https://doi.org/10.1016/j.bspc.2021.103456

[10] Y. Li, C. Stasinakis, and W. M. Yeo, “A Hybrid XGBoost-MLP Model for Credit Risk Assessment on Digital Supply Chain Finance,” Forecasting, vol. 4, no. 1, pp. 184–207, Mar. 2022, doi: 10.3390/forecast4010011. DOI: https://doi.org/10.3390/forecast4010011

[11] R. Syahputra, G. J. Yanris, and D. Irmayani, “SVM and Naïve Bayes Algorithm Comparison for User Sentiment Analysis on Twitter,” Sinkron, vol. 7, no. 2, pp. 671–678, May 2022, doi: 10.33395/sinkron.v7i2.11430. DOI: https://doi.org/10.33395/sinkron.v7i2.11430

[12] R. Pambudi and F. Madani, “Analysis of public opinion sentiment against COVID-19 in Indonesia on twitter using the k-nearest neighbor algorithm and decision tree,” Journal of Soft Computing Exploration, vol. 3, no. 2, pp. 117–122, Sep. 2022, doi: 10.52465/joscex.v3i2.88. DOI: https://doi.org/10.52465/joscex.v3i2.88

[13] I. Setiawan et al., “Utilizing Random Forest Algorithm for Sentiment Prediction Based on Twitter Data,” in Proceedings of the First Mandalika International Multi-Conference on Science and Engineering 2022, MIMSE 2022 (Informatics and Computer Science), Atlantis Press International BV, 2022, pp. 446–456. doi: 10.2991/978-94-6463-084-8_37. DOI: https://doi.org/10.2991/978-94-6463-084-8_37

[14] A. Saekhu and D. Intan Surya Saputra, “Enhancing Student Sentiment Classification on AI in Education using SMOTE and Naive Bayes,” Technology and Science (BITS), vol. 6, no. 4, pp. 2165–2174, 2025, doi: 10.47065/bits.v6i4.6469.

[15] M. Idris, A. Rifai, and K. D. Tania, “Sentiment Analysis of Tokopedia App Reviews using Machine Learning and Word Embeddings,” sinkron, vol. 9, no. 1, pp. 210–219, Jan. 2025, doi: 10.33395/sinkron.v9i1.14278. DOI: https://doi.org/10.33395/sinkron.v9i1.14278

[16] A. Samih, A. Ghadi, and A. Fennan, “Enhanced sentiment analysis based on improved word embeddings and XGboost,” International Journal of Electrical and Computer Engineering, vol. 13, no. 2, pp. 1827–1836, Apr. 2023, doi: 10.11591/ijece.v13i2.pp1827-1836. DOI: https://doi.org/10.11591/ijece.v13i2.pp1827-1836

[17] M. K. Anam et al., “Sara Detection on Social Media Using Deep Learning Algorithm Development,” Journal of Applied Engineering and Technological Science, vol. 6, no. 1, pp. 225–237, Dec. 2024, doi: 10.37385/jaets.v6i1.5390. DOI: https://doi.org/10.37385/jaets.v6i1.5390

[18] V. Talasila, M. V Mohan, and N. M. R, “Enhancing Text-to-Image Synthesis with an Improved Semi-Supervised Image Generation Model Incorporating N-Gram, Enhanced TF-IDF, and BOW Techniques,” International Journal of Intelligent Systems and Applications in Engineering , vol. 11, no. 7s, pp. 381–397, 2023, [Online]. Available: www.ijisae.org

[19] A. Masitha and M. Kunta Biddinika, “Preparing Dual Data Normalization for KNN Classfication in Prediction of Heart Failure,” KLIK, vol. 4, no. 3, pp. 1227–1234, 2023, doi: 10.30865/klik.v4i3.1382.

[20] Y. A. Singgalen, “Performance Evaluation of Sentiment Classification Models: A Comparative Study of NBC, SVM, and DT with SMOTE,” Media Online, vol. 4, no. 5, pp. 2539–2548, Apr. 2024, doi: 10.30865/klik.v4i5.1827.

[21] T. P. Handayani, W. Hasyim, and Nursetia, “Preliminary Evaluation of Gaussian Naive Bayes for Multi-Label Hate Speech and Abusive Language Detection on Indonesian Twitter,” Journal of International Multidisciplinary Research, vol. 1, no. 1, pp. 159–165, Nov. 2023, doi: 10.62504/jimr532. DOI: https://doi.org/10.62504/jimr532

[22] P. P. Putra, M. K. Anam, A. S. Chan, A. Hadi, N. Hendri, and A. Masnur, “Optimizing Sentiment Analysis on Imbalanced Hotel Review Data Using SMOTE and Ensemble Machine Learning Techniques,” Journal of Applied Data Sciences, vol. 6, no. 2, pp. 936–951, May 2025, doi: 10.47738/jads.v6i2.618. DOI: https://doi.org/10.47738/jads.v6i2.618

[23] M. K. Anam, T. P. Lestari, H. Yenni, T. Nasution, and M. B. Firdaus, “Enhancement of Machine Learning Algorithm in Fine-grained Sentiment Analysis Using the Ensemble,” ECTI Transactions on Computer and Information Technology (ECTI-CIT), vol. 19, no. 2, pp. 159–167, Mar. 2025, doi: 10.37936/ecti-cit.2025192.257815. DOI: https://doi.org/10.37936/ecti-cit.2025192.257815

[24] L. L. Van Fc, M. K. Anam, M. B. Firdaus, Y. Yunefri, and N. A. Rahmi, “Enhancing Machine Learning Model Performance in Addressing Class Imbalance,” COGITO Smart Journal, vol. 10, no. 1, pp. 478–490, 2024. DOI: https://doi.org/10.31154/cogito.v10i1.626.478-490

[25] A. N. Ulfah, M. K. Anam, N. Y. S. Munti, S. Yaakub, and M. B. Firdaus, “Sentiment Analysis of the Convict Assimilation Program on Handling Covid-19,” JUITA: Jurnal Informatika, vol. 10, no. 2, pp. 209–216, 2022, doi: 10.30595/juita.v10i2.12308. DOI: https://doi.org/10.30595/juita.v10i2.12308

[26] R. Rahmaddeni, M. K. Anam, Y. Irawan, S. Susanti, and M. Jamaris, “Comparison of Support Vector Machine and XGBSVM in Analyzing Public Opinion on Covid-19 Vaccination,” ILKOM Jurnal Ilmiah, vol. 14, no. 1, pp. 32–38, Apr. 2022, doi: 10.33096/ilkom.v14i1.1090.32-38. DOI: https://doi.org/10.33096/ilkom.v14i1.1090.32-38

[27] C. G. Özmen and S. Gündüz, “Comparison of Machine Learning Models for Sentiment Analysis of Big Turkish Web-Based Data †,” Applied Sciences (Switzerland), vol. 15, no. 5, pp. 1–20, Mar. 2025, doi: 10.3390/app15052297. DOI: https://doi.org/10.3390/app15052297