Performance Comparison of Hybrid Indobert-Self-Paced Ensemble and Indobert- Rusboost for Imbalanced Text Classification (Case Study: Insecurity Detection on Social Media X)

Emeylia Safitri; I Gusti Ngurah Sentana  Putra; Nuramaliyah Nuramaliyah; Ika Nur Laily  Fitriana; Ria  Faulina

doi:10.59395/ijadis.v7i2.1572

Authors

Emeylia Safitri Statistics Study Program, Faculty of Science and Technology, Universitas Terbuka, Indonesia
I Gusti Ngurah Sentana Putra Department of Statistic and Data Science, School of Data Science, Mathematics, and Informatics, IPB University, Indonesia
Nuramaliyah Nuramaliyah Statistics Study Program, Faculty of Science and Technology, Universitas Terbuka, Indonesia
Ika Nur Laily Fitriana Statistics Study Program, Faculty of Science and Technology, Universitas Terbuka, Indonesia
Ria Faulina Statistics Study Program, Faculty of Science and Technology, Universitas Terbuka, Indonesia

DOI:

https://doi.org/10.59395/ijadis.v7i2.1572

Keywords:

Insecurity detection, IndoBERT, RUSBoost, Self-Paced Ensemble, Imbalanced text classification

Abstract

Insecurity detection on social media is crucial for early mental health intervention, yet imbalanced data distribution poses significant challenges for text classification. This study compares the performance of hybrid IndoBERT-Self-Paced Ensemble (SPE) and IndoBERT- RUSBoost for imbalanced text classification in insecurity detection on social media X. A dataset of 5,179 Indonesian tweets was collected using keywords "insecure" and "overthinking", preprocessed through text cleaning and slang normalization, and labeled automatically using a fine-tuned IndoBERT model as insecurity (positive) or non-insecurity (negative). Results demonstrated that RUSBoost significantly outperformed SPE across 8 of 13 evaluation metrics, achieving balanced accuracy of 0.8521, F1-score of 0.6763, and AUC-ROC of 0.9459 compared to SPE's 0.8465, 0.5171, and 0.9242 respectively. RUSBoost showed superior specificity (0.9460 vs 0.8381) and precision (0.6104 vs 0.3706), while SPE excelled in recall (0.8548 vs 0.7581). Gradient Boosting achieved the highest AUC-PR (0.7100) and specificity (0.9622). The findings confirm that hybrid IndoBERT- RUSBoost provides more balanced and robust performance for imbalanced text classification, offering practical implications for early insecurity detection systems on social media platforms.

436 163

Downloads

Download data is not yet available.

References

[1] K. Burghardt, D. M. T. Fessler, C. Tang, A. Pisor, and K. Lerman, “Posts of Peril: Detecting Information About Hazards in Text,” 2025, [Online]. Available: http://arxiv.org/abs/2405.17838

[2] R. Sazama, A. Muttaqin, R. Ananda, R. Julianansa, M. Ula, and M. Eng, “Klasifikasi Cyberbullying Di Twitter Menggunakan Metode Gradient Boosting,” SENASTIKA Univ. Malikussaleh, pp. 1–9, 2024, [Online]. Available: https://proceedings.unimal.ac.id/senastika/issue/view/15

[3] S. Nouas, L. Oukid, and F. Boumahdi, “Enhancing imbalanced text classification: an overlap-based refinement approach,” Data Sci. Manag., vol. 8, no. 4, pp. 474–484, 2025, doi: 10.1016/j.dsm.2025.03.001. DOI: https://doi.org/10.1016/j.dsm.2025.03.001

[4] D. O. Putri and A. Qoiriah, “Klasifikasi Cyberbullying Pengguna Twitter Menggunakan Random Forest Dengan Penyematan Kata FastText,” J. Informatics Comput. Sci., vol. 6, no. 01, pp. 265–272, 2024, doi: 10.26740/jinacs.v6n01.p265-272. DOI: https://doi.org/10.26740/jinacs.v6n01.p265-272

[5] N. Abdulloh and A. F. Hidayatullah, “Deteksi Cyberbullying pada Cuitan Media Sosial Twitter,” Automata, vol. Vol 1, no. 1, pp. 1–5, 2021.

[6] S. F. Taskiran, B. Turkoglu, E. Kaya, and T. Asuroglu, “A comprehensive evaluation of oversampling techniques for enhancing text classification performance,” Sci. Rep., vol. 15, no. 1, pp. 1–21, 2025, doi: 10.1038/s41598-025-05791-7. DOI: https://doi.org/10.1038/s41598-025-05791-7

[7] A. Nawaz, A. Ahmad, and S. S. Khan, “Beyond Rebalancing: Benchmarking Binary Classifiers Under Class Imbalance Without Rebalancing Techniques,” 2025, [Online]. Available: http://arxiv.org/abs/2509.07605

[8] I. G. N. Sentana Putra, M. Yusran, J. R. Sari, C. Suhaeni, B. Sartono, and G. A. Dito, “Evaluating Fasttext and Glove Embeddings for Sentiment Analysis of AI-Generated Ghibli-Style Images,” J. Appl. Informatics Comput., vol. 9, no. 5, pp. 2574–2587, 2025, doi: 10.30871/jaic.v9i5.10600. DOI: https://doi.org/10.30871/jaic.v9i5.10600

[9] Z. Liu et al., “Self-paced ensemble for highly imbalanced massive data classification,” Proc. - Int. Conf. Data Eng., vol. 2020-April, pp. 841–852, 2020, doi: 10.1109/ICDE48307.2020.00078. DOI: https://doi.org/10.1109/ICDE48307.2020.00078

[10] S. Bano et al., “Self-paced ensemble and big data identification: a classification of substantial imbalance computational analysis,” J. Supercomput., vol. 80, no. 7, pp. 9848–9869, 2024, doi: 10.1007/s11227-023-05828-6. DOI: https://doi.org/10.1007/s11227-023-05828-6

[11] N. C. Ristea and R. T. Ionescu, “Self-paced ensemble learning for speech and audio classification,” Proc. Annu. Conf. Int. Speech Commun. Assoc. INTERSPEECH, vol. 2, pp. 1276–1280, 2021, doi: 10.21437/Interspeech.2021-155. DOI: https://doi.org/10.21437/Interspeech.2021-155

[12] C. Seiffert, T. M. Khoshgoftaar, J. Van Hulse, and A. Napolitano, “ RUSBoost: A hybrid approach to alleviating class imbalance,” IEEE Trans. Syst. Man, Cybern. Part ASystems Humans, vol. 40, no. 1, pp. 185–197, 2010, doi: 10.1109/TSMCA.2009.2029559. DOI: https://doi.org/10.1109/TSMCA.2009.2029559

[13] T. Fulazzaky, A. Saefuddin, and A. M. Soleh, “Evaluating Ensemble Learning Techniques for Class Imbalance in Machine Learning: A Comparative Analysis of Balanced Random Forest, SMOTE-RF, SMOTEBoost, and RUSBoost,” Sci. J. Informatics, vol. 11, no. 4, pp. 969–980, 2024, doi: 10.15294/sji.v11i4.15937. DOI: https://doi.org/10.15294/sji.v11i4.15937

[14] Y. Permatasari, “PENANGANAN MASALAH KELAS TIDAK SEIMBANG DENGAN RUSBOOST DAN UNDERBAGGING (STUDI KASUS: MAHASISWA DROP OUT SPs IPB PROGRAM MAGISTER),” Thesis, 2016, [Online]. Available: http://repository.ipb.ac.id/handle/123456789/80118

[15] N. A. Setiabudi, B. Sartono, U. D. Syafitri, and K. B. Aryasa, “Comparing Self-Paced Ensemble and RUSBoost for Imbalanced Poverty Classification in West Java,” Indones. J. Stat. Its Appl., vol. 9, no. 2, pp. 218–229, 2025, doi: 10.29244/ijsa.v9i2p218-229. DOI: https://doi.org/10.29244/ijsa.v9i2p218-229

[16] H. Chen and M. Ji, “Experimental Comparison of Classification Methods under Class Imbalance,” EAI Endorsed Trans. Scalable Inf. Syst., vol. 8, no. 33, pp. 1–20, 2021, doi: 10.4108/EAI.11-6-2021.170234. DOI: https://doi.org/10.4108/eai.11-6-2021.170234

[17] N. Putu, D. Agustina, I. D. Ayu, P. Pratiwi, I. G. Ngurah, and L. Wijayakusuma, “Public Sentiment Analysis on Demonstration Actions Using IndoBERT Based on Transfer Learning,” vol. 9, no. 6, 2026.

[18] A. A. Qolbu, N. Fitriyati, and N. Inayah, “Performa Naïve Bayes , SVM , dan IndoBERT pada Analisis Sentimen Twitter IndiHome dengan Strategi Penanganan Data Tidak Seimbang,” J. Fourier, vol. 814, no. 1, pp. 29–44, 2025, doi: 10.14421/fourier.2025.141.29-44.

[19] M. G. Al-Kadzim, R. Rasim, and H. Herbert, “Analisis Perubahan Sentimen Publik di Media Sosial X terhadap Konflik Palestina-Israel Menggunakan Model IndoBERT,” Digit. Transform. Technol., vol. 4, no. 2, pp. 1167–1174, 2025, doi: 10.47709/digitech.v4i2.5312. DOI: https://doi.org/10.47709/digitech.v4i2.5312

[20] A. F. Abdillah, C. B. P. Putra, A. Apriantoni, S. Juanita, and D. Purwitasari, “Ensemble-based Methods for Multi-label Classification on Biomedical Question-Answer Data,” J. Inf. Syst. Eng. Bus. Intell., vol. 8, no. 1, pp. 42–50, 2022, doi: 10.20473/jisebi.8.1.42-50. DOI: https://doi.org/10.20473/jisebi.8.1.42-50

[21] M. Anusha, “Hybrid Approach to Fake News Detection: Leveraging BERT-Based Model with Word Embedding Features for Sentiment Classification in Social Media Content,” J. Electr. Syst., vol. 20, no. 11s, pp. 1044–1060, 2024, doi: 10.52783/jes.7363. DOI: https://doi.org/10.52783/jes.7363

[22] M. F. Arifah, Irnawati, Ruslin, K. Nisa, A. Windarsih, and A. Rohman, “The Application of FTIR Spectroscopy and Chemometrics for the Authentication Analysis of Horse Milk,” Int. J. Food Sci., vol. 2022, 2022, doi: 10.1155/2022/7643959. DOI: https://doi.org/10.1155/2022/7643959

[23] Y. K. Sari, J. Iskandar, and A. Prasetya, “Penerapan Metode Naive Bayes Dalam Analisis Sentimen Terhadap Cyberbullying,” vol. 5, no. 1, 2026. DOI: https://doi.org/10.57218/juster.v5i1.2556

[24] M. Hassanzad and K. Hajian-Tilaki, “Methods of determining optimal cut-point of diagnostic biomarkers with application of clinical data in ROC analysis: an update review,” BMC Med. Res. Methodol., vol. 24, no. 1, pp. 1–14, 2024, doi: 10.1186/s12874-024-02198-2. DOI: https://doi.org/10.1186/s12874-024-02198-2

[25] D. Eko Putro, D. Juarsa, B. Putra Hermana, B. Bagastian, and H. Sulistiani, “Analisis Sentimen Publik terhadap ‘Save Raja Ampat’ di Media Sosial Menggunakan Model IndoBERT,” Bull. Comput. Sci. Res., vol. 5, no. 5, pp. 1067–1075, 2025, doi: 10.47065/bulletincsr.v5i5.621. DOI: https://doi.org/10.47065/bulletincsr.v5i5.621

[26] C. H. Miranda, G. Sanchez-Torres, and D. Salcedo, “Exploring the Evolution of Sentiment in Spanish Pandemic Tweets: A Data Analysis Based on a Fine-Tuned BERT Architecture,” Data, vol. 8, no. 6, 2023, doi: 10.3390/data8060096. DOI: https://doi.org/10.3390/data8060096