Performance Evaluation of the XGBoost Method for Measuring Uncertainty in Small Area Estimation

Fida Fariha Amatullah; Khairil Anwar Notodiputro; Anwar Fitrianto

doi:10.59395/ijadis.v7i2.1621

Authors

Fida Fariha Amatullah Statistics and Data Science, School of Data Science, Mathematics, and Informatics, IPB University, Bogor, Indonesia
Khairil Anwar Notodiputro Statistics and Data Science, School of Data Science, Mathematics, and Informatics, IPB University, Bogor, Indonesia
Anwar Fitrianto Statistics and Data Science, School of Data Science, Mathematics, and Informatics, IPB University, Bogor, Indonesia

DOI:

https://doi.org/10.59395/ijadis.v7i2.1621

Keywords:

EBLUP-FH, Bootstrap, Small Area Estimation, Winsorization, XGBoost

Abstract

This study looks at using Small Area Estimation (SAE) to assess per capita spending in subdistricts of Jambi Province. Direct estimators in small areas usually show high variance because of small sample sizes, so model-based methods are needed. The Fay–Herriot model based on EBLUP–FH was used as the primary indirect estimation method, while XGBoost served as a flexible machine learning benchmark. These approaches show fundamentally different methods. EBLUP-FH acts as a model-based estimator with a clear sampling-error framework, while XGBoost functions as a prediction model. Its uncertainty is evaluated through bootstrap resampling, so their standard errors can't be directly compared. Before modeling, data was cleaned and winsorized to lessen outlier impact. The normality assumption for the sampling error in EBLUP-FH was breached, and a logarithmic transformation was used which improved the model components' distribution. XGBoost achieved the smallest bootstrap-based standard error, indicating reduced variability in predictions across resamples rather than greater estimation accuracy. Subdistrict-level analysis revealed that XGBoost tends to produce homogeneous estimates that fail to capture contextual socioeconomic variation, confirming that a small bootstrap standard error does not necessarily indicate accurate small-area estimates.

288 139

Downloads

Download data is not yet available.

References

[1] J. N. K. Rao and I. Molina, Small Area Estimation, 2nd ed. United States of America: John Wiley & Sons, Inc., 2015.

[2] A. O. Fengki, K. A. Notodiputro, and K. Sadik, Bisakah Memperoleh Statistik Indeks Harga Konsumen Tingkat Provinsi Di Indonesia Dengan Ketelitian Yang Lebih Baik?, Semin. Nas. Off. Stat., vol. 2019, no. 1, pp. 297306, 2020, doi: 10.34123/semnasoffstat.v2019i1.178. DOI: https://doi.org/10.34123/semnasoffstat.v2019i1.178

[3] S. M. Mah, M. Brown, R. C. Colley, L. C. Rosella, G. Schellenberg, and C. Sanmartin, Exploring the use of experimental small area estimates to examine the relationship between individual-level and area-level community belonging and self-rated health, Heal. Reports, vol. 35, no. 3, 2024, doi: 10.25318/82-003-x202400300001-eng.

[4] A. Ubaidillah et al., Multivariate Fay-Herriot models for small area estimation with application to household consumption per capita expenditure in Indonesia, J. Appl. Stat., vol. 0, no. 0, pp. 117, 2019, doi: 10.1080/02664763.2019.1615420. DOI: https://doi.org/10.1080/02664763.2019.1615420

[5] S. Sugasawa and T. Kubokawa, Mixed-Effects Models and Small Area Estimation. 2023. DOI: https://doi.org/10.1007/978-981-19-9486-9

[6] S. Elahe, D. Shahsavani, and M. Reza, Small area estimation with partially linear mixed-t model with measurement error, J. Comput. Appl. Math., vol. 446, pp. 112, 2024, doi: 10.1016/j.cam.2024.115871. DOI: https://doi.org/10.1016/j.cam.2024.115871

[7] T. C. Nokeri, Data Science Solutions with Python. New York: Apress, 2022. DOI: https://doi.org/10.1007/978-1-4842-7762-1

[8] C. Robinson, B. Dilkina, W. Zhang, S. Guhathakurta, M. A. Brown, and R. M. Pendyala, Machine learning approaches for estimating commercial building energy consumption, Appl. Energy, no. May, pp. 889904, 2017, doi: 10.1016/j.apenergy.2017.09.060. DOI: https://doi.org/10.1016/j.apenergy.2017.09.060

[9] C. E. Kontokosta, B. Hong, N. E. Johnson, and D. Starobin, Computers , Environment and Urban Systems Using machine learning and small area estimation to predict building-level municipal solid waste generation in cities, Comput. Environ. Urban Syst., no. March, pp. 151162, 2018, doi: 10.1016/j.compenvurbsys.2018.03.004. DOI: https://doi.org/10.1016/j.compenvurbsys.2018.03.004

[10] T. Chen and C. Guestrin, XGBoost: A scalable tree boosting system, Proc. ACM SIGKDD Int. Conf. Knowl. Discov. Data Min., vol. 13-17-Augu, pp. 785794, 2016, doi: 10.1145/2939672.2939785. DOI: https://doi.org/10.1145/2939672.2939785

[11] M. Viljanen, L. Meijerink, L. Zwakhals, and J. Van De Kassteele, A machine learning approach to small area estimation: predicting the health , housing and well being of the population of Netherlands, Int. J. Health Geogr., pp. 118, 2022, doi: 10.1186/s12942-022-00304-5. DOI: https://doi.org/10.1186/s12942-022-00304-5

[12] W. Zhang, C. Wu, H. Zhong, Y. Li, and L. Wang, Prediction of undrained shear strength using extreme gradient boosting and random forest based on Bayesian optimization, Geosci. Front., vol. 12, no. 1, pp. 469477, 2021, doi: 10.1016/j.gsf.2020.03.007. DOI: https://doi.org/10.1016/j.gsf.2020.03.007

[13] Z. Shao, M. N. Ahmad, and A. Javed, Comparison of Random Forest and XGBoost Classifiers Using Integrated Optical and SAR Features for Mapping Urban Impervious Surface, Remote Sens., vol. 16, no. 4, 2024, doi: 10.3390/rs16040665. DOI: https://doi.org/10.3390/rs16040665

[14] J. Frank E. Harrell, Regression Modeling Strategies, 2nd ed. New York: Springer, 2015. DOI: https://doi.org/10.1007/978-3-319-19425-7

[15] C. F. Dormann et al., Collinearity: A review of methods to deal with it and a simulation study evaluating their performance, Ecography (Cop.)., vol. 36, no. 1, pp. 2746, 2013, doi: 10.1111/j.1600-0587.2012.07348.x. DOI: https://doi.org/10.1111/j.1600-0587.2012.07348.x

[16] A. C. Davison and D. V. Hinkley, Bootstrap methods and their application. New York: Cambridge University Press, 1997. DOI: https://doi.org/10.1017/CBO9780511802843

[17] S. Hermawan and Amirullah, Metode Penelitian Pendekatan Kuantitatif Kualitatif. Sidoarjo: Universitas Muhammadiyah Sidoarjo, 2016.

[18] W. A. Nurizza, Penerapan Model Fay-Herriot pada Small Area Estimation Studi Simulasi Pengeluaran Per Kapita Level Kabupaten/Kota Provinsi Kalimantan Timur 2020, BESTARI Bul. Statisitika dan Apl. Terkini, vol. 1, no. 1, pp. 17, 2021.

[19] R. Ananda, K. A. Notodiputro, and M. N. Aidi, Modified Mixed Effects Random Forest in Small Area Estimation Using PCA and Rotation Forest with Correlated Auxiliary Variables, Sci. J. Informatics, vol. 11, no. 3, pp. 705720, 2024, doi: 10.15294/sji.v11i3.10633. DOI: https://doi.org/10.15294/sji.v11i3.10633

[20] S. L. Lohr, Sample Surveys: Model-Based Approaches, Int. Encycl. Soc. Behav. Sci., vol. 20, pp. 886889, 2015, doi: 10.1016/B978-0-08-097086-8.42166-5. DOI: https://doi.org/10.1016/B978-0-08-097086-8.42166-5

[22] M. Sagming, R. Heymann, and M. V. Visaya, Using topological data analysis and machine learning to predict customer churn, J. Big Data, vol. 11, pp. 131, 2024, doi: 10.1186/s40537-024-01020-6. DOI: https://doi.org/10.1186/s40537-024-01020-6

[23] L. Yang, X. Zhang, and J. Chen, Winsorization greatly reduces false positives by popular differential expression methods when analyzing human population samples, Genome Biol., vol. 25, pp. 14, 2024, doi: 10.1186/s13059-024-03230-w. DOI: https://doi.org/10.1186/s13059-024-03230-w

[24] H. Bichri, A. Chergui, and M. Hain, Investigating the Impact of Train / Test Split Ratio on the Performance of Pre-Trained Models with Custom Datasets, Int. J. Adv. Comput. Sci. Appl., vol. 15, no. 2, pp. 331339, 2024, doi: 10.14569/IJACSA.2024.0150235. DOI: https://doi.org/10.14569/IJACSA.2024.0150235

[25] E. Vivas, H. Allende-Cid, and R. Salas, A systematic review of statistical and machine learning methods for electrical power forecasting with reported mape score, Entropy, vol. 22, no. 12, pp. 124, 2020, doi: 10.3390/e22121412. DOI: https://doi.org/10.3390/e22121412

[26] BPS Jambi, Provinsi Jambi Dalam Angka 2022. Jambi: BPS Provinsi Jambi, 2022.

[27] A. Moretti and A. Whitworth, Estimating the Uncertainty of a Small Area Estimator Based on a Microsimulation Approach, Sociol. Methods Res., vol. 52, no. 4, pp. 17851815, 2023, doi: 10.1177/0049124120986199. DOI: https://doi.org/10.1177/0049124120986199

[28] N. Tzavidis, L. Zhang, and A. Luna, From start to finish: a framework for the production of small area official statistics, J. R. Stat. Soc. Ser. A, vol. 4, pp. 927979, 2018, doi: https://doi.org/10.1111/rssa.12364. DOI: https://doi.org/10.1111/rssa.12364

[29] G. James, D. Witten, T. Hastie, and R. Tibshirani, An Introduction to Statistical Learning, 2nd ed. New York: Springer Nature, 2021. DOI: https://doi.org/10.32614/CRAN.package.ISLR2

[30] P. Corral, I. Molina, A. Cojocaru, and S. Segovia, GUIDELINES TO SMALL AREA ESTIMATION FOR. Washington DC: The World Bank, 2022. DOI: https://doi.org/10.1596/37728