An Optimization Framework for Retrieval Augmented Generation in Indonesian Educational Question Answering

I Ketut Resika Arthana; Nyoman  Gunantara; Made  Sudarma; I Made Sukarsa

doi:10.59395/ijadis.v7i2.1633

Authors

I Ketut Resika Arthana Dept. of Informatics Universitas Pendidikan Ganesha, Indonesia
Nyoman Gunantara Faculty of Engineering, Universitas Udayana , Indonesia
Made Sudarma Faculty of Engineering, Universitas Udayana , Indonesia
I Made Sukarsa Faculty of Engineering, Universitas Udayana , Indonesia

DOI:

https://doi.org/10.59395/ijadis.v7i2.1633

Keywords:

RAG, LLM, Chunking, Embedding, RAGAS

Abstract

The Retrieval-Augmented Generation (RAG) approach has been widely adopted to produce responses that are more closely aligned with a predefined knowledge context. However, many RAG implementations have not undergone systematic optimization of their retrieval and generation components, resulting in outputs that do not always correspond accurately to the reference context. This study developed a RAG optimization framework for Indonesian-language educational question answering using a Human-Computer Interaction learning corpus as a case study. In the retrieval stage, the study evaluated chunking strategies, multilingual embedding models, and the use of a reranker. Evaluation was conducted using Mean Reciprocal Rank (MRR), Normalized Discounted Cumulative Gain (nDCG@K), and Hit@K. In the generation stage, candidate Large Language Models (LLMs) were assessed using RAGAS metrics, namely Context Precision (CP), Context Recall (CR), Faithfulness (F), Answer Relevancy (AR), and Answer Correctness (AC). Experimental results showed that the GTE configuration with fixed-size chunking and a reranker yielded the best retrieval performance, achieving an MRR of 0.9082, nDCG@5 of 0.9215, and Hit@5 of 0.9655. In the generation stage, Gemma 4 E4B exhibited the most balanced answer quality. The resulting framework provides a procedure for selecting retrieval and generation settings for a given corpus.

358 170

Downloads

Download data is not yet available.

References

[1] J. Chen, H. Lin, X. Han, and L. Sun, Benchmarking large language models in retrieval-augmented generation, in Proceedings of the AAAI Conference on Artificial Intelligence, 2024, vol. 38, no. 16. doi: 10.1609/aaai.v38i16.29728. DOI: https://doi.org/10.1609/aaai.v38i16.29728

[2] L. Huang et al., A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions, ACM Transactions on Information Systems, vol. 43, no. 2, pp. 155, Mar. 2025, doi: 10.1145/3703155. DOI: https://doi.org/10.1145/3703155

[3] I. K. R. Arthana, N. Gunantara, M. Sudarma, and M. Sukarsa, LoRA-based fine-tuning of local LLMs for hallucination detection in Indonesian RAG systems, International Journal of Advanced Computer Science and Applications, vol. 17, no. 3, 2026, doi: 10.14569/IJACSA.2026.0170389. DOI: https://doi.org/10.14569/IJACSA.2026.0170389

[4] Y. Gao et al., Retrieval-Augmented Generation for Large Language Models: A Survey, 2024, doi: https://doi.org/10.48550/arXiv.2312.10997.

[5] D. Danter, H. Mhle, and A. Stckl, Advanced Chunking and Search Methods for Improved Retrieval-Augmented Generation (RAG) System Performance in E-Learning, 2024. doi: 10.54941/ahfe1005756. DOI: https://doi.org/10.54941/ahfe1005756

[6] F. Golla, Enhancing Student Engagement Through AI-Powered Educational Chatbots: A Retrieval-Augmented Generation Approach, in 2024 21st International Conference on Information Technology Based Higher Education and Training, ITHET 2024, 2024. doi: 10.1109/ITHET61869.2024.10837678. DOI: https://doi.org/10.1109/ITHET61869.2024.10837678

[7] I. K. R. Arthana, N. Gunantara, M. Sudarma, and M. Sukarsa, A Systematic Literature Review of Retrieval-Augmented Generation Implementation for Enhancing Large Language Models in Education, Jurnal Nasional Pendidikan Teknik Informatika (JANAPATI), vol. 15, no. 1, pp. 91109, Mar. 2026, doi: 10.23887/janapati.v15i1.112281. DOI: https://doi.org/10.23887/janapati.v15i1.112281

[8] C. Sarmiento and E. J. M. Laura, Investigating Flavors of RAG for Applications in College Chatbots, in International Conference on Computer Supported Education, CSEDU - Proceedings, 2025, vol. 2, pp. 421428. doi: 10.5220/0013468200003932. DOI: https://doi.org/10.5220/0013468200003932

[9] S. Liang, X. Gong, F. Li, and C. Liao, UniEval-RAG: A unified end-to-end evaluation framework for RAG systems, in Proceedings - 2025 International Conference on Computer, Internet of Things and Smart City (CIoTSC 2025), 2025. doi: 10.1109/CIoTSC67482.2025.11412981. DOI: https://doi.org/10.1109/CIoTSC67482.2025.11412981

[10] J. Bossenz et al., Evaluation of Chunking and Embedding Strategies for Local Document Retrieval Using an Open-Source LLM in a Hospital, 2025. doi: 10.3233/SHTI251383. DOI: https://doi.org/10.3233/SHTI251383

[11] Y. Song, L. Liu, H. Wang, and J. Liu, A survey of retrieval granularity in retrieval-augmented generation, Data Analysis and Knowledge Discovery, 2026, doi: 10.11925/infotech.2096-3467.2025.0349.

[12] R. D. Pesl, J. G. Mathew, M. Mecella, and M. Aiello, Retrieval-Augmented Generation for Service Discovery: Chunking Strategies and Benchmarking, IEEE Transactions on Services Computing, pp. 115, 2026, doi: 10.1109/TSC.2026.3665441. DOI: https://doi.org/10.1109/TSC.2026.3665441

[13] X.-K. Koay, L.-Y. Ong, and P.-Y. Goh, Structure-Aware Chunking for Complex Tables in Retrieval-Augmented Generation Systems, Emerging Science Journal, vol. 10, no. 1, pp. 184205, Feb. 2026, doi: 10.28991/ESJ-2026-010-01-09. DOI: https://doi.org/10.28991/ESJ-2026-010-01-09

[14] Y. Niu and X. Rong, Improving retrieval-augmented generation for educational policy understanding via structure-aware text chunking, in IECA 2026, 2026. doi: 10.1145/3802133.3802307. DOI: https://doi.org/10.1145/3802133.3802307

[15] Z. Yu, S. Liu, P. Denny, A. Bergen, and M. Liut, Integrating Small Language Models with Retrieval-Augmented Generation in Computing Education: Key Takeaways, Setup, and Practical Insights, in SIGCSE TS 2025 - Proceedings of the 56th ACM Technical Symposium on Computer Science Education, 2025, vol. 1, pp. 13021308. doi: 10.1145/3641554.3701844. DOI: https://doi.org/10.1145/3641554.3701844

[16] S. F. Chaerul Haviana, M. Riyadi, and R. Kusumaningrum, Evaluation of chunking strategies in RAG application for explicit retrieval on Indonesian language scientific papers, in Proceedings of EECSI 2025, 2025. doi: 10.1109/EECSI67060.2025.11290624. DOI: https://doi.org/10.1109/EECSI67060.2025.11290624

[17] J. Cohen, A Coefficient of Agreement for Nominal Scales, Educational and Psychological Measurement, vol. 20, no. 1, pp. 3746, Apr. 1960, doi: 10.1177/001316446002000104. DOI: https://doi.org/10.1177/001316446002000104

[18] K. Doxolodeo and A. A. Krisnadhi, AC-IQuAD: Automatically Constructed Indonesian Question Answering Dataset by Leveraging Wikidata, Language Resources and Evaluation, vol. 59, no. 1, pp. 135160, Mar. 2025, doi: 10.1007/s10579-023-09702-y. DOI: https://doi.org/10.1007/s10579-023-09702-y

[19] L. Zhang and Y. Ning, Improving construction contract question answering through embedding optimization and semantic chunking in large language models, Advanced Engineering Informatics, vol. 69, p. 104027, Jan. 2026, doi: 10.1016/j.aei.2025.104027. DOI: https://doi.org/10.1016/j.aei.2025.104027

[20] N. Muennighoff, N. Tazi, L. Magne, and N. Reimers, MTEB: Massive Text Embedding Benchmark, in Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, 2023, pp. 20142037. doi: 10.18653/v1/2023.eacl-main.148. DOI: https://doi.org/10.18653/v1/2023.eacl-main.148

[21] K. Jrvelin and J. Keklinen, Cumulated gain-based evaluation of IR techniques, ACM Transactions on Information Systems, vol. 20, no. 4, pp. 422446, Oct. 2002, doi: 10.1145/582415.582418. DOI: https://doi.org/10.1145/582415.582418

[22] J. Chen, S. Xiao, P. Zhang, K. Luo, D. Lian, and Z. Liu, M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillation, in Findings of the Association for Computational Linguistics ACL 2024, 2024, pp. 23182335. doi: 10.18653/v1/2024.findings-acl.137. DOI: https://doi.org/10.18653/v1/2024.findings-acl.137

[23] R. Nogueira and K. Cho, Passage Re-ranking with BERT, Apr. 2020, doi: https://doi.org/10.48550/arXiv.1901.04085.

[24] H. Yu, A. Gan, K. Zhang, S. Tong, Q. Liu, and Z. Liu, Evaluation of Retrieval-Augmented Generation: A Survey, Jul. 2024, doi: 10.1007/978-981-96-1024-2_8. DOI: https://doi.org/10.1007/978-981-96-1024-2_8

[25] A. Salemi and H. Zamani, Evaluating Retrieval Quality in Retrieval-Augmented Generation, in Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, Jul. 2024, pp. 23952400. doi: 10.1145/3626772.3657957. DOI: https://doi.org/10.1145/3626772.3657957

[26] S. Es, J. James, L. Espinosa Anke, and S. Schockaert, RAGAs: Automated Evaluation of Retrieval Augmented Generation, in Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations, 2024, pp. 150158. doi: 10.18653/v1/2024.eacl-demo.16. DOI: https://doi.org/10.18653/v1/2024.eacl-demo.16

[27] J. R. Landis and G. G. Koch, The Measurement of Observer Agreement for Categorical Data, Biometrics, vol. 33, no. 1, p. 159, Mar. 1977, doi: 10.2307/2529310. DOI: https://doi.org/10.2307/2529310

[28] S. Robertson and H. Zaragoza, The Probabilistic Relevance Framework: BM25 and Beyond, Foundations and Trends in Information Retrieval, vol. 4, no. 12, pp. 1174, Sep. 2009, doi: 10.1561/1500000019. DOI: https://doi.org/10.1561/1500000019

[29] T. akar and H. Emekci, Maximizing RAG efficiency: A comparative analysis of RAG methods, Natural Language Processing, vol. 31, no. 1, pp. 125, Jan. 2025, doi: 10.1017/nlp.2024.53. DOI: https://doi.org/10.1017/nlp.2024.53

[30] M. Zhou, J. Tang, W. Zeng, and X. Zhao, INKER: Adaptive dynamic retrieval augmented generation with internal-external knowledge integration, Information Processing & Management, vol. 63, no. 3, p. 104534, Apr. 2026, doi: 10.1016/j.ipm.2025.104534. DOI: https://doi.org/10.1016/j.ipm.2025.104534