Enhancing AI-driven Cybersecurity Awareness Smart Consultant using RAG Method with Hybrid Knowledge Based

I Putu Agus Eka Pratama; I Made Oka  Widyantara; Linawati Linawati; Nyoman Gunantara

doi:10.59395/ijadis.v7i1.1488

Authors

I Putu Agus Eka Pratama Doctoral Program of Engineering Sciences, Faculty of Engineering, Udayana University, Bali, Indonesia
I Made Oka Widyantara Department of Electrical Engineering, Faculty of Engineering, Udayana University, Bali, Indonesia
Linawati Linawati Department of Electrical Engineering, Faculty of Engineering, Udayana University, Bali, Indonesia
Nyoman Gunantara Department of Electrical Engineering, Faculty of Engineering, Udayana University, Bali, Indonesia

DOI:

https://doi.org/10.59395/ijadis.v7i1.1488

Keywords:

Answer Accuracy Score (AAS), AI-driven , Cybersecurity awareness, Hybrid knowledge based, Retrieval Augmented Generation (RAG)

Abstract

The rapid advancement of Artificial Intelligence (AI), particularly in the realm of Large Language Model (LLM), holds significant potential for addressing the escalating issue of cyberattacks that exploit users' insufficient cybersecurity awareness. This study involves the design and development of a prototype cybersecurity awareness smart consultant, leveraging AI through the Retrieval Augmented Generation (RAG) method. This approach integrates hybrid knowledge derived from both user-specific internal cybersecurity documents and internet resources, thereby enhancing the validity of system responses and mitigating the risk of hallucinations. The prototype was evaluated using the Answer Accuracy Score (AAS) method, based on Black Box Testing and human evaluation, across four cybersecurity-related questions, yielding an average score of 0.925, accompanied by comprehensive analysis and discussion. The findings indicate that the system's response accuracy improves when knowledge is synthesized from both internal document resources and internet sources. Future research may focus on incorporating deliberative thinking to further enhance system performance in generating responses.

1322 643

Downloads

Download data is not yet available.

References

[1] R. Tashfiq, “Human Factors in Cybersecurity: A Scoping Review,” 12th Int. Conf. Adv. Inf. Technol. IAIT2021, 2021, doi: 10.1145/3468784.3468789. DOI: https://doi.org/10.1145/3468784.3468789

[2] T. Ostonya, B. M. Muhaimin, M. T. Jean, and R. Mizanur, “Transportation Cyber Incident Awareness through Generative AI-Based Incident Analysis and Retrieval-Augmented Question-Answering Systems,” ArXiv250802523 Cryptogr. Secur., vol. 1, 2025, doi: 10.48550/arXiv.2508.02523.

[3] C. Silvia, C. Francesco, N. Fabio, and P. Giulia, “Transforming threats into opportunities: The role of human factors in enhancing cybersecurity,” J. Innov. Knowl., vol. 10, no. 3, 2025, doi: 10.1016/j.jik.2025.100695. DOI: https://doi.org/10.1016/j.jik.2025.100695

[4] S. Alahmari, K. Renaud, and I. Omoronyia, “Moving beyond cyber security awareness and training to engendering security knowledge sharing,” Inf Syst E-Bus Manage, vol. 21, pp. 123–158, 2023, doi: 10.1007/s10257-022-00575-2. DOI: https://doi.org/10.1007/s10257-022-00575-2

[5] D. C. Nikitha, L. Anthony, and M. Azab, “Understanding User Behavior for Enhancing Cybersecurity Training with Immersive Gamified Platforms,” Information, vol. 15, no. 12, p. 814, 2024, doi: 10.3390/info15120814. DOI: https://doi.org/10.3390/info15120814

[6] O. A. Temitayo, A. F. Oluwatoyin, S. Kaggwa, and U. U. Prisca, “Cybersecurity Awareness and Education Programs: A Review of Employee Engagement and Accountability,” Comput. Sci. IT Res. J., vol. 5, no. 1, 2024, doi: 10.51594/csitrj.v5i.708. DOI: https://doi.org/10.51594/csitrj.v5i1.708

[7] A. T. Neumann, Y. Yin, S. Sowe, and S. Decker, “An LLM-Driven Chatbot in Higher Education for Databases and Information Systems,” IEEE Trans. Educ., vol. 68, no. 1, pp. 103–116, 2025, doi: 10.1109/TE.2024.3467912. DOI: https://doi.org/10.1109/TE.2024.3467912

[8] L. Desta, M. Afifuddin, and A. Adriyanto, “Challenges and Cybersecurity Threats in Digital Economic Transformation,” Int. J. Humanit. Educ. Soc. Sci. IJHESS, vol. 6, no. 2, 2023, doi: 10.55227/ijhess.v2i6.515. DOI: https://doi.org/10.55227/ijhess.v2i6.515

[9] B. Sarmah, B. Hall, R. Rao, and S. Patel, “HybridRAG: Integrating Knowledge Graphs and Vector Retrieval Augmented Generation for Efficient Information Extraction,” Comput. Lang. CsCL ArXiv240804948, vol. 8, 2024, doi: 10.48550/arXiv.2408.04948. DOI: https://doi.org/10.1145/3677052.3698671

[10] G. Byun, S. Lee, N. Choi, and J. D. Choi, “Secure Multifaceted-RAG: Hybrid Knowledge Retrieval with Security Filtering,” Information, vol. 16, p. 804, 2025, doi: 10.3390/info16090804. DOI: https://doi.org/10.3390/info16090804

[11] Y. Shang, Z. Ke, P. Lin, and Q. Ren, “Empowering knowledge graphs with hybrid retrieval-augmented generation for the intelligent mix scheme of mass concrete,” Case Stud. Constr. Mater., vol. 23, p. 4979, 2025, doi: 10.1016/j.cscm.2025.e04979. DOI: https://doi.org/10.1016/j.cscm.2025.e04979

[12] S. Kiran, “Hybrid Retrieval-Augmented Generation (RAG) Systems with Embedding Vector Databases,” Int. J. Sci. Res. Comput. Sci. Eng. Inf. Technol., vol. 11, no. 2, pp. 2694–2702, 2025, doi: 10.32628/CSEIT25112702. DOI: https://doi.org/10.32628/CSEIT25112702

[13] R. Kalra, Z. Wu, A. Gulley, and A. Hilliard, “HyPA-RAG: A Hybrid Parameter Adaptive Retrieval-Augmented Generation System for AI Legal and Policy Applications,” Proc. 1st Workshop Cust. NLP Prog. Chall. Cust. NLP Domain Appl. Group Individ. Cust., pp. 237–256, 2024, doi: 10.18653/v1/2024.customnlp4u-1.18. DOI: https://doi.org/10.18653/v1/2024.customnlp4u-1.18

[14] Z. Shahbazi, R. Jalali, and M. Maryam, “AI-Based Phishing Detection and Student Cybersecurity Awareness in the Digital Age,” Big Data Cogn Comput, vol. 9, no. 8, p. 210, 2025, doi: 10.3390/bdcc9080210. DOI: https://doi.org/10.3390/bdcc9080210

[15] M. M. Yamin, H. Ehtesham, U. Mohib, and B. Katt, “Applications of LLMs for Generating Cyber Security Exercise Scenarios,” IEEE Access, vol. 12, pp. 143806–143822, 2024, doi: 10.1109/ACCESS.2024.3468914. DOI: https://doi.org/10.1109/ACCESS.2024.3468914

[16] A. Zineddine, A. Abdeslam, M. Zaoui, and Y. Belfaik, “LLM-based assessment of HTTPS cybersecurity awareness: Dataset from moroccan web users and webmasters,” Data Brief, vol. 62, 2025, doi: 10.1016/j.dib.2025.112024. DOI: https://doi.org/10.1016/j.dib.2025.112024

[17] O. Cohen, G. A. Agmon, S. Asaf, and R. Puzis, “The Information Security Awareness of Large Language Models,” ArXiv Cryptogr. Secur., vol. 2411, 2024, doi: 10.48550/arXiv.2411.13207.

[18] J. Antony, M. Trovati, and S. Bolton, “Retrieval-Augmented Generation to Generate Knowledge Assets and Creation of Action Drivers,” Appl Sci, vol. 15, no. 11, p. 6247, 2025, doi: 10.3390/app15116247. DOI: https://doi.org/10.3390/app15116247

[19] J. P. Singh and S. Agrawal, “Automating Threat Intelligence Analysis with Retrieval Augmented Generation (RAG) for Enhanced Cybersecurity Posture,” Int. J. Sci. Res. IJSR, vol. 13, no. 5, pp. 251–255, 2024, doi: 10.21275/SR24502103758. DOI: https://doi.org/10.21275/SR24502103758

[20] Z. Chengshuai, G. Agrawal, F. Zhang, and K. Tharindu, “Ontology-Aware RAG for Improved Question-Answering in Cybersecurity Education,” ArXiv241214191 Comput. Sci., vol. 1, no. 2, 2024, doi: 10.48550/arXiv.2412.14191.

[21] F. Blefari, C. Cosentino, F. A. Pironti, and F. Angelo, “CyberRAG: An agentic RAG cyber attack classification and reporting tool,” Future Gener. Comput. Syst., vol. 76, p. 108186, 2026, doi: 10.1016/j.future.2025.108186. DOI: https://doi.org/10.1016/j.future.2025.108186

[22] S. Rajapaksha, R. Rani, and E. Karafili, “A RAG-Based Question-Answering Solution for Cyber-Attack Investigation and Attribution,” ArXiv Prepr. ArXiv240806272, no. Query date: 2024-09-02 23:48:01, 2024, [Online]. Available: https://arxiv.org/abs/2408.06272

[23] I. P. A. E. Pratama, I. M. O. Widyantara, L. Linawati, and N. Gunantara, “Bibliometric Analysis of AI-Based Prototype Proposal for User Security Awareness in Healthcare,” Int. J. Inform. Vis. JOIV, vol. 9, no. 3, 2025, doi: 10.62527/joiv.9.3.3319. DOI: https://doi.org/10.62527/joiv.9.3.3319

[24] I. P. A. E. Pratama, I. M. O. Widyantara, L. Linawati, N. Gunantara, S. Suakanto, and E. T. Irawan, “Technology-Enhanced Learning for User Security Awareness Using AI-based Naive RAG: A Design and Prototype,” 2025 Int. Conf. Adv. Data Sci. E-Learn. Inf. Syst. ICADEIS, vol. 2025, pp. 1–6, 2025, doi: 10.1109/ICADEIS65852.2025.10933283. DOI: https://doi.org/10.1109/ICADEIS65852.2025.10933283

[25] I. P. A. E. Pratama, I. M. O. Widyantara, L. Linawati, N. Gunantara, and E. T. Irawan, “A Proof of Concept for Fixed Chunking in an Aidriven Cybersecurity Awareness Prototype,” Int. Conf. Smart-Green Technol. Electr. Inf. Syst. ICSGTEIS, pp. 134–139, 2025, doi: 10.1109/ICSGTEIS68532.2025.11284323. DOI: https://doi.org/10.1109/ICSGTEIS68532.2025.11284323

[26] M. U. Hadi, Q. A. Tashi, and R. Qureshi, “A Survey on Large Language Models: Applications, Challenges, Limitations, and Practical Usage,” TechRxiv, vol. 10, 2023, doi: 10.36227/techrxiv.23589741.v1. DOI: https://doi.org/10.36227/techrxiv.23589741.v1

[27] P. Lewis, E. Perez, A. Piktus, and F. Petroni, “Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks,” Proceeding 34th Conf. Neural Inf. Process. Syst. NeurIPS 2020 Vanc. Can., 2020, doi: 10.48550/arXiv.2005.11401.

[28] Y. Gao, X. Yun, G. Xinyu, and K. Jia, “Retrieval-Augmented Generation for Large Language Models: A Survey,” ArXiv231210997 Comput. Lang., vol. 23, 2023, doi: 10.48550/arXiv.2312.10997.

[29] J. H. A. Alvaro and J. G. Barreda, “An advanced retrieval-augmented generation system for manufacturing quality control,” Adv. Eng. Inform., vol. 64, p. 103007, 2025, doi: 10.1016/j.aei.2024.103007. DOI: https://doi.org/10.1016/j.aei.2024.103007

[30] Y. Shuo, C. Mingyue, L. Qi, W. Daoyu, and J. Yang, “Multi-Source Knowledge Pruning for Retrieval-Augmented Generation: A Benchmark and Empirical Study,” Comput. Lang. CsCL ArXiv240913694, vol. 24, 2025, doi: 10.48550/arXiv.2409.13694.

[31] D. O. Opoku, M. Sheng, and Y. Zhang, “DO-RAG: A Domain-Specific QA Framework Using Knowledge Graph-Enhanced Retrieval-Augmented Generation,” Comput. Lang. CsCL ArXiv250517058, vol. 25, 2025, doi: 10.48550/arXiv.2505.17058. DOI: https://doi.org/10.36227/techrxiv.174837976.69904638/v1

[32] M. C. Lee, Q. Zhu, C. Mavromatis, Z. Han, and S. Adeshina, “HybGRAG: Hybrid Retrieval-Augmented Generation on Textual and Relational Knowledge Bases,” Proc. 63rd Annu. Meet. Assoc. Comput. Linguist., vol. 1, pp. 879–893, 2025, doi: 10.18653/v1/2025.acl-long.43. DOI: https://doi.org/10.18653/v1/2025.acl-long.43

[33] M. Murtiyoso, I. Tahyudin, and B. Berlilana, “A Systematic Review of Retrieval-Augmented Generation for Enhancing Domain-Specific Knowledge in Large Language Models,” Sink. J. Dan Penelit. Tek. Inform., vol. 9, no. 2, 2025, doi: 10.33395/sinkron.v9i2.14824. DOI: https://doi.org/10.33395/sinkron.v9i2.14824

[34] Y. Wan, Z. Chen, L. Ying, C. Chen, and M. Packianather, “Empowering LLMs by hybrid retrieval-augmented generation for domain-centric Q&A in smart manufacturing,” Adv. Eng. Inform., vol. 65, p. 103212, 2025, doi: 10.1016/j.aei.2025.103212. DOI: https://doi.org/10.1016/j.aei.2025.103212

[35] E. Z. Selviana, H. E. Wahanani, and E. P. Aditiawan, “Black Box Testing in Community Service Systems Using Boundary Value Analysis and Cause Effect Graph Methods (Case Study of Jombang District Community and Village Empowerment Service),” East Asian J. Multidiscip. Res., vol. 2, no. 10, pp. 4161–4184, 2023, doi: 10.55927/eajmr.v2i10.6435. DOI: https://doi.org/10.55927/eajmr.v2i10.6435

[36] T. Gupta and E. Kumar, “Answer Relevance Score (ARS): Novel Evaluation Metric for Question Answering System,” Int. Conf. Adv. Comput. Commun. Inf. Technol. ICAICCIT, pp. 292–296, 2023, doi: 10.1109/ICAICCIT60255.2023.10466080. DOI: https://doi.org/10.1109/ICAICCIT60255.2023.10466080

[37] A. Wang, K. Cho, and M. Lewis, “Asking and Answering Questions to Evaluate the Factual Consistency of Summaries,” Proc. 58th Annu. Meet. Assoc. Comput. Linguist., pp. 5008–5020, doi: 10.18653/v1/2020.acl-main.450. DOI: https://doi.org/10.18653/v1/2020.acl-main.450

[38] F. Nan, C. N. D. Santos, H. Zhu, P. Ng, and K. M. Keown, “Improving Factual Consistency of Abstractive Summarization via Question Answering,” Proc. 59th Annu. Meet. Assoc. Comput. Linguist. 11th Int. Jt. Conf. Nat. Lang. Process., vol. 1, pp. 6881–6894, 2021, doi: 10.18653/v1/2021.acl-long.536. DOI: https://doi.org/10.18653/v1/2021.acl-long.536

[39] A. Fabbri, C. S. Wu, W. Liu, and C. Xiong, “QAFactEval: Improved QA-Based Factual Consistency Evaluation for Summarization,” Proc. 2022 Conf. North Am. Chapter Assoc. Comput. Linguist. Hum. Lang. Technol., pp. 2587–2601, 2022, doi: 10.18653/v1/2022.naacl-main.187. DOI: https://doi.org/10.18653/v1/2022.naacl-main.187

[40] P. Yao and D. Barbosa, “Accurate and Nuanced Open-QA Evaluation Through Textual Entailment,” Find. Assoc. Comput. Linguist. ACL 2024, vol. 151, pp. 2575–2587, 2024, doi: 10.18653/v1/2024.findings-acl.151. DOI: https://doi.org/10.18653/v1/2024.findings-acl.151

[41] C. W. Huang and Y. N. Chen, “FactAlign: Long-form Factuality Alignment of Large Language Models,” Find. Assoc. Comput. Linguist. EMNLP 2024, vol. 955, pp. 16363–16375, 2024, doi: 10.18653/v1/2024.findings-emnlp.955. DOI: https://doi.org/10.18653/v1/2024.findings-emnlp.955

[42] T. Zheng, S. Shen, and C. Zeng, “A Retrieval-Augmented Generation Method for Question Answering on Airworthiness Regulations,” Electronics, vol. 14, no. 16, p. 3314, 2025, doi: 10.3390/electronics14163314. DOI: https://doi.org/10.3390/electronics14163314

[43] R. B. Permadi and K. Ramli, “Analysis of Measuring Information Security Awareness for Employees at Institution XYZ,” MALCOM Indones. J. Mach. Learn. Comput. Sci., vol. 4, no. 4, 2024, [Online]. Available: https://journal.irpi.or.id/index.php/malcom DOI: https://doi.org/10.57152/malcom.v4i4.1453

[44] D. T. J. Jun, A. S. Rafsanjani, S. Aslam, and M. Behjati, “Human Factors in Information Security: A Quantitative Study with Technical Solutions to Prevent Social Engineering Attacks,” Digit. Threats Res. Pract., vol. 6, no. 4, pp. 1–35, 2025, doi: 10.1145/37673. DOI: https://doi.org/10.1145/3767320