A Usability Evaluation of an E-Commerce Checkout System Using SUS and Usability Testing

Reynald Daffa Pahlevi; Harry Budi  Santoso

doi:10.59395/ijadis.v7i1.1498

Authors

Reynald Daffa Pahlevi Faculty of Computer Science, Universitas Indonesia, Indonesia
Harry Budi Santoso Faculty of Computer Science, Universitas Indonesia, Indonesia https://orcid.org/0000-0003-0459-0493

DOI:

https://doi.org/10.59395/ijadis.v7i1.1498

Keywords:

Usability Evaluation, System Usability Scale, Checkout System , User-Centered Design , E-Commerce

Abstract

This study focuses on evaluating and improving the usability of the XYZ Checkout System, which previously exhibited several issues that negatively affected transaction completion, including unclear information hierarchy, inconsistent interface behaviour, and inefficient checkout flows. To address these challenges, the research adopted a User-Centered Design (UCD) approach supported by a mixed-method evaluation strategy, combining the System Usability Scale (SUS), usability testing, and open-ended questions. A between-subject research design was employed, involving two independent groups of participants. The initial as-is evaluation was conducted with 35 users and produced an average SUS score of 69.57, indicating marginal usability and the presence of notable interaction barriers. Based on the findings, a high-fidelity to-be prototype was designed using established usability heuristics and UCD principles. The redesigned interface was subsequently evaluated by a different group of 35 users, resulting in a significantly improved average SUS score of 96.64, classified as Excellent with high acceptability. Overall, the results demonstrate that the proposed design improvements successfully enhanced interface consistency, clarity of interaction, and checkout efficiency, emphasizing the critical role of systematic usability evaluation in optimizing e-commerce checkout systems.

1708 878

Downloads

Download data is not yet available.

References

[1] Kedah, Z. (2023). Use of e-commerce in the world of business. Startupreneur Business Digital (SABDA Journal), 2(1), 5160. DOI: https://doi.org/10.33050/sabda.v2i1.273

[2] Kementerian Perdagangan. (2024). Perdagangan Digital (E-Commerce) Indonesia Periode 2023. https://satudata.kemendag.go.id/ringkasan/produk/perdagangandigital-e-commerce-indonesia-periode-2023

[3] Andini, E. (2023). Survei: Fee Marketplace. SleekFlow

[4] Nielsen, J. (1999, February 6). Why people shop on the web. Nielsen Norman Group.

[5] Audrey, Y. A., Wiguna, A. S., Ramadhani, S. A., & Gui, A. (2022). Factors influencing online shopping intention in Indonesia. 2022 International Conference on Information Management and Technology (ICIMTech), 357362. DOI: https://doi.org/10.1109/ICIMTech55957.2022.9915081

[6] Yahya, S., & Sugiyanto, C. (2020). Indonesian demand for online shopping. Journal of Indonesian Economy and Business, 35(2), 188203. DOI: https://doi.org/10.22146/jieb.55358

[7] Huang, G., Korfiatis, N., & Chang, C. (2018). Mobile shopping cart abandonment: The roles of conflicts, ambivalence, and hesitation. Journal of Business Research, 85, 165174. DOI: https://doi.org/10.1016/j.jbusres.2017.12.008

[8] Wang, S., Ye, Y., Ning, B., Cheah, J., & Lim, X. (2022). Why Do Some Consumers Still Prefer In-Store Shopping? An Exploration of Online Shopping Cart Abandonment Behavior. Frontiers in Psychology, 12 DOI: https://doi.org/10.3389/fpsyg.2021.829696

[9] Marchewka, J. T. (2015). Information Technology Project Management: Providing Measurable Organizational Value (5th ed.). Wiley Publishing.

[10] Conyer, M. (1995). User and Usability Testing--How It Should Be Undertaken? Australasian Journal of Educational Technology, 11, 3851. DOI: https://doi.org/10.14742/ajet.2075

[11] Maulana, H. T., & Rosyidah, U. (2022). User Experience Model using Concise User-Centered Design in Small and Medium Enterprise E-Commerce. 2022 International Seminar on Application for Technology of Information and Communication: Technology 4.0 for Smart Ecosystem: A New Way of Doing Digital Business, ISemantic 2022, 377382. https://doi.org/10.1109/iSemantic55962.2022.9920381 DOI: https://doi.org/10.1109/iSemantic55962.2022.9920381

[12] Rozanah, & Santoso, H. B. (2024). Usability Evaluation and Alternative Design Recommendations for HSIS Mobile Application: A Case Study of PT HNI. The Indonesian Journal of Computer Science, 13(4). https://doi.org/10.33022/ijcs.v13i4.4260 DOI: https://doi.org/10.33022/ijcs.v13i4.4260

[13] Hasan, T. I., Silalahi, C. I., Rumagit, R. Y., & Pratama, G. D. (2024). UI/UX Design Impact on E-Commerce Attracting Users. Procedia Computer Science, 245(C), 10751082. https://doi.org/10.1016/j.procs.2024.10.336 DOI: https://doi.org/10.1016/j.procs.2024.10.336

[14] Interaction Design Foundation. (2025). User Centered Design (UCD).

[15] Interaction Design Foundation. (2024). Heuristic Evaluation (HE).

[16] Sharfina, Z., & Santoso, H. B. (2016). An Indonesian Adaptation of the System Usability Scale (SUS) DOI: https://doi.org/10.1109/ICACSIS.2016.7872776

[17] Hanifa, M. R., Santoso, H. B., & Kasiyah. (2019). 2019 International Conference on Advanced Computer Science and Information Systems (ICACSIS): Meli Bali Hotel, Bali, Indonesia, October 12th-13th, 2019. IEEE.

[18] Uma, S., & Roger, B. (2016). Research methods for business: A skill-building approach (7th ed.). John Wiley & Sons

[19] Moser, A., & Korstjens, I. (2018). Series: Practical guidance to qualitative research. Part 3: Sampling, data collection and analysis. In European Journal of General Practice (Vol. 24, Issue 1, pp. 918). Taylor and Francis Ltd. https://doi.org/10.1080/13814788.2017.1375091 DOI: https://doi.org/10.1080/13814788.2017.1375091

[20] Elliott, V. (2018). Thinking about the coding process in qualitative data analysis. Qualitative Report, 23(11), 28502861. https://doi.org/10.46743/2160- 3715/2018.3560 DOI: https://doi.org/10.46743/2160-3715/2018.3560

[21] Hyzy, M., Bond, R., Mulvenna, M., Bai, L., Dix, A., Leigh, S., & Hunt, S. (2022). System Usability Scale Benchmarking for Digital Health Apps: Meta-analysis. JMIR MHealth and UHealth, 10(8). https://doi.org/10.2196/37290 DOI: https://doi.org/10.2196/37290

[22] Bangor, A., Kortum, P. T., & Miller, J. T. (2008). An empirical evaluation of the system usability scale. International Journal of Human-Computer Interaction, 24(6), 574 594. https://doi.org/10.1080/10447310802205776 DOI: https://doi.org/10.1080/10447310802205776

[23] Sauro, J., & Lewis, J. R. (2016). Chapter 8 - Standardized usability questionnaires DOI: https://doi.org/10.1016/B978-0-12-802308-2.00008-4

[24] Gunawan, R., Anthony, G., Vendly, & Anggreainy, M. S. (2021). The Effect of Design User Interface (UI) E-Commerce on User Experience (UX). Proceedings of 2021 6th International Conference on New Media Studies, CONMEDIA 2021, 9598. https://doi.org/10.1109/CONMEDIA53104.2021.9617199

[25] Dzuhriyan, A., Permana, S., & Gufron, M. (2025). Consumer Legal Protection in Online Transactions: Challenges and Opportunities in Indonesia's Digital Economy. Justice Voice. https://doi.org/10.37893/jv.v3i1.1017. DOI: https://doi.org/10.37893/jv.v3i1.1017

[26] Safrida, S. (2025). Dynamics of Consumer Protection Law in the Digital Era: Case Study on E-commerce in Indonesia. International Journal of Educational Research & Social Sciences. https://doi.org/10.51601/ijersc.v6i1.957. DOI: https://doi.org/10.51601/ijersc.v6i1.957

[27] Erlangga, H., Setiawati, T., Riadi, F., Hindarsah, I., & Riani, D. (2024). Consumer Behavior in The Digital Age: A Qualitative Analysis of Online Shopping Patterns in Indonesia. Majalah Ilmiah Bijak. https://doi.org/10.31334/bijak.v21i2.4334. DOI: https://doi.org/10.31334/bijak.v21i2.4334

[28] Arum, M., Lishobrina, L., Purbaya, M., Iffah, T., & Cristianti, O. (2023). Customer Shopping Behavior on E-Commerce Effectiveness Using the Information System Success Theory Approach. Telaah Bisnis. https://doi.org/10.35917/tb.v24i2.439. DOI: https://doi.org/10.35917/tb.v24i2.439

[29] Assegaff, S., Jasmir, J., Abidin, D., & Sharipuddin, S. (2017). User Behavior in Using Mobile Commerce (Scale Development: Perspective of Trust and Risk). , 3, 320-325. https://doi.org/10.25077/teknosi.v3i3.2017.320-325. DOI: https://doi.org/10.25077/TEKNOSI.v3i3.2017.320-325

[30] Azhar, T., Santoso, H., & Putra, P. (2022). Evaluation of Usability and User Experience of Shopee as One of the Top E-Marketplaces in Indonesia. 2022 10th International Conference on Information and Communication Technology (ICoICT), 305-309. https://doi.org/10.1109/icoict55009.2022.9914895 DOI: https://doi.org/10.1109/ICoICT55009.2022.9914895