Multi-Task Learning for Traffic Sign Recognition using Multi-Scale Convolutional Neural Networks

Mutaqin Akbar; Indah Susilawati; Budi Sulistiyo Jati; Nur Alamsyah

doi:10.59395/ijadis.v6i2.1406

Authors

Mutaqin Akbar Universitas Mercu Buana Yogyakarta
Indah Susilawati Universitas Mercu Buana Yogyakarta
Budi Sulistiyo Jati Universitas Mercu Buana Yogyakarta
Nur Alamsyah Universitas Informatika dan Bisnis Indonesia

DOI:

https://doi.org/10.59395/ijadis.v6i2.1406

Keywords:

Convolutional neural network, Multi-scale, Multi-task, Recognition, Traffic sign

Abstract

Traffic signs are an essential component of road infrastructure. According to the Department of Transportation, Indonesia has over 300 distinct traffic signs, categorized based on their functions and purposes. TSR systems have been widely integrated into various intelligent transportation technologies, such as Driver Assistance Systems (DAS), Advanced Driver Assistance Systems (ADAS), and Autonomous Driving Systems (ADS). The output generated by TSR serves as a critical input for DAS, ADAS, ADS, and other intelligent systems. This article presents a CNN-based classification for traffic sign recognition using multi-task learning (MTL), focusing on traffic signs in Indonesia. The dataset was collected from direct capture with the help of a cellphone camera, indirect capture by utilizing screenshots on a digital map application, and they are captured from several different angles, during the day and at night. The proposed CNN architecture incorporates multi-scale within an MTL framework. The use of a multi-scale approach will hopefully enhance the model’s ability to recognize traffic signs in varied and complex environments. And the integration of MTL will enable the model to handle multiple related tasks concurrently, sharing learned features across tasks. During the training stage, the MS-CNN outperformed a standard CNN model by demonstrating lower initial loss, higher starting accuracy, and achieving 100% accuracy by the 8th epoch with a minimal error rate of just 0.003. In the testing stage, the model achieved exceptional results, as shown by the confusion matrix, it successfully classified all traffic sign types (10 classes) and accurately categorized each sign into one of two categories—warning or prohibition. All performance metrics, including precision, recall, and F1-score, reached 100% for both output tasks, confirming the robustness and reliability of the model.

1102 639

Downloads

Download data is not yet available.

References

[1] K. Bengler, K. Dietmayer, B. Farber, M. Maurer, C. Stiller, and H. Winner, Three Decades of Driver Assistance Systems: Review and Future Perspectives, IEEE Intell. Transport. Syst. Mag., vol. 6, no. 4, pp. 622, 2014, doi: 10.1109/MITS.2014.2336271. https://doi.org/10.1109/MITS.2014.2336271 DOI: https://doi.org/10.1109/MITS.2014.2336271

[2] B. Reimer, Driver Assistance Systems and the Transition to Automated Vehicles: A Path to Increase Older Adult Safety and Mobility?, Publ. Pol. Aging. Rep., vol. 24, no. 1, pp. 2731, Mar. 2014, doi: 10.1093/ppar/prt006. https://doi.org/10.1093/ppar/prt006 DOI: https://doi.org/10.1093/ppar/prt006

[3] F. Jimnez, J. E. Naranjo, J. J. Anaya, F. Garca, A. Ponz, and J. M. Armingol, Advanced Driver Assistance System for Road Environments to Improve Safety and Efficiency, Transportation Research Procedia, vol. 14, pp. 22452254, 2016, doi: 10.1016/j.trpro.2016.05.240. https://doi.org/10.1016/j.trpro.2016.05.240 DOI: https://doi.org/10.1016/j.trpro.2016.05.240

[4] A. Ziebinski, R. Cupek, D. Grzechca, and L. Chruszczyk, Review of advanced driver assistance systems (ADAS), presented at the PROCEEDINGS OF THE INTERNATIONAL CONFERENCE OF COMPUTATIONAL METHODS IN SCIENCES AND ENGINEERING 2017 (ICCMSE-2017), Thessaloniki, Greece, 2017, p. 120002. doi: 10.1063/1.5012394. https://doi.org/10.1063/1.5012394 DOI: https://doi.org/10.1063/1.5012394

[5] V. K. Kukkala, J. Tunnell, S. Pasricha, and T. Bradley, Advanced Driver-Assistance Systems: A Path Toward Autonomous Vehicles, IEEE Consumer Electron. Mag., vol. 7, no. 5, pp. 1825, Sep. 2018, doi: 10.1109/MCE.2018.2828440. https://doi.org/10.1109/MCE.2018.2828440 DOI: https://doi.org/10.1109/MCE.2018.2828440

[6] G. De-Las-Heras, J. Snchez-Soriano, and E. Puertas, Advanced Driver Assistance Systems (ADAS) Based on Machine Learning Techniques for the Detection and Transcription of Variable Message Signs on Roads, Sensors, vol. 21, no. 17, p. 5866, Aug. 2021, doi: 10.3390/s21175866. https://doi.org/10.3390/s21175866 DOI: https://doi.org/10.3390/s21175866

[7] J. Levinson et al., Towards fully autonomous driving: Systems and algorithms, in 2011 IEEE Intelligent Vehicles Symposium (IV), Baden-Baden, Germany: IEEE, Jun. 2011, pp. 163168. doi: 10.1109/IVS.2011.5940562. https://doi.org/10.1109/IVS.2011.5940562 DOI: https://doi.org/10.1109/IVS.2011.5940562

[8] K. Othman, Exploring the implications of autonomous vehicles: a comprehensive review, Innov. Infrastruct. Solut., vol. 7, no. 2, p. 165, Apr. 2022, doi: 10.1007/s41062-022-00763-6. https://doi.org/10.1007/s41062-022-00763-6 DOI: https://doi.org/10.1007/s41062-022-00763-6

[9] J. Zhao et al., Autonomous driving system: A comprehensive survey, Expert Systems with Applications, vol. 242, p. 122836, May 2024, doi: 10.1016/j.eswa.2023.122836. https://doi.org/10.1016/j.eswa.2023.122836 DOI: https://doi.org/10.1016/j.eswa.2023.122836

[10] B. Padmaja, Ch. V. K. N. S. N. Moorthy, N. Venkateswarulu, and M. M. Bala, Exploration of issues, challenges and latest developments in autonomous cars, J Big Data, vol. 10, no. 1, p. 61, May 2023, doi: 10.1186/s40537-023-00701-y. https://doi.org/10.1186/s40537-023-00701-y DOI: https://doi.org/10.1186/s40537-023-00701-y

[11] D. Garikapati and S. S. Shetiya, Autonomous Vehicles: Evolution of Artificial Intelligence and the Current Industry Landscape, BDCC, vol. 8, no. 4, p. 42, Apr. 2024, doi: 10.3390/bdcc8040042. https://doi.org/10.3390/bdcc8040042 DOI: https://doi.org/10.3390/bdcc8040042

[12] O. R. Sitanggang, H. Fitriyah, and F. Utaminingrum, Sistem Deteksi dan Pengenalan Jenis Rambu Lalu Lintas Menggunakan Metode Shape Detection Pada Raspberry Pi, JPTIIK, vol. 2, no. 12, pp. 61086117, Dec. 2018.

[13] T. Oddy Chrisdwianto, H. Fitriyah, and E. Rosana Widasari, Perancangan Sistem Deteksi dan Pengenalan Rambu Peringatan Menggunakan Metode Template Matching, JPTIIK, vol. 2, no. 3, pp. 12651274, Mar. 2018.

[14] G. Romadhon and Murinto, Aplikasi pengenalan citra rambu lalu lintas berbentuk lingkaran menggunakan metode jarak city-block, JSTIE, vol. 2, no. 2, pp. 286294, Jun. 2014.

[15] C. Rahmad, I. Fauziah Rahmah, and R. Andrie Asmara, Deteksi dan pengenalan rambu lalu lintas di indonesia menggunakan RGBN dan Gabor, in SENTRINOV, 2017, pp. TI13-22.

[16] Y. LeCun et al., Backpropagation Applied to Handwritten Zip Code Recognition, Neural Computation, vol. 1, no. 4, pp. 541551, Dec. 1989, doi: 10.1162/neco.1989.1.4.541. https://doi.org/10.1162/neco.1989.1.4.541 DOI: https://doi.org/10.1162/neco.1989.1.4.541

[17] Y. LeCun, P. Haffner, L. Bottou, and Y. Bengio, Object Recognition with Gradient-Based Learning, in Shape, Contour and Grouping in Computer Vision, vol. 1681, in Lecture Notes in Computer Science, vol. 1681. , Berlin, Heidelberg: Springer Berlin Heidelberg, 1999, pp. 319345. doi: 10.1007/3-540-46805-6_19. https://doi.org/10.1007/3-540-46805-6_19 DOI: https://doi.org/10.1007/3-540-46805-6_19

[18] Y. LeCun, Y. Bengio, and G. Hinton, Deep learning, Nature, vol. 521, no. 7553, pp. 436444, May 2015, doi: 10.1038/nature14539.

https://doi.org/10.1038/nature14539 DOI: https://doi.org/10.1038/nature14539

[19] H. H. Aghdam and E. J. Heravi, Guide to Convolutional Neural Networks: A Practical Application to Traffic-Sign Detection and Classification, Softcover reprint of the original 1st edition 2017. Cham: Springer International Publishing, 2017.

https://doi.org/10.1007/978-3-319-57550-6_1 DOI: https://doi.org/10.1007/978-3-319-57550-6_1

[20] R. Venkatesan and B. Li, Convolutional neural networks in visual computing: a concise guide. Boca Raton, FL: CRC Press, 2018.

https://doi.org/10.4324/9781315154282 DOI: https://doi.org/10.4324/9781315154282

[21] V. E. Balas, R. Kumar, and R. Srivastava, Eds., Recent trends and advances in artificial intelligence and Internet of Things. in Intelligent systems reference library, no. 172. Cham: Springer, 2020. https://doi.org/10.1007/978-3-030-32644-9 DOI: https://doi.org/10.1007/978-3-030-32644-9

[22] Y. Zhang, H. G. Soon, D. Ye, J. Y. H. Fuh, and K. Zhu, Powder-Bed Fusion Process Monitoring by Machine Vision With Hybrid Convolutional Neural Networks, IEEE Trans. Ind. Inf., vol. 16, no. 9, pp. 57695779, Sep. 2020, doi: 10.1109/TII.2019.2956078.

https://doi.org/10.1109/TII.2019.2956078 DOI: https://doi.org/10.1109/TII.2019.2956078

[23] N. I. Chervyakov, P. A. Lyakhov, M. A. Deryabin, N. N. Nagornov, M. V. Valueva, and G. V. Valuev, Residue Number System-Based Solution for Reducing the Hardware Cost of a Convolutional Neural Network, Neurocomputing, vol. 407, pp. 439453, Sep. 2020, doi: 10.1016/j.neucom.2020.04.018. https://doi.org/10.1016/j.neucom.2020.04.018 DOI: https://doi.org/10.1016/j.neucom.2020.04.018

[24] M. Akbar, Traffic sign recognition using convolutional neural networks, J. Teknol. dan Sist. Komput, vol. 9, no. 2, pp. 120125, Apr. 2021, doi: 10.14710/jtsiskom.2021.13959. https://doi.org/10.14710/jtsiskom.2021.13959 DOI: https://doi.org/10.14710/jtsiskom.2021.13959

[25] M. Akbar, Pengenalan Rambu Lalu-lintas menggunakan Convolutional Neural Network (Studi Kasus: Rambu Lalu-lintas Indonesia), InfoTekJar, vol. 6, no. 2, pp. 272276, Mar. 2022, doi: http://dx.doi.org/10.30743/infotekjar.v6i2.4564.

[26] P. Sermanet and Y. LeCun, Traffic sign recognition with multi-scale Convolutional Networks, in The 2011 International Joint Conference on Neural Networks, San Jose, CA, USA: IEEE, Jul. 2011, pp. 28092813. doi: 10.1109/IJCNN.2011.6033589. https://doi.org/10.1109/IJCNN.2011.6033589 DOI: https://doi.org/10.1109/IJCNN.2011.6033589

[27] M. Akbar, A. S. Purnomo, and S. Supatman, Multi-Scale Convolutional Networks untuk Pengenalan Rambu Lalu Lintas di Indonesia, SISFOKOM, vol. 11, no. 3, pp. 310315, Dec. 2022, doi: 10.32736/sisfokom.v11i3.1452. https://doi.org/10.32736/sisfokom.v11i3.1452 DOI: https://doi.org/10.32736/sisfokom.v11i3.1452

[28] A. Krizhevsky, I. Sutskever, and G. E. Hinton, ImageNet classification with deep convolutional neural networks, Commun. ACM, vol. 60, no. 6, pp. 8490, May 2017, doi: 10.1145/3065386. https://doi.org/10.1145/3065386 DOI: https://doi.org/10.1145/3065386

[29] M. D. Zeiler and R. Fergus, Visualizing and Understanding Convolutional Networks, 2013, arXiv. doi: 10.48550/ARXIV.1311.2901.

https://doi.org/10.1007/978-3-319-10590-1_53 DOI: https://doi.org/10.1007/978-3-319-10590-1_53

[30] C. Szegedy et al., Going Deeper with Convolutions, 2014, arXiv. doi: 10.48550/ARXIV.1409.4842.

[31] K. Simonyan and A. Zisserman, Very Deep Convolutional Networks for Large-Scale Image Recognition, 2014, arXiv. doi: 10.48550/ARXIV.1409.1556.

[32] K. He, X. Zhang, S. Ren, and J. Sun, Deep Residual Learning for Image Recognition, 2015, arXiv. doi: 10.48550/ARXIV.1512.03385.

https://doi.org/10.1109/CVPR.2016.90 DOI: https://doi.org/10.1109/CVPR.2016.90

[33] C. Szegedy, S. Ioffe, V. Vanhoucke, and A. Alemi, Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning, AAAI, vol. 31, no. 1, Feb. 2017, doi: 10.1609/aaai.v31i1.11231. https://doi.org/10.1609/aaai.v31i1.11231 DOI: https://doi.org/10.1609/aaai.v31i1.11231

[34] G. Huang, Z. Liu, L. van der Maaten, and K. Q. Weinberger, Densely Connected Convolutional Networks, 2016, arXiv. doi: 10.48550/ARXIV.1608.06993. https://doi.org/10.1109/CVPR.2017.243 DOI: https://doi.org/10.1109/CVPR.2017.243

[35] S. Ramaneswaran, K. Srinivasan, P. M. D. R. Vincent, and C.-Y. Chang, Hybrid Inception v3 XGBoost Model for Acute Lymphoblastic Leukemia Classification, Computational and Mathematical Methods in Medicine, vol. 2021, pp. 110, Jul. 2021, doi: 10.1155/2021/2577375.

https://doi.org/10.1155/2021/2577375 DOI: https://doi.org/10.1155/2021/2577375

[36] S. Ruder, An Overview of Multi-Task Learning in Deep Neural Networks, Jun. 15, 2017, arXiv: arXiv:1706.05098. Accessed: Feb. 13, 2024. [Online]. Available: http://arxiv.org/abs/1706.05098

[37] Z. Zhang, P. Luo, C. C. Loy, and X. Tang, Facial Landmark Detection by Deep Multi-task Learning, in Computer Vision ECCV 2014, vol. 8694, D. Fleet, T. Pajdla, B. Schiele, and T. Tuytelaars, Eds., in Lecture Notes in Computer Science, vol. 8694. , Cham: Springer International Publishing, 2014, pp. 94108. doi: 10.1007/978-3-319-10599-4_7. https://doi.org/10.1007/978-3-319-10599-4_7 DOI: https://doi.org/10.1007/978-3-319-10599-4_7

[38] X. Yin and X. Liu, Multi-Task Convolutional Neural Network for Pose-Invariant Face Recognition, IEEE Trans. on Image Process., vol. 27, no. 2, pp. 964975, Feb. 2018, doi: 10.1109/TIP.2017.2765830. https://doi.org/10.1109/TIP.2017.2765830 DOI: https://doi.org/10.1109/TIP.2017.2765830

[39] Z. Wang et al., A Multi-Task Convolutional Neural Network for Lesion Region Segmentation and Classification of Non-Small Cell Lung Carcinoma, Diagnostics, vol. 12, no. 8, p. 1849, Jul. 2022, doi: 10.3390/diagnostics12081849. https://doi.org/10.3390/diagnostics12081849 DOI: https://doi.org/10.3390/diagnostics12081849

[40] S. Starke et al., Multitask Learning with Convolutional Neural Networks and Vision Transformers Can Improve Outcome Prediction for Head and Neck Cancer Patients, Cancers, vol. 15, no. 19, p. 4897, Oct. 2023, doi: 10.3390/cancers15194897. https://doi.org/10.3390/cancers15194897 DOI: https://doi.org/10.3390/cancers15194897

[41] B. Oliveira et al., A multi-task convolutional neural network for classification and segmentation of chronic venous disorders, Sci Rep, vol. 13, no. 1, p. 761, Jan. 2023, doi: 10.1038/s41598-022-27089-8. https://doi.org/10.1038/s41598-022-27089-8 DOI: https://doi.org/10.1038/s41598-022-27089-8

[42] D. Avianto, A. Harjoko, and Afiahayati, CNN-Based Classification for Highly Similar Vehicle Model Using Multi-Task Learning, J. Imaging, vol. 8, no. 11, p. 293, Oct. 2022, doi: 10.3390/jimaging8110293. https://doi.org/10.3390/jimaging8110293 DOI: https://doi.org/10.3390/jimaging8110293

[43] Y. H. Kim and K. R. Park, MTS-CNN: Multi-task semantic segmentation-convolutional neural network for detecting crops and weeds, Computers and Electronics in Agriculture, vol. 199, p. 107146, Aug. 2022, doi: 10.1016/j.compag.2022.107146. https://doi.org/10.1016/j.compag.2022.107146 DOI: https://doi.org/10.1016/j.compag.2022.107146

[44] S. W. Smith, The scientist and engineers guide to digital signal processing. San Diego, Calif.: California Technical Pub., 1999.

[45] D. Hutchison et al., Evaluation of Pooling Operations in Convolutional Architectures for Object Recognition, in Artificial Neural Networks ICANN 2010, vol. 6354, K. Diamantaras, W. Duch, and L. S. Iliadis, Eds., in Lecture Notes in Computer Science, vol. 6354. , Berlin, Heidelberg: Springer Berlin Heidelberg, 2010, pp. 92101. doi: 10.1007/978-3-642-15825-4_10. https://doi.org/10.1007/978-3-642-15825-4_10 DOI: https://doi.org/10.1007/978-3-642-15825-4_10

[46] A. Garcia-Garcia, S. Orts-Escolano, S. Oprea, V. Villena-Martinez, P. Martinez-Gonzalez, and J. Garcia-Rodriguez, A survey on deep learning techniques for image and video semantic segmentation, Applied Soft Computing, vol. 70, pp. 4165, Sep. 2018, doi: 10.1016/j.asoc.2018.05.018. https://doi.org/10.1016/j.asoc.2018.05.018 DOI: https://doi.org/10.1016/j.asoc.2018.05.018

[47] Y. Chen, L. Li, W. Li, Q. Guo, Z. Du, and Z. Xu, Fundamentals of neural networks, in AI Computing Systems, Elsevier, 2024, pp. 1751. doi: 10.1016/B978-0-32-395399-3.00008-1. https://doi.org/10.1016/B978-0-32-395399-3.00008-1 DOI: https://doi.org/10.1016/B978-0-32-395399-3.00008-1

[48] A. Kulkarni, D. Chong, and F. A. Batarseh, Foundations of data imbalance and solutions for a data democracy, in Data Democracy, Elsevier, 2020, pp. 83106. doi: 10.1016/B978-0-12-818366-3.00005-8. https://doi.org/10.1016/B978-0-12-818366-3.00005-8 DOI: https://doi.org/10.1016/B978-0-12-818366-3.00005-8

[49] P. Singh, N. Singh, K. K. Singh, and A. Singh, Diagnosing of disease using machine learning, in Machine Learning and the Internet of Medical Things in Healthcare, Elsevier, 2021, pp. 89111. doi: 10.1016/B978-0-12-821229-5.00003-3. https://doi.org/10.1016/B978-0-12-821229-5.00003-3 DOI: https://doi.org/10.1016/B978-0-12-821229-5.00003-3

[50] A. A. Lawan, N. Cavus, R. Yunusa, U. I. Abdulrazak, and S. Tahir, Fundamentals of machine-learning modeling for behavioral screening and diagnosis of autism spectrum disorder, in Neural Engineering Techniques for Autism Spectrum Disorder, Volume 2, Elsevier, 2023, pp. 253268. doi: 10.1016/B978-0-12-824421-0.00020-5. https://doi.org/10.1016/B978-0-12-824421-0.00020-5 DOI: https://doi.org/10.1016/B978-0-12-824421-0.00020-5

[51] S. Sathyanarayanan, Confusion Matrix-Based Performance Evaluation Metrics, AJBR, pp. 40234031, Nov. 2024, doi: 10.53555/AJBR.v27i4S.4345. https://doi.org/10.53555/AJBR.v27i4S.4345 DOI: https://doi.org/10.53555/AJBR.v27i4S.4345