Generalization Analysis of YAMNet-DNN Architectures in Deepfake Audio Classification
DOI:
https://doi.org/10.59395/ijadis.v7i2.1567Keywords:
Deepfake Audio, YAMNet Embeddings, Deep Neural Networks, Domain Generalization, Fine-Tuning StrategiesAbstract
The rapid advancement of speech synthesis and voice conversion technologies has increased the risk of audio deepfake attacks, necessitating robust and generalizable detection systems. This study proposes a deepfake audio detection framework that leverages pretrained YAMNet embeddings as a feature extractor, combined with statistical aggregation (Mean, Standard Deviation, and Mean + Standard Deviation) and Deep Neural Network classifiers to form compact utterance-level representations without temporal modeling. Experiments on a public dataset show that Mean and Mean + Standard Deviation representations achieve classification accuracy of up to 99.24%, outperforming Standard Deviation–based features, while deeper architectures provide only marginal performance gains. However, cross-dataset evaluation on a newly collected primary dataset reveals a significant generalization gap, with accuracy dropping to approximately 62% under direct testing. To address this issue, fine-tuning strategies with and without layer freezing are explored under limited target-domain data conditions, where the best-performing model achieves an accuracy of 0.93 on the primary dataset and 0.91 on the secondary dataset, with macro-averaged precision, recall, and F1-score reaching 0.93 and 0.91, respectively. Further evaluation on external datasets demonstrates that the model benefits from fine-tuning even without prior exposure to these data sources, indicating its ability to generalize across different synthesis conditions. These findings confirm that controlled fine-tuning effectively balances adaptation and generalization, while also highlighting the potential of incremental learning strategies to maintain robustness against evolving deepfake generation techniques.
Downloads
References
[1] A. van den Oord et al., “WaveNet: A Generative Model for Raw Audio,” Sep. 19, 2016, arXiv: arXiv:1609.03499. doi: 10.48550/arXiv.1609.03499.
[2] Y. Wang et al., “Tacotron: Towards End-to-End Speech Synthesis,” in Interspeech 2017, ISCA, Aug. 2017, pp. 4006–4010. doi: 10.21437/Interspeech.2017-1452. DOI: https://doi.org/10.21437/Interspeech.2017-1452
[3] Y. Ren et al., “FastSpeech: Fast, Robust and Controllable Text to Speech”.
[4] Y. Jia, Y. Zhang, R. Weiss, Q. Wang, J. Shen, and F. Ren, “Transfer Learning from Speaker Verification to Multispeaker Text-To-Speech Synthesis”.
[5] T. Kaneko and H. Kameoka, “CycleGAN-VC: Non-parallel Voice Conversion Using Cycle-Consistent Adversarial Networks,” in 2018 26th European Signal Processing Conference (EUSIPCO), Rome: IEEE, Sep. 2018, pp. 2100–2104. doi: 10.23919/EUSIPCO.2018.8553236. DOI: https://doi.org/10.23919/EUSIPCO.2018.8553236
[6] H. Kameoka, T. Kaneko, K. Tanaka, and N. Hojo, “StarGAN-VC: non-parallel many-to-many Voice Conversion Using Star Generative Adversarial Networks,” in 2018 IEEE Spoken Language Technology Workshop (SLT), Athens, Greece: IEEE, Dec. 2018, pp. 266–273. doi: 10.1109/SLT.2018.8639535. DOI: https://doi.org/10.1109/SLT.2018.8639535
[7] S. Liu, Y. Cao, D. Su, and H. Meng, “DiffSVC: A Diffusion Probabilistic Model for Singing Voice Conversion,” in 2021 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), Cartagena, Colombia: IEEE, Dec. 2021, pp. 741–748. doi: 10.1109/ASRU51503.2021.9688219. DOI: https://doi.org/10.1109/ASRU51503.2021.9688219
[8] C. Wang et al., “Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers,” Jan. 05, 2023, arXiv: arXiv:2301.02111. doi: 10.48550/arXiv.2301.02111.
[9] R. Mubarak, T. Alsboui, O. Alshaikh, I. Inuwa-Dutse, S. Khan, and S. Parkinson, “A Survey on the Detection and Impacts of Deepfakes in Visual, Audio, and Textual Formats,” IEEE Access, vol. 11, pp. 144497–144529, 2023, doi: 10.1109/ACCESS.2023.3344653. DOI: https://doi.org/10.1109/ACCESS.2023.3344653
[10] Z. Almutairi and H. Elgibreen, “A Review of Modern Audio Deepfake Detection Methods: Challenges and Future Directions,” Algorithms, vol. 15, no. 5, p. 155, May 2022, doi: 10.3390/a15050155. DOI: https://doi.org/10.3390/a15050155
[11] B. Zhang, H. Cui, V. Nguyen, and M. Whitty, “Audio Deepfake Detection: What Has Been Achieved and What Lies Ahead,” Sensors, vol. 25, no. 7, p. 1989, Mar. 2025, doi: 10.3390/s25071989. DOI: https://doi.org/10.3390/s25071989
[12] F. Romero-Moreno, “Deepfake detection in generative AI: A legal framework proposal to protect human rights,” Comput. Law Secur. Rev., vol. 58, p. 106162, Sep. 2025, doi: 10.1016/j.clsr.2025.106162. DOI: https://doi.org/10.1016/j.clsr.2025.106162
[13] O. A. Shaaban, R. Yildirim, and A. A. Alguttar, “Audio Deepfake Approaches,” IEEE Access, vol. 11, pp. 132652–132682, 2023, doi: 10.1109/ACCESS.2023.3333866. DOI: https://doi.org/10.1109/ACCESS.2023.3333866
[14] S. Kopecky, “Challenges of Deepfakes,” in Intelligent Computing, vol. 1016, K. Arai, Ed., in Lecture Notes in Networks and Systems, vol. 1016. , Cham: Springer Nature Switzerland, 2024, pp. 158–166. doi: 10.1007/978-3-031-62281-6_11. DOI: https://doi.org/10.1007/978-3-031-62281-6_11
[15] Dr. B. Dhiman, “Exploding AI-Generated Deepfakes and Misinformation: A Threat to Global Concern in the 21st Century,” Qeios, Dec. 2023, doi: 10.32388/DPLE2L. DOI: https://doi.org/10.32388/DPLE2L
[16] A. Hery, O. Joseph, O. Femi, and H. Luz, “Audio deepfakes: threats to voice assistants and voice-activated systems”.
[17] A. O. Vaidya, M. Dangore, V. K. Borate, N. Raut, Y. K. Mali, and A. Chaudhari, “Deep Fake Detection for Preventing Audio and Video Frauds Using Advanced Deep Learning Techniques,” in 2024 IEEE Recent Advances in Intelligent Computational Systems (RAICS), Kothamangalam, Kerala, India: IEEE, May 2024, pp. 1–6. doi: 10.1109/RAICS61201.2024.10689785. DOI: https://doi.org/10.1109/RAICS61201.2024.10689785
[18] I. Amerini et al., “Deepfake Media Forensics: Status and Future Challenges,” J. Imaging, vol. 11, no. 3, p. 73, Feb. 2025, doi: 10.3390/jimaging11030073. DOI: https://doi.org/10.3390/jimaging11030073
[19] X. Li, P.-Y. Chen, and W. Wei, “Measuring the Robustness of Audio Deepfake Detectors,” Mar. 21, 2025, arXiv: arXiv:2503.17577. doi: 10.48550/arXiv.2503.17577.
[20] X. Zhang, J. Yi, and J. Tao, “Towards Robust Audio Deepfake Detection: A Evolving Benchmark for Continual Learning,” Aug. 13, 2024, arXiv: arXiv:2405.08596. doi: 10.48550/arXiv.2405.08596.
[21] L. Liu et al., “YAMNet-based transfer learning for compact noise classification in urban and wireless systems,” EURASIP J. Wirel. Commun. Netw., vol. 2025, no. 1, p. 74, Sep. 2025, doi: 10.1186/s13638-025-02483-8. DOI: https://doi.org/10.1186/s13638-025-02483-8
[22] H. Zhong, J. M. Buchholz, J. Maclaren, S. Carlile, and R. Lyon, “A dataset and model for auditory scene recognition for hearing devices: AHEAD-DS and OpenYAMNet,” Jan. 13, 2026, arXiv: arXiv:2508.10360. doi: 10.48550/arXiv.2508.10360.
[23] S. Lachenani, H. Kheddar, and M. Ouldzmirli, “Improving Pretrained YAMNet for Enhanced Speech Command Detection via Transfer Learning,” in 2024 International Conference on Telecommunications and Intelligent Systems (ICTIS), Djelfa, Algeria: IEEE, Dec. 2024, pp. 1–6. doi: 10.1109/ICTIS62692.2024.10894266. DOI: https://doi.org/10.1109/ICTIS62692.2024.10894266
[24] M. Lamrini, M. Y. Chkouri, and A. Touhafi, “Evaluating the Performance of Pre-Trained Convolutional Neural Network for Audio Classification on Embedded Systems for Anomaly Detection in Smart Cities,” Sensors, vol. 23, no. 13, p. 6227, Jul. 2023, doi: 10.3390/s23136227. DOI: https://doi.org/10.3390/s23136227
[25] K. Shahzad, S. Farhan, Yasin-Ul-Haq, R. Sana, and S. Pathan, “Enhancing Voice Spoofing Detection: A Hybrid Approach With VGGish-LSTM Model for Improved Security in Automatic Speaker Verification Systems,” IEEE Access, vol. 13, pp. 40682–40702, 2025, doi: 10.1109/ACCESS.2025.3544639. DOI: https://doi.org/10.1109/ACCESS.2025.3544639
[26] D. A. Prasetya, A. Sanusi, G. Chandrarin, E. Roikhah, I. Mujahidin, and R. Arifuddin, “Small and Medium Enterprises Problem and Potential Solutions for Waste Management,” J. Southwest Jiaotong Univ., vol. 54, no. 6, p. 21, 2019, doi: 10.35741/issn.0258-2724.54.6.21. DOI: https://doi.org/10.35741/issn.0258-2724.54.6.21
[27] N. Müller, P. Czempin, F. Diekmann, A. Froghyar, and K. Böttinger, “Does Audio Deepfake Detection Generalize?,” in Interspeech 2022, ISCA, Sep. 2022, pp. 2783–2787. doi: 10.21437/Interspeech.2022-108. DOI: https://doi.org/10.21437/Interspeech.2022-108
[28] S. Liao et al., “Fish-Speech: Leveraging Large Language Models for Advanced Multilingual Text-to-Speech Synthesis,” Nov. 09, 2024, arXiv: arXiv:2411.01156. doi: 10.48550/arXiv.2411.01156.
[29] A. Muhaimin, W. Wibowo, and P. A. Riyantoko, “Multi-label Classification Using Vector Generalized Additive Model via Cross-Validation,” J. Inf. Commun. Technol., vol. 22, 2023, doi: 10.32890/jict2023.22.4.5. DOI: https://doi.org/10.32890/jict2023.22.4.5
[30] A. F. Adziima, A. Muhaimin, A. T. Damaliana, A. R. Pratama, K. M. Hindrayani, and D. A. Prasetya, “Incremental Learning for Drought Prediction with CatBoost on Time-Series Data,” in 2025 IEEE 11th Information Technology International Seminar (ITIS), Mataram, Indonesia: IEEE, Oct. 2025, pp. 400–405. doi: 10.1109/ITIS67966.2025.11308862. DOI: https://doi.org/10.1109/ITIS67966.2025.11308862
[31] D. Soekhoe, P. Van Der Putten, and A. Plaat, “On the Impact of Data Set Size in Transfer Learning Using Deep Neural Networks,” in Advances in Intelligent Data Analysis XV, vol. 9897, H. Boström, A. Knobbe, C. Soares, and P. Papapetrou, Eds., in Lecture Notes in Computer Science, vol. 9897. , Cham: Springer International Publishing, 2016, pp. 50–60. doi: 10.1007/978-3-319-46349-0_5. DOI: https://doi.org/10.1007/978-3-319-46349-0_5
[32] D. A. Prasetya, A. P. Sari, P. A. Riyantoko, and T. M. Fahrudin, “The Effect of Information Quality and Service Quality on User Satisfaction of the Government of Kabupaten Malang,” TIERS Inf. Technol. J., vol. 4, no. 1, pp. 32–42, Jun. 2023, doi: 10.38043/tiers.v4i1.4328. DOI: https://doi.org/10.38043/tiers.v4i1.4328
[33] M. Nasrudin, S. S. M. Wara, A. Muhaimin, N. I. Nirmalasari, and M. R. Arfiana, “Detection of Ventricular Septal Defect in Pediatric Cardiac Ultrasound Videos Using Parasternal View and Faster R-CNN,” vol. 15, no. 1, 2026.
[34] T. M. Fahrudin, I. Syarif, and A. R. Barakbah, “Data Mining Approach for Breast Cancer Patient Recovery,” Emit. Int. J. Eng. Technol., vol. 5, no. 1, pp. 36–71, Jul. 2017, doi: 10.24003/emitter.v5i1.190. DOI: https://doi.org/10.24003/emitter.v5i1.190
[35] M. Idhom, D. A. Prasetya, P. A. Riyantoko, T. M. Fahrudin, and A. P. Sari, “Pneumonia Classification Utilizing VGG-16 Architecture and Convolutional Neural Network Algorithm for Imbalanced Datasets,” TIERS Inf. Technol. J., vol. 4, no. 1, pp. 73–82, Jun. 2023, doi: 10.38043/tiers.v4i1.4380. DOI: https://doi.org/10.38043/tiers.v4i1.4380
[36] E. Issa, “AFAD-MSA: Dataset and Models for Arabic Fake Audio Detection,” Computation, vol. 14, no. 1, p. 20, Jan. 2026, doi: 10.3390/computation14010020. DOI: https://doi.org/10.3390/computation14010020
[37] M. KOKLU, B. Isgor, and C. Tumer, “Hybrid Feature-Based Two-Stage Framework for Audio Deepfake Detection and Generative Model Attribution,” Adv. Eng. Intell. Syst., vol. 005, no. 01, Mar. 2026, doi: 10.22034/aeis.2026.566863.1400.
Downloads
Published
Issue
Section
License
Copyright (c) 2026 Hakam Dzakwan Diash, Dwi Arman Prasetya, Alfan Rizaldy Pratama, Tresna Maulana Fahrudin
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
How to Cite
Share
Plum Analytics