Automatic Detection of Toxic Content in Short Videos Using Deep Learning-Based Text and Audio Feature Integration

Heri Agus Supriyanto; Ryan Ari Setyawan; Jemmy Edwin Bororing

doi:10.61255/decoding.v4i2.1540

Authors

Heri Agus Supriyanto Janabadra University
Ryan Ari Setyawan Janabadra University
Jemmy Edwin Bororing Janabadra University

DOI:

https://doi.org/10.61255/decoding.v4i2.1540

Keywords:

Audio-text classification, BiLSTM, Multimodal fusion, Short-form video, Toxic video detection

Abstract

Purpose – This study develops and evaluates a multimodal classification model combining audio and text features via configurable weighted late fusion to detect toxic content in keyword-retrieved Indonesian short videos containing profanity-related contexts, addressing text-only detection limitations.
Methods – The pipeline utilized FFmpeg for audio extraction, MFCC for audio features, and Google Speech Recognition for text. Three fusion configurations (Audio-Text: 40%–60%, 50%–50%, and 60%–40%) combining a DNN and BiLSTM were evaluated on 1,484 manually labeled Indonesian short videos.
Findings – The Audio 60%–Text 40% configuration achieved the numerically highest test accuracy of 93.94% with a 95% confidence interval of 90.62%–96.13%, using a decision threshold of 0.60 selected from the validation set. The model obtained an F1-score of 0.95 for the toxic class. Compared with unimodal baselines, all fusion models achieved higher accuracy, indicating the benefit of integrating audio and text features.
Research implications – The findings suggest that multimodal audio-text fusion can improve Indonesian short-form video toxicity detection compared with audio-only or text-only models. However, the differences among the three fusion weighting schemes were not statistically significant based on McNemar’s test, so the audio-dominant configuration should be interpreted as the numerically best configuration in this dataset rather than as a universally superior setting.
Originality – This study systematically compares unimodal baselines and configurable audio-text late-fusion weighting strategies for Indonesian short-form video toxicity detection. The study also applies validation-based threshold selection, confidence intervals, and pairwise McNemar testing to provide a more reliable evaluation of multimodal model performance.

Abstract views: 20 ,

PDF downloads: 13

Downloads

Download data is not yet available.

References

T. I. Sari, Z. N. Ardilla, N. Hayatin, and R. Maskat, “Abusive comment identification on Indonesian social media data using hybrid deep learning,” IAES International Journal of Artificial Intelligence, vol. 11, no. 3, pp. 895–904, Sep. 2022, doi: 10.11591/ijai.v11.i3.pp895-904.

M. O. Ibrohim and I. Budi, “Hate speech and abusive language detection in Indonesian social media: Progress and challenges,” Aug. 01, 2023, Elsevier Ltd. doi: 10.1016/j.heliyon.2023.e18647.

A. Chhabra and D. K. Vishwakarma, “A literature survey on multimodal and multilingual automatic hate speech identification,” Multimed. Syst., vol. 29, no. 3, pp. 1203–1230, Jun. 2023, doi: 10.1007/s00530-023-01051-8.

M. Subramanian, V. Easwaramoorthy Sathiskumar, G. Deepalakshmi, J. Cho, and G. Manikandan, “A survey on hate speech detection and sentiment analysis using machine learning and deep learning models,” Oct. 01, 2023, Elsevier B.V. doi: 10.1016/j.aej.2023.08.038.

G. A. Koushik, D. Kanojia, and H. Treharne, “Towards a Robust Framework for Multimodal Hate Detection: A Study on Video vs. Image-based Content,” in WWW Companion 2025 - Companion Proceedings of the ACM Web Conference 2025, Association for Computing Machinery, Inc, May 2025, pp. 2014–2023. doi: 10.1145/3701716.3718382.

H. S. Wicaksana, K. Huda, and G. Airlangga, “Improved Text Classification for Indonesian Hate Speech Detection: FastText-LSTM Model with Easy Data Augmentation,” Jurnal Sistem Komputer dan Informatika (JSON), vol. 7, no. 3, pp. 1074–1083, Mar. 2026, doi: 10.30865/json.v7i3.9637.

A. B. Gumelar, E. M. Yuniarno, A. Nugroho, D. P. Adi, I. Sugiarto, and M. H. Purnomo, “An Improved Toxic Speech Detection on Multimodal Scam Confrontation Data Using LSTM-Based Deep Learning,” International Journal of Intelligent Engineering and Systems, vol. 17, no. 6, pp. 880–904, 2024, doi: 10.22266/ijies2024.1231.67.

R. Sharma and A. Arya, “MMHFND: Fusing Modalities for Multimodal Multiclass Hindi Fake News Detection via Contrastive Learning,” ACM Transactions on Asian and Low-Resource Language Information Processing, vol. 23, no. 11, Nov. 2024, doi: 10.1145/3686797.

K. Yousaf and T. Nawaz, “A Deep Learning-Based Approach for Inappropriate Content Detection and Classification of YouTube Videos,” IEEE Access, vol. 10, pp. 16283–16298, 2022, doi: 10.1109/ACCESS.2022.3147519.

R. Prabhu and V. Seethalakshmi, “A comprehensive framework for multi-modal hate speech detection in social media using deep learning,” Scientific Reports, vol. 15, Art. no. 13020, Apr. 2025, doi: 10.1038/s41598-025-94069-z.

H. Zennou, R. Ouadad, M. Ouhda, and M. Baslam, “Real-Time Speech Emotion Recognition with a CNN-BiLSTM-Attention Deep Learning Model,” Engineering, Technology & Applied Science Research, vol. 16, no. 3, pp. 35047–35055, Jun. 2026, doi: 10.48084/etasr.17495.

F. Andayani, L. B. Theng, M. T. Tsun, and C. Chua, “Hybrid LSTM-Transformer Model for Emotion Recognition From Speech Audio Files,” IEEE Access, vol. 10, pp. 36018–36027, 2022, doi: 10.1109/ACCESS.2022.3163856.

M. Das, R. Raj, P. Saha, B. Mathew, M. Gupta, and A. Mukherjee, “HateMM: A Multi-Modal Dataset for Hate Video Classification,” Zenodo, Jun. 2023, doi: 10.5281/zenodo.7799469.

M. K. Gourisaria, R. Agrawal, M. Sahni, and P. K. Singh, “Comparative analysis of audio classification with MFCC and STFT features using machine learning techniques,” Discover Internet of Things, vol. 4, no. 1, Dec. 2024, doi: 10.1007/s43926-023-00049-y.

A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, and I. Sutskever, “Robust Speech Recognition via Large-Scale Weak Supervision,” in Proceedings of the 40th International Conference on Machine Learning, vol. 202, pp. 28492–28518, Jul. 2023.

O. Rainio, J. Teuho, and R. Klén, “Evaluation metrics and statistical tests for machine learning,” Scientific Reports, vol. 14, Art. no. 6086, Mar. 2024, doi: 10.1038/s41598-024-56706-x.

J. An, W. Lee, Y. Jeon, J. Ok, Y. Kim, and G. G. Lee, “An Investigation into Explainable Audio Hate Speech Detection,” in Proceedings of the 25th Annual Meeting of the Special Interest Group on Discourse and Dialogue, 2024, pp. 533–543, doi: 10.18653/v1/2024.sigdial-1.45.

M. Alfaro-Contreras, J. J. Valero-Mas, J. M. Iñesta, and J. Calvo-Zaragoza, “Late multimodal fusion for image and audio music transcription,” Expert Syst. Appl., vol. 216, Apr. 2023, doi: 10.1016/j.eswa.2022.119491.

L. H. Moeslich and C. B. Santoso, “Pengembangan Sistem Media Intelligence ESG Berbasis NLP Bahasa Indonesia Menggunakan TF-IDF dan IndoBERT,” JSAI: Journal Scientific and Applied Informatics, vol. 9, no. 2, pp. 213–220, Jun. 2026, doi: 10.36085/jsai.v9i2.10586.

N. M. D. Sikiandani, I. M. A. Dwi Suarjaya, and Y. Perdana Putra, “Browser-Based Detection of Harmful Content with Deep Learning Model,” Journal of Applied Informatics and Computing, vol. 9, no. 4, pp. 1800–1811, Aug. 2025, doi: 10.30871/jaic.v9i4.9804.

P. Rawat, M. Bajaj, S. Vats, and V. Sharma, “A comprehensive study based on MFCC and spectrogram for audio classification,” Journal of Information and Optimization Sciences, vol. 44, no. 6, pp. 1057–1074, 2023, doi: 10.47974/jios-1431.

W. Mu, B. Yin, X. Huang, J. Xu, and Z. Du, “Environmental sound classification using temporal-frequency attention based convolutional neural network,” Sci. Rep., vol. 11, no. 1, Dec. 2021, doi: 10.1038/s41598-021-01045-4.

S. M. Saleem Abdullah, S. Y. Ameen, M. A. M. Sadeeq, and S. R. M. Zeebaree, “Multimodal Emotion Recognition using Deep Learning,” Journal of Applied Science and Technology Trends, vol. 2, no. 1, pp. 73–79, Jan. 2021, doi: 10.38094/jastt20291.

L. L. Manihuruk, D. Ginting, S. Manik, and L. W. Manurung, “Slang Words Used by the Millennial Indonesian Generation in TikTok,” Journal of Applied Linguistics, vol. 4, no. 2, pp. 302–308, Jan. 2025, doi: 10.52622/joal.v4i2.362.

E. S. Alshahrani and M. S. Aksoy, “Adversarially Robust Multitask Learning for Offensive and Hate Speech Detection in Arabic Text Using Transformer-Based Models and RNN Architectures,” Applied Sciences (Switzerland), vol. 15, no. 17, Sep. 2025, doi: 10.3390/app15179602.