Depression Intensity Identification using Transformer Ensemble Technique for the Resource-constrained Bengali Language
DOI:
https://doi.org/10.38032/jea.2024.02.001Keywords:
Bengali, Classification, Depression, Resource-constrained Language, Transformer EnsembleAbstract
Depression is an ordinary mental health-related disorder that hampers people’s daily activities, and sometimes, it destroys an individual’s life. It is one of the major social issues at present. Since depressed people use various social networking sites for sharing their thoughts and feelings, many scholars have tried to identify depression texts in highly resourced languages like English; however, only a small quantity of papers are detected in the resource-constrained Bengali language. This paper focuses on developing a depression intensity detection system from Bengali text data. In this regard, this study experiments on a 2,596 sample-sized dataset with four levels of depression by utilizing five state-of-the-art transformer models, including multilingual Bidirectional Encoder Representations from Transformers, DistilmBERT, XLM-RoBERTa, Bangla-BERT-Base, and BanglaBERT, and suggests a new ensemble method called MaxOfAvgProb. This method goes beyond the performance of the previous work on the same dataset, scoring 63.47% F1-score and 62.90% accuracy. To increase the reliability of the proposed method, we utilize this approach on another available dataset with 4,897 entries. In this case, our recommended method also surpasses the performance of the existing work on the same dataset, with accuracy at 86.45% and F1-score at 86.35%. Identifying the intensity of depression, depressed people may get proper counseling or treatment from their respected guardians or psychologists according to the victims’ level of depression.
References
Depressive disorder (depression), World Health Organization, Mar. 2023, accessed: 20 December 2023. [Online]. Available: https://www.who.int/news-room/fact-sheets/detail/depression
Kulsoom, B. and Afsar, N.A., 2015. Stress, anxiety, and depression among medical students in a multiethnic setting. Neuropsychiatric Disease and Treatment, pp.1713-1722. DOI: https://doi.org/10.2147/NDT.S83577
Hossain, M.D., Ahmed, H.U., Chowdhury, W.A., Niessen, L.W. and Alam, D.S., 2014. Mental disorders in Bangladesh: a systematic review. BMC psychiatry, 14, pp.1-8. DOI: https://doi.org/10.1186/s12888-014-0216-9
Arusha, A.R. and Biswas, R.K., 2020. Prevalence of stress, anxiety and depression due to examination in Bangladeshi youths: A pilot study. Children and youth services review, 116, p.105254. DOI: https://doi.org/10.1016/j.childyouth.2020.105254
Choudhury, A.A., Khan, M.R.H., Nahim, N.Z., Tulon, S.R., Islam, S. and Chakrabarty, A., 2019, June. Predicting depression in Bangladeshi undergraduates using machine learning. In 2019 IEEE Region 10 Symposium (TENSYMP) (pp. 789-794). IEEE. DOI: https://doi.org/10.1109/TENSYMP46218.2019.8971369
Hoque, R., 2015. Major mental health problems of undergraduate students in a private university of Dhaka, Bangladesh. European Psychiatry, 30, p.1880. DOI: https://doi.org/10.1016/S0924-9338(15)31442-5
Hoque, M.N. and Seddiqui, M.H., 2024. Detecting cyberbullying text using the approaches with machine learning models for the low-resource Bengali language. Int J Artif Intell ISSN, 2252(8938), p.358-367. DOI: https://doi.org/10.11591/ijai.v13.i1.pp358-367
Tanjim, K.F.H., Hoque, M.N. and Seddiqui, M.H., 2023. A Benchmark Dataset with Developing a Strong Baseline Accident Text Classification System for the Low-resource Bengali Language. In 2023 26th International Conference on Computer and Information Technology (ICCIT) (pp. 1-6). IEEE.
Uddin, A.H., Bapery, D. and Arif, A.S.M., 2019. Depression analysis from social media data in Bangla language using long short term memory (LSTM) recurrent neural network technique. In 2019 international conference on computer, communication, chemical, materials and electronic engineering (IC4ME2) (pp. 1-4). IEEE. DOI: https://doi.org/10.1109/IC4ME247184.2019.9036528
Mumu, T.F., Munni, I.J. and Das, A.K., 2021. Depressed people detection from bangla social media status using lstm and cnn approach. Journal of Engineering Advancements, 2(01), pp.41-47. DOI: https://doi.org/10.38032/jea.2021.01.006
Mohammed, M.B., Abir, A.S.M., Salsabil, L., Shahriar, M. and Fahmin, A., 2021, December. Depression Analysis from Social Media Data in Bangla Language: An Ensemble Approach. In 2021 Emerging Technology in Computing, Communication and Electronics (ETCCE) (pp. 1-6). IEEE. DOI: https://doi.org/10.1109/ETCCE54784.2021.9689887
Ghosh, T. and Kaiser, M.S., 2022, February. Bangla depressive social media text detection using hybrid deep learning approach. In Proceedings of the Third International Conference on Trends in Computational and Cognitive Engineering: TCCE 2021 (pp. 111-120). Singapore: Springer Nature Singapore. DOI: https://doi.org/10.1007/978-981-16-7597-3_9
Ahmed, A., Sultana, R., Ullas, M.T.R., Begom, M., Rahi, M.M.I. and Alam, M.A., 2020, December. A machine learning approach to detect depression and anxiety using supervised learning. In 2020 IEEE Asia-Pacific Conference on Computer Science and Data Engineering (CSDE) (pp. 1-6). IEEE. DOI: https://doi.org/10.1109/CSDE50874.2020.9411642
Das, A., Sharif, O., Hoque, M.M. and Sarker, I.H., 2021. Emotion classification in a resource constrained language using transformer-based approach. arXiv preprint arXiv:2104.08613. DOI: https://doi.org/10.18653/v1/2021.naacl-srw.19
Hossen, I., Islam, T., Rashed, M.G. and Das, D., 2022, October. Early Suicide Prevention: Depression Level Prediction Using Machine Learning and Deep Learning Techniques for Bangladeshi Facebook Users. In Proceedings of International Conference on Fourth Industrial Revolution and Beyond 2021 (pp. 735-747). Singapore: Springer Nature Singapore. DOI: https://doi.org/10.1007/978-981-19-2445-3_52
Kabir, M.K., Islam, M., Kabir, A.N.B., Haque, A. and Rhaman, M.K., 2022. Detection of Depression Severity Using Bengali Social Media Posts on Mental Health: Study Using Natural Language Processing Techniques. JMIR Formative Research, 6(9), p.e36118. DOI: https://doi.org/10.2196/36118
Hoque, M.N. and Salma, U., 2023. Detecting level of depression from social media posts for the low-resource bengali language. Journal of Engineering Advancements, 4(02), pp.49-56. DOI: https://doi.org/10.38032/jea.2023.02.003
Khan, S. and Alqahtani, S., 2023. Hybrid machine learning models to detect signs of depression. Multimedia Tools and Applications, pp.1-19. DOI: https://doi.org/10.1007/s11042-023-16221-z
Mustafa, R.U., Ashraf, N., Ahmed, F.S., Ferzund, J., Shahzad, B. and Gelbukh, A., 2020. A multiclass depression detection in social media based on sentiment analysis. In 17th International Conference on Information Technology–New Generations (ITNG 2020) (pp. 659-662). Springer International Publishing. DOI: https://doi.org/10.1007/978-3-030-43020-7_89
de Jesús Titla-Tlatelpa, J., Ortega-Mendoza, R.M., Montes-y-Gómez, M. and Villaseñor-Pineda, L., 2021. A profile-based sentiment-aware approach for depression detection in social media. EPJ data science, 10(1), p.54. DOI: https://doi.org/10.1140/epjds/s13688-021-00309-3
Chiu, C.Y., Lane, H.Y., Koh, J.L. and Chen, A.L., 2021. Multimodal depression detection on instagram considering time interval of posts. Journal of Intelligent Information Systems, 56(1), pp.25-47. DOI: https://doi.org/10.1007/s10844-020-00599-5
Abd El-Jawad, M.H., Hodhod, R. and Omar, Y.M., 2018, December. Sentiment analysis of social media networks using machine learning. In 2018 14th international computer engineering conference (ICENCO) (pp. 174-176). IEEE. DOI: https://doi.org/10.1109/ICENCO.2018.8636124
Paul, P. C., Ahmed, M. T., Hasan, M. R., Rajee, A., and Sultana, K., 2023. Analyzing depression on social media utilizing machine learning and deep learning methods. Indian Journal of Computer Science and Engineering, 14(5), pp.740– 746. DOI: https://doi.org/10.21817/indjcse/2023/v14i5/231405049
Soliman, T.H., Elmasry, M.A., Hedar, A. and Doss, M.M., 2014. Sentiment analysis of Arabic slang comments on facebook. International Journal of Computers & Technology, 12(5), pp.3470-3478. DOI: https://doi.org/10.24297/ijct.v12i5.2917
Seddiqui, M.H., Maruf, A.A.M. and Chy, A.N., 2016. Recursive suffix stripping to augment bangla stemmer. In International Conference Advanced Information and Communication Technology (ICAICT).
Kudo, T. and Richardson, J., 2018. Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. arXiv preprint arXiv:1808.06226. DOI: https://doi.org/10.18653/v1/D18-2012
Kudo, T., 2018. Subword regularization: Improving neural network translation models with multiple subword candidates. arXiv preprint arXiv:1804.10959. DOI: https://doi.org/10.18653/v1/P18-1007
Sennrich, R., Haddow, B. and Birch, A., 2015. Neural machine translation of rare words with subword units. arXiv preprint arXiv:1508.07909. DOI: https://doi.org/10.18653/v1/P16-1162
Hoque, M.N. and Seddiqui, M.H., 2023, December. Leveraging Transformer Models in the Cyberbullying Text Classification System for the Low-resource Bengali Language. In 2023 26th International Conference on Computer and Information Technology (ICCIT) (pp. 1-6). IEEE. DOI: https://doi.org/10.1109/ICCIT60459.2023.10441412
Pires, T., Schlinger, E. and Garrette, D., 2019. How multilingual is multilingual BERT?. arXiv preprint arXiv:1906.01502. DOI: https://doi.org/10.18653/v1/P19-1493
Devlin, J., Chang, M.W., Lee, K. and Toutanova, K., 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł. and Polosukhin, I., 2017. Attention is all you need. Advances in neural information processing systems, 30.
Conneau, A., Khandelwal, K., Goyal, N., Chaudhary, V., Wenzek, G., Guzmán, F., Grave, E., Ott, M., Zettlemoyer, L. and Stoyanov, V., 2019. Unsupervised cross-lingual representation learning at scale. arXiv preprint arXiv:1911.02116. DOI: https://doi.org/10.18653/v1/2020.acl-main.747
Sarker, S., 2022. BanglaBERT: Bengali mask language model for Bengali language understanding (2020). URL: https://github. com/sagorbrur/bangla-bert.
Bhattacharjee, A., Hasan, T., Ahmad, W.U., Samin, K., Islam, M.S., Iqbal, A., Rahman, M.S. and Shahriyar, R., 2021. BanglaBERT: Language model pretraining and benchmarks for low-resource language understanding evaluation in Bangla. arXiv preprint arXiv:2101.00204. DOI: https://doi.org/10.18653/v1/2022.findings-naacl.98
Clark, K., Luong, M.T., Le, Q.V. and Manning, C.D., 2020. Electra: Pre-training text encoders as discriminators rather than generators. arXiv preprint arXiv:2003.10555.
Rafi-Ur-Rashid, M., Mahbub, M. and Adnan, M.A., 2022. Breaking the curse of class imbalance: Bangla text classification. Transactions on Asian and Low-Resource Language Information Processing, 21(5), pp.1-21. DOI: https://doi.org/10.1145/3511601
Maiya, A.S., 2022. ktrain: A low-code library for augmented machine learning. Journal of Machine Learning Research, 23(158), pp.1-6.
Downloads
Published
Issue
Section
License
Copyright (c) 2024 Md. Nesarul Hoque, Umme Salma, Md. Jamal Uddin, Sadia Afrin Shampa
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
All the articles published by this journal are licensed under a Creative Commons Attribution-NonCommercial 4.0 International License
