Bangla language modeling algorithm for automatic recognition of hand-sign-spelled Bangla sign language

Abstract

Because of using traditional hand-sign segmentation and classification algorithm, many diversities of Bangla language including joint-letters, dependent vowels etc. and representing 51 Bangla written characters by using only 36 hand-signs, continuous hand-sign-spelled Bangla sign language (BdSL) recognition is challenging. This paper presents a Bangla language modeling algorithm for automatic recognition of hand-sign-spelled Bangla sign language which consists of two phases. First phase is designed for hand-sign classification and the second phase is designed for Bangla language modeling algorithm (BLMA) for automatic recognition of hand-sign-spelledBangla sign language. In first phase, we have proposed two step classifiers for hand-sign classification using normalized outer boundary vector (NOBV) and window-grid vector (WGV) by calculating maximum inter correlation coefficient (ICC) between test feature vector and pre-trained feature vectors. At first, the system classifies hand-signs using NOBV. If classification score does not satisfy specific threshold then another classifier based on WGV is used. The system is trained using 5,200 images and tested using another (5, 200 × 6) images of 52 hand-signs from 10 signers in 6 different challenging environments achieving mean accuracy of 95.83% for classification with the computational cost of 39.972 milliseconds per frame. In the Second Phase, we have proposed Bangla language modeling algorithm (BLMA) which discovers all “hidden characters” based on “recognized characters” from 52 hand-signs of BdSL to make any Bangla words, composite numerals and sentences in BdSL with no training, only based on the result of first phase. To the best of our knowledge, the proposed system is the first system in BdSL designed on automatic recognition of hand-sign-spelled BdSL for large lexicon. The system is tested for BLMA using hand-sign-spelled 500 words, 100 composite numerals and 80 sentences in BdSL achieving mean accuracy of 93.50%, 95.50% and 90.50% respectively.

Keywords

Bangla sign language (BdSL) / hand-sign / classification / Bangla language modeling rules (BLMR) / Bangla language modeling algorithm (BLMA)

References

[1] Sutton V. Introduction on deafness, sign language & sign-writing. See Signwritingorg Website, 2016

[2] WFD. Sign language. See World Federation of the Deaf Website, 2016

[3] Dey S. Bangladesh sign language day. See DPI-AP Website, 2014

[4] Jasim M, Zhang T, Hasanuzzaman M. A real-time computer visionbased static and dynamic hand gesture recognition system. International Journal on Image and Graphics, 2014, 14(01n02): 145006

[5] Majumder M, Hossain M, Mamtaz M, Ahmed M, Khan H, Ali M H, Khaled O, Iqbal M. Bengali Sign Language Dictionary. 2nd ed. Dhaka, Bangladesh: National Centre for Special Education and Ministry of SocialWelfare in Cooperation with The Norwegian Association of The Deaf and Bangladersh National Federation of The Deaf, 1994

[6] Begum S, Hasanuzzaman M. Computer vision-based bangladeshi sign language recognition system. In: Proceedings of the 12th International Conference on Computer and Information Technology. 2009, 414–419

[7] Rahaman M A, Jasim M, Zhang T, Ali MH, Hasanuzzaman M. A realtime hand-signs segmentation and classification system using fuzzy rule based RGB model and grid-pattern analysis. Frontiers of Computer Science, 2018, 12(6): 1258–1260

[8] Rahaman M A, Jasim M, Zhang T, Ali M H, Hasanuzzaman M. Realtime bengali and chinese numeral signs recognition using contour matching. In: Proceedings of the IEEE International Conference on Robotics and Biomimetics. 2015, 1215–1220

[9] Li J, Wang J, Ju Z. A novel hand gesture recognition based on highlevel features. International Journal of Humanoid Robotics, 2018, 15(2): 1750022

[10] Dong L, Liang Y, Kong G, Zhang Q, Cao X, Izquierdo E. Holons visual representation for image retrieval. IEEE Transactions on Multimedia, 2016, 18(4): 714–725

[11] Dong L, He L, Zhang Q. Discriminative light unsupervised learning network for image representation and classification. In: Proceedings of the 23rd ACM International Conference on Multimedia. 2015, 1235–1238

[12] Garcia-Ceja E, Brena R F. An improved three-stage classifier for activity recognition. International Journal of Pattern and Recognition and Artifical Intelligence, 2018, 32(1): 1860003

[13] Lee G, Mallipecldi R, Lee M. Trajectory-based vehicle tracking at low frame rates. Expert System with Application, 2017, 80: 46–57

[14] Mills M T, Bourbakis N G. A comparative survey on NLP/U methodologies for processing multi-documents. International Journal on Artificial Intelligence Tools, 2012, 21(4): 1250017

[15] Mills M, Psarologou A, Bourbakis N. Modeling natural language sentences into SPN graphs. In: Proceedings of the 25th IEEE International Conference on Tools with Artificial Intelligence. 2013, 889–896

[16] Santoni D, Pourabbas E. Automatic detection of words associations in texts based on joint distribution of words occurrences. Computational Intelligence, 2016, 32(4): 535–560

[17] Lee G, Mallipeddi R, Lee M. Trajectory-based vehicle tracking at low frame rates. Expert Systems with Applications, 2017, 80: 46–57

[18] Liberman Y, Perry A. A visual tracking scheme for accurate object retrieval in low frame rate videos. International Journal on Artificial Intelligence Tools, 2016, 25(5): 1640003

[19] Dong L, Feng N, Mao M, He L, Wang J. E-grabcut: an economic method of iterative video object extraction. Frontiers of Computer Science, 2017, 11(4): 649–660

[20] Dong L, Feng N, Zhang Q. LSI: semantic label inference for nature image segmentation. Pattern Recognition, 2016, 59: 282–291

[21] Li C, Lin L, Zuo W, Wang W, Tang J. An approach to streaming video segmentation with sub-optimal low-rank decomposition. IEEE Transactions on Image Processing, 2016, 25(4): 1947–1960

[22] Chen F S, Fu C M, Huang C L. Hand gesture recognition using a realtime tracking method and hidden markov models. Image Vision Computer, 2003, 21(8): 745–758

[23] Alon J, Athitsos V, Yuan Q, Sclaroff S. A unified framework for gesture recognition and spatiotemporal gesture segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2009, 31(9): 1685–1699

[24] AsaariM SM, Rosdi B A, Suandi S A. Adaptive kalman filter incorporated eigenhand (akfie) for real-time hand tracking system. Multimedia Tools and Applications, 2015, 74(21): 9231–9257

[25] Khaled H, Sayed S G, Saad E SM, Ali H. Hand gesture recognition using modified 1 and background subtraction algorithms. Mathematical Problems in Engineering, 2015, (Article ID: 741068): 1–8

[26] Gurav R M, Kadbe P K. Vision based hand gesture recognition with haar classifier and adaboost algorithm. International Journal of Latest Trends in Engineering and Technology, 2015, 5(2): 155–160

[27] Rahaman M A, Jasim M, Ali M H, Hasanuzzaman M. Realtime computer vision-based bengali sign language recognition. In: Proceedings of the 17th International Conference on Computer and Information Technology. 2014, 192–197

[28] Weng S K, Kuo C M, Tu S K. Video object tracking using adaptive kalman filter. Journal of Visual Communication and Image Representation, 2006, 17(6): 1190–1208

[29] Li N, Liu L, Xu D. Corner feature based object tracking using adaptive kalman filter. In: Proceedings of the 9th International Conference on Signal Processing. 2008, 1432–1435

[30] Luo Y, Celenk M. A new adaptive kalman filtering method for blockbased motion estimation. In: Proceedings of the 15th International Conference on Systems, Signals and Image Processing. 2008, 89–92

[31] Karmokar B C, Alam K M R, Siddiquee M K. Bangladeshi sign language recognition employing neural network ensemble. International Journal of Computer Applications, 2012, 58(16): 43–46

[32] Jarman A M, Arshad S, Alam N, Islam M J. An automated bengali sign language recognition system based on fingertip finder algorithm. International Journal of Electronics and Informatics, 2015, 4(1): 1–10

[33] Yasir F, Prasad P WC, Alsadoon A, Elchouemi A. Sift based approach on bangla sign language recognition. In: Proceedings of the 8th IEEE International Workshop on Computational Intelligence and Applications. 2015, 35–39

[34] Ayshee T F, Raka S A, Hasib Q R, Hossain M, Rahman R M. Fuzzy rule-based hand gesture recognition for bengali characters. In: Proceedings of the IEEE International Advance Computing Conference. 2014, 484–489

[35] Wang, Qiao Y, Tang X. Action recognition with trajectorypooled deepconvolutional descriptors. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2015, 4305–4314

[36] Feichtenhofer, Pinz A, Wildes R. Spatiotemporal residual networks for video action recognition. In: Proceedings of the 30th International Conference on Neural Information Processing Systems. 2016, 3476–3484

[37] Asadi-Aghbolaghi M, Clapes A, Bellantonio M, Escalante H J, Ponce López V, Baró X, Guyon I, Kasaei S, Escalera S. A survey on deep learning based approaches for action and gesture recognition in image sequences. In: Proceedings of the 12th IEEE Conference on Automatic Face and Gesture Recognition (FG 2017). 2017, 476–483

[38] Liu Z, Zhang C, Tian Y. 3D-based deep convolutional neural network for action recognition with depth sequences. Image and Vision Computing, 2016, 55(2): 93–100

[39] Varol G, Laptev I, Schmid C. Long-term temporal convolutions for action recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017, 40(6): 1510–1517

[40] Zhu G, Zhang L, Mei L, Shao J, Song J, Shen P. Large-scale isolated gesture recognition using pyramidal 3D convolutional networks. In: Proceedings of the 23rd International Conference on Pattern Recognition. 2016, 19–24

[41] Wang P, Li W, Liu S, Gao Z, Tang C, Ogunbona P. Largescale isolated gesture recognition using convolutional neural networks. In: Proceedings of the 23rd International Conference on Pattern Recognition. 2016, 7–12

[42] Xu P. A real-time hand gesture recognition and humancomputer interaction system. 2017, arXiv preprint arXiv:1704.07296

[43] Rahaman M A, Jasim M, Ali M H, Hasanuzzaman M. Computer vision based bengali sign words recognition using contour analysis. In: Proceedings of the 18th International Conference on Computer and Information Technology. 2015, 335–340

[44] Park A, Yun S, Kim J, Min S, Jung K. Real-time visionbased korean finger spelling recognition system. International Journal of Computer, Electrical, Automation, Control and Information Engineering, 2008, 2(8): 2623–2628

[45] Kane L, Khanna P. A framework for live and cross platform fingerspelling recognition using modified shape matrix variants on depth silhouettes. Computer Vision and Image Understanding, 2015, 141: 138–151

[46] Fang G, Gao W, Zhao D. Large-vocabulary continuous sign language recognition based on transition-movement models. IEEE Transactions on Systems, Man and Cybernetics, Part A, 2007, 37(1): 1–9

[47] Liwicki S, Everingham M. Automatic recognition of fingerspelled words in British sign language. In: Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops. 2009, 50–57

[48] Koller O, Forster V J, Ney V H. Continuous sign language recognition: towards large vocabulary statistical recognition systems handling multiple signers. Computer Vision and Image Understanding, 2015, 141: 108–125

[49] Rameshar S. Sadaran Vhasabiggan and Bangla Vhasa Dhaka, Bangladesh: Ananda Press, 1996

[50] Mukhopaddhay A. Adorsho Bangla banan: ekti prostabona. See Galpersamay Website, 2017

[51] Kibria G. Bangla joint letter. See Daffodilvarsity.edu Website, 2013

[52] Chowdhuri M, Chowdhuri M H. Bangla Bhashar Bakoron (Grammar of Bengali language). 2nd ed. Dhaka, Bangladesh: Jatio Shikkhakromo Opatthopustok Board, 2014

[53] Choudhury J. Bangla Banan Abhidhan. 3rd ed. Dhaka, Bangladesh: Bangla Academy, 2008

[54] Ishaaque A. Samakalin Bangla Bhashar Abhidhan, 2nd ed. Dhaka, Bangladesh: Bangla Academy, 2003

[55] CDD. Manual on Sign Supported Bangla. Dhaka, Bangladesh: Center for Disability in Development (CDD), 2002

[56] Sourceforge.net. EmguCV. See Sourceforge Website, 2015

[57] Nagarajan S, Subashini T S. Static hand gesture recognition for sign language alphabets using edge oriented histogram and multi class SVM. International Journal of Computer Applications, 2013, 82(4): 28–35

RIGHTS & PERMISSIONS

2019 Higher Education Press and Springer-Verlag GmbH Germany, part of Springer Nature