Multiple Transformer-Based Models for Phase-Sensitive Masking-Based Single-Channel Speech Enhancement

Abstract

Leveraging both global contextual dependencies and local temporal-spectral patterns can further enhance speech quality and intelligibility, motivating the integration of diverse neural network structures for improved mask estimation. Furthermore, due to the limitations of existing time-frequency phase-aware masks, a new constrained phase-sensitive mask is introduced and estimated using the proposed architectures. In this paper, we first propose a modified transformer model for constrained phase-sensitive ideal magnitude ratio mask (cPSIRM) estimation (TRF-cPSIRM) to incorporate both magnitude and phase information and improve speech enhancement quality. The transformer model can extract global information, whereas the convolutional neural network (CNN) and convolutional recurrent network (CRN) are effective in capturing local content. CNNs have feature extraction capability due to their convolutional layers, and CRNs benefit from the temporal modelling strength of LSTM recurrent layers, which is useful for enhancement. Therefore, in this paper, to exploit both transformer (TRF) and CNN/CRN capabilities, we propose the cascaded structures (models) of CNN and TRF layers (CNN-TRF-cPSIRM) and also TRF and CRN layers (TRF-CRN-cPSIRM) for cPSIRM estimation and, consequently, speech enhancement. The CNN is used for feature extraction in CNN-TRF-cPSIRM and CRN for enhancement in TRF-CRN-cPSIRM. Moreover, considering the harmonic property of speech in the frequency spectrum, we present the transposed transformer-based model (TTRF), in which the neighbourhood relationship between the different frequency sub-bands is used as a sequence in the model. Then, to model both the long-term and short-term dependencies, the cascaded model of TTRF (intra-transformer) and TRF (inter-transformer) is proposed for cPSIRM estimation (TTRF-TRF-cPSIRM). The experimental results show that among these transformer-based models, the CNN-TRF-cPSIRM has the best performance, achieving up to about 0.44 perceptual evaluation of speech quality (PESQ) improvement over the baseline and 1.5 over the noisy speech.

Keywords

phase-sensitive ideal magnitude ratio mask (cPSIRM) / speech enhancement (SE) / transformer

Funding

The authors have nothing to report.

Conflicts of Interest

The authors declare no conflicts of interest.

Data Availability Statement

Research data are not shared.

References

[1] A. S. Al-Zubaidi, B. M. Mahmmod, S. H. Abdulhussain, M. A. Naser, and A. Hussain, “Low-Distortion MMSE Estimator for Speech Enhancement Based on Hahn Moments,” in 2023 15th International Conference on Developments in eSystems Engineering, (2023), 322-327.

[2] C. Lan, S. Zhao, H. Chen, et al., Single-Channel Speech Enhancement Algorithm Based on ME-MGCRN in Low Signal-to-Noise Scenario,” IEEE Access 12, (2024): 101342-101355, https://doi.org/10.1109/access.2024.3431713.

[3] S. A. Jerjees, H. J. Mohammed, H. S. Radeaf, B. M. Mahmmod, and S. H. Abdulhussain, “Deep Learning-Based Speech Enhancement Algorithm Using Charlier Transform,” in 2023 15th International Conference on Developments in eSystems Engineering (2023), 100-105.

[4] T.Yanhui, D.Jun, X.Yong, D.Lirong, and L.Chin-hui , “ Deep Neural Network Based Speech Separation for Robust Speech Recognition,” in 2014 12th International Conference on Signal Processing (ICSP), (IEEE, 2014), 532-536.

[5] M. Pashaian and S. Seyedin, “Speech Enhancement Using Joint DNN-NMF Model Learned With Multi-Objective Frequency Differential Spectrum Loss Function,” IET Signal Processing 2024 (2024): 1-10, https://doi.org/10.1049/2024/8881007.

[6] M. Pashaian, S. Seyedin, and S. M. Ahadi , “A Novel Jointly Optimized Cooperative DAE-DNN Approach Based on a New Multi-Target Step-Wise Learning for Speech Enhancement,” IEEE Access 11 (2023): 21669-21685, https://doi.org/10.1109/access.2023.3250820.

[7] S. W. Fu, T. Y. Hu, Y. Tsao, and X. Lu, “Complex Spectrogram Enhancement by Convolutional Neural Network With Multi-Metrics Learning,” in 2026 IEEE International Workshop on Machine Learning for Signal Processing, (2017), 1-6.

[8] Y. Shi, W. Yuan, S. Hu, and Y. Lou, “A Convolutional Recurrent Neural Network for Real-Time Speech Enhancement,” International Speech Communication Association (2018): 3229-3233.

[9] K. Tan and D. Wang, “Learning Complex Spectral Mapping With Gated Convolutional Recurrent Networks for Monaural Speech Enhancement,” IEEE/ACM Transactions on Audio, Speech and Language Processing 28 (2020): 380-390, https://doi.org/10.1109/taslp.2019.2955276.

[10] M. Pashaian and S. Seyedin, “Speech Enhancement Based on a Joint Two-Stage CRN+DNN-DEC Model and a New Constrained Phase-Sensitive Magnitude Ratio Mask,” IEEE Access 12 (2024): 98567-98583.

[11] M. Strake, B. Defraene, K. Fluyt, W. Tirry, and T. Fingscheidt, “Speech Enhancement by LSTM-based Noise Suppression Followed by CNN-Based Speech Restoration,” EURASIP Journal on Advances in Signal Processing 49 (2020): 1-26, https://doi.org/10.1186/s13634-020-00707-1.

[12] A. Vaswani, N. Shazeer, N. Parmar, et al. , “Attention Is All You Need,” Advances in Neural Information Processing Systems (2017): 5998-6008.

[13] W. Yu, J. Zhou, H. Wang, and L. Tao, “SETransformer: Speech Enhancement Transformer,” Cognitive Computation (2022), https://link.springer.com/article/10.1007/s12559-020-09817-2.

[14] N. Borjian, E. Kabir, S. Seyedin, and E. Masehian, “A Query-By-Example Music Retrieval System Using Feature and Decision Fusion,” Multimedia Tools and Applications 77 (2018): 6165-6189.

[15] N. C. Hoang, T. N. L. Nguyen, T. K. Doan, and Q. C. Nguyen, “Multi-Stage Temporal Representation Learning via Global and Local Perspectives for Real-Time Speech Enhancement,” Applied Acoustics 223 (2024): 110067, https://doi.org/10.1016/j.apacoust.2024.110067.

[16] Z. Zhang, L. Zhang, X. Zhuang, Y. Qian, and M. Wang, “Supervised Attention Multi-Scale Temporal Convolutional Network for Monaural Speech Enhancement,” EURASIP Journal on Audio Speech and Music Processing 2024, no. 1 (2024): 20, https://doi.org/10.1186/s13636-024-00341-x.

[17] S. Yecchuri and S. D. Vanambathina, “Sub-Convolutional U-Net With Transformer Attention Network for End-to-End Single-Channel Speech Enhancement,” EURASIP Journal on Audio Speech and Music Processing 2024, no. 1 (2024): 8, https://doi.org/10.1186/s13636-024-00331-z.

[18] S. Yechuri, T. R. Komati, R. K. Yellapragada, and S. Vanambathina, “A Multi-Scale Subconvolutional U-Net With Time-Frequency Attention Mechanism for Single Channel Speech Enhancement,” Circuits, Systems, and Signal Processing (2024), https://link.springer.com/article/10.1007/s00034-024-02721-2.

[19] K. Wang, B. He, and W. P. Zhu, “Tstnn: Two-Stage Transformer Based Neural Network for Speech Enhancement in the Time Domain,” in IEEE International Conference on Acoustics, Speech, and Signal Processing, (2021), 7098-7102.

[20] D. de Oliveira, T. Peer, and T. Gerkmann, “Efficient Transformer-Based Speech Enhancement Using Long Frames and STFT Magnitudes,” Proceedings of the Annual Conference of the International Speech Communication Association, InterSpeech (2022): 2948-2952, https://doi.org/10.48550/arXiv.2206.11703.

[21] Y. Guo, W. Yu, and J. Zhou, “ZTrans: A New Transformer for Speech Enhancement,” in 2021 4th International Conference on Information Communication and Signal Processing (ICICSP), (2021), 178-182.

[22] A. Gulati, J. Qin, C. C. Chiu, et al., Conformer: Convolution-Augmented Transformer for Speech Recognition,” Proceedings of the Annual Conference of the International Speech Communication Association, InterSpeech 2020, (2020): 5036-5040, https://doi.org/10.48550/arXiv.2005.08100.

[23] C. Subakan, M. Ravanelli, S. Cornell, M. Bronzi, and J. Zhong, “Attention Is all You Need in Speech Separation,” in ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing, (2021), 21-25.

[24] J. Kim, M. El-Khamy, and J. Lee, “T-GSA: Transformer With Gaussian-Weighted Self-Attention for Speech Enhancement,” in ICASSP 2020 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), (2020), 6649-6653.

[25] A. Nicolson and K. K. Paliwal, “Masked Multi-Head Self-Attention for Causal Speech Enhancement,” Speech Communication 125 (2020): 80-96, https://doi.org/10.1016/j.specom.2020.10.004.

[26] Q. Zhang, X. Qian, Z. Ni, A. Nicolson, E. Ambikairajah, and H. Li , “A Time-Frequency Attention Module for Neural Speech Enhancement,” IEEE/ACM Transactions on Audio, Speech and Language Processing 31 (2023): 462-475, https://doi.org/10.1109/taslp.2022.3225649.

[27] L. Deng, “Deep Learning: From Speech Recognition to Language and Multimodal Processing,” APSIPA Transactions on Signal and Information Processing 5, no. 1 (2016): e1, https://doi.org/10.1017/atsip.2015.22.

[28] A. B. Nassif, I. Shahin, I. Attili, M. Azzeh, and K. Shaalan, “Speech Recognition Using Deep Neural Networks: A Systematic Review,” IEEE Access 7 (2019): 19143-19165, https://doi.org/10.1109/access.2019.2896880.

[29] X. Tan, T. Qin, F. K. Soong, and T.-Y. Liu, “A Survey on Neural Speech Synthesis,” ArXiv, vol. abs/2106.1 (2021).

[30] K. B. Bhangale and M. Kothandaraman, “Survey of Deep Learning Paradigms for Speech Processing,” Wireless Personal Communications 125, no. 2 (2022): 1913-1949, https://doi.org/10.1007/s11277-022-09640-y.

[31] S. Latif, A. Zaidi, H. Cuayáhuitl, F. Shamshad, M. Shoukat, and J. Qadir, “Transformers in Speech Processing: A Survey,” (2023).

[32] S. Zhang, M. Chadwick, A. Ramos, and S. Bhattacharya, “Cross-Attention Is all You Need: Real-Time Streaming Transformers for Personalised Speech Enhancement,” (2022).

[33] Y. Zhao, C. Luo, Z.-J. Zha, and W. Zeng, “Multi-Scale Group Transformer for Long Sequence Modeling in Speech Separation,” in Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence (IJCAI-PRICAI 2020), (2020), 3251-3257.

[34] W. Jiang, C. Sun, F. Chen, et al., Low Complexity Speech Enhancement Network Based on Frame-Level Swin Transformer,” Electronics 12, no. 6 (2023): 1330, https://doi.org/10.3390/electronics12061330.

[35] D. L. J. Chen and Q. Mao, “Dual-Path Transformer Network: Direct Context-Aware Modeling for End-to-End Monaural Speech Separation,” in Proceedings of Interspeech, (2020), 2642-2646.

[36] Y. Wang, A. Narayanan, and D. L. Wang, “On Training Targets for Supervised Speech Separation,” IEEE/ACM Transactions on Audio, Speech and Language Processing 22, no. 12 (2014): 1849-1858, https://doi.org/10.1109/TASLP.2014.2352935.

[37] H. Erdogan, J. R. Hershey, S. Watanabe, and J. Le Roux, “Phase-Sensitive and Recognition-Boosted Speech Separation Using Deep Recurrent Neural Networks,” in IEEE International Conference on Acoustics, Speech, and Signal Processing, (2015), 708-712.

[38] D. S. Williamson, Y. Wang, and D. Wang, “Complex Ratio Masking for Monaural Speech Separation,” IEEE/ACM Transactions on Audio, Speech and Language Processing 24, no. 3 (2016): 483-492, https://doi.org/10.1109/taslp.2015.2512042.

[39] Z. Wang, X. Wang, X. Li, Q. Fu, and Y. Yan, “Oracle Performance Investigation of the Ideal Masks,” IEEE Int. Work. Acoust. Signal Enhanc. (2016): 1-5.

[40] D. Yin, C. Luo, Z. Xiong, and W. Zeng, “PHASEN: A Phase-and-Harmonics-Aware Speech Enhancement Network,” Proceedings of the AAAI Conference on Artificial Intelligence 34, no. 5 (2020): 9458-9465, https://doi.org/10.1609/aaai.v34i05.6489.

[41] M. Hasannezhad, Z. Ouyang, W.-P. Zhu, and B. Champagne, “An Integrated CNN-GRU Framework for Complex Ratio Mask Estimation in Speech Enhancement,” in Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC), (2020), 764-768.

[42] S. Xia, H. Li, and X. Zhang, “Using Optimal Ratio Mask as Training Target for Supervised Speech Separation,” in Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC), (2017), 5-8.

[43] M. Hasannezhad , Speech Enhancement with Improved Deep Learning Methods’. Ph.D. Thesis.

[44] K. He, X. Zhang, S. Ren, and J. Sun, “Deep Residual Learning for Image Recognition,” in 2016 IEEE Conference on Computer Vision and Pattern Recognition, (2016), 770-778.

[45] J. L. Ba, J. R. Kiros, and G. E. Hinton, “Layer Normalization,” (2016).

[46] A. L. Maas, “Rectifier Nonlinearities Improve Neural Network Acoustic Models,” (2013).

[47] X. Glorot, A. Bordes, and Y. Bengio, “Deep Sparse Rectifier Neural Networks,” in Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, (2011), 315-323.

[48] K. He, X. Zhang, S. Ren, and J. Sun, “Delving Deep Into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification,” in 2015 IEEE International Conference on Computer Vision, (2015), 1026-1034.

[49] M. Ravanelli, T. Parcollet, P. Plantinga, et al., {SpeechBrain}: A General-Purpose Speech Toolkit,” 2021.

[50] J. Garofolo, L. Lamel, W. Fisher, J. Fiscus, and D. Pallett, “DARPA TIMIT Acoustic-Phonetic Continuous Speech Corpus CD-ROM. NIST Speech Disc 1-1.1,” in NASA STI/Recon Technical Report N 93, (1993).

[51] A. Varga and H. J. M. Steeneken, “Assessment for Automatic Speech Recognition: II. NOISEX-92: A Database and an Experiment to Study the Effect of Additive Noise on Speech Recognition Systems,” Speech Communication 12, no. 3 (1993): 247-251, https://doi.org/10.1016/0167-6393(93)90095-3.

[52] Sounds Data, (July 2020) [Online], https://freesound.org/.

[53] D.-A. Clevert, T. Unterthiner, and S. Hochreiter , Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs) (Under Rev. ICLR2016, 2015).

[54] D. Kingma and J. Ba, “Adam: A Method for Stochastic Optimization,” in International Conference on Learning Representations, (2014).

[55] A. W. Rix, J. G. Beerends, M. P. Hollier, and A. P. Hekstra, “Perceptual Evaluation of Speech Quality (PESQ)-a New Method for Speech Quality Assessment of Telephone Networks and Codecs,” in IEEE International Conference on Acoustics, Speech, and Signal Processing, (May 2001), 749-752.

[56] C. H. Taal, R. C. Hendriks, R. Heusdens, and J. Jensen, “An Algorithm for Intelligibility Prediction of time-frequency Weighted Noisy Speech,” IEEE Transactions on Audio, Speech and Language Processing 19, no. 7 (2011): 2125-2136, https://doi.org/10.1109/tasl.2011.2114881.

[57] Y. Hu and P. C. Loizou, “Evaluation of Objective Quality Measures for Speech Enhancement,” IEEE Transactions on Audio, Speech and Language Processing 16, no. 1 (January 2008): 229-238, https://doi.org/10.1109/tasl.2007.911054.

[58] J. Yun, J. Lee, C. G. Kim, et al., A Novel Performance Prediction Model for Mobile GPUs,” IEEE Access 6 (2018): 16235-16245, https://doi.org/10.1109/access.2018.2816040.

[59] S. Wang, A. Pathania, and T. Mitra, “Neural Network Inference on Mobile SoCs,” IEEE Design & Test 37, no. 5 (2020): 50-57, https://doi.org/10.1109/mdat.2020.2968258.