Measuring code maintainability with deep neural networks

Abstract

The maintainability of source code is a key quality characteristic for software quality. Many approaches have been proposed to quantitatively measure code maintainability. Such approaches rely heavily on code metrics, e.g., the number of Lines of Code and McCabe’s Cyclomatic Complexity. The employed code metrics are essentially statistics regarding code elements, e.g., the numbers of tokens, lines, references, and branch statements. However, natural language in source code, especially identifiers, is rarely exploited by such approaches. As a result, replacing meaningful identifiers with nonsense tokens would not significantly influence their outputs, although the replacement should have significantly reduced code maintainability. To this end, in this paper, we propose a novel approach (called DeepM) to measure code maintainability by exploiting the lexical semantics of text in source code. DeepM leverages deep learning techniques (e.g., LSTM and attention mechanism) to exploit these lexical semantics in measuring code maintainability. Another key rationale of DeepM is that measuring code maintainability is complex and often far beyond the capabilities of statistics or simple heuristics. Consequently, DeepM leverages deep learning techniques to automatically select useful features from complex and lengthy inputs and to construct a complex mapping (rather than simple heuristics) from the input to the output (code maintainability index). DeepM is evaluated on a manually-assessed dataset. The evaluation results suggest that DeepM is accurate, and it generates the same rankings of code maintainability as those of experienced programmers on 87.5% of manually ranked pairs of Java classes.

Graphical abstract

Keywords

code maintainability / lexical semantics / deep learning / neural networks

References

[1] Schneidewind N F. The state of software maintenance. IEEE Transactions on Software Engineering, 1987, SE-13( 3): 303–310

[2] Bennett K H, Rajlich V T. Software maintenance and evolution: a roadmap. In: Proceedings of the Conference on the Future of Software Engineering. 2000, 73−87

[3] Lientz B P, Swanson E B. Software Maintenance Management: A Study of the Maintenance of Computer Application Software in 487 Data Processing Organizations. Reading: Addison-Wesley, 1980

[4] Yau S S, Collofello J S. Some stability measures for software maintenance. IEEE Transactions on Software Engineering, 1980, SE-6( 6): 545–552

[5] Reiss S P. Incremental maintenance of software artifacts. IEEE Transactions on Software Engineering, 2006, 32( 9): 682–697

[6] ISO. ISO/IEC 5055: 2021 Information technology — Software measurement — Software quality measurement — Automated source code quality measures. Geneva: ISO, 2021

[7] Grubb P, Takang A A. Software Maintenance: Concepts and Practice. 2nd ed. London: World Scientific Publishing, 2003

[8] Al Dallal J. Object-oriented class maintainability prediction using internal quality attributes. Information and Software Technology, 2013, 55( 11): 2028–2048

[9] Myers G J, Badgett T, Thomas T M, Sandler C. The Art of Software Testing. 2nd ed. Hoboken: John Wiley & Sons, 2004

[10] Mari M, Eila N. The impact of maintainability on component-based software systems. In: Proceedings of the 29th Euromicro Conference. 2003, 25−32

[11] Li W, Henry S. Object-oriented metrics that predict maintainability. Journal of Systems and Software, 1993, 23( 2): 111–122

[12] Oman P, Hagemeister J. Construction and testing of polynomials predicting software maintainability. Journal of Systems and Software, 1994, 24( 3): 251–266

[13] Heitlager I, Kuipers T, Visser J. A practical model for measuring maintainability. In: Proceedings of the 6th International Conference on the Quality of Information and Communications Technology (QUATIC 2007). 2007, 30−39

[14] ISO. ISO/IEC 9126-1: 2001 Software engineering—Product quality — Part 1: Quality model. Geneva: ISO, 2001

[15] Zhou Y, Leung H. Predicting object-oriented software maintainability using multivariate adaptive regression splines. Journal of Systems and Software, 2007, 80( 8): 1349–1361

[16] Bavota G, Dit B, Oliveto R, Di Penta M, Poshyvanyk D, De Lucia A. An empirical study on the developers’ perception of software coupling. In: Proceedings of the 35th International Conference on Software Engineering (ICSE). 2013, 692−701

[17] Pantiuchina J, Lanza M, Bavota G. Improving code: the (Mis) perception of quality metrics. In: Proceedings of 2018 IEEE International Conference on Software Maintenance and Evolution (ICSME). 2018, 80−91

[18] Lenhard J, Blom M, Herold S. Exploring the suitability of source code metrics for indicating architectural inconsistencies. Software Quality Journal, 2019, 27( 1): 241–274

[19] Liu H, Xu Z, Zou Y. Deep learning based feature envy detection. In: Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering. 2018, 385−396

[20] Scalabrino S, Linares-Vasquez M, Poshyvanyk D, Oliveto R. Improving code readability models with textual features. In: Proceedings of the 24th IEEE International Conference on Program Comprehension. 2016, 1−10

[21] Kim S, Kim D. Automatic identifier inconsistency detection using code dictionary. Empirical Software Engineering, 2016, 21( 2): 565–604

[22] Schankin A, Berger A, Holt D V, Hofmeister J C, Riedel T, Beigl M. Descriptive compound identifier names improve source code comprehension. In: Proceedings of the 26th IEEE/ACM International Conference on Program Comprehension (ICPC). 2018, 31−40

[23] Lai S, Liu K, He S, Zhao J. How to generate a good word embedding. IEEE Intelligent Systems, 2016, 31( 6): 5–14

[24] Greff K, Srivastava R K, Koutník J, Steunebrink B R, Schmidhuber J. LSTM: a search space odyssey. IEEE Transactions on Neural Networks and Learning Systems, 2017, 28( 10): 2222–2232

[25] Tai K S, Socher R, Manning C D. Improved semantic representations from tree-structured long short-term memory networks. In: Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). 2015, 1556−1566

[26] Raffel C, Ellis D P. Feed-forward networks with attention can solve some long-term memory problems. 2016, arXiv preprint arXiv: 1512.08756

[27] McCulloch W S, Pitts W. A logical calculus of the ideas immanent in nervous activity. The Bulletin of Mathematical Biophysics, 1943, 5( 4): 115–133

[28] Nuñez-Varela A S, Pérez-Gonzalez H G, Martínez-Perez F E, Soubervielle-Montalvo C. Source code metrics: a systematic mapping study. Journal of Systems and Software, 2017, 128: 164–197

[29] Przybyłek A. Where the truth lies: AOP and its impact on software modularity. In: Proceedings of the 14th International Conference on Fundamental Approaches to Software Engineering. 2011, 447−461

[30] Goel B M, Bhatia P K. Analysis of reusability of object-oriented systems using object-oriented metrics. ACM SIGSOFT Software Engineering Notes, 2013, 38( 4): 1–5

[31] Bruntink M, Van Deursen A. An empirical study into class testability. Journal of Systems and Software, 2006, 79( 9): 1219–1232

[32] Poort E R, Martens N, Van De Weerd I, Van Vliet H. How architects see non-functional requirements: beware of modifiability. In: Proceedings of the 18th International Conference on Requirements Engineering: Foundation for Software Quality. 2012, 37−51

[33] Alzahrani M, Alqithami S, Melton A. Using client-based class cohesion metrics to predict class maintainability. In: Proceedings of the 43rd IEEE Annual Computer Software and Applications Conference (COMPSAC). 2019, 72−80

[34] Kanellopoulos Y, Antonellis P, Antoniou D, Makris C, Theodoridis E, Tjortjis C, Tsirakis N. Code quality evaluation methodology using the ISO/IEC 9126 standard. International Journal of Software Engineering & Applications, 2010, 1( 3): 17–36

[35] Malhotra R, Lata K. An empirical study to investigate the impact of data resampling techniques on the performance of class maintainability prediction models. Neurocomputing, 2021, 459: 432–453

[36] Padhy N, Panigrahi R, Neeraja K. Threshold estimation from software metrics by using evolutionary techniques and its proposed algorithms, models. Evolutionary Intelligence, 2019, 14( 2): 315–329

[37] Shatnawi R. Comparison of threshold identification techniques for object-oriented software metrics. IET Software, 2020, 14( 6): 727–738

[38] Ferreira K A M, Bigonha M A S, Bigonha R S, Mendes L F O, Almeida H C. Identifying thresholds for object-oriented software metrics. Journal of Systems and Software, 2012, 85( 2): 244–257

[39] Hofmeister J, Siegmund J, Holt D V. Shorter identifier names take longer to comprehend. In: Proceedings of the 24th IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER). 2017, 217−227

[40] Morales R, Khomh F, Antoniol G. RePOR: mimicking humans on refactoring tasks. Are we there yet? Empirical Software Engineering, 2020, 25( 4): 2960–2996

[41] Hussain Y, Huang Z, Zhou Y. Improving source code suggestion with code embedding and enhanced convolutional long short-term memory. IET Software, 2021, 15( 3): 199–213

[42] Scalabrino S, Bavota G, Vendome C, Linares-Vasquez M, Poshyvanyk D, Oliveto R. Automatically assessing code understandability. IEEE Transactions on Software Engineering, 2021, 47( 3): 595–613

[43] Smith N, Van Bruggen D, Tomassetti F. JavaParser: Visited. Victoria: Leanpub, 2021

[44] Jiang Y, Liu H, Zhu J, Zhang L. Automatic and accurate expansion of abbreviations in parameters. IEEE Transactions on Software Engineering, 2020, 46( 7): 732–747

[45] Butler S. The effect of identifier naming on source code readability and quality. In: Proceedings of the Doctoral Symposium for ESEC/FSE on Doctoral Symposium. 2009, 33−34

[46] Lawrie D, Feild H, Binkley D. Quantifying identifier quality: an analysis of trends. Empirical Software Engineering, 2007, 12( 4): 359–388

[47] ORACLE. Naming Conventions. See oracle-base/articles/misc/naming-conventions website, 2017

[48] Olsson M. Constants. In: Olsson M, ed. Java 17 Quick Syntax Reference. 3rd ed. Berkele: Apress Berkele, 2022, 85−87

[49] Fowler M. Refactoring: Improving the Design of Existing Code. 2nd ed. Reading: Addison-Wesley Professional, 2018

[50] Mi Q, Xiao Y, Cai Z, Jia X. The effectiveness of data augmentation in code readability classification. Information and Software Technology, 2021, 129: 106378

[51] Alon U, Zilberstein M, Levy O, Yahav E. Code2vec: learning distributed representations of code. Proceedings of the ACM on Programming Languages, 2019, 3(POPL): 40

[52] Lee T, Lee J B, In H P. A study of different coding styles affecting code readability. International Journal of Software Engineering and its Applications, 2013, 7( 5): 413–422

[53] Sun H, Wang R, Chen K, Utiyama M, Sumita E, Zhao T. Unsupervised bilingual word embedding agreement for unsupervised neural machine translation. In: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. 2019, 1235−1245

[54] Chidamber S R, Kemerer C F. A metrics suite for object oriented design. IEEE Transactions on Software Engineering, 1994, 20( 6): 476–493

[55] Aniche M. Java code metrics calculator (CK), 2015

[56] Svyatkovskiy A, Deng S K, Fu S, Sundaresan N. IntelliCode compose: code generation using transformer. In: Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering. 2020, 1433−1443

[57] Svyatkovskiy A, Zhao Y, Fu S, Sundaresan N. Pythia: AI-assisted code completion system. In: Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. 2019, 2727−2735

[58] Li Y, Yang Z, Guo Y, Chen X. Humanoid: a deep learning-based approach to automated black-box android app testing. In: Proceedings of the 34th IEEE/ACM International Conference on Automated Software Engineering (ASE). 2019, 1070−1073

[59] Gu X, Zhang H, Zhang D, Kim S. Deep API learning. In: Proceedings of the 24th ACM SIGSOFT International Symposium on Foundations of Software Engineering. 2016, 631−642

[60] Goodfellow I, Bengio Y, Courville A. Deep Learning. Cambridge: MIT Press, 2016

[61] Lever J, Krzywinski M, Altman N. Logistic regression. Nature Methods, 2016, 13( 7): 541–543

[62] Zhang Y, Zhou M, Mockus A, Jin Z. Companies’ participation in OSS development−an empirical study of OpenStack. IEEE Transactions on Software Engineering, 2021, 47( 10): 2242–2259

[63] Amreen S, Mockus A, Zaretzki R, Bogart C, Zhang Y X. ALFAA: active learning fingerprint based anti-aliasing for correcting developer identity errors in version control systems. Empirical Software Engineering, 2020, 25( 2): 1136–1167

[64] El Emam K, Benlarbi S, Goel N, Rai S N. The confounding effect of class size on the validity of object-oriented metrics. IEEE Transactions on Software Engineering, 2001, 27( 7): 630–650

[65] Bengio Y. Practical recommendations for gradient-based training of deep architectures. In: Montavon G, Orr G B, Müller K R, eds. Neural Networks: Tricks of the Trade. Berlin: Springer, 2012, 437−478

[66] Wang W, Li G, Shen S, Xia X, Jin Z. Modular tree network for source code representation learning. ACM Transactions on Software Engineering and Methodology, 2020, 29( 4): 31

[67] Allamanis M, Barr E T, Bird C, Sutton C. Suggesting accurate method and class names. In: Proceedings of the 10th Joint Meeting on Foundations of Software Engineering. 2015, 38−49

[68] Baranasuriya N. Java Coding Standard. See se-education/guides/ conventions/java/index website, 2022

[69] Triguero I, González S, Moyano J M, García S, Alcalá-Fdez J, Luengo J, Fernández A, Del Jesús M J, Sánchez L, Herrera F. KEEL 3.0: an open source software for multi-stage analysis in data mining. International Journal of Computational Intelligence Systems, 2017, 10( 1): 1238–1249

[70] Zerouali A, Mens T. Analyzing the evolution of testing library usage in open source java projects. In: Proceedings of the 24th IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER). 2017, 417−421

RIGHTS & PERMISSIONS

Higher Education Press