Iterative Android automated testing

Abstract

With the benefits of reducing time and workforce, automated testing has been widely used for the quality assurance of mobile applications (APPs). Compared with automated testing, manual testing can achieve higher coverage in complex interactive Activities. And the effectiveness of manual testing is highly dependent on the user operation process (UOP) of experienced testers. Based on the UOP, we propose an iterative Android automated testing (IAAT) method that automatically records, extracts, and integrates UOPs to guide the test logic of the tool across the complex Activity iteratively. The feedback test results can train the UOPs to achieve higher coverage in each iteration. We extracted 50 UOPs and conducted experiments on 10 popular mobile APPs to demonstrate IAAT’s effectiveness compared with Monkey and the initial automated tests. The experimental results show a noticeable improvement in the IAAT compared with the test logic without human knowledge. Under the 60 minutes test time, the average code coverage is improved by 13.98% to 37.83%, higher than the 27.48% of Monkey under the same conditions.

Graphical abstract

Keywords

quality assurance / automated testing / UOP / test coverage

Yi Zhong received the BS degree in Computer Application Technology from Chongqing University of Posts and Telecommunications, China in 2009 and the MS degree in Computer Application Technology from Chongqing University of Posts and Telecommunications, China in 2013. She is working toward the PhD degree in Software Engineering of Nanjing University, China. Her research interests include artificial intelligence and software testing

Mengyu Shi received the BS degree in Computer Science and Technology from Southwest University, China in 2021. She is currently working toward the MS degree in Software Engineering in Nanjing University, China. Her research interest is software testing

Youran Xu received the BS degree in Software Engineering from Soochow University, China, Business College in 2021 and the MS degree in Software Engineering from Nanjing University, China. His research interest is software testing and mobile application testing

Chunrong Fang, the Research Assistant of Software Institute, Nanjing University, China. His teaching include Foundations of Computing Systems(Freshman), Software Engineering and Computing II(Sophomore), Software Engineering and Computing III(Sophomore) and Automation Test(Junior). His research interest is Bigcode Quality and AITesting

Zhenyu Chen, the Full Professor of Software Institute, Nanjing University, China. He is the main teacher of Statistical Methods and Data Analytics and the Software Testing: Methods and Techniques at Nanjing University, China. He has published a total of 86 papers as the first author or co-author. He is the sociate Editor of IEEE Transactions on Reliability. He is also the Contest Co-Chair in China at QRS 2018, ICST 2019, ISSTA 2019. Besides, he is the Industrial Track Co-Chair of SANER 2019, PC member of ISSRE 2018. His research interests include collective intelligence, deep learning testing and optimization, big data quality, and mobile application testing

References

[1] Pecorelli F, Catolino G, Ferrucci F, De Lucia A, Palomba F . Software testing and Android applications: a large-scale empirical study. Empirical Software Engineering, 2022, 27( 2): 31

[2] Peng C, Rajan A, Cai T. CAT: change-focused android GUI testing. In: Proceedings of 2021 IEEE International Conference on Software Maintenance and Evolution (ICSME). 2021, 460–470

[3] Salehnamadi N, Alshayban A, Lin J W, Ahmed I, Branham S, Malek S. Latte: use-case and assistive-service driven automated accessibility testing framework for android. In: Proceedings of 2021 CHI Conference on Human Factors in Computing Systems. 2021, 274

[4] Ravelo-Méndez W, Escobar-Velásquez C, Linares-Vásquez M . Kraken: a framework for enabling multi-device interaction-based testing of Android APPs. Science of Computer Programming, 2021, 206: 102627

[5] Noh M J, Lee K T . An analysis of the relationship between quality and user acceptance in smartphone APPs. Information Systems and e-Business Management, 2016, 14( 2): 273–291

[6] Sun S, Fu X, Ruan H, Du X, Luo B, Guizani M . Real-time behavior analysis and identification for android application. IEEE Access, 2018, 6: 38041–38051

[7] Amalfitano D, Fasolino A R, Tramontana P, De Carmine S, Memon A M. Using GUI ripping for automated testing of Android applications. In: Proceedings of the 27th IEEE/ACM International Conference on Automated Software Engineering. 2012, 258–261

[8] Huang R, Zhang Q, Towey D, Sun W, Chen J . Regression test case prioritization by code combinations coverage. Journal of Systems and Software, 2020, 169: 110712

[9] Cai G, Su Q, Hu Z . Automated test case generation for path coverage by using grey prediction evolution algorithm with improved scatter search strategy. Engineering Applications of Artificial Intelligence, 2021, 106: 104454

[10] Liu Z, Chen C, Wang J, Huang Y, Hu J, Wang Q. Guided bug crush: assist manual GUI testing of android APPs via hint moves. In: Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems. 2022, 557

[11] Yasin H N, Ab Hamid S H, Yusof R J R . DroidbotX: test case generation tool for android applications using Q-learning. Symmetry, 2021, 13( 2): 310

[12] Li L, Bissyandé T F, Papadakis M, Rasthofer S, Bartel A, Octeau D, Klein J, Traon L . Static analysis of android APPs: a systematic literature review. Information and Software Technology, 2017, 88: 67–95

[13] Kong P, Li L, Gao J, Liu K, Bissyandé T F, Klein J . Automated testing of android APPs: a systematic literature review. IEEE Transactions on Reliability, 2019, 68( 1): 45–66

[14] Méndez-Porras A, Quesada-López C, Jenkins M. Automated testing of mobile applications: a systematic map and review. In: Proceedings of the XVIII IberoAmerican Conference on Software Engineering. 2015, 195

[15] Pilgun A, Gadyatskaya O, Zhauniarovich Y, Dashevskyi S, Kushniarou A, Mauw S . Fine-grained code coverage measurement in automated black-box android testing. ACM Transactions on Software Engineering and Methodology, 2020, 29( 4): 23

[16] Liu S. Improvement and implementation of android Robotium automated testing framework system. Southeast University, Dissertation, 2017

[17] Geng Z. Study and improvement of android automatic testing. Beijing University of Posts and Telecommunications, Dissertation, 2017

[18] Choudhary S R, Gorla A, Orso A. Automated test input generation for android: are we there yet? (E). In: Proceedings of the 30th IEEE/ACM International Conference on Automated Software Engineering. 2015, 429–440

[19] Mirzaei N, Garcia J, Bagheri H, Sadeghi A, Malek S. Reducing combinatorics in GUI testing of android applications. In: Proceedings of the 38th IEEE/ACM International Conference on Software Engineering. 2016, 559–570

[20] Hu Y, Neamtiu I, Alavi A. Automatically verifying and reproducing event-based races in Android APPs. In: Proceedings of the 25th International Symposium on Software Testing and Analysis. 2016, 377–388

[21] Clapp L, Bastani O, Anand S, Aiken A. Minimizing GUI event traces. In: Proceedings of the 24th ACM SIGSOFT International Symposium on Foundations of Software Engineering. 2016, 422–434

[22] Heiskanen H, Maunumaa M, Katara M. A test process improvement model for automated test generation. In: Proceedings of the 13th International Conference on Product-Focused Software Process Improvement. 2012, 17–31

[23] Yu S, Fang C, Feng Y, Zhao W, Chen Z. LIRAT: layout and image recognition driving automated mobile testing of cross-platform. In: Proceedings of the 34th IEEE/ACM International Conference on Automated Software Engineering. 2019, 1066–1069

[24] Grano G, Ciurumelea A, Panichella S, Palomba F, Gall H C. Exploring the integration of user feedback in automated testing of Android applications. In: Proceedings of the 25th IEEE International Conference on Software Analysis, Evolution and Reengineering. 2018, 72–83

[25] Gu Y, Shi J L. Generality for Technology of Software Testing. Beijing: Tsinghua University Press, 2004

[26] Mahmood R, Mirzaei N, Malek S. EvoDroid: segmented evolutionary testing of Android APPs. In: Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering. 2014, 599–609

[27] Su T, Meng G, Chen Y, Wu K, Yang W, Yao Y, Pu G, Liu Y, Su Z. Guided, stochastic model-based GUI testing of Android APPs. In: Proceedings of the 11th Joint Meeting on Foundations of Software Engineering. 2017, 245−256

[28] Mao K, Harman M, Jia Y. Sapienz: multi-objective automated testing for Android applications. In: Proceedings of the 25th International Symposium on Software Testing and Analysis. 2016, 94−105

[29] Behrang F, Orso A. AppTestMigrator: a tool for automated test migration for Android APPs. In: Proceedings of the 42nd IEEE/ACM International Conference on Software Engineering: Companion Proceedings. 2020, 17−20

[30] Chen S, Fan L, Chen C, Su T, Li W, Liu Y, Xu L. StoryDroid: automated generation of storyboard for android APPs. In: Proceedings of the 41st IEEE/ACM International Conference on Software Engineering. 2019, 596−607

[31] Fan L, Su T, Chen S, Meng G, Liu Y, Xu L, Pu G. Efficiently manifesting asynchronous programming errors in Android APPs. In: Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering. 2018, 486−497

[32] Pan M, Huang A, Wang G, Zhang T, Li X. Reinforcement learning based curiosity-driven testing of Android applications. In: Proceedings of the 29th ACM SIGSOFT International Symposium on Software Testing and Analysis. 2020, 153−164

[33] Dong Z, Böhme M, Cojocaru L, Roychoudhury A. Time-travel testing of android APPs. In: Proceedings of the 42nd IEEE/ACM International Conference on Software Engineering. 2020, 481−492

[34] Zhang X, Chen Z, Fang C, Liu Z. Guiding the crowds for Android testing. In: Proceedings of the 38th International Conference on Software Engineering Companion. 2016, 752−753

[35] Meng C. A research on android test automation technology based on dependency injection. Nanjing University, Dissertation, 2017

[36] Mao K, Harman M, Jia Y. Crowd intelligence enhances automated mobile testing. In: Proceedings of the 32nd IEEE/ACM International Conference on Automated Software Engineering. 2017, 16−26

Acknowledgements

This work was supported in part by the National Natural Science Foundation of China (Grant No. 62141215); the National Key R&D Program of China: R&D and Application of Integrated Crowdsourcing Test Service Platform for Information Products and Technology Services (2018YFB1403400); and the Science, Technology and Innovation Commission of Shenzhen Municipality (CJGJZD20200617103001003).

RIGHTS & PERMISSIONS

2023 Higher Education Press