Machine vision-based automatic fruit quality detection and grading

Highlights

	● A machine vision-based prototype system was developed for fruit grading.
	● Deep learning and image processing algorithms are used for defective fruit detection.
	● The mechanical system is controlled by microcontroller guided by computer vision.
	● Maximum validation accuracies for mangoes and tomatoes were around 94%.

Abstract

Artificial intelligence-based automatic systems can reduce time, human error and post-harvest operations. By using such systems, food items can be successfully classified and graded based on defects. For this context, a machine vision system was developed for fruit grading based on defects. The prototype consisted of defective fruit detection and mechanical sorting systems. Image processing algorithms and deep learning frameworks were used for detection of defective fruit. Different image processing algorithms including pre-processing, thresholding, morphological and bitwise operations combined with a deep leaning algorithm, i.e., convolutional neural network (CNN), were applied to fruit images for the detection of defective fruit. The data set used for training CNN model consisted of fruit images collected from a publicly-available data set and captured fruit images: 1799 and 1017 for mangoes and tomatoes, respectively. Subsequent to defective fruit detection, the information obtained was communicated to microcontroller that further actuated the mechanical sorting system accordingly. In addition, the system was evaluated experimentally in terms of detection accuracy, sorting accuracy and computational time. For the image processing algorithms scheme, the detection accuracy for mango and tomato was 89% and 92%, respectively, and for CNN architecture used, the validation accuracy for mangoes and tomatoes was 95% and 94%, respectively.

Graphical abstract

Keywords

Computer and machine vision / convolution neural network / deep learning / defective fruit detection / fruit grading / microcontroller

References

[1] Wang A, Zhang W, Wei X. A review on weed detection using ground-based machine vision and image processing techniques. Computers and Electronics in Agriculture, 2019, 158: 226–240

[2] Elmasry G, Kamruzzaman M, Sun D W, Allen P. Principles and applications of hyperspectral imaging in quality evaluation of agro-food products: a review. Critical Reviews in Food Science and Nutrition, 2012, 52(11): 999–1023

[3] Pawar S S, Dale M P. Computer vision based fruit detection and sorting system. International Journal of Electrical, Electronics and Computer Systems, 2016, 12–15

[4] Jayas D, Karunakaran C. Machine vision system in postharvest technology. Stewart Postharvest Review, 2005, 1(2): 1–9

[5] Wang F, Zheng J, Tian X, Wang J, Niu L, Feng W. An automatic sorting system for fresh white button mushrooms based on image processing. Computers and Electronics in Agriculture, 2018, 151: 416–425

[6] Arah I K, Ahorbo G K, Anku E K, Kumah E K, Amaglo H. Postharvest handling practices and treatment methods for tomato handlers in developing countries: a mini review. Advances in Agriculture, 2016, 2016: 6436945

[7] Thong N D, Thinh N T, Cong H T. Mango sorting mechanical system uses machine vision and artificial intelligence. IACSIT International Journal of Engineering and Technology, 2019, 11(5): 321–327

[8] Al Ohali Y. Computer vision based date fruit grading system: design and implementation. Journal of King Saud University - Computer and Information Sciences, 2011, 23(1): 29–36

[9] Yossy E H, Pranata J, Wijaya T, Hermawan H, Budiharto W. Mango fruit sortation system using neural network and computer vision. Procedia Computer Science, 2017, 116: 596–603

[10] Ismail N, Malik O A. Real-time visual inspection system for grading fruits using computer vision and deep learning techniques. Information Processing in Agriculture, 2022, 9(1): 24–37

[11] Xie W, Wang F, Yang D. Research on carrot surface defect detection methods based on machine vision. IFAC-PapersOnLine, 2019, 52(30): 24–29

[12] Patria L, Sambas A. Image processing technology for edge detection based on vision and Raspberry Pi. IOP Conference Series. Materials Science and Engineering, 2021, 1115(1): 012044

[13] Ding Y, Lee W S, Li M. Feature extraction of hyperspectral images for detecting immature green citrus fruit. Frontiers of Agricultural Science and Engineering, 2018, 5(4): 475–484

[14] Kim D H, Lee K H, Choi C H, Choi T H, Kim Y J. Development of real-time onion disease monitoring system using image acquisition. Frontiers of Agricultural Science and Engineering, 2018, 5(4): 469–474

[15] Aliteh N A, Minakata K, Tashiro K, Wakiwaka H, Kobayashi K, Nagata H, Misron N. Fruit battery method for oil palm fruit ripeness sensor and comparison with computer vision method. Sensors, 2020, 20(3): 637

[16] Nazulan W N S W, Asnawi A L, Ramli H A M, Jusoh A Z, Ibrahim S N, Azmin N F M. Detection of sweetness level for fruits (watermelon) with machine learning. Kota Kinabalu, Malaysia: 2020 IEEE Conference on Big Data and Analytics (ICBDA), 2020, 79–83

[17] Patil S V, Jadhav V M, Dalvi K K, Kulkarni B P. Fruit quality detection using opencv/python. International Research Journal of Engineering and Technology, 2020, 7(5): 6658–6660

[18] Vandana S, Sai K S S, Rohila P, Manideep V. PLC operated colour based product sorting machine. IOP Conference Series. Materials Science and Engineering, 2021, 1119(1): 012016

[19] Wasule S M, Deshmukh S M. Quality determination and grading of tomatoes using Raspberry Pi. International Journal on Recent and Innovation Trends in Computing and Communication, 2018, 6(7): 86–89

[20] Nandi C S, Tudu B, Koley C. An automated machine vision based system for fruit sorting and grading. 2012 Sixth International Conference on Sensing Technology (ICST), 2012, 195–200

[21] Tan K, Lee W S, Gan H, Wang S. Recognising blueberry fruit of different maturity using histogram oriented gradients and colour features in outdoor scenes. Biosystems Engineering, 2018, 176: 59–72

[22] Abasi S, Minaei S, Jamshidi B, Fathi D. Development of an optical smart portable instrument for fruit quality detection. IEEE Transactions on Instrumentation and Measurement, 2021, 70: 7000109

[23] Ghazal S, Qureshi W S, Khan U S, Iqbal J, Rashid N, Tiwana M I. Analysis of visual features and classifiers for fruit classification problem. Computers and Electronics in Agriculture, 2021, 187: 106267

[24] Chopra H, Singh H, Bamrah M S, Mahbubani F, Verma A, Hooda N, Rana P S, Singla R K, Singh K A. Efficient fruit grading system using spectrophotometry and machine learning approaches. IEEE Sensors Journal, 2021, 21(14): 16162–16169

[25] Chavhan P R, Rode S V. Review on colour based quality analysis of fruits for automatic grading using Raspberry Pi. International Journal of Creative Research Thoughts. Control Engineering, 2018, 6(2): 631–636

[26] Chithra P L, Henila M. Fruits classification using image processing techniques. International Journal on Computer Science and Engineering, 2019, 7(5): 131–135

[27] Sullca C, Molina C, Rodríguez C, Fernández T. Diseases detection in blueberry leaves using computer vision and machine learning techniques. International Journal of Machine Learning, 2019, 9(5): 656–661

[28] Kumar A G S, Aathisha S, Dharani S, Revathi N. Machine vision technique based smart fruit sorter. Iconic Research and Engineering Journals, 2019, 2(9): 166–168

[29] Moon T, Park J, Son J E. Prediction of the fruit development stage of sweet pepper (Capsicum annum Var. annuum) by an ensemble model of convolutional and multilayer perceptron. Biosystems Engineering, 2021, 210: 171–180

[30] Patil P U, Lande S B, Nagalkar V J, Nikam S B, Wakchaure G C. Grading and sorting technique of dragon fruits using machine learning algorithms. Journal of Agriculture and Food Research, 2021, 4: 100118

[31] Melesse T Y, Bollo M, Pasquale V D, Centro F, Riemma S. Machine learning-based digital twin for monitoring fruit quality evolution. Procedia Computer Science, 2022, 200: 13–20

[32] Rokunuzzaman M, Jayasuriya H P. Defects and calyx detection using rule-based and neural network approaches for sorting of tomatoes by Machine Vision. Agricultural Engineering International: CIGR Ejournal, 2013, 15(1): 173–180

[33] Shilpashree K S, Lokesha H, Shivkumar H. Implementation of image processing on Raspberry Pi. International Journal of Advanced Research in Computer and Communication Engineering, 2015, 4(5): 199–202

[34] Mishra P, Brouwer B, Meesters L. Improved understanding and prediction of pear fruit firmness with variation partitioning and sequential multi-block modelling. Chemometrics and Intelligent Laboratory Systems, 2022, 222: 104517

[35] Khan S A, Anika T Z, Sultana N, Hossain F, Uddin M N. Color Sorting Robotic Arm. Dhaka, Bangladesh: 2019 International Conference on Robotics, Electrical and Signal Processing Techniques (ICREST), 2019, 507–510

[36] Santo L A G E, Placido R J D, Linsangan N B. Potato skin defect detection and classification through image processing. Thirteenth International Conference on Graphics and Image Processing (ICGIP 2021), 2022

[37] Jayasankar K, Karthika B, Jeyashree T, Deepalakshmi R, Karthika G. Fruit freshness detection using Raspberry Pi. International Journal of Innovative Research in Applied Sciences and Engineering, 2018, 1(10): 202–208(IJIRASE)

[38] Eswaran S, Sathyanarayanan N, Sathyanarayanan N, Suriyakumaran E, Vigneshguhan E. Automatic sorting machine for brinjal using Raspberry PI. Aegaeum Journal, 2020, 8(4): 1396–1406

[39] Dairath M H, Akram M W, Mehmood M A, Sarwar H U, Akram M Z, Omar M M, Faheem M. Computer vision-based prototype robotic picking cum grading system for fruits. Smart Agricultural Technology, 2023, 4: 100210

[40] Akram M W, Li G, Jin Y, Chen X, Zhu C, Zhao X, Khaliq A, Faheem M, Ahmad A. CNN based automatic detection of photovoltaic cell defects in electroluminescence images. Energy, 2019, 189: 116319

[41] Li K, Jin Y, Akram M W, Han R, Chen J. Facial expression recognition with convolutional neural networks via a new face cropping and rotation strategy. Visual Computer, 2020, 36(2): 391–404

[42] Jin G, Zhu T, Akram M W, Jin Y, Zhu C. An adaptive anti-noise neural network for bearing fault diagnosis under noise and varying load conditions. IEEE Access: Practical Innovations, Open Solutions, 2020, 8: 74793–74807

[43] Akram M W, Li G, Jin Y, Chen X, Zhu C, Ahmad A. Automatic detection of photovoltaic module defects in infrared images with isolated and develop-model transfer deep learning. Solar Energy, 2020, 198: 175–186

[44] Arakeri M P, Lakshmana. Computer vision based fruit grading system for quality evaluation of tomato in agriculture industry. Procedia Computer Science, 2016, 79: 426–433

[45] Yu Y, An X, Lin J, Li S, Chen Y. A vision system based on CNN-LSTM for robotic citrus sorting. Information Processing in Agriculture, 2022 [Published Online] doi: 10.1016/j.inpa.2022.06.002

[46] Hossain M S, Al-Hammadi M, Muhammad G. Automatic fruit classification using deep learning for industrial applications. IEEE Transactions on Industrial Informatics, 2019, 15(2): 1027–1034

[47] Bazame H C, Molin J P, Althoff D, Martello M. Detection, classification, and mapping of coffee fruits during harvest with computer vision. Computers and Electronics in Agriculture, 2021, 183: 106066

[48] Nithya R, Santhi B, Manikandan R, Rahimi M, Gandomi A H. Computer vision system for mango fruit defect detection using deep convolutional neural network. Foods, 2022, 11(21): 3483

[49] Mamat N, Othman M F, Abdulghafor R, Alwan A A, Gulzar Y. Enhancing image annotation technique of fruit classification using a deep learning approach. Sustainability, 2023, 15(2): 901

[50] Hamid Y, Wani S, Soomro A B, Alwan A A, Gulzar Y. Smart seed classification system based on mobileNetV2 architecture. Tabuk, Saudi Arabia: 2022 2nd International Conference on Computing and Information Technology (ICCIT), 2022, 217–222

[51] Ibrahim N M, Gabr D G I, Rahman A, Dash S, Nayyar A. A deep learning approach to intelligent fruit identification and family classification. Multimedia Tools and Applications, 2022, 81(19): 27783–27798

[52] Shaik K B, Ganesan P, Kalist V, Sathish B S, Jenitha J M M. Comparative study of skin color detection and segmentation in HSV and YCbCr color space. Procedia Computer Science, 2015, 57: 41–48

[53] Mohsin H, Abdullah S H. Human face detection using skin color segmentation and morphological operations. Journal of Al-Nisour University Collage, 2018, 7: 63–80

[54] HATCHAT. Tomatoes-for-analysis. Kaggle, 2021. Available at Kaggle website on January 31, 2023

[55] Mukhiddinov M, Muminov A, Cho J. Improved classification approach for fruits and vegetables freshness based on deep learning. Sensors, 2022, 22(21): 8192

Compliance with ethics guidelines

Amna, Muhammad Waqar Akram, Guiqiang Li, Muhammad Zuhaib Akram, Muhammad Faheem, Muhammad Mubashar Omar, and Muhammad Ghulman Hassan declare that they have no conflicts of interest or financial conflicts to disclose. This article does not contain any studies with human or animal subjects performed by any of the authors.

RIGHTS & PERMISSIONS

The Author(s) 2023. Published by Higher Education Press. This is an open access article under the CC BY license (http://creativecommons.org/licenses/by/4.0)