Machine vision-based automatic fruit quality detection and grading

Abstract

Artificial intelligence-based automatic systems can reduce time, human error and post-harvest operations. By using such systems, food items can be successfully classified and graded based on defects. For this context, a machine vision system was developed for fruit grading based on defects. The prototype consisted of defective fruit detection and mechanical sorting systems. Image processing algorithms and deep learning frameworks were used for detection of defective fruit. Different image processing algorithms including pre-processing, thresholding, morphological and bitwise operations combined with a deep leaning algorithm, i.e., convolutional neural network (CNN), were applied to fruit images for the detection of defective fruit. The data set used for training CNN model consisted of fruit images collected from a publicly-available data set and captured fruit images: 1799 and 1017 for mangoes and tomatoes, respectively. Subsequent to defective fruit detection, the information obtained was communicated to microcontroller that further actuated the mechanical sorting system accordingly. In addition, the system was evaluated experimentally in terms of detection accuracy, sorting accuracy and computational time. For the image processing algorithms scheme, the detection accuracy for mango and tomato was 89% and 92%, respectively, and for CNN architecture used, the validation accuracy for mangoes and tomatoes was 95% and 94%, respectively.

Graphical abstract

Keywords

Computer and machine vision / convolution neural network / deep learning / defective fruit detection / fruit grading / microcontroller

Highlight

	● A machine vision-based prototype system was developed for fruit grading.
	● Deep learning and image processing algorithms are used for defective fruit detection.
	● The mechanical system is controlled by microcontroller guided by computer vision.
	● Maximum validation accuracies for mangoes and tomatoes were around 94%.

1 Introduction

The global population is increasing exponentially at a rate of nearly 1.09% per year, leading to increasing demand for food and other basic needs. The sufficient and safe access to food is greatly affected by rapid population growth, dwindling resources, climate change, food losses, availability and quality deterioration. Substantial post-harvest losses in current processes are one of the major components of food loss in the agriculture sector. However, the requirement for the food, fiber and other commodities increases constantly with the population increase^[1]. One group food particularly beneficial to humans is fruit and vegetables that provide various vitamins, minerals and antioxidants that are necessary for human health and help to avoid various diseases. These benefits can be best obtained if the fruit is are of the good quality^[2]. Fruit quality is determined by nutrients, composition, contamination, color, flavor, texture, smell, taste and elasticity.

Fruit and vegetables are perishable so their efficient and proper handling is important to avoid damage. The steps and processes involved in the post-harvest of the perishable products, such as fruit, are washing, sorting, grading, storage, packaging and transportation^[3]. Sorting and grading are the most important, difficult and time-consuming steps in the post-harvest chain that are based on product shape, size, color, maturity, freshness and apparent defects^[4]. Sorting of agricultural products is more difficult than other objects because they vary in numerous characteristics including size, quality, yield, appearance, composition, texture, taste, smell and maturity, due to difference in their cultivation, plantation, nutritional value and environmental factors^[5]. Manual inspection and handling used for the food and agricultural products can be damaging and inconsistent due to careless and inexperienced workers. Also, these methods are mainly dependent upon human abilities and often involve human error, and only practicable on small-scale. Manual handling involves several other challenging problems that include higher cost, labor shortage and inefficiency, less accuracy, arduousness, unreliability and time losses^[6]. The use of automatic and semi-automatic systems for the food product handling and post-harvest processes helps to avoid these issues^[7].

The apparent attributes of fruit are important factors for determining their quality^[8]. Color, size, and amount of ethylene produced depict the maturity level of fruit^[9]. These quality determining attributes can be effectively identified by computer vision algorithms. Computer/machine vision-based systems have gained significant attention for fruit quality determination and grading in recent past. These techniques and methods are efficient, fast, consistent, time saving, reliable and cost effective to handle the products according to the market demands^[10]. Also, they reduce labor dependence and availability issues with requirement of less or no expertise once developed, and can be applied on large-scale. These vision-based systems are non-destructive and can determine different external and internal features including shape, size, color, texture, defects, maturity and moisture content^[11,^12]. They employ different image recognition techniques like simple image processing algorithms, pattern recognition, feature extraction methods followed by classification, and other machine/deep learning algorithms^[13,^14]. These vision-based systems mostly use RGB color model in their widespread applications. Other color assessment methods, such as hue, saturation and intensity, can also be used^[15].

Simple image processing algorithms, sensor-based detection, and embedded systems are widely studied for application in post-harvest operations of different kinds of fruit^[16–^20]. Machine learning algorithms are also widely applied in this field for different tasks and kinds of fruit^[21–^26]. There are several reports about deep learning applications for classification and/or grading of different kinds of fruit that used different sensor-based detection and machine/deep learning-based techniques^[27–^31].

In the present research, an automatic, embedded fruit-grading system based on image processing and deep learning frameworks was developed and evaluated. This system consists of two subsystems, i.e., defective fruit detection and mechanical sorting systems. The defective fruit detection system uses image processing and deep learning algorithms. Different image processing algorithms including pre-processing, thresholding, morphological and bitwise operations as well as deep leaning algorithm, i.e., CNN were applied to the acquired fruit images for the detection of fruit defects. The detection results are subsequently communicated to microcontroller which accordingly actuates the mechanical sorting system. Subsequent to the development, the system was evaluated for detection accuracy, sorting accuracy and computational time. The automatic quality determination and grading system developed is relatively cheap, reliable, fast, error-free and accurate compared to existing systems.

2 Related work

In this section, previous related studies are reviewed.

2.1 Image processing and sensor-based detection

Simple image processing algorithms, sensors-based detection, and embedded systems are widely studied for application in post-harvest operations on different kinds of fruit. Nazulan et al.^[16] conducted the research on the inspection and grading of watermelon. They used the sweetness of watermelon as a parameter for detection. For the detection of color and shape of fruit, different image processing algorithms were used. Patil et al.^[17] described the use of computer vision-based technology in the fruit industry. They used the tomato hybrid Rishika 255 for the quality determination process, classifying the fruit two categories based on normal and defective skin attributes. Feature extraction techniques were used for the identification of fruit. Vandana et al.^[18] developed an S71200 PLC-based sorting system. They utilized electromagnetic actuators and a TCS2300 color sensor which automated the sorting system. Items were carried on a conveyor belt where a color sensor was used to determine its color, and the items was classified and move to a predetermined position using three electromagnetic actuators. Wasule and Deshmukh et al.^[19] used image processing algorithms and techniques for sorting and grading tomatoes. The system was developed using Raspberry Pi, computer vision technology, and digital image processing techniques. Nandi et al.^[20] used LabVIEW Real-Time for fruit grading and sorting based on the computer vision technique. A pseudo-median filter was used to remove the noise from the images that provide a better boundary of the fruit. The estimation of characteristics to anticipate the maturity was done by the application of Gaussian model. Rokunuzzaman and Jayasuriya^[32] developed a machine vision-based algorithm for the detection of blossom end rot and different cracks in tomatoes. The sorting decision was made on the basis of rule base method and neural network method. Shilpashree et al.^[33] presented the use of image processing algorithms on Raspberry Pi. The Rudin-Ocher-Fatemi model was used for the removing noise from the images. This model helped to obtain the smooth images and preserve the edges. Mishra et al.^[34] studied the firmness level of pear at various stages of dehydration and ripening of the fruit. They used Vis-NIR spectrometry to predict the firmness of pear. Khan et al.^[35] developed a robotic arm using TCS34725FN color sensor and an Arduino microcontroller to sort the fruit based on color. Santo et al.^[36] developed a prototype to detect the potato quality using machine learning algorithms that differentiated potatoes from carrots. Thumbnail cracks on the potato were detected by using Gaussian blur, canny edge detection, thresholding and contour drawing. Jayasankar et al.^[37] inspected the fruit quality based on shape, size, color and weight. They developed an embedded system prototype that was cost effective. A proximity sensor, load sensor and gas sensor were used in their prototype. Eswaran et al.^[38] designed the system for automatic sorting of eggplants. The system used RGB of the eggplants and extracted the features for their classification into different categories. They have used Raspberry Pi 2 with a camera module and LCD. Color segmentation was done by using a look-up table. The clustering was performed in RGB color space by a C-means algorithm. Dairath et al.^[39] developed an image processing based system for fruit picking and grading. They used an Arduino microcontroller for communication of fruit quality detection system with robotic arm servo motors. To date, standard image processing algorithms scheme including color scheme conversion, masking and dilation processes have been used. The present study extends this to a system which employs deep learning algorithms.

2.2 Machine learning-based detection

Machine learning algorithms are widely applied in this field for different purposes. Tan et al.^[21] developed an intelligent system for blueberry fruit maturity level recognition. They used histogram oriented gradients (HOG) feature vectors technique followed by classification using a support vector machine (SVM) algorithm. Also, they also used K-nearest neighbor (KNN) and template matching with weighted Euclidean distance methods for classification purposes. Abasi et al.^[22] designed and developed a portable optical instrument for real-time non-contact determination of ripeness of apple fruit using a decision tree classification algorithm. Ripeness was detected on the basis of soluble solids, moisture contents and pH. Ghazal et al.^[23] studied classification of fruit using a custom feature extraction method followed by classification algorithms, i.e., KNN, SVM, naive Bayes, decision trees, linear discriminant analysis, and feed forward back propagation neural network. Chopra et al.^[24] developed an intelligent system using ensemble learning and spectrophotometry for apple fruit grading. They used a cloud-computing platform for processing purpose. Chavhan and Rode^[25] designed system based on the size and color of fruit. Background removal was done by HOG and SVM was used for color classification of fruit. Chithra and Henila^[26] extracted features from fruit images for classification using SVM and KNN.

2.3 Deep learning-based detection

Deep learning is widely used in different fields for image recognition tasks^[40–^43]. There are several other existing studies involving deep learning applications for classification and/or grading of different fruit. Sullca et al.^[27] developed a model to detect the damage to the blueberry plant by disease and pests. Their model uses gray scale conversion, binarization, Gaussian filtration, medium filtration and other transformations. They also applied different feature extraction techniques including HOG and LBP on the image to get vectors for the image. Then four algorithms including SVM, neural network, CNN and random forest were used to process these vectors. Kumar et al.^[28] designed a system for automatic grading of mangoes based on size, shape and texture. The images were processed by Raspberry Pi using CNN technique. Moon et al.^[29] developed sweet capsicum development stage (immature, breaking and mature) detection system using ensemble learning. They used CNN and multilayer perceptron (MLP) algorithms. The immature stage of the fruit was further classified into four stages using MLP. Ismail and Malik^[10] trained and tested four deep learning models (DenseNet, ResNet, MobileNetV2, EfficientNet and NASNet) for classification of apples and bananas. Their real-time classification system was developed using Raspberry Pi connected to a camera and display device. Patil et al.^[30] used machine/deep learning algorithms, including CNN, artificial neural network (ANN), and support vector machine (SVM), for developing grading/sorting system for dragon fruit. Their grading system was based on the fruit size, shape, weight, color and presence of disease symptoms using a depth camera which was interfaced with Raspberry Pi. Arakeri and Lakshmana^[44] extracted contrast, energy, correlation and homogeneity features using a gray level co-occurrence matrix for classification of tomatoes by ANN. The features obtained were used as input to a neural network. Melesse et al.^[31] described a machine learning-based digital twin method for detection of banana quality. Their model was trained using SAP technologies. Deep convolutional neural network was used to observe the fruit, with an 99% accuracy of prediction. Yu et al.^[45] developed a robotic gripper for sorting citrus fruit using a CNN-LSTM technique. Defected oranges were detected with an CNN-based detector, and the position of oranges was identified by LSTM-based predictor. Detection accuracy of 94.1% was achieved. Hossain et al.^[46] developed a deep learning-based fruit classification framework. They used a light CNN model and fine-tuned VGG-16 network for classification tasks. Bazame et al.^[47] detected and classified coffee fruit using deep learning networks. They used darknet open source framework and YOLOv3-tiny network. Nithya et al.^[48] employed CNN for mango fruit defect detection. They used publicly-available data set of 800 mango images to train the CNN model for their experiment. The proposed system was tested and found able to classify the mangoes with an accuracy of 98.5%. Mamat et al.^[49] classified oil palm fruit ripeness and identified the cultivar using a you only look once (YOLO) method. They trained the model using 100 oil palm fruit images and 400 RGB images of a fruit cultivar with model accuracy was 98.7% for oil palm fruit and 99.5% for fruit cultivar. Hamid et al.^[50] used MobileNetV2 for classification of 14 types of seed. The accuracy achieved for the training and test data was 98% and 95%, respectively. Ibrahim et al.^[51] used CNN for fruit feature extraction and their classification by botanical family. A data set of 3800 images containing 2660 training and 1440 testing images of fruit was used for the experiment with an accuracy of 99.8%. The model was evaluated by comparing its results using ResNet-20 and SVM.

The summary of the literature on machine/deep learning-based assessment of fruit is given in Tab.1.

3 Materials and methods

The complete workflow of the study is shown in Fig.1. Images of fruit moving on a conveyor was captured with a camera interfaced to a laptop or processor, where two methods, image processing algorithms scheme and CNN-based model, were used to detect defective fruit. The, the result was communicated to microcontroller that actuates the mechanical sorting system. The sorting system consists of servo attached arm that moves the fruit in respective bin.

3.1 Components of the developed system

The automatic system shown in Fig.2 was developed using different materials and components. This system included a conveyor operated by the DC motor, arms attached to the servo motor, image acquisition chamber, infrared sensor, Arduino Uno, speed controller and 5-V single channel relay module. The hardware components used in the system are described below.

3.1.1 Belt conveyor

A belt conveyor was used in order to capture fruit images by a camera installed in the image acquisition chamber. The belt conveyor system consisted of three main components, belt support, drive unit and pulley system. A 12-V DC motor was used to drive the rollers and conveyor belt.

3.1.2 Image acquisition chamber

Image acquisition system acquire images under appropriate conditions using an RGB type USB camera for image capturing and an LED for lighting. A Kodak T130 USB camera was used in the system. The resolutions for image and video capture are 2 and 1.3 megapixels, respectively.

3.1.3 Microcontroller

Arduino Uno was used as a microcontroller in the grading system. A microcontroller is a compact single board integrated circuit designed for embedded system control. Arduino is an open-source microcontroller compact board based on ATmega microchip. Arduino reads the input signals from sensors and provides the output for controlling different devices based on the received inputs.

3.1.4 Servo motor

Servo motor was used for controlling the mechanical sorting arm that is actuated to perform the sorting action. The arm attached to the servo moves fruit to the appropriate bin according to its classification. A servo motor regulates the servo attached arm and is comprised of a DC/AC motor, control unit and a potentiometer. The system of servo motor is a controlled system that works according to the given instructions. Servo rotation is not continuous; they rotate at fixed angles. These angles could be different for different servos, e.g., 0°‒90°, 0°‒180° or 0°‒270°.

3.2 Algorithm development

Two platforms were used to develop the algorithms required for the automatic system: Spyder IDE by Anaconda and Arduino IDE. Python language libraries including NumPy, matplotlib, time, OpenCV, tflearn and PySerial, and Arduino environment extension libraries including servo library were used.

3.3 Image processing and deep learning for defective fruit detection

For defective fruit detection, image processing algorithms and deep learning frameworks were applied in Python. Image processing algorithms including pre-processing, thresholding, morphological and bitwise operations combined with a deep leaning algorithm, i.e., convolutional neural network (CNN), were used to acquire the fruit images. The information obtained, i.e., detection decision, was then communicated to mechanical system through Arduino UNO microcontroller.

The first step in the defective fruit detection system was image acquisition. The camera acts as a sensor that converts the light information into the digital form. An RGB camera (Kodak T130) connected with a laptop was used to capture the fruit image. LED illumination source (12 V) was provided right above the image acquisition chamber to provide better lightening condition and minimum shadow. The fruit images were captured at a resolution of 2 megapixels. The image captured by the camera is in the RGB color model. The images were transferred to the laptop/processor for the processing. The image acquisition process is shown in the Fig.3.

3.3.1 Image processing algorithms scheme

The algorithm scheme for defective fruit detection employed a series of steps to determine fruit quality on the basis of presence of defects as shown in Fig.4.

The steps involved in the image processing algorithms scheme were:

(a) Image pre-processing

Pre-processing of the fruit image included several steps. First, the image was enhanced by using different techniques to highlight the details of the image. These techniques could be image enhancement, restoration, color processing, compression and morphological operations implementation. Image captured by the camera was read in the BGR color space by Open CV library. The image was then converted to three color spaces: grayscale, HSV and RGB. The conversion of RGB image to HSV is a time-consuming step. The Cartesian coordinate system of the RGB image is converted into polar coordinate of HSV image. The images with a uniform background are best suited for this process^[52].

(b) Image segmentation

Segmentation was conducted on the basis of range of pixel values in the HSV color space. It is one of the most important and difficult steps in image processing. It divides the digital image to multiple sections. For image segmentation, different techniques are available including thresholding, edge-based segmentation and region-based segmentation methods. The image was first converted to grayscale image. Then thresholding converted the grayscale image to binary (black/white) image.

(c) Morphological processes

Morphological processes on obtained binary images from above step were then applied. Morphological operations included the mathematical and set operators; dilation and erosion, and union and intersection, respectively^[53]. These operators helped to remove noise, improve image and edge detection.

(d) Contour drawing and percentage calculation

The contours around the fresh and defected fruit parts were detected and drawn using open CV library was used. The areas under each contour were then examined and their ratio was calculated to find out the percentage of defected portion of the fruit. A threshold value was set, and percentage calculated was used for fruit quality detection. The fruit images with a value greater than threshold were deemed defected fruit.

3.3.2 Deep learning-based scheme

Herein, CNN algorithm is used for developing defective fruit detection model. The data set, pre-processing, architecture, and training details are given below.

(a) Image data set

The mango and tomato image data sets used for the training of CNN model consists of publicly-available images ^[54,^55] combined with captured images. The mango data set included 1799 images comprising 950 fresh and 849 rotten mango fruit images, of these 1439 images were used for training and 360 for testing the model. The tomato data set included 1017 images comprising 507 fresh and 512 rotten tomato fruit images, of these 817 images were used for training and 200 for testing.

(b) Data augmentation

The performance of deep learning models is largely dependent on the size of data set. Training of CNN requires a large number of images in the data set for improving the classification ability and to avoid over fitting. To achieve this, data augmentation methods are used. Data augmentation is an approach to expand the available data set by employing a range of techniques, with rotation, cropping, scaling, flipping, noise addition, translation, saturation and contrast used in this study and resulting performance is checked. Based in this, rotation, resizing, blurring and flipping techniques were selected for augmenting the data set according to their performance. Specifically, flipping (horizontal and vertical), 90° rotation (clockwise, counter-clockwise and inverted), rotation (from ‒45° to +45°), cropping (zoom from 0% to 40%) and blurring.

(c) CNN architecture and training

CNN is a deep learning algorithm that is commonly used for image classification tasks. It takes the images as input and performs a range of tasks including feature extraction and classification. CNN network usually consists of different layers including convolutional layer, pooling layer, flattened and fully connected layer. Different CNN architectures were trained and tested for the mango and tomato data sets, and the best architecture selected for the respective fruit. Differing numbers of layers were used in CNN architectures. Changing the number of layers by changing hyper-parameters gave different results. The architectures yielding the best results were selected and used for defective fruit detection. The process and layers used are discussed below.

Images were first fed into the convolutional layer. The convolutional layer is the main block of CNN, which bears most of the computational load. It convolves filters and input images; and results in an output layer called feature map. These layers employed rectified linear (ReLU) activation function. Pooling layer is subsequently used to down sample the output of convolutional layer, i.e., feature map. This helps to reduce the dimension, computation and over fitting. Max pooling was employed in the CNN architecture used. Flattening converts the 2D matrix of feature maps into a single long vector (vertical map). These vectors are finally fed into fully connected layers. The fully connected layers employed ReLU and softmax activation functions to compute the output. The output for each image is generated by normalizing the fully connected layers into a probability distribution, i.e., normalized output. The probability for each element of the vector ranges from 0 to 1 resulting in a probability distribution value of1. In machine learning classification models, different metrics are used to measure the performance. The cross-entropy loss function, used in our study, measures the difference between predicted and actual class of an image.

Tab.2 provides details of final selected CNN architecture for mango and tomato models.

3.4 Actuation of mechanical sorting system

After the defective fruit detection, the decision made was communicated to the microcontroller (Arduino Uno) through Arduino IDE for mechanical grading. Arduino IDE received the commands from Anaconda Spyder IDE through serial communication. The servo attached arm was actuated according to the decision made to grade the fruit based on presence of defects to their respective bins.

4 Results and discussion

4.1 Image processing algorithm scheme

The results of image processing algorithm scheme are shown in the Fig.5, which provides a few representative examples used for the testing of the system. RGB images (after pre-processing/color scheme conversion) of mangoes and tomatoes are shown in the first column. The contours drawn around fresh and defected parts are shown in the subsequent columns.

The RGB images captured from camera were read in BGR format by OpenCV library of the Python. These images were converted to different color models including HSV and grayscale according to the requirements of algorithms. Images were then segmented to differentiate the objects from background. After applying morphological operations, contours around the fresh and defected parts were found and drawn. Subsequent to contour drawing, ratio and percentage of defected portion of the fruit were calculated.

From these results of the application of algorithm scheme, it is evident that the results are affected by the diversity of images obtained, which are primarily dependent on the variation in fruit color, types and illumination. The images (Fig.5) show the variation in color of the fruit samples. These differences contribute to the accuracy and efficiency of the algorithm scheme developed. The algorithm scheme was solely based on the RGB color of the images, so any variations can alter the results.

As mentioned above, the performance of the developed automatic fruit grading system was evaluated based on various parameters. A number of samples containing mangoes and tomatoes were tested and time taken by the individual steps of algorithm was noted. Tab.3 shows the time taken by different operations. The time taken for the mango images was more than tomato images because of the difference in their skin texture and color. The mango skin color was more diverse than tomatoes which required more time for the processing of the images captured.

In addition to the time measurement, detection and sorting accuracies were measured. These accuracies were measured using the following formulas.

(1){\mathrm{A}}_{\mathrm{D}\mathrm{e}\mathrm{t}}=\frac{\mathrm{N}\mathrm{o}.\mathrm{o}\mathrm{f}\mathrm{c}\mathrm{o}\mathrm{r}\mathrm{r}\mathrm{e}\mathrm{c}\mathrm{t}\mathrm{p}\mathrm{r}\mathrm{e}\mathrm{d}\mathrm{i}\mathrm{c}\mathrm{t}\mathrm{i}\mathrm{o}\mathrm{n}\mathrm{s}}{\mathrm{T}\mathrm{o}\mathrm{t}\mathrm{a}\mathrm{l}\mathrm{n}\mathrm{o}.\mathrm{o}\mathrm{f}\mathrm{f}\mathrm{r}\mathrm{u}\mathrm{i}\mathrm{t}\mathrm{s}}\times 100 \text{\%}

(2){\mathrm{A}}_{\mathrm{S}\mathrm{o}\mathrm{r}}=\frac{\mathrm{N}\mathrm{o}.\mathrm{o}\mathrm{f}\mathrm{c}\mathrm{o}\mathrm{r}\mathrm{r}\mathrm{e}\mathrm{c}\mathrm{t}\mathrm{l}\mathrm{y}\mathrm{s}\mathrm{o}\mathrm{r}\mathrm{t}\mathrm{e}\mathrm{d}\mathrm{f}\mathrm{r}\mathrm{u}\mathrm{i}\mathrm{t}\mathrm{s}}{\mathrm{T}\mathrm{o}\mathrm{t}\mathrm{a}\mathrm{l}\mathrm{n}\mathrm{o}.\mathrm{o}\mathrm{f}\mathrm{f}\mathrm{r}\mathrm{u}\mathrm{i}\mathrm{t}\mathrm{s}}\times 100\text{\%}

Tab.4 shows the results of detection and sorting accuracies. The detection accuracy of 89% and 92% was achieved for the mangoes and tomatoes, respectively. The variation in the attributes of fruit affect the detection accuracy. The detection accuracy for mangoes was less than tomato because of the more variation in the skin color of mangoes. The sorting accuracy achieved for the mangoes and tomatoes was 87% and 89%, respectively. The actuation of servo attached arm was delayed a few times due to the interruption in serial communication. Due to this factor, the sorting accuracy achieved is less than detection accuracy.

4.2 CNN architecture

As discussed above, different CNN architectures were tested for mango and tomato data sets and were evaluated in terms of validation accuracy and losses. Details of the results obtained from the few tested CNN architectures for mangoes and tomatoes are shown in Tab.5 and Tab.6, respectively.

The validation accuracy and losses for seven convolutional layers and 0.6 dropout value were found best, i.e., 95% and 0.1541 for mangoes at 100th epoch. The precision, recall, and F1 scores for this model were 94.1%, 95.2% and 94.6%, respectively. This architecture was finally selected for mango quality recognition.

The validation accuracy and losses for five convolutional layers and 0.5 dropout value were found best, i.e., 93.5% and 0.2119 for tomatoes at 100th epoch. The precision, recall and F1 scores for this model were 91.7%, 94.6% and 93.1%, respectively. This architecture was finally selected for the tomato quality recognition task. The deep learning model for tomato quality recognition shows less validation accuracy relatively due to less number of images in the data set. The performance of deep learning models can be further improved by expanding the data set with more collected images. Additionally, transfer learning technique can be used to enhance the model performance.

Overall, the developed system provided high accuracy with less computational power requirements and low-cost hardware. Both the image processing and deep learning frameworks have advantages as well as limitations, and suitability for actual and practical conditions. For example, the use of deep learning-based CNN technique improved the accuracy, but also increases the computational power requirements, complexity, and hardware cost. However, there are several other benefits associated with deep learning, such as universality, end-to-end processing (including feature extraction, classification and cost-effectiveness). In contrast, the image processing methods are not universal and less accurate; but they can be preferable in case of data availability issue, requiring satisfying results at low cost, short training and annotation time availability.

The grading system developed can be used to automate this task field, farm and industry contexts. Also, the developed image processing algorithm scheme and proposed CNN architecture can be used for image recognition purposes in agriculture preforming different operations including object (fruit/vegetables) detection, weed detection and robotic picking.

5 Conclusions and future work

In present study, an embedded system prototype based on artificial (computer/machine) vision was developed for automatic grading of fruit on the basis of defects. This prototype performed the defective fruit detection and mechanical grading of fruit. Image processing algorithms scheme including pre-processing, thresholding, morphological and bitwise operations in combination with a deep learning algorithm CNN was used for defective fruit detection. For image processing, the percentage of defected fruit area was calculated to classify individual fruit; while, for deep learning, the best CNN architecture was selected and trained by using a combination of publicly-available and captured images; 1799 and 1017 images of mangoes and tomatoes, respectively. The decision made was then communicated to the microcontroller that actuated the mechanical grading system. Subsequently, the performance of developed system was evaluated with respect to grading accuracy, detection accuracy, validation accuracy and losses and computational time.

It is concluded from this research that the use of both image processing and CNN model are useful and efficient ways for fruit grading. Both these frameworks are associated with several benefits as well as limitations as discussed above. The detection accuracy achieved for the mangoes and tomatoes by image processing algorithms was 89% and 92% respectively. The proposed CNN architectures resulted in relatively higher accuracies of 95% and 93.5% for mangoes and tomatoes, respectively, compared to image processing scheme. The overall system including defective fruit detection and mechanical grading of the fruit worked successfully. The automatic defective fruit detection and grading system developed is relatively cheaper, reliable, fast, error-free and accurate compared to current systems.

The proposed system can be modified to sort and grade more types and cultivars of fruit based on their apparent quality. Some defects that have the color similar to the fruit skin color were difficult to detect accurately. This issue can be resolved by using larger data set having greater variation. The grading of the fruit can be further enhanced by acquiring full360° images using multiple cameras and roller type conveyors. Images taken from different sides should allow for better defect detection.

References

[1] Wang A, Zhang W, Wei X. A review on weed detection using ground-based machine vision and image processing techniques. Computers and Electronics in Agriculture, 2019, 158: 226–240

[2] Elmasry G, Kamruzzaman M, Sun D W, Allen P. Principles and applications of hyperspectral imaging in quality evaluation of agro-food products: a review. Critical Reviews in Food Science and Nutrition, 2012, 52(11): 999–1023

[3] Pawar S S, Dale M P. Computer vision based fruit detection and sorting system. International Journal of Electrical, Electronics and Computer Systems, 2016, 12–15

[4] Jayas D, Karunakaran C. Machine vision system in postharvest technology. Stewart Postharvest Review, 2005, 1(2): 1–9

[5] Wang F, Zheng J, Tian X, Wang J, Niu L, Feng W. An automatic sorting system for fresh white button mushrooms based on image processing. Computers and Electronics in Agriculture, 2018, 151: 416–425

[6] Arah I K, Ahorbo G K, Anku E K, Kumah E K, Amaglo H. Postharvest handling practices and treatment methods for tomato handlers in developing countries: a mini review. Advances in Agriculture, 2016, 2016: 6436945

[7] Thong N D, Thinh N T, Cong H T. Mango sorting mechanical system uses machine vision and artificial intelligence. IACSIT International Journal of Engineering and Technology, 2019, 11(5): 321–327

[8] Al Ohali Y. Computer vision based date fruit grading system: design and implementation. Journal of King Saud University - Computer and Information Sciences, 2011, 23(1): 29–36

[9] Yossy E H, Pranata J, Wijaya T, Hermawan H, Budiharto W. Mango fruit sortation system using neural network and computer vision. Procedia Computer Science, 2017, 116: 596–603

[10] Ismail N, Malik O A. Real-time visual inspection system for grading fruits using computer vision and deep learning techniques. Information Processing in Agriculture, 2022, 9(1): 24–37

[11] Xie W, Wang F, Yang D. Research on carrot surface defect detection methods based on machine vision. IFAC-PapersOnLine, 2019, 52(30): 24–29

[12] Patria L, Sambas A. Image processing technology for edge detection based on vision and Raspberry Pi. IOP Conference Series. Materials Science and Engineering, 2021, 1115(1): 012044

[13] Ding Y, Lee W S, Li M. Feature extraction of hyperspectral images for detecting immature green citrus fruit. Frontiers of Agricultural Science and Engineering, 2018, 5(4): 475–484

[14] Kim D H, Lee K H, Choi C H, Choi T H, Kim Y J. Development of real-time onion disease monitoring system using image acquisition. Frontiers of Agricultural Science and Engineering, 2018, 5(4): 469–474

[15] Aliteh N A, Minakata K, Tashiro K, Wakiwaka H, Kobayashi K, Nagata H, Misron N. Fruit battery method for oil palm fruit ripeness sensor and comparison with computer vision method. Sensors, 2020, 20(3): 637

[16] Nazulan W N S W, Asnawi A L, Ramli H A M, Jusoh A Z, Ibrahim S N, Azmin N F M. Detection of sweetness level for fruits (watermelon) with machine learning. Kota Kinabalu, Malaysia: 2020 IEEE Conference on Big Data and Analytics (ICBDA), 2020, 79–83

[17] Patil S V, Jadhav V M, Dalvi K K, Kulkarni B P. Fruit quality detection using opencv/python. International Research Journal of Engineering and Technology, 2020, 7(5): 6658–6660

[18] Vandana S, Sai K S S, Rohila P, Manideep V. PLC operated colour based product sorting machine. IOP Conference Series. Materials Science and Engineering, 2021, 1119(1): 012016

[19] Wasule S M, Deshmukh S M. Quality determination and grading of tomatoes using Raspberry Pi. International Journal on Recent and Innovation Trends in Computing and Communication, 2018, 6(7): 86–89

[20] Nandi C S, Tudu B, Koley C. An automated machine vision based system for fruit sorting and grading. 2012 Sixth International Conference on Sensing Technology (ICST), 2012, 195–200

[21] Tan K, Lee W S, Gan H, Wang S. Recognising blueberry fruit of different maturity using histogram oriented gradients and colour features in outdoor scenes. Biosystems Engineering, 2018, 176: 59–72

[22] Abasi S, Minaei S, Jamshidi B, Fathi D. Development of an optical smart portable instrument for fruit quality detection. IEEE Transactions on Instrumentation and Measurement, 2021, 70: 7000109

[23] Ghazal S, Qureshi W S, Khan U S, Iqbal J, Rashid N, Tiwana M I. Analysis of visual features and classifiers for fruit classification problem. Computers and Electronics in Agriculture, 2021, 187: 106267

[24] Chopra H, Singh H, Bamrah M S, Mahbubani F, Verma A, Hooda N, Rana P S, Singla R K, Singh K A. Efficient fruit grading system using spectrophotometry and machine learning approaches. IEEE Sensors Journal, 2021, 21(14): 16162–16169

[25] Chavhan P R, Rode S V. Review on colour based quality analysis of fruits for automatic grading using Raspberry Pi. International Journal of Creative Research Thoughts. Control Engineering, 2018, 6(2): 631–636

[26] Chithra P L, Henila M. Fruits classification using image processing techniques. International Journal on Computer Science and Engineering, 2019, 7(5): 131–135

[27] Sullca C, Molina C, Rodríguez C, Fernández T. Diseases detection in blueberry leaves using computer vision and machine learning techniques. International Journal of Machine Learning, 2019, 9(5): 656–661

[28] Kumar A G S, Aathisha S, Dharani S, Revathi N. Machine vision technique based smart fruit sorter. Iconic Research and Engineering Journals, 2019, 2(9): 166–168

[29] Moon T, Park J, Son J E. Prediction of the fruit development stage of sweet pepper (Capsicum annum Var. annuum) by an ensemble model of convolutional and multilayer perceptron. Biosystems Engineering, 2021, 210: 171–180

[30] Patil P U, Lande S B, Nagalkar V J, Nikam S B, Wakchaure G C. Grading and sorting technique of dragon fruits using machine learning algorithms. Journal of Agriculture and Food Research, 2021, 4: 100118

[31] Melesse T Y, Bollo M, Pasquale V D, Centro F, Riemma S. Machine learning-based digital twin for monitoring fruit quality evolution. Procedia Computer Science, 2022, 200: 13–20

[32] Rokunuzzaman M, Jayasuriya H P. Defects and calyx detection using rule-based and neural network approaches for sorting of tomatoes by Machine Vision. Agricultural Engineering International: CIGR Ejournal, 2013, 15(1): 173–180

[33] Shilpashree K S, Lokesha H, Shivkumar H. Implementation of image processing on Raspberry Pi. International Journal of Advanced Research in Computer and Communication Engineering, 2015, 4(5): 199–202

[34] Mishra P, Brouwer B, Meesters L. Improved understanding and prediction of pear fruit firmness with variation partitioning and sequential multi-block modelling. Chemometrics and Intelligent Laboratory Systems, 2022, 222: 104517

[35] Khan S A, Anika T Z, Sultana N, Hossain F, Uddin M N. Color Sorting Robotic Arm. Dhaka, Bangladesh: 2019 International Conference on Robotics, Electrical and Signal Processing Techniques (ICREST), 2019, 507–510

[36] Santo L A G E, Placido R J D, Linsangan N B. Potato skin defect detection and classification through image processing. Thirteenth International Conference on Graphics and Image Processing (ICGIP 2021), 2022

[37] Jayasankar K, Karthika B, Jeyashree T, Deepalakshmi R, Karthika G. Fruit freshness detection using Raspberry Pi. International Journal of Innovative Research in Applied Sciences and Engineering, 2018, 1(10): 202–208(IJIRASE)

[38] Eswaran S, Sathyanarayanan N, Sathyanarayanan N, Suriyakumaran E, Vigneshguhan E. Automatic sorting machine for brinjal using Raspberry PI. Aegaeum Journal, 2020, 8(4): 1396–1406

[39] Dairath M H, Akram M W, Mehmood M A, Sarwar H U, Akram M Z, Omar M M, Faheem M. Computer vision-based prototype robotic picking cum grading system for fruits. Smart Agricultural Technology, 2023, 4: 100210

[40] Akram M W, Li G, Jin Y, Chen X, Zhu C, Zhao X, Khaliq A, Faheem M, Ahmad A. CNN based automatic detection of photovoltaic cell defects in electroluminescence images. Energy, 2019, 189: 116319

[41] Li K, Jin Y, Akram M W, Han R, Chen J. Facial expression recognition with convolutional neural networks via a new face cropping and rotation strategy. Visual Computer, 2020, 36(2): 391–404

[42] Jin G, Zhu T, Akram M W, Jin Y, Zhu C. An adaptive anti-noise neural network for bearing fault diagnosis under noise and varying load conditions. IEEE Access: Practical Innovations, Open Solutions, 2020, 8: 74793–74807

[43] Akram M W, Li G, Jin Y, Chen X, Zhu C, Ahmad A. Automatic detection of photovoltaic module defects in infrared images with isolated and develop-model transfer deep learning. Solar Energy, 2020, 198: 175–186

[44] Arakeri M P, Lakshmana. Computer vision based fruit grading system for quality evaluation of tomato in agriculture industry. Procedia Computer Science, 2016, 79: 426–433

[45] Yu Y, An X, Lin J, Li S, Chen Y. A vision system based on CNN-LSTM for robotic citrus sorting. Information Processing in Agriculture, 2022 [Published Online]

[46] Hossain M S, Al-Hammadi M, Muhammad G. Automatic fruit classification using deep learning for industrial applications. IEEE Transactions on Industrial Informatics, 2019, 15(2): 1027–1034

[47] Bazame H C, Molin J P, Althoff D, Martello M. Detection, classification, and mapping of coffee fruits during harvest with computer vision. Computers and Electronics in Agriculture, 2021, 183: 106066

[48] Nithya R, Santhi B, Manikandan R, Rahimi M, Gandomi A H. Computer vision system for mango fruit defect detection using deep convolutional neural network. Foods, 2022, 11(21): 3483

[49] Mamat N, Othman M F, Abdulghafor R, Alwan A A, Gulzar Y. Enhancing image annotation technique of fruit classification using a deep learning approach. Sustainability, 2023, 15(2): 901

[50] Hamid Y, Wani S, Soomro A B, Alwan A A, Gulzar Y. Smart seed classification system based on mobileNetV2 architecture. Tabuk, Saudi Arabia: 2022 2nd International Conference on Computing and Information Technology (ICCIT), 2022, 217–222

[51] Ibrahim N M, Gabr D G I, Rahman A, Dash S, Nayyar A. A deep learning approach to intelligent fruit identification and family classification. Multimedia Tools and Applications, 2022, 81(19): 27783–27798

[52] Shaik K B, Ganesan P, Kalist V, Sathish B S, Jenitha J M M. Comparative study of skin color detection and segmentation in HSV and YCbCr color space. Procedia Computer Science, 2015, 57: 41–48

[53] Mohsin H, Abdullah S H. Human face detection using skin color segmentation and morphological operations. Journal of Al-Nisour University Collage, 2018, 7: 63–80

[54] HATCHAT. Tomatoes-for-analysis. Kaggle, 2021. Available at Kaggle website on January 31, 2023

[55] Mukhiddinov M, Muminov A, Cho J. Improved classification approach for fruits and vegetables freshness based on deep learning. Sensors, 2022, 22(21): 8192

RIGHTS & PERMISSIONS

The Author(s) 2023. Published by Higher Education Press. This is an open access article under the CC BY license (http://creativecommons.org/licenses/by/4.0)