Introduction
The near infrared (NIR) spectroscopy technique is a powerful tool in the field of quality assessment for agro-products, because it has the advantages of being nondestructive, sensitive, safe and rapid. The first sector of commerce to give it credibility was its application in the grain industry
[1]. Now, livestock products
[2–6], fruit and vegetables
[7–15], marine products
[16] are gradually being evaluated using this technique. The analytical process is illustrated by an example in Fig. 1, including steps of NIR spectra acquisition, reference measurement as well as multivariate calibration and prediction
[7]. Beside determination of multivariate attributes of objects, discrimination of objects from different seasons, cultivars, origins and varieties have also been successfully studied
[17–21]. With the growing attention paid to agro-product security, the NIR spectroscopy technique also has a broad application in identification of adulteration in the food industry
[22]. Since research in the laboratory have been carried out for a long time and is already yielded many satisfying results, a growing range of portable and in-field NIR equipment has come into commercial use in recent years
[23–29]. The ability and feasibility of NIR quality assessment have been improved by optimizing the performance of prediction model.
Although the NIR spectroscopy technique involves reasonably simple steps to analyze or predict the quality of products, there are still more than 40 possible error sources in the whole process of analysis
[30], which will affect the results in various ways. These error sources have been classified here into three types: errors from the external environment, the sample and the instrument. Environmental factors affect both the instrument and the sample, which consequently influences the entire analytical process and results. The sample is considered as the most significant influencing factor in the use of NIR spectroscopy, which can influence the results through the process of choosing representative samples, operations on samples
[30] and biological variations between samples
[18,19]. For different kinds of agro-products, there are various error sources related to the sample. Indeed, with different types of instruments, the accuracy and robustness of the model will differ even though measurement parameters are kept same. It is also important to ensure that the influence of the instrument can be effectively controlled or avoided.
There have been few reviews systematically examining how these factors affect the NIR analysis and how to compensate for them in practical applications. In this paper, recent studies on error source and their influence on the NIR spectroscopy analysis are reviewed, the corresponding solutions and applications are discussed and solutions and recommendations made for solving these problems on the basis of this analysis.
Environmental factors
Higher feasibility and efficiency for measurement of agro-products with the NIR spectroscopy technique could be achieved, with only slight reduction in accuracy, if the robustness of calibration could be improved, especially for on-site applications. However, in the application of NIR spectroscopy it is difficult to keep all measurement conditions and parameters constant and the robustness of the calibration model will be influenced by various external factors, such as temperature, humidity and other uncertain variables. In this section, the effects of these environmental factors on NIR spectra and analysis are explained. In addition, some suggestions for compensating for these effects are proposed and compared.
Temperature
Unavoidable temperature fluctuation may occur because of varying weather conditions or improper conditioning of agro-products after harvest and these can have a crucial influence on NIR analysis. Ambient temperature is important in the whole process of NIR measurement and analysis by affecting the temperature of the instrument, the sample or a sub sample
[30]. From a spectral perspective, the wavelength that is most affected by temperature is the moisture-dominant part. The absorbance value in NIR spectra changes because vibration and rotation of O–H in water molecular are changed due to temperature fluctuations
[30,31]. Thus, objects with higher moisture content will suffer more from temperature fluctuations.
Factors that have an important influence on the instrument, such as temperature fluctuation, can affect both lamp output (intensity and quality) and detector response
[32] (wavelength sensitivity), as well as electronic noise
[33,34]. Any drift of the designed wavelength range in the instrument will be followed by a change to the regression coefficients of the calibration model. Martinsen et al.
[35] studied the effect of drift in wavelength calibration. Although the drift was small (<0.1 nm) over a 12-month period, it was noted that a partial least squares regression (PLSR) model based on short-wavelength NIR could be very sensitive to such fluctuations, e.g., a kiwifruit dry matter (DM) model showed a bias shift of 7% DM for a 1 nm perturbation. However, introducing the small wavelength perturbation into the training set proved to be an effective way to reduce this error to less than 1.7% per nm. When operating an NIR instrument in different places in different ambient temperatures, changes in temperature will lead to spectral variation. Hayes et al.
[34] monitored the spectral variation of quartz halogen lamps due to temperature fluctuations. The lamp output was spectrally stable for the time of the first measurement (10 s), although total illumination was not stable until approximately 40 min from start-up. However, when predictive models were developed using second derivative absorbance data, there was no statistically significant impact in root mean square error of prediction (RMSEP) on time of lamp warm-up (after 10 s). In practical applications of the NIR spectroscopy technique, up to 50°C temperature variation can occur, where the previous reference measurement is no longer suitable for the current conditions. Thus, it is necessary to operate in the same conditions where a reference measurement has been taken in order to get high-accuracy of analytical results. Hence, explanations of the procedure and frequency to conduct reference measurements should be given to ensure and maintain the optimal performance of an NIR measurement system. However, this also limits the versatility of a specific NIR measurement instrument and can be inconvenient for the measurement process.
There have been a large number of studies on the influence of temperature on the NIR spectra of samples and the research on wheat conducted by Williams et al.
[36] provides a representative example. It was shown that when varying sample temperature from −10 to 45°C, the read protein was significantly linearly reduced due to the sensitivity to temperature of the effective wavelength range in the calibration model. The analysis also verified that this reduction was not caused by changes in the moisture content that occurred during grinding but was strictly a result of temperature change, and was totally independent of the algorithm. Yao et al.
[37] collected and analyzed the spectra of watermelon juice sample at nine temperatures, varying from 0 to 40°C at intervals of 5°C to predict soluble solid content (SSC). The result indicated that there was an influence of temperature on NIR spectra in a nonlinear way. The average absorbance shifted with the change of temperature, is shown in Fig. 2.
There have been many studies on compensating for the effect of temperature in calibration models
[38]. For primary applications, Williams et al.
[36] proposed several methods to compensate, including choosing a wavelength insensitive to temperature for calibration, adjusting sample temperature to the calibration temperature and construction of a correction chart. Of these, the third can be considered as the most effective and direct way to solve this problem, although it suffers from a lack of automation and convenience. It is more satisfactory and convenient to utilize mathematical algorithms for compensating this bias automatically. Yao et al.
[37] used discriminant analysis and PLSR to establish calibration models for nine different temperatures. It turned out that the PLSR model established at 20°C performed better than at any other temperature. The result also indicated that the local model was sensitive to fluctuations in temperature. Additionally, the global model showed useful prediction ability, which made temperature fluctuation a negligible interference. Besides, there are many other studies
[37–43] that have proved that the global model can significantly reduce the effect of temperature on NIR measurement. However, the drawback of this approach is that the required data collection is quite a large undertaking because the local temperature model uses a calibration set for a single selected temperature and a validation set for other temperatures, while the global temperature model contains all of the data sets. Based on large scale of data sets, Acharya et al.
[41] designed several methods of population structuring for detecting the dry matter and color of tomatoes with the aim of producing robust models that take account of sample temperature. It was concluded that temperature compensation created by adding spectra of the same set of samples at different temperatures was overwhelmed by continuing addition of 500 spectra at a uniform temperature, resulting in a model that was not robust to temperature. The use of a repeatability file was regarded as the best way for temperature compensation to predict dry matter of fruit, while selection of a wavelength region to avoid water absorption features was recommended for color prediction. Chauchard et al.
[42] categorized these temperature-compensating approaches into two major types depending on whether the temperature is measurable or not. For measurable temperatures, they tested three methods. The first used a spectrum correction while the second and third were based on regression coefficients which vary with temperature fluctuations. Studies where temperatures are non-measurable, have led to robust calibration models and to a self-correcting model, where fruit temperature is estimated using spectral data. The efficient ways to compensate for temperature fluctuation have been explored and these can also be potentially useful to minimize other external parameters in NIR calibration. Also, spectral pretreatment can be adopted to reduce the influence of temperature change in samples. Roger et al.
[43] presented a pretreatment method called external parameter orthogonalization (EPO) which aimed at removing from the calculated space the part mostly influenced by the external parameter variations. This method estimated this corresponding subspace by computing a principal component analysis on a small set of spectra measured on the same objects, while the external parameter was varied. An application for detecting the influence of the fruit temperature on the sugar measurement of intact apples is presented to better illustrate this method. Without any spectral pretreatment, the bias in the prediction of SSC was about 8 °Bx for a temperature variation of 20°C. After EPO pretreatment, the bias was less than 0.3 °Bx for the same temperature change. This approach can also be considered for use in other similar circumstances.
Research on the influence of temperature will lead to more optimized temporal conditions where the measurement should be conducted to attain a more accurate and robust calibration model for the applications in the field of food quality and security, which is similar for other external parameters. For example, De Benedictis & Huck
[44] optimized NIR spectroscopy parameters by varying investigated factors like layer thickness, number of scans and temperature during measurement, in order to obtain optimal response variables in the full factorial design consisting of absorption intensity, signal-to-noise ratio and reproducibility of the spectra. Optimized combinations for liquid milk measurement have been found to be 0.5 mm layer thickness, 64 scans and 25°C ambient temperature.
Humidity
Humidity is one of the environmental factors that influence the water status, which directly affects the absorbance of near infrared spectra. However, there have been few studies on the influence of humidity fluctuations on NIR spectra and analytical model performance. Two studies
[45,46] suggested that humidity made little difference to NIR spectra, so there are only minor improvements to be made by obtaining a better quality assessment model in these cases. Compared to the high moisture of analyzed objects, such as livestock products, fruit and vegetables, dairy products and marine products, the amount of water in the air can be neglected, therefore it has little impact. To determine whether the change in humidity can influence the NIR spectrum, Zhou et al.
[45] established an airtight, humidity-controllable test bed to change the humidity of the environment. Final sample spectra were obtained by removing the background spectra from the samples at 40%, 50%, 60%, 70% and 80% relative humidity (RH). It was not known whether the influences of the sample spectrum and the background were equal, so the trial was divided into two groups: detecting background and sample at each humidity level (Group 1) and detecting background at 40% RH only (Group 2). The result showed that in both Groups 1 and 2, humidity had little influence on NIR spectra. However, humidity fluctuation dominates absorption fluctuation, but temperature and humidity affect refraction fluctuation. If humidity fluctuations are sufficiently strong they can dominate the refraction fluctuation for some infrared frequencies, but not for visible frequencies. Thus it will have little influence on NIR spectra
[46]. There are no specific studies showing the optimal humidity condition for NIR analysis. Thus, more in-depth and appropriate research should be conducted in this area.
Other uncertain factors
Sometimes different working environments will generate problems. For instruments in a laboratory, the influence of external light can be effectively eliminated by ensuring an effective seal in order to ensure stable working conditions. When it comes to the in-field applications, especially with portable NIR equipment, the spectra acquired may be influenced by external light and other uncertain factors. With the development of NIR spectroscopy technique, although slightly better results were obtained under laboratory conditions, the results obtained in the field were also accurate enough to determine some quality attributes of fruit
[23–28], which improved the efficiency of the evaluation process. For example, by comparing spectra data recorded in two different experimental condition: on trees in the field and under laboratory conditions with the same portable NIR machine, Gracia & León
[23] and Leónmoreno
[24] proved the feasibility of use of portable NIR instrument for determination of oil and moisture contents in intact olive fruit. Bessho et al.
[25] also demonstrated that a portable nondestructive measurement system might be a powerful tool to easily estimate fruit quality in the apple canopy. In addition, the external variables affecting online agro-product NIR assessment lines are very complicated and vary under dynamic circumstances. Sun et al.
[29] examined the effect of movement speed of fruit on SSC measurement. The results showed that a speed of 0.3–0.7 m·s
−1 had little effect on spectra and model performance, however these can be considered as relatively slow speeds. At 0.5 m·s
−1, the best model for SSC was a PLSR model established with original spectra, with its coefficient of determination and RMSEP being 0.916 and 0.53, respectively.
Sample factors
Agro-products are object with high variation in their physical attributes and chemical composition. They vary in shape, color, density, storage condition and so on. Most error sources come from the sample itself and the process of dealing with it. In the beginning, sample preparations (e.g., smashing and mixing) were required for the analysis of NIR measurement for agro-product, especially for seeds and grains. Gradually, with the development of NIR noninvasive detection, the need for this was reduced and now there is no need for sample preparation. And the error sources in this process are consequently eliminated. This sector will consider different factors influencing livestock products, fruit and vegetables, which lead to variations in NIR spectra and influence the performance of prediction model.
Livestock products
For livestock products, it is suggested that mincing preparations is the best way to prepare material to analyze meat by NIR spectroscopy technique
[5,47]. In addition, the performance of prediction models depends on the animal species. Comapring the analysis of chicken breast, beef and lamb, chicken breast had the best statistical analytical model for crude protein and moisture, and those for beef and lamb are worse, according to Prevolnik et al.
[5]. However, for fat, beef was better predicted than chicken breast while lamb remained the worst. This is because major constituents are easier to predict than minor compounds and beef obviously has the highest fat content. Even muscle type can have an influence on the result and this may be interpreted as the influence of texture and different models have been prepared that vary several influencing factors including muscle type. According to calibration statistics, the prediction using a common equation for several different muscle types seemed more reliable than the equations obtained with studies of single muscle, but the latter showed less bias. Mcdevitt et al.
[48] showed that NIR spectroscopy was successful able to discriminate ground carcasses from three different genotypes: fast-growing broiler, slow-growing broiler and layer-type chicken. The varied genotypes of chicken determined different contents of fat, crude protein and ash, which explains the variation of NIR spectra. To provide people with the opportunity to buy worthwhile, healthy and safe merchandise, these differences are combined with appropriate algorithms to realize the classification of livestock product quality and identification of adulteration in the industry. Several advanced algorithms have been employed to identify adulteration and sort livestock products, including artificial neural network, discriminant analysis and other techniques
[49–51]. One of the most widespread applications is to monitor and predict the spoilage of livestock products during storage and transportation. Argyri et al.
[50] used a machine learning strategy in the form of a multilayer perceptron neural network to correlate Fourier transform near infrared (FT-NIR) spectral data with beef spoilage during aerobic storage at chilling and abuse temperatures. The result of this work indicated that the biochemical fingerprints during beef spoilage obtained by FT-NIR spectroscopy in combination with the appropriate machine learning strategy have significant potential for rapid assessment of meat spoilage. Chen et al.
[51] demonstrated that the FT-NIR spectroscopy technique combined with a classification algorithm had the potential to determine pork storage time associated with its freshness.
Fruit and vegetables
For fruit and vegetables, there are more influencing factors from the growth stage and postharvest storage period. Samples of various cultivars, orchards, districts and seasons can all have an influence on NIR spectra. Peirs et al.
[18] researched the effect of cultivar, orchard and season. Cultivar and season were responsible for a major part of the spectral variability, whereas the influence of the orchard was low, and only occurred for certain cultivars during specific seasons. Bobelyn et al.
[19] used functional analysis of variance to interpret the variance in spectra with respect to biological variability. The effect of cultivar, storage period, origin and season on the accuracy of NIR calibration models for the SSC and firmness of apple was studied based on a large spectral data set for about 6000 apples of different cultivars and storage periods, and from different origins and seasons, which all proved to be significant. The largest difference in spectra was found around the water absorption peaks (970, 1170 and 1450 nm). External validations using an independent data set showed that the accuracy of the models increased considerably when more variability was included in the calibration data set. Guthrie et al.
[20] found that calibrations for total soluble sugar in intact pineapple were not transferrable between the summer and winter growing seasons. A combined calibration (data of three harvest dates) validated reasonably well against a population set drawn from all harvest dates. However, they came to the conclusion that a lack of robustness of calibration was indicated by poor validation within populations of fruit harvested at different time. León et al.
[21] studied the influence of parent and harvest year on NIR determinations of oil, moisture, oleic acid and linoleic acid contents in intact olive fruit. When samples for each year and female parent were analyzed separately by a PLS method with validation against the other groups, calibration models were accurate enough to predict all constituents in new samples from a different female parents but were not transferable across years. However, a calibration equation of sufficient accuracy was obtained from the combined data sets, which should prove to be a useful tool in olive breeding programs.
Deciding how to choose samples appropriately is the first step in obtaining a robust and representative prediction model
[52]. It was shown that having more variability between samples could, to some extent, improve the robustness of calibration models. Peirs et al.
[18] tested the robustness of the calibration models of SSC with respect to cultivars, orchards and seasons based on external validations. It was found that the accuracy of the models increased considerably when more variability was included in the calibration set. Furthermore, overfitting of the calibration model was avoided. However, adding more data to the calibration set increased the chance of adding atypical data, which can reduce the accuracy of model. Thus, it was suggested that when the effect of a certain factors is known a priori, e.g., cultivar, a specific calibration model should be used. It is important to collect calibration data that are sufficiently representative for future samples to be analyzed with the calibration model developed and to develop simple procedure for model adaptation during application
[19].
Also, a non-homogenized spectral distribution of measured values of objects may make a difference to the analysis for intact fruit and vegetables
[53]. Thus, the selection of measurement position can influence the accuracy and robustness of NIR analytical results. Slaughter et al.
[54] determined SSC of intact tomato using NIR and suggested that when used at a random position along the equator of the fruit, rather than at the blossom-end, the technique had significantly greater accuracy. Thus, it was recommended that the fruit should be oriented to allow measurement at the equator when developing high-speed sorting equipment for SSC. Fan et al.
[55] studied the influence of variation of spectrum measurement position on the NIR analysis of SSC for apples (Fig. 3). This result indicated that the measurement position influenced the prediction accuracy of SSC. Compared with the local position model, the global position model was well suited to controlling the prediction accuracy of the calibration model for SSC with respect to the variation of spectrum measurement position. Next, competitive adaptive reweighted sampling was used for the robust global position model to select the most effective wavelengths. This indicated that a global model established with effective wavelengths achieved more promising results.
Last but not least, the size and texture of different fruit types differ a lot, which will influence the choice of NIR spectra’s transmission mode. Transmittance mode can acquire the most complete internal information of fruit, which will improve the prediction ability of NIR calibration models. Due to limited penetration depth of NIR light, the transmittance mode is more suitable for small fruits, such as jujube and cherry. However, by enhancing the intensity of light source, even watermelon can be analyzed using transmittance mode. On the other hand, for those fruits with a core, reflectance mode will be more suitable because it will carry more information from the part near the surface. Therefore, it is suggested that the transmission path should be well designed according to the features of the fruit sample to be tested.
Instrument and algorithm factors
To highlight the factors that may influence the accuracy and robustness of measurement, spectral pretreatment, detection mode, spectral range, modeling method and instrument all need to be taken into consideration. The recent studies for quality analysis of livestock products (Table 1), and fruit and vegetables (Table 2) are listed here to illustrate these factors.
From the instrument perspective, for different samples the wavelength range and detection mode vary, which is consistent with the previous discussion. When it comes to detection mode, reflectance is the most common spectral acquisition mode in NIR analysis shown in the tables. This may be due to its convenience for configuration and transmittance mode will sometimes lead to overheating of samples. However, the transmittance mode shows better prediction ability than reflectance mode. Interactance mode combines the features of reflectance and transmittance mode, which can also yield satisfying results. The NIR instrumental system normally consists of light source, monochromator, detector, sample compartment and optical accessory
[59]. In recent years, the instrument has become portable and automated. From the tables, it can also be seen that online and portable equipments have shown reasonable prediction ability while laboratory studies yielded better prediction models and results.
From the algorithm perspective,parts of pretreatment, variable selection and modeling are important. There are various spectral pretreatment algorithms, such as Savitzky-Golay (S-G) smoothing, standard normal variable (SNV) transformation and multiplicative scatter correction, all with the same aim of removing useless information in NIR spectra. An S-G filter and SNV are commonly used to preprocess spectra before establishing models. S-G filters can be used to minimize noise effects. SNV can be applied to remove slope variation and correct for scatter effects
[2,60,61]. Variable selection methods are used to make the model more efficient and effective. However, these two steps are not always necessary for prediction. Last but not least, mathematical algorithms, such as partial least squares, multilinear regression and principal component analysis are employed to develop calibration models. Sometimes different algorithms will be combined to improve the performance of a calibration model.
Conclusions
The influencing factors that can affect NIR analysis have been summarized and classified into several types in this review. Some appropriate methods are discussed that can be considered to compensate for these influences in order to improve the robustness and accuracy of NIR prediction model.
Among environment factors, temperature has the most obvious influence on NIR spectra, while humidity has a minor influence. Ambient temperature affects both spectral output of instruments and also the sample. To minimize the influence of temperature, it is suggested to carry out experiments at a controlled temperature. However, for occasions with inevitable temperature fluctuation, global temperature models can be used to compensate for its influence. In current applications, NIR analysis is usually conducted in a specific temperature range according to the guidelines for the conditions where reference measurements are taken.
For livestock products, sample status (being minced or intact), muscle species (texture) and storage time will affect NIR spectra. These differences are combined with appropriate algorithms to realize the classification of livestock product quality and identification of adulteration in the industry. For fruit and vegetables, cultivar, parent, orchard, district, season, storage period, measurement position, size, texture and corresponding transmittance mode are all factors that will influence the performance of NIR spectra and analysis. Therefore, robust models can only be developed with sufficiently representative samples varying in the corresponding variables.
In addition, instrument and algorithm also play an important role in the process of NIR spectroscopy analysis. The choice of instrument and algorithm should suit the features of the samples being tested. There is a trend that the range of portable and online equipment will grow fast, in order to meet the needs of the agro-products industry. Improving model accuracy and robustness when applied to samples that are moving is still a crucial problem to solve.
To conclude, the performance of NIR prediction model can be influenced by many environmental factors that affect the instrument and the sample itself. To improve model performance in NIR analysis, choosing an appropriate instrument, spectral range, modeling method, spectral pretreatment and representative sample is crucial. Sometimes the outcome depends greatly on the experience and knowledge of the operator and the guidelines are highly recommended in order to minimize sources of error for different batches of experiments. Thus, a database for NIR spectra of different objects should be established, based on standard and fixed reference measurement procedures. Calibration models in NIR instruments and systems should be updated regularly and in accordance with the specific operating conditions.
The Author(s) 2019. Published by Higher Education Press. This is an open access article under the CC BY license (http://creativecommons.org/licenses/by/4.0)
PDF
(1546KB)
