Prediction of flowering time based on multimodal fusion and temporal meteorological data in rapeseed

Abstract

Accurately predicting the days to flowering for individual rapeseed plants is crucial for supporting operations such as emasculation, bagging, and pollination. However, traditional phenological prediction methods have rarely been applied at the individual plant level. To address this challenge, this study proposes a multimodal deep learning approach that integrates RGB images with future time-series meteorological data. First, the YOLOv8s-seg model segments rapeseed flower regions, reducing interference from complex field backgrounds, and data augmentation enhances data diversity. In the visual branch, five convolutional neural networks—MobileNet V2, MobileNet V3, ResNet18, EfficientNet-B0, and DenseNet 121—are compared. Results show that DenseNet 121 achieves the best balance between prediction accuracy and model complexity, and is selected as the visual backbone network. The meteorological branch uses a long short-term memory model to extract features from future time-series meteorological data, which are then fused with visual features via a multilayer perceptron. To address large-scale differences among meteorological variables, Z-score normalization is used to normalize the meteorological data. Experimental results show that the proposed model performs best on the test set, with a coefficient of determination of 0.8979, a root mean square error of 2.15 days, a mean absolute error of 1.51 days, and a ratio of performance to deviation of 3.13. Therefore, this study provides an effective method for predicting days to flowering in individual rapeseed plants, with potential applications in coordinating flowering periods between parental lines and supporting breeding decision-making.

Graphical abstract

Keywords

Rapeseed / flowering time prediction / image analysis / deep learning / future time-series meteorological data

Highlight

	● Multimodal RGB and future weather time-series fusion predicts rapeseed flowering time.
	● YOLOv8s-seg segmented corolla to remove field background, augmentation improved robustness.
	● DenseNet121 gave the best accuracy-complexity balance among five CNN backbones.
	● LSTM extracted future 15-day weather time-series, Z-score normalization improved performance.
	● Ablation shows future weather data, LSTM, and normalization work synergistically.

1 Introduction

Rapeseed (Brassica napus L.) is one of the world’s four major oilseed crops and plays a significant role in ensuring oil supply and promoting livestock development as a primary source of edible vegetable oil and plant protein. Hybrid vigor, an important theory in modern breeding, can significantly enhance agronomic traits such as yield and stress resistance in self-pollinated crops like rapeseed, increasing yields by 20%–50%^[1]. Hybrid breeding relies on cytoplasmic male sterility^[2], requiring sterile lines (female parents) and restorer lines (male parents) to be grown separately and isolated, with hybrid seeds obtained through natural or artificial pollination. In practice, the success of artificial hybridization depends heavily on synchronizing flowering between parental lines. Accurate prediction of flowering time for individual rapeseed plants is therefore essential for formulating effective hybridization plans, coordinating parental flowering periods, and improving breeding efficiency.

For predicting the days remaining until crop phenological stages, traditional approaches mainly rely on manual field observations and empirical estimation. For example, breeders usually estimate time to flowering based on morphological traits such as flower bud expansion and inflorescence elongation^[3], combined with their field experience. However, in breeding populations, different genotypes exhibit substantial variations in developmental rates and inflorescence morphology, making manual estimation highly subjective and prone to considerable errors. Therefore, developing objective, non-destructive, and rapid approaches to predict crop phenological stages are essential to improve breeding management efficiency under precision agriculture frameworks. Currently, studies on crop phenological prediction can be broadly categorized into four groups: methods based on remote sensing or near-ground imaging data, methods driven by meteorological information, multi-source approaches integrating remote sensing data with thermal accumulation models, and multimodal deep learning approaches.

With the development of sensor technologies, non-destructive approaches based on remote sensing, ground-based imaging, and spectral data have been increasingly applied to predict crop phenological timing. These methods typically acquire single-date images or multi-temporal image sequences using unmanned aerial vehicles (UAVs) or ground-based cameras, and employ deep learning models to extract phenotypic features for predicting crop developmental stages. Teodoro et al.^[4] used a UAV-mounted multispectral sensor to acquire canopy spectral information at 60 days after soybean emergence and combined spectral bands with a deep learning model to predict soybean days to maturity. The model achieved a Pearson correlation coefficient of 0.66, demonstrating the potential of remote sensing spectral information for quantitatively predicting crop developmental status. Huang et al.^[5] continuously collected time-series images of rice from transplanting to maturity using ground-based cameras and integrated deep convolutional neural network and long short-term memory (LSTM) networks to predict the time required for rice to reach subsequent phenological stages. However, these approaches mainly rely on phenotypic information and often overlook environmental drivers, such as temperature, solar radiation, and precipitation, that regulate crop growth and development.

Weather-based phenological prediction methods model crop phenology using the thermal time model, the nonlinear product model, and the CW3 model. The thermal time model calculates growing degree days (GDD) by assuming a fixed base temperature^[6], which describes the crop development process. However, the research by M.R. Salazar-Gutierrez et al.^[7] indicates that the base temperatures for winter wheat at different growth stages (from sowing to heading, from heading to maturity, and from sowing to maturity) are 3.1–8.1 °C, 10.6–18.4 °C, and 1.6–8.4 °C, respectively, and there are significant differences among different varieties. These findings suggest that base temperature is highly dependent on both developmental stage and cultivar. Moreover, this model considers only temperature and ignores other meteorological variables and environmental differences, so its generalizability is limited. To enhance model performance, Xue et al.^[8] compared the predictive performance of the nonlinear multiplication model based on Wang & Engel^[9] and CERES-Wheat V3.0^[10,^11] in phenological prediction, and in the prediction of the booting to maturity stage of three winter wheat varieties, the root mean square error (RMSE) of this model was 2.9–4.1 days, significantly better than 4.8–5.9 days for CERES-Wheat V3.0. However, these three biological meteorological methods rely on large amounts of historical phenological data and variety-specific parameters, and data availability is limited and acquisition costs are high in cases of frequent variety updates.

Remote sensing integrated with temperature accumulation models has been widely employed to predict crop phenological stages. For instance, Dížková et al.^[6] derived the start of season for wheat using MODIS-derived vegetation indices, specifically the normalized difference vegetation index and the two-band enhanced vegetation index, and subsequently applied a temperature accumulation model to forecast key developmental stages, including jointing, flowering, and physiological maturity. The model yielded an average RMSE ranging from 6.2 to 15.2 days. Similarly, Liu et al.^[12] coupled MODIS time-series data with a temperature-based phenological model to estimate winter wheat maturity, achieving an RMSE of 2.69 days and a coefficient of determination (R²) of 0.63. Nevertheless, this approach is inherently constrained by its dependence on satellite remote sensing data, which typically have coarse spatial resolution and are prone to signal degradation from cloud cover and precipitation, limitations that impact its applicability to fine-scale, individual-plant-level phenological monitoring.

With advances in deep learning models for feature extraction and multi-source information fusion, multimodal deep learning approaches have been increasingly applied to crop phenological prediction. Through multilayer architectures, deep learning models can automatically learn complex patterns from large-scale datasets via backpropagation and iterative parameter optimization^[13]. This capability enables the extraction of nonlinear relationships between visual phenotypic features and meteorological variables, providing an effective framework for predicting the time remaining until specific phenological stages. For example, Shen et al.^[14] developed a GSP-AI dual-task multimodal deep learning network that integrated meteorological data with multi-scale image information, achieving an accuracy of 93.4% for wheat growth stage recognition and an RMSE of 4.7 days for flowering prediction. Xie et al.^[15] combined historical and future meteorological data with image information to develop a small-sample learning model for predicting flowering in individual wheat stems, achieving an F1-score above 0.8 in the binary classification task. These studies demonstrate that multimodal deep learning approaches can effectively integrate visual phenotypic information and meteorological drivers, providing a more robust and reliable solution for crop phenological prediction.

However, existing studies still have several limitations. First, most studies focus on phenological analysis at the population scale, while investigations at the individual-plant level remain limited. Second, existing multimodal models generally rely on long-term historical meteorological sequences as inputs. Future meteorological conditions may provide relevant signals for predicting imminent flowering events. Therefore, incorporating future meteorological time-series information is expected to improve flowering-time prediction accuracy.

Most existing methods are designed for population or regional scales, rely primarily on historical meteorological data or single-source phenotypic data, and focus on phenological stage recognition or flowering date estimation. In contrast, research on predicting the remaining days to flowering at the individual-plant level by combining current visual phenotypes and future meteorological data remains relatively limited. For the prediction task in this study, the time required for an individual rapeseed plant to reach flowering from its current stage depends on two aspects: first, the current morphological development status of the plant, such as the degree of flower bud differentiation, which can be characterized by phenotypic features reflected in RGB images; and second, the meteorological conditions on and after the observation date, as future weather may affect subsequent developmental rates. Therefore, incorporating future meteorological time-series information alongside current phenotypic information is expected to improve flowering-time prediction accuracy for individual plants. Accordingly, this study aims to address a core question: how to accurately predict the continuous number of days remaining until flowering for individual rapeseed plants based on their current visual phenotypic status and future environmental conditions? It should be emphasized that our task is fundamentally different from conventional phenological stage recognition. Phenological stage recognition assigns a discrete categorical label to the current developmental stage, whereas our task predicts the number of days from the current observation to the flowering day—A continuous regression target. This information can directly support breeding decisions by enabling breeders to compare flowering dates across different genotypes and adjust field management accordingly.

To address the challenge of predicting days to flowering at the individual plant level in oilseed rape, this study aims to: (1) Construct a single-plant oilseed rape flowering prediction dataset integrating image and meteorological information, and employ image segmentation and data augmentation strategies to reduce the influence of complex field backgrounds and variations in image acquisition conditions on model performance. (2) Evaluate different visual feature extraction networks for oilseed rape flowering prediction and identify an optimal backbone network with high prediction accuracy and generalization capability. (3) Develop a multimodal deep learning model by integrating visual phenotypic features and meteorological information to investigate the contribution of environmental factors to predicting days to flowering.

2 Materials and methods

2.1 Experimental design

Three independent field experiments were conducted at three ecological sites in Sichuan Province from 2024 to 2025. The experiments included diverse growth environments and used various rapeseed materials (inbred lines, restorer lines, and hybrid varieties) to represent different genotypes and environmental conditions. In each experiment, multiple rapeseed plants were randomly selected and individually tagged for continuous observation. Image acquisition and phenological recording were conducted throughout the pre-flowering developmental stages. The detailed information on the study sites, varieties, and image acquisition dates is summarized in Table 1.

Experiment 1 was conducted at the Agricultural and Forestry Science and Technology Innovation Base in Yangma Town, Chengdu City (30°40′53″N, 103°43′59″E) in spring 2024. The experimental materials were male sterile lines, and a total of 5 genotypes were included. Each plot was marked, and observations were made on 20 plants.

Experiment 2 was conducted at the Maerkang Experimental Station of Chengdu Agricultural and Forestry Sciences Academy (31°53′51″N, 102°07′23″E) in summer 2024. The experimental materials were hybrid varieties, consisting of 3 genotypes. Each plot was marked, and observations were made on 20 plants in each plot.

Experiment 3 was conducted at the Main Experimental Station of Chengdu Agricultural and Forestry Research Institute, Wenjiang (30°42'26"N, 103°51'38"E) in spring 2025. The experimental materials included restorer and male sterile lines, totaling 10 genotypes. Each plot was marked with 10 plants, among which the number of plants for some genotypes was 30.

2.2 Image acquisition

The oilseed plant consisted of a main inflorescence and multiple branch inflorescences. The main inflorescence of each individual plant was selected as the research object. This approach enabled precise recording of the flowering process of each plant, which facilitated flowering time prediction. Once flower buds became visible, image data collection began. During this stage, the inflorescences elongated, and the flower buds gradually expanded. The termination time of data collection for each plant was determined by its developmental stage. Once the plant entered the flowering stage (BBCH60)^[3], the date was recorded, and subsequent image collection was terminated. In this experiment, the iPhone 12 Pro Max was used to capture images of the labeled plants. Data collection was conducted under natural light conditions without any artificial light sources. During image acquisition, it was ensured that the entire plant canopy was fully captured without any obstructions (Fig. 1). One image was taken for each plant at each time point, and the images were saved in JPEG format. The original image resolution was 3024 × 4032 pixels. Data collection was uniformly scheduled between 9:00 and 16:00 to reduce the influence of lighting variation on image quality.

2.3 Definition of regression labels

To build the regression task for predicting the flowering period, we defined a continuous numerical label for each image representing the time interval (days between the image date and the plant’s actual flowering date. The specific steps are as follows: Firstly, through field observation, the actual flowering date of each individual plant was recorded, and this date served as the true value of the actual flowering date for that plant. Then, for each image captured of a single plant before it flowered, the difference between the image capture date and flowering date was calculated.

(1)\text{Label}={\text{Date}}_{\text{flowering}}-{\text{Date}}_{\text{capture}}

As shown in Eq. (1), this difference represented the number of days until flowering. Through the aforementioned method, an independent regression label was assigned to each image, which is used to learn the mapping from multimodal data to the remaining days until flowering from the supervised model.

2.4 Dataset construction

2.4.1 Image Segmentation

To eliminate the interference of complex backgrounds (such as leaves, soil, etc.) on the prediction task of remaining flowering days, this experiment adopted the YOLOv8s-seg model to segment the corolla area in the original images^[16]. 370 original images were randomly selected (covering different varieties and spanning from the start of image acquisition to the day before flowering) from the images collected in the Yangma and Maerkang areas, and then randomly divided into a training set and a validation set at a ratio of 8:2. Since YOLOv8s-seg was used as an auxiliary preprocessing module to generate flower-region images, rather than the final prediction model, an independent test set was not established for the segmentation model. After the model was trained, the optimal segmentation model was applied to the original images, and 125 images with abnormal segmentation results were removed, finally obtaining 3439 images.

2.4.2 Image enhancement

To expand the dataset, enhance the model’s robustness to changes in shooting direction, and prevent overfitting, data augmentation was performed on the segmented image data: for each original segmented image, in addition to retaining the original image, one of the three operations—“horizontal flip”, “vertical flip”, or “horizontal and vertical flip”—was randomly selected to generate a corresponding augmented image. Ultimately, this experiment obtained 6878 images as the dataset for the prediction model of remaining flowering days in rapeseed.

2.4.3 Meteorological data acquisition

Since the flowering time of rapeseed is closely related to environmental factors, the Open-Meteo open-source meteorological application programming interface (API) was used to obtain the weather data for each shooting location on the current day and the next 15 days. Specifically, we used the forecast API endpoint, which provides genuine future forecasts rather than historical reanalysis or observational data. This API integrates advanced local (approximately 1 km resolution) and global (approximately 11 km resolution) numerical weather prediction models from multiple national meteorological service agencies (e.g., DWD, ECMWF, MeteoFrance, etc.), and provided high-spatial-temporal resolution meteorological data worldwide.

Meteorological data acquisition was strictly synchronized with image acquisition time and spatial location. For each oilseed rape plant, meteorological data were obtained as 16-day daily forecast data (from the acquisition day through the next 15 days) based on the image collection date, as shown in Fig. 2. For multiple images of different plants collected at the same location on the same day, they were all associated with the same set of meteorological data. The data acquisition process was implemented through Python programming by calling APIs. For each experimental site, API requests were constructed using its latitude and longitude coordinates and the required date range, and the required variables were obtained in batches. The meteorological variables and their descriptions are listed in Table 2. After obtaining the relevant variables, this study normalized all meteorological data using the Z-score method to remove the influence of different scales and units.

2.5 Deep learning framework and model architectures

2.5.1 Deep learning framework

This study developed a multimodal deep learning framework that integrates visual information with future meteorological time-series data to predict the remaining days until flowering for individual oilseed rape plants. As illustrated in Fig. 3, the proposed framework consists of three components: visual feature extraction, meteorological temporal feature extraction, and multimodal feature fusion for regression prediction. First, segmented rapeseed inflorescence images are processed by a CNN to extract high-level visual features. Meanwhile, an LSTM network models’ meteorological conditions from the image acquisition day and the subsequent 15 days, enabling extraction of temporal environmental features associated with flowering development. The visual and meteorological features are then fused and fed into a multilayer perceptron (MLP) to predict the remaining days until flowering. To identify the most suitable visual backbone for this task, five representative CNN architectures, including MobileNet V2^[17], MobileNet V3^[18], ResNet 18^[19], EfficientNet-B0^[20], and DenseNet 121^[21], were comparatively evaluated.

2.5.2 DenseNet-LSTM network architecture

DenseNet121 is a convolutional neural network architecture proposed at CVPR 2017. Through its dense connectivity mechanism, DenseNet effectively alleviates the vanishing gradient problem and improves feature reuse in deep networks^[21]. In this study, DenseNet 121 was selected as the visual backbone network for feature extraction due to its efficient feature reuse capability and favorable trade-off between representation capacity and computational complexity. Before training, the original classification layer of DenseNet 121 was removed, allowing the network to serve as a feature extractor for learning high-level semantic representations from segmented rapeseed inflorescence images. The input images were uniformly resized to 224 × 224 pixels, and the global average pooling layer output was used as the visual representation. Consequently, we obtained a 1024-dimensional image feature vector.

To model the temporal characteristics of future meteorological information, an LSTM network was employed, which is characterized by its gating mechanism and ability to capture temporal dependencies^[22]. The meteorological time-series data of each sample were input into the LSTM network to learn temporal variation patterns, and the final hidden state was extracted as a 64-dimensional meteorological feature representation. During the multimodal feature fusion stage, the visual and meteorological feature vectors were concatenated along the feature dimension to form a 1088-dimensional joint representation. This fused feature vector was subsequently fed into a two-layer MLP regression head. The first fully connected layer reduced the feature dimension from 1088 to 128, followed by a rectified linear unit activation function and dropout regularization (dropout rate = 0.2). Finally, the second fully connected layer mapped the 128-dimensional representation to a single output neuron, corresponding to the predicted remaining days until flowering.

2.6 Model training strategies and evaluation metrics

The training process was conducted on a mobile workstation equipped with an NVIDIA RTX 2000 Ada GPU and an Intel (R) Core (TM) i7-14700HX CPU. The framework was PyTorch 2.5.1, with CUDA 12.1 and cuDNN 9.1.0 used for accelerated computation. During training, the dataset was randomly split into training, validation, and test sets in proportions of 80%, 10%, and 10%, respectively, ensuring that the original images and their flipped augmented counterparts were assigned to the same subset.

The Adam optimizer was used for model training, with an initial learning rate of 1×10⁻³, a batch size of 32, a weight decay coefficient of 1×10⁻⁴, and a total of 150 training epochs. EfficientNet-B0, MobileNet V2, MobileNet V3, and ResNet-18 were included as baseline models. To ensure fairness and comparability of experimental results, all models were trained using identical hyperparameter settings and random seeds. For model evaluation, R², RMSE, Mean Absolute Error (MAE), and Ratio of Performance to Deviation (RPD) were used as evaluation metrics, whose calculation formulas are shown in equations (2) to (5).

(2)R^2=1-\frac{{{\displaystyle \sum }}_{\mathit{i}=1}^n{\left(\widetilde{y_i}-y_i\right)}^2}{{{\displaystyle \sum }}_{\mathit{i}=1}^n{\left(\overline{y_i}-y_i\right)}^2}

(3)RMSE=\sqrt{\frac{1}{n}{{\displaystyle \sum }}_{i=1}^n{\left({\hat{y}}_i-y_i\right)}^2}

(4)MAE=\frac{1}{n}{\sum }_{i=1}^n\left|y_i-{\hat{y}}_i\right|

(5)RPD=\frac{SD}{RMSEP}

where n represents the total number of samples, y_i denotes the true value of the remaining days to flowering, {\hat{y}}_i denotes the predicted value of the remaining days to flowering, {\overline{y}}_i represents the mean of the true values, SD denotes the standard deviation of the true remaining flowering days, and RMSEP represents the root mean square error of prediction.

3 Results

3.1 Segmentation of rapeseed flower regions is achieved using YOLOv8s-seg

To reduce the interference of the complex background in the field scenes on the subsequent flowering period tasks, this study first employed the YOLOv8s-seg model to segment the oilseed rape corolla area. Table 3 presents the segmentation performance. The model achieved precision, recall, and mAP50 of 99.93%, 100%, and 99.5%, respectively, on the validation set. This result indicates that the model can stably extract the oilseed rape corolla area in complex field scenes. In Fig. 4, the segmentation effects of the model in different regions and different varieties are further demonstrated. The figure shows that the model effectively preserves corolla information while removing redundant background regions such as leaves, thereby providing high-quality visual input for subsequent rapeseed flowering prediction.

3.2 Data augmentation significantly improves model robustness and reduces prediction error

Data augmentation is a widely employed technique in deep learning designed to improve model generalization and mitigate overfitting. To increase the diversity and representativeness of the training dataset, and thereby enhance the model’s robustness to geometric variations induced by differing imaging perspectives, this study applied stochastic data augmentation to the segmented images. Specifically, each image was subjected to exactly one of three geometric transformations: horizontal flip, vertical flip, or concurrent horizontal and vertical flips, selected uniformly at random. To verify the effectiveness of the data augmentation strategy, this study compared the effects of the original dataset and the augmented dataset. As shown in Table 4, data augmentation has brought an overall performance improvement to the model. The model based on the augmented dataset has an R² that is approximately 0.033 higher than that of the original dataset; the RMSE has decreased from 2.83 to 2.56 days (about 10%), and the RPD has increased from 2.37 to 2.63; the smallest decrease was in the MAE, which slightly decreased from 1.94 to 1.93 days. These results indicate that the data augmentation strategy effectively reduced prediction errors and improved model performance.

3.3 Comparative analysis of different visual models

This study compared five different convolutional neural networks, including MobileNet V2^[17], ResNet 18^[19], DenseNet 121^[21], MobileNet V3^[18], and EfficientNet-B0^[20], and conducted comparisons on four indicators (R², RMSE, MAE, and RPD) in the validation set and test set, respectively. As shown in Table 5, DenseNet 121 achieved the best performance in both the validation set and the test set. On the test set, the R² of DenseNet 121 reached 0.8555, with RMSE and MAE being 2.56 and 1.93 days, respectively, and RPD reaching 2.63. These results outperformed those of other models. The prediction performance of MobileNet V2 and MobileNet V3 was relatively weak, with R² in the test set being lower than 0.80, and RMSE and MAE being relatively high. Additionally, the performance gap between the DenseNet 121 model and other models was small in the validation set and test set, with R² only decreasing by 0.0048, indicating that the model did not overfit and exhibited strong predictive performance and stability on the held-out test set. In contrast, the performance fluctuations of other models between the validation set and test set were more obvious, which also demonstrated the limited predictive ability of other models and the presence of slight overfitting.

To visually compare the actual performance of the model during the testing phase, we plotted marginal distribution graphs to compare the true values and predicted values of different models on the validation set and the test set; as shown in Fig. 5, the predicted values of the DenseNet 121 model are overall closer to the true values, and the predicted values are more concentrated, with a smaller degree of dispersion, indicating that the DenseNet 121 model has better prediction ability and fitting ability; in contrast, the dispersion and error of the predicted values of MobileNet V2 and MobileNet V3 are larger. Additionally, as shown in Table 6, in terms of model complexity, the parameter size of DenseNet 121 is 6.95 MB, and the computational cost is 2.90 GFLOPs, with an overall complexity at a medium level, achieving a good balance between model performance and computational cost. Furthermore, we measured the per-image inference time, FPS, and GPU memory consumption for each model, with detailed results provided in Table S1. DenseNet 121 achieves a per-image inference time of 24.74 ms (40.43 FPS) and consumes 45.22 MB of GPU memory. Although DenseNet 121 is computationally slower than lightweight architectures such as MobileNet V2 and ResNet18, it yields substantially higher prediction accuracy, demonstrating a favorable balance between model performance and computational cost for practical breeding applications.

3.4 Meteorological fusion improves performance only after feature standardization

Single-modal visual features can only reflect the current growth status of plants. To further distinguish the contributions of meteorological data fusion, LSTM-based temporal feature extraction, and Z-score normalization to model performance, five model configurations were evaluated in an ablation study: DenseNet 121, DenseNet 121-weather, DenseNet 121-weather-Z-score, DenseNet 121-LSTM, and DenseNet 121-LSTM-Z-score, as shown in Table 7. For the DenseNet 121-weather and DenseNet 121-weather-Z-score models, the 16-day × 9-variable meteorological data were flattened into one-dimensional feature vectors and directly concatenated with the image features extracted by the DenseNet 121 encoder, without LSTM-based temporal encoding, to separately evaluate the contributions of meteorological information fusion and Z-score normalization. First, directly fusing the raw meteorological variables with the image features of DenseNet 121 did not improve model performance. Compared with the image-only DenseNet 121 model, the test-set R² of DenseNet 121-weather decreased from 0.8555 to 0.8258, while RMSE increased from 2.56 to 2.81 days and MAE increased from 1.93 to 2.09 days. This result suggests that although future meteorological variables contain potentially useful information related to subsequent plant development, substantial differences in their numerical ranges and scales may hinder effective use when the raw variables are directly fused. After Z-score normalization, the DenseNet 121-weather-Z-score model showed a clear improvement, achieving a test-set R² of 0.8637, RMSE of 2.48 days, MAE of 1.75 days, and RPD of 2.71. Compared with the non-normalized DenseNet 121-weather model, R² increased by 0.0379, and RMSE decreased by 0.33 days, indicating that Z-score normalization plays an important role in effectively integrating meteorological variables with different scales. Building on this, the introduction of LSTM for temporal feature extraction from future meteorological time series enabled the DenseNet 121-LSTM-Z-score model to achieve the best performance, with a test-set R² of 0.8979, RMSE of 2.15 days, MAE of 1.51 days, and RPD of 3.13. Notably, without normalization, the test-set R² of DenseNet 121-LSTM was 0.8548, which was almost identical to that of the image-only DenseNet 121 model (0.8555), indicating that simply introducing LSTM did not provide a substantial additional benefit when raw meteorological variables were used. In contrast, after Z-score normalization, the introduction of LSTM further increased R² from 0.8637 to 0.8979 and reduced RMSE from 2.48 to 2.15 days. This finding suggests that, once the meteorological variables are on comparable scales, LSTM can better exploit temporal dependencies in the future meteorological sequence.

The loss curves further illustrate the convergence characteristics of the five models during training, as shown in Fig. 6. Overall, all five models showed a gradual decline in training loss as the number of training epochs increased, although training stability differed among the models. In particular, the DenseNet 121-weather model showed pronounced fluctuations in both training and validation losses, suggesting that directly fusing meteorological variables with image features without normalization may lead to unstable training. In contrast, the loss curves of DenseNet 121, DenseNet 121-weather-Z-score, DenseNet 121-LSTM, and DenseNet 121-LSTM-Z-score were generally smoother and gradually approached convergence. Figure 7 further compares the prediction results of the five models on the test set. As shown in Fig. 7(a–e), differences in consistency between predicted and true values can be observed across the strategies. The scatter points of the DenseNet 121-weather model show greater dispersion relative to the 1:1 reference line, with its test-set R² decreasing to 0.8258, consistent with the performance degradation observed after directly fusing the raw meteorological variables in Table 7. After Z-score normalization, the scatter distribution of the DenseNet 121-weather-Z-score model is closer to the 1:1 reference line, with R² increasing to 0.8637. The DenseNet 121-LSTM model achieved a test-set R² of 0.8548 without normalization, which was nearly identical to that of the image-only DenseNet 121 model (0.8555). In contrast, the DenseNet 121-LSTM-Z-score model showed the highest consistency between the predicted and true values, with its scatter points generally distributed closer to the 1:1 reference line and an R² of 0.8979. Fig. 7(f) further summarizes the RMSE, MAE, and RPD of the five models. The DenseNet 121-LSTM-Z-score model achieved the lowest RMSE (2.15 days) and MAE (1.51 days), as well as the highest RPD (3.13), further confirming its best prediction performance among the five strategies.

To evaluate the prediction reliability of the DenseNet 121-LSTM-Z-score model, a statistical analysis of prediction errors on the test set was conducted (Fig. 8). Overall, the prediction errors were primarily concentrated around 0–1 day, indicating a high consistency between predicted and actual values, with a relatively tight error distribution. Further assessment revealed that 72.13% of samples had prediction errors within ±2 days, while 85.63% fell within ±3 days, suggesting that most samples yielded stable predictions, with only a small proportion exhibiting significant errors. The regression plot showed no evident systematic overestimation or underestimation, demonstrating overall good stability. To further evaluate how prediction accuracy varies with the remaining days to flowering, we stratified the test set by lead time and calculated RMSE and MAE for each group (Table S2, Fig S1). The model maintained high accuracy within 0–14 days (RMSE ≤ 2.07 days, MAE ≤ 1.48 days), with the best performance in the 5–9 days interval (RMSE = 1.63 days, MAE = 1.23 days). Beyond 15 days, prediction errors increased progressively (RMSE: 2.19 days for 15–19 days, 2.86 days for 20–24 days, and 4.89 days for 25–29 days), indicating that the 15-day forecast window defines the effective temporal boundary for reliable predictions. These results suggest that the model can provide practically useful flowering-time predictions within 15 days before anthesis, with lead-time-dependent precision that supports time-sensitive breeding operations.

4 Discussion

4.1 The influence of the selection of the DenseNet 121 backbone network on the performance of the rapeseed flowering period prediction task

Deep learning methods based on RGB images provide an objective and effective approach to phenotypic analysis for predicting the remaining days to flowering in rapeseed. Compared with traditional machine learning methods that rely on manually designed features, CNNs can automatically learn hierarchical visual representations from images through an end-to-end learning framework, thereby reducing the subjectivity and potential information loss associated with manual feature engineering^[23]. During the transition from vegetative to reproductive growth, rapeseed plants undergo continuous changes in morphology and inflorescence appearance, which RGB imaging can capture. Therefore, deep learning models can extract informative visual features from RGB images and establish nonlinear relationships between image-based representations and the remaining days to flowering.

This study compared five convolutional neural networks, including MobileNet V2, MobileNet V3, ResNet 18, EfficientNet-B0, and DenseNet 121, for predicting the remaining days to flowering in rapeseed. Among these models, DenseNet 121 achieved the best overall performance, likely due to its dense connectivity architecture. This architecture integrates low-level details and high-level structural information, better characterizing visual differences among developmental stages. At the same time, these visual features also correspond to morphological indicators of rapeseed before flowering: in early stages, the inflorescence continuously elongates and buds differentiate sequentially; as flowering approaches, bud volume increases, and color becomes lighter. These changes form distinguishable patterns in the images, as shown in Fig. 1. DenseNet 121, through its multi-level feature integration, effectively captures these phenotypic changes associated with the flowering progression, thereby improving the prediction accuracy of the remaining days to flowering.

In contrast, lightweight networks such as MobileNet V2^[17] and MobileNet V3^[18] have advantages in terms of reduced parameters and computational efficiency; however, their compact architectures may limit their capacity to represent complex visual information, resulting in relatively weaker performance in this regression task. Although ResNet 18^[19] and EfficientNet-B0^[20] also exhibit strong feature extraction capabilities, their ability to integrate visual information under the dataset scale and task characteristics of this study was inferior to that of DenseNet 121. Therefore, DenseNet 121 provides a favorable balance between predictive performance and computational complexity, making it a more suitable visual backbone for predicting the remaining days to flowering at the individual plant level in rapeseed. Future work can visualize the key image regions the model focuses on using techniques such as Gradient-CAM, further revealing the biological basis for its predictions.

4.2 Analysis of the impact of meteorological data fusion on model performance

Crop growth and development are governed by the interactions among genotype, environmental conditions, and cultivation practices, with environmental factors, particularly temperature, light, and water availability, playing important roles in regulating developmental rates. Visual features obtained from a single time point mainly represent the current phenotypic state of plants and cannot fully reflect the influence of subsequent environmental conditions on future developmental progression. Therefore, prediction models based solely on RGB images may not adequately consider the effects of future environmental variations on rapeseed flowering progression. In this study, future meteorological time-series information was incorporated into the DenseNet 121-based visual feature extraction model, allowing the model to jointly utilize current phenotypic information and future environmental conditions to improve the prediction accuracy of remaining days to flowering.

Current methods for predicting the remaining days to phenological stages using multimodal data mainly include two approaches: the integration of remote sensing data with GDD models, and the combination of image data with meteorological information. The remote sensing-based approach typically uses continuous monitoring of canopy spectral changes to identify the onset of specific phenological stages and then applies GDD models to estimate the remaining days until the corresponding stages are reached^[6,¹²,^24]. However, this approach is mainly designed for regional-scale phenology monitoring and prediction. Its performance depends on predefined temperature-response relationships, which may limit its ability to capture variations among individual plants and account for complex environmental effects. Therefore, it remains challenging to apply this approach for precise phenological prediction at the individual-plant level under diverse field conditions.

The prediction of remaining days to phenological stages based on image and meteorological data is mainly achieved by constructing multimodal deep learning models. For example, Shen et al.^[14] combined historical weather data from sowing to the observation date with image data to identify the current phenological stage of wheat and predict the remaining days to heading. By incorporating accumulated environmental effects during crop development, their approach provided additional information for characterizing the current growth status of plants. Furthermore, Xie et al.^[15] integrated historical weather information (previous 90 days), short-term future weather conditions (next 7 days), and image data for wheat heading prediction, thereby considering both the cumulative effects of past environments and the potential influence of future environmental variations on subsequent development. However, for predicting the remaining days to flowering, future meteorological conditions may provide additional information on environmental variations after image acquisition and may affect subsequent flowering progression, whereas historical weather data primarily reflect conditions that have already influenced the plant before the current observation.

Beyond the methodological limitations discussed above, the biological rationale for incorporating meteorological variables into flowering time prediction should also be considered. Temperature is widely recognized as the core factor regulating rapeseed flowering progression. Luo et al.^[25] demonstrated that the developmental rate during the pre-floral initiation phase exhibits an asymmetrical distribution against mean daily temperature, decreasing rapidly when the temperature exceeds the optimum, with different genotypes showing distinct optimal temperatures for floral initiation. Photoperiod (daylength) also plays a profound role: Nelson et al.^[26] found that long-day conditions accelerated flowering by an average of 765 degree-days compared with short-day treatments, and early-maturity genotypes exhibit greater photoperiod sensitivity than late-maturity ones. In addition to temperature and photoperiod, solar radiation has been identified as an independent factor affecting flowering time, with evaporation serving as a primary proxy for radiation effects^[27]. Furthermore, precipitation and relative humidity are negatively correlated with flowering date and form the major components of the weather factor in flowering prediction models^[27]. These findings collectively suggest that temperature, radiation, photoperiod, precipitation, and humidity each regulate flowering progression through distinct physiological pathways. Therefore, incorporating future meteorological time-series information, rather than relying solely on historical weather or current visual phenotypes, provides a more comprehensive representation of the environmental conditions that influence subsequent flowering development.

Based on the above findings, this study developed a multimodal deep learning framework that integrates RGB visual information with future meteorological data to predict the remaining days to flowering at the individual-plant level in rapeseed. Compared with approaches based on remote sensing data and accumulated temperature models, the proposed method directly extracts fine-scale phenotypic features of individual rapeseed plants from RGB images using deep learning techniques. In addition, it incorporates multiple meteorological variables, including radiation, temperature, precipitation, and humidity, allowing the model to consider the potential effects of future environmental variations on plant developmental rates and achieve more precise predictions at the individual-plant scale. Furthermore, existing multimodal phenological prediction methods mainly utilize historical meteorological information. In contrast, future meteorological conditions can provide complementary information regarding environmental changes after image acquisition and their potential impacts on subsequent developmental processes. Consequently, integrating future meteorological time-series information with current visual phenotypes enables a more comprehensive representation of the dynamic transition of rapeseed from the current developmental stage to flowering.

4.3 Limitations of this study and future work

This study proposes a multimodal deep learning framework that integrates RGB images with future meteorological time-series data to predict the remaining days until flowering at the individual plant level, providing a novel technical approach for smart phenotyping-assisted rapeseed breeding. Compared with traditional phenological assessment methods that rely on manual field observations, the proposed approach automatically extracts phenotypic information describing the current developmental status of individual inflorescences from RGB images and combines it with future meteorological conditions to estimate flowering time. This multimodal strategy reduces the subjectivity and labor intensity associated with manual observations while improving the accuracy and consistency of flowering time prediction. Consequently, it provides timely decision support for critical hybrid breeding operations, including emasculation, bagging, and pollination.

However, this method still has limitations and requires further improvement. First, regarding model reliability and generalization, this study employed random splitting of the training and test sets, which may allow samples from the same genotype or similar environmental conditions to appear in both sets, potentially leading to an overestimation of model performance. Therefore, the generalization ability of the model to unseen genotypes, new locations, and different environmental conditions still requires further validation. Second, future meteorological information is inherently uncertain. In practical applications, errors in weather forecasts may propagate into flowering-time predictions, and this effect may increase with longer forecast lead times, thereby affecting the stability and reliability of the model in real-world breeding scenarios. Third, the current model does not explicitly incorporate genotype and crop-management factors. Because crop phenology results from the combined effects of genotype, environment, and management, differences among genotypes, as well as factors such as sowing date, planting density, fertilization, and irrigation, may influence plant development and environmental responses, thereby limiting the applicability of the model under complex agricultural conditions. Fourth, the model’s biological interpretability remains limited. This study did not systematically analyze the key image regions attended to by the model or the relative contributions of different meteorological variables to the predictions. Therefore, the relationships between the model’s internal decision-making process and the physiological and phenological processes of rapeseed still require further investigation.

To overcome these limitations, future research in this area can be conducted in the following ways: First, more rigorous validation strategies, such as genotype-grouped splitting, leave-one-genotype-out validation, and leave-one-site-out validation, should be adopted to more comprehensively assess model generalization to unseen genotypes and environmental conditions. Second, the uncertainty associated with future weather forecasts should be explicitly considered by evaluating model performance under different forecast lead times (e.g., 3-, 7-, and 15-day forecasts) and, where possible, incorporating ensemble weather forecasts to quantify the propagation of meteorological uncertainty into flowering-time predictions. Third, genotype information and crop-management variables, such as sowing date, planting density, fertilization, and irrigation, could be incorporated to better represent genotype–environment–management interactions and improve model robustness under diverse agricultural conditions. Fourth, model interpretability could be strengthened by combining methods such as Grad-CAM for identifying informative image regions with SHAP, permutation feature importance, or variable-ablation analyses to quantify the contributions of individual meteorological variables, thereby improving the biological interpretability and practical credibility of the model.

5 Conclusions

This study proposes a multimodal prediction framework that integrates RGB images with future meteorological time series to predict the remaining flowering days of individual rapeseed plants. In the visual branch, YOLOv8s-seg is employed to accurately segment flower regions, and data augmentation strategies are applied to enhance model robustness. Among various backbone networks compared, DenseNet 121 achieves the best balance in terms of accuracy, generalization capability, and complexity, and is therefore selected as the visual feature extraction backbone network. Subsequently, LSTM is introduced to model future meteorological sequences, which are then fused with visual features. However, directly incorporating raw meteorological data yields only marginal performance gains due to substantial feature scale discrepancies. After applying Z-score normalization, model performance significantly improves, achieving R² = 0.8979, RMSE = 2.15 days, MAE = 1.51 days, and RPD = 3.13 on the test set—outperforming other approaches substantially. These results indicate that standardized multimodal fusion effectively enhances the model’s ability to utilize meteorological information and improves generalization performance. This method can support flowering period prediction and coordination of parental flowering in rapeseed hybrid breeding, providing a methodological reference for integrating multimodal learning into crop phenology prediction and breeding decision support.

References

[1] Tu M X, Wang R S, Guo W H, Xu S Q, Zhu Y, Dong J, Yao X T, Jiang L X . A CRISPR/Cas9-induced male-sterile line facilitating easy hybrid production in polyploid rapeseed (Brassica napus). Horticulture Research, 2024, 11(7): uhae139

[2] Ali A, Bakhsh M Z M, Li S P, Zhang X Y, Tu J X, Yi B . Cytoplasmic male sterility: sterility induction, fertility restoration and gene-for-gene interaction of CMS/Rf genes. Plant Science, 2025, 360: 112721

[3] Meier U, Bleiholder H, Buhr L, Feller C, Hack H, Heß M, Lancashire P D, Schnock U, Stauß R, Van Den Boom T, Weber E, Zwerger P . The BBCH system to coding the phenological growth stages of plants—history and publications. Journal für Kulturpflanzen, 2009, 61(2): 41–52

[4] Teodoro P E, Teodoro L P R, Baio F H R, da Silva Junior C A, dos Santos R G, Ramos A P M, Pinheiro M M F, Osco L P, Gonçalves W N, Carneiro A M, Junior J M, Pistori H, Shiratsuchi L S . Predicting days to maturity, plant height, and grain yield in soybean: a machine and deep learning approach using multispectral data. Remote Sensing, 2021, 13(22): 4632

[5] Huang X M. Research and Application of Rice Phenology Recognition and Prediction Based on DCNN and LSTM. Dali: Dali University, 2024 (in Chinese)

[6] Dížková P, Bartošová L, Bláhová M, Balek J, Hájková L, Semerádová D, Bohuslav J, Pohanková E, Žalud Z, Trnka M . Modeling phenological phases of winter wheat based on temperature and the start of the growing season. Atmosphere, 2022, 13(11): 1854

[7] Salazar-Gutierrez M R, Johnson J, Chaves-Cordoba B, Hoogenboom G . Relationship of base temperature to development of winter wheat. International Journal of Plant Production, 2013, 7(4): 741–762

[8] Xue Q W, Weiss A, Baenziger P S . Predicting phenological development in winter wheat. Climate Research, 2004, 25(3): 243–252

[9] Wang E L, Engel T . Simulation of phenological development of wheat crops. Agricultural Systems, 1998, 58(1): 1–24

[10] Ritchie J T. Wheat phasic development. In: Hanks J, Ritchie J T, eds. Modeling Plant and Soil Systems. Madison: ASA, 1991, 31–54

[11] Jones J W, Hoogenboom G, Porter C H, Boote K J, Batchelor W D, Hunt L A, Wilkens P W, Singh U, Gijsman A J, Ritchie J T . The DSSAT cropping system model. European Journal of Agronomy, 2003, 18(3–4): 235–265

[12] Liu R X, Zhang F Z, Gao Y H, Zhang J B, Liu Z X, Li Z H, Yang J T . Winter wheat maturity date prediction using MODIS/ECMWF data: accuracy evaluation and spatiotemporal variation analysis. European Journal of Agronomy, 2025, 167: 127581

[13] LeCun Y, Bengio Y, Hinton G . Deep learning. Nature, 2015, 521(7553): 436–444

[14] Shen L Y, Ding G H, Jackson R, Ali M, Liu S C, Mitchell A, Shi Y Y, Lu X Q, Dai J, Deakin G, Frels K, Cen H Y, Ge Y F, Zhou J . GSP-AI: an AI-powered platform for identifying key growth stages and the vegetative-to-reproductive transition in wheat using trilateral drone imagery and meteorological data. Plant Phenomics, 2024, 6: 0255

[15] Xie Y T, Roy S J, Schilling R K, Liu H J . Multi-modal few-shot learning for anthesis prediction of individual wheat plants. Plant Phenomics, 2025, 7(3): 100091

[16] Varghese R, Sambath M. YOLOv8: a novel object detection algorithm with enhanced performance and robustness. In: Proceedings of 2024 International Conference on Advances in Data Engineering and Intelligent Computing Systems (ADICS). Chennai, India: IEEE, 2024, 1–6

[17] Sandler M, Howard A, Zhu M L, Zhmoginov A, Chen L C. MobileNetV2: inverted residuals and linear bottlenecks. In: Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City, UT, USA: IEEE, 2018, 4510–4520

[18] Howard A, Sandler M, Chen B, Wang W J, Chen L C, Tan M X, Chu G, Vasudevan V, Zhu Y K, Pang R M, Adam H, Le Q V. Searching for MobileNetV3. In: Proceedings of the 2019 IEEE/CVF International Conference on Computer Vision (ICCV). Seoul, Korea (South): IEEE, 2019, 1314–1324

[19] He K M, Zhang X Y, Ren S Q, Sun J. Deep residual learning for image recognition. In: Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Las Vegas, NV, USA: IEEE, 2016, 770–778

[20] Tan M X, Le Q V. EfficientNet: rethinking model scaling for convolutional neural networks. In: Proceedings of the 36th International Conference on Machine Learning (ICML). Long Beach, CA, USA: PMLR, 2019, 6105–6114

[21] Huang G, Liu Z, Van Der Maaten L, Weinberger K Q. Densely connected convolutional networks. In: Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Honolulu, HI, USA: IEEE, 2017, 2261–2269

[22] Gers F A, Schraudolph N N, Schmidhuber J . Learning precise timing with LSTM recurrent networks. The Journal of Machine Learning Research, 2003, 3: 115–143

[23] LeCun Y, Bottou L, Bengio Y, Haffner P . Gradient-based learning applied to document recognition. Proceedings of the IEEE, 1998, 86(11): 2278–2324

[24] Huang X, Zhu W Q, Wang X Y, Zhan P, Liu Q F, Li X Y, Sun L X . A method for monitoring and forecasting the heading and flowering dates of winter wheat combining satellite-derived green-up dates and accumulated temperature. Remote Sensing, 2020, 12(21): 3536

[25] Luo T, Zhang J, Khan M N, Liu J H, Xu Z H, Hu L Y . Temperature variation caused by sowing dates significantly affects floral initiation and floral bud differentiation processes in rapeseed (Brassica napus L.). Plant Science, 2018, 271: 40–51

[26] Nelson M N, Rajasekaran R, Smith A, Chen S, Beeck C P, Siddique K H M, Cowling W A . Quantitative trait loci for thermal time to flowering and photoperiod responsiveness discovered in summer annual-type Brassica napus L. PLoS One, 2014, 9(7): e102611

[27] Sun J Q, Zhang Z W, Ai W W. Application of BP neural network in the prediction of oilseed rape florescence. Meteorological and Environmental Sciences, 2019, 42(4): 22–26 (in Chinese)

Rights & permissions

The Author(s) 2026. Published by Higher Education Press. This is an open access article under the CC BY license (http://creativecommons.org/licenses/by/4.0)