A Cross-Resolution Teacher-Student Framework for Hyperspectral and Multispectral Image Registration

Abstract

Hyperspectral (HS) and multispectral (MS) image registration is an essential precondition for their joint exploitation, yet remains challenging under their large spatial-spectral resolution discrepancy. A cross-resolution teacher-student framework (CTSF) is proposed in this paper, where HS and MS image registration is degraded as cross-resolution single-band registration. Especially, a hierarchical shared teacher-student network is designed to learn paired descriptors from these cross-resolution pairs. More specifically, a dual-path hierarchical shared teacher and a single-path hierarchical student net are proposed to reduce gaps of content representation and receptive field between cross-resolution pairs. Furthermore, a confidence-guided selective distillation strategy and a cross-resolution alignment regularization are investigated to perform high-consistency description. Experimental results demonstrate that our CTSF shows obvious advantages as the resolution gap grows.

Graphical abstract

Keywords

cross-resolution image registration / hyperspectral image / multispectral image / knowledge distillation

1 Introduction

Generally, hyperspectral (HS) images provide rich spectral information but low spatial resolution, while multispectral (MS) images offer high spatial resolution but limited spectral information. Spatial-spectral fusion of the two modalities can generate high-resolution (HR) HS data so that they are widely used in environmental monitoring and precision agriculture. However, the fusion effect relies heavily on accurate geometric registration which is highly challenging because of their significant discrepancy in spatial and spectral resolution.

Existing image registration can be broadly categorized into feature-based and template-based approaches. Feature-based methods rely on local feature detection, description, and matching, such as SIFT [1] and PIIFD [2]. They probably suffer from insufficient keypoints and unreliable matches when registering images of large resolution discrepancy because local gradients and structural cues could greatly degrade in low-resolution (LR) images. Template-based methods achieve registration using transformation parameters estimated by optimizing a similarity measure function. For example, the sum-of-squared difference (SSD) [3] is measured to register single-modality images well, but it is sensitive to radiometric differences of multi-modality data. Subsequently, some improvements, for instance, normalized mutual information (NMI) [4], have been developed to mitigate the negative impact of radiometric differences on registration performance. Similarly, template-based methods are of limited ability to address cross-resolution image registration.

In order to achieve better feature/template matching, various deep learning methods [5, 6] have been investigated. XFeat [7] which applied a novel match refinement module based on coarse local descriptors, achieved the first semi-dense matching. By introducing a multi-scale and multi-homography approach to boost the ability for interest point detection and cross-domain adaptation, a self-supervised interest point detection and description (SuperPoint) [8] was proposed. SuperGlue network [9] achieved outstanding local feature matching, where the feature assignment was achieved via a flexible context aggregation mechanism. To avoid ambiguous areas, Revaud et al. proposed to jointly learn keypoint detection and description together with a predictor of the local descriptor discrimination, resulting in repeatable and reliable detector and descriptor (R2D2) [10]. Different from the above sparse paradigm, a dense kernelized feature matching (DKM) [11] was designed through a kernel regression global matcher, warp refinement, and dense confidence maps. Despite the excellent performance of existing general-purpose feature extraction and matching methods, severe performance degradation may still occur when they are directly applied to HS-MS image registration. Due to the large resolution gap between HS and MS images, the extracted and matched features may suffer from risks of loss and mismatch. It is an alternative [12] to perform coarse super-resolution on the HS image via up-sampling operation, but the up-sampled one is registered rather than the original HS image.

Recently, knowledge distillation [13] has been applied to excellent local feature descriptions [14] by enabling a lightweight student model to inherit discriminative knowledge from a large teacher model. Lee et al. [15] distilled intermediate convolutional features into compact local descriptors (COLD), while Guo et al. [16] introduced a teacher-student framework to learn low-dimensional descriptors for efficient multi-robot simultaneous localization and mapping (SLAM). Nevertheless, cross-resolution descriptor learning for HS-MS registration remains insufficiently studied. Therefore, a cross-resolution teacher-student framework (CTSF) is proposed for HS-MS registration, where knowledge distillation is introduced to address cross-resolution descriptor learning for the first time. Since all bands of HS/MS images capture the same scene from the same perspective, geometric deviations are assumed to be identical among different bands. Naturally, the affine transformation between LR and HR single-bands projected from HS and MS images, respectively, can be used to align them band-by-band. Then, a hierarchical shared teacher-student network (HSNet) is designed to perform local feature descriptions for HR and LR projections. Considering the gaps of spatial resolution and receptive field between the two projections, a dual-path hierarchical shared teacher network that is equipped with a cross-resolution alignment regularization, is constructed for discriminative representation of HR projection as well as initialized alignment of LR descriptors. In addition, a single-path hierarchical shared student network is designed with more lightweight structure to extract LR descriptors aligning to HR ones generated by teacher net. Different from general knowledge distillation that student net is trained from scratch using full distillation, part parameters of our student net are retrained via a confidence-guided selective distillation strategy after the parameter sharing from teacher net. Thus, not only training cost is effectively reduced, but also negative interference caused by resolution degradation could be alleviated.

The main contributions of our novel CTSF can be summarized as follows:

1) An HSNet containing a dual-path hierarchical shared teacher net and a single-path hierarchical shared student net is designed to extract paired descriptors for cross-resolution inputs.

2) A hierarchical parameter sharing strategy between dual paths of teacher net and between teacher and student nets is proposed for effective transfer of discriminative features and reduction of network training cost.

3) A cross-resolution alignment regularization is established to optimize the spatial description consistency for teacher net.

4) A confidence-guided selective distillation loss is constructed to transfer reliable soft labels as well as suppress possible unreliability of resolution degradation.

2 Methodology

HS and MS images exhibit significant differences in spatial resolution, spectral resolution, and radiometric characteristics, which make direct geometric registration highly challenging. Assuming the uniformity of geometric deviation among bands, both HS and MS images are first projected into single-band representations by averaging across spectrum following [12]. Then, the registration between HS and MS images can be simplified as the cross-resolution alignment between their corresponding HR and LR projections. As illustrated by our CTSF in Fig. 1(a), HSNet is designed to perform cross-resolution descriptor establishment for pairs of the HR and LR projections. After initial correspondence establishment and outliers removal which are executed via nearest-neighbor matching and RANSAC [17], respectively, the final affine matrix is calculated to align HS image to the corresponding MS one band-by-band.

2.1 HSNet

A key challenge in cross-resolution descriptor establishment lies in the consistent feature representations for HR and LR projections. One convolution kernel that captures fine local structures on dense HR grids, could extract blurred and mixed texture when imposed on sparse LR grids because much broader region is spanned. If different convolution filters are applied in projections of high and low spatial resolution, consistency cannot be ensured for their feature representations. In addition, the standard convolution might cause limited receptive field, since the rich structures and high-frequency details are contained in HR projection. In order to meet the demands of feature consistency and different receptive fields, a descriptor estimation network is characterized by a hierarchical shared structure, which is designed in Fig. 1. Affine parameters between real HS and MS observations are unknown, which signifies that corresponding coordinates are unknown before and after geometrical transformation, so paired descriptor estimation becomes fairly difficult. Inspired by the strong discriminability preservation of the teacher-student transfer, a HSNet is eventually constructed in this paper.

As shown in Fig. 1(b), the teacher network consists of a dual-path backbone with shared parameters and two path-specific modules with independent parameters. By projecting the HR and LR features onto a unified feature grid, the path-specific modules facilitate the extraction of paired descriptors. Thus, asymmetric convolutions are applied in the dual-path backbone. Further to ensure feature unification, parameters of convolution kernels in the dual paths are shared. More specifically, the dilated convolution is employed in the high path to capture fine details and broader context from the HR projection using bigger receptive field. The strided convolution is adopted in the low path to match the sparse sampling characteristics of LR projection. Besides sharing same convolution parameters, similar network architectures are designed for the two paths. As shown in Fig. 1(b), both paths are constructed by the cascade of six dense blocks which is linked via a convolution layer of size 3\times 3, a batch normalization (BN) layer, and the rectified linear unit (ReLU) function in succession. In addition, feature dimensionalities of both paths increase uniformly along the network depth, enabling hierarchical feature abstraction. These designs map the HR/LR inputs into their feature spaces through asymmetric convolutions matching their respective resolutions, filters sharing the same parameters, and backbones employing similar structures, thereby preliminarily reducing their gaps in resolution and representation.

Subsequently, the two paths branch into path-specific mapping modules that adapt features to their respective resolutions and project them onto a unified spatial grid of size m\times n. The high and low path-specific modules each consist of two dense blocks followed by a shortcut branch. The dense blocks and shortcut branch in the high path-specific module use 2\times 2 and 1\times 1 convolution layers, respectively, whereas all convolution layers in the low path-specific module use 3\times 3 kernels. In order to achieve the final resolution unification, the strided and dilated convolutions are applied in the high and low path-specific modules, respectively. Space sizes, dilation rates, and stride configurations of convolutional operations in both paths and modules are illustrated in Fig. 1(b). After resolution and representation unification of the high and low path-specific modules, the corresponding two features {\mathcal{F}}^h\in {\mathbb{R}}^{m\times n\times 128} and {\mathcal{F}}^l\in {\mathbb{R}}^{m\times n\times 128} are obtained. After imposing dimensionality reduction on the element-wise square of {\mathcal{F}}^p via 1\times 1 convolutions, {\mathit{T}}^p\in {\mathbb{R}}^{m\times n} and {\mathit{C}}^p\in {\mathbb{R}}^{m\times n\times 2} are produced to calculate the repeatability and reliability maps of an image, respectively. Under L_2 normalization, sigmoid, and softmax function, the descriptor, repeatability, and reliability are individually formulated as follows

(1){\mathcal{D}}_{ijk}^p=\frac{{\mathcal{F}}_{ijk}^p}{\sqrt{{\displaystyle {\sum }_{k=1}^{128}({\mathcal{F}}_{ijk}^p{)}^2}}}

(2){\mathit{S}}_{ij}^p=\frac{1}{1+{\mathrm{e}}^{-{\mathit{T}}_{\mathrm{i}\mathrm{j}}^{\mathrm{p}}}}

(3){\mathit{R}}_{ij}^p=\frac{{\mathrm{e}}^{{\mathcal{C}}_{\mathcal{i}\mathcal{j}\mathcal{1}}^{\mathcal{p}}}}{{\mathrm{e}}^{{\mathcal{C}}_{\mathcal{i}\mathcal{j}\mathcal{1}}^{\mathcal{p}}}+{\mathrm{e}}^{{\mathcal{C}}_{\mathcal{i}\mathcal{j}\mathcal{0}}^{\mathcal{p}}}}

where p=[h,l] corresponds to the HR/LR inputs; {\mathcal{D}}^p\in {\mathbb{R}}^{m\times n\times 128}, {\mathit{S}}^p\in {\mathbb{R}}^{m\times n}, and {\mathit{R}}^p\in {\mathbb{R}}^{m\times n} denote the descriptor, repeatability, and reliability maps, respectively; i, j, and k denote the i-th row, j-th column, k-th channel, respectively.

Under the supervision of soft labels that are extracted from projections of MS observations via teacher net, student net is naturally responsible for the abstraction of the corresponding LR descriptors from projections of HS measurements. To ensure the consistency of feature representation, student net maintains a structure consistent with the LR branch of teacher net. Namely, it is cascaded by a low-path and low path-specific module, as presented in Fig. 1(b).

2.2 Knowledge Distillation-based Training Strategy

A two-stage training strategy established on knowledge distillation is proposed to optimize our HSNet to pose the extraction of paired descriptors from cross-resolution inputs. General knowledge distillation which always performs the full distillation aims to teach a small student network via a trained and large teacher net. In our work, a selective distillation strategy is proposed to reduce propagation of unimportant information, and part parameters are shared from teacher to student net for a faster student net training.

2.2.1 Teacher Stage: Cross-resolution Self-Alignment

In order to learn discriminative features of high resolution while establishing cross-resolution consistency, teacher net is trained using HR projections, their affine transformations and downsamplings during the teacher stage. Let the MS and HS measurements be denoted as {\mathcal{I}}^h\in {\mathbb{R}}^{H\times W\times b} and {\mathcal{I}}^l\in {\mathbb{R}}^{h\times w\times B}, respectively, where H>h, W>w, and B>b are individually the number of the image row, column, and band. The HR projection {\mathit{I}}^h\in {\mathbb{R}}^{H\times W} is the spectral average of {\mathcal{I}}^h. The training set consists of {\mathit{I}}^h, and its random affine transformation {\mathit{I}}^{\prime h}\in {\mathbb{R}}^{H\times W} and downsampling result {\mathit{I}}^{\prime l}\in {\mathbb{R}}^{h\times w}. As the teacher stage in Fig. 1 shown, three map sets \{{\mathcal{D}}^h\in {\mathbb{R}}^{m\times n\times 128},{\mathit{S}}^h\in {\mathbb{R}}^{m\times n},{\mathit{R}}^h\in {\mathbb{R}}^{m\times n}\}, \{{\mathcal{D}}^{\prime h}\in {\mathbb{R}}^{m\times n\times 128},{\mathit{S}}^{\prime h}\in {\mathbb{R}}^{m\times n},{\mathit{R}}^{\prime h}\in {\mathbb{R}}^{m\times n}\}, and \{{\mathcal{D}}^{\prime l}\in {\mathbb{R}}^{m\times n\times 128},{\mathit{S}}^{\prime l}\in {\mathbb{R}}^{m\times n},{\mathit{R}}^{\prime l}\in {\mathbb{R}}^{m\times n}\} are generated as the descriptor, repeatability, and reliability of {\mathit{I}}^h, {\mathit{I}}^{\prime h}, and {\mathit{I}}^{\prime l} in sequence.

Since dense pixel-wise correspondences between {\mathit{I}}^h and {\mathit{I}}^{\prime h} provided by known affine parameters can be used as supervision labels, a training objective L_{\mathrm{b}\mathrm{a}\mathrm{s}\mathrm{e}} is calculated following R2D2 [10]

(4)\begin{array}{rl}{\mathcal{L}}_{\mathrm{b}\mathrm{a}\mathrm{s}\mathrm{e}}=& \frac{1}{|\mathrm{\Omega }|}\sum _{(ij,i^{\prime }{j}^{\prime })\in \mathrm{\Omega }}{\ell }_{bce}({\mathit{R}}_{ij},{\mathit{Y}}_{i^{\prime }{j}^{\prime }})+\\ & [1-\frac{1}{|\mathcal{Q}|}\sum _{q\in \mathcal{Q}}\mathrm{c}\mathrm{o}\mathrm{s}\mathrm{i}\mathrm{m}({\mathit{S}}_q^h,{\mathit{S}}_q^u)]+\\ & \{1-\frac{1}{|\mathcal{Q}|}\sum _{q\in \mathcal{Q}}[max({\mathit{S}}_q)-\mathrm{m}\mathrm{e}\mathrm{a}\mathrm{n}({\mathit{S}}_q)]\}\end{array}

with

(5){\mathit{Y}}_{i^{\prime }{j}^{\prime }}=\{\begin{array}{ll}0,& \mathrm{A}\mathrm{P}({\mathcal{D}}_{ij:}^h,{\mathcal{D}}_{i^{\prime }{j}^{\prime }:}^{\prime h})⩽\tau \\ 1,& \mathrm{A}\mathrm{P}({\mathcal{D}}_{ij:}^h,{\mathcal{D}}_{i^{\prime }{j}^{\prime }:}^{\prime h})>\tau \end{array}

where \mathit{R}=({\mathit{R}}^h+{\mathit{R}}^{\prime h})/2; \mathrm{\Omega } denotes the set of valid coordinate pairs induced by the known affine transformation between {\mathit{I}}^h and {\mathit{I}}^{\prime h}, \mathrm{A}\mathrm{P}(\cdot ) is the pixel-wise average precision[18]; {\ell }_{bce} denotes the binary cross-entropy loss[19]; \mathrm{c}\mathrm{o}\mathrm{s}\mathrm{i}\mathrm{m}(a,b) calculates the cosine similarity between vectors a and b; \mathcal{Q}=\{q\} is the set of overlapping patches of size N\times N extracted from a m\times n matrix; {\mathit{S}}_q^h,{\mathit{S}}_q^u\in {\mathbb{R}}^{N^2} are the q-th vectorized patches extracted from {\mathit{S}}^h and {\mathit{S}}^u, respectively; {\mathit{S}}^u denotes the affine transformation of {\mathit{S}}^{\prime h}. Under the stable supervision of local features before geometric transformation, the first and second terms in {\mathcal{L}}_{\mathrm{b}\mathrm{a}\mathrm{s}\mathrm{e}} minimize the gap between the two reliability maps and maximize the similarity between two repeatability patches across different transformed views, respectively, while the third term preserves the locally peaked of repeatability patches of the original HR projection. Comprehensively, the R2D2-style feature-learning loss {\mathcal{L}}_{\mathrm{b}\mathrm{a}\mathrm{s}\mathrm{e}} ensures better global reliability and local repeatability under geometric transformations.

By minimizing the base loss L_{\mathrm{b}\mathrm{a}\mathrm{s}\mathrm{e}}, the dual-path backbone and the high path-specific module can be optimized for effective HR feature representation. For further refining representation and cross-resolution alignment, a consistency loss is defined as

(6)\begin{array}{rl}{\mathcal{L}}_{\mathrm{c}\mathrm{r}\mathrm{c}}=& \frac{1}{|\mathrm{\Omega }|}\sum _{(ij,i^{\prime }{j}^{\prime })\in \mathrm{\Omega }}[1-{\mathcal{D}}_{ij:}^h({\mathcal{D}}_{i^{\prime }{j}^{\prime }:}^{\prime h}{)}^{\mathrm{\top }}]+\\ & \frac{1}{mn}\sum _{i=1,j=1}^{m,n}[1-\mathrm{S}\mathrm{G}({\mathcal{D}}_{ij:}^h)({\mathcal{D}}_{ij:}^{\prime l}{)}^{\mathrm{\top }}]\end{array}

where \mathrm{S}\mathrm{G}(\cdot ) denotes the stop-gradient operator [20]. By using the \mathrm{S}\mathrm{G}(\cdot ), the HR descriptor is treated as a fixed target; thus, the LR descriptor is encouraged to approach the HR representation. The first term enforces geometric consistency within the HR domain, while the second term aligns LR descriptors to the HR one. Combining the R2D2-style loss {\mathcal{L}}_{\mathrm{b}\mathrm{a}\mathrm{s}\mathrm{e}} and the consistency loss {\mathcal{L}}_{\mathrm{c}\mathrm{r}\mathrm{c}}, the teacher-stage loss is formulated as

(7){\mathcal{L}}_{\mathcal{T}}\mathcal{=}{\mathcal{L}}_{\mathrm{b}\mathrm{a}\mathrm{s}\mathrm{e}}\mathcal{+}\lambda {\mathcal{L}}_{\mathrm{c}\mathrm{r}\mathrm{c}}

where \lambda >0 controls the strength of cross-resolution consistency. Through minimizing the total teacher-stage loss, teacher net is optimized to learn robust HR feature representations while providing an aligned initialization for the LR branch.

2.2.2 Student Stage: Confidence-Guided Selective Distillation

With the unified HR representation and cross-resolution feature alignment learned by the trained parameter-sharing dual-path teacher network and the low path-specific module, the student network is further partially optimized through teacher-to-student distillation, so that the LR descriptors extracted from the projection of {\mathcal{I}}^l can be aligned with the HR descriptors generated from the projection of {\mathcal{I}}^h. The LR projection corresponding to the HS image {\mathcal{I}}^l is denoted as {\mathit{I}}^l\in {\mathbb{R}}^{h\times w}. Considering that LR projections contain blurred structures and mixed spectrums, the generally full distillation could be suboptimal for cross-resolution matching. In addition, it may propagate noisy supervision from degraded regions when forcing student stage to mimic all teacher features under large resolution discrepancy. For this, a confidence-guided selective distillation strategy is introduced, in which only high-confidence teacher regions participate in feature transfer.

Specifically, teacher net is frozen after {\mathcal{L}}_{\mathrm{b}\mathrm{a}\mathrm{s}\mathrm{e}} has converged. As to the student net, its low-path backbone is fixed to the low-path backbone of teacher net, and the low path-specific module is initialized as the low path-specific module of teacher net. To avoid unreliable supervision from degraded LR regions, a selective distillation loss guided by teacher confidence map {\mathit{C}}^t is constructed as

(8){\mathcal{L}}_{\mathrm{s}\mathrm{d}\mathrm{l}}=\frac{\sum _{(i,j)\in P_{\mathrm{t}\mathrm{o}\mathrm{p}}}{\mathit{C}}_{ij}^t[1-{\mathcal{D}}_{i^{\prime }{j}^{\prime }:}^l({\mathcal{D}}_{ij:}^h{)}^{\mathrm{\top }}]}{\sum _{(i,j)\in P_{\mathrm{t}\mathrm{o}\mathrm{p}}}{\mathit{C}}_{ij}^t}

where P_{\mathrm{t}\mathrm{o}\mathrm{p}}=\left\{(i,j)\mid {\mathit{C}}_{ij}^t>\gamma \right\} with {\mathit{C}}^t being the element-wise multiplication of {\mathit{S}}^h and {\mathit{R}}^h, and \gamma >0 controlling distillation region. Eq. (8) shows that only positions within P_{\mathrm{t}\mathrm{o}\mathrm{p}} participate in such descriptor distillation. Moreover, teacher confidence {\mathit{C}}^t acts as a soft weighting factor, assigning stronger supervision to reliable regions. To further align LR confidence to HR counterpart, a full confidence distillation loss is defined as

(9){\mathcal{L}}_{\mathrm{c}\mathrm{d}\mathrm{l}}=\frac{1}{mn}\sum _{i,j=1}^{m,n}({\mathit{C}}_{ij}^s-{\mathit{C}}_{ij}^t{)}^2

where {\mathit{C}}^s={\mathit{S}}^l\cdot {\mathit{R}}^l with symbol “\cdot” being the element-wise multiplication. Therefore, the student-stage loss is

(10){\mathcal{L}}_S={\mathcal{L}}_{\mathrm{s}\mathrm{d}\mathrm{l}}+\omega {\mathcal{L}}_{\mathrm{c}\mathrm{d}\mathrm{l}}

where \omega serves as a balance factor.

3 Experiments and Analysis

3.1 Experimental Setup

Experiments are conducted on the Harvard dataset [21], which provides 50 hyperspectral images of size 512\times 512\times 31. To simulate cross-resolution HS-MS registration scenarios, each original image is degraded into LR-HS observations through Gaussian blur followed by spatial downsampling of ratio s=2 and s=4 [22]. HR-MS observations of size 512\times 512\times 3 are generated by spectrally degrading 31 to 3 bands [23]. Among the 50 scenes, 39 pairs are used for training and 11 pairs for testing. Each training HS image is augmented by random rotation in angle range [-{90}^{\circ },{90}^{\circ }], random scale in variation range [0.9,1.1], and random horizontal and vertical translations within [-0.25w,0.25w] and [-0.25h,0.25h], respectively, to generate 200 image pairs. All testing HS images are rotated in an angle randomly selected from [0^{\circ },{180}^{\circ }] and translated by displacements within [-64,64] along horizontal and vertical directions respectively. Different settings between training and testing data help evaluate generalization of all methods.

Both teacher and student networks are optimized using Adam with an initial learning rate of 1\times {10}^{-4}, a weight decay of 5\times {10}^{-4}, and a batch size of 8. Teacher net is trained for 20 epochs; subsequently, student net is trained for 10 epochs. Patch size N is set as 16. During the distillation stage, \gamma is empirically set as the 85-th percentile of teacher confidence score.

Four metrics are adopted to evaluate registration performance, including the number of correct matches (NCM), registration root mean square error (RMSE), area under the curve (AUC), and success rate (SR). Higher NCM, AUC, and SR indicate better registration performance, whereas lower RMSE corresponds to more accurate geometric estimation. Since our work focuses on the paired descriptor establishment, NCM is used as a priority metric. Others are used as auxiliaries for comprehensive evaluation as well as a comparative analysis with template-based methods.

3.2 Regularization Parameters

Regularization parameters \lambda and \omega are manually tuned, and both are determined experimentally. Under a fixed \omega, \lambda is searched within [0,2] with a step size of 0.2. As shown by the NCM and RMSE values listed in Tab. 1, the proposed CTSF performs the best registration when \lambda =1.4. Then, \omega is searched in the same way when fixing \lambda as 1.4. According to the best performance, \lambda and \omega are set as 1.4 and 1.0, respectively, for the following experiments.

3.3 Comparative Experiments

For a comparative analysis, CTSF is evaluated against traditional feature-based methods such as SIFT [1] and PIIFD [2], template-based methods such as SSD [3] and NMI [4], and recent deep methods such as LoFTR [6] and R2D2 [10]. Quantitative metrics between HS image before affine transformation and HS image aligned by compared methods are calculated in Tab. 2, where “2\times” and “4\times” denote different spatial resolution ratios between MS and HS, corresponding to s=2 and s=4, respectively. Some corresponding visualized results are presented in Fig. 2, while feature matching results of feature-based methods are shown in Fig. 3.

As shown in Tab. 2, all considered methods remain reasonably effective under the 2\times setting. It is worth noting that LoFTR and R2D2 achieve stronger matching performance than SIFT, as shown by their NCM values. This can be attributed to the stronger feature representation capability of deep networks. Although LoFTR achieves a relatively high NCM, its large RMSE suggests occasional failures under challenging affine conditions, which significantly affect the average error. The proposed CTSF achieves excellent and competitive NCM, RMSE, AUC, and SR, demonstrating our establishment of denser and more reliable correspondences.

When the resolution ratio increases to 4\times, the performance of all competing methods deteriorates noticeably. Traditional feature-based and template-based methods become markedly less reliable because local structures and intensity consistency are severely degraded in LR observations. Deep learning-based methods also exhibit noticeable performance degradation, especially LoFTR which is primarily designed for iso-resolution matching scenarios. Although R2D2 remains comparatively competitive, its lack of explicit cross-resolution modeling still limits its robustness under large resolution discrepancies. Our CTSF achieves the best NCM, RMSE, and SR while maintaining competitive AUC, which can be attributed to the proposed hierarchical shared representation and confidence-guided selective distillation. Fig. 2 and Fig. 3 demonstrate that the proposed CTSF produces denser and more spatially consistent correspondences, thereby enabling more accurate geometric estimation under severe cross-resolution conditions. These results indicate that CTSF is particularly suitable for cross-resolution image registration with large resolution discrepancies.

3.4 Ablation Study

To verify the contribution of each key component in our CTSF, ablation experiments are conducted under both 2\times and 4\times settings. Corresponding metrics are listed in Tab. 3, where “w/o Align”, “w/o CGD”, and “w/o HSW” correspond to removing the self-alignment constraint {\mathcal{L}}_{\mathrm{c}\mathrm{r}\mathrm{c}}, the confidence-guided distillation loss \mathcal{L}{}_{\mathrm{s}\mathrm{d}\mathrm{l}}, and the parameter sharing mechanism from the whole CTSF.

As shown in Tab. 3, the full model consistently delivers the strongest performance under both the 2\times and 4\times settings. “w/o Align” shows the most severe performance degradation, demonstrating that explicit alignment constraint of cross-resolution descriptors is essential to reduce description inconsistency caused by resolution difference. The ablation of hierarchical parameter sharing degrades registration performance significantly, particularly under the more challenging 4\times setting. Therefore, the shared learning between cross-resolution representations plays a critical role in improving feature transferability from HR to LR scenes. In addition, NCM and AUC are reduced while RMSE is increased when removing selective distillation, although SR remains competitive under the 2\times setting. It reveals that confidence-guided knowledge transfer is effective to suppress unreliable supervision from degraded LR regions and then provide reliable descriptor correspondences.

In order to study the robustness of above ablation components to geometric transformation, ablation experiments under different translations and rotations are conducted in Fig. 4. Obviously, the full CTSF is consistently superior to all ablated variants when changing translation pixel and rotation angle, which further demonstrates the effectiveness of each technological improvement. Smoother NCM and RMSE curves are generated by our CTSF than others, revealing its stronger robustness to geometric transformation. Moreover, CTSF still preserves stable registration performance within the rotation range [{90}^{\circ },{180}^{\circ }] that is not covered by the training set. It certainly indicates that the proposed CTSF could be generalized to different geometric transformations.

3.5 Robustness Under Random Noise

To evaluate noise robustness, Poisson and Gaussian noises with intensities randomly sampled from [0,0.03] are added to both normalized HR and LR observations during testing. The quantitative and visual results are reported in Tab. 4 and Figs. 5–6, respectively.

Compared with Tab. 2, all methods show performance degradation, but several methods still preserve acceptable registration accuracy under the 2\times setting. As the resolution ratio increases to 4\times, CTSF achieves the best RMSE and NCM while maintaining competitive AUC and SR, indicating stronger robustness under severe cross-resolution noise perturbations. Furthermore, Fig. 5 shows that our CTSF achieves more stable cross-resolution registration under noise perturbations, and Fig. 6 presents its corresponding matching results. Overall, CTSF shows competitive noise robustness.

4 Conclusion

A cross-resolution registration framework CTSF is proposed for hyper- and multi-spectral imagery in this paper. Besides a dual-path hierarchical shared teacher network, a single-path hierarchical shared student network is designed for cross-resolution description. In addition, a confidence-guided distillation strategy and a cross-resolution alignment regularization are constructed to enhance robustness and effectiveness of cross-resolution description. Simulated experiments verified the effectiveness and superiority of the proposed framework, particularly under large resolution discrepancies. Future work will study the possible generalization of CTSF to register and fuse images with larger resolution ratios.

References

[1] D. G. Lowe . Distinctive image features from scale-invariant keypoints. International Journal of Computer Vision, 2004, 60(2): 91–110

[2] C. Gao and W. Li, “Multi-scale HARRIS-PIIFD features for registration of visible and infrared images,” in 2021 IEEE International Geoscience and Remote Sensing Symposium (IGARSS), pp. 5437-5440, 2021.

[3] B. Zitová , J. Flusser . Image registration methods: A survey. Image and Vision Computing, 2003, 21(11): 977–1000

[4] C. Studholme , D. L. G. Hill , D. J. Hawkes . An overlap invariant entropy measure of 3D medical image alignment. Pattern Recognition, 1999, 32(1): 71–86

[5] P. Lindenberger, P. E. Sarlin, and M. Pollefeys, “LightGlue: Local feature matching at light speed,” in Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 17627-17638, 2023.

[6] J. Sun, Z. Shen, Y. Wang, H. Bao, and X. Zhou, “LoFTR: Detector-free local feature matching with transformers,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 8922-8931, 2021.

[7] G. A. Potje, F. Cadar, A. Araujo, R. Martins, and E. R. Nascimento, “XFeat: Accelerated features for lightweight image matching,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2682-2691, 2024.

[8] D. DeTone, T. Malisiewicz, and A. Rabinovich, “SuperPoint: Self-supervised interest point detection and description,” in 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 224-236, 2018.

[9] P. E. Sarlin, D. DeTone, T. Malisiewicz, and A. Rabinovich, “SuperGlue: Learning feature matching with graph neural networks,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4938-4947, 2020.

[10] J. Revaud, C. R. de Souza, M. Humenberger, and P. Weinzaepfel, “R2D2: Reliable and repeatable detector and descriptor,” in Advances in Neural Information Processing Systems 32 (NeurIPS 2019), pp. 12405-12415, 2019.

[11] J. Edstedt, I. Athanasiadis, M. Wadenbäck, and M. Felsberg, “DKM: Dense kernelized feature matching for geometry estimation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 17765-17775, 2023.

[12] K. Ren , W. Sun , X. Meng , G. Yang , J. Peng , J. Huang , J. Li . CDFSL: Image registration for spaceborne hyperspectral and multispectral data having large spatial-resolution difference. IEEE Transactions on Geoscience and Remote Sensing, 2023, 61: 1–15

[13] G. Hinton, O. Vinyals, and J. Dean, “Distilling the knowledge in a neural network,” in NIPS 2014 Deep Learning and Representation Learning Workshop, 2014.

[14] C. Shi , J. Li , J. Gong , B. Yang , G. Zhang . An improved lightweight deep neural network with knowledge distillation for local feature extraction and visual localization using images and LiDAR point clouds. ISPRS Journal of Photogrammetry and Remote Sensing, 2022, 184: 177–188

[15] J. Lee, Y. Jeong, S. Kim, J. Min, and M. Cho, “Learning to distill convolutional features into compact local descriptors,” in Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pp. 898-908, 2021.

[16] X. Guo, J. Hu, H. Bao, and G. Zhang, “Descriptor distillation for efficient multi-robot SLAM,” in 2023 IEEE International Conference on Robotics and Automation (ICRA), pp. 6210-6216, 2023.

[17] M. A. Fischler , R. C. Bolles . Random sample consensus: A paradigm for model fitting with applications to image analysis and automated cartography. Communications of the ACM, 1981, 24(6): 381–395

[18] K. He, Y. Lu, and S. Sclaroff, “Local descriptors optimized for average precision,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 596-605, 2018.

[19] E. B. Baum and F. Wilczek, “Supervised learning of probability distributions by neural networks,” in Neural Information Processing Systems, pp. 52-61, 1988.

[20] X. Chen and K. He, “Exploring simple siamese representation learning,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 15750-15758, 2021.

[21] A. Chakrabarti and T. Zickler, “Statistics of real-world hyperspectral images,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 193-200, 2011.

[22] N. Yokoya , T. Yairi , A. Iwasaki . Coupled nonnegative matrix factorization unmixing for hyperspectral and multispectral data fusion. IEEE Transactions on Geoscience and Remote Sensing, 2012, 50(2): 528–537

[23] R. Dian , S. Li , L. Fang . Learning a low tensor-train rank representation for hyperspectral image super-resolution. IEEE Transactions on Neural Networks and Learning Systems, 2019, 30(9): 2672–2683

Rights & permissions

The associate editor coordinating the review of this manuscript was Dr. Yixiao Yang