An Extended Depth-of-Field Microscope Field-of-View Stitching-Based High-Throughput Multimodal Imaging Method

Abstract

Optical microscopes are essential tools for scientific research, but traditional microscopes are restricted to capturing only twodimensional (2D) texture information, lacking comprehensive threedimensional (3D) morphology capabilities. Additionally, traditional microscopes are inherently constrained by the limited spacebandwidth product of optical systems, resulting in restricted depth of field (DOF) and field of view (FOV). Attempts to expand DOF and FOV typically come at the cost of diminished resolution. In this paper, we propose a texturedriven FOV stitching algorithm specifically designed for extended depthoffield (EDOF) microscopy, allowing for the integration of 2D texture and 3D depth data to achieve highresolution, highthroughput multimodal imaging. Experimental results demonstrate an 11fold enhancement in DOF and an 8fold expansion in FOV compared to traditional microscopes, while maintaining axial resolution after FOV extension.

Keywords

extended depth-of-field microscopy / field-of-view stitching / iterative closest point

1 Introduction

Optical microscopes have played a crucial role as ability to capture only low-throughput, single-modality texture images, which prevents them from indispensable tools in scientific research, particularly in observing the microscopic world. Nevertheless, traditional microscopes are limited by their inability to meet the demands of high-throughput and complex sample analysis [ 1 ]. Specifically, on the one hand, traditional microscopes primarily rely on \(2\mathrm{D}\) imaging, which can only capture the surface texture information of objects without providing comprehensive \(3\mathrm{D}\) structural and morphological details. This limitation poses significant challenges in fields such as biomedicine and materials science, where \(3\mathrm{D}\) morphological analysis is often essential [ 2 ]. On the other hand, due to the inherent limitations of the space-bandwidth product in optical systems, traditional microscopes generally have a constrained depth of field (DOF) and a limited field of view (FOV). Attempts to expand the DOF or FOV often come at the expense of image resolution [ 3 ]. These inherent limitations in imaging modalities and information throughput significantly hinder the applicability of traditional optical microscopes in high-throughput, multimodal imaging scenarios that are required for complex applications.

To address these challenges, extended depth-of-field (EDOF) microscopy has gained considerable attention in recent years [ 4 ]. By fusing image data from multiple focal planes, EDOF microscopy can achieve an increased DOF by fusing image data from multiple focal planes, thus enabling the capture of high-resolution images with an extended depth range [ 5 ]. This approach effectively mitigates the limitations of traditional microscopes in terms of DOF and resolution. Additionally, EDOF microscopy enables 3D depth reconstruction through focus metric analysis across focal planes [ 6 ], providing new opportunities for studying complex samples. However, despite the significant advantages of EDOF microscopy in improving DOF and resolution, its FOV remains limited. In scenarios that require extensive coverage, such as circuit board, wafer, biological tissue samples, and chip inspection, EDOF microscopy still falls short [ 3 ]. Existing research has largely neglected the development of methods for expanding the FOV in EDOF microscopy, creating a gap in addressing the need for wide-area, high-throughput imaging.

Research on \(2\mathrm{D}\) texture image stitching is extensive. Current methods focus on feature extraction and matching [ 7 ], handling stitching errors [ 8 ], and ensuring color consistency and seamless visual transitions [ 9 ] in the stitched images. State-of-the-art approaches include feature-based matching algorithms such as scale-invariant feature transform (SIFT)[ 10 ] and speeded-up robust features (SURF)[ 11 ], optimization algorithms for image registration and stitching like random sample consensus (RANSAC)[ 12 ]. To ensure the seamlessness of the final stitched image, weighted blending, multi-band blending [ 13 ], or optimal seam blending methods [ 14 ] are often employed to optimize the fusion effect. Recent advancements incorporate deep learning for parallax-tolerant stitching [ 15 , 16 ] , achieving sub-pixel alignment accuracy even in low-texture scenarios.

In comparison, stitching 3D depth data poses even greater challenges. On the one hand, 3D depth data are typically weak in features [ 17 ], making it difficult to identify significant feature points during the stitching and matching process. This makes direct feature matching of depth images particularly challenging [ 18 ]. On the other hand, the 3D depth data reconstructed from EDOF microscopy suffer from limited axial resolution, making it difficult to align them directly using classic algorithms like the iterative closest point (ICP)[ 19 ] algorithm. When initial alignment errors are significant, ICP tends to get trapped in local optima, leading to substantial alignment inaccuracies. In recent years, most research on rigid registration of \(3\mathrm{D}\) data has focused on improving the ICP algorithm. Researchers have developed ICP variants targeting weak features, such as normal-based iterative closest point (NICP)[ 20 ], which enhances robustness by entire solution space to ensure that a global optimum is reached. Other methods combine ICP with feature-based point cloud matching algorithms, employing descriptors like fast point feature histogram (FPFH)[ 21 ] and Signature of histograms of orientations (SHOT)[ 22 ] to enhance global information in point cloud matching. However, these approaches overlook critical EDOF-specific advantages -particularly the co-registered high resolution texture information that could drive multimodal registration. To the best of our knowledge, no targeted algorithm specifically addresses EDOF microscope FOV stitching through texture-guided 3D alignment.

Based on the current state of research, this paper proposes a novel high-throughput multimodal imaging technique based on field stitching technology using an EDOF microscope. This approach fully leverages the large space-bandwidth product of the EDOF microscope, along with its combined texture and depth imaging capabilities. We designed a stitching algorithm named texture-driven FOV stitching algorithm specifically for the EDOF microscope. The algorithm drives multimodal data registration and fusion through image texture, ultimately achieving large FOV, large DOF, and high-resolution multimodal imaging.

2 Proposed Method

As shown in Fig. 1 , we use an EDOF microscope to acquire a multi-focus image sequence. By evaluating the focus quality and reconstructing the information from the multi-focus images, we obtain a high resolution, large DOF texture image and a 3D depth map for the local FOV. Concurrently, the confidence of the \(3\mathrm{D}\) depth map is estimated based on the focus evaluation of the multi-focus image sequence. By integrating the aforementioned confidence information, the texture-feature driven FOV stitching algorithm is then applied to solve the translational and rotational parameters between different FOV, driven by the EDOF texture information from different views. Ultimately, a 3D affine transformation matrix between different FOV is derived.

2.1 Principle of the EDOF

As shown in Fig. 2 , the EDOF microscope drives the objective lens to move axially via a translation stage, causing the focal plane to also move along the axial direction. Through layer-by-layer scanning and capturing images at different focal planes, a multi-focus image sequence \({I}_{1},{I}_{2},\cdots ,{I}_{N}\) is obtained, where \(N\) represents the number of images. Subsequently, the focus quality of the images is evaluated on a pixel-by-pixel basis using a focus evaluation method based on gray-level variance:

(1) \[{H}_{n}\left({x, y}\right)= \frac{1}{MN}\mathop{\sum }\limits_{x}\mathop{\sum }\limits_{y}{\left({I}_{n}\left( x, y\right)- \mu \right)}^{2}\]

where(x, y)denotes the pixel coordinates, \(M\) and \(N\) represent the width and height of the image, respectively, and \(\mu\) represents the mean gray level within the pixel neighborhood. The variable \(H\) denotes the focus evaluation value. Through the focus evaluation, we obtain a focus evaluation volume \({H}_{1},{H}_{2},\cdots ,{H}_{N}\) .

For any given pixel coordinate \(\mathbf{p}= \left({x, y}\right)\) , the variation of the focus evaluation value with respect to the image index forms a focus evaluation curve \({H}_{\left( p\right)}\left( n\right)\) . As the sequence moves from focus to defocus and back to focus, the focus evaluation curve exhibits a Gaussian-shaped curve. By extracting the image index corresponding to the maximum value of the focus evaluation curve:

(2) \[ G\left({x, y}\right)= \mathop{\operatorname{argmax}}\limits_{n}{H}_{n}\left({x, y}\right)\]

We can obtain the depth information of the focal plane and construct a 3D depth map:

(3) \[ T\left({x, y}\right)= G\left({x, y}\right)\text{ step }\]

where step represents the interval between the focal planes of consecutive images during layer-by-layer scanning. Furthermore, by extracting the pixel values from the images at the corresponding focal plane positions, an all-in-focus image can be obtained.

2.2 Texture-Driven FOV Stitching Algorithm

As shown in Fig. 3 , by driving the target object to scan along the radial direction using a translation stage, results from different FOV can be acquired. In this paper, we propose a FOV stitching algorithm tailored for the EDOF microscope, which performs registration and fusion of the \(2\mathrm{D}\) texture images and 3D depth maps from different FOV. This approach enables large FOV imaging and measurement, further enhancing the information throughput of the target acquisition.

2.2.1 2D Coarse Registration

Before image stitching process, the input images are first preprocessed to compensate the brightness, saturation to make the two images basically consistent under the visual effect of the human eye. After preprocessing, the core of image stitching lies in determining the correspondences between adjacent images, which involves calculating the transformation matrix through feature point matching. In this study, SIFT [ 10 ] is used to detect and describe key feature points within adjacent images. Following feature extraction, \(k\) -nearest neighbor (KNN) matching [ 23 ] of the descriptor sets based on the L2 norm is utilized to establish feature matching. To ensure reliable matches, Lowe’s ratio test [ 24 ] is employed, which deems a match reliable if the distance of the closest match is below a certain threshold compared to the second closest match. Given potential issues such as noise, repetitive structures, or occlusion, RANSAC [ 12 ] is used to robustly estimate the transformation relationship between the images. For any two adjacent FOV, a 2D affine transformation matrix is ultimately obtained:

(4) \[{\mathbf{T}}_{2;{m}_{1}{m}_{2}}= \left\lbrack \begin{matrix}{R}_{11}^{\left( 0\right)} &{R}_{12}^{\left( 0\right)} &{t}_{x}^{\left( 0\right)} \\{R}_{21}^{\left( 0\right)} &{R}_{22}^{\left( 0\right)} &{t}_{y}^{\left( 0\right)} \\ 0 & 0 & 1 \end{matrix}\right\rbrack \]

where \({R}_{11}^{\left( 0\right)},{R}_{12}^{\left( 0\right)},{R}_{21}^{\left( 0\right)}\) , and \({R}_{22}^{\left( 0\right)}\) collectively describe the rotation of the target along the optical axis ( \(z\) -axis) between different FOV, while \({t}_{x}^{\left( 0\right)}\) and \({t}_{y}^{\left( 0\right)}\) represent the translational shifts along the radial directions ( \(x\) - and \(y\) -axis). However, mechanical imperfections in the translation stage may introduce \(3\mathrm{D}\) rotational misalignment or axial positional errors, which cannot be fully corrected by a 2D affine model.

2.2.2 3D Fine Registration

We further treat the imaging results from the EDOF microscope as a point cloud, and perform registration in 3D space. Based on the ICP algorithm, the 2D affine transformation obtained from 2D texture features is used as the initial parameterization. To initialize the \(3\mathrm{D}\) registration process, the 2D affine transformation is extended to \(3\mathrm{D}\) space by assuming zero initial displacement and rotation in the depth direction. Combined with the peak value information from the focus evaluation curve used in the 3D depth reconstruction of the EDOF microscope, a more accurate 3D rigid transformation matrix is computed. Specifically, on the one hand, the registration result based on \(2\mathrm{D}\) texture features takes full advantage of the strong texture characteristics. We use this result to construct the initial value for the iteration of the ICP algorithm, driving the algorithm to start from a position close to the optimal solution:

(5) \[{\mathbf{T}}_{3;{m}_{1}{m}_{2}}^{\left( 0\right)} =\left\lbrack \begin{matrix}{R}_{11}^{\left( 0\right)} &{R}_{12}^{\left( 0\right)} & 0 &{t}_{x}^{\left( 0\right)} \\{R}_{21}^{\left( 0\right)} &{R}_{22}^{\left( 0\right)} & 0 &{t}_{y}^{\left( 0\right)} \\ 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 1 \end{matrix}\right\rbrack \]

The superscript (0) denotes the 0 th iteration, which represents the initial value for iteration. In the standard ICP algorithm, all point pairs are treated equally, which works well when data quality is relatively uniform. However, when dealing with noisy or unevenly sampled data, equal weighting may cause the algorithm to be sensitive to outliers, thereby affecting the registration results. According to [ 25 ], since regions with rich texture details often more clearly reflect the differences in the focus-defocus process on the focus evaluation curve, the estimated depth is thus more reliable. We use the maximum value on the focus evaluation curve to reflect the confidence level of the depth reconstruction result at each pixel coordinate:

(6) \[ C\left({x, y}\right)= \max {H}_{n}\left({x, y}\right)\]

Based on this confidence, we further construct the weighting values for point cloud matching:

(7) \[{w}_{i}= {C}_{m1}\left({p}_{i}\right){C}_{m2}\left({q}_{i}\right)\]

where \({m}_{1}\) and \({m}_{2}\) represent the confidence levels of two adjacent FOV, \({p}_{i}\) represents the \(i\) -th point in the source point cloud, and \({q}_{i}\) represents any matching point in the target point cloud corresponding to \({p}_{i}\) .

Using these weights, the weighted ICP method iteratively minimizes the objective function to achieve better registration results:

(8) \[ E\left({\mathbf{R},\mathbf{t}}\right)= \mathop{\sum }\limits_{{i = 1}}{w}_{i}\mathbf{R}{p}_{i}+ \mathbf{t}- {q}_{i}^{2}\]

(9) \[\mathbf{R}= \left\lbrack \begin{array}{lll}{R}_{11}& {R}_{12}& {R}_{13}\\{R}_{21}& {R}_{22}& {R}_{23}\\{R}_{31}& {R}_{32}& {R}_{33}\end{array}\right\rbrack \]

(10) \[\mathbf{t}= {\left\lbrack \begin{array}{lll}{t}_{x}& {t}_{y}& {t}_{z}\end{array}\right\rbrack }^{\mathrm{T}}\]

where \(\mathbf{R}\) represents the \(3\mathrm{D}\) rotation matrix, and \(\mathbf{t}\) represents the 3D translation matrix. \(E\left({\mathbf{R},\mathbf{t}}\right)\) represents the sum of the point-to-point distance errors between the source point cloud and the target point cloud. After solving the weighted ICP, we obtain the 3D rigid transformation matrix:

(11) \[{\mathbf{T}}_{3;{m1m2}}= \left\lbrack \begin{matrix}\mathbf{R}& \mathbf{t}\\\mathbf{0}& \mathbf{1}\end{matrix}\right\rbrack \]

Once this matrix is obtained, the spatial alignment between two adjacent FOV is achieved. Furthermore, in this paper, we apply some well-established techniques to fuse the overlapping information between two FOV to eliminate perceptible differences in the stitched images. For the \(2\mathrm{D}\) texture images, we employ optimal seamline optimization [ 9 ] and feather blending. Optimal seamline optimization technique which is particularly effective in complex image stitching scenarios, not only considers direct pixel differences but also incorporates gradient and texture information to ensure a visually inconspicuous seam. To further smooth the seam area, feather blending is utilized to enhance the overall visual quality of the stitched image. Feather blending not only smooths the transition region but also retains important visual information near edges, making the stitched image appear natural and coherent. For the \(3\mathrm{D}\) depth data fusion, we implement a linear weighted averaging method. The weight for each depth value is determined by the spatial proximity between corresponding points in adjacent FOVs.

3 Experiment

We built a system based on the EDOF microscope and texture feature-driven FOV stitching algorithm to verify the high-throughput multimodal imaging performance. As illustrated in Fig. 4 , the system consists of an EDOF microscope and an \({XYZ}\) translation stage. The system’s magnification options include \(\times {0.5},\times {1.0}\) , and \(\times {2.0}\) . The \({XYZ}\) translation stage has an accuracy of \({1\mu }\mathrm{m}\) . Resolution preservation after multi-focus fusion was confirmed by imaging a resolution board, showing no degradation from the native resolution. As shown in Fig. 5 , the uppermost image presents the single-frame imaging result reconstruction utilizing a multi-focus image sequence comprising 20 frames, with each frame’s focal plane separated by \({0.05}\mathrm{\;{mm}}\) . This achieves an effective DOF of \({1.0}\mathrm{\;{mm}}\) , representing an \({11}\times\) increase over the baseline single-image DOF of \({90.9\mu }\mathrm{m}\) . The lowermost row shows the stitching result of a \(5 \times 3\) grid of local fields, totaling 15 regions with \({4.88}\mathrm{\;{mm}}\) spacing. After stitching, the effective FOV reaches \({87.6}{\mathrm{\;{mm}}}^{2}\) , an \(8 \times\) expansion from the original \({11.2}{\mathrm{\;{mm}}}^{2}\) single-frame FOV.

To quantitatively analyze the changes after stitching, we selected a standard gauge block as the test sample for evaluating axial resolution before and after field stitching. The gauge block has a tolerance grade of 0 , flatness less than or equal to \({0.5\mu }\mathrm{m}\) , and height values of1.05,1.10, and \({1.15}\mathrm{\;{mm}}\) . Fig. 6 (a) shows the results with EDOF and field stitching for a gauge block with a height of \({1.10}\mathrm{\;{mm}}\) in two adjacent FOVs. Fig. 6 (b) presents the height profiles of a local cross-section, where the red and blue lines represent the sectional height profiles of the two local fields, and the black line represents the sectional height profile after stitching. The abrupt change on the left side of the curve indicates the measured height of the gauge block, and the overlapping region of the red and blue curves corresponds to the overlapping area between the two fields. Fig. 6 (c) shows the results from multiple measurements performed on the gauge block. During the measurement process, we ensured uniform and consistent illumination conditions, randomly translating or rotating the gauge block sample, and conducted at least three sets of repeatability experiments for each sample.

The sample was moved \(5\mathrm{\;{mm}}\) per step in the \(x\) and \(y\) directions, and \({0.2}\mathrm{\;{mm}}\) per step in the \(z\) direction. The single-field imaging results indicate that the average surface height of the gauge block is \({1.101}\mathrm{\;{mm}}\) , with a roughness root mean square error (RMSE) of \({0.019}\mathrm{\;{mm}}\) . After stitching the two fields, the measured average height is \({1.102}\mathrm{\;{mm}}\) , and the roughness RMSE is \({0.018}\mathrm{\;{mm}}\) , which is essentially consistent with the measurement results. The experiment indicates that the axial resolution of the system remains unchanged before and after FOV stitching.

4 Conclusion

In this study, we proposed a novel high-throughput, multimodal imaging method based on EDOF microscopy with a texture-driven FOV stitching algorithm. By integrating multi-focus images, this technique achieves high-resolution, large DOF texture imaging and constructs \(3\mathrm{D}\) depth data of the target. The texture-driven field stitching algorithm we developed allows for accurate registration and seamless merging of \(2\mathrm{D}\) texture and 3D depth data across multiple FOVs. Experimental results demonstrate an 11-fold improvement in DOF and an 8-fold FOV expansion compared to conventional microscopy, with axial resolution remaining consistent pre- and post-stitching. This high-throughput multimodal imaging method provides a powerful tool for applications requiring extensive field coverage and depth information, such as biomedical and materials sciences, where detailed structural analysis is essential.

The current algorithm is tailored for EDOF microscopy, and its cross-platform applicability requires further validation. Future efforts will explore deep learning integration and hardware optimization to enable realtime processing of high-resolution data.

References

[1] M. Oheim . High-throughput microscopy must re-invent the microscope rather than speed up its functions [J]. British Journal of Pharmacology , 2007 . 152 ( 1 ): 1 - 4 .

[2] J. Mertz . Optical sectioning microscopy with planar or structured illumination [J]. Nature Methods , 2011 . 8 ( 10 ): 811 - 819 .

[3] S. Jeon , J. Lee , K. Kim , S. M. Hong , B. H. Oh , K. H. Kim . Extended depth-of-field wide-field fluorescence microscopy with a micro-mirror array lens system for versatile cellular examination [J]. Optics Letters , 2024 . 49 ( 12 ): 3368 - 3371 .

[4] A. S. Ambikumar , D. G. Bailey , G. Sen Gupta . Extending the depth of field in microscopy: A review [C]// International Conference on Image and Vision Computing, New Zealand , 2016 . 185 - 190 .

[5] E. J. Botcherby , M. J. Booth , R. Juškaitis , T. Wilson . Real-time extended depth of field microscopy [J]. Optics Express , 2008 . 16 ( 26 ): 21843 - 21848 .

[6] A. G. Valdecasas , D. Marshall , J. M. Becerra , J. J. Terrero . On the extended depth of focus algorithms for bright field microscopy [J]. Micron , 2001 . 32 ( 6 ): 559 - 569 .

[7] Y. Xie , Q. Wang , Y. Chang , X. Zhang . Fast target recognition based on improved ORB feature [J]. Applied Sciences , 2022 . 12 ( 2 ): 786 -

[8] W. Y. Lin , S. Liu , Y. Matsushita , T. T. Ng , L. F. Cheong . Smoothly varying affine stitching [C]// IEEE Conference on Computer Vision and Pattern Recognition , 2011 . 345 - 352 .

[9] A. Levin , A. Zomet , S. Peleg , Y. Weiss . Seamless image stitching in the gradient domain [C]// Computer Vision Eccv 2004, Pt 4, Lecture Notes in Computer Science, T. Pajdla and J. Matas, eds , 2004 . 377 - 389 .

[10] D. G. Lowe . Distinctive image features from scale-invariant keypoints [J]. International Journal of Computer Vision , 2004 . 60 ( 2 ): 91 - 110 .

[11] H. Bay , A. Ess , T. Tuytelaars , L. Van Gool . Speeded-up robust features (SURF) [J]. computer vision and image understanding , 2008 . 110 ( 3 ): 346 - 359 .

[12] M. A. Fischler , R. C. Bolles . Random sample consensus: A paradigm for model fitting with applications to image analysis and automated cartography [J]. Communications of the ACM , 1987 . 24 ( 6 ): 381 - 395 .

[13] Y. Zhao , W. Qian , D. Xu , Ieee . Fast multi-band blending using run-length encoding [C]// International Conference on Computer-Aided Design and Computer Graphics CAD GRAPHICS , 2015 . 224 - 225 .

[14] S. Devitsyna T. Balabanova . Seamless image stitching in the gradient domain [C]// ITM Web of Conferences , 2019 . 30 . 04017 -

[15] L. Nie , C. Lin , K. Liao , S. Liu , Y. Zhao . Unsupervised deep image stitching: reconstructing stitched features to images [J]. Ieee Transactions on Image Processing , 2021 . 30 . 6184 - 6197 .

[16] Q. Zhao , Y. Ma , C. Zhu , C. Yao , B. Feng , F. Dai . Image stitching via deep homography estimation [J]. Neurocomputing , 2021 . 450 . 219 - 229 .

[17] F. Tombari , S. Salti , L. Di Stefano . Performance evaluation of 3D keypoint detectors [J]. International Journal of Computer Vision , 2013 . 102 ( 1-3 ): 198 - 220 .

[18] Y. Guo , F. Sohel , M. Bennamoun , M. Lu , J. Wan . Rotational projection statistics for 3D local surface description and object recognition [J]. International Journal of Computer Vision , 2013 . 105 ( 1 ): 63 - 86 .

[19] P. J. Besl , N. D. McKay . A method for registration of 3-D shapes [J]. IEEE Transactions on Pattern Analysis and Machine Intelligence , 1992 . 14 ( 2 ): 239 - 256 .

[20] J. Scrafin G. Grisetti . NICP: Dense normal based point cloud registration [C]// IEEE International Conference on Intelligent Robots and Systems , 2015 . 742 - 749 .

[21] R. B. Rusu , N. Blodow , M. Beetz , Ieee . Fast point feature histograms (FPFH) for 3D registration [C]// IEEE International Conference on Robotics and Automation ICRA , 2009 . 1848 - 1853 .

[22] F. Tombari , S. Salti , L. Di Stefano . Unique signatures of histograms for local surface description [C]// Lecture Notes in Computer Science , 2010 . 6313 . 356 - 369 .

[23] S. Liu , X. Liang , L. Liu , X. Shen , J. Yang , C. Xu , L. Lin , X. Cao , S. Yang . Matching-CNN meets KNN: Quasi-parametric human parsing [C]// IEEE Conference on Computer Vision and Pattern Recognition , 2015 . 1419 - 1427 .

[24] M. Muja , D. G. Lowe . Fast approximate nearest neighbors with automatic algorithm configuration [C]// International Conference on Computer Vision Theory and Applications , 2009 . 331 - 340 .

[25] S. Onogi , T. Kawase , T. Sugino , Y. Nakajima . Investigation of shape-from-focus precision by texture frequency analysis [J]. Electronics , 2021 . 10 ( 16 ): 1870 -

Funding

Science Foundation of Donghai Laboratory(DH-2022KF01001)