Emerging theories and technologies on computational imaging

Abstract

Computational imaging describes the whole imaging process fromthe perspective of light transport and information transmission, featurestraditional optical computing capabilities, and assists in breakingthrough the limitations of visual information recording. Progressin computational imaging promotes the development of diverse basicand applied disciplines. In this review, we provide an overview ofthe fundamental principles and methods in computational imaging, thehistory of this field, and the important roles that it plays in thedevelopment of science. We review the most recent and promising advancesin computational imaging, from the perspective of different dimensionsof visual signals, including spatial dimension, temporal dimension,angular dimension, spectral dimension, and phase. We also discusssome topics worth studying for future developments in computationalimaging.

Keywords

Computational imaging / Multi-scaleand multi-dimensional / Super-resolution / Femto-photography / 3D reconstruction / Hyperspectral imaging

References

[1] Assion, A., Baumert, T., Bergt, M., , 1998. Control of chemical reactions by feedback-optimized phase-shapedfemtosecond laser pulses. Science, 282(5390):919–922. https://doi.org/10.1126/science.282.5390.919

[2] Backman, V., Wallace, M.B., Perelman, L., , 2000. Detection of preinvasive cancer cells. Nature, 406(6791): 35–36. https://doi.org/10.1038/35017638

[3] Bao, J., Bawendi, M.G., 2015. A colloidal quantum dot spectrometer. Nature, 523(7558):67–70. https://doi.org/10.1038/nature14576

[4] Bifano, T., 2011. Adaptive imaging: MEMS deformablemirrors. Nat. Photon., 5(1):21–23. https://doi.org/10.1038/nphoton.2010.297

[5] Bina, M., Magatti, D., Molteni, M., , 2013. Backscattering differential ghost imaging in turbid media. Phys. Rev. Lett., 110(8), Article 083901. https://doi.org/10.1103/PhysRevLett.110.083901

[6] Brady, D., Gehm, M., Stack, R., , 2012. Multiscalegigapixel photography. Nature, 486(7403):386–389. https://doi.org/10.1038/nature11150

[7] Brenner, D.J., Hall, E.J., 2007. Computed tomography—an increasing source of radiationexposure. New Engl.J. Med., 357:2277–2284. https://doi.org/10.1056/NEJMra072149

[8] Candès, E.J., Romberg, J., Tao, T., 2006. Robustuncertainty principles: exact signal reconstruction from highly incompletefrequency information. IEEE Trans. Inform. Theory, 52(2):489–509. https://doi.org/10.1109/TIT.2005.862083

[9] Chaigne, T., Katz, O., Boccara, A.C., , 2014. Controlling light in scattering media non-invasively using the photoacoustictransmission matrix. Nat. Photon., 8(1):58–64. https://doi.org/10.1038/nphoton.2013.307

[10] Chakrabarti, A., Zickler, T., 2011. Statistics of real-world hyperspectral images. IEEE Conf. on Computer Vision and Pattern Recognition, p.193–200. https://doi.org/10.1109/CVPR.2011.5995660

[11] Chao, T.H., Zhou, H., Xia, X., , 2005. Near IR electroopticimaging Fourier transform spectrometer. Proc. Optical Pattern Recognition, p.163–172. https://doi.org/10.1117/12.607968

[12] Charles, A.S., Olshausen, B.A., Rozell, C.J., 2011. Learningsparse codes for hyperspectral imagery. IEEE J. Sel. Topics Signal Process., 5(5):963–978. https://doi.org/10.1109/JSTSP.2011.2149497

[13] Choi, W., Fang-Yen, C., Badizadegan, K.R., , 2007. Tomographic phase microscopy. Nat. Meth., 4(9):717–719. https://doi.org/10.1038/NMETH1078

[14] Cotte, Y., Toy, F., Jourdain, P., , 2013. Marker-free phase nanoscopy. Nat. Photon., 7(2):113–117. https://doi.org/10.1038/nphoton.2012.329

[15] Cuche, E., Bevilacqua, F., Depeursinge, C., 1999. Digitalholography for quantitative phase-contrast imaging. Opt. Lett., 24(5):291–293. https://doi.org/10.1364/OL.24.000291

[16] Delalieux, S., Auwerkerken, A., Verstraeten, W.W., , 2009. Hyperspectral reflectance and fluorescence imaging to detect scabinduced stress in apple leaves. Remote Sens., 1(4):858–874. https://doi.org/10.3390/rs1040858

[17] Descour, M., Dereniak, E., 1995. Computed-tomography imaging spectrometer: experimentalcalibration and reconstruction results. Appl. Opt., 34(22):4817–4826. https://doi.org/10.1364/AO.34.004817

[18] Diaspro, A., Chirico, G., Collini, M., 2005. Two-photonfluorescence excitation and related techniques in biological microscopy. Q. Rev. Biophys., 38(02):97–166. https://doi.org/10.1017/S0033583505004129

[19] Ding, W., Wang, Y., Chen, H., , 2014. Plasmonicnanocavity organic light-emitting diode with significantly enhancedlight extraction, contrast, viewing angle, brightness, and low-glare. Adv. Funct. Mater., 24(40):6329–6339. https://doi.org/10.1017/S0033583505004129

[20] Ferguson, R., Phillips, W., 1967. High-resolution nuclear magnetic resonance spectroscopy. Science, 157(3786): 257–267. https://doi.org/10.1126/science.157.3786.257

[21] Fienup, J.R., 1982. Phase retrieval algorithms: a comparison. Appl. Opt., 21(15):2758–2769. https://doi.org/10.1364/AO.21.002758

[22] Fienup, J.R., 2013. Phase retrieval algorithms: a personaltour [invited]. Appl. Opt., 52(1):45–56. https://doi.org/10.1364/AO.52.000045

[23] Frenkel, K.A., 2010. Panning for science. Science, 330(6005):748–749. https://doi.org/10.1126/science.330.6005.748

[24] Gatti, A., Brambilla, E., Bache, M., , 2004. Ghost imaging with thermal light: comparing entanglement and classicalcorrelation. Phys.Rev. Lett., 93(9), Article 093602. https://doi.org/10.1103/PhysRevLett.93.093602

[25] Gebbie, H., 1961. Molecular emission spectroscopy from2μ to 12μ by a michelson interferometer. Nature, 191:264–265. https://doi.org/10.1038/191264a0

[26] Goda, K., Tsia, K., Jalali, B., 2009. Serial time-encodedamplified imaging for real-time observation of fast dynamic phenomena. Nature, 458(7242):1145–1149. https://doi.org/10.1038/nature07980

[27] Greenbaum, A., Luo, W., Su, T.W., , 2012. Imagingwithout lenses: achievements and remaining challenges of wide-fieldon-chip microscopy. Nat. Meth., 9(9):889–895. https://doi.org/10.1038/nmeth.2114

[28] Gustafsson, M.G., 2005. Nonlinear structured-illuminationmicroscopy: wide-field fluorescence imaging with theoretically unlimitedresolution. PNAS, 102(37):13081–13086. https://doi.org/10.1073/pnas.0406877102

[29] Heide, F., Hullin, M.B., Gregson, J., , 2013. Low-budget transient imaging using photonic mixer devices. ACM Trans. Graph., 32(4), Article 45. https://doi.org/10.1145/2461912.2461945

[30] Hein, B., Willig, K.I., Hell, S.W., 2008. Stimulatedemission depletion (STED) nanoscopy of a fluorescent protein-labeledorganelle inside a living cell. PNAS, 105(38):14271–14276. https://doi.org/10.1073/pnas.0807705105

[31] Hell, S.W., Wichmann, J., 1994. Breaking the diffraction resolution limit by stimulatedemission: stimulatedemission-depletion fluorescence microscopy. Opt. Lett., 19(11):780–782. https://doi.org/10.1364/OL.19.000780

[32] Helmchen, F., Denk, W., 2005. Deep tissue two-photon microscopy. Nat. Meth., 2(12):932–940. https://doi.org/10.1038/nmeth818

[33] Hess, S.T., Girirajan, T.P., Mason, M.D., 2006. Ultrahighresolution imaging by fluorescence photoactivation localization microscopy. Biophys. J., 91(11):4258–4272. https://doi.org/10.1529/biophysj.106.091116

[34] Horton, N.G., Wang, K., Kobat, D., , 2013. In vivo three-photonmicroscopy of subcortical structures within an intact mouse brain. Nat. Photon., 7(3):205–209. https://doi.org/10.1038/nphoton.2012.336

[35] Howard, S.S., Straub, A., Horton, N.G., , 2013. Frequency-multiplexed in vivo multiphoton phosphorescence lifetimemicroscopy. Nat.Photon., 7(1):33–37. https://doi.org/0.1038/nphoton.2012.307

[36] Jahr, W., Schmid, B., Schmied, C., , 2015. Hyperspectral light sheet microscopy. Nat. Commun., 6, Article 7990. https://doi.org/10.1038/ncomms8990

[37] Ji, N., Milkie, D.E., Betzig, E., 2010. Adaptiveoptics via pupil segmentation for high-resolution imaging in biologicaltissues. Nat. Meth., 7(2):141–147. https://doi.org/10.1038/nmeth.1411

[38] Kester, R.T., Bedard, N., Gao, L., , 2011. Real-time snapshot hyperspectral imaging endoscope. J. Biomed. Opt., 16(5), Article 056005. https://doi.org/10.1117/1.3574756

[39] Kim, T., Zhou, R., Mir, M., , 2014. White-lightdiffraction tomography of unlabelled live cells. Nat. Photon., 8(3):256–263. https://doi.org/10.1038/nphoton.2013.350

[40] Levoy, M., Hanrahan, P., 1996. Light field rendering. Proc.23rd Annual Conf. on Computer Graphics and Interactive Techniques, p.31–42. https://doi.org/10.1145/237170.237199

[41] Levoy, M., Ng, R., Adams, A., , 2006. Light fieldmicroscopy. ACMTrans. Graph., 25(3):924–934. https://doi.org/10.1145/1141911.1141976

[42] Lin, X., Liu, Y., Wu, J., , 2014. Spatial-spectralencoded compressive hyperspectral imaging. ACM Trans. Graph., 33(6), Article 233. https://doi.org/10.1145/2661229.2661262

[43] Lin, X., Wu, J., Zheng, G., , 2015. Camera arraybased light field microscopy. Biomed. Opt. Expr., 6(9):3179–3189. https://doi.org/10.1364/BODA.2015.JT3A.48

[44] Ma, C., Cao, X., Tong, X., , 2014. Acquisitionof high spatial and spectral resolution video with a hybrid camerasystem. Int. J.Comput. Vis., 110(2):141–155. https://doi.org/10.1007/s11263-013-0690-4

[45] Manley, S., Gillette, J.M., Patterson, G.H., , 2008. High-density mapping of single-molecule trajectories with photoactivatedlocalization microscopy. Nat. Meth., 5(2):155–157. https://doi.org/10.1038/nmeth.1176

[46] Marks, D.L., Son, H.S., Kim, J.,, 2012. Engineeringa gigapixel monocentric multiscale camera. Opt. Eng., 51(8), Article 083202. https://doi.org/10.1117/1.OE.51.8.083202

[47] Morris, P.A., Aspden, R.S., Bell, J.E., , 2015. Imaging with a small number of photons. Nat. Commun., 6, Article 5913. https://doi.org/10.1038/ncomms6913

[48] Nakagawa, K., Iwasaki, A., Oishi, Y., , 2014. Sequentially timed all-optical mapping photography (STAMP). Nat. Photon., 8(9):695–700. https://doi.org/10.1038/nphoton.2014.163

[49] Neifeld, M.A., Shankar, P., 2003. Feature-specific imaging. Appl. Opt., 42(17):3379–3389. https://doi.org/10.1364/AO.42.003379

[50] Ng, R., Levoy, M., Brédif, M., , 2005. Light field photography with a hand-held plenoptic camera. Comput. Sci. Techn. Rep., 2(11):1–11.

[51] Orth, A., Tomaszewski, M.J., Ghosh, R.N., , 2015. Gigapixel multispectral microscopy. Optica, 2(7):654–662. https://doi.org/10.1364/OPTICA.2.000654

[52] Pal, H., Neifeld, M., 2003. Multispectral principal component imaging. Opt. Expr., 11(18):2118–2125. https://doi.org/10.1364/OE.11.002118

[53] Popescu, G., Deflores, L.P., Vaughan, J.C., , 2004. Fourier phase microscopy for investigation of biological structuresand dynamics. Opt.Lett., 29(21):2503–2505. https://doi.org/10.1364/OL.29.002503

[54] Prevedel, R., Yoon, Y.G., Hoffmann, M., , 2014. Simultaneous whole-animal 3D imaging of neuronal activity using light-fieldmicroscopy. Nat.Meth., 11(7):727–730. https://doi.org/10.1038/nmeth.2964

[55] Rust, M.J., Bates, M., Zhuang, X., 2006. Sub-diffractionlimitimaging by stochastic optical reconstruction microscopy (STORM). Nat. Meth., 3(10):793–796. https://doi.org/10.1038/nmeth929

[56] Ryle, M., 1972. The 5-km radio telescope at Cambridge. Nature, 239:435–438. https://doi.org/10.1038/239435a0

[57] Schermelleh, L., Heintzmann, R., Leonhardt, H., 2010. A guideto super-resolution fluorescence microscopy. J. Cell Biol., 190(2):165–175. https://doi.org/10.1083/jcb.201002018

[58] Stoklasa, B., Motka, L., Rehacek, J., , 2014. Wavefrontsensing reveals optical coherence. Nat. Commun., 5, Article 3275. https://doi.org/10.1038/ncomms4275

[59] Strack, R., 2016. Highly multiplexed imaging. Nat. Meth., 13(1), Article 35. https://doi.org/10.1038/nmeth.3706

[60] Suo, J., Bian, L., Chen, F., , 2014. Bispectralcoding: compressive and high-quality acquisition of fluorescence andreflectance. Opt.Expr., 22(2):1697–1712. https://doi.org/10.1364/OE.22.001697

[61] Teague, M.R., 1983. Deterministic phase retrieval: a green’sfunction solution. JOSA, 73(11):1434–1441. https://doi.org/10.1364/JOSA.73.001434

[62] van Tilbeurgh, H., Egloff, M., Martinez, C., , 1993. Interfacial activation of the lipase-procolipase complex by mixedmicelles revealed by X-ray crystallography. Nature, 362(6423):814–820. https://doi.org/10.1038/362814a0

[63] Vellekoop, I., Lagendijk, A., Mosk, A., 2010. Exploitingdisorder for perfect focusing. Nat. Photon., 4(5):320–322. https://doi.org/10.1038/nphoton.2010.3

[64] Velten, A., Willwacher, T., Gupta, O., , 2012. Recovering three-dimensional shape around a corner using ultrafasttime-of-flight imaging. Nat. Commun., 3, Article 745. https://doi.org/10.1038/ncomms1747

[65] Velten, A., Wu, D., Jarabo, A., , 2013. Femtophotography:capturing and visualizing the propagation of light. ACM Trans. Graph., 32(4), Article 44. https://doi.org/10.1145/2461912.2461928

[66] Waller, L., Kou, S.S., Sheppard, C.J., , 2010a. Phase from chromatic aberrations. Opt. Expr., 18(22):22817–22825. https://doi.org/10.1364/OE.18.022817

[67] Waller, L., Tian, L., Barbastathis, G., 2010b. Transportof intensity phase-amplitude imaging with higher order intensity derivatives. Opt. Expr., 18(12):12552–12561. https://doi.org/10.1364/OE.18.012552

[68] Waller, L., Situ, G., Fleischer, J.W., 2012. Phase-spacemeasurement and coherence synthesis of optical beams. Nat. Photon., 6(7):474–479. https://doi.org/10.1038/nphoton.2012.144

[69] Wang, L.V., Hu, S., 2012. Photoacoustic tomography: in vivo imaging from organelles to organs. Science, 335(6075):1458–1462. https://doi.org/10.1126/science.1216210

[70] Wilburn, B., Joshi, N., Vaish, V., , 2004. Highspeedvideography using a dense camera array. Proc. IEEE Computer Society Conf. on Computer Vision and PatternRecognition, p.294–301. https://doi.org/10.1109/CVPR.2004.1315176

[71] Willett, R., Gehm, M.E., Brady, D.J., 2007. Multiscale reconstructionfor computational spectral imaging. Proc.Electronic Imaging, Article 64980L. https://doi.org/10.1117/12.715711

[72] Wong, G., 2009. Snapshot hyperspectral imaging andpractical applications. J. Phys., 178(1), Article 012048. https://doi.org/10.1088/1742-6596/178/1/012048

[73] Zernike, F., 1955. How I discovered phase contrast. Science, 121(3141):345–349.

[74] Zheng, G., Horstmeyer, R., Yang, C., 2013. Wide-field,high-resolution Fourier ptychographic microscopy. Nat. Photon., 7(9):739–745. https://doi.org/10.1038/nphoton.2013.187

RIGHTS & PERMISSIONS

2017 Zhejiang University and Springer-Verlag GmbHGermany