Light field imaging: models, calibrations, reconstructions, and applications

Abstract

Light field imaging is an emerging technology in computationalphotography areas. Based on innovative designs of the imaging modeland the optical path, light field cameras not only record the spatialintensity of threedimensional (3D) objects, but also capture the angularinformation of the physical world, which provides new ways to addressvarious problems in computer vision, such as 3D reconstruction, saliencydetection, and object recognition. In this paper, three key aspectsof light field cameras, i.e., model, calibration, and reconstruction,are reviewed extensively. Furthermore, light field based applicationson informatics, physics, medicine, and biology are exhibited. Finally,open issues in light field imaging and long-term application prospectsin other natural sciences are discussed.

Keywords

Light field imaging / Plenopticfunction / Imaging model / Calibration / Reconstruction

References

[1] Babacan, S.D., Ansorge, R., Luessi, M., , 2012. Compressive light field sensing. IEEE Trans. Image Process., 21(12):4746–4757. https://doi.org/10.1109/tip.2012.2210237

[2] Belden, J., Truscott, T.T., Axiak, M.C., , 2010. Threedimensional synthetic aperture particle image velocimetry. Meas. Sci. Technol., 21(12):125403. https://doi.org/10.1088/0957-0233/21/12/125403

[3] Bergamasco, F., Albarelli, A., Cosmo, L., , 2015. Adopting an unconstrained ray model in light-field cameras for 3Dshape reconstruction. Proc. IEEE Conf. on Computer Vision and Pattern Recognition, p.3003–3012. https://doi.org/10.1109/cvpr.2015.7298919

[4] Birklbauer, C., Bimber, O., 2014. Panorama light-field imaging. Comput. Graph. Forum, 33(2):43–52. https://doi.org/10.1111/cgf.12289

[5] Birklbauer, C., Opelt, S., Bimber, O., 2013. Rendering gigaraylight fields. Comput.Graph. Forum, 32(2pt4):469–478. https://doi.org/10.1111/cgf.12067

[6] Bishop, T.E., Favaro, P., 2012. The light field camera: extended depth of field, aliasing,and superresolution. IEEE Trans. Patt. Anal. Mach. Intell., 34(5):972–986. https://doi.org/10.1109/tpami.2011.168

[7] Bok, Y., Jeon, H.G., Kweon, I.S., 2014. Geometric calibrationof micro-lens-based light-field cameras using line features. Proc. European Conf. onComputer Vision, p.47–61. https://doi.org/10.1007/978-3-319-10599-4_4

[8] Broxton, M., Grosenick, L., Yang, S., , 2013. Wave optics theory and 3-D deconvolution for the light field microscope. Opt. Expr., 21(21):25418–25439. https://doi.org/10.1364/oe.21.025418

[9] Buehler, C., Bosse, M., McMillan, L., , 2001. Unstructured lumigraph rendering. Proc.28th Annual Conf. on Computer Graphics and Interactive Techniques, p.425–432. https://doi.org/10.1145/383259.383309

[10] Chen, C., Lin, H., Yu, Z., , 2014. Light fieldstereo matching using bilateral statistics of surface cameras. Proc. IEEE Conf. on Computer Vision and PatternRecognition, p.1518–1525. https://doi.org/10.1109/cvpr.2014.197

[11] Cho, D., Lee, M., Kim, S., , 2013. Modelingthe calibration pipeline of the Lytro camera for high quality light-fieldimage reconstruction. Proc. IEEE Int. Conf.on Computer Vision, p.3280–3287. https://doi.org/10.1109/iccv.2013.407

[12] Dansereau, D.G., Pizarro, O., Williams, S.B., 2013. Decoding,calibration and rectification for lenselet-based plenoptic cameras. Proc. IEEE Conf. on Computer Vision and PatternRecognition, p.1027–1034. https://doi.org/10.1109/cvpr.2013.137

[13] Dansereau, D.G., Pizarro, O., Williams, S.B., 2015. Linearvolumetric focus for light field cameras. ACM Trans. Graph., 34(2):15.1–15.20. https://doi.org/10.1145/2665074

[14] Edussooriya, C.U.S., 2015. Low-Complexity MultidimensionalFilters for Plenoptic Signal Processing. PhD Thesis, University of Victoria, Canada. http://hdl.handle.net/1828/6894

[15] Fahringer, T., Thurow, B.S., 2012. Tomographic reconstruction of a 3-D flow field usinga plenoptic camera. Proc. 42nd AIAA FluidDynamics Conf. and Exhibit, p.1–13. https://doi.org/10.2514/6.2012-2826

[16] Georgiev, T., Lumsdaine, A., 2009. Superresolution with Plenoptic 2.0 cameras. Proc. Frontiers in Optics / Laser Science XXV /Fall OSA Optics & Photonics Technical Digest. https://doi.org/10.1364/srs.2009.stua6

[17] Georgiev, T., Lumsdaine, A., 2010. Focused plenoptic camera and rendering. J. Electron. Imag., 19(2):021106. https://doi.org/10.1117/1.3442712

[18] Georgiev, T., Lumsdaine, A., 2012. The multifocus plenoptic camera. Proc. Digital Photography VIII. https://doi.org/10.1117/12.908667

[19] Georgiev, T., Zheng, K.C., Curless, B., , 2006. Spatioangularresolution tradeoffs in integral photography. Proc. 17th Eurographics Conf. on Rendering Techniques, p.263–272. https://doi.org/10.2312/EGWR/EGSR06/263-272

[20] Georgiev, T., Chunev, G., Lumsdaine, A., 2011. Superresolutionwith the focused plenoptic camera. Proc.Computational Imaging IX, p.78730X. https://doi.org/10.1117/12.872666

[21] Ghasemi, A., Vetterli, M., 2014. Detecting planar surface using a light-field camera withapplication to distinguishing real scenes from printed photos. Proc. IEEE Int. Conf. on Acoustics, Speech andSignal Processing, p.4588–4592. https://doi.org/10.1109/icassp.2014.6854471

[22] Gortler, S.J., Grzeszczuk, R., Szeliski, R., , 1996. The lumigraph. Proc. 23rd Annual Conf.on Computer Graphics and Interactive Techniques,p.43–54. https://doi.org/10.1145/237170.237200

[23] Guo, X., Yu, Z., Kang, S.B., , 2016. Enhancinglight fields through ray-space stitching. IEEE Trans. Vis. Comput. Graph., 22(7):1852–1861. https://doi.org/10.1109/tvcg.2015.2476805

[24] Hahne, C., Aggoun, A., Haxha, S., , 2014. Light field geometry of a standard plenoptic camera. Opt. Expr., 22(22):26659–26673. https://doi.org/10.1364/oe.22.026659

[25] Hahne, C., Aggoun, A., Velisavljevic, V., 2015. The refocusingdistance of a standard plenoptic photograph. Proc. 3DTV-Conf.: the True Vision-Capture, Transmission and Displayof 3D Video, p.1–4. https://doi.org/10.1109/3dtv.2015.7169363

[26] Iffa, E., Wetzstein, G., Heidrich, W., 2012. Lightfield optical flow for refractive surface reconstruction. Proc. Applications of Digital Image ProcessingXXXV, p.84992H. https://doi.org/10.1117/12.981608

[27] Isaksen, A., McMillan, L., Gortler, S.J., 2000. Dynamicallyreparameterized light fields. Proc. 27thAnnual Conf. on Computer Graphics and Interactive Techniques, p.297–306. https://doi.org/10.1145/344779.344929

[28] Jeon, H.G., Park, J., Choe, G., , 2015. Accuratedepth map estimation from a lenslet light field camera. Proc. IEEE Conf. on Computer Vision and Pattern Recognition, p.1547–1555. https://doi.org/10.1109/cvpr.2015.7298762

[29] Johannsen, O., Heinze, C., Goldluecke, B., , 2013. On the calibration of focused plenoptic cameras. In: Grzegorzek, M., Theobalt,C., Koch, R., et al. (Eds.), Timeof-Flightand Depth Imaging: Sensors, Algorithms, and Applications, p.302–317. https://doi.org/10.1007/978-3-642-44964-2_15

[30] Johannsen, O., Sulc, A., Goldluecke, B., 2015. On linearstructure from motion for light field cameras. Proc. IEEE Int. Conf. on Computer Vision, p.720–728. https://doi.org/10.1109/iccv.2015.89

[31] Johannsen, O., Sulc, A., Goldluecke, B., 2016. What sparselight field coding reveals about scene structure. Proc. IEEE Conf. on Computer Vision and Pattern Recognition, p.3262–3270. https://doi.org/10.1109/cvpr.2016.355

[32] Kalantari, N.K., Wang, T.C., Ramamoorthi, R., 2016. Learning-basedview synthesis for light field cameras. ACM Trans. Graph., 35(6):193.1–193.10. https://doi.org/10.1145/2980179.2980251

[33] Kim, C., Zimmer, H., Pritch, Y., , 2013. Scene reconstruction from high spatio-angular resolution light fields. ACM Trans. Graph., 32(4):73.1–73.12. https://doi.org/10.1145/2461912.2461926

[34] Kim, S., Ban, Y., Lee, S., 2014. Face liveness detectionusing a light field camera. Sensors, 14(12):22471–22499. https://doi.org/10.3390/s141222471

[35] Landy, M., Movshon, J.A., 1991. The Plenoptic Function and the Elements of Early Vision. MIT Press, USA, p.3–20.

[36] Levin, A., Durand, F., 2010. Linear view synthesis using a dimensionality gap lightfield prior. Proc. IEEE Conf. on ComputerVision and Pattern Recognition, p.1831–1838. https://doi.org/10.1109/cvpr.2010.5539854

[37] Levin, A., Fergus, R., Durand, F., , 2007. Image and depth from a conventional camera with a coded aperture. ACM Trans. Graph., 26(3):70. https://doi.org/10.1145/1239451.1239521

[38] Levoy, M., Hanrahan, P., 1996. Light field rendering. Proc.23rd Annual Conf. on Computer Graphics and Interactive Techniques, p.31–42. https://doi.org/10.1145/237170.237199

[39] Levoy, M., Ng, R., Adams, A., , 2006. Light fieldmicroscopy. ACMTrans. Graph., 25(3):924–934. https://doi.org/10.1145/1141911.1141976

[40] Li, J., Lu, M., Li, Z.N., 2015. Continuous depth mapreconstruction from light fields. IEEE Trans. Image Process., 24(11):3257–3265. https://doi.org/10.1109/tip.2015.2440760

[41] Li, N., Ye, J., Ji, Y., , 2014. Saliencydetection on light field. Proc. IEEE Conf.on Computer Vision and Pattern Recognition, p.2806–2813. https://doi.org/10.1109/cvpr.2014.359

[42] Liang, C.K., Shih, Y.C., Chen, H.H., 2011. Light field analysisfor modeling image formation. IEEE Trans. Image Process., 20(2):446–460. https://doi.org/10.1109/tip.2010.2063036

[43] Lin, H., Chen, C., Kang, S.B., , 2015. Depth recoveryfrom light field using focal stack symmetry. Proc. IEEE Int. Conf. on Computer Vision, p.3451–3459. https://doi.org/10.1109/iccv.2015.394

[44] Liu, J., Xu, T., Yue, W., , 2015. Light-fieldmoment microscopy with noise reduction. Opt. Expr., 23(22):29154–29162. https://doi.org/10.1364/OE.23.029154

[45] Lumsdaine, A., Georgiev, T., 2008. Full Resolution Lightfield Rendering. Indiana University and Adobe Systems, Technical Report.

[46] Lytro Inc., 2011. Lytro Cinema Brings RevolutionaryLight Field Technology to Film and TV Production. Technical Report. http://www.lytro.com

[47] Maeno, K., Nagahara, H., Shimada, A., , 2013. Light field distortion feature for transparent object recognition. Proc. IEEE Conf. on Computer Vision and PatternRecognition, p.2786–2793. https://doi.org/10.1109/cvpr.2013.359

[48] Marwah, K., Wetzstein, G., Bando, Y., , 2013. Compressive light field photography using overcomplete dictionariesand optimized projections. ACM Trans. Graph., 32(4):46.1–46.12. https://doi.org/10.1145/2461912.2461914

[49] Maximilian, D., 2016. Light-Field Imaging and HeterogeneousLight Fields. PhD Thesis, Heidelberg University,Germany.

[50] Mignard-Debise, L., Ihrke, I., 2015. Light-field microscopy with a consumer light-field camera. Proc. Int. Conf. on 3D Vision,p.335–343. https://doi.org/10.1109/3dv.2015.45

[51] Mihara, H., Funatomi, T., Tanaka, K., , 2016. 4D light field segmentation with spatial and angular consistencies. Proc. Int. Conf. on Computational Photography, p.54–61. https://doi.org/10.1109/iccphot.2016.7492872

[52] Ng, R., 2005. Fourier slice photography. ACM Trans. Graph., 24(3):735–744. https://doi.org/10.1145/1073204.1073256

[53] Ng, R., 2006. Digital Light Field Photography. PhD Thesis, Stanford University, USA.

[54] Ng, R., Levoy, M., Brédif, M., , 2005. Light Field Photography with a Hand-Held Plenoptic Camera. Technical Report, CTSR 2005-02, Stanford University,USA.

[55] Niu, C.Y., Qi, H., Huang, X., , 2016. Efficientand robust method for simultaneous reconstruction of the temperaturedistribution and radiative properties in absorbing, emitting, andscattering media. J. Quant. Spectros. Rad.Transfer, 184:44–57. https://doi.org/10.1016/j.jqsrt.2016.06.032

[56] Orth, A., Crozier, K.B., 2013. Light field moment imaging. Opt. Lett., 38(15):2666–2668. https://doi.org/10.1364/ol.38.002666

[57] Perwaß, C., Wietzke, L., 2012. Single lens 3D-camera with extended depth-of-field. Proc. Human Vision and Electronic Imaging XVII. https://doi.org/10.1117/12.909882

[58] Perwaß, U., Perwaß, C., 2013. Digital Imaging System, Plenoptic Optical Device andImage Data Processing Method. US Patents.

[59] Pérez, F., Pérez, A., Rodríguez, M., ., 2012. Fourier slice super-resolution in plenoptic cameras. Proc. IEEE Int. Conf. on Computational Photography, p.1–11. https://doi.org/10.1109/iccphot.2012.6215210

[60] Raghavendra, R., Raja, K.B., Busch, C., 2015. Presentation attackdetection for face recognition using light field camera. IEEE Trans. Image Process., 24(3):1060–1075. https://doi.org/10.1109/tip.2015.2395951

[61] Sabater, N., Drazic, V., Seifi, M., , 2014. Light-Field Demultiplexing and Disparity Estimation. Technical Report, Technicolor Research and Innovation, France.

[62] Seifi, M., Sabater, N., Drazic, V., , 2014. Disparityguided demosaicking of light field images. Proc. IEEE Int. Conf. on Image Processing, p.5482–5486. https://doi.org/10.1109/icip.2014.7026109

[63] Shi, L., Hassanieh, H., Davis, A., , 2014. Light field reconstruction using sparsity in the continuous Fourierdomain. ACM Trans.Graph., 34(1):12.1–12.13. https://doi.org/10.1145/2682631

[64] Shum, H., Kang, S.B., 2000. Review of image-based rendering techniques. Proc. Visual Communications and Image Processing, p.2–13. https://doi.org/10.1117/12.386541

[65] Skupsch, C., Brücker, C., 2013. Multiple-plane particle image velocimetry using a light-fieldcamera. Opt. Expr., 21(2):1726–1740. https://doi.org/10.1364/oe.21.001726

[66] Srinivasan, P.P., Tao, M.W., Ng, R., , 2015. Oriented light-field windows for scene flow. Proc. IEEE Int. Conf. on Computer Vision, p.3496–3504. https://doi.org/10.1109/iccv.2015.399

[67] Tao, M.W., Hadap, S., Malik, J., , 2013. Depth fromcombining defocus and correspondence using light-field cameras. Proc. IEEE Int. Conf. on Computer Vision, p.673–680. https://doi.org/10.1109/iccv.2013.89

[68] Tao, M.W., Su, J.C., Wang, T.C., , 2016. Depth estimationand specular removal for glossy surfaces using point and line consistencywith light-field cameras. IEEE Trans. Patt. Anal. Mach. Intell., 38(6):1155–1169. https://doi.org/10.1109/tpami.2015.2477811

[69] Thomason, C.M., Thurow, B.S., Fahringer, T., 2014. Calibrationof a microlens array for a plenoptic camera. Proc. 52nd Aerospace Sciences Meeting, p.1456–1460. https://doi.org/10.2514/6.2014-0396

[70] Thurow, B.S., Fahringer, T., 2013. Recent development of volumetric PIV with a plenopticcamera. Proc. 10th Int. Symp. on ParticleImage Velocimetry, p.1–7.

[71] Tosic, I., Berkner, K., 2014. Light field scale-depth space transform for dense depthestimation. Proc. IEEE Conf. on ComputerVision and Pattern Recognition, p.435–442. https://doi.org/10.1109/cvprw.2014.71

[72] Vaish, V., 2007. Synthetic Aperture Imaging Using DenseCamera Arrays. PhD Thesis, Stanford University,USA.

[73] Vaish, V., Garg, G., Talvala, E., , 2005. Syntheticaperture focusing using a shear-warp factorization of the viewingtransform. Proc. IEEE Computer SocietyConf. on Computer Vision and Pattern Recognition, 3:129. https://doi.org/10.1109/cvpr.2005.537

[74] Vaish, V., Levoy, M., Szeliski, R., , 2006. Reconstructing occluded surfaces using synthetic apertures: stereo,focus and robust measures. Proc. IEEE ComputerSociety Conf. on Computer Vision and Pattern Recognition, p.2331–2338. https://doi.org/10.1109/cvpr.2006.244

[75] Venkataraman, K., Lelescu, D., Duparré, J., , 2013. Pi-Cam: an ultra-thin high performance monolithic camera array. ACM Trans. Graph., 32(6):166.1–166.13. https://doi.org/10.1145/2508363.2508390

[76] Wang, T.C., Efros, A.A., Ramamoorthi, R., 2015. Occlusionawaredepth estimation using light-field cameras. Proc. IEEE Int. Conf. on Computer Vision, p.3487–3495. https://doi.org/10.1109/iccv.2015.398

[77] Wang, T.C., Chandraker, M., Efros, A.A., , 2016a. SVBRDF-invariant shape and reflectance estimation from light-fieldcameras. Proc. IEEE Conf. on Computer Visionand Pattern Recognition, p.5451–5459. https://doi.org/10.1109/cvpr.2016.588

[78] Wang, T.C., Zhu, J.Y., Hiroaki, E., , 2016b. A 4D light-fielddataset and CNN architectures for material recognition. Proc. European Conf. on Computer Vision, p.121–138. https://doi.org/10.1007/978-3-319-46487-9_8

[79] Wanner, S., Goldluecke, B., 2012a. Globally consistent depth labeling of 4D light fields. Proc. IEEE Conf. on Computer Vision and PatternRecognition, p.41–48. https://doi.org/10.1109/cvpr.2012.6247656

[80] Wanner, S., Goldluecke, B., 2012b. Spatial and angular variational super-resolution of 4Dlight fields. Proc. European Conf. on ComputerVision, p.608–621. https://doi.org/10.1007/978-3-642-33715-4_44

[81] Wanner, S., Goldluecke, B., 2014. Variational light field analysis for disparity estimationand super-resolution. IEEE Trans. Patt. Anal. Mach. Intell., 36(3):606–619. https://doi.org/10.1109/tpami.2013.147

[82] Wanner, S., Fehr, J., Jähne, B., 2011. GeneratingEPI representations of 4D light fields with a single lens focusedplenoptic camera. Proc. Int. Symp. on VisualComputing, p.90–101. https://doi.org/10.1007/978-3-642-24028-7_9

[83] Wanner, S., Meister, S., Goldluecke, B., 2013. Datasetsand benchmarks for densely sampled 4D light fields. In: Bronstein, M., Favre, J.,Hormann, K. (Eds.), Vision, Modeling and Visualization, p.225–226. https://doi.org/10.2312/PE.VMV.VMV13.225-226

[84] Wilburn, B., 2004. High Performance Imaging Using Arraysof Inexpensive Cameras. PhD Thesis, StanfordUniversity, USA.

[85] Williem, W., Park, I.K., 2016. Robust light field depth estimation for noisy scene withocclusion. Proc. IEEE Conf. on ComputerVision and Pattern Recognition, p.4396–4404. https://doi.org/10.1109/cvpr.2016.476

[86] Xiao, Z., Wang, Q., Si, L., , 2014. Reconstructingscene depth and appearance behind foreground occlusion using cameraarray. Proc. IEEE Int. Conf. on Image Processing, p.41–45. https://doi.org/10.1109/icip.2014.7025007

[87] Xu, Y., Nagahara, H., Shimada, A., , 2015. TransCut: transparent object segmentation from a light-field image. Proc. IEEE Int. Conf. on Computer Vision, p.3442–3450. https://doi.org/10.1109/iccv.2015.393

[88] Yoon, Y., Jeon, H.G., Yoo, D., , 2015. Learninga deep convolutional network for light-field image superresolution. Proc. IEEE Int. Conf. on Computer Vision, p.24–32. https://doi.org/10.1109/iccvw.2015.17

[89] Yu, J., McMillan, L., 2004. General linear cameras. Proc.European Conf. on Computer Vision, p.14–27. https://doi.org/10.1007/978-3-540-24671-8_2

[90] Yu, Z., Yu, J., Lumsdaine, A., , 2012. An analysis of color demosaicing in plenoptic cameras. Proc. IEEE Conf. on Computer Vision and Pattern Recognition, p.901–908. https://doi.org/10.1109/cvpr.2012.6247764

[91] Yu, Z., Guo, X., Lin, H., , 2013. Line assistedlight field triangulation and stereo matching. Proc. IEEE Int. Conf. on Computer Vision, p.2792–2799. https://doi.org/10.1109/iccv.2013.347

[92] Yuan, Y., Liu, B., Li, S., , 2016. Light-field-cameraimaging simulation of participatory media using Monte Carlo method. Int. J. Heat Mass Transfer, 102:518–527. https://doi.org/10.1016/j.ijheatmasstransfer.2016.06.053

[93] Zhang, C., Ji, Z., Wang, Q., 2016. Rectifying projectivedistortion in 4D light field. Proc. IEEEInt. Conf. on Image Processing, p.1464–1468. https://doi.org/10.1109/icip.2016.7532601

[94] Zhang, Z., Liu, Y., Dai, Q., 2015. Light field from microbaselineimage pair. Proc. IEEE Conf. on ComputerVision and Pattern Recognition, p.3800–3809. https://doi.org/10.1109/cvpr.2015.7299004

[95] Zhou, C., Miau, D., Nayar, S.K., 2012. Focal Sweep Camerafor Space-Time Refocusing. Technical ReportCUCS-021-12, Department of Computure Science, Columbia University,USA.

RIGHTS & PERMISSIONS

2017 Zhejiang University and Springer-Verlag GmbHGermany