REVIEW ARTICLE

Retinal projection head-mounted display

  • Junguo LIN ,
  • Dewen CHENG ,
  • Cheng YAO ,
  • Yongtian WANG
Expand
  • Beijing Engineering Research Center of Mixed Reality and Advanced Display, School of Optoelectronics, Beijing Institute of Technology, Beijing 100081, China

Received date: 29 Jul 2016

Accepted date: 15 Oct 2016

Published date: 17 Mar 2017

Copyright

2016 Higher Education Press and Springer-Verlag Berlin Heidelberg

Abstract

Retinal projection displays (RPDs) are an important development direction for head-mounted displays (HMDs). This paper reviews the literature on optical engineering aspects based on the data on advanced technology in RPD design and development. The review includes the principles and applications of four theories, e.g., the Maxwellian view and its modified modality and the monocular and binocular depth cues of stereoscopic objects in the physiology of the human visual system. To support the Maxwellian view and achieve retinal projection systems with depth cues, results of previous design works were summarized using different methods and their advantages and disadvantages are analyzed. With an extremely long focal depth, a prototype of a full-color stereoscopic see-through RPD system was discussed. Finally, a brief outlook of the future development trends and applications of the RPDs was presented

Cite this article

Junguo LIN , Dewen CHENG , Cheng YAO , Yongtian WANG . Retinal projection head-mounted display[J]. Frontiers of Optoelectronics, 2017 , 10(1) : 1 -8 . DOI: 10.1007/s12200-016-0662-8

1 Introduction

Head-mounted display (HMD) is of great interest in current research, including augmented reality (AR) smart glasses [1], virtual reality (VR) devices [2], and mixed reality (MR) technology. The accommodation and convergence of conventional HMDs, which generates eye fatigue and compromises image quality, is immanently dissociated [3]. However, retinal projection displays (RPDs), as a recent important development in HMDs [4], can overcome this immanent dissociation between the accommodation and convergence of conventional HMDs [5,6] and support extremely long focal depths.
Retinal projection technology was developed from retinal scanning technology, which originated from the scanning laser ophthalmoscope (SLO) set developed by Webb et al. in 1980 [7]. To support the SLO, a special design of scanning and detecting devices that control the laser beam in scanning two-dimensionally through the pupil of the eye onto the retina, which reconstructs the retinal image through the fundus reflex light. This technology has been widely used in medical images to obtain the fundus of the retina [810]. Webb et al. concluded that the user can observe the digital image if the scanning beam is modulated by loading the virtual digital information [7].
In 1986, Yoshinaka proposed the use of a scanning device to control the light beam projected on the retina in image display in order to reduce the volume and energy consumption of the display device. He pointed out that the device can potentially display the stereoscopic image, which can be used in the near eye display [11,12]. Based on the concept of image scanning directly on the eye retina, a retinal scanning display prototype was developed in the Human-computer Interface Technology Laboratory (HIT Laboratory) by Kollin et al. in 1992 [13]. Kollin also noted that the spatial position of the focal point on the retina was determined by the angle of the incident beam. Thus, the accommodation of the eye does not affect the blurring of digital images [1315].
Several companies have completed RPD prototypes as near-eye displays. Based on the technology spun off from the HIT Laboratory, Microvision Inc., which was founded in 1993, built a prototype called Nomad Personal Display System in 2001 using micro electro-mechanical system (MEMS) scanning mirrors for laser projection [16,17]. Brother Inc. demonstrated its eyewear named AiRScouter at the Brother World Japan 2010 Expo [16,18]. However, both companies failed to commercially produce their prototypes and ceased development probably because the use of eyewear terminal equipment did not gain popularity until recently. In 2014, Avegant Int. formally issued a VR device called Glyph, which provided stereoscopic image vision by means of retinal projection technology [19]. In the same year, QD Laser Int. demonstrated its AR eyewear, Laser Eyewear (LEW), based on laser retinal imaging optics [20,21] and composed primarily of a MEMS, an RGB laser module, and a free-surface reflection mirror [16]. With these, retinal projection HMD technology is likely to be used as display devices in the next generation of computer terminals.
Based on the data on the advanced technology of RPD design and development, this work aims to examine aspects of the literature on optical engineering. Principles and applications of four theories [4,22] were studied, e.g., the Maxwellian view and its modified modality [23], as well as monocular and binocular depth cues of stereoscopic objects in the physiology of a human visual system [22]. To support the Maxwellian view and achieve retinal projection systems, results from previous designing works using different methods [2332] were analyzed. With an extremely long focal depth, a prototype of a full-color stereoscopic see-through RPD system was discussed. Finally, a brief outlook on future development trends and applications of RPDs was presented.

2 Principles

This section explores the geometric optics of the Maxwellian view, the key technology of the retinal projection HMD [33,34]. Technology with the imaging method of the conventional HMD is compared to illustrate how virtual information is projected on the human retina in the Maxwellian view. A modified Maxwellian view [23], which overcomes the drawbacks of the previous one, is introduced.

2.1 Maxwellian view

Figure 1 shows that in the Maxwellian view, thin parallel beams are emitted from a spatial light modulator (SLM, such as liquid crystal display (LCD) and organic light emitting diode (OLED), all converged at the center of the pupil by the lens system and directly projected on the retina. Each pixel of the SLM that forms a clear image at any position behind the SLM stimulates a certain unique point on the retina. An image pattern modulated from the SLM is thus recognized. Note that the image pattern can be observed in the Maxwellian view without ocular accommodation because all the beams pass through the center of the crystalline lens. Thus, the image pattern supports an extremely long focal depth.
Fig.1 Principle of the Maxwellian view

Full size|PPT slide

However, in conventional HMD, beams are emitted from the display, adjusted by the lens system, converged at the crystalline lens, and projected on the retina, as shown in Fig. 2. Ocular accommodation is indispensable in conventional HMD to obtain a clear image.
Fig.2 Normal viewing of HMD

Full size|PPT slide

In addition, the conventional HMD has a fixed focal point of virtual images on display, whereas the eye dynamically accommodates the changing fixation points of real objects. Therefore, both real and virtual objects cannot be steadily observed because the observer can only accommodate one of them [23], as indicated in Fig. 3(a). The use of such an AR-Smart Glasses in complex environments, requiring the eyes of the observer to accommodate continually, is also unsafe. Compared with the conventional HMD, thin parallel beams from the RPD support an extremely long focal depth, rendering a clear view of the virtual object to the observer when his eyes accommodate real objects as shown in Fig. 3(b).
Fig.3 (a) Dissociation between real and virtual objects [23]; (b) integration of real and virtual objects

Full size|PPT slide

The Maxwellian view makes it possible to fix the immanent dissociation between accommodation and convergence, support an extremely long focal depth, and minimize the effect of optical aberrations on the eye. However, the Maxwellian view is limited by its extremely narrow viewing area that requires the beams to strictly converge at the center of the crystalline lens.

2.2 Modified Maxwellian view

To overcome the narrow viewing area of the Maxwellian view, Takahashi used a holographic optical element (HOE) to propose the modified Maxwellian view [23]. This innovation allows the observer to keep viewing an image with eyes rotating at a limited extent. Compared to the Maxwellian view, the converging point of the beams in the modified Maxwellian view is located at the rotation center of the human eyeball, as shown in Fig. 4. The observer can keep viewing an image with rotating eyes as the field of view angle narrows through the aperture of the pupil [23]. This method, to a certain extent, overcomes the narrow viewing area problem but neglects the crystalline accommodation, narrows the pupil aperture, and reduces the view angle field.
Fig.4 Principle of the modified Maxwellian view. (a) Side view; (b) top view [23]

Full size|PPT slide

3 Methods supporting the Maxwellian view

From the discussion on the Maxwellian view in Section 2, obtaining the thin light beams and their convergence are conclusively the key points of the approach. This section summarizes the methods to achieve the Maxwellian view from previous design works by examining the literature on the state-of-the-art in retinal projection; their advantages and disadvantages are analyzed as well.

3.1 Obtaining thin light beams

The importance of the thin light beams is considered in Fig. 3(b). Each thin light beam of the image pixel stimulates a certain unique point on the retina to recognize the image pattern. The two methods to obtain thin light beams are as follows.
The first method is the pinhole imaging model shown in Fig. 5(a). The light source image is modulated by the pinhole and projected on the retina in the Maxwellian view. The thin light beam from each image pixel of the light source passes through the pinhole, converges at the center of the crystalline lens, and is projected on the retina. Image clarity is independent of the accommodation function and the focus position of the crystalline lens. Given that only a small position of light energy comes through the pinhole, a dark image is observed, and the energy efficiency of the light source is low.
The other method to obtain thin light beams is laser image, as shown in Fig. 5(b) [35]. This method has the advantages of good collimation and miniaturization. The combination of these advantages with MEMS [36] results in the laser’s capacity to support a two-dimensional image by scanning the human retina through the center of the crystalline lens. The laser image can achieve very high lumens for day and night operations; however, the speckle effect blurs the laser image.
Fig.5 Process of obtaining thin light beams. (a) Pinhole image; (b) laser image [16]

Full size|PPT slide

The current mainstream approach is a combination of the two methods. In this approach, the laser is used to overcome the low light intensity of the pinhole image, and the pinhole serves as a modulator to reduce the speckle effect and as a filter to limit the senior diffraction of light.

3.2 Methods of convergence

The convergence of thin light beams at the center of crystalline lens is another outstanding feature of RPD. The methods of convergence are summarized from previous design works, such as the uses of freeform free-space magnifier, half mirror, non-axisymmetric free-surface reflection mirror, and HOEs. Other methods that have the potential to support the convergence are also presented in this paper.
A feasible strategy to converge thin light beams is through the use of a magnifier to produce a distinguishable image of SLM. Glyph, released by Avegant Int. [19], used a patented freeform free-space magnifier to obtain an occlusion view. By means of the freeform free-space magnifier, low power LEDs with digital light processing (DLP) provide a virtual image converged at the center of crystalline lens. The magnifier provides convergence rather than non-see-through view unless it cooperates with other optical elements.
As illustrated in Fig. 6, Yang et al. [24,25] used a half-mirror with a convex lens to reflect the image from the liquid crystal on silicon (LCoS) display, converge thin light beams at the center of crystalline lens, project them on the retina, and render real-world objects available for viewing through the RPD system. Half-mirror is used to superimpose a virtual image on a real world object. However, the RPD with half mirror cannot effectively produce both virtual image and real object rays with appropriate intensity because of the tradeoff between reflectance and transmittance.
Fig.6 Optical path of the RPD using a half-mirror [25]

Full size|PPT slide

Ando et al. [27] presented another method that uses the HOE to combine superimpositions of a virtual image on a real scene. The HOE is a diffraction grating accomplished by the holography technique, and the light source is an LCD modulated by digital micro-mirror device (DMD). Figure 7 shows the specification of the HOE, where the parallel rays from left and right are diffracted by the HOE and converged at the corresponding eye. Therefore, the appropriate intensities of both real world object and diffracted image are seen simultaneously because the HOE diffracts only rays of specific wavelengths, without reducing the intensity of visible rays. This binocular system can potentially support a stereoscopic see-through RPD, as Takahashi hypothesized [23]. The angular and wavelength selectivity of the HOE makes the virtual image from the HOE monochromatic. The accurate production of such a HOE is particularly demanding because of the reflection and aberration by the curved shape of the spectacle lens. Moreover, it is also difficult to accurately superimpose the optical axis of the HOE on the optical axis of the HMD [28]. These drawbacks hinder the application of HOEs to some extent.
Fig.7 Specification of the HOE [27]

Full size|PPT slide

The universal design compact eyewear of the RPD composed of a free-surface reflection mirror from QD Laser Int. was presented by Sugawara et al. [16]. As demonstrated in Fig. 8, the free-surface reflecting mirror is designed to collimate the RGB laser beam initially scanned by the MEMS mirror. Afterwards, the beam is converged at the center of crystalline lens. Like other methods of convergence, its capacities to match the convergence point and the center of crystalline lens were considered.
Fig.8 Compact optics of the RPD constituted by a free-surface reflection mirror [16]

Full size|PPT slide

To support the convergence, the methods of obtaining the Maxwellian view were summarized based on the analysis above. The premise of this study continues to be that the free form total internal reflection (TIR), the prism mirror, and the waveguide, which have been used in conventional HMD, can potentially support the Maxwellian view.

4 Realization of depth cues

In the Sections 2 and 3, the principles of the Maxwellian view and its modified modality were explored, summarized, and applied to prove the theory and to obtain the two-dimensional virtual information. However, when the two-dimensional virtual image is displayed, viewers can observe it even without any relation to real world objects. The viewer needs to perceive the virtual information in accordance with the real world object. The present study seeks to understand how RPD with depth cues provides stereoscopic vision for the viewer in accordance with HMD research.
In the physiology of the human visual system, the realization of depth cues can be divided into monocular and binocular depth cues [22]. Based on the theories, the current methods of depth cues in the RPD [2,23,29] are analyzed in this section.

4.1 Monocular depth cues and its realization in the RPD

To obtain monocular depth cues, the viewer accommodates the focal length of crystalline lens by stretching the muscles of the ciliary body, so that different parts of the scene are seen as revealed in Fig. 9.
Fig.9 Schematic of accommodation [22]

Full size|PPT slide

In practice, Takahashi et al. used a HOE composed of three types of HOEs (H1, H2, and H3), as demonstrated in Fig. 10(a), to provide a monocular multi-view with three different depth cue images [29]. Each type of HOE converges the corresponding projected images at respective convergent points (C1,C2, and C3) on the center plane of the crystalline lens, as shown in Fig. 10(b). However, this multi-view system displays the virtual image at different distances within eye accommodation, which do not strictly agree with the Maxwellian view. As the HOE is composed of finite number types of HOEs to support the corresponding number of depth cues, the multi-view system can only support a limited number of depths. Monocular depth cues can only support an effective distance of perception within a few meters, unless it cooperates with binocular convergence. Hence, the system is confined to a limited perception distance.
Fig.10 (a) Structure of the HOE; (b) schematic of the multi-view system [29]

Full size|PPT slide

4.2 Binocular parallax and its realization in RPD

When one views an object, the light beams of the object converge at the center of the retina. Owing to pupil spacing (average value of 6.5 cm), the relative position of the same scene is different in the left and right eyes, which provides the binocular parallax as indicated in Fig. 11. The human brain constructs stereoscopic impressions by binocular parallax. This mechanism provides for the use of two cameras with a certain distance to obtain a stereogram of side-by-side format as shown in Fig. 12. Two-dimensional images with depth parallax information are transmitted to the left and right eyes of the viewer, which can be perceived by the human brain to structure a stereogram.
Fig.11 Schematic of the binocular parallax [22]

Full size|PPT slide

Fig.12 Stereogram of the side-by-side format

Full size|PPT slide

Based on the binocular parallax mechanism, a prototype of a full-color stereoscopic see-through RPD system, especially with an extremely long focal depth, is presented. With regard to the disadvantages of previous stereoscopic see-through RPDs, non-fusion, and mono-chromaticity [23], a universal design optical structure demonstrated in Fig. 13 is used to achieve the Maxwellian view. A specific size of micro-pore P to modulate the thin light beam is located on the front focus plane of lens L3. The pupil of the human eye is located on the back focus plane of the lens system, which means that the micro-pore and pupil conjugate. To overcome the drawback that RPD with half-mirror cannot effectively produce both rays with appropriate intensities, the intensity of the laser is controlled and adjusted to the reflectance and transmittance of the half-mirror. Figure 14 shows the schematic diagram of the binocular stereoscopic RPD. The screens display the corresponding binocular parallax images of the stereogram. On the left side, P1 and P2 indicate the different points of the stereogram projected through the lens system, the center of the crystalline lens, and finally on the retina of the viewer. In the symmetric system, when the binocular light beams projected on the retina extend reversely, they intersect at P1″ and P2″ and construct a stereogram in the human brain. To demonstrate the effectiveness of the binocular stereoscopic see-through RPD, the experimental symmetric prototype in Fig. 15 is constructed. This prototype is not the head-mounted type, but a desktop type limited to the miniaturization of the laser projection. The long focal depth of the Maxwellian view that focuses on either a near or a far scene, is verified to obtain a clear view of the virtual image as demonstrated in Fig. 16.
Fig.13 Universal design optical structure of the RPD

Full size|PPT slide

Fig.14 Schematic of the binocular stereoscopic RPD

Full size|PPT slide

Fig.15 Experimental symmetric prototype

Full size|PPT slide

Fig.16 Image by the Maxwellian view. Focuses on (a) nearby and (b) distant

Full size|PPT slide

5 Conclusions

This paper presented a review of the state-of-the-art RPD design, which focuses on optical engineering aspects including the Maxwellian view, its modified modality, and human depth cue perception realization. Furthermore, the key differences between the optical design of RPDs and conventional HMDs were discussed with regard to the methods to obtain thin light beams and realize convergence. To solve these two problems, methods from previous design works, including pinhole image, laser image, magnifier, half-mirror, HOE, and free-surface reflecting mirror, were summarized and analyzed. With regard to the disadvantages of previous design works, non-fusion [23] and mono-chromaticity, a prototype of a full-color stereoscopic see-through RPD system, especially with an extremely long focal depth, were presented.
Compared with conventional HMDs, RPD can relieve eye fatigue by matching accommodation and convergence. Thus, RPD can be widely used in entertainment, education, industrial, surgical assistance, and military applications, as well as other long time works. Opportunities for further research are observed in terms of matching the convergence and the crystalline lens, expanding the viewing area, and refining the models for depth perception. This indicates the following research direction in RPD. We are confident that the RPD development will have a bright future in the fields of AR and VR.

Acknowledgements

This work was partially supported by the National High Technology Research and Development Program of China (No. 2015AA016301) and the Beijing Natural Science Foundation Program (No. 4152044).
1
Rauschnabel P A, Brem A, Ro Y. Augmented reality smart glasses: definition, conceptual insights, and managerial importance. Working paper, The University of Michigan-Dearborn, 2015

2
Kress B. The segmentation of the HMD market: optics for smart glasses, smart eyewear, AR and VR headsets. In: Proceedings of SPIE-The International Society for Optical Engineering, 2014, 9202: 92020D-1–92020D-14

3
Schowengerdt B T, Seibel E J, Furness T A. Binocular retinal scanning laser display with integrated focus cues for ocular accommodation. In: Proceedings of SPIE-The International Society for Optical Engineering, 2003, 5006: 1–9

4
Cakmakci O, Rolland J. Head-worn displays: a review. Journal of Display Technology, 2006, 2(3): 199–216

DOI

5
Wit G C D. A retinal scanning display for virtual reality. Dissertation for the Doctoral Degree. TU Delft: Delft University of Technology, 1997

6
Hakan U. Optical advantages of retinal scanning displays. In: Proceedings of SPIE-The International Society for Optical Engineering, 2000, 4021: 20–26

7
Webb R H, Hughes G W, Pomerantzeff O. Flying spot TV ophthalmoscope. Applied Optics, 1980, 19(17): 2991–2997

DOI PMID

8
Webb R H, Hughes G W, Delori F C. Confocal scanning laser ophthalmoscope. Applied Optics, 1987, 26(8): 1492–1499

DOI PMID

9
Ellingford A. The Rodenstock scanning laser ophthalmoscope in clinical practice. Journal of Audiovisual Media in Medicine, 1994, 17(2): 67–70

DOI PMID

10
Varano M, Scassa C. Scanning laser ophthalmoscope microperimetry. Seminars in Ophthalmology, 1998, 13(4): 203–209

DOI PMID

11
Yoshinaka K. Display Device, Japanese Patent: 61198892. 1986–09–03

12
Hu X, Liu Y, Wang J, Liu B. Current progress in head-mounted display based on retinal scanning. Hongwai Yu Jiguang Gongcheng, 2014, 43(3): 871–878 (in Chinese)

13
Jacobs R J, Bailey I L, Bullimore M A. Artificial pupils and Maxwellian view. Applied Optics, 1992, 31(19): 3668–3677

DOI PMID

14
Kollin J. A retinal display for virtual-environment applications. SID International Symposium Digest of Technical Papers, 1993, 24: 827–827

15
Kollin J S, Tidwell M R. Optical engineering challenges of the virtual retinal display. In: Proceedings of SPIE-The International Society for Optical Engineering, 1995, 2537: 48–60

16
Sugawara M, Suzuki M, Miyauchi N. Late-News Paper: Retinal imaging laser eyewear with focus-free and augmented reality. Sid Symposium Digest of Technical Papers, 2016, 47(1): 164–167

17
Davis W O, Brown D, Ma Y, Coy J W. Evolution of MEMS scanning mirrors for laser projection in compact consumer electronics. International Society for Optics and Photonics, 2010, 7594(5): 75940A-1–75940A-12

18
Watanabe M, Takayama H, Asai N, Matsuda R, Yamada S. A retinal scanning display with a wavefront curvature modulator. Journal of the Society for Information Display, 2003, 11(3): 511–515

DOI

19
https://www.avegant.com/

20
Wearable see-through display “Laser Eyewear” developed based on original laser retina imaging optics, http://www.qdlaser.com/pdf/Press-Release-Laser-Eye-Wear-QD-Laser-Inc.20140605-finalfinal.pdf

21
ttp://www.laserfocusworld.com/articles/2014/05/new-smart-glasses-from-qd-laser-rely-on-laser-retinal-imaging.html

22
Jia J. Fast algorithm for CGH calculation and system design for holographic 3D display. Dissertation for the Doctoral Degree. Beijing: Beijing Institute of Technology, 2013

23
Takahashi H. Stereoscopic see-through retinal projection head-mounted display. In: Proceedings of SPIE-The International Society for Optical Engineering, 2008, 6803: 68031N-1–68031N-8

24
Yang Z, Guo Z, Yang Z. Design of retinal projection display system. Journal of Applied Optics, 2011, 32(2): 222–225

25
Yang M, Guo Z, Yang Z. Helmet-mounted displays based on retinal projection display. Journal of Applied Optics, 2012, 33(2): 370–373

26
Ando T, Yamasaki K, Okamoto M, Matsumoto T, Shimizu E. Evaluation of HOE for head-mounted display. SPIE Proceedings, Practical Holography XIII, 1999, 3637: 110–118

DOI

27
Ando T, Yamasaki K, Okamoto M, Matsumoto T, Shimizu E. Retinal projection display using holographic optical element. SPIE Proceedings, Practical Holography XIV and Holographic Materials VI, 2000, 3956: 211–216

DOI

28
Takatsuka Y, Yabu H, Yoshimoto K, Takahashi H. Retinal projection display using diffractive optical element. In: Proceedings of Intelligent Information Hiding and Multimedia Signal, 2014: 403–406

29
Takahashi H, Ito Y, Nakata S, Yamada K. Retinal projection type super multi-view head-mounted display. Society of Photo-Optical Instrumentation Engineers (SPIE), Conference Series, 2014, 9012: 90120L-1–90120L-6

30
Bayer M M. Retinal scanning display: a novel HMD approach for army aviation. In: Proceedings of SPIE-The International Society for Optical Engineering, 2002, 4711: 202–213

31
McQuaide S C, Seibel E J, Burstein R, Furness T A III. 50.4: Three-dimensional virtual retinal display system using a deformable membrane mirror. SID Symposium Digest of Technical Papers, 2002, 33(1): 1324–1327

32
McQuaide S C, Seibel E J, Kelly J P, Schowengerdt B T, Furness T A III. A retinal scanning display system that produces multiple focal planes with a deformable membrane mirror. Displays, 2003, 24(2): 65–72

DOI

33
Westheimer G. The Maxwellian view. Vision Research, 1966, 6(11–12): 669–682

DOI PMID

34
Maxwell J C, Zaidi Q. On the theory of compound colours, and the relations of the colours of the spectrum. Color Research and Application, 1993, 18(4): 270–287

DOI

35
Viirre E, Johnston R, Pryor H, Nagata S, Furness T A 3rd. Laser safety analysis of a retinal scanning display system. Journal of Laser Applications, 1997, 9(5): 253–260

DOI PMID

36
Urey H, DeWitt F A IV, Lopez P A, Tauscher J. MEMS raster correction scanner for SXGA resolution retinal scanning display. SPIE Proceedings, 2003, 4985: 106–114

Outlines

/