Techniques to Expand the Exit Pupil of Maxwellian Display: A Review ^†

Abstract

Near-eye display (NED) devices are required to provide visual instructions in the fields of education, navigation, military operations, construction, healthcare, etc. The issues with conventional NEDs are the form factor and vergence–accommodation conflict (VAC). The Maxwellian display alleviates the VAC in NEDs by providing consistently focused virtual images to the viewer, regardless of the depth of focus of the human eye. The main limitation of the Maxwellian display is its limited exit pupil size. Due to misalignment of the device or eyeball rotation, the user may miss the eye box, and the image will become lost. To mitigate this limitation, exit pupil expansion can be obtained either statically or dynamically. This paper reviews the various techniques employed to expand the exit pupil. The review includes the principle, advantages, and drawbacks of various techniques for expanding the exit pupil of the Maxwellian display. The structure of the paper starts with an introduction and the principle of the Maxwellian display, followed by a discussion of the main limitations that arise with various techniques, along with potential solutions.

1. Introduction

The development of augmented reality (AR) technology has gained a lot of attention in recent years or decades. Augmented reality, which combines digital images with a physical world in three dimensions (3D), has recently grown in popularity in the scientific field [1]. The near-eye display is the most popular AR device due to its best immersive effect and portable design. NED devices are employed in the fields such as architecture, construction, education, navigation, military operations, and gaming [1]. Since existing NEDs focus virtual images on a fixed focus plane, there is a fixed accommodation distance. The distance from the focus plane to the NEDs does not match the vergence distance, known as vergence–accommodation conflict (VAC), which results in discomfort and visual fatigue [2]. The VAC issue is resolved by the Maxwellian display, offering consistently in-focus images to the user independent of the optical power of the human eye [3]. The Maxwellian display is based on the Maxwellian view, in which projection of images directly onto the retina is carried out by focusing the light rays on the eye’s pupil instead of providing the proper depth cues. The light beams from the display are converged by the eyepiece lens at the eye pupil plane and then projected onto the retina. However, the eye box size is limited to the size of the eye pupil [2,3]. This tiny eye box is uncomfortable for the wearer because a slight misalignment of the device or eyeball rotation makes the image disappear entirely. However, the limited eye box of Maxwellian displays remains a major challenge in the development of AR. Several methods have been proposed to enlarge the eye box.

This review includes various techniques to expand the exit pupil in active and passive ways. The review includes the principle, advantages, and drawbacks of various approaches for expanding the exit pupil of the Maxwellian display that have been reported in the literature, and is divided into three sections: static displays, tunable displays, and dynamic displays.

2. Techniques

2.1. Static Viewpoints

Exit pupil expansion of Maxwellian displays can be achieved by generating multiple viewpoints. Multiple viewpoints can be generated using the multiplexing technique. As an enlarged eye box is crucial for comfortable viewing in NEDs, angular multiplexing is used in Maxwellian displays. Multiple concave mirrors are recorded in a holographic optical element (HOE) as an out-coupler of a waveguide [4]. The HOE focuses the displayed images into multiple spots in the eye pupil plane, enhancing the exit pupil in the configuration. However, in this work, the exit pupil expansion is only carried out horizontally. The exit pupil can be extended in a vertical direction using the vertical high diffraction orders of the spatial light modulator (SLM) by enlarging the aperture in the Fourier plane of the 4-f system [5]. In the study reported by Shrestha et al., an array of beam splitters was used to expand the exit pupil, which resulted in an increase in weight or form factor due to bulky optics [6]. In the previous research, setups are bulky and have a high cost. A computational imaging-based holographic Maxwellian near-eye display addressed these shortcomings [7]. In this paper, encoding a complex wavefront into amplitude-only signals produces an all-in-focus virtual image. The hologram is multiplexed with several off-axis plane waves, which duplicate the pupils into an array to enlarge the exit pupil. As a result, this approach has a small form factor and only needs one active electrical component, which supports wearable applications. The aforementioned methods are difficult to use in a full-color NED due to the narrow bandwidth of diffractive optical elements [8]. A Maxwellian NED in full color with an expanded exit pupil in two-dimensional (2D) space is used to overcome the tiny eye box restriction [2]. With the use of a quarter-wave plate (QWP) and two Pancharatnam–Berry deflectors (PBDs), a broadband 2D beam deflector can be used to multiplex the one viewpoint into a 3 × 3 array of viewpoints, as shown in Figure 1a. There is a demerit of aforementioned methods in that each viewpoint image is overlapped, or the area becomes blank when changing the position from one viewpoint to another viewpoint. In order to prevent the images from overlapping on the retina, this effect is removed by creating multiple independent viewpoints [9]. In this method, a high-speed MEMS mirror can be used as an aperture stop for a narrow exit pupil, which provides many views based on the time-multiplexing approach, and gives a continuous image over a wide eye box without an eye tracker. To acquire a different area and perspective of a scene to each viewpoint, multiple HOEs are designed accordingly and spatially located. These viewpoint images do not overlap on the retina.

2.2. Tunable Viewpoints

Most Maxwellian NEDs with eye box replication give fixed intervals between the viewpoints, which must be tunable according the variation in eye pupil size among users. Focal spot steering is accomplished by synthesizing the CGH with various plane carrier waves [10]. In this method, the transverse position offset in the plane of the eye’s pupil depends on the spatial frequency of the plane wave. By changing the frequency of carrier wave in the CGH synthesis, multiple focal spots can be steered, as shown in Figure 1b. There is another technique to extend the eye box in Maxwellian see-through NEDs, which uses polarized gratings (PGs) and a multiplexed HOE [11]. The transmission PGs selectively diffract light beams with different polarization states and have high diffraction efficiency in ±1 orders. Two viewpoints are generated by the multiplexed HOE and are further duplicated to four viewpoints at different locations by using these two PGs, as shown in Figure 1c. By mechanically moving the PG, these viewpoints can be tuned accordingly. However, the tuning of image viewpoints and image shifts is still limited, which hinders its practicability. To overcome this limitation, an adjustable and continuous replication of eye box of a holographic Maxwellian near-eye display system has been proposed, in which different frequencies are employed to the hologram using spatial multiplexing to guide the beams in the required directions [12]. This allows us to create a pupil array, generate the sub-holograms corresponding to the viewpoints, then multiplex them all to make the composite hologram. The interval of the focus spots is dynamically adjusted, making it feasible to adapt to any eye pupil size and prevent the image overlapping and blind region issues. To minimize the operating speed and the bulky form factor, and to eliminate the problem with double or blank images, Yoo et al. [13] designed a light guide based on switchable viewpoints in the Maxwellian display. With this method, using the polarization grating, multiplexed HOEs, and polarization-dependent eyepiece lens, the expansion of the eye box is carried out without an additional mechanical movement of elements, as represented in Figure 1d. With the polarization-multiplexing approach, the polarizer rotator independently activates two different groups of viewpoints and is synchronized with the eye tracker.

Figure 1. (a) Eye box expansion using polarization-dependent HOEs (Reprinted with permission from Ref. [2]. Copyright 2020 The Optical Society); (b) Eye box expansion using multiplexed HOE (Reprinted with permission from Ref. [10]. Copyright 2018 The Optical Society); (c) Tunable viewpoints using polarization-dependent HOEs (Reprinted with permission from Ref. [11]. Copyright 2021 The Optical Society); (d) Switchable eye box using polarization multiplexing (Reprinted with permission from Ref. [13]. Copyright 2020 The Optical Society); (e) Viewing point steering with backlight modulation using LED array (Reprinted with permission from Ref. [14]. Copyright 2019 The Optical Society).

Figure 1. (a) Eye box expansion using polarization-dependent HOEs (Reprinted with permission from Ref. [2]. Copyright 2020 The Optical Society); (b) Eye box expansion using multiplexed HOE (Reprinted with permission from Ref. [10]. Copyright 2018 The Optical Society); (c) Tunable viewpoints using polarization-dependent HOEs (Reprinted with permission from Ref. [11]. Copyright 2021 The Optical Society); (d) Switchable eye box using polarization multiplexing (Reprinted with permission from Ref. [13]. Copyright 2020 The Optical Society); (e) Viewing point steering with backlight modulation using LED array (Reprinted with permission from Ref. [14]. Copyright 2019 The Optical Society).

2.3. Dynamic Viewpoints

Most of the Maxwellian display setups in the previous section do not include an eye-tracking device, which is needed for more realistic applications. By incorporating pupil-tracking, exit-pupil steering has been proposed as a light field projection-type display [15]. However, exit-pupil shifting for a holographic display has not yet been developed. Takaki et al. proposed a technique based on wave optics [16]. This technique can electrically modify the light’s convergence point in response to an eye’s movement without needing mechanical components, and produces a steerable eye box. An eye pupil-tracking Maxwellian system using a steering mirror and a pupil-shifting HOE is proposed, which provides a dynamic eye box that can be accomplished by using the pupil-shifting HOE combiner to laterally shift a view point in order to follow the movement of eye pupil [17]. However, the eye box is shifted accordingly by changing the incidence angle to the HOE in accordance with the detected eye location. Although the eye box is capable of 2D shifting, the possible incidence angle range is constrained by the aberrations and angular selectivity, which also restricts the eye box’s shifting range. In the study reported by Kim et al. [18], the dynamic eye box is generated by laterally translating the HOE. Both of the aforementioned techniques make use of the optical element’s mechanical movement, which results in an increase in the weight or form factor. The backlight modulation technique has been reported in the literature to generate a dynamic eye box in full-color Maxwellian displays [14]. An array of light-emitting diodes (LEDs) and a pupil tracker are synchronized to generate a steerable eye box based on the pinhole imaging principle represented in Figure 1e. This approach can switch fixed focal spots with low motion-to-photon latency. The aforementioned pupil-steering techniques rely on modifying the incident light angle. However, the lens coupler’s diffraction-limited performance is only achievable at one incidence angle; these techniques result in significant aberration. A new pupil-steering approach employs a switchable polarization converter and a cholesteric liquid crystal holographic lens (CLCHL) [19]. The polarization converter controls the light’s polarization and chooses the appropriate holographic lens to operate, as opposed to depending on changing the incidence angle to change the focal point.

3. Conclusions

This paper reviewed the state-of-the-art Maxwellian display design, focusing on two comfort features, form factor and a large eye box. We have introduced the conventional Maxwellian display and its principles, and then discussed the multiplexing techniques available to expand the exit pupil. Techniques for enlarging the exit pupil of Maxwellian displays, such as spatial and angular multiplexing of HOEs, polarization multiplexing, backlight modulation, and materials, are reviewed. Our paper discusses the relative merits and demerits of the methods, along with potential solutions in terms of achieving the goals of AR displays.