See ‐ Through Near ‐ Eye Display with Built ‐ in Prescrip ‐ tion and Two ‐ Dimensional Exit Pupil Expansion

Featured Application: The proposed see ‐ through near ‐ eye display is applicable for the aug ‐ mented/mixed reality devices, e.g., smart glasses or headsets etc. In addition to being a wearable computing device, it can be used as a pair of everyday eyeglasses with a built ‐ in prescription. Abstract: We propose a see ‐ through near ‐ eye display featuring an exit pupil expander (EPE), which is composed of two multiplexed slanted gratings. Via a two ‐ dimensional expansion, the exit pupil (EP) is able to be enlarged up to 10 × 8 mm 2 . Besides, the prescription for correcting the refractive errors can be integrated as well. The design rules are set forth in detail, followed by the results and discussion regarding the efficiency, field of view (FOV), exit pupil, angular resolution modu


Introduction
Following in the footsteps of smartphones, smart glasses are highly anticipated as the next mobile computing device. Unlike smartphones, which widely adopt the flat panel displays (FPDs) [1][2][3], smart glasses are equipped with see-through near-eye displays (NEDs) [4][5][6]. The major difference between these two types of display technologies is something called an exit pupil (EP). In FPDs, there is no such thing. In NEDs, an exit pupil defines the boundaries to which the eye can be moved. Unfortunately, for the sake of its compact form factor, almost every optical component of NEDs needs to be miniaturized. This inevitably results in a tiny exit pupil. Even when the form factor is overridden, it will not be easy to achieve a big exit pupil owing to the paradox between the field of view (FOV) and the exit pupil [7]. To enlarge the exit pupil, the most common method is to duplicate a single exit pupil into many through multiple reflections and/or diffractions. Of all the techniques that have been reported [8][9][10][11][12][13][14][15][16], the most successful ones are the cascaded semi-reflectors [8] and slanted gratings [10]. The former is a proprietary technology patented by Lumus. However, the cascaded semi-reflectors are only able to duplicate the exit pupil in one dimension. The latter was originally developed by Nokia and later commercialized by Microsoft's HoloLens. Though HoloLens pulled off the two-dimensional exit pupil duplication, it requires three layers of waveguides with slanted gratings to realize the full color [15]. Another gripe about HoloLens is it is not very friendly for those who have to wear the prescription lens for correcting the refractive errors [17]. For wearable devices, bulkiness is always a deal breaker. Towards the said issues, we hereby introduce a see-through NED that integrates the prescription lens with a two-dimensional exit pupil expander (EPE). In what follows, its structure, design rules, and overall performance are to be expounded. Figure 1 is a schematic of the proposed see-through NED, which consisted of three major components: a pico projector, a plano-concave lens, and two multiplexed slanted gratings. Mounted inside the pico projector was a microdisplay and a projection lens. The front surface (facing forward) of the plano-concave lens was a curved surface, which assumed a concave shape to yield a negative power to compensate the myopia. The right side of plano-concave lens was a bevel, to which the beam of the pico projector was incident. On the bevel, a multiplexed slanted grating was fabricated. This multiplexed slanted grating was able to couple the light from the pico projector into the lens and elongate the beam along the vertical direction. On the back surface (facing the eye) of the planoconcave lens, there was another multiplexed slanted grating that could diffract the light out of the lens and elongate the beam along the horizontal direction. To avoid being confused between these two multiplexed slanted gratings, the former is referred to as the in-coupling slanted grating, or ISG for short, whereas the latter is referred to as the out-coupling slanted grating, or OSG for short. W1, W2, H, T, and γ are the front width, back width, height, thickness and bevel angle of plano-concave lens, respectively.

Plano-Concave Lens
The design of the plano-concave lens was related to the visual acuity, and it plays an important role in imaging the real object, as shown in Figure 2, where R is the radius of curvature of the front surface of lens, sr the real object distance, and s′ the image distance. For the real image, rays emitted from the real object were diverged by the plano-concave lens so as to offset the over-focusing of the eye. A "real" image-by which we mean that it is the image of a real object, albeit this image is technically virtual-was formed at a closer distance. The object distance s, image distance s′ and diopter or optical power of the lens P shall be correlated via the lens-maker's equation [18] 1 (1) where n is the refractive index of the lens. Suppose a user has only 3 diopters of myopia, disregarding other types of refractive errors. Then, P = −3 m −1 when sr = ∞ m and s′ = −0.333 m. It should be mentioned that the "real" image was not observed by the eye. Rather, the eye saw the real object, as the rays derailed by the lens got back on track through the accommodation of the eye [19]. The front width W1 of the lens rested partly upon the interpupillary distance dip, which was on average 64 mm [20]. If the front surface of the lens is center-aligned with the eye, then the front width W1 of lens is (3) where db is the width of the bridge of the smart glasses. Say db = 18 mm, W1 = 46 mm. Further, the front width W1, back width W2, thickness T, and bevel angle γ shall be correlated via (4) Say γ = 75.52° (as will be discussed later), W2 = 47.93 mm, and T = 7.47 mm. The height H was a freelance parameter as long as it met the ergonomics. Pursuant to the above design rules, a planoconcave lens can be tentatively designed with the parameters itemized in Table 1.  Figure 3 draws the optical path diagram of the pico projector, of which the microdisplay was self-emissive, i.e., organic light-emitting diode panel on silicon [21], and the projection lens was a singlet-namely the simple magnifier. The viewing cone, within which the eye must be placed to see the full image, is highlighted. For the beam to be perfectly collimated, the image distance was set to be infinite by equating the virtual object (microdisplay) distance, sv, and the focal length, f, of the projection lens. It would be straightforward to write the FOV of the pico projector as

Pico Projector
where Dm is the size of the microdisplay. Say Dm = 0.165 inch (4.191 mm) for both horizontal and vertical dimensions and sv = f = 10 mm, then FOV = 24° (horizontal) × 24° (vertical), i.e., 33° (diagonal). Once the FOV was given, the exit pupil measured at the eye relief (ER)-the distance starting from the last surface of the pico projector to the pupil of eye-could be determined with where Ap is the aperture of the projection lens, and also the entrance pupil of our system. If the ER = 12 mm and Ap = 6 mm, then EP = 1 mm, which is unacceptably small. Since there is not much room to further shorten the eye relief-especially for NEDs without a built-in prescription-the simplest way to expand the exit pupil was to scale up the projection lens. That being said, the wearability was compromised due to the added volume and weight. Table 2 lists the customized specifications necessary for designing the pico projector. Other than the values already mentioned, the resolution of the microdisplay was 640 × 640, the pixel size was 6.5 μm, and the contrast ratio (CR) was 100,000.

Multiplexed Slanted Grating
Slanted grating, a type of asymmetric grating, lends itself to folding the optical path of seethrough NEDs for the reason that the energy of diffracted light can be concentrated to a certain diffraction order [22]. Its cross-sectional profile is outlined in Figure 4, where p is the grating period, h the grating height, w the grating width, and θg the slant angle. To couple light into and out of the lens, both the ISG and OSG were constructed as the transmission gratings, whose allowable diffraction angles shall satisfy [22] (7) where ni is the refractive index of the incident medium, ne the refractive index of the exit medium, θi the incident angle (relative to the grating normal), θm the diffraction angle of m th order, m the diffraction order, and λ the wavelength. As Equation (7) indicates, for a single-period slanted grating, both of its spectral and angular bandwidths were intrinsically narrow. To solve this issue, a multiple of waveguides with different gratings can be stacked together for the full color [15]. As an alternative solution, we resorted to the multiplexing of plural gratings on a single layer [23,24]. The multiplexed grating can be decomposed into three sub-gratings of the same slant angle but of different periods, widths and heights, as shown in Figure 5. Each sub-grating was designed for certain wavelength and incident angle. Say the periods of sub-gratings are pA, pB, and pC, respectively, then the collective period Pm of the multiplexed grating shall be the least common multiple of the above three. To minimize the reflection at the interface between the grating and the lens, it was desirable to etch the slanted gratings out of the lens so that there is no mismatch in their refractive indices. Speaking of the fabrication, photolithography, electron-beam lithography, and focused ion beam are among the feasible lithography techniques [25].  Decomposition of multiplexed grating into three sub-gratings. Say the periods of sub-gratings are pA, pB, and pC, respectively, then the collective period Pm of the multiplexed grating shall be the least common multiple of the above three.

Exit Pupil Expansion
Instead of duplicating the exit pupil into many, our EPE leveraged the oblique intersection of the viewing cone. As shown in Figure 6, the plane of the ISG intersected the viewing cone at angle α, thereby forming an elliptical exit pupil, which had a vertical length Ai that can be written as 2 (8) and a horizontal length that is defined by the width Wi of the ISG. Say Ap = 6 mm and α = 14.48°, Ai = 13.25 mm. Let β symbolize the angle of beam incident to the OSG, as shown in Figure 7, where only the axial rays are depicted. The horizontal length of the elliptical exit pupil will be elongated to Ao For Wi = 3.8 mm and β = 75.52°, Ao = 15.2 mm. As a result, the ISG is of 3.8 × 13.25 mm 2 and the OSG of 15.2 × 13.25 mm 2 . In case of misalignment, their actual sizes were supposed to be bigger. If the beam incident to the OSG is perpendicular to the bevel or the ISG, then β is equal to the bevel angle γ, which affects the thickness of the plano-concave lens. Incidentally, for the off-axis rays, problems such as pupil mismatch, vignetting, etc. merit special care.

Simulation Settings
The performance of our NED is quantitatively analyzed with Code V (Synopsys) and VirtualLab Fusion (Wyrowski Photonics). The former deals with the modulation transfer function (MTF), distortion, and simulated images. The latter handles the diffraction efficiency (DE) of multiplexed slanted gratings. The design wavelengths include 495, 565, and 655 nm. The real and virtual images are simulated in forward and backward directions, respectively. The parameters of optical surfaces used for both the real and virtual images are summarized in Tables 3 and 4. More detailed parameters for defining aspherical surfaces can be found in Tables 5 and 6.

Diffraction Efficiency
The algorithm for calculating the grating is the Fourier modal method [26]. The simulated annealing method [27] and Nelder-Mead or downhill simplex method [28] are in turn employed for the global and local optimizations. During the optimization, the slant angle, grating height and width are assigned as variables. In order to widen the spectral and angular bandwidths of a single slanted grating, both the ISG and OSG are multiplexed, which are formed through the fusion of plural sub-gratings, as itemized in Table 7. As for the spectral bandwidth, the DEs of all possible diffraction orders combined are calculated with respect to the wavelengths, as shown in Figure 8, where the average DEs of the ISG and OSG over the entire spectrum (460 to 660 nm) are 48.0% and 16.7%, respectively. As for the angular bandwidth, the DEs are calculated with respect to the out-coupling angles, at which the light is coupled out of the lens, the range of which equals the FOV, as shown in Figure 9. It can be seen that the average DEs of the ISG and OSG for the wavelength of 565 nm over the full horizontal/vertical FOV (±12°) are 14.40% and 10.53%, respectively.

Total Efficiency
Total efficiency, η, which measures the overall light utilization, is defined as the ratio of illuminance at the plane of the microdisplay to that at the plane of the exit pupil. Provided that the absorption, reflection and scattering of both projection and the prescription lenses are neglected, total efficiency η could be roughly estimated with 1 where (11) where

Field of View
The FOV of the real image, FOVr, describes the angular extent of the curved front surface of the plano-concave lens, which is given by It can be seen that FOVr is limited by the size of the plano-concave lens. As W1 = 46 mm, H = 30 mm (see Table 1), and ER = 12 mm, FOVr = 133° (125° × 103°). The FOV of the virtual image, FOVv, hinges on both the pico projector and gratings. As the out-coupling angles of the ISG and OSG span over the input FOV of the pico projector, the output FOVv is therefore conserved as 33° (24° × 24°), which is comparable to HoloLens 1′s FOV [15].

Exit Pupil
Without the exit pupil expansion, the original exit pupil at the eye relief of 12 mm is 1 × 1 mm 2 . Revisiting Figure 7 and Equation (6), the final exit pupil becomes For ER = 12 mm, Ao/i = 15.2 or 13.25 mm, and FOV = 24° × 24°, EP = 10 × 8 mm 2 . It shall be noted that although both the expanded and duplicated exit pupils can be calculated in the same way, the expanded exit pupil without gaps is undisputedly more solid than the duplicated exit pupils with gaps in between [8][9][10][11][12][13][14][15][16].

Angular Resolution
For the virtual image, angular resolution (AR) is defined as the average angular subtense of a single pixel. It can be calculated by dividing the FOV-measured in arcminutes (′)-by the number of pixels N along the diagonal, which is stated as [29] 60 • 60 • where Nh and Nv are the number of pixels along the horizontal and vertical directions, respectively. For FOVv = 33°, Nh = 640, and Nv = 640, AR = 2.19′.

Modulation Transfer Function
As shown in Figure 10, the MTFs are calculated as a function of spatial frequency in cycle/degree for the fields of 0° and 12° (tangential and radial). At 30 cycle/degree, the MTFs for all fields of the real and virtual images are above 0.252 and 0.032, respectively. The reason why the MTFs at 30 cycle/degree are selected as a benchmark is linked to the visual acuity [30]. A normal visual acuity of 1.0-stated as a decimal number-means that the eye is able to read an optotype at a spatial frequency of 30 cycle/degree.

Contrast Ratio
CR-the ratio of maximum intensity to minimum intensity-can be deduced as [31] where CRo is the CR of the real/virtual object. As the horizontal/vertical resolution is 640 and the FOV = 24°, the corresponding spatial frequency shall be 13.33 cycle/degree. According to Figure 10, at the field of 0° for the real image, CRr = 5 (CRo = ∞ and MTF = 0.69), while for the virtual image, CRv = 3 (CRo = 100,000 and MTF = 0.44).

Distortion
Distortion-the displacement of image height or ray location-is plotted in Figure 11, from which it can be seen that the distortions are 0.02% and 3.4% for the real and virtual images, respectively. Figure 11. Calculated distortion with respect to the field angle. It can be seen that the distortions are 0.02% and 3.4% for the real and virtual images, respectively. Figure 12 shows the original image alongside the real and virtual images. Compared to the original one, the real image is virtually lossless, while the virtual image has a mild pincushion distortion and slightly decreased brightness.

Conclusions
A see-through NED with a built-in prescription and two-dimensional EPE and design rules thereof have been proposed. Via a two-dimensional expansion, its exit pupil is enlarged up to 10 × 8 mm 2 . As opposed to the conventional EPEs, in which the exit pupil is duplicated or cloned into many, our solution leverages the oblique intersection of the viewing cone. More importantly, full color can be realized by the multiplexed gratings on a single layer. Other than the EPE, another benefit is the integration of prescriptions for a better user experience. Based on the simulation, its overall performance, including the efficiency, FOV, AR, MTF, CR, distortion, and simulated imaging have been investigated. For the real image, the FOV is 133° (diagonal), MTF is above 0.252 at 30 cycle/degree, CR is 5, and distortion is 0.02%. For the virtual image, the total efficiency is 0.55%, FOV is 33° (diagonal), AR is 2.19′, MTF is above 0.032 at 30 cycle/degree, CR is 3, and distortion is 3.4%. Merits aside, low efficiency, pupil mismatch, vignetting, etc. are identified as the remaining issues and will be dealt with in our future work.