Smart Glasses for Visually Evoked Potential Applications: Characterisation of the Optical Output for Different Display Technologies ^†

Abstract

Off-the-shelf consumer-grade smart glasses are being increasingly used in extended reality and brain–computer interface applications that are based on the detection of visually evoked potentials from the user’s brain. The displays of these kinds of devices can be based on different technologies, which may affect the nature of the visual stimulus received by the user. This aspect has substantial impact in the field of applications based on wearable sensors and devices. We measured the optical output of three models of smart glasses with different display technologies using a photo-transducer in order to gain insight on their exploitability in brain–computer interface applications. The results suggest that preferring a particular model of smart glasses may strongly depend on the specific application requirements.

1. Introduction

Emerging brain–computer interfaces (BCIs) are being extensively investigated in the scientific community [1]. BCIs provide a novel means of communication which relies on the direct detection of brain signals. The applications of BCIs are oriented to impaired and able-bodied people, with several already explored applications such as robot control [2], industrial inspection [3], and neurological rehabilitation [4,5,6]. Among various paradigms, the so-called reactive BCI, in which the user is exposed to a stimulus and the evoked brain response is detected, are the best-performing ones [7,8,9]. In particular, BCIs based on visually evoked potentials (VEPs) are well suited for communication and control applications [3,10,11]. Visual stimulation in VEP-BCI can be performed by means of off-the-shelf smart glasses, which can generate icons of different colour, shape, position and blinking rate in the user’s field of view [3,12]. Smart glasses based on different display technologies are available on the market. Typically, smart glasses exploit video see-through or optical see-through technology. The former consists of displaying virtual objects in superposition with a video recording of the real environment, whereas the latter exploits semi-transparent displays allowing normal vision with superimposed virtual elements [13]. There exist commercial devices that rely on either liquid crystal displays (LCDs) with active matrices of poly-crystalline silicon thin-film-transistors, silicon-based organic light emitting diode (OLED) matrices, planar waveguides, or other technologies at various development stages [14,15,16]. Independently of the display technology, the visual stimuli must have a number of specific characteristics to be suitable for VEP-BCI applications. These characteristics may strongly depend on the technology and the implementation of the smart glasses, first of all on the ability of generating stimuli with specific and distinguishable harmonic contents. In this paper, we report and discuss measurements of the optical output of smart glasses based on three different technologies and discuss the results in terms of VEP-BCI applications.

2. Materials and Methods

We characterised three models of commercially-available smart glasses based on different technology: (i) Epson BT-200, (ii) Epson BT-350, and (iii) Microsoft Hololens. The BT-200 employs an LCD display, the BT-350 employs an OLED display, and the Hololens’s display is based on waveguides. The presented devices are representative of the most recent consumer-grade technologies available on the market. The smart glasses were programmed to generate a white rectangular icon blinking at frequency f on one lens. To characterise the optical output of the above-mentioned smart glasses, we implemented a phototransducer (our approach is similar to the one described by international standards series IEC 62087 concerning measurement of the brightness of displays [17]). The photo-transducer (PT), schematically shown in Figure 1, was based on a commercial photodiode (OPT-101) integrated with a transimpedance amplifier [18]. The PT was battery-powered to avoid power line interference. The microcontroller $\mathsf{\mu }$C STM32F401RE [19] was used to measure the output voltage $V_{\mathrm{PT}}$ of PT and to transmit the measured values to a PC. The microcontroller’s ADC has a resolution of 12 bit and was operated at a sampling frequency of 1 kHz. The transducer PT was placed in front of the smart glasses display with the blinking icon extending on the whole sensor area. In each case, the positioning was empirically determined by looking for the maximum measurable voltage, so that the sensor was uniformly illuminated by the icon.

Figure 1. Schematic representation of the experiment, with the smart glasses and the phototransducer PT. The dashed arrows represent the light coming from the blinking icon generated by the smart glasses. The filled black square represents the PD-sensitive area.

In order to obtain meaningful results, we checked the overall bandwidth of our phototransducer by measuring a 12 $\mathrm{Hz}$ square-wave optical signal generated with an LED driven by a function generator SRS DS360. The DS360 had a nominal rise time of $1.3$ $\mathsf{\mu }$$\mathrm{s}$ [20], whereas the LED rise time was comparably negligible, making the test signal bandwidth large enough with respect to the PT’s bandwidth (limited by the ADC). The 1 $\mathrm{k}$$\mathsf{\Omega }$ LED series resistance was high enough to achieve a negligible current load for the signal generator, thus avoiding signal distortion. These preliminary measurements highlighted a discrepancy between the nominal and the measured harmonic ratios for a test square wave of less than 0.5% up to the 11th component of the measured signal $V_{\mathrm{PT}}$.

The optical output of each of the tested smart glasses was recorded in a dark room for 10 $\mathrm{s}$ and analysed up to a frequency of 100 $\mathrm{Hz}$, high enough for the requirements of VEP-BCI applications. This maximum frequency was well below the ADC’s Nyquist frequency to avoid artefacts due to aliasing.

3. Results

Figure 2 shows the optical output, represented by the ADC voltage $V_{\mathrm{PT}}$, of the three smart glasses when programmed to generate a blinking icon (a square wave visual stimulus) at $f=10\mathrm{Hz}$. The signals in the time domain are shown in Figure 2a–c within a time frame of 1 $\mathrm{s}$. The corresponding plots in the frequency domain are shown in Figure 2d–f in the range from 0 $\mathrm{Hz}$–100 $\mathrm{Hz}$. From Figure 2a–c, it can be seen that the three smart glasses, programmed to generate the same nominal signal, produce substantially different outputs. In particular, the BT-200 (LCD) produces a rather symmetric wave profile, whereas the BT-350 (OLED) produces a strongly asymmetric profile, but in both cases, the intensity of their output changes in a smooth way. By contrast, the output of the Hololens smart glasses appears to be composed of a series of fast pulses. Furthermore, there are three distinct sets of pulses with different intensities, corresponding to the red, green, and blue colour components, as verified with separate measurements. Finally, in the time-domain plots of Figure 2a,b, the waveforms have different rise and fall profiles, probably related to the different on and off switching times of the display elements.

Figure 2. Optical output $V_{\mathrm{PT}}$ of the characterised smart glasses. Time domain signals of BT-200 (a), BT-350 (b), and Hololens (c). Frequency domain spectra of BT-200 (d), BT-350 (e), and Hololens (f). Each spectrum (d–f) is normalised to its first harmonic.

Figure 2d–f show the output harmonic content of the tested smart glasses. The BT-200 and BT-350 present a rather well-defined harmonic content, whereas the Hololens does not allow one to distinguish any frequency component, whether the fundamental at frequency f or the higher harmonics. Here, the asymmetric distortion seen in the time domain plots of Figure 2a,b is reflected in the spectra shown in Figure 2d,e, which in addition to the odd-harmonics of a symmetric square-wave ( 12 $\mathrm{Hz}$, 36 $\mathrm{Hz}$, ...), also contain even harmonics ( 24 $\mathrm{Hz}$, 48 $\mathrm{Hz}$, ...).

4. Discussion

Given the present results, some considerations in terms of the applicability of these off-the-shelf smart glasses to VEP-BCI can be drawn. None of the tested devices produce an optical output which is a square wave, compared to the LED-generated signal used to test PT. In particular, even harmonics are present in both the BT-200 and BT-350 smart glasses (see Figure 2d,e) and the harmonic ratios of the odd harmonics are substantially greater than 0.5% with respect to the reference harmonic ratios given by the test signal. The Hololens smart glasses mark a separate case not showing any distinguishable harmonic component.

It appears that the display technology of the compared smart glasses has a strong effect on the nature of the generated visual stimuli, for the same given test waveform. This can be related to the specific behaviour of the display matrix and to the practical implementation of the control electronics. For example, the nominal effective rate of the BT-200 is 60 $\mathrm{Hz}$, while it is 30 $\mathrm{Hz}$ for the BT-350 and 240 $\mathrm{Hz}$ for the Hololens, with four sub-frames corresponding to a 60 $\mathrm{Hz}$ effective rate. It is evident from Figure 2d,f that, besides having the same effective effective rate of 60 $\mathrm{Hz}$, the BT-200 and the Hololens do not produce a similar optical output at all.

However, these features do not imply an equivalence at all costs between the optical output and the effect of the visual stimulus on the user brain potentials. In fact, in typical VEP-BCI applications, multiple icons blinking at different frequencies, for instance 10 $\mathrm{Hz}$ and 12 $\mathrm{Hz}$, are shown to the user. The user’s evoked brain potentials are then measured to detect their relative attention to the two icons. In this case, the distinguishability of the harmonic component can be sufficient and any other spurious content may not be detrimental for the task [3,6]. Moreover, it cannot be excluded that in terms of the net effect of the visual stimulus on the user, even the Hololens can be perfectly functional to VEP-BCI applications.

In a different sense, if these devices would be considered to study quantitative effects of visual stimuli on brain activity [21], the above-discussed features may be a stronger discriminant.

5. Conclusions

In principle, commercial smart glasses can be programmed to generate a blinking icon with a square wave profile, typically used as a visual stimulus in VEP-BCI applications. However, the ability to generate well-separated harmonics is dependent on the implemented display technology. This may be not a problem in terms of sensory stimulation for the elicitation of specific brain potentials. Nevertheless, the choice of the visual stimulation technology may strongly depend on the particular target application. As a future work, it is worth studying how the different optical outputs may affect the brain potentials during VEP-BCI experiments.