Effects of Depth-Based Object Isolation in Simulated Retinal Prosthetic Vision

Abstract

Visual retinal prostheses aim to restore vision for blind individuals who suffer from outer retinal degenerative diseases, such as retinitis pigmentosa and age-related macular degeneration. Perception through retinal prostheses is very limited, but it can be improved by applying object isolation. We used an object isolation algorithm based on integral imaging to isolate objects of interest according to their depth from the camera and applied image processing manipulation to the isolated-object images. Subsequently, we applied a spatial prosthetic vision simulation that converted the isolated-object images to phosphene images. We compared the phosphene images for two types of input images, the original image (before applying object isolation), and the isolated-object image to illustrate the effects of object isolation on simulated prosthetic vision without and with multiple spatial variations of phosphenes, such as size and shape variations, spatial shifts, and dropout rate. The results show an improvement in the perceived shape, contrast, and dynamic range (number of gray levels) of objects in the phosphene image.

1. Introduction

Retinitis pigmentosa and geographic atrophy are outer retinal progressive degenerative diseases that cause a loss of photoreceptor cells. Both diseases currently have no cure [1,2] and together affect almost 0.7% of the global population [3,4]. One approach intended to help with blindness caused by outer retinal diseases is the use of a retinal prosthesis, which produces artificial electrical stimulation of the remaining healthy cells in the other layers of the retina.

During the late sixties, several researchers demonstrated that blind people can perceive electrically elicited light blobs, referred to as “phosphenes”, by stimulating electrodes in contact with the occipital pole of the right cerebral hemisphere [5]. Since then, many visual prostheses have been developed. Retinal prostheses are a type of visual prostheses in which a microelectrode array is placed on the outer surface of the retina (subretinal prostheses), the inner surface of the retina (epiretinal prostheses), or the suprachoroidal layer (suprachoroidal prostheses), and remaining functioning cells of the retina are stimulated. Retinal prostheses aim to restore vision to blind individuals suffering from outer retinal progressive degenerative disorders, such as retinitis pigmentosa and age-related macular degeneration [6,7,8,9,10,11].

Spatial perception with current retinal prostheses is primarily limited by the low resolution (small number of electrodes). This has been addressed and demonstrated in most prior simulations [12,13,14,15]. Several other spatial variations of prosthetic vision have been reported. The first is phosphene size variation, which may be caused by variations in the stimulation amplitude, electrical conductivity, the efficiency of the electrodes [7,16,17,18,19], or different sizes of ganglion receptive fields [20]. The second is phosphene shape variation. Phosphenes frequently appear round, but they may also appear as elongated shapes [8,17,21,22,23]; the elongated or elliptical shapes may be caused by unintended activation of axon bundles [20,24]. The phosphene shape may also be affected by the current amplitude of the stimulation [25]. The third factor influencing the perception is represented by dynamic random spatial shifts of the phosphenes. The spatial arrangement of the phosphenes in the subject’s visual field was found to be correlated to the spatial arrangement of the electrodes [7,16,26,27,28,29]. However, this correspondence may be distorted by a mismatch between the electrode array and the underlying retinal ganglion cell (RGC) array [7,17].

Spatial variations have already been implemented in previous simulations. Dagnelie et al. [12] simulated variations in the sizes of phosphenes, the gap between phosphenes, varying number of electrodes (10 × 10, 16 × 16, and 25 × 25 electrodes), varying dynamic ranges (2, 4, and 8 gray levels), and different dropout rates (30, 50, and 70% of electrodes not eliciting a phosphene upon stimulation) and examined their impact on reading ability. Xia et al. [30] studied the simulated spatial shifts (phosphenes appear shifted with respect to the retinotopic position of the electrode array), variations in dynamic range (2, 4, 6, and 8), and different phosphene dropout rates (10, 20, 30, and 40%) on object recognition. Wu et al. [31] simulated spatial shifts, shape variations (50% of phosphenes reshaped into 12 different predefined elliptical shapes), and phosphene dropout (20%).

Epiretinal and suprachoroidal prosthetic devices use a video camera that captures the scene [6,10,11,32]. Image processing is then performed to fit this captured video to the prosthetic device. During this process, the 3D scene is converted into an appropriate 2D image. This conversion adds a distortion to the scene in addition to the spatial limitations of the prosthesis. Among these limitations are low resolution (small number of electrodes) [33], limited visual field [34], limited dynamic range (about 4–12 gray levels) [35], and cluttering. Cluttering is a situation in which an object of interest (OI) is confused or barely recognized because of the background or other objects in the original 3D scene.

Integral imaging (InI), originally proposed about a century ago [36], has become popular in the last two decades because of advances in digital imaging and optoelectronic technologies. In InI, 3D scene data are produced by capturing elemental images (EIs) optically using a pickup lens array and a detector array or by using multiple cameras arranged in a matrix form [37]. Integral imaging acquisition may be viewed as a multichannel system in which each channel generates an EI, which is a 2D image with its own perspective. These over-informative data permit the 3D visualization of objects. A 3D object isolation (or decluttering) algorithm using computational InI data has already been developed [38,39].

The limited number of retinal implant users (approximately 500 globally) [33] makes it difficult to conduct psychophysical studies with implant users and highlights the importance of simulating prosthetic vision. Prosthetic vision simulations are based on past psychophysical studies and vision research and should mimic the perception properties of retinal implants. These simulations demonstrate to normally sighted people how a blind person with a prosthetic device sees, and they enable the isolation of each spatial or temporal property and the investigation of its effects on prosthetic vision [40]. Simulations also help in examining how different image acquisition and image processing techniques affect prosthetic vision, which may also help in the specification of future developments of prosthetic devices and make them more practical for their users [35].

Because of the abovementioned spatial limitations and distortions in prosthetic vision, as well as several temporal distortions, such as persistence and perceptual fading of phosphenes [40,41], there is a strong demand for improving the quality of prosthetic vision. Here, we address this issue by applying an object isolation algorithm based on the depth information obtained from a computational InI, a 3D imaging technique that is currently not used in any of the available prosthetic devices. Subsequently, several image processing techniques were used to achieve additional improvement of the prosthetic images. This study presents results from multiple simulations.

The effects of object isolation in prosthetic vision were previously proposed and implemented in a binary representation [42]. However, this technique was not tested in a simulated prosthetic vision application. In this paper, we reveal the advantages of isolated-object input images compared with original scene input images (that contain multiple objects) in simulated prosthetic vision characterized by two phosphene models: symmetric phosphenes (i.e., without spatial variations) and asymmetric phosphenes (i.e., in the presence of multiple spatial variations). Our simulations are presented in both the original and reversed polarities (i.e., bright pixels are presented as dark and vice versa), and we discuss the advantages of these two presentations.

2. Methods

2.1. Scene Capture

In the experimental setup, a digital single-lens reflex (DSLR) camera (Nikon D500) was attached to a translator and took 100 EIs (10 rows of 10 EIs) by a shift and capture process. The horizontal and vertical translations between every two EIs were 5 mm. Each EI was taken at a resolution of 1856 × 2784 pixels, converted into grayscale, and then cropped and resized into a resolution of 500 × 1000 pixels (Figure 1) to reduce the algorithm running time. The captured scene is of a room that contains a laptop, a school bag, posters that were placed behind the school bag, and various less prominent objects. The laptop and the school bag were placed at distances of approximately 0.5 m and 1.2 m from the camera, respectively. The posters were placed behind the bag, approximately 2.3 m from the camera, to test the algorithm with a non-uniform background.

2.2. Object Isolation Algorithm

The benefits of object isolation on simulated prosthetic vision images were examined by using a method developed for depth-based 3D object isolation [38,39]. For this goal, computational integral imaging was performed using a matrix of EIs. The depth range available for the reconstruction of the depth planes was defined according to the depth of field of a single EI. A reconstructed image of the integral imaging system at $z_0$ depth is [37]:

$$f^{BP}\left(x,y,z_0\right)=\frac{1}{KL}{\displaystyle \sum }_{k=0}^{K-1}{\displaystyle \sum }_{l=0}^{L-1}{g}_{kl}\left(x+\left(\frac{1}{M_0}\right)S_{x}k,y+\left(\frac{1}{M_0}\right)S_{y}l\right),$$

(1)

where $g_{kl}$ is the $K$ × $L$ EI array; $k$ and $l$ are indices for the particular EI; $M_0$ is the magnification factor, which depends on the distance between the camera and the reconstructed plane $z_0$, $S_x$ and $S_y$ are the translations between two consecutive locations of the camera along $x$ and $y$ directions, respectively; and $f^{BP}\left(x,y,z_0\right)$ is a 2D reconstructed image at a distance $z_0$ from the camera. Objects within the scene appear sharp when reconstructing the image at their depth, whereas objects far away from the chosen depth are blurred.

In the next phase, the 3D depth locations of objects in the 3D space were determined [39]. According to the assumption that the focused regions are obtained at higher frequencies [38], the higher local gradient values of the gradient image along the depth axis were determined:

$$f_{Grad}\left(x,y,z_0\right)=\nabla ·\left(f^{BP}\left(x,y,z_0\right)\right).$$

(2)

The threshold value was calculated according to the average gradient magnitude of the reconstructed images (AGMR) [39]:

$$AGMR\left(z_0\right)=\frac{1}{N_x·N_y}{{\displaystyle \sum }}_y{{\displaystyle \sum }}_x\left|f_{Grad}\left(x,y,z_0\right)\right|,$$

(3)

where $N_x$ and $N_y$ are the numbers of pixels along x and y directions, respectively. Plotting the average gradient magnitude values against the depth locations on a graph produces local maxima in depths that include focused regions (i.e., locations of objects). Then, a spatial segmentation based on a sharpness criterion via an adaptive threshold was applied to the reconstructed images at the depths determined using the AGMR function [38,39].

The segmented planes obtained in this process are the depth-based isolated-object images intended to be the isolated-object inputs to the prosthesis. Another option for a prosthesis input can include all the isolated objects that may represent prominent objects in the observed scene. For this case, using the knowledge that computational integral imaging may observe regions hidden behind the front objects, and that the background in the isolated-object images is set to black in all the isolated-object images, the following procedure was carried out. The pixels that represented the front object (all the non-zero pixels) in the isolated-object image of the closest depth plane (to the camera) were identified. Next, the corresponding pixels were zeroed in the next-closest isolated-object images (i.e., the next-closest object’s region hidden behind the one in the front). This process was repeated for all the isolated-object images of the farther depth planes until arriving at the isolated-object image of the farthest depth plane. Thus, in the case of spatial overlap between OIs, the algorithm uses the front object, similarly to normal vision. The following equation is used to produce an image that includes all the isolated objects:

$$I_{OIs}={\displaystyle \sum }_{i=1}^n{I}_i$$

(4)

where $I_{OIs}$ is the image that contains all the isolated objects of interest, n is the number of OIs in the scene (or the number of depth planes detected with prominent objects), and $I_i$ is the isolated-object image at the ith depth plane.

2.3. Prosthetic Vision Simulation

A prosthetic vision simulation was built in MATLAB R2018a (Mathworks Inc., Natick, MA, USA). The simulation input was a grayscale image; the output was a simulated prosthetic vision image that includes discrete or overlapping phosphenes in grayscale and was presented on a desktop display.

Phosphenes usually appear round [8,16,17,21,29] and white or lightly colored [8,16,21]; thus, they were implemented accordingly. Phosphenes do not appear with sharp edges [12], and thus a two-dimensional elliptical Gaussian function was used to present a phosphene as follows:

$$\begin{gathered} f\left(x,y\right)=Aexp\left\{-\left[a{\left(x-x_0\right)}^2+2b\left(x-x_0\right)\left(y-y_0\right)+c{\left(y-y_0\right)}^2\right]\right\}, \\ a=\frac{{\cos }^2\theta }{2{\sigma }_x^2}+\frac{{\sin }^2\theta }{2{\sigma }_y^2}, \\ b=-\frac{\sin 2\theta }{4{\sigma }_x^2}+\frac{\sin 2\theta }{4{\sigma }_y^2}, \\ c=\frac{{\sin }^2\theta }{2{\sigma }_x^2}+\frac{{\cos }^2\theta }{2{\sigma }_y^2}, \end{gathered}$$

(5)

where $f\left(x,y\right)$ is the pixel’s value in the location (x,y); $A$ represents the phosphene brightness at the center and is the mean value of a pixel block in the high-resolution camera’s frame, which, in our case, was quantized into four normalized values (0, 1/3, 2/3, and 1); ($x_0,y_0$) is the phosphene’s center position coordinates; ${\sigma }_x$ and ${\sigma }_y$ represent the spread of the phosphene along both axes; and the angle $\theta$ represents the clockwise rotation of the ellipsoid-shaped phosphene. If ${\sigma }_x={\sigma }_y$, then b = 0, and the phosphene is round; otherwise, the phosphene is elliptical. Adding the spatial variations to Equation (5) gives:

$$\begin{array}{c}f\left(x,y\right)=Aexp\{-S[a{\left(x-\left(i{x}_0+x_1\right)\right)}^2+2b\left(x-\left(i{x}_0+x_1\right)\right)\left(y-(j{y}_0+y_1)\right)\\ +c{\left(y-(j{y}_0+y_1)\right)}^2 ]\},\end{array}$$

(6)

where the indices $i, j$ represent the coordinates of the phosphene’s center point on the x and y axes, respectively; $S$ represents the size variation; and $x_1,y_1$ represent the spatial shifts. If these variations are disabled in the simulation, then the values of $S$, $x_1, y_1,$ and $\theta$ are equal to zero, and ${\sigma }_x \mathrm{and} {\sigma }_y$ are equal to 1. If the variations were enabled, $A$ was equal to zero for 20% of the phosphenes (randomly), $S$ was uniformly distributed between 0.5 (i.e., 50% smaller phosphene) and 1.5 (i.e., 50% larger phosphene), and $x_1$ and $y_1$ were uniformly distributed within a range from 0 to 4 pixels. ${\sigma }_x$ was uniformly distributed between 1 and 3, ${\sigma }_y$ was uniformly distributed between 1/3 and 1, and $\theta$ was uniformly distributed between zero (i.e., the phosphene is horizontal) and 180 (i.e., the phosphene is horizontal again after a clockwise rotation of 180 degrees).

Table 1. The spatial parameters of the prosthetic vision simulation.

Spatial Parameter	Symbol	Description
Size variations	$S$	Random uniformly distributed variation of the phosphenes’ sizes between 30% smaller than nominal and 30% larger than nominal.
Shape variations	${\sigma }_x,{\sigma }_y,\theta$	Random change in the shape of 15% of the phosphenes into an ellipse with a random angle (0–180°) of rotation.
Spatial Shifts	$x_1, y_1$	Random uniformly distributed shifts of phosphenes by 0–4 pixels with respect to the retinotopic position of their corresponding electrodes in the array, in both the vertical and horizontal directions.
Dropout	$A$	Random elimination of 20% of the phosphenes.

summarizes the spatial parameters and describe them, and

Table 2. The values employed for the spatial parameters of the prosthetic vision simulation.

		$\mathit{S}$	${\mathit{\sigma }}_{\mathit{x}}$	${\mathit{\sigma }}_{\mathit{y}}$	$\mathit{\theta }$ (deg)	${\mathit{x}}_1\left(\mathbf{pxls}\right)$	${\mathit{y}}_1\left(\mathbf{pxls}\right)$	$\mathit{A}$
Size variations	Disabled	1
	Enabled	0.5–1.5
Shape variations	Disabled		1	1	0
	Enabled		1–3	1/3–1	0–180
Spatial Shifts	Disabled					0	0
	Enabled					0–4	0–4
Dropout	Disabled							0, 1/3, 2/3, 1
	Enabled							0

provides the specific values for each parameter.

Size variations and spatial shifts were applied to all phosphenes, shape variations were applied to only 15% because it is likely that only a few of the elicited phosphenes appear elliptical rather than round [7,16,17,21,22], and dropout ($A$) was applied to 20% of the phosphenes because it is unlikely that most of the electrodes are unable to evoke phosphenes. Figure 2 shows the different phosphene sizes and shapes.

An electrode array that includes 15 × 30 electrodes in a rectangular structure was chosen to demonstrate the advantages of our technique on simulated prosthetic vision at a reasonable spatial resolution. Each stimulating electrode elicited one phosphene in the visual field. The phosphenes were rendered in a full 8-bit display, but the range of their maximum intensities was quantized into a 2-bit dynamic range (i.e., 4 different phosphene intensities).

The input image was divided into square pixel blocks so that the number of blocks matched the number and organization of the electrodes [40]. The original resolution of the input image (500 × 1000 pixels) was decreased to the low prosthetic vision resolution, which was set to 105 × 210 pixels to grossly fit the field of view of the Argus II prosthesis—22° diagonally [34] on a head-mounted display (HMD) screen with a resolution of 960 × 1080 pixels and a field of view of 90° × 110° [43]. The average gray value of each pixel block in the image was calculated, and a phosphene was generated accordingly within the block region. The brightness of each phosphene was set based on the corresponding quantized value of the pixel block average, for which a higher average result in a brighter phosphene [40].

The phosphene images were rendered in two different ways: using discrete or overlapping phosphenes. The overlap rendering added uniformly distributed random size variations of phosphenes that resulted in overlaps among phosphenes. The simulation has a feature that enables switching between the original and reversed polarities of the input grayscale image, similarly to Argus II [44]. In the reversed polarity, each pixel has a new value equal to 255 minus the original value. A flow chart of the entire process, including the object isolation algorithm and the prosthetic vision simulation, is presented in Figure 3.

3. Results

3.1. Object Isolation

First, we determined the depths of the objects presented in Figure 1b. Figure 4 depicts the AGMR graph; the peaks in the graph represent the average gradient magnitude of the sharpest reconstructed images, which are at the depth locations of the significant objects in the 3D scene. The AGMR graph shows a peak at 190 mm and a significant change in the AGMR slope at 1190 mm. These values were correlated with the depths of the laptop and school bag, respectively. The reconstructed images at the depths identified in the AGMR graph are presented in Figure 5.

3.2. Prosthetic Vision Views of the Isolated Objects

The rest of the object isolation process was performed [39]. The isolated-object images of the laptop and school bag were obtained, as presented in Figure 6. Direct prosthetic vision simulations of these images yielded a partial or no vision of the OIs because of the low dynamic range in prosthetic vision. After we applied contrast stretch to the isolated-object images, the school bag became visible, but the laptop remained only partially visible. Then, we changed the black background to white before we applied contrast stretch. This made both objects visible under the low dynamic range and resolution conditions of the simulated prosthetic vision.

3.3. Prosthetic Vision Views before and after Isolation

Figure 7 compares the simulated prosthetic vision images of symmetric phosphenes before and after the object isolation, background adjustment, and contrast stretch in both the original and reversed polarities. Figure 7a presents the original and simulated images of the scene from Figure 1b before applying any image processing manipulation to the input image. Figure 7b presents the original and simulated images of the isolated laptop from Figure 6(1)c. Figure 7c presents the original and simulated images of the isolated bag from Figure 6(4)c. Figure 7d presents the original and simulated images of the isolated OIs(according to Equation (4)).

Figure 7b shows the contrast and dynamic range improvement that allows the perception of the white sticker on the laptop (in the red circle), as well as an improved perception of the keyboard and the screen. Figure 7c depicts the significant improvement in the perception of the school bag due to object isolation.

3.4. Prosthetic Vision Views before and after Isolation with Spatial Phosphene Variations

Figure 8 illustrates the effects of size variations, dropout, shape variations, and spatial shifts (non-uniform spacing) of phosphenes and a combination thereof in creating asymmetric phosphenes for a white input image. Figure 9 is the same as Figure 7 but with the addition of the spatial phosphene variations presented in Figure 8.

Figure 10 shows another example of the algorithm for a different scene. In this scene, the algorithm isolated the first two objects that were closest to the camera.

4. Discussion and Conclusions

Despite the rapid progress in visual prosthetic devices, the spatial resolution remained low, with a visual acuity of 20/4275 for the Gen 2 suprachoroidal device [32], 20/1260 for the Argus II epiretinal prosthesis [28], and 20/546 for the ALPHA-IMS subretinal prosthesis [9]. Increasing the number of electrodes to improve the spatial resolution is challenging because of the limited power dissipation [45], crosstalk between electrodes [45,46], tissue heating [45], etc. [45,47]. Additionally, using sequential (asynchronous) stimulation, which reduces the temporal noise with many electrodes, limits the stimulation frequency [46]. It has also been found that the number of electrodes has a limited contribution to visual performance [48,49]. In addition to the low spatial resolution, numerous spatial and temporal effects can reduce the quality of prosthetic vision [7,18,19,22,40,41,50]. Using image processing approaches, such as the method proposed in this study, may be a desirable solution to improve the clarity of objects in prosthetic vision and reduce some of the distracting spatial and temporal effects.

We demonstrated the potential advantages of object isolation based on InI in simulated prosthetic phosphene vision. As part of the object isolation algorithm, a segmentation process based on a sharpness criterion was implemented to isolate the OIs in their depth planes. Our results agree with the conclusion that the depth-based object isolation process enables improved perception of the OI [42] by generating an isolated-object input, which avoids overlaps between the OI and the background or other objects.

A subject implanted with an epiretinal prosthesis has a low perceived dynamic range, which was reported to be as small as 2–4 gray levels [35], suggesting that the retinal neurons distinguish between a small number of distant stimulation levels within the safe charge density, which is $0.35 mC/c{m}^2$ for platinum electrodes [51,52] and 3$mC/c{m}^2$ for iridium oxide electrodes [53]. In our prosthetic vision simulation, the number of gray levels decreased from the standard 256 to 4. Therefore, enhancing the contrast and dynamic range of the OI in the isolated-object input image by applying contrast stretch allows the prosthesis to use four stimulation currents that are as far from each other as possible.

In the isolated-object image, the whole image except for the OI becomes uniform, usually black or white. When choosing between a black or a white background, the algorithm should consider the brightness of the OI. As shown in Figure 7b and Figure 9b, applying contrast stretch to the isolated-object input image at this stage affects the simulated prosthetic image by increasing the OI’s contrast (light gray phosphenes are being perceived in addition to dark gray phosphenes) and dynamic range (three levels instead of two in our example). This may enable brightness changes inside the OI, such as the sticker on the laptop, to be distinguished.

The proposed method may involve automatic isolation of predefined objects, which are of interest to implanted patients. This could be achieved by an object classification algorithm using machine learning. The applied object isolation technique may allow users to control the depth of their vision and thus increase their performance in multiple-object scenes. For instance, controlling the depth of vision could be achieved by a scroll wheel that would enable the users to see only objects within a predefined depth range; the objects outside this range would not be visible [42] (Figure 7b,c and Figure 9b,c). Alternatively, the input image of the prosthesis could contain objects at various depths, whereas other less prominent information associated with the background of the scene are removed (Figure 7d and Figure 9d).

Applying polarity reversal, in which the object is bright and the background is dark, has the advantage of involving fewer electrodes that stimulate bright phosphenes, thus decreasing the spread of distortions caused by spatial variations, such as phosphene shape [20], size [7,8], non-uniform spatial shifts between phosphenes [7,53], and temporal variations of phosphenes, such as persistence and perceptual fading of phosphenes, in retinal prosthetic devices [40].

Our results suggest that object isolation may improve prosthetic vision regardless of the spatial variations, that is, in the simulated prosthetic vision of both symmetric and asymmetric phosphenes. The improvement in the simulated prosthetic vision images demonstrated in our study suggests that the technique presented in this work may improve prosthetic vision, and it should be considered in the development of future prosthetic devices.

A possible future direction of this study involves conducting experiments with normally sighted subjects to measure the performance improvement in different visual tasks, such as object detection, recognition, and identification. Another future direction is implementing the simulation in real time with an HMD. This would require a decrease in the running time by optimizing the algorithm. Such a real-time simulation will enable the testing of algorithm performance in other visual tasks, such as navigation and object scanning.