Time Sequential Motion-to-Photon Latency Measurement System for Virtual Reality Head-Mounted Displays

Because the interest in virtual reality (VR) has increased recently, studies on head-mounted displays (HMDs) have been actively conducted. However, HMD causes motion sickness and dizziness to the user, who is most affected by motion-to-photon latency. Therefore, equipment for measuring and quantifying this occurrence is very necessary. This paper proposes a novel system to measure and visualize the time sequential motion-to-photon latency in real time for HMDs. Conventional motion-to-photon latency measurement methods can measure the latency only at the beginning of the physical motion. On the other hand, the proposed method can measure the latency in real time at every input time. Specifically, it generates the rotation data with intensity levels of pixels on the measurement area, and it can obtain the motion-to-photon latency data in all temporal ranges. Concurrently, encoders measure the actual motion from a motion generator designed to control the actual posture of the HMD device. The proposed system conducts a comparison between two motions from encoders and the output image on a display. Finally, it calculates the motion-to-photon latency for all time points. The experiment shows that the latency increases from a minimum of 46.55 ms to a maximum of 154.63 ms according to the workload levels.


Introduction
Because the personal computer's performance has been greatly improved, real-time rendering of high-quality images has become possible, and virtual reality (VR) technology has become a reality [1].According to this trend, a variety of VR devices utilizing 3D rendering and sensor-based technology have been released.The head-mounted display (HMD) device, which is a system designed to improve immersion by mounting a wide field-of-view display within the user's sight, has been gaining popularity [2].However, because of the mismatch between visual and vestibular systems, users wearing HMD devices experience motion sickness and dizziness, which can be an obstacle to the VR market.The motion-to-photon latency, which is one of the causes for this mismatch, is the time delay for a user movement to be fully reflected on a display screen [3].Generally, three steps are required to render an image, as shown in Figure 1.When head motion occurs, the motion detection unit samples the orientation data for the view generation.After the motion detection, the visual processing unit renders a 3D image.Finally, the rendered image is outputted to a display corresponding to the head orientation of the user measured by the sensor.As these steps take time, the delay results in motion-to-photon latency.In this case, the image does not exactly correspond to the actual head orientation of the user, thereby causing the user to experience motion sickness [4].Therefore, many HMD makers such as Oculus VR have conducted studies to minimize the mismatch caused by the motion-to-photon latency.
Numerous studies such as prediction techniques based on the data acquired by inertial measurement unit (IMU) sensors [5] and asynchronous time warp (ATW) [6] have been conducted to overcome this limitation.The quantitative evaluation of the above methods required the authors to measure the motion-to-photon latency.They could then improve and evaluate these methods with reference data.
Electronics 2018, 7, x FOR PEER REVIEW 2 of 13 minimize the mismatch caused by the motion-to-photon latency.Numerous studies such as prediction techniques based on the data acquired by inertial measurement unit (IMU) sensors [5] and asynchronous time warp (ATW) [6] have been conducted to overcome this limitation.The quantitative evaluation of the above methods required the authors to measure the motion-to-photon latency.They could then improve and evaluate these methods with reference data.To measure this motion-to-photon latency, the method of [7] presents a low-cost system with high accuracy which measures the latency.However, it can be used for the mobile device.The method of [8] proposes the measurement system and a technique to reduce the latency for the optical see-through display.The authors had previously proposed a new measurement system by using multiple sensors such as an optical sensor and encoders and by comparing the physical signal to the luminance signal from a VR scene reflected on the display [9]. Figure 2 shows the overall architecture of this measurement system.A rotary platform, which controls the physical motion, was proposed in that method to simulate the rotation of the neck.It measured the change of the image brightness when the physical motion commenced.The method could directly measure the motion-to-photon latency because it calculated the time difference between the point at which the brightness startedchanging and that at which the physical motion started However, although this method provided accurate measurement results, its limitation was that the latency could only be measured when the physical motion began.To measure this motion-to-photon latency, the method of [7] presents a low-cost system with high accuracy which measures the latency.However, it can be used for the mobile device.The method of [8] proposes the measurement system and a technique to reduce the latency for the optical see-through display.The authors had previously proposed a new measurement system by using multiple sensors such as an optical sensor and encoders and by comparing the physical signal to the luminance signal from a VR scene reflected on the display [9]. Figure 2 shows the overall architecture of this measurement system.A rotary platform, which controls the physical motion, was proposed in that method to simulate the rotation of the neck.It measured the change of the image brightness when the physical motion commenced.The method could directly measure the motion-to-photon latency because it calculated the time difference between the point at which the brightness startedchanging and that at which the physical motion started However, although this method provided accurate measurement results, its limitation was that the latency could only be measured when the physical motion began.minimize the mismatch caused by the motion-to-photon latency.Numerous studies such as prediction techniques based on the data acquired by inertial measurement unit (IMU) sensors [5] and asynchronous time warp (ATW) [6] have been conducted to overcome this limitation.The quantitative evaluation of the above methods required the authors to measure the motion-to-photon latency.They could then improve and evaluate these methods with reference data.To measure this motion-to-photon latency, the method of [7] presents a low-cost system with high accuracy which measures the latency.However, it can be used for the mobile device.The method of [8] proposes the measurement system and a technique to reduce the latency for the optical see-through display.The authors had previously proposed a new measurement system by using multiple sensors such as an optical sensor and encoders and by comparing the physical signal to the luminance signal from a VR scene reflected on the display [9]. Figure 2 shows the overall architecture of this measurement system.A rotary platform, which controls the physical motion, was proposed in that method to simulate the rotation of the neck.It measured the change of the image brightness when the physical motion commenced.The method could directly measure the motion-to-photon latency because it calculated the time difference between the point at which the brightness startedchanging and that at which the physical motion started However, although this method provided accurate measurement results, its limitation was that the latency could only be measured when the physical motion began.In this paper, the authors propose a novel time sequential measurement system for the motion-to-photon latency by comparing the physical motion and the motion reflected on the image by 3D rendering.Therefore, it is possible to measure and record the time sequential latency more precisely in real-time, while reflecting the change in workload over time, unlike the conventional method mentioned in [9].Specifically, the proposed method renders the rotation data with the intensity levels of pixels on the measurement area, which is specially invented for the HMD system, and it can obtain the motion-to-photon latency data in all temporal ranges.Therefore, it has a great advantage over the existing method.
This paper is organized as follows.Section 2 describes the proposed latency measurement system with five subsections.Section 3 presents the experimental environment and the results using the proposed system.Section 4 presents the paper's conclusion.

Proposed Method
Figure 3 shows the overall block diagram of the proposed system.The proposed method measured the actual angular change of the encoder when physical movement occurred.Concurrently, when the physical movement was measured by the IMU sensor, the rendered image was outputted to the display reflecting the measured angle, and the corresponding angle value was converted into the intensity image.The proposed method compared these two values.Specifically, a measurement area, which was constructed with multiple 2D objects mapped with an intensity level converting the rotation angle measured from the IMU, was implemented.Then, the photodetector converted the luminance reflected in the measurement area into a voltage.Finally, the oscilloscope measured this voltage value to determine the intensity level.At the same time, the pulse obtained from the encoder was converted into a position value and compared with the luminance-based position value obtained from the photodetector in real time.Finally, the motion-to-photon latency was calculated by measuring the time difference at which the angle measured from the display reached the same rotation angle of the encoder, as shown in Figure 3.The detailed processes are explained in following sections.
Electronics 2018, 7, x FOR PEER REVIEW 3 of 13 In this paper, the authors propose a novel time sequential measurement system for the motionto-photon latency by comparing the physical motion and the motion reflected on the image by 3D rendering.Therefore, it is possible to measure and record the time sequential latency more precisely in real-time, while reflecting the change in workload over time, unlike the conventional method mentioned in [9].Specifically, the proposed method renders the rotation data with the intensity levels of pixels on the measurement area, which is specially invented for the HMD system, and it can obtain the motion-to-photon latency data in all temporal ranges.Therefore, it has a great advantage over the existing method.
This paper is organized as follows.Section 2 describes the proposed latency measurement system with five subsections.Section 3 presents the experimental environment and the results using the proposed system.Section 4 presents the paper's conclusion.

Proposed Method
Figure 3 shows the overall block diagram of the proposed system.The proposed method measured the actual angular change of the encoder when physical movement occurred.Concurrently, when the physical movement was measured by the IMU sensor, the rendered image was outputted to the display reflecting the measured angle, and the corresponding angle value was converted into the intensity image.The proposed method compared these two values.Specifically, a measurement area, which was constructed with multiple 2D objects mapped with an intensity level converting the rotation angle measured from the IMU, was implemented.Then, the photodetector converted the luminance reflected in the measurement area into a voltage.Finally, the oscilloscope measured this voltage value to determine the intensity level.At the same time, the pulse obtained from the encoder was converted into a position value and compared with the luminance-based position value obtained from the photodetector in real time.Finally, the motion-to-photon latency was calculated by measuring the time difference at which the angle measured from the display reached the same rotation angle of the encoder, as shown in Figure 3.The detailed processes are explained in following sections.

Motion Generator-Based on Human Neck Movement
The authors used the rotary platform proposed in [9] to model the physical rotation of the user's neck.Specifically, in order to simulate a human neck, a two-degree-of-freedom rotary platform was

Motion Generator-Based on Human Neck Movement
The authors used the rotary platform proposed in [9] to model the physical rotation of the user's neck.Specifically, in order to simulate a human neck, a two-degree-of-freedom rotary platform was proposed, built with joints and links based on head kinematics.Therefore, the motion at the end-effector could be estimated through the motion in each joint based on forward kinematics [10,11], as shown in Figure 4. proposed, built with joints and links based on head kinematics.Therefore, the motion at the endeffector could be estimated through the motion in each joint based on forward kinematics [10,11], as shown in Figure 4.For this paper, the authors implemented the kinematic analysis of the model using the Denavit-Hartenberg parameters [12], and the angle of the end-effector was estimated by calculating the angle of each joint [13].In the proposed method, the rotation of the end-effector was estimated by performing rotation transformation on the displacement and rotation in the previous link, and coordinate transformation with the translation matrix.The transformation was carried out on number of links with the following matrix multiplication: where denotes the transformation matrix of the end-effector, denotes a transformation matrix at the -th joint, and denotes a transformation matrix of at the -th link.The transformation matrix of the -th joint is as follows: where denotes the rotation angle between the previous joint and the next -th joint along the z axis, and denotes the displacement between each joint.The transformation matrix of the -th link is as follows: where , denotes an angle between each link along the x-axis, and , denotes the displacement between each link.Finally, the transformation matrix for the forward kinematics from the 1 -th link to the -th link is as follows: = cos sin cos sin sin cos cos cos sin sin 0 s i n cos 0 0 0 1 .
Therefore, the Euler angles [14] of the end-effector were calculated from the inner rotation matrix.The Euler angles were used in an interface of the proposed system.

Measurement Area Design
In the proposed method, photodetectors measured the rotation angles from the rendered image.The advantage of photodetectors is that they can be measured more accurately while modeling the human visual function owing to their fast response time.However, the photodetectors only measure For this paper, the authors implemented the kinematic analysis of the model using the Denavit-Hartenberg parameters [12], and the angle of the end-effector was estimated by calculating the angle of each joint [13].In the proposed method, the rotation of the end-effector was estimated by performing rotation transformation on the displacement and rotation in the previous link, and coordinate transformation with the translation matrix.The transformation was carried out on n number of links with the following matrix multiplication: where [T] denotes the transformation matrix of the end-effector, [Z n ] denotes a transformation matrix at the n-th joint, and [X n ] denotes a transformation matrix of at the n-th link.The transformation matrix of the i-th joint is as follows: where θ i denotes the rotation angle between the previous joint and the next i-th joint along the z axis, and d i denotes the displacement between each joint.The transformation matrix of the i-th link is as follows: where α i,i+1 denotes an angle between each link along the x-axis, and r i,i+1 denotes the displacement between each link.Finally, the transformation matrix for the forward kinematics from the (n − 1)-th link to the n-th link is as follows: Therefore, the Euler angles [14] of the end-effector were calculated from the inner rotation matrix.The Euler angles were used in an interface of the proposed system.

Measurement Area Design
In the proposed method, photodetectors measured the rotation angles from the rendered image.The advantage of photodetectors is that they can be measured more accurately while modeling the human visual function owing to their fast response time.However, the photodetectors only measure the intensity of light within the specific range.Hence, the authors proposed specially invented objects that displayed the current rotation angle on a VR scene.Specifically, the objects did not have any colors but had grayscale intensities, and the intensity levels represented the rotation angle with the pre-defined conversion rule that converted the rotation angle with a float type into the intensity value with an unsigned integer type.Therefore, by measuring the luminance for the objects in the measurement area, the current rotation angle was obtained.
When designing the objects in the measurement area, the shapes of the object could influence not only the rendering performance but also the measurement performance.In 3D rendering, there is back-face culling in that the game engine does not calculate the region occluded by an object [15].Therefore, due to occlusion in the measurement area, the rendering performance is changed.Since the back-face culling affects the latency measurement, the size of the measurement area to be projected into the display is optimally adjusted to minimize the back-face culling and to maximize the voltage that the active layer of the photodetector can measure.
The brightness of the measurement area was determined by the rotation value calculated from the IMU sensor of the HMD device when rendering the current frame.To enhance the measurement accuracy, two objects represented the current rotation angle at a single rotation axis.Arrangement of the four objects enabled both the yaw and pitch, which were the target rotation directions, to be calculated.
The scanline of the display for the area to be measured should also be considered.Since the entire image is not drawn at once but sequentially drawn along the scanline, there is a time difference depending on the arrangement of the measurement area [16].In order to estimate the correct angle, two measurement areas must be synchronized considering the physical location.

Voltage Mapping Table
When a change in light is detected in the photodetector installed in the measurement area, the current (the output voltage as a result) emitted by the internal photodiode changes.If the experimental environment and parameters are controlled constantly, the output voltage of the photodetector to the image luminance is constant.This means that the output voltage can be converted back to the original luminance information.Therefore, if the conversion between the luminance and the voltage is known in advance, the rotation angle applied in the rendering process can be estimated.In the proposed method, the relationship between the voltage and luminance was acquired in advance, and the mapping table for the relationship was obtained by using these data.Figure 5 shows how to estimate intensity levels from the luminance of the measurement area on the display using the mapping table.
converted back to the original luminance information.Therefore, if the conversion between the luminance and the voltage is known in advance, the rotation angle applied in the rendering process can be estimated.In the proposed method, the relationship between the voltage and luminance was acquired in advance, and the mapping table for the relationship was obtained by using these data.Figure 5 shows how to estimate intensity levels from the luminance of the measurement area on the display using the mapping table.The voltage of the photodetector is proportional to the luminance, and therefore, the low luminance has a low output voltage.In this case, the amount of the voltage according to the brightness was not distinguishable from the noise when the voltage outputted from the measurement region was low at the stage of constructing the mapping table.Figure 6 illustrates the above problem.Therefore, low voltage levels are excluded from the measurement, and the mapping table is estimated from the upper level.Then, the current rotation value was inferred.There is a problem that if only the 128 levels of intensity are used for 256 gray levels with an 8-bit depth, the measurement resolution is halved.To compensate for this problem, an additional photodetector was used.An additional sensor restored the resolution to the original resolution by using double photodetectors to measure one rotation of the axis.The voltage of the photodetector is proportional to the luminance, and therefore, the low luminance has a low output voltage.In this case, the amount of the voltage according to the brightness was not distinguishable from the noise when the voltage outputted from the measurement region was low at the stage of constructing the mapping table.Figure 6 illustrates the above problem.Therefore, low voltage levels are excluded from the measurement, and the mapping table is estimated from the upper level.Then, the current rotation value was inferred.There is a problem that if only the 128 levels of intensity are used for 256 gray levels with an 8-bit depth, the measurement resolution is halved.To compensate for this problem, an additional photodetector was used.An additional sensor restored the resolution to the original resolution by using double photodetectors to measure one rotation of the axis.For double photodetectors, a new formula was needed which converted a rotation value into two intensity levels.The rotation angle was split into two intensity levels, and these intensity levels were reflected on the measurement area so that the photodetectors measured each intensity level to convert them back to the rotation angle.The specific conversion formula is as follows: where denotes the maximum Euler angle in the measurement system, denotes the current angle measured from HMD, and and denote the upper and lower 7 bits of the converted value, respectively.In the measurement area, 14-bit data were mapped onto the objects.
For example, as shown in Figure 7, the current rotation of the yaw axis and maximum Euler Luminance Voltage(V) Measurable gap immeasurable gap on measurement For double photodetectors, a new formula was needed which converted a rotation value into two intensity levels.The rotation angle was split into two intensity levels, and these intensity levels were reflected on the measurement area so that the photodetectors measured each intensity level to convert them back to the rotation angle.The specific conversion formula is as follows: where M euler denotes the maximum Euler angle in the measurement system, θ denotes the current angle measured from HMD, and MSBs 7bit and LSBs 7bit denote the upper and lower 7 bits of the converted value, respectively.In the measurement area, 14-bit data were mapped onto the objects.For example, as shown in Figure 7, the current rotation of the yaw axis and maximum Euler angle are given.Then, these values were calculated with a conversion formula.The current rotation angle was converted into binary data and split into two parts.Each part was the intensity level of the measurement area.After estimating these intensity levels by the photodetectors, the reverse calculation was performed with the inverse of the conversion formula.Finally, the current rotation was obtained, which is used for the generation of a VR scene.where denotes the number of points, denotes an output voltage from the photodetector, denotes an 8-bit unsigned integer value of the intensity level projected on the measurement area, denotes a curvature parameter, and denotes a random error.The polynomial curve is composed as follows.
In (7), if the random error term is excluded, the curvature parameters can be estimated after the least-squares estimation [17].The least-squares estimation of these parameters is as follows: where denotes a vector composed of the output voltages from the photodetector, denotes the curvature parameter vector of the voltage-luminance curve, and denotes the matrix of intensity levels from the repeated measurements.Using this regression model, the motion-to-photon latency was measured and is described in the following section.

Measurement of Motion-to-Photon Latency
The output image on the display was rendered by rotating the HMD using a motion generator designed to control the rotation of the HMD device.It controlled the HMD device with the desired rotation angle using a highly accurate DC servo motor.An internal encoder in the motor, which was used to control the motor, could also be used for measuring the current physical movement.However, in order to more precisely measure the physical movement over time, the incremental type encoders, additionally mounted on the axes, detected the rotation angles and outputted them as pulses.Concurrently, by measuring the luminance of the objects, the current angle data could be obtained by the conversion rule with the mapping table described above.The motion data about the current frame outputted from the display and the motion data outputted from the motion generator were compared.Due to the motion-to-photon latency, the rotation angle from the display was temporally lagged with respect to the angle outputted from the motion generator.To quantify the motion-to-photon latency, the proposed system calculated the time at which the angle, outputted However, in some cases, accurate mapping tables could not be formed because of changes in the external environment during the measurement process and changes in the output voltage due to the electrical noise.To solve this problem, a polynomial regression to store the data was performed from the voltage data obtained by repeated experiments.In the proposed method, an optimal curve was obtained by repeatedly performing a polynomial regression of less than third-order to avoid overfitting.Then, the obtained curve was used for the voltage estimation for a luminance level.
The third polynomial regression used in this paper is described below.The regression model is as follows: where n denotes the number of points, v i denotes an output voltage from the photodetector, I i denotes an 8-bit unsigned integer value of the intensity level projected on the measurement area, a i denotes a curvature parameter, and ε i denotes a random error.The polynomial curve is composed as follows.
In (7), if the random error term is excluded, the curvature parameters can be estimated after the least-squares estimation [17].The least-squares estimation of these parameters is as follows: where v denotes a vector composed of the output voltages from the photodetector, a denotes the curvature parameter vector of the voltage-luminance curve, and I denotes the matrix of intensity levels from the repeated measurements.Using this regression model, the motion-to-photon latency was measured and is described in the following section.

Measurement of Motion-to-Photon Latency
The output image on the display was rendered by rotating the HMD using a motion generator designed to control the rotation of the HMD device.It controlled the HMD device with the desired rotation angle using a highly accurate DC servo motor.An internal encoder in the motor, which was used to control the motor, could also be used for measuring the current physical movement.However, in order to more precisely measure the physical movement over time, the incremental type encoders, additionally mounted on the axes, detected the rotation angles and outputted them as pulses.Concurrently, by measuring the luminance of the objects, the current angle data could be obtained by the conversion rule with the mapping table described above.The motion data about the current frame outputted from the display and the motion data outputted from the motion generator were compared.Due to the motion-to-photon latency, the rotation angle from the display was temporally lagged with respect to the angle outputted from the motion generator.To quantify the motion-to-photon latency, the proposed system calculated the time at which the angle, outputted from the display, matched the angle outputted from encoders of the motion generator.Finally, it calculated the difference, which was the motion-to-photon latency between the two time points that had the same rotation angle.When calculating the time points, the authors found the indices of two rotation buffers with the same rotation angles, as shown in Figure 8.To avoid finding multiple points of the same value, the searching range was limited.There were multiple points that had the same value in all ranges.By limiting the searching range, the method allowed the buffers to be searched in the limited range so that only one same value was selected.value in all ranges.By limiting the searching range, the method allowed the buffers to be searched in the limited range so that only one same value was selected.

Real-Time Measurement and Interface
The proposed method used buffers to store the data from the encoders and the display.Specifically, it synchronized and stored all data in an oscilloscope, and the time delays were compared by searching points that had the same values.Once each time difference was found, it was converted to a time value to obtain the final motion-to-photon latency.In addition, a user interface was proposed to control the system and to integrate and run several modules.Figure 9 shows the configuration of the interface.The sequence-based motion generation part provided the information for controlling the motion generator.In the data plotting part, the graphs for the rotation angles are plotted.The latency measurement part provides the current, mean, and max values for the motionto-photon latency.

Real-Time Measurement and Interface
The proposed method used buffers to store the data from the encoders and the display.Specifically, it synchronized and stored all data in an oscilloscope, and the time delays were compared by searching points that had the same values.Once each time difference was found, it was converted to a time value to obtain the final motion-to-photon latency.In addition, a user interface was proposed to control the system and to integrate and run several modules.Figure 9 shows the configuration of the interface.The sequence-based motion generation part provided the information for controlling the motion generator.In the data plotting part, the graphs for the rotation angles are plotted.The latency measurement part provides the current, mean, and max values for the motion-to-photon latency.
The proposed method used buffers to store the data from the encoders and the display.Specifically, it synchronized and stored all data in an oscilloscope, and the time delays were compared by searching points that had the same values.Once each time difference was found, it was converted to a time value to obtain the final motion-to-photon latency.In addition, a user interface was proposed to control the system and to integrate and run several modules.Figure 9 shows the configuration of the interface.The sequence-based motion generation part provided the information for controlling the motion generator.In the data plotting part, the graphs for the rotation angles are plotted.The latency measurement part provides the current, mean, and max values for the motionto-photon latency.

Experimental Results
The experimental environment is as follows.First, the authors used an Oculus Rift DK 2 HMD device (Oculus VR, Menlo park, CA, USA) for the VR implementation [18].To drive a motion generator, two RE 40 (Maxon Motor, Sachseln, Switzerland) [19] were used as a DC motor along two axes, and the same number of EPOS2 50/5 (Maxon Motor, Sachseln, Switzerland) were used as controllers to control them [20].However, since the position estimated from the motor itself had mechanical latency, additional encoders (EIL 580 Baumer, Frauenfeld, Switzerland), which are of the incremental type with 500 steps/turn and 300 kHz frequency, were used to measure more accurate current rotation angles [21].SM05PDs (THORLABS, Newton, NJ, USA) were used as the photodetectors, which have a spectral range from 200 nm to 1000 nm [22].A PicoScope 4824 (Pico Technology, Saint Neots, UK) was used to measure the voltages from encoders and photodetectors [23].It measured the voltages of the encoder and photodetector with a sampling rate of 100,000 times per second and a bandwidth of 5.0 Gbps, thereby allowing nearly continuous data description and processing in real time.To render the VR scenes and drive the entire proposed system, a PC, which had an Intel i7-6700k @ 4.4 GHz CPU [24] and an NVIDIA Geforce GTX 1080 graphic card [25], were used.The experiment was conducted by the following two methods.First, the authors measured the changes in the latency according to the change in the position sequence while the initial workload was fixed.In this case, the workload was assigned the minimum level.The second method used only one of the motion sequences for different initial workload levels.Figure 10 shows the overall experimental process described above.
had an Intel i7-6700k @ 4.4 GHz CPU [24] and an NVIDIA Geforce GTX 1080 graphic card [25], were used.The experiment was conducted by the following two methods.First, the authors measured the changes in the latency according to the change in the position sequence while the initial workload was fixed.In this case, the workload was assigned the minimum level.The second method used only one of the motion sequences for different initial workload levels.Figure 10 shows the overall experimental process described above.

Position Sequence
Table 1 lists the position sequences defined in the experiment for the yaw and pitch axes.Six peak points were inputted to the motion generator controlled by the input angle, and the system was controlled according to the sequence shown in Figure 11.In this case, as shown in Table 2, the average latency was up to 46.55 ms and the maximum latency was up to 63.72 ms.In this experiment, the positions and shapes of the 3D objects in a VR scene were changed according to the orientation of the HMD.By reflecting these changes of the objects, the rendering workload was changed with time, thereby changing the motion-to-photon latency.As shown in Table 2, the motion-to-photon latency was varied according to the motion sequence.These results show that the proposed system could measure the motion-to-photon latency in all temporal ranges when the position sequence changed.

Position Sequence
Table 1 lists the position sequences defined in the experiment for the yaw and pitch axes.Six peak points were inputted to the motion generator controlled by the input angle, and the system was controlled according to the sequence shown in Figure 11.In this case, as shown in Table 2, the average latency was up to 46.55 ms and the maximum latency was up to 63.72 ms.In this experiment, the positions and shapes of the 3D objects in a VR scene were changed according to the orientation of the HMD.By reflecting these changes of the objects, the rendering workload was changed with time, thereby changing the motion-to-photon latency.As shown in Table 2, the motion-to-photon latency was varied according to the motion sequence.These results show that the proposed system could measure the motion-to-photon latency in all temporal ranges when the position sequence changed.

Figure 1 .
Figure 1.Image generation process in the HMD (head-mounted display) system.

Figure 2 .
Figure 2. Overall architecture of the previously proposed motion-to-photon latency measurement system.

Figure 1 .
Figure 1.Image generation process in the HMD (head-mounted display) system.

Figure 1 .
Figure 1.Image generation process in the HMD (head-mounted display) system.

Figure 2 .
Figure 2. Overall architecture of the previously proposed motion-to-photon latency measurement system.Figure 2. Overall architecture of the previously proposed motion-to-photon latency measurement system.

Figure 2 .
Figure 2. Overall architecture of the previously proposed motion-to-photon latency measurement system.Figure 2. Overall architecture of the previously proposed motion-to-photon latency measurement system.

Figure 3 .
Figure 3. Overall block diagram of the proposed system.

Figure 4 .
Figure 4. Kinematics of the head-model-based measurement platform.

Figure 4 .
Figure 4. Kinematics of the head-model-based measurement platform.

Figure 5 .
Figure 5. Luminance estimation from an object rendered by using the photodetector system and the voltage mapping table.Figure 5. Luminance estimation from an object rendered by using the photodetector system and the voltage mapping table.

Figure 5 .
Figure 5. Luminance estimation from an object rendered by using the photodetector system and the voltage mapping table.Figure 5. Luminance estimation from an object rendered by using the photodetector system and the voltage mapping table.

Figure 6 .
Figure 6.Measurement error and noise for an output voltage in the photodetector system.

Figure 6 .
Figure 6.Measurement error and noise for an output voltage in the photodetector system.

Electronics 2018, 7 , 13 Figure 7 .
Figure 7. Example of the intensity level conversion and its inverse for measuring the current rotation angle.

Figure 7 .
Figure 7. Example of the intensity level conversion and its inverse for measuring the current rotation angle.

Figure 8 .
Figure 8. Example of the motion-to-photon latency calculation.

Figure 8 .
Figure 8. Example of the motion-to-photon latency calculation.

Figure 9 .
Figure 9. Proposed user interface of the motion-to-photon latency measurement system.Figure 9. Proposed user interface of the motion-to-photon latency measurement system.

Figure 9 .
Figure 9. Proposed user interface of the motion-to-photon latency measurement system.Figure 9. Proposed user interface of the motion-to-photon latency measurement system.

Figure 10 .
Figure 10.Overall experimental processes for calculating the motion-to-photon latency in (a) different sequences and (b) different workload levels.

Figure 10 .
Figure 10.Overall experimental processes for calculating the motion-to-photon latency in (a) different sequences and (b) different workload levels.

Table 1 .
Peak points of simulated head orientations in yaw and pitch directions.

Table 2 .
Minimum, maximum, and average motion-to-photon latencies by different position sequence.