Automatic, Qualitative Scoring of the Interlocking Pentagon Drawing Test (PDT) Based on U-Net and Mobile Sensor Data

We implemented a mobile phone application of the pentagon drawing test (PDT), called mPDT, with a novel, automatic, and qualitative scoring method for the application based on U-Net (a convolutional network for biomedical image segmentation) coupled with mobile sensor data obtained with the mPDT. For the scoring protocol, the U-Net was trained with 199 PDT hand-drawn images of 512 × 512 resolution obtained via the mPDT in order to generate a trained model, Deep5, for segmenting a drawn right or left pentagon. The U-Net was also trained with 199 images of 512 × 512 resolution to attain the trained model, DeepLock, for segmenting an interlocking figure. Here, the epochs were iterated until the accuracy was greater than 98% and saturated. The mobile senor data primarily consisted of x and y coordinates, timestamps, and touch-events of all the samples with a 20 ms sampling period. The velocities were then calculated using the primary sensor data. With Deep5, DeepLock, and the sensor data, four parameters were extracted. These included the number of angles (0–4 points), distance/intersection between the two drawn figures (0–4 points), closure/opening of the drawn figure contours (0–2 points), and tremors detected (0–1 points). The parameters gave a scaling of 11 points in total. The performance evaluation for the mPDT included 230 images from subjects and their associated sensor data. The results of the performance test indicated, respectively, a sensitivity, specificity, accuracy, and precision of 97.53%, 92.62%, 94.35%, and 87.78% for the number of angles parameter; 93.10%, 97.90%, 96.09%, and 96.43% for the distance/intersection parameter; 94.03%, 90.63%, 92.61%, and 93.33% for the closure/opening parameter; and 100.00%, 100.00%, 100.00%, and 100.00% for the detected tremor parameter. These results suggest that the mPDT is very robust in differentiating dementia disease subtypes and is able to contribute to clinical practice and field studies.


Introduction
The pentagon drawing test (PDT) is a sub-test of the Mini-Mental State Examination (MMSE), used extensively in clinical and research settings as a measure of cognitive impairment [1]. The MMSE is a general screening tool for cognitive impairment. However it shows low sensitivity for detecting cognitive impairment in Parkinson's disease [2]. The mobile application, mPDT, for the interlocking pentagon test was developed using the Android Studio development environment. While the source code of mPDT was implemented to be able to be built in any mobile device, including smartphones, tablets, or notebooks, we built and tested the mPDT in a Samsung Galaxy Note 4 smartphone with a resolution of 640 dots per inch and a spatial accuracy of 0.004 cm. The mPDT allows for a user to copy two interlocking pentagons (with the interlocking shape being a rhombus) on the screen, and scores the drawing image qualitatively based on the sensor data of the drawing image and the pre-trained models, Deep5 and DeepLock, developed in this study. Using U-Net, a convolutional network architecture for fast and precise segmentation of images, the pre-trained models Deep5 and DeepLock were generated for segmenting the pentagon shapes and the interlocking shape, respectively.
The sensor data collected by the touch sensors embedded in a smartphone consisted of timestamps in seconds, the x and y coordinates in pixels, and touch-events of the samples of the drawing image with a 50 Hz sampling frequency. Figure 1a shows the flow diagram of the mPDT operation in, and Figure 1b-d shows the screen shots of the registration window, the PDT window, and the result window for mPDT, respectively. At the launch of mPDT, an informed consent prompt appears, following this, a registration window is displayed, where it is possible to enter the subject's information, such as name, age, gender, and handedness, plus the optional parameters.

Pre-Trained Models, Deep5 and DeepLock based on the U-Net
Novel pre-trained models of Deep5 and DeepLock were developed for segmentation of the drawn pentagon portions and the interlocking domains of the images, respectively. Deep5 and DeepLock were created based on the U-Net convolutional network architecture in keras [32]. The Following the user pressing the start button in the registration window, the PDT window then appears, in which the user is asked to copy two interlocking pentagons provided on a paper by an examiner or draw them while recalling from an image provided on a previous window. Values of measured parameters for the time in seconds and the x and y coordinates in pixels are obtained from the drawing image with a 50 Hz sampling frequency. The values are saved as sensor data when the subject copies or draws pentagons on the touch screen of the PDT window. The results window then displays the sensor data along with the drawn image, and/or a plot of speeds in mm/sec of inter-samples over time. The results could then be sent to the email address entered at the registration window.

Pre-Trained Models, Deep5 and DeepLock based on the U-Net
Novel pre-trained models of Deep5 and DeepLock were developed for segmentation of the drawn pentagon portions and the interlocking domains of the images, respectively. Deep5 and DeepLock were created based on the U-Net convolutional network architecture in keras [32]. The network architecture implemented in this study is illustrated in Figure 2. It consists of a contracting path, an expansive path, and a final layer. The contracting path consists of repeated applications of two 3 × 3 convolutions and a 2 × 2 max pooling operation with stride 2 for down-sampling. At each repetition, the number of feature channels is doubled. The expansive path consists of two 3 × 3 convolutions and a 2 × 2 convolution ("up-convolution") for up-sampling to recover the size of the segmentation map. At the final layer, a 1 × 1 convolution was used to map each 16-component feature vector to the desired number of classes. In total, the network has 23 convolutional layers. The training data for both Deep5 and DeepLock contain 960 images of 128 × 128 resolution, which were augmented using a module called ImageDataGenerator in keras.preprocessing.image and resized from the original 199 images of 1600 × 1320 resolution. Deep5 and DeepLock were generated by training the network architecture for five and seven epochs with accuracies of approximately 0.977 and 0.979, respectively. The loss function used for the training was essentially binary cross entropy.

Pre-Trained Models, Deep5 and DeepLock based on the U-Net
Novel pre-trained models of Deep5 and DeepLock were developed for segmentation of the drawn pentagon portions and the interlocking domains of the images, respectively. Deep5 and DeepLock were created based on the U-Net convolutional network architecture in keras [32]. The network architecture implemented in this study is illustrated in Figure 2. It consists of a contracting path, an expansive path, and a final layer. The contracting path consists of repeated applications of two 3 × 3 convolutions and a 2 × 2 max pooling operation with stride 2 for down-sampling. At each repetition, the number of feature channels is doubled. The expansive path consists of two 3 × 3 convolutions and a 2 × 2 convolution ("up-convolution") for up-sampling to recover the size of the segmentation map. At the final layer, a 1 × 1 convolution was used to map each 16-component feature vector to the desired number of classes. In total, the network has 23 convolutional layers. The training data for both Deep5 and DeepLock contain 960 images of 128 × 128 resolution, which were augmented using a module called ImageDataGenerator in keras.preprocessing.image and resized from the original 199 images of 1600 × 1320 resolution. Deep5 and DeepLock were generated by training the network architecture for five and seven epochs with accuracies of approximately 0.977 and 0.979, respectively. The loss function used for the training was essentially binary cross entropy.

Scoring Method of mPDT
The novel, automatic, and qualitative scoring method for the mPDT was developed based on the sensor data and the pre-trained models, Deep5 and DeepLock. Four parameters were included in the scoring method: the number of angles (0-4 points), distance/intersection between the two figures (0-4 points), closure/opening of the image contour (0-2 points), and detected tremors (0-1 points). All the assigned scores for the parameters are integers. A total score corresponding to the sum of individual scores of each parameter ranged from 0 to 11. The parameters, number of angles (0-4 points), distance/intersection between the two figures (0-4 points), and closure/opening of the image contour (0-2 points) were adopted from a previous study by Paolo Caffarra et al. [5]. When a subject executes more than one copy of the pentagons, the last copy is then scored. A detailed list of the parameters Sensors 2020, 20, 1283 6 of 24 used is presented in Table 2 and the overall flowchart and the schematic diagram of the scoring method are shown in Figure 3a,b, respectively.  The scoring method consists of a series of processes that include manipulation of the sensor data and segmentation of the drawn pentagon and the interlocking shapes using Deep5 and DeepLock, respectively. There is then the extraction of variables (the number of angles, distance/intersection, closure/opening, and presence of tremors) and the assignment of scores according to the performance scores for each parameter. The sensor data primary obtained from the drawn image samples during the mPDT interaction include timestamps, x-and y-coordinates, and touch events. The time index of moving from a figure drawing to another is detected from the primary sensor data in the process of  The scoring method consists of a series of processes that include manipulation of the sensor data and segmentation of the drawn pentagon and the interlocking shapes using Deep5 and DeepLock, respectively. There is then the extraction of variables (the number of angles, distance/intersection, closure/opening, and presence of tremors) and the assignment of scores according to the performance scores for each parameter. The sensor data primary obtained from the drawn image samples during the mPDT interaction include timestamps, x-and y-coordinates, and touch events. The time index of moving from a figure drawing to another is detected from the primary sensor data in the process of the manipulation of the sensor data. Velocity values are also obtained from the primary sensor data in the process of the manipulation of sensor data.
The drawn image is then analyzed by Deep5 and DeepLock to respectively segment the pentagon and the interlocking shapes. It is here that the percentages of the segmented pentagon shapes and the interlocking shape overlapping the drawn image are calculated. Next, the values for each parameter are calculated. For the total number of angles of each figure, the percentages and the number of peaks in velocities are determined. For the distance/intersection, the interlocking distance between two figures and the absolute ratio of the differences in x and y coordinates of the two points used in the calculation of the interlocking distance are extracted. After clustering the primary sensor data of zero velocities, the mean distance between the cluster center and the points belonging to each cluster are calculated for the closure/opening parameter. For presence of tremors, the frequency of consecutive 'up and down' touch events from the sensor data is estimated. Finally, there is the assignment of integer scores according to each parameter score, as in Table 2 (also see Appendix B for the examples of drawn images and the corresponding performance scores and assigned integer scores). The details are described in the following subsections.

Sensor Data Manipulation and Shape Segmentation Using Deep5 and Deeplock
Once the mPDT has been completed, three pieces of data, including the drawing image and the primary and secondary sensor data, are generated for output. The size of the drawing image I D is 128 × 128 pixels, which is resized from the original 1600 × 1320 drawing image of the PDT window. The primary sensor data consist of times t[n] in seconds, x-and y-coordinates, x[n] and y[n] in pixels, and touch-events e[n] of the sample points of the 128 × 128 drawing image, where the sampling rate was set at 50 Hz and n is the index of a sample point. The secondary sensor data are velocities v[n] in pixels/sec which are calculated from the primary sensor data. Each of the touch-events e[n] has one of the values, such as -1, 0, 1, where the assigned value of -1 is for the event 'down', as in touching on the screen; 1 for the event 'up', as in touching off the screen; and 0 for the event 'move', as in moving and touching on the screen. For the drawing image I D , it is supposed to be of two interlocking pentagons.
The sensor data for each of the two interlocking shapes is separately obtained using the times t[n f 1 ] of the last sample point of the first drawn figure and t[n f 2 ] of the first sample point of the second drawn figure. In other words, the sample points for t[n], n ≤ n f 1 belong to the first figure I D,n≤n f 1 and those for t[n], n ≥ n f 2 to the second figure I D,n≥n f 2 . The time t[n f 1 ] can be estimated by the touch-events shifting from the event 'move' into the event 'up' and then successively staying at the event 'up' for the longest time if such a period occurs more than once. Therefore, the index n f 1 can be determined by finding the longest chain of events consisting of a 0 (the event 'move') and consecutive 1s (the 'up' events) in the sequence of touch-events e[n]. The time t[n f 2 ] can be obtained by adding the longest period of time to the time t[n f 1 ]. Sensor data for the interlocking image I DL of the 128 × 128 drawing image I D is obtained from the sample points between two data points, (x[n 1 ], y[n 1 ]) and (x[n 2 ], y[n 2 ]), where n 1 is the index at which the value of x[n 1 ] is the maximum of those belonging to x[n] (n < n f 1 ) and n 2 is the index at which the value of x[n 2 ] is the minimum of those belonging to x[n] (n ≥ n f 2 ).
The two pentagon shapes I f i (i = 1, 2) are then separately segmented using indices n f i (i = 1, 2) and the Deep5 pre-trained model from the 128 × 128 drawing image I D . An interlocking shape I L is segmented using the DeepLock pre-trained model from the 128 × 128 drawing image I D as well. Next, percentage p f i (i = 1, 2) of the segmented image I f i (i = 1, 2) matching to the corresponding figure of the 128 × 128 drawing image I D is estimated as below: , and where n(I DL ∩ I L ) is the number of pixel coordinates that I L and I DL have in common; n(I DL ) is the total number of pixel coordinates in I DL .

Number of Angles
Number of angles is the sum of the number N Ai (i = 1, 2) of angles of each figure of the 128 × 128 drawing image I D , which is estimated using the percentage, p f i (i = 1, 2), velocity v[n], and the index n f i (i = 1, 2) as follows: where N pi (i = 1, 2) is the number of peaks in velocity v[n] for each figure. If the percentage p f i (i = 1, 2) of the segmented image I f i (i = 1, 2) matching to the corresponding figure of the 128 × 128 drawing image I D is larger than a given threshold p th1 , then the shape of the figure is identified as a pentagon and the number N Ai (i = 1, 2) of angles is estimated to be 5. If not, then the number N A1 and N A2 are estimated by the numbers N p1 and N p2 of peaks in velocity v[n] (n ≤ n f 1 ) and v[n] (n ≥ n f 2 ), respectively, since the velocity of the drawing stroke increases and then decreases to zero at the point of change in direction.

Distance/Intersection between Two Figures
The distance/intersection between two figures is evaluated by the percentage p L of the image I L matching to the interlocking image I DL of the 128 × 128 drawing image I D , the distance d L between two figures in the interlocking image, and the ratio ∆x/∆y of differences of two data points (x[n 1 ], y[n 1 ]) and (x[n 2 ], y[n 2 ]). The distance d L in cm between two figures in the interlocking image is estimated by the distance between two data points, ( ), where n 1 is the index at which the value of x[n 1 ] is the maximum of those belonging to x[n] (n ≤ n f 1 ) and n 2 is the index at which the value of x[n 2 ] is the minimum of those belonging to x[n] (n ≥ n f 2 ). The ratio ∆x/∆y is estimated by the ratio of the differences ∆x and ∆y, where ∆x and ∆y are the differences in cm in x-and y-axis directions, respectively, between two data points, ( ). The existence and the shape of the interlocking between two figures can be discriminated by the percentage p L of the segmented interlocking image I L matching to the interlocking image I DL of the 128 × 128 drawing image I D , the distance d L , and the ratio ∆x/∆y.
If the percentage p L is larger than a given threshold p th2 , the distance d L has a negative value and the absolute value of the ratio ∆x/∆y is larger than a given threshold r th , the two figures are then evaluated to be interlocked with a shape of a rhombus. If the percentage p L is larger than a threshold p th2 and the distance d L has a negative value but the absolute value of the ratio ∆x/∆y is less than the given threshold r th , then the two figures are evaluated to be interlocked without the shape of a rhombus. On the other hand, if the distance d L has a positive value regardless of the percentage p L and the absolute value of the ratio ∆x/∆y , then the two figures are evaluated as not to be interlocked and apart from each other by the distance d L . In such a case, if the value of the distance d L is nearly positively zero, being less than ε, a very small positive value, then the two figures are evaluated as to be attached to each other. The evaluation of the distance d L and the absolute ratio ∆x/∆y are formulated as follows: , and

Existence of Closure/Opening
Existence of opening in a figure is determined by the fact that an opening could exist in a region with a relatively larger variance of distances between sample points, having consecutive 0s of velocity in a figure, as there are changes in the stroke direction as well as in the stroke position within the region where an opening occurs. Parameters of the existence of openings N op1 and N op2 in the figure images of I D,n≤n f 1 and I D,n≥n f 2 are estimated by k-means clustering of the sample points (x[n], y[n]), n ≤ n f 1 and (x[n], y[n]), n ≥ n f 2 with zero velocity, respectively, where the target numbers for the parameters N op1 and N op2 are set to the numbers of angles N A1 and N A2 , respectively. Then, the cluster parameter k ij (i = 1, 2, j = 1, . . . N Ai ) is calculated by averaging the distances between the sample points belonging to the cluster δ ij and the cluster center δ c,ij as follows: The parameters of the existence of openings N op1 and N op2 in the figure images of I D,n≤n f 1 and I D,n≥n f 2 are then estimated as follows: where δ th is a given threshold.

Existence of Tremors
A tremor is an involuntary quivering movement or shake which could be caused by age-associated weakness, neurodegenerative diseases, or mental health conditions. A tremor can be detected from the frequency of a 'down' event happening after an 'up' event in the chain of touch-events, which cannot be observed in a paper and pencil test. The existence of tremors N tri (i = 1, 2) in each figure is determined by the total number of 'up' events followed by 'down' events being larger than a given threshold e th in the touch-events e[n] of the sample points in the corresponding figure. An 'up' event followed by a 'down' event in the touch-events e[n] can be detected when the multiplication of two neighboring values in touch-events e[n] is equal to -1 as the values of 'up' and 'down' events are set to 1 and -1, respectively. The number of tremors N tri (i = 1, 2) in each figure is evaluated as follows:   Table 3 lists the conditions for the assigned integer scores for each parameter in the mPDT. The score of the number of angles is via percentages, p f 1 and p f 2 , in addition to the numbers of angles, N A1 and N A2 . The score is a 4 if both of the percentages, p f 1 and p f 2 , are equal or greater than a given threshold p th1 ; the score is a 3 if at least one of the percentages, p f 1 or p f 2 , is less than p th1 as well as the sum number of angles, N A1 + N A2 , is 9 or 11; the score is a 2 if at least one of the percentages, p f 1 or p f 2 , is less than p th1 as well as the sum number of angles, N A1 + N A2 , is 8 or 12; the score is a 1 if at least one of the percentages, p f 1 or p f 2 , is less than p th1 as well as the sum number of angles, N A1 + N A2 , is between or including 5 and 7; and the score is a 0 if at least one of the percentages, p f 1 or p f 2 , is less than p th1 as well as the sum number of angles, N A1 + N A2 , is less than 5 or greater than 12.
The score for distance/intersection is obtained by using the percentage p L , the distance d L between the two figures in the interlocking image, and the absolute value of the ratio ∆x/∆y . The score is a 4 if the percentage p L is equal to or larger than a given threshold p th2 , the distance d L is less than −ε, and the absolute value of the ratio ∆x/∆y is equal to or larger than a given threshold r th ; the score is a 3 if the percentage p L is less than p th2 , the distance d L is less than −ε, and the absolute value of the ratio ∆x/∆y is less than a given threshold r th ; the score is a 2 if the percentage p L is less than p th2 as well as the absolute of the distance d L is equal to and less than ε; the score is a 1 if the percentage p L is less than p th2 as well as the distance d L is between ε and 1 cm exclusive; and the score is a 0 if the percentage p L is less than p th2 as well as the distance d L is equal to or greater than 1 cm.
The score of closure/opening is determined by the parameters N op1 and N op2 . The score is a 2 if there are no openings in both figures (N op1 N op2 = 1 and N op1 +N op2 = 2); the score is a 1 if there is an opening in one of the two figures (N op1 N op2 = 0 and N op1 +N op2 = 1); and the score is a 0 if there are openings in both figures ( N op1 N op2 = 0 and N op1 +N op2 = 0).
The score for tremors is determined by the values N tr1 and N tr2 . It is given a score of 1 if both N tr1 and N tr2 are equal to 1 (N tr1 N tr2 = 1); and 0 if any of N tr1 and N tr2 values is equal to 0 (N tr1 N tr2 = 0). Table 3. Details for the assignment of scores.

Parameters Assigned Integer Scores Conditions (Scoring Method)
Number of angles 4 Figure 4 depicts separate examples of original drawings, each with their own characteristic shapes. The analytical ability of the Deep5 pre-trained model with its segmented images for a detected pentagon and the generated velocity graphs with the detected peaks for shape analysis are then demonstrated.

Scoring of the Number of Angles
In Figure 4a where the original image is of two interlocking figures (left), both being pentagons, the segmented image (center) perceived by Deep5 has the estimated percentages p f 1 and p f 2 of 100.00% and 90.71%, respectively. The number of angles in each pentagon were both evaluated to be 5, as p f 1 and p f 2 were greater than the 0.75 score for p th1 , a threshold heuristically set by the two clinical experts in PDT scaling during the process of the ground truth scorings of all the images used in this study. For the velocity graph, the number of detected peaks corresponding to the number of angles in each figure was evaluated to be 5 (right). Figure 4b has the original drawing image of two interlocking figures, but only one is a pentagon (left figure). The segmented image by Deep5 has the estimated percentages p f 1 and p f 2 of 91.5% and 23.88%, respectively (center). The number of angles in the pentagon portion was evaluated to be 5, as the estimated percentage p f 1 was greater than 0.75; however, the number of angles in the non-pentagon portion was gauged to be 4, as the estimated percentage p f 2 was less than the 0.75 score and the number of peaks detected in the velocity graph was estimated to be 4 as well (right). Figure 4c similarly shows an example of an original drawing (left) of two interlocking figures with only one being a pentagon. The segmented image (middle) by Deep5 where the estimated percentages p f 1 and p f 2 were 29.10% and 98.52%, respectively. The number of angles in the non-pentagon portion was evaluated to be 4 from the estimated percentage score, p f 1 , being less than 0.75. The number of peaks in the non-pentagon portion of the figure from the velocity graph (right) was estimated to be 4 as well. On the other hand, the number of angles of the pentagon portion was evaluated to be 5, as the estimated percentage score, p f 2 , was greater than 0.75. Finally, the original drawing image (left) of Figure 4d depicts the example of two interlocking figures, with none of them being a pentagon. For the segmented image (middle) by Deep5, the estimated percentages, p f 1 and p f 2 were estimated to be 18.18 and 0.00%, respectively. In this case, the number of angles of both non-pentagons was evaluated to be 4, as both estimated percentages were less than 0.75 and the numbers of peaks detected in the velocity graph (right) were also estimated to be 4 for both non-pentagons. Figure 4 depicts separate examples of original drawings, each with their own characteristic shapes. The analytical ability of the Deep5 pre-trained model with its segmented images for a detected pentagon and the generated velocity graphs with the detected peaks for shape analysis are then demonstrated.  . The number of angles of the left-side figure was evaluated to be 5, as p f 1 was 91.5% and it was greater than 75%. For the right-side figure, the number of angles was given as 4, as the percentage p f 2 of 23.88% was less than 75% and the number of peaks detected in the velocity graph is 4. (c) The image of the right portion is a pentagon with the number of angles evaluated as 5 as p f 2 was 98.52% and it was greater than 75%. The image of the left portion was evaluated as a 4, as p f 1 was 29.10% and it was less than 75% and the number of peaks detected in the velocity graph is 4. (d) The image, composite of two figures, none of them being a pentagon. This is because the number of angles for each of the right and left-hand portions was evaluated to be a 4, since p f 1 and p f 2 were 18.18% and 0.00%, respectively, both being less than 75%. The numbers of angles in both the right and left portions were evaluated to be 4, matching the number of peaks detected in the velocity graphs for both.

Scoreing of Distance/Interlocking
The analytical ability of the pre-trained model DeepLock is demonstrated in Figure 5 with five separate examples of original drawing image along with the segmented image of interlocking generated by the program. Figure 5a has the original drawing image example (left) with the interlocking shape of a rhombus, its segmented image (middle) interlocking, and the overlap of the original drawing image along with the segmented image (right). In this case, the percentage p L and the distance d L were evaluated to be 97.67% and −2.27 cm, respectively. The absolute value ∆x/∆y of the application was estimated to be 4.49. As such, the two figures were evaluated to be interlocked with a shape of a rhombus from three parameters: (1) percentage score, p L , being greater than p th2 of 0.75, a heuristically given threshold; (2) the absolute value ∆x/∆y being larger than 1.12, a heuristically given threshold of r th , and (3) the distance d L of a negative value equal to and less than 0.01 cm, as the threshold value of ε. Here, the threshold values for p th2 and r th were chosen by the two clinical experts in PDT scaling during the process of the ground truth scorings of all the images used in this study. The threshold value ε was set to be 0.01 cm, considering a 2 pixel diagonal distance, 0.004 × 2 × sqrt(2), with a spatial resolution 0.004 cm of the device used in the implementation of the mPDT. The example in Figure 5b has the original drawing image of an interlocking shape that is not a rhombus (left) along with the resulting segmented image (middle) of the interlocking, and the overlap of the original drawing image and the segmented image (right). In this case, the percentage p L was evaluated to be 99.79% and the distance d L to be -2.36 cm. The absolute value ∆x/∆y for the plot was estimated to be 0.86. From these parameters, the two figures were evaluated to be interlocked without a shape of a rhombus, having the percentage p L of greater than 0.75 for p th2 and the distance d L of a negative value equal to and less than 0.01 cm for ε, but the absolute value ∆x/∆y being less than 1.12 for r th .
were evaluated to be interlocked without a shape of a rhombus, having the percentage L p of greater than 75 . 0 for 2 th p and the distance L d of a negative value equal to and less than 01 . 0 cm for ε , but the absolute value | / | y x Δ Δ being less than 1.12 for th r .
In the example for Figure 5c, the two figures are not intersecting, but are making contact. The percentage L p was estimated to be 0.00% and the distance L d was calculated to be 0.001 cm, less than 01 . 0 cm for ε . In Figure 5d, the original image depicts two component figures that are apart from each other; the percentage L p was estimated to be 0.00% and the distance L d was calculated as to be 0.24 cm, greater than ε of 0.01 cm. The example in Figure 5e has two figures that are also apart from each other, giving the percentage L p estimated as 0.00% and the distance L d calculated as 1.50 cm.  (c) The case of two figures with no interlocking but still touching each other. There, the percentage p L and the distance d L were evaluated to be 0.00% ( < 0.75%) and 0.001 cm (being in between −0.01 and 0.01 cm range), respectively. (d) The display case of two figures, drawn so that they are separated from each other, but are within 1.00 cm of each other. The percentage p L and the distance d L were evaluated to be 0.00% (< 0.75%) and 0.24 cm, respectively. (e) The example of two drawn figures that are more than 1.00 cm apart with the percentage p L and the distance d L were evaluated to be 0.00% (< 0.75%) and 1.50 cm, respectively.
In the example for Figure 5c, the two figures are not intersecting, but are making contact. The percentage p L was estimated to be 0.00% and the distance d L was calculated to be 0.001 cm, less than 0.01 cm for ε. In Figure 5d, the original image depicts two component figures that are apart from each other; the percentage p L was estimated to be 0.00% and the distance d L was calculated as to be 0.24 cm, greater than ε of 0.01 cm. The example in Figure 5e has two figures that are also apart from each other, giving the percentage p L estimated as 0.00% and the distance d L calculated as 1.50 cm.

Scoring of Closure/Opening
In three representative examples, Figure 6 demonstrates how the openings in original images are detected and assigned. Figure 6a

Scoring of Tremors
For the presence of tremors in the hand drawn images, Figure 7 represents two cases with one having no tremors and the other having tremors. For Figure 7a with no significant tremors being present in either of the two interlocking figures, the total numbers of the 'up' events followed by

Scoring of Tremors
For the presence of tremors in the hand drawn images, Figure 7 represents two cases with one having no tremors and the other having tremors. For Figure 7a with no significant tremors being present in either of the two interlocking figures, the total numbers of the 'up' events followed by 'down' events in touch-events e[n] of the sample points for the left and right portions of the interlocking figure were 0 and 1, respectively, and both with a given threshold e th of less than 5. The threshold value e th here was set by the two clinical experts in PDT scaling during the process of the ground truth scorings of all the images used in this study. In contrast, Figure 7b displays drawings with some tremors present in each of the two interlocking figures. In the figure, the 'up' events followed by 'down' events in the touch-events e[n] were 5 and 19 for the left and right figures, respectively, both being equal to or greater than 5, the given threshold e th . threshold value th e here was set by the two clinical experts in PDT scaling during the process of the ground truth scorings of all the images used in this study. In contrast, Figure 7(b) displays drawings with some tremors present in each of the two interlocking figures. In the figure, the 'up' events followed by 'down' events in the touch-events ] [n e were 5 and 19 for the left and right figures, respectively, both being equal to or greater than 5, the given threshold th e .

Performance Test Results
A total of 230 drawing images were used to test the performance of the scoring method with the mPDT. Table 4 summarizes the frequency of the ground truth for the 230 images with the score in each of the parameters. For the number of angles detected, the scores of 0 through 4 were 55, 32, 33, 30, and 80 events. For the distance/intersection parameter, the scores of 0 through 4 were for 33, 38, 34, 38, and 87 measures. Similarly, for the closure/opening measure, the scores of 0 through 2 were for 39, 61, and 130 detections, in the given order. The total numbers of instances for absence or presence of tremors, with a score of 0 or 1, were 16 and 214, respectively. Table 5 lists the performance of each scoring parameter in mPDT. For the angle number parameter, the sensitivity, specificity, accuracy, and precision values were 97.53%, 92.62%, 94.35%, and 87.78%; for distance/intersection, they were 93.10%, 97.90%, 96.09%, and 96.43%; for closure/opening, they were 94.03%, 90.63%, 92.61%, and 93.33%; and for tremor reads, they were 100.00%, 100.00%, 100.00%, and 100.00%, respectively.

Performance Test Results
A total of 230 drawing images were used to test the performance of the scoring method with the mPDT. Table 4 summarizes the frequency of the ground truth for the 230 images with the score in each of the parameters. For the number of angles detected, the scores of 0 through 4 were 55, 32, 33, 30, and 80 events. For the distance/intersection parameter, the scores of 0 through 4 were for 33, 38, 34, 38, and 87 measures. Similarly, for the closure/opening measure, the scores of 0 through 2 were for 39, 61, and 130 detections, in the given order. The total numbers of instances for absence or presence of tremors, with a score of 0 or 1, were 16 and 214, respectively.  Table 5 lists the performance of each scoring parameter in mPDT. For the angle number parameter, the sensitivity, specificity, accuracy, and precision values were 97.53%, 92.62%, 94.35%, and 87.78%; for distance/intersection, they were 93.10%, 97.90%, 96.09%, and 96.43%; for closure/opening, they were 94.03%, 90.63%, 92.61%, and 93.33%; and for tremor reads, they were 100.00%, 100.00%, 100.00%, and 100.00%, respectively.

Discussion
Conventional PDT based on a paper and pencil test is not readily suitable for evaluation of the dynamic components of cognitive function, as there are limitations in the real-time tracking of the orders, the stroke patterns, the speed variations, and so on, while the subjects are copying or drawing from recall. When subjects participate in a conventional PDT, many fMRI studies have shown multiple brain areas becoming active in the subject, including the bilateral parietal lobe, sensorimotor cortex, cerebellum, thalamus, premotor area, and inferior temporal sulcus [33][34][35][36]. However, it is not exactly clear what components of the cognitive function are associated with the activation of these areas as the conventional PDT is difficult to quantify objectively. To address this issue, our study focused on implementation of the PDT as a mobile phone application, namely mPDT, with a novel, automatic, and qualitative scoring method based on U-Net, a convolutional network for biomedical image segmentation of sensor data. The sensor data is also obtained by the mPDT.
The performance test proved that the scoring protocol suggested by the mPDT is reasonable and practical when compared with those of the traditional PDT. Further, the mPDT was shown to be capable of evaluation of the dynamic components of cognitive function. In our study, the subjects used a smartpen provided for the smartphone when copying figures in order to create an environment similar to the conventional paper and pencil test of the PDT. This also increased the accuracy and avoided undesirable noise in the activated brain function assay. The performance test was restricted to right-handed subjects to avoid a bias in statistical analysis and also due to relatively small number of left-handed subjects available. However, the mPDT scoring worked quite in the same way when two left-handed subjects (a 27 year old male and a 26 year old female) were initially included in the younger volunteer group, both samples showing proper results and accuracies (see Appendix C).
The conventional PDT is a sub-item of MMSE, which is usually used in assessing Alzheimer's disease [25]. However, the mPDT was developed to be applicable in better detection of cognitive impairment in Parkinson's disease. For this reason, the tremor parameter was included in the scoring of the mPDT, instead of the closing-in parameter suggested in previous qualitative scoring of the pentagon test [5], and as the closing-in sign is a characteristic sign in all dementia, but not for Parkinson's disease [37]. A tremor is an involuntary quivering movement or shake which may be due to age-associated weakness, a neurodegenerative disease, or a mental health condition. Therefore, there are several types of tremors recognized, such as essential tremors, Parkinsonian tremors, dystonic tremors, cerebellum tremors, psychogenic tremors, orthostatic tremors, and physiologic tremors [38]. In this study, the tremor symptom could be detected by the frequency of the occurrence of a 'down' event after an 'up' event in the touch-event series. A future study using mPDT for correlations between the pattern of tremoring and the underlying disease or condition could make a case for an early and differential diagnosis of a given neurodegenerative disease, such as Parkinson's.
Conventional screening tools, including MMSE, do not detect early cognitive impairment in Parkinson's disease, while the PDT is known to detect cognitive impairment earlier in Parkinson's than in Alzheimer's disease [39]. Using this fact, we aimed to develop a more sensitive screening tool to detect cognitive impairment in Parkinson's. In addition, as the smartphone could evaluate the motor-related indicators such as the speed at which the test was performed or the number of pauses, we could also measure the effects on motor ability as a cognitive measurement tool that could not be detected by the conventional PDT pencil and paper test. Therefore, our study indicates that the developed mPDT tool is specifically applicable in increasing the accuracy of cognitive function assessment in Parkinson's disease.

Conclusions
Even though the qualitative scorings of the PDT have been essential in establishing it as a prognostic marker in the assessment of visuospatial functions and in the differentiation of various neuronal degenerative diseases including Parkinson's, the evaluation is done manually, which is not subjective, is prone to human error, and it is not able to provide parametric and dynamic information on a specific neuronal degenerative disease. In this study, we developed a smartphone application, named mPDT, with an automatic scoring method based on mobile sensors and image segmentation using the U-Net deep learning algorithm. A tremor read, not in the standard PDT, was also included, allowing for the detection of early Parkinson's along with the other parameters tested.
The mPDT is also relatively environment independent as it is applicable for different types of mobile devices, such as smartphones, tablets, and notebooks. It is also relatively fast. The execution time was 0.73 ± 0.21 seconds (mean ± std.) for the total score after a drawn PDT image was submitted in the performance test for a machine with Processor Intel(R) Core(TM) i7-8700 CPU at 3.20 GHz, 3192 Mhz 6 Core(s), 12 Logical Processor(s) with 8 GB of RAM running 64-bit Windows Version 10 specifications.
The mPDT is very easy, simple and intuitive to use and it can be convenient for use by the elderly. Using mPDT for the PDT test also allows evaluation of the results objectively and qualitatively as well as for parametric assessment of the results. This can also allow differentiation of the dynamic components of the cognitive function underlying a given neurodegenerative disease. Furthermore, because redrawing and saving of the sensor data along with the images drawn by subjects are possible in any mobile, electronic device, the onset and time course of brain neuronal degeneration could be detected and monitored as a basis of a personal lifelog as well as in real time. Therefore, this tool is to evaluate the current cognitive functions of the examinee and better distinguish the causes of cognitive decline.
For future work, we are currently developing qualitative and automatic scoring algorithms for the CDT and the ROCF tests by expanding the algorithms used in mPDT. Directions for this future work include various drawing tests, such as a draw-a-person test, a draw-a-family test, and so on, which would need more specific deep neural networks for image segmentation, feature extraction and classification, and clustering correlations between features.

Conflicts of Interest:
The authors declare no conflict of interest. closure/opening, tremor automatic interpreting using U-net deep learning algorithm and sensor data system Appendix B closure/opening, tremor automatic interpreting using U-net deep learning algorithm and sensor data system Appendix B closure/opening, tremor automatic interpreting using U-net deep learning algorithm and sensor data system Appendix B closure/opening, tremor automatic interpreting using U-net deep learning algorithm and sensor data system Appendix B closure/opening, tremor automatic interpreting using U-net deep learning algorithm and sensor data system Appendix B closure/opening, tremor automatic interpreting using U-net deep learning algorithm and sensor data system Appendix B closure/opening, tremor automatic interpreting using U-net deep learning algorithm and sensor data system Appendix B closure/opening, tremor automatic interpreting using U-net deep learning algorithm and sensor data system Appendix B closure/opening, tremor automatic interpreting using U-net deep learning algorithm and sensor data system Appendix B automatic interpreting using U-net deep learning algorithm and sensor data Appendix B automatic interpreting using U-net deep learning algorithm and sensor data Appendix B