Quantifying Effects of Dataset Size, Data Variability, and Data Curvature on Modelling of Simulated Age-Related Motor Development Data

Abstract

Performance modeling using robotic tasks can be used to identify motor control impairments. Motor development in children and youth occurs non-linearly, with higher accuracy and decreased variability in performance as they reach adulthood. However, past models have not accounted for these differences. The objective of this research was to create an algorithm that accounts for variability in sample size and age, modifying the curvature and variability of the data to test the accuracy and repeatability. While increased sample size improves the model, data collection is often limited by funding or population interest in participation. The simulations provide models of variability relative to sample size of the typically developing population from 5 to 18 years. The algorithm was evaluated with a sample of two-hundred and eighty-eight children. Using these models based on varying sample size, one can have greater confidence when identifying motor deficits from outlying data. Future researchers can use the model accuracy and repeatability information from this work to assess confidence in their own models based on the dataset size, data variability, and data curvature.

1. Introduction

The purpose of this work was to model variability in motor performance across the duration of childhood (5–18 years) and identify age-specific confidence intervals using both uniform and non-uniform sample datasets. These models were then evaluated using data collected from a visually guided reaching task. First, a non-linear model of motor performance was simulated based on asymptotic decay and asymptotic growth observed in measurements of motor development with differing uniform and non-uniform sample sizes. Variability across the age groups was evaluated. The models are standardized and can be used with any task for which there is a neurological learning effect over time. The models provide data for both exponential growth (for example, reaching tasks) and decay curves (such as reaction time), which can be used to identify the range of variability (standard deviation) at each age, and can also be used to predict the number of participants required to detect differences from the typical curves. To provide a proof of concept that the models are effective, the results were applied to data collected from 288 participants from a visually guided reaching task using the Kinarm robot.

2. Background

Children and youth experience motor development in a manner that quickly evolves at younger ages, and the rate of improvement slows until they have reached “near-adult” performance [1,2,3]. For example, proprioception reaches “near-adult” performance around 8 years of age, whereas other aspects of coordination, such as reaction time, do not mature until 12–14 years of age [1,4,5]. Like height and weight, measures of motor performance (e.g., reaction time) of typically developing children and youth can be plotted against age to generate curves that reflect developmental expectations of motor function. These models of typical development are often used to identify deficits in performance in various patient populations [6,7,8,9,10,11] based on a comparison to some performance interval of a healthy control population (i.e., 95% of the range of healthy performance or mean ± 2 standard deviations). Normative models allow clinicians to more easily identify deficits in motor control at a single subject level [12,13]. If the results of one individual significantly differ from the typical population, an individual may be under- (or over-) performing. The ability to effectively detect deficits relies on the accuracy and reliability of the models of typical development. The first step in developing these normative models is the generation of reliable and reproducible curves of developmental behaviour, to accurately identify deficits in performance. This article provides clinicians with simulated developmental curves that include the range of standard deviations at each age, providing data that demonstrates the gradual decrease in variability of typical performance as children progress to adulthood. We further inform scientists about the number of participants required to generate accurate models of development from motor function data.

The generation of these developmental curves, including the mean and variance of performance at different ages, can be affected by various factors including the underlying variability of the data and the number of participants included in the analyses. There is evidence that variability in motor performance changes with age [1]. For example, as children reach “near-adult” performance, the variability in their movement decreases [1]. Therefore, using a fixed performance interval width for the whole curve (i.e., one that is calculated based on the entire sample mean) may result in a performance interval that is over-estimated for older ages. This could result in the misidentification of deficits when comparing behavior from patient groups especially those at the younger ages. Explicitly, one should not use the standard deviation data from eighteen-year-olds to diagnose an abnormality in a seven-year-old as there is greater variability in the younger population and misdiagnosis may occur.

As well, although it is recognized as good statistical practice, very few studies tend to include power analyses a priori to determine the appropriate number of participants required to achieve desired power [9]. The use of power analyses is particularly pertinent to studies of motor development given the non-linearity of age-related performance. For example, studies that recruit participants at older age groups may not require as many participants to achieve sufficient power to identify differences compared to those that recruit younger children.

Previous work has highlighted the importance of confidence intervals on fitted curves to determine the reliability of the modelled behavioural fits [9]. Including confidence intervals at each age from 5 to 18 years creates confidence bounds to reliably identify deficits in individual patient performance.

Robotic assessments of motor function have been shown to be highly objective and repeatable, and they can be effective therapeutic devices [14,15]. Using robotic assessments to measure motor function in children and youth can provide researchers with quantitative results that can be repeated across multiple participants and sessions. The data recorded from these robotic assessments can be used to generate normative models of motor development, and these models can be used to identify deficits on an individual level. As a result, the simulated models were compared to data collected from the Kinarm robot.

3. Methods

Non-linear modeling of motor development included curve fitting of both uniform and non-uniform distributions, followed by an assessment of accuracy and repeatability. All data used to assess the impact of dataset size, variability in data, and data curvature on the models were simulated, not data collected from participants. While the size of the uniform data was developed using simulations, the production of a non-uniform dataset represented a sample size similar to data collected in a visually guided reaching task (288 participants from ages 5–18 years) using the Kinarm robot. The reason to develop a non-uniform dataset with the same number of participants to the data previously collected was to be able to test the model with real data. Once all the models were developed, the confidence intervals calculated with the simulations were then applied to real data as a proof-of-concept.

3.1. Simulated Data with Uniform Distributions

A curve fitting algorithm was developed for non-linear regression models. Simulated curves were created with an equal number of datapoints at each integer age, Gaussian white noise was added, and exponential curves were fit to the simulated data. For simulations, the input curves, dataset sizes, and standard deviation of the Gaussian white noise were varied. Two types of curves were chosen to reflect clinically published data on motor coordination of reaching movements: asymptotic growth and asymptotic decay [2,11,16]. For example, the path smoothness of a reaching motion in children follows a growth curve with age, whereas reaction time follows a decay curve [2,11,16]. The dataset size ranged from 14 to 1792 datapoints to represent both small and larger studies that quantify sensorimotor or cognitive function in children and youth [2,5,9,10,17,18]. The standard deviations of the Gaussian white noise (i.e., input SDs) were chosen to reflect reported standard deviations of children and youth performance on motor and proprioception tasks [2,3,17,19,20,21,22].

The input curves increased or decreased asymptotically to a value of 1 from 0.5 or 2. The rates of growth or decay reflected different developmental trajectories where data reached “near-adult” performance by age 8, 10, 12, or 14 years.

A base curve, represented by the equation below, was used to generate simulated data for participants from ages 5–18:

$$y=a_0\ast e^{a_1\ast age}+a_2$$

(1)

The variable “age” represents the integer age. The coefficients a₀, a₁, and a₂ were calculated to simulate the desired ages of “near-adult” performance. “Near-adult” performance was defined as the curve being within 5% of the final value (0.95 for growth curves, 1.05 for decay curves). Given these parameters, eight input curves, eight dataset sizes, and 15 standard deviations for the Gaussian white noise (input SD) were used to total 960 simulation combinations. Ten-thousand curve fits were applied for each simulation combination. For each curve fit, new simulated data were created by adding a different random sample of Gaussian white noise to the input curve. For each combination, the 95-percentile range of fitted curves were calculated across the age range to generate confidence intervals for the curve fit.

The MATLAB function “nlinfit.m” was used to fit a non-linear multivariable function to the simulated data (Mathwork, Inc., MA, USA, version 2021b). The function requires the input data, initial estimations of the coefficients, the independent variable(s) (age in this case), and the desired equation used for curve fitting. After a curve was fit to the data, the standard deviation of the residuals relative to the fitted curve was calculated.

3.2. Simulated Data with Non-Uniform Distributions

Typically, a study does not have an equal distribution of integer ages and sex among the participants. To examine the effects of different numbers of participants, a representative sample of 288 child participants was simulated as non-uniform data. The same process was followed as mentioned above, but using the distribution of participants as shown in Figure 1. The chosen dataset was based on data that had been collected using a Visually Guided Reaching (VGR) task in the Kinarm robot (Kinarm, Kingston, ON, Canada). The coefficients for creating the input curves, the dataset sizes, and standard deviations can be found in the Supplementary Material.

3.3. Applying Confidence Intervals from Both Uniform and Non-Uniform Data Distributions to Data Collected from Child Participants

Data were collected using a Visually Guided Reaching (VGR) task in the Kinarm Exoskeleton Lab (Kinarm, Kingston, ON, Canada). All participants gave informed assent, and informed consent was obtained from the guardians prior to participation. This study was approved by the Health Sciences and Affiliated Hospitals Research Ethics Board at Queen’s University, Kingston, ON, Canada (application number 6004951) in accordance with the Helsinki Declaration of 1975, as revised in 2000 and 2008.

Participants sat in a system-integrated modified wheelchair base with their arms supported against gravity. The robot allows free planar movement of the upper limbs while recording elbow and shoulder joint kinematics. Hand position and kinematics were calculated from the joint kinematics. Participants sat in front of a virtual reality display, with vision of their hands and arms occluded, as seen in Figure 2A. Hand position feedback was displayed in each task on the virtual reality display. The robot has been described in detail previously [23].

The Visually Guided Reaching (VGR) task assesses single handed goal-directed reaching [24]. There are four peripheral targets set in a square, 6 cm apart, with a fifth starting target in the centre of the square. Participants are instructed to reach as quickly and accurately as possible to each target as they appear. The peripheral targets appear in a pseudo-random order, six times each for a total of 24 reaches per arm. Each reach consists of moving out to the peripheral target, then back to the central target when it reappears. Participants complete the task once with both their dominant and non-dominant hands. This task is represented in Figure 2B.

4. Results

4.1. Simulated Data with Uniform Distributions

Figure 3A,B display growth and decay input curves when no noise is added to the data. The four different curves in each represent the growth or decay to ages 8, 10, 12, and 14. Figure 3C–F show how variations in standard deviation (SD) alter the dispersion of data for the growth curve associated with near-adult performance at age 8.

Figure 4A highlights the range of model fits for 10,000 runs for the early development curve (growth to age 8), with a SD of 0.21 and a sample size of 56 datapoints. The estimated mean and associated 95% confidence interval are plotted for each age, as well as the estimated mean ± 2SD and its 95% confidence interval (CI). The estimated 2SD around the mean represents 95% of healthy performance range and can be used as a cut-off value. If an individual datapoint lies outside this 2SD range around the mean, the participant would be identified as having a deficit. On average, both the mean and 2SD closely matched the values for the input curves (<0.7% error), although the individual estimates display considerable error when compared to the input curve.

At age 5, the 95% CI of the curve fits ranged from 0.3 to 0.7 (range = 0.4) even though the correct value was 0.5. At age 11, this range reduced to 0.12. The true 2SD value is 0.92 (0.5 + 2 × 0.21SD) at age 5 for this dataset and the estimated values averaged to 0.91; however, the CI for 2SD varied substantially. Figure 4A shows the range of estimates for 2SD (top curve) is 0.4 (0.72–1.12) which then drops to 0.15 at age 11. Similar patterns of variability in the mean and SD are observed when SD and the data sample size are modified (Figure 4B–D). For example, Figure 4B highlights that the 95% CI for the mean and 2SD ranges reduce to 0.04 and 0.05, respectively, at age 11.

Age differences around the mean at the 95% CIs and dataset size together impact whether an individual child is identified as within the range of typical performance for that age. The 95% CI is largest at age 5, regardless of dataset size, and the CI range generally decreases with increasing age. The range of all 95% CIs decrease and become flatter, regardless of age, as dataset size increases. For different input SDs, the range of the 95% CI for the same age and dataset size is affected in smaller dataset sizes, but not in larger datasets. For example, with the input SD of 0.61 and 112 datapoints, the 95% CI ranges are 0.95, 0.31, and 0.40, at age 5, 11 and 18, respectively.

The impact of all the different input SDs on the 95% CIs is shown in Figure 5C. The 95% CIs were calculated for each input SD, then the distribution of these CIs was calculated and represented by the error bars. This distribution represented by the error bars will be called the CI span to avoid confusion. For clarity of the image, only four dataset sizes are shown: 14, 56, 224, and 1792 datapoints. Smaller datasets show more variation in the CI spans for different input SDs, and the variation decreases with increasing dataset size. The largest CI span for the datasets with 14, 56, 224, and 1792 were 0.77, 0.38, 0.19, and 0.04 SDs, respectively. Although the repeatability of the curve fitting was always poorest for the 5-year-old datapoints, the largest variability among input SDs was typically at ages 6 or 7.

The same trends were found for all input curves (see Supplementary Material).

4.2. Simulated Data with Non-Uniform Distributions

The same simulations were run with a non-uniform distribution of sample size. The simulations were run with the eight input curves, 15 SDs, but only two dataset sizes. The datasets had 288 and 576 datapoints to reflect the number of datapoints recorded for bimanual and unimanual tasks, respectively. The participant ages were not uniformly distributed across the age range (mean = 13.4, SD = 3.13, 190 male, 98 female) as they matched the demographics of participants collected using the Kinarm robot. Most participants were between ages 11 and 17 with few participants under age 8. There were also nearly twice as many male participants as female participants.

Figure 6 shows the same information as Figure 5, but for the non-uniform dataset simulations with the decaying to age 8 input curve. Unlike the uniform datasets, the non-uniform datasets show a larger difference between the 95% CIs.

Figure 6C shows the same information as Figure 5C, but for the non-uniform dataset. The larger differences in 95% CIs among input SDs resulted in wider CI spans of the input SDs at age 5, but similar spans for the other ages when comparing the uniform and non-uniform datasets. The largest CI span for the 288 and 576 datapoint datasets were 0.81 and 0.51 SD, respectively. The CI spans for the non-uniform datasets are most comparable to the spans for the uniform datasets with 14 datapoints and 56 datapoints. The largest CI spans for the 14 and 56 datapoints were 0.77 and 0.38, respectively. The average CI spans at each age within the uniform datasets with 14 and 56 datapoints were 0.28 and 0.19, compared to 0.24 and 0.15 for the non-uniform 288 and 576 datapoint datasets.

To evaluate differences, the 95% CIs for uniform datasets and the closest (relative to size) non-uniform datasets at each age are plotted in Figure 7. The comparisons were made with the input SD of 0.21. There were five participants who were five years old, so the 95% CIs of the dotted line at age 5 are from the uniform dataset with 56 datapoints, as there would be four datapoints at age 5.

Figure 7A shows the results from the 288 datapoint simulations, whereas Figure 7B shows the results for the 576 datapoints. The 95% confidence intervals on the curve fitting differed when calculated from the simulations with a uniform vs. non-uniform dataset. These differences were more evident in the younger ages.

4.3. Assessing Accuracy and Repeatability of Curve Fitting: Fitting Confidence Intervals to Participant Data

The confidence intervals created by the simulations were applied to curves produced by data collected from participants. These confidence intervals can be used to quantify the certainty of the identification of a deficit in a motor function performance measure. Curves were fit to Reaction Time (RT) collected from the Visually Guided Reaching (VGR) task. Figure 8 shows the 95% confidence intervals on the curve fit to RT using the uniform and non-uniform datasets. Figure 8A shows the 95% confidence intervals from the uniform datasets, whereas Figure 8B shows the intervals from the non-uniform datasets. The difference between the 95% confidence interval widths from uniform and non-uniform datasets is 0.07 s at age 5, and decreases to 0.03 s at age 6. The differences are then below 0.01 s for all other ages, with the smallest difference, 0.0006 s, at age 16. These differences in the 95% confidence interval width range from 0.21% to 12% of the fitted curve’s value at each age.

From ages 9–18, the average difference between the 95% CIs generated from the uniform and non-uniform datasets with matched numbers of participants were 0.18 and 0.12 input SD for the 288 and 576 datapoint datasets, respectively.

5. Discussion

An algorithm for creating non-linear developmental models of motor performance that accounted for changing variability was created. The repeatability of the curve fitting was measured using simulated data with both a uniform age distribution and a non-uniform distribution similar to the participant data. The simulations tested how dataset size, variability in the data, and curvature of the data affected the curve fitting.

5.1. Repeatability of Curve Fitting

Rather than setting desired goals for accuracy, curve fitting performance was determined to inform future work (see Supplementary Material). When collecting data for clinical studies, the number of participants is often difficult to determine a priori; however, the current findings could provide insight into limitations and benefits of different sized groups and to inform power analyses. Similarly, the information could be used to determine minimum participant numbers for effective curve fitting depending on the expected outcomes.

As expected, sample size has the largest impact on accuracy and repeatability for non-linear regressions for both uniform and non-uniform data distributions. The repeatability of the curve fitting improved in a decaying exponential fashion with increasing dataset size. The average 95% CIs across all input SDs at the beginning of the curve decreased from 2.7 to 0.24 input SD, a ten-fold decrease, from 14 to 1792 datapoints, respectively, for the input curve representing growth to age 8 to reach “near-adult” performance. Most of the decrease was observed for datasets smaller than 224. Once the dataset size reached 224 datapoints, the average 95% CI was 0.68, so 82% of the overall improvement occurred between sample sizes from 14 to 224 datapoints.

The smaller improvements in repeatability above a dataset of 224 datapoints are consistent with another study on logistic regression for modelling complications from radiotherapy [25]. The authors found that improvements in predictive power of the models were marginal among datasets with more than 200 datapoints but were larger among smaller datasets [25]. Schönbrodt and Perugini determined stabilized correlations occurred at 250 datapoints which is similar to our 224 datapoints [26]. These results indicate that using non-linear models to create typical performance ranges requires above 200 datapoints to create ranges which could accurately differentiate typical from atypical performance. More simulations would be necessary to determine an exact number of datapoints required, but the outcomes would differ depending on the model and output parameters.

Regardless of input dataset size, SD, or type of input curve, the poorest accuracy and repeatability was always found at the beginning of the curve (age 5). These results indicate that it may be useful to either use weighting methods in curve fitting to improve accuracy at the beginning of the curve, or to collect more datapoints at the boundaries of the curve.

The 95% confidence intervals on the curve fitting were similar between the non-uniform and uniform datasets from ages 9 and up (within 0.18 input SD). These results indicate that the simulations from the uniform dataset may be suitable to create 95% confidence intervals on new fitted curves for ages 9 and up for this particular dataset. The poor repeatability from ages 5 to 8 indicate that using these normative models to identify motor deficits in children that age may not be appropriate.

The curve fitting algorithm has been shown to be a reliable tool for fitting non-linear functions to simulated data and determining distributions of the input data. The algorithm could prove to be a useful tool for creating future models of experimental data with known accuracy and repeatability based on dataset size and data variability.

5.2. Confidence Intervals Based on Normative Models

Normative models were created for the Reaction Time (RT) data collected from the Visually Guided Reaching (VGR) task. The confidence intervals generated from the uniformly distributed and non-uniformly distributed datasets were compared across the fitted curve to the RT data. The confidence intervals were similar from ages 9–18 and the simulations from the uniform distributions could be used for future normative models, even if they are generated from non-uniformly distributed data. A more thorough analysis of the impact of non-uniformity of input data distributions on curve fitting should be considered.

The authors have not found previous studies that included confidence intervals in normative models of motor development in children [9]. Including confidence intervals quantifies the reliability of the curve fitting and improves accuracy in the identification of motor function deficits. Reporting confidence intervals on statistical tests is common in other disciplines and should be more common for these types of normative models to allow for a full understanding of the strength of the conclusions drawn from the results.

5.3. Limitations

The curve fitting algorithm was provided a very good initial estimate of the coefficients used to generate the data (within 1% of each coefficient), and the accuracy and repeatability were not compared for different ranges of input coefficients. Poor initial estimates would most likely result in poorer fits to the data. The issue of poorer fits can be partially fixed by changing parts of the curve fitting function, such as the number of optimization iterations or optimization parameters. These additional solutions have not been explored in this work.

5.4. Clinical Implications

The results of the simulations can be applied to normative models of motor development to inform the certainty of identification of a deficit. Clinicians can use normative models to identify impairments and target and track therapies. The inclusion of confidence intervals on these models will help clinicians decide if a deficit is suspected.

Simulated data were used to assess how dataset size, curvature in the data, and variability in the data affect the accuracy and repeatability of the curve fitting. The simulations for curve fitting were completed to fill the gap in which there was a lack in reporting of true confidence in normative ranges of motor development in children. The accuracy and repeatability were proportional to the variability in the data; however, when normalized to the variability, the accuracy and repeatability were similar among the different curve fits. Increasing dataset size improved the accuracy and repeatability of the curve fitting. Most of the improvement from the uniformly distributed data was observed at the dataset size of 224 datapoints. The results of these simulations can be used to determine a necessary sample size for a desired confidence in a curve fit, or to assess the confidence in a curve fit based on the collected dataset size. The identification of the important parameters to explore when performing these types of simulations may encourage other researchers to complete these simulations and report confidence intervals in their results.

The results from the simulations also showed that the confidence intervals of an exponential fit are not constant across the entire range of the input variable, age in this case. These results highlight that using a single value for confidence interval width may be inappropriate for this type of analysis. Confidence intervals should be generated for different input values, rather than using one value across the entire range of input values.