Reliability and Validity of a Dog Personality and Unwanted Behavior Survey

Simple Summary Dogs have distinct personalities, meaning differences between individuals that persist throughout their lives. However, it is still unclear what traits are required to define the whole personality of dogs. Personality and unwanted behavior are often studied using behavioral questionnaires, but researchers should ensure that these questionnaires are reliable and valid, meaning that they measure the behavior traits they were intended to measure. In this study, we first examined what traits define a dog’s personality. We discovered seven personality traits: Insecurity, Training focus, Energy, Aggressiveness/dominance, Human sociability, Dog sociability and Perseverance. We also studied six unwanted behavior traits: noise sensitivity, fearfulness, aggression (including barking, stranger directed aggression, owner directed aggression and dog directed aggression), fear of surfaces and heights, separation anxiety, and impulsivity/inattention (including hyperactivity/impulsivity and inattention). We examined the reliability of these traits by asking some dog owners to answer to the questionnaire twice, several weeks apart, and by asking another family member to answer the questionnaire of the same dog. Furthermore, we studied the validity of these traits by forming predictions based on previous literature. Based on our results, this personality and unwanted behavior questionnaire is a good tool to study dog behavior. Abstract Dogs have distinct, consistent personalities, but the structure of dog personality is still unclear. Dog personality and unwanted behavior are often studied with behavioral questionnaires. Even though many questionnaires are reliable and valid measures of behavior, all new questionnaire tools should be extensively validated. Here, we examined the structure of personality and six unwanted behavior questionnaire sections: noise sensitivity, fearfulness, aggression, fear of surfaces and heights, separation anxiety and impulsivity/inattention with factor analyses. Personality consisted of seven factors: Insecurity, Training focus, Energy, Aggressiveness/dominance, Human sociability, Dog sociability and Perseverance. Most unwanted behavior sections included only one factor, but the impulsivity/inattention section divided into two factors (Hyperactivity/impulsivity and Inattention) and the aggression section into four factors (Barking, Stranger directed aggression, Owner directed aggression and Dog directed aggression). We also examined the internal consistency, test-retest reliability, inter-rater reliability and convergent validity of the 17 personality and unwanted behavior traits and discovered excellent reliability and validity. Finally, we investigated the discriminant validity of the personality traits, which was good. Our findings indicate that this personality and unwanted behavior questionnaire is a reliable and valid tool that can be used to study personality and behavior extensively.


Introduction
Animal personality traits are behavior traits that are relatively stable over time and across contexts [1][2][3]. Especially in humans and captive animals, the concept of personality often encompasses the combination of traits needed to define an individual's personality and distinguish individuals from each other [4,5]. Thus, the structure of personality in different animal species interests researchers.
It is still unclear what traits form personality in dogs, as different studies have discovered a different number of traits [6]. For example, the well-validated Dog Personality Questionnaire (DPQ) consists of five personality factors: Fearfulness, Aggression towards people, Aggression towards animals, Activity/excitability, and Responsiveness to training [7]. Similarly well-validated Revised Monash Canine Personality Questionnaire (MCPQ-R) includes five factors: Extraversion, Motivation, Training focus, Amicability, and Neuroticism [8]. Other studies have discovered, for example, four [9,10], five [11,12] or eleven [13] factors. Jones and Gosling [6] categorized traits measured in different studies to six wider domains: activity, which mostly involves motor activity; aggression, including both human directed and dog directed displays of aggressive behavior; sociability, including social behaviors towards both humans and other dogs; responsiveness to training, which includes the tendency to stay focused, willingness to work with people and quickness of learning; submissiveness, the opposite of dominance; and fearfulness, also involving reactivity.
Questionnaires are a commonly used method of collecting behavioral data from companion animals. Despite their frequent use, they are somewhat subjective, as they do not collect data straight from the animals, but their owners. Many surveys, however, are reliable [14][15][16][17][18][19]. Still, new surveys and previously validated surveys that are translated or edited should be extensively validated to ensure that they measure what they are designed to measure.
Nearly a decade ago we designed and validated a behavior questionnaire focused on fearfulness in pet dogs [14]. This was later expanded to include aggressive behavior [20] and a previously used compulsive behavior section [21] and later included sections about fear of surfaces and heights, separation anxiety [22] and impulsive and inattentive behavior adapted from Vas et al. [23].
As we examined the environmental factors associated with behavior [24][25][26][27], it became evident that the survey could use another revision to allow for analysis of variation in behavior traits instead of only case-control studies. Therefore, we redesigned most sections of the questionnaire to utilize factor analysis for forming the scores for each dog. Here, we report the factor structure, reliability and validity of these redesigned survey sections.

Questionnaire
The questionnaire included nine behavioral sections: personality, noise sensitivity, fearfulness, separation-related behavior, fear of surfaces and heights, aggression, impulsivity/inattention, cognition, and compulsive behavior. Additionally, the questionnaire included an extensive background section as well as a health section. The questionnaire can be found in Supplementary File: Questionnaire. Here, we report results from other behavior sections except for compulsive behavior, as compulsions are largely separate traits, making scale construction questionable, and cognition, which is based on a validated survey which does not utilize factor analysis [28].

Personality Questionnaire
We developed an adjective-based dog personality questionnaire using a combination of top-down and bottom-up approaches. Firstly, we searched for and utilized adjectivebased personality questionnaires designed to be answered by caretakers of different pets and captive animals [8,15,[29][30][31][32][33][34][35][36][37]. From these questionnaires, we selected behaviors that were applicable to dogs. After this top-down approach, we added adjectives relevant for dogs particularly. This combined bottom-up/top-down approach allows the inclusion of species-specific behaviors and simultaneously enables comparison between species [35].
After excluding irrelevant, completely overlapping or ambiguous adjectives, the questionnaire included 63 adjectives and definitions (Supplementary File: Questionnaire) in a randomized order. Owners were asked to indicate how strongly they agreed with the answering options being "strongly disagree", "somewhat disagree", "neither agree or disagree", "somewhat agree", "strongly agree" and "I don't know".

Unwanted Behavior Questionnaires
Other behavioral sections focused on possibly unwanted or problematic behaviors, including noise sensitivity (fear of thunder, fireworks, gunshots and other noises), fearfulness (towards unfamiliar people, dogs and situations), separation-related behavior (both when the owner is leaving and when the dog is alone), fear of surfaces and heights, aggression (towards strangers, the owner and other dogs), impulsivity/inattention and compulsive behavior (including, for example, tail chasing, pacing, flank sucking and light/shadow chasing).
The impulsivity/inattention section was developed by Vas and colleagues [23] and translated to Finnish. Other sections were based on our previous questionnaire [22]. We redesigned some sections, and the new questionnaire can be found in Supplementary File: Questionnaire.

Subjects
Before analyses, we excluded dogs with missing basic information and duplicate answers ( Figure 1). Firstly, we excluded dogs that were deceased more than 3 months before answering (206 dogs). We also excluded dogs whose birthdays were not reported by their owners and could not be verified from other sources (182 dogs) and dogs whose birthday was reported to be the date of answering (2 dogs). Some owners did not report their dog's sex. For these, we tried to verify their sex from other sources and when we could not, we classified the dogs as male/female based on their calling names (39 dogs). Of these, purebred dogs were assumed intact and mixed breed dogs neutered, as this was the case with most dogs in our study population. Finally, some owners had answered for their dogs more than once. From these, we selected the most complete or newest answer.
The final dataset included responses from 15,371 dogs in 329 breeds and breed variants. As many breeds were represented by only a few individuals, we grouped many of them based on the genetic relatedness [38], the purpose of the breed and known similarities in behavior. As a result of this grouping, the final sample included 19 individual breeds, 32 breed groups and mixed breed dogs (Table S1).
During the last months of the data collection, we selected a set of owners who had answered all or most questionnaire sections 1-3 months prior and sent them a request to participate in the test-retest reliability study. These owners were requested to answer the questionnaire sections again.
Finally, we selected a set of owners who had answered all or most questionnaire sections 1-3 months prior and reported to live with another adult. We requested that they participate in the inter-rater reliability study by allowing the other adult family member to answer the questionnaire sections. For this set of questionnaires, we included a question asking how long the other respondent has known the dog. The answering options were "less than 3 months", "3-6 months", "6 months to 1 year", "1-5 years" and "over 5 years". The participants also had to declare that they had not discussed their dog's behavior with the other owner when filling the questionnaires.
The participants also had to declare that they had not discussed their dog's behavior with the other owner when filling the questionnaires.

Convergent and Discriminant Validity
We evaluated the convergent validity of the questionnaire by hypothesis testing (concurrent validity). We collected hypotheses from previous literature and included 1-4 hypotheses per factor (Table S2). For example, based on previous literature, we hypothesized that female dogs would, on average, be more fearful (towards strangers, dogs and situations) than male dogs [9,[39][40][41][42][43]. We also hypothesized that dogs whose owners feel that their dog shows a particular unwanted behavior, for example, aggression towards strangers or fear of thunder, would, on average, have higher factor scores than dogs that do not show a particular behavior (Table S2).
For discriminant validity, we evaluated the correlations between factors to see whether factors that should not be correlated. Our previous study indicated that all the unwanted behavior traits we studied are correlated [22] and therefore, we only evaluated the discriminant validity of the personality section.

Factor Analyses
We performed factor analysis for all questionnaire sections to reduce the questionnaire items into a smaller number of biologically meaningful traits. We excluded ques-

Convergent and Discriminant Validity
We evaluated the convergent validity of the questionnaire by hypothesis testing (concurrent validity). We collected hypotheses from previous literature and included 1-4 hypotheses per factor (Table S2). For example, based on previous literature, we hypothesized that female dogs would, on average, be more fearful (towards strangers, dogs and situations) than male dogs [9,[39][40][41][42][43]. We also hypothesized that dogs whose owners feel that their dog shows a particular unwanted behavior, for example, aggression towards strangers or fear of thunder, would, on average, have higher factor scores than dogs that do not show a particular behavior (Table S2).
For discriminant validity, we evaluated the correlations between factors to see whether factors that should not be correlated. Our previous study indicated that all the unwanted behavior traits we studied are correlated [22] and therefore, we only evaluated the discriminant validity of the personality section.

Factor Analyses
We performed factor analysis for all questionnaire sections to reduce the questionnaire items into a smaller number of biologically meaningful traits. We excluded questions with more than 20% missing responses and dogs with more than 20% missing responses in the remaining items. Before factor analyses, we tested the suitability of our datasets for factor analysis with the Kaiser-Meyer-Olkin test for sampling adequacy from the package psych [44].
We conducted the factor analyses with the package psych [44]. We used polychoric correlation matrices (as all questionnaire items were coded on a Likert scale) and conducted Animals 2021, 11, 1234 5 of 16 the factor analyses without rotation and with mean imputation. We evaluated the number of factors to be extracted with the scree test and Velicer's minimum average partial (MAP) test. Furthermore, we evaluated the quality of the factor structure by extracting all possible structures (Goldberg's hierarchical tree) starting from 1 factor up to at least two factors more than recommended by the scree test. We evaluated the conceptual interpretability of the competing factor structures, as well as compared the root mean square error of approximation (RMSEA) and the Tucker-Lewis index between these structures.

Internal Consistency, Test-Retest Reliability and Inter-Rater Reliability
We calculated Cronbach's Alpha and Guttman's Lambda 6 with the package psych [44] for all factors. For test-retest reliability, we used the package psych [44] to calculate the correlations between the first and second time of answering. These correlations were calculated for all items and extracted factors. Finally, for inter-rater reliability, we used the package psych [44] to estimate the inter-rater reliabilities of factors and items based on intraclass correlation coefficients.

Convergent and Discriminant Validity
Before validity analyses, the items that did not load onto any factor (all loadings <0.3) and items that were unreliable based on test-retest and inter-rater reliabilities were removed and the factor analyses were conducted again for the reduced set of items. We then extracted the factor scores for all individual dogs with the package psych [44], using the "tenBerge" estimation method for multifactorial structures and "Thurstone" for unifactorial structures. These factor scores were used in subsequent validity analyses.
To calculate the validity coefficients, we used Pearson correlations for continuous predictors (for example, dog's age) and Welch t-tests for discrete predictors (for example, dog's sex). We corrected all p-values for false discovery rate (FDR) to decrease the probability of type I error. The significance cut-off p-value was set at p < 0.05.

Descriptive Statistics
In total, the final dataset included 15,371 dog individuals from 11,498 owners. This sample included dogs in 329 breeds and breed variants, which were grouped to form 31 breed groups and 21 individual breeds (Table S1). Of the dogs, 52.9% were females and 47.1% males. In total, 23.5% of dogs (26.0% of females and 20.7% of males) were neutered. Age varied between 0.16 (2 months) and 18.1 years, with a mean age of 5.23 years (sd 3.47). The number of dogs varied between questionnaire sections (Table 1). In the test-retest reliability and inter-rater reliability datasets, the number of dogs and time between the two answers also varied between sections (Table 1).

Factor Structure
Personality questionnaire items formed seven factors, which were named Insecurity, Training focus, Energy, Aggressiveness/dominance, Human sociability, Dog sociability and Perseverance (Table 2). This factor structure accounted for 54% of the variance in behavior.
Noise sensitivity, fearfulness, fear of surfaces and heights and separation anxiety sections each formed one factor, which included all or most of the items in the particular section. These factors were correspondingly named Noise sensitivity, Fearfulness, Fear of surfaces and heights and Separation anxiety (Tables S3-S6) and accounted for 51%, 34%, 60% and 51% of the variance, respectively. The impulsivity section, in which the items were translated from [23], constituted two factors, as in the original study: Inattention and Hyperactivity/impulsivity (Table S7), which explained 52% of the variance. Aggression section items formed four factors, which were named Barking, Stranger directed aggression, Owner directed aggression and Dog directed aggression (Table S8) and accounted for 63% of the variance.

Reliability and Internal Consistency
Internal consistency of most factors was adequate (Table 3). In the personality questionnaire, Cronbach's alpha varied from 0.61 (Perseverance) to 0.89 (Insecurity) and Guttmann's lambda 6 from 0.67 (Perseverance) to 0.91 (Insecurity). In other questionnaire sections, Cronbach's alpha varied from 0.63 (Owner directed aggression) to 0.95 (Noise sensitivity), with a mean of 0.78. Guttman's lambda 6 varied from 0.68 (Owner directed aggression) to 0.97 (Noise sensitivity), with a mean of 0.81. Besides personality traits Human sociability and Perseverance, and other traits Owner directed aggression and Dog directed aggression, all estimates were over 0.70. Test-retest reliability of all factors was good (Table 3). In the personality questionnaire, the correlation between the two timepoints varied from 0.70 (Perseverance) to 0.91 (Insecurity, Training focus and Aggressiveness/dominance). In other sections, the correlation varied from 0.77 (Hyperactivity/impulsivity) to 0.93 (Noise sensitivity). Mean test-retest reliability of all factors was 0.84. Test-retest reliability estimates of individual items can be found in Tables S3-S9. Inter-rater reliabilities of all factors were similarly good (Table 3). In the personality questionnaire, highest inter-rater reliability was obtained by the Aggressiveness/dominance factor (ICC(1,1) = 0.81, ICC(1,k) = 0.90) and the lowest by the Human sociability factor (ICC(1,1) = 0.48, ICC(1,k) = 0.65). In other sections, the highest inter-rater reliability was achieved by the Noise sensitivity factor (ICC(1,1) = 0.80, ICC(1,k) = 0.89) and the lowest by the Separation anxiety factor (ICC(1,1) = 0.50, ICC(1,k) = 0.66). Mean ICC(1,1) of all factors was 0.69 and mean ICC(1,k) was 0.81. Inter-rater reliability estimates of individual items can be found in Tables S3-S9.

Convergent and Discriminant Validity
In total, we formed 51 hypotheses to validate 17 factors (Table 4 and Table S2). Of these 51 hypotheses, 47 were met and only 4 did not hold true. These four hypotheses were in Human sociability, Separation anxiety, Barking and Owner directed aggression.  We assessed the discriminant validity of the personality section by evaluating the factor correlations between the factors (Table 5). Moderate correlations (>0.3) were observed only between Training focus and Insecurity and between Aggressiveness/dominance and Dog sociability.

Discussion
We developed a dog personality and unwanted behavior questionnaire and examined the reliability and validity of the survey sections. Based on our analyses, the survey has excellent reliability and validity. Thus, it should be a reliable and objective method of collecting behavioral data from dog owners.
Of the other questionnaire sections, noise sensitivity, fearfulness, separation anxiety, and fear of surfaces/heights each formed only one factor. In noise sensitivity, this is not surprising, as noise sensitive dogs often display fearfulness towards many different noises [20,22,[46][47][48]. In contrast, unexpectedly, fearfulness also comprised only one trait, despite it is often divided into social and non-social fear [49] or fear towards specific targets [13,45]. Therefore, it seems that at least in our dataset and with our questions, fear towards both unfamiliar people, dogs and situations are highly correlated.
The impulsivity/inattention section was divided into two factors and the aggression section into four factors. The factors of the impulsivity/inattention section were Inattention and Hyperactivity/impulsivity, which was also discovered by Vas et al. [23] and by our previous study using their questionnaire (but translated to Finnish) [22]. The factors of the aggression section were Barking, Stranger directed aggression, Owner directed aggression and Dog directed aggression. Another, nearly equally good factor structure split the items into two factors: Meeting aggression (including items related to barking, stranger directed aggression and aggression towards unfamiliar dogs) and Resource/handling aggression (including items related to owner directed aggression and resource aggression towards familiar dogs). We opted to use this four-factor structure as we felt that Barking might be more related to fearfulness. Indeed, despite that Barking was highly correlated with Stranger directed aggression, the correlation between Barking and Fearfulness was much higher than the correlation between Stranger directed aggression and Fearfulness (0.57 and 0.39, respectively). Some previous studies have also indicated that aggression traits are separate [22,50], supported by our results.
The internal consistency of all factors was adequate. In most factors, Cronbach's alpha exceeded the suggested [51] cut-off of 0.70. Four factors, Human sociability, Perseverance, Owner directed aggression and Dog directed aggression achieved values less than 0.70. Cronbach's alpha values ranged from 0.61 (Perseverance) to 0.95 (Noise sensitivity), with a mean of 0.78. These Cronbach's alpha values were similar to previous studies, in which Cronbach's alpha has varied from less than 0.60 to more than 0.90 [7][8][9][11][12][13]37] and estimates falling between 0.60 and 0.70 are common [9,[11][12][13]. Cronbach's alpha is highly dependent on the length of the scale [7]. Perseverance and Dog directed aggression included only four items and Owner directed aggression only five items, explaining this low internal consistency. Human sociability, on the other hand, included eight items but the loadings of many items were low. Therefore, these scales could be improved by including additional questions that resemble high loading items.
Our personality and unwanted behavior factors had good test-retest reliability. Testretest reliability, measured by Pearson correlation coefficient, ranged from 0.70 (Perseverance) to 0.93 (Noise sensitivity), with a mean of 0.84. Jones [7] reported even higher test-retest reliability estimates, ranging from 0.88 to 0.94 with a similar retest interval than our study. However, their extensive literature search discovered that the mean test-retest reliability estimate for dog behavior was 0.63 [7].
The inter-rater reliability of our extracted factors was good, with ICC(1,1) values ranging from 0.48 (Human sociability) to 0.81 (Aggressiveness/dominance) and ICC(1,k) values ranging from 0.65 (Human sociability) to 0.90 (Aggressiveness/dominance). The mean ICC(1,1) of factors was 0.69 and the mean ICC(1,k) was 0.81. In previous studies, ICC values have ranged between 0.07 and 0.98 [7,11,[52][53][54], with most values varying between 0.50 and 0.80. In human and animal personality studies, the mean interobserver agreement is around 0.50 [55]. For example, in one study, ICC values of human five-factor model subscales varied between 0.30 and 0.65 [56]. In human personality research, Cicchetti [51] suggested having ICC values of at least 0.4 but preferably over 0.6. Thus, our inter-rater reliability estimates are mostly excellent and in line with previous studies.
To examine the convergent validity of our questionnaire, we formed 51 hypotheses primarily based on previous literature. Of these 51 hypotheses, only four were not met. Based on one study [8], we hypothesized that large dogs would have a higher mean Human sociability than small dogs but instead discovered the opposite association, even though the Pearson correlation coefficient was small (−0.09). Unlike we hypothesized, large dogs did not display less separation anxiety [41,57,58], as separation anxiety was not associated with body size. Furthermore, we hypothesized that male dogs would bark more [42] and that older dogs would show more owner directed aggression [59,60], but these factors were not associated with these behaviors. Other hypotheses held true, and all of these four traits also included hypotheses that were met. Furthermore, in all unwanted behavior sections, we hypothesized that dogs who were reported to display the particular trait would show a higher mean score than dogs reported not to display the behavior. Indeed, the difference in means between these groups was large and highly significant.
Finally, we examined the discriminant validity of our personality questionnaire by examining the correlations between personality factors. Moderate (Pearson correlation coefficient more than 0.30) were observed only between Dog sociability and Aggressiveness/dominance, and between Insecurity and Training focus. This correlation between Dog sociability and Aggressiveness/dominance is not surprising, as Aggressiveness/dominance was comprised of items related to behavior towards other dogs and as in one previous study, similar items have loaded onto a single factor [9]. The correlation between Insecurity and Training focus is, however, interesting, as it has not been observed previously. Training focus included items related to predictability and reactivity, which Jones and Gosling [6] categorized into fearfulness. Most correlations between factors were small, as in previous studies [10,61], indicating good discriminant validity.
This study has limitations, especially regarding the broader use of the questionnaire. Firstly, dog owners reported their dog's age, sex, and breed, and we did not confirm the accuracy of their answers. Secondly, the questionnaire has been translated to English, but the validity of this translated version has not been examined. Furthermore, all the respondents included in the test-retest and inter-rater reliability datasets were Finnish speakers residing in Finland. Therefore, before collecting behavioral data outside Finland, the reliability and validity of the translated questionnaire should be confirmed. The suitability of the questions about the living environment and background of the dog should be carefully assessed as well. Effort should also be made to ensure that dog owners understand their dog's personality traits correctly. For example, dominance is a true concept, which is, however, often misunderstood by owners to mean that dogs would purposefully strive for dominance in dog-dog and dog-owner interactions [62].
In the future, we plan to analyze the dataset collected with this questionnaire, for example, examine the environmental factors associated with personality and unwanted behavior and the heritability of these traits. Behavior and personality traits are highly complex, as they are influenced by possibly hundreds of genes, tens of environmental factors in different life stages and interactions between these different factors [63], and behavior may change during life as well [64]. Therefore, longitudinal studies would be needed to assess the effect of these factors and trait change throughout life.

Conclusions
This study examined the structure, test-retest reliability, inter-rater reliability, convergent validity and discriminant validity of a dog personality and unwanted behavior questionnaire. We discovered that dog personality included seven personality traits, which paralleled previous dog personality traits. This personality and unwanted behavior questionnaire was shown to have good test-retest and inter-rater reliability. We examined the convergent validity with hypotheses formed based on previous research and most of these hypotheses held true, indicating excellent validity of this questionnaire tool. In conclusion, this questionnaire was shown to be a reliable and valid measure of dog personality and unwanted behavior.
Supplementary Materials: The following are available online at https://www.mdpi.com/article/10 .3390/ani11051234/s1, Table S1. Breeds and breed groups, Table S2. Hypotheses and studies they were derived from, Table S3. Item loadings, test-retest reliabilities and inter-rater reliabilities in the noise sensitivity questionnaire, Table S4. Item loadings, test-retest reliabilities and inter-rater reliabilities in the fearfulness questionnaire, Table S5. Item loadings, test-retest reliabilities and inter-rater reliabilities in the fear of surfaces and heights questionnaire, Table S6. Item loadings, test-retest reliabilities and inter-rater reliabilities in the separation anxiety questionnaire, Table  S7. Item loadings, test-retest reliabilities and inter-rater reliabilities in the impulsivity/inattention questionnaire. Questionnaire translated from [50], Table S8. Item loadings, test-retest reliabilities and inter-rater reliabilities in the aggression questionnaire, Table S9. Test-retest reliabilities and inter-rater reliabilities in the personality questionnaire, Supplementary File: Questionnaire.