Construct Validity of a New Health Assessment Questionnaire for the National Screening Program of Older Adults in Japan: The SONIC Study

The Japanese government has implemented a new screening program to promote measures to avoid worsening lifestyle-related diseases and frailty among the older population. In this effort, the government formulated a new health assessment questionnaire for the screening program of old-old adults aged ≥75 years. The questionnaire comprises 15 items, of which 12 address frailty, two address general health status, and one addresses smoking habits. This study examined the construct validity of this questionnaire, using the explanatory factor analysis (EFA) and confirmatory factor analysis (CFA). The data used in this study were drawn from a mail-in survey conducted in 2020 as part of the Septuagenarians, Octogenarians, Nonagenarians Investigation with Centenarians study. A total of 1576 respondents (range, 78–99 years of age) were included in the study. Although the EFA did not show an interpretable factor structure of the questionnaire with 15 items, the CFA using only 12 frailty-related items showed the goodness of fit for a higher-order factor “frailty”, and the five frailty-related sub-factors model was acceptable. These results suggest that the total score of the 12 frailty-related items in the questionnaire can be used as an indicator of the degree of “frailty”.


Introduction
In April 2008, the Japanese government implemented a nationwide annual screening program to identify the individuals who were at risk of metabolic syndrome and to provide health guidance to help reduce such a risk for all residents aged 40-74 years [1]. The annual screening program was extended to those aged 75 years and above (old-old adults). As most old-old adults have been undergoing drug therapy for their chronic conditions, such as hypertension, dyslipidemia, coronary heart disease, and osteoarthritis, and have undergone screening at the time [2][3][4], we believe that the annual screening program was of poor value for the old-old adults who were already receiving treatment for such diseases. In April 2016, the Japanese government amended the health policy for old-old adults to implement a new screening program that promotes measures to avoid the worsening of lifestyle-related diseases and frailty in each municipality [5]. Frailty is a state in which the standby capacity of the body decreases because of aging; older adults with frailty are vulnerable to various stresses, and their degree of independence in functioning is 2 of 10 reduced [6]. As advanced frailty interferes with independence in performing the activities of daily living and increases the risk of disability, it is necessary for older adults to assess their health conditions comprehensively, to avoid frailty and worsening chronic diseases.
In April 2020, the national government formulated a new health assessment questionnaire for the annual screening program of old-old adults and asked each municipality to use it (Table 1) [7]. Note: The scores shown in Satake and Arai [7] were modified by the authors.
This questionnaire comprises 15 items, including two items on general health status, such as self-rated health and life satisfaction, 12 items on multidimensional aspects of frailty, such as physical function, nutritional status, oral function, cognitive function, and social participation, and one item on smoking habits. Because this questionnaire is aimed at providing a comprehensive assessment of the health status of older adults using each of the 15 items individually, the expert committee that developed this questionnaire did not initially envision that the items would be scored to identify the older adults with health risks [8]. However, some of the public health nurses in charge of community health expressed demands to assess the priority of the need for health guidance among the community-dwelling old-old adults by calculating the total score on this questionnaire.
A recent study examined the construct validity of the questionnaire using all 15 items [9]. This study had a few limitations. First, the authors assumed six factors were extracted from the explanatory factor analysis (EFA), including nine with factor loadings <0.4, which were rather small to the point that they should have been excluded [10]. Second, because the authors did not show the results of the tests for the statistical significance of the factor loadings in the confirmatory factor analysis (CFA), we were unable to statistically and logically interpret as independent the meanings of the sub-factors shown in the study. As the health assessment questionnaire for old-old adults was not developed with the intention of using all of the 15 items as a rating scale in the first place, we believe that the construct validity of the questionnaire does not need to be examined using all of the 15 items and should have been evaluated in a statistically appropriate manner. The current study aimed to re-examine the construct validity of this questionnaire and discuss its applicability as a comprehensive health assessment scale for old-old adults.

Study Design and Participants
The data used in this study were drawn from a mail-in survey conducted in August and September 2020 in the Septuagenarians, Octogenarians, Nonagenarians Investigation with Centenarians (SONIC) study [11]. The SONIC study used a narrow age-range cohort design: all of the residents in four areas (urban and nonurban) in the Tokyo Metropolitan and Hyogo Prefectures; aged 69-71 years (70s group); 79-81 years (80s group); and 89-91 years (90s group) in the initial survey, were recruited for participation in an in-venue survey. The follow-up with the participants of the in-venue survey was conducted once every 3 or 4 years. The 70s group was recruited for the initial survey in 2010. The additional participants were recruited during the first follow-up survey conducted in 2013. The 80s group was recruited for the initial survey in 2011, and additional participants were recruited during the first follow-up survey in 2014. The 90s group was recruited for the initial survey in 2012, and additional participants were recruited during the first and second follow-up surveys, conducted in 2015 and 2018, respectively. The total number of participants at the time of the baseline survey was 3346 (1592 men and 1754 women; 70s = 1129; 80s = 1234; 90s = 883). For the present mail-in survey, a letter of request and questionnaire were sent via postal mail to 2649 people (1230 men and 1419 women; 70s = 1053; 80s = 961; 90s = 635), excluding 693 dropouts from the original list of 3346 participants, due to death, institutionalization, falling in a state requiring nursing care, unknown new address, refusal to participate further, and other reasons.

The Health Assessment Questionnaire for Old-Old Adults
A health assessment questionnaire for the old-old adults (Table 1) [7] was used in this study. Although this was self-administered, when the respondents could not respond by themselves or answer the questions, a family member or a third person who knew them was allowed to fill it on their behalf. The responses to the individual items were converted into scores in the following manner. For Question 1, a score scale of 0 "excellent" to 4 points "very poor" in 1-point increments was used for each option. For Question 2, a score scale of 0 "satisfied" to 3 points "dissatisfied" in 1-point increments was used for each option. For Question 12, "Yes" was scored 2, "No" was scored 0, and "I quit" was scored 1. For questions 3, 9, 13, 14, and 15, "Yes" was scored 0, and "No" was scored 1. For questions 4-8, 10, and 11, "Yes" was scored 1, and "No" was 0.

Basic Characteristics of the Participants
To identify the basic characteristics of the participants, we collected data on sex, age, educational attainment, living arrangements, instrumental activities of daily living (IADL), and eligibility for public long-term care (LTC) insurance benefits. The items of the IADL included being able to use public transportation by oneself, shop for daily necessities, prepare meals, pay bills, and manage one's own bank account [12]. For each item, a score of 1 was assigned for "yes" and 0 for "no", and the total score for the five items was calculated. The higher the score, the more independent the participant was in the IADL. Regarding eligibility for LTC insurance benefits, all of the residents aged ≥40 years in Japan are mandatorily enrolled in the public LTC insurance system. The municipality assesses whether an insured person is eligible for insurance benefits based on an evaluation of the person's current physical and cognitive status [13]. When an insured person is certified as having LTC needs by the local government, they are categorized into either two or five levels of support or LTC needs (Levels 1 to 5).

Statistical Analysis
The factor structure of the health assessment questionnaire items for the old-old adults was examined using EFA and CFA. The principal factor method was used for factor extraction from the 15 items of the questionnaire, with Promax rotation in the EFA. We used both the Kaiser-Guttman test and the scree test to determine the number of factors to be extracted in the EFA [14]. We decided that any item with a factor loading of ≥0.4 was interpretable for each factor in the EFA [10]. Because the health assessment questionnaire for old-old adults includes 12 items to assess the multidimensional aspects of frailty, such as physical, cognitive, and social aspects, we then used CFA to examine whether the 12 frailty-related items in this questionnaire fit the multidimensional model of frailty [15]. Recent review articles have described that an assessment tool for frailty should include physical, psychological, cognitive, and social dimensions [16,17], and the physical domain includes three factors: physical function, nutritional status [15], and oral health status [18]. We assume that, behind these five frailty-related domains, there is a concept of multidimensional frailty that affects each of the five domains. Therefore, we assumed that the model of frailty for this CFA had five sub-factors as the primary factors: physical function (Q7, Q8, Q9, and Q13); nutritional status (Q3 and Q6); oral function (Q4 and Q5); cognitive function (Q10 and Q11); and social aspects (Q14 and Q15); with the secondary factor of "a comprehensive concept of frailty" on top of these five sub-factors. In the CFA, using structural equation modeling (SEM), the parameter estimates in the model were led by the maximum likelihood method [19], using only 12 frailty-related items. The χ 2 score, goodness of fit index (GFI), adjusted GFI (AGFI), root mean square error of approximation (RMSEA), and Akaike's information criterion (AIC) were used to assess the model's goodness of fit for the CFA. We also calculated the Cronbach's alpha coefficient using these 12 items. A two-tailed p-value < 0.05 was considered statistically significant in the CFA. IBM SPSS Statistics ver.25 was used for the fundamental statistical analysis and EFA, and IBM SPSS Amos ver.25 was used for the CFA.

Characteristics of the Participants
Among the 2653 people, 1783 responded to the questionnaire (response rate, 67.2%). A total of 1576 (88.4%) respondents who provided answers to all 15 items of the health assessment questionnaire were included in the analysis. The respondents' basic characteristics are listed in Table 2. The men accounted for 47.1% of the total study population. The mean age was 85.6 years (standard deviation 5.9), and the age range was 78-99 years. Of the total, 87.8% responded on their own, 6.5% responded through a proxy, 1.9% relied on a third person, and 3.9% had an unknown response status. Table 3 shows the mean values for each item stratified by sex and age group. First, to investigate the factor structure of the health assessment questionnaire for old-old adults, we conducted an EFA using the 15 items identified using the principal factor method. Based on the Kaiser-Guttman test, we extracted five factors with eigenvalues of one or more. Next, the eigenvalues of the identified factors were plotted on a scree plot (Figure 1). Because the shape of the scree plot showed a smooth decrease after the fifth factor, we determined that there were five factors in this EFA. factor method. Based on the Kaiser-Guttman test, we extracted five factors with eigenvalues of one or more. Next, the eigenvalues of the identified factors were plotted on a scree plot (Figure 1). Because the shape of the scree plot showed a smooth decrease after the fifth factor, we determined that there were five factors in this EFA. The proportion of variance accounted for by these five factors was 25.1%. The principal factor method was used to identify the factor loadings of each item by setting the number of factors to five. Promax rotation was performed to obtain the factor pattern matrix. Table 4 shows the factor loading of each item after factor rotation, and the estimated communality after factor identification. The factor loadings were interpretable with respect to the items of both Factor 1 (general health assessment: self-rated health status; life satisfaction) and Factor 4 (cognitive function: always asking the same thing; not remembering today's date). However, the other factors were difficult to interpret because Factor 3 included various aspects, such as oral function, weight loss, and physical function, and The proportion of variance accounted for by these five factors was 25.1%. The principal factor method was used to identify the factor loadings of each item by setting the number of factors to five. Promax rotation was performed to obtain the factor pattern matrix. Table 4 shows the factor loading of each item after factor rotation, and the estimated communality after factor identification. The factor loadings were interpretable with respect to the items of both Factor 1 (general health assessment: self-rated health status; life satisfaction) and Factor 4 (cognitive function: always asking the same thing; not remembering today's date). However, the other factors were difficult to interpret because Factor 3 included various aspects, such as oral function, weight loss, and physical function, and Factor 5 included social relationships, nutritional status, and smoking habits. In addition, Factor 2 was difficult to interpret because there was only one item with a factor loading of ≥0.4. Although the estimated communality of items Q1 and Q2 were the highest, at 0.72 and 0.40, respectively, the communality of the eight items was approximately 0.2. Specifically, the estimated communality of item Q12 "smoking habits" was the lowest (0.02). These results indicate that the questionnaire with 15 items was not explained by the five factors, and the EFA using 15 items did not extract interpretable factor structures. Table 4. Factor loadings and estimated communality in the EFA using the principal factor method with Promax rotation of the health assessment questionnaire for old-old adults (15 items

CFA
The next step was the CFA, using only 12 frailty-related items. These items were developed as representatives of the five frailty-related domains: physical function (Q7, Q8, Q9, and Q13); nutritional status (Q3 and Q6); oral function (Q4 and Q5); cognitive function (Q10 and Q11); and social aspects (Q14 and Q15). CFA assumed that the model had a higher-order factor (frailty) above these five frailty sub-factors. Figure 2 shows that all of the estimated factor loadings on the 12 items were 0.34 or more, except for one item (Q 15), and all of them were statistically significant (p < 0.05).
Each item had a factor loading at almost the same level as initially assumed. The factor loadings of the five sub-factors from the higher-order factor were sufficiently high (i.e., 0.46 or more), and all of them were statistically significant (p < 0.05). The goodness of fit of the model was sufficiently, with χ 2 (49) = 156.2 (p < 0.001), with GFI, AGFI, RMSEA, AIC being 0.983, 0.973, 0.037, and 214.2, respectively. The Cronbach's alpha coefficient for the 12 items was 0.569. The next step was the CFA, using only 12 frailty-related items. These items were developed as representatives of the five frailty-related domains: physical function (Q7, Q8, Q9, and Q13); nutritional status (Q3 and Q6); oral function (Q4 and Q5); cognitive function (Q10 and Q11); and social aspects (Q14 and Q15). CFA assumed that the model had a higher-order factor (frailty) above these five frailty sub-factors. Figure 2 shows that all of the estimated factor loadings on the 12 items were 0.34 or more, except for one item (Q 15), and all of them were statistically significant (p < 0.05).

Discussion
The responses to the new health assessment questionnaire for old-old adults were obtained from community-dwelling older adults through a mail-in survey that covered the participants in the SONIC study. The factor structure of the questionnaire was then examined. The EFA revealed that an interpretable factor structure was not obtained through the analysis of all 15 items. In contrast, the results from the CFA showed that there were five sub-factors: physical function; nutritional status; oral function; cognitive function; and social aspects, indicating the goodness of fit of the model with the higher-order factor of "frailty" on top of these five sub-factors. In addition, the goodness of fit of the model that assumes frailty as the secondary factor may indicate that the total score of the 12 frailtyrelated items in the health assessment questionnaire for old-old adults can be used as an indicator of the degree of "frailty".
This study indicated that the total score of the 12 frailty-related items in the questionnaire showed the possibility of screening older adults with frailty. This questionnaire comprehensively covered the physical, cognitive, psychological, and social aspects of frailty. Although there are several validated frailty measurement tools [20], most of them have focused on the physical aspects of frailty. In addition to this questionnaire, only some comprehensively assessed the physical, psychological, and social aspects of frailty, such as the Groningen Frailty Indicator [21][22][23] from Germany, the Edmonton Frail Scale [24,25] from Canada, the Tilburg Frailty Indicator [26][27][28] from The Netherlands, and the Kihon Checklist [29,30] and Kaigo Yobo Checklist [31] from Japan. These comprehensive frailty assessment instruments were reported to examine convergent validity and/or concurrent validity and to compare their predictive validity: a comparison of predictive validity for disability between the Groningen Frailty Indicator and the Tilburg Frailty Indicator [32]; a comparison of predictive value for identifying frailty between the Groningen Frailty Indicator and the Tilburg Frailty Indicator [33]; an examination of the convergent validity between the electronic frailty index and the Edmonton Frail Scale [34]; a comparison of diagnostic test accuracy for identifying frailty among the Groningen Frailty Indicator, the Edmonton Frail Scale, and the Kihon Checklist [35]; a comparison of predictive value for frailty between the Kihon Checklist and the Kaigo Yobo Checklist [36]; an examination of convergent validity between the Kihon Checklist (Portuguese version) and the Edmonton Frail Scale [37].
Compared with the two existing Japanese checklists for comprehensive frailty risk assessment, the health assessment questionnaire for the old-old adults has at least two unique features. First, the questionnaire was used across Japan: 91.9% (1600/1741) of the municipalities used it in 2021 [38], and 93.9% (1503/1600) of them used it in the course of the annual screening program for old-old adults. The national government expects all of the municipalities to use the questionnaire within a few years. Second, the data from the questionnaire are routinely registered in the national health insurance database system, known as "the KDB system" [39], which links the data from the national screening program and the claims data from health and LTC insurance on an individual basis. Using the data registered in the KDB system, such as blood test data at the time of the screening program, and treatment and prescription status, each municipality can provide health guidance to those at risk of frailty to prevent them from becoming frail.
We limited CFA to 12 frailty-related items out of the 15 items in the questionnaire and excluded the other 3 (subjective health, life satisfaction, and smoking habits). However, these 3 items, and the other 12, are important for the sake of the comprehensive health assessment of old-old adults living in the community. As is true for all of the items in the questionnaire for old-old adults, it is not appropriate in health guidance for them to focus only on the items with unfavorable responses and to focus too heavily on guidance for the shift to favorable health behaviors. What is important is for the health guidance instructor to acknowledge that there are favorable health behaviors that are currently being implemented and support their continuation [5]. For the items that received unfavorable responses, it is important to consider the background and causes for such unfavorable conditions, while taking into account the subject's injuries and psychosocial conditions, and to provide health guidance that addresses the issues while taking care not to cause physical and/or cognitive impairment.
This study has at least two limitations. First, the alpha coefficient of the total score of the 12 items in the health assessment questionnaire was relatively low at 0.57, indicating a low internal consistency. We consider the reason for the low internal consistency to be because the five domains measure different aspects of frailty, which are likely to be diverse and have high variance, namely physical, oral, nutritional, cognitive, and social aspects. Therefore, it is within our expectation that the within-subject consistency among the 12 items was limited to some extent. Second, because the analyzable subjects in this study were survivors of the SONIC study, they may be healthier and have better health behaviors than the general population who participate in the annual health screening program in Japan. We believe that the construct validity of this questionnaire needs to be re-examined using the questionnaire data registered in the KDB system, which are from old-old adults who underwent health screening programs in their communities.

Conclusions
We propose that the total score of the 12 frailty-related items in the health assessment questionnaire for old-old adults can be used as an indicator of the degree of "frailty". We expect that health professionals working in municipalities will be able to use the total score of the 12 frailty-related items out of the 15 items of the questionnaire to identify those who are eligible for health guidance aimed at preventing frailty among community-dwelling older adults. Future research should examine the criterion validity of the health assessment questionnaire for old-old adults, and the extent to which this questionnaire correlates with actual frailty status among community-dwelling old-old adults.