Next Article in Journal
High Refractive Index Electromagnetic Devices in Printed Technology Based on Glide-Symmetric Periodic Structures
Previous Article in Journal
An Agent-Based Simulation to Minimize Losses during a Terrorist Attack
Previous Article in Special Issue
Identification of Breast Malignancy by Marker-Controlled Watershed Transformation and Hybrid Feature Set for Healthcare
 
 
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

Evaluation of Quality and Readability of Online Health Information on High Blood Pressure Using DISCERN and Flesch-Kincaid Tools

1
Department of Computer Software Engineering, University of Engineering and Technology Mardan, Mardan 23200, Pakistan
2
Department of Electrical Engineering, City University of Science & Information Technology Peshawar, Peshawar 25000, Pakistan
3
Department of Computer Science, University of Engineering and Technology Mardan, Mardan 23200, Pakistan
4
Department of Electrical Engineering, University of Engineering and Technology Mardan, Mardan 23200, Pakistan
5
Health Department, Government of Khyber Pakhtunkhwa, Malakand 23050, Pakistan
6
Electrical Engineering Department, College of Engineering, Najran University Saudi Arabia, Najran 61441, Saudi Arabia
7
Department of Automatic, Control and Robotics, AGH University of Science and Technology, 30-059 Kraków, Poland
*
Authors to whom correspondence should be addressed.
Appl. Sci. 2020, 10(9), 3214; https://doi.org/10.3390/app10093214
Submission received: 28 March 2020 / Revised: 28 April 2020 / Accepted: 29 April 2020 / Published: 5 May 2020
(This article belongs to the Special Issue Recent Developments in Smart Healthcare)

Abstract

:
High Blood Pressure (BP) is a vital factor in the development of cardiovascular diseases worldwide. For more than a decade now, patients search for quality and easy-to-read Online Health Information (OHI) for symptoms, preventions, therapy and other medical conditions. In this paper, we evaluate the quality and readability of OHI about high BP. In order that the first 20 clicks of three top-rated search engines have been used to collect the pertinent data. Using the exclusion criteria, 25 unique websites are selected for evaluation. The quality of all included links is evaluated through DISCERN checklist, a questionnaire for assessing the quality of written information for a health problem. To enhance the reliability of evaluation, all links are separately assessed by two different groups—a group of Health Professional (HPs) and a group of Lay Subjects (LS). A readability test is performed using Flesch-Kincaid tool. Fleiss’ kappa has been calculated before considering average value of each group. After evaluation, the average DISCERN value of HPs is 49.43 ± 14.0 (fair quality) while for LS, it is 48.7 ± 12.2; the mean Flesch-Reading Ease Score (FRES) is 58.5 ± 11.1, which is fairly difficult to read and the Average Grade Level (AGL) is 8.8 ± 1.9. None of the websites scored more than 73 (90%). In both groups, only 4 (16%) websites achieved DISCERN score over 80%. Mann-Whitney and Cronbach’s alpha have been computed to check the statistical significance of the difference between two groups and internal consistency of DISCERN checklist, respectively. Normality and homoscedasticity tests have been performed to check the distribution of scores of both evaluating groups. In both groups, information category websites achieved high DISCERN score but their readability level is worse. Highest scoring websites have clear aim, succinct source and high quality of information on treatment options. High BP is a pervasive disease, yet most of the websites did not produce precise or high-quality information on treatment options.

1. Introduction

1.1. Motivation and Background

Blood pressure (BP) is the pressure exerted by blood against the artery wall during its circulation. The normal level of BP in a healthy person is 140/90 mm Hg [1]. High BP is a prevalent chronic disease; when BP exceeds than normal level, results in detriment to the body’s essential organs [2]. High BP which is also termed as hypertension is a major disease with a pervasiveness of 40.8 percent and control rate of 32.3 percent worldwide [3]. High BP is considered the most portentous factor in the development of cardiac diseases. The high pervasiveness of high BP makes it a significant factor for mortality and morbidity in both developed and developing countries. Due to its high mortality rates and lack of early symptoms, it is also known as a major chronic, a ‘silent killer’ and non-communicable disease [4]. Research shows that more than nine million deaths are associated with the high BP complications; and 45% of them are deaths which is caused due to coronary artery disease, while the death of 51% are caused by stroke [5].
Presently, around one billion individuals suffer from high BP worldwide and the number is expected to surge up to 1.56 billion by 2025 [6]. Nowadays, the use of Information and Communication Technologies (ICT) is rapidly growing in medical education [5]. Internet world statistics [7] show that web users are increasing day by day. As of June 2014, there were 3.035 billion Internet users around the globe and the number reached to 4.208 billion in June 2018, representing 55.1% of the World’s population. Every day, thousands of users visit the Internet to get health-related information [5]. Health clients use Internet as the first choice to get instant information about their health issues [8]. According to the latest statistics [9], approximately 7% of Google’s daily searches are related to health, which is equivalent to 70,000 per minute. Studies show that people up to 80% use Internet for treatment purposes [10]. Nowadays, more than 60% of people in Europe and 80% people in the US [11] use online health resources for their health-related issues and among them 90% people are convinced that Internet has improved their health-related knowledge [12]. Furthermore, physicians motivate patients to search for material related to their medical condition [13]. Studies show that the primary reason to use online health resources are to verify the information from the physicians and to get answers to questions in their mind and find out alternative treatment options [14]. In a study by the US physician showed that 85% of patients visit the hospital because of the information perceived through Internet [15].
Similarly, the authors of Reference [16] investigated that people search for OHI and trust the information. However, to reduce potential risks that are associated with online-health content and to facilitate health professionals, patients and health seekers to access precise, complete and accurate information, various quality tools are designed to assess the quality of OHI [17]. DISCERN instrument is one of the tool that is widely used for evaluation of OHI [18]. In addition to quality of the information, readability also plays a key role in comprehending written information effectively. It is the metric that a reader can use to understand a line [19]. Previous studies show that although some websites have good quality of information but their content is complex for most of the Internet users [19]. In addition, some web-based material for educating patients is written at above than recommendation grade [20,21]. Some of them are written at grade level-15 [22], while others are written at grade level-17 [23]. There are many tools, such as Flesch-Kincaid, Gunning fog index, Simple Measure of Gobbledygook (SMOG) and so forth, for readability assessment. These tools measure the complexity level of any written information.

1.2. Literature Review

Studies show that seventy percent of the research studies related to the evaluation of websites providing health information conclude that the quality of such websites is a major concern [24]. Various studies have been conducted for the assessment of OHI on different medical conditions such as lupus erythematosus [10], diverticulitis [25], breast cancer treatment [26], childhood epilepsy [27], endoscopic retrograde cholangio pancreatography procedure (ERCP) [28], anxiety disorder [29], cataracts [30] and childhood depression [31]. In Reference [10], the authors search the term ‘systemic lupus erythematosus’ through Google, Bing and Yahoo and collect first 25 links in each search engine and assess only 25 unique websites (obtained through exclusion criteria) by evaluators. All websites are rated (No: does not fulfil criterion (1 point) to Yes: fulfils criterion (5 points)) through 16 questions of DISCERN checklist which has maximum score of 80. The authors find that none of the websites is categorized as ‘excellent website’ because none achieve a DISCERN score above 66. Among the 26 websites, 6 websites are poor and the content of 2 websites is very poor. Similarly, the authors of Reference [27] investigate the term ‘Childhood Epilepsy’ through Google and only 42 links are included in the study after removing advertisements, health forums, duplicated websites and personal experience websites. The assessment is carried out by two different groups through DISCERN tool. Hence, only 9.6% of the websites are of good quality while 26.2% are marked unreliable. Furthermore, 42.8% of the websites are unable to expatiate treatment choices effectively, whereas only 7 out of 42 websites provide clear information and access to additional sources. Furthermore, the authors conclude that source attribution in all websites is poor. The quality of 80% websites has serious deficiencies [32]. In study of ERCP, the authors examine the search term ‘ERCP’ in top three search engines [28]. A total of 60 websites are retrieved and 24 unique websites (excluding the websites with visual content, websites with completely scientific articles at advanced level or those which did not contain relevant information) are selected for study. Authors demonstrate that the websites are unable to provide proof of reliable source of information and the date on which the information is described. Furthermore, none of the websites achieve 5 out of 5 rating in all indicators of DISCERN. The included websites also fail to describe the effect for health consumers and whether the additional treatment information is available or not. In Reference [29], Google, Yahoo and MSN search engines yield a total of 540 websites for anxiety disorder, only 67 websites are found eligible for the study. All the websites are categorized as per its affiliation. The result shows that 29 out of 67 websites provide poor quality of information, 15 websites are of good quality and only 3 websites produce excellent quality of information on anxiety disorder.
In References [10,25,26,27,28,29,33,34,35], it is found that inconsistent, unreliable and unauthentic information co-exist with actual information. Since the Internet has a great possible impact on the treatment decisions of patients; it is, therefore, very significant that patients access accurate and reliable OHI to bring down the risks of Internet-based detrimental clinical decisions [36]. In addition, studies have repeatedly showed that OHI are being developed to a degree that is too complex for most web users [19].
Therefore, this paper focuses on evaluating the quality and readability of OHI related to high BP.

1.3. Paper Contributions

High BP is a common disease and is a worldwide leading cause of premature death. According to the statistics issued by World Health Organization (WHO) in 2015, every fourth man and every fifth woman suffer from hypertension [37]. Nowadays, people are more health conscious and usually use Internet for health-related issues before and after consultation with health professionals. So it is very important to evaluate the quality of information on high BP available on the World Wide Web (WWW). Therefore, in this paper we will evaluate the quality of OHI about high BP by using the DISCERN tool.
Good quality of information does not guarantee its readability. Some websites providing quality information about OHI are not easy to read/understand. The authors of Reference [33] found that Government websites providing OHI are more readable whereas the readability level of non-profit category is extremely technical (not easily understandable by common man) [33]. Hence, in this paper, we determine readability test via Flesch-Reading Ease and Flesch-Kincaid tools. In this paper, we identify websites associated with high BP which has high quality of information based on standard criterion. In addition, we also highlight those websites which described benefits of each treatment, risks of different treatments and sources on which the publication is based. Similarly, we mark those websites which has low quality written information about high BP. Furthermore, we examine the readability level of all included websites.

1.4. Paper Organization

The rest of the paper is organized as follows—Section 2 shows methodology of our work. Section 3 presents the results and its discussion in terms of quality and readability. Section 4 shows the strengths and shortcomings of our work. Section 5 concludes the paper.

2. Methodology

2.1. Selection of Search Engines and Web Browser

In this study, we use the most widely used search engines, that is, Google, Yahoo and Bing [28]. At the time of our research study, based on usage of all kinds of electronic devices, the topmost search engines around the globe are Google (77%), Baidu (14%) and Bing (4%). Since the second commonly used search engine is available only in Chinese language, it is not applicable for this study. As such the fourth most commonly used search engine, Yahoo (2.48%), is selected. All searches were performed through Google Chrome version 69.0.3497.81 (for 64-bit OS). Language settings and location was set to default mode and the browser is set to incognito mode to avoid personalized results.

2.2. Websites Selection for the Study

After repeated consultation with the health professionals (HPs) and lay subjects (LS) and Internet users the keyword ‘high blood pressure’ is searched separately in the three already mentioned search engines to obtain relevant information on high BP. Most of the people view their search results to a maximum of first 20 hits [33]. Study shows that more than 91% do not move to next page [38]. Similarly, the study on health-specific Internet search behaviour shows that 97.2% clicks are based on first 10 links [39].
Therefore, we limited our search to the first 20 hits from each search engine, so a total of 60 search results are rigorously reviewed. During this process, advertisements and sponsored links were ignored. Similarly, websites with irrelevant, non-English content, those asking for payment, completely scientific based articles and videos links are excluded from the list. Furthermore, identical websites are also filtered out. As a result of this filtering process, 25 unique websites are chosen for analysis (See Table A1 in Appendix A). The workflow of methodology is shown in Figure 1.
In order to check the quality level of websites, all 25 websites have been split into their respective categories—Government, Information, Institution and Non-profit organization. Most of the websites are from information portal as shown in Table 1.

2.3. Quality Assessment Instrument

The quality of all distinct websites is separately evaluated through widely used DISCERN instrument. This tool is used for the assessment of written health care information about health diseases [10,25,26,27,28,29,33]. It consists of short questionnaire that can be used by HPs, patients and LS. The DISCERN checklist consists of sixteen questions which has the total score 16 (minimum)–80 (maximum), where each question is rated through an ordinal Likert scale of 1 (for No) to 5 (for Yes) [18].
This instrument has been set up in three parts; the first part of the DISCERN questionnaire (Questions 1 to 8) is about the reliability of publication—which consists of aim of the publication, source of information, relevance of publication, additional source of information about the disease and so forth. The second part (Questions 9 to 15) concerns the description of treatment. For example, how a specific treatment works, what are the risks and benefits of a particular treatment or is there any other treatment choice for patients and so forth. The third part (Question 16) describes overall quality of the publication as a source of information for treatment options. The DISCERN instrument is used and validated in combination with a handbook [18]. A grading mechanism, as shown in Table 2, has been generated on the basis of score achieved by each website [33]. The detail of the DISCERN questionnaire is given in Table A2).

2.4. Readability Assessment

In this study, readability of the studied websites is determined using Flesch-Reading Ease Score (FRES) [29,40]. The built-in function of this tool is available in Microsoft Word 2010. FRES gives us numeric value which does not clarify the level of the grade, so in order to get grade level; we use Flesch-Kincaid Grade Level or Average Grade Level (AGL).
All content of each web page is copied and pasted in Microsoft Word 2010 separately. Each webpage is then analysed thoroughly and irrelevant content is removed. Hyperlinks such as webpage link, colons, semicolons, hyphens, decimals and abbreviations (such as ‘U.S’) are also removed [41]. Thus, we obtained the numeric value of FRES and AGL for each website as below:
F R E S = 206.835 1.015 × ( w o r d s s e n t e n c e s ) 84.6 × ( s y l l a b l e s w o r d s ) .
The range of values of FRES varies from 0 to 100 (very difficult to read to very easy to read). The content which has high FRES value means that it is easy to read [42].
The value of AGL can be obtained as follows:
A G L = 0.39 × ( w o r d s s e n t e n c e s ) + 11.8 × ( s y l l a b l e s w o r d s ) 15.59 .
FRES depends on average sentence length and average word length. The results of the above two tools are related inversely, means a text having a high FRES score should have a lower score of AGL.

2.5. Data Collection

The data for this work were extracted on 10 September 2018. Inclusion and exclusion of all the included websites was carried out between 10 and 15 September 2018, while the quality and readability assessments are performed between 15 September and 30 September 2018.

2.6. Websites Evaluation

For evaluation of all included links, two groups are created. The first group is from the medical profession, consisting of a total of five HPs, with a minimum degree in Bachelor of Medicine, Bachelor of Surgery (MBBS). All HPs are hailing from different areas of Khyber Pakhtunkhwa—Peshawar, Malakand, Swat. Similarly, the second group is also composed of five members but had different educational background, that is, engineering, natural sciences and so forth. They are also from various regions of Khyber Pakhtunkhwa. All group members are computer literate. The reason for selecting two groups with different academic background is that each member evaluate the included websites independently and then compare and correlate the results of medical group with the second group.
All evaluators were fully trained on DISCERN questionnaire. Each member in both groups has been provided all included websites along with a copy of the DISCERN manual, which explains all 16 questions with examples. Responses from the group members are collected through Google sheet which is shared with all group members. To distinguish a specific evaluation response from a group, all members have been assigned a unique ID. So, after fully understanding of DISCERN checklist and online link, all group members uploaded their DISCERN ratings for each website separately. This way, participants independently evaluated all 25 websites. Average (mean) value and standard deviation (SD) for both groups are computed. SD is a measure of dispersion, which shows that how much data is spread out from the mean of each group. If the data spreads farther, it means SD is greater. Furthermore, all distinctive 25 websites are classified into different quality levels based on the DISCERN score according to Table 2.

2.7. Data Analysis

For statistical analysis, we used IBM SPSS Statistics v-20 software. Cronbach’s α (alpha) (coefficient of reliability) is used to measure the internal consistency (reliability) of DISCERN questionnaire based on Likert scale that how closely are set of items connected. The range of coefficient of reliability (α) varies from 0 to 1. For Cronbach’s α, the value over 0.70 is minimum acceptable range [43]. Similarly, before taking average value of each group, Fleiss’ Kappa statistics has been carried out to assess the inter-rater agreement for both evaluating groups. The strength of agreement according to the Fleiss’ Kappa (k) statistics is given in Table 3 [44]:
Further, two-tailed Mann-Whitney test or Wilcoxon rank-sum test has been conducted to assess the statistical significance of the difference between two groups for each website evaluated by both groups. Similarly, for two indicators (Question 4 and Question 7) of DISCERN, Chi-squared test has been performed and compared for both groups.
Further, normality and homoscedasticity tests have been performed to check the distribution of websites scores for both groups. With not statistically significant results of these tests, we carried out independent samples t-test to assess the difference between two groups of evaluators.

3. Results

As discussed earlier, websites evaluation is accomplished by two groups. The responses from Group 1 (HPs) are compiled and results along with discussion are presented in Section 3.1, which is followed by the feedback from Group 2 (LS) and results are discussed in Section 3.2.

3.1. Group 1: Health Professional

Responses received from HPs are analysed in this section as follows:

3.1.1. High Scoring Websites

The evaluated websites are ranked based on the DISCERN score. Summary of the DISCERN score of the websites on the basis of responses obtained from HPs is given in Table A3 (Column: HPs). The DISCERN score ranges from 27.0 (being the lowest) to 72.2 (being the highest), with the average value 49.43 ± 14.0. The average DISCERN score indicates fair quality of information on high BP.

3.1.2. Quality of Included URLs

Fair to good quality is exhibited by the reviewed websites. Details of the evaluation of candidate websites by HPs are given in Table 4.
A.
DISCERN’s Part-I (Question 1–8)
In first part of DISCERN questionnaire, ‘Excellent’ results are shown by only 5 (20%) websites as can be seen from the Table 4. A good quality publication has clear aims and explains what it is about and what it covers. After assessment of Questions 1–8 of DISCERN, it is found that only 3 out of 25 websites (mentioned in Table A3: S.No.1–3) has clearly elaborated aim of publication and achieve 5 out of 5 in DISCERN points. Similarly, the websites mentioned in Table A3: S.No. 23–25 vaguely or unclearly describe aim of publication. The content of the two websites (mentioned in Table A3: S.No. 3–4) attain 4.8 and 4.7 points out of 5, respectively, in terms of relevance of publication. Thus, overall 10 (40%) of the websites demonstrate ‘fair’ evaluation on reliability of publication (Part- I) and 4 (16%) websites are established unreliable during the assessment period as is evident from Table 4. Six websites are unable to produce the date of publication and dates of main sources of evidences. While 11 websites have sufficiently produced date of publication and achieved 3–4 points in DISCERN ratings.
B.
DISCERN’s Part-II (Question 9–15)
It is observed from Table 4 Part-II that 5 (20%) websites had an excellent quality of information on treatment choices. Six websites have thoroughly explained each treatment which is useful for high BP. According to HPs, these websites have scored 4 to 5 out of 5 points of DISCERN. On treatment options, the quality of 7 (28%) websites is substandard. Furthermore, six websites had not described the risks or disadvantages of particular treatment, although it is very essential to be conscious of the risks in order to understand what to expect from the treatment and to be able to make further decisions. Quality of 9 (36%) websites is fair. Eight websites did not properly describe that what would happen if no treatment is used. According to HPs, it is the most dominant question because a publication of good quality will always include a description of what would happen if the condition remains untreated.
C.
DISCERN’s Part-III (Question 16)
From Question 16, overall quality of the website is assessed. As is apparent from Table 4, only 5 (20%) websites (out of 25) have ‘excellent’ quality and have produced appropriate sources of information on high BP. The overall quality of 5 (20%) websites is poor. Similarly, 2 (8%) websites are deficient in content.

3.1.3. DISCERN Indicator: Clarity of Sources

One of the important and essential indicators for reliability of health websites is clarity of sources which mean identification of the source to compile the publication other than the originator. It may be an expert opinion, a reference or any other evidence. The DISCERN rating for Clarity of sources is shown in Figure 2. According to health experts, as DISCERN cannot be used to tell whether the article is true or not, so via this indicator, users can check it easily against some other credible source. Through DISCERN indicator ‘clarity of sources,’ it is found that 9 out of 25 websites had no evidences about their publications. Only 3 (12%) websites provided clear sources of information about high BP which had obtained maximum score, that is 5 out of 5, for this indicator. 13 out of 25 websites provided somehow reference for this indicator.

3.1.4. DISCERN Indicator: Additional Source of Information

The indicator for additional source of information searches recommendation from other organization to provide further information about the treatment options, conditions or additional information about the disease. It can be seen from Figure 3 that 40% of the websites provided satisfactory additional sources of information on high BP whereas 16% of the websites were unable to provide additional information on high BP.
Amongst the 25 websites, only 4 (having overall high DISCERN score) have provided details of the DISCERN indicator additional sources of information on high BP. For further studies, it is essential to have detail of other support sources about treatment choices. Websites with high DISCERN score produce adequate additional information on high BP.

3.2. Group 2: Lay Subjects

Responses received from LS are analysed in this section as follows:

3.2.1. High Scoring Websites

As per assessment of Group 2 (Lay subjects), all websites are marked according to their DISCERN score. The range of DISCERN scores varies from 25.2 (the lowest score; Table A3: S.No. 25 (LS)) to 69.6 (the highest score; Table A3: S.No. 5 (LS)) with average 48.7 ± 12.2 (fair quality). The average score of LS is close to the average score of HPs group (49.43). The average value indicates that the overall quality of websites is ‘fair.’ DISCERN score of all websites are given in Table A3.

3.2.2. Quality of Included URLs

Fair to Good quality is exhibited by the reviewed websites. Detail analysis of the evaluation of candidate websites by Group 2 (LS) is given in Table 5.
A.
DISCERN’s Part-I (Questions 1–8)
After further dissecting the results of DISCERN questionnaire by LS, it is found that, only 5 out of 25 websites have produced reliable publication as shown in Table 5. On analysing the first eight questions of DISCERN, it is known that seven websites (mentioned in Table A3: S.No. 13, S.No. 15, S.No. 16, S.No. 18–19, S.No. 24–25) achieved the goal of the publication of the DISCERN questionnaire and four websites (mentioned in Table A3: S.No. 16, S.No. 23, S.No. 24, S.No. 25) are unreliable in terms of content. According to assessment of LS through DISCERN Question (8), there are ten such websites (mentioned in Table A3: S.No. 8, S.No. 14–16, S.No. 19–20, S.No. 22–25) which are either uncertain or the information is unclear or incomplete. Only one website (mentioned in Table A3: S.No. 5) has produced publication with clear reference to ‘grey’ areas for the uncertainty about effective treatment. 7 out of 25 websites failed to produce the date of publication and dates of main sources of evidences and only one website (mentioned in Table A3: S.No. 5) achieved 5 out of 5 points by publishing dates for every source or reference.
B.
DISCERN’s Part-II (Questions 9–15):
Only 3 (12%) websites provide excellent content with information on treatment choices, whereas 8 out of 25 websites have poor content on treatment options for high BP as shown in Table 5. Good publication always describes the benefits of each treatment. It is found from analysis of Questions 9–15 that 3 out of 25 websites (mentioned in Table A3: S.No. 20, S.No. 22, S.No. 25) has not discussed benefits of any of the treatment and three websites (mentioned in Table A3: S.No. 1–2, S.No. 5) described the benefits of each treatment effectively. It is also found that two websites (mentioned in Table A3: S.No. 23, S.No. 25) achieved 1 out of 5 DISCERN points by not discussing what would happen if no treatment is used for high BP. For support shared decision-making, only one website (mentioned in Table A3: S.No. 25) does not provided any information.
C.
DISCERN’s Part-III (Question 16):
Similarly, if we look at Question 16, which is related to the overall quality of the websites, we can see that only 6 websites provide excellent quality of information about high BP as shown in Table 5. Analysis shows that overall 8 out of 25 websites produce poor quality of information about the disease.

3.2.3. DISCERN Indicator: Clarity of Sources

After analysis of the data assessed by LS, it was found that only one website (4%) shows references at the end of publication which show that the author has produced clear evidence of its publication and is thus highly rated for the DISCERN indicator of ‘clarity of sources.’ Seven websites (28%) partially show their sources of information and 10 websites (40%) have a poor rating for this indicator because they have not shown any link, sources or any other references to confirm authentication. The statistics are summarized in Figure 4.

3.2.4. DISCERN Indicator: Additional Source of Information

After the assessment of websites by Group 2 (LS), it was found that only two websites (out of 25) have not discussed additional sources of information about high BP and hence achieved a poor score in the DISCERN indicator ‘Additional source of information.’ The remaining websites have shown an additional source of information. The findings are summarized in Figure 5.

3.3. Results Comparison of Both Groups (HPs & LS)

In this section we compare the analysis of Group 1 and Group 2 to evaluate the two-group results. Section wise comparison of the DISCERN questionnaire by HPs and LSs is shown in Figure 6. It is clear from the figure that the assessments of part-I and part-II of the DISCERN questionnaire in both groups are closely related compared to a slightly greater difference in part-III of the questionnaire where HPs (Group 1) achieved 60.4% and LS (Group 2) achieved 64.4% average DISCERN scores.

3.4. Website’s Category Wise Comparison of Both Groups (HPs and LS)

Figure 7 shows the performance of each category of website by both groups. It is obvious from the figure that the content of the Information category websites is highly rated by both groups and thus achieved the highest DISCERN score. The number of websites from Information category is also highest which is 9. It means that the Information category websites have highly visibility rate due to their good quality of information on health issues. Similarly, non-profit category websites failed to provide clear aims, citations, treatment options and any additional information about high BP. Consequently, the content of the Non-profit category is marked the lowest in both groups. The performance of Government and Institution category websites is average.

3.5. Statistical Analysis

Before considering the average value of all evaluated websites, we performed Fleiss’ kappa (k) analysis for both groups to assess the five raters’ scores for all websites. The average values of Fleiss Kappa (k) for HPs and LS are 0.3912 and 0.3421, respectively. According to Table 3, the strength of agreement amongst both groups is ‘fair.’ Kappa (k) values for each website are shown in Table A1 in Appendix A (Column: HPs & LS). As there are five measurements per group, so the calculated values of Kappa (k) show good consistency among the evaluators of each group. Further, to measure the internal consistency of the DISCERN questionnaire, we calculated Cronbach’s α for both evaluating groups. After measuring all questions of DISCERN by each evaluator in HP group, the value of α is 0.876 which shows higher internal consistency. Similarly, the value of α for LS is 0.783 which also reflects a good range for consistency of LS group. Both values of α show that the tool is reliable for evaluation.
Similarly, Wilcoxon-rank sum test has been used to compute the value of p at significance level (p < 0.05) for each website evaluated by both groups. The values are shown in last column in the table given on Table A3. Furthermore, the Chi-squared test has been calculated for the DISCERN indicator regarding clarity of sources (Question 4 of DISCERN questionnaire given in Table A2). For HPs, the value of Chi-squared is 0.199148 whereas for LS, it is 0.014612 for statistical significance level (p < 0.05). Similarly, the same Chi-squared test has been carried out for additional source of information (Question 7 of DISCERN questionnaire given in Table A2). So, the value of Chi-Square is 0.024406 for HPs while for LS it is 0.005135 at p < 0.05.
Normality and homoscedasticity tests were performed to check whether the scores are normally distributed for both groups of evaluators. After performing these tests, it is found from Shapiro-Wilk’s test (p > 0.05) [45,46], visual inspection of their histograms and box plots that all the websites scores were normally distributed for both groups. For HPs, the value of skewness is −0.078 (Standard Error (SE) = 0.464) and a kurtosis is −1.448 (SE = 0.902). Whereas for LS, the value of skewness is 0.197 (SE = 0.481) and a kurtosis is −0.943 (SE = 0.935) [47]. Similarly, after examining the Q-Q plots of both groups, it has been observed that all the dots were along the line which indicates that the data are approximately normally distributed. Further, the analysis of both groups showed that they are homoscedastic because they have the same level of variance.
After confirming normality and homoscedasticity, subsequently, we performed independent samples t-test to find out t-values for the statistical differences of DISCERN questionnaire assessed by both groups. The Independent Samples t-test is a parametric test that compares the means of two independent groups in order to find out statistical evidence for the significant difference between means of the associated populations. It is found that the t-value is 0.09854 and p-value is 0.921914. The result shows that it is not significant at p < 0.05. Lastly, we calculated t-values for the statistical differences of DISCERN questionnaire assessed for all categories of websites. The results are shown in Table 6. It is clear from the results the value of p is significant at p < 0.05 for only Information category websites.

3.6. Readability Level of All Included Websites

The median FRES value score of the websites is 58.5 (fairly difficult to read) with standard deviation (SD) 11.1 and the mean AGL is 8.8 with SD (1.9). The overall readability level of all websites is ‘fairly difficult to read.’ There are certain websites (for example Wikipedia) whose readability level is very low as per criterion of Flesch-Kincaid tool. The FRES and AGL score of all included websites are shown in Table A4.
Moreover, the average readability level of all categories of websites was calculated separately. The content of government websites is in plain English which can be easily understood. Table 7 shows the average FRES and AGL of all categories of websites. The Information category websites, which achieved a high DISCERN score in both assessments, however had a worse performance in readability.

4. Strengths and Limitations

To the best of our knowledge, this is the first time the DISCERN instrument has been used for evaluating the content of the World Wide Web. From the study, we learned that those websites that appeared on 3rd and 4th click of the search engine are not necessarily of good quality.
On the other hand, there are several constraints that must be kept in mind when applying the findings of this work.
The search engines are continuously updating and improving their search strategies. The same query for a search may produce different search items in the future which may lead to different results. Also, because we chose the first 20 hits (first 2 pages) from each search engine, we may have missed high quality websites just because they were not amongst the first 20 search results. Although DISCERN is a reliable tool, it does have its own limitations. For example, it does not tell anything about display/presentation of information or browsing and locating information on a website. We also excluded scientific websites from the list of candidate websites due to the limitations of DISCERN.
Similarly, there are certain limitations of the Flesch-Kincaid tool. For example, most of the medical terminologies are long in length, due to which the number of characters in a sentence increases. Thus, the complexity of an article increases according to the Flesch-Kincaid tool which is not a fair reflection of the actual result. Furthermore, a website may have a greater level of complexity if it has large information content compared to a smaller one. Similarly, complexity and readability of a document does not depend only on the length of words and sentences but also on other factors [17]. Previous studies show that readability formulas do not take graphic or pictorial material into account [40], so it may also affect the overall readability of a website. Furthermore, this study is restricted to websites written in the English language only; quality information may exist on non-English language websites too. There may also be a difference in search results depending on region. Our findings are based on search queries conducted in Malakand, in the Khyber Pakhtunkhwa region of Pakistan. The mentioned reasons may produce variation in results.

5. Conclusions

In this study, we carried out an assessment of the quality and readability of OHI on high BP, a common disease known to everyone and its treatment is not as complex as that of diseases such as Recurrent Respiratory papillomatosis, maxillofacialtrauma, diverticulitis and Endoscopic retrograde cholangiopancreatography (ERCP).
The study considers two different groups for evaluation and each group consists of 5 evaluators. It was found from the analysis that the quality of OHI on high BP is moderate.
To ensure internal consistency of the DISCERN questionnaire, the coefficient of reliability is computed. The overall average DISCERN score of all studied websites is 48.1 (60.1%) which is below the norm for a disease such as hypertension. The quality of three websites (Table A3: S.No. 23–25) is very poor and five websites have high quality of information on hypertension. Similarly, one thing is common in the assessment of both groups—that the top ten websites in each group are the same but they have been marked differently (see Table A3: S.No. 1–10). Overall the quality of 11 out of 25 (44%) websites varies from good to excellent. Both groups concluded that references and supplementary sources are not mentioned by most of the websites.
Moreover, 11 out of 25 websites do not meet the recommended score of readability. Among these 11 websites, 6 achieved an excellent DISCERN score but their readability score is less than 50 which is difficult for a college student to understand properly. Hence, in order to make it more productive, there is a dire need to improve the readability level of the assessed websites. It was also observed that the websites with a high visibility rate (appeared on 4th and 5th hit of search engine) do not necessarily have good quality of health information compared to subsequent websites.

Author Contributions

Conceptualization, M.T.; Data curation, M.U.; Formal analysis, M.T., F.M. and I.K.; Investigation, I.K. and F.M.; Methodology, M.T., M.U., S.u.R.; Project administration, M.U., F.M. and A.G.; Resources, M.U. and I.K.; Software, M.T. and M.U.; Supervision, M.U. and F.M; Validation, F.M., M.I. (Muhammad Irfan), M.I. (Muhammad Idrees) and S.u.R.; Visualization, S.u.R., M.I. (Muhammad Irfan) (Muhammad Idrees); Writing – original draft, M.T. and M.I. (Muhammad Irfan); Writing – review & editing, M.U., M.I. (Muhammad Irfan), F.M. and A.G. All authors have read and agreed to the published version of the manuscript.

Funding

For this study: there was no funding.

Acknowledgments

Thanks to Muhammad Idrees, M. Sohail Khan and Habib Ullah for their assessment about the research work. Special appreciation of Awais Naeem Assistant Professor at Khyber Teaching Hospital Peshawar and Fawad Rahim Assistant Professor at Hayat Abad Medical Complex Peshawar for their valuable inputs during the evaluation process of this work.

Conflicts of Interest

All authors declare that they have no conflict of interest.

Appendix A

Table A1. Unique Websites and Corresponding Kappa Values.
Table A1. Unique Websites and Corresponding Kappa Values.
S. No.URLsKappa (k)
HPsLS
1https://www.mayoclinic.org/diseases-conditions/high-blood-pressure/symptoms-causes/syc-203734100.40180.2204
2https://www.medicalnewstoday.com/articles/159283.php0.39600.3314
3http://www.bloodpressureuk.org/BloodPressureandyou/Thebasics/Whatishigh0.42500.3960
4https://www.heart.org/en/health-topics/high-blood-pressure/why-high-blood-pressure-is-a-silent-killer/what-are-the-symptoms-of-high-blood-pressure0.36900.3688
5https://www.medicinenet.com/high_blood_pressure_hypertension/article.htm#what_is_high_blood_pressure_wh at_is_normal_blood_pressure0.23100.2310
6https://www.healthline.com/health/high-blood-pressure-hypertension0.30500.3050
7https://www.webmd.com/hypertension-high-blood-pressure/guide/blood-pressure-causes#10.41800.4246
8https://www.nhlbi.nih.gov/health-topics/high-blood-pressure0.41800.2418
9https://www.cdc.gov/bloodpressure/about.htm0.47000.2310
10https://www.nhs.uk/conditions/high-blood-pressure-hypertension/0.36700.3047
11https://www.heartfoundation.org.au/your-heart/know-your-risks/blood-pressure0.50100.3956
12https://www.fda.gov/forconsumers/byaudience/forwomen/ucm118529.htm0.44000.4018
13https://medlineplus.gov/ency/article/000468.htm0.36600.2610
14https://www.bhf.org.uk/informationsupport/risk-factors/high-blood-pressure0.38100.2450
15https://en.wikipedia.org/wiki/Hypertension0.41800.3960
16https://www.texasheart.org/heart-health/heart-information-center/topics/high-blood-pressure-hypertension/0.39150.2750
17http://healthywa.wa.gov.au/Articles/F_I/High-blood-pressure0.36700.4247
18http://www.heartandstroke.ca/heart/risk-and-prevention/condition-risk-factors/high-blood-pressure0.44100.4513
19https://www.onhealth.com/content/1/high_blood_pressure_hypertension0.36880.4105
20https://www.emedicinehealth.com/high_blood_pressure/article_em.htm0.40180.3670
21https://www.drugs.com/mcd/high-blood-pressure-hypertension0.40180.3960
22https://www.nia.nih.gov/health/high-blood-pressure0.35050.4248
23https://familydoctor.org/condition/high-blood-pressure/0.43020.3020
24https://draxe.com/high-blood-pressure-symptoms/0.36070.3803
25https://www.betterhealth.vic.gov.au/health/conditionsandtreatments/blood-pressure-highhypertension0.36070.3690
Table A2. DISCERN Questionnaire.
Table A2. DISCERN Questionnaire.
Section 1: Reliability of Publication
Q1: Are the aims clear?
Q2: Does it achieve its aims?
Q3: Is it relevant?
Q4: Is it clear what sources of information were used to compile the publication (other than the author or producer)?
Q5: Is it clear when the information used to reported in the publication was produced?
Q6: Is it balanced and unbiased?
Q7: Does it provide details of additional sources of support and information?
Q8: Does it refer to areas of uncertainty?
Rating: No: give 1 point  Partially: give 2, 3 or 4 points  Yes: give 5 points
------------------------------------------------------------------------
Section 2: Quality of information on treatment choices
Q9: Does it describe how each treatment works?
Q10: Does it describe the benefits of each treatment?
Q11: Does it describe the risks of each treatment?
Q12: Does it describe what would happen if no treatment is used?
Q13: Does it describe how the treatment choices affect overall quality of life?
Q14: Is it clear there may be more than one possible treatment choice?
Q15: Does it provide support for shared decision-making?
Rating: No: give 1 point  Partially: give 2, 3 or 4 points  Yes: give 5 points
------------------------------------------------------------------------
Section 3: Overall rating of the publication
Q16: On the basis of the answers to all of the above questions, rate the overall quality of the publication as a source of information about treatment choices.
Rating: Low: give 1 point (serious extensive deficiencies
Moderate: give 2, 3 or 4 points (potentially important but not serious deficiencies)
Yes: give 5 points (minimum deficiencies)
Table A3. DISCERN Score and p Values.
Table A3. DISCERN Score and p Values.
S. No.URLsDISCERN Score (out of 80)p Value
HPsLSat p < 0.05
1https://www.medicalnewstoday.com/articles/159283.php72.266.80.39532
2https://www.nhs.uk/conditions/high-blood-pressure-hypertension/69.668.80.23404
3https://www.drugs.com/mcd/high-blood-pressure-hypertension68.059.40.00228
4https://www.emedicinehealth.com/high_blood_pressure/article_em.htm66.464.60.24200
5https://en.wikipedia.org/wiki/Hypertension65.269.60.03662
6https://www.healthline.com/health/high-blood-pressure-hypertension63.258.80.06288
7https://draxe.com/high-blood-pressure-symptoms/60.557.00.03000
8https://www.onhealth.com/content/1/high_blood_pressure_hypertension60.455.60.20340
9https://www.mayoclinic.org/diseases-conditions/high-blood-pressure/symptoms- causes/syc-2037341057.754.00.03662
10https://www.medicinenet.com/high_blood_pressure_hypertension/article htm#what_is_high_blood_pressure_what_is_normal_blood_pressure56.360.60.56192
11https://www.betterhealth.vic.gov.au/health/conditionsandtreatments/blood-pressure-high-hypertension48.947.00.22628
12https://www.texasheart.org/heart-health/heart-information-center/topics/high-blood-pressure-hypertension/48.653.71.00000
13http://www.heartandstroke.ca/heart/risk-and-prevention/condition-risk-factors/high-blood-pressure46.839.80.01596
14https://healthywa.wa.gov.au/Articles/F_I/High-blood-pressure46.641.20.19360
15https://www.webmd.com/hypertension-high-bloodpressure/guide/blood-pressure-causes#146.137.80.00001
16https://www.nia.nih.gov/health/high-blood-pressure45.940.80.35758
17https://www.nhlbi.nih.gov/health-topics/high-blood-pressure45.645.50.49650
18https://medlineplus.gov/ency/article/000468.htm45.644.10.00001
19https://familydoctor.org/condition/high-blood-pressure36.642.60.76418
20https://www.bhf.org.uk/informationsupport/risk-factors/high-blood-pressure35.839.40.70394
21https://www.fda.gov/forconsumers/byaudience/forwomen/ucm118529.htm33.838.80.68918
22https://www.cdc.gov/bloodpressure/control.htm31.740.00.08364
23https://www.heartfoundation.org.au/your-heart/know-your-risks/blood-pressure30.035.40.10960
24http://www.bloodpressureuk.org/BloodPressureandyou/Thebasics/Whatishigh27.231.80.31732
25https://www.heart.org/en/health-topics/high-blood-pressure/why-high-blood-pressure-is-a-silent-killer/what-are-the-symptoms-of-high-blood-pressure27.025.20.20054
Table A4. Readability scores of websites.
Table A4. Readability scores of websites.
S. No.URLsFRESAGLp Value at p < 0.05
1https://www.fda.gov/forconsumers/byaudience/forwomen/ucm118529.htm74.75.50.68918
2https://familydoctor.org/condition/high-blood-pressure/71.56.00.76418
3https://www.heartfoundation.org.au/your-heart/know-your-risks/blood-pressure70.76.80.10960
4https://medlineplus.gov/ency/article/000468.htm69.17.30.00001
5http://www.bloodpressureuk.org/BloodPressureandyou/Thebasics/Whatishigh69.17.70.31732
6https://www.cdc.gov/bloodpressure/about.htm67.77.00.08364
7https://www.nia.nih.gov/health/high-blood-pressure67.67.30.35758
8https://www.webmd.com/hypertension-high-blood-pressure/guide/blood-pressure-causes#166.97.30.00001
9http://www.heartandstroke.ca/heart/risk-and-prevention/condition-risk-factors/high-blood-pressure64.18.20.01596
10http://healthywa.wa.gov.au/Articles/F_I/High-blood-pressure63.08.00.19360
11https://www.bhf.org.uk/informationsupport/risk-factors/high-blood-pressure62.99.00.70394
12https://www.texasheart.org/heart-health/heart-information-center/topics/high-blood-pressure-hypertension/62.18.11.00000
13https://www.mayoclinic.org/diseases-conditions/high-blood-pressure/symptoms-causes/syc-2037341061.58.00.03662
14https://www.nhs.uk/conditions/high-blood-pressure-hypertension/61.58.50.23404
15https://www.heart.org/en/health-topics/high-blood-pressure/why-high-blood-pressure-is-a-silent-killer/what-are-the-symptoms-of-high-blood-pressure56.69.60.20054
16https://www.healthline.com/health/high-blood-pressure-hypertension55.58.90.06288
17https://www.drugs.com/mcd/high-blood-pressure-hypertension53.89.40.00228
18https://www.nhlbi.nih.gov/health-topics/high-blood-pressure52.210.70.49650
19https://www.betterhealth.vic.gov.au/health/conditionsandtreatments/blood-pressure-high-hypertension50.59.30.22628
20https://draxe.com/high-blood-pressure-symptoms/49.811.00.03000
21.https://www.medicinenet.com/high_blood_pressure_hypertension/article.htm#what_is_high_blood_pressure_what_is_normal_blood_pressure47.810.50.56192
22https://www.medicalnewstoday.com/articles/159283.php47.311.10.39532
23https://www.onhealth.com/content/1/high_blood_pressure_hypertension47.210.30.20340
24https://www.emedicinehealth.com/high_blood_pressure/article_em.htm44.711.40.24200
25https://en.wikipedia.org/wiki/Hypertension26.014.50.03662

References

  1. World Health Organization. Non Communicable Disease WHO Experts Warn Against Inadequate Prevention; World Health Organization: Geneva, Switzerland, 1996; p. 1. [Google Scholar]
  2. Mirrezaie, S.M.; Jalil, S.K.; Mirrezaie, N. The Quality of Websites Related to Hypertension in Iranian Internet Space. Int. J. Health Stud. 2015, 1, 12. [Google Scholar]
  3. Chow, C.K.; Teo, K.K.; Rangarajan, S.; Islam, S.; Gupta, R.; Avezum, A.; Bahonar, A.; Chifamba, J.; Dagenais, G.; Diaz, R.; et al. Prevalence, awareness, treatment, and control of hypertension in rural and urban communities in high-, middle-, and low-income countries. Jama 2013, 310, 959–968. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  4. Shah, N.; Shah, Q.; Shah, A.J. The burden and high prevalence of hypertension in Pakistani adolescents: A meta-analysis of the published studies. Arch. Public Health 2018, 76, 20. [Google Scholar] [CrossRef] [Green Version]
  5. Lim, S.S.; Vos, T.; Flaxman, A.D.; Danaei, G.; Shibuya, K.; Adair-Rohani, H.; Amann, M.; Anderson, H.R.; Andrews, K.G.; Aryee, M.; et al. A comparative risk assessment of burden of disease and injury attributable to 67 risk factors and risk factor clusters in 21 regions, 1990–2010: A systematic analysis for the Global Burden of Disease Study 2010. Lancet 2012, 380, 2224–2260. [Google Scholar] [CrossRef] [Green Version]
  6. Diviani, N.; van den Putte, B.; Giani, S.; van Weert, J.C. Low health literacy and evaluation of online health information: A systematic review of the literature. J. Medical. Int. Res. 2015, 17, e112. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  7. Internet World Stats – Usage and Populations Statistics. Available online: https://www.internetworldstats.com/stats.htm (accessed on 15 October 2018).
  8. Cerminara, C.N.; Malhany, E. Website and Headache: Assessment of the Information Quality Using the DISCERN Tool. J. Neurol. Neurosci. 2016, 7, 1–5. [Google Scholar]
  9. Dr Google Will See You Now. Available online: https://www.telegraph.co.uk/technology/2019/03/10/google-sifting-one-billion-health-questions-day/ (accessed on 13 August 2019).
  10. Reynolds, M.; Hoi, A.; Buchanan, R.R.C. Assessing the quality, reliability and readability of online health information regarding systemic lupus erythematosus. Lupus 2018, 27, 1911–1917. [Google Scholar] [CrossRef]
  11. More People Search for Health Online. Available online: http://www.nbcnews.com/id/3077086/t/more-people-search-health-online/#.W8o4kkszbIU (accessed on 19 October 2018).
  12. Europeans Becoming Enthusiastic Users of Online Health Information: Shaping Europe’s Digital Future. Available online: https://ec.europa.eu/digital-single-market/en/news/europeans-becoming-enthusiastic-users-online-health-information. (accessed on 19 October 2018).
  13. Patel, P.P.; Hoppe, I.C.; Ahuja, N.K.; Ciminello, F.S. Analysis of Comprehensibility of Patient Information Regarding Complex Craniofacial Conditions. J. Craniofacial Surg. 2011, 22, 1179–1182. [Google Scholar] [CrossRef]
  14. Kocyigit, B.F.; Koca, T.T.; Akaltun, M.S. Quality and Readability of Online Information on Ankylosing Spondylitis. Clin. Rheumatol. 2019, 38, 3269–3274. [Google Scholar] [CrossRef]
  15. Murray, E.; Lo, B.; Pollack, L.; Donelan, K.; Catania, J.; Lee, K.; Zapert, K.; Turner, R. The impact of health information on the Internet on health care and the physician-patient relationship: National US survey among 1.050 US physicians. J. Med. Internet Res. 2003, 5, e17. [Google Scholar] [CrossRef]
  16. Mead, N.; Varnam, R.; Rogers, A.; Roland, M. What predicts patients’ interest in the Internet as a health resource in primary care in England? J. Health Serv. Res. Policy 2003, 8, 33–39. [Google Scholar] [CrossRef] [PubMed]
  17. Seidman, J.J.; Steinwachs, D.; Rubin, H.R. Conceptual framework for a new tool for evaluating the quality of diabetes consumer-information Web sites. J. Med. Internet Res. 2003, 5, e29. [Google Scholar] [CrossRef] [PubMed]
  18. Charnock, D. The DISCERN Handbook: Quality Criteria for Consumer Health Information on Treatment Choices. Available online: http://www.discern.org.uk (accessed on 12 September 2018).
  19. Friedman, D.B.; Hoffman-Goetz, L.; Arocha, J.F. Health literacy and the World Wide Web: Comparing the readability of leading incident cancers on the Internet. Med. Inform. Internet Med. 2006, 31, 67–87. [Google Scholar] [CrossRef] [PubMed]
  20. McKenzie, J.F.; Neiger, B.L.; Thackeray, R. Planning, Implementing & Evaluating Health Promotion Programs: A Primer; Pearson: London, UK, 2016. [Google Scholar]
  21. Meleo-Erwin, Z.; Basch, C.; Fera, J.; Ethan, D.; Garcia, P. Readability of Online Patient-Based Information on Bariatric Surgery. Health Promot. Perspect. 2019, 9, 156–160. [Google Scholar] [CrossRef] [PubMed]
  22. Groller, K.D. Systematic Review of Patient Education Practices in Weight Loss Surgery. Surg. Obes. Relat. Dis. 2017, 13, 1072–1085. [Google Scholar] [CrossRef] [PubMed]
  23. Hansberry, D.R.; Agarwal, N.; Shah, R.; Schmitt, P.J.; Baredes, S.; Setzen, M.; Carmel, P.W.; Prestigiacomo, C.J.; Liu, J.K.; Eloy, J.A. Analysis of the Readability of Patient Education Materials from Surgical Subspecialties. Laryngoscope 2013, 124, 405–412. [Google Scholar] [CrossRef] [Green Version]
  24. Eysenbach, G.; Powell, J.; Kuss, O.; Sa, E.R. Empirical studies assessing the quality of health information for consumers on the world wide web: A systematic review. Jama 2002, 287, 2691–2700. [Google Scholar] [CrossRef]
  25. Connelly, T.M.; Khan, M.S.; Victory, L.; Mehmood, A.; Cooke, F. An assessment of the quality and content of information on diverticulitis on the Internet. Surgeon 2018, 16, 359–364. [Google Scholar] [CrossRef]
  26. Arif, N.; Ghezzi, P. Quality of online information on breast cancer treatment options. Breast 2018, 37, 6–12. [Google Scholar] [CrossRef] [Green Version]
  27. Cerminara, C.; Santarone, M.E.; Casarelli, L.; Curatolo, P.; El Malhany, N. Use of the DISCERN tool for evaluating web searches in childhood epilepsy. Epilepsy Behav. 2014, 41, 119–121. [Google Scholar] [CrossRef]
  28. Kaicker, J.; Dang, W.; Mondal, T. Assessing the quality and reliability of health information on ERCP using the DISCERN instrument. Health Care Curr. Rev. 2013, 1, 104. [Google Scholar] [CrossRef] [Green Version]
  29. Ipser, J.C.; Dewing, S.; Stein, D.J. A systematic review of the quality of information on the treatment of anxiety disorders on the Internet. Curr. Psychiatry Rep. 2007, 9, 303–309. [Google Scholar] [CrossRef] [PubMed]
  30. Alshehri, M.G.; Joury, A.U. Quality, Readability, and Understandability of Internet-Based Information on Cataract. Health Technol. 2019, 9, 791–795. [Google Scholar] [CrossRef]
  31. Dickerson, C.; Kumar, S. Quality and Readability of Online Patient Education Information and the Parents Comprehension for Childhood Depression. J. Health Med Inform. 2016, 7, 2. [Google Scholar] [CrossRef]
  32. Saithna, A.; Ajayi, O.O.; Davis, E.T. The Quality of Internet Sites Providing Information Relating to Hip Resurfacing. Surgeon 2008, 6, 85–87. [Google Scholar] [CrossRef]
  33. San Giorgi, M.R.M.; de Groot, O.S.D.; Dikkers, F.G. Quality and readability assessment of websites related to recurrent respiratory papillomatosis. Laryngoscope 2017, 127, 2293–2297. [Google Scholar] [CrossRef] [Green Version]
  34. Joury, A.; Joraid, A.; Alqahtani, F.; Alghamdi, A.; Batwa, A.; Pines, J.M. The Variation in Quality and Content of Patient-Focused Health Information on the Internet for Otitis Media. Child Care Health Dev. 2017, 44, 221–226. [Google Scholar] [CrossRef]
  35. Kumar, V.S.; Subramani, S.; Veerapan, S.; Khan, S.A. Evaluation of Online Health Information on Clubfoot Using the DISCERN Tool. J. Pediatric Orthop. B 2014, 23, 135–138. [Google Scholar] [CrossRef]
  36. Clarke, M.A.; Moore, J.L.; Steege, L.M.; Koopman, R.J.; Belden, J.L.; Canfield, S.M.; Meadows, S.E.; Elliott, S.G.; Kim, M.S. Health information needs, sources, and barriers of primary care patients to achieve patient-centered care: A literature review. Health Inform. J. 2016, 22, 992–1016. [Google Scholar] [CrossRef]
  37. World Health Organization Hypertension. Available online: https://www.who.int/news-room/fact-sheets/detail/hypertension (accessed on 10 March 2020).
  38. Van Deursen, A.J.A.M.; van Dijk, J.A.G.M. Using the Internet: Skill related problems in users’ online behavior. Interact. Comput. 2009, 21, 393–402. [Google Scholar] [CrossRef]
  39. Eysenbach, G. How Do Consumers Search for and Appraise Health Information on the World Wide Web? Qualitative Study Using Focus Groups, Usability Tests, and in-Depth Interviews. BMJ 2002, 324, 573–577. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  40. Grabeel, K.L.; Russomanno, J.; Oelschlegel, S.; Tester, E.; Heidel, R.E. Computerized versus hand-scored health literacy tools: A comparison of Simple Measure of Gobbledygook (SMOG) and Flesch-Kincaid in printed patient education materials. J. Med Libr. Assoc. JMLA 2018, 106, 38. [Google Scholar] [PubMed] [Green Version]
  41. Sabharwal, S.; Badarudeen, S.; Kunju, S.U. Readability of online patient education materials from the AAOS web site. Clin. Orthop. Relat. Res. 2008, 466, 1245–1250. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  42. Protheroe, J.; Estacio, E.V.; Saidy-Khan, S. Patient Information Materials in General Practices and Promotion of Health Literacy: An Observational Study of Their Effectiveness. Br. J. Gen. Pract. 2015, 65, e192–e197. [Google Scholar] [CrossRef] [Green Version]
  43. Wong, A.; Yamat, H. Testing the Validity and Reliability of the “Learn, Pick, Flip, Check, Reward” (LPFCR) Card Game in Homophone Comprehension. Int. J. Acad. Res. Bus. Soc. Sci. 2020, 10, 22–32. [Google Scholar] [CrossRef]
  44. Landis, J.R.; Koch, G.G. The measurement of observer agreement for categorical data. Biometrics 1977, 159–174. [Google Scholar] [CrossRef] [Green Version]
  45. Shapiro, S.S.; Wilk, M.B. An analysis of variance test for normality (complete samples). Biometrika 1965, 52, 591–611. [Google Scholar] [CrossRef]
  46. Razali, N.M.; Wah, Y.B. Power comparisons of shapiro-wilk, kolmogorov-smirnov, lilliefors and anderson-darling tests. J. Stat. Modeling Anal. 2011, 2, 21–33. [Google Scholar]
  47. Cramer, D. Fundamental Statistics for Social Research: Step-By-Step Calculations and Computer Techniques Using SPSS for Windows; Psychology Press: London, UK, 1998. [Google Scholar]
Figure 1. Flow chart of research work.
Figure 1. Flow chart of research work.
Applsci 10 03214 g001
Figure 2. Clarity of sources of all included websites by HPs.
Figure 2. Clarity of sources of all included websites by HPs.
Applsci 10 03214 g002
Figure 3. Assessment of additional sources of information by HPs.
Figure 3. Assessment of additional sources of information by HPs.
Applsci 10 03214 g003
Figure 4. Clarity of sources of all included websites by LS.
Figure 4. Clarity of sources of all included websites by LS.
Applsci 10 03214 g004
Figure 5. Evaluation of additional sources of information by LS.
Figure 5. Evaluation of additional sources of information by LS.
Applsci 10 03214 g005
Figure 6. Results comparison by both groups (HPs and LS).
Figure 6. Results comparison by both groups (HPs and LS).
Applsci 10 03214 g006
Figure 7. Performance of categories of websites by both Groups.
Figure 7. Performance of categories of websites by both Groups.
Applsci 10 03214 g007
Table 1. Categories of websites.
Table 1. Categories of websites.
Category of WebsitePercentage
Information36%
Government24%
Institution24%
Non-Profit Organization16%
Table 2. DISCERN grading and their quality level.
Table 2. DISCERN grading and their quality level.
DISCERN ScoreOut of 100Quality Level
64–8080% & aboveExcellent
52–6365% to 79%Good
41–5151% to 64%Fair
30–4037% to 50%Poor
16–2920% to 36%Very Poor
Table 3. Kappa Statistics and Strength of Agreement [44].
Table 3. Kappa Statistics and Strength of Agreement [44].
Kappa StatisticsStrength of Agreement
<0.00Poor
0.00–0.20Slight
0.21–0.40Fair
0.41–0.60Moderate
0.61–0.80Substantial
0.81–1.00Almost Perfect
Table 4. Evaluation of all included websites by Group 1 (Health Professional (HPs)).
Table 4. Evaluation of all included websites by Group 1 (Health Professional (HPs)).
N = 25Very Poor (Score = 16–29) (20% to 36%)Poor (Score: 30–40) (37% to 50%)Fair (Score: 41–51) (51% to 64%)Good (Score: 52–63) (65 % to 79%)Excellent (score: 64–80) (80%&above)
DISCERN: Part-I (Q1 to Q8) Reliability of Publication2(8%)2(8%)10(40%)6 (24%)5 (20%)
DISCERN: Part-II (Q9 to Q15) Quality of Information on Treatment5(20%)2(8%)9(36%)4 (16%)5 (20%)
DISCERN: Part-III (Q16) Overall Quality2(8%)5(20%)8(32%)5(20%)5(20%)
Table 5. Evaluation of all included websites by Group 2 (Lay Subjects (LS)).
Table 5. Evaluation of all included websites by Group 2 (Lay Subjects (LS)).
N = 25Very Poor (Score = 16–29) (20% to 36%)Poor (Score:30–40) (37% to 50%)Fair (Score: 41–51) (51% to 64%)Good (Score: 52–63) (65% to 79%)Excellent (Score: 64–80) (80% & above)
DISCERN: Part-I (Q1 to Q8) Reliability of Publication2(8%)2(8%)10(40%)6 (24%)5 (20%)
DISCERN: Part-II (Q9 to Q15) Quality of Information on Treatment1(4%)8(32%)5(20%)8(32%)3 (12%)
DISCERN: Part-III(Q16) Overall Quality1(4%)4(16%)9(36%)5(20%)6(24%)
Table 6. t-values for all categories of websites at p < 0.05.
Table 6. t-values for all categories of websites at p < 0.05.
Category of Websitest-Valuep-Value at p < 0.05
Information1.801230.040863
Government−0.11650.908034
Institution−0.523810.60426
Non-profit organization−0.709270.483292
Table 7. Average Readability score of all categories of websites.
Table 7. Average Readability score of all categories of websites.
CategoryNo. of WebsitesFRESAGLp-Value at p < 0.05
Government664.4 ± 8.27.6 ± 1.30.908034
Information948.7± 10.810.5 ± 1.90.040863
Institution460.8 ± 6.38.5 ± 1.40.60426
Non-Profit665.8 ± 5.77.8 ± 1.30.483292

Share and Cite

MDPI and ACS Style

Tahir, M.; Usman, M.; Muhammad, F.; Rehman, S.u.; Khan, I.; Idrees, M.; Irfan, M.; Glowacz, A. Evaluation of Quality and Readability of Online Health Information on High Blood Pressure Using DISCERN and Flesch-Kincaid Tools. Appl. Sci. 2020, 10, 3214. https://doi.org/10.3390/app10093214

AMA Style

Tahir M, Usman M, Muhammad F, Rehman Su, Khan I, Idrees M, Irfan M, Glowacz A. Evaluation of Quality and Readability of Online Health Information on High Blood Pressure Using DISCERN and Flesch-Kincaid Tools. Applied Sciences. 2020; 10(9):3214. https://doi.org/10.3390/app10093214

Chicago/Turabian Style

Tahir, Muhammad, Muhammad Usman, Fazal Muhammad, Shams ur Rehman, Imran Khan, Muhammad Idrees, Muhammad Irfan, and Adam Glowacz. 2020. "Evaluation of Quality and Readability of Online Health Information on High Blood Pressure Using DISCERN and Flesch-Kincaid Tools" Applied Sciences 10, no. 9: 3214. https://doi.org/10.3390/app10093214

APA Style

Tahir, M., Usman, M., Muhammad, F., Rehman, S. u., Khan, I., Idrees, M., Irfan, M., & Glowacz, A. (2020). Evaluation of Quality and Readability of Online Health Information on High Blood Pressure Using DISCERN and Flesch-Kincaid Tools. Applied Sciences, 10(9), 3214. https://doi.org/10.3390/app10093214

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop