Open Innovation in Digital Healthcare: Users’ Discrimination between Certified and Non-Certified mHealth Applications

Abstract

Mobile applications users consider it especially important to have quality certification, and particularly in an e-health context. However, if the app is certified, will user satisfaction be greater? This paper is intended to help answer that question. Therefore, with user satisfaction as a determinant of the quality of services, the purpose of the paper is to understand the extent to which certified and non-certified mHealth apps influence user experience. This was measured by online reviews and downloads. An empirical study was carried out with 50 mHealth apps (21 certified, 29 non-certified), which is a convenient sampling procedure. The Mahalanobis matching technique was used to test for significant differences between the groups. A comparative content analysis was performed on 6188 user comments and on app downloads. Non-parametric tests were preferred because the variables analyzed had a categorical structure. A log-linear model and advanced contingency tables were used to examine the more than two-way relationships of variables. The results show differences between the two samples (certified and non-certified mHealth apps). However, quality certification labels as a guarantee of the quality of apps do not have a modulating effect on the online reviews and downloads relationship. The model showed a statistically significant partial association between “certification” and “online user reviews” (Z = 2.174; sig = 0.042)] and between “download” and “online user reviews” (Z = 2.387; sig = 0.017).

1. Introduction

The exponential growth of mobile health (“mHealth”) applications (“apps”) in recent years has not been accompanied by mandatory control or regulation processes to mitigate risk or to guarantee quality [1,2,3,4]. These apps can collect a variety of personal data and, because they are designed to influence health, it is important to ensure their safety, validity, reliability, privacy, and security. In most countries, medical device regulation applies only to a subset of high-risk mHealth apps that have well-defined medical purposes. However, most mHealth apps available on the market target a wide range of health-related issues (monitoring, control, medical advice, and so on), while still being considered non-medical. In addition, some apps have been evaluated based on specific regulation procedures, validation or accreditation systems, or standards. Initiatives by the United States’ Food and Drug Administration, the American Medicines Agency, and the European Union are clear examples of this. They have already put themselves to work to regulate and set minimum quality criteria for mHealth apps. These processes allow the quality of mHealth apps to be measured. Compliance with certain requirements (usability, privacy, security, safety, and so on) help to guarantee proper use and integration into medical practice; otherwise, because they collect a variety of personal data, the apps may be indirectly responsible for misinformed self-care [5]. Through quality assurance, the fulfilment of the purpose of the apps, such as monitoring, goal-setting and schedule management, gathering health-related information, and managing health conditions, can be guaranteed [6]. This helps to satisfy the needs of users (e.g., patients, health professionals, or the general public), who in many cases do not possess media and technology literacy skills.

In general, for many, the certification of a product or service is synonymous with quality. Obviously, the health-apps are no exception to this phenomenon. Therefore, this should result in a positive evaluation of its users based on their personal experiences. However, even when the app is included in a repository or catalogue that guarantees compliance with different requirements widely recognised in the literature [7], and is tested by specific accreditation programs for accessibility, usability, reliability, privacy, security, and so on, if it does not meet users’ needs, their perception of the app’s quality will differ. Consumer satisfaction indicates the degree to which products and services meet or exceed customer expectations [8] and is the result of service quality evaluation [9]. In the present research, we consider consumer satisfaction as the overall affective response based on the use of apps and/or related attributes. This can change after each use of the apps. The users of the app constitute the key element; they are the ultimate “testers” in evaluating the service offered by the devices.

The purpose of this study is to evaluate the implementation and certification of quality and its impact on user experience in m-health apps. We explore the extent to which user experience differs according to whether an mHealth app is certified or not. User satisfaction is the foundation of quality. Therefore, it is expected that these variables (user online reviews and downloads) will discriminate certified apps from non-certified ones.

The paper is organised as follows. First, a literature review of quality certification and online user comments is conducted. The methodology is then described, and the results presented and discussed. The conclusion includes some of the paper’s limitations, and finally a number of suggestions are made for relevant future research.

2. Theoretical Background and Related Work

Two areas of research are particularly relevant to the current study. The first covers the certification programs or regulations systems that verify app quality. The second is online user review evaluation.

2.1. Apps and Certified Quality

The term “quality” itself hinders the development of any work in this area. The subjectivity of the concept, and therefore the lack of a universally accepted definition, may be the main reason for this obstacle. In addition, the features or attributes that have traditionally been used to evaluate products and services are either insufficient or inappropriate in today’s digital environment. The products and services to be evaluated are different in the health field. Here, the technology enables more information to be gathered, and at less cost. Furthermore, the process takes place in real time.

This situation perhaps makes it difficult to establish a system or standard that allows the regulation of a verification process that could guarantee accessibility, security, and so on. However, quality criteria and measurements have recently been introduced, and these can be used by users and academics to evaluate the quality of apps.

Although there is still no consensus on the matter, the United States Food and Drug Administration (FDA) and the American Medicines Agency launched a guide with the objective of regulating and setting minimum quality criteria for health apps, proposing a series of recommendations that give guidelines on which apps would be on track and which ones would not. Likewise, it indicates that the mHealth apps that function as medical devices and that therefore might “put the patient at risk” will be regulated by the FDA. For its part, the European Union has created a directory of mHealth apps to “support patients to find useful and reliable apps.”

In addition, alternatives have been introduced to guarantee basic quality and safety standards for mHealth applications. In the United States, for example, a system similar to the one used to accredit and certify laboratory tests is laid out in the Clinical Laboratory Improvement Amendment [10]. In Europe, there are companies such as the Société Générale de Surveillance (SGS) that are dedicated to testing and verifying products, including medical devices. These certify health software which is considered safe for use by applying the new health software products security brand [11]. In addition, in the United Kingdom, the National Health Service (NHS) Applications Library helped users find reliable health and wellness applications that had been certified as clinically safe and secure. In fact, several studies examining mHealth apps have used NHS certification as a criterion for inclusion, and recommended users access NHS-certified applications [3,12,13,14]. However, the library was shut down in 2015 due to security concerns [15].

A number of other mHealth app certification initiatives have been taken to integrate mobile health in the medical field with a view to guaranteeing safety and quality. One was the Health App Certification Program (HACP seal). This was developed by Happtique, a mobile health solutions company focused on applications targeting the United States market. It was suspended in December 2013, after the CEO of a health IT firm posted a blog post exposing security issues with two apps Happtique had certified as secure. However, several papers have used Happtique-certified apps, and have recommend the use of Happtique-certified applications [2,16,17].

Users in different six countries (Estonia, Germany, Portugal, Slovenia, the United Kingdom, and Spain) have the option of using accredited health apps from either private organisations or public entities [18]. Amongst them, the United Kingdom has a more formalised process developed by the government. In Spain, there are two certification models, Appsalut and Appsaludable, both developed by public health agencies (in the Catalonian and Andalusian regions, respectively), that judge the quality and safety of health apps. These are free and available for all public or private apps, Spanish or otherwise. The importance of these badges justifies the selection criteria of apps analyzed in several studies [19,20,21,22,23,24,25]. This is evidence of the interest in security seals, which allow people to place more confidence in new technological solutions. Llorens-Vernet and Miró (2020) provide a standard for mHealth apps based on the information that is available in published studies, guidelines, and standards [4].

Despite all these initiatives, greater transparency and knowledge of the benefits and limitations of using these digital tools are necessary. In the Table 1 we can analyze some of the commented initiatives.

Table 1. Initiatives of accreditation de mHealth apps.

Initiative	Origin	Year	Features/Requirements
FDA (US Food and Drug Administration)	United State (US)	2013	Oversight of device software functions. Three-tier classification system: - Class I: lowest risk. Exempt from FDA review - Class II: intermediate level of risk. Developers are required to submit a premarket notification. - Class III: highest risk level. More complex, time-consuming, and expensive premarket approval process
https://apps.healthskouts.com/
NHS Health Apps Library	United Kingdom (UK)	2013	Technical assessment tries to check the following items: available evidence on outcomes, clinical safety, data protection, security, usability and accessibility, interoperability and technical stability.
https://www.nhs.uk/apps-library/
European Directory of Health Apps	European Union (EU)	2012	Health apps it lists have all been recommended by patient groups and empowered consumers, then categorised and indexed in several ways (including by local language), to make the details easy for readers to find.
https://g3ict.org/publication/european-directory-of-health-apps-2012-2013
Clinical Laboratory Improvement Amendment (CLIA)	US	1988	Certifies and ensures the quality of testing in US laboratories. Over time a similar approach is applied to m-Health apps. They must comply with basic standards in three areas: accessibility, privacy and content.
https://www.cms.gov/Regulations-and-Guidance/Legislation/CLIA
Happtique	US	2010	Requirements assess: operability, privacy, security and content.
Suspended since December 2013
mHONcode	Switzerland	1996	Reflect the accountability, trustworthiness, and transparency of health application. The Honcode app certification has 8 principles: authority, complementarity, confidentiality, validity, justifiability and objectivity, user‘s practice, financial disclosure and advertisement policy.
https://www.hon.ch/imgs/2020/ENGuidelinesApp_compressed.pdf
AppSaludable	Spain	2012	This seal is based on the 31 recommendations published in the Guide of Recommendations on Design, Use and Assessment of Health Apps (see Appendix A).
http://www.calidadappsalud.com/distintivo/catalogue
Appsalut	Spain	2015	It is based on criteria published in the criteria guide of the accreditation process. The criteria are structured in 4 groups: usability and design, functionality, technology, and security. At the end of the accreditation process, the mHealth.cat office grants those applications that have successfully passed the mandatory minimum criteria to publish on the AppSalut portal, a numerical note resulting from the evaluation of the criteria.
https://appsalut.gencat.cat/

Source: own elaboration.

All initiatives have the same limitations, in that accreditation should ensure compliance with industry standard levels of encryption and authentication. Besides, the regulators should consider establishing standards for accreditation processes and be ready to intervene if accreditation programs cannot manage risks effectively. A general agreement is necessary to regulate the use of health applications with full guarantee and quality [26].

2.2. Online User Reviews

The irruption of new technologies enables the collection of large amounts of data about app users. As an alternative to conventional methods, studies from different fields have shown that user-generated content (including online user reviews) can be used as an information source to understand customer preferences and demands [27]. Usually, online consumer reviews are provided by customers who had direct (and usually recent) experience with a product or service; they are reported in multiple forms including online ratings (e.g., number of stars) and online reviews (e.g., personal opinion in text format) [28,29,30].

The data are used for a variety of purposes. The form of interaction with the app, the user’s profile and behaviour, activity, downloads, location, and so on, are amongst the variables most commonly analyzed in evaluation of apps. It may be argued that this type of analysis is essential if apps are to be improved. In addition, mobile app reviews are valuable repositories of information, because they are written by the users themselves. Therefore, they constitute a valuable source of data, the analysis of which can allow the detection of possible defects and aspects that might be improved. Also, the app can be classified according to the positive or negative connotation that are expressed in the language. Hence, the evaluation and comparison of mHealth apps based on the opinions of its users has become a research objective, with the results used for different purposes. The user reviews contain a wealth of information about user experience and expectations and may provide valuable information for making decision about applications and their features [31,32]. For research, results help to increase the understanding of customer expectations formation in online settings [33].

A great number of papers have been published on this topic, and a range of approaches have been taken. Specialist journals have shown their interest by presenting articles that examine qualitative feedback based on the analysis of user reviews [34,35,36,37]. Several researchers have studied the utility for users of mHealth apps aimed at certain diseases. These address unmet needs and determine users’ expectations [38,39,40]. A number of researchers have examined the perception of users of mHealth apps [1,41,42,43]. All these studies further prove the degree of interest in the subject.

On the other hand, today, the integration of the public in research and development in health care is considered essential for the advancement of innovation. This is a paradigmatic shift away from the traditional assumption that solely health care professionals are able to devise, develop, and disseminate novel concepts and solutions in health care [44]. The active participation of other groups is essential.

Thus, app reviews given by users usually contain active, heterogeneous and real life user experience of mobile app which can be useful to improve the quality of app [32,45]. The participation of the users, through the online reviews, allows obtaining very valuable information from the user regarding the application: what is wrong, what could be improved, what is missing, what is left over, etc. Take advantage of this feedback to prioritize the new features to be developed, according to what your audience is waiting for. Furthermore, it has been proven that platforms, online communities and applications are an important intermediary for patient empowerment [44,46]. All of this demonstrates the importance of the open innovation in analysis of mobile health applications. The user review and, consequently, the improvements that can be carried out by them being types of user-lead innovations, are an important source of innovation [47]. Open innovation models bring added value resulting in the use of technical potential and ensuring the growth of economic value [48,49,50,51]. This could benefit healthcare delivery to improve individual well-being and the overall quality of life of the population.

In general, stakeholders need to be involved in the development and implementation of eHealth via co-creation processes, and design should be mindful of vulnerable groups and eHealth illiteracy [52].

Another related area of research is the relationship between user online reviews and downloads. Some authors have studied the connection between app downloads and users’ perception of quality [31]. There is considerable disparity here; some studies claim there is a direct link between these variables while others do not. For example, Chen et al. (2014), Duan et al. (2008) and Liu (2006) argue that user ratings do not influence choice, and therefore downloads [53,54,55]. Chen and Liu (2011) demonstrate that apps at the top of the download rankings are not the ones with the best user ratings [56].

However, other authors, such as Harman et al. (2012) [57], who analyzed more than 30,000 Blackberry apps, have identified a strong correlation between app ratings and the number of downloads. Chevalier and Mayzlin (2006) show this to be the case [58]. It should be kept in mind that, although many users trust other users’ ratings to decide whether to use a particular app, sometimes misleading comments and false reviews can lead to poor decisions. Automated systems exist to identify these, but their use is limited and they are not fully developed. Computer models such as AR-Miner perform exhaustive analyses of user reviews to provide app developers with clear and concise information about user opinions so they can subsequently improve the product [53]. The variety of opinions of a particular app may sometimes be the consequence of a failure to analyze user reviews uniformly across studies [59].

Attempts to understand the relationship between online reviews and downloads are not been made within a regulatory framework. The present study adds to the existing literature by providing a new view of the quality of apps that is linked directly to the user experience, which is mediated through online reviews and downloads [60]. The following objectives are proposed in order to achieve the general purpose of this study: (1) verify if there are any significant differences between certified and non-certified apps; and (2) determine if the apps’ quality label affects the user experience, according to the type of applications (certified and non-certified).

Under these considerations the following hypotheses are proposed:

Hypothesis 1a (H1a).

There are user online review differences between apps with quality certification and non-certified.

Hypothesis 1b (H1b).

There are user online review differences between apps with respect to variables (user, developer, stars rating and categories).

Hypothesis 1c (H1c).

There are download differences between apps with quality certification and non-certified.

Hypothesis 2a (H2a).

User online review discriminates certified apps from non-certified ones.

Hypothesis 2b (H2b).

Downloads discriminates certified apps from non-certified ones.

Hypothesis 2c (H2c).

The quality labels affect the relationship between downloads and online user feedback.

3. Materials and Methods

3.1. Sample

First of all, we focused our efforts on collecting a list of accredited and non-accredited apps. All AppSaludable-accredited apps were selected for inclusion in the study. In this sense, we also chose a set of unaccredited Google Play applications with significant recognition by users, so each of these had to meet two criteria: classification in the medical category, and inclusion in at least two of the three existing rankings (top apps; top grossing; or trending). The end result was a set of 55 apps (22 certified and 33 non-certified). In order to eliminate selection bias from the sample of non-certified apps, to ensure an appropriate comparison, a matching technique was used (Mahalanobis distances) [61]. Finally, despite the alleged lack of randomness, the final sample retains 50 apps with comparable recognitions, of which only 5 of the original apps were left out.

The AppSaludable Quality Seal is the first Spanish seal that recognises quality and safety of health apps. It is free and open to all public or private apps, both Spanish and from other countries.

Those apps that accomplish the Andalusian Agency for Healthcare Quality (ACSA) validation process will award the AppSaludable Quality Seal.

The certification process is mainly based on the self-assessment of the app in accordance with recommendations included in the guide, and the assessment carried out by a committee of the Agency’s experts in order to identify possible improvements. Once the seal is awarded, the app becomes part of a list of mobile health apps with remarkable safety and quality. This seal will be in force while the app is active or until its content and functions change. The responsible persons of the app will commit themselves to a revision of the process carried out by ACSA when the app suffers important changes.

The quality seal is based on 31 recommendations, grouped into four broad categories containing 16 criteria and published in the Guide for Recommendations on the Design, Use, and Assessment of Health Apps [62] (see Figure 1).

Figure 1. Recommendations on design, use and assessment of health apps. Source: Andalusian Agency for Healthcare Quality.

In the Appendix A, the complete list of recommendations for the design, use and evaluation of health applications can be consulted.

This certification program is part of Safety and Quality Strategy in Mobile Health Apps (ACSA). It is a dynamic and integrated process that counts with the collaboration of those users (developer, professionals, patients and general public) who want to share any suggestions or ideas to contribute to this project, through the open innovation project “Collaborate With Us”.

3.2. Data Analysis and Collection

To analyze the apps, we collected descriptive information (e.g., price, platform, certification date, and so on), as well as technical details (e.g., developer, sharing capabilities, and so on). Information was also gathered on the target user group and other aspects of the apps that may have been of interest. These data domains could be adapted to include or exclude specific content areas as needed. Next, web scraping was carried out to convert the unstructured web data into a structured format that could be easily accessed and used. For this study we used R language for scraping the reviews [63]. We used the rvest and RSelenium packages in R and scraped the following data from each app: app name; date; user’s name; start; review; likes; reply. All comments recorded for the 55 apps were collected from Google Play. The total number amounted to 6188 user reviews, and all were analyzed. The strengths of researching customer textual reviews are that they can show customer consumption experiences, highlight the product and service attributes customers care about, and provide customers’ perceptions in a detailed way through the open-structure form [64].

Before we performed the text mining, we cleaned up all the text to render the results more accurate [65,66]. Our goal at this stage was to create independent terms (i.e., words) from the data file before we could begin counting their frequency. To achieve our goal, we used the tm and SnowballC packages, which support all text mining functions. We removed websites, emojis, punctuations, numbers, whitespace, and stop words from the data set. After the cleaning process, we were left with 13,881 independent terms that exist throughout the online reviews. These were stored in a matrix that shows each of their occurrences. This matrix (called a term matrix) logs the number of times the term appears in our clean data set. We used sentiment analysis for data labelling. This was carried out automatically by the R programme using a set of more frequently-occurring positive words to extract the meaning from the reviews and to define their polarity. Word weighting accorded to frequency of occurrence. The more often a word appeared in the reviews overall, the greater its weight; the term was then considered to be a strong candidate for inclusion in the content analysis [67]. The set of positive word selected amounted to 29 terms, and, according to the presence or otherwise of positive words, we rated each review. We rated a total of 2509 positive comments.

After that, we created an app quartile ranking based on the Q ratio. We determined the ratio as the total number of positive words from the app’s positive comments divided by the total number of words from all positive comments. Finally, each app was assigned to a group of Q depending on the value of that ratio. Thus, Q4 represented those apps with the highest value of Q, and Q1 the opposite.

To contextualise the study, descriptive analyses of the variables were carried out using the data obtained. Two-sample t and chi-square tests were used to assess the differences between certified and non-certified apps based on online user reviews and downloads. To check the possible relationship between the variables, non-parametric tests of chi-square were carried out.

With regard to categorical data, a log-linear model was applied to study the possible modulating effect of the certification process between user reviews and downloads. This model has been suggested in the literature as a way of determining the relationship structures of multi-dimensional contingency tables [68]. The log-linear (conditional independence) model was chosen to model the association between the X and Y variables conditioned by the Z variable because it was the model with the best fit; it is used in cases when one of the variables is dependent on the other two variables (X and Y), and those two variables are independent of each other (Z). The model (XY, YZ) is expressed as:

$$\ln {\mu }_{ijk}= \lambda + {\lambda }_i^X+{\lambda }_j^Y+{\lambda }_k^Z+{\lambda }_{ij}^{XY}+{\lambda }_{jk}^{YZ}$$

(1)

donde:

• X → 1, 2, …, i
• Y → 1, 2, …, j
• Z → 1, 2, …, k
• ${\mu }_{ijk}$ = frequency determined for the ijk cell in the contingency table
• ${\lambda }_i^X$ = Main impact of the X variable
• ${\lambda }_j^Y$ = Main impact of the Y variable
• ${\lambda }_k^Z$ = Main impact of the Z variable
• ${\lambda }_{ij}^{XY}$ = Second degree interaction impact of variables X and Y
• ${\lambda }_{jk}^{YZ}$ = Second degree interaction impact of variables Y and Z

4. Results. Analysis and discussion

4.1. Sample Characteristics

The study found a variety of mHealth apps that had been classified by different users, category, developer, and type of communication (bidirectional or unidirectional). Table 2 summarizes the main characteristics of the 55 mHealth apps.

Table 2. Details of h-apps (n = 50) (frequency).

Characteristics	Certified		Top (Non-Certified)
	No.	%	No.	%
Users
Patients	7	33.33	15	51.72
Professionals	8	38.10	1	3.45
General public	6	28.57	13	44.83
Category
Monofunction	13	61.90	14	48.27
Multifunction	8	38.10	15	51.73
Developer
Health sector company	4	19.05	11	37.93
Technological company	8	38.10	16	55.17
Sanitary professional	6	28.57	1	3.45
Particular	1	4.76	0	0.00
Public administration	2	9.52	1	3.45
Communication (bidirectional)
Yes	4	19.05	5	17.24
No	17	80.95	24	82.76

4.2. Hypotheses Testing

Figure 2 shows the distribution of apps according to the ranking established by user ratings in online reviews (0 ≤ Q1 ≤ 0.015; 0.015 < Q2 ≤ 0.038; 0.038 < Q3 ≤ 0.114; 0.114 < Q4 ≤ 0.75) and descriptive statistical summaries.

Figure 2. Distribution of apps according to the user opinion ranking (n = 50 apps).

The results of Student’s t-test for the two samples (certified apps and non-certified apps) showed a statistically significant difference (t = 4.22; p = 0.000). For this reason, H1a hypothesis is confirmed.

This significant difference highlights the effectiveness of the application quality certification processes. This is likely to be justified by meeting the seal requirements in terms as important to the user as ensuring the quality and security of information (requirements 5–15, see Appendix A) as well as privacy and data protection (requirements 21–31, see Appendix A).

On the other hand, the chi-square tests carried out to assess the possible relationships between the different variables (user, developer, category, and stars) and user comments showed a significant relationship between all the variables, except for the categories of functions (see Table 3). Then, the results obtained confirmed the H1b hypothesis except for the category variable.

Table 3. Chi-square tests.

Variables	Chi-Squared	p
User	11.738	0.047 *
Developer	19.516	0.04 *
Stars	122.832	0.045 *
Categories (mono-multi function)	2.579	0.453
Health and general well-being	9.179	0.027 *
Medical information	3.792	0.285
Healthcare administration	12.940	0.044 *
Remote and sensor-based monitoring	8.289	0.040 *
Other	11.167	0.011 *

* p < 0.05.

The higher percentage of h-apps used by professionals (as a group supposed to have more objective opinion about the applicability of a h-app) in the group of certified apps, perhaps marks the possible relationship between the type of users and online comments. While the satisfaction of general public users and patients is of great importance, it has less impact from the view of the scientific applicability of an h-app.

Certified apps, whose main developers are tech companies and healthcare workers, get the highest proportion of positive reviews (Q4). This demonstrates the importance of technical and health aspects for users. These aspects provide a high level of usability, guaranteed, in the case that it is being studied, through compliance with requirements, such as accessibility, design, authorship, updating or revision of information, among others (see Appendix A).

Opinions such as the following are a clear example:

“It’s excellent. The design of the app makes it easy to use. The interface is very intuitive. I recommend it”.

Furthermore, as expected, and in line with other studies [64,69], the more stars there are, the more positive comments will be obtained from users. In this study, chi-square test results show a significant difference in the star ratings. The fact that certified apps gather the highest positive response rates (Q4) may lead us to the claim that the star ratings pose trusty the user’s experience and the validity of the reviews posted by them. It could be due to there is an adequate mechanism to control user’s feedback that is guaranteed by the certification program. Consequently, this will contribute to reaching a high level of user satisfaction and thus, go on increasing positive reviews.

Finally, in relation to functionality, apps that dealt with general health and well-being show a higher proportion of positive´s comments. By contrast, apps manage health get the lower positive´s comments ratio.

Regarding H1c hypothesis, Figure 3 shows the distribution of apps according to classification by downloads (D1 ≤ 10,000; 10,000 < D2 ≤ 50,000; 50,000 < D3 ≤ 100,000; D4 > 100,000) and descriptive statistical summaries.

Figure 3. Distribution of apps according to download behaviour (n = 50 apps).

The results of Student’s t-test of the two samples (certified and non-certified apps) showed a statistically significant difference (t = 2.32; p = 0.025) confirming the H1c hypothesis.

In contrast with online reviews, the average download was significantly higher in the group of non-certified applications. This does not imply that users consciously opt for downloading non-certified apps. It can be explained by the fact that the sample of non-certified apps was made up of the top apps on Google Play and, obviously, these generate a greater number of downloads. Both user downloads and user ratings offer an arbitrary indicator of the popularity, acceptability, and satisfaction with apps. In addition, other factors such as, e.g., the fact that the number of apps used by professional users (supposedly a smaller group), was much higher in the certified group, could explain these results.

To test the H2a hypothesis, a cross-tabulations test was performed resulted in an association between online user reviews and app quality certification (chi-square = 41.570; p = 0.000). Although the percentages can help us to intuit possible association patterns, the corrected standardised residuals allow us to determine precisely the relationship between both variables [70] (see Table 4). Thus, for the apps in categories Q2 and Q3, the proportion of non-certified apps was significantly higher than the certified (2.900 and 3.000 > 1.96, respectively; 95% confidence level); and for Q4, the proportion of certified apps was significantly higher than the non-certified (4.9 > 1.96). Of the highest rank apps (Q4), 100% were certified. These represented 61.9% of the total certified apps.

Table 4. Contingency table user reviews and quality seal.

Health Apps		Online User Reviews 1				Total
		Q1	Q2	Q3	Q4
Non-certified	Observed count	7	12	10	0	29
	Expected count	8.1	7.5	5.8	7.5	29.0
	Standardized residual	−0.4	1.6	1.7	−2.7
	Adjusted residual	−0.7	2.9	3.0	−4.9
Certified	Observed count	7	1	0	13	21
	Expected count	5.9	5.5	4.2	5.5	21.0
	Standardized residual	0.5	−1.9	−2.0	3.2
	Adjusted residual	0.7	−2.9	−3.0	4.9	0.7
Total	Observed count	14	13	10	13	50
	Expected count	14.0	13.0	10.0	13.0	50.0

¹ 0 ≤ Q1 ≤ 0.015; 0.015 < Q2 ≤ 0.038; 0.038 < Q3 ≤ 0.114; 0.114 < Q4 ≤ 0.75.

Besides, the results of the chi-square tests and the statistics associated with the cross tables also showed a significant association between certified apps and downloads (103.447; GL = 8; p = 0.000), confirming the H2b hypothesis. The analysis of corrected standardised residuals enables us to identify apps with fewer downloads (D1), a more significant proportion in certified apps (5.2 > 1.96; 95% confidence level); for those in D3, the proportion of non-certified apps was significantly higher than the certified (2.9 > 1.96) and in the group of most downloaded non-certified apps (D4), there was significant association (4.1 > 1.96) (see Table 5).

Table 5. Contingency table downloads and quality seal.

Health Apps		Downloads 1				Total
		D1	D2	D3	D4
Non-certified	Observed count	5	5	8	11	29
	Expected count	6.2	2.8	3.8	4.5	29.0
	Standardized residual	−0.5	1.3	2.2	3.1
	Adjusted residual	−0.7	1.7	2.9	4.1
Certified	Observed count	13	3	3	2	21
	Expected count	4.5	2.0	2.8	3.3	21.0
	Standardized residual	4.0	0.7	0.2	−0.7
	Adjusted residual	5.2	0.9	0.2	−0.9
Total	Observed count	18	8	11	13	50
	Expected count	18.0	8.0	11.0	13.0	50.0

¹ D1 ≤ 10,000; 10,000 < D2 ≤ 50,000; 50,000 < D3 ≤ 100,000; D4 > 100,000.

Additionally, as was stated previously, the log-linear conditional independence model was used to analyze the association between the three variables (downloads, online reviews, and certification) and to test the H2c hypothesis. The download variable was converted to dichotomous with two categories, D1 (≤100,000 downloads) and D2 (>100,000 downloads). Goodness of fit results can be seen in Table 6.

Table 6. Goodness of fit results for the model.

	Test Statistics	df	Sig.
Likelihood ratio	0.980	4	0.913
Pearson chi-square	0.749	4	0.945

The model design is: constant + certification * user online reviews + user online reviews * download. Table 7 shows the parameter estimates and standardised errors.

Table 7. Model parameter estimates.

Parameter	Estimate	Std. Error	Z	Sig.	95% Confidence Interval
					Lower Bound	Upper Bound
Constant	4.002 × 10−17	1.000	0.000	1.000	−1.960	1.960
[certification = 0] × [Qi = 1]	0.405	1.185	0.342	0.732	−1.918	2.728
[certification = 0] × [Qi = 2]	1.019	1.157	0.880	0.379	−1.250	3.287
[certification = 0] × [Qi = 3]	1.792	1.080	2.174	0.042	−0.325	3.909
[certification = 0] × [Qi = 4]	−20.378	4475.634	−0.005	0.996	−8792.460	8751.705
[certification = 1] × [Qi = 1]	0.405	1.185	0.342	0.732	−1.918	2.728
[certification = 1] × [Qi = 2]	−1.466	1.502	−0.976	0.329	−4.410	1.478
[certification = 1] × [Qi = 3]	−18.665	5308.578	−0.004	0.997	−10,423.281	10,385.957
[certification = 1] × [Qi = 4]	0
[Qi = 1] × [Di = 0]	1.299	0.651	1.995	0.046	0.023	2.576
[Qi = 1] × [Di = 1]	0
[Qi = 2] × [Di = 0]	1.204	0.658	1.829	0.067	−0.086	2.494
[Qi = 2] × [Di = 1]	0
[Qi = 3] × [Di = 0]	−0.405	0.645	−0.628	0.530	−1.671	0.860
[Qi = 3] × [Di = 1]	0
[Qi = 4] × [Di = 0]	2.485	1.041	2.387	0.017	0.445	4.525
[Qi = 4] × [Di = 1]	0

It was observed as a result of the partial relations test, that the statistically significant parameters were the relationship between “certification” and “online user reviews” and the relationship between “downloads” and “online user reviews.” When the Z values of double-way relationship terms were examined, it was determined that ‘download” and “online user reviews” had the highest Z value (2.387; the dependency was between [online user reviews (Qi) = 4] and [downloads (Di) = 0]). Therefore, it was the most important factor in determining the frequencies in the contingency table. The more the number of positive comment increased, the more the probability of a download increased, regardless of whether the app was certified or not. Additionally, the model showed a statistically significant association between “certification” and “online user reviews” ([online user reviews (Qi) = 3] and [certification = 0]; Z = 2.174; sig = 0.042).

5. Conclusions

This paper is a preliminary analysis of the impact of mHealth app certification on user experience. It complements previous research on studies of the association between user opinions and app downloads, the differentiating element being an analysis of the role played by a quality seal that guarantees the quality and safety of certain mHealth apps.

The first finding is the significant difference between the sample of apps analyzed with regard to user reviews and download behaviour. It could be due to the particular features of each group of apps which present different users and functionality. The sample selection criterion could also affect to this results. In respect of possible differences in the user experience revealed by online reviews, the results show a significant association between the type of user and the developers of the app; this confirms the results of previous studies. It is also noteworthy to point out, the highest proportion of positive comments belong to certified apps which the effectiveness of the self and external evaluation process is demonstrated.

Another important finding is contained within the results of the relationship between user experience and certified app quality. Although the quality of what is offered (i.e., the product or service) must prevail if the user experience is to be satisfactory, the existence of a quality mark is not a significant or relevant factor. This conclusion is based on the joint evaluation of user comments and app downloads using a log-linear conditional independence model. The model showed a statistically significant partial association between “certification” and “online user reviews” (Z = 2.174; sig = 0.042)] and between ‘download” and “online user reviews” (Z = 2.387; sig = 0.017).

5.1. Implications: Open Innovation and Certification in mHealth Application Industry

Having taken the results of our investigation as a whole into account, it is possible to consider their practical implications and subsequently make some recommendations. First, questions about the relevance of the app quality validation process are a reminder that it needs to be modified or revised to ensure that a satisfactory level of quality (usability, security, reliability, and so on) will be perceived by users. Therefore, new quality and security seals for mHealth apps need to be developed for people to have greater confidence in them. What is more, they should cover the design, development, and deployment stages. Second, as is widely accepted, the clearer and simpler a technology is, the easier and more useful it will be for users, increasing their user experience [50]. Information from user reviews related to this aspect can provide valuable information for quality assurance agencies involved in validation to improve their accreditation processes. This way, users become in active participants in health apps open innovation projects settings.

In this context, to embrace efficient, equitable, and accountable values in culture for open innovation dynamics is important to conquer cultural barriers in addition to legal and institutional barriers in public organizations. The increased culture of multi-stakeholder collaboration in this sector will allow for more active collaboration with external parties and open innovation in public administration could be used to explore collective action to improve healthcare management for the benefit of all the agents involved (patients, professionals, healthcare industry, public administrations, etc.) [71]. This model is followed by the AppSaludable seal which is accessible to any interested agent who wishes to share a suggestion or idea, our work being an example of academic involvement as a concerned party. In this field, the pioneering project in mobile health of the European mHealth Hub is a reference in innovation and knowledge in mobile health for the European territories of the World Health Organization.

For all the above, according to the concept of the quadruple helix model this sets the direction in which open innovation policy makers should move [50]. This model suggests four main players engaged to the application of open innovation: university, government, industry and society. Therefore, within the overall health context, open innovation policy makers should focus on providing enough information feedback loops to open innovation systems and at the same time take actions to strengthen the open innovation culture of the organization.

Obviously, in the digital age, all of this contributes to the progressive shift from the traditional knowledge management approach to a network approach to knowledge exchange.

In short, the applications have come to the health field to stay and evolve towards a new model of medicine: mHealth, which will help the patient and the doctor to improve well-being indices and greater efficiency in prevention, diagnosis and treatment. Research in mHealth can ensure that important social, behavioral and environmental data is used to understand the determinants of health, improve health outcomes, and prevent the development of disorders. This new area of research has the potential to be a transformative force, because it is dynamic, based on a continuous process of data entry and evaluation.

5.2. Limitations and Further Research

Finally, this study has limitations that should be considered when interpreting the results. We would like to note these limitations come mainly from the sampling methodology. To eliminate selection and analysis bias, the initial sample was adjusted. A matching technique was used to test the significant differences between the groups and thus guarantee an appropriate comparison between them.

A second limitation lies in the fact that the survey is based on the users’ experience, without focusing on distinguishing between “aware users” or “non-aware users” in relation to the accreditation of apps. Probably some aware users download the application because the distinction of quality offers them a greater guarantee and security, being their preferences and expectations completely different. A study that considers this situation can provide interesting information for a deeper analysis of user satisfaction. This also involves the use of a fit methodology for this aim far from the purpose of this paper.

Logically, the lack of trust associated with the risks of use an app is present in the choice of the application by the user and the existence of a quality seal can mitigate this perceived risk [72]. However, the seal or distinction of quality of an app is not among the most influential factors to download an app. In a Redbox Mobile study of mobile app searches, based on responses from 1500 mobile users from the US, China, UK, Spain, Germany and Italy, the most influential factor is price followed by other factors that would also be quite relevant such as a clear description, the score that the application has obtained and its reviews [73]. Some of these factors are included in the App Store Optimization OFF-METADATA factors, which are those that the developers do not control directly, and that depend on the reaction of the market and the marketing of the application that has been carried out [74]. These are the most important factors that influence Google and Apple’s positioning algorithms in their app stores, and that directly impact users’ installation decisions. One again, a quality distinction or seal is not a relevant factor, so it could be supported by the statement that the quality, trust, or safety certification culture and implementation is at an early stage with a long way to go.

Despite these limitations, this study provides a framework for others working to create sustainable health solutions in the developing world.