COVIDTAS COVID-19 Tracing App Scale— An Evaluation Framework

: Mobile apps play an important role in COVID-19 tracing and tracking, with different countries taking different approaches. Our study focuses on 17 government owned COVID-19 Contact Tracing Apps (CTAs) and analyze them using a proposed COVIDTAS framework. User satisfaction is not directly related to the COVIDTAS score or the interaction between users and the app developers. To increase adoption of CTAs, government leadership must offer assurance to its citizens that their identify will be concealed and emphasize the beneﬁts of CTAs as it relates to shared public health. While no country has topped the list on all three major factors (COVIDTAS Score, User Reviews, and User Ratings), the CTA from India seems to have above average performance on all three factors.


Introduction
The COVID-19 pandemic, through its sheer scale has highlighted and widened the socioeconomic divides that exist today and made vulnerable populations such as those with chronic illnesses, elderly, minorities, people of lower socioeconomic backgrounds, etc., more susceptible [1]. Mobile phones and digital technology has been in the forefront of addressing this crisis in fields varying from education to [2] healthcare [3]. One such approach is contact tracing using mobile applications. The concept of contact tracing in relation to infectious diseases in not new. For years, practitioners of community medicine have isolated those most likely to be infected by a certain infectious disease from any infected individual, isolated these potential candidates, and provided treatment as needed [4]. This approach is extremely effective when done at a local level, with a relatively low number of cases [5]. Contact tracing can be done by local health care workers in person, or by use of technologies ranging from RFID [6] to wearable tracking devices [7].
In recent times, with the advent of smart phones and the high penetration of mobile technology, cell-phone apps have been widely used for contact tracing. Mobile contact tracing applications can be used to identify the proximity of each user to others, including those infected with a contagious disease. Once an infected person is identified, some mobile applications even warn its users when contact is imminent [8]. Clearly, such applications have a potential to minimize person to person transmission of diseases. In addition, these applications allow healthcare workers to identify those at risk of infection, allowing for pre-emptive isolation and treatment to minimize further spread as well as the intensity of infestation.
These applications have become more mainstream in light of the ongoing COVID-19 pandemic. It should be noted that the use of such applications is not without controversy, and it has been argued that extensive data gathering by governments can eventually do more harm than good [9]. However, mobile apps have been employed successfully across the world for years to prevent the spread of contagious diseases. Therefore, a methodology to assess the applications currently employed for contact tracing on criteria such as data security, privacy, data retention, etc., is of significance. Understanding these applications in the context of various discrete criteria can help assess their risk to citizens and governments should the data be misused. In that light, this study presents a novel COVIDTAS framework for assessing COVID-19 tracing applications. The evaluations rely on public data and reviews on various platforms for analysis. However, before analysis it is essential to gain a basic understanding of contact tracing applications and their use in medicine. In that light, the next section explores a few such examples with a focus on how government mechanisms in the past have employed mobile contact tracing apps. This is followed by a discussion on digital apps used for COVID-19 tracing specifically. The study then focuses on COVID-19 applications that are initiated by governments and the concerns associated with a government driven approach. The data used in this study is then presented and analyzed before discussing the results of the analysis. In addition, the perceptions surrounding government controlled COVID-19 applications is also discussed in this paper before presenting the conclusion and future research direction.

History of Contact Tracing
Contact tracing becomes essential when the potential of a disease for person to person spread is high. Naturally, the earlier applications of mobile apps for contact tracing has been for diseases such as Ebola, SARS/MERS, TB, etc. Broadly speaking, these applications fall under m-health or mobile health interventions, where mobile technology is used to aid health care. It should be noted that while this paper focuses on contact tracing, mhealth encapsulates applications such as remote diagnosis, data sharing, and information dissemination in addition to contact tracing. Contact tracing apps have been employed extensively in managing the spread of Ebola in African countries over the last decade. For example, in the 2014-15 outbreak of Ebola in West Africa, mobile interventions were used for contact tracing in addition to community level manual tracing. In Sierra Leone, a mobile app termed as Ebola Contact Tracing application (ECT app) was employed for contact tracing [10]. Results show that app-based interventions allowed for more concrete, accurate, and timely updating of the centralized database if done correctly. In addition, the manpower and paperwork required for contact tracing was also drastically reduced [10]. A study of 58 Ebola contact tracing apps by [11] identifies three applications, Community Care, Sense Ebola Follow up, and Surveillance and Outbreak Response Management and Analysis System as capable of carrying out surveillance, contact tracing, management of cases, and data management for laboratory data, thereby making them complete data management systems for Ebola and similar contagious diseases. A more generic software Go.data [12] which can be employed for any disease outbreak with peer to peer transmission has been used successfully in Democratic Republic of Congo for Ebola contact tracing successfully. Similarly, mobile apps have been employed for contact tracing in case of TB, SARS, and other respiratory diseases [13,14].

Present Use of Contact Tracing Apps for COVID-19
The potential use of contact tracing in the control of COVID-19 has been well established by studies, especially in cases where there was less chance of transmission before the onset of symptoms [15]. The use of mobile apps for contact tracing has come to the mainstream with the recent COVID-19 pandemic. Governments and healthcare agencies across the world are employing mobile apps for identifying potential paths of infection and isolating those at risk. The MIT project on COVID-19 tracking apps lists over 25 apps at this stage, which are employed by governments across the world for contact tracing during this pandemic [16]. The project lists a number of applications ranging from those that are compulsory to those that are optional. Many vary widely in terms of transparency, A few examples include the COVIDSafe app by Australian government, which faced resistance from the public in general due to concerns about privacy and was not entirely successful in providing contact tracing due to lack of widespread adoption of the application [17]. India's Aarogya Setu app, which is the most downloaded contact tracing application, effectively collects personal information and constantly adds to location information in real time for its users [18]. On the other hand, the Corona-Warn-App used in Germany posed no major privacy concerns as it was developed with potential privacy infringements in mind. Data is depersonalized and stored locally to avoid any concerns in this regard [19]. While many countries such as India have made contact-tracing apps practically compulsory for activities such as air travel [20], other countries such as the UK have refrained from employing similar procedures due to public concerns over privacy [17,21,22]. There are clearly diverse approaches taken by governments during this pandemic when it comes to mobile contact tracing. However, in general, contact tracing apps have been employed as a novel tool for curtailing the spread of this disease. The concerns raised by many with regards to privacy and data security in relation to these applications are valid and need to be discussed in detail.

Government Interventions in Contact Tracing
The role of governments in contact tracing has been a matter of much discussion. Multiple surveys and studies find varying degrees of public trust in government contact tracing applications. For example, a survey carried out in the Republic of Ireland with over 8000 participants shows a general acceptability for a contact tracing application. In fact, 54% of the respondents were positive about downloading such an app, while another 30% were moderately likely to do the same [23]. This is in sharp contrast with a survey on over 2000 participants from the UK, where the acceptance rate for a government-controlled application was under 55%. However, there was greater acceptability for an app controlled by the NHS [24], clearly highlighting a lack of trust in the government handling sensitive medical information. Citizens of many other countries are effectively left without a choice in the matter. For example, while data collection for public good needs no individual consent in China, it is possible for the government to provide private parties access to the same data, thereby leading to potential privacy concerns [19]. Similarly, a study on a COVID-19 contact tracing app propagated by the government of France highlights how such applications, though immediately beneficial, may be a serious challenge to data privacy in the future. The study also suggests that the government may be downplaying this adverse effect for the immediate purpose of contact tracing [25].
Similarly, the use of contact tracing in Russia has raised serious concerns about it being used for mass movement surveillance. The application suggested in Moscow may require citizens to generate a unique code each time they exit their living space, and share the code with law enforcement, should they be asked to do so [26]. Some countries such as South Africa and the UAE are yet to have concrete data privacy laws, making the deployment of such applications all the more concerning as they do not have to adhere to existing data laws [27]. Amnesty international (2020) has raised privacy concerns related to contact tracing apps and identifies the applications employed in Bahrain, Kuwait, and Norway to be among the least safe for data privacy. There have also been significant differences in how governments across the world handled these concerns related to surveillance and data security. For example, Singapore chose to publicly acknowledge the potential security concerns with its application and asked its citizens to concentrate on the immediate contact tracing requirement [28]. Hong Kong employed geo fencing instead of GPS tracking to provide some degree of privacy to its citizens while South Korea addressed such concerns by stringently restricting access to the database [29]. In other countries such as India and Russia the concerns regarding misuse of data from tracking apps is still significant [26,30,31]. The limited number of studies available on this issue highlights that mass adoption of these applications is necessary for them to be successful. This can either be achieved through legislation (making it compulsory to use the application), or through trust building with the public. In addition, with regards to COVID-19, the disease disproportionately affects the elderly, who, despite having smart phones, are often technology averse and unlikely to install and use mobile applications [32]. In general, at least in democratic systems, such mobile applications are unlikely to succeed without adequate public trust as illustrated by cases in the UK [33] and Australia 17].

Challenges and Apprehensions Regarding Government Contact Tracing
The papers discussed so far clearly illustrate the challenges of balancing individual rights regarding data privacy, with the immediate need for contact tracing [34]. From literature, it is also seen that specific policy and technology measures can help alleviate and minimize these concerns. This is best illustrated by the approach taken by Germany, where data privacy was built into the contact tracing application while also providing decentralized storage of data, thereby minimizing the loss or misuse of largescale data [35]. Similarly, by employing geo fencing instead of GPS tracking, Hong Kong has managed to address concerns of governments tracking its citizens to a great extent. Existing privacy and data security laws also play a part in protecting citizens of any country against the misuse of this sensitive information. In general, it is safe to conclude that in democratic systems, it is best that the government spend adequate resources on educating the public about the importance of contact tracing applications while simultaneously being transparent in addressing any concerns regarding privacy and data security.

Materials and Methods
Contact Tracing Apps (CTAs) are designed to automate the process of gathering data of contacts of people with infected individuals, so as to identify the risks based on the context, proximity and duration of contact. CTAs estimate proximity through Bluetooth or Global Positioning System (GPS). GPS is not suitable as the accuracy is low in built-up spaces. Bluetooth is available in smartphones and can support proximity estimation, however the distance estimation may vary based on surrounding objects and the orientation of the phone [30], thus leading to false positive and negative alerts.
Contact tracing apps may be designed with a centralized architecture, where all activities revolve around a trusted server that supports storing encrypted information, analyzing risk of contacts and notifying such identified contacts [36,37]. In this case, the identity of the users is known to the trusted server. India's Aarogya Setu and Singapore's TraceTogether are examples of centralized architecture (Table 1). In contrast, the decentralized architecture minimizes the information exchange and anonymizes user identification with the server [38]. Australia's COVIDSafe [39] is built with decentralized architecture. There is also a hybrid model, where only the personal ID and data of COVID-19 positive users or volunteers are saved at the server [40].
This study is focused on contact tracing applications used by governments for COVID-19 mitigation. Data from Contact Tracing Apps of 17 countries with at least one country from each of the seven continents were aggregated as part of this study ( Table 2). A multistage approach was adopted to collect the data.

Asia Israel
CTAs, privacy policies and their mapping to the mandatory requirements of the state. Additionally, policies on transparency were verified through open-source availability of CTA software databases and adherence to GDPR requirements. The adherence to various GDPR requirements were assessed for non-European countries from publicly available authentic and reputable sources that provided critical analysis of CTA's security and privacy features. Cross-validations with news articles and/or research publications were carried out wherever possible.

•
Stage III involved evaluation of publicly available reviews inclusive of those shared by user, legal firms and the media. countries from N. America, three countries from Europe, two from South America, one country from Africa, Australia, and New Zealand.

•
Stage II involved comprehensive research that involved analyzing the design of CTAs, privacy policies and their mapping to the mandatory requirements of the state. Additionally, policies on transparency were verified through open-source availability of CTA software databases and adherence to GDPR requirements. The adherence to various GDPR requirements were assessed for non-European countries from publicly available authentic and reputable sources that provided critical analysis of CTA's security and privacy features. Cross-validations with news articles and/or research publications were carried out wherever possible.

•
Stage III involved evaluation of publicly available reviews inclusive of those shared by user, legal firms and the media. The process followed for data selection and analysis is presented below.
• Stage I involved selection of countries were primarily based on two factors (1) countries that had proactive involvement of state-owned bodies in the control, coordination and reporting of their national status and (2) additionally, to ensure good geographical spread, countries were chosen from various continents. This resulted in three countries being chosen within Asia, three countries from Middle East, two countries from N. America, three countries from Europe, two from South America, one country from Africa, Australia, and New Zealand. • Stage II involved comprehensive research that involved analyzing the design of CTAs, privacy policies and their mapping to the mandatory requirements of the state. Additionally, policies on transparency were verified through open-source availability of CTA software databases and adherence to GDPR requirements. The adherence to various GDPR requirements were assessed for non-European countries from publicly available authentic and reputable sources that provided critical analysis of CTA's security and privacy features. Cross-validations with news articles and/or research publications were carried out wherever possible.

•
Stage III involved evaluation of publicly available reviews inclusive of those shared by user, legal firms and the media.
Certain countries such as USA, Russia, China, S. Africa, etc., were excluded from this study due to lack of adoption of CTAs or reliance on private or multiple entities for contact tracing apps. India, France, Spain, Brazil, Colombia, and Mexico had the highest number of infected people in their respective continents, i.e., Asia, Europe, South America, and North America. Additionally, they were also amongst the Top 10 countries worldwide from the highest number of total cases and total number of deaths perspective. The reason Qatar and Israel were chosen was because of their high density of COVID-19 cases i.e., total cases per million population. The interesting aspect of analyzing data from Singapore was that in spite of having a relatively large number of cases i.e., close to 10,000 per million population, the number of deaths per million is amongst the lowest worldwide i.e., five per million. On the other hand, Vietnam and New Zealand were amongst the countries that had the lowest number of COVID-19 cases per million population and two to three orders of magnitude lower deaths per million in comparison to the highly infected countries. On user ratings and reviews, data was collected primarily from Google app store rather than Apple app store due to the former being a much larger repository of user downloads and usage statistics.

Qualitative and Quantitative Parameters in the COVIDTAS Framework
A reliable and sound method for analyzing the various applications selected was essential for this study. The framework should be able to assess the effectiveness of the application on various fronts and provide an overall score for the application for comparison with its peers. This study proposes The Covid Tracing App Scale (COVIDTAS) framework as a platform to compare the various dimensions of these apps. This includes elements of human-centric design from usability, technology and privacy perspectives, their effectiveness in tracking and communicating proximity to COVID-19 patients and ultimately the user perception and acceptance based on their direct feedback. This framework has the potential to serve as an effective tool to key stake holders i.e., the public, medical personnel and federal authorities and assist with enhancing adoption, usage and impact from such apps. COVIDTAS framework was adapted from the sociotechnical framework proposed by [41] that pertained to analyzing the design configurations of the app from technology, governance and user perspectives. Our framework extends this even further by including factors that reflect user experience and sentiment that are critical to its success.
The data collected included both qualitative and quantitative parameters. The qualitative parameters included: the extent of data security, transparency, ease of use, access to information, architecture, and so on. While synthesizing the data, emphasis was placed on the congruence between the qualitative parameters and the policies governing CTAs. Particular attention was paid to good and suboptimal practices that led to the identification of technological, organizational, cultural, and individual factors that were captured in the final empirical framework. The scoring rubric for all qualitative factors ranged between 0 and 2 with 2 representing the highest rating for the most preferred option, 1 representing moderate or partial compliance, and 0 representing the lowest rating for inadequate coverage or poor compliance (Table 3) Table 3. Factors and scoring rubric for the COVID-19 CTAs.

Factors
Score 2 Score 1 Score 0 Data collection should be compliant with the General Data Protection Regulation (GDPR) [6] and respect the privacy of the individual. A Data Protection Impact Assessment (DPIA) must be carried out before the deployment of any contact-tracing system. The purpose of the app and the mechanisms to assess its usage need to be clearly defined Some of the other factors such as legislative policies, incidental finding or dual-use policy mentioned in Vinuesa et al. (2020) were difficult to compare due to nonavailability of information to make reasonable comparisons between the 17 CTAs. It is also assumed factors such as status of fundamental rights of individuals and any discrimination or stigmatization are low or fairly similar across countries not influential in this analysis.
Several quantitative data points were systematically analyzed from the 17 countries. These included considering their total population, the number of COVID-19 cases in their countries and the number of CTA downloads to derive the percentage of adoptees. Additionally, the total number of cases, total number of deaths and total number of COVID-19 tests taken per million of the population were tabulated. Viewing this data set helped ensure and confirm our objective of selecting countries that captured a wide spectrum of impact from COVID-19 in their respective countries. The quantitative factors coded into the framework included the user review and star ratings of the respective apps. This involved looking at both reviews and star ratings of 17 CTAs between March and September 2020. Analyzing the total number of reviews and responses to the reviews revealed that there was a direct correlation between the number of responses and the percentage of adoption. CTAs from six countries (i.e., Vietnam, Japan, France, Spain, Brazil, and Australia) in this cohort had provided no responses to reviews.
Previous studies [42] have shown that user acceptability is critical in adoption of CTAs resulting in tangible management of the spread of COVID-19. More specifically the privacy concerns were found to be the largest impediment for acceptance and usage in developed countries such as Germany and US. The model used in their work, i.e., health belief model, anchors on the user perception of both i.e., consequences of a health threat and how effective the proposed ways of its management or protection from it are. Thus, direct experiences and opinions of users are important to accommodate in viewing the impact of CTAs holistically. The user ratings utilized in our framework included their review-based ratings and star ratings. A one-or two-star rating was considered inadequate and earned a score of 0 while a four-star or five-star rating was assigned a score of 2. Additionally, Google's ratings between 0 and 5 were also captured and linearly extrapolated and mapped to score of 0-2. Comparisons between Google's app ratings and user's review ratings showed reasonable congruence between them in most cases.
We first provided a characterization of all the CTAs from Google Play Store. Then we analyzed the parameters like number of installs, average star ratings, total reviews, reviews by star ratings from the website applfow.io and androidrank.org. The COVIDTAS score was derived by averaging scores of all 12 factors in the framework. All factors included in the framework were given equal weightage.

Results and Discussion
The COVIDTAS framework evaluated whether the 17 state-owned CTAs selected for this study satisfied parameters such as privacy and data protection, transparency rights, data management, etc. as shown in Table 4. Each CTA was thoroughly studied to identify if the apps were compliant with the 12 parameters that were the foundation of the COVIDTAS scale.
Overall, the New Zealand app scored the highest (1.82), followed by the apps from Canada and Australia (1.67), and those from Japan and Germany (1.57). The apps that scored less than 1 were from Qatar and Columbia. Though the EHTERAZ tracing app from Qatar was legally enforced on the public, it was not available in the local language and did not support app-related education and tutorials for the public. The app scored 0 in other aspects as well such as transparency rights, ease of accessibility and security. The South American apps also scored low on the COVIDTAS scale (Brazil 0.67, Columbia 0.58) as they scored 0 in all other parameters. It was also unclear if these state-owned apps can be removed or deactivated easily. All these apps (Middle East and South America) and COVID-19MX from Mexico did not comply with data minimization principles and were found to be a security risk to their users.
The research involved analyzing the average star rating given to the apps by users. To do so, the 1 to 5 scale of Google Play Store were converted into 0, 1, 2, wherein 1-2 was scored 0, 2.01-3.50 was scored 1, and 3.51-5.0 was scored 2. It was determined that the apps from Asia, North America, Columbia, and Africa received the highest rating of 2, whereas apps from Japan, Israel, UAE, New Zealand, Australia, Europe, and Brazil received a one-star rating. When exploring the review ratings for these apps, only six apps scored a rating of 2, namely, India, Vietnam, Canada, Mexico, Columbia, and Morocco. A higher percentage of users (68% to 89%) gave these six apps a four-or five-star rating. The other 11 CTAs scored 1 for this parameter.  Some interesting observations can be made regarding the performance of these COVID-19 apps by employing the COVIDTAS framework. The stress placed on privacy, transparency, and data security seems to be dependent on the state of development of the country involved. Each CTA was examined under the "Privacy and data protection" COVIDTAS parameter to assess if data collection was in agreement with the General Data Protection Regulation (GDPR), user's privacy was respected, data protection impact assessment was done, and other such variables given in the Table 3. It was seen that CTAs from Europe, Australia, New Zealand, and Canada completely fulfilled these requirements, whereas countries from the Middle East (UAE, Qatar) were not adequate, which, as mentioned above, was reflected in their low COVIDTAS score (1, 0.5, respectively).
Israel, Canada, Australia, New Zealand, and European countries score high in parameters related to privacy, transparency, data management, and security. New Zealand, Germany, Australia, and France score a maximum possible score of 8 (2 in each category) while Israel, Canada, and Spain score 7. It is also interesting to note that these 'Western' countries score higher than their Asian counterparts of similar socioeconomic development. Singapore and Japan, the two developed Asian economies considered in this study score 6 points in this regard. Vietnam and India, the two developing Asian economies also score 6 each in this regard. Countries in the Middle East (with the exception of Israel, which is a democracy) and South America (with relatively unstable governments) seem to fare the lowest in this regard, with Brazil, Columbia, UAE, and Qatar scoring zero across these parameters. It is safe to assume that healthy democracies fare better in terms of data integrity and security due to higher levels of accountability to the public. In addition, economic development usually allows for greater data protection and citizens of developing countries face greater challenges in this regard [43]. This trend continues with COVID-19 applications as well.
When factors that determine the effectiveness of the application such as decentralization, accessibility, and awareness raising (through education) are considered, New Zealand and Japan score the highest points, while India, Vietnam, Australia, Germany, and Spain are close behind. Large scale advertisements and interventions regarding COVID-19 and CTAs in general have been shown to improve acceptability of the application [44]. This characteristic does not seem to be dependent on geography, socioeconomic development, or type of government. Ten apps were observed to be educative and user friendly as they provided reading materials and tutorials within the app and external access to websites to improve awareness on the disease as well as information on using the app correctly. However, the apps from Qatar, Mexico, Europe (Germany provided website access), South America, and Africa did not provide any public health awareness or app information. In countries such as India, use of CTAs was mandated in certain locations such as airports, government offices while it was not mandated at other public places such as malls and recreational sites.
The ease of removing the app from the device and whether the installation is forced legally in any way are both measures of the degree of control citizens have over the application. In this regard, much like with data privacy, developed 'Western' economies (Australia, New Zealand, Canada, Germany, and France) score the highest values along with Japan, and UAE. It should be noted that Singapore, a developed economy from Asia does not score as highly in this regard, much like with data security. The only serious deviation in the trend between data security and the degree of control citizens have over the application itself is in UAE, where the data security and privacy itself is very low, but citizens are not legally required to install the application and can deactivate the application at will. This may be because the UAE government relied more on ground teams for testing rather than contact tracing for managing the pandemic [38,45].
The significant trends to be observed with regards to privacy is that both the form of government and state of socioeconomic development seems to have a relation to the importance placed on data privacy. Stable democracies with better development indicators seem to fare better in this regard while developing countries and unstable democracies and monarchies are less so. Within developed democracies, Western countries fare better than their eastern and Middle Eastern counterparts. With exceptions, these same countries also give a greater degree of control to its citizens regarding the installation and utilization of the application. The effectiveness of the application in terms accessibility, awareness generation and decentralization does not seem to have any dependence on the developmental status or governance style of the country.

Public Perception of CTAs
At the beginning, there was a lot of uncertainty and inexperience in managing the COVID-19 pandemic. Countries around the world tackled it in different ways. Vietnam and Singapore developed strategies to eradicate the virus, Australia tried to arrest and control the transmission, whereas Sweden tried to build herd immunity. Another point to note was the response time taken by the authorities to suppress virus transmission, Italy had a comparatively slower response in contrast to South Korea which had a faster response in preventing the infection. The study reviewed 17 CTAs that targeted the COVID-19 disease from all over the world. The customer ratings of these applications are provided in Table 5.
CTAs that were centrally, federally and government sponsored were included in the study. Based on average customer ratings (>4) on Google Play Store, it was observed that certain country-specific apps had scored the higher ratings, namely Bluezone-Contact detection (Vietnam Bluezone-Contact detection app was launched in April and has the second highest ratings on Google Play (total ratings: 67,422) after India. Overall average ratings demonstrated that a high proportion of users were satisfied with the quality of the app for both Vietnamese (84%) and English (61%) versions, with the Vietnamese app scoring higher. In total, 12% gave one or two stars for the Vietnamese version, and for the English version 27%. Total reviews received from users was 19,196 with 1600 reviews/month. In total, 2914 reviews received replies with 243 replies coming every month.
Aarogya Setu app from India is one of the most popular CTAs with more than 100 million downloads and the highest number of ratings from Play Store (1,335,785). The app also received the most reviews (54,043; 4504 reviews/month) and replies to reviews (33,385; 2782 replies/month). A total of 79% of users were satisfied with the performance of this app with 16% giving it a one-or two-star rating.
Compared to all other CTAs, the average star rating (4/5) for Wiqaytna CTA (Morocco) was the highest (89%) indicating that users were satisfied with the quality and performance of this app, whereas only a small percentage (9%) disliked or hated the app. In the Play Store, 12,300 total ratings, and 4723 total reviews with 394 reviews/month was observed. The app received 309 overall replies to reviews and 26 replies/month.
CoronApp from Columbia was launched in March, making it one of the earliest CTA launches. After India, this app received the most replies to reviews. About 64,511 users rated this app, which received 35,968 reviews and 2997 reviews/month. There were 31,202 replies to reviews and 2600 replies/month. Users gave a four-or five-star rating (68%), a few gave one-or two-star ratings (23%).
Tracing apps that received the lowest user ratings (<3) were NZ COVID Tracer (2.48; New Zealand), COVIDSafe (2.75; Australia), StopCovid (2.88; France), and COCOA-COVID-19 (2.99; Japan). note was the response time taken by the authorities to suppress virus transmission, Italy had a comparatively slower response in contrast to South Korea which had a faster response in preventing the infection. The study reviewed 17 CTAs that targeted the COVID-19 disease from all over the world. The customer ratings of these applications are provided in Table 5. CTAs that were centrally, federally and government sponsored were included in the study. Based on average customer ratings (>4) on Google Play Store, it was observed that certain country-specific apps had scored the higher ratings, namely Bluezone-Contact detection (Vietnam Bluezone-Contact detection app was launched in April and has the second highest ratings on Google Play (total ratings: 67,422) after India. Overall average ratings demonstrated that a high proportion of users were satisfied with the quality of the app for both Vietnamese (84%) and English (61%) versions, with the Vietnamese app scoring higher. Implementing strict lockdown strategies, improved rapid testing facilities, and case management has enabled New Zealand to contain the COVID-19 outbreak twice. For the purpose of tracing and isolating COVID-19 contacts, New Zealand health authorities developed the NZ COVID Tracer CTA. The app received 2816 ratings on Google. The number of total reviews received was 2200 and 183 reviews/month. In total, this app received 1038 replies to reviews and 87 replies/month. User satisfaction rating (four or five stars) was observed to be 23% in contrast to 68% who gave one-or two-star ratings (hated or disliked the app).
COVIDSafe from Australia received a total of 12,909 ratings, with 7353 reviews in total and 613 reviews/month. This app received no replies to reviews. Half of the users gave a rating of four or five stars and 44% gave it one star or two stars.
In June, the StopCovid app from France was released to the public in English and received 9121 ratings. Out of 4741 reviews, this app received 395 reviews/month. An interesting observation noted with this app was that there were no replies to reviews. In total, 53% of the public gave it four or five stars and 38% gave it a one-or two-star rating.
In Japan, COCOA-COVID-19 was launched in June in the local language. It received 6295 overall ratings. This app received 3157 reviews with 263 reviews/month. Like the StopCovid app from France, this app also received no replies to reviews at all. Users who gave it a four-or five-star rating (38%) were lower than those who rated it one or two stars (46%).
It is evident that the public perception of the application has a correlation to the utility or data privacy measures taken by the application. Countries like India, Vietnam, and Canada which scored above 1.5 also scored high in customer reviews. Morocco, which scored 1.25 in the framework also has the highest overall review score. However, it is evident that the COVIDTAS framework cannot directly measure customer experience. Countries like Australia, New Zealand, Germany, France, and Spain which had COVIDTAS scores above 1.5 have relatively low average ratings, with the application in New Zealand scoring a low 2.12 average score. Surprisingly, the application from Mexico which had a low COVIDTAS score of 0.75 has a remarkably high average customer review score of 4.02. Therefore, it is best to view customer experience and the effectiveness of the application as viewed by the COVIDTAS framework as two separate metrics for assessing the application.
A more detailed analysis of the CTAs from the perspective of customer experience is provided in Table 6.
India, Vietnam, Canada, Mexico, and Morocco have a higher percentage of four-and five-star reviews, indicating greater customer satisfaction. Similarly, the number of times the application has responded to its customers is also a key indicator. When normalized by the number of reviews, it is seen that application from Colombia has responded to 83% of the reviews. Similarly, India and Israel also have high response rates of 61% and 54%, respectively. However, no direct correlation to customer satisfaction could be observed. In fact, highly rated applications from Morocco and Canada have relatively low response rates. This could partly be because customer complaints are more likely to warrant a response from the developer, leading to a larger number of responses to applications. In addition, the approach taken by the nation towards promoting the application could also play a part in the response rate. For example, in a country like India with relatively low technology literacy, the application was made necessary for various travel processes. Therefore, the application had to engage with its users and ensure that any difficulties in using the application be sorted out. Such interventions may not have been necessary in Morocco or Canada, where the application was entirely voluntary.      Some of the challenges in our study were the lack of organized and comprehensive information of CTAs and the inability to download several of them due to lack of accessibility outside of their respective countries. Hence our reliance on collection of data had depended on publicly available information. The rapidity in launch and usage of CTAs was the need of the hour and ability to inculcate the right processes or approaches to CTAs in some countries could have been unintentionally compromised.
Awareness and adoption of CTAs is no silver bullet in the battle against COVID-19 pandemic. In the last few weeks, announcements about COVID-19 vaccines is very encouraging but the citizens should not become complacent about the use of CTAs. In this regard, Govt. leadership must also offer assurance to its citizens that their identify will be concealed and emphasize the benefits of CTAs as it relates to shared public health.

Conclusions and Outlook
The current COVID-19 pandemic has shown the importance of expert-based decision making from governments, and highlighted the importance of developing a transparent and inclusive communication campaign. An excellent example of this is the development of COVID-19 smartphone apps, which are aimed at performing contact tracing at a massive scale. As discussed in this study, there are a number of technical solutions to deploy COVID-19 apps which are privacy-preserving, as well as effective when it comes to providing relevant epidemiological information to fight the pandemic. In many cases, the governments have not succeeded in conveying the message that decentralized digital contact tracing is not only able to fulfill all the ethical requirements, but it also provides invaluable information to contain the spread of the infection. This is particularly relevant in countries where less strict measures have been taken. For instance, it is surprising that Sweden has not developed a CTA, which could have significantly helped the lenient approach to the pandemic adopted by their Public Health Agency. Clearly, the current situation confronts the interests of governments, which want to gather as much information as possible to effectively tackle the COVID-19 spread, and individuals, who are concerned about their privacy and other essential ethical issues such as discrimination and stigmatization.
In this work we proposed an evaluation framework, the COVID Tracing App Scale (COVIDTAS), to assess the compliance of CTAs with all these principles. This framework builds on other ethical studies [41], which were based on the 17 Sustainable Development Goals (SDGs) of the United Nations (UN) [46], adding other aspects such as user review and ratings. This study features one of the largest sociotechnical assessments of CTAs, where 17 government-owned technical solutions are thoroughly evaluated. Our analysis indicates that the CTA by New Zealand reached the highest score (1.82), followed by the CTAs by Canada and Australia (1.67), and those from Japan and Germany (1.57). Some important comments regarding the CTAs that scored less than 1 are given next, i.e., the ones from Qatar, Brazil and Colombia. For example, the EHTERAZ tracing app from Qatar was legally enforced to the public, but it was not available in the local language. Interestingly, the CTAs that obtained low scores are generally located in areas with significant data gaps and digital divides [47], which are critical not only terms of handling the pandemic but also regarding a more sustainable global future.
The present framework can be used to analyze newly developed CTAs, so that the public can assess them and make informed decisions based on the compliance with the COVIDTAS methodology. Furthermore, we expect that a wide range of applications will rely on smartphone apps in the near future, ranging from economic issues to measurements of pollution. These apps may constitute an integral part of our lives, and we want to ensure that they will fulfill all the ethical requirements in terms of privacy and equality. Therefore, this framework can form the basis for the study of the future apps that will disrupt our societies. Future research in this area should also aim at gaining a more comprehensive understanding of the role of CTAs in managing this pandemic. In addition, as data stabilizes, more significant statistical analysis on each CTA can be carried out, identifying the best approach for data privacy, user trust, and disease tracking.