Quality Assurance Technologies of Big Data Applications: A Systematic Literature Review

Big data applications are currently used in many application domains, ranging from statistical applications to prediction systems and smart cities. However, the quality of these applications is far from perfect, leading to a large amount of issues and problems. Consequently, assuring the overall quality for big data applications plays an increasingly important role. This paper aims at summarizing and assessing existing quality assurance (QA) technologies addressing quality issues in big data applications. We have conducted a systematic literature review (SLR) by searching major scientific databases, resulting in 83 primary and relevant studies on QA technologies for big data applications. The SLR results reveal the following main findings: 1) the impact of the big data attributes of volume, velocity, and variety on the quality of big data applications; 2) the quality attributes that determine the quality for big data applications include correctness, performance, availability, scalability, reliability and so on; 3) the existing QA technologies, including analysis, specification, model-driven architecture (MDA), verification, fault tolerance, testing, monitoring and fault&failure prediction; 4) existing strengths and limitations of each kind of QA technology; 5) the existing empirical evidence of each QA technology. This study provides a solid foundation for research on QA technologies of big data applications. However, many challenges of big data applications regarding quality still remain.


Introduction
The big data technology market grows at a 27% compound annual growth rate (CAGR), and big data market opportunities will reach over 203 billion $ in 2020 [1,2].Big data application systems [3,4], abbreviated as big data applications, refer to the software systems that can collect, process, analyze or predict a large amount of data by means of different platforms, tools and mechanisms.Big data applications are now increasing being used in many areas, such as recommendation systems, monitoring systems, and statistical applications [5,6].Big data applications are associated with the so-called 4V attributes, e.g., volume, velocity, variety and veracity [7].Due to the large amount of generated data, the fast velocity of arriving data, and the various types of heterogeneous data, the quality of data is far from ideal, which makes the software quality of big data applications far from perfect [8].For example, due to the volume and velocity attributes [9,10], the generated data of big data applications are extremely large and increasing with high speed, which may affect data accuracy and data timeliness [11], and consequently lead to software quality problems, such as performance and availability issues [11,12].Due to the huge variety of heterogeneous data [13,14], data types and formats are increasingly rich, including structured, semi-structured, and unstructured, which may affect data accessibility and data scalability, and hence lead to usability and scalability problems.
In general, quality assurance (QA) is a general way to detect or prevent mistakes or defects in manufactured software/products and avoid problems when solutions or services are delivered to customers [15].However, compared with traditional software systems, big data applications raise new challenges for QA technologies due to the four big data attributes (for example, the velocity of arriving data, and the volume of data) [16].Many scholars have illustrated current QA problems for big data applications [17], [P39].For example, it is a hard task to validate the performance, availability and accuracy of a big data prediction system due to the large-scale data size and the feature of timeliness.Because of the volume and variety attributes, keeping big data recommendation systems scalable is very difficult.Therefore, QA technologies of big data applications is now becoming a hot research topic currently.
Compared with traditional applications, big data applications have the following special characteristics: a) statistical computation based on large-scale, diverse formats, with structured and non-structured data; b) machine learning and knowledge-based system evolution; c) intelligent decision-making with uncertainty and d) more complex visualization requirements.These Numbers Key findings Implications

F1
The three main big data attributes (volume velocity, and variety) have a direct impact on the quality of the applications.
The volume, velocity and variety can affect data quality, and thus also affect the quality of big data applications.

F2
Some quality attributes, such as correctness, performance, availability, scalability and reliability, can determine the quality of big data applications.
Through our research, we can identify the most important quality attributes that state-of-the-art works address.

F3
Existing QA technologies include Analysis, Specification, Model-Driven Architecture (MDA), Testing, Fault tolerance, Fault and Failure Prediction, Monitoring and Verification.
Surveying and summarizing existing quality assurance technologies for big data applications.

F4
Existing strengths and limitations of each kind of QA technique.
Through the systematic review, strengths and limitations of each kind of QA technique are discussed and compared.

F5
Existing empirical evidence of each kind of QA technique.
Validating the proposed QA technologies through real cases and providing a reference for big data practitioners.
new features of big data applications need novel QA technologies to ensure quality.For example, compared with data in traditional applications (such as graphics, images, sounds, documents, etc.), there is a substantial amount of unstructured data in big data applications.These data are usually heterogeneous and lack of integration.Consequently, traditional testing processes lack testing methods for unstructured data and cannot adapt to the diversity of data processing requirements.Some novel QA technologies are urgently needed to solve these problems.
In the literature, many scholars have investigated the use of different QA technologies to assure the quality of big data applications [18,19,16,20].Some papers have presented overviews on quality problems of big data applications.Zhou et al. [18] presented the first comprehensive study on the quality of the big data platform.For example, they have investigated the common symptoms, causes, and mitigation of quality issues, including hardware faults, code defects and so on.Juddoo [19] et al. have systematically studied the challenges of data quality in the context of big data.Gao et al. [16] did a profound research on validation of big data and QA, including the basic concepts, issues, and validation process.They also discussed the big data QA focuses, challenges and requirements.Zhang et al. [20] introduced big data attributes and quality attributes, some quality assurance technologies like testing and monitoring were also discussed.Although these authors have proposed a few QA technologies for big data applications, publications on QA technologies for big data applications remain scattered in the literature, and this hampers the analysis of the advanced technologies and the identification of novel research directions.
Therefore, a systematic study of QA technologies for big data applications is still necessary and critical.
In this paper, we provide an exhaustive survey of QA technologies that perform a significant role in big data applications, covering 83 papers published from Jan. 2012 to Dec. 2019.The major purpose of this paper is to look into literature that is related to QA technologies for big data applications.Then, a comprehensive reference concerning challenges of QA technologies for big data applications is also proposed.In summary, the major contributions of the paper are described in the following: • The elicitation of big data attributes, and the quality problems they introduce to big data applications; • The identification of the most frequently used big data QA technologies, together with an analysis of their strengths and limitations; • A discussion of existing strengths and limitations of each kind of QA technologies.
• The proposed QA technologies are generally validated through real cases, which provides a reference for big data practitioners.
Our research results in five overall main findings that are summarized in Table 1.The findings of this paper contribute general information for future research as the quality of big data applications becomes increasingly more important.Existing QA technologies have a certain effect on the quality of big data applications; however, some challenges still exist, such as the lack of quantitative models and algorithms.
The rest of the paper is structured as follows.The next section reviews related background and previous studies.Section 3 describes our systematic approach for conducting the review.Section 4 reports the results of themes based on five research questions raised in Section 3. Section 5 provides the main findings of the survey and provides existing research challenges.Section 6 describes some threats in this study.Conclusions and future research directions are given in the final section.

Related work
To begin our study, we searched Google Scholar, Baidu Scholar, Bing Academic, IEEE, ACM and other search engines and databases (using the search strings: (systematic study OR literature review OR SLR OR SMS OR systematic literature review OR systematic mapping study) AND (Big data) AND (application OR system) AND (quality OR performance OR quality assurance OR QA))).We finally found that there is no systematic literature review (including a systematic mapping study, a systematic study, and a literature review) that focuses on QA technologies for big data applications.However, quality issues are prevalent in big data [21], and the quality of big data applications has attracted attention and been the focus of research in previous studies.In the following, we first try to describe all the relevant reviews that are truly related to the quality of big data applications.
Zhou et al. [18] firstly present comprehensive study on the quality of the big data platform.They have investigated the familiar symptoms, causes, and mitigation of quality problems.In addition, big data computing also presents different types of problems, including hardware failure, code defects and so on.Their discovery is of great significance to the design and maintenance of big data platforms in the future.
Juddoo et al. [19] systematically study the challenges of data quality in the context of big data.They mainly analyze and propose the data quality technologies that would be more suitable for big data in a general context.Their goal is to probe diverse components and activities forming part of data quality management, metrics, dimensions, data quality rules, data profiling, and data cleansing.In addition, the volume, velocity, and variety of data may make it impossible to determine the data quality rules.They believe that the measurement of big data attributes is very important to the users' decision-making.Finally, they also list existing data quality challenges.
Gao and Tao [5,16] first provide detailed discussions for QA problems and big data validation, including the basic concepts and key points.Then they discuss big data applications influenced by big data features.Furthermore, they also discuss big data validation processes, including data collection, data cleaning, data cleansing, data analysis, etc.In addition, they summarize the big data QA issues, challenges and needs.
Zhang et al. [20] further consider QA of big data applications, combined QA technologies with big data attributes, and explore the big data 4V attributes of existing QA technologies.
Liu et al. [22] point out and summarize the issues faced by big data research in data collection, processing and analysis in the current big data area, including uncertain data collection, incomplete information, and big data noise, representability, consistency, reliability and so on.
To sum up, big data applications offer many opportunities to adjust businesses and enhance promotion models.In addition, big data applications can also help governments perform accurate prediction, such as weather forecasts, preventing natural disasters, and developing appropriate policies to improve the quality of human life.The survey of existing literature (such as paper [17,18,19,20,22]) shows that there have been many studies to introduce big data QA, but little scientific research has focused on comprehending, defining, classifying and communicating QA technologies for big data applications.Consequently, there is no definite way to address QA of big data applications.As a result, it is necessary to conduct a systematic study of QA technologies for big data applications.

Research Method
In this work, a systematic literature review (SLR) approach proposed by Kitchenham et al. [23] is used to extract QA technologies for big data applications and related questions.Based on the SLR and our research problem, research steps can be raised as shown in Fig. 1.Through these research steps, we can obtain the desired results.

Research Questions
We used the Goal-Question-Metric (GQM) perspectives (i.e., purpose, issue, object, and viewpoint) [24] to draw up the aim of this study.The result of the application of the Goal-Question-Metric approach is the specification of a measurement system targeting the given set of problems and a set of rules for understanding the measurement data [25].Table 2 provides the purpose, issue, object and viewpoint of the research topics.
Research questions can usually help us to perform an indepth study and achieve purposeful research.Based on this research, there are five research questions.Table 3 shows the research questions that we translated from Table 2.The five research questions are in Table 3, and their subresearch questions are detailed below.For each research question, we also propose the primary objective of the investigation.RQ1: How do the big data attributes affect the quality of big data applications?Generally, this problem involves all big data attributes.Objective: Discuss the influence of the big data attributes on the quality of big data applications.RQ2: Which kind of important quality attributes do big data applications need to ensure?Objective: Identify and classify existing common quality attributes and understand the implication of them.RQ3: Which kinds of technologies are used to guarantee the quality of big data applications?Objective: Identify and classify existing QA technologies and understand the effect of them.Objective: Validate the proposed QA technologies through real cases and provide a reference for big data practitioners.

Search Strategy
The goal of this systematic review is thoroughly examining the literature on QA technologies for big data applications.Three main phases of SLR are presented by EBSE (evidencebased software engineering) [26] guidelines that include planning, execution, and reporting results.Moreover, the search strategy is an indispensable part and consists of two different stages.Stage 1: Database search.Digital Library the definition and validation of the search string to be used for automated search.This process started with pilot searches on seven databases as shown in Table 4.We combined different keywords that are related to research questions.Table 5 shows the search terms we used in the seven databases, and the search string is defined in the following: (a AND (b OR c) AND (d OR e OR f OR g)) IN (Title or Abstract or Keyword).
We used a "quasi-gold standard" [27] to validate and guarantee the search string,.We use IEEE and ACM libraries as representative search engines to perform automatic search and refine the search string until all the search items meet the requirements and the number of remaining papers was minimal.Then, we use the defined search string to carry out automatic searches.We choose ACM Digital Library, IEEE Xplore Digital Library, Engineering Village, Springer Link, Scopus, ISI Web of Science and Science Direct because these seven databases are the largest and most complete scientific databases that include computer science.We manually downloaded and searched the proceedings if venues not included in the digital libraries.After the automatic search, a total of 3328 papers were collected.Stage 2: Grey literature.
To cover grey literature, some alternative sources were investigated as follows: • Google Scholar In order to adapt the search terms to Google Scholar and improve the efficiency of the search process, search terms • Checking the personal websites of all the authors of primary studies, in search of other related sources (e.g., unpublished or latest progress).
Through two stages, we found 4548 related papers.Only 102 articles met the selection strategy (discussed below) and are chosen in the next stage.Then, we scanned all the related results according to the snowball method [28], and we referred to the references cited by the selected paper and include them if they are appropriate.We expanded the number of papers to 121; for example, we used this technique to find P72, which corresponds to our research questions from the references in P10.
To better manage the paper data, we used NoteExpress 1 , which is a professional-level document retrieval and management system.Its core functions cover all aspects of "knowledge acquisition, management, application, and mining".It is a perfect tool for academic research and knowledge management.However, the number of these results is too large and therefore detrimental to the study.Consequently, we filtered the results by using the selection strategy described in the next section.

Selection Strategy
In this subsection, we focus on the selection of research literature.According to the search strategy, much of the returned 1 https://noteexpress.apponic.com/literature is unnecessary.It is essential to define the selection criteria (inclusion and exclusion criteria) for selecting the related literature.We describe each step of our selection process in the following: Step 1: Combination and duplicates removal.In this step, we sort out the results that we obtain from stage 1 and stage 2 and remove the duplicate content.
Step 2: Selection of studies.In this step, the main objective is to filter all the selected literature in the light of a set of rigorous inclusion and exclusion criteria.There are five inclusion and four exclusion selection criteria we have defined as described below.
Step 3: Exclusion of literature during data extraction.When we read a study carefully, it can be selected or rejected according to the inclusion and exclusion criteria in the end.
When all inclusion criteria are met, the study is selected; otherwise, it is discarded if any exclusion criteria are met.According to the research questions and research purposes, we identified the following inclusion and exclusion criteria.Inclusion criteria: a study should be chosen if it satisfies each inclusion criteria.
1) The study of the literature focuses on the quality of big data applications or big data systems, in order to be aligned with the theme of our study.
2) One or more of our research questions must be directly answered.
3) The selected literature must be in English.4) The literature must consist of journal papers or papers published as part of conference or workshop proceedings.5) Studies are published in or after 2012.From a simple search (for which we use the search item "big data application") in the EI (engineering index) search library, we can see that most of the papers on big data applications or big data systems are published after 2011, as shown in Fig. 2. The literature published before 2012 rarely takes into account the quality of big data applications or systems.By reading the relevant literature abstracts, we found that these earlier papers were not relevant to our subject, so we excluded them.
The main objective of our study is to determine the current technologies of ensuring the quality of big data applications and the challenges associated with the quality of big data applications.This means that the content of the article should be related to the research questions of this paper.Exclusion criteria: a study should be discarded if it satisfies any one of the following exclusion criteria.1) It is related to big data but not related to the quality of big data applications.Our goal is to study the quality of big data applications or services, rather than the data quality of big data, although data quality can affect application quality.
2) It does not explicitly discuss the quality of big data applications and the impact of big data applications or quality factors of big data systems.
3) Duplicated literature.Many articles have been included in different databases, and the search results contain repeated articles.For conference papers that meet our selection criteria but are also extended to journal publications, we choose journal publications because they are more comprehensive in content.4) Studies that are not related to the research questions.
S e a r c h : (Y'epcn^"u(z ^• ) + ( 2 0 1 9^tb 2 0 1 8^tb 2 0 1 7^tb 2 0 1 6^tb 2 0 1 5^tb 2 0 1 4^tb 2 0 1 3^tb 2 0 1 2^tb 2 0 1 1^tb 2 0 1 0^tb 2 0 0 9^t ) Click to limit your results  Inclusion criteria and exclusion criteria are complementary.Consequently, both the inclusion and exclusion criteria are considered.In this way, we can achieve the desired results.A detailed process of identifying relevant literature is presented in Figure 3. Obviously, the analysis of all the literature presents a certain degree of difficulty.First, by applying these inclusion and exclusion criteria, two researchers separately read the abstracts of all studies selected in the previous step to avoid prejudice as much as possible.Of the initial studies, 488 were selected in this process.Second, for the final selection, we read the entire initial papers and then selected 102 studies.Third, we expanded the number of final studies to 121 according to our snowball method.Conflicts were resolved by extensive discussion.We excluded numbers of papers because they were not related and had 83 primary studies at the end of this step.Details of the selected papers are shown in Table 17 of Appendix B.
In the process, the first author and the second author worked together to develop research questions and search strategies, and the second author and four students of the first author executed the search plan together.During the process of finalizing the primary articles, all members of the group had a detailed discussion on whether the articles excluded by only few researchers are in line with our research topic.

Quality Assessment
After screening the final primary studies by inclusion and exclusion criteria, the criteria for the quality of the study were determined according to the guidelines proposed by Kitchenham and Charters [29].The corresponding quality checklist is shown in Table 6.The table includes 12 questions that consider four research quality dimensions, including research design, behavior, analysis, and conclusions.For each quality item we set a value of 1 if the authors put forward an explicit description, 0.5 if there is a vague description, and 0 if there is no description at all.The author and his research assistant applied the quality assessment method to each major article, compared the results, and discussed any differences until a consensus was reached.We scored each possible answer for each question in the main article and converted it into a percentage after coming to an agreement.The presentation quality assessment results of the  Nevertheless, some studies do not describe the specific division of labor in the method introduction, and there is a lack of discussion of the limitations of the proposed method.However, the total average score of 8.8 out of 12 indicates that the quality of the research report is good, supporting the validity of the extracted data and the conclusions drawn therefrom.

Data Extraction
The goal of this step is to design forms to identify and collect useful and relevant information from the selected primary studies so that it can answer our research questions proposed in Section 3.1.To carry out an in-depth analysis, we can apply the data extraction form to all selected primary studies.Table 7 shows the data extraction form.According to the data extraction form, we collect specific information in an excel file 2 .In this process, the first author and the second author jointly developed an information extraction strategy to lay the foundation for subsequent analysis.In addition, the third author validated and confirmed this research strategy.

Data Synthesis
Data synthesis is used to collect and summarize the data extracted from primary studies.Moreover, the main goal is to understand, analyze and extract current QA technologies for big  7 and some are indispensable in the research process) to determine the trends and collect information about our research questions and record them.In addition, we classify and analyze articles according to the research questions proposed in Section 3.1.
Phase 2: We classify the literature according to different research questions.The most important task is to classify the articles according to the QA technologies through the relevant analysis.

Results
This section, by deeply analyzing the primary studies listed in Appendix B, provides an answer to the five research questions presented in Section 3.For readability, we use [Px] refers to a surveyed paper listed in Appendix B.
In addition, Figures 4, 5, and 6 provide some simple statistics.Figure 4 presents how our primary studies are distributed over the years.Figure 5 groups the primary studies according to the type of publication.Figure 6 counts the number or studies retrieved from different databases.
While Section 4.1 provides an overview of the main concepts discussed in this section, Sections 4.2 to 4.6 report the answer to the research questions.

Overview of the main concepts
While answering the five research questions identified in previous sections, we will utilize and correlate three different dimensions: big data attributes, data quality parameters, and   Big data attributes: big data applications are associated with the so-called 4V attributes, e.g., volume, velocity, variety and veracity [7].In this study, we take into account only three of the 4V big data attributes (excluding the veracity one) for the following reasons: First, through the initial reading of the literature, many papers are not concerned about the veracity.Second, big data currently have multi-V attributes; only three attributes (volume, variety and velocity) are recognized extensively [30] [31].
Data quality parameters: data quality parameters describe the measure of the quality of data.Since data are an increasingly vital part of applications, data quality becomes an important concern.Poor data quality could affect enterprise revenue, waste company resources, introduce lost productivity, and even lead to wrong business decisions [16].According to the Experian Data Quality global benchmark report 3 , U.S. organizations claim that 32 percent of their data is wrong on average.Since data quality parameters are not universally agreed upon, we extracted them by analyzing papers [16,32,33].
Software quality attributes: software quality attributes describe the attributes that software systems shall expose.We start from the list provided in the ISO/IEC 25010:2011 standard and select those quality attributes that are mostly recurrent in the primary studies.Improved software quality attributes definitions are as follows: 1) A quality model consists of five characteristics (correctness, performance, availability, scalability, and reliability) that relate to the outcome of the interaction when a product is used in a particular context of use.This system model is applicable to the complete human-computer system, including both computer systems in use and software products in use.
2) A product quality model composed of eight characteristics (specification, analysis, MDA, fault tolerance, verification, testing, monitoring, fault and failure prediction) that relate to static attributes of software and dynamic attributes of the computer system.The model is applicable to both computer systems and software products.

Identify the Effect of Big Data Properties (RQ1)
The goal of this section is to answer RQ1 (how do the big data attributes affect the quality of big data applications?).Table 8 lists the general challenges proposed by big data attributes.These challenges produce great difficulties regarding the quality of the data, thus affecting the quality of big data applications.
The volume data property poses storage and scale challenges.The size of data sets used in industrial environments is huge, usually measured in terabytes, or even exabytes.Various types of data sets need a huge space to be stored and processed [P40].Application performance will decline as data volume grows.When the amount of data reaches a certain size, the application crashes and cannot provide mission services [P41].Therefore, massive amounts of data will inevitably affect the processing performance of big data applications.
The velocity data property poses fast analysis and processing challenges.With the flood of data generated quickly from smart phones and sensors, the new trend of big data analysis has shifted the focus to "what can we do with data" [34].Rapid analysis and processing of data need to be considered [P52].The data generates and processes quickly and is therefore prone to errors.Mapping MapReduce frameworks to the cloud architecture became imperative in recent years because of the need to manage large data sets in a fast, reliable (and as cheap as possible) way [P22].
The variety property poses the heterogeneity challenge, which leads to higher requirements on the data processing capacity of big data applications [17,18].The increasing amount of sensors that are deployed on the Internet makes the generated data complex.It is impossible for human beings to write every rule for each type of data to identify relevant information.As a result, most of the events in these data are unknown, abnormal and indescribable.The collection, analysis, auditing, management and testing of such a complex amount of data by industry, researchers, government and media has become a major problem [P39].
To answer RQ1, we extracted the relationships existing between the big data attributes and the data quality parameters.Table 9 identifies the primary studies discussing the relationship between couples of big data attributes and data quality parameters.The relationships that we found on the primary studies are reported in Table 9 and discussed below.
• Volume-Data Correctness: According to [P39] and [P42], the larger the volume of the data, the greater the probability that the data will be modified, deleted, and so on.In other words, a large volume of data has a high probability of errors in transmission, processing, and storage.
• Volume-Data Completeness: Data completeness is a quantitative measurement that is used to evaluate how much valid analytical data are obtained compared to the planned number [16].Data completeness is usually expressed as a percentage of usable analytical data.In general, increased data reduces data completeness [P23][P54].
• Volume-Data Timeliness: In the era of big data, people are not only concerned with the size of data but also with the way to process data.Because the amount of data to be processed is too large, the business needs and competitive pressures require a real-time and effective data processing [P24], and response time is critical in most situations [P53].Thus how to deal with a massive amount of data in a very short time is a vast challenge.If these data cannot be processed in a timely manner, the value of these data will decrease and the original goal of building big data systems will be lost [P41].
• Variety-Data Accuracy: For big data applications, the sources of data are varied, including structured, semistructured, and unstructured data.A part of these data has no statistical significance, which greatly influences the accuracy of big data application results [P55].In addition, the contents of the database became corrupted by erroneous programs storing incorrect values and deleting essential records.It is hard to recognize such quality erosion in large databases, but over time, it spreads similar to a cancerous infection, causing ever-increasing big data system failures.Thus, not only data quality but also the quality of applications suffers under erosion [P42].Overall, as summarized in Table 9, the volume property has a significant impact on all aspects of data quality, including data

Properties Challenge Description
Volume Storage/Scale The data scale has a significant effect on the performance of big data applications.
Velocity Fast ProcessingThe data generates quickly, the data are processed quickly, and the data are prone to errors.

Variety
Heterogeneity Multitype data need higher requirements on the data processing capacity of big data applications.
correctness, data timeliness, data completeness, and so on.Variety affects the data consistency, data accuracy, etc. Velocity plays an important role in data timeliness and data correctness.
We mainly determine the influences of big data attributes on big data applications.Specifically, the major data quality issues are mostly because of volume, and we have finalized five major quality parameters, including data timeliness, data completeness, data correctness, data accuracy and data consistency.

Identify the Important Quality Attributes in Big Data Applications (RQ2)
The goal of this section is to answer RQ2 (which kind of important quality attributes do big data applications need to ensure?).We use the ISO/IEC 25010:2011 to extract the quality attributes of big data applications.
Table 11 provides the statistical distribution of different quality attributes.According to statistics, we identify related quality attributes, as shown in Figure 7.For some articles that may involve more than one quality attribute, such as [P41] and [P58], we choose the main quality attribute that they convey.
From the 83 primary studies, some quality attributes are discussed, including correctness, performance, availability, scalability, reliability, efficiency, flexibility, robustness, stability, interoperability, and consistency.
From Fig. 7, we can see that the five main quality attributes have been discussed in the 83 primary studies.However, there are fewer articles focused on other attributes, such as stability,  10 shows some of the technologies that may affect correctness, performance, availability, scalability, and reliability.These technologies can help us to further understand these quality attributes.We will now focus on the main quality attributes.
• Correctness: Correctness measures the probability that big data applications can 'get things right'.If the big data application cannot guarantee the correctness, then it will have no value at all.For example, a weather forecast system that always provides the wrong weather is obviously not of any use.Therefore, correctness is the first attribute to be considered in big data applications.The papers [P25] and [P23] provide the fault tolerance mechanism to guarantee the normal operation of applications.If the big data application runs incorrectly, it will cause inconvenience or even loss to the user.The papers [P40] and [P43] provide the testing method to check the fault of big data applications to assure the correctness.
• Performance: Performance refers to the ability of big data applications to provide timely services, specifically in three areas, including the average response time, the number of transactions per unit time and the ability to maintain high-speed processing.However, volume, variety and velocity attributes of big data attributes have an impact on these three aspects.Due to the large amounts of data, performance is a key topic in big data applications.In Table 11, we show the many relative papers that refer to the performance of big data applications.The major purpose of focusing on the performance problem is to handle big data with limited resources in big data applications.To be precise, the processing performance of big data applications under massive data scenarios is its major selling point and breakthrough.According to the relevant literature, we can see that common performance optimization technologies for big data applications are generally divided into two parts . The first one is hardware and system-level observations to find specific bottlenecks and make hardware or system-level adjustments.The second one is to achieve optimization mainly through adjustments to specific software usage methods.
• Availability: Availability refers to the ability of big data applications to run without any issue for a long time.The rapid growth of data has made it necessary for big data applications to manage data streams and handle an impressive volume, and since these data types are complex (variety), the operation process may create different kinds of problems.Consequently, it is important to ensure the availability of big data applications.
• Scalability: Scalability refers to the ability of large data applications to maintain service quality when users and data volumes increase.The volume of big data attributes will inevitably bring about the scalability issue of big data applications.Specifically, the scalability of big data applications includes system scalability and data scalability.
For a continuous stream of big data, processing systems, storage systems, etc. should be able to handle these data in a scalable manner.Moreover, the system would be very complex for big data applications.For better improvement, the system must be scalable.Paper P7 proposes a flexible data analytic framework for big data applications, and the framework can flexibly handle big data with scalability.
• Reliability: Reliability refers to the ability of big data applications to apply the specified functions within the specified conditions and within the specified time.Reliability issues are usually caused by unexpected exceptions in the design and undetected code defects.For example, paper [P58] uses a monitoring technique to monitor the operational status of big data applications in real time so that failures can occur in real time and developers can effectively resolve these problems.
Although big data applications have many other related quality attributes, the most important ones are the five mentioned above.Therefore, these five quality attributes are critical to ensuring the quality of big data applications and are the main focus of this survey.Attributes Studies 4.4.Technologies for Assuring the QA of Big Data Applications (RQ3) This section answers RQ3 (which kinds of technologies are used to guarantee the quality of big data applications?).We extracted the quality assurance technologies used in the primary articles.In Fig. 8, we show the distribution of papers for these different types of QA technologies.These technologies cover the entire development process for big data applications.According to the papers we collected, we identified eight technologies in our study, i.e., specification, analysis, model-driven architecture (MDA), fault tolerance, testing, verification, monitoring, and fault and failure prediction.
In fact, quality assurance is an activity that applies to the entire big data application process.The development of big data applications is a system engineering task that includes requirements, analysis, design, implementation, and testing.Accordingly, we mainly divide the QA technologies into design time and runtime.Based on the papers we surveyed and the development process of big data applications, we divide the quality

Existing Strengths and Limitations (RQ4)
The main purpose of RQ4 (what are the advantages and limitations of the proposed technologies?) is to comprehend the strengths and limitations of QA approaches.To answer RQ4, we first compare quality assurance technologies from six aspects.Finally, we discuss the strengths and limitations of each technique.

Comparison
We compare the quality assurance technologies from six different aspects, including suitable stage, application domain, quality attribute, effectiveness, usability and efficiency, according to the study published by Patel and Hierons [35].
Suitable stage refers to the possible stage using these quality assurance technologies, including design time, runtime or both.Application domain means the specific area that big data applications belong to, such as recommendations systems and prediction systems.Quality attribute identifies the most used quality attributes being addressed by the corresponding technologies.Effectiveness means that big data application quality assurance technologies can guarantee the quality of big data applications to a certain extent.Usability refers to the extent to which quality assurance technology can guarantee the quality of big data applications in the quality assurance environment of big data applications to achieve specific goals, such that effectiveness, efficiency, and satisfaction are achieved.Efficiency refers to the improvement of the quality of big data application through the QA technologies.For better comparisons, we present the results in Table 13  4 .
From Table 13, we can see that specification, analysis, verification and MDA are used at design time.Testing, monitoring and fault and failure prediction are used at runtime.Fault tolerance covers both design time and runtime.Design-time technologies are commonly used in MapReduce, which is a great 4 the QA are those presented in Table 11.help when designing big data application frameworks.The runtime technologies are usually used after the generation of big data applications, and their application is very extensive, including intelligent systems, storage systems, cloud computing, etc.
For quality attributes, while most technologies can contend with performance and reliability, some technologies focus on correctness, scalability, etc.To a certain extent, these eight quality assurance technologies assure the quality of big data applications during their application phase, although their effectiveness, usability, and efficiency are different.To better illustrate these three quality parameters, we carry out a specific analysis through two quality assurance technologies.Specification establishes complete descriptions of information, detailed functional and behavioral descriptions, and performance requirements for big data applications to ensure the quality.Therefore, it can guarantee the integrity of the function of big data applications to achieve the designated goal of big data applications and guarantee satisfaction.Although it does not guarantee the quality of the application runtime, it guarantees the quality of the application in the initial stage.As a vital technology in the QA of big data applications, the main function of testing is to test big data applications at runtime.As a well-known concept, the purpose of testing is to detect errors and measure quality, thus ensuring effectiveness and usability.In addition, the efficiency of testing is largely concerned with testing methods and testing tools.The detailed description of other technologies is shown in Table 13.

Strengths and Limitations
• Specification: Due to the large amounts of data generated for big data applications, suitable specifications can be used to select the most useful data at hand.This technology can effectively improve the efficiency, performance and scalability of big data applications by using UML [P5], ADL [P4] and so on.The quality of the system can be guaranteed at the design stage.In addition, the specification is also used to ensure that system functions in big data applications can be implemented correctly.However, all articles are aimed at a specific application or scenario, and do not generalize to different types of big data applications [P4], [P2].
• Analysis: The main factors that affect the big data applications' quality analysis are the size of the data, the speed of data processing, and data diversity.Analysis technologies can analyze major factors that may affect the quality of software operations during the design phase of big data applications.Current approaches only focus on analyzing performance attributes [P8], [P7].There is a need to develop approaches for analyzing other quality attributes.
In addition, it is impossible to analyze all quality impact factors for big data applications.The specific conditions should be specified before analysis.
• MDA: MDA uses a single model to generate and export most codes for big data applications and can greatly reduce MDA provides a way (through related tools) to standardize a platform-independent application and select a specific implementation platform for the application, transforming application specifications to a specific implementation platform.
Fault tolerance Fault tolerance serves as an effective means to address reliability and availability concerns of big data applications.
The purpose of verification is to verify that the design of the output ensures that the design phase of the input requirements is met. Testing The purpose of testing is not only acknowledging application levels of correctness, performance and other quality attributes but also checking the testability of big data applications. [P39],

Monitoring
Monitoring can detect failures or potential anomalies at runtime and is an effective method to guarantee the quality of big data applications.
Fault and Failure Prediction The goal is to achieve the prediction of the performance status and potential anomalies of big data applications and to provide the important and abundant information for real-time control.
[  Based on the prediction results, varying degrees of effectiveness can be achieved.trast, big data application testing focuses on involute algorithms, large-scale data input, complicated models and so on.Furthermore, conventional testing and big data application testing are different in the test input, the testing execution and the results.As an example, learning-based testing approaches [P73] can test the velocity attribute of big data applications.
• Monitoring: Monitoring can obtain accurate status and behavior information for big data applications in a real operating environment.For big data applications running in a complex and variable network environment, their operating environment will affect the operation of the software system and produce some unexpected problems.Therefore, monitoring technologies will be more conducive to the timely response to the emergence of anomalies to prevent failures [P59], [P52].A stable and reliable big data application relies on monitoring technologies that not only monitor whether the service is alive or not but also monitor the operation of the system and data quality.The high velocity of big data engendered the challenge of monitoring accuracy issues and may produce overhead problems for big data applications.
• Fault and Failure Prediction: Prediction technologies can predict errors that may occur in the operation of big data applications so that errors can be prevented in advance.Due to the complexity of big data applications, the accuracy of prediction is still a substantial problem that we need to consider in the future.Deep learning-based approaches [P28], [P26], [P25] can be combined with other technologies to improve prediction accuracy due to the large amounts of data.

Empirical Evidence (RQ5)
The goal of RQ5 (what are the real cases of using the proposed technologies?) is to elicit empirical evidence on the use of QA technologies.We organize the discussion along the QA technologies discussed in Section 4.4.
• Specification.The approach in [P4] is explained by a case study of specifying and modeling a Vehicular Ad hoc NETwork (VANET).The major merits of the posed method are its capacity to take into consideration big data attributes and cyber physical system attributes through customized concepts and models in a strict, simple and expressive approach.
• Analysis.The experiments in [P7] show that the two factors that are the most important concern the quality of scientific data compression and remote visualization, which are analyzed by latency and throughput.Experiments in [P8] were conducted to analyze the connection between the performance measures of several MapReduce applications and performance concepts, such as CPU processing time.The consequences of performance analysis illustrate that the major performance measures are processing time, job turnaround and so on.Therefore, in order to improve the performance of big data applications, we must take into consideration these measures.
• MDA.In [P15], the investigators demonstrate the effectiveness of the proposed approach by using a case study.This approach can overcome accidental complexities in analytics-intensive big data applications.Paper P18 conducts a series of tests using Amazon's AWS cloud platform to evaluate performance and scalability of the observable architecture by considering the CPU, memory, and network utilization levels.Paper P19 uses a simple case study to evaluate the proposed architecture and a metamodel in the Word Count Application.show that the performance of all cases (MapReduce data computing applications) can be significantly improved.
• Verification.In [P33], the author uses CMA(cell morphology assay) as an example to describe the design of the framework.Verifying and validating datasets, software systems and algorithms in CMA demonstrates the effectiveness of the framework.
• Testing.In [P41], the authors use a number of virtual users to simulate real users and observe the average response time and CPU performance in a network public opinion monitoring system.In [P50], the experiment verifies the effectiveness and correctness of the proposed technique in alleviating the Oracle problem in a region growth program.The testing method successfully detects all the embedded mutants.
• Monitoring.The experiments in [P56] show that a large queue can increase the write speed and that the proposed framework supports a very high throughput in a reasonable amount of time in a cloud monitoring system.The authors also provide comparative tests to show the effectiveness of the framework.In [P55], the comparison experiment shows that the method is reliable and fast, especially with the increase of the data volume, and the speed advantage is obvious.
• Fault and Failure Prediction.In [P67], the authors implement the proactive failure management system and test the performance in a production cloud computing environment.Experimental results show that the approach can reach a high true positive rate and a low false positive rate for failure prediction.In [P65], the authors provide emulation-based evaluations for different sets of data traces, and the results show that the new prediction system is accurate and efficient.

Discussion
The key findings are already provided in Table 1.In this Section, we mainly discuss the cross-cutting findings and existing challenges of this review.

Cross-cutting Findings
This subsection discusses some cross-cutting findings deduced from the key findings.
• Relations between big data attributes and quality attributes.The collected results based on main findings show that big data attributes sometimes have contradictory impacts on quality attributes.Some big data attributes are found to improve some quality attributes and weaken others.These findings lead to a conclusion that big data attributes do not always improve all quality attributes of big data applications.To a certain extent, this conclusion matches the definition of big data attributes stated by most of the researchers involved in a study regarding the challenges and benefits between big data attributes and quality attributes in practice.
• Relations among big data attributes, quality attributes and big data applications.In this study, researchers have proposed some quality attributes to effectively assess the impact of big data attributes on applications.Therefore, we believe that it is incorrect to limit the research on the quality of big data applications to a certain big data property, obtain some negative results, and then state a general conclusion that comprehensive consideration of big data attributes causes big data applications' quality to weaken.For example, considering that the data that the system need to process has a large volume, fast velocity, and huge variety, a number of companies have built sophisticated monitoring and analyzing tools that go far beyond simple resource utilization reports.The continuous improvement of monitoring and analyzing tools enables big data application systems to occupy more resources, which means longer response times and lower performance[P53].Consequently, most big data applications that take into account big data attributes can cause the system to be more complex.We believe that it is incorrect to draw a general conclusion that comprehensive consideration of big data attributes negatively affects big data applications' quality.Moreover, such a conclusion does not consider other quality attributes such as reliability, scalability, and correctness.
• Relations between quality attributes and QA technologies.
It is important to note that researchers may also use different QA technologies when considering the same quality attributes.That is, there may be some empirical experience enlisted in practice.It can be inferred that the relations between quality attributes and QA technologies is not one-to-one.For example, correctness can be achieved through a variety of QA technologies, including testing, fault tolerance, verification, monitoring, and fault and failure prediction, as analyzed from Tables 11 and 12. On the other hand, when using the same QA technology, different researchers design different methods and evaluation indicators.Therefore, when a study finds that there is a negative or a positive relation between quality attributes and QA technologies, we cannot conclude a specific finding regarding the relation between them.We need to consider investigating this problem for various types of big data applications.For example, for big data applications concerned with privacy, monitoring technologies can improve the reliability of the system, but in some common big data applications, an excessive emphasis on monitoring technologies may degrade the performance of the system.Therefore, the specification of QA technologies and the relationship between QA technologies and quality attributes need to be further studied.

Existing Challenges
Based on the key findings and cross-cutting findings aforementioned, we discuss some research challenges in this subsection.
• Challenge 1: The existing problems brought by big data attributes.
Despite that many technologies have been proposed to address big data attributes, existing technologies cannot provide adequate scalability and face major difficulties.Based on the SLR results, Table 15 summarizes the challenges and the possible solutions for 3V attributes.For example, the distributed file system has a high fault tolerance, high throughput and other excellent characteristics.It can use multiple storage servers to share the storage load to store a large amount of data and support linear expansion.When the storage space is insufficient, it can use hot swap to increase storage devices and expand conveniently.These capabilities address the storage and scalability challenges of big data applications caused by the volume attribute.Many studies [37,38] show that distributed file systems can handle large-scale data very well.
For large-scale optimization and high-speed data transmission of big data applications, a decomposition-based distributed parallel programming algorithm [39] is proposed and an online algorithm is designed to dynamically adjust data partitioning and aggregation.Dobre et al. [40] review various parallel and distributed programming paradigms, analyze how they fit into the big data era, and present modern emerging paradigms and frameworks.Consequently, parallel programming is particularly effective in big data applications, especially for addressing thevelocity of data.In addition, the NoSQL [41] database is created to solve the challenges brought by the multiple data types of large-scale data collection, especially the big data application problems.NoSQL's flexible storage structure fundamentally solves the problem of variety and unstructured data storage [P23].At the same time, distributed file systems solve the problem of data storage and greatly reduce costs.It can be seen that these technologies can be combined with existing QA technologies for big data applications in the future.As mentioned in Section 5.1, because different professional skills and understanding of the field exist, big data practitioners tend to use different QA technologies when considering the same quality attributes; therefore, the QA technologies that are chosen according to experience may not be the most appropriate.Moreover, an incorrect application of QA technologies may cause extensive losses.For example, because of an incorrect transaction algorithm, the electronic trading system led to the purchase of 150 different stocks at a low price by the United States KCP (Knight Capital Group) financial companies, resulting in the company suffering a loss of 440 million US dollars with the day shares falling 62% 5 .Therefore, a clear understanding of QA technologies can reduce the implementation of incorrect algorithms and technologies in big data applications, thus avoiding huge losses.Nevertheless, the variety and diversity of big data applications makes it difficult to enact a theory of QA technologies to normalize them, which creates the challenge regarding a lack of awareness of QA technologies.In general, fully understanding the capabilities and limitations of QA technologies can address the specific needs of big data applications.Consequently, researchers are advised to fill this gap by deeply exploring theoretical research, considering more mature QA technologies, and making use of the studies frequently applied in practice.
• Challenge 3: Lack of quantitative models and algorithms to measure the relations among big data attributes, data quality parameters and software quality attributes.
The SLR results show that big data attributes are related to the quality of software.However, big data attributes should first affect multiple data quality parameters; then, the quality of data attributes affects the quality of software.Figure 9 shows our primary study on the relations among big data attributes, data quality parameters, and software quality attributes.However, the change of an attribute is often accompanied by the change of multiple attributes.More detailed theories, models and algorithms are needed to precisely understand the different kinds of relations.To specify quality requirements in the context of big data applications, paper P1 presents a novel approach to address some unique requirements of engineering challenges in big data to specify quality requirements in the context of big data applications.The approach intersects big data attributes with software quality attributes, and then it identifies the system quality requirements that apply to the intersection.Nevertheless, the approach is still in the early stages and has not been applied to the development environment of big data applications.Hence, it is still a considerable challenge and a trending research issue.
• Challenge 4: Lack of mature tools for QA technologies for big data applications.
In Section 4.4, we have summed up eight QA technologies for big data applications based on the selected 83 primary studies.Nevertheless, many authors discussed existing limitations and needed improvements.Therefore, existing technologies can solve quality problems to a certain extent.From Table 16, we can see that the 3V properties will result in software quality issues, and the corresponding technologies can partially address those problems.
However, with the wide application of machine learning in the field of big data application, the quality attributes of big data application gradually appear some new non functional attributes, such as fairness and interpretability.Processing a large amount of data needs to consume more resources of the application system.The performance of big data application is an urgent matter to be considered.Fair distribution of the resources of big data application can greatly improve the quality of big data application.For example, in distributed cloud computing, storage and bandwidth resources are usually limited, and such resources are usually very expensive, so collaborative users need to use resources fairly [P81].In [P82], an elastic online scheduling framework is proposed to guarantee big data applications fairness.Another attribute is interpretability.Inter- pretability is a subjective concept which is hard to reach a consensus.Considering the two different dimensions of semantics and complexity, when dealing with big data, researchers often pay attention to the performance indicators of big data applications, such as accuracy, but these indicators can only explain some problems, and the black box part of big data applications is still difficult to explain clearly.Due to the rapid increase of the amount of data to be processed as time goes by, the structure of big data application will gradually become more complex, which increases the difficulty of interpretation system.At this time, it is very important to study the interpretability of big data applications[P83].However, we have collected few papers about these non functional attributes so far, and the research is still in its infancy, lacking mature tools or technologies.In addition, although there are many mature QA tools for traditional software systems, none of the surveyed approaches discusses any mature tools that are dedicated to big data applications.Indeed, if practitioners want to apply QA technologies for big data applications today, they would have to implement their own tool, as there are no publicly available and maintained tools.This is also a very significant obstacle for the widespread use of QA technologies for big data applications in empirical research as well as in practice.

Threats to Validity
In the design of this study, several threats are encountered.Similar to all SLR studies, a common threat to validity regards the coverage of all relevant studies.In the following, we discuss the main threats of our study and the ways we mitigated them.
External validity: In the data collection process, most of the data are collected by three researchers; this may lead to incomplete data collection, as some related articles may be missing.Although all authors have reduced the threat by determining unclear questions and discussing them together, this threat still exists.In addition, each researcher may be biased and inflexible when he extracts the data, so at each stage of the study, we have ensured that at least two other reviewers have reviewed the work.Another potential threat is the consideration of studies which are only published in English.However, since the English language is the main used language for academical papers, this threat is considered as as minimal reasonably.
Internal validity: This SLR may have missed some related novel research papers.To alleviate this threat, we have searched for papers in big data-related journals/conferences/workshops.In total, 83 primary studies are selected by using the SLR.The possible threat is that QA technologies are not clearly shown for the selected primary studies.In addition, we endeavored as much as possible to extract information to analyze each article, which helps to avoid missing important information.This approach can minimize the threats as much as possible.
Construct validity: This concept relates to the validity of obtained data and the research questions.For the systematic literature review, it mainly addresses the selection of the main studies and how they represent the population of the questions.We have taken steps to reduce this threat in several ways.For example, the automatic search is performed on several electronic databases so as to avoid the potential biases.
Reliability: Reliability focuses on ensuring that the results are the same if our review would be conducted again.Different researchers who participated in the survey may be biased in collecting and analyzing data.To solve this threat, the two researchers simultaneously extracted and analyzed data strictly according to the screening strategy, and further discussed the differences of opinions in order to enhance the objectivity of the research results.Nevertheless, the background and experience of the researchers may have produced some prejudices and introduced a certain degree of subjectivity in some cases.This threat is also related to replicating the same findings, which in turn affects the validity of the conclusion.

Conclusions and Future Work
In this paper, we conducted a systematic literature review on QA technologies for big data applications.It mainly discusses the state-of-art technologies to ensure the quality of big data applications.Based on this, we present five research questions.To reach our goal, we applied a database search approach to identify the most relevant studies on the topic of study, and 83 primary studies are selected.Finally, we analyze the data collected from these studies to answer the research questions presented earlier in Section 3.1.
Using the SLR, a list of eight QA technologies has been identified.These technologies not only play an important role in the research of big data applications but also impact actual big data applications.Although researchers have proposed these technologies to ensure quality, the research on big data quality is, however, still in its infancy stage, and problems regarding quality still exist in big data applications.The results of this study are useful for future research in QA technologies for big data applications.Based on our discussions, the following topics may be part of our future work: • Considering quality attributes with big data properties together to ensure the quality of big data applications.
• Understanding and tapping into the limitations, advantages and applicable scenarios of QA technologies.
• Researching quantitative models and algorithms to measure the relations among big data properties, data quality attributes and software quality attributes.
• Developing mature tools to support QA technologies for big data applications.
Appendix A.
• Specification A specification refers to a type of technical standard.The specification includes requirements specification, functional specification, and non-functional specification of big data applications.Paper P3 thinks that many useless data may lead to a lot of time for data processing.
Only the correct specifications can collect valuable data and improve the efficiency of big data applications.Paper P4 proposes an approach which integrates Architecture Analysis & Design Language (AADL) to consider big data properties through customized concepts and models in a rigorous way.Paper P5 uses UML 2.3 with formal methods to standardize cloud-based applications.The semantic models can fully describe cloud security functions and ensure the secrecy of data which are stored in the cloud.Paper P6 defines a vocabulary that can integrate security requirements specifications into cloud architecture based on the unified modeling language model, which significantly improves the productivity and success rate of distributed cloud application and system development.Paper P1 presents a novel approach to address requirement engineering challenges in big data scopes.The approach intersects the big data properties with quality attributes, and then it identifies the system quality requirements that apply to the intersection.Paper P2 proposes a bloat-ware design paradigm for the development of efficient and scalable big data applications in object-oriented Garbage Collection enabled languages.

• Analysis
The analysis technique is used to analyze the main factors which can affect the big data application quality such as performance, correctness, and others, which plays an important role in the design phase of big data applications.
Paper P13 conducts a comprehensive survey on threat analysis techniques of existing software systems and analyze their applicability.Paper P8 proposes a performance analysis model based on the concept of metrology and aspects of software quality that is directly related to performance.Paper P7 proposes a framework to find the best strategy to reduce data movement in specific situations created by the large volume of data.Paper P12 restores the architecture of the system and guarantees the quality of the data intensive systems through different methods based on static analysis.Paper P9 proposes the data confidentiality challenges in big data applications.Paper P10 uses the fault tree model to model the reliability of big data applications on the cloud.The fault tree analysis (FTA) technology can provide good references for the fault processing as well as quality assurance of big data applications.Paper P11 focuses on security aspects of data acquisition, transmission, storage, decision making and other control phases.The approach is validated on some industrial big data applications with security risk.Paper P75 develops a system called OntoDMA, which is based on the cloud environment via big data analysis technique.

• Verification
Verification relies on existing technologies to validate whether the big data applications satisfy desired quality attributes.Many challenges for big data applications appear due to the 3V properties.For example, the volume and the velocity of data may bring a difficult task to validate the correctness attribute of big data applications.
To solve the challenges brought by volume and velocity of big data, paper P33 has developed a big data system called CMA (Cell Morphology Assay), and have developed a framework that rigorously can verify large-scale image data software systems and the corresponding machine learning algorithms.Paper P34 presents MtMR (merkle tree-based MapReduce), which utilize existing verification method to verify the integrity of MapReduce applications.Their analysis shows that MtMR can give a high result integrity while incurring a moderate performance overhead.Paper P35 proposes a dynamic security framework via a shared key which is driven from some synchronized numbers.Paper P37 proposes a distributed approach which exploits cloud computing facilities to verify complex systems.Paper P38 has worked with BDA application developers in industry for three years to find that traditional verification methods cannot solve the runtime and debugging problems of the applications during deployment phase.Consequently, they propose a lightweight approach which can uncover the differences between some pseudo and the large-scale cloud deployments.In this way, the efficiency and accuracy of deployment verification are improved.

• Testing
The SLR results show that different kinds of testing techniques are used to validate whether big data applications conform to requirements specification.Paper P54 introduces the data quality issues and discuss ways to monitor and control data quality.Because of the flood of data generated from smart phones and sensors, paper P52 proposes a service framework for the implementation of cloud surveillance.This framework supports realtime QoS monitoring to ensure the availability and performance of big data applications.Paper P64 proposes a system-level monitoring method for a heterogeneous big data streaming system to flexibly meet the quality of the systems used in different environments.Paper P56 proposes a monitoring framework based on big data solutions for efficiently storing and managing the data.This makes monitoring big data applications a challenging task due to the lack of standards and techniques for modeling and analysis of execution data (i.e., logs) generated by big data applications.Paper P61 presents their monitoring solution that performs real-time fault detection in big data applications.They analyze the complexity, scalability and availability of the proposed solution, which proves that it can efficiently monitor in big data applications.Based on the characteristics of big data and big data of power system, paper P58 proposes an integrated condition monitoring system of the transformer, GIS and power cable to assess system status.The technique greatly improves the accuracy of condition monitoring and diagnostics of power equipment.Paper P80 uses the Elasticsearch system to monitor and analyze big data application.

• Fault and Failure Prediction
Big data applications faces many failures.If the upcoming failure can be predicted, he overall quality of big data applications may be greatly improved.
To address this problem, Paper P72 makes a detailed failure analysis on unsuccessful execution and develops three online prediction models to improve the accuracy of fault prediction for big data systems.Paper P74 proposes a fast data update protocol for distributed storage systems that handle big data.The fault prediction mechanism proposed in this protocol improves the data update performance by 30% and the prediction accuracy reaches 80% Ṗaper P66 proposes a method to disk failure prediction that combines manufacturer telemetry data with performance counters of the server and then converts them and use a two-stage machine learning model to achieve a high degree of accuracy in predicting disk failures.Paper P67 presents an unsupervised failure detection approach using Bayesian models.It can not only characterize the normal system execution states but also can detect the anomalous behaviors.Paper P68 designs a prediction system based on a recurrent neural network (LSTM), which is a log-driven failure prediction system.

Figure
Figure 1: SLR Protocol

Figure 2 :
Figure 2: Distribution of Papers in EI until Dec 2019

Figure 3 :
Figure 3: The Search Process

Figure 6 :
Figure 6: The Number Papers from Different Databases quality attributes including stability, consistency and so on

Figure 7 :
Figure 7: The Frequency Distribution of the Quality Attributes

Figure 8 :
Figure 8: Distribution of Different QA Technologies

Figure 9 :
Figure 9: Relations among Big Data Properties, Data Quality Parameters and Software Quality Attributes

Table 1 :
Key Findings and Implications of this Research

Table 2 :
Goal of this Research

Table 3 :
Research Questions

Table 4 :
Studies Resource

Table 5 :
Search Terms -(big data AND (application OR system) AND ((quality OR performance OR QA) OR testing OR analysis OR verification OR validation)) -(big data AND (application OR system) AND (quality OR performance OR QA) AND (technique OR method)) -(big data AND (application OR system) AND (quality OR performance OR QA) AND (problem OR issue OR question))

Table 6 :
Quality Assessment Questions and Results We analyze the extracted data (most of which are included in Table

Table 7 :
Data Extraction Form Generally, the greater the velocity with which data can be analyzed is, the larger the profit for the organization [P39].Low-speed data may result in the fact that the big data systems are unable to respond effectively to any negative change (speed)[P56].Therefore, the velocity engenders challenges to data timeliness.
• Variety-Data Consistency: Data consistency is useful to evaluate the consistency of given data sets from different perspectives [P33].The consistency of various data has a positive impact on the content validity and consistency of large data application systems [P42], [P55].Unstructured data will produce the consistency problem such as continuous availability and data security issues in big data applications [P39].• Velocity-Data Timeliness: How to effectively handle high-speed data transmission and ensure the timeliness of data processing are very important.These data must be analyzed in time because the velocity of data generation is very quick,[P65], [P23].

Table 8 :
The Challenge and Description of Big Data Properties

Table 9 :
Distribution of the Relation between Big Data Properties and Data Quality Parameters

Table 10 :
Techniques for Addressing Quality Attributes Scalability Flexible data analytic framework, distributed storage system, bloat-aware design and so on Reliability Fault-tolerance mechanism, Heterogeneous NoSQL databases, Condition monitoring consistency, efficiency, and so on.In addition, from the relevant literature, we can analyze which technologies can affect the corresponding quality attributes, although there is no clear statement in the relevant literature.Table

Table 11 :
Distribution of the Quality Attributes Table 12, we have listed the simple descriptions of different types of QA technologies for big data applications.Detailed explanations of the eight technologies are provided in Appendix A.

Table 12 :
Identified QA Technologies

Table 13 :
Comparison of QA Technologies

Table 14 :
Experimental summary and statistics

Table 15 :
Properties, Challenges and Technologies

Table 16 :
3V Properties, Software Quality Attributes and Techniques of Big Data Application If the data is not filtered, it is difficult to achieve runtime monitoring.As a result, one of the biggest challenges for such applications is the real-time analysis and processing of the data.Runtime monitoring is an effective way to ensure the overall quality of big data applications.However, Runtime monitoring may occur additional loading problems for big data applications.Hence, it is necessary to improve the performance of big data monitoring.
• MonitoringPlenty of structured, unstructured, and semi-structured data are generated in big data applications.These data are very complex, huge, as well as rapidly changing.
Many machine learning methods apply to fault prediction in big data application, paper P69 aims at the comparison of 14 machine learning techniques for development of effective defect prediction models.Paper P73 analyses and compares six kinds of machine learning fault prediction methods, and confirmed the predictive ability of machine learning methods for software fault prediction in big data applications.To solve the macro integrity of software programs in software defect prediction and the correlation between local defects and surrounding program elements, paper P71 introduces complex network technology into defect prediction, establishes a software network model, and uses complex network metrics to design a set of defects that can reflect local and global measure of features, and a dynamic predictive model based on threshold filtering algorithm is proposed.