The World Wide Web is characterized by user-generated content which has resulted in various online encyclopedias. The most popular is the free online encyclopedia Wikipedia, which contains more than 51,600,000 million articles in over 300 languages. The English Wikipedia with around 6 million articles is the largest language chapter, followed by the German Wikipedia with over 2.3 million articles [1
The main feature of the website is the wiki tool, with which viewers can edit the articles directly in the web browser. In traditional encyclopedias, articles are written by experts, but in Wikipedia any web user can write an article. A system that is so open attracts many users who voluntarily write articles and keep them up to date. A major problem is that open access is very difficult to protect from manipulation and vandalism, so for example inaccurate information can be published by opportunistic or inexperienced contributors without any verification [2
Therefore, a high quality and accuracy of a Wikipedia article cannot be guaranteed. For this reason, Wikipedia has developed several user-oriented approaches for evaluating articles. The users of the platform can mark high quality articles as “featured articles” or “good articles” and mark inferior articles as “articles to be deleted”. However, this system only partially solves the problem, as only a very small part of Wikipedia is rated by them. With a view to improving the rating of articles in online encyclopedias, relevant metrics (or indicators, features, measures) can be developed to determine the quality of Wikipedia articles and their sources [3
]. Numerous approaches have already been developed on this topic. Current publications focus mainly on technical and mathematical solutions for the development of metrics. Knowledge oriented or economic factors are hardly considered [5
For this purpose, the results of a qualitative study in the form of interviews with international knowledge management experts on the topic “Benefits of metrics for determining the quality of collaborative open data pages and their sources and the main influencing factors” will be presented and explained. The collected data from the semi-structured interviews will be analysed using the Grounded Theory method in order to get verifiable and comprehensible results. The aim of this research paper is to collect interview data from which hypotheses about potentials of determining quality of open data pages can be identified.
In Section 2
, the research method and design used in the data collection is explained. Afterwards, the data collection and the selection of experts will be described. Subsequently, the collected data is analyzed and coded, hypotheses are set up with the help of a designed model in Section 3
. Finally, the results are summarized. The limitations and the conclusion complete the study. Material in the Appendix A
provide more detailed insights.
3. The Potentials of Quality Determination of Collaborative Open Data Pages
Based on data analysis with Grounded Theory, the authors defined six parameters which have a positive effect on the potentials of quality determination. The parameters used in the presented model are the results of the interviews, which have been transcribed and afterwards coded and analyzed. The difficulties have a negative impact. The designed model therefore consists of eight constructs (Figure 1
The description of the parameters is based on the expert statements within the qualitative research. To explain the designed model, each construct is considered in detail. Figure 1
shows the designed model which was developed on the basis of the analyzed interviews. Included are six constructs that have a positive influence on the benefits of metrics to determine the quality of collaborative open data pages and their sources. According to the model, the difficulties reduce the benefits examined. The designed model describes the developed relations between constructs and the dependent variable within the research question.
3.1. Potentials of Quality Determination
As can be seen in the designed model, the quality determination of collaborative open data pages has the following potentials:
Reliable and trustworthy utilization of data,
More efficient workflows for article creation and review,
Better decision making.
There might be also limitations and difficulties in implementation. The following subsections describe the individual potentials and difficulties in detail. In Section 3.9
descriptive contents are listed. These contents are not represented in the construct, but because of their relevance they are described.
3.2. Quality Improvement
A frequently named potential of the quality determination of collaborative open data pages is the resulting quality improvement of these pages. This goes in accordance with the principle ‘improvement through measurement’ or like Stefan said: “What cannot be measured cannot be improved.” Especially automated metrics are considered by some experts to be useful indicators for quality issues. Matthias is convinced that “it is at least possible to improve the quality with machine learning […]” and also Stefan thinks that “automated surface metrics are quite useful as proxies for deeper senses of quality”. But he emphasizes that “values on automated measures can only with caution be taken as indicators for true quality.” An important part of quality improvement is according to some of the experts also “[…] to find coverage gaps in Wikipedia” to achieve a “more complete encyclopedia […]”.
3.3. Better Outcomes
A further potential, which was discussed in a large part of the expert interviews, is better outcomes. If quality can be improved through measurement (Section 3.2
), data driven organizations will become more efficient in return “[…] and (so) we have the ability to have better outcomes
”. This was confirmed by expert Roland, who specifically addresses the improvement of the “accuracy of outcomes
”. Michael confirmed that “high quality information may increase the performance/ outcome of the task and low quality may cause the task to fail
.” Another part of better outcomes is, according to some experts, also the possibility of automated knowledge extraction. Because “automated extraction can be used as a baseline and to verify if that what you have is correct
.” That is what expert Daniel said in the interview, among other things.
3.4. Reliable and Trustworthy Utilization of Data
Nearly all the experts named the reliable and trustworthy utilization of data as an important potential of the quality determination of collaborative open data pages. Quality determination can be helpful for the “[…] identification of false data” (Anna) and can provide information on whether “[…] some sources (or) some information are reliable” (Karl). “Without any quality determination people may lose the trust into open data pages”, said expert Johannes, who emphasizes the importance of data trustworthiness for people.
3.5. Process Efficiency
The two most important statements regarding process efficiency as a potential of the quality determination of collaborative open data pages are firstly the paradigm ‘Garbage in, garbage out’ and secondly the statement, that the quality determination helps to uncover the root cause of the problem much earlier in the process.
The paradigm ‘Garbage in, garbage out’ was used by many experts to describe the problem of dealing with a bad matter source. Roland stated: “We can waste a lot of time and effort, because we end up doing a lot of work many times, fixing problems and reports instead of fixing the matter source.” Anna also underlines that “as the paradigm ‘garbage in, garbage out’ suggests, one needs to ensure that the quality of data is high such that the outputs of data analyses are valuable”.
This also goes along with discovering the root cause of the problem, rather than, as Roland described, trying to fix the data/information later in the process. Anna thinks that “data quality assessment helps uncover the root cause of the problem which allows one to improve the quality of the data”.
3.6. More Efficient Workflows for Article Creation and Review
Another potential of the quality determination is to allow for more efficient workflows for article creation and review. This includes better service for service providers and higher editor satisfaction. Karl states that automated metrics can help in letting the community know, when to have a look on certain pages. “The metrics will not tell if the page is bad, but metrics can tell, ‘Well this page could have issues, so have a look.’ It is kind of a tool for the collaborative process to draw attention to certain areas.”
Johannes even sees this potential as the most important one: “[…] I think the most interesting potential is as a service provider providing quality determination as a service to their customers”. He names DeepL as one similar example for an automated (translation) platform, which offers their (translation) services to any customer in the business and private sector. For Simon the term ‘Editor satisfaction’ particularly stands out: “Internally [ed.: a potential of quality determination is], more efficient workflows for article creation and review, and thus more editor satisfaction, if they can focus more on things that matter and less on management or irrelevant edits.”
3.7. Better Decision Making
In order to make high quality decisions firstly the information, that your decision is based upon, must be right. Secondly, often times you have to make the decision fast. The quality determination of open data pages facilitates both these points by giving the decision-makers a tool to weight and compare the information.
Richard, for example, says one of the most important potentials of the quality determination is to “weight the data accordingly” and Michael asserts “High quality information may increase the performance/ outcome of the task; low quality may cause the task to fail.”
Anna mentioned a major barrier with not knowing the ‘gold standard’ when determining the quality of an open data page. She says: “We don’t know the real true values (i.e., the gold standard) to determine whether it is a quality issue.”
For Johannes the biggest challenge is checking the correctness of an article, as one would need a verified source. This can be done automatically or by crowd evaluation. However, “in particular, automatically determining correctness (precision) is currently far beyond the ability of automated methods” (Stefan). So both of these methods are limited to a certain extent. Only an evaluation by an expert in the field could really check for the correctness of an article, but as Johannes states, this is “basically impossible”. This is also confirmed by Adrian: “The biggest challenge is around human labor hours. It takes a lot of time and energy to manually assess the quality of an article. Worse, the expertise necessary to assess the complete coverage of a topic is hard-won and thus rare.” He continues, “in the context of using machine classifiers to lessen the cost of human labor hours, the biggest hurdle is the limitation in NLP techniques. It is difficult to have a machine classification model assess the readability of an article. Structural characteristics are much easier to assess.” Johannes even goes as far as saying: “Furthermore, criteria regarding text quality are hard to define and capture even with humans. This is a common problem in natural language generation, where evaluation criteria like fluency are hard to interpret as even humans often do not agree.”
Finally, Stefan summarizes the difficulties in the quality determination of open data pages quite well: “More generally, I believe a challenge of automated metrics is their narrowness, i.e., they only measure a specific quantifiable dimension (e.g., number of references or number of completeness statements), yet these are only proxies for true quality, and only crudely correlate with it. Thus, good values on automated measures can only with caution be taken as indicators for true quality.”
3.9. Descriptive Content
The experts are unanimous that “Data Quality is […] a multi-dimensional construct and defined as fitness for use. […] Data quality of a free online encyclopedia page can be defined with several different dimensions (e.g., completeness, consistency, trustworthiness) but the use case determines which dimensions are relevant.” This is the opinion of the expert Anna, whose statement represents the majority of the experts. Expert Johannes added that he thinks it is useful to “separate quality into content quality (Correctness, valid sources) and text quality (well written, structure), [because] […] they are both important for a high quality article.”
When we asked about possible metrics (indicators) for determining the quality of collaborative open data pages, the experts gave us a huge variety of different answers.
Daniel mentioned that “automated metrics like the size of the page, the number of characters, the number of references or the number of links […] can maybe indicate some quality, but in the end, you need a human to review the text.” This statement coincided with the opinions of many other experts.
In addition to the second part of Daniel’s statement, some experts referred to the current collaborative rating system (Crowd evaluation) of Wikipedia for this question. Expert Alexander suggested to maintain this principle and “[…] borrow (a) reputation system as in Tripadvisor to evaluate the information quality of online encyclopedia, where the users of the information will rate it for its quality.”
The correlation between the proportion of citations per citable text and the quality of those citations is controversial among the experts. While Stefan thinks that “correct and comprehensive pages can easily be written without links and references”, Adrian thinks that the correlation between the citations and article quality is very high. Roland agrees with Adrian’s statement and finds that “the provenance of information is really an important part of quality.” Daniel notes that “having more sources […] is a good thing, but the sources should be verified that they are correct.”
Nearly half of the experts referred directly or indirectly to the editing history of the page as an indicator of its quality. Richard mentioned the “analysis of changes” as a metric for quality. Karl told us that it is important to look (by cross-tracking) “which […] pages are edited by which users. If cross-tracking says high traffic on the page or is there only one author and no one else cared this are indicators.”
Nearly all experts are of the opinion that “SEO tools seem not very well suited for assessing the overall quality of free encyclopedia pages” (Johannes). They “[…] do not think that purely statistical metric[s] can on itself give [information about] the quality of the page” (Karl).
3.9.3. Relevant Groups
The experts agree that almost everyone can benefit from the potential that the quality determination brings. No matter if you are a researcher, student, politician or a data provider.
The authors developed a model on the basis of evaluated data that investigates the potential of quality assessment of collaborative open data base pages and their sources. Therefore, the authors first conducted a semi-structured guideline interview to obtain empirical data from international experts in knowledge management. The evaluation of this study is based on Grounded Theory, which uncovered important influencing factors. As a result, the authors found six main factors that can be used as potentials of determining the quality of collaborative open data pages: Quality improvement, better outcomes, reliable and trustworthy utilization of data, process efficiency, more efficient workflows for article creation and review and better decision making.
In addition, the analysis of the data shows that difficulties can arise in determining the quality of collaborative open data base pages, which moderate the influencing factors within the designed model. i.e., there is also an influence on the relationship between each influencing factor (as an independent variable) and the potentials of quality determination of collaborative open data base pages (dependent variable). The full interview transcripts can be obtained from the corresponding author. The interview questions are listed in Table A1
5. Limitations and Threats to Validity
Nevertheless, there are still some limitations to be considered within the research. First, there are a number of very weak influencing factors that have been neglected in the designed model during the iterative approach of Grounded Theory. Because only a minority of experts have noticed such a factor or other experts argue that this factor does not play an important role for the potentials of quality determination of open data base pages.
The analysis of the results finally revealed that the experts sometimes have very different views on these comprehensive concepts. Another limitation of our qualitative research approach is the focus on a sample of only twelve different international experts. Although the authors tried to put together a representative sample of experts with different expertise in knowledge management, the designed model may differ when interviewing experts with different emphases within knowledge management. As implications for science, the model which was developed serves as the basis for a quantitative approach to validate the relationship between each influencing factor and the potentials of quality determination of collaborative open data base pages.
In a qualitative study using interviews, individual statements from experts are analyzed and their value processed accordingly. Therefore, no hard facts can be generated using statistical models. As a further investigation, a quantitative study can be carried out, which can bring in a larger amount of data by means of a survey and the developed model. This allows hard facts to be analyzed and the subject matter to be examined more closely. It would be interesting to consider another study that is carried out in five or ten years. The comparison could lead to new findings and further research areas.