A Systematic Review to Inform the Development of a Reporting Guideline for Concept Mapping Research

Concept mapping is a phased, mixed-method approach that is increasingly used in health research to develop an understanding of complex phenomena. The six phases of concept mapping are preparation, idea generation, structuring (clustering and prioritization), data analysis, interpretation, and utilization of the map. The reporting of concept mapping research requires the development of a specific reporting guideline. We conducted a systematic review to identify candidate reporting items for inclusion in a reporting guideline. Three databases (MEDLINE, CINAHL, and PsycInfo) were searched to identify studies that used concept mapping methodology. We included 75 concept mapping studies published since 2019 from which we extracted information about the quality of reporting. A third of the studies focused on public health. We identified 71 candidate items that relate to the quality of reporting concept mapping research. The rationale for the study, the focus prompt, procedures for brainstorming, and structuring statements were consistently reported across the included studies. The process for developing the focus prompt, the rationale for the size of the stakeholder groups, and the process for determining the final concept map were generally not reported. The findings from the review will be used to inform the development of our reporting guideline for concept mapping research.


Introduction
Concept mapping is a specific type of mixed-method research that involves qualitative and quantitative procedures for data collection and analysis [1,2].Concept mapping is used to develop deep a understanding of a complex phenomenon in a visual form [3]. Our manuscript focuses on the concept mapping methodology described by Trochim [1,[4][5][6].There are other approaches to generate concept maps, such as Novak's method [7]; however, this is not the focus of this manuscript.There are six phases in concept mapping-preparation, idea generation, structuring of statements, data analysis, interpretation, and utilization of the map [1].
The quality of reporting concept mapping studies has not been evaluated comprehensively.Two systematic reviews have examined-albeit in a limited way-the quality of reporting of concept mapping research [8,9].For example, Rosas and Kane [8] reported a review of 69 concept mapping studies using the Concept Systems (www.groupwisdom.com,accessed on 10 April 2022) software package and reported that authors frequently omitted important information about the research, including the flow of participants through the study and stress value, a measure of the fit between the final concept map and how participants sorted the data [8].Donnelly [9] conducted a systematic review of 104 doctoral dissertations that used concept mapping, reporting that the included studies did not provide a comprehensive description of the methods that they applied in their research.The authors of both reviews argue a need to improve the quality of reporting of concept mapping research and suggest that a reporting guideline may be helpful in achieving this [8,9].
Guidelines are intended to enhance the quality and transparency of research reporting [10].Generally, taking the form of a list of recommendations, reporting guidelines ensure that authors provide a detailed description of the design, conduct, and reporting of research [11] so that consumers of research (researchers, clinicians, the public) can appraise the quality of the work and determine its relevance to health care practice [12].The first reporting guideline, CONsolidated Standards Of Reporting Trials (CONSORT), was described in 1996 [13].Subsequently, there has been a rapid expansion in the number of reporting guidelines across a range of different research methodologies, including observational (STROBE [14], TRIPOD [15], STARD [16]), qualitative (COREQ [17], SRQR [18]), and mixed-method (GRAMMS [19]) designs.
Several authors have investigated compliance with reporting guidelines in published manuscripts [20][21][22][23][24].For example, Ziemann et al. (2022) evaluated adherence to the STROBE checklist (Strengthening The Reporting of OBservational studies in Epidemiology) in observational studies of COVID-19 treatments [21].The review authors reported that around half of the checklist items were addressed in the included studies [21].Participant recruitment, potential sources of bias, and study limitations were not reported in about 9 out of 10 studies [21].Walsh et al. [22] examined compliance with the 32-item COREQ (COmprehensive REporting of Qualitative studies) checklist in a review of 197 qualitative studies published in nursing journals [22].Adherence with the reporting guidelines was rated depending on the number of items addressed in the manuscript (good ≥25, moderate 17-24, poor 9-16, and very poor ≤8).Compliance with the reporting guidelines was rated moderate in half of the included studies [22].
The central repository for reporting guidelines in health research is the EQUATOR (Enhancing the QUAlity and Transparency Of health Research) network.The EQUATOR is a searchable repository of 550 reporting guidelines (equator-network.org,accessed on 28 November 2022) across different methodologies.We searched the EQUATOR network for mixed-method and concept mapping reporting guidelines.None of the 16 reporting guidelines related to mixed-method research (www.equator-network.org,accessed on 28 November 2022) had a specific focus on concept mapping design.
The "concept systems" software package website (www.groupwisdom.com/gcmrgassessed on 10 April 2022) has a 10-point checklist to inform reports of concept mapping studies.The checklist is limited to studies using the concept systems software package.In addition, the process of developing the checklist has not been reported.
Given the complexity and specific methodological procedures for concept mapping, there is a robust scientific justification for developing a precise reporting guideline.Moher et al. [25] describe the process for developing a reporting guideline for inclusion in the EQUATOR network.A review of the existing literature to identify the need for a reporting guideline, determine the quality of reporting, and generate a list of potential candidate items is a pre-requisite for the development of a reporting guideline [25].

Aim of the Review
Our systematic review has two aims: 1.
Review the quality of reporting of concept mapping studies to establish key areas where reporting may be sub-optimal.2.
Identify candidate items for inclusion in the reporting guideline for concept mapping research.

Method
The reporting of our review adheres to the Preferred Reporting guidelines for Systematic review and Meta-Analysis (PRISMA) [26] (Supplementary Document S1 is a copy of the completed PRISMA checklist).We prospectively registered our protocol for developing a concept mapping reporting guideline with the Open Science Framework (OSF) (https://osf.io/h54k6/,accessed on 1 July 2023) on 17 July 2021.Our study is listed with the EQUATOR network as a guideline under development (https://www.equator-network.org,accessed on 1 July 2023).Additionally, we published a study protocol for developing concept mapping reporting guidelines, including the details of the systematic review methodology we intended to follow [27].

Eligibility Criteria
We included any study where the authors indicated that they followed the concept mapping methodology described by Trochim (1989) [1] and where the focus of the study was on a health topic.Studies were included in the review if they were published in English in a peer-reviewed journal.No date restrictions were applied.

Information Sources
Three online databases-Medical Literature Analysis and Retrieval System (MEDLINE), PsycInfo, and Cumulative Index to Nursing and Allied Health Literature (CINAHL)-were searched through the Ovid platform.Although this is the minimum number of databases recommended by Cochrane when conducting a systematic review, we considered that to achieve the aims of our review, these databases would capture most of the relevant concept mapping research.Our initial search was carried out on 8 June 2021.We contemplated updating our initial search; however, the Cochrane handbook suggests (p.89) that if a substantial number of papers are generated from the initial search, there is little value to conducting additional further searches [28].

Search Strategy
Donnelly [9] reported a systematic review of the quality of reporting of concept mapping in doctoral dissertations.The search concepts that we used were derived from this work.The search constructs 'concept map*', 'structured conceptualization', 'concept systems', and 'Aridane' were combined with the bullion operator 'OR'.Our search was initially developed for MEDLINE and subsequently tailored for other databases.Complete search strategies for the three databases can be accessed as a supplementary Document (Supplementary Document S2).

Selection Process
We used Covidence, an online software package for systematic reviews, to support our research.Citations from the three databases were exported to Covidence, and duplicate citations removed.Two reviewers (SP, PG) completed title and abstract and full-text screening, again in Covidence, against eligibility criteria.A third reviewer resolved conflicts (RG).If there were multiple papers from the same study, as per the Cochrane handbook we included the first reported paper [28].

Data Collection Process
Our data extraction tool was developed based on the detailed description of concept mapping provided by Trochim [1,[4][5][6].One researcher (SP) completed data extraction against our 46-item data extraction spreadsheet [27].All data items were coded 'Reported', 'Not Reported', or 'Not applicable'.After completing data extraction for the first ten studies, we noted that important information about the reporting of concept mapping was not being captured.Consequently, we revised our data extraction form, adding 44 additional items, for example, whether authors indicated that they used a reporting guideline when drafting the manuscript, stakeholder involvement in statement reduction, and data interpretation.The final data extraction form has 90 items, of which 19 relate to study characteristics and 71 to the quality of reporting.Both data extraction forms are available as supplementary Documents (Supplementary Documents S3 and S4).

Synthesis Method
Characteristics of included studies were summarized using standard descriptive statistics.We reported the number and percentage of manuscripts that addressed each data item.Some data items did not apply to all included studies (for example, some studies did not include a focus prompt because statements were generated from a secondary source [a literature review]); this reduced the number of studies in the denominator.The number of studies used as the denominator is noted in the data extraction table.

Synthesis Method
Characteristics of included studies were summarized using standard descriptive statistics.We reported the number and percentage of manuscripts that addressed each data item.Some data items did not apply to all included studies (for example, some studies did not include a focus prompt because statements were generated from a secondary source [a literature review]); this reduced the number of studies in the denominator.The number of studies used as the denominator is noted in the data extraction table.

Amendments to the Review Protocol
We had 258 studies for data extraction.Given that this review aimed to examine the quality of reporting of concept mapping research and generate candidate items for inclusion in a reporting guideline, we considered that it was not necessary or feasible to extract data from all 258 papers to address the aim of the review.Therefore, we made a post hoc amendment to the inclusion criteria in our protocol to extract data only from studies published after 2018 (2019-2021, three years).This was primarily performed to limit the number of published concept mapping studies for data extraction.Our revised inclusion criteria limited the number of papers included in the review to 88 (Supplementary Document S6).

Studies Excluded during Data Extraction
During data extraction, we excluded an additional thirteen papers (Supplementary Document S7).Ten did not have a health focus, and two were secondary reports.Two manuscripts reported results from a single study (possible duplicate publication) [29,30]; we included the first paper (by date of submission) [29].As the overlap between these two papers was substantial, we informed the editors of both journals on 4 January 2023.As of 30 June 2023, we had not received a response.The final list of 75 included studies can be accessed in Supplementary Document S8.

Characteristics of Included Studies
Table 1 shows the characteristics of the 75 included studies.Between one and five stakeholder groups were involved in the included studies (mean = 1.7,SD = 0.9).Published studies had a focus predominantly in public health, occupational health, and rehabilitation.Two-thirds of the included studies were conducted in Europe or North America.
1 Five studies reported multiple concept maps. 2 Indexed in database up to 7 June 2021.
The authors of 12 studies did not report the total number of participants in the study.The average number of statements used for structuring (prioritization and clustering) was 94.The final concept map, on average, had seven clusters.Seven studies included reports of the peer-review process [31][32][33][34][35][36][37].The authors of one study indicated that they followed a reporting guideline when preparing their manuscript [38].Two studies were prospectively registered [38,39] and an equal number reported publishing a study protocol [38,40].

Quality of Reporting of Concept Mapping Studies
There were seventy-one data extraction items related to the reporting of concept mapping research.We presented data items under eight subheadings: 1. Title and abstract, 2. Background, 3. Methods, 4. Ethics, 5. Results, 6. Discussion, 7. Limitations, and 8. Registration and study protocol.Table 2 is a summary of the quality of reporting of the included studies.The full dataset is available as Supplementary Document S4.The authors of two-thirds of the included studies stated, in the title of the manuscript, that they were reporting concept mapping research.Virtually all studies provided a scientific justification for the study and reported details of the stakeholder groups involved in the abstract.One in four studies gave information on the focus prompt and details of where fieldwork for the study was conducted.

Background (Three Items)
Most studies reported the aim of the study.A rationale for concept mapping as a study design was included in half of the papers.

Methodology
Twenty-five data items were related to the reporting of concept mapping methods, ganized under four subheadings (which related to the first four phases of concept mapping): preparation, idea generation, structuring (clustering and prioritization), and data-analysis.

Preparation (Eight Items)
Essentially all study authors reported a focus prompt for their research and gave a description of the study stakeholder groups.Fewer than a quarter of studies gave a rationale for selecting stakeholder groups for the research.The authors of 19 studies reported the processes they followed to develop the focus prompt; of these, 15 (79%) reported how stakeholders were engaged in the work.

Idea Generation (Five Items)
A description of the idea generation (brainstorming and statement reduction) procedures was included in most of the included studies.A rationale for the sample size was reported in a third of the studies.Three-quarters of studies reported detailed processes for idea synthesis.

Structuring of Statements (Six Items)
Details about whether the structuring of statements was conducted online, face-toface, or hybrid was reported in almost all included studies.A rationale for the number of participants required for this phase of concept mapping was reported in around a third of studies.

Data Analysis (Six Items)
The three steps in the analysis of concept mapping data (binomial distance matrix, multidimensional scaling, and cluster analysis) were generally described by all study authors.Stakeholder engagement in data analysis was reported in half of the studies.The authors of one in three studies did not outline the process for determining the final cluster solution.

Additional Information (Four Items)
Information on the Institutional Review Board (IRB) or ethics committee that provided approval for the research was reported in more than eight out of ten studies.The ethics approval number was reported in half of the studies.Three-quarters of the studies stated the process of obtaining consent from the study participants.

Results
There were twenty-two data items related to the reporting of results of concept mapping studies, and these were detailed under three subheadings: participants, statements, and clusters.the Concept Systems' reporting checklist.One of the reasons for non-compliance with the checklist could be that it is not listed on the EQUATOR network, which is the publicly available repository of the reporting guidelines.
One of the aims of the review to identify candidate items to include in a reporting guideline for concept mapping research.Moher et al. [25] suggest that a comprehensive literature review is an important part of the guideline development process, informing the identification of candidate items for inclusion in the final document.It has been common when describing the development of reporting guidelines for authors to provide scant details of the literature review that underpinned guideline development (see, for example, [15,17,[42][43][44]). Tong et al. [17] report that they conducted a literature review to inform the development of the COREQ (COmprehensive REporting of Qualitative studies) reporting guidelines but did not provide a detailed explanation of how the findings from the review were incorporated in the development of the checklist [17].Our review is intended to be consistent with the Moher et al. [25] recommendation for guideline development, providing a comprehensive and transparent summary of the strengths and limitations of the reporting of concept mapping research and candidate items for inclusion.
Our manuscript focuses on the quality of the reporting of concept mapping described by Trochim [1,[4][5][6].We acknowledge that there are other techniques to generate concept maps, such as relational concept maps, as described by Novak [7].Novak and other colleagues have acknowledged practical differences in various approaches of concept mapping [3].We want to reinforce that our study does not capture information on the quality of the reporting of other approaches to concept maps.

Limitations of the Study
Our review has some important limitations that warrant consideration.We made two major post hoc amendments to our study protocol that may have impacted the findings of our review.First, we amended our review inclusion criteria to only include studies published from 2019.This was performed primarily to restrict the number of studies in the review, as we identified substantially more studies than predicted.We would expect more complete reporting of studies published more recently, in part because of the movement to encourage the use of reporting guidelines [41], and, consequently, our observations about the quality of the reporting of concept mapping research may be an overestimate.Second, we made substantial amendments to our data extraction template whilst reviewing the included papers, potentially introducing bias into how we determined which data items to include.It would have been preferential if we had undertaken a pilot data extraction exercise to comprehensively determine the data items we intended to use for this review.
Data extraction was performed by one researcher.It would have been more rigorous if two researchers had completed this task and checked the consistency of ratings.We acknowledge this is a limitation of our review methodology.We did not update our search.As per Cochrane handbook recommendations, it would have been ideal to update the search after 12 months of the initial search.However, given that the aim of the study was to evaluate the quality of research reporting, updating the search, we argue, would have provided little additional value to the review.

Conclusions
Our systematic review evaluated the quality of reporting concept mapping studies in health research and identified candidate items for inclusion in a reporting guideline.Our systematic review evaluated 258 studies, of which we extracted data from 75 manuscripts published after 2018.The 71 items that relate to the quality of reporting will be used to inform the development of a reporting guideline for concept mapping research.

Figure 1 (
Figure 1 (PRISMA flowchart) shows the flow of studies through the different phases of our review.Our search identified 5260 citations, of which 258 met the inclusion criteria.Supplementary Document S5 is a list of all 258 included studies.

Figure 1 (
Figure 1 (PRISMA flowchart) shows the flow of studies through the different phases of our review.Our search identified 5260 citations, of which 258 met the inclusion criteria.Supplementary Document S5 is a list of all 258 included studies.

Table 1 .
Characteristics of included studies (n = 75 unless otherwise stated).

Table 2 .
Frequency and percentage of reporting of the data items in included studies.