Next Article in Journal
Isolation of the Sarcoplasmic Reticulum Ca2+-ATPase from Rabbit Fast-Twitch Muscle
Previous Article in Journal
A Simple Protocol for Isolating Hemolymph from Single Drosophila melanogaster Adult Flies
 
 
Search for Articles:
Title / Keyword
Author / Affiliation / Email
Journal
Article Type
 
 
Advanced Search
 
Section
Special Issue
Volume
Issue
Number
Page
 
Journals
MPs
Volume 6
Issue 5
10.3390/mps6050101
mps-logo
Submit to this Journal Review for this Journal Propose a Special Issue
► Article Menu

Article Menu

share Share announcement Help format_quote Cite question_answer Discuss in SciProfiles thumb_up
...
Endorse textsms
...
Comment
first_page
settings
Order Article Reprints
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Systematic Review

A Systematic Review to Inform the Development of a Reporting Guideline for Concept Mapping Research

by
Sandesh Pantha
1,*,
Martin Jones
2,
Pragya Gartoulla
3 and
Richard Gray
1
1
School of Nursing and Midwifery, La Trobe University, Bundoora, VIC 3086, Australia
2
Department of Rural Health, University of South Australia, Whyalla Campus, Whyalla Norrie, SA 5608, Australia
3
Australian Institute of Family Studies, Melbourne, VIC 3000, Australia
*
Author to whom correspondence should be addressed.
Methods Protoc. 2023, 6(5), 101; https://doi.org/10.3390/mps6050101
Submission received: 2 July 2023 / Revised: 28 September 2023 / Accepted: 10 October 2023 / Published: 17 October 2023
Browse Figure
Versions Notes

Abstract

:
Concept mapping is a phased, mixed-method approach that is increasingly used in health research to develop an understanding of complex phenomena. The six phases of concept mapping are preparation, idea generation, structuring (clustering and prioritization), data analysis, interpretation, and utilization of the map. The reporting of concept mapping research requires the development of a specific reporting guideline. We conducted a systematic review to identify candidate reporting items for inclusion in a reporting guideline. Three databases (MEDLINE, CINAHL, and PsycInfo) were searched to identify studies that used concept mapping methodology. We included 75 concept mapping studies published since 2019 from which we extracted information about the quality of reporting. A third of the studies focused on public health. We identified 71 candidate items that relate to the quality of reporting concept mapping research. The rationale for the study, the focus prompt, procedures for brainstorming, and structuring statements were consistently reported across the included studies. The process for developing the focus prompt, the rationale for the size of the stakeholder groups, and the process for determining the final concept map were generally not reported. The findings from the review will be used to inform the development of our reporting guideline for concept mapping research.

1. Introduction

Concept mapping is a specific type of mixed-method research that involves qualitative and quantitative procedures for data collection and analysis [1,2]. Concept mapping is used to develop deep a understanding of a complex phenomenon in a visual form [3]. Our manuscript focuses on the concept mapping methodology described by Trochim [1,4,5,6]. There are other approaches to generate concept maps, such as Novak’s method [7]; however, this is not the focus of this manuscript. There are six phases in concept mapping—preparation, idea generation, structuring of statements, data analysis, interpretation, and utilization of the map [1].
The quality of reporting concept mapping studies has not been evaluated comprehensively. Two systematic reviews have examined—albeit in a limited way—the quality of reporting of concept mapping research [8,9]. For example, Rosas and Kane [8] reported a review of 69 concept mapping studies using the Concept Systems (www.groupwisdom.com, accessed on 10 April 2022) software package and reported that authors frequently omitted important information about the research, including the flow of participants through the study and stress value, a measure of the fit between the final concept map and how participants sorted the data [8]. Donnelly [9] conducted a systematic review of 104 doctoral dissertations that used concept mapping, reporting that the included studies did not provide a comprehensive description of the methods that they applied in their research. The authors of both reviews argue a need to improve the quality of reporting of concept mapping research and suggest that a reporting guideline may be helpful in achieving this [8,9].
Guidelines are intended to enhance the quality and transparency of research reporting [10]. Generally, taking the form of a list of recommendations, reporting guidelines ensure that authors provide a detailed description of the design, conduct, and reporting of research [11] so that consumers of research (researchers, clinicians, the public) can appraise the quality of the work and determine its relevance to health care practice [12]. The first reporting guideline, CONsolidated Standards Of Reporting Trials (CONSORT), was described in 1996 [13]. Subsequently, there has been a rapid expansion in the number of reporting guidelines across a range of different research methodologies, including observational (STROBE [14], TRIPOD [15], STARD [16]), qualitative (COREQ [17], SRQR [18]), and mixed-method (GRAMMS [19]) designs.
Several authors have investigated compliance with reporting guidelines in published manuscripts [20,21,22,23,24]. For example, Ziemann et al. (2022) evaluated adherence to the STROBE checklist (Strengthening The Reporting of OBservational studies in Epidemiology) in observational studies of COVID-19 treatments [21]. The review authors reported that around half of the checklist items were addressed in the included studies [21]. Participant recruitment, potential sources of bias, and study limitations were not reported in about 9 out of 10 studies [21]. Walsh et al. [22] examined compliance with the 32-item COREQ (COmprehensive REporting of Qualitative studies) checklist in a review of 197 qualitative studies published in nursing journals [22]. Adherence with the reporting guidelines was rated depending on the number of items addressed in the manuscript (good ≥25, moderate 17–24, poor 9–16, and very poor ≤8). Compliance with the reporting guidelines was rated moderate in half of the included studies [22].
The central repository for reporting guidelines in health research is the EQUATOR (Enhancing the QUAlity and Transparency Of health Research) network. The EQUATOR is a searchable repository of 550 reporting guidelines (equator-network.org, accessed on 28 November 2022) across different methodologies. We searched the EQUATOR network for mixed-method and concept mapping reporting guidelines. None of the 16 reporting guidelines related to mixed-method research (www.equator-network.org, accessed on 28 November 2022) had a specific focus on concept mapping design.
The “concept systems” software package website (www.groupwisdom.com/gcmrg assessed on 10 April 2022) has a 10-point checklist to inform reports of concept mapping studies. The checklist is limited to studies using the concept systems software package. In addition, the process of developing the checklist has not been reported.
Given the complexity and specific methodological procedures for concept mapping, there is a robust scientific justification for developing a precise reporting guideline. Moher et al. [25] describe the process for developing a reporting guideline for inclusion in the EQUATOR network. A review of the existing literature to identify the need for a reporting guideline, determine the quality of reporting, and generate a list of potential candidate items is a pre-requisite for the development of a reporting guideline [25].

Aim of the Review

Our systematic review has two aims:
  • Review the quality of reporting of concept mapping studies to establish key areas where reporting may be sub-optimal.
  • Identify candidate items for inclusion in the reporting guideline for concept mapping research.

2. Method

The reporting of our review adheres to the Preferred Reporting guidelines for Systematic review and Meta-Analysis (PRISMA) [26] (Supplementary Document S1 is a copy of the completed PRISMA checklist). We prospectively registered our protocol for developing a concept mapping reporting guideline with the Open Science Framework (OSF) (https://osf.io/h54k6/, accessed on 1 July 2023) on 17 July 2021. Our study is listed with the EQUATOR network as a guideline under development (https://www.equator-network.org, accessed on 1 July 2023). Additionally, we published a study protocol for developing concept mapping reporting guidelines, including the details of the systematic review methodology we intended to follow [27].

2.1. Eligibility Criteria

We included any study where the authors indicated that they followed the concept mapping methodology described by Trochim (1989) [1] and where the focus of the study was on a health topic. Studies were included in the review if they were published in English in a peer-reviewed journal. No date restrictions were applied.

2.2. Information Sources

Three online databases—Medical Literature Analysis and Retrieval System (MEDLINE), PsycInfo, and Cumulative Index to Nursing and Allied Health Literature (CINAHL)—were searched through the Ovid platform. Although this is the minimum number of databases recommended by Cochrane when conducting a systematic review, we considered that to achieve the aims of our review, these databases would capture most of the relevant concept mapping research. Our initial search was carried out on 8 June 2021. We contemplated updating our initial search; however, the Cochrane handbook suggests (p. 89) that if a substantial number of papers are generated from the initial search, there is little value to conducting additional further searches [28].

2.3. Search Strategy

Donnelly [9] reported a systematic review of the quality of reporting of concept mapping in doctoral dissertations. The search concepts that we used were derived from this work. The search constructs ‘concept map*’, ‘structured conceptualization’, ‘concept systems’, and ‘Aridane’ were combined with the bullion operator ‘OR’. Our search was initially developed for MEDLINE and subsequently tailored for other databases. Complete search strategies for the three databases can be accessed as a supplementary Document (Supplementary Document S2).

2.4. Selection Process

We used Covidence, an online software package for systematic reviews, to support our research. Citations from the three databases were exported to Covidence, and duplicate citations removed. Two reviewers (SP, PG) completed title and abstract and full-text screening, again in Covidence, against eligibility criteria. A third reviewer resolved conflicts (RG). If there were multiple papers from the same study, as per the Cochrane handbook we included the first reported paper [28].

2.5. Data Collection Process

Our data extraction tool was developed based on the detailed description of concept mapping provided by Trochim [1,4,5,6]. One researcher (SP) completed data extraction against our 46-item data extraction spreadsheet [27]. All data items were coded ‘Reported’, ‘Not Reported’, or ‘Not applicable’. After completing data extraction for the first ten studies, we noted that important information about the reporting of concept mapping was not being captured. Consequently, we revised our data extraction form, adding 44 additional items, for example, whether authors indicated that they used a reporting guideline when drafting the manuscript, stakeholder involvement in statement reduction, and data interpretation. The final data extraction form has 90 items, of which 19 relate to study characteristics and 71 to the quality of reporting. Both data extraction forms are available as supplementary Documents (Supplementary Documents S3 and S4).

2.6. Synthesis Method

Characteristics of included studies were summarized using standard descriptive statistics. We reported the number and percentage of manuscripts that addressed each data item. Some data items did not apply to all included studies (for example, some studies did not include a focus prompt because statements were generated from a secondary source [a literature review]); this reduced the number of studies in the denominator. The number of studies used as the denominator is noted in the data extraction table.

3. Results

3.1. Study Selection

Figure 1 (PRISMA flowchart) shows the flow of studies through the different phases of our review. Our search identified 5260 citations, of which 258 met the inclusion criteria. Supplementary Document S5 is a list of all 258 included studies.

3.2. Amendments to the Review Protocol

We had 258 studies for data extraction. Given that this review aimed to examine the quality of reporting of concept mapping research and generate candidate items for inclusion in a reporting guideline, we considered that it was not necessary or feasible to extract data from all 258 papers to address the aim of the review. Therefore, we made a post hoc amendment to the inclusion criteria in our protocol to extract data only from studies published after 2018 (2019–2021, three years). This was primarily performed to limit the number of published concept mapping studies for data extraction. Our revised inclusion criteria limited the number of papers included in the review to 88 (Supplementary Document S6).

3.3. Studies Excluded during Data Extraction

During data extraction, we excluded an additional thirteen papers (Supplementary Document S7). Ten did not have a health focus, and two were secondary reports. Two manuscripts reported results from a single study (possible duplicate publication) [29,30]; we included the first paper (by date of submission) [29]. As the overlap between these two papers was substantial, we informed the editors of both journals on 4 January 2023. As of 30 June 2023, we had not received a response. The final list of 75 included studies can be accessed in Supplementary Document S8.

3.4. Characteristics of Included Studies

Table 1 shows the characteristics of the 75 included studies. Between one and five stakeholder groups were involved in the included studies (mean = 1.7, SD = 0.9). Published studies had a focus predominantly in public health, occupational health, and rehabilitation. Two-thirds of the included studies were conducted in Europe or North America.
The authors of 12 studies did not report the total number of participants in the study. The average number of statements used for structuring (prioritization and clustering) was 94. The final concept map, on average, had seven clusters. Seven studies included reports of the peer-review process [31,32,33,34,35,36,37]. The authors of one study indicated that they followed a reporting guideline when preparing their manuscript [38]. Two studies were prospectively registered [38,39] and an equal number reported publishing a study protocol [38,40].

3.5. Quality of Reporting of Concept Mapping Studies

There were seventy-one data extraction items related to the reporting of concept mapping research. We presented data items under eight subheadings: 1. Title and abstract, 2. Background, 3. Methods, 4. Ethics, 5. Results, 6. Discussion, 7. Limitations, and 8. Registration and study protocol. Table 2 is a summary of the quality of reporting of the included studies. The full dataset is available as Supplementary Document S4.

3.5.1. Title and Abstract (12 Items)

The authors of two-thirds of the included studies stated, in the title of the manuscript, that they were reporting concept mapping research. Virtually all studies provided a scientific justification for the study and reported details of the stakeholder groups involved in the abstract. One in four studies gave information on the focus prompt and details of where fieldwork for the study was conducted.

3.5.2. Background (Three Items)

Most studies reported the aim of the study. A rationale for concept mapping as a study design was included in half of the papers.

3.5.3. Methodology

Twenty-five data items were related to the reporting of concept mapping methods, organized under four subheadings (which related to the first four phases of concept mapping): preparation, idea generation, structuring (clustering and prioritization), and data-analysis.

Preparation (Eight Items)

Essentially all study authors reported a focus prompt for their research and gave a description of the study stakeholder groups. Fewer than a quarter of studies gave a rationale for selecting stakeholder groups for the research. The authors of 19 studies reported the processes they followed to develop the focus prompt; of these, 15 (79%) reported how stakeholders were engaged in the work.

Idea Generation (Five Items)

A description of the idea generation (brainstorming and statement reduction) procedures was included in most of the included studies. A rationale for the sample size was reported in a third of the studies. Three-quarters of studies reported detailed processes for idea synthesis.

Structuring of Statements (Six Items)

Details about whether the structuring of statements was conducted online, face-to-face, or hybrid was reported in almost all included studies. A rationale for the number of participants required for this phase of concept mapping was reported in around a third of studies.

Data Analysis (Six Items)

The three steps in the analysis of concept mapping data (binomial distance matrix, multidimensional scaling, and cluster analysis) were generally described by all study authors. Stakeholder engagement in data analysis was reported in half of the studies. The authors of one in three studies did not outline the process for determining the final cluster solution.

3.5.4. Additional Information (Four Items)

Information on the Institutional Review Board (IRB) or ethics committee that provided approval for the research was reported in more than eight out of ten studies. The ethics approval number was reported in half of the studies. Three-quarters of the studies stated the process of obtaining consent from the study participants.

3.5.5. Results

There were twenty-two data items related to the reporting of results of concept mapping studies, and these were detailed under three subheadings: participants, statements, and clusters.

Participants (Seven Items)

The demographic and clinical (where relevant) characteristics of participants were described in 68 studies. Nine out of ten studies reported the number of people who participated in idea generation (brainstorming) and the structuring of statements (prioritization and clustering). The participant response rate (the percentage of invited people who participated) was reported in half of the included studies.

Statements Generated (Six Items)

Three-quarters of the studies reported the total number of statements generated by participants during the idea generation (brainstorming) phase of the research. A complete list of included statements was reported in 90% of studies. Less than a third (n = 20, 29%) of the 68 studies (that had a rating task) reported scores for each of the included statements.

Clusters (Nine Items)

Cluster labels were reported in all studies. The authors of fewer than a quarter of the included studies reported information on the average number of clusters generated by participants.

3.5.6. Discussion (Three Items)

The authors of all studies appropriately considered the findings of their work within broader scientific literature. Half of the studies reported how the findings of the research might be utilized in clinical practice. A summary of the key findings from the study was reported in eight out of ten studies.

3.5.7. Limitations (One Item)

The limitations of the study were fully described and discussed by most authors of the included concept mapping studies.

3.5.8. Registration and Study Protocol (One Item)

The authors of three studies reported that they had registered their study or had published the study protocol.

4. Discussion

The aim of our systematic review was to describe the quality of reporting concept mapping research and identify potential candidate items to be included in a reporting checklist. Data were extracted from 75 studies published between 2019 and 2021. Overall, the authors of concept mapping research omitted important information about the conduct and reporting of different phases of concept mapping. Specifically, it was common for studies to omit a rationale for including stakeholder groups, details of participant inclusion/exclusion criteria, and information on how stakeholders were engaged in the development of the focus prompt and reduction of statements (idea synthesis). Our findings are consistent with the two previous reviews evaluating the quality of the reporting of concept mapping studies [8,9]. However, these two reviews had a discrete focus. Rosas and Kane (2012) examined studies conducted using a specific concept mapping software package [8]. Donnelly (2017) exclusively reviewed doctoral dissertations using concept mapping methodology [9]. Our study is the first review that looked at health research broadly, extending the evidence on the quality of reporting concept mapping research. The review identified potential areas for improvement in the reporting of concept mapping. Enhancing the quality of reporting enables readers to critically appraise the work [41]. The findings of this and previous reviews potentially provide a strong rationale for developing a guideline to enhance the quality and completeness of reporting concept mapping research in health.
More than half of the studies included in the data extraction were carried out using Concept Systems software. However, none of the studies that used this software followed the Concept Systems’ reporting checklist. One of the reasons for non-compliance with the checklist could be that it is not listed on the EQUATOR network, which is the publicly available repository of the reporting guidelines.
One of the aims of the review was to identify candidate items to include in a reporting guideline for concept mapping research. Moher et al. [25] suggest that a comprehensive literature review is an important part of the guideline development process, informing the identification of candidate items for inclusion in the final document. It has been common when describing the development of reporting guidelines for authors to provide scant details of the literature review that underpinned guideline development (see, for example, [15,17,42,43,44]). Tong et al. [17] report that they conducted a literature review to inform the development of the COREQ (COmprehensive REporting of Qualitative studies) reporting guidelines but did not provide a detailed explanation of how the findings from the review were incorporated in the development of the checklist [17]. Our review is intended to be consistent with the Moher et al. [25] recommendation for guideline development, providing a comprehensive and transparent summary of the strengths and limitations of the reporting of concept mapping research and candidate items for inclusion.
Our manuscript focuses on the quality of the reporting of concept mapping described by Trochim [1,4,5,6]. We acknowledge that there are other techniques to generate concept maps, such as relational concept maps, as described by Novak [7]. Novak and other colleagues have acknowledged practical differences in various approaches of concept mapping [3]. We want to reinforce that our study does not capture information on the quality of the reporting of other approaches to concept maps.

5. Limitations of the Study

Our review has some important limitations that warrant consideration. We made two major post hoc amendments to our study protocol that may have impacted the findings of our review. First, we amended our review inclusion criteria to only include studies published from 2019. This was performed primarily to restrict the number of studies in the review, as we identified substantially more studies than predicted. We would expect more complete reporting of studies published more recently, in part because of the movement to encourage the use of reporting guidelines [41], and, consequently, our observations about the quality of the reporting of concept mapping research may be an overestimate. Second, we made substantial amendments to our data extraction template whilst reviewing the included papers, potentially introducing bias into how we determined which data items to include. It would have been preferential if we had undertaken a pilot data extraction exercise to comprehensively determine the data items we intended to use for this review.
Data extraction was performed by one researcher. It would have been more rigorous if two researchers had completed this task and checked the consistency of ratings. We acknowledge this is a limitation of our review methodology. We did not update our search. As per Cochrane handbook recommendations, it would have been ideal to update the search after 12 months of the initial search. However, given that the aim of the study was to evaluate the quality of research reporting, updating the search, we argue, would have provided little additional value to the review.

6. Conclusions

Our systematic review evaluated the quality of reporting concept mapping studies in health research and identified candidate items for inclusion in a reporting guideline. Our systematic review evaluated 258 studies, of which we extracted data from 75 manuscripts published after 2018. The 71 items that relate to the quality of reporting will be used to inform the development of a reporting guideline for concept mapping research.

Supplementary Materials

The following supporting information can be downloaded at: https://www.mdpi.com/article/10.3390/mps6050101/s1, Supplementary Document S1, PRISMA checklist; Supplementary Document S2, Complete search strategy; Supplementary Document S3, Pilot data extraction tool; Supplementary Document S4, Final data extraction tool; Supplementary Document S5, List of 259 included studies; Supplementary Document S6, List of 88 studies; Supplementary Document S7, List of 13 excluded studies; and Supplementary Document S8, List of 75 studies for data extraction.

Author Contributions

S.P., M.J. and R.G. were involved at every stage of the review, including conceptualization, development of search strategy, screening, manuscript drafting, and editing. S.P. and P.G. conducted title and abstract screening and full-text screening. R.G. resolved conflicts. S.P. performed data extraction, verification, analysis, and prepared the first draft of the manuscript. RG undertook a substantial revision of the draft manuscript. All authors have read and agreed to the published version of the manuscript.

Funding

The corresponding author is supported by a La Trobe University Post Graduate Research Scholarship (LTUPRS) and a La Trobe University Full Fee Research Scholarship (LTUFFRS). This research received no external funding.

Institutional Review Board Statement

This review is exempt from institutional review board approval.

Data Availability Statement

Data extracted from the studies included in the review can be assessed in the Supplementary Documents provided with the manuscript.

Acknowledgments

We would like to acknowledge Nompilo Moyo at La Trobe University for his feedback and support of this review during search strategy development and screening articles.

Conflicts of Interest

Richard Gray is an Editor of IJERPH and Editor-in-Chief of Nursing Reports.

Review Registration

This review was registered prospectively with the Open Science Framework (osf.io/h54k6/).

References

  1. Trochim, W.M. An introduction to concept mapping for planning and evaluation. Eval. Program Plan. 1989, 12, 1–16. [Google Scholar] [CrossRef]
  2. Trochim, W.; Kane, M. Concept mapping: An introduction to structured conceptualization in health care. Int. J. Qual. Health Care 2005, 17, 187–191. [Google Scholar] [CrossRef]
  3. Cañas, A.J.; Coffey, J.W.; Carnot, M.J.; Feltovich, P.; Hoffman, R.R.; Feltovich, J.; Novak, J.D. A Summary of Literature Pertaining to the Use of Concept Mapping Techniques and Technologies for Education and Performance Support; The Institute for Human and Machine Cognition: Pensacola, FL, USA, 2003; p. 108. [Google Scholar]
  4. Kane, M.; Trochim, W. Concept Mapping for Planning and Evaluation; Sage Publications: Thousand Oaks, CA, USA, 2007; Volume 50, p. 217. [Google Scholar]
  5. Trochim, W.M. Hindsight is 20/20: Reflections on the evolution of concept mapping. Eval. Program Plan. 2017, 60, 176–185. [Google Scholar] [CrossRef]
  6. Trochim, W.M.; McLinden, D. Introduction to a special issue on concept mapping. Eval. Program Plan. 2017, 60, 166–175. [Google Scholar] [CrossRef]
  7. Novak, J.D. Concept mapping: A useful tool for science education. J. Res. Sci. Teach. 1990, 27, 937–949. [Google Scholar] [CrossRef]
  8. Rosas, S.R.; Kane, M. Quality and rigor of the concept mapping methodology: A pooled study analysis. Eval. Program Plan. 2012, 35, 236–245. [Google Scholar] [CrossRef]
  9. Donnelly, J.P. A systematic review of concept mapping dissertations. Eval. Program Plan. 2017, 60, 186–193. [Google Scholar] [CrossRef]
  10. Simera, I.; Moher, D.; Hirst, A.; Hoey, J.; Schulz, K.F.; Altman, D.G. Transparent and accurate reporting increases reliability, utility, and impact of your research: Reporting guidelines and the EQUATOR Network. BMC Med. 2010, 8, 24. [Google Scholar] [CrossRef]
  11. Simera, I.; Moher, D.; Hoey, J.; Schulz, K.F.; Altman, D.G. The EQUATOR Network and reporting guidelines: Helping to achieve high standards in reporting health research studies. Maturitas 2009, 63, 4–6. [Google Scholar] [CrossRef]
  12. Benchimol, E.I.; Manuel, D.G.; To, T.; Griffiths, A.M.; Rabeneck, L.; Guttmann, A. Development and use of reporting guidelines for assessing the quality of validation studies of health administrative data. J. Clin. Epidemiol. 2011, 64, 821–829. [Google Scholar] [CrossRef]
  13. Begg, C.; Cho, M.; Eastwood, S.; Horton, R.; Moher, D.; Olkin, I.; Pitkin, R.; Rennie, D.; Schulz, K.F.; Simel, D.; et al. Improving the quality of reporting of randomized controlled trials. JAMA 1996, 276, 637–639. [Google Scholar] [CrossRef]
  14. Elm, V.E.; Altman, D.G.; Egger, M.; Pocock, S.J.; Gøtzsche, P.C.; Vandenbroucke, J.P. The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) Statement: Guidelines for Reporting Observational Studies. Ann. Intern. Med. 2007, 147, 573–577. [Google Scholar] [CrossRef] [PubMed]
  15. Collins, G.S.; Reitsma, J.B.; Altman, D.G.; Moons, K.G. Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): The TRIPOD statement. J. Br. Surg. 2015, 102, 148–158. [Google Scholar] [CrossRef] [PubMed]
  16. Bossuyt, P.M.; Reitsma, J.B.; Bruns, D.E.; Gatsonis, C.A.; Glasziou, P.P.; Irwig, L.M.; Lijmer, J.G.; Moher, D.; Rennie, D.; De Vet, H.C. Towards complete and accurate reporting of studies of diagnostic accuracy: The STARD initiative. Radiology 2003, 226, 24–28. [Google Scholar] [CrossRef]
  17. Tong, A.; Sainsbury, P.; Craig, J. Consolidated criteria for reporting qualitative research (COREQ): A 32-item checklist for interviews and focus groups. Int. J. Qual. Health Care 2007, 19, 349–357. [Google Scholar] [CrossRef]
  18. O’Brien, B.C.; Harris, I.B.; Beckman, T.J.; Reed, D.A.; Cook, D.A. Standards for reporting qualitative research: A synthesis of recommendations. Acad. Med. 2014, 89, 1245–1251. [Google Scholar] [CrossRef]
  19. O’cathain, A.; Murphy, E.; Nicholl, J. The quality of mixed methods studies in health services research. J. Health Serv. Res. Policy 2008, 13, 92–98. [Google Scholar] [CrossRef] [PubMed]
  20. Jin, Y.; Sanger, N.; Shams, I.; Luo, C.; Shahid, H.; Li, G.; Bhatt, M.; Zielinski, L.; Bantoto, B.; Wang, M. Does the medical literature remain inadequately described despite having reporting guidelines for 21 years?–A systematic review of reviews: An update. J. Multidiscip. Healthc. 2018, 11, 495. [Google Scholar] [CrossRef] [PubMed]
  21. Ziemann, S.; Paetzolt, I.; Grüßer, L.; Coburn, M.; Rossaint, R.; Kowark, A. Poor reporting quality of observational clinical studies comparing treatments of COVID-19—A retrospective cross-sectional study. BMC Med. Res. Methodol. 2022, 22, 23. [Google Scholar] [CrossRef] [PubMed]
  22. Walsh, S.; Jones, M.; Bressington, D.; McKenna, L.; Brown, E.; Terhaag, S.; Shrestha, M.; Al-Ghareeb, A.; Gray, R. Adherence to COREQ Reporting Guidelines for Qualitative Research: A Scientometric Study in Nursing Social Science. Int. J. Qual. Methods 2020, 19, 1–9. [Google Scholar] [CrossRef]
  23. Yu, J.; Li, X.; Li, Y.; Sun, X. Quality of reporting in surgical randomized clinical trials. J. Br. Surg. 2017, 104, 296–303. [Google Scholar] [CrossRef]
  24. Hendriksma, M.; Joosten, M.H.M.A.; Peters, J.P.M.; Grolman, W.; Stegeman, I. Evaluation of the Quality of Reporting of Observational Studies in Otorhinolaryngology—Based on the STROBE Statement. PLoS ONE 2017, 12, e0169316. [Google Scholar] [CrossRef] [PubMed]
  25. Moher, D.; Schulz, K.F.; Simera, I.; Altman, D.G. Guidance for developers of health research reporting guidelines. PLoS Med. 2010, 7, e1000217. [Google Scholar] [CrossRef] [PubMed]
  26. Page, M.J.; McKenzie, J.E.; Bossuyt, P.M.; Boutron, I.; Hoffmann, T.C.; Mulrow, C.D.; Shamseer, L.; Tetzlaff, J.M.; Akl, E.A.; Brennan, S.E. The PRISMA 2020 statement: An updated guideline for reporting systematic reviews. BMJ 2021, 372, n71. [Google Scholar] [CrossRef] [PubMed]
  27. Pantha, S.; Jones, M.; Gray, R. Development of a Guideline to Enhance the Reporting of Concept Mapping Research: Study Protocol. Int. J. Environ. Res. Public Health 2022, 19, 7273. [Google Scholar] [CrossRef]
  28. Higgins, J.P.; Thomas, J.; Chandler, J.; Cumpston, M.; Li, T.; Page, M.J.; Welch, V.A. Cochrane Handbook for Systematic Reviews of Interventions; John Wiley & Sons: Hoboken, NJ, USA, 2019. [Google Scholar]
  29. Mahabir, D.F.; O’Campo, P.; Lofters, A.; Shankardass, K.; Salmon, C.; Muntaner, C. Experiences of everyday racism in Toronto’s health care system: A concept mapping study. Int. J. Equity Health 2021, 20, 74. [Google Scholar] [CrossRef]
  30. Mahabir, D.F.; O’Campo, P.; Lofters, A.; Shankardass, K.; Salmon, C.; Muntaner, C. Classism and everyday racism as experienced by racialized health care users: A concept mapping study. Int. J. Soc. Determ. Health Health Serv. 2021, 51, 350–363. [Google Scholar] [CrossRef]
  31. Aarons, G.A.; Reeder, K.; Sam-Agudu, N.A.; Vorkoper, S.; Sturke, R. Implementation determinants and mechanisms for the prevention and treatment of adolescent HIV in sub-Saharan Africa: Concept mapping of the NIH Fogarty International Center Adolescent HIV Implementation Science Alliance (AHISA) initiative. Implement. Sci. Commun. 2021, 2, 53. [Google Scholar] [CrossRef]
  32. Nielsen, K.T.; Rasmussen, M.U.; Overgaard, A.F.; Klokker, L.; Christensen, R.; Waehrens, E.E. Identifying values and preferences around the choice of analgesia for patients with acute trauma pain in emergency and prehospital settings: Using group concept mapping methodology. BMJ Open 2020, 10, e031863. [Google Scholar] [CrossRef]
  33. Urbanoski, K.; Pauly, B.; Inglis, D.; Cameron, F.; Haddad, T.; Phillips, J.; Phillips, P.; Rosen, C.; Schlotter, G.; Hartney, E.; et al. Defining culturally safe primary care for people who use substances: A participatory concept mapping study. BMC Health Serv. Res. 2020, 20, 1060. [Google Scholar] [CrossRef]
  34. Belmon, L.S.; Busch, V.; van Stralen, M.M.; Stijnman, D.P.M.; Hidding, L.M.; Harmsen, I.A.; Chinapaw, M.J.M. Child and Parent Perceived Determinants of Children’s Inadequate Sleep Health. A Concept Mapping Study. Int. J. Environ. Res. Public Health 2020, 17, 1583. [Google Scholar] [CrossRef] [PubMed]
  35. Bennett, R.J.; Fletcher, S.; Conway, N.; Barr, C. The role of the general practitioner in managing age-related hearing loss: Perspectives of general practitioners, patients and practice staff. BMC Fam. Pract. 2020, 21, 87. [Google Scholar] [CrossRef] [PubMed]
  36. Antoniou, T.; Mishra, S.; Matheson, F.; Smith-Merrill, D.; Challacombe, L.; Rowe, J.; DiCenso, A.M.; Kouyoumdjian, F.G.; Wobeser, W.; Kendall, C.; et al. Using concept mapping to inform the development of a transitional reintegration intervention program for formerly incarcerated people with HIV. BMC Health Serv. Res. 2019, 19, 761. [Google Scholar] [CrossRef]
  37. Rouhi, M.; Stirling, C.M.; Crisp, E.P. Mothers’ views of health problems in the 12 months after childbirth: A concept mapping study. J. Adv. Nurs. 2019, 75, 3702–3714. [Google Scholar] [CrossRef]
  38. Cardwell, R.; McKenna, L.; Davis, J.; Gray, R. How is clinical credibility defined in nursing? A concept mapping study. J. Clin. Nurs. 2021, 30, 2441–2452. [Google Scholar] [CrossRef] [PubMed]
  39. Strassheim, V.; Deary, V.; Webster, D.A.; Douglas, J.; Newton, J.L.; Hackett, K.L. Conceptualizing the benefits of a group exercise program developed for those with chronic fatigue: A mixed methods clinical evaluation. Disabil. Rehabil. 2021, 43, 657–667. [Google Scholar] [CrossRef] [PubMed]
  40. Gausman, J.; Othman, A.; Daas, I.; Hamad, I.; Dabobe, M.; Langer, A. How jordanian and syrian youth conceptualise their sexual and reproductive health needs: A visual exploration using concept mapping. Cult. Health Sex. 2020, 23, 176–191. [Google Scholar] [CrossRef]
  41. Moher, D. Reporting guidelines: Doing better for readers. BMC Med. 2018, 16, 233. [Google Scholar] [CrossRef]
  42. Moher, D.; Cook, D.J.; Eastwood, S.; Olkin, I.; Rennie, D.; Stroup, D.F. Improving the quality of reports of meta-analyses of randomised controlled trials: The QUOROM statement. Oncol. Res. Treat. 2000, 23, 597–602. [Google Scholar] [CrossRef]
  43. Grant, S.P.; Mayo-Wilson, E.; Melendez-Torres, G.; Montgomery, P. Reporting quality of social and psychological intervention trials: A systematic review of reporting guidelines and trial publications. PLoS ONE 2013, 8, e65442. [Google Scholar] [CrossRef]
  44. Hoffmann, T.C.; Glasziou, P.P.; Boutron, I.; Milne, R.; Perera, R.; Moher, D.; Altman, D.G.; Barbour, V.; Macdonald, H.; Johnston, M. Better reporting of interventions: Template for intervention description and replication (TIDieR) checklist and guide. BMJ 2014, 348, g1687. [Google Scholar] [CrossRef] [PubMed]
Mps 06 00101 g001
Figure 1. PRISMA flowchart.
Figure 1. PRISMA flowchart.
Mps 06 00101 g001
Table 1. Characteristics of included studies (n = 75 unless otherwise stated).
Table 1. Characteristics of included studies (n = 75 unless otherwise stated).
CharacteristicMean (SD)RangeNumber (%)
Stakeholder groups1.7 (0.9)1–5--
Participants included in the research (n = 63)60 (51)13–292--
Participants included in brainstorming (idea generation) (n = 69)66 (97)6–736--
Participants included in sorting (clustering task) (n = 73)43 (40)4–271--
Participants included in ranking (rating task) (n = 68)53 (55)13–271--
Statements generated by participants (n = 56)331 (437)45–2704--
Statements used for sorting and ranting94 (65)14–455 1--
Clusters in the final concept map7 (2)3–15--
Geographical region
North America----31 (41)
Europe----19 (25)
Australasia----12 (16)
Asia----7 (9)
Africa----3 (4)
South America----1 (1)
Global----2 (3)
Field of research
Public health----25 (33)
Occupational health----11 (15)
Rehabilitation----9 (12)
Health policy----8 (11)
Sexual and reproductive health----7 (9)
HIV Aids----6 (8)
Audiology----5 (7)
Mental health----4 (5)
Year of publication
2021 2----24 (32)
2020----25 (33)
2019----26 (35)
Use of any reporting guideline
Yes----1 (1)
No----74 (99)
Availability of peer-review report
Yes----7 (9)
No----68 (91)
Study pre-registration
Yes----2 (3)
No----73 (97)
Publication of study protocol
Yes----2 (3)
No----73 (97
Use of a reporting guideline
Yes----1 (1)
No----74 (99)
1 Five studies reported multiple concept maps. 2 Indexed in database up to 7 June 2021.
Table 2. Frequency and percentage of reporting of the data items in included studies.
Table 2. Frequency and percentage of reporting of the data items in included studies.
S. No.Reporting ItemComponentReportedNot ReportedNot ApplicablePercentage Reported 1
1Concept mapping was stated in the study titleTitle and abstract4926065
2Concept mapping was reported as a methodology in the abstractTitle and abstract714095
3Rationale (background information) of study was provided in the abstractTitle and abstract696092
4Focus question/prompt was reported in the abstractTitle and abstract1659021
5Stakeholders who participated in the study were identified in the abstractTitle and abstract687091
6Information on the phases of concept mapping was provided in the abstract Title and abstract2154028
7Study site was reported in the abstractTitle and abstract2055027
8Number of participants in the study was provided in the abstractTitle and abstract6312084
9Information on total number of statements generated in the study was provided in the abstractTitle and abstract4827064
10Number of clusters in the concept map was reported in the abstractTitle and abstract6411085
11Label for all clusters was provided in the abstractTitle and abstract4035053
12Concept mapping software used in the study was stated in the abstractTitle and abstract76809
13Rationale for the study was explainedBackground705093
14Rationale/justification for concept mapping as a study design was providedBackground4134055
15A clear aim/objective of the study was reportedBackground678089
16The development of the focus prompt was elaboratedMethods—preparation1949728
17Involvement of the stakeholders in the development of focus prompt was reportedMethods—preparation1553722
18Focus prompt used in the study was statedMethods—preparation671799
19All the stakeholder groups were identified in the manuscriptMethods—preparation723096
20Rationale for the stakeholder groups was providedMethods—preparation1758023
21Participant recruitment was elaboratedMethods—preparation6312084
22Inclusion and exclusion criteria were providedMethods—preparation3540047
23Data collection period was reported in the manuscriptMethods—preparation3540047
24The process of idea generation was outlinedMethods—idea generation669088
25Rationale was provided for the number of participants in the idea generation phaseMethods—idea generation2548234
26Information was provided on how brainstorming session was conducted (face-to-face, remote, or both)Methods—idea generation712297
27The process of idea synthesis (statement reduction) was detailedMethods—idea generation5420173
28Involvement of stakeholders in idea synthesis was reportedMethods—idea generation66818
29Rationale was provided for the number of participants engaged in structuring the statementsMethods—structuring the statements2253029
30Instructions for structuring the statements was reportedMethods—structuring the statements5223069
31Information on how statements were structured (face-to-face, remote, or both) was reported Methods—structuring the statements705093
32Web application/software used to structure the statements remotely was reportedMethods—structuring the statements38122576
33Information was provided on the duration of structuring of the statementsMethods—structuring the statements11442020
34Information was provided on the number of prioritization task and type of Likert scaleMethods—structuring the statements6807100
35Name of the software used for data analysis was reportedMethods—data analysis705093
36Authors outline the steps (statistical procedures) involved in the analysis of concept mapping dataMethods—data analysis732097
37Information was provided on how cluster solution was identifiedMethods—data analysis4926065
38The process of providing cluster labels was reportedMethods—data analysis6114081
39Information was provided on who interpreted the dataMethods—data analysis678089
40Study participants and/or stakeholders were engaged in data interpretationMethods—data analysis6510087
41Name of the review board providing ethics approval was mentionedAdditional information669088
42Authors reported the ethics approval numberAdditional information3441045
43The process of obtaining consent from participants was reportedAdditional information5520073
44Information on participant reimbursement was providedAdditional information2649035
45Authors reported the total number of participants in the studyResults—participants5025067
46Flow of participants through the different phases of concept mapping was providedResults—participants6114081
47Sample size for idea generation was reportedResults—participants685193
48Participant response rate for idea generation was statedResults—participants2251130
49Number of participants who structured the statements was reportedResults—participants741099
50Response rate was provided for the statement structuring phase of concept mappingResults—participants3342044
51Demographic characteristics were reported for all stakeholder groupsResults—participants687091
52Number (total) of statements generated by the participants was reportedResults—statements5519074
53The number of statements used for structuring phase was reportedResults—statements723096
54Number of statements beyond those generated by participants was reportedResults—statements1164015
55List of statements used to generate the concept map was providedResults—statements6312084
56Information was provided on the most and least important statementsResults—statements2048529
57Statements were classified on the basis of a go-zone graphResults—statements2840541
58Number of clusters generated by the participants (example, mean) was reportedResults—clusters1857024
59The number of cluster solutions considered for interpretation was reportedResults—clusters3441045
60All clusters were identified in the reportResults—clusters7500100
61Authors provided characteristics of the clusters identified in the studyResults—clusters4431059
62The most and least important clusters were reportedResults—clusters5117575
63Authors report the cluster bridging valueResults—clusters1164015
64Information was provided on the stress value and its significanceResults—clusters5124068
65Information was provided on the number of statements in each clusterResults—clusters2847037
66A ladder graph was computed to report prioritization between stakeholder groups or prioritization tasksResults—clusters2840541
67Authors discussed the relevance of the study resultsDiscussion741099
68A summary of findings from the study was providedDiscussion6213083
69The possible use of the results from the study was reportedDiscussion4134055
70A discussion was provided on the limitations of the studyLimitations714095
71Study was pre-registered, or protocol was published before resultsRegistration and protocol37204
1 The denominator for percentage reported is sum of ‘yes’ and ‘no’.
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

Share and Cite

MDPI and ACS Style

Pantha, S.; Jones, M.; Gartoulla, P.; Gray, R. A Systematic Review to Inform the Development of a Reporting Guideline for Concept Mapping Research. Methods Protoc. 2023, 6, 101. https://doi.org/10.3390/mps6050101

AMA Style

Pantha S, Jones M, Gartoulla P, Gray R. A Systematic Review to Inform the Development of a Reporting Guideline for Concept Mapping Research. Methods and Protocols. 2023; 6(5):101. https://doi.org/10.3390/mps6050101

Chicago/Turabian Style

Pantha, Sandesh, Martin Jones, Pragya Gartoulla, and Richard Gray. 2023. "A Systematic Review to Inform the Development of a Reporting Guideline for Concept Mapping Research" Methods and Protocols 6, no. 5: 101. https://doi.org/10.3390/mps6050101

Article Metrics

Back to TopTop