Next Article in Journal
Toward Standardized Massive Transfusion Protocols: A Multicenter Evaluation of Practice Variability Within a National Trauma System
Previous Article in Journal
A Systematic Review of Tuberculosis Stigma Reduction Interventions
 
 
Review
Peer-Review Record

Evaluating Maturity Models in Healthcare Information Systems: A Comprehensive Review

Healthcare 2025, 13(15), 1847; https://doi.org/10.3390/healthcare13151847
by Jorge Gomes 1,* and Mário Romão 2
Reviewer 1:
Reviewer 2: Anonymous
Reviewer 3:
Healthcare 2025, 13(15), 1847; https://doi.org/10.3390/healthcare13151847
Submission received: 2 July 2025 / Revised: 22 July 2025 / Accepted: 23 July 2025 / Published: 29 July 2025

Round 1

Reviewer 1 Report

Comments and Suggestions for Authors

Title : Evaluating Maturity Models in Healthcare Information Systems: A Comprehensive Review

This manuscript offers a timely and valuable literature review on the use of Maturity Models (MMs) in guiding the implementation and optimization of Healthcare Information Systems (HIS). By identifying 45 distinct MMs across various domains such as telemedicine, analytics, and electronic health records, the authors highlight the diversity and applicability of these models. The review underscores how structured maturity assessments, often grounded in CMMI principles, can enhance organizational readiness, process efficiency, and digital transformation efforts. The paper contributes meaningfully to the field by providing a comprehensive reference point for researchers and practitioners seeking to align HIS implementation with specific technological and organizational goals.

Introduction :

The introduction provides a clear and well-structured overview of the manuscript’s scope and relevance. It effectively outlines the significance of Maturity Models (MMs) in supporting Healthcare Information Systems (HIS) implementation and digital transformation. However, several points would benefit from clarification and structural improvement:

  • Some references (e.g., Aerts, 2021) are cited briefly without sufficient context or explanation of their contribution.

  • The sentence beginning with "To deliver successful projects, organizations must possess..." is duplicated (lines 98–100) and should be corrected.

  • The paragraph on CSFs (lines 130–135) introduces a new concept but does so abruptly. Consider integrating it earlier as a positive counterpart to failure factors, and possibly linking it to the maturity model framework later discussed.

In the section about Maturity Models For HIS :

  • The concept of maturity is described in multiple ways (e.g., process maturity, IS infrastructure maturity), but these dimensions are not clearly distinguished or defined.

  • Recommendation: Offer a concise typology or classification (e.g., technical maturity vs. organizational/process maturity) to clarify how different aspects of HIS maturity are evaluated.

Méthodology : 

Discussion :

This section provides a comprehensive and well-sourced overview of maturity models (MMs) in healthcare, highlighting their growing relevance for improving Health Information Systems (HIS). However, the content would benefit from clearer internal structure, reduced redundancy, and deeper analysis of key themes. The current version mixes definitions, benefits, criticisms, and technical insights without clear transitions, making it difficult to follow. To improve clarity and impact, the authors should organize the section thematically (e.g., definitions, applications, limitations), eliminate repeated points, and better contextualize MMs within healthcare-specific challenges. Adding concrete examples, distinguishing between types of maturity (e.g., technical vs. organizational), and linking back to previously identified HIS implementation challenges would enhance both coherence and practical relevance.

 

Author Response

  1. Introduction Section

 Reviewer Comment:

Some references (e.g., Aerts, 2021) are cited briefly without sufficient context or explanation of their contribution.

Response:
We revised the relevant sentences in the Introduction to provide clearer context for Aerts (2021) and other early citations. Their specific contributions to HIS implementation and maturity model adaptation are now explained to enhance reader understanding.

Reviewer Comment:

The sentence beginning with "To deliver successful projects, organizations must possess..." is duplicated (lines 98–100) and should be corrected.

Response:
The duplicated sentence has been removed. The paragraph was restructured for clarity and improved flow.

Reviewer Comment:

The paragraph on CSFs (lines 130–135) introduces a new concept but does so abruptly. Consider integrating it earlier as a positive counterpart to failure factors, and possibly linking it to the maturity model framework later discussed.

Response:
The CSFs paragraph has been moved earlier in the Introduction section and now appears immediately after the discussion of HIS implementation challenges. It is positioned as a complement to failure factors and includes a transition sentence that links CSFs to the role of maturity models, creating continuity between sections.

  1. Maturity Models for HIS Section

Reviewer Comment:

The concept of maturity is described in multiple ways (e.g., process maturity, IS infrastructure maturity), but these dimensions are not clearly distinguished or defined.
Recommendation: Offer a concise typology or classification (e.g., technical maturity vs. organizational/process maturity) to clarify how different aspects of HIS maturity are evaluated.

Response:
We added a paragraph directly following the sentence:

“In the literature, the concept of maturity is described in various ways…”

The new paragraph introduces a typology with three core dimensions: technical maturity, organizational maturity, and process maturity. Each is defined with references (e.g., Gomes & Romão, 2018a; Liaw & Godinho, 2023; Paulk et al., 1993), clarifying how these aspects interact with HIS development.

  1. Discussion Section

Reviewer Comment:

The section would benefit from clearer internal structure, reduced redundancy, and deeper analysis of key themes.
Suggestion: Organize thematically (definitions, applications, limitations), eliminate repeated points, and better contextualize MMs within healthcare-specific challenges. Add concrete examples, distinguish between types of maturity, and link back to HIS implementation challenges.

Response:
We substantially revised the Discussion as follows:

  • Clearer Structure:
    The section now contains the following subheadings:
    • 6.1 Overview of the Models Identified
    • 6.2 Comparative Analysis of the Models
    • 6.3 Practical Implications for HIS Implementation (new)
  • Eliminated Redundancy:
    We removed repetitive descriptions of CMMI, staged progression, and model structures.
  • Deeper Thematic Analysis:
    Subsection 6.2 compares models along several dimensions: scope, specialization, technical vs. strategic focus, healthcare-specific origin, and interoperability inclusion.
  • Concrete Examples:
    We highlighted applied models such as EMRAM (EHR maturity), TMSMM (telemedicine), PACS MM (imaging), MMHiAP and GDHI (policy-level assessment).
  • Link to HIS Implementation Challenges:
    A new section (6.3) was added. It explicitly connects maturity models to HIS barriers such as interoperability gaps, change resistance, and lack of contextual fit. The paragraph underscores how models offer actionable roadmaps to guide transformation.

Language Quality

Reviewer Assessment:

The English is fine and does not require any improvement.

Response:
Acknowledged with thanks. We made minor stylistic improvements during revision to further enhance clarity and consistency.

Author Response File: Author Response.pdf

Reviewer 2 Report

Comments and Suggestions for Authors

Dear Authors,
First of all, congratulations on this comprehensive effort. You've clearly invested significant time into systematically reviewing and categorizing maturity models in healthcare information systems (HIS). Your choice of organizing the models into clear categories process-oriented, technology-focused, specialized domains, data analytics, and policy-oriented is highly effective and user-friendly. I particularly liked Table 5, as it provides an excellent summary and can serve as a useful reference guide.

The paper is well-structured, and your methodology is clear and sound. Following the systematic review guidelines by Tranfield and colleagues strengthened your approach significantly. Additionally, the wide range of references you've provided 161 total demonstrates thorough knowledge of the field, real good work.

However, there are several technical issues you should carefully address some example are given below. Also, double-check all citations and references for accuracy, as there seem to be unusual date errors (such as citations dated beyond the publication year).

Overall, your manuscript represents valuable work in a timely and important area of healthcare informatics. With some careful editing to resolve technical glitches and accuracy issues, your paper will be significantly stronger. 

Your systematic approach to identifying and categorizing 45 different maturity models is noteworthy, providing a valuable resource for researchers and professionals in digital health transformation.

From a methodological standpoint, your choice of Tranfield's five-step systematic review process is sound and strengthens your paper significantly. Clearly stating your search terms, databases, and criteria, such as AIS Library, Scopus, Springer, IEEE, and similar, adds rigor and transparency. However, it would be helpful to explain more clearly your inclusion and exclusion criteria for selected papers, which are critical for replicability and validity in systematic reviews.

In terms of content, your decision to classify maturity models into five distinct categories, like process-oriented, technology-focused, specialized, data/analytics-driven, and policy-oriented, is logical and practical, but does the commercial view take those benchmarks as well?

Still, some models, particularly those adapted from other fields like general IT, require deeper scientific approaches regarding their direct applicability and validity in the healthcare context. I think discussing the limitations of these adapted models more explicitly would further strengthen your analysis.

Your selected HIS papers broadly represent the field. Yet, given the fast evolution of HIS research, more recent or emerging models, like models integrating AI or FHIR standards explicitly, might be underrepresented. Addressing cutting-edge models would enhance your review’s currency and practical relevance.

Additionally, while your review provides a solid descriptive summary of the models, it could benefit from a deeper analysis. Providing an explicit comparative framework or evaluation criteria, such as ease of implementation, validation status, empirical effectiveness, commercial viability, and market status, would add significant scientific value.

Best regards,
Recommended Corrections and Updates,

  • Use a better naming for Table 1, repeating twice the name
  • Page 3 line 99-101, "To deliver successfull Projects" repeated twice
  • Line 535 is null line check
  • Table 3 is orphan check for the headers and table on same page
  • Line 930 “consent management, and audit trails to safeguard patient dat” data is “a” missing

Author Response

Reviewer Comment:

“It would be helpful to explain more clearly your inclusion and exclusion criteria for selected papers...”

Response:
We added a dedicated subsection in the Methodology titled “4.1 Inclusion and Exclusion Criteria.” This subsection now clearly outlines:

  • What types of studies were included (peer-reviewed, HIS/MM-related, English language, etc.)

  • What was excluded (non-healthcare contexts, non-empirical studies, etc.)

Action Taken: Section 4.1 – Inclusion and Exclusion Criteria (Page 4)

2. Deepen Discussion on Adapted Models

Reviewer Comment:

“Some models... require deeper scientific approaches regarding their direct applicability..."

Response:
We expanded Section 6.3.4: Limitations of Adapted Models from Other Sectors to include:

  • Scientific limitations of generic models (e.g., lack of patient-centered indicators, workflow alignment)

  • A concrete example using CMMI and CSCMM

  • Discussion of empirical validation gaps in clinical settings

Action Taken: Expanded Section 6.3.4 (Page 29)

3. Mention Emerging Models (AI, FHIR)

Reviewer Comment:

“More recent or emerging models, like models integrating AI or FHIR standards explicitly, might be underrepresented..."

Response:
We updated Section 6.6.5: Adaptation for Emerging Technologies and Challenges to explicitly mention:

  • AI decision support tools

  • The Fast Healthcare Interoperability Resources (FHIR) standard

  • The need for future models to assess ethical and real-time integration

Action Taken: Revised Section 6.6.5 (Page 31)

? 4. Add a Comparative Evaluation Framework

Reviewer Comment:

“Providing an explicit comparative framework or evaluation criteria... would add significant scientific value.”

Response:
To address this, we added a paragraph in Section 6.2: Comparative Analysis, outlining five potential evaluation criteria:

  • Ease of implementation

  • Empirical validation

  • Contextual adaptability

  • Stakeholder usability

  • Market adoption

We also noted the challenge of standardized benchmarking due to model heterogeneity.

Action Taken: Paragraph added to Section 6.2 (Page 27)

5. Address the “Commercial View” Question

Reviewer Comment:

“Does the commercial view take those benchmarks as well?”

Response:
We added a new subsection 6.5.6: Limited Coverage of Commercial Benchmarking Models, discussing:

  • Benchmark scoring in commercial models like EMRAM and INFRAM

  • Their widespread use among providers

  • The difference in focus compared to academic models (e.g., certification vs. clinical impact)

Action Taken: New Section 6.5.6 (Page 30)

6. Fix Technical and Formatting Errors

Issue Action
Repetition at line 99–101  Fixed
Table 1 naming redundancy  Fixed
Blank line at 535  Fixed
Orphaned Table 3  Fixed
Typo “dat” at line 930  Fixed (“data”)

 

All formatting and typographic issues were resolved.

Action Taken: Across manuscript (Pages 3, 7, 12, etc.)

Author Response File: Author Response.pdf

Reviewer 3 Report

Comments and Suggestions for Authors

This manuscript presents a comprehensive review of maturity models in healthcare information systems (HIS), a topic of great relevance in the current era of digital transformation in healthcare. The paper is clearly structured, well-written, and supported by a thorough literature review. The methodological approach is appropriate, and the inclusion of various maturity models and their comparison provides valuable insights.

However, to improve the overall quality and scientific rigor of the manuscript, I suggest the following:

Clarify the Selection Criteria:
The manuscript lacks clear inclusion and exclusion criteria for selecting the maturity models. Please detail how the models were selected, and what methodological framework (e.g., PRISMA, PICOS) was followed to ensure systematicity.

Expand on Comparative Analysis:
While various models are described, a deeper comparative synthesis (e.g., using a summary matrix or scoring system) would enhance the reader’s ability to distinguish their strengths, limitations, and applicability across contexts (e.g., hospitals vs. outpatient care).

Improve Methodological Transparency:
It is unclear whether a formal scoping review or systematic review methodology was used. Explicitly stating the type of review and adhering to recognized reporting standards (e.g., PRISMA-ScR) would strengthen scientific soundness.

Limitations Section:
A formal section discussing the limitations of the review is lacking. This would be particularly important to acknowledge any publication bias, limited geographic diversity, or model validation issues.

Figure/Table Enhancement:
The visual elements (figures/tables) are informative but could benefit from standardization in terms of formatting and structure. Consider using a unified legend or layout to improve clarity.

References and Terminology:
While the references are largely appropriate, ensure that the terminology used in defining maturity models is consistent and aligns with standard frameworks in the literature (e.g., ISO/IEC 33001 family for process assessment).

Once these issues are addressed, the manuscript has the potential to serve as a key reference for both scholars and practitioners in health informatics and healthcare management.

Author Response

  1. Clarify the Selection Criteria

Reviewer Comment:

"The manuscript lacks clear inclusion and exclusion criteria. It would be helpful to detail how the models were selected..."

Response:
We have added a dedicated subsection titled “4.1 Inclusion and Exclusion Criteria” within the Methodology section. This clearly outlines the criteria used to include or exclude studies, covering publication types, time frame, language, domain relevance, and empirical grounding. This change increases the transparency and reproducibility of our review process.

Action Taken: See Section 4.1 in the revised manuscript

  1. Expand on Comparative Analysis

Reviewer Comment:

"A deeper comparative synthesis of the maturity models... would enhance the reader’s ability to distinguish their strengths, limitations, and applicability."

Response:
We retained our structured narrative synthesis of the models while also expanding the discussion in Section 6.2 to outline five key dimensions for future comparative evaluations:

  • Ease of implementation
  • Level of empirical validation
  • Contextual adaptability
  • Stakeholder usability
  • Market or institutional adoption

Although we did not build a quantitative comparison table due to heterogeneity in reporting, this paragraph frames a formal evaluation approach for future studies.

Action Taken: Paragraph added to Section 6.2 – Comparative Analysis of the Models

Additionally, Table 5 was reviewed and formatted for consistency.

  1. Improve Methodological Transparency

Reviewer Comment:

"It is unclear whether a formal scoping review or systematic review methodology was used."

Response:
We clarified our methodological stance by explicitly stating that the study follows a scoping review methodology, aligned with the PRISMA Extension for Scoping Reviews (PRISMA-ScR). This clarification has been added to the opening paragraph of the Methodology section, and the PRISMA-ScR citation has been included in the references.

Action Taken: See Section 4 – Methodology
Reference added: Tricco et al. (2018)

  1. Add a Formal Limitations Section

Reviewer Comment:

"A formal section discussing limitations is lacking..."

Response:
We developed a new section titled “6.5 Limitations of the Review” which outlines five key limitations:

  1. Scope of the literature search
  2. Heterogeneity of model definitions
  3. Limited empirical validation
  4. Geographical and sectoral bias
  5. Focus on model characteristics over healthcare outcomes

This section directly addresses concerns about research scope and generalizability.

Action Taken: See Section 6.5 – Limitations of the Review

  1. Enhance Figure and Table Formatting

Reviewer Comment:

"Visual elements could benefit from standardization and a unified layout."

Response:
We carefully reviewed and standardized all tables, including captions, font sizes, and alignment. Table headings were consistently formatted above the tables, spacing was unified, and any orphaned headers (e.g., Table 3) were corrected. We also ensured consistent use of terminology across the tables and narrative text.

Action Taken: Tables 1 through 5 reformatted and standardized

  1. Ensure Terminology and Reference Consistency

Reviewer Comment:

"Ensure terminology aligns with standard frameworks such as ISO/IEC 33001..."

Response:
We reviewed the manuscript to ensure consistent use of terminology related to maturity models (e.g., "dimensions," "levels," "validation"). Where appropriate, we clarified that some models are conceptually inspired by process frameworks such as the ISO/IEC 33001 series, but typically adapted for healthcare contexts with varying degrees of standardization.

Action Taken: Clarification added in Section 5 and terminology made consistent throughout

 

Author Response File: Author Response.pdf

Round 2

Reviewer 1 Report

Comments and Suggestions for Authors

Dear authors,

Having reviewed the revised version of your manuscript, I find that my previous comments have been satisfactorily addressed, and the manuscript has been improved as a result.
I have no further comments at this stage.

Sincerely,

Back to TopTop