Research on Comprehensive Performance Evaluation Method for Frontier Fundamental Research Project for Future Aircraft Engines

Abstract

The evaluation and management of frontier fundamental research projects for future advanced aircraft engines are challenging due to the need to balance assessing the innovative potential and technical risks with considering their long-term effects and inherent uncertainties. This study presents a comprehensive evaluation indicator system for evaluating frontier fundamental research projects for future advanced aircraft engines, integrating the Analytic Hierarchy Process (AHP) and fuzzy comprehensive evaluation (FCE) to balance innovative potential with technical risks. The AHP is used to determine weights for the evaluation indicator system based on a survey of technical experts. By incorporating expert ratings and weighted criteria, the FCE method synthesizes comprehensive evaluations and effectively avoids traditional scoring biases and simplistic averaging methods. A case study on a major project is conducted to demonstrate the effectiveness of the proposed method in highlighting the significant achievements and potential for innovation gaps. The results show that the AHP-FCE method proves robust in identifying complex, prospective research, providing a strategic tool for policymakers to prioritize impactful aircraft engine research and ensuring investment in projects with significant breakthrough potential.

1. Introduction

Aircraft engines, regarded as the ‘heart’ of airplanes, are crucial to a nation’s overall strength, industrial capacity, and technological sophistication. In the arena of future aerial combat, aircraft equipped with engines capable of operating across expansive spatial domains, spanning a wide range of speeds, and achieving extended operational ranges hold substantial strategic deterrence and practical combat value. As early as 2000, the United States Air Force articulated a new strategic vision for the 21st century, summarized in the principle of ‘global vigilance, global reach, and global power’, aiming to achieve preliminary combat capabilities by approximately 2030 [1]. Similarly, Europe and Japan have initiated research into sixth-generation fighter jets, underscoring a range of enhancements over fifth-generation models. These advancements include capabilities for rapid long-range endurance, improved survivability, enhanced strike power, and superior sensory and intelligence integration, placing demanding requirements on their propulsion systems [2,3].

These advanced engines must not only accommodate a wide range of flight speeds, enabling both high sustained and sprint capabilities, but also adapt to varied spatial domains and maintain operational efficiency over extended periods. Given these challenges, there is a clear need for a new generation of aircraft engines. These engines must be versatile enough to support low-altitude, low-speed flights over considerable distances, ensuring eco-friendly travel. Simultaneously, they should possess the capacity for strategic superiority in higher domains, being capable of operating at altitudes exceeding 20 km and surpassing velocity thresholds of Mach 4.

To counter challenges in the desired aircraft necessitates high sustained and sprint flight speeds, alongside the ability to operate over diverse spatial domains and for extended periods, the Aeroengine and Gas Turbine Science Center (abbreviated as “center”) was established in 2018 [4], and the center initiated the Future Aeroengine & Turbine Innovation Project plan (and hereinafter referred to as the FATIP plan) in 2020. The plan is a typical frontier fundamental research project focusing on crafting aero-thermodynamic configurations for aircraft engines that are not just viable for the supersonic aircraft of tomorrow but are also ultra-economic in operation. The FATIP funding system set up a top-level general performance scheme, including an ecological velocity domain from Mach 1.3 to 2.5, accommodating airspace from 0 to 30 km, and a velocity domain from 0 to Mach 5. The top-level general performance scheme contains a series of leading technical indicators, including, in general, a broad adaptive bypass ratio range, a wide adaptive supercharger ratio adjustment rate, a high maximum turbine inlet temperature, a wide economic supercruise velocity domain, and a low interior noise. The realization of the top-level general performance scheme requires multidisciplinary collaborative research and development. Therefore, the plan establishes several multidisciplinary specialty groups, including overall design and simulation, aerodynamics and acoustics, heat transfer and combustion, structural strength and reliability, testing, measurement and control, advanced materials, advanced manufacturing, and safety and airworthiness, as shown in Figure 1. Under the FATIP funding system, the top-level general performance scheme and leading technical indicators are decomposed into specific indicators required by each discipline group. A series of major project groups and key project groups are established within each discipline group, collaborating around the project’s specific requirements and the leading technical indicators of the top-level general performance scheme. The critical deliverables of the top-level general performance scheme include achieving an electronic prototype of the new concept aircraft engine based on the original innovations of various disciplines.

Given the complex interdependencies within this technical system and the necessity for interdisciplinary collaboration—as well as the frontier fundamental research projects of aircraft engine facing a higher technical risk with more uncertainty and complexity during the research process due to lower technical readiness levels, long research cycles, and interdisciplinary integration—the objectives of the work include developing a comprehensive evaluation indicator system to evaluate the performance of the FATIP plan, focusing on the directionality and sustainability of its initiatives, as well as the integration of the overall and component design and new technologies. On this basis, a combined quantitative and qualitative performance evaluation method should be developed to manage the risks associated with systemic exploration and to scientifically assess the technical contributions and progress of the entities involved, which can effectively avoid subjective bias existing in traditional scoring mechanisms, thereby enhancing the rigor and objectivity of the evaluation process.

The paper is structured into five sections: The initial section provides an introduction. Section 2 provides a comprehensive review of the existing literature. Section 3 demonstrates the methodology, including developing an evaluation indicator system, formulating the Analytic Hierarchy Process (AHP), and integrating a fuzzy comprehensive evaluation (FCE) method. Section 4 presents the findings of the AHP-FCE methodology to a typical project, offering a refined understanding of its efficacy. Section 5 synthesizes the study’s conclusions, clarifies its theoretical and practical implications, and outlines the limitations and prospects for future research.

2. Related Works

The performance evaluation methods for research projects include qualitative and quantitative methods. The qualitative evaluation method is a research or assessment approach focused on describing and interpreting phenomena, observed characteristics, or situations rather than analyzing them through quantitative data. This method typically involves the in-depth analysis of observations, interviews, or case studies to gain a deeper understanding of the implications of phenomena rather than merely conducting numerical measurements or calculations. Qualitative evaluation includes peer review and the Delphi method. Peer review is most frequently used based on a scoring system for assessing scientific research funding projects. It is expressed through quantitative scores based on peer review, and the decisions are made based on ranking the scores after comprehensive reviewers’ opinions integration [5]. The Delphi method is a structured technique used to harness and synthesize expert opinions iteratively, aiming to achieve consensus or systematically forecast future trends through multiple anonymous rounds of surveys and iterative feedback [6].

Quantitative evaluation methods are the systematic process of collecting and analyzing numerical data to understand, assess, and quantify outcomes, behaviors, or phenomena. These methods rely on statistical, mathematical, or computational techniques to convert data into measurable, objective evidence that can be used to test hypotheses, make predictions, or inform decisions. Quantitative evaluation methods include bibliometric analysis [7], principal component analysis [8], analytic hierarchy process [9], and fuzzy comprehensive evaluation [10]. However, mixed methods research, by combining elements of qualitative and quantitative research approaches, can achieve a more detailed and accurate understanding of research questions [11].

Recent studies have developed various frameworks and methodologies, including fuzzy analytic hierarchy processes, Bayesian best-worst methods, and fuzzy-set qualitative comparative analyses, to evaluate and enhance the performance and decision-making processes in fields ranging from industrial technology development to green finance and emergency management, underscoring the importance of innovative and analytical approaches in improving competitiveness, project management, and industry ecosystems. Huang et al. (2008) presented a fuzzy analytic hierarchy process method and utilized a crisp judgment matrix to evaluate subjective expert judgments made by the technical committee of the Industrial Technology Development Program in Taiwan [12]. Bitman and Sharif (2008) developed a framework for R&D project ranking to help a firm improve its competitiveness and advantages from reasonableness, attractiveness, responsiveness, competitiveness, and innovativeness [13]. Amiri (2010) proposed an AHP and fuzzy TOPSIS approach to assess alternative projects and help the decision-maker select the best one for the National Iranian Oil Company by using six criteria for comparing investment alternatives [14]. Barbosa et al. (2021) empirically examined the configuration of project management practices associated with better performance using fuzzy-set qualitative comparative analysis [15]. Gul and Yucesan (2022) developed a university ranking model with the aid of performance measures in the “University Monitoring and Evaluation Reports-2019” published by the Council of Higher Education Institutions in Turkey, and performance criteria stated were weighted using the Bayesian best-worst method (BBWM) and TOPSIS multicriteria decision-making (MCDM) method [16]. Shi et al. (2023) conducted the theoretical analysis and field research to determine the evaluation system of the emergency management capability of coal mining companies by utilizing the entropy weight technique to determine the weight of individual evaluation system indicators, and the convolutional neural network (CNN) model was adopted to predict the samples and provide suggestions for improving coal mine emergency management [17]. Li et al. (2023) utilized the Delphi and fuzzy Analytical Hierarchy Process (FAHP) method to identify and rank the main factors and sub-factors of green finance [18]. Wang et al. (2023) proposed an evaluation indicator system based on target detection combination with equipment test video data [19]. AlMalki and Durugbo (2023) used the Delphi method to identify and prioritize critical institutional enablers and barriers of Industry 4.0 for education from the viewpoint of experts within the triple helix, i.e., university, industry, and government experts [20]. Zhang et al. (2023) constructed the evaluation indicator system of the blockchain industry ecosystem from the “target layer-criterion layer-index layer” and made an empirical analysis based on the panel data of 15 provinces in China from 2018 to 2020, using the methods of entropy weight and TOPSIS [21].

As a typical representative of basic research projects, the National Natural Science Foundation Committee has established a comprehensive funding system based on technological development trends and national strategic needs, of which performance evaluation methods hold significant referential value for the FATIP project due to the similar attributes of the projects. For the project approval assessment phase of the National Natural Science Foundation, Zhu et al. (2016) proposed a systematic evaluation decision model based on evidence reasoning rules, which can fully consider the weight of evaluation indicators and the diversity of evaluation grades [22]. Zheng and Ren (2016) developed an evaluation indicator system for the Youth Science Fund project based on the Analytic Hierarchy Process and analyzed the influencing factors of research outcomes through an ordinal logistic regression model [23]. Du and Wang (2017) designed a performance evaluation mechanism and proposed a method for evaluating the performance of projects using DEMATEL (decision-making trial and evaluation laboratory) integrated with fuzzy comprehensive evaluation, which can handle the interrelationship among indicators and integrate expert evaluation information comprehensively [24]. The National Natural Science Foundation’s major research initiative targets significant foundational issues in the field of aviation propulsion, such as efficient flight, aiming to address the fundamental scientific principles and mechanisms behind “bottleneck” technologies in the aviation industry. Despite numerous projects supported by this funding mechanism aiming for the same system objective, there remains a gap between research outcomes and achieving a highly integrated aircraft engine prototype. Therefore, it is necessary to integrate the attributes and characteristics of the development and management of weapon equipment systems and aircraft engine projects into developing the performance evaluation management method for the FATIP plan.

Since there is little research on performance evaluation on frontier fundamental research projects for aircraft engines, the evaluation of weapon equipment R&D projects can provide a technical reference. In such projects, the evaluation of technical performance is often more important than the evaluation of economic benefits. Hence, it is necessary to focus on the aircraft engine’s top-level technical indicators for the top-level general performance scheme, as well as its systemic attributes within the operational and usage systems. Besides, the funding for equipment research projects can generally be sufficiently guaranteed. The project management department needs to exercise real-time control over the progress and budget during the implementation process. Thus, in the project evaluation, it is necessary to assess the control effects during the project implementation process to determine the effectiveness of the control [25]. Hou et al. (2008) designed the principle of the process evaluation system for aerial weapon equipment projects and the indicator system’s weight was determined by using the grey relating theory [26]. Tohumcu and Karasakal (2010) developed an approach based on analytic network process (ANP) and data envelopment analysis (DEA) to evaluate the performance of R&D projects and ongoing projects in a Defense Research and Development Institute [27]. Xu et al. (2014) established a post-evaluation indicator system for aircraft engine R&D capability-building projects based on the preliminary expert survey and group decision-making maximum eigenvalue method, and the correlation analysis and selection of evaluation indicators are determined based on statistical methods [28].

From the literature above, current research has made significant progress in quantifying evaluation indicators for basic research and weapon equipment projects, establishing evaluation systems, and developing methods for project performance evaluation. However, in the evaluation process, they neglect the degree of support to the top-level general performance scheme and indicators of the projects and whether the research results are well applied to the integration process of the overall performance scheme of the engine, which is crucial for the frontier fundamental research project for future aircraft engines with low technology readiness levels. Therefore, a comprehensive evaluation indicator system, which emphasizes collaborative research on developing the integrated top-level general performance scheme spanning these technologies, is necessary for the FATIP plan. To overcome the shortcomings of the existing scoring mechanism used in project assessment, we propose a comprehensive qualitative and quantitative evaluation method, which is of great significance for controlling systemic risks in prospective basic research projects and addressing the challenges of balancing the divergence of basic research and the feasibility of system integration.

3. Methodology

In this section, we conducted the survey with the project technical experts and project technical experts from the Aero-Engine and Gas Turbine Basic Science Center (abbreviated as “center”) and developed a robust evaluation system. This system evaluates fundamental research projects across three key dimensions: project management quality, technical level, and the potential benefits and prospects of the project. Furthermore, to enhance the accuracy and reliability of the evaluation, a comprehensive qualitative and quantitative evaluation method including the AHP method and the FCE method was adopted to provide a structured approach to assessing the multifaceted aspects of projects.

3.1. Establishment of the Evaluation Indicator System

Since the aircraft engine fundamental research projects face a higher technical risk with more uncertainty and complexity during the research process due to lower levels of technological maturity, long research cycles, and interdisciplinary integration, a comprehensive evaluation indicator system is the prerequisite for effectively managing these programs. Based on the comprehensive performance evaluation indicator system of the key special project under the National Key Research and Development Program [29,30] and combining the management experience of personnel and senior technical experts from the center, we have developed a comprehensive evaluation indicator system for frontier fundamental research projects. The system incorporates common factors, such as the project progress, significant achievements and benefits, and utilizing personnel and financial resources in routine performance evaluations, and emphasizes collaborative research on developing the integrated top-level general performance scheme, which is categorized into three primary indicators: project management quality, project technical quality, and benefit and prospect of the project. Multiple secondary and tertiary indicators are established under each primary indicator to assess project performance comprehensively.

3.1.1. Project Management Quality A₁

Project management refers to the overall and effective planning, organization, management, and control of the project in the project implementation process. Effective project management ensures the project is completed according to schedule, quality, and cost and achieves the expected objectives. Two secondary indicators, including organization management and fund management, are determined in the evaluation system. Several tertiary indicators are developed to refine the evaluation criteria of each secondary indicator. The detailed description of tertiary indicators underlying the secondary indicators is listed as follows.

1.

Organization management B₁₁

Organizational management refers to effectively managing the organizational structure, roles, responsibilities, and collaborative relationships of the project research group during project implementation. It is necessary to ensure that the organizational structure of the project research group is rational, the communication and collaboration among the members are efficient, and the project has the potential to be completed on schedule under the clear division of labor of members and well-established management. Organization management is measured using three tertiary indicators.

• Project organization strategy C₁₁₁

This indicator can reflect and assess the effective cooperation and implementation of the project, as well as the situation of regular internal and inter-group communication to solve technical difficulties in the project. It places a significant emphasis on the assessment of how effectively project groups communicate internally and interact with other groups regularly. This includes regular progress reports and results presentations and encompasses exchanging knowledge and experience through seminars, online forums, and working groups when technical challenges arise. By focusing on these areas, the indicator seeks to identify strengths and areas for improvement in project organization, ensuring that all group members have the necessary skills and strategies for effective problem-solving and project advancement.

• Research progress management C₁₁₂

This indicator involves meticulously tracking the milestones achieved against the planned timeline, examining the fulfillment of the set objectives, and assessing the overall progress with the project’s scope. The evaluation process includes examining the effectiveness of the methodologies employed, the quality of data collected, the thorough analyses conducted, and the extent of research findings documented. Moreover, this process should identify any discrepancies between the expected and actual outcomes, obstacles encountered, and the strategies implemented to overcome them. It also encompasses reviewing the resource utilization, including time, budget, and personnel, ensuring that the project adheres to its allocated resources while maintaining high standards of research integrity.

• Execution recording management C₁₁₃

This indicator assesses the authenticity and effectiveness of archival materials within a project, emphasizing these records’ integrity and operational impact. It examines the completeness of project documentation, ensuring that all relevant records are accurately maintained and readily available. Additionally, it evaluates the procedures for archiving certification, confirming that documents are stored correctly and certified to attest to their authenticity and compliance with legal and regulatory standards.

2.

Fund management B₁₂

Financial management refers to the process of effectively managing and controlling the financial resources of a project during its implementation. Effective financial management ensures the rational use of the project’s financial resources, controls costs, reduces risks, and achieves the expected economic benefits. Additionally, sound financial management improves the project’s transparency and sustainability and enhances its standardization. The following two tertiary indicators are established under the financial management factor.

• Fund management conformity C₁₂₁

This indicator scrutinizes the project’s administrative department, focusing on established financial systems, daily management practices, and the readiness for financial performance evaluation. It ensures compliance with financial regulations, emphasizing transparency, accountability, and efficiency. After assessing budgeting, accounting, reporting, and risk management, this standard fosters a sustainable, effective fund management environment.

• Fund execution situation C₁₂₂

This indicator evaluates a project fund’s allocation, transfer, and overall management, focusing on precise resource distribution and financial efficiency. It examines total and individual sub-item expenditures against planned budgets, providing insight into financial utilization and accountability. Additionally, it assesses the handling of fund surpluses, highlighting strategies for reinvestment or saving for future needs.

3.1.2. Project Technical Quality A₂

The quality of outcomes and the level of technical advancement produced during the project execution are crucial for project leaders and management professionals to assess whether the project has achieved its expected goals. This assessment primarily focuses on whether the project outputs satisfy the intended objectives and reach the anticipated quality and effectiveness. Evaluation can be conducted using the following two secondary indicators and eight tertiary indicators:

1.

Research status and quality B₂₁

The “research status and quality” indicator refers to the quality level of the project outputs. It can measure whether the results obtained from conducting the project conform to the expected requirements and quality standards. In this research, completion quality can be evaluated using six tertiary indicators:

• Achievement and quality of research goals and milestones C₂₁₁

This indicator is designed to meticulously evaluate the extent to which research goals have been met and the caliber of the project’s achievements and milestones. It delves into the completion status of the outlined objectives, measuring progress against predefined benchmarks to ascertain the degree of success achieved. Furthermore, it examines the quality of the outcomes, ensuring that they not only fulfill but potentially exceed the initial expectations in terms of innovation, relevance, and impact.

• Achievement and quality of technical indicators C₂₁₂

This indicator is used to evaluate the extent to which technical indicators have been achieved and the quality of these achievements. These technical indicators are crucial for assessing a project’s technical dimensions, progress, and ultimate results during its lifecycle. They act as concrete milestones that are directly tied to the project’s objectives, reflecting the project’s technical performance, advancement, and accomplishments.

• Achievement and quality of research deliverables C₂₁₃

This indicator focuses on assessing the progression and excellence of key research deliverables within a project, including but not limited to sophisticated mathematical models, innovative algorithms, and advanced software programs. It aims to precisely gauge the completion status of these deliverables, ensuring they meet or exceed their intended milestones and specifications. Additionally, it evaluates the quality of these outputs, considering their innovation, functionality, and applicability to real-world problems.

• Supporting level for the top-level general performance scheme C₂₁₄

This indicator is used to evaluate the project’s ongoing progress and its substantial role in facilitating the achievement of the top-level general performance scheme and indicators, specifically focusing on the general performance aspects of the engine. This assessment delves into how well the project conforms to and supports the core objectives of the top-level general performance scheme, ensuring that its contributions are both relevant and pivotal to the engine’s overall performance enhancement. Furthermore, this indicator examines the project’s potential for seamless integration within the broader scope of the project funding system’s deliverables, highlighting its capacity to add value and complement the existing framework of funded initiatives. By doing so, it not only gauges the current state of progress but also forecasts the project’s ability to influence and contribute to the cumulative success of the engine’s performance goals.

• Technical readiness level of the significant research deliverables C₂₁₅

This indicator evaluates the progression and technical readiness level of significant research deliverables, including pivotal experimental components and methodologies, ensuring alignment with the project’s contractual obligation stipulations. The Technology Readiness Level (TRL) framework [31] is a methodical metric used for gauging the development stage of a technology, spanning from the initial idea phase to its full-scale application. Originating from NASA for the appraisal of space exploration technologies, the TRL model has since been embraced across various sectors, encompassing both commercial and defense-related innovations. The framework is structured into distinct stages, detailed in

Table 1. Technical readiness level.

Level	Definition
Level 1 Basic Research	The fundamental principles of the technology concept and/or application are understood.
Level 2 Applied Research	The technology concept and/or application have been verified through experimentation and theoretical analysis.
Level 3 Technology Development	Prototypes of key functions or characteristics at the laboratory scale have been developed and tested.
Level 4 System/Subsystem Prototype Demonstration	Prototypes of complete systems or subsystems have been demonstrated in a laboratory or controlled environment.
Level 5 Practical Application	The technology has been verified in a real-world environment through prototypes or demonstration systems.
Level 6 Production Readiness	The technology is ready for production and/or deployment.
Level 7 Actual Operation	The technology has been proven in operational use, demonstrating the expected performance.

.

2.

Innovative impact and technological advancement B₂₂

The technical level refers to evaluating the technology’s real-world applicability, innovation, and progressive development within a project. It reflects how the project’s execution promotes technological quality, market competitiveness, and innovative growth. The technical level is mainly evaluated through the following two tertiary indicators:

• Innovation and advancement C₂₂₁

This indicator is strategically designed to comprehensively evaluate a project’s contribution to the field of innovation and its role in pushing the boundaries of technological advancement. It meticulously measures the project’s achievements, not just locally within national confines but also its recognition and impact on a global scale, highlighting its international relevance and contribution to the global knowledge base. This assessment goes beyond merely comparing the project to existing industry benchmarks; it critically examines the project’s capacity to set new standards and influences future market directions with innovative solutions and pioneering research outcomes.

• Achievement application and transformation C₂₂₂

This indicator is designed to quantify how effectively a project’s outputs transition from theoretical breakthroughs to practical applications within the industry, thereby revolutionizing current practices or forging new market paths. It evaluates the tangible impact of research, scrutinizing the extent to which project discoveries are transformed into implementable solutions that catalyze industry progress and spur innovation. This assessment highlights the project’s role in bridging the gap between research and practical application, underlining its contribution to the dynamic evolution of industry standards and practices. Focusing on the real-world utility of research findings underscores the project’s potential to drive significant industry advancements.

3.1.3. Benefit and Prospect A₃

This indicator meticulously monitors the augmentation of the project group’s capabilities, emphasizing technological advancements, experience, and knowledge for project refinement, competitive differentiation, and organizational expansion. It focuses on the critical importance of knowledge accumulation and developing capacities essential for achieving enduring excellence and success. The benefits derived from effective implementation are manifold, spanning economic and industrial progress, societal advancement, further capacity enhancement, and environmental sustainability.

1.

Academic system and talents development B₃₁

This metric represents a critical metric for evaluating the enhancement of skills, technological advancements, experiential learning, and knowledge base expansion that a project group gains throughout a project. This indicator highlights the immediate gains in expertise and information and emphasizes the strategic importance of these advancements in fostering continuous project improvement. It underscores the critical role that knowledge and capacity development play in augmenting the research group’s competitive edge, thereby contributing significantly to the overarching goals of organizational growth and sustainable development in the long run. Two tertiary indicators are identified:

• Academic system development C₃₁₁

This indicator is used to measure the project’s effectiveness in systematically developing an academic system and inheriting knowledge since one project’s contributions also significantly affect the development of multi-disciplinary research in aircraft engine technology. In other words, this indicator not only evaluates the project’s contribution to the academic level within its field but also examines the extent to which the project execution supports a comprehensive interdisciplinary knowledge system and the degree of innovation.

• Talent development C₃₁₂

This indicator refers to the strategic cultivation of human resources within the research group, emphasizing the nurturing and development of talent, such as master’s and doctoral students, and the advancement of group members’ professional titles or the acquisition of distinguished talent recognitions. It assesses the group’s effectiveness in building a high-quality research group through dedicated efforts in educational mentoring, professional development, and the recognition of individual achievements.

2.

Implementation benefits B₃₂

Implementation benefits outline both the tangible and intangible gains accrued from the execution of a project. These benefits are critical as they offer a comprehensive view of the value added through project activities, extending beyond mere financial returns to include enhancements in efficiency, knowledge, stakeholder satisfaction, and market positioning. The process of evaluating these benefits serves a dual purpose. Firstly, it provides project management groups with actionable insights and empirical data, aiding in the refinement of strategies, optimization of resource allocation, and enhancement of operational efficiencies for current and future projects. Secondly, it contributes to a body of knowledge that can inform decision-making, strategic planning, and policy formulation, ensuring that lessons learned are integrated into the organizational practices:

• Economic impact and industry contribution C₃₂₁

This indicator evaluates the project’s economic impact, both in terms of direct financial gains and indirect economic value. It encompasses the recognition received by the research project through industry awards, as well as the enhancement of market competitiveness resulting from the project’s outcomes.

• Societal advancement and structural innovation C₃₂₂

This indicator assesses the project’s contribution to societal progress, including promoting technological innovation, optimizing industrial structures, and generating educational and training programs. It specifically considers the project’s role in spurring local job creation.

• Sustainability and environmental impact C₃₂₃

This indicator considers the environmental footprint of the project, especially fuel efficiency and noise for aircraft engines, which could be important for long-term sustainability goals. Moreover, it can assess the project’s contribution to sustainable practices within the industry, such as the advancement of green technologies, the promotion of renewable energy utilization, and the implementation of eco-friendly manufacturing and testing processes.

By meticulously determining the primary, secondary, and tertiary indicators outlined above, we have established a robust and multifaceted evaluation indicator system for performance assessment within the FATIP plan. This integrative indicator system, which serves as the cornerstone for a comprehensive appraisal of managerial efficacy, is presented in Figure 2.

3.2. Establishment of the Analytic Hierarchy Process

As depicted in Figure 2, the comprehensive management evaluation indicator system for the projects under the FATIP plan is utilized. We employ the Analytic Hierarchy Process (AHP), a structured technique pioneered by Saaty in the 1970s [32], to determine the relative weights of indicators. AHP simplifies complex decision-making by organizing objectives into a hierarchical framework, which is then decomposed into levels of indicators and factors. This method quantifies the significance of each factor and calculates its weight, enabling a systematic evaluation of alternatives and prioritization of objectives. The calculation process of AHP is shown as follows:

Step 1. Problem decomposition

The decision problem is initially broken down into a structured hierarchy of smaller, more manageable sub-problems, enabling a clearer and more systematic evaluation process. This hierarchical structure is typically organized into several levels: At the top are the primary indicators of the decision-making process, such as level 1 shown in Figure 2; the secondary indicators comprise various criteria or factors that need to be considered, providing a framework for assessing the options, such as level 2 shown in Figure 2; and the tertiary indicators at the bottom are the alternatives or options being evaluated, such as level 3 shown in Figure 2. This breakdown not only facilitates a comprehensive understanding of the complex decision at hand but also aids in systematically comparing the alternatives against the set criteria, ensuring a thorough and balanced decision-making approach.

Step 2. Construction of pairwise comparison matrix

In each hierarchical level, the elements are comparatively assessed to gauge their relative significance through a pairwise comparison matrix. Expert judgments shape this matrix with elements C_ij indicating the importance of element i relative to j under the judgment criterion B_k. The matrix contrasts indicators C₁, C₂, ……, C_n to establish their precedence. The comparison matrix is shown in

Table 2. Pairwise comparison matrix.

Bk	C1	C2	C3	……	Cn
C1	C11	C12	C13	……	C1n
C2	C21	C22	C23	……	C2n
C3	C31	C32	C33	……	C3n
……	……	……	……	……	……
Cn	Cn1	Cn2	Cn3	……	Cnn

, and the importance score of C_ij is shown in

Table 3. Importance score of element C_ij.

Scale	Implicature
Cij = 1	Two elements are of equal importance
Cij = 3	The former element is slightly more important compared with the latter
Cij = 5	The former element is clearly important compared with the latter
Cij = 7	The former element is strongly important compared with the latter
Cij = 9	The former element is definitely important compared with the latter
Cij = 2, 4, 6, 8	The middle value of the above two adjacent judgments
Cij = 1, 1/2, 1/3, ……, 1/9	The degree of importance of the latter compared with the former

.

Step 3. Calculation of weights and consistency check

For each comparison matrix, calculate the eigenvector (which gives the weights) and the largest eigenvalue (used in the consistency check). The calculation process is shown as follows:

(1)

Calculate the product of the elements of each row in the matrix, as shown in Equation (1):

$${\mathit{M}}_{\mathit{i}}={\prod }_{\mathit{j}=1}^{\mathit{n}}{\mathit{C}}_{\mathit{i}\mathit{j}}(\mathit{i}=\mathrm{1,2}...,\mathit{n})$$

(1)

where ${\mathit{M}}_{\mathit{i}}$ is the product of the elements of the i-th row, $\mathit{n}$ is the number of rows within the matrix.

(2)

Solve the eigenvectors $\overline{\mathit{W}}$ as the n-th root of ${\mathit{M}}_{\mathit{i}}$, as shown in Equation (2):

$$\overline{\mathit{W}}=\sqrt[\mathit{n}]{{\mathit{M}}_{\mathit{i}}}$$

(2)

(3)

Normalize the eigenvectors $\overline{\mathit{W}}$ resulting in the weights of each indicator, as shown in Equation (3).

$${\mathit{W}}_{\mathit{i}}={\overline{\mathit{W}}}_{\mathit{i}}/{\sum }_{\mathit{j}=1}^{\mathit{n}}{\overline{\mathit{W}}}_{\mathit{j}}$$

(3)

where ${\mathit{W}}_{\mathit{i}}$ is the normalized weight of the i-th criterion.

(4)

Calculate the maximum eigenvalue of the judgment matrix ${\mathit{\lambda }}_{\mathit{m}\mathit{a}\mathit{x}}$, as shown in Equation (4).

$${\mathit{\lambda }}_{\mathit{m}\mathit{a}\mathit{x}}={\displaystyle \frac{1}{\mathit{n}}}{\sum }_{\mathit{i}=1}^{\mathit{n}}\left((\mathit{A}\mathit{w}{)}_{\mathit{i}}/{\mathit{W}}_{\mathit{i}}\right)$$

(4)

where $(\mathit{A}\mathit{w}{)}_{\mathit{i}}$ is the i-th element of the product of the comparison matrix A and the weight vector W.

(5)

The consistency index (CI) of the judgment matrix is shown in Equation (5) below.

$$\mathrm{C}\mathrm{I}=({\mathit{\lambda }}_{\mathit{m}\mathit{a}\mathit{x}}-\mathit{n})/(\mathit{n}-1)$$

(5)

(6)

The random index (RI) values are used in the AHP for consistency checks. These values depend on the number of criteria (or elements) being compared in the pairwise comparison matrices. The RI values are derived from randomly generated pairwise comparison matrices. Here are the RI values for matrices up to nine criteria, where 1~9 indicates the dimension of the matrix, as shown in

Table 4. Average randomized consistency indicator values.

Order n	1	2	3	4	5	6	7	8	9
RI	0.00	0.00	0.58	0.9	1.12	1.24	1.32	1.41	1.45

.

(7)

These values are typically referenced in AHP to calculate the consistency ratio (CR), which helps determine the reliability of the pairwise comparisons made within the matrix. A CR value less than 0.10 is generally considered acceptable, indicating a reasonable level of consistency in the pairwise comparisons. Correspondingly, this provides a basis for ranking indicators. The CR calculation is shown in Formula (6):

$$\mathrm{CR}=\mathrm{C}\mathrm{I}/\mathrm{R}\mathrm{I}$$

(6)

3.3. Fuzzy Comprehensive Evaluation Method

The fuzzy comprehensive evaluation (FCE) method represents a sophisticated approach to multi-criteria decision-making, seamlessly incorporating fuzzy logic into the evaluation process. This methodology is invaluable in navigating scenarios fraught with uncertainty, ambiguity, or subjective judgments, offering a robust framework for decision-makers grappling with complex factors. Rooted in the foundational principles of fuzzy set theory, pioneered by Zadeh in the 1960s [33], FCE stands apart for its capacity to accommodate complex assessments, transcending the limitations of conventional binary or crisp evaluations. FCE is widely embraced across diverse domains such as environmental management, risk assessment, and performance evaluation, serving as a preferred solution for addressing the intricate interplay of criteria and uncertain or subjective data. Based on the AHP-weighted quantification of indicator systems, the FCE method integrates expert ratings (excellent, good, fair, marginal, and poor) across indicators, and aggregates weighted criteria to derive comprehensive evaluations, effectively avoiding traditional scoring mechanisms like subjective bias and simple averaging, thereby enhancing the rigor and objectivity of the evaluation process. The calculation process of the FCE method is shown as follows:

Step 1: Determine the set of evaluation factors $\mathit{U}$ associated with a membership function that assigns it a grade of membership ranging from 0 to 1, as shown in Equation (7). This grade indicates the degree to which the factor conforms to certain qualitative or quantitative criteria within the evaluation context. By incorporating a spectrum of values rather than a dichotomous yes-or-no decision, the evaluation process acknowledges and utilizes the complexity of the data it analyzes.

$$\mathit{U}=\left\{{\mathit{u}}_1,{\mathit{u}}_2,...,{\mathit{u}}_{\mathit{i}}\right\}$$

(7)

where u_i is the evaluation factors.

Step 2: Determine the set of evaluation terms $\mathit{V}$ and assign the values $\mathit{P}$ to the set $\mathit{V}$:

$$\mathit{V}=\left\{{\mathit{V}}_1,{\mathit{V}}_2,...,{\mathit{V}}_{\mathit{m}}\right\}$$

(8)

$$\mathit{P}=\left\{{\mathit{P}}_1,{\mathit{P}}_2,...,{\mathit{P}}_{\mathit{m}}\right\}$$

(9)

where ${\mathit{V}}_{\mathit{m}}$ is a collection of assessment results, meticulously curated to include all relevant evaluation terms that comprehensively cover the criteria necessary for a robust assessment of the subject, and each term within this collection is a descriptor or a label that encapsulates a specific aspect of performance or quality to be evaluated. ${\mathit{P}}_{\mathit{m}}$ is the assigned value of the corresponding evaluation level. The assignment of ${\mathit{P}}_{\mathit{m}}$ to each term in ${\mathit{V}}_{\mathit{m}}$ is a critical task that demands attention to detail and a deep understanding of the evaluation framework, which involves the application of specific criteria and standards that have been established prior to the assessment. These standards ensure that the evaluation is objective, fair, and relevant.

Step 3: Determine the vector of indicator weights at each level, which are obtained by the AHP method. Determining these weights is a crucial aspect of the assessment process as it reflects the importance or influence of each indicator on the overall evaluation.

$$\mathit{W}=\left\{{\mathit{W}}_1,{\mathit{W}}_2,...,{\mathit{W}}_{\mathit{i}}\right\}$$

(10)

where ${\mathit{W}}_{\mathit{i}}$ is the relative weight value of the corresponding ${\mathit{u}}_{\mathit{i}}$.

Step 4: The collected data were summarized to obtain the fuzzy judgment matrix R. In fuzzy comprehensive evaluation, the matrix R is typically represented as a two-dimensional array with elements ${\mathit{r}}_{\mathit{i}\mathit{j}}$, as shown in Equation (11). In this matrix, i stands for the i-th evaluation subject and j represents the j-th characteristic. Each element ${\mathit{r}}_{\mathit{i}\mathit{j}}$ within this matrix indicates the extent to which the i-th evaluation subject possesses the j-th characteristic or meets the j-th criterion, based on a predefined scale, often ranging from 0 to 1.

$$\mathit{R}=\left[\begin{array}{cccc}{\mathit{r}}_{11}& {\mathit{r}}_{12}& \dots & {\mathit{r}}_{1\mathit{m}}\\ {\mathit{r}}_{21}& {\mathit{r}}_{22}& \dots & {\mathit{r}}_{2\mathit{m}}\\ ⋮& ⋮& \ddots & ⋮\\ {\mathit{r}}_{\mathit{i}1}& {\mathit{r}}_{\mathit{i}2}& \cdots & {\mathit{r}}_{\mathit{i}\mathit{m}}\end{array}\right]$$

(11)

Step 5: Perform a fuzzy comprehensive evaluation

In the fuzzy comprehensive evaluation process, the weight vector W, indicative of each evaluation index’s relative significance, merges with the evaluation matrix R, delineating performance assessments across diverse standards, via fuzzy synthesis. This complex interaction seamlessly converts qualitative evaluations into measurable indicators, guaranteeing a detailed analysis. The synthesis results in the fuzzy evaluation set B, an all-encompassing portrayal of the entity’s overall performance, encapsulating the combined importance and evaluated scores of each criterion, thereby offering a layered perspective on its quality and efficiency.

$$\mathit{B}=\mathit{W}\times \mathit{R}$$

(12)

Then, the fuzzy evaluation set B, representing the synthesized outcome of the evaluation process, is multiplied by the predetermined values P, which are assigned to different evaluation levels to reflect their respective importance or impact. This multiplication integrates the comprehensive assessment with the significance of each evaluation level, ensuring that the outcome, the fuzzy comprehensive evaluation score Q, accurately embodies the overall performance and quality metrics. Consequently, Q emerges as a singular, quantifiable score that encapsulates the entity’s evaluated performance across all criteria, offering a clear, consolidated measure of its effectiveness or efficiency.

$$\mathit{Q}=\mathit{B}\times \mathit{P}$$

(13)

According to the FCE evaluation method’s hierarchical structure, the project’s overall performance score is computed ascendingly across each indicator level from the bottom to the top. This method proves more efficacious when the indicators are fewer in number, more independent, and of similar types, which minimizes complexity and potential bias.

4. Case Study

In this case study, we selected a major project under the FATIP funding system as a typical frontier fundamental research project as an example and conducted a mid-term performance evaluation. The project aims at proposing novel aerothermodynamic cycle principles and overall technical solutions, specifically focusing on “airspace from 0 to 30 km, velocity domain from Mach 0 to Mach 5, a broad adjustable bypass ratio, wide adaptive supercharger ratio adjustment rate, and maximum turbine inlet temperature.” Based on the experts’ remarks, we developed the project’s overall performance score to reveal the effectiveness of the developed evaluation indicator system and the proposed AHP-FCE method.

4.1. Weight Calculation of the Evaluation System Based on the AHP Method

Based on the established comprehensive evaluation indicator system, questionnaires were distributed to 12 project technical experts at the center. After collecting the relevant data, a judgment matrix was constructed. The AHP method was then used to compare the importance of each criterion pairwise to determine the indicators’ weights at different levels, following the calculation process in Section 3.2.

4.1.1. Weights of Primary Indicators

Based on Table 2 and the evaluation results of the experts, the comparison matrix of primary indicators of the research is shown in

Table 5. Comparison matrix of primary indicators.

Indicator	A1	A2	A3	$\mathbf{Weights} {\mathit{W}}_{\mathit{i}}$	Other Indexes
A1	1	1/7	3	0.1488	${\mathit{\lambda }}_{\mathit{m}\mathit{a}\mathit{x}}$ = 3.08 $\mathrm{CI}=0.04$ $\mathrm{RI}=0.58$ $\mathrm{CR}=0.069$
A2	7	1	9	0.7854
A3	1/3	1/9	1	0.0658

. From Table 5, we can conclude that A₁ is strongly less important than A₂ (with a value of 1/7), and slightly more important than A₃ (with a value of 3). A₂ is definitely more important than A₃ (with a value of 9). Derived from Equations (1)–(3), we can derive that A₁ weigh 0.1488, A₂ has a weight of 0.7854, and A₃ has a weight of 0.0658. These weights are normalized and sum up to 1. According to Equation (4), the maximum eigenvalue λ_max = 3.08. Derived from Equations (5) and (6), the consistency index (CI) is calculated to be 0.04, and when compared to the random index (RI) of 0.58 (a value that depends on the matrix size and is derived from random matrices), it yields a consistency ratio (CR) of 0.069. Given that the CR is less than 0.10, it suggests that the matrix has an acceptable level of consistency.

4.1.2. Weights of Secondary Indicators

According to the evaluation results from the experts, we calculated the corresponding indicator weights for the judgment matrix for the secondary indicators of A₁ ~ A₃ according to Equations (1)–(6), and conducted a consistency check on the calculation results, as shown in

Table 6. Comparison matrix of secondary indicators of A₁~A₃.

Secondary Indicator of A1	B11	B12	$\mathbf{Weights} {\mathit{W}}_{\mathit{i}}$	Other Indexes
B11	1	1	0.5	${\mathit{\lambda }}_{\mathit{m}\mathit{a}\mathit{x}}$ = 2 $\mathrm{CI}=0$ $\mathrm{RI}=0$ $\mathrm{CR}=N/A$
B12	1	1	0.5
Secondary Indicator of A2	B21	B22	$\mathbf{Weights} {\mathit{W}}_{\mathit{i}}$	Other indexes
B21	1	4	0.8	${\mathit{\lambda }}_{\mathit{m}\mathit{a}\mathit{x}}$ = 2 $\mathrm{CI}=0$ $\mathrm{RI}=0$ $\mathrm{CR}=N/A$
B22	1/4	1	0.2
Secondary Indicator of A3	B31	B32	$\mathbf{Weights} {\mathit{W}}_{\mathit{i}}$	Other indexes
B31	1	3	0.75	${\mathit{\lambda }}_{\mathit{m}\mathit{a}\mathit{x}}$ = 2 $\mathrm{CI}=0$ $\mathrm{RI}=0$ $\mathrm{CR}=N/A$
B32	1/3	1	0.25

.

From Table 6, we can conclude that B₁₁ and B₁₂ are of equal importance, with equal weights of 0.5. The eigenvalue ${\mathit{\lambda }}_{\mathit{m}\mathit{a}\mathit{x}}$ = 2, which is the expected value for a 2 × 2 matrix, suggesting that the matrix is consistent. The second-order matrix only involves the judgment of the relative importance of two elements and does not require a complex consistency check, because they naturally meet the consistency requirements. Therefore, this matrix passes the consistency test. B₂₁ is deemed slightly more important than B₂₂. The weight of B₂₁ is 0.8, with B₂₂ weighting 0.2. B₃₁ is slightly more important than B₃₂. Based on the pairwise comparisons, B₃₁ received a weight of 0.75 and B₃₂ received a weight of 0.25.

4.1.3. Weights of Tertiary Indicators

Similarly, we can calculate the corresponding indicator weights for the judgment matrix for the tertiary indicators of B₁₁ ~ B₃₂, and conduct a consistency check on the calculation results.

• Weights of Tertiary Indicators of B₁

The indicator weights for the judgment matrix for the tertiary indicators of B₁₁ ~ B₁₂ are shown in

Table 7. Comparison matrix of tertiary indicators of B₁₁.

Tertiary Indicator of B11	C111	C112	C113	$\mathbf{Weights} {\mathit{W}}_{\mathit{i}}$	Other Indexes
C111	1	1	2	0.4	${\mathit{\lambda }}_{\mathit{m}\mathit{a}\mathit{x}}$ = 3 $\mathrm{CI}=0$ $\mathrm{RI}=0.58$ $\mathrm{CR}=0$
C112	1	1	2	0.4
C113	1/2	1/2	1	0.2

and

Table 8. Comparison matrix of tertiary indicators of B₁₂.

Tertiary Indicator of B12	C121	C122	$\mathbf{Weights} {\mathit{W}}_{\mathit{i}}$	Other Indexes
C121	1	1/3	0.25	${\mathit{\lambda }}_{\mathit{m}\mathit{a}\mathit{x}}$ = 2 $\mathrm{CI}=0$ $\mathrm{RI}=0$ $\mathrm{CR}=N/A$
C122	3	1	0.75

.

From Table 7, the pairwise comparison shows that C₁₁₁ is considered equally important as C₁₁₂, with the same weight of 0.4. C₁₁₃ is less slightly important than C₁₁₁ and C₁₁₂, with a weight of 0.2. From Table 8, C₁₂₂ is slightly more important than C₁₂₁, assigning a weight of 0.25 to C₁₂₁ and 0.75 to C₁₂₂. The CI and RI of the second-order matrix are 0.000, indicating that the pairwise comparisons are entirely consistent.

• Weights of Tertiary Indicators of B₂.

The corresponding indicator weights for the judgment matrix for the tertiary indicators of B₂₁ and B₂₂ are shown in

Table 9. Comparison matrix of tertiary indicators of B₂₁.

Tertiary Indicator of B21	C211	C212	C213	C214	C215	$\mathbf{Weights} {\mathit{W}}_{\mathit{i}}$	Other Indexes
C211	1	1	1	1/3	2	0.158	${\mathit{\lambda }}_{\mathit{m}\mathit{a}\mathit{x}}$ = 5.02 $\mathrm{CI}=0.0496$ $\mathrm{RI}=1.12$ $\mathrm{CR}=0.0044$
C212	1	1	1	1/3	2	0.158
C213	1	1	1	1/3	2	0.158
C214	3	3	3	1	4	0.440
C215	1/2	1/2	1/2	1/4	1	0.086

and

Table 10. Comparison matrix of tertiary indicators of B_22.

Tertiary Indicator of B22	C221	C222	$\mathbf{Weights} {\mathit{W}}_{\mathit{i}}$	Other Indexes
C221	1	1	0.5	${\mathit{\lambda }}_{\mathit{m}\mathit{a}\mathit{x}}$ = 2.000 $\mathrm{CI}=0.000$ $\mathrm{RI}=0.000$ $\mathrm{CR}=N/A$
C222	1	1	0.5

.

From Table 9, we can conclude that the computed weights W_i for the indicators C₂₁₁ to C₂₁₅ are 0.158, 0.158, 0.158, 0.440, and 0.086, respectively. The value of CR is 0.0044, showing that the consistency of the pairwise comparisons is acceptable. From Table 10, C₂₂₁ is as important as C₂₂₂; both C₂₂₁ and C₂₂₂ are assigned equal importance with the same weights of 0.5. However, for a 2 × 2 matrix, a CR is not needed as the matrix is inherently consistent.

• Weights of Tertiary Indicators of B₃.

The corresponding indicator weights for the judgment matrix for the tertiary indicators of B₃₁ and B₃₂ are shown in

Table 11. Comparison matrix of tertiary indicators of B₃₁.

Tertiary Indicator of B31	C311	C312	$\mathbf{Weights} {\mathit{W}}_{\mathit{i}}$	Other Indexes
C311	1	1/2	0.333	${\mathit{\lambda }}_{\mathit{m}\mathit{a}\mathit{x}}$ = 2.000 $\mathrm{CI}=0.000$ $\mathrm{RI}=0.000$ $\mathrm{CR}=N/A$
C312	2	1	0.667

and

Table 12. Comparison matrix of tertiary indicators of B₃₂.

Tertiary Indicator of B32	C321	C322	C323	$\mathbf{Weights} {\mathit{W}}_{\mathit{i}}$	Other Indexes
C321	1	1	1	0.333	${\mathit{\lambda }}_{\mathit{m}\mathit{a}\mathit{x}}$ = 3.000 $\mathrm{CI}=0$ $\mathrm{RI}=0.58$ $\mathrm{CR}=0$
C322	1	1	1	0.333
C323	1	1	1	0.333

.

Table 11 shows that C₃₁₂ is twice as important as C₃₁₁, assigning a weight of 0.333 to C₃₁₁ and 0.667 to C₃₁₂. From Table 12, C₃₂₁ is as important as C₃₂₂ and C₃₂₃, and, consequently, the derived weight of these three indicators is 0.333.

After obtaining the weights of the indicators at each level and conducting consistency tests on them respectively, it is necessary to integrate and calculate the final weights so that the indicator system is more intuitive and concise. The calculation method is that the weights of the indicators at this level are multiplied by the weights at the higher level. The final weights of the indicators of the evaluation system are shown in

Table 13. Final weights of the indicators of the evaluation system.

Primary Indicator	Weights	Secondary Indicator	Weights	Tertiary Indicator	Weights
Project management quality A1	0.1488	Organization management B11	0.0744	Project organization strategy C111	0.0298
				Research progress management C112	0.0298
				Execution recording management C113	0.0149
		Fund management B12	0.0744	Fund management conformity C121	0.0186
				Fund execution situation C122	0.0558
Project technical quality A2	0.7854	Research status and quality B21	0.6283	Achievement and quality of research goals and milestones C211	0.0993
				Achievement and quality of technical indicators C212	0.0993
				Achievement and quality of research deliverables C213	0.0993
				Supporting level for the top-level general performance scheme C214	0.2764
				Technical readiness level of the significant research deliverables C215	0.054
		Innovative impact and technological advancement B22	0.1571	Innovation and advancement C221	0.0786
				Achievement application and transformation C222	0.0786
Benefit and prospect A3	0.0658	Academic system and talents development B31	0.0494	Academic system development C311	0.0164
				Talent development C312	0.0329
		Implementation benefits B32	0.0165	Economic impact and industry contribution C321	0.0055
				Societal advancement and structural innovation C322	0.0055
				Sustainability and environmental impact C323	0.0055

. For weight of each indicator, four significant digits are preserved.

4.2. Fuzzy Comprehensive Evaluation of the Typical Project

The FCE method uses a well-established set of criteria for assessing aircraft engine technology projects. This method was refined with the input from leading experts at the center, ensuring a professional and thorough evaluation. The assessment relies on various sources, including mid-term technical reports, compiled outcomes from the projects, collections of notable project results, progress archives, and project group management reports. The results of the project evaluations can be found in

Table 14. Experts’ evaluation results of the project.

	Indicators of Achievement	Expert 1	Expert 2	Expert 3	Expert 4	Expert 5
1	Project organization strategy C111	Excellent	Excellent	Excellent	Excellent	Excellent
2	Research progress management C112	Excellent	Excellent	Excellent	Excellent	Excellent
3	Execution recording management C113	Excellent	Excellent	Excellent	Excellent	Excellent
4	Fund management conformity C121	Excellent	Excellent	Excellent	Excellent	Excellent
5	Fund execution situation C122	Excellent	Excellent	Excellent	Excellent	Excellent
6	Achievement and quality of research goals and milestones C211	Good	Excellent	Good	Excellent	Excellent
7	Achievement and quality of technical indicators C212	Excellent	Excellent	Good	Excellent	Good
8	Achievement and quality of research deliverables C213	Good	Excellent	Excellent	Excellent	Excellent
9	Supporting level for the top-level general performance scheme C214	Good	Excellent	Good	Excellent	Excellent
10	Technical readiness level of the significant research deliverables C215	Good	Excellent	Excellent	Excellent	Excellent
11	Innovation and advancement C221	Good	Excellent	Good	Good	Excellent
12	Achievement application and transformation C222	Good	Good	Good	Good	Excellent
13	Academic system development C311	Excellent	Good	Good	Good	Good
14	Talent development C312	Excellent	Good	Excellent	Good	Good
15	Economic impact and industry contribution C321	Excellent	Excellent	Excellent	Excellent	Good
16	Societal advancement and structural innovation C322	Good	Excellent	Excellent	Excellent	Excellent
17	Sustainability and environmental impact C323	Good	Excellent	Good	Good	Good

.

The expert evaluation results were statistically organized to carry out a fuzzy comprehensive evaluation of the mid-term situation of the new generation of aircraft engine aerodynamic thermodynamic principal configuration project and followed the calculation process in Section 3.3:

(1) According to Equation (7), the evaluation factor set $\mathit{U}$ for fuzzy comprehensive evaluation is $\mathit{U}=\left\{{\mathit{u}}_1,{\mathit{u}}_2,{\dots ,\mathit{u}}_{20}\right\}$;

(2) According to Equation (8), we constructed a fuzzy comprehensive evaluation rubric set V and assigned values to the rubric set:

V = {Excellent, Good, Fair, Marginal, Poor}

where the score of “Excellent” ranges from 90 to 100, the score of “Good” ranges from 80 to 90, the score of “Fair” ranges from 70 to 80, the score of “Marginal” ranges from 60 to 70, the score of “Poor” is below 60. Then, the values P to the set V can be written as P = {100,90,80,70,60};

(3) Construct fuzzy judgment matrix R according to Equation (11):

$$\mathit{R}=\left[\begin{array}{ccccc}1& 0& 0& 0& 0\\ 1& 0& 0& 0& 0\\ 1& 0& 0& 0& 0\\ 1& 0& 0& 0& 0\\ 1& 0& 0& 0& 0\\ 0.6& 0.4& 0& 0& 0\\ 0.6& 0.4& 0& 0& 0\\ 0.8& 0.2& 0& 0& 0\\ 0.6& 0.4& 0& 0& 0\\ 0.8& 0.2& 0& 0& 0\\ 0.4& 0.6& 0& 0& 0\\ 0.2& 0.8& 0& 0& 0\\ 0.2& 0.8& 0& 0& 0\\ 0.4& 0.6& 0& 0& 0\\ 0.8& 0.2& 0& 0& 0\\ 0.8& 0.2& 0& 0& 0\\ 0.2& 0.8& 0& 0& 0\end{array}\right]$$

(4) Based on the index weights of the established performance evaluation indicator system for aircraft engine science and technology projects in Table 14, we establish the evaluation indicator weight vector $\mathit{W}$;

(5) Multiply the weight vector W with the judgment matrix R to calculate the evaluation result B according to Equation (12).

B =W × R = [0.629924,0.370126, 0.0000, 0.0000, 0.0000]

(6) Multiply the evaluation result B with the evaluation grade assignment P to finally obtain the corresponding fuzzy comprehensive evaluation score Q according to Equation (13).

Q =B × P = [0.629924, 0.370126, 0.0000, 0.0000, 0.0000] × [100, 90, 80, 70, 60] = 96.30

Referring to the method of calculating evaluation scores based on primary indicators, judgment matrices were established for secondary and tertiary indicators respectively, and after conducting a fuzzy comprehensive evaluation, the overall score for the performance evaluation of aircraft engine technology projects was obtained, as shown in

Table 15. Detailed score of project performance evaluation.

Primary Indicator	Score	Secondary Indicator	Score	Tertiary Indicator	Score
Project management quality A1	14.33	Organization management B11	7.165	Project organization strategy C111	2.866
				Research progress management C112	2.866
				Execution recording management C113	1.433
		Fund management B12	7.165	Fund management conformity C121	1.791
				Fund execution situation C122	5.374
Project technical quality A2	75.63	Research status and quality B21	60.505	Achievement and quality of research goals and milestones C211	9.560
				Achievement and quality of technical indicators C212	9.560
				Achievement and quality of research deliverables C213	9.560
				Supporting level for the top-level general performance scheme C214	26.622
				Technical readiness level of the significant research deliverables C215	5.203
		Innovative impact and technological advancement B22	15.129	Innovation and advancement C221	7.5645
				Achievement application and transformation C222	7.5645
Benefit and prospect A3	6.34	Academic system and talents development B31	4.752	Academic system development C311	1.582
				Talent development C312	3.170
		Implementation benefits B32	1.584	Economic impact and industry contribution C321	0.528
				Societal advancement and structural innovation C322	0.528
				Sustainability and environmental impact C323	0.528

. In Table 15, we have applied the following approximations to the scores: For primary indicators, two significant digits are retained; for secondary and tertiary indicators, three significant digits are preserved. Additionally, for two tertiary indicators carrying the same weight under an identical secondary indicator, four significant digits are maintained.

Table 15 presents the mid-term performance evaluation of the project, indicating an excellent performance consistent with the project’s overall evaluation. Throughout the project implementation, the research group has maintained regular exchanges and documentation of supporting materials, such as meeting records and photographs, at intervals of no more than six months. Consequently, the group has successfully shared all agreed-upon results and achieved milestone objectives by the mid-term assessment. The management of project funds demonstrates a commendable level of reasonableness and standardization, resulting in a project management score of 14.33, thus earning an “excellent” classification in the primary indicator of project management.

In research progress, the mid-term phase has witnessed the fulfillment of research content and objectives, with 90% of the leading targets already accomplished. These leading targets will be fully realized within the next three months. Notably, the project has addressed a significant gap in domestic innovation and advancement, which is reflected in its outstanding technical quality score of 75.63, classified as “excellent”. However, enhancing the technology’s leading technical indicators remains necessary to fulfill contractual requirements and achieve optimal practical application. Furthermore, the research group has cultivated exceptional talent and amassed substantial knowledge, culminating in forming a leading domestic research group with a score of 6.34. Consequently, the project exhibits promising benefits and prospects in capability accumulation and implementation. The evaluation results underscore the methodology’s effectiveness as a comprehensive evaluation tool that is both systematic and adaptable to the subjective nature of certain assessments. This characteristic renders it particularly valuable in evaluating conceptual design scenarios for aircraft engines, where decisions are influenced by a combination of quantitative and qualitative factors.

5. Conclusions

In this research, we propose a comprehensive performance evaluation method to assess frontier fundamental research projects for future advanced aircraft engines. This evaluation method combines quantitative and qualitative assessments to thoroughly evaluate project strengths and challenges, and a case study on a major project under the FATIP funding system aiming at developing the aerothermodynamics configuration for aircraft engines was conducted to validate the proposed method. The key contributions of the research include:

(1)

To balance assessing the innovative potential and technical risks for frontier fundamental research projects, we propose a comprehensive evaluation indicator system, which emphasizes collaborative research on developing the integrated top-level general performance scheme spanning these technologies for evaluating frontier fundamental research projects.

(2)

We adopt the Analytic Hierarchy Process (AHP) to determine the appropriate weights for the established comprehensive evaluation indicator system derived from a survey conducted among technical experts at the center.

(3)

Based on the AHP-weighted quantification of indicator systems, the fuzzy comprehensive evaluation method integrates expert ratings (excellent, good, fair, marginal, and poor) across indicators. It aggregates weighted criteria to derive comprehensive evaluations, effectively avoiding traditional scoring mechanisms like subjective bias and simple averaging, thereby enhancing the rigor and objectivity of the evaluation process.

(4)

A case study validates the evaluation system’s effectiveness, highlighting its relevance and efficacy in aligning basic research with practical needs in aircraft engine development, which is essential for transitioning innovative concepts from theory to practice.

Therefore, this research establishes a crucial foundation for advancing aerospace engineering and innovation in next-generation aircraft engine technologies. It provides theoretical support not only for evaluating frontier fundamental research projects related to aircraft engines but also for promoting similar advancements in other major equipment sectors. In future research, we will refine the evaluation indicator system to cover aircraft engine performance under various conditions. Simulation scenarios will be developed to validate the performance evaluations of overall and specific components. Additionally, integrating advanced analytical techniques, such as machine learning and predictive modeling, will enhance the evaluation system’s robustness and accuracy. These data-driven insights will provide policymakers with a strategic tool to prioritize impactful frontier research projects in advanced engines.