Quantitative Measurement of Digital Maturity in Manufacturing Enterprises: An Application Scenario-Based Study

Abstract

With the rapid advancement of intelligent manufacturing and digital transformation, assessing the digital transformation and maturity of manufacturing enterprises enables firms to evaluate their digital achievements and establish appropriate transformation pathways. Existing maturity assessment models for manufacturing enterprises predominantly emphasize strategic-level evaluation and rely heavily on survey-based data, while paying limited attention to the business-function level. However, in practice, enterprises often initiate digital transformation by addressing specific business challenges and then gradually advance through concrete application scenarios. To address this gap, this paper proposes a scenario-based approach for measuring the digital transformation and maturity of manufacturing enterprises. With this approach, a differentiated weighting system is constructed based on “core keywords–extended keywords–negative keywords”, and a semantic similarity model is used to identify and quantify digital application scenarios in corporate annual reports. Building on this, a three-dimensional evaluation framework, comprising scenario coverage, scenario depth, and scenario consistency, is developed to comprehensively assess the extent of digital transformation from the perspective of application scenarios. With the proposed method, different business units can be evaluated independently, thereby capturing transformation progress across heterogeneous levels. Since it relies on publicly available corporate annual reports, the evaluation process is transparent and traceable and generates quantitative results. By shifting from survey-based, strategy-oriented assessments to function-oriented, data-driven, and modular evaluations, the method not only enhances accuracy and interpretability but also provides practical guidance for resource allocation, cross-functional complexity management, and the progressive expansion of digital transformation.

1. Introduction

With the accelerated progress in intelligent manufacturing and industrial digital transformation, manufacturing enterprises are undergoing a profound transformation from traditional production models to deeply integrated systems of physical and digital operations. Digital technologies have been widely adopted as critical tools to enhance flexibility, reduce operating costs, and restructure strategic models. However, scientifically evaluating the maturity and evolutionary pathways of a manufacturing enterprise’s digital transformation remains a major challenge.

In contrast to holistic transformation, manufacturing enterprises typically initiate digital transformation by addressing specific operational problems and then gradually implement it through application scenarios. Corporate annual reports serve as an important and reliable source of information for disclosing the degree of digital transformation, but their texts are often characterized by high semantic variability and unstructured expressions, making it difficult for traditional methods to effectively extract and quantify the underlying digital capabilities. Therefore, there is an urgent need for a measurement approach that can identify application scenarios from authentic texts, reflect the development depth, and track transformation trends to better align with enterprise practices and policy evaluation needs.

To address this issue, this paper proposes a scenario-based method for measuring the digital transformation of manufacturing enterprises. The method employs keywords and semantic similarity models to identify and quantitatively evaluate digital application scenarios in corporate annual reports. First, application scenarios are classified according to evaluation objectives. Then, three categories of keywords—“core words, extended words, and negation words”—are constructed and assigned differentiated weights. Next, based on keyword intensity, semantic similarity is applied to evaluate the alignment between digital transformation objectives and outcomes reflected in specific business scenarios. For each predefined scenario, a three-dimensional evaluation framework—comprising scenario coverage, scenario depth, and consistency—is established using keyword frequency, keyword intensity, and semantic similarity, thereby providing a systematic assessment of digital transformation from the perspective of application scenarios.

The proposed measurement framework for digital transformation is based on “scenario division–keyword guidance–semantic model embedding.” The framework demonstrates strong model adaptability and interpretability, as well as reliable trend tracking. It can be used to quantify digital transformation maturity across different business scenarios and reveal the transformation trajectories in functional dimensions such as R&D, manufacturing, operations, and collaboration, thereby providing a foundation for multidimensional assessment. To validate its effectiveness, the method was applied to ten years of annual reports from manufacturing enterprises in Zhejiang Province. The results show that it can consistently capture the trajectories of digital transformation across scenarios and align closely with regional policy developments. Compared with traditional approaches, this study introduces a novel paradigm for evaluating digital maturity and offers enterprises, governments, and research institutions a practical tool for assessing transformation progress and supporting decision-making, with the potential for cross-industry and cross-regional applications.

2. Literature Review

Maturity models (MMs) have been reviewed from various perspectives by several authors (for example, [1,2,3]. Senna et al. [3] provided a literature review of 55 digital MMs in the manufacturing sector. Therefore, we used these reviews as a starting point to build our model and collected research results on MMs for manufacturing enterprises. The reviewed literature on MMs in the manufacturing sector was published between 2023 and 2025 and obtained from the WoS and Scopus databases.

In the following subsections, existing MMs across four key dimensions are analyzed: assessment level (strategic, organizational, functional), evaluation purpose (readiness, maturity, benchmarking), data sources (survey, expert, system-generated, text), and methodology (qualitative, MCDA/statistical, hybrid, data-driven). This framework allows for a structured comparison of key models, identifying gaps that motivate the proposed scenario-based approach.

2.1. Evaluation Level and Scope

From a structural perspective, layered frameworks with hierarchical stages are commonly adopted in the 105 reviewed MMs. Five- and six-level maturity structures are frequently used and have become relatively standardized, providing a clear basis for stakeholders to understand how capability progresses across stages. Some research applies such structures in their models to describe capability gaps and evolution path [4,5,6].

While some models focus primarily on technological aspects, most incorporate multiple assessment domains, including strategy, organization, technology, human factors, and operations, reflecting the multidimensional nature of Industry 4.0 readiness. For instance, several of these domains are used in the assessments at the same time [7,8,9,10]. In addition, some models are tailored for specific contexts, such as simplified frameworks for small and medium-sized enterprises [11] or models with targeted objectives such as green readiness [12] or workforce evaluation [13].

However, existing MMs predominantly define and assess digital maturity at the organizational level using macro-level dimensions, with limited attention to the transformation processes of specific business functions in manufacturing firms. In practice, manufacturing digital transformation often starts with a single function or a critical use case and then expands progressively across functional modules. Relying solely on organizational-level stage definitions and aggregated scoring can obscure maturity disparities among functions, making it difficult to identify function-specific implementation pace and capability gaps.

2.2. Purpose of Model Evaluation

The reviewed MMs vary in their focus and intended purpose, which can be broadly grouped into three categories: (1) readiness assessment, which evaluates an organization’s preparedness to initiate or advance Industry 4.0 transformation [9,14,15]; (2) maturity measurement, which determines the current state of digital maturity across selected domains or technologies [8,10,16]; and (3) capability benchmarking and strategic planning, which aims to identify capability gaps and support the formulation of strategic roadmaps or related decision-making inputs [17,18].

Of the 105 models analyzed, more than 70% are primarily designed for readiness assessment or maturity evaluation, indicating the strong emphasis on understanding organizational status relative to Industry 4.0 goals. Models with a strategic orientation are typically used to support transformation planning through methods such as capability gap analysis or weighted performance evaluation [11,17,18,19].

In addition, models are increasingly being developed for policy support and sustainability integration. These models have a range of purposes, such as education and workforce development [13,20], green and sustainable transformation [12,21], and SME-oriented policy frameworks [11].

Most existing evaluation targets adopt a firm-level perspective and primarily address the question of “where the organization stands overall in its digital transformation.” Consequently, they provide limited visibility into the performance and status of specific functions or processes in manufacturing firms and fail to reveal heterogeneity in transformation outcomes across functional modules.

2.3. Data for Model Evaluation

The evaluation data used can be categorized as: (1) survey-based data, which are collected via structured questionnaires, typically Likert scales [15,22]; (2) expert input and interviews, which are collected via expert judgment, interviews, and workshops [16]; (3) literature-driven or framework-derived data, which are based on existing models, standards, and theoretical mapping [23]; and (4) operational or system-generated data, which are based on digital systems such as MES, IoT, or performance logs [16].

Most (over 60%) of the examined models rely primarily on survey instruments. While this method is easy to implement, it may be limited in terms of objectivity and scalability. Recent developments indicate a gradual shift towards the use of system-based, real-time, or hybrid data sources, with the aim of improving reliability, automation, and integration with operational technologies. Beyond the above four data categories, there is the case-study-oriented data collection strategy, which draws directly from empirical evidence in multiple enterprises [24].

Among the commonly used data types, survey- and expert-based inputs are inherently subjective, while literature-driven or framework-derived data often fail to accurately represent firm-specific contexts. Moreover, substantial heterogeneity across manufacturing sectors and the diversity of digital software systems makes it challenging to leverage operational or system-generated data effectively, as doing so requires standardized information-extraction methods and unified data formats, which are difficult to design and implement.

2.4. Evaluation Methods

Evaluation methods were grouped into the following four categories: (1) expert-based and self-assessment tools, which involve subjective scoring by external experts or internal stakeholders using checklists, Likert scales, or maturity matrices; (2) quantitative and statistical methods, which include AHP (Analytic Hierarchy Process), PROMETHEE (Preference Ranking Organization METHod for Enrichment Evaluations), SEM (Structural Equation Modeling), PLS-SEM (Partial Least Squares Structural Equation Modeling), CCA (Canonical Correlation Analysis), IPMA (Importance–Performance Map Analysis), fuzzy logic, and regression-based scoring; (3) hybrid or simulation-based approaches, which combine expert input with decision models, simulations, or model-driven reasoning (e.g., fuzzy AHP, job profile scoring); and (4) process-based or value-oriented assessment, using VSM (Value Stream Mapping), carbon footprint, or value-added analysis to link maturity with sustainability or process efficiency.

Among the reviewed literature, the expert-based assessment is prevalent; many models prioritize usability and adopt quick self-assessment tools—especially suited for SMEs (Small and Medium Enterprises) and initial diagnostic stages [3,25,26]. The use of formal MCDA (Multi-Criteria Decision Analysis) and statistical approaches is also rising. For instance, PLS-SEM, AHP, or PROMETHEE are incorporated into various models for more reliable validation and scoring, especially in academic contexts [4,22,27]. Hybrid models are also emerging. A growing number of studies [13,28] mix expert scoring with algorithmic computation, such as job role alignment and fuzzy logic, aiming for scalability and personalization. Newer approaches linking maturity to performance/sustainability integrate environmental KPIs (e.g., [12]) or production performance (e.g., [29]) to assess real-world impact and implementation depth. The evaluation of these methods reflects a shift from manual checklists to hybrid, data-supported strategies.

In the assessment of digital transformation in manufacturing enterprises, the underlying evidence is predominantly textual and available at a considerable scale. Most existing approaches remain traditional, small-data–oriented evaluation methods, which are insufficient to fully leverage massive text-based evidence. Against the backdrop of rapid advances in artificial intelligence, it is necessary to explore automated analytics for large-scale, multi-source textual data to characterize firms’ digital practices in a more objective and reproducible manner and to derive robust assessment conclusions.

2.5. Summary of the Research State and Trends

To provide a clearer understanding of the current landscape of digital transformation MMs,

Table 1. Comparative overview of existing digital transformation MMs.

Model	Level/Scope	Main Purpose	Data Source	Method Type	Function-/Scenario-Level?
Mittal et al. (2018) [10]	Organization/plant	Industry 4.0 readiness/maturity	Survey	Expert/MCDA	No
De Carolis et al. (DREAMY) (2025) [7]	Organization (manufacturing)	Digital readiness/roadmap	Survey + interviews	Expert/MCDA	Limited
Battistoni et al. (2023) [14]	Manufacturing SMEs/firm	DT adoption paths	Survey	PLS-SEM + NCA	No
Gürdür et al. (2019) [18]	Cross-industry/organization	Data analytics readiness	Web-based survey	Survey-based scoring	No
Manavalan & Jayakrishna (2019) [23]	Supply chain/organization	I4.0/sustainable SC readiness	Literature review	Conceptual framework	No (supply-chain level)
Rahamaddulla et al. (2021) [11]	Manufacturing SMEs/organization	Smart manufacturing readiness	Literature + conceptual synthesis	Conceptual readiness–MM	Limited
Saad et al. (2020) [27]	SMEs/value chain/design	I4.0 technology readiness (SSTRA)	Case/practitioner data	AHP-based assessment	Partial (design-related)
Fareri et al. (2020) [13]	Organization/HR	I4.0 impact on skills/profiles	Job descriptions/text	Text mining/NLP	No
Jamwal et al. (2025) [4]	Manufacturing SMEs/organization	I4.0 practices maturity	Survey	Fuzzy/multi-criteria maturity	No
Latino, M.E. (2025) [8]	Manufacturing SMEs/organization	Industry 5.0 implementation maturity	Self-assessment/survey	Expert/MCDA-style model	Limited
This study	Function/scenario (firm)	Scenario-level digital maturity	Public annual reports (text)	LLM-based semantic model	Yes (scenario-level)

compares 10 key models based on their assessment levels, evaluation purposes, data sources, and method types. While over 100 models are described in the literature, these 10 were specifically chosen to highlight the most relevant approaches, gaps, and limitations in line with the objectives of this study. The selected models encompass various assessment levels, data sources, and evaluation methods commonly used in MMs.

This comparative framework illustrates the dominant trends in existing research and highlights some inherent limitations. Many models focus on high-level organizational assessments, rely on subjective data sources such as surveys or expert judgment, and lack detailed analysis at the functional or scenario level. These gaps hinder a comprehensive assessment of digital transformation across various business functions. The following sections will explore these limitations in more detail.

(1)

Dominant focus on maturity and readiness evaluation

Over 60% of the reviewed models are primarily designed to evaluate readiness or maturity. This indicates that most frameworks are oriented toward identifying the current state of digital adoption, particularly in relation to emerging technologies such as IoT and AI [7,30,31,32]. Academic studies tend to emphasize descriptive frameworks, paying limited attention to how firms progress between maturity stages. While some models support basic benchmarking, they often provide limited guidance on improvement actions or resource allocation [3,33]. In most cases, the emphasis remains on answering “Where are you now?” rather than “How are different parts of the organization evolving, and what should be improved next?”

(2)

Emphasis on strategic-level evaluation

From the scope perspective, many of the models (over 40%) focus on strategic-level assessment. In contrast, the number of models addressing operational and technology-level evaluations is considerably smaller. Strategic assessments can help initiate high-level discussions but often lack the specificity required for implementation in concrete areas such as production, IT, and operations [34]. Most models do not provide diagnostics for specific organizational functions, such as manufacturing, IT, logistics, or human resources [29]. Cross-functional integration is also limited, particularly in terms of linking strategic intent with coordinated actions across operational and technological domains. This is partly because many models are designed to support top-level decision-making, helping executives align Industry 4.0 initiatives with business goals and digital investment planning [9,30]. In addition, conceptual elements such as vision, leadership, and innovation are more readily modeled and generalized across different organizations and sectors [3]. As a result, function-level diagnostics—such as those for manufacturing, IT, logistics, or HR—are addressed to a lesser extent in many models.

(3)

Prevalence of survey-based data and underuse of real-time or system data

Survey-based models far outnumber other types. Surveys are a commonly used method for deployment and scaling, especially across sectors with different digital maturity levels [7,29], and many models follow management research paradigms that favor self-reported questionnaires [28,33]. This reliance on survey instruments is also related to access limitations: researchers often lack access to live factory data due to privacy, system integration, or technical constraints [35].

However, surveys typically involve cross-sectional data collection and can only capture information at a specific moment in time, reducing their usefulness for continuous improvement or automated decision-making [16]. Without continuous data streams, most models provide one-off diagnostics with limited feedback mechanisms [36], and current models rarely explore the use of intelligent algorithms to process and interpret operational or text-based data.

(4)

Synthesized gaps

The review identifies three key gaps in current MM research for digital transformation in manufacturing industry:

• Lack of Function- and Scenario-Level Diagnostics: Existing models predominantly focus on organization- or strategy-level assessments, providing limited insight into how specific business functions or application scenarios progress asynchronously during digital transformation.
• Dependence on Subjective, Survey-Based Data: The dominance of self-reported questionnaires and expert scoring limits scalability, continuity, and objectivity, while underutilizing the potential of publicly available or system-generated data.
• Limited Use of Data-Driven and Text-Based Semantic Modeling: Although some models integrate quantitative and hybrid approaches, the systematic use of text mining and advanced semantic models (e.g., LLMs) to extract and quantify digital practices from unstructured data is still rare.

To address these limitations, future MMs should adopt a more modular, function-oriented, and data-driven architecture. Models ought to be structured around key organizational functions—such as production, supply chain, IT, maintenance, R&D, and HR—to enable their independent assessment using tailored criteria. This allows organizations to prioritize areas aligned with their strategic goals and transformation priorities. In addition, models should become more configurable, enabling the selection of maturity dimensions based on specific objectives, capabilities, and resource constraints. To improve accuracy and credibility, objective system-generated or publicly available data should be used, supporting real-time or at least regularly updated assessments of digital maturity in rapidly changing environments.

In manufacturing, digital transformation is typically incremental rather than uniform, often starting in areas such as production, maintenance, or logistics, where the value is clear and measurable. These pilot implementations gradually expand as technologies mature and organizational readiness grows. Successful transformation also depends on integrating enterprise software systems such as MES, which are inherently modular and support discrete operations (e.g., production scheduling, inventory control, quality management, customer service). Therefore, assessing transformation at the business-function level is often more effective than relying solely on aggregate organizational scores.

Accordingly, this study proposes a business-function-oriented maturity evaluation model. The model is structured around independent business functions, each assessed through quantifiable indicators derived from objective and publicly available data. This design has four advantages:

• Modularity—Each function is evaluated independently, capturing asynchronous transformation progress across different areas.
• Objectivity—System-generated or publicly available data are used, rather than purely subjective judgments.
• Practicality—The model employs clear evaluation logic with a low data collection burden, supporting benchmarking, monitoring, and cross-functional comparison without imposing excessive reporting requirements on firms.
• Quantifiability—Targeted, quantitative diagnostics guide stepwise transformation aligned with varied maturity levels and functional priorities.

With these features, the proposed model addresses the identified gaps and complements existing MMs, extending the literature from predominantly strategic, survey-based assessments toward a more fine-grained, function- and scenario-oriented, data-driven evaluation of digital transformation.

3. Scenario-Based Framework for Quantitative Evaluation of Digital Transformation in Manufacturing Enterprises

3.1. Digital Transformation Application Scenarios in Manufacturing Enterprises

In the context of digital transformation, application scenarios focus on addressing specific business problems by integrating digital software functions with enterprise processes.

For manufacturing enterprises, digital transformation often begins with operational challenges or tasks that offer significant potential to improve efficiency; these opportunities then serve as entry points for broader organizational expansion. From the perspective of efficiency and value creation, application scenarios represent concrete solutions that drive performance improvement in manufacturing firms. A digital application scenario can therefore be understood as a practical instance in which digital technologies and advanced manufacturing technologies are embedded into specific business functions or production processes to achieve process optimization, efficiency gains, or value creation. Examples include predictive maintenance, digital quality traceability systems, and automated material delivery. Beyond demonstrating technological application, these scenarios also reflect a firm’s digital capability and illustrate how technology can be embedded into core operations to reduce costs, enhance flexibility, and strengthen customer responsiveness. Thus, application scenarios act as a bridge between technological potential and business value, providing a vehicle for practical application while significantly influencing the effectiveness and sustainability of digital transformation initiatives.

Digital transformation’s gradual and uneven nature further underscores the importance of scenario analysis. Transformation rarely unfolds uniformly across an organization; instead, it progresses incrementally, often beginning in functions such as production, maintenance, or logistics—domains where digital interventions generate immediate and measurable value. These pilot practices typically expand as technological capabilities mature and organizational readiness improves. From this perspective, application scenarios can be regarded as structured practices that integrate one or more digital technologies into business functions or processes to drive improvement or innovation. They are not only solutions to business problems but also structured mechanisms that enable digital transformation to scale in line with enterprise capabilities.

The essence of scenario-based digital transformation lies in deriving technological implementation from the need to improve process efficiency. By identifying and continuously expanding application scenarios, enterprises can clarify transformation objectives, deeply integrate digital technologies with business processes, and pursue sustained innovation oriented toward value creation. Moreover, the evaluation of application scenarios can reveal their breadth (functional coverage), depth (technological maturity), and integration (extent of business alignment), thereby reflecting the staged characteristics of digital transformation and identifying weaknesses across departments or processes. These insights provide a basis for resource allocation and optimization. Importantly, the selection of application scenarios is highly flexible and can be adjusted according to the evaluation object, purpose, and scope.

3.2. Challenges in Using Annual Report Data to Evaluate Digital Transformation

One of the key challenges in evaluating enterprise digital transformation from the perspective of business scenarios lies in obtaining stable and reliable data on business operations. Consequently, corporate annual reports have become an important data source and have been used for text analyses in several studies [37,38,39]. However, in existing studies that employ annual report data to measure digital transformation [40,41], keyword-matching methods are typically based on string equivalence or containment, which cannot capture the semantic similarity among different textual expressions. Moreover, the keywords used are primarily names of digital technologies, which fail to reflect the actual digital transformation outcomes within enterprises.

To evaluate whether manufacturing enterprises have undertaken digital transformation in specific business scenarios and to determine the degree of such transformation, this study builds on existing research by integrating scenario-specific keywords with semantic analysis. Beyond literal keyword matching, LLMs (large language models) are employed to uncover broader semantic relationships, thereby complementing keyword matching and enhancing the comprehensiveness of the evaluation.

3.3. Evaluation Framework and Procedure

The proposed method as shown in Figure 1 employs annual reports as the primary data source. Typical manufacturing business scenarios are first selected as the analytical units, and textual analysis is performed accordingly. Keyword extraction and semantic similarity computation are then applied to derive three indicators—scenario coverage, scenario depth, and scenario consistency—forming a three-dimensional evaluation system for assessing the degree of digital transformation.

Compared with traditional approaches that rely on process-based segmentation or manually defined indices, the innovations of this framework can be summarized as follows: (1) business scenarios are adopted as the fundamental analytical unit, which better captures the practical logic of digital transformation in manufacturing enterprises; (2) a keyword–semantic model integration mechanism that combines expert knowledge with semantic embeddings is introduced to enhance the detection of target information in unstructured texts; and (3) a three-dimensional evaluation structure (coverage, depth, and consistency) is designed, which enables a systematic measurement from local semantic expressions to overall trend tracking. The evaluation procedure is outlined as follows:

(1)

Annual reports data collection and preprocessing

A cluster of listed manufacturing enterprises is identified based on the evaluation objectives. Their annual reports are then collected and preprocessed, with the Management Discussion and Analysis (MD&A) section retained for further analysis.

(2)

Definition of scenario keywords

First, scenarios are selected according to the current evaluation objectives. For each scenario, core keywords (CKs) are identified, and then corresponding extended keywords (EKs) and negative keywords (NKs) are derived. The number of CKs, EKs, and NKs can be flexibly determined based on the evaluation purpose and specific conditions.

CKs represent the fundamental characteristics of digital transformation, typically expressed through key processes, technologies, or methods. They explicitly capture the outcomes of digital application, integration, and transformation. EKs complement the Core Keywords and generally include concepts and domain-specific descriptions related to the core technologies or methods. They provide additional detail on the specific business processes or scope of digital transformation, thereby enriching the description of each scenario. NKs denote negations, adverse terms, or exclusionary expressions (e.g., “underperformed,” “lack of resources,” “non-strategic direction”). The combined use of CKs, EKs, and NKs—where CKs define the direction, EKs broaden the scope, and NKs eliminate noise—enables a more comprehensive understanding of textual semantics.

(3)

Calculation of scenario keyword intensity

When calculating keyword intensity for each scenario in annual reports, different weights are assigned to core keywords (CKs), extended keywords (EKs), and negative keywords (NKs). For example, CKs are assigned a weight of 1.5, EKs a weight of 1.0, and NKs a weight of −0.8. The intensity score for each scenario is then computed by combining the frequency of each keyword type with its assigned weight.

CKs carry the highest weight to anchor the core theme of the scenario. Even in lengthy texts, CKs keep the evaluation focused and allow rapid identification of the central elements of digital transformation, ensuring strong alignment between the assessment and the business scenario. EKs are assigned moderate weights and are used to capture derivative expressions and indirect associations of the core theme. They help recognize different formulations of the same concept, supplementing industry-specific terminology, synonyms, or scenario extensions, thereby enhancing the reliability of the assessment. NKs are assigned negative weights to suppress noise or adverse information by detecting negation patterns and lowering the corresponding scenario score. This adjustment improves the ability to identify the true strategic intent expressed in annual reports.

When CKs, EKs, and NKs are jointly used to calculate keyword intensity, each of their contributions can be traced individually, ensuring interpretability and facilitating verification. In addition, by adjusting the keyword library and weights, the sensitivity of the evaluation can be flexibly calibrated, making the method adaptable to different assessment targets.

(4)

Semantic similarity calculation

The keyword matching is based on string equivalence or containment. However, the number of keywords is limited, and their semantics are often narrow. Directly searching for a keyword’s occurrence in annual reports fails to adequately account for synonyms, near-synonyms, or alternative expressions.

This study employs LLMs to learn semantic associations among word vectors, thereby capturing deeper semantic information. By calculating semantic similarity, more comprehensive scenario-related information can be extracted from annual reports. Even when specific words do not appear explicitly, semantic similarity allows them to be recognized as related, thus complementing the limitations of simple keyword matching.

The results of semantic similarity computation further improve keyword intensity analysis by evaluating the clarity of digital transformation expressions. They also assess scenario consistency—the consistency between the intended objectives and actual outcomes of digital transformation in specific business scenarios.

For the computation, LLMs (e.g., Word2Vec, BERT, Transformer) are first employed to extract core semantic anchors, which serve as the foundation for semantic understanding, from the scenario keyword table. Next, the LLMs transform the annual report texts into multidimensional vectors, representing their overall semantics rather than superficial lexical forms. Cosine similarity is then applied to measure the similarity between semantic vectors, producing a semantic similarity score, which is normalized to a value between 0 and 1.

In this paper, the normalized semantic similarity score between the annual report text and the scenario keywords is denoted by ${\mathrm{S}\mathrm{S}}_{\mathrm{i}}$, and the processing steps are outlined in Appendix A. This score is used as the scenario consistency measurement.

Through the above steps, deep semantic recognition and matching between the MD&A section of annual reports and the scenario keywords can be achieved.

In the three core tasks involved in the above process—semantic anchor understanding, text semantic vectorization, and semantic similarity computation—different language models significantly differ in their adaptability to Chinese and English corpora. Specifically, semantic anchor understanding places high demands on a model’s capacity for contextual modeling and semantic completion. This is particularly critical when dealing with short keywords with insufficient contextual information, where the model must possess a certain level of “semantic extrapolation” ability. Text semantic vectorization, in contrast, emphasizes the model’s ability to achieve consistent encoding across texts of varying lengths while maintaining semantic structural stability. Semantic similarity computation requires the embedding space to exhibit well-organized semantic distribution characteristics, such that semantically similar text pairs are represented with smaller distances in the vector space. For tasks involving semantic anchor understanding and semantic similarity computation, different language models demonstrate distinct strengths in language coverage, application scenarios, and performance.

(5)

Three-dimensional evaluation

During the evaluation, the consolidated annual report text is treated as a whole and assessed for each predefined scenario separately. For each scenario, a three-dimensional evaluation framework is constructed based on keyword frequency, keyword strength, and semantic similarity. This framework comprises scenario coverage, scenario depth, and scenario consistency and is designed to comprehensively assess the degree of digital transformation. Scenario coverage is measured by the number of scenario-related keywords appearing in the annual report, thereby evaluating the breadth of digital transformation. Scenario depth is determined using keyword weights together with the frequency and number of keywords within each scenario, thus evaluating transformation depth. Scenario consistency is assessed by supplementing the analysis with semantic similarity, which evaluates the coherence and clarity of transformation-related keyword expressions within a scenario. The computational logic of these three indicators is summarized in

Table 2. The calculation principles of the three indicators.

Indicator Name	Calculating Principles	Equation No
Scenario coverage	(The number of keywords appearing in the text/Total keywords) × 100%	Equation (A2) in Appendix B
Scenario Depth	Σ(Weights of different types of keywords × Frequencies of different types of keywords)/(Weight of the core keywords × keyword frequency) × 100%	Equation (A3) in Appendix B
Scenario consistency	Normalized semantic similarity value	Equation (A4) in Appendix B

, and the detailed process of calculating scenario coverage and scenario depth is presented in Appendix B.

The scores across different dimensions reflect a firm’s transformation status within each scenario. By integrating the values of scenario coverage, scenario depth, and scenario consistency, the Digital Transformation Index (DTI) can be derived to evaluate the degree of digital transformation for an individual scenario.

$${\mathrm{D}\mathrm{T}\mathrm{I}}_i={\displaystyle \frac{{\mathrm{S}\mathrm{C}}_{\mathrm{i}}+{\mathrm{S}\mathrm{D}}_{\mathrm{i}}+{\mathrm{S}\mathrm{S}}_{\mathrm{i}}}{3}}.$$

(1)

The evaluation value of a single scenario can be used to assess different transformation scenarios for a target group of enterprises. Furthermore, aggregating the evaluation values across all scenarios provides an overall assessment of a firm’s digital transformation. In this study, the mean value of all scenario-specific evaluation scores is defined as the Overall Digital Transformation Index (ODTI).

$$\mathrm{ODTI}={\displaystyle \frac{\sum _{\mathrm{i}=1}^{\mathrm{n}\mathrm{s}}{\mathrm{D}\mathrm{T}\mathrm{I}}_{\mathrm{i}}}{\mathrm{n}\mathrm{s}}}$$

(2)

where $ns$ is the predefined number of scenarios.

4. Data Example and Analysis

To validate the effectiveness of the proposed method, listed manufacturing enterprises in Zhejiang Province are used as an empirical case study for evaluation and analysis. Manufacturing in Zhejiang accounts for 35% of the provincial GDP, exceeding the national average of approximately 27%. The province hosts 45 national-level intelligent manufacturing demonstration factories (over 10% of the national total), 32 global “lighthouse factories” (representing one-fifth of China’s total), and 18 national-level industrial clusters, covering both traditional and emerging industries. In addition, Zhejiang has more than 450 A-share listed manufacturing firms. With its strong manufacturing base, advanced digital transformation process, comprehensive policy support, and high data completeness, Zhejiang constitutes a representative case for examining the digital transformation of China’s manufacturing sector.

4.1. Annual Report Data Acquisition and Sampling Strategy

Following prior studies on annual report sampling [40,42], manufacturing firms listed on the Shanghai and Shenzhen A-share markets with registered addresses in Zhejiang Province between 2014 and 2023 were identified using the Choice data platform. Industry classification was based on three systems: (1) the CSRC “manufacturing” category; (2) selected SW categories, including machinery, automotive, electrical equipment, electronics, petrochemicals, and basic chemicals; and (3) the CSI “industry” category. The results from the three systems were merged and manually screened to exclude non-manufacturing firms, yielding the final list of Zhejiang manufacturing enterprises. To ensure data completeness, only firms with at least five consecutive years of annual report data were retained. The full texts of annual reports were then downloaded from the Choice database “http://choice.eastmoney.com (accessed on 10 June 2025)”. Prior to evaluation, annual reports were preprocessed and only companies meeting the five-year continuity requirement were included in the final sample.

4.2. Three-Dimensional Keyword Mapping Table Construction

According to the Implementation Guidelines for Digital Transformation of Manufacturing Enterprises, jointly issued by the Ministry of Industry and Information Technology and two other departments, the value-creation processes of manufacturing enterprises can be categorized into six typical digital transformation scenarios: research and design, production and manufacturing, operation and maintenance services, business management, supply chain management, and cross-linkage collaboration. In addition to defining these scenarios, the guidelines also delineate their functional scopes. The literature on the digital transformation of manufacturing enterprises further discusses scenario descriptions and transformation-related evaluation keywords.

By building on the scenario definitions and descriptions provided in the policy guidelines and integrating relevant domestic and international research, a mapping table of scenario keywords adapted for annual report texts was developed. As shown in Appendix C, the mapping table includes CKs, EKs, and NKs.

Given the continuous development and iteration of digital technologies and their applications in manufacturing, as well as the varying purposes of digital transformation evaluation, scenario definitions are not immutable. They may be adjusted according to specific evaluation objectives and prevailing conditions.

4.3. Evaluation Process

4.3.1. Calculation of Keyword Intensity

From the previously collected ten years of annual reports, the “Management Discussion and Analysis” (MD&A) sections were extracted for analysis. Since the reports are written in Chinese, the JIEBA segmentation tool was first applied to perform word segmentation and remove stop words. The predefined keyword mapping table was then loaded, and weights of 1.0, 0.6, and −0.8 were assigned to “CKs” “EKs” and “NKs” respectively. For the six manufacturing scenarios: research and design (S1), production and manufacturing (S2), operation and maintenance services (S3), business management (S4), supply chain management (S5), and cross-linkage collaboration (S6), the frequency of each keyword appearing in the corresponding scenario was calculated, and the total number of identified keywords within each scenario was recorded. Finally, by applying Equations (A2) and (A3) in Appendix B, the keyword frequencies, counts, and weights were integrated to obtain the evaluation values of scenario coverage and depth.

4.3.2. Calculation of Semantic Similarity

As noted above, computing semantic similarity based on keywords involves three steps: semantic anchor understanding, text semantic vectorization, and semantic similarity computation. Since the case study entails evaluating semantic similarity in Chinese annual reports of manufacturing enterprises, the text structure is relatively stable, the anchor keywords have clear semantic meanings, and noise interference is limited. Therefore, employing a single model to perform semantic anchor understanding, text semantic vectorization, and semantic similarity computation in a unified manner provides a cost-effective solution.

To compare the results of different large language models, three representative sentence embedding models were selected for the semantic similarity evaluation experiments: SimCSE-chinese-roberta-wwm-ext, paraphrase-multilingual-MiniLM-L12-v2, and text2vec-base-chinese. These models differ in network architecture, pre-training corpora, and training objectives, thereby reflecting their performance in semantic alignment tasks from different perspectives.

By introducing models with different architectures and training mechanisms for comparative analysis and examining the evaluation results from multiple perspectives, such as inter-model differences, the robustness and generalizability of the evaluation method can be validated, which enhances the reliability of the overall analysis and the persuasiveness of the conclusions.

Based on the annual report data of the manufacturing enterprise group described in Section 4.1, the three Chinese sentence embedding models were applied to evaluate digital transformation levels across six typical manufacturing scenarios (S1–S6) over the ten-year period from 2014 to 2023. The scenarios include research and design (S1), production and manufacturing (S2), business management (S3), operation and maintenance services (S4), supply chain management (S5), and cross-linkage collaboration (S6). This process yielded three score matrices of size 10 × 6. The evaluation results are aggregated at the “year–scenario” level, resulting in a relatively small outcome matrix. However, each value is derived from a large corpus of annual report texts, ensuring that the results capture macro-level trends. The evaluation results are analyzed below.

5. Evaluation Results and Trend Analysis

Although the primary contribution of this study is methodological—developing a scenario-based DTI/ODTI evaluation framework—measurement research should also demonstrate that the resulting index behaves in theoretically meaningful ways. Therefore, the empirical patterns reported in this section are presented descriptively, but they also serve as construct-validation evidence rather than purely narrative description. In particular, we examine whether (i) the temporal evolution of DTI/ODTI is consistent with widely discussed diffusion and deepening processes of digital transformation, and (ii) the observed scenario-level dynamics correspond to staged advancement of intelligent manufacturing in Zhejiang Province. We also report cross-model consistency as reliability-oriented validation evidence.

5.1. Comparative Analysis Across Models

Table 3. Average evaluation values of S1–S6 per model from 2014 to 2023.

	SimCSE	Text2Vec	Paraphrase
S1	0.4031	0.4989	0.3983
S2	0.1792	0.269	0.2011
S3	0.4205	0.5415	0.4488
S4	0.1159	0.1814	0.1331
S5	0.2354	0.2802	0.2691
S6	0.0566	0.1188	0.0744

and Figure 2 present the ten-year average scores of the three models across the six evaluation scenarios (S1–S6). The results indicate noticeable differences in score ranges among the models. The Text2Vec model consistently achieves higher scores, with the highest score in S3 (0.5415) and the lowest in S6 (0.1188). In contrast, the SimCSE and Paraphrase models exhibit relatively low average scores, primarily distributed between 0.07 and 0.45.

Table 4. Average evaluation values of S1-S6 per model for each year from 2014 to 2023.

	2014	2015	2016	2017	2018	2019	2020	2021	2022	2023
SimCSE	0.1826	0.1601	0.1762	0.1871	0.1806	0.1911	0.1935	0.3727	0.3548	0.3526
Paraphrase	0.1966	0.1759	0.1915	0.1877	0.1977	0.2113	0.2173	0.3999	0.3836	0.3799
Text2Vec	0.2515	0.2309	0.2496	0.2641	0.2568	0.2680	0.2742	0.4639	0.4457	0.4450

and Figure 3 show the time series of the overall digital transformation evaluation index (ODTI) from 2014 to 2023.

From a trend perspective, although the three models exhibit systematic shifts in their absolute scores, their evaluation results demonstrate a high degree of consistency in relative temporal patterns (see Table 4 and Figure 3). Specifically, the average annual scores of all three models gradually increased between 2014–2020, followed by a pronounced rise between 2020–2021. In the subsequent two years, the scores of all models slightly fluctuated but remained relatively stable, without any clear trend divergence. These consistent temporal dynamics suggest that, while the models differ in their scoring scales, they respond to semantic feature variations in a largely uniform direction when applied to the same time period.

A closer examination of the trends shown in Figure 4 for the three models (SimCSE, Paraphrase, and Text2Vec) across scenarios (S1–S6) reveals that, despite differences in absolute scores due to model architecture and training strategies, SimCSE, Paraphrase, and Text2Vec exhibit highly consistent temporal patterns across S1–S6. Notably, all three models peaked in S1 in 2021, with scores remaining high but slightly declining thereafter; a comparable surge was observed for S3 in the same period. These results indicate that mainstream Chinese sentence embedding models possess strong substitutability and cross-validation capabilities when applied to structured semantic texts such as corporate annual reports, reinforcing the reliability of similarity-based approaches for index modeling, trend detection, and intelligent classification and validating the effectiveness of the proposed methodology.

Altogether, the high consistency in relative temporal patterns across models provides reliability-oriented validation evidence for the proposed framework, suggesting that the observed trends are not driven by a particular embedding model.

5.2. Overall Analysis of Digital Transformation Trends Across Enterprise Scenarios

To comprehensively capture the digital semantic expression across different business segments in corporate annual reports, the annual trend curves of six scenarios were plotted based on the overall evaluation values produced by the SimCSE, Paraphrase, and Text2Vec models (see Figure 5).

S1 (R&D design) remained at a relatively low level (0.2–0.4) between 2014 and 2020, indicating that digital transformation in R&D was still in its early stage. After 2021, scores rose sharply and remained at their highest levels through 2023, reflecting the concentrated release of investments and achievements in R&D related digital transformation in annual report. This trend suggests that R&D has recently become critical for breakthroughs in manufacturing enterprises.

S2 (Production and manufacturing) followed a steady upward trajectory, markedly accelerating after 2019 and reaching nearly 0.5 in 2023. This indicates that production, as the core domain of digital transformation, is progressively advancing from equipment upgrades toward more systematic transformation.

S3 (Business management) remained at a relatively high level throughout the period and rose rapidly after 2020, surpassing 0.7. This reflects the increasing recognition and application of digital value in strategic planning and resource allocation by management, positioning this scenario as a leading driver of digital transformation.

S4 (Operations and maintenance services) remained low overall but grew slowly, reaching only about 0.3 in 2023. This suggests that although technologies such as remote operations and intelligent maintenance are being gradually adopted, digital transformation in this domain has yet to see widespread diffusion.

S5 (Supply chain management) displayed a nonlinear trajectory, with relatively high scores during 2014–2016 but a decline to the lowest point in 2020, followed by stabilization. This pattern may be influenced by the increasing external complexity of supply chains and rising challenges in upstream–downstream coordination, highlighting the instability and lack of standardized expressions in supply chain digital transformation.

S6 (Cross-process collaboration) consistently had the lowest scores, with limited growth and poor interannual consistency. This indicates that cross-departmental and inter-firm data integration and business collaboration remain major challenges in digital transformation, requiring further breakthroughs in organizational mechanisms and technical standards.

5.3. Correspondence Between Enterprise Digital Maturity Trends and Stages of Intelligent Manufacturing Advancement in Zhejiang Province

As shown in Figure 6, the trends in the average digital transformation evaluation scores of the six key scenarios (S1–S6) for manufacturing enterprises from 2014 to 2023 reveal that digital semantic expression in annual reports is evolving in alignment with the staged advancement path of intelligent manufacturing in Zhejiang Province. This finding indicates that the evaluation based on semantic models not only captures the internal evolution of linguistic standardization within enterprises but also serves as an external mapping tool for assessing the effectiveness of regional digital transformation initiatives.

Phase I (2014–2017): National and provincial initiatives emphasized informatization and automation, such as the Machine Substitution program (2013) and Made in China 2025 (2015). Correspondingly, S1 (R&D) and S3 (management) showed early and stable improvements, indicating initial progress in digitalized R&D processes and management systems. However, S2 (production) did not exhibit a significant increase, suggesting lagging digital transformation in manufacturing.

Phase II (2018–2020): Policies focused on technological integration and system development, exemplified by the Intelligent Manufacturing Action Plan (2018–2020) and the provincial program on industrial internet platform construction (2019). The model results, however, show only limited gains in S1 and S2, while S4 (operations) steadily improved and S5 (supply chain) declined before stabilizing. This reflects the organizational complexity and external disruptions constraining digital transformation in production and collaboration scenarios.

Phase III (2021–2023): Policy emphasis shifted toward system integration and deeper digital application, as evidenced by the Opinions on Accelerating the Construction of “Future Factories” issued by Hangzhou (2021) and the national Industrial Internet Innovation and Development Action Plan (2021–2023). During this period, core scenarios S1, S2, S3, and S4 experienced substantial increases, reflecting the accumulated effects of earlier policy efforts. S5 (supply chain) rebounded slightly, but S6 (cross-process collaboration) remained at a low level, underscoring the persistent challenges of integration and inter-organizational collaboration.

Overall, the ten-year trend analysis reveals that the enterprise digital maturity in Zhejiang Province evolves following a distinct staged trajectory: beginning with internal management as a priority, progressing through production and operations, and advancing toward breakthroughs in system-wide collaboration. The digital and intelligent transformation of manufacturing enterprises is gradually shifting from isolated digital transformation of individual processes to holistic process-wide collaboration. This correspondence provides theory-consistent validation that the proposed index captures meaningful staged evolution rather than arbitrary narrative variation.

5.4. Other Issues Identified in the Ten-Year Scenario-Based Digital Transformation Evaluation

(1)

Variations in the Clarity of Digital Transformation Expressions across Business Scenarios

The scoring trends and semantic analysis reveal marked differences in the clarity of digital transformation expressions across business scenarios. These differences are evident not only in semantic density but also in the degree of linguistic standardization and the recognizability of technical terminology. Scenarios such as S2 (production) and S6 (cross-process collaboration) consistently fall within the lower score range, not solely due to weak enterprise capabilities but, more likely, because of ambiguous expression combined with model adaptation challenges.

For S2, persistently lower scores can be attributed to two main factors. On the one hand, production systems are largely internal infrastructure and thus are rarely elaborated upon in annual reports. On the other hand, terminology in this domain is highly heterogeneous and industry-specific, with little semantic standardization, making it difficult for semantic models to capture accurately. Furthermore, unlike strategic or management modules that benefit from a more communicable narrative logic, production-related practices lack systematic and standardized linguistic representation, hindering the extraction of underlying capabilities.

Collaboration capability, meanwhile, is widely regarded in both theory and policy as a hallmark of digital maturity—particularly since the industrial internet and platform-based collaboration have become strategic priorities. Yet, the results show that S6 scores have remained low over the decade, without sustained upward movement. This highlights the inherent difficulty of expressing cross-process collaboration in textual form. Its semantics involve multi-system and multi-organization interactions, with key expressions scattered across contexts such as “supply chain integration,” “system interconnection,” and “upstream–downstream coordination.” These elements lack standardized terminology, rely heavily on contextual inference and cross-paragraph linkage, and thus are not readily captured in semantic similarity calculations. Moreover, cross-process collaboration often entails organizational restructuring, data-sharing protocols, and platform-based mechanisms that are difficult to implement and slow to diffuse, making them less likely to be explicitly reported in annual documents.

Taken together, the persistently low scores of S2 and S6 reflect both limitations in enterprise reporting practices and methodological challenges in handling terminological diversity, semantic fragmentation, and dispersed logic. This suggests that future research should focus on dynamic keyword modeling, semantic standardization, and enhanced adaptation mechanisms for complex scenarios, thereby improving the model’s ability to capture and represent digital transformation across diverse business contexts.

(2)

Time Lag between Policy Priorities and the Transformation Effects Reflected in Textual Expressions

Although the overall scoring trends broadly align with Zhejiang’s trajectory of intelligent manufacturing, there is a noticeable temporal lag between policy deployment and enterprise textual expressions. In Phase I (2014–2017), S3 (management) and S1 (R&D) improved first, reflecting the orientation toward informatization, while S2 (production) did not respond to the goal of intelligent transformation. In Phase II (2018–2020), despite policies promoting ERP–MES integration and industrial internet platforms, gains in S1 and S2 remained limited. Only in Phase III (2021–2023) did the scores of core scenarios rise sharply, with S1 and S3 reaching high levels. This suggests that the semantic reflections of digital transformation in annual reports are not immediate responses, but typically lag behind policy pilots and system deployment, closely linked to the accumulation of prior technological efforts. The findings further indicate a consistent 1–2-year “policy-first, expression-lagging” window, reflecting a multi-level trajectory of digital transformation from institutional design to organizational practice and finally to external disclosure. This also highlights the proposed method’s sensitivity in structural alignment and phase detection, meaning that the enterprise transformation pace and strategic consistency can be inferred from publicly available texts.

5.5. Sensitivity and Robustness Analysis

To evaluate the robustness and reliability of the proposed DTI/ODTI framework, we conducted a sensitivity analysis in three ways: cross model robustness, keyword weights sensitivity, and external validity through comparison with benchmark enterprises.

5.5.1. Cross Model Robustness

Three representative sentence embedding models were used: SimCSE-chinese-roberta-wwm-ext, paraphrase-multilingual-MiniLM-L12-v2, and text2vec-base-chinese were used in this article. Table 4 and Figure 3 show the ODTI from 2014 to 2023 for these three models.

Spearman correlation coefficients showed highly consistent temporal trends among the three models, with values ranging from ρ = 0.98 to 0.99 (p < 0.01), indicating stable alignment in their temporal evaluations. The overall ranking pattern was consistent, with Text2Vec consistently outperforming the other two models, Paraphrase in the middle, and SimCSE yielding the lowest scores. This demonstrates that framework is robust across different models.

5.5.2. Sensitivity to Keyword Weights

To assess the DTI’s sensitivity to keyword weights, two tests are conducted:

(1)

Single-factor perturbation (±20% weight change):

The weights of each keyword category were adjusted by ±20%. The experimental results are shown in Figure 7. Adjusting the weight of core keywords by ±20% leads to significant changes in evaluation scores across all scenarios. In contrast, changes in the weights of extended and negative keywords had minimal effects, indicating that the framework is highly sensitive to the weight of core keywords, while other categories have little impact on the final score.

(2)

Extreme Weight Configurations

Several extreme weight configurations are also tested, including core keyword dominant (CK-dominant), balanced weights (Equivalent), and extended keywords dominant (EK-dominant). The evaluation results corresponding to these configurations are presented in Figure 8.

The core keywords dominant configuration showed stable results, similar to the baseline, while the extended keywords dominant configuration resulted in significant score reductions, particularly in the R&D scenario. This suggests that core keywords have the most influence, while extended words contribute little in the current dataset.

5.5.3. External Validity: Benchmark Enterprises vs. Other Firms

To validate the external applicability of the framework, we compared the DTI/ODTI scores of seven demonstration enterprises selected from the Future Factory Cultivation List in Zhejiang Province (2021) against the industry average from 2017 to 2023.

Figure 9 and Figure 10 show that the DTI/ODTI scores for these enterprises consistently exceeded the industry average, confirming that the framework effectively identifies companies with advanced digital transformation capabilities.

The above section shows that the sensitivity and robustness analysis demonstrate that the DTI/ODTI framework is reliable across different models, highly sensitive to changes in keyword weights, and capable of distinguishing leading digital transformation firms through external validation. These results highlight the practical applicability and external validity of the framework in real-world settings.

6. Discussion

6.1. Summary of Methods

The methodological contributions of this study are threefold. First, it introduces “application scenarios” as the fundamental unit of analysis, going beyond traditional approaches that rely on business processes or manually defined indicators. This design is more closely aligned with the practical logic of digital transformation in manufacturing enterprises and enables the identification of heterogeneous trajectories across different functional domains. While its application was demonstrated in the manufacturing sector, the framework can also be adapted to other industries for digital transformation assessment. Second, the method incorporates a “core–extended–excluded” keyword mechanism, which directs the semantic embedding process for unstructured texts. This mechanism not only enhances the ability to capture target concepts but also increases the transparency and interpretability of the evaluation, with adjustable weighting parameters to accommodate different analytical needs. By maintaining semantic accuracy while significantly improving interpretability and traceability, it provides a replicable pathway for subsequent semantic analysis. Third, the framework’s stability was further validated through multi-model comparison. Comparative experiments using three structurally distinct models—SimCSE, Paraphrase, and Text2Vec—demonstrated strong consistency in temporal trends (average Spearman correlation = 0.89), despite differences in raw scores. This indicates the cross-model generalizability of the approach and lays a foundation for its application in multilingual, cross-industry, and multi-task environments.

In sum, the proposed framework introduces methodological innovations in scenario-based structuring, keyword-enhanced semantic modeling, and cross-model validation, providing a generalizable and scalable tool for quantifying enterprise digital transformation. Moreover, the framework has the potential to be extended beyond manufacturing to applications across industries and international contexts.

6.2. Limitations and Future Improvements

Although the evaluation method proposed in this study performs well in trend identification and cross-model stability, several limitations remain. The keyword system heavily relies on expert knowledge, which limits its adaptability to emerging fields and rapidly evolving terminology. For example, new technical terms may not be recognized, leading to underestimated evaluation results. Additionally, keywords are typically short phrases with sparse semantics, which can cause misalignment when matching them to long, complex annual report texts, especially in cross-paragraph and cross-functional contexts. The framework also uses a single semantic model for keyword parsing, text encoding, and similarity computation, resulting in high task coupling and limiting scalability as corpus size or domain diversity increases. Finally, the method relies on static annual reports, which often lag behind actual organizational practices, potentially underestimating recent progress in digital transformation efforts.

Lastly, the framework does not integrate operational data, such as IoT or MES logs, which could offer more granular and real-time insights into digital transformation efforts. The integration of such proprietary operational data presents significant challenges, particularly in industries with varied data formats and structures. Moreover, the LLM-based semantic similarity approach may face difficulties when applied to emerging industries or cross-regional contexts. The current study is focused on a specific dataset from the manufacturing sector in Zhejiang Province, and applying the method to other industries or regions would likely require substantial re-engineering of keywords and adjustments to the model, limiting the framework’s generalizability and its ability to handle industry- or region-specific terminology effectively.

Given these limitations, future research could focus on the following areas: (1) Automatic Keyword Generation and Semantic Lexicon Expansion: Develop automatic mechanisms for keyword and semantic lexicon expansion, such as topic modeling and knowledge graph-based methods, to address the limitations of manually curated keywords and improve the framework’s adaptability to emerging and unstandardized terminology. (2) Context-Aware and Document-Level Modeling: Incorporate context-aware mechanisms, including paragraph- and document-level embeddings, to enhance the framework’s ability to recognize implicit statements and capture cross-paragraph relationships, especially in complex, multi-functional scenarios. (3) Model Architecture Optimization: Explore modular model architectures, where different models handle specific tasks, such as anchor extraction, text encoding, and semantic matching. Additionally, integrating graph neural networks or attention mechanisms to construct semantic dependency graphs could improve the scalability and consistency of the evaluation process. (4) Integration of Real-Time and Operational Data: Integrate real-time data sources, such as quarterly reports, social media, and IoT logs, to provide more up-to-date insights into digital transformation efforts. Time-series forecasting models (e.g., ARIMA or LSTM) could also be used to predict future trends and adjust evaluations to account for delays in policy implementation and reporting.

In conclusion, addressing these limitations through future optimization measures will enhance the method’s adaptability, scalability, and applicability across diverse industries, improving its effectiveness in digital transformation assessment and policy monitoring.