Systematic Annotation Framework for Robust Speech Recognition

Abstract

This study proposes a systematic annotation framework to improve the robustness of end-to-end automatic speech recognition (ASR) in a complex low-resource dialect setting, using Hainan Lingao dialect as a case study. The framework consists of three components: semantically complete utterance segmentation instead of fixed-duration clipping; structured annotation at the lexical, sentence, and pragmatic-behavior levels, including explicit tags for dialectal variation, environmental noise, and unintelligible speech as well as rules for handling overlapping speech; and a three-stage quality-assurance workflow with iterative guideline refinement. The framework was implemented in the construction of a Hainan Lingao dialect corpus from 16 speakers and evaluated using 80 h/10 h/10 h training, validation, and test splits under an identical Conformer-based ASR configuration. Compared with a plain-transcription baseline using no special tags and fixed 3 s segmentation, the full specification reduced character error rate (CER) from 8.7% to 7.9%, 24.3% to 18.5%, 19.5% to 15.2%, and 15.2% to 13.1% on clean, noisy, dialogue, and dialect-variation test sets, respectively. The corresponding sentence error rate (SER) decreased from 17.5% to 15.2%, 39.6% to 32.1%, 34.2% to 27.8%, and 28.3% to 24.5%. Ablation experiments further examined the individual contributions of pragmatic-behavior tags, noise tags, semantic segmentation, and dialect-feature annotation. Paired bootstrap testing with 10,000 resamples showed that all baseline-to-full-specification improvements were statistically significant (p < 0.01). These results indicate that systematic annotation can improve ASR robustness in this Lingao low-resource dialect setting, with the largest relative CER reductions observed in the noisy (23.7%) and dialogue (22.1%) scenarios.

1. Introduction

Automatic speech recognition (ASR) has shown strong potential in human–computer interaction, smart homes, in-vehicle systems, and accessibility services. Under quiet conditions with standard Mandarin, transcription accuracy has approached human performance. In real-world deployment, however, robustness remains a major challenge because background noise, device variability, accent diversity, speaking-rate variation, and overlapping speech jointly degrade recognition accuracy [1,2,3]. This limitation is especially severe in low-resource dialect scenarios, where the lack of high-quality annotated data remains a fundamental bottleneck [4,5].

Existing speech annotation schemes are still largely oriented toward standard pronunciation and quiet environments and treat plain-text transcription as the primary objective. As a result, they do not systematically capture dialectal sound shifts, environmental noise, or overlapping speech, all of which are common in practical ASR scenarios. In China, this problem is particularly pronounced because dialect resources are diverse and highly variable across initials, finals, tones, and mixed regional varieties [6]. As noted in prior surveys and corpus-construction studies of Chinese dialect speech recognition [7,8,9], the lack of unified and systematic annotation standards remains a major obstacle to progress.

Industrial guidelines such as Microsoft Azure [10] provide detailed text normalization. Academic schemes such as Discourse-CASS [11] offer deep linguistic annotation. However, existing frameworks still do not jointly address explicit noise tagging, pragmatic-behavior annotation, dialectal variation handling, and engineering scalability. To address this gap, this paper proposes a systematic annotation framework for robust speech recognition in low-resource dialect settings. The central research question of this study is whether a linguistically grounded, machine-readable, and scalable annotation framework can improve the robustness of end-to-end ASR under complex conditions, compared with conventional plain-transcription annotation. We formulate three working hypotheses. H1: A multi-level annotation framework integrating lexical, sentence-level, and pragmatic-behavior information will reduce CER and SER relative to a conventional annotation baseline under the same model configuration. H2: The performance gains will be most pronounced in acoustically and interactionally challenging scenarios, especially noisy and dialogue speech, because structured tags provide supervision beyond plain text. H3: The major components of the framework, including pragmatic-behavior tags, noise tags, semantic segmentation, and dialect-feature annotation, will each make an independent contribution to recognition robustness. As later shown in the case study, under the same model configuration, the conventional annotation baseline yields markedly higher CER and SER in noisy and dialogue conditions than the proposed specification.

Relative to existing industrial and academic annotation schemes, the novelty of this study lies in integrating semantically complete segmentation, machine-readable pragmatic and noise-event tagging, and a deployable three-stage quality-control workflow within a single operational specification for low-resource dialect ASR. Based on these hypotheses, this paper makes four main contributions. First, it formulates annotation-design principles centered on linguistic consistency, machine readability, and scalability. Second, it develops a hierarchical annotation scheme covering lexical, sentence-level, and pragmatic-behavior information. Third, it specifies fine-grained handling rules for dialectal variation, environmental noise, and overlapping speech, together with a three-stage quality-assurance and dynamic-optimization workflow. Fourth, it evaluates the framework through the construction of a Hainan Lingao dialect corpus and a set of comparative and ablation experiments.

The remainder of this paper is organized as follows. Section 2 reviews related work on speech annotation, dialect corpora, and robustness-oriented ASR and identifies the gap addressed in this study. Section 3 presents the proposed annotation framework and its design rationale. Section 4 describes the construction of the Hainan Lingao dialect corpus and the implementation of the annotation workflow. Section 5 reports the experimental results and discussion, and Section 6 concludes the paper.

2. Related Work

2.1. Evolution of Speech Recognition and Annotation Requirements

Speech recognition has evolved from isolated-word recognition to large-scale continuous speech recognition and then to end-to-end deep learning models, and the demand for data annotation has increased accordingly. In early studies based on dynamic time warping or hidden Markov models, annotation focused mainly on phone boundaries and isolated-word transcripts, and the schemes were relatively simple [12]. With the rise of deep learning, especially end-to-end models, the research paradigm changed fundamentally. End-to-end architectures represented by the Transformer [13] and Conformer [14] can learn the mapping from acoustic features to text sequences directly, thereby reducing reliance on forced alignment, but they also impose higher demands on the scale, quality, and contextual richness of training data [15]. Specifically, end-to-end models require longer context and coherent semantic units for effective sequence learning, whereas fragmentary, inconsistent, and low-information-density traditional annotation can no longer unlock their full potential. In recent years, researchers have also explored robustness improvements through decoupled speech-enhancement front ends, recognition back ends, and multimodal fusion. These architectural innovations implicitly require cleaner, more consistent training data as well as richer multimodal annotations, making annotation increasingly central to the development of speech recognition technology.

2.2. Dialect and Low-Resource Speech Corpora

In dialect and low-resource speech recognition, corpus construction is especially critical. In recent years, several representative dialect corpora have emerged, including THUYG-20 for Uyghur [16], ASR-CNanDiaCSC for Nanchang Gan Chinese [17], MDCC for Cantonese [18], and the large-scale dialect corpus WenetSpeech-Chuan [19]. These efforts have explored speaker diversity, domain-balanced text selection, and standardized recording conditions, thereby accumulating valuable resources for dialect speech recognition [20]. However, while corpus scale has expanded, research on annotation methods and specifications has lagged behind [21]. Three problems remain. First, annotation standards are fragmented and often lack design principles centered on linguistic consistency and machine readability, which makes corpora difficult to interoperate. Second, the handling of dialectal sound change, environmental noise, and overlapping speech often depends on annotators’ subjective judgment, which weakens internal consistency. Third, annotation is often limited to surface transcription and lacks structured descriptions of deeper information such as pragmatic behavior and acoustic scene. These issues directly constrain performance gains in dialect speech recognition, and in low-resource settings the shortage of high-quality annotated data has become a key bottleneck.

2.3. Large-Scale Speech Annotation Standards

As multilingual and low-resource speech recognition has gained attention, several large-scale speech corpora have emerged internationally, and their annotation philosophies and practices have significantly shaped the field [22]. Common Voice is a large-scale crowdsourced multilingual speech corpus for speech technology research and development, but it does not systematically specify fine-grained annotation for noise, accents, or dialogue structure [23]. VoxPopuli [24], built from European Parliament recordings, provides a large-scale multilingual corpus for ASR, semi-supervised learning, and speech translation, but its scenarios are relatively homogeneous and its coverage of everyday dialogue is limited. In industry, Microsoft’s Azure speech transcription guidelines [10] provide detailed text-normalization guidance for commercial ASR training, but they mainly target general scenarios and lack systematic annotation methods for dialogue behavior and complex acoustic events. In academia, Discourse-CASS [11] offers 18 layers of deep annotation for spoken Chinese discourse and is linguistically rich. However, its complex workflow and reliance on expert knowledge make direct engineering deployment difficult. This reveals a clear gap in current mainstream international standards between “engineering efficiency” and “linguistic depth”: industrial specifications emphasize scalability but offer limited information dimensions, whereas academic schemes emphasize theoretical completeness but are costly to implement. What is missing is a systematic annotation method that can balance both.

2.4. Annotation Requirements for Robust ASR

To improve ASR robustness in complex scenarios such as noise and accent variation, researchers have mainly focused on front-end signal enhancement, acoustic-model optimization, and data augmentation [25,26,27]. These methods, however, share a common limitation: annotated data are typically treated as fixed inputs, with insufficient attention paid to whether annotation information itself can further improve performance. Existing approaches, for example, rarely include fine-grained annotation of real noise, making it difficult for models to distinguish noise type, intensity, and speech content [28]. In dialogue settings, they also generally lack consistent tagging of interruptions, backchannels, and silence, so models cannot effectively learn the dynamic structure of conversation [29]. From an information-theoretic perspective, systematic annotation of noise events, dialogue behavior, and dialectal features can provide not only more precise supervision, but also direct support for modules such as environment classification and dialogue understanding, thereby improving robustness at the data level. In other words, further advances in robustness depend not only on algorithmic innovation, but also on a conceptual upgrade in data annotation. Overall, dialect speech recognition still lacks a scientific, systematic, and operational annotation scheme for building complex corpora. Against this background, this paper offers a systematic attempt to construct an annotation system spanning design, implementation, and validation, thereby providing a solid data foundation for robust speech recognition in real-world settings.

2.5. Research Gap and Problem Statement

The above review shows that existing work has made important progress in speech recognition, dialect corpus construction, and speech annotation, but three gaps remain, as summarized in

Table 1. Comparison of representative speech corpora and annotation approaches with respect to requirements for robust ASR.

Resource	Strength	Noise/Event	Dialogue/Pragmatic	Dialect/Low-Resource	Scalability	Main Limitation for This Study
Common Voice [23]	Massively multilingual crowdsourced speech corpus	Not systematic	Not designed	Broad coverage, but limited structured variation annotation	High	Lacks fine-grained annotation of acoustic events, interactional phenomena, and dialect-specific variation
VoxPopuli [24]	Large multilingual parliamentary speech corpus	Not systematic	Not designed for conversational interaction	Multilingual, but mainly formal parliamentary speech	High	Relatively homogeneous domain and interaction structure; limited coverage of spontaneous dialogue
Microsoft Azure [10] text-normalization guidelines	Practical normalization and customization support	Not designed	Not a primary layer	General ASR customization rather than dialect-specific annotation	High	Strong engineering usability but limited linguistic and interactional annotation depth
Discourse-CASS [11]	Rich discourse and spoken-interaction representation	Not a primary focus	Strong	Not optimized for low-resource dialect engineering workflows	Low to moderate	High annotation complexity and limited scalability for large-scale corpus construction
Representative dialect speech corpora (THUYG-20 [16], MDCC [18], WenetSpeech-Chuan [19])	Valuable low-resource and regional speech resources	Inconsistent across corpora	Rarely included systematically	Strong dialect relevance	Moderate	Typically emphasize corpus collection and transcription rather than unified, systematic annotation frameworks
This study	Multi-level annotation framework for robust dialect ASR	Explicit	Explicit	Designed for low-resource dialect scenarios	Moderate to high	Currently validated on a single dialect corpus; broader cross-corpus evaluation remains future work

. First, large-scale multilingual corpora and industrial transcription guidelines emphasize openness, scale, and engineering usability, yet they usually lack systematic annotation of complex phenomena that are critical to robust ASR, such as environmental noise, overlapping speech, dialogue dynamics, and dialect-specific variation. Second, linguistically rich academic annotation schemes provide deeper discourse and interactional information, but their complexity and annotation cost limit their direct applicability to large-scale engineering-oriented corpus construction. Third, existing dialect corpora provide valuable resources for low-resource ASR. However, most of them focus on corpus collection and transcription rather than on a unified, operational, and quality-controlled framework that jointly addresses lexical normalization, semantic segmentation, pragmatic behavior, and special acoustic events.

Therefore, the core problem addressed in this study is not merely how to build another dialect corpus, but how to design an annotation framework that simultaneously satisfies linguistic validity, machine readability, and engineering scalability for robust ASR in low-resource dialect settings. On this basis, the present work develops and evaluates a systematic multi-level annotation framework using the Hainan Lingao dialect corpus as a case study.

3. Proposed Method

Current speech-annotation practice often suffers from fragmented standards, inconsistent content, and limited information dimensionality. To quantify the effect of annotation quality on model robustness, we introduce an error-propagation model for annotation noise. Let X denote the acoustic features, Y the ground-truth labels, and Ŷ the noisy labels obtained in practice. If L_ann measures annotation error, the upper bound of the model generalization error can be written as follows:

$$\begin{array}{c}\mathsf{\epsilon }\le {\mathsf{\epsilon }}_{\mathrm{opt}} + \mathsf{\alpha }·\mathrm{E}\left[{\mathrm{L}}_{\mathrm{ann}}\right(\mathrm{Y},\widehat{\mathrm{Y}}\left)\right] +\sqrt{{\displaystyle \frac{\ln (1/\mathsf{\delta })}{2\mathrm{N}}}}\end{array}$$

(1)

where ε_opt is the model’s Bayes-optimal error, α is the model’s sensitivity coefficient to annotation noise, and δ is the confidence level. This expression shows that, once model architecture and data size are fixed, minimizing expected annotation error is the key to improving system robustness. Furthermore, to analyze the composition of annotation noise more deeply, we decompose it into systematic bias and random error. Let H(Y|X) be the conditional entropy of the true labels given the acoustic signal. Then, the uncertainty in the annotation process can be modeled as

$$\begin{array}{c}{\mathsf{\eta }}_{\mathrm{ann}}={ \mathrm{D}}_{\mathrm{KL}}\left({\mathrm{P}}_{\mathrm{gt}}\right|\left|{\mathrm{P}}_{\mathrm{ann}}\right)+\mathrm{H}\left(\widehat{\mathrm{Y}}\right|\mathrm{Y})\end{array}$$

(2)

where D_KL(P_gt‖P_ann) denotes the KL divergence between the true distribution and the distribution induced by the annotation specification, capturing the extent to which the specification itself departs from linguistic reality, that is, the systematic bias; and H(Ŷ|Y) denotes the random entropy introduced by annotator behavior after the specification has been fixed, reflecting the degree of annotation consistency. Traditional annotation methods often overlook the first term and fail to constrain the second effectively. The systematic annotation method proposed in this paper is intended to reduce the former through linguistic consistency and the latter through a strict quality-assurance mechanism, thereby systematically reducing annotation noise and ultimately improving speech-recognition robustness. The above formulations provide a conceptual and analytical justification for the proposed annotation framework.

3.1. Design Principles

3.1.1. Linguistic Consistency

An annotation system is more than a set of engineering symbols. It must reflect the internal structure of the target language. Before annotation begins, the target language or dialect should be analyzed at the phonological, grammatical, and pragmatic levels. For the Lingao dialect, the annotation units should not simply reuse the Mandarin initial-final system. They should instead follow the dialect’s own phonemic contrasts and syllable structure. Transcripts should also respect local syntactic patterns. Spoken phenomena such as repetition, repair, and ellipsis should be normalized only when necessary to preserve authenticity. They should not be forced into written-language forms. At the pragmatic level, the guidelines should capture conversational dynamics through tags for listener feedback, shifts in interactional control, and silence-to-speech transitions between turns. This consistency allows annotated data to preserve interactional information and to provide reliable learning targets for models [30,31].

3.1.2. Machine Readability

Machine readability is essential if annotation is to support model training effectively. All outputs should therefore be clear, unambiguous, and easy to parse. This requires a restricted symbol inventory. In this framework, only four punctuation marks are permitted: the comma, period, question mark, and exclamation mark. Their usage is explicitly defined to avoid symbols that are difficult for models to learn. English content is written in lowercase with spaces between words to reduce formatting noise. Non-textual speech events and pragmatic behavior are represented with structured tags, such as [*] for unintelligible speech, [ENS] for sudden noise events, and [节奏指令] (instruction-response tag) for responses to interactional prompts. These tags provide supervision beyond plain text. Each segment is also linked to detailed metadata, including speaker ID, gender, age, place of origin, recording device, and environmental noise level. Such information supports speaker and environment adaptation [32]. Without machine readability, even linguistically precise annotation cannot be fully exploited in model training [33].

3.1.3. Scalability

Scalability ensures that the annotation system can adapt to future developments and changing requirements [34]. Language is dynamic, and speech technology continues to evolve, so the annotation system must support modularity and version evolution. In this paper, the system is divided into a core layer and an extension layer. The core layer contains basic annotation such as lexical and sentence-level labels and must be followed in all tasks. The extension layer contains dialect tags, fine-grained noise tags, pragmatic-behavior tags, and the like, which can be selectively enabled according to specific research tasks, such as dialect recognition or meeting transcription. In addition, a formal version-management mechanism should be established. When new speech phenomena or research needs arise, new tags may be introduced after expert review, and their definitions and usage should be clarified through version updates. This allows the specification to evolve as understanding deepens and tasks change, while strict version control preserves the reusability of historical data and compatibility between old and new annotations, thereby maximizing the long-term value of the corpus [35]. For example, if “emotion tags” need to be added in the future, they can be defined in the extension layer without changing the core structure.

3.2. Multi-Level Annotation Framework

Building on the above principles, this paper proposes a three-level annotation framework spanning lexical units to pragmatic behavior. The proposed annotation framework can be interpreted within a structured-prediction view, in which lexical units, sentence structure, and pragmatic behavior provide complementary levels of supervision. Under this view, the speech-recognition task can be described as a structured prediction problem. This formulation makes the dependencies among annotation levels explicit. Let O denote the acoustic observation sequence and W the target text sequence. In end-to-end modeling, the posterior probability is maximized directly. Because direct mapping in complex scenarios often fails to capture fine-grained characteristics, we introduce a set of multi-level latent variables Θ = {V, S, U}, representing lexical units, sentence structure, and pragmatic behavior, respectively. The speech-recognition process can then be decomposed into the following joint-probability maximization problem:

$$\begin{array}{c}\mathrm{P}(\mathrm{W},\mathsf{\Theta }|\mathrm{O}) = \mathrm{P}(\mathrm{V}\left|\mathrm{O}\right)·\mathrm{P}\left(\mathrm{S}\right|\mathrm{V})·\mathrm{P}(\mathrm{U}\left|\mathrm{S}\right)·\mathrm{P}\left(\mathrm{W}\right|\mathsf{\Theta })\end{array}$$

(3)

In Equation (3), P(V|O) models the mapping from acoustic observations to lexical units and handles lexical boundaries and ambiguity resolution; P(S|V) characterizes syntactic dependency and topic coherence; and P(U|S) captures dialogue intention, affective tendency, and interaction strategy. Together, these levels capture information in the speech signal at different granularities, providing the model with supervision ranging from low-level acoustic structure to high-level semantics and pragmatics. At the same time, the levels are coupled through hierarchical conditional dependencies V → S → U, forming a bottom-up path of semantic enrichment while also supporting top-down attention guidance, thereby improving robustness and interpretability. The mathematical definitions and constraints of each annotation level are as follows:

3.2.1. Lexical-Level Annotation

In the multi-level latent-variable model Θ = {V, S, U}, lexical-level annotation corresponds to latent variable V. Its role is to map the acoustic observation sequence O to discrete lexical units and to provide a structured linguistic basis for subsequent sentence-level semantic modeling (S). To balance the openness of natural language with machine readability, lexical-level annotation reduces model uncertainty through a restricted symbol space. According to the machine-readability principle above, we define a restricted transcription symbol space as a normalized lexical set V, covering standard Chinese characters, normalized number forms, and lowercase English words. For any transcription sequence T, the legality constraint is

T ∈ V* ⇔ ∀x ∈ T, x ∈ V

(4)

By mapping open-vocabulary language into a normalized lexical set, this constraint effectively compresses the model’s search space, reduces the conditional entropy of the predicted text sequence, and makes training more stable and faster to converge. In practice, subsets of V should be specified more precisely for special categories such as numbers, English words, and modal particles so that linguistic consistency and machine readability can be maintained simultaneously.

Lexical-level annotation addresses lexical diversity and uncertainty by mapping continuous speech into discrete, normalized lexical units. Numbers must be transcribed into Chinese characters according to actual pronunciation. For example, ‘2381832’ should be rendered as ‘二三八幺八三二’ (the Chinese character sequence read digit by digit for ‘2381832’), which is more faithful to the nature of speech recognition than retaining Arabic numerals. Proper nouns such as personal and place names should use standard written forms when these are well-established; for less common items, transcription should follow pronunciation and commonly used characters, with explanations recorded in the metadata. As language contact intensifies, loanwords and mixed Chinese–English usage have become increasingly common. English words are written uniformly in lowercase, for example, ‘good’; abbreviations spelled out letter by letter should include spaces, for example, ‘a p p’; and complete English phrases should preserve inter-word spacing, for example, ‘good morning’.

Modal particles and fillers such as ‘呃’ (‘uh’), ‘啊’ (‘ah’), and ‘嗯’ (‘mm-hmm’ or ‘yes’) are important carriers of hesitation, thinking, affirmation, and related emotions or attitudes in speech. In this paper, they are transcribed uniformly with standardized characters bearing the mouth-radical form so that they can be distinguished from content words and annotated consistently. More specifically, the thinking sounds ‘e~~’ and ‘eng’ are uniformly rendered as ‘呃’ (‘uh’). The ‘en4’ sound used for affirmation is rendered as ‘嗯’ (‘mm-hmm/yes’). Internet spellings such as ‘额’ and ‘昂’ are normalized to ‘呃’ and ‘嗯’. Meaningless sounds such as ‘ang4’, ‘a4’, and ‘eng4’ are transcribed as ‘啊’ or ‘嗯’ according to actual pronunciation. The pronunciation ‘ei4 ei2’ is uniformly rendered as ‘诶’ (‘hey/eh’). Other fillers, including common Chinese discourse particles such as ‘呢、吧、啊、呃、呗、啵、嗯、哦、噢、唉、呐、哎嘛、诶、的、了、么’, are likewise normalized according to actual pronunciation, with mouth-radical forms used where applicable. In addition, clearly audible erhua forms should explicitly retain the character ‘儿’, as in ‘您今儿打算去哪儿啊?’ (‘Where are you planning to go today?’). This standardized transcription prevents the model from being confused by multiple written forms for the same sound. Examples are given in

Table 2. Lexical-level transcription annotation specifications.

Category	Original Content/Example	Formatting Requirement	Converted Example/Notes
Numbers and Symbols	e.g., “2022年” (“year 2022”), “超过90%” (“more than 90%”), “@用户” (“@user”)	Arabic numerals are transcribed according to their spoken Chinese pronunciation; special symbols are transliterated according to common readings.	“二零二二年” (spoken Chinese form of “2022”), “超过百分之九十” (“more than ninety percent”), “艾特用户” (“at user”)
English Words and Letters	e.g., “good morning”, “PPT”, “A区” (“Zone A”)	English words are uniformly lowercase, and letters are separated by spaces; common abbreviations are written as one lowercase word if pronounced as a word, or spaced letter by letter if spelled out.	“good morning” (word), “p p t” (letter spelling), “a区” (“Zone A”; lowercase letter + Chinese)
Modal Particles and Fillers	e.g., “呃…我想想” (“Uh…let me think”), “嗯, 是的” (“Mm-hmm, yes”), “这个…那个…” (“this…that…”)	Use standardized characters with the “mouth” radical to distinguish modal particles; preserve natural pauses and prosody.	“呃…我想想” (“Uh…let me think”), “嗯, 是的” (“Mm-hmm, yes”), “这个…那个…” (“this…that…”) (distinguish hesitation/thinking from affirmation)
Proper Nouns	e.g., “清华大学” (“Tsinghua University”), “COVID-19”, slang term “hin好” (“hin-good”; hin = very)	Well-known entities should use their standard names; dialectal or colloquial items with no established Chinese characters should, after investigation, be represented with standardized surrogate or homophonous characters.	“清华大学” (“Tsinghua University”), “c o v i d 一九” (“COVID-19” spelled letter by letter + numeral), “hin好” (using “hin” to mean “very”)

.

3.2.2. Sentence-Level Annotation

Sentence-level semantic annotation ensures that each annotation unit, typically an audio clip, is semantically and structurally complete. In the multi-level latent-variable model Θ = {V, S, U}, sentence-level annotation corresponds to latent variable S, which maps the continuous acoustic observation sequence O to a sequence of semantic units. Fixed-duration segmentation inevitably disrupts semantic continuity and cannot meet the long-context requirements of end-to-end models. In this paper, we define a set of boundary points B and seek the optimal segmentation sequence by maximizing the joint score of semantic completeness and acoustic purity:

$${\mathrm{B}}^{*} = \arg \underset{\mathrm{B}}{\max }\left[\mathsf{\lambda }·{\mathrm{Score}}_{\mathrm{sem}}\left(\mathrm{O}\right|\mathrm{B})+(1-\mathsf{\lambda })·{\mathrm{Score}}_{\mathrm{acou}}\left(\mathrm{O}\right|\mathrm{B})\right]$$

(5)

where Score_sem quantifies the semantic coherence of each segment, such as logical sentence structure and the appropriateness of semantic pauses, and can be evaluated using a language model or syntactic analysis; and Score_acou quantifies signal-to-noise ratio and speaker purity, such as SNR and speaker exclusivity, and can be measured using voice activity detection (VAD) and speaker separation. If the original observation sequence O is segmented into a set of non-overlapping audio fragments, then any valid fragment must satisfy the following:

$$\begin{array}{l}\left\{\begin{array}{l}\mathrm{S}\mathrm{p}\mathrm{e}\mathrm{a}\mathrm{k}\mathrm{e}\mathrm{r}\left({\mathrm{s}}_{\mathrm{i}}\right)=1\hspace{1em}(\mathrm{s}\mathrm{p}\mathrm{e}\mathrm{a}\mathrm{k}\mathrm{e}\mathrm{r} \mathrm{e}\mathrm{x}\mathrm{c}\mathrm{l}\mathrm{u}\mathrm{s}\mathrm{i}\mathrm{v}\mathrm{i}\mathrm{t}\mathrm{y})\\ \mathrm{L}\mathrm{e}\mathrm{n}\left({\mathrm{s}}_{\mathrm{i}}\right)\ge {\mathrm{T}}_{\mathrm{m}\mathrm{i}\mathrm{n}}\hspace{1em}(\mathrm{s}\mathrm{e}\mathrm{m}\mathrm{a}\mathrm{n}\mathrm{t}\mathrm{i}\mathrm{c}-\mathrm{c}\mathrm{o}\mathrm{m}\mathrm{p}\mathrm{l}\mathrm{e}\mathrm{t}\mathrm{e}\mathrm{n}\mathrm{e}\mathrm{s}\mathrm{s} \mathrm{d}\mathrm{u}\mathrm{r}\mathrm{a}\mathrm{t}\mathrm{i}\mathrm{o}\mathrm{n})\\ \mathrm{S}\mathrm{i}\mathrm{l}\mathrm{e}\mathrm{n}\mathrm{c}\mathrm{e}\left({\mathrm{s}}_{\mathrm{i}}\right)\cap {\mathrm{s}}_{\mathrm{i}}\le {\mathsf{\tau }}_{\mathrm{m}\mathrm{a}\mathrm{x}}\hspace{1em}(\mathrm{s}\mathrm{i}\mathrm{l}\mathrm{e}\mathrm{n}\mathrm{c}\mathrm{e}/\mathrm{b}\mathrm{l}\mathrm{a}\mathrm{n}\mathrm{k} \mathrm{c}\mathrm{o}\mathrm{n}\mathrm{t}\mathrm{r}\mathrm{o}\mathrm{l})\end{array}\right.\end{array}$$

(6)

Here, Speaker(s_i) = 1 indicates that fragment s_i contains speech from only one speaker. T_min is the minimum duration threshold for semantic completeness and is set to 6 s in this paper. Silence(s_i) denotes the silent intervals within fragment s_i. τ_max is the maximum allowed silence duration and is set to 1.5 s. These mathematical constraints ensure both acoustic purity and semantic coherence in the training samples. Based on the above optimization idea, we abandon traditional mechanical segmentation and instead treat audio segmentation as a systematic process that deeply integrates acoustic features with linguistic context, while formulating corresponding rules for segmenting audio signals. In the present implementation, semantic completeness was operationalized through punctuation-guided sentence boundaries and manual verification, while acoustic purity was enforced through VAD-based boundary detection, single-speaker exclusivity, and SNR screening.

Punctuation is an important part of sentence-level annotation because it structures the text and conveys intonation. This paper allows only four marks: the comma, period, question mark, and exclamation mark. Their mapping to speech prosody is explicitly defined so that punctuation corresponds directly to pauses and semantic structure in speech, providing the language model with prosodic and syntactic boundary cues and reducing uncertainty in text-structure prediction. The sentence-level semantic annotation specifications are shown in

Table 3. Sentence-level semantic annotation specifications.

Specification Dimension	Core Standard	Design Intent
Speaker Exclusivity	Strictly follow the principle that one audio segment contains clear speech from only one speaker; no overlap or mixed dialogue is allowed.	Eliminates interference from overlapping speech and improves the purity of the training data.
Semantic Completeness	Segmentation should be based on semantic units (such as complete sentences or clauses); recommended duration > 6 s and >20 characters.	Provide long-range context for end-to-end models and strengthen sequence learning.
Acoustic Environment Control	Retain natural pauses shorter than 1.5 s as speech-flow features; silence within a segment must not exceed 1.5 s; discard segments with overly low SNR (<50% recognizable content).	Faithfully reflect natural dialogue rhythm while removing invalid noise interference.
Symbol Set Restriction	Only comma (,), period (.), question mark (?), and exclamation mark (!) are allowed.	Avoid special symbols that are difficult for models to parse and ensure standardized, consistent input text.
Prosody Mapping	1. Sentence-type mapping: statements use periods, questions use question marks, and exclamations use exclamation marks.	Make punctuation correspond strictly to speech pauses and semantic structure, provide accurate prosodic and syntactic boundary information, and improve the naturalness and coherence of synthesis or recognition results.
	2. Within-sentence pause mapping: use commas to mark short pauses according to natural semantic breathing points.
	3. “Truncated” punctuation is forbidden: repair semantically incoherent short fragments (e.g., “小微, 等下, 出去, 加好友。” (“Xiaowei, later, go out, add friend.”) should be revised to “小微, 等下我们出去加个好友。” (“Xiaowei, let’s go out and add a friend later.”)).

.

3.2.3. Pragmatic-Behavior Annotation

Real-world speech interaction is full of irregular nonverbal behavior. To describe the interactional dynamics of dialogue, we introduce a set of pragmatic-behavior tags as additional supervision. For a dialogue segment, a mapping function is defined to structure non-textual interactional information. Mathematically, this is equivalent to introducing additional mutual information, such that

$$\begin{array}{c}\mathrm{I}(\mathrm{W};\mathrm{O},\mathrm{U})=\mathrm{I}(\mathrm{W};\mathrm{O})+\mathrm{I}(\mathrm{W};\mathrm{U}|\mathrm{O}) \ge \mathrm{I}(\mathrm{W};\mathrm{O})\end{array}$$

(7)

This shows that pragmatic-behavior tags can provide information beyond pure acoustic features and thereby improve recognition accuracy. For example, the [附和] (backchannel) tag marks short agreeing or responsive expressions in dialogue, such as ‘对’ (‘right/yes’) and ‘嗯嗯’ (‘mm-hmm’), helping the model capture turn-taking behavior; the [节奏指令] (instruction-response) tag marks direct responses to interactional prompts, such as ‘到’ (‘present’) during roll call, and provides precise supervision for intent recognition; and the [打破沉默] (silence-breaking utterance) tag marks utterances that reopen a conversation after a long silence, such as ‘诶, 人呢?’ (‘Hey, where is everyone?’), helping the model handle long-range dependencies and dialogue-state tracking. These tags turn plain text transcription into a richer record of dialogue behavior, substantially increasing the value of the data for training dialogue systems and enabling models to learn interaction patterns that more closely resemble real human communication. The pragmatic-behavior annotation specifications are summarized in Figure 1.

3.3. Handling Special Speech Phenomena

In real-world speech interaction, phenomena such as dialect-specific pronunciation, environmental noise, and overlapping speech are pervasive and pose serious challenges to annotation consistency and accuracy. To address these complexities systematically, this paper develops an annotation model based on decision boundaries, transforming subjective auditory judgment into quantifiable and verifiable criteria, thereby reducing annotator arbitrariness and improving both the theoretical consistency and the engineering robustness of the annotated data.

(1) Annotation of dialect-specific pronunciation: Dialect-specific pronunciation is one of the central challenges in Chinese dialect recognition. For languages such as the Lingao dialect and Southern Min, which are spoken but lack a standardized script or have only incomplete writing systems, sufficient phonetic investigation must be conducted before annotation to determine the phonological framework. International Phonetic Alphabet transcription may be used, or near-sounding Chinese characters may be borrowed as phonographic surrogates, but a single transcription strategy must be maintained consistently across the entire corpus. More importantly, speaker information must be tightly linked to the speech data. Metadata should record not only the speaker’s township, but also sociolinguistic variables such as age, gender, education level, and language acquisition history. These data make it possible to analyze the effects of age grading and geographic variation on speech and also provide a basis for training models that adapt to different speaker characteristics. For accent strength, optional labels such as mild, moderate, and strong can be introduced to provide the model with finer-grained learning targets.

(2) Speech annotation in noisy environments: Speech annotation under noisy conditions requires a clear distinction between speech and non-speech components and between transcribable and non-transcribable segments. To formalize this logic, the decision boundary is expressed in terms of SNR, speech activity, and noise-event likelihood. For an audio segment, the annotation decision function is defined as follows:

$$\mathrm{\phi }({\mathit{o}}_{\mathit{t}}) = \left\{\begin{array}{ll}\mathrm{Transcribe}\left(\mathrm{w}\right),& \mathrm{if} \mathsf{\gamma }\left(\mathrm{t}\right) > {\mathsf{\tau }}_1\mathrm{and} \mathrm{P}\left(\mathrm{speech}\right|{\mathrm{o}}_{\mathrm{t}}) > {\mathsf{\theta }}_1\\ \mathrm{Tag}\left(\right[\mathrm{ENS}\left]\right),& \mathrm{if} \mathrm{P}\left(\mathrm{noise}\right|{\mathrm{o}}_{\mathrm{t}}) > {\mathsf{\theta }}_2\\ \mathrm{Tag}\left({[}^{*}\right]),& \mathrm{if}{ \mathsf{\tau }}_0 < \mathsf{\gamma }\left(\mathrm{t}\right) \le {\mathsf{\tau }}_1\end{array}\right.$$

(8)

Here, τ₁ is the SNR threshold for transcribable speech, θ₁ is the speech-confidence threshold, θ₂ is the threshold for identifying a noise event, and τ₀ is the lower bound of minimally intelligible SNR. This decision function converts traditional auditory judgment into a threshold-based decision problem grounded in acoustic features, effectively reducing annotator subjectivity. When the SNR exceeds τ₁ and speech confidence exceeds θ₁, the segment is transcribed as text. When the noise-event probability exceeds θ₂, the [ENS] label is assigned and no text is transcribed. When the SNR falls between τ₀ and τ₁ and the speech content cannot be identified reliably, the [*] label is assigned to avoid speculative transcription errors. In addition, metadata should record detailed noise types, such as traffic noise, background conversation, and fan noise, and, where possible, graded noise levels. This provides direct support for training environment classifiers and targeted data augmentation, thereby systematically improving model robustness in the corresponding acoustic scenarios.

(3) Handling overlapping speech and pauses: The treatment of overlapping speech and pauses is critical to conversational speech recognition. Suppose the audio signal is S(t), where K is the number of speakers. When K ≥ 2, speech overlap exists. This paper follows the dominant-speaker principle: the signal component with the largest energy and complete semantics is retained as the target speech, that is,

$${\mathrm{S}}_{\mathrm{target}}=\arg \underset{{\mathrm{S}}_{\mathrm{k}}}{\max }\left(\int {|{\mathrm{S}}_{\mathrm{k}}\left(\mathrm{t}\right)|}^2\mathrm{dt}·\mathrm{Integrity}({\mathrm{S}}_{\mathrm{k}})\right)$$

(9)

where Integrity(S_k) denotes semantic completeness and can be binarized according to linguistic rules, such as whether the signal forms a complete sentence or clause. If no separated component satisfies the completeness requirement, the segment is discarded according to the speaker-exclusivity constraint in Equation (6). Although this may result in a small amount of data loss, it aligns the distribution of the training data with the ideal recognition distribution and prevents the model from learning spurious mixed features.

For pauses in speech, we adopt a duration-based strategy: short pauses of less than 1.5 s are retained within the same utterance and are usually represented by commas so as to preserve breathing and rhythmic features in the natural speech stream, whereas long pauses exceeding 1.5 s serve as natural sentence boundaries. For independent non-linguistic sound events, such as coughing or throat-clearing, if they can be cleanly isolated, they are marked independently with sound-event tags and are not accompanied by text transcription. Although these fine-grained treatments increase annotation complexity, they preserve natural fluency to the greatest extent possible and provide a high-quality foundation for models to learn rhythm, turn-taking, and interaction structure in dialogue.

3.4. Quality Assurance and Validation

To ensure high accuracy and consistency of annotated data, this paper establishes a human–machine collaborative quality-assurance and validation mechanism that covers the entire workflow (see Figure 2) and defines strict acceptance criteria (see

Table 4. Speech acceptance criteria.

Acceptance Type	Acceptance Dimension	Acceptance Standard	Acceptance Method
Single Speaker	Language	Pure dialect; pronunciation covers the various local varieties of the dialect region in equal proportions.	Manual
	Format	WAV	Automatic
	Channel	Mono	Automatic
	Sampling Rate	16 kHz, 16 bit	Automatic
	Speech Amplitude	3000–20,000 smpl; checked in Cool Edit.	Automatic
	Background Noise	<300 smpl; no abnormal interference sources such as echo or reverberation; checked in Cool Edit.	Automatic
	SNR	SNR > 20 (SNR = speech energy/noise energy); 15–20 may still pass if transcription is normal; checked in Cool Edit.	Automatic
	Duration	Typically 6–12 s per utterance after semantic segmentation; clips shorter than 6 s are automatically flagged for review.	Automatic
	Style	Use keyword-guided free speaking to closely reproduce the naturalness of free conversation; recording must be real, natural, and fluent, with no reading style.	Manual
	Audio Quality	Pronunciation is clear and full and can be transcribed normally.	Manual
Batch	Gender	Male:female = 1:1	Automatic
	Age	Ages 18–25 account for 20%, 26–40 for 40%, and over 40 for 40%.	Automatic
	Number of Speakers Covered	Effective duration per speaker must not exceed 40 min (customizable as needed).	Automatic
	Accepted Duration	Counted after annotation. Effective duration = total duration with annotation results after removing invalid, non-transcribable speech.	Manual

). Operating at three coordinated levels, consistency checking, expert review, and dynamic optimization, this mechanism forms a closed quality-control loop spanning initial annotation, review, and acceptance, upgrading traditional experience-driven quality control into a data-driven scientific management framework.

At the consistency-checking stage, we developed automated validation scripts to examine the formatting of annotation results, covering key dimensions such as tag compliance, punctuation usage, matching between audio files and transcripts, and the precision of silence trimming. Cross-validation is also conducted regularly: 5–10% of the annotated data are randomly sampled and reassigned to different annotators for independent re-annotation, and inter-annotator agreement (IAA) is calculated to quantify consistency. Items with low agreement are analyzed to clarify ambiguities in the guidelines and to generate supplementary explanations, thereby informing subsequent revision of the annotation specification.

At the expert-review stage, a two-tier process consisting of annotation review and annotation acceptance is implemented. During review, senior annotators conduct either full inspection or intensive sampling of the initial annotations, focusing on transcription accuracy, appropriateness of tag usage, and suitability of sentence segmentation. During acceptance, the project lead or linguistic experts perform final sample-based evaluation. To ensure scientific rigor and representativeness in sampling, this paper introduces a model for determining the sample size under stratified sampling. Let the total batch size be N. If the acceptance result is required to have an error no greater than E under confidence level 1 − α, the minimum sample size is given by

$${\mathrm{n}}_{\min }={\displaystyle \frac{{\mathrm{Z}}_{1-\mathsf{\alpha }/2}^2·\mathrm{p}(1-\mathrm{p})}{{\mathrm{E}}^2}}$$

(10)

where Z_(1−α/2) is the quantile of the standard normal distribution (here, α = 0.05, corresponding to Z = 1.96), p is the expected pass rate (set to 0.95 based on historical data), and E is the acceptable error bound (set to 0.03 in this paper). The resulting sample size balances statistical power and acceptance cost, ensuring that the acceptance conclusion can represent overall batch quality with 95% confidence. In addition to checking individual items, acceptance also evaluates the macro-level balance of the batch, such as whether speaker gender ratio, age coverage, and geographic origin satisfy the corpus-construction objectives.

At the dynamic-optimization stage, a mechanism for specification feedback and version evolution is established. An immediate feedback channel is provided so that annotators can report cases not covered by the guidelines or cases that remain ambiguous in practice. The expert team responds quickly and issues rulings, dynamically updating the specification in the form of FAQs or supplementary notes. Strict version management is also introduced: every change is explicitly logged with its content, rationale, and effective date, ensuring traceability of annotation history and compatibility between versions. At the same time, typical error cases made by the speech-recognition model on the test set are analyzed in reverse to locate potential annotation defects or blind spots in the training data, enabling targeted revision of the specification. This closed-loop iteration among data, model, and specification gives the annotation system sustained evolutionary capability and provides long-term support for iterative optimization of robust speech-recognition systems.

A qualitative comparison among the proposed specification, a representative industrial general specification (Microsoft Azure), and an academically deep annotation scheme (Discourse-CASS) are shown in Figure 3 across their main design, annotation, and deployment dimensions.

4. Case Study: Construction of the Hainan Lingao Dialect Corpus

To validate the proposed method in practice, this paper applied it throughout the construction of the Hainan Lingao Dialect Speech Corpus. The Lingao dialect (Ong Be), spoken in northern Hainan Island by about 600,000 people, belongs to the Kra-Dai language family. It is spoken but lacks a standardized writing system, exhibits substantial internal regional accent variation, and suffers from an extreme scarcity of digital text resources. It is therefore a typical endangered low-resource dialect. During corpus construction, we strictly followed the transcription specifications described above. The text materials were drawn mainly from authoritative works such as Research on the Lingao Language and Collected Materials of Lingao Utterances, with additional material taken from local radio news programs. The text corpus covers seven domains: Lingao place names, common words, common sentences, daily dialogues, Lingao news, novels/stories, and distinctive slang. Domain proportions were set according to the natural distribution of everyday spoken language (see Figure 4a), and some phonologically important but low-frequency words were manually added to alleviate data sparsity.

Speaker selection followed three principles: dialect authenticity, age stratification, and geographic coverage. Sixteen speakers were recruited from eight core townships in Lingao County, Diaolou, Nanbao, Duowen, Heshe, Dongying, Lincheng, Bolian, and Xinying, with ages ranging from 18 to 52 years and a perfectly balanced gender distribution (8 male and 8 female). To support subsequent sociolinguistic analysis and speaker-adaptive modeling, a fine-grained metadata management system was established. Each speaker was assigned a unique structured ID encoding key sociodemographic information. The coding rule was “region code + name code + gender code + age-group code + file serial number.” The speaker-ID structure is illustrated in Figure 4b. This design standardizes metadata management throughout the data lifecycle, from collection to downstream use, and provides a solid basis for later analysis of the effects of age grading and geographic variation on speech recognition.

During speech collection and front-end preprocessing, we followed two principles: faithful reproduction of real acoustic scenes and prioritization of end-to-end data diversity. Recording devices included professional microphones, portable recorders, and smartphones. Recording environments covered diverse real-world scenes, including quiet indoor rooms, offices, household courtyards, and outdoor streets, and the metadata for each audio segment explicitly recorded noise type, estimated SNR, and other acoustic-environment attributes. Audio was standardized to mono, 16-bit quantization, 16 kHz sampling, and WAV format to satisfy the input requirements of mainstream end-to-end models. Each speaker completed both read speech and semi-spontaneous recordings based on the corpus pool. After collection, all raw audio was screened in Cool Edit Pro for SNR and background noise. Segments with SNR below 15 dB or irreparable reverberation were re-recorded or discarded. The core processing step was semantic-level intelligent segmentation based on voice activity detection (VAD). Using the pretrained Silero VAD model together with a self-developed Python 3.10script (Figure 5), long recordings were first processed to obtain timestamps for all speech-active intervals. The corresponding source text was then loaded and split into candidate sentences by punctuation and spaces. The timestamps were traversed one by one to crop the audio into semantically complete sentence segments with clean boundaries. These segments were exported under filenames in the form ‘speakerID_startTime_endTime.wav’, thereby establishing a one-to-one mapping between audio files and transcripts. This method reduced the average duration of each utterance to 6–12 s and expanded the corpus from dozens of long recordings to 42,000 utterances.

During annotation and quality control, we used the self-developed Dialect Annotation Management Platform shown in Figure 6 to operationalize the three-level quality-control mechanism across the entire workflow. All annotation tasks were organized around three levels: lexical, sentence-level, and pragmatic-behavior annotation. The platform adopts an RBAC permission model and defines four roles, annotator, reviewer, acceptor, and administrator, fully supporting the business loop from batch management to annotation control, review, and acceptance. The annotation-control module provides core functions such as corpus assignment, transcription editing, pragmatic-behavior tagging, and audio playback for verification. Following the lexical transcription guidelines, annotators transcribe Lingao speech fragments into agreed surrogate Chinese-character sequences and attach structured tags for special phenomena such as backchannels, instruction responses, environmental noise, and unintelligible speech. The review module allows reviewers to inspect annotated data item by item or in batches; items that fail review can be returned with one click to the annotator task pool together with rejection comments. The acceptance module is handled by linguistic experts, who perform sampled final review at the batch level from macro perspectives such as corpus–distribution balance, consistency of tag usage, and transcription accuracy. The platform standardizes, visualizes, and renders traceable what would otherwise be a complex annotation-management process, substantially reducing annotation cost.

5. Experimental Results and Analysis

5.1. Experimental Setup

To systematically evaluate the effectiveness of the multi-level annotation specification proposed in this paper, we constructed five training sets, one validation set, and four test sets, from the original Hainan Lingao dialect recordings. All datasets were drawn from the same source material and involved 16 speakers, evenly split by gender (8 male, 8 female), aged 18–52, from the eight core townships of Diaolou, Nanbao, Duowen, Heshe, Dongying, Lincheng, Bolian, and Xinying. Recording devices included professional microphones, portable recorders, and smartphones, and recording environments covered diverse real-world scenes, including quiet indoor spaces, offices, courtyards, and outdoor streets, so as to reflect practical acoustic variability. Audio was uniformly stored as mono 16-bit, 16 kHz WAV files. Before dataset partitioning, segments with SNR below 15 dB or with irreparable reverberation were re-recorded or discarded, and semantic-level segmentation was performed using the pretrained Silero VAD model together with a custom Python script to generate sentence-complete utterances. From the collected material, 80 h of speech were used for training, 10 h for validation, and 10 h for testing, with balanced distributions of speakers and scenes. All five training sets contained 80 h of speech. All models used the same Conformer-based end-to-end architecture with a 12-layer encoder, 4 attention heads, and 256 hidden units, jointly optimized with CTC/Attention. The batch size was 32, the initial learning rate was 0.002, the warmup schedule used 25,000 steps, and training lasted for 80 epochs with early stopping and model selection on the validation set. Character error rate (CER) and sentence error rate (SER) were used as evaluation metrics, and all results were averaged over three independent runs. Statistical significance was assessed by paired bootstrap resampling at the utterance level. For each test set, the baseline and full-specification systems were compared on the same utterances using 10,000 paired resamples of the recognition outputs, and CER/SER differences were considered statistically significant at p < 0.01.

The five training sets were defined as follows. The full-specification set strictly followed the three annotation levels proposed in this paper: lexical, sentence-level, and pragmatic-behavior annotation. The traditional-specification set contained only plain transcription, no special tags, and fixed 3 s segmentation, thereby simulating the current mainstream baseline. The three ablation sets respectively removed pragmatic-behavior tags (Ablation A), removed noise tags (Ablation B), and replaced semantic segmentation with fixed-duration segmentation (Ablation C). The test data totaled 10 h and covered four scenarios, each accounting for about 2.5 h: a clean test set (quiet indoor environments, SNR > 25 dB), a noisy test set (traffic, fan, and crowd noise added at 15-5 dB SNR), a dialogue test set (including natural overlap, backchannels, and interruptions), and a dialect-variation test set (covering accents from eight townships and three age groups). Each test set contained about 1250 utterances held out from training, ensuring an objective evaluation under matched speaker and scene distributions. All annotation followed the three-level quality-control mechanism described in Section 3.4, and the inter-annotator agreement (IAA) of the full-specification set exceeded 0.95, ensuring consistent data quality.

5.2. Comparative Results

The comparative experiments assessed the performance difference between the full annotation specification and the traditional baseline; the results are summarized in Figure 7a and Table 5. On the clean test set, the full specification reduced CER from 8.7% to 7.9%, corresponding to an absolute reduction of 0.8 percentage points and a relative reduction of 9.2%. On the noisy test set, CER decreased from 24.3% to 18.5%, corresponding to an absolute reduction of 5.8 percentage points and a relative reduction of 23.7%. On the dialogue test set, CER decreased from 19.5% to 15.2%, corresponding to an absolute reduction of 4.3 percentage points and a relative reduction of 22.1%. On the dialect-variation test set, CER decreased from 15.2% to 13.1%, corresponding to an absolute reduction of 2.1 percentage points and a relative reduction of 13.8%.

SER showed the same overall pattern. Under the full specification, SER on the clean, noisy, dialogue, and dialect-variation test sets was 15.2%, 32.1%, 27.8%, and 24.5%, respectively, compared with 17.5%, 39.6%, 34.2%, and 28.3% under the traditional specification. The average relative SER reduction was 16.4%. Paired bootstrap testing with 10,000 resamples showed that all baseline-to-full-specification differences were statistically significant (p < 0.01).

5.3. Ablation Study

To examine the contribution of individual annotation components, we conducted a systematic ablation study; the results are summarized in Figure 7b and Table 6. Removing pragmatic-behavior tags (Ablation A) increased CER on the dialogue test set from 15.2% to 16.8%, an absolute increase of 1.6 percentage points. Removing noise tags (Ablation B) increased CER on the noisy test set from 18.5% to 20.5%, an absolute increase of 2.0 percentage points. Replacing semantic segmentation with fixed-duration segmentation (Ablation C) increased CER from 7.9% to 8.3% on the clean test set and from 18.5% to 19.2% on the noisy test set, corresponding to absolute increases of 0.4 and 0.7 percentage points, respectively.

Across the ablation conditions, performance degradation was observed in all cases, with the largest change for Ablation A occurring on the dialogue test set, the largest change for Ablation B occurring on the noisy test set, and the largest change for Ablation C occurring on the noisy test set.

5.4. Analysis of Dialect-Feature Annotation

To quantify the contribution of dialect-feature annotation, we conducted an additional ablation study in which dialect-feature annotation was removed from the full specification; the results are summarized in Figure 7c. After removing dialect features, CER on the dialect-variation test set increased from 13.1% to 14.2%, an absolute increase of 1.1 percentage points. Stratified analysis showed that, for speakers older than 45 years and for samples from Diaolou and Xinying, the error-rate increase reached up to 2.8%, whereas for younger speakers (<25 years) and county-seat speakers from Lincheng, the increase was 0.4%.

At the lexical level, the adoption of a unified surrogate-character convention increased inter-annotator agreement from 0.89 to 0.96.

Table 5. Main CER and SER results across the baseline and full specification.

Scenario	CER (Base)	CER (Full)	ΔCER	SER (Base)	SER (Full)	ΔSER
Clean	8.7	7.9	−0.8	17.5	15.2	−2.3
Noisy	24.3	18.5	−5.8	39.6	32.1	−7.5
Dialogue	19.5	15.2	−4.3	34.2	27.8	−6.4
Dialect variation	15.2	13.1	−2.1	28.3	24.5	−3.8

Table 5. Main CER and SER results across the baseline and full specification.

Scenario	CER (Base)	CER (Full)	ΔCER	SER (Base)	SER (Full)	ΔSER
Clean	8.7	7.9	−0.8	17.5	15.2	−2.3
Noisy	24.3	18.5	−5.8	39.6	32.1	−7.5
Dialogue	19.5	15.2	−4.3	34.2	27.8	−6.4
Dialect variation	15.2	13.1	−2.1	28.3	24.5	−3.8

Table 6. Ablation results relative to the full specification.

Setting	Scenario	CER	Δ vs. Full
Ablation A	Dialogue	16.8	+1.6
Ablation B	Noisy	20.5	+2.0
Ablation C	Clean	8.3	+0.4
Ablation C	Noisy	19.2	+0.7
No dialect features	Dialect variation	14.2	+1.1

Table 6. Ablation results relative to the full specification.

Setting	Scenario	CER	Δ vs. Full
Ablation A	Dialogue	16.8	+1.6
Ablation B	Noisy	20.5	+2.0
Ablation C	Clean	8.3	+0.4
Ablation C	Noisy	19.2	+0.7
No dialect features	Dialect variation	14.2	+1.1

5.5. Discussion

Overall, the experimental results support four main conclusions. First, the systematic annotation method proposed in this paper substantially improves the robustness of speech-recognition models in complex environments, achieving relative performance gains exceeding 20% in noisy and dialogue scenarios, with all improvements statistically significant. This gain arises because traditional specifications provide only sparse text-level supervision, whereas the proposed specification injects rich structured prior knowledge through multi-level tags, enabling more precise disentanglement of acoustic interference and linguistic content. The larger gains in noisy conditions are consistent with the use of explicit noise-event and unintelligible-speech tags. The larger gains in dialogue conditions are consistent with the addition of pragmatic-behavior annotation for backchannels, interruptions, and interactional responses. Second, each component of the specification contributes independently to final performance. Pragmatic-behavior tags, noise tags, fine-grained semantic segmentation, and dialect-feature annotation respectively target dialogue interaction, acoustic interference, semantic boundaries, and accent variation, together forming a comprehensive data-enhancement strategy. Third, the experimental results are highly consistent with the annotation-error theory expressed in Equations (1) and (2). Equation (1) indicates that minimizing expected annotation error is key to improving system robustness, and the ablation results show that removing any component increases annotation error and worsens the generalization bound. Equation (2) decomposes annotation noise into systematic bias and random error; the targeted analysis of dialect-feature annotation shows that unified surrogate-character conventions reduce random error, while speaker metadata reduces systematic bias by providing a more accurate conditional distribution. Fourth, the present evidence supports the effectiveness of the proposed annotation framework within the Hainan Lingao corpus. It also suggests potential transferability to similar low-resource dialect settings. However, broader validation across additional dialect corpora and public benchmarks is still required. The current experiments nevertheless demonstrate its engineering feasibility for corpus construction under the conditions examined in this study.

This study has several limitations. First, the empirical validation was conducted on a single dialect corpus with 16 speakers from one geographical area, which limits the external generalizability of the findings. Second, the current experiments do not include evaluation on public multilingual or dialect ASR benchmarks. Third, the comparison is limited to a conventional plain-transcription baseline and does not yet include stronger alternative annotation or training baselines. Fourth, although the framework is general in design, its transferability to additional dialects and languages still requires direct validation through multiple case studies.

6. Conclusions

This paper systematically investigated annotation specifications for robust speech recognition, especially in complex Chinese dialect scenarios. A multi-level annotation framework grounded in linguistic principles and guided by machine readability and extensibility was proposed, covering lexical, sentence-level, and pragmatic-behavior levels and providing fine-grained handling rules for dialect variation, environmental noise, and overlapping speech. A three-stage quality-assurance mechanism consisting of initial annotation, review, and acceptance, together with a dynamic optimization process, ensured executability and continued evolution of the specification. Using the Hainan Lingao dialect corpus as a case study, this paper demonstrated the complete workflow from specification design to quality control. Comparative and ablation experiments showed that models trained with the proposed specification achieved relative CER reductions exceeding 20% in noisy and dialogue scenarios. They also showed that pragmatic-behavior tags, noise tags, fine-grained segmentation, and dialect-feature annotation each made significant contributions. Together, these results confirm both the effectiveness and the systematic nature of the method. Methodologically, the proposed specification bridges the gap between industrial general specifications and academically deep annotation schemes. Compared with industrial specifications such as Microsoft Azure, the framework strengthens the modeling of dialogue interaction through systematic pragmatic-behavior tags. Compared with academic systems such as Discourse-CASS, it retains key linguistic dimensions while improving engineering practicality through a streamlined hierarchy, quantitative decision criteria, and a three-stage quality-control mechanism. Under the case study conditions examined here, this design also demonstrates practical feasibility for engineering use. The proposed framework provides a practically validated annotation strategy for building more robust speech-recognition systems in the Hainan Lingao dialect corpus and, more broadly, in similar low-resource dialect settings. Future work will explore integration with self-supervised learning and large language models. It will also examine how structured annotation can enhance model cognition and extend the framework to the digital preservation of more endangered languages and dialects. In this way, this work may contribute to the protection and transmission of linguistic and cultural diversity.