Evidence-Aware Human-in-the-Loop LLM Review for Requirements-to-Planning Decisions

Abstract

Large language models can produce fluent requirements refinements and planning artifacts while still leaving information unresolved for implementation, testing, or planning commitment. This paper presents ReqPlan-Eval, an evidence-aware human-in-the-loop architecture that connects NFR disagreement, weak-word cues, planning-relevant ambiguity, and role-specialized hypotheses to inspectable planning-support records and configurable review routes. The empirical study evaluates the principal mechanisms and role-based routing signals on separate datasets; it does not constitute an end-to-end evaluation of the complete pipeline on a common set of requirements. A held-out 500-requirement NFR diagnostic achieved exact-match accuracy of 0.716, micro F1 of 0.702, and quality accuracy of 0.874. Pattern-aware arbitration increased weak-word specificity from 0.572 to 0.676 and review precision from 0.678 to 0.724, while component/goal gating increased ambiguity specificity from 0.380 to 0.908 and F1 from 0.748 to 0.855; both mechanisms lost recall. In Experiment 3, four role-specialized outputs were compared with a model-seeded reference reviewed and adjudicated by three human reviewers. On the 20 public stories, role-level exact agreement for validation_needed was 0.825, and a two-or-more-vote policy reviewed 12 stories and captured 12 of 14 reference positives without routing any of the six reference negatives. Open-ended planning artifacts showed very low exact normalized-item overlap with the reference for tasks, acceptance criteria, and test ideas, so no semantic-agreement claim is made for those artifacts. Seven external participants provided initial face-validity evidence for selective review and human control. The results support the evaluated component mechanisms and selective review routing under the frozen configurations, but they do not establish end-to-end workflow effectiveness, model invariance, better planning decisions, or industrial effectiveness.

1. Introduction

LLM-assisted software engineering increasingly places generated refinements, tasks, test ideas, and other planning artifacts beside human-authored requirements. The difficulty is not limited to obviously incorrect text. A requirement can look complete while leaving a quality target, business rule, dependency, acceptance condition, or stakeholder decision unresolved.

Requirements engineering already provides useful foundations for addressing parts of this problem. NFR research studies quality concerns and softgoals [1,2]; automated NFR identification has been examined using several approaches [3,4,5,6,7,8]; and requirements-quality research covers ambiguity, incompleteness, smells, weak wording, and user-story quality [9,10,11,12,13,14,15,16,17,18]. LLMs extend this setting by generating candidate refinements and planning artifacts, but their outputs remain sensitive to prompting, model choice, and the organization of human oversight [19,20,21,22,23,24,25]. For planning, the practical question is therefore not only whether an artifact can be generated, but whether unresolved evidence is made visible before a commitment is made.

ReqPlan-Eval addresses this question as a human-controlled review-routing architecture. NFR disagreement, weak-word arbitration, and ambiguity gating provide review evidence. Four role-specialized LLM outputs provide planning perspectives. These elements are represented in an inspectable planning-support record, and a project-selected policy determines whether additional human review is requested. The route is not a planning decision, and the reviewer may change any model-derived field.

The scope is deliberately bounded. The paper does not introduce a new NFR taxonomy, a foundation model, or a general conflict detector. Its two new decision mechanisms are pattern-aware weak-word arbitration and component/goal ambiguity gating. NFR classification and role-specialized planning outputs are evidence sources within the wider architecture.

Consider the user story:

As a customer, I want to upload documents so that my loan application can be processed quickly.

The function is clear, but the planning conditions are not. The story does not state which documents are allowed, which formats and size limits apply, how files are validated and stored, who may access them, what must be logged, or what “quickly” means. ReqPlan-Eval is intended to expose such open points and route the story for focused clarification when they affect planning, testing, dependencies, or stakeholder validation.

The study addresses four research questions:

• RQ1: How accurately can LLM-supported configurations identify NFR labels, weak-word quality defects, and planning-relevant user-story ambiguity?
• RQ2: Do pattern-aware arbitration and component/goal gating improve the reliability of the corresponding LLM-supported judgments?
• RQ3: How do alternative review policies trade review effort against problem capture and missed problems?
• RQ4: To what extent do role-specialized LLM outputs align with a human-adjudicated planning-support reference on categorical review signals, and what surface-level similarity is observed for generated planning artifacts?

The empirical evaluation is intentionally component-wise. Experiment 1 evaluates NFR classification and selective validation, including a held-out taxonomy-aware diagnostic. Experiment 2 evaluates weak-word arbitration and user-story ambiguity gating. Experiment 3 evaluates role-specialized planning-support and review-signal alignment against a 30-story, model-seeded reference reviewed and adjudicated by three human reviewers. A separate seven-participant evaluation provides face-validity evidence for the framework presentation. The study does not evaluate the complete ReqPlan-Eval pipeline end to end on one common set of requirements, and it does not measure final planning decisions or workflow productivity.

The paper makes the following three contributions:

• A coherent evidence-aware architecture that connects analysis signals and role-specialized planning hypotheses to inspectable records and configurable human review routes;
• Two planning-aware decision mechanisms, pattern-aware weak-word arbitration and component/goal ambiguity gating, with explicit review trade-offs;
• A multi-part empirical evaluation of the principal component mechanisms and role-based routing signals, supported by a human-adjudicated reference, external-participant face-validity evidence, and a reproducible replication package.

The remainder of the paper presents related work, the framework, the empirical design and results, the implications and evidence boundaries, and threats to validity.

2. Related Work

ReqPlan-Eval draws on three lines of work: LLM support for requirements engineering, requirements-quality analysis, and selective human review.

2.1. LLM Support and Requirements Quality

LLMs have been studied for elicitation, classification, summarization, refinement, ambiguity analysis, formal requirements, and broader software-engineering tasks [19,20,21,22,23,24,25]. These capabilities are useful, but they do not remove concerns about reliability, reproducibility, prompt sensitivity, and human oversight. Practitioner evidence similarly places AI mainly in a supporting rather than fully autonomous role [26]. Quality-assurance work on LLM-generated software also distinguishes functional success from broader quality [27].

Requirements engineering provides established concepts for the evidence used in this paper. NFR and quality models define quality concerns [1,2,28], while automated NFR classification has been studied with rule-based, statistical, and learning-based approaches [3,4,5,6,7,8]. Requirements-quality research addresses ambiguity, incompleteness, vague wording, smells, and defects [9,10,11,12,13,14]. User-story research further shows that syntactic completeness does not guarantee planning readiness [15,16,17,18]. ReqPlan-Eval uses these signals as reasons for inspection when they affect planning rather than assuming that each signal is an independent final decision.

2.2. Selective Human Review

Explainable AI and human–AI interaction research emphasizes that users should be able to inspect, challenge, and correct automated recommendations [29,30]. Selective-review methods similarly direct limited human effort toward uncertain or informative cases [31,32,33,34]. ReqPlan-Eval applies this principle to requirements-to-planning work by linking a route to inspectable evidence and a focused validation question rather than to a confidence value alone.

2.3. Positioning of ReqPlan-Eval

Existing approaches provide useful labels, quality signals, selective-review strategies, and generated artifacts. ReqPlan-Eval differs in organizing NFR disagreement, weak-word context, planning-relevant ambiguity, and role-specialized planning hypotheses around one inspectable review-routing record. Pattern-aware weak-word arbitration and component/goal ambiguity gating are the two mechanisms introduced and evaluated directly.

3. ReqPlan-Eval Framework

ReqPlan-Eval creates an inspectable planning-support record from a natural-language requirement or user story. Every LLM output remains a hypothesis, explanations are tied to observable cues and planning consequences, review routing is explicit, and the routing policy can change with project risk and review capacity. Pattern-aware weak-word arbitration and component/goal ambiguity gating are the two new mechanisms. NFR analysis and four role-specialized LLM perspectives provide additional evidence within the same architecture.

Figure 1 shows the conceptual workflow. The original text is sent independently to the derived-evidence layer and to the four role-specialized perspectives. Their signals and hypotheses are then integrated in the planning-support record. The selected policy routes the record without approving it. Final authority remains with the human reviewer, who may revise any model-derived field.

3.1. Operational Pipeline

Algorithm 1 gives the procedural form of the workflow in Figure 1. Each stage adds either an analysis signal, planning artifact, inspectable evidence, or routing decision. The review status for layer $\mathit{k}$ and record $\mathit{i}$ is represented as

$${\mathit{H}}_{\mathit{k},\mathit{i}}\in \{\mathrm{No}\mathrm{Review},\mathrm{Optional}\mathrm{Review},\mathrm{Required}\mathrm{Review}\}.$$

Algorithm 1 ReqPlan-Eval requirements-to-planning pipeline

Require: A natural-language requirement or user story ${\mathit{r}}_{\mathit{i}}$

Ensure: A planning-support record and review-routing status

1: Create an initial record ${\mathit{R}}_{\mathit{i}}$ containing the original text and available metadata.   2: Derive NFR boundary or label signals, weak-word cues, and planning-relevant ambiguity cues from the original text.   3: Independently generate role-specialized planning hypotheses from the original text, including tasks, acceptance criteria, test ideas, and dependencies.   4: Integrate the derived signals and role-specialized hypotheses in ${\mathit{R}}_{\mathit{i}}$, together with evidence, rationale, assumptions, alternative labels, and role disagreement.   5: Assign operational risk and uncertainty levels and identify unresolved high-impact issues.   6: Derive validation need, recommended action, and review priority under the selected policy.   7: if human review is required then   8:      Route ${\mathit{R}}_{\mathit{i}}$ with the evidence and focused validation questions.   9:      Record corrections, clarified constraints, accepted or rejected assumptions, and the final human decision. 10: else 11:       Mark ${\mathit{R}}_{\mathit{i}}$ as not requiring additional review under the selected policy; this status does not constitute final planning approval. 12: end if 13: return the final planning-support record and review-routing status.

3.2. Analytical Layers

The evidence sources serve different purposes. NFR analysis exposes functional and quality boundaries, alternative labels, disagreement, and overlap-prone categories. Weak-word analysis records the matched family, context cue, and protected patterns. Ambiguity analysis records missing or weak actor, action, object, or goal components when they affect planning. Role-specialized outputs contribute tasks, acceptance criteria, dependencies, assumptions, risks, uncertainty, and clarification questions. These signals are combined only for review support; the human reviewer remains responsible for any final change to the planning record.

3.3. Planning-Support Record

The planning-support record may include a refined requirement, tasks, acceptance criteria, test ideas, dependencies, assumptions, clarification questions, quality issues, risk and uncertainty levels, validation need, high-risk unresolved status, and a recommended action. For the document-upload example, the record would expose the missing file constraints, privacy and retention rules, access conditions, measurable processing target, and the focused questions that must be resolved before planning commitment.

The categorical fields are fixed routing categories rather than calibrated probabilities. They support review and do not replace the final project judgment.

Table 1. Operational interpretation of the main planning-support fields.

Field	Operational Interpretation
risk_level	Low: Ordinary refinement is sufficient. Medium: A planning-relevant dependency, quality expectation, acceptance condition, or assumption should be checked. High: An unresolved issue may materially affect scope, architecture, testing, security, compliance, or feasibility.
uncertainty_level	Low: Intent, scope, behavior, and main acceptance expectations are sufficiently clear. Moderate: Information is missing, but the likely interpretation remains constrained. High: Key intent, object, rule, constraint, dependency, quality target, or acceptance condition is missing or ambiguous.
validation_needed	Yes when the model would otherwise have to assume a planning-relevant fact; otherwise No. Normal backlog refinement alone does not imply Yes.
high_risk_unresolved	Yes only when at least one high-impact issue remains open and should require human review before planning acceptance under the selected policy.
recommended_action	Accept for planning, Refine before planning, Clarify with stakeholder, Decompose, or Defer, ordered from least to most restrictive for action-distance analysis.

gives the interpretation used in Experiment 3.

A High label does not make the final decision. It asks the reviewer to inspect the linked evidence, assumptions, and questions before planning commitment.

3.4. Role-Specialized Perspectives and Aggregation

Experiment 3 uses four LLM role-agents: requirements engineering, agile planning, QA/acceptance criteria, and architecture/technical risk. These are model perspectives, not human reviewers or a simulation of a full project team. The human reference was produced separately through two-reviewer consensus, third-reviewer review, and final adjudication.

The role-agents provide different evidence views of the same story. Agreement provides converging evidence for a field, while disagreement becomes a review signal. Aggregation is used mainly for routing rather than for creating a universal majority label. For example, the story-level policy routes a story when at least two role-agents mark validation_needed as Yes. A large gap in recommended_action, such as Accept for planning versus Defer, is also treated as a review signal. The human reviewer remains responsible for the final decision.

3.5. Review Policies and Scope

A balanced policy routes fewer items when review capacity is limited. A high-safety policy routes more items when the cost of a missed defect, ambiguity, or planning risk is high. The experiments report review rate, problem capture, missed-problem rate, over-review, and review precision so that the trade-off remains visible rather than being reduced to one “best” threshold.

Three terms are kept distinct throughout the paper. Review routing is the policy output that determines whether additional human inspection is requested. Validation is the human act of checking the linked evidence or answering a focused question. The planning decision is the human outcome, such as accepting, refining, clarifying, decomposing, or deferring the item. A routing status or a validation_needed signal is therefore not itself a planning decision.

ReqPlan-Eval is limited to requirements-to-planning review support. It does not claim formal verification, complete traceability, automated effort estimation, or final sprint commitment. Cross-requirement conflict detection is a relevant future extension, but it is not included as an evaluated layer in this study because the available datasets do not provide a suitable confirmed-conflict reference.

4. Materials and Methods

The evaluation contains three empirical studies rather than one end-to-end benchmark. No public dataset used in this study provides the reference fields required by every layer. Experiments 1 and 2 evaluate focused analysis and routing mechanisms, while Experiment 3 examines role-specialized outputs and review signals against a model-seeded, human-adjudicated planning reference. The complete ReqPlan-Eval pipeline is therefore not evaluated on a common requirement set, and the evidence is interpreted for the component or routing layer that each experiment actually tests.

4.1. Evaluation Overview and Datasets

The evaluation uses separate datasets because no dataset in the study contains all reference fields required by the complete framework.

Table 2. ReqPlan-Eval experiments, datasets, and evaluation use.

Experiment	Input Unit	Records	Reference Signal	Evaluation Use
1a and 1b	Requirement	622	Functional/quality status and NFR labels	Comparative NFR classification and selective validation; not independent held-out validation.
1c	Requirement	500 held-out; 122 calibration	Functional/quality status and NFR labels	Held-out taxonomy-aware diagnostic.
2a	Requirement	500	Defect/non-defect label	Weak-word defect detection, arbitration, and routing.
2b	User story	500	Ambiguous/non-ambiguous label and ambiguity evidence	Component/goal gating and review routing.
3	User story	30 stories; 120 role outputs	Model-seeded planning records reviewed and adjudicated by three human reviewers	Role-specialized planning fields, review-signal alignment, and routing policies.

summarizes the units, sample sizes, reference signals, and purpose of each experiment. Experiment 1 uses the NICE relabeled PROMISE NFR dataset [35]; Experiment 2a uses the QuRE industrial requirements-quality dataset [36,37]; and Experiments 2b and 3 use stories from the Dalpiaz collection [17].

Experiment 3 contains two deliberately different subsets. Twenty stories from the public Dalpiaz collection form the planning-readiness stress set. Ten simple, single-capability stories assembled by the study team form a calibration set used before freezing to check over-routing. The calibration stories are reported separately and are not treated as independent held-out evidence. Both subsets were fixed before the human reference was created.

4.2. Human Reference for Experiment 3

The Experiment 3 reference is human-adjudicated but not an independently authored human-only benchmark. Candidate planning records were prepared from the stored role-agent outputs and an optional LLM synthesis step. Two human reviewers jointly reviewed and corrected those candidate records to produce the initial consensus. Because the initial reference was produced jointly, the study does not report a pre-consensus inter-rater reliability statistic. The first review covered requirements and planning judgment. A third human reviewer examined all 30 records with particular attention to QA and testability. Third-reviewer comments were recorded for 26 records. After adjudication, at least one structured field changed in 20 records. The most frequently changed fields were uncertainty_level (13 records), human_review_priority (8), risk_level (7), secondary_risks (5), and quality_issues (4); validation_needed changed in two records and recommended_action in one. This procedure provides a traceable human-corrected reference, but the model-seeded starting point may increase apparent alignment with the role-agent outputs. Experiment 3 is therefore interpreted as an evaluation of alignment with a human-adjudicated planning reference, not as a comparison with an independent human-only ground truth. During prediction, the role-agents received only the story identifier, dataset split, and original story text. Reference fields, including risk, uncertainty, validation need, recommended action, tasks, acceptance criteria, dependencies, assumptions, and clarification questions, were withheld. The stored role-agent outputs were not regenerated after adjudication.

The reference construction produced an initial two-reviewer consensus and a final three-reviewer adjudicated reference. A sensitivity analysis compares the categorical fields that changed during the later adjudication to measure how that adjudication altered role-reference agreement. The experiment did not record the candidate state immediately before the first two reviewers’ corrections or separate pre-consensus records for those reviewers, so the full contribution of the original model seed cannot be isolated. The sensitivity analysis uses the previously generated role outputs and fixed reference states; it does not regenerate predictions or alter reference labels.

Reviewer Roles and Participant Provenance

As shown in

Table 3. Anonymized participant roles.

Identifier	Expertise Area	Study Role
R1 and R2	Software engineering practitioners with requirements and planning experience	Joint correction and initial consensus for the model-seeded reference.
R3	Software engineering practitioner with experience in QA and testing of agile projects	Review of all 30 records with emphasis on QA and testability, followed by adjudication input.
E1	Requirements engineering researcher	Framework evaluation panel.
E2	Senior requirements analyst/business analyst	Framework evaluation panel.
E3	Agile product owner/software project manager	Framework evaluation panel.
E4	Software QA and acceptance-testing specialist	Framework evaluation panel.
E5	Software architect/technical-risk specialist	Framework evaluation panel.
E6	Human-AI interaction/explainable AI researcher	Framework evaluation panel.
E7	Empirical software-engineering reviewer	Framework evaluation panel.

, the three reference reviewers and the seven external panel members took part in two different study activities. The reference reviewers were software engineering practitioners. The first two jointly corrected the model-seeded candidate records from requirements and planning perspectives. The third reviewer examined all 30 records and had specific experience in quality assurance and testing for agile projects. All three reference reviewers saw the model-generated starting records. The seven panel members were external to the author team. They evaluated the framework presentation and did not create or modify any reported role-agent output.

4.3. Freezing, LLM Configuration, and Reproducibility

The experiments use pretrained LLMs; no model parameters were updated. Configuration therefore refers to prompts, output schemas, deterministic gates, arbitration rules, and review policies. Where a separate development or calibration set was defined, it was used before the final configuration was frozen, and final held-out errors were not used to revise the frozen decision rules. Experiment 1a has a different boundary: C2, C4, and C4TAX_R were developed within the same 622-record study context. Its results are therefore treated as a frozen comparative component evaluation rather than independent held-out validation. Experiment 1c provides the separate held-out diagnostic for the NFR layer.

Table 4. Exact submitted API identifiers and LLM configurations used for the reported outputs.

Study	Exact API Identifier Submitted	Recorded Run Date	Decoding/Output Settings	Retries and Completion
1a and 1b	gpt-5.2	13 May 2026	Temperature 0.0; maximum 1200 output tokens; full 622-record set	No automatic API retry loop; complete outputs for C2, C4, and C4TAX_R.
1c	gpt-5-mini	17 May 2026	Temperature not sent; API maximum-token value not set; seed 42 for the split	At most two JSON retries, then deterministic fallback; 499 records needed no retry, one needed one retry, and no fallback was used.
2a	gpt-5	16 May 2026	Temperature and API maximum-token value not sent; fixed 500-record set	No repeated API call; tolerant JSON extraction; 500 complete predictions.
2b	gpt-5-mini	16 May 2026	Temperature and API maximum-token value not sent; fixed 500-record set	No repeated API call; tolerant JSON repair/extraction; 500 complete predictions.
3	gpt-5.2 for all four roles	15 May 2026	Temperature 0.2; maximum 4500 output tokens; seed 42 for sampling before the 30-story set was fixed	No automatic API retry; structured-output validation and error logging; 120 complete role outputs. The optional LLM synthesis was used only in candidate-reference preparation, not as a fifth evaluated role.

reports the exact API identifier strings and execution settings used in the reported runs. All LLM calls used the OpenAI Responses API. A seed affected only record sampling where shown; it was not treated as a guarantee of deterministic model generation. Prediction generation was separated from metric computation.

The model identifiers differ because each study was developed and frozen for its own component task rather than as part of a single-model benchmark. Within each before/after mechanism comparison, model choice is held fixed: Experiment 2a applies arbitration to the same stored gpt-5 judgments used for its LLM-only baseline, and Experiment 2b applies gating to the same stored gpt-5-mini judgments used for its LLM-only baseline. Experiment 1a compares its frozen prompt configurations under gpt-5.2; Experiment 1c is a separate held-out diagnostic under gpt-5-mini; and all four Experiment 3 roles use gpt-5.2. The experiments were not rerun under a single common model. The conclusions are therefore conditional on the reported frozen configurations, and cross-model invariance is not claimed.

Structured outputs were stored before aggregate analysis, and all reported metrics were recomputed from those stored outputs. The replication package includes processed inputs, frozen prompts and configurations, stored outputs, workbooks, verification tables, and scripts that recompute the reported metrics and tables.

4.4. Evaluation Measures and Statistical Robustness

For binary classification tasks, the paper reports accuracy, precision, recall, F1-score, and specificity using standard definitions [38,39]. For multi-label NFR classification, the evaluation reports exact-match accuracy, micro-averaged precision/recall/F1, macro F1, supported macro F1, and weighted F1.

Review-routing performance is central to ReqPlan-Eval. Let ${\mathit{V}}_{\mathit{i}}=1$ indicate that record $\mathit{i}$ is routed to human review, and let ${\mathit{P}}_{\mathit{i}}=1$ indicate that it is problematic according to the relevant reference. The main routing measures are:

$$\mathrm{Review}\mathrm{Rate}={\displaystyle \frac{{\sum }_{\mathit{i}}{\mathit{V}}_{\mathit{i}}}{\mathit{n}}},\mathrm{Problem}\mathrm{Capture}={\displaystyle \frac{{\sum }_{\mathit{i}}{\mathit{V}}_{\mathit{i}}{\mathit{P}}_{\mathit{i}}}{{\sum }_{\mathit{i}}{\mathit{P}}_{\mathit{i}}}},\mathrm{Review}\mathrm{Precision}={\displaystyle \frac{{\sum }_{\mathit{i}}{\mathit{V}}_{\mathit{i}}{\mathit{P}}_{\mathit{i}}}{{\sum }_{\mathit{i}}{\mathit{V}}_{\mathit{i}}}}.$$

(1)

The missed-problem rate is $1-\mathrm{Problem}\mathrm{Capture}$, and over-review is the share of non-problematic records that are nevertheless routed.

For Experiment 3 binary fields, exact agreement is supplemented with the reference class distribution, confusion counts, positive-class precision and recall, specificity, balanced accuracy, and macro F1. For the ordinal risk_level and uncertainty_level fields, the analysis reports exact agreement, linear weighted kappa, mean absolute ordinal distance, and agreement within one level [40]. This avoids relying on exact match alone.

Bootstrap confidence intervals are reported for the classification results and for role-level Experiment 3 agreement, where stories are resampled as clusters so that each selected story contributes all four role outputs. Story-level review-policy proportions use Wilson score intervals [41]. Wilson intervals remain non-degenerate when the observed numerator is zero or equals the denominator, unlike a percentile bootstrap applied to a small sample with no observed errors. Paired McNemar tests are used for selected before/after arbitration or gating comparisons and are treated as robustness checks rather than population-level generalization claims.

4.5. Explainability and Operational Measurement

ReqPlan-Eval is evaluated as review support rather than as a black-box classifier. The NFR layer exposes predicted labels, alternatives, configuration disagreement, and overlap-prone categories. The weak-word layer exposes the weak-word family, context cue, protected quality or safety pattern, and arbitration rationale. The ambiguity layer exposes missing or weak story components and the reason for applying or suppressing a flag. The planning-support layer exposes tasks, acceptance criteria, dependencies, assumptions, risks, uncertainty, recommended actions, and clarification questions.

Quality labels are operational proxies rather than absolute semantic truths. This is especially important for NFR categories that overlap in natural language. Disagreement and overlap-prone categories are therefore interpreted both as possible errors and as evidence for review. Explainability is assessed through the presence and use of inspectable evidence; the study does not claim a separate user-centered evaluation of explanation quality.

4.6. Experiment-Specific Setup

4.6.1. Experiment 1: NFR Classification and Selective Validation

Experiment 1a evaluates strict NFR classification on all 622 records using the frozen C2, C4, and C4TAX_R configurations. Because these configurations were developed within the same 622-record study context rather than under a separate held-out design, Experiment 1a is interpreted as a frozen comparative component evaluation, not as independent held-out validation. Experiment 1b evaluates disagreement, overlap-prone categories, and functional/quality-boundary disagreement as review signals. Experiment 1c applies the taxonomy-aware verifier to 500 held-out records after using a separate 122-record calibration subset. It provides the held-out generalization diagnostic for the NFR layer, while Experiment 1a provides comparative component evidence.

4.6.2. Experiment 2a: Weak-Word Requirement Defect Detection

Experiment 2a compares the LLM-only decision with the final pattern-arbitrated decision on 500 QuRE records. It reports classification and routing trade-offs rather than treating every weak expression as a defect.

4.6.3. Experiment 2b: User-Story Ambiguity Detection

Experiment 2b applies actor/action/object/goal extraction and the component/goal gate to 500 user stories. The gate is intended to suppress flags for short but planning-usable stories while retaining ambiguity that affects acceptance, dependencies, assumptions, or stakeholder validation.

4.6.4. Experiment 3: Role-Specialized Planning-Support and Review-Signal Alignment

Each of the 30 stories produced four LLM role outputs: requirements engineering, agile planning, QA/acceptance criteria, and architecture/technical risk. The analysis keeps the categorical review signals separate rather than collapsing them into one overall agreement score. Risk and uncertainty are evaluated as ordinal fields, validation_needed as a binary review signal, and recommended_action on the ordered action scale Accept for planning, Refine before planning, Clarify with stakeholder, Decompose, and Defer. The field high_risk_unresolved remains part of the operational record, but the reference contains only one positive story; it is therefore not treated as a separately supported detection outcome.

For list-valued planning fields, the analysis uses strict normalized-item Jaccard as a surface diagnostic [42]. Each stored list item is reduced to its textual item representation, stripped of surrounding whitespace, and lower-cased; structured items use the recorded title, criterion, description, or risk text when present. Jaccard is then computed over the resulting sets of complete items. The measure therefore captures exact normalized-item overlap, not token-level similarity or semantic equivalence.

For story $\mathit{i}$, let ${\mathit{v}}_{\mathit{i}}$ be the number of roles that set validation_needed to Yes. Let ${\mathit{h}}_{\mathit{i}}=1$ when any role sets high_risk_unresolved to Yes or risk_level to High. Let ${\mathit{a}}_{\mathit{i}}$ count roles that recommend any action other than Accept for planning, and let ${\mathit{e}}_{\mathit{i}}$ count roles that recommend Clarify with stakeholder, Decompose, or Defer.

Table 5. Deterministic review-policy definitions used in Experiment 3.

Policy	Exact Deterministic Condition	Operating Interpretation
Any validation-needed vote	${\mathit{v}}_{\mathit{i}}\ge 1$	Maximum validation sensitivity.
Two-or-more validation-needed votes	${\mathit{v}}_{\mathit{i}}\ge 2$	Balanced validation routing.
Strong validation-needed vote	${\mathit{v}}_{\mathit{i}}\ge 3$	More selective validation routing.
Any high-risk safety signal	${\mathit{h}}_{\mathit{i}}=1$	High-impact safety safeguard.
Two-or-more action-review votes	${\mathit{a}}_{\mathit{i}}\ge 2$	Broad action-based routing.
Hybrid safety rule	$({\mathit{v}}_{\mathit{i}}\ge 1)\vee ({\mathit{h}}_{\mathit{i}}=1)\vee ({\mathit{e}}_{\mathit{i}}\ge 2)$	Combined validation and safety routing.
Conservative hybrid rule	$({\mathit{v}}_{\mathit{i}}\ge 2)\vee ({\mathit{h}}_{\mathit{i}}=1)\vee ({\mathit{e}}_{\mathit{i}}\ge 2)$	Lower-burden combined routing.

gives the deterministic policy definitions used in the frozen analysis.

These policies request human inspection; they do not make the final planning decision. The high-risk-containing policies are retained as operational definitions for reproducibility, but the sparse positive support for high_risk_unresolved prevents a separate detection-capability claim for that field.

4.7. External-Participant Evaluation Protocol

Seven external participants reviewed the framework presentation. The anonymized profiles cover requirements engineering, business analysis, agile planning, QA and acceptance testing, software architecture and technical risk, human–AI interaction, and empirical software engineering. Table 3 summarizes the anonymized participant roles used in the study.

The external panel rated 17 criteria on a five-point ordinal scale and could add open comments. The criteria covered problem relevance, contribution, clarity, framework structure, explainability, selective validation, planning usefulness, risk and uncertainty support, empirical design, practical adoption, and overall strength. Because the panel was small and the responses were ordinal, the results are reported only as descriptive face-validity and perceived-usefulness evidence. They do not establish workflow effectiveness or statistical generalization. The replication package contains the anonymized scores and profile key.

5. Results

The results are reported by study component. For each component, the analysis considers both predictive performance and review-routing value. The aim is to show what the configuration predicts and whether its signals help identify records that deserve closer inspection.

5.1. Experiment 1: NFR Classification and Selective Validation

Experiment 1 asks two practical questions about the NFR layer. First, how stable are the labels produced by the frozen configurations? Second, can disagreement and difficult taxonomy boundaries identify cases that need review? Experiment 1a uses the complete 622-record reference set from the same configuration-development context, so its estimates are comparative component results. Experiment 1c supplies the separate held-out diagnostic.

5.1.1. Experiment 1a: NFR Classification

Experiment 1a is a comparative component study within the same 622-record configuration-development context. Across the three frozen configurations, quality-level F1 was above 0.93, while exact NFR subtype performance was lower. C4TAX_R produced the highest supported macro F1 (0.691) and weighted NFR F1 (0.709), but the confidence intervals overlapped those of the other configurations. Performance also varied by category, with operability remaining particularly difficult. These results are treated as comparative component evidence rather than a generalization estimate; the held-out evidence is provided by Experiment 1c.

5.1.2. Experiment 1b: Selective Validation and Review Routing

Experiment 1b examines the review-routing value of the NFR layer. The question is whether signals such as configuration disagreement and taxonomy-boundary behavior can identify many of the NFR judgments that are likely to be wrong or unstable, without sending all requirements to human review.

Table 6. Experiment 1b NFR selective-validation results with bootstrap 95% confidence intervals.

Policy	Review Rate	Error Capture	Over-Review	Review Precision
Default: configuration disagreement	0.310 [0.275, 0.347]	0.599 [0.530, 0.665]	0.181	0.596 [0.525, 0.663]
High-safety: disagreement + overlap-prone class	0.426 [0.387, 0.465]	0.755 [0.695, 0.816]	0.279	0.547 [0.489, 0.606]
Extended high-safety: + FR/quality disagreement	0.461 [0.423, 0.500]	0.792 [0.732, 0.847]	0.314	0.530 [0.471, 0.585]

compares three policies. The default policy routes only cases with configuration disagreement. The high-safety policy also routes overlap-prone categories. The extended high-safety policy adds cases where the functional/quality boundary is unclear.

The policies expose a direct capacity trade-off. The default rule reviews 31.0% of the requirements and captures 59.9% of the strict NFR errors. The two broader rules capture more errors but also route more correct cases to review.

5.1.3. Experiment 1c: Taxonomy-Aware Verification

Experiment 1c is the held-out diagnostic for the NFR layer and provides the clearest evidence about performance outside the 622-record configuration-development context used in Experiment 1a. On 500 held-out requirements, exact-match accuracy was 0.716, micro F1 was 0.702, and quality accuracy was 0.874. These values confirm that broad quality-related identification is more stable than exact subtype assignment, especially for overlapping and multi-label cases.

Table 7. Experiment 1c taxonomy-aware held-out results with bootstrap 95% confidence intervals.

Metric	Result
Exact-match accuracy	0.716 [0.678, 0.758]
Micro precision	0.703 [0.652, 0.752]
Micro recall	0.701 [0.650, 0.748]
Micro F1	0.702 [0.658, 0.742]
Specificity	0.982 [0.978, 0.986]
Quality accuracy	0.874 [0.844, 0.902]
Raw errors/retries	0/1

reports the complete results.

Exact match was 0.858 for records with no gold NFR label, 0.675 for single-label records, and 0.235 for two-label records. The sharp decline for two-label cases supports review of taxonomy boundaries and multi-label requirements. It does not show that the taxonomy guide resolves exact NFR classification.

5.2. Experiment 2a: Weak-Word Requirement Defect Detection

Experiment 2a tests whether the weak-word layer can separate actual defects from acceptable weak expressions.

Table 8. Experiment 2a weak-word defect detection results with bootstrap 95% confidence intervals.

Metric	LLM-Only	Pattern-Arbitrated	Change
Accuracy	0.736 [0.696, 0.774]	0.762 [0.726, 0.798]	+0.026
Precision	0.678 [0.628, 0.726]	0.724 [0.672, 0.773]	+0.046
Recall	0.900 [0.862, 0.935]	0.848 [0.803, 0.893]	−0.052
F1	0.773 [0.734, 0.809]	0.781 [0.742, 0.820]	+0.008
Specificity	0.572 [0.512, 0.635]	0.676 [0.619, 0.733]	+0.104

compares the initial LLM decision with the final pattern-arbitrated decision.

Pattern arbitration increased accuracy from 0.736 to 0.762, precision from 0.678 to 0.724, specificity from 0.572 to 0.676, and F1 from 0.773 to 0.781. Recall fell from 0.900 to 0.848. The rule therefore removed some false alarms but also missed additional true defects. The McNemar exact test was borderline ($\mathit{p}=0.053$). The result is therefore interpreted as a useful specificity and review-precision trade-off under this rule, not as evidence that the arbitrated decision is better on every measure.

Weak-Word Error Analysis

The confusion counts make the trade-off more concrete. The LLM-only decision produced 225 true positives, 107 false positives, 25 false negatives, and 143 true negatives. The pattern-arbitrated decision produced 212 true positives, 81 false positives, 38 false negatives, and 169 true negatives. Arbitration changed 39 records, all from defect to ok: 26 changes corrected false positives, while 13 changes introduced false negatives. The specificity gain therefore reflects a real reduction in false alarms, while the recall loss can be traced directly to the override rules.

Among the 81 final false positives, 51 belonged to dataset-specific or uncategorized weak word patterns and 21 to the adequacy/sufficiency family. Among the 38 final false negatives, 19 were dataset-specific or uncategorized and 18 were in the adequacy/sufficiency family. The remaining error burden is therefore concentrated in these two families rather than distributed evenly across weak-word types. This error analysis is diagnostic of the frozen mechanism; the arbitration rules were not returned after examining these failures.

Table 9. Representative weak-word failure cases from Experiment 2a.

Case	Reference/Final	Mechanism Detail	Why the Case Failed
QURE-0004	OK/defect	No override; adequacy/sufficiency family	The phrase “adequate velocity independent cross-fading” was treated as insufficiently measurable, while the dataset reference marks the requirement acceptable.
QURE-0034	Defect/OK	technical_appropriate_low_risk_override	The protected technical-context rule suppressed a reference defect in “control logic is designed appropriately”, illustrating the recall cost of the override.

gives representative cases.

Table 10. Experiment 2a weak-word review-routing policies.

Policy	Review Rate	Defect Capture	Over-Review	Review Precision
Final predicted defect	0.586	0.848	0.324	0.724
Final defect or protected pattern	0.600	0.864	0.336	0.720
Required or optional review	0.824	0.960	0.688	0.583

summarizes the review-routing policies. The final defect-or-protected-pattern policy reviewed 60% of records and captured 86.4% of defects. The required-or-optional policy increased defect capture to 96%, but at a substantially higher review rate of 82.4%.

The missed-defect rate is the complement of defect capture. The required-or-optional policy missed 4.0% of defects but increased over-review to 68.8%. ReqPlan-Eval records the weak-word family, matched pattern, protected cue, and routing rationale for inspection.

5.3. Experiment 2b: User-Story Ambiguity Detection

Experiment 2b evaluates ambiguity detection for user stories.

Table 11. Experiment 2b ambiguity detection results with bootstrap 95% confidence intervals.

Metric	LLM-Only	Component/Goal-Gated	Change
Accuracy	0.674 [0.630, 0.716]	0.862 [0.832, 0.890]	+0.188
Precision	0.610 [0.562, 0.658]	0.899 [0.859, 0.937]	+0.289
Recall	0.968 [0.944, 0.988]	0.816 [0.764, 0.863]	−0.152
F1	0.748 [0.709, 0.784]	0.855 [0.820, 0.887]	+0.107
Specificity	0.380 [0.320, 0.441]	0.908 [0.872, 0.944]	+0.528

compares the LLM-only ambiguity decision with the final component/goal-gated decision. The component/goal gate produced the largest change. Accuracy rose from 0.674 to 0.862, precision from 0.610 to 0.899, specificity from 0.380 to 0.908, and F1 from 0.748 to 0.855. Recall fell from 0.968 to 0.816. The paired McNemar test was significant ($\mathit{p}<0.001$), but the practical result is the reduction in false alarms. The gate gives up some recall in exchange for much less over-flagging.

Table 12. Experiment 2b ambiguity review-routing policies.

Policy	Review Rate	Ambiguity Capture	Over-Review	Review Precision
Final gated ambiguous	0.454	0.816	0.092	0.899
LLM-only ambiguous	0.794	0.968	0.620	0.610
Required or optional review	0.846	0.972	0.720	0.574

reports the review-routing policies. The final gated policy reviewed 45.4% of stories and captured 81.6% of ambiguous stories with high review precision. Broader high-safety policies captured more ambiguity, but required substantially more review.

The final gated rule reduced the review rate from 79.4% for the LLM-only policy to 45.4% and reduced over-review from 62.0% to 9.2%, while ambiguity capture fell from 96.8% to 81.6%. The gate does not treat a weak business goal alone as a blocking defect when the actor, action, object, and planning scope are still clear.

Ambiguity Error Analysis

The LLM-only decision produced 242 true positives, 155 false positives, 8 false negatives, and 95 true negatives. After component/goal gating, the counts were 204 true positives, 23 false positives, 46 false negatives, and 227 true negatives. The gate suppressed 170 LLM-positive ambiguity flags. Of these, 132 were reference false positives that the gate correctly removed, while 38 were reference positives that became new false negatives. These transitions account for both the large specificity gain and the observed loss in recall.

Nineteen of the twenty-three final false positives involved a goal classified as unclear, and twenty-one involved a planning-sensitive goal. Among the 46 final false negatives, 24 involved multiple component issues and 13 involved an unclear goal; 33 were produced by the clear_components_no_cue_override rule. The main recall cost is therefore associated with the override that suppresses an ambiguity flag when components appear sufficiently clear, and no additional cue is present. This error analysis is diagnostic of the frozen gate; the gating rules were not retuned after examining these failures.

Table 13. Representative ambiguity failure cases from Experiment 2b.

Case	Reference/Final	Mechanism Detail	Why the Case Failed
US-C1-1961	Non-ambiguous/ ambiguous	LLM ambiguity retained; planning-sensitive goal	The goal “improve security” triggered a planning concern even though the dataset reference treats the story as non-ambiguous.
US-C8-12790	Ambiguous/ non-ambiguous	clear_components_no_cue_override	The story “monitor production so that I can improve efficiency” was treated as having sufficiently concrete components, while the dataset reference marks it ambiguous.

shows two representative cases.

5.4. Experiment 3: Role-Specialized Planning-Support and Review-Signal Alignment

Experiment 3 compares the four role outputs with the model-seeded, human-adjudicated reference described in Section 4. It contains 30 stories, including 20 public stories used for the main practical interpretation. The 120 role-level comparisons are clustered within stories and are not treated as 120 independent experimental units. The analysis supports review-signal and routing interpretation, not human-level planning generation.

5.4.1. Surface Agreement of Generated Planning Artifacts

Table 14. Experiment 3 strict normalized-item Jaccard for list-valued planning fields.

Field	Role-Agent Items	Reference Items	Mean Normalized-Item Jaccard
Quality issues	1.550	1.867	0.713 [0.638, 0.786]
Generated tasks	2.900	3.500	0.011 [0.001, 0.025]
Acceptance criteria	5.500	6.133	0.005 [0.002, 0.009]
Test ideas	6.108	6.467	0.001 [0.000, 0.003]
Dependencies	1.758	2.900	0.032 [0.002, 0.085]
Planning assumptions	3.725	4.000	0.003 [0.000, 0.008]
Clarification questions	5.183	5.367	0.004 [0.001, 0.007]
Secondary risks	1.983	2.067	0.606 [0.509, 0.701]

reports strict normalized-item Jaccard for the list-valued planning fields. Quality issues and secondary risks showed substantial wording overlap, but generated tasks, acceptance criteria, test ideas, dependencies, assumptions, and clarification questions showed very little.

The task, acceptance-criteria, and test-idea values are evidence of strong surface-level mismatch with the reference under this strict exact-item measure. The metric does not establish semantic disagreement, and the study did not perform a semantic-equivalence evaluation. No claim of semantic agreement or artifact usefulness is therefore made from these results.

5.4.2. Sensitivity to Human Adjudication of the Model-Seeded Reference

The two observed reference states allow a direct sensitivity check for the categorical fields that changed during later adjudication.

Table 15. Experiment 3 sensitivity of categorical agreement to later human adjudication.

Field	Stories Changed	Agreement with Initial Consensus	Agreement with Final Reference	Agreement on Changed Stories, Initial → Final
risk_level	7	0.925	0.725	0.929 → 0.071 (26/28 to 2/28)
uncertainty_level	13	0.883	0.533	0.904 → 0.096 (47/52 to 5/52)
validation_needed	2	0.933	0.883	0.875 → 0.125 (7/8 to 1/8)
recommended_action	1	0.775	0.758	0.500 → 0.000 (2/4 to 0/4)

Note: “Stories changed” counts fields whose value differs between Initial_Human_Consensus and Human_Adjudicated_Reference. Each changed story contributes four role-level comparisons for the field. The high_risk_unresolved field did not change during this adjudication and is not included as a performance result because its reference contains only one positive story.

compares role-level exact agreement with the initial two-reviewer consensus and the final three-reviewer adjudicated reference. The purpose is to measure how the observed later adjudication changed role-reference agreement, not to estimate the full causal contribution of the original model seed.

For every field changed during the third-reviewer stage, agreement on those stories was lower against the final reference than against the initial consensus. The largest movements were for risk and uncertainty. Later adjudication therefore did not simply preserve the role outputs when it changed a field. The experiment did not record the candidate state immediately before the first two reviewers’ corrections or separate pre-consensus records for those reviewers, so the total agreement attributable to the shared model seed cannot be isolated. Experiment 3 remains an evaluation against a model-seeded, human-adjudicated reference rather than an independent correctness benchmark.

5.4.3. Ordinal and Validation-Needed Review Signals

Risk exact agreement was 0.725 [0.575, 0.858], with linear weighted $\kappa =0.558$ and mean ordinal distance 0.275. Uncertainty exact agreement was 0.533 [0.391, 0.683], with $\kappa =0.308$ and mean distance 0.467. All risk and uncertainty disagreements were within one level.

For validation_needed, the reference contained 14 positive and 16 negative stories. Across the four role outputs, the confusion counts were TP = 44, FP = 2, FN = 12, and TN = 62, giving precision 0.957, recall 0.786, specificity 0.969, balanced accuracy 0.877, and macro F1 0.881. The reference contained only one high_risk_unresolved positive story. That sample is insufficient to evaluate high-risk detection capability, so no performance conclusion is drawn for that field.

5.4.4. Review-Policy Sensitivity

Table 16. Experiment 3 story-level review-policy sensitivity with Wilson 95% intervals.

Policy	Review Rate	Capture	Over-Review	Precision
Any validation-needed vote	0.500 [0.332, 0.668]	0.929 [0.685, 0.987]	0.125 [0.035, 0.360]	0.867 [0.621, 0.963]
Two-or-more validation-needed votes	0.400 [0.246, 0.577]	0.857 [0.601, 0.960]	0.000 [0.000, 0.194]	1.000 [0.758, 1.000]
Strong validation-needed vote	0.367 [0.219, 0.545]	0.786 [0.524, 0.924]	0.000 [0.000, 0.194]	1.000 [0.741, 1.000]
Two-or-more action-review votes	0.633 [0.455, 0.781]	1.000 [0.785, 1.000]	0.312 [0.142, 0.556]	0.737 [0.512, 0.882]

reports the policies that can be interpreted directly from the current validation reference. High-risk-containing variants are not used as evidence of high-risk detection because that field has only one positive reference story.

For every policy in Table 16, capture, over-review, and precision are evaluated against the story-level human-adjudicated validation_needed reference. A policy may use validation, risk, or action signals as its trigger, but the common comparison target is whether the reference indicates that human validation is needed.

The two-or-more validation-needed rule reviewed 12 of 30 stories and captured 12 of the 14 reference positives. All 12 routed stories were reference positives, although the Wilson interval shows the uncertainty of this small sample. The action-review rule captured all 14 positives, but its aggregate burden is misleading because the calibration stories were simple and none were routed.

5.4.5. Public-Versus-Calibration Sensitivity

The 20 public stories contained all 14 validation-needed reference positives. On that subset, role-level exact agreement was 0.688 for risk, 0.525 for uncertainty, 0.825 for validation need, and 0.713 for recommended action; 8.8% of action comparisons differed by more than one level. The 10 calibration stories contained no validation-needed reference positives and showed 0.800 risk agreement, 0.550 uncertainty agreement, 1.000 validation agreement, and 0.850 action agreement. Their easier structure and absence of validation positives make them unsuitable as independent generalization evidence.

On the public subset, the two-or-more validation-needed rule reviewed 12 stories, captured 12 of 14 positives, and routed none of the six negatives. By contrast, the broad action-review rule routed 19 of 20 public stories, including five of the six reference negatives. A 95% review rate provides almost no practical selectivity. The action-review rule is therefore not supported as a default routing policy in this dataset; it is useful only as a sensitivity comparator showing the cost of broad action disagreement.

5.4.6. Recommended-Action Distance

Across the 120 role-reference comparisons for recommended_action, 91 were exact matches, 22 differed by one level, and 7 were severe mismatches differing by more than one level. All seven severe mismatches occurred in the public subset. Thus, 113 of 120 comparisons were within one level, but the severe errors were concentrated in the more demanding stories.

Experiment 3 therefore provides preliminary evidence for validation-needed routing on the public stories. It does not provide evidence of high-risk detection capability or semantic agreement of the open-ended planning artifacts.

5.5. External-Participant Evaluation Results

The 17 criteria are reported individually rather than collapsed into an unweighted overall mean because they cover different constructs.

Table 17. Selected external-participant evaluation results for the central review-support claim.

Criterion	Mean	Interpretation
Problem relevance	4.86	The requirements-to-planning problem was judged highly relevant.
Analytical-layer table	4.86	Participants found the evidence and routing structure clear and useful.
Selective human validation	4.71	Review routing was viewed as a main strength of the framework.
Human-AI interaction	4.71	Participants agreed that the framework keeps the human reviewer in control.
Risk and uncertainty support	3.57	Useful as review signals, but the thresholds remain partly judgment-based.
Practical adoption	3.57	Workflow fit and review burden still require task-based study.
Overall framework strength	4.00	This is a single questionnaire item, not an aggregate across the 17 criteria.

shows the items closest to the central review-support claim; the replication package contains the complete anonymized matrix. The strongest ratings concerned problem relevance, the analytical-layer presentation, selective human validation, and human control. Lower ratings for risk and uncertainty support and practical adoption are consistent with the remaining need for clearer thresholds and task-based workflow evidence.

6. Discussion

Table 18. Claim, direct evidence, supported interpretation, and boundary.

Claim	Direct Evidence	Supported Interpretation	Explicit Boundary
Pattern-aware weak-word arbitration	Experiment 2a	Improves specificity and precision under the evaluated rule.	Recall decreases, and the paired test is borderline.
Component/goal ambiguity gating	Experiment 2b	Reduces over-flagging and improves balanced classification.	Recall decreases, and the result is dataset-specific.
NFR selective validation	Experiments 1a to 1c	Disagreement and difficult taxonomy boundaries can prioritize review.	Exact subtype classification remains difficult.
Validation-needed routing	Experiment 3 public subset	Multiple role signals can support selective review routing.	The reference is small and model-seeded.
Open-ended planning artifacts	Experiment 3 lexical diagnostic	Tasks, acceptance criteria, and test ideas have strong surface mismatch with the reference.	Semantic equivalence and usefulness were not evaluated.
Integrated ReqPlan- Eval pipeline	Architectural specification only	The components are organized in one coherent review-routing design.	The complete pipeline was not evaluated end-to-end on a common requirement set.
Human-control design	Seven-person external-participant evaluation	Provides initial face-validity support for selective review and override.	No task-based workflow or productivity evidence is available.

separates the architecture from the evidence directly provided by the experiments.

6.1. What the Current Evidence Supports

The clearest component evidence concerns the two planning-aware mechanisms. Pattern arbitration reduced false weak-word alarms and improved review precision, with a loss in recall. Component/goal gating produced a larger reduction in ambiguity over-flagging, again with lower recall. The held-out 500-requirement NFR diagnostic achieved 0.716 exact-match accuracy, 0.702 micro F1, and 0.874 quality accuracy; the sharp decline on two-label records shows why NFR disagreement is more defensible as review evidence than as an authoritative subtype decision.

Experiment 3 supports a narrower conclusion. The two-or-more validation-needed policy provided the most credible selective-routing trade-off on the public subset, reviewing 12 of 20 stories and capturing 12 of 14 positives without routing any of the six negatives. The broad action-review rule, in contrast, reviewed 19 of 20 public stories. That burden largely defeats the purpose of selective review, so the current evidence argues against using broad action-based routing as the default policy.

The generated planning artifacts require a different interpretation. Near-zero strict normalized-item Jaccard values for tasks, acceptance criteria, and test ideas show strong surface-level mismatch with the reference. The study did not evaluate semantic equivalence, so it provides no basis for claiming that differently worded artifacts nevertheless express the same planning content. A semantic or task-based evaluation is required before making that claim.

6.2. Architecture Versus End-to-End Evidence

ReqPlan-Eval is integrated as an architecture: NFR evidence, weak-word evidence, ambiguity signals, and role-specialized planning hypotheses meet in one planning-support record and feed configurable review policies. The experiments do not, however, test that complete sequence on one common set of requirements. Experiments 1 and 2 evaluate separate component mechanisms, while Experiment 3 evaluates role outputs and routing signals. The contribution should therefore be read as an integrated review-routing design with component-level and routing-layer evidence, not as an end-to-end validated planning system.

This distinction also limits claims about human benefit. The seven-participant evaluation provides perceived-relevance and perceived-usefulness evidence, but the study does not measure review time, changed decisions, workload, or workflow performance.

6.3. Reference Boundaries and Next Evaluation

The Experiment 3 planning reference is model-seeded and human-adjudicated. The adjudication-sensitivity analysis shows that later human adjudication often moved changed categorical fields away from the role outputs, particularly for risk and uncertainty. However, the experiment did not collect a parallel human-only reference created without exposure to the seeded candidates or separate pre-consensus records for the first two reviewers. The full contribution of the original model seed therefore cannot be isolated.

The next evaluation should test the complete workflow on a common requirement set with independently authored reference judgments and human review outcomes. It should compare review with and without ReqPlan-Eval and measure missed issues, changed decisions, review time, workload, confidence, and override behavior. A semantic assessment of the open-ended planning artifacts should be included in the same design. The component studies also used different frozen model configurations, so cross-model invariance remains outside the current evidence.

7. Threats to Validity

7.1. Construct Validity

Several reported fields are operational review judgments rather than fixed project facts. NFR labels can overlap, and Experiment 3 risk, uncertainty, validation, and action fields depend on the adopted definitions. Class-sensitive measures and ordinal distance are used where the reference supports them. The high_risk_unresolved reference contains only one positive story, which is insufficient for evaluating high-risk detection capability; the field is retained as an operational part of the framework but not used to support a detection claim. Strict normalized-item Jaccard measures exact surface item overlap only. The very low values for several open-ended planning fields establish surface mismatch under that measure, not semantic equivalence or usefulness.

7.2. Reference and Internal Validity

Prompts, schemas, gates, arbitration rules, and routing policies were frozen before the reported runs. Experiment 3 reference fields were withheld from the role agents. Experiment 1a is a comparative component study within the same 622-record development context, not independent held-out validation; Experiment 1c provides the held-out NFR diagnostic.

Experiment 3 has a separate reference-validity limitation. Candidate planning records were model-seeded and then jointly reviewed and corrected by two human software engineering practitioners. A third human reviewer examined all 30 records, with particular attention to quality assurance and testability, and provided adjudication input. All three reviewers had access to the model-generated starting records. The resulting reference is therefore human-adjudicated, but it is not an independently authored human-only benchmark created without exposure to the model-seeded candidates.

The adjudication-sensitivity analysis characterizes part of this dependence. For fields changed during the later adjudication, agreement with the role-agent outputs generally decreased sharply. On stories whose risk_level changed, agreement fell from 0.929 with the initial consensus to 0.071 with the final reference; for changed uncertainty_level, it fell from 0.904 to 0.096. These movements show that later human adjudication did not simply preserve the role-agent outputs when a field was changed. The experiment did not record the candidate state immediately before the first two reviewers’ corrections or separate pre-consensus records for those reviewers, and it did not collect a parallel human-only reference constructed without exposure to the seeded records. The total contribution of the initial model seed therefore cannot be isolated experimentally. Experiment 3 is interpreted as alignment with a model-seeded, human-adjudicated planning reference rather than agreement with independent human ground truth.

The 10 simple stories were used during calibration, are reported separately, and are not independent held-out evidence. Prompt and model sensitivity also remain limitations. The component studies used different frozen model identifiers and were not rerun under one common configuration. The before/after comparisons in Experiments 2a and 2b reuse the same stored model judgments, so their mechanism effects are not caused by a model change; however, cross-model invariance of the framework-level findings is not established. The study also does not compare all prompting or self-consistency strategies.

Because all three reference reviewers saw the model-generated starting records, the reference-review process is not treated as independent of the model seed.

7.3. External and Practical Validity

The datasets do not represent all domains, regulated settings, large backlogs, or team practices. Experiment 3 contains 30 stories, and the main practical interpretation relies on 20 public stories. Because the component studies use separate datasets, combined workflow performance remains outside the present evidence. The seven-participant evaluation does not replace a practitioner task study, and the paper does not measure review time, better decisions, adoption effort, or integration into a live backlog workflow. The 95% review rate of the broad action rule on the public subset also shows that some routing policies can be impractical even when they capture many positives. Automation bias and review fatigue remain practical risks.

7.4. Conclusion Validity

Bootstrap intervals and paired tests support the component comparisons but do not remove dataset-specific limits. Experiment 3 role-level intervals resample complete stories as clusters, and story-level policy proportions use Wilson intervals. The conclusions remain component-specific: ambiguity gating and weak-word arbitration have the strongest direct mechanism evidence; exact NFR subtype classification remains difficult; validation-needed routing has preliminary support on the public subset; and the study does not establish high-risk detection, semantic agreement of open-ended artifacts, combined workflow effectiveness, or industrial effectiveness.

8. Conclusions

ReqPlan-Eval organizes NFR disagreement, weak-word evidence, planning-relevant ambiguity, and role-specialized planning hypotheses into an inspectable human review-routing architecture. The empirical evidence is deliberately narrower than the architecture. A held-out NFR diagnostic showed that exact subtype classification remains difficult; pattern-aware arbitration improved the weak-word specificity and precision trade-off; and component/goal gating sharply reduced ambiguity over-flagging, with recall losses in both mechanisms.

For Experiment 3, the most defensible result is validation-needed routing on the public stories. The two-or-more-vote rule reviewed 12 of 20 public stories and captured 12 of 14 reference positives without routing a reference negative. Broad action routing is not supported as a default in this dataset because it reviewed 19 of 20 public stories. The reference contains only one high_risk_unresolved positive story, so no high-risk detection conclusion is drawn. Generated tasks, acceptance criteria, and test ideas showed strong surface-level mismatch with the reference, and semantic agreement was not evaluated.

The planning reference remains model-seeded and human-adjudicated, and the full contribution of the initial model seed cannot be isolated. The component studies also used different frozen model configurations. ReqPlan-Eval should therefore be read as an integrated review-routing design with component-level and routing-layer evidence, not as an end-to-end or industrially validated planning system. The next study should evaluate the complete workflow on a common requirement set with independent reference construction and task-based measures of decision quality, review time, workload, and override behavior.