Toward Ludic-Aware Narrative Generation: A Neuro-Symbolic Framework for Evaluating Playability in LLM-Generated Backstories

Round 1

Reviewer 1 Report

Comments and Suggestions for Authors

In the manuscript titled “Toward Ludic-Aware Narrative Generation: A Neuro-Symbolic Framework for Evaluating Playability in LLM-Generated Backstories” (applsci-4575921), authors propose a structure-enhanced and ensemble-stabilized framework for training-sample contribution valuation. Extensive experimental results demonstrate their efficiency. However, I have several concerns outlined below.

1. In this manuscript, overall playability shows a significant correlation with LPI after multiple-comparison correction, whereas openness, conflict potential, and preparation burden do not. Moreover, the reported ICC values are also low.
2. The authors do not provide the sensitivity analysis for the parameters in PCV. These parameters, such as w_I，w_G, w_V，w_D, PCV threshold, and the conflict-saturation rule, are set in human choices.
3. TH reaches its maximum at three thematic holes, while its median value in the experimental corpus is already 0.925. I think the contribution of TH is limited.
4. Instability_prompt directly push the model to add a living NPC, a Tier-1 hook, and a proactive motivation. It seems cloud increase RTN, UCR, and MV.
5. The related work section is incomplete, some recent noisy label work such as urct: a two-stage noisy label learning framework with uniform consistency selection and robust training should be discussed and contrasted with the proposed approach.
6. The organization and formatting of the manuscript need further improvement. For example, the non-standard section number “3.0.1” appears on Page 5 and is immediately followed by Section “3.1”. In addition, the manuscript contains overly frequent paragraph breaks in several places, such as Page 10.

Author Response

The answers and details are in the attached PDF file for easier reading.

Author Response File: Author Response.pdf

Reviewer 2 Report

Comments and Suggestions for Authors

Comments and recommendations to the authors are provided in the file.

Comments for author File: Comments.pdf

Author Response

The answers and details are in the attached PDF file for easier reading.

Author Response File: Author Response.pdf

Reviewer 3 Report

Comments and Suggestions for Authors

< Major Concerns >

1. Weak construct validity between LPI and the downstream outcome it is meant to predict The theoretical backbone of the paper is that LPI (and the gate built on it) should reduce downstream "Adaptation Effort" (Ea) — i.e., that a more "open" backstory requires less improvisation from a Game Master. Yet under the paper's own semantic Ea formulation, the LPI–Ea correlation is essentially zero (pooled r = -0.206 in Study 1; r ≈ +0.09 in the pilot). The authors appropriately withdrew the correlational hypothesis, but this leaves the interventional gate effect as the sole evidence for the framework's practical value, and that effect (see #3 below) is itself small and inconsistent across generators.

2. Human construct-validity results are marginal In the n=60, 11-rater human study, LPI correlates with overall human-rated playability at only r=0.36 (95% CI [0.14, 0.58]) — the only one of four convergent hypotheses to survive Holm–Bonferroni correction. Inter-rater reliability (ICC(1)) fails to reach the pre-registered acceptance threshold of 0.50 for any item (best case 0.26). The authors are candid that this is a "first wave," but as it stands the claim that LPI/PCV track expert judgments of playability rests on thin evidence.

3. Underpowered/weak downstream effects. H1b (the gating×generator interaction, one of the paper's primary hypotheses) is directional but not significant (F(1,156)=2.25, p=0.136) at n=40/cell.

In the expanded nonparametric analysis (Section 5.6), Ea differentiates only 4 of 45 pairwise comparisons (vs. 24/45 for LPI and 16/45 for PCV). The authors themselves state this shows "gating reshapes the metrics it selects on strongly and moves the downstream Synthetic-DM load weakly" — an important qualifier that somewhat undercuts the abstract's framing ("meaningful ... reduction").

4. Generalizability is narrow. All experiments use a single Forgotten Realms (D&D-style) Story Bible and a single fantasy/conflict-oriented genre convention. The authors' own "Pro-Conflict Bias" discussion (Section 5.10) acknowledges PCV, as defined, is unsuitable for cozy/life-sim genres and would require re-derivation of weights — a genuinely useful admission, but one that leaves a core generalizability question open rather than resolved.

5. PCV weights are unvalidated by construction. Unlike LPI's weights (informed at least by empirical draw-frequency data from Synthetic-DM sessions), PCV's weights (wI=wG=4, wV=wD=1) were fixed before any data collection and are explicitly excluded from the sensitivity analysis. One of the paper's two central metrics therefore rests entirely on an unvalidated design choice.

< Minor Comments >

The original branching-diversity metric (H2) saturated at ceiling and had to be replaced post hoc by Divα — a reasonable fix, but it suggests the original metric was insufficiently piloted before being pre-registered.

The explanation for Opus 4.7's higher Ea (it emits more total entities, inflating both numerator
and denominator) is plausible but exposes that Ea is not normalized for generation verbosity — a design limitation the authors correctly flag as future work but does affect interpretation of the
current headline contrast.

Please state exact model version/build dates for "Opus 4.7," "claude-sonnet-4-6," and "qwen3"
variants to support reproducibility, since these are fast-moving, versioned commercial systems.

Consider softening abstract language ("meaningful ... reduction," "successfully reining in
variability") to match the more qualified, honest tone of the Results and Discussion sections.

Author Response

The answers and details are in the attached PDF file for easier reading.

Author Response File: Author Response.pdf

Round 2

Reviewer 1 Report

Comments and Suggestions for Authors

All my concerns have been addressed.

Reviewer 3 Report

Comments and Suggestions for Authors

The authors have thoroughly and honestly addressed all of my previous comments, and I recommend Accept.