Review Reports

Round 1

Reviewer 1 Report

Comments and Suggestions for Authors

Dear Authors,

I read with interest the article devoted to the development and application of a three-layer backpropagation neural network for multi-output regression in the context of assessing the impact of citizen-science projects. The topic is timely, and the focus on a practically oriented solution within the MICS framework makes the study significant and well positioned in the contemporary scientific and applied context. It is a positive impression that you do not limit yourselves to a purely conceptual description, but present a real architecture, data, a training procedure, an evaluation protocol, and a reproducible implementation. This gives the manuscript concreteness and demonstrates a serious attitude toward the technical side of the development. The article is structured clearly and coherently, and the methodological section is particularly impressive. The introduction succeeds in outlining the motivation for the study and its place within the MICS framework, while the discussion demonstrates that you are aware of the limitations of the proposed model and do not attempt to present the results as stronger than they actually are. This creates an impression of scientific correctness and balance. Another positive aspect is that the work has a real applied contribution, as you propose an initial, transparent baseline for the automated assessment of impact in citizen science.

Alongside these undeniable strengths, however, some weaknesses can also be observed in the text. First, the empirical basis remains too limited, since the model is trained on a very small number of projects, which inevitably raises questions about the degree to which the results can be generalized. In addition, although the methods are described in considerable technical detail, in several places the methodological justification for the choices made is not sufficiently strong. The literature review could also be developed in greater depth so that the article can be positioned even more convincingly in the international scientific context. There are also certain conceptual and presentational ambiguities that could make the text more difficult for the reader and weaken its overall impact.

Although, overall, the article gives the impression of work well done, unfortunately I also find a considerable number of omissions. I have several comments and questions about them that I would like to discuss; they are described in the attached file.

Comments for author File: Comments.pdf

Author Response

Response to reviewers

Manuscript: Design and Implementation of a Three-Layer Backpropagation Neural Network for Multi-Output Regression in Citizen-Science Impact Assessment

Authors: Luigi Ceccaroni, Lyle Visa, Iain Visa

Manuscript ID: ai-4248044

We thank both reviewers for their careful, constructive, and encouraging reading of the manuscript. Their comments have substantially improved the clarity, rigour, and completeness of the work. Below we address each point in turn. Changes to the manuscript are indicated by section reference; all revisions are in the version submitted alongside this response.

Response to Reviewer 1

Comment 1.1 — One-hot padding for questions with fewer than 20 options: it would be good to state explicitly in the main text what happens to the unused positions.

Response: We have added an explicit statement in Section 2.1 clarifying that questions with fewer than 20 response options are represented with the unused positions set to zero, so that all input rows are padded to the same width and the 4,460-dimensional structure is preserved uniformly across all projects. The same clarification is repeated in Section 2.10 (Data Preprocessing and Schema) for completeness.

Revised text (Section 2.1): "Questions with fewer than 20 response options are represented with the unused positions set to zero, so that all input rows are padded to the same width and the 4,460-dimensional structure is preserved uniformly across projects."

Comment 1.2 — Clarify whether there was more than one expert evaluator and whether inter-rater agreement was measured.

Response: We agree this was insufficiently described. Section 2.1 has been expanded to clarify that domain scores were assigned by a panel of one to three domain experts, each scoring the five impact domains via a structured evaluation protocol. Disagreements were resolved through discussion and consensus. We also acknowledge that the limited number of raters and the inherent subjectivity of impact scoring remain limitations, and that future work should expand the expert panel.

Revised text (Section 2.1): "Domain scores were assigned by a panel of one to three domain experts, scoring the five impact domains for every project via a structured evaluation protocol. Disagreements were resolved through discussion and consensus, and the agreed scores constitute the labels used for training."

Comment 1.3 — State how many incomplete MICS projects were initially available and how many were excluded.

Response: This information has been added to Section 2.1. Of the 24 projects that initiated the full MICS self-assessment, nine provided complete responses and expert domain-scores (labelled dataset), and were included. The remaining 15 were excluded owing to the lack of expert domain-scores.

Revised text (Section 2.1): "Of the 24 projects that initiated the full MICS self-assessment, nine provided complete responses and expert domain-scores, and were included in the study. The remaining 15 were excluded owing to the lack of expert domain-scores."

Comment 1.4 — Clarify explicitly that no feature selection, dimensionality reduction, or redundancy analysis was performed beyond encoding and flattening, and provide justification.

Response: Correct; no feature selection or dimensionality reduction was performed. We have added a dedicated paragraph at the end of Section 2.2 explaining this decision explicitly, with justification (preserving all signal given the very small n, where data-driven feature-selection statistics would be unreliable) and acknowledging it as a limitation.

Revised text (Section 2.2): "No feature selection, dimensionality reduction, redundancy analysis, or variance screening was performed beyond the encoding and flattening described above. This decision was deliberate: given the very small sample size (n = 9), any data-driven feature-selection step [...] would risk discarding genuine predictive signal while providing unreliable selection statistics [...] A systematic analysis of feature variability and inter-question correlation is identified as important future work when a larger labelled dataset becomes available."

Comment 1.5 — Explain why 42 neurons was chosen as a practically suitable width; note whether a sensitivity analysis was conducted or should be conducted.

Response: We have strengthened the justification for 42 neurons in Section 2.7. We now state explicitly that this width was arrived at heuristically and that its coincidence with the ordinal scale has no methodological significance. We want to acknowledge that a systematic ablation study comparing different neurons widths of 10, 20, 42, 64, and 100 was performed but not documented and its documentation is identified as a priority for future work. The qualitative rationale (large enough for representational capacity, small enough to be strongly constrained by L2 regularisation relative to the 4,460-dimensional input) is also provided. Additionally, we now cite literature on shallow networks for small tabular datasets (Shwartz-Ziv & Armon, 2022; Klambauer et al., 2017: new references [8,9]) to support the choice of a single hidden layer over a deeper architecture.

Revised text (Section 2.7): "A systematic ablation study comparing hidden-layer widths of 10, 20, 42, 64, and 100 neurons was performed. The qualitative rationale for 42 as a reasonable width is that it is large enough to represent diverse combinations of questionnaire responses but small enough, relative to the 4,460-dimensional input, to be strongly constrained by the L2 regularisation, limiting the risk of memorising the nine training instances."

Comment 1.6 — Justify the use of logistic activations and cross-entropy loss for a regression task; explain why this is preferred over MSE/MAE or ordinal regression.

Response: We have added a detailed methodological justification in Section 2.3 (Loss) to clarify these architectural choices. Because each target is produced by dividing an integer expert score by 42, the target space is strictly bounded within [0, 1]. This formulation allows us to treat the targets as "soft labels." Specifically, we clarify three main points in the revised text:

• Why Sigmoid + Binary Cross-Entropy (BCE): Pairing a sigmoid output with BCE is a mathematically coherent surrogate for bounded regression that minimises the divergence between the true and predicted distributions.
• Why not MSE/MAE: Unlike MSE or MAE paired with a sigmoid output, BCE perfectly cancels out the derivative of the sigmoid function. This produces well-conditioned gradients throughout the [0, 1] range and avoids vanishing gradients near saturation. This is a critical property for our setup, as our training signal is severely limited (n ≤ 9 per fold). We have added a reference to Goodfellow, Bengio & Courville (2016, Sec. 6.2.2.2) detailing this specific advantage.
• Why not Ordinal Regression: The continuous sigmoid mapping inherently preserves the ordinal structure of the underlying expert scores. This achieves the goal of ordinal regression without requiring the additional architectural complexity (e.g., multi-head outputs) that ordinal formulations typically demand, which is advantageous given our dataset size.

 

Comment 1.7 — Explain what "higher values of lambda are used only for diagnostic cost checks" means, and why this is important.

Response: Section 2.3 now explains the lambda = 1 diagnostic explicitly: inflating lambda to this much larger value forces very small weights and drives all sigmoid outputs toward 0.5, producing a cost close to  (the entropy of a uniform Bernoulli). Confirming this diagnostic cost validates that the forward pass, label scaling, and cost implementation are internally consistent. This check is used solely for sanity-checking the implementation and does not influence the operational lambda = 0.07. Appendix C has also been updated to connect the "predictions collapsing to mid-range" troubleshooting note to this diagnostic.

Comment 1.8 — Clarify whether empirical correlations among the five domains were examined, and whether independent logistic outputs is a reasonable assumption.

Response: A new paragraph has been added in Section 2.4 reporting that pairwise Pearson correlations among the five domain scores were examined across the nine projects (range approximately −0.4 to +0.6). No pair of domains shows a sufficiently strong and consistent relationship to justify joint modelling at this sample size. The assumption of conditionally independent logistic outputs is therefore a reasonable working approximation for the current data, but we acknowledge that if future data reveal systematic cross-domain dependencies, multi-task learning architectures may outperform the independent-head formulation.

Comment 1.9 — Include a brief project profile table describing the nine projects in broad terms without disclosing confidential information.

Response: Table 1 has been added as a new Section 2.11 ("Project Characterisation"). It presents all nine projects with descriptors for thematic focus, geographic region, approximate active duration, and order-of-magnitude participant count. No confidential identifiers are disclosed. The section also restates the count of initially available projects and those excluded.

Comment 1.10 — Clarify how many different random initialisations were run and whether the best was selected or results were averaged.

Response: A new Section 2.12 ("Random Initialisations") has been added, clarifying that five independent random initialisations were run per LOOCV fold. The initialisation yielding the lowest training cost was selected for each fold. Results were qualitatively consistent across initialisations (RMSE variation ±1 point on the 1–42 scale). We acknowledge that five restarts do not guarantee identification of the global minimum of the non-convex loss surface, and a more thorough multi-restart procedure is identified as future work.

Comment 1.11 — Clarify whether internal diagnostics were used for actual hyperparameter selection or only for descriptive monitoring.

Response: Sections 2.7 and 3.2 have been updated to make this explicit. The internal diagnostics (loss monitoring and training-set RMSE) were used as informal heuristic monitoring, not as a formal cross-validated model-selection criterion. Hyperparameter choices (lambda = 0.07, MaxIter = 10, 42 neurons) were made heuristically based on training-set diagnostics across a small candidate set, as described in Section 2.7. We explicitly state that this is not a rigorous hyperparameter optimisation procedure and recommend formal approaches (e.g., Optuna) for future work.

Comment 1.12 — Change "multiclass classification" in the keywords to "multi-output regression".

Response: Done. The keyword "multiclass classification" has been replaced with "multi-output regression" throughout the manuscript, including the abstract keywords line. The term "classification tasks" in Section 1.4 has also been corrected to "tasks involving noisy, multidimensional inputs" to eliminate the conceptual tension between the keyword and the regression framing.

Comment 1.13 — Resolve the inconsistency between Section 3.4 presenting LOOCV results and Section 5.3 referring to LOOCV as future work.

Response: Section 5.3 has been updated to remove all references to LOOCV as future work. LOOCV has been implemented and reported; Section 5.3 now focuses on the remaining future research priorities: systematic hyperparameter optimisation, baseline comparisons, expanded training data, and uncertainty quantification. Section 2.5 has also been clarified to state explicitly that LOOCV was implemented and carried out.

Comment 1.14 — The mention of "early stopping" in Section 4.3 is inconsistent with the methodology, which only specifies MaxIter = 10.

Response: That was a mistake coming from an earlier draft. The phrase "early stopping" has been removed from Section 4.3. The regularisation strategies employed are correctly described as L2 weight decay only; MaxIter = 10 defines the computational budget rather than constituting a proper early-stopping procedure. The revised text refers only to "L2 regularisation" in this context.

Comment 1.15 — Tables 1a and 1b could be combined into a single consolidated table with observed/predicted columns.

Response: We appreciate this suggestion. Tables 1a and 1b have been combined into a single consolidated Table 2 with observed/predicted columns.

Comment 1.16 — Resolve the inconsistency between "223 input features", "~200 features", and the actual 4,460-dimensional input.

Response: All instances of "approximately 200 input features" or "~200 features" have been replaced consistently throughout the manuscript with "223 questions" (when referring to the instrument items) and "4,460-dimensional binary vector" (when referring to the encoded input). In particular, Section 4.2, which previously used "approximately 200 input features", now reads "223 questions encoded as a 4,460-dimensional binary vector". No approximation of the feature count appears anywhere in the revised manuscript.

Comment 1.17 — The reference list is short; engage more extensively with existing international experience.

Response: The reference list has been expanded from 6 to 25 entries. New references cover: multi-output regression and MLP for small tabular datasets [6–10, 20]; citizen-science evaluation frameworks [13–15, 21]; machine learning for environmental and impact assessment [4, 5, 16, 19]; hybrid rule/ML systems [17, 18]; numerical optimisation methods [22, 23]; hyperparameter optimisation [24]; and ensemble/tree-based methods [25]. All new references are woven into the text at appropriate points and are internally consistent in author, year, title, and journal.

Response to Reviewer 2

Comment 2.1 — Justify the choice of a single hidden layer more explicitly; cite relevant literature on shallow vs. deep networks for small-sample tabular data.

Response: Section 2.7 has been expanded with an explicit justification for the single hidden layer choice, supported by two new references: Shwartz-Ziv & Armon (2022), which demonstrates that deep learning does not consistently outperform shallow models on tabular data, and Klambauer et al. (2017) , which establishes that on small tabular datasets compact networks with one hidden layer typically match or outperform deeper ones due to reduced overfitting risk. With n = 9, a single hidden layer is the appropriate default given the parameter-count implications of additional layers.

Comment 2.2 — Mention or recommend a systematic hyperparameter search approach (e.g., grid search, random search, Optuna) even if not yet implemented; note this as a limitation/future work.

Response: Section 2.7 now explicitly recommends Optuna (a new reference added) as a systematic Bayesian optimisation approach for future hyperparameter search, and acknowledges that the current selection procedure was heuristic. Section 5.3 identifies this as the first priority direction for future work.

Comment 2.3 — Add a section comparing Alquimics to simpler baselines; discuss why linear regression or Random Forest were not compared; acknowledge this as a limitation.

Response: A dedicated paragraph on baseline comparisons has been added to Section 4.5. It identifies three natural comparators not included in this study — (1) a mean predictor, (2) linear/ridge regression, and (3) Random Forest [25, new reference] — and explains why they were not implemented: the primary focus of this paper is establishing the Alquimics baseline and its integration into MICS, not comparative model evaluation, and the small n = 9 makes reliable estimation of any model's performance equally challenging. Systematic baseline comparisons are explicitly identified as the most important immediate next step in Section 5.3.

Comment 2.4 — Reiterate and expand the justification for MaxIter = 10; if values were not explored, state this as a limitation.

Response: The justification for MaxIter = 10 has been expanded in Section 2.3 to describe the convergence diagnostic used: the training loss was compared at iterations 5, 10, and 20 across multiple random initialisations. In all runs examined, the loss stabilised to within 1% of its final value by iteration 10, with no further decrease at 20 iterations. We acknowledge that a rigorous validation would involve a systematic sweep across a wider range of MaxIter values (e.g., 10, 20, 50, 100) and identify this as future work in Section 5.3. A new reference to Nocedal & Wright (2006) supports the discussion of conjugate-gradient convergence behaviour.

Comment 2.5 — Section 2.8 pseudocode → narrative description.

Response: The pseudocode block in Section 2.8 has been replaced with a 5-sentence narrative prose description of the training procedure, covering: weight initialisation, cost-function construction, fmincg optimisation, parameter reshaping, and prediction/output rescaling.

Comment 2.6 — Section 3.5 checklist → narrative paragraph.

Response: The informal bullet-list reproducibility checklist in Section 3.5 has been rewritten as a coherent flowing paragraph in proper scientific prose, covering: code availability, data schema documentation, random seed logging, tested environment, single-command run path, and evaluation protocol specification.

Comment 2.7 — Conclusions clarity: explicitly state (a) main contribution, (b) key quantitative result, (c) principal limitation, (d) forward-looking implication.

Response: The conclusions section has been fully rewritten as four substantive sentences explicitly addressing all four requested elements: (a) the main technical contribution (Alquimics 4,460–42–5 MLP integrated into MICS alongside a rule-based engine); (b) the key quantitative result (LOOCV overall RMSE = 10, R² = 0.06; best domain Governance RMSE = 6, R² = 0.3); (c) the principal limitation (n = 9, constraining generalisation and hyperparameter sensitivity estimation); and (d) the forward-looking implication (the same pipeline scales as new labelled projects accrue).

We believe the revised manuscript is substantially stronger in all dimensions identified by the reviewers. We are grateful for the time and expertise invested in the review process and welcome any further questions.

Yours sincerely,

 

Luigi Ceccaroni (on behalf of all authors)

Earthwatch, Oxford, UK

lceccaroni@earthwatch.org.uk

 

Reviewer 2 Report

Comments and Suggestions for Authors

The paper proposes Alquimics, a feedforward neural network that uses structured descriptors of citizen science projects to predict multidimensional impact across five domains, formulated as a multi-output regression problem. The results obtained through LOOCV validation show limited predictive performance, but establish a transparent and reproducible baseline for future extensions.

Although the article is well structured, several aspects require improvement:

• The choice of a single hidden layer MLP network is not sufficiently justified. Also, the selection of 42 neurons in the hidden layer requires a more rigorous justification. A systematic method for hyperparameter optimization (e.g. tools like Optuna) should be used.
• Comparisons with alternative models, both from the neural network category and simpler baselines (e.g., linear regression, Random Forest), are missing. Such comparisons are essential to objectively evaluate the performance of the proposed model.
• The number of iterations for optimization (MaxIter = 10) is fixed without justification. Several values ​​must be explored and the optimal value identified.
• Section 2.8 should be reconsidered, as the use of pseudocode is relatively uncommon and could be replaced with a more concise description.
• Section 3.5 requires reformulation, removing the informal checklist structure and adopting a narrative style appropriate for a scientific article.
• The conclusions should be stated more clearly and concisely, explicitly highlighting the main contributions.

Author Response

Response to reviewers

Manuscript: Design and Implementation of a Three-Layer Backpropagation Neural Network for Multi-Output Regression in Citizen-Science Impact Assessment

Authors: Luigi Ceccaroni, Lyle Visa, Iain Visa

Manuscript ID: ai-4248044

We thank both reviewers for their careful, constructive, and encouraging reading of the manuscript. Their comments have substantially improved the clarity, rigour, and completeness of the work. Below we address each point in turn. Changes to the manuscript are indicated by section reference; all revisions are in the version submitted alongside this response.

Response to Reviewer 1

Comment 1.1 — One-hot padding for questions with fewer than 20 options: it would be good to state explicitly in the main text what happens to the unused positions.

Response: We have added an explicit statement in Section 2.1 clarifying that questions with fewer than 20 response options are represented with the unused positions set to zero, so that all input rows are padded to the same width and the 4,460-dimensional structure is preserved uniformly across all projects. The same clarification is repeated in Section 2.10 (Data Preprocessing and Schema) for completeness.

Revised text (Section 2.1): "Questions with fewer than 20 response options are represented with the unused positions set to zero, so that all input rows are padded to the same width and the 4,460-dimensional structure is preserved uniformly across projects."

Comment 1.2 — Clarify whether there was more than one expert evaluator and whether inter-rater agreement was measured.

Response: We agree this was insufficiently described. Section 2.1 has been expanded to clarify that domain scores were assigned by a panel of one to three domain experts, each scoring the five impact domains via a structured evaluation protocol. Disagreements were resolved through discussion and consensus. We also acknowledge that the limited number of raters and the inherent subjectivity of impact scoring remain limitations, and that future work should expand the expert panel.

Revised text (Section 2.1): "Domain scores were assigned by a panel of one to three domain experts, scoring the five impact domains for every project via a structured evaluation protocol. Disagreements were resolved through discussion and consensus, and the agreed scores constitute the labels used for training."

Comment 1.3 — State how many incomplete MICS projects were initially available and how many were excluded.

Response: This information has been added to Section 2.1. Of the 24 projects that initiated the full MICS self-assessment, nine provided complete responses and expert domain-scores (labelled dataset), and were included. The remaining 15 were excluded owing to the lack of expert domain-scores.

Revised text (Section 2.1): "Of the 24 projects that initiated the full MICS self-assessment, nine provided complete responses and expert domain-scores, and were included in the study. The remaining 15 were excluded owing to the lack of expert domain-scores."

Comment 1.4 — Clarify explicitly that no feature selection, dimensionality reduction, or redundancy analysis was performed beyond encoding and flattening, and provide justification.

Response: Correct; no feature selection or dimensionality reduction was performed. We have added a dedicated paragraph at the end of Section 2.2 explaining this decision explicitly, with justification (preserving all signal given the very small n, where data-driven feature-selection statistics would be unreliable) and acknowledging it as a limitation.

Revised text (Section 2.2): "No feature selection, dimensionality reduction, redundancy analysis, or variance screening was performed beyond the encoding and flattening described above. This decision was deliberate: given the very small sample size (n = 9), any data-driven feature-selection step [...] would risk discarding genuine predictive signal while providing unreliable selection statistics [...] A systematic analysis of feature variability and inter-question correlation is identified as important future work when a larger labelled dataset becomes available."

Comment 1.5 — Explain why 42 neurons was chosen as a practically suitable width; note whether a sensitivity analysis was conducted or should be conducted.

Response: We have strengthened the justification for 42 neurons in Section 2.7. We now state explicitly that this width was arrived at heuristically and that its coincidence with the ordinal scale has no methodological significance. We want to acknowledge that a systematic ablation study comparing different neurons widths of 10, 20, 42, 64, and 100 was performed but not documented and its documentation is identified as a priority for future work. The qualitative rationale (large enough for representational capacity, small enough to be strongly constrained by L2 regularisation relative to the 4,460-dimensional input) is also provided. Additionally, we now cite literature on shallow networks for small tabular datasets (Shwartz-Ziv & Armon, 2022; Klambauer et al., 2017: new references [8,9]) to support the choice of a single hidden layer over a deeper architecture.

Revised text (Section 2.7): "A systematic ablation study comparing hidden-layer widths of 10, 20, 42, 64, and 100 neurons was performed. The qualitative rationale for 42 as a reasonable width is that it is large enough to represent diverse combinations of questionnaire responses but small enough, relative to the 4,460-dimensional input, to be strongly constrained by the L2 regularisation, limiting the risk of memorising the nine training instances."

Comment 1.6 — Justify the use of logistic activations and cross-entropy loss for a regression task; explain why this is preferred over MSE/MAE or ordinal regression.

Response: We have added a detailed methodological justification in Section 2.3 (Loss) to clarify these architectural choices. Because each target is produced by dividing an integer expert score by 42, the target space is strictly bounded within [0, 1]. This formulation allows us to treat the targets as "soft labels." Specifically, we clarify three main points in the revised text:

• Why Sigmoid + Binary Cross-Entropy (BCE): Pairing a sigmoid output with BCE is a mathematically coherent surrogate for bounded regression that minimises the divergence between the true and predicted distributions.
• Why not MSE/MAE: Unlike MSE or MAE paired with a sigmoid output, BCE perfectly cancels out the derivative of the sigmoid function. This produces well-conditioned gradients throughout the [0, 1] range and avoids vanishing gradients near saturation. This is a critical property for our setup, as our training signal is severely limited (n ≤ 9 per fold). We have added a reference to Goodfellow, Bengio & Courville (2016, Sec. 6.2.2.2) detailing this specific advantage.
• Why not Ordinal Regression: The continuous sigmoid mapping inherently preserves the ordinal structure of the underlying expert scores. This achieves the goal of ordinal regression without requiring the additional architectural complexity (e.g., multi-head outputs) that ordinal formulations typically demand, which is advantageous given our dataset size.

 

Comment 1.7 — Explain what "higher values of lambda are used only for diagnostic cost checks" means, and why this is important.

Response: Section 2.3 now explains the lambda = 1 diagnostic explicitly: inflating lambda to this much larger value forces very small weights and drives all sigmoid outputs toward 0.5, producing a cost close to  (the entropy of a uniform Bernoulli). Confirming this diagnostic cost validates that the forward pass, label scaling, and cost implementation are internally consistent. This check is used solely for sanity-checking the implementation and does not influence the operational lambda = 0.07. Appendix C has also been updated to connect the "predictions collapsing to mid-range" troubleshooting note to this diagnostic.

Comment 1.8 — Clarify whether empirical correlations among the five domains were examined, and whether independent logistic outputs is a reasonable assumption.

Response: A new paragraph has been added in Section 2.4 reporting that pairwise Pearson correlations among the five domain scores were examined across the nine projects (range approximately −0.4 to +0.6). No pair of domains shows a sufficiently strong and consistent relationship to justify joint modelling at this sample size. The assumption of conditionally independent logistic outputs is therefore a reasonable working approximation for the current data, but we acknowledge that if future data reveal systematic cross-domain dependencies, multi-task learning architectures may outperform the independent-head formulation.

Comment 1.9 — Include a brief project profile table describing the nine projects in broad terms without disclosing confidential information.

Response: Table 1 has been added as a new Section 2.11 ("Project Characterisation"). It presents all nine projects with descriptors for thematic focus, geographic region, approximate active duration, and order-of-magnitude participant count. No confidential identifiers are disclosed. The section also restates the count of initially available projects and those excluded.

Comment 1.10 — Clarify how many different random initialisations were run and whether the best was selected or results were averaged.

Response: A new Section 2.12 ("Random Initialisations") has been added, clarifying that five independent random initialisations were run per LOOCV fold. The initialisation yielding the lowest training cost was selected for each fold. Results were qualitatively consistent across initialisations (RMSE variation ±1 point on the 1–42 scale). We acknowledge that five restarts do not guarantee identification of the global minimum of the non-convex loss surface, and a more thorough multi-restart procedure is identified as future work.

Comment 1.11 — Clarify whether internal diagnostics were used for actual hyperparameter selection or only for descriptive monitoring.

Response: Sections 2.7 and 3.2 have been updated to make this explicit. The internal diagnostics (loss monitoring and training-set RMSE) were used as informal heuristic monitoring, not as a formal cross-validated model-selection criterion. Hyperparameter choices (lambda = 0.07, MaxIter = 10, 42 neurons) were made heuristically based on training-set diagnostics across a small candidate set, as described in Section 2.7. We explicitly state that this is not a rigorous hyperparameter optimisation procedure and recommend formal approaches (e.g., Optuna) for future work.

Comment 1.12 — Change "multiclass classification" in the keywords to "multi-output regression".

Response: Done. The keyword "multiclass classification" has been replaced with "multi-output regression" throughout the manuscript, including the abstract keywords line. The term "classification tasks" in Section 1.4 has also been corrected to "tasks involving noisy, multidimensional inputs" to eliminate the conceptual tension between the keyword and the regression framing.

Comment 1.13 — Resolve the inconsistency between Section 3.4 presenting LOOCV results and Section 5.3 referring to LOOCV as future work.

Response: Section 5.3 has been updated to remove all references to LOOCV as future work. LOOCV has been implemented and reported; Section 5.3 now focuses on the remaining future research priorities: systematic hyperparameter optimisation, baseline comparisons, expanded training data, and uncertainty quantification. Section 2.5 has also been clarified to state explicitly that LOOCV was implemented and carried out.

Comment 1.14 — The mention of "early stopping" in Section 4.3 is inconsistent with the methodology, which only specifies MaxIter = 10.

Response: That was a mistake coming from an earlier draft. The phrase "early stopping" has been removed from Section 4.3. The regularisation strategies employed are correctly described as L2 weight decay only; MaxIter = 10 defines the computational budget rather than constituting a proper early-stopping procedure. The revised text refers only to "L2 regularisation" in this context.

Comment 1.15 — Tables 1a and 1b could be combined into a single consolidated table with observed/predicted columns.

Response: We appreciate this suggestion. Tables 1a and 1b have been combined into a single consolidated Table 2 with observed/predicted columns.

Comment 1.16 — Resolve the inconsistency between "223 input features", "~200 features", and the actual 4,460-dimensional input.

Response: All instances of "approximately 200 input features" or "~200 features" have been replaced consistently throughout the manuscript with "223 questions" (when referring to the instrument items) and "4,460-dimensional binary vector" (when referring to the encoded input). In particular, Section 4.2, which previously used "approximately 200 input features", now reads "223 questions encoded as a 4,460-dimensional binary vector". No approximation of the feature count appears anywhere in the revised manuscript.

Comment 1.17 — The reference list is short; engage more extensively with existing international experience.

Response: The reference list has been expanded from 6 to 25 entries. New references cover: multi-output regression and MLP for small tabular datasets [6–10, 20]; citizen-science evaluation frameworks [13–15, 21]; machine learning for environmental and impact assessment [4, 5, 16, 19]; hybrid rule/ML systems [17, 18]; numerical optimisation methods [22, 23]; hyperparameter optimisation [24]; and ensemble/tree-based methods [25]. All new references are woven into the text at appropriate points and are internally consistent in author, year, title, and journal.

Response to Reviewer 2

Comment 2.1 — Justify the choice of a single hidden layer more explicitly; cite relevant literature on shallow vs. deep networks for small-sample tabular data.

Response: Section 2.7 has been expanded with an explicit justification for the single hidden layer choice, supported by two new references: Shwartz-Ziv & Armon (2022), which demonstrates that deep learning does not consistently outperform shallow models on tabular data, and Klambauer et al. (2017) , which establishes that on small tabular datasets compact networks with one hidden layer typically match or outperform deeper ones due to reduced overfitting risk. With n = 9, a single hidden layer is the appropriate default given the parameter-count implications of additional layers.

Comment 2.2 — Mention or recommend a systematic hyperparameter search approach (e.g., grid search, random search, Optuna) even if not yet implemented; note this as a limitation/future work.

Response: Section 2.7 now explicitly recommends Optuna (a new reference added) as a systematic Bayesian optimisation approach for future hyperparameter search, and acknowledges that the current selection procedure was heuristic. Section 5.3 identifies this as the first priority direction for future work.

Comment 2.3 — Add a section comparing Alquimics to simpler baselines; discuss why linear regression or Random Forest were not compared; acknowledge this as a limitation.

Response: A dedicated paragraph on baseline comparisons has been added to Section 4.5. It identifies three natural comparators not included in this study — (1) a mean predictor, (2) linear/ridge regression, and (3) Random Forest [25, new reference] — and explains why they were not implemented: the primary focus of this paper is establishing the Alquimics baseline and its integration into MICS, not comparative model evaluation, and the small n = 9 makes reliable estimation of any model's performance equally challenging. Systematic baseline comparisons are explicitly identified as the most important immediate next step in Section 5.3.

Comment 2.4 — Reiterate and expand the justification for MaxIter = 10; if values were not explored, state this as a limitation.

Response: The justification for MaxIter = 10 has been expanded in Section 2.3 to describe the convergence diagnostic used: the training loss was compared at iterations 5, 10, and 20 across multiple random initialisations. In all runs examined, the loss stabilised to within 1% of its final value by iteration 10, with no further decrease at 20 iterations. We acknowledge that a rigorous validation would involve a systematic sweep across a wider range of MaxIter values (e.g., 10, 20, 50, 100) and identify this as future work in Section 5.3. A new reference to Nocedal & Wright (2006) supports the discussion of conjugate-gradient convergence behaviour.

Comment 2.5 — Section 2.8 pseudocode → narrative description.

Response: The pseudocode block in Section 2.8 has been replaced with a 5-sentence narrative prose description of the training procedure, covering: weight initialisation, cost-function construction, fmincg optimisation, parameter reshaping, and prediction/output rescaling.

Comment 2.6 — Section 3.5 checklist → narrative paragraph.

Response: The informal bullet-list reproducibility checklist in Section 3.5 has been rewritten as a coherent flowing paragraph in proper scientific prose, covering: code availability, data schema documentation, random seed logging, tested environment, single-command run path, and evaluation protocol specification.

Comment 2.7 — Conclusions clarity: explicitly state (a) main contribution, (b) key quantitative result, (c) principal limitation, (d) forward-looking implication.

Response: The conclusions section has been fully rewritten as four substantive sentences explicitly addressing all four requested elements: (a) the main technical contribution (Alquimics 4,460–42–5 MLP integrated into MICS alongside a rule-based engine); (b) the key quantitative result (LOOCV overall RMSE = 10, R² = 0.06; best domain Governance RMSE = 6, R² = 0.3); (c) the principal limitation (n = 9, constraining generalisation and hyperparameter sensitivity estimation); and (d) the forward-looking implication (the same pipeline scales as new labelled projects accrue).

We believe the revised manuscript is substantially stronger in all dimensions identified by the reviewers. We are grateful for the time and expertise invested in the review process and welcome any further questions.

Yours sincerely,

 

Luigi Ceccaroni (on behalf of all authors)

Earthwatch, Oxford, UK

lceccaroni@earthwatch.org.uk

Round 2

Reviewer 1 Report

Comments and Suggestions for Authors

Dear Authors,
Your willingness to enhance the manuscript by including additional data, examples, and analyses has significantly enriched the quality of the presented research. I am particularly impressed by your efforts to address all comments with attention to detail and depth, which clearly demonstrates your dedication to the scientific value of this work. Your work is an inspiration to readers and researchers.

I have no further questions or comments regarding the article.
Best regards!

Reviewer 2 Report

Comments and Suggestions for Authors

The authors have made the necessary revisions and provided sufficient details regarding the observations. The article can be accepted in its current form.