Review Reports

Round 1

Reviewer 1 Report

Comments and Suggestions for Authors

This manuscript offers timely and practically relevant contribution on early wildfire detection. The greatest asset of the paper is the introduction of the PYRONEAR2025 dataset, which consists of both image and video data from multiple geographic regions and is embedded for early smoke plume detection and not as much for late stage fire detection. I also thank the attempts at comparing the proposed dataset to other public datasets and the evaluation of single frame and sequential models in a lightweight deployment scenario.

Not to say I don't believe the manuscript is ready for publication as is, though. The contribution is good but some problems with the methodology, reporting, and presentation must be resolved to give the work a fair assessment.

The first step is to make the description of the data set more precise and internally consistent. The abstract claims that they have approximately 150,000 manual annotations on 50,000 images, but the numbers differ in Table 1 for PYRONEAR2025 I and PYRONEAR2025 V. Relationships between the raw collected images, unique annotated images, annotations, videos, image subsets and the train/validation/test splits must be clearly explained by the authors. It is not always clear which numbers are related to annotations, images, frames, wildfire events or video sequences.

Second, it's necessary to improve the documentation of the annotation process. The use of five annotators per image and Krippendorff’s alpha is a good practice, but the manuscript only says that the agreement values were “good”. The unit of agreement (alpha), how disagreements are dealt with, the rules for bounding box acceptance, and the guidelines for annotations should be reported. The dataset is the main contribution of the paper, so the quality of the annotation should not be addressed in a cursory manner.

Thirdly, there is a need to improve the reproducibility. The authors write that code and data will be made available, but it should clearly be indicated what code and data are made available, on which basis, whether all video frames are available, whether third party data is available to be redistributed, how the splits used in the experiments can be reproduced, etc. Accessibility of data and repeatable experimental settings are of paramount importance for a benchmark paper.

Fourth, the experimental design is reasonably spirited but a few essentials are not included. The event level split of PYRONEAR2025 is suitable and there is also reference to some external datasets where this was not feasible. This introduces a potential leakage issue or comparability issue which needs to be thought through more carefully. More details should be provided about the perceptual hash procedure, such as the Hamming distance threshold and the number of images that were eliminated.

Fifth, there should be a better explanation of the selection of evaluation metrics. Precision, recall, and F1 score are helpful for an operational detection system, particularly when the primary goal is to provide early warning. The task remains to be object detection with Bounding Boxes though, so it would be nice to have at least one IoU based detection metric reported or a more precise explanation of why mAP is not suitable for this task. The manuscript should also define the count of true positives and false positives in the scenario that the ground truth smoke region does not fully overlap with the predicted region.

Sixth, the sequential model is interesting, but description still too light. The authors should state the window size and the sampling method, number of training sequences, class balance, LSTM hyper-parameters, optimization algorithm and its parameters, as well as a better description of the computation of the earlier detection time. The sequential model has an important advantage in terms of improved recall and reduced time to detection as suggested in Table 5; however, the evaluation protocol should be clearly stated so as to be easily replicable.

Seventh, there were some claims that need to be taken down or backed up. Claims like “the most diverse” or “surpasses existing datasets” could be correct, but must be backed up by a more careful comparison of the size, geographic coverage, type of annotations, public availability, and difficulty of the task provided by the dataset. The synthetic image ablation is also too short. It refers to a decrease of 2 per cent in F1, but the manuscript needs to explain whether this is significant or not, and whether it is consistent or not across splits or seeds.

8th: Most references are relevant; not fully current for the 2026 submission. The manuscript could be improved by adding recent work done on YOLO based wildfire or smoke detection, early smoke plume detection, video based detection and benchmark datasets from the year 2024, 2025, and 2026. It is also recommended that the authors verify if there are newer versions of any of the datasets or methods that they have cited.

Lastly, the manuscript must be carefully edited in language and formatting. There are several grammatical and typographical issues, such as “scrapping” instead of “scraping”, “wildifre”, “synthethic”, “bounding boxe”, and several awkward sentences. In addition there is “Figure ??” that could not be resolved and the header to the manuscript reads “submitted to Mathematics”, but it looks like it was submitted to Electronics. Such matters diminish the confidence in the final presentation and need to be corrected prior to publication.

This is generally a helpful and promising data paper but needs a good deal of work. I will reconsider it if the authors offer a more complete description of their data, validation of their annotations, details of their experiments that are more reproducible, more up-to-date references, and a well edited manuscript.

This manuscript offers timely and practically relevant contribution on early wildfire detection. The greatest asset of the paper is the introduction of the PYRONEAR2025 dataset, which consists of both image and video data from multiple geographic regions and is embedded for early smoke plume detection and not as much for late stage fire detection. I also thank the attempts at comparing the proposed dataset to other public datasets and the evaluation of single frame and sequential models in a lightweight deployment scenario.

Not to say I don't believe the manuscript is ready for publication as is, though. The contribution is good but some problems with the methodology, reporting, and presentation must be resolved to give the work a fair assessment.

The first step is to make the description of the data set more precise and internally consistent. The abstract claims that they have approximately 150,000 manual annotations on 50,000 images, but the numbers differ in Table 1 for PYRONEAR2025 I and PYRONEAR2025 V. Relationships between the raw collected images, unique annotated images, annotations, videos, image subsets and the train/validation/test splits must be clearly explained by the authors. It is not always clear which numbers are related to annotations, images, frames, wildfire events or video sequences.

Second, it's necessary to improve the documentation of the annotation process. The use of five annotators per image and Krippendorff’s alpha is a good practice, but the manuscript only says that the agreement values were “good”. The unit of agreement (alpha), how disagreements are dealt with, the rules for bounding box acceptance, and the guidelines for annotations should be reported. The dataset is the main contribution of the paper, so the quality of the annotation should not be addressed in a cursory manner.

Thirdly, there is a need to improve the reproducibility. The authors write that code and data will be made available, but it should clearly be indicated what code and data are made available, on which basis, whether all video frames are available, whether third party data is available to be redistributed, how the splits used in the experiments can be reproduced, etc. Accessibility of data and repeatable experimental settings are of paramount importance for a benchmark paper.

Fourth, the experimental design is reasonably spirited but a few essentials are not included. The event level split of PYRONEAR2025 is suitable and there is also reference to some external datasets where this was not feasible. This introduces a potential leakage issue or comparability issue which needs to be thought through more carefully. More details should be provided about the perceptual hash procedure, such as the Hamming distance threshold and the number of images that were eliminated.

Fifth, there should be a better explanation of the selection of evaluation metrics. Precision, recall, and F1 score are helpful for an operational detection system, particularly when the primary goal is to provide early warning. The task remains to be object detection with Bounding Boxes though, so it would be nice to have at least one IoU based detection metric reported or a more precise explanation of why mAP is not suitable for this task. The manuscript should also define the count of true positives and false positives in the scenario that the ground truth smoke region does not fully overlap with the predicted region.

Sixth, the sequential model is interesting, but description still too light. The authors should state the window size and the sampling method, number of training sequences, class balance, LSTM hyper-parameters, optimization algorithm and its parameters, as well as a better description of the computation of the earlier detection time. The sequential model has an important advantage in terms of improved recall and reduced time to detection as suggested in Table 5; however, the evaluation protocol should be clearly stated so as to be easily replicable.

Seventh, there were some claims that need to be taken down or backed up. Claims like “the most diverse” or “surpasses existing datasets” could be correct, but must be backed up by a more careful comparison of the size, geographic coverage, type of annotations, public availability, and difficulty of the task provided by the dataset. The synthetic image ablation is also too short. It refers to a decrease of 2 per cent in F1, but the manuscript needs to explain whether this is significant or not, and whether it is consistent or not across splits or seeds.

8th: Most references are relevant; not fully current for the 2026 submission. The manuscript could be improved by adding recent work done on YOLO based wildfire or smoke detection, early smoke plume detection, video based detection and benchmark datasets from the year 2024, 2025, and 2026. It is also recommended that the authors verify if there are newer versions of any of the datasets or methods that they have cited.

Lastly, the manuscript must be carefully edited in language and formatting. There are several grammatical and typographical issues, such as “scrapping” instead of “scraping”, “wildifre”, “synthethic”, “bounding boxe”, and several awkward sentences. In addition there is “Figure ??” that could not be resolved and the header to the manuscript reads “submitted to Mathematics”, but it looks like it was submitted to Electronics. Such matters diminish the confidence in the final presentation and need to be corrected prior to publication.

This is generally a helpful and promising data paper but needs a good deal of work. I will reconsider it if the authors offer a more complete description of their data, validation of their annotations, details of their experiments that are more reproducible, more up-to-date references, and a well edited manuscript.

Author Response

Please see the attachment.

Author Response File: Author Response.pdf

Reviewer 2 Report

Comments and Suggestions for Authors

This paper introduces PYRONEAR2025 dataset for early wildfire detection that surpasses existing datasets in size, geographic diversity, and complexity, and is the only one to include video sequences suitable for training sequential models. The dataset contribution merits publication, however the experimental evaluation requires significant revision.

My main concern is Section 3.1, where the authors explicitly acknowledge that for Nemo and SmokeFrames datasets it was not possible to ensure that the same wildfire event does not appear in both train and test splits simultaneously. This is a fundamental data leakage problem that directly inflates model performance on those datasets - models are effectively memorizing visual patterns from specific fire events seen during training rather than learning to generalize. The authors are aware of this, yet the cross-dataset comparison in Table 3 proceeds as if this were a minor technical note rather than a critical methodological flaw.

The consequence is that comparing F1 scores obtained under data leakage conditions (Nemo: 86.8%, SmokeFrames: 82.8%) against scores obtained under a correct event-level split (PYRONEAR2025: 68.3%) is not a meaningful comparison - this simply follows from the fact that the splits are not comparable. The apparent performance gap does not reflect genuine differences in dataset quality or model generalization. Furthermore, the authors do not provide sufficient detail about how event-level integrity was ensured for AiForMankind and Fuego, leaving it unclear whether those datasets are affected by the same issue. Given that this is the central methodological concern of the paper, splitting procedures should be explicitly described and justified for all datasets.

Since the authors cannot retroactively enforce event-level splits on existing datasets due to missing metadata, the cross-dataset comparison in Table 3 is difficult to interpret as it stands. A more appropriate alternative would be to take published models from the literature that were originally trained and evaluated on those datasets (e.g. SmokeyNet [13], Jeong et al. [19], Yazdi et al. [17]), if their code or pretrained weights are publicly available, and evaluate them directly on the PYRONEAR2025 test set under identical conditions. This would eliminate the data leakage problem entirely - all models would be evaluated on the same correctly split test set, and differences in performance would reflect genuine differences in generalization. Where pretrained weights or code are unavailable, this should be explicitly noted as a limitation. Even a partial comparison of this kind would be substantially more informative than the current design.

It is also worth noting that this reframing would strengthen rather than weaken the paper's contribution. The authors could make a more honest and compelling claim: that PYRONEAR2025 is the only publicly available dataset that enables methodologically sound evaluation of early wildfire detection models - and that alone would be a meaningful contribution.

Other minor issues are:
- Section 3.4 contains an unfilled reference and Appendix A is empty (some error probably in latex?)
- The sentence describing the dataset as containing both images and videos is repeated verbatim twice (lines 3-5 and 11-12)

Author Response

Thanks for the review! 

On splitting

There was a misunderstanding because of lack of details in the paper. We will reformulate what is written to make it clearer. When the sources of the events used in the datasets where similar (only valid for SmokeFrames and Nemo; there is no contamination in-between the datasets) but the names between the datasets where different, we ensured their independance, in particular in the test sets, through perceptual hash along with Hamming distance, which removed highly similar images (i.e., we remove the events from SmokeFrames test set that were in Nemo train set). The names in between a single dataset allow for event-level splits. In this context, the comparison between the datasets stands, as Pyronear events are different than the ones from Nemo and SmokeFrames. 

We released the official splits with our code and data. 

Other minor issues

Thanks for checking the typos, we will take this into account in the revised version.  

Reviewer 3 Report

Comments and Suggestions for Authors

A timely and interesting paper which is generally well-written, illustrated and logically ordered. The approach appears to be quite novel and current with potential to advance fire detection monitoring. The context is also well presented - both from the point of view of the literature and also the detail re: datasets etc. A few minor things - English is mostly fine except in a few places - a minor final check needed by a proof-reader/native English speaker. The illustrations are generally very informative and clear. Figure 8 - perhaps some more caption annotation to explain what each shows and comparison? Figure 9 is fine - but seems a little oddly placed with little explanation?

Comments on the Quality of English Language

A few minor things - English is mostly fine except in a few places - a minor final check needed by a proof-reader/native English speaker. 

Author Response

Thanks for the review! 

English comment:  

The English typos and formulations (such as “scrapping” instead of “scraping”, “wildifre”, “synthethic”, “bounding boxe”) will be fixed.  

Captions comment:

We will improve Figure 8 and Figure 9 captions:

Figure 8: "Comparison of the predictions of a single-frame model versus a video model on a set of videos from FigLib. In green are the good predictions and in red are the wrong predictions, with respect to time. Frame 0 is the start of the wildfire."

Figure 9: "Installation of one of our early wildfire detection stations in Chile on a watchtower of the Corporación Nacional Forestal (National Park Office; San Javier, Region of Maule). The horizon is of 80 km with clean weather."

Round 2

Reviewer 1 Report

Comments and Suggestions for Authors I recommend acceptance. A few minor points follow, but are not necessary experiments to run; they serve to aid clarity. 1- Dataset-size figures. It would be helpful to briefly explain how the annotations, the 24,000 unique images that have been validated, and the final image counts in Table 1 are related to each other to make it easy for readers to follow the composition of the released dataset. 2- Inter-annotator agreement. The α = 0.66 (Krippendorff) is reported as "strong"; consider using the term "moderate-to-substantial" or justifying the term, since this range is considered acceptable, though not “strong”. 3- Minor count inconsistencies. The synthetic (200 vs 287) and web-scraped (442 vs 87) image counts differ between Sections 2.2.1 and 2.2.3. This would be cleared up with a brief explanation (such as subsets included in final dataset). 4- Section 4 wording. The sentence "...which aims to semi-annotation process" seems to be incomplete and a few mentions of "PYRONEAR2025" in this section appear to be for "PYRONEAR2026. Please correct.
5- Appendix formatting. There are no entries in Appendix A and the "Appendix B Appendix" is formatting; please check. 6- Unit consistency. Time to detection needs to be harmonized throughout the text (e.g., 35 seconds), Table 5 (minutes) and Appendix Table A1. 7- Figures. The small axis labels and axis legends in Figures 2, 5-8 could be increased in size to be more readable. 8- Minor language editing. Simple grammatical problems (e.g., "requiring on heavy computations", "is strongly link to", "in-real-life") would be better corrected with a light language pass. Overall this is a timely, well done, and effective paper; it will be of interest to the readership. I wish to congratulate the authors on a quality and practical book.

Author Response

Thanks for the update and the review process! 

We took all the comments below into account in our new version. 

Reviewer 2 Report

Comments and Suggestions for Authors

Thank you for the clarification regarding the splitting strategy. After re-reading Section 3.1 more carefully, I can see that the event-level split issue is addressed in the text, though the phrasing could be made more explicit. Specifically, the sentence "This was not possible for datasets like Nemo or SmokeFrames, as the files were named with different names, from the same wildfire but different perspectives" could be misread as implying that event-level splits were not possible within those datasets, rather than only across them. I would suggest rephrasing to make it unambiguous that event-level splits were successfully applied within each individual dataset, and that the naming issue only concerned the cross-dataset overlap between Nemo and SmokeFrames. With these minor clarifications added to the text, I am satisfied that the methodological concerns have been addressed and recommend acceptance.

Author Response

Thanks for the update and the review process! We changed the wording so now it is clearer.