Validating Foundation Models for Automated Cattle Detection

Round 1

Reviewer 1 Report

Comments and Suggestions for Authors

The manuscript addresses an important and practically relevant problem in precision livestock farming, namely reducing the manual annotation burden for cattle detection using foundation models. The EMA dataset and the comparison between manual annotations and SAM 3-generated annotations have potential value. However, the current manuscript overstates the generality of the findings, and several methodological details require clarification or additional analysis before the conclusions can be fully supported.

1. In the Abstract, especially lines 4–16, the statement that automated annotation can “reliably replace manual labeling” is too strong. The current experiments only demonstrate that SAM 3-generated masks/OBBs can be useful for training cattle detectors under the tested barn-camera conditions. The authors should revise the abstract to avoid implying that all manual annotation can be replaced, and should state more precisely that the study evaluates automated annotation for cattle detection only. In lines 15–16 and lines 69–73, the manuscript states that foundation models can be “trusted” and can enable scalable monitoring pipelines. This conclusion is broader than the presented evidence, because the experiments evaluate detection only and do not validate tracking, re-identification, behavior analysis, or a complete cattle monitoring pipeline. The authors should narrow these claims and clearly state that tracking, re-identification, and behavior analysis are downstream applications that remain to be validated. In lines 543–566, the conclusion that only 5–10% of automatically annotated data is sufficient should be softened. This finding is based on one dataset and may depend on the sampled images. The authors should phrase this conclusion as applying to the present dataset and should avoid generalizing it to new farms or unseen camera conditions without further validation.
2. In lines 56–67 and the contribution list in lines 75–90, the manuscript does not clearly distinguish between the full EMA annotation set and the SAM 3-generated annotations. EMA contains oriented bounding boxes, head orientation, posture labels, and visibility status, whereas SAM 3 only generates segmentation masks that are converted into OBBs. The authors should explicitly state that the automated annotation evaluated in this paper refers only to cow detection masks/OBBs, not head orientation, posture, visibility, identity, or behavior labels.
3. Section 2.1, especially lines 126–202, is too lengthy and reads more like a broad dataset survey than a focused related work section. The authors should shorten the long dataset descriptions and add a concise comparison table summarizing the most relevant aspects, such as public availability, viewpoint, annotation type, OBB availability, detection suitability, and whether automated/foundation-model annotation is involved.
4. In Section 3.1, lines 242–260, the image acquisition and exclusion process should be clarified. The manuscript states that 6,296 images were suitable for annotation, while later subsets are named EMA_6295 and SAM_6295. The authors should explain this discrepancy and state whether one image was removed, corrupted, empty, or excluded for another reason.
5. In Section 3.2, lines 261–288, the manual annotation process appears to rely on a single annotator. Since the main evaluation compares SAM 3 annotations against manual annotations, the reliability of the manual reference labels is critical. The authors should add an inter-annotator agreement analysis on a subset of images or include expert verification. If this is not possible, the manuscript should describe the annotations as “manual reference annotations” rather than unquestioned “ground truth”, and the limitation should be clearly discussed.
6. In lines 293–335 and Table 1, the relationship among EMA_6295, EMA_1096, SAM_6295, and SAM_1096 is difficult to follow. The authors should add a clearer table or schematic diagram showing, for each subset, the source images, number of images, number of annotations, annotation type, visibility category, and experimental purpose. This would make the dataset organization and experimental design much easier to understand.
7. In Section 4.2, lines 387–394, the YOLO11-OBB training setup is not sufficiently reproducible. The authors should specify the exact YOLO11-OBB variant, pretrained checkpoint, optimizer/default settings, random seed, hardware, training time, and whether early stopping was actually triggered. The current statement that training used up to 100 epochs with early stopping patience of 100 is confusing, because the patience value equals the maximum epoch number.
8. In Section 5.1.1, lines 414–438 and Table 2, the SAM 3 confidence and MRA threshold selection procedure should be clarified. It is not clear whether these parameters were selected on a validation set or on the same data used for final evaluation. The authors should clearly separate threshold tuning from final testing. If the same data were used for both, this should be corrected or discussed as a limitation.
9. In Section 5.2, lines 497–515 and Table 3, the comparison between YOLO models trained on manual and SAM-generated annotations is based on single reported values. Because the differences are small and YOLO training can vary with random seed, augmentation, and initialization, the authors should repeat the main experiments several times and report mean ± standard deviation. If repeated experiments are not feasible, the manuscript should state that the results come from a single run and discuss this limitation.
10. The figure captions should be carefully revised. In Figure 2, “green represent ground truth images” should be corrected to “green boxes represent ground truth annotations”. The phrase “little boxes” should be changed to “small boxes”. For Figures 3–5, the color meanings should be made clearer, either through figure legends or more precise captions.

Comments on the Quality of English Language

Several language and formatting errors should be corrected. For example, line 148 should read “The MultiCamCows2024 dataset was introduced...”; lines 156–158 should read “our dataset complements the MultiCamCows2024 dataset”; line 171 should be “206,100 frames”; line 180 contains a serious formatting error (“AuthValidating Foundation Models for Automated Cattle Detectionors”); lines 182–183 should read “entrance area”; lines 320–321 should read “the latter contains”; and line 614 should read “eliminating”.

Author Response

Please see the attachment.

Author Response File: Author Response.pdf

Reviewer 2 Report

Comments and Suggestions for Authors

The manuscript addresses an important and timely problem: reducing the manual annotation burden in computer-vision-based cattle monitoring. The proposed EMA dataset, the comparison between manual and SAM 3-generated annotations, and the subsequent YOLO11-OBB training experiments are potentially valuable for precision livestock farming. The work is practically relevant, and the reported performance suggests that foundation-model-assisted annotation may substantially reduce annotation costs.

However, I recommend a major revision before the manuscript can be considered. My main concern: the conclusions are currently stronger than the experimental design supports.

I summarized my remarks and notices in 9 points below:

1. The manuscript repeatedly suggests that automated annotation can “replace” manual annotation and that foundation models can be “trusted” for cattle detection pipelines. These claims are too broad. The experiments are conducted in a single-farm, single-barn environment over a two-week data-collection period, using selected cameras and images with sufficient illumination. Therefore, the results support the feasibility of automated annotation under the tested conditions. Still, they do not yet demonstrate general robustness across farms, breeds, seasons, camera positions, night-time conditions, dirty lenses, different barn architectures, or more severe occlusions. The authors should substantially soften these claims or add external validation.
2. The validation design requires clarification and improvement. The ground truth appears to have been produced by a single annotator. This is a serious limitation for a study whose central claim is the reliability of automated annotations. A second expert should independently annotate a subset of the dataset, and inter-annotator agreement should be reported for oriented bounding boxes, visibility status, posture labels, and head orientation. Without this, it is difficult to determine whether disagreements between SAM 3 and manual annotations reflect model error or annotation ambiguity.
3. The manuscript must clarify possible data leakage in the experiment using the larger SAM_6295 dataset. Since SAM_1096 is described as a subset of SAM_6295, it is essential to state explicitly whether the EMA_1096 test images were excluded from SAM_6295 during training. If the test images or temporally adjacent frames highly similar to them were included in the training set, the reported performance for the SAM_6295-trained model is invalid. This experiment should be repeated using a strictly leakage-free split, preferably separated by camera, date, or time block.
4. The matching criterion combining IoU and IoM needs more careful justification. The IoM criterion is useful when SAM produces tighter boxes. Still, it can also be overly permissive, especially when one predicted box covers two animals or when partial containment is counted as a correct detection. The authors should report results using standard IoU-only criteria in addition to the combined IoU/IoM criterion. A sensitivity analysis for IoM thresholds and a detailed error analysis of merged cows, split detections, partial animals, and severe occlusions would strengthen the study.
5. The methodological description is not sufficiently reproducible. The authors should provide the exact SAM 3 version, checkpoint, implementation source, prompt settings, confidence calculation, mask-to-OBB conversion procedure, filtering rules, hardware and software environments, GPU type, training hyperparameters, augmentation settings, optimizer details, random seeds, and the number of repeated training runs. Stating that the default Ultralytics configuration was used is not enough for a reproducible Sensors paper.
6. The manuscript should better separate the dataset contribution from the model validation. The EMA dataset includes head orientation, posture labels, and visibility status, but the experiments mainly evaluate cattle detection. If posture, visibility, and head orientation are presented as important contributions, their annotation quality and potential use should be more explicitly evaluated or discussed. Otherwise, the paper should clearly state that the current experiments validate only detection.
7. The results should include confidence intervals or standard deviations from repeated runs. A single training run per setting is not sufficient to support strong claims about comparable performance between manual and automated annotations. The authors should report variability across multiple runs and, where possible, statistical comparisons between models trained on manual and SAM-generated labels.
8. The dataset description contains inconsistencies. It alternates between 6,295 and 6,296 images, and the annotation-time statement is ambiguous regarding whether the reported 250 hours refers only to EMA_6295 or also includes EMA_1096. The distinction between EMA_6295, EMA_1096, SAM_6295, and SAM_1096 should be made clearer, especially because only some subsets are directly comparable.
9. The figures and tables are generally useful, but several visualizations should be improved. Figures showing bounding boxes are dark and visually dense, and the color coding may be difficult to interpret. Captions should be more precise; for example, “ground truth images” should be corrected to “ground truth annotations.” Tables should include clearer explanations of evaluation sets and whether training and test images are fully independent.

Overall, this is a promising, practically relevant manuscript, but the experimental design, reproducibility, validation strategy, and the strength of the conclusions must be improved before publication.

Comments on the Quality of English Language

The English is understandable, but several grammatical and typographical errors remain. Examples include “Sensorsfor,” “our datasets complements,” “the later,” “entrance are,” “elliminating,” and awkward phrases such as “little boxes.” The manuscript would benefit from thorough language editing.

Author Response

Please see the attachment.

Author Response File: Author Response.pdf

Reviewer 3 Report

Comments and Suggestions for Authors

The manuscript presents an interesting contribution by validating the use of the foundation model SAM 3 for automated annotation in cattle detection, demonstrating that training on these annotations yields competitive performance compared to manual labeling. While the work is timely and relevant for scaling livestock monitoring systems, several important issues require clarification and deeper analysis to strengthen the scientific rigor and practical relevance.

Questions

1. The EMA dataset originates from a single farm over a limited two-week period. How well would SAM 3 annotations and the YOLO11-OBB detectors trained on this data generalize to other farms with different camera setups, cattle breeds, lighting conditions, and barn layouts? Including cross-site validation or at least discussion on this aspect is essential.

2. The manuscript acknowledges challenges arising from severe occlusions, lighting extremes, and partial visibility. How do these factors affect the quality of SAM 3’s automatic annotations and the downstream detector performance? A quantitative or qualitative evaluation focusing on these difficult scenarios would provide valuable insight.

3. While the dataset and models cover head orientation and standing versus lying posture, other important behaviors such as feeding, drinking, and lameness indicators remain unaddressed. What are the authors’ plans or prospects for extending behavioral annotations and for automating their extraction? This would significantly enhance the usefulness of the system for precision livestock farming.

4. The “manual” ground truth annotations were produced by a single expert annotator. Without inter-annotator agreement analysis, it is difficult to estimate annotation consistency and thus the upper bound of detector performance. Could the authors comment on this limitation or provide measures of annotation reliability?

5. Although SAM 3 generates high-quality annotations, it demands substantial GPU memory and processing time, limiting real-time applicability. Could the authors elaborate on trade-offs in real-world deployments, including hardware requirements, annotation throughput, and how often retraining or re-annotation may be needed in dynamic farm environments?

6. Have you considered including Average Recall (AR) alongside mAP to better capture how completely objects are detected, especially in crowded scenes where coverage might be challenging?

7. Since SAM 3 generates segmentation masks, did you evaluate annotation quality using pixel-level metrics like Boundary F1 Score or Mask IoU, assuming manual masks might be available? Such metrics could provide a more detailed assessment than bounding boxes alone.

8. Please consider including a detailed discussion and quantitative comparison of your approach and results with relevant recent studies in automated or semi-automated livestock monitoring, such as:

https://doi.org/10.5187/jast.2500286
https://doi.org/10.1016/j.atech.2025.100770
https://doi.org/10.1016/j.atech.2025.101054

Author Response

Please see the attachment.

Author Response File: Author Response.pdf

Round 2

Reviewer 1 Report

Comments and Suggestions for Authors

The authors have provided point-by-point responses and thorough revisions to all the issues I raised in the previous round. The quality of the revised manuscript has been markedly improved. In my opinion, the current version meets the journal's publication standards, and I recommend acceptance.

Reviewer 2 Report

Comments and Suggestions for Authors

I would like to thank the authors for their thorough work and for incorporating my suggestions into the manuscript. In its current form, I accept the manuscript.

Reviewer 3 Report

Comments and Suggestions for Authors

The authors have addressed most of the reviewers' comments and suggestions satisfactorily. The revised manuscript has been improved accordingly. Therefore, I have a favorable opinion regarding the publication of this paper in Sensors (MDPI).