RF-GoatDet: An Occlusion-Aware Instance Segmentation Framework for Dairy Goats with an Adaptive Receptive Field Encoder

Abstract

Accurate instance segmentation is essential for precision livestock farming, as it supports individual tracking, posture analysis, body-shape measurement, and other downstream visual monitoring tasks. However, dairy goat segmentation in dense barn scenes remains challenging because frequent mutual occlusion, instance adhesion, partial visibility, and non-rigid posture variation often lead to incomplete masks and ambiguous instance boundaries. To address these challenges, this study develops RF-GoatDet, a real-time instance segmentation framework for dairy goats built upon RT-DETR. The main component of the proposed framework is an Adaptive Receptive Field Encoder (ARFE), which enhances feature encoding by adapting the effective receptive field to irregular goat contours, scale variation, and partially visible body regions. In addition, Coordinate Attention is introduced to strengthen direction-aware spatial representation, while a Query-Conditioned Dynamic Mask Head is used to generate instance-specific masks and improve the separation of adjacent goats. A dairy goat instance segmentation dataset containing 2288 annotated images was constructed, and a two-stage cleaning procedure was applied to reduce redundancy and visual anomalies. Experimental results show that RF-GoatDet achieves 46.5% mask AP and 62 FPS on this dataset, improving the RT-DETR baseline by 4.2 percentage points in mask AP while maintaining real-time inference. These results demonstrate that the proposed ARFE-centered framework effectively improves mask quality and instance discrimination in dense dairy goat scenes, providing a robust and efficient solution for real-time visual monitoring in precision livestock farming.

1. Introduction

Precision livestock farming (PLF) has become an important direction in modern animal husbandry because it supports continuous and data-driven management of animal health, welfare, and production efficiency. Berckmans [1] described the core idea of PLF as a shift from periodic manual observation to real-time and individual-level monitoring supported by sensors and intelligent analysis. In this context, computer vision and deep learning have been increasingly used for livestock monitoring tasks such as behavior recognition, individual identification, body measurement, and welfare assessment, because they provide non-contact and scalable observation in practical farm environments [2,3,4]. Recent reviews on livestock biometrics and body measurement further indicate that image-based analysis has become a promising tool for individual recognition, farm automation, precision breeding, and phenotypic assessment [3,4].

Among vision-based perception tasks, instance segmentation is particularly important for precision livestock farming. Compared with object detection, which only provides bounding boxes, instance segmentation can delineate each animal at the pixel level. This fine-grained representation is more suitable for downstream tasks such as posture analysis, gait assessment, body-shape measurement, and individual tracking in crowded enclosures. Mask R-CNN established a representative instance segmentation framework and demonstrated the practical value of mask-level prediction over box-only detection [5]. In livestock applications, Qiao et al. [6] showed that cattle segmentation and contour extraction based on Mask R-CNN can support precision livestock analysis. Zhang et al. [7] developed an interactive dairy goat image segmentation method and showed that fine-grained goat segmentation is useful for annotation efficiency and downstream precision-farming applications. In video-based livestock monitoring, one-shot cattle video segmentation has also been shown to provide useful visual support for phenotype analysis and practical monitoring [8]. In addition, multi-scene cattle segmentation based on improved semantic segmentation further confirms the importance of accurate animal-region delineation in livestock management [9].

For commercial dairy-goat management, accurate individual-level masks are not only an intermediate computer-vision output but also a structural input for practical decision support. A complete and correctly separated goat mask can be used to estimate body contour, body length, back posture, visible area, animal location, and animal-to-animal contact. These cues are closely related to individual health monitoring, feeding behavior assessment, welfare evaluation, automatic counting, and long-term tracking in crowded pens. Therefore, improving instance segmentation under occlusion has direct practical relevance for precision livestock interventions rather than serving only as a generic segmentation benchmark.

Despite these advances, accurate instance segmentation of dairy goats remains challenging in dense barn environments. In intensive breeding scenes, goats often appear close to each other, which causes frequent mutual occlusion and severe instance adhesion. At the same time, dairy goats show substantial non-rigid posture variation during standing, turning, feeding, and lying. These behaviors lead to irregular contours, truncated limbs, and partially visible body regions. As a result, dense dairy goat segmentation is more difficult than segmentation on standard scenes with clearer foreground-background separation. Similar difficulties have also been reported in other livestock vision tasks. For example, goat-face detection and cattle behavior detection studies have shown that small targets, cluttered backgrounds, weather variation, and inter-object overlap can reduce model robustness in farm scenes [10,11]. An Applied Sciences study on cattle behavior analysis also noted that complex real-farm conditions remain challenging even when lightweight and real-time detection models are used [12].

One key limitation of many existing segmentation pipelines is that their feature extraction process still relies heavily on fixed-grid convolution. Although fixed-grid convolution is effective in many generic vision tasks, it has limited flexibility when objects undergo large geometric deformation or become only partially visible. Deformable ConvNets V2 learn adaptive sampling offsets and modulation weights, allowing convolutional sampling locations to better match object structures under spatial deformation [13]. This idea is highly relevant to dairy goat instance segmentation, because goats in crowded pens often present curved backs, elongated bodies, occluded limbs, and incomplete visible regions. Therefore, adaptive receptive-field modeling is needed to improve feature encoding for irregular and partially visible goat structures.

In addition to feature extraction, separating adjacent individuals is another major challenge in crowded dairy goat scenes. Traditional detection-and-segmentation frameworks often rely on heuristic post-processing to suppress duplicated predictions and refine masks. However, such strategies may become unstable when multiple goats overlap heavily and share similar appearance. Transformer-based query learning provides a promising alternative by representing instances through learnable object queries. Cheng et al. [14] showed that query-conditioned mask prediction can provide a unified framework for image segmentation, while Zhao et al. [15] demonstrated that end-to-end transformer detectors can achieve competitive real-time detection performance with global context modeling. However, these general-purpose methods are not specifically designed for dense livestock scenes, where neighboring animals have similar texture, fragmented visible regions, and ambiguous boundaries caused by fences, feeding troughs, shadows, and crowding. Therefore, directly transferring generic segmentation architectures to dairy goat scenes may still lead to incomplete masks, boundary confusion, or instance adhesion.

Practical livestock monitoring systems also require a favorable balance between segmentation accuracy and inference efficiency. Farm-side monitoring is often continuous and may need to operate under limited computational resources. Therefore, a useful dairy goat segmentation model should not only produce accurate masks but also maintain real-time inference speed. Although recent livestock studies have pursued lightweight or real-time visual models, many of them focus on detection, semantic segmentation, behavior recognition, or annotation support rather than dense instance segmentation under severe occlusion [7,10,11,12]. This leaves a clear need for a real-time instance segmentation framework tailored to dense dairy goat scenes.

To address these challenges, this study develops RF-GoatDet, an occlusion-aware real-time instance segmentation framework for dairy goats built upon RT-DETR. The proposed framework focuses on task-specific adaptation for dense dairy goat segmentation by strengthening receptive-field flexibility, spatial localization, and query-conditioned mask generation. Specifically, Coordinate Attention (CA) is incorporated into the ResNet-50 backbone to enhance direction-aware spatial representation [16]. An Adaptive Receptive Field Encoder (ARFE) is introduced to align feature encoding with irregular goat contours, scale variation, and partially visible body regions. In addition, a Query-Conditioned Dynamic Mask Head is designed to generate instance-specific masks and improve the separation of adjacent goats during mask prediction. Through these components, RF-GoatDet aims to improve both mask quality and instance discrimination in scenes characterized by mutual occlusion, instance adhesion, and non-rigid posture variation.

In addition, we construct and annotate a dairy goat instance segmentation dataset containing 2288 images collected under different illumination conditions, herd densities, and behavioral states. A two-stage cleaning procedure is applied to reduce redundant frames and visual anomalies, thereby improving dataset reliability. Extensive experiments on the constructed dataset show that RF-GoatDet improves mask AP over the RT-DETR baseline while maintaining real-time inference capability. The main contributions of this work are summarized as follows:

• We develop RF-GoatDet, an occlusion-aware real-time instance segmentation framework for dairy goats in dense barn environments. The framework is designed to address mutual occlusion, instance adhesion, partial visibility, and non-rigid posture variation.
• We propose an Adaptive Receptive Field Encoder (ARFE) to improve feature encoding for irregular and partially visible goat structures. By adaptively aligning the effective receptive field with visible animal regions, ARFE enhances the representation of scale-varying and deformed goat bodies.
• We integrate Coordinate Attention and a Query-Conditioned Dynamic Mask Head into the ARFE-centered framework. These components strengthen direction-aware spatial localization and instance-specific mask generation, leading to better separation of adjacent goats in crowded scenes.
• We construct a dairy goat instance segmentation dataset and conduct ablation, comparative, training-dynamics, and qualitative analyses. The experimental results demonstrate that the proposed method achieves consistent gains over the RT-DETR baseline while preserving real-time inference performance.

2. Related Work

The research related to this study can be grouped into three main areas: livestock visual perception in precision farming, feature modeling for dense and irregular animal targets, and query-based real-time instance segmentation. This section reviews representative studies in these areas and clarifies the remaining gaps that motivate the proposed RF-GoatDet framework.

2.1. Livestock Visual Perception in Precision Farming

With the development of precision livestock farming, computer vision has become an important tool for animal monitoring, phenotyping, and intelligent farm management. Brocklehurst et al. reviewed quantitative frameworks for real-time livestock auto-monitoring and pointed out that image-based and sensor-based observation are increasingly important for continuous on-farm decision support. Their study also emphasized that practical livestock monitoring systems should consider sensing frequency, deployment feasibility, and the reliability of downstream prediction models. Adam et al. further surveyed deep learning approaches for livestock detection in UAV imagery and showed that recent livestock vision systems often rely on YOLO-, R-CNN-, and transformer-based pipelines. However, they also noted that balancing detection accuracy, model size, and inference speed remains a key challenge in real applications [17,18].

Beyond coarse object detection, segmentation and recognition have received increasing attention because many livestock-management tasks require fine-grained visual information. Bello et al. proposed a transfer-learning-based cattle instance segmentation framework in Agriculture and showed that instance segmentation can support livestock inventory and welfare monitoring more effectively than box-level recognition alone. Feng et al. introduced DBCA-Net for cattle face segmentation and recognition, demonstrating that complex farm scenes require stronger context modeling and multi-scale feature extraction for reliable segmentation and identification [19,20]. These studies show that segmentation provides richer structural information for livestock analysis, but they also indicate that background clutter, feature ambiguity, and animal similarity remain major difficulties in practical farm environments.

Goat-related visual analysis has shown similar challenges. Pu et al. developed an automatic recognition method for Chengdu ma goats and reported that dense target distribution and large scale variation increase the difficulty of robust recognition in barns. Guo et al. proposed a YOLOv4-based method for detecting and localizing Albas velvet goats, showing that lightweight goat localization can be achieved under practical breeding conditions. Fang et al. proposed CCS-YOLOv8 for livestock detection in complex grassland scenes and pointed out that densely clustered animals and complex backgrounds remain important obstacles for practical livestock detection [21,22,23]. Although these methods improve livestock detection and recognition, most of them still focus on object localization or identity analysis. They do not fully address dense instance-level mask prediction, where adjacent animals must be separated at the pixel level.

2.2. Dense Livestock Targets and Feature Modeling

Dense livestock scenes are difficult because neighboring animals often overlap, share similar appearance, and appear at different scales. Guo et al. proposed YOLO-SDD for single-class dense livestock detection and showed that detection robustness can be improved in crowded livestock scenarios involving broilers, geese, and sheep. This finding is relevant to intensive breeding environments, where the main difficulty is not only detecting animals but also distinguishing strongly adjacent individuals in crowded layouts [24].

In goat-related applications, high target density and scale variation also strongly affect visual perception. Pu et al. showed that dense goat distributions and large scale changes can reduce individual recognition performance in barns, while Guo et al. reported that effective goat localization depends on robust feature extraction under real farm conditions [21,22]. These observations suggest that standard feature extractors may be insufficient when animal bodies are partially visible, irregularly shaped, or mixed with background structures such as troughs, fences, shadows, and floor textures.

Recent livestock behavior recognition studies have further highlighted the importance of discriminative feature modeling in crowded farm scenes. Long et al. proposed FSCA-YOLO for multi-target dairy cow behavior recognition and showed that feature enhancement and attention mechanisms improve performance under high inter-cow similarity, nighttime conditions, and crowded outdoor scenes [25]. These results indicate that stronger feature representation is helpful when visual cues are weak, incomplete, or occluded. However, most livestock-oriented feature enhancement methods are still designed for detection or behavior recognition. They improve target sensitivity, but they do not directly solve dense instance segmentation, where contour recovery and adjacent-instance separation are both required.

For dairy goat instance segmentation, this limitation is especially important. Goats in dense barns often present curved backs, elongated bodies, truncated limbs, and locally visible regions caused by mutual occlusion. Fixed-grid convolution has limited flexibility in such cases because its sampling pattern is spatially regular and cannot adapt to irregular animal contours. Therefore, dense goat segmentation requires feature encoding that can better align the effective receptive field with visible body structures and suppress interference from cluttered backgrounds.

2.3. Query-Based and Real-Time Instance Segmentation

In general computer vision, query-based architectures have become an important direction for reducing hand-crafted post-processing and improving instance-level reasoning. Carion et al. proposed DETR and reformulated object detection as a direct set prediction problem with bipartite matching. This design reduces the dependence on anchor design and duplicate suppression through transformer-based decoding [26]. Following this idea, Sun et al. introduced Sparse R-CNN, which replaces dense candidates with a small set of learnable proposals and demonstrates the effectiveness of sparse object representations [27].

For instance segmentation, query-based design has been further extended to mask-aware prediction. Fang et al. proposed QueryInst and showed that dynamic mask heads can be naturally combined with query-based instance modeling to improve mask generation in end-to-end frameworks. He et al. later introduced FastInst as a simple query-based real-time instance segmentation framework, showing that query-based methods can also achieve strong efficiency under real-time constraints [28,29]. These studies indicate that query-based prediction is attractive for dense scenes because it provides instance-specific representations and reduces reliance on heuristic post-processing.

Efficient real-time perception has also been investigated in related dense-prediction tasks. Zhang et al. proposed FasterSal, a robust and real-time single-stream architecture for RGB-D salient object detection, demonstrating that detail-aware feature modeling and efficient single-stream design can improve dense visual prediction while maintaining real-time inference efficiency [30]. Although RGB-D salient object detection differs from livestock instance segmentation, FasterSal is relevant to this study because it highlights a similar computational principle: dense prediction models should enhance feature discrimination and boundary/detail awareness while controlling encoder–decoder complexity. This principle is consistent with the efficiency-oriented design of RF-GoatDet.

Nevertheless, most existing query-based segmentation frameworks are designed for generic visual benchmarks rather than livestock-specific crowded environments. In dense dairy goat scenes, neighboring individuals often have highly similar textures and shapes. Their visible body parts may also be fragmented by mutual occlusion, posture variation, and background interference. Under these conditions, directly applying a generic query-based segmentation model may still lead to mask adhesion, incomplete contours, or inaccurate boundary assignment. Therefore, query-based decoding should be combined with stronger spatial feature localization and more adaptive feature encoding when applied to dense livestock imagery.

2.4. Summary of Research Gaps

The above studies reveal several remaining gaps. First, existing livestock vision methods have made clear progress in detection, recognition, semantic segmentation, and behavior analysis, but relatively few studies focus on dense instance-level separation of goats under severe overlap [19,20,21,22]. Second, studies on crowded or complex farm scenes repeatedly show that target similarity, background clutter, scale variation, and partial visibility remain major barriers to robust performance, indicating the need for more adaptive and discriminative feature modeling [23,24,25]. Third, although query-based frameworks provide an effective route toward end-to-end and real-time instance reasoning, their application to dense livestock instance segmentation remains limited [26,27,28,29].

Motivated by these gaps, this study develops a real-time dairy goat instance segmentation framework that combines direction-aware spatial representation, adaptive receptive-field modeling, and query-conditioned mask prediction. The proposed design aims to improve mask quality and instance discrimination in dense barn environments with mutual occlusion, instance adhesion, partial visibility, and non-rigid posture variation.

3. Proposed Method

Dense dairy goat scenes in practical breeding environments are particularly challenging for instance segmentation. As illustrated in Figure 1, dairy goats often appear in close contact with each other, resulting in mutual occlusion, instance adhesion, and partial visibility. These factors make it difficult for a model to recover complete instance masks and to assign accurate boundaries between adjacent individuals. To address these problems, this study proposes RF-GoatDet, an occlusion-aware real-time instance segmentation framework for dairy goats.

3.1. Problem Formulation

Given an input image $I\in {\mathbb{R}}^{H\times W\times 3}$ containing multiple dairy goats, the goal of instance segmentation is to predict a set of instance-level outputs:

$$\mathcal{Y}={\left\{(c_i,b_i,m_i)\right\}}_{i=1}^N$$

(1)

where N denotes the number of detected instances, $c_i$ is the category label, $b_i$ is the bounding box, and $m_i\in {[0,1]}^{H\times W}$ is the predicted mask of the i-th goat. Since this study focuses on dairy goat instance segmentation, all foreground instances belong to the same semantic category. The main difficulty lies not in category classification, but in accurately separating adjacent goats and recovering visible body regions under occlusion and posture variation.

In dense barn scenes, three visual factors are especially important. First, mutual occlusion causes only part of a goat body to be visible. Second, instance adhesion blurs the boundary between adjacent goats with similar color and texture. Third, non-rigid posture variation changes the apparent shape of the target, leading to curved backs, extended limbs, and irregular contours. These factors require the model to have both adaptive feature encoding and instance-specific mask prediction.

3.2. Overview of RF-GoatDet Architecture

The overall architecture of RF-GoatDet is shown in Figure 2. Built upon RT-DETR [15], the framework combines a Coordinate Attention-augmented backbone, the proposed Adaptive Receptive Field Encoder (ARFE), a transformer decoder with learnable queries, and a Query-Conditioned Dynamic Mask Head for instance-specific mask prediction.

First, the input image is processed by a ResNet-50 backbone to generate multi-scale feature maps. To improve spatial localization in cluttered barn scenes, Coordinate Attention (CA) modules are inserted into the backbone. CA helps the network preserve direction-aware positional information, which is useful for distinguishing elongated goat bodies from fences, troughs, and floor textures.

The proposed framework consists of three stages: hierarchical feature extraction, adaptive feature encoding, and instance-aware prediction. First, the input image is processed by a ResNet-50 backbone [31] to generate multi-scale feature maps. ResNet-style backbones remain widely used in agricultural and livestock computer vision because they provide a favorable accuracy–efficiency trade-off and a mature transfer-learning basis, as noted in recent reviews of animal-farming vision systems and livestock phenotyping pipelines [32,33]. To improve feature localization in cluttered and crowded farm scenes, Coordinate Attention (CA) modules [16] are inserted into the residual blocks of the backbone.

Next, the multi-scale features are fed into the proposed Adaptive Receptive Field Encoder. ARFE is designed to improve the alignment between the effective receptive field and the irregular visible structure of dairy goats. By combining adaptive sampling and multi-scale context aggregation, the encoder strengthens feature representation under posture variation, scale change, and partial occlusion. The encoded features are then passed to a transformer decoder with N learnable object queries. Through self-attention and cross-attention, the decoder produces refined query embeddings that represent potential goat instances.

Finally, each refined query is delivered to three parallel prediction branches, including the class head, the box head, and the Query-Conditioned Dynamic Mask Head. In particular, the mask branch transforms each decoder query into an instance-specific dynamic kernel, which is then applied to the shared mask feature map to generate the final binary mask. This design allows different goat instances to be predicted with different mask parameters, which is important for separating adjacent animals in dense scenes.

3.3. Dataset Acquisition and Cleaning Pipeline

3.3.1. Data Acquisition

The dataset used in this study was collected from the experimental dairy goat facility at the Animal Husbandry Teaching and Experimental Base of Northwest A&F University (NWAFU), located in Yangling District, Xianyang City, Shaanxi Province, China (109.94° E, 34.82° N). The farm houses 134 lactating Saanen dairy goats ranging in age from 2 to 5 years. To improve the robustness of the proposed model, the data collection process covered different illumination conditions, herd densities, and behavioral states.

As shown in

Table 1. Dataset acquisition settings and dense-scene characteristics.

Item	Description
Farm location	Experimental dairy goat facility, Northwest A&F University, China.
Animal breed	Saanen dairy goats.
Number of animals	134 lactating dairy goats housed in an indoor barn environment.
Scene type	Indoor long-aisle barn with feeding fences, concrete floor, side wall, and fixed surveillance viewpoint.
Camera type	Fixed RGB surveillance camera installed above the goat pen.
Camera viewpoint	High-position oblique top-view perspective with an approximately 40° downward viewing angle.
Image coverage	The camera covered the feeding area, walking aisle, fence region, side wall, and partial standing/resting area of the goat pen.
Original image resolution	1920 × 1080 pixels.
Input preprocessing for training and inference	Multi-scale resizing from 480 to 800 pixels was used during training. During inference, all compared models were evaluated under the same input-size setting to ensure fairness.
Illumination condition	Mixed indoor illumination, including natural side light, weak barn illumination, shadowed regions, and uneven lighting caused by walls and fences.
Animal density in images	Most images contained multiple goats simultaneously, commonly ranging from 3 to 8 visible goats per frame.
Typical occlusion types	Body-body overlap, fence occlusion, partial body truncation, head-body adhesion, and incomplete visibility near image boundaries.
Posture variation	Standing, walking, feeding, turning, lowering head, lateral body exposure, and non-rigid body deformation.
Dense-scene definition	Images with at least 4 visible goats or with obvious body overlap/instance adhesion were regarded as dense-scene samples.
Dataset size after cleaning	2288 indoor dairy goat images after image quality filtering and annotation correction.
Main segmentation challenge	Separating adjacent goats with similar appearance under mutual occlusion, irregular body contours, and partial visibility.

, the dataset was collected from a fixed high-position surveillance viewpoint in an indoor barn. Such a setting produced typical dense-scene challenges, including multiple goats appearing in the same frame, fence occlusion, partial body truncation, similar white body appearance, and irregular contours caused by non-rigid posture variation. Therefore, the dataset is suitable for evaluating instance segmentation performance under dense livestock-monitoring conditions.

3.3.2. Data Cleaning and Quality Control

To ensure dataset reliability, a two-stage data cleaning and quality control strategy was adopted. In the first stage, a perceptual hashing (pHash) algorithm [34] was used to measure similarity between adjacent frames and remove redundant or near-duplicate samples generated during video sampling. After this automated filtering, manual inspection was further performed to remove samples with severe motion blur or missing target instances.

In the second stage, the CleanVision library [35] was used to identify and exclude samples containing visual anomalies. The anomaly categories considered during this process are summarized in

Table 2. Anomaly Categories and Their Definitions.

Anomaly Category	Characteristic Definition
Extreme Aspect Ratio	Samples exhibiting significant deviation from standard width-to-height proportions.
Exact Redundancy	Instances where identical pixel-level frames appear multiple times.
High Similarity	Images retaining high structural resemblance with negligible photometric variations.
Severe Blur	Frames degraded by defocusing or rapid subject motion artifacts.
Monochromatic	Single-channel images lacking necessary color information.
Content Insufficiency	Samples dominated by background void or lacking discernible target features.
Resolution Mismatch	Images deviating from the standardized spatial dimensions of the dataset.

. These categories include extreme aspect ratio, exact redundancy, high similarity, severe blur, monochromatic rendering, content insufficiency, and resolution mismatch. After this two-stage cleaning procedure, approximately 7% of the raw samples were discarded, resulting in a finalized dataset of 2288 indoor images. Ground-truth masks were annotated using LabelMe v5.5.0 [36], and only the visible body regions of each goat were delineated to avoid ambiguous supervision caused by heavy occlusion. The dataset was finally divided into training, validation, and test subsets with a ratio of 7:2:1.

To reduce temporal redundancy between data subsets, near-duplicate frames identified by perceptual hashing and manual review were removed before dataset splitting. The train, validation, and test partitions were then generated from the cleaned image pool so that highly similar frames were not intentionally assigned to different subsets. This procedure was adopted to make the test results reflect segmentation generalization to unseen barn appearances rather than memorization of adjacent video frames.

3.4. Backbone with Coordinate Attention

In dense breeding environments, standard convolutional backbones may struggle to preserve discriminative spatial cues because of background clutter, animal overlap, and limited long-range dependency modeling. To address this issue, we integrate the Coordinate Attention (CA) module [16] into the residual blocks of the ResNet-50 backbone. The structure of the CA module is illustrated in Figure 3.

Given an input feature tensor $X=\left[x_c(i,j)\right]\in {\mathbb{R}}^{C\times H\times W}$, the CA module first performs one-dimensional global pooling along the horizontal and vertical directions to preserve long-range dependency while retaining positional information. The two directional descriptors are formulated as

$$z_c^h\left(i\right)={\displaystyle \frac{1}{W}}{\displaystyle \sum _{j=1}^W}{x}_c(i,j),z_c^h\in {\mathbb{R}}^{C\times H\times 1}$$

(2)

and

$$z_c^w\left(j\right)={\displaystyle \frac{1}{H}}{\displaystyle \sum _{i=1}^H}{x}_c(i,j),z_c^w\in {\mathbb{R}}^{C\times 1\times W}$$

(3)

The pooled features are concatenated and passed through a shared transformation consisting of a $1\times 1$ convolution, batch normalization, and a nonlinear activation function:

$$f=\delta \left(\mathrm{BN}\left(F_1\left([z^h,z^w]\right)\right)\right)$$

(4)

where $F_1(·)$ denotes the shared $1\times 1$ convolution and $\delta (·)$ denotes the nonlinear activation. The transformed feature is then split into two branches to generate the horizontal and vertical attention maps:

$$g^h=\sigma \left(F_h(f^h)\right),g^w=\sigma \left(F_w\left(f^w\right)\right)$$

(5)

where $F_h(·)$ and $F_w(·)$ denote the branch-specific $1\times 1$ convolutions and $\sigma (·)$ denotes the sigmoid function. Finally, the output feature is obtained by direction-aware reweighting:

$$y_c(i,j)=x_c(i,j)·g_c^h\left(i\right)·g_c^w\left(j\right)$$

(6)

In this way, the CA module enhances the backbone with both channel dependency and coordinate-sensitive spatial information. For dense dairy goat scenes, this design helps the network emphasize elongated body structures and suppress background interference from fences, troughs, and floor textures, thereby improving feature localization under occlusion and mutual adhesion.

3.5. Adaptive Receptive Field Encoder

Although the CA-enhanced backbone improves spatial feature localization, dense dairy goat scenes still involve substantial geometric variation caused by bending, turning, feeding, and partial occlusion. Standard convolutions use fixed sampling grids and therefore cannot adapt their receptive field to such irregular shapes. To address this issue, we propose an Adaptive Receptive Field Encoder (ARFE), as shown in Figure 4. ARFE is the core methodological innovation of this study. It is designed to adapt the feature encoding process to the visible structure of each target, rather than relying on a fixed receptive-field pattern throughout the encoder.

The proposed ARFE is built upon deformable convolution, but its role in this work is not merely to insert an existing operator into the network. Instead, deformable sampling is organized here as an adaptive receptive-field encoding strategy for dense goat segmentation, so that receptive-field responses can better match incomplete body regions, elongated contours, and large posture variation. Given an input feature map $F_{in}\in {\mathbb{R}}^{C\times H\times W}$, ARFE first uses lightweight convolutional branches to predict sampling offsets and modulation scalars:

$$\Delta P={\varphi }_{off}\left(F_{in}\right),$$

(7)

$$\Delta M=\sigma \left({\varphi }_{mod}\left(F_{in}\right)\right)$$

(8)

where ${\varphi }_{off}(·)$ denotes the offset learning branch, ${\varphi }_{mod}(·)$ denotes the modulation learning branch, and $\sigma (·)$ constrains the modulation values to the range of $[0,1]$.

Concretely, the deformable feature response at spatial position $p_0$ is computed as

$$F_{def}\left(p_0\right)=\sum _{p_k\in R}{w}_k·F_{in}(p_0+p_k+\Delta p_k)·\Delta m_k$$

(9)

where R denotes the regular sampling grid, $w_k$ denotes the kernel weight, $\Delta p_k$ is the learned offset, and $\Delta m_k$ is the modulation scalar. Through this operation, the sampling locations are no longer restricted to a fixed grid and can shift toward informative regions such as curved backs, extended limbs, and partially visible contours.

To further capture dairy goats at different apparent scales, ARFE introduces a multi-scale context aggregation branch. This branch applies several $3\times 3$ convolutions with different dilation rates:

$$F_s=Con{v}_{3\times 3}^{d=s}\left(F_{in}\right),s\in \{1,2,3,4\}$$

(10)

where d denotes the dilation rate. The multi-scale responses are concatenated and fused by a $1\times 1$ convolution:

$$F_{ms}=Con{v}_{1\times 1}\left([F_1,F_2,F_3,F_4]\right)$$

(11)

The final ARFE output is obtained by fusing the deformable feature and the multi-scale context feature:

$$F_{out}=Con{v}_{1\times 1}\left([F_{def},F_{ms}]\right)$$

(12)

The function of each ARFE component is summarized in

Table 3. Functional roles of the main components in the proposed ARFE module.

Component	Main Function	Targeted Visual Challenge
Offset learner	Predicts adaptive sampling locations from the input feature map.	Irregular contours and posture deformation.
Modulation learner	Reweights sampled feature responses according to local structure.	Occlusion and background interference.
Multi-scale context branch	Aggregates features from different receptive-field ranges.	Scale variation and partial visibility.
Feature fusion	Integrates deformable responses and multi-scale context information.	Feature ambiguity in dense and cluttered scenes.

. Compared with directly using a single deformable convolution layer, ARFE organizes offset learning, modulation weighting, multi-scale context aggregation, and feature fusion into a unified encoding block. This structure is designed for dense goat scenes where visible regions are incomplete, object contours are irregular, and neighboring instances are visually similar.

From a mathematical viewpoint, ARFE changes the local feature operator from a fixed spatial mapping to an input-conditioned mapping. In a standard convolution, the sampling coordinates $p_0+p_k$ are fixed for all images and all local structures. In ARFE, the sampling coordinates become $p_0+p_k+\Delta p_k$, and each sampled response is further controlled by $\Delta m_k$. Therefore, the effective receptive field is conditioned on the observed feature pattern. When a goat body is bent, partially occluded, or truncated by another goat, the learned offsets can move the sampling positions toward informative visible contours, while the modulation scalars can reduce the influence of background or occluded regions. The multi-scale branch complements this local adaptation by collecting contextual information from several dilation ranges. Computationally, this design improves representation flexibility while keeping the additional cost local and convolutional, rather than introducing a much deeper encoder.

3.6. Query-Conditioned Dynamic Mask Head

A common failure mode in crowded barns is instance adhesion, where adjacent goats are merged into a single predicted mask. To alleviate this issue, we design a Query-Conditioned Dynamic Mask Head. The structure of this module is shown in Figure 5, where the decoder query is first mapped by an MLP into an instance-specific dynamic kernel and then applied to the mask feature map for mask generation.

Let $q_i\in {\mathbb{R}}^D$ denote the refined query embedding of the i-th predicted instance. The dynamic mask head maps $q_i$ into an instance-specific kernel and bias:

$${\omega }_i=ML{P}_{\omega }\left(q_i\right)$$

(13)

$${\beta }_i=ML{P}_{\beta }\left(q_i\right)$$

(14)

where ${\omega }_i$ and ${\beta }_i$ are used to generate the mask for the corresponding instance.

Meanwhile, multi-scale image features are fused to construct a shared mask feature map:

$$F_m=\psi (F_3,F_4,F_5)$$

(15)

where $F_3$, $F_4$, and $F_5$ denote multi-scale feature maps, and $\psi (·)$ represents the mask feature construction process consisting of lateral convolution, upsampling, and feature fusion.

For the i-th instance, the final mask response at location $(u,v)$ is formulated as

$$M_i(u,v)=\sigma \left({\displaystyle \sum _{c=1}^{C_m}}{\omega }_{i,c}·F_{m,c}(u,v)+{\beta }_i\right)$$

(16)

where $C_m$ is the channel number of the mask feature map, and $\sigma (·)$ is the sigmoid activation function.

Compared with a static mask head, this design makes mask prediction directly dependent on the semantic representation of each object query. Therefore, different goat instances are predicted with different mask parameters. This instance-specific prediction mechanism improves the model’s ability to distinguish closely interacting goats, especially when they share similar appearance and overlap heavily in space.

This formulation also provides a computational interpretation of instance separation. The shared feature map $F_m$ contains common spatial evidence for all goats, whereas each query $q_i$ generates a different kernel ${\omega }_i$ and bias ${\beta }_i$. Thus, the mask of the i-th goat can be viewed as a query-conditioned classifier over the same mask feature space. Adjacent goats may share similar texture and color, but their decoder queries encode different instance-level locations and semantics. Conditioning the mask kernel on $q_i$ therefore reduces the tendency of a static mask head to merge adjacent animals and improves separation under body contact, posture variation, and partial visibility.

3.7. Training Objective

The proposed framework is trained in an end-to-end manner. Following the query-based detection and segmentation paradigm, predicted instances are matched with ground-truth instances through bipartite matching. The overall training objective consists of classification loss, box regression loss, generalized IoU loss, and mask prediction loss:

$$\mathcal{L}={\lambda }_{cls}{\mathcal{L}}_{cls}+{\lambda }_{box}{\mathcal{L}}_{box}+{\lambda }_{giou}{\mathcal{L}}_{giou}+{\lambda }_{mask}{\mathcal{L}}_{mask}+{\lambda }_{dice}{\mathcal{L}}_{dice}$$

(17)

where ${\mathcal{L}}_{cls}$ denotes the classification loss, ${\mathcal{L}}_{box}$ denotes the bounding-box regression loss, ${\mathcal{L}}_{giou}$ denotes the generalized IoU loss, ${\mathcal{L}}_{mask}$ denotes the pixel-wise mask loss, and ${\mathcal{L}}_{dice}$ denotes the Dice loss. The coefficients ${\lambda }_{cls}$, ${\lambda }_{box}$, ${\lambda }_{giou}$, ${\lambda }_{mask}$, and ${\lambda }_{dice}$ are used to balance different loss terms.

For the mask branch, the pixel-wise mask loss encourages accurate foreground-background assignment, while the Dice loss improves region-level overlap between the predicted mask and the ground-truth mask. This combination is beneficial for dense dairy goat segmentation because many target masks are incomplete or partially occluded, and boundary quality is important for separating adjacent animals.

3.8. Inference Procedure

During inference, the input image is first processed by the CA-enhanced backbone and the ARFE to obtain spatially enhanced feature representations. The transformer decoder then produces a fixed set of query embeddings. Each query generates a category score, a bounding box, and an instance mask. Predictions with low confidence scores are filtered out, and the remaining masks are resized to the original image resolution. The final output is a set of dairy goat instance masks with corresponding confidence scores and bounding boxes.

Because RF-GoatDet follows the query-based prediction paradigm, it reduces dependence on dense anchors and heavy post-processing. This design helps maintain real-time inference while improving mask quality in dense and occlusion-heavy dairy goat scenes.

3.9. Computational Complexity Consideration

The proposed modules were designed to improve dense-scene representation without changing the overall real-time nature of the RT-DETR-based framework. The CA module introduces only lightweight one-dimensional pooling and $1\times 1$ transformations. ARFE adds offset prediction, modulation prediction, deformable sampling, and several dilated $3\times 3$ convolution branches. Its additional cost is local to the feature maps and is therefore controlled by the spatial resolution of the encoder features. The Q-DMH introduces a query-conditioned dynamic kernel for each retained object query. If the mask feature map has spatial size $H_m\times W_m$ and channel number $C_m$, the main mask computation is approximately proportional to $N{H}_m{W}_m{C}_m$, where N is the number of object queries. This cost is moderate because N is fixed and much smaller than the number of dense anchor locations used in anchor-based pipelines.

The ablation results further quantify the accuracy-efficiency trade-off. Compared with the RT-DETR baseline, the full RF-GoatDet increases the parameter count from 32.0 M to 35.8 M, but improves AP from 42.3% to 46.5% and AP₇₅ from 44.1% to 49.8%. The measured inference speed remains 62 FPS on an RTX 3090 under the same input setting. Thus, the added computational cost is exchanged for stronger boundary quality and instance discrimination, while the model still satisfies real-time inference requirements in the in-house GPU benchmark.

4. Experiments and Results

In this section, we evaluate the proposed RF-GoatDet framework on the constructed dairy goat instance segmentation dataset. The experiments include implementation details, evaluation metrics, ablation analysis, training dynamics, comparison with representative instance segmentation methods, and qualitative visualization. These experiments are designed to verify whether the proposed framework can improve mask quality and instance discrimination while maintaining real-time inference performance in dense barn scenes.

4.1. Experimental Setup

All experiments were implemented in the PyTorch version 2.7.0 with CUDA 12.8 framework on a workstation equipped with an Intel Core i9-12900K CPU and a single NVIDIA GeForce RTX 3090 GPU with 24 GB memory. For optimization, the AdamW optimizer was adopted with an initial learning rate of $1\times {10}^{-4}$ and a weight decay of $1\times {10}^{-4}$. The batch size was set to 8, and the model was trained for 100 epochs. To improve generalization and reduce overfitting, standard data augmentation strategies were employed, including random horizontal flipping, multi-scale resizing from 480 to 800 pixels, and Mosaic augmentation. The learning rate was decayed using a cosine annealing schedule.

The proposed RF-GoatDet was trained and evaluated on the dairy goat dataset described in Section 3.3. The final dataset was divided into training, validation, and test subsets with a ratio of 7:2:1. The same data split was used for ablation experiments and comparative evaluation to ensure consistency. Unless otherwise specified, all reported AP values refer to the test set results, while the training dynamics curves are used to analyze the optimization behavior on the validation set.

For comparative experiments, all representative methods were trained and evaluated on the same train/validation/test split. The same image-resolution range and the same basic augmentation strategy were used whenever supported by the official implementation. Model-specific hyperparameters were kept consistent with the official or commonly used settings, while the training schedule and evaluation protocol were aligned as much as possible. This setting was adopted to make the comparison reflect architectural differences rather than differences in data partitioning or evaluation criteria.

4.2. Evaluation Metrics

To comprehensively evaluate instance segmentation performance, we adopted standard COCO-style mask evaluation metrics. The primary metric was mask AP, which is computed as the mean precision over IoU thresholds from 0.50 to 0.95 with a step size of 0.05. In addition, mask AP₅₀ and mask AP₇₅ were reported to evaluate segmentation quality under relatively loose and strict IoU thresholds, respectively. AP₅₀ mainly reflects coarse mask localization ability, whereas AP₇₅ provides a stricter measure of boundary quality and instance-level mask accuracy. In the remainder of this paper, “AP” refers to mask AP unless otherwise specified.

To evaluate deployment feasibility, inference speed was measured in frames per second (FPS). FPS was tested on the RTX 3090 GPU with a batch size of 1, and data loading time was excluded. This setting provides a controlled runtime comparison among different models. Since inference speed can be affected by hardware, implementation details, and input resolution, the reported FPS values should be interpreted as an in-house benchmark under the same experimental environment rather than as hardware-independent results.

We further note that the current runtime benchmark was obtained on a high-end desktop GPU and does not directly represent deployment speed on embedded farm devices. For edge deployment, the memory footprint, power consumption, and acceleration backend, such as TensorRT or ONNX Runtime, should be evaluated on standard devices such as NVIDIA Jetson or other industrial edge platforms. Because such hardware was not available in the current experimental setting, edge-device benchmarking is treated as an important direction for future work rather than an unsupported claim in this study.

4.3. Ablation Study

To evaluate the contribution of each proposed component, we conducted a stepwise ablation study on the dairy goat dataset. Starting from the RT-DETR baseline with a ResNet-50 backbone, we progressively introduced the Coordinate Attention (CA) module, the Adaptive Receptive Field Encoder (ARFE), and the Query-Conditioned Dynamic Mask Head (Q-DMH). The quantitative results are summarized in

Table 4. Ablation study of the proposed components.

Model	CA	ARFE	Q-DMH	AP (%)	AP50 (%)	AP75 (%)	Params (M)
Baseline (RT-DETR)				42.3	76.5	44.1	32.0
+CA	✓			43.5 (+1.2)	77.8	45.6	32.2
+ARFE	✓	✓		45.3 (+1.8)	80.2	48.1	34.5
RF-GoatDet (Ours)	✓	✓	✓	46.5 (+1.2)	81.5	49.8	35.8

.

As shown in Table 4, the RT-DETR baseline achieved 42.3% AP, 76.5% AP₅₀, and 44.1% AP₇₅. After introducing CA into the backbone, the AP increased to 43.5%, indicating that direction-aware spatial representation helps improve feature localization in cluttered barn scenes. When ARFE was further added, the AP increased to 45.3%, and AP₅₀ reached 80.2%. This improvement shows that adaptive receptive-field modeling is beneficial for handling irregular goat contours, scale variation, and partial visibility. After adding Q-DMH, the complete RF-GoatDet model achieved 46.5% AP and 49.8% AP₇₅. The improvement in AP₇₅ suggests that query-conditioned dynamic mask generation contributes to more accurate boundary delineation and better separation of adjacent goats.

Overall, RF-GoatDet improved the baseline by 4.2 percentage points in AP, while the parameter count increased from 32.0 M to 35.8 M. This indicates that the added modules introduce only a moderate increase in model size, but bring clear gains in segmentation accuracy. More importantly, the improvements are reflected not only in AP₅₀ but also in AP₇₅, which means that the proposed components improve both coarse instance localization and stricter mask quality. These results support the effectiveness of the ARFE-centered design for dense dairy goat instance segmentation.

The practical meaning of these gains is that more goat instances can be separated at a stricter mask-overlap threshold. In crowded barns, AP₇₅ is particularly relevant because small boundary errors may merge two contacting goats or assign one goat’s visible body region to another. Therefore, the increase from 44.1% to 49.8% in AP₇₅ suggests improved reliability for downstream contour-based measurement, posture estimation, and individual tracking, where mask boundary quality is more important than coarse detection alone.

To further analyze the computational influence of each proposed component, we evaluated the accuracy–efficiency trade-off of different RF-GoatDet variants. As summarized in

Table 5. Accuracy–efficiency trade-off of different RF-GoatDet variants.

Model Variant	Main Added Component	FPS	AP (%)	AP75 (%)
RT-DETR baseline	None	70	42.3	44.1
RT-DETR + CA	Direction-aware spatial attention	66	43.5	45.6
RT-DETR + CA + ARFE	Adaptive receptive-field encoding	64	45.3	48.1
RF-GoatDet	CA + ARFE + Q-DMH	62	46.5	49.8

, the introduction of CA, ARFE, and Q-DMH gradually improved segmentation accuracy, while the inference speed only decreased slightly from 70 FPS to 62 FPS. This indicates that the proposed modules improve feature representation and instance-level mask discrimination without severely sacrificing real-time inference efficiency.

Compared with the RT-DETR baseline, the complete RF-GoatDet model improved AP by 4.2 percentage points and AP₇₅ by 5.7 percentage points, while maintaining a real-time inference speed of 62 FPS on the same experimental platform. Although the adaptive receptive-field operation and query-conditioned mask generation introduce additional computation, their cost is relatively limited compared with the improvement in dense-scene segmentation accuracy. Therefore, the proposed framework achieves a favorable balance between segmentation performance and inference efficiency for real-time dairy goat monitoring.

4.4. Training Dynamics Analysis

To further analyze the optimization behavior of RF-GoatDet, we compared the training loss and validation AP curves of RF-GoatDet and the RT-DETR baseline. The results are shown in Figure 6.

As shown in Figure 6a, both models converge stably during training. The training loss decreases rapidly in the early stage and gradually approaches a stable level in later epochs. Compared with the baseline, RF-GoatDet maintains a lower training loss throughout most of the training process. This indicates that the proposed modules do not make optimization unstable. Instead, they help the model learn more effective feature representations for dense dairy goat segmentation.

Figure 6b shows that RF-GoatDet also achieves consistently higher validation AP than the baseline. The performance gap appears in the early training stage and remains clear in the middle and later stages. At the end of training, RF-GoatDet reaches a validation AP of 0.465, while the baseline reaches 0.423. In addition, RF-GoatDet obtains a higher peak validation AP than the baseline. These results indicate that the proposed framework improves both convergence behavior and final segmentation performance.

Although both models show mild AP fluctuations in the later training stage, RF-GoatDet maintains a clear advantage over the baseline. This behavior is common in instance segmentation training, especially when dense scenes contain severe occlusion and ambiguous boundaries. Overall, the training dynamics provide additional evidence that the proposed RF-GoatDet framework achieves a more favorable optimization trajectory than the RT-DETR baseline.

4.5. Comparison with Representative Methods

To evaluate the overall effectiveness of RF-GoatDet, we compared it with representative instance segmentation methods, including two-stage, one-stage, and transformer-based approaches. The quantitative comparison is reported in

Table 6. Performance comparison with representative instance segmentation methods.

Method	Backbone	AP (%)	AP50 (%)	AP75 (%)	FPS
Two-stage methods
Mask R-CNN	ResNet-50	38.2	68.5	40.1	14
Mask R-CNN	ResNet-101	40.1	70.8	42.3	11
Cascade Mask R-CNN	ResNet-101	46.8	81.0	50.2	18
One-stage methods
YOLACT	ResNet-50	34.5	58.2	35.6	45
SOLOv2	ResNet-50	40.5	72.1	42.8	32
YOLOv8-Seg (L)	CSP-Darknet	45.0	79.8	48.5	55
Transformer-based methods
RT-DETR-L (Baseline)	ResNet-50	42.3	76.5	44.1	70
RF-GoatDet (Ours)	ResNet-50	46.5	81.5	49.8	62

.

Among the two-stage methods, Mask R-CNN with ResNet-50 and ResNet-101 backbones achieved AP values of 38.2% and 40.1%, respectively, which are lower than RF-GoatDet. Cascade Mask R-CNN [37] achieved the highest AP of 46.8% and AP₇₅ of 50.2%, but its inference speed was only 18 FPS. This result shows that two-stage cascade refinement can improve mask accuracy, but it is less suitable for real-time barn monitoring. In contrast, RF-GoatDet achieved a comparable AP of 46.5% while reaching 62 FPS, showing a better balance between accuracy and efficiency.

Compared with one-stage methods, RF-GoatDet also showed clear advantages. YOLACT [38] and SOLOv2 [39] achieved AP values of 34.5% and 40.5%, respectively. YOLOv8-Seg (L), built on the YOLOv8 family [40], achieved 45.0% AP and 79.8% AP₅₀. RF-GoatDet surpassed YOLOv8-Seg (L) by 1.5 percentage points in AP and 1.7 percentage points in AP₅₀. This suggests that the proposed adaptive feature encoding and query-conditioned mask prediction are more effective for irregular goat contours and dense occlusion than standard one-stage segmentation pipelines.

Compared with the RT-DETR-L baseline, RF-GoatDet improved AP from 42.3% to 46.5% and AP₅₀ from 76.5% to 81.5%. Although the additional modules reduced FPS from 70 to 62, the model still maintained real-time inference capability. Therefore, the proposed method improves segmentation quality without sacrificing practical runtime performance. These results show that RF-GoatDet provides a favorable accuracy–efficiency trade-off for dense dairy goat instance segmentation.

4.6. Robustness Under Different Density and Occlusion Conditions

Since the proposed framework is mainly designed for dense barn environments, we further analyzed its segmentation performance under different animal-density and occlusion conditions. The test images were grouped according to the number of visible goats and the degree of body overlap or truncation. The corresponding results are reported in

Table 7. Segmentation performance under different density and occlusion conditions.

Scenario	Criterion	Test Images	AP (%)	AP75 (%)
Low-density scenes	1–2 visible goats with weak or no overlap	46	49.2	52.1
Medium-density scenes	3–5 visible goats with partial body overlap	110	46.8	50.3
High-density scenes	≥6 visible goats or obvious instance adhesion	73	43.7	46.5
Mild occlusion	Most body contours are visible and separable	61	49.6	52.7
Moderate occlusion	Partial overlap among adjacent goats or fence occlusion	104	46.4	49.6
Severe occlusion	Large body overlap, truncation, or incomplete visibility	64	42.9	45.8

.

As shown in Table 7, segmentation performance decreased as animal density and occlusion severity increased. This trend is expected because adjacent dairy goats usually have similar white body appearance, overlapping contours, and partial visibility in crowded barn scenes. Nevertheless, RF-GoatDet still maintained an AP of 43.7% in high-density scenes and 42.9% under severe occlusion, demonstrating its robustness in challenging livestock-monitoring scenarios. These results further support the effectiveness of adaptive receptive-field modeling and query-conditioned mask generation for instance separation under dense and occlusion-heavy conditions.

4.7. Qualitative Analysis

To further evaluate the effectiveness of RF-GoatDet under challenging farm conditions, we visualized the segmentation outputs and feature activation maps on representative samples from the Dairy Goat test set. Figure 7 compares RF-GoatDet with Mask R-CNN, YOLOv8-Seg, and RT-DETR-L. Figure 8 further presents Grad-CAM activation maps for the same or similar scenes [41]. These visual results provide qualitative support for the quantitative findings.

The first two rows of Figure 7 show dense feeding scenes with severe mutual occlusion. In these cases, adjacent goats are close to each other, and their visible body regions overlap substantially. Mask R-CNN tends to produce under-segmented masks in crowded regions, while YOLOv8-Seg sometimes generates fragmented or incomplete masks. RT-DETR-L improves instance separation to some extent, but it still misses some partially visible targets or produces incomplete boundaries. In contrast, RF-GoatDet produces more complete masks and clearer separation between adjacent goats. This visual difference is consistent with the role of Q-DMH, which generates instance-specific mask parameters from object queries.

The bottom rows of Figure 7 show scenes with more complex posture variation and irregular body contours. In these cases, conventional methods often produce coarse or incomplete mask boundaries around curved backs, legs, and head regions. RF-GoatDet better follows the visible semantic contours of the goats and generates masks that are more consistent with the actual body shapes. This observation supports the effectiveness of ARFE, which improves adaptive feature sampling for non-rigid and partially visible animal structures.

Figure 8 provides additional insight into the feature response of different models. Compared with the competing methods, RF-GoatDet shows more concentrated activation on goat body regions and less activation on background structures such as fences, troughs, and floor textures. The activation maps of Mask R-CNN and YOLOv8-Seg are relatively scattered in some crowded regions, while RT-DETR-L shows less focused responses under heavy occlusion. These results suggest that the combination of Coordinate Attention and ARFE helps improve target localization and suppress background interference.

Overall, the quantitative and qualitative results consistently show that RF-GoatDet improves dense dairy goat instance segmentation. The ablation study verifies the contribution of each component, the training curves show stable optimization, the comparison with representative methods demonstrates a favorable accuracy–efficiency trade-off, and the visualization results further show clearer instance separation and more focused feature activation in challenging barn scenes.

5. Discussion

The experimental results show that RF-GoatDet provides an effective solution for dense dairy goat instance segmentation. Compared with the RT-DETR baseline, the proposed framework improves mask AP from 42.3% to 46.5% while maintaining a real-time inference speed of 62 FPS. This improvement indicates that the proposed modifications enhance segmentation quality without sacrificing practical efficiency. In dense barn environments, where goats frequently overlap and appear in close contact, this balance between accuracy and speed is important for continuous visual monitoring.

The performance gain of RF-GoatDet can be explained by the complementary roles of the three main components. Coordinate Attention strengthens direction-aware spatial representation and helps preserve positional cues in cluttered scenes. This is useful when goat bodies are elongated and easily confused with fences, troughs, and floor textures. The Adaptive Receptive Field Encoder further improves feature encoding by allowing the receptive field to adapt to irregular and partially visible goat structures. As shown in the ablation study, adding ARFE increases AP from 43.5% to 45.3%, which suggests that adaptive receptive-field modeling is beneficial for handling scale variation, posture deformation, and incomplete visible regions. Finally, the Query-Conditioned Dynamic Mask Head improves instance-specific mask generation. Its contribution is reflected in the increase in AP₇₅ from 48.1% to 49.8%, indicating better boundary quality and stronger separation of adjacent goats.

The above behavior can be interpreted mathematically as a combination of input-conditioned spatial sampling and query-conditioned mask classification. ARFE replaces a fixed receptive-field operator with an adaptive operator whose sampling offsets and modulation weights depend on the local feature pattern. This makes the encoded feature less sensitive to rigid sampling grids and more suitable for goats with curved backs, truncated limbs, and partially visible body regions. Q-DMH then uses the decoder query as a conditioning vector to generate instance-specific mask parameters. Instead of applying one static mask classifier to all nearby goats, the model learns a different mask decision function for each predicted instance. This mechanism is particularly useful when two goats touch each other and share similar texture, because query-level conditioning helps associate pixels with the correct instance.

The qualitative results further support these findings. In the dense feeding scenes shown in Figure 7, conventional methods tend to produce incomplete masks, fragmented predictions, or merged instances when adjacent goats overlap. RF-GoatDet generates more complete masks and clearer instance boundaries. This visual improvement is consistent with the role of Q-DMH, because each object query produces an instance-specific dynamic kernel for mask prediction. In scenes with complex posture variation, RF-GoatDet also better follows visible animal contours, which supports the effectiveness of ARFE in representing non-rigid goat structures. The Grad-CAM visualization in Figure 8 also shows that RF-GoatDet produces more concentrated activation on goat body regions and less response to irrelevant background areas. This suggests that the combination of Coordinate Attention and ARFE improves feature localization in dense and cluttered barn scenes.

Compared with representative instance segmentation methods, RF-GoatDet achieves a favorable accuracy–efficiency trade-off. Two-stage methods such as Mask R-CNN and Cascade Mask R-CNN can provide competitive mask quality, but their inference speed is relatively low. Cascade Mask R-CNN achieves slightly higher AP than RF-GoatDet, but its FPS is much lower, which limits its suitability for real-time farm monitoring. One-stage methods such as YOLACT, SOLOv2, and YOLOv8-Seg are faster, but their mask accuracy is lower in dense dairy goat scenes. RF-GoatDet achieves comparable or better segmentation accuracy while preserving real-time speed, showing that enhancing a query-based detector with adaptive feature encoding and dynamic mask prediction is a practical design route for livestock instance segmentation.

The computational trade-off should also be considered. The proposed modules increase parameters from 32.0 M to 35.8 M, corresponding to a moderate model-size increase. This additional cost is mainly introduced by ARFE and Q-DMH, which improve adaptive feature encoding and mask separation. In return, the model gains 4.2 percentage points in AP and 5.7 percentage points in AP₇₅. Because the final speed remains 62 FPS in the in-house GPU benchmark, the proposed design achieves a practical balance between segmentation quality and real-time inference. However, this does not mean that the same speed can be directly transferred to edge devices. Real farm deployment should further evaluate memory consumption, power budget, and hardware acceleration on embedded devices such as Jetson-class platforms.

From an application perspective, accurate dairy goat instance segmentation can support several downstream precision livestock farming tasks. Pixel-level masks provide more detailed animal contours than bounding boxes, which is useful for body measurement, posture analysis, behavior understanding, and individual tracking. In dense barn scenes, the ability to separate adjacent animals is especially important because errors in mask assignment may directly affect subsequent phenotyping or behavior analysis. Therefore, the improved mask quality and instance discrimination of RF-GoatDet provide a useful visual foundation for intelligent dairy goat monitoring systems. This is consistent with recent livestock vision studies, which emphasize that robust localization, fine-grained segmentation, and deployment efficiency are important for practical animal monitoring and phenotyping [42,43,44,45,46].

More specifically, separated instance masks can provide three types of downstream information. First, the visible contour and mask area can be used for body-shape measurement and growth-state estimation when combined with camera calibration or depth information. Second, the orientation and contour deformation of the mask can support posture analysis, such as standing, feeding, turning, or lying-related behavior. Third, temporally consistent masks can serve as the input for multi-object tracking and individual activity analysis. These applications require reliable instance separation in crowded scenes; therefore, the AP₇₅ improvement and the qualitative reduction of merged masks are practically meaningful. The present study does not yet include a complete downstream task benchmark, and this limitation is explicitly acknowledged below.

Several limitations should be noted. First, the current dataset was collected from a single experimental dairy goat facility. Although the images include different illumination conditions, herd densities, and behavioral states, multi-farm validation is still necessary to test generalization under different barn layouts, camera viewpoints, backgrounds, and herd phenotypes. Second, the reported FPS was measured on an RTX 3090 GPU. Edge-device benchmarking on standard farm-deployable hardware remains necessary before large-scale practical deployment. Third, the study evaluates segmentation accuracy and speed, but it does not directly quantify downstream benefits for body measurement, posture recognition, or tracking. Fourth, very severe occlusion may still cause failure when only a small fragment of the goat body is visible or when several goats form a continuous visual region. Future work will therefore focus on multi-farm data collection, lightweight model compression, edge inference evaluation, temporal tracking, and downstream task validation.

6. Conclusions

This study proposed RF-GoatDet, an occlusion-aware real-time instance segmentation framework for dairy goats in dense barn environments. The framework is built upon RT-DETR and integrates Coordinate Attention, an Adaptive Receptive Field Encoder, and a Query-Conditioned Dynamic Mask Head. These components are designed to improve spatial localization, adaptive feature encoding, and instance-specific mask prediction under mutual occlusion, instance adhesion, partial visibility, and non-rigid posture variation.

On the constructed dairy goat instance segmentation dataset, RF-GoatDet achieved 46.5% mask AP and 62 FPS. Compared with the RT-DETR baseline, it improved mask AP by 4.2 percentage points while maintaining real-time inference capability. The ablation results show that each component contributes to the final performance, and the complete framework achieves the best overall result. The comparison with representative two-stage, one-stage, and transformer-based methods further demonstrates that RF-GoatDet provides a strong balance between segmentation accuracy and inference efficiency.

The qualitative results also confirm the effectiveness of the proposed design. RF-GoatDet produces more complete masks, clearer instance boundaries, and more focused feature activation in dense and cluttered dairy goat scenes. These results indicate that adaptive receptive-field modeling and query-conditioned dynamic mask generation are effective for separating adjacent goats and improving mask quality under challenging farm conditions.

Overall, RF-GoatDet provides a robust and efficient instance segmentation solution for real-time dairy goat visual monitoring. The proposed framework can serve as a useful basis for downstream precision livestock farming applications, including body measurement, posture analysis, behavior understanding, and individual tracking. Future work will focus on lightweight deployment and the integration of segmentation results with practical farm management tasks.

In future studies, we will extend the dataset to multiple farms with different barn layouts, illumination conditions, camera configurations, and herd phenotypes. We will also benchmark the model on edge hardware and evaluate whether the improved segmentation masks can quantitatively improve downstream tasks such as contour-based body measurement, posture classification, and long-term individual tracking.