Probabilistic Orchestrator for Indeterministic Multi-Agent Systems in Real-Time Environments

Abstract

Multi-agent perception systems must operate under fundamental asymmetries: some agents provide fast but unreliable observations, while others deliver higher-quality evidence with delay and uncertain correspondence. Traditional deterministic orchestration and rule-based fusion struggle to manage these trade-offs, often producing brittle or unstable behavior. We introduce a probabilistic orchestration framework that treats coordination as an epistemic generation problem—constructing and updating belief states under uncertainty—rather than a selection problem. Instead of committing to a single agent’s output, the orchestrator constructs a belief state that explicitly represents uncertainty, evidential provenance, and temporal relevance. Decisions are produced through latency-aware, association-weighted fusion, and uncertainty itself becomes a first-class signal governing action, deferral, and learning. Crucially, the orchestrator enables controlled teacher–student adaptation: high-confidence, well-associated stationary observations are gated into a feedback loop that improves ego perception over time while mitigating error amplification. We demonstrate the approach on an infrastructure-assisted dual-camera obstacle-recognition task. Experimental results show improved robustness to distance, occlusion, and delayed evidence compared to ego-only and deterministic orchestration baselines. By operationalizing orchestration as epistemic generation, this work provides a unifying framework for robust decision-making and safe adaptation in multi-agent systems, with implications that extend beyond perception to agentic and generative AI architectures.

1. Introduction

Consider an autonomous vehicle approaching an intersection. Its onboard camera detects a potential obstacle, but the object is distant and partially hidden behind another car. Meanwhile, a roadside camera with a better vantage point has already recognized the same obstacle with high confidence—but this information arrives with a 200-millisecond delay due to network transmission. Which source should the vehicle trust? How should it combine these conflicting signals into a safe decision? This scenario illustrates a fundamental challenge in multi-agent perception systems: different sensors offer complementary strengths, yet they also differ in accuracy, timing, and reliability. The onboard camera provides immediate feedback but struggles with distant or occluded objects. The infrastructure sensor offers superior recognition but introduces latency. Neither source alone is sufficient; the real challenge lies in orchestrating them effectively.

Current approaches to this problem typically rely on simple rules: trust the source with higher confidence, or always prefer the faster sensor for time-critical decisions. While such rule-based methods are easy to implement, they break down when conditions become complex. What happens when both sources report conflicting results with similar confidence? Or when the “reliable” sensor occasionally produces overconfident errors? In practice, these rigid approaches lead to unstable behavior—flickering detections, missed obstacles, and false alarms that undermine system reliability.

This paper proposes a different approach: a probabilistic orchestrator that treats each sensor’s output not as a definitive answer but as uncertain evidence to be weighed and combined. The key insight is that effective multi-source fusion requires reasoning about three interrelated factors: how confident each source is in its prediction, how likely it is that both sources are observing the same object, and how much we should discount information that arrives late? By modeling these factors explicitly, the orchestrator can make principled decisions even when individual sources disagree or fail. Beyond fusion, the orchestrator enables something more ambitious: continuous self-improvement. When the infrastructure sensor provides a high-confidence detection that the vehicle’s own camera missed or misclassified, this becomes a learning opportunity. The orchestrator can use such cases as training signals to gradually improve the onboard model, but only when it is confident that the teaching signal is reliable and correctly matched to the same physical object.

We demonstrate and evaluate this approach using a dual-camera setup for road obstacle detection. A vehicle-mounted camera handles real-time perception, while stationary roadside cameras provide delayed but often more accurate detections. Our experiments show that probabilistic orchestration improves the detection of distant and occluded obstacles compared to either source alone, while the learning mechanism allows the onboard model to improve over time without manual retraining.

Importantly, while this work is motivated by a concrete perception problem, its scope is broader than sensor fusion. We view orchestration as the generation of an epistemic state rather than the selection of a single “correct” output. In this sense, the orchestrator does not merely aggregate predictions but constructs a temporally grounded belief that explicitly represents uncertainty, evidential provenance, and the conditions under which decisions should be deferred, revised, or used for learning. This perspective aligns the proposed framework with recent developments in agentic and generative AI, where systems are increasingly expected to reason under incomplete information, manage heterogeneous agents, and transform partial evidence into structured, actionable interpretations rather than deterministic answers.

Four core contributions are put forward in this paper:

• Probabilistic orchestration for heterogeneous agents. We introduce an orchestration framework that fuses fast ego perception with delayed, high-confidence external evidence using calibrated confidence, association reliability, and latency-aware policies.
• Uncertainty-aware decision policy. We define an interpretable fusion mechanism that produces an uncertainty-aware output distribution, reducing brittleness compared to deterministic selection and enabling stable behavior under delayed evidence.
• Controlled teacher–student adaptation. We integrate pseudo-labeling and knowledge distillation into the orchestration layer through explicit gating, enabling continual improvement while reducing the risk of error amplification.
• Dual-camera obstacle recognition case study. We demonstrate the approach in an infrastructure-assisted perception setting and analyze robustness under distance, occlusion, and latency variability.

Probabilistic Orchestrator advantages:

• For high-speed cars, it is very important to detect (classification, objectness, and bounding box) obstacles that are far off, because the braking distance of a car can be very long. Probabilistic orchestration helps to improve obstacle detection for far-away objects.
• The Probabilistic orchestration improves decision quality compared with ego-only detection under long-range and occlusion conditions.
• Probabilistic orchestration improves robustness compared with ego-only and deterministic fusion, especially for long-range and occluded obstacles.
• Latency-aware mixing stabilizes decisions and avoids abrupt behavior under delayed evidence.
• Gated teacher–student adaptation improves ego performance over time while controlling false alarms, reducing reliance on infrastructure signals.

Figure 1 provides an overview of the proposed multi-agent architecture, showing how the probabilistic orchestrator fuses ego and delayed stationary evidence and how high-confidence teacher outputs are gated to support controlled student adaptation.

In our prior work [1], we extended Rodney Brooks’ subsumption architecture by replacing its traditional priority-based arbiter with a generative orchestrator designed for semantic mediation. Instead of arbitrating between competing agent outputs, the orchestrator interprets and integrates heterogeneous agent representations into coherent, human-interpretable actions. Specifically, it synthesizes reflexive, model-based, goal-oriented, and utility-driven epistemic perspectives using large language models to produce unified decisions. That study established orchestration as a process of semantic integration, rather than hierarchical control, and validated the approach through an autonomous driving case study.

The present work advances this line of research by moving from architectural demonstration to operational realization. Specifically, we address how generative orchestration can function under real-world constraints such as calibrated uncertainty, latency-aware decision deadlines, and association ambiguity across heterogeneous sensing agents.

Given the scope of the study, the rest of this paper is organized as follows: Section 2 reviews related work, Section 3 formalizes the problem and introduces the dual-camera use case, Section 4 presents the orchestrator architecture and decision policy, Section 5 describes the controlled adaptation loop, Section 6 details the implementation, Section 7 evaluates the proposed approach, Section 8 discusses generalization and broader implications, and Section 9 concludes the paper.

2. Related Work

This work engages with several intersecting research threads that address complementary aspects of complex, real-time decision systems. Specifically, it relates to prior work on multi-agent orchestration for heterogeneous decision sources, uncertainty-aware fusion under imperfect confidence and latency constraints, teacher–student learning through pseudo-labeling and knowledge distillation, and multi-camera or infrastructure-assisted perception in autonomous driving. While each of these lines of research addresses a subset of the challenges considered here, they are typically studied in isolation or combined through fixed, rule-based mechanisms. In contrast, our approach integrates these perspectives within a unified probabilistic orchestration framework, enabling semantic mediation between heterogeneous agents. The following sections review these research directions and clarify how our method departs from and extends existing approaches.

2.1. Multi-Agent Orchestration and Decision Coordination

Multi-agent systems are widely adopted in domains where complex tasks are decomposed into specialized components (agents) and coordinated through a controller or distributed policies [1,2]. In many deployed pipelines, orchestration remains largely deterministic—implemented as rule-based gating, hard thresholds, or state-machine logic that selects one agent’s output or fuses outputs using fixed heuristics [3]. These mechanisms are attractive because they are interpretable, easy to implement, and straightforward to debug.

However, deterministic orchestration becomes brittle when agents have different error modes and timing characteristics. In real-time perception stacks, conflicts can arise when one detector produces a confident but incorrect prediction under occlusion or low resolution, while another (delayed) source produces a more accurate hypothesis. If the orchestrator treats confidence as directly comparable or treats association as binary, late-arriving signals can cause abrupt decision overrides, oscillations (decision churn), and inconsistent labels that reduce trust in the system [4].

More recent research has explored learned mechanisms for coordinating multiple decision components, including adaptive routing and decision optimization across heterogeneous models [5,6]. While these approaches move beyond fixed priority schemes, they typically assume well-calibrated confidence estimates or abstract away timing effects, limiting their suitability for real-time, safety-critical perception pipelines. In contrast, our work explicitly models both uncertainty and latency within a probabilistic orchestration framework, incorporating association reliability directly into the decision policy.

2.2. Uncertainty-Aware Fusion and Confidence Calibration

Fusion methods combine information from multiple sources to improve robustness beyond what any single sensor or model can provide. Classical approaches include Bayesian fusion, Kalman filtering, particle filtering, and probabilistic graphical models that update belief states based on uncertain observations [7]. In tracking and state estimation, uncertainty is explicit and naturally represented as part of the system state.

In deep learning-based perception systems, uncertainty is often handled implicitly, and confidence scores produced by neural networks are frequently miscalibrated, particularly under distribution shift, long-range sensing, occlusion, and rare conditions [8,9]. While calibration techniques and uncertainty estimation methods aim to improve the reliability of predicted probabilities [10,11,12], calibrated confidence alone is insufficient when evidence sources differ in latency and association reliability. In multi-view perception, late fusion and confidence-based selection are commonly used, but these approaches typically rely on fixed thresholds and neglect explicit timing effects. Our approach extends existing fusion strategies by treating uncertainty, association reliability, and latency as jointly interacting factors within a probabilistic orchestration framework.

2.3. Teacher–Student Learning, Pseudo-Labeling, and Knowledge Distillation

Teacher–student learning provides an effective mechanism for transferring knowledge from a high-capacity model (the teacher) to a smaller or faster model (the student), supporting efficient deployment and often improving generalization [13,14,15]. Knowledge distillation typically trains the student to match the teacher’s soft probability outputs, sometimes in combination with ground-truth supervision [14]. Related approaches (such as pseudo-labeling and self-training) use teacher predictions as supervisory signals to expand training data in semi-supervised settings, reducing annotation costs and enabling adaptation to new domains [16,17,18].

A central challenge in these approaches is that naïve pseudo-labeling can reinforce teacher errors, introduce confirmation bias, and amplify drift, particularly under distribution shift or when the teacher’s predictions are uncertain [19]. Consequently, effective teacher–student systems rely on careful sample selection, confidence thresholds, and validation strategies. In many perception pipelines, however, teacher–student learning remains an offline training procedure, decoupled from operational inference. In contrast, our work integrates teacher–student adaptation directly into the orchestration layer, where training signals are gated based on association reliability, teacher confidence, and timing constraints, enabling controlled and continual adaptation while limiting error propagation.

2.4. Multi-Camera and Infrastructure-Assisted Perception for Autonomous Driving

Multi-camera perception is a central component of modern autonomous driving systems, encompassing camera-only pipelines, multi-view 3D reconstruction, temporal modeling, and cross-modal sensor fusion [20,21]. By combining multiple viewpoints, such systems mitigate occlusion and improve long-range recognition, but they must also address challenges related to calibration, synchronization, and cross-view association [22].

Infrastructure-assisted perception extends multi-camera approaches by incorporating stationary sensors such as roadside cameras or smart-city infrastructure to support autonomous vehicles [23,24]. These sensors can provide advantageous viewpoints, earlier visibility of occluded objects, and improved recognition of distant obstacles. However, infrastructure-based signals introduce additional constraints, including network latency and jitter, uncertain cross-view association, partial coverage, and heterogeneity in sensor quality and model performance [25]. Many existing methods treat infrastructure inputs as auxiliary signals or fuse them using deterministic rules. In contrast, our work formulates infrastructure-assisted perception as a multi-agent orchestration problem, in which delayed external evidence is integrated probabilistically based on both association reliability and timing.

2.5. Positioning of the Proposed Approach

Prior work has established strong foundations in multi-agent coordination, probabilistic fusion, and teacher–student learning. In practice, however, many deployed systems continue to rely on deterministic orchestration mechanisms and treat learning as a separate, offline process. This paper addresses these limitations by introducing a probabilistic orchestrator that integrates heterogeneous agent outputs under uncertainty and latency constraints while simultaneously regulating teacher–student adaptation through a gated feedback mechanism. The dual-camera obstacle recognition case study illustrates how uncertainty-aware orchestration can unify real-time decision fusion with controlled, continual learning in multi-agent perception systems.

2.6. Implications for Generative Multi-Agent Systems

Recent work on generative multi-agent systems has focused on coordinating heterogeneous models and tools, such as lightweight and high-capacity language models, retrieval components, verifiers, and planners, into reliable decision workflows [26,27]. In such settings, deterministic orchestration strategies, such as fixed tool ordering or unconditional reliance on a single high-capacity model, often prove brittle in the presence of prompt variability, hallucinations, tool failures, and shifting task distributions [28]. To address these issues, several approaches have explored adaptive routing and selection policies that determine when to invoke particular models or tools, when to verify intermediate results, and when to defer decisions or request additional evidence [29].

From this perspective, our framework suggests a more general orchestration principle that is applicable beyond perception. Agent outputs are treated as uncertain evidence rather than definitive decisions, and orchestration is framed as a probabilistic process that accounts for reliability and timing. In this analogy, a high-quality model or verified tool functions as a teacher, lower-latency or lower-capacity models act as students, and auxiliary verification components provide reliability signals analogous to association confidence. The orchestrator integrates these signals to perform uncertainty-aware routing and aggregation, and it can selectively trigger adaptation using high-confidence feedback. Although our experiments focus on perception, this view positions probabilistic orchestration as a unifying design pattern for both perception-driven and generative multi-agent systems.

3. Use Case and Problem Formulation: Dual-Camera Road Obstacle Recognition

This section introduces the motivating use case and formalizes the multi-agent perception setting addressed in this paper. We focus on road obstacle recognition for autonomous driving, where an ego vehicle must detect obstacles under long-range, low-contrast, and partially occluded conditions. While state-of-the-art object detectors achieve strong performance under favorable conditions, their reliability degrades significantly for small or distant objects, motion blur, strong illumination changes, and occlusions caused by vehicles or infrastructure. At the same time, stationary roadside cameras can provide advantageous viewpoints and improved object visibility, but their outputs arrive with non-negligible latency due to communication and processing delays. This combination creates an ideal setting for studying orchestration across heterogeneous agents with different latency and confidence profiles.

In this work, we model obstacle recognition as an object detection task, where each obstacle hypothesis consists of a class label (or a class probability distribution) and an associated bounding box. The goal of orchestration is to generate a detection output under real-time constraints while maintaining robustness to partial, noisy, or conflicting sensory evidence and to explicitly represent uncertainty when the available information is insufficient for confident classification or localization.

3.1. Problem Motivation and Failure Modes in Ego-Only Perception

Ego camera perception provides low-latency situational awareness and is a common input modality for real-time driving systems. However, ego-only detection exhibits several persistent failure modes:

• Long-range degradation: Distant obstacles occupy few pixels, reducing discriminative features and increasing confusion with background patterns.
• Occlusion and partial visibility: Obstacles may be partially blocked by other vehicles or road geometry, resulting in fragmented detections or missed objects.
• Illumination changes: Shadows, glare, and nighttime conditions can distort appearance and reduce detector confidence.
• Motion blur: Rapid ego motion and vibration degrade image quality, especially at higher speeds.
• Domain shift: Weather, camera settings, and geographic differences lead to distribution shift and degraded generalization.

These failure modes are especially critical for safety because they can cause missed detections of obstacles that require early reaction. Prior work has explored multi-sensor fusion and multi-view perception to mitigate these effects [20,21], but infrastructure-assisted perception introduces additional orchestration challenges, including latency, uncertain association, and heterogeneous model reliability.

3.2. Infrastructure-Assisted Perception Scenario

This work considers an infrastructure-assisted perception scenario in which ego sensing is augmented by a stationary roadside camera. The setup, illustrated in Figure 1, assumes a dual-camera configuration combining an ego-mounted sensor with an external, fixed-viewpoint camera.

Stationary cameras provide several complementary advantages over ego-mounted sensors: (i) stable mounting that reduces motion blur, (ii) broader or elevated viewpoints, and (iii) earlier visibility of obstacles that may be occluded from the ego perspective. As a result, stationary sensors can act as high-confidence but temporally delayed evidence sources, supplying obstacle hypotheses that may refine or correct ego perception.

Despite these advantages, the integration of stationary camera outputs introduces several constraints that are central to the orchestration problem:

• Latency and jitter: Sensor outputs are delayed by acquisition, processing, and network transmission, and they may arrive with variable delay.
• Association uncertainty: A stationary camera’s detection must be matched to the ego camera’s observation of the same physical object across different perspectives.
• Partial coverage: Not all environments are instrumented; stationary evidence may be intermittent.
• Heterogeneous failure modes: Stationary detectors can also fail due to weather, occlusion, or miscalibration, and their confidence may not be directly comparable with ego confidence.

These constraints motivate an orchestration layer that can reason probabilistically about the reliability and relevance of external evidence under timing uncertainty.

3.3. System Overview and Multi-Agent Perception Pipeline

We consider a pipeline with four primary agents that produce complementary outputs:

• An ego detection agent that performs real-time object detection from the vehicle’s forward-facing camera.
• A geometric localization agent that maps detection outputs into a shared spatial representation (e.g., using homography or camera calibration).
• A stationary detection agent that detects obstacles from a roadside camera and communicates predictions to the vehicle.
• A cross-view association agent that establishes correspondence between the ego and the stationary detections.

These agents feed a probabilistic orchestrator that selects or fuses decisions under uncertainty and latency constraints. The orchestrator produces the final obstacle label and confidence score, and it may trigger a learning loop to improve the ego detector over time.

Table 1. Agent role summary.

Agent	Function	Typical Latency	Typical Confidence	Output
A1: Ego Detector	Real-time obstacle detection on ego camera	Low	Medium (drops at distance/occlusion)	Bounding boxes + class probabilities
A2: Ego Localization	Map ego detections to ground-plane/shared coordinates	Low	Medium	Localized boxes/estimated position
A3: Stationary Detector	Obstacle detection from roadside camera	Medium	High (better long-range)	Bounding boxes + class probabilities
A4: Cross-View Association	Match detections across ego and stationary views	Medium	High when confident match exists	Association score + matched object IDs

summarizes the roles of the individual agents involved in this perception pipeline, including their primary functions, typical latency characteristics, confidence profiles, and output representations.

The orchestrator consumes the heterogeneous outputs of these agents, including class probability distributions, spatial bounding boxes, localization signals, cross-view association scores, and latency metadata. Based on this information, it produces a consolidated detection result comprising a final object class label (or distribution), a selected or fused bounding box, an explicit uncertainty estimate, and a decision trace that supports explainability and audit.

3.4. Scope and Assumptions

To keep the problem well defined and operationally realistic, we adopt the following scope assumptions:

In scope:

• Vision-based detection/classification of road obstacles using ego and stationary cameras.
• Confidence-aware and latency-aware orchestration across heterogeneous agents.
• Cross-view association and reliability scoring.
• Offline/nearline adaptation through pseudo-labeling and knowledge distillation.

Out of scope:

• End-to-end motion planning and control.
• Multi-modal fusion with lidar/radar (future work).
• High-definition mapping and full SLAM integration.
• Online model updates without validation (unsafe in practice).

We assume that cameras are calibrated and that a planar road approximation is acceptable for homography-based projection within the region of interest. We also assume that stationary camera connectivity is available, but subject to variable latency.

Taken together, the dual-camera setting, agent roles, uncertainty sources, and scope limitations introduced in this section define a perception problem characterized by heterogeneous evidence, temporal misalignment, and non-uniform reliability. Addressing this setting requires an orchestration mechanism that is capable of integrating delayed and uncertain information while preserving real-time responsiveness. The next section introduces the probabilistic orchestrator architecture and decision policy developed to meet these requirements.

4. Probabilistic Orchestrator Architecture and Decision Policy

This section presents the probabilistic orchestrator that coordinates heterogeneous agents under uncertainty and latency constraints. The orchestrator integrates predictions from ego and stationary detection agents, geometric localization, and cross-view association into an uncertainty-aware decision state. Unlike deterministic orchestration (e.g., hard thresholds or fixed precedence rules), the orchestrator produces calibrated confidence estimates, preserves uncertainty when evidence is weak, and supports a gated feedback loop for improving agents over time (Section 5).

As was described in the Introduction, we meet challenging problems related to weak object recognition when only cars’ cameras are leveraged for inferencing/object classification. When a camera (stationed on the car) is capturing video from far enough away (more than 60–80 m), the size of an object on the road (in pixels) is small enough (less than 16–32 pixels for the standard 640 × 640 image resolution used for object classification/segmentation tasks) that it can be almost impossible to classify (cars, humans, animals, etc.) with sufficient accuracy. To improve this accuracy (see [24,25]), an additional stationary camera can be arranged near the pedestrian crossing or near the traffic lights.

Multi-view fusion with a moving car camera + a fixed intersection camera can boost recall and reduce false positives. The key is to put both cameras’ detections into the same space/time and then associate them robustly. The stationary camera captures images from compressed video stream (in h264/265 format, 1–2 Mbit/s) and sends to the client (the car’s client app running on the car’s CPU + GPU) using a client–server communication technology based on 5G 1Gbit/s communication architecture. Obviously, 5G adds latency and jittering, but below we will discuss how to achieve the temporal alignment of two agents: the car’s images and object recognition (the student agent) and object recognition from the stationary camera (the teacher agent).

The appropriate agents (teacher/student) and the matching agent (using ground-plane fusion algorithms explained in Appendix A) provide ID tracking of student- and teacher-recognized objects. If there are (in the current small time slot of ~100 ms) two objects (still not classified) from both the teacher and student cameras, the algorithm should choose the appropriate object and its bounding box from the teacher’s object queue, and the corresponding student object should be deprecated from the student’s queue.

This information is obtained by our central agent, named the orchestrator decision-maker. The goal of this central agent is to manage appropriate student, teacher, and matcher agents to achieve optimal object classification. Moreover, this orchestrator agent manages the fine-tuning learning process of popular datasets that are used for object classification with autonomic cars and self-driving assistance (YOLO, Ultralytics models, etc.).

The main target of the present publication is the orchestrator decision-maker agent. Below, we provide the technical description, innovative algorithms, and model that allow us to create an orchestrator decision-maker agent. We suppose that this orchestrator concept can be used for a wide class of neural network models, for example, reinforced teacher–student models for training of robot walking and robot navigation (reinforced teacher–student models are used very widely in NVidia’s Isaac Robot simulators). For robot walking, robot vision, and robot navigation, the tasks often involve two different agents solving the same problem but with different natures: for example, the first agent calculates the robot’s next step based on a deterministic algorithm (stereometric vision, 3D homography analysis, etc.), but the second agent is based on a reinforced teacher–student network (probabilistic model) and provides the solution for the robot’s next step at the same time. Our model of the orchestrator agent can make the decision in this situation. We intend to describe the usage of the orchestrator agent for robotics navigation in a future article.

In summary, our orchestrator agent can be used as a decision-maker when there are multiple agents solving the same tasks at the same time and providing different decisions.

4.1. Cross-View Association and Shared Coordinate Representation

The orchestrator is designed to satisfy four operational goals:

• Robustness under uncertainty: Reconcile conflicting predictions and preserve uncertainty when evidence is insufficient.
• Latency-aware decision-making: Produce real-time outputs even when higher-confidence evidence is delayed.
• Association-aware fusion: Incorporate cross-view correspondence reliability as first-class evidence.
• Controlled learning enablement: Generate decision traces and high-confidence supervision signals without amplifying errors.

These goals reflect the central tension in infrastructure-assisted perception: responsiveness requires fast ego decisions, while reliability improves when external evidence becomes available—provided that the association is correct and confidence is calibrated.

4.2. Teacher–Student Orchestrator Design, Flow, and Realization

What is the orchestrator architecture, and what components are needed to create an orchestrator decision-maker?

An orchestrator is a very improved student agent (improved neuron object classification). The following standard student components must be improved to transform the standard student agent into an orchestrator agent. Of course, the student agent works in parallel to the orchestrator. We do not replace the simple student agent with the orchestrator decision-maker.

The orchestrator’s improved components are as follows:

• Fine-tuning/transfer learning for YOLO.
• Knowledge distillation (KD) and KD integration.
• KD loss functions (improved YOLO loss function).

Fine-tuning and transfer learning for YOLO

We start from one of the following pre-trained checkpoints:

• The public ones: yolov8n.pt, yolov8s.pt, yolo11n.pt, etc.
• The car’s own camera model: carcam_yolov8n.pt.

Then, we keep those weights and train further on our new dataset (real + pseudo), where “real” is the standard YOLO dataset and “pseudo” is obtained from the teacher camera.

Transfer learning can be described as follows:

• We do not re-learn basic things, e.g., “What is an edge? What is a car shape?”
• We adapt to the following:

  ○

  Our camera viewpoint (car + intersection geometry).

  ○

  Small/distant humans and cars.

  ○

  Optionally freezing early layers (mainly to adapt high-level features and the head).

Knowledge Distillation (KD) and KD integration

Instead of using only the hard 0/1 labels from the teacher, more advanced setups use teacher logits (full probability over classes) and a special loss to match student predictions. Out-of-the-box Ultralytics does not expose a simple “teacher model” argument, so this would require a custom training loop, which we will treat as the next step.

What is knowledge distillation?

In normal pseudo-labeling, the following takes place:

• The teacher says, “This is PERSON”.
• The student uses a hard label: class = person, 1 vs. 0.

In knowledge distillation, the teacher provides soft labels:

• The teacher outputs the following:

  ○

  Class probabilities:

  P_teacher = [person: 0.82, car: 0.17, dog: 0.01, …].

  ○

  Bounding-box regression confidences.

• The student is trained to match the entire probability distribution, not just the winning class.
• This contains much richer information, especially for ambiguous cases (e.g., small, distant, or partially occluded objects).
• KD was invented for training small models from large models, and it works very well for our case:

• Stationary camera—stronger teacher (better angle, stable, less motion blur).
• Car camera—weaker student (far objects, motion blur, etc.).

How does KD fit the dual-camera scenario?

The pipeline described earlier performs tracking + cross-camera association:

Car camera → low confidence → candidate sample.

Stationary camera → same track → high confidence → teacher logits available.

We pair each ego detection with its corresponding teacher detection (via the track ID + ground-plane fusion).

This process is paired cross-view distillation.

KD Loss Functions (improved YOLO loss function)

YOLO internal loss has three components:

• Classification loss.
• Box regression loss.
• Objectness loss.

KD adds new terms for the orchestrator loss function:

• Classification KD Loss:

Formula of KD Loss component:

$${\mathcal{L}}_{\mathrm{KD-cls}}={\tau }^2\cdot KL\left(\mathrm{softmax}\right(z_t/\tau ) , \mathrm{softmax}(z_s/\tau \left)\right),$$

where:

Teacher logits (vector of un-normalized scores, one per class).

$z_t$-Teacher logits (vector of un-normalized scores, one per class).

$z_s$-Student logits.

$KL(P,Q)$-Divergence between distributions $P$ and $Q$.

KL divergence is:

$$KL(P\mid \mid Q)=\sum _i{P}_i \mathrm{l}\mathrm{o}\mathrm{g}{\displaystyle \frac{P_i}{Q_i}}$$

2.

Bounding-Box Distillation Loss

Bounding-box loss component formula:

$${\mathcal{L}}_{\mathrm{KD-bbox}}={\lambda }_{\mathrm{box}}\cdot SmoothL1(B_s,B_t)$$

where:

• $B_s$ = student bounding-box prediction.
• $B_t$ = teacher bounding-box prediction.

Boxes are usually in the following format:

$$[x_c,y_c,w,h]$$

or YOLO’s internal parameterization.

We now describe the meaning of bounding-box distillation.

The stationary teacher has a better viewpoint → more accurate boxes.

Thus,

• The student (car camera) learns better geometry, even when their own camera view is poor.
• This is especially helpful for the following:

  ○

  Distant pedestrians.

  ○

  Side-view objects.

  ○

  Partially occluded objects.

  ○

  Small objects (<16 px).

3.

Objectness Distillation Loss

Formula of object loss component:

$${\mathcal{L}}_{\mathrm{KD-obj}}={\lambda }_{\mathrm{obj}}\cdot (p_{\mathrm{obj}}^s-p_{\mathrm{obj}}^t{)}^2$$

Why is objectness KD important?

Especially in far-distance scenes, the following points are relevant:

• The ego (car) camera obtains more false positives (reflections, poles, lights, etc.).
• The stationary camera sees objects more clearly → gives reliable objectness.

Teaching students to copy the teacher’s objectness means the following:

• Fewer false positives.
• Stronger positive signals for distant but real objects.
• Better small object recall.

Full Orchestrator Loss and how KD Loss components will be integrated with YOLO’s normal loss

Final orchestrator student loss:

$$L_{\mathrm{total}}=L_{\mathrm{YOLO}}^{cls}+L_{\mathrm{YOLO}}^{box}+L_{\mathrm{YOLO}}^{obj}+\alpha L_{\mathrm{KD-cls}}+\beta L_{\mathrm{KD-bbox}}+\gamma L_{\mathrm{KD-obj}}$$

The purpose of each component is illustrated in

Table 2. The purpose of each loss component.

Component	Purpose
YOLO classification	Student still learns hard labels
YOLO bounding box	Student learns from their own view
YOLO objectness	Student filters own FP/TP
KD-cls	Learns teacher’s “soft opinion”
KD-bounding-box	Learns teacher’s geometry
KD-obj	Learns teacher’s sense of objectness

.

4.3. Mathematical Model of the Probabilistic Orchestrator (Under Simplifying Assumptions)

In this section, we describe the mathematical model of the probabilistic orchestrator. Our orchestrator agent analyzes image data obtained from different sources. In our case, the first source is periodic images captured by the ego camera, and the second source is the stationary camera, which also sends captured images to the orchestrator. So, the first component of the orchestrator is the storage of two different kinds of image data (from two different cameras). The orchestrator tries to analyze which image characteristics are better suited for use.

The second component of the orchestrator (named student) is an inference neural network (CNN) with KD (knowledge distillation). This student CNN performs the following functions:

• Regular inference CNN for image classification and object/bounding-box detection.
• Transfer learning CNN with improved KD (knowledge distillation).

The pre-training process is based on the teacher–student transfer learning model (fine-tuning learning). Details of the teacher–student model for probabilistic orchestrator are described below. For our practical cases, we used the Ultralytics YOLO v8 dataset, pre-training the YOLO network and the YOLO training API. Our mathematical model is independent of which image inference CNN (convolution network) was leveraged.

The agent that sends detected images to the orchestrator is the teacher. In our practical case, the teacher agent consisted of a stationary camera and image inference with the teacher convolution network (usually, it comes with a similar CNN architecture, like the student). Both sources (cameras) capture the same scenes or obstacles (in many cases), but the object confidence of the teacher’s inference is usually better than that of the student.

Moreover, the orchestrator includes an important component: the image matcher (mathematical and algorithmic aspects of which are described in Appendix A). The matcher is intended to find similar pairs of images that were identified by both the teacher and student CNNs. Such image pairs should be synchronized by timestamps (from each camera). Furthermore, the bounding boxes of appropriate objects/obstacles detected in each image should have good matching in global ground system coordinates (details will be considered in Appendix A).

Considering the probabilistic orchestrator as a more abstract object, we present two fundamental characteristics of the orchestrator that seem quite obvious but will be fully proven in future work. These propositions are proposed and formulated in two lemmas:

• Lemma 1: The probabilistic orchestrator makes better decisions than the deterministic decision-maker agent.
• Lemma 2: The probabilistic orchestrator (based on our KD training and teacher–student transfer learning model) provides much better object classification confidence.

Based on these lemmas, as well as the tests obtained by KD pre-trained YOLO v8 and using datasets of pedestrian/car images captured by the stationary camera (positioned near a crossroads), we can guarantee that our orchestrator provides the classification results with much better confidence, even when a road’s obstacles are far from the ego camera.

4.4. Algorithm General Description

The main goal of our article is to demonstrate and present the advantages of the probabilistic orchestrator. The probabilistic orchestrator (student side) gets input data from both the ego camera frames and the stationary camera captures images (teacher side). The following steps describe the work of the probabilistic orchestrator. A full explanation in detail and flow will be described below in the central algorithm.

General description of probabilistic orchestrator work by steps:

• Receive current ego image and corresponding prediction $\left(p_e,b_e^t\right)$, where $p_e$ is object confidence and $b_e^t is the bounding box.$(Student side).
• Put $pair \left(p_e,b_e^t\right)$ into the student confidence queue data structure SQ (student queue).
• Receive the current stationary camera image and corresponding prediction $\left(p_s,b_s^{t-\Delta }\right),$ where $p_s$ is the object confidence and $b_s^{t-\Delta } is the bounding box.$ (Teacher side).
• Put $pair \left(p_s,b_s^{t-\Delta }\right)$ into the teacher confidence queue data structure TQ (teacher queue).
• Matching both the student confidence queue and the teacher confidence queue accordingly: If $\Delta < T_{max}$ and $p_e<p_s$ set $p_o=p_s$, otherwise, set $p_e$.
• Transfer learning step: Provide the transfer learning process (student side) by means of stationary camera images with confidences $\left(p_s,b_s^{t-\Delta }\right)$.

4.5. Physical Orchestrator Model and Simplified Orchestrator Simulator (Math Model of Both Cameras Simulation)

Above in Section 4.4, steps 1–6 include probabilistic orchestrator work. This is a Physical Orchestrator model, where ego and stationary cameras are real cameras arranged at the car and at the road crossing accordingly. To show the probabilistic orchestrator advantages compared to the deterministic orchestrator and approve it by our computations, we will replace this physical model with a simplified simulator. Here is the simplified orchestrator model (simulator or math model) description:

• Stationary camera: Stationary camera simulation will be replaced with a video stream (taken by a YouTube link). Instead of the stationary camera capturing, the central algorithm (math simplifying model) extracts images from this video with a distance to an object not farther than 5 m distance from the camera (this corresponds to the cross road model) and puts corresponding $pair \left(p_s,b_s^{t-\Delta }\right)$ into the teacher queue structure TQ with appropriate time stamps.
• Ego camera: The images extracted from the video file will also be used for ego camera simulation, but they should pass the following procedure: each image (extracted from an appropriate video file) will undergo the following transformation:

  ▪

  Downscaling with factor scales: 0.2 and 0.1 (that will correspond to distances of 50 and 80 m approximately).

  ▪

  Upscaling with factors 5 and 10. The images should be returned to the same pixel sizes to pass the student inference step (as it was described for the ego camera).

Our Orchestrator simulation and our computations will be done with the simplified orchestrator model (math model). Of course, this model (compared to a physical one) does not take into account the following factors:

• Distant objects experience (optical blur):

  ▪

  Lens blur.

  ▪

  Atmospheric blur.

• Sensor noise amplification. Small objects obtained from the ego camera at a far distance (50–100 m) lead to fewer pixels of the actual object, and this can amplify high-frequency noise.
• Resampling (downscaling and upscaling) can create additional problems of image aliasing.
• Contrast reduction can happen with fog/haze at a far distance.

Problems 1–4 are not taken into account in our described algorithm and computations. We are going to analyze them in future work. It is very important to note:

Again, all our following computations and results related to probabilistic orchestrator work, teacher–student inference, teacher–student confidential data matching, transfer learning of student YOLO network, etc., will be obtained only within the framework of the simplified orchestrator model described (above) named simplified orchestrator simulator.

The central algorithm and the associated orchestration flow in Figure 2 operationalize the probabilistic framework developed throughout this section. Together, they specify how heterogeneous agent outputs—ego predictions, delayed infrastructure evidence, localization signals, and association confidence—are calibrated, weighted, and fused under explicit latency and reliability constraints. Beyond producing uncertainty-aware obstacle detections, the orchestrator exposes internal decision traces and gating signals that regulate when external evidence may be used to support learning. These mechanisms form the basis for the feedback-driven adaptation process described in the next section.

5. Feedback-Driven Adaptation: Teacher–Student Fine-Tuning and Knowledge Distillation

While the probabilistic orchestrator improves decision quality at inference time by fusing heterogeneous agent outputs (Section 4), its longer-term value is in enabling controlled improvement of weaker agents. In our use case, the ego detector (Agent A1) exhibits degraded performance for distant and partially occluded obstacles, whereas the stationary detector (Agent A3) typically provides higher-confidence predictions but with a delay. This asymmetry naturally motivates a teacher–student adaptation mechanism, in which stationary predictions serve as supervision to improve the ego model—provided that (i) the association is reliable and (ii) teacher confidence is high.

A central challenge is that naïve self-training can amplify errors: teacher failures, mismatched associations, or delayed evidence can generate noisy pseudo-labels that degrade performance over time. We therefore design an orchestrator-gated learning loop that uses explicit quality and safety gates to decide when learning is allowed. This section describes pseudo-label curation, student fine-tuning, knowledge distillation, and controlled deployment of updated models.

5.1. Learning Loop Overview

The adaptation loop is triggered and managed by the orchestrator, which produces both fused decisions and decision traces. High-confidence teacher outputs are gated by association and uncertainty constraints before being used for student fine-tuning and knowledge distillation (Figure 3).

The learning loop consists of four stages:

• Candidate collection: Identify time steps where stationary predictions are available and association confidence is high.
• Pseudo-label gating: Filter candidates using teacher confidence, latency constraints, and uncertainty thresholds to reduce label noise.
• Student adaptation: Fine-tune the ego detector using pseudo-labels, optionally combined with a small amount of ground-truth labeled data.
• Controlled deployment: Evaluate and register the updated model; deploy only if it improves key metrics without increasing false alarms beyond an acceptable limit.

• Crucially, we do not perform uncontrolled online learning: model updates are performed offline or nearline and are deployed only after validation and gating checks.

5.2. Student Model and Training Objective

The student is the ego detector $f_{\theta }$ (Agent A1). The teacher is the stationary detector $g_{\varphi }$ (Agent A3). The goal is to improve ego detection quality in difficult cases (distance/occlusion) while preserving responsiveness and controlling false alarms.

We optimize a combined objective:

$$\mathcal{L}={\mathcal{L}}_{det}+{\lambda }_{KD}{\mathcal{L}}_{KD}$$

where

• ${\mathcal{L}}_{det}$ is the standard detector loss under pseudo-label supervision;
• ${\mathcal{L}}_{KD}$ transfers teacher soft labels to the student;
• ${\lambda }_{KD}$ controls the distillation strength.

5.3. Detection Loss (Pseudo-Label Supervision)

For a pseudo-labeled training instance $(x_e^t,{\tilde{y}}^t,{\tilde{b}}^t)$, the detector loss is

$${\mathcal{L}}_{\det }={\mathcal{L}}_{\mathrm{cls}}(\widehat{y},\tilde{y})+{\lambda }_{\mathrm{box}}{\mathcal{L}}_{\mathrm{box}}(\widehat{b},\tilde{b})+{\lambda }_{\mathrm{obj}}{\mathcal{L}}_{\mathrm{obj}}$$

where

• $\tilde{y}$ and $\tilde{b}$ are the pseudo-label class and box from the teacher;
• $\widehat{y}$ and $\widehat{b}$ are student outputs;
• ${\mathcal{L}}_{\mathrm{cls}}$ is typically cross-entropy or focal loss;
• ${\mathcal{L}}_{\mathrm{box}}$ is IoU/GIoU/DIoU loss;
• ${\mathcal{L}}_{\mathrm{obj}}$ models objectness confidence.

Because pseudo-labels may contain residual noise, we optionally apply confidence-weighted supervision:

$${\mathcal{L}}_{\mathrm{cls}}\leftarrow w^t\cdot {\mathcal{L}}_{\mathrm{cls}} \mathrm{where}{ w}^t=\underset{y}{max} p_s^{\prime t}(y)$$

so that high-confidence teacher predictions contribute more strongly to training.

5.4. Knowledge Distillation Loss (Soft Label Transfer)

To transfer richer information than hard pseudo-labels, we distill the teacher’s soft distribution into the student output distribution. We use temperature-scaled distillation [14]:

$${\mathcal{L}}_{\mathrm{KD}}=T^2\cdot KL\left(Softmax(z_s/T) \parallel Softmax(z_e/T)\right)$$

where

• $z_s$ and $z_e$ are teacher and student logits;
• $T$ is the distillation temperature;
• $KL(\cdot )$ is the Kullback–Leibler divergence.

The distillation term encourages the student to match the teacher’s uncertainty, which is especially valuable for borderline cases (e.g., distant obstacles) where the teacher provides a smoother probability structure than hard labels alone.

6. Implementation

This section describes the implementation of the dual-camera multi-agent pipeline, including agent modules, orchestration runtime, cross-view association, pseudo-label logging, and the controlled adaptation workflow. The implementation is designed as an offline/nearline proof of concept that reflects practical constraints of infrastructure-assisted perception, including variable latency, imperfect association, and the need for safe model update gating.

6.1. System Components and Runtime Pipeline

The pipeline is implemented in Python 3.13 as a modular multi-agent system, where each agent exposes a consistent interface:

• Predict (input) → output;
• Metadata → confidence/latency statistics;
• Trace → evidence and internal signals used for auditing.

Figure 4 summarizes the testing (inference) pipeline, highlighting the real-time ego detection flow and the integration of delayed stationary evidence through the probabilistic orchestrator.

The orchestrator consumes agent outputs and produces

$${\mathcal{D}}^t=({\widehat{y}}^t, p_o(y\mid {\mathcal{E}}^t), {\widehat{b}}^t, {\widehat{u}}^t, {\tau }^t)$$

as defined in Section 4.2, where ${\tau }^t$ is a decision trace.

Runtime timing model:

• Ego inference runs synchronously, frame-by-frame, at time index $t$.
• Stationary camera predictions arrive asynchronously with delay $\Delta$ (and optional jitter).
• The orchestrator performs event-driven fusion whenever new stationary evidence becomes available within the fusion window $\left[t-T_{max},t\right]$.

This execution model ensures that ego decisions can be produced without waiting for delayed evidence while still benefiting from stationary corrections when they arrive in time.

6.2. Ego Detector (Student Model)

Model: The ego detector (Agent A1) is implemented using a YOLO-family object detector (e.g., YOLOv5/YOLOv8) configured for obstacle detection. The model is initialized from a publicly available pre-trained checkpoint and adapted to the target domain through fine-tuning using pseudo-labels curated by the orchestrator.

Input Preprocessing:

• Resize to $\left[W×H\right]$ (e.g., 640 × 640).
• Normalize pixel values to $\left[0,1\right]$.
• Optional augmentation for robustness (color jitter, blur, random crop).

Output: Per frame $t$, the ego detector outputs the following:

• Bounding boxes $b_e^t$;
• Class distribution $p_e(y\mid x_e^t)$;
• Objectness score $p\left(\mathrm{obj}\right)$.

6.3. Stationary Detector (Teacher Agent A3)

The stationary detector uses the same model family but is treated as a higher-confidence teacher due to the following:

• Stable viewpoint;
• Reduced motion blur;
• Improved long-range visibility.

It outputs the following:

• Bounding boxes $b_s^{t-\Delta }$;
• Class distribution $p_s(y\mid x_s^{t-\Delta })$;
• Objectness scores.

• Communication Payload: Stationary predictions are transmitted along with metadata:

• Timestamp $t-\Delta$;
• Estimated delay $\Delta$;
• Calibration parameters or confidence summary statistics.

In the proof of concept, stationary delay is modeled as one of the following:

• Fixed delay;
• Stochastic delay with jitter: $\Delta \sim \mathcal{N}(\mu ,{\sigma }^2)$, truncated to positive values.

6.4. Geometric Localization (Agent A2) and Shared Coordinate Frame

To enable cross-view association, detections are mapped into a shared representation using planar homography projection.

Ego projection:

$${\mathcal{l}}_e^t={\Pi }_e(b_e^t;H_e)$$

Stationary projection:

$${\mathcal{l}}_s^{t-\Delta }={\Pi }_s(b_s^{t-\Delta };H_s)$$

where $H_e$ and $H_s$ are homography matrices computed from camera calibration. Projection is implemented using OpenCV routines. Because planar homography is approximate (non-planar road surfaces, calibration drift), the projected localization is used as a coarse alignment signal, not as a deterministic match.

This will be described in detail in Appendix A.

6.5. Student Fine-Tuning and Distillation Workflow

Adaptation is triggered when a minimum dataset size $N_{min}$ is reached or after a fixed collection window $W$. Fine-tuning runs offline/nearline for a bounded number of epochs and produces a new student checkpoint $f_{{\theta }^{\prime }}$.

The student is trained using

$$\mathcal{L}={\mathcal{L}}_{det}+{\lambda }_{KD}{\mathcal{L}}_{KD}$$

as defined in Section 5.3 and Section 5.4. The training workflow includes the following:

• Checkpoint versioning;
• Validation-based early stopping;
• A deployment gate requiring recall improvement while bounding false positive increase and latency regression (Section 5.4).

7. Evaluation

This section evaluates the proposed probabilistic orchestrator and the feedback-driven adaptation loop on the dual-camera obstacle recognition task. We compare against ego-only perception and deterministic orchestration baselines, and we analyze robustness under long-range detection, occlusion, and variable latency. We further quantify the benefit of the teacher–student adaptation loop through ablation experiments.

7.1. Evaluation Goals

We designed the evaluation to answer four questions:

• Q1. Does probabilistic orchestration improve decision quality compared with ego-only detection under long-range and occlusion conditions?
• Q2. Does probabilistic orchestration outperform deterministic fusion/state-machine baselines under delayed stationary evidence?
• Q3. Does the teacher–student learning loop improve ego detector performance over time without increasing false alarms?
• Q4. How sensitive is performance to latency and association uncertainty?

7.2. Dataset and Experimental Setup

Data sources and split

We evaluated on a dataset of synchronized ego and stationary camera sequences containing obstacle events under diverse conditions (distance variation, illumination changes, partial occlusions). The dataset was split into the following subsets:

• Training set: Used for initial ego fine-tuning (if applicable) and for collecting pseudo-labels.
• Validation set: Used for confidence calibration, threshold tuning, and gating parameter selection.
• Test set: Held out for final evaluation and latency sensitivity analysis.

Where ground-truth labels were unavailable for all sequences, we relied on partial annotation for key events and used stationary detections as weak supervision for evaluation of long-range improvements (clearly stated as a limitation).

7.3. Baselines

We compared against the following baselines:

• B1: Ego-only detector.
• The ego detector (A1) without stationary assistance.
• This baseline reflects common camera-only real-time perception.
• B2: Deterministic orchestrator (rule-based/state machine).

A deterministic fusion policy that selects stationary predictions when available and uses ego predictions otherwise, using hard thresholds and fixed precedence rules. This represents common system engineering practice.

• B3: Probabilistic orchestrator (ours, no learning).

The probabilistic orchestrator described in Section 4, with calibrated confidence, association-aware weighting, and latency-aware mixing, but without the adaptation loop.

• B4: Probabilistic orchestrator + teacher–student learning (ours, full).
• B3 plus the gated pseudo-labeling and knowledge distillation adaptation loop described in Section 5.

7.4. Experimental and Synthetic Data

One of the options that allows for human-in-the-loop validation or formal safety approvals is the following: Instead of taking only real stationary camera capture, we take recorded video with very good object confidence (e.g., pedestrians, cars) for testing and validation. First, we extract key frames (from video MPEG files), and these images simulate the teacher (stationary camera). In parallel, the student task obtains the same images (downscaled as though they were captured as distant objects). Moreover, we can simulate 5G network jittering at 10–100 ms (by Gauss/Poisson distribution). This is an excellent simulator that can be used for KD learning and our probabilistic orchestrator. In fact, we used this simulator for our results.

7.5. Latency Modeling

Stationary evidence delay $\Delta$ was evaluated under the following:

• Fixed delay conditions (e.g., 100, 300, 500 ms);
• Jittered delay conditions (e.g., $\Delta \sim \mathcal{N}(\mu ,{\sigma }^2)$).

This allows for the analysis of robustness under realistic communication uncertainty.

7.6. Results

We developed the project Prob Orchestrator; all presented results for probabilistic orchestrator were obtained by means of the probabilistic orchestrator model (based on the simplified simulator described in Section 4.4.) This Github project can be found at the link: https://github.com/akojukhov-bit/Prob-Orchestrator.git (accessed on 1 March 2026).

Also, the open-source dataset Custom Training YOLOv8n was used to detect vehicles, pedestrians, and signboards (https://github.com/RezuwanHassan262/Custom-Training-YOLOv8-to-detect-Vehicle-Pedestrians-and-Signboards/tree/main (accessed on 1 March 2026)).

This open-source dataset was taken as a basic transfer learning project, and we adapted it for simulation with our teacher–student models for the probabilistic orchestrator.

The dataset was developed by extracting only 150 frames (setting the stride = 100) from a YouTube video and was annotated mostly using Roboflow’s auto-labeling feature, using both Python scripts and the Roboflow platform.

We fine-tuned a pre-trained YOLOv8n model, as described in the Readme.md document in the original project.

7.6.1. Ego/Stationary Cameras’ Model Training Based on the Teacher–Student Transfer Learning Simulation

Data for teacher–student transfer learning were obtained from a YouTube video (https://youtu.be/7HaJArMDKgI (accessed on 1 March 2026)). The following steps explain how the data were extracted and how the extracted video images were used for both the student and teacher models:

• Key or IDR image extraction (from H.264vformat encoding).
• Making inference (classification and bounding boxes), this side of the teacher model (stationary camera simulation). Next, we chose only such images where the sizes of the obtained bounding boxes (according to the detected objects) corresponded to distances of less than 5–10 m. Then, the images were extracted according to our simulation of the stationary camera and teacher model (we extracted ~150 images). Below, the stationary image examples were presented: Figure 5. Photo 1, Figure 6. Photo 2 and Figure 7. Photo 3. The image on Figure 8. – this is example that ilustrates the Bounding Boxes inference (stationary camera, Teacher side) for pedestrians and cars objects. The bounding boxes coordinates were used estimating distans from stationar camera to crossroad obstacles.
• The images used for the teacher model (stationary camera) were sent to the student model (ego camera), but with appropriate preprocessing of distant-object simulation. This means that we transformed appropriate images before they were sent to the student model. We used a simple distant-object simulation based on downscaling and immediately upscaled the images. We used two scale factors according to 100 and 50 m distances from the ego camera. The following examples: Figure 9 (downscale with factor 0.2, distanse 50 m), Figure 10 (downscale with factor 0.2, distanse 50 m), Figure 11 (downscale with factor 0.2, distanse 50 m) and Figure 12 (downscale with factor 0.1, distanse 100 m)–these images were undergone downscale and upscale and they simulate ego camera input for transfer learning.
• We used the latency simulation on the teacher side (stationary camera). The evidence delay $\Delta$ was evaluated under the following:
  • Fixed delay conditions (e.g., 100, 300, 500 ms);
  • Jittered delay conditions (e.g., $\Delta \sim \mathcal{N}(\mu ,{\sigma }^2)$).

7.6.2. Original Images from Stationary Camera

In this section, we present input images obtained from a stationary camera video (teacher side). These images were extracted from the video file and used for inference, simulating a stationary camera setup on the teacher side.

7.6.3. Scaled (Far Object Distance) Images

This section is devoted to the illustration of images obtained through downscaling and upscaling procedures that simulate an egocentric camera perspective (student side, distant objects).

7.6.4. Loss Analysis Results and Comparison Chart

Figure 13 shows loss components, their comparison, and analysis: classification loss and bounding-box loss for average loss (before vs. after), where “Before” represents the teacher model’s inference for close distances (5–10 m), and “After”—with scale factors of 0.1 and 0.2 for far distances (50–100 m)—represents the student-side detection. The results indicate that the loss and confidence achieved by the student model are good enough (especially for Scale 2, ~50 m distance) and similar to those achieved by the stationary camera. This indicates that student transfer learning with KD was carried out successfully.

The following data provides a comparison report: confidence average for near and far distant object detection (obtained by testing our model).

There were done Classification Loss and. Bounding-Box Loss Analysis.

The following compare was done: Before Scaling vs. After Scaling (50 and 100 m distances).

• Regular inference (50 m distance to objects, ego camera, see Figure 14.):

  Classification Loss: 0.0860.

  Bounding-Box Loss: 0.0430.

  Total Objects: 2.

  Average Confidence: 0.9140.

• After Scaling (scale factor: 0.1, ~100 m distance):

  Classification Loss: 0.6526.

  Bounding-Box Loss: 0.3263.

  Total Objects: 3.

  Average Confidence: 0.3474.

  Change vs. Original:

    cls_loss: +0.5666 (+658.7%).

    box_loss: +0.2833 (+658.7%).

    detections: +1.

• After Scaling (scale factor: 0.2, ~50 m distance):

  Classification Loss: 0.2018.

  Bounding-Box Loss: 0.1009.

  Total Objects: 2.

  Average Confidence: 0.7982.

  Change vs. Original:

    cls_loss: +0.1158 (+134.6%).

    box_loss: +0.0579 (+134.6%).

    detections: +0.

• Regular inference (100 m distance to objects, ego camera, see Figure 15.):

  Classification Loss: 0.3650.

  Bounding-Box Loss: 0.1825.

  Total Objects: 16.

  Average Confidence: 0.6350.

• After Scaling (~100 m distance):

  Classification Loss: 0.5661.

  Bounding-Box Loss: 0.2831.

  Total Objects: 10.

  Average Confidence: 0.4339.

  Change vs. Original:

    cls_loss: +0.2012 (+55.1%).

    box_loss: +0.1006 (+55.1%).

    detections: −6.

• After Scaling (~50 m distance):

  Classification Loss: 0.3695.

  Bounding-Box Loss: 0.1848.

  Total Objects: 15.

  Average Confidence: 0.6305.

  Change vs. Original:

    cls_loss: +0.0046 (+1.3%).

    box_loss: +0.0023 (+1.3%).

    detections: −1.

• Summary Statistics
• Average Classification Loss:

  Original: 0.3376.

  Scale 0.1 (10%): 0.6131 (+81.6%).

  Scale 0.2 (20%): 0.3726 (+10.4%).

• Average Bounding-Box Loss:

  Original: 0.1688.

• ~100 m distance (10%): 0.3066 (+81.6%).

  (~50 m distance (20%): 0.1863 (+10.4%).

7.6.5. Benefit of Distillation and Pseudo-Label Gating

Beyond B1–B4, there are some important baselines related to the feature level and activation map of the YOLO model with and without KD transfer learning. It is very important how the activation map results can explain why KD transfer learning improves far-object diagnostics.

We generated real activation maps from our YOLOv8 model using a forward hook (P3/P4/P5), and below we compare the ego-only vs. KD-trained approaches on the same frames for different distances: 30, 50, and 80 m.

Figure 14, Figure 15 and Figure 16 demonstrate activation maps before KD usage (pedestrians’ distances: 30, 50, 80 m).

Figure 16 demonstrates KD-trained activation maps (pedestrians’ distances: 30, 50, 80 m).

The activation maps related to the KD-trained YOLO network provide much better detection of pedestrian body images. Activation maps related to diagnostics for far-distance objects (before KD training) contain noise, and head features are less recognizable compared with KD learning, where pedestrian inference provides good confidence, even for 50–100 m distances.

7.7. Discussion of Findings

The results support three conclusions:

• Probabilistic orchestration improves robustness compared with ego-only and deterministic fusion, especially for long-range and occluded obstacles.
• Latency-aware mixing stabilizes decisions and avoids abrupt behavior under delayed evidence.
• Gated teacher–student adaptation improves ego performance over time while controlling false alarms, reducing reliance on infrastructure signals.

8. Discussion

This work argues that multi-agent perception systems should be designed as uncertainty-aware orchestration problems rather than deterministic routing problems. The dual-camera obstacle recognition use case highlights a recurring pattern in real-world agent ecosystems: some agents are fast but unreliable in edge cases (ego detector), while others are slow but more accurate (stationary camera). Effective systems must therefore (i) decide under incomplete evidence and (ii) improve future decision-making by learning from high-quality feedback. In this section, we discuss how the proposed probabilistic orchestrator addresses these requirements, how it generalizes beyond the case study, and what limitations remain.

8.1. The Contribution of Probabilistic Orchestration

This section describes the practical recommendations for how to leverage the probabilistic orchestrator.

• Self-driving vehicles, especially robotaxis and city buses, could be appropriate candidates for the usage of the probabilistic orchestrator. Why could our orchestrator, based on teacher–student KD transfer learning, be used for these self-driving vehicles? The corresponding vehicles, by definition, come with a 5G client–server network communication infrastructure, so it would be easy to integrate our improved transfer learning model (YOLO dataset, for example) for such configurations. As mentioned in Section 4.4, the roles of the teacher and student cameras for vehicle pair (i, j) will be changed depending on whether vehicle [i] or vehicle [j] is closer to the appropriate crossroads. This means that if we are matching the same objects/obstacles captured by both cameras from these vehicles, the teacher camera (temporary) will be situated at the closest car to this object/obstacle.
• For high-speed trains, it is very important to detect (classification, objectness, and bounding box) obstacles that are far off, because the braking distance of a train is very long. Here, the ego camera can be positioned on the train’s roof, and the stationary camera could be positioned at a railway station and/or railway crossing.

8.2. Generalization Beyond the Dual-Camera Use Case

Although evaluated on dual-camera obstacle recognition, the proposed orchestration pattern generalizes to a broad class of systems that combine the following:

• Fast/low-cost agents (real-time edge models);
• Slow/high-quality agents (infrastructure sensors, cloud services, heavy models);
• Association/verification agents (matching, checking consistency);
• An orchestrator that must decide under uncertainty and timing constraints.

• Potential applications include the following:

• Cooperative perception and V2X sensor networks.
• Robotics perception with multi-view cameras and remote supervision.
• Anomaly detection pipelines, where slow forensic analysis informs fast detection.
• Multi-sensor medical imaging triage pipelines.

The essential principle is unchanged: probabilistic orchestration enables robust decisions and safe adaptation when agents differ in reliability and latency.

8.3. Future Directions and Works

In this section, we propose future directions and works where probabilistic orchestration patterns could be applied to multi-agent GenAI workflows.

• Probabilistic orchestration patterns could be extended to ego car and stationary cameras for semi-uncertain situations where teacher images suffer from noise (vibrations, bad weather, fog, etc.).
• The infrastructure of self-driving cars (robotaxis, buses) is going to be developed as a multi-client–server 5G network (for a town/area), where each car sends ego camera information to the server and receives teacher camera data in return. For example, if one car is near a crossroads, it sends the ego camera to the server with very strong inference confidence. Another car that is far from this crossroads but driving in its direction can receive appropriate teacher detection data that can be used for both better inference and KD transfer learning, as described in this paper. We presume that this extension of our orchestrator will constitute a prospective model for future self-driving infrastructure.
• The probabilistic orchestration pattern could be extended to human robotics learning processes. The robot walking process is mostly learned in simulators (e.g., NVidia’s Isaac) by means of teacher–student models (RL models). Today, more and more works aim to fuse synthetic simulation learning (in robot simulators) with real-world model learning. Our orchestrator could be used for such a fusion model, and we are planning to research this in future work.

Other future directions include the following:

• Learned or adaptive gating policies that adjust thresholds under drift.
• Integration with additional sensors (lidar/radar) and temporal tracking.
• Richer uncertainty models (ensembles or evidential learning).
• Broader evaluation across environments and weather conditions.
• Extension to full multi-agent GenAI orchestration pipelines with verification and policy gating.

9. Conclusions

This paper introduces a probabilistic orchestrator for multi-agent perception pipelines that must operate under uncertainty, conflicting predictions, and variable latency. Motivated by dual-camera road obstacle recognition, we have shown how deterministic orchestration mechanisms—such as hard thresholds and rule-based state machines—are brittle when combining fast ego camera predictions with delayed but higher-confidence stationary camera evidence. Our orchestrator addresses these limitations by fusing agent outputs using calibrated confidence, cross-view association reliability, and latency-aware policies, producing uncertainty-aware decisions and decision traces that support auditability.

Beyond inference-time fusion, we integrated a controlled teacher–student adaptation loop within the orchestration layer. High-confidence stationary detections serve as teacher signals for fine-tuning and knowledge distillation of the ego detector, while orchestrator gating (association confidence, teacher confidence, latency constraints, and uncertainty thresholds) mitigates label noise and reduces the risk of error amplification. Evaluation across distance, occlusion, and latency conditions demonstrates that probabilistic orchestration improves robustness and stability compared with ego-only and deterministic baselines and that the gated learning loop further strengthens ego performance over time while controlling false alarms.

The proposed framework generalizes beyond the autonomous driving case study to broader multi-agent systems where evidence sources differ in accuracy and timing, including infrastructure-assisted perception, robotics, and multi-agent generative AI workflows. Future work will explore richer uncertainty models, learned gating strategies, online adaptation with stronger safeguards, and expanded evaluation across diverse environments and sensor configurations.