Resolving the Perceptual Aliasing Problem by Multi-Agent Reinforcement Learning

Abstract

Perceptual aliasing occurs when distinct environmental states generate the same observation even though different actions may be appropriate. This paper studies whether the observable presence of other agents can mitigate such ambiguity without adding an explicit communication channel or internal memory. We formalize other-agent perception as a refinement of the focal agent’s observation and characterize when the resulting augmented observations are action-consistent. The analysis distinguishes local resolution of observation-induced action conflicts from the stronger requirements needed for a global goal-reaching policy. We evaluate the mechanism with Profit Sharing and Q-learning in three grid environments and under variations in agent number, execution protocol, and sensor noise and compare it with a one-step-history baseline. The results show that other-agent cues can be useful when they create action-relevant distinctions, but they may also increase representation cost or fail when informative configurations are rare or unstable. Statistical comparisons treat non-terminating trials as right-censored observations rather than completed runs. Overall, the results support a conditional conclusion: other agents can provide useful perceptual information, but the benefit depends on environment structure, learner dynamics, action selection, and the temporal availability of the cue.

1. Introduction

Reinforcement Learning (RL) [1] acquires behavioral policies through interaction with an environment using rewards as evaluative signals. Its practical use, however, is often complicated by partial observability: distinct environment states can generate the same observation even when different actions are appropriate. This is the perceptual aliasing problem, which is naturally formulated within a Partially Observable Markov Decision Process (POMDP) [2]. Classical approaches address missing information by maintaining beliefs or by splitting observations with history [3,4]; more recent work uses recurrent or latent representations, and recurrent model-free RL can provide a strong baseline across many POMDPs [5].

Partial observability is also central to Multi-Agent Reinforcement Learning (MARL). Modern MARL research addresses it through several distinct mechanisms. Communication-learning methods exchange learned messages; centralized-training/decentralized-execution methods exploit additional information during training; recurrent agents summarize observation histories; and cooperative-perception systems explicitly share sensor data or features. Recent surveys document the rapid development of MARL, communication-based MARL, scalability, non-stationarity, and partial observability [6,7,8,9]. A recent application in heterogeneous mobile-edge computing combines federated learning, safe multi-agent DRL, and convex optimization, illustrating how current MARL systems often add substantial coordination and optimization machinery around the learners [10].

The mechanism considered here is deliberately narrower. We ask whether another agent’s physical presence, when it is already observable through the focal agent’s ordinary local sensor, can itself refine an aliased observation. No message is sent, the field of view is not extended, and no centralized critic is introduced. The potential benefit is therefore not that the proposed mechanism dominates modern MARL methods in performance. Rather, it identifies a structural case in which part of the missing information is supplied by the multi-agent environment itself. Conversely, if the other agents do not create action-relevant distinctions, the augmentation can be useless or even costly because it splits a tabular representation into more observation entries.

This distinction motivates the main question of this paper: when does other-agent occupancy make an aliased observation sufficiently informative for a memoryless action rule, and when does it not? We formalize the base and augmented observation functions and use action consistency over an entire perceptual class, rather than only pairwise state comparisons, to characterize observation-induced action conflict. The formal results concern local action compatibility; they do not by themselves establish a global optimal policy or all conditions of an existing Profit Sharing rationality theorem. Those stronger conclusions require additional reachability and learning assumptions and are treated separately.

The main contributions are as follows.

(C1)

We formalize perceptual augmentation as a refinement of the observation function and show that distinctions already available in the base observation are not lost (Proposition 1).

(C2)

We define action consistency for perceptual classes and give a sufficient condition under which augmentation removes a local observation-induced action conflict (Proposition 3). For intermittent local resolution, Proposition 4 quantifies the expected number of unresolved visits using an event probability q.

(C3)

We distinguish this event probability from the empirical total-variation distance $D_{\mathrm{TV}}$ used later as a descriptive distributional-separability index. We also show that augmentation can increase the number of independently learned tabular entries (Proposition 5).

(C4)

We evaluate the mechanism in three grid environments and under changes in the number of agents, execution protocol, and occupancy-sensor noise. The experiments include both favorable and unfavorable cases and a one-step-history baseline.

(C5)

We separate training-time completion from final-policy quality and analyze non-terminating runs as right-censored observations. Non-completion counts are reported directly, and time-to-completion comparisons use censoring-aware analysis where right-censoring occurs.

Related preliminary experiments were reported in [11]. This article is a revised and expanded version of a paper entitled “Resolving the Perceptual Aliasing Problem by Multi-Agent Reinforcement Learning,” which was presented at BICA*AI 2026, the 17th Annual International Conference on Biologically Inspired Cognitive Architectures, Mérida, Yucatán, Mexico, 9–12 September 2026 [12]. The conference paper focused on Environment 1 and a limited subset of the experiments. The present article adds the formal analysis, Environments 2 and 3, statistical analysis, population-size and execution-protocol studies, sensor-noise experiments, and the memory-based baseline. In addition, the simulator was re-audited, and all affected Q-learning results were rerun with the corrected successor-state update and strict occupancy-based perceptual augmentation described in Section 5.3.

The rest of this paper is organized as follows. Section 2 positions this study with respect to related work. Section 3 describes the problem setting. Section 4 presents the formal framework. Section 5 reports the numerical experiments and statistical analysis, Section 6 reports the extended evaluation, Section 7 discusses limitations, and Section 8 concludes this paper.

2. Related Work and Positioning

2.1. Perceptual Aliasing and Memory-Based Remedies

Perceptual aliasing has been recognized since the early development of RL. Chrisman [3] introduced perceptual distinctions based on history, and McCallum [4] proposed Utile Distinction Memory to split observations when distinctions are useful for control. POMDP belief-state methods provide a principled alternative [2], while point-based methods approximate planning in large belief spaces [13]. In modern model-free RL, recurrent networks provide an implicit state representation; Ni et al. [5] showed that carefully tuned recurrent model-free RL is a strong baseline across a broad collection of POMDP tasks. These approaches compensate for missing information by adding state or memory inside the agent.

The present work asks a different question: whether some missing information is already present in the multi-agent environment. The observation function is refined by locally observable occupancy of other agents, while the learner itself may remain tabular and memoryless. Section 6.5 therefore includes a deliberately simple one-step-history baseline as a controlled empirical comparison. It is not intended to reproduce Utile Distinction Memory, PS-r*, or a recurrent neural method.

2.2. Recent MARL, Communication, and Partial Observability

Recent surveys show that contemporary MARL spans centralized training, decentralized execution, learned communication, recurrent agents, graph-based methods, and large-scale application-specific systems [6,7,8,9]. Communication is particularly relevant under partial observability because agents can exchange observations, intentions, hidden states, or learned features; Zhu et al. [6] provide a recent systematic treatment of this design space. Representative learned-communication methods include RIAL/DIAL-style communication [14] and CommNet [15]. CTDE methods such as MADDPG [16] and QMIX [17] use additional joint information during training to improve decentralized policies.

Recent work also illustrates the increasing complexity of practical MARL systems. For example, Liu et al. [10] combine federated learning, safe multi-agent DRL, and convex optimization for age-of-information minimization in heterogeneous mobile-edge computing. Hady et al. [9] review MARL for resource-allocation optimization and highlight current challenges including partial observability, scalability, communication, coordination, and dynamic environments. These works pursue high-performance decision making in complex applications; the present study instead isolates one elementary information mechanism in a controlled setting.

2.3. How the Present Mechanism Differs, and Why the Difference Matters

The mechanism studied here differs from communication learning, cooperative perception, CTDE, and memory-based POMDP remedies in three operational respects.

• No message channel or learned protocol. The cue is produced by another agent physically occupying a cell already inside the focal agent’s sensor footprint. This removes the need to design or learn message semantics, recipients, bandwidth allocation, or a communication protocol. The consequences are lower mechanism complexity as well as less control over when informative cues are available.
• No privileged centralized information is required. The focal learner acts only on its local observation. This differs from CTDE, where joint information can be available during training. The consequence is that the mechanism is applicable to fully decentralized execution and learning, but it cannot exploit information that is outside the local field of view.
• The information structure is changed rather than the learner. History-based and recurrent methods add internal state; here, the learner can remain memoryless while the observable environment supplies an additional distinction. The consequence is interpretability and low representational machinery when the cue is informative, but no general advantage when history is cheap or when other-agent occupancy is uninformative.

Table 1 summarizes these differences. The table is not intended as a performance ranking, because the methods solve different information problems and make different assumptions. It makes explicit which resources each approach adds and what trade-off follows.

Table 1. Conceptual comparison of approaches to partial observability in multi-agent settings.

Approach	Additional Information	Internal Memory	Explicit Communication	Main Trade-Off Relative to This Study
History/recurrent POMDP methods [4,5]	Past observations/actions	Yes	No	Can resolve ambiguity even without other agents, but adds internal state and learning capacity.
Learned communication [6,14,15]	Messages from other agents	Often	Yes	Information can be deliberately transmitted, but a channel/protocol and associated learning or design are required.
CTDE (MADDPG/ QMIX) [16,17]	Joint or centralized training information	Method-dependent	Not necessarily	Strong training signal, but uses information or coordination structure beyond purely local decentralized learning.
Cooperative perception	Shared sensor data/features	Method-dependent	Yes	Extends effective sensing through networked sharing; requires communication infrastructure.
Present perceptual augmentation	Locally visible physical occupancy	No (in the tested tabular form)	No	Very simple and interpretable, but useful only when task-driven occupancy creates action-relevant distinctions.

The empirical comparison in Section 6.5 complements this conceptual table: one-step history is substantially faster than perceptual augmentation in Environment 2, demonstrating that the proposed mechanism is not a generally superior remedy. Its contribution is instead to characterize when a low-complexity, memoryless source of information is structurally available. The extended experiments further show that this availability depends on topology, population density, execution protocol, and sensor noise.

3. Problem Setting

In this paper, we consider reinforcement learning agents operating with a discrete-time perception–action cycle. At each time step, an agent receives sensory input from the environment and selects and executes an action based on it. As a result, the environment provides a reward or a penalty. This setting follows the standard reinforcement learning framework [1].

Sensory inputs are discretized as needed, and the discretized inputs are referred to as states. Coarse discretization reduces the number of states but increases the likelihood that different environment situations are perceived as the same state, thereby making perceptual aliasing more likely. Fine discretization can mitigate perceptual aliasing but increases the number of states and thus the required learning time. This trade-off is a fundamental issue in POMDP environments, and it reappears in a different guise in Proposition 5.

A pair of a state and an action is called a rule. A sequence of rules from an initial state (or immediately after receiving a reward) until the next reward is defined as an episode. If different actions are selected for the same state within an episode, the rule sequence between those selections is called a detour sequence. Rules that always appear on detour sequences in all episodes are called ineffective rules, and the others are called effective rules [18].

In Profit Sharing (PS), a representative exploitation-oriented learning method, the distinction between effective and ineffective rules has a significant impact on learning performance. Following the terminology of the original PS theory [18,19], a policy is called rational when its expected reward per action is positive. In particular, the phenomenon in which a rule that was effective at some time later appears on detour sequences in all subsequent episodes is called Type-2 confusion. The PS rationality theorem [18,19] states, under its reinforcement-function condition, that irrational rules are not reinforced over rational rules in classes without Type-2 confusion; consequently, a rational policy can be acquired in a class broader than MDPs. This broader guarantee is one reason for using PS as a principal learner in the present study. By contrast, the standard convergence guarantee of Q-learning [20] is established for Markov Decision Processes (MDPs) and does not generally extend to POMDP environments with perceptual aliasing.

This paper targets a multi-agent environment in which multiple agents exist. In the main experiments we assume a synchronous setting in which only one agent performs an action per unit time, and only a single type of reward is used; this setting is consistent with prior work [21]. The restriction is made for comparability with [21] and not because the mechanism requires it: Section 6.3 evaluates a simultaneous-observation execution variant experimentally and shows that cue persistence is an additional requirement in practice.

In general, in MARL, non-stationarity due to mutual influences among agents’ learning processes is known to be a key issue [22,23,24]. Although this study does not aim to solve Dec-POMDPs exactly, we use the Dec-POMDP notation of [25] in the next section in order to state precisely what “the other agent complements perception” means.

4. Perceptual Augmentation by Other Agents: A Formal Analysis

4.1. Factored States and Observation Functions

Let $I=\{1,\dots ,n\}$ be the set of agents. We consider a factored state

$$s=(c,x_1,\dots ,x_n)\in S,x_i\in X,$$

(1)

where c is the static layout of the environment (the set of cells occupied by walls), which is fixed throughout a run, and $x_i$ is the position of agent i. We write $x_{-i}={\left(x_j\right)}_{j\ne i}$ for the joint position of the other agents. For example, if there are four agents and the focal agent is $i=2$, then $x_{-2}=(x_1,x_3,x_4)$; $x_2$ is kept separate because it is the position of the agent whose observation is being defined.

Each agent selects an action from a finite set A ($\left|A\right|=5$ in our environments). We denote by T the transition function that maps the current joint state and the agents’ actions to the next joint state, and $R_i$ denotes the reward function of agent i. This notation follows the standard Dec-POMDP formulation [25]; the analysis below uses the notation only to describe the information available to an individual agent and does not require solving the Dec-POMDP itself.

The perception of agent i is determined by its perceptual field $N\left(x_i\right)$. In this paper we write

$$N\left(x_i\right)=(u_1,u_2,u_3,u_4),$$

where $u_1,u_2,u_3,u_4$ are, in this order, the cells above, below, to the left of, and to the right of $x_i$. Thus, each $u_k$ denotes one cell in the local perceptual field, and $\left|N\right(x_i\left)\right|=4$. The base observation function is

$${\omega }_i\left(x_i\right)=\left(\beta \left(u_1\right),\beta \left(u_2\right),\beta \left(u_3\right),\beta \left(u_4\right)\right),\beta \left(u_k\right)=\left\{\begin{array}{cc}0\hfill & u_k\mathrm{is}\mathrm{a}\mathrm{wall},\hfill \\ 1\hfill & \mathrm{otherwise},\hfill \end{array}\right.$$

(2)

so that ${\omega }_i:X\to \mathsf{\Omega }={\{0,1\}}^4$, and $\left|\mathsf{\Omega }\right|\le 16$. When other agents can be perceived, the augmented observation function is defined using both the focal position $x_i$ and the joint other-agent position $x_{-i}$. In the example $i=2$ above, the test “$u_k$ is occupied by another agent” means that $u_k$ coincides with one of $x_1,x_3,x_4$. Thus,

$${\omega }_i^{+}(x_i,x_{-i})=\left({\beta }^{+}(u_1,x_{-i}),\dots ,{\beta }^{+}(u_4,x_{-i})\right),$$

(3)

where

$${\beta }^{+}(u_k,x_{-i})=\left\{\begin{array}{cc}0\hfill & u_k\mathrm{is}\mathrm{a}\mathrm{wall},\hfill \\ 2\hfill & u_k\mathrm{is}\mathrm{occupied}\mathrm{by}\mathrm{some}\mathrm{agent}j\ne i,\hfill \\ 1\hfill & \mathrm{otherwise}.\hfill \end{array}\right.$$

so that ${\omega }_i^{+}:X\times X^{n-1}\to {\mathsf{\Omega }}^{+}={\{0,1,2\}}^4$, and $\left|{\mathsf{\Omega }}^{+}\right|\le 81$.

Two remarks are important. First, (3) encodes presence or absence, not identity: the agent does not know which other agent occupies the cell, nor how many. Consequently $\left|{\mathsf{\Omega }}^{+}\right|\le 3^{\left|N\right(x_i\left)\right|}$ independently of the number of agents n. Second, no communication act occurs in (3): the value 2 is produced by the same sensor that produces 0 and 1.

The base observation ${\omega }_i$ induces a partition of the reachable positions into perceptual classes $X_z=\{x_i\in X\mid {\omega }_i\left(x_i\right)=z\}$. Perceptual aliasing occurs exactly when $\left|X_z\right|>1$ for some $z\in \mathsf{\Omega }$.

4.2. The Refinement Property

Proposition 1

(Refinement). For every fixed $x_{-i}$, the partition of X induced by the mapping $x_i\mapsto {\omega }_i^{+}(x_i,x_{-i})$ is a refinement of the partition induced by ${\omega }_i$. In particular, perceptual augmentation never merges two positions of agent i that were distinguishable without it.

Proof. 

Let $\phi :\{0,1,2\}\to \{0,1\}$ be defined by $\phi \left(0\right)=0$, $\phi \left(1\right)=\phi \left(2\right)=1$, and extend it componentwise to ${\mathsf{\Omega }}^{+}\to \mathsf{\Omega }$. By (2) and (3), for each $k\in \{1,2,3,4\}$, ${\beta }^{+}(u_k,x_{-i})=0$ if and only if $\beta \left(u_k\right)=0$, and ${\beta }^{+}(u_k,x_{-i})\in \{1,2\}$ if and only if $\beta \left(u_k\right)=1$. Hence, ${\omega }_i\left(x_i\right)=\phi \left({\omega }_i^{+}(x_i,x_{-i})\right)$ for all $x_i$ and all $x_{-i}$. Therefore, ${\omega }_i^{+}(x_i,x_{-i})={\omega }_i^{+}(x_i^{\prime },x_{-i})$ implies ${\omega }_i\left(x_i\right)={\omega }_i\left(x_i^{\prime }\right)$, which proves the refinement property. □

Proposition 1 formalizes the sense in which the presence of other agents “expands the effective perceptual space”: the mapping ${\omega }_i^{+}$ is a strictly finer description whenever some other agent is inside the perceptual field, and coincides with ${\omega }_i$ otherwise. It also shows that the mechanism cannot be harmful in terms of information; as Proposition 5 shows, it can nevertheless be harmful in terms of learning cost.

4.3. Action-Conflicting Perceptual Classes and Local Resolution

We next formalize when aliasing creates an action-level ambiguity. Fix agent i with goal position $x_i^{\mathrm{g}}$. For the analysis only, consider the static free-cell graph of the environment, treating cells temporarily occupied by other agents as free. Let $d_i\left(x_i\right)$ denote the length of a shortest path from $x_i$ to $x_i^{\mathrm{g}}$ in this graph, and let ${\tau }_i(x_i,a)$ denote the successor position produced by action a in the same static graph. Define

$$G_i\left(x_i\right)=\left\{a\in A\mid d_i\left({\tau }_i(x_i,a)\right)=d_i\left(x_i\right)-1\right\}.$$

(4)

Thus, $G_i\left(x_i\right)$ is the set of actions that can serve as the first action of at least one shortest path from $x_i$ to the goal. This definition is an analytical device only; the learning agent is neither given $d_i$ nor assumed to compute a shortest path.

For a base observation $z\in \mathsf{\Omega }$, let

$$C_i\left(z\right)=\{x_i\in X\mid {\omega }_i\left(x_i\right)=z\}$$

(5)

denote its perceptual class. For any subset $S\subseteq C_i\left(z\right)$, define the common shortest-path action set

$$G_i\left(S\right)=\bigcap _{x_i\in S}{G}_i\left(x_i\right).$$

(6)

Definition 1

(Action consistency). A set S of positions sharing the same observation is action-consistent if $G_i\left(S\right)\ne \varnothing$ and is action-conflicting otherwise. Hence, a deterministic memoryless rule can select one shortest-path action that is valid at every member of S if and only if S is action-consistent.

This class-level definition is important. Pairwise overlap of the sets $G_i\left(x_i\right)$ is not sufficient in general: three sets can intersect pairwise while having an empty global intersection. Accordingly, pairwise conflict is used below only as a convenient sufficient witness of action conflict in the two-state structures of Environments 1 and 2, not as a complete characterization for arbitrary perceptual classes.

Definition 2

(Conflict pair). An unordered pair $(x_i,x_i^{\prime })$ of aliased positions is a conflict pair if $G_i\left(x_i\right)\cap G_i\left(x_i^{\prime }\right)=\varnothing$. Such a pair is a sufficient witness that the perceptual class containing it is action-conflicting. A conflict pair is unavoidable for agent i if every route from the start position to the goal position visits both members and is avoidable otherwise. An avoidable pair need not obstruct successful behavior, because a route that bypasses one member may exist; the detour in Environment 1 is an example, whereas in Environment 2 the pair is unavoidable.

Proposition 2

(Memoryless action conflict). Let S be a set of positions that share one observation and that are all relevant to the behavior being evaluated. If S is action-conflicting, no deterministic memoryless policy that assigns a single action to that observation can select a shortest-path action at every member of S.

Proof. 

A deterministic memoryless policy assigns one action a to the shared observation. Such an action is a shortest-path action at every member of S exactly when $a\in {\bigcap }_{x_i\in S}{G}_i\left(x_i\right)=G_i\left(S\right)$. If S is action-conflicting, this intersection is empty, so no such action exists. □

Proposition 2 is deliberately local. It identifies an observation-induced action conflict, but it does not by itself establish that every member of the class must be visited, that a goal-reaching policy exists globally, or that any particular learning algorithm will converge.

Definition 3

(Action-resolving augmentation). For an augmented observation $z^{+}\in {\mathsf{\Omega }}^{+}$, let

$$C_i^{+}\left(z^{+}\right)=\{x_i\in X\mid \exists x_{-i}\mathit{with}{\omega }_i^{+}(x_i,x_{-i})=z^{+}\}$$

(7)

be the set of focal-agent positions that can produce $z^{+}$ under configurations of the other agents that occur in the setting under consideration. Perceptual augmentation is action-resolving on a collection of visited augmented observations if every corresponding set $C_i^{+}\left(z^{+}\right)$ is action-consistent.

This definition requires only distinctions that matter for action choice. Positions that admit a common shortest-path action may remain aliased without creating the local conflict considered here.

Proposition 3

(Sufficient condition for local action resolution). If perceptual augmentation is action-resolving on every augmented observation encountered in the collection under consideration, then there exists a deterministic memoryless action assignment on those augmented observations that selects a shortest-path action for every represented position in the static free-cell graph.

Proof. 

For each encountered augmented observation $z^{+}$, action resolution gives ${\bigcap }_{x_i\in C_i^{+}\left(z^{+}\right)}{G}_i\left(x_i\right)\ne \varnothing$. Choose any action from this nonempty intersection and assign it to $z^{+}$. The chosen action is then a shortest-path action for every focal-agent position that can generate that augmented observation. Repeating this construction independently for each encountered $z^{+}$ defines the required deterministic memoryless action assignment. □

The scope of Proposition 3 is intentionally narrower than in the previous formulation. It removes the local observation-induced action conflict; it does not, by itself, prove the existence of a global goal-reaching policy in the dynamic multi-agent system, nor does it establish all assumptions of the Profit Sharing rationality theorem. A connection to the no-Type-2-confusion class of [18,19] can be made only when the additional task-level assumptions required by that result are also satisfied. Temporary blocking, reachability, recurrence of relevant states, and other sources of partial observability remain separate issues.

4.4. Intermittent Local Resolution

In practice, the configuration of the other agents may resolve an action conflict only on some visits. Let $R_j$ be the indicator that, at the jth visit to an action-conflicting base perceptual class, the current augmented observation belongs to an action-consistent augmented class in the sense of Definition 1. For a collection of visits under consideration, let

$$q=\mathrm{Pr}(R_j=1)$$

(8)

denote the marginal event probability of local action resolution at each such visit.

Proposition 4

(Expected unresolved visits under intermittent resolution). Conditional on the focal agent visiting an action-conflicting base perceptual class exactly k times, suppose that the marginal probability of local action resolution at each of these visits is q. Then, the expected number of unresolved visits is $k(1-q)$.

Proof. 

Let $I_j=1-R_j$ be the indicator that the jth visit remains unresolved. Since $E\left[I_j\right]=1-q$ for each of the k visits, linearity of expectation gives $E\left[{\sum }_{j=1}^k{I}_j\right]=k(1-q)$. No independence among the $I_j$ is required. □

This proposition is a conditional, event-level statement for a fixed number of visits. It does not identify q with a distance between two empirical observation distributions, and it does not imply a particular run-level failure probability. In the extended evaluation we therefore keep q as a theoretical quantity and report total-variation distance separately as a descriptive distributional separability index $D_{\mathrm{TV}}$. The two quantities answer different questions: q is the probability that a particular visit is locally action-resolved, whereas $D_{\mathrm{TV}}$ measures how different two empirical observation distributions are in aggregate.

4.5. The Cost of Augmentation

Proposition 5

(Representation cost). Perceptual augmentation replaces the rule set $\mathsf{\Omega }\times A$ by ${\mathsf{\Omega }}^{+}\times A$ with $|{\mathsf{\Omega }}^{+}|\ge |\mathsf{\Omega }|$. If a base observation is encountered under m distinct occupancy patterns, a tabular learner may maintain up to m independently updated value or weight vectors where the base learner maintained one.

Proof. 

Each distinct augmented observation indexes a distinct row of the tabular representation. Updates applied to one row do not automatically transfer to another. The number of independently updated rows can therefore grow with the number of occupancy patterns encountered. □

Proposition 5 is a representation statement, not a universal sample-complexity theorem. A larger representation can slow learning because experience is split across more entries, but it can also help when the additional distinctions remove harmful aliasing. Environment 1 provides an empirical example in which PS with roulette selection becomes slower, whereas the corrected Q-learning results show that the effect depends strongly on the learner and action-selection strategy.

4.6. Execution Protocol and Number of Agents

The definitions above concern the observation available when an action is selected. If other agents may move during the same decision interval, a cue that was informative at decision time may no longer correspond to the configuration after all moves have been applied. Rather than asserting a general bound without additional assumptions on the transition protocol, we measure this effect experimentally through $p_{\mathrm{leave}}$, the conditional probability that a cue visible at decision time is absent after the joint step. Section 6.3 reports this robustness test.

For the occupancy encoding of (3), the maximum number of local occupancy patterns is bounded by the size of the perceptual field rather than directly by the number of agents. This does not imply monotonic improvement with more agents: added agents change collision rates, occupancy distributions, and learning non-stationarity. We therefore treat the $n=2,3,4,5$ study in Section 6.1 as a sensitivity analysis within one small environment, not as a general scalability result.

4.7. When the Mechanism Fails

Collecting the results above, perceptual augmentation by other agents fails, or is not worth its cost, in the following identifiable situations.

• No task-relevant action conflict (the relevant aliased positions admit a common action, or behavior can avoid the conflicting portion of the class). Proposition 5: the representation may grow without providing a necessary disambiguation benefit.
• Low local-resolution probability q. Proposition 4: lower q implies more unresolved visits in expectation. This occurs when the other agent’s task keeps it away from the perceptual field of the conflict positions.
• Symmetric cues. If the other agent is equally likely to appear at the corresponding cells of both members of the conflict pair, the augmented observations coincide, and the sufficient separation condition of Proposition 3 is not satisfied. Separation requires an asymmetry between the two positions, not merely the presence of another agent.
• Non-stationarity of the other agent’s policy. Early in learning, ${\mathsf{\Pi }}_{-i}$ changes, so the effective q is time-varying; rules reinforced under one regime may become misleading under another.
• Sensor noise. If the occupancy label is reported incorrectly with probability $\eta$, the separating observation is itself aliased with the ambiguous one, and Proposition 4 applies with q replaced by an effective value decreasing in $\eta$.

Items 1–3 are structural and can be checked at design time; items 4–5 are statistical and are evaluated experimentally in Section 6.

5. Numerical Experiments

5.1. Objective

In this section we examine three empirical expectations motivated by Section 4 in two grid environments: (P1) in Environment 2, where the relevant conflict pair is unavoidable, we examine whether occupancy-based perceptual augmentation can reduce training-criterion non-completions by providing action-resolving distinctions; (P2) in Environment 1, where the identified conflict pair is avoidable and exhaustive search confirms that a deterministic memoryless goal-reaching policy exists on the base observation, perceptual augmentation is not structurally required and may instead incur a representation cost; (P3) lower local-resolution probability q implies more unresolved visits conditional on a fixed number of visits, while the run-level outcome may still depend on the learning and action-selection rules.

The purpose is not to demonstrate improved optimal-policy performance, but to clarify whether the prespecified training criterion becomes easier to reach in the presence of perceptual aliasing and whether the non-completion rate can be reduced. Accordingly, we use as evaluation metrics the number of action selections required to obtain a reward and the frequency of training-criterion non-completions. We compare Profit Sharing (PS) and Q-learning (QL) in order to examine whether exploitation-oriented learning is particularly suited to exploiting the augmented observation.

5.2. Experimental Environments

We consider environments with two agents as shown in Figure 1 and Figure 2 (hereafter, Environment 1 and Environment 2). Each agent can perceive only the four neighboring cells (up, down, left, and right), i.e., $N\left(x_i\right)$ is as in Section 4. As a result, if no other agent can be perceived, the hatched states are perceived as the same state, and perceptual aliasing occurs. If, on the other hand, the other agent is located in one of the four neighboring cells and can be perceived, the hatched states can be recognized as different states in the sense of Definition 3.

Figure 1. Environment 1: an environment used to illustrate the advantage of multi-agent learning. The hatched cells are perceived identically by an agent that senses only its four neighboring cells, because the wall pattern of their four neighbors is the same; this is where perceptual aliasing occurs. Agent 2 must select “right” at the hatched cell two cells below G1 and “up” at the hatched cell one cell below G2, so the two hatched cells have disjoint shortest-path action sets and form a conflict pair in the sense of Definition 2. The dashed-dotted line is the detour path, which allows Agent 2 to avoid the first hatched cell; the conflict pair is therefore avoidable. The blue solid and green dashed lines indicate the paths of Agent 1 and Agent 2, respectively; the purple dash-dotted line indicates the detour of Agent 2. S1/S2 are starting positions and G1/G2 are goals. Black boundaries indicate walls, white cells are traversable, and the blue hatched cells identify aliased positions.

Figure 2. Environment 2: an environment in which no detour path exists to avoid Type-2 confusion. The hatched cells and the required actions are as in Figure 1, but the black cells block the lower corridor, so every path of Agent 2 from S2 to G2 passes through both hatched cells. The two hatched cells again form a conflict pair, and the pair is unavoidable; by Proposition 2 no deterministic memoryless policy can select a shortest-path action at both cells. Black boundaries and filled black cells indicate walls or blocked cells; white cells are traversable. S1/S2 are starting positions, G1/G2 are goals, and the blue solid and green dashed lines indicate the paths of Agent 1 and Agent 2, respectively.

The start positions of Agent 1 and Agent 2 are S1 and S2, respectively. When an agent reaches its goal (G1 or G2), it receives a reward and returns to its start position. Each agent can select one of five actions: {up, down, left, right, stay}. Under these settings, Agent 1 has no conflict pair in either environment. For Agent 2 to obtain a reward, it must select “right” at the hatched state two cells below G1 and “up” at the hatched state one cell below G2, so the shortest-path action sets of these two hatched states are disjoint, and they form a conflict pair in the sense of Definition 2.

In Environment 1 a detour path (dashed-dotted line in Figure 1) allows Agent 2 to avoid the first hatched state, so the conflict pair is avoidable. In Environment 2 the lower corridor is blocked and no such detour exists, so the pair is unavoidable. An exhaustive search over deterministic memoryless policies on the base observation (Agent 2 alone, the other agent ignored) confirms this: it finds a policy that reaches the goal in Environment 1 and none in Environment 2. A standalone verification script and its output are provided as Supplementary Files S1 and S2; Supplementary File S3 contains the Environment 3 transition/observation definition used by the same script. By Propositions 2 and 5, Environment 1 and Environment 2 are therefore expected to behave qualitatively differently, and this is the reason for studying both.

5.3. Experimental Settings and Reproducibility

The following specification is given in full so that the experiments can be reproduced.

5.3.1. Environments

Both environments are grids in which each cell is either free or a wall; the reachable cells are numbered $0,\dots ,20$, and the transition function is a deterministic table. An action that would move an agent into a wall leaves it in place, and an action that would move it into a cell occupied by another agent also leaves it in place. The episode of an agent ends when it performs the goal action at its goal cell, at which point it receives the reward and is returned to its start cell (indicated by “reset” in Figure 1 and Figure 2); the other agents are not reset. Environment 2 differs from Environment 1 only in that the two cells of the lower corridor below the start of Agent 2 are walls, which removes the detour path.

5.3.2. Observations

“No perceptual aliasing” means that the agent observes the cell index itself. “No other agent perception” means that the agent observes ${\omega }_i\left(x_i\right)$ of (2); concretely, the three cells of the aliased class (the cell two below G1, the cell one below G1, and the cell one below G2) are reported as one and the same observation. “Other agent perception” means that the agent observes ${\omega }_i^{+}(x_i,x_{-i})$ of (3), implemented as the aliased observation together with the four-bit occupancy mask of the perceptual field. The three conditions differ only in the observation function; the environment dynamics are identical.

5.3.3. Learning Rules

The initial weight of every rule was set to $10.0$, and the reward value was set to $100.0$. Profit Sharing distributes the reward over the rules of the episode by a geometrically decreasing credit assignment function: if the episode is $(z_1,a_1),\dots ,(z_L,a_L)$ and a reward R is obtained after $(z_L,a_L)$, the weight of $(z_{L-t},a_{L-t})$ is increased by $R{\gamma }_{\mathrm{PS}}^t$. The discount factor was set to ${\gamma }_{\mathrm{PS}}={\displaystyle \frac{1}{5}}$, which satisfies the rationality theorem of [18] for $\left|A\right|=5$, and the episode buffer retains the most recent 10 rules. Q-learning uses $Q(z,a)\leftarrow (1-\alpha )Q(z,a)+\alpha \left(r+\gamma {\max }_{a^{\prime }}Q(z^{\prime },a^{\prime })\right)$ with learning rate $\alpha =0.5$ and discount factor $\gamma =0.9$, chosen on the basis of preliminary experiments. The bootstrap term is evaluated at the successor observation $z^{\prime }$. In the other-agent-perception condition, the occupancy cue is appended to the aliased base observation as the four-bit occupancy mask defined above; the underlying unaliased cell index is not included in the observation.

5.3.4. Action Selection

Roulette selection selects action a at observation z with probability proportional to the rule weight, i.e., $\mathrm{Pr}(a\mid z)=w(z,a)/{\sum }_{a^{\prime }}w(z,a^{\prime })$ for PS and $\mathrm{Pr}(a\mid z)=Q(z,a)/{\sum }_{a^{\prime }}Q(z,a^{\prime })$ for QL; all weights are positive by construction. The greedy strategy selects $\arg {\max }_{a}w(z,a)$ and falls back to a uniformly random choice while all weights of the observation are still at their initial value. Roulette selection represents performance when stochastic action choice is retained during learning and interaction, whereas greedy selection suppresses this stochasticity and more directly exposes the action preferences encoded in the learned weights or Q-values. Accordingly, the two strategies are used here as complementary evaluation modes rather than as competing deployment recommendations. Stochastic action selection may prevent an accurate evaluation of the inherent capability of the learner, which is why both strategies are reported.

5.3.5. Protocol and Metrics

One agent acts per time step, the agents acting in turn. Each condition was run for 100 trials with different random seeds. A trial reached the prespecified training criterion when every agent had obtained its reward 101 times. This criterion measures cumulative training-phase performance; it is not a direct evaluation of the quality of a frozen final policy. The time-to-criterion variable is the number of action selections required. Trials that had not reached the criterion by $\mathrm{100,000}$ action selections were right-censored at that budget and are reported as non-completions. The censoring limit is not treated as an observed completion time. Accordingly, a cell with any censored trials is summarized by its number of non-completions rather than by a mean and standard deviation. The uniformly random policy is reported as a reference.

5.4. Experimental Results

Table 2 and Table 3 report the results for Environment 1 and Environment 2 separately, and Table 4 collects the training-criterion non-completion rates with exact $95\%$ confidence intervals. The two environments are reported separately because they differ structurally, and learning speed and training-criterion non-completion is reported separately because they are not commensurable outcomes.

Table 2. Environment 1 (a detour path exists, hence the conflict pair of Agent 2 is avoidable). For conditions with no censoring, the mean number of action selections required over 100 trials is reported, with the standard deviation in parentheses. For conditions containing right-censored trials, the number of non-completions out of 100 trials is reported instead (see Table 4).

	No Perceptual Aliasing	With Perceptual Aliasing
		No Other Agent Perception	Other Agent Perception
PS (roulette)	5355.42 (573.20)	17,348.86 (1844.48)	18,418.39 (2038.94)
PS (greedy)	non-completions: 6	non-completions: 27	non-completions: 28
QL (roulette)	12,603.63 (1687.27)	33,659.36 (4144.07)	29,195.47 (4623.41)
QL (greedy)	1653.00 (417.14)	29,797.43 (12,412.41)	non-completions: 99
Random	51,016.92 (4508.01)

Table 3. Environment 2 (no detour path, hence an unavoidable conflict pair for Agent 2). Format as in Table 2.

	No Perceptual Aliasing	With Perceptual Aliasing
		No Other Agent Perception	Other Agent Perception
PS (roulette)	6798.47 (644.04)	non-completions: 100	51,589.19 (4963.87)
PS (greedy)	1346.71 (497.69)	non-completions: 100	non-completions: 59
QL (roulette)	13,890.88 (1762.31)	41,058.31 (4836.51)	32,940.06 (6459.41)
QL (greedy)	1719.26 (534.52)	38,932.14 (3487.41)	non-completions: 100
Random	67,496.29 (6495.73)

Table 4. Training-criterion non-completion rates (number of right-censored trials out of 100) with exact (Clopper–Pearson) $95\%$ confidence intervals, and Fisher’s exact test comparing the two aliased conditions. “—” indicates that no non-completion occurred.

Environment	Method	No Other Agent	Other Agent	Fisher’s Exact Test
Env. 1	PS (greedy)	27 [18.6, 36.8]%	28 [19.5, 37.9]%	$p=1.00$
Env. 2	PS (roulette)	100 [96.4, 100]%	0 (—)	$p=2.2\times {10}^{-59}$
Env. 2	PS (greedy)	100 [96.4, 100]%	59 [48.7, 68.7]%	$p=5.0\times {10}^{-15}$
Env. 1	QL (greedy)	0 (—)	99 [94.6, 100]%	$p=2.2\times {10}^{-57}$
Env. 2	QL (greedy)	0 (—)	100 [96.4, 100]%	$p=2.2\times {10}^{-59}$

First, when perceptual aliasing does not exist, most conditions reach the training criterion with far fewer action selections than the uniformly random policy. The exception is PS (greedy) in Environment 1, for which 6 of 100 trials are right-censored at the action budget. We therefore report the non-completion count rather than a mean obtained by substituting the censoring limit. A greedy learner whose weights are still tied may repeat an unproductive loop for a long time, which explains the long right tail in this condition.

Second, and in agreement with prediction (P1), in Environment 2 the effect of other-agent perception is large. PS with roulette selection changes from 100 non-completions out of 100 trials to no non-completion at all, and PS with the greedy strategy changes from 100 to 59 non-completions. Correctly implemented Q-learning with roulette selection also benefits from the additional observation (from 41,058 to 32,940 action selections in Environment 2). In contrast, greedy Q-learning is destabilized by the enlarged observation: it changes from no non-completions without other-agent perception to $100/100$ non-completions with it.

Third, and in agreement with prediction (P2), in Environment 1 the effect is absent. The non-completion rate of PS (greedy) is 27 versus 28, and the number of action selections of PS (roulette) moves in the unfavourable direction.

Fourth, the empirical augmented-observation distributions are method-dependent because the learned policies change the state–occupancy distribution. For PS (roulette), the total-variation distance between the compared distributions is $D_{\mathrm{TV}}=0.22$. This quantity is a descriptive distributional-separability index and is not an estimator of the event probability q in Proposition 4. The different run-level outcomes of roulette and greedy selection further show that neither quantity alone is a run-level failure-probability model.

5.5. Statistical Analysis

The action budget creates right-censored time-to-criterion observations: a run that has not reached the training criterion by $\mathrm{100,000}$ action selections is known only to require more than the budget. Treating the budget as an observed completion time would bias means, standard deviations, and tests based on them. We therefore analyze the time to the prespecified training criterion using Kaplan–Meier survival curves and two-sided log-rank tests. As a descriptive time summary, we report the Kaplan–Meier median when it is estimable and the restricted mean time to criterion (RMST) up to $\mathrm{100,000}$ action selections. The RMST incorporates the common censoring horizon without asserting that censored trials completed at the horizon. For the binary non-completion counts we retain Fisher’s exact test and exact Clopper–Pearson confidence intervals (Table 4).

Table 5 reports the principal comparisons between the two aliased conditions. “NR” means that the Kaplan–Meier median was not reached within the observation horizon.

Table 5. Right-censored time-to-training-criterion analysis comparing “no other agent perception” with “other agent perception” (100 trials per condition; censoring horizon $\mathrm{100,000}$ action selections). KM median is the Kaplan–Meier median; RMST is the restricted mean time to criterion up to the common horizon; p is from the two-sided log-rank test.

Environment	Method	Non-Completions (No/Other)	KM Median (No/Other)	RMST No	RMST Other	Log-Rank p
Env. 1	PS (roulette)	0/0	17,198/18,230	17,348.86	18,418.39	$2.7\times {10}^{-4}$
Env. 1	PS (greedy)	27/28	1728/1994	28,158.44	29,342.27	$0.252$
Env. 1	QL (roulette)	0/0	33,301/28,797	33,659.36	29,195.47	$6.5\times {10}^{-9}$
Env. 1	QL (greedy)	0/99	34,678/NR	29,797.43	99,098.34	$1.7\times {10}^{-53}$
Env. 2	PS (roulette)	100/0	NR/51,388	100,000.00	51,589.19	$2.6\times {10}^{-55}$
Env. 2	PS (greedy)	100/59	NR/NR	100,000.00	60,434.54	$5.4\times {10}^{-13}$
Env. 2	QL (roulette)	0/0	40,648/32,811	41,058.31	32,940.06	$8.2\times {10}^{-17}$
Env. 2	QL (greedy)	0/100	39,159/NR	38,932.14	100,000.00	$2.6\times {10}^{-55}$

Three observations follow.

• In Environment 2, other-agent perception substantially changes completion of the prespecified training criterion for PS: PS (roulette) changes from $100/100$ censored trials to $0/100$, and PS (greedy) changes from $100/100$ to $59/100$. The corresponding log-rank tests are $p=2.6\times {10}^{-55}$ and $p=5.4\times {10}^{-13}$, respectively. These statements concern training-phase time to the specified criterion, not the quality of a frozen final policy.
• In Environment 1, PS (greedy) shows no detectable difference in the censored time-to-criterion distribution (27 versus 28 censored trials; log-rank $p=0.252$). PS (roulette), for which no censoring occurs, is slower with other-agent perception (RMST 17,348.86 versus 18,418.39; log-rank $p=2.7\times {10}^{-4}$). This negative result is consistent with Proposition 5: the additional distinctions are not required to remove an action conflict in this environment, while the tabular representation is nevertheless split across additional observations.
• Corrected Q-learning remains strongly strategy-dependent. With roulette selection, other-agent perception reduces the censored time to criterion in both environments (Env. 1: $p=6.5\times {10}^{-9}$; Env. 2: $p=8.2\times {10}^{-17}$). With greedy selection, it produces severe lock-in under the synchronous protocol (Env. 1: 0 versus 99 censored trials; Env. 2: 0 versus 100). Thus, the experiments do not support a blanket claim that one learner is intrinsically superior under aliasing; they show an interaction among observation refinement, bootstrapping, action selection, and execution protocol.

This survival-analysis treatment replaces the earlier practice of entering the $\mathrm{100,000}$-action cutoff as though it were an observed completion time. The principal favorable, null, and unfavorable conclusions are unchanged, but their statistical interpretation is now appropriate for censoring.

6. Extended Evaluation

The experiments of Section 5 are confined to two environments with two agents under synchronous execution and noiseless perception. This section evaluates the predictions of Section 4 outside that envelope. The protocol is identical to Section 5.3 unless stated otherwise: 100 trials per condition and a budget of $\mathrm{100,000}$ action selections. Throughout this section we report, in addition to the outcome, two measured quantities that make the theory checkable: the cue rate $\widehat{c}$, the fraction of visits to a cell of a conflict pair at which some other agent occupies a cell of the perceptual field, and the distributional separability index $D_{\mathrm{TV}}$, the total-variation distance between the empirical distributions of augmented observations at the compared aliased positions. This quantity is not an estimator of the event probability q in (8). The two quantities are different, and $D_{\mathrm{TV}}$ depends on which pair of positions is compared; it is, therefore, always reported together with the pair.

6.1. Number of Agents

The occupancy encoding has a local representation bound that does not depend directly on n, but increasing n also changes density, collisions, and occupancy statistics. We therefore use $n\in \{2,3,4,5\}$ only as a sensitivity analysis in this fixed small environment. We replicate Environment 2 with $n\in \{2,3,4,5\}$; the additional agents are given the task of Agent 1, with distinct start cells, so that the conflict pair of Agent 2 is unchanged, and only the occupancy statistics of its perceptual field change. Table 6 reports the resulting sensitivity analysis.

Table 6. Effect of the number of agents in Environment 2 with other-agent perception. $\widehat{c}$ is the measured cue rate, and $D_{\mathrm{TV}}$ is the distributional separability index at the compared aliased positions of Agent 2. For PS (greedy) the number of non-completions out of 100 trials is reported.

n	$\widehat{\mathit{c}}$	${\mathit{D}}_{\mathbf{TV}}$	PS (Roulette)	PS (Greedy)	QL (Roulette)
2	0.20	0.22	51,589.19 (4963.87)	non-completions: 59	32,940.06 (6459.41)
3	0.30	0.21	44,092.34 (5401.52)	non-completions: 58	23,709.88 (6220.74)
4	0.44	0.21	36,562.49 (5559.03)	non-completions: 94	18,709.66 (4996.33)
5	0.59	0.21	27,304.87 (3301.97)	non-completions: 100	15,489.35 (2858.00)

The result splits in two, and the split is informative. Within this particular environment, the stochastic strategies become faster as more helper agents are added: as n grows from 2 to 5, the cue rate rises from $0.20$ to $0.59$, and PS (roulette) needs $47\%$ fewer action selections (51,589 to 27,305), with corrected QL (roulette) also improving strongly. The distributional separability index $D_{\mathrm{TV}}$, in contrast, stays essentially constant near $0.21$. Thus, in this particular experiment, additional agents make occupancy cues available more often without materially changing the aggregate difference between the augmented-observation distributions at the compared aliased positions. This is a descriptive observation, not an estimate of the event probability q and not a general monotonicity claim about agent number.

For the greedy strategy the mechanism scales badly: the non-completion rate rises from 59 to 100. This is consistent with the representation-cost mechanism of Proposition 5, although the present experiment also confounds agent number with density and collision frequency. The bound on the augmented observation space is independent of n, but the number of augmented observations actually visited grows with n, because more occupancy patterns of the perceptual field occur; a greedy learner acts at random until every visited rule has been reinforced at least once, so its exploration burden grows with n even though its representation does not. Accordingly, these runs should not be read as a general scalability result. They show only that the local encoding remains bounded while learning performance can improve or deteriorate as the population changes in this small topology.

6.2. A Third Environment with a Different Topology

Environments 1 and 2 share a corridor topology and an aliased class with three members. Environment 3 (Figure 3) is an open room of $8\times 3$ cells with two doors in the ceiling, one of which is the goal of Agent 2; the alcove above the other door is the goal of Agent 1. Because the interior of a room has no distinguishing walls, the four-neighbor observation (2) puts eight cells into a single aliased class.

Figure 3. Environment 3: an open room of $8\times 3$ cells with two doors in the ceiling. G1 (reached through the alcove D1) is the goal of Agent 1 (solid), and G2 (through D2) is the goal of Agent 2. The hatched cells are perceived identically by an agent that senses only its four neighboring cells, because none of those neighbors is a wall; the aliased class has eight members. The shortest route of Agent 2 (dashed, 9 actions) crosses the class, whose shortest-path action sets conflict: the top-row cell below D1 admits only “right”, the top-row cell below D2 and the middle-row cell below it admit only “up”, and the five other middle-row cells admit “right” or “up”. The dash-dotted route along the bottom row (13 actions) visits only one cell of the class, the top-row cell below D2, where “up” is required; a deterministic memoryless policy that reaches the goal therefore exists on the base observation, and the conflict is avoidable.

Evaluating (4) on the static graph gives the following shortest-path action sets of Agent 2 on the eight cells of the class: the top-row cell below door D1 admits only “right”; the top-row cell below D2 and the middle-row cell below it admit only “up”; and the five remaining middle-row cells admit “right” or “up”. The class is therefore action-conflicting, and its conflict pairs are those formed by the cell below D1 with each of the two cells that require “up”. The conflict is, however, avoidable. A deterministic memoryless policy that reaches the goal exists on the base observation: Agent 2 can follow the bottom row (the dash-dotted route in Figure 3; 13 actions instead of 9 for the shortest route), which visits only one cell of the class and requires “up” there. We verified this both by an exhaustive search over deterministic memoryless policies and by direct simulation of the resulting policy; the same verification script is provided as Supplementary File S1. Environment 3 is thus, like Environment 1, an environment with an avoidable conflict pair; the difference is that the avoiding route is longer.

By Proposition 5 and item 1 of Section 4.7, perceptual augmentation is, therefore, not expected to be necessary here, and Environment 3 is used as an empirical stress test with a different topology, a larger aliased class, and a longer avoiding route. It is not a test of the sufficient condition of Proposition 3, which would require an environment in which the conflict is unavoidable.

The training results for this topology are reported in Table 7.

Table 7. Environment 3 (open room; aliased class of eight members). Format as in Table 2.

Method	No Perceptual Aliasing	No Other Agent Perception	Other Agent Perception
PS (roulette)	21,631.88 (7303.48)	non-completions: 100	non-completions: 100
PS (greedy)	non-completions: 76	non-completions: 100	non-completions: 99
QL (roulette)	15,592.83 (1515.59)	32,776.72 (3085.36)	31,373.52 (2785.28)
QL (greedy)	2564.60 (628.43)	42,087.16 (4204.55)	non-completions: 46
Random	40,405.27 (3909.96)

The results are strongly negative for Profit Sharing. With roulette selection PS has 100 of 100 non-completions both with and without other-agent perception (Fisher’s exact test, $p=1.00$), and with the greedy strategy it has 100 and 99. PS (greedy) also fails in 76 of 100 trials without aliasing, so Environment 3 is difficult for this learner even before aliasing is introduced. Corrected Q-learning with roulette selection completes the training criterion in every trial and changes only modestly with other-agent perception, whereas greedy Q-learning changes from no non-completions to 46 out of 100.

The measured quantities are descriptive. Another agent occupies a cell of the perceptual field at $36\%$ of the visits to the three cells that form conflict pairs ($\widehat{c}=0.36$ for PS with roulette selection and $n=2$, rising to $0.59$ for $n=5$). The distributional separability index depends strongly on the pair compared: for PS (roulette) with $n=2$, it is $D_{\mathrm{TV}}=0.011$ for the pair formed by the cell below D1 and the middle-row cell below D2 and $0.98$ for the pair formed by the cell below D1 and the top-row cell below D2; for $n=5$ the values are $0.44$ and $0.98$. No single value summarizes the separation in this environment, and, because the conflict is avoidable, none of them is a prediction target of Section 4.

A null effect of other-agent perception is what item 1 of Section 4.7 leads one to expect when the conflict need not be resolved, and these data do not test the sufficient condition of Proposition 3. They also do not explain why Profit Sharing fails under aliasing although a deterministic memoryless goal-reaching policy exists. A sensitivity check excluded one candidate explanation: lengthening the credit-assignment buffer from 10 to 20 or 40 rules, so that the 13-action route can be fully credited, left PS (roulette) at $100/100$ non-completions and PS (greedy) at 99–100 out of 100 under both aliased conditions. We therefore report Environment 3 as an unexplained empirical result and leave the identification of its cause to future work.

6.3. Asynchronous Execution

To test robustness to the execution protocol, we repeat Environment 2 with a simultaneous-move protocol in which all agents select their action from the same snapshot of the configuration, and the moves are then applied sequentially using the same fixed agent order as the simulator. This is a simultaneous-observation variant rather than a fully general asynchronous scheduler. We measure $p_{\mathrm{leave}}$ directly. Table 8 compares the two execution protocols.

Table 8. Synchronous versus asynchronous execution in Environment 2 with other-agent perception. The measured $p_{\mathrm{leave}}$ ranges from $0.44$ to $0.71$ in these conditions.

Method	Synchronous	Asynchronous	${\mathit{p}}_{\mathbf{leave}}$
PS (roulette)	51,589.19 (4963.87)	non-completions: 100	0.44
PS (greedy)	non-completions: 59	non-completions: 80	0.46
QL (roulette)	32,940.06 (6459.41)	38,380.87 (5359.95)	0.46
QL (greedy)	non-completions: 100	33,239.55 (18,122.31)	0.71

The result shows that the mechanism is not protocol-independent. PS (roulette), which succeeds in all synchronous trials, has $100/100$ non-completions under the simultaneous-observation variant; PS (greedy) also deteriorates. Corrected Q-learning responds differently: roulette selection becomes slower but remains successful, while greedy Q-learning changes from $100/100$ non-completions to completion of the training criterion in all trials, again showing that protocol changes can alter exploration dynamics as well as cue persistence.

The measured $p_{\mathrm{leave}}$ is large ($0.44$ for PS roulette and $0.46$ for QL roulette), indicating that cues are often temporally unstable across a joint step. We therefore use $p_{\mathrm{leave}}$ as a descriptive robustness diagnostic, not as a universal bound on the effective separation probability. The design implication is nevertheless clear: if other-agent presence is to serve as perceptual information, the cue must remain informative on the time scale at which the focal agent’s action is evaluated.

6.4. Imperfect Perception

We repeat Environment 2 with an occupancy sensor that reports the wrong label with probability $\eta$, independently for each of the four cells of the perceptual field at each time step. Table 9 summarizes the noise-sensitivity results.

Table 9. Effect of occupancy-sensor noise $\eta$ in Environment 2 with other-agent perception.

Method	$\mathit{\eta }=0.0$	$\mathit{\eta }=0.05$	$\mathit{\eta }=0.1$	$\mathit{\eta }=0.2$
$D_{\mathrm{TV}}$	0.22	0.14	0.10	0.05
PS (roulette)	51,589.19 (4963.87)	non-completions: 19	non-completions: 25	non-completions: 29
PS (greedy)	non-completions: 59	10,263.86 (11,285.94)	10,424.23 (5850.47)	11,711.45 (5251.10)

For PS with roulette selection, the distributional separability index $D_{\mathrm{TV}}$ falls monotonically from $0.22$ to $0.05$ as sensor noise increases, while the non-completion count rises from 0 to 29 out of 100 ($p=1.5\times {10}^{-6}$ by Fisher’s exact test already at $\eta =0.05$). This is consistent with the qualitative prediction that corrupted occupancy labels reduce action-relevant perceptual information, but $D_{\mathrm{TV}}$ is used here as a descriptive distributional measure rather than as the event probability q of Proposition 4.

For PS with the greedy strategy, the effect is the opposite and, at first sight, paradoxical: noise removes the non-completions entirely (59 out of 100 at $\eta =0$, none at $\eta =0.05$, $p=1.3\times {10}^{-23}$) and reduces the number of action selections by a factor of five. The explanation is that the two strategies fail for different reasons. The greedy strategy is deterministic, and its dominant non-completion mechanism in this environment is not the conflict pair but lock-in: once the weights of a visited observation are no longer tied, the agent repeats the same action indefinitely. A noisy occupancy label makes the observation itself stochastic, which makes the action stochastic, and thereby supplies the exploration that the greedy strategy lacks. The effect is a property of the action-selection strategy rather than of perceptual augmentation, and it should not be read as evidence that noise is beneficial: for the strategy whose behavior actually depends on the separating information, noise is harmful, exactly as predicted.

6.5. Comparison with a Memory-Based Baseline

Finally we compare the mechanism against a simple memory-based baseline. The baseline agent has no other-agent perception. Instead, we augment its observation with one step of history and use the triple (previous observation, previous action, and current base observation). This baseline is not intended to reproduce a particular existing memory-based algorithm; it is a controlled comparison for testing whether a short history alone can resolve the perceptual aliasing in Environment 2. More sophisticated approaches to incomplete perception include utile distinction memory [4] and PS-r* [26]. Table 10 compares the two representations.

Table 10. Perceptual augmentation versus a one-step-memory baseline in Environment 2.

Method	Base Observation	One-Step Memory	Other Agent Perception
PS (roulette)	non-completions: 100	10,920.11 (3187.59)	51,589.19 (4963.87)
PS (greedy)	non-completions: 100	non-completions: 3	non-completions: 59
Distinct observations	19	2205	336

The baseline is clearly better on this task: one step of history provides the action-relevant distinction deterministically in this environment, so PS (roulette) reaches the training criterion in 10,920 action selections on average against 51,589 with perceptual augmentation, while the greedy strategy has 3 non-completions rather than 59.

We report this result because it corrects the claim that a reader might otherwise take from Section 5. Perceptual augmentation is not the better remedy for aliasing in this environment; where a short history supplies a stable action-relevant distinction, history is better. By contrast, under other-agent perception the empirical augmented-observation distributions at the compared aliased positions remain substantially overlapping ($D_{\mathrm{TV}}=0.22$). What the comparison does show is the trade-off that the analysis predicts. The memory-based agent requires an internal state and, in this tabular setting, an observation space two orders of magnitude larger (2205 against 19 rules); the augmented agent remains memoryless, and its tabular observation space grows only through occupancy patterns. Moreover, one step of history is sufficient here only because the conflict states of Environment 2 are entered from different predecessors; in environments where aliased states share their recent histories, memory of bounded length does not separate them, while a separating configuration, when one exists, does so instantaneously. The two remedies are therefore complementary, and the contribution of this paper is the condition that decides when the second one is available at all.

7. Limitations

Four limitations should be stated explicitly.

First, the analysis of Section 4 characterizes when augmented observations remove a local observation-induced action conflict. It does not by itself establish a global goal-reaching or rational policy, nor does it provide a sample-complexity bound. The event probability q is a property of the joint process rather than a design variable, and applying the Profit Sharing rationality theorem requires its additional task-level assumptions. Section 6.1 shows why a sample-complexity account is needed: in the fixed small Environment 2, increasing the number of agents improves roulette-based learners but drives PS with greedy selection from 59 to 100 non-completions; density and collision effects are confounded with population size. Making q an object of optimization—that is, inducing an agent to be visible where it is informative, for instance through the indirect rewards analyzed in [21]—is the natural next step.

Second, the mechanism is fragile in ways the experiments quantify. It provides no consistent benefit when the extra distinctions are not needed (Section 5.5), is not shown to help in an open room whose conflict pairs are avoidable, where Profit Sharing fails for reasons that remain unexplained (Section 6.2), and can collapse under a simultaneous-observation protocol when cues are temporally unstable (Section 6.3). It is also not competitive with a one-step-memory baseline on the task where it does work (Section 6.5). The contribution of this paper is accordingly a characterization, not a method that should be preferred to existing remedies.

Third, the environments are grid worlds with a four-cell perceptual field. The formal framework is not restricted to grids: $N\left(x_i\right)$, ${\omega }_i$, and ${\omega }_i^{+}$ are defined for arbitrary perceptual fields and label sets. However, increasing environmental complexity can change several factors simultaneously: the number and geometry of aliased states, the number of distinct action requirements within a perceptual class, the probability that other agents occupy informative locations, collision density, and the temporal stability of cues. Environment 3 varies topology and aliasing-class structure, but it does not span these dimensions independently. The empirical claims should therefore be read as claims about small discrete environments rather than evidence of scale-independent effectiveness.

Fourth, the mechanism has been studied with tabular learners. Whether the same effect survives with function approximation is open; Proposition 5 is stated only for tabular representation growth, and generalization could substantially change the cost of augmentation.

8. Conclusions

Reinforcement learning suffers from the perceptual aliasing problem, in which different states are perceived as identical. This paper asked when, and at what cost, the mere presence of other agents mitigates that problem.

We formalized the presence of other agents as a refinement of the observation function (Proposition 1), characterized action conflict at the level of an entire perceptual class (Proposition 2), and gave a sufficient condition under which augmented observations locally remove that observation-induced action conflict (Proposition 3). This local result does not by itself imply a global goal-reaching policy or the assumptions required by the Profit Sharing rationality theorem; those require additional task-level conditions. For intermittent separation, Proposition 4 quantifies the expected number of visits that remain ambiguous; Proposition 5 identifies the growth of independently learned tabular entries; and the simultaneous-observation experiment shows that cue persistence across a joint step matters in practice.

The experiments support this account in both directions, and the negative directions are as informative as the positive one. Where no detour path exists, other-agent perception changes the Profit Sharing roulette condition from $100/100$ budget-censored runs to $0/100$; where a detour path exists, it yields no improvement and a small slow-down in training completion time, consistent with the representation-cost mechanism of Proposition 5. In the open-room environment, whose conflict pairs can be avoided by a longer route, other-agent perception does not improve Profit Sharing, as expected when the conflict need not be resolved; Profit Sharing nevertheless fails there under aliasing although a deterministic memoryless goal-reaching policy exists, and the present analysis does not explain this. Under the simultaneous-observation protocol, where the measured $p_{\mathrm{leave}}$ lies between $0.44$ and $0.71$ across conditions, the Profit Sharing roulette condition changes from $0/100$ to $100/100$ non-completions. Increasing the number of agents raises the cue rate while the measured distributional-separability index changes little in this topology; the resulting training behavior remains strategy-dependent. A one-step-memory baseline outperforms the mechanism where both apply, at the price of an internal state and a far larger rule set.

Taken together, these results replace the intuition that additional agents automatically provide useful information with a narrower criterion: other-agent occupancy is useful only when the resulting augmented observations create action-relevant distinctions at the states that matter for behavior. Environment layout, agent placement, task assignment, learner dynamics, and execution protocol jointly determine how often such distinctions arise and whether the learner can exploit them. Future work includes making q an object of design through indirect rewards, extending the analysis to function approximation, and characterizing the perceptual-field size required for aliased classes of a given cardinality.