Reinforcement Learning in Swarms
Reinforcement learning trains an agent by having it act, observe a reward, and adjust. Applied to a swarm, it promises what hand-design cannot deliver: local rules that produce a specified collective outcome, discovered rather than invented.
The promise is real and the difficulties are severe. Almost everything that makes single-agent reinforcement learning work stops working when there is more than one learner.
Why it is hard
Non-stationarity. Each agent’s environment includes the other agents, and they are learning too. From any one agent’s perspective the world’s dynamics change underneath it as its peers update their policies. The Markov property — on which nearly every convergence guarantee rests — is violated. This is the central problem, and it does not go away.
Credit assignment. The swarm succeeded. Which agent’s action caused it? With a shared global reward, an agent that did nothing useful receives the same signal as the one that solved the problem. Structural credit assignment across agents compounds the temporal credit assignment that single-agent reinforcement learning already struggles with.
Partial observability. Agents see neighbourhoods, not states. Each faces a partially observable problem whose hidden state includes the internal state of every other agent.
Scale. The joint action space grows exponentially in agent count. Any method that reasons about it directly is finished at a dozen agents.
Paradigms
Centralized training, decentralized execution
The dominant framing. During training, a critic sees the global state and every agent’s action, which restores stationarity from the critic’s point of view. At execution time each agent runs a policy that reads only its own observation.
CTDE is a genuinely good idea and it has a hard boundary: it requires a training-time oracle with global state. That is available in simulation and unavailable in the field, which is fine — training happens in simulation — but it means the fidelity of your simulator determines the quality of your policy.
Fully decentralized learning
Each agent learns independently, treating others as part of the environment. Independent Q-learning is the simplest instance: run single-agent Q-learning per agent and hope. It has no convergence guarantee, because of non-stationarity, and it frequently works anyway, which is one of the field’s more irritating empirical facts.
Communication-based methods
Agents learn what to communicate alongside what to do, with the message channel differentiable end-to-end so that gradients flow through it. Learning a protocol rather than specifying one is elegant. It also means the learned protocol assumes whatever channel properties the training environment had — and if that channel was free and instantaneous, the protocol will not survive a real one.
Algorithms
Independent actor-critic methods run a per-agent policy and value function. Simple, scalable, and subject to all of non-stationarity’s consequences.
MADDPG applies CTDE to continuous control: each agent has a decentralized actor and a centralized critic conditioned on all agents’ observations and actions. It handles cooperative, competitive, and mixed settings.
MAPPO — multi-agent proximal policy optimization with a centralized value function and, typically, parameter sharing across identical agents — is the strong, boring baseline. Parameter sharing collapses the learning problem to a single policy and makes agent count almost free, at the cost of foreclosing heterogeneity.
Mean-field reinforcement learning approximates an agent’s interaction with the whole population by its interaction with the average of its neighbours, converting a many-body problem into a two-body one. It is the principled way to scale, and it presumes agents are interchangeable and that the mean is a sufficient statistic — an assumption that fails exactly when a few agents matter more than the rest.
Rewards
Global rewards align every agent with the collective objective and destroy credit assignment. Local rewards give clean credit and encourage agents to optimize something that is not the collective objective. Neither works alone, and the field’s history is largely a history of splitting the difference.
Difference rewards compute an agent’s contribution as the collective outcome minus the counterfactual outcome had that agent not acted (or acted by default). This gives an aligned signal and clean credit. Computing the counterfactual is the difficulty.
Reward shaping adds intermediate signal to a sparse objective. Potential-based shaping is the safe form: adding a term for any potential provably leaves the optimal policy unchanged. Non-potential shaping optimizes something you did not intend, reliably and often invisibly.
Architectures
Graph neural networks are the natural inductive bias. An agent aggregates messages from its neighbours through a permutation-invariant function, which builds locality and interchangeability into the architecture rather than hoping they are learned. A network trained at one swarm size then transfers to another, because nothing in it depends on agent count.
Attention lets an agent weight neighbours by relevance rather than uniformly, which matters when a few neighbours dominate the decision.
Experience sharing across identical agents multiplies sample efficiency by the swarm size — every agent’s trajectory trains the shared policy. This is why parameter sharing is so hard to beat.
Applications
Robotic swarm coordination (formation, coverage, pursuit), distributed sensing and monitoring under an energy budget, traffic signal control, vehicle platooning, and demand-responsive fleet positioning. Most published results run at tens of agents, and the ones that claim thousands are usually running mean-field or parameter-shared policies where the agent count is nearly free.
Open problems
Scale. Above roughly ten thousand agents, per-agent computation, sample efficiency, and the sheer cost of simulation all bind. Mean-field approximation, hierarchical abstraction, and locality-preserving architectures are the available responses, and none of them makes the problem go away.
Guarantees and safety. Convergence is not guaranteed in the multi-agent setting. Safety during learning — a robot that must not crash while discovering that crashing is bad — requires constrained methods. Robustness certification under agent failure or adversarial behaviour is largely open, as is explaining a learned emergent behaviour to a human operator. Compare formal verification generally.
Sim-to-real. Policies learned in simulation fail on hardware in proportion to the simulator’s infidelity. Domain randomization — training across many perturbed simulators — is the standard hedge; simulation pre-training with real-world fine-tuning is the standard practice.
The gap this lab works in
There is a fourth open problem that gets less attention, and it invalidates more results than the other three.
Nearly every multi-agent reinforcement learning environment gives each agent an instantaneous view of its neighbours. Policies learned under that assumption are policies for a world that does not exist at any interesting scale — and the failure is not graceful. Our measurement is that coordination quality collapses through a delay boundary near 10–20 steps of latency, identically for learned and hand-designed policies, identically at 500 and 2,000 agents. A policy trained without delay does not degrade under delay; it stops coordinating.
The tractable response is not a better architecture. It is to let the agent predict where its neighbours have gone since it last saw them — and, empirically, constant-velocity extrapolation beats a Kalman filter at doing so, because bounded actuation keeps trajectories nearly linear over the horizon and the filter’s priors cost more variance than they buy.
For the modern architectural toolkit, see MARL, GNNs, and world models. For what these methods are compared against, see the Arboria Swarm Benchmark.