Modern MARL, GNNs, and World Models
The other Techniques pages cover the classical algorithms — Boids, PSO, ACO, evolutionary computation, and reinforcement learning. Contemporary research runs a second layer of methods on top of, or in place of, those policies. This page is the map.
Multi-agent reinforcement learning
MAPPO — multi-agent PPO with parameter sharing and a centralized critic — is the current default baseline for continuous-control swarm tasks. It scales, it is stable, and it produces sensible policies on modest sample budgets. Yu et al. (2021), The Surprising Effectiveness of PPO in Cooperative Multi-Agent Games, is the reference, and the title is the finding: a straightforward method, carefully implemented, beats most of what was proposed to replace it.
QMIX, VDN, and MADDPG are the value-decomposition and mixed actor-critic families. They earn their place on discrete or sparse-reward problems. For continuous physics they are generally worse than MAPPO and considerably more fiddly.
IPPO — independent PPO, no centralized critic — is worth running as an ablation, since it measures exactly what the centralized critic buys. On some tasks the answer is embarrassingly little.
Graph neural network policies
A swarm is a graph: agents are nodes, near-neighbour interactions are edges, and a coordination rule is message passing over them. Treating it that way is not a metaphor but an architectural commitment with three consequences.
It handles variable agent counts by construction — no padding, no masking, no fixed input dimension. It is permutation-invariant, so a policy trained on 100 agents runs on 100,000 without retraining, because nothing in the architecture references agent identity or count. And it composes: consensus and flocking chain into a single forward pass.
The commitment worth making explicit is that classical policies fit the same interface. Boids is a zero-parameter graph neural network layer. Laplacian consensus is another. When a hand-designed baseline and a learned policy run behind the same interface, a comparison between them measures algorithms rather than plumbing — which is the design principle behind Gossamer’s graph substrate.
The standard libraries are PyTorch Geometric and, for JAX, Jraph .
World models
Classical reinforcement learning learns a policy directly from experience. A world model instead learns a predictive model of the environment, then plans against it or trains a policy on imagined rollouts. The payoff is sample efficiency, which matters most when the environment is expensive to simulate — the regime most swarm questions live in.
JEPA — Joint-Embedding Predictive Architecture, Yann LeCun’s 2022 proposal — predicts representations of future observations in a latent space rather than reconstructing observations. This suits swarms structurally: per-agent observations are low-dimensional while the joint state is enormous, and latent prediction sidesteps the mismatch instead of paying for it.
Dreamer (Hafner et al., through V3) pairs a recurrent latent world model with an actor-critic trained entirely on imagined rollouts. It is the most battle-tested model-based baseline available, with strong sample efficiency across a very wide task suite.
Graph world models predict future node states directly over the interaction graph, in the manner of GraphCast. Architecturally this is the best fit for swarms, and it is where the open research is — considerably fewer shipping implementations than JEPA or Dreamer.
When they help. Pure consensus convergence does not need a world model; classical control theory is sharper and comes with proofs. Delayed or partitioned communication does, because under delay an agent must reason about where its peers have gone rather than react to where they were. Anticipation is worth modelling explicitly.
Our own result on that is a caution for anyone reaching straight for a learned model. Across both tasks and every delay we tested, the ranking was constant-velocity extrapolation ≳ linear fit > Kalman filter > no prediction. The simplest possible model of a peer beat the principled filter, because bounded actuation keeps trajectories nearly linear over the prediction horizon and the filter’s process priors add variance they never repay. A learned graph predictor is a plausible improvement on that baseline. It is not a substitute for checking whether the baseline was already sufficient.
Emergent communication
Give reinforcement-learning agents a differentiable channel and they learn a protocol. Whether the protocol generalizes and transfers is the research question.
Channel constraints determine protocol structure. Bandwidth, latency, loss, and per-message energy cost each produce structurally different learned languages. A protocol learned on a free, instantaneous, lossless channel is a protocol for a world that does not exist, and this is the most common unforced error in the literature.
The standard instruments for analyzing a learned protocol are topographic similarity, context independence, compositionality, and disentanglement.
The question we find most interesting: do learned protocols rediscover CRDT -like abstractions under delay-tolerant conditions? Convergence under partition without a central arbiter is a strong constraint, and it has a small number of solutions. If gradient descent finds one of them, that is a fact about the problem rather than about the architecture.
Foundational references: Foerster et al. (2016) on differentiable inter-agent learning; Lazaridou and Baroni (2020) for a review; Chaabouni et al. (2020) on compositionality.
Putting it together
A defensible contemporary swarm stack:
Physics with a large-N engine, differentiable where gradient-based shaping is required. Policy substrate exposing one message-passing interface for classical and learned policies alike. Learned policy by MAPPO under centralized-training/decentralized-execution, optionally with a learnable communication channel. World model where anticipatory behaviour is needed — Dreamer, or a graph JEPA. Evaluation against a fixed scenario and baseline suite with published seeds; see the Arboria Swarm Benchmark.
That is the surface area modern swarm research must cover to be legible to reviewers. The part reviewers do not yet ask for, and should, is what any of it does when an agent’s view of its neighbours is an hour old.