The Arboria Swarm Benchmark
A fixed set of scenarios and reference baselines. Every coordination policy we evaluate reports against the same matrix, which is what separates “we tuned flocking” from “we made measurable progress against prior art.”
Availability. The benchmark currently ships inside Gossamer, which is proprietary. A versioned, publicly runnable release — with standardized fault models and a leaderboard harness that requires no access to our internal stack — is the deliverable of our constellation-benchmark work. Until it lands, treat this page as a description of our internal evaluation protocol rather than as something you can execute today. See Reproducibility and Data Availability.
Scenarios
Each scenario fixes an initial state, a per-step reward, and a terminal metric.
| Name | Question | Terminal metric |
|---|---|---|
dispersal | How fast can a clumped swarm spread without colliding? | Mean nearest-neighbour distance at termination |
rendezvous | How fast does a scattered swarm meet at a common point? | Final mean distance to centroid (lower is better) |
coverage | Explore a bounded region; maximize cells visited per unit time | Unique cells visited over total cells |
leader_follower | One agent is exogenously driven; keep the swarm within range | Mean follower distance to the leader’s path |
byzantine | Inject a fraction of silently faulty agents | Terminal metric of the base scenario, under perturbation |
Benchmark scenarios are a different thing from Gossamer’s tasks, and the distinction matters. Scenarios exist to rank policies across a breadth of objectives. Tasks exist to define the coordination quality that the delay papers measure, and each is scored against a peer-derived target so that coordination is provably necessary to score well. Scenarios are for comparison; tasks are for measurement.
Baselines
Every new policy reports against the same reference set, so a reviewer has a stable point of comparison rather than a number in isolation.
random applies uniform random accelerations. It is the lower bound, and a
surprising number of published policies fail to clear it convincingly on at least
one scenario.
greedy is a per-scenario hand-crafted heuristic — move toward the centroid
for rendezvous, push away from the nearest neighbour for dispersal, a persistent
random walk for coverage. It is what a competent engineer writes in an afternoon,
and beating it is the real bar.
gossamer_flocking is classical
Boids, included because it is the
field’s shared reference point.
Learned policies — MAPPO and its relatives — are trained and evaluated against the same scenarios through Gossamer’s graph substrate, which deliberately presents classical and learned policies behind one interface so a comparison between them is not confounded by plumbing.
Harness
The harness runs a policy against a scenario for a fixed step budget across several seeds and emits one row per run: scenario, baseline, agent count, steps, seed, terminal metric, mean reward, and elapsed wall-clock. An aggregator collapses those rows into a leaderboard table.
By default the stepper is pure NumPy, so the suite carries no dependency on the C++ engine at test time and runs anywhere. For numbers that feed a paper, the same scenarios re-run through Leviathan, so that collisions, the communication cost model, and the energy and fault modules are the real ones. The two paths are not interchangeable and we do not mix them within a single table.
Reproducibility
Seeds are recorded with each row, and rerunning at the same seed yields identical numbers — Gossamer takes an explicit random generator at every stochastic entry point and makes no module-level random calls anywhere.
What comes next
The constellation benchmark extends this suite with real orbital geometry: contact plans derived from propagated orbits rather than an assumed always-on network, eclipse and power budgets, radiation-induced transient faults, and adversarial bidders. That release is intended to be neutral, versioned, and publicly runnable — the point being that a benchmark only functions as a community standard if the community can actually run it.