SPACeR: Self-Play Anchoring with Centralized Reference Models

Existing approaches face a fundamental trade-off:

• Imitation learning (IL): Realistic and human-like, but tokenized or diffusion models are large (GPU memory heavy), slow, and difficult to scale.
• Self-play reinforcement learning (RL): Efficient and scalable, but requires reward shaping and often diverges from human norms.

SPACeR is an RL-first approach designed to bridge these gaps—combining the scalability of self-play with the realism of IL. Our policies are ~50× smaller than tokenized models and run 10× faster (or more!), enabling lightweight, human-like multi-agent simulation.

We propose to anchor self-play reinforcement learning to a pretrained tokenized reference model, which provides a human-likeness distributional signal. The resulting SPACeR policy is decentralized and conditioned only on local observations, while the reference model is centralized and conditioned on the full scene context—allowing scalable training without sacrificing realism.

Formally, we augment self-play reinforcement learning with a pretrained reference policy \(\pi_{\text{ref}}\) that captures the human driving distribution and provides a realism signal during training. The overall objective is

\[ r_t = r_t^{\text{task}} + \alpha \, r_{\text{humanlike}}(s_t, a_t), \quad L(\theta) = L_{\text{PPO}}(\theta; A[r]) - \beta \, D_{\text{KL}}\!\left(\pi_{\text{ref}}(\cdot \mid s_t) \,\|\, \pi_\theta(\cdot \mid o_t)\right), \]

where (1) \(L_{\text{PPO}}\) optimizes task performance (goal-reaching, collision avoidance, off-road avoidance), (2) the human-likeness reward \(r_{\text{humanlike}}(s_t, a_t) = \log \pi_{\text{ref}}(a_t \mid s_t)\) provides dense per-timestep likelihood feedback, and (3) the distributional alignment term enforces KL-regularization between the reference and learned policies. This formulation ensures agents learn from experience in a closed-loop manner while also remaining closely aligned with realistic human driving behaviors

To incorporate human-likeness into self-play, we introduce a pretrained reference tokenized model (\(\pi_{\text{ref}}\)), trained on real-world driving trajectories as a proxy for the human driving distribution. Tokenized models (e.g., SMART) factorize the joint action distribution under a conditional independence assumption:

$$ p(a_t \mid a_{<t}, c) = \prod_{i=1}^{N} p(a_t^i \mid a_{<t}, c). $$

This yields per-agent (i) distributions at each timestep (t). Unlike autoregressive generation, our approach only requires tractable training a single forward pass per rollout. By aligning the action space of \(\pi_\theta\) and \(\pi_{\text{ref}}\), we obtain a direct distributional signal that guides scalable self-play toward human-like behavior.

We measure whether SPACeR self-play policies are human-like by adopting the Waymo Sim Agents Challenge (WOSAC). We compare SPACeR against two self-play RL baselines:

• PPO: Only using task reward (goal reaching, collision avoidance, off-road avoidance)
• HR-PPO: Previous works that use Behavior Cloning as KL-regularization
• SMART*: State of the art tokenized traffic models
• CAT-K*: Supervised Fine-tuning on top of SMART

SPACeR outperforms other self-play approaches significantly across all realism metrics. In addition, compared to imitation-learning methods, SPACeR is lightweight(65k params , 50x smaller than SMART 3M) and achieves ~10× throughput while maintaining competitive performance.

*Note: We retrain SMART and CAT-K with the same token size of 200 at 5Hz, where CAT-K is used as a reference model.

We evaluate SPACeR by comparing how planners perform under different sim agent policies. Specifically, we test 22 self-play–trained policies, 10 sampling-based Frenet planners, and 10 IDM-based planners. For each planner, we compute PDM scores across diverse scenes under three simulation modes: ground-truth log replay, Cat-K rollouts, and SPACeR agent policies. We then measure the correlation of PDM scores across sim agent strategies to understand how similarly different agents assess planner behaviors.

The correlation of collision scores is low between SPACeR against Log-Replay and CAT-K. Qualitatively, both Log-Replay and CAT-K yield more false positive collisions, giving a wrong estimate of the planner's safety performance. In contrast, SPACeR agents provide more reactive, realistic behavior that better reflects real-world driving scenarios.

In summary, although CAT-K agents achieve higher distributional realism on the Sim Agents Challenge, we observe that in closed-loop planner evaluation, self-play–trained agents exhibit more reactive and adaptive behaviors, especially in collision and negotiation scenarios. This likely explains why CAT-K and the non-reactive Log-Replay agents show higher correlation in collision scores.