TerraTransfer:
Learning End-to-End Driving Policies Without Expert Demonstrations

End-to-end autonomous driving has achieved state-of-the-art performance on benchmarks and real-world deployments. Its standard training recipe, however, is expensive across all stages: collecting and labeling millions of driving frames is costly, and closed-loop RL on images is bottlenecked by the per-step cost of photorealistic rendering plus a forward pass through a large vision backbone. Self-play in vectorized simulators changes the economics: millions of rollout steps per second, and a state distribution naturally rich in collisions, near-misses, and recoveries that no driving log contains.

Our approach exploits this asymmetry by decoupling learning to drive from learning to see. We pretrain a single policy by self-play, then align its latent space with a pretrained vision backbone, through the action KL divergence and a batch-relational low-rank structural loss. The action target comes from the self-play policy, so alignment never supervises against a logged trajectory: a paired dataset of (image, scene-state) frames suffices, with no need for the curated expert demonstrations that imitation pretraining is built on. On photorealistic 3D Gaussian splatting closed-loop scenarios, the resulting end-to-end policy matches or exceeds prior end-to-end methods.

Phase 1 — Learn to drive (self-play). A single policy is trained end-to-end with PPO by multi-agent self-play in TerraZero, our vectorized simulator. One shared parameter set controls every agent in the scene, so the interaction distribution — collisions, near-misses, recoveries — co-evolves with the policy. Ego, map, and partner encoders feed a shared trunk and an action head; per-episode reward weights are sampled and exposed to the policy, yielding a single agent that spans a continuum of driving preferences.

Phase 2 — Learn to see (vision alignment). The Phase-1 policy is frozen and used purely as a source of per-frame supervision. For each log frame, a frozen DINOv3 backbone extracts image features, and two linear adapters map them to the road and partner features the policy consumes, while the ego encoder and action head are inherited and kept frozen. Two losses tie student to teacher: a structural loss that matches the teacher's relational scene geometry within its low-rank subspace, and an action loss (KL) on the output action distribution. No logged ego trajectory is ever used as a supervision target.

The two loss terms are also complementary: action-only reaches 0.319 and structural-only 0.307, but combined they reach 0.490. The batch-wise subspace is itself principled — at batch size 1024 it faithfully recovers the dominant teacher modes while still separating small geometric scene perturbations (see appendix diagnostics).

We evaluate on HUGSim, a photorealistic 3D Gaussian-splatting closed-loop benchmark over 88 nuScenes-derived scenarios across four difficulty tiers, and report the aggregate closed-loop HD-Score (higher is better). Our vision policy is aligned on nuPlan and evaluated on nuScenes scenarios, so these scores also reflect cross-dataset generalization.

Bold marks the best per column among comparison methods. The italicized self-play teacher drives from privileged vectorized state (not images) and is excluded from best-per-column counts. Our vision policy is the strongest method on aggregate (0.490 All) — beating the best imitation-trained baseline, LTF, by 0.130 and the strongest ECO variant by 0.038 — while approaching its own teacher (0.520) without any logged-trajectory supervision.

The main exception is the Extreme tier, where ECO leads. These scenarios are strongly out of distribution: surrounding vehicles may actively collide with the ego, and in some cases pass through other vehicles before doing so. Our policy responds conservatively, which preserves safe behavior but often sacrifices route completion — the resulting low route completion pulls down the aggregate HD-Score on this split.