TraVEL: Trajectory-Guided Video Embedding Learning for Driving-Video Retrieval

Yi-Chung Chen^1,2, Philip Jacobson¹ , Tom Lampo¹, Yiren Lu¹, Jin Yao¹, David I. Inouye², Jing Gao², Danhua Guo¹, Burhan Yaman¹^†

¹Uber AV Labs
²Purdue University
Corresponding author
^†denotes project lead

Paper arXiv

Before TraVEL, two daytime clips with opposite ego motion have high similarity while a day/night pair sharing the same trajectory has low similarity; after TraVEL, the ordering is reversed.

TraVEL reshapes the video embedding space around ego motion instead of visual context. Before trajectory-aware fine-tuning (left), two daytime clips receive a relatively high cosine similarity (0.420) despite exhibiting different ego motion — an accelerating left turn versus a decelerating right turn — while daytime and nighttime clips that share the same accelerating left-turn trajectory score only 0.123. TraVEL reverses this context bias (right), dropping the motion-incompatible pair to 0.094 and lifting the matched-trajectory pair to 0.462.

Abstract

Efficiently retrieving relevant clips from large-scale driving logs is essential for data curation, model development, and safety analysis. Structured and rule-based retrieval systems can explicitly target driving events, but typically require expert-defined rules, auxiliary data, and multi-stage perception pipelines. Multimodal embedding models offer a simpler and more efficient alternative by representing each video with a single searchable vector. However, general-purpose models often rely on shortcuts from static scene context and struggle to distinguish motion-centric events, such as turning left versus right or accelerating versus decelerating. In this work, we study how to adapt a general-purpose multimodal embedding model to driving-video retrieval. We first fine-tune Qwen3-VL-Embedding on paired clips and reasoning traces from nuReasoning using an InfoNCE objective. While this stage substantially improves overall retrieval, caption supervision alone remains insufficient for fine-grained motion understanding. We therefore introduce TraVEL (Trajectory-Guided Video Embedding Learning), a motion-aware fine-tuning framework that uses ego-trajectory similarity as a reward within Group Relative Policy Optimization. Trajectories serve only as privileged training supervision; retrieval still operates on single-vector video embeddings without ego poses, expert rules, or auxiliary perception outputs. We further construct a driving-video retrieval benchmark from nuReasoning. Experiments show that TraVEL improves motion-centric retrieval across model scales: relative to SFT, it raises longitudinal and lateral mAP by 9.8 and 4.7 points at 2B, with corresponding gains of 7.2 and 1.5 points at 8B. TraVEL thus combines physically grounded supervision with efficient embedding-based search.

Retrieving the Right Maneuver

The same reasoning-trace query — “gently accelerate to execute the left turn at the green light while monitoring the pedestrian to ensure they safely clear the crosswalk” — issued against the same 1,715-video pool. Each cell plays the retrieved clip; the labels under it are that clip's lateral and longitudinal ego motion, checked against the ground-truth clip's motion.

Pretrained — ground truth ranked #115, 0/5 of the top-5 share the queried ego motion

TraVEL — ground truth ranked #1, 5/5 of the top-5 share the queried ego motion

The pretrained embedding retrieves plausible intersections at green lights, but the ego vehicle in them is stopping or driving straight rather than accelerating through a left turn, and the paired clip falls to rank 115. After trajectory-aware fine-tuning the paired clip is ranked first and all five of the top results execute the same maneuver — accelerating through a left turn — including night-time clips whose appearance differs sharply from the query's daytime ground truth.

Diagnosing the Pretrained Embedding Space

Density of cosine similarity between each caption and its paired video versus non-paired videos, for Pretrained and TraVEL.

(a) Caption–video similarity distributions

t-SNE projection of video and caption embeddings coloured by longitudinal motion category, for Pretrained and TraVEL.

(b) Joint caption–video embedding space

Running Qwen3-VL-Embedding on driving videos without adaptation exposes two failures. In (a), the cosine similarity between a reasoning-trace caption and its paired ground-truth video is concentrated at nearly the same high values as its similarity to every other video: the two densities overlap heavily, so paired and non-paired clips are almost indistinguishable and there is little discriminative margin left for retrieval. In (b), a t-SNE projection shows a wide modality gap between caption and video embeddings, and video embeddings from different longitudinal-motion categories are densely intermixed — the pretrained space simply does not organize driving clips by ego motion. We attribute this to a domain gap between general-purpose pretraining data and driving video, plus the fact that text supervision is only an indirect signal for fine-grained vehicle dynamics. The TraVEL panels in each plot preview what the two-stage pipeline recovers.

Method

TraVEL's trajectory-aware GRPO stage: online retrieval forms a text-to-video group with binary instance rewards and a video-to-video group with graded ego-trajectory rewards, both optimized with group-standardized advantages.

Stage 1 — driving-domain supervised fine-tuning. We first close the domain gap with a contrastive text-to-video InfoNCE objective over reasoning-trace captions paired with 8-second clips, where every non-matching video in the batch is a negative. This aligns driving language with visual content and yields the checkpoint that initializes the reinforcement stage. But InfoNCE gives two clips with nearly identical ego motion no more affinity than two physically unrelated clips — binary supervision poorly matched to the continuous structure of vehicle dynamics.

Stage 2 — trajectory-aware GRPO. Each clip is paired with a synchronized ego-pose sequence (T = 80 poses at 10 Hz), from which we extract two scalar motion channels — ego-frame lateral displacement and speed — kept separate so the larger-scale channel cannot dominate. A logarithmic warp compresses large values to emphasize behavior changes near zero, such as the transition from stopped to accelerating. Channels are compared with dynamic time warping, which tolerates the same maneuver occurring at slightly different times within a clip, then normalized by a robust corpus-level percentile scale and combined into a graded reward in [0, R].

Two groups, two reward types. At each step the current policy retrieves over a candidate pool to form the groups on policy. The text-to-video group holds the paired video plus the highest-scoring hard negatives and uses a binary instance reward, preserving the cross-modal alignment learned in stage 1. The video-to-video group uses the graded trajectory reward, organizing the video space by ego motion. Keeping the graded reward off the text-to-video pathway matters: a motion description legitimately matches many clips, and forcing that many-to-many relation onto text-to-video retrieval would conflict with paired-caption supervision. GRPO standardizes rewards within each group and updates the shared embedding model.

Trajectories are privileged training signal only. Ego poses are never inputs to the embedding model. At retrieval time each clip is still a single vector, searched by cosine similarity — no ego poses, expert-defined rules, or auxiliary perception outputs.

Instance-Level Retrieval

Each reasoning-trace caption ranks all 1,715 videos in the held-out nuReasoning pool by cosine similarity. Recall is a percentage; MdR and MnR are raw rank positions. The chance row is the analytical expectation under a uniformly random ranking — the honest floor for every column.

Model	#Params	R@1 ↑	R@5 ↑	R@10 ↑	MdR ↓	MnR ↓
Chance-level reference
Uniform random ranking	–	0.1	0.3	0.6	858	858.0
Off-the-shelf models
CLIP4Clip	151M	0.6	2.6	4.7	415	545.0
InternVideo2	6B	0.1	0.6	1.0	860	858.6
Cosmos-Embed1	1.2B	2.4	9.2	14.8	106	231.7
Qwen3-VL-Embed	2B	1.3	4.5	7.4	266	400.5
Qwen3-VL-Embed	8B	1.6	5.2	7.8	264	371.7
Fine-tuned models
Qwen3-VL-Embed + TraVEL	2B	8.7	24.5	35.3	23	87.8
Qwen3-VL-Embed + TraVEL	8B	10.1	28.3	39.8	17	70.6

Every off-the-shelf model shows a substantial driving-domain gap: R@1 is at most 2.4 across 151M to 8B parameters, and InternVideo2 sits at the uniform-random reference. Scaling Qwen3-VL-Embed from 2B to 8B lifts zero-shot R@1 only from 1.3 to 1.6, so model scale alone does not close the gap. TraVEL raises R@1 to 8.7 at 2B and 10.1 at 8B, cutting the median rank of the paired video from 266 to 23 and from 264 to 17.

Motion-Centric Retrieval

To measure motion understanding independently of exact caption matching, each longitudinal or lateral motion phrase is used directly as a text query against the same video pool. We report per-phrase average precision and the mAP within each motion family. Best within each model scale in bold.

Motion query	2B			8B
Motion query	Pretrained	SFT	TraVEL	Pretrained	SFT	TraVEL
Longitudinal motion
Gently accelerate	34.1	49.9	55.7	29.2	49.9	56.6
Maintain speed	39.0	44.2	49.2	30.0	44.1	49.7
Slow down gently	25.2	30.7	30.4	32.3	32.0	34.2
Gently come to a stop	14.7	33.7	53.8	18.4	43.3	57.4
Remain stopped	10.6	70.8	89.4	21.8	82.0	89.4
mAP	24.7	45.9	55.7	26.3	50.3	57.5
Lateral motion
No lateral action	66.2	77.8	80.5	63.4	77.1	78.9
Turn right	13.1	56.0	69.8	19.1	70.5	72.0
Turn left	16.5	49.7	63.6	14.4	65.6	68.3
Right lane change	9.8	9.0	10.2	10.5	19.4	20.2
Left lane change	5.7	11.1	11.7	7.6	19.3	22.5
Slightly move right in the lane	4.6	4.2	4.7	7.7	6.4	6.8
Slightly move left in the lane	8.1	9.1	9.6	4.9	10.2	10.6
mAP	17.7	31.0	35.7	18.2	38.4	39.9

The table separates what supervised fine-tuning buys from what trajectory-aware optimization adds. At 2B, SFT lifts longitudinal and lateral mAP from 24.7 and 17.7 to 45.9 and 31.0; TraVEL pushes them to 55.7 and 35.7, a further +9.8 and +4.7 points over SFT. The same trend holds at 8B, where TraVEL adds +7.2 and +1.5 points on top of SFT's 50.3 and 38.4. The largest gains land on stopping and turning behaviors, exactly where a trajectory carries information a caption does not. Subtle within-lane shifts stay hard in every setting, including TraVEL.