Findings and Insights

We conducted comprehensive evaluation of 24 state-of-the-art multimodal foundation models in SAW-Bench. Our experiments reveal critical insights about the current state and limitations of situated awareness understanding in MFMs.

Camera rotation as a source of trajectory errors. We identify a systematic failure mode in Route Shape occurring when changes in camera rotation are decoupled from the observer's translational movement. To isolate this effect, we compare three controlled scenarios: (1) a straight path with stable head ori- entation (Figure 5 Left); (2) the same straight path with frequent head rotations (Figure 5 Middle); and (3) a true zigzag trajectory (Figure 5 Right). Despite identical translational motion in cases (1) and (2), even top-performing models frequently misclassify case (2) as a zigzag trajectory: Gemini 3 Flash does so in 60.0% of instances, while Qwen3-VL 235B misclassifies 53.3% of cases. As illustrated in Figure 4 Right, models justify these predictions by erroneously attributing camera orientation shifts to physical body displacement. This failure highlights a fundamental limitation in current MFMs: the inability to maintain a robust observer-centric coordinate system that distinguishes egocentric rotational pans from global positional updates.

Trajectory complexity and error accumulation. Spatial updating is an inherently accumulative process, where errors in estimating egocentric motion compound as the observer moves through an environment. In human navigation, this integration is highly sensitive to "noise" introduced by changes in orientation. To investigate whether MFMs exhibit a similar sensitivity to trajectory complexity, we stratify results on the Relative Direction task by geometric complexity: (1) Straight (pure translation), (2) Single Turn (one rotational update), and (3) Two Turns (multiple rotational updates). As shown in Table 1, increasing geometric complexity is associated with a substantial accuracy degradation for most models, particularly when trajectories involve multiple orientation changes. When quantified using relative performance drop with respective to straight trajectories, MFMs often exhibit significant degradation under multi-turn conditions, while human performance remains largely stable. This human-model gap suggests that current MFMs struggle to reliable integrate successive egocentric orientation changes over time, resulting in compounding errors as trajectories move away from simple translational motion.

Failure to maintain persistent object memory. A recurring failure mode across Spatial Memory tasks arises from models' difficulty in maintaining object persistence across egocentric motion. Although models often provide accurate descriptions of what is visible in individual frames or short temporal windows, they fail to reason about objects that leave the camera's field of view. As shown in Figure 6, models tend to infer that objects are absent in earlier frames simply because they are not visible, incorrectly treating first observation as object appearance rather than recognizing that the object may have existed outside the field of view. These errors suggest that current MFMs rely primarily on view-dependent evidence, rather than maintaining a persistent world-state representation over time.

Effect of openness on situated awareness. Figure 7 summarizes model performance across indoor and outdoor environments. Contrary to the intuition that larger and more dynamic outdoor environments may increase spatial reasoning difficulty, no consistent performance degradation is observed in outdoor scenes. Across the four selected models, outdoor performance is often comparable to, and in several cases higher than, indoor performance. On average, the indoor-outdoor performance gap remains small. These results suggest that environment scale alone does not determine spatial reasoning difficulty. While outdoor scenes typically span larger spatial extents, they often contain fewer objects and exhibit less structural clutter than indoor environments, which may reduce relational ambiguity. As a result, spatial reasoning difficulty is not monotonically correlated with scene size or openness. Instead, indoor environments can pose equally, if not more, complex spatial challenges due to higher object density and more intricate layout structures.

@article{
}