G2VLM: Geometry Grounded Vision Language Model with Unified 3D Reconstruction and Spatial Reasoning

G2VLM is a geometry-grounded vision-language model designed to address one of the most persistent weaknesses of modern vision-language models (VLMs): spatial understanding and reasoning. While recent VLMs have achieved remarkable success in image understanding, multimodal dialogue, and visual question answering, they often struggle with tasks that require a deeper grasp of three-dimensional structure, object relationships, depth, and spatial reasoning. The authors argue that this limitation arises because most VLMs learn primarily from large collections of two-dimensional images paired with text and therefore lack an explicit mechanism for reconstructing and understanding the geometry of the physical world. 

To overcome this problem, the paper proposes a unified framework that combines 3D visual geometry learning with high-level multimodal reasoning. Rather than treating images as flat visual inputs, G2VLM learns to infer the underlying three-dimensional structure of scenes and then uses this geometric knowledge to support spatial reasoning. According to the authors, spatial intelligence depends on both identifying objects and understanding their locations and physical relationships. G2VLM is designed to model both capabilities simultaneously. 

The architecture is inspired by the neuroscience concept of the two-streams hypothesis, which proposes separate pathways for recognizing objects ("what") and understanding spatial relationships ("where"). Reflecting this idea, G2VLM uses a Mixture-of-Transformer-Experts (MoT) architecture containing two specialized components. The first is a geometric perception expert, responsible for learning 3D geometry, reconstructing point clouds, estimating depth, and predicting camera poses from image inputs. The second is a semantic perception expert, built upon a pretrained vision-language model, which performs multimodal understanding and language reasoning. Rather than operating independently, these experts share attention mechanisms that allow geometric and semantic information to interact and reinforce one another. 

A key contribution of the work is its ability to learn geometry from ordinary images and videos without requiring extensive handcrafted 3D supervision during downstream reasoning tasks. The model is trained in two stages. First, the semantic component is frozen while the geometric expert learns rich 3D representations from large-scale datasets containing geometric annotations such as depth and camera information. Next, the semantic expert is unfrozen and jointly trained with the geometric expert on spatial understanding tasks. This second stage allows the language model to incorporate learned geometric representations into its reasoning process, producing a system that can both reconstruct scenes and answer spatial questions about them. 

The geometric learning process is sophisticated and centers on reconstructing the three-dimensional structure of scenes. The model predicts pixel-aligned 3D point maps and camera poses, using losses that measure point reconstruction accuracy, camera pose estimation quality, and surface normal consistency. Through these objectives, the geometric expert acquires a detailed understanding of spatial layouts, depth relationships, and scene structure. Unlike systems focused solely on geometric reconstruction, however, G2VLM integrates these capabilities directly into a language model, enabling geometry to influence reasoning and decision-making. 

Experimental results demonstrate the effectiveness of this unified approach. On visual geometry benchmarks, G2VLM achieves performance comparable to state-of-the-art reconstruction systems such as VGGT. The model shows strong results on tasks including monocular depth estimation, point reconstruction, and camera pose estimation. For example, the paper reports an improvement in Absolute Relative Error on the Sintel depth benchmark, reducing the error from 0.335 to 0.297 compared with VGGT. 

The model also performs exceptionally well on spatial reasoning tasks. Across multiple benchmarks designed to evaluate understanding of spatial relationships and physical scene structure, G2VLM matches or exceeds the performance of significantly larger models. The paper highlights particularly strong performance on SPAR-Bench, where G2VLM surpasses GPT-4o by 18.5 percentage points. Notably, these results are achieved with a relatively compact 2-billion-parameter model, suggesting that explicit geometric grounding may be more valuable for spatial reasoning than simply scaling model size. 

An important finding from the study is the positive interaction between geometric and semantic learning. As the geometric expert improves its ability to reconstruct 3D scenes, the semantic reasoning capabilities of the model also improve. This supports the authors’ central hypothesis that robust spatial reasoning requires a dedicated understanding of geometry rather than reliance solely on language patterns or two-dimensional visual cues. 

In conclusion, the paper presents G2VLM as the first unified model that simultaneously performs high-quality 3D reconstruction and advanced spatial reasoning within a single vision-language framework. By incorporating an explicit geometric perception pathway and allowing it to interact closely with semantic reasoning, the model bridges a gap between low-level visual geometry and high-level multimodal intelligence. The results suggest that future progress in spatial intelligence, robotics, embodied AI, and potentially applications such as 3D scene editing may benefit significantly from integrating geometric understanding directly into foundation models. 

References: https://arxiv.org/html/2511.21688v1