Metric-semantic 3D goal grounding

Meanings and Measurements

Multi-Agent Probabilistic Grounding for Vision-Language Navigation

MAPG decomposes metric-semantic instructions into referent, directional, and metric components, grounds them against an online 3D scene graph, and composes continuous spatial kernels into a planner-ready goal distribution.

100 MAPG-Bench queries41 HM3D scenesContinuous goal distributions
MAPG motivating example
A metric-semantic instruction requires simultaneous object grounding, spatial-relation reasoning, and metric constraint enforcement.
01

Method

From language and observations to an executable spatial goal.

Orchestrate

Parse the query into anchor, relation, and metric clauses.

Ground

Resolve the anchor instance using scene-graph and visual evidence.

Compose

Combine semantic, directional, and metric kernels in log space.

Plan

Mask to navigable space and return the highest-density goal.

MAPG system overview
MAPG system overview and agent interaction loop.
02

MAPG-Bench results

Trace-derived evaluation using one consistent adapter across stored outputs.

2.09 mLowest O-O errorMAPG, Claude Opus 4.6
2.57 mLowest O-W errorMAPG, Gemini 3.7 Flash
69.16°Lowest MAPG angular errorMAPG, GPT-5.6 Luna
0.93Highest completionMAPG, GPT-5.6 Luna
Table I. Metric-semantic grounding on MAPG-Bench.
MethodO-O ↓O-W ↓Angle ↓Obj. Sel. ↑Intrinsic Rel. ↑Completion ↑Anchor ↑Traj. ↓
SpatialRGPT, VILA1.5-8B7.036.8781.05°0.000.610.95N/AN/A
GraphEQA, GPT-5.6 Luna4.164.3087.46°0.040.080.270.2014.96
GraphEQA, Gemini 3.7 Flash4.514.7280.77°0.050.520.810.4012.33
MAPG, GPT-5.22.452.9272.93°0.220.590.860.564.24
MAPG, Gemini 3.7 Flash2.132.5773.87°0.270.530.830.575.38
MAPG, Claude Opus 4.62.092.6673.18°0.260.470.810.554.33
MAPG, Claude Sonnet 52.172.5976.59°0.240.470.740.516.64
MAPG, GPT-5.6 Luna2.532.9269.16°0.230.670.930.614.29

Distance and trajectory values are in meters. Angle is in degrees. Completion records a confident non-null output and is not ground-truth waypoint correctness. Anchor selection uses category and 3D-center remapping for GraphEQA local object identifiers.

Table II. HM-EQA question answering.
MethodAccuracy ↑Trajectory ↓
Explore-EQA, Llama4-Mav0.4410.4
Explore-EQA, Gemini 2.5 Pro0.5412.3
GraphEQA, GPT-5.20.637.1
GraphEQA, Claude Opus 4.60.647.4
MAPG, GPT-5.20.606.9
MAPG, Claude Opus 4.60.716.6
Tables III and IV. Object-selection ablations.
ConfigurationFull ↑Occluded ↑
GraphEQA base0.340.30
MAPG CoT, no spatial reasoner0.200.30
MAPG with spatial reasoner0.420.50
03

Composed grounding

Each analytic component contributes to the final goal distribution.

Semantic grounding
Semantic grounding
Directional kernel
Directional kernel
Metric kernel
Metric kernel
Composed goal
Composed goal
04

References

Prior work and evaluation resources discussed on this page.

  1. Kim et al. LINGO-Space: Language-Conditioned Incremental Grounding for Space. AAAI 2024.
  2. Saxena et al. GraphEQA: Using 3D Semantic Scene Graphs for Real-time Embodied Question Answering. 2024.
  3. Cheng et al. SpatialRGPT: Grounded Spatial Reasoning in Vision Language Models. 2024.
  4. Ren et al. Explore until Confident: Efficient Exploration for Embodied Question Answering. 2024.
  5. Ramakrishnan et al. Habitat-Matterport 3D Dataset. 2021.