Details
- NVIDIA AI highlights a new diagnostic benchmark called Spatial-IQ for evaluating multimodal models on 3D object counting.
- The post compares human performance at 82.1% accuracy against 17.7% for the best off-the-shelf multimodal model on a hidden-box counting task.
- Spatial-IQ decomposes stacked 3D object counting into nine perceptual and cognitive sub-tasks aligned with stages of human spatial cognition.
- The benchmark is built on roughly 80,000 procedurally generated 3D structures using NVIDIA Isaac Sim, each with task-specific ground truth labels.
- NVIDIA evaluates models in multiple output formats, including free-response text, multiple-choice images, and image editing, alongside human baselines.
- The research shows models can succeed at final object-count tasks while failing underlying sub-tasks, exposing shortcut reasoning that standard accuracy metrics miss.
- Training with chain-of-thought supervision over the hierarchical sub-tasks, combined with reinforcement learning using verifiable rewards, significantly improves spatial consistency and counting accuracy.
- NVIDIA releases the Spatial-IQ paper, dataset, and code publicly, positioning it as both a diagnostic tool and a training signal for future multimodal and vision-language models.
Impact
Spatial-IQ strengthens NVIDIA’s role in multimodal AI evaluation by targeting a core weakness in current models: reliable spatial reasoning in 3D scenes. By decomposing object counting into cognitive sub-tasks and showing measurable gains from structured supervision, the benchmark could influence how competitors design training regimes and benchmarks for next-generation vision-language systems.
