SpaceMind: Camera-Guided Modality Fusion for Spatial Reasoning in Vision-Language Models — arXiv:2511.23075 · PDF
SpaceMind is a vision-language model for 3D spatial reasoning from RGB observations, aiming to answer questions involving distance/size comparison, relative direction, route planning, counting, and cross-view consistency without requiring explicit 3D sensors at inference time.
- problem: general VLMs are strong at perception/reasoning but often fail on explicit 3D-aware spatial reasoning.
- idea: treat the camera representation as a guiding modality (not just metadata) to control how geometry cues are fused before the LLM.
- architecture: a dual-encoder design with InternViT (2D visual encoder) + VGGT (spatial encoder) and a lightweight Camera-Guided Modality Fusion (CGMF) module.
We reproduce SpaceMind training on Huawei Ascend NPU (昇腾) 910C with 32 NPUs (4 nodes × 8 NPUs/node) using HCCL.
This repo is based on InternVL (internvl_chat/). Upstream documentation: InternVL readthedocs.
Install CANN and torch_npu matching your 910C firmware/driver, following the official Ascend PyTorch guide.
pip install -r requirements.txtThe launcher auto-selects HCCL when npu-smi is available; you can also set DIST_BACKEND=hccl explicitly.
From the repository root, set paths if they differ from the defaults (SPACEMIND_ROOT, DATA_URL, OUTPUT_DIR, MODEL_NAME_OR_PATH; weights include InternVL3-8B and VGGT under ${SPACEMIND_ROOT}/models/).
Example 32-NPU (4 nodes × 8 NPUs/node) launch (run on each node with the same MASTER_ADDR / MASTER_PORT / NNODES and a unique NODE_RANK):
NNODES=4 NODE_RANK=0 MASTER_ADDR="xx.xx.xx.xx" MASTER_PORT="56789" bash train_nv_version.shThe script cds into internvl_chat/, sets LAUNCHER=pytorch, uses HCCL when npu-smi is available, and runs torchrun on internvl/train/internvl_chat_finetune.py with --model_type spacemind.
Offline evaluation uses the same training entrypoint with --do_train False --do_eval True, loads a fine-tuned checkpoint (HF safetensors preferred), and runs VSI-Bench.
Place weights under ${SPACEMIND_ROOT}/weight/ (or pass explicit paths):
weight/
InternVL3-8B/ # base model
VGGT/model.pt # spatial encoder
best_ckpt/ # fine-tuned checkpoint (prefer HF safetensors)
${DATA_URL} (default ${SPACEMIND_ROOT}/data) must contain:
opensource/questions.jsonl
data/video_sampled/<dataset>/<scene>/
# Activate your Ascend / torch_npu environment first, then:
bash eval.sh
bash eval.sh --max_questions=8 --nproc_per_node=4
bash eval.sh --data_url=/path/to/vsibench_root --checkpoint=/path/to/best_ckpt
# optional: ACTIVATE_CONDA=/path/to/activate.sh bash eval.shResults are written to ${OUTPUT_DIR}/eval_results_standalone/<step>_eval.txt.
Prefer HF best_ckpt over DeepSpeed ZeRO shards: ZeRO load requires the same world size used when saving.
- Overall Score: 0.721
- object_rel_direction: 0.896
- object_rel_distance: 0.713
- route_planning: 0.454
- obj_appearance_order: 0.804
- object_abs_distance: 0.627
- object_counting: 0.724
- object_size_estimation: 0.780
- room_size_estimation: 0.775
- Data processing (documented pipeline + scripts for preparing training/eval data).
- Open-source model weights (HF links + checksums).
| Path | Role |
|---|---|
eval.sh |
Standalone VSI-Bench evaluation launcher |
internvl_chat/internvl/model/spacemind/ |
SpaceMind model |
internvl_chat/internvl/model/fusion/cgmf.py |
CGMF fusion |
internvl_chat/internvl/model/vggt/ |
VGGT spatial encoder integration |
internvl_chat/internvl/train/vsibench_eval_callback.py |
VSI-Bench eval (train-time + offline) |
internvl_chat/internvl/train/checkpoint_loader.py |
Offline checkpoint loading for eval |
internvl_chat/shell/data/ |
Dataset meta JSON for --meta_path |
internvl_chat/zero_stage2_config.json |
DeepSpeed ZeRO-2 config used by train_nv_version.sh |
SpaceMind contributions in this repository are licensed under the Apache License 2.0 (LICENSE). Third-party components (for example InternVL and VGGT) remain under their respective licenses; see NOTICE.
@article{zhao2025spacemind,
title={SpaceMind: Camera-Guided Modality Fusion for Spatial Reasoning in Vision-Language Models},
author={Zhao, Ruosen and Zhang, Zhikang and Xu, Jialei and Chang, Jiahao and Chen, Dong and Li, Lingyun and Sun, Weijian and Wei, Zizhuang},
journal={arXiv preprint arXiv:2511.23075},
year={2025}
}Built on InternVL and related open components; see NOTICE and per-directory license files.