Note
This is the video2morph integration fork used by X-Morph. It preserves the upstream Fast SAM 3D Body implementation and adds live monocular video-to-robot retargeting and reference publishing. The X-Morph additions are documented under X-Morph Live Retargeting; see the upstream repository for the original project.
Timing Yang1, Sicheng He1, Hongyi Jing1, Jiawei Yang1, Zhijian Liu2,3, Chuhang Zou4†, Yue Wang1,3†
1USC Physical Superintelligence (PSI) Lab 2University of California, San Diego 3NVIDIA 4Meta Reality Labs
† Joint corresponding authors
Speed-accuracy overview of Fast SAM 3D Body. Top left: Qualitative results on in-the-wild images show our framework preserves high-fidelity reconstruction. Top right: Our method achieves up to a 10.25x end-to-end speedup over SAM 3D Body and replaces the iterative MHR-to-SMPL bottleneck with a 10,000x faster neural mapping. Bottom: Our system enables real-time humanoid robot control from a single RGB stream at ~65 ms per frame on an NVIDIA RTX 5090.
SAM 3D Body (3DB) achieves state-of-the-art accuracy in monocular 3D human mesh recovery, yet its inference latency of several seconds per image precludes real-time application. We present Fast SAM 3D Body, a training-free acceleration framework that reformulates the 3DB inference pathway to achieve interactive rates. By decoupling serial spatial dependencies and applying architecture-aware pruning, we enable parallelized multi-crop feature extraction and streamlined transformer decoding. Moreover, to extract the joint-level kinematics (SMPL) compatible with existing humanoid control and policy learning frameworks, we replace the iterative mesh fitting with a direct feedforward mapping, accelerating this specific conversion by over 10,000x. Overall, our framework delivers up to a 10.9x end-to-end speedup while maintaining on-par reconstruction fidelity, even surpassing 3DB on benchmarks such as LSPET. We demonstrate its utility by deploying Fast SAM 3D Body in a vision-only teleoperation system that enables real-time humanoid control and the direct collection of manipulation policies from a single RGB stream.
Qualitative comparison. The original SAM 3D Body (left) and our Fast variant (right) yield visually comparable mesh reconstructions across diverse poses and multi-person scenes on 3DPW and EMDB.
Please refer to SAM 3D Body for environment setup, or use our setup script:
bash setup_env.sh
conda activate fast_sam_3d_bodycheckpoints/
├── sam-3d-body-dinov3/ # Auto-downloaded from HuggingFace on first run
│ ├── model.ckpt
│ └── assets/
│ └── mhr_model.pt
├── yolo/ # Place YOLO-Pose weights here
│ ├── yolo11m-pose.pt
│ └── yolo11m-pose.engine # Generated by convert_yolo_pose_trt.py (optional)
└── moge_trt/ # Generated by build_tensorrt.sh (optional)
└── moge_dinov2_encoder_fp16.engine
# Optimized (torch.compile + TensorRT)
bash run_demo.sh# Convert all models (YOLO-Pose + MoGe encoder + DINOv3 backbone)
bash build_tensorrt.sh
# Or convert individually
python convert_yolo_pose_trt.py --model yolo11m-pose.pt --imgsz 640 --half
python convert_moge_encoder_trt.py --all
python convert_backbone_tensorrt.py --allAll generated engines are stored under ./checkpoints/.
This fork contains the live video-to-robot interface for X-Morph. The
morph repository owns retargeting,
physics correction, and causal-student training; this repository owns Fast SAM
3D Body inference, optional GMR conversion, target-robot visualization, and ZMQ
reference publishing. The complementary
kimodo_morph repository provides
the text-conditioned interface.
Set the local checkout paths once:
export MORPH_ROOT=/path/to/morph
export GMR_ROOT=/path/to/GMR
export MORPH_STUDENT_CKPT=$MORPH_ROOT/runs/student_rt_g1_go2/best.ptIf morph and GMR are sibling directories of this repo, the scripts will
find them automatically. Otherwise pass --morph-root, --gmr-root, and
--student-ckpt explicitly.
Trained X-Morph student and downstream tracking-policy checkpoints are released
separately and are not committed to this repository.
Example GMR-to-RT-student live command:
python scripts/video_to_go2_student_rt_live.py \
--video 0 \
--morph-root "$MORPH_ROOT" \
--gmr-root "$GMR_ROOT" \
--processed-dir "$MORPH_ROOT/data/processed/loco_g1_go2" \
--task-family locomotion \
--pair-id g1_to_go2 \
--student-ckpt "$MORPH_STUDENT_CKPT" \
--student-device cuda:0 \
--root-motion-mode student \
--heading-mode integrate \
--publish-zmq tcp://127.0.0.1:5555 \
--publish-quat-convention wxyz \
--publish-ref-offsets 0,1For headless publishing, add:
--no-viewerFor direct SMPL-student inference, set:
--student-input-mode smplThe optional dual-mode Yuna config at
configs/yuna_dual_live.example.yaml uses ${MORPH_ROOT} placeholders and is
expanded by the live script at runtime.
For instructions on running the publisher, see docs/realworld_deployment.md.
We demonstrate a real-time, vision-only teleoperation system for the Unitree G1 humanoid robot using a single RGB camera, operating at ~65 ms end-to-end latency on an NVIDIA RTX 5090.
Humanoid teleoperation. The system tracks diverse whole-body motions including upper-body gestures (a), body rotations (b-e), walking (f), wide stance (g), single-leg standing (h), squatting (i), and kneeling (j).
Humanoid policy rollout. The robot grasps a box on the table with both hands, squats down, and steps to the right. Achieving 80% task success rate with 40 demonstrations collected via our system.
Single-View vs Multi-View. Multi-view fusion resolves depth ambiguities inherent in single-view reconstruction, producing more accurate SMPL body estimates.
@article{yang2026fastsam3dbody,
title={Fast SAM 3D Body: Accelerating SAM 3D Body for Real-Time Full-Body Human Mesh Recovery},
author={Yang, Timing and He, Sicheng and Jing, Hongyi and Yang, Jiawei and Liu, Zhijian and Zou, Chuhang and Wang, Yue},
journal={arXiv preprint arXiv:2603.15603},
year={2026}
}This project builds upon SAM 3D Body (3DB) and Multi-HMR (MHR). We thank the original authors for releasing their models and codebases, which served as the foundation for our acceleration framework.




