TCP4D reconstructs scenes, tracks TCP poses and gripper states, predicts future TCP trajectories, and supports cross-embodiment data augmentation.
TCP4D shares a 4D scene representation between its Tracker and Generator. The tracker recovers metric TCP poses, gripper states, scene geometry and camera parameters from RGB videos and initial 2D TCP queries. The generator samples features along the recovered tracks and combines them with current robot observations and language to generate future TCP trajectories. The paper also introduces a cross-embodiment augmentation pipeline that retargets recovered motion and synthesizes target-robot observations.
Run commands from the repository root. The tested stack is Python 3.11, PyTorch 2.9.1 + CUDA 12.8, torchvision 0.24.1, Transformers 4.55.4 and Accelerate 1.10.1.
git clone https://github.com/Livioni/TCP4D.git
cd TCP4D
conda create -n tcp4d python=3.11 -y
conda activate tcp4d
# Install a matching PyTorch / torchvision pair first.
python -m pip install torch==2.9.1 torchvision==0.24.1 \
--index-url https://download.pytorch.org/whl/cu128
python -m pip install -e '.[visualization]'Released assets are listed in the Hugging Face model collection and dataset and benchmark collection.
| Checkpoint | Usage | Local directory |
|---|---|---|
| TCP4D-stage1-DROID | Single-arm TCP4D tracking trained on DROID | checkpoints/TCP4D-stage1-DROID/ |
| TCP4D-stage1-RoboTwin | Dual-arm TCP4D tracking trained on RoboTwin; Stage2 initialization | checkpoints/TCP4D-stage1-RoboTwin/ |
| TCP4D-stage2-RoboTwin | Dual-arm action generation with a 32-step prediction horizon | checkpoints/TCP4D-stage2-RoboTwin/ |
| Dataset | Published layout | Local directory |
|---|---|---|
| DROID Training Set | 33,966 episodes in 1,018 TAR shards, with episode and shard indexes | datasets/droid/ after extraction |
| AUTOLAB-UR5 Dataset | TCP4D-AUTOLAB-UR5.part0001.tar.gz |
datasets/TCP4D-AUTOLAB-UR5/ after extraction |
| TCP4D Benchmark | droid_eval_sets/ and robolab_eval_sets/ |
datasets/TCP4D-Benchmark/ |
Select the TCP in the interface, then run inference:
python infer_stage1.py --checkpoint /path/to/checkpoint --input /path/to/episode --interactive
# For example:
python infer_stage1.py --checkpoint checkpoints/TCP4D-stage1-DROID \
--input assets/samples/droid/AUTOLab__Fri_Jul_14_16:44:09_2023 \
--interactive
python infer_stage1.py --checkpoint checkpoints/TCP4D-stage1-RoboTwin \
--input assets/samples/robotwin/adjust_bottle/episode_0000000 \
--interactiveEach inference predicts 32 action steps. By default, the policy replans every
20 steps using the latest observations (--window-stride 20).
Select the TCP in the interface, then run inference:
python infer_stage2.py --checkpoint /path/to/checkpoint \
--input /path/to/episode \
--interactive
# For example:
python infer_stage2.py --checkpoint checkpoints/TCP4D-stage2-RoboTwin \
--input assets/samples/robotwin/adjust_bottle/episode_0000000 \
--interactiveAfter inference, the interactive 3D window viewer appears below the controls on
the same page. For remote access, forward only the UI port (default 7861);
viewer assets and WebSocket connections use that same port.
1. Prepare the data. Extract the DROID training set to datasets/droid/,
with one directory per episode.
datasets/droid_episodes/
└── <episode_name>/
├── metadata.json # frame_count、 language_instruction
├── images/<camera_serial>/000000.png
├── depths/metadata.json
├── depths/<camera_serial>/000000.png
├── intrinsic/<camera_serial>.npy # [3,3]
├── extrinsic/<camera_serial>.npy # [4,4] world-to-camera
└── TCP/<camera_serial>/
├── metadata.json
└── state.npy # [T,7]Build a local SQLite index and episode-level train/validation lists (seed 42, approximately 90%/10%); existing TXT lists are reused:
conda activate tcp4d
python - <<'PYTHON'
from tcp4d.datasets.droid_index import ensure_index, write_splits
index = ensure_index("datasets/droid", "cache/tcp4d_droid.sqlite")
write_splits("datasets/droid", index)
PYTHONThe lists are saved under datasets/splits/droid/. To use the released split,
copy train_set.txt and val_set.txt from assets/samples/splits/DROID/ there
before preparing the index. Build the index on each machine; if the dataset path
or indexing thresholds change, pass rebuild=True to ensure_index.
2. Configure and train. Review configs/stage1_droid.py, especially the data
paths, train_batch_images, num_workers and max_train_steps. The defaults use
BF16, 2–18 frames per clip and 500,000 optimizer steps. New runs initialize from
Luo-Yihang/4RC; use --pretrained-model /path/to/checkpoint for local weights.
accelerate launch train_stage1.py \
--config configs/stage1_droid.pyThe DROID augmentation workbench provides a local Example Web UI
for tracking, robot retargeting, URDF/SAM3 mask refinement, background
inpainting and paired dataset export. Its README includes installation from
scratch in the shared tcp4d environment.
We provide a WebSocket server template for deploying the TCP4D Generator (Stage2) with the released RoboTwin checkpoint. It receives RGB observation history, initial TCP queries, current wrist images, robot state and a language instruction, and returns 32-step dual-arm TCP trajectories and gripper states. See deploy/README.md for setup, the request/response protocol and a Python client example.
- Stage1,2 Checkpoint, DROID training dataset, Benchmark.
- Training and inference scripts.
- Example Data augumentation pipeline.
- Paper, project page, author list and citation.
@misc{peng2026tcp4dunifiedtoolcenterpointtracking,
title = {TCP4D: A Unified Tool-Center-Point Tracking and Generation Model for Robot Manipulation},
author = {Haosong Peng and Xiangyang Liu and Zhengshen Zhang and Yunfeng Fan and Tianyu Qi and Xin Xia and Yufeng Zhan and Wenchao Xu},
year = {2026},
eprint = {2611.23143},
archivePrefix = {arXiv},
primaryClass = {cs.RO},
url = {https://arxiv.org/abs/2611.23143},
}The TCP4D codebase and released model weights are licensed under CC BY 4.0. Third-party code, model checkpoints and datasets retain their original licenses. See NOTICE for third-party attribution and licensing details.
The TCP4D model is based on 4RC. We are especially grateful to Yihang Luo and Zhiwei Wu for their insightful discussions, and to Xingxin He and WONG Yuk To for their help.


