We provide a WebSocket server template for deploying the TCP4D Generator (Stage2) with the released TCP4D-stage2-RoboTwin checkpoint. It takes third-person RGB history, initial 2D TCP queries, current wrist RGB, current robot state and a language instruction, and returns 32 future steps of dual-arm TCP poses and gripper states. The released checkpoint uses 4 history frames and 4 sampling steps; history length and prediction horizon are read from the checkpoint.
The server is stateless: the client supplies the complete observation window on every request and updates the first-frame TCP queries whenever that window moves. Robot observation capture, camera-to-base transforms, inverse kinematics and action execution belong in the client integration.
Follow the installation instructions in the repository README
first. Use the same tcp4d environment: Python 3.11, PyTorch 2.9.1 + CUDA 12.8,
torchvision 0.24.1, Transformers 4.55.4 and Accelerate 1.10.1. Download the Stage2
checkpoint to checkpoints/TCP4D-stage2-RoboTwin/.
Run all commands from the repository root:
conda activate tcp4d
python -m pip install -r deploy/requirements.txt
python -m deploy.server \
--checkpoint checkpoints/TCP4D-stage2-RoboTwin \
--device cuda:0 --host 0.0.0.0 --port 8000python deploy/server.py is also supported. The default checkpoint is
checkpoints/TCP4D-stage2-RoboTwin. --checkpoint accepts either a directory
containing the checkpoint files or an experiment directory containing
final_checkpoint/.
| Option | Default | Meaning |
|---|---|---|
--checkpoint |
checkpoints/TCP4D-stage2-RoboTwin |
Stage2 checkpoint directory |
--device |
auto |
CUDA when available, otherwise CPU; also accepts cuda:0 or cpu |
--dtype |
auto |
BF16/FP16 on CUDA, FP32 on CPU; explicit choices are float32, float16, bfloat16 |
--sampling-steps |
Checkpoint value (4 for the released model) | Action sampling steps |
--seed |
42 |
Sampling seed, reset for each request |
--host / --port |
0.0.0.0 / 8000 |
WebSocket bind address |
--t5-model |
Checkpoint configuration, with resource fallback | Optional local T5-base directory or model identifier |
The server loads the model once and starts listening only after it is ready. A
complete checkpoint must include config.json, model.safetensors,
action_norm.json and state_norm.json; normalization statistics are checked
against the saved model buffers. The wrist DINOv2 configuration and weights are
restored from the checkpoint. T5 resources are loaded separately: if the saved
absolute T5 path is unavailable, the loader checks for t5-base/ inside or beside
the checkpoint, then falls back to google-t5/t5-base through Hugging Face. For
offline deployment, provide an available local directory:
python -m deploy.server \
--checkpoint checkpoints/TCP4D-stage2-RoboTwin \
--device cuda:0 --t5-model /path/to/local/t5-baseOne process loads one policy and serializes inference across connections. Inference runs in a worker thread, allowing other connections to send health checks while it is running. Shutdown waits for an active inference to finish. Each request uses an independently seeded sampling generator; results do not depend on earlier requests' random state.
All image sizes below are width × height. Send original RGB images; the server applies the preprocessing used during training.
| Input | Client representation | Server / model preprocessing |
|---|---|---|
| Third-person history | 320×240 RGB, uint8 [0,255] |
Reflect-pad 1 pixel left/right and 6 top/bottom to 322×252; normalize to [-1,1] |
| Current left/right wrist images | 320×240 RGB, uint8 [0,255] |
Convert to [0,1]; the wrist encoder uses bicubic resize to 256×192, reflect-padding to 266×196 and ImageNet normalization |
| Camera intrinsics and TCP queries | Pixel coordinates in the original, unpadded 320×240 image | Shift the principal point and queries by (1,6) |
| Current robot state | 12 joint angles in radians and two grippers in [0,1] |
Apply the checkpoint's state normalization inside the model |
Images must be RGB HWC NumPy arrays. Convert OpenCV BGR images to RGB on the client. Encoded JPEG bytes, floating-point images and other resolutions are not accepted. Do not pre-pad, pre-normalize or resize wrist images to 256×192.
Supply 1–4 third-person frames, oldest first. Short histories repeat their earliest
frame: [0] → [0,0,0,0], [0,1] → [0,0,0,1], [0,1,2] → [0,0,1,2].
More than 4 frames are rejected for this checkpoint. Wrist images and current
robot state must correspond to the latest history frame. TCP queries must locate
the left and right TCP in the first supplied frame.
The API uses frame indices and does not accept timestamps. Match the observation
cadence to the data used for your checkpoint. The bundled RoboTwin sample records
frequency_hz=15 in metadata.json; its next 32 steps refer to subsequent sampled
frames. This does not imply a 15 Hz server inference rate. Replanning and how many
predicted steps to execute are client decisions; the offline inference script's
20-step window stride is not applied by this server.
Connect to ws://SERVER_IP:8000. The first server message has type="metadata"
and describes the protocol version, history length, prediction horizon, image
sizes, coordinate frames, arm order and gripper convention. Read and validate it
before sending requests.
Encode and decode every message with deploy.protocol.pack_message and
unpack_message, and send binary WebSocket messages. Clients need only
deploy/protocol.py, NumPy, MessagePack and websockets; model dependencies and
PyTorch are not required. This is a custom protocol, distinct from OpenPI and
msgpack-numpy.
The message limit is 16 MiB. Use max_size=MAX_MESSAGE_BYTES and
compression=None on the client; a full observation exceeds websockets' default
1 MiB limit. Responses follow request order within a connection. Wait for a
prediction before sending the next prediction request.
| Field | Type / shape | Meaning |
|---|---|---|
type |
"predict" |
Request type |
request_id |
String or integer | Returned unchanged in the response |
images |
uint8 [T,240,320,3] |
Third-person history, 1≤T≤4, oldest first |
tcp_query_points |
Numeric [2,2] |
Left/right (u,v) in the first unpadded frame; 0≤u<320, 0≤v<240 |
intrinsics |
Numeric [3,3] |
OpenCV camera matrix K for the 320×240 view; shared across the history window |
wrist_images |
uint8 [2,240,320,3] |
Current left/right wrist RGB |
current_robot_state |
Numeric [14] |
Left six joints, left gripper, right six joints, right gripper |
instruction |
Non-empty string | Task instruction |
Small numeric fields may be arrays or lists; images must be uint8 NumPy arrays.
All numeric inputs must be finite. K must be nonsingular, have positive fx,fy
and have last row [0,0,1].
Grippers occupy state indices 6 and 13: 0=closed, 1=open. The bundled
RoboTwin robot_state.npy already uses [0,1]; send it directly. For another
robot, map measured gripper openings into this convention on the client according
to its calibration. The server does not infer units or apply the old task-specific
0.10 m conversion. Do not normalize the joint angles yourself.
No ground-truth history TCP poses, future images, depth or camera extrinsics are required by the server.
Common fields are type="prediction", the original request_id, success=True
and inference_ms. Timing includes preprocessing, model execution and transfer
back to CPU, and excludes queuing and network transmission.
Shapes below use H=32 for the released checkpoint and T for the number of real
input frames, excluding repeated padding frames. Every arm dimension is ordered
left, right.
| Field | Type / shape | Meaning |
|---|---|---|
action_position |
float32 [H,2,3] |
Absolute future TCP positions in meters |
action_rotation |
float32 [H,2,3,3] |
Future TCP rotation matrices |
action_gripper |
int64 [H,2] |
1=open, 0=closed |
action_gripper_score |
float32 [H,2] |
Raw score; >=0 means open, not a probability |
future_step_indices |
int64 [H] |
Steps 1 through H |
history_position |
float32 [T,2,3] |
Recovered TCP positions for real history frames |
history_rotation |
float32 [T,2,3,3] |
Recovered history TCP rotations |
history_confidence |
float32 [T,2] |
Raw TCP-head confidence, not a probability |
history_query_points_px |
float32 [T,2,2] |
Recovered TCP projections in unpadded image pixels |
history_valid |
bool [T,2] |
Validity mask for recovered history entries |
history_padding |
Integer | Number of repeated earliest frames, 4-T |
Future poses use the latest observation's OpenCV camera frame: x right, y down, z forward. Positions are absolute, not deltas. Rotation matrices transform TCP-local vectors into that camera frame. Convert poses to the robot base frame and solve IK in the client, using the same TCP definition as the training data.
History poses use each observation's own camera frame. Invalid history entries
have zero-filled positions, rotations, confidences and projections; always inspect
history_valid. Projections outside the image are not clipped to its border.
For a moving window [f0,f1,f2,f3] → [f1,f2,f3,f4], use the previous result's
history_query_points_px[1] as the new first-frame queries only if both arms in
history_valid[1] are true. Otherwise, locate the TCPs again on the client. Use
recovered history projections for this update; future predicted actions do not
provide the observed first-frame queries. The server does not propagate queries.
Send {"type": "health", "request_id": "ping"} with the same binary codec. The
response is {"type": "health", "request_id": "ping", "success": True, "ready": True}. The server accepts connections only after model loading finishes.
Recoverable errors have this structure:
{
"type": "error", "request_id": "window-0", "success": False,
"error": {"code": "invalid_request", "message": "..."},
}Codes are invalid_request, prediction_failed and inference_error. Error
responses contain no action_* fields; check success before consuming actions.
The connection remains usable after these errors. Messages exceeding 16 MiB cause
the WebSocket layer to close the connection. If the request identifier cannot be
decoded, the error response uses request_id=None.
The root message is a MessagePack map with string keys. NumPy arrays use this map:
{"__ndarray__": True, "dtype": "<f4", "shape": [2, 2], "data": raw_bytes}dtype follows NumPy dtype.str, e.g. |u1, <f4, <i8, |b1. data is a
binary payload in C order. Only bool, integer and real floating-point arrays are
supported; byte length must match shape and dtype. Object arrays and pickle are
not supported.
Run this example from the repository root after starting the server. It uses the same RoboTwin sample as the root README and obtains initial queries by projecting the sample's ground-truth TCP positions. A live robot client can initialize queries with manual selection or calibrated TCP localization instead. The example needs Pillow to read the sample PNG files; it is included in the project environment.
import asyncio
import json
from pathlib import Path
import numpy as np
from PIL import Image
from websockets.asyncio.client import connect
from deploy.protocol import MAX_MESSAGE_BYTES, PROTOCOL_VERSION, pack_message, unpack_message
episode = Path("assets/samples/robotwin/adjust_bottle/episode_0000000")
frame_indices = [0, 1, 2, 3]
metadata = json.loads((episode / "metadata.json").read_text())
def rgb(view, frame):
with Image.open(episode / "images" / view / f"{frame:06d}.png") as image:
return np.asarray(image.convert("RGB")).copy()
K = np.load(episode / "intrinsics/third_views.npy").astype(np.float32)
xyz = np.stack([
np.load(episode / "TCP_third" / f"{arm}_state.npy")[frame_indices[0], :3]
for arm in ("left", "right")
])
pixels = xyz @ K.T
queries = pixels[:, :2] / pixels[:, 2:3]
state = np.load(episode / "robot_state.npy")[frame_indices[-1]].astype(np.float32)
request = {
"type": "predict", "request_id": "window-0",
"images": np.stack([rgb("third_views", i) for i in frame_indices]),
"tcp_query_points": queries, "intrinsics": K,
"wrist_images": np.stack([
rgb(side, frame_indices[-1]) for side in ("left_view", "right_view")
]),
"current_robot_state": state,
"instruction": metadata["instructions"][0],
}
async def predict():
async with connect("ws://127.0.0.1:8000", max_size=MAX_MESSAGE_BYTES,
compression=None, proxy=None) as ws:
server_metadata = unpack_message(await ws.recv())
assert server_metadata["type"] == "metadata"
assert server_metadata["protocol_version"] == PROTOCOL_VERSION
assert server_metadata["history_frames"] == 4
assert server_metadata["prediction_horizon"] == 32
await ws.send(pack_message({"type": "health", "request_id": "ping"}))
health = unpack_message(await ws.recv())
assert health["success"] and health["ready"]
await ws.send(pack_message(request))
result = unpack_message(await ws.recv())
if not result["success"]:
raise RuntimeError(result["error"])
print(result["action_position"].shape) # (32, 2, 3)
print(result["action_rotation"].shape) # (32, 2, 3, 3)
print(result["action_gripper"].shape) # (32, 2), 0/1
print(result["history_valid"])
asyncio.run(predict())This checks metadata, readiness and one inference request without executing any
robot actions. For another machine, change the WebSocket URL to the server's
address. A client running outside the model environment can install its transport
dependencies with python -m pip install -r deploy/requirements.txt and use a copy
of deploy/protocol.py; Pillow is needed only when reading images as in this
example.