Configuration

This page lists every setting TANDEM reads. For setup, see Installation; for commands, the Command reference.

Profiles

A profile is one task in one YAML file:

# ~/tandem-data/profiles/my-task.yml
version: 3
description: 'Cup onto plate'
task:
  prompt: put the cup on the plate       # the episode's language label
  target_episodes: 30
hitl:                                    # phase planning (see below)
  enabled: true
  on_verification_failure: label
planner:
  backend: tiptop
  options:
    tamp:                                # TiPToP's task settings
      num_particles: 512
recording:
  enabled: true
export:
  hf_repo: my-org/cup-on-plate

Keys you leave out take their defaults. Usually you start from the paper's settings or a copy:

tandem profile create my-task --prompt "put the cup on the plate"   # the paper's settings, your task
tandem profile create my-box --from store-bread-in-closed-box      # a copy of any profile
tandem profile edit my-task                                        # opens $EDITOR; validated on save

Robot, camera and calibration settings aren't in a profile; they're in the rig. Every key is documented in src/tandem/resources/profile_template.yml. An unknown key fails at load, and a misspelled tamp key gets a suggestion. Profiles and their trajectories live under ~/tandem-data/:

~/tandem-data/
├── profiles/
│   ├── my-task.yml             one file per profile
│   └── .planner-options/       a previous planner's options (tandem planners use)
└── trajectories/
    └── my-task/
        ├── eval/               not yet labeled
        ├── success/
        └── failure/            failures, and settled trials (no label)

Profile keys

The paper's five

tandem init adds the paper's five tasks, each with the settings the paper collected it with, plus the three switches.

• They're yours to edit. tandem init never overwrites one, and restores one you deleted.
• profile create --prompt gives a new profile what the five share: phase planning, the TAMP and DATAFARM settings, and the three switches.
• Safety: they set tamp.time_dilation_factor_literal: 1.0, so planned motions ignore the rig's time_dilation_factor (homing and capture moves don't). Keep a hand on the E-stop.

Older profiles

Before version 3, a profile was a directory (profiles/<name>/profile.yml) with its own robot, cameras and calibration.json. They don't show up until you move them:

tandem profile migrate   # tandem init also does this

Each profile's task goes to profiles/<name>.yml and its trajectories to trajectories/<name>/. The robot and cameras seed the rig (from the active profile) if rig.yml doesn't exist yet. Extrinsics are added to the rig's calibration.json only for cameras without an entry.

• Old directories aren't deleted. Each is archived in profiles/.migrated/<name>/ with a migration.json.
• If the rig can't be set up from them, no profile is moved. A profile that fails to move is left as it was, and running the command again finishes a partial move.
• tandem rig set and rig edit wait until old profiles are moved. Older TANDEM can't read version 3.
• Version-1 profiles keep hitl.on_robot_phase_failure: teleop, the old default. Set abort unless you chose teleop.

Phase planning (hitl)

Phase planning is on in the paper's profiles and in every new profile, and off in a profile with no hitl: block. The defaults match the paper. Some common changes:

# Label failed human steps yourself, instead of excluding the trial from the dataset
hitl:
  enabled: true
  on_verification_failure: label

# When the planner can't plan a robot step, let a person do it (checked like a human step)
hitl:
  enabled: true
  on_robot_phase_failure: teleop    # or replan: ask the model for a new plan

# Turn phase planning off: the planner does the whole task, with no human steps or checks
hitl:
  enabled: false

Each check costs one model call per checkable atom, with the arm parked. Checkable atoms are invented predicates plus the planner's checkable_predicates (TiPToP: On; capabilities).

Planning

Checks

Failures

Human executor

The rig

The rig holds this workstation's robot, camera and calibration settings. All profiles use it. tandem init writes it to ~/.config/tandem/rig.yml:

version: 1
robot:
  type: fr3_robotiq
  host: 172.16.0.2          # the NUC
cameras:
  perception: external
  hand:     {serial: '14846828'}
  external: {serial: '32439448'}
calibration: calibration.json
planners: {}                # a planner's machine settings, where they differ from its defaults

To view or change it:

tandem rig show                                   # also --json; the web UI's Settings page shows it too
tandem rig set robot.host NUC_ADDRESS             # `null` removes a setting
tandem rig set planners.tiptop.perception.m2t2.url http://HOST:8123
tandem rig edit                                   # opens $EDITOR; validated on save
tandem rig path --calibration                     # where the extrinsics are

Calibration file

calibration.json holds one pose per camera, keyed by serial:

{
  "14846828": {"pose": [0.0266, 0.0705, -0.1392, -0.4509, -0.0045, -1.5620]},
  "32439448": {"pose": [0.1532, -0.5827, 0.4410, -2.0843, 0.0126, 0.1964]}
}

• pose is [x, y, z, roll, pitch, yaw], in meters and radians (xyz Euler angles).
• The wrist camera's pose is relative to the end effector (ee_from_cam).
• An external camera's pose is relative to the robot's base (world_from_cam).
• Other keys in an entry, such as timestamp, are ignored.

A session won't start if a camera has no entry. To get the poses, see Cameras and calibration.

Related settings outside the rig

• TiPToP's scripts (calibrate-wrist-cam, viz-calibration, cutamp-demo, …) run through tandem runtime run or shell use your rig settings, through $TIPTOP_CONFIG and $TIPTOP_CALIBRATION. --raw uses tiptop's stock config instead.
• Teleop uses robot.host too; TANDEM passes it to the teleop driver.
• Gripper mask (perception: hand only): make yours with tandem runtime run compute-gripper-mask (or paint-gripper-mask). It writes the runtime's tiptop/tiptop/config/assets/gripper_mask.png.

Planner settings

A profile names its planner and that planner's settings for the task:

planner:
  backend: tiptop     # which planner
  options:            # its task settings, checked by the planner at load
    tamp:
      num_particles: 512

Task settings go in the profile's planner.options. Machine settings (a server's address, a robot's ports) go in the rig under planners.<name>. If you put a key in the wrong file, TANDEM refuses it and says where it goes. tandem planners info NAME lists both (how planners declare them).

Did my setting apply? tandem profile show NAME --planner prints exactly what the planner receives.

Switching planners

tandem planners use myplanner                  # the active profile now plans with it
tandem planners use myplanner -o KEY=VALUE     # also set an option it requires (repeatable)
tandem planners default myplanner              # the planner new profiles get

use sets planner.backend and stashes the old planner's options in profiles/.planner-options/<profile>.<planner>.yml, restoring them when you switch back. It warns if the runtime isn't installed, and it repairs a profile naming a planner this machine lacks.

TiPToP options

Machine settings live in rig.yml under planners.tiptop:

tandem rig set planners.tiptop.robot.time_dilation_factor 0.3          # arm at 30% speed
tandem rig set planners.tiptop.perception.m2t2.url http://HOST:8123
tandem rig set planners.tiptop.perception.depth_smoothing_frames 1

The robot's address and type come from robot.host and robot.type.

robot:

perception:

Task settings live in the profile's planner.options.tamp:

planner:
  backend: tiptop
  options:
    tamp:
      num_particles: 512
      voxel_downsample_size: 0.005   # finer than the rig's perception setting
      placement_support: true        # see Surface-fitted placement

• These are cuTAMP and cuRobo overrides with tiptop's key names, so a cfg/tamp/*.yml's tamp_overrides paste in as is. Accepted keys: src/tandem/planners/tiptop/tamp_keys.py.
• For contact_threshold_m and voxel_downsample_size, the tamp value overrides the rig's.
• tandem doctor warns about a key that needs another (retime_ops without retime_trajectory).
• Relative checkpoint paths (encoder_path, posture_ref) resolve beside the profile, then in the runtime, where the install puts the DATAFARM checkpoints (vae/checkpoints/vae_full_v2.pt, rnd/checkpoints/rnd_droid.pt).
• Stroke re-timing is the trajectory encoder's: retime_trajectory with encoder_path and the other retime_* keys. A profile written with the older names (vae_path, vae_manifold_weight, blend_*) still loads, read under the new names; vae_retiming: false and blend_mode: vae are dropped, and a setting of a removed mode (the VAE-owned clock, spline or flow re-timing) is refused with the reason.

Surface-fitted placement

By default, cuTAMP can place an object anywhere in a surface's bounding box, at the top of it, such as on an open box's rim or a plate's edge. placement_support: true places only on observed level patches that fit the footprint:

tamp:
  placement_support: true
  placement_fill_occluded: true    # e.g. a box whose floor the camera can't see

In the paper's profiles, solve-constrained-puzzle sets placement_support, placement_support_required and placement_into_surface. store-bread-in-closed-box sets all seven, with margin 0.005, flatness 0.012 and placement_fill_occluded: true.

The three switches

These tamp: keys are off unless set. The paper's five and every new profile turn them on, because every run in the paper used them.

tamp:
  table_plane_support_vote: true
  disjoint_object_masks: true
  retime_stretch_to_caps: true

TANDEM settings and credentials

TANDEM's own settings live in ~/.config/tandem/config.toml:

tandem config list
tandem config set ui.port 8800
tandem config set data_root /mnt/data/tandem
tandem config edit                 # also: get KEY, path

Credentials

Credentials are stored in credentials.toml next to it, readable only by you:

tandem config set-gemini-key    # for phase planning and TiPToP's perception
tandem config set-hf-token      # for tandem export lerobot --push

• GEMINI_API_KEY or GOOGLE_API_KEY overrides the stored Gemini key.
• The Hugging Face token is looked up in order: stored, HF_TOKEN or HUGGING_FACE_HUB_TOKEN, then ~/.cache/huggingface/token (huggingface-cli login).

Directories

These are the Linux defaults; other systems differ.

The planner runtime

tandem planners install tiptop    # build or update TiPToP's runtime (tandem init runs this)
tandem planners info tiptop       # pinned vs installed commits
tandem planners install tiptop --force   # refetch and rebuild everything

TiPToP's runtime is a pixi environment (torch, cuRobo's CUDA kernels, cuTAMP, tiptop). The first build takes about 25 GB and 5–20 minutes, and asks before installing pixi (--yes accepts). Its pinned sources are below; the TANDEM branches add surface-fitted placement and the three switches.

The install applies one patch (so $TIPTOP_CONFIG and $TIPTOP_CALIBRATION can point TiPToP at the rig) and places the DATAFARM checkpoints. How sources are fetched is in Adding a planner, and where the build log goes is in Logs and session files.

• Updating. A moved pin shows outdated in tandem planners list. Reinstalling replaces only the moved sources.
• ZED. With the ZED SDK installed (/usr/local/zed/get_python_api.py), the install adds pyzed. Without it the install succeeds but ZED cameras won't open, and tandem doctor says so. Install the SDK, then reinstall.

Offline install

If the workstation can't reach GitHub, bundle the sources on a machine that can:

tandem planners bundle tiptop --out /media/usb/planner-sources      # on a machine with network
tandem planners install tiptop --sources /media/usb/planner-sources # on the workstation

• Use the same TANDEM version on both: commits are checked against their pins and files against the digest.
• $TANDEM_PLANNER_SOURCES can replace --sources (one directory per source: tiptop/, cuTAMP/, curobo/). While set, nothing is fetched.
• Only sources are bundled. pixi install still needs conda-forge, PyPI and GitHub (to build SAM-2), and the first warm-up downloads the SAM-2 checkpoint (about 0.9 GB).