Data format

This page describes what a trial leaves on disk, and where TANDEM writes its logs. Paths are the Linux defaults (moving them). For the terms used here, see Concepts.

Reading an episode

import json
from pathlib import Path

import numpy as np

ep = Path("~/tandem-data/trajectories/my-task/success/2026-09-25_14-03-12").expanduser()
meta = json.loads((ep / "_meta.json").read_text())
state = np.load(ep / "robot_state.npz")
print(meta["instruction"], meta["n_frames"], "frames at", meta["fps"], "fps")
print(state["joint_position"].shape)          # (F, 7)

# Who did each phase, and did its camera checks pass?
hitl = json.loads((ep / "hitl.json").read_text())
for phase in hitl["phases"]:
    verdicts = [v["satisfied"] for v in hitl["verifications"] if v.get("phase") == phase["index"]]
    print(phase["index"], phase["executor"], phase["description"], verdicts)

# The frame shown 4.2 s into the videos (merged episodes have video_time).
frame = int(np.searchsorted(state["video_time"], 4.2))

Episode layout

Each trial is one directory, under success/ or failure/:

~/tandem-data/trajectories/<profile>/<status>/<YYYY-MM-DD_HH-MM-SS>/
├── external_cam.mp4  external_cam_2.mp4  hand_cam.mp4   exterior 1, exterior 2 (optional), wrist
├── robot_state.npz        per-frame arrays
├── _meta.json             lineage, timing, and which leg and phase each frame belongs to
├── hitl.json              the phase record
├── vlm/                   model queries
├── tiptop_plan.json, …    the primary leg's other files
└── segments/NN_<source>_<leg dir>/   the raw legs, in order

The videos and arrays join every leg of the trial in order. The episode takes the primary leg's directory name, its _meta.json, and its other files except *.log. For TiPToP those are tiptop_plan.json, metadata.json, rgb.png and perception/.

A trial with a single leg is not merged: the leg is the episode. The format is the same as hitl-tamp-vla's. To review and export episodes, see Reviewing and exporting.

robot_state.npz

The file has one row per frame. The per-leg arrays are listed in the recording contract. An episode may also have:

To match frames to video in a merged episode, use video_time. Don't use frame_time, which keeps the gaps between legs. In a single leg, frame i sits (frame_time[i] − record_start) / (record_stop − record_start) of the way through each video.

The arrays the export uses are listed in Exporting.

_meta.json

{
  "trajectory_id": "3f9c0a1b7d2e4c55",
  "planner": "tiptop",
  "instruction": "place the bread inside the box",
  "n_frames": 912, "fps": 15,
  "cameras": {"exterior_image_1_left": "external_cam.mp4", "wrist_image_left": "hand_cam.mp4"},
  "segments": [
    {"source": "tamp",   "timestamp": "2026-09-25_14-03-12", "phase_index": 0, "video_start": 0.0,  "video_stop": 31.4},
    {"source": "teleop", "timestamp": "2026-09-25_14-03-50", "phase_index": 1, "video_start": 31.4, "video_stop": 60.8}
  ]
}

A merged episode's _meta.json is the primary leg's, with the keys below set. A single-leg episode keeps its own (per-leg keys).

Each segments[k] has:

• source (tamp, teleop or policy), timestamp (the leg's directory name), n_frames and n_video_frames.
• video_start and video_stop (seconds into the joined videos), and record_start and record_stop.
• If the leg stated them: phase_index, phase_description, n_phases and config_id (teleop: teleop/teleop). phase_index and phase_description never appear at the top level.
• After a replan: plan_generation (0 is the first plan).

A conjoined robot leg gets the first phase it covers. Its phase_description joins the covered phases' descriptions with ;.

hitl.json

This file is written when phase planning is on and the trial has a plan. It goes into the primary leg, then beside the merged episode. A trial stopped at the label prompt keeps it in its primary leg in eval/.

A trimmed example, for a trial where the robot moves the bread and a person closes the lid:

{
  "instruction": "put the bread in the box and close it",
  "trajectory_id": "3f9c0a1b7d2e4c55",
  "planner": "tiptop",
  "human_executor": "teleop",
  "outcome": "success", "excluded": false, "failure_stage": null, "filed_under": "success",
  "specification": {
    "invented_predicates": [{"name": "IsClosed", "types": ["surface"], "instructions": "Is the lid of {0} closed?"}],
    "unrepresented": [],
    "coverage": [{"clause": "put the bread in the box", "phase": 0}, {"clause": "close it", "phase": 1}]
  },
  "phases": [
    {"index": 0, "executor": "robot", "description": "Put the bread in the box",
     "atoms": ["On(bread, box)"], "goal": [{"predicate": "on", "args": ["bread", "box"]}],
     "planning_seconds": 41.2},
    {"index": 1, "executor": "human", "description": "Close the box",
     "atoms": ["IsClosed(box)"], "instructions": "Close the box's lid.",
     "carried_out": [{"attempt": 1, "carried_out_by": "teleop", "status": "done", "leg_recorded": true, "n_frames": 440}]}
  ],
  "verifications": [
    {"atom": "IsClosed(box)", "statement": "Is the lid of box closed?", "holds": true, "expected": true,
     "satisfied": true, "role": "effect", "reason": "The lid is down.", "phase": 1}
  ],
  "phase_index": 2
}

Top-level keys

provenance has phases_and_their_order, phase_sub_goals, invented_predicates and human_instructions (the model), robot_phases (the planner), who_does_what (TANDEM), and human_steps (the intended executor; each phase's carried_out says who actually did it). human_operators and robot_operators are {by, signatures}, with signatures written Name(param: type).

specification

The proposal's reading of the instruction.

checks

phases[k]

Every phase has index, executor, description, atoms and planned_by. The rest depends on the kind:

verifications[k]

One entry per camera verdict, failing ones included: {atom, statement, holds, expected, satisfied, role, reason, phase}. role is effect, effect (deleted) or precondition.

Read satisfied. holds is what the model saw and expected is what the plan wanted, so satisfied == (holds == expected). For a delete effect, holds: true is the failure.

Only the attempt that settled a phase is kept. For an excluded trial, those are the verdicts it was excluded on. Earlier attempts are in the human_phase_verified events.

vlm/

vlm/
├── 001_task-plan_input.png
├── 001_task-plan_output.png
├── 002_classify-LidClosed-red_bin_input.png
├── 002_classify-LidClosed-red_bin_output.png
└── index.jsonl

This directory holds every model query of the trial. It is written when phase planning is on and hitl.save_vlm_io is true (the default): first into the session directory, then copied into the episode when it is filed. Each query leaves:

• NNN_<label>_input.png: the image sent.
• NNN_<label>_output.png: that image above the answer. It is marked REJECTED if TANDEM rejected the answer, or CACHED if it was replayed from hitl.cache_path.
• An index.jsonl line with seq, label, attempt, model, input_image, output_image, rejected (why, or null), cached, prompt and response.

The labels are task plan (the proposal), classify <atom> (a camera check), and detect objects (tandem plan without --object). A retry adds _attempt<N>. NNN continues from the directory's highest number, so nothing is overwritten.

The events file

{"event": "rollout_saved", "ts": 1790270037.40, "dir": "…/eval/2026-09-25_14-03-12", "n_frames": 12}
{"event": "human_phase_verified", "ts": 1790270101.12, "phase_index": 1, "attempt": 1, "ok": true, "verdicts": […]}
{"event": "trial_outcome", "ts": 1790270110.87, "outcome": "excluded", "failure_stage": "verification", "reason": "…"}

A session appends one JSON line per event to ~/.local/state/tandem/sessions/<profile>/<session id>/events.jsonl. A sidecar planner writes to it too. The web UI streams each event on /api/sessions/{id}/stream, adding "type": "event" and "at": <ts>.

From the phase loop

From the session

Below, trial summary means trajectory_id, outcome, excluded, filed_under, failure_stage, reason and labeled.

Session summary

A session's live state. The web UI gets it from GET /api/sessions/{id}, and tandem collect saves it in session-<id>.json. The common fields are below; there are more.

Logs and session files

A trial that failed at invention usually has no legs. Only its events file and its session vlm/ and perception/ directories remain.