TANDEM: Task and Motion Planning with As‑Needed Demonstrations for Efficient Vision‑Language‑Action Model Fine‑tuning

Samrat Sahoo1,*, Liang Ji2,*, Tom Silver2, Yixuan Huang2
1Stanford University · 2Princeton University
*Equal contribution · Paper under review
Paper · coming soon Code · coming soon Appendix
Full-task teleoperation (left) vs. TANDEM (right). With TANDEM, the human only teleoperates the covering stage.

Abstract

Human teleoperators spend substantial time demonstrating behaviors that robots can already perform autonomously, limiting the scalability of data collection for robot foundation models. Task and motion planning (TAMP) can automate many of these behaviors, but a fixed planning domain may not support every stage of a long-horizon manipulation task.

We present TANDEM (Tamp with As-Needed Demonstrations for Efficient Model fine-tuning), a system that combines TAMP with selective human teleoperation to collect demonstrations for tasks beyond the planner's capabilities. Our key idea is to represent human assistance as an on-demand planning capability. Given a language instruction and visual observation, TANDEM uses pretrained vision-language models to extend the planning domain with missing predicates and human-executed magic operators. This allows the planner to interleave autonomous and human-executed stages without task-specific intervention points. After each human stage, TANDEM re-perceives the scene and checks whether the intended effects hold before resuming autonomous planning. To support fine-tuning vision-language-action (VLA) models, TANDEM also uses example pretraining trajectories to align planner-generated motions with the target model's pretraining distribution.

We evaluate TANDEM on five long-horizon manipulation tasks beyond the TAMP domain's capabilities. On a representative long-horizon task, TANDEM collects 2.9× as many demonstrations as full-task teleoperation at the same human intervention time. Fine-tuning a pretrained π0.5-DROID model on 20 TANDEM demonstrations per task increases average task success from 0% to 60% across the five tasks.

Goal: place the bread rolls on the plate and cover them. TAMP can pick and place but cannot cover, with no human effort; teleoperation executes every stage with huge human effort; TANDEM uses TAMP for pick and place and a human only for covering, with low human effort. The collected dataset supervises π0.5 fine-tuning, and an illustrative plot shows policy success rate against human time for TANDEM, teleop, and TAMP.
TAMP executes pick-and-place behaviors, and humans handle behaviors beyond the planner’s capabilities. The plot is illustrative.

Method

A VLM extends the TAMP domain with missing predicates and human-executed magic operators, so one plan can interleave autonomous and teleoperated phases.

TANDEM overview for “Place the bread inside the box”: (1) task and initial observation; (2) the existing TAMP domain has On, Grasped, Pick and Place, and LiftOpen(box) is detected as a missing capability; (3) a VLM invents the predicate IsOpen(box) and a magic operator LiftOpen(box) executed by a human; (4) the task plan is TAMP Pick(bread), TAMP Place(bread, plate), Human LiftOpen(box), TAMP Pick(bread), TAMP Place(bread, box); (5) execution, where after the human opens the box a VLM verifies IsOpen(box) from a new observation before TAMP resumes.
Overview of TANDEM. TAMP executes the supported stages and a human teleoperates the rest. After each human phase, TANDEM verifies the intended effect before TAMP resumes.
1 · Predicates

Predicate Invention

A VLM invents predicates the domain is missing, each with a natural-language definition, e.g. IsOpen(x): “x is open”. The definition also lets a VLM check the predicate from an image.

2 · Operators

Magic Operators

Operators executed by a human through teleoperation. They have the same symbolic structure as TAMP operators, e.g. LiftOpen(box) with effect IsOpen(box), so the planner can use them.

3 · Execution

Task Plan Generation and Execution

Robot phases run on TAMP and human phases go to teleoperation. After each human phase, TANDEM re-perceives the scene and verifies the effects before planning resumes.

4 · Alignment

Demonstration Generation

All phases of a successful trial form one complete demonstration. DATAFARM aligns the TAMP segments with the VLA’s pretraining data.

Tasks

Five long-horizon tasks, each with at least one stage beyond the TAMP domain.

Frame sequences for the five tasks: Cover Bread Rolls, Solve Constrained Puzzle, Sort & Cover Snacks, Open Obstructed Book, and Store Bread in Closed Box, with bars marking the segments executed by TAMP (green) and by human teleoperation (blue).
  • Cover Bread RollsPlace three breads on a plate, then cover them with a cloth.TAMP→Teleop
  • Solve Constrained PuzzleRemove an obstructing toy, then solve the puzzle.TAMP→Teleop
  • Sort & Cover SnacksSort snacks into bowls, then cover the bread bowl with a cloth.TAMP→Teleop
  • Open Obstructed BookRemove a marker from a book, then open the book.TAMP→Teleop
  • Store Bread in Closed BoxMove the bread off the lid, open the box, then place the bread inside.TAMP→Teleop→TAMP

Demonstrations collected by TANDEM

The label in each video shows who executes the current phase: TAMP or a human.

Cover Bread Rolls
Solve Constrained Puzzle
Sort & Cover Snacks
Open Obstructed Book
Store Bread in Closed Box

Results

π0.5-DROID fine-tuned on 20 TANDEM demonstrations per task, compared with the pretrained model and HITL-TAMP (20 trials each).

Downstream policy performance
Show full results table
Taskπ0.5-DROIDHITL-TAMPTANDEM
Succ.Prog.Succ.Prog.Succ.Prog.
Cover Bread Rolls0%41.3%30%68.8%45%67.5%
Solve Constrained Puzzle0%30.0%0%30.0%50%65.0%
Sort & Cover Snacks0%31.7%15%68.3%75%85.0%
Open Obstructed Book0%38.3%30%71.7%50%76.7%
Store Bread in Closed Box0%33.3%10%40.0%80%93.3%
Average0%34.9%17.0%55.8%60.0%77.5%

Success: every subgoal met. Progress: fraction of subgoals met.

60% average success. Up from 0% for the pretrained model, and above HITL-TAMP (17%) on every task.
No planner at inference. The TANDEM-trained VLA runs at 15 Hz; HITL-TAMP keeps a slow TAMP planner.

Fine-tuned policy rollouts

The fine-tuned VLA executes each full task on its own, without TAMP or a human.

Cover Bread Rolls
Solve Constrained Puzzle
Sort & Cover Snacks
Open Obstructed Book
Store Bread in Closed Box

Human effort

On Cover Bread Rolls, TANDEM and full-task teleoperation get the same human time.

Scaling with matched human effort on Cover Bread Rolls

Demonstration collection

Raw TAMP completes none of the five tasks. TANDEM collected 100 demonstrations from 130 attempts (76.9%).

Demonstration collection attempts by outcome

93.3% of failures come from TAMP execution. The remaining 6.7% come from predicate/operator invention, all on Store Bread in Closed Box.

BibTeX

@article{tandem2026,
  title   = {TANDEM: Task and Motion Planning with As-Needed Demonstrations
             for Efficient Vision-Language-Action Model Fine-tuning},
  author  = {Sahoo, Samrat and Ji, Liang and Silver, Tom and Huang, Yixuan},
  year    = {2026},
  note    = {Under review}
}