Physical AI Notes
KO

Leader arm: lessons

An input device that lets a human operate the robot in place of the policy, recording the motion as observation, action and timestamp data for policy training.

Around the robot

No lessons are published yet. Here are the planned topics.

Overview

Where teleop sits in the robot stack (a loop the human closes in place of the policy), comparing teleop interfaces from leader arms to human demonstration videos

  1. Where teleoperation fits in the robot stack: a loop the human closes in place of the policyComing soon

    A human temporarily stands in for the policy that will later take over, and what is recorded meanwhile becomes the dataset.

  2. Comparing teleoperation interfaces: from leader arms to human demonstration videos, the pros and cons of each approachComing soon

    Data collection methods compared along six axes, and what each one gives up.

Basics

Recording a first demo with an SO-101 leader-follower, the data structure of LeRobotDataset, VR and hand-tracking teleop basics

  1. Recording your first demonstration with an SO-101 leader-follower pairComing soon

    How calibration, teleoperation and recording settings show up in the data.

  2. The structure of recorded data: episodes, frames and timestamps in LeRobotDatasetComing soon

    Parquet and MP4 files, metadata, recomputing normalization statistics, checking for dropped frames, and a comparison with RLDS and MCAP.

  3. VR and hand-tracking teleoperation basics: frame alignment, scaling, clutching and per-cycle IKComing soon

    How the chain of coordinate frames and the IK weights change what ends up in the recorded data.

Advanced

Operating styles under latency (move-and-wait, bilateral teleop), leader-arm dynamics and gravity compensation, the action logging schema and time alignment, collecting data without a robot (UMI, gloves, exoskeletons, human demonstration videos)

  1. Latency changes how operators move: move-and-wait and the stability of bilateral teleoperationComing soon

    The pausing strategies operators adopt under latency, along with passivity and wave variables.

  2. Leader arm dynamics: joint mirroring, gravity compensation and follower sagComing soon

    How the operator's corrections get recorded in the data as a difference between action and state.

  3. What to record as the action: logging schema and time alignmentComing soon

    Comparing candidate action definitions, and estimating latency within a dataset using cross-correlation.

  4. Collecting data without a robot: UMI, gloves, exoskeletons and human demonstration videosComing soon

    The feedback and labels given up in exchange for data scale, and feasibility filters.

In practice

What makes a good demo, intervention data and DAgger-style methods, what goes wrong during data collection

  1. What makes a good demonstration: consistency, diversity and the problems with scaling up data volumeComing soon

    Reproducing the robomimic PH/MH results, signals of demonstration quality, and the bias that filtering introduces.

  2. Learning from human interventions: intervention data and the DAgger familyComing soon

    Bumpless transfer, intervention flags and HG-DAgger rounds.

  3. What goes wrong during data collection: calibration drift, frame drops and operator fatigueComing soon

    Symptoms and the traces they leave in the data, a data linter, and train/validation splits by session.