Physical AI Notes
KO

Cloud: lessons

Retrains policies on demos and failures collected from robots, evaluates whether performance actually improved, and deploys them back to the robots. One of the slowest feedback loops.

Around the robot

No lessons are published yet. Here are the planned topics.

Overview

The training pipeline from data collection to deployment, comparing robot foundation models (action output, openness, required resources)

  1. The training pipeline is one of the slowest feedback loops: from data collection to deploymentComing soon

    The training pipeline viewed as a feedback loop, with evaluation as the sensor and deployment as the actuator.

  2. Comparing robot foundation models: action output methods, openness and resource requirementsComing soon

    Robot foundation models compared along three axes: how they output actions, how openly they are released and what resources they require.

Basics

Training a first policy with ACT or Diffusion Policy and tracking experiments, trial counts and confidence intervals for success rates

  1. Training and logging your first policy: ACT, Diffusion Policy and experiment trackingComing soon

    One pass through the training loop, the disconnect between loss and success rate, and records that make a run reproducible.

  2. Can you trust a 9/10 success rate: trial counts and confidence intervalsComing soon

    Wilson score intervals (about 60–98% for 9/10), the number of trials needed (about 290 per policy to distinguish 70% from 80%) and a one-page evaluation protocol.

Advanced

Why ACT and Diffusion Policy output actions in chunks, how VLAs output actions (regression, tokens, flow matching), fine-tuning VLAs in practice, RL fine-tuning with demos and experience data

  1. Why ACT and Diffusion Policy output actions in chunksComing soon

    Compounding error and multimodality, action chunks, temporal ensembling, receding horizon and a toy experiment on mode averaging.

  2. How VLAs output actions: regression, tokens and flow matchingComing soon

    Three families of action heads, and the split between a slow System 2 and a fast System 1.

  3. VLA fine-tuning in practice: memory estimation, what to freeze and robot-specific action configurationComing soon

    Memory requirements of LoRA versus full fine-tuning, and matching action dimensions and normalization to your robot.

  4. RL fine-tuning from demonstrations and experience data: DAPG, RLPD and residual RLComing soon

    DAPG, RLPD, residual RL, advantage conditioning and the effect of success-detector errors.

In practice

Debugging train-deploy mismatch, designing policy evaluations, building a data flywheel, staged rollout and rollback

  1. Debugging training–deployment mismatches: normalization, joint order, fps and preprocessingComing soon

    A symptom table from deliberately injected mismatches in normalization, joint order, fps and preprocessing, and a policy bundle that packages weights, normalization statistics and config together.

  2. Designing policy evaluations: paired comparisons, failure taxonomy and benchmark saturationComing soon

    McNemar's test, blind A/B evaluation and sequential testing.

  3. Building a data flywheel: collecting, curating and retraining on failure casesComing soon

    Event-triggered collection, failure mining, dataset versioning and survivorship bias.

  4. Deploying a new policy to robots: versioned bundles, staged rollouts and rollbackComing soon

    Canary deployment and SPRT-based rollback, worked through in a toy fleet simulation.