Physical AI Notes
KO

Head: lessons

The vision system that turns light into images tagged with capture time and coordinate frame, and uses them to compute the 3D positions of objects.

On the robot

No lessons are published yet. Here are the planned topics.

Overview

What a vision system gives the robot: pixels, capture time, lens, mounting position

  1. What a vision system gives the robot: pixels, capture time, lens and mounting positionComing soon

    Why a camera frame is only useful when it comes with its timestamp and its intrinsic and extrinsic parameters, and where the camera loop sits in the robot as a whole.

Basics

Camera interfaces and bandwidth calculation, depth measurement with the RealSense D435i, sending and receiving camera images in ROS 2, intrinsic calibration and back-projection

  1. Camera interfaces and bandwidth calculations: UVC, MIPI and GMSLComing soon

    Working out the USB bandwidth limit from W×H×B×fps and verifying it with v4l2-ctl and GStreamer.

  2. Measuring depth with the RealSense D435i: IR dot pattern, disparity and error that grows with the square of distanceComing soon

    Checking σ_z ≈ z²·σ_d/(f·B) against the RMS depth error measured on a wall.

  3. Sending and receiving camera images in ROS 2: Image, CameraInfo, TF and QoSComing soon

    Conventions for stamp, frame_id and the optical frame, bandwidth math, and reproducing a QoS mismatch.

  4. Converting pixels to 3D points: intrinsic calibration and back-projectionComing soon

    Checkerboard calibration, and back-projecting the center of a bounding box into 3D.

Advanced

Extrinsic and hand-eye calibration, camera timestamps and exposure time, zero-copy frame handling on Jetson (GStreamer, NVMM), detection and tracking with DeepStream

  1. Estimating the camera mounting pose: extrinsics and hand-eye calibrationComing soon

    AX = XB, why calibration poses must be varied, and how many millimeters of end-effector error a 1° rotation error produces.

  2. Which moment a camera timestamp records: exposure time, latency and rolling shutterComing soon

    Estimating the camera–IMU time offset and testing the prediction that a wall will appear doubled.

  3. Zero-copy frame pipelines on Jetson: GStreamer and NVMMComing soon

    Counting how many times frames cross memory boundaries and get color-converted, then measuring the difference in CPU load and latency.

  4. Detecting and tracking at 30 fps: DeepStream and trackersComing soon

    Trade-offs among the nvinfer interval, tracker type and ID switches, and how SORT and ByteTrack work.

In practice

The error budget from pixels to end-effector coordinates, common camera problems in the field, evaluating open-vocabulary detectors and VLMs

  1. The error budget from pixels to end-effector coordinates: where centimeter-level errors come fromComing soon

    Building an error budget in millimeters from intrinsic and extrinsic calibration error, depth noise, time offset and flying pixels.

  2. How cameras fail in the field: bandwidth, lighting, reflections and vibrationComing soon

    A symptom → cause → diagnostic command table, plus record-and-replay debugging.

  3. Evaluating open-vocabulary detectors and VLMs for robot perception: measuring performance on 100 photos you take yourselfComing soon

    Evaluating the models standalone for prompt sensitivity, score calibration and spatial-relation failures, reported with Wilson score intervals.