Understanding EgoZero by Running It: Turning First-Person Human Video into Robot Imitation Data

EgoZero is an interesting idea because it asks a very practical question: can we teach robots from videos of humans doing everyday tasks, without first collecting robot demonstrations?

In this post, I walk through a small hands-on experiment using Meta's HOT3D dataset. The goal is not to evaluate robot performance. Instead, the goal is to understand the learning representation used by EgoZero: object points, fingertip trajectories, and grasp labels expressed as 3D data.

https://egozero-robot.github.io/

By the end, we will have taken HOT3D annotations, converted them into an EgoZero-like preprocessing format, loaded them with EgoZero's BCDataset, run a short smoke training job for the point policy, and visualized what 3D augmentation is doing.

Image credit: https://egozero-robot.github.io/

---

1. What is EgoZero?

EgoZero is a method proposed by researchers from New York University / NYU and UC Berkeley. It learns robot manipulation policies from human first-person demonstrations recorded with smart glasses. Its key feature is simple but powerful: it does not require collecting demonstrations with the robot itself.

In many robot learning workflows, a human teaches the robot by teleoperating it or by recording demonstrations directly on the robot. This gives us robot states and robot actions, but it also makes data collection expensive. We need the robot hardware, a teleoperation setup, careful safety procedures, calibration, and often a controlled environment.

EgoZero changes the starting point. Instead of first collecting robot demonstrations, it records humans naturally performing tasks with Project Aria smart glasses and tries to learn a robot policy from those human demonstrations.

The important point is that EgoZero does not simply feed raw human video frames into a robot policy. That would be difficult because human hands and robot grippers look very different. The camera viewpoints are different too: the human demonstration comes from a smart-glasses perspective, while the robot may use a separate camera at test time. Learning directly from images would make the appearance gap and morphology gap too large.

So EgoZero converts the human demonstration into a 3D point-based representation instead:

state:
  3D keypoints representing the object

action:
  3D position of the thumb tip
  3D position of the index-finger tip
  gripper closure label

At a high level, the pipeline looks like this:

human first-person demonstration
  ↓
object points + fingertip points + grasp label
  ↓
3D point sequence
  ↓
Transformer policy
  ↓
robot execution

---

2. What this post covers

In this post, I use Meta's HOT3D dataset. HOT3D is a first-person dataset for hand-object interaction. It includes data captured with Project Aria and Quest 3, along with 3D hand and object annotations, 3D object models, 2D bounding boxes, and related metadata.

Here is what we will do:

What we will do:
- Read an Aria sequence from HOT3D
- Build EgoZero-style 3D point data from HOT3D hand/object ground truth
- Create thumb/index trajectories and grasp labels
- Create object keypoints from object poses and object meshes
- Save the data in the preprocess format expected by EgoZero's BCDataset
- Run a smoke training job for point_policy
- Visualize the meaning of 3D augmentation

This experiment is meant to help us understand EgoZero's learning representation and training pipeline. It is not a performance evaluation.

---

3. Why use HOT3D?

The official EgoZero pipeline uses .vrs recordings from Project Aria together with Aria MPS outputs. From Aria MPS, EgoZero can obtain camera poses, hand poses, calibration information, and other signals needed to recover 3D information.

If you do not have a Project Aria device, reproducing that preprocessing pipeline exactly can be difficult.

HOT3D gives us a useful alternative. It includes 3D ground truth for hands and objects. In other words, some of the intermediate 3D information that EgoZero would normally estimate can be reconstructed directly from public annotations.

In the original EgoZero pipeline, the processing looks roughly like this:

object points:
  2D labels + DIFT/CoTracker + Aria SLAM trajectory → triangulation

fingertip points:
  HaMeR + Aria MPS hand pose → estimated 3D thumb/index points

In this post, we replace those estimation steps with HOT3D ground truth:

object points:
  built from HOT3D object pose + object mesh

fingertip points:
  taken from HOT3D UmeTrack hand landmarks

gripper closure label:
  created by thresholding the thumb-index distance

So we are not fully reproducing the official EgoZero preprocessing pipeline. Instead, we are reconstructing the final 3D state/action format that EgoZero uses for learning.

---

4. Working directory layout

I use the following directory layout:

~/workdir/
  egozero/                # official EgoZero repository
  hot3d/                  # HOT3D toolkit
  hot3d_dataset/          # actual HOT3D data
  egozero-hot3d-tools/    # helper scripts for this post

The commands below assume these environment variables:

WORKDIR="$HOME/workdir"
EGOZERO_DIR="$WORKDIR/egozero"
HOT3D_TOOLKIT_DIR="$WORKDIR/hot3d/hot3d"
HOT3D_DATA_DIR="$WORKDIR/hot3d_dataset"
TOOLS_DIR="$WORKDIR/egozero-hot3d-tools"

I assume that the helper scripts have been cloned like this:

cd "$WORKDIR"
git clone https://github.com/fromfactory/egozero-hot3d-tools

---

5. Setting up the environment

First, create an environment for the HOT3D toolkit:

cd "$WORKDIR"
git clone https://github.com/facebookresearch/hot3d.git
cd "$HOT3D_TOOLKIT_DIR"

conda create -y -n hot3d python=3.10
conda activate hot3d
python -m pip install --upgrade pip setuptools wheel
python -m pip install -r "$TOOLS_DIR/requirements-hot3d.txt"

Next, clone and set up EgoZero by following the official EgoZero README. I will not repeat the full EgoZero setup here, but the rest of this post assumes that the EgoZero environment is available with:

conda activate egozero

For the smoke training run in this post, I also install a few small extra dependencies:

cd "$EGOZERO_DIR"
conda activate egozero
python -m pip install -r "$TOOLS_DIR/requirements-egozero-extra.txt"

One small practical note: I do not recommend blindly reinstalling the entire requirements.txt if your environment is already working. Depending on your machine, dependencies related to robosuite or EGL rendering can cause setup issues. In this experiment, we do not use real-robot evaluation or EGL rendering.

---

6. Downloading the HOT3D data

To download HOT3D, you first need to accept the terms on the official site and obtain the download URL JSON files. These files may contain time-limited URLs, so do not commit them to GitHub.

For this experiment, we need the following two files:

Hot3DAssets_download_urls.json
Hot3DAria_download_urls.json

Place them here:

$HOT3D_TOOLKIT_DIR/data_downloader/Hot3DAssets_download_urls.json
$HOT3D_TOOLKIT_DIR/data_downloader/Hot3DAria_download_urls.json

For the first sanity check, I use one Aria sequence that also appears in the HOT3D README examples:

cd "$HOT3D_TOOLKIT_DIR/data_downloader"
mkdir -p "$HOT3D_DATA_DIR"

python3 dataset_downloader_base_main.py \
  -c Hot3DAssets_download_urls.json \
  -o "$HOT3D_DATA_DIR" \
  --sequence_name all

python3 dataset_downloader_base_main.py \
  -c Hot3DAria_download_urls.json \
  -o "$HOT3D_DATA_DIR" \
  --sequence_name P0003_c701bd11 \
  --data_types all

Check the downloaded folders and files:

find "$HOT3D_DATA_DIR" -maxdepth 2 -type d | sort | head
find "$HOT3D_DATA_DIR/P0003_c701bd11" -maxdepth 2 -type f | sort | head

---

7. Inspecting what is inside HOT3D

Before converting anything, it is useful to make sure we can actually read the HOT3D sequence.

conda activate hot3d
python "$TOOLS_DIR/scripts/hot3d/scan_hot3d.py" \
  --hot3d-toolkit-root "$HOT3D_TOOLKIT_DIR" \
  --sequence-folder "$HOT3D_DATA_DIR/P0003_c701bd11" \
  --object-library-folder "$HOT3D_DATA_DIR/assets" \
  --stream-id 214-1

Next, export a grid of first-person RGB images with hand/object bounding boxes:

python "$TOOLS_DIR/scripts/hot3d/export_rgb_bbox_grid.py" \
  --hot3d-toolkit-root "$HOT3D_TOOLKIT_DIR" \
  --sequence-folder "$HOT3D_DATA_DIR/P0003_c701bd11" \
  --object-library-folder "$HOT3D_DATA_DIR/assets" \
  --stream-id 214-1 \
  --out-dir "$EGOZERO_DIR/hot3d_egozero/plots/rgb_bbox"

At this point, we are only checking that HOT3D can be read and that the sequence contains visible hands and objects. We have not converted anything into EgoZero format yet.

---

8. Creating EgoZero-style preprocess data from HOT3D

Now we convert HOT3D hand/object ground truth into the format expected by EgoZero's BCDataset.

The output directory will be:

$EGOZERO_DIR/hot3d_egozero/preprocess

Run the conversion:

conda activate hot3d
python "$TOOLS_DIR/scripts/hot3d/hot3d_to_egozero_preprocess.py" \
  --hot3d-toolkit-root "$HOT3D_TOOLKIT_DIR" \
  --sequence-folder "$HOT3D_DATA_DIR/P0003_c701bd11" \
  --object-library-folder "$HOT3D_DATA_DIR/assets" \
  --stream-id 214-1 \
  --out-preprocess "$EGOZERO_DIR/hot3d_egozero/preprocess" \
  --hand auto \
  --keypoint-mode bbox9 \
  --grasp-threshold-m 0.08 \
  --contact-dist-m 999 \
  --min-grasp-frames 5 \
  --pre-frames 75 \
  --post-frames 75 \
  --max-demos 30

For each demonstration, the script creates files like these:

demonstration_00001/
  triangulation.json
  index.npy
  thumb.npy
  grasp.npy
  pose.npy
  first_frame_g2w.npy
  first_frame.png
  hot3d_preview.mp4
  hot3d_meta.json

The name triangulation.json is a little misleading in this experiment. In the original EgoZero pipeline, object points are produced through triangulation. Here, however, the points are created from HOT3D ground-truth object poses and object meshes.

So the file format matches EgoZero, but the way we produce the data is different from the official preprocessing pipeline.

---

9. Diagnosing grasp labels and BCDataset compatibility

Not every converted demonstration can be used for training. EgoZero's BCDataset assumes that each demo has a meaningful grasp segment, a sufficiently long trajectory, and a grasp that still makes sense after stationary-point removal.

First, run the diagnostic script:

conda activate egozero
python "$TOOLS_DIR/scripts/egozero/diagnose_hot3d_grasp.py" \
  "$EGOZERO_DIR/hot3d_egozero/preprocess"

In my experiment, the conversion produced 13 candidate demos, and 8 of them were compatible with BCDataset:

candidate demos: 13
BCDataset-compatible demos: 8

Copy only the compatible demos into a separate folder:

python "$TOOLS_DIR/scripts/egozero/filter_bc_compatible.py" \
  "$EGOZERO_DIR/hot3d_egozero/preprocess" \
  "$EGOZERO_DIR/hot3d_egozero/preprocess_bc"

The preprocess_bc folder includes guard demos at the beginning and end. This matches EgoZero's BCDataset implementation, which skips the first and last demos:

preprocess_bc/
  demonstration_00000      # guard
  demonstration_00001      # valid
  demonstration_00002      # valid
  ...
  demonstration_00008      # valid
  demonstration_99999      # guard

Finally, check whether EgoZero can read the data through BCDataset:

python "$TOOLS_DIR/scripts/egozero/check_bc_dataset.py" \
  "$EGOZERO_DIR/hot3d_egozero/preprocess_bc" \
  --egozero-root "$EGOZERO_DIR"

In my run, EgoZero's internal outlier filter reduced the final dataset to 4 training examples:

BCDataset-compatible demos: 8
After BCDataset outlier filtering: 4 training examples

That is a very small dataset, but it is enough for a smoke test. Again, the purpose here is not to report performance. The purpose is to check that the data format, dataloader, model, loss, optimizer, and checkpointing path all work together.

---

10. Running a short EgoZero point-policy training job

Here we only test whether the EgoZero 3D point policy can enter the training loop with HOT3D-derived data. We are not measuring robot success rate.

bash "$TOOLS_DIR/scripts/training/train_hot3d_egozero_smoke.sh" \
  --egozero-root "$EGOZERO_DIR" \
  --data-dir "$EGOZERO_DIR/hot3d_egozero/preprocess_bc" \
  --conda-env egozero \
  --num-steps 1000 \
  --save-every 500 \
  2>&1 | tee "$EGOZERO_DIR/hot3d_egozero_train.log"

When training starts, the log prints the workspace path:

workspace: .../point_policy/exp_local/.../hot3d_egozero/...

Inside that workspace, you should see files such as train.csv and snapshot/500.pt:

WORKSPACE=$(grep -oP 'workspace: \K.*' "$EGOZERO_DIR/hot3d_egozero_train.log" | tail -1)
echo "$WORKSPACE"
find "$WORKSPACE" -maxdepth 3 -type f | sort

In my experiment, snapshot/500.pt was successfully saved.

---

11. Inspecting the training output

First, plot the loss:

python "$TOOLS_DIR/scripts/egozero/plot_train_loss.py" "$WORKSPACE"

In my run, the actor loss dropped from around 25 near step 0 to roughly the 1.x range around step 900. This tells us that the model, optimizer, dataloader, loss computation, and backpropagation were working, and that the model could fit this tiny dataset.

However, this result needs to be interpreted carefully. The final number of training examples was only 4. So the loss decrease does not demonstrate generalization or robot performance. It is simply a sanity check that the learning loop runs correctly.

Next, inspect the checkpoint:

python "$TOOLS_DIR/scripts/egozero/inspect_snapshot.py" "$WORKSPACE/snapshot/500.pt"

In my experiment, the checkpoint contained entries such as:

_global_step: 500
stats: past_tracks, actions
actor: Transformer-style weights
point_projector: point projection layer weights
actor_opt: optimizer state
point_opt: optimizer state

This confirms that training reached checkpoint saving successfully.

---

12. Looking at data diagnostic plots

Figures such as 0_demonstration_00001.png are not training results. They are diagnostic plots for checking the quality of the converted EgoZero-style demonstrations.

These plots help us inspect things like frame-to-frame fingertip movement, grasp intervals, distance from the grasp frame, and distance to object points. They are useful for catching problems such as all-False grasp labels, trajectories that are too short, or large discontinuities.

You can also rank demonstrations by trajectory smoothness:

python "$TOOLS_DIR/scripts/egozero/rank_demo_smoothness.py" \
  "$EGOZERO_DIR/hot3d_egozero/preprocess_bc"

One caveat: a smooth-looking trajectory does not always mean the demo will be used for training. EgoZero's BCDataset also applies internal outlier filtering based on distances between object points and grasp points.

---

13. Visualizing 3D augmentation

One of the important ideas in EgoZero is 3D augmentation.

Image augmentation usually means changing the image itself: rotating it slightly, shifting brightness, cropping it, and so on. EgoZero's 3D augmentation is different. It applies the same 3D transformation to both the object points and the fingertip actions.

First, visualize how the distribution of object points spreads under augmentation:

python "$TOOLS_DIR/scripts/visualization/visualize_3d_augmentation_three_views.py" \
  "$EGOZERO_DIR/hot3d_egozero/preprocess_bc" \
  --out "$EGOZERO_DIR/hot3d_egozero/plots/3d_aug_three_views_bc.png"

This applies random 3D rotations and translations to the object points in preprocess_bc, then visualizes how the point distribution expands.

Next, visualize one demo where not only the object points but also the fingertip trajectory are transformed together:

python "$TOOLS_DIR/scripts/visualization/visualize_one_demo_state_action_aug.py" \
  "$EGOZERO_DIR/hot3d_egozero/preprocess_bc/demonstration_00001" \
  --out "$EGOZERO_DIR/hot3d_egozero/plots/one_demo_state_action_aug.png"

The important detail is that we are not moving the object independently. The object points and fingertip trajectory are moved by the same 3D transform, so the relative manipulation relationship is preserved.

Intuitively, the augmentation creates examples like this:

Original data:
  the object is on the left
  the hand reaches left and grasps it

After 3D augmentation:
  the object is shifted slightly to the right
  the hand trajectory is shifted by the same amount and still grasps it

This augmented scene did not literally appear in HOT3D, so in that sense it is synthetic data. But because the object and fingertip trajectory are transformed together as a rigid 3D relationship, the hand-object relationship is still meaningful.

The caution is that not every transformation is valid. If an augmentation makes the object pass through the table, float in the air, change the task meaning, or move the object without moving the hand trajectory, it can become broken supervision.

---

14. Summary

EgoZero does not try to learn from raw human first-person video frames directly. Instead, it converts human demonstrations into a 3D state/action representation made of object points, fingertip points, and grasp labels, then trains a policy on that representation.

Without a Project Aria device, reproducing the official EgoZero preprocessing pipeline exactly is difficult. But because HOT3D includes 3D ground truth for hands and objects, we can reconstruct the kind of 3D point representation that EgoZero eventually uses for training.

In this experiment, we confirmed the following flow using only HOT3D:

HOT3D
  ↓
EgoZero-style 3D state/action data
  ↓
BCDataset
  ↓
point_policy smoke training with a dummy environment
  ↓
loss inspection, checkpoint inspection, and 3D augmentation visualization

The main lesson is that the core idea of EgoZero is not simply “give smart-glasses video to a robot.” The more important idea is to use 3D point representations to bridge the gap between human hands and robot grippers, and also between different camera viewpoints.

That is what makes EgoZero worth studying: it reframes human video as structured 3D manipulation data that a robot policy can learn from.