# Understanding EgoZero by Running It: Turning First-Person Human Video into Robot Imitation Data

EgoZero is an interesting idea because it asks a very practical question: **can we teach robots from videos of humans doing everyday tasks, without first collecting robot demonstrations?**

In this post, I walk through a small hands-on experiment using Meta's HOT3D dataset. The goal is not to evaluate robot performance. Instead, the goal is to understand the learning representation used by EgoZero: object points, fingertip trajectories, and grasp labels expressed as 3D data.

[https://egozero-robot.github.io/](https://egozero-robot.github.io/)

By the end, we will have taken HOT3D annotations, converted them into an EgoZero-like preprocessing format, loaded them with EgoZero's `BCDataset`, run a short smoke training job for the point policy, and visualized what 3D augmentation is doing.

![](https://cdn.hashnode.com/uploads/covers/69ce85bb0ff860b6def2c2ab/015607f7-f261-407f-86ed-b3aa525451cd.png align="center")

Image credit: https://egozero-robot.github.io/

* * *

## 1\. What is EgoZero?

EgoZero is a method proposed by researchers from **New York University / NYU** and **UC Berkeley**. It learns robot manipulation policies from human first-person demonstrations recorded with smart glasses. Its key feature is simple but powerful: **it does not require collecting demonstrations with the robot itself**.

In many robot learning workflows, a human teaches the robot by teleoperating it or by recording demonstrations directly on the robot. This gives us robot states and robot actions, but it also makes data collection expensive. We need the robot hardware, a teleoperation setup, careful safety procedures, calibration, and often a controlled environment.

EgoZero changes the starting point. Instead of first collecting robot demonstrations, it records humans naturally performing tasks with Project Aria smart glasses and tries to learn a robot policy from those human demonstrations.

The important point is that EgoZero does **not** simply feed raw human video frames into a robot policy. That would be difficult because human hands and robot grippers look very different. The camera viewpoints are different too: the human demonstration comes from a smart-glasses perspective, while the robot may use a separate camera at test time. Learning directly from images would make the appearance gap and morphology gap too large.

So EgoZero converts the human demonstration into a **3D point-based representation** instead:

```text
state:
  3D keypoints representing the object

action:
  3D position of the thumb tip
  3D position of the index-finger tip
  gripper closure label
```

At a high level, the pipeline looks like this:

```text
human first-person demonstration
  ↓
object points + fingertip points + grasp label
  ↓
3D point sequence
  ↓
Transformer policy
  ↓
robot execution
```

* * *

## 2\. What this post covers

In this post, I use Meta's **HOT3D** dataset. HOT3D is a first-person dataset for hand-object interaction. It includes data captured with Project Aria and Quest 3, along with 3D hand and object annotations, 3D object models, 2D bounding boxes, and related metadata.

Here is what we will do:

```text
What we will do:
- Read an Aria sequence from HOT3D
- Build EgoZero-style 3D point data from HOT3D hand/object ground truth
- Create thumb/index trajectories and grasp labels
- Create object keypoints from object poses and object meshes
- Save the data in the preprocess format expected by EgoZero's BCDataset
- Run a smoke training job for point_policy
- Visualize the meaning of 3D augmentation
```

This experiment is meant to help us understand **EgoZero's learning representation and training pipeline**. It is **not** a performance evaluation.

* * *

## 3\. Why use HOT3D?

The official EgoZero pipeline uses `.vrs` recordings from Project Aria together with Aria MPS outputs. From Aria MPS, EgoZero can obtain camera poses, hand poses, calibration information, and other signals needed to recover 3D information.

If you do not have a Project Aria device, reproducing that preprocessing pipeline exactly can be difficult.

HOT3D gives us a useful alternative. It includes 3D ground truth for hands and objects. In other words, some of the intermediate 3D information that EgoZero would normally estimate can be reconstructed directly from public annotations.

In the original EgoZero pipeline, the processing looks roughly like this:

```text
object points:
  2D labels + DIFT/CoTracker + Aria SLAM trajectory → triangulation

fingertip points:
  HaMeR + Aria MPS hand pose → estimated 3D thumb/index points
```

In this post, we replace those estimation steps with HOT3D ground truth:

```text
object points:
  built from HOT3D object pose + object mesh

fingertip points:
  taken from HOT3D UmeTrack hand landmarks

gripper closure label:
  created by thresholding the thumb-index distance
```

So we are not fully reproducing the official EgoZero preprocessing pipeline. Instead, we are reconstructing the **final 3D state/action format** that EgoZero uses for learning.

![](https://cdn.hashnode.com/uploads/covers/69ce85bb0ff860b6def2c2ab/5a941603-ba27-4ffa-88d2-dc2c059f520f.png align="center")

* * *

## 4\. Working directory layout

I use the following directory layout:

```text
~/workdir/
  egozero/                # official EgoZero repository
  hot3d/                  # HOT3D toolkit
  hot3d_dataset/          # actual HOT3D data
  egozero-hot3d-tools/    # helper scripts for this post
```

The commands below assume these environment variables:

```bash
WORKDIR="$HOME/workdir"
EGOZERO_DIR="$WORKDIR/egozero"
HOT3D_TOOLKIT_DIR="$WORKDIR/hot3d/hot3d"
HOT3D_DATA_DIR="$WORKDIR/hot3d_dataset"
TOOLS_DIR="$WORKDIR/egozero-hot3d-tools"
```

I assume that the helper scripts have been cloned like this:

```bash
cd "$WORKDIR"
git clone https://github.com/fromfactory/egozero-hot3d-tools
```

* * *

## 5\. Setting up the environment

First, create an environment for the HOT3D toolkit:

```bash
cd "$WORKDIR"
git clone https://github.com/facebookresearch/hot3d.git
cd "$HOT3D_TOOLKIT_DIR"

conda create -y -n hot3d python=3.10
conda activate hot3d
python -m pip install --upgrade pip setuptools wheel
python -m pip install -r "$TOOLS_DIR/requirements-hot3d.txt"
```

Next, clone and set up EgoZero by following the official EgoZero README. I will not repeat the full EgoZero setup here, but the rest of this post assumes that the EgoZero environment is available with:

```bash
conda activate egozero
```

For the smoke training run in this post, I also install a few small extra dependencies:

```bash
cd "$EGOZERO_DIR"
conda activate egozero
python -m pip install -r "$TOOLS_DIR/requirements-egozero-extra.txt"
```

One small practical note: I do **not** recommend blindly reinstalling the entire `requirements.txt` if your environment is already working. Depending on your machine, dependencies related to robosuite or EGL rendering can cause setup issues. In this experiment, we do not use real-robot evaluation or EGL rendering.

* * *

## 6\. Downloading the HOT3D data

To download HOT3D, you first need to accept the terms on the official site and obtain the download URL JSON files. These files may contain time-limited URLs, so **do not commit them to GitHub**.

For this experiment, we need the following two files:

```text
Hot3DAssets_download_urls.json
Hot3DAria_download_urls.json
```

Place them here:

```text
$HOT3D_TOOLKIT_DIR/data_downloader/Hot3DAssets_download_urls.json
$HOT3D_TOOLKIT_DIR/data_downloader/Hot3DAria_download_urls.json
```

For the first sanity check, I use one Aria sequence that also appears in the HOT3D README examples:

```bash
cd "$HOT3D_TOOLKIT_DIR/data_downloader"
mkdir -p "$HOT3D_DATA_DIR"

python3 dataset_downloader_base_main.py \
  -c Hot3DAssets_download_urls.json \
  -o "$HOT3D_DATA_DIR" \
  --sequence_name all

python3 dataset_downloader_base_main.py \
  -c Hot3DAria_download_urls.json \
  -o "$HOT3D_DATA_DIR" \
  --sequence_name P0003_c701bd11 \
  --data_types all
```

Check the downloaded folders and files:

```bash
find "$HOT3D_DATA_DIR" -maxdepth 2 -type d | sort | head
find "$HOT3D_DATA_DIR/P0003_c701bd11" -maxdepth 2 -type f | sort | head
```

![](https://cdn.hashnode.com/uploads/covers/69ce85bb0ff860b6def2c2ab/2e937e00-1605-4c86-8060-f4e14d2a901d.png align="center")

* * *

## 7\. Inspecting what is inside HOT3D

Before converting anything, it is useful to make sure we can actually read the HOT3D sequence.

```bash
conda activate hot3d
python "$TOOLS_DIR/scripts/hot3d/scan_hot3d.py" \
  --hot3d-toolkit-root "$HOT3D_TOOLKIT_DIR" \
  --sequence-folder "$HOT3D_DATA_DIR/P0003_c701bd11" \
  --object-library-folder "$HOT3D_DATA_DIR/assets" \
  --stream-id 214-1
```

Next, export a grid of first-person RGB images with hand/object bounding boxes:

```bash
python "$TOOLS_DIR/scripts/hot3d/export_rgb_bbox_grid.py" \
  --hot3d-toolkit-root "$HOT3D_TOOLKIT_DIR" \
  --sequence-folder "$HOT3D_DATA_DIR/P0003_c701bd11" \
  --object-library-folder "$HOT3D_DATA_DIR/assets" \
  --stream-id 214-1 \
  --out-dir "$EGOZERO_DIR/hot3d_egozero/plots/rgb_bbox"
```

![](https://cdn.hashnode.com/uploads/covers/69ce85bb0ff860b6def2c2ab/70d72f95-c9d0-409b-860c-6a87c5acb4d0.jpg align="center")

At this point, we are only checking that HOT3D can be read and that the sequence contains visible hands and objects. We have not converted anything into EgoZero format yet.

* * *

## 8\. Creating EgoZero-style preprocess data from HOT3D

Now we convert HOT3D hand/object ground truth into the format expected by EgoZero's `BCDataset`.

The output directory will be:

```text
$EGOZERO_DIR/hot3d_egozero/preprocess
```

Run the conversion:

```bash
conda activate hot3d
python "$TOOLS_DIR/scripts/hot3d/hot3d_to_egozero_preprocess.py" \
  --hot3d-toolkit-root "$HOT3D_TOOLKIT_DIR" \
  --sequence-folder "$HOT3D_DATA_DIR/P0003_c701bd11" \
  --object-library-folder "$HOT3D_DATA_DIR/assets" \
  --stream-id 214-1 \
  --out-preprocess "$EGOZERO_DIR/hot3d_egozero/preprocess" \
  --hand auto \
  --keypoint-mode bbox9 \
  --grasp-threshold-m 0.08 \
  --contact-dist-m 999 \
  --min-grasp-frames 5 \
  --pre-frames 75 \
  --post-frames 75 \
  --max-demos 30
```

For each demonstration, the script creates files like these:

```text
demonstration_00001/
  triangulation.json
  index.npy
  thumb.npy
  grasp.npy
  pose.npy
  first_frame_g2w.npy
  first_frame.png
  hot3d_preview.mp4
  hot3d_meta.json
```

The name `triangulation.json` is a little misleading in this experiment. In the original EgoZero pipeline, object points are produced through triangulation. Here, however, the points are created from HOT3D ground-truth object poses and object meshes.

So the file format matches EgoZero, but the way we produce the data is different from the official preprocessing pipeline.

* * *

## 9\. Diagnosing grasp labels and BCDataset compatibility

Not every converted demonstration can be used for training. EgoZero's `BCDataset` assumes that each demo has a meaningful grasp segment, a sufficiently long trajectory, and a grasp that still makes sense after stationary-point removal.

First, run the diagnostic script:

```bash
conda activate egozero
python "$TOOLS_DIR/scripts/egozero/diagnose_hot3d_grasp.py" \
  "$EGOZERO_DIR/hot3d_egozero/preprocess"
```

In my experiment, the conversion produced 13 candidate demos, and 8 of them were compatible with `BCDataset`:

```text
candidate demos: 13
BCDataset-compatible demos: 8
```

Copy only the compatible demos into a separate folder:

```bash
python "$TOOLS_DIR/scripts/egozero/filter_bc_compatible.py" \
  "$EGOZERO_DIR/hot3d_egozero/preprocess" \
  "$EGOZERO_DIR/hot3d_egozero/preprocess_bc"
```

The `preprocess_bc` folder includes guard demos at the beginning and end. This matches EgoZero's `BCDataset` implementation, which skips the first and last demos:

```text
preprocess_bc/
  demonstration_00000      # guard
  demonstration_00001      # valid
  demonstration_00002      # valid
  ...
  demonstration_00008      # valid
  demonstration_99999      # guard
```

Finally, check whether EgoZero can read the data through `BCDataset`:

```bash
python "$TOOLS_DIR/scripts/egozero/check_bc_dataset.py" \
  "$EGOZERO_DIR/hot3d_egozero/preprocess_bc" \
  --egozero-root "$EGOZERO_DIR"
```

In my run, EgoZero's internal outlier filter reduced the final dataset to 4 training examples:

```text
BCDataset-compatible demos: 8
After BCDataset outlier filtering: 4 training examples
```

That is a very small dataset, but it is enough for a smoke test. Again, the purpose here is not to report performance. The purpose is to check that the data format, dataloader, model, loss, optimizer, and checkpointing path all work together.

* * *

## 10\. Running a short EgoZero point-policy training job

Here we only test whether the **EgoZero 3D point policy can enter the training loop with HOT3D-derived data**. We are not measuring robot success rate.

```bash
bash "$TOOLS_DIR/scripts/training/train_hot3d_egozero_smoke.sh" \
  --egozero-root "$EGOZERO_DIR" \
  --data-dir "$EGOZERO_DIR/hot3d_egozero/preprocess_bc" \
  --conda-env egozero \
  --num-steps 1000 \
  --save-every 500 \
  2>&1 | tee "$EGOZERO_DIR/hot3d_egozero_train.log"
```

When training starts, the log prints the workspace path:

```text
workspace: .../point_policy/exp_local/.../hot3d_egozero/...
```

Inside that workspace, you should see files such as `train.csv` and `snapshot/500.pt`:

```bash
WORKSPACE=$(grep -oP 'workspace: \K.*' "$EGOZERO_DIR/hot3d_egozero_train.log" | tail -1)
echo "$WORKSPACE"
find "$WORKSPACE" -maxdepth 3 -type f | sort
```

In my experiment, `snapshot/500.pt` was successfully saved.

* * *

## 11\. Inspecting the training output

First, plot the loss:

```bash
python "$TOOLS_DIR/scripts/egozero/plot_train_loss.py" "$WORKSPACE"
```

![](https://cdn.hashnode.com/uploads/covers/69ce85bb0ff860b6def2c2ab/49286262-8ded-42f9-b9d8-516729d2f36e.png align="center")

In my run, the actor loss dropped from around 25 near step 0 to roughly the 1.x range around step 900. This tells us that the model, optimizer, dataloader, loss computation, and backpropagation were working, and that the model could fit this tiny dataset.

However, this result needs to be interpreted carefully. The final number of training examples was only 4. So the loss decrease does **not** demonstrate generalization or robot performance. It is simply a sanity check that the learning loop runs correctly.

Next, inspect the checkpoint:

```bash
python "$TOOLS_DIR/scripts/egozero/inspect_snapshot.py" "$WORKSPACE/snapshot/500.pt"
```

In my experiment, the checkpoint contained entries such as:

```text
_global_step: 500
stats: past_tracks, actions
actor: Transformer-style weights
point_projector: point projection layer weights
actor_opt: optimizer state
point_opt: optimizer state
```

This confirms that training reached checkpoint saving successfully.

* * *

## 12\. Looking at data diagnostic plots

Figures such as `0_demonstration_00001.png` are not training results. They are diagnostic plots for checking the quality of the converted EgoZero-style demonstrations.

![](https://cdn.hashnode.com/uploads/covers/69ce85bb0ff860b6def2c2ab/1e2d74f9-55c2-48b2-92a7-114e6dcd0c1b.png align="center")

These plots help us inspect things like frame-to-frame fingertip movement, grasp intervals, distance from the grasp frame, and distance to object points. They are useful for catching problems such as all-False grasp labels, trajectories that are too short, or large discontinuities.

You can also rank demonstrations by trajectory smoothness:

```bash
python "$TOOLS_DIR/scripts/egozero/rank_demo_smoothness.py" \
  "$EGOZERO_DIR/hot3d_egozero/preprocess_bc"
```

One caveat: a smooth-looking trajectory does not always mean the demo will be used for training. EgoZero's `BCDataset` also applies internal outlier filtering based on distances between object points and grasp points.

* * *

## 13\. Visualizing 3D augmentation

One of the important ideas in EgoZero is **3D augmentation**.

Image augmentation usually means changing the image itself: rotating it slightly, shifting brightness, cropping it, and so on. EgoZero's 3D augmentation is different. It applies the same 3D transformation to both the **object points** and the **fingertip actions**.

First, visualize how the distribution of object points spreads under augmentation:

```bash
python "$TOOLS_DIR/scripts/visualization/visualize_3d_augmentation_three_views.py" \
  "$EGOZERO_DIR/hot3d_egozero/preprocess_bc" \
  --out "$EGOZERO_DIR/hot3d_egozero/plots/3d_aug_three_views_bc.png"
```

![](https://cdn.hashnode.com/uploads/covers/69ce85bb0ff860b6def2c2ab/1ac8baa0-c318-4ba7-b3d7-ecd26649f86f.png align="center")

This applies random 3D rotations and translations to the object points in `preprocess_bc`, then visualizes how the point distribution expands.

Next, visualize one demo where not only the object points but also the fingertip trajectory are transformed together:

```bash
python "$TOOLS_DIR/scripts/visualization/visualize_one_demo_state_action_aug.py" \
  "$EGOZERO_DIR/hot3d_egozero/preprocess_bc/demonstration_00001" \
  --out "$EGOZERO_DIR/hot3d_egozero/plots/one_demo_state_action_aug.png"
```

![](https://cdn.hashnode.com/uploads/covers/69ce85bb0ff860b6def2c2ab/dbcae473-ac1b-43b9-b77d-9771ae719c11.png align="center")

The important detail is that we are not moving the object independently. The object points and fingertip trajectory are moved by the same 3D transform, so the relative manipulation relationship is preserved.

Intuitively, the augmentation creates examples like this:

```text
Original data:
  the object is on the left
  the hand reaches left and grasps it

After 3D augmentation:
  the object is shifted slightly to the right
  the hand trajectory is shifted by the same amount and still grasps it
```

This augmented scene did not literally appear in HOT3D, so in that sense it is synthetic data. But because the object and fingertip trajectory are transformed together as a rigid 3D relationship, the hand-object relationship is still meaningful.

The caution is that not every transformation is valid. If an augmentation makes the object pass through the table, float in the air, change the task meaning, or move the object without moving the hand trajectory, it can become broken supervision.

* * *

## 14\. Summary

EgoZero does not try to learn from raw human first-person video frames directly. Instead, it converts human demonstrations into a 3D state/action representation made of object points, fingertip points, and grasp labels, then trains a policy on that representation.

Without a Project Aria device, reproducing the official EgoZero preprocessing pipeline exactly is difficult. But because HOT3D includes 3D ground truth for hands and objects, we can reconstruct the kind of 3D point representation that EgoZero eventually uses for training.

In this experiment, we confirmed the following flow using only HOT3D:

```text
HOT3D
  ↓
EgoZero-style 3D state/action data
  ↓
BCDataset
  ↓
point_policy smoke training with a dummy environment
  ↓
loss inspection, checkpoint inspection, and 3D augmentation visualization
```

The main lesson is that the core idea of EgoZero is not simply “give smart-glasses video to a robot.” The more important idea is to use **3D point representations** to bridge the gap between human hands and robot grippers, and also between different camera viewpoints.

That is what makes EgoZero worth studying: it reframes human video as structured 3D manipulation data that a robot policy can learn from.