Automatic Ground Truth Generation (AGTG)

The AGTG pipeline describes the stages for automating the annotation process of a dataset. There are two modes of operation.

1. Fully Automatic: This is invoked at the time of importing the dataset as a snapshot and as a background process which deploys a detection model to drive SAM-2.
2. Semi-Automatic: This is invoked when users trigger the AI assisted annotations in the dataset gallery. Users can select portions of the dataset to auto-annotate, but SAM-2 requires initial annotations from the users as prompts.

Fully Automatic Ground Truth Generation

This annotation feature is available at the time of importing the dataset via snapshot restoration. The auto-annotation process is done in the background allowing the user to focus on separate tasks. This section will show the steps for performing this type of auto-annotation in EdgeFirst Studio. However, this feature can also be deployed using the EdgeFirst Client in the command line.

A complete description of this feature along with the buttons associated in this tutorial can be found under Studio.

This feature is a two-step process: Create Snapshot and Restore Snapshot.

Create Snapshot

Recall that snapshots are frozen and compact form of datasets. There are three ways of creating snapshots.

1. Create from Existing Dataset

   On the dataset card, open the dataset context menu and select "Create Snapshot".

   

   Create Snapshot From Dataset

   Next enter the name for the dataset snapshot. Click on "Add Snapshot" to begin the snapshot creation.

   

   Name the Snapshot

   The snapshot creation process will be shown like the following.

   

   Snapshot Process

2. Upload from MCAP File

   An MCAP file is a recording captured by an EdgeFirst Platform (Maivin or Raivin).

   To create a snapshot from an MCAP file, visit the "Data Snapshots" page.

   

   Data Snapshots

   Either drag and drop an MCAP file into the page OR click on the "From File" button to allow selection of the MCAP file from your directories.

   

   MCAP Import Snapshot

3. Upload from Zip/Arrow (EdgeFirst Dataset) Files

   The Zip/Arrow files together creates the EdgeFirst Dataset Format.

   To create a snapshot from a Zip/Arrow files, visit the "Data Snapshots" page.

   

   Data Snapshots

   Either drag and drop the Zip/Arrow files into the page OR click on the "From File" button to allow selection of the Zip/Arrow files from your directories.

   

   Zip/Arrow Import Snapshot

   Uniform Names

   The name of corresponding Zip and Arrow files must be same.

   Zip/Arrow File pairs

   The upload must come into Zip and Arrow file pairs. If there are multiple Zip and Arrow pairs, then each pair will become a sequence.

A snapshot can be created from one of the three ways as described above: Create from Existing Dataset, Upload from MCAP File, Upload from Zip/Arrow File. The next section will show how to restore these created snapshots for auto-annotations.

Restore Snapshot

The snapshot restoration process involves several dataset transformations such as the frame rate specification, depth map generation, and auto-annotations. More information can be found in Studio.

COCO Annotations

The labels supported during the auto-annotation process for the Fully Automatic Ground Truth Generation are the COCO labels listed.

The created snapshots can be found under "Data Snapshots".

Data Snapshots

To restore the snapshot, click on the snapshot context menu and select "Restore".

Restore Snapshots

Restoring a snapshot will create a new dataset entirely with annotations. Specify the project to contain this new dataset and specify the name and the description of the dataset. Furthermore, toggle the "AI Ground Truth Generation" to auto annotate the dataset samples. The rest of the settings can be kept in their defaults for this tutorial. Click "Restore" to start the restoration process.

Restore Snapshots Fields

The snapshot restore process can be found under the project datasets.

Restore Snapshots Progress

Once completed, the dataset will now contain annotations that resulted from the auto-annotation process.

[Insert Image]

Next navigate to the gallery of the dataset by clicking on the gallery button. The figure below shows a side-by-side display of the annotations from frames 1-3. The 2D annotations for both segmentation masks and bounding boxes were auto-generated in the background.

Frame 1	Frame 2	Frame 3

The next section will describe the Semi-Automatic Ground Truth Generation for additional user control during the auto-annotation process.

Semi-Automatic Ground Truth Generation

This annotation feature is available after capturing the dataset into EdgeFirst Studio. This process occurs in the dataset gallery where the user has more control over the annotation process. This feature will preload all frames of a video sequence in the dataset into SAM-2 to generate segmentation masks, 2D bounding boxes, 3D bounding boxes (For Raivin/LiDAR Only) by tracking the object across the frames.

A complete description of this feature along with the functionalities of each buttons shown in this tutorial can be found under Studio.

This feature is a three-step process: Initialize AGTG Server, Annotate Starting Frame, Propagate.

Initialize AGTG Server

Inside the dataset gallery, click on the "AI Segment Tool" to start the AGTG server.

Auto Segment Mode

If there is currently no AGTG server available, go ahead and click on "Launch AGTG Server".

Launch AGTG Server

Please wait while the server is being initialized.

Launch AGTG Server

Active AGTG Server

This server is costing credits to run. An inactivity of 15 minutes will auto-terminate this server. Otherwise, once you have completed the annotations, please ensure to terminate the AGTG server to avoid spending more of your credits.

Annotate Starting Frame

Once the server has been initialized, annotate the starting frame. This is the only annotation required by the user. The rest of the frames will be annotated by SAM-2 and the AGTG process. This step is also known as initializing the SAM-2 state. Each object must be annotated independently so the tracker can assign a unique ID.

Start by drawing a bounding box for the first object by clicking and dragging. For multiple objects in the frame, click "+" to add a new object as shown in red below. The process for each object should be: Add a new object -> Draw object prompt.

By default, the prompts provided to SAM-2 are bounding boxes (mouse click and drag) which should cover the object to be annotated in the frame. However, points can also be provided (mouse clicks) by clicking areas that are part of the object.

AGTG Initial Prompts

Propagate

Once the first frame has been annotated (prompts for SAM-2), specify the "End Frame" which marks the point where SAM-2 stops propagating. By default this is the end of the sequence (last frame). Once this has been specified, click on "Propagate" to start the propagation process.

As the frames propagate, you should see the frames being auto-annotated. To stop the propagation process click on "Stop Propagation".

Propagation Process

Once the propagation completes as it reached the end frame, click on "Save Annotations" to save the generated annotations.

Propagation Completed

Video Tutorials

This is a high level video tutorial shown an MCAP recording restored as a snapshot.

Next Steps

If you notice any errors on the annotations that require adjustments or any missing annotations that needs to be added, follow these tutorials that describe the manual annotation features.