On Target Validation

This tutorial will show the steps for running validation on target. This type of validation is hosted as a user-managed validation session in EdgeFirst Studio. A user-managed validation session is hosted in an embedded platform for a proper assessment of the model performance and timings when deployed on target. In this tutorial, you will validate a Vision model that was trained through the end-to-end workflows or Training Vision.

Another type of validation is the On Cloud Validation which is hosted as a managed validation session in EdgeFirst Studio. A managed validation session creates an EC2 server to deploy the model for validation.

i.MX 95 Validation

To deploy TFLite models in the i.MX 95, first convert the models using eIQ's Neutron Converter prior to deploying the model in the platform. This modifies the model's architecture to allow deployment using the device's Neutron NPU delegate.

Specify Project Experiments

From the projects page, choose the project that contains the training session with the models you want to validate. In this example, the project chosen is the "Object Detection" project that was created in the Getting Started. Next click the "Model Experiments" button as indicated in red.

Model Experiments

Create Validation Session

In the experiment card, click the "Validate Sessions" button as indicated in red below.

Validate Sessions

You will be greeted to the "Validate Sessions" page as shown below.

Validate Sessions Page

Start a validation session by clicking on the "New Session" button on the top right corner of the page.

New Session Button

You will be greeted with a validation session dialog. In this dialog, check the "User Managed Validator" checkbox. Next specify the name of the validation session, the model to validate, and the dataset to deploy. In this example, the TFLite model will be validated and the "Coffee Cup" dataset with the validation partition will be used. Next specify, the validation parameters on the right. Additional information on these parameters are provided by hovering over the info button .

Validation Session Fields

Once the configurations have been made, go ahead and click on the "Start Session" button on the bottom right of the window. This will create the validation session to track validation progress that will run in the embedded platform.

Session Progress

Once the validation session has been created, SSH into the platform and install the following dependencies.

Virtual Environment

To avoid re-installation of existing system packages, we recommend setting up a Python virtual environment prior to running the pip installations below. Append --system-site-packages when creating the environment to include existing packages in the system. For example:

• Linux python3 -m venv /path/to/myvenv --system-site-packages
• Windows python -m venv /path/to/myenv --system-site-packages

Activate the environment via:

• Linux: source /path/to/myenv/bin/activate
• Windows: /path/to/myenv/Scripts/activate

pip install edgefirst-validator

Next login to your account in EdgeFirst Studio by using the EdgeFirst Client which comes installed with the validator package. The command below will prompt you to enter your EdgeFirst Studio credentials.

edgefirst-client login

Server Specification

Specify the EdgeFirst Studio server using --server among these variations: "test", "stage", "saas". This is an optional parameter as the default is set to "saas". To modify the studio server, first edgefirst-client logout then edgefirst-client --server <server> login.

EdgeFirst Studio Token

Once logged in, an EdgeFirst Studio Token will be saved under ~/.config/edgefirststudio/token granting access to the EdgeFirst Studio API which will remain valid for a period of time, usually 12 hours. Using this token will refresh the expiration timer.

Once the validator is installed and authenticated, run validation using the following command.

edgefirst-validator --session-id v-c1f

Note

Replace the session ID parameter specific to the validation session ID in your session.

If the model already exists in your system, you can run this command edgefirst-validator /path/to/mymodel.tflite --session-id v-c1f. Otherwise, the model will be downloaded as an artifact from the EdgeFirst Studio training session.

Once entered, the following validation progress should now be indicated in EdgeFirst Studio as shown below.

Validation Session

Completed Session

The completed session will look as follows with the status set to "Complete".

Completed Session

The attributes of the validation sessions in EdgeFirst Studio are labeled below.

Validation Session Attributes

Validation Metrics

Once the validation session completes, you can view the validation metrics by clicking the "View Validation Charts" button on the top of the session card.

Validation Charts

Info

See detection and segmentation metrics for further details.

You can go back to the validation session card by pressing the "Back" button as indicated in red below on the top left corner of the page.

Back to the Session Card

Comparing Metrics

It is also possible to compare validation metrics for multiple sessions. See Validation Sessions in the Model Experiments Dashboard.

Next Steps

Now that you have validated your Vision model, you can find examples for deploying your model in the EVK or the Maivin Platform. Furthermore, you can also find examples for running your ModelPack model or Ultralytics model in your PC.