Metadata-Version: 2.4
Name: studio-runner-py
Version: 0.0.0
Summary: Studio Runner Python
Requires-Python: >=3.8
Description-Content-Type: text/markdown
License-File: LICENSE
License-File: NOTICE
Requires-Dist: edgefirst-hal
Requires-Dist: polars
Requires-Dist: edgefirst-client
Requires-Dist: lmdb
Requires-Dist: tqdm
Requires-Dist: pillow
Requires-Dist: matplotlib
Requires-Dist: PyYAML
Provides-Extra: pc
Requires-Dist: onnxruntime-gpu; extra == "pc"
Requires-Dist: tensorflow; extra == "pc"
Requires-Dist: opencv-python; extra == "pc"
Provides-Extra: onnx
Requires-Dist: onnxruntime-gpu; extra == "onnx"
Provides-Extra: tflite
Requires-Dist: tensorflow; extra == "tflite"
Dynamic: license-file

# Studio Runner Python

This repository contains the Python implementation for demonstrating live model inference on camera feeds with real-time native display rendering.  It supports Windows PCs, NVIDIA Jetson boards, and NXP i.MX embedded systems using either OpenCV or GStreamer with PyCairo for display and inference acceleration.

## General Architecture

This application requires a monitor connected to the device for rendering the camera stream and model inference results using either OpenCV's native window display (`imshow`) or PyCairo. 

The application:

1. Captures video frames from the camera using [OpenCV-Python](https://docs.opencv.org/4.x/d0/de3/tutorial_py_intro.html) `VideoCapture` or [GStreamer](https://gstreamer.freedesktop.org/) `appsink`
2. Preprocesses images (resizing with letterbox/pad mode, normalization)
3. Runs model inference using either [TensorFlow (Keras)](https://www.tensorflow.org/guide/keras), [ONNX Runtime](https://onnxruntime.ai/), [TFLite Runtime](https://ai.google.dev/edge/litert), [TensorRT](https://developer.nvidia.com/tensorrt), [Kinara ARA-2](https://www.nxp.com/products/ARA240), or [Hailo](https://hailo.ai/products/hailo-software/model-explorer-vision/)
4. Postprocesses outputs (decoding, NMS for object detection) from the model embedded metadata
5. Renders annotated results on the display with bounding boxes and/or segmentation masks
6. Displays results in real-time with performance metrics

Note: TFLite runtime is pre-installed on NXP i.MX platforms; ONNX Runtime is available as an optional GPU-accelerated package. The Hardware Abstraction Layer (HAL) will provide future optimizations for preprocessing and postprocessing stages.

```mermaid
flowchart TD
    server[Start] --> camera[Read from Camera]
    camera --> preprocess[Preprocess Frame] --> inference[Run Model Inference]
    inference --> decode[Decode Model Outputs] --> postprocess[Run NMS]
    postprocess --> draw[Draw Outputs] --> render[Render Frame]
    render --> stop{CTRL-C?}
    stop -- Yes --> endServer[End]
    stop -- No --> camera    
```

If you are looking for examples in line with production setups, you can refer to the [EdgeFirst Perception Guide](https://doc.edgefirst.ai/latest/perception/) which builds applications using Zenoh.

## Installation

*This application can be run with Python>=3.8.0*.

1. Upgrade pip to the latest version `python -m pip install --upgrade pip`

2. It is recommended to first [create a python virtual environment](https://docs.python.org/3/library/venv.html#creating-virtual-environments). If you are on an embedded platform such as NXP's i.MX or NVIDIA Jetson platforms, create the virtual environment with `--system-site-packages` option enabled to include existing system packages such as `tflite_runtime` or `onnxruntime` which are needed for running the models. *Note: OpenCV should already come pre-installed in the device's BSP.*

3. Install the application via pip **on target** `pip install studio-runner-py` or **on PC** `pip install studio-runner-py[pc]`

4. Install edgefirst-validator without any dependencies `pip install edgefirst-validator --no-deps`

Otherwise, if the repository is cloned, execute the following commands.

```shell
pip install -r requirements.txt
pip install edgefirst-validator --no-deps
pip install onnxruntime-gpu tensorflow opencv-python
```

## Usage

The table below shows a check list of supported platforms. For example, on the PC running the ONNX model is the only model supported. TFLite has been observed to have high inference times on the CPU which makes its usage impractical on the PC. 

| Platform            | ONNX | TFLite | In Development |
|---------------------|------|--------|----------------|
| PC / Linux          |  ✓   |        |                |
| Mac/MacOS           |      |        |        ✓       |
| i.MX 8M Plus EVK    |  ✓   |   ✓    |                |
| NVIDIA Orin         |  ✓   |        |                |
| Kinara ARA-2        |      |        |        ✓       |
| Raivin Radar Fusion |      |        |        ✓       |
| i.MX 95 EVK         |      |        |        ✓       |

*Note: Low power devices such as the **i.MX 8M Plus** should deploy quantized [TFLite](https://doc.edgefirst.ai/latest/models/modelpack/quantize/#onnx-to-tflite) and ONNX models to take advantage of NPU performance.*

*Note: For the **NVIDIA ORIN**, power savings are enabled by default which results in slower model performance. You can disable any power savings and allow maximum performance by running the following commands.*

```shell
sudo nvpmodel -m 0
sudo jetson_clocks
```

### TFLite

1. If you have a model trained in EdgeFirst Studio, install the model using [edgefirst-client](https://doc.edgefirst.ai/latest/perception/studio/).  However, you will need to first login via `edgefirst-client login` using your [EdgeFirst Studio](https://edgefirst.studio/login) credentials.

```shell
edgefirst-client download-artifact <training session ID> <model name>
```

```shell
edgefirst-client download-artifact t-123 model.tflite
```

2. If you have a model in the system, pass the path to the model.  The [`labels.txt` and `edgefirst.yaml` files](https://doc.edgefirst.ai/latest/models/metadata/#reading-metadata) need to be embedded inside the model.

```shell
studio-runner-py /path/to/model.tflite
```

```shell
python -m studio.runner /path/to/model.tflite
```

### ONNX

1. If you have a model trained in EdgeFirst Studio, install the following model artifacts using `edgefirst-client download-artifact`.  However, you will need to first login via `edgefirst-client login`.

```shell
edgefirst-client download-artifact t-123 model.onnx
```

```shell
edgefirst-client download-artifact t-123 labels.txt
```

```shell
edgefirst-client download-artifact t-123 edgefirst.yaml
```

3. If you have a model in the system, pass the path to the model.  If the model does not have embedded metadata, you can specify the path to the "edgefirst.yaml" and "labels.txt". 

```shell
studio-runner-py /path/to/model.onnx --config /path/to/edgefirst.yaml --labels /path/to/labels.txt
```

```shell
python -m studio.runner /path/to/model.onnx --config /path/to/edgefirst.yaml --labels /path/to/labels.txt
```

## Examples

| Application |
|-------------|
| Setup Application<br><img src="docs/assets/command-line.jpg"/> |

### Windows PC 🖥️

![Windows PC Demo](docs/assets/WindowsPC.gif)

### NVIDIA Jetson Orin <img src="docs/assets/NVIDIA_ORIN.jpg" width="60" style="vertical-align:middle; margin-right:5px;">

![NVIDIA ORIN Demo](docs/assets/NVIDIA_ORIN.gif)

### NXP i.MX 8M Plus <img src="docs/assets/imx8mpevk.png" width="60" style="vertical-align:middle; margin-right:5px;">

![i.MX 8M Plus Demo](docs/assets/i.MX8MPlus.gif)
