Last edited 2 weeks ago

Object detection

Applicable for STM32MP13x lines, STM32MP15x lines, STM32MP25x lines

This article explains how to use the stai_mpu API for object detection applications supporting OpenVX [1], TensorFlow Lite[2], ONNX [3] or Coral [4] back-ends.

1. Description[edit source]

The object detection neural network model allows identification and localization of a known object within an image.

Object detection application

The application demonstrates a computer vision use case for object detection where frames are grabbed from a camera input (/dev/videox) and analyzed by a neural network model interpreted by OpenVX, TFLite or ONNX framework.
A Gstreamer pipeline is used to stream camera frames (using v4l2src), to display a preview (using gtkwaylandsink) and to execute neural network inference (using appsink).
The result of the inference is displayed in the preview. The overlay is done using GtkWidget with cairo.

The models used with this application are the SSD MobileNet V1 downloaded from the TensorFlow Lite Hub[5] and the SSD MobileNet V2 FPNLite downloaded from the ST model zoo[6]

Info white.png Information
On STM32MP2 series' boards More info green.png the SSD MobileNet V2 FPNLite used for this application, is a TensorFlow Lite per-tensor asymmetric quantized model, which is accelerated using the neural processing unit (NPU). Then, the model has been converted to NBG format using ST Edge AI tool. For more information about this tool, refer to the dedicated article.

2. Installation[edit source]

2.1. Install from the OpenSTLinux AI package repository[edit source]

Warning white.png Warning
The software package is provided AS IS, and by downloading it, you agree to be bound to the terms of the software license agreement (SLA0048). The detailed content licenses can be found here.

After having configured the AI OpenSTLinux package you can install X-LINUX-AI components for object detection application:

2.1.1. Install on STM32MP2x board[edit source]

The OpenVX application will be installed to take advantage of the neural processing unit (NPU) and graphics processing unit (GPU).

  • To install this application, please use the following command:
 x-linux-ai -i stai-mpu-object-detection-cpp-ovx
Warning DB.png Important
You can install the Python version of this application by installing this package: stai-mpu-object-detection-python-ovx


  • Then, restart the demo launcher:
 systemctl restart weston-graphical-session.service

2.1.2. Install on STM32MP1x board[edit source]

The TFLite application will be installed with the XNNPACK delegate to accelerate the neural network inference on the CPU.


  • To install this application, please use the following command:
 x-linux-ai -i stai-mpu-object-detection-cpp-tflite
Warning DB.png Important
You can install the Python version of this application by installing this package: stai-mpu-object-detection-python-tflite


  • Then, restart the demo launcher:
 systemctl restart weston-graphical-session.service

2.2. Source code location[edit source]

  • in the OpenSTLinux Distribution with X-LINUX-AI Expansion Package:
<Distribution Package installation directory>/layers/meta-st/meta-st-x-linux-ai/recipes-samples/object-detection/files/stai_mpu
  • on GitHub:
recipes-samples/object-detection/files/stai_mpu

2.3. Regenerate the package from OpenSTLinux Distribution (optional)[edit source]

Using the OpenSTLinux Distribution, you are able to rebuild the application.

Info white.png Information
If not already installed, the X-LINUX-AI OpenSTLinux Distribution need to be installed by following this link


  • Set up the build environment:
 cd <Distribution Package installation directory>
 source layers/meta-st/scripts/envsetup.sh
Warning.png Be careful to select the right board in the script to setup the correct environment
  • Rebuild the application on STM32MP2x:
 bitbake stai-mpu-object-detection-cpp-ovx -c compile

The generated binary is available here:

<Distribution Package installation directory>/<build directory>/tmp-glibc/work/cortexa35-ostl-linux/stai-mpu-object-detection-cpp/5.0.0-r0/stai-mpu-object-detection-cpp-5.0.0/stai_mpu
Warning DB.png Important
You can generate the Python version of this application: stai-mpu-object-detection-python-ovx


  • Rebuild the application on STM32MP1x:

For more information about building the application for STM32MP1x, please expand this section.

 bitbake stai-mpu-object-detection-cpp-tflite -c compile

The generated binary is available here:

<Distribution Package installation directory>/<build directory>/tmp-glibc/work/cortexa7t2hf-neon-vfpv4-ostl-linux-gnueabi/stai-mpu-object-detection-cpp/5.0.0-r0/stai-mpu-object-detection-cpp-5.0.0/stai_mpu
Warning DB.png Important
You can generate the Python version of this application: stai-mpu-object-detection-python-tflite

3. How to use the application[edit source]

3.1. Launching via the demo launcher[edit source]

You can click on the icon to run either the C/C++ or Python application. By default the OpenVX applications will be installed on STM32MP2x, and the TFLite applications will be installed on STM32MP1x.

Demo launcher

3.2. Executing with the command line[edit source]

The object detection C++ & Python applications are located in the userfs partition:

/usr/local/x-linux-ai/object-detection/stai_mpu_object_detection
/usr/local/x-linux-ai/object-detection/stai_mpu_object_detection.py

It accepts the following input parameters:

  • In C/C++ application:
 
Usage: stai_mpu_object_detection -m <model .tflite> -l <label .txt file>

-m --model_file <.tflite file path>:  .tflite model to be executed
-l --label_file <label file path>:    name of file containing labels
-i --image <directory path>:          image directory with image to be classified
-v --video_device <n>:                video device is automatically detected but can be set (example video0)
--frame_width  <val>:                 width of the camera frame (default is 640)
--frame_height <val>:                 height of the camera frame (default is 480)
--framerate <val>:                    framerate of the camera (default is 15fps)
--input_mean <val>:                   model input mean (default is 127.5)
--input_std  <val>:                   model input standard deviation (default is 127.5)
--verbose:                            enable verbose mode
--validation:                         enable the validation mode
--val_run:                            set the number of draws in the validation mode
--conf_threshold <val>:               confidence_thresh of accuracy above which the boxes are displayed (default 0.70)
--iou_threshold <val>:                threshold of intersection over union above which the boxes are displayed (default 0.45)
--dual_camera_pipeline                Use dual camera post-processing pipeline, one dedicated for the NN and the other dedicated for the display 
--help:                               show this help
  • In Python application:
usage: stai_mpu_object_detection.py [-h] [-i IMAGE] [-v VIDEO_DEVICE]
                                    [--frame_width FRAME_WIDTH] [--frame_height FRAME_HEIGHT]
                                    [--framerate FRAMERATE] [-m MODEL_FILE] [-l LABEL_FILE]
                                    [--input_mean INPUT_MEAN] [--input_std INPUT_STD]
                                    [--validation] [--val_run VAL_RUN]
                                    [--num_threads NUM_THREADS]
                                    [--conf_threshold CONF_THRESHOLD]
                                    [--iou_threshold IOU_THRESHOLD]

options:
  -h, --help            show this help message and exit
  -i IMAGE, --image IMAGE
                        image directory with image to be classified
  -v VIDEO_DEVICE, --video_device VIDEO_DEVICE
                        video device ex: video0
  --frame_width FRAME_WIDTH
                        width of the camera frame (default is 320)
  --frame_height FRAME_HEIGHT
                        height of the camera frame (default is 240)
  --framerate FRAMERATE
                        framerate of the camera (default is 15fps)
  -m MODEL_FILE, --model_file MODEL_FILE
                        .tflite model to be executed
  -l LABEL_FILE, --label_file LABEL_FILE
                        name of file containing labels
  --input_mean INPUT_MEAN
                        input mean
  --input_std INPUT_STD
                        input standard deviation
  --validation          enable the validation mode
  --val_run VAL_RUN     set the number of draws in the validation mode
  --num_threads NUM_THREADS
                        Select the number of threads used by tflite interpreter to run inference
  --conf_threshold CONF_THRESHOLD
                        threshold of accuracy above which the boxes are displayed (default 0.70)
  --iou_threshold IOU_THRESHOLD
                        threshold of intersection over union above which the boxes are displayed
                        (default 0.45)

4. Testing withSSD MobileNet[edit source]

4.1. Test with SSD MobileNetV2 on STM32MP2x[edit source]

The model used for testing is the ssd_mobilenet_v2_fpnlite_10_256_int8_per_tensor.nb

Info white.png Information
The different objects the neural network is able to classify are listed in the labels.txt file located in the target:

/usr/local/x-linux-ai/object-detection/models/coco_ssd_mobilenet/labels_coco_dataset_80.txt

To ease launching of the application, two shell scripts are available for both C/C++ and Python applications on the board:

  • For C/C++ application:
    • launch object detection based on camera frame inputs:
    /usr/local/x-linux-ai/object-detection/launch_bin_object_detection.sh
    • launch object detection based on the pictures located in /usr/local/demo-ai/object-detection/models/coco_ssd_mobilenet/testdata directory
    /usr/local/x-linux-ai/object-detection/launch_bin_object_detection_testdata.sh
  • For Python application:
    • launch object detection based on camera frame inputs:
    /usr/local/x-linux-ai/object-detection/launch_python_object_detection.sh
    • launch object detection based on the pictures located in /usr/local/demo-ai/object-detection/models/coco_ssd_mobilenet/testdata directory
    /usr/local/x-linux-ai/object-detection/launch_python_object_detection_testdata.sh
Warning DB.png Important
Note that you need to populate the testdata directory with your own data sets. The pictures are then randomly read from the testdata directory

4.2. Test with SSD MobileNetV1 on STM32MP1x[edit source]

The model used for testing is the ssd_mobilenet_v1_10_300_int8.tflite downloaded from Tensorflow Lite Hub[5].

Info white.png Information
The different objects the neural network is able to classify are listed in the labels.txt file located in the target:

/usr/local/x-linux-ai/object-detection/models/coco_ssd_mobilenet/labels_coco_dataset.txt

To ease launching of the application, two shell scripts are available for both C/C++ and Python applications on the board:

  • For C/C++ application:
    • launch object detection based on camera frame inputs:
    /usr/local/x-linux-ai/object-detection/launch_bin_object_detection.sh
    • launch object detection based on the pictures located in /usr/local/demo-ai/object-detection/models/coco_ssd_mobilenet/testdata directory
    /usr/local/x-linux-ai/object-detection/launch_bin_object_detection_testdata.sh
  • For Python application:
    • launch object detection based on camera frame inputs:
    /usr/local/x-linux-ai/object-detection/launch_python_object_detection.sh
    • launch object detection based on the pictures located in /usr/local/demo-ai/object-detection/models/coco_ssd_mobilenet/testdata directory
    /usr/local/x-linux-ai/object-detection/launch_python_object_detection_testdata.sh
Warning DB.png Important
Note that you need to populate the testdata directory with your own data sets. The pictures are then randomly read from the testdata directory

5. Going further[edit source]

5.1. Install all object detection packages[edit source]

Warning white.png Warning
The software package is provided AS IS, and by downloading it, you agree to be bound to the terms of the software license agreement (SLA0048). The detailed content licenses can be found here.

After having configured the AI OpenSTLinux package you can install directly all the X-LINUX-AI packages related to object detection by using the following command::

 x-linux-ai -i stai-mpu-object-detection-*

Then restart the demo launcher:

 systemctl restart weston-graphical-session.service

5.2. Launching via the demo launcher[edit source]

Now, it's possible to launch all the object detection application on your board by clicking on it in the demo launcher. The Python & C/C++ application are now available using TFLite, ONNX and Coral backend. Moreover, on STM32MP2x the OpenVX backend is also available.

Demo launcher

5.3. Launching via the command line interface (CLI)[edit source]

5.3.1. CLI on STM32MP2x[edit source]

The model used for testing is the SSD MobilenetV2 FPNLite.

Info white.png Information
The different objects the neural network is able to classify are listed in the labels.txt file located in the target:

/usr/local/x-linux-ai/object-detection/models/coco_ssd_mobilenet/labels_coco_dataset_80.txt

The two shell scripts described before offers the possibility to select the framework directly between OpenVX, TFlite, ONNX and Coral. To be able to run the application using all supported frameworks, the models for each frameworks must be available in the /usr/local/x-linux-ai/object-detection/models/coco_ssd_mobilenet/ directory. Then, you will need to specify the framework as an argument of the launch scripts as follow.

  • For C/C++ application:
    • Run object detection based on camera input with the chosen framework. Available framework options are: nbg, tflite, onnx, coral.
    /usr/local/x-linux-ai/object-detection/launch_bin_object_detection.sh nbg
    • Run object detection based on picture located in the /usr/local/demo-ai/object-detection/models/coco_ssd_mobilenet/testdata directory with the chosen framework. Available framework options are: nbg, tflite, onnx, coral.
    /usr/local/x-linux-ai/object-detection/launch_bin_object_detection_testdata.sh nbg
  • For Python application:
    • Run object detection based on camera input with the chosen framework. Available framework options are: nbg, tflite, onnx, coral.
    /usr/local/x-linux-ai/object-detection/launch_python_object_detection.sh nbg
    • Run object detection based on picture located in the /usr/local/demo-ai/object-detection/models/coco_ssd_mobilenet/testdata directory with the chosen framework. Available framework options are: nbg, tflite, onnx, coral.
    /usr/local/x-linux-ai/object-detection/launch_python_object_detection_testdata.sh nbg
Warning DB.png Important
Note that you need to populate the testdata directory with your own data sets. The pictures are then randomly read from the testdata directory

5.3.2. CLI on STM32MP1x[edit source]

The model used for testing is the SSD MobilenetV1.

Info white.png Information
The different objects the neural network is able to classify are listed in the labels.txt file located in the target:

/usr/local/x-linux-ai/object-detection/models/coco_ssd_mobilenet/labels_coco_dataset.txt

The two shell scripts described before offers the possibility to select the framework directly between TFlite, ONNX and Coral. To be able to run the application using all supported frameworks, the models for each frameworks must be available in the /usr/local/x-linux-ai/object-detection/models/coco_ssd_mobilenet/ directory. Then, you will need to specify the framework as an argument of the launch scripts as follow.

  • For C/C++ application:
    • Run object detection based on camera input with the chosen framework. Available framework options are: tflite, onnx, coral.
    /usr/local/x-linux-ai/object-detection/launch_bin_object_detection.sh tflite
    • Run object detection based on picture located in the /usr/local/demo-ai/object-detection/models/coco_ssd_mobilenet/testdata directory with the chosen framework. Available framework options are: tflite, onnx, coral.
    /usr/local/x-linux-ai/object-detection/launch_bin_object_detection_testdata.sh tflite
  • For Python application:
    • Run object detection based on camera input with the chosen framework. Available framework options are: tflite, onnx, coral.
    /usr/local/x-linux-ai/object-detection/launch_python_object_detection.sh tflite
    • Run object detection based on picture located in the /usr/local/demo-ai/object-detection/models/coco_ssd_mobilenet/testdata directory with the chosen framework. Available framework options are: tflite, onnx, coral.
    /usr/local/x-linux-ai/object-detection/launch_python_object_detection_testdata.sh tflite

6. References[edit source]