This article explains how to experiment with ONNX Runtime [1] applications for object detection based on the COCO SSD MobileNet v1 model using ONNX C++ API runtime.
1. Description[edit source]
The object detection' neural network model allows the identification and localization of a known object within an image.
The application demonstrates a computer vision use case for object detection where frames are grabbed from a camera input (/dev/videox) and analyzed by a neural network model interpreted by the ONNX Runtime [1] framework.
A Gstreamer pipeline is used to stream camera frames (using v4l2src), to display a preview (using waylandsink) and to execute a neural network inference (using appsink).
The result of the inference is displayed on the preview. The overlay is done using GtkWidget with cairo.
This combination is quite simple and efficient in terms of CPU overhead.
The performance depends on the number of CPUs available. The camera preview is limited to one CPU core while the ONNX runtime[1] interpreter is configured to use the maximum of the available resources.
The model used with this application is the COCO SSD MobileNet v1 downloaded from the Tensorflow Lite Hub[2] and converted to ONNX opset 16 format using tf2onnx.
1.1. Convert a Tensorflow™ Lite model to ONNX using tf2onnx[edit source]
To convert a .tflite model to the ONNX format, ONNX provides a tool named tf2onnx [3], which is very simple to use.
The first step is to install Tensorflow on the host computer. For test purposes it might be useful to install also ONNX Runtime. The tool tf2onnx uses the versions of Tensorflow and ONNX Runtime already installed. If it does not find any, it installs the most recent versions.
The second step is to install tf2onnx:
- Install from pypi :
pip install -U tf2onnx
or
- Install latest from GitHub:
pip install git+https://github.com/onnx/tensorflow-onnx
After the installation, the user is able to convert the .tflite model directly using the following command line:
python -m tf2onnx.convert --opset 16 --tflite path/to/tflite/model.tflite --output path/to/onnx/model/model.onnx
Native ONNX models are also available in the ONNX Model Zoo [4].
2. Installation[edit source]
2.1. Install from the OpenSTLinux AI package repository[edit source]
After having configured the AI OpenSTLinux package, the user can install the X-LINUX-AI components for this application:
apt-get install onnx-cv-apps-object-detection-c++
Then, the user can restart the demo launcher:
- For OpenSTLinux distribution with a version lower than 4.0 use:
systemctl restart weston@root
- For other OpenSTLinux distribution use:
systemctl restart weston-launch
2.2. Source code location[edit source]
- in the Openembedded OpenSTLinux Distribution with the X-LINUX-AI Expansion Package:
- <Distribution Package installation directory>/layers/meta-st/meta-st-stm32mpu-ai/recipes-samples/onnx-cv-apps/files/object-detection/src
- on GitHub:
2.3. Re-generate the package from OpenSTLinux Distribution (optional)[edit source]
Using the Openembedded OpenSTLinux distribution, you are able to rebuild the application.
- Set up the build environment:
cd <Distribution Package installation directory> source layers/meta-st/scripts/envsetup.sh
- Rebuild the application:
bitbake onnx-cv-apps-object-detection-c++ -c compile
The generated binary is available here:
<Distribution Package installation directory>/<build directory>/tmp-glibc/work/cortexa7t2hf-neon-vfpv4-ostl-linux-gnueabi/onnx-cv-apps-object-detection-c++/3.0.0-r0/onnx-cv-apps-object-detection-c++-3.0.0/object-detection/src/
3. How to use the application[edit source]
3.1. Launching via the demo launcher[edit source]
3.2. Executing with the command line[edit source]
The Python script objdetect_onnx.py application is located in the userfs partition:
/usr/local/demo-ai/computer-vision/onnx-object-detection/python/objdetect_onnx.py
It accepts the following input parameters:
usage: objdetect_onnx.py[-h] [-i IMAGE] [-v VIDEO_DEVICE] [--frame_width FRAME_WIDTH] [--frame_height FRAME_HEIGHT] [--framerate FRAMERATE] [-m MODEL_FILE] [-l LABEL_FILE]
[--input_mean INPUT_MEAN] [--input_std INPUT_STD] [--validation] [--num_threads NUM_THREADS]
[--maximum_detection MAXIMUM_DETECTION] [--threshold THRESHOLD]
options:
-h, --help show this help message and exit
-i IMAGE, --image IMAGE
image directory with image to be classified
-v VIDEO_DEVICE, --video_device VIDEO_DEVICE
video device (default /dev/video0)
--frame_width FRAME_WIDTH
width of the camera frame (default is 640)
--frame_height FRAME_HEIGHT
height of the camera frame (default is 480)
--framerate FRAMERATE
framerate of the camera (default is 15fps)
-m MODEL_FILE, --model_file MODEL_FILE
.onnx model to be executed
-l LABEL_FILE, --label_file LABEL_FILE
name of file containing labels
--input_mean INPUT_MEAN
input mean
--input_std INPUT_STD
input standard deviation
--validation enable the validation mode
--num_threads NUM_THREADS
Select the number of threads used by ONNX interpreter to run inference
--maximum_detection MAXIMUM_DETECTION
Adjust the maximum number of object detected in a frame accordingly to your NN model (default is 10)
--threshold THRESHOLD
threshold of accuracy above which the boxes are displayed (default 0.62)
3.3. Testing with COCO ssd MobileNet v1[edit source]
The model used for test is the detect.onnx downloaded from Tensorflow Lite Hub[2] and converted to ONNX format.
To launch the Python script more easily, two shell scripts are available:
- launch object detection based on camera frame inputs
/usr/local/demo-ai/computer-vision/onnx-object-detection/python/launch_python_objdetect_onnx_coco_ssd_mobilenet.sh
- launch object detection based on the pictures located in /usr/local/demo-ai/computer-vision/models/coco_ssd_mobilenet/testdata directory:
/usr/local/demo-ai/computer-vision/onnx-object-detection/python/launch_python_objdetect_onnx_coco_ssd_mobilenet_testdata.sh
4. References[edit source]