This article explains how to use a OpenVX [1], TensorFlow Lite[2] or ONNX [3] model in applications for image classification based on MobileNetV1 and MobileNetV2 neural network using the stai_mpu API on our STM32MPUs serie.
1. Description[edit source]
The image classification neural network model allows identification of the subject represented by an image. It classifies an image into various classes.
The application demonstrates a computer vision use case for image classification where frames are grabbed from a camera input (/dev/videox) and analyzed by a neural network model interpreted by OpenVX, TFLite or ONNX framework.
A Gstreamer pipeline is used to stream camera frames (using v4l2src), to display a preview (using gtkwaylandsink) and to execute neural network inference (using appsink).
The result of the inference is displayed in the preview. The overlay is done using GtkWidget with cairo.
The models used with this application are the MobileNet v1 downloaded from the TensorFlow Lite Hub[4] and the MobileNet v2 downloaded from the ST model zoo[5]
2. Installation[edit source]
2.1. Install from the OpenSTLinux AI package repository[edit source]
After having configured the AI OpenSTLinux package you can install X-LINUX-AI components for image classification application:
2.1.1. On STM32MP2x board[edit source]
The OpenVX application will be installed to take advantage of the neural processing unit (NPU) and graphics processing unit (GPU).
- To install this application, please use the following command:
x-linux-ai -i stai-mpu-image-classification-cpp-ovx
- Then, restart the demo launcher:
systemctl restart weston-graphical-session.service
2.1.2. On STM32MP1x board[edit source]
The TFLite application will be installed with the XNNPACK delegate to accelerate the neural network inference on the CPU. For more information, please expand this section.
- To install this application, please use the following command:
x-linux-ai -i stai-mpu-image-classification-cpp-tflite
- Then, restart the demo launcher:
systemctl restart weston-graphical-session.service
2.2. Source code location[edit source]
- in the OpenSTLinux Distribution with X-LINUX-AI Expansion Package:
- <Distribution Package installation directory>/layers/meta-st/meta-st-x-linux-ai/recipes-samples/image-classification/files/stai_mpu
- on GitHub:
2.3. Regenerate the package from OpenSTLinux Distribution (optional)[edit source]
Using the OpenSTLinux Distribution, you are able to rebuild the application.
- Set up the build environment:
cd <Distribution Package installation directory> source layers/meta-st/scripts/envsetup.shBe careful to select the right board in the script to setup the correct environment
- Rebuild the application on STM32MP2x:
bitbake stai-mpu-image-classification-cpp-ovx -c compile
The generated binary is available here:
<Distribution Package installation directory>/<build directory>/tmp-glibc/work/cortexa35-ostl-linux/stai-mpu-image-classification-cpp/5.0.0-r0/stai-mpu-image-classification-cpp-5.0.0/stai_mpu
- Rebuild the application on STM32MP1x:
For more information about building the application for STM32MP1x, please expand this section.
bitbake stai-mpu-image-classification-cpp-tflite -c compile
The generated binary is available here:
<Distribution Package installation directory>/<build directory>/tmp-glibc/work/cortexa7t2hf-neon-vfpv4-ostl-linux-gnueabi/stai-mpu-image-classification-cpp/5.0.0-r0/stai-mpu-image-classification-cpp-5.0.0/stai_mpu