Difference between revisions of "X-LINUX-AI - object detection using armNN TensorFlow Lite parser"

[unchecked revision] [quality revision]
m
m
 

This article explains how to experiment with armNN[1] applications for object detection based on the COCO SSD MobileNet v1 model using the TensorFlow Lite parser.

1 Description[edit]

The object detection[2] neural network model allows identification and localization of a known object within an image.

C/C++ armNN Lite object detection application using TensorFlow Lite parser

The application demonstrates demonstrate a computer vision use case for object detection where frames are grabbed from a camera input (/dev/videox) and analyzed by a neural network model interpreted by the armNN[1] framework.
A Gstreamer pipeline is used to stream camera frames (using v4l2src), to display a preview (using waylandsink) and to execute neural network inference (using appsink).
The result of the inference is displayed on the preview. The overlay is done using GtkWidget GTK widget with cairo.
This combination is quite simple and efficient in terms term of CPU overheadconsumption.

The model used with this application is the COCO SSD MobileNet v1 downloaded from downloaded from the object detection overview[2].

2 Installation[edit]

2.1 Install from the OpenSTLinux AI package repository[edit]

Warning.png The software package is provided AS IS, and by downloading it, you agree to be bound to the terms of the software license agreement (SLA). The detailed content licenses can be found here.

After having configured the AI OpenSTLinux package you can install X-LINUX-AI components for this application:

Board $> apt-get install armnn-tfl-cv-apps-object-detection-c++

Then And restart the demo launcher:

Board $> systemctl restart weston@root

2.2 Source code location[edit]

  • in the Openembedded OpenSTLinux Distributiondistribution:
<Distribution Package installation directory>/layers/meta-st/meta-st-stm32mpu-ai/recipes-samples/armnn-cv-apps/files/tfl-object-detection/src
  • on GitHub:
https://github.com/STMicroelectronics/meta-st-stm32mpu-ai/tree/dunfell/recipes-samples/armnn-cv-apps/files/tfl-object-detection/src

2.3 Re-generate the package from OpenSTLinux Distribution distribution (optional)[edit]

Using the Openembedded OpenSTLinux Distributiondistribution, you are able to rebuild the application.

Info.png If not already installed, the X-LINUX-AI OpenSTLinux Distribution distribution need to be installed by following this link


  • Set up the build environment:
PC $> cd <Distribution Package installation directory>
PC $> source layers/meta-st/scripts/envsetup.sh
  • Rebuild the application:
PC $> bitbake armnn-tfl-cv-apps-object-detection-c++ -c compile

The generated binary is available here:

<Distribution Package installation directory>/<build directory>/tmp-glibc/work/cortexa7t2hf-neon-vfpv4-ostl-linux-gnueabi/armnn-tfl-cv-apps-object-detection-c++/1.0-r0/armnn-tfl-cv-apps-object-detection-c++-1.0/tfl-object-detection/src

3 How to use the applicationit[edit]

3.1 Launching Launch application via the demo launcher[edit]

Launch cpp armnn tfl object detection.png

3.2 Executing Or execute with the command line[edit]

The objdetect_armnn_tfl_gst_gtk C/C++ application is located in the userfs partition:

/usr/local/demo-ai/computer-vision/armnn-tfl-object-detection/bin/objdetect_armnn_tfl_gst_gtk

It accepts the following input parameters:

Usage: ./objdetect_armnn_tfl_gst_gtk -m <model .tflite> -l <label .txt file>                                                          
                                                                                                                                      
-m --model_file <.tflite file path>:  .tflite model to be executed                                                                    
-l --label_file <label file path>:    name of file containing labels                                                                  
-b --backend <device>:                preferred backend device to run layers on by default. Possible choices: CpuAcc, CpuRef          
-i --image <directory path>:          image directory with image to be classified                                                     
-v --video_device <n>:                video device (default /dev/video0)                                                              
--crop:                               if set, the nn input image is cropped (with the expected nn aspect ratio) before being resized, 
                                      else the nn imput image is only resized to the nn input size (could cause picture deformation). 
--frame_width  <val>:                 width of the camera frame (default is 640)                                                      
--frame_height <val>:                 height of the camera frame (default is 480)                                                     
--framerate <val>:                    framerate of the camera (default is 15fps)                                                      
--input_mean <val>:                   model input mean (default is 127.5)                                                             
--input_std  <val>:                   model input standard deviation (default is 127.5)                                               
--help:                               show this help 

3.3 Testing with COCO SSD MobileNet V1[edit]

The model used for test is the detect.tflite downloaded from object detection overview[2]

Info.png The different objects the neural network is able to detect are listed in the labels.txt file located in the target:

/usr/local/demo-ai/computer-vision/models/coco_ssd_mobilenet/labels.txt


To ease launching of the application, two shell scripts are available:

  • launch object detection based on camera frame inputs
Board $> /usr/local/demo-ai/computer-vision/armnn-tfl-object-detection/bin/launch_bin_objdetect_armnn_tfl_coco_ssd_mobilenet.sh
  • launch object detection based on the pictures located in /usr/local/demo-ai/computer-vision/models/mobilenet/testdata directory
Board $> /usr/local/demo-ai/computer-vision/armnn-tfl-object-detection/bin/launch_bin_objdetect_armnn_tfl_coco_ssd_mobilenet_testdata.sh
Info.png Note that you need to populate the testdata directory with your own data sets.

The pictures are then randomly read from the testdata directory

4 References[edit]


This article explains how to experiment with {{Highlight|'''armNN'''<ref name=armnn_url>[https://developer.arm.com/ip-products/processors/machine-learning/arm-nn armNN]</ref>}} applications for object detection based on the COCO SSD MobileNet v1 model using the TensorFlow Lite parser.

==Description==
The '''object detection'''<ref name=tflite_obj_detect_url>[https://www.tensorflow.org/lite/models/object_detection/overview TFLite object detection overview]</ref> neural network model allows identification and localization of a known object within an image.

[[File: cpp_armnn_tfl_object_detection_application_screenshot.png|thumb|upright=2|center|link=|C/C++ armNN Lite object detection application using TensorFlow Lite parser]]

The application demonstratesdemonstrate a computer vision use case for object detection where frames are grabbed from a camera input (/dev/videox) and analyzed by a neural network model interpreted by the {{Highlight|'''armNN'''<ref name=armnn_url></ref>}} framework.<br>
A Gstreamer pipeline is used to stream camera frames (using v4l2src), to display a preview (using waylandsink) and to execute neural network inference (using appsink).<br>

The result of the inference is displayed on the preview. The overlay is done using GtkWidget GTK widget with cairo.<br>

This combination is quite simple and efficient in termsterm of CPU overheadconsumption.<br>


The model used with this application is the {{Highlight|COCO SSD MobileNet v1}} downloaded from downloaded from the '''object detection overview'''<ref name=tflite_obj_detect_url>[https://www.tensorflow.org/lite/models/object_detection/overview TFLite object detection overview]</ref>.

==Installation==
===Install from the OpenSTLinux AI package repository===
{{Warning|{{SoftwareLicenseAgreement | distribution=X-LINUX-AI}}}}
After having [[X-LINUX-AI_OpenSTLinux_Expansion_Package#Configure the AI OpenSTLinux package repository|configured the AI OpenSTLinux package]] you can install X-LINUX-AI components for this application:
 {{Board$}} apt-get install armnn-tfl-cv-apps-object-detection-c++ThenAnd restart the demo launcher:
 {{Board$}} systemctl restart weston@root

===Source code location===
* in the Openembedded OpenSTLinux Distributiondistribution:
:'''<Distribution Package installation directory>/layers/meta-st/meta-st-stm32mpu-ai/recipes-samples/armnn-cv-apps/files/tfl-object-detection/src'''
* on GitHub:
:https://github.com/STMicroelectronics/meta-st-stm32mpu-ai/tree/dunfell/recipes-samples/armnn-cv-apps/files/tfl-object-detection/src

===Re-generate the package from OpenSTLinux Distributiondistribution (optional)===
Using the Openembedded OpenSTLinux Distributiondistribution, you are able to rebuild the application.

{{Info|If not already installed, the X-LINUX-AI OpenSTLinux Distributiondistribution need to be installed by following this [[X-LINUX-AI_OpenSTLinux_Expansion_Package#Re-generate_X-LINUX-AI_OpenSTLinux_distribution|link]]}}

* Set up the build environment:
 {{PC$}} cd <Distribution Package installation directory>

 {{PC$}} source layers/meta-st/scripts/envsetup.sh
* Rebuild the application:
 {{PC$}} bitbake armnn-tfl-cv-apps-object-detection-c++ -c compile
The generated binary is available here:<Distribution Package installation directory>/<build directory>/tmp-glibc/work/cortexa7t2hf-neon-vfpv4-ostl-linux-gnueabi/armnn-tfl-cv-apps-object-detection-c++/1.0-r0/armnn-tfl-cv-apps-object-detection-c++-1.0/tfl-object-detection/src

==How to use the application==
===Launching it==
===Launch application via the demo launcher===
[[File: launch_cpp_armnn_tfl_object_detection.png|thumb|upright=2|center|link=]]

===Executing Or execute with the command line===
The '''objdetect_armnn_tfl_gst_gtk''' C/C++ application is located in the userfs partition:
 /usr/local/demo-ai/computer-vision/armnn-tfl-object-detection/bin/objdetect_armnn_tfl_gst_gtk
It accepts the following input parameters:
<pre>

Usage: ./objdetect_armnn_tfl_gst_gtk -m <model .tflite> -l <label .txt file>                                                          


-m --model_file <.tflite file path>:  .tflite model to be executed                                                                    
-l --label_file <label file path>:    name of file containing labels                                                                  
-b --backend <device>:                preferred backend device to run layers on by default. Possible choices: CpuAcc, CpuRef          
-i --image <directory path>:          image directory with image to be classified                                                     
-v --video_device <n>:                video device (default /dev/video0)                                                              
--crop:                               if set, the nn input image is cropped (with the expected nn aspect ratio) before being resized, 
                                      else the nn imput image is only resized to the nn input size (could cause picture deformation). 
--frame_width  <val>:                 width of the camera frame (default is 640)                                                      
--frame_height <val>:                 height of the camera frame (default is 480)                                                     
--framerate <val>:                    framerate of the camera (default is 15fps)                                                      
--input_mean <val>:                   model input mean (default is 127.5)                                                             
--input_std  <val>:                   model input standard deviation (default is 127.5)                                               
--help:                               show this help </pre>


===Testing with COCO SSD MobileNet V1===
The model used for test is the {{Highlight|detect.tflite}} downloaded from '''object detection overview'''<ref name=tflite_obj_detect_url>[https://www.tensorflow.org/lite/models/object_detection/overview TFLite object detection overview]</ref>


{{Info|The different objects the neural network is able to detect are listed in the labels.txt file located in the target:
/usr/local/demo-ai/computer-vision/models/coco_ssd_mobilenet/labels.txt}}

To ease launching of the application, two shell scripts are available:
* launch object detection based on camera frame inputs
 {{Board$}} /usr/local/demo-ai/computer-vision/armnn-tfl-object-detection/bin/launch_bin_objdetect_armnn_tfl_coco_ssd_mobilenet.sh

*launch object detection based on the pictures located in '''/usr/local/demo-ai/computer-vision/models/mobilenet/testdata''' directory
 {{Board$}} /usr/local/demo-ai/computer-vision/armnn-tfl-object-detection/bin/launch_bin_objdetect_armnn_tfl_coco_ssd_mobilenet_testdata.sh

{{Info|Note that you need to populate the testdata directory with your own data sets.<br>

The pictures are then randomly read from the testdata directory}}

==References==<references />

<noinclude>

[[Category:Artificial intelligence sample apps|35]]
{{PublicationRequestId | 16541 | 24Jun'20}}</noinclude>
Line 1: Line 1:
This article explains how to experiment {{Highlight|'''armNN'''<ref name=armnn_url>[https://developer.arm.com/ip-products/processors/machine-learning/arm-nn armNN]</ref>}} applications for object detection based on COCO SSD MobileNet v1 model using TensorFlow Lite parser.
+
This article explains how to experiment with {{Highlight|'''armNN'''<ref name=armnn_url>[https://developer.arm.com/ip-products/processors/machine-learning/arm-nn armNN]</ref>}} applications for object detection based on the COCO SSD MobileNet v1 model using the TensorFlow Lite parser.
   
 
==Description==
 
==Description==
Line 6: Line 6:
 
[[File: cpp_armnn_tfl_object_detection_application_screenshot.png|thumb|upright=2|center|link=|C/C++ armNN Lite object detection application using TensorFlow Lite parser]]
 
[[File: cpp_armnn_tfl_object_detection_application_screenshot.png|thumb|upright=2|center|link=|C/C++ armNN Lite object detection application using TensorFlow Lite parser]]
   
The application demonstrate a computer vision use case for object detection where frames are grabbed from a camera input (/dev/videox) and analyzed by a neural network model interpreted by the {{Highlight|'''armNN'''<ref name=armnn_url></ref>}} framework.<br>
+
The application demonstrates a computer vision use case for object detection where frames are grabbed from a camera input (/dev/videox) and analyzed by a neural network model interpreted by the {{Highlight|'''armNN'''<ref name=armnn_url></ref>}} framework.<br>
Gstreamer pipeline is used to stream camera frames (using v4l2src), to display a preview (using waylandsink) and to execute neural network inference (using appsink).<br>
+
A Gstreamer pipeline is used to stream camera frames (using v4l2src), to display a preview (using waylandsink) and to execute neural network inference (using appsink).<br>
The result of the inference is displayed on the preview. The overlay is done using GTK widget with cairo.<br>
+
The result of the inference is displayed on the preview. The overlay is done using GtkWidget with cairo.<br>
This combination is quite simple and efficient in term of CPU consumption.<br>
+
This combination is quite simple and efficient in terms of CPU overhead.<br>
   
The model used with this application is the {{Highlight|COCO SSD MobileNet v1}} downloaded from downloaded from the '''object detection overview'''<ref name=tflite_obj_detect_url>[https://www.tensorflow.org/lite/models/object_detection/overview TFLite object detection overview]</ref>.
+
The model used with this application is the {{Highlight|COCO SSD MobileNet v1}} downloaded from the '''object detection overview'''<ref name=tflite_obj_detect_url>[https://www.tensorflow.org/lite/models/object_detection/overview TFLite object detection overview]</ref>.
   
 
==Installation==
 
==Installation==
Line 18: Line 18:
 
After having [[X-LINUX-AI_OpenSTLinux_Expansion_Package#Configure the AI OpenSTLinux package repository|configured the AI OpenSTLinux package]] you can install X-LINUX-AI components for this application:
 
After having [[X-LINUX-AI_OpenSTLinux_Expansion_Package#Configure the AI OpenSTLinux package repository|configured the AI OpenSTLinux package]] you can install X-LINUX-AI components for this application:
 
  {{Board$}} apt-get install armnn-tfl-cv-apps-object-detection-c++
 
  {{Board$}} apt-get install armnn-tfl-cv-apps-object-detection-c++
And restart the demo launcher:
+
Then restart the demo launcher:
 
  {{Board$}} systemctl restart weston@root
 
  {{Board$}} systemctl restart weston@root
   
 
===Source code location===
 
===Source code location===
* in the Openembedded OpenSTLinux distribution:
+
* in the Openembedded OpenSTLinux Distribution:
 
:'''<Distribution Package installation directory>/layers/meta-st/meta-st-stm32mpu-ai/recipes-samples/armnn-cv-apps/files/tfl-object-detection/src'''
 
:'''<Distribution Package installation directory>/layers/meta-st/meta-st-stm32mpu-ai/recipes-samples/armnn-cv-apps/files/tfl-object-detection/src'''
 
* on GitHub:
 
* on GitHub:
 
:https://github.com/STMicroelectronics/meta-st-stm32mpu-ai/tree/dunfell/recipes-samples/armnn-cv-apps/files/tfl-object-detection/src
 
:https://github.com/STMicroelectronics/meta-st-stm32mpu-ai/tree/dunfell/recipes-samples/armnn-cv-apps/files/tfl-object-detection/src
   
===Re-generate the package from OpenSTLinux distribution (optional)===
+
===Re-generate the package from OpenSTLinux Distribution (optional)===
Using the Openembedded OpenSTLinux distribution, you are able to rebuild the application.
+
Using the Openembedded OpenSTLinux Distribution, you are able to rebuild the application.
   
{{Info|If not already installed, the X-LINUX-AI OpenSTLinux distribution need to be installed by following this [[X-LINUX-AI_OpenSTLinux_Expansion_Package#Re-generate_X-LINUX-AI_OpenSTLinux_distribution|link]]}}
+
{{Info|If not already installed, the X-LINUX-AI OpenSTLinux Distribution need to be installed by following this [[X-LINUX-AI_OpenSTLinux_Expansion_Package#Re-generate_X-LINUX-AI_OpenSTLinux_distribution|link]]}}
   
   
Line 41: Line 41:
 
  <Distribution Package installation directory>/<build directory>/tmp-glibc/work/cortexa7t2hf-neon-vfpv4-ostl-linux-gnueabi/armnn-tfl-cv-apps-object-detection-c++/1.0-r0/armnn-tfl-cv-apps-object-detection-c++-1.0/tfl-object-detection/src
 
  <Distribution Package installation directory>/<build directory>/tmp-glibc/work/cortexa7t2hf-neon-vfpv4-ostl-linux-gnueabi/armnn-tfl-cv-apps-object-detection-c++/1.0-r0/armnn-tfl-cv-apps-object-detection-c++-1.0/tfl-object-detection/src
   
==How to use it==
+
==How to use the application==
===Launch application via the demo launcher===
+
===Launching via the demo launcher===
 
[[File: launch_cpp_armnn_tfl_object_detection.png|thumb|upright=2|center|link=]]
 
[[File: launch_cpp_armnn_tfl_object_detection.png|thumb|upright=2|center|link=]]
   
===Or execute with command line===
+
===Executing with the command line===
 
The '''objdetect_armnn_tfl_gst_gtk''' C/C++ application is located in the userfs partition:
 
The '''objdetect_armnn_tfl_gst_gtk''' C/C++ application is located in the userfs partition:
 
  /usr/local/demo-ai/computer-vision/armnn-tfl-object-detection/bin/objdetect_armnn_tfl_gst_gtk
 
  /usr/local/demo-ai/computer-vision/armnn-tfl-object-detection/bin/objdetect_armnn_tfl_gst_gtk