Difference between revisions of "X-LINUX-AI - face recognition using TensorFlow Lite C++ API"

[quality revision] [quality revision]
m
m
 

This article explains how to get started on the face recognition application.

Warning.png The face recognition binary is available on demand. Please contact the local STMicroelectronics support for more information about this application or send a request to edge.ai@st.com

1 Description[edit]

The face recognition application is capable of recognizing faces of known (enrolled) users.

C/C++ TensorFlow™ Lite face recognition application


The application demonstrates a computer vision use case for face recognition

  • In camera mode, frames are grabbed from a camera input (/dev/videox) and processed by two neural network models (face detection and face recognition) interpreted by the TensorFlow™ Lite[1] framework. A GStreamer pipeline is used to stream camera frames (using v4l2src), to display a preview (using waylandsink), and to execute neural network inference (using appsink).
  • In file system mode, pictures are read from the file system. In that case Gstreamer pipeline is replaced by OpenCV functions.

The result of the inference is displayed using overlay generated by GtkWidgets with Cairo.
This combination is quite simple and efficient in terms of CPU overhead.


Face recognition pipeline

1.1 Frame processing flow[edit]

The figure below shows the different frame processing stages involved in the face recognition application:

Data flow for the face recognition application.

1.1.1 Camera frame capture[edit]

The camera frames are captured by the GStreamer pipeline in a YUV422 VGA (640 x 480) resolution (this is the default resolution of the application).

1.1.2 Frame preprocessing[edit]

The main preprocessing stage is the pixel color format conversion from YUV422 captured frames into RGB888 frames.

1.1.3 Face detection[edit]

The face detection block is in charge of finding the faces present in the input frame (VGA, RGB888). The output of this block is a frame of resolution 96 x 96 that contains the faces found in the input captured frame.

1.1.4 Face recognition[edit]

The face Recognition block is in charge of extracting features from the face and computing a signature (embedding vector) corresponding to the input face.

1.1.5 Face identification[edit]

The face identification block is in charge of computing the distance between:

  • the vector produced by the face recognition block, and
  • each of the vectors stored in memory (and corresponding to the enrolled faces)

The output face identification block generates the two following outputs:

  • a User Face ID corresponding to the minimum distance
  • a similarity score

2 Installation[edit]

Info.png The face recognition binary is available on demand. Please contact the local STMicroelectronics support for more information about this application or send a request to edge.ai@st.com

2.1 copy the face recognition binary[edit]

PC $> scp path/to/the/deb/package/tflite-cv-apps-face-recognition-c++_1.21.0-r0.0_armhf.deb root@<board_ip_address>:/home/root

2.2 deploy the face recognition binary[edit]

Warning.png The software package is provided AS IS, and by downloading it, you agree to be bound to the terms of the software license agreement (SLA). The detailed content licenses can be found here.

After having configured the AI OpenSTLinux package, you can deploy the application that will need to download dependencies:

Board $> cd /home/root
Board $> apt-get install ./tflite-cv-apps-face-recognition-c++_1.21.0-r0.0_armhf.deb

Then restart the demo launcher:

Board $> systemctl restart weston@root

3 How to use the application[edit]

3.1 Launching via the demo launcher[edit]

Demo launcher

3.2 Executing with the command line[edit]

The facereco_tfl_gst_gtk C/C++ application is located in the userfs partition:

/usr/local/demo-ai/computer-vision/tflite-face-recognition/bin/facereco_tfl_gst_gtk

It accepts the following input parameters:

 
Usage: ./facereco_tfl_gst_gtk [option]

--reco_simultaneous_faces:       enable multiple face recognition (default is single face recognition)
--reco_threshold <val>:          face recognition threshold for face similarity (default is 0.70 = 70%)
--max_db_faces <val>:            maximum number of faces to be stored in the database (default is 200)
-d --database <directory path>:  provide the directory where the face recognition database is stored (else default location is used)
-i --image <directory path>:     image directory with images to be classified
-v --video_device <n>:           video device (default /dev/video0)
--frame_width  <val>:            width of the camera frame (default is 640 pixels)
--frame_height <val>:            height of the camera frame (default is 480 pixels)
--framerate <val>:               frame rate of the camera (default is 15 fps)
--validation:                    enable the validation mode
--help:                          show this help


  • launch face recognition based on the pictures located in /usr/local/demo-ai/computer-vision/tflite-face-recognition/testdata directory
Board $> /usr/local/demo-ai/computer-vision/tflite-face-recognition/bin/launch_bin_facereco_tfl_model_testdata.sh
Info.png Note that you need to populate the testdata directory with your own datasets.

The pictures are then randomly read from the testdata directory

3.3 In practice[edit]

As soon as a face is detected within the camera captured frame, a rectangle box is drawn around it.

If the system is not able to match the detected face with one of the enrolled faces (either because the user's face is not yet enrolled or because the face identification similarity score is lower than the default recognition threshold), the rectangle box is drawn in red with the unknown identity.

Unknown person


To enroll a new user, simply touch (or click) inside the red rectangle. The virtual keyboard is then displayed to enter the user's name. To finish the enrollment process, simply press the return key. Note that the face picture is captured when the red rectangle is touched (or clicked).

Info.png
  • The maximum number of people to be enrolled is configurable and is by default set to 200 people.
  • As the enrollment database is stored in the file system, the database is persistent upon reset.
Virtual keyboard displayed after having touched an unknown face


If the system is able to match the detected face with one of the enrolled faces, the rectangle box is drawn in green and the registered user's name is displayed with the similarity score expressed in percentage (%). The thumbnail (representing the user's enrolled face matching the detected face) is displayed and highlighted with a green rectangle in the banner located at the bottom of the preview.

Info.png The banner at the bottom of the preview is displaying the thumbnails of each users previously detected, ordered by detection history.
Jack is detected


The information displayed at the top of the display provide performance figures:

  • disp. fps is the average frame rate of the preview expressed in frame per second (display when camera input mode is selected)
  • inf. fps is the average frame rate for both face detection and face recognition inferences grouped together (display when camera input mode is selected)
  • inf. time is the instant measure of inference time for the face detection processing and the face recognition processing.

3.4 The database[edit]

  • The database is stored in the file system:
Board $> ls -l /usr/local/demo-ai/computer-vision/tflite-face-recognition/database
total 115
-rw-r--r-- 1 root root 23976 Dec 16 09:37 john.png
-rw-r--r-- 1 root root 23336 Dec 16 09:37 jack.png
-rw-r--r-- 1 root root 24881 Dec 16 09:37 lily.png
-rw-r--r-- 1 root root 20928 Dec 16 09:37 jen.png
-rw-r--r-- 1 root root 21550 Dec 16 09:37 marc.png

Every user is represented with a .png file representing the user's registered face. The name of the .png file is the name of the user the face is referring to.


  • To get the number of registered users execute the following command:
Board $> find /usr/local/demo-ai/computer-vision/tflite-face-recognition/database -type f | wc -l
5
  • To unregister users from the database, two solutions exist:
    • In the thumbnail banner, touch the thumbnail face of the user you want to remove
    • In the database directory, delete the file of the user you want to remove

3.5 Performance[edit]

  • The average execution frame rate to execute both face detection and face recognition on 1 face is around 5 fps:
    • face detection execution time ~70 ms
    • face recognition execution time ~55 ms
Info.png The inference times of the face detection and face recognition processing vary according to:
  • Linux® system load
  • The number of faces that are detected simultaneously. By default single face detection is set. For multiple face detection --reco_simultaneous_faces parameter must be set.
  • The recognition threshold corresponds to the similarity score above which the input face is successfully mapped to one of the enrolled faces.
    • The default recognition threshold is set to 0.70 and its value is configurable.
    • TAR (True Acceptance Rate) and FAR (False Acceptance Rate) are the metrics used to compute the recognition threshold default value.
    • The True Acceptance Rate represents the degree at which the system can correctly match the biometric information from the same person.
    • The False Acceptance Rate is the probability of cases for which the system fallaciously authorizes an unauthorized person.
    • A recognition threshold of 0.70 corresponds to a FAR of ~1% when plotting the TAR=f(FAR) graph using a dedicated (non-public) test database.
Info.png The user must keep in mind that the following parameters impact the accuracy of the recognition:
  • Ambient light and illumination
  • Distance between the camera and the user (best results are achieved at a distance <1.5 m)

4 References[edit]


This article explains how to get started on the face recognition application.

{{Warning | The face recognition binary is available on demand. Please contact the local STMicroelectronics support for more information about this application or send a request to [mailto:edge.ai@st.com edge.ai@st.com]}}

==Description==
The face recognition application is capable of recognizing faces of known (enrolled) users.<br>

<div class="res-img">

[[File: cpp_tfl_face_recognition_application_screenshot.png|center|C/C++ TensorFlow™ Lite face recognition application]]</div>


The application demonstrates a computer vision use case for face recognition
* In camera mode, frames are grabbed from a camera input (/dev/videox) and processed by two neural network models (face detection and face recognition) interpreted by the {{Highlight|TensorFlow™ Lite<ref name=tensorflowlite_url>[https://www.tensorflow.org/lite TensorFlow™ Lite]</ref>}} framework. A GStreamer pipeline is used to stream camera frames (using v4l2src), to display a preview (using waylandsink), and to execute neural network inference (using appsink).<br>

* In file system mode, pictures are read from the file system. In that case Gstreamer pipeline is replaced by OpenCV functions.
The result of the inference is displayed using overlay generated by GtkWidgets with Cairo.<br>

This combination is quite simple and efficient in terms of CPU overhead.<br>

<div class="res-img">

[[File: cpp_tfl_face_recognition_application_pipeline.png|center| Face recognition pipeline]]</div>


=== Frame processing flow ===
The figure below shows the different frame processing stages involved in the face recognition application:
<div class="res-img">

[[File: cpp_tfl_face_recognition_dataflow.png|center||Data flow for the face recognition application.]]</div>


====Camera  frame capture====
The camera frames are captured by the GStreamer pipeline in a YUV422 VGA (640 x 480) resolution (this is the default resolution of the application).

==== Frame preprocessing====
The main preprocessing stage is the pixel color format conversion from YUV422 captured frames into RGB888 frames.

==== Face detection====
The face detection block is in charge of finding the faces present in the input frame (VGA, RGB888).
The output of this block is a frame of resolution 96 x 96 that contains the faces found in the input captured frame.

==== Face recognition====
The face Recognition block is in charge of extracting features from the face and computing a signature (embedding vector) corresponding to the input face.

====Face identification====
The face identification block is in charge of computing the distance between:
* the vector produced by the face recognition block, and
* each of the vectors stored in memory (and corresponding to the enrolled faces)

The output face identification block generates the two following outputs:
* a ''User Face ID'' corresponding to the minimum distance
* a similarity score

==Installation==
{{Info | The face recognition binary is available on demand. Please contact the local STMicroelectronics support for more information about this application or send a request to [mailto:edge.ai@st.com edge.ai@st.com]}}
===copy the face recognition binary===
 {{PC$}} scp path/to/the/deb/package/tflite-cv-apps-face-recognition-c++_1.21.0-r0.0_armhf.deb root@<board_ip_address>:/home/root
===deploy the face recognition binary===
{{Warning|{{SoftwareLicenseAgreement | distribution=X-LINUX-AI | chapter=Face recognition application}}}}
After having [[X-LINUX-AI_OpenSTLinux_Expansion_Package#Configure the AI OpenSTLinux package repository|configured the AI OpenSTLinux package]], you can deploy the application that will need to download dependencies:

 {{Board$}} cd /home/root
 {{Board$}} apt-get install ./tflite-cv-apps-face-recognition-c++_1.21.0-r0.0_armhf.deb
Then restart the demo launcher:
 {{Board$}} systemctl restart weston@root

==How to use the application==
===Launching via the demo launcher===<div class="res-img">

[[File: launch_cpp_tfl_face_recognition.png|center|550px|Demo launcher]]</div>


===Executing with the command line===
The '''facereco_tfl_gst_gtk''' C/C++ application is located in the userfs partition:
 /usr/local/demo-ai/computer-vision/tflite-face-recognition/bin/facereco_tfl_gst_gtk
It accepts the following input parameters:
<pre> 

Usage: ./facereco_tfl_gst_gtk [option]

--reco_simultaneous_faces:       enable multiple face recognition (default is single face recognition)
--reco_threshold <val>:          face recognition threshold for face similarity (default is 0.70 = 70%)
--max_db_faces <val>:            maximum number of faces to be stored in the database (default is 200)
-d --database <directory path>:  provide the directory where the face recognition database is stored (else default location is used)
-i --image <directory path>:     image directory with images to be classified
-v --video_device <n>:           video device (default /dev/video0)
--frame_width  <val>:            width of the camera frame (default is 640 pixels)
--frame_height <val>:            height of the camera frame (default is 480 pixels)
--framerate <val>:               frame rate of the camera (default is 15 fps)
--validation:                    enable the validation mode
--help:                          show this help</pre>


*launch face recognition based on the pictures located in '''/usr/local/demo-ai/computer-vision/tflite-face-recognition/testdata''' directory
 {{Board$}} /usr/local/demo-ai/computer-vision/tflite-face-recognition/bin/launch_bin_facereco_tfl_model_testdata.sh

{{Info|Note that you need to populate the testdata directory with your own datasets.<br>

The pictures are then randomly read from the testdata directory}}

===In practice===
As soon as a face is detected within the camera captured frame, a rectangle box is drawn around it.

If the system is not able to match the detected face with one of the enrolled faces (either because the user's face is not yet enrolled or because the face identification similarity score is lower than the default recognition threshold),  the rectangle box is drawn in red with the '''unknown''' identity. 
<div class="res-img">

[[File: cpp_tfl_face_recognition_application_unknown.png|center|Unknown person]]</div>


To enroll a new user, simply touch (or click) inside the red rectangle. The virtual keyboard is then displayed to enter the user's name. To finish the enrollment process, simply press the return key.
Note that the face picture is captured when the red rectangle is touched (or clicked).
{{Info |
*The maximum number of people to be enrolled is configurable and is by default set to 200 people. 
*As the enrollment database is stored in the file system, the database is persistent upon reset.}}
<div class="res-img">

[[File: cpp_tfl_face_recognition_application_enrollment.png|center|Virtual keyboard displayed after having touched an unknown face]]</div>


If the system is able to match the detected face with one of the enrolled faces, the rectangle box is drawn in green and the registered '''user's name''' is displayed with the '''similarity score expressed in percentage (%)'''. The thumbnail (representing the user's enrolled face matching the detected face) is displayed and highlighted with a green rectangle in the banner located at the bottom of the preview.

{{Info | The banner at the bottom of the preview is displaying the thumbnails of each users previously detected, ordered by detection history.}}
<div class="res-img">

[[File: cpp_tfl_face_recognition_application_user_detected.png|center|Jack is detected]]</div>


The information displayed at the top of the display provide performance figures:
* '''disp. fps''' is the average frame rate of the preview expressed in frame per second (display when camera input mode is selected)
* '''inf. fps''' is the average frame rate for both face detection and face recognition inferences grouped together (display when camera input mode is selected)
* '''inf. time''' is the instant measure of inference time for the face detection processing and the face recognition processing.

===The database===
* The database is stored in the file system:
 {{Board$}} ls -l /usr/local/demo-ai/computer-vision/tflite-face-recognition/database
 total 115
 -rw-r--r-- 1 root root 23976 Dec 16 09:37 john.png
 -rw-r--r-- 1 root root 23336 Dec 16 09:37 jack.png
 -rw-r--r-- 1 root root 24881 Dec 16 09:37 lily.png
 -rw-r--r-- 1 root root 20928 Dec 16 09:37 jen.png
 -rw-r--r-- 1 root root 21550 Dec 16 09:37 marc.png

Every user is represented with a .png file representing the user's registered face. The name of the .png file is the name of the user the face is referring to.

* To get the number of registered users execute the following command:
 {{Board$}} find /usr/local/demo-ai/computer-vision/tflite-face-recognition/database -type f | wc -l
 5

* To unregister users from the database, two solutions exist:
** In the thumbnail banner, touch the thumbnail face of the user you want to remove
** In the database directory, delete the file of the user you want to remove

===Performance===
*The average execution frame rate to execute both face detection and face recognition on '''1 face''' is around 5 fps:
** face detection execution time ~70 ms
** face recognition execution time ~55 ms

{{Info | The inference times of the face detection and face recognition processing vary according to:
* Linux<sup>®</sup> system load
* The number of faces that are detected simultaneously. By default single face detection is set. For multiple face detection --reco_simultaneous_faces parameter must be set.}}

*The recognition threshold corresponds to the similarity score above which the input face is successfully mapped to one of the enrolled faces.
** The default recognition threshold is set to 0.70 and its value is configurable.
** TAR (''True Acceptance Rate'') and FAR (''False Acceptance Rate'') are the metrics used to compute the recognition threshold default value.  
** The ''True Acceptance Rate'' represents the degree at which the system can correctly match the biometric information from the same person. 
** The ''False Acceptance Rate'' is the probability of cases for which the system fallaciously authorizes an unauthorized person.
** A recognition threshold of 0.70 corresponds to a FAR of ~1% when plotting the TAR=f(FAR) graph using a dedicated (non-public) test database.

{{Info | The user must keep in mind that the following parameters impact the accuracy of the recognition:
* Ambient light and illumination
* Distance between the camera and the user (best results are achieved at a distance <1.5 m)}}

==References==<references />
<noinclude>

[[Category:Artificial intelligence sample apps|8]]
{{PublicationRequestId | 18384 | 10Dec'20}}</noinclude>
Line 54: Line 54:
 
{{Info | The face recognition binary is available on demand. Please contact the local STMicroelectronics support for more information about this application or send a request to [mailto:edge.ai@st.com edge.ai@st.com]}}
 
{{Info | The face recognition binary is available on demand. Please contact the local STMicroelectronics support for more information about this application or send a request to [mailto:edge.ai@st.com edge.ai@st.com]}}
 
===copy the face recognition binary===
 
===copy the face recognition binary===
  {{PC$}} scp path/to/the/deb/package/tflite-cv-apps-face-recognition-c++_1.1.0-r0_armhf.deb root@<board_ip_address>:/home/root
+
  {{PC$}} scp path/to/the/deb/package/tflite-cv-apps-face-recognition-c++_1.2.0-r0.0_armhf.deb root@<board_ip_address>:/home/root
 
===deploy the face recognition binary===
 
===deploy the face recognition binary===
 
{{Warning|{{SoftwareLicenseAgreement | distribution=X-LINUX-AI | chapter=Face recognition application}}}}
 
{{Warning|{{SoftwareLicenseAgreement | distribution=X-LINUX-AI | chapter=Face recognition application}}}}
Line 60: Line 60:
   
 
  {{Board$}} cd /home/root
 
  {{Board$}} cd /home/root
  {{Board$}} apt-get install ./tflite-cv-apps-face-recognition-c++_1.1.0-r0_armhf.deb
+
  {{Board$}} apt-get install ./tflite-cv-apps-face-recognition-c++_1.2.0-r0.0_armhf.deb
 
Then restart the demo launcher:
 
Then restart the demo launcher:
 
  {{Board$}} systemctl restart weston@root
 
  {{Board$}} systemctl restart weston@root