This article explains how to use the stai_mpu API for human face detection and recognition application supporting OpenVX [1] back-end.
1. Description[edit | edit source]
The human face recognition application is capable of detect the position of human faces in the frame and recognizing faces of known (enrolled) users.
The application demonstrates a computer vision use case for human face recognition where frames are grabbed from a camera input (/dev/videox) and analyzed by a neural network model interpreted by OpenVX framework.
A Gstreamer pipeline is used to stream camera frames (using v4l2src), to display a preview (using gtkwaylandsink) and to execute neural network inference (using appsink).
The result of the inference is displayed in the preview. The overlay is done using GtkWidget with cairo.
The models used with this application are the Blazeface for face detection and the Facenet512 for the face recognition downloaded respectively from the PINTO model zoo github [2] and Deepface github [3]
1.1. Frame processing flow[edit | edit source]
The figure below shows the different frame processing stages involved in the face recognition application:
1.1.1. Camera frame capture[edit | edit source]
The camera sensor has a default resolution of 5MPx.
The DCMIPP configuration set for the main pipeline set the format to RGB24 with a resolution of 760x568 at 30FPS which is the default resolution of the STM32MP25x evaluation board screen, the auxiliary pipeline is set to the face recognition NN input size 160x160 at 30FPS.
The camera frames are captured by the two GStreamer pipelines one dedicated to display the video stream and the NN inference outputs the second is dedicated to provide frames to the NN model.
1.1.2. Face detection[edit | edit source]
The face detection block is in charge of finding the faces present in the input frame.
The blazeface model allows the detection of the position of faces as well as certain points of interest such as the eyes, nose, mouth, and ears. As part of this application, we only use the position of the face and not all the facial key-points.
The coordinates of the faces detected can be used to extract the faces found in the input captured frame.
1.1.3. Face recognition[edit | edit source]
The face Recognition block is in charge of extracting features from the face and computing a signature (embedding vector) corresponding to the input face.
1.1.4. Face identification[edit | edit source]
The face identification block is in charge of computing the distance between:
- the vector produced by the face recognition block, and
- each of the vectors stored in memory (and corresponding to the enrolled faces)
The output face identification block generates the two following outputs:
- a User Face ID corresponding to the minimum distance
- a similarity score
2. Installation[edit | edit source]
2.1. Install from the OpenSTLinux AI package repository[edit | edit source]
After having configured the AI OpenSTLinux package you can install X-LINUX-AI components for image classification application:
2.2. Source code location[edit | edit source]
- in the OpenSTLinux Distribution with X-LINUX-AI Expansion Package:
- <Distribution Package installation directory>/layers/meta-st/meta-st-x-linux-ai/recipes-samples/face-recognition/files/stai_mpu
- on GitHub:
2.3. Regenerate the package from OpenSTLinux Distribution (optional)[edit | edit source]
Using the OpenSTLinux Distribution, you are able to rebuild the application.
 Information
|
| If not already installed, the X-LINUX-AI OpenSTLinux Distribution need to be installed by following this link |
- Set up the build environment:
cd <Distribution Package installation directory> source layers/meta-st/scripts/envsetup.shBe careful to select the right board in the script to setup the correct environment
3. How to use the application[edit | edit source]
3.1. Launching via the demo launcher[edit | edit source]
You can click on the icon to run the C/C++ application. By default the OpenVX applications will be installed on STM32MP2x.
3.1.1. Executing with the command line[edit | edit source]
The stai_mpu_face_recognition C/C++ application is located in the userfs partition:
/usr/local/x-linux-ai/face-recognition/stai_mpu_face_recognition
It accepts the following input parameters:
Usage: ./stai_mpu_face_recognition [option] --reco_simultaneous_faces: enable multiple face recognition (default is single face recognition) --reco_threshold <val>: face recognition threshold for face similarity (default is 0.70 = 70%) --max_db_faces <val>: maximum number of faces to be stored in the database (default is 200) -d --database <directory path>: provide the directory where the face recognition database is stored (else default location is used) -i --image <directory path>: image directory with images to be classified -v --video_device <n>: video device (default /dev/video0) --frame_width <val>: width of the camera frame (default is 640 pixels) --frame_height <val>: height of the camera frame (default is 480 pixels) --framerate <val>: frame rate of the camera (default is 15 fps) --validation: enable the validation mode --help: show this help
- launch face recognition based on the pictures located in /usr/local/x-linux-ai/face-recognition/models/facenet/testdata directory
/usr/local/x-linux-ai/face-recognition/launch_bin_face_recognition_testdata.sh
| Information
|
| Note that you need to populate the testdata directory with your own datasets. The pictures are then randomly read from the testdata directory |
3.1.2. In practice[edit | edit source]
As soon as a face is detected within the camera captured frame, a rectangle box is drawn around it.
If the system is not able to match the detected face with one of the enrolled faces (either because the user's face is not yet enrolled or because the face identification similarity score is lower than the default recognition threshold), the rectangle box is drawn in red with the unknown identity.
To enroll a new user, simply touch (or click) inside the red rectangle. The virtual keyboard is then displayed to enter the user's name. To finish the enrollment process, simply press the return key. Note that the face picture is captured when the red rectangle is touched (or clicked).
| Information
|
|
If the system is able to match the detected face with one of the enrolled faces, the rectangle box is drawn in green and the registered user's name is displayed. The thumbnail (representing the user's enrolled face matching the detected face) is displayed and highlighted with a green rectangle in the banner located at the bottom of the preview.
| Information
|
| The banner at the bottom of the preview is displaying the thumbnails of each users previously detected, ordered by detection history. To remove an already enrolled user, touch the thumbnail face of the user you want to remove. |
The information displayed at the top of the display provide performance figures:
- disp. fps is the average frame rate of the preview expressed in frame per second (display when camera input mode is selected)
- inf. fps is the average frame rate for both face detection and face recognition inferences grouped together (display when camera input mode is selected)
- inf. time is the instant measure of inference time for the face detection processing and the face recognition processing.
3.1.3. The database[edit | edit source]
- The database is stored in the file system:
ls -l /usr/local/x-linux-ai/face-recognition/database total 115 -rw-r--r-- 1 root root 23976 Dec 16 09:37 john.png -rw-r--r-- 1 root root 23336 Dec 16 09:37 jack.png -rw-r--r-- 1 root root 24881 Dec 16 09:37 lily.png -rw-r--r-- 1 root root 20928 Dec 16 09:37 jen.png -rw-r--r-- 1 root root 21550 Dec 16 09:37 marc.png
Every user is represented with a .png file representing the user's registered face. The name of the .png file is the name of the user the face is referring to.
- To get the number of registered users execute the following command:
find /usr/local/x-linux-ai/face-recognition/database -type f | wc -l 5
- To unregister users from the database, two solutions exist:
- In the thumbnail banner, touch the thumbnail face of the user you want to remove
- In the database directory, delete the file of the user you want to remove
3.1.4. Performance[edit | edit source]
- The average execution frame rate to execute both face detection and face recognition on 1 face is around 33 fps:
- face detection execution time ~30 ms
- face recognition execution time ~3 ms
Inferences are executed on the neural processing unit (NPU).
| Information
|
The inference times of the face detection and face recognition processing vary according to:
|
- The recognition threshold corresponds to the similarity score above which the input face is successfully mapped to one of the enrolled faces.
- The default recognition threshold is set to 0.70 and its value is configurable.
| Information
|
The user must keep in mind that the following parameters impact the accuracy of the recognition:
|
4. References[edit | edit source]