How to use Teachable Machine to create an image classification application on STM32

Revision as of 16:45, 21 September 2022 by Registered User (Cropping the image)

This article shows how to use the Teachable Machine online tool with STM32Cube.AI and the FP-AI-VISION function pack to create an image classifier running on the STM32H747I-DISCO board.

This tutorial is divided into three parts: the first part shows how to use the Teachable Machine to train and export a deep learning model, then STM32Cube.AI is used to convert this model into optimized C code for STM32 MCUs. The last part explains how to integrate this new model into the FP-AI-VISION1 to run live inference on an STM32 board with a camera. The whole process is described below:

CLI_as_back-end


Info white.png Information
  • Teachable Machine is an online tool allowing to quickly train a deep learning model for various tasks including image classification. It is an educational tool not suitable for production purposes.
  • STM32Cube.AI[ST 1] is an expansion software for STM32CubeMX that generates optimized C code for STM32 microcontrollers and neural network inference. It is delivered under the Mix Ultimate Liberty+OSS+3rd-party V1 software license agreement[ST 2] with the additional component license schemes listed in the product data brief[ST 3]
  • The FP-AI-VISION1 Function Pack is a software example of an image classifier running on the STM32H747I-DISCO board. It is delivered under the Mix Ultimate Liberty+OSS+3rd-party V1 software license agreement[ST 4] with the additional component license schemes listed in the product data brief[ST 5]

1 Prerequisites

1.1 Hardware

1.2 Software

2 Training a model using Teachable Machine

In this section, we will train deep neural network in the browser using Teachable Machine. We first need to choose something to classify. In this example, we will classify ST boards and modules. The chosen boards are shown in the figure below:

Boards used for classification

You can choose whatever object you want to classify it: fruits, pasta, animals, people, etc...

Info white.png Information
Note: the STM32H747 discovery kit combined with the B-CAMS-OMV camera daughter board can be used as a USB webcam to be used directly with Teachable Machine. However, Teachable Machine allows also to import images from your computer.

Let's get started. Open https://teachablemachine.withgoogle.com/, preferably from Chrome browser.

Click Get started, then select Image Project, then Standard image model (224x244px color images). You will be presented with the following interface.

Teachable Machine interface

2.1 Adding training data

For each category you want to classify, edit the class name by clicking the pencil icon. In this example, we choose to start with SensorTile.

To add images with your webcam, click the webcam icon and record some images. If you have image files on your computer, click upload and select the directory containing your images.

The STM32H747 discovery kit combined with the B-CAMS-OMV camera daughter board can be used as a USB webcam. Using the ST kit for data collection will help to get better results as the same camera will be used for data collection and inference when the model will have been trained.

To use the ST kit as a webcam, simply program the board with the following binary of the function pack. First plug your board onto your computer using the ST-LINK port. Make sure the JP6 jumper is set to ST-LINK.

Discovery board ST-LINK setup

After plugging the USB cable onto your computer, the board appears as a mounted device.

The binary is located under FP-AI-VISION1_V3.1.0\Projects\STM32H747I-DISCO\Applications\USB_Webcam\Binary.

Drag and drop it onto the board mounted device. This flashes the binary on the board.

Unplug the board, change the JP6 jumper to the HS position, and plug your board using the USB OTG port.

Discovery board USB OTG setup

For convenience, you can plug simply two USB cables, one on the USB OTG port, the other on the USB ST-LINK and set the JP6 jumper to ST-LINK. In this case, the board can be programmed and switch from USB webcam mode to test mode without the need to change the jumper position.

Depending on how you oriented the camera board, you might prefer to flip the image. If you do so you need to use the same option when generating the code on STM32.

Adding images with a webcam
Info white.png Information
No need to capture too many images, ~100 images per class is usually well enough. Try to vary camera angle, subject pose and scale as much as possible. To obtain the best results, a uniform background is recommended.

Once you have a satisfactory amount of images for this class, repeat the process for the next one until your dataset is complete.

Info white.png Information
Note: It can be nice to have a "Background/Nothing" class, so that the model is able to tell when nothing is presented to the camera.

2.2 Training the model

Now that we have a good amount of data, we are going to train a deep learning model for classifying these different objects. To do this, click the Train Model button as shown below:

Train a model

This process can take a while, depending on the amount of data you have. To monitor the training progress, you can select Advanced and click Under the hood. A side panel displays training metrics.

When the training is complete, you can see the predictions of your network on the "Preview" panel. You can either choose a webcam input or an imported file.

Predictions from the model

2.2.1 What happens under the hood (for the curious)

Teachable Machine is based on Tensorflow.js to allow neural network training and inference in the browser. However, as image classification is a task that requires a lot of training time, Teachable Machine uses a technique called transfer learning: The webpage downloads a MobileNetV2 model that was previously trained on a big image dataset of 1000 categories. The convolution layers of this pre-trained model are very good at doing feature extraction so they do not need to be trained again. Only the last layers of the neural network are trained using Tensorflow.js, thus saving a lot of time.

2.3 Exporting the model


If you are happy with your model, it is time to export it. To do so, click the Export Model button. In the pop-up window, select Tensorflow Lite, check Quantized and click Download my model.

Export the model

Since the model conversion is done in the cloud, this step can take a few minutes.

Your browser downloads a zip file containing the model as a .tflite file and a .txt file containing your label. Extract these two files in an empty directory that we will call workspace in the rest of this tutorial.

2.3.1 Inspecting the model using Netron (optional)

It is always interesting to take a look at a model architecture as well as its input and output formats and shapes. To do this, use the Netron webapp.

Visit https://lutzroeder.github.io/netron/ and select Open model, then choose the model.tflite file from Teachable Machine. Click sequental_1_input: we observe that the input is of type uint8 and of size [1, 244, 244, 3]. Now let's look at the outputs: in this example we have 6 classes, so we see that the output shape is [1,6]. The quantization parameters are also reported. Refer to part 3 for how to use them.

Model visualization

3 Porting to a target board

3.1 STM32H747I-DISCO

In this part we will use the stm32ai command line tool to convert the TensorflowLite model to optimized C code for STM32.

Warning white.png Warning
Warning: FP-AI-VISION1 v3.1.0 is based on X-Cube-AI version 7.1.0, but can be updated to support a newer version. You can check your version of Cube.AI by running stm32ai --version


For ease of usage, add the X-Cube-AI installation folder to your path, for Windows:

  • For X-CUBE-AI v7.1.0:
 set CUBE_FW_DIR=C:\Users\<USERNAME>\STM32Cube\Repository
 set X_CUBE_AI_DIR=%CUBE_FW_DIR%\Packs\STMicroelectronics\X-CUBE-AI\7.1.0
 set PATH=%X_CUBE_AI_DIR%\Utilities\windows;%PATH%
  • For X-CUBE-AI v7.2.0:
 set CUBE_FW_DIR=C:\Users\<USERNAME>\STM32Cube\Repository
 set X_CUBE_AI_DIR=%CUBE_FW_DIR%\Packs\STMicroelectronics\X-CUBE-AI\7.2.0
 set PATH=%X_CUBE_AI_DIR%\Utilities\windows;%PATH%

Start by opening a shell in your workspace directory, then execute the following command:

 cd <path to your workspace>
 stm32ai generate -m model.tflite -v 2

The expected output is:

Neural Network Tools for STM32AI v1.6.0 (STM.ai v7.1.0-RC3)

 Exec/report summary (generate)
 ------------------------------------------------------------------------------------------------------------------------
 model file           : C:\path_to_workspace\model.tflite
 type                 : tflite
 c_name               : network
 compression          : lossless
 workspace dir        : C:\path_to_workspace\stm32ai_ws
 output dir           : C:\path_to_workspace\stm32ai_output
 model_name           : model
 model_hash           : 3e04a924905fea0274099abd7153e500
 input 1/1            : 'serving_default_sequential_1_input0'
                        150528 items, 147.00 KiB, ai_u8, scale=0.00747405, zero_point=134, (1,224,224,3), domain:user/
 output 1/1           : 'nl_71_0_conversion'
                        5 items, 5 B, ai_u8, scale=0.00390625, zero_point=0, (1,1,1,5), domain:user/
 params #             : 517,688 items (526.47 KiB)
 macc                 : 58,587,644
 weights (ro)         : 539,128 B (526.49 KiB) / +20(+0.0%) vs original model (1 segment)
 activations (rw)     : 610,784 B (596.47 KiB) (1 segment)
 ram (total)          : 761,317 B (743.47 KiB) = 610,784 + 150,528 + 5


 Generated C-graph summary
 ------------------------------------------------------------------------------------------------------------------------
 model name           : model
 c-name               : network
 c-node #             : 68
 c-array #            : 264
 activations size     : 610784 (1 segments)
 weights size         : 539128 (1 segments)
 macc                 : 58587644
 inputs               : ['serving_default_sequential_1_input0_output']
 outputs              : ['nl_71_0_conversion_output']

 C-Arrays (264)
 ------------------------------------------------------------------------------------------------------------------------------------------------
 c_id   name (*_array)                               item/size       domain/mem-pool           c-type          fmt                       comment
 ------------------------------------------------------------------------------------------------------------------------------------------------
 0      serving_default_sequential_1_input0_output   150528/150528   user/                     uint8_t         int8/ua                   /input
 1      conversion_0_output                          150529/150529   activations/**default**   int8_t          int8/sa  
 2      conv2d_2_output                              200704/200704   activations/**default**   int8_t          int8/sa  

( ... )

 261    conv2d_67_scratch0                           13248/13248     activations/**default**   int8_t          int/ss   
 262    conv2d_67_scratch1                           62720/62720     activations/**default**   int8_t          int8/sa  
 263    conv2d_67_scratch2                           62720/62720     activations/**default**   int8_t          int8/sa  
 ------------------------------------------------------------------------------------------------------------------------------------------------

 C-Layers (68)
 -----------------------------------------------------------------------------------------------------------------------------------------------
 c_id   name (*_layer)          id   layer_type         macc      rom      tensors                                         shape (array id)
 -----------------------------------------------------------------------------------------------------------------------------------------------
 0      conversion_0            0    conv               301056    0        I: serving_default_sequential_1_input0_output   (1,224,224,3) (0)
                                                                           O: conversion_0_output                          (1,224,224,3) (1)
 -----------------------------------------------------------------------------------------------------------------------------------------------
 1      conv2d_2                2    conv2d             5419024   496      I: conversion_0_output                          (1,224,224,3) (1)
                                                                           S: conv2d_2_scratch0                         
                                                                           S: conv2d_2_scratch1                         
                                                                           W: conv2d_2_weights                             (3,16,3,3) (69)
                                                                           W: conv2d_2_bias                                (1,1,1,16) (70)
                                                                           O: conv2d_2_output                              (1,112,112,16) (2)
 -----------------------------------------------------------------------------------------------------------------------------------------------

( ... )

Complexity report per layer - macc=58,587,764 weights=539,232 act=610,784 ram_io=150,534
---------------------------------------------------------------------------------
id   name           c_macc                    c_rom                     c_id    
---------------------------------------------------------------------------------
0    conversion_0   |                  0.5%   |                  0.0%   [0]     
2    conv2d_2       ||||||||||||       9.2%   |                  0.1%   [1]     
3    conv2d_3       ||||               3.1%   |                  0.0%   [2]     
4    conv2d_4       ||||               2.7%   |                  0.0%   [3]     
5    conv2d_5       |||||||||||        8.2%   |                  0.1%   [4]     
7    conv2d_7       |||                2.3%   |                  0.1%   [5]     
  
( ... )
  
-----------------------------------------------------------------------------------------------------------------------------------------------
 66     nl_71                   71   nl                 75        0        I: dense_70_0_conversion_output                 (1,1,1,5) (66)
                                                                           O: nl_71_output                                 (1,1,1,5) (67)
 -----------------------------------------------------------------------------------------------------------------------------------------------
 67     nl_71_0_conversion      71   conv               10        0        I: nl_71_output                                 (1,1,1,5) (67)
                                                                           O: nl_71_0_conversion_output                    (1,1,1,5) (68)
 -----------------------------------------------------------------------------------------------------------------------------------------------

This command generates five files under workspace/stm32ai_ouptut/:

  • network_config.h
  • network.c
  • network_data.c
  • network_data_params.c (for X-CUBE-AI v7.2.0 and above)
  • network.h
  • network_data.h
  • network_data_params.h (for X-CUBE-AI v7.2.0 and above)

Let's take a look at the highlighted lines: we learn that the model uses 526.49 Kbytes of weights (read-only memory) and 596.47 Kbytes of activations. The STM32H747xx MCUs do not have 596.47 Kbytes of contiguous RAM, we need to use the external SDRAM present on the STM32H747-DISCO board. Refer to UM2411 section 5.8 "SDRAM" for more information.

Info white.png Information
These figures about memory footprint might be different for your model as it depends on the number of classes you have.
Info white.png Information
From X-CUBE-AI v7.2.0, for a given model, the value of the weights (c-table) are now defined in a new specific c-file: network_data_params.c\.h files. Previous network_data.c\.h is always generated but only with the intermediate functions to manage the weights.

3.1.1 Integration with FP-AI-VISION1

In this part we will import our brand-new model into the FP-AI-VISION1 function pack. This function pack provides a software example for a food classification application. For more information on FP-AI-VISION1, go here.

The main objective of this section is to replace the network and network_data files in FP-AI-VISION1 by the newly generated files and make a few adjustments to the code.

3.1.1.1 Open the project

If it is not already done, download the zip file from ST website and extract the content to your workspace. It must now contain the following elements:

  • model.tflite
  • labels.txt
  • stm32ai_output
  • FP_AI_VISION1

If we take a look inside the function pack, we'll start from the FoodReco_MobileNetDerivative application we can see two configurations for the model data type, as shown below.

FP-AI-VISION1 model data types

Since our model is a quantized one, we have to select the Quantized_Model directory.

Go into workspace/FP_AI_VISION1_V3.1.0/Projects/STM32H747I-DISCO/Applications/FoodReco_MobileNetDerivative/Quantized_Model/STM32CubeIDE and double-click .project. STM32CubeIDE starts with the project loaded. You will notice 2 sub-project for each core of the microcontroller : CM4 and CM7, as we don't use CM4, ignore it and work with the CM7 project.

3.1.1.2 Replacing the network files

The model files are located in workspace/FP-AI-VISION1_V3.1.0/Projects/STM32H747I-DISCO/Applications/FoodReco_MobileNetDerivative/Quantized_Model/CM7/ Src and Inc directory.

Delete the following files and replace them with the ones from workspace/stm32ai_output:

In Src:

  • network.c
  • network_data.c
  • network_data_params.c (for X-CUBE-AI v7.2 and above)

In Inc:

  • network.h
  • network_data.h
  • network_data_params.c (for X-CUBE-AI v7.2 and above)

3.1.1.3 Updating to a newer version of X-CUBE-AI

FP-AI-VISION1 v3.1.0 is based on X-Cube-AI version 7.1.0, You can check your version of Cube.AI by running stm32ai --version. If the c-model files were generated with a newer version of X-Cube-AI (> 7.1.0), you need also to update the X-CUBE-AI library (NetworkRuntime720_CM7_GCC.a for X-CUBE-AI version 7.2).

Go to workspace/FP-AI-VISION1_V3.1.0/Middlewares/ST/STM32_AI_Runtime then for a Windows command prompt:

 copy %X_CUBE_AI_DIR%\Middlewares\ST\AI\Inc\*  .\Inc\
 copy %X_CUBE_AI_DIR%\Middlewares\ST\AI\Lib\GCC\ARMCortexM7\NetworkRuntime720_CM7_GCC.a .\lib\NetworkRuntime710_CM7_GCC.a
Info white.png Information
Copying the new runtime library NetworkRuntime720_CM7_GCC.a (for X-CUBE-AI version 7.2) using the same name of the previous version (NetworkRuntime710_CM7_GCC.a) avoids to update the project properties. if not you will need to update the project properties in the panel C/C++ General / Paths and Symbols / Librairies. Edit the library and update with the new name.

You also need to add to the project tree the new generated file network_data_params.c. Simply drag and drop the network_data_params.c within the STM32CubeIDE project in the Applications folder and link to file as shown below:

Adding the file to the project

3.1.1.4 Updating the labels and display

In this step we will update the labels for the network output. The label.txt file downloaded with Teachable Machine can help you doing this. In our example, the content of this file looks like this:

0 SensorTile
1 IoTNode
2 STLink
3 Craddle Ext
4 Fanout
5 Background

From STM32CubeIDE, open fp_vision_app.c. Go to line 125 where the output_labels is defined and update this variable with your label names:

This snippet is provided AS IS, and by taking it, you agree to be bound to the license terms that can be found here for the component: Application.
// fp_vision_app.c line 125
const char* output_labels[AI_NET_OUTPUT_SIZE] = {
    "SensorTile", "IoTNode", "STLink", "Craddle Ext", "Fanout", "Background"};

While we're here, we'll update the display mode that it shows camera image instead of food logos. Go around line 200 and update the App_Output_Display function. At the top of the function, the display_mode variable should be set to 1.

This snippet is provided AS IS, and by taking it, you agree to be bound to the license terms that can be found here for the component: Application.
static void App_Output_Display(AppContext_TypeDef *App_Context_Ptr)
{
  static uint32_t occurrence_number = NN_OUTPUT_DISPLAY_REFRESH_RATE;
  static uint32_t display_mode = 1; // Was 0

3.1.1.5 Cropping the image

Teachable Machine crops the webcam image to fit the model input size. In FP-AI-VISION1, the image is resized to the model input size, hence losing the aspect ratio. We will change this default behavior and implement a crop of the camera image.

In order to have square images and avoid image deformation we are going to crop the camera image using the DCMI. The goal of this step is to go from the 640x480 resolution to a 480x480 resolution.

First, edit fp_vision_camera.h (located in FP-AI-VISION1_V3.1.0/Projects/STM32H747I-DISCO/Applications/Common/CM7/Inc) at line 59 to update the CAMERA_WIDTH define to 480 pixels:

This snippet is provided AS IS, and by taking it, you agree to be bound to the license terms that can be found here for the component: Application.
//fp_vision_camera.h line 57
#if CAMERA_CAPTURE_RES == VGA_640_480_RES
#define CAMERA_RESOLUTION CAMERA_R640x480
#define CAM_RES_WIDTH 480 // Was 640
#define CAM_RES_HEIGHT 480

Then, edit fp_vision_camera.c located in FP-AI-VISION1_V3.1.0/Projects/STM32H747I-DISCO/Applications/Common/CM7/Src.

Modify the CAMERA_Init function (line 58) to configure DCMI cropping (update the function with the highlighted code bellow) :

This snippet is provided AS IS, and by taking it, you agree to be bound to the license terms that can be found here for the component: Application.
void CAMERA_Init(CameraContext_TypeDef* Camera_Context_Ptr)
{
  CAMERA_Context_Init(Camera_Context_Ptr);

  __HAL_RCC_MDMA_CLK_ENABLE();

  (...)

  /* Set camera mirror / flip configuration */
  CAMERA_Set_MirrorFlip(Camera_Context_Ptr, Camera_Context_Ptr->mirror_flip);

  /* With default mounting of the B-CMAS-OMV, the image should be flipped */
  /* If at the end the image is not inn the right position, comment the line below */
  CAMERA_Set_MirrorFlip(Camera_Context_Ptr, CAMERA_MIRRORFLIP_FLIP);

  HAL_Delay(100);
  
  /* Center-crop the 640x480 frame to 480x480 */
  const uint32_t x0 = (640 - 480) / 2;
  const uint32_t y0 = 0;

  /* Note: 1 px every 2 DCMI_PXCLK (8-bit interface in RGB565) */
  HAL_DCMI_ConfigCrop(&hcamera_dcmi,
                      x0 * 2,
                      y0,
                      CAM_RES_WIDTH * 2 - 1,
                      CAM_RES_HEIGHT - 1);

  HAL_DCMI_EnableCrop(&hcamera_dcmi);
  /* Wait for the camera initialization after HW reset */ 
  HAL_Delay(200);
  
  /*
   * Start the Camera Capture
   * Using intermediate line buffer in D2-AHB domain to support high pixel clocks.
   */
  if (HAL_DCMIEx_Start_DMA_MDMA(&hcamera_dcmi, CAMERA_MODE_CONTINUOUS,
                                (uint8_t *)Camera_Context_Ptr->camera_capture_buffer,
                                CAM_LINE_SIZE, CAM_RES_HEIGHT) != HAL_OK)
  {
    while(1);
  }

Now image cropping is enabled and the image is square.

3.1.1.6 Adapt to the NN input data range

The neural network input needs to be normalized accordingly to the training phase. This is achieved by updating the value of both the nn_input_norm_scale and nn_input_norm_zp variables during initialization. The nn_input_norm_scale and nn_input_norm_zp variables affect the pixel format adaptation stage. The scale, zero point values should be set {127.5, 127} if the NN model was trained using input data normalized in the range [-1, 1]. They should be set to {255, 0} if the NN model was trained using input data normalized in the range [0, 1]. The food recognition model was trained with input data normalized in the range [0, 1] whereas the Teachable Model was trained in the range of [-1, 1].

Edit the file fp_vision_app.c and modify the App_Context_Init function (line 328) to update the scale and zero-point values (update the function with the highlighted code bellow) :

This snippet is provided AS IS, and by taking it, you agree to be bound to the license terms that can be found here for the component: Application.
  /*{scale,zero-point} set to {127.5, 127} since NN model was trained using input data normalized in the range [-1, 1]*/
  App_Context_Ptr->Ai_ContextPtr->nn_input_norm_scale=127.5f; //was 255.0f
  App_Context_Ptr->Ai_ContextPtr->nn_input_norm_zp=127; //was 0

3.1.2 Compiling the project

The function pack for quantized models comes in four different memory configurations :

  • Quantized_Ext
  • Quantized_Int_Fps
  • Quantized_Int_Mem
  • Quantized_Int_Split

As we saw in Part 2, the activation buffer requires more than 512 Kbytes of RAM. For this reason, we can only use the Quantized_Ext configuration to place activation buffer. For more details on the memory configuration, refer to UM2611 section 3.2.5 "Memory requirements".

To compile only the Quantized_Ext configuration, select Project > Properties from the top bar. Then select C/C++ Build from the left pane. Click manage configuration and then delete all configurations that are not Quantized_Ext. Only one configuration is left.

Memory configuration settings

Clean the project by selecting Project > Clean... and clicking Clean.

Eventually, build the project by clicking Project > Build All.

When the compilation is complete, a file named STM32H747I_DISCO_CM7.hex is generated in

workspace > FP-AI-VISION1_V3.1.0 > Projects > STM32H747I-DISCO > Applications > FoodReco_MobileNetDerivative > Quantized_Model > STM32CubeIDE > STM32H747I_DISCO > Quantized_Ext

3.1.3 Flashing the board

Connect the STM32H747I-DISCO to your PC via a Micro-USB to USB cable. Make sure that your board is connected to the ST-LINK port and properly powered (set the jumper JP6 jumper is set to ST-LINK).

Build the project STM32H747I_DISCO_FoodReco_Quantized_CM7 and run it with the play button or use STM32CubeProgrammer to program the board with the generated .hex file.

3.1.4 Testing the model

Connect the camera to the STM32H747I-DISCO board using a flex cable. To have the image in the upright position, the camera must be placed with the flex cable facing up as shown in the figure below. Once the camera is connected, power on the board and press the reset button. After the "Welcome Screen", you will see the camera preview and output prediction of the model on the LCD Screen.

Model inference running on target

3.2 Troubleshooting

You may notice that once the model is running on STM32, the performance of the deep learning model is not as expected. The rationale is the following:

  • Quantization: the quantization process can reduce the performance of the model, as going from a 32-bit floating point to a 8-bit integer representation means a loss in precision.
  • Camera: if the webcam used for training the model is different from the the camera on the Discovery board. This difference of data between the training and the inference can explain a loss in performance.

4 STMicroelectronics references

See also: