How to create a dual-tone multi-frequency classifier using NanoEdge AI Studio

Revision as of 18:37, 18 November 2021 by Registered User (Fixed the broken link to NanoEdge AI Studio)

This article is a tutorial on how to use NanoEdge™ AI Studio to create a dual tone multi frequency (DTMF)[1] classifier running on the STWIN SensorTile Wireless Industrial Node development kit (STEVAL-STWINKT1B).

This tutorial shows you how to program the STWIN for datalogging; how to use NanoEdge™ AI Studio to collect data, label data, find the best algorithm, run the Emulator, generate the library files for deployment; how to use STM32CubeIDE to build project and program the STWIN, and finally run the classifier on STWIN. The main process of this tutorial is shown in the flowchart below:

Flowchart of this article


1 Prerequisites

1.1 Hardware

  • STEVAL-STWINKT1B, the STWIN development kit,
  • Windows® powered laptop/PC (Windows® 7, 8, or 10),
  • Two Micro-USB cables (one to connect the STWIN to PC, and another one for the STLINK-V3MINI)
  • STLINK-V3MINI, which is included in the dev kit.
FP-AI-MONITOR1-hardware.png

1.2 Software

1.2.1 Source Code

25px Important
Firmware package delivered in this article is delivered under the Mix Ultimate Liberty+OSS+3rd-party V1 software license agreement (SLA0048).
  • A firmware package STWIN-DTMF-Classifier-wiki-package.zip is ready to download from st.com, please download the zip file here. The package includes:
  • Project source code in subfolder STWIN_DTMF_Classifier
  • Two Pre-compiled binary files:
  • STWIN_DTMF_Classifier-datalog.bin for datalogging
  • STWIN_DTMF_Classifier-detection.bin for tone detection
  • Library files generated by NanoEdge™ AI Studio libneai_STWIN-DTMF-Classifier.zip
Info white.png Information
In this article, we will only use the datalogging binary and the project source code. The other files are for your reference

1.2.2 NanoEdge™ AI Studio

  • NanoEdge™ AI Studio is a new Machine Learning (ML) technology to bring true innovation easily to the end-users. In just a few steps, developers can create an optimal ML library for their project, based on a minimal amount of data.
  • Download NanoEdge™ AI Studio as a free trial version for the STMicroelectronics boards.

1.2.3 STM32CubeIDE

  • STM32CubeIDE is an all-in-one multi-OS development tool, which is part of the STM32Cube software ecosystem. It is an advanced C/C++ development platform with peripheral configuration, code generation, code compilation, and debug features for STM32 microcontrollers and microprocessors.
  • Download STM32CubeIDE from st.com

1.2.4 Tera Term

  • Tera Term is an open-source and freely available software terminal emulator, which is used to display the classification results.
  • Users can download and install the latest version available from Tera Term.

1.2.5 Tone Generation App

Install the corresponding app on your smart phone for DTMF tone generation:

2 Step 1 – Program STWIN for datalogging

The first step is to connect the STWIN to your PC and program it for datalogging mode with the provided binary file.

2.1 Board connection

Cable connection on STWIN

There are three connections to make on the STWIN, as shown in the picture:

  1. Connect the STDC14 cable between the STWIN and STLINK-V3MINI.
  2. Plug the USB cable micro-B side into the STWIN.
  3. Plug the USB cable micro-B side into the STLINK-V3MINI.
  4. Plug the STWIN USB cable into PC.
  5. Plug the STLINK-V3MINI USB cable into PC.

2.2 Program STWIN using binary file

The easiest way to program the STWIN is using the drag-and-drop. Once you have the pre-built binary file in your local disk, the STWIN can be very easily programmed with the binary by simply performing a drag-and-drop action:

Drag and drop to program


  1. In Windows File Explorer, find the binary file ..\STWIN-DTMF-Classifier-wiki-package\binary\STWIN_DTMF_Classifier-datalog.bin
  2. Left click on the file and hold the left button on your mouse
  3. Find the STLINK_V3M disk in the Navigation pane on the left hand side, drag the binary to that disk and release the left button
  4. When the transfer is done, the STWIN is programmed successfully.
Info white.png Information
If the drag-and-drop method doesn't work, there are two other alternative methods to program the STWIN:

3 Step 2 – Create a Classification Project in NanoEdge™ AI Studio

In this step, we will create a classfication project in NanoEdge™ AI Studio, and configure the project settings.

Open NanoEdge™ AI Studio on PC, you should see the welcome interface below and then follow the steps:


  1. Click on Create a new project.
  2. Choose Classification for Project type.
  3. Give a project name.
  4. Add a brief description for the project.
  5. Click on the drop down menu for Target to open the pop-up window, then:
    • Click on STMicroelectronics tab
    • Choose STEVAL-STWINKT1B Cortex-M4 as target
    • Click on Validate to go back to the main interface.
  6. Leave the Max RAM and Max Flash as default.
  7. Choose Microphone sensor for Sensor type.
  8. Finally, click on Create to create the project.

4 Step 3 – Datalogging in NanoEdge™ AI Studio

In this step, we will use the programmed STWIN and NanoEdge™ AI Studio to collect the data for each class. The tone generation app will be used to generate the sound when each key is pressed.

Once the project was created, you should see the window as shown below, and you are ready for datalogging.

    main window for data logging in NanoEdge AI Studio

  1. By default, there is already a Class1 added.
  2. Click on Choose Signals to open the "Import your signal" window for this class. Then choose From serial (USB). You should see the "Import your signal" window as shown below:
  3. data logging via serial port

  4. Select your Serial / COM port. Refresh if needed.
  5. Info white.png Information
    COM port number varies on different PCs, you can go to Windows Device Manager to look for STMicroelectronics STLink Virtual COM Port
  6. Select 921600 for Baudrate
  7. Check the box to define Max lines of 200 signals. By doing this, the data logger will automatically stop when there are 200 signals collected.
  8. Leave the Delimiter as default. Open the Tone Generation app on your smart phone, adjust phone volume to about 50%, let the phone speaker face to the STWIN, and press on key "1". Keep pressing the key during data logging. Then click on the Record button to start data logging. The real-time data and plotting will be shown.
  9. Info white.png Information
    • If there's no data shown, unplug the two USB cables and reconnect them to your PC to force the STWIN reset.
    • During data logging, move your phone around STWIN, e.g. circling around, moving far and near, to add variation to the signals.


  10. Once 200 signals are collected, the data logger will stop and you can release the key on the smart phone. click on the Validate import for to validate the data.
  11. Info white.png Information
    If any error is found during the real-time data logging via serial, a pop-up window will appear, letting you delete or edit the lines where issues were detected. Just click Delete this line and then click Validate import again until there's no more error reported.

  12. Then you will return to the main interface, Click on "pencil" icon to rename the class, e.g. "key1". The information of data is shown in the middle column, and the plotting of data is shown on the right column. You can also turn on "Show FFT" to show the FFT plotting.
  13. Click on Add Class and repeat 2-8 to add classes for all the keys, as well as a class for "no key is pressed", so you should have 13 classes in total. The final result is shown below and you are ready for next step.
  14. All 13 classes for DTMF classifier

5 Step 4 – Benchmarking in NanoEdge™ AI Studio

In this step, we will use NanoEdge™ AI Studio to quickly and automatically find the optimized algorithm for our use case.

After collecting the data for all classes, click on "Optimize and Benchmark" in NanoEdge™ AI Studio. You should see the welcome interface as shown below, with the explanations for each measuring metrics.

    NanoEdge AI Studio step 3

  1. Click on the START button to initiate the library selection process
  2. A "Select your Class" window will pop up. Select All classes and click on Validate.
  3. NanoEdge AI Studio select classes
  4. After validation, the Studio will start the searching, you should see the window below:
  5. NanoEdge AI Studio benchmarking in progress
    1. This area indicates the benchmark process:
      • Progress bar
      • Elapsed time
      • Number of libraries tested
      • You can pause the benchmark and resume later
      • You can stop the benchmark when the desired performance is reached
    2. This area shows the performance indicators. The higher balanced accuracy and confidence level are better, while the lower RAM and Flash sizes are better.
    3. This small log window will display notable information and events such as benchmark status, search speed per thread, new libraries found, etc.
    4. This graph shows the evolution of the performance indicators over time, as thousands of candidate libraries are tested.
    Info white.png Information
    Regarding the performance indicators, you can refer to this NanoEdge™ AI Studio documentation for more details.

  6. Once you are satisfied with the performance during the library selection process, click on "Stop" to end the benchmarking. The optimized algorithm will be selected for deployment later.

6 Step 5 – Emulator in NanoEdge™ AI Studio

In this step, we will use the Emulator in NanoEdge™ AI Studio to verify the selected library before we generate and deploy the library files on STWIN.

After finish the benchmarking, click on "Emulator" in NanoEdge™ AI Studio to open the Emulator interface.

NanoEdge AI Studio Emulator


  • Click on the Initialize Emulator button to open the Emulator, as shown below and then the process is similar to Step 3
NanoEdge AI Studio Emulator main interface
  • Select Serial data as we did in Step 3.
  • Select the same COM port and Baudrate, check Max lines for 200 lines.
  • Press a key in the tone generation app on your smart phone, and click on the "Record" button. The classification result should be shown in the interface.
  • Try not pressing any key to see if the "no-key-pressed" is correctly classified, and you can try all other keys.
  • If the classification results are not accute, you should go back to Step 3 to start over the datalogging and benchmarking process.
  • Otherwise you can go to the next step to generate the libarary files for deployment.
Info white.png Information
The picture above only demonstrates classification results of 5 classes. For more information about Emulator, please refer to this NanoEdge™ AI Studio documentation

7 Step 6 – Deploy in NanoEdge™ AI Studio

In this step, we will generate the library files using NanoEdge™ AI Studio.

7.1 Generate library files in NanoEdge™ AI Studio

When you finish the Emulator, click on "Deploy" in NanoEdge™ AI Studio to open the Deploy interface for library generation.

NanoEdge AI Studio  deploy step


  • Check the box for -mfloat-abi (hard) to enable the hardware FPU operations.
  • Click on the Compile button, and there will be a pop-up window as shown below:
NanoEdge AI Studio compile version


  • Click on Development version, then there will be a pop-up window to let you choose a place to save the libraries:
NanoEdge AI Studio save lib files


  • Locate to a local directory you prefer, rename the file if you want, then click on Save.
  • Go to the directory you chose, make sure the file libneai_*.zip is successfully saved there, extract the file, and you are ready for the next step.

7.2 Library files explanation

The NanoEdge AI static library for classification is the code that contains an AI model (for example, as a bundle of signal treatment, machine learning model, optimally tuned hyperparameters, etc.) designed to identify a sensor pattern in a class. All classes are defined by the user in NanoEdge™ AI Studio and are used during the training process of the AI model.

After the zip file is extracted, you should find these folders and files included:

NanoEdge AI Studio generated library files


There are three files we will need for next step: libneai.a, knowledge.h and NanoEdgeAI.h.

  • libneai.a is the C/C++ static library for the classifer.
  • knowledge.h is the header file that contains the knowledge buffer declaration.
  • NanoEdgeAI.h is the header file that contains functions and variable definitions, as well as the class number and names. An example of the code is shown below:
NanoEdge AI library header file code
Info white.png Information
  • The class number may not be in the same order as you recorded in Step 3.
  • There is an "unknown" class automatically generated by the Studio. When calling the classifer NanoEdgeAI_classifier() in the code later, we can define the attribute class_threshold to "1". So that, the function returns the identifier of the class which has a probability strictly greater than 0.5. If there is no probability strictly greater than 0.5 it returns 0 which means that the class is unknown.
  • For more details, refer to this NanoEdge™ AI Library documentation.

8 Step 7 – Program STWIN for Detection

In this step, we will add the library files into the project, import the project into STM32CubeIDE, modify the source code, then program the STWIN for detection.

  1. Replace the old library files by the new ones
    • Go to the library folder, copy the file libneai.a
    • Go to the project folder ..\STWIN_DTMF_Classifier\Core\Src, paste the file here, and click on Replace the file in the destination in the pop-up window.
    • Go back to the library folder, copy the files knowledge.h and NanoEdgeAI.h.
    • Go back to the project folder ..\STWIN_DTMF_Classifier\Core\Inc, paste the files here, and click on Replace the files in the destination in the pop-up window.
  2. Import project into STM32CubeIDE workspace
    • Open STM32CubeIDE on your PC, create or choose a workspace and click Launch to launch your workspace.
    • In the menu bar, click File -> Import...
    • In the pop-up window, select Existing Projects into Workspace, and click Next, as shown below:
    Pop-up window when importing project in STM32CubeIDE

    • In the next interface, click Browse..., find the project folder where you saved and then click Select Folder, then click Finish and wait for the project to be imported, as shown in the picture below:
    Selecting folder when importing project in STM32CubeIDE

  3. Build Project and Program the STWIN
    • In STM32CubeIDE, unfold the project in Project Explorer, find ..\Core\Src\main.c and double click to open it in the Editor, as shown below:
    STM32CubeIDE Project Explorer

    • In main.c, go to line 48 to find #define NEAI_MODE. Change the value to "1" if it is not, as shown below:
    NanoEdge project main.c line 48

    • Press CTRL + S to save the change. Make sure the STWIN and STLINK_V3MINI are connected to PC via USB cables, and then click on the "Run" button in the Toolbar to build and program the project:
    STM32CubeIDE Run button

    • You can see the log information in the Console window. When the project is successfully built and programmed on the STWIN, you should see the following information:
    Download verified successfully
    
    Debugger connection lost.
    Shutting down...
    
  4. Now you are ready for the final step - to run the DTMF classifier on STWIN.

9 Step 8 – Run DTMF classifier on STWIN

In this step, we will run the DTMF classifier on STWIN and use Tera Term to display the classification results.

  • Once the STWIN is successfully programmed for detection, you can open the Tera Term on PC.
  • In Tera Term's menu bar, click on Setup -> Serial port.... The serial port setup window will pop up as shown below:
Tera Term Serial port setup
  • Select the same COM port as we did in Step 3 for datalogging.
  • Select the baudrate value "921600" for Speed.
  • Leave other settings as default, and click on New open.
  • Bring out your smart phone, and press any key in the tone generation app. The classification results will show in the interface, along with the probability of each class. When there's a key pressed, it will show the corresponding class. When the key is released, it will show the "no-key-pressed" class, as shown below:
Tera Term classification results

10 Resources

25px Important
Firmware package delivered in this article is delivered under the Mix Ultimate Liberty+OSS+3rd-party V1 software license agreement (SLA0048).

11 References