How to create a dual-tone multi-frequency classifier using NanoEdge AI Studio

This article is a tutorial on how to use NanoEdge™ AI Studio to create a dual tone multi frequency (DTMF) classifier running on the STWIN SensorTile Wireless Industrial Node development kit (STEVAL-STWINKT1B).

This tutorial shows you how to program the STWIN for datalogging; how to use NanoEdge™ AI Studio to collect data, label data, find the best algorithm, run the Emulator, generate the library files for deployment; how to use STM32CubeIDE to modify codes, build the project, program the STWIN; and finally run the classifier on STWIN. The main process of this tutorial is shown in the flowchart below:

1. Prerequisites

1.1. Hardware

• STEVAL-STWINKT1B, the STWIN development kit,
• Windows^® powered laptop/PC (Windows^® 7, 8, or 10),
• Two Micro-USB cables (one to connect the STWIN to PC, and another one for the STLINK-V3MINI)
• STLINK-V3MINI, which is included in the dev kit.

1.2. Software

1.2.1. Source Code

• A firmware package stwin-dtmf-classifier-wiki-package.zip is ready to download from st.com, please download the zip file here. The package includes:

• Project source code in subfolder STWIN_DTMF_Classifier
• Two Pre-compiled binary files in subfolder binary:

• STWIN_DTMF_Classifier-datalog.bin for datalogging
• STWIN_DTMF_Classifier-detection.bin for tone detection

• Library files generated by NanoEdge™ AI Studio in subfolder libneai_STWIN-DTMF-Classifier.zip

Information

If you have the firmware package from ST's Machine Learning at the Edge with NanoEdge AI Studio workshop, the contents are almost the same except some additional documentations. In this article, we will only use the datalogging binary and the project source code. The other files are for your reference.

1.2.2. NanoEdge™ AI Studio

• NanoEdge™ AI Studio is a new Machine Learning (ML) technology to bring true innovation easily to the end-users. In just a few steps, developers can create an optimal ML library for their project, based on a minimal amount of data.
• Download NanoEdge™ AI Studio as a free trial version for the STMicroelectronics boards.

Information

If you need to run NanoEdge™ AI Studio on a virtual machine, please submit a request on our Online Support platform. To ensure your request is quickly routed to the correct support team, please select Product / Application for Category, and NanoEdgeAIStudio for Product Category/Part Number.

1.2.3. STM32CubeIDE

• STM32CubeIDE is an all-in-one multi-OS development tool, which is part of the STM32Cube software ecosystem. It is an advanced C/C++ development platform with peripheral configuration, code generation, code compilation, and debug features for STM32 microcontrollers and microprocessors.
• Download STM32CubeIDE from st.com

1.2.4. Tera Term

• Tera Term is an open-source and freely available software terminal emulator, which is used to display the classification results.
• Users can download and install the latest version available from Tera Term.

1.2.5. Tone Generation App

Install the corresponding app on your smart phone for DTMF tone generation:

• iOS: Dialer by DigitalSand.
• Android DTMF Tone Generator by Guillaume Desbieys.

2. Step 1 - Program STWIN for datalogging

The first step is to connect the STWIN to your PC and program it for datalogging mode with the provided binary file.

2.1. Board connection

There are three connections to make on the STWIN, as shown in the picture:

1. Connect the STDC14 cable between the STWIN and STLINK-V3MINI.
2. Plug the USB cable micro-B side into the STWIN.
3. Plug the USB cable micro-B side into the STLINK-V3MINI.
4. Plug the STWIN USB cable into PC.
5. Plug the STLINK-V3MINI USB cable into PC.

2.2. Program STWIN using binary file

The easiest way to program the STWIN is using the drag-and-drop. Once you have the pre-built binary file in your local disk, the STWIN can be very easily programmed with the binary by simply performing a drag-and-drop action:

1. In Windows File Explorer, find the binary file ..\STWIN-DTMF-Classifier-wiki-package\binary\STWIN_DTMF_Classifier-datalog.bin
2. Left click on the file and hold the left button on your mouse
3. Find the STLINK_V3M disk in the Navigation pane on the left hand side, drag the binary to that disk and release the left button
4. When the transfer is done, the STWIN is programmed successfully.

Information

If the drag-and-drop method doesn't work, there are two other alternative methods to program the STWIN: Using STM32CubeProgramer with the binary. Refer to user manual UM2237 for how to use STM32CubeProgramer. Using STM32CubeIDE with source code. Refer to #Step 7 – Program STWIN for Detection in this article. If you use this method, please make sure #define NEAI_MODE in main.c is defined as datalogging mode.

3. Step 2 - Create a Classification Project in NanoEdge™ AI Studio

In this step, we will create a classification project in NanoEdge™ AI Studio, and configure the project settings.

Open NanoEdge™ AI Studio on PC, you should see the welcome interface below and then follow the steps:

1. Click on n-Class Classification to choose it as project type.
2. Click on CREATE NEW PROJECT to create the project.

You should see the Project settings page as below:

1. Give the project a name, e.g. STWIN-DTMF-Classifier
2. Add a brief description for the project.
3. Choose STEVAL-STWINKT1B for Target.
4. Leave the Max RAM as default.
5. Choose No Flash limit
6. Choose Microphone sensor for Sensor type.
7. Finally, click on SAVE & NEXT to go to the next step.

4. Step 3 - Datalogging in NanoEdge™ AI Studio

In this step, we will use the programmed STWIN and NanoEdge™ AI Studio to collect the data for each class. The tone generation app will be used to generate the sound when each key is pressed.

Once the project was created, you should see the window as shown below, and you are ready for datalogging.

1. Click on ADD SIGNAL to open the "Import signal" window.



2. In the pop-up window, choose FROM SERIAL (USB) as Type of signal source.



3. In the next window, select your Serial / COM port. Refresh if needed.

Information

Virtual COM port in Windows Device Manager COM port number varies on different PCs. In this example, it is COM20. You can go to Windows Device Manager to look for STMicroelectronics STLink Virtual COM Port under Ports (COM & LPT)

4. Select 921600 for Baudrate
5. Check the box to define Max lines of 100 signals. By doing this, the data logger will automatically stop when 100 lines of signal are collected.
6. Give a class name, which will be used to shown as classification result later.
7. Open the Tone Generation app on your smart phone, adjust phone volume to about 50%, let the phone speaker face to the STWIN, and press on key "1". Keep pressing the key during data logging. Then click on the START/STOP button to start data logging. The real-time data and plotting will be shown.

Information

If there's no data shown, unplug the two USB cables and reconnect them to your PC to force the STWIN reset. During data logging, move your phone around STWIN, e.g. circling around, moving far and near, to add variation to the signals.





8. Once 100 lines of signal are collected, the data logger will stop and you can release the key on the smart phone. click on CONTINUE to the next step.



9. In the signal preview step, you may see a colored box, which means there's an error in that line. Simply click on that box to delete that line of signal
10. Then, click on IMPORT to finish datalogging for this class.

Repeat step 1-10 to add classes for all the keys, as well as a class for "no key is pressed", so you should have 13 classes in total. The final result is shown below and you are ready for next step.

5. Step 4 - Benchmarking in NanoEdge™ AI Studio

In this step, we will use NanoEdge™ AI Studio to quickly and automatically find the optimized algorithm for our use case.

After collecting the data for all classes, click on "Benchmark" in NanoEdge™ AI Studio. You should see the welcome interface as shown below:

1. Click on RUN NEW BENCHMARK


2. A "Select your Class" window will pop up. Select All classes and click on START.

3. Then the Studio will start the benchmarking, you should see the window below:




1. This area indicates the benchmark status:
   • How many benchmarks have run
   • Number of libraries tested
   • Elapsed time
2. This area shows the process of the running benchmark, and the time stamp of it
3. This area shows the performance indicators. The higher balanced accuracy and confidence level are better, while the lower RAM and Flash sizes are better.
4. This small log window will display notable information and events such as benchmark status, search speed per thread, new libraries found, etc.
5. This graph shows the evolution of the performance indicators over time, as thousands of candidate libraries are tested.
6. This area contains control buttons:
   • You can pause the benchmark and resume later
   • You can stop the benchmark when the desired performance is reached

Information

Regarding the performance indicators, you can refer to this NanoEdge™ AI Studio wiki for more details.



4. Once you are satisfied with the performance during the library selection process, click on the "STOP" button to end the benchmarking. The optimized algorithm will be selected for deployment later. You should see the details of this benchmark as below:

6. Step 5 - Emulator in NanoEdge™ AI Studio

In this step, we will use the Emulator in NanoEdge™ AI Studio to verify the selected library before we generate and deploy the library files on STWIN.

After finish the benchmarking, click on "Emulator" in NanoEdge™ AI Studio to open the Emulator interface.

• Click on the INITIALIZE EMULATOR button to open the Emulator, as shown below and then the process is similar to Step 3.

• Select Serial data as we did in Step 3.
• Select the same COM port and Baudrate.
• Press a key in the tone generation app on your smart phone, and click on the "START/STOP" button. The classification result should be shown in the interface.
• Try not pressing any key to see if the "no-key-pressed" is correctly classified, and you can try all other keys.

• If the classification results are not accurate, you should go back to Step 3 to start over the datalogging and benchmarking process.
• Otherwise you can go to the next step to generate the library files for deployment.

Information

For more information about the Emulator, please refer to this NanoEdge™ AI Studio wiki

7. Step 6 - Deploy in NanoEdge™ AI Studio

In this step, we will generate the library files using NanoEdge™ AI Studio.

7.1. Generate library files in NanoEdge™ AI Studio

When you finish the Emulator, click on "Deployment" in NanoEdge™ AI Studio to open the Deploy interface for library generation.

• If you have multiple benchmarks, choose the one you want to compile. Otherwise, leave it default.
• Check the box for Float abi to enable the hardware FPU operations.
• Click on the COMPILE LIBRARY button, and there will be a pop-up window as shown below:

• Choose the DEMO VERSION by clicking on it, then there will be another pop-up window letting you to choose a local directory to save the libraries:

• Locate to a local directory you prefer, rename the file as you want, then click on Save.
• Go to the directory you chose, make sure the file libneai_*.zip is successfully saved there, extract the file, and you are ready for the next step.

7.2. Library files explanation

The NanoEdge AI static library for classification is the code that contains an AI model (for example, as a bundle of signal treatment, machine learning model, optimally tuned hyperparameters, etc.) designed to identify a sensor pattern in a class. All classes are defined by the user in NanoEdge™ AI Studio and are used during the training process of the AI model.

After the zip file is extracted, you should find these folders and files included:

There are three files we will need for next step: libneai.a, knowledge.h and NanoEdgeAI.h.

• libneai.a is the C/C++ static library for the classifier.
• knowledge.h is the header file that contains the knowledge buffer declaration.
• NanoEdgeAI.h is the header file that contains function prototypes and sample declarations. An example of the code is shown below:

Information

Starting from NanoEdge™ AI Studio v3, user has to declare the input, output and class buffer in the main program. The class number may not be in the same order as you recorded in Step 3. There is an automatically generated "unknown" class, which indicates when the probability is less than 50% for each known class. For more details, refer to this NanoEdge™ AI Library for n-class classification wiki.

8. Step 7 - Program STWIN for Detection

In this step, we will add the library files into the project, import the project into STM32CubeIDE, modify the source code, then program the STWIN for detection.

1. Replace the old library files by the new ones

• Go to the library folder, copy the file libneai.a
• Go to the project folder ..\STWIN_DTMF_Classifier\Core\Src, paste the file here, and click on Replace the file in the destination in the pop-up window.
• Go back to the library folder, copy the files knowledge.h and NanoEdgeAI.h.
• Go back to the project folder ..\STWIN_DTMF_Classifier\Core\Inc, paste the files here, and click on Replace the files in the destination in the pop-up window.

2. Import project into STM32CubeIDE workspace

• Open STM32CubeIDE on your PC, create or choose a workspace and click Launch to launch your workspace.
• In the menu bar, click File -> Import...
• In the pop-up window, select Existing Projects into Workspace, and click Next, as shown below:





• In the next interface, click Browse..., find the project folder where you saved and then click Select Folder, then click Finish and wait for the project to be imported, as shown in the picture below:




3. Build Project and Program the STWIN

• In STM32CubeIDE, unfold the project in Project Explorer, find ..\Core\Src\main.c and ..\Core\Inc\NanoEdgeAI.h. Double click on the two files separately to open them in the Editor, as shown below:





• In main.c, go to line 48 to find #define NEAI_MODE. Change the value to "1" if it is not, as shown below:





• Then go to NanoEdgeAI.h, find the buffer declarations in the commented part underneath, select the codes. Right click and choose "Copy" or simply press Ctrl+C on the keyboard to copy the codes:





• Go back to main.c, paste the copied codes between /* USER CODE BEGIN PV */ and /* USER CODE END PV */. If there are existing declarations, delete them and paste your own codes from previous step. As shown below:





• Press CTRL + S to save the changes. Make sure the STWIN and STLINK_V3MINI are connected to PC via USB cables, and then click on the "Run" button in the Toolbar to build and program the project:





• You can see the log information in the Console window. When the project is successfully built and programmed on the STWIN, you should see the following information:

Download verified successfully

Debugger connection lost.
Shutting down...

4. Now you are ready for the final step - to run the DTMF classifier on STWIN.

9. Step 8 - Run DTMF classifier on STWIN

In this step, we will run the DTMF classifier on STWIN and use Tera Term to display the classification results.

• Once the STWIN is successfully programmed for detection, you can open the Tera Term on PC.
• In Tera Term's menu bar, click on Setup -> Serial port.... The serial port setup window will pop up as shown below:

• Select the same COM port as we did in Step 3 for datalogging.
• Select the Baudrate value "921600" for Speed.
• Leave other settings as default, and click on New open.

• Bring out your smart phone, and press any key in the tone generation app. The classification results will show in the interface, along with the probability of each class. When there's a key pressed, it will show the corresponding class. When the key is released, it will show the "no-key-pressed" class, as shown below:

10. Resources

• Firmware Package - Project source code and pre-compiled binary files
• NanoEdge™ AI Studio wiki
• NanoEdge™ AI Library for n-class classification (nCC) wiki
• UM2777 - How to use the STEVAL-STWINKT1B SensorTile Wireless Industrial Node for condition monitoring and predictive maintenance applications
• UM2237 STM32CubeProgrammer software description
• UM2609 STM32CubeIDE user guide