|
Delivery for this function pack is being prepared |
Sensing and condition monitoring are two of the major components of the IoT and predictive maintenance systems, enabling context awareness, production performance improvement, maintenance cost reduction and a drastic decrease of the downtime due to routine maintenance.
The FP-AI-MONITOR1 function pack helps to jump-start the implementation and development of classification and condition monitoring applications designed with X-CUBE-AI Expansion Package for STM32Cube and with the NanoEdge™ AI Studio solution.
FP-AI-MONITOR1 supports on-device learning for the anomaly detection using NanoEdge AI libraries, and on-device inference for both classification problems based on classical machine learning and Neural Networks as well as for the anomaly detection using NanoEdge AI studio. The evaluation board supported is a sensor node STEVAL-STWINKT1B with STM32L4R9ZI ultra-low power microcontroller. It covers the entire design of the Machine Learning cycle from the data set acquisition to the integration on a physical node.
FP-AI-MONITOR1 implements a wired interactive CLI to configure the node, and manage the learn and detect phases. In addition, for simple operation in the field, a standalone battery-operated mode is supported, which enables basic controls through the user button without using the console.
This article provides an overview of the following topics:
- The required hardware and software,
- Setting up the hardware and software components,
- Performing the vibration sensor data collection using a prebuilt binary of FP-SNS-DATALOG1,
- Running a One-class SVM model for anomaly detection on the device
- Building the model using Scikit-learn,
- Exporting as onnx model using onnx-runtime,
- Generating C code using X-CUBE-AI, and
- Running on STEVAL-STWINKT1B .
- Generating the library using NanoEdge™ AI Studio, installing and testing the NanoEdge™ AI machine learning libraries on the STEVAL-STWINKT1B using the provided function pack, and
- Some links to useful online resources, to help the user better understand and customize the project for her/his own needs.
For full FP-AI-MONITOR1 user instructions, please refer to this article.
1. Hardware and software overview
1.1. SensorTile wireless industrial node Evaluation Kit STEVAL-STWINKT1B
The STWIN SensorTile wireless industrial node (STEVAL-STWINKT1B) is a development kit and reference design that simplifies the prototyping and testing of advanced industrial IoT applications such as condition monitoring and predictive maintenance. It is powered with an ultra-low-power Arm® Cortex®-M4 MCU at 120 MHz with FPU, 2048-Kbyte Flash memory (STM32L4R9). STWIN SensorTile is equipped with a microSD™ card slot for standalone data logging applications. STWIN SensorTile also features a wide range of industrial IoT sensors including:
- an ultra-wide bandwidth (up to 6 kHz), low-noise, 3-axis digital vibration sensor (IIS3DWB)
- a 6-axis digital accelerometer and gyroscope iNEMO inertial measurement unit (IMU) with machine learning core (ISM330DHCX)
and much more. Please refer to this link for all the sensors and features supported by STWIN. Other attractive features include:
- a 480 mAh Li-Po battery to enable standalone working mode
- STLINK-V3MINI debugger with programming cable to flash the board
- a plastic box for ease of placing and planting the SensorTile on the machines for condition monitoring. For further details, visit this link
1.2. FP-AI-MONITOR1 software description
The top-level architecture of the FP-AI-MONITOR1 function pack is shown in the following figure.
2. Prerequisites and setup
2.1. Hardware prerequisites and setup
To use the FP-AI-MONITOR1 function pack on STEVAL-STWINKT1B, the following hardware items are required:
- STEVAL-STWINKT1B development kit board,
- Windows® powered laptop/PC (Windows® 7, 8, or 10),
- Two Micro-USB cables, one to connect the sensor-board to the PC, and another one for the STLINK-V3MINI, and
- an STLINK-V3MINI.
2.2. Software requirements
2.2.1. FP-AI-MONITOR1
- Download the FP-AI-MONITOR1 package from ST website, extract and copy the .zip file content into a folder on your PC. The package contains binaries and source code for the sensor board STEVAL-STWINKT1B.
2.2.2. IDE
- Install one of the following IDEs:
- STMicroelectronics STM32CubeIDE version 1.5.0
- IAR Embedded Workbench for Arm (EWARM) toolchain V8.50.5 or later
- RealView Microcontroller Development Kit (MDK-ARM) toolchain V5.31
2.2.3. STM32CubeProgrammer
- STM32CubeProgrammer (STM32CubeProg) is an all-in-one multi-OS software tool for programming STM32 products. It provides an easy-to-use and efficient environment for reading, writing and verifying device memory through both the debug interface (JTAG and SWD) and the bootloader interface (UART, USB DFU, I2C, SPI, and CAN). STM32CubeProgrammer offers a wide range of features to program STM32 internal memories (such as Flash memory, RAM and OTP) as well as external memories.
- This software can be downloaded from STM32CubeProg.
2.2.4. TeraTerm
- TeraTerm is an open-source and freely available software terminal emulator, which is used to host the CLI of the FP-AI-MONITOR1 through a serial connection.
- Download and install the latest version available from TeraTerm.
2.2.5. STM32CubeMX
STM32CubeMX is a graphical tool that allows a very easy configuration of STM32 microcontrollers and microprocessors, as well as the generation of the corresponding initialization C code for the Arm® Cortex®-M core or a partial Linux® Device Tree for Arm® Cortex®-A core), through a step-by-step process. Its salient features include:
- Intuitive STM32 microcontroller and microprocessor selection.
- Generation of initialization C code project, compliant with IAR™, Keil® and STM32CubeIDE (GCC compilers) for Arm® Cortex®-M core
- Development of enhanced STM32Cube Expansion Packages thanks to STM32PackCreator, and
- Integration of STM32Cube Expansion Packages into the project.
For downloading and details of all the features please visit st.com.
2.2.6. X-Cube-AI
X-CUBE-AI is an STM32Cube Expansion Package part of the STM32Cube.AI ecosystem and extending STM32CubeMX capabilities with automatic conversion of pre-trained Artificial Intelligence models and integration of generated optimized library into the user's project. The easiest way to use it is to download it inside the STM32CubeMX tool (version 7.0.0 or newer) as described in user manual Getting started with X-CUBE-AI Expansion Package for Artificial Intelligence (AI) (UM2526). The X-CUBE-AI Expansion Package also offers several means to validate the AI models (Neural Network and Scikit-Learn models) both on desktop PC and STM32, as well as measure performance on STM32 devices (computational and memory footprints) without user handmade ad-hoc C code.
2.2.7. PythonTM 3.7.X
Python is an interpreted high-level general-purpose programming language. Python's design philosophy emphasizes code readability with its notable use of significant indentation. Its language constructs as well as its object-oriented approach aim at helping programmers write clear, logical code for small and large-scale projects. To build and export the Onnx models, set up a Python environment with a list of packages. The list of the required packages along with their versions can be found as a text file in the package at location /FP-AI-MONITOR1/Utilities/AI_resources/requirements.txt.
2.2.8. NanoEdge AI Studio
The function pack lets the user use the AI libraries generated by NanoEdgeTM AI Studio powered by Cartesiam. Generate the libraries from the NanoEdge AI Studio and then embed these libraries in the FP-AI-MONITOR1. The Studio can be downloaded from Cartesiam.ai as a free trial version for the STMicroelectronics boards.
2.3. Installing the function pack to STWIN
2.3.1. Getting the function pack
The first step is to get the function pack. Download the FP-AI-MONITOR1 from ST website. Once the pack is downloaded, unpack/unzip it and copy the content to a folder on the PC. The steps of the process along with the content of the folder are shown in the following image.
2.3.2. Flashing the application on the sensor board STEVAL-STWINKT1B
Once the package has been downloaded and unpacked, the next step is to program the sensor node with the binary of the function pack. For the convenience of the users, the function pack is equipped with a pre-built binary file of the project. This binary file can be found at path /FP-AI-MONITOR1/Projects/Binary/FP-AI-MONITOR1.bin. The sensor board can be very easily programmed with the provided binary by simply performing a drag-and-drop action as shown in the figure below.
3. FP-AI-MONITOR1 console application
FP-AI-MONITOR1 provides an interactive command-line interface (CLI) application. This CLI application equips a user with the ability to configure and control the sensor node, to perform learning (for NanoEdge™ AI libraries), and (anomaly) detection operations on the edge. The following sections provide a small guide on how to install this CLI application on a sensor board and control it through the CLI from TeraTerm.
3.1. Setting up the console
Once the sensor board is programmed with the binary of the project (as shown in section 2), the next step is to set up the serial connection of the board with the PC through TeraTerm. To do so, start TeraTerm and create a new connection by either selecting it from the toolbar or by selecting the proper port to establish the serial communication with the board. In the figure below, this is COM10 - USB Serial Device (COM 10).
Once the connection is established, the message below is displayed. If this is not the case, reset the board.
Typing help shows the list of all the available commands along with their usage instructions.
3.2. Configuring the sensors for sensing
Through the CLI interface, a user can configure the supported sensors for sensing and condition monitoring applications. The list of all the supported sensors can be displayed on the CLI console by entering the command:
$ sensor_info
This command prints the list of the supported sensors along with their ids as shown in the image below. The user can configure these sensors using these ids. The configurable options for these sensors and subsensors include:
- enable: to activate or deactivate the sensor,
- ODR: to set the output data rate of the sensor from the list of available options, and
- FS: to set the full-scale range from the list of available options.
The current value of any of the parameters for a given sensor can be printed using the command,
$ sensor_get <sensor_id> <param>
or all the information about the sensor can be printed using the command:
$ sensor_get <sensor_id> all
Similarly, the values for any of the available configurable parameters can be set through the command:
$ sensor_set <sensor_id.sub-sensor_id> <param> <val>
The figure above shows the complete example of getting and setting these values along with old and changed values.
4. Classification applications for sensing
4.1. Running the AI application
The CLI application comes with a prebuilt Human Activity Recognition model. This functionality can be started by typing the command:
$ start ai
Note that the provided HAR model is built with a dataset created using the IHM330DHCX_ACC sensor with ODR = 26, and FS = 4. To achieve good performance, the user is required to set these parameters to the sensor configurations using the instructions provided in section Sensor Configuration.
Running the $ start ai command starts the inference on the accelerometer data and predicts the performed activity along with the confidence. The supported activities are:
- Stationary,
- Walking,
- Jogging, and
- Biking.
The following screenshot shows the normal working session of the AI command in CLI application.
4.2. Replacing the provided classification AI model
A set of Python scripts is provided in the package to prepare the SVC and CNN models for Human Activity Recognition (HAR), although to facilitate the usage, two pretrained models are also provided in the /FP-AI-MONITOR1/Utilities/AI_Resources/models/, along with generated C-code. The more advanced users can use these scripts and build their own models by playing with different parameters or by using their own datasets.
The prebuilt model provided in the CLI example binary is based on the Support Vector Machine. The Python script along with detailed instructions on how to prepare the dataset, build the model and convert it to onnx, is provided in the Jupyter Notebook available in /FP-AI-MONITOR1/Utilities/AI_Resources/Training Scripts/HAR/HAR_With_SVC.ipynb. The building of these models and C-Code generation is out of the scope of this article and readers are invited to refer to the user manual for detailed instructions on how to do so. This article describes how to replace the model provided in the function pack by default by the user-generated model once the C-code has been generated using STM32CubeMX extension X-CUBE-AI. In this example, the provided SVC model is replaced with the prebuilt and converted CNN model available at path /FP-AI-MONITOR1/Utilities/AI_Resources/models/IGN4class/. Full details on the data used for training and the topology can be found in the training scripts provided under /FP-AI-MONITOR1/Utilities/AI_Resources/Training Scripts/HAR/HAR_With_CNN.ipynb.
To update the model, copy and replace the following files in the folder /FP-AI-MONITOR1_V1.0.0/Projects/STM32L4R9ZI-STWIN/Applications/FP-AI-MONITOR1/X-CUBE-AI/App/ as shown in the figure above:
- app_x-cube-ai.c
- app_x-cube-ai.h
- network.c
- network.h
- network_config.h
- netowrk_data.h
- network_generate_report.h
Once copied the files, open the project with the STM32CubeIDE. To do so, go to the folder /FP-AI-MONITOR1/Projects/STWINL4R9ZI-STWIN/Applications/FP-AI-MONITOR1/STM32CubeIDE/ and double click .project file as shown in the figure below.
Once the project is opened, go to FP-AI-MONITOR1_prj/Application/X-CUBE-AI/app/app_x-cube-ai.c file and comment the following lines of the code as shown in the figure below.
4.2.1. Building and installing the project
Then build and install the project in STWIN sensor board by pressing the play button as shown in the figure below.
A message saying Download verified successfully indicates the new firmware is programmed in the sensor board.
5. Condition monitoring using NanoEdge AI Machine Learning library
This section provides a complete method to set up an application for condition monitoring using FP-AI-MONITOR1. The diagram below provides an overview of the required steps:
5.1. Generating a condition monitoring library
This function pack also supports the AI condition monitoring library generated and provided by NanoEdge AI Studio, a solution provided by Cartesiam. This section provides a step-by-step guide on how to generate and install these libraries by linking them to the provided project in FP-AI-MONITOR1, and then use them to perform the condition monitoring on the edge by running them first in training mode and then in detection mode.
5.1.1. Data logging for normal and abnormal conditions
The details on how to acquire data are provided in section Datalogging using HSDatalog.
5.1.2. Data preparation for library generation with NanoEdge AI Studio
The data logged through the datalogger is in binary format. As it is, it is not user-readable nor compliant with the NanoEdge AI studio format. To convert these data to a useful form, FP-AI-MONITOR1 provides Python™ utility scripts available at path /FP-AI-MONITOR1_V1.0.0/Utilities/AI-resources/NanoEdgeAi/.
The provided Jupyter Notebook NanoEdgeAI_Utilities.ipynb provides a complete example of data preparation for library generation for a fan with three speeds running in normal and clogged conditions. In addition a HSD_2_NEAISegments.py is provided if the user wants to prepare segments for a given data acquisition. This script can be used by issuing the following command for a data acquisition with ISM330DHCX_ACC sensor:
python HSD_2_NEAISegments.py ../Datasets/Fan12CM/ISM330DHCX/normal/1000RPM/
This command generates a file named ISM330DHCX_Cartesiam_segments_0.csv using the default parameter set. The file is generated with segments of length 1024 and stride 1024, with the first 512 samples skipped from the file.
5.1.3. Library generation using NanoEdge AI Studio
Running the scripts in Jupyter Notebook generates the normal_WL1024_segments.csv and clogged_WL1024_segments.csv files. This section describes how to generate the libraries using these normal and abnormal segment files.
The sequence to generate the libraries with Cartesiam NanoEdge AI studio consists of five steps.
- Hardware description
- Choose the target platform or a microcontroller type such as STEVAL-STWINKT1B Cortex-M4
- Configure the maximum amount of RAM to be allocated for the library: usually a few Kbytes suffices but it depends on the data frame length used in the data preparation process. Start with 32 Kbytes.
- Define the sensor type : 3-axis accelerometer
- Provide the sample contextual data to adjust and gauge the performance of the chosen model. This step requires data for:
- Nominal or normal case (
normal_WL1024_segments.csv) - Abnormal case (
clogged_WL1024_segments.csv)
- Nominal or normal case (
- Benchmark available models and choose the one that complies with the requirements.
- Validate the model for learning and testing through the provided emulator, which emulates the behavior of the library on the edge.
- Then compile and download the libraries. In this process, the flag
"-mfloat-abi"has to becheckedfor using libraries with hardware FPU. All the other flags can be left to the default state.- Note: For using the library with μKeil also check -fshort-wchar in addition to -mfloat-abi.
Detailed documentation is available at NanoEdge AI Studio.
.
5.2. Installing the NanoEdgeAI Machine Learning library
Once the libraries are generated and downloaded from Cartesiam NanoEdge AI Studio, the next step is to link these libraries to FP-AI-MONITOR1 and run them on the STWIN. The FP-AI-MONITOR1 comes with the library stubs in the place of the actual libraries generated by NanoEdge AI Studio. This is done to ease the linking of the generated libraries. In order to link the actual libraries, copy the generated libraries and replace the existing stub/dummy libraries and header files NanoEdgeAI.h, and libneai.a files present in the folders Inc, and lib, respectively. The relative paths of these folders are /FP_AI_MONITOR1_V1.0.0/Middlewares/ST/NanoEdge_AI_Library/ as shown in the figure below.
Once these files are copied, the project must be reconstructed and programmed on the sensor board to link the libraries. For this, open the project in STM32CubeIDE. To do so, go to the folder /FP-AI-MONITOR1/Projects/STWINL4R9ZI-STWIN/Applications/FP-AI-MONITOR1/STM32CubeIDE/ and double click .project file as shown in the figure below.
To build and install the project, click the play button and wait for the successful download message as shown in the section Build and Install Project.
Once the sensor board is successfully programmed, the welcome message is displayed in the CLI (TeraTerm terminal). If the message does not appear, reset the board by pressing the RESET button.
5.3. Testing the NanoEdgeAI Machine Learning library
Once the STWIN is programmed with the firmware containing a valid library, the condition monitoring libraries are ready to be tested on the sensor board.
The learning phase can be started by simply issuing a start neai_learn command from the CLI console. Launching this command shows the process of learning and displays a message on the console every time learning is performed on a new signal (see the below snippet):
$ start neai_learn
NanoEdgeAI: starting
$ {"signal": 1, "status": success}
{"signal": 2, "status": success}
{"signal": 3, "status": success}
:
:
The process can be stopped by pressing the ESC key on the keyboard.
The learning can also be started by performing a long-press of the user button. Pressing the user button again stops the learning process.
Once the normal conditions are learned, the user can start the condition monitoring process by issuing the command:
$ start neai_detect NanoEdgeAI: starting
or by double-pressing the user button on the sensor board. If the similarity of the current signal is less than the 90% threshold, the similarity is printed on the CLI console indicating that the signal is an anomaly as shown in the snippet below. Nothing is printed in the CLI console if the similarity is higher than 90% to the normal signals.
5.3.1. Additional parameters in condition monitoring
For the convenience of the users, the CLI application also provides some handy options to easily fine-tune the inference and learning process of the condition monitoring. Users can see all the variables they can play with by issuing the following command:
$ neai_get all NanoEdgeAI: signals = 0 NanoEdgeAI: sensitivity = 1.000000 NanoEdgeAI: threshold = 95 NanoEdgeAI: timer = 0
Each of the these parameters can be configured using the neai_set <param> <val> command. For details on these parameters and commands go to FP-AI-MONITOR1 user-manual.
This section provides usage information on neai_set signals <nr of signals> . Using this command, a user can start the learning or condition monitoring mode for a given number of samples. For example, to learn on 10 signals, issue this command before issuing the learn command as shown below:
$ neai_set signals 10
NanoEdge AI: signals set to 10
$ start neai_learn
NanoEdgeAI: starting
$ {"signal": 1, "status": success}
{"signal": 2, "status": success}
{"signal": 3, "status": success}
:
:
{"signal": 10, "status": success}
NanoEdge AI: stopped
The threshold parameter is used to report the anomalies. For any signal which has similarities below the threshold value is reported as anomaly. The default value used in the CLI application is 90. Users can change this value by using the neai_set threshold <val> command.
The sensitivity parameter is used as an emphasis parameter. The default value is set to 1. Increasing this sensitivity means that the matching of the signals is to be performed more strictly, reducing it also reduces the matching calculation process and results in higher matching values.
For further details on how NanoEdge AI libraries work, read the detailed documentation of NanoEdge AI studio.
6. Data collection
The data collection functionality is out of the scope of this function pack. However, to facilitate the users and equip them with the possibility to perform a datalog, a precompiled .bin file for FP-SNS-DATALOG1 is provided in the Utilities and can be found under path /FP-AI-MONITOR1_V1.0.0/Utilities/Datalog/. To acquire the data from any of the available sensors on the sensor-board, the user can program the sensor-board with this binary using the drag-drop method shown in section 2.3.2.
Once the sensor-board is programmed, the data can be logged using the following instructions.
- Place a
DeviceConfig.jsonfile with the configurations of the sensors to be used for the datalogging in the root folder of the microSDTM card. Some sample.jsonfiles are provided in the package and can be found at the locationFP-AI-MONITOR1_V1.0.0/Utilities/Datalog/Sample_STWIN_Config_Files/. These files are to be named exactlyDeviceConfig.json. - Insert the SD card into the STWIN board.
- Reset the board. Orange LED blinks once per second. The custom sensor configuration provided in
DeviceConfig.jsonis loaded from the file. - Press the [USR] button to start data acquisition on the SD card. The orange LED turns off and the green LED starts blinking to signal sensor data are being written into the SD card.
- Press the [USR] button again to stop data acquisition. Do not unplug the SD card or turn the board off before stopping the acquisition or the data on the SD card will be corrupted.
- Remove the SD card and insert it into an appropriate SD card slot on your PC. The log files are stored in
STWIN_###folders for every acquisition, where###is a sequential number determined by the application to ensure log file names are unique. Each folder contains a file for each active subsensor calledSensorName_subSensorName.dat. Each file contains raw sensor data coupled with timestamps, aDeviceConfig.jsonwith specific information about the device configuration that are necessary for correct data interpretation, and anAcquisitionInfo.jsonwith information about the acquisition.
7. Button operated modes
To facilitate the usage, the FP-AI-MONITOR1 is equipped with a button-operated mode. The purpose of the button-operated mode is to enable the FP-AI-MONITOR1 to be operated even in the absence of the CLI console. All the (default) values required for running different functions are provided in the firmware. In button-operated mode, the user can start/stop different execution phases by using only the user button. The following table shows the user instruction for the button-operated mode.
| Button Press | Description | Action |
|---|---|---|
| SHORT_PRESS | The button is pressed for less than 200 ms and released | Starts AI inferences for X-CubeAI model. |
| LONG_PRESS | The button is pressed for more than 200 ms and released | Starts the learning for NanoEdgeTM AI Library. |
| DOUBLE_PRESS | A succession of two SHORT_PRESS in less than 500 ms | Starts the inference for NanoEdgeTM AI Library. |
| ANY_PRESS | The button is pressed and released (overlaps with the three other modes) | Stops the current running execution phase. |
The status of the current execution phase is indicated to the user through the onboard LEDs. The LEDs are allocated as shown in the table below:
| Pattern | Green | Orange |
|---|---|---|
| OFF | Power OFF | |
| ON | idle | System error |
| BLINK | - | MicroSD™ card missing |
| BLINK_SHORT | Detecting, no anomaly | Anomaly detected |
| BLINK_LONG | Learning, status OK | Learning, status FAILED |
- Put Data brief of FP-AI-MONITOR1 Link: Artificial Intelligence (AI) Sensing and monitoring function pack for STM32Cube
- User Manual : User Manual for FP-AI-MONITOR1
- STEVAL-STWINKT1B
- STM32CubeMX : STM32Cube initialization code generator
- X-CUBE-AI : expansion pack for STM32CubeMX
- NanoEdge AI Studio: NanoEdge AI™ by Cartesiam the first Machine Learning Software, specifically developed to entirely run on microcontrollers.
- DB4345: Data brief for STEVAL-STWINKT1B.
- UM2777: How to use the STEVALSTWINKT1B SensorTile Wireless Industrial Node for condition monitoring and predictive maintenance applications.