Sensing and condition monitoring are two of the major components of the IoT and predictive maintenance systems, enabling context awareness, production performance improvement, a drastic decrease of the downtime due to routine maintenance, and so, maintenance cost reduction.
The FP-AI-MONITOR1 function pack is a Multi-sensor AI data monitoring framework on the wireless industrial node, function pack for STM32Cube. It helps to jump-start the implementation and development for sensor-monitoring-based applications designed with the X-CUBE-AI Expansion Package for STM32Cube or with the NanoEdge™ AI Studio. It covers the entire design of the Machine Learning cycle from the data set acquisition to the integration on a physical node.
FP-AI-MONITOR1 runs learning and inference sessions in real-time on SensorTile Wireless Industrial Node development kit (STEVAL-STWINKT1B), taking data from onboard sensors as input. FP-AI-MONITOR1 implements a wired interactive CLI to configure the node, and manage the learn and detect phases. For simple in the field operation, a standalone battery-operated mode allows basic controls through the user button, without using the console.
The STEVAL-STWINKT1B has an STM32L4R9ZI ultra-low-power microcontroller (Arm® Cortex®‑M4 at 120 MHz with 2 Mbytes of Flash memory and 640 Kbytes of SRAM). In addition, the STEVAL-STWINKT1B embeds industrial-grade sensors, including 6-axis IMU, 3-axis accelerometer and vibrometer and analog microphones to record any inertial, vibrational and acoustic data with high accuracy at high sampling frequencies.
The rest of the article discusses the following topics:
- The required hardware and software,
- Pre-requisites and setup,
- FP-AI-MONITOR1 console application,
- Running a human activity recognition (HAR) application for sensing on the device,
- Running an anomaly detection application using NanoEdge™ AI libraries on the device,
- Running an n-class Classification application using NanoEdge™ AI libraries on the device,
- Running an advance dual model on the device, which combines anomaly detection using NanoEdge AI libraries based on vibration data and CNN based n-class classification based on ultrasound data, and finally,
- Performing the vibration sensor data collection using a prebuilt binary of FP-SNS-DATALOG1,
- Button operated modes, and
- Some links to useful online resources, to help the user better understand and customize the project for her/his own needs.
This article is just to serve as the quick starting guide and for full FP-AI-MONITOR1 user instructions, readers are invited to please refer to FP-AI-MONITOR1 User Manual.
1. Hardware and software overview
1.1. SensorTile wireless industrial node Evaluation Kit STEVAL-STWINKT1B
The SensorTile Wireless Industrial Node (STEVAL-STWINKT1B) is a development kit and reference design that simplifies the prototyping and testing of advanced industrial IoT applications such as condition monitoring and predictive maintenance. It is powered with an ultra-low-power Arm® Cortex®-M4 MCU at 120 MHz with FPU, 2048-Kbyte Flash memory (STM32L4R9). STEVAL-STWINKT1B is equipped with a microSD™ card slot for standalone data logging applications. STEVAL-STWINKT1B also features a wide range of industrial IoT sensors including:
- an ultra-wide bandwidth (up to 6 kHz), low-noise, 3-axis digital vibration sensor (IIS3DWB)
- a 6-axis digital accelerometer and gyroscope iNEMO inertial measurement unit (IMU) with machine learning core (ISM330DHCX),
- an analog MEMS microphone with frequency response up to 80 kHz (IMP23ABSU),
and much more. Please refer to this link for all the sensors and features supported by STEVAL-STWINKT1B. Other attractive features include:
- a 480 mAh Li-Po battery to enable standalone working mode
- STLINK-V3MINI debugger with programming cable to flash the board
- a plastic box for ease of placing and planting the SensorTile on the machines for condition monitoring. For further details, visit this link
1.2. FP-AI-MONITOR1 software description
The top-level architecture of the FP-AI-MONITOR1 function pack is shown in the following figure.
2. Prerequisites and setup
2.1. Hardware prerequisites and setup
To use the FP-AI-MONITOR1 function pack on STEVAL-STWINKT1B, the following hardware items are required:
- STEVAL-STWINKT1B development kit board,
- Windows® powered laptop/PC (Windows® 7, 8, or 10),
- Two Micro-USB cables, one to connect the sensor-board to the PC, and another one for the STLINK-V3MINI, and
- an STLINK-V3MINI.
2.2. Software requirements
2.2.1. FP-AI-MONITOR1
- Download the FP-AI-MONITOR1 package from ST website, extract and copy the .zip file content into a folder on your PC. The package contains binaries and source code for the sensor board STEVAL-STWINKT1B.
2.2.2. IDE
- Install one of the following IDEs:
- STMicroelectronics STM32CubeIDE version 1.9.0,
- IAR Embedded Workbench for Arm (EWARM) toolchain version 9.20.1 or later, or
- RealView Microcontroller Development Kit (MDK-ARM) toolchain version 5.32.
2.2.3. STM32CubeProgrammer
- STM32CubeProgrammer (STM32CubeProg) is an all-in-one multi-OS software tool for programming STM32 products. It provides an easy-to-use and efficient environment for reading, writing, and verifying device memory through both the debug interface (JTAG and SWD) and the bootloader interface (UART, USB DFU, I2C, SPI, and CAN). STM32CubeProgrammer offers a wide range of features to program STM32 internal memories (such as Flash memory, RAM, and OTP) as well as external memories. FP-AI-MONOTOR1 is tested with STM32CubeProgrammer version 2.10.0.
- This software can be downloaded from STM32CubeProg.
2.2.4. TeraTerm
- TeraTerm is an open-source and freely available software terminal emulator, which is used to host the CLI of the FP-AI-MONITOR1 through a serial connection.
- Download and install the latest version available from TeraTerm.
2.2.5. STM32CubeMX
STM32CubeMX is a graphical tool that allows a very easy configuration of STM32 microcontrollers and microprocessors, as well as the generation of the corresponding initialization C code for the Arm® Cortex®-M core or a partial Linux® Device Tree for Arm® Cortex®-A core), through a step-by-step process. Its salient features include:
- Intuitive STM32 microcontroller and microprocessor selection.
- Generation of initialization C code project, compliant with IAR Systems®, Keil®, and STM32CubeIDE for Arm® Cortex®-M core
- Development of enhanced STM32Cube Expansion Packages thanks to STM32PackCreator, and
- Integration of STM32Cube Expansion Packages into the project.
FP-AI-MONITOR1 requires STM32CubeMX V 6.5.0. To download the STM32CubeMX and obtain details of all the features please visit st.com.
2.2.6. X-CUBE-AI
X-CUBE-AI is an STM32Cube Expansion Package part of the STM32Cube.AI ecosystem and extending STM32CubeMX capabilities with automatic conversion of pre-trained Artificial Intelligence models and integration of generated optimized library into the user's project. The easiest way to use it is to download it inside the STM32CubeMX tool (version 7.1.0 or newer) as described in the user manual "Getting started with X-CUBE-AI Expansion Package for Artificial Intelligence (AI)" (UM2526). The X-CUBE-AI Expansion Package also offers several means to validate the AI models (Neural Network and Scikit-Learn models in .h5, .onnx, and .tflite format) both on desktop PC and STM32, as well as to measure performance on STM32 devices (computational and memory footprints) without user handmade ad-hoc C code.
2.2.7. Python 3.7.3
Python is an interpreted high-level general-purpose programming language. Python's design philosophy emphasizes code readability with its notable use of significant indentation. Its language constructs as well as its object-oriented approach aim to help programmers write clear, logical code for small and large-scale projects. To build and export AI models the reader requires to set up a Python environment with a list of packages. The list of the required packages along with their versions is available as a text file in /FP-AI-MONITOR1_V2.0.0/Utilities/AI_resources/requirements.txt directory. The following command is used in the command terminal of the anaconda prompt or Ubuntu to install all the packages specified in the configuration file requirements.txt:
pip install -r requirements.txt
2.2.8. NanoEdge™ AI Studio
NanoEdge™ AI Studio is a new Machine Learning (ML) technology to bring true innovation easily to the end-users. In just a few steps, developers can create optimal ML libraries for:
- Anomaly Detection,
- 1-class classification,
- n-class classification, and
- extrapolation, based on a minimal amount of data.
In addition, users can also program and configure the STEVAL-STWINKT1B for the data logging from Studio through a high-speed data logger.
This function pack supports only the Anomaly Detection and n-Class Classification libraries generated by NanoEdge™ AI Studio for STEVAL-STWINKT1B. FP-AI-MONITOR1 is tested using NanoEdge AI Studio V3.0.2.
.
2.3. Installing the function pack to STEVAL-STWINKT1B
2.3.1. Getting the function pack
The first step is to get the function pack. Download the FP-AI-MONITOR1 from ST website. Once the pack is downloaded, unpack/unzip it and copy the content to a folder on the PC. The steps of the process along with the content of the folder are shown in the following image.
2.3.2. Flashing the application on the sensor board STEVAL-STWINKT1B
Once the package has been downloaded and unpacked, the next step is to program the sensor node with the binary of the function pack. For the convenience of the users, the function pack is equipped with a pre-built binary file of the project. This binary file can be found at path /FP-AI-MONITOR1_V2.0.0/Projects/STM32L4R9ZI-STWIN/Applications/FP-AI-MONITOR1/Binary/FP-AI-MONITOR1.bin. The sensor board can be very easily programmed with the provided binary by simply performing a drag-and-drop action as shown in the figure below.
3. FP-AI-MONITOR1 console application
FP-AI-MONITOR1 provides an interactive command-line interface (CLI) application. This CLI application equips a user with the ability to configure and control the sensor node, perform learning (for NanoEdge™ AI libraries), and (anomaly) detection operations on the edge. The following sections provide a small guide on how to install this CLI application on a sensor board and control it through the CLI from TeraTerm.
3.1. Setting up the console
Once the sensor board is programmed with the binary of the project (as shown in section 2), the next step is to set up the serial connection of the board with the PC through TeraTerm. To do so, start TeraTerm and create a new connection by either selecting it from the toolbar or by selecting the proper port to establish the serial communication with the board. In the figure below, this is COM10 - USB Serial Device (COM 10), but it could vary for different users.
Once the connection is established, the message below is displayed. If this is not the case, reset the board.
Typing help shows the list of all the available commands along with their usage instructions.
3.2. Configuring the sensors
Through the CLI interface, a user can configure the supported sensors for sensing and condition monitoring applications. The list of all the supported sensors can be displayed on the CLI console by entering the command sensor_info. This command prints the list of the supported sensors along with their ids as shown in the image below. The user can configure these sensors using these ids. The configurable options for these sensors include:
- enable: to activate or deactivate the sensor,
- ODR: to set the output data rate of the sensor from the list of available options, and
- FS: to set the full-scale range from the list of available options.
The current value of any of the parameters for a given sensor can be printed using the command,
$ sensor_get <sensor_id> <param>
or all the information about the sensor can be printed using the command:
$ sensor_get <sensor_id> all
Similarly, the values for any of the available configurable parameters can be set through the command:
$ sensor_set <sensor_id> <param> <val>
The figure below shows the complete example of getting and setting these values along with old and changed values.
4. Classification applications for sensing
The CLI application comes with a prebuilt Human Activity Recognition model. This functionality can be started by typing the command:
$ start ai
Note that the provided HAR model is built with a dataset created using the IHM330DHCX_ACC sensor with ODR = 26, and FS = 4. To achieve good performance, the user is required to set these parameters to the sensor configurations using the instructions provided in section FP-AI-MONITOR1_getting_started#Configuring_the_sensors Configuring the sensors.
Running the $ start ai command starts the inference on the accelerometer data and predicts the performed activity along with the confidence. The supported activities are:
- Stationary,
- Walking,
- Jogging, and
- Biking.
The following screenshot shows the normal working session of the AI command in the CLI application.
4.1. NanoEdge AI Anomaly Detection
This section provides a complete anomaly detection method using the NanoEdge™ AI library. FP-AI-MONITOR1 includes a pre-integrated stub which is easily replaced by an AI condition monitoring library generated and provided by NanoEdge™ AI Studio. This stub simulates the NanoEdge™ AI-related functionalities, such as running learning and detection phases on the edge.
The learning phase is started by issuing a command $ start neai_learn from the CLI console or by long-pressing the [USR] button. The learning process is reported either by slowly blinking the green LED light on STEVAL-STWINKT1B or in the CLI as shown below:
NanoEdge AI: learn
$ CTRL: ! This is a stubbed version, please install NanoEdge AI library !
{"signal": 1, "status": need more signals}
{"signal": 2, "status": need more signals}
:
:
{"signal": 10, "status": need more signals}
{"signal": 11, "status": success}
{"signal": 12, "status": success}
{"signal": 13, "status": success}
:
:
:
End of execution phase
The CLI shows that the learning is to be performed for at least 10 signals and reports that more signals are needed until it has learned 10 signals. After 10 signals are learned, the status message of learning changes to success for every learning of the new signal. The learning can be stopped by pressing the ESC key on the keyboard or simply by pressing the [USR] button.
Similarly, the user starts the condition monitoring process by issuing the command $ start neai_detect command or by double-pressing the [USR] button. This starts the inference phase and checks the similarity of the presented signal with the learned normal signals. If the similarity is less than the set threshold default: 90%, a message is printed in the CLI showing the occurrence of an anomaly along with the similarity of the anomaly signal.
The process is stopped by pressing the ESC key on the keyboard or pressing the [USR] button.
This behavior is shown in the snippet below:
$ start neai_detect
NanoEdge AI: starting detect phase.
NanoEdge AI: detect
$ CTRL: ! This is a stubbed version, please install NanoEdge AI library !
{"signal": 1, "similarity": 0, "status": anomaly}
{"signal": 2, "similarity": 1, "status": anomaly}
{"signal": 3, "similarity": 2, "status": anomaly}
:
:
{"signal": 90, "similarity": 89, "status": anomaly}
{"signal": 91, "similarity": 90, "status": anomaly}
{"signal": 102, "similarity": 0, "status": anomaly}
{"signal": 103, "similarity": 1, "status": anomaly}
{"signal": 104, "similarity": 2, "status": anomaly}
End of execution phase
Other than CLI, the status is also presented using the LED lights on the STEVAL-STWINKT1B. Fast blinking green LED light shows the detection is in progress. Whenever an anomaly is detected, the orange LED light is blinked twice to report an anomaly.
NOTE : This behavior is simulated using a STUB library where the similarity starts from 0 when the detection phase is started and increments with the signal count. Once the similarity is reached to 100 it resets to 0. One can see that the anomalies are not reported when similarity is between 90 and 100.
4.1.1. Additional parameters in condition monitoring
For user convenience, the CLI application also provides handy options to easily fine-tune the inference and learning processes. The list of all the configurable variables is available by issuing the following command:
$ neai_get all NanoEdgeAI: signals = 0 NanoEdgeAI: sensitivity = 1.000000 NanoEdgeAI: threshold = 95 NanoEdgeAI: timer = 0
Each of the these parameters is configurable using the neai_set <param> <val> command.
This section provides information on how to use these parameters to control the learning and detection phase. By setting the "signals" and "timer" parameters, the user can control how many signals or for how long the learning and detection is performed (if both parameters are set the learning or detection phase stops whenever the first condition is met). For example, to learn 10 signals, the user issues this command, before starting the learning phase as shown below.
$ neai_set signals 10
NanoEdge AI: signals set to 10
$ start neai_learn
NanoEdgeAI: starting
$ {"signal": 1, "status": success}
{"signal": 2, "status": success}
{"signal": 3, "status": success}
:
:
{"signal": 10, "status": success}
NanoEdge AI: stopped
If both of these parameters are set to "0" (default value), the learning and detection phases will run indefinitely.
The threshold parameter is used to report any anomalies. For any signal which has similarities below the threshold value, an anomaly is reported. The default threshold value used in the CLI application is 90. Users can change this value by using neai_set threshold <val> command.
Finally, the sensitivity parameter is used as an emphasis parameter. The default value is set to 1. Increasing this sensitivity will mean that the signal matching is to be performed more strictly, reducing it will relax the similarity calculation process, i.e. resulting in higher matching values.
For further details on how NanoEdge™ AI libraries work users are invited to read the detailed documentation of NanoEdge™ AI Studio.
5. Data collection
The data collection functionality is out of the scope of this function pack, however, to simplify this for the users this section provides the step-by-step guide to log the data on STEVAL-STWINKT1B using the high-speed data logger.
Step 1: Program the STEVAL-STWINKT1 with the HSDatalog FW
In the scope of this function pack and article, the STEVAL-STWINKT1 can be programmed with HSDatalog firmware in two ways.
- Using the binary provided in the function pack
- To simplify the task of the users and to allow them the possibility to perform a data log, the precompiled
HSDatalog.binfile for FP-SNS-DATALOG1 is provided in theUtilitiesdirectory, which is located under path/FP-AI-MONITOR1_V1.1.0/Utilities/Datalog/. The sensor tile can be programmed by simply following the drag-and-drop action shown in section 2.3.
- To simplify the task of the users and to allow them the possibility to perform a data log, the precompiled
- Using the NanoEdge AI Studio
- The new version of NanoEdgeTM AI Studio has a new feature to program the high-speed datalogger in the STEVAL-STWINKT1B. To do this follow the steps in the figure below.
- click on the Datalogger button in the home menu,
- connect the STEVAL-STWINKT1B to the computer as shown in the figure below (connect the USB to STWIN first and STLINK after),
- wait for the status to change to connect,
- click the LOAD BINARY button,
- weight for the Success message,
- the STWIN is now programmed with the high-speed datalogger firmware.
- The new version of NanoEdgeTM AI Studio has a new feature to program the high-speed datalogger in the STEVAL-STWINKT1B. To do this follow the steps in the figure below.
Step 2: Place a DeviceConfig.json file on the SD card
The next step is to copy a DeviceConfig.json on the SD card. This file contains the sensors configurations to be used for the data logging. These sample files can be either:
- interactively generated through studio by clicking on the Configure datalogger button in the datalogger menu as shown in the figure below, or,
- the users can simply use one of the provided some sample
.jsonfiles in the package inFP-AI-MONITOR1_V1.1.0/Utilities/Datalog/Sample_STWIN_Config_Files/directory.
Step 3: Insert the SD card into the STWIN board Insert an SD card in the STEVAL-STWINKT1B sensor tile.
Step 4: Reset the board
Reset the board. Orange LED blinks once per second. The custom sensor configurations provided in DeviceConfig.json are loaded from the file.
Step 5: Start the data log
Press the [USR] button to start data acquisition on the SD card. The orange LED turns off and the green LED starts blinking to signal sensor data is being written into the SD card.
Step 6: Stop the data logging
Press the [USR] button again to stop data acquisition. Do not unplug the SD card, turn the board off or perform a [RESET] before stopping the acquisition otherwise the data on the SD card will be corrupted.
Step 7: Retrieve data from SD card
Remove the SD card and insert it into an appropriate SD card slot on the PC. The log files are stored in STWIN_### folders for every acquisition, where ### is a sequential number determined by the application to ensure log file names are unique. Each folder contains a file for each active sub-sensor called SensorName_subSensorName.dat containing raw sensor data coupled with timestamps, a DeviceConfig.json with specific information about the device configuration (confirm if the sensor configurations in the DeviceConfig.json are the ones you desired), necessary for correct data interpretation, and an AcquisitionInfo.json with information about the acquisition.
6. Button operated modes
To facilitate the usage, of the FP-AI-MONITOR1 when there is no CLI connection, the FP is equipped with a button-operated mode. The purpose of the button-operated mode is to enable the FP-AI-MONITOR1 to work even in the absence of the CLI console. All the (default) values required for running different functionalities are provided in the firmware. In button-operated mode, the user can start/stop different execution phases by using only the user button. The following section describe the allocation of the buttons and LEDs to display the currently running contexts.
6.1. LED Allocation
In the function pack six execution phases exist:
- idle: the system waits for user input.
- ai detect: all data coming from the sensors are passed to the X-CUBE-AI library to detect anomalies.
- neai learn: all data coming from the sensors are passed to the NanoEdge™ AI library to train the model.
- neai detect: all data coming from the sensors are passed to the NanoEdge™ AI library to detect anomalies.
- neai class: all data coming from the sensors are passed to the NanoEdge™ AI library to perform classification.
- dual mode: all data coming from the sensors are passed to the NanoEdge™ AI library to detect anomalies and to X-CUBE-AI library to classify them.
At any given time, the user needs to be aware of the current active execution phase. We also need to report on the outcome of the detection when the detect execution phase is active, telling the user if an anomaly has been detected, or what activity is being performed by the user with the HAR application when the "ai" context is running.
The onboard LEDs indicate the status of the current execution phase by showing which context is running and also by showing the output of the context (anomaly or one of the four activities in the HAR case).
The green LED is used to show the user which execution context is being run.
| Pattern | Task |
|---|---|
| OFF | - |
| ON | IDLE |
| BLINK_SHORT | X-CUBE-AI Running |
| BLINK_NORMAL | NanoEdge™ AI learn |
| BLINK_LONG | NanoEdge™ AI detection or classification or Dual Mode |
The Orange LED is used to indicate the output of the running context as shown in the table below:
| Pattern | Reporting |
|---|---|
| OFF | Stationary (HAR) if in X-CUBE-AI mode, Normal Behavior when in NEAI mode |
| ON | Biking (HAR) |
| BLINK_SHORT | Jogging (HAR) |
| BLINK_LONG | Walking (HAR) or anomaly detected (NanoEdge™ AI detection or Dual Mode) |
Looking at these LED patterns the user is aware of the state of the sensor node even when CLI is not connected.
6.2. Button Allocation
In the button operated mode mode, the user can trigger only four of the available execution phases. The button supported modes are:
- idle: the system is waiting for a command.
- ai: run the X-CUBE-AI library and print the results of the live inference on the CLI (if CLI is available).
- neai_learn: all data coming from the sensor is passed to the NanoEdge™ AI library to train the model.
- neai_detect: all data coming from the sensor is passed to the NanoEdge™ AI library to detect anomalies.
To trigger these phases, the FP-AI-MONITOR1 is equipped with the support of the user button. In the STEVAL-STWINKT1B sensor node, there are two software usable buttons:
- The user button: This button is fully programmable and is under the control of the application developer.
- The reset button: This button is used to reset the sensor node and is connected to the hardware reset pin, thus is used to control the software reset. It resets the knowledge of the NanoEdge™ AI libraries and reset all the context variables and sensor configurations to the default values.
To control the executions phases, we need to define and detect at least three different button press modes of the user button.
The following are the types of the press available for the user button and their assignments to perform different operations:
| Button Press | Description | Action |
|---|---|---|
| SHORT_PRESS | The button is pressed for less than (200 ms) and released | Start AI inferences for X-CUBE-AI model. |
| LONG_PRESS | The button is pressed for more than (200 ms) and released | Starts the learning for NanoEdge™ AI Library. |
| DOUBLE_PRESS | A succession of two SHORT_PRESS in less than (500 ms) | Starts the inference for NanoEdge™ AI Library. |
| ANY_PRESS | The button is pressed and released (overlaps with the three other modes) | Stops the current running execution phase. |
- DB4494: Data brief for FP-AI-MONITOR1
- User Manual: User Manual for FP-AI-MONITOR1
- STEVAL-STWINKT1B: SensorTile Wireless Industrial Node STEVAL-STWINKT1B development kit and reference design for industrial IoT applications
- STM32CubeMX: STM32Cube initialization code generator
- X-CUBE-AI : expansion pack for STM32CubeMX
- NanoEdge™ AI Studio: NanoEdge AI™ Studio the first Machine Learning Software, specifically developed to entirely run on microcontrollers.
- DB4345: Data brief for STEVAL-STWINKT1B.
- UM2777: How to use the STEVAL-STWINKT1B SensorTile Wireless Industrial Node for condition monitoring and predictive maintenance applications.