How to perform anomaly detection using FP-AI-MONITOR1

This article aims to provide a step-by-step guide to set up a vibration-based condition monitoring solution. This demonstration uses an easily available USB fan and implements a condition monitoring proof of concept using FP-AI-MONITOR1 and an STEVAL-STWINKT1B board. The anomaly detection AI libraries to be used in this tutorial are generated using NanoEdge^TM AI Studio and the software used to program the sensor board is provided as a function pack that can be downloaded from the ST website.

The following steps are explained in this article:

• How to use high-speed datalogger binary to log the vibration data on microSD^TM card on STEVAL-STWINKT1B,
• How to generate the AI library for condition monitoring using NanoEdge^TM AI Studio,
• How to integrate the generated libraries to FP-AI-MONITOR1, and program the sensor-node for condition monitoring, and
• How to perform condition monitoring on the USB fan setup using live vibration data.

1 Requirements

To follow and reproduce the steps provided in this article, the following parts are required:

1.1 Hardware

To perform all the steps in this document, users need the following hardware components.

• An STEVAL-STWINKT1B development kit board,
• A Windows^® powered laptop/PC (Windows^® 7, 8, or 10),
• An STLINK-V3MINI,
• Two micro USB cables, one to connect the STEVAL-STWINKT1B to the PC, and another one for the STLINK-V3MINI,
• A microSD^TM card formatted as FAT32-FS for the data logging operation,
• A USB microSD^TM card reader to read the data from the microSD^TM card to the computer, and
• A USB powered fan that can be ordered from this link.

1.2 Software

• FP-AI-MONITOR1

  Download the FP-AI-MONITOR1, package from ST website, extract and copy the .zip file contents into a folder on the PC. The package contains binaries and source code, for the sensor-board STEVAL-STWINKT1B.

• STMicroelectronics - STM32CubeIDE version 1.7.0

  STM32CubeIDE is an all-in-one multi-OS development tool, which is part of the STM32Cube software ecosystem.

• Tera Term

  Tera Term is an open-source and freely available software terminal emulator, which is used to host the CLI of the FP-AI-MONITOR1 through a serial connection. Users can download and install the latest version available from Tera Term.

• NanoEdge^TM AI Studio 2.1.*

  The anomaly detection libraries used in this demonstration are generated by NanoEdge^TM AI Studio. For this, the user needs to generate a project in the Studio and get the libraries from it. Then, these libraries can be embedded in the FP-AI-MONITOR1. More information on the studio can be found on st.com.

• Python^TM 3.7.X

  To process the data logs and to prepare the data in the form which is accepted by the NanoEdge^TM AI Studio, Python^TM utility scripts are provided. These scripts are available under path /FP-AI-MONITOR1_V1.0.0/Utilities/AI_Resources/NanoEdgeAi/. This also requires installing some dependencies for Python^TM. The list of the required packages along with their versions is available as a text file in the package at location /FP-AI-MONITOR1_V1.0.0/Utilities/AI_resources/requirements.txt. The following command is used in the command terminal of the anaconda prompt or Ubuntu to install all the packages specified in the configuration file requirements.txt:

pip install -r requirements.txt

Information

In this document all the steps presented are carried out with STM32CubeIDE but the same can also be done with the IAR EWARM and MDK Keil.

2 Condition monitoring

The following section shows how to set up a condition monitoring project. The following figure shows all the steps with brief details.

2.1 Data acquisition

FP-AI-MONITOR1 contains as a utility, a precompiled HSDatalog.bin binary file for FP-SNS-DATALOG1. The binary file is located under /FP-AI-MONITOR1_V1.0.0/Utilities/Datalog/ directory. This file can be used to program the STEVAL-STWINKT1B to enable the data-logging functionality. This section briefly describes how a user can use this binary to log the data using onboard sensors of STEVAL-STWINKT1B. Connect the STEVAL-STWINKT1B for programming. This needs to connect the STLINKV3-Mini as well as the USB of the STEVAL-STWINKT1B as shown in the figure below to the computer (both connections are made using micro USB cables).

File:how to AD stwin connections.png

Users can program the sensor-board using this file by simply performing a drag-drop action as shown in the figure below.

Information

For data logging using the high speed datalogger user needs a FAT32-FS formatted microSDTM card.

Once the sensor-board is programmed log the data using the following instructions.

• Place a DeviceConfig.json file with the sensors configurations to be used for the data logging in the root folder of the microSD^TM card. DeviceConfig.json is a json file which has to be in a predefined format and contains the configurations of all the sensors to be used during the data logging. FP-AI-MONITOR1_V1.0.0/Utilities/Datalog/Sample_STWIN_Config_Files/ directory contains some sample configuration files. Note that these files are not named as DeviceConfig and the file to be used for the data acquisition has to be precisely named as DeviceConfig.json. Otherwise, the data is logged using the default configurations (all sensors active at the highest data-rate).

Information

Details on the syntax and all the elements present in DeviceConfig.json file can be found in the section 2.5.1 of user manual of FP-SNS-DATALOG1.

• Insert the SD card into the STEVAL-STWINKT1B board.
• Reset the board. Orange LED blinks once per second. The custom sensor configurations provided in DeviceConfig.json are loaded from the file.
• Press the [USR] button to start data acquisition on the SD card. The orange LED turns off and the green LED starts blinking to signal sensor data is being written into the SD card.
• Press the [USR] button again to stop data acquisition. Do not unplug the SD card or turn the board off before stopping the acquisition otherwise the data on the SD card is corrupted.
• Remove the SD card and insert it into an appropriate SD card slot on the PC. The log files are stored in STWIN_### folders for every acquisition, where ### is a sequential number determined by the application to ensure log file names are unique. Each folder contains a file for each active sub-sensor called SensorName_subSensorName.dat containing raw sensor data coupled with timestamps, a DeviceConfig.json with specific information about the device configuration, necessary for correct data interpretation, and an AcquisitionInfo.json with information about the acquisition.

Information

For details on how to use all the features of the provided HSDatalog.bin binary the users are invited to refer to the user manual of FP-SNS-DATALOG1.

2.1.1 Logging for normal and abnormal conditions

The anomaly detection library generation in NanoEdge^TM AI Studio requires some example data for the normal and abnormal conditions to select an optimal algorithm and library for the anomaly detection for any given setup. Following the steps in this section the users can log the data for the normal and abnormal conditions. HSDatalog.bin enables us to log data using multiple sensors on the board at the same time. For this demonstration, only data from ISM330DHCX sensor is used, and all the other sensors are disactivated. To do so the user can take one of the example .json files and copy it at the root of a microSD^TM card. Rename the file as DeviceConfig.json and edit it deactivate all the sensors and sub sensors but specifying the values for isActive flag as false. After this enable the ISM330DHCX sensors by setting the value of isActive flag to true for the accelerometer sub sensor. Set the odr (output data-rate) for this sensor to 1667 and Fs (full-scale) to 4. Insert the microSD^TM card in sensor-board, and restart the board by pressing RESET button. After this step, everything is set and ready for data logging using accelerometer data.

Put the STEVAL-STWINKT1B on the fan as shown in the figure above, while it is running in normal condition (with no problems). The fan being used in this demonstration has a handy controller which lets users turn on and turn off the fan, as well as run it at one of the three-speed settings. The following figure is showing the controller image.

As the readers can see the settings are labeled on the controller are:

• O: Fan is in Off state,
• L: Fan is running at Low speed (1000 RPM),
• M: Fan is running at Medium speed (1250 RPM), and
• H: Fan is running at High speed (1500 RPM).

Start the fan at the low speed when the button is at L settings, when the fan is stable, start the data logging by pressing the user button on the STEVAL-STWINKT1B the green LED light starts flashing. Wait for 40 seconds, then press the user button again to stop the logging of the data, the green LED light stops blinking. Repeat the process for medium and high speed. This generates three data logs for fan speeds L (1000RPM), M (1250RPM), and H (1500RPM), named as ST0001, ST0002, and ST0003 respectively. Now we can simulate the clogged condition by putting a piece of paper on the inlet side (where the airflow is inside) of the fan. Repeat the steps for the data logging for logging data in abnormal conditions at the three speeds. This generates three more data folders named as ST0004, ST0005, and ST0006 for speeds L (1000RPM), M (1250RPM) and H (1500RPM) respectively.

Information

Note that we removed the grill of the fan from the outlet side (from the side which throws the air out) to be able to test unbalance and friction faults on the fan setup. This is done before the data logging.

2.1.2 Data preparation for library generation with NanoEdge^TM AI Studio

The data logged through the datalogger is in the binary format and is neither user-readable nor compliant with the NanoEdge^TM AI Studio format as it is. To convert this data to a useful form, FP-AI-MONITOR1 provides set of Python^TM utility scripts on the path /FP-AI-MONITOR1_V1.0.0/Utilities/AI-resources/NanoEdgeAi/. There are two possibilities provided for the data parsing and preparation. A Jupyter™ Notebook and a Python^TM function as a Python^TM script file.

The Jupyter™ Notebook named as NanoEdgeAI_Utilities.ipynb provides a complete example of data preparation for a three-speed fan library generation running in normal and clogged condition. The script out of the package uses the example data provided in the package. This section guides users to exercise these scripts to generate an anomaly detection library using the six data logs generated following the instruction in the section above.

The NanoEdge^TM AI Studio expects as input one normal and one abnormal file as input for anomaly detection library generation. Hence, we need to combine the data for the three speeds for normal and clogged conditions. To use the provided scripts as it is without changing too much, we advise readers to rename and arrange their six data acquisitions as shown in the folder below.

Once the datalogs are renamed, arranged and grouped as shown in the figure above, copy and paste the folder named as Fan12CMNew at the path FP-AI-MONITOR1_V1.0.0/Utilities/AI_Resources/HSD_Logged_Data/Fan12CMNew/ and open the Jupyter™ Notebook named NanoEdgeAI_Utilities.ipynb. Edit the code as shown in the screenshot below:

After making these changes, save the file, run all the cells either manually one-by-one or by going to the Cell menu in the toolbar, and select Run All the sections option, as shown in the figure below.

After the last section run, there are two files generated in the folder Fan12CMNew named as clogged_WL1024_segments.csv and normal_WL1024_segments.csv. Note that 1024 is an indication that the window length is 1024 samples.

The Jupyter™ Notebook is user-friendly and users can control the process of normal and abnormal file generation by changing a few parameters according to their will. As the FP-AI-MONITOR1 supports anomaly detection through both ISM330DHCX as well as IIS3DWB, the provided script can generate normal and abnormal files for inputting to NanoEdge^TM AI Studio for any of the sensors by just changing the name of the sensor. The default variables for the frame length and stride are set for both sensors to result in non-overlapping windows of length 1024 and 4096 for ISM330DHCX and IIS3DWB sensor. Similarly, a user can change the default values to generate overlapping frames or can change the window lengths to have longer or shorter frames. The user can also choose to add or remove some more acquisitions for the file generation process.

Note: In addition to this Jupyter™ Notebook there is a simple option provided for generating the NanoEdge AI complaint segment files in the file HSD_2_NEAISegments.py. This script can generate the frames for a single acquisition file by issuing the following command for data acquisition with ISM330DHCX_ACC sensor, but can not combine the data from multiple files.

python HSD_2_NEAISegments.py ../Datasets/Fan12CM/ISM330DHCX/normal/1000RPM/

This command generates a file named as ISM330DHCX_Cartesiam_segments_0.csv using the default parameter set. The file is generated with segments of length 1024 and stride 1024 plus the first 512 samples are skipped from the file.

2.2 Generating a condition monitoring library

The process of generating the libraries with NanoEdge^TM AI Studio consists of six steps. These steps can be seen in the figure below.

1. Hardware description
   • Choosing the target platform or a microcontroller type: STEVAL-STWINKT1B Cortex-M4
   • Maximum amount of RAM to be allocated for the library: Usually a few Kbytes are enough (but it depends on the data frame length used in the process of data preparation, 32 Kbytes is a good starting point) so we start from 32 Kbytes.
   • Sensor type: select 3-axis accelerometer for this project
2. Providing the sample contextual data for normal segments generated in the previous section. This data is used to find the best parameters to have the best performances for the given setup. meaning the normal_WL1024_segments.csv file.
3. Providing the sample contextual data for abnormal segments generated in the previous section for fine-tuning the model to have the best performance on the given setup, meaning the clogged_WL1024_segments.csv file.
4. Benchmarking of available models and choosing the one that complies with the requirements.
5. Validating the model for learning and testing through the provided emulator which emulates the behavior of the library on the edge.
6. The final step is to compile and download the libraries. In this process, the flag -mfloat-abi has to be checked for using libraries with hardware FPU. All the other flags can be left to the default state.

Note: For using the library with μKeil also check -fshort-wchar in addition to -mfloat-abi.

The detailed documentation on the NanoEdge^TM AI Studio regarding the AI library generation and working with these libraries can be found here.

2.3 Installing the NanoEdge^TM Machine Learning library

Once the libraries are generated and downloaded from NanoEdge^TM AI Studio, the next step is to link these libraries to FP-AI-MONITOR1 and run them on the STEVAL-STWINKT1B. The FP-AI-MONITOR1, comes with the library stubs in the place of the actual libraries generated by NanoEdge^TM AI Studio. This is done to simplify the linking process of the generated libraries. In order to link the generated libraries, the user needs to copy the generated libraries and replace the existing stub/dummy libraries libneai.a and header files NanoEdgeAI.h present in the folders lib and Inc, respectively as shown in the figure below. Both of these folders can be found at the path /FP_AI_MONITOR1_V1.0.0/Middlewares/ST/NanoEdge_AI_Library/.

Once these files are copied, the project must be reconstructed and programmed on the sensor board to link the libraries correctly. For this, the user must open the project in STM32CubeIDE located in the /FP-AI-MONITOR1_V1.0.0/Projects/STWINL4R9ZI-STWIN/Applications/FP-AI-MONITOR1/STM32CubeIDE/ folder and double click .project file as shown in the figure below.

Then build and install the project in STEVAL-STWINKT1B sensor-board by pressing the play button as shown in the figure below.

A message saying Download verified successfully indicates the new firmware is programmed in the sensor-board.

Information

NOTE: To be sure that new software is installed in STEVAL-STWINKT1B, the user can issue the command $ info. This shows the compile-time and the user can confirm if the sensor-board is programmed with new or old binary. If executing the info command shows an older time and date, try to source clean option from the CubeIDE and then rebuild and flash the project by pressing the play button.

2.4 Testing the NanoEdge^TM AI Machine Learning library

Once the STEVAL-STWINKT1B is programmed with the FW containing a valid library, the condition monitoring libraries are ready to be tested on the sensor board. The libraries can now to used to perform the condition monitoring using the STEVAL-STWINKT1B programmed with FP-AI-MONITOR1.

2.4.1 Connecting the command line interface (CLI) application

FP-AI-MONITOR1 comes with an interactive CLI application. The CLI uses a Virtual COM port over USB to interact with the sensor-board. This section shows how to connect the sensor-board using this and enable the CLI.

On a windows powered machine start the Tera Term, select the proper connection (featuring the STMicroelectronics name) as shown in the figure below,

Set the parameters:

• Terminal
  • [New line]:
    • [Receive]: CR
    • [Transmit]: CR

• Serial

  The interactive console can be used with the default values.

Restart the board by pressing the RESET button. The following welcome screen is displayed on the terminal.

From this point, the users can start entering the commands directly or type help to get the list of available commands along with their usage guidelines.

2.4.2 Learning normal conditions

To perform the on-device learning the user can start the learning phase by simply entering the command start neai_learn but before doing this the user needs to configure the sensor to use. To achieve the best performances the user should use the same sensor configurations as used for the data for data acquisition, meaning enable ISM330DHCX sensor with ODR = 1666 and FS = 4, and make sure the other sensors are disabled. This can be achieved by issuing the command sequence shown in the following figure.

File:how to AD configuring sensor.png

Once the sensor is configured the user needs to learn the normal running conditions for the fan at all three speeds. The learning can be started either by issuing the start neai_learn command or by long-pressing the user button (press for at least 200 msec). Once the normal behavior under consideration has been learned, the learning can be stopped by simply pressing the ESC button on the keyboard or by pressing the USR button on the sensor-board. Every library generated from NanoEdge^TM AI Studio comes with some guidelines on how many minimum training samples are to be used to get optimal performance. It is recommended that the user exercises at least three to five times more samples than reported. For example, if the minimum iteration value is 40, the user has to perform learning at each of the three speeds using 120 to 200 samples.

Information

Bonus: FP-AI-MONITOR1 has a command call neai_set signals, which enables users to run the learn and detect phases for provided number of iterations. So, if the user issues command neai_set signals 120, before issue start neai_learn command, the learning automatically stops after 120 iterations.

2.4.3 Performing condition monitoring

Once the normal state has been learned the condition monitoring or detect phase can be started by either issuing the start neai_detect command or double pressing the user button. During this process, the similarity of the vibration pattern is compared with the learned patterns, and whenever the similarity is below the set threshold (default value for threshold is 90%), a message is printed in the console showing the occurrence of the anomaly. This threshold can be changed and set using neai_set threshold <threshold_value> command, where threshold_value can be an integer value between [0 and 100].

Information

Note: The NanoEdgeTM AI library comes with an incremental learning capability, so the learning and detection phases can be toggled depending on whether the needs to be without forgetting the previous learning or not.

As the modeling of normal conditions is based on the data used for learning and STEVAL-STWINKT1B is not physically fixed with the fan the slight movements or changes of the orientation might result in bad results for the detecting part. Thanks to the on-device incremental learning capability of the NanoEdge^TM anomaly detection libraries, this problem can be very easily fixed by relearning the new vibration patterns without losing the information learned in the first learning phase. Performing a few relearning with slight changes of the position and orientation of STEVAL-STWINKT1B makes the solution robust and stable.

Information

To avoid the relearning phases the user can fix the STEVAL-STWINKT1B on the fan set up by using a metallic tape and magnets in STEVAL-STWINKT1B. This keeps the relative conditions stable hence the performances of the detection phase always stay almost the same.

2.5 Additional parameters in condition monitoring

For user convenience, the CLI application also provides handy options to easily fine-tune the inference and learning processes. The list of all the configurable variables is available by issuing the following command:

$ neai_get all
NanoEdgeAI: signals = 0
NanoEdgeAI: sensitivity = 1.000000
NanoEdgeAI: threshold = 95
NanoEdgeAI: timer = 0

Each of the these parameters is configurable using the neai_set <param> <val> command.

This section provides information on how to use these parameters to control the learning and detection phase. By setting the "signals" and "timer" parameters, the user can control how many signals or for how long the learning and detection is performed (if both parameters are set the learning or detection phase stops whenever the first condition is met). For example, to learn 10 signals, the user issues this command, before starting the learning phase as shown below.

$ neai_set signals 10
NanoEdgeAI: signals set to 10
$ start neai_learn
NanoEdgeAI: starting
$ {"signal": 1, "status": success}
{"signal": 2, "status": success}
{"signal": 3, "status": success}
:
:
{"signal": 10, "status": success}
NanoEdgeAI: stopped

If both of these parameters are set to "0" (default value), the learning and detection phases run indefinitely.

The threshold parameter is used to report any anomalies. For any signal which has similarities below the threshold value, an anomaly is reported. The default threshold value used in the CLI application is 90. Users can change this value by using neai_set threshold <val> command.

Finally, the sensitivity parameter is used as an emphasis parameter. The default value is set to 1. Increasing this sensitivity means that the signal matching is to be performed more strictly, reducing it relaxes the similarity calculation process, meaning resulting in higher matching values.

Information

Note: For the best performance, the user must expose all the normal conditions to the sensor-board during the learning and library generation process, meaning in the case of motor monitoring, the required speeds and ramps that need to be monitored must be exposed.

For further details on how NanoEdge^TM AI libraries and FP-AI-MONITOR1 work users are invited to read the detailed documentation of NanoEdge^TM AI Studio and user manual of FP-AI-MONITOR1.

3 Documents and related resources

Following are some resources which can be helpful for the readers.

• FP-AI-MONITOR1: Multi-sensor AI data monitoring framework on wireless industrial node, function pack for STM32Cube
• User Manual: User Manual for FP-AI-MONITOR1
• QSG:Quick Start Guide for FP-AI-MONITOR1
• STEVAL-STWINKT1B
• STM32CubeMX: STM32Cube initialization code generator
• X-CUBE-AI : expansion pack for STM32CubeMX
• NanoEdge^TM AI Studio: NanoEdge^TM AI the first Machine Learning Software, specifically developed to entirely run on microcontrollers.
• DB4345: Data brief for STEVAL-STWINKT1B.
• UM2777: How to use the STEVAL-STWINKT1B SensorTile Wireless Industrial Node for condition monitoring and predictive maintenance applications.