FP-AI-MONITOR1 getting started

Revision as of 07:35, 23 September 2021 by Registered User

Sensing and condition monitoring are two of the major components of the IoT and predictive maintenance systems, enabling context awareness, production performance improvement, a drastic decrease of the downtime due to routine maintenance, and so, maintenance cost reduction.

The FP-AI-MONITOR1 function pack is a Multi-sensor AI data monitoring framework on wireless industrial node, function pack for STM32Cube. It helps to jump-start the implementation and development for sensor-monitoring-based applications designed with the X-CUBE-AI Expansion Package for STM32Cube or with the NanoEdge™ AI Studio. It covers the entire design of the Machine Learning cycle from the data set acquisition to the integration on a physical node.

FP-AI-MONITOR1 runs learning and inference sessions in real time on SensorTile Wireless Industrial Node development kit (STEVAL-STWINKT1B) , taking data from onboard sensors as input. FP-AI-MONITOR1 implements a wired interactive CLI to configure the node, and manage the learn and detect phases. For simple in the field operation, a standalone battery-operated mode allows basic controls through the user button, without using the console.

STEVAL-STWINKT1B has an STM32L4R9ZI ultra-low-power microcontroller (Arm® Cortex®‑M4 at 120 MHz with 2 Mbytes of Flash memory and 640 Kbytes of SRAM). In addition the STEVAL-STWINKT1B embeds industrial-grade sensors, including 6-axis IMU, 3-axis accelerometer and vibrometer to record any inertial and vibrational data with high accuracy at high frequencies.

The rest of the article discusses the following topics:

  • The required hardware and software,
  • Pre-requisites and setup,
  • FP-AI-MONITOR1 console application,
  • Running a classification application for sensing on the device,
  • Running condition monitoring application using NanoEdge™ AI libraries on the device,
  • Performing the vibration sensor data collection using a prebuilt binary of FP-SNS-DATALOG1,
  • Button operated modes, and
  • Some links to useful online resources, to help the user better understand and customize the project for her/his own needs.

This article is just to serve as the quick starting guide and for full FP-AI-MONITOR1 user instructions, readers are invited to please refer to FP-AI-MONITOR1 User Manual.

Info white.png Information
NOTE: The NanoEdge™ library generation itself is out of the scope of this function pack and must be generated using NanoEdge™ AI Studio.

1 Hardware and software overview

1.1 SensorTile wireless industrial node Evaluation Kit STEVAL-STWINKT1B

The SensorTile Wireless Industrial Node (STEVAL-STWINKT1B) is a development kit and reference design that simplifies the prototyping and testing of advanced industrial IoT applications such as condition monitoring and predictive maintenance. It is powered with an ultra-low-power Arm® Cortex®-M4 MCU at 120 MHz with FPU, 2048-Kbyte Flash memory (STM32L4R9). STEVAL-STWINKT1B is equipped with a microSD™ card slot for standalone data logging applications. STEVAL-STWINKT1B also features a wide range of industrial IoT sensors including:

  • an ultra-wide bandwidth (up to 6 kHz), low-noise, 3-axis digital vibration sensor (IIS3DWB)
  • a 6-axis digital accelerometer and gyroscope iNEMO inertial measurement unit (IMU) with machine learning core (ISM330DHCX)

and much more. Please refer to this link for all the sensors and features supported by STEVAL-STWINKT1B. Other attractive features include:

  • a 480 mAh Li-Po battery to enable standalone working mode
  • STLINK-V3MINI debugger with programming cable to flash the board
  • a plastic box for ease of placing and planting the SensorTile on the machines for condition monitoring. For further details, visit this link

1.2 FP-AI-MONITOR1 software description

The top-level architecture of the FP-AI-MONITOR1 function pack is shown in the following figure.

FP-AI-MONITOR1 architecture.png

2 Prerequisites and setup

2.1 Hardware prerequisites and setup

To use the FP-AI-MONITOR1 function pack on STEVAL-STWINKT1B, the following hardware items are required:

  • STEVAL-STWINKT1B development kit board,
  • Windows® powered laptop/PC (Windows® 7, 8, or 10),
  • Two Micro-USB cables, one to connect the sensor-board to the PC, and another one for the STLINK-V3MINI, and
  • an STLINK-V3MINI.
FP-AI-MONITOR1-hardware.png

2.2 Software requirements

2.2.1 FP-AI-MONITOR1

  • Download the FP-AI-MONITOR1 package from ST website, extract and copy the .zip file content into a folder on your PC. The package contains binaries and source code for the sensor board STEVAL-STWINKT1B.

2.2.2 IDE

Info white.png Information
All the steps presented in this document are carried out with STM32CubeIDE, but any of the other two IDEs could have been used.

2.2.3 STM32CubeProgrammer

  • STM32CubeProgrammer (STM32CubeProg) is an all-in-one multi-OS software tool for programming STM32 products. It provides an easy-to-use and efficient environment for reading, writing and verifying device memory through both the debug interface (JTAG and SWD) and the bootloader interface (UART, USB DFU, I2C, SPI, and CAN). STM32CubeProgrammer offers a wide range of features to program STM32 internal memories (such as Flash memory, RAM and OTP) as well as external memories.
  • This software can be downloaded from STM32CubeProg.

2.2.4 TeraTerm

  • TeraTerm is an open-source and freely available software terminal emulator, which is used to host the CLI of the FP-AI-MONITOR1 through a serial connection.
  • Download and install the latest version available from TeraTerm.

2.2.5 STM32CubeMX

STM32CubeMX is a graphical tool that allows a very easy configuration of STM32 microcontrollers and microprocessors, as well as the generation of the corresponding initialization C code for the Arm® Cortex®-M core or a partial Linux® Device Tree for Arm® Cortex®-A core), through a step-by-step process. Its salient features include:

  • Intuitive STM32 microcontroller and microprocessor selection.
  • Generation of initialization C code project, compliant with IAR™, Keil® and STM32CubeIDE (GCC compilers) for Arm® Cortex®-M core
  • Development of enhanced STM32Cube Expansion Packages thanks to STM32PackCreator, and
  • Integration of STM32Cube Expansion Packages into the project.

For downloading and details of all the features please visit st.com.

2.2.6 X-CUBE-AI

X-CUBE-AI is an STM32Cube Expansion Package part of the STM32Cube.AI ecosystem and extending STM32CubeMX capabilities with automatic conversion of pre-trained Artificial Intelligence models and integration of generated optimized library into the user's project. The easiest way to use it is to download it inside the STM32CubeMX tool (version 7.0.0 or newer) as described in user manual Getting started with X-CUBE-AI Expansion Package for Artificial Intelligence (AI) (UM2526). The X-CUBE-AI Expansion Package also offers several means to validate the AI models (Neural Network and Scikit-Learn models) both on desktop PC and STM32, as well as measure performance on STM32 devices (computational and memory footprints) without user handmade ad-hoc C code.

2.2.7 PythonTM 3.7.X

Python is an interpreted high-level general-purpose programming language. Python's design philosophy emphasizes code readability with its notable use of significant indentation. Its language constructs as well as its object-oriented approach aim at helping programmers write clear, logical code for small and large-scale projects. To build and export the Onnx models, set up a Python environment with a list of packages. The list of the required packages along with their versions can be found as a text file in the package at location /FP-AI-MONITOR1_V1.0.0/Utilities/AI_resources/requirements.txt. The following command can be used in the command terminal of the anacondas prompt or Ubuntu to install all the packages specified in the configuration file requirements.txt:

pip install -r requirements.txt

2.2.8 NanoEdge™ AI Studio

The function pack lets the user use the AI libraries generated by NanoEdge™ AI Studio. The Studio can be downloaded from NanoEdge™ AI Studio as a free trial version for the STMicroelectronics boards.

2.3 Installing the function pack to STEVAL-STWINKT1B

2.3.1 Getting the function pack

The first step is to get the function pack. Download the FP-AI-MONITOR1 from ST website. Once the pack is downloaded, unpack/unzip it and copy the content to a folder on the PC. The steps of the process along with the content of the folder are shown in the following image.

FP-AI-MONITOR1 folder contents.png

2.3.2 Flashing the application on the sensor board STEVAL-STWINKT1B

Once the package has been downloaded and unpacked, the next step is to program the sensor node with the binary of the function pack. For the convenience of the users, the function pack is equipped with a pre-built binary file of the project. This binary file can be found at path /FP-AI-MONITOR1_V1.0.0/Projects/Binary/FP-AI-MONITOR1.bin. The sensor board can be very easily programmed with the provided binary by simply performing a drag-and-drop action as shown in the figure below.

FP-AI-MONITOR1 drag drop installation.png

3 FP-AI-MONITOR1 console application

FP-AI-MONITOR1 provides an interactive command-line interface (CLI) application. This CLI application equips a user with the ability to configure and control the sensor node, to perform learning (for NanoEdge™ AI libraries), and (anomaly) detection operations on the edge. The following sections provide a small guide on how to install this CLI application on a sensor board and control it through the CLI from TeraTerm.

3.1 Setting up the console

Once the sensor board is programmed with the binary of the project (as shown in section 2), the next step is to set up the serial connection of the board with the PC through TeraTerm. To do so, start TeraTerm and create a new connection by either selecting it from the toolbar or by selecting the proper port to establish the serial communication with the board. In the figure below, this is COM10 - USB Serial Device (COM 10).

FP-AI-MONITOR1-teraterm connection.png

Once the connection is established, the message below is displayed. If this is not the case, reset the board.

FP-AI-MONITOR1-welcome screen.png

Typing help shows the list of all the available commands along with their usage instructions.

3.2 Configuring the sensors

Through the CLI interface, a user can configure the supported sensors for sensing and condition monitoring applications. The list of all the supported sensors can be displayed on the CLI console by entering the command sensor_info. This command prints the list of the supported sensors along with their ids as shown in the image below. The user can configure these sensors using these ids. The configurable options for these sensors include:

  • enable: to activate or deactivate the sensor,
  • ODR: to set the output data rate of the sensor from the list of available options, and
  • FS: to set the full-scale range from the list of available options.

The current value of any of the parameters for a given sensor can be printed using the command,

$ sensor_get <sensor_id> <param>

or all the information about the sensor can be printed using the command:

$ sensor_get <sensor_id> all

Similarly, the values for any of the available configurable parameters can be set through the command:

$ sensor_set <sensor_id> <param> <val>

The figure below shows the complete example of getting and setting these values along with old and changed values.

FP-AI-MONITOR1 sensor configurations.png

4 Classification applications for sensing

The CLI application comes with a prebuilt Human Activity Recognition model. This functionality can be started by typing the command:

$ start ai

Note that the provided HAR model is built with a dataset created using the IHM330DHCX_ACC sensor with ODR = 26, and FS = 4. To achieve good performance, the user is required to set these parameters to the sensor configurations using the instructions provided in section Configuring the sensors.

Running the $ start ai command starts the inference on the accelerometer data and predicts the performed activity along with the confidence. The supported activities are:

  • Stationary,
  • Walking,
  • Jogging, and
  • Biking.

The following screenshot shows the normal working session of the AI command in CLI application.

AI HAR.png

5 Condition monitoring using NanoEdge™ AI Machine Learning library

FP-AI-MONITOR1 includes a pre-integrated stub that can be easily replaced by an AI condition monitoring library generated and provided by NanoEdge™ AI Studio. Please refer to User Manual for the full details on how to generate and install a NanoEdge™ AI library for accelerometer data using FP-AI-MONITOR1.

For further details on how NanoEdge™ AI libraries work, read the detailed documentation of NanoEdge™ AI studio.

The provided stub simulates the NanoEdge™ AI Studio related functionalities, such as running learning and detection phases on the edge.

The learning phase can be started by simply issuing a command $ start neai_learn from the CLI console. Launching this command starts the process of learning and displays a message on the console every time learning is performed on a new signal (see the below snippet):

 
$ CTRL: ! This is a stubbed version, please install NanoEdge AI library !
NanoEdge AI: learn
{"signal": 1, "status": success}
{"signal": 2, "status": success}
{"signal": 3, "status": success}
:
:
:
End of execution phase

The process can be stopped by pressing the ESC key on the keyboard.

The learning can also be started by performing a long-press of the user button. Pressing the user button again stops the learning process.

Once the normal conditions are learned, the user can start the condition monitoring process by issuing the command $ start neai_detect command. This command starts the inference phase and checks the similarity of the presented signal with the learned normal signals. If the similarity is less then the set threshold defulat: 90%, the similarity value is printed showing the occurrence of an anomaly. This behavior is shown in the snippet below:

$ start neai_detect
NanoEdgeAI: starting detect phase...

$ CTRL: ! This is a stubbed version, please install NanoEdge AI library !
NanoEdge AI: detect
{"signal": 1, "similarity": 0, "status": anomaly}
{"signal": 2, "similarity": 1, "status": anomaly}
{"signal": 3, "similarity": 2, "status": anomaly}
:
:
{"signal": 90, "similarity": 89, "status": anomaly}
{"signal": 91, "similarity": 90, "status": anomaly}
{"signal": 102, "similarity": 0, "status": anomaly}
{"signal": 103, "similarity": 1, "status": anomaly}
{"signal": 104, "similarity": 2, "status": anomaly}
End of execution phase

or by double-pressing the user button on the sensor board. The process can be stopped by pressing the ESC key on the keyboard or alternatively pressing the user button .

Info white.png Information
Note  : The message CTRL: ! This is a stubbed version, please install NanoEdge AI library ! shows that the library embedded in the function pack is just a stub and a real library is not present. This message will disappear once a real library is embedded. The simulations show the similarity for the detection phase is starting as zero and then increasing until 100. The signals where similarity is less than or equal to 90% are reported as anomaly and signals where the similarity is higher than 90 are considered normal. This can be changed by changing the threshold (see user manual for more details).

6 Data collection

The data collection functionality is out of the scope of this function pack. However, to facilitate the users and equip them with the possibility to perform a datalog, a precompiled .bin file from FP-SNS-DATALOG1 is provided in the Utilities and can be found under path /FP-AI-MONITOR1_V1.0.0/Utilities/Datalog/. To acquire the data from any of the available sensors on the sensor-board, the user can program the sensor-board with this binary using the drag-drop method shown in section 2.3.2.

Info white.png Information
For data logging using the high-speed datalogger, the user needs a FAT32-FS formatted microSDTM card.

Once the sensor-board is programmed, the data can be logged using the following instructions.

  • Place a DeviceConfig.json file with the configurations of the sensors to be used for the datalogging in the root folder of the microSDTM card. Some sample .json files are provided in the package and can be found at the location FP-AI-MONITOR1_V1.0.0/Utilities/Datalog/Sample_STWIN_Config_Files/. These files are to be named exactly DeviceConfig.json.
  • Insert the SD card into the STEVAL-STWINKT1B board.
  • Reset the board. Orange LED blinks once per second. The custom sensor configuration provided in DeviceConfig.json is loaded from the file.
  • Press the [USR] button to start data acquisition on the SD card. The orange LED turns off and the green LED starts blinking to signal sensor data are being written into the SD card.
  • Press the [USR] button again to stop data acquisition. Do not unplug the SD card or turn the board off before stopping the acquisition or the data on the SD card will be corrupted.
  • Remove the SD card and insert it into an appropriate SD card slot on your PC. The log files are stored in STWIN_### folders for every acquisition, where ### is a sequential number determined by the application to ensure log file names are unique. Each folder contains a file for each active subsensor called SensorName_subSensorName.dat. Each file contains raw sensor data coupled with timestamps, a DeviceConfig.json with specific information about the device configuration that are necessary for correct data interpretation, and an AcquisitionInfo.json with information about the acquisition.
Info white.png Information
For details on how to use all the features of the provided HSDatalog.bin binary, users are invited to refer to the user manual of FP-SNS-DATALOG1.

7 Button operated modes

To facilitate the usage, the FP-AI-MONITOR1 is equipped with a button-operated mode. The purpose of the button-operated mode is to enable the FP-AI-MONITOR1 to be operated even in the absence of the CLI console. All the (default) values required for running different functions are provided in the firmware. In button-operated mode, the user can start/stop different execution phases by using only the user button. The following table shows the user instruction for the button-operated mode.

Button Press Description Action
SHORT_PRESS The button is pressed for less than 200 ms and released Starts AI inferences for X-CubeAI model.
LONG_PRESS The button is pressed for more than 200 ms and released Starts the learning for NanoEdge™ AI Library.
DOUBLE_PRESS A succession of two SHORT_PRESS in less than 500 ms Starts the inference for NanoEdge™ AI Library.
ANY_PRESS The button is pressed and released (overlaps with the three other modes) Stops the current running execution phase.

The onboard LEDs indicate the status of the current execution phase and the classes during classification. The green LED is used to show the user which execution context is being run.

Pattern Task
OFF -
ON IDLE
BLINK_SHORT X-CUBE-AI Running
BLINK_NORMAL NanoEdge™ AI detection
BLINK_LONG NanoEdge™ AI learning

The Orange LED is used to indicate the output of the running context as shown in the table below:

Pattern Reporting
OFF Stationary if in X-CUBE-AI mode, Normal Behavior when in NEAI mode
ON Biking
BLINK_SHORT Jogging
BLINK_LONG Walking
Double Blink Anomaly detected

Looking at these LED patterns the users is aware of the state of the sensor node even when CLI is not connected.

8 Documents and related resources