NanoEdge AI Library for anomaly detection (AD)

Revision as of 07:11, 15 May 2024 by Registered User (→‎How to get an AI library)
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Warning white.png Warning

Major update for NanoEdge AI v3.
Pay close attention to the new function names, new arguments and new return types.


1. What is NanoEdge AI Library for anomaly detection?

NanoEdge™ AI Library is an Artificial Intelligence (AI) static library for embedded C software running on Arm® Cortex® microcontrollers.

A NanoEdge AI Library is compiled in the last step of Any NanoEdge AI Studio project.

The goal of Anomaly detection libraries is to distinguish normal and abnormal behavior defined during its training in NanoEdge AI Studio. A library contains everything needed to be embedded on a microcontrollers:

  • The AI model and its hyperparameters
  • The preprocessing of the signals

Few files are given to make a use of it:

  • libneai.a contains the ML model as an obfuscated static library
  • NanoEdgeAI.h contains the variables and functions declaration
  • (optional in AD) Knowledge.h model and preprocessing parameters

1.1. Embedded learning

Anomaly detection libraries have the particularity to be retrainable directly on a microcontroller. It is the only kind of library able to do so in NanoEdge AI Studio.

Compared to other kinds of libraries, Anomaly detection libraries need to be retrained before being used. You can do it in three different ways:

  • Deploy the model (untrained), then collect and learn new signals directly on the microcontroller before doing detection. (do not use the Knowledge.h)
  • Deploy the "non-trained" model and also the knowledge. Use the knowledge to retrain the model chmark and then do detection
  • Deploy the "non-trained" model and also the knowledge. Use the knowledge but also collect new signals and use both as training before doing detection.

If you use the knowledge from the benchmark, you don't exploit the advantages of anomaly detection. If you decide to retrain the model with new signal, you should get a model that is more specifically made for its final environment ad thus should give better results.

2. Install / Getting started

The main functions available via the library are:

init() run first before learning/detecting, or to reset the knowledge of the library/emulator
learn() start a number of learning iterations (to establish an initial knowledge, or enrich an existing one)
detect() start a number of detection iterations (inference), once a minimum knowledge base has been established
Warning DB.png Important

When building a smart device, the final features heavily depend on the way those functions are called. It is entirely up to the developer to design relevant learning and detection strategies, depending on the project specificities and constraints.

NanoEdgeAI ild.png

For example for a hypothetical machine, one possible strategy is to:

  1. initialize the model
  2. establish an initial knowledge base by calling learn() every minute for 24 hours on that machine
  3. switch to the inference mode by calling detect() 10 times every hour (and averaging the returned scores), each day
  4. blink a LED and ring alarms whenever detect() returns any anomaly (average score < 90%)
  5. run another learning cycle to enrich the existing knowledge, if the temperature rises above 60°C (and the machine is still OK)
  6. send a daily report (average number of anomalies per hour, with date, time, and machine ID for instance) using Bluetooth® or LoRa®.

In summary, those smart functions can be triggered by external data (for example from sensors, buttons, to account for and adapt to environment changes).
The scores returned by the smart functions can trigger all kinds of behaviors on your device.
The possibilities are endless.

2.1. How to get an AI library

  • In NanoEdge AI Studio, after obtaining a library, click Compile (on the "Deployment" screen, which follows the "Benchmark" and "Validation" screens)
  • (optional) You can check the option to get the knowledge before compiling. the knowledge can be learn to get the model as it was in the benchmark.


The .zip file obtained contains:

  • the static precompiled NanoEdge AI library file libneai.a
  • the NanoEdge AI header file NanoEdgeAI.h
  • the knowledge header file knowledge.h (if checked before the compilation)
  • the NanoEdge AI Emulators (both Windows® and Linux® versions)
  • some library metadata information in metadata.json

To use it, simply add libneai.a, NanoEdgeAI.h and optionally the knowledge.h to your project. Then, link the library for the compilation in your IDE. For example in STM32CubeIDE, go to Project -> Properties -> C/C++ Build -> Settings -> MCU GCC Linker -> Libraries. Add neai in the Libraries section and the libneai.a' path in the Library search path section. Click Apply and Close.

2.2. NanoEdge AI Library functions

Most NanoEdge AI function return the status of the library in the following enum, neai_state:

enum neai_state {
	NEAI_OK = 0,
	NEAI_INIT_FCT_NOT_CALLED = 123,
	NEAI_BOARD_ERROR, 
	NEAI_KNOWLEDGE_BUFFER_ERROR,
	NEAI_NOT_ENOUGH_CALL_TO_LEARNING,
	NEAI_MINIMAL_RECOMMENDED_LEARNING_DONE,
	NEAI_UNKNOWN_ERROR
};
#endif

Here are the possible statuses:

NEAI_OK: the library is working as expected
NEAI_INIT_FCT_NOT_CALLED: the learn or detect function has been called without running the init function before. Initialize your library.
NEAI_BOARD_ERROR: the board detected is not authorized. For instance, it may happen if you are trying to use a limited library (for instance obtained from the free version of NanoEdge AI Studio) with a non-supported STM32 board.
NEAI_KNOWLEDGE_BUFFER_ERROR: the knowledge loaded is not compatible with this library. Make sure that the knowledge being used is the one obtained with this exact library.
NEAI_NOT_ENOUGH_CALL_TO_LEARNING: this is a fail-safe to prevent users from running an insufficient (only one or a few) number of iterations of the learning function. Run more learning iterations.
NEAI_MINIMAL_RECOMMENDED_LEARNING_DONE: Minimal amount of recommended learning have been done.
NEAI_UNKNOWN_ERROR: there is an unknown error with the library.

2.2.1. Initialization

enum neai_state neai_anomalydetection_init(void);

Initialization can be run at the beginning to initialize the model and/or later to initialize a new model and reset all knowledge.
Returns the neai_state enum (NEAI_OK == 0, in case of success).

2.2.2. Include knowledge

The main advantage of anomaly detection libraries is that they can be re-trained directly on the edge. By default, if you use the library on a microcontroller, the model must be retrained to better fit the data in its real environment. But it is also possible to add the knowledge acquired during the benchmark (corresponding to the training data of the project). To do so, after the init() function, call this function to add the knowledge to the model:

enum neai_state neai_anomalydetection_knowledge(const float knowledge_buffer[]);
  • Input:
const float knowledge_buffer[], this buffer is defined in the header file knowledge.h provided in the .zip file containing the static NanoEdge AI Library.
  • Output:
the neai_state enum (NEAI_OK == 0, in case of success).

2.2.3. Learning

enum neai_state neai_anomalydetection_learn(float data_input[]);

This function is used to learn patterns in your data. It can be used at any time, in the beginning to build the original knowledge base of the AI model, but also later, as an additional learning phase to complement the existing knowledge.

  • Input:
float data_input[], the length of the data is BUFFER_SIZE * NB_AXES .
  • Output:
the neai_state enum (NEAI_MINIMAL_RECOMMENDED_LEARNING_DONE or NEAI_NOT_ENOUGH_CALL_TO_LEARNING).
Info white.png Information

The learning function can be called:

  1. initially, before any inference, to establish some reference knowledge base
  2. subsequently, whenever needed, to complete the existing knowledge and enrich it (for example, to take into account some new nominal environment conditions)
Warning white.png Warning

NanoEdge AI Library uses float data types instead of int. If you are using int data types, convert (cast) them into float.

2.2.4. Detection

enum neai_state neai_anomalydetection_detect(float data_input[], uint8_t *similarity);

This function returns returns a similarity percentage, measure of the mathematical distance between the incoming signal and the existing knowledge, learned by the library.

  • Input:
float data_input[], the length of the data is BUFFER_SIZE * NB_AXES.
uint8_t *similarity, the variable that contains the similarity score returned by the function.
  • Output:
The percentage of similarity [0-100] between the new signal and learned patterns ("100" means completely similar, and "0" completely dissimilar).
The neai_state enum.
Info white.png Information
  • The uint8_t *similarity variable must be defined prior to calling the detection function, and pointed to using &similarity when passed as an argument (see code example below).
  • The recommended threshold percentage is 90. Values under this threshold reveal a behavior that differs from the usual behavior learned by the AI model. This threshold can be defined by the user, depending on the final application sensitivity requirements.

2.2.5. Setting sensitivity

enum neai_state neai_anomalydetection_set_sensitivity(float sensitivity);

This function sets the sensitivity of the model in detection mode. It can be tuned at any time without having to go through a new learning phase. This sensitivity has no influence on the knowledge acquired during the learning steps. It only plays a role in the detection step, by influencing the similarity percentages that are returned by the detect function (acts as a linear scaler of the preset internal sensitivity).

  • Input: float sensitivity
  • Output: the neai_state enum
Info white.png Information
  • The default sensitivity value is 1. A sensitivity value between 0 and 1 (excluded) decreases the sensitivity of the model, while a value in between 1 and 100 increases it. STMicroelectronics recommend increasing or decreasing sensitivity by steps of 0.1.
  • Sensitivity 1.1 - 100 tends to decrease the percentages of similarity returned (the algorithm is more sensitive to perturbations), while sensitivity 0 - 0.9 tends to increase them (the algorithm is less sensitive to perturbations).

2.2.6. Getting sensitivity

float neai_anomalydetection_get_sensitivity(void);

This function returns the current sensitivity of the model.
The recommended/default sensitivity is 1. However, this value can be defined by the user depending on his application sensitivity requirements.

  • Input: None
  • Output: The sensitivity of model.

2.3. Backing up and restoring the library knowledge

2.3.1. Creating backups

When using NanoEdge AI Library, knowledge is created on the go. It means that after each learning iteration, the Machine Learning model is incrementally getting richer and richer.

For performance reasons, this knowledge is saved into the microcontroller RAM. Since RAM is volatile, better copy this knowledge into non-volatile memory and prevent its loss (for example, in case of power failure).

NanoEdge AI Library attributes a specific memory section called .neai for the knowledge variables (model hyperparameters). To use it, you need to create a memory section in your linker script according to your microcontroller architecture.

The idea is to create a section called .neai (for NanoEdge AI) in your RAM, using the linker script that is specific to the board / MCU that you use (it is a .ld file).

For example, using a STM32F401RE:

  1. First of all, modify your linker script "STM32F401RETX_FLASH.ld":

At the end of file, just after ._user_heap_stack section, add .neai section :

.neai :
{
  _sneai_knowledge = .;	/* Start address of .neai section */
  KEEP(*(.neai))          /* Keep my variable even if not referenced */
  _eneai_knowledge = .;	/* End address of .neai section */
} >RAM
  1. Once the linker script has been edited, we've to declare our "_neai_knowledge" variable in main.c :
extern void *_sneai_knowledge;
extern void *_eneai_knowledge;
  1. Now, we've to define .neai size and address in Flash where we want to store the knowledge. In your main.c defines section, add :
#define NEAI_KNOWLEDGE_SIZE	((unsigned int)&_eneai_knowledge - (unsigned int)&_sneai_knowledge) /* To determine .neai section size */
[...]
#define WORD_SIZE 4 /* A word is 4 bytes */
#define FLASH_USER_START_ADDR ADDR_FLASH_SECTOR_4 /* Start @ of user Flash area */
#define FLASH_USER_END_ADDR FLASH_USER_START_ADDR + (WORD_SIZE * NEAI_KNOWLEDGE_SIZE) /* End @ of user Flash area */

Make attention to not overwrite .data section in Flash memory. In my case, the start address is at 0x08010000 which works fine. Overwriting .data section could lead to hardfault. The available sectors on NUCLEO-F401RE dev board are:

AI:neai flash sectors.png
2.3.1.1. Save knowledge

Saving knowledge consists to write _neai_knowledge in Flash memory. To perform this operation, you've to erase the sector(s) you want to write then, write in :

erase_flash_memory();
write_flash_memory(_neai_knowledge);

You can have a look to these two functions in the main.c. Check (optionnal) If you want to check the write operation in Flash, you can use STM32CubeProgrammer. Before write, you should see something like :

AI:neai erase.png

After:

AI:neai write begin.png
AI:neai write end.png

Data is written until address 0x08010220 which is normal because : 0x08010000 (start address) + 4 (word size) * 136 (my .neai section size) = 0x08010220.

2.3.1.2. Restore knowledge

After a reboot or a reset, in order to get back the knowledge into RAM, you just have to do :

memcpy((uint32_t *) &_sneai_knowledge, (uint32_t *) FLASH_USER_START_ADDR, NEAI_KNOWLEDGE_SIZE);

With this operation, you copy data from FLASH_USER_START_ADDR in Flash to _sneai_knowledge in RAM. After performing this operation, you can pass in detect mode without doing a new learning phase.

2.4. Example "Hello World!"

Header file: NanoEdgeAI.h

Example of NanoEdge AI Library header file:

This snippet is provided AS IS, and by taking it, you agree to be bound to the license terms that can be found here for the component: Application.
/* =============
Copyright (c) 2021, STMicroelectronics

All rights reserved.

Redistribution and use in source and binary forms, with or without modification, are permitted provided that
the following conditions are met:

* Redistributions of source code must retain the above copyright notice, this list of conditions and the
  following disclaimer.

* Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the
  following disclaimer in the documentation and/or other materials provided with the distribution.

* Neither the name of the copyright holders nor the names of its contributors may be used to endorse or promote
  products derived from this software without specific prior written permission.

*THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES,
INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER / OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY,
WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE
USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.*
*/

#ifndef __NANOEDGEAI_H__
#define __NANOEDGEAI_H__

/* Includes */
#include <stdint.h>

/* Define */

#define AXIS_NUMBER 3
#define DATA_INPUT_USER 256

#ifndef __NEAI_STATE__
#define __NEAI_STATE__
enum neai_state { 
    NEAI_OK = 0,
    NEAI_INIT_FCT_NOT_CALLED = 123,
    NEAI_BOARD_ERROR,
    NEAI_KNOWLEDGE_BUFFER_ERROR,
    NEAI_NOT_ENOUGH_CALL_TO_LEARNING, //This is a fail-safe to prevent users from learning one or even no signals.
    NEAI_UNKNOWN_ERROR};
#endif

/* Function prototypes */
#ifdef __cplusplus
extern "C" {
#endif
	enum neai_state neai_anomalydetection_init(void);
	enum neai_state neai_anomalydetection_learn(float data_input[]);
	enum neai_state neai_anomalydetection_detect(float data_input[], uint8_t *similarity);
	enum neai_state neai_anomalydetection_set_sensitivity(float sensitivity);
	float neai_anomalydetection_get_sensitivity(void);
#ifdef __cplusplus
}
#endif

#endif

/* =============
Here some sample declaration added in your main program for the use of the NanoEdge AI library.
You can directly copy these declarations or modify the names.
* WARNING: respect the size of the buffer.

uint8_t similarity = 0; // Point to similarity (see argument of neai_anomalydetection_detect fct)
float input_user_buffer[DATA_INPUT_USER * AXIS_NUMBER]; // Buffer of input values
*/

Main program: main.c
This program must be completed by the user (depending for instance on the applications or the desired features).

This snippet is provided AS IS, and by taking it, you agree to be bound to the license terms that can be found here for the component: Application.
/* =============
Copyright (c) 2020, STMicroelectronics

All rights reserved.

Redistribution and use in source and binary forms, with or without modification, are permitted provided that
the following conditions are met:

* Redistributions of source code must retain the above copyright notice, this list of conditions and the
  following disclaimer.

* Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the
  following disclaimer in the documentation and/or other materials provided with the distribution.

* Neither the name of the copyright holders nor the names of its contributors may be used to endorse or promote
  products derived from this software without specific prior written permission.

*THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES,
INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER / OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY,
WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE
USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.*
*/
/**
  **************************************************************************
  * Demo: NanoEdge AI process to include in main program body
  *
  * @note  This program must be completed and customized by the user
  **************************************************************************
  */

/* Includes --------------------------------------------------------------------*/
#include "NanoEdgeAI.h"
/* Number of samples for learning: set by user ---------------------------------*/
#define LEARNING_ITERATIONS 256
float input_user_buffer[DATA_INPUT_USER * AXIS_NUMBER]; // Buffer of input values

/* Private function prototypes defined by user ---------------------------------*/
/*
 * @brief Collect data process
 *
 * This function is defined by user, depends on applications and sensors
 *
 * @param sample_buffer: [in, out] buffer of sample values
 * @retval None
 * @note   If AXIS_NUMBER = 3 (cf NanoEdgeAI.h), the buffer must be
 *         ordered as follow:
 *         [x0 y0 z0 x1 y1 z1 ... xn yn zn], where xi, yi and zi
 *         are the values for x, y and z axes, n is equal to
 *         DATA_INPUT_USER (cf NanoEdgeAI.h)
 */
void fill_buffer(float input_buffer[])
{
	/* USER BEGIN */
	/* USER END */
}

/* -----------------------------------------------------------------------------*/
int main(void)
{
	/* Initialization ------------------------------------------------------------*/
	enum neai_state error_code = neai_anomalydetection_init();
	uint8_t similarity = 0;

	if (error_code != NEAI_OK) {
		/* This happens if the library works into a not supported board. */
	}

	/* Learning process ----------------------------------------------------------*/
	for (uint16_t iteration = 0 ; iteration < LEARNING_ITERATIONS ; iteration++) {
		fill_buffer(input_user_buffer);
		neai_anomalydetection_learn(input_user_buffer);
	}
	
	/* Detection process ---------------------------------------------------------*/
	while (1) {
		fill_buffer(input_user_buffer);
		neai_anomalydetection_detect(input_user_buffer, &similarity);
		/* USER BEGIN */
		/*
		 * e.g.: Trigger functions depending on similarity
		 * (blink LED, ring alarm, etc.).
		 */
		/* USER END */
	}
}

3. Resources

Documentation
All NanoEdge AI Studio documentation is available here.

Tutorials
Step-by-step tutorials to use NanoEdge AI Studio to build a smart device from A to Z.