NanoEdge AI Anomaly Detection library for ISPU

This message will disappear after all relevant tasks have been resolved.

Semantic MediaWiki

There are 1 incomplete or pending task to finish installation of Semantic MediaWiki. An administrator or user with sufficient rights can complete it. This should be done before adding new data to avoid inconsistencies.

1. What is NanoEdge AI Library for anomaly detection?

NanoEdge™ AI Library is an Artificial Intelligence (AI) static library originally developed by Cartesiam, for embedded C software running on Arm^® Cortex^® microcontrollers.

When embedded on microcontrollers, it gives them the ability to easily "learn" and "understand" sensor patterns, by themselves, without the need for the user to have additional skills in Mathematics, Machine Learning, or data science.

The NanoEdge AI static library for anomaly detection is the code that contains an AI model (for example, as a bundle of signal treatment, Machine Learning model, and optimally tuned hyperparameters). It is designed to gather knowledge incrementally during a learning phase to become able to detect potential anomalous machine behaviors, and possibly predict them.

Information

In the free version of NanoEdgeAIStudio the NanoEdge AI library is only compatible with a selection of STM32 evaluation boards. See the full list here in the Studio documentation.

2. What is the Intelligent Sensor Processing Unit (ISPU)?

The ISPU (intelligent sensor processing unit) is an embedded programmable core that allows reading sensor data and processing it inside the ISM330ISN device and can directly provide, when necessary, the results of said processing to an external microcontroller. The ISPU can run any type of processing, from basic signal processing to AI algorithms. NanoEdgeAI Studio provides ready-to-use Anomaly Detection binary for the ispu.

3. How it works ?

As described above, NanoEdgeAI Library is designed to detect anomalies from a signal. It will run continuously in the ISPU as it is an always-on device. A learning phase in needed to learn the nominal behavior of the monitoring device. This phase can be start and stopped sending commands from the main MCU to the ISPU. This learning phase will create the knowledge of the NanoEdgeAI Library. Considering that the ISPU does not have any on-board nonvolatile memory, the knowledge will be lost each time power to the device is removed.

Once the learning phase is done, the ISPU can be set in detection mode. The monitoring will be done continuously. Each time a signal is detect as abnormal, the ISPU will rise an interruption. If wanted, the ISPU could send the raw data of the corresponding signal.

4. Getting Started

4.1. NanoEdgeAI Studio with the ISPU

In NanoEdge Studio select the ISM330ISN in the target list.

Information

To learn how to use the Studio, please refer to the following documentation

Once a benchmark is complete, the Deployement section is enable. Select the corresponding benchmark, and the measures settings as below.

Important

Datarate and Full Scale must correspond with the ones used during the datalogging step.

After clicking Compile and selecting your library type, a .zip file is downloaded to your computer.

It contains:

the ispu binary NanoEdge AI ispu_neai.h

This file contains the complete binary. It should not be modified. It declares an array used in the initialization phase to load the binary in the ISPU's memory.

the ispu control source file ispu_ctrl.c
the ispu control header file ispu_ctrl.h
the shared memory file shared_mem.h

Those files are used to control the ispu. It includes all the functions to initialize, send commands, process the answers etc.

the NanoEdge AI Emulators (both Windows^® and Linux^® versions)
some library metadata information in metadata.json

4.2. MCU configurations

4.2.1. I2C/SPI

The ISPU can be connected via I2C or SPI. In both case, to simplify the use of ispu_ctrl library, ispu_write and ispu_read has to be filled with the I2C or SPI read and write functions according to the library used.

void ispu_read(uint8_t reg, uint8_t *val, uint16_t len)
{
	/* USER BEGIN */

	/* USER END */
}

void ispu_write(uint8_t reg, uint8_t val)
{
	/* USER BEGIN */

	/* USER END */
}

4.2.2. Interruptions

The ISPU use two GPIOs to rise interruptions. Using those interruptions, it is possible to let the main MCU in a sleep mode and only be waken if an anomaly happen. Both GPIOs must be configured in rising edge only, no pull.

  GPIO_InitStruct.Pin = INT2_Pin;
  GPIO_InitStruct.Mode = GPIO_MODE_IT_RISING;
  GPIO_InitStruct.Pull = GPIO_NOPULL;
  HAL_GPIO_Init(INT2_GPIO_Port, &GPIO_InitStruct);

5. Developping an always-on monitoring device

5.1. Load the device

As mentioned above, the ISPU does not have on-board non volatile memory. The ispu_neai.h contain an array with all the registers values that has to be send though the i2c/spi.

	for (uint32_t i = 0; i < sizeof(ispu_conf) / sizeof(ucf_line_t_neai); i++) {
		if (ispu_conf[i].op == MEMS_UCF_OP_WRITE)
			ispu_write(ispu_conf[i].address, ispu_conf[i].data);
		else if (ispu_conf[i].op == MEMS_UCF_OP_DELAY)
			HAL_Delay(ispu_conf[i].data);
	}

5.2. Learning mode

NanoEdgeAI Library in Anomaly Detection needs a learning phase. This learning in the ISPU, will create the knowledge of the library. All those signals learned will be considered as nominal. The learning mode can be set at anytime to add new nominal signals to the knowledge. The ISPU does not have on-board nonvolatile memory, hence, the knowledge will be lost each each time the ISPU's power is cut.

In learning mode, the ISPU get new datas from the accelerometer and complete the knowledge as long as the mode is changed. INT2 is rised each time a signal has been learned. It helps to count the number of learning done. However, the most important is to create a complete learning with all the nominal regimes. The more the best

5.3. Detection mode

In detection, the ISPU will monitor continuously.

Mode Detect:

Each time the signal is identified as abnormal, the ISPU rise INT2 and continuous its monitoring.

ispu_send_cmd(SET_MODE, MODE_DETECT);

Mode Detect Consecutively:

This mode will trigger an interruption only if the signal is identified several times consecutively as abnormal.

ispu_send_cmd(SET_MODE, MODE_DETECT_CONSECUTIVE, 5);

5.4. Fetching data

If the ISPU rise an interruption in detection mode (meaning that the last signal was abnormal), it is possible to fetch this signal in order to save it in the MCU memory for a postpone analysis. To do so, a FETCH_DATA command has to be send. When ready, the ISPU will answer by rising the INT1 pin. This mechanism is used to communicate between ISPU and MCU, it is used as long as the signal is fully fetched. The process_answer() available in ispu_ctrl.c is called each time INT1 is rised to process this asynchronous communication. Please refer to the hello_world.

5.5. Example "Hello World!"

/* =============
Copyright (c) 2022, STMicroelectronics

All rights reserved.

Redistribution and use in source and binary forms, with or without modification, are permitted provided that
the following conditions are met:

* Redistributions of source code must retain the above copyright notice, this list of conditions and the
  following disclaimer.

* Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the
  following disclaimer in the documentation and/or other materials provided with the distribution.

* Neither the name of the copyright holders nor the names of its contributors may be used to endorse or promote
  products derived from this software without specific prior written permission.

*THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES,
INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER / OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY,
WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE
USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.*
*/
/* Private includes ----------------------------------------------------------*/
#include <stdio.h>
#include <string.h>
#include "ispu_ctrl.h"
#include "ispu_neai.h"
#include "shared_mem.h"


enum ispu_mode ispu_mode;
enum ispu_comm ispu_comm;
enum neai_state neai_state;
int anomaly_detected = 0;
int learn_count;


void ispu_read(uint8_t reg, uint8_t *val, uint16_t len)
{
	/* USER BEGIN */

	/* USER END */
}

void ispu_write(uint8_t reg, uint8_t val)
{
	/* USER BEGIN */

	/* USER END */
}

int ispu_init(void)
{
	uint8_t who_am_i;

	// Reset the ISPU
	ispu_write(0x12, 0x81);
	while (1) {
		uint8_t ctrl;

		ispu_read(0x12, &ctrl, 1);
		if (!(ctrl & 1))
			break;
	}

	// Make sure the user bank is selected
	ispu_write(0x01, 0x00);

	// Verify the ISPU code
	ispu_read(0x0F, &who_am_i, 1);

	if (who_am_i != 0x22) {
		while (1) {
			HAL_Delay(1000);
			printf("Error: adapter not recognized (%02x)\n", who_am_i);
			return -1;
		}
	}

	initialize_ctrl_interface();

	// load device configuration
	for (uint32_t i = 0; i < sizeof(ispu_conf) / sizeof(ucf_line_t_neai); i++) {
		if (ispu_conf[i].op == MEMS_UCF_OP_WRITE)
			ispu_write(ispu_conf[i].address, ispu_conf[i].data);
		else if (ispu_conf[i].op == MEMS_UCF_OP_DELAY)
			HAL_Delay(ispu_conf[i].data);
	}

	// wait until the ISPU raises the boot flag
	uint8_t status;
	ispu_write(0x01, 0x80);
	do {
		ispu_read(0x04, &status, 1);
		printf("status: %#02x\n", status);
	} while (!(status & (1 << 2)));
	ispu_write(0x01, 0x00);

	return 0;
}

/**
  * @brief  The application entry point.
  * @retval int
  */
int main(void)
{
	int ret;

	ret = ispu_init();
	if (ret)
		return ret;

	neai_state = get_neai_status();

	if (neai_state == NEAI_BOARD_ERROR) {
		// NanoEdgeAI is not loaded on a compatible ISPU board.
	}

	do {
		ispu_comm = ispu_send_cmd(SET_MODE, MODE_LEARNING);
	} while (ispu_comm != ISPU_COMM_OK);

	// ... Depending on the use case, make sure your learning phase is long enough ...
	HAL_Delay(20000);
	// ... few hours later ...

	do {
		ispu_comm = ispu_send_cmd(SET_MODE, MODE_DETECT);
	} while (ispu_comm != ISPU_COMM_OK);


  /* Infinite loop */
  while (1)
  {
	  // ... Now it's time to sleep ...
	  // Do not disturb me unless there is an anomaly 
	  // If there is an anomaly, you can fetch the corresponding data   
	  if (anomaly_detected) {
		  neai_state = get_neai_status();
		  if (neai_state == NEAI_NOT_ENOUGH_CALL_TO_LEARNING) {
			  // Please make sure that the ISPU spend enough time in learning mode
		  }
			  
		  do {
			  ispu_comm = ispu_send_cmd(FETCH_DATA);
		  } while (ispu_comm != ISPU_COMM_OK);
	  }
  }
}

// Example of an interrupt callback function 
// --------------------------------------------
void HAL_GPIO_EXTI_Callback(uint16_t GPIO_Pin)
{
	int16_t acc_x, acc_y, acc_z;

	switch (GPIO_Pin) {
	case INT1_Pin:
		process_answer();
		break;
	case INT2_Pin:
		switch (ispu_mode) {
		case MODE_SAMPLING:
			get_accelerometer_samples(&acc_x, &acc_y, &acc_z);
			printf("ACC x: %d, y: %d, z: %d\n", acc_x, acc_y, acc_z);
			break;
		case MODE_LEARNING:
			learn_count++;
			printf("Learning in progress %d !!!\n", learn_count);
			break;
		case MODE_DETECT:
			printf("Anomaly detected !!!\n");
			anomaly_detected = 1;
			break;
		default:
			break;
		}
		break;
	}
}

6. Other commands

Trigger percentage

By default a signal is considered as abnormal when its similarity is below 90%. This value can be modified using the command below :

ispu_send_cmd(SET_TRIGGER_PERCENTAGE, int percentage_value);

7. Resources

Documentation
All NanoEdge AI Studio documentation is available here. he