FP-AI-MONITOR1 an introduction to the technology behind

Comming Soon!

Sensing is a major part of the smart objects and equipment, for example, condition monitoring for predictive maintenance, which enables context awareness, production performance improvement, and results in a drastic decrease in the downtime due to preventive maintenance.

The FP-AI-MONITOR1 function pack for STM32Cube is a multi-sensor AI data monitoring framework on the wireless industrial node. It helps to jump-start the implementation and development of sensor-monitoring-based applications designed with the X-CUBE-AI Expansion Package for STM32Cube or with the NanoEdge™ AI Studio. It covers the entire design of the Machine Learning cycle from the data set acquisition to the integration on a physical node.

The FP-AI-MONITOR1 runs learning and inference sessions in real-time on the SensorTile Wireless Industrial Node development kit (STEVAL-STWINKT1B), taking data from onboard sensors as input. The FP-AI-MONITOR1 implements a wired interactive CLI to configure the node and manages the learn and detect phases. For simple in the field operation, a standalone battery-operated mode allows basic controls through the user button, without using the console.

The STEVAL-STWINKT1B has an STM32L4R9ZI ultra-low-power microcontroller (Arm® Cortex®‑M4 at 120 MHz with 2 Mbytes of Flash memory and 640 Kbytes of SRAM). In addition, the STEVAL-STWINKT1B embeds industrial-grade sensors, including 6-axis IMU, 3-axis accelerometer and vibrometer, and analog microphones to record any inertial, vibrational and acoustic data with high accuracy at high sampling frequencies.

For more information about how to use FP-AI-MONITOR1 look at the FP-AI-MONITOR1 user manual

The rest of the article discusses how to use the source code of the function pack to implement a custom use case like Acoustic Scene Classification.

1. Introduction

The firmware in FP-AI-MONITOR1 implements a complex multi tasking embedded application that takes full advantage of STM32 power. The following image gives an idea of the complexity managed by the firmware: the left side of the image displays fourteen tasks (each task is an independent execution flow) running in parallel, together with several Interrupt Service Routine (ISR), in the single core of STM32L4R9ZI microcontroller.

FP-AI-MONITOR1 performance

At the same time we can see in the right part of the image that, despite the high number of context switches, the average load percentage of the MCU core, with a clock frequency set to 120MHz, is below 50%. These results are made possible by the use of an application framework optimized for STM32, and a set of firmware components designed on top of the framework.

1.1. Who should read this document?

This document is intended for firmware developers who develop or support the development of AI applications on the edge. Users of this document should have some knowledge of the following:

• Real Time Operating System (RTOS). FreeRTOS^TM is part of the kernel of FP-AI-MONITOR1.
• Principle of Object Oriented programming (like inheritance, polymorphism, virtual function).
• Unified Modeling Language (UML)

1.2. Main components of the firmware

We can look at the complex architecture of FP-AI-MONITOR1 from different perspectives, from the high level view presented in the User Manual up to the more detailed diagram that we can find, in the developer documentation available in FP-AI-MONITOR1 local installation folder at this location:

Projects\STM32L4R9ZI-STWIN\Applications\FP-AI-MONITOR1\doc

The next sections give a look at the main components that we need to understand and use for this tutorial. They are:

• eLooM: the application framework.
• Sensor Manager: an eLooM component to configure and get raw data from the sensors on the board.
• Digital Processing Unit (DPU): an eLooM component to process data.

eLooM components - packaging diagram

On top of the above main components the application implements a set of AI related processing tasks:

• AI_USC_Task_t
• AITask_t
• MfccTask_t
• NeaiTask_t

2. eLooM application framework (main concepts)

eLooM stands for embedded Light object-oriented fraMework for STM32​. It is an application framework designed for STM32 and specifically for soft real-time, event drived, multi-tasking and low power embedded applications. It has been optimized over several years.

eLooM - high level class diagram

2.1. The Managed Task abstraction

The framework introduces the concept of Managed Task, that is a RTOS native task extended with new features. These new features are the result of analysis and generalization of problems common to STM32 applications, that leads us to a generic solution with an optimized trade-off between memory footprint and performance. The class diagram for the managed task is displayed in the following image:

eLooM - AManagedTask class diagram

If we think of a Managed Task as an OO class, then it implements a set of methods used by the framework to coordinate the tasks activities at runtime, like:

• System initialization: at startup the system initializes all hardware resources as well as the software ones used by the Managed Tasks.
• Power management: the system coordinates the Managed Tasks to switch from a state to the other of a Power Mode State Machine (PM State Machine, or simply State Machine) defined by the application.
• Error management: the system handles the errors reported by a Managed Task, or its sub-components, and it checks that all Managed Tasks are working fine.

Not all methods are implemented by the AManagedTask or its subclass AManagedtTaskEx. Indeed, in the above image, the methods displayed in italic inside the Virtual methods block are pure virtual functions. Nota that the prefix 'A' in front of the class name (AManagedTask) stands for abstract. This means that we cannot instantiate an object of the class (because it would be an incomplete object), but a developer can subclass one of the Managed Task classes and provide an application specific implementation. This is what happens in FP-AI-MONITOR1. The following image shows the application specific class we provide with the function pack as white rectangles, while the blue ones are provided by the framework.

eLooM - FP classes extending AManagedClassExt

By extending one of the base classes provided by the framework and implementing the pure virtual functions, the application tasks are well integrated and the application code is split from the code base provided by ST.

2.1.1. AI_USC_Task, a concrete example

Let's give a look at one of the managed task created for the function pack, the AI Ultrasound Classification Task ( AI_USC_Task_t). As a general recommendation, the implementation is split in three files, two header files and one source files:

• AI_USC_Task.h: the main header file declares the type for the new managed task and its public API.
• AI_USC_Task_vtbl.h: the header file containing only all the virtual functions implemented by the managed task. Declaring all virtual functions in a separate file is a practical choice to understand at a glance which behaviors are redefined by the class. It is not a mandatory rule. Another way is to search the Outline view of the IDE for all function names that contain _vtbl (that stands for virtual table) as showed in the below image.
• AI_USC_Task.c: the source file defines the managed task. This file contains the implementation of the public as well as the private methods of the managed task.

AI_USC_Task - virtual functions.

How an user defined managed task extends the AManagedTask or the AManagedTaskEx base classes provided by the framework? The technique used is the inheritance by composition because it is easy to implement in C.

This snippet is provided AS IS, and by taking it, you agree to be bound to the license terms that can be found here for the component: Application.

/* file:AI_USC_Task.h */
 
/**
 * Create  type name for _AI_USC_Task_t.
 */
typedef struct _AI_USC_Task_t AI_USC_Task_t;

/**
 *  AI_USC_Task_t internal structure.
 */
struct _AI_USC_Task_t
{
  /**
   * Base class object.
   */
  AManagedTaskEx super;

  /* class specific members must be declared  here */

  /**
   * Task input message queue. The task receives message of type struct AIMessage_t in this queue.
   * This is one of the way the task expose its services at application level.
   */
  QueueHandle_t in_queue;

  /* Task variables should be added here. */

  /**
   * Digital processing Unit specialized for the CubeAI library.
   */
  AiUSC_DPU_t dpu;

  /**
   * Data buffer used by the DPU but allocated by the task. The size of the buffer depend
   * on many parameters like:
   * - the type of the data used as input by the DPU
   * - the length of the signal
   * - the number of signals to manage in a circular way in order to decouple the data producer from the data process.
   * The correct size in byte is computed by the DPU with the method AiUSC_DPUSetStreamsParam().
   */
  void *p_dpu_buff;
};

The first member in the AI_USC_Task_t structure is a variable of the base class (AManagedTaskEx super), so, thanks to the way AManagedTask is defined, it is possible to use a pointer of type AI_USC_Task_t* with all the functions that require a parameters of type AManagedTask* or AManagedTaskEx*. For example:

AI_USC_Task_t my_task;
ACAddTask(pAppContext, (AManagedTask*)&my_task);

In the above example we use the explicit cast of the pointer to avoid compiler warning.

There is one more thing to do: we need to use the virtual functions defined by the AI_USC_Task_t class to override those defined in the base class. This is done during the allocation of an instance of the class, as highlighted in the following snippet:

This snippet is provided AS IS, and by taking it, you agree to be bound to the license terms that can be found here for the component: Application.

/* file:AI_USC_Task.c */

/**
 * The only instance of the task object.
 */
static AI_USC_Task_t sTaskObj;

AManagedTaskEx *AI_USC_TaskAlloc(void)
{
  /* In this application there is only one instance of the task,
   * so this allocator implement the singleton design pattern.
   */

  /* Initialize the super class */
  AMTInitEx(&sTaskObj.super);

  sTaskObj.super.vptr = &sTheClass.vtbl;

  return (AManagedTaskEx*)&sTaskObj;
}

The virtual pointer (sTaskObj.super.vptr) points to a table, named virtual table, containing all the pointers to the virtual functions in the correct order, as defined by the base class:

This snippet is provided AS IS, and by taking it, you agree to be bound to the license terms that can be found here for the component: Application.

/* file:AManagedTaskEx_vtbl.h */

/**
 * Create  type name for _AManagedTask_vtb.
 */
typedef struct _AManagedTaskEx_vtbl AManagedTaskEx_vtbl;

struct _AManagedTaskEx_vtbl {
  sys_error_code_t (*HardwareInit)(AManagedTask *_this, void *pParams);
  sys_error_code_t (*OnCreateTask)(AManagedTask *_this, TaskFunction_t *pvTaskCode, const char **pcName, unsigned short *pnStackDepth, void **pParams, UBaseType_t *pxPriority);
  sys_error_code_t (*DoEnterPowerMode)(AManagedTask *_this, const EPowerMode eActivePowerMode, const EPowerMode eNewPowerMode);
  sys_error_code_t (*HandleError)(AManagedTask *_this, SysEvent xError);
  sys_error_code_t (*OnEnterTaskControlLoop)(AManagedTask *_this);
  sys_error_code_t (*ForceExecuteStep)(AManagedTaskEx *_this, EPowerMode eActivePowerMode);
  sys_error_code_t (*OnEnterPowerMode)(AManagedTaskEx *_this, const EPowerMode eActivePowerMode, const EPowerMode eNewPowerMode);
};

For the AI_USC_Task_t the virtual table is defined in the structure AI_USC_TaskClass_t, and in its initialization code, highlighted in the following snippet, we can see how the virtual functions of the AI_USC_Task_t class are linked with the base class:

This snippet is provided AS IS, and by taking it, you agree to be bound to the license terms that can be found here for the component: Application.

/* file:AI_USC_Task.c */

/**
 * Class object declaration. The class object encapsulates members that are shared between
 * all instance of the class.
 */
typedef struct _AI_USC_TaskClass_t {
  /**
   * AI_USC_Task class virtual table.
   */
  AManagedTaskEx_vtbl vtbl;

  /**
   * AI_USC_Task class (PM_STATE, ExecuteStepFunc) map. The map is implemente with an array and
   * the key is the index. Number of items of this array must be equal to the number of PM state
   * of the application. If the managed task does nothing in a PM state, then set to NULL the
   * relative entry in the map.
   */
  pExecuteStepFunc_t p_pm_state2func_map[];
} AI_USC_TaskClass_t;
};

/**
 * The class object initialization.
 */
static const AI_USC_TaskClass_t sTheClass = {
    /* Class virtual table */
    {
        AI_USC_Task_vtblHardwareInit,
        AI_USC_Task_vtblOnCreateTask,
        AI_USC_Task_vtblDoEnterPowerMode,
        AI_USC_Task_vtblHandleError,
        AI_USC_Task_vtblOnEnterTaskControlLoop,
        AI_USC_Task_vtblForceExecuteStep,
        AI_USC_Task_vtblOnEnterPowerMode
    },

    /* class (PM_STATE, ExecuteStepFunc) map */
    {
        AI_USC_TaskExecuteStepState1,
        NULL,
        NULL,
        NULL,
        AI_USC_TaskExecuteStepAIActive,
        NULL,
        AI_USC_TaskExecuteStepAIActive
    }
};

At this point the AI_USC_Task behaves as a managed task.

There is another question to be answered: how the framework knows about the managed tasks created by the application?

2.2. Application entry points

In a typical bare-metal application with a procedural programming model, a developer starts coding inside the main() function, that is the initial entry point. This is not the case for an eLooM based application where the main.c file is very simple like this:

This snippet is provided AS IS, and by taking it, you agree to be bound to the license terms that can be found here for the component: Application.

/* file main.c */

#include <stdio.h>
#include "services/sysinit.h"
#include "task.h"


/* The main function initializes a minimum set of hardware resources and create the INIT task, before giving the control to the scheduler.
 * The main application entry points are defined in the file App.c
 */

int main()
{
  // System initialization.
  SysInit(FALSE);

  vTaskStartScheduler();

  while (1);
}

Normally the main.c file doesn't need to be modified. eLooM, instead, defines four entry points as weak functions as reported in the following table.

Call order	function prototype	Note
1	IApplicationErrorDelegate *SysGetErrorDelegate(void)	Optional. This function is used by the system during the application startup in order to get an application specific object that implements the IApplicationErrorDelegate interface.
2	IAppPowerModeHelper *SysGetPowerModeHelper(void)	Mandatory. This function is used by the system during the application startup in order to get an application specific object that implements the IAppPowerModeHelper interface.
3	sys_error_code_t SysLoadApplicationContext(ApplicationContext *pAppContext)	Mandatory. Add all managed tasks to the application context
4	sys_error_code_t SysOnStartApplication(ApplicationContext *pAppContext)	Optional. This function is called by the framework at the end of the initialization process and before the INIT task releases the control to the application tasks.

A developer must provide at least the SysGetPowerModeHelper() and the SysLoadApplicationContext() functions. They can be defined in an application file. For example FP-AI-MONITOR1 uses the file App.c for this purpose.

Let's give a look at the SysLoadApplicationContext() function defined in the function pack.

This snippet is provided AS IS, and by taking it, you agree to be bound to the license terms that can be found here for the component: Application.

/* file: App.c */

sys_error_code_t SysLoadApplicationContext(ApplicationContext *pAppContext)
{
  assert_param(pAppContext);
  sys_error_code_t xRes = SYS_NO_ERROR_CODE;

  // Allocate the task objects
  spSPIBusObj     = SPIBusTaskAlloc(&MX_SPI3InitParams);
  spIMP23ABSUObj  = IMP23ABSUTaskAlloc(&MX_DFSDMCH0F1InitParams, &MX_ADC1InitParams);
  spISM330DHCXObj = ISM330DHCXTaskAlloc();
  spIIS3DWBObj    = IIS3DWBTaskAlloc();
  spUtilObj       = UtilTaskAlloc(&MX_TIM5InitParams, &MX_TIM16InitParams, &MX_GPIO_PF6InitParams);
  spNeaiObj       = NeaiTaskAlloc();
  spAIObj         = AITaskAlloc();
  spAI_USC_Obj    = AI_USC_TaskAlloc();
  spMfccObj       = MfccTaskAlloc();
  spControllerObj = AppControllerAlloc();
  DataInjectorTaskAllocStatic(&sDataInjObj);

  // Add the task object to the context.
  xRes = ACAddTask(pAppContext, (AManagedTask*)spSPIBusObj);
  xRes = ACAddTask(pAppContext, (AManagedTask*)spIMP23ABSUObj);
  xRes = ACAddTask(pAppContext, (AManagedTask*)spISM330DHCXObj);
  xRes = ACAddTask(pAppContext, (AManagedTask*)spIIS3DWBObj);
  xRes = ACAddTask(pAppContext, (AManagedTask*)spUtilObj);
  xRes = ACAddTask(pAppContext, (AManagedTask*)spNeaiObj);
  xRes = ACAddTask(pAppContext, (AManagedTask*)spAIObj);
  xRes = ACAddTask(pAppContext, (AManagedTask*)spAI_USC_Obj);
  xRes = ACAddTask(pAppContext, (AManagedTask*)spMfccObj);
  xRes = ACAddTask(pAppContext, (AManagedTask*)spControllerObj);
  xRes = ACAddTask(pAppContext, (AManagedTask*)&sDataInjObj);

  return xRes;
}

At the beginning all the managed task objects are allocated. The allocation strategy is an important topics for an embedded application, but it is not covered in this tutorial. Nevertheless let's note that we prefer to use, whenever it is possible, static allocation, and the Singleton design pattern when it make sense for the specific class.
Once a managed task object is allocated, we tell to the framework that this task is part of the application by using the function ACAddTask() provided by the Application Context API declared in the ApplicationContext.h file , for example:

xRes = ACAddTask(pAppContext, (AManagedTask*)spNeaiObj);

If a managed task is not added to the application context then it is not initialized, its resources are not allocated, the native RTOS task is not created and so on. Basically it doesn't exist during the application execution.

In the above example the six tasks highlighted in yellow are the specific ones created for this function pack, the other tasks are provided by the Sensor Manager component, and we will introduce them in the next section.

The rest of the system initialization is done by the framework thanks to the INIT task that coordinates all managed tasks added to the application context as described in the section eLooM framework > System initialization of the developer documentation available in FP-AI-MONITOR1 local installation folder at this location:

Projects\STM32L4R9ZI-STWIN\Applications\FP-AI-MONITOR1\doc

3. Sensor Manager: main concepts

Sensor Manager is an eLooM-based application-level module that interfaces sensors and offers their data to other application modules. It is implemented as an acquisition engine that:

• Orchestrates multiple managed tasks accesses to sensor bus data as follows:
  • One or more sensor for each managed task
  • Read/write requests via queue to handle concurrency on common buses (a physicals bus is represented as a resource controlled by a managed task acting as gate keeper)
• Defines interfaces to avoid implementation dependencies
• Dispatches events to notify when data ready

The following image is an high level class diagram showing how the Sensor Manager (red boxes ??) extend the framework.

Sensor Manager - high level class diagram.

The current implementation of the Sensor Manager supports these sensors:

• HTS221
• IIS3DWB
• IMP23ABSU
• ISM330DHCX
• LPS22HB
• LPS22HH

More sensors will be supported with further releases of the eLooM component.

You find more information about the Sensor Manager in the the section Sensor Manager of the developer documentation available in FP-AI-MONITOR1 local installation folder at this location:

Projects\STM32L4R9ZI-STWIN\Applications\FP-AI-MONITOR1\doc

3.1. Simple and advanced API exposed by the Sensor Manager

The first thing we need to understand, for the purpose of this tutorial, is how to interact with a sensor supported by the Sensor Manager. We saw that sensors and buses are modeled as subclasses of AManagedTaskEx. It means that working with a sensor is a two steps process:

1. To add the sensor to the Application Context, so that the framework knows about it, and it initializes the sensor subsystem.
2. To interact with the sensor using the proper API.

A sensor task is added to the Application Context during the system initialization as each other managed tasks, as we saw in the code snippet of the function SysLoadApplicationContext(). If the physical sensor is connected to a bus then we need to do the same in the application code. This is normally done after all the objects have been initialized, in the SysOnStartApplication() entry point like in the FP code.

This snippet is provided AS IS, and by taking it, you agree to be bound to the license terms that can be found here for the component: Application.

/* file: App.c */

sys_error_code_t SysOnStartApplication(ApplicationContext *pAppContext) {
  UNUSED(pAppContext);

  /* Re-map the state machine of the Sensor Manager tasks */
  SensorManagerStateMachineRemap(spAppPMState2SMPMStateMap);

  /* Disable the automatic low power mode timer */
  UtilTaskSetAutoLowPowerModePeriod((UtilTask_t*)spUtilObj, 0);

  /* connect the sensors tasks to the SPI bus. */
  SPIBusTaskConnectDevice((SPIBusTask*)spSPIBusObj, ISM330DHCXTaskGetSensorIF((ISM330DHCXTask*)spISM330DHCXObj));
  SPIBusTaskConnectDevice((SPIBusTask*)spSPIBusObj, IIS3DWBTaskGetSensorIF((IIS3DWBTask*)spIIS3DWBObj));

  /* set the output queue for the USB CDC device. It is used to deliver all the incoming input */
  CDC_SetOutQueue(AppControllerGetInQueue((AppController_t*)spControllerObj));

  /* Connect the Util Task with the AppController task to propagate the push button event. */
  UtilTaskSetCtrlInQueue((UtilTask_t*)spUtilObj, AppControllerGetInQueue((AppController_t*)spControllerObj));
  /* register AI processing with the application controller. */
  QueueHandle_t queueAi = AITaskGetInQueue((AITask_t*)spAIObj);
  QueueHandle_t queueMfcc = MfccTaskGetInQueue((MfccTask_t*)spMfccObj);
  QueueHandle_t queueUsc = AI_USC_TaskGetInQueue((AI_USC_Task_t*)spAI_USC_Obj);
  QueueHandle_t queueNeai = NeaiTaskGetInQueue((NeaiTask_t*)spNeaiObj);
  QueueHandle_t queueUtil = UtilTaskGetInQueue((UtilTask_t*)spUtilObj);
  AppControllerSetAIProcessesInQueue((AppController_t*)spControllerObj, queueAi, queueMfcc, queueUsc,queueNeai,queueUtil);

  /* set Mfcc DPU to controller */
  IDPU * p_dpu =  MfccTaskGetDpu((MfccTask_t*)spMfccObj);
  AppControllerSetPreprocessDPU((AppController_t*)spControllerObj,p_dpu);

  /* Set the memory storage area for the Data Injector task */
  DataInjectorTaskSetMemStorageLocation(&sDataInjObj, GET_AI_DATA_STORAGE_ADDR(), AI_DATA_STORAGE_SIZE);

  return SYS_NO_ERROR_CODE;
}

For the second step, the Sensor Manager provides different set of API targeting a simple use of the sensor or a more advanced one. The simple way to operate a sensor is to use the API exported by the SensorManager class.

SensorManager - simple API.

Given the ID of a sensor, it is possible to set its basic parameter like ODR, full scale and so on. To get the actual value of the sensor 's parameter we can use the SMGetSensorObserver() function to get the ISourceObservable interface of the sensor. This interface has been designed to provide an uniform and immutable interface between all sensors. All SensorManager API are based on the idea that a sensor has an ID. To know the ID of a sensor it is possible to use the SIterator or the SQuery services provided by the Sensor Manager. The SIterator is used to iterate trough the sensors collection managed by the Sensor Manager. The SQuery is used to search for sensors that match a given query. This is an example:

This snippet is provided AS IS, and by taking it, you agree to be bound to the license terms that can be found here for the component: Application.

/* using a SIterator object. */
SIterator_t iterator;
SIInit(&iterator, SMGetSensorManager());
while(SIHasNext(&iterator))
{
   uint16_t sensor_id = SINext(&iterator);
   // do something with the sensor
   ISourceObservable *p_sensor_observer = SMGetSensorObserver(sensor_id);
   float odr = ISourceGetODR(p_sensor_observer);
   SMSensorSetODR(sensor_id,  odr+1000);
}

/* using a SQuery object. */
SQuery_t query;
SQInit(&query, SMGetSensorManager());
sensor_id = SQNextByNameAndType(&query, "ism330dhcx",  COM_TYPE_ACC);
if (sensor_id != SI_NULL_SENSOR_ID)
{
  SMSensorEnable(sensor_id);
  SMSensorSetODR(sensor_id, 1666);
  SMSensorSetFS(sensor_id, 4.0);
}

Of course the Sensor Manager allow an advanced use of the sensor. In fact it is possible to subclass a sensor class to access all the sensor internal data and also the low level PID, but this is out of the scope of this tutorial.