How to create Arduino Rock-Paper-Scissors game using NanoEdge AI Studio

This message will disappear after all relevant tasks have been resolved.

Semantic MediaWiki

There are 1 incomplete or pending task to finish installation of Semantic MediaWiki. An administrator or user with sufficient rights can complete it. This should be done before adding new data to avoid inconsistencies.

In this tutorial, we will use AI and a Time Of Flight to create a Rock Paper Scissors game.

The goal is to demonstrate how to use NanoEdge AI Studio and Arduino IDE to be able to create any project you can think of using AI.

NanoEdge AI Studio is a tool developed by STMicroelectronics especially designed for embedded profiles to help them acquire an AI library to embed in their project, only using their data.

Through a simple and step by step process, we will collect data related to your use case, use the tool to get the best model with little effort.

NanoEdge AI Studio libraries are compatible with any Cortex M and since version 4.4, the tool is able to compile library ready to import directly in Arduino IDE.

1. Goals

Create a Rock Paper Scissors game using TOF data and AI from scratch.
Learn how to use NanoEdge AI Studio to easily integrate AI in any future projects through this example.

2. Hardware & Software Needed

Hardware:

An Arduino GIGA R1 board
An Arduino GiGA display shield
An ST time of flight X-nucleo-53L5A1
A Micro USB cable to connect the Arduino board to your desktop machine

Software:

To program your board, you can use the Arduino Web Editor or install the Arduino IDE. We’ll give you more details on how to set these up in the following sections
To create and get the AI model for the shifumi sign recognition, you need NanoEdge AI Studio v4.4 or above

note: For windows user, we recommend using Arduino IDE v1.8.19. Do not use the Microsoft store version.

2.1. Hardware setup:

Just a word concerning the setup montage. You need to plug the display screen on top of the GIGA R1 board and the TOF expansion board under the board like this:

Important:

We need to modify the I2C communication pins between the TOF and the Arduino board to avoid conflict with the display screen: Plug a jumper wire from the SDA1 pin to the 20SDA pin Plug another wire from the SCL1 pin to the 21SCL pin

3. NanoEdge AI Studio

For any part linked with NanoEdge AI Studio, you can take a look at the documentation for more information: https://wiki.st.com/stm32mcu/wiki/AI:NanoEdge_AI_Studio

3.1. Install:

First, we need to install NanoEdge AI Studio. After filling the form you will receive an email with your license. After launching the .exe file for the install, you will be asked to enter your license and you will be good with the install.

In case of troubles, here are useful links: NanoEdge AI Studio wiki: license activation Troubles shootings

3.2. Create a project:

4 kinds of projects are available in NanoEdge AI Studio:

Anomaly detection (AD): to detect a nominal behavior and an abnormal one. Can be retrained directly on board
1 class classification (1c): Create a model to detect also a nominal behavior and an abnormal one but with only nominal data. (in case you cannot collect abnormal example)
N class classification (Nc): Create a model to classify data in multiple classes that you define
Extrapolation (Ex): Regression in short. To predict a value instead of a class from the input data (a speed or temperature for example)

In our case, we want to create an AI able to recognize three signs (scissor, paper and rock) so we click on N class classification >Create new project

In the project settings:

Enter a project name
Define a RAM and FLASH limit if you need
Click SELECT TARGET, then go to the ARDUINO BOARD tab and select the GIGA R1 WiFi
In sensor type, put GENERIC and 1 as the number of axis

Then click SAVE AND NEXT

note: we are working with the TOF matrix of 64 values (8x8), but each measure are taking independently. That the reason why we use 1 axis.

3.3. Data collection:

In the Signal part of NanoEdge AI Studio, we need to import 4 datasets:

Nothing: When we are not playing
Paper
Scissor
Rock

Our TOF can collect data in a matrix of size 8x8, so every signal in our dataset if of size 64. We then need to collect data to create 4 datasets containing various example of one sign each.

Arduino code for data collection:

To collect data, create a new project in Arduino IDE and copy the following code:

#include <Wire.h>
#include <SparkFun_VL53L5CX_Library.h> //http://librarymanager/All#SparkFun_VL53L5CX

SparkFun_VL53L5CX myImager;
VL53L5CX_ResultsData measurementData; // Result data class structure, 1356 bytes of RAM
int imageResolution = 0; //Used to pretty print output
float neai_buffer[64];

void setup() {    
  Serial.begin(115200);
  delay(100);
  Wire.begin(); //This resets to 100kHz I2C
  Wire.setClock(400000); //Sensor has max I2C freq of 400kHz 
  if (myImager.begin() == false)
  {
    Serial.println(F("Sensor not found - check your wiring. Freezing"));
    while (1);
  }
  myImager.setResolution(8*8); //Enable all 64 pads
  myImager.setRangingFrequency(15); //Ranging frequency = 15Hz
  imageResolution = myImager.getResolution(); //Query sensor for current resolution - either 4x4 or 8x8
  myImager.startRanging();
}

void loop() {  
  if (myImager.isDataReady() == true)
  {
    
    if (myImager.getRangingData(&measurementData)) //Read distance data into array
    {
      for(int i = 0 ; i < imageResolution ; i++) {
        neai_buffer[i] = (float)measurementData.distance_mm[i];
      }
      for(int i = 0 ; i < imageResolution ; i++) {
        Serial.print(measurementData.distance_mm[i]);
        Serial.print(" ");
      }
      Serial.println();
    }
  }
}

We need to add 2 libraries in the project:

Wire.h for I2C communication: click on Sketch > Include Library > Wire
SparkFun_VL53L5CX_Library: to use the TOF: Sketch > Include Library > Manage Library > SparkFun_VL53L5CX_Library and click install

Once ready, click on the verify sign to compile the code and then to the right arrow to flash the code on the board. Make sure that the board is connected to the pc beforehand.

Make sure that the right COM port is selected. Click on Tools > Port to select it. If the board is plugged, you should see its name displayed.

Back in NanoEdge:

In the step SIGNALS:

Click on ADD SIGNAL
Click on FROM SERIAL

Make sure to select the right COM port
Leave the Baudrate as it is
Click on maximum number of lines and enter 500
Click START/STOP to log data
Once finished click CONTINUE and then IMPORT

VERY IMPORTANT: Do not log data as if playing sifumi multiple times, log a sign continuously at different positions below the captor (up, down, left, right etc). For example, do the scissor sign and move below the captor while collecting the 500 signal without ever leaving the captor sight.

We really need to only have data corresponding to the class only. If you simulate playing to log data, you will have data corresponding to no signs (because you are not in the captor sight) and then data corresponding to the class.

Make also sure not to be to close to the TOF as it will not be able to see anything but a big object covering the whole matrix.

3.4. Finding the best model

Go to the BENCHMARK step.

In this step, NanoEdge AI Studio will look for the best pretraitment of your data, model and parameters for this model in order to find the best combination for your usecase.

Once done, you will be able at the end of the project to compile the combination found as an AI Library to import in Arduino IDE.

Click NEW BECHMARK
Select the four classes collected previously
Click start

The studio display few metrics:

Balanced accuracy which is the weighted mean of good classification per classes
RAM and FLASH needs
Score: this metrics takes in account the performance and size of the model found

The time required for the benchmark is heavily correlated to the size of buffer used and their quantity. The benchmark will improve fast at the beginning at tends to slow down to find the most optimized library at the end. So you can stop the benchmark when you are satisfied of the results (above 95% is a good reference).

Validate the model found The validation and emulator steps are made to make sure that the library found is indeed the best. To achieve that, it is recommended to test the few best libraries with new data and make sure that the one selected is the best.

note: To collect new dataset for the validation, you can go back to the step signal, import new dataset via serial and download them to use them in validation.

Go to the Validation step
Select 1 to 10 libraries
Click NEW EXPERIMENT
Import new dataset for the 4 classes
click START

You also have the emulator to push tests further if needed on one library and also to test it live using serial for a quick demo for example:

Getting the Arduino library In the last step, the compilation, you can directly get from NanoEdge AI Studio a library ready to be imported in Arduino IDE.

Important: leave the Float abi uncheck for the GIGA R1 WIFI.

Arduino IDE After getting the zip containing the AI library from NanoEdge AI Studio, Create a new project by clicking File > New

3.5. Library setup

To make this project, we need few libraries to make it work.

Click on Sketch > Include Library > Wire to include the Wire.h library for I2C communication (it is installed by default)

To add a standard libraries click on Sketch > Include Library > Manage Library and install the following libraries:

ArduinoGraphics: for display on the screen
SparkFun_VL53L5CX_Library: to use the TOF

To add the NanoEdge AI Studio library:

Extract the .zip provided by NanoEdge AI Studio After compilation
In Arduino IDE click on Sketch > Include Library > Add .ZIP Library…
Find the extracted file and import the zip in the Arduino folder

We also need to add incbin.h:

You can find incbin.h here: https://github.com/graphitemaster/incbin/blob/main/incbin.h
Copy and paste it inside the folder containing the .ino file of the project.

Lastly, we need to select the GIGA R1 WIFI as our board:

Click Tools > Board: “your actual board” > Boards Manager…
Search and Install Arduino Mbed OS Giga Boards
Then go back to Tools > > Board: “your actual board” > Arduino Mbed OS Giga Boards >Arduino Giga R1

Note: Arduino_H7_Video.h is automatically added when selecting the board, you don't need to add it manually.

3.6. Code:

The following code takes care of 3 main parts: Collect data from the TOF Use the NanoEdge AI Library every time we collect data from the TOF to detect what sign is being played Load and display images corresponding to the sign detect by NanoEdge AI Studio

here is the code:

#include "Arduino_H7_Video.h"
#include "ArduinoGraphics.h"
#include "incbin.h"
#include <Wire.h>
#include <SparkFun_VL53L5CX_Library.h> //http://librarymanager/All#SparkFun_VL53L5CX
#include "NanoEdgeAI.h"
#include "knowledge.h"

// Online image converter: https://lvgl.io/tools/imageconverter (Output format: Binary RGB565)

//#define DATALOG
#define SCREEN_WIDTH  800
#define SCREEN_HEIGHT 480
#define SIGN_WIDTH    150
#define SIGN_HEIGHT   200
#define LEFT_SIGN_X   115
#define RIGHT_SIGN_X  530
#define SIGN_Y        200

#define INCBIN_PREFIX
INCBIN(backgnd, "C:/Users/boissona/OneDrive - STMicroelectronics/Documents/POCs/DEMOS_Arduino_2024/Shifumi_2024/Shifumi_Arduino_2024/Pics/backgnd.bin");
INCBIN(rock, "C:/Users/boissona/OneDrive - STMicroelectronics/Documents/POCs/DEMOS_Arduino_2024/Shifumi_2024/Shifumi_Arduino_2024/Pics/rock.bin");
INCBIN(paper, "C:/Users/boissona/OneDrive - STMicroelectronics/Documents/POCs/DEMOS_Arduino_2024/Shifumi_2024/Shifumi_Arduino_2024/Pics/paper.bin");
INCBIN(scissors, "C:/Users/boissona/OneDrive - STMicroelectronics/Documents/POCs/DEMOS_Arduino_2024/Shifumi_2024/Shifumi_Arduino_2024/Pics/scissors.bin");

void signs_wheel(void);
void signs_result(uint16_t neaiclass);
uint16_t mostFrequent(uint16_t arr[], int n);

Arduino_H7_Video Display(SCREEN_WIDTH, SCREEN_HEIGHT, GigaDisplayShield);

Image img_backgnd(ENCODING_RGB16, (uint8_t *) backgndData, SCREEN_WIDTH, SCREEN_HEIGHT);
Image img_rock(ENCODING_RGB16, (uint8_t *) rockData, SIGN_WIDTH, SIGN_HEIGHT);
Image img_paper(ENCODING_RGB16, (uint8_t *) paperData, SIGN_WIDTH, SIGN_HEIGHT);
Image img_scissors(ENCODING_RGB16, (uint8_t *) scissorsData, SIGN_WIDTH, SIGN_HEIGHT);

Image img_classes[CLASS_NUMBER - 1] = {img_paper, img_rock, img_scissors};

SparkFun_VL53L5CX myImager;
VL53L5CX_ResultsData measurementData; // Result data class structure, 1356 bytes of RAM
int imageResolution = 0; //Used to pretty print output
int imageWidth = 0; //Used to pretty print output

float neai_buffer[DATA_INPUT_USER];
float output_buffer[CLASS_NUMBER]; // Buffer of class probabilities
uint16_t neai_class = 0;
uint16_t previous_neai_class = 0;
int class_index = 0;
uint16_t neai_class_array[10] = {0};

void setup() {    
  randomSeed(analogRead(0));
  Display.begin();
  neai_classification_init(knowledge);

  Serial.begin(115200);
  delay(100);
  Wire.begin(); //This resets to 100kHz I2C
  Wire.setClock(400000); //Sensor has max I2C freq of 400kHz 
  if (myImager.begin() == false)
  {
    Serial.println(F("Sensor not found - check your wiring. Freezing"));
    while (1);
  }
  myImager.setResolution(8*8); //Enable all 64 pads
  myImager.setRangingFrequency(15); //Ranging frequency = 15Hz
  imageResolution = myImager.getResolution(); //Query sensor for current resolution - either 4x4 or 8x8
  imageWidth = sqrt(imageResolution); //Calculate printing width
  myImager.startRanging();

  Display.beginDraw();
  Display.image(img_backgnd, (Display.width() - img_backgnd.width())/2, (Display.height() - img_backgnd.height())/2);
  Display.endDraw();
  delay(500);
}

void loop() {  
  if (myImager.isDataReady() == true)
  {
    
    if (myImager.getRangingData(&measurementData)) //Read distance data into array
    {
      for(int i = 0 ; i < DATA_INPUT_USER ; i++) {
        neai_buffer[i] = (float)measurementData.distance_mm[i];
      }

#ifdef DATALOG
      for(int i = 0 ; i < DATA_INPUT_USER ; i++) {
        Serial.print(measurementData.distance_mm[i]);
        Serial.print(" ");
      }
      Serial.println();
#else
      
      neai_classification(neai_buffer, output_buffer, &neai_class);
      
      if(class_index < 10) {
        neai_class_array[class_index] = neai_class;
        Serial.print(F("class_index "));
        Serial.print(class_index);
        Serial.print(F(" = class"));
        Serial.println(neai_class);
        class_index++;
      } else {
        neai_class = mostFrequent(neai_class_array, 10);
        Serial.print(F("Most frequent class = "));
        Serial.println(neai_class);
        class_index = 0;

        if (neai_class == 4 && previous_neai_class != 4)    // EMPTY
        {
          previous_neai_class = neai_class;
          Display.beginDraw();
          Display.image(img_backgnd, (Display.width() - img_backgnd.width())/2, (Display.height() - img_backgnd.height())/2);
          Display.endDraw();
          Serial.println(F("Empty class detected!"));
        }
        else if (neai_class == 1 && previous_neai_class != 1)   // PAPER
        {
          previous_neai_class = neai_class;
          Serial.println(F("Paper class detected!"));
          signs_wheel();
          signs_result(neai_class);
        }
        else if (neai_class == 3 && previous_neai_class != 3)   // SCISSORS
        {
          previous_neai_class = neai_class;
          Serial.println(F("SCISSORS class detected!"));
          signs_wheel();
          signs_result(neai_class);
        }
        else if (neai_class == 2 && previous_neai_class != 2)   // ROCK
        {
          previous_neai_class = neai_class;
          Serial.println(F("Rock class detected!"));
          signs_wheel();
          signs_result(neai_class);
        }
      }
#endif
    }
  }
}


void signs_wheel(void)
{
  for(int i = 0 ; i < 10 ; i++) {
    Display.beginDraw();
    Display.image(img_backgnd, (Display.width() - img_backgnd.width())/2, (Display.height() - img_backgnd.height())/2);
    Display.image(img_SCISSORS, LEFT_SIGN_X, SIGN_Y);
    if(i % (CLASS_NUMBER - 1) == 0) {
      Display.image(img_rock, RIGHT_SIGN_X, SIGN_Y);
    } else if(i % (CLASS_NUMBER - 1) == 1) {
      Display.image(img_paper, RIGHT_SIGN_X, SIGN_Y);
    } else {
      Display.image(img_SCISSORS, RIGHT_SIGN_X, SIGN_Y);
    }
    Display.endDraw();
  }
}


void signs_result(uint16_t neaiclass)
{
  Display.beginDraw();
  Display.image(img_backgnd, (Display.width() - img_backgnd.width())/2, (Display.height() - img_backgnd.height())/2);
  int random_img = random(0, CLASS_NUMBER - 1);
  if(random_img == neaiclass - 1) {
    Display.fill(255, 127, 127);
    Display.rect(LEFT_SIGN_X - 10, SIGN_Y - 10, SIGN_WIDTH + 20, SIGN_HEIGHT + 20);
    Display.rect(RIGHT_SIGN_X - 10, SIGN_Y - 10, SIGN_WIDTH + 20, SIGN_HEIGHT + 20);
  } else if(random_img == (neaiclass == 1) ? 2 : (neaiclass == 2) ? 0 : 1) {
    Display.fill(255, 0, 0);
    Display.rect(LEFT_SIGN_X - 10, SIGN_Y - 10, SIGN_WIDTH + 20, SIGN_HEIGHT + 20);
    Display.fill(0, 255, 0);
    Display.rect(RIGHT_SIGN_X - 10, SIGN_Y - 10, SIGN_WIDTH + 20, SIGN_HEIGHT + 20);         
  } else {
    Display.fill(0, 255, 0);
    Display.rect(LEFT_SIGN_X - 10, SIGN_Y - 10, SIGN_WIDTH + 20, SIGN_HEIGHT + 20);
    Display.fill(255, 0, 0);
    Display.rect(RIGHT_SIGN_X - 10, SIGN_Y - 10, SIGN_WIDTH + 20, SIGN_HEIGHT + 20); 
  }
  Display.image(img_classes[neaiclass - 1], LEFT_SIGN_X, SIGN_Y);
  Display.image(img_classes[random_img], RIGHT_SIGN_X, SIGN_Y);
  Display.endDraw();
  delay(1000);
}


uint16_t mostFrequent(uint16_t arr[], int n)
{
    int count = 1, tempCount;
    uint16_t temp = 0,i = 0,j = 0;
    //Get first element
    uint16_t popular = arr[0];
    for (i = 0; i < (n- 1); i++)
    {
        temp = arr[i];
        tempCount = 0;
        for (j = 1; j < n; j++)
        {
            if (temp == arr[j])
                tempCount++;
        }
        if (tempCount > count)
        {
            popular = temp;
            count = tempCount;
        }
    }
    return popular;
}

Once ready, click on the verify sign to compile the code and then to the right arrow to flash the code on the board. Make sure that the board is connected to the pc beforehand.

Make sure that the right COM port is selected. Click on Tools > Port to select it. If the board is plugged, you should see its name displayed.

3.7. Screen display:

We display on the screen a background and the signs to play SHIFUMI (scissor, paper and rock). You can download these images:

We also need to convert them to bin files using this website:

Online image converter - BMP, JPG or PNG to C array or binary | LVGL
Select the images
Change the output format to Binary RGB565
Click convert

Get all the binary images and copy them in the project folder. You can create a folder named images for example and put them in it. In the code, update the image path. You may need to add the full path of the images

NanoEdge:

Conserning the use of NanoEdge AI Library, it is really simple: We use the function neai_classification_init(knowledge) in the setup() to load the model with the knowledge acquired during the benchmark We use the function neai_classification(neai_buffer, output_buffer, &neai_class) to make the detection. This function takes as input 3 variable that we created as well:

float neai_buffer[64]: the input data for the detection, which are the TOF data
float output_buffer[CLASS_NUMBER]: An output array of size 4 containing the probability for the input signal to be part of each class
uint16_t neai_class = 0: the variable we use to get the class detected. It correspond to the class with the highest probability.

That’s it!

4. Demo Setup:

If you want to reproduce this demo setup, here are the resources used:

3 support plates:

Support.svg
4x Brackets 20.stl: https://www.printables.com/model/113989-corner-bracket-optimized-for-3d-printing/files
M3 nylon 12mm screws and M3 nylon nuts