Desk-Robot Assistant Based on M5Stick-V and M5Stack

This project shows in a practical way how you can perform the classification of emotions, using a development system such as the M5StickV. To perform this classification it is necessary to have a database in which emotions are divided by classes which represent the different emotions. The KDEF database (http://kdef.se/) was used to perform the training, this database has a total of 4, 900 images of human facial expressions. The set of images contains 70 individuals showing 7 different emotional expressions.

Before starting the training phase, it was necessary to carry out a pre-processing stage, in which the images were resized. This is necessary because when using a mobile network architecture, it is necessary to resize each image to 224x224.

Windows 10 was used as operating system in which the training phase was executed, this phase can take between 3 and 4 hours (depending on the characteristics of the computer). Once it is done training, the system generates the model *.h5, once you have this model it is necessary to convert it to the format of tensorflow lite *.tflite. This is done using the command:

tflite_convert --keras_model_file " + keras_model_path + " --output_file " + tflite_path

Already having our *.tflite model, the following is transform it to *.kmodel. this is necessary because the KPIU of the M5StickV, only accepts model with this extension. This is done using ncc (https://github.com/kendryte/nncase/releases/tag/v0.2.0-alpha2).

ncc.exe -i tflite -o k210model --dataset " + path_all_dataset + " " + tflite_path + " " + kmodel_path

In this step it is very important that all the images of the dataset are located in a single folder called dataset.

At this point we should have something like this:

 

                                          Model structure

 

Once you have the model in *.kmodel format, you can access the model in two ways. The first using SD memory (in my case my M5StatickV no longer reads it ☹), this can be done in the following way:

task = kpu.load("/sd/model.kmodel")

Or we can use the application Kflash_gui and download the model to the address 0x300000, as shown in the following figure.

 

Kflash_gui

 

Now we need the Micro-pytho program, this was done using the development environment maixpyide_2.3. The following is to convert our emotion classification code into boot.py, this is shown in the following video.

Video1

Finally, the following video shows the classification of emotions, using images searched by google.

Video2

Video3

Robot1

Robot2

Robot3

Robot4

Robot5

Connection diagram

Bibliography

http://jrincon.gti-ia.upv.es/

https://www.sciencedirect.com/science/article/pii/S0925231217311153

http://ceur-ws.org/Vol-2166/afcai18-paper11.pdf

Schematics

Diagram