MCU development -- face recognition application based on esp32-cam

background

I learned a little OpenCV, Familiar with some basic concepts , I looked for whether the single chip computer can support face recognition , Used to make some gadgets .
As a result, a module called ESP32-CAM.ESP32-CAM It can be regarded as the cheapest MCU development board supporting face recognition , The performance is quite good in the single chip computer , Although it only reluctantly supports face recognition . But its advantages are also huge , It's the price , It's too cheap ！！

And in B I saw the work of this big guy on the station ：

be based on esp32cam Face recognition unlock complete tutorial & Exclusive tutorial

So let's learn how to use this module . By the way
Dozens of dollars of face recognition door locks , Dare you use .

But it can be used to make a toy for the child .

Hardware and software preparation

• Hardware aspect

Taobao has a base ESP32-CAM modular , This base mainly solves the problems of power supply and burning , Plug it in directly to connect the computer .

• Software aspect

Refer to the user manual , Set the arduino Development environment of , Installed the latest ESP32 Support package , You can run through the official example , Capable of face recognition . This tutorial will be given away whenever you buy a development board , I won't go into details here .

But the official example , The code is a bit confusing , This version of the code is still open-source by the big guys. The logic is relatively clear .

Detailed face recognition door lock

The big guy in the video has dedicated his code , I would like to express my thanks , You can support .

The code can be downloaded , Understand for yourself .

But when the big guy's code runs here , Will report a mistake ,fr_flash Error of , I haven't found the cause of the problem yet , It may be that the recognized face data cannot be , Store it , Lead to , You need to re-enter your face every time you restart .

Let's talk about the key points of face recognition .
The first is to get the video , The following interfaces are used

fb = esp_camera_fb_get();

Then convert the video to RGB888, The following interfaces are used

fmt2rgb888(fb->buf, fb->len, fb->format, out_res.image);

Operation of human face , It is divided into detection （ See if there are faces ）, entry （ Store the characteristic values of the face ）, And identify （ Identify whether the face captured by the camera is in the library ）. All three operations are required Face detection . The interface used is

out_res.net_boxes = face_detect(image_matrix, &mtmn_config);

And then Face alignment

align_face(out_res.net_boxes, image_matrix, aligned_face)

Again Extract eigenvalues

out_res.face_id = get_face_id(aligned_face);

Be careful , This face_id Namely At the core The data of , Is the eigenvalue of your face . With this data , You can judge the face .
If it is entered , You need to store this data ;
If it is identification , You need to compare this data with the stored features , It means that the right person has been photographed .
Characteristic contrast The core functions of are as follows

face_id_node *f = recognize_face_with_name(&st_face_list, out_res.face_id);

among st_face_list It's just a linked list , The data inside is already stored face_id.

After the whole code idea is sorted out , It is easy to modify . Let's take a look at this face_id What is the structure of the data .

typedef struct tag_face_id_node
{
    
    struct tag_face_id_node *next;           /*!< next face id node */
    char id_name[ENROLL_NAME_LEN];           /*!< name corresponding to the face id */
    dl_matrix3d_t *id_vec;                   /*!< face id */
} face_id_node;

typedef struct
{
    
    int w;        /*!< Width */
    int h;        /*!< Height */
    int c;        /*!< Channel */
    int n;        /*!< Number of filter, input and output must be 1 */
    int stride;   /*!< Step between lines */
    fptp_t *item; /*!< Data */
} dl_matrix3d_t;

You can see this eigenvalue , It is the structure below .
I tried to print a group of data , Find out item For one 512 Of length float Array . This is the feature of your face , It should be one of your biometric fingerprints , It's so easy to get .

flash Problem solving

So let's solve it flash Failed to store... In face_id The question of ,
When it starts , The system calls this function

read_face_id_from_flash_with_name(&st_face_list);

It should be the reason for the error report , Then we don't need this flash Store , Use an array or file to store directly .
There is just a memory card that can be used . The reference code is SD_MMC This example code .
The general principle is to build user data into a directory , Each user enters a set of characteristic values , namely faceID, When you enter it , Store them in the corresponding directory .
When it starts up , By reading each directory , Form core comparative data , Put it in st_face_list variable .

Two core functions , preservation ID

void SD_save_face_id(char* name,int number,dl_matrix3d_t *face_id)
{
    
	int num=0;
	char filename[32]={
    0};
	unsigned char* data=NULL;

	num=(face_id->w)*(face_id->h)*(face_id->c)*(face_id->n);
	sprintf(filename,"/%s",name);
	SD_createDir(SD_MMC,filename);
	sprintf(filename,"/%s/1.txt",name);
	data=(unsigned char*)face_id->item;
	SD_writeFile(SD_MMC, filename, data,num*4);
}

read out ID

void SD_read_all_face_id(fs::FS &fs)
{
    
	st_face_list.count=0;
	st_face_list.head=NULL;
	st_face_list.tail=NULL;
	st_face_list.confirm_times=5;

	face_id_node* faceidnode_now=NULL;

    File root = fs.open("/");
    if(!root)
    {
    
        Serial.println("Failed to open directory");
        return;
    }
    if(!root.isDirectory())
    {
    
        Serial.println("Not a directory");
        return;
    }

    File file = root.openNextFile();
    while(file)
    {
    
        if(file.isDirectory())
        {
    
			face_id_node* faceidnode=NULL;
			char fullname[64]={
    0};
			
            Serial.printf("find user:[%s]\n",file.name());

			sprintf(fullname,"%s/1.txt",file.name());
			
			File tryopen = fs.open(fullname);
			if(!tryopen)
			{
    
				Serial.printf("user has no face id[%s]\n",fullname);
				file = root.openNextFile();
				continue;
			}
			else
			{
    
				faceidnode = (face_id_node*)dl_lib_calloc(1, sizeof(face_id_node), 0);
				if(faceidnode)
				{
    
					dl_matrix3d_t* id_vec;
					strcpy(faceidnode->id_name,file.name()+1);
					id_vec= (dl_matrix3d_t*)dl_lib_calloc(1, sizeof(dl_matrix3d_t), 0);
					if(id_vec)
					{
    
						float* face=NULL;
						
						faceidnode->id_vec=id_vec;
						faceidnode->next=NULL;
						id_vec->w=1;
						id_vec->h=1;
						id_vec->c=512;
						id_vec->n=1;
						id_vec->stride=512;
						
						face= (float*)dl_lib_calloc(512, sizeof(float), 0);
						tryopen.read((unsigned char*)face,512*4);
						id_vec->item=face;

						if(st_face_list.head==NULL)
						{
    
							st_face_list.head=faceidnode;
							st_face_list.tail=faceidnode;
							faceidnode_now = faceidnode;
						}
						else
						{
    
							st_face_list.tail=faceidnode;
							faceidnode_now->next=faceidnode;
							faceidnode_now=faceidnode;
						}
						st_face_list.count++;
					}
				}
			}
        } 
        file = root.openNextFile();
    }
}

See Download for other operation functions , It's too much to put on .

Come on Exhibition

After burning , visit web.

You can enter the face first
Input name , Click on ADD USER, I'll use a picture here instead of .

And then click ACCESS CONTROL, Face recognition

Recognition success , Will prompt DOOR OPEN FOR test.
After reboot , Information will never be lost .
Someone asked , Who is this beautiful woman , That's a little embarrassing , This is a man .

Caton problem

The signal of this module may not be very good , After being far away from the router , It is too laggy. , It's strange that , My family used a router to enhance the signal coverage , This module is connected to the extended router , It will also become Caton .
So I used a separate router , The camera module and the computer form a LAN , And you have to put the module near the router , So there will be no more jamming .

In the future, we will consider whether to use an external antenna , See if it is caused by the problem of the antenna .

Replace the door lock with a light

This module RST There is an onboard red beside the button LED. The LED Internally connected to GPIO 33. You can use this LED Indicate what is happening . for example , If it's connected Wi-Fi, be LED In red , vice versa .
The LED With reverse logic , So you sent a LOW The signal turns it on HIGH The signal turns it off .

So maybe the simulated red light is on , Means unlock , The same thing GPIO It can be used to trigger other operations, not just unlocking , turn on the light , Turn on the tap , Drive what you want .

The code download

Big guy's code , Please go B Stand and watch , Then you can see the download link in the comment area .
My code download
It took a day to change this storage , So I still use a little careful machine .

Conclusion

Do you know what the meaning of life is ？ Share with you a picture

Actually, I think life is meaningful , What we mean , In what we mean to others .