How to convert mindspire model to onnx format and use onnxruntime reasoning - development test

Tips

1 This document is based on “onnx Preparation ” Based on the documentation , By default, operator statistics have been completed ,models and mindspore The warehouse clone Work , If not, please review the document , Or check out Web version course ：https://bbs.huaweicloud.com/blogs/360051.

2 In document Basic concepts Etc. if Have mastered and can skip .

3 Students who do not need development ： If all operators have been implemented , Please start directly from Chapter 2 .

4 Students who need development ： Used in document Pad operator To introduce Development process The part of the is written in great detail , Some students may feel very wordy , But I still hope to Read it patiently , In this way, the whole development process can be clarified .

5 This document is also available on the web ：, You can support me if you like , Brush the amount of reading .

6 If there is a mistake , Welcome correction and feedback

Common website

1 How to integrate MindSpore Model transfer ONNX Format and use OnnxRuntime Reasoning --- Preliminary preparation

2 How to integrate MindSpore Model transfer ONNX Format and use OnnxRuntime Reasoning --- Development and testing

3 CNN+CTC ONNX export onnx_exporter.cc file PR

4 CNN+CTC ONNX Export script + Reasoning scripts PR

5 MindSpore official Model Warehouse

6 MindSpore Official model checkpoint Download url

7 MindSpore Official operator query website

8 Onnx Operator Query Library

Catalog

Tips    

Common website     

One 、     Operator mapping      1

1.     Basic concepts      1

（1）     chart      1

（2）     node      2

2.     Development process      3

（1）     Operator mapping analysis      3

（2）     Declare mapping methods      6

（3）     Register operator mapping table      7

（4）     Implement the mapping method      7

（5）     Get node input and attributes      7

（6）     Numerical mapping      7

（7）     Add a value node      8

（8）     Add operator node      9

（9）     Onnx Figure adding operator nodes      9

3.     Official case studies      11

（1）     one-on-one （ Exactly the same ）：MS Conv2D -> Onnx Conv     11

（2）     one-on-one （ Not exactly ）：MS ExpandDims -> Onxx Reshape     11

（3）     One to many ：MS BatchMatMul -> ONNX Transpose + MatMul     13

Two 、     Compile and export      15

1.     MindSpore compile      16

2.     MindSpore Package reinstallation or Python Run path settings （ recommend ）     16

（1）     MindSpore Package reinstallation      16

（2）     Python Run path settings      16

3.     Onnx Model export      17

3、 ... and 、     Reasoning test      17

1.     Dataset Download      17

（1）     basis Models README download      17

（2）     According to the source code of the original paper of the model README download      17

2.     MindSpore Model reasoning      18

（1）     MindSpore Reasoning      18

3.     Onnx Model reasoning      19

（1）     Onnx Inference file      19

（2）     Onnx Reasoning scripts      19

4.     PR Submit      20

（1）     Leader Create a branch      20

（2）     Download new MindSpore and Models Warehouse      20

（3）     Update replacement file      20

（4）     Upload code      20

5.     establish PR     22

6.     Access control test      23

1、

2、 Operator mapping

1. Basic concepts

（1） chart

The Huawei official website defines a graph as follows ：

Actually develop onnx You don't need to be completely familiar with Clear picture The concept of , Instead, the graph is abstractly understood as a Calculation The flow chart will better understand its role and call process , Suppose we now need to compute the following expression ：F(x, y, z)=（x+y）*z-b, Then it corresponds to Calculation The flow chart is as follows ：

chart 1F(x, y, z) Calculation flow chart

chart 1 Every circle or ellipse in is a node , and All these nodes together make up us F（x, y, z） The calculation flow chart of this function , and “ chart ” Is to use computer code to represent and record this kind of calculation process . And in this picture , We can see that there are three different colors of nodes , For us this time onnx In terms of mapping, they are the three most commonly used nodes ： Enter the value node （ Orange x, y, z）、 Constant value node （ gray b） and Operator nodes （ Blue +, *, -）. You have also noticed that I put Blue node （+, *, -） Call it an operator, not an operator , This is because these operators are actually most The basic operator , In fact, the operators we use are the superposition of such basic operators , Which is what we call Library method And library functions . Therefore, the graph can be 1 Convert to operator calculation graph 2：

chart 2 Operator calculation diagram

So far we are right “ chart ” I have a simple understanding that I personally feel is completely adequate , Next, the nodes Merge into two categories ： Numerical nodes and operator nodes , And introduce the related concepts of nodes .

（1） node

1 Node name

It can be noted that chart 2 in I can only put the operator （+, *, -） Replace with the name of the operator , A sequence number is also marked on the edge of each node , This sequence number can be understood as the unique identifier of each node , That is, a mark used by the computer to store and record the operation flow . This mark is used in this onnx Set as this node in the operator map “name”, And for this The picture says There is a variable to store all the nodes in it “name”, That's what the official document says ：“node_map_ptr”< node , Node number >.

2 Numerical node

The previous input value node and constant value node are used to store data values , It mainly includes the following information ：

Name（ Serial number ）： Unique identifier of the node （ If it is not easy to understand, it is all understood as serial number ）.

Value（ The number ）： Enter or set the value of the constant , Such as 1,10,1000 etc. .

Type（ type ）： The data type of the value stored in this node .

Output（ Output ）： Numerical nodes need to be transmitted to other nodes for calculation , Therefore, you need to set the output of this node （ That is to say, what is it To downstream nodes ）, actually In most cases All are add_output(name), That is, pass down your serial number , At that time, the downstream nodes can pass name Access the values .

3 Operator nodes

The operator node defines the calculation logic of the data , The calculation part is “ Operator development ” What needs to be done , This time onnx The mapping only needs to adjust and map the following information ：

Name（ Serial number ）： Unique identifier of the node （ If it is not easy to understand, it is all understood as serial number ）.

OpType（ Operator type ）： This can be simply understood as the name of this operator , such as “+” His OpType Set to “Add”.

Input（ Input ）： Define this What is the input to the operator , That is, the output of any numerical node or other operator node is passed to this operator , Most of the time add_input(name_xxx), That is, tell the computer that I want the serial number to be name_xxx As my input .

Attr（ attribute ）： Some operators have attribute values in addition to inputs , Attribute values are assigned during operator initialization , And in the In this project, it can be regarded as the same thing as input , It's just that the methods of obtaining are different .

Output（ Output ）： Define this Operator's Output What is it? , the What values are passed to downstream operators , In fact, most of the cases are also add_output(name1), That is, the output of this operator is its unique identifier , If the downstream operator needs to be called, you can add_input(name1) To set this operator as input .

By setting up each Of numeric nodes value,type,output And operator nodes Of input and output After the message , The whole picture It's connected .

1. Development process

this Development process This time I will onnx Reasoned CNN+CTC Missing from the model Pad For example, operators , Describe with function + Code + Mapping map Explain the methods that need to be used in the development in the form of , After learning these methods, it can be completed through splicing and combination other Mapping of operators .

（1） Operator mapping analysis

1 MindSporePad Operator interface analysis

MindSpore Official operator query website ：

https://www.mindspore.cn/docs/zh-CN/r1.7/index.html

This operator is actually very simple , Enter an arbitrary dimension of Tensor, Based on the paddings Attribute in The corresponding dimensions are added before and after 0 value , With 2 dimension Tensor For example ：

Input input_x as follows ：

attribute paddings as follows ：

input_x Of shape yes （2,4）, This paddings The meaning of attribute value is , stay input_x Of the 0 Dimensions increase 3（ front 1, Back 2）, For this example, add three lines （ above 1 That's ok , below 2 That's ok ）, And then in input_x Of the 1 Dimensions （ Column ） increase 3 Column （ On the left 2 Column , The right column ）, All added rows and columns are filled with values 0, give the result as follows ：

and mode The attribute value is used to set the filling method , Divided into three , You can Official website Case study , This network uses the default value, so it is not analyzed here .OnnxPad Operator interface analysis

Onnx Operator query URL ：

https://github.com/onnx/onnx/blob/main/docs/Operators.md

You can see Onnx There is also this in Pad operator , After a simple review, it is found that the functions are the same , Then go to the next step .

4 Difference analysis

Through comparative analysis, we can see MindSpore Medium Pad Operator and Onnx Medium Pad The operator functions are the same , But there are differences in input and attributes , Specific for ：

MindSpore Used to set to add dimensions paddings Is an attribute , and Onnx Which have the same function pads It's an input , therefore We need to write paddings To pads Mapping .

meanwhile paddings The request is a shape by [N,2]（N by input_x Dimensions ） Of tuple, Its The first D That's ok Of The first 0 Elements represent in input_x Of the D dimension The quantity increased in front , And the first 1 Elements represent the elements added later . and pads Must be a 1D Of tensor Its shape by [2 * N], Its numerical form is [dim1_begin, dim2_begin,…,dim1_end, dim2_end] among dim1_begin and dim1_end It's on behalf of input_x Of the 1 The number of dimensions added before and after .

The words may be dizzy , Let's look at the picture below ：

This picture clearly shows paddings and pads The correspondence between , At the same time, you can get paddings To pads Mapping what needs to be done ：

• MindSpore attribute paddings It maps to Onnx Input pads

• take paddings from 2D tuple type It maps to 1D tensor type

• take paddings Of The values are correctly mapped to pads Of The number

and because CNN+CTC The default is used in the network mode Property value CONSTANT, Fill in the new row and column 0, Therefore, there is no analysis here , Again, no mapping is required , If mode If the attribute value is set to a specific value, you need to add a pair of mode Mapping of attributes .

So far, we have completed the analysis of the mapping operator , Now you can enter the development stage , If it should be in that line or Where to write corresponding The code can be viewed CNN+CTC ONNX Derived PR link , There are clear addition and deletion codes and code line numbers ：

https://gitee.com/mindspore/mindspore/pulls/35915/files

（ If you find that the operator you want to map can map perfectly , namely MindSporeXXX Operators and OnnxXXX The input and output of the operator are identical , Then you can skip this chapter , Go straight into The first 3 section Official case studies ）.

（2） Statement Mapping method

stay OnnxExporter Class Class private method ExportPrimXXX：

The meanings of the four parameters are ：

• func_graph yes MindSpore Graph .

• node Is the current node in the graph , In this function is Pad Operator nodes .

• node_map_ptr Store all nodes in the graph < node , Node number >.

• graph_proto yes ONNX Graph .

（3） Register operator mapping table

stay OnnxExporter::ExportCNode Method to register the operator mapping table ：

（4） Implement the mapping method

Realization of the first （2） Mapping method declared in step ExportPrimXXX：

（5） Get node input and attributes

obtain MindSporePad Operator No 1 Inputs input_x Of name：

obtain MindSporePad The properties of the operator paddings value ：

（6） The number mapping

take paddings（2Dtuple） Turn into pads_sequence（1D vector）, And complete the numerical mapping

（7） Add a value node

Numerical value pads_sequence Register as a numeric node pads：

stay node_map_ptr in registration pads Constant Numerical node And get it name：

stay ONNX chart graph_proto Create a new constant value node in pads_node, And specify its output as pads Constant value node name：

by pads_node Add a value Numerical properties , And the value node pads The value in is converted to onnx：：tensor after Assign a value to value：

（8） Add operator node

This time Pad Operator mapping does not involve the addition of operator nodes , The first 3 section Of BatchMatMul operator How to add operator nodes will be described in the mapping .

（9） Onnx Figure adding operator nodes

stay node_map_ptr Registered in At present MindSporePad operator node And get it name：

stay ONNX chart graph_proto New China ONNXPad Operator nodes ：

Appoint ONNXPad The output of the operator node is MindSporePad Operator node Output （name） namely ms_pad_node_name：

Appoint ONNXPad The th... Of the operator node 1 The first input is MindSporePad Second order of operator 1 Inputs input_x Of name namely x_name：

Appoint ONNXPad The th... Of the operator node 2 The first input is （7） Created in the pads Of numeric nodes name namely pads_name：

When the mapping is complete ：

2. official case analysis

The official case will not be drawn , Analyze with function description and code , At the same time, the declaration of mapping method and operator registration will not be repeated , Analyze the core mapping code directly .

（1） one-on-one （ Exactly the same ）：MSConv2D -> Onnx Conv

1 Mapping analysis

Official case , No analysis , The result is MindSpore Conv2D and Onnx Conv The input and attributes of can correspond exactly to , So just write One Segment code , Just map the parameter names with the same function ：

（2） one-on-one （ Not exactly ）：MS ExpandDims -> OnxxReshape

1 Mapping analysis

MindSporeExpandDims operator -> ONNX Reshape operator , Input and output can correspond to , Attributes have different meanings , Cannot directly correspond to , ExpanDims Input axis(int) Is the axis to extend the dimension ,Reshape Input shape Is the expanded dimension , Special treatment is required during conversion . So we need to be clear about what the mapping needs to do ：

• obtain MindSporeExpandDims The input of the operator axis Combined with input input_x Of shape Calculation After dimension expansion Of new_shape, And this new_shape Namely OnnxReshape The input required by the operator shape.

5 obtain Node inputs and attributes

obtain MindSporeExpandDims Second order of operator 1 Inputs input_x and The first 2 Inputs axis：

6 MindSpore Input axis It maps to Onnx Input shape

obtain MindSpore The first 1 Inputs input_x Of shape value x_shape：

adopt MindSpore The input of axis and x_shape Derive the correct Onnx Input new_shape, The specific algorithm is not analyzed , If you are interested, you can check it yourself ：

7 Add a value node

Numerical value new_shape Register as a numeric node shape：

stay node_map_ptr Registered in shape Constant Numerical node And get it name（name_shape）：

stay ONNX chart graph_proto Create a new constant value node in node_proto, And specify that its output is a constant value node name（name_shape）：

by node_proto Add a value Numerical properties , And the value node shape The value in is converted to onnx：：tensor Assigned to value：

8 Onnx Figure adding operator nodes

stay node_map_ptr Register the current MindSporeExpandDims Operator nodes And get it name：

stay ONNX chart graph_proto New China ONNXReshape Operator nodes ：

Appoint ONNXReshape The output of the operator node is MindSporeExpandDims The output of the operator node （name） namely node_name：

Appoint ONNXPad The th... Of the operator node 1 The first input is MindSporeExpandDims Second order of operator 1 Inputs input_x Of name namely input_x：

Appoint ONNXPad The th... Of the operator node 2 The first input is ④ Created in the shape Of numeric nodes name namely name_shape：

This completes the mapping

（3） One to many ：MSBatchMatMul -> ONNX Transpose + MatMul

1 Mapping analysis

MindSpore Of BatchMatMul operator Yes transpose_a,transpose_b attribute , Controls whether input is transposed ,ONNX Of MatMul operator No transpose attribute , therefore , You need to determine BatchMatMul Of transpose_a,transpose_b Whether the property is true, If true, You need to in MatMul Add... Before corresponding input Transpose Operator to convert . So we can define what needs to be done ：

• obtain MindSporeBatchMatMul The properties of the operator transpose_a and transpose_b

• obtain MindSporeBatchMatMul The input of the operator input_x and input_y

• if transpose_a by true, You need to add a OnnxTranspose The node will enter input_x To transpose

• if transpose_b by true, You need to add a OnnxTranspose The node will enter input_y To transpose

9 Get node input and attributes

obtain MindSpore BatchMatMul Second order of operator 1 Inputs input_x And the 2 Inputs input_y：

obtain MindSporeBatchMatMul Second order of operator 1 Attributes transpose_a And the 2 Attributes transpose_b：

10 if transpose_a by True It's in input_x Add... After Transpose Operator nodes

Get input input_x Of shape：

stay node_map_ptr Registered in One transpose_input_x_name Constant Numerical node To store the transposed input_x The number ：

stay ONNX chart graph_proto New China ONNXTranspose Operator nodes transpose_inputx_node_proto, And specify its input as MindSporeBatchMatMul The th... Of the operator node 1 Inputs input_x, The output is the constant node just created transpose_input_x_name：

by transpose_inputx_node_proto Add a perm Numerical properties , and basis input_x Of shape For the assignment ：

thus Yes input_x Transposed OnnxTranspose Operator node addition completed .

11 if transpose_b by True It's in input_y Add... After Transpose Operator nodes

Get input input_y Of shape：

stay node_map_ptr Register one in transpose_input_y_name Constant Numerical node To store the transposed input_y The number ：

stay ONNX chart graph_proto New China ONNXTranspose Operator nodes transpose_inputy_node_proto, And specify its input as MindSporeBatchMatMul The th... Of the operator node 1 Inputs input_y, The output is the constant node just created transpose_input_y_name：

by transpose_inputy_node_proto Add a perm Numerical properties , And based on input_y Of shape For the assignment ：

So far, yes input_y Transposed OnnxTranspose Operator node addition completed .

12 Onnx chart Add operator node

stay node_map_ptr Register the current MindSpore BatchMatMul operator node And get it name（node_name）：

stay ONNX chart graph_proto New China ONNX MatMul Operator nodes ：

Appoint ONNX Reshape The output of the operator node is MindSpore BatchMatMul The output of the operator node （name） namely node_name：

I don't know what the following line of code means , If any students know about it, please contact me to revise it ：

if transpose_a by True, shows input_x Need to transpose , The specified ONNX MatMul The th... Of the operator node 1 A constant value node whose input is transposed transpose_input_x_name; if False, shows input_x There is no need to transpose , The specified ONNX MatMul The th... Of the operator node 1 The inputs are original input_x：

if transpose_b by True, shows input_y Need to transpose , The specified ONNX MatMul The th... Of the operator node 1 A constant value node whose input is transposed transpose_input_y_name： if False, shows input_y There is no need to transpose , The specified ONNX MatMul The th... Of the operator node 1 The inputs are original input_y：

This completes the mapping .

1、 compile export

1. MindSpore compile

stay MindSpore Enter the following command under path to start compilation ：

sh build.sh -e gpu -j32

First compilation requires downloading many third parties So Very slowly , Then compile again soon .

If an error is reported during compilation, modify it according to the error , If the compilation is successful, the following interface will be displayed ：

3. MindSpore package heavy install or Python function Path settings （ recommend ）

（1） MindSpore Package reinstallation

Compiled MindSpore The package will be generated in mindspore/build/package/ Under the path ：

Can be in mindspore Enter the following command under the path to reinstall mindspore package ：

pip uninstall build/package/xxx.whl

pip install build/package/xxx.whl

（2） Python Run path settings

Enter the following command to set Python Running path ：

export PYTHONPATH=/disk1/user14/wzb/test/mindspore/build/package:$PYTHONPATH

Replace the path with one compiled by yourself mindspore The absolute path of storage , In this way You can avoid repeat Uninstall and then install mindspore The trouble of the bag , It only needs Set up once After that, you can directly call Every time compiled new mindspore It's packed .

4. Onnx Model export

Update complete MindSpore After package , stay Models Under your own model path in the folder call README In the document export command , Set export format --file_format by ONNX, If the export is successful, the... Will be generated under the model path xxx.onnx file ：

If an error is reported, it shall be reported according to Misinformation Modify the operator mapping file onnx_exporter.cc And repeat （1）（2）（3） step , Until you can export ONNX Until the document .

2、 Reasoning test

1. Dataset Download

（1） basis ModelsREADME download

see Models in README Description of the dataset in the file , Download the data set through the hyperlinks provided and put it in the corresponding path ：

（2） According to the source code of the original paper of the model README download

If you find Models Of README Hyperlinks given in the file The data set in is too large 、 Downloading is slow or saving on Google cloud disk requires climbing over the wall , Then you can refer to the source code link given in the original paper of the model , View the source code README The data download address provided .

Such as my CNN+CTC Model Models The hyperlinks given in are stored in Google cloud disk , And put the training set 、 Validation sets and test sets are packaged together , Results in data sets up to 18G, You need to climb over the wall again , Direct downloads are often interrupted ：

After many unsuccessful attempts , I looked up the code of the original paper and found its readme The data set download link given in the file ：

After clicking on it, I found the existing in the original author's data set Models Set package given in , There is also a separate training set 、 Validation set and test set packages , Since we only deal with reasoning, we only need to download the test set Just compress the package , It's only the size of 160M, The download went very smoothly ：

2. MindSpore Model reasoning

（1） MindSpore Reasoning

This step only needs to refer to yourself Models In the model README The document is about MindSpore Model eval It's just a description of , Just note that some model scripts may be MindSpore The update of results in some operators The usage of has changed , If an error is reported, you can look at it and revise it yourself ：

3. Onnx Model reasoning

（1） Onnx Inference file

Onnx Inference file is to use onnxruntime The interface imports the... Exported in Chapter 2 onnx file , Then pass in the input that meets the network requirements , Final output Onnx Model accuracy of off-line reasoning .

This step is actually very simple , There are mainly two steps , The complete inference file can Refer to this PR Medium eval.py and infer_cnnctc_onnx.py：https://gitee.com/mindspore/models/pulls/2913/files

1 Data acquisition dataloader

Use Models Model eval Datasets in scripts dataloader get data , And then take it. from MindSpore::Tesnor Turn into Onnx General input numpy Format ：

13 Network replacement ：MindSpore Net -> Onnx

take Models Model eval The definition of the network in the script is replaced by Onnx in onnxruntime Imported network ：

MindSpore：

Onnx：

（2） Onnx Reasoning scripts

The reasoning script is to imitate eval Write a script that can use bash Called .sh file , Make it possible for users to access bash Command one click reasoning , Please refer to this for details PR Medium run_infer_onnx.sh：

https://gitee.com/mindspore/models/pulls/2913/files

5. PR Submit

（1） Leader Create a branch

Contact group Leaderfork official MindSpore and Models Warehouse and create branches for each model .

（2） Download new MindSpore and Models Warehouse

Download group to submit PR The new MindSpore and Models Warehouse ：

git clone https://gitee.com/xxxx/mindspore.git

git clone https://gitee.com/xxxx/models.git

（3） Update replacement file

Get into MindSpore and Models Switch to your own branch after the path ：

git checkputcnnctc

Transfer the files you need to modify to the newly downloaded MindSpore and Models Replace the folder under the corresponding path .

（4） Upload code

Use the following command to submit pr：

1 View modified files ：

git status

2 Submit to the local repository ：

basis ① Modified file exported in , Check whether they are all self modified , If yes, direct git add . If not, then one by one git add xxxx file .

MindSPore Warehouse ：

git add .

git commit -am "ONNX converter: CNN+CTC support"

Save and exit ： Click on ESC, Then press and hold shift and ”:” , Input wq, enter

Models Warehouse ：

git add .

git commit -am "ONNX infer: CNN+CTC support"

Save and exit ： Click on ESC, Then press and hold shift and ”:” , Input wq, enter

3 Submit to remote Warehouse Branch ：

git push origin cnnctc

6. establish PR

contact Leader establish PR, Or go to the warehouse where the code is submitted to create PR：

7. Access control test

1 Turn on the test

In myself PR Of Comment area Input /retest You can start the test ：

Click... In the system automatic comment Link You can open the access control interface by hyperlink ：

2 test result

The access control test will be interrupted after an error is reported , stay Comment area Find the error information , Click on newspaper You can view the hyperlink in the error comment Access control report Misinformation ：

One Segment code Mistakes can be made in two Different format output of the same code , The following is the code you are currently writing , And the above paragraph is the modification opinion of the access control system , Just follow the above paragraph （ The words are a little small. Zoom in ）.

If yes, the system will automatically reply as follows ：