mysql load data infile

1 Introduce
    LOAD DATA INFILE  Statement reads rows from a text file to a table at a high speed . It is SELECT ... INTO OUTFILE A supplement to ,SELECT ... INTO OUTFILE Is to write the data in the table to a file ,LOAD DATA INFILE Import the contents of the file into the table . Use both FIELDS And LINES sentence , Two statements are optional , At the same time FIELDS To be in LINES front .
    We can also pass mysqlimport Tools to import data , It is essentially sending LOAD DATA INFILE Statement to the server .
2 grammar

    LOAD DATA The grammar is as follows （mysql5.6 And above ）：

[sql]  view plain   copy

1. LOAD DATA [LOW_PRIORITY | CONCURRENT] [LOCAL] INFILE 'file_name'  
2.     [REPLACE | IGNORE]  
3.     INTO TABLE tbl_name  
4.     [PARTITION (partition_name,...)]  
5.     [CHARACTER SET charset_name]  
6.     [{FIELDS | COLUMNS}  
7.         [TERMINATED BY 'string']  
8.         [[OPTIONALLY] ENCLOSED BY 'char']  
9.         [ESCAPED BY 'char']  
10.     ]  
11.     [LINES  
12.         [STARTING BY 'string']  
13.         [TERMINATED BY 'string']  
14.     ]  
15.     [IGNORE number {LINES | ROWS}]  
16.     [(col_name_or_user_var,...)]  
17.     [SET col_name = expr,...]  

3 Presentation data preparation

    3.1 establish test database And table

[sql]  view plain   copy

1. mysql> create database loaddata;  
2. mysql> use loaddata;  
3. mysql> create table loadtest (c1 int(10), c2 varchar(20), c3 varchar(20), c4 varchar(20));  

    3.2 Insert test data

[sql]  view plain   copy

1. mysql> insert into loadtest values (100, 'column2', 'column3', 'column4');  
2. mysql> insert into loadtest values (200, 'line2', 'line3', 'line4');  

    3.3 Export data

[sql]  view plain   copy

1. [[email protected] /]# mysqldump --tab=/tmp/ loaddata --fields-terminated-by=',' --fields-optionally-enclosed-by='"' --fields-escaped-by='#' --lines-terminated-by='\n'  

    View the exported data file

[sql]  view plain   copy

1. [[email protected] /]# cat /tmp/loadtest.txt   
2. 100,"column2","column3","column4"  
3. 200,"line2","line3","line4"  

4 Grammar explanation and example analysis

    file_name： file name file_name A string is required , stay windows Next , The path can be a slash '/' Or double backslash '\\'（windows Verify below ）.
    PARTITION：MySQL 5.6.2 And subsequent versions ,LOAD DATA INFILE Start supporting PARTITION Partition options . For partitioned tables, a storage engine using table locks should be used , for example myisam, Not applicable to the storage engine with row lock , Such as Innodb, because LOAD DATA You cannot remove any partition locks .
    CHARACTER SET： from character_set_database The character set indicated by the system variable is used to interpret the information in the file .SET NAMES and character_set_client The setting of does not affect the interpretation of the input . If the character encoding in the file is inconsistent with the default , It is best to CHARACTER SET Statement specifies the encoding method , This requires us to go through mysqldump or SELECT ... INTO OUTFILE When exporting a file, use the parameter --default-character-set Specify the encoding format of the exported file .
    Be careful ： Importing files does not support ucs2, utf16, utf16le, or utf32 Coding format .
    LOW_PRIORITY： This parameter is applicable to the table lock storage engine , such as MyISAM, MEMORY, and MERGE, If a client program reads a table during writing , Writing will be delayed , Continue writing until no client program reads the table .
    CONCURRENT： Use this parameter , Allow other client programs to read table contents during writing .
    The impact on replication ： Use statement based formatting （statement-based） Copy , stay mysql5.5.1 Before ,CONCURRENT Parameters cannot be copied . Use row based formatting （row-based） Copy , stay mysql5.1 And subsequent versions , Can be copied correctly .mysql5.6 Think of using LOAD DATA INFILE It's not safe , When using statement-based A warning will be issued when the log is formatted , Use mixed Format log will be stored as row Format .
    LOCAL： This keyword affects the expected read location and error handle of the file . This option should be enabled on both the client and the server , If mysqld The startup parameters for include local-infile=0, Then specifying this parameter is invalid .
    Impact on file lookup ：
    Specify this option ： The file will be searched by the client program on the client host , Then send it to the server . You can specify the absolute path and relative path of the file , If you specify a relative path, you will find it in the client program directory .
    When importing, the file will be copied to the temporary folder of the server first （linux Next is /tmp, windows Next is C:\WINDOWS\TEMP）, If the temporary folder space is insufficient, it will cause LOAD DATA Failure .
    The option is not specified ： The file will be searched by the server program on the server host , The search rules are as follows ：
    If an absolute path name is given , Then the server uses this path name .
    If a relative path name with one or more boot components is given , Then the server will search relative to the server （ Data directory ） The file of .
    If you give a file name without a boot component , The server will default to （ database ） Of （ Database directory ） Looking for files in .
    for example ：./myfile.txt Will be in the database data Directory lookup ,myfile.txt The default database that will be used when we connect , Such as test The data directory of .
    LOCAL Impact on error handles ：
    Use LOAD DATA INFILE, Data interpretation and primary key conflict error Will terminate operation .
    Use LOAD DATA LOCAL INFILE Data interpretation and primary key conflict error Will become warning, Operation continues , because mysql server There is no way to end the transfer of data during the operation . For primary key conflicts error Speaking of , This method is similar to specifying IGNORE.
    REPLACE And IGNORE： Control the repetition of the entered row and the unique primary key .
    REPLACE： The input line replaces the existing line . That is, the repetition with the primary key or unique index column .
    IGNORE： The input row duplicates the existing row primary key or unique index column , Then discard .
    If neither is specified ： The behavior will depend on whether or not it is specified LOCAL Parameters .

    Ignored during data import （ Foreign key constraints ） Can be in LOAD DATA Pre execution SET foreign_key_checks = 0 .

    LINES STARTING BY 'prefix_string'： If all the rows you want to read in contain a common prefix that we want to ignore , You can use 'prefix_string' To skip the prefix （ And the prefix
All characters before ）. If a row does not include a prefix , The entire line is skipped
    for example ：

[sql]  view plain   copy

1. mysql> LOAD DATA INFILE '/tmp/test.txt' INTO TABLE test LINES STARTING BY "xxx";  

    If /tmp/test.txt The contents of the document are as follows

[sql]  view plain   copy

1. xxx"abc",1  
2. something xxx"def",2  
3. "ghi",3  

    Then the contents we read include ("abc",1) and ("def",2), The third line is skipped directly .
    IGNORE number LINES： This option can be used to ignore lines at the beginning of the file . for example , We can use IGNORE 1 LINES To skip a starting header row containing column names .

[sql]  view plain   copy

1. LOAD DATA INFILE '/tmp/test.txt' INTO TABLE test IGNORE 1 LINES;  

    FIELDS and LINES：
    about LOAD DATA INFILE and SELECT ... INTO OUTFILE,FIELDS and LINES The syntax of each statement is the same , Both are optional , But at the same time FIELDS The sentence should be put in LINES In front of the statement .
    If we specify FIELDS sentence , Its clause (TERMINATED BY, [OPTIONALLY] ENCLOSED BY, and ESCAPED BY) Also optional , However, at least one must be specified .
    If FIELDS and LINES None of us specified , The default syntax is ：

[sql]  view plain   copy

1. FIELDS TERMINATED BY '\t' ENCLOSED BY '' ESCAPED BY '\\'   
2. LINES TERMINATED BY '\n' STARTING BY ''  

    all field- or line-handling Options can specify an empty string (''). If the string is not empty , be FIELDS [OPTIONALLY] ENCLOSED BY and FIELDS ESCAPED BY The value must be a single character .FIELDS TERMINATED BY, LINES STARTING BY and LINES TERMINATED BY The value can exceed one character . for example , To write by carriage return / A line that ends with a pair of characters , Or read
Files containing such lines , Then one should be specified LINES TERMINATED BY ‘\r\n’ Clause .
    FIELDS [OPTIONALLY] ENCLOSED BY ： about SELECT ... INTO OUTFILE Output , If not OPTIONALLY Options , All fields will be ENCLOSED BY Specified character wrap , for example ：

[sql]  view plain   copy

1. "1","a string","100.20"  
2. "2","a string containing a , comma","102.20"  
3. "3","a string containing a \" quote","102.20"  
4. "4","a string containing a \", quote and comma","102.20"  

    If we specify OPTIONALLY, Only string data type （ Such as CHAR, BINARY, TEXT, or ENUM） The field of will be ENCLOSED BY Specified character wrap , for example ：

[sql]  view plain   copy

1. 1,"a string",100.20  
2. 2,"a string containing a , comma",102.20  
3. 3,"a string containing a \" quote",102.20  
4. 4,"a string containing a \", quote and comma",102.20  

    Be careful , If... Appears in the field value ENCLOSED BY character , By using ESCAPED BY Character as prefix , Yes ENCLOSED BY Character escape . in addition , it is to be noted that , If an empty... Is specified ESCAPED BY value , It may be generated that cannot be LOAD DATAINFILE Output value read correctly . for example ：

[sql]  view plain   copy

1. 1,"a string",100.20  
2. 2,"a string containing a , comma",102.20  
3. 3,"a string containing a " quote",102.20  
4. 4,"a string containing a ", quote and comma",102.20  

    FIELDS ESCAPED BY： Used to control how to read and write special characters , As an example above , Specify... When exporting and importing FIELDS ESCAPED BY Is double quotation mark ["] Can be correctly imported , The export format is as follows , Double quotation marks inside the field ["] It's escaped .

[sql]  view plain   copy

1. 1,"a string",100.20  
2. 2,"a string containing a , comma",102.20  
3. 3,"a string containing a #" quote",102.20  
4. 4,"a string containing a #", quote and comma",102.20  

    For input ：
    If FIELDS ESCAPED BY The specified character is not empty , Then the character is removed during input , Subsequent content is added to the field . Some two character string sequences where the first character is an escape character '\' It's an exception , Such as \0,\b,\n,\r,\t,\Z\,\N etc. , These character sequences themselves have a special meaning .
    If FIELDS ESCAPED BY The specified character is empty , There will be no interpretation of escape sequences .
    For output ：
    If FIELDS ESCAPED BY The specified character is not empty , Characters are prefixed to the following output .
    【********** Here to be improved ***********】
    In certain circumstances ,field- and line-handling Options interact ：
    If LINES TERMINATED BY Is an empty string ,FIELDS TERMINATED BY Non empty , Line to FIELDS TERMINATED BY End with the specified string .
    If FIELDS TERMINATED BY And FIELDS ENCLOSED BY Values are all empty (''), Fixed rows will be used （ No segmentation ） Format . Use fixed line format , There will be no delimiters between fields （ Line terminators can still be used ）, The column field data is read and written according to the width defined by the field , Such as TINYINT, SMALLINT, MEDIUMINT, INT, and BIGINT, The field widths are 4, 6, 8, 11, and 20.
    Unsuitable for use LOAD DATA INFILE The situation of
    Use fixed line format （ namely FIELDS TERMINATED BY and FIELDS ENCLOSED BY All empty. ） The column field type is BLOB or TEXT.
    The specified delimiter is the same as other option prefixes ,LOAD DATA INFILE Unable to interpret input correctly . for example ：

[sql]  view plain   copy

1. FIELDS TERMINATED BY '"' ENCLOSED BY '"'  

    If FIELDS ESCAPED BY It's empty , The field value contains FIELDS ENCLOSED BY Specify the characters , perhaps LINES TERMINATED BY The characters in FIELDS TERMINATED BY Before , Will lead to premature stop LOAD DATA INFILE operation . because LOAD DATA INFILE Cannot accurately determine the end of a row or column .

    Select the imported Columns ：

    The following statement will import all the columns of the file

LOAD DATA INFILE 'persondata.txt' INTO TABLE persondata;
    If we want to import some columns of the table , You need to specify a list of columns
LOAD DATA INFILE 'persondata.txt' INTO TABLE persondata (col1,col2,...);
    Be careful ： If sql-mode For the strict mode of things STRICT_TRANS_TABLES, Errors will be reported when importing some columns .
    If the column order of the input file is different from that of the table , We must specify a list of columns , otherwise mysql You cannot match the fields of the input file with the columns of the table .

[sql]  view plain   copy

1. mysql> LOAD DATA INFILE '/tmp/loadtest.txt' INTO TABLE loadtest FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '"' ESCAPED BY '#' LINES TERMINATED BY '\n' (c1, c2, c4, c3);  

    The list of columns can contain column names or user variables , Before writing a column, we need to use SET Statement to convert user variables . Yes set Statements and user variables can be used as follows ：
    Method 1： Before the user variable is used for the first column , First assign the value of the first column to the user variable , After the division operation, input to c1.

[sql]  view plain   copy

1. mysql> LOAD DATA INFILE '/tmp/loadtest.txt' INTO TABLE loadtest FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '"' ESCAPED BY '#' LINES TERMINATED BY '\n' (@var1, c2, c3, c4) set c1 = @var1/2;  

    Method 2： hold c3 Set the column to the current time （sql-mode Use strict thing patterns STRICT_TRANS_TABLES Will report a mistake ）

[sql]  view plain   copy

1. mysql> LOAD DATA INFILE '/tmp/loadtest.txt' INTO TABLE loadtest FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '"' ESCAPED BY '#' LINES TERMINATED BY '\n' (c1, c2, c4) set c3 = CURRENT_DATE;  

    

    Method 3： Assign input to user variables , Instead of assigning user variables to columns in the table , To discard this input value .

[sql]  view plain   copy

1. mysql> LOAD DATA INFILE '/tmp/loadtest.txt' INTO TABLE loadtest FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '"' ESCAPED BY '#' LINES TERMINATED BY '\n' (@dummy, c2, @dummy, c4);  

    

    Import data through pipes ：

    stay unix In the system , If we want to go from the pipe （pipe） in load data, You need to use the following methods ：

[sql]  view plain   copy

1. mkfifo /tmp/ls.dat  
2. chmod 666 /tmp/ls.dat  
3. find / -ls > /tmp/ls.dat &  
4. [[email protected] tmp]# mysql -e "LOAD DATA INFILE '/tmp/ls.dat' INTO TABLE test.tb1 FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '\"' ESCAPED BY '#' LINES TERMINATED BY '\n'"  

    Be careful ：sql-mode Use strict thing patterns STRICT_TRANS_TABLES Will report a mistake
    Write data to the pipeline in another window

[sql]  view plain   copy

1. [[email protected] /]# cat /tmp/loadtest.txt > /tmp/ls.dat   

    notes ： You can read or write the pipeline first , Whoever comes first can , Before all the data written into the pipeline is read out , In a blocking state .
5 matters needing attention
    LOAD DATA INFILE You can read external files , Such as comma separated by other databases or programs CSV File format .
    When we use SELECT ... INTO OUTFILE Export data from database to file , And then through LOAD DATA INFILE Read the file to the database , The of these two statements field and lines Options must match , otherwise LOAD DATA INFILE Will not correctly interpret the contents of the file .
6 Common mistakes
    6.1 sql_mode by STRICT_TRANS_TABLES, That is, strict transaction mode , It's easy to make mistakes , for example ：

[sql]  view plain   copy

1. mysql> LOAD DATA INFILE '/tmp/loadtest.txt' INTO TABLE loadtest FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '"' ESCAPED BY '#' LINES TERMINATED BY '\n' (c1, c2, c4) set c3 = CURRENT_DATE;   
2. R 1262 (01000): Row 1 was truncated; it contained more data than there were input columns  

    see sql_mode

[sql]  view plain   copy

1. mysql> show variables like '%sql_mode%';  
2. +---------------+--------------------------------------------+  
3. | Variable_name | Value                                      |  
4. +---------------+--------------------------------------------+  
5. | sql_mode      | STRICT_TRANS_TABLES,NO_ENGINE_SUBSTITUTION |  
6. +---------------+--------------------------------------------+  

    Only when the strict transaction mode is cancelled can

[sql]  view plain   copy

1. set sql_mode='NO_ENGINE_SUBSTITUTION';  

    6.2 windows Some mistakes on

[sql]  view plain   copy

1. C:\Program Files\MySQL\MySQL Server 5.5\bin>mysqldump -uroot -pactionsky --tab=/tmp/ loaddata --fields-terminated-by=',' --fields-optionally-enclosed-by='"' --fields-escaped-by='#' --lines-terminated-by='\r\n'  
2. mysqldump: Got error: 1083: Field separator argument is not what is expected; check the manual when executing 'SELECT INTO OUTFILE'  

    mysql5.5.30