Tencent cloud database tdsql elite challenge Q & A (real-time update)

Ladies and gentlemen ：

Hello everyone ！ In response to the questions mentioned by the contestants, the technical team gave a passionate answer , The contents are as follows . The current questions and answers have been very comprehensive , If you have any questions, please check the following carefully Q&A, And the document content will be updated to everyone in real time .

Consider the fairness of the competition and the significance of the competition , In the course of the competition, the topic analysis is involved 、 Specific details, etc , The technical team will not answer , Expect to understand . I wish you all the best in your competition , come on. ！

One 、 Rules of entry / Violations, ：

Q： Can Tencent cloud account compete without real name authentication ？ If you can compete , Can the voucher be reissued ？

A：

1） No real name certification , Submit the results of the contest and preliminary contest that cannot receive the contest voucher .

2） The real name certification can be carried out at any time within the deadline of the preliminary contest , It is recommended not to delay the real name authentication to a later stage .

3） Vouchers will not be reissued , In the early stage, the contestants have been reminded for many times to complete the real name authentication and receive the voucher as soon as possible within the specified time .

4） It is impossible to submit the results without real name authentication .

Q： Ahead of time int Within the scope of id The corresponding relationship with the partition is saved in the file , Read directly in the file when running the test id And the corresponding relation of the partition , Is that illegal ？

A： count , The running test program should complete all functions , It is unfair to others to calculate in advance , So it's illegal ; It can be calculated during the test run , Then import .

Q： Ask a question about the use of a tool . I've been thinking about how to improve performance , notice tdsql load_data Introduction to tools , Although there are already post replies, you can't use this tool directly , I would like to ask if I can carry this tool in the compressed package , To use its mode0 For local processing of files ？

A： Don't allow .

Two 、 test / evaluating ：

Auxiliary understanding of logs

【 important 】 Supplementary description of evaluation rules （ The event announcement has been issued , Game description has been configured ）：

Contestants will source code and its dependent code base （ It is not allowed to submit compiled binaries and libraries , Such as .jar file ,.a Files and ELF Executable documents, etc ） Compressed into zip The package is submitted to the evaluation system .

1. The code written by the contestants themselves , The source code must be submitted .

2. If the uploaded code refers to a third-party library ( for example java Of jar The library files , perhaps C++ Of .a file ), You need to package the downloaded third-party library files into the submitted code . If the player changes or recompiles the third-party library , You need to provide the source code .

The above-mentioned third-party libraries refer to open source , And can be downloaded 、 Traceability 、 Verifiable Libraries .

3. Uploading code is not allowed to contain ELF Executable file .

attach ： Sample file compression

1） Select file compression

2） After the compression

3） Rename tdsql.zip

Q：kill failure,maybe run program not exist Under what circumstances will it happen ？

Use C++, The executable has been named run 了 , Check the log to see that the migration program has also started , But there was kill failure The situation of , So I want to ask kill failure Under what circumstances will it happen ？

A：kill failure explain run The program is not running .

Q： Whether the log has a size limit ？ Even programs larger than this size will be kill fall ？

A：log Limit to 100MB, Beyond this size you will not see this log , Only less than 100MB Log , The program will not be kill fall .

Q： There is no... On the evaluation platform /mnt/datadisk0 This data disk , To use ssd Disk as a temporary file directory where to use ？

A： There is no difference between the two disks , The current path is OK , Space is enough .

Q： How long does the evaluation timeout ？

A： The evaluation time is uniformly limited to 100 Within minutes, , If it exceeds the time limit, it will be judged as failure , The reason for the failure can be found in the log .

Q： How to set the relative path of test data ？

A： If you test by yourself, you can set the path by yourself , If you submit the sponsor evaluation, you need to obtain the data source path , Refer to the example demo.

Q： Is the directory structure of the actual test data completely consistent with that of the given test data ？

A： The directory structure is similar to , For details, please refer to the description of the competition .

Q： The code running environment supports pip Well ？

A： I won't support it .

Q： Are multiple sources local environments that are evaluated online ？ Is it possible to come from multiple external servers ？

A： The first round is a data source .

Q：make.sh Is the system environment allowed to be configured ？ It can be used apt Pretend to depend on ? For example, without access to the network , modify /etc/sysctl.conf Kernel parameters in optimize memory management , Or use mkdir create a file , Use mkswap establish swap Virtual memory, etc （ need CVM Mechanical root jurisdiction ）

A： The contestant code runs in non root Under the user , So there was no root jurisdiction .

Q： Connect to database , It's the players who find three sides jar package , Or is it that the official has provided corresponding sdk ah ？

A： Just find it yourself , In addition, please refer to the supplementary notes on the evaluation rules in the competition questions in detail .

Q： Will the program running log not be provided to us for relevant analysis during the test ？

A： After submitting the sponsor evaluation system for evaluation , success / If it fails, there will be a download log .

Q： It can be used on the evaluation machine maven Compile ？

A： It is not provided by default maven Compiler tools , Please refer to the supplementary description of the evaluation rule for details .

Q： java compile , What are the tools on the tester , Where is the path ？

A：java-1.8.0-openjdk,java-1.8.0-openjdk-devel, The path is the default path .

Q： What is the configuration of the tester server ？ Memory 、cpu What is the configuration ？

A：4c/8g Mirror image centos7.5, The recommendations are consistent with the evaluation environment .

Q： The dataset does not answer？ So how to check whether your answer is right ？ Whether there is correctness self-test code provided ？

A： Test data sets and examples of self built test environments have been provided for contestants , Players can test themselves , You can also submit your answers to the official evaluation system of the competition , The evaluation is successful / Failure has log feedback . Be careful ： Each team can submit 2 Evaluation times , You need to go to PC End “ My grades ” Page view （ In login status, you can view ）.

Q：Key 'id' doesn't exist in table '4' surface 4 There must be id This index ？ The title is given to sql Not in it .

A： There are differences between the test set and the actual data set .

Q： Must the table structure of the actual test be the same as that in the test data ？ It's still possible to have different needs according to .sql Judge for yourself ？

A： There are differences between the test set and the actual data set , The number of tables is the same , The table structure is different .

Q： On the test machine , Yes Python The basic library of ？ such as sys,os This is the most basic library , Is this kind of library also packaged ？

A： yes , we have .

Q： Please tell me about the evaluation environment TDSQL What is the specification ？

A： Please look at the questions carefully , The competition questions have been explained in detail . The target example is TDSQL Distributed instances , Version is 8.0,2 A shard , Individual specifications are 2C4G example , Capacity of 100G.

Q： Please tell me what is used for evaluation tdsql Is it strongly synchronous or asynchronous ？

A： Strong synchronization .

Q： The test data are from 1 Start , Gradually increase ？

A： yes . Such problems are not directly helpful to the solution .

Q： How to view the log of player code addition ？

A： Players need to output the log as standard , The evaluation program will redirect the player log to make.log and start.log.

---

3、 ... and 、CVM/ example

CVM Voucher usage guide ：https://cloud.tencent.com/developer/article/1922573

Q： Is there a CVM Provide , When to provide ？

A： Provided by the sponsor CVM The voucher will be on 12.22-27 Issued between days , At the same time, it will provide purchase instructions .

Q-1： What did you evaluate cvm Which model is it ？

Q-2： You can provide the same server environment as the test environment for development ？

A： above Q-1-2 The questions are answered uniformly -- Code runs CVM The machine specification is 4C8G, It is in the same zone as the target database instance , The system disk capacity is 100G, The capacity of the data disk is 100G.

Q： When evaluating cvm The system image of id Whether it can be made public ？ What software and dynamic libraries are preinstalled in the system ？

A： Sponsor's evaluation cvm The system image used is centos7.5, The system is pre loaded with the basic language packs supported in the competition questions , about C/C++ Pre installed mysql-devel,gcc4.8.5, about python Pre installed MySQL-python/python-devel/PyMySQL.

Q： The machine used for the test goes to mysql How much bandwidth does the machine have ？

A： Enough bandwidth .

---

Four 、 Merge

Q-1： The meaning of the contest question is to merge tables from different sources with the same database name and table name , Or do you want to merge all the tables with the same name from different sources and libraries ？

Q-2： The actual test data are from two sources Or there may be multiple sources Three or more ？

Q-3： The primary key of each table of the test data is different , What about the primary key after consolidation ？

Q-4： Is to merge all local tables into one table and write TDSQL in ？

Q-5： At present, the test data given are src_a and src_b Two data sources , Each source has a-g Seven database, Every database has 1-4 Four tables , According to our understanding , The data of all tables with the same name should be , No matter from which data source database, Are aggregated and merged into the table with the same name in the target database as required , Whether it is right ？ Still say src_a And src_b Which are aggregated into different target databases ？

Q-6： The merger , Is multithreading involved ？

Q-7： Is to merge src_a and src_b Is the database table data of to the destination ？

A： above Q-1-7 The questions are answered uniformly -- What's merged is src_a Corresponding tables and src_b The corresponding table , Is one of the two sources , Merge the same name library with the same name table . such as src_a in a Table of the library 1 and src_b in a Table of the library 1 A merger ,src_a in a Table of the library 2 and src_b in a Table of the library 2 A merger . Please refer to the description of the competition questions for specific merger rules .

Q：src_a/src_b also a-g Is constant ？ Or is it just an example , You need to traverse all the files during the evaluation ？

A： Sample descriptions are provided in the title and test data , The preliminaries are just about two in one , There must be only two Source Source , The number of databases under each source should not exceed 7 individual , The tables under each database are variable .

Q： Is this the goal of the topic ？

A： Yes , Is the combination of intuitive meaning understanding .

Q： Input to ensure that the same library table structure in different source instances is the same ？

A： Ensure that the , The structure of the same library table is consistent .

---

5、 ... and 、 Primary key ：

Q： When there is no primary key , Do you want to update updated_at The field also depends on updated_at Newer or older ？

A： Yes .

Q： Do you really have to have a primary key ？ Test environment last night sql Unprocessed tables created directly , Then each table has been checked in batches , But nothing unusual .

A： Please refer to the description in the title , It is not recommended to make subjective assumptions if the topic is not explained .

Q： When there is no primary key ,“ Update only updated_at Field ” What does that mean? ？ Keep it up to date ？ Directly covered ？

A： Keep it up to date .

Q： Is the data in the data file out of order , For example, two pieces of data with the same primary key value , Their sequence in the document , and updata_at Is the value of the field related , Is it the second data update_at It's going to be bigger ？

A： The latter is not necessarily better than the former update_at Big , If there is no clear explanation in the title , No such assumption can be made .

Q： In the real world sql_require_primary_key What is the value of the variable ？ Because there are tables without primary keys .

A： The actual environment is OFF, The self-test environment needs to be manually modified .

Q： Currently, there is no primary key table , and tdsql Table must have primary key , Insufficient permission to modify parameters , Cannot create a table without a primary key .

A： Self test instances can be managed in the cloud platform database -> Self modifying in parameter setting , The sponsor's evaluation environment has been handled .

---

6、 ... and 、 The Internet ：

Q： What is the network delay , I test locally , Delay 300 millisecond , Very bad performance , Concurrent 28 Tables ,10 I wrote in minutes 28w, There's something wrong with it .

A： Network problems need not be considered . It is recommended to use... In the same zone CVM To test , The network latency is very low , Local tests are not informative .

Q： The error log prompt should be in VPC Network access , How to access this ？

" To ensure data security , At present, only the intranet address is provided , The address is valid for 15 minute , After expiration, please refresh the page to get it again ,VPC The network address can be found at VPC Network access " How can I access this log ？ in addition TDSQL Your server can ssh Log in ？

A： You can view the operation log added in your code , Don't look at this .

---

7、 ... and 、 other

Q： Is the preliminary contest to develop a complete data migration ？ Or some partial statements ?

A： A complete migration program is required .

Q： Direct use jdbc Operate the database ？mybatis Would you? ？ Here you are sql Interface documentation for ？ It uses Tencent sdk To operate the database ？

A： Please refer to the supplementary description of the evaluation rule for details , Just follow the official interface .

Q： Excuse me Is the table name case sensitive ？

A： Insensitivity

Q： Can you send the official interface document ？ It can be used maven Go to the management pack

A： You can find it yourself ,maven No .

Q： What could be the reason ？ see start.log It's all normal .evaluate.log Can you add time to the log ？

A： The machine environment is tested , The log is clear , Try to solve .

Q：evaluate The process is to take set1 and set2 Check all the data on the, right , Even if the data of a table is only in one of them set It doesn't matter if you go to ？

A： I feel that my understanding is a little biased , Validation only verifies the final result set .

Q： How to solve the problem of precision loss ？10.135204300235847 This number is written to mysql It becomes 10.1352

A： This is a problem that players need to consider and solve .

Q： The example provides table creation sql It seems that some of the documents are not specified primary key, If we build the table according to it, we will report an error . If you add a primary key by yourself, it is inconsistent with the original data , Please give the official solution . A： The target side can create tables , We need to find a way .

Q： What does each column of the log mean ？

start check data consistent

checkSum:Inconsistent data is: e 1 0 172.16.0.105 InnoDB 6652449 21b33d33 19 0 NULL NULL

checkSum:Inconsistent data is: e 1 0 172.16.0.161 InnoDB 6652496 1b8d43f2 6 0 NULL NULL

checkSum:Inconsistent data is: e 2 0 172.16.0.105 InnoDB 6657624 2771d986 19 0 NULL NULL

checkSum:Inconsistent data is: e 2 0 172.16.0.161 InnoDB 6657624 6df50d97 5 0 NULL NULL

A: This is the log of the evaluation system , If you don't understand it, you can ignore it .

Q： What is the given destination SQL The account has access information_schema Of SELECT Jurisdiction? ？

A： The account permissions of the target instance are clearly explained in the title , Please examine the topic carefully .

Q：Go mod build abnormal ？ go: github.com/go-sql-driver/[email protected]: reading github.com/go-sql-driver/mysql/go.mod at revision v1.5.0: unknown revision v1.6.0

Evaluation environment git Version, and go env Can configuration parameters be provided , Because this exception has been wasted 4 This evaluation opportunity ！！！

A： Access to the Internet is not provided, and the use of git,go Support only 1.15 edition ,go env The configuration parameters are the default configuration .

Q： System disk and data disk , How to distinguish ？ It seems that the mounting path is not given ？

A： Data disk mounting path /mnt/datadisk0.

Q：tdsql-mysql Should be compatible mysql Well , That is to say, I used common in early debugging mysql+（ such as cpp Next mysql-devel） It's OK, too ？

A： Sure .

Q： Can I use Rust Yeah ？

A： You can't . Q： use java Development ,jdbc The driver is to use tdsql Dedicated or mysql Self contained ? A：mysql Just bring your own ,8.0 above .

Q： If you use mysql drive , Which version is used ？

A：java Words 8.0 above .

Q： If the mysql v8 drive ,jdbc The target database name in the connection string must be specified , In the current title start.sh The parameter is not in the parameter , How to specify ？ Or is the test environment database name fixed , If fixed , What is it? ？

A： You can specify MySQL The default library that comes with , There is no need to be in start.sh Add , You can add... To your program .

Q： In target database , Whether the corresponding table has been established , Or do you need to use it by yourself .sql The statements in the file go to the target database to create a table ？

A： You need to create your own table .

Q：C++ Related compilation environment , May I introduce you to you ？ such as cmake edition 、make、GCC/clang Version, etc .

A： Based on the feedback from the contestants, we need support cmake, For a better experience , The technical mission has configured ,cmake 2.8.12 edition ,make The version is 3.82,gcc yes 4.8.5.

Q：CMake2.8.12 Too old , It's already 8 Years ago . Can you provide a higher version CMake？ In this way, many libraries cannot be compiled . for example Mysql Connector, Required CMake The minimum version is also 3.1 above

A： You can't , At present, only mysql-devel library , Please change your mind to another way .

Q： Will there be game problem analysis ？ This is to copy multi-source data to tdsql Inside , Then see who uses the shortest time ？

A： There will be no detailed explanation of the competition questions during the competition , The evaluation rules of the competition questions are mainly based on the issued explanation of the competition questions .

Q： Where source data files and structures are provided ?

A： The organizer has provided the test data set to the contestants , Log in to your account to download .

Q： java Only 1.8 Well ？ I think Intel About pmem The manual says v10 To support pmem, Isn't it useless java? This will use idea Programming ？（ I'm a little confused , see Intel Manuals programming persistent memory Only as heap Use needs 10 above , Maybe not heap 1.8 Just ok 了 ）

A：java Only 1.8,pmem Only for the finals .

Q：java Language restrictions ： The evaluation environment includes maven Do you ？ have access to maven To manage dependencies ？

A： It is not provided by default maven Compiler tools .

Q：mysql-devel What's the version of ?

A：mariadb-devel-5.5.68.

Q： When data is migrated to an instance , Is the database and table on the ECS built in advance ？

A：“ Cloud server ” You mean instance , Not built in advance .

Q： Is there any for SQL The format of the document is guaranteed ？ Is there a guarantee SQL There is only one... In the file CREATE TABLE sentence , And has a fixed format ？ In the sample, the format is similar to this , Will there be more than one final evaluation data set CREATE TABLE Or some reference,auto_increment,constraint Something like that ？

A：SQL In the document SQL The statement is correct , stay MySQL and TDSQL All instances can be executed . In the document SQL Only the statement of creating the current table , Nothing else , Refer to the test data set for specific format .

Q： Yes make Well ？ What about the test environment ？ What is a compiler ？ What is the version ？

A：make The version is 3.82, Please refer to the game description for the test environment .

Q： This competition allows us to use load in local file How to import ？

A： This setting of the target instance is off , Can not use load data.

Q： The structure provided is not tdsql The creation statement of , So there was no shardkey, I create tdsql Formulated at shardkey Is it consistent ？

A： The difference caused by the distributed database is not within the verification range .

Q：checksum The principle of is to check the table .ibd Documents ？

A： All the data on both sides are read out in segments to do CRC check , Ensure that the data and structure on both sides are completely consistent , Ignore the differences caused by distributed systems .

Q： What is wrong with data accuracy ？ A： Floating point numbers need to be written to the database with the highest possible precision ,MySQL The processing of floating point numbers in the database follows certain international standards IEEE-754, But reading on the implementation 、 Presentation and storage may also have some special features , The handling of floating point numbers is also an examination point of the topic , Floating point numbers participate in the verification in the form of strings .

Q： Why is it /mnt Cannot create... Under directory ？ A： The submitted code is in non root Under the user , Specify the path to run , Can't access other paths , To create a file directory , Create in the current path .

Q：TDSQL The competition needs to involve TDSQL Configuration changes ？ Or can I change the database variables that affect the insertion performance ？

A： Don't allow TDSQL Change of configuration parameters , All default configurations .