Tencent cloud database tdsql elite challenge --q & A

You are interested and have signed up for 【2021·Change SQL- Tencent cloud database TDSQL Elite challenge 】 My friends ： Hello everyone ！ At present, the competition registration is still in progress , For the questions that everyone cares about and raises , The organizer of the competition will Q&A In the form of , Update regularly to everyone .

If the following answers do not meet your needs , Or you have other concerns , Welcome to the 【 Tencent cloud + Community 】 cloud + Discussion area of the competition Leaving a message. , We will answer your questions as soon as possible , Thank you for your awesome support for the competition .

---

One 、 About signing up

Q： Can't find a teammate to form a team , Can I enter for the competition by myself ？

A： The organizer suggested that a team be formed to participate in the competition , It can give play to the advantages of group warfare , Of course, individual competition is also possible . Q： Will there be any pre competition guidance for the entry ？

A：11 month 18 Japan 、11 month 25 The Japanese organizers will carry out database technology training , Stay tuned 【 Tencent cloud + Community - Salon 】 or CSDN“ University Club ” Live broadcast on the platform .

Q： What basic skills do you need to learn ？

A： Basic programming skills , Anyone who knows about the database can participate in the competition , In the near future, you can pay attention to the knowledge points related to database migration .

Q、 When will the competition vouchers be issued ？

A、 During the period from the end of registration to the announcement of the preliminary questions .

---

Two 、 About the game

Q： What are the limitations of language ？

A： Language involves go 1.15、java 8 、c、 c++ 11 、python3.

Q： What is the main test of the competition ？ Are the preliminaries and finals different topics ？ What needs to be prepared in advance ？

A： This competition mainly focuses on the direction of database migration , Preliminaries 、 The final question is in the same direction , Of advanced nature , The inspection points of each stage are different . Before the competition, you can learn about database migration in advance .

Q： This time it is TDSQL PG edition , still TDSQL MySQL edition ?

A：TDSQL MySQL edition .

Q： If the distributed database implements the consistency protocol by itself instead of zk Will the performance be higher ？

A： The topic itself does not involve the design and implementation of distributed consistency protocols , The target side uses TDSQL Distributed database .

Q： If memory is used , So it's better to run on a separate server ？

A： The compiled program provided by the contestant runs on a separate server , It is separate from the machine where the database instance is located .

Q： How difficult is the problem ？ Is it suitable to attend the undergraduate course ？

A： The competition is open to all , Of course, people at different stages have different skills , We can cope with this competition by strengthening our study .

Q： Is there a private cloud ？

A： This competition is mainly aimed at public cloud scenarios .

Q：1、 The final goal is to implement a middleware that is responsible for data migration to TDSQL, Or implement a class TDSQL It also stores itself ？2、 For the game TDSQL yes innodb still ydstore sharedkey hash What is the algorithm ？ Can the local code calculate by itself ？ transparent transmission sql How to use the link library ？ Directly preceded by a comment ？

A： About the game 、 data 、 Evaluation and other details , Coming soon 2021 year 12 month 16 Japan 【 Tencent cloud + Community 】 cloud + The competition questions are released on the competition platform .

Q： Migration is multi-source migration ？

A： Yes , It is a scene of multi-source integration .

Q： Are there any downtime scenarios ？

A： Downtime scenarios are not excluded , Contestants need to consider this disaster scenario . Q： Is the preliminary contest to develop a complete data migration ？ Or some partial statements ? A： A complete migration program is required .

Q：1、 The migration program needs to be able to run on the source side , You can also run at the target end , You can also run on an intermediate server ? 2、 Is data migration on one machine or is the original data and target data on different machines ? A： The migration program runs on the middle server , Source end 、 Destination 、 The intermediate servers are separate , But they are all in the same zone on the cloud .

Q： Will strong synchronization be a little slow ?

A：TDSQL The performance is optimized for the strong synchronization function .

Q： Are there any simulation questions before the competition ?

A： I'm sorry , There is no simulation question before the competition .

Q： If you log in from the Internet, will your permission be restricted to some extent ？

A： Used by contestants CVM Instance and database do not restrict Internet access , You can access instances according to your actual needs . But in the assessment environment , The code submitted by the contestant cannot access the Internet .

Q： What are the violations in the database competition ？

A： See cloud + Violation instructions issued by the community competition platform https://cloud.tencent.com/developer/competition/introduction/10027

---

3、 ... and 、 About evaluation

Q： Is there any performance index for the competition ？

A： yes , we have , When the correctness verification is passed , Overall timing of the performance evaluation phase , Rank from low to high according to the total time （ The shorter the time, the higher the ranking ）. Q： What are the evaluation indicators , To evaluate the software ?

A： The organizer of the competition will provide a unified evaluation system , Details about evaluation, etc , Coming soon 2021 year 12 month 16 Japan 【 cloud + Community 】 cloud + The competition questions are released on the competition platform .

Q： To submit code is to submit the source code or the executable package ? A： Contestants will source code and its dependent code base （ It is not allowed to submit compiled binaries and libraries , Such as .jar file ,.a Files and ELF Executable documents, etc ） Compressed into zip The package is submitted to the evaluation system .

Q：C Only glibc Calculation standard library ？ All the others need source code ？

A： have access to .

Q： Deployment scripts can yum Installation dependency ？

A： You can't yum install .

Q：java Need to pack all jdk The source code of the library ？ A： Yes .

Q： Can the test environment and evaluation environment be described in detail ？ A： About the test environment , The main purpose is to give the contestants a better competition experience , The organizer of the competition will arrange the participating teams to get real examples to experience and verify the answers to the competition questions . In addition, the evaluation of the environment is mainly for the sake of fairness and justice , The organizer will build a unified evaluation system , Get the results and rank after running the code file of the contestant .

---

Four 、 Tools

Q： What database software does the competition need ？

A： Not dependent on specific software , Players focus on code and dependencies , The cloud resources required for the competition will be provided to the contestants by issuing vouchers .

Q： Data migration , Do you need a consistency verification tool ?

A： Players don't need to make this tool , But we need to ensure the correctness of the data , The data consistency before and after migration is used as the scoring item of the competition results .

---

5、 ... and 、intel relevant

Q： This time and intel cooperation , What needs to be used is intel The basic hardware of is still in the process of migration , To use intel Some functions of ?

A： The program running environment is intel On the basic hardware , Players can take advantage of the characteristics of the hardware to improve the performance of the program .

Q：intel Memory is also provided by you ？ On the cloud host ？

A： Provided by the competition organizer , Instructions and requirements will be provided when the competition questions are announced .

Q： Persistent memory does not need to be log It was written to the disk ？

A： Competitors can choose according to their own tuning results log Persistence of .

Q： The test machine is intel belt NVMe Hardware environment ?

A： The machines used in this competition do not cover .

Q： Now the choice is high IO edition , Do you want to choose PMEM Example ？

A：PMEM Instances will only be used in the finals .

Q： The game will need to be modified TDSQL Your kernel ？

A： Unwanted .

Q： Does the competition need to write its own network library ？

A： Unwanted , As long as the code provided in the form of source code is compiled and run together .

Q： Persistent memory is treated as ssd Use the words , How about stability ？ Memory is not unstable storage ？

A： The stability of persistent memory is very good , The characteristics of the storage medium are persistent , and DRAM There is an essential difference between using capacitors to maintain refresh . And in the persistent memory ECC Verification mechanism .

Q： Persistent memory and ssd What's the difference between storage ？

A： The main features of persistent memory include byte access , Write in place , The delay is greater than that of the normal SSD A lot lower , This sum Nand SSD The block devices represented by are different .

Q: A new generation of products , After execution clwb Similar instructions , It will also carry out sfence Is the order guaranteed ？

A： existing X86 Under the instruction framework ,CLWB and CLFLUSHOPT After the instruction is executed, you must use SFENCE. See Intel ® for instruction details 64 and 32 A software architecture developer's Manual ：https://software.intel.com/en-us/articles/intel-sdm

Q： Is there a big difference between the access rate of the file system and that of the raw device ？ Is there any particular advantage of access with file systems ？

A：fsdax Sure Yes 2 Two ways to access ： One is through the file system API visit , One is through mmap The way . among mmap The way and devdax There is no difference in the access rate .

After the newly created file under the file system, there are zero-out The operation of . The file system itself can help the application manage the operation of metadata . stay devdax In the mode, applications are required to manage the corresponding metadata . therefore , Which mode to use can be decided according to the requirements .

Q： If the package manager cannot be found , When compiling manually , Are there any other dependencies that need to be installed ？

A：ndctl and PMDK All have git Address of the , stay readme It lists the required dependencies . Newer package managers will basically include these dependencies . If the package manager does not , You can manually compile and install .PMDK It is recommended to install a newer version from the source code . The following is a list of common source libraries related to persistent memory ：

https://pmem.io/ndctl/

https://github.com/pmem/pmdk

https://github.com/memkind/memkind

https://github.com/pmem/llpl

Q： May I use it pmem_map_file Open a large file that already exists , Wrong report , What's the cause of this ？ It's OK to open a smaller file .

A： It is necessary to identify the problem according to the specific situation , especially flag and mapped_penp Whether the parameters are set correctly . Here is pmem_map_file Function prototype of . For more information, please refer to pmem.io/pmdk/libpmem/ perhaps github Of PMDK library .

void *pmem_map_file(const char *path, size_t len, int flags, mode_t mode, size_t *mapped_lenp, int *is_pmemp)

Q：pmem When the file system accesses , There is no need to read or write 4K Align ？

A： One of the characteristics of persistent memory is byte addressable access , So you can access... More finely .

Q：libpmemobj When not in business , The performance and libpmem Is there any difference ？

A：libpmem Is the lowest level and the most granular , Therefore, the efficiency is the highest .Libpmemobj The data structure of the object is used , The cost is slightly higher than libpmem.

Q： Can persistent memory replace all current database writes wal The disk on which the file is located , The read and write speed of persistent memory is the same as that of disk or ssd Is there any performance comparison between ？

A：wal In theory, they can be placed on persistent memory , The method can be direct file read / write and use mmap Read and write in a different way . In practical application, it needs to be based on wal Evaluate the performance benefits of the overhead ratio in the application .

The latency of persistent memory is in the hundreds of nanoseconds ,Nand SSD The delay is more than tens of microseconds .

---

6、 ... and 、 Technology side

Q：mysql-conector How to execute delivery sql？ A： You can refer to Tencent cloud TDSQL Official development manual , Or check ：https://cloud.tencent.com/document/product/557/47559

Q：TDSQL The primary and standby databases of support consistency , Is it an extreme case of discarding high availability selection consistency ?

A： stay 1 Lord 2 Under the standby architecture , By default, the consistency of priority data is guaranteed . But in extreme cases and to the extent that the business is acceptable , Availability is optional .

Q： Relationship between master library and slave Library ？ Is it a backup from the master database , One master and three slaves are three backups ？

A： The slave database synchronizes the data of the master database in real time . Yes .

Q： If a master-slave switch occurs ,TDSQL Master and slave event How to deal with ？

A： On each node agent Judge the active and standby nodes of the current primary node , To decide event The state of .

Q： Are both master and slave nodes partitioned ？

A： Each slice is specified by the user when purchasing 1 Lord X From constituted , One TDSQL The instance consists of such multiple slices .

Q： Is sharding a physical storage location ？ If it's three slices , What is the difference between the use or storage of three partitions and one partition ？

A：TDSQL It will automatically divide databases and tables , A slice stores part of the data of a table after horizontal splitting , If there are three slices , Then each slice stores one third of the data . Each slice is designated by the at the time of purchase 1 Lord X Make up of . attach TDSQL Product documentation entry ：https://cloud.tencent.com/document/product/557/7700

Q： Can sharding cross availability zones ？ If you can cross availability zones , How many kilometers can you reach ？

A： Can span availability zones . The city is within 100 kilometers .

Q： The difference between slice and table ？

A：TDSQL The slicing function is realized automatically , Save part of the data of the whole table on each partition , The access engine will automatically route the user's request to the corresponding partition , So as to realize the horizontal sub table . This sub table is transparent to the business , The business only needs to operate one table , Reduce business development complexity .

Q： After the separation of computation and storage ,tps and qps How to improve 100%？

A： This question should be answered in detail for specific scenarios or more detailed background descriptions ,TDSQL Strictly speaking, an instance cannot be regarded as a storage computing separated architecture .

Q： Will data be lost in case of power failure ？

A： Don't lose data , You can configure parameters to ensure that the log data falls to the disk .

Q：TDSQL Can the database support hundreds of billions of data ？

A： Support , Trillions of rows of data are accessed in milliseconds , You can refer to the documentation to learn more about the product ：https://cloud.tencent.com/document/product/557/8763

Q： The source is SQL The file or MySQL database ？

A： When the competition questions are announced, they will be explained in detail in the competition questions .

Q： How to deal with foreign keys in a table creation statement ？

A： You can mask foreign key constraints first , After the table is built , And on again .

Q： Distributed database , What are the advantages and disadvantages compared with the centralized database mounting a large disk ？

A： Distributed database , You can freely choose the disaster tolerance level and availability zone , Single slice failure has little impact on the overall request . However, more resources will be consumed in analysis scenarios such as multi slice data aggregation .

Q： Multi center remote consistency is achieved through zk Is it realized ？

A： If it is a multi center in the same city , Data consistency is achieved through MySQL Synchronization of is guaranteed . If cross city disaster recovery , Cross instance asynchronous disaster recovery is required .

Q： Global... Can be allowed group by Operation ？

A： You can do it , When there is a large amount of data, it will consume resources .