Event related | reveal how Ti-One's support ability for large-scale events is developed

2021 The preliminary stage of Tencent advertising algorithm competition has ended , The competition received from thousands of universities, enterprises and institutions at home and abroad  4,335 people   entrants . As the only special machine learning platform designated by the official competition , Tencent cloud TI Machine learning platform in the platform TI-ONE（ hereinafter referred to as TI-ONE） Not only for all contestants to provide data preprocessing 、 model building 、 model training 、 The whole process training ability of model evaluation , And provides up to  1000 Zhang   Of GPU Card and  500T  High performance storage , Support  1500+  The contestants successfully completed the competition .

that , It has been the supporting platform for Tencent advertising algorithm contest for four consecutive years ,TI-ONE What are the advantages ？ Based on this , Xiaobian invited people who participated in the whole competition process TI-ONE Experts in technology , Xie Bowen, senior engineer of Tencent cloud , He will reveal the secrets for everyone from a technical point of view TI-ONE How to develop the supporting ability of large-scale events step by step .

 Q1： What is the difference between this year's event and last year's event ？ Where is the challenge ？

The theme of this year's competition is video advertising , The data set used is sensitive video data , This is right TI-ONE A number of challenges have been raised .

One side The reading and training of video data have a great impact on the computing power of the platform 、 Data reading and writing performance A great challenge , We need to be prepared from the beginning of the game GPU Calculate the force , And high-throughput data storage services .

On the other hand, the event organizers Security of data There are strict requirements , need TI-ONE Ensure that data will not be easily leaked to the Internet environment .

in addition , Because the video training task is very important to the contestants 、 Research 、 The algorithm development ability has put forward higher requirements , Contestants should complete data processing during the preliminary contest 、 Algorithm implementation 、 Model tuning 、 Result test and other steps , Need to improve 、 Stable deep learning and training environment .

 Q2： To support this year's event ,TI-ONE In what aspects have you prepared and improved ？

First , We have prepared a lot of  GPU  resources , And through reasonable Plan how to issue vouchers Optimize the time distribution of players' use of calculating power , It ensures that players can successfully apply for computing resources when needed .

secondly , We did before the game Adequate measurement and drilling , Evaluate the typical computing power and memory overhead required for the game , Based on this, the best default configuration is recommended for players when they create computing resources , This saves the cost of vouchers for the contestants , It also effectively improves the performance of the whole platform GPU utilization .

Again , To ensure data security and read / write performance , We introduced the high performance version CFS  Storage service , This scheme limits that players can only access data in the Intranet environment , And can provide a total of up to 10GB/s Read and write throughput of . At the beginning of the tournament , A large number of players have concurrent access to the intranet 50GB Video data for , Every player can also achieve 100MB/s Read speed of .

Last , We provide Open the box Of  Jupyter Notebook  Version of Baseline, It also provides a way to directly execute  Baseline  Complete program conda  Environment initialization script , Contestant here Notebook Under the guidance of , It can efficiently complete the construction of algorithm development environment , And easily realize model training 、 Results testing and other tasks .

 Q3：TI-ONE As a professional competition platform , What are the advantages ？

First ,TI-ONE As the only official designated platform for algorithm competition , It can provide rich computing resources Support thousands of people to compete at the same time , To provide players with an excellent event experience , Compatible with TensorFlow、Pytorch、Pycaffe And other mainstream open source machine learning frameworks , Players can flexibly define algorithm modules on the platform .

secondly ,TI-ONE Support Multiple structures / Adaptation of unstructured data source and data storage Store as a platform , At the same time, it supports docking with a variety of external data sources to obtain training samples and business production data , Except Tencent cloud used in the competition COS Storage , It also includes database storage 、Ceph Storage, etc , For the players to eliminate worries .

Again ,TI-ONE built-in exceed 150 An algorithmic operator , Covering data access , Data processing , Feature Engineering , Machine learning framework / Algorithm , Deep learning framework / Algorithm , Visual evaluation and other model building links , It can fully meet the requirements of the contestants .

Last , Use TI-ONE Play the game , Players can Save time in building machine learning platform and managing physical resources , Focus on more valuable modeling work , Improve efficiency .

---

After reading the blog post, I will share with you , Are you right about the event “ Behind the scenes ” as well as TI-ONE With more understanding ？

TIPS：

2021 The second round of the Tencent advertising algorithm contest has been held on 6 month 16 Japan 12 The curtain officially opens at ,TI-ONE It will still be the only official designated platform to escort the semi-finals , I wish you all the more fighting and braver , Achieve satisfactory results .

in addition ,TI-ONE It will also serve as The second round of wechat big data challenge The only official designated platform , Provide power support for all semi-finals players . I believe you will see it in more and more competitions in the future TI-ONE The figure of ,TI-ONE It will also continuously improve the user experience in this process , May everyone be in TI-ONE You can complete one challenge after another with the help of , Keep breaking through yourself , make progress ！