Turning and anti-climbing attack and defense

一、背景

某日,Data group to report,前几日DAUIncrement is amazing,WangPingTai verify whether accurate.GMr Looked at the data,Think this wave of growth should be affected by a week before the new media on the traffic,Is thinking,A voice from leisurely in the corner：“Did they come back again？”

GMr Instant straightened back,熟悉的人都知道,This is something he has encountered in special dangerous,Such as the Tibetan mastiff will pose when the beast posture.不知过了多久,He slowly weakened body stand in the chair：“是的,是它们,Hateful insect！”

二、现状

GMr Filled up around,Listen carefully in his analysis.

“Now our situation is like this.”G先生说：“In fact, we are under a lot of every day in crawler intrusion,Sometimes they are just our defense mechanism in the system,Sometimes they just suspended the attack.不幸的是,The latter this kind of situation seems to be a bit more.”

“总结起来”

“They typically have two characteristics”

• 规律性

• 高频次

“They generally have two kinds of attack mode”

• 正面进攻
  ：This pattern is characterized by a large number of requests,Rough camouflage,Focusing on an interface,By the number and diversity to win.

• espionage operation
  ：Less but more regular request,They will try to disguised as a real user,Mimic the behavior of users,To obtain the key interface more core information.

“The harm of them mainly reflected in four aspects:”

• 数据安全性
  : 作为电商平台,The key commodity information,And user information once crawl,Most likely resulting in the loss of goods、用户信息泄露、Even telecommunications fraud and a series of safety problems.

• 大数据异常
  : Statistics such asdau,pv,uvEtc are dependent on the interface request log every day,Once these log records the data of real users creeper,You will lose statistical effect.

• 服务稳定性
  : Due to the above mentioned the first attack mode,The crawler will be a large number of requests,Some even close to yu hong pan attack,This will greatly increase the load on the server,If there are new activities at the same time,Can cause traffic soared,Leading to system paralysis.

• Personal failure
  : appAccording to provide all the user's search content to the general user search keywords,The crawler if intrusive search interface,And a large number of malicious keyword search,Is expected to push for normal user keywords less accurate,从而影响用户体验.

“Turn around nowappAlready in the crawler strategy is based,The legitimacy of the only contains some necessary check,Hour level delay risk control and manual banned strategy.”GMr Shook his head：“To guard against attacks by means of the existing,Blocking the harm is not enough.”

三、前人研究

“In the light of the characteristics of the above mentioned crawler,We will make around construction is a based on the characteristics of the crawler,Enough to avoid the two attack mode,尽量不依赖前端,Can accurately distinguish normal users and the crawler request,And achieve the second level real-time banned the crawler system.”

“We need to learn from predecessors explore”

“Several common backend back crawler method”

• 登录限制
  : Requires the user to request interface must login,This approach can largely increase the cost of the crawler,But more arbitrary,In the key nodes are easy to affect the user experience.

• cookie校验
  : 用户请求cookieCan carry some used to identify the identity of the data in the,These data have their own a set of generating rules,Support the validity check,对这些数据进行验证,Can identify whether the real user.But when the crawler cracked generate rules,Or use of real userscookie进行请求时,This method cannot be effective defense.

• Frequency check
  : Use the common features of each request（由于ip的成本最高,所以一般选择ip）frequency statistics,To decide whether the characteristics of banned for.Due to the crawler regularity,高频次的特点,This method can effectively prevent a lot of the crawler request,But it also brings another problem,同一ipSometimes more than one user in with,May be injured user.而不选择ip,Select other dimensions,Fake and low cost.

• 验证码校验
  : 同一ipMany times or user request reaches a certain threshold,Requires the user to enter the verification code,Verification code there are many kinds of,如文字、图形、滑动等.The sliding graphical verification code works best,Because of the high cost of image recognition,But need front end with,But also will affect the user experience.

• 数据加密
  : The front of the request data is encrypted calculate,And the encrypted value as a parameter to the server,在服务器端同样有一段加密逻辑,生成一串编码,With the request parameters matching,匹配通过则会返回数据.This method still need the client to participate in,And the encryption algorithm expressly written in the bookJS里,The crawler or can be analyzed out.

“These methods all have their own advantages of existing,But their disadvantages are significant.”GMr Frowned：“可以肯定的是,The use of any a,Or simply stitching this several ways,Does not ensure that the user experience while,达到我们需要的效果.”

四、CleanerReptile cleaner

“We need the crawler system has the characteristics of what？”GMr Look around,Simply said：

“The basic characteristics of the crawler system”

• 准确性
  ：Have the ability to grasp the crawler,To avoid friendly fire again.

• 实时性
  ：秒级别的响应,If catch all reptiles such as the data,满载而归了,Again it's meaningless to ban.

“Combining with the attack mode,To cope with the crawler method”

• 正面进攻
  ：合法性校验,频次控制.

• espionage operation
  ：用户行为分析.

“Based on the characteristics of the crawler and the climbing method,In fact we already have the prototype of the system,Just haven't put into development.”GMr Evil spirit's smile：“这就是我们的

CleanerReptile cleaner

系统.”

4.1 系统模型

“我们的Cleaner系统主要有3个模块.”GMr Lifted2根手指.

“

数据处理中心

”,And he bent a finger.

“

Block Center

”,He bent a finger.

“以及,

封禁库

”,Then he found that not enough,But there is no interrupted by a little mind,Turned and took out a picture,继续说道：

“The three modules as shown.Cleaner利用mq进行解耦,用户日志,redis,The local cache for the realization of the function.Below I to explain in detail each module”

4.1.1 数据处理中心

“Based on the accuracy and real-time,Want to consider is that,Alone the behavior characteristic of a request is not enough,Must contact all of the same user requests over a period of time,This means going to gather all the log to identify a period of time,In huge amounts of data and analysis of the characteristics of each beam request chain、Degree of regularity and dangerous.”GThe expression of Mr See everybody as wise as he,点了点头：“是的,The key here is that huge amounts of data,While huge amounts of data real-time processing,You can think of what？”

“对！就是flink.”GMr Turned loudly：“flink作为一款分布式、高性能、实时、Accurate origin handling frame,完美符合我们的需求.CleanerThe data processing center was based onflinkReal-time processing logging implementation,The log after it in the treatment of useful information aggregation intomqSent to the downstream of the banned center for further processing.”

“flinkWe won't discuss the principle of the,We just use it to feature for data aggregation.The following picture show isCleaner系统中flink处理日志的过程.”

“Get the user's data processing center each request log,Depending on the agreed format,生成一个对应的EntryRequest对象,This object contains banned center need to identify、Analysis of user characteristics,Including request interface name,用户唯一标识,ip,版本,时间戳等等.”

“接下来,Request collectorRequestCollectorWill be one by one to theseEntryRequest对象收集起来,In its interior in a bounded blocking queue.The use of bounded,In the process is to prevent the next,Due to some time processing is not timely,Data backlog caused by,Thus affect the system function”

“MQSend assembled classSecurityMQPuzzlerWill constantly from the bounded queue outEntryRequest对象,In the same way in order to prevent data backlog,It maintains a maximum capacity of10w个key,Write an hour after the failure of the local cache.”

“缓存中的keyCan be set to want to identify the dimensions,A unique identifier can be either user,Can also is the most commonly used,Cost of replacement of the highestip.缓存中的value是同一key下EntryRequest的list,In the order to get anEntryRequestis put into thislist中,这个listTake a sort of sliding window,当listTo the length of the pre-made threshold,就会将这个list放入MQ实体中,发送出去.”

“总结一下”

“Data processing center is to make sureCleanerSystem real time the key.Because of its characteristics to deal with huge amounts of data,flinkAs its core data analysis work,Data processing center and downstream banned use betweenMQ进行削峰,解耦,In order to prevent the possible between modules of vicious chain reaction.”

4.1.2 Block Center

“To resolve is the problem of real-time data processing center,And banned center is mainly to solve the accuracy problems.Its design the diagram below.”

“Banned centre core is to set up

策略

与

计分标准

.”

“Upon receiving the data processing center ofMQ后,分析数据之前,First of all we need a switch,If one day, after all, the system convulsions,At least can close it in time.”

“Strategy center contains many analysis of user behavior,它会对这些MQIn the user's behaviorlistAccording to these strategies to determine,打分,Each strategy will have a score.If the user the final score is greater than the sum of set punishment threshold,Will the ban.”

“Several kinds of main strategy”

• The only user identity legitimacy
  : Due to the formation of the user's unique identification has a certain rule,Around is no exception,Naturally we can use these rules to determine whether a request for the real user.A large number of illegal request don't need other judgment,Use only generate rules can block,This strategy is mainly used to stop the frontal attack.

• 频次
  : 同样地,This strategy is mainly used to stop the frontal attack.But it is used to make up for the inadequacy of identity legitimacy strategies,When the crawler using real user's identity to a large number of requests,We can use them the characteristics of high frequency,Setting the threshold value for a particular interface frequency,When the request more than limit,To specify a user characteristics were banned.

• El表达式
  : The above two strategies can significantly weaken the influence of the frontal attack,But for spying operation is almost powerless,Because when some cunning crawler repeatedly frustrated,Ascertained the climbing strategy,after the frequency threshold.They will choose to imitate the behavior of real users to request,Give up a short time to get a lot of information fantasy,Turning to though time is long, can access to the complete data.这时ELExpressions are used,The core can be banned users a request list,CleanerCan analyze the user behavior in this request,Look at them any deviation from the normal user request,At this time of the crawler although did not have the characteristics of high frequency,但周期性,The characteristics of regularity,Is the crawler sin,They will never be able to avoid.According to some different interface request order,Frequency ratio of different characteristics such as whether the user can be deduced illegal.

• 封禁记录
  : 辅助策略,To hit the banned user dimension into the database,As an indicator of determine whether banned in the future.

• 黑白名单
  : Specify the characteristics of skip or forced banned,Support manually add,In case of system failure or disorder.

• Access to other banned library
  :辅助策略,Combined with other business banned information,Improve the judgment result.

“计分标准”

“Scoring criteria is actually a score for each policy fine-tuning,This needs to be derived from many times in the actual combat against the crawler,And in the end to a certain value of dynamic balance.”GMr Was lost：“There is no perfect determine the numerical,We can do is the infinite.”

“总结一下”

“Banned center isCleaner系统的核心,Is key to ensure the accuracy of the whole system.Due to the crawler will evolve,Offensive way varied also,So the key to the banned center is iterated,Dynamically adjust various strategies and parameters,In practical application of perfecting,达到最好的效果.”

4.1.3 封禁库

“To be banned users,We will put the dimension of the banned and gateway connectedredis中”

“When the user requests into the gateway,We need to quickly determine the request is illegal,由于数据量庞大,Time consuming to as low as possible,So the design of the banned library must lightweight,The response must be fast enough.CleanerBans the library only use local cache andredis,Every once in a certain time interval,The local cache will be fromredisUpdated in bannedkey,The local cache takes almost negligible.”

“System module is something like this,Anyone interested to develop？”GLook around Mr Yu inside,Find a pair of warm eyes：“小C,就你了.”

小CSuddenly stood up,和GMr Plot up into the little black house together.

4.2 效果

A thousand years later...

“实际应用后,Experience several crawler cleaning tasks,The process also has many parameters and strategy adjustment.”GMr Patted the littleCshoulder said：“Indeed the final effect is remarkable,你看这个图.The horizontal axis is every every dayipThe number of requests a core interface,Vertical axis is every request the interface within the scope of a certain number ofip总数.”

“蓝色的线是CleanerShut down one day,Orange isCleanerStart the day.Can see in closeCleaner时,Request a day the interface50次-100次的ip个数有100多个,500次以上的有400个左右.The overall trend of rising.而启动Cleaner之后,50次-100次的ipNumber up to1000个以上,但200The more times the number of requests to0,这说明什么？”

“说明我们成功了.”小C激动地说：“50次ipShows the rise of the original200次以上的ipare requesting50Was banned in time after time to live！”

“是呀,Now it seems at least at this stage we are successful.The crawler to block the attack has been we.Lived up to the time of hard ah！”GMr and JrCBrilliant smile,Laugh like two200A pound of mastiffs.

五、总结

We are in the process of dealing with crawler,Examines the predecessors' research achievements of,Summarizes the crawler has and the climbing system should have the characteristics of,Final planning out a practicable specific anti crawler system solutions.

反爬虫系统

• 基本特性
  ：实时性,准确性.

• 基本功能
  ：合法性校验,频次控制,用户行为分析.

• 基本模块
  ：大数据处理中心,Banned strategy center.

• 基本策略
  ：Legitimacy check strategy,frequency strategy,ELExpression strategy,黑白名单策略.

• Two dynamics
  ：The perfection of the climbing strategy、插拔;The adjustment of the scoring standard.

• a balance
  ：The crawler and the game process between the crawler is a long,Both will eventually reach a state of balance,In the face of the crawler continuous rebound,We can do is to continue to monitor,suppress quickly.

目前,CleanerThe crawler system can achieve suspicious user10sIn the rapid banned,In the search interface, for example high-risk request block with millions of times a day.

CleanerData security is always guarded around.

作者简介

Gao Mengning,转转-平台-basic ecology-后端开发.

转转研发中心及业界小伙伴们的技术学习交流平台,定期分享一线的实战经验及业界前沿的技术话题.

关注公众号「转转技术」（综合性）、「大转转FE」（专注于FE）、「转转QA」（专注于QA）,更多干货实践,欢迎交流分享~