How to shield yandexbot

We often get a lot of  YandexBot  visit , We know YandexBot yes Yandex Search engine Reptiles .

a large number of YandexBot visit , It not only causes great pressure on the server , At the same time, it will also make normal visitors visit the website slower , Influence the experience .

For the above reasons , We have to limit YandexBot The interview of , But one thing to note ： Limit Yandex Visit will lose Yandex Bring us traffic , Because of the limitation Yandex Crawler index our website , So when users are in Yandex When searching our website for topic words , There will be no show , So it won't bring us any traffic .

If Yandex Didn't bring us any visitors , Then let's block it ！

We can shield in the following three ways Yandex：

IP Address masking

We go through  list crawlers YandexBot  Check out YandexBot Of IP Address , Let's put these directly IP Just add it to the blacklist , I take Ubuntu Operating system as an example , How to integrate IP Join the blacklist of firewall ：

sudo iptables -A INPUT -s 213.180.203.82 -j DROP

above IP The address is that we passed  list crawlers YandexBot  Page query IP , Directly one by one IP Just type in , One line at a time .

benefits ： Direct and rapid .

shortcoming ： Some may be missed YandexBot Of IP Address .

adopt User-agent shielding YandexBot

Through my article ：yandex bot user agent, We can see each one YandexBot The specific User-agent, We are Nginx It can shield specific User-agent：

if ($http_user_agent ~* "Yandex") 
{ 
    return 403; 
}

So we can get through User-agent shielding YandexBot The purpose of .

advantage ： Can quickly and directly shield YandexBot.

adopt robots.txt shielding YandexBot

In the previous article, I have introduced how to pass robots.txt shielding YandexBot 了 , I won't say much here , You can read this article if you need ：block yandex bot

advantage ： Comply with official specifications .

shortcoming ：Yandex Some of the reptiles do not obey robots.txt agreement , Use  robots.txt agreement It can't be shielded .

summary

This article uses three ways to shield YandexBot, And introduced the advantages and disadvantages , There is always a way for you .

Finally, I need to reiterate ： If Yandex It doesn't bring you any traffic , You can safely shield , If Yandex It brings you traffic , Think twice before you act .