Selenium is detected as a crawler. How to shield and bypass it

Thank you for your reading and company , I put the automatic test content that I usually use more into a small volume of the system , You can directly click the text below to read , I hope to provide a little help for those who are destined .

• ️Selenium Web automation test practice
• ️Appium Mobile terminal automation test practice

Selenium Operation is blocked

Use selenium When automating web pages , There is a certain probability that it will be recognized by the target website , Once detected , The target website will block the web page operations made by the client .

For example, the landing pages of Taobao and public comments , When opening the browser manually , When you enter a user name and password , Can normally enter the home page , But if it's through selenium open , It will directly prompt that the verification fails , Click the box to try again .

This paper introduces a method , There is no need to modify browser properties , There's no need to inject JavaScript Script , It can also easily bypass website detection .

Selenium Why was it detected

Every time a browser visits a website , Will bring specific fingerprint features , The website will parse these features , To determine whether this visit is an automated program .

One of the most well-known features is window.navigator.webdriver, This feature directly indicates that this browser is webdriver Program . When a browser passes selenium After starting , Enter this property in the developer tool , You'll find it marked true, The browser opened manually is false.

actually , The browser was detected as webdriver This is not the only feature of the program , It means , Even if you modify the properties , It may not be able to bypass the detection of the website .

We can go through sannysoft To detect browser fingerprints , If the browser is through selenium When the automation program opens , After visiting this website, many features will expose these fingerprints , The values of these features are different from those after manual opening , So it can be easily detected by others .

Some people also try not to selenium, Switch to puppeter and playwright Such automation tools , But the ending is the same .

Selenium Methods to avoid being detected

The browser starts with a fingerprint , If an automated program is used, it has been marked when it is started , Then why not start it manually , And then use Selenium Connect to the browser that has been started ？

What should I do to open the browser manually to make Selenium How about the connection? ？Selenium How to connect to the browser opened manually ？

I am here Selenium Connect to an existing browser 1 This article details the steps required , Sum up ：

1、 Add the following parameters when opening the browser ：

--remote-debugging-port=9222 --user-data-dir="C:\selenium\ChromeProfile"

2、selenium Set browser options in , Through the... Set above 9222 Port connection browser ：

from selenium.webdriver.chrome.options import Options

chrome_options = Options()
chrome_options.add_experimental_option("debuggerAddress", "127.0.0.1:9222")
driver = webdriver.Chrome(options=chrome_options)

adopt subprocess Run the browser

Of course , To do automated programs, you usually don't click the icon manually to open the browser , We can start the browser from the command line , And then use selenium Connect .

import subprocess
cmd = '"C:\Program Files (x86)\Google\Chrome\Application\chrome.exe" ' \
'--remote-debugging-port=9222 ' \
'--user-data-dir="C:\selenium\ChromeProfile"'

subprocess.run(cmd)