当前位置：网站首页>Future trends in automated testing

Future trends in automated testing

2022-06-24 08:11:00 【ThoughtWorks China】

background

Automated testing begins with the earliest recording and playback technologies , Gradually develop into DOM Object recognition and hierarchical automation , And based on POM（Page Object Model） To improve use case reuse , To the current hot based on AI Automation of technology , It shows that the development trend of automated testing is more intelligent , More accurate , More efficient .

Here we introduce two intelligent automatic testing technologies that have been widely used in the industry ：

self-healing （Self-Healing） technology
machine learning （Machine Learning） technology

Self healing technology

1.1 What is self-healing technology

self-healing （Self-Healing） Technology in computer terminology means ： A self-healing management mechanism .

Analogy to life , When a living body suffers some small damage , Their bodies can often achieve self-healing through their own repair mechanism , Without outside intervention . Such as gecko's broken tail regeneration , Or the body regeneration ability of crabs .

Back to the computer world , Self healing technology is also widely used , For example, the information channel of the chip is self-healing , Software system fault self-healing, etc . So what we want to introduce here is a self-healing technology in the direction of automated testing ：

You can find unexpected errors in the execution of its test script , And change it without manual intervention , So as to restore itself to a better running state .

1.2 Principle introduction

Problem domain ： Using self-healing technology in automated testing mainly solves ** Object recognition （object identification）** problem .

Traditional automated testing frameworks and tools , Use the application model to define the components and objects of the application and their properties . These definitions are then used to identify and manipulate application components . But applications change frequently when they are updated . May be intentional developer changes or immediate （ By the application system or build process ） the . These changes undermine our traditional automation approach based on static definitions .

natural language processing （NLP） And machine learning (ML) And other intelligent technologies have been developed to test scripts “ Study ” and “ To adapt to ” The point of ; Use of self-healing automated testing tools AI And machine learning technology , According to the user interface (UI) Or changes in the application environment , The test process can be updated and adjusted automatically .

When running tests , They scan the application's user interface to see if there are any objects . They then compare these objects to the application model previously generated for automated testing . If there are any changes to the application , There is a technology that allows tests to adapt and update automatically . This ability is called “ Self repair ”. Property changes are automatically sensed , Internal scripts repair themselves at run time by repairing themselves .

The self-healing function has the following two remarkable features ：

In the process of execution , If a test step locator cannot be detected by its default locator value , Other locator policies in the list will be automatically applied , No manual intervention by the tester is required . Execution will continue , It's like there's no fault .
In the process of execution , If the test step locator fails , And cannot be automatically detected using any other locator policy , The test will be suspended , Allows the user to select the relevant element and continue . The new locator policy will be automatically updated the next time it is executed .

The following figure describes the main steps of self-healing technology ：

( picture source ：https://www.impactqa.com/blog/5-great-ways-to-achieve-complete-automation-with-ai-and-ml/)

The advantages of automatic test self-healing technology mainly include ：

Reduce test failure rate
It is normal that the test execution fails , But sometimes the root cause of failure is simply the user community The surface changes and the test script does not change synchronously . After using self-healing technology , It is unlikely that script execution failure will be affected by the object location that cannot be correctly recognized . However, the traditional automation method cannot recognize these changes and update them automatically .
Improve test stability

If we have flaky tests exist , It is difficult for us to determine whether our test is stable .”NoSuchElementException“ Errors are one of the several errors that make test design unstable , It is very difficult for the test team to completely control such phenomena . And when our test design is consistent with the application , The test is less likely to fail during execution , And the execution process is smoother .

Improve script maintainability
Changes in the test code are positively related to changes made by the developer in the application . The reason for the test failure may change , And does not reflect AUT The true state of ; Therefore, failed test results will limit testers' access to meaningful insights about their tests . By identifying and updating any changed test cases in the user interface , Self healing technology saves the time and energy of agile test and development teams . In addition to time and energy , Self repair in test automation also significantly reduces the cost of test script maintenance .

1.3 Industry practice

We can see , There are already some good practices in the industry , such as Healenium project .

With Healenium Project as an example , See how automated test self-healing technology works ：

Suppose we go through id To locate a button on the application interface , The positioner should be ：#button

You can see from the above picture that , Elements can be correctly positioned to .Healnium Will save the correct positioner , As the benchmark value for the next test execution .

Now? , Let's assume that the developer changes the... Of the application UI Interface , Changed the of this button id attribute , from **#button Changed to: #green_button**. But for some reason , The test team was not notified of this change , So the test script has not been updated . So when we execute the script again , Trying to use **#button** When the old positioner goes to position the button , The script will report an error , Tips “NoSuchElement” The error exception of .

under these circumstances , standards-of-use Selenium The implementation test will fail , But use Healenium Will not be . Healenium Capture NoSuchElement abnormal , Trigger Machine learning algorithm , Pass the current page status , Get the previously successful locator path , Compare them , And generate a list of fixed locators . It takes the locator with the highest score and uses it to perform operations . As we can see, the element was successfully found and passed the test .

After the test runs ,Healenium Generate a report , It contains information about repairing locators 、 Screenshot and all details of the repair success feedback button .

If the repair is successful , We can use Healenium Idea Plug in updates our automated test code ： The plug-in uses the repair locator to find, repair, and update test code .

Healenium Use a machine learning algorithm to analyze the changes of the current web page ： The longest common subsequence algorithm based on weight
More details about this project , You can visit the official website of this project ：https://healenium.io/

Through this case , Let's compare the differences in object maintenance between traditional automated testing methods and automated testing methods based on self-healing technology （ As shown in the figure below ）：

2. machine learning

In the previous section , We introduce the self-healing technology to solve the problem of object recognition in the process of automated testing .

But in automated testing , We still face other problems ：

It is still necessary to obtain the positioning mode manually ;
If it's through Canvas Drawn objects , How to identify elements ( Such as Flutter Web).

wait .

Although the self-healing technology adds some fault tolerance in the traditional automated testing . But in essence, it is an object recognition technology based on element location , It is still traditional DOM Location Technology , Such as XPath Or is it CSS positioner .

And we know that , Mobile and Web The operating environment of the terminal is complex , External interference factors can easily destroy the stability of automated script operation , This is a Element locator brittleness By nature .

Therefore, the industry has been continuously exploring more stable object recognition technology . In the early days we used CV（Computer Vision） Computer vision +OCR（Optical Character Recognition） Optical character recognition technology .

and CV and OCR Is based on Image processing and statistical machine learning methods .

For example, the more popular automated testing framework in the industry airtest, Is based on CV Technology to identify intelligent controls .

adopt OCR And image recognition ability , Under the same process , The ability of an automated script to execute on multiple platforms , It greatly reduces the cost of script writing and post maintenance .

At present, the industry is also practicing and exploring machine learning technology CV and OCR To solve the high learning cost of automated testing 、 Maintenance costs are high 、Hybird Poor recognition 、 Poor cross application capability , And does not support cross platform .

2.1 Intelligent identification

stay UI On the page , Our information is mainly composed of images and words . How to efficiently identify control objects based on images and text , It is a problem that current automated testing has to face . In order to solve the problem based on DOM The element positioning of technology is unstable , And later maintenance costs . Currently in UI There are two ways to identify controls ： One is based on CV Technology of image recognition , One is based on OCR Technical character recognition .

2.1.1 be based on CV Image recognition

Conventional CV

In the traditional way , We are mainly based on image feature recognition technology ：
When it comes to image recognition algorithms, we can't get around them OpenCV, What is commonly used is SIFT Algorithm , The core is to extract some key features from the image , These features are available in different models , It has good adaptability under different resolutions . The processing process based on traditional feature recognition technology is shown in the following figure ：

Based on deep learning CV

This method is better than the current positioning strategy （ image ） More flexible , Because we can use CNN, Or other in-depth learning framework AI Model to identify control icons , There is no need to know the context , You don't need to match the control icon exactly . That is to say, we can find one such as... Across applications and platforms “ The shopping cart ” Icons for such controls , There is no need to care about the subtle differences . The processing flow based on deep learning is shown in the following figure ：

Although based on deep learning CV Have stronger ability , But the traditional way still has irreplaceable advantages , It's worth learning . For example, at present, deep learning still needs a lot of data , And the traditional way will save a lot of cost in this respect . For some simple identification tasks , We prefer traditional CV The way .

2.1.2 be based on OCR Character recognition

OCR Can recognize predefined characters on the screen . Use OCR The software of will adopt “ Best guess ” To determine whether the image matches the character , In order to convert the image into text that the computer can process . Conventional OCR Based on image processing （ Two valued 、 Connected domain analysis 、 Projection analysis, etc ） And statistical machine learning （Adaboost、SVM）.

Conventional OCR Only relatively simple scenarios can be handled , Such as ： Simple page layout 、 The foreground and background information are easy to distinguish and each text character is easy to segment . As our test subjects become more complex over the months , The page layout , In various scenes such as styles , Tradition OCR The accuracy of has also been challenged .

With the development of deep learning , We can solve the traditional problem through new algorithm technology OCR The limitations of .

2.2 Case practice

In the industry , Mostly based on AI The automatic test platforms of all the systems adopt CV+OCR Combined intelligent identification technology , To reduce the cost of writing automated test scripts and later maintenance costs .

We are based on OpenCV Of airtest Platform as an example ：

be based on OpenCV Of UI automation - Airtest

Airtest It mainly uses two traditional OpenCV matching algorithm ：1. Template matching , 2 Feature matching

The characteristics of the two matching algorithms are as follows ：

Template matching

Unable to recognize across resolutions
There must be a relatively best match
Method name ：“tpl”, “mstpl”

Feature point matching

Cross resolution recognition
There may not be a match
List of method names ：[“kaze”, “brisk”, “akaze”, “orb”, “sift”, “surf”, “brief”]

stay Airtest You can configure the matching algorithm you want to use . Because the two matching algorithms have their own advantages and disadvantages , Therefore, the general default is to select the combination of these matching algorithms , The algorithm carries out image recognition in turn , Finding the result will stop identifying , If the result is not found, it will be recognized circularly according to the recognition sequence of this algorithm until it times out .

How to judge the success or failure of image recognition ？Airtest There are two important nouns in it ： threshold and credibility , threshold It can be configured , The general default is 0.7, credibility Is the probability of probability calculated after the algorithm is executed , When credibility ＞ threshold When , The program will think Found the best match ; And when credibility ＜ threshold When , The program will think No best match found .

Here's an example ： stay Airtest Operate Netease cloud music in APP ：

Touch(“ picture ”) The principle is as follows ：

Airtest It does not directly provide OCR Mode identification , But we can integrate open source Tesseract-OCR Library to support OCR Recognition ability .

challenges ：

Traditional based OpenCV Image recognition , The main problem is that the image feature recognition is not accurate enough , In particular, there are few features in the image itself , If there is a large white background , Or dynamic elements . At the same time, the average success rate of traditional recognition is 80% about , It can't reach the artificial level 95% The accuracy of , So in the traditional way , We can only optimize the recognition rate by adding more feature information , But I want to match the accuracy of manual work , The traditional statistical machine learning method is difficult to achieve .

To solve this problem, we need stronger generalization ability , At present, it is more based on CNN And other deep learning technologies to solve such problems . We don't have much to do here , You can refer to Appium with AI project .

details ：

https://appiumpro.com/editions/39-early-stage-ai-for-appium-test-automation

2.3 Future outlook

With DL,RL and NLP And so on , Will our user stories be automatically transformed into automated test cases in the future , Did it real The zero code of , At present, there are also such explorations in the industry , We are also following up .

CV and AI The blessing of the algorithm makes UI Automated testing has made a new breakthrough in object recognition , But still can not get rid of the software layer API Limitations of operation , Limited by the operating system , There is still the problem that elements in some specific scenarios cannot be recognized （ As in the system Push Message operation ）;

We can see that the leading companies in the industry are trying to use the automatic manipulator , To solve this problem .

As shown in the figure below, Ali's Robot-XT：

( Picture provenance ：https://mp.weixin.qq.com/s/5ZngQyJiRZy6714-CC498g)

Are these technologies owned by big manufacturers “ patent ” Well ? I think the answer is No , future AI Technology will certainly be like hydropower and coal , Become the most basic infrastructure , We just need to be able to use it .

3. summary

In this paper , We introduce two new automatic testing technologies which are widely used , The purpose is to help you understand the future development trend of automated testing , So as to make better use of new technologies to improve our testing efficiency .

The future trend of automated testing is not just these two , There are also intelligent exploratory tests , Intelligent traversal test and intelligent verification . On the technology used in intelligent traversal , You can refer to DQN Introduction to .