Continuous testing | making testing more free: practicing automated execution of use cases in coding

The author of this article ： Cheng Shengcong - CODING The product manager

Automated testing is the foundation of continuous testing

stay DevOps In the context of high frequency delivery , Teams tend to fall between speed and quality “ A choice ” Plight ： To embrace demand change , Adopt a shorter lead time , Then frequent changes lead to more problems , So the development test is delayed , Finally, the test time is compressed 、 It's difficult to do adequate testing . Let's face it , How can the team improve the efficiency of test execution ？ The first thing you will think of is automated testing —— Replace repetitive manual testing with automated testing , Faster execution saves test time . Besides , Because of automation, the execution time is relatively fixed , Moreover, the preset test behavior of the program brings high consistency , Let the stability and repeatability of the test reach a high standard , It can be realized very well “ Rapid reproduction of software defects ” The goal of .

If we say that in the traditional waterfall mode with relatively sufficient test time , The biggest value of automated testing for regression testing scenarios is to save manpower cost , So in agile and DevOps Time , The greater value of automated testing lies in frequent verification and fast feedback . It can be said that the foundation of continuous testing practice is automated testing , Only automation is high enough , In order to meet the needs of continuous delivery of high-frequency version .

Automated testing strategy

Automated testing has important value , But it doesn't mean that we should devote ourselves to all kinds of automated testing . The purpose of automated testing is to verify that the given logic is as expected , In a scenario where requirements change frequently , The maintenance cost of automated code is huge . therefore , We need the right strategy to guide the implementation of Automation —— Pyramid model .

come from 《The Test Automation Pyramid: Your essential guide for test automation》

The automated test pyramid was first created by Mike Cohn On 2009 In 《Succeeding with Agile: Software Development using Scrum》 It is proposed that , At that time, the three layers from top to bottom were UI、Service and Unit. Then with the implementation of agile testing practice , The cognition gradually formed in the industry is that from top to bottom, they are User interface testing （UI Tests）、 Interface integration testing （API Tests）、 unit testing （Unit Tests）, Plus the manual exploratory test at the top , Further enrich the pyramid for testing （ Including automation and manual ）. This triangle of narrow top and wide bottom provides a visual guide for our automation investment in each layer ： The underlying unit tests are the most , Interface test center ,UI Least testing .

As the pyramid model shows , The lower level of unit testing / Interface testing is better than the upper level UI The advantages of testing are ： Because it's closer to production code, it's easier to write and locate defects in the code ; because The granularity of the test object is smaller 、 Rely less , So execution is more efficient ; Because the test object is more stable, the maintenance cost is lower and so on , Of course, the advantage of testing closer to the top is , Because it's more reflective of business needs , So it's easier to see the value of testing . So in DevOps Time , Based on the balance of speed and mass , The interface integration test of the middle layer can keep relatively low maintenance cost , At the same time, it can reflect the value of business logic , It should be our focus , Especially when automation is still in its infancy .

The test pyramid originated in agile practice , Take it as a reference to continuously adjust our automation test investment , The team's test cases and execution status will gradually form a good balance .

The value of precision testing

Although it can be seen from the survey reports of the industry in recent years , As for DevOps Recognition , The investment of enterprises in automated testing continues to increase , The direct result is that there are more and more code for automated testing . But with a rapidly increasing number of automation code , Can automation achieve what we expect ？

In terms of the actual effect , The enterprise did not get the expected value due to the improvement of automation test coverage , Because automated code execution is not as much as we think “ free ”, Often for two reasons ：

1. Most teams think of automated code execution as CI A part of , It's just used as a regression scenario , But the time of full regression is too long, which limits the frequency of implementation ;
2. There is still a technical threshold for building an assembly line and matching corresponding tools on this basis , Plus the amount of work required by the operations themselves , It also makes it difficult for everyone to perform automated tests at any time .

This leads to an increase in automation coverage , Instead, regression testing takes longer to execute , So the frequency of automated testing can only be reduced , Finally, the value of automated code is questioned . In fact, in addition to improving automation coverage , We need to change “ Each test execution covers as many cases as possible ” Idea ： We shouldn't because “ Don't worry ” And make the test set redundant , But needs Optimize test coverage based on business risk , In order to achieve a higher test input-output ratio in a limited range , Achieve accurate testing .

CODING Make test execution more free

To make test execution more “ free ”,CODING Build the ability of cloud automated execution , Expect to solve the problem of automated testing “ Last mile ” problem , So as to achieve ：

1. Each execution has the freedom to choose the sample set flexibly ;
2. Everyone has the freedom to perform tests .

Next , Let's see How to be in CODING Test management implementation “ free ” Run the test smoothly ：

1. First , stay CODING Automatic code registration in automatic use case library , Make sure the automation code already exists in the code hosting , Register the existing automated code base , And set the relevant language / frame .

2. Parse the test function list of the automation code base , And establish the matching relationship between function use cases and automation functions in use case management , Get automation coverage . By matching automation code and function use cases , It can help us build an intuitive sense of the value generated by automated code , Achieve “ know ”.

3. Create test plans that are executed automatically , Circle the use cases . Selecting a suitable subset of automated tests requires business test knowledge , It takes too long to perform a fixed range of full regression tests , Or repeated mechanical implementation of smoke test can not reflect the test situation of new requirements in time , This is the reason why the expected value is still not achieved after the improvement of automatic test coverage .CODING Provides a way to circle test subsets on demand to create test plans , Precise execution of the relevant subset of automation code 、 Fast feedback results , This eliminates the worry about the long running time of automation , Maximize the value of the automated code that the team strives to produce .

4. Execute the test plan , The matching automation use cases are executed in the background and the execution results of the corresponding function use cases are updated . After the automation is completed , You can manually verify the untested or failed use cases 、 And update the use case task status .

5. Click generate test report , The system will automatically analyze and evaluate the quality according to the collected data . Test traceability makes test reports more convincing , Help the team control the risk to a low level .