Image classification, AI and automatic performance test

Original by williamfzc Published in TesterHome Community , Click on [ Link to the original text ] Go directly to the original post and communicate with the author online .

Project address ：https://github.com/williamfzc/stagesepx
Official documents ：https://williamfzc.github.io/stagesepx/#/
Directions for use ：https://github.com/williamfzc/work_with_stagesepx

Preface

Before that, I have shared some application practices of image recognition in the testing field , Both functional test and performance test are involved . Some time ago, I wrote So that everyone can use image recognition to do UI automation And Based on image recognition UI Automation solutions after , As he gradually stabilized , The personal goal of functional testing has finally been basically completed .

In the direction of performance testing , Recently, many students have been paying close attention to the application of this technology in performance testing . These are the two versions that have been made successively before ：

• Using image recognition and OCR Conduct speed test
• Again, image recognition and OCR Perform performance testing

Although later versions have been basically available （ Later, with the iteration , Efficiency has become too low , It has become less usable again ）, But it always feels , This is not an ideal version .

After this time , I finally made a plan that was more in line with my expectations ： stagesep-x.

Why open a new pit

Compared with the previous version , Its principle is completely different , The usage scenarios are not completely consistent . So I chose to start another project instead of continuing the iteration .

stagesepx What can be done

In software engineering , Video is a kind of universal UI（ The phenomenon ） Describe the method . It can record what the user has done , And what happened to the interface . for example , The following example describes opening from the desktop chrome Get into amazon The process of the home page ：

stagesepx can Automatic detection And extract the stable or unstable stages in the video （ In the example ,stagesepx Think that the video contains three stable stages , Before clicking 、 When clicking and after the page is loaded ）：

then , Automatically get the time interval corresponding to each stage ：

for example , As you can see from the diagram ：

• The video starts until 0.76s At the stage of 0
• stay 0.76s Time from phase 0 Switch to phase 1
• stay 0.92s Time from phase 1 Switch to phase 0, Then enter the changing state （ When stagesepx Frames cannot be divided into a specific category 、 Or when the frame is not within the range to be analyzed , Will be marked as -1, It usually appears when the page changes ）
• stay 1.16s Reach the stage 2
• …

And so on , We can make a very detailed evaluation of each stage of the video . By watching the video, you can also find , The recognition effect is completely consistent with the actual situation .

At run time ,stagesepx The powerful snapshot function makes it easy for you to know what happened in each stage ：

All you need is a video , No pre template is required 、 No need to learn in advance .

Application, for example,

all stagesepx All you need is a video , And it is essentially only related to video , There are no specific usage scenarios ！ therefore , You can use your imagination , Use it to help you achieve more functions .

APP

• The application startup speed calculation mentioned above
• So in the same way , Page switching speed and other aspects can be applied
• In addition to performance , You can use the cutter to cut the video , Using, for example findit And other image recognition schemes to verify the functionality
• In addition to the application , The game, which cannot be tested by traditional methods, is its home court
• …

except APP？

• Except for mobile terminal , Of course PC、 Web pages can also calculate the results in the same way
• Even any video ？
  

Do whatever you want:)

Use

install

Python >= 3.6

pip install stagesepx 

Example

• Quick start
• Full tutorial

Want more features ？

Of course ,stagesepx More Than This . But before we begin the following reading , You need to understand Cutter （cutter） And classifier （classifier）.stagesepx It mainly consists of these two concepts .

Cutter

seeing the name of a thing one thinks of its function , The function of the cutter is to cut a video into multiple parts according to certain rules . He is responsible for video stage division and sampling , As a data collector for other tools （ for example AI Model ） Provide automated data support . It should provide friendly interfaces or other forms for external （ Including classifiers ） Provide support . for example ,pick_and_save Method is simply to enable data to be directly keras Designed for use .

The positioning of the cutter is a pretreatment , Reduce the operation cost and repeatability of other modules . After we get the stable interval , We can see that there are several stable stages in the video 、 Extract the frame corresponding to the stable phase, etc . On this basis , You can easily sample pictures of phases （ In the example, for each stage 3 A picture , Altogether 3 A stable stage , Respectively called 0、1、2） Keep it in the future , For his use （ for example AI Training 、 Function detection, etc ）：

classifier

For the example above , Classifiers came into being . It mainly loads （ stay AI The classifier may be learning ） Some classified pictures , And the frame （ picture ） To classify .

for example , After loading the frame corresponding to the stabilization phase in the above example , The classifier can classify the video at the frame level , Get the exact time-consuming of each stage .

The location of the classifier is to perform frame level classification on the video 、 High accuracy image classification , And be able to use the sampling results . It should have different forms of existence （ For example, machine learning model ）、 To achieve different classification effects . for example , You can use the sampled data in the previous videos to train your AI Model , When it converges, you can directly use the trained model for classification in your future analysis , There is no need for the pre sampling process .stagesep2 It is essentially a classifier .

Classifiers of different forms

stagesepx Two different types of classifiers are provided , Used to process the results after cutting ：

• Conventional SSIM The classifier needs no training and is lightweight , More for stages less 、 Simpler video ;
• SVM + HoG The classifier performs well on videos with complex stages , You can train it with different videos and gradually improve its recognition effect , Make it enough to be used in the production environment ;

Currently based CNN The classifier of has been preliminarily completed , It will be added after stabilization ：） But for now , The application of the first two classifiers in shorter videos is enough （ You may need to tune , But in principle, it is enough ）.

in fact ,stagesepx Encourage developers in design According to their actual needs Design and use your own classifier , To achieve the best results .

Rich charts

It takes time to get ？stagesepx It has been calculated for you ：

The snapshot function can let you know the situation of each stage intuitively ：

…

Excellent performance

In terms of efficiency , Absorbed stagesep2 Lesson （ He is really slow , This makes it difficult to be used in the production environment ）, During the project planning period, we will increase the priority of performance . For this video , You can see from the log , Its time-consuming is amazing 300 Millisecond or so （windows7 i7-6700 3.4GHz 16G）：

2019-07-17 10:52:03.429 | INFO     | stagesepx.cutter:cut:200 - start cutting: test.mp4
...
2019-07-17 10:52:03.792 | INFO     | stagesepx.cutter:cut:203 - cut finished: test.mp4

In addition to the conventional optimization methods based on the image itself ,stagesepx The sampling mechanism is mainly used for performance optimization , It refers to the process of converting the continuous quantity in time domain or space domain into discrete quantity . Because of the high accuracy of the classifier , This mechanism is more commonly used in the cutter section , For accelerating the cutting process . Its optimization range in the amount of computation is very considerable , With 5 Take the step size of a frame as an example , It saves 80% Amount of computation .

Of course , There will be some errors between sampling and continuous calculation , If your video changes a lot or you want to have higher accuracy , You can also turn off sampling .

More stability

stagesep2 Another problem is , High requirements for video itself , The anti-interference ability is not strong . This is mainly the module it uses （template matching、OCR etc. ） As a result of , rotate 、 The resolution of the 、 Illumination will affect the recognition effect ; Because it relies heavily on pre - prepared template images , If the recording environment of the template picture is different from that of the video , It can easily lead to miscarriage of justice .

and SSIM Its anti-interference ability is relatively strong . If you use the default SSIM classifier , All data （ Training sets and test sets ） All from the same video , Ensure the consistency of the environment , Avoid different environments （ For example, rotation 、 light 、 Resolution, etc ） The impact , The occurrence of misjudgment is greatly reduced .

Bug Report

It is conceivable that , It is very difficult to consider all the scenarios , It is difficult to do this in the early stage of the project .

If you have any suggestions or problems, you can go through issue Give me feedback

Project address

https://github.com/williamfzc/stagesepx

Original by williamfzc Published in TesterHome Community , Click on [ Link to the original text ] Go directly to the original post and communicate with the author online .

Today's knowledge has been absorbed ！ Want to learn more about dry goods 、 Get to know quality industry leaders and industry elites ？
The 10th China Internet testing and Development Conference · Shenzhen , Get to know >>