Save: software analysis, verification and test platform

Abstract ： This paper will introduce a new software analysis tool CI The ecological system SAVE (Software Analysis Verification and Evaluation)

This article is shared from Huawei cloud community 《SAVE: Software analysis, verification and test platform 》, author ： Huawei cloud software analysis Lab .

A long-term practical problem that puzzles software analysis researchers is the lack of a unified test framework and test case set . This paper will introduce a new software analysis tool CI The ecological system SAVE (Software Analysis Verification and Evaluation).SAVE The purpose of is to provide a unified software analysis, verification and testing platform , It provides convenience for verifying and comparing software analysis tools .SAVE At present, it mainly has the following functions ：

• Provide a unified testing and verification framework , It is convenient to verify the functions of software analysis tools locally ;
• Provide a unified platform , Users can compare the effectiveness of different software analysis tools according to the common test suites that have been integrated or by uploading test suites themselves , Or evaluate its scope of application ;
• Provides REST API, Test verification can be integrated into the development process of software analysis tools （ Such as CI/CD）.

Near future ,SAVE The community will also be supported to organize competitions related to software analysis technology .

One . motivation

An important task of our team is to develop software for developers 、 Static analysis tools to improve development efficiency and quality , Including defect inspection and repair 、 Open source component risk identification 、 Third party library conflict identification, etc . To verify the effectiveness of the tool , It is necessary to build a testing framework and platform for such tools , Watch the evolution of static analysis tools , Comprehensively evaluate the capability and efficiency of static analysis tools . The current situation is that each static analysis tool is building its own special test framework again and again .

We have investigated the testing frameworks of several open source static analysis tools , At present, the most widely used is LLVM Lit[1]. Other frameworks have limited capabilities 、 The open source community is not active enough 、 Narrow focus and other issues .LLVM Lit It's for LLVM Ecological tools （ Such as Clang、LLVM backend、CSA etc. ） Test framework for . But after research , We found that LLVM Lit Some disadvantages , There are mainly ：

• Lit By some Python Script composition , It is difficult to implement at any time without installation ;
• Lit It was designed for the compiler , And the extensibility of application logic is not considered ;
• Lit Created by community system developers , It's just a simple one CLI Tools , There is no ecosystem

Besides , Most developers of defect checking tools face similar problems ： How to evaluate the effectiveness of the tool ？ The currently recognized test case set is NIST[2]. but NIST No test platform is provided , So that developers can more intuitively see the evolution of the tool and how it compares with similar tools . Besides ,NIST The test cases in are also mainly artificially constructed , It is difficult to evaluate the actual effect of inspection tools .

Change the current situation 、 The idea of providing convenience for software analysis practitioners , We developed SAVE frame （SAVE-cli[3]） And the service （SAVE-cloud[4]）. Developers of software analysis tools do not have to repeat the development of test frameworks 、 The existing test case set can be reused to test the effectiveness of the tool .

actually , The software analysis tool test scenarios are similar , It makes it possible to build a unified testing framework . Common test scenarios are as follows ：

• Pass the source code in the form of string to the analysis tool ;
• The analysis tool executes the internal analysis logic in the form of a black box ;
• Analysis tool output ： The middle of the code indicates as AST、 Static check alarm 、 Fixed code 、 Code execution results, etc . All output data can be represented in string form .

  

Two . SAVE-cli

2.1 Command line application

SAVE-cli Is a simple local command line application , It can be used to test the functions of analysis tools .SAVE It's a unified framework , Not dependent on a particular ecosystem （ Such as JVM、Python）. Users just need to follow SAVE Format to write test cases , There is no need to think about anything else .

We also provide a plug-in like interface to make SAVE-cli With scalability . Everyone can create their own plug-ins . Now we have Fix and Warn Two plug-ins .

2.2 The plugin is introduced

Fix plug-in unit [5]. The plug-in runs the tool on the original test file , And compare the output with the expected results . We use what we build diff library[6] To complete the comparison function .

Warn plug-in unit [7]. The interesting thing about the plug-in is that it provides an easy to read and user-friendly DSL To verify common static analysis tool alarms . It can be used in the source code of test cases DSL To describe the desired alarm information . The DSL Support regular 、 Multi line alarm . Here is a concrete example .

2.3 Resource detection mechanism

SAVE-cli The recursive resource detection mechanism is implemented . In order to make SAVE Can detect the test suite , Users need to configure save.toml file , Include the name of the test suite , Describe information, etc . And put the file in the directory where the test cases are stored .SAVE Support multi-level configuration , The configuration of the lower layer will inherit the configuration of the upper layer . for example , For the following directory levels , Catalog B Medium save.toml Will automatically inherit the directory A Configuration of .

|  A
   |  save.toml
   |  B
      |  save.toml

SAVE Follow a test suite save.toml Rules for configuration files . In other words , Only those that contain save.toml The directory of the file will be SAVE Treat as a test case .SAVE Will automatically detect the contents with Test The final file will be used as a test resource , And use... In the same directory save.toml File as test configuration .

|  A
   |  save.toml
   |  B               <<< test suite
      |  myTest.java   <<< test resource
      |  save.toml

3、 ... and . SAVE-cloud

3.1 What's new ？

There are many test cases for large projects , for example gcc and clang There are more than 50 Million test cases . Imagine , In this case ,CI How slow the processing will be ？ Consider that these test cases are often independent of each other , Using the batch concurrent processing mechanism of cloud computing can effectively improve the testing efficiency .SAVE-cloud It is such a testing framework based on cloud computing , It includes the following features ：

1. Unified testing framework ;
2. cloud CI Platform to perform test tasks ;
3. Visual Kanban of test results ;
4. Store historical test results ;
5. Support regression testing ;
6. Can be used to validate or compare analysis tools Benchmark aggregate

3.2 SAVE-cloud Workflow

SAVE-cloud utilize SAVE-cli Tests validate software analysis tools and store historical results . The workflow is as follows ：

• Users can use SAVE-cloud Provided REST API perhaps WEB Interface ;
• Users can select existing benchmark Or upload your own benchmark;
• Trigger processing flow , Start a docker Containers , call SAVE-cli To perform tests ;
• Store all historical execution results in the database , Easy to analyze

The flow chart is shown in the figure below ：

3.3 SAVE-cloud How to use it? ？

SAVE-cloud Already deployed on site https://saveourtool.com.

1. Users can click “Sign in”, At this point through github Authorized login . If you haven't logged in before , SAVE A user will be automatically created . Of course , Users can also click “Continue” Visit as a tourist .

2. Create or select an organization .

3. Create a new test project . You can use private git warehouse . Fill in the basic information , Including the organization 、 Name of the tested tool 、 Website of the tested tool 、 Of the test suite git Address 、 Description information . To protect privacy ,SAVE use Token To access git Warehouse . Users can also choose whether to expose the test project , If it's not public , This item will not appear on the Kanban , No one else can see .

4. After the project is created , You can upload binary files and other related files of the tool under test （ Such as auxiliary script ）. It should be noted that , The execution command of the tool under test needs to be configured in toml In file , See configuration item “execCmd”. Last , Click on “Test the tool now” To perform tests .

5. The results of each execution are recorded . Users can see the historical execution results on the kanban , You can also view the detailed results of each execution .

3.4 SAVE API

SAVE-cloud Also provided REST API, It can be easily integrated into various CI/CD platform , Include GitHub Action、Jenkins、TeamCity etc. . For details, please refer to the documentation [8].

Four . summary

We hope to have a unified benchmark framework and platform in the field of static analysis in the future .SAVE-cli and SAVE-cloud A step in this direction , It provides convenience for researchers in the field of software analysis to test and verify their analysis tools . We believe that SAVE There is still a lot to improve , There is still a lot of work to do , Welcome to try and provide suggestions . at present SAVE All components of are open source , We sincerely invite you to join us in making contributions ！

PaaS Technological innovation Lab Home link ：PaaS Technological innovation Lab- Hua Wei Yun

reference ：

1. LLVM Lit: https://llvm.org/docs/CommandGuide/lit.html
2. NIST Juliet Test Suites: https://samate.nist.gov/SARD/test-suites
3. SAVE-cli: https://github.com/saveourtool/save-cli
4. SAVE-cloud: https://github.com/saveourtool/save-cloud
5. SAVE fix plugin: https://github.com/saveourtool/save-cli/blob/main/save-plugins/fix-plugin/README.md
6. Diff library: https://github.com/petertrr/kotlin-multiplatform-diff
7. SAVE warn plugin: https://github.com/saveourtool/save-cli/blob/main/save-plugins/warn-plugin/README.md
8. SAVE API：https://github.com/saveourtool/save-cloud/blob/master/save-backend/Backend-API.md

Click to follow , The first time to learn about Huawei's new cloud technology ~