[C++] Wings - Let unit tests be generated intelligently and automatically

Unit testing is a very effective means to ensure software quality, whether it is from the perspective of the concept of early intervention in testing or from the characteristics of unit tests that can be verified at high speed without being affected by UI, so the test-driven development advocated by the industry, the test driver mentioned in this refers more to the unit test driver. However, the general development team still rarely executes unit tests systematically, and the test for application software is more performed by professional testing teams to perform black box tests. The biggest difficulty of unit testing is not that the input and output cannot be determined, after all, it is already determined in the module development stage, but that the writing of unit test cases will consume a lot of developer man-hours, and according to relevant statistics, the time of unit test cases will even far exceed the development time of the function itself. Here are a few of the most common reasons why development doesn't write unit tests:

●The requirements are always endless, and there are still functional requirements to be realized in the next stage, and there is no time to fill the unit

●There are too many unit tests to be supplemented, and there is no way to start, so I resist subjectively.

● Unit tests are difficult to write. On the one hand, the reason may be that the functional function implementation is not reasonable enough, and on the other hand, there are no (or unknown) useful unit test frameworks and mock frameworks.

● Unit tests are not included in the workload.

Secondly, the functional requirements are still unstable, and the cost performance of writing unit tests is not high. In other words, if the requirements change tomorrow, not only the functional code will be scrapped, but also the unit tests. If you don't write unit tests, then this part of the effort will not be in vain.

In fact, the root cause of the above points is that the unit test writing is too time-consuming, which eventually leads to the loss of power of the test-driven engine, causing the beautiful vision of test-driven development to be stalled in the real scenario, because it is too difficult and expensive to build the engine for this drive. The various "x" units on the market and unit testing frameworks only solve the problem of generating test-driven outer frames, without any use case logic and data generation capabilities based on deep program understanding. Therefore, it makes developers resistant in various development-related scenarios. The release of Wings (currently for C) solves one of the biggest problems for programmers, and has the potential to fundamentally change the status quo of unit testing, which will effectively alleviate the pressure of system-level black box testing and automated testing based on massive human resources.

The constraint test cases are automatically generated by programs, and the most critical underlying technology is complex parameter parsing technology. That is, it can arbitrarily define nested level recursive parsing at the compiler level for arbitrarily complex types. Without this breakthrough in this critical technology, the automatic test case generation system would either be commercially incapable or evolve to produce compliant test data with very low efficiency. For example, the famous fuzzing tool American Fuzzy Lop cannot identify the type of structure required by the user's program, and needs to evolve the search algorithm based on the outermost layer. The characteristics of the program are that the input at the interface level and the data requirements of an internal module are far away, and the external data is usually transformed layer by layer of complex transformation to become the data structure type required by the internal module, so the amount of computation and time required to explore from the outside layer will be unimaginable. Based on the American Fuzzy Lop, in order to be able to generate a legitimate SQL statement, the internal module of the program needs to be explored in days, far from being minutes or hours. Another constraint is that the inputs that each program can take over are carefully structured and compiled data with a large number of rules, and it is very unrealistic and extremely time-consuming to generate these data through random + exploratory methods. Therefore, it is not feasible to generate auto-generated use cases from the black box as well as the outermost input.

If the use case-driven is generated from the analysis of the internal structure of the software, it is necessary to have a deep understanding of the compilation structure of the software. A viable test case generation system should be based on the middle of the program (key entry point) as the most appropriate test entry point. The inputs of these modules have turned fuzzy inputs into highly structured parameters. As long as these complex structures can be identified, the complex data types can be degraded into simple data types step by step, and the parameter construction can be completed at the same time, the generation of driving use cases can be automatically completed.

Module-based testing, which can be classified as traditional unit testing, is the best way to find and contain defects in the R&D phase. However, due to the limitations of unit testing, a large number of drivers need to be developed, and the promotion and application in the industry are greatly limited. Of course, unit tests can also be executed after the system is integrated to avoid building virtual stub programs.

Nebulas Testing's Wings product, which was launched in the world for the first time a few days ago, is an intelligent and fully automated unit test case generation system, which has studied and solved the following difficulties, and is now shared with you.

(1) In-depth analysis of program parameters

Wings uses the underlying technology of the compiler to form module objects based on the input source file according to the function. The object contains the input parameters of the function, the return value type, and other information, which can be used by the driver function module and the test case module. Each file is a unit that performs in-depth analysis of each parameter of each function in it, and can achieve accurate parsing and decomposition for nested types, complex types, etc., explain complex types layer by layer as basic data types, and generate a description file (PSD) of parameter structure.

(2) Function drive automatic generation of modules

According to the format information of the PSD file, all driver functions of the source program under test are automatically generated, and the unit testing process no longer relies on developers to manually write test functions, but only needs to compile the generated driver functions and the source files under test together, and the test results can be executed and the test results can be viewed. The test driver automatically generates the program based on the PSD description, fully automatically builds all the parameters and necessary global variables that drive the test under test, and can generate a structured test driver according to the hierarchy of complex variables, which can save a lot of time in writing unit test cases.

(3) Automatic generation and management of test data

It is used to automatically generate test data, which corresponds to the information extracted by the test function and the data is stored in a JSON file with a certain hierarchical logical relationship. The data and the data type after decomposition and expansion correspond to each other. Users can arbitrarily marginalize these data according to business requirements, and use JSON files to display them in a structured and hierarchical manner, which is very clear. The test data includes the values of global variables and the parameter values when the function under test is called.

Wings provides a unit testing method for automatically generating driver functions, which mainly includes the following steps:

Figure 1: Unit test-driven build flow

The structure information of the program under test mainly includes the global variables and function information in the program, and the function information mainly includes the number of parameters, parameter types and return value types of the function. The most important thing is to extract the symbol information and type information for some complex types, and analyze them into basic data types layer by layer to complete the construction of global variables and function parameters.

The types of variables are generally divided into basic types, construction types, pointer types and null types. Wings uses the underlying compilation technology to handle different variable types in different ways.

(1) Basic types, such as unsigned int u_int=20, Wings will parse the name of the variable to u_int and the data type to unsigned int.

(2) Construction types, construction types are roughly divided into arrays, structs, commons, and enumeration types.

● Array type, such as intarray[2][3], array name is array, type int and length of 2D array, behavior 2, column 3.

●Structure type, for structs as arrays, struct linked lists, etc., different markers are divided.

(3) Pointer type, e.g. int **ptr = 0; , parses the pointer as a level 2 pointer of int type.

(4) Null type, which is resolved to be NULL.

(5) System types, such as File, size_t, etc., are marked as system types, and will be added to the template and assigned by the user.

(6) Function pointer type, analyze the return value type, parameter type and number of parameters of the function

For each compilation unit of the source program under test, the parsed function information is stored in the corresponding PSD structure, and the following source code examples are described:

   

In the above program, void StructTypeTest3(myy_struct mm_struct[2])The saved PSD structure is as follows:

The meanings of each node in the PSD file are as follows:

●StructTypeTest3 represents the function name, parmType0 represents the parameter type, and parmNum represents the number of parameters

●mm_struct represents the symbol of the function parameter, baseType1 represents the classification of types (basic data type, construction type, pointer type, null type), type represents specific types, including int, char, short, long, double, float, bool, and these types of unsigned types and other basic types, and there are some special types such as: ZOA_FUN type represents function type, StructureOrClassType represents the struct type, etc., and name represents the name of the struct, union, and enum type

●i_int represents the base type, which is the smallest assignment unit

●array_one represents the array type, RowSize represents the length of the array, and the array can be divided into one-dimensional arrays, two-dimensional arrays, etc

●point represents the pointer type, the pointer is divided into first-level pointer, second-level pointer, etc., and the general pointer is used as a function parameter as an array, so for the basic type of pointer, the dynamic allocation array method is used to assign values, and the user can modify the corresponding value file according to the needs.

● w represents the type of bit field, and bitfileld represents the number of digits

●functionPtr represents the function pointer type, which analyzes the parameter type, number of parameters, and return value information respectively

●Dem stands for consortium type

● dy represents the enum type, and value represents the value of the enum type

●file represents the structure type, SystemVar represents this variable belongs to the variable in the system header file, for this type of variable, Wings adds template variables to the template library, users can assign special values according to specific needs. For example, the File type is handled as follows:

Users can also add their own assignment methods. For system types, Wings can be distinguished from ordinary user-defined types, and when parsing to the built-in type of the system, it can stop recursive analysis downwards.

●g_int represents global variables, and globalType represents global variables

●next represents the linked list structure, and NodeType represents this structure as a linked list

●returnType represents the return value type of the function.

In the above paper, the structural information of global variables and functions is analyzed and extracted, and the following information is used to save in PSD to complete the overall generation of the driving framework of the source program under test.

Generation is mainly divided into the following aspects:

Ø Declaration of global variables

Ø Assignment operation of function parameters, according to the number of function parameters, assign values in turn

Ø The assignment of global variables is performed sequentially according to the number of global variables used by the analysis

Ø Call of the original function

Some points to note are as follows:

●During the driver generation process, some special functions, such as main functions, static functions, etc., are not processed temporarily because they cannot be accessed by the outside world.

● For each source file under test, a corresponding driver file is generated.

● Drive control is included in the Driver_main.cpp to automatically configure the number of tests of the function through macros

The driver function generated by the above source program is as follows:

● All variables are named before the name of the original variable, add _

●By obtaining the corresponding test data, the variables are assigned in turn

●For the built-in parameters of the system and the special parameters of the user, the assignment method is uniformly configured through the template method.

●Assign and call parameters to the function under test.

The following is a set of data generated in PSD format in Figure 3, each set of data is saved in JSON format, making it easier to see the hierarchical relationship of the data.

For each compilation unit, a set of test data files corresponding to all functions is generated by default, and the value generation can be modified by the number of configurations.

How to complete the generation of the driver framework, the following is a detailed explanation of the complete generation process of the open source program MySQL.

The following is the main interface diagram of Wings testing Mysql:

Click the File button to set the project directory of the source program under test. After the settings are completed, click the function operation, which mainly includes parameter parsing, driver generation, value file generation, and template addition. The following folders are generated for the analysis:

Among them, the parameter parsing module generates FunXml and GlobalXml, which store the function information and global variable information of each extracted compilation unit respectively.

The driver generation module will be generated Wings_Projects corresponding folder, which stores the driver files for each compilation unit

The value generation module stores the generated test data for each compilation unit.

The following figure shows the driver file structure information loaded by Mysql, and the navigation tree on the left is the generated driver file, which contains the functions of each compilation unit, as well as the parameters and global variables of the functions. Click on one of the compilation units to load the corresponding driver file and the corresponding value file.

The above is the driver file and value file corresponding to the overall generation of Mysql, and the driver file is described in detail in the following code.

● For each compilation unit, the reference of the global variable is by extern.

●The driver function is uniformly named as the Driver_XXX method, JSON is used as the way to obtain test data, and times represents the number of tests of a single function.

●For each parameter assignment operation, the parsed PSD storage format is used to assign values to each layer structure in turn.

The application of Wings is very simple, the following is a statistical index of the generated test data using Mysql code that can be compiled normally in Visual Studio 2015 as an example, the whole generation process does not require any manual intervention, only needs to formulate the path of the source code that needs to be generated and driven.

mysqlTest data

Below are the results obtained using the source code statistics tool, with over 4 million lines of valid unit test code generated by Wings fully automatically. What's even more interesting is that it can be seen that the cost of manual development of these codes is as high as 1,079 man-months, and the cost is as much as 10.79 million.

Wings has realized the first step of exploration by the program to automatically generate the program, the first version is currently released, interested developers can download it directly on the code cloud platform (https://gitee.com/teststars/wings_release), commercial licensing provides a one-month unlimited function experience period, you can quickly experience the magical power of Wings, Wings c language version supports multiple platforms, such as visual studio, vxworks, gcc, qt, etc. Wings is designed and developed by the Nebulas testing (www.teststar.cc) team, and interested developers can get in touch with the Nebulas testing team through Codecloud's interactive platform to contribute their design ideas and product usage feedback (for the excellent suggestions adopted, Nebulas can extend its free use period for at least three months). Wings has a strong, underlying gene to greatly improve software quality, and in the future, Wings will deeply optimize the readability of automatically written programs (closer to the writing level of good programmers) and support for the C++ language.