Studies have shown that combinatorial testing (CT) can be effective for detecting faults in software systems. By focusing on the interactions between different factors of a system, CT shows its potential for detecting faults, especially those that can be revealed only by the specific combinations of values of multiple factors (multi-factor faults). These research questions remain in the context of industrial settings. In this paper, we present an empirical study of CT on five industrial systems with real faults.

The number of (distributed) features and hence the risk of undesired feature interaction within this distributed system rises significantly. Such distributed automotive features pose a huge challenge in terms of efficient testing. Bringing together Combinatorial Testing with Automated Feature-Interaction Testing reduces the testing effort for such features significantly.

All-pairs testing

The number of tests produced by ACTS or other covering array generators is proportional to vt log n, for v values per parameter, with n parameters, for a t-way covering array. Note that this is not the exact number of tests produced; the test set size is proportional to this value, i.e., the number of tests grows exponentially with the number of values, but only logarithmically with the number of parameters. This size is a characteristic of covering arrays, and holds for all covering array generating tools, not just ACTS.

Results showed that the pairwise tests detected “many bugs which are not detected by existing test methods based on predicates in the query”. Another intuitive tool for performing combinatorial testing is testcover.com where factors, values, and constraints are simply written in the editor, and test configurations are generated. This tool has an extremely fast and efficient algorithm and can generate about 15 test cases in 1 second.

Practical Combinatorial Testing

Computer scientists and mathematicians both work on algorithms to generate pairwise test suites. Numerous exist to generate such test suites as there is no efficient exact solution for every possible input and constraints scenarios. An early researcher in this area created a short one-hour Combinatorial Testing course that covers the theory of combinatorial testing (of which pairwise testing is a special case) and shows learners how to use a free tool from NIST to generate their own combinatorial test suites quickly.

Combinatorial Test Design is a technique that helps plan test data as input to the manual or automation test process, ensuring 100% test coverage based on the several combinations of test data inputs for a test suite. The mathematical concept of “Orthogonal Arrays” sits at the what is combinatorial testing base of the combinatorial test design. A mathematician professor called Taguchi coined the mathematical algorithm term, Orthogonal Arrays. It measures the degree to which combinations of input values have been covered in tests, which is a static property of the test set.

Multiple studies have found 4-way to 6-way combination coverage was able to detect all faults found with exhaustive testing. Thus we can refer to this type of testing as “effectively exhaustive” (within reason). Develops a method of applying combinatorial testing for use with SQL database query programs.

• Combinatorial testing tools are easy-to-use test case generators that allow to provision the input and constraints to the input parameter model and then generate the test configurations using the model.
• We are investigating a new test development method that aims to maximize the confidence to be achieved by combining Assurance Cases with High Throughput Testing (HTT).
• A model of dependencies between input parameters of NEWTRNX is created.
• The test method and input model described in this paper have immediate application to other systems that provide complex full text search.
• Another advantage of this tool is that it can use three different algorithms (FIPOG, FIPOG-F, FIPOG-F2) to generate the combinatorial object array.

In this case, you can feed the test data inputs to the CTD tool to help you generate a combination of the test data input interactions. This improved set of test data goes as an input to the manual test or automation test processes. Many researchers use the OA concept for their industrial experiments in many domains all over the world. Combinatorial methods can be applied to all types of software, but are especially effective where interactions between parameters are significant. The primary industry applications for ACTS are in database and e-commerce, aerospace, finance, telecommunications, industrial controls, and video game software, but we have users in probably every industry.

It is very unlikely that pairwise tests would detect this unusual case; we would need to test 3-way and 4-way combinations of values. What degree of interaction occurs in real failures in real systems? Surprisingly, this question had not been studied when NIST began investigating interaction failures in 1999. Results showed that across a variety of domains, all failures could be triggered by a maximum of 4-way to 6-way interactions. As shown in Figure 2, the detection rate (y axis) increased rapidly with interaction strength (the interaction level t in t-way combinations is often referred to as strength). Studies by other researchers have been consistent with these results.

In the first case, we select combinations of values of configurable parameters. For example, telecommunications software may be configured to work with different types of call (local, long distance, international), billing (caller, phone card, 800), access (ISDN, VOIP, PBX), and server for billing (Windows Server, Linux/MySQL, Oracle). But what if some failure is triggered only by a very unusual combination of 3, 4, or more sensor values?