Optimal Combinatorial Testing with Constraints: The Balancing Act

Imagine that you are in front of a cockpit with several on-off buttons.

If you were to thoroughly test it, you would need to try a prohibitive number of configurations.

But since most bugs in practice can be isolated to interactions among few components, having tests that cover every possible pairwise configuration is a good start.

However, this is a problem that goes from easy to NP-hard as soon as some pairwise configurations are forbidden.

In this paper, we revisit unconstrained combinatorial testing with pairwise coverage on binary parameters and contrast it with the constrained case, showing and conjecturing properties that either are upheld or change from one to the other.

In particular, we discuss the extent to which it remains a good idea -- and sometimes indeed optimal -- to have every button almost as many times on as off to minimize testing.

We propose the first exact algorithm based on integer programming and a faster heuristic that often produces optimal solutions, both outperforming or competitive with their baselines.