Static Data Trap. All Tests Passed. But Why is the Output Wrong?

Code coverage is not enough

Code coverage is one of those metrics which can easily give you a false sense of security. In real life, code achieving 100% code coverage is often not possible, but that is the least of the problems with using code coverage as a metric of how well-tested some code is.

There are some parts of the code and logic that code coverage tools simply cannot cover. Even if you have achieved 100% code coverage for some code, you might still have serious flaws in that code.

Every branch is taken, every line is green, all your tests are passing. There is nothing left to do. Right?

Wrong?

Probably one of the simplest examples that can show this problem is converting a decimal integer number to some other base:

const
  BaseChars = '0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZ';

function IntToBase(Value, Base: UInt32): string;
begin
  Result := '';
  while Value > 0 do
    begin
      Result := (BaseChars[1 + (Value mod Base)]) + Result;
      Value := Value div Base;
    end;
  if Result = '' then
    Result := '0';
end;

To achieve 100% code coverage for the above code, we need only two test cases:

Value	Base	Expected Result
0	16	0
205	16	CD

It is beyond obvious that a mere two test cases cannot possibly constitute a proper set of test cases for such a function. This is one of the reasons code coverage is an extremely weak measure for adequate testing. It can only tell us when we do have some omissions in our test suite, and show us code which was not tested at all, but it cannot tell us how good our set of test cases is, and whether we have adequately covered a particular functionality with tests.

The blind spot of traditional testing methodologies

Full code coverage commonly gives us less than white-box testing methodologies could. In this particular situation, test cases which give us 100% code coverage are almost the same we would get with white-box methodologies. The only missing, non-critical, test case is the one in which a while loop would only have a single iteration.

Adding that one would extend our test set to:

Value	Base	Expected Result
0	16	0
1	16	1
205	16	CD

To get a more exhaustive set of test cases we need to reach to the black-box testing methodologies, like equivalence partitioning and boundary value analysis.

For the Value parameter, those give us two partitions from which we can take our boundary values:

Value Partitition	Category
0	valid
1 .. 4294967295	valid

For the Base parameter, we get the following partitions:

Value Partitition	Category
0	invalid
1	invalid
2 .. 36	valid
37 .. 4294967295	invalid

Using those we can extend our initial test cases and get the following set of tests:

Value	Base	Expected Result
0	16	0
1	16	1
4294967295	16	FFFFFFFF
10	2	1010
35	36	Z
1	0	division by zero
1	1	infinite loop
36	37	range error

Above set of test cases exercise the functionality of the IntToBase function more thoroughly, and they show the function fails on basic input validation, if we pass an invalid base value. In the context of testing, the most problematic is the infinite loop, which will hang the test suite.

Based on the previous analysis we might add an additional range check for the Base parameter input. But we might well have written the code like that on purpose, because we don't expect that any code calling the function will use invalid input values, and for performance reasons, we don't want to introduce additional verification in the code at this level.

But the real problem with our minimal test suite lies elsewhere. It is hidden in the BaseChars constant. Yes, we used that constant when determining valid and invalid input for the Base parameter, but our test cases forgot to cover another, extremely important aspect. This constant has direct impact on the correctness of the integer-to-string conversion. Changes in the constant—for instance, a simple typo where some characters are swapped and are not in the proper order—will give us incorrect results for values which contain that wrong character in their converted output.

Our set of test cases can only catch those if the BaseChars has incorrect values for 0, 1, F, or Z characters. For all other 32 characters, we can change values in the constant and our tests would still pass.

Even if we had included an additional set of test cases, beyond the minimal set shown above, they will most likely not cover all characters defined in the constant, unless we have intentionally written them to do so.

If your code relies on static data, where a change in that data changes the functionality of the code, then even with 100% code coverage, and even with test cases provided by other testing methodologies, you might still have serious flaws lurking in that code.

Code coverage tools can only measure the execution of instructions, but they cannot measure whether all the data behind some declared constant has been exercised during testing.

Considering the impact lookup tables and other kinds of constants can have on code correctness, we need to make sure our unit tests fully cover such scenarios, too.

The only way to properly test code which relies on static data is to make an exhaustive set of test cases which will cover every piece of the data. If any part of the data changes, the tests must catch the change and fail.

This is especially important in the AI era, where writing code and tests has been often delegated to the AI. An AI can more easily introduce accidental changes in places a person would not touch. If AI-made changes are not properly monitored, you can easily end up with bugs in previously working code, which will not be caught in testing.

Even when you instruct AI to write an exhaustive suite of tests, it may still fail to properly detect a potential issue with static data. One of the problems with AI-written tests—even in scenarios where AI will actually try to exercise the whole static dataset—is that it will simply take a copy of the original data and use it for test verification. (Yes, people will do that kind of thing, too.)

If the original data had correct values, then such tests will be able to catch future changes and related issues, but if the original data was flawed, then those tests will be flawed, too.

Testing functionality which uses static data is one of those situations where more care is always needed, regardless of who actually writes the tests.