Today's Episode of Testing with AI

One of the good things in the AI era is that more and more code is now actually being tested.

AI writes code, AI writes tests, everyone lived happily ever after.

Or maybe not...

I needed to implement the SHA256 algorithm, and naturally such algorithms need to be well-tested. And nowadays, the fastest way to get some test vectors is to ask AI. And this is what it gave me:

Input	SHA-256 digest (hex)
(empty string)	e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
abc	ba7816bf8f01cfea414140de5dae2223b00361a396177a9cb410ff61f20015ad
The quick brown fox jumps over the lazy dog	d7a8fbb307d7809469ca9abcb0082e4f8d5651e46d3cdb762d02d0bf37c9e592
The quick brown fox jumps over the lazy dog.	ef537f25c895bfa782526529a9b63d97aa631564d5d789c2b765448c8635fb6c
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa (63 × 'a')	7d3e74a05d7db15bce4ad9ec0658ea98e3f06eeecf16b4c6fff2da457ddc2f34
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa (64 × 'a')	ffe054fe7ae0cb6dc65c3af9b61d5209f439851db43d0ba5997337df154668eb
...	...

I am not mentioning the exact AI used, because I asked several models (local, free, paid) the same question, and the responses were quite consistent across the board.

Now the problem with the above set of test cases (I haven't included them all, as they were mostly variants of using different amounts of the letter 'a', or different sentences similar to the one shown above), is not that AI made mistakes in the hash digest — calculation of those was delegated by AI to a known good implementation like Python's hashlib, but the fact that after giving a few reasonable test cases (like an empty string, 'abc', some sentence with slight variations to show drastic differences in the result), it resorted to repeating a single character.

My response to those was: "Wouldn't it be better to have different characters and phrases used for longer tests, instead of merely repeating the letter 'a'? What if the code has an error and copies the wrong part of the buffer? If it is all 'a', it might not reveal the bug."

---

Yes, you're absolutely right.

Using only repeated identical characters (like all 'a's) is a weak test for buffer-handling bugs, off-by-one errors, incorrect length calculations, or partial copies. Varied content with mixed case, digits, punctuation, and real phrases is much better for catching those issues.

Here are improved test vectors with diverse characters:

Length 63

Input:
ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789!

SHA-256: aaa7932dcc5db1e35047bcd1bb857f85c23fc647aebd08290b626b797a336e24

Length 64

Input:
ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789!@

SHA-256: ee38b305103d0e3afb8c0162efbff13e6731bd6426130221c5324104edea2b50

...

---

Of course, I am right. Producing appropriate test cases for any kind of code requires more consideration than just letting an AI invent some and being satisfied with that.

Now, you might think that for a hashing algorithm, a mere few test cases could be enough to determine whether it works properly or not. From there, you'd decide the initial four were more than enough to catch bugs in the SHA256 implementation. But if you take a closer look at how the algorithm works, you will find that it is done on 512-bit chunks of input data. So the first four test cases won't test the loop where data is being split into those chunks before being processed.

It is not that such poorly defined test cases cannot catch bugs. They certainly can, but that can also depend greatly on the nature of the bug. The problem is not in the possible bugs they can catch, but in the possible bugs they might miss.

The whole point of testing is not only to show that the bare minimum functionality works as intended, but to minimize the amount of predictable, and therefore preventable, bugs releasing to production. It is about figuring out what could possibly go wrong, and making sure that it doesn't happen.

It is about test quality, not quantity. A small, but carefully selected collection of test cases might be much better than a much larger collection of poorly chosen ones. AI likes to add too much cruft when writing tests, and if you are not careful, the sheer amount of test cases produced can easily lull you into a false sense of security.