Shortly about tests generated by LLMs

LLMs can write a lot of tests. But let’s not confuse a lot of tests with good tests.

For instance, a single well designed test could prevent a major failure that fifty auto-generated ones might miss (of course there the proportion is relative).

Too many tests can bring a couple of issues:

1. They increase the time to run your test suite. Even a fast test suite cannot keep up with the number of tests an LLM will generate by default.

2. They have to be maintained. If you use LLMs to write both tests and code, you still need to review the changes. The more tests you have, the more will change, and the more lines you must read in the diff.

3. They have to be understood. When debugging, tests help you understand current behavior, but the more tests you have, the more time you spend reading possibly duplicate documented behavior.

4. They add to the context. The more tests you have, the more an LLM needs to read. If you generate tests with LLMs and use them as is, you are feeding back what the LLM generated and only increasing the context.

Keep tests that matter: those that verify requirements, document specific behavior, or help replicate edge cases.