Use LLMs to generate either tests or code, but not both

When using LLMs in software development, it’s important to be aware that generating both tests and code simultaneously can lead to significant risks. Therefore, use LLMs to generate either tests or code, but not both. For complex problems or business logic, I recommend using LLMs to generate either tests or code, but not both.

Using LLM to generate both in the same session has two main risks:

1. Source of truth: When an LLM writes tests, it assumes the source code is correct and aims to confirm the code works as intended. This is even more real when the domain where you are trying to solve a problem that maybe is not so common or diverges from some common assumption in some specific way.
2. Confirmation bias: This can lead to confirmation bias and make it difficult to identify bugs, especially when reviewing large volumes of generated code.

A more effective approach is to use LLMs for either test case generation or code writing, but not both simultaneously.

When LLMs generate test cases, remember that we generally lack automated methods to verify their accuracy or ensure they assess the intended requirements.

You might want to use LLMs to generate both tests and code, but you should do that in stages: generate the first asset, review it, change it if needed and then generate the second asset. Human-in-the-loop review if essential to have maintain a good quality and lower the risk of failures.

To help LLMs generate effective tests that target critical areas, ensure the features you are building are thoroughly documented. Enhance the feature specification by including:

1. Examples relevant to the feature
2. A clear definition of the target user group
3. Decision tables or truth tables for any relevant conditions

The more detailed the specification, the better context the LLM receives, resulting in more relevant tests. However, avoid providing excessive context, as this may cause the LLM to overfit to narrow scenarios.