RSS Feed for Good Enough Testing

RSS Feed for Good Enough Testing Good Enough Testing - live practical workshops to get better at writing effective test cases https://goodenoughtesting.com/articles/feed Tests Verify Behavior, Types Verify Structure Working with LLMs to generate code has made me think deeply about the relationship between tests and types. Both matter for code correctness, but they provide fundamentally different kinds of confidence. <h2>Tests verify behavior</h2> Tests answer a straightforward question: Does the system behave as expected in these scenarios? They provide concrete evidence that specific cases work. When you run your test suite and see green, you know those particular scenarios pass. Tests also protect against regressions. When you refactor code or add new features, your existing tests catch breaks in functionality you already built. During development, they give you fast feedback about whether your changes work. <a href="https://booklet.goodenoughtesting.com/1/good-enough-testing-for-developers/1/what-is-testing-for-a-developer">I wrote about how to think about testing as developer</a> and what purpose does the tests have for your development process. But tests have clear limits. They only check what you thought to test. If you miss an edge case or make wrong assumptions about how the system should behave, your tests will pass while bugs remain. Passing tests do not mean there are no bugs. So they are as strong as your understanding of the product, target market, business goals, and user needs. Tests give you confidence through examples. They show that specific inputs produce expected outputs. <h2>Types verify structure</h2> Types answer a different question: Can this category of errors happen? They provide guarantees before running the code. A type checker validates your program structure without executing it. Types eliminate entire categories of invalid states. If your type system says you cannot pass a string where an integer is expected, that error simply cannot occur at runtime. They also make refactoring safer. When you change a function signature, the type checker finds every place that needs updating. But types also have limits. They express structure, not business logic. A function that takes two integers and returns an integer might be typed correctly but still implement the wrong calculation. Types cannot encode real-world correctness. They prevent structural mistakes but allow logical ones. Types give you confidence through constraints. They restrict what operations are possible on your data. <h2>The reality with LLMs</h2> When generating code with LLMs, this distinction becomes crucial. LLMs are surprisingly good at producing structurally valid code. Modern language models understand syntax, common patterns, and basic type requirements. The real problem is they often generate code that compiles and runs but does the wrong thing. They make wrong assumptions about your requirements. They miss edge cases you forgot to mention. They misunderstand the business logic you tried to describe. The code looks right and passes basic structural validation, but it implements the wrong behavior. This is where tests become essential. A test suite that captures what the code should actually do catches these subtle logic errors that types cannot see. Types help ensure the code is structurally sound, but tests validate that the LLM understood what you needed. <h2>What developers have proven</h2> Developers in dynamically-typed languages like Ruby have built massive, reliable systems with comprehensive tests and no static types for decades. These systems handle millions of users and billions of dollars in transactions. The pattern is consistent: Strong test coverage catches the errors that matter in production. Tests verify the actual behavior users depend on. When working with LLM-generated code, this pattern holds even more strongly. The generated code might satisfy type constraints easily, but only tests tell you if it solves your actual problem. <h2>Both matter, differently</h2> I am not arguing against types. In statically-typed languages, types catch whole classes of errors automatically. They make code easier to refactor and maintain. They provide valuable documentation about structure. But when evaluating LLM-generated code, tests provide the confidence you need most. They verify that the system does what you intended, not just that it satisfies structural requirements. The code might be perfectly typed and completely wrong. Tests catch that. The code might lack type annotations but work exactly as needed. Tests verify that too. Both tests and types improve code quality, but they do it through different mechanisms. Understanding this difference helps you work more effectively with LLMs to generate reliable code. 2026-01-17 https://goodenoughtesting.com/articles/testing-and-types-in-context-of-llms https://goodenoughtesting.com/articles/testing-and-types-in-context-of-llms A short review of study "How Many Instructions can LLMs Follow at Once?" While trying to determine how many instructions are enough in an Agents.md file, I found the following paper: <a href="https://arxiv.org/abs/2507.11538">Jaroslawicz, D., Whiting, B., Shah, P., & Maamari, K. (2025). How Many Instructions Can LLMs Follow at Once? arXiv:2507.11538</a> Before reading what I extracted from there I think it is important to understand the limitations defined by authors: <blockquote> Our study has several important limitations. We focus exclusively on professional report generation with simple keyword-inclusion instructions, which may not generalize to other task types or domains, or more complex instruction types. Our business vocabulary from SEC 10-K filings limits insights into other instruction formats common in real applications. Results are specific to English-language, business-domain instruction following, with cross-lingual performance and other paradigms requiring future investigation. </blockquote> I still think the study is relevant for developers when thinking about writing guidelines LLM agents or working with an agent and writing more complex prompts. <h2>Findings from the study “How Many Instructions can LLMs Follow at Once.”</h2> <h3>There is a bias toward earlier instructions and lower accuracy as the instruction count approaches 500.</h3> <blockquote> We evaluate 20 state-of-the-art models across seven major providers and find that even the best frontier models only achieve 68% accuracy at the max density of 500 instructions Our analysis reveals model size and reasoning capability to correlate with 3 distinct performance degradation patterns, bias towards earlier instructions, and distinct categories of instruction-following errors </blockquote> Be conservative with the total number of instructions. Even under controlled conditions, performance degrades as the number of instructions increases. Limit the context provided in AGENTS.md or CLAUDE.md files. One way to do this while still providing necessary information is to use progressive disclosure. <h3>Few instructions are better.</h3> <blockquote> Threshold decay: Performance remains stable until a threshold, then transitions to a different (steeper) degradation slope and displays increased variance. The top two models ( gemini-2.5-pro , o3 ) demonstrate this clearly, maintaining near-perfect performance through 150 or more instructions before declining. Notably, these are both reasoning models, indicating that deliberative processing architectures provide robust instruction tracking up to critical thresholds, beyond which systematic degradation occurs. </blockquote> The best reasoning models in this paper, such as gemini-2.5-pro and o3, show a performance decay pattern. They maintain performance up to about 150 or more instructions before a systematic decline, especially for simple, repetitive constraints. In the paper, they used a list of keywords for a business report, as tested in the IFScale benchmark. <h3>Models exhibit a primacy effect, meaning they are generally better at satisfying instructions that appear earlier in the list.</h3> <blockquote> Primacy effects refer to the tendency of models to better satisfy instructions appearing earlier versus later in the instruction list Primacy effects display an interesting pattern across all models: they start low at minimal instruction densities indicating almost no bias for earlier instructions, peak around 150-200 instructions, then level off or decrease at extreme densities. This mid-range peak suggests that models exhibit the most bias as they begin to struggle under cognitive load at moderate densities. </blockquote> One takeaway is to place the most critical instructions at the beginning of the prompt. This bias is stronger when the model is under moderate cognitive load, around 150 to 200 instructions. However, relying on ordering is less effective at extreme densities, above 300 instructions, where models are overwhelmed and fail uniformly. <h2>My takeaways</h2> After reading that study here are some takeaways I noted down mostly for me: 1. Be conservative with how many instructions you include in AGENTS.md/CLAUDE.md Long instruction lists degrade model performance. Aim for under 150 instructions if possible. If you reach 200+, expect accuracy drops. In case of prompting, avoid giant “superprompts.” 2. Put the most important rules first Models consistently show a primacy effect: they are better at following instructions that appear earlier in the list. Always put your most critical requirements first. Place safety rules and project-critical constraints in the first 10–20 items. 3. Use progressive disclosure Don’t dump everything into one big file and break guidelines into layers or steps. Practical ways to do this: <ul> <li>Split instructions by themes (coding, style, architecture).</li> <li>Send only the section relevant to the current task.</li> <li>Use “ask-if-needed” patterns instead of hard-coding everything upfront.</li> </ul> 4. Prefer fewer, better instructions Redundant, verbose, or repetitive rules increase instruction count and reduce accuracy, even though they degrade in effectiveness when the number of instructions is high. Simple, repetitive instructions hold up better than complex or varied ones, but even these degrade in effectiveness at high counts. 5. Use structured formats instead of long prose Models track clearer instructions when the structure is consistent. Examples: <ul> <li>Use bullet points instead of paragraphs.</li> <li>Use tables for dos/don’ts.</li> <li>Use “If X, then Y” rules.</li> <li>Use numbered steps</li> </ul> This improves recall in most normal cases. 6. Let the model summarize your instructions for internal use One way to reduce the number of rules is to ask the model to summarize or restate your instruction set in its own words. Make sure you read them and check if they maintain the same spirit or intention as the original prompt. 2025-12-04 https://goodenoughtesting.com/articles/how-many-instructions-are-enough https://goodenoughtesting.com/articles/how-many-instructions-are-enough Highlights from "Do LLMs Generate Test Oracles That Capture The Actual Or The Expected Program Behaviour?" There’s a lot of discussion about using LLMs to build applications, including generating test cases. It’s interesting to read people’s experiences and see how LLMs are affecting software development. Opinions on the usefulness of LLMs range from very positive to quite negative. To make sense of these different experiences, I decided to look at research studies about using LLMs for testing. I’ll focus on development in another article. I plan to share some highlights as I review different studies on how generative LLMs are used for testing. I’ll start with a 2024 study titled “Do LLMs generate test oracles that capture the actual or the expected program behaviour?” <pre> Konstantinou, Michael & Degiovanni, Renzo & Papadakis, Mike. (2024). Do LLMs generate test oracles that capture the actual or the expected program behaviour?. 10.48550/arXiv.2410.21136. </pre> Before I continue, I want to clarify two important points: <ol> <li>Test Oracle: “A test oracle is a mechanism that determines whether software executed correctly for a test case” (<a href="https://ieeexplore.ieee.org/document/1240304">source</a>) or “A source to determine an expected result” (<a href="https://glossary.istqb.org/en_US/term/test-oracle">source</a>)</li> <li>Content of the study: The paper’s results are based on tests with OpenAI GPT 3.5-turbo in 2024, so newer models may have improved. Still, I think the insights from this study are valuable for understanding AI-driven test generation.</li> </ol> I’ve picked out some key points from the study and will share my thoughts on each of them. <h3>LLMs Mirror Code, Not Intentions</h3> <blockquote> Interestingly, our results show that LLMs are more likely to generate test oracles that capture the actual program behaviour (what is actually implemented) rather than the expected one, i.e., the intended behaviour. Additionally, we find that the overall performance of the LLMs is relatively low (less than 50% accuracy) meaning that LLMs do not provide a strong oracle correctness signal. Therefore, all LLMs suggestions will need human inspection. </blockquote> This suggests that when LLMs are asked to generate test cases and assertions without clear requirements, they tend to follow the code itself. In my view, to use LLMs effectively for test generation, you need well-written requirements so the LLM can use them to create test oracles. Another situation is when your codebase lacks test cases and you want to add tests before refactoring. If you focus on the functional behavior, given certain inputs, the code should produce specific results, an LLM can help generate these kinds of test cases. <h3>LLMs can boost code coverage but require human help for correct behavior</h3> <blockquote> While effective at covering code (or killing mutants), automatic test generation falls short in finding faults, particularly business-logic related faults. This is because of the inherent inability of these techniques to compose test oracles (test assertions) that capture the expected program behaviour. This means that the fault detection ability of these techniques is limited to zero-level oracles, such program crashes or memory violations (when applied at system level). </blockquote> This supports the idea I mentioned earlier: you can use LLMs to increase test coverage and focus on documenting your code’s current functional behavior. <blockquote> To reveal business-logic software faults with test generation techniques one need to manually validate and correct, when needed, the generated tests and their respective oracles </blockquote> This is the approach I recommend (for now) in my workshops: when using LLMs to generate test cases, a human should always review them to ensure the tests check for the intended behavior, not just what the code currently does. <blockquote> Our experiments showed that the LLM’s accuracy to correctly classify a correct assertion as positive, drops when the given code is buggy. This suggest that the LLM is prone to follow the actual implementation to classify the test oracle rather than the expected behaviour. </blockquote> If your code has a bug that isn’t obvious business logic, the LLM will likely generate test cases that follow the bug, since it can’t guess your intentions without requirements. However, if you’re working with common patterns or business logic that LLMs have seen during training, they might spot issues. I’ve seen cases where LLMs, when asked to write tests, also reviewed the code and pointed out bugs that needed fixing. But if you’re working on something specific, the LLM may not be able to point out bugs that need to be fixed. If you’re writing code that’s specific to your domain and not just boilerplate, you can’t rely on LLMs to write good assertions. You’ll need to review what they generate. <h3>Meaningful names help LLMs in generating tests</h3> <blockquote> Taken together, our results corroborate the conclusion that unless having meaningful test or variable names LLMs can mainly be used to capture the actual program behaviour (thus to be used for regression testing). </blockquote> Although the paper focused on Java, I think the idea can apply to other languages too. For example, in Ruby, where developers often use meaningful names for variables and methods and pay attention to domain context, this should help LLMs generate better test oracles, according to the study. <h3>Two-Step Workflow</h3> In TDD, you usually have requirements before writing code. You can use a two-step workflow: first, give the requirements and ask the LLM to identify what’s testable, then ask it to write tests. This way, the requirements act as the test oracle, and the LLM is more likely to write good test assertions based on them instead of just the implementation. If you write the implementation first and then want to generate test cases, you can still use a two-step workflow. First, describe the purpose of your code and provide any other sources as a test oracle. Ask the LLM to review them and identify what should be tested, then give it the code and ask it to generate test cases. 2025-11-07 https://goodenoughtesting.com/articles/llms-test-generation-research-insights https://goodenoughtesting.com/articles/llms-test-generation-research-insights Use LLMs to generate either tests or code, but not both When using LLMs in software development, it’s important to be aware that generating both tests and code simultaneously can lead to significant risks. Therefore, use LLMs to generate either tests or code, but not both. For complex problems or business logic, I recommend using LLMs to generate either tests or code, but not both. Using LLM to generate both in the same session has two main risks: <ol> <li>Source of truth: When an LLM writes tests, it assumes the source code is correct and aims to confirm the code works as intended. This is even more real when the domain where you are trying to solve a problem that maybe is not so common or diverges from some common assumption in some specific way.</li> <li>Confirmation bias: This can lead to confirmation bias and make it difficult to identify bugs, especially when reviewing large volumes of generated code.</li> </ol> A more effective approach is to use LLMs for either test case generation or code writing, but not both simultaneously. When LLMs generate test cases, remember that we generally lack automated methods to verify their accuracy or ensure they assess the intended requirements. You might want to use LLMs to generate both tests and code, but you should do that in stages: generate the first asset, review it, change it if needed and then generate the second asset. Human-in-the-loop review if essential to have maintain a good quality and lower the risk of failures. To help LLMs generate effective tests that target critical areas, ensure the features you are building are thoroughly documented. Enhance the feature specification by including: <ol> <li>Examples relevant to the feature</li> <li>A clear definition of the target user group</li> <li>Decision tables or truth tables for any relevant conditions</li> </ol> The more detailed the specification, the better context the LLM receives, resulting in more relevant tests. However, avoid providing excessive context, as this may cause the LLM to overfit to narrow scenarios. 2025-11-04 https://goodenoughtesting.com/articles/use-llms-to-generate-test-or-code-but-not-both https://goodenoughtesting.com/articles/use-llms-to-generate-test-or-code-but-not-both Shortly about tests generated by LLMs LLMs can write a lot of tests. But let’s not confuse a lot of tests with good tests. For instance, a single well designed test could prevent a major failure that fifty auto-generated ones might miss (of course there the proportion is relative). Too many tests can bring a couple of issues: <ol> <li>They increase the time to run your test suite. Even a fast test suite cannot keep up with the number of tests an LLM will generate by default.</li> <li>They have to be maintained. If you use LLMs to write both tests and code, you still need to review the changes. The more tests you have, the more will change, and the more lines you must read in the diff.</li> <li>They have to be understood. When debugging, tests help you understand current behavior, but the more tests you have, the more time you spend reading possibly duplicate documented behavior.</li> <li>They add to the context. The more tests you have, the more an LLM needs to read. If you generate tests with LLMs and use them as is, you are feeding back what the LLM generated and only increasing the context.</li> </ol> Keep tests that matter: those that verify requirements, document specific behavior, or help replicate edge cases. 2025-10-27 https://goodenoughtesting.com/articles/on-tests-generated-by-llms https://goodenoughtesting.com/articles/on-tests-generated-by-llms Add comments to your database for humans and LLMs Add comments to your Rails migrations. This helps the next developer understand your intent and also helps an LLM follow your product decisions. Explaining why a column exists or how you chose its name turns your schema into a mini README. This small step helps both people and machines better understand your code. <h3>Example</h3> Here’s an example of a topics table that stores themes used for talks at different events. While the following example is from Ruby on Rails you should do this no matter the programming language or framework used. <pre class="ruby">class CreateTopics < ActiveRecord::Migration[8.0] def change create_table :topics, comment: "Reusable themes that group talks across events. Support discovery by theme." do |t| t.integer :canonical_id, comment: "References the canonical topic when duplicates are merged. Used for redirects and metric consolidation." t.string :name, limit: 80, comment: "Unique public name. Human readable" t.string :slug, null: false, default: "", comment: "Slug used in URLs and API paths. Auto generated from name. Changing this will break existing links." t.string :status, null: false, default: "pending", comment: "Moderation state. Moves from pending to approved to archived after review." t.boolean :published, default: false, comment: "Controls public visibility. Visible only if status is approved and this flag is true." t.text :description, comment: "Public summary. One to three sentences. Shown on topic pages and in search results." t.integer :talks_count, null: false, default: 0, comment: "Counter cache for associated talks. Updated automatically through the Talk model." t.datetime :created_at, null: false, comment: "Managed by Rails. Used for ordering and audit trails." t.datetime :updated_at, null: false, comment: "Managed by Rails. Updated when topic content changes." end end end </pre> In case you already have created your tables and run your migrations you can add comments in Ruby on Rails to your schema using <code>change_table_comment</code> and <code>change_column_comment</code> <pre class="ruby">class AddCommentsToSchema < ActiveRecord::Migration[6.1] def change change_table_comment :topics, "Reusable themes that group talks across events. Support discovery by theme" change_column_comment :topics, :name, "Unique public name. Human readable" change_column_comment :topics, :slug, "Slug used in URLs and API paths. Auto generated from name. Changing this will break existing links." end end </pre> <h3>Why some of these comments matter</h3> Let’s take a look at some of them and what is the intention behind those comments. <h4>Documenting the main table <code>topics</code></h4> <blockquote> Reusable themes that group talks across events. Support discovery by theme </blockquote> This comment explains the reason behind the table, not just its purpose. It helps newcomers see how this table fits into the product. <h4>Documenting the <code>canonical_id</code>:</h4> <blockquote> References the canonical topic when duplicates are merged. Used for redirects and metric consolidation. </blockquote> Without this note, a developer might assume it’s a foreign key to a different table or maybe a self-reference on the same table. But with the comment, it’s clear that it’s a self-reference used during merges. <h4>Documentation on <code>slug</code></h4> <blockquote> Slug used in URLs and API paths. Auto generated from name. Changing this will break existing links. </blockquote> This comment makes it clear that changing the slug could break links. It also lets you know that slugs usually aren’t changed after publishing. <h4>Documentation on <code>status</code> and <code>published</code>:</h4> For the status: <blockquote> Moderation state. Moves from pending to approved to archived after review </blockquote> It explains what is the purpose of the status (“moderation”) and how it changes over time with some examples without going full into the enum for that status. For the published field: <blockquote> Controls public visibility. Visible only if status is approved and this flag is true </blockquote> It explains what is the purpose of the published field (“visibility”) and how it should be considered together with <code>status</code>. These fields control visibility. The comment explains that a topic needs to be both approved and published to show up, which helps future developers avoid mistakes. <h3>Writing these comments well takes practice</h3> You might not get every comment perfect the first time. Some are tricky because they cover both business and technical details. Some others are tricky if you, like me, are not a native English speaker. That’s okay. The goal isn’t perfect comments, it’s to reduce confusion and make your intent clear. Even a short note like ‘used for redirects’ or ‘set by background job’ is better than nothing. Each comment you add now means fewer mysteries for you or others later. Start with small comments and improve them over time. Your future self, your teammates, and even your LLMs will appreciate it. 2025-10-25 https://goodenoughtesting.com/articles/comment-your-database-for-humans-and-llms https://goodenoughtesting.com/articles/comment-your-database-for-humans-and-llms Tests as the line of defense for AI generated code Being good at testing is a skill developers should sharpen even more when using LLMs to generate code. Especially if you: <ol> <li>Use the LLM in agentic mode.</li> <li>Use LLMs to solve hard problems or implement complicated business logic.</li> </ol> Your test cases are the best line of defense to ensure the generated code works as intended. Good tests should be fast, isolated, and intention-revealing. They should quickly identify faults in the code without dependencies that can skew results and clearly communicate what they are testing. Here’s a brief checklist to determine if your tests are effective in defending against faulty LLM outputs: <ol> <li>Is the test fast enough to run frequently?</li> <li>Does it function independently without impacting or being affected by other tests?</li> <li>Does it clearly state the intention of what it is verifying?</li> <li>Does it cover critical business requirements or edge cases?</li> <li>Is it easy to understand and maintain over the long term?</li> </ol> Always review the code generated by an LLM. While writing good test cases is essential, it does not replace the need for a thorough code review. Targeted tests can effectively guide your LLMs to create better solutions with fewer chances of dangerous bugs. In the end, you still have to assume and own the solution created with LLMs. 2025-10-21 https://goodenoughtesting.com/articles/tests-as-line-of-defense-when-using-llms https://goodenoughtesting.com/articles/tests-as-line-of-defense-when-using-llms Resources to use when you want to test strings <h2>Resources useful for testing strings</h2> In case you really need in your test suite to handle real strings of various types, here are some resources you can use: <h4>The Big List of Naughty Strings:</h4> <a href="https://github.com/minimaxir/big-list-of-naughty-strings">https://github.com/minimaxir/big-list-of-naughty-strings</a> <blockquote> The Big List of Naughty Strings is an evolving list of strings which have a high probability of causing issues when used as user-input data. </blockquote> <h4>Faker gem:</h4> <a href="https://github.com/faker-ruby/faker">https://github.com/faker-ruby/faker</a> <blockquote> A library for generating fake data such as names, addresses, and phone numbers. </blockquote> <h4>SecLists Username and Passwords:</h4> SecList sections about Username and Passwords if you want to test authentication: <ul> <li>Passwords: <a href="https://github.com/danielmiessler/SecLists/tree/master/Passwords">https://github.com/danielmiessler/SecLists/tree/master/Passwords</a> -> “The Passwords directory will hold a number of password lists that can be used by multiple tools when attempting to guess credentials for a given targeted service”</li> <li>Usernames: <a href="https://github.com/danielmiessler/SecLists/tree/master/Usernames">https://github.com/danielmiessler/SecLists/tree/master/Usernames</a> -> “The Usernames directory will hold a number of username lists that can be used by multiple tools when attempting to guess credentials for a given targeted service”</li> </ul> <h4>List of Dirty, Naughty, Obscene, and Otherwise Bad Words</h4> This is an old list of dirty, naughty, obscene, and otherwise bad words that can be used to test your application’s ability to handle such content. <a href="https://github.com/LDNOOBW/List-of-Dirty-Naughty-Obscene-and-Otherwise-Bad-Words">https://github.com/LDNOOBW/List-of-Dirty-Naughty-Obscene-and-Otherwise-Bad-Words</a> <h4>A list of hostnames and usernames to reserve</h4> <a href="https://ldpreload.com/blog/names-to-reserve">https://ldpreload.com/blog/names-to-reserve</a> I tried to compile what is in that article and make simple .txt files <a href="https://github.com/lucianghinda/list-of-usernames-hostnames-and-emails-to-reserve">in this repo</a> where I added a bit more. <h4>UI Testing Checklist</h4> Some ideas about Strings and other formats to use when submitting UI forms: <a href="https://www.scribd.com/document/507622474/UI-Testing-Checklist">https://www.scribd.com/document/507622474/UI-Testing-Checklist</a> 2025-10-13 https://goodenoughtesting.com/articles/resources-for-developers/strings https://goodenoughtesting.com/articles/resources-for-developers/strings Good Tests Begin with Good Samples Make your future easier by spending time today to properly set up test data. The time you spend today setting up test data that matches real and important user cases is an investment. It will help your daily work but it will be a great help during a priority zero incident, and you will be glad you invested that time early. To achieve this, understand your users, your product, and your business. Work closely with your product, support, and sales teams, then apply sampling. Choosing good test data is about sampling well. Borrowing ideas from statistics helps us make smarter, more deliberate choices. <h2>1. Population vs. Sample</h2> Think of your code’s input space as a population: every possible combination of data and state. Your tests are a sample of that population. The goal isn’t to test everything, but to test a representative subset that helps you discover likely problems early. <h2>2. Random vs. Systematic Sampling</h2> You can: <ul> <li>Pick random inputs (great for fuzz testing or probabilistic code), or</li> <li>Pick systematic samples, one from each meaningful category or boundary.</li> </ul> For most business logic, structured sampling works best. Example: If your API accepts ages 0 to 120, you don’t need 100 random ages. You need a few key samples: <ul> <li>0 (lower boundary)</li> <li>1 (typical valid)</li> <li>119 and 120 (upper boundaries)</li> <li>121 (invalid)</li> </ul> This approach is actually the folundation for boundary value analysis testing technique. <h2>3. Stratified Sampling: Covering All Groups</h2> Not all inputs are equal. Divide your input space into logical groups and pick samples from each one. Example: <ul> <li>For a login form, you might stratify by “valid credentials,” “invalid password,” “nonexistent user,” “empty input.” Then pick one or two cases from each.</li> </ul> This prevents over-testing one area and missing others. In testing terms: that’s a part of the equivalence partitioning testing technique. <h2>4. Sampling Bias: The Trap of “Happy Paths”</h2> Most developers unintentionally sample the same region, sometimes called the happy path. The result is biased tests that miss the tricky edge cases where bugs often hide. Ask yourself: <blockquote> Are my test cases representative of real-world usage and rare conditions? </blockquote> If 90% of your users are on mobile Safari, don’t just test on desktop Chrome. If your code processes files, test both small and large ones, and consider different file extensions. See the stratified sampling. <h2>5. How Many Samples Are Enough?</h2> Adding more test data isn’t always better. Beyond a certain point, new samples might reveal new information, but adding more to an already covered space increases testing costs. Think of sampling like this: <ul> <li>Start broad: cover each risk area once.</li> <li>Go deeper only where you see instability or recent changes.</li> </ul> This is called risk-based coverage: you focus your effort where it matters most and avoid unnecessary work. <h2>6. Always Include Extremes and Weirdness</h2> Include a few “troublemakers”: <ul> <li>Empty strings</li> <li>Nulls</li> <li>Maximum lengths</li> <li>Invalid characters</li> <li>Very large numbers</li> </ul> These aren’t just edge cases. They’re bug magnets. <h2>7. Realistic vs. Synthetic Data</h2> <ul> <li>Synthetic data helps you test logic cleanly and deterministically.</li> <li>Realistic data reveals integration issues, performance surprises, and messy real-world inputs.</li> </ul> You need both. For example, your happy path tests can use synthetic data, but your end-to-end suite should include a realistic snapshot of production-like data. <h2>8. Adaptive Sampling</h2> You don’t have to get it right from the start. Start small. When a bug appears, expand your sample around that area. Think of it as zooming in with a magnifying glass to spot where problems start to appear. <h2>9. Things to keep in mind</h2> Good test data isn’t random. Good test data is chosen carefully, based on structure, boundaries, and risk. Next time you write a test, ask: <ul> <li>What’s the space input here?</li> <li>Which categories or boundaries matter most?</li> <li>Am I sampling fairly across them?</li> <li>Where might the system behave differently?</li> </ul> That’s not over-engineering. That’s just good sampling, and it makes testing much easier. 2025-10-08 https://goodenoughtesting.com/articles/test-data-and-sampling https://goodenoughtesting.com/articles/test-data-and-sampling The Art of Knowing Which TDD Tests to Delete Not all tests written during TDD belong in your codebase. While this might be familiar wisdom, it’s important to reiterate it here. <h2>A commit is an artifact</h2> I see a commit as an artifact of my work: it can deliver value or be examined to understand its functionality and how to fix issues. From this perspective, the tests included in the commit are essential for understanding how the code works and ensuring it performs as intended. In this case the tests included in a commit should be carefully selected. <h2>TDD is a practice not a test design technique</h2> TDD is a software development practice, not a testing or test design technique. Tests written during TDD serve different purposes depending on when they are written. At the end of this practice, you will have tests that drove your development and a piece of business logic that satisfies some functional or non-functional requirements. During your TDD process, you might have written a range of test types at various levels: <ul> <li>You may have written some functional unit testing</li> <li>You may have written some non-functional integration testing</li> <li>You may have written some functional system or acceptance testing</li> </ul> <h2>Keep the tests that satisfy these criteria</h2> When preparing to commit, you have to remove the tests that were driving your process and keep the ones that satisfy at a minimum the following criteria: <ol> <li>They verify that the implementation satisfies the requirements. Ask: “Does this test ensure my code accomplishes what it’s supposed to do based on the specifications?”</li> <li>They document code behaviou to help maintainability (including debugging, support, and changing. Consider: “Will this test help a developer understand the code or diagnose issues in the future?”</li> <li>They check that a specific shape, attribute, or form of the code or architecture ei important for the component you are developing , such astesting that a new object inheriting from a parent needs to define a specific method). Ask: “Is this structural test crucial for ensuring the component functions correctly in its intended context?”</li> </ol> For example, a test that verifies a method outputs correct results based on given inputs would meet these criteria, as it ensures the functionality aligns with the requirements.If you have tests that guided your code design but offer no further benefit, you should delete them. <h2>Example of tests to remove</h2> Here are some categories of tests that could be created during a TDD session and that you might consider removing. This is not an exhaustive list, and depending on your context, you might want to keep some, for example, to pin specific implementation details for performance or security reasons. There are instances when keeping such tests is justified, such as when dealing with legacy code where comprehensive documentation is lacking. Compliance may also require specific tests to meet regulatory standards or to demonstrate that certain conditions have been met. Additionally, in scenarios where performance is critical, keeping some of these tests can help ensure that the system performs optimally under various conditions. Scaffolding tests used purely for design exploration: <ul> <li>Existence/scaffolding tests (from the category of make it work, like testing that an object exists)</li> <li>Tests written to explore different API designs before settling on the final interface</li> <li>Experimental tests for approaches you ultimately rejected</li> <li>“Spike” tests used to understand how a third-party library works</li> </ul> Tests that verify implementation details rather than behaviour: <ul> <li>Tests that check private method implementations</li> <li>Tests that verify the exact sequence of internal method calls</li> <li>Tests that assert on specific data structure choices (e.g., “must use a HashMap”) when the requirement is just “fast lookups”</li> </ul> Over-specified tests that constrain refactoring: <ul> <li>Tests that verify exact error message wording when the requirement is just “must fail gracefully”</li> <li>Tests that assert on specific class structures when the requirement is about behaviour</li> <li>Tests that mock every dependency when testing at a higher level would be more valuable</li> </ul> Duplicate tests at the wrong abstraction level: <ul> <li>Low-level unit tests that duplicate what acceptance tests already verify</li> <li>Tests for trivial getters/setters with no business logic</li> <li>Tests for framework behavior rather than your code’s behavior</li> </ul> The key question to ask: <blockquote> If this test breaks, does it tell me something valuable broke, or just that I changed how I implemented something? </blockquote> 2025-10-06 https://goodenoughtesting.com/articles/the-art-of-knowing-which-tdd-tests-to-delete https://goodenoughtesting.com/articles/the-art-of-knowing-which-tdd-tests-to-delete What We Lost When QA Teams Disappeared In many companies over the last 15 years, QA teams have diminished or even disappeared. The tech industry decided we do not need special QA teams because everyone should care about quality. As companies aimed to be more flexible and moved away from old step-by-step methods, they stopped using QA as the final check before releases and instead encouraged the whole team to work together. I think this concept became popular around 2009. The best resource on this is Lisa Crispin and Janet Gregory’s book, Agile Testing: A Practical Guide for Testers and Agile Teams (2009). Thinking about quality from the beginning is great. It is like handling security or privacy: you have to start as soon as you define what the product should do. Adding quality, security, or privacy at the end is expensive, takes longer, and delays benefits for users. <h3>Developers doing more testing</h3> I want to focus on the impact this has on developers, especially regarding testing practices. Focusing on quality early led to more automated testing. Developers began testing their own work, mostly using automated tools. This is good. To achieve high product quality, we need to start with quality from the beginning, and everyone involved should pay attention to it. Developers are good at writing tools and automating tasks, so more testing tools and better automation environments were created. As a result, more code is now covered by tests. <h3>Do we have the right tests in place?</h3> Having close to 100% code coverage with automated tests might sound like a great metric. It is useful because any line of code not executed could cause a failure in production. However, these metrics mostly refer to line coverage, which can be achieved in different ways, sometimes without testing important cases for certain users. This shows a team can have good test coverage but still miss important quality or functional targets. <h3>Developers and testing skills</h3> Most schools and programs that teach software development do not have a proper curriculum for testing. Testing is usually taught as writing unit tests or learning a framework, not as a skill. It is mostly seen as a way to evaluate code, not the holistic product or its value from a quality perspective. In practice, developers pick up testing skills as they go. If you were in a good culture led by someone who knew testing philosophy and practices, you probably have decent or great testing skills. If not, you may still write tests just to show the code works as written. This is better than having no tests, but these tests might miss important or critical cases for the business and end-users. Reflect on your own testing habits. Are there areas where your skills might be lacking? Are your tests effectively capturing the needs of your end-users? By regularly evaluating your testing strategies, you can identify gaps in your knowledge and take steps to improve. Here is an example. Let’s look at the following piece of code: <pre class="ruby"># https://github.com/basecamp/once-campfire/blob/main/app/models/user.rb#L26 class User < ApplicationRecord # t.string "name", null: false def initials name.scan(/\b\w/).join end end </pre> This code is supposed to generate initials from the user’s name. What tests would you write for it? <ul> <li>If you focus on achieving 100% code coverage of <code>Users#initials</code>, a single test is enough.</li> <li>You could write two tests by adding a negative one that might throw an exception.</li> </ul> Both cases overlook an important point: Who are the users of our application, and which countries are they from? Depending on the country, this method might often return empty. For example, if the name is <code>José Ángel</code>, this would return only <code>J</code>. If the name is <code>佐藤陽翔</code>, it will return empty. Maybe this is correct, but my point is that we need test design to uncover for whom we are writing these tests, and then write the tests accordingly. Knowing who the end-users are and their most common or important use cases is a crucial part of a tester’s approach. What is currently missing is test design. We know how to implement and execute tests. We can organize and optimize test code, but asking whether these are the right tests is missing from conversations. <h3>With QA vanished, testing best practices, knowledge and wisdom also vanished</h3> Here is the problem today: In companies where the testing team is gone, there is no focus on testing as a skill. Whether the tests are good or not is random. It just becomes about numbers: if you write enough tests, you might cover important situations. I do not like this approach because it is not systematic. We need to increase developers’ knowledge about testing, especially as developers use LLMs. If you cannot properly ask for a testing concept, you probably will not get it from an LLM. You may get many tests written by an LLM that you have to maintain and execute, but without understanding test design and what to ask for, you might end up with bloat. <h3>The solution: learn test design</h3> The solution is to learn about test design, test design techniques, and discuss how to approach testing in the current context. Here are some starting points in case you want to learn more about test design: <ul> <li>Understand more about the entire test process and where test design fits in.</li> <li>Clarify the role of testing: it is about risk, priorities, product context, and project constraints. Accept that testing has tradeoffs, and when designing test cases you have to consider this.</li> <li>Get comfortable with test levels and test types. Choosing what to test depends on where and what you are planning to test.</li> <li>How to set test objectives and traceability to requirements, use cases, code, or important artifacts</li> <li>Test Design Techniques: mathematically backed ways to cover important areas of your code or requirements systematically. The core of test design is to know these techniques and when to apply them.</li> <li>Prioritization and risk: ranking test objectives and possible test cases based on risk, and choosing to apply a test design technique to generate more or fewer test cases, is crucial.</li> <li>Read about preconditions, postconditions, and the difference between them and assertions. This will help organize your tests even before writing the first line of code.</li> </ul> 2025-10-02 https://goodenoughtesting.com/articles/what-we-lost-when-qa-teams-dissapeared https://goodenoughtesting.com/articles/what-we-lost-when-qa-teams-dissapeared The smallest files of any type to be used in automated testing In case you really need in your test suite to handle real files of various types, here are some resources you can use: <ol> <li>The smallest possible valid PNG image: <a href="https://evanhahn.com/worlds-smallest-png/">https://evanhahn.com/worlds-smallest-png</a></li> <li>The smallest PDF file: <a href="https://pdfa.org/the-smallest-possible-valid-pdf/">https://pdfa.org/the-smallest-possible-valid-pdf/</a></li> </ol> But what you should actually bookmark is this repository: <a href="https://github.com/mathiasbynens/small">https://github.com/mathiasbynens/small</a> It has a great collection of the smallest valid files of any type to be used in automated testing. 2025-09-29 https://goodenoughtesting.com/articles/resources-for-developers/small-files https://goodenoughtesting.com/articles/resources-for-developers/small-files New AI Workshop at EuRuKo 2025 I started working on the workshop for <a href="https://2025.euruko.org">EuRuKo 2025</a>, “Don’t Let Your AI Guess: Teach It to Test.” <img alt="Image with announcement about Lucian Ghinda's workshop at EuRuKo" src="https://images.goodenoughtesting.com/articles/new-workshop-ai-wip.webp" /> The goal of this workshop is to introduce techniques for prompting that reduce the hallucinations AI may experience when asked to write tests. This is intended for individuals using AI to enhance their coding, allowing them to remain in control of which changes are applied. It’s likely unsuitable for fully agentic coding (unsupervised) unless you plan to use these prompts as system instructions for your agent. Let me start with my core belief: current LLMs are NOT ready to independently write tests that achieve the following goals: <ol> <li>Verify requirements</li> <li>Ensure the software behaves predictably and consistently over time</li> <li>Specify current behavior</li> </ol> They can generate unit tests that pass by examining the code, whether you or they wrote it. While this is useful, it seems like merely documenting the current code behavior. Without supervision, it’s just creating more tests to confirm the code works as it is written. This resembles verifying requirements, but it is NOT! Verifying requirements involves writing tests based on expected behavior, not on the current business logic in the code. The surse of truth (called test oracle) for a test should be the product requirements, the users needs and the customer expectations and not the code. If you care about correctness, consider using AI to generate tests, but maintain control: <ul> <li>Select which tests the AI will write</li> <li>Review the test data</li> <li>Review the assertions</li> </ul> Otherwise, you might end up with tests that pass but don’t effectively test what’s important, leaving your users unprotected from potential bugs. I plan to do a dry-run of this workshop before the conference and if you want to get invited at a discounted price join <a href="https://goodenoughtesting.com/subscribers/new">the GoodEnoughTesting newsletter</a>. 2025-08-21 https://goodenoughtesting.com/articles/announcing-the-new-euruko-workshop https://goodenoughtesting.com/articles/announcing-the-new-euruko-workshop How many tests are enough for this piece of code? Let’s look at the following Ruby code, but it could be in any language, as the logic is more important than the actual code. <img alt="Screenshot of a simple Ruby code with one condition" src="https://images.goodenoughtesting.com/articles/how-many-tests-would-you-write-1.webp" /> I think of looking at the code and focusing on the line coverage, we can say that two tests might be enough. <img alt="Screenshot of a the two test cases that would cover line coverage" src="https://images.goodenoughtesting.com/articles/how-many-tests-would-you-write-2.webp" /> But this also might be misleading. We can see that if we want to test the entire truth table, then we need 4 test cases: <img alt="Screenshot of the 4 test cases that would cover the entire truth table" src="https://images.goodenoughtesting.com/articles/how-many-tests-would-you-write-3.webp" /> There are various coverage criteria that we can apply and each one will result in a different number of test cases: <img alt="So how many tests do we need? 4 answers depending on our goal" src="https://images.goodenoughtesting.com/articles/how-many-tests-would-you-write-4.webp" /> <h2>How do we choose between these coverage criteria?</h2> We choose based on risk and project context. <h3>Choosing criteria taking risk into consideration</h3> <img alt="Choosing coverage criteria based on risk" src="https://images.goodenoughtesting.com/articles/how-many-tests-would-you-write-5.webp" /> When talking about the risk, let’s use a simple graph having a Y axis going from low risk to high risk. Branch coverage In this case, for low risk, we have branch coverage. That is just making sure the entire decision (the entire expression used in the <code>if</code> = <code>(A ||B)</code>) is one time true and one time false. We could execute, for example, <code>A = true, B = does not matter</code> and <code>A = false, B = false</code>, and that will achieve 100% branch coverage. Condition Coverage Then, as the risk increases, we can apply condition coverage. That means ensuring that each independent condition (<code>A</code> or <code>B</code>) is both accurate and false. In this case, we need 2 test cases: <ul> <li><code>A = false, B = true</code></li> <li><code>A = true, B = false</code></li> </ul> You can notice that actually you might want to make also the decision (the entire IF) true and false so you might choose 3: <ul> <li><code>A = true, B = does not matter</code></li> <li><code>A = false, B = true</code></li> <li><code>A = false, B = false</code></li> </ul> This is because of the short circuit logic that most programming languages have where if for example in case of <code>OR</code> if the first condition is true, the other ones are not evaluated. Modified Condition Decision Coverage The more the risk, the more we want to make sure to cover. In this case, we want to choose so that we can also check how each condition affects the decision. So we might choose the following test cases: <ul> <li><code>A = false, B = false</code> => A is decisive and B is decisive</li> <li><code>A = true, B = false</code> => A is decisive, B is not</li> <li><code>A = false, B = true</code> => A is not decisive, B is decisive</li> </ul> Truth table Of course if we want to cover all combinations, we have to make the entire truth table: <ul> <li><code>A = true, B = false</code></li> <li><code>A = true, B = true</code></li> <li><code>A = false, B = false</code></li> <li><code>A = false, B = true</code></li> </ul> And this one will be choosen when the risk is high as we want to cover the possible logical cases here. <h3>Choosing criteria, taking features or the project in consideration</h3> <img alt="Choosing coverage criteria based on project or feature" src="https://images.goodenoughtesting.com/articles/how-many-tests-would-you-write-6.webp" /> Imagine that you are implementing a feature that is related to compliance with a standard or legal compliance. In this case, you should apply either a truth table or modified condition decision coverage as they have higher chances of catching more bugs and exercises more paths in your code. But if you are working on something that is medium or low from the point of view of the end-users or clients, you might want to apply condition coverage or even just branch coverage. <h2>Good Enough Testing means matching coverage to risk</h2> <img alt="Summary of Good Enough Testing mindset" src="https://images.goodenoughtesting.com/articles/how-many-tests-would-you-write-7.webp" /> A core concept in Good Enough Testing is adapting to the risk and project context and choosing the test cases so that it lowers the risk and it covers what’s important in our project for the end-users, for the company and for the team. That means being systematic so we can rely on the test suite, focusing on what’s important to avoid overtesting or under-testing, and keeping it lean so the test suite is easy to maintain. Don’t overtest. Don’t guess. Use structure. 2025-06-24 https://goodenoughtesting.com/articles/how-many-tests-would-you-write https://goodenoughtesting.com/articles/how-many-tests-would-you-write Tests should be simple The <a href="/">Good Enough Workshops</a> focus on creating the minimum number of test cases to ensure that the requirements, business logic, and code logic are adequately covered. So, during the workshops, we discuss how many tests are needed to cover a feature or piece of logic, how to reduce them when you don’t have time or in the context of the risk, and more related to test case design. But we don’t talk about writing test code. There is a lot of information about how to write test code; there are good books and articles about this if you need a deep dive. I am sharing my concise perspective on establishing a style guide or guidelines for writing tests in Minitest or RSpec. <img alt="A screenshot of 5 principles for simple testing" src="https://images.goodenoughtesting.com/tests-should-be-simple.webp" /> <h3>1. Keep tests plain and simple</h3> Avoid complexity. A big part of avoiding complexity is avoiding abstractions in test code as much as possible. Don’t create abstractions with test code. You should write explicit, straightforward code. Focus on writing tests so that when read, they can fit in your mind in a single pass. There should not be a need to open other files or go back and forth between various sections in the same file. Don’t try to be smart with your code. Don’t apply SOLID. Test code should flow like a recipe from top to bottom. <h3>2. Each test should be read like a story</h3> The story should describe clearly: <ul> <li>preconditions (state of the system before running the main action)</li> <li>the action</li> <li>the expectation or assertion</li> <li>postconditions when needed (the state of the system after the test run)</li> </ul> When you need to write preconditions and postconditions, make sure the test will fail if they are not matched. Example: <ul> <li>Use <code>create!</code> instead of <code>create</code> if you need to create AR</li> <li>Use <code>validate!</code> on a new instance when you don’t want to create it </li> </ul> <h3>3. Having the same code across multiple tests is allowed and encouraged</h3> Duplicated code is allowed because it makes the system resilient to unintended changes. Make the assertions explicit and, depending on the test’s purpose, assert them with concrete values rather than references to elements from the tested code. <h3>4. There should be positive and negative tests to limit the confirmation bias</h3> Make sure you have also written negative scenarios to cover the features/logic that you are testing. You should verify how your system handles invalid, unexpected, error-prone input or erroneous conditions. Write at least one test that will cover the case where your feature/business logic should not work and will throw an error, return false/nil/empty … <h3>5. Tests that never failed cannot be trusted</h3> Consider that any tests that never fail cannot be trusted; thus, each test should have failed at least once on your computer before pushing the code. For each test, ensure you execute it at least once in a scenario where you have modified the code so that the test is expected to fail. Don’t commit a test that never failed on your local machine. 2024-11-20 https://goodenoughtesting.com/articles/tests-should-be-simple https://goodenoughtesting.com/articles/tests-should-be-simple Two new sessions for December 2024 I launched two new sessions of the workshop for December 2024: <ul> <li>One session is scheduled for <a href="https://lu.ma/zqp6zco1">11 December 2024 - 08:00 UTC / 09:00 CET</a></li> <li>The other for <a href="https://lu.ma/533zg5zq">12 December 2024 - 15:00 UTC / 16:00 CET / 10:00 EST</a></li> </ul> I choose two different times to accommodate more participants for the workshop. One slot is scheduled for morning Europe time, and the other for morning US time. This was requested by many people that wrote to me asking about different times for the workshop so more people can attend. <h2>Pricing</h2> Based on the experience with the November workshops (<a href="https://lu.ma/rx31g5h9">session one</a> and <a href="https://lu.ma/hgs6eb3t">session two</a>) that were fully booked very quickly because there was a discount offered for joining earlier, I will continue with the Early Bird pricing for the ones in December. The price for the December workshops is: <ul> <li>USD 80 for anyone buying before 30 November</li> <li>USD 100 for anyone buying after 30 November</li> </ul> <h2>Workshop details</h2> You can read more about the workshop on the <a href="/#get-tickets">homepage</a>. And if you have any questions please <a href="/contact">contact me</a>. 2024-11-19 https://goodenoughtesting.com/articles/two-new-sessions-in-December-2024 https://goodenoughtesting.com/articles/two-new-sessions-in-December-2024 What is the difference between TDD and the testing process? This is a question I sometimes receive, and I want to share my thoughts on it. <h2>TDD</h2> TDD is a software development methodology that drives development through tests, facilitating the writing of code and building of features or products. Without doing here a full review of TDD, for this article we take the main idea that is about first writing tests and then writing the code that makes the tests pass. <h2>Testing process</h2> From a developer’s perspective, the testing process is a series of activities done to intentionally write tests with the purpose of verifying code correctness, documenting expected behavior, identifying defects or bugs, and improving maintainability. <h2>What is the difference?</h2> First, while doing TDD you should write tests by using test design techniques which are part of the testing process. Rather than viewing testing and development as two separate phases, consider that TDD involves writing tests so it will include applying testing activities while doing TDD. Still, we talk about doing testing and doing TDD. I think the main difference comes from what is called a development mindset and a testing mindset. This difference exists in both TDD methodology and in many others because it is about people and points of view while working. <h3>Development mindset or developer mindset</h3> While being in a development mindset, the focus is more on trying to write code that implements the requirements in any form or shape they were provided. Thus the tests written while in the development mindset are more focused on verifying requirements or proving that the code works as expected and requirements are met. From practice while being in development mindset there are more positive scenarios written than negative scenarios. Keep in mind that it is considered positive scenario also when you are verifying for example than an API will return 404 when record not found. While this might be a failure from point of view of the user trying to access that page, is a positive scenario because it tries to assess an expected behavior of the product. The same goes for thinking about tests as documenting behavior. They will mostly be focused on the positive scenarios. There is also a bias there called developer bias. That means that once a solution is created (either written, explained or thought about) the developer is more likely to write tests that are passing, thus it is more inclined to demonstrate (with tests) that the code is working as expected or that the solution is the right one instead of trying to find the solution that solves the requirements. <h3>Testing mindset or tester mindset</h3> In the testing mindset, the focus is more on making sure the code is reliable, consistent predictable, and it will not return unexpected results. Thus, tests written while being in the testing mindset are focused more on edge cases of undocumented behavior; it is more concerned with the questions “What happens if …?” and “What happens when …?”. The tests written are mostly negative scenarios, and so they are inspired by the idea of what could possibly trigger a bug. <h3>TDD and Testing Process</h3> In the first phase of building a feature and applying TDD, we mostly live in the development mindset. That is both natural and good for making progress and being efficient as a developer. We want to first make the code work as expected with a focus more on the functional characteristics of the code and less on the non-functional attributes (performance, security, …). Less does not mean not paying attention to non-functional attributes while coding but it is more about what kind of tests are written during this first phase. Then, after the functional part of the requirements is implemented, we want to make sure the code is, for example: reliable, consistent, predictable, maintainable, and performant. You can choose other non-functional/code quality attributes. During this second phase you can still use TDD (and if you are familiar with it, please do so) but the tests driving the development should be generated while being in the testing mindset. Refactor the code to make these tests pass, thus improving the quality of the code. <h2>One trick for the testing mindset</h2> One trick to switch to a testing mindset is to re-read the requirements, open a test file, think about what cases you want to test, and just write the names of the tests. The tests will be higher quality if you write all of the cases first and then fill in the test code (preconditions, initial state, assertions) instead of writing the first test, run it, then write the second test, run it, and so on. Whether we like it or not, writing test code is still development, and you will get back in the developer mindset before you get to think about what to test next. 2024-10-24 https://goodenoughtesting.com/articles/difference-between-tdd-and-testing-activity https://goodenoughtesting.com/articles/difference-between-tdd-and-testing-activity How to think about testing as a developer Here is the video version of this article: <iframe src="https://www.youtube.com/embed/b8wHuEufDAw" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen class="w-full aspect-w-16 aspect-h-9" height="600"></iframe> <h2>High level view</h2> In any project, there is an area called Quality Management that has two main activities: <ul> <li>Quality Assurance - mostly means measuring quality via various processes and tools either during the production and/or looking at the end result</li> <li>Quality Control - making sure we have processes in place that will guide, verify and change the organisation and in-place processes to assure a certain level of quality built-in</li> </ul> From this perspective, Testing is the outcome of a Quality Control decision and is an activity part of Quality Assurance. When doing side projects, for example, Quality Management is informal, and you mainly represent it. You decide (or not - but still a decision) what quality you want to have and how much, if anything, you want to do about it. I will not go too deep into the Quality Management process, but it is good to know that testing is not just dropped into a project. Still, it should be part of a bigger strategy that looks into how much time/effort/money/people the organisation want to put in and how in order to get to a specific level of quality of our building process and the quality of our end result. Let’s focus now on how developers should see testing and what can it bring to them. <h2>First, what is testing?</h2> There are multiple definitions of testing. Let me share some of them here for a good and informed starting point when considering testing and I will comment on them while trying to extract key points. <h3>Making sure the software is predictable and consistent</h3> <blockquote> Software testing is a process, or a series of processes, designed to make sure computer code does what is designed to do and, conversely, that it does not anything unintended. Software should be predictable and consistent, presenting no surprises to users. </blockquote> <cite> <a href="#the-art-of-testing-reference">Mayers, 2011</a> </cite> I like this definition because it does not even use the words bugs, errors and we can focus more on what expectations we have from our code: <ul> <li>To be predictable</li> <li>To be consistent</li> <li>To catch surprises or unexpected behavior</li> </ul> Key point As a developer, I should assess if the execution of my code is predictable and consistent. Key point As a developer, I should make sure that my code will only return expected outcomes/outputs and there should be no surprise. <h3>Verifying the correctness of functionality and finding defects</h3> <blockquote> Testing is both a constructive activity in that it is verifying the correctness of functionality, and it may be a destructive activity in that the objective is to find defects in the implemented software. Testing verifies that the requirements are correctly implemented, and it yields the presence or absence of defects. </blockquote> <cite> <a href="#fundamentals-of-software-testing-reference">O’Regan, 2019</a> </cite> Key point As a developer I should verify that requirements are implemented Key point As a developer, I should check that there are no hidden bugs or cases improperly handled by my code by testing invalid or out-of-boundary inputs, logic, or state The “destructive activity” part is a good point of view to keep in mind while writing tests and could drive the effort also to execute/find test cases that might break the code/logic/app. <h3>Comparing what-is with what-ought-to-be</h3> <blockquote> At its core, testing is the process of comparing “what is” with “what ought to be.” </blockquote> <cite> <a href="#a-practitioners-guide-reference">Copeland, L. (2003)</a> </cite> Key point As a developer, I want to verify that what was requested is what I implemented => my code works the way it is expected <h2>Why do we test?</h2> Based on the definitions above (and many more), it is good to define testing from the perspective of developers. We test because we want to: <ol> <li>Verify that requirements are implemented correctly (what was requested is what it was implemented)</li> <li>Check that the software behaves predictable and consistent over time (taking the same inputs and initial state, it should give the same result every time)</li> <li>Specify the current behavior to improve maintainability</li> </ol> <h2>References</h2> <a name="the-art-of-testing-reference">[Mayers, 2011]</a> Glenford J. Myers, Corey Sandler, and Tom Badgett. 2011. The Art of Software Testing (3rd. ed.). Wiley Publishing. <a href="https://onlinelibrary.wiley.com/doi/book/10.1002/9781119202486">DOI 10.1002/9781119202486</a> <a name="fundamentals-of-software-testing-reference">[O’Regan, 2019]</a> O'Regan, Gerard. “Fundamentals of Software Testing.” Concise Guide to Software Testing (2019). <a name="a-practitioners-guide-reference">[Copeland, L. (2003)]</a> Lee Copeland, A Practitioner’s Guide to Software Test design - ISBN:158053791x 2024-10-24 https://goodenoughtesting.com/articles/how-to-think-about-testing-as-a-developer https://goodenoughtesting.com/articles/how-to-think-about-testing-as-a-developer Session scheduled for 1st November is fully booked. 🙏 The GoodEnoughTesting workshop session <a href="https://lu.ma/hgs6eb3t">planned for 1st November is now fully booked</a>. <img alt="Good Enough Testing - fully booked" src="https://images.goodenoughtesting.com/1st-november-good-enough-testing-fully-booked.webp" /> For anyone that still wants to join, I just opened the registration for the next session - happening on 22 November. As I announced this workshop is at full price. If you join until 31 October you will get 20% discount Register at: <a href="https://lu.ma/rx31g5h9">https://lu.ma/rx31g5h9</a> <a href="https://lu.ma/rx31g5h9"><img alt="Get 20% discount until 31 October" src="https://images.goodenoughtesting.com/22-november-20-percent-discount.webp" /></a> 2024-10-24 https://goodenoughtesting.com/articles/news-fully-booked-1st-november https://goodenoughtesting.com/articles/news-fully-booked-1st-november A short video introduction I created a short video to introduce the workshop and explain what questions I am going to help you answer during the the workshop. <iframe src="https://www.youtube.com/embed/YqhgqzzNRuc" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen class="w-full aspect-w-16 aspect-h-9" height="600"></iframe> 2024-10-23 https://goodenoughtesting.com/articles/what-is-this-workshop https://goodenoughtesting.com/articles/what-is-this-workshop What is good enough testing? When I was trying to search for this website online and see how Google renders it, I discovered that <a href="https://en.wikipedia.org/wiki/James_Marcus_Bach">James Bach</a> coined the term “Good Enough Testing” in 1998. I know about James Bach from teaching Exploratory testing techniques when I delivered the Certified ISTQB Foundation Level course so I think it is important to review the paper he published and see if there is something that I can use in my workshop. James’ paper <a href="https://www.satisfice.com/download/a-framework-for-good-enough-testing">available online at James’ website</a> sets an excellent base for thinking about Good Enough Testing. <h2>What is Good Enough Testing?</h2> He shares there the following definition for Good Enough Testing: <blockquote> Good Enough testing is the process of developing a sufficient assessment of quality, at a reasonable cost, to enable wise and timely decisions to be made concerning the product. </blockquote> <cite> <a href="#james-back-reference">James Bach. 1998</a> </cite> I agree strongly with the definition; it feels more explicit than mine. The definition I use in the workshop is the following: <blockquote> Considering the level of risk, we define coverage criteria that will drive the test design so that we can achieve 100% coverage with the minimum number of test cases. </blockquote> <img alt="Definition of Good Enough Testing" src="https://images.goodenoughtesting.com/what_is_good_enough_testing_max_1600.webp" /> My workshops are focused on developers, and I try to show a series of tools or test design techniques that they can pick and use depending on the feature, the risk, and the time. I talked about the time axis in the workshop, but I see it closely connected with the risk. The main idea is that we can use systematic test design techniques, and based on the level of risk and/or time constraints, we can apply the techniques, strict or loose. We can choose the maximum coverage or pick a coverage criteria that fits the current context. The second point is that my workshop is dedicated to developers starting from the assumption that the purpose of testing in their specific case is: <ul> <li>To verify that the code works as expected</li> <li>To document the code behavior</li> </ul> There are also other important objectives for testing (like finding bugs and validating that the solution is helping the end-user), but I think those first 2 are closer to developers and a good fit to get started. I plan to create a very concrete and practical workshop, so I will continue to use my own due to the nature of the workshop. There is not enough time to discuss how each part of the context can drive the testing and influence objectives, techniques, and execution. But I will definitely present and talk a bit also about James’ one. <h2>Basic questions</h2> In my workshop, I start with some questions that will drive what we learn: <ol> <li>How do you know how many tests are enough?</li> <li>How do you know if your tests cover enough business logic?</li> <li>How confident are you that your tests cover the critical parts of your code?</li> </ol> And the main idea is that we want to have a systematic approach to answering them. A systematic approach means finding a repeatable process or heuristic that does not depend on the developer’s experience. We want to predict the total number of test cases and ensure we can always identify and explain the current criteria. In the same paper “A Framework for Good Enough Testing”, James starts with the following fundamental question: <blockquote> How do I know if I’m doing, or have done, enough of the right type of testing? </blockquote> <cite> <a href="#james-back-reference">James Bach. 1998</a> </cite> He adds more questions that will help assess the value of the testing: <blockquote> How accurate and complete is it? How reasonable is it? Is it within project constraints? Is there a good return on investment, such as the extent of information gained per test? How well does the assessment serve the project and the business? Is it soon enough to be useful? </blockquote> <cite> <a href="#james-back-reference">James Bach. 1998</a> </cite> These are very useful questions to consider and would help create a testing strategy. James’ chapter “Components of Assessment” covers various aspects of the product, testing costs, decision-making, and timing and in case you are preparing a testing strategy or a document about how to adapt testing to various project realities, I invite you to read it and try to answer all those questions. <h2>How to answer all these questions about testing</h2> <blockquote> Unfortunately, there is no objective or rigorous calculus for answering this question, but we can identify what to consider in attempting to answer it. We can begin to build at least a heuristic framework around the problem. </blockquote> <cite> <a href="#james-back-reference">James Bach. 1998</a> </cite> I agree that it is hard to create a generally applicable or objective way to answer these questions for all projects and contexts and this is why I narrowed down my focus in the workshop to help developers have a couple of test design techniques and teach them how to use and adapt them when trying to test their own code. While not enough to ensure good quality, it could be - in case of an engineering or development department - the base or starting point of good-quality product engineering. It provides the foundation for developing stronger and more user-focused quality processes for such department. <h2>References</h2> <a name="james-back-reference">[James Back, 1998]</a> A Framework for Good Enough Testing. Computer 31, 10 (October 1998), 124–126. <a href="https://doi.org/10.1109/2.722304">https://doi.org/10.1109/2.722304</a> - Full PDF available at <a href="https://www.satisfice.com/download/a-framework-for-good-enough-testing">satisfice.com</a> 2024-09-24 https://goodenoughtesting.com/articles/what-is-good-enough-testing https://goodenoughtesting.com/articles/what-is-good-enough-testing Announcing the Good Enough Workshops A while back, I submitted a proposal for a talk or workshop at the <a href="https://2024.euruko.org">EuRuKo</a> conference. Here is what I wrote in the abstract where I described the workshop: <blockquote> When learning Ruby or Ruby on Rails, you learn about RSpec or Minitest, you get to know the DSL and how to use the library features to write model, controller, or integration tests. But you don’t learn much about testing: What does it mean to test a feature or a piece of code? How do you know that the tests you wrote will actually catch bugs? This talk is about test design techniques that help you write the minimum number of tests covering the widest surface of your app. You will learn about Decision Tables, Boundary Value Analysis, Equivalence Partitioning, State Transition testing and more, all with Ruby code examples. </blockquote> And as the main pitch I wrote: <blockquote> There are a lot of resources out there about how to write RSpec or Minitest and about speeding up test case execution, but there is precious little knowledge shared online about how to decide what to test and how to write a good test case in Ruby. This talk plans to close this knowledge gap by showing easy-to-use and practical test design techniques applied in Ruby and Ruby on Rails. I have worked with Ruby since 2007 and am passionate about learning and teaching. I have over 11 of experience delivering technical training/workshops for teams on learning Ruby or testing. I am also an ISTQB-certified trainer, delivering international courses about testing to a wide range of audiences.** </blockquote> Along with this workshop I also submitted a talk called “The Modern Rubyist.” <h2>A response</h2> To my complete surprise, I got a response from EuRuKo organizers that both my proposals were accepted by the program committee. And because both of them were subjects I was passionate about I decided to accept the challenge and deliver both of them. <h2>Early feedback</h2> EuRuKo happened in September 2024, so I started working on the workshop around the end of July. I somehow had a version ready 2 weeks before the conference, and I wanted to test it and get some honest feedback. That is because the workshop supposed to be 2 hours and so I wanted to make sure that I manage to deliver some practical tips about how to test. I though the best way to see the value of this workshop is to organise an online live version thus, I created one. I opened it for 10 participants because I wanted to make sure I have time to ask for feedback: <img alt="The first workshop session" src="https://images.goodenoughtesting.com/first-session.webp" /> I thought about putting a price or not on it, but I decided to price it for 20 USD as the first session. I learned that when a workshop has a price, the participants are more focused and give honest feedback. If it is not helpful, you can find out very quickly. It sold in less than 24 hours and was the first hint that this might be useful I also got some very good feedback. Here is a part of it (but I will add more in one of the pages of the website) <img alt="Feedback about the workshop" src="https://images.goodenoughtesting.com/workshop-feedback.webp" /> <blockquote> “Lucian’s understanding of the whole, the experience is noticeable and the demonstrations were smooth” “Seeing the same testing strategies in various forms: drawings, tables, test files, graphs, etc. I like when I can view something differently, it helps to find blind spots” “The content was great! I had not seen actual definitions for some of the techniques that were presented!” “The delivery, the way you presented it - not arrogant at all; like we are colleagues, you researched a topic and we hop on a call to help me understand it!” </blockquote> I also got more practical and valuable feedback and so I was prepared for the EuRuKo workshop. <h2>EuRuKo workshop</h2> At EuRuKo I found out that the workshop was fully booked. We discussed over email one week before the workshop, and I agreed to a maximum of 50 participants because I wanted to make it a bit more interactive. And all seats were booked before the conference started. That put a bit of pressure on me. I delivered the workshop, and had a great interaction with the participants, and I will publish the video here because it was recorded. Of course having 50 participants makes the workshop be more toward me explaining the techniques and we did not had enough time to write the actual tests in Minitest/Rspec. Anyhow as the workshop is more about the how to design test cases than how to write in Minitest or RSpec this worked out well. <h2>More sessions</h2> I think learning how to design better test cases is important and so I will launch a series of live workshops. I will increase the time to 3 hours per workshop and probably a maximum 15 participants per session. The price will also be increased step by step until it reaches 100 USD per workshop. I think 100 USD is a good enough price to ensure everyone is involved, and I think this is the value that people could get out of my workshop. If you, as a participant, are able to save 2-3 hours while writing tests at your job next week after my workshop, or if writing better tests improves the quality of your code, I think this would make paying for the workshop a no-brainer. If you like this idea, subscribe to the newsletter 👇 2024-09-22 https://goodenoughtesting.com/articles/website-launch https://goodenoughtesting.com/articles/website-launch