Log in

No account? Create an account
When is it OK to Find a Bug

"How many times have you seen an email praising the heroic efforts of the developer who fixed some last-minute major issues in a huge new feature that were discovered in final user acceptance testing? This should not be seen as a heroic deed—rather it should be seen as a tragic failure. " — Real World DevOps

Let's be clear, finding issues is always best before it exists. However, we have layers of testing specifically because we want to find issues. The fact that you found the issue during testing is a good thing and a success.

Quick side note on User Acceptance Testing, at this point I believe that the tests should be focusing on, now that the system is in place rather than testing that features meet business requirements, you want to be testing that the system as a whole is meeting the value/proposition expected. The types of issues I hope to find here aren't ones where the behavior deviates from the specified requirements, but that the requirements were not satisfactory with either what was actually expected or that expectations changed due to other factors.

The two main points I want to note is

  • Reproducing the issue quickly in lower environments
  • Getting the fix to the environment quickly

If either of these things don't happen, then you have a failure. These eliminate the "heroic" efforts because it reduces time taken to fix a bug.

Read more...Collapse )

UI Automation by Mocking

The testing pyramid is a popular communication tool to show where the quantity of testing should occur. Some people have to complain about the analogy. In essences the push is to have a reduction in tests which happen at the UI and with full system integration (end-to-end testing). There is plenty of information out there on why you want to do that, like the article I linked above.

While I haven't actually made this path happen, I'm looking at yet another layer of separation to help facilitate the UI testing, mock the backend interactions. The goal here is to identify the primary state changes the UI is expected to under go, verify the UI requests to the backend and to return events back to the UI to validate its behavior to that response. This keeps the end-to-end part of the setup to be less complete, though a good mocking of backend system is still an undertaking.

The other aspect to UI testing is that Selenium is quite popular, but isn't really stable or consistent. So I'm also going to focus on utilizing only headless Chrome for the majority of UI testing. Once that testing is complete I can look at what is left to test with the end-to-end environment.