Testing is driven by Human Judgements, Observations, Analysis, Heuristics & Biases.
However, like in any other field, Biases and prejudices can be very risky for Testing. It’s good to know about them in advance and develop a questioning / investigative attitude whenever you feel you are getting biased.
I have seen poor products that have come for testing to me in projects or in bugathons where the developers and product owners claimed that they didn’t have many serious problems as they have already performed their tests (developer tests + tester tests) and all the tests apparently passed for them. They just want to have beta testing done before a major release so that if there are some minor issues, they can find those too.
However, the results from testing such products were totally different as I could find over 100+ issues by using simple (read “powerful”) oracles such as FEW HICCUPS. The team had to postpone the release plans, or plan for a quick next release (hotfix) to accommodate the key issues.
Questions that came up in my mind:
- What’s going on?
- What could be the cause of this gap?
- How are the developer tests passing?
- How are the tester’s tests passing?
- Is it inefficiency?
- How are the results all green?
- Why are regression tests not finding any new bugs?
- What are the blind spots that might be hitting them?
Never had an answer to these questions till I read about the Experimenter Effect (or Observer Effect) in Psychology.
This refers to the influence that experimenters (read ‘testers’) who conduct an experiment (read ‘tests’) have on the performance of participants (read ‘testing’) and the interpretation of the results. It is a form of bias that affects the validity of experiments as the experimenters (read ‘testers’), either deliberately or otherwise, influence the test results. It is one of the reasons why the results do not get replicated by future evaluations (read ‘bugathons’), and good experimenters (read ‘testers’) look for various ways to negate it.
Biases can hit our work, and its outcome by a big margin!
- Are we designing our tests (scripts) to PASS?
- Are we modifying our tests (scripts) by discarding some factors?
- Are we not doing the right assertions?
- Are we even covering all significant tests for our testing (scripting)?
- Are we aware of blind spots that could be hitting us?
- Are we discussing our test strategy & testing often? Should we start doing it?
- What’s the primary purpose of our scripts? Demo? or Testing?
- Are the results biased? modified? manipulated?
- Are we just re-executing scripts till they PASS somehow?
- Are we blindly trusting the green report?
Wondering, Why a green report cannot be trusted? Check out 10 Reasons why a green report might not be green!
Interested to know some common biases, that might be affecting your testing? Check out the summary of this talk on Biases Driving 5 Misconceptions that Testers Face by Brijesh Deb at TestFlix 2020.
Does it mean, scripts are bad? NO. It’s just that everything comes with an effect. So, Be Aware! Be Critical! Be Observant!
Do Script Your Tests! But Test Your Scripts & Biases Too!