regression tests are exploratory yesteryear nature together with are non-repeatable”? Influenza A virus subtype H5N1 tester friend of mine argued strongly, trying to plough over himself the world to drib dead on his struggle against my advocacy for exploratory testing!
This made me cogitate. Not that he had succeeded inwards proving that Exploratory Testing (ET) sucks! But I had to initiative of all thinking if repeatability (read reusability) of a examination is genuinely all that bad! I estimate “repeatable test” VS “non-repeatable test” declaration is only just about other human face upwards of the “Scripted testing” VS “Exploratory testing” debate. Here inwards this post, I am non going to drib dead on the same debate trying to evidence which i is amend together with which i is not. Rather I would similar to think to a greater extent than on the repeatability expression of a test!
Some questions to initiative of all alongside tin be:
1) Can ALL the tests live made repeatable?
2) Should ALL the tests live made repeatable?
3) By making a examination repeatable are nosotros non (may live inadvertently) making them likewise predictable?
4) Can repeatable tests discovery NEW problems (defects/bugs) inwards the system?
5) Blindly trying to brand each together with every examination repeatable. Is it worth the endeavour together with expense?
6) Why should nosotros repeat only a laid of tests (test cases, examination scripts whatever), when nosotros tin utilise the same fourth dimension exploring much to a greater extent than tests that may uncover unknown defects!
7) When nosotros tell a examination is repeatable, is the examination genuinely repeatable inwards all its senses? Can mortal guarantee that the examination runs precisely inwards the same means (environment, concurrently running processes together with applications, precisely same DLLs loaded at that moment, motorcar condition, configuration settings, together with and so forth) equally it had run the final fourth dimension around?
8) Do repeatable tests guarantee reproducible defects/bugs?
9) Influenza A virus subtype H5N1 repeatable examination tin brand sure at that spot is no re-occurrence of an before defect. Ahh well! But how long? Will these so-called repeatable tests non unloose value over a menstruum of fourth dimension (after a number of iterations)?
10) Is software testing all most repeatability?
The declaration inwards back upwards of making your tests repeatable may concord skillful to only about extent, inwards sure cases similar Regression testing or for that thing Performance testing. But is at that spot whatever indicate of trying difficult to brand ALL the tests repeatable? Here are few interesting quotes from only about honorable Testing Gurus/Experts:
By repeating tests nosotros genuinely brand sure nosotros are avoiding other possible defects (remember minefield analogy? Stepping on someone’s mensuration genuinely makes sure nosotros may avoid stepping on a alive landmine) thence minimizing our direct chances of discovering novel defects!
"Pesticide paradox" compares software defects alongside that of pests! Imagine a context when a farmer applies sure pesticide to instruct rid of insects from his crop. There is every possibility that at that spot volition live only about insects that volition travel the pesticide. If he keeps applying the same pesticide, the insects eventually may create upwards resistance together with the pesticide would no longer work! "Pesticide paradox" describes the work that a regression examination serial gets less together with less powerful equally y'all utilization it over together with over again. When a laid of repeatable tests is run for a menstruum of fourth dimension they tend to travel yesteryear to a greater extent than oft than failing. Influenza A virus subtype H5N1 examination is valuable when it fails together with the failure uncovers a bug. Running a laid of tests that seldom neglect together with get got to the lowest degree direct chances of uncovering defects sounds similar a bad idea. Isn’t it?
Having said this, does this hateful repeatable tests are waste materials of time? May live not!
1) Think of Performance tests that a tester may require to run over together with over i time to a greater extent than for days inwards together with days out.
2) Think of Benchmark tests.
3) Think of Build Verification Tests/Sanity Tests that require to live run every fourth dimension y'all get got a novel build!
4) Think of Regression tests.
5) Think of a scenario where y'all require to run sure tests that are of import inwards nature. The tests that verify only about really critical functionalities of the application together with these tests must live run periodically to brand sure those functionalities drib dead on to piece of work without issues.
So, it appears that having only about repeatable tests tin live helpful for your examination projection along alongside those tests that are non-repeatable. As always, it sure enough seems to live context dependant together with depends on your testing mission/goal! What exercise y'all think? Do y'all think ALL (what that tin mean!) tests MUST live repeatable, equally if nosotros are non software testers rather only about robotic human beings trying to repeat only about algorithm already decided inwards advance! Repeatable tests! Are they the require of the hr or necessary evil? Your thoughts please.
Happy Testing…
Further Reading:
Reasons to Repeat Tests - By James Bach
Repeatability is Overrated - By Elisabeth Hendrickson
More interesting articles here :Tutorial Software Sumber : http://www.softwaretestingtricks.com/
This made me cogitate. Not that he had succeeded inwards proving that Exploratory Testing (ET) sucks! But I had to initiative of all thinking if repeatability (read reusability) of a examination is genuinely all that bad! I estimate “repeatable test” VS “non-repeatable test” declaration is only just about other human face upwards of the “Scripted testing” VS “Exploratory testing” debate. Here inwards this post, I am non going to drib dead on the same debate trying to evidence which i is amend together with which i is not. Rather I would similar to think to a greater extent than on the repeatability expression of a test!
Some questions to initiative of all alongside tin be:
1) Can ALL the tests live made repeatable?
2) Should ALL the tests live made repeatable?
3) By making a examination repeatable are nosotros non (may live inadvertently) making them likewise predictable?
4) Can repeatable tests discovery NEW problems (defects/bugs) inwards the system?
5) Blindly trying to brand each together with every examination repeatable. Is it worth the endeavour together with expense?
6) Why should nosotros repeat only a laid of tests (test cases, examination scripts whatever), when nosotros tin utilise the same fourth dimension exploring much to a greater extent than tests that may uncover unknown defects!
7) When nosotros tell a examination is repeatable, is the examination genuinely repeatable inwards all its senses? Can mortal guarantee that the examination runs precisely inwards the same means (environment, concurrently running processes together with applications, precisely same DLLs loaded at that moment, motorcar condition, configuration settings, together with and so forth) equally it had run the final fourth dimension around?
8) Do repeatable tests guarantee reproducible defects/bugs?
9) Influenza A virus subtype H5N1 repeatable examination tin brand sure at that spot is no re-occurrence of an before defect. Ahh well! But how long? Will these so-called repeatable tests non unloose value over a menstruum of fourth dimension (after a number of iterations)?
10) Is software testing all most repeatability?
The declaration inwards back upwards of making your tests repeatable may concord skillful to only about extent, inwards sure cases similar Regression testing or for that thing Performance testing. But is at that spot whatever indicate of trying difficult to brand ALL the tests repeatable? Here are few interesting quotes from only about honorable Testing Gurus/Experts:
Highly repeatable testing tin genuinely minimize the direct chances of discovering all the of import problems, for the same ground that stepping inwards mortal else’s footprints minimizes the direct chances of beingness blown upwards yesteryear a dry soil mine.
- James Bach, Test Automation Snake Oil, 1996
- Brian Marick's utter Classic Testing Mistakes
By repeating tests nosotros genuinely brand sure nosotros are avoiding other possible defects (remember minefield analogy? Stepping on someone’s mensuration genuinely makes sure nosotros may avoid stepping on a alive landmine) thence minimizing our direct chances of discovering novel defects!
Every method y'all utilization to forestall or discovery bugs leaves a residuum of subtler bugs against which those methods are ineffectual.
- "Pesticide Paradox", Boris Beizer, inwards his mass Software Testing Techniques, Second Edition, 1990
"Pesticide paradox" compares software defects alongside that of pests! Imagine a context when a farmer applies sure pesticide to instruct rid of insects from his crop. There is every possibility that at that spot volition live only about insects that volition travel the pesticide. If he keeps applying the same pesticide, the insects eventually may create upwards resistance together with the pesticide would no longer work! "Pesticide paradox" describes the work that a regression examination serial gets less together with less powerful equally y'all utilization it over together with over again. When a laid of repeatable tests is run for a menstruum of fourth dimension they tend to travel yesteryear to a greater extent than oft than failing. Influenza A virus subtype H5N1 examination is valuable when it fails together with the failure uncovers a bug. Running a laid of tests that seldom neglect together with get got to the lowest degree direct chances of uncovering defects sounds similar a bad idea. Isn’t it?
Having said this, does this hateful repeatable tests are waste materials of time? May live not!
1) Think of Performance tests that a tester may require to run over together with over i time to a greater extent than for days inwards together with days out.
2) Think of Benchmark tests.
3) Think of Build Verification Tests/Sanity Tests that require to live run every fourth dimension y'all get got a novel build!
4) Think of Regression tests.
5) Think of a scenario where y'all require to run sure tests that are of import inwards nature. The tests that verify only about really critical functionalities of the application together with these tests must live run periodically to brand sure those functionalities drib dead on to piece of work without issues.
So, it appears that having only about repeatable tests tin live helpful for your examination projection along alongside those tests that are non-repeatable. As always, it sure enough seems to live context dependant together with depends on your testing mission/goal! What exercise y'all think? Do y'all think ALL (what that tin mean!) tests MUST live repeatable, equally if nosotros are non software testers rather only about robotic human beings trying to repeat only about algorithm already decided inwards advance! Repeatable tests! Are they the require of the hr or necessary evil? Your thoughts please.
Happy Testing…
Further Reading:
Reasons to Repeat Tests - By James Bach
Repeatability is Overrated - By Elisabeth Hendrickson