QA plays a crucial
part in product quality and product delivery. In agile development
methodologies like scrum, QA needs to work in highly compressed test execution
cycles. One of the challenges the QA engineer's face is that the code that
worked in previous sprints often gets churned by the new features and bug fixes
in each subsequent sprint, increasing the risk of regression. Without test
automation it might be hard to achieve sufficient test coverage and provide
rapid feedback.
So
is test automation a bullet?
Before
we jump to the conclusion...we need to understand the following:
WHEN TO AUTOMATE?
Check
if your application is automation ready? Below are the questions you need to
ask yourself before rushing to automate the tests.
1. Is your application ready for automation?
We need to be careful
when automating GUI. Never start automating at the start of the project or you
might end up re-writing the automation scripts that you have written.
To start automating,
you need to have a set of stable and functional features ready. It is wise to
start automating when you are sure that these features and the GUI will not
change. If you proceed and automate the features that are undergoing continuous
changes in GUI then the cost of maintaining the script will be too high.
Predicting what will change and what will remain stable is tough to determine
when working in an agile environment as we need to embrace and welcome the
changes.
Another important
thing is not to overlook is the maintenance that is required to keep all your
scripts up to date and working. Even after spending hours on programming the
automation scripts, your automation scripts might fail when you get a new
version of the product. You need to design the scripts in such a way that it
will require the smallest amount of maintenance in the future. We should have a
plan of how we are going to accomplish that in our 'automation strategy'.
2. Do
you have skilled testers in your team?
Your QA team need
tester's with the skill-set to utilise the capabilities of the tool. No tool
can help you unless you have skilled resources to implement automation. Your
test team should capable of handling challenges that might arise in automating
the test suite. Most of the tools demand testers to code. Most of the
automation tool vendors provide training and support to the resources else you
should consider bringing in an expert from outside. Test Automation is a long
term investment. It will take time and expertise in developing and maintaining
test automation frameworks and automated scripts.
3. Have
you selected the right tool?
Choosing the right
tool is crucial to succeed in test automation. There are several factors that
need to be considered when selecting a test automation tool. Some of the tools
are free, some are expensive.
Several times after
few days of initial excitement and after automating few tests the tool often
becomes "shelf-ware" and testing returns to Manual.
Here are some
factors that you might need to consider before selecting the tool:
- Does the tool fit into current process
and infrastructure?
- Does the tool vendor support training
and provide good services,
- Do you have the budget for the tool?
- Product features: How does the tool
identify the objects? Image based recognition? Object-based recognition?
- Which programming languages does it
support? Does the QA team have the knowledge on these languages?
- If using an open source tool: Does the
tool have good community support and enough documentation?
- The list goes on…
There are risks of
the vendor going out of stock or the tool being acquired or retired. It's a
good practice to explore the tool on a small scale with a pilot project and
determine if the tool can accomplish what is needed.
We should keep in
mind that no tool is perfect, be it an open-source or commercial. We need to
consider the above factors and select the tool that serves our purpose and
provides good ROI.
WHAT TO AUTOMATE??
Did
you say "Everything"?
Then
it is an unrealistic and impractical expectation!!
1. AUTOMATE REPETITIVE STUFF!!
Testing software involves a lot of repetitive work which is tedious to do manually. People become bored and make mistakes when doing the same task over and over. People tend to do the same task differently when they are repeating it again and again. A tool will exactly reproduce what it did before, so each time it is run the result is consistent. The regression suite is the ideal candidate for automation.
Repetitive work is
excellent job for a computer to do.
2. AUTOMATE
THE TESTS THAT ARE HARD TO BE PERFORMED MANUALLY.
Assume
you are testing an app which has a feature called ‘People’ that sync's all your
contacts from various social media and email accounts. You are testing if
the ‘contact count’ displayed is accurate. The test account has 5000+ contacts.
It is not feasible count the contacts manually and check if the count displayed
is accurate.
Assume
your testing an email app and the app crashes frequently when you
'archive' around 30-35 emails in succession (one after another). When
developers ask you to replicate this issue, think how tiresome it would be to
do it manually. Let's be positive and think that you replicated the issue
after few attempts..and the developer fixes it. But that's not the end of the
story.... you need to confirmation test if the issue is really fixed…Good luck
replicating that bug again!
Automation tool
eases the life of a QA engineer in such situations and can save a lot of time
and effort by automating these annoying tests.
3.
AUTOMATE BVT/SMOKE TESTS
BVT(Build
Verification tests) are cursory tests that are helpful to determine the
stability and testability of the build. These tests need to be run every time a
build is deployed. The BVT’s are ideally quick tests that concentrate on core
functionality.
Automating BVT's and
integrating them to the continuous integration server will ensure that the
changes made have not led to regressions. One thing to keep in mind while
designing these tests is that the test should be short and should execute in
few minutes. There may be multiple commits a day it does not make sense if the
automated BVT's run for hours.
Also consider the
following while automating:
- Automation
is a great option when we have to validate the same feature with large set of
different inputs and data (Data-driven testing).
- Load
and performance testing should be ideally automated where we test the system’s
ability to handle load by creating virtual users.
WHAT NOT TO AUTOMATE?
Do not try to automate
the tests that would require human intelligence and intuition.
- Tests
which are not executed frequently or would be too cumbersome to automate.
- Tests
like exploratory testing which is based on intuition and expertise of the
tester.
- Tests
that need human intervention. An automated test should be able to run and
verify the outcome entirely unattended.
- Do
not automate the volatile features which will have frequent changes.
- Prioritize your
automation using the 20-80-20 rule. Automate 20
percent of the scripts that take 80 percent of the execution time and 20
percent of the resource skills.
CAN
TEST AUTOMATION REPLACE HUMAN TESTERS?
One of the developers
who watched me automate was fascinated and said “This
is cool stuff! Now you need not have to do anything, the computer will do
everything for you. I feel that computers will replace the testers soon”.
I replied “The
day when testers will be replaced by automated tester's is the day when
developer's will be replaced by an auto-developer or a CEO will be replaced by
an auto-CEO ...”
QA experts do not use
the word 'test automation'. They rather call it ‘automation
checks’ as
they feel testing cannot be automated, but checking can be completely
automated. Humans have the ability to do things, notice things, and
analyze things that computers cannot. Our humanity as testers plays a big role
in identifying defects while testing. A computer cannot experience or cannot
express emotions.
A
computer cannot 'determine if
the game is fun or not?' ..... 'if an app
has good user experience or a pleasant interface'..... Or to 'take decisions
using heuristics' etc. Automated test can
only find defects that you already thought of or predicted might happen.
You may ask if the
test automation can run the same test more quickly, more accurately and with
same consistency every time, is it not a better than a human tester?
No! Because running a
test quickly with more precision and consistency does not produce better
software but it would definitely reduce the time required to complete testing.
Test Automation is probably one of the most useful weapons in tester's
inventory. It has its own place in the inventory. The weapon is useless without
its wielder. Automated tests require scripts to be written and maintained;
this in turn requires a HUMAN.
Automation tools are
here to assist the testers not to replace them.
Test
Automation is certainly not a "Silver bullet".
About the Author
Prashant Hegde is a passionate tester. He leads the QA team
in Razorthink software and ensures that the products he works on meet the
highest standards in tight schedules and deadlines. Prashant loves to write and
he has authored articles for leading software testing magazines. He writes a
blog on best practices in software testing in his spare time. Prashant is a
gamer, a biker, and an author.
