A seasoned doctor could diagnose a patient by relying on
some observations and a few questions. Only relevant
laboratory tests may be required to confirm the diagnosis
and not all the tests available in the list published by the
lab. Could the same be said of ‘Software Testers?’ Yes,
experienced testers could identify defect prone areas by
gathering a few vital inputs, without having to visit the
entire application. Intuition born out of in-depth domain
knowledge and considerable experience with the relevant
application makes this possible.
Is there anything to be gained by identifying defect prone
areas in advance? Yes, it saves time in cutting through
the chaff to the heart of sensitive areas. This becomes
critical when testing timelines are to be crunched, as is
invariably the case!
Testing is never enough and it is difficult to draw a line on
when and where to end such processes. They become
significant when schedules are crunched due to several
factors, such as :
• Late delivery of the system
• Phased delivery of code
• Lack of readiness of the environment/interfaces
• Data migration issues
In the Software Development Life Cycle (SDLC), when it
is crunch time, the ‘Testing’ phase is what absorbs the
delays caused by other phases. But then again, ‘Testing’
is the last chance to uncover any defects or shortcomings
in the system.
The challenge is to use the crunched window efficiently
and effectively. Towards this, are there ways to ensure
that the most critical areas are tested adequately in the
shortest possible time after due prioritization? Is there a
scientific way or is it to be purely intuitive?
The ‘Pareto Principle’ could be the answer.
‘Doing more with less’ and contrasting the ‘Vital Few
against the Trivial Many'
Pareto says that if 20% of efforts are directed to the right
areas almost 80% of the results could be achieved. This
finding is context independent. The question that remains
is how to identify the right areas? Where are they to be
found; in the beginning, end, middle or all jumbled up?
The art and science of zeroing in on the right areas is
called the
‘Defect Prediction Technique’ aka
‘Risk
Based Testing’ (RBT).
This helps arrive at an optimal coverage/framework to
prioritize testing based on information relating to the
business, past trends in performance of the application
and the testing complexity.
‘Risk based testing’ (RBT), based on the above principle,
speeds up the process of unearthing critical defects at a
very early stage, thus providing enough time for
rectification of the problem and re-testing the process.
This ensures that all critical functionalities are adequately
tested and the roll-out is devoid of any critical defects
despite a compressed time schedule.
RBT is based on an optimally weighted combination of
the Technical Perspective (TP) and the Business
Perspectives (BP)
For example, in the credit card business, address printing
in a card statement or intimation of change in credit limits
may be a simple functionality and may not have a high
score on technical complexity. However, as both are
customer touch points, they carry a relatively high score
on business criticality.
• Not receiving the monthly credit card statement is bad
enough, but consider your receiving your boss’
statement and the boss coming to know about it.
Though there is no financial loss, imagine the
embarrassment and the reputational risk.
• Unlikely though, take the case of a customer’s credit
limit being lowered without advance intimation.
Imagine his embarrassment when his card is refused
for having exceeded the lowered limit. Both examples
have a low impact on the technical perspective (LTP)
and a high impact on the business perspective (HBP).
When dealing with Treasury and Capital Markets,
capturing of settlement instructions properly in respect of
payments to be made through third-party banks,
‘settlement-Instructions’ setup, ‘limits management’ and
Know your Customer (KYC) compliance are
functionalities that are moderate on technical complexity
but very high on business criticality.
• Imagine a situation where a customer’s current
address is not updated as per KYC norms and a new
checkbook meant for him does not reach him because
it is mailed (not hand delivered henceforth as per
current policy) to his earlier address. This is a case that
is moderate on technical perspective (MTP) and high
on business perspective (HBP)!
In the insurance business, an error in the premium
calculation logic would have a low impact on the technical
perspective (LTP), as it could be easily corrected
retrospectively from a programming perspective. But it
could have a high impact on the business perspective
(HBP) if the error results in over-recoveries through
inflated EMIs, however small they may be, from the
insurer and if she/he comes to know about it years later.
There could be some functionalities that are critical from
both perspectives such as General Ledger (GL)
reconciliations, interest calculations, fee calculations, real
time exposure monitoring, user privileges and handling
the information handshake routine between the core
application and interfaces like sending chip related
embosser information to the card embosser. They carry
the highest risk rating; HTP and HBP and would carry the
highest priority in mitigating the risks.
In practice the gradations across TP and BP are not just
three; low, medium and high, but could be much wider
and finer, ranging from TP1 at the lower end to perhaps
as high as TP15 at the higher end. It is the same case
with the business perspective! The weightages given to
the TP attributes could also be different from those given
to BP attributes and could vary from application to
application and across time, depending on changes in
the regulatory environment and market conditions. Hence
the need for informed decision making in the risk
prioritization process should be backed by the collective
wisdom gained through years of data-flow,pattern
recognition experience and in-depth domain knowledge -
which in turn translates into intuition.
The factors that determine the value of the above two
parameters may vary, based on:
• Testing Complexity
• Risk disposition of the client
• Maturity of the product/application
• Level of customization
• Initial roll-out or Upgrade/Release
• Product release or client specific release
• Business/operational areas impacted
• Defect history of the areas to be tested
A sample list of factors used to arrive at the risk score are:
Probability of Failure:
• Changes in core application
• Interface related changes
• Testing complexity
• Past history of defects
Consequence of Failure:
• Loss of revenue and / or increase in operational costs
• Loss of reputation
• Customer Service breakdowns
• Legal or regulatory impact
• Impact on Go-live date
Theories are well known in the industry; however their
application often leaves much to be desired. Some tips to
increase the effectiveness:
1. The risk score of each module could determine the
code-delivery schedule, so that the most critical
module is delivered, tested and made ready on priority.
2. The amount of regression testing to be done can be
determined by the risk rating.
3. The test execution plan can be drawn based on the
RPM , so that high risk functionalities are tested first.
4. More importantly, if the progress of testing is reported
based on the order of the risk scores of the
functionalities, then it helps to determine the health of
the application more than going merely by the volume
of tests completed.
A diligent approach to risk based testing could bring up to
50% savings in time, money and effort. This is illustrated
in the Case Study below. By executing only 53% of the
planned test cases, 100% of Priority-1 defects and 95%
of Priority - 2 defects were unearthed.
Case Study :
Comprehensive Vs RPM Based
Testing
“You know more of a road by having traveled it than by all
the conjectures and descriptions in the world”
-William Hazlitt
No comments:
Post a Comment