Metrics Used In Testing

The Product Quality Measures –
1. Customer satisfaction index
2. Delivered defect quantities
3. Responsiveness (turnaround time) to users
4. Product volatility
5. Defect ratios
6. Defect removal efficiency
7. Complexity of delivered product
8. Test coverage
9. Cost of defects,
10. Costs of quality activities
11. Re-work
12. Reliability and Metrics for Evaluating Application System Testing.

The Product Quality Measures:
1. Customer satisfaction index : This index is surveyed before product delivery and after product delivery (and on-going on a periodic basis, using standard questionnaires).The following are analyzed:

• Number of system enhancement requests per year
• Number of maintenance fix requests per year
• User friendliness: call volume to customer service hotline
• User friendliness: training time per new user
• Number of product recalls or fix releases (software vendors)
• Number of production re-runs (in-house information systems groups)

2. Delivered defect quantities : They are normalized per function point (or per LOC) at product delivery (first 3 months or first year of operation) or Ongoing (per year of operation) by level of severity, by category or cause, e.g.: requirements defect, design defect, code defect, documentation/on-line help defect, defect introduced by fixes, etc.

3. Responsiveness (turnaround time) to users :
• Turnaround time for defect fixes, by level of severity
• Time for minor vs. major enhancements; actual vs. planned elapsed time

4. Product volatility
• Ratio of maintenance fixes (to repair the system & bring it into compliance with specifications), vs. enhancement requests (requests by users to enhance or change functionality)

5. Defect ratios
• Defects found after product delivery per function point.
• Defects found after product delivery per LOC
• Pre-delivery defects: annual post-delivery defects
• Defects per function point of the system modifications

6. Defect removal efficiency
• Number of post-release defects (found by clients in field operation), categorized by
level of severity
• Ratio of defects found internally prior to release (via inspections and testing), as a
percentage of all defects
• All defects include defects found internally plus externally (by customers) in the first
year after product delivery

7. Complexity of delivered product
• McCabe’s cyclomatic complexity counts across the system
• Halstead’s measure
• Card’s design complexity measures
• Predicted defects and maintenance costs, based on complexity measures

8. Test coverage
• Breadth of functional coverage
• Percentage of paths, branches or conditions that were actually tested
• Percentage by criticality level: perceived level of risk of paths
• The ratio of the number of detected faults to the number of predicted faults.

9. Cost of defects
• Business losses per defect that occurs during operation
• Business interruption costs; costs of work-arounds
• Lost sales and lost goodwill
• Litigation costs resulting from defects
• Annual maintenance cost (per function point)
• Annual operating cost (per function point)
• Measurable damage to your boss’s career

10. Costs of quality activities
• Costs of reviews, inspections and preventive measures
• Costs of test planning and preparation
• Costs of test execution, defect tracking, version and change control
• Costs of diagnostics, debugging and fixing
• Costs of tools and tool support
• Costs of test case library maintenance
• Costs of testing & QA education associated with the product
• Costs of monitoring and oversight by the QA organization (if separate from the
development and test organizations)

11. Re-work
• Re-work effort (hours, as a percentage of the original coding hours)
• Re-worked LOC (source lines of code, as a percentage of the total delivered LOC)
• Re-worked software components (as a percentage of the total delivered components)

12. Reliability

• Availability (percentage of time a system is available, versus the time the system is
needed to be available)
• Mean time between failure (MTBF).
• Man time to repair (MTTR)
• Reliability ratio (MTBF / MTTR)
• Number of product recalls or fix releases
• Number of production re-runs as a ratio of production runs

Metrics for Evaluating Application System Testing:
Metric = Formula Test Coverage = Number of units (KLOC/FP) tested / total size of the system.

(LOC represents Lines of Code)
Number of tests per unit size = Number of test cases per KLOC/FP (LOC represents Lines of Code).

Acceptance criteria tested = Acceptance criteria tested / total acceptance criteria

Defects per size = Defects detected / system size

Test cost (in %) = Cost of testing / total cost *100

Cost to locate defect = Cost of testing / the number of defects located

Achieving Budget = Actual cost of testing / Budgeted cost of testing

Defects detected in testing = Defects detected in testing / total system defects

Defects detected in production = Defects detected in production/system size

Quality of Testing = No of defects found during Testing/(No of defects found during testing + No of acceptance defects found after delivery) *100

Effectiveness of testing to business = Loss due to problems / total resources processed by the system.

System complaints = Number of third party complaints / number of transactions processed

Scale of Ten = Assessment of testing by giving rating in scale of 1 to 10

Source Code Analysis = Number of source code statements changed / total number of tests.

Effort Productivity = Test Planning Productivity = No of Test cases designed / Actual Effort for Design and Documentation

Test Execution Productivity = No of Test cycles executed / Actual Effort for testing

Introduction of Software Quality Management

This article gives an overview of Software Quality Management and various processes that are a part of Software Quality Management. Software Quality is a highly overused term and it may mean different things to different people. This article covers Software Quality Management, Manage Software Quality, Quality Planning, Quality Assurance, Quality Control, Importance of Documentation and What is Defect Tracking?

The definition of the ISO 8204 for quality:

“Totality of characteristics of an entity that bears on its ability to satisfy stated and implied needs.”

This means that the Software product delivered should be as per the requirements defined. We now examine a few more terms used in association with Software Quality.

Quality Planning: In the Planning Process we determine the standards that are relevant for the Software Product, the Organization and the means to achieve them.

Quality Assurance : Once the standards are defined and we start building the product. It is very important to have processes that evaluate the project performance and aim to assure that the Quality standards are being followed and the final product will be in compliance.

Quality Control: Once the software components are built the results are monitored to determine if they comply with the standards. The data collected helps in measuring the performance trends and as needed help in identifying defective pieces of code.

Software Quality Management : Software Quality Management simply stated comprises of processes that ensure that the Software Project would reach its goals. In other words the Software Project would meet the clients expectations.

The key processes of Software Quality Management fall into the following three categories:

1) Quality Planning
2) Quality Assurance
3) Quality Control

The Software Quality Management comprises of Quality Planning, Quality Assurance and Quality Control Processes. We shall now take a closer look at each of them.

1) Quality Planning : Quality Planning is the most important step in Software Quality Management. Proper planning ensures that the remaining Quality processes make sense and achieve the desired results. The starting point for the Planning process is the standards followed by the Organization. This is expressed in the Quality Policy and Documentation defining the Organization-wide standards. Sometimes additional industry standards relevant to the Software Project may be referred to as needed. Using these as inputs the Standards for the specific project are decided. The Scope of the effort is also clearly defined. The inputs for the Planning are as summarized as follows:

a. Company’s Quality Policy
b. Organization Standards
c. Relevant Industry Standards
d. Regulations
e. Scope of Work
f. Project Requirements

Using these as Inputs the Quality Planning process creates a plan to ensure that standards agreed upon are met. Hence the outputs of the Quality Planning process are:

a. Standards defined for the Project
b. Quality Plan

To create these outputs namely the Quality Plan various tools and techniques are used. These tools and techniques are huge topics and Quality Experts dedicate years of research on these topics. We would briefly introduce these tools and techniques in this article.

a. Benchmarking: The proposed product standards can be decided using the existing
performance benchmarks of similar products that already exist in the market.
b. Design of Experiments: Using statistics we determine what factors influence the Quality or
features of the end product
c. Cost of Quality: This includes all the costs needed to achieve the required Quality levels. It
includes prevention costs, appraisal costs and failure costs.
d. Other tools: There are various other tools used in the Planning process such as Cause and
Effect Diagrams, System Flow Charts, Cost Benefit Analysis, etc.

All these help us to create a Quality Management Plan for the project.

2) Quality Assurance : The Input to the Quality Assurance Processes is the Quality Plan created during Planning. Quality Audits and various other techniques are used to evaluate the performance of the project. This helps us to ensure that the Project is following the Quality Management Plan. The tools and techniques used in the Planning Process such as Design of Experiments, Cause and Effect Diagrams may also be used here, as required.

3) Quality Control :

Following are the inputs to the Quality Control Process:
- Quality Management Plan.
- Quality Standards defined for the Project
- Actual Observations and Measurements of the Work done or in Progress

The Quality Control Processes use various tools to study the Work done. If the Work done is found unsatisfactory it may be sent back to the development team for fixes. Changes to the Development process may be done if necessary.
If the work done meets the standards defined then the work done is accepted and released to the clients.

Importance of Documentation: In all the Quality Management Processes special emphasis is put on documentation. Many software shops fail to document the project at various levels. Consider a scenario where the Requirements of the Software Project are not sufficiently documented. In this case it is quiet possible that the client has a set of expectations and the tester may not know about them. Hence the testing team would not be able test the software developed for these expectations or requirements. This may lead to poor “Software Quality” as the product does not meet the expectations.

Similarly consider a scenario where the development team does not document the installation instructions. If a different person or a team is responsible for future installations they may end up making mistakes during installation, thereby failing to deliver as promised.
Once again consider a scenario where a tester fails to document the test results after executing the test cases. This may lead to confusion later. If there were an error, we would not be sure at what stage the error was introduced in the software at a component level or when integrating it with another component or due to environment on a particular server etc. Hence documentation is the key for future analysis and all Quality Management efforts.

Steps:
In a typical Software Development Life Cycle the following steps are necessary for Quality Management:

1) Document the Requirements
2) Define and Document Quality Standards
3) Define and Document the Scope of Work
4) Document the Software Created and dependencies
5) Define and Document the Quality Management Plan
6) Define and Document the Test Strategy
7) Create and Document the Test Cases  Execute Test Cases and (log) Document the Results
9) Fix Defects and document the fixes
10) Quality Assurance audits the Documents and Test Logs

Various Software Tools have been development for Quality Management. These Tools can help us track Requirements and map Test Cases to the Requirements. They also help in Defect Tracking.

What is Defect Tracking?

This is very important to ensure the Quality of the end Product. As test cases are executed at various levels defects if any are found in the Software being tested. The Defects are logged and data is collected. The Software Development fixes these defects and documents how they were fixed The testing team verifies whether the defect was really fixed and closes the defects. This information is very useful. Proper tracking ensures that all Defects were fixed. The information also helps us for future projects.
The Capability Maturity Model defines various levels of Organization based on the processes that they follow.

Level 0
The following is true for “Level 0” Organizations -
There are no Processes, tracking mechanisms, no plans. It is left to the developer or any person responsible for Quality to ensure that the product meets expectations.

Level 1 – Performed Informally
The following is true for “Level 1” Organizations –
In Such Organizations, Typically the teams work extra hard to achieve the results. There are no tracking mechanisms, standards defined. The work is done but is informal and not well documented.

Level 2 – Planned and Tracked
The following is true for “Level 2” Organizations –
There are processes within a team and the team can repeat them or follow the processes for all projects that it handles.
However the process is not standardized throughout the Organization. All the teams within the organization do not follow the same standard.

Level 3 – Well-Defined
In “Level 3” Organizations the processes are well defined and followed throughout the organization.

Level 4 – Quantitatively Controlled
In “Level 4” Organizations –
- The processes are well defined and followed throughout the organization
- The Goals are defined and the actual output is measured
- Metrics are collected and future performance can predicted

Level 5 – Continuously Improving
“Level 5” Organizations have well defined processes, which are measured and the organization has a good understanding of IT projects affect the Organizational goals.
The Organization is able to continuously improve its processes based on this understanding.

How to Start SAS Preparation

The basic book to start SAS is “SAS Certification Prep Guide- Base Programming”. This is very basic book and very use full for Beginers. Better arrange this book first.

How to Learn Business Object?

Arrange Business Object Software and install in your computer. Business Object has very good PDF book in its help, which you can be saved in you local computer. Read it, its very easy and practice on Business Object.

How to Paste Picture Hyperlink in a Cell of Excel

Set objExcel = CreateObject(”Excel.Application”)
Set objWorkbook = objExcel.WorkBooks.Open(”c:\Execution_Result_3_2_2011_7_41_41.xls”)
Set objDriverSheet = objWorkbook.Worksheets(”Global”)
objExcel.Visible=True
RowNumber=4
C = “N” & RowNumber
objExcel.ActiveSheet.range(C).Select
objDriverSheet.Hyperlinks.Add objExcel.Selection, “C:\2_3_2011 7_44 16 AM.jpg”
objDriverSheet.Range(C).WrapText = True