As a Software Test Engineer, when I’m hired to test an application under development, I am aware of the looming testing tsunami. Unfortunately, many organizations are not aware of this phenomenon, so they don’t plan for it, which puts their projects at risk.
What is the testing tsunami? A tsunami is a tidal wave. The testing tsunami is the tidal wave of testing work that occurs at the end of development. Consider the workload curves in software development. The developers’ workload starts high and progressively decreases until all work is completed. I will ignore feature creep here, because this does not change the fundamental curve. Ditto for those who might say, “Doesn’t it spike here and there?” Fluctuations are not important – just the overall trend.
This is just the opposite of the testers’ workload. As more code is completed, the testers’ workload increases. New features must be tested. Old features must be retested. A full regression test should be performed before each release into production. Integration testing is required if this interfaces with other systems. Specialized testing such as load, stress, performance and security usually begins towards the end of development. Making matters worse, the largest amount of testing happens as the delivery deadline and/or budget is coming to an end. Armed with this knowledge, what should an organization do to mitigate the effects of the testing tsunami?
The way to lessen the effects of the testing tsunami is to move testing activities up as soon as possible. This means hiring testers early on in the project. While this advice is by no means new, few companies follow this practice (I may have a skewed view of this, since I am under an impression that testers tend to be hired late in the project’s lifecycle or on projects already in trouble). Make sure testers are budgeted for starting from day one of the project.
What would testers do this early in development, when the requirements haven’t even been gathered? Set up the testing infrastructure. Software development is a chaotic process. Testing requires a controlled environment, which translates into a large investment in testing infrastructure. The infrastructure needs will vary depending on the company’s testing maturity. Are defect tracking processes, dedicated test servers, testers’ workstations, test tool licenses, test databases and development processes already in-place? Project archeology (digging and understanding artifacts) may be needed if this is another attempt at a failed project or a new version of an old system.
Testers should be involved in requirements gathering meetings, but as an observer. During this time, they would be getting familiar with the application and end-users and beginning to think of some test case ideas. Requirements and specifications are usually the source material for test cases, but the information density in these documents is usually sparse and needs to be boiled down into test cases. These documents usually focus on positive scenarios. For each positive scenario, usually many negative scenarios exist that must be developed and tested.
A Proof of Concept (PoC) is another best practice for reducing the testing tsunami, especially if specs are weak or non-existent. The sooner the testers get their hands on the application under test (AUT), the better. If test automation tools are being used, the PoC becomes key, as these tools should be evaluated against it. If the tool has already been purchased, you will get a good idea of how well it understands the AUT. What will you do if your test automation tool doesn’t work with your application? Select a different tool, change your application to increase its testability or abandon test automation?
Daily builds are another best practice, especially automated daily builds. Developers check in their code whenever they finish it and each night a build is made. This has many advantages. First, developers get used to performing builds (or having builds performed). Builds are no longer a big deal, fraught with errors. Secondly, this allows testers to log bugs against these features earlier, rather than later. Why wait for five features to be completed before building and releasing into test; why create a mini-tsunami? Get it into test ASAP. Finally, it gives faster feedback to the developers, which also gives an accurate status of the project to management. It’s frightening to hear about testing organizations that will not accept code to test until the application is complete or that stop testing as soon as they find a bug. Both of these are real examples I’ve encountered.
Instead of viewing testing as a quality function, view it as a project management function. Testing is a tool that provides a true, up-to-date status of the project. When wondering, “When should we start testing?” just ask yourself “When would I like to find out something is wrong or incomplete?”
By being aware of the massive workload that awaits testers at the end of the project, do everything possible to move up testing activities as early as possible. This will lessen, although not eliminate, the testing tsunami.
Psychology Of Testing
Psychology Of Testing is a very important for identifying potential defects in a product or project. Independent thinking is very important because different perspectives judge the product or the project in different ways. Usually the developers think that the code would work and the testers would think that the code would not work. A person who becomes suddenly a tester from being a developer, he doesn't discover more defects because the psychology of a developer still exists within him. So psychology of the persons involved in testing also matters for bringing out quality in the product.
posted by ITQA at 11:41 PM 0 comments
Friday, September 15, 2006
What is software Testing
By definition Software Testing is the process of executing an application with the intent of finding defects in it. But Testing of Today's world is much more than this. Today's tester not only concentrates on only finding defects but often adds suggestions to bring out quality in the system. I feel a tester should be called as a Quality Engineer rather than a Tester as he brings in lot of quality into the system.
posted by ITQA at 6:26 AM 0 comments
Wednesday, September 06, 2006
Database Testing
Database Testing has become a hugely discussed word in the software testing industry. Database Testing has gained so much importance because every application be it client server or multitiered, has a backend. So without testing stored procedures which often have business logic in it, if the application gets released it lacks quality. Database Testers often need to know SQL queries and Database architecture. A Database checklist also helps the testers to attack the various areas in a stored procedure.
posted by ITQA at 6:26 AM 0 comments
Importance of Unit Testing
Unit Testing has always been a neglected area in many software companies. But Unit testing is very much important. The cost of fixing the defect in the Unit testing is always the least than when it gets identified in other phases. Like any other testing, Unit testing also focusses on both black box and White box testing techniques. Blackbox testing involves techniques like Boundary Value Analysis and Equivalence partitioning. And whitebox testing involves techniques like Loop coverage,path coverage,condition coverage and Error guessing etc. Unit testing if done properly can identify lot of defects in the early phase of SDLC and can save lot of revenue for the organization.
posted by ITQA at 6:17 AM 0 comments
Monday, August 07, 2006
Classes Postponement
Hi all,
After seeing large number of queries, we wanted to bring out this notice that our institute has been closed for sometime as we both the faculties are busy in our current assignments at our workplace. Once we are free we would announce programs.
Till then we will come out with different articles subjected to testing which would help the job seekers.
Regards
ITQA team
posted by ITQA at 4:31 AM 0 comments
Thursday, August 18, 2005
Why a Systematic education in Testing and Quality Assurance is needed?
Testing software systems to make sure they perform as expected and to ensure their reliability is a very critical quality control activity. Unfortunately this critical activity is being performed on almost every software project in a very ad hoc, informal manner by developers, testers, and users. The testing phase in many software projects mostly goes without planning and test teams are unable to perform adequate testing. Software testers have been given very little training and guidance on how to perform effective testing. There is much for developers and test professionals to learn.So, a systematic education is very important for a Test or QA professional.
ITQA IS IN THE PROCESS OF PROVIDING SYSTEMATIC EDUCATION TO MILLIONS OF TEST AND QA ENGINEERS BY EXPERIENCED FACULITES.SINCE THE FACULTIES ARE INDUSTRY EXPERTS, SO IT IS PROVIDING COURSES IN ADVANCED QUALITY CONCEPTS LIKE CMM,SIX SIGMA,CMMI,PROJECT MANAGEMENT.BUT SINCE THE FACULTIES ARE ACTUAL INDUSTRY PROFESSIONALS, SO PROGRAMS ARE NOT RUNNING IN PARALLEL.
AT PRESENT, CMM HAS BEEN ANNOUNCED.THE TARGET AUDIENCE WOULD BE TEST ENGINEERS AND QA ENGINEERS WITH LESS THAN 3 YEARS OF EXPERIENCE.
THIS WILL GIVE THEM ENORMOUS INSIGHT ABOUT HOW CMM WORKS IN THE ORGANIZATION
Fears within software development and testing
When working on new projects and specifically projects that deal with new technologies and approaches there is always a risk that the project does not go according to plan. When starting a new job or project that is quite new to me I always wonder what it would be like if 'I can't find any bugs' or to be more specific, 'if I can't find the important bugs'. It's probably a testers worst nightmare!Risk based selection for agile iterative lifecycle methods This overview helps project management and teams in selecting the best iterative method(s) for their projects, based on the major risks they want to tackle. The overview highlights the focus of the different methods, ie. which risks are mainly addressed. This is not a detailed course on all the methods, but references to literature are provided.
During the last decade agile software development methods and lifecycles (SDLC) became very successful and as such a lot of hype was created around them. The abundance of methods that have been developed looks scaring at first sight: agile, incremental, iterative, evolutionary, lean and extreme are some of the terms used. It is not easy to select the right method for the problem at hand.
As a project team your focus is on tackling your project's risk. Choosing the right iterative development method is an important activity early in your development lifecycle.
Compared to the waterfall approach all iterative methods have in common that their main deliverable in (almost) every iteration is an integrated, tested, executable system. All development processes (requirements, analysis and design, coding, integration and test) are performed in every iteration.
Differences between the methods lie in the kind of risks where they put the focus on, the length of iterations, the amount of documentation, the importance of modeling.
What follows is an overview of the best-known methods, their focus, and main references. Use this overview to make an educated selection for your projects.
Spiral Model (Boehm 1985)
Although not the first application of iterative development, Barry Boehm was the first to explain why iteration matters. The spiral model has explicit activities to identify risks, tackle them via prototyping and evaluate them before going to the next iteration.
A Spiral Model of Software Development and Enhancement, by Barry Boehm, IEEE Computer, May 1988
Incremental Development
Often used together with iterative development. The main focus is on a staged integration. The major risk of a big bang integration near the end of development is tackled. In traditional waterfall oriented organisation, incremental development has the advantage that you can work in an iterative way only at the backend of your lifecycle activities (coding, integration, testing) and do the front part activities (requirements, analysis & design) in a waterfall way. No difficult cultural and organisational change is needed with your business partners. A disadvantage of this approach is that you remain inflexible to requirements changes. With a fully iterative process you also do your requirements activities with every iteration, and you can cope with requirements changes, but it also requires more involvement from your customer.
Rational Unified Process (RUP)
An iterative incremental development process by IBM-Rational which is widely accepted by industry. The process is architecture driven, use case driven, and risk driven, in right balance. Compared to the agile methods there is more emphasis on documentation and visual modelling. IBM-Rational provides a whole toolsuite and web-based process asset library to support the process. RUP is highly tailorable and scalable, although the tailoring can be a challenge for somewhat smaller projects.
The Rational Unified Process: An Introduction, Philippe Kruchten,
eXtreme Programming (XP)
XP is the best known of the agile methods. The aim is on coping with unstable requirements. Emphasis is put on communication, simplicity, early deliveries and testing. The method is based on a set of core practices: test-driven development, continuous integration, pair programming, small releases, customer tests, simple design, refactoring, metaphor, collective ownership, coding standard, planning game, whole team and sustainable pace. XP works very effective and is largely productive for small highly skilled teams. The method is rather difficult to scale to large distributed teams. It can also be difficult to use in traditional development organisations.
Task-Based Software Testing
Introduction
There is a plethora of software testing techniques available to a development team. A survey by Zhu, identified over 200 unit testing techniques. However, for the services’ operational test agencies, there has been a continuing, unanswered question of how to test software’s impact on a system’s mission effectiveness. I propose a task-based approach as part of an integrated test strategy in an effort to answer this long-standing question.
Why Test?
From a speech by Lloyd K. Mosemann II, at the time the Deputy Assistant Secretary for the Air Force (Communications, Computers, and Support Systems), a customer’s concerns are:
They want systems that are on-time, within budget, that satisfy user requirements, and are reliable.
A report from the National Research Council refines the latter two concerns in his statement by presenting two broad objectives for operational testing:
1. to help certify, through significance testing, that a system’s performance satisfies its requirements as specified in the ORD and related documents, and
2. to identify any serious deficiencies in the system design that need correction before full rate production
Following the path from the system level to software, these two reasons are consistent with the two primary reasons for testing software or software intensive systems. Stated generically, these are:
1. test for defects so they can be fixed, and
2. test for confidence in the software
The literature often refers to these as “debug” and “operational” testing, respectively. Debug testing is usually conducted using a combination of functional test techniques and structural test techniques. The goal is to locate defects in the most cost-effective manner and correct the defects, ensuring the performance satisfies the user requirements. Operational testing is based on the expected usage profile for a system. The goal is to estimate the confidence in a system, ensuring the system is reliable for its intended use.
Task-Based Testing
Task-based testing, as I define it here, is a variation on operational testing. It uses current DoD doctrine and policy to build a framework for designing tests. The particular techniques are not new, rather it leverages commonly accepted techniques by placing them within the context of current DoD operational and acquisition strategies.
Task Analysis
Task-based testing, as the name implies, uses task analysis. Within the DoD, this begins with the Uniform Joint Task List and, in the case of the Air Force, is closely aligned with the Air Force Task List (AFTL). The AFTL “...provides a comprehensive framework for all of the tasks that the Air Force performs.” Through a series of hierarchical task analyses, each unit within the service creates a Mission Essential Task List (METL). The Mission Essential Tasks (METs) are “...only those tasks that represent the indispensable tasks to that particular organization.”
METLs, however, only describe “what” needs to be done, not “how” or “who.” Further task decomposition identifies the system(s) and people required to carry out a mission essential task. Another level of decomposition results in the system tasks (i.e. functions) a system must provide. This is, naturally, the level in which developers and testers are most interested. From a tester’s perspective, this framework identifies the most important functions to test by correlating functions against the mission essential tasks a system is designed to support.
This is distinctly different from the typical functional testing or “test-to-spec” approach where each function or specification carries equal importance. Ideally, there should be no function or specification which does not contribute to a task, but in reality there are often requirements, specifications, and capabilities which do not or minimally support a mission essential task. Using task analysis, one identifies those functions impacting the successful completion of mission essential tasks and highlights them for testing.
Operational Profiles
The above process alone has great benefit in identifying what functions are the most important to test. However, the task analysis above only identifies the mission essential tasks and functions, not their frequency of use. Greater utility can be gained by combining the mission essential tasks with an operational profile an estimate of the relative frequency of inputs that represent field use. This has several benefits:
“...offers a basis for reliability assessment, so that the developer can have not only the assurance of having tried to improve the software, but also has an estimate of the reliability actually achieved.”
“...provides a common base for communicating with the developers about the intended use of the system and how it will be evaluated.”
“When testing schedules and budgets are tightly constrained, this design yields the highest practical reliability because if failures are seen they would be the high frequency failures.”
The first benefit has the advantage of applying statistical techniques, both in the design of tests and in the analysis of resulting data. Software reliability estimation methods such as those are available to estimate both the expected field reliability and the rate of growth in reliability. This directly supports an answer to the long-standing question about software’s mpact on a system’s mission effectiveness as well as answering Mr. Mosemann II’s fourth concern a customer has (is it reliable).
Operational profiles are criticized as being difficult to develop. However, as part of its current operations and acquisition strategy, the DoD inherently develops an operational profile. At higher levels, this is reflected in such documents as the Analysis of Alternatives (AOA), the Operational Requirements Document (ORD), Operations Plans, Concept of Operations (CONOPS), etc. Closer to the tester’s realm is the interaction between the user and the developer which the current acquisition strategy encourages. The tester can act as a facilitator in helping the user refine his or her needs while providing insight to the developer on expected use. This highlights the second benefit above the communication between the user, developer, and tester.
The third benefit is certainly of interest in today’s environment of shrinking budgets and manpower, shorter schedules (spiral acquisition), and greater demands on a system. Despite years of improvement in the software development process, one still sees systems which have gone through intensive debug testing (statement coverage, branch coverage, etc.) and “test-to-spec,” but still fail to satisfy the customer’s concerns as stated by Mr. Mosemann II. By involving a customer early in the process to develop an operational profile, the most needed functions to support a task will be developed and tested first, increasing the likelihood of satisfying the customer’s four concerns.
Task-Based Software Testing
Task-based software testing, as defined herein, is the combination of a task analysis and an operational profile. The task analysis helps partition the input domain into mission essential tasks and the system functions which support them. Operational profiles, based on these tasks, are developed to further focus the testing effort.
Integrated Testing
Operational testing is not without its weaknesses. As a rather obvious example of this, one can raise the question, “What about a critical feature that is seldom executed?” Operational testing, or task-based testing as defined herein, does not address such questions well. Debug testing, with the explicit goal of locating defects in a cost-effective manner, is more suited to this.
Debug Testing
Debug testing is “...directed at finding as many bugs as possible, by either sampling all situations likely to produce failures (e.g., methods informed by code coverage or specification criteria), or concentrating on those that are considered most likely to produce failures (e.g., stress testing or boundary testing methods).” survey of unit testing methods are examples of debug testing methods. These include such techniques as statement testing, branch testing, basis path testing, etc. Typically associated with these methods are some criteria based on coverage, thus they are sometimes referred to as coverage methods. Debug testing is based on a tester’s hypothesis of the likely types and locations of bugs. Consequently, the effectiveness of this method depends heavily on whether the tester’s assumptions are correct.
If a developer and/or tester has a process in place to correctly identify the potential types and locations of bugs, then debug testing may be very effective at finding bugs. If a “standard” or “blind” approach is used, such as statement testing for its own sake, the testing effort may be ineffectual and wasted. A subtle hazard of debug testing is that it may uncover many failures, but in the process wastes test and repair effort without notably improving the software because the failures occur at a negligible rate during field use.
Integration of Test Methods
Historically, a system’s developer relied on debug testing (which includes functional or “test-to-spec” testing). Testing with the perspective of how the system would by employed was not seen until an operational test agency (OTA) became involved. Even on the occasions when developmental test took on an operational flavor, this is viewed as too late in the process. This historical approach to testing amplifies the weaknesses of both operational and debug testing. I propose that task-based software testing be accelerated to a much earlier point in the acquisition process. This has the potential of countering each respective method’s weaknesses with the other’s strengths. This view is supported by the current philosophy in the test community, to develop a combined test force spanning contractor, developmental, and operational test (CT/DT/OT).
Summary
Task-based software evaluation is a combination of demonstrated, existing methods (task analysis and operational testing). Its strength lies in matching well with the DoD’s current operational strategy of mission essential tasks and the acquisition community’s goal to deliver operational capability quickly. By integrating task-based software testing with existing debug testing, the risk of meeting the customer’s four concerns (on-time, within budget, satisfies requirements, and is reliable) can be reduced.Test Efficiency Vs Test Effectiveness
S. No
Test Efficiency
Test Effectiveness
1
Test efficiency=
internal in the organization
how much resources were consumed
how much of these resources were utilized
Test effectiveness =
how much the customer's requirements are satisfied by the system
how well the customer specifications are achieved by the system
how much effort is put in developing the system
2
Number of test cases executed divided by unit of time (generally per hour).
Number of defects found divided by number of test cases executed.
3
Test efficiency =
(total number of defects found in unit+integration+system) / (total number of defects found in unit+integration+system+User acceptance testing)
Test effectiveness =
(total number of defects injected +total number of defect found) / (total number of defect escaped)* 100
4
Test Efficiency: Test the amount of code and testing resources required by a program to perfirm a function
Test Effectivness: It judge the Effect of the test enviornment on the application
5
Testing Efficiency = (No. of defects Resolved / Total No. of Defects Submitted)* 100
Test Effectiveness = Loss due to problems / Total resources processed by the system
6
Test Efficiency is the rate of bugs found by the tester to the total bugs found.
Let me explain it more clearly:
When the build is sent to the customer side people for the testing (Alpha and Beta Testing), the customer side people also find some bugs.
Test Efficiency = A/(A+B)
Here A= Number of bugs found by the tester
B= Number of bugs found by the custome side people
This test efficiency should always be greater than 90%
What is a Test Case?
Definition of Test Case
- In software engineering, a test case is a set of conditions or variables under which a tester will determine if a requirement upon an application is partially or fully satisfied. It may take many test cases to determine that a requirement is fully satisfied. In order to fully test that all the requirements of an application are met, there must be at least one test case for each requirement unless a requirement has sub requirements. In that situation, each sub requirement must have at least one test case.
More Definitions of a Test Case.
- A test case is also defined as a sequence of steps to test the correct behavior of a functionality/feature of an application.
- A set of inputs, execution preconditions, and expected outcomes developed for a particular objective, such as to exercise a particular program path or to verify compliance with a specific requirement.
- A test case is a list of the conditions or issues of what the tester want to test in a software. Test case helps to come up with test data. A test case has an input description, Test sequence and an expected behavior.
The characteristics of a test case is that there is a known input and an expected output, which is worked out before the test. The known input should test a pre-condition and the expected output should test a post-condition.
Under special circumstances, there could be a need to run the test, produce results - and a team of experts evaluate if the results can be considered as passed. The first test is taken as the base line for subsequent test / product release cycles.
Test cases include a description of the functionality to be tested taken from either the requirements or use cases, and the preparation required to ensure that the test can be conducted.What is a Test Plan?
Definition of a Test Plan
A test plan can be defined as a document describing the scope, approach, resources, and schedule of intended testing activities. It identifies test items, the features to be tested, the testing tasks, who will do each task, and any risks requiring contingency planning.
In software testing, a test plan gives detailed testing information regarding an upcoming testing effort, including
l Scope of testing
l Schedule
l Test Deliverables
l Release Criteria
l Risks and Contingencies
It is also be described as a detail of how the testing will proceed, who will do the testing, what will be tested, in how much time the test will take place, and to what quality level the test will be performed.
Few other definitions –
The process of defining a test project so that it can be properly measured and controlled. The test planning process generates a high level test plan document that identifies the software items to be tested, the degree of tester independence, the test environment, the test case design and test measurement techniques to be used, and the rationale for their choice.
A testing plan is a methodological and systematic approach to testing a system such as a machine or software. It can be effective in finding errors and flaws in a system. In order to find relevant results, the plan typically contains experiments with a range of operations and values, including an understanding of what the eventual workflow will be.
Test plan is a document which includes, introduction, assumptions, list of test cases, list of features to be tested, approach, deliverables, resources, risks and scheduling.
A test plan is a systematic approach to testing a system such as a machine or software. The plan typically contains a detailed understanding of what the eventual workflow will be.
A record of the test planning process detailing the degree of tester indedendence, the test environment, the test case design techniques and test measurement techniques to be used, and the rationale for their choice.
Fundamentals of Software Testing
Objectives of Testing
Finding of Errors - Primary Goal
Trying to prove that software does not work. Thus, indirectly verifying that software meets requirements
Software Testing
Software testing is the process of testing the functionality and correctness of software by running it. Software testing is usually performed for one of two reasons:
(1) defect detection
(2) reliability or Process of executing a computer program and comparing the actual behavior with the expected behavior
What is the goal of Software Testing?
* Demonstrate That Faults Are Not Present
* Find Errors
* Ensure That All The Functionality Is Implemented
* Ensure The Customer Will Be Able To Get His Work Done
Modes of Testing
* Static Static Analysis doesn¡¦t involve actual program execution. The code is examined, it is tested without being executed Ex: - Reviews
* Dynamic In Dynamic, The code is executed. Ex:- Unit testing
Testing methods
* White box testing Use the control structure of the procedural design to derive test cases.
* Black box testing Derive sets of input conditions that will fully exercise the functional requirements for a program.
* Integration Assembling parts of a system
Verification and Validation
* Verification: Are we doing the job right? The set of activities that ensure that software correctly implements a specific function. (i.e. The process of determining whether or not products of a given phase of the software development cycle fulfill the requirements established during previous phase). Ex: - Technical reviews, quality & configuration audits, performance monitoring, simulation, feasibility study, documentation review, database review, algorithm analysis etc
* Validation: Are we doing the right job? The set of activities that ensure that the software that has been built is traceable to customer requirements.(An attempt to find errors by executing the program in a real environment ). Ex: - Unit testing, system testing and installation testing etc
What's a 'test case'?
A test case is a document that describes an input, action, or event and an expected response, to determine if a feature of an application is working correctly. A test case should contain particulars such as test case identifier, test case name, objective, test conditions/setup, input data requirements, steps, and expected results
What is a software error ?
A mismatch between the program and its specification is an error in the program if and only if the specifications exists and is correct.
Risk Driven Testing
What if there isn't enough time for thorough testing?
Use risk analysis to determine where testing should be focused. Since it's rarely possible to test every possible aspect of an application, every possible combination of events, every dependency, or everything that could go wrong, risk analysis is appropriate to most software development projects. This requires judgement skills, common sense, and experience.
Considerations can include:
- Which functionality is most important to the project's intended purpose?
- Which functionality is most visible to the user?
- Which aspects of the application are most important to the customer?
- Which parts of the code are most complex, and thus most subject to errors?
- What do the developers think are the highest-risk aspects of the application?
- What kinds of tests could easily cover multiple functionality?
Whenever there's too much to do and not enough time to do it, we have to prioritize so that at least the most important things get done. So prioritization has received a lot of attention. The approach is called Risk Driven Testing. Here's how you do it: Take the pieces of your system, whatever you use - modules, functions, section of the requirements - and rate each piece on two variables, Impact and Likelihood.
Risk has two components: Impact and Likelihood
Impact
is what would happen if this piece somehow malfunctioned. Would it destroy the customer database? Or would it just mean that the column headings in a report didn't quite line up?
Likelihood
is an estimate of how probable it is that this piece would fail. Together, Impact and Likelihood determine Risk for the piece.
Test Planning
What is a test plan?
A software project test plan is a document that describes the objectives, scope, approach, and focus of a software testing effort. The process of preparing a test plan is a useful way to think through the efforts needed to validate the acceptability of a software product.
Elements of test planning
* Establish objectives for each test phase
* Establish schedules for each test activity
* Determine the availability of tools, resources
* Establish the standards and procedures to be used for planning and conducting the tests and reporting test results
* Set the criteria for test completion as well as for the success of each test
The Structured Approach to Testing
Test Planning
* Define what to test
* Identify Functions to be tested
* Test conditions
* Manual or Automated
* Prioritize to identify Most Important Tests
* Record Document References
Test Design
* Define how to test
* Identify Test Specifications
* Build detailed test scripts
* Quick Script generation
* Documents
Test Execution
* Define when to test
* Build test execution schedule
* Record test results
Bug Overview
What is a software error?
A mismatch between the program and its specification is an error in the Program if and only if the specification exists and is correct.
Example: -
* The date on the report title is wrong
* The system hangs if more than 20 users try to commit at the same time
* The user interface is not standard across programs
Categories of Software errors
* User Interface errors
* Functionality errors
* Performance errors
* Output errors
* documentation errors
What Do You Do When You Find a Bug?
IF A BUG IS FOUND,
* alert the developers that a bug exists
* show them how to reproduce the bug
* ensure that if the developer fixes the bug it is fixed correctly and the fix
* didn't break anything else
* keep management apprised of the outstanding bugs and correction trends
Bug Writing Tips
Ideally you should be able to write bug report clearly enough for a developer to reproduce and fix the problem, and another QA engineer to verify the fix without them having to go back to you, the author, for more information.
To write a fully effective report you must :-
* Explain how to reproduce the problem
* Analyze the error so you can describe it in a minimum number of steps
* Write a report that is complete and easy to understand
Product Test Phase - Product Testing Cycle
Pre-Alpha
Pre-Alpha is the test period during which QA, Information Development and other internal users make the product available for internal testing.
Alpha
Alpha is the test period during which the product is complete and usable in a test environment but not necessarily bug-free. It is the final chance to get verification from customers that the tradeoffs made in the final development stage are coherent.
Entry to Alpha
* All features complete/testable (no urgent bugs or QA blockers)
* High bugs on primary platforms fixed/verified
* 50% of medium bugs on primary platforms fixed/verified
* All features tested on primary platforms
* Alpha sites ready for install
* Final product feature set Determined
Beta
Beta is the test period during which the product should be of "FCS quality" (it is complete and usable in a production environment). The purpose of the Beta ship and test period is to test the company's ability to deliver and support the product (and not to test the product itself). Beta also serves as a chance to get a final "vote of confidence" from a few customers to help validate our own belief that the product is now ready for volume shipment to all customers.
Entry to Beta
* At least 50% positive response from Alpha sites
* All customer bugs addressed via patches/drops in Alpha
* All bugs fixed/verified
* Bug fixes regression tested
* Bug fix rate exceeds find rate consistently for two weeks
* Beta sites ready for install
GM (Golden Master)
GM is the test period during which the product should require minimal work, since everything was done prior to Beta. The only planned work should be to revise part numbers and version numbers, prepare documentation for final printing, and sanity testing of the final bits.
Entry to Golden Master
* Beta sites declare the product is ready to ship
* All customer bugs addressed via patches/drops in Beta
* All negative responses from sites tracked and evaluated
* Support declares the product is supportable/ready to ship
* Bug find rate is lower than fix rate and steadily decreasing
FCS (First Customer Ship)
FCS is the period which signifies entry into the final phase of a project. At this point, the product is considered wholly complete and ready for purchase and usage by the customers.
Entry to FCS
* Product tested for two weeks with no new urgent bugs
* Product team declares the product is ready to ship
Thread Based Integration Testing
Introduction
Our organization has recently completed the development of a large-scale command and control system through the implementation and formal qualification phases of the project. This development involved over eighty software engineers developing roughly 1.5 million source lines of code using multiple languages and platforms. In order to deliver the product within the projected schedule, parallel development and rapid integration occurred over many related software functional areas. To facilitate the decomposition of our design into manageable components we chose the concept of a “functional thread” as the elementary building block for integration. In this context, a “functional thread” is defined as a logical execution sequence through a series of interfacing software components resulting from or ending in the receipt of a message, event or operator interaction.
Threads not only serve as the basis for integration, they also tend to drive the entire software development effort from scheduling to status reporting. Each thread itself represents a microcosm of the system in that each has a documented definition and general execution path, an internal design and an associated test. Thread definition intends to communicate functional background and execution details between developers and from developers to testers. More importantly, the desired independence of threads supports incremental integration and system testing while the corresponding thread definition substantiates the results. Finally, since all system development activity progresses in relation to threads, management has an accurate method of judging the status of individual tasks, functional areas and requirements.
Threads
Keeping the goals of iterative development and testing in mind, each thread has its own lifecycle with autonomous states and a formal process for state transitions (see Figure 1). Individual team leaders usually decompose general requirements into groups of threads at the beginning of formal, six month software builds and assign threads to developers. Developers maintain ownership of their threads and are responsible for documenting a scenario under which an integrator can verify the basic functionality, providing rudimentary definition to the thread. Following implementation and unit test, the developer releases the corresponding software components to a daily integration build, at which point the thread enters a “testable” state. After verifying the functionality in the integration build, the developer marks the thread “ready” for an integrator who performs more extensive testing and eventually “integrates” the thread and corresponding software components into the system. At the end of each formal build, a team of key engineers in conjunction with quality assurance checks all threads against requirements as a regression test and “finalizes” those threads which pass.
While the development team originally tracked threads manually, we quickly developed a shared database application to serve as a central repository for thread development, maintenance and tracking. The database provides a formal mechanism for defining and documenting threads, changing thread status and reporting status to project management. Moreover, the database manages references between threads: threads can serve as preconditions to other threads and developers may incorporate thread test steps from previous threads. Most importantly, the interface helps enforce the process by demonstrating the autonomy of thread status and establishing clearly defined responsibilities among developers and testers.
Thread Test Steps
Thread test steps and other background information from the database serve as a contract between developers and integrators. Integrators use thread test steps as a simple scenario to identify the scope of a thread rather than as a rigid test case that may only rubber-stamp a developer’s unit test. Consequently, the integrators are responsible for developing several execution scenarios within the boundaries of the thread and applying appropriate testing mechanisms such as known exceptional cases and boundary checking. Furthermore, the integration team often stresses exercising subsystem interfaces during integration testing, which was an area that thread steps often overlooked.
In addition to helping formalize the implementation process, the thread testing approach standardizes the integration testing process as well. As a result, the number of detected coding errors increased almost 250 percent over three formal builds after thread testing had been introduced. Although errors attributable to integration doubled during the first formal build during which our group used threads, that number has subsequently dropped to almost fifty percent below the level at which we started using threads.
While thread-based development certainly contributes greatly to the main goals of early, rapid integration and iterative development, we have also identified several potential areas of further process improvement. Perhaps most notably, developers and testers shared concerns that thread scope lacked uniformity among subsystems. At times, thread definitions were far too specific and a conscientious integrator could verify the basic functionality in fewer steps than the developer identified. Likewise, developers sometimes defined threads at too high a level, requiring the integrator to seek further information from the developer to ensure a meaningful test. A thread review process, perhaps as part of a design walk through, may answer this problem. Likewise, we recommend requiring completion of a code walk through as a prerequisite to thread completion due to the implications of walk through initiated design and code changes.
Thread Maintenance
A related area of improvement is thread maintenance. While the process encouraged (and the database supported) threads referencing other threads, maintaining consistency was not always an easy task. Furthermore, while software that composes a thread often changes after a thread has been integrated, there is no formal update process for the thread. The changes to process here are obvious and one could modify the tool to help enforce these concerns. For example, the tool would benefit from the ability to attach references to source code units so that changes to code might trigger the need for associated thread changes.
In this project the thread process focused on the integration activities rather than the full development lifecycle. This is certainly the main difference between our thread-based approach and use-case analysis. The thread database requires references to user interface specifications where applicable, but the process did not link the thread directly to the requirements database. Thus software testing and overall system testing were somewhat disjoint in that system testers found it difficult to use the thread database as a reference when creating test cases. Though it might be desirable to shift thread definition to the requirements analysis phases of the project, such analysis usually occurs at a higher level than what we had used for our threads and almost always span subsystem boundaries. Instead we suggest a more hierarchical approach to thread definition rooted in requirement-based parent threads. This would directly link the software thread repository to system requirements and better facilitate a similar iterative approach to system-wide testing. Finally, by linking threads directly to requirements, project management would have better insight about the status of entire requirements.
Since threads drove the software efforts and status, developers viewed threads as the most visible formal process in place. The simplicity of the process, accurate status and integration efficiency contributed to the development team’s acceptance of the process and enthusiasm to suggest improvements. In addition, the empirical results suggest that the introduction of thread-based testing exposed design and coding errors earlier and attributed fewer errors to the integration process itself, probably due to the enhanced communication between developers and testers. In short, our method appears to have synchronized the notion of task completion among developers, testers and management.
Summary
Thread-based integration testing played a key role in the success of this software project. At the lowest level, it provided integrators with better knowledge of the scope of what to test, in effect a contract between developers and testers. At the highest level, it provided a unified status tracking method and facilitated an agreement between management and the developers as to what would be delivered during each formal build. Furthermore, instead of testing software components directly, it required integrators to focus on testing logical execution paths in the context of the entire system. Because of this, it strongly supported the goals of early, rapid integration coupled with an iterative development approach. In summary, the thread approach resulted in tangible executable scenarios driving development and integration while the autonomous, well-defined thread states strengthened the use of threads as an accurate method of scheduling and tracking status.Testing for Zero bugs
The Software Quality Myth
A widely accepted premise on software quality is that software is so complex (in combinatorial terms) that it is impossible to have defect free software. We often hear maxims like "there's always one more bug", and "software testing can reveal the existence of bugs, but never prove their absence".
While the above maxims may be true, I'm afraid they tend to lead us to a state of mind where we accept them as an inevitable evil. If there's no way to make software clean - why bother.
Having some prior experience with software testing, I'm certain that we can do much better than we do today. Once we do this right, the results would pleasantly surprise us.
Conventional testing: How we test software?
Looking at the ways we test software, we see the following methods:
· Known scenario replay (test suites, regression tests)
· Random early exposure (Campus alphas, selected customer betas)
The known scenario replay is the single most commonly used method to test software. Unfortunately, it is the least effective method to uncover bugs as proven by the large number of bugs uncovered in later stages.
Regression tests and test suites are necessary but insufficient. They're a good way for conducting sanity checking and ensuring that popular and commonly compiled code runs correctly. However, they have two striking flaws:
· Coverage is extremely limited. When you run the same suite so many times, obviously your bugs tend to hide elsewhere.
· It is difficult to track bugs in case of failure. Test suites tend to be big pieces of code. When something goes wrong, we apply some further trial runs and search techniques until the problem is narrowed down to a single routine or line of source code.
Early exposure like alpha (and beta) testing has the advantage that it is much more random than the known-scenario replay. It provides more "real world" coverage, but it has its own flaws:
· Since it is an informal testing method, reproducibility is a problem (e.g asking a beta customer: what exactly did you do to experience the problem?")
· It relies on people's good will to investigate problems as they occur, and report bugs accurately and with enough detail when they are found.
· It suffers for a small scope (time, number of machines, and people employed) compared to the real installed base and its usage patterns.
· At least the Alpha part doesn't really emulate our customer environment: our environment is far from heterogenic: almost no other vendors' machines (Sun, HP, IBM, Mac, PCs) exist on campus.
Fortunately, there is a complementary testing method that covers the above flaws well.
Monkeys at work: Random Testing
It is said that if you give a zillion monkeys keyboards and let them type long enough, one of them would eventually produce (insert your favorite magnum opus here). Finding bugs is probably much easier than writing magnum opii. If you think this is absurd, just substitute the word "monkeys" with "customers".
For years we have been sending our software to much less than a zillion customers, and yet, without exception, they eventually hit hundreds of undiscovered bugs.
Optimization imperative #1 calls for applying zillions of computer cycles before shipment to try and beat the customers in the race to uncover the bugs. This is what random testing is all about. This is also obvious. The magic is in the details.
Constrained Random Testing: a zillion monkeys, optimized
Since the combinatorial space of software is so huge we would like the proverbial monkeys to be constrained in some way and not purely random. To give an example: if we had our monkeys test UNIX for us, we would want them to type stuff like ls, make, cc etc. rather than having them type stuff like %^3gkl'$#*(&% (*).
``Elementary,'' you may say, but read on. White Box Testing
Definition of White Box Testing - A software testing technique whereby explicit knowledge of the internal workings of the item being tested are used to select the test data.
Unlike black box testing, white box testing uses specific knowledge of programming code to examine outputs. The test is accurate only if the tester knows what the program is supposed to do. He or she can then see if the program diverges from its intended goal. White box testing does not account for errors caused by omission, and all visible code must also be readable.
Contrary to black-box testing, software is viewed as a white-box, or glass-box in white-box testing, as the structure and flow of the software under test are visible to the tester.
Testing plans are made according to the details of the software implementation, such as programming language, logic, and styles. Test cases are derived from the program structure. White-box testing is also called glass-box testing, logic-driven testing or design-based testing.
There are many techniques available in white-box testing, because the problem of intractability is eased by specific knowledge and attention on the structure of the software under test. The intention of exhausting some aspect of the software is still strong in white-box testing, and some degree of exhaustion can be achieved, such as executing each line of code at least once (statement coverage), traverse every branch statements (branch coverage), or cover all the possible combinations of true and false condition predicates (Multiple condition coverage).
Control-flow testing, loop testing, and data-flow testing, all maps the corresponding flow structure of the software into a directed graph. Test cases are carefully selected based on the criterion that all the nodes or paths are covered or traversed at least once. By doing so we may discover unnecessary "dead" code -- code that is of no use, or never get executed at all, which can not be discovered by functional testing.
In mutation testing, the original program code is perturbed and many mutated programs are created, each contains one fault. Each faulty version of the program is called a mutant. Test data are selected based on the effectiveness of failing the mutants. The more mutants a test case can kill, the better the test case is considered. The problem with mutation testing is that it is too computationally expensive to use. The boundary between black-box approach and white-box approach is not clear-cut. Many testing strategies mentioned above, may not be safely classified into black-box testing or white-box testing. It is also true for transaction-flow testing, syntax testing, finite-state testing, and many other testing strategies not discussed in this text. One reason is that all the above techniques will need some knowledge of the specification of the software under test. Another reason is that the idea of specification itself is broad -- it may contain any requirement including the structure, programming language, and programming style as part of the specification content.
We may be reluctant to consider random testing as a testing technique. The test case selection is simple and straightforward: they are randomly chosen. Study indicates that random testing is more cost effective for many programs. Some very subtle errors can be discovered with low cost. And it is also not inferior in coverage than other carefully designed testing techniques. One can also obtain reliability estimate using random testing results based on operational profiles. Effectively combining random testing with other testing techniques may yield more powerful and cost-effective testing strategies.
What is User Acceptance Testing?
Concept
UAT goes under many names. As well as user acceptance testing, it is also known as Beta Testing (usually in the PC world) QA Testing, Application Testing, End User Testing or as it is known in the company where I work as Model Office Testing.
Developing software is a expensive business. It is expensive in ;
· time, as the software must be analysed, specified, designed and written
· people, as very few development projects are one man jobs
· money, the people responsible for the analysis, specification and development of software do not come cheap (look at the current rates for contractors!)
If having expended all these peoples time and the company's money, if the resulting software is not completely suitable to the purpose required, then that time and money has not been fully utilised.
If the software is suitable to the purpose, but;
· does not dovetail precisely with the business processes
· makes processes more difficult to do than before
· causes business processes to take longer than previously
· makes additional processes necessary, with making other processes obsolete
then you may not will not see return on your investment in the software until much later, or may not even see a return on your investment.
Question : how do we ensure that we do not end up in this situation?
Answer : we test the software against objective criteria to ensure that we don't
Previously, most testing was left in the hands of the development teams, with the end users trusting those teams to deliver applications that were not only fully functional and stable, but also applications that would dovetail into business processes, and support those processes (maybe even make things a little bit easier)
However, the testing executed by developers is to ensure that the code they have created is stable and functional. They will test that;
· they cover all the lines and logic paths through their code
· all the screens flow backwards and forwards in the correct order
· the software meets the functions specified (eg calculations are correct, reports have correct columns, screens have correct validation etc)
This testing might not be done through the application itself (often because it has not been completely built while they are testing), so they will only add a few records, maybe by editing the file/table and adding the records, rather than using the 'Record Entry Screen'.
THIS IS NOT A PROBLEM.
I AM NOT DEGRADING THE SYSTEM AND UNIT TESTING DONE BY THE DEVELOPERS
As we will see later, this does not pose us a problem, because the UAT department will cover this testing. This system testing and unit testing by the developers is still very valid and useful. I would rather take delivery of an application where the development team say "We have done this, done that, hacked this file and added a few records, ran a few test cases through here - and everything seems OK", then take an application which has not gone through any system testing.
The application that has been tested by the developers will have had most of the obvious flaws identified and ironed out, and only the types of issues the testing was designed for should be identified. The second application will be completely unknown, and some of the time allocated for UAT will be spent identifying and fixing problems problems that could have been easily identified and rectified by the developers.
Also, because the developers are testing their own work, there is a tendency for them to skip areas because they 'know that there won't be a problem there'.
I have spoke to developers who have came to our company from places that do not do UAT, and they are both impressed with how we do things, but also like the idea of an independent third party testing their software.
These people are professional software developers, and they do not want to supply something that isn't exactly what's wanted, and they feel that the UA testing gives them a large comfort zone that any problems with their work will be identified and escalated back to them for correction.
As I said, these issues do not prove a problem to the user acceptance tester.
The four issues of the software delivered not matching the business process, making things more difficult etc are circumvented by the user acceptance tester.
While the developer tests against the system specification and technical documentation, the user acceptance tester test against the business requirements. The former tests the code, the latter the application. We will come to the test planning in a bit.
The issue of the developer testing their own work ceases to be an issue, as the UAT team will design a testing strategy that covers all areas of the business requirements, whether or not the developer feels there may be problems in a specific area.
The issue of additional processes being necessary should also not be a problem. As I said before, the UAT team tests the application against the business requirements, so all testing is done through the use of the proper system transactions.
The UAT team do not hack tables / file to create data, if a client record is needed for a test, then the UAT will create this client, by use of the formal client maintenance transaction, not by adding a record to the 'Client_Details' file.
This use of the formal application transaction transactions serves two purposes
· it tests all the transactions that the business users shall run, giving complete 'business' coverage (as opposed to code coverage, or logic path coverage)
· it will highlight any potential areas of adverse impacts on the business processes. If the contents of a form (eg an application form for a new life assurance policy) is used as the basis for creating a new life assurance policy record, then the use of the formal 'New Life Assurance Policy' transaction will determine whether the transaction works, and also whether the form holds the requisite information to create the policy records.
The 'New Life Assurance Policy' system may require the client to declare whether they smoke or not, however, if this question is not on the application form, then the business users will have to re-contact every new client, to determine whether or not they smoke!
We can see then, that it is the role of user acceptance testing to not only prove whether or not an application works, but also to prove how it will fit with business processes.
User Acceptance Testing Processes
OK, now we have determined what UAT is, now we need to look at HOW we achieve these objectives.
The user acceptance test life cycle follows the path shown below (obviously at a very high level);
· analysis of business requirements We can't do anything concerning testing until we understand what the developments are supposed to achieve. This is quite an intangible step in the process, and consists mostly of thought processes, meeting etc. The end result, is a clear vision,in the testers mind, of what they are going to be expected to prove, and why it is necessary.
· analysis of testing requirements. This is more tangible than the first stage, and consists of documenting the areas of the development that require testing, the methodologies you will need to use to test them, and the results to expect to be returned when you test them.
· Execution of testing. Doing the business. This is what it all boils down to. Every development project will be different, and you will have had enough experience in this part of the cycle to not need any pointers from me!
· Getting the testing signed off. There is no use going through all of these processes, raising problems to developments teams, having more work done by the development teams in fixing those problems, re-testing the changes and re-doing all your regression scripts, unless at the end of the day, you can the users to sign off the changes.
Black Box Testing
The black box testing approach is a testing method in which test data are derived from the specified functional requirements without regard to the final program structure.
It is also termed data-driven, input/output driven, or requirements-based testing. Because only the functionality of the software module is of concern, black-box testing also mainly refers to functional testing -- a testing method emphasized on executing the functions and examination of their input and output data.
The tester treats the software under test as a black box -- only the inputs, outputs and specification are visible, and the functionality is determined by observing the outputs to corresponding inputs. In testing, various inputs are exercised and the outputs are compared against specification to validate the correctness. All test cases are derived from the specification. No implementation details of the code are considered.
It is obvious that the more we have covered in the input space, the more problems we will find and therefore we will be more confident about the quality of the software. Ideally we would be tempted to exhaustively test the input space. But as stated above, exhaustively testing the combinations of valid inputs will be impossible for most of the programs, let alone considering invalid inputs, timing, sequence, and resource variables. Combinatorial explosion is the major roadblock in functional testing. To make things worse, we can never be sure whether the specification is either correct or complete.
Due to limitations of the language used in the specifications (usually natural language), ambiguity is often inevitable. Even if we use some type of formal or restricted language, we may still fail to write down all the possible cases in the specification. Sometimes, the specification itself becomes an intractable problem: it is not possible to specify precisely every situation that can be encountered using limited words. And people can seldom specify clearly what they want -- they usually can tell whether a prototype is, or is not, what they want after they have been finished. Specification problems contributes approximately 30 percent of all bugs in software.
The research in black-box testing mainly focuses on how to maximize the effectiveness of testing with minimum cost, usually the number of test cases. It is not possible to exhaust the input space, but it is possible to exhaustively test a subset of the input space. Partitioning is one of the common techniques. If we have partitioned the input space and assume all the input values in a partition is equivalent, then we only need to test one representative value in each partition to sufficiently cover the whole input space.
Domain testing partitions the input domain into regions, and consider the input values in each domain an equivalent class. Domains can be exhaustively tested and covered by selecting a representative value(s) in each domain. Boundary values are of special interest. Experience shows that test cases that explore boundary conditions have a higher payoff than test cases that do not. Boundary value analysis requires one or more boundary values selected as representative test cases. The difficulties with domain testing are that incorrect domain definitions in the specification can not be efficiently discovered.
Good partitioning requires knowledge of the software structure.
A good testing plan will not only contain black-box testing, but also white-box approaches, and combinations of the two.What is a Nighlty Test Case
Why is a nightly called a nightly?
A nightly is a test case that must be run every time there is a new build. Generally, a new build is published every night, and these tests are run nightly, hence the name, nightly tests. Other names are Acceptance or Self-Host, but the concept remains the same.
What should a nightly cover?
A good nightly test case verifies the following:
1. Is the feature usable?
2. Can the user perform basic end-to-end scenarios?
3. Most importantly, if a certain feature/scenario is broken, does the dev need to drop everything and immediately fix the issue?
Think of nightlies as regression tests. The nightlies cover the most important scenarios, so if the test case fails, a regression has occurred that the dev must investigate immediately. (If you’re wondering why don’t devs just run tests before checking in, they do – which is a topic for a later time. Or if you’re wondering why don’t devs just have the testers test the build before checking in, we do – called buddy tests – which is a topic for a later time.)
Consider creating a new text file. Putting the above into practice, a create text file nightly may verify the following:
1. Is a Text File template available on the New File Dialog? Is the Text File created upon ok’ing the New File Dialog with the text file selected?
2. Can the user insert text into the Text File?
3. Can the user perform basic editing scenarios, like cut, copy, paste, and selection?
4. Can the user save the Text File? Does the File – Save As dialog appear upon save?
5. Does the text file appear dirty upon editing it?
6. Can the user close the file?
7. Does the file appear in the Most Recently Used list?
Why doesn’t a nightly cover more?
One might think, “that isn’t very much testing for a text file,” but consider how many different languages we support, how many different OS we support, the different SKUs, and the Express Skus. After a while, the testing matrix really starts to get big, especially given that we’re not only running these tests with every new build, but we’re also analyzing the results for each new build. And, there are more than 5 builds a week, believe me, when you add in all the different flavors of runs I’ve mentioned above.
To reiterate, the point of a nightly is to ensure that the feature is “testable”, also called “Self-Test”, and to find any regressions. If nightlies are passing at 100%, the feature is said to be “Self-Host” or testable for other scenarios. Once QA signs off on a feature / build, the feature / build is said to be shippable. Once nightlies are finished running, we are able to say, “This build or feature is Self-Host." If the build fails (obviously implying no features are testable), the build is considered Self-Toast, a term used by the build lab, which is a great play-on-words. Automation Testing versus Manual Testing
I met with my team’s automation experts a few weeks back to get their input on when to automate and when to manually test. The general rule of thumb has always been to use common sense. If you’re only going to run the test one or two times or the test is really expensive to automation, it is most likely a manual test. But then again, what good is saying “use common sense” when you need to come up with deterministic set of guidelines on how and when to automate?
Pros of Automation
1. If you have to run a set of tests repeatedly, automation is a huge win for you
2. It gives you the ability to run automation against code that frequently changes to catch regressions in a timely manner
3. It gives you the ability to run automation in mainstream scenarios to catch regressions in a timely manner (see What is a Nighlty)
4. Aids in testing a large test matrix (different languages on different OS platforms). Automated tests can be run at the same time on different machines, whereas the manual tests would have to be run sequentially.
Cons of Automation
1. It costs more to automate. Writing the test cases and writing or configuring the automate framework you’re using costs more initially than running the test manually.
2. Can’t automate visual references, for example, if you can’t tell the font color via code or the automation tool, it is a manual test.
Pros of Manual
1. If the test case only runs twice a coding milestone, it most likely should be a manual test. Less cost than automating it.
2. It allows the tester to perform more ad-hoc (random testing). In my experiences, more bugs are found via ad-hoc than via automation. And, the more time a tester spends playing with the feature, the greater the odds of finding real user bugs.
Cons of Manual
1. Running tests manually can be very time consuming
2. Each time there is a new build, the tester must rerun all required tests - which after a while would become very mundane and tiresome.
Other deciding factors
1. What you automate depends on the tools you use. If the tools have any limitations, those tests are manual.
2. Is the return on investment worth automating? Is what you get out of automation worth the cost of setting up and supporting the test cases, the automation framework, and the system that runs the test cases?
Criteria for automating
There are two sets of questions to determine whether automation is right for your test case:
Is this test scenario automatable?
1. Yes, and it will cost a little
2. Yes, but it will cost a lot
3. No, it is no possible to automate
How important is this test scenario?
1. I must absolutely test this scenario whenever possible
2. I need to test this scenario regularly
3. I only need to test this scenario once in a while
If you answered #1 to both questions – definitely automate that test
If you answered #1 or #2 to both questions – you should automate that test
If you answered #2 to both questions – you need to consider if it is really worth the investment to automate
What happens if you can’t automate?
Let’s say that you have a test that you absolutely need to run whenever possible, but it isn’t possible to automate. Your options are
1. Reevaluate – do I really need to run this test this often?
2. What’s the cost of doing this test manually?
3. Look for new testing tools
4. Consider test hooksDeveloping a Test Specification
Definitions
I’ve seen the terms “Test Plan” and “Test Specification” mean slightly different things over the years. In a formal sense (at this given point in time for me), we can define the terms as follows:
1. Test Specification – a detailed summary of what scenarios will be tested, how they will be tested, how often they will be tested, and so on and so forth, for a given feature. Examples of a given feature include, “Intellisense, Code Snippets, Tool Window Docking, IDE Navigator.” Trying to include all Editor Features or all Window Management Features into one Test Specification would make it too large to effectively read.
2. Test Plan – a collection of all test specifications for a given area. The Test Plan contains a high-level overview of what is tested (and what is tested by others) for the given feature area. For example, I might want to see how Tool Window Docking is being tested. I can glance at the Window Management Test Plan for an overview of how Tool Window Docking is tested, and if I want more info, I can view that particular test specification.
If you ask a tester on another team what’s the difference between the two, you might receive different answers. In addition, I use the terms interchangeably all the time at work, so if you see me using the term “Test Plan”, think “Test Specification.”
Parts of a Test Specification
A Test Specification should consist of the following parts:
· History / Revision - Who created the test spec? Who were the developers and Program Managers (Usability Engineers, Documentation Writers, etc) at the time when the test spec was created? When was it created? When was the last time it was updated? What were the major changes at the time of the last update?
· Feature Description – a brief description of what area is being tested.
· What is tested? – a quick overview of what scenarios are tested, so people looking through this specification know that they are at the correct place.
· What is not tested? - are there any areas being covered by different people or different test specs? If so, include a pointer to these test specs.
· Nightly Test Cases – a list of the test cases and high-level description of what is tested each night (or whenever a new build becomes available). This bullet merits its own blog entry. I’ll link to it here once it is written.
· Breakout of Major Test Areas - This section is the most interesting part of the test spec where testers arrange test cases according to what they are testing. Note: in no way do I claim this to be a complete list of all possible Major Test Areas. These areas are examples to get you going.
· Specific Functionality Tests – Tests to verify the feature is working according to the design specification. This area also includes verifying error conditions.
· Security tests – any tests that are related to security. An excellent source for populating this area comes from the Writing Secure Code book.
· Accessibility Tests – This section shouldn’t be a surprised to any of my blog readers.
· Stress Tests – This section talks about what tests you would apply to stress the feature.
· Performance Tests - this section includes verifying any perf requirements for your feature.
· Edge cases – This is something I do specifically for my feature areas. I like walking through books like How to break software, looking for ideas to better test my features. I jot those ideas down under this section
· Localization / Globalization - tests to ensure you’re meeting your product’s International requirements.
Setting Test Case Priority
A Test Specification may have a couple of hundred test cases, depending on how the test cases were defined, how large the feature area is, and so forth. It is important to be able to query for the most important test cases (nightly), the next most important test cases (weekly), the next most important test cases (full test pass), and so forth. A sample prioritization for test cases may look like:
1. Highest priority (Nightly) – Must run whenever a new build is available
2. Second highest priority (Weekly) – Other major functionality tests run once every three or four builds
3. Lower priority – Run once every major coding milestoneDefect Severity and Defet Priority
This document defines the defect Severity scale for determining defect criticality and the associated defect Priority levels to be assigned to errors found in software. It is a scale which can be easily adapted to other automated test management tools.
ANSI/IEEE Std 729-1983 Glossary of Software Engineering Terminology defines Criticality as,
"A classification of a software error or fault based on an evaluation of the degree of impact that error or fault on the development or operation of a system (often used to determine whether or when a fault will be corrected)."
The severity framework for assigning defect criticality that has proven most useful in actual testing practice is a five level scale. The criticality associated with each level is based on the answers to several questions.
First, it must be determined if the defect resulted in a system failure. ANSI/IEEE Std 729-1983 defines a failure as,
"The termination of the ability of a functional unit to perform its required function."
Second, the probability of failure recovery must be determined. ANSI/IEEE 729-1983 defines failure recovery as,
"The return of a system to a reliable operating state after failure."
Third, it must be determined if the system can do this on its own or if remedial measures must be implemented in order to return the system to reliable operation.
Fourth, it must be determined if the system can operate reliably with the defect present if it is not manifested as a failure.
Fifth, it must be determined if the defect should or should not be repaired.
The following five level scale of defect criticality addresses the these questions.
The five Levels are:
1. Critical
2. Major
3. Average
4. Minor
5. Exception
1. Critical - The defect results in the failure of the complete software system, of a subsystem, or of a software unit (program or module) within the system.
2. Major - The defect results in the failure of the complete software system, of a subsystem, or of a software unit (program or module) within the system. There is no way to make the failed component(s), however, there are acceptable processing alternatives which will yield the desired result.
3. Average - The defect does not result in a failure, but causes the system to produce incorrect, incomplete, or inconsistent results, or the defect impairs the systems usability.
4. Minor - The defect does not cause a failure, does not impair usability, and the desired processing results are easily obtained by working around the defect.
5. Exception - The defect is the result of non-conformance to a standard, is related to the aesthetics of the system, or is a request for an enhancement. Defects at this level may be deferred or even ignored.
In addition to the defect severity level defined above, defect priority level can be used with severity categories to determine the immediacy of repair.
A five repair priority scale has also be used in common testing practice. The levels are:
1. Resolve Immediately
2. Give High Attention
3. Normal Queue
4. Low Priority
5. Defer
1. Resolve Immediately - Further development and/or testing cannot occur until the defect has been repaired. The system cannot be used until the repair has been effected.
2. Give High Attention - The defect must be resolved as soon as possible because it is impairing development/and or testing activities. System use will be severely affected until the defect is fixed.
3. Normal Queue - The defect should be resolved in the normal course of development activities. It can wait until a new build or version is created.
4. Low Priority - The defect is an irritant which should be repaired but which can be repaired after more serious defect have been fixed.
5. Defer - The defect repair can be put of indefinitely. It can be resolved in a future major system revision or not resolved at all.Regression Testing
Definition - Regression testing is the re-running of Test cases that a program has previously executed correctly, in order to detect failures spawned by changes or corrections made during software development and maintenance.
These failures arise from incomplete or incorrect changes and are often witnessed as (unexpected) side effects in apparently unrelated application areas. It is common in the IT industry, that one in six (or seventeen percent*) of correction attempts are themselves defective.
This high rate of introduced defects is exaggerated when developers maintain a large number of poorly documented, integrated systems where they of ten have little or no experience of these systems. Regression testing may then be used to great effect at detecting subtle side effects and unconsidered inter-relationships within these environments, thus reducing risk.
In regression testing standard actions in a test procedure are carried out and the expected responses are checked for correctness. Failure of the system to reproduce any of the expected responses imply that the system may have regressed (there may have been one or more introduced defects), or that the regression test itself may be out of date or incorrect.
If all responses are as expected, there may still have been introduced defects. In this case they escaped detection. Each defect that is reported from live or field use, having escaped detection during regression testing must be carefully analysed and the regression test suite(s) updated to catch these or similar defects in the future.
The main source of regression test cases is usually from re-use of unit, integration or system test cases. It is good practice to batch test cases into logically cohesive test suites, rather than have a single huge regression test. This allows different sub-sets of tests to be executed when there is time-pressure, or where there is confidence that only certain tests need to be run.
When first creating a regression testing suite, the choice of tests to use can be guided by the 80/20 principle. Twenty percent of system functions are likely to be used eighty percent of the time. Thus these highly used screens, transactions, menus, or fields ought to be the first candidates for regression tests. This is easy to understand if we consider one of these popular functions experiencing failure. The company call centre will be inundated with calls and the majority of users will be negatively affected. If, however, one of the less common functions has a problem, fewer users would have used it and thus discovered a lack.
Further tests to add to regression suites may be guided by risk considerations. Certain failures may not occur often, but should they occur, would result in a highly negative business impact. These higher risk business areas, modules, screens, or transactions should therefore be tested each and every time there are changes in the system and/or its environment.
Additional regression tests can be added to application areas that are known to be difficult to maintain, and have a history of high failure rates.
Regression testing can begin at the unit level, where unit tests may be adapted and rerun after changes to the unit have been effected. Regression testing should then continue through integration, system, user acceptance and operational software development life cycle phases.
As a minimum, regression tests should be run prior to build releases into the broader community and/or company live Environment. These tests will help detect major anomalies that could have serious cost, schedule, productivity and/or company image Implications.
Web systems, and other multi-user systems might have ongoing regression tests run at regular intervals. For example, one such test may check that all hyperlinks on a web site remain correct and reachable. Links to other sites may become outdated, or may even be corrupted by hackers in a security breach.
Regression testing at regular intervals can also answer production questions such as: "Is the performance of our major transactions within acceptable time limits?", "or", is some factor slowing our response times on an important transaction?"
Regression tests of non-functional application attributes such as performance, usability or security, is also very important. A very small change to the code or design may have a significant impact on system performance for example. Also please take note that debilitating changes may not even be within the application software. Changes known to have had dire consequences include an update of PC BIOS, operating system software, network cards, or updates of third party database versions for example.
Regression testing is by definition repetitive, and thus many of the tests are likely to be suited to test automation. Test automation can deliver reduced testing costs after a few test iterations when compared to labour-intensive manual testing processes.
Many companies who use regression testing conduct a very abbreviated check test (sometimes called a 'smoke' or 'sanity' test) on newly delivered code, prior to starting their formal regression tests. This often saves time as the abbreviated test commonly exposes obvious errors (for example: a whole form may not be displayed because it failed to compile against a changed database format). Removing this type of problem prior to running the time-consuming regression testing scenarios can result in getting earlier developer help, and prevent testers completing a significant portion of the regression testing before finding such problems.
It should be noted that the control and make-up of the test environment is as critical for regression testing as it is for other testing types within the same software development life cycle phase. Refer to the article on page eight that discusses creating a testing environment.
Regression test suites, be they manual or automated, are an important company asset. They therefore need to be backed up, configuration-managed, and kept current to deliver maximum benefit to their owners. Specific ownership of and responsibility for the regression test suites therefore needs to be clearly defined. Software Testing Terminology
A few simple definitions of Testing Terminology.
Error - The difference between a computed, observed, or measured value or condition and the true, specified, or theoretically correct value or condition.
Fault - An incorrect step, process, or data definition in a computer program.
Debug - To detect, locate, and correct faults in a computer program.
Failure - The inability of a system or component to perform its required functions within specified performance requirements. It is manifested as a fault.
Testing - The process of analyzing a software item to detect the differences between existing and required conditions (that is, bugs) and to evaluate the features of the software items.
Static analysis - The process of evaluating a system or component based on its form, structure, content, or documentation.
Dynamic analysis - The process of evaluating a system or component based on its behavior during execution.
Correctness - The degree to which a system or component is free from faults in its specification, design, and implementation. The degree to which software, documentation, or other items meet specified requirements. The degree to which software, documentation, or other items meet user needs and expectations, whether specified or not.
Verification - The process of evaluating a system or component to determine whether the products of a given development phase satisfy the conditions imposed at the start of that phase. Formal proof of program correctness.
Validation - The process of evaluating a system or component during or at the end of the development process to determine whether it satisfies specified requirements.
Usability Testing
What is Usability Testing?
Usability testing is the process of working with end-users directly and indirectly to asses how the user perceives a software package and how they interact with it. This process will uncover areas of difficulty for users as well as areas of strength. The goal of usability testing should be to limit and remove difficulties for users and to leverage areas of strength for maximum usability.
This testing should ideally involve direct user feedback, indirect feedback (observed behavior), and when possible computer supported feedback. Computer supported feedback is often (if not always) left out of this process. Computer supported feedback can be as simple as a timer on a dialog to monitor how long it takes users to use the dialog and counters to determine how often certain conditions occur (ie. error messages, help messages, etc). Often, this involves trivial modifications to existing software, but can result in tremendous return on investment.
Ultimately, usability testing should result in changes to the delivered product in line with the discoveries made regarding usability. These changes should be directly related to real-world usability by average users. As much as possible, documentation should be written supporting changes so that in the future, similar situations can be handled with ease.
When to Begin
This process can and should begin as early in the development process as possible. It need not involve a large number of users, in fact 3-10 is probably ideal depending on the scale of the product. These users should ideally be intended users for the product (ie. alpha/beta testers) and should represent a decent cross section of users targeted by the application.
The real question should be "when to end". I believe that this is an incremental process with many beginnings and endings. Ultimately, the job is not done until the product has reached the end of its lifecycle (not just development cycle). Each incremental step should be relatively short with changes and supporting documentation made often (up until initial delivery of the product.) Once a product is initially delivered it can be difficult to make these kind of changes without affecting active users. Once the product is shipped, changes should be considered more carefully with special concern for how active users will be affected and how future users will benefit.
However, it is never too late to start. Even if you are nearing the end of the development cycle, usability testing can still yield enormous results. Sometimes even minor changes to the UI, help system, reports, etc can make the product more appealing to users.
How to Begin
Usability testing can be quite simple. There are 4 basic ingredients that are essential to success. These are
1. The usability testing person/team needs to include a software developer who is open-minded about changes and not offended by criticism. The goal of usability testing is not to criticize, but improve and learn. If any member of the team is not ready to receive criticism with an open-mind, the testing will almost certainly fail. This person needs to have a good working knowledge of the workflow process the application is designed to facilitate and needs to have good communication skills. Good note taking skills are also essential.
2. The users selected as test subjects need to be average users (not all power users, and not all entry-level users.) They should understand what the application is designed to do and should be able to communicate their needs reasonably well. Often, these users will not be able to communicate their needs in technical terms, but will be able to identify problems they are having.
3. The development person/team needs to be prepared to make changes on an incremental basis in relatively fast paced environment. The best situation is to make needed changes quickly so that those changes can be incorporated into the continuing testing process.
4. Patience. The usability testing and refinement process can take some time and will sometimes go in the wrong direction. By keeping changes small and incremental, it is usually easy to back-track and rework the problem without significant setbacks.
There are 3 methods of feedback that need to be incorporated into our testing. These are:
1. Direct user feedback. This type of feedback usually occurs by having the test users use the software alone and report back to the usability team/person. Ideally, reporting back should occur on a regular basis (daily or weekly at most).
2. Observed behavior. This type of feedback can occur in conjunction with direct user feedback. This occurs when the testing team/person observes how users use the software.
3. Computer supported feedback. This type of feedback can occur on an ongoing basis throughout the testing process. As mentioned this is usually quite simple involving timers and hit counters.
Each of these feedback methods should be used to achieve the ultimate goal.Unit, Component and Integration Testing
Unit, Component and Integration testing
The following definitions are from a posting by Boris Beizer on
the topic of "integration testing" in the c.s.t. newsgroup.
The definitions of integration tests are after Leung and White.
Note that the definitions of unit, component, integration, and
integration testing are recursive:
Unit - The smallest compilable component. A unit typically is the
work of one programmer (At least in principle). As defined, it does
not include any called sub-components (for procedural languages) or
communicating components in general.
Unit Testing: in unit testing called components (or communicating
components) are replaced with stubs, simulators, or trusted
components. Calling components are replaced with drivers or trusted
super-components. The unit is tested in isolation.
Component: a unit is a component. The integration of one or more
components is a component.
Note: The reason for "one or more" as contrasted to "Two or
more" is to allow for components that call themselves
recursively.
Component testing: the same as unit testing except that all stubs
and simulators are replaced with the real thing.
Two components (actually one or more) are said to be integrated when:
a. They have been compiled, linked, and loaded together.
b. They have successfully passed the integration tests at the
interface between them.
Thus, components A and B are integrated to create a new, larger,
component (A,B). Note that this does not conflict with the idea of
incremental integration -- it just means that A is a big component
and B, the component added, is a small one.
Integration testing: carrying out integration tests.
Integration tests (After Leung and White) for procedural languages.
This is easily generalized for OO languages by using the equivalent
constructs for message passing. In the following, the word "call"
is to be understood in the most general sense of a data flow and is
not restricted to just formal subroutine calls and returns -- for
example, passage of data through global data structures and/or the
use of pointers.
Let A and B be two components in which A calls B.
Let Ta be the component level tests of A
Let Tb be the component level tests of B
Tab The tests in A's suite that cause A to call B.
Tbsa The tests in B's suite for which it is possible to sensitize A
-- the inputs are to A, not B.
Tbsa + Tab == the integration test suite (+ = union).
Note: Sensitize is a technical term. It means inputs that will
cause a routine to go down a specified path. The inputs are to
A. Not every input to A will cause A to traverse a path in
which B is called. Tbsa is the set of tests which do cause A to
follow a path in which B is called. The outcome of the test of
B may or may not be affected.
There have been variations on these definitions, but the key point is
that it is pretty darn formal and there's a goodly hunk of testing
theory, especially as concerns integration testing, OO testing, and
regression testing, based on them.
As to the difference between integration testing and system testing.
System testing specifically goes after behaviors and bugs that are
properties of the entire system as distinct from properties
attributable to components (unless, of course, the component in
question is the entire system). Examples of system testing issues:
resource loss bugs, throughput bugs, performance, security, recovery,
transaction synchronization bugs (often misnamed "timing bugs").
Tasks of a Test Engineer
The Test Engineer should participate on teams that design and develop the products. They should develop test protocols and test reports for the verification testing of the product to ensure that they meet product requirements and specifications. Tasks and responsibilities include:
1. Prepare verification and validation test plans, test procedures and test reports.
2. Participate in the development of product requirements and specifications.
3. Perform verification and validation testing of hardware and software.
4. Participate in customer meetings on project status and design reviews.
5. Facilitate risk analysis activities.
6. Contribute to the overall improvement in procedures.
7. Ensure that testing is performed
8. Ensure that testing is documented
9. Ensure that testing methodology, techniques, standards are established and developed
Testing personnel are encouraged to be creative & innovative, while at the same time are expected to learn and follow our structured processes.
The basic skills for a qualified test engineer:
l Controlled - Organized individual, systematic planing - Good planing on testing
l Competent - Technical awareness of testing methods, tools, and criteria
l Critical - Inner determination to discover problems
l Comprehensive - Total understanding of the given system and specifications - Pay good attention to the details
l Considerate - Ability to related to others and resolve conflicts
The basic tasks of a test engineer include:
1. Prepare testing plans and organize testing activities
2. Design test cases and document them using well-defined test methods
3. Develop test procedures and test data
4. Write problem reports and text execution records
5. Use testing tools and aids
6. Review test designs, results and problems
7. Test maintenance changes
8. Oversee acceptance tests.Fundamentals of Software Testing - In Brief
Some of the fundamentals of software testing include
- Software Test Requirements
- Software Test Design
- Software Test Planning
Let us deal each of the above one by one.
Software Test Requirements
Identifying and defining software requirements in general is a difficult job. Requirements Management is seen as the key to success in software development.
A formal approach to specifying test requirements goes a long way toward satisfying the two major complaints cited in the survey above. It formalizes the translation of software requirements to test requirements and makes the process repeatable (Capability Maturity Model level 2).
Even with the availability of the ANSI/IEEE standards, Gerrard holds that the majority of requirements documents are "often badly organized, incomplete, inaccurate and inconsistent." He also believes that the majority of documented requirements are "untestable" as written because they are presented in a form that is difficult for "testers" to understand and test against their expectations. It is up to the test engineers themselves to translate those requirements into testable ones.
There are no standards documents to guide the specification of requirements for software testing (and/or to guide the translation of existing software requirements specifications). Refining existing software requirements into software testing requirements is a very difficult task. The information a test engineer must have in order to properly test a software component is highly detailed and very specific. Granted, there are different levels and approaches to testing and test data specification (Black Box, Gray Box, and White Box views of the software) and the nature of the test requirements depends on the point of view of the test engineer. The stance of the test engineer is also dependent on the level of depth and details the software requirements specification contains. Thus, it is possible to specify test requirements from both Black Box and White Box perspectives.
Before starting test design, we must identify our test objectives, focuses, and test items. The major purpose is to help us understand what are the targets of software testing.
This step can be done based on:
1. Requirements specifications
2. Inputs from developers
3. Feedback from customers Benefits are:
4. Identify and rank the major focus of software testing
5. Check the requirements to see if they are correct, completed, and testable
6. Enhance and update system requirements to make sure they are testable
7. Support the decision on selecting or defining test strategy
Software Test Design
Software test design is an important task for software test engineers. A good test engineer always know how to come out quality test cases and perform effective tests to uncover as many as bugs in a very tight schedule.
What do you need to come out an effective test set ? -
· Choose a good test model and an effective testing method
· Apply a well-defined test criteria
· Generate a cost-effective test set based on the selected test criteria
· Write a good test case specification document What is a good test case?
It must have a high probability to discover a software error -
· It is designed to aim at a specific test requirement
· It is generated by following an effective test method
· It must be well documented and easily tracked
· It is easy to be performed and simple to spot the expected results
· It avoids the redundancy of test cases
Software Test Planing
This includes the following activities.
1. Testing activities and schedule
2. Testing tasks and assignments
3. Selected test strategy and test models
4. Test methods and criteria
5. Required test tools and environment
6. Problem tracking and reporting
7. Test cost estimationSoftware Errors
Definition of Software Error - What is a software error?
One common definition of a software error is a mismatch between the program and its specification.
Definition #1:
“A mismatch between the program and its specification is an error in the program if and only if the specification exists and is correct.”
Definition #2:
“A software error is present for when the program does not do what its end user reasonability expects to do.” (Myers, 1976)
Categories of Software Errors
1. User interface errors, such as output errors, incorrect user messages.
2. Function errors
3. Defect hardware
4. Incorrect program version
5. Testing errors
6. Requirements errors
7. Design errors
8. Documentation errors
9. Architecture errors
10. Module interface errors
11. Performance errors
12. Error handling
13. Boundary-related errors
14. Logic errors such as
· calculation errors
15. State-based behavior errors
16. Communication errors
17. Program structure errors, such as control-flow errors
Most programmers are rather cavalier about controlling the quality of the software they write. They bang out some code, run it through some fairly obvious ad hoc tests, and if it seems okay, they’re done. While this approach may work all right for small, personal programs, it doesn’t cut the mustard for professional software development. Modern software engineering practices include considerable effort directed toward software quality assurance and testing. The idea, of course, is to produce completed software systems that have a high probability of satisfying the customer’s needs.
There are two ways to deliver software free of errors. The first is to prevent the introduction of errors in the first place. And the second is to identify the bugs lurking in your code, seek them out, and destroy them. Obviously, the first method is superior. A big part of software quality comes from doing a good job of defining the requirements for the system you’re building and designing a software solution that will satisfy those requirements. Testing concentrates on detecting those errors that creep in despite your best efforts to keep them out.Few FAQ's about Testing
1. What is testing?
Software Testing can be defined as: Testing is an activity that helps in finding out bugs/defects/errors in a software system under development, in order to provide a bug free and reliable system/solution to the customer.
In other words, you can consider an example as: suppose you are a good cook and are expecting some guests at dinner. You start making dinner; you make few very very very delicious dishes (off-course, those which you already know how to make). And finally, when you are about to finish making the dishes, you ask someone (or you yourself) to check if everything is fine and there is no extra salt/chili/anything, which if is not in balance, can ruin your evening (This is what called 'TESTING').
This procedure you follow in order to make it sure that you do not serve your guests something that is not tasty! Otherwise your collar will go down and you will regret over your failure!
2. Why we go for testing?
Well, while making food, its ok to have something extra, people might understand and eat the things you made and may well appreciate your work. But this isn't the case with Software Project Development. If you fail to deliver a reliable, good and problem free software solution, you fail in your project and probably you may loose your client. This can get even worse!
So in order to make it sure, that you provide your client a proper software solution, you go for TESTING. You check out if there is any problem, any error in the system, which can make software unusable by the client. You make software testers test the system and help in finding out the bugs in the system to fix them on time. You find out the problems and fix them and again try to find out all the potential problems.
3. Why there is need of testing?
OR
Why there is a need of 'independent/separate testing'?
This is a right question because, prior to the concept of TESTING software as a ‘Testing Project’, the testing process existed, but the developer(s) did that at the time of development.
But you must know the fact that, if you make something, you hardly feel that there can be something wrong with what you have developed. It's a common trait of human nature, we feel that there is no problem in our designed system as we have developed it and it is perfectly functional and fully working. So the hidden bugs or errors or problems of the system remain hidden and they raise their head when the system goes into production.
On the other hand, its a fact that, when one person starts checking something which is made by some other person, there are 99% chances that checker/observer will find some problem with the system (even if the problem is with some spelling that by mistake has been written in wrong way.). Really weird, isn't it? But that’s a truth!
Even though its wrong in terms of human behavior, this thing has been used for the benefit of software projects (or you may say, any type of project). When you develop something, you give it to get checked (TEST) and to find out any problem, which never aroused while development of the system. Because, after all, if you could minimize the problems with the system you developed, it’s beneficial for yourself. Your client will be happy if your system works without any problem and will generate more revenues for you.
BINGO, it's really great, isn't it? That's why we need testing!
4. What is the role of "a tester"?
A tester is a person who tries to find out all possible errors/bugs in the system with the help of various inputs to it. A tester plays an important part in finding out the problems with system and helps in improving its quality.
If you could find all the bugs and fix them all, your system becomes more and more reliable.
A tester has to understand the limits, which can make the system break and work abruptly. The more number of VALID BUGS tester finds out, the better tester he is.
Testing Basics
Although it doesn't get a lot of importance, testing is arguably the most important phase in any software development project. Certainly testing is —the only— way to know whether your project or product is destined for success or doomed to failure before it goes live.
Testing involves operation of a system or application under controlled conditions and evaluating the results. The controlled conditions should include both normal and abnormal conditions. Testing should intentionally attempt to make things go wrong to determine if things happen when they shouldn't or things don't happen when they should. It is oriented to 'detection'.
The U.S. Commerce Department has estimated that buggy software costs nearly $60 billion annually, and that at least $22.2 billion worth of those bugs could have been prevented with more thorough testing.
And although you might like to think that your newly coded application is virtually bug-free, we all know by now that it just ain't so. Note that there's simply no such thing as an error-free application. There is no software that is perfect.
Defect can be caused by a flaw in the application software or by a flaw in the application specification. For example, unexpected (incorrect) results can be from errors made during the construction phase, or from an algorithm incorrectly defined in the specification. Testing is commonly assumed to mean executing software and finding errors. This type of testing is known as dynamic testing, and while valid, it is not the most effective way of testing. Static testing, the review, inspection and validation of development requirements, is the most effective and cost efficient way of testing. A structured approach to testing should use both dynamic and static testing techniques.
There are many, many different ways to test software, but most tend to fall into two broad categories: functional testing and performance testing.
Functional Testing
The functional testing focuses on the program's functionality against the specification.
Functional testing covers how well the system executes the functions it is supposed to execute—including user commands, data manipulation, searches and business processes, user screens, and integrations.
Although functional testing is often done toward the end of the development cycle, it can—and should, say experts—be started much earlier. Individual components and processes can be tested early on, even before it's possible to do functional testing on the entire system.
Performance Testing
This term is often used interchangeably with 'stress' and 'load' testing. Ideally 'performance' testing (and any other 'type' of testing) is defined in requirements documentation or QA or Test Plans. Performance testing is perhaps the biggest area of testing.
Performance testing can be defined as the testing conducted to evaluate the compliance of a system or component with specified performance requirements. Often this is performed using an automated test tool to simulate large number of users.
The different types of performance testing that can be done are:
· Classic performance testing, which measures response times and transaction rates
· Load testing, which measures the system's ability to handle varied workloads
· Stress testing, which looks for errors produced by low resources or competition for resources
· Volume testing, which subjects the software to larger and larger amounts of data to determine its point of failure
Key Objectives of Performance Testing
· Plan Approach - What is it that you are trying to measure and why? You need to understand your motive of performance testing your application
· Create a plan and a model - After planning your approach, invest time and effort into creating your capacity and performance model. The approach should include the key inputs , key items to measure and how you plan to measure.
· Implement Approach - Roll out your performance testing approach, including your measurement, profiling and benchmarking tools and processes, to start feeding your capcity model. At this point of your testing, you will most likely iterate for a defined period, inputting the data from profiling and benchmarking into your performance testing model.
How to Report Bugs Effectively
Introduction
Anybody who has written software for public use will probably have received at least one bad bug report. Reports that say nothing ("It doesn't work!"); reports that make no sense; reports that don't give enough information; reports that give wrong information. Reports of problems that turn out to be user error; reports of problems that turn out to be the fault of somebody else's program; reports of problems that turn out to be network failures.
There's a reason why technical support is seen as a horrible job to be in, and that reason is bad bug reports. However, not all bug reports are unpleasant: I maintain free software, when I'm not earning my living, and sometimes I receive wonderfully clear, helpful, informative bug reports.
In this essay I'll try to state clearly what makes a good bug report. Ideally I would like everybody in the world to read this essay before reporting any bugs to anybody. Certainly I would like everybody who reports bugs to me to have read it.
In a nutshell, the aim of a bug report is to enable the programmer to see the program failing in front of them. You can either show them in person, or give them careful and detailed instructions on how to make it fail. If they can make it fail, they will try to gather extra information until they know the cause. If they can't make it fail, they will have to ask you to gather that information for them.
In bug reports, try to make very clear what are actual facts ("I was at the computer and this happened") and what are speculations ("I think the problem might be this"). Leave out speculations if you want to, but don't leave out facts.
When you report a bug, you are doing so because you want the bug fixed. There is no point in swearing at the programmer or being deliberately unhelpful: it may be their fault and your problem, and you might be right to be angry with them, but the bug will get fixed faster if you help them by supplying all the information they need. Remember also that if the program is free, then the author is providing it out of kindness, so if too many people are rude to them
then they may stop feeling kind.
"It doesn't work."
Give the programmer some credit for basic intelligence: if the program really didn't work at all, they would probably have noticed. Since they haven't noticed, it must be working for them. Therefore, either you are doing something differently from them, or your environment is different from theirs. They need information; providing this information is the purpose of a bug report. More information is almost always better than less.
Many programs, particularly free ones, publish their list of known bugs. If you can find a list of known bugs, it's worth reading it to see if the bug you've just found is already known or not. If it's already known, it probably isn't worth reporting again, but if you think you have more information than the report in the bug list, you might want to contact the programmer anyway. They might be able to fix the bug more easily if you can give them information they didn't
already have.
This essay is full of guidelines. None of them is an absolute rule. Particular programmers have particular ways they like bugs to be reported. If the program comes with its own set of bug-reporting guidelines, read them. If the guidelines that come with the program contradict the guidelines in this essay, follow the ones that come with the program!
If you are not reporting a bug but just asking for help using the program, you should state where you have already looked for the answer to your question. ("I looked in chapter 4 and section 5.2 but couldn't find anything that told me if this is possible.") This will let the programmer know where people will expect to find the answer, so they can make the documentation easier to use.
"Show me."
One of the very best ways you can report a bug is by showing it to the programmer. Stand them in front of your computer, fire up their software, and demonstrate the thing that goes wrong. Let them watch you start the machine, watch you run the software, watch how you interact with the software, and watch what the software does in response to your inputs.
They know that software like the back of their hand. They know which parts they trust, and they know which parts are likely to have faults. They know intuitively what to watch for. By the time the software does something obviously wrong, they may well have already noticed something subtly wrong earlier which might give them a clue. They can observe everything the computer does during the test run, and they can pick out the important bits for themselves.
This may not be enough. They may decide they need more information, and ask you to show them the same thing again. They may ask you to talk them through the procedure, so that they can reproduce the bug for themselves as many times as they want. They might try varying
the procedure a few times, to see whether the problem occurs in only one case or in a family of related cases. If you're unlucky, they may need to sit down for a couple of hours with a set of
development tools and really start investigating. But the most important thing is to have the programmer looking at the computer when it goes wrong. Once they can see the problem
happening, they can usually take it from there and start trying to fix it.
"Show me how to show myself."
This is the era of the Internet. This is the era of worldwide communication. This is the era in which I can send my software to somebody in Russia at the touch of a button, and he can send me comments about it just as easily. But if he has a problem with my program, he can't have me standing in front of it while it fails. "Show me" is good when you can, but often you can't.
If you have to report a bug to a programmer who can't be present in person, the aim of the exercise is to enable them to reproduce the problem. You want the programmer to run their
own copy of the program, do the same things to it, and make it fail in the same way. When they can see the problem happening in front of their eyes, then they can deal with it.
So tell them exactly what you did. If it's a graphical program, tell them which buttons you pressed and what order you pressed them in. If it's a program you run by typing a command, show them precisely what command you typed. Wherever possible, you should provide a
verbatim transcript of the session, showing what commands you typed and what the computer output in response.
Give the programmer all the input you can think of. If the program reads from a file, you will probably need to send a copy of the file. If the program talks to another computer over a network, you probably can't send a copy of that computer, but you can at least say what kind of computer it is, and (if you can) what software is running on it.
"Works for me. So what goes wrong?"
If you give the programmer a long list of inputs and actions, and they fire up their own copy of the program and nothing goes wrong, then you haven't given them enough information. Possibly the fault doesn't show up on every computer; your system and theirs may differ in some way. Possibly you have misunderstood what the program is supposed to do, and you are both looking at exactly the same display but you think it's wrong and they know it's right.
So also describe what happened. Tell them exactly what you saw. Tell them why you think what you saw is wrong; better still, tell them exactly what you expected to see. If you say "and then it went wrong", you have left out some very important information.
If you saw error messages then tell the programmer, carefully and precisely, what they were. They are important! At this stage, the programmer is not trying to fix the problem: they're just
trying to find it. They need to know what has gone wrong, and those error messages are the computer's best effort to tell you that. Write the errors down if you have no other easy way to remember them, but it's not worth reporting that the program generated an error unless you can also report what the error message was.
In particular, if the error message has numbers in it, do let the programmer have those numbers. Just because you can't see any meaning in them doesn't mean there isn't any. Numbers contain all kinds of information that can be read by programmers, and they
are likely to contain vital clues. Numbers in error messages are there because the computer is too confused to report the error in words, but is doing the best it can to get the important information to you somehow.
At this stage, the programmer is effectively doing detective work. They don't know what's happened, and they can't get close enough to watch it happening for themselves, so they are searching for clues that might give it away. Error messages, incomprehensible strings of
numbers, and even unexplained delays are all just as important as fingerprints at the scene of a crime. Keep them!
If you are using Unix, the program may have produced a core dump. Core dumps are a particularly good source of clues, so don't throw them away. On the other hand, most programmers don't like to receive huge core files by e-mail without warning, so ask before mailing one to anybody. Also, be aware that the core file contains a record of the complete state of the program: any "secrets" involved (maybe the program was handling a personal message, or dealing with confidential data) may be contained in the core file.
"So then I tried . . ."
There are a lot of things you might do when an error or bug comes up. Many of them make the problem worse. A friend of mine at school deleted all her Word documents by mistake, and before calling in any expert help, she tried reinstalling Word, and then she tried running
Defrag. Neither of these helped recover her files, and between them they scrambled her disk to the extent that no Undelete program in the world would have been able to recover anything. If she'd only left it alone, she might have had a chance.
Users like this are like a mongoose backed into a corner: with its back to the wall and seeing certain death staring it in the face, it attacks frantically, because doing something has to be better than doing nothing. This is not well adapted to the type of problems
computers produce.
Instead of being a mongoose, be an antelope. When an antelope is confronted with something unexpected or frightening, it freezes. It stays absolutely still and tries not to attract any attention, while it stops and thinks and works out the best thing to do. (If antelopes had a technical support line, it would be telephoning it at this point.) Then, once it has decided what the safest thing to do is, it does it.
When something goes wrong, immediately stop doing anything. Don't touch any buttons at all. Look at the screen and notice everything out of the ordinary, and remember it or write it down.
Then perhaps start cautiously pressing "OK" or "Cancel", whichever seems safest. Try to develop a reflex reaction - if a computer does anything unexpected, freeze.
If you manage to get out of the problem, whether by closing down the affected program or by rebooting the computer, a good thing to do is to try to make it happen again. Programmers like problems that they can reproduce more than once. Happy programmers fix bugs faster and
more efficiently.
"I think the tachyon modulation must be
wrongly polarised."
It isn't only non-programmers who produce bad bug reports. Some of the worst bug reports I've ever seen come from programmers, and even from good programmers.
I worked with another programmer once, who kept finding bugs in his own code and trying to fix them. Every so often he'd hit a bug he couldn't solve, and he'd call me over to help. "What's gone wrong?" I'd ask. He would reply by telling me his current opinion of what needed to be fixed.
This worked fine when his current opinion was right. It meant he'd already done half the work and we were able to finish the job together. It was efficient and useful.
But quite often he was wrong. We would work for some time trying to figure out why some particular part of the program was producing incorrect data, and eventually we would discover that it wasn't, that we'd been investigating a perfectly good piece of code for half an hour, and that the actual problem was somewhere else.
I'm sure he wouldn't do that to a doctor. "Doctor, I need a prescription for Hydroyoyodyne." People know not to say that to a doctor: you describe the symptoms, the actual discomforts and aches and pains and rashes and fevers, and you let the doctor do the diagnosis of what the problem is and what to do about it. Otherwise the doctor dismisses you as a hypochondriac or crackpot, and quite rightly so.
It's the same with programmers. Providing your own diagnosis might be helpful sometimes, but always state the symptoms. The diagnosis is an optional extra, and not an alternative to giving the symptoms. Equally, sending a modification to the code to fix the problem is a useful addition to a bug report but not an adequate substitute for one.
If a programmer asks you for extra information, don't make it up! Somebody reported a bug to me once, and I asked him to try a command that I knew wouldn't work. The reason I asked him to try it was that I wanted to know which of two different error messages it would give. Knowing which error message came back would give a vital clue. But he didn't actually try it - he just mailed me back and said "No, that won't work". It took me some time to persuade him to try it for real.
Using your intelligence to help the programmer is fine. Even if your deductions are wrong, the programmer should be grateful that you at least tried to make their life easier. But report the
symptoms as well, or you may well make their life much more difficult instead.
"That's funny, it did it a moment ago."
Say "intermittent fault" to any programmer and watch their face fall. The easy problems are the ones where performing a simple sequence of actions will cause the failure to occur. The programmer can then repeat those actions under closely observed test conditions and watch what happens in great detail. Too many problems simply don't work that way: there will be programs that fail once a week, or fail once in a blue moon, or never fail when you try them in
front of the programmer but always fail when you have a deadline coming up.
Most intermittent faults are not truly intermittent. Most of them have some logic somewhere. Some might occur when the machine is running out of memory, some might occur when another program tries to modify a critical file at the wrong moment, and some might occur only in the first half of every hour! (I've actually seen one of these.)
Also, if you can reproduce the bug but the programmer can't, it could very well be that their computer and your computer are different in some way and this difference is causing the problem. I had a program once whose window curled up into a little ball in the top left corner of the screen, and sat there and sulked. But it only did it on 800x600 screens; it was fine on my 1024x768 monitor.
The programmer will want to know anything you can find out about the problem. Try it on another machine, perhaps. Try it twice or three times and see how often it fails. If it goes wrong when you're doing serious work but not when you're trying to demonstrate it, it might
be long running times or large files that make it fall over. Try to remember as much detail as you can about what you were doing to it when it did fall over, and if you see any patterns, mention them. Anything you can provide has to be some help. Even if it's only
probabilistic (such as "it tends to crash more often when Emacs is running"), it might not provide direct clues to the cause of the problem, but it might help the programmer reproduce it.
Most importantly, the programmer will want to be sure of whether they're dealing with a true intermittent fault or a machine-specific fault. They will want to know lots of details about your computer, so they can work out how it differs from theirs. A lot of these details will depend on the particular program, but one thing you should definitely be ready to provide is version numbers. The version number of the program itself, and the version number of the
operating system, and probably the version numbers of any other programs that are involved in the problem.
"So I loaded the disk on to my Windows . . ."
Writing clearly is essential in a bug report. If the programmer can't tell what you meant, you might as well not have said anything.
I get bug reports from all around the world. Many of them are from non-native English speakers, and a lot of those apologise for their poor English. In general, the bug reports with apologies for their poor English are actually very clear and useful. All the most unclear reports come from native English speakers who assume that I will understand them even if they don't make any effort to be clear or precise.
· Be specific. If you can do the same thing two different ways, state which one you used. "I selected Load" might mean "I clicked on Load" or "I pressed Alt-L". Say which you did. Sometimes it matters.
· Be verbose. Give more information rather than less. If you say too much, the programmer can ignore some of it. If you say too little, they have to come back and ask more questions. One bug report I received was a single sentence; every time I asked for more
information, the reporter would reply with another single sentence. It took me several weeks to get a useful amount of information, because it turned up one short sentence at a time.
· Be careful of pronouns. Don't use words like "it", or references like "the window", when it's unclear what they mean. Consider this: "I started FooApp. It put up a warning window. I
tried to close it and it crashed." It isn't clear what the user tried to close. Did they try to close the warning window, or the whole of FooApp? It makes a difference. Instead, you could say "I started FooApp, which put up a warning window. I tried to close the
warning window, and FooApp crashed." This is longer and more repetitive, but also clearer and less easy to misunderstand.
· Read what you wrote. Read the report back to yourself, and see if you think it's clear. If you have listed a sequence of actions which should produce the failure, try following them
yourself, to see if you missed a step.
Summary
· The first aim of a bug report is to let the programmer see the failure with their own eyes. If you can't be with them to make it fail in front of them, give them detailed instructions so that they can make it fail for themselves.
· In case the first aim doesn't succeed, and the programmer can't see it failing themselves, the second aim of a bug report is to describe what went wrong. Describe everything in
detail. State what you saw, and also state what you expected to see. Write down the error messages, especially if they have numbers in.
· When your computer does something unexpected, freeze. Do nothing until you're calm, and don't do anything that you think might be dangerous.
· By all means try to diagnose the fault yourself if you think you can, but if you do, you should still report the symptoms as well.
· Be ready to provide extra information if the programmer needs it. If they didn't need it, they wouldn't be asking for it. They aren't being deliberately awkward. Have version numbers at your fingertips, because they will probably be needed.
· Write clearly. Say what you mean, and make sure it can't be misinterpreted.
· Above all, be precise. Programmers like precision.Testing is easy, what’s the big deal?
Deepak had come down for an interview at a top MNC. He sat down in a chair, glanced around the conference hall.
“Actually,” He confessed, “even though I’m here for an interview for the testing position, I really want a job in development.” He gave his interviewer, Satish, a conspiratorial smile. “I’ve been in testing for two years,” He explained, “and I think I’ve pretty much done it all at this point. I’m ready to move on.”
Satish, who had been in testing for four times as many years and felt as if he’d just scratched the surface, was taken aback. Hmmm, he thought. Let’s see how good he is. Standing at an empty whiteboard, Satish sketched a dialog box with four fields and a button. He labeled the fields and then stepped back. “Here’s a form from a functional system. Given what you see, what tests would you want to run?” Satish handed him the pen in case he wanted to draw his ideas on the board.
“Well,” Deepak began without moving from his seat, “I’d try some boundary conditions, some negative tests, a use case or two, and I’d be done.” He looked very satisfied with his answer. Satish smiled. “And you’d be done?”
Deepak frowned as he thought for a moment, and then smiled in return. “Yup. I’d be done.” Satish’s smile dissolved. “Deepak, most of the candidates we have through here would have filled that whiteboard with tests. You’ve barely begun to explore the testing problem I posed to you.” Satish paused to let his criticism sink in.
“Want to try again?”
Deepak came up with a few more tests, but not many. He had no idea what Satish was looking for, and He didn’t particularly appreciate his attitude.
Testing is easy, He thought. What’s the big deal?
The big deal isn’t how many tests Deepak surfaced as much as his attitude toward testing. He thinks he already knows all there is to know. Satish, on the other hand, recognizes that professional growth can take a lifetime. He knows that no matter how long he spends in the testing field, he’ll still have more to learn. He takes time to nurture his skills, grow his abilities, and feed his mind. As a result, Satish will continue to progress in his testing career. Deepak’s career stalled after just two short years, while Satish is continuing to grow professionally and personally. How can you assure your continued professional growth?
Testing is almost infinite - Avoid contentment
Deepak was quite satisfied with his answer to Satish’s testing problem. He felt that he’d covered everything. But since testing is an almost infinite problem, no answer would be complete. Deepak had fallen into the trap of complacency. Testers like Satish know that no matter how many clever tests they’ve already thought up, there are more tests they could be running and that testing is infinite.
Consider the above example. How many test cases can you think you can write for this?
If we want to write one test case for each and every path that the above flowchart shows, there are 10 to the power of 14 possible paths! If we execute one test per millisecond, it would take 3.17 years to test this program!!
Whenever there is a lot of testing to be done and not enough time to do the testing, we have to prioritize so that at least the most important things get done. In testing, there's never enough time or resources. In addition, the consequences of skipping something important are severe. In such a scenario, we have to prioritize and make out how to find the most important bugs first.So prioritization has received a lot of attention. The approach is called Risk Driven Testing, where Risk has very specific meaning.
We have to prioritize testing by Risk. That makes sure that we're doing the most important things first. But we also need to allow some time to get some coverage of the less risky features and usage scenarios. Risk drives good testing!!!
Study Bugs
Satish has more than a passing curiosity about bugs. He seeks to understand them:
· Why did the software behave that way?
· How can I search out related bugs in other areas?
· What was the root cause?
It’s good if you study the bugs that have already been found in the software you’re testing. It’s even better if you study other people’s bugs, too. The next time you get together with a group of testers, swap bug stories. Meetings with other team members provide an ideal forum for a friendly chat about bugs with other testers working on different kinds of software.Risk Driven Testing
Whenever there is a lot of testing to be done and not enough time to do the testing, we have to prioritize so that at least the most important things get done.
In such a scenario, we have to prioritize and make out How to find the most important bugs first.
How to find the most important bugs first?
In testing, there's never enough time or resources. In addition, the consequences of skipping something important are severe.
So prioritization has received a lot of attention. The approach is called Risk Driven Testing, where Risk has very specific meaning.
Here's how you do it:
Take the pieces of your system, whatever you use - modules, functions, section of the requirements - and rate each piece on two variables, Impact and Likelihood.
Impact is what would happen if this piece somehow malfunctioned. Would it destroy the customer database? Or would it just mean that the column headings in a report didn't quite line up?
Likelihood is an estimate of how probable it is that this piece would fail.
Together, Impact and Likelihood determine Risk for the piece.
For example, suppose we're testing new two wheeler as they come off the production line. We only have one hour to test each vehicle.
We could easily justify spending all that time testing the brakes. After all, if the brakes don't work right, there'll be a really large Impact - literally, in this case. Therefore, while it's important to see that the vehicle runs, it's even more important to be sure that if it does run, it will stop.
Just as in testing software, we say that something "works" if it works every time, under all combinations of conditions.
So we could soak the brakes with water, freeze them, overheat them, and so on - all the standard engineering tests. We could spend the whole time doing this.
But we also have to make sure that the vehicle starts, accelerates, check that the lights, horn etc work.
Unfortunately, in the real world there's never enough time to do all the testing we'd like to do. So, as in the above example, we have to prioritize testing by Risk. That makes sure that we're doing the most important things first. But we also need to allow some time to get some coverage of the less risky features and usage scenarios.
Risk drives good testing!!!
Rapid Testing
Rapid Software Testing - The style of rapid testing is tester-centric (as opposed to technique-centric) and is a blend of heuristic testing, risk-based testing, and exploratory testing.
How is Rapid Software Testing different from normal software testing?
Software Testing practice differs from industry to industry, company to company, tester to tester and person to person. But there are some elements that most test projects have in common. Let's call those common elements "normal testing". In our experience, normal testing involves writing test cases against some kind of specification. These test cases are fragmentary
plans or procedures that loosely specify what a tester will do to test the product. The tester is then expected to perform these test cases on the product, repeatedly, throughout the course of the project.
Rapid testing differs from traditional testing in several major ways:
· Don’t Waste Time. The most rapid action is no action at all. So, in rapid testing we eliminate any activity that isn't necessary. Traditional testing, by comparison, is a bloated
mess. It takes some training and experience to know what to eliminate, of course. In general, streamline documentation and devote testing to primarily to risk areas. Don't repeat tests just because someone told you that repetition is good. Make sure that you get good information value from every test. Consider the opportunity cost of each testing activitiy.
· Mission. In Rapid Testing we don't start with a task ("write test cases"), we start with a
mission. Our mission may be "find important problems fast". If so, then writing test cases may not be the best approach to the test process. If, on the other hand, our mission is "please the FDA auditors", then we not only will have to write test cases, we'll have to write certain kinds of test cases and present them in a specifically approved format. Proceeding from an understanding of our mission, we take stock of our situation and look for the most efficient and effective actions we can take right now to move towards fulfilling that mission.
· Skills. To do any testing well requires skill. Normal testing downplays the importance of skill by focusing on the format of test documentation rather than the robustness of
tests. Rapid Testing, as we describe it, highlights skill. It isn't a mechanical technique like making microwave popcorn, or filling out forms at the DMV. Robust tests are very important, so we practice critical thinking and experimental design skills. A novice tester will not do RT very well unless supervised and coached by a senior tester who is trained (or self-trained) in the art. We hope the articles and presentations on this site will help you work on those skills.
· Risk. Normal testing is focused on functional and structural product coverage. In other words, if the product can do X, then try X. Rapid Testing focuses on important problems. We gain an understanding of the product we're testing to the point where we can imagine what kinds of problems are more likely to happen and what problems would have more impact if they happened.Then we put most of our effort into testing for those problems. Rapid Testing is concerned with uncovering the most important information as soon as possible.
· Heuristics. We must beware of overthinking the testing problem, so we use heuristics (loose translation: rules of thumb) to help us get out of analysis paralysis and test
more quickly. Heuristics are essentially reflexes-- biases to respond in a certain way-- that generally help us test the right things at the right time. Rapid Testers collect, memorize and practice using helpful heuristics. In normal testing, heuristics are also used, but testers are often not aware of them and do not fully control them.
· Exploration. Rapid Testing is also rapid learning, so we use exploratory testing. We avoid pre-scripting test cases unless there is a clear and compelling need. We prefer
to let the next test we do be influenced by the last test we did. This is a good thing, because we're not imprisoned by pre-conceived notions about what we should test, but let ourselves develop better ideas as we go. Other approaches to doing testing quickly, such as extensive test automation, run the risk that you'll end up with a lot of very fast tests that don't help you find the important problems in the product.
· Teaming. Rapid Testers cheat. That is, they do things that our former elementary school teachers would consider cheating: we figure things out in advance where possible, we borrow other people's work, we use every resource and tool we can find. For example,
one important technique of Rapid Testing is pair testing: two heads, one computer. This idea has proven valuable in the practice of Extreme Programming, and it works for testing just as well. In our experience of normal testing, testers usually work quietly and alone, rather than hunting for bugs like a pack of rapid wolves.Different Types of Testing
Functionality Testing : The functionality of the application ( i.e GUI components ). against the specifications ( eg, if we click " submit" button, the application should display ..... " confirmation dialog box")
Acceptance testing: Formal testing conducted to determine whether a system satisfies its acceptance criteria and thus whether the customer should accept the system.
Regression Testing: Testing the application for checking whether the previous features are working properly or not, after adding new features to the application.
Stress Testing: The idea of stress testing is to find the breaking point in order to find bugs that will make that break potentially harmful.
Load Testing: This is merely testing at the highest transaction arrival rate in performance testing to see the resource contention, database locks etc...
Black-box Testing: Focuses on the program's functionality against the specification.
White-box Testing: Focuses on the paths of logic.
Unit Testing: The most 'micro' scale of testing; to test particular functions or code modules. Typically done by the programmer and not by testers, as it requires detailed knowledge of the internal program design and code. Not always easily done unless the application has a well-designed architecture with tight code; may require developing test driver modules or test harnesses.
Integration Testing - testing of combined parts of an application to determine if they function together correctly. The 'parts' can be code modules, individual applications, client and server applications on a network, etc. This type of testing is especially relevant to client/server and distributed systems.
Incremental Integration Testing: Continuous testing of an application as new functionality is added; requires that various aspects of an application's functionality be independent enough to work separately before all parts of the program are completed, or that test drivers be developed as needed; done by programmers or by testers.
Functional Testing: Black-box type testing geared to functional requirements of an application; this type of testing should be done by testers.
System Testing: Black-box type testing that is based on overall requirements specifications; covers all combined parts of a system.
Sanity Testing: Typically an initial testing effort to determine if a new software version is performing well enough to accept it for a major testing effort. For example, if the new software is crashing systems every 5 minutes, bogging down systems to a crawl, or destroying databases, the software may not be in a 'sane' enough condition to warrant further testing in its current state.
Performance Testing: This term is often used interchangeably with 'stress' and 'load' testing. Ideally 'performance' testing (and any other 'type' of testing) is defined in requirements documentation or QA or Test Plans.
Usability Testing: Testing for 'user-friendliness'. Clearly this is subjective, and will depend on the targeted end-user or customer. User interviews, surveys, video recording of user sessions, and other techniques can be used. Programmers and testers are usually not appropriate as usability testers.
Installation/Uninstallation Testing: Testing of full, partial, or upgrade install/uninstall processes.
Security Testing: Testing how well the system protects against unauthorized internal or external access, willful damage, etc; may require sophisticated testing techniques.
Compatability Testing: Testing how well software performs in a particular hardware/software/operating system/network/etc. environment.
Ad-hoc Testing: Testing the application in a random manner.
Alpha Testing: Testing of an application when development is nearing completion; minor design changes may still be made as a result of such testing. Typically done by end-users or others, not by programmers or testers.
Beta Testing: Testing when development and testing are essentially completed and final bugs and problems need to be found before final release. Typically done by end-users or others, not by programmers or testers.
Bugzilla
What is BugZilla?
BugZilla is a bug tracking system(also called as issue tracking system).
Bug tracking systems allow individual or group of developers effectively to keep track of outstanding problems with their product. Bugzilla was originally in a programming language called TCL, to replace a rudimentary bug-tracking database used internally by Netscape Communications. Terry later ported Bugzilla to Perl from TCL, and in Perl it remains to this day. Most commercial defect-tracking software vendors at the time charged enormous licensing fees, and Bugzilla quickly became a favorite of the open-source crowd (with its genesis in the open-source browser project, Mozilla). It is now the de-facto standard defect-tracking system against which all others are measured.
Bugzilla boasts many advanced features. These include:
· Powerful searching
· User-configurable email notifications of bug changes
· Full change history
· Inter-bug dependency tracking and graphing
· Excellent attachment management
· Integrated, product-based, granular security schema
· Fully security-audited, and runs under Perl's taint mode
· A robust, stable RDBMS back-end
· Web, XML, email and console interfaces
· Completely customisable and/or localisable web user interface
· Extensive configurability
· Smooth upgrade pathway between versions
Why Should We Use Bugzilla?
For many years, defect-tracking software has remained principally the domain of large software development houses. Even then, most shops never bothered with bug-tracking software, and instead simply relied on shared lists and email to monitor the status of defects. This procedure is error-prone and tends to cause those bugs judged least significant by developers to be dropped or ignored.
These days, many companies are finding that integrated defect-tracking systems reduce downtime, increase productivity, and raise customer satisfaction with their systems. Along with full disclosure, an open bug-tracker allows manufacturers to keep in touch with their clients and resellers, to communicate about problems effectively throughout the data management chain. Many corporations have also discovered that defect-tracking helps reduce costs by providing IT support accountability, telephone support knowledge bases, and a common, well-understood system for accounting for unusual system or software issues.
But why should you use Bugzilla?
Bugzilla is very adaptable to various situations. Known uses currently include IT support queues, Systems Administration deployment management, chip design and development problem tracking (both pre-and-post fabrication), and software and hardware bug tracking for luminaries such as Redhat, NASA, Linux-Mandrake, and VA Systems. Combined with systems such as CVS, Bonsai, or Perforce SCM, Bugzilla provides a powerful, easy-to-use solution to configuration management and replication problems.
Bugzilla can dramatically increase the productivity and accountability of individual employees by providing a documented workflow and positive feedback for good performance. How many times do you wake up in the morning, remembering that you were supposed to do something today, but you just can't quite remember? Put it in Bugzilla, and you have a record of it from which you can extrapolate milestones, predict product versions for integration, and follow the discussion trail that led to critical decisions.
Ultimately, Bugzilla puts the power in your hands to improve your value to your employer or business while providing a usable framework for your natural attention to detail and knowledge store to flourish.Functional Testing Vs Non-Functional Testing
S. No
Functional Testing
Non-Functional Testing
1
Testing developed application against business requirements.
Functional testing is done using the functional specifications provided by the client or by using the design specifications like use cases provided by the design team.
Testing the application based on the clients and performance
requirement.
Non-Functioning testing is done based on the requirements and test scenarios defined by the client.
2
Functional testing covers
· Unit Testing
· Smoke testing / Sanity testing
· Integration Testing (Top Down,
Bottom up Testing)
· Interface & Usability Testing
· System Testing
· Regression Testing
· Pre User Acceptance Testing
(Alpha & Beta)
· User Acceptance Testing
· White Box & Black Box Testing
· Globalization & Localization
Testing
Non-Functional testing covers
· Load and Performance Testing
· Ergonomics Testing
· Stress & Volume Testing
· Compatibility & Migration Testing
· Data Conversion Testing
· Security / Penetration Testing
· Operational Readiness Testing
· Installation Testing
· Security Testing (Application
Security, Network Security,
System Security)
What is Software Testing?
The principal aim of testing is to detect faults so that they can be removed before the product is made available to customers. It is often referred to as Quality Control.
The British Standards Institution, in their standard BS7925-1, define testing as:
The process of exercising software to verify that it satisfies specified requirements and to detect faults; the measurement of software quality.
Faults in software are made for a variety of reasons, from misinterpreting the requirements through to simple typing mistakes. It is the role of software testing to reduce those faults by identifying the failures.
What is Quality Assurance?
Quality Assurance consists of the the processes and methods used to ensure quality. This may include processes such as reviewing requirements documents, source code control, change management, configuration management and of course, software testing.
How Long Does Testing Take :
The job of testing is to find bugs.
Bugs come about because people make mistakes.
People make mistakes because it’s not clear what they should be doing (poor specifications), they are not good at what they are doing (lack of experience, familiarity with system/technology) or they are not doing it well (time pressure, lack of tools or support, distractions, general bad day).
So one factor in how long this takes is how many bugs you have.
Another factor is where these bugs are, if they’re spelling mistakes then they can be identified, reproduced, resolved and retested quickly. They also won’t hold up any further testing. So these bugs are unlikely to affect the schedule too much unless there are lots of them.
However, if they’re intermittent, data or time related, if they prevent other areas of the System Under Test from being tested then they could have a dramatic affect on the schedule.
Some things that can affect the schedule:
1. Number of bugs
2. Whether bugs prevent other tests from being run
3. Speed of fixing bugs
4. Accuracy of fixing bugs
5. Ability of Test Team to learn product (from Functional Requirements or from Exploratory Testing)
6. Experience of Test Team to find bugs
7.
Why you can't test everything
Imagine you have a login screen, and the username field can be 8 characters in length.
Now, assume that you only accept the standard 128 ASCII characters.
To test every combination means you have to enter every character in each position for example
1. aaaaaaaa
2. aaaaaaab
3. aaaaaaac
4. aaaaaaba
5. etc.
Now assuming that you do this in a clever way, using some automated tool and it can check 100 combinations every second.
To test every combination will take 22,849,313 years.
Here’s the maths:
8 spaces, each one can take 128 characters
= 128 * 128 * 128 * 128 * 128 * 128 * 128 * 128 = 72057594037927936 combinations.
72057594037927936/ (100 * 60 * 60 * 24 * 365)
100 a second, 60 seconds a minute, 60 minutes an hour, 24 hours a day, 365 days a year)
Testing Standards
Table of Contents· IEEE 829 Testing Documentation
· Test Planning
· Test Execution
· Test Reporting IEEE 829 Testing Documentation
IEEE 829-1998 , also known as the 829 Standard for Software Test Documentation is an IEEE standard that specifies the form of a set of documents for use in eight defined stages of software testing. The standard specifies the format of these documents but does not stipulate whether they all must be produced, nor does it include any criteria regarding adequate content for these documents. The documents can be divided into 3 stages
Test Planning
· Test Plan : a management planning document that shows:
· How the testing will be done
· Who will do it
· What will be tested
· How long it will take
· What the test coverage will be, i.e. what quality level is required
· Test Design Specification : deciding what needs to be tested, identifying the Test Conditions.
· Test Case Specification : creating the tests to be run to cover the Test Conditions
· Test Procedure Specification : detailing how to run each test, including any set-up preconditions and the steps that need to be followed
· Test Item Transmittal Report : reporting on when tested software components have progressed from one stage of testing to the next
Test Execution
· Test Log : recording which tests cases were run, who ran them, in what order, and whether each test passed or failed
· Test Incident Report : detailing, for any test that failed, the actual versus expected result, and other information intended to throw light on why a test has failed
Test Reporting
· Test Summary Report : A management report providing any important information uncovered by the tests accomplished, and including assessments of the quality of the testing effort, the quality of the software system under test, and statistics derived from Incident Reports. The report also records what testing was done and how long it took, in order to improve any future test planning. This final document is used to indicate whether the software system under test is fit for purpose according to whether or not it has met acceptance criteria defined by project stakeholders.
Web Testing Checklist
Table of Contents· General
· Code
· Appearance
· Localisation and Language
· Navigation
· Entering a Page
· Page Functionality
· Submitting Information - Forms
· Error Prevention and Correction
· Client Environment
· See Also This page provides a number of things to think about when testing web applications. It’s not an exhaustive list but may help spark thoughts specific to the application you’re testing.
General
· Who are the users (potential users) of the application?
· On what purpose the application is going to be used
Code
· Full and Correct Doctype
· No javascript Errors (check for error symbol at bottom left of IE)
· Valid (X)HTML and CSS
· Where possible CSS is used
· Well structured code
· favicon - is one needed, does it exist?
· Metadata
Appearance
· Brand awareness
· Consistent Layout
· Look and feel
· Consistent colours and Typeface
· Page Title (also shown on taskbar when minimised)
· Status Bar messages
· Minimize, maximise and resize browser window
· Appropriate use of images, colours, available screen space
Localisation and Language
· What languages are supported?
· Are extended ASCII characters (e.g. accented characters) supported?
· Reading level and style - Does the site use language in a way that is familiar to and comfortable for its readers?
· If you have to submit information do you know in which format to enter it?
· Spelling mistakes, grammar, punctuation, acronyms?
· Limited use of Jargon?
· Cultural bias
· Are dates, currencies etc in the correct locale, is it clear what they are?
· Is sorting correct for different date formats?
· Is leap years considered in calendar controls?
· Is sorting correct for different number formats (e.g. comma is used as the decimal point in some countries).
Navigation
· Where am I? Where have I been? Where can I go?
· Clear indication of current page, if a menu is used, is the correct area highlighted?
· Clearly identifiable link back to the home page, or back the section start.
· Are links clear in their appearance and where they go?
· Links (internal and external) – is there popup ALT text?
· Can favourites/bookmarks be created, do they correctly take the user to the site without error?
· Back/Forward/Refresh/Print buttons – do they function correctly?
· Browser Back and Forward buttons – do they function correctly?
· Does logo take you to home page?
· Dead ends
· Dead links
Entering a Page
· Is it clear why you are at this page?
· Where can you navigate to the page from? Is the page always the same or is it dependent on where you came from?
· What are the default values of any fields? Are they persisted if you use the back button, or return using some functionality?
Page Functionality
· Is it clear what to do on the page?
· If a multi-step process is it clear what step you are at?
· Is end clearly identified?
Submitting Information - Forms
· Is it clear what information is required?
· Check field validation
· Minimum information (enter nothing)
· Maximum information (you can try pasting a document into the field).
· Try to enter invalid characters in fields such as ‘, %, “, *, -, [space], <, >, –,