My Best View of a Topic in Software Engineering

My Best View of A Topic in Software Engineering

Hendy Tanata

PAPER DISCUSSED

An Empirical Study of Regression Test Application Frequency

Jung-Min Kim; Porter, A.; Rothermel, G.

Page(s): 126 -135

Proceedings of the 22nd International Conference on Software Engineering, 2000. ICSE 2000

ABSTRACT

Regression testing is an expensive maintenance process used to revalidate modified software. Regression test selection (RTS) techniques try to lower the cost of regression testing by selecting and running a subset of all existing test cases. The authors of the paper believe that issues such as the frequency with which testing is done have a strong effect on the behavior of these techniques. Therefore, they conducted an experiment to assess the effects of test application frequency on the cost and benefits of regression test selection techniques. Their results expose essential tradeoffs that should be considered when using these techniques over a series of software releases.

Keywords: regression testing, application frequency, empirical study

1 INTRODUCTION

After modifying software, developers typically want to know that unmodified code has not been adversely affected. When such unmodified code is adversely affected, we say that regression error has occurred.

Regression test selection (RTS) technique has been proposed to reduce regression testing cost. With this approach, only a subset of test cases are selected and rerun.

The cost-benefit tradeoffs of RTS techniques is a central concern of regression testing research and techniques since in general, optimal test selection (i.e. selecting exactly the fault revealing test cases) is impossible.

A common way to study this problem has been to find or create a base and modified versions of a system and accompanying test suites. Next, a test selection algorithm is run and the size and effectiveness of the selected test suite is compared to the size and effectiveness of the original test suites.

However, one limitation to this approach is that it models regression testing as a one-time activity rather than as the continuous process it is. In practice, many companies integrate changes and then regression test on a monthly, weekly, or even daily basis.

The authors' hypothesis is that the amount of change made between regression testing sessions strongly affects the costs and benefits of different regression test techniques.

If this hypothesis is true, testing practitioners may be able to better manage and coordinate their integration and regression testing process, thereby saving time and money.

2 RELATED LECTURE TOPICS

The paper is related to the topic Software Testing Strategies in our Software Engineering course, although it is not covered in the lecture itself. It describes some regression test selection (RTS) techniques used to save cost and time and the experiment result shows the relation of testing frequency and effectiveness of RTS techniques.

3 NEW CONTRIBUTION TO THE FIELD OF SOFTWARE ENGINEERING

From the result of the experiment, testing practitioners may be able to choose a more effective regression test selection (RTS) technique depending on their project requirement. They may better manage and coordinate their integration and regression testing processes, thereby saving time and money.

However, the experiment is not a good enough representative of industrial practice because it neglects some factors. Thus, more experiment can be conducted to better observe the relation between regression test frequency and the effectiveness of regression test selection techniques.

4 ELABORATION OF TECHNIQUES

4.1 Regression Testing

Let P be a procedure or program, let P' be a modified version of P and let T be a test suite for P. A typical regression test proceeds in five steps. Each of these steps involves important problems. However, the experiment restrict its attention to step 1 - the regression test selection problem, which is to select T'T, a set of test cases to execute on P'.

4.2 Regression Test Selection Techniques

Here is a brief description of several regression test selection techniques and a representative example of each.

4.2.1 Retest-All Technique

This method reruns all test cases in T. This method may be used when test effectiveness is the utmost priority with little regard to cost.

4.2.2 Random/Ad-hoc Technique

Testers often select test cases randomly or rely on their prior knowledge or experience. One such technique is to randomly select a percentage of test cases from T.

4.2.3 Minimization Technique

This method aims to select a minimal set of test cases from T that covers all modified elements of P'. One such technique selects at least one test case in T that exercise every program statement added or modified to create P'.

4.2.4 Safe Techniques

These techniques select, under certain conditions, every test case in T that covers changed program entities in P'. One such technique selects every test case in T that exercises at least one statement that was added or modified to create P', or that has been deleted from P.

4.3 Hypotheses

H1: Test selection ratios change as test application frequency changes.

H2: Test effectiveness ratios changes as test application frequency changes.

4.4 Experimental Design

4.4.1 Variables

The experiment manipulated four independent variables:

1. The subject program (there are 8 programs, each with a variety of modified versions).

2. The test selection