Presented at ESCOM 7, Wilmslow 1996
Martin Shepperd and Chris Schofield, Department of Computing
Bournemouth University, Talbot Campus
Poole, BH12 5BB, UK
Email : cschofie@bournemouth.ac.uk
Abstract
Accurate effort prediction for software projects has long been a thorny problem for software developers. Recent work has shown that estimation using analogy is a technique that can yield useful results and is in many instances more reliable and robust than the traditional approach of algorithmic modelling. This paper describes how organisations can set up effort prediction programmes based upon analogy. We emphasise the need for data collection and data collection procedures. We analyse the data collected from 33 projects at a large telecommunications company and show by tuning the analogy selection method and the variables used to characterise each project, the accuracy of the predictions can be enhanced. An overall mean magnitude of relative error (MMRE) of 37% was obtained. This is solely using data available at the specification stage of a project and thus compares very favourably with other methods.
Keywords: Effort prediction, analogy, empirical, estimation process.
1. Introduction
The goal of accurate effort prediction for software projects has long been sought both by academics and practitioners alike. Many potential solutions have been suggested to remedy this situation such as algorithmic models (e.g. COCOMO, SLIM) and expert judgement (e.g. bottom up). None of these have been shown to be convincing or consistent in solving the problem. Recent research [5], however, has demonstrated the potential of the use of analogy based estimation to provide both consistently more accurate estimates and to be more robust in dealing with datasets which contain outliers or lack continuous linear relationships.
The remainder of this short paper goes onto explain the process of estimating by analogy. This is followed by a comparison of analogy based and traditional estimation methods. We then outline the steps involved in setting up an estimation by analogy programme and illustrate some of these points with an industrial case study. We conclude by highlighting areas requiring further research and investigation.
2. Analogy
Estimating by analogy involves the characterisation of software projects with information available at the point when estimation is required. For instance a project could be characterised by the number of inputs and outputs, the number of screens and the programming language used to code it. The choice of variables will depend upon the application type, development environment and on the pragmatic basis of what is available. Once a database of completed (or source) projects has been built up, it is possible to estimate a new (or target) project by looking for similar or analogous source projects. The effort figure(s) from the source project(s) then forms the basis for the estimation, possibly with adjustments.
There are a variety of methods of measuring similarity. For our work we have chosen the simplest approach of standardising each variable so that each variable contributes an equal weight. Projects are then plotted in n-dimensional space (where there is one dimension for each variable) and Euclidean distance measured. Projects that are most similar will be nearest in the n-dimensional space.
Before an estimate is made, the process of searching for and using analogous projects can be tuned in two ways. First, the inclusion of certain project attributes can create `noise' when searching for source projects, thus a subset of attributes needs to be found that best characterises the environment. Second, the number of source analogies to find is important. If too few are used, there is the risk that a maverick project will be found. If too many are used the effects of the closest analogies may be masked. To counter this problem it is possible to apply weightings to the closest analogies.
3. Results of Estimation by Analogy
Originally, analogy estimates were created by a very labour intensive manual search. It quickly became obvious, however, that the process would need to be automated if it were to be used seriously. To this end ANGEL (ANaloGy softwarE tooL ) was developed in Visual Basic. ANGEL is an environment where data can be stored, analogies found and estimates generated.
A technique known as jack-knifing is employed by ANGEL when assessing the level of confidence that can be placed in the predictions from a given dataset. This involves successively removing each project from the database and using the remaining projects as the analogy source. The Mean Magnitude of Relative Error (MMRE) is then found by computing the mean of all the percentage errors which result from comparing the estimate with the actual effort figure. The MMRE figure gives an indication of the likely accuracy that can be expected when estimating future projects. It is also useful for technique comparison because, unlike R2, it is estimation method independent.
Name n Description Source
Albrecht 24 IBM DP Services projects [1]
Atkinson 21 Builds to a large telecomms [2]
product
Desharnais 77 Canadian software house - [3]
commercial projects
Finnish 38 Data collected by the TIEKE Finnish Dataset:
organisation from IS projects dataset made available
from 9 different Finnish to the ESPRIT Mermaid
companies. Project by the TIEKE
organisation
Kemerer 15 Large business applications [4]
Mermaid 28 New and enhancement projects MM2 Dataset: Dataset
made available to the
ESPRIT Mermaid Project
anonymously
Table 1: Datasets used to compare effort prediction methods
Six industrial datasets (Table 1) were analysed using ANGEL alongside linear and stepwise regression. The regression techniques, seen by many as `state of the art', are used to place the analogy values into perspective.
Dataset Analogy Linear Regression Stepwise
Regression
Albrecht 62% 85% 90%
Atkinson 39% 57% 45%
Desharnais 64% 66% 66%
Finnish 62% 133% 101%
Kemerer 62% 107% 107%
Mermaid 78% 252% 252%
Table 2: MMRE Values for Effort Estimation
Table 2 shows the accuracy of the respective methods. In all cases the performance of analogy is equal to or better than that of the regression methods. This suggests that analogy is a superior techniques, at least for these datasets. The Mermaid dataset is particularly interesting as it shows a dataset for which no statistically significant relationships could be found between any of the independent variables and effort. This results in the exceptionally poor MMRE of 252%. By contrast the analogy method is able to produce an overall estimation accuracy of 78%: considerably better.
4. Setting Up an Effort Prediction Programme
The following are the main stages in setting up an estimation by analogy programme:
* identify the data to collect
* agree data definitions and collection mechanisms
* populate the database
* tune the estimation method
* estimate new project
The first stage, that of identifying what data to collect will be very dependent upon the nature of the projects for which estimates are required. Because of these variations, ANGEL was designed to be very flexible in the data that is used to characterise analogies and the user is able to define a template describing the data that will be supplied. Factors to be taken into account include beliefs as to what features significantly impact development effort (and are measurable at the time the estimate is required) and what features can easily be collected. There is little sense in identifying huge numbers of variables that can not be easily or reliably collected in practice. ANGEL is able to cope with both continuous and categorical data although categorical data has to be held as binary values. For instance, programming language would be represented as a series of truth valued variables e.g. COBOL, 4GL, C++, etc..
The second stage is to agree definitions as to what is being collected. Even within organisations there may be no shared understanding of what is meant by effort. Any estimation programme will be fatally flawed if different projects are measuring the same attributes in different ways. It is also important to identify who is responsible for the data collection and when they should collect the data. Sometimes it can be beneficial to have the same person collecting the data across projects in order to increase the level of consistency.
Next, the database must be populated. Like all estimation methods, other than inspired guess work, analogy requires some data collection. Our experience suggests that a minimum of 8 projects are required in order to provide a stable basis for estimation. Clearly more data is preferable although in most cases data collection will be an on-going process as projects are completed and their effort data becomes available.
The penultimate stage is to tune the estimation method. The user also will need to experiment with the optimum number of analogies searched for and whether to use a subset of variables. As the case study demonstrates (in the next section) this can make quite a difference to the quality of predictions.
The last stage is to estimate for a new project. It must be possible to characterise the project in terms of the variables that have been identified at the first stage of the estimation process. From these variables ANGEL can be used to find similar projects and the user can make a subjective judgement as to the value of the analogies. Where they are believed to be trustworthy the prediction can be relied on to greater extent than where they are thought to be doubtful. The other indicator of likely prediction quality is the average MMRE figure obtained through jack knifing the dataset. Again a low figure will indicate more confidence than a high figure.
5. An Industrial Case Study
Data derived from 33 projects at a large telecommunications company was analysed using the ANGEL tool. The analogy technique is self calibrating, however, a decision still needed to be made concerning the number of analogies to search for (the ANGEL tool will currently search for up to three). A second consideration was the fact that as discussed earlier, some attributes will inevitably create `noise' and can be weeded out by an exhaustive search. At present the ANGEL tool employs a brute force search trying out every attribute combination possible. Where there are a large number of variables this can take a long time.
Attribute Description
C1 effort in work hours to implement a function
C2 no. of parameters in operator commands
C3 no. of parameters used by the subscriber input procedures
C4 outputs which triggers messages
C5 no. of parameters in output/enquiry subscriber procs
C6 no. of parameters on print-outs
C7 no. of messages passed between blocks
C8 count of timers used by function
C9 months of experience of the block designer
C10 months of experience of the function designer
C11 subs - C3+C5
C12 experience - C9+C10
C13 no. of LOC patched in the base version of the code that are
incorporated into the enhancement.
C14 subsystem indicator - Binary 1 or 0.
Table 3: Attributes used to characterise projects
No. of Analogies Attributes MMRE One Analogy All attributes 49% Two Analogies All attributes 48% Three Analogies All attributes 55% Two Analogies (1st weighted double) All attributes 48%
Table 4: Number of Analogies Used and Estimation Accuracy
Tables 4 shows the MMRE figures obtained when differing numbers of analogies are searched for using all the available attributes. These values are comparable to the best regression based model which gives an MMRE of 51%. At this point it is interesting to note that the values, although quite similar, seem to indicate that, at least for this dataset, fewer analogies may work better.
No. of Analogies Attributes MMRE
One Analogy C4, C5, C8, C14 37%
Two Analogies C2, C3, C5, C8, C9, 39%
C11, C12, C14
Three Analogies C3-C5, C7-C9, C12, 37%
C14
Two Analogies (1st weighted double) C4-C6, C8, C11, C14 39%
Table 5: Optimum attribute subsets
Table 5 demonstrates the improvement that is achievable when the best attribute combination is found. On average the MMRE value improves by 12%. Again the four values are very similar, this is made even more remarkable by the fact that each value is derived by using a different set of attributes, although it is interesting to note that some variables (e.g. C8 and C14) are common to all four methods. This similarity leaves us with a problem i.e. which method to use when making an estimate. This ultimately will be the decision of the user based upon the nature of the data.
6. Conclusions
We believe that the analogy approach offers a number of advantages. First the approach is simple and easy to understand (unlike say neural nets). Estimation by analogy has a lot in common with expert judgement. This leads to the next advantage of the analogy approach over other techniques in that it can be regarded as a "glass box", i.e. it is possible to directly relate output to the inputs. This is desirable as it allows the user to judge the quality of the estimate by viewing its source. Next, the method is more robust in that it can cope with outliers and datasets for which no statistical relationships can be found. Last, but not least, it has proved to be more accurate than the traditional algorithmic methods for each dataset that we have studied.
Acknowledgements:
The authors are grateful to the Finish TIEKE organisation for granting leave to use the Finnish dataset and to the anonymous organisation for supplying the data for the case study. This work has been supported by BT and the Enterprise in Higher Education initiative
References:
[1] Albrecht, A.J. and J.R. Gaffney, 'Software function, source lines of code, and development effort prediction: a software science validation', IEEE Trans. on Softw. Eng., 9(6), pp639-648, 1983.
[2] Atkinson, K. and M.J. Shepperd, 'The use of function points to find cost analogies', in Proc. European Software Cost Modelling Meeting. Ivrea, Italy: 1994.
[3] Desharnais, J.M., Analyse statistique de la productivitie des projects de development en informatique apartir de la technique des points de fonction. 1988, Unpublished Master's Thesis Universite du Quebec.
[4] Kemerer, C.F., 'An empirical validation of software cost estimation models', CACM, 30(5), pp416-429, 1987.
[5] Shepperd, M.J., C. Schofield, and B.A. Kitchenham, 'Effort estimation using analogy', in Proc. 18th Intl. Conf. on Softw. Eng. Berlin: IEEE Computer Press, 1996.