SOFTWARE PROCESS AND PROJECT METRICS
Chapter 4
ITU DEPARTMENT OF COMPUTER ENGINEERING SOFTWARE ENGINEERING
Software metrics
Measurement can be applied to the software process with the intent of improving to assist in estimation, quality control, productivity assessment, and project control to help assess the quality of technical work products and to assist in tactical decision making as a project proceeds
ITU DEPARTMENT OF COMPUTER ENGINEERING SOFTWARE ENGINEERING
MEASURES , METRICS , AND INDICATORS
a measure provides a quantitative indication of the extent, amount, dimensions, capacity, or size of some attribute of a product or process. a metric as " a quantitative measure of the degree to which a system, component, or process possesses a given attribute. an indicator is a metric or combination of metrics that provide insight into the software process, a software project, or the product itself A software engineer collects measures and develops metrics so that indicators will be obtained .
ITU DEPARTMENT OF COMPUTER ENGINEERING SOFTWARE ENGINEERING
METRICS IN THE PROCESS AND PROJECT DOMAINS
Process indicators
enable a software engineering organization to gain insight into the efficacy of an existing process ( i.e. , the paradigm, software engineering tasks , work products , and milestones ) .
enable managers and practitioners to assess what works and what doesn't. Process metrics are collected across all projects and over long periods of time. Their intent is to provide indicators that lead to longterm software process improvement.
ITU DEPARTMENT OF COMPUTER ENGINEERING SOFTWARE ENGINEERING
Project indicators
enable a software project manager to assess the status of an ongoing project track potential risks uncover problem areas before they "go critical" adjust work flow or tasks evaluate the project team's ability to control quality of software engineering work products.
ITU DEPARTMENT OF COMPUTER ENGINEERING SOFTWARE ENGINEERING
Process Metrics and Software Process Improvement The only rational way to improve any process is to measure specific attributes of the process develop a set of meaningful metrics based on these attributes use the metrics to provide indicators that will lead to a strategy for improvement
ITU DEPARTMENT OF COMPUTER ENGINEERING SOFTWARE ENGINEERING
Product
Customer Characteristics
Business Conditions
Process
People
Development Environment
Technology oduct
Determinants for software quality and organizational effectiveness.
ITU DEPARTMENT OF COMPUTER ENGINEERING SOFTWARE ENGINEERING
Determinants for software quality and organizational effectiveness
process sits at the center of a triangle connecting three factors that have a profound influence on software quality and organizational performance the skill and motivation of people has most influential factor in quality and performance the complexity of the product have impact on quality and team performance the technology (the software engineering methods) the process triangle exists within a circle of environmental conditions that include the development environment, business conditions, customer characteristics (e.g., ease of communication)
ITU DEPARTMENT OF COMPUTER ENGINEERING SOFTWARE ENGINEERING
Outcomes
We measure the efficacy of a software process indirectly based on the outcomes that can be derived from the process. Outcomes :
measures of errors uncovered before release of the software efects delivered to and reported by end users work products delivered human effort expended calendar time expended schedule conformance
ITU DEPARTMENT OF COMPUTER ENGINEERING SOFTWARE ENGINEERING
We derive process metrics by measuring the characteristics of specific software engineering tasks.
measure the effort and time spent performing the umbrella activities measure the generic software engineering activities
ITU DEPARTMENT OF COMPUTER ENGINEERING SOFTWARE ENGINEERING
Private metric
There are "private and public" uses for different types of process data : Data private to the individual
serve as an indicator for the individual only
Examples of metrics private to the individual defect rates (by individual) defect rates (by module) errors found during development
ITU DEPARTMENT OF COMPUTER ENGINEERING SOFTWARE ENGINEERING
Personal Software Process
A structured set of process descriptions , measurements , and methods that can help engineers to improve their personal performance Some process metrics are private to the software project team but public to all team members
Defects reported for major software functions Errors found during formal technical reviews Lines of code or function points per module and function
ITU DEPARTMENT OF COMPUTER ENGINEERING SOFTWARE ENGINEERING
Public metrics
Public metrics assimilate information that originally was private to individuals and teams. Project-level defect
rates , effort, calendar times, related data
are collected and evaluated in an attempt to uncover indicators that can improve organizational process performance.
ITU DEPARTMENT OF COMPUTER ENGINEERING SOFTWARE ENGINEERING
Software metrics etiquette Software process metrics can be misused.
Software metrics etiquette:
Use common sense and organizational sensitivity when interpreting metrics data. Provide regular feedback to the individuals and teams who have worked to collect measures and metries. Don't use metrics to appraise individuals. Work with practitioners and teams to set clear goals and metrics that will be used to achieve them Never use metrics to threaten individuals or teams. Metrics data that indicate a problem area should not be considered "negative." These data are merely an indicator for process improvement. Don't obsess on a single metric to the exclusion of other important metrics.
ITU DEPARTMENT OF COMPUTER ENGINEERING SOFTWARE ENGINEERING
A simple defect distribution
Hardware interface 8% Error checking 11% Data handling 11% User interface 12% Standards 7% Specifications 25%
Causes of defects and their origin for four software projects
Software interface 6%
Logic 20%
Origin of errors/defects specification/requirem ents desig n code
ITU DEPARTMENT OF COMPUTER ENGINEERING SOFTWARE ENGINEERING
A fishbone diagram showing the causes of one class of defects
Missing
Ambiguous
Specification defects wrong customer queried customer gave wrong info inadequate inquiries used outdated info Incorrect Changes
ITU DEPARTMENT OF COMPUTER ENGINEERING SOFTWARE ENGINEERING
Project Metrics
Used for strategic purposes
by a project manager and a software team to adapt project work flow and technical activities .
Project metrics on estimation
effort and time duration estimates
Production rates
pages of documentation, review hours, function points, delivered source lines
ITU DEPARTMENT OF COMPUTER ENGINEERING SOFTWARE ENGINEERING
The intent of project metrics is twofold :
to minimize the development schedule, to assess product quality on an ongoing basis and when necessary, modify the technical approach to improve quality.
Every project should measure:
inputs - measures of the resources (e.g., people, environment) required to do the work, outputs - measures of the deliverables or work products created during the software engineering process, results - measures that indicate the effectiveness of the deliverables
ITU DEPARTMENT OF COMPUTER ENGINEERING SOFTWARE ENGINEERING
SOFTWARE MEASUREMENT
Direct measures (e.g., the length of a bolt) Indirect measures (e.g., the "quality" ) Direct measures
lines of code (LOC) execution speed memory size defects reported over some set period of time
Indirect measures
functionality, quality, complexity, efficiency, reliability, maintainability and many other "abilities"
ITU DEPARTMENT OF COMPUTER ENGINEERING SOFTWARE ENGINEERING
Size-Oriented Metrics
Derived by normalizingquality and or productivity measures by considering the "size" of the software
ITU DEPARTMENT OF COMPUTER ENGINEERING SOFTWARE ENGINEERING
10
Size Oriented Metrics
a set of simple size -oriented metrics can be developed for each project:
errors per KLOC (thousand lines of code) defects per KLOC $ per LOC pages of documentation per YLOC
other interesting metrics:
errors/person-month LOC per person-month $/page of documentation
ITU DEPARTMENT OF COMPUTER ENGINEERING SOFTWARE ENGINEERING
Lines of Code (LOC)
proponents claim
LOC can be easily counted, many existing software estimation models use LOC or KLOC as a key input, a large body of literature and data predicated on LOC already exists.
opponents claim
LOC measures are programming language dependent, they penalize well-designed but shorter programs, they cannot easily accommodate nonprocedural languages, their use in estimation requires a level of detail that may be difficult to achieve (i.e., the planner must estimate the LOC to be produced long before analysis and design have been completed).
ITU DEPARTMENT OF COMPUTER ENGINEERING SOFTWARE ENGINEERING
11
Function-Oriented Metrics
Use a measure of the functionality delivered by the application as a normalization value. Function-oriented metrics were first proposed by Albrecht.
Function points
measurement parameter number of user inputs number of user outputs number of user inquiries number of files number of externel interfaces count = total count
Weighting Factor simple 3 4 3 7 5 average 4 5 4 10 7 complex 6 7 6 15 10 = = = = =
Computing function point metrics.
ITU DEPARTMENT OF COMPUTER ENGINEERING SOFTWARE ENGINEERING
Number of user inputs . Each user input that provides distinct applicationoriented data to the software is counted. Inputs should be distinguished from inquiries, which are counted separately. Number of user outputs. Each user output that provides application-oriented information to the user is counted. In this context output refers to reports, screens, error messages, and so on. Individual data items within a report are not counted separately. Number of user inquiries. An inquiry is defined as an on-line input that results in the generation of some immediate software response in the form of an on-line output. Each distinct inquiry is counted. Number of files . Each logical master file (i.e., a logical grouping of data that may be one part of a large database or a separate file), is counted. Number of external interfaces . All machine readable interfaces (e.g., data files on tape or disk) that are used to transmit information to another system ITU DEPARTMENT are counted. OF
COMPUTER ENGINEERING SOFTWARE ENGINEERING
12
COMPUTING FUNCTION POINTS
Once the above data have been collected, a complexity value is associated with each count. For determining whether a particular entry,
simple, average, complex.
To compute function points (FP),
FP = count-total x [0.65 + 0.01 x Fi]
The Fi ( i = 1 to 14) are " complexity adjustment values "
ITU DEPARTMENT OF COMPUTER ENGINEERING SOFTWARE ENGINEERING
COMPUTING FUNCTION POINTS Rate each factor on a scale of 0
0 1 2 3 4 5
No influenc
Incident
Moderat
Average
Significant
Essentia
1. Does the system require reliable backup and recovery? 2. Are data communications required? 3. Are there distributed processing functions? 4. Is performance critical? 5. Will the system run in an existing, heavily utilized operational environment? 6. Does the system require on-line data entry? 7. Does the on-line data entry require the input transaction to be built over multiple screens or operations? 8. Are the master files updated on-line? 9. Are the inputs, outputs, files, or inquiries complex? 10.. Is the internal processing complex? 11. Is the code designed to be reusable? 12. Are conversion and installation included in the design? 13. Is the system designed for multiple installations in different organizations? 14. Is the application designed to facilitate change and ease o f use by the user? ITU DEPARTMENT OF COMPUTER ENGINEERING SOFTWARE ENGINEERING
13
Function Point
Once function points have been calculated, they are used in a manner analogous to LOC to normalize measures of software productivity, quality, and other attributes : errors per FP defects per FP $ per FP page of documentation per FP FP per person-month
ITU DEPARTMENT OF COMPUTER ENGINEERING SOFTWARE ENGINEERING
An Example of LOC-Based Estimation
The CAD software will accept two- and three-dimensional geometric data from an engineer. The engineer will interact and control the CAD system through a user interface that will exhibit characteristics of good humanmachine interface design. All geometric data and other supporting information will be maintained in a CAD database. Design analysis modules will be developed to produce required output which will be displayed on a variety of graphics devices. The software will be designed to control and interact with peripheral devices that include a mouse, digitizer, and laser printer. user interface and control facilities (UICF) two-dimensional geometric analysis (2DGA) three-dimensional geometric analysis (3DGA) database management (DBM) computer graphics display facilities (CGDF) peripheral control (PC) design analysis modules (DAM)
ITU DEPARTMENT OF COMPUTER ENGINEERING SOFTWARE ENGINEERING
14
Function
Estimated LOC 2,300 5,300 6,800 3,350 4,950 2,100 8,400 33,200
User interface and control facilities (UICF) Two-dimensional geometric analysis (2DGA) Three-dimensional geometric analysis (3DGA) Database management (DBM) Computer graphics display facilities (CGDF) Peripheral control (PC) Design analysis modules (DAM) Estimated lines of code
ITU DEPARTMENT OF COMPUTER ENGINEERING SOFTWARE ENGINEERING
Estimation table for LOC method.
optimistic: 4600 most likely: 6900 pessimistic: 8600 AN EXAMPLE OF FP-BASED ESTIMATION
Information Domain Value
opt.
likely pess.
est. count
weight
FP-count
Number of inputs Number of outputs Number of inquiries Number of files Number of external interfaces
20 12 16 4 2
24 15 22 4 2
30 22 28 5 3
24 16 22 4 2
4 5 4 10 7
96 80 88 40 14 318
Count-total
ITU DEPARTMENT OF COMPUTER ENGINEERING SOFTWARE ENGINEERING
15
Estimating information domain values. Factor
Backup and recovery Data communications Distributed processing Performance critical Existing operating environment On-line data entry Input transaction over multiple screens Master files updated on-line Information domain values complex Internal processing complex Code designed for reuse Conversion/installation in design Multiple installations Application designed for change Complexity adjustment factor
Value
4 2 0 4 3 4 5 3 5 5 4 3 5 5 1.17
ITU DEPARTMENT OF COMPUTER ENGINEERING SOFTWARE ENGINEERING
The estimated number of FP is derived: FPestimated = count-total [0.65 + 0.01 Fi ] FPestimated = 372
Historical data normalized using function points indicate that the organizational average productivity for systems of this type is 6.5 FP/p m. Burdened labor rate of $8000 per month, the cost per FP is approximately $1230. Based on the LOC estimate and the historical productivity data, the total estimated project cost is $457,000 and the estimated effort is 58 personmonths.
ITU DEPARTMENT OF COMPUTER ENGINEERING SOFTWARE ENGINEERING
16
Extended Function Point Metrics
The function point metric was originally designed to be applied to business information systems applications. The data dimension was emphasized to the exclusion of the functional and behavioral (control) dimensions. The function point measure was inadequate for many engineering and embedded systems Feature points
a superset of the function point applications in which algorithmic complexity is high. real-time process control embedded software applications
ITU DEPARTMENT OF COMPUTER ENGINEERING SOFTWARE ENGINEERING
To compute the feature point information domain values are again counted and weighted as described In addition, the feature point metric counts a new software characteristic, algorithms. An algorithm is defined as "a bounded computational problem that is included within a specific computer program"
ITU DEPARTMENT OF COMPUTER ENGINEERING SOFTWARE ENGINEERING
17
3D Function Point
3D function point integrates Data dimension of software with the functional and control dimensions. Data Dimension Counts of retained data (the internal program data structure, e.g., files) External data (inputs, outputs, inquiries, and external references) Functional Dimension Measured by considering 'the number of internal operations required to transform input to output data'
ITU DEPARTMENT OF COMPUTER ENGINEERING SOFTWARE ENGINEERING
Transformation A series of processing steps that are constrained by a set of semantic statements. A transformation is accomplished with an algorithm that results in a fundamental change to input data as it is processed to become output data. Acquire data from a file Simply place that data into program memory
ITU DEPARTMENT OF COMPUTER ENGINEERING SOFTWARE ENGINEERING
18
The level of complexity assigned to each transformation is a function of : The number of processing steps The number of semantic statements that control the processing steps.
Semantic Statements 1-5 Processing Steps 1-10 low low low 6-10 11+
11-20
low
average
high
21+
average
high
high
Determining the complexity of a transformation for 3D function points
ITU DEPARTMENT OF COMPUTER ENGINEERING SOFTWARE ENGINEERING
Control Dimension
Measured by counting the number of transitions between states
A state represents some externally observable mode of behavior, and a transition occurs as a result of some event that causes the software or system to change its mode of behavior. For example, a cellular phone auto-dial state resting state dialing state
ITU DEPARTMENT OF COMPUTER ENGINEERING SOFTWARE ENGINEERING
19
To compute 3D function points index = I+ O +Q+F+ E+ T+ R where I, 0, Q, F, E, T, and R represent inputs, outputs, inquiries, internal data structures, external files, transformations, and transitions, respectively.. Each complexity weighted value is computed using the following relationship complexity weighted value = Nil Wil + Nia Wia + Nih Wih where Nil , Nia , and Nih represent the number of occurrences of element i (e.g., outputs) for each level of complexity (low, average, high) Wil , Wia and Wih are the corresponding weights.
ITU DEPARTMENT OF COMPUTER ENGINEERING SOFTWARE ENGINEERING
Function Point
The function point ,like the LOC, measure is controversial Proponents claim that FP is programming language independent, making it ideal for applications using conventional and nonprocedural languages It is based on data that are more likely to be known early in the evolution of a project Opponents claim that The method requires some "sleight of hand" in that computation s i based on subjective, rather than objective, data Counts of the information domain (and other dimensions) can be difficult to collect after-the fact FP has no direct physical meaning - it's just a number.
ITU DEPARTMENT OF COMPUTER ENGINEERING SOFTWARE ENGINEERING
20
Computing 3-D Function Point
Complexity Weighting measurement parameter internal data structures external data number of user inputs number of user outputs number of user inquiries transformations transitions 3D function point index
low 7 5 3 4 3 7
+
average
high
10 + 7 4 5 4
+
15 = 10 = 6 7 6 = = =
10 + n/a +
15 = n/a =
n/a +
ITU DEPARTMENT OF COMPUTER ENGINEERING SOFTWARE ENGINEERING
RECONCILING DIFFERENT METRICS APPROACHES
The relationship between lines of code and function points depends programming language Rough estimates of the average number of lines of code required to build one function point in various programming languages Programming Language LOC/FP (average) assembly language 320 C 128 Cobol 105 Fortran 105 Pascal 90 Ada 70 object -oriented languages 30 fourth generation languages (4GLs) 20 code generators 15 spreadsheets 6 graphical languages (icons) 4 LOC and FP measures are often used to derive productivity metrics. Computing the 3D function point index
ITU DEPARTMENT OF COMPUTER ENGINEERING SOFTWARE ENGINEERING
21
Software Productivity
Five important factors that influence software productivity: People factors. The size and expertise of the development organization. Problem factors . The complexity of the problem to be solved and the number of changes in design constraints or requirements. Process factors . Analysis and design techniques that are used, languages and CASE tools available, and review techniques. Product factors. Reliability and performance of the computer-based system. Resource factors . Availability of CASE tools and hardware and software resources.
ITU DEPARTMENT OF COMPUTER ENGINEERING SOFTWARE ENGINEERING
METRICS FOR SOFTWARE QUALITY
The overriding goal of software engineering is to produce a high-quality system, application, or product. The quality of a system, application, or product is only as good as The requirements that describe the problem The design that models the solution The code that leads to an executable program The tests that exercise the software to uncover errors. To accomplish this real-time quality assessment, the engineer must use technical measures to evaluate quality in objective, rather than subjective, ways.
ITU DEPARTMENT OF COMPUTER ENGINEERING SOFTWARE ENGINEERING
22
Measuring Quality
Correctness : The degree to which the software performs its required function. Common measure: Defects per KLOC , where a defect is defined as a verified lack of conformance to requirements. Maintainability: The ease with which a program can be corrected if an error is encountered, adapted if its environment changes, or enhanced if the customer desires a change in requirements. A simple time -oriented metric Mean-time -to-change (MTTC), the time it takes to analyze the change request, design an appropriate modification, implement the change, test it, and distribute the change to all users
ITU DEPARTMENT OF COMPUTER ENGINEERING SOFTWARE ENGINEERING
Measuring Quality
Integrity Measures a system's ability to withstand attacks (both accidental and intentional) on its security. Attacks programs, data, and documents. To measure integrity, two additional attributes must be defined Threat Security Threat: The probability that an attack of a specific type will occur within a given time.
Security The probability that the attack of a specific type will be repelled. integrity = [1 - threat x (1 - security)] where threat and security are summed over each type of attack.
ITU DEPARTMENT OF COMPUTER ENGINEERING SOFTWARE ENGINEERING
23
Measuring Quality
Usability User friendliness. If a program is not "user friendly," it is often doomed to failure, even if the functions that it performs are valuable User friendliness can be measured in terms of four characteristics
the physical and/or intellectual skill required to learn the system the time required to become moderately efficient in the use of the system the net increase in productivity measured when the system is used by someone who is moderately efficient a subjective assessment of users attitudes toward the system.
ITU DEPARTMENT OF COMPUTER ENGINEERING SOFTWARE ENGINEERING
Defect Removal Efficiency
DRE = E / (E + D) where E = number of errors found before delivery of the software to the end user D = number of defects found after delivery The ideal value for DRE is 1. No defects are found in the software Realistically, D will be greater than zero, but the value of DRE can still approach 1 as E increases As E increases it is likely that the final value of D will decrease
ITU DEPARTMENT OF COMPUTER ENGINEERING SOFTWARE ENGINEERING
24
Defect Removal Efficiency
DRE can also be used within the project to assess a team's ability to find errors before they are passed to the next framework activity
DREi = Ei / (Ei + Ei + 1 ) where Ei = number of errors found during software engineering activity i. Ei + 1 = number of errors found during software engineering activity i + 1 that are traceable to errors that were not discovered in software engineering activity i.
ITU DEPARTMENT OF COMPUTER ENGINEERING SOFTWARE ENGINEERING
INTEGRATING METRICS WITHIN THE SOFTWARE PROCESS
The majority of software developers still do not measure. The problem is cultural. If we do not measure there is no real way of determining whether we are improving. Software project managers are concerned with more mundane issues :
developing meaningful project estimates producing higher-quality systems getting product out the door on time
by using measurement to establish a project baseline, each of these issues becomes more manageable.
ITU DEPARTMENT OF COMPUTER ENGINEERING SOFTWARE ENGINEERING
25
INTEGRATING METRICS WITHIN THE SOFTWARE PROCESS
As the software design is completed
Which user requirements are most likely to change? Which modules in this system are most error prone? How much testing should be planned for each module? How many errors (of specific types) can I expect when testing commences?
Answers to these questions can be determined if metrics have been collected and used as a technical guide.
ITU DEPARTMENT OF COMPUTER ENGINEERING SOFTWARE ENGINEERING
INTEGRATING METRICS WITHIN THE SOFTWARE PROCESS
Data collection requires an historical investigation of past projects to reconstruct required data. Metrics computation can span a broad range of LOC or FP metrics as well as other quality and project-oriented metrics Metrics evaluation focuses on the underlying reasons for the results obtained and produces a set of indicators that guide the project or process.
ITU DEPARTMENT OF COMPUTER ENGINEERING SOFTWARE ENGINEERING
26
FURTHER READINGS AND OTHER INFORMATION SOURCES
On the Internet, Research reports and pointers to information on software metries are available at the Software Engineering Laboratory: [Link] The U.S. Army software Metric System Web site contains a variety of useful information on process metrics: [Link] /[Link] A comprehensive listing of textbooks and papers on process metrics can be obtained at: [Link]
ITU DEPARTMENT OF COMPUTER ENGINEERING SOFTWARE ENGINEERING
FURTHER READINGS AND OTHER INFORMATION SOURCES
A Listserv mailing list that addresses function point metrics has been established. To subscribe, send mail to: cim@[Link] SUBJECT : "none" (this field must be empty) CONTENT: SUB [Link] "Your name" An up-to-date list of World Wide Web references for software process metrics can be found at: [Link]
ITU DEPARTMENT OF COMPUTER ENGINEERING SOFTWARE ENGINEERING
27
