duncan@ctt.bellcore.com (Scott Duncan) (04/05/91)
In article <1991Apr4.224939.20793@cbnewsm.att.com> lfd@cbnewsm.att.com (Lee Derbenwick) writes: > >A metric isn't necessarily bogus, though I suspect that any _single_ >metric of the software process, in isolation, omits vastly more than it >measures. And the opposite can be true: one can measure so many things that people cannot effectively analyze the output and select some subset to focus on, resulting in the same thing. It doesn't have to, but if measures are imposed on people rather than being self-selected, choice of a subset is likely to be as much of a problem. >On the other hand, sometimes _truly_ bogus metrics get introduced... There does seem to be a trend in the software industry to measure for the sake of measuring because that sounds scientific somehow, e.g., other fields of engineering measure stuff, so software better do it so it looks like more engineering is going on. > rewarding the _opposite_ of desired >behavior, I believe I have seen very little obvious assessment of what behavior many metrics are intended to encourage. I think this is one of the things that is not clear in software. People do not talk about metrics and behavior in the same breath. I think they expect people to change behavior based on what measurement shows, but it is often "left as an exercise to the reader" to de- termine what that should be. That is, the assumption seems to be that, faced with some measurement results, true professionals will know what to do. This suggests that the conclusion should be obvious and that no perverse or opposite behavior will occur. Measurement seems to start with a focus on the product (lines of code volumes and defects per some volume) because that is easy to do. I do not doubt there are correlations between such measures and some large trends that organizations find instructive. But such measures have little or no important value in moti- vating behavior for individuals. As you note, people will produce what the measure suggests is important or be confused about what should be done in response to the measures. In general, productivity measures seem to be more guilty than quality ones. It is fairly clear that the message sent by a defect/KLOC ratio is to reduce the number of defects. But what message is sent by a KLOC/staff-year measure of productivity? In the former case, we have agreed that defects are bad in- dependent of the measurement which just tells us how many we have in some den- sity format. In the latter case, is the production of individual lines of code valuable such that it should be a standalone measure? What behavior does this hope to encourage? It is a widely measured thing. (Function Points may be more useful because it is clear you want function added, rather than lines of code. However, I have no personal experience with using FP measures though I know others who have and find them much more to their liking. They have also told me that, as opposed to implementation phase use of LOC measures, FPs are useful at requirements and design phases to help work with rough estimates of cost/schedule/effort.) More sophisticated use of measurement seems to move into a reflection of the process (to encourage change). Measures which reflect on the product rather than the process ultimately reflect on the producer and become measures of personal/individual value rather than organizational/project value. Using product measures to suggest where in the process defects are occurring and then working to fix the process seems to be the real "engineering" approach to the use of measurement. Unfortunately, in the craft atmosphere of indivi- dual prowess of many software organizations, "process" is a dirty word that smacks of low technical merit. Well...I'd better stop here before this gets too long, but I'd be very inter- ested in heariong about how people think mesurement should be used, both technically and as a motivator of behavior. Speaking only for myself, of course, I am... Scott P. Duncan (duncan@ctt.bellcore.com OR ...!bellcore!ctt!duncan) (Bellcore, 444 Hoes Lane RRC 1H-210, Piscataway, NJ 08854) (908-699-3910 (w) 609-737-2945 (h))