- Email: [email protected]
Controlling Software Production, from a Customer Point of View
Copyright © [FAC SAFECOMP'90, London, UK, 1990
CONTROLLING SOFTWARE PRODUCTION, FROM A CUSTOMER POINT OF VIEW F. Ficheux*, Y. Mayadoux** and C. Pain*** *Elerlrifill dl' Fral/Cf , DirfclivlI des Dlldfs 1'1 Rffhnch es ""Sf ll 'iff IlIfoml!lliqlll' fl lIIalhhnaliql/fS Appliljll fl'.\ I , al'l'I/lle dll Gellha/ de Gal/lie, 921-11 , Ciwl/arl Cedl'X , FrailCl' ***Sell'ice El/.\flllbll'., de Prodll(liulI, 6, ljllai \1'alin, 78 -1(1/ ClllIlulI (;l'dl'X, Frallce
Abstract. EDF/DER, as a customer, must control several development teams on many software projects. This control is done on three directions : management, tools, effective production of software. We divide development in two agreements, one addressing the specifications, the second the production of software. The software workshop must be realistic and based on already known products, to avoid risks during development. The results of code analyzers give good indications of the quality level achieved, but can not be used as formal metrics.
Keywords. Software development. Maintenance Engineering. Quality control.
A CONTEXT TO PRODUCE AND TO CONTROL SOFTWARE. The Research and Development Division of EDF (EDF/DER) is notably in charge to deliver or to control software products to the other Directorates.This may be either scientific codes, ie numerical calculus, or software to control systems, ie software connected to hardware equipments.In any case, EDF/DER is faced with an increasing amount of software that are subcontracted for their production. The duration of development as well as the long time of use of these products (over ten years) implies that EDF/DER must take into account not only the development aspects but also the maintenance ones from the beginning of the software life cycle, in order to restrain costs after the delivery of products.
Inside this organization, the Software Quality Group is in charge to promote methodologies, to provide advice and to control when requested.
THE STATE OF THE INDUSTRY. An ideal answer to the quality problem is to be able to apply the Software Development Plan and Software Quality Plan that may be established and standardized, like the ISO 9000 series, or the French AFNOR and AFCIQ Plans. It provides frameworks to set up quality systems using well structured methods We are faced with two kinds of situations. EDF/DER may be its own producers, and then must control itself, often when new areas are expl o red; this is illustrated by the research side. On the opposite, EDF / DER may contract agreements with suppliers, and is well illustrated in computer based systems side. We are now
15 1
F. Ficheux, Y. Mayadoux and C. Pain
152
going to explain some of the major difficulties encountered in the two areas then we will propose a strategy. The Research Side. The goal of research engineers is to experiment new techniques, and so the first version of a product is often without the quality attributes that may be expected; then, the set up of a new qualified version implies a major modification in the mind of developers. The request of a well planned system is sometimes difficult in a research project, where the objective can not be established in terms of Q.A. attributes, but is determined by the experiment itself. The system looks like a set of prototypes, the results gained from each one having an influence on the next step of developments. Some of these softwares remain as prototypes and do not need any quality procedure; in other cases, the softwares are requested by industry and then must have all the required quality attributes. The Computer Based Systems Side. In computer controlled systems, software is part of a larger problem, and can not be completely defined, while the hardware is not entirely chosen. The managers of this type of projects are scarcely software engineers, and the software aspects must be delegated to specialized teams, increasing the cost of communication and so bringing up new constraints between people having different interests. To be able to control, the quality team need a well established reference and the access to the project informations. The EDF/DER Strategy.
The following procedures to improve quality are now adopted for computer based systems. - A strategy to separate the Requirements from the following phases of a project in two separate agreements; using such a technique, the customer gets a much better view of the specific software aspects of his problem. It is recommended to use two separate agreements, a first one to provide the Specifications from the Requirement, and a second one to develop the software according to the Specifications. A possibility from this two time production is to change of supplier between the two agreements. This strategy gives different benefits : - a better scale in terms of cost and delay; - a more detailed second agreement, according to a accurate specification; it must also include a detailed Software Quality Plan and a detailed Software Development Plan. - The introduction of generic Software Quality and Software Development Plans which provides frameworks to guide the redaction or the evaluation of the supplier plans. A guidance is provided for each section of the plan, and highlights the possible pitfalls; a description of different types of tools existing at each phase of a project allows the customer to verify the veracity of the supplier proposal. As many project managers are not software engineers, a set of course sessions informs of the specific aspects of software production, and emphasizes the necessary knowledge to understand the process.
BEFORE CONCLUDING AN AGREEMENT. A good "supplier-customer" relationship is preponderant to achieve the quality of the prod-
Controlling Software Production
uct, either the specifications or the final delivery. The customer must carefully select his supplier within different directions. Eyaluation of the Supplier Team. The experience shows that it is extremely difficult to rectify a development process with a "unreliable" supplier team; in such an occasion, the A.Q. rules become ineffective, or inapplicable. We must note that a particularity of the software area is the important turn over of the staff; it is difficult to be insured of the stability of a supplier team. On small projects, we must be care of some human behaviors, where people tend to be too much confident on one person, sometimes a "hacker" unable to follow formal A.Q. rules.
The configuration management must reflect the coc~leteness of the process. The s i ~e of this environment must be connected to the size of the project and to the knowledge of developers. All these matters addressed, we must remind that all that intentions may become ineffective if the actual developers, who usually do not know this development plan in advance, do not agree with them and are not ready to apply theses rules. Subcontracting. It is recommended to avoid any subcontracting activity. Otherwise, the same quality procedures must be applied to the subcontractors. A control of subcontractors must be laid down in the agreement.
Eyaluation of Methods and Tools.
Forecast of Controls.
The development system to be used must satisfy some confidence to the stability and duration of the chosen tools: the cost of maintenance is already affected by these choices.The possibility of immature tools, that would affect the schedule by the delay introduced in waiting for a valuable release of them, implies the greatest care in their selection. In order to allow maintenance, it is also crucial to determine the property of the tools and of their environment, the training of the maintenance team.
All the quality control procedures must be included in the agreement. We have to mention what documents will be controlled, when it will be done. It is very important not to forget the consequences in terms of delay and cost. Otherwise, it will be dificult to perform any action, either to control or to correct, as no budget and time have been dedicated to the task.
The development Plan must be correct in terms of scheduling, manpower allocation.
In EDF/DER life cycle model, this phase is essential, as its results, the specifications become the references to judge the final delivery.
The connection between the different phases and the tools associated must be verified. It does not mean that an integrated workshop is better than the use of toolkit, it only points out that a definite production of documents is planned.
CONTROLLING THE REQUIREMENTS: THE SPECIFICATION PHASE.
The following issues must be addressed : - The specifications must not be merged with some design components.
153
F. Ficheux . Y. Mayadoux and C. Pain
154
- A pretty way to insure a good communication between supplier and customer is to share a common tool, to define requirements and specifications. This practice implies that the customer get enough knowledge about the tools. No tool is currently widely agreed within the software community, so it appears difficult to select one of them and to force suppliers to use it.
GOING ON THE PROJECT. Once the realization phase of the project started, the job of the customer is mainly to verify the actual mapping with the scheduled one. We are going to focus on two points : the tool usage and the analyzers. Tool Usage. Our main advice is to survey the effective usage that is made of the tools. Roughly, there is no major problem during the Design and the Coding phase : tools like high level editors are now well established, and easy to use. Situation is very different during the test and integration phases; tools are practically not used. We may sometimes think that the obtained results of the tools are only produced to demonstrate the real usage of the tool (but on small portions of software), justifying the invoice. To answer this problem, we must insure, by agreement, that the results of tools are available to the customer. This point becomes crucial if the maintenance is going to be done by a customer team, and not the development one. In particular, a description of the regression test system is mandatory to enable the maintenance team to reuse the same materials as during development. Unfortunately, few commercial products are available; so the goal is to set up a system based on an integra-
tion of standard tools provided by the test system. We also must i ~ Jur~ that recovery training from the configuration management are done fL8m time to time, to avoid the waste of time in case of a disk failure. The Analyzers. Many static and dynamic analyzers are now available, for most of languages. Several aspects must be clearly set to establish the rules. - The responsibility to achieve the analyze: is the work done by the supplier or the customer? In the first case, the supplier is associated to the Q.A. process.This method provides good results if the tools are available to the coding team. Unfortunately, the high cost of analyzers forbids such a technique for small projects. In case of an analyze by the customer, the role of Q.A. may become ineffective, if only late versions are analyzed, at a time where no significant modification can be done. The interest is to be closer to a real situation, where the maintenance team must take into account the (hidden) structure of a software. On the opposite, the analyzed version should not be a too earlier one, where it is claimed that the next versions would erase all the detected problems, and setting the control out of the scope of a Q.A. validation process. - The expected results must be clearly defined. The goals may be considered only as highlights and warnings, or may be hold points to achieve. In the first case, the analyze tends to be only a registration time (the production process is then supposed to be concluded), but in the second case, no metrics are today universally agreed. The current usable metrics (Mac Cabe and
Controlling Software Production
Halstead complexity metrics) gives results for each module of a system; it must be care that a not too modular software has been produced, in a way to reach "a perfect unit metrics", to the prejudice of a very complex system, when integrated. Practically, we are not using derived metrics, but only the basic ones, like the cyclomatic number, the number of operators and operands. This should always be verified against the structure of control graphs, and the mandatory actions. It must be noted that most of these tools are only concerned with the process control, and not with the data usage, like the transmission of parameters between procedures. Anyway, the results from analyzers always gives indications of the quality level of the software, and the acceptance of their usage is also an indicator or an alarm of the confidence that the supplier team may have of itself.
CONCLUSION. The division of a software production in two contracts, one for the specification and one for the development, allows the customer to have a better understanding and a more accurate anticipation of its software. The software workshop must be realistic and based on already known products, to avoid risks during development. The static and dynamic analyzes give indications but the software metrology is still not enough mature to produce laws that can never been trespassed.
155
CONTROLLING SOFTWARE PRODUCTION, FROM A CUSTOMER POINT OF VIEW F. Ficheux*, Y. Mayadoux** and C. Pain*** *Elerlrifill dl' Fral/Cf , DirfclivlI des Dlldfs 1'1 Rffhnch es ""Sf ll 'iff IlIfoml!lliqlll' fl lIIalhhnaliql/fS Appliljll fl'.\ I , al'l'I/lle dll Gellha/ de Gal/lie, 921-11 , Ciwl/arl Cedl'X , FrailCl' ***Sell'ice El/.\flllbll'., de Prodll(liulI, 6, ljllai \1'alin, 78 -1(1/ ClllIlulI (;l'dl'X, Frallce
Abstract. EDF/DER, as a customer, must control several development teams on many software projects. This control is done on three directions : management, tools, effective production of software. We divide development in two agreements, one addressing the specifications, the second the production of software. The software workshop must be realistic and based on already known products, to avoid risks during development. The results of code analyzers give good indications of the quality level achieved, but can not be used as formal metrics.
Keywords. Software development. Maintenance Engineering. Quality control.
A CONTEXT TO PRODUCE AND TO CONTROL SOFTWARE. The Research and Development Division of EDF (EDF/DER) is notably in charge to deliver or to control software products to the other Directorates.This may be either scientific codes, ie numerical calculus, or software to control systems, ie software connected to hardware equipments.In any case, EDF/DER is faced with an increasing amount of software that are subcontracted for their production. The duration of development as well as the long time of use of these products (over ten years) implies that EDF/DER must take into account not only the development aspects but also the maintenance ones from the beginning of the software life cycle, in order to restrain costs after the delivery of products.
Inside this organization, the Software Quality Group is in charge to promote methodologies, to provide advice and to control when requested.
THE STATE OF THE INDUSTRY. An ideal answer to the quality problem is to be able to apply the Software Development Plan and Software Quality Plan that may be established and standardized, like the ISO 9000 series, or the French AFNOR and AFCIQ Plans. It provides frameworks to set up quality systems using well structured methods We are faced with two kinds of situations. EDF/DER may be its own producers, and then must control itself, often when new areas are expl o red; this is illustrated by the research side. On the opposite, EDF / DER may contract agreements with suppliers, and is well illustrated in computer based systems side. We are now
15 1
F. Ficheux, Y. Mayadoux and C. Pain
152
going to explain some of the major difficulties encountered in the two areas then we will propose a strategy. The Research Side. The goal of research engineers is to experiment new techniques, and so the first version of a product is often without the quality attributes that may be expected; then, the set up of a new qualified version implies a major modification in the mind of developers. The request of a well planned system is sometimes difficult in a research project, where the objective can not be established in terms of Q.A. attributes, but is determined by the experiment itself. The system looks like a set of prototypes, the results gained from each one having an influence on the next step of developments. Some of these softwares remain as prototypes and do not need any quality procedure; in other cases, the softwares are requested by industry and then must have all the required quality attributes. The Computer Based Systems Side. In computer controlled systems, software is part of a larger problem, and can not be completely defined, while the hardware is not entirely chosen. The managers of this type of projects are scarcely software engineers, and the software aspects must be delegated to specialized teams, increasing the cost of communication and so bringing up new constraints between people having different interests. To be able to control, the quality team need a well established reference and the access to the project informations. The EDF/DER Strategy.
The following procedures to improve quality are now adopted for computer based systems. - A strategy to separate the Requirements from the following phases of a project in two separate agreements; using such a technique, the customer gets a much better view of the specific software aspects of his problem. It is recommended to use two separate agreements, a first one to provide the Specifications from the Requirement, and a second one to develop the software according to the Specifications. A possibility from this two time production is to change of supplier between the two agreements. This strategy gives different benefits : - a better scale in terms of cost and delay; - a more detailed second agreement, according to a accurate specification; it must also include a detailed Software Quality Plan and a detailed Software Development Plan. - The introduction of generic Software Quality and Software Development Plans which provides frameworks to guide the redaction or the evaluation of the supplier plans. A guidance is provided for each section of the plan, and highlights the possible pitfalls; a description of different types of tools existing at each phase of a project allows the customer to verify the veracity of the supplier proposal. As many project managers are not software engineers, a set of course sessions informs of the specific aspects of software production, and emphasizes the necessary knowledge to understand the process.
BEFORE CONCLUDING AN AGREEMENT. A good "supplier-customer" relationship is preponderant to achieve the quality of the prod-
Controlling Software Production
uct, either the specifications or the final delivery. The customer must carefully select his supplier within different directions. Eyaluation of the Supplier Team. The experience shows that it is extremely difficult to rectify a development process with a "unreliable" supplier team; in such an occasion, the A.Q. rules become ineffective, or inapplicable. We must note that a particularity of the software area is the important turn over of the staff; it is difficult to be insured of the stability of a supplier team. On small projects, we must be care of some human behaviors, where people tend to be too much confident on one person, sometimes a "hacker" unable to follow formal A.Q. rules.
The configuration management must reflect the coc~leteness of the process. The s i ~e of this environment must be connected to the size of the project and to the knowledge of developers. All these matters addressed, we must remind that all that intentions may become ineffective if the actual developers, who usually do not know this development plan in advance, do not agree with them and are not ready to apply theses rules. Subcontracting. It is recommended to avoid any subcontracting activity. Otherwise, the same quality procedures must be applied to the subcontractors. A control of subcontractors must be laid down in the agreement.
Eyaluation of Methods and Tools.
Forecast of Controls.
The development system to be used must satisfy some confidence to the stability and duration of the chosen tools: the cost of maintenance is already affected by these choices.The possibility of immature tools, that would affect the schedule by the delay introduced in waiting for a valuable release of them, implies the greatest care in their selection. In order to allow maintenance, it is also crucial to determine the property of the tools and of their environment, the training of the maintenance team.
All the quality control procedures must be included in the agreement. We have to mention what documents will be controlled, when it will be done. It is very important not to forget the consequences in terms of delay and cost. Otherwise, it will be dificult to perform any action, either to control or to correct, as no budget and time have been dedicated to the task.
The development Plan must be correct in terms of scheduling, manpower allocation.
In EDF/DER life cycle model, this phase is essential, as its results, the specifications become the references to judge the final delivery.
The connection between the different phases and the tools associated must be verified. It does not mean that an integrated workshop is better than the use of toolkit, it only points out that a definite production of documents is planned.
CONTROLLING THE REQUIREMENTS: THE SPECIFICATION PHASE.
The following issues must be addressed : - The specifications must not be merged with some design components.
153
F. Ficheux . Y. Mayadoux and C. Pain
154
- A pretty way to insure a good communication between supplier and customer is to share a common tool, to define requirements and specifications. This practice implies that the customer get enough knowledge about the tools. No tool is currently widely agreed within the software community, so it appears difficult to select one of them and to force suppliers to use it.
GOING ON THE PROJECT. Once the realization phase of the project started, the job of the customer is mainly to verify the actual mapping with the scheduled one. We are going to focus on two points : the tool usage and the analyzers. Tool Usage. Our main advice is to survey the effective usage that is made of the tools. Roughly, there is no major problem during the Design and the Coding phase : tools like high level editors are now well established, and easy to use. Situation is very different during the test and integration phases; tools are practically not used. We may sometimes think that the obtained results of the tools are only produced to demonstrate the real usage of the tool (but on small portions of software), justifying the invoice. To answer this problem, we must insure, by agreement, that the results of tools are available to the customer. This point becomes crucial if the maintenance is going to be done by a customer team, and not the development one. In particular, a description of the regression test system is mandatory to enable the maintenance team to reuse the same materials as during development. Unfortunately, few commercial products are available; so the goal is to set up a system based on an integra-
tion of standard tools provided by the test system. We also must i ~ Jur~ that recovery training from the configuration management are done fL8m time to time, to avoid the waste of time in case of a disk failure. The Analyzers. Many static and dynamic analyzers are now available, for most of languages. Several aspects must be clearly set to establish the rules. - The responsibility to achieve the analyze: is the work done by the supplier or the customer? In the first case, the supplier is associated to the Q.A. process.This method provides good results if the tools are available to the coding team. Unfortunately, the high cost of analyzers forbids such a technique for small projects. In case of an analyze by the customer, the role of Q.A. may become ineffective, if only late versions are analyzed, at a time where no significant modification can be done. The interest is to be closer to a real situation, where the maintenance team must take into account the (hidden) structure of a software. On the opposite, the analyzed version should not be a too earlier one, where it is claimed that the next versions would erase all the detected problems, and setting the control out of the scope of a Q.A. validation process. - The expected results must be clearly defined. The goals may be considered only as highlights and warnings, or may be hold points to achieve. In the first case, the analyze tends to be only a registration time (the production process is then supposed to be concluded), but in the second case, no metrics are today universally agreed. The current usable metrics (Mac Cabe and
Controlling Software Production
Halstead complexity metrics) gives results for each module of a system; it must be care that a not too modular software has been produced, in a way to reach "a perfect unit metrics", to the prejudice of a very complex system, when integrated. Practically, we are not using derived metrics, but only the basic ones, like the cyclomatic number, the number of operators and operands. This should always be verified against the structure of control graphs, and the mandatory actions. It must be noted that most of these tools are only concerned with the process control, and not with the data usage, like the transmission of parameters between procedures. Anyway, the results from analyzers always gives indications of the quality level of the software, and the acceptance of their usage is also an indicator or an alarm of the confidence that the supplier team may have of itself.
CONCLUSION. The division of a software production in two contracts, one for the specification and one for the development, allows the customer to have a better understanding and a more accurate anticipation of its software. The software workshop must be realistic and based on already known products, to avoid risks during development. The static and dynamic analyzes give indications but the software metrology is still not enough mature to produce laws that can never been trespassed.
155