- Email: [email protected]
Results of a Safety Software Validation: SACEM
C:opvri)(ht
©
I FAC Control.
(:0 111 I'" t ,.,.,
,
( :(HllIllUlli(
P
RESULTS OF A SAFETY SOFTWARE V ALIDA TION: SAC EM P. Chapront* and C. GaliveF * """ Sl/ n 'I'\,
"'Sigllll llillg S\" III' II/,I , A 1.111/1111/, SI -Ol/I'II , F I'IIIII'I' /)1'lmrll/I/'111 "l Ihl' 'Ch il'! l:'11'clriml 1~' lIgilll'n, N.-ITI', 1'lIri " F l'lllIu'
Abstract : In 1988, HAT P commiss ioned a new ATP sys tem, SACEM, on one of the busiest suburban lines in the world, the Paris HER line A. Microprocessors are extensively u s ed, including to realize vital functions. Of course, the demands on s oftware safety are particularly high. The present paper explains the system and the philosophy of software validation. Then, the resul t s of different detection methods used in Sacem software validation are presente d.
Keywords
Automatic Train Protec tion Safety - Software Validat ion
ASA
SpE'"cif i cation
\
Lump tests
SADT + Grafcet
+
pIai n tE"rms
Contrac tors Lump t es ts
/
I
Ana:ysis of the E"ffects
B Forma l
~f' f> xprf'ss
RATP
':)f
i on
~:,f t"'.'3!"'e
Semi - lui.lp -;:esi:.s
errors
Detai led design
Cod e
Hoa re
Formal proo f
Asseortions
SACEM
D;:;VcLOPMENT AND VALIDATION PROCESS
91
1'. Ch a p ro nt a nd C. (;a li H'1
92
A -
SACEM Introduction
In 1967, for the first time in the word, RATP started the ATC revenue service on a subway line. After having equiped 12 lines of its system in about 10 years, RAT P had to face the need of automatizing the middle section of the line A of its express system where the questions of ATP and ATC were asked in new words : - headways of 120 s wi th 225 meters (738") long trains and dwelling times of 50 s, - dwelling locations along the platform related to the (variable) length of the trains, - preservation of existing wayside signals as a downgraded operating mode, - two types of rolling stock whose characteristics are very different. - compatibility wi th French National Railroad requirements related to the mutual connection of th e systems. No existing system met these specifications. Furthermore, new transit modes were appearing requiring a wholly automated operation (unattended subway trains . . . ) which led to th e development of a new generation of systems. This is SACEM (Sys teme d' Aide it la Conduite, a l'Exploitation et it la Maintenance - Driving, operation and maintenance assisting system) developed by a group of French equipment builders (ALSTHOM. CSEE. MATRA) conducted by RAT P jointly with SNCF : On october 1988, SACEM was commissioned on the line A of Paris express sys tem, one of the busies t suburban lines in the world.
The delivery involves 30 km of double track line, 10 stations and 200 trainsets of 3 or 4 cars. SAC EM permi ts to bring back the headway from 150 to 120 s with 225 meters long trains, dwelling 50 seconds along the platform. Targets To realize a modular system The primary target has been the development of a new generati<>n of systems able to be matched to every possible case, from the simplest one - driving assistance, e.g accurate stopping along the platform or super vision data provided to the driverto the most sophisticated one - e.gthorough fully automated operation. A l a rge functional as well as t e chnical modularity has therefore been demanded which, according to the type of application, makes the implementation possible, without extensive development costs, of a system strictly and perfectly matched to the specifications. Mo r eov e r. it affords an easier matching to every requirement evolution in the time, thanks to the addition of new functions without mod i fying the existing ones. To integrate the technical progress
most
recent
For a new system generation of the 80' s i t was demanded to profi t by the most recent technical progress. Specially, microprocessors are extensively used including to perform vital functions. To be cost competitive The ini tial target was to divide by two the investment costs in comparison to Paris subway ATC, the functions remaining identical . For that purpose, the realization of the system has been the subject of a "design to cost" analysis in order to make technical choices that lead to the smallest net cost. Furthermore, reliability, but above all maintainability, requirements were imposed as soon as the
begin ning of the develo pment in order to ensure a large availa bility and a reduc ed opera ting cost. Speci ally all the equipm ents were desig ned to under go autom ated diagn oses and check s in case of failur e, in order to drasti cally cut down the size of the mainte nance crew. 1 ~pl ementation
of SACEM on the PARIS
Line A On the PARIS Line A, the functi ons of SACEM are : - autom atic train protec tion, - cab-s ignall ing, trafic contro l assist ance, cance llatio n of waysid e signa ls by appro achin g trains in order to opera te at a shorte r headwa y thanks to cab-s ignall ing, - real time failur e diagn osis and dire ct with supe rvis ion transm ission to the mainte nanc e cente r. Gener al Princ iples The trains are still manua lly driven , the waysid e Signa ls are cance lled and the driver s follow the c ab signal displa y. Should a train overri de the ATP compu ted curve , the emerge ncy brakin g is trippe d. For econom y reason s it was decide d to imple ment as many proce sses as by softw are mean s. poss ible , Furthe rmore most of these proces ses are made on board rather than on the waysid e. It is thus easier to match these proces ses to the rollin g stock chara cteris tics and severa l rollin g stock types are thus able to be simul taneou sly operat ed on the same line. Waysid e equipm ent collec ts necess ary track circu its and data from in terloc kings and transm i ts them to the carbor ne equipm ent which adjust s train opera tion and contr ols depen dent of the instan taneou s line status and of perma nent data which are also transm i t ted to the train, let us howev er notice that when trains are always operat ed on the
same line (subw ay) it is then possib le to store all the perma nent d a ta on board. Waysid e Equipe ment The cont i nuous transm ission of coded digit al data is carrie d by the rails. This indepe ndent transm ission is compa tible with all types of track circu its with or witho ut insula ted joints implem ented by SNCF and RAT P. other carrie r is possib le Any r a diatin g cable, radio ... to 6 meters long beacon s make the system initia lizati on possib le as well as the distan ce offse tting (to clear small unacc uracie s of the wheel turn counti ng device ) and the of supe rvisi on trans miss ion t e l egram s .
~
VHF At deter mine d locat ions, recei vers make the recep tion possi ble of mainte nance relate d data transm itted by the trains . th e se equip ments are not All neces sary every where . On branc h lines it is possib le to insta ll only a part of them. Carbo rne Equipm ent Train s carry vario us modu les corres pondin g to chosen functi ons : - ATP with pick- up coils tachom e ter toothe d wheel s,
and
This functi on is the core of the system and is the only compu lsory subass embly . It carrie s out train protec tion wi th respec t to block and to speed limita tions. In case of oversp eed detect ion as well as in case of train or system the speed super vision failur e, system contr ols the emerg ency brakin g down to the train stop, cab-si gnal data proce ssing, - interf aces versus train equipm ent, - mainte nance assist ing.
Safety Concepts All circuits interfacing with wayside signalling equipments, respectively the emergency brake control loop, are realized according to conventional fail-safe principles. But most of the system is realized with integrated digital circuits and microprocessors : new concepts had to be developed as well as realization and certification methods suitable to those new technologies. Safety in case of hardware failure The chosen principle is to insert a heavy redundancy in all processed and transmitted data and then to code these data in the form of an arithmetic code. Thus at the end of each operation cycle (312 milliseconds), a fail safe circuit carries out a signature check on the ultimate result of the processes which makes possible the detection of any error whatever the cause is. Importance of software validation The formely described concept which widely involves safe data processing, reduces the influence of hardware aspects. Therefore, the quality of the software becomes primordial. The highly innovative concepts as well as their implementation made it more necessary as ever to be very cautious during the process and made it compulsory to implement multiple checks before considering the system is operational. RAT P and the contractors had therefore agreed that a double sided software validation process would be performed. The check processes were independent from each other and performed by different teams without any information interchange about the methods implemented in order to get a complete decorrelation of their works.
B - SOFTWARE VALIDATION Validation Process by RAT P
Perfonned
Due to its user role, perfectly knowing the targets of the system but less knowing the realization details, RAT P was only able to design a global check process according to a "black box" approach bearing on self-contained equipments. The main difficulty of such an approach is to correctly define the set of tests to be implemented by a correlative evaluation of the attainable covering rate. To reach the wished result, RAT P realized a complete system model by using SADT combined formalization and fini te s ta te graphs. This was the opportunity of developing ASA tool turned out by VERILOG Ltd. Thanks to this model which took fonctional targets into account, especially those related to safetyas well as realization constraints (at macroscopic level) it has been possible : a) to validate the specifications, b) to simulate system behaviour prior to its completion, c) to determine the mos t efficient test scenarii. Tests have been implemented on actual equipments fitted with their softwares, activated by a simulation system making it possible to inject the desired stimuli and to collect data related to hardware behaviour. Observed resul ts could be compared to forecasts, and deviations could be analyzed by a close survey of the effects on safety. Validation Process Perfonned by the Contractors From the be~inning of the survey, the interven~s were clearly aware that the main difficulty was to get a perfect specification available as
Re sult s of;t S;liet \ Soi't\\;tre \ ';tlid;ttioll : S,\CE\[
well at the level of the needs as at the level of the realizntion constraints. For this reason. from the beginningthe contractOrS decided to implement the maximal means to achieve this target. Those means are of two kinds : a) From the beginning, th e implementation, by the designing teams, of a need reexpression process. SADT formalization method has been introduced on this occasion to improve readableness, b) the implemen ta tion of a mul tipl e check process by independent teams. Therefore the specification were : a) Written by the designing teams by using : - SADT - GRAFCET - plain text
This kind of test (grey box approach) takes data into account which are related to program s tructure as well as external targe ts. An effort has been made in order t o travel through 100 % of s afety-involving paths. It is ma inly aimed to disclose problems ['elated to uncontrolled side e ff e cts and to real time ope r a tion (from cycle to cycle), c ) Lump-test on a target machine (whol e equipment) with the help o f a generator of stimuli. Te s t scenarii arise from a need s pec ification analysis guided by the sy s t em sarety analysis and analyses o J' the effects of software errors ( I\ EEL) .
b) read and criticized by the safety check team. That was the first stage of AEEL (Analyse de l'effet des erreurs dans le logicielAnalysis of the effects of software errors). During this inductive critical approach, research is not oriented towards error detection but towards the analysis of the effects of a possible error, c) reformalized according assertion method.
b ) Tes t on developing machine. This t est is said "semi-lump "because i t is relaced to modules that perform self-contained f unctions.
the Hoare
The specification check being so performed three times, the check of a good implementation of the development was carried out by using three parallel methods : a) Program formal proof performed from the specification under the form of assertions with the assistance of an assertion transforming tool (Floyd and Hoare method). This process was aimed to guarantee the best possible covering rate as far as the range of entering data and travelable paths were concerned (black-box approach) .
F or malized
specification
Prior to decide whether the system wa s able to be implemented, an ul timate check, bearing on specifications was decided. A quite new team (both Contractors and RAT P members) undertook to c o mpletely rewrite the system s pecification, taking into account tIle implementation choices the specification language selected was tlte B lunguage (VDM line developed by Mr ABRIAL). This process made it pos s ible then to derive the whole of the assertions formerly · used and thu s , to validate a great part of the validation process. Validation Results To appreciate the results, the val i dated software need to be quantiEed first. This software is made of two self-contained blocks - the on-board equipment software, - the wayside equipment software.
The on-board software includes approximately 14 000 lines written in "MODULA 2" language and includes roughly 170 procedures .
procedures involved in semi-lump t e s ts ..................... 153 number of test sets ....... 146
The wayside software includes 7 000 lines ("MODULA 2" as well) i.e. roughly 70 procedures.
- lump tests : nUIllber of test sets ....... 357 clustered in 23 scenario classes
Development has been carried out by successive stages corresponding to an evolution of users need specifiation and to the necessary corrections as wel l.
The amoun ts shown above do not inc lude reiteration performed to chock the non decrement.
Thus, this evolution required, and because of that, made it possible to test the non-decrement check related to those modifications. The validation results appreciated at three levels - quan t i ta ti vely , by detect problems,
can
the abil i ty
be
(.'0" HAT P's part, 880 tests were cae ried-out including 600 ones for on -boird software and 280 for wayside softwares;
Concerning the wayside software tests, let us notice that there are about twenty wayside computers along th,::! .Line A.
to
RAT P has performed test on each wayside computer.
- qualitatively, thanks to the confrontation of the kind of problems encountered,
Thus, one test may have been applied to several wayside computers.
quantitatevely again, by measuring working loads involved by each method. To make such an analysis possible the information is used which is con tained in error de tec tion shee ts filled on the occasion of various checks. These sheets classes :
are sorted
into
three
- irregularity sheets related to the disclosure of a well characterized error, - observation sheets related to a doubtfully interpreted report that needs a more detailed analysis, - remark sheets related to quality. To make a quantitative balance we only use the firt two classes of sheets sorting them according to the fact whether the code needed or did not need a correction. 1 - Validation effort - the only vital procedures passed the formal proof ............ 132
2 - Unformatted returns contractors made validation
of
Detection Formal Semi- Lump mode proof lump tests Total tests Number of irregularities and obser vations having led to code correc l.Lons
39
9
20
68
Other ieregularities or observations
106
85
98
189
Dueing the validation, successive checks of quality non-decrement made it possible to prove the expected decrement of the amount of irregularities.
97
ReslIlt s of a Safel \ So!'t"";lI e \ "; didali () lI : S"HT\I
Number of' irregulari ties leading to corrections
43
3-2 - Semi-lump test AJIlong the irregularities detected, 75 % we re related to discrepancies b e tween specifications and r e alization. ( Er r ors detected by the formal proof wld als o prove d by the t ests are not t ake n into account here) .
8
o
- "quality" defects such as discrepancies between comments and specification or code ...
1
2
3
4
Mos t of these defects are related to r e al time operation problems (from cycle to cycle) and to a few border e f f e cts) . Thi s kind of irregularity is not e as i ly detected by a formal method.
3 - Qualitative analysis At the level of the nature of stated errors the results hereunder have been noticed 3-1 - Formal proof results - percentage of errors related to a discrepancy between specification and realization (having led to correct the code) : 95 %
25 % of the irregularities are r e l a t e d to irregularities in test s e t s the mselves. That emphasizes the d:i. f f iculty to make up adequate test s e t s whe n those are not unit tests. Non e of the proved defects was r( ~ l n t e d to s a fety.
3- 3 - Results
of lump tests pe rformed by the contractors
logical faults wrong variabl e initialization
Most problems encountered during t hi s kind of test are related to the c o mb i nation hardware software. Pl'oved de fects have been stated on the occas ion of scenarii putting in p Lac e irregular circumstances (colnb i na tion of particular failures wld operation occurences).
- percentage of errors in th e specification itself detected on the occasion of its Hoar e rewriting : 5 %
I n such cases operators have to depe nd on their own judgment to i magine the occurences in question w\a l yze the sta t e d behaviour
Among those errors which made a correction necessary, roughly one quarter were related to safety.
Ro u g hly half the defects which n e eded a code correction were r e l a ted to safety.
Among the irregularities whose correction was considered unnecessary one may quote
3-4 - Results
Among those errors the main trend s were :
- minor discrepancies between specifications and realization perturbing only characteristics or data issued in case of trouble,
of lump performed by RATP
tests
The s e tests are mainly related to bo th aspects - validation of the software, - v a lidation of data, concerning e very specific feature of the l i ne A.
Equipment
Total number of irregularities having led to code corrections
on-board equipment software
wayside equipment software
30
23
Roughly, half of these defects were related to safety. Most of the defects detected by RATP were related to functionnal analysis. Both specific RAT P know-how (ATP, signalling systems ... ) and methods (ASA tool, RAT P software tools ... ) were particularly efficient to complete the software validation performed by the contractors. Conclusion The main lessons which can be drawn from this experiment are the following ones : specification essential.
quality
is
quite
Only it makes it possible - to implement and efficient proofs,
automatize
- to make up correct test sets. For this reason we consider that it is quite necessary to have methods available which possibilitate the problem partition hierarchical manner,
in
a
- its quasi-mathematic formalization. - check mul tiplici ty (and especially those of specification analysis) performed by independent teams is quite fundamental, especially as far as safety is concerned, - used methods complete mutually as far as the kind of defects they make it possible to detect are different,
- lump experiment that implements the whole of simulated equipment and requires a thorough analYSis of results remains quite necessary. Only this method makes it possible in the last resort to detect what could have been imperfectly specifed, especially as far as what the sys tem is due NOT TO DO is concerned, - as far as costs are concerned, the validation effort for a vital software represents finally between 100 and 150 %of the cost of realization including debugging and preparation. However it can be significantly reduced by using from the beginning, formal methods of specification and software development.
©
I FAC Control.
(:0 111 I'" t ,.,.,
,
( :(HllIllUlli(
P
RESULTS OF A SAFETY SOFTWARE V ALIDA TION: SAC EM P. Chapront* and C. GaliveF * """ Sl/ n 'I'\,
"'Sigllll llillg S\" III' II/,I , A 1.111/1111/, SI -Ol/I'II , F I'IIIII'I' /)1'lmrll/I/'111 "l Ihl' 'Ch il'! l:'11'clriml 1~' lIgilll'n, N.-ITI', 1'lIri " F l'lllIu'
Abstract : In 1988, HAT P commiss ioned a new ATP sys tem, SACEM, on one of the busiest suburban lines in the world, the Paris HER line A. Microprocessors are extensively u s ed, including to realize vital functions. Of course, the demands on s oftware safety are particularly high. The present paper explains the system and the philosophy of software validation. Then, the resul t s of different detection methods used in Sacem software validation are presente d.
Keywords
Automatic Train Protec tion Safety - Software Validat ion
ASA
SpE'"cif i cation
\
Lump tests
SADT + Grafcet
+
pIai n tE"rms
Contrac tors Lump t es ts
/
I
Ana:ysis of the E"ffects
B Forma l
~f' f> xprf'ss
RATP
':)f
i on
~:,f t"'.'3!"'e
Semi - lui.lp -;:esi:.s
errors
Detai led design
Cod e
Hoa re
Formal proo f
Asseortions
SACEM
D;:;VcLOPMENT AND VALIDATION PROCESS
91
1'. Ch a p ro nt a nd C. (;a li H'1
92
A -
SACEM Introduction
In 1967, for the first time in the word, RATP started the ATC revenue service on a subway line. After having equiped 12 lines of its system in about 10 years, RAT P had to face the need of automatizing the middle section of the line A of its express system where the questions of ATP and ATC were asked in new words : - headways of 120 s wi th 225 meters (738") long trains and dwelling times of 50 s, - dwelling locations along the platform related to the (variable) length of the trains, - preservation of existing wayside signals as a downgraded operating mode, - two types of rolling stock whose characteristics are very different. - compatibility wi th French National Railroad requirements related to the mutual connection of th e systems. No existing system met these specifications. Furthermore, new transit modes were appearing requiring a wholly automated operation (unattended subway trains . . . ) which led to th e development of a new generation of systems. This is SACEM (Sys teme d' Aide it la Conduite, a l'Exploitation et it la Maintenance - Driving, operation and maintenance assisting system) developed by a group of French equipment builders (ALSTHOM. CSEE. MATRA) conducted by RAT P jointly with SNCF : On october 1988, SACEM was commissioned on the line A of Paris express sys tem, one of the busies t suburban lines in the world.
The delivery involves 30 km of double track line, 10 stations and 200 trainsets of 3 or 4 cars. SAC EM permi ts to bring back the headway from 150 to 120 s with 225 meters long trains, dwelling 50 seconds along the platform. Targets To realize a modular system The primary target has been the development of a new generati<>n of systems able to be matched to every possible case, from the simplest one - driving assistance, e.g accurate stopping along the platform or super vision data provided to the driverto the most sophisticated one - e.gthorough fully automated operation. A l a rge functional as well as t e chnical modularity has therefore been demanded which, according to the type of application, makes the implementation possible, without extensive development costs, of a system strictly and perfectly matched to the specifications. Mo r eov e r. it affords an easier matching to every requirement evolution in the time, thanks to the addition of new functions without mod i fying the existing ones. To integrate the technical progress
most
recent
For a new system generation of the 80' s i t was demanded to profi t by the most recent technical progress. Specially, microprocessors are extensively used including to perform vital functions. To be cost competitive The ini tial target was to divide by two the investment costs in comparison to Paris subway ATC, the functions remaining identical . For that purpose, the realization of the system has been the subject of a "design to cost" analysis in order to make technical choices that lead to the smallest net cost. Furthermore, reliability, but above all maintainability, requirements were imposed as soon as the
begin ning of the develo pment in order to ensure a large availa bility and a reduc ed opera ting cost. Speci ally all the equipm ents were desig ned to under go autom ated diagn oses and check s in case of failur e, in order to drasti cally cut down the size of the mainte nance crew. 1 ~pl ementation
of SACEM on the PARIS
Line A On the PARIS Line A, the functi ons of SACEM are : - autom atic train protec tion, - cab-s ignall ing, trafic contro l assist ance, cance llatio n of waysid e signa ls by appro achin g trains in order to opera te at a shorte r headwa y thanks to cab-s ignall ing, - real time failur e diagn osis and dire ct with supe rvis ion transm ission to the mainte nanc e cente r. Gener al Princ iples The trains are still manua lly driven , the waysid e Signa ls are cance lled and the driver s follow the c ab signal displa y. Should a train overri de the ATP compu ted curve , the emerge ncy brakin g is trippe d. For econom y reason s it was decide d to imple ment as many proce sses as by softw are mean s. poss ible , Furthe rmore most of these proces ses are made on board rather than on the waysid e. It is thus easier to match these proces ses to the rollin g stock chara cteris tics and severa l rollin g stock types are thus able to be simul taneou sly operat ed on the same line. Waysid e equipm ent collec ts necess ary track circu its and data from in terloc kings and transm i ts them to the carbor ne equipm ent which adjust s train opera tion and contr ols depen dent of the instan taneou s line status and of perma nent data which are also transm i t ted to the train, let us howev er notice that when trains are always operat ed on the
same line (subw ay) it is then possib le to store all the perma nent d a ta on board. Waysid e Equipe ment The cont i nuous transm ission of coded digit al data is carrie d by the rails. This indepe ndent transm ission is compa tible with all types of track circu its with or witho ut insula ted joints implem ented by SNCF and RAT P. other carrie r is possib le Any r a diatin g cable, radio ... to 6 meters long beacon s make the system initia lizati on possib le as well as the distan ce offse tting (to clear small unacc uracie s of the wheel turn counti ng device ) and the of supe rvisi on trans miss ion t e l egram s .
~
VHF At deter mine d locat ions, recei vers make the recep tion possi ble of mainte nance relate d data transm itted by the trains . th e se equip ments are not All neces sary every where . On branc h lines it is possib le to insta ll only a part of them. Carbo rne Equipm ent Train s carry vario us modu les corres pondin g to chosen functi ons : - ATP with pick- up coils tachom e ter toothe d wheel s,
and
This functi on is the core of the system and is the only compu lsory subass embly . It carrie s out train protec tion wi th respec t to block and to speed limita tions. In case of oversp eed detect ion as well as in case of train or system the speed super vision failur e, system contr ols the emerg ency brakin g down to the train stop, cab-si gnal data proce ssing, - interf aces versus train equipm ent, - mainte nance assist ing.
Safety Concepts All circuits interfacing with wayside signalling equipments, respectively the emergency brake control loop, are realized according to conventional fail-safe principles. But most of the system is realized with integrated digital circuits and microprocessors : new concepts had to be developed as well as realization and certification methods suitable to those new technologies. Safety in case of hardware failure The chosen principle is to insert a heavy redundancy in all processed and transmitted data and then to code these data in the form of an arithmetic code. Thus at the end of each operation cycle (312 milliseconds), a fail safe circuit carries out a signature check on the ultimate result of the processes which makes possible the detection of any error whatever the cause is. Importance of software validation The formely described concept which widely involves safe data processing, reduces the influence of hardware aspects. Therefore, the quality of the software becomes primordial. The highly innovative concepts as well as their implementation made it more necessary as ever to be very cautious during the process and made it compulsory to implement multiple checks before considering the system is operational. RAT P and the contractors had therefore agreed that a double sided software validation process would be performed. The check processes were independent from each other and performed by different teams without any information interchange about the methods implemented in order to get a complete decorrelation of their works.
B - SOFTWARE VALIDATION Validation Process by RAT P
Perfonned
Due to its user role, perfectly knowing the targets of the system but less knowing the realization details, RAT P was only able to design a global check process according to a "black box" approach bearing on self-contained equipments. The main difficulty of such an approach is to correctly define the set of tests to be implemented by a correlative evaluation of the attainable covering rate. To reach the wished result, RAT P realized a complete system model by using SADT combined formalization and fini te s ta te graphs. This was the opportunity of developing ASA tool turned out by VERILOG Ltd. Thanks to this model which took fonctional targets into account, especially those related to safetyas well as realization constraints (at macroscopic level) it has been possible : a) to validate the specifications, b) to simulate system behaviour prior to its completion, c) to determine the mos t efficient test scenarii. Tests have been implemented on actual equipments fitted with their softwares, activated by a simulation system making it possible to inject the desired stimuli and to collect data related to hardware behaviour. Observed resul ts could be compared to forecasts, and deviations could be analyzed by a close survey of the effects on safety. Validation Process Perfonned by the Contractors From the be~inning of the survey, the interven~s were clearly aware that the main difficulty was to get a perfect specification available as
Re sult s of;t S;liet \ Soi't\\;tre \ ';tlid;ttioll : S,\CE\[
well at the level of the needs as at the level of the realizntion constraints. For this reason. from the beginningthe contractOrS decided to implement the maximal means to achieve this target. Those means are of two kinds : a) From the beginning, th e implementation, by the designing teams, of a need reexpression process. SADT formalization method has been introduced on this occasion to improve readableness, b) the implemen ta tion of a mul tipl e check process by independent teams. Therefore the specification were : a) Written by the designing teams by using : - SADT - GRAFCET - plain text
This kind of test (grey box approach) takes data into account which are related to program s tructure as well as external targe ts. An effort has been made in order t o travel through 100 % of s afety-involving paths. It is ma inly aimed to disclose problems ['elated to uncontrolled side e ff e cts and to real time ope r a tion (from cycle to cycle), c ) Lump-test on a target machine (whol e equipment) with the help o f a generator of stimuli. Te s t scenarii arise from a need s pec ification analysis guided by the sy s t em sarety analysis and analyses o J' the effects of software errors ( I\ EEL) .
b) read and criticized by the safety check team. That was the first stage of AEEL (Analyse de l'effet des erreurs dans le logicielAnalysis of the effects of software errors). During this inductive critical approach, research is not oriented towards error detection but towards the analysis of the effects of a possible error, c) reformalized according assertion method.
b ) Tes t on developing machine. This t est is said "semi-lump "because i t is relaced to modules that perform self-contained f unctions.
the Hoare
The specification check being so performed three times, the check of a good implementation of the development was carried out by using three parallel methods : a) Program formal proof performed from the specification under the form of assertions with the assistance of an assertion transforming tool (Floyd and Hoare method). This process was aimed to guarantee the best possible covering rate as far as the range of entering data and travelable paths were concerned (black-box approach) .
F or malized
specification
Prior to decide whether the system wa s able to be implemented, an ul timate check, bearing on specifications was decided. A quite new team (both Contractors and RAT P members) undertook to c o mpletely rewrite the system s pecification, taking into account tIle implementation choices the specification language selected was tlte B lunguage (VDM line developed by Mr ABRIAL). This process made it pos s ible then to derive the whole of the assertions formerly · used and thu s , to validate a great part of the validation process. Validation Results To appreciate the results, the val i dated software need to be quantiEed first. This software is made of two self-contained blocks - the on-board equipment software, - the wayside equipment software.
The on-board software includes approximately 14 000 lines written in "MODULA 2" language and includes roughly 170 procedures .
procedures involved in semi-lump t e s ts ..................... 153 number of test sets ....... 146
The wayside software includes 7 000 lines ("MODULA 2" as well) i.e. roughly 70 procedures.
- lump tests : nUIllber of test sets ....... 357 clustered in 23 scenario classes
Development has been carried out by successive stages corresponding to an evolution of users need specifiation and to the necessary corrections as wel l.
The amoun ts shown above do not inc lude reiteration performed to chock the non decrement.
Thus, this evolution required, and because of that, made it possible to test the non-decrement check related to those modifications. The validation results appreciated at three levels - quan t i ta ti vely , by detect problems,
can
the abil i ty
be
(.'0" HAT P's part, 880 tests were cae ried-out including 600 ones for on -boird software and 280 for wayside softwares;
Concerning the wayside software tests, let us notice that there are about twenty wayside computers along th,::! .Line A.
to
RAT P has performed test on each wayside computer.
- qualitatively, thanks to the confrontation of the kind of problems encountered,
Thus, one test may have been applied to several wayside computers.
quantitatevely again, by measuring working loads involved by each method. To make such an analysis possible the information is used which is con tained in error de tec tion shee ts filled on the occasion of various checks. These sheets classes :
are sorted
into
three
- irregularity sheets related to the disclosure of a well characterized error, - observation sheets related to a doubtfully interpreted report that needs a more detailed analysis, - remark sheets related to quality. To make a quantitative balance we only use the firt two classes of sheets sorting them according to the fact whether the code needed or did not need a correction. 1 - Validation effort - the only vital procedures passed the formal proof ............ 132
2 - Unformatted returns contractors made validation
of
Detection Formal Semi- Lump mode proof lump tests Total tests Number of irregularities and obser vations having led to code correc l.Lons
39
9
20
68
Other ieregularities or observations
106
85
98
189
Dueing the validation, successive checks of quality non-decrement made it possible to prove the expected decrement of the amount of irregularities.
97
ReslIlt s of a Safel \ So!'t"";lI e \ "; didali () lI : S"HT\I
Number of' irregulari ties leading to corrections
43
3-2 - Semi-lump test AJIlong the irregularities detected, 75 % we re related to discrepancies b e tween specifications and r e alization. ( Er r ors detected by the formal proof wld als o prove d by the t ests are not t ake n into account here) .
8
o
- "quality" defects such as discrepancies between comments and specification or code ...
1
2
3
4
Mos t of these defects are related to r e al time operation problems (from cycle to cycle) and to a few border e f f e cts) . Thi s kind of irregularity is not e as i ly detected by a formal method.
3 - Qualitative analysis At the level of the nature of stated errors the results hereunder have been noticed 3-1 - Formal proof results - percentage of errors related to a discrepancy between specification and realization (having led to correct the code) : 95 %
25 % of the irregularities are r e l a t e d to irregularities in test s e t s the mselves. That emphasizes the d:i. f f iculty to make up adequate test s e t s whe n those are not unit tests. Non e of the proved defects was r( ~ l n t e d to s a fety.
3- 3 - Results
of lump tests pe rformed by the contractors
logical faults wrong variabl e initialization
Most problems encountered during t hi s kind of test are related to the c o mb i nation hardware software. Pl'oved de fects have been stated on the occas ion of scenarii putting in p Lac e irregular circumstances (colnb i na tion of particular failures wld operation occurences).
- percentage of errors in th e specification itself detected on the occasion of its Hoar e rewriting : 5 %
I n such cases operators have to depe nd on their own judgment to i magine the occurences in question w\a l yze the sta t e d behaviour
Among those errors which made a correction necessary, roughly one quarter were related to safety.
Ro u g hly half the defects which n e eded a code correction were r e l a ted to safety.
Among the irregularities whose correction was considered unnecessary one may quote
3-4 - Results
Among those errors the main trend s were :
- minor discrepancies between specifications and realization perturbing only characteristics or data issued in case of trouble,
of lump performed by RATP
tests
The s e tests are mainly related to bo th aspects - validation of the software, - v a lidation of data, concerning e very specific feature of the l i ne A.
Equipment
Total number of irregularities having led to code corrections
on-board equipment software
wayside equipment software
30
23
Roughly, half of these defects were related to safety. Most of the defects detected by RATP were related to functionnal analysis. Both specific RAT P know-how (ATP, signalling systems ... ) and methods (ASA tool, RAT P software tools ... ) were particularly efficient to complete the software validation performed by the contractors. Conclusion The main lessons which can be drawn from this experiment are the following ones : specification essential.
quality
is
quite
Only it makes it possible - to implement and efficient proofs,
automatize
- to make up correct test sets. For this reason we consider that it is quite necessary to have methods available which possibilitate the problem partition hierarchical manner,
in
a
- its quasi-mathematic formalization. - check mul tiplici ty (and especially those of specification analysis) performed by independent teams is quite fundamental, especially as far as safety is concerned, - used methods complete mutually as far as the kind of defects they make it possible to detect are different,
- lump experiment that implements the whole of simulated equipment and requires a thorough analYSis of results remains quite necessary. Only this method makes it possible in the last resort to detect what could have been imperfectly specifed, especially as far as what the sys tem is due NOT TO DO is concerned, - as far as costs are concerned, the validation effort for a vital software represents finally between 100 and 150 %of the cost of realization including debugging and preparation. However it can be significantly reduced by using from the beginning, formal methods of specification and software development.