A Light-Weight Fault Injection Approach to Test Automated Production System PLC Software in Industrial Practice

0967-0661/ © 2016 Elsevier Ltd. All rights reserved.

Control Engineering Practice 58 (2017) 12–23

Contents lists available at ScienceDirect

Control Engineering Practice journal homepage: www.elsevier.com/locate/conengprac

A Light-Weight Fault Injection Approach to Test Automated Production System PLC Software in Industrial Practice

crossmark

⁎

Susanne Rösch , Birgit Vogel-Heuser Technical University Munich, Boltzmannstr. 15, 85748 Garching near Munich, Germany

A R T I C L E I N F O

A BS T RAC T

Keywords: Automated production system Software-implemented fault injection Automated software testing Programmable logic controller Error handling

A light-weight software-implemented fault injection (SWIFI) testing approach is introduced, focusing on technical process faults and system faults. The reaction of automated production systems (aPSs) and their programmable logic controller (PLC) software to these faults is tested. In order to tailor the testing approach to the aPS domain in industrial practice, our test generation is based on a classification of possible deviations, i.e. a classification of possible technical process and system faults as the PLC perceives them. As a result, both specification and test execution become more efficient for practitioners. Furthermore, the test specification is tailored for execution on IEC 61131-3 programming environments. In this, the execution of test cases both against simulation or the real aPS, is enabled.

1. Introduction Trends, such as flexible production (Vyatkin, 2013), are causing a rise in the amount of control software and the complexity of automated production systems (aPSs). Consequently, quality assurance is becoming more challenging because testing costs increase proportionally as software complexity increases (Afzal, Alone, Glocksien, & Torkar, 2016). In Vogel-Heuser, Fay, Schaefer, and Tichy (2015), current challenges concerning the engineering process of aPSs are analyzed. Concerning quality assurance, or respectively, verification and validation, it is stated that “the integration of simulation and testing approaches for mechanical and electrical properties in combination with formal analysis approaches for software properties is a challenging question for future research” Vogel-Heuser et al. (2015). Hussain and Frey (2006) assert that “formal verification is also a popular practice that can complement or substitute the need for testing but will require additional effort building up the formal model as well as huge computational power while realizing it”. For this reason, VogelHeuser et al. (2015) suggest to focus on incremental and compositional verification techniques. In this way, the overall quality of aPSs is assured and less complex parts, i.e. the changes detected by analyzing the change impact (Angerer, Grimmer, Prähofer, & Grunbacher, 2015), can be verified. Approaches and methods for verifying specific parts of the aPS software have been developed in several research works (Soliman & Frey, 2011; Yang & Vyatkin, 2012). The acceptance of specification languages in practice or the focus on safety critical functions is considered in these publications.

⁎

Testing is defined as the process “in which a system or component is executed under specified conditions, the results are observed or recorded, and an evaluation is made of some aspect of the system or component” (ISO/IEC/IEEE 24765, 2010). In aPS, testing is a laborious task with both test specification and test execution still being done manually. Established tools for the development of industrial control software were analyzed in Dubey (2011). The results show that testing activities are rarely automated and need to be conducted manually. Another main challenge in production automation is that “additional system aspects have to be considered in the design of automation software […], resulting from the automation-affected technical process and physical structure of the technical system” (Vogel-Heuser et al., 2014). Therefore, it is necessary to consider that the scope of aPS software exceeds the handling of faults or exceptions as practiced in the classical software domains. APS software also has to handle faults and failures that occur in the technical process, the failure of hardware and influences of the environment. Subsequently, to avoid dangers to humans or the aPS itself, the aPS has to react to faults with a suitable strategy. This strategy must be realized in error handling routines, which need to be implemented both correctly and at the correct places within the PLC software. Up to now, software is mainly implemented using the IEC 61131-3 standard for aPS (Thramboulidis, 2012). The fault detection and error handling routines within the control software comprise the largest part of the software, as it is suggested that “approximately 90% of the overall control logic is used for exception handling” (Park, Tilbury, & Khargonekar, 2001). Testing of error

Corresponding author. E-mail addresses: [email protected] (S. Rösch), [email protected] (B. Vogel-Heuser).

http://dx.doi.org/10.1016/j.conengprac.2016.09.012 Received 24 March 2016; Received in revised form 13 September 2016; Accepted 21 September 2016

Control Engineering Practice 58 (2017) 12–23

S. Rösch, B. Vogel-Heuser

code. Jamro (2015) presents an independent framework for testing IEC 61131-3 units. Although a few automatic testing procedures have been established, most commonly, partially formal models and informal requirement specifications are used within the aPS area. Because of this lack of adequate testing models, it is essential to create model notations for testing to support system and test engineers. Therefore, modeling languages and notations are being further developed and standardized to create a basis for code generation (Bonfè, Fantuzzi, & Secchi, 2013; Schumacher & Fay, 2014). Using these and similar models for test case generation is the logical next step to further support and improve the development process. The Unified Modeling Language (UML) is one of the most widely used notations for modeling the structure and behavior of software up to now. Numerous approaches focus on deriving test cases from this language (Hametner, Kormann, VogelHeuser, Winkler, & Zoitl, 2011; Hametner, Hegny, & Zoitl, 2014; Kumar, Czybik, & Jasperneite, 2011; Racchetti, Fantuzzi, & Tacconi, 2015; Lochau et al., 2014). In contrast to the works focusing on UML, Grafcet (IEC 60848), which is a graphical specification language for describing logic sequential systems, is used as the basic model for test case generation in Provost, Roussel, and Faure (2011, 2014). Sinha, Pang, Martínez, and Vyatkin (2016) use an ontology-based requirement specification as a basis for test generation. None of the approaches described above focus on fault injection. Making the UML models executable, especially sequence diagrams as in Kormann, Tikhonov, and Vogel-Heuser (2012), is another challenge when using UML in the testing process. In Kormann and Vogel-Heuser (2011), this approach is extended by a method to test error handling routines. To support testing the reaction to faults from the technical process level, a UML state chart simulation model is used. Within the simulation model, one component is marked as defective. Subsequently, in this case, the test selection does not occur automatically, but rather must be executed by hand for each test case.

handling routines, however, is often neglected in aPS industrial practice. This may also be due to the fact that according to domain experts within production automation, only 25–30% of the overall time for software development is reserved for testing (Rösch, Tikhonov, Schütz, & Vogel-Heuser, 2014). Therefore, new and efficient strategies to test aPSs’ reaction to technical process and system faults must be developed. To prove the validity and correctness of systems, fault injection has been established to measure the dependability of technical systems. The main contribution of this paper is to provide an approach to test the reaction of aPS to technical process and system faults using software-implemented fault injection (SWIFI). SWIFI was developed in collaboration with 10 leading companies of the German machine and plant manufacturing industry companies. A main focus of our paper is the presentation a light-weight specification that is applicable in the industrial area. Therefore, execution of SWIFI as Hardware-in-theLoop (HiL) is considered using commercially available tools. In this way, industrial companies focusing on special purpose machinery are supported in the application of our approach. A second focus of our paper is the classification of aPS fault models as the PLC perceives them. The third focus of the paper is an evaluation of the feasibility of SWIFI to test aPS PLC software. The paper is structured as follows. In Section 2, the state of the art of testing, model-based testing and fault injection techniques are presented. Section 3 introduces faults as the PLC perceives them from the technical process level. In Section 4, the test case specification for SWIFI testing of the PLC software is described. Section 5 presents a case study evaluation of our approach to testing aPS as well as an expert evaluation of this testing. We summarize our work and present an outlook on future work in the area in Section 6. 2. Testing of aPS and fault injection techniques

2.2. Fault injection techniques

A comprehensive overview on testing and model-based testing of aPS as well as fault injection techniques may be found in Rösch, Ulewicz, Provost, and Vogel-Heuser (2015). The complementary techniques to testing, which are based on proving that the implementation behaves equivalently to its specification, are formal verification, model-checking and theorem-proving. These techniques prove “that the internal semantics of a model is consistent, independently from the modeled system” Roussel and Lesage (1996). This means that rules such as that there should be no deadlocks in the software are proven using a software model or representation. However, as stated in Corriveau and Shi (2013), verification is rarely established within the industry in contrast to testing in computer science; this also holds true for production automation. As stated in our paper introduction, promising approaches relating to specific parts of the software, such as safety critical functions, have been researched and developed. Research works related to this paper are presented in the following. We first discuss automated testing and model-based testing of aPS. Fault injection can be included when explicitly defined in the test case; support for specifying test cases for testing error handling of aPS has not, however, been a focus of previous research work. Because of this lack in previous works, we include work from other domains.

Fault injection approaches can be divided into three types. The first type is hardware-implemented fault injection (HWIFI) in which faults are injected by forcing pins or electromagnetic interference. In software-implemented fault injection (SWIFI), faults of the system are emulated by the software. The third type of fault injection is modelimplemented fault injection (MIFI), also called simulation-based fault injection (Svenningsson, Eriksson, Vinter, & Törngren, 2011). While HWIFI and SWIFI are commonly used on prototypes or for system testing, MIFI is used in conceptual and design phases to give early feedback to engineers (Hsueh, Tsai, & Iyer, 1997). Fault injection is defined by Arlat et al. (1990) as:

• • • •

the faults (F) that are injected, a set of activations (A), the readouts (R), i.e. the logging of the system reaction or the outputs, and the actions or measures (M) that are derived from the analysis of F, A and R.

Testing approaches may also be divided into approaches aimed at finding some classes of faults and approaches focused on testing the reaction of a system to these classes. The latter testing approach explicitly defines a fault model, i.e. the possible faults, which are described as mutants or saboteurs. A fault model defines the types of possible faults of a system in respect to several different criteria. These criteria include the phase of creation (design, implementation, etc.), the dimension (hardware faults, software faults) or the system boundary (introduced from within or outside the system) (Avizienis, Laprie, Randell, & Landwehr, 2004). The faults F, which are commonly tested by fault injection, are frequently the hardware fault type and not software fault, human made fault, or other types of faults (for a

2.1. Testing of aPS Until recently, the automation of testing aPS has been largely neglected in practice. However, programming environment suppliers have identified the need to support the specification of tests and the implementation of test scripts for testing IEC 61131-3 software applications and they have begun to supply tools for these purposes. The CODESYS Test Manager (3S-Smart Software Solutions, 2015), for example, supports creating test scripts. These scripts include not only the test code, but also actions to start programs, set up communication, create test actions and integrate IEC-Unit-Tests using IEC 61131-3 13

Control Engineering Practice 58 (2017) 12–23

Company-wide unique equipment ID with component ID

Machine-wide unique fault ID

classification of faults see Avizienis et al., 2004). The activation A is differentiated by whether the fault is injected before runtime or during operation. By injecting faults during operation, the opportunity to activate the fault by time-triggered or event-triggered conditions is created allowing realization of more complex fault scenarios. In the context of this work, technical system and process faults are defined as the faults injected into the system. The activation is triggered based on a predefined state of the aPS. The readouts consist of the appropriate alarms and error handling. If an incorrect alarm is signaled, an adequate measure would be to correct the corresponding alarm in the related PLC software. A specific example is a pneumatic cylinder that gets stuck while extending. The injected fault would be the blocking of the corresponding sensor value, which indicates that the cylinder is extended. The activation is triggered based on the previous process step before the extension. The readout should be an alarm that the pneumatic cylinder has failed to perform its job. If the readout is correct, no measures need to be taken. The faults injected for HWIFI in related work also stem from the technical process level, but only represent the electrical/electronical domain (Hsueh et al., 1997; Ziade, Ayoubi, & Velazco, 2004). The tests are executed, but do not comply with the constraints for production automation, including the IEC 61131-3 execution environment. As for MIFI and HWIFI, tools for injecting faults in integrated circuits have been evaluated and are available for use (Carreira, Madeira, & Silva, 1998; Yuste, Ruiz, Lemus, & Gil, 2003). In Powell, Arlat, Chu, Ingrand, and Killijian (2012), a SWIFI approach for embedded systems is introduced in order to validate specified safety functions. Specific fault scenarios or PLCs are not the main focus of that approach. Schlingloff and Vulinovic (2005) inject faults during co-simulation. They also propose to use model-based approaches in the automotive domain to introduce faults into the code during code-generation from MATLAB/Simulink models making it a SWIFI approach. The components to be tested as failing are selected in the model. Afterwards, code is generated that realizes the occurrence of this fault during execution. The generated code is tailored to the automotive domain generating CCode and has not been evaluated for IEC 61131-3 applications. aPS PLCs are also not evaluated. In Sung, Choi, Wong, and Debroy (2011) a SWIFI method to inject faults at runtime is presented. Kernel-based fault injection is suggested on different architectural levels of systems automated by PLCs (nuclear power plants, in particular). An analysis is made of how and where different faults may be injected in order to test the integration of application, operating system, and hardware. The integration of application and operating system is tested by manipulating the communication protocols between them. The fault models do not represent technical process faults but rather specify manipulated signals. This method seems to be a viable means of testing the integration of components such as a PLC as proposed in this paper. Testing against simulations by combining SWIFI with a HiL approach similar to that proposed by Barth and Fay (2013) that also supports the automatic generation of simulations is possible. However, support to automatically generate simulation models in the aPS special purpose machinery domain remains an open challenge. In the automotive domain, several MIFI approaches have been suggested (Puntel-Schmidt & Fay, 2015; Svenningsson et al., 2011). Svenningsson et al. (2011) make use of the fact that MATLAB/ Simulink models are commonly used for modeling automotive systems. In the paper the authors do not aim at testing solely on a simulation level: the result of the test runs against simulation are used for test case generation for the real systems. Test execution is not within the constraints of production automation (incompatibility with IEC 61131-3 standard). Puntel-Schmidt and Fay (2015) show a systematization for modeling and simulating models of aPS regarding faults and failures representing a first step towards establishing the basis for MIFI approaches within aPS.

approx. 100 PLCopen compliant

ST, SFC

PLC vendor specific, similar to OMAC Company specific approx. 20 PLCopen compliant

ST

Machine/ company-wide unique fault ID OMAC approx. 450 PLCopen compliant

FBD, ST, SFC

Fault assignment to a determined address – Siemens S7

Case study A: M – Medical and automotive component special purpose machinery/ transportation with work piece carriers Case study B: PU – Medical and pharmaceutical special purpose machinery/ packaging Case study C: M – Platform and programming environment supporter/sorting station Case study D: M – Automotive component special purpose machinery/ assembly station

LD, IL approx. 200

PLC type and supplier Case study – Area (M: Machine, P: Plant Unit) – Type of aPS

Table 1 Industry-based case studies within the aPS domain.

No. of POUs

IEC 61131-3 languages

State machine per module

Fault handling

S. Rösch, B. Vogel-Heuser

14

Control Engineering Practice 58 (2017) 12–23

S. Rösch, B. Vogel-Heuser

hinder the correct error handling of technical process and system faults can be eliminated. aPS software must deal with component failures, electro-magnetic interference and corruption of software through hardware failures that may lead to system failures. These causes must be anticipated and handled correctly in order to prevent dangerous situations and damages to the aPS. In addition to component failures we identified that dangers to safety and quality are also paramount to aPS. Any deviations from the expected process and operation need to be considered and handled by the aPS software and its error handling routines. There may be additional deviations that need to be dealt with, such as those in the field of process industry, which were not analyzed in our case studies. The four case studies were analyzed regarding fault identification and error handling. The analysis was realized by examining the PLC code and identifying the code implemented for fault identification of the technical system and process faults. Subsequently, a classification of faults as the PLC perceives them was deduced. In total, 416 fault identification implementations were extracted from the case studies. The fault identification marks the first entry point of a fault from process level being recognized by the PLC application. In the four case studies, we analyzed which mechanisms are used for identifying these faults. The overall result was that these mechanisms are similar and use the same underlying algorithms. The different mechanisms are illustrated using IEC 61131-3 Function Block Diagrams (FBDs) in the following. In case study B, fault identification was primarily implemented in the way illustrated. However, the PLC software from the case studies implemented in other languages can be translated accordingly. In Fig. 3 two real examples translated from case study B are shown. In this example fault identification is commonly implemented using the Ladder Diagram language. The first method of fault identification is quite straightforward. Several sensors throughout a machine are installed to directly detect errors that endanger production. Fault identification is achieved by detecting a signal or a missing signal from these sensors (see Fig. 1, left). 35% of all analyzed fault identification mechanisms are implemented in this way. One group of sensors is installed, for example, for ensuring safety. Each safety door has at least one sensor controlling whether the door is open (when the door is open a contact is broken and the missing signal is detected). In real applications, both case study A and B include safety doors and fault identification algorithms as shown in Fig. 1. One example of a fault identification mechanism translated into English from case study B is shown in Fig. 3. In this case, a cover for a conveyor belt is observed. Another common mechanism to detect errors is the surveillance of interlocks (12% of all analyzed fault identification mechanisms of the case studies). Interlocks are an important means to prevent and detect collisions on the technical process level. One example from case study B is the position for transferring work pieces. If such a position is occupied by a component picking up the work piece, there is an interlock constraint for every other component that does not allow access to the position. In Fig. 1, middle, this is illustrated as “SensorValue1” indicating the position of one component and “SensorValue2” indicating the position of another component. Several combinations of “not in position SensorValue” and “in position SensorValue” are possible. Typical sensors within an aPS that provide the means to control the technical process include sensors monitoring positions, pressures, and temperature. A temperature that is too high or too low endangers electrical components making it necessary to monitor temperature throughout most production processes. A violation of a valid interval is detected as

3. Characteristics and requirements for SWIFI testing of aPS using industrial case studies In order to establish the basis for the approach of SWIFI testing and to analyze the most important fault identification mechanisms requiring testing, an industrial case study was conducted. We present several case studies on faults, fault identification, and error handling of aPS and their PLC software. The four IEC 61131-3 case studies are taken from four different companies and vary regarding the application domain, and in size of the analyzed software. More specifically, the four case studies used in our work originate from the domain of special purpose machinery (three case studies), and from a platform supporter (see also case studies within Rösch, 2016). The case studies are introduced in the following and presented in Table 1. One of the case studies is also further analyzed in Vogel-Heuser, Rösch, Fischer, Simon, and Ulewicz (2016). Case study A uses a Siemens S7 PLC application for a transporting system with synchronous transportation belts and work piece carriers. The case study software in the case study consists of about 200 Program Organization Units (POUs) and is implemented using the Ladder Diagram and Instruction List IEC 61131-3 languages. There is no specific state machine implemented within the program and the alarm handling is primarily realized using determined addresses. Case study B is an example from the packaging industry where the machine consists of two plant units. While the first plant unit of the machine implements a continuous process, the second unit implements a discrete process. Case study B is put into practice using the IEC 61131-3 languages Function Block Diagram, Structured Text, and Sequential Function Chart in a PLCopen compliant programming environment, and the software used consists of about 450 POUs. This case study software conforms to the OMAC (Organization for Machine Automation and Control) state machine, which is popular in the packaging machine domain. Case study C includes a small sorting station with discrete processes implemented using a PLCopen compliant programming environment. The software consists of about 20 POUs implemented in Structured Text. Case Study C also contains a state machine that conforms to the PLC vendor's suggestion of how to implement state machines Case study D contains PLCopen compliant implementation as well, controlling an assembly station with discrete processes implemented in the IEC 61131-3 languages Sequential Function Chart and Structured Text. The software project contains about 100 POUs. The state machine used is company-specific. Except for case study B, none of the case studies have been implemented using the IEC 61131-3 object oriented extension because their programming environment does not yet support the object oriented extension although this support became available in later versions.

3.1. Faults and fault identification in aPS Testing of the reaction of aPS to faults that should be handled by the PLC software is a main focus of this work. Fault, error, and failure are defined in Avizienis et al. (2004). A failure refers to “an event that occurs when the delivered service deviates from correct service” (Avizienis et al., 2004), which was originally specified or expected. A failure is caused by a deviation from the expected system state. This “deviation is called an error” and “the adjudged or hypothesized cause of an error is called a fault”. In the context of this work, the injected faults are considered to be technical process and system faults in contrast to software bugs. By testing the aPS with its software bugs that

Fig. 1. Fault identification mechanisms for possible deviations from the process image.

15

Control Engineering Practice 58 (2017) 12–23

S. Rösch, B. Vogel-Heuser

unexpected signal. The same is true for axis errors. In case study B, a detailed diagnosis of axis errors is done by the firmware of the axis supplier. These errors are also reported in the same manner as an unexpected signal. To detect invalid inputs from the operator, parameters are checked within the application as used to check the interlock. In conclusion, typical faults and errors from the technical process level include unexpected (missing) values (also taking specific durations into account), interlocks, interval violations and group faults. Group faults and fault sequences may be viewed as the combination of several faults. Pre-diagnosed faults can also be considered as signals and are therefore handled in the same way as unexpected and unexpected missing signals.

Fig. 2. Fault identification mechanisms for possible deviations from the process image including timing aspects.

shown in Fig. 1, right (6% of all fault identification mechanisms in the case studies). The surveillance of the timing of different process steps is also common in aPS PLC software programs (13.5% of all fault identifications from the case studies were classified as such). A typical example is the basic movement of a pneumatic cylinder. As soon as the command to move has been given or a previous process step has finished, the pneumatic cylinder is expected to reach its opposite end position in a specified time (within a certain tolerance). The successful arrival at the end position is signaled by an end position sensor. If the end position is not reached, an error has occurred (e.g. pressure not high enough, sensor failure, actuator failure, etc.). An error is diagnosed by monitoring the specified variable that signals the start of the transition (Fig. 2, “StateX”) and the sensor signal (SensorValue1), e.g. the end position, in this case in combination with a timer measuring the allowed duration (represented by a TON in Fig. 2). The sensor variable may also be of another type (such as REAL). The fault detection is then a combination of the one shown in Fig. 2 with a limit checking as shown in Fig. 1. An example of this type of cylinder observation implemented in case study B is shown in Fig. 3. In some of our case studies the root cause of an alarm or a failure is complex to diagnose. Hence, some programs have error detection mechanisms to help identify the root cause. Identification is realized by combining the information from two errors, which have arisen at the same time or where one is the result of the other. A possible scenario is that first an error of a missing signal is detected, followed by a pneumatic cylinder receiving the command to pull in. If the cylinder does not reach the inner position and sets off an alarm, a group error may indicate that the entire pneumatic cylinder is not working properly. In the case studies, fault identification was primarily limited to combining two errors to one group error. The identification of subsequent faults results in a change of the error reaction leading to a shutting down of the operation of the machine. An integrated routine collects and stores all identified faults. When looking at the percentages of the different fault identifications mechanisms implemented, testing of single faults showed the highest relevance for the aPS domain. When analyzing the fault identification mechanisms within the case studies, there are some faults that allow a distinction to be made between detected faults from process level and other faults. Some faults are detected using information not directly stemming from sensors. First, some faults are detected by routines implementing the surveillance of the communication, which are often predefined routines not implemented by the application engineer, but integrated using corresponding libraries. A result of an observed fault delivered by the surveillance is reported to the application in the same manner as an

3.2. Requirements for SWIFI testing of aPS As soon as a fault has been identified by an aPS, it must be managed (and treated, from this point on, as an error because of deviations in the operation of the system). The analysis of error handling routines of the industry-based case studies revealed several characteristics that must be taken into account for SWIFI testing of aPS. In all of the case studies described above, the aPS almost never has an error reaction exactly tailored to the error that occurs. Rather, the errors are classified according to their severity, which often includes lower product quality in addition to safety and danger issues regarding the aPS. If an identified error is critical, the aPS is immediately stopped; if the error is not critical at the moment but may cause a problem in the future, a warning is given. There remains an individual part to the error handling process; namely, the alarm that is given in the form of a specified ID or global variable. Alarms are unique and give feedback on where and what kind of error occurred. In the case studies A, B, and C the alarm is simply directly written into global variables. By contrast, in case D the alarm is an error ID containing the ID of the affected component or module with its complete hierarchy. Seven case studies from Vogel-Heuser et al. (2016) show the same splitting of error handling into an individual alarm and a reaction to the error. In conclusion, these findings show that the: the evaluation of the reaction of an aPS may be split into two parts:

• •

the correct identification of a fault and its correct alarm, which is individual for each fault and the corresponding error, and the correct reaction of the aPS according to the severity of a fault and its error handling class.

(Requirement 1 – Req1). The advantage of splitting identification and reaction is that the effort for specifying the correct reaction to a fault and its subsequent error can be immensely reduced and the different error handling classes can be defined once. This allows for the specification of the error handling to be reused within each test case, where only the error handling class needs to be specified. As the identification of faults is often linked to a specific state, from which a subsequent state is expected within a specified time frame (see Fig. 2), the aPS must be brought into this state to start the fault

Fig. 3. Examples of fault identification mechanisms as implemented in case study B using LD.

16

Control Engineering Practice 58 (2017) 12–23

S. Rösch, B. Vogel-Heuser

dropping to “v2″ (valid between 39 °C and 35 °C) until time “x2″. In this case, the state change would be successful if the temperature is within the valid range and no exact value must be reached. The match for possible deviations from expected state/variable changes may be found when looking at the error detection mechanism “missing expected signal within specified duration”. In Fig. 2, only one example is shown (“StateX” and not “SensorValue1”). The different combinations of “StateX” and “SensorValue1”, and “StateX” and “SensorValue2” complete the match to the possible deviations. Furthermore, analogous to the other detection mechanisms, analogous signals must be considered for “SensorValue1”. The comparison of the different possible signals and the fault identification mechanisms leaves one gap – the definition of group errors. However, as group errors are a combination of at least two detected errors, the combination of different FIOs needs to be enabled to include all possible errors making up the following fault classification:

injection. Furthermore, different states can only be reached in their respective mode of operation (automatic mode, manual mode, etc.). To this end, we analyzed how aPSs are usually brought into these states or respectively switch from one mode of operation to another. The findings were that each case study included a specific base position. A base position is possible when the components within the aPS consist of drives, pneumatic cylinders, conveyor belts, and other components which may hold several positions. The base positions are then specified positions for all components. A situation assumed to be almost impossible within process industry plants, as they include many components lacking specific positions. In order to start another mode of operation, the aPS is brought into the base position. Therefore, to reach the base position an initialization/resetting routine is included in the case study examples and may be used for testing. (Requirement 2 – Req2). Existing aPS do have such initialization routines, which may be used for the test cases, in order to subsequently execute the fault injection. After one fault scenario has been tested and the aPS has for example stopped operation, the initialization or resetting procedure may be used to bring the aPS into position for the next test case. One problem when not testing against a simulation but against a real aPS and feigning wrong signals is that the system is brought into a possibly unknown state and which may lead to a dangerous situation. Subsequently, the test case must inherently guarantee that dangerous states never occur during test case execution (Requirement 3 – Req3). This routine ensures that the aPS PLC software can both be tested with the actual aPS and against a simulation.

1. Unexpected state change: Missing value when value is expected, or value when no value is expected, also called random (missing) value. Unexpected state change can be realized by injecting variations for a missing expected (Boolean) signal. An example of an FIO of this category is: 1. SensorValue1 ≔FALSE; 2. Interval violation: (Missing) value considering non-Boolean variables. An example of an FIO of this category is: 1. SensorValue1≔〈SpecifiedIntervalUpperBound〉 + 1; 3. State change block: State/signal change does not occur within expected time. An example of an FIO of this category is: 1. SensorValue1≔FALSE; 4. State change force: State/value change occurs earlier than expected if a minimal amount of time is given. An example of an FIO of this category is: 1. SensorValue2≔TRUE;

4. Concept for SWIFI test specification and execution on aPS 4.1. Fault injection operators for testing aPS PLC software The approach presented in this paper is aimed at testing the reaction of an aPS to faults from the technical system and process level. Therefore, a fault from the technical process level is feigned during test execution. In order to achieve this, the process image received by the PLC needs to be manipulated according to possible faults and fault identification algorithms. In this way, the FIOs do not change the program itself, but rather the process image. To be more specific, the sensor values received in each PLC cycle are manipulated. First, the FIOs are identified and defined by considering the possible deviations of the process image received by the PLC. In addition these deviations were identified by considering the results of the analyses of the case studies from the previous section. When considering possible signal sequences of sensor variables there are two possibilities: either a variable remains constant within a defined time frame or the variable changes. In the next step, the different degrees of freedom must be considered. During an expected constant signal, deviations include the absence of said signal or the occurrence of a signal when it is not expected. However, as sensor signals do not only include Boolean variables, analogous signals also need to be considered. For these variables valid ranges and subsequently the violations of valid ranges need to be taken into account. When checking the different possible deviations against the analysis of fault identification, matches may be found. The missing constant signal or the existing signal, when no signal is expected, finds a match in the error detection mechanisms unexpected (missing) signal, interlock constraint and interval violation. The interlock constraint must be considered with a defined state for the interlock (identified by the values of other variables during that time) of the aPS. For state changes, again the absence of an expected signal within the expected time frame must be considered. Furthermore, during the occurrence of a state change, it may be possible that the change takes place earlier than expected if a minimal amount of time is given, or that the state change does not occur at all. Valid ranges must also be considered for state changes. This is necessary, when at time “x1″ a temperature “v1″ is expected (valid between 40 °C and 45 °C)

Some of the FIOs are the same but are differentiated from one another by the activation A explained in the following section. Concluding this section, each possible deviation is defined as an FIO: the fault model as the PLC perceives technical system and process faults. For each FIO both Boolean and analogous values are considered. The existence of valid intervals is considered for analogous variables. Each of the defined FIOs is also reflected in the error detection actually implemented as analyzed in the case studies. Furthermore, combinations of fault operators to allow the injection of multiple faults and therefore group errors are also supported. 4.2. Test specification for SWIFI testing on PLCs The test case specification was developed according to the imposed requirements so that all information needed for test code generation is provided. This means that an initialization routine has to be included (Req2) and that no dangerous state may occur (Req3). Furthermore, the constraints from the case studies, splitting fault identification and error handling, were taken into account (Req1). The basic structure is depicted in Fig. 4 in the form of a UML sequence diagram model to illustrate the test case specification. While there is no sequence diagram in the implementation, there is a table format and the generated code. Basically, a test setup consists of three components, which is true not only for SWIFI testing but also for every test setup with a real aPS or a simulation (see Kormann et al., 2012): 1. a test system and test bed realizing the execution of test cases (TMEEngine), 2. the system under test, which is the IEC 61131-3 PLC software in this case, and more specifically the IEC 61131-3 main program (testobject:PLCControlType), 17

Control Engineering Practice 58 (2017) 12–23

S. Rösch, B. Vogel-Heuser

engineer. As soon as the activation A has been triggered, the fault is injected. This is done according to the FIO and is exemplarily shown as a call “!Sensor2” from TMEEngine to the system under test of the PLCControlType in Fig. 4, C. For the evaluation of the test verdict or respectively the reaction R, two features of the reaction of an aPS are to be analyzed. As ascertained in the case studies, the reaction may either be a general reaction, such as a full stop, or an individual reaction, such as a reconfiguration, which is rather uncommon up to now. If both reactions have been observed and are equal to the test specification, the test verdict is set to “Pass” (Fig. 4, D). The requirement that no test case may lead to a dangerous state of the aPS (Req3) is adhered to by including an “Abort”-routine within the test case. This is predefined in the actual PLC programs like the routine for driving into base position. If the aPS does not react to a fault injection within the specified time, it is stopped by the “Abort”-routine, which guarantees that no dangerous states may occur. Finally, the test case is closed using the TC_Exit()routine (Fig. 4, E).

4.3. Concept for the SWIFI test execution on PLCs The execution of test cases is related to building an appropriate test script that executes the complete test cases using the generated code. The test script is responsible for building the correct communication and executing both the test cases and the system under test with additional components such as the three actors TMEEngine, PLCControlType and APSType. The fact that a SWIFI is to be executed in a PLC environment is a decisive factor for test execution. Furthermore, the program that is tested is an IEC 61131-3 program. To make the SWIFI on a PLC possible, it is crucial that the IEC 61131-3 program receives the manipulated sensor value before its execution. This characteristic is addressed by building a test script that reshapes the program for test execution. The reshaping must be realized by replacing the targeted program in the according PLC task by a program which first calls and executes the test cases that have been generated for testing the fault scenarios. Within a normal PLC cycle, the inputs I are read, the IEC 61131-3 program X is executed, the outputs O are written and the cycle time is completed with a waiting time. For the SWIFI test execution, this process is adapted as depicted in Fig. 5. In each cycle, one test case is additionally executed during time T with the IEC 61131-3 program adding to the program execution time Tx. When executing the program, each function block representing a test case can be subsequently executed concertedly with the program. In order for the aPS to function normally, the test case must call the actual main program. The call must be directly integrated in the test case during test case generation. In this manner, the requirements for the test cases to function correctly, namely the actual program only receiving the manipulated sensor values, is fulfilled. This procedure also realizes a system test as the test case is integrated into the complete application with every task being normally executed during the test run.

Fig. 4. Structure of test cases depicted within a UML sequence diagram.

3. the aPS as an additional test component, which gives further feedback (PLCAPSType). This may either be the real aPS or a simulation. Test cases in general typically include an initialization (of the test case itself), a parameterization of the system under test (sometimes within several test steps), and routines to compare the actual behavior to the nominal behavior. In the approach presented in this paper, test cases must be initialized first (see call to self TC_Init() of TMEEngine, Fig. 4, A). Furthermore, each test case tests a specific fault scenario with the respective fault identification and error handling. In order to test the specific fault, a precondition is introduced in each test case (Fig. 4, B). The precondition observes and evaluates whether the right state for starting the fault injection has been reached. Consequently, the precondition represents the activation A of the fault injection process. In the case studies (see Section 3), it has been stated that most aPS first move into a base position, a predefined routine in every analyzed example. In this, the aPS gets into the correct mode of operation for the specific state. This characteristic is considered in the test case specification by introducing an initialization procedure for the aPS. The aPS is initialized according to the specification of a test

Fig. 5. Test case execution on a PLC (picture PLC: Systemtechnik, 2014).

18

Control Engineering Practice 58 (2017) 12–23

S. Rösch, B. Vogel-Heuser

Fig. 6. The SWIFI test specification in practice.

4.4. Specification and generation of SWIFI test cases for aPS We developed a table format to specify SWIFI test cases for PLC applications (see Fig. 6). This table enables an easy specification of the test cases and an automatic code generation of the IEC 61131-3 code. Each row in the table represents one test case. The table columns contain the specified parts for SWIFI test cases to test IEC 61131-3 PLC software applications. Some of the parts of the specification must be implemented using IEC 61131-3 structured text, which can be specified in another window and can then be selected within the table. Subsequently, these selections are included in each test case according to the specification. In particular, the following information needs to be specified:

• • • • • • • •

In the column precondition, the variable, marking the fulfillment of the precondition is specified as the activation A. The column FI Target marks the component or variable where a fault is injected. Furthermore, the column FI Code shows which exact sensor variable is overwritten. In TC Type the applied FIO is shown, marking the fault F. The Time Constraint specifies the allowed time until the reaction of the aPS. In the column Init, a reference on structured text code can be included. It is also possible to specify a semi-automatic initialization where the test case is halted and an operator may complete some actions to start the automatic operation of the aPS. The column Abort is also a reference on structured text code. In this case, a user decides how a test case should be canceled using predefined routines already available (see case study). The third predefined code is the evaluation of the aPS’ reaction (Expected Reaction). The criteria to evaluate the aPS reaction must be specified; that is, the observed reaction is equal to the specified reaction. The reaction is considered a failure if it does not meet this criterion. The evaluation of the alarm expression is implemented with the column Expected Alarm. Thus, the second constraint for the evaluation of an expected fault identification extracted from the case study is fulfilled. In combination with the expected reaction, the alarm expression marks the expected readouts R.

Fig. 7. The SWIFI test specification in practice.

5. Implementation and evaluation of SWIFI on aPS Using a prototypical editor providing the table and an IEC 61131-3 test code generator, the test cases for testing aPS can be defined and imported into PLCopen compliant programming environments. The evaluation of the approach presented in this paper contains an evaluation regarding test case execution time showing the feasibility of the approach. Furthermore, an expert evaluation with industry experts is brought forward. 5.1. Application example for SWIFI testing The feasibility of SWIFI testing aPS was evaluated using a laboratory pick and place unit of Vogel-Heuser, Legat, Folmer, & Feldmann (2014) also shown in Fig. 5. The pick and place unit consists of four main modules: (1) a stack in which work pieces are stored and separated; (2) a stamp used to stamp work pieces; (3) a conveyor used to sort work pieces into different slides according to material; and (4) a crane to transport work pieces. After initialization, using a pneumatic cylinder, one work piece from the stack is separated and pushed to the pickup position of the crane. The crane then picks up the work piece and turns it to the desired position. The crane turns 90°-counterclockwise to transport work pieces from the stack to the conveyor belt. For transportation to the stamp, the crane turns 180°. Once the work piece is placed on the conveyor it is detected by a sensor. As soon as the work piece reaches the first pair of opposing optical and inductive sensors, located in front of the first ramp, the material and color of the work piece are checked. Subsequently, the work piece is transported to one of the ramps according to material and color. It is then pushed from the conveyor belt into the ramps using pneumatic cylinders for the first and second ramp. The third ramp is filled by the conveyor belt itself. The software to control the technical process is programmed in the IEC 61131-3 languages using CODESYS. Considering the technical process, the definition using several FIOs is possible. For the evaluation with the pick and place unit, 19 test cases were defined by neglecting random values and random missing values. The test cases were disregarded in favor of testing requirements concerning the pick and place unit, focusing on timing constraints (e.g. after the slider has moved in, the stamp should be lowered within 1 s). The test cases include deviations from constant values as well as from changing values, deviations from changing values with timing

In order to develop the tests in practice, the test cases can either be specified manually in the table format (see Fig. 6) or the test cases can be automatically generated using a model-based approach. In our previous research complementing this work, we use timing diagrams and the FIOs to automatically generate the test cases (Rösch, 2016; Rösch et al., 2014). However, by using the FIOs and the test specification, other model-based approaches based on different model notations would also be possible. In order to ensure the correct implementation, a template specifying the basic structure of the test cases has been defined. For automatically generating test cases from timing diagrams, the notation has been defined using the Meta Object Facility Model to Text Transformation Language. An excerpt showing the structure for the evaluation of the test readouts is shown in Fig. 7. Within the template, the information specified, such as “ExpectedAlarm”, is individually filled in for each test case. Further information concerning test generation and the templates used in present work can be found in Rösch (2016).

19

Control Engineering Practice 58 (2017) 12–23

S. Rösch, B. Vogel-Heuser

tested by the empty “PLCPRG”-program. The Test Manager automatically generates the call of the test cases into the “PLCPRG” program. In this manner, the reshaping of the program for SWIFI and system testing is achieved.

constraints, interval violations and a group error. Hence, the test cases can still be considered representative. Therefore, the evaluation helps to validate whether the planned SWIFI with the different FIOs is possible. The test cases were implemented manually using IEC 611313 structured text code in accordance with the requirements on the test cases and predefined templates. For initializing and reinitializing, a routine “Reset” already existing within the pick and place unit software was used. For canceling (aborting), the same routine was used. This routine is semi-automatic and was implemented in all cases due to the necessity of removal and placing of work pieces by the operator. Additionally, a clearance signal has to be given by the operator for each module. The expected reaction in all cases included stopping of the machine and the setting of an individual alarm defined within the global variables. The 19 test cases were executed semi-automatically against the pick and place unit. Of the 19 test cases, only three were successful meaning that the error handling routine and the setting of the correct alarm after occurrence of an error were correct. A manual inspection of the code verified that the error handling routines of 16 failed test cases were not implemented correctly.

5.3. Evaluation results on test case execution time Several laboratory and industrial case studies, also including a timing diagram model-based approach (Rösch, 2016; Rösch et al., 2014), have been conducted on the feasibility of SWIFI testing. One of the industrial examples includes a machine from the packaging industry (from the company of case study B). This industrial case study contains a software project with about 200 POUs with ca. 14 MB memory size on a Schneider Electric PLC. The test cases in this case mainly comprise a position control and drive error handling within the IEC 61131-3 application. Using case study B no runtime failures occurred during testing also using the Test Manager. One of the laboratory examples includes the machine shown in Fig. 5 with 38 sensors and 26 actuators. The laboratory example IEC 61131-3 program contains only 7 POUs, ca. 124 kB memory size code and ca. 1 MB memory size including I/O. Besides the correct execution of test sequences, the feasibility regarding the execution on a PLC and a SoftPLC was further analyzed with the laboratory example. To evaluate the execution under the constraint of meeting real-time requirements of PLCs and aPSs, several experimental runs with a Soft-PLC and a small PLC (CPM 20 Systemtechnik, 2014) were performed. The small PLC has 128 MB DDR2-SDRAM, 32 MB NOR-Flash, 64 kB FRAM and 1 GB NAND-Flash. For both soft-PLC and PLC test runs with the program only, one standard test case and runs with test cases testing very complex scenarios were done. Complex scenarios were simulated by adding additional conditions (200, 400 and 600) to the test code, which had to be checked during execution (x-axis of the diagrams shown in Figs. 9–12). Furthermore, the influence of the Test Manager on execution times was checked for both the Soft-PLC and the PLC. The execution without the Test Manager was implemented by manipulating the program beforehand, so that the test cases were executed at runtime. For each setup, three test runs were performed and 44 test runs were performed in total. The results of the test runs including both average cycle time and maximal cycle time are shown in Figs. 9 and 10 for the Soft-PLC and in Figs. 11 and 12 for the PLC. The PLC task execution time was set to 10 ms. For the Soft-PLC, the cycle times are far from violating these 10 ms. Both the execution with and without the Test Manager stay below 0.1 ms even for complex test cases. The cycle times were higher for execution on the CPM-PLC. They are never critical, though, as even the most complex test case stayed below 2 ms in its maximal cycle time when considering the standard deviation (Fig. 12; one of the test runs had a very high maximal cycle time, the reasons are unknown). However, in case study B (see Table 1), the tasks were set to 1 ms. In this case, the test case

5.2. Implementation of SWIFI on PLCs Test execution within IEC 61131-3 programming environments and against aPS also including documentation is supported within the CODESYS Test Manager Plugin (3S-Smart Software Solutions, 2015). Therefore, our work was not aimed at developing a new test execution framework but rather at using and adapting the existing functionality. Generated test cases as described in the application example can be imported into the CODESYS environment using the PLCopen import function. As a prerequisite for execution, they were generated in conformance to Test Manager unit Function Block test cases. Having included the test code FBs, a test script can be executed. As explained in Section 2, the Test Manager supports the creation of test scripts including test actions to build communication. The unit test functionality of the Test Manager can be used to manually include test code that has been implemented using the IEC 61131-3 programming languages, which is what we have done in our approach (and adapted and used also for testing PLC software system testing). The test script, which is generally applicable to every aPS by simply changing the communication channel and the project that is to be tested, is shown in Fig. 8. As shown in Fig. 8, five prepare-actions are included in the test script. The actions “LoadProject”, “UpdateDevice” and “Set communication to local PLC” are standard prepare-actions typically included for every unit test. On the right-hand side of the figure an example of an action's configuration is shown. To realize the reshaping of the PLC program, the actions “AddPLCPRG” and “ChangeMainTask” are included in the test script. As described in the concept Section 4, this is necessary for SWIFI. The first action adds an empty program called “PLCPRG” in the program application. The second action replaces the program to be

Fig. 8. The Test Manager test script for SWIFI testing.

20

Control Engineering Practice 58 (2017) 12–23

S. Rösch, B. Vogel-Heuser

Fig. 13. Expert evaluation of the SWIFI testing approach (1: fully satisfied, 6: not satisfied at all). Fig. 9. Average cycle time with different execution setups on a Soft-PLC.

platform support company possesses detailed knowledge about PLC software testing and implementing tests. At the workshop, first a training using a modeling guideline including the example of the laboratory machine was given (see also Rösch et al., 2015). All phases of the approach from modeling to test execution and evaluation were discussed. Afterwards, the participants modeled another application example, generated and selected test cases, and executed the test cases on the application example. The workshop concluded with an expert discussion to summarize the results of the workshop. The approach and the workshop evaluation were presented to ten industry experts from ten different companies (including four of the participants from the workshop). In this way, the applicability of the approach and the improvement of the new approach for testing the aPS, compared to current approaches, were evaluated. Fig. 13 shows the results regarding the test cases and test case generation of the approach. As assessed by the experts, the generation of test cases is well managed using the approach. According to the experts, the structure of the test cases and the fault scenarios are appropriate and useful. Criticisms of the approach were that test case prioritization of the generated test cases is still a manual process in the SWIFI testing approach up to now. Automated and SWIFI testing approaches are seldom applied within the production automation industry. Hence, the overall quality assurance for SWIFI PLC software testing was rated in comparison to current quality assurance approaches. According to the experts, testing fault scenarios could be improved (see Fig. 14).

Fig. 10. Maximal cycle time with different execution setups on a Soft-PLC.

Fig. 11. Average cycle time with different execution setups on a PLC (CPM 20).

6. Summary and outlook on future work The rising amount and complexity of software in production automation necessitate the improvement of quality assurance procedures. This applies particularly to the efficiency of test specification and automation. Until now, industrial aPS developers in practice have solely supported manual testing processes. As a result, technical process and system faults and their error handling routines are often neglected. When testing aPS, special focus should be placed on fault detection and error handling as they account for the main parts of the aPS software. Faults in aPS do not only arise from the software itself or incorrect parameterization but can also arise from the technical process level at which mechanical and electrical component and process failures lead to deviations from correct functionality. In this paper, we presented a light-weight SWIFI testing approach using a table specification as a basis. This approach is implemented for technical process and system faults and the corresponding reaction of the aPS and its PLC software. In order to tailor the testing approach to the aPS domain, the test generation is based on a classification of possible deviations, i.e. a classification of possible technical process and system faults as the PLC perceives them. Furthermore, the test specification has been tailored for execution on IEC 61131-3 programming environments, enabling the execution of test cases against simulation or the real aPS. Currently, the approach is tailored to support testing the

Fig. 12. Maximal cycle time with different execution setups on a PLC (CPM 20).

complexity should be considered so as not to violate real-time constraints while taking into account the specific PLCs used. Furthermore, the cycle times are slightly higher using the Test Manager. Therefore, for very critical real-time constraints, manipulating the program beforehand and not using the Test Manager may be an option. 5.4. Expert evaluation regarding improvement of the testing process of aPS Besides the feasibility analysis, an expert evaluation with two parts was conducted. In the first part, a seven hour workshop with industry experts was held. The goal of the workshop was to provide further understanding of our approach, to receive detailed feedback on specific drawbacks, and to receive suggestions for improvement. The experts were from five different companies. Three of the experts are from a leading packaging machine producing company (head of automation, leading automation software architecture employee, automation software commissioning employee), one expert from a leading automotive and medical component machine producing company, one expert from an automotive component machine producing company, one expert from a servo-drive producing company (head) and one expert from a company providing platform support (programming environment). Except for the servo-drive company expert, all experts have an extensive understanding of PLCs and their programming. The experts from the aPS producing companies also have detailed knowledge about these systems including their technical processes. The expert from the

Fig. 14. Expert comparison of the SWIFI testing approach and current approaches (1: fully satisfied, 6: not satisfied at all).

21

Control Engineering Practice 58 (2017) 12–23

S. Rösch, B. Vogel-Heuser

applications. In 2011 9th IEEE international conference on industrial informatics (INDIN) (pp. 585–590). Hametner, R., Kormann, B., Vogel-Heuser, B., Winkler, D., & Zoitl, A. (2011). Test case generation approach for industrial automation systems. In The 5th international conference on automation, robotics and applications (pp. 57–62), December. Hametner, R., Hegny, I., & Zoitl, A. (2014). A unit-test framework for event-driven control components modeled in IEC 61499. In Emerging technology and factory automation (ETFA) (pp. 1–8). Hsueh, M.-C., Tsai, T. K., & Iyer, R. K. (1997). Fault injection techniques and tools. Computer, 30(4), 75–82. Hussain, T., & Frey, G. (2006). UML-based development process for IEC 61499 with automatic test-case generation. In IEEE conference on emerging technologies and factory automation (ETFA’06) (pp. 1277–1284). Iriondo, N., Estévez, E., & Marcos, M. (2013). Automatic generation of the supervisor code for industrial switched-mode systems. IEEE Transactions on Industrial Informatics, 9(4), 1868–1878. ISO/IEC/IEEE 24765 (2010). Systems and software engineering – Vocabulary. Jamro, M. (2015). POU-oriented unit testing of IEC 61131-3 control software. IEEE Transactions on Industrial Informatics, 11(5), 1119–1129. Kormann, B., Tikhonov, D., & Vogel-Heuser, B. (2012). Automated PLC software testing using adapted UML sequence diagrams. In14th IFAC symposium of information control problems in manufacturing, Bucharest, Romania (pp. 1615–1621). Kormann, B., & Vogel-Heuser, B. (2011). Automated test case generation approach for PLC control software exception handling using fault injection. In IEEE Industrial Electronics Society, IECON (pp. 365–372). IEEE. Kumar, B., Czybik, B., & Jasperneite, J. (2011). Model based TTCN-3 testing of industrial automation systems. First results. In IEEE conference on emerging technologies and factory automation 2011 (pp. 1–4). IEEE, September. Lettner, D., Eder, K., Grunbacher, P., & Prähofer, H. (2015). Feature modeling of two large-scale industrial software systems: Experiences and lessons learned. In18th international conference on model driven engineering languages and systems (MODELS) (pp. 386–395). ACM/IEEE. Lochau, M., Bürdek, J., Lity, S., Hagner, M., Legat, C., Golz, U., et al. (2014). Applying model-based software product line testing approaches to the automation engineering domain. At - Automatisierungstechnik – Special Issue on SPP1593 62 (11), 771– 780. MKT Systemtechnik (2014). CODESYS PLC. mkt. WebVisu. Control Web Visu. android app CPM 20. Handbook. URL 〈http://docplayer.org/1880954-Codesys-plc-mktsystemtechnik-webvisu-control-web-visu-android-app-cpm-20-handbuch〉 Park, E., Tilbury, D., & Khargonekar, P. (2001). A modeling and analysis methodology for modular logic controllers of machining systems using Petri net formalism. IEEE Transactions on Systems, Man and Cybernetics, Part C, 31(2), 168–188. Powell, D., Arlat, J., Chu, H. N., Ingrand, F., & Killijian, M. (2012). Testing the input timing robustness of real-time control software for autonomous systems. In 2012 9th European dependable computing conference (pp. 73–83). IEEE. Provost, J., Roussel, J.-M., & Faure, J.-M. (2011). Translating Grafcet specifications into mealy machines for conformance test purposes. Control Engineering Practice, 19(9), 947–957. Provost, J., Roussel, J.-M., & Faure, J.-M. (2014). Generation of single input change test sequences for conformance test of programmable logic controllers. IEEE Transactions on Industrial Informatics, 10(3), 1696–1704. Puntel-Schmidt, P., & Fay, A. (2015). Levels of detail and appropriate model types for virtual commissioning in manufacturing engineering. In 8th Vienna international conference on mathematical modelling MATHMOD 2015 (Vol. 48, pp. 922–927). Elsevier Ltd. Racchetti, L., Fantuzzi, C., & Tacconi, L. (2015). Verification and validation based on the generation of testing sequences from timing diagram specifications in industrial automation. In IEEE Industrial Electronics Society conference (IECON) (pp. 2816– 2821). Rösch, S. (2016). Model-based testing of fault scenarios in production automation. Lehrstuhl für Automatisierung und Informationssysteme, Technische Universität München, Sierke Verlag (pp. 1–30). Rösch, S., Tikhonov, D., Schütz, D., & Vogel-Heuser, B. (2014). Model-based testing of PLC software: Test of plants’ reliability by using fault injection on component level. In IFAC world conference. Rösch, S., Ulewicz, S., Provost, J., & Vogel-Heuser, B. (2015). Review of model-based testing approaches in production automation and adjacent domains – current challenges and research gaps. Journal of Software Engineering and Applications, 8(9), 499–519. Roussel, J.-M., & Lesage, J.-J. (1996). Validation and verification of Grafcet using state machine. In Proceedings of IMACS-IEEE “CESA'96” (pp. 758–764). Schlingloff, H., & Vulinovic, S. (2005). Model based dependability evaluation for automotive control functions. In Modeling and simulation for public safety. Schumacher, F., & Fay, A. (2014). Formal representation of GRAFCET to automatically generate control code. Control Engineering Practice, 33, 84–93. Sinha, R., Pang, C., Martínez, G. S., & Vyatkin, V. (2016). Automatic test case generation from requirements for industrial cyber-physical systems. At Automatisierungstechnik, 64(3), 216–230. Soliman, D., & Frey, G. (2011). Verification and validation of safety applications based on plcopen safety function blocks. Control Engineering Practice, 19(9), 929–946. Sung, A., Choi, B., Wong, W. E., & Debroy, V. (2011). Mutant generation for embedded systems using kernel-based software and hardware fault simulation. Information and Software Technology, 53(10), 1153–1164. Svenningsson, R., Eriksson, H., Vinter, J., & Törngren, M. (2011). Generic fault modelling for fault injection. In Formal methods for components and objects (pp. 287–296). Berlin, Heidelberg: Springer.

reaction to single faults. However, group errors can also be specified by combining multiple FIOs and specifying the expected reaction. Our future work will aim at supporting the reaction test to multiple faults resulting in, for example, alarm floods as described in Vogel-Heuser, Schütz, & Folmer (2015). Our evaluation with three application example machines showed that our approach is successful when applied to aPS with real-time requirements no higher than a 2 ms cycle time. Finally, results showed the acceptance of the new approach by industrial practitioners. Another result of the expert evaluation is that the testing process is greatly improved for special purpose aPS manufacturers in contrast to process technology (Vogel-Heuser et al., 2016). This especially holds true for testing of fault scenarios. However, supporting testing of process and chemical industry aPS remains an open challenge. Research work providing the means to include even more complicated supervisory functions also requiring testing has been published in recent years (Iriondo, Estévez, & Marcos, 2013). The approach presented in this paper does not cover such complex scenarios. Future work should include research on structuring and modularizing aPS in order to enable a more structured testing approach. One example for structuring IEC 61499 software may be found in Zoitl and Prähofer (2013). Variability in industrial applications is particularly considered in Lettner, Eder, Grunbacher, and Prähofer (2015). However, as first results of an ongoing survey and case studies show, there are still many challenges within the aPS domain, such as the details of modularizing fault handling and modes of operation (Vogel-Heuser et al., 2016). Acknowledgments The IGF-project 16906 N of the research association electrical engineering of the ZVEI e.V. was sponsored by the AiF as part of the program to support cooperative industrial research (IGF), with funds from the Federal Ministry of Economics and Technology (BMWi) following an Order by the German Federal Parliament. The authors would like to thank the members of the project board: Harro Höfliger Verpackungsmaschinen GmbH, teamtechnik Maschinen und Anlagen GmbH, Beckhoff Automation GmbH, Robert Bosch GmbH, and Phoenix Contact GmbH & Co. KG who supplied the industrial case study examples. We would like to thank 3S-Smart Software Solutions GmbH, and ESR Pollmeier GmbH for their participation in the workshop evaluation. Schneider Electric, Festo AG & Co. KG, and Validas AG, who supported the project during requirements elicitation and expert and survey evaluation. References 3S-Smart Software Solutions (2015). CODESYS test manager. URL 〈http://store. codesys.com/codesys-test-manager〉 Afzal, W., Alone, S., Glocksien, K., & Torkar, R. (2016). Software test process improvement approaches: A systematic literature review and an industrial case study. Journal of Systems and Software, 111, 1–33. Angerer, F., Grimmer, A., Prähofer, H., & Grunbacher, P. (2015). Configuration-aware change impact analysis (t). In30th IEEE/ACM international conference on automated software engineering (ASE). p. 385–395. Arlat, J., Aguera, M., Amat, L., Crouzet, Y., Fabre, J.-C., Laprie, J.-C. Powell, D. (1990). Fault injection for dependability validation: A methodology and some applications. IEEE Transactions on Software Engineering, 16(2), 166–182. Avizienis, A., Laprie, J.-C., Randell, B., & Landwehr, C. (2004). Basic concepts and taxonomy of dependable and secure computing. IEEE Transactions on Dependable and Secure Computing, 1(1), 11–33. Barth, M., & Fay, A. (2013). Automated generation of simulation models for control code tests. Control Engineering Practice, 21(2), 218–230. Bonfè, M., Fantuzzi, C., & Secchi, C. (2013). Design patterns for model-based automation software design and implementation. Control Engineering Practice, 21(11), 1608–1619. Carreira, J., Madeira, H., & Silva, J. G. (1998). Xception: Software fault injection and monitoring in processor functional units. In Dependable computing and fault tolerant systems (Vol. 10. pp. 245–266). Corriveau, J.-P., & Shi, W. (2013). Traceability in acceptance testing. Journal of Software Engineering and Applications, 2013(October), 36–46. Dubey, A. (2011). Evaluating software engineering methods in the context of automation

22

Control Engineering Practice 58 (2017) 12–23

S. Rösch, B. Vogel-Heuser

in plc-based industry 4.0 automated production systems as a basis for restart and self-configuration and its evaluation. Journal of Software Engineering and Applications, 9(January), 1–43. Vyatkin, V. (2013). Software engineering in industrial automation: State-of-the-art review. IEEE Transactions on Industrial Informatics, 9(3), 1234–1249. Yang, C.-H., & Vyatkin, V. (2012). Control engineering practice transformation of simulink models to IEC 61499 function blocks for verification of distributed control systems. Control Engineering Practice, 20, 1259–1269. Yuste, P., Ruiz, J. C., Lemus, L., & Gil, P. (2003). Non-intrusive software-implemented fault injection. In Dependable computing (pp. 23–38). Berlin, Heidelberg: Springer. Ziade, H., Ayoubi, R., & Velazco, R. (2004). A survey on fault injection techniques. International Arab Journal of Information Technology, 1(2), 171–186. Zoitl, A., & Prähofer, H. (2013). Guidelines and patterns for building hierarchical automation solutions in the IEC 61499 modeling language. IEEE Transactions on Industrial Informatics, 9(4), 2387–2396.

Thramboulidis, K. (2012). IEC 61499: Back to the well proven practice of IEC 61131? In IEEE 17th conference on emerging technologies factory automation (ETFA) (pp. 1– 8). Vogel-Heuser, B., Diedrich, C., Fay, A., Jeschke, S., Kowalewski, S., Wollschläger, M., & Göhner, P. (2014). Challenges for software engineering in automation. Journal of Software Engineering and Applications, 07(05), 440–451. Vogel-Heuser, B., Fay, A., Schaefer, I., & Tichy, M. (2015). Evolution of software in automated production systems: Challenges and research directions. Journal of Systems and Software, 110, 54–84. Vogel-Heuser, B., Legat, C., Folmer, J., & Feldmann, S. (2014). Researching evolution in industrial plant automation: scenarios and documentation of the pick and place unit. Technical Report TUM-AIS-TR-01-14-02. Vogel-Heuser, B., Schütz, D., & Folmer, J. (2015). Criteria-based alarm flood pattern recognition using historical data from automated production systems (aPS). Mechatronics 31, 89–100. Vogel-Heuser, B., Rösch, S., Fischer, J., Simon, T., & Ulewicz, S. (2016). Fault handling

23