- Email: [email protected]
Test case generation for production systems with model-implemented fault injection consideration
Available online at www.sciencedirect.com Available online at www.sciencedirect.com
ScienceDirect ScienceDirect Procedia CIRP 00 (2018) 000–000 ScienceDirect Procedia CIRP 00 (2018) 000–000 ScienceDirect
Available online atonline www.sciencedirect.com Available at www.sciencedirect.com
Procedia CIRP 00 (2017) 000–000 Procedia CIRP 79 (2019) 268–273
www.elsevier.com/locate/procedia www.elsevier.com/locate/procedia
www.elsevier.com/locate/procedia
12th 2018, 12thCIRP CIRPConference Conferenceon onIntelligent IntelligentComputation ComputationininManufacturing ManufacturingEngineering, Engineering,18-20 CIRPJuly ICME '18 12th CIRP Conference on Intelligent Computation in Manufacturing Engineering, CIRP ICME '18 Gulf of Naples, Italy 28th CIRP Design Conference, May 2018, Nantes, France Test case generation for production systems with model-implemented fault
Test case generation for production systems with model-implemented fault injection consideration A new methodology to analyze the consideration functional and physical architecture of injection a, a Kübler *, Elmar oriented Schwarza, Alexander existing products Karl for an assembly product Verl family identification a, a a Karl Kübler *, Elmar Schwarz , Alexander Verl PaulofStief *,Tools Jean-Yves Dantan, Alain Etienne, AliSeidenstr. Siadat36, 70174 Stuttgart, Germany Institute for Control Engineering Machine and Manufacturing Units (ISW), University of Stuttgart, Institute for Control Engineering of Machine Tools and Manufacturing Units (ISW), University of Stuttgart, Seidenstr. 36, 70174 Stuttgart, Germany
a a
* Corresponding Tel.:Supérieure +49 711 685-82393. E-mail address: [email protected] Écoleauthor. Nationale d’Arts et Métiers, Arts et Métiers ParisTech, LCFC EA 4495, 4 Rue Augustin Fresnel, Metz 57078, France * Corresponding author. Tel.: +49 711 685-82393. E-mail address: [email protected]
* Corresponding author. Tel.: +33 3 87 37 54 30; E-mail address: [email protected]
Abstract Abstract Complex production systems, which have to handle product diversity and short product life cycles, can only be stable and efficient when Abstract successfully tested prior to theirwhich start-up. changes in automation within the life of can a production systemand should be verified Complex production systems, haveAlso, to handle product diversitysoftware and short product life cycle cycles, only be stable efficient when through repetitive applying Also, the new software version. Therefore, automation in combination with fault injection virtual successfully testedtesting prior tobefore their start-up. changes in automation software test within the life cycle of a production system shouldon be averified Inthrough today’s business environment, the trend towards moretoproduct variety and customization is unbroken. Due towith this development, the of model ofrepetitive the production system seen as the next improve theTherefore, verification validation a production system. This work on presents an testing beforeisapplying new step software version. testand automation inofcombination fault injection aneed virtual agile and production systems emerged withthe various products productof families. To design andThis optimize production approach to achieve negative testiscases forthe a test automation framework. model of reconfigurable the production system seen as next steptotocope improve verification andand validation a production system. work presents an systems as tonegative choose the optimal matches, product analysis methods are needed. Indeed, most of the known methods aim to © 2018 as The Authors. Published Elsevier approach towell achieve testby cases forproduct aB.V. test automation framework. analyze a product or one product family on the physical level. Different families, however, may differ largely ininterms of the number and Peer-review under responsibilityby of Elsevier the scientific committee of the 12thproduct CIRP Conference on Intelligent Computation Manufacturing © The Authors. B.V. © 2018 2019 The Authors. Published Published by Elsevier B.V. nature of components. This fact impedes an efficient comparison and choice of appropriate product family combinations for the production Engineering. Peer-review under responsibility responsibilityofofthe thescientific scientificcommittee committee 12th CIRP Conference on Intelligent Computation in Manufacturing Peer-review under of of thethe 12th CIRP Conference on Intelligent Computation in Manufacturing Engineering. system. A new methodology is proposed to analyze existing products in view of their functional and physical architecture. The aim is to cluster Engineering. these products in new assembly oriented product for the optimization of existing assembly lines and the creation of future reconfigurable Keywords: Negative testing; Test automation; Virtualfamilies commissiong; Hardware-in-the-loop assembly systems. Based on Datum Flow Chain, the physical structure of the products is analyzed. Functional subassemblies are identified, and Keywords: Negative testing; Test automation; Virtual commissiong; Hardware-in-the-loop a functional analysis is performed. Moreover, a hybrid functional and physical architecture graph (HyFPAG) is the output which depicts the similarity between product families by providing design support to both, production system planners and product designers. An illustrative example of a nail-clipper is used to explain the proposed methodology. Ancommissioning industrial case study twotoproduct of steering of “go on back controlfamilies software issues”.columns The PS’s 1. Introduction thyssenkrupp Presta France is then carried out to give a first industrial evaluation of the proposed approach. reliability and “go availability, see [6]software for definition, is more 1. Introduction commissioning back to control issues”. The PS’s © 2017 The Authors. Published by Elsevier dependent software quality thanfor ever.definition, Risk of lifeisfor the Automated production systems haveB.V. the characteristics of a reliability on andtheavailability, see [6] more Peer-review under responsibility ofin the[1]. scientific committee of theof28th Design 2018. quality worker asConference well economic damage consequences of mechatronic system, shown As mechatronic Automated production systems have the most characteristics a CIRP dependent on theassoftware thancan ever.beRisk of life for the
systems, the automated production system is developed in a mechatronic system, shown in [1]. As most mechatronic sequentialtheand independent manner, divided into the three systems, automated production system is developed in a disciplines:and mechanical, electrical anddivided software sequential independent manner, intoengineering the three [2,3]. Therefore, the first integration of allsoftware components and the disciplines: mechanical, electrical and engineering test of its interdisciplinary during Therefore, the first integrationfunctionality of all components and the 1.[2,3]. Introduction development is important. functionality during the test of its process interdisciplinary Mechatronic systems, which were informerly mechanical development process Due to the fastis important. development the domain of systems, gain even more in software proportion through Mechatronic systems, which were mechanical communication and an ongoing trend formerly of digitization and realization of new functions software the substitution of systems, gain even more in in softwareand proportion through digitalization, manufacturing enterprises are facing important functions realized in mechanical electronic realization previously of functions in software and the substitution of challenges in new today’s market environments: aor continuing components [1]. A conducted ofdevelopment over 70 or companies in functionstowards previously realized in mechanical electronic tendency reduction of survey product times and the field of machine tools andInproduction systems revealed that components [1]. Alifecycles. conducted survey of over 70 companies in shortened product addition, there is an increasing [4]: Firstly, most tools of the flexible high-performance the field of customization, machine and production systems that demand of being at the and same timerevealed in a global functionality demanded onlyworld. realizable [4]: Firstly,with most of by thecustomers flexible high-performance competition competitors all overisand the Thisthrough trend, more software employment. Secondly, keeping known functionality demanded customers isfrom only realizable which is inducing the by development macrothe tothrough micro quality from mechanical components is a rather job to more software employment. Secondly, keeping the known markets, results in diminished lot sizes due todifficult augmenting do for varieties production system (PS) companies. Kormann and quality from mechanical components is a ratherproduction) difficult job[1]. to product (high-volume to low-volume Vogel-Heuser [5] augmenting point out that thecompanies. majority during do cope for with production system (PS) Kormann and To this variety as wellofasfailures to be able to Vogel-Heuser [5] point out that the majority of during identify possible optimization potentials in failures the existing 2212-8271 ©system, 2017 The it Authors. Publishedtobyhave Elsevier B.V. production is important a precise knowledge Keywords: Assembly; Design method; Family identification
faulty the PS. damage Since the is delegated by worker behavior as well asofeconomic canPS be consequences of automation software running on different of controls faulty behavior of the PS. Since the PS types is delegated by (numeric programmable logic automationcontrols, software robot runningcontrols, on different types of controls controllers, motion controls master controls) the behavior (numeric controls, robot and controls, programmable logic of the product PS ismotion determined byand themaster automation software. If the controllers, controls) the behavior of the rangecontrols and characteristics manufactured and/or software notsystem. recognize faulty behavior correctly, onIftime of the PSdoes is this determined byathis thecontext, automation software. the assembled in In the main challenge in and does does not correctly on behavior time, thecorrectly, wholewith PSonsingle might software not recognize aand faulty time modelling andreact analysis is now not only to cope get and might betime, hurt. theproduct anddamaged does anot reactpersonnel correctlyrange and on whole PS might products, limited product or existing families, current challenges companies have cope withtoindefine order getThe damaged personnel might hurt. to products but also to be and able to analyze and tobe compare to equilibrate the software overhead are machine downtime and The current challenges companies have to cope with existing in order new product families. It can be observed that classical too much timethe consumption during the process. to equilibrate software overhead arecommissioning machine downtime and product families are regrouped in function of clients or features. At the same time oriented it shows the the importance testing the too much time consumption during commissioning process. However, assembly product families areof hardly to find. automation software in level, interplay withdiffer the of rest of the AtOn thethesame timefamily it shows the importance testing product products mainly in two mechatronic system. automation software(i) in the rest main characteristics: the interplay number of with components and of (ii) the mechatronic system.(e.g. mechanical, electrical, electronical). type of components Classical methodologies considering mainly single products or solitary, already existing product families analyze the product structure on a physical level (components level) which causes difficulties regarding an efficient definition and comparison of different product families. Addressing this
Peer-review the scientific committee 2212-8271 ©under 2017responsibility The Authors. of Published by Elsevier B.V. of the 11th CIRP Conference on Intelligent Computation in Manufacturing Engineering. Peer-review under responsibility of the scientific committee of the 11th CIRP Conference on Intelligent Computation in Manufacturing Engineering. 2212-8271©©2017 2019The The Authors. Published by Elsevier 2212-8271 Authors. Published by Elsevier B.V. B.V. Peer-reviewunder underresponsibility responsibility scientific committee of the CIRP Conference on 2018. Intelligent Computation in Manufacturing Engineering. Peer-review of of thethe scientific committee of the 28th12th CIRP Design Conference 10.1016/j.procir.2019.02.065
Karl Kübler et al. / Procedia CIRP 79 (2019) 268–273 K. Kübler et al. / Procedia CIRP 00 (2018) 000–000
269
Fig. 1. Life cycle of a production system with virtual commissioning and test automation
Eventually, these tendencies lead to (i) the need of new measures to secure software quality with respect to reliability and availability of the whole PS. We do not want to change the whole and well-established engineering process at PS companies. Thus, (ii) we focus on the existing phase of integration (virtual commissioning, see Fig. 1) to implement our method, as it is the moment in the life cycle of the PS where all mechatronic components interact for the first time. The remainder of this paper is structured as follows. In the next section the current state of the development process of production systems is presented. The methods virtual commissioning and test automation as methods of the digital factory [7] used during the integration phase of the overall PS are explained. A glimpse on the concurrent work is given as well as the focus of the method presented in this paper. Eventually, the implementation of the approach is described and discussed. 2. Production systems’ development process Mechatronic components from the different disciplines (mechanical, electrical, electronics and software) are developed in separate departments within a PS company. The fact that engineers and technicians work separately on different components of a PS leads to poor information exchange between the disciplines and thus to sequential development, see dark gray bars in Fig. 1. Integration of all components happens in a late phase of the PS’s life cycle. This late phase of system testing in the life cycle is called virtual commissioning (VCOM) [7]. In this phase, the mechatronic overall system is for the first time brought into service virtually, depicted as red bars in Fig. 1. In order to make this possible, the real control components and the real communication components are plugged together with a virtual model of the PS. The virtual model contains the behavior of mechanical, electrical and electronic components: sensors and actors as well as transportation, handling, tools, workpieces and so on. The virtual model is used in a simulation tool running on a personal computer with an operating system with realtime
capabilities. Such a system is referred to as realtime hardwarein-the-loop (RT-HIL) simulation [8]. The advantage of a RTHIL compared to a non-realtime HIL is the exact behavior of the simulation concerning the time-deterministic bus communication between the simulation and the control system. This makes RT-HIL a VCOM scenario, which is closer to the real behavior of the PS than every other VCOM technique known at present. We would like to make use of the benefits of a VCOM using a RT-HIL system for our approach, therefore (iii) the virtual environment is a RT-HIL system. 2.1. Virtual commissioning as an important life cycle phase of a production system The purpose of VCOM is testing the PS prior to its real assembly and avoiding the issues normally expected during the start-up phase, see [9]. The significant advantage of the VCOM phase, is its parallel execution. Noticeable is the composition of the VCOM phase (see Fig. 1): It consists of the “Modeling” phase and the “Verification & validation” phase. During modeling, the virtual model is created from information and data contributed from previous and currently running development phases of the mechatronic components. The last phase of development is the software phase. The program code retrieved from this phase is the key component to automate the PS. After the automation software development is finished it is verified and validated by testing its functions on the RT-HIL system. In 2015 an advancement for the VCOM process was proposed in [10]. Leaning on a Computer Aided Software Testing (CAST) tool the subprocess during system testing of the PS was automated. Two years later this test automation approach was brought into the market [11] and is currently being developed further. In this work, (iv) we use the framework-based CAST tool approach as test environment to integrate the selection and generation of negative test cases. The CAST tool provides all features necessary for the system testing process: interfaces towards the RT-HIL, test case libraries, logging and graphical test case modeling. The blue bars in Fig. 1 show the current integration of test automation
270
Karl Kübler et al. / Procedia CIRP 79 (2019) 268–273 K. Kübler et al./ Procedia CIRP 00 (2018) 000–000
into the life cycle of a PS. As defined for most cases [12], automation is also here limited to the subprocess of “Execution”.
4. Related work
3. Definition of negative testing using model-implemented fault injection
• (i) Securing software quality with respect to reliability and availability of a PS, therefore using model-implemented fault injection to generate negative test cases. • (ii) Applying the approach in the phase of system testing, as it integrates all components of the PS for the first time. • (iii) Using a RT-HIL system, as it provides the highest accuracy during VCOM. • (iv) Using a framework-based testing environment to run the system tests.
In this paragraph we want to define some expressions in the context of this work. Starting with negative testing and negative test case. Multiple definitions, which are similar but not the same, are given in the information technology and software engineering disciplines. Zelenov and Zelenova [13] describe the term negative test for parsers as follows: ”A negative test for a parser is a sequence of tokens for which the parser returns false […]”. Cauevic, Punnekkat et al. [14] describe negative test cases as „…how the program is behaving for a non-given requirement.“. Takagi and Arao [15] generate negative test cases from mutant sources. ISO 29119 [16] notes that negative test cases are also referred to as “invalid” cases. From the domain of test-driven development negative testing refers to “…exercising a program in a way that was not explicitly specified in the requirement.” [17]. For the process of VCOM of a PS we define negative testing as a method to test the behavior of the automation software while internally forcing the virtual model of the PS into a faulty state. The negative test case holds the description of the single steps to perform the method, which are executed by an automated test framework, for example. The negative test cases also include the steps to put the virtual PS into a faulty state via fault injection, where e.g. sensors or actuators of the PS fail due to simulated wear out or pollution. In software engineering the fault injection testing technique is used to gather information about the reliability of a system [18]. It can be used on a blackbox testing level altering the inputs and outputs of a system, as well as on a white-box testing level by injection code into existing code [19]. Svenningsson, Vinter et al. [20] define model-implemented fault injection as a technique to extend models of hardware, software or systems with model blocks to achieve deviating and faulty behavior. In [21] Svenningsson, Eriksson et al. point out hardware and software faults, taken from [6], which are most suitable for fault injection: production effects, component wear-out and external sources. Avizienis, Laprie et al. [6] describe the interdependence between fault, error and failure. Faults appear within or outside of a system. They might become active and cause errors to evolve in the system. As soon as an error propagates through the system and causes the system to deviate from its correct behavior a failure is caused. Failures themselves can create faults in a system. As a result, all three terms have a causality relationship. We focus on failures of single components of a PS, which evolve as errors in the overall system. The task of the automation software is to detect those errors and to handle them accordingly to the requirements before they cause the whole PS to fail. We consider generation of negative test cases using model-implemented fault injection for the virtual model of the PS as a suitable approach for the automated verification and validation of the reliability of a PS.
Our requirements for the derived approach are as follows:
To the knowledge of the authors there are only a few contributions that can meet more than one of the requirements. Kormann and Vogel-Heuser [5] present a fault injection testing method for programmable logic controllers (PLC). Fault injection in the context of this work means changing one or more of the controller’s input signals to deviate from their standard values and thereby forcing the controller software to react to an irregular situation. The authors of this paper integrated a simulation environment into the runtime of the PLC. Using a Failure Mode and Effects Analysis (FMEA) or Fault Tree Analysis (FTA) relevant faults can be identified and injected into the simulation model by manipulating transitions or states. “The main focus of faults to be examined in this approach is restricted to mechanical malfunctions […].” This contribution falls short to our requirements (i), (ii), (iii) and (iv), as it only simulates a small proportion of the whole PS and is neither running a RT-HIL system nor a framework-based test tool. In [22] Rösch, Tikhonov suggest using fault injection as a means of testing the error handling routines of PLCs. The normal behavior is derived from an adapted timing sequence diagram. The point in execution for the fault injection is extracted from a control flow graph (CFG) of the PLC code. With all this information a test case can be generated. This proposed solution is then validated by manually transferring the test cases into test (PLC) code on the platform for testing of a stamping module in a laboratory plant. This contribution falls short to our requirements (i), (iii) and (iv), as it changes the PLC code using software-implemented fault injection rather than model-implemented fault injection. Rajabpour and Sedaghat [23] describe a technique to monitor the execution flow of distributed supervisory control and data acquisition (SCADA) controllers in a manufacturing environment. The monitoring technique is verified by using fault injection on a real manufacturing system. Although this technique has good results concerning the fault coverage it does not propose any measures to improve automation software quality. This contribution falls short to our requirements (i), (iii) and (iv), as the fault injection it uses is not modelimplemented but software-implemented and thus changes the real PLC code. Also, no RT-HIL system is used as virtual environment for the tests.
Karl Kübler et al. / Procedia CIRP 79 (2019) 268–273 K. Kübler et al. / Procedia CIRP 00 (2018) 000–000
271
Wiebe, Rösch et al. [24] present an approach for system testing on a HIL system using a Simulink virtual model. The goal is to inject faults by signal manipulation in the virtual model. First, a library of test cases is manually created allowing to disturb a single signal or multiple signals as well as a single block or multiple blocks in the virtual model. The library is then combined with information of a manual inspection of the machine and a so-called test suite is generated automatically. The test suite can be executed and manipulates the signals in the simulation during the test run. The approach was implemented and evaluated against a real machine. This contribution falls short to our requirements (iii) and (iv), as it is not using a RT-HIL system to run the tests nor is it using a framework-based test tool. 5. Approach for generation of negative test cases using model-implemented fault injection All steps of the proposed approach are depicted in the flowchart in Fig. 2. The steps are modeled as “point of manual operation” starting with step 1, “Research on failure types of production system components”. The research was performed on literature from component manufacturer’s manuals, from standards and technical literature. Based on [25] a component-based failure classification considering a complete PS was created (step 2 in the flowchart). For each failure type a signal deviation was identified and the signal deviation was modeled in the simulation tool as a reusable signal manipulation block (SMB) (steps 3 and 4). Eight different manipulation types were identified: • • • • • • • •
Drift Zero point failure Time delay Random noise Hard-over Complete failure Freezing/stuck Sign error
By replacing signal paths of components in an existing virtual model (steps 5 and 6) an enhanced virtual model was created (step 7 in the flowchart). The blocks of these eight SMBs can be described as a black-box with inputs and outputs (I/Os). Inputs are 1) the regular signal 2) the port to activate the SMB, 3) the trigger condition and 4) inputs for corresponding parameters, e.g. the gradient of the drift. The trigger condition can be chosen from multiple options, e.g. timing with/ without delay and conditions derived from I/Os of other blocks. Since these trigger conditions are verified within the realtime environment it is possible to achieve a realtime testing behavior: The activation and trigger conditions are preloaded by the CAST tool when starting the test run, while the execution is triggered by the realtime environment of the simulation. Finally, the enhanced virtual model is used in system tests for the PS.
Fig. 2. Flowchart of the presented approach
Therefore, negative test cases were created in the graphical user interface of the CAST tool, see steps 8, 9 and 10 in Fig. 2. The procedure is to match functions from PLC code with components in the virtual model. In this manner test cases were created triggering failures of the components just when they were in use by the executed PLC code. Fig. 3 shows the modeled classification of failures in a PS, in the back. The model was created as a Unified Markup Language (UML) class diagram. It is a universally valid diagram, it contains all mechatronic components of a PS and each component is associated with its typical failure types. Additionally the failure types are linked with one of the eight signal manipulations. Exemplary, the electric failure types of a motor are presented, Fig. 3 in the front. The electric motor is part of the path “Production system”, “Inherent faults”, “Component faults”, “Drive faults”, “Electric motor /servo motor faults”, “Motor faults”, “Faults of the electronic parts” and is associated with three failures.
272
Karl Kübler et al. / Procedia CIRP 79 (2019) 268–273 K. Kübler et al./ Procedia CIRP 00 (2018) 000–000
Fig. 3. Class diagram with modeled failure classification (back), zoom (front)
5.1. Example application on a virtual machining center As an example to showcase the approach, a machining center controlled by a computerized numerical control (CNC) was virtually commissioned using a RT-HIL system. Fig. 4 shows the test environment with RT-HIL, CAST tool and the negative test cases. The numbers 1 to 3 show the places of intervention within the test automation environment: • (1) Exchange of the virtual model (behavioral part only) with the enhanced virtual model which contains the SMBs. • (2) Manual creation of the negative test cases via the CAST tool’s graphical user interface. • (3) Automated execution of the negative test cases. By using one integrated SMB from the virtual model per negative test case, 27 negative test cases could be created for one machining center. Negative test cases, which combine multiple SMBs, were not included yet. During manufacturing, tool change is a critical situation in a machining tool. Especially the correct continuation after a tool change got interrupted is very critical. Therefore, some of the negative test cases were aimed at parts involved in a tool change, e.g. the tool handling arm or the pneumatic system.
Fig. 5. Signal traces of test case “pressure loss during tool change”
During automatic tool change (ATC) the tool door is opened and the tool holder is moved to the tool magazine. After the correct tool is placed in the tool holder it is moved to the staging place where the tool handling arm flips it towards the spindle were it is clamped. In one exemplary negative test case the pneumatic pressure in the machining center is interrupted. Fig. 5 shows signal-time diagrams of the I/Os involved in the ATC. The ATC is interrupted by an overall pressure loss in the machining tools, see third diagram from the bottom “compressed air”. At the markings 1 and 2 the CNC issued an error message indicating the loss of pressure 969 milliseconds after the failure was injected. The ATC is not finished correctly as the signal “tool is present” at the spindle never becomes “1” (true), see last diagram in Fig. 5. More error messages appear indicating the timeout of the ATC process and an open tool door. In this test the PLC code on the CNC worked correctly. 6. Discussion of the approach and the derived results The authors see the following advantages in the approach presented:
Fig. 4. Integration of the approach into the automated test environment
• Independence of the toolchain: by modeling the SMBs for the virtual model within another simulation tool and creating
Karl Kübler et al. / Procedia CIRP 79 (2019) 268–273 K. Kübler et al. / Procedia CIRP 00 (2018) 000–000
the negative test cases within another CAST tool the use of the approach is independent from the toolchain. • Applicable in existing development processes: by manually enhancing the virtual model with the SMBs a fault injected model can be created even in an existing project. • Extending the knowledge of each mechatronic component: By integrating the corresponding failure types. This is helpful during the design of new production systems as the engineers can rely on the previously added failure types. The tester operating the CAST tool during the verification and validation phase does not need to know about specific failure types. It is sufficient to trigger the block to generate the typical faulty behavior of a component. • With the failure types integrated into the virtual model in the RT-HIL a realtime testing capability is given in this approach. Time-deterministic triggering and lossless tracing during a test run can be guaranteed by the realtime environment. Improvements and future work on the approach are: • Automation of the presented workflow in Fig. 2. • Application of new methods and algorithms to generate negative test cases from the enhanced virtual model. • When applying the approach at a PS company one should consider the experience of the service personnel. They are aware of incidents, which happen during commissioning and production on site. These incidents can be mapped with failure types and negative test cases can be derived using the presented approach. 7. Summary and outlook In this paper the current state of the development process of production systems was lined out. With focus on the phase of virtual commissioning, using test automation to perform verification and validation, an approach for creating negative test cases from enhanced virtual models was presented. Within the approach, causes for faulty behavior (failure types) of mechatronic components from a PS were researched and classified. From the identified behavior a library was derived containing signal manipulation blocks (SMB) according to the failure types. These SMBs are universally valid for virtual models of production systems. The SMBs were included into an existing virtual model of a machining center (modelimplemented fault injection). Each SMB can then be triggered within a negative testing test run to verify the automation software’s reaction of error recognition and error handling. The implementation of the complete procedure was presented on an existing tool chain. Findings of the approach were discussed. For efficient use of the approach an adapted procedure should be established to be able to automate the complete process according to the workflow in Fig. 2. This can only be done by taking into account the specific tools in use. Current work is done on expanding the possibilities on the realtime testing capabilities given by the presented approach.
273
References [1] Verein Deutscher Ingenieure. Design methodology for mechatronic systems;03.100.40;31.220(2206). Berlin: Beuth Verlag; 2004. [2] Rzevski G. On conceptual design of intelligent mechatronic systems. Mechatronics 2003;13(10):1029–44. [3] Thramboulidis K. The 3+1 SysML View-Model in Model Integrated Mechatronics. JSEA 2010;03(02):109–18. [4] Linke J. Der SPS-Benchmark. Computer Automation 2011(9):55–8. [5] Kormann B, Vogel-Heuser B. Automated test case generation approach for PLC control software exception handling using fault injection. In: 37th IECON 2011. 2011, p. 365–372. [6] Avizienis A, Laprie J-C, Randell B, Landwehr C. Basic concepts and taxonomy of dependable and secure computing. IEEE Trans.Dependable and Secure Comput. 2004;1(1):11–33. [7] Verein Deutscher Ingenieure. Digital Factory - Digital Factory Operations; (4499 Part 2). Berlin: Beuth Verlag; 2011. [8] Pritschow G, Röck S. “Hardware in the Loop” Simulation of Machine Tools. CIRP Annals - Manufacturing Technology 2004;53(1):295–8. [9] Gamblin R. Machine tools: Specification, purchase, and installation. New York: McGraw-Hill Ed; 2014. [10] Kübler K, Neyrinck A, Schlechtendahl J, Lechler A, Verl A. Approach for Manufacturer Independent Automated Machine Tool Control Software Test. In: Wulfsberg JP, editor. WGP Congress 2015. Pfaffikon: TTP Inc; 2015, p. 347–354. [11] Kübler K. Presentation at Cluster Mechatronik & Automation: Automatisiertes Testen im Umfeld funktionskritischer Systeme 18.03.2018; Nuremburg; 2018. [12] International standard. Software and systems engineering - Software testing Part 1: Concepts and definitions. 1st ed;35.080(29119-1); 2013. [13] Zelenov SV, Zelenova SA. Generation of Positive and Negative Tests for Parsers. Program Comput Soft 2005;31(6):310–20. [14] Cauevic A, Punnekkat S, Sundmark D. Quality of Testing in Test Driven Development. In: 2012 Eighth International Conference on the Quality of Information and Communications Technology. 2012, p. 266–271. [15] Takagi T, Arao T. Overview of a place/transition net-based mutation testing framework to obtain test cases effective for concurrent software. In: 2015 IEEE/ACIS 16th SNPD.2015, p. 1–3. [16] International standard. Software and systems engineering - Software testing Part 4: Test techniques. 1st ed;35.080(29119-4); 2015. [17] Causevic A, Shukla R, Punnekkat S, Sundmark D. Effects of Negative Testing on TDD: An Industrial Experiment. In: van der Aalst W et al., editors. Agile Processes in Software Engineering and Extreme Programming. Springer; 2013, p. 91–105. [18] Sommerville I. Software engineering. 9th ed. Boston MA: AddisonWesley; 2011. [19] Voas J. Fault injection for the masses. Computer 1997;30(12):129–30. [20] Svenningsson R, Vinter J, Eriksson H, Törngren M. MODIFI: A MODelImplemented Fault Injection Tool. In: Hutchison D et al., editors. Computer Safety, Reliability, and Security. Springer Berlin Heidelberg; 2010, p. 210–222. [21] Svenningsson R, Eriksson H, Vinter J, Törngren M. Model-Implemented Fault Injection for Hardware Fault Simulation. In: 2010 Workshop on Model-Driven Engineering, Verification, and Validation.2010, p. 31–36. [22] Rösch S, Tikhonov D, Schütz D, Vogel-Heuser B. Model-based testing of PLC software: Test of plants' reliability by using fault injection on component level. IFAC Proceedings Volumes 2014;47(3):3509–15. [23] Rajabpour N, Sedaghat Y. A hybrid-based error detection technique for PLC-based Industrial Control Systems. In: 2015 IEEE 20th Conference on Emerging Technologies & Factory Automation (ETFA).2015, p. 1–7. [24] Wiebe F, Rösch S, Rehberger S, Vogel-Heuser B. Automated test suite generation to test modular designed packaging machines using Fault Injection and a simulink-based simulation approach. In: 2016 IEEE CASE; p. 758–765. [25] TUM, Lehrstuhl für Rechnertechnik und Rechnerorganisation. IMoMeSA Abschlussbericht. München; 2015.
ScienceDirect ScienceDirect Procedia CIRP 00 (2018) 000–000 ScienceDirect Procedia CIRP 00 (2018) 000–000 ScienceDirect
Available online atonline www.sciencedirect.com Available at www.sciencedirect.com
Procedia CIRP 00 (2017) 000–000 Procedia CIRP 79 (2019) 268–273
www.elsevier.com/locate/procedia www.elsevier.com/locate/procedia
www.elsevier.com/locate/procedia
12th 2018, 12thCIRP CIRPConference Conferenceon onIntelligent IntelligentComputation ComputationininManufacturing ManufacturingEngineering, Engineering,18-20 CIRPJuly ICME '18 12th CIRP Conference on Intelligent Computation in Manufacturing Engineering, CIRP ICME '18 Gulf of Naples, Italy 28th CIRP Design Conference, May 2018, Nantes, France Test case generation for production systems with model-implemented fault
Test case generation for production systems with model-implemented fault injection consideration A new methodology to analyze the consideration functional and physical architecture of injection a, a Kübler *, Elmar oriented Schwarza, Alexander existing products Karl for an assembly product Verl family identification a, a a Karl Kübler *, Elmar Schwarz , Alexander Verl PaulofStief *,Tools Jean-Yves Dantan, Alain Etienne, AliSeidenstr. Siadat36, 70174 Stuttgart, Germany Institute for Control Engineering Machine and Manufacturing Units (ISW), University of Stuttgart, Institute for Control Engineering of Machine Tools and Manufacturing Units (ISW), University of Stuttgart, Seidenstr. 36, 70174 Stuttgart, Germany
a a
* Corresponding Tel.:Supérieure +49 711 685-82393. E-mail address: [email protected] Écoleauthor. Nationale d’Arts et Métiers, Arts et Métiers ParisTech, LCFC EA 4495, 4 Rue Augustin Fresnel, Metz 57078, France * Corresponding author. Tel.: +49 711 685-82393. E-mail address: [email protected]
* Corresponding author. Tel.: +33 3 87 37 54 30; E-mail address: [email protected]
Abstract Abstract Complex production systems, which have to handle product diversity and short product life cycles, can only be stable and efficient when Abstract successfully tested prior to theirwhich start-up. changes in automation within the life of can a production systemand should be verified Complex production systems, haveAlso, to handle product diversitysoftware and short product life cycle cycles, only be stable efficient when through repetitive applying Also, the new software version. Therefore, automation in combination with fault injection virtual successfully testedtesting prior tobefore their start-up. changes in automation software test within the life cycle of a production system shouldon be averified Inthrough today’s business environment, the trend towards moretoproduct variety and customization is unbroken. Due towith this development, the of model ofrepetitive the production system seen as the next improve theTherefore, verification validation a production system. This work on presents an testing beforeisapplying new step software version. testand automation inofcombination fault injection aneed virtual agile and production systems emerged withthe various products productof families. To design andThis optimize production approach to achieve negative testiscases forthe a test automation framework. model of reconfigurable the production system seen as next steptotocope improve verification andand validation a production system. work presents an systems as tonegative choose the optimal matches, product analysis methods are needed. Indeed, most of the known methods aim to © 2018 as The Authors. Published Elsevier approach towell achieve testby cases forproduct aB.V. test automation framework. analyze a product or one product family on the physical level. Different families, however, may differ largely ininterms of the number and Peer-review under responsibilityby of Elsevier the scientific committee of the 12thproduct CIRP Conference on Intelligent Computation Manufacturing © The Authors. B.V. © 2018 2019 The Authors. Published Published by Elsevier B.V. nature of components. This fact impedes an efficient comparison and choice of appropriate product family combinations for the production Engineering. Peer-review under responsibility responsibilityofofthe thescientific scientificcommittee committee 12th CIRP Conference on Intelligent Computation in Manufacturing Peer-review under of of thethe 12th CIRP Conference on Intelligent Computation in Manufacturing Engineering. system. A new methodology is proposed to analyze existing products in view of their functional and physical architecture. The aim is to cluster Engineering. these products in new assembly oriented product for the optimization of existing assembly lines and the creation of future reconfigurable Keywords: Negative testing; Test automation; Virtualfamilies commissiong; Hardware-in-the-loop assembly systems. Based on Datum Flow Chain, the physical structure of the products is analyzed. Functional subassemblies are identified, and Keywords: Negative testing; Test automation; Virtual commissiong; Hardware-in-the-loop a functional analysis is performed. Moreover, a hybrid functional and physical architecture graph (HyFPAG) is the output which depicts the similarity between product families by providing design support to both, production system planners and product designers. An illustrative example of a nail-clipper is used to explain the proposed methodology. Ancommissioning industrial case study twotoproduct of steering of “go on back controlfamilies software issues”.columns The PS’s 1. Introduction thyssenkrupp Presta France is then carried out to give a first industrial evaluation of the proposed approach. reliability and “go availability, see [6]software for definition, is more 1. Introduction commissioning back to control issues”. The PS’s © 2017 The Authors. Published by Elsevier dependent software quality thanfor ever.definition, Risk of lifeisfor the Automated production systems haveB.V. the characteristics of a reliability on andtheavailability, see [6] more Peer-review under responsibility ofin the[1]. scientific committee of theof28th Design 2018. quality worker asConference well economic damage consequences of mechatronic system, shown As mechatronic Automated production systems have the most characteristics a CIRP dependent on theassoftware thancan ever.beRisk of life for the
systems, the automated production system is developed in a mechatronic system, shown in [1]. As most mechatronic sequentialtheand independent manner, divided into the three systems, automated production system is developed in a disciplines:and mechanical, electrical anddivided software sequential independent manner, intoengineering the three [2,3]. Therefore, the first integration of allsoftware components and the disciplines: mechanical, electrical and engineering test of its interdisciplinary during Therefore, the first integrationfunctionality of all components and the 1.[2,3]. Introduction development is important. functionality during the test of its process interdisciplinary Mechatronic systems, which were informerly mechanical development process Due to the fastis important. development the domain of systems, gain even more in software proportion through Mechatronic systems, which were mechanical communication and an ongoing trend formerly of digitization and realization of new functions software the substitution of systems, gain even more in in softwareand proportion through digitalization, manufacturing enterprises are facing important functions realized in mechanical electronic realization previously of functions in software and the substitution of challenges in new today’s market environments: aor continuing components [1]. A conducted ofdevelopment over 70 or companies in functionstowards previously realized in mechanical electronic tendency reduction of survey product times and the field of machine tools andInproduction systems revealed that components [1]. Alifecycles. conducted survey of over 70 companies in shortened product addition, there is an increasing [4]: Firstly, most tools of the flexible high-performance the field of customization, machine and production systems that demand of being at the and same timerevealed in a global functionality demanded onlyworld. realizable [4]: Firstly,with most of by thecustomers flexible high-performance competition competitors all overisand the Thisthrough trend, more software employment. Secondly, keeping known functionality demanded customers isfrom only realizable which is inducing the by development macrothe tothrough micro quality from mechanical components is a rather job to more software employment. Secondly, keeping the known markets, results in diminished lot sizes due todifficult augmenting do for varieties production system (PS) companies. Kormann and quality from mechanical components is a ratherproduction) difficult job[1]. to product (high-volume to low-volume Vogel-Heuser [5] augmenting point out that thecompanies. majority during do cope for with production system (PS) Kormann and To this variety as wellofasfailures to be able to Vogel-Heuser [5] point out that the majority of during identify possible optimization potentials in failures the existing 2212-8271 ©system, 2017 The it Authors. Publishedtobyhave Elsevier B.V. production is important a precise knowledge Keywords: Assembly; Design method; Family identification
faulty the PS. damage Since the is delegated by worker behavior as well asofeconomic canPS be consequences of automation software running on different of controls faulty behavior of the PS. Since the PS types is delegated by (numeric programmable logic automationcontrols, software robot runningcontrols, on different types of controls controllers, motion controls master controls) the behavior (numeric controls, robot and controls, programmable logic of the product PS ismotion determined byand themaster automation software. If the controllers, controls) the behavior of the rangecontrols and characteristics manufactured and/or software notsystem. recognize faulty behavior correctly, onIftime of the PSdoes is this determined byathis thecontext, automation software. the assembled in In the main challenge in and does does not correctly on behavior time, thecorrectly, wholewith PSonsingle might software not recognize aand faulty time modelling andreact analysis is now not only to cope get and might betime, hurt. theproduct anddamaged does anot reactpersonnel correctlyrange and on whole PS might products, limited product or existing families, current challenges companies have cope withtoindefine order getThe damaged personnel might hurt. to products but also to be and able to analyze and tobe compare to equilibrate the software overhead are machine downtime and The current challenges companies have to cope with existing in order new product families. It can be observed that classical too much timethe consumption during the process. to equilibrate software overhead arecommissioning machine downtime and product families are regrouped in function of clients or features. At the same time oriented it shows the the importance testing the too much time consumption during commissioning process. However, assembly product families areof hardly to find. automation software in level, interplay withdiffer the of rest of the AtOn thethesame timefamily it shows the importance testing product products mainly in two mechatronic system. automation software(i) in the rest main characteristics: the interplay number of with components and of (ii) the mechatronic system.(e.g. mechanical, electrical, electronical). type of components Classical methodologies considering mainly single products or solitary, already existing product families analyze the product structure on a physical level (components level) which causes difficulties regarding an efficient definition and comparison of different product families. Addressing this
Peer-review the scientific committee 2212-8271 ©under 2017responsibility The Authors. of Published by Elsevier B.V. of the 11th CIRP Conference on Intelligent Computation in Manufacturing Engineering. Peer-review under responsibility of the scientific committee of the 11th CIRP Conference on Intelligent Computation in Manufacturing Engineering. 2212-8271©©2017 2019The The Authors. Published by Elsevier 2212-8271 Authors. Published by Elsevier B.V. B.V. Peer-reviewunder underresponsibility responsibility scientific committee of the CIRP Conference on 2018. Intelligent Computation in Manufacturing Engineering. Peer-review of of thethe scientific committee of the 28th12th CIRP Design Conference 10.1016/j.procir.2019.02.065
Karl Kübler et al. / Procedia CIRP 79 (2019) 268–273 K. Kübler et al. / Procedia CIRP 00 (2018) 000–000
269
Fig. 1. Life cycle of a production system with virtual commissioning and test automation
Eventually, these tendencies lead to (i) the need of new measures to secure software quality with respect to reliability and availability of the whole PS. We do not want to change the whole and well-established engineering process at PS companies. Thus, (ii) we focus on the existing phase of integration (virtual commissioning, see Fig. 1) to implement our method, as it is the moment in the life cycle of the PS where all mechatronic components interact for the first time. The remainder of this paper is structured as follows. In the next section the current state of the development process of production systems is presented. The methods virtual commissioning and test automation as methods of the digital factory [7] used during the integration phase of the overall PS are explained. A glimpse on the concurrent work is given as well as the focus of the method presented in this paper. Eventually, the implementation of the approach is described and discussed. 2. Production systems’ development process Mechatronic components from the different disciplines (mechanical, electrical, electronics and software) are developed in separate departments within a PS company. The fact that engineers and technicians work separately on different components of a PS leads to poor information exchange between the disciplines and thus to sequential development, see dark gray bars in Fig. 1. Integration of all components happens in a late phase of the PS’s life cycle. This late phase of system testing in the life cycle is called virtual commissioning (VCOM) [7]. In this phase, the mechatronic overall system is for the first time brought into service virtually, depicted as red bars in Fig. 1. In order to make this possible, the real control components and the real communication components are plugged together with a virtual model of the PS. The virtual model contains the behavior of mechanical, electrical and electronic components: sensors and actors as well as transportation, handling, tools, workpieces and so on. The virtual model is used in a simulation tool running on a personal computer with an operating system with realtime
capabilities. Such a system is referred to as realtime hardwarein-the-loop (RT-HIL) simulation [8]. The advantage of a RTHIL compared to a non-realtime HIL is the exact behavior of the simulation concerning the time-deterministic bus communication between the simulation and the control system. This makes RT-HIL a VCOM scenario, which is closer to the real behavior of the PS than every other VCOM technique known at present. We would like to make use of the benefits of a VCOM using a RT-HIL system for our approach, therefore (iii) the virtual environment is a RT-HIL system. 2.1. Virtual commissioning as an important life cycle phase of a production system The purpose of VCOM is testing the PS prior to its real assembly and avoiding the issues normally expected during the start-up phase, see [9]. The significant advantage of the VCOM phase, is its parallel execution. Noticeable is the composition of the VCOM phase (see Fig. 1): It consists of the “Modeling” phase and the “Verification & validation” phase. During modeling, the virtual model is created from information and data contributed from previous and currently running development phases of the mechatronic components. The last phase of development is the software phase. The program code retrieved from this phase is the key component to automate the PS. After the automation software development is finished it is verified and validated by testing its functions on the RT-HIL system. In 2015 an advancement for the VCOM process was proposed in [10]. Leaning on a Computer Aided Software Testing (CAST) tool the subprocess during system testing of the PS was automated. Two years later this test automation approach was brought into the market [11] and is currently being developed further. In this work, (iv) we use the framework-based CAST tool approach as test environment to integrate the selection and generation of negative test cases. The CAST tool provides all features necessary for the system testing process: interfaces towards the RT-HIL, test case libraries, logging and graphical test case modeling. The blue bars in Fig. 1 show the current integration of test automation
270
Karl Kübler et al. / Procedia CIRP 79 (2019) 268–273 K. Kübler et al./ Procedia CIRP 00 (2018) 000–000
into the life cycle of a PS. As defined for most cases [12], automation is also here limited to the subprocess of “Execution”.
4. Related work
3. Definition of negative testing using model-implemented fault injection
• (i) Securing software quality with respect to reliability and availability of a PS, therefore using model-implemented fault injection to generate negative test cases. • (ii) Applying the approach in the phase of system testing, as it integrates all components of the PS for the first time. • (iii) Using a RT-HIL system, as it provides the highest accuracy during VCOM. • (iv) Using a framework-based testing environment to run the system tests.
In this paragraph we want to define some expressions in the context of this work. Starting with negative testing and negative test case. Multiple definitions, which are similar but not the same, are given in the information technology and software engineering disciplines. Zelenov and Zelenova [13] describe the term negative test for parsers as follows: ”A negative test for a parser is a sequence of tokens for which the parser returns false […]”. Cauevic, Punnekkat et al. [14] describe negative test cases as „…how the program is behaving for a non-given requirement.“. Takagi and Arao [15] generate negative test cases from mutant sources. ISO 29119 [16] notes that negative test cases are also referred to as “invalid” cases. From the domain of test-driven development negative testing refers to “…exercising a program in a way that was not explicitly specified in the requirement.” [17]. For the process of VCOM of a PS we define negative testing as a method to test the behavior of the automation software while internally forcing the virtual model of the PS into a faulty state. The negative test case holds the description of the single steps to perform the method, which are executed by an automated test framework, for example. The negative test cases also include the steps to put the virtual PS into a faulty state via fault injection, where e.g. sensors or actuators of the PS fail due to simulated wear out or pollution. In software engineering the fault injection testing technique is used to gather information about the reliability of a system [18]. It can be used on a blackbox testing level altering the inputs and outputs of a system, as well as on a white-box testing level by injection code into existing code [19]. Svenningsson, Vinter et al. [20] define model-implemented fault injection as a technique to extend models of hardware, software or systems with model blocks to achieve deviating and faulty behavior. In [21] Svenningsson, Eriksson et al. point out hardware and software faults, taken from [6], which are most suitable for fault injection: production effects, component wear-out and external sources. Avizienis, Laprie et al. [6] describe the interdependence between fault, error and failure. Faults appear within or outside of a system. They might become active and cause errors to evolve in the system. As soon as an error propagates through the system and causes the system to deviate from its correct behavior a failure is caused. Failures themselves can create faults in a system. As a result, all three terms have a causality relationship. We focus on failures of single components of a PS, which evolve as errors in the overall system. The task of the automation software is to detect those errors and to handle them accordingly to the requirements before they cause the whole PS to fail. We consider generation of negative test cases using model-implemented fault injection for the virtual model of the PS as a suitable approach for the automated verification and validation of the reliability of a PS.
Our requirements for the derived approach are as follows:
To the knowledge of the authors there are only a few contributions that can meet more than one of the requirements. Kormann and Vogel-Heuser [5] present a fault injection testing method for programmable logic controllers (PLC). Fault injection in the context of this work means changing one or more of the controller’s input signals to deviate from their standard values and thereby forcing the controller software to react to an irregular situation. The authors of this paper integrated a simulation environment into the runtime of the PLC. Using a Failure Mode and Effects Analysis (FMEA) or Fault Tree Analysis (FTA) relevant faults can be identified and injected into the simulation model by manipulating transitions or states. “The main focus of faults to be examined in this approach is restricted to mechanical malfunctions […].” This contribution falls short to our requirements (i), (ii), (iii) and (iv), as it only simulates a small proportion of the whole PS and is neither running a RT-HIL system nor a framework-based test tool. In [22] Rösch, Tikhonov suggest using fault injection as a means of testing the error handling routines of PLCs. The normal behavior is derived from an adapted timing sequence diagram. The point in execution for the fault injection is extracted from a control flow graph (CFG) of the PLC code. With all this information a test case can be generated. This proposed solution is then validated by manually transferring the test cases into test (PLC) code on the platform for testing of a stamping module in a laboratory plant. This contribution falls short to our requirements (i), (iii) and (iv), as it changes the PLC code using software-implemented fault injection rather than model-implemented fault injection. Rajabpour and Sedaghat [23] describe a technique to monitor the execution flow of distributed supervisory control and data acquisition (SCADA) controllers in a manufacturing environment. The monitoring technique is verified by using fault injection on a real manufacturing system. Although this technique has good results concerning the fault coverage it does not propose any measures to improve automation software quality. This contribution falls short to our requirements (i), (iii) and (iv), as the fault injection it uses is not modelimplemented but software-implemented and thus changes the real PLC code. Also, no RT-HIL system is used as virtual environment for the tests.
Karl Kübler et al. / Procedia CIRP 79 (2019) 268–273 K. Kübler et al. / Procedia CIRP 00 (2018) 000–000
271
Wiebe, Rösch et al. [24] present an approach for system testing on a HIL system using a Simulink virtual model. The goal is to inject faults by signal manipulation in the virtual model. First, a library of test cases is manually created allowing to disturb a single signal or multiple signals as well as a single block or multiple blocks in the virtual model. The library is then combined with information of a manual inspection of the machine and a so-called test suite is generated automatically. The test suite can be executed and manipulates the signals in the simulation during the test run. The approach was implemented and evaluated against a real machine. This contribution falls short to our requirements (iii) and (iv), as it is not using a RT-HIL system to run the tests nor is it using a framework-based test tool. 5. Approach for generation of negative test cases using model-implemented fault injection All steps of the proposed approach are depicted in the flowchart in Fig. 2. The steps are modeled as “point of manual operation” starting with step 1, “Research on failure types of production system components”. The research was performed on literature from component manufacturer’s manuals, from standards and technical literature. Based on [25] a component-based failure classification considering a complete PS was created (step 2 in the flowchart). For each failure type a signal deviation was identified and the signal deviation was modeled in the simulation tool as a reusable signal manipulation block (SMB) (steps 3 and 4). Eight different manipulation types were identified: • • • • • • • •
Drift Zero point failure Time delay Random noise Hard-over Complete failure Freezing/stuck Sign error
By replacing signal paths of components in an existing virtual model (steps 5 and 6) an enhanced virtual model was created (step 7 in the flowchart). The blocks of these eight SMBs can be described as a black-box with inputs and outputs (I/Os). Inputs are 1) the regular signal 2) the port to activate the SMB, 3) the trigger condition and 4) inputs for corresponding parameters, e.g. the gradient of the drift. The trigger condition can be chosen from multiple options, e.g. timing with/ without delay and conditions derived from I/Os of other blocks. Since these trigger conditions are verified within the realtime environment it is possible to achieve a realtime testing behavior: The activation and trigger conditions are preloaded by the CAST tool when starting the test run, while the execution is triggered by the realtime environment of the simulation. Finally, the enhanced virtual model is used in system tests for the PS.
Fig. 2. Flowchart of the presented approach
Therefore, negative test cases were created in the graphical user interface of the CAST tool, see steps 8, 9 and 10 in Fig. 2. The procedure is to match functions from PLC code with components in the virtual model. In this manner test cases were created triggering failures of the components just when they were in use by the executed PLC code. Fig. 3 shows the modeled classification of failures in a PS, in the back. The model was created as a Unified Markup Language (UML) class diagram. It is a universally valid diagram, it contains all mechatronic components of a PS and each component is associated with its typical failure types. Additionally the failure types are linked with one of the eight signal manipulations. Exemplary, the electric failure types of a motor are presented, Fig. 3 in the front. The electric motor is part of the path “Production system”, “Inherent faults”, “Component faults”, “Drive faults”, “Electric motor /servo motor faults”, “Motor faults”, “Faults of the electronic parts” and is associated with three failures.
272
Karl Kübler et al. / Procedia CIRP 79 (2019) 268–273 K. Kübler et al./ Procedia CIRP 00 (2018) 000–000
Fig. 3. Class diagram with modeled failure classification (back), zoom (front)
5.1. Example application on a virtual machining center As an example to showcase the approach, a machining center controlled by a computerized numerical control (CNC) was virtually commissioned using a RT-HIL system. Fig. 4 shows the test environment with RT-HIL, CAST tool and the negative test cases. The numbers 1 to 3 show the places of intervention within the test automation environment: • (1) Exchange of the virtual model (behavioral part only) with the enhanced virtual model which contains the SMBs. • (2) Manual creation of the negative test cases via the CAST tool’s graphical user interface. • (3) Automated execution of the negative test cases. By using one integrated SMB from the virtual model per negative test case, 27 negative test cases could be created for one machining center. Negative test cases, which combine multiple SMBs, were not included yet. During manufacturing, tool change is a critical situation in a machining tool. Especially the correct continuation after a tool change got interrupted is very critical. Therefore, some of the negative test cases were aimed at parts involved in a tool change, e.g. the tool handling arm or the pneumatic system.
Fig. 5. Signal traces of test case “pressure loss during tool change”
During automatic tool change (ATC) the tool door is opened and the tool holder is moved to the tool magazine. After the correct tool is placed in the tool holder it is moved to the staging place where the tool handling arm flips it towards the spindle were it is clamped. In one exemplary negative test case the pneumatic pressure in the machining center is interrupted. Fig. 5 shows signal-time diagrams of the I/Os involved in the ATC. The ATC is interrupted by an overall pressure loss in the machining tools, see third diagram from the bottom “compressed air”. At the markings 1 and 2 the CNC issued an error message indicating the loss of pressure 969 milliseconds after the failure was injected. The ATC is not finished correctly as the signal “tool is present” at the spindle never becomes “1” (true), see last diagram in Fig. 5. More error messages appear indicating the timeout of the ATC process and an open tool door. In this test the PLC code on the CNC worked correctly. 6. Discussion of the approach and the derived results The authors see the following advantages in the approach presented:
Fig. 4. Integration of the approach into the automated test environment
• Independence of the toolchain: by modeling the SMBs for the virtual model within another simulation tool and creating
Karl Kübler et al. / Procedia CIRP 79 (2019) 268–273 K. Kübler et al. / Procedia CIRP 00 (2018) 000–000
the negative test cases within another CAST tool the use of the approach is independent from the toolchain. • Applicable in existing development processes: by manually enhancing the virtual model with the SMBs a fault injected model can be created even in an existing project. • Extending the knowledge of each mechatronic component: By integrating the corresponding failure types. This is helpful during the design of new production systems as the engineers can rely on the previously added failure types. The tester operating the CAST tool during the verification and validation phase does not need to know about specific failure types. It is sufficient to trigger the block to generate the typical faulty behavior of a component. • With the failure types integrated into the virtual model in the RT-HIL a realtime testing capability is given in this approach. Time-deterministic triggering and lossless tracing during a test run can be guaranteed by the realtime environment. Improvements and future work on the approach are: • Automation of the presented workflow in Fig. 2. • Application of new methods and algorithms to generate negative test cases from the enhanced virtual model. • When applying the approach at a PS company one should consider the experience of the service personnel. They are aware of incidents, which happen during commissioning and production on site. These incidents can be mapped with failure types and negative test cases can be derived using the presented approach. 7. Summary and outlook In this paper the current state of the development process of production systems was lined out. With focus on the phase of virtual commissioning, using test automation to perform verification and validation, an approach for creating negative test cases from enhanced virtual models was presented. Within the approach, causes for faulty behavior (failure types) of mechatronic components from a PS were researched and classified. From the identified behavior a library was derived containing signal manipulation blocks (SMB) according to the failure types. These SMBs are universally valid for virtual models of production systems. The SMBs were included into an existing virtual model of a machining center (modelimplemented fault injection). Each SMB can then be triggered within a negative testing test run to verify the automation software’s reaction of error recognition and error handling. The implementation of the complete procedure was presented on an existing tool chain. Findings of the approach were discussed. For efficient use of the approach an adapted procedure should be established to be able to automate the complete process according to the workflow in Fig. 2. This can only be done by taking into account the specific tools in use. Current work is done on expanding the possibilities on the realtime testing capabilities given by the presented approach.
273
References [1] Verein Deutscher Ingenieure. Design methodology for mechatronic systems;03.100.40;31.220(2206). Berlin: Beuth Verlag; 2004. [2] Rzevski G. On conceptual design of intelligent mechatronic systems. Mechatronics 2003;13(10):1029–44. [3] Thramboulidis K. The 3+1 SysML View-Model in Model Integrated Mechatronics. JSEA 2010;03(02):109–18. [4] Linke J. Der SPS-Benchmark. Computer Automation 2011(9):55–8. [5] Kormann B, Vogel-Heuser B. Automated test case generation approach for PLC control software exception handling using fault injection. In: 37th IECON 2011. 2011, p. 365–372. [6] Avizienis A, Laprie J-C, Randell B, Landwehr C. Basic concepts and taxonomy of dependable and secure computing. IEEE Trans.Dependable and Secure Comput. 2004;1(1):11–33. [7] Verein Deutscher Ingenieure. Digital Factory - Digital Factory Operations; (4499 Part 2). Berlin: Beuth Verlag; 2011. [8] Pritschow G, Röck S. “Hardware in the Loop” Simulation of Machine Tools. CIRP Annals - Manufacturing Technology 2004;53(1):295–8. [9] Gamblin R. Machine tools: Specification, purchase, and installation. New York: McGraw-Hill Ed; 2014. [10] Kübler K, Neyrinck A, Schlechtendahl J, Lechler A, Verl A. Approach for Manufacturer Independent Automated Machine Tool Control Software Test. In: Wulfsberg JP, editor. WGP Congress 2015. Pfaffikon: TTP Inc; 2015, p. 347–354. [11] Kübler K. Presentation at Cluster Mechatronik & Automation: Automatisiertes Testen im Umfeld funktionskritischer Systeme 18.03.2018; Nuremburg; 2018. [12] International standard. Software and systems engineering - Software testing Part 1: Concepts and definitions. 1st ed;35.080(29119-1); 2013. [13] Zelenov SV, Zelenova SA. Generation of Positive and Negative Tests for Parsers. Program Comput Soft 2005;31(6):310–20. [14] Cauevic A, Punnekkat S, Sundmark D. Quality of Testing in Test Driven Development. In: 2012 Eighth International Conference on the Quality of Information and Communications Technology. 2012, p. 266–271. [15] Takagi T, Arao T. Overview of a place/transition net-based mutation testing framework to obtain test cases effective for concurrent software. In: 2015 IEEE/ACIS 16th SNPD.2015, p. 1–3. [16] International standard. Software and systems engineering - Software testing Part 4: Test techniques. 1st ed;35.080(29119-4); 2015. [17] Causevic A, Shukla R, Punnekkat S, Sundmark D. Effects of Negative Testing on TDD: An Industrial Experiment. In: van der Aalst W et al., editors. Agile Processes in Software Engineering and Extreme Programming. Springer; 2013, p. 91–105. [18] Sommerville I. Software engineering. 9th ed. Boston MA: AddisonWesley; 2011. [19] Voas J. Fault injection for the masses. Computer 1997;30(12):129–30. [20] Svenningsson R, Vinter J, Eriksson H, Törngren M. MODIFI: A MODelImplemented Fault Injection Tool. In: Hutchison D et al., editors. Computer Safety, Reliability, and Security. Springer Berlin Heidelberg; 2010, p. 210–222. [21] Svenningsson R, Eriksson H, Vinter J, Törngren M. Model-Implemented Fault Injection for Hardware Fault Simulation. In: 2010 Workshop on Model-Driven Engineering, Verification, and Validation.2010, p. 31–36. [22] Rösch S, Tikhonov D, Schütz D, Vogel-Heuser B. Model-based testing of PLC software: Test of plants' reliability by using fault injection on component level. IFAC Proceedings Volumes 2014;47(3):3509–15. [23] Rajabpour N, Sedaghat Y. A hybrid-based error detection technique for PLC-based Industrial Control Systems. In: 2015 IEEE 20th Conference on Emerging Technologies & Factory Automation (ETFA).2015, p. 1–7. [24] Wiebe F, Rösch S, Rehberger S, Vogel-Heuser B. Automated test suite generation to test modular designed packaging machines using Fault Injection and a simulink-based simulation approach. In: 2016 IEEE CASE; p. 758–765. [25] TUM, Lehrstuhl für Rechnertechnik und Rechnerorganisation. IMoMeSA Abschlussbericht. München; 2015.