Introspection mechanisms for runtime verification in a system-level design environment

ARTICLE IN PRESS Microelectronics Journal 40 (2009) 1124– 1134

Contents lists available at ScienceDirect

Microelectronics Journal journal homepage: www.elsevier.com/locate/mejo

Introspection mechanisms for runtime verification in a system-level design environment M. Metzger a, A. Anane a,, F. Rousseau b, J. Vachon a, E.M. Aboulhamid a a b

´al, Canada Universite´ de Montre Laboratoire TIMA, France

a r t i c l e in f o

a b s t r a c t

Article history: Received 28 March 2007 Received in revised form 26 March 2008 Accepted 15 April 2008 Available online 13 June 2008

A new generation of computer-aided design (CAD) tools is mandatory to cope with the growing complexity of System-On-Chip. We believe that they should be built on top of a modern and standard framework. ESys.NET is a design environment based on the .NET Framework. It takes advantage of advanced programming features which facilitates the integration of external tools. This paper presents a runtime verification tool for ESys.NET. Introspection ability is emphasized together with its capabilities to cooperate with third party tools. Introspection is used to retrieve the state of the model during simulation and to check a set of user defined properties. Neither the model nor the simulator is modified by the verification process. Experimentations on an AMBA bus model highlight the effectiveness of this approach. & 2008 Elsevier Ltd. All rights reserved.

Keywords: Net introspection Runtime verification Simulation System-level design ESys.NET SystemC

1. Introduction The complexity of today’s electronic systems necessitates of a new electronic design automation (EDA) tools generation. High levels of abstraction and efficient verification techniques are the key features of modern design platforms. Although traditional hardware description language (e.g. VHDL and Verilog) are well suited to address hardware synthesis, their low-level of abstraction is often highly discouraging. SystemC [1] and SystemVerilog [2] constitute promising evolutions of traditional development environments by providing higher level modeling and verification constructs taken from object-oriented programming and assertion-based verification. SystemC supports hardware/software co-design by extending C++ with system-level modeling constructs. However, C++ may not be the best for this task, mainly because of its lack of introspection capabilities [5]. Moreover, it is recommended [6] that modeling and simulation environments should be built inside larger standard software platforms like Java [3] or .NET [4]. Advanced programming features like introspection can greatly facilitate the development of such environments as well as their cooperation with external tools [5]. Introspection is the ability of a program to  Corresponding author.

E-mail addresses: [email protected] (M. Metzger), [email protected] (A. Anane), [email protected] (F. Rousseau), [email protected] (J. Vachon), [email protected] (E.M. Aboulhamid). 0026-2692/$ - see front matter & 2008 Elsevier Ltd. All rights reserved. doi:10.1016/j.mejo.2008.04.010

provide information about its own structure during runtime. Automated introspection is possible thanks to reflection mechanisms provided by programming environments [5]. ESys.NET [6] is an open-source system-level modeling, verification and simulation environment based on the .NET framework which is standardized by ISO. It can be seen as an evolution of SystemC taking advantages of .NET capabilities. ESys.NET hence enables introspection, multi-lingual model definition, remote processing and refinement of models. This paper, an extended version of Metzger et al. [20], demonstrates how the conception of a verification tool is facilitated by ESys.NET’s architecture and its inherited .NET’s introspection capabilities. This tool uses a runtime verification approach based on observers. The designer expresses the desired behavior of the model to be verified by specifying a set of properties formulated using the linear temporal logic (LTL). These properties are transformed into observers which watch the model during simulation and make sure that it conforms to the corresponding properties. It is important to mention that neither the model nor the simulator is modified by the verification process. In fact the source code of the system model is not needed to perform verification since introspection mechanisms apply on standard intermediate format code to retrieve all the required information. A similar tool built for SystemC would necessitate an additional layer to provide introspection [5]. One of the main contributions of this work is to show how separation of concerns can help in developing collaborative tools

ARTICLE IN PRESS M. Metzger et al. / Microelectronics Journal 40 (2009) 1124–1134

revolving around a model. Introspection is used instead of tedious parsing in order to explore a model and register the threads and classes pertinent to a simulator; it can also help for the construction of a set of observers which constitute a runtime verifier. The verifier, the model and the simulator remain separate from each other and we can envision multiple simulators and multiple verifiers working together around the same model, each one taking into account one aspect of the model. This will be illustrated using one specific simulator belonging to the ESys.NET environment. The paper is organized as follows. Section 2 introduces the verification environment based on ESys.NET. Section 3 compares our approach to SystemC. Section 4 details the implementation of the verification process and tools. Section 5 presents the case study and the performances analysis. Finally, Section 6 concludes this work.

2. ESys.NET verification flow 2.1. The .NET framework and ESys.NET The .NET Framework is a modern environment created to ease the development and the deployment of distributed applications [4]. The core of .NET standardizes the development and execution of applications. It includes:

(simulator, verifier, etc.), has to provide either hook points or callback functions to allow for collaboration. It is the role of the integrator to instantiate the different models and tools required to run a simulation/verification task. Note that frontiers are well established between the different concerns, which are modeling simulation and verification. This separation increases potential for reuse, extension and collaboration. In the following we will present different tools that were built around this approach including a simulator and a runtime verification tool. More precisely, the focus of this paper is on the runtime verification tool. 2.3. ESys.NET modeling Built on top of the .NET Framework, ESys.NET is a system-level modeling, verification and simulation environment which therefore benefits from the above facilities. Systems are modeled by a hierarchy of modules communicating via communication channels [6]. The behavior of a module is defined by processes. In terms of programming, ESys.NET is a class library. The following classes constitute the core of ESys.NET:

 BaseModule: The base class of all user modules. It is also the

 A Common Intermediate Language (CIL), which is an instruc-



tion set for a virtual machine. This enables interoperability across multiple programming languages (C#, VB.NET, managed C++, etc.), and portability. Metadata definitions, which are additional information about the program embedded into the CIL code. Metadata includes description of types and attributes. The description of types provides information about the structure of the application such as class hierarchy, interface implementation and members (methods, fields, etc.). Attributes, also supported by .NET, are used to annotate the data structures including user defined attributes. Metadata can be consulted using the .NET’s reflection API.

1125





base class of most of the model components (signals and channels). During model elaboration, all subcomponents of a module are registered by the simulator, using the .NET reflection mechanism. BaseSignal: This is an abstract class that models the transmission of a single data on a bus. ESys.NET provides a set of specialized signals for common data types (integers, strings, characters, floating-point numbers, etc.). Designers can implement signals for their own data types by extending BaseSignal. Event: It is used to synchronize processes. For instance, BaseSignal owns a transaction event and a sensitive event. The first one is triggered each time a process writes the signal. The second one is only triggered when the new value is different from the previous one.

2.4. ESys.NET scheduler 2.2. ESys.NET motivation As stated in the Introduction, one of the main objectives of ESys.NET is the separation of concerns. Models are written in a .NET language such as C# which allows metadata (including attributes) to be assigned to classes, methods and objects. The modeler has to be aware of these attributes and of the semantics he associates to them. The model is completely independent from the tools. By contrast, SystemC’s simulator is completely interweaved with the model through the macros that are embedded in the model. In an ESys.NET model we do not have to be aware whether one simulator or many simulators will need to collaborate during the simulation of the model. To elaborate runtime verification observers, one only needs the model to be explored by introspection in order to construct the different observers. Each observer is triggered by an event which may be different for each observer. The simulator only needs to be aware of a subset of the attributes included in the model in order to simulate an aspect of the model (we can imagine a subset of attributes describing some software part of the system, while the other subset describes one of its hardware part at some given level of abstraction). Each tool

Similar to SystemC, the ESys.NET scheduler is based on a discrete event execution model where concurrent processes interact together by event notifications [6]. Fig. 1 shows the different phases of the ESys.NET scheduler. During the Initialization phase all the processes tagged with a special attribute are executed. Next, the scheduler begins the Evaluation phase by executing each process whose sensitive events are notified. Due to immediate notification [6], other processes can become ready to run and executed in this phase. When all the processes are executed, the ESys.NET scheduler executes the Update phase. The interest of the Update phase is to delay the modification of states of channels, especially signals, until the end of the Evaluation phase. The sequential execution of the Evaluation and Update phase forms a delta cycle. So, at the end of a delta cycle, the scheduler checks if there are new processes whose sensitive events are notified. If so it begins a new delta cycle. Otherwise the scheduler advances the time to the next timed notification and thus begin a new discrete simulation cycle. To indicate that a particular method must be considered as a process by the simulator, the designer tags it with attributes. Attributes indicate the process’s type and the events it is sensitive to. Attributes can also be used by designers to request the

ARTICLE IN PRESS 1126

M. Metzger et al. / Microelectronics Journal 40 (2009) 1124–1134

Initialization

Immediate Notification Cycle Evaluation

Delta Cycle Discrete Time Cycle

Update

Advance Time

Fig. 1. ESys.NET scheduler execution phases.

execution of a method at given points in a simulation (e.g. beginning of simulation, end of a discrete simulation cycle, etc.). External tools can also be plugged in the simulator by registering callback methods. ESys.NET offers a set of hook-points where callbacks can be hooked. Using callbacks is one way to implement the observer paradigm. For instance, the observers of the AHB-Lite model (Section 5.1) require to be notified at the end of each discrete simulation cycle. For this purpose a callback method will be hooked at this point and called each time the simulator reaches the end of the discrete simulation cycle. Each observer, independently of the others can be hooked either at the end of the initialization phase, or at each iteration of the different cycles (immediate, delta or discrete) or at the complete end of simulation. The model can thus be observed flexibility according to different time precision levels allowing for various abstraction needs. For example, an ‘‘untimed’’ TLM model may only use immediate and delta cycles, thus, for example, allowing observers to hook only at delta cycles in this case. Each observer is made sensitive to a single event of the model; this event may be different from one observer to the other. Therefore, each observer will have a consistent NEXT operation, related to the triggering of this single event. In other words, the state transition of the automaton corresponding to an observer will take place only when the sensitive event has been triggered. Triggers are verified according to the specified time granularity, that is at the end of immediate, delta or discrete cycles. By opposition, in the RTL verification paradigm, time granularity and triggering events are superposed notions which both correspond to the edge of a clock. Therefore our approach can be seen as different from the RTL verification process in which all the observers are triggered by the same clock event.

2.5. Overview of the ESys.NET verification flow The verification layer we propose for Esys.Net is based on the observer paradigm. The state of the model under simulation is

verified by the verification engine which observes the runtime evolution of the system and checks it against a set of formal properties: the observers. Fig. 2 shows the links between simulation and verification flows. Two simulation flows are pictured. The first one (A) is the actual ESys.NET simulation process. On the right side, a hypothetical simulation flow (C) based on SystemC is presented. The middle lane (B) describes our verification process. As illustrated, the simulation flow of Esys.Net (A) remains independent of the verification process (B). The properties to be verified are expressed using LTL [12] (see Section 4.2). LTL formulae are stored in a text file apart from the model. Each formula is then transformed into an observer automaton to be later executed in parallel with the simulation. For each observer, an event of the system model is used to synchronize the system model under simulation with the observer automaton. When the given event occurs, the observer automaton executes a transition whose labels match the new current state of the model. If an automaton has no such transition, the validation is said to fail or succeed depending on the type of the automaton. As later explained in Section 4.2, execution traces observed by such automata will either be validating, violating or non-conclusive for the LTL property being checked.

2.6. ESys.NET simulation flow An ESys.NET system model can be written in any language supported by .NET such as C#, J#, managed C++, etc. Note that over 20 languages are supported. The model is compiled into CIL code by existing compilers. CIL code is used during the transformation of properties to perform syntax and type checking (operation 1a on Fig. 2) to ensure, for example, that a specific signal exists in the model. Prior to simulation, the class describing the model is instantiated and elaborated by the ESys.NET environment. The resulting executable model is browsed (Fig. 2-2a) to retrieve data

ARTICLE IN PRESS M. Metzger et al. / Microelectronics Journal 40 (2009) 1124–1134

1127

B

C

Verification Flow

SystemC Simulation Flow

Esys.NET System Model C# source

Properties Editor

Internal representation generation

A ESys.NET Simulation Flow

SystemC Model C++ Source

Compilation

C++ Parser

C++ Compiler

Abstract Syntax Tree

SystemC Compiled Model

Properties File Esys.NET System Model CIL code & metadata

Intro

s pe c

1a tion

1b Transformation into automata

Me

od th

all sC

Instanciation And Elaboration

Binding phase

Model Instantiation And Elaboration

Executable System Model

Link

2a Introspection

Observers Internal Representation

3b Methods Call

Execution Phase

Esys.NET Simulator

Callbacks

Executable System Model

Internal Representation

Meth

3a

2b Binary

od s

Call 4b

Verification Engine

Modified SystemC Simulator & Verification Engine

SystemC Reflection Provider

Fig. 2. The simulation and verifications flows.

that will be used during simulation to evaluate the state of the model. The callback methods registered within the ESys.NET simulator are called when the verification engine applies the observers on the simulated model (Fig. 2-3a).

3. Comparison with SystemC

verification since the internal behavior of the system is not taken into consideration. Nonetheless, SCV uses introspection on input data to allow for arbitrary data types. The strategy exploits template mechanisms to extend data types with introspection capabilities. The drawback is that introspection support must be described manually for user defined types.

3.1. Adding reflection to SystemC

3.2. SystemC verification

Other frameworks such as SystemC are more difficult to interact with. The main drawback of SystemC is its lack of introspection. This is mainly due to the fact that SystemC is based on C++ which has limited introspection capabilities [5]. Additional libraries to support reflection are needed. In SystemCXML [8], Doxygen’s parser is used to extract an abstract system-level description (ASLD) and to capture it in a XML file. The PINAPA project [9] adds a patch to the GNU C++ compiler and to SystemC. The compiler is used to get the abstract syntax tree (AST) of SystemC processes. Internal SystemC structures resulting from the elaboration phase are exported to extract the structure of models. Nevertheless, SystemC’s core library allows some basic introspection thanks to the get_child_objects method [1], which returns all subcomponents of a given element in a design. However, this approach only permits to retrieve SystemC objects (modules, signals, clocks, etc.); internal variables are thus still hidden. Besides, the design of SystemC’s simulator makes it difficult to integrate external tools. The modification of the simulator source code is often the only solution [8]. Basic callbacks are available inside the design but are not usable from external libraries. The SystemC verification (SCV) standard provides an API to develop test benches at the transaction level [10]. Contrary to the approach we present in this paper, SCV implements black-box

Fig. 2 (lanes B and C) presents a possible simulation/ verification flow to emphasis the usefulness of introspection and hook-points. The technique used to add introspection to SystemC is inspired by the one proposed by PINAPA. The main drawback of this approach is the lack of introspection which prevents the binary form of a SystemC model to be exploited. Indeed a lot of information is lost during the compilation process. The binary format resulting from the compilation highly depends on the compiler used and the architecture of the computer it targets. Developing a tool to directly explore the object file would be tedious. Thus, it is almost mandatory to use the source code files to get introspection information. This implies either to write a complete C++ parser or to use an existing one. The PINAPA project uses the C++ parser of gcc to build the AST. It can be used to get information about the structure of the model (Fig. 2-1b). After the instantiation and the elaboration of the model, the executable model and the AST need to be linked, i.e. we need to link every entry in the AST (that can be introspected) to its actual instance. This is crucial to be able to recover the state of the model during simulation. To do so, one can use the information about binary structure of classes generated by the compiler (Fig. 2-2b). The result of this process can be used to retrieve the state of the model for verification purposes (Fig. 2-3b). Besides the additional introspection layer, the SystemC simulator needs to be modified to allow interaction with third

ARTICLE IN PRESS 1128

M. Metzger et al. / Microelectronics Journal 40 (2009) 1124–1134

party tools. The callbacks provided by SystemC 2.1 are virtual methods defined in system model components (modules, ports, channels, etc.). Therefore, callbacks are accessible only from the model and not from an external environment. There are several solutions to get around this limitation. One is to add an artificial module to export the callbacks. One could also add hook-points to the SystemC simulator, similar to the ones found in ESys.NET. Finally the verification engine could be merged with the simulator (Fig. 2-4b). As we can see, significant work is required to provide introspection and extend SystemC. On the opposite, developing a verification tool with ESys.NET is facilitated by introspection and hook-points which are parts of the core libraries.

4. Implementation of the verification layer 4.1. Construction of the design structure tree The structure of the model is needed for the construction of the observers. This information is available in the type structure that defines the model. It corresponds to the operation (b) of Fig. 5. To avoid redundant introspection operations, we build a design structure tree which represents the modules hierarchy and contains the signals, the events and the observable variables of the model. This structure is quite similar to the ASLD of Patel et al. [8]. Compared to PINAPA, this can be seen as a higher level abstraction structure where only relevant information is kept (i.e. information used for hierarchy exploration and evaluation of the model’s state during simulation). The pseudo-code below shows the basic structure of the exploration algorithm used to build the tree. Note that this pseudo-code is very close to the actual source code. 01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 16 17 18 19

object BuildTree(Type moduleType){ foreach(FieldInfo f in moduleType.GetFields()){ if(f.FieldType ¼ ¼ typeof(Event)) AddEvent(f.FieldType) else if(f.FieldType ¼ ¼ typeof(Clock)) AddClock(f.FieldName); else if(f.FieldType ¼ ¼ typeof(BaseSignal)) AddSignal(f.FieldName,f.FieldType); else if(f.FieldType ¼ ¼ typeof(BaseModule)) { AddModule(f.FieldName); BuildTree(f.FieldType); }else{ AddVariable(f.FieldName,f.FieldType); } } } BuildTree(typeof(MyTopLevel));

The core of the algorithm is a loop which iterates through all of the subcomponents of a module (line 2). A Type object contains the information offered by .NET about a specific type. The GetFields() method returns a collection of objects describing all the instance variables of a given type. Depending on the type of the field, a node is added to the abstract tree. Event and Clock objects are simply added to the tree (lines 4–7). For signals, the type of the signal must be kept since it defines the data type carried by the signal. It will be used to perform type checking. If a field is a module extending the BaseModule, it is recursively explored (line 13). Other variables are simply registered in the tree along with their type. Line 19 is the initial call of the method on the root of the model defined by class MyTopLevel.

The tree construction is greatly facilitated by the use of reflection. It does not require the tedious implementation of a parser. Indeed the source file is not needed at all. All the information is extracted from the compiled version of the model, no matter which language is used, as long as it is supported by .NET (Fig. 6 shows the representation of this tree within our verification tool). 4.2. Building observers 4.2.1. Observer background Observers are built from LTL formulae. LTL and Boolean logic are the base layers of Property Specification Language (PSL) [11]. Our tool can be seen as a first step towards the integration of a complete PSL within a CAD platform. LTL was originally designed to express complex requirements on infinite execution paths [12,15]. LTL allows designers to formalize the temporal behavior of the model. It is composed of atomic properties, Boolean operators (and, or, not, imply) and temporal operators (Next, Always and Eventually). The semantics of temporal operators over an infinite execution path s0s1s2y is:

 The Next operator is noted X: X j holds over path s0s1s2y if j holds for s1s2y.

 The Always operator is noted G: Gj holds if j holds in each sate of the execution path.

 The Eventually operator is noted F: F j holds if j holds in some future state of the path.

 The Until operator is noted U: j U c holds if j holds until c holds over the path. As an example, the following property specifies that a request must occur and be followed by an acknowledgement some time after: F ðsigðreqÞ ¼¼ true && X F sigðackÞ ¼¼ trueÞ where sig(req) ¼ ¼ true and sig(ack) ¼ ¼ true are atomic properties. They are evaluated during simulation and reflect the state of the system model. While in formal verification an LTL property is either valid or not, in runtime verification we cannot guarantee the validity of the property since we are limited by the observation of some finite execution paths (i.e. finite traces). As an example, it is impossible to validate or violate property G F p with a single partial simulation. To better cope with these limitations, many semantics were proposed which adapt the interpretation of LTL properties for runtime verification [7,12,13,19]. In our implementation we consider a three-valued semantics equivalent to the strong semantics of LTL formulae over finite executions traces described in [13]. Given a finite execution trace w and an LTL property F, an observer ends with one of the three following output values: 1. w is validating: If all infinite extensions of w satisfy property F. We also say that w is a ‘‘good prefix’’ for property F. 2. w is violating: If all infinite extensions of w violate property F. We say that w is a ‘‘bad prefix’’ for property F. 3. w is non-conclusive: This value is output by an observer that cannot conclude if w is violating or validating. This may happen, for example, when we try to validate a liveness property such as G F p which admits no good prefix since any finite execution could possibly be prolonged with (p)o which violates the property. It may also happen for safety properties such as p-(X q-X r) if the simulation is inadvertently stopped at the second state after recognizing p-X q. In this case we cannot predict if during the third state the atomic

ARTICLE IN PRESS M. Metzger et al. / Microelectronics Journal 40 (2009) 1124–1134

property q will be true or false. Consequently, we cannot conclude on the validity of the property and the result is thus non-conclusive. When monitoring a property over a simulation, (1) any violation (or validation) should be detected as soon as possible and (2) simulation monitoring should be stopped as soon as we are sure that pursuing the observation any further would not change the current result. Our tool implementation is partly based on the work of d’ Amorim and Rosu [19], which describes how to build a monitoring automaton M jviolating recognizing the minimal bad prefixed of an LTL formula j. Automaton M jviolating is constructed from a Bu¨chi automaton Aj of j. They show how Aj can be reduced and transformed to recognize violating finite traces for j (minimal bad prefixes), thus allowing for the specific needs of runtime verification. At each step of a simulation, all enabled transitions in the observer automaton Mjviolating are executed and new states are updated. If no transition is possible, this implicitly means the observer evolves to a ‘‘violate’’ state, and that a bad prefix for j has just been detected. The recognition of validating finite traces is also possible. In that case, we must be able to recognize the good prefixes of j, i.e. the bad prefixes of j: For this we simply need to construct an observer automaton M jviolating . As an example, Fig. 3 shows the observer automaton Mjviolating that recognizes all the good prefixes of the LTL property j (i.e. all the bad prefixes of j): F(sig(req) ¼ ¼ true&& X F sig(ack) ¼ ¼ true). Mjviolating is used to validate the property j. Since liveness properties cannot be violated by finite traces, it would have been useless to try violating j with an observer automaton M jviolating . The validation of property j is executed as follows. Starting from the Init state, we consider the execution trace {sig(req) ¼ ¼ true, sig(ack) ¼ ¼ false}{sig(req) ¼ ¼ false, sig(ack) ¼ ¼ true}. In this case, the sequence of states maintained by the observer automaton Mjviolating during the simulation is {init}{1}{}. The observer ends with {}, i.e. the implicit violate state. This means a bad prefix for j has been recognized and that the execution trace therefore validates j. However, for the execution trace {sig(req) ¼ ¼ false, sig(ack) ¼ ¼ true}{sig(req) ¼ ¼ true, sig(ack) ¼ ¼ true}, the sequence of states maintained by the observer during the simulation is {init}{init,1}{1}. This execution does not allow any since the observer ends in state 1 without having recognized any bad prefixes for j. In this case the output of the observer is said to be nonconclusive. 4.2.2. Implementation considerations The building of observers requires the construction of atomic properties followed by the transformation of LTL formulae into finite state automata (operation (c) of Fig. 5). The first part consists in building the atomic properties composing the LTL properties to verify. These atomic properties are comparisons between two objects like sig(req) ! ¼ sig(ack), or between an object and a constant value, like

Sig(req) == false

Sig(ack) == false

true Init

1

Fig. 3. An example of automaton M jviolating to validate j.

1129

sig(req) ¼ ¼ true. The first step is to build the two operands. We use the abstract tree presented in Section 4.1 to check that references to model objects are correct (e.g. the req signal exists) and to retrieve the type of objects. If the right operand is a constant, the type must be resolved according to the type of the left operand. The obvious method to build an object from a string is to call a constructor with a string parameter, or to look for a static method Parse which takes a string argument and returns an object. When the two operands are built, the whole atomic property can be constructed. Two checks are required. First the two operands must be of the same type. Then one need to make sure that the comparison operation can be performed, which means that the CompareTo or Equals methods must be implemented. This mechanism allows to implicitly support all .NET primitive data types (integers, Booleans, strings, floating-point numbers, etc.), as well as a wide range of other types (DateTime, IPAddress, Enumerated types, etc.). Custom types can also be used in the verification process as long as the type complies with the object construction procedure and the comparison methods as previously given. The second part is achieved by executing the LTL2BA algorithm presented in [14] and the transformations presented in [19]. LTL2BA is a tool written in unmanaged code C. Unlike managed code, an unmanaged code is compiled directly to the machine language bypassing the common language runtime (CLR). However, .NET framework facilitates the interoperability between unmanaged and managed code. For this, we developed a managed C++ interface which can communicate easily with an unmanaged code like C. The role of this interface is to read an LTL property in the Unicode format and to pass it to LTL2BA in ASCII format. Then, it converts the unmanaged data structure, corresponding to the Bu¨chi automaton generated by LTL2BA, to some C# data structure. Fig. 4 shows the interaction between the Verification Environment and the LTL2BA tool.

4.3. Binding observers to the model While the observers are built, the model is instantiated and elaborated by the ESys.NET kernel. Atomic properties need to be connected to the elaborated model before the simulation can start (see operation (d) of Fig. 5). For now, properties operands are described by paths localizing objects within the model; but they are not yet bound to values of a specific model instance. The current reference to the object is required to get the value of the operands and thus evaluate the property. Once again introspection is used; given a reference to an object and the name of one of its field, .NET offers a mechanism to obtain the value of the field. Since a reference to the top-level and the path of the operands are available, a reference to the object pointed by the path can be obtained.

4.4. Binding observers to the simulator As explained in Section 2.4, each observer is synchronized by a predefined event of the model. This synchronization event defines the execution step of the automaton and a basis for the sampling of tested variables and signals. Thus the observer has to be notified each time his synchronization event is been triggered. The ESys.NET simulator offers a method to bind a procedure to some given event in the model. This has the great advantage to extend the model dynamically without the need to modify the source code (see operation (e) of Fig. 5). To do so, a reference to the event object is needed. For this, the introspection approach

ARTICLE IN PRESS 1130

M. Metzger et al. / Microelectronics Journal 40 (2009) 1124–1134

LTL Property (Unicode)

managed data structures

LTL Property (ASCII)

Unmanaged data structures

Fig. 4. Interaction with LTL2BA.

presented in Section 4.3 is used. Once a reference to the event is retrieved, a method is bound to it as follows: Simulator.BindMethod( new RunnableMethod(FlagObserver), eventObject, methodID);

where FlagObserver is the name of the method to bind to the event eventObject and methodID is a unique string used to identify the link between the event an d the method. When the synchronization event occurs the observer is flagged to be executed. It is worth noting that the occurrence of the sensitive event does not imperatively compel the immediate execution of the corresponding automaton. In fact, the evaluation of atomic properties and the execution of automaton transitions can be delayed to some further point in the simulation, that is the end of an immediate, delta or discrete cycle. For instance, in the AHB-Lite model presented in Section 5.1, the state of the model during a discrete simulation cycle (i.e. clock cycle) is difficult to evaluate since it highly depends on the scheduling of processes. One therefore needs to wait until the end of the discrete cycle when all the signals are stabilized [6]. For this purpose, the ESys.NET simulator provides a cycleEnd hook point triggered when the model reaches the end of discrete simulation cycle. The following statement is used to bind a method to this hook point: simulator.cycleEnd + ¼ new RunnableMethod(UpdateAutomata);

The UpdateAutomata method will run all flagged observers whose sensitive events have been triggered.

Moreover, the editor offers a graphical tree representation of the system model, using the information collected in the Design Structure Tree (Section 4.1). LTL properties are stored in a text file. This text file is read by the verification application. This verification application provides a graphical user interface to monitor the ESys.NET simulation engine and the verification engine (see Fig. 7). It displays the list of observers and their associated automata. A waveform viewer is also implemented to display the evolution of signal values during simulation. According to the verifier’s perspective, a sole task needs to be done, that is the specification of the desired properties to verify against the model. To do this, the verifier can rely on the editor application to explore the model design tree and to edit the LTL formulae. Thereafter, all the verification steps are executed automatically. The verifier only needs to specify the name of the LTL file already generated by the editor, to set the simulation duration and to launch the simulation. If a property is invalidated, the model can be debugged by examining the violating execution and visualizing the different transitions performed by the automaton.

5. Experimentation The verification layer implementation has been validated on a case-study describing an AMBA bus model. The case study aimed to validate the verification flow proposed, find its limitations and show the usefulness of this type of verification. An evaluation of performances was also performed to identify bottlenecks in the verification engine. The properties that were checked are the ones that a well-known EDA company previously used to validate its own AMBA bus model.

4.5. Evaluation of properties

5.1. The AHB-Lite model

To perform a step in the observer automaton, the properties of all outgoing transitions from the current states are evaluated. This evaluation consists in retrieving the operand values from the model and calling the Equals or CompareTo method. It is important to mention that a limited number of introspection operations are executed at this point in order to minimize the overhead.

Our case study is based on a specification from ARM [17] for the AMBA on-chip bus. The specification contains the description of a high performance bus called AHB. For this first experiment, a light version of this bus, called AHB-Lite [18], was considered: it implements a single master and does not support split transactions. The model implements burst transfers, single clock edge operation and supports data width from 8 to 1024 bits. Additional features implemented by the full AHB bus concern the support of split transactions, the handover to the master in a single clock cycle and the support of up to 16 masters. The model was optimized to decrease simulation time. The model contains a master process, a decoder process and as many as 16 slave processes (Fig. 8).

4.6. Tools The verification layer is made of two distinct applications: an advanced editor to specify properties and a verification application. The editor provides high-level facilities such as syntax highlighting; auto-indentation and auto-completion (see Fig. 6). As the formalization of complex LTL properties can be difficult, a pattern instantiation mechanism is provided to ease the specification of LTL properties. Introduced in [16], patterns are LTL formula templates corresponding to typical properties that often occur in formal specifications. Defining properties by instantiating these patterns reduces the risk of errors in the specification process.

5.2. Verification process The system model was first debugged using traditional in-code assertions and trace analysis. Thus, minor and obvious bugs were fixed. Then a set of properties written in English were formalized

ARTICLE IN PRESS M. Metzger et al. / Microelectronics Journal 40 (2009) 1124–1134

1131

EsSys.NET Verification Engine

LTL Editor Properties Editor

EsSys.NET Simulator Esys.NET System Model CIL code & metadata

Introspection

LTL Edition

a Properties File

Int

Parse Model b

Design Structure Tree

Construction of Atomic Properties

c

LTL2BA Construction of Observer Automata

Model Instantiation And Elaboration d Binding Observers to Model

Introspection

Executable System Model

Register FlagObserver Method to be sensible to each Observer Event

Evaluations of Properties

e

f

Debugging

Building Observers to Simulator

Binding Observers to Model

Building Observers

Design Structure Tree

Build Design Tree

ros pe cti

on

ANTLR Parser

g

Bind the call back Method Update Automata at the end of simulation cycle (delta or discrete)

Evaluation of Properties

Esys.NET Simulator

Callbacks

Run Simulation

Valid, invalid or non conclusive

View Waveform and Debug Automaton

Fig. 5. Detailed verification flow.

in LTL using the editor presented in Section 4.6. The following example shows the textual description of a property prop2 together with the LTL formula defining it. ‘‘In a write transfer, after the address phase of the transfer is sampled, the master should provide valid data in the next immediate cycle’’, is translated into: ltl_observer prop2(clk.event(posedge)){ G (((sig(htrans) ¼ ¼ "NSEQ" ||sig(htrans) ¼ ¼ "SEQ")

&&sig(hwrite) ¼ ¼ true &&sig(hready) ¼ ¼ true) X(sig(hwdata) ¼ ¼ "DATA")) }

The formula is made of two parts. The first one specifies that the model must be in a transfer phase. The htrans signal identifies the type of the frame currently transferred. It can be either not sequential or sequential, if the frame is the first of a burst sequence or not. The hwrite signal indicates that the master is

ARTICLE IN PRESS 1132

M. Metzger et al. / Microelectronics Journal 40 (2009) 1124–1134

Fig. 6. LTL editor.

Fig. 7. Verification application.

writing data to a slave and the hready signal is the answer of the slave to signify it is ready to transfer. The second part of the formula (X (sig(hwdata) ¼ ¼ ‘‘DATA’’) checks that a valid data is provided by the master. The two parts are connected by an implication operator. The always (‘‘G’’) operator is put in front of the formula to ensure that it is valid during the whole simulation. The final specification file contains 23 main properties and 19 additional properties to evaluate the functional coverage of the

simulation (e.g. each slave has been activated; all transaction types have been performed, etc.). The specification file is loaded by the simulation application, which initiates the transformation of each LTL formula into an automaton. Fig. 9 shows the automaton which recognizes all the bad prefixes related to the formula prop2 given above. Labels p0, p1, p2, p3 and p4 respectively denote the following properties: sig(htrans) ¼ ¼ "NSEQ", sig(htrans) ¼ ¼

ARTICLE IN PRESS M. Metzger et al. / Microelectronics Journal 40 (2009) 1124–1134

Slave #1

Decoder

select

1133

spection (i.e. introspection during the evaluation of properties). In this case, the overhead rate due to introspection increased to 90%.

addr & ctrl data in Master data out & response

addr & ctrl data out

Slave #16

data in & response

select Read data / Response mux

addr & ctrl data in data out & response

Fig. 8. High-level representation of an AHB-lite model.

!p0 && !p1

p4

true

!p3

!p2 && p4 Init

1

6. Discussions Concerning the model, the verification allowed us to find limit cases and synchronization bugs in our implementation. The hardest and longest part was not the construction of the observers but the construction of the model itself. Correction of bugs remains especially difficult. This case study was also a good opportunity to pinpoint weaknesses of the verification engine and find solutions to improve its performances. Among others, the formulation of complex behaviors with LTL is a difficulty frequently encountered in formal verification. However, the use of formula patterns [16] can alleviate this task. This is why our LTL formula editor proposes a set of such patterns to assist the designer. Furthermore, we plan to extend our work by implementing the IEEE standard for PSL [11] as a specification language. PSL specifies a LTL, the Foundation Language (FL). It extends the LTL logic by inheriting the LTL temporal operators and offering new powerful constructs such as regular expressions used to define finite linear temporal patterns.

7. Conclusion and future work

!p0 && !p1 && p4

!p2

!p3 && p4 Fig. 9. Automaton Mprop2 . violating

"SEQ", sig(hwrite) ¼ ¼ true, sig(hready) ¼ ¼ true and sig(hwdata) ¼ ¼ "DATA". Just before the simulation, automata are bound to the model and to the simulator. After the simulation, the user is informed about the validity of each formula.

5.3. Performances Performances analysis was achieved on the AHB-Lite model with 16 slaves and a set of 42 observers. A profiler was used to record execution time and memory allocation. The overhead due to observers’ execution was evaluated during simulation. Approximately 67% of the simulation time was dedicated to observers. A simulation of 20 000 clock cycles took approximately 1.6 s without observers (250 k events/s). Adding observers raised the execution time to 5 s. At first sight, the simulation time rate used by observers can seem quite important. The situation can, however, be explained by the important number of observers compared to the fairly low complexity of the model. Indeed, the time overhead is directly proportional to the number of observers (to be more precise it is proportional to the number of atomic properties). The case study implied many observers (i.e. LTL formulae to check) verifying a fairly simple model. Of course, verifying the same properties on a more complex model would lower the impact of the observers overhead. Furthermore the size of each formula tends not to exceed a certain size in practice. On the other hand, to evaluate advantages brought by introspection during the initialization phase (i.e. static introspection), the simulation was also performed using dynamic intro-

This paper presents a runtime verification method and tools for a flexible introspective system-level design environment. Our approach brought to light the steps of the verification process, which could benefit from introspection and therefore greatly simplify the development of modeling and simulation environments. Introspection is used to get the structure of the model and to retrieve data during simulation. ESys.NET could easily be extended thanks to hook-points used to synchronize the execution of observers and the model simulation. Performances were evaluated on a concrete case study. The overhead observed during simulation only depends on the number of observers. Performances should be improved by the next version of the .NET framework, especially by using generics classes for properties’ operands. Different execution models for temporal logics will also be explored, thus allowing transaction level verification. Parallelization of the verification engine and the simulation on different CPUs (as well as on separate hosts) is also envisaged. References [1] SystemC, Version 2.1, /http://www.systemc.org/S. [2] D.I. Rich, The evolution of SystemVerilog, IEEE Des. Test Comput. 20 (4) (2003). [3] Java Technology, /http://www.sun.com/java/S, SUN. [4] .NET Framework, /http://www.microsoft.com/netS, 2003. [5] F. Doucet, S. Shulka, R. Gupta, Introspection in system-level language frameworks: meta-level vs. integrated, 2003. [6] J. Lapalme, E.M. Aboulhamid, G. Nicolescu, A new efficient EDA tools design methodology, ACM Trans. Embedded Comput. Syst. 5(2)(2006)408–430 (available also at /http://esyssim.sourceforge.netS). [7] D. Giannakopoulou, K. Havelund, Automata-based verification of temporal properties on running programs, in: Annual International Conference on Automated Software Engineering, November 2001. [8] H.D. Patel, D.A. Mathaikutty, D. Berner, S.K. Shukla, SystemCXML: an extensible SystemC front end using XML, Formal Engineering Research using Methods, Abstractions and Transformations, Technical Report no. 2005–06. /http://systemcxml.sourceforge.net/S. [9] M. Moy, F.Maraninchi, L. Maillet-Contoz, PINAPA: the extraction tool for SystemC descriptions of systems-on-a-chip, in: Proceedings EMSOFT’05. [10] C. Norris Ip, S. Swan, A tutorial introduction on the new SystemC verification standard, /http://www.systemc.orgS.

ARTICLE IN PRESS 1134

M. Metzger et al. / Microelectronics Journal 40 (2009) 1124–1134

[11] IEEE Standard for Property Specification Language (PSL) Reference Manual. [12] A. Pnueli, The temporal logic of programs, in: Proceedings of the 18th IEEE Symposium on Foundation of Computer Science, 1977. [13] C. Eisner, D. Fisman, J. Havlicek, Y. Lustig, A. McIsaac, D. et Van Campenhout, Reasoning with temporal logic on truncated paths, in: International Workshop on Computer Aided Verification, Lecture Notes in Computer Science, vol. 2725, Springer, Berlin, 2003. [14] P. Gastin, D. Oddoux, Fast LTL to Bu¨chi Automata Translation, in: Conference on Computer Aided Verification, July 2001, pp. 53–65. [15] P. Wolper, M.Y. Vardi, A.P. Sistla, Reasoning about infinite computation paths, IEEE Symposium on the Foundations of Computer Science, 1983, pp. 185–194.

[16] M.B. Dwyer, G.S. Avrunin, J.C. Corbett, Property specification patterns for finite-state verification, in: The Second Workshop on Formal Methods in Software Practice, March 1998. [17] ARM company /http://www.arm.com/S. [18] AHB-Lite /http://www.arm.com/miscPDFs/1744.pdfS. [19] M. d’Amorim, G. Rosu, Efficient monitoring of O-languages, in: The 17th International Conference on Computer Aided Verification (CAV’05), Edinburgh, Scotland, UK, 9 July 2005. [20] M. Metzger, F. Bastien, F. Rousseau, J. Vachon, E.M. Aboulhamid, Introspection mechanisms for semi-formal verification in a system-level design environment, in: Proceedings of the 17th IEEE International Workshop on Rapid System Prototyping (RSP 2006), Chania, Greece, 14–16 June, 2006, pp. 169–178.