- Email: [email protected]
Control Environment Implementation Issues - With Case Studies
Copyright © IFAC Algorithms and Architectures for Real- Time ControL SeouL Korea, 1992
. REAL - TIME CONTROL APPUCA nONS
Control Environment Implementation Issues - with Case Studies Robert E. Betz*, Peter J. Moylan*, Len J. Sciacca**
* Department of Electrical and Computer Engineering, University of Newcastle, Callaghan , NSW ,2308, Australia * Department of Electrical Engineering , University of M elbourne, Parkville, V ie, 3052, Australia
Abstract: This paper discusses a number of practical issues related to the software environment required for the successful implementation of medium to large scale control systems. It is argued that for all but trival systems it is desirable to implement the control system within a software environment which formally supports real-time multi-tasking. Keywords; Real-time computer systems, Operating Systems
Introduction
important. In some system consequential suppression is also supported. (e) Task support. This code provides the underlying multi-tasking support for the system. (t) Communications support. This code provides the software interface to serial channels, networks etc. that the system must communicate with. (g) Plant interface modules. This code provides the software interfaces to the hardware devices which directly interact with the environment. Examples are NO converters, relay control logic and other more specialised I/O devices.
The development of a computer control system involves a number of aspects: the design of a control algorithm, the selection of suitable hardware, the software design and implementation, the development of actuator and sensor mechanisms, and so on. Looking at the control theory literature, one can get the impression that the algorithm design is the biggest part of the job, or at least the most intellectually demanding. In practice, one tends to find that the software development phase is where most of the development time goes; and that the complexity of the work is such that it cannot be trusted to "average" computer programmers.
The form of the underlying concurrent task support is often an area of confusion. A Computer Scientist for example assumes that a real-time system is any system that has to react to external asynchronous events. However there needs to be a differentiation between, say an airline booking system, and a aircraft control system. An airline booking ystem is an example of a soft real-time system, because the consequences of slower response are not significant. An aircraft control system is an example of a hard real-time system. If timing constraints are not met all the time then the control system will not achieve acceptable performance, or catastrophic failure may result. Soft real-time applications can usually be supported with conventional operating systems, whereas hard real-time applications require special operating system kernels which can offer deterministic task response times.
Typically, the control algorithm itself can, even if it is a complex algorithm, be expressed in just a few pages of straightforward code in a high level language. The total software package, in contrast, is likely to be hundreds of pages long; and its design involves some complex and often subtle problems. The majority of the code in most medium to large control systems is related to functions such as:(a) Operator interface - usually a large section of code. Important to the user as it is the means of controlling and monitoring the system . (b) Logical decisions. Usually surrounds the controller code. Governs control transitions such as the change from linear to slewing mode. (c) Sequencing of events. Usually a large and complex section of code. Contains logical decisions which also contain timing constraints. For example, timed sequences are required to start and stop most control systems. (d) Error handling and reporting. This code interacts with most of the other code in the system. Context sensitive error handling
The remainder of this paper will concentrate on the concurrent task support requirements for hard realtime operating system kernels (RTOSK). Section 2 will discuss the facilities that are required in a RTOSK. Section 3 will describe the structure of two RTOS's developed at the University of Newcastle. 313
when the low priority task claims a resource and at some time prior to its release a high priority task requests the same resource. The high priority task then becomes blocked on the resource and must wait until the low priority task releases the resource. The length of time for which the effective priority alteration occurs is very much application dependent.
Finally, section 4 will discuss two control system case studies using one of these kernels.
Real-time Kernel Requirements A real-time kernel should provide the following features:(i) Multi-tasking support - i.e. the ability to run a number of separate asynchronous tasks. (ii) A scheduling algorithm which attempts to ensure that the timing constraints required for correct control system operation can be satisfied at all times. (iii) It should not adversely affect the interrupt latency times for the system. (iv) A mechanism to precisely time and trigger software execution. (v) If timing constraints are not being satisfied then the kernel should provide basic monitoring support to allow the rea~on for the timing failure to be identified. (vi) The kernel should be backed up by a good development environment which amongst other things should allow debugging in a multitasking environment.
The solution adopted in UNOS to overcome this problem is to implement dynamic priorities for tasks. When a task is created it is assigned two priorities - a dynamic priority and a static priority. Initially these two are the same. However in the situation when a low priority task has claimed a resource which a high priority task requires the dynamic priority of the low priority task is raised to that of the high priority requesting task. The low priority task will then continue to execute at this new priority until it releases the resource, at which time its dynamic priority will drop back to its static priority. All task scheduling is carried out based on the task dynamic priority. The inclusion of dynamic priorities adds considerable complexity to the kernel. Another undesirable property of the dynamic priority mechanism is that there is a possibility of multiple recursion inside the kernel. Consider the following example. A low priority task claims a resource and then is blocked itself on another resource, prior to releasing the first resource. A high priority task then requests the first resource. This means that the priority of the claiming task must be raised to that of the high priority task. Because the first task is blocked on a second resource this means that its position in a priority ordered semaphore queue has to change to reflect the new dynamic priority. This scenario can then be repeated again for the task that is claiming the second resource, and so on. Obviously such behaviour leads to uncertainty about the time that a semaphore operation takes. The solution taken in UNOS to solve this problem is to only allow a limited number of recursions to occur.
Most commercial real-time kernels do not offer all of these facilities, or only offer some in a limited way.
Example Real-time Kernels Over the last 4 years two real-time operating systems kernels have been developed at the University of Newcastle. The UNOS kernel The development the University of Newcastle Operating System (UNOS) was primarily motivated by the requirement to have a real-time kernel for a large project. One of the main design objectives of the UNOS kernel was that it be portable across different hardware platforms. The first version was written in PL/M, but was subsequently translated into 'C'.
Another feature which is currently being added to the UNOS kernel is support for timing violation monitoring. If a task has timing violation monitoring turned on then a time window is defined (around an absolute time) which the task has to execute within, else it has not satisfied the desired timing. If the timing is violated then at the time of violation the state of the system is saved into a timing violation circular buffer, which can be interrogated to see what tasks were running prior to the violation. This should be an invaluable aid in debugging timing violations that result from unanticipated task sequences.
UNOS is essentially a conventional pre-emptive priority based real-time kernel [1]. However it does contain a few unusual features which are not found in most commercial real-time kernels. One problem that commonly arises in any moderate to large size real-time programming task is the sharing of resources between low priority and high priority tasks. Generally it is a good idea to try and avoid these situations, but in some cases this cannot be achieved. The result of such an interaction is that the high priority task's priority can transiently become that of the low priority task. This can occur 314
The PM OS kernel
The timing of tasks in UNOS can be achieved in two ways. Since the operating system supports time slicing then some tasks can be run every time slice. This situation often occurs in control systems applications since the control algorithm task almost always fall into the category of tasks that operate this way. More sophisticated task timing requires another mechanism. In UNOS an object called a timer is provided by the operating system. There are two types of timers - single shot timers, and repetitive timers. When a timer is created by a task the time to time out and the task to be executed at time out are initialised. In the case of a single shot timer when time out occurs the time out function is executed and the timer is placed back into a queue of inactive timers. Repetitive timers on the other hand place the timer immediately back onto the active time queue with the time out period reinitialised and then execute the time out function . This means that repetitive timers can support very accurate repetitive timing. The clock tick used for the timers is the basic clock tick for the system. However UNOS allows the tick frequency to be separated from the time slice frequency, therefore subject to tick routine overheads arbitrary timing accuracy can be achieved.
The second kernel, known as PMOS (peter Moylan's Operating System), was motivated primarily from an educational requirement This kernel is being used in a number of undergraduate courses within the department, and is also used as a vehicle to test new scheduling ideas. PMOS is much larger than UNOS in that it supports file systems and has extensive display text and graphics support. The implementation language is Modula-2 [2, 3]. The key feature of PMOS is its modularity. It is presumed that the applications programmers will at times need to make modifications to the system software - adding new device drivers, for example, or changing the scheduling algorithm for a special application - and the system is designed to make this easy to do. Users can add, remove, or modify modules with confidence that there will be no unexpected side-effects in the rest of the system. The modularity ensures, too, that applications programs get only the features that they need. Some applications require features such as a file system, graphics, screen menus, and the like; for others, such features would represent an unacceptable overhead. To allow for this, PMOS is implemented as a large number of small modules, which can be linked in with the applications programs as needed.
Timers are user transparently by the operating for such operations as " timed waits" on semaphores and time constraints on the receipt of messages from mail boxes (see below).
The file system, for example, does not import anything from the device driver modules. Instead, each device driver has a procedure call (in its initialization code) to "install" itself into the file system. This means that only those devices whose driver modules are imported at a higher level will become known to the file system. The programmer also has the option of omitting the file system and calling device drivers directly. This is usually a better option in embedded system applications, where special-purpose devices are more common than file-oriented devices.
UNOS uses mail boxes as the inter-task communication mechanism. Although mail boxes are a common technique of inter-task communication in many operating systems, the implementation in UNOS has some novelty. When a task is created a mail box is created and bound to that task. This means that any task can always communicate with any other task because the mail box always exists. In order to locate the mail box for another task it has associated with it an address, which is the task name for the task bound to the mail box. A mail exchange agent within the operating system kernel routes sent messages to the correct task using this address.
The modularity extends to the point where the notion of a "task" or "process" is largely hidden from the programmer. There are, for example, no kernel mail boxes. Tasks are synchronized using semaphores, and they communicate via shared data structures. The shared data structures, implemented through "abstract data type" modules, are typically not visible at the application level. To the application programmer, intertask communication occurs through procedure calls.
The mail boxes themselves are conventional in structure. They contain a user definable of message slots of user definable size. Messages are read from the mail boxes in a FIFO fashion, except in the case of an express message (for which there is a special message location). The mail boxes are designed to support concurrent operation of tasks, and do not force a "rendezvous" synchronization between the sending and receiving tasks. A return address is always attached to any sent message.
Scheduling methods in PMOS are currently under review. The present arrangement is a pure prioritybased preemptive scheduler. We are now working on methods for implementing priority inheritance 315
relationship between the sequencer and the other tasks in the system.
and - if possible - deadline inheritance. There are still some unsolved theoretical problems in this area.
Case Studies The Pipeline Crack Investigation Vehicle
This project, which began in 1988, involved the design and development of a computer controlled vehicle to travel down underground gas pipelines and search for areas of stress corrosion cracking. The mechanics of the system were designed by an industrial partner, and the University of Newcastle built the computing and electronics hardware and wrote the software. Due to the vague design specification during the initial stages of the development it was decided to base the system on a real-time operating system. The main reason for this was the flexibility offered by the operating system approach. The ease with which new tasks can be added to an operating system based software strucutre meant that unforeseen design changes would not result in significant changes to the existing software. It was decided to develop our own operating system, and hence the UNOS operating system was born. Figure 1 : PCIV task relationships.
The hardware for PCIV was based on an Intel 8OC186/187 microprocessor and a Texas Instruments TMS320C25 DSP. The software for the system could be divided into three main sections:-
The sampling task was the major periodic task. It fired the ultrasonic heads used to detect the cracking in the pipes, and then collected the return echo data, pre-processed and stored it for later retrieval and post processing. It was executed every 5 msec. The pre-processing and storage was carried out concurrently with sampling by the data processing task. The vehicle speed was controlled using a PI controller and a 2 quadrant reversing OC drive system. The limit task was also running concurrently when the vehicle was operating in sampling mode, constantly checking to see if the sampling start and stop distances had occurred, if the batteries still have sufficient capacity to retrieve the vehicle, or if there was enough water couplant to lubricate the ultrasonic heads.
(i) The user interface (ii) The signal processing and crack detection. (Hi) The vehicle control.
A simple menu based user interface was used to monitor and program the system. Ancillary modules added to aid in debugging and validation of the software were:a)
A context sensitive error logging and display system. b) An operating system monitor which allowed one to view the status of the tasks in the system.
Of the tasks shown in Figure 1, six of them operate concurrently when sampling is in progress. In addition to the tasks shown here, there are two tasks handling the serial communications via a radio modem link. This degree of concurrency would be difficult to handle in a traditional looping executive environment. The use of the multi-tasking environment also aided in code developement, since it was easy to write tasks that were able to simulate such things as the vehicle's motion. This allowed code development and testing to be carried out on a bench.
The completed software system contained approximately 30 tasks and was 60,000 lines of documented code. The main task was the sequencer task whose function was to control the other tasks in the system by sending mail box messages to them. The communications was carried out on a send acknowledge basis. If an acknowledgment did not arrive in a specified time then an error action would occur. Additional sequencer complexity resulted from the fact that it could accept asynchronous messages from other tasks. Figure 1 shows the 316
The major components in this system are:
The version of UNOS used for the PCIV project was primitive in relation to the current UNOS versions. The experience gained during this project influenced the development of the subsequent 'C' UNOS version.
(a) Supervisory Control and Data Acquisition (SCADA) systems for operator use. (b) Antenna Control and Tracking Unit (ATCU) This monitored critical antenna parameters and carried out closed loop position control. Control output was via a digital-to-analogue converter to two sets four-quadrant regenerative DC motor drives, one for each axis. (c) PLC to monitor less important antenna structural and electrical parameters.
Antenna Tracking Control System This project involved the design and implementation of a real-time control and supervisory system for four medium sized (approximately 26m diameter) satellite tracking antennae for low-inclined orbit satellites. Capability for star tracking was also required to carry out gain/temperature calibration of the receiver.
Altogether there were some 23 tasks in the final system comprising communication serial drivers, PLC protocol converters, parsers, the sequencer, Kalman filter updates for the satellite tracking algorithms, and operating system monitoring tasks.
The design process involved a quality assurance plan, rigid project management, and software development procedures to military standards. Also tight budget and timing constraints meant a development path of least risk was needed. All of these factors highlighted how closely coupled a control system design is to other aspects of a project. As well as these project management issues, we had to take a systems view of the project ensuring the proper integration of structural and mechanical components.
Only one month after starting the project a working system was implemented with a sequencer shell, dummy tasks, and antenna models for simulation. This proved to be an extremely good software development model in which to approach a medium size project. From the very beginning we could see the system evolve and, as new software modules were written, tasks could be added, and bugs detected and removed at each stage of software writing. In taking this approach, the momentum of the project development was sustained, keeping morale high in the development team, and highlighting possible problems that even critical design principles could not foresee.
Although many commercially available real-time operating systems were evaluated, it was decided early in the project to use UNOS. The reason for this was that other operating systems available at the time did not satisfy our requirement for real-time control. Also, although they offered apparently convenient graphic user interfaces, they failed to address such issues as high resolution timer support, deadlock in intertask communication, preemptive kernel task scheduling, and a suitable development environment.
ReaI·tine aoct
10-
f--
I/O InI"'_
5eqIiencer T~
Pan« TDIk
To further reduce risk and maintain a policy of a multi-sourced product, industrial grade 80386/80387 PCs were used with commercially available bus interfaces. A Programmable Logic Controller (PLC) was requested by the client to provide monitoring of other mechanical and electrical components around the structure. The Antenna Control and Tracking Unit (ATCU) was required to communicate with other computers via serial lines. One of the supervisory systems was located hundreds of meters away, communicating via a fibre-optic cable.
-
I 11
· AnlIMO latus • Safely • COIbIaIion
f - PetItion/Rat. ModI cAgcritlmt
· SiIUaIors
Pr04oco1ll AdqIIIv. Pceition locp
-
r--
SIll" Tracking cAgcritlm
I Sal.... Troctlnll
An advantage of using an in-house kernel was that it could be tailored to the precise needs of the project. Having the source code for the RTOSK also meant bugs could be quickly found and rectified without having to wait on a third party supplier to solve problems.
~
(<*nil"! FII. Model updale
Figure 2 : Antenna controller task relationships. As with the previous case study, a sequencer was employed to supervise antenna control. Figure 2 provides an overview of sequencer operation. The 317
sequencer in this case, however, was a single task modelled as a sequential state machine driven by various internal states on each control sampling period. To reduce jitter in control sampling and digital-to-analogue output, the sequencer was signalled every control interval using the UNOS repetitive timer facility and was designated as the highest priority task in the system. This resulted in the sequencer being guaranteed to run with only the timing jitter due to background interrupt routines and kernel task switch times. Being a sequential state machine, state changes were synchronised with the sampling period making controller changeovers and integrator and adaptive filter resetting simpler. This was particularly important considering that there were over 100 states and modes that the sequencer could be in and smooth controller changeover was essential.
entire real-time system running - something present day simulation packages can not achieve easily and modem controller theory has not addressed. We found the use of a simulation facility built into the code beneficial from a development viewpoint. With a well thought out simulation capability, commissioning becomes merely a fine tuning exercise. This raises another important point: in practice, the plant is usually well known beforehand. All a priori knowledge of the plant, even before it is built, should be used in the design. Controller analysis does form part of the design process, especially where adaptive loops are involved; but most of the control loop coding is related to engineering a solution based on hundreds or even thousands of non-linear conditions, boolean states, as well as computer and operating system issues.
The sequencer was primarily concerned with the state of the control system, whereas other housekeeping duties were performed by other tasks. The sequencer contained no resource waits. This ensured that, regardless of the status of other tasks - and provided that the kernel itself did not crash - the sequencer could maintain antenna control, and take appropriate action on system failure. As a further precaution a hardware watchdog had to be refreshed by the sequencer at regular intervals. On sequencer task failure the system would reset, and the antenna motor drives could be automatically shut down.
Conclusions Experience with the above-mentioned as well as other projects leads to the following conclusions:(i) All but trival real-time programming projects should be implemented using a operating system kernel specifically designed for realtime applications. The use of a RTOSK enhances the modularity of the software, and allows structured development even if there is an ill-defined specification. (ii) Access to the source of the kernel is very important since is allows the kernel to be customized and bugs to be fixed. (iii) Provision of debugging aids in the form of error display, timing violation and operating system monitor facilities is important in large projects. The latter two imply special RTOSK support for their implementation.
The sequencer was also responsible for calling the position control loops. These consisted of adaptive feed forward PI controllers. The adaptive algorithm is described elsewhere [4]. An extensive simulation facility was built into the system from the start. This facility was built into the sequencer and other tasks enabling the entire system to be run on any PC. NVRAM, digital inputs and outputs, antenna structural resonance, wind disturbances, serial input, and satellite beacon locations were all simulated. This proved advantageous as no changes in software were required, even after 12 months development without a real antenna, during the commissioning stage.
References [1] Betz, R., and Sciacca L. (1992). UNOS User Manual (version 1.5a). Tech. Rept., University of Newcastle, Australia. [2] Moylan, P. (1991). The PMOS definition modules. Tech. Rept. EE91 07, University of Newcastle, Australia. [3] Moylan, P. (1992). The PMOS real-time kernel. IntI. Workshop on Real-Time Programming, Bruges, Belgium. [4] Sciacca L. J. , Evans R. J., Betz R. E.,and Cooper D. N. (1990). Adaptive Control for the Australia Telescope, 11 th IFAC World Congress, Tallinn, Estonia.
What this case study has told us is that a controller can not be thought of in isolation of other system components, even the more intangible aspects of project management and software management become factors in its design. The boundaries of where the linear controller code ends and the operating system begins is a vague one. The ATCU alone consisted of over lOO,()()() lines of documented code, of which only 0.1 % was for the adaptive controller. However the behaviour of the controller in this case could only be properly studied with the
318
. REAL - TIME CONTROL APPUCA nONS
Control Environment Implementation Issues - with Case Studies Robert E. Betz*, Peter J. Moylan*, Len J. Sciacca**
* Department of Electrical and Computer Engineering, University of Newcastle, Callaghan , NSW ,2308, Australia * Department of Electrical Engineering , University of M elbourne, Parkville, V ie, 3052, Australia
Abstract: This paper discusses a number of practical issues related to the software environment required for the successful implementation of medium to large scale control systems. It is argued that for all but trival systems it is desirable to implement the control system within a software environment which formally supports real-time multi-tasking. Keywords; Real-time computer systems, Operating Systems
Introduction
important. In some system consequential suppression is also supported. (e) Task support. This code provides the underlying multi-tasking support for the system. (t) Communications support. This code provides the software interface to serial channels, networks etc. that the system must communicate with. (g) Plant interface modules. This code provides the software interfaces to the hardware devices which directly interact with the environment. Examples are NO converters, relay control logic and other more specialised I/O devices.
The development of a computer control system involves a number of aspects: the design of a control algorithm, the selection of suitable hardware, the software design and implementation, the development of actuator and sensor mechanisms, and so on. Looking at the control theory literature, one can get the impression that the algorithm design is the biggest part of the job, or at least the most intellectually demanding. In practice, one tends to find that the software development phase is where most of the development time goes; and that the complexity of the work is such that it cannot be trusted to "average" computer programmers.
The form of the underlying concurrent task support is often an area of confusion. A Computer Scientist for example assumes that a real-time system is any system that has to react to external asynchronous events. However there needs to be a differentiation between, say an airline booking system, and a aircraft control system. An airline booking ystem is an example of a soft real-time system, because the consequences of slower response are not significant. An aircraft control system is an example of a hard real-time system. If timing constraints are not met all the time then the control system will not achieve acceptable performance, or catastrophic failure may result. Soft real-time applications can usually be supported with conventional operating systems, whereas hard real-time applications require special operating system kernels which can offer deterministic task response times.
Typically, the control algorithm itself can, even if it is a complex algorithm, be expressed in just a few pages of straightforward code in a high level language. The total software package, in contrast, is likely to be hundreds of pages long; and its design involves some complex and often subtle problems. The majority of the code in most medium to large control systems is related to functions such as:(a) Operator interface - usually a large section of code. Important to the user as it is the means of controlling and monitoring the system . (b) Logical decisions. Usually surrounds the controller code. Governs control transitions such as the change from linear to slewing mode. (c) Sequencing of events. Usually a large and complex section of code. Contains logical decisions which also contain timing constraints. For example, timed sequences are required to start and stop most control systems. (d) Error handling and reporting. This code interacts with most of the other code in the system. Context sensitive error handling
The remainder of this paper will concentrate on the concurrent task support requirements for hard realtime operating system kernels (RTOSK). Section 2 will discuss the facilities that are required in a RTOSK. Section 3 will describe the structure of two RTOS's developed at the University of Newcastle. 313
when the low priority task claims a resource and at some time prior to its release a high priority task requests the same resource. The high priority task then becomes blocked on the resource and must wait until the low priority task releases the resource. The length of time for which the effective priority alteration occurs is very much application dependent.
Finally, section 4 will discuss two control system case studies using one of these kernels.
Real-time Kernel Requirements A real-time kernel should provide the following features:(i) Multi-tasking support - i.e. the ability to run a number of separate asynchronous tasks. (ii) A scheduling algorithm which attempts to ensure that the timing constraints required for correct control system operation can be satisfied at all times. (iii) It should not adversely affect the interrupt latency times for the system. (iv) A mechanism to precisely time and trigger software execution. (v) If timing constraints are not being satisfied then the kernel should provide basic monitoring support to allow the rea~on for the timing failure to be identified. (vi) The kernel should be backed up by a good development environment which amongst other things should allow debugging in a multitasking environment.
The solution adopted in UNOS to overcome this problem is to implement dynamic priorities for tasks. When a task is created it is assigned two priorities - a dynamic priority and a static priority. Initially these two are the same. However in the situation when a low priority task has claimed a resource which a high priority task requires the dynamic priority of the low priority task is raised to that of the high priority requesting task. The low priority task will then continue to execute at this new priority until it releases the resource, at which time its dynamic priority will drop back to its static priority. All task scheduling is carried out based on the task dynamic priority. The inclusion of dynamic priorities adds considerable complexity to the kernel. Another undesirable property of the dynamic priority mechanism is that there is a possibility of multiple recursion inside the kernel. Consider the following example. A low priority task claims a resource and then is blocked itself on another resource, prior to releasing the first resource. A high priority task then requests the first resource. This means that the priority of the claiming task must be raised to that of the high priority task. Because the first task is blocked on a second resource this means that its position in a priority ordered semaphore queue has to change to reflect the new dynamic priority. This scenario can then be repeated again for the task that is claiming the second resource, and so on. Obviously such behaviour leads to uncertainty about the time that a semaphore operation takes. The solution taken in UNOS to solve this problem is to only allow a limited number of recursions to occur.
Most commercial real-time kernels do not offer all of these facilities, or only offer some in a limited way.
Example Real-time Kernels Over the last 4 years two real-time operating systems kernels have been developed at the University of Newcastle. The UNOS kernel The development the University of Newcastle Operating System (UNOS) was primarily motivated by the requirement to have a real-time kernel for a large project. One of the main design objectives of the UNOS kernel was that it be portable across different hardware platforms. The first version was written in PL/M, but was subsequently translated into 'C'.
Another feature which is currently being added to the UNOS kernel is support for timing violation monitoring. If a task has timing violation monitoring turned on then a time window is defined (around an absolute time) which the task has to execute within, else it has not satisfied the desired timing. If the timing is violated then at the time of violation the state of the system is saved into a timing violation circular buffer, which can be interrogated to see what tasks were running prior to the violation. This should be an invaluable aid in debugging timing violations that result from unanticipated task sequences.
UNOS is essentially a conventional pre-emptive priority based real-time kernel [1]. However it does contain a few unusual features which are not found in most commercial real-time kernels. One problem that commonly arises in any moderate to large size real-time programming task is the sharing of resources between low priority and high priority tasks. Generally it is a good idea to try and avoid these situations, but in some cases this cannot be achieved. The result of such an interaction is that the high priority task's priority can transiently become that of the low priority task. This can occur 314
The PM OS kernel
The timing of tasks in UNOS can be achieved in two ways. Since the operating system supports time slicing then some tasks can be run every time slice. This situation often occurs in control systems applications since the control algorithm task almost always fall into the category of tasks that operate this way. More sophisticated task timing requires another mechanism. In UNOS an object called a timer is provided by the operating system. There are two types of timers - single shot timers, and repetitive timers. When a timer is created by a task the time to time out and the task to be executed at time out are initialised. In the case of a single shot timer when time out occurs the time out function is executed and the timer is placed back into a queue of inactive timers. Repetitive timers on the other hand place the timer immediately back onto the active time queue with the time out period reinitialised and then execute the time out function . This means that repetitive timers can support very accurate repetitive timing. The clock tick used for the timers is the basic clock tick for the system. However UNOS allows the tick frequency to be separated from the time slice frequency, therefore subject to tick routine overheads arbitrary timing accuracy can be achieved.
The second kernel, known as PMOS (peter Moylan's Operating System), was motivated primarily from an educational requirement This kernel is being used in a number of undergraduate courses within the department, and is also used as a vehicle to test new scheduling ideas. PMOS is much larger than UNOS in that it supports file systems and has extensive display text and graphics support. The implementation language is Modula-2 [2, 3]. The key feature of PMOS is its modularity. It is presumed that the applications programmers will at times need to make modifications to the system software - adding new device drivers, for example, or changing the scheduling algorithm for a special application - and the system is designed to make this easy to do. Users can add, remove, or modify modules with confidence that there will be no unexpected side-effects in the rest of the system. The modularity ensures, too, that applications programs get only the features that they need. Some applications require features such as a file system, graphics, screen menus, and the like; for others, such features would represent an unacceptable overhead. To allow for this, PMOS is implemented as a large number of small modules, which can be linked in with the applications programs as needed.
Timers are user transparently by the operating for such operations as " timed waits" on semaphores and time constraints on the receipt of messages from mail boxes (see below).
The file system, for example, does not import anything from the device driver modules. Instead, each device driver has a procedure call (in its initialization code) to "install" itself into the file system. This means that only those devices whose driver modules are imported at a higher level will become known to the file system. The programmer also has the option of omitting the file system and calling device drivers directly. This is usually a better option in embedded system applications, where special-purpose devices are more common than file-oriented devices.
UNOS uses mail boxes as the inter-task communication mechanism. Although mail boxes are a common technique of inter-task communication in many operating systems, the implementation in UNOS has some novelty. When a task is created a mail box is created and bound to that task. This means that any task can always communicate with any other task because the mail box always exists. In order to locate the mail box for another task it has associated with it an address, which is the task name for the task bound to the mail box. A mail exchange agent within the operating system kernel routes sent messages to the correct task using this address.
The modularity extends to the point where the notion of a "task" or "process" is largely hidden from the programmer. There are, for example, no kernel mail boxes. Tasks are synchronized using semaphores, and they communicate via shared data structures. The shared data structures, implemented through "abstract data type" modules, are typically not visible at the application level. To the application programmer, intertask communication occurs through procedure calls.
The mail boxes themselves are conventional in structure. They contain a user definable of message slots of user definable size. Messages are read from the mail boxes in a FIFO fashion, except in the case of an express message (for which there is a special message location). The mail boxes are designed to support concurrent operation of tasks, and do not force a "rendezvous" synchronization between the sending and receiving tasks. A return address is always attached to any sent message.
Scheduling methods in PMOS are currently under review. The present arrangement is a pure prioritybased preemptive scheduler. We are now working on methods for implementing priority inheritance 315
relationship between the sequencer and the other tasks in the system.
and - if possible - deadline inheritance. There are still some unsolved theoretical problems in this area.
Case Studies The Pipeline Crack Investigation Vehicle
This project, which began in 1988, involved the design and development of a computer controlled vehicle to travel down underground gas pipelines and search for areas of stress corrosion cracking. The mechanics of the system were designed by an industrial partner, and the University of Newcastle built the computing and electronics hardware and wrote the software. Due to the vague design specification during the initial stages of the development it was decided to base the system on a real-time operating system. The main reason for this was the flexibility offered by the operating system approach. The ease with which new tasks can be added to an operating system based software strucutre meant that unforeseen design changes would not result in significant changes to the existing software. It was decided to develop our own operating system, and hence the UNOS operating system was born. Figure 1 : PCIV task relationships.
The hardware for PCIV was based on an Intel 8OC186/187 microprocessor and a Texas Instruments TMS320C25 DSP. The software for the system could be divided into three main sections:-
The sampling task was the major periodic task. It fired the ultrasonic heads used to detect the cracking in the pipes, and then collected the return echo data, pre-processed and stored it for later retrieval and post processing. It was executed every 5 msec. The pre-processing and storage was carried out concurrently with sampling by the data processing task. The vehicle speed was controlled using a PI controller and a 2 quadrant reversing OC drive system. The limit task was also running concurrently when the vehicle was operating in sampling mode, constantly checking to see if the sampling start and stop distances had occurred, if the batteries still have sufficient capacity to retrieve the vehicle, or if there was enough water couplant to lubricate the ultrasonic heads.
(i) The user interface (ii) The signal processing and crack detection. (Hi) The vehicle control.
A simple menu based user interface was used to monitor and program the system. Ancillary modules added to aid in debugging and validation of the software were:a)
A context sensitive error logging and display system. b) An operating system monitor which allowed one to view the status of the tasks in the system.
Of the tasks shown in Figure 1, six of them operate concurrently when sampling is in progress. In addition to the tasks shown here, there are two tasks handling the serial communications via a radio modem link. This degree of concurrency would be difficult to handle in a traditional looping executive environment. The use of the multi-tasking environment also aided in code developement, since it was easy to write tasks that were able to simulate such things as the vehicle's motion. This allowed code development and testing to be carried out on a bench.
The completed software system contained approximately 30 tasks and was 60,000 lines of documented code. The main task was the sequencer task whose function was to control the other tasks in the system by sending mail box messages to them. The communications was carried out on a send acknowledge basis. If an acknowledgment did not arrive in a specified time then an error action would occur. Additional sequencer complexity resulted from the fact that it could accept asynchronous messages from other tasks. Figure 1 shows the 316
The major components in this system are:
The version of UNOS used for the PCIV project was primitive in relation to the current UNOS versions. The experience gained during this project influenced the development of the subsequent 'C' UNOS version.
(a) Supervisory Control and Data Acquisition (SCADA) systems for operator use. (b) Antenna Control and Tracking Unit (ATCU) This monitored critical antenna parameters and carried out closed loop position control. Control output was via a digital-to-analogue converter to two sets four-quadrant regenerative DC motor drives, one for each axis. (c) PLC to monitor less important antenna structural and electrical parameters.
Antenna Tracking Control System This project involved the design and implementation of a real-time control and supervisory system for four medium sized (approximately 26m diameter) satellite tracking antennae for low-inclined orbit satellites. Capability for star tracking was also required to carry out gain/temperature calibration of the receiver.
Altogether there were some 23 tasks in the final system comprising communication serial drivers, PLC protocol converters, parsers, the sequencer, Kalman filter updates for the satellite tracking algorithms, and operating system monitoring tasks.
The design process involved a quality assurance plan, rigid project management, and software development procedures to military standards. Also tight budget and timing constraints meant a development path of least risk was needed. All of these factors highlighted how closely coupled a control system design is to other aspects of a project. As well as these project management issues, we had to take a systems view of the project ensuring the proper integration of structural and mechanical components.
Only one month after starting the project a working system was implemented with a sequencer shell, dummy tasks, and antenna models for simulation. This proved to be an extremely good software development model in which to approach a medium size project. From the very beginning we could see the system evolve and, as new software modules were written, tasks could be added, and bugs detected and removed at each stage of software writing. In taking this approach, the momentum of the project development was sustained, keeping morale high in the development team, and highlighting possible problems that even critical design principles could not foresee.
Although many commercially available real-time operating systems were evaluated, it was decided early in the project to use UNOS. The reason for this was that other operating systems available at the time did not satisfy our requirement for real-time control. Also, although they offered apparently convenient graphic user interfaces, they failed to address such issues as high resolution timer support, deadlock in intertask communication, preemptive kernel task scheduling, and a suitable development environment.
ReaI·tine aoct
10-
f--
I/O InI"'_
5eqIiencer T~
Pan« TDIk
To further reduce risk and maintain a policy of a multi-sourced product, industrial grade 80386/80387 PCs were used with commercially available bus interfaces. A Programmable Logic Controller (PLC) was requested by the client to provide monitoring of other mechanical and electrical components around the structure. The Antenna Control and Tracking Unit (ATCU) was required to communicate with other computers via serial lines. One of the supervisory systems was located hundreds of meters away, communicating via a fibre-optic cable.
-
I 11
· AnlIMO latus • Safely • COIbIaIion
f - PetItion/Rat. ModI cAgcritlmt
· SiIUaIors
Pr04oco1ll AdqIIIv. Pceition locp
-
r--
SIll" Tracking cAgcritlm
I Sal.... Troctlnll
An advantage of using an in-house kernel was that it could be tailored to the precise needs of the project. Having the source code for the RTOSK also meant bugs could be quickly found and rectified without having to wait on a third party supplier to solve problems.
~
(<*nil"! FII. Model updale
Figure 2 : Antenna controller task relationships. As with the previous case study, a sequencer was employed to supervise antenna control. Figure 2 provides an overview of sequencer operation. The 317
sequencer in this case, however, was a single task modelled as a sequential state machine driven by various internal states on each control sampling period. To reduce jitter in control sampling and digital-to-analogue output, the sequencer was signalled every control interval using the UNOS repetitive timer facility and was designated as the highest priority task in the system. This resulted in the sequencer being guaranteed to run with only the timing jitter due to background interrupt routines and kernel task switch times. Being a sequential state machine, state changes were synchronised with the sampling period making controller changeovers and integrator and adaptive filter resetting simpler. This was particularly important considering that there were over 100 states and modes that the sequencer could be in and smooth controller changeover was essential.
entire real-time system running - something present day simulation packages can not achieve easily and modem controller theory has not addressed. We found the use of a simulation facility built into the code beneficial from a development viewpoint. With a well thought out simulation capability, commissioning becomes merely a fine tuning exercise. This raises another important point: in practice, the plant is usually well known beforehand. All a priori knowledge of the plant, even before it is built, should be used in the design. Controller analysis does form part of the design process, especially where adaptive loops are involved; but most of the control loop coding is related to engineering a solution based on hundreds or even thousands of non-linear conditions, boolean states, as well as computer and operating system issues.
The sequencer was primarily concerned with the state of the control system, whereas other housekeeping duties were performed by other tasks. The sequencer contained no resource waits. This ensured that, regardless of the status of other tasks - and provided that the kernel itself did not crash - the sequencer could maintain antenna control, and take appropriate action on system failure. As a further precaution a hardware watchdog had to be refreshed by the sequencer at regular intervals. On sequencer task failure the system would reset, and the antenna motor drives could be automatically shut down.
Conclusions Experience with the above-mentioned as well as other projects leads to the following conclusions:(i) All but trival real-time programming projects should be implemented using a operating system kernel specifically designed for realtime applications. The use of a RTOSK enhances the modularity of the software, and allows structured development even if there is an ill-defined specification. (ii) Access to the source of the kernel is very important since is allows the kernel to be customized and bugs to be fixed. (iii) Provision of debugging aids in the form of error display, timing violation and operating system monitor facilities is important in large projects. The latter two imply special RTOSK support for their implementation.
The sequencer was also responsible for calling the position control loops. These consisted of adaptive feed forward PI controllers. The adaptive algorithm is described elsewhere [4]. An extensive simulation facility was built into the system from the start. This facility was built into the sequencer and other tasks enabling the entire system to be run on any PC. NVRAM, digital inputs and outputs, antenna structural resonance, wind disturbances, serial input, and satellite beacon locations were all simulated. This proved advantageous as no changes in software were required, even after 12 months development without a real antenna, during the commissioning stage.
References [1] Betz, R., and Sciacca L. (1992). UNOS User Manual (version 1.5a). Tech. Rept., University of Newcastle, Australia. [2] Moylan, P. (1991). The PMOS definition modules. Tech. Rept. EE91 07, University of Newcastle, Australia. [3] Moylan, P. (1992). The PMOS real-time kernel. IntI. Workshop on Real-Time Programming, Bruges, Belgium. [4] Sciacca L. J. , Evans R. J., Betz R. E.,and Cooper D. N. (1990). Adaptive Control for the Australia Telescope, 11 th IFAC World Congress, Tallinn, Estonia.
What this case study has told us is that a controller can not be thought of in isolation of other system components, even the more intangible aspects of project management and software management become factors in its design. The boundaries of where the linear controller code ends and the operating system begins is a vague one. The ATCU alone consisted of over lOO,()()() lines of documented code, of which only 0.1 % was for the adaptive controller. However the behaviour of the controller in this case could only be properly studied with the
318