Automatic bioprocess control. 1. A general concept

Journal of Biotechnology, 19 (1991) 0 1991 Elsevier Science Publishers ADONIS 016816569100091G

BIOTEC

1-18 B.V. 0168-1656/91/$03.50

00613

Automatic bioprocess control. 1. A general concept Bernhard Sonnleitner, Georg Lecher and Armin Fiechter Institute

for Biotechnology,

(Received

5 April

ETH 1990;

Hijnggerberg,

revision

Ziirich,

accepted

9 June

Switzerland 1990)

Summary Automation of bioprocesses is presented and discussed. A general concept is applied to laboratory scale reactors as well as to large scale production facilities consisting of many unit operations with a hierarchical and highly modular structure. The implementation of non-dedicated and intelligent analytical subsystems is foreseen. Hard- and software requirements are discussed in view of the functional requirements of both scientific research and production engineering. Some practical experience is reported using several different components in parallel installations. Bioprocess automation; DDC (direct digital control); Computer application; Hardand software; Modular concept; Reproducitivity of bioprocesses; Performance

Introduction The higher the value of a product from biotechnology, the higher are the demands for a constant and excellent product quality. The obvious consequences for process engineering, quality control as well as environmental control are: an urgent need for increased safety of the product, decreased pollution of the environment and reproducibility of the processing steps. The recent history of disastrous accidents in the chemical industries teaches that ignorance and human error are the major obstacles to achieving the above goals. Better processing will come with the improvement of physiological and biochemical knowledge and the application of automatic control in biotechnological processes (Schiigerl, 1988). Correspondence Switzerland.

to: B. Sonnleitner.

Institute

for Biotechnology,

ETH

Hikggerberg

HPT.

CH 8093 Ziirich,

2

A bioprocess encompasses several subprocesses (unit operations), i.e. the biotransformation, up stream and down stream processing of substrates, products and byproducts, and an appropriate disposal of wastes. Effective control of these subprocesses requires a sound understanding of the (bio)chemical reaction or metabolic control mechanisms and a precise and accurate knowledge of the state of a process. But this means that we must improve methodology, tools and equipment to know what is qualitatively and quantitatively going on in a biotechnological (sub)process. Representative measurement of as many state variables as possible is decisive since this is the only way to obtain abundant information about a current process. Enrichment of biological knowledge allows decision as to which measures are really important for process control. Excellent automation will come from exploitation of currently available logistic and electronic tools and the development of sensors and intelligent analytical systems that are appropriate for use under aseptic conditions in a technical environment. These analytical data should be produced in situ, on-line, continuously in real time and cover a wide dynamic range (of activities and/or concentrations) in order to exploit them for process control. In order to gain maximal flexibility and efficiency for process control, modem technology takes advantage of direct digital control (DDC) and computers on different (hierarchical) levels: as slaves, masters and coordinators.

Objectives

The most urgent needs to be realistically served by such a concept are: excellent knowledge of the process state; excellent data(-base) for the elucidation of origin-effect-mechanisms; highly precise control of the so called culture parameters; thorough documentation of processes; extreme flexibility for on-line improvement of process behaviour; reproducibility of process and product and increased safety of process and environment. Sufficient information about the process state is a prerequisite for process control. This includes an up to date vector describing the actual state, as well as an historical trend of variables (matrix). In most cases, this information will only be used to confirm the expected time evolution of a bioprocess. However, in case of deviations, only a complete set of data, not just a few expected variables but also the time course of so-called parameters and usually unexploited secondary variables such as controller outputs etc., enables successful trouble-shooting. These data are also decisive for configuration, startup and modification procedures. Data should be accessible in a graphic environment, e.g., as well-sorted pictograms including alphanumeric displays, bar graphs or trend curves, in order to facilitate an immediate and complete overview of logical groups of primary and secondary data, i.e. measured variables, set points, margins, calculated values, errors and output variables.

3

In some cases, it might be important to have also access to engineering data such as calibration factors, regulator parameters or controller attributes. Such data may change because of alterations of the physical system (e.g. OD-sensor signal depends also on aeration rate or stirrer speed) or because of supervisory control (e.g. during a step change of a parameter set point a PI-controller might be forced to act as a pure P-controller). This extension of “information about the process state” to include the behavior of the control equipment is essential to trace the origins (biological or technical) of measured effects. Novel basic scientific knowledge is mostly derived from accurate, measured data. The value of scientific experimentation is directly proportional to the qualtity, quantity and retrievability of analytical (dynamical) data. It is, therefore, necessary to store the original data in a well organized way (data base). With reliable collected data, chances are good for unravelling unknown dependencies or mechanisms; however, the danger of getting lost in an overwhelming flood of data is also greater. Consequently, the data to be stored must be carefully selected. A practicable solution must be a compromise between quantity and quality of data. Modern database management systems can provide a professional solution. However, such systems cannot decide which data are valuable and decisive, and at what frequency they are required. They are just a technical tool to improve the data accessibility and organization. Effective rules have to be developed to optimize data selection (data quality) and the frequency of data acquisition (data quantity). More rules have to be developed to reduce the necessary amount of data without a significant loss of information; the restriction is no longer the availability of computer memory or storage capacity, it is primarily the necessity to concentrate on important and decisive data (and to avoid unintentional data losses). Culture “parameters” are process variables that are made constant by means of closed-loop controllers. Theoretically, there is not much difference between classical analogue and digital controllers. Practically, there are significant differences: plants operated with discrete measurement and control units very seldom have free (spare) units to expand the equipment to a new controlled variable. Electronic units in classical technique cannot be changed very much after installation as there are always certain constraints due to the physical elements used in construction (a beneficial side effect therefore is that a fatal mistuning is normally not possible). Whereas such a configuration is not very flexible, a DDC system normally provides for system extensions with a minimum of hardware installations (by just adding the necessary wiring from a new sensor via the amplifier to the input connectors or eventually by also adding another board). Moreover, calibration factors and controller parameters can be updated via a supervisory (computer) system. This allows full advantage of the controllers’ capabilities and their optimal tuning to be taken. As a consequence, controller performance is no longer a compromise, it can be made excellent with respect to precision and dynamics of controlled variables. Process documentation is not just good practice, it is often necessary for regulatory and forensic reasons. Automated systems are governed by rules that can be easily extended to contain print or store commands after completion or failure of important actions. These data, together with the temporal state matrix (process

4

variables and parameters), provide a sufficiently complete documentation and a sound basis for trace back and trouble-shooting. Any type of rule or data based expert system can influence the process from the supervisory system. Or, in simple cases, proper actions to improve process performance can be foreseen (programmed or configured) directly in the process dedicated, i.e. front-end component (slave). New experiences can be added to either slave or master with a minimum of effort (and even automatically in self-learning systems). This reduces efficiently the time delay for progress. Moreover, alternative strategies can be calculated in parallel and validated against each other. It is important that the high degree of flexibility can only be realized in a hierarchically well structured system: (1) All functions necessary for basic process operation must be implemented in the front-end component. (2) Supervisory as well as extended command and visualization functions should be implemented on a powerful workstation. (3) More complex analytical functions should be included as intelligent subsystems able to control the analytical equipment completely (actuators, timing, numerical analyses) and to evaluate results. (4) In case of (analytical) functions that are shared (multiplexed) among different subprocesses, direct connections to process dedicated controllers should be avoided since “process-confidential” data are to be handled; instead, a coordinating computer must be responsible for the correct distribution of such data. Automation is designed to improve the reproducibility of processes and, hence, of the products. Actions and reactions triggered by either dependent (process or threshold data) or independent (time) variables are performed identically. Human errors and individual variations can be markedly reduced. Even built-in systematic errors may be tolerated since they can be identified (quantitatively) and compensated for by calibrations or calculations. The implementation of prediction methods (time series extrapolation, mechanistic models, comparison with known patterns) allows the planning and eventually the validation in advance, of specific actions to be taken in response to a process starting to diverge from an expected behavior. Increased reproducibility of processes must result in increased safety of both, processes and the environment.

L.ogistic and physical hierarchy

Automation of (bio)processes requires the realization of several different functional levels (Schneider, 1987, 1989). The primary tasks of a bioreactor will not be discussed here, but the strictly aseptic conditions of bioprocesses impose extreme constraints on the use of on-line sensors and analyzers, drastically reducing the number of suitable available instruments. Similar restrictions apply to most actuators. Most of the historical problems with converters and amplifiers have been solved satisfactorily and a series of alternatives is commercially available. The front-end (slave) controller makes up the third level. Its tasks are relatively unintelligent and have been formerly performed by discrete electrical, electromechanical or electropneumatical units and programming was by hardware, i.e. discrete wiring. A

5

major demand is the real time behavior of this component. Some of the simple control loops such as those for pressure control, flux control, speed control, etc., require cycle times of < 1 s for laboratory and pilot scale equipment. A simple operator’s interface is mandatory. The next higher level component, the master or supervisor, has to perform more intelligent and demanding tasks. It should have a comfortable man machine interface (operator’s command and visualization station) with extra computing power to store, retrieve and organize different sets of data, to calculate resource-consuming algorithms (filters, models, patterns, etc.) and to prepare the desired documentation. It must check the users’ authorization hierarchy and manage the communication with other systems (e.g., coordinating computers or intelligent subsystems). Further levels may increase flexibility, power and comfort deliberately (Fig. la, b). On-line

anafjues,

e.special[y sensors

The current measurements of process variables in strictly aseptic bioprocesses do not, unfortunately, cover the biologically most decisive factors. Instead, only physical and some chemical variables are accessible by on-line, in situ probes (Sonnleitner and Fiechter, 1989; Fig. 2). However, there are ways to exploit indirect measures to estimate the important state variables and, in a mid term range, they are quite satisfactory. Although sensors are not the topic of this contribution, one has to keep in mind that indirect measures need models to be fully exploited (for the conversion of the indirect measure into a value for the desired variable). Since these models are normally very simple, e.g. a linear correlation, they can be easily implemented on the slave system. However, experimentally verified parameters must be provided in order to apply the models correctly; check lists implemented as “on-line help” functions or routines on the supervisory system proved extremely valuable and efficient in assuring this. Hardware

interface

The electrical interface from the sensor components to the slave system does not pose problems. However, there are hidden pitfalls built-in if low cost is over-emphasized. Saving money at the wrong place costs, in the long term, much more (time and money). Such key elements are: preamplifiers for high impedance sensors (e.g. pH, redox); galvanic isolation for excitation and signal loops powered from different supplies; inadequate shielding or partitioning of cables; improper distribution or connection of earth/ground potential. Front-end

component

The logistic center of measurement and control is the direct digital controller (DDC-slave). Its tasks are manifold. After conversion of the electrical input values to engineering units, all these raw data should be fed into calibration blocks. Minor drifts or nonlinearities of

to be “optimally”

intelligent analytical subsystems

physical Interface

II

algebraic dlfferentlatlon,

operated

and chemical from/to the

calcilatlo~s, Integration

I

user

local area nctwork

CONNECTIONS

authorlzatlon, command transmlsslon, data vlsuallratlon, complex control, data base tasks

data archlve + documentation, coordlnatlon of subprocesses & non-dedicated analytics, observers, modelllng,

economics, plannlng, . .. .. super

...

computing

Fig. la. Overview of the hierarchy of bioprocess automation components: structural and functional aspects. The left hand side shows the single modules (hardware components) as well as their connections; the right hand side shows the major tasks and key elements. The lower 2 elements need not necessarily be present for a sound automatic bioprocess operation. The principally high flexibility in extending and (inter)connecting various components is indicated by grey shadowed background boxes.

b

process level

specialized

front

end

components:

analytical

indispensable control

level

supervisory level: operating 6 visualization

coordination level

d

supervisor

U

alarm prinlers

specia’ :

software

parallel or serial

(analog or digital) lines (e.g. RS 232

connections P

II

i!IlzIl super computer 1 .. . . . . . . ,.. . . . . . . :

Fig. lb. Possible scheme to automate an entire production plant (from top to bottom) with several components on different hierarchical levels (left to right): structural and communicational aspects. The interconnections are depicted. where the directly wired analogue and digital signals (either electrical or pneumatic between sensors/actuators and amplifier/converters) are drawn as single lines; the other data paths are serial transmission lines working with different throughput (from a few kbit s-’ to several Mbit s-l). The slow terminal lines (4800 or 9600 baud) are often a severe bottleneck between slave and supervisor although the amount of data transferred decreases from left to right; however, neither the amount nor the required rates are low. At ETH, we can transfer data directly via network to computers with special software installations of super computers (indicated as special and super on the right side).

amplifiers and moderate changes of sensors can be elegantly accounted for by software calibration because this procedure can be automated, e.g. by the implementation of standard calibration sequences that must be successfully completed prior to further processing. (A 2-point linear transfer function suffices in most cases, see Fig. 3. However, static function blocks allowing a 5- or lo-point calibration are to be preferred as a standard). Thus the, step from a stand-by condition to the sterilization sequence can be logically locked unless all (preprogrammed) calibration steps have been passed. Controllers in DDC systems can normally be tuned over the unclipped range of

8

I

physical variables

characteristics measurement

chemical variables

of I

by pass I exit on line continuous

FIA size distribution

discontinuous

Fig. 2. Overview of the present state of the art for measuring relevant data in bioprocesses. The left hand part shows the measured items grouped in physical, chemical and the most interesting biological variables. The right hand part displays the characteristics of signal generation with respect to place, manner and time. The presently dominating diagonal (top left to bottom right) must be overcome in the near future, i.e. methods must be developed so that the biological variables are shifted (upwards in this scheme) towards on-line, continuous and in situ measurement. OD, optical density; ARD, acoustic resonance densitometry; “/3ugs” are cells, i.e.; fluid elements entirely sourrounded by a membrane being polarized in a radio frequency field, measured with the “/3ugmeter”; RQ, respiratory quotient; FIG, flow injection analysis; GC, gas chromatography; HPLC, high performance liquid chromatography; ELISA, enzyme linked immunosorbent assay.

zalibration

of pH electrode

reactor-ID

in

1

preconditions: - electrode has been cleaned and tested - buffers, rinsing water and electrode are at the same calibration temperature - calibration and operation temperature will not differ by more than 10 K - enter the correct oH-values as indicated

1. adjust DH to

6.99

insert electrode in buffer and wait for a stable signal, then w STORE the correct value for “pH 7

6.99

I 4.0

2. adjust pH to 4.01 I

insert electrode in buffer and wait for a stable signal, then STORE the correct value for “oH 4”

3.0 4.0

1

l-

I--0

t

fminl

5

Fig. 3. Example of a user interface picture for pH electrode calibration. This check list acts as on-line help and is installed on a visualization station. The picture can be activated via an operating menu or activated aulomatically during a “reactor preparation” sequence; successful completion would then be the condition to switch into the next step. The tasks of the actually shown example are: to remind the user to make sure that all necessary preconditions are satisfied; to make sure that the exact pH values of the calibration buffers are available (known); to provide the user with a graphical representation of the time course of the electrodes response; this allows the user to estimate the actual response time characteristics of the electrode, and to decide when the automatic conversion of the measured variables into calibration parameters is due; stores the measured (input) values and the correct pH values in a software calibration block of the DDC controller after pressing 1 key only. Invisibly, there is a check procedure active that decides whether the gain of the electrode is sufficient or not; if not, another picture can be activated to remind the user lo replace the electrode. Normal font and line width indicates static picture elements, bold indicates dynamically updated elements.

parameters (from 0 to co). Per se, this is advantageous because a maximum of flexibility is provided. However, for inexperienced users (the initial) tuning may be laborious. It is a tremendous advantage if the controller can also be functionally switched, e.g. from PI to P or to PID characteristics and vice versa. Using simple window comparators for measured value deviations allows the triggering of functional changes of a controller easily but very effectively. We have implemented such an “anti reset wind up” mechanism for all controller loops and experienced both an increased stability and a higher speed of convergence. Also less experienced students no longer mistune (until divergence) controllers. Therefore, a functional change of controller algorithms should be regarded as essential (in a certain implementation) and is recommended. Interlocking of functions (via enable or disable flags, e.g. for pumps, valves or motors) based on the results of Boolean arithmetics proves an effective means of avoiding malfunction or destruction of mechanical elements. For instance, a stirrer

must be dominantly disabled if either amount or temperature of the lubricant (e.g. condensate) for the mechanical seals are out of range, or the circulation of lubricant does not work properly, or lubricant circuit is not pressurized, or the speed set point is out of range (too low or too high) or illegal vibrations of the shaft are detected. Also, “out of normal time” conditions can be supervised effectively; such detectors are extremely helpful in maintenance and trouble-shooting operations. As many alarms as desired can be issued. Differential limiters are very helpful in avoiding unallowable accelerations of mechanical parts by converting step changes of controlled variables into sufficiently smooth trajectories. They are also essential for the improvement in the performance of physiological controllers (cf. controller for maximal biomass productivity according to Meyer and Beyeler, 1984). Multiplexers triggered by logical variables provide correct values from a set of alternatives, either calculated values or fixed values. There should be sufficient storage cells (in a non-volatile memory) to provide default values, e.g. for set points valid in different phases of a process. For instance, temperature-set points differ significantly during sterilization (heating, holding and cooling phase), cultivation, harvest, separation or extraction sequence. At least one storage cell should be reserved for an “emergency” value which must be dominantly and irreversibly (manual unlock only) selected in case of undesired process behavior (e.g. power fault or equipment malfunction) instead of just giving an alarm, e.g. simple RS-flipflops are very efficient to promote this strategy of forcing a process into a predetermined secure state. Simple algebraic calculations can be implemented. This allows derivation of decisive variables directly from auxiliary variables, e.g. the RQ value from exhaust gas analysis data. Also, simple models correlating key variables with indirect measures can be evaluated on-line (provided that correct parameters have been entered). Numerical integration and differentiation (with respect to time) should be possible on the front-end level. These methods are required to calculate either mass or energy balances (e.g. carbon recovery on-line) or to determine fluxes from integral measures (e.g. weight of the contents of a buffer vessel). If the formulae are not contained in the firmware the algorithms can be composed from simple mathematical and logical elementary function blocks. However, it is mandatory to store such compound functions as new, user defined blocks. A more flexible solution is to program the desired algorithm in a high-level language, compile it, load it to memory and execute it as a firmware component. This functionality is essential for the integrity of the concept of an individual front-end control system. It is often said that bioprocesses are “slow” processes with typical relaxation times in the range of minutes to days. This may hold true for some processes with biological systems that react slowly (small conversion rates) and that are not subject to rapid regulatory mechanisms (such as mass action law or enzyme inhibition; for detailed discussion of relaxation times, see Roels, 1983). In general, a biological system will respond in terms of seconds rather than minutes to environmental excitations, e.g. too high a temperature, an extreme pH value, oxygen limitation or

11

toxification, too high a shear stress or nutrient depletion. Further, the system is, of course, expected to handle very rapid physical control loops such as speed, flux or pressure controllers; at least on a small (laboratory and pilot) scale, these loops have typical relaxation times of seconds or even less. It is therefore essential that the DDC slave can be operated with a cycle time of less than 1 s realistically, typically 100 ms. Our experience proved that a minimal cycle time of 2.5 s was definitely impractical for a 30 1 bioreactor; however, a 200 to 1000 ms cycle time resulted in an overall satisfactory performance. The front-end component needs a man-machine interface that facilitates the operation of this compdnent. The supervisory component can play this role efficiently, however, a redundant operator’s console should allow a minimal amount of communication. This console will not normally be in use, but if so, it will be located very close to the process in an environment, not well protected. Therefore, some attention should be paid to the mechanical construction of the input devices: keys should be waterproof and large enough (also sufficiently widely spaced) to be operated with gloves; locator devices such as a light pen or a track ball are always helpful, mechanical mice or touch screens may not serve as well in a plant environment. Supervisory component This component has 3 major tasks that must cooperate perfectly (Fig. 4; see also Schneider, 1989): (1) check the user command and operating access authorization; (2) provide an efficient and comfortable (i.e. safe) man-machine interface; (3) provide computing and storage power to evaluate, sort, organize and store process, control and model data. Security and protection mechanisms are generally mandatory for biotechnological hardware (containments, laboratory equipment, exhaust and waste treatment). It is obvious that similar precautions must also be foreseen to guarantee process integrity on the logistic level. Depending on qualification, training, experience and function, different persons can be granted appropriate rights for operating, tuning, programming, data retrieval and the like by means of (multiple) passwords instead of mechanical keys. This increases flexibility in every day use and, in case of trouble, accelerates the first decisive emergency actions tremendously because these are now independent of the site: both information and operating commands may be exchanged via network connections or modem lines. Functional man-machine interface Well-organized presentation of primary and additional process information helps technicians, operators, engineers and managers. However, the contents of the messages and the ways to present them efficiently are considerably different. The same is, in analogy, true for the commands and data issued by these users. Graphical presentation is generally preferred because it is easy to comprehend, is less prone to verbal misinterpretation (of both technical terms and every day

;p:::::::::::::jg ~~

d

unldlrectional

-

bldirectlonal

dataflow

Fig. 4. Basic concept of distributed programs and shared memory areas on a supervisory station. Only data flow is indicated, not control interactions for inter-task communication and synchronization. Such a universal implementation with presently 10 to 12 programs and 6 to 8 common shared memory areas is running on 3 independent workstations at ETH. A communication driver program handles all requests and commands to. and responses and other messages from the front-end controller; since data transfer is always embedded in a distinct protocol, the driver has to translate transmitted data into “plain text” data and vice versa. Information to the slave component is taken from a sorted list of requests and commands, information from the slave is tested for plausibility and written to either the “plausible data” section or into another section to be treated further by special recovery programs. This driver runs with high priority and adds a time stamp to every data item. A data logger program transfers plausible data into a sorted circular memory section that holds the latest history of plausible data and places a pointer to the most recent data record. It is the only program with write access to this memory section and has also high priority to make new data rapidly available to other programs. Any user specific program, some examples are indicated, may retrieve recent data directly from the circular buffer (with unlimited read only access rights). A special program performs data reduction and writes selected data into the permanent database; this database manager is the only program with write access to the database. Although other programs may retrieve data from the database directly it is convenient to let the database manager handle also read requests for those programs. These programs may be required to communicate with each other; they might even run on external CPUs. To close the loop of data flow, a request and command handier sorts the information from controller programs, eventually adds information from the database and places the requests according to their priority in the outgoing list. There is a central supervisor program that organizes and schedules all the single programs but it does not directly participate in the handling of data.

phrases) and is readable from variable positions and distances (and not only from the operator’s chair). Color displays are extremely helpful since further information can be effectively encoded by color. However, translation of colors to their meanings is non-standard and confusing (if one compares the commercial or home-made products of different companies and institutions); an international standardization (general guide lines) would be helpful to improve comprehensiveness. Nevertheless, accurate information in numerical presentation is (eventually along with graphical) mandatory for some special duties such as initializing and tuning, or evaluating and

13

trouble-shooting. Printing hard copies (screen dumps, but not necessarily colored) is an absolute requirement for documentation and training purposes. Equally important is the command (language) interface. Items must be unequivocal. Small, dedicated menus with default choices are valuable. However, they require a very carefully prepared logistic concept and tedious programming because their use must never cause the process to destabilize. Therefore, defaults must be variable and depend on expert knowledge, less important is whether they are fixed in a program or calculated on-line. For the sake of security and the time required, the choices and other menus must be offered and traced by the program and only rarely by the user. Furthermore, the number and type of keys that must be pressed to trigger a desired action should be minimized and standardized to reduce the probability of errors. It may not be easy to find the best compromise between security and flexibility. Most of the decisive commands must be made subject to confirmation (i.e. corrections enabled). Depending on the user’s function, these menus, pictures and interaction tools may differ greatly, provide more or less information, or be locked. Scenes to be displayed live, with or without the facility to issue commands or change values interactively, include: (1) trend curves of floating variables; live or from historical data or mixed; automatic as well as individual scaling; including the possibility of superimposing historical or calculated data; (2) time diagrams of binary variables; (3) phase plane plots; (4) overview pictures displaying both the functional and physical state of: the entire process (multiple screens or windows); parts of a process (single screens; eventually zoom functions); single elementary groups and blocks including global and local parameters; (5) overviews showing step sequences including branching, continuation or termination conditions; (6) special loop pictures to promote routine manual procedures, e.g.: cleaning, testing and calibration; either standing alone or embedded in sequences; (7) history of alarms (issued and acknowledged) and response actions started. It is very convenient for engineers and managers to have more than one active window on the screen but operating personnel may prefer just one single picture at a time; then the way through the operator’s pictures tree must be made transparent (e.g. by pictograms or icons). We have had excellent experience with a window user interface (DEC windows on VAX stations). This is a very versatile and easy to learn interface; acceptance by differently trained users is high. However, it requires a large computer memory to be fully exploited at high speed. Data logging and the organization Since these procedures run continuously, i.e. usually unattended, they must be extremely robust and safe. The programs should be able to reinitialize data transfer in case of communication loss and must also be able to reactivate data transmission of the slave which may occasionally be suspended in case of high working load (of the slave); we have experienced this, unfortunately too often, on different front-end components.

14

When data are received, they must be marked with a time stamp and should be checked for plausibility. In order to make all these data available to many different programs, they are written into a shared common memory section. This is designed as a circular data buffer and provides sufficient space to hold the history of the last half day or more (we presently hold the last 50 h); older data are overwritten. The main advantage of this method is that the most interesting data can be retrieved deliberately, often at a very high speed. One program does copy the new data into the respective table spaces of a relational database system. There are programs to visualize the data, others use them for calculations or distribute them over the network; this, of course, requires a multi-tasking environment with perfect inter-task communication and synchronization tools (mechanisms involving event flags, priority control and hibernation/ wake techniques). It is not efficient to save all data although todays disks and tapes provide very high storage capacity. The major problem is to keep track of a huge amount of data and to retrieve historical data within a useful time. If this cannot be guaranteed, the data are practically lost. It is, therefore, a good decision to pretreat process data prior to long term storage, e.g. in the following way. The entire data set transferred to the supervisory station is always written into the above mentioned circular short-term storage buffer; all data are so far treated as variables. These data are further checked for their temporal changes: if they remain within a predefined window for a given period, only the first and the last value are transferred into the data base in order to reduce the amount for long-term storage. However, if any value starts to vary rapidly with time (i.e. permanently leaves the respective window) all its data items are saved with the respective sampling frequency until a next invariant period. This is not a predefined but an on-line distinction between actual process parameters and process variables. There is another shared common memory section that collects all requests for and commands to the slave .component issued by any program such as a model supervisor, pattern recognizer, trajectory generator, state estimator or, generally, supervisory controller or expert system. It is again the communication driver program that handles all these pendencies on a sequential priority basis. It may happen that even a powerful workstation does not have sufficient computing capacity to calculate some (user specific) programs without changing the quasi real-time characteristics of the station. In a cluster environment, however, such tasks can be performed on more powerful machines (e.g. the boot or resource server or another specialized cluster member) with the local station used just as an input/output interface; transfer of even large packets of data over the network is not expected to limit the performance. The coordinating

computer

Possibly the most important feature - at least for larger facilities with many process units installed - is to hold the operating system, libraries and application software: all the supervisory stations are connected as satellites to this boot and

15

resource server via a cluster network. The larger such an installation grows, the more effective is this concept. Investment costs decrease, maintenance demands for licenses, software and computer operating also decrease significantly; user-specific programs have to be developed, tested and installed only once which greatly improves their robustness and safety. System and user backup can be done from one central site with maximal efficiency. The network allows simple but safe distribm’ion of “confidential” information to the appropriate receiver (station/process controller); this is of great importance if non-dedicated (i.e. shared or multiplexed) intelligent subsystems (complex analytical stations) are implemented in the installation. For the sake of the entire systems integrity, a second, redundant boot server should be installed in this cluster. However, placed in a different location, in case of a system crash, the cluster can be rebooted from this machine within an acceptable time and normal operation resumed since data table spaces are written to local disks. This component has some key functions within the cluster. For instance, there must be only one single time basis for the entire system; this computer has to synchronize all others periodically (e.g. every 6 h). It coordinates especially the correct routing of data streams to and from single process units, e.g. from a shared analytical subsystem to the actually analyzed process units. This requires a small, special database to keep track of addresses and time delays of samples and signals. Although not realized in our present implementation, we are planning to centralize the recipe management on this level using the database system. This will greatly improve the overview and maintenance of recipes. Other tasks are not time critical, so most of the backup operations can be placed in batch job queues and executed either during standby operation of the physical (bio)processes or during steady-state periods when the amount of data to be stored is low. Furthermore, program development, re-sorting of historical data or formatting high quality output should be restricted to this computer because its real-time demands are lowest in the described environment. Provided this computer is very powerful it may substitute some of the satellite computing resources or, at least, function as a routing node to exchange data and results with (up to) super computers. Certainly, this computer has to serve as a source of generally required data. It must hold a collection of process recipes and make them available to any authorized user (managers, scientists, engineers, operators). Typical trajectories and patterns of “perfect” process behavior should there be compiled as well as characteristic numeric key values (organisms and chemical parameters). In a larger, distributed environment, this computer is also the proper element to hold analytical reference and library data, e.g. up-to-date IR-, GC-, LC-, MS- or NMR spectra. Satellite subsystems should pose their request for library searches to a powerful server instead of inefficiently wasting space and time on their own resources. Engineers and managers may wish to keep track of the single machines load, the maintenance schedule or the like; it is again this coordinating component that should collect and statistically analyze this various information. Last but not least, this computer may effectively enhance human coordination and information exchange via electronic mail.

16

Achievements and benefits In fact, biotechnological product development requires a longer time than the development of products in other areas, say, in electronics. Consequently, biotechnological research and development must be designed to plan further ahead and mu% be prepared to implement current and near-future insights, tools and techniques; in other words, a high degree of flexibility is mandatory for the concepts and the equipment in bioprocess engineering. The tremendous progress and experience in other fields (electronics, informatics or physical sciences) can and must be also exploited in biotechnology (Wooley, 1989). Automation of bioprocesses reduces the load of boring tasks and jobs from the personnel, thus increasing motivation and decreasing the fatal factor of human insufficiency. Thus, safety of process, product and environment is significantly improved and the value of either scientific investigations or of commercialized products is enhanced. The significant achievement of process automation is the increased precision of process control over an extended period. This ensures a high quality. On the other hand, it promotes decisive progress in bioprocess analytics based on actual measured values. Accuracy and speed of measurements have been improved and can be expected to be even better in the future. The outstanding character of a discipline is essentially based on know-how. In biotechnology, this is experimental process data and expert knowledge. This treasure has to be well organized and must be accessible to be exploited. Therefore, the core of a functional biotechnological unit is a carefully maintained and diligently updated data- and rulebase. A very flexible physical and logistical structure is a sound basis on which to realize these objectives. The structure must be easy to extend in order to keep up with current progress in analytics, electronics and informatics. Modern software installed on efficient operating systems (real-time, multi-tasking, multi-user) allows the achievement of a maximum of comfort and efficiency. Window-based man-machine interfaces guarantee a high user acceptance; graphical representation reduces the probability of misinterpretation; robustness of performance and low investments for personnel training benefit from both factors. Network or cluster operation minimizes the necessary efforts for maintenance and upgrade procedures and facilitates the essential backup tasks. It is not wise to save money at the wrong place. We have used these concepts for 4 distinct installations at the laboratory scale (Lecher et al., 1991 and Sonnleitner et al., in preparation).

References Lecher, G., Sonnleitner, practical experiences.

B. and Fiechter, A. (1991) J. Biotechnol. (accepted).

Automatic

bioprocess

control.

2. Implementation

and

17 Meyer, C. and Beyeler, W. (1984) Control strategies for continuous bioprocesses based on biological activities. Biotechnol. Bioeng. 26, 916-925. Roels, J.A. (1983) Energetics and Kinetics in Biotechnology. Elsevier Science Publishers, Amsterdam. Schneider, K. (1987) Computeranwendung in der Biotechnologie. Swiss Biotechnol. 5, 13-21. Schneider, K. (1989) CAROLINE I: a new DDC-system for biotechnical process control. Experientia 45, 1030-1034. Schtigerl, K. (1988) Measurement and bioreactor control. Proc. 8th Int. Biotechnol. Symp., Durand, G., Bobichon, L. and Florent, J. (Eds.) 1, 547-562. Sonnleitner, B. and Fiechter, A. (1989) Process control in chemostat experiments. GBF Monogr. 13, 75-83. Wooley. J.C. (1989) Computational biology for biotechnology: part II. TIBTECH 7. 126-132.