- Email: [email protected]
Coupling a machine vision sensor and a neural net supervised controller: controlling microbial cultivations
journal of
Motechndogy ELSEVIER
Journal
of Biotechnology
38 (1995) 219-228
Coupling a machine vision sensor and a neural net supervised controller: controlling microbial cultivations Qin Zhang a, J. Bruce Litchfield b,*, John F. Reid b, Jinliang Ren b, Shiuan-Wu Chang b aAutomated Analysis Corporation, 4516 North Sterling Avenue, Peoria, IL 61614, USA b Department of Agricultural Engineering, Uniuersity of Illinois at Urbana-Champaign, 1304 West Penmyiuania Avenue, Urbana, IL 61801, USA Received
1 April 1994; accepted
10 October
1994
Abstract This paper describes the application of a machine vision sensor in a neural network-based supervisory control system for a microbial cultivation. The vision sensor was capable of counting the number of microbial cells and acquiring other microbial growth information like sporulation activities from the sample medium. The information was useful for classifying the state of the microbial cultivation and could be used for control and determination of nutrient addition. A neural network supervisory unit was used to tune PID controllers in order to obtain an optimal cellular growth environment throughout the process. The process was simulated with a neural network-based program and tested with a bench scale reactor. Promising results were obtained from both the computer simulation and experimental tests. The results suggest that application of machine vision sensors and neural network-based supervisory controls are attractive for microbial cultivations. Keywords:
Microbial cultivation control; Neural network; Fuzzy logic; Machine vision
1. Introduction Increasing microbial growth rate, maintaining consistent production, improving process yield, and reducing production costs are some of the most desired goals for microbial cultivations. The development of reliable automatic control systems for microbial cultivations is a promising approach to attain those goals.
* Corresponding
author.
016%1656/95/$09.50 0 1995 Elsevier SSDI 0168-1656(94)00123-5
Science
Since living systems are involved, microbial cultivation is highly dynamic, and also involves uncertainty. Consequently, there are difficulties in developing reliable automatic control systems for microbial cultivation, including: (1) the lack of reliable mathematical models for describing those bioprocesses precisely (Shimizu and Morisue, 1989), (2) the lack of appropriate on-line sensors for detecting cellular growth information and the state of the bioprocess (Williams, 1990), and (3) the lack of understanding regarding the relationships between the controlled process variables and the process output (Rolf and Lim, 1985).
B.V. All rights resewed
Q. Zhang et ul. / Journul of Bwtechnology 38 (1995) 219-228
220
Conventional control systems for microbial cultivation rely on mathematical models of the process for development of control algorithms. With this approach, a control system acquires measurable process variables and decides how to adjust the manipulated variables to achieve the performance objectives based on the process model. Due to the limitations of available sensors, the measurable variables are usually indirect indicators of the state of the cultivation, and often include variables such as reactor temperature, substrate concentration, and pH (Fordyce et al., 1990). Use of such indirect process variables will result in some uncertainties in estimating the real state of the cultivation process. The mathematical model can not fully account for the uncertainty and diversity that occur in the bioprocess. Conventional control technologies, therefore, have an inherent weakness in handling complex living systems. In the past few years, two technologies which hold promise for this application have emerged in the automation of complex industrial processes. Machine vision sensors are capable of providing direct visual information from the process. Neu-
Digital Measurement and Control Unl
ral network-based devices can often make excellent control decisions in situations where accurate mathematical methods do not exist, but historic process data do. By combining machine vision and neural networks technologies, the control system is capable of accomplishing improved control performance based on direct visual process information and historic process data. Such a system provides a possible solution for addressing the difficulties in controlling complex microbial cultivations. The primary purpose of this research was to investigate the feasibility of applying a machine vision/neural network-based supervisory control system to microbial cultivations. This was accomplished through (1) the development of a prototype control system, (2) the analysis of image information acquired from cultivations using neural network and fuzzy logic technology, and (3) the simulation and implementation of the control system with a laboratory bioreactor. The objective of the control system was to regulate the microbial cultivation at the optimal condition throughout the process to achieve the highest possible microbial yields. While this objective is impre-
--__ ---__ _----__.. -_- -____-__ I _-.______.__..__-.___________-._-_____.__. II
_.__--____-
Sampling Tub
m0noc0upb3 PI-IProba r%XRob Water Jack* Water to Jackat
Laboratory Bioreactor
Fig. 1. Neural
network
supervisory
control
system with a machine
vision sensing system for microbial
cultivation.
Q. Zhang et al. /Journal
of Biotechnology 38 (1995) 219-228
Cl
cisely stated by traditional standards, part of the power of the fuzzy logic control technique is its ability to handle imprecise and subjective goals.
2. Development
C)
of a prototype system
The prototype system consisted of a bench scale bioreactor (BIOSTAT MD, B. Braun Biotech Inc. ‘), a digital measurement and control unit (DMCU), a machine vision sensing system (MVSS), and a neural network-based supervisory unit (NNSU). Fig. 1 shows how these elements were integrated in this system. 2.1. Bioreactor control unit
221
and
associated
C) Cl Fig. 2. Architecture of the neural network simulator with real input pattern and qualitative output pattern.
measurement /
The laboratory bioreactor had a working volume of 2.0 liters. An agitator, driven by a dc motor, was installed in the center of the vessel. The dissolved oxygen level in the medium was measured by a p0, probe, and controlled by adjusting the agitating speed of the stirrer. The process temperature was monitored by a thermocouple, and regulated by changing the temperature of water through a glass jacket. The pH of the medium was detected by a pH probe, and adjusted by supplying either acid or alkali through two peristaltic pumps. The foam level inside the vessel was checked by a foam sensor, and controlled by pumping antifoam solution into the vessel as needed. The digital measurement and control unit (DMCU) was a stand-alone control system. A software interface offered all functions necessary for automation including: data acquisition, PID control loops, and alarm monitoring. The data acquisition system was capable of acquiring and handling process variables through traditional
1 Trade names used in this publication are solely for the purpose of providing specific information. Mention of a trade name, proprietary product, or specific equipment does not constitute a guarantee or warranty by the University of Illinois and does not imply the approval of the named product to the exclusion of other products that may be suitable.
sensors such as a p0, probe, a pH probe, and a thermocouple. The PID controllers were used to regulate the process by adjusting stirrer rate, jacket water temperature, acid/alkali supply, and antifoam solution feeding based on corresponding control algorithms. The DMCU was capable of adapting set-points of the PID controllers either from local human-machine interface (key board) or from the supervisory control unit. All the set-points and related measured values could be displayed on a monitor. 2.2. Neural network supervisory unit The neural network supervisory unit (NNSU), hosted in a Sun SPARCstation 1 + + workstation, consisted of a neural network simulator and a supervisory controller. This unit supervised the DMCU which regulated the bioreactor operation according to an initial schedule at start up. After the start up, the NNSU requested operational information from a human operator, the DMCU, and the MVSS. The NNSU performed process simulation and adjusted set-points for the standby PID controllers in the DMCU at one hour intervals. Based on the results obtained from previous studies (Zhang et al., 19941, the neural network simulator consisted of a four-layer backpropagation neural network (Fig. 2) with six inputs of process information, and nine outputs of pre-
222
Q. Zhang et 01. /Jourrml
o/Biotechmlogy
dieted levels of ceil counts. The process information inputs included: (1) whether the cultivation inoculum was vegetation cells or spores (the only binary-valued process input), (2) accumulated process time (up to 10 h), (3) temperature (ranged from 26 to 32”C), (4) pH (ranged from 6.0 to 8.01, (5) current cell count per image from the medium sample (ranged from 0 to 26 cells per image), and (6) the cell count increase since last sampling. A total of 28 sets of historic data were used as the training set. The outputs of the neural network were the weights of each predicted grade of image cell counts. These outputs contained some degree of uncertainty. As a mathematical tool for handling vague information, fuzzy logic (Zadeh, 1976) can be used to interpret these uncertain outputs from the network. The predicted cell counts for the next sampling time were therefore presented in the form of fuzzy qualitative values of Ll to L9. The domains for those fuzzy values were defined so that Ll contained one or less cells in each image, L2 contained zero to two cells, L3 contained one to five cells, L4 contained two to eight cells, L.5 contained five to 13 cells, L6 contained eight to 18 cells, L7 contained 13 to 24 cells, L8 contained 18 to 30 cells, and L9 contained more than 24 cells. The degree of overlap for domain was indicated by fuzzy memberships (Zhang et al., 1994). The neural network was trained off line, using 18 sets of historic process data as the training set. The simulator was used to perform an optimization, searching for the optimal setpoints for the PID controller at l-h intervals. The supervisory controller (Fig. 3) coordinated the operation of all subsystems: (1) informing the operator to take samples from the bioreactor, (2) starting the MVSS to check the cell counts from the sample and sending the information to the neural network simulator, (3) driving the DMCU to measure other process variables and to deliver the information to the neural network simulator, (4) prompting the simulator to search for optimal set-points for the PID controllers, and (5) modifying the set-points for the PID controllers so those controllers could regulate the process at optimal conditions. The communications between NNSU and DMCU were through an RS-232 serial line.
38 (1995) 219-22X
~pi!F
0
NeWal-net
optimization
initialization
Fig. 3. Primary functions performed
by the supervisory control
unit.
The MVSS was connected to the NNSU directly through a VME bus extension connector. A graphical interface on the NNSU provided information about the status of the NNSU and of the other subsystems. 2.3. Machine u&ion sensing system The machine vision sensing system (MVSS) consisted of (1) a microscope, (2) a monochrome video camera, and (3) an image processing sys-
pzT
& Vision system initialization
Fig.
4. The
operations
sensing system.
performed
by the
machine
vision
Q. Zhang et al. /Journal of Biotechnology 38 (1995) 219-228
tern. The microscope (OLYMPUS BHZRFCA, with an adapter for mounting a video camera) was equipped for phase contrast microscopy, which provided the greatest amount of contrast between objects and background for inspecting cellular information in the medium. A Model
223
CCD-72 monochrome video camera (DAGEMTI, Inc.) was used for image acquisition. The video signal generated by the camera was sent to the image processing system and pre-conditioned by a video control box which optimized the contrast, either automatically or manually. The spa-
Fig. 5. Images acquired from the sample medium collected from the B. thuringiensis cultivation at hour 6.
Q. Zhang et al. /Journal ofBiotechnology 38 (I 9951 219-228
224
tial resolution of the camera system was 512 x 480 picture elements (pixels) and the gray level resolution was eight bits (256 gray levels). The image processing system was selected for its fast image acquisition rate and processing speed. Modules in the system were DIGIMAX, MAX-GRAPH, MAX-MUX, MAX-SP, ROISTORE and APA-512. DIGIMAX is an A/D and D/A conversion module which accepts standard video signal, digitizes and displays images in real-time. MAX-GRAPG is a graphics generator and display module which enables various graphics functions. MAX-MUX is a multiplexer with an onboard look-up table which allows programmable control of the source and destination of input and output data. MAX-SP is a signal processing module which allows real-time application of temporal and spatial filters, image merging, image subtraction and addition, and maximum/minimum processing. ROI-STORE is a 16-bit deep image buffer module which provides two megabytes of region-of-interest (ROD video memory. APAis a preprocessing module (Vision Systems International, Ltd.) which provides real-time feature analysis on a full image. All MaxVideo modules operated at the speed of 10 MHz, and the image scanning rate was up to 33 ms per frame. This system (Inovision, Inc.)
was installed in a separate box and linked to the Sun SPARCstation with a VME bus extension connector (Ren, 1992). The machine vision system was used to detect cellular properties from the medium during the cultivation. The operation of the system followed the sequence: (1) acquire cell images from medium sample, (2) execute appropriate vision algorithms for performing image feature enhancement, (31 perform image feature extraction and classification, and (4) feed information about the medium to the neural simulator in the supervisory unit (Fig. 4). The vision system software was developed to threshold the image to create binary images of the objects. Feature extraction and classification algorithms were used to categorize objects as (a) vegetable cells, (b) spores, (cl crystals, or (d) debris, based on the shape and the brightness of the object. The number of vegetable cell per image was related to the cell count and the number of spores and crystals per image were related to the yield of the process.
3. Analysis of image information The analysis of image information consisted of (1) the determination of the number of images
Table 1 The cell count of the sequential images and the mean cell count over the number of images at selected sampling time Sequent of the image
1 2 3 4 5 6 7 8 9 10 11 12 I3 14 15 I6
Sample of hour 6
Sample of hour 4 cell count
mean value
standard deviation
cell count
mean value
standard deviation
2 10 4 Y Y 13 8 14
2.0 6.0 5.3 6.3 6.X 7.8 7.9 8.6 7.8 7.9 8.0 7.8 7.0 7.6 7.7 7.6
_ 4.0 3.4 3.3 3.2 3.1 3.4 3.8 4.3 4.1 3.‘) 3.9 3.8 3.X 3.6 3.6
19 33 20 18 22 17 31 21 I3 23 20 14 24 15 26 21
19.0 26.0 24.0 22.5 22.4 21.5 22.9 22.6 21.6 21.7 21.5 20.9 21.2 20.7 21.1 21.1
_ 7.0 6.4 6.1 5.5 5.4 6.0
1 9 Y s IO 4 9 6
5.6
6.1 5.8 5.6 5.7 5.6 5.6 5.6
5.4
Q. Zhang et al. /Journal
of Biotechnology 38 (1995) 219-228
required at each sample time to obtain reliable cell counts and (2) the comparison of cell counts and optical density of the medium for use as process information in the supervisory controller. 3.1. Image number and cell count stability To acquire cellular growth information from the sample, several images of the sample were taken. Only those fully separable cells in the image were counted. The distribution of vegetative cells in the images was not always uniform, which caused difficulties in acquiring reliable cell concentration information. Fig. 5 shows four images from a sample medium collected from the reactor vessel at hour 6. As shown in these images, the number of cells per image varied widely even when they were acquired from the same sample. To obtain a reliable estimation of cell count from a sample, it was necessary to use the average number of cells of several images from that particular sample. Table 1 lists individual cell counts, average values, and standard deviation over 16 sequential images from samples collected at hours 4 and 6. As listed, cell counts per image randomly varied between 1 and 14 from the sample of hour 4 and between 14 and 33 from the sample of hour 6. The average values stabilized as the number of images analyzed increased. The standard deviations were quite stable around 3.6 for the sample of hour 4 and around 5.6 for the sample of hour 6 when at least five images were acquired. Similar results were obtained each time samples were collected. 3.2. Comparison of cell counts and optical density To train the neural network simulator, a large set of historic process data is required. On actual practice, it may be difficult to collect enough training data for neural network simulator training from prototype processes. Optical density is a traditional, indirect means of measuring cell concentration in bioreaction medium. Converting optical density data into cell count data for initial neural network simulator training purpose could
“n
-2
0 0
12
I
I
C&vat&
,
&nap
I
,
7
,
8
/
9
0
h
Fig. 6. Cell counts and optical density of samples collected during a B. thuringiensis cultivation.
reduce the cost and time of system development significantly. To determine the relationship between optical density and cell counts, the same samples were analyzed by using both means. The optical density of the sample was measured using a Spectronic 20 (Bausch and Lomb, USA) at 600 nm. The average cell count was obtained from 16 images of the same sample. The resulting relationship was almost linear in the lag and exponential phases. Fig. 6 shows optical density and cell counts vs. cultivation time in the lag and exponential phases of one cultivation of Bacillus thuringiensis. Linear regression showed that the average cell count and optical density of samples collected during the lag and exponential phases were linearly related W2 of 0.98). Cell counts = 0.3 + 2.08 X optical density
4. Simulation
and implementation
(1)
of the control
system
Simulation and implementation of the supervisory control system consisted of (1) on-line simulation of the microbial cultivation using the neural network-based simulator and (2) testing of the supervisory control system with the laboratory bioreactor. Part of the training data were the converted cell counts from optical densities according to Eq. 1. Thus, the implementation was limited to the lag and exponential phases.
Q. Zhang et al. /Journul
726
4.1. Simulation
of the
microbial
of Biotechnology38
cultication
The input information to perform the process simulation included: (1) the type of inoculum, (2) the accumulated process time, (3) the possible set-point range for temperature, (4) the possible set-point range for pH, (51 the cell count in the medium at last sample time, and (6) the increase in cell count since the previous sample. The outputs of the simulation were the optimal set-points for temperature and pH and the predicted cell count at the next sample time. The neural network simulator was used to select conditions to optimize the process, but some initial process conditions had to be provided to the simulator. For example, the initial process conditions for one of the control experiments included (1) cultivation inocula of B. thuringiensis spores, (2) accumulated process time of zero hours, (3) medium temperature of 30°C (4) medium pH of 7.0, and (5) initial cell concentration of 0.2 cells per image (mean value of ten images). The neural simulator predicted the cell count for the next sample time based on current process information and all possible future set-points. The set-points which predicted the highest cell count increase were selected as the optimal setpoints for that moment. After the process was begun, the neural network simulator used the actual acquired cell counts as input data. 4.2. Implementation tem
of the supenisory
(1995) 219-228
procedures 3 and 4 each hour until the end of the exponential phase of growth. Spores of B. thuringiensis were used as inocula in both experiments. The growth medium used was described by Chang (1993). Solutions of 2 N NaOH and 2 M H,PO, were used as the acid and alkali source for adjusting the pH. The dissolved oxygen concentration of the medium was controlled by a cascade control loop through adjusting the agitation rate while the air supply was constant at 1 wm. The initial set-points for this experiment were pH 7.0 and temperature 30°C. The dissolved oxygen concentration was kept at 50% throughout the experiments. The temperature, pH, dissolved oxygen level, and agitation rate were measured and controlled by the stand-by digital measurement and control unit (DMCU), which was supervised by the neural network supervisory unit (NNSU). To acquire cell count information, samples of the growth medium were collected from the reactor vessel hourly starting at hour zero. After collection, the sample was placed on a slide and shielded with a cover slip. Ten images from each sample were acquired and the mean number of vegetative cells was used as an indication of cell count. The neural simulator searched for the optimal set-points for temperature and pH at hour zero. The supervisory unit would adjust the PID controllers according to the optimal set-points, so the
control sys-
Two experiments were conducted. The purposes were to test the performance of the supervisory control system and to inspect the accuracy of the neural network simulator in the supervisory unit. The experimental procedures were to (1) fill and autoclave the growth medium in the bioreactor vessel, (2) inoculate and start up the control loop managed by the supervisory unit, (3) acquire cell count information from medium samples and perform on-line neural optimization to search for optimal set-points, (4) adjust set-points for temperature and pH controllers according to the predicted optimal set-points, and (5) repeat
Fig. 7. Predicted work supervisory
and observed cell counts control experiments.
from
neural
net-
Q. Zhang et al. /Journal
of Biotechnology 38 (1995) 219-228
227
were held constant. The cell counts from the PID controlled tests were converted from the measured optical density based on the method discussed in the previous section. The supervisory controller achieved higher peak cell counts in a shorter time than the PID controller in both experiments (Fig. 8). This demonstrated the advantage of the supervisory controller in that it was capable of adjusting the process to its optimal conditions throughout the operation. D’
0
1
2
3
4
,
e
7
CultivationTime, h
Fig. 8. Comparison of cell counts from experiments neural network supervisory control and fixed set-points control.
using PID
PID controllers regulated operating temperature and pH to the new levels. The predicted cell counts matched the observed ones well in the course of both experiments (Fig. 7). The first experiment started at controller set-points of 3O.O”C for temperature and 7.0 for pH. The neural network-based supervisory unit adjusted the set-points to 32°C and 6.0 at hour 1. As the number of cells increased, the temperature set-point was decreased to 30, 28, and 26°C at hours 2, 3, and 4, respectively. Meanwhile, the pH set-point was maintained at 6.0. After hour 4, during the exponential phase, the neural supervisory unit maintained the temperature set-point at 26.O”C and increased the pH set-point to 7.0, 7.5, and 8.0 at hours 5, 6, and 7, respectively, and then maintained pH 8.0 thereafter. As designed, this experiment was terminated at hour 8 when the process reached the end of the exponential phase. At this point, the cell number had reached its peak value according to the simulation result from the neural network simulator. The mean squares error between the predicted and the observed values was 1.0 cell per image with the largest error of 1.7 cell per image at hour 5. The second experiment was similar to the first, except the operation was terminated at hour 6 due to mechanical failure of the stirrer in the reactor vessel (Fig. 7). Two other PID control experiments were conducted in which temperature and pH set-points
5. Conclusions The application of a machine vision/neural network supervisory control system for microbial cultivation showed substantial potential. Using information captured by the machine vision system, the neural network simulator was capable of predicting microbial cultivation states reliably. Based on the predicted microbial cultivation behavior, the supervisory unit was capable of optimizing the set-points for the stand-by PID controllers. Experiments using this supervisory control achieved faster cell growth rates and higher product yields than experiments using conventional PID control only. So, the combination of machine vision sensing and neural network simulation are attractive for automatic control of pharmaceutical, biological, and food processes.
References Chang, SW. (1993) Studies of Growth Kinetics and Estimation of Oxygen Uptake Rate for Bacillus thuringiensis. PhD Thesis, University of Illinois at Urbana-Champaign, IL. Fordyce, A.P., Rawlings, J.B. and Edgar, T.F. (1990) Control Strategies for Fermentation Processes. In: D.R. Omstead, (Ed.), Computer Control of Fermentation Processes, CRC Press, Inc., Boca Raton, FL, pp. 165-206. Ren, J. (1992) Computer Control of Fermentation with a Vision-Based Sensing System. PhD Thesis, University of Illinois at Urbana-Champaign, IL. Rolf, M.J. and Lim, H.C. (1985) Systems for fermentation process control. In: M. Murry (Ed.), Comprehensive Biotechnology: the Principles, Applications, and Regulations of Biotechnology in Industry, Agriculture, and Machine. Pergamon Press, Oxford, UK, pp. 165-174.
228
Q. Zhang et al. /Journal
of Biotechnology 38 (1995) 219-228
Shimizu, K. and Morisue, T. (1989) Efficient control strategies for bioreactor systems. Proceedings of SICE’ 89, pp. 11591162. Williams, D. (1990) Problems of measurement and control in biotechnological processes. In: M.A. Winkler (Ed.). Chemical Engineering Problems in Biotechnology, Elsevier Applied Science, London, pp. 167-213.
Zadeh, L.A. (1976) The concept of a linguistic variable and its application to approximate reasoning - 1. Inform. Sci. 8, 199-249. Zhang, Q., Reid, J.F., Litchfield, J.B., Ren, J. and Chang, SW. (1994) A prototype neural network supervised control system for Bacillus thuringiensis fermentations. Biotechnol. Bioengin. 43, 483-489.
Motechndogy ELSEVIER
Journal
of Biotechnology
38 (1995) 219-228
Coupling a machine vision sensor and a neural net supervised controller: controlling microbial cultivations Qin Zhang a, J. Bruce Litchfield b,*, John F. Reid b, Jinliang Ren b, Shiuan-Wu Chang b aAutomated Analysis Corporation, 4516 North Sterling Avenue, Peoria, IL 61614, USA b Department of Agricultural Engineering, Uniuersity of Illinois at Urbana-Champaign, 1304 West Penmyiuania Avenue, Urbana, IL 61801, USA Received
1 April 1994; accepted
10 October
1994
Abstract This paper describes the application of a machine vision sensor in a neural network-based supervisory control system for a microbial cultivation. The vision sensor was capable of counting the number of microbial cells and acquiring other microbial growth information like sporulation activities from the sample medium. The information was useful for classifying the state of the microbial cultivation and could be used for control and determination of nutrient addition. A neural network supervisory unit was used to tune PID controllers in order to obtain an optimal cellular growth environment throughout the process. The process was simulated with a neural network-based program and tested with a bench scale reactor. Promising results were obtained from both the computer simulation and experimental tests. The results suggest that application of machine vision sensors and neural network-based supervisory controls are attractive for microbial cultivations. Keywords:
Microbial cultivation control; Neural network; Fuzzy logic; Machine vision
1. Introduction Increasing microbial growth rate, maintaining consistent production, improving process yield, and reducing production costs are some of the most desired goals for microbial cultivations. The development of reliable automatic control systems for microbial cultivations is a promising approach to attain those goals.
* Corresponding
author.
016%1656/95/$09.50 0 1995 Elsevier SSDI 0168-1656(94)00123-5
Science
Since living systems are involved, microbial cultivation is highly dynamic, and also involves uncertainty. Consequently, there are difficulties in developing reliable automatic control systems for microbial cultivation, including: (1) the lack of reliable mathematical models for describing those bioprocesses precisely (Shimizu and Morisue, 1989), (2) the lack of appropriate on-line sensors for detecting cellular growth information and the state of the bioprocess (Williams, 1990), and (3) the lack of understanding regarding the relationships between the controlled process variables and the process output (Rolf and Lim, 1985).
B.V. All rights resewed
Q. Zhang et ul. / Journul of Bwtechnology 38 (1995) 219-228
220
Conventional control systems for microbial cultivation rely on mathematical models of the process for development of control algorithms. With this approach, a control system acquires measurable process variables and decides how to adjust the manipulated variables to achieve the performance objectives based on the process model. Due to the limitations of available sensors, the measurable variables are usually indirect indicators of the state of the cultivation, and often include variables such as reactor temperature, substrate concentration, and pH (Fordyce et al., 1990). Use of such indirect process variables will result in some uncertainties in estimating the real state of the cultivation process. The mathematical model can not fully account for the uncertainty and diversity that occur in the bioprocess. Conventional control technologies, therefore, have an inherent weakness in handling complex living systems. In the past few years, two technologies which hold promise for this application have emerged in the automation of complex industrial processes. Machine vision sensors are capable of providing direct visual information from the process. Neu-
Digital Measurement and Control Unl
ral network-based devices can often make excellent control decisions in situations where accurate mathematical methods do not exist, but historic process data do. By combining machine vision and neural networks technologies, the control system is capable of accomplishing improved control performance based on direct visual process information and historic process data. Such a system provides a possible solution for addressing the difficulties in controlling complex microbial cultivations. The primary purpose of this research was to investigate the feasibility of applying a machine vision/neural network-based supervisory control system to microbial cultivations. This was accomplished through (1) the development of a prototype control system, (2) the analysis of image information acquired from cultivations using neural network and fuzzy logic technology, and (3) the simulation and implementation of the control system with a laboratory bioreactor. The objective of the control system was to regulate the microbial cultivation at the optimal condition throughout the process to achieve the highest possible microbial yields. While this objective is impre-
--__ ---__ _----__.. -_- -____-__ I _-.______.__..__-.___________-._-_____.__. II
_.__--____-
Sampling Tub
m0noc0upb3 PI-IProba r%XRob Water Jack* Water to Jackat
Laboratory Bioreactor
Fig. 1. Neural
network
supervisory
control
system with a machine
vision sensing system for microbial
cultivation.
Q. Zhang et al. /Journal
of Biotechnology 38 (1995) 219-228
Cl
cisely stated by traditional standards, part of the power of the fuzzy logic control technique is its ability to handle imprecise and subjective goals.
2. Development
C)
of a prototype system
The prototype system consisted of a bench scale bioreactor (BIOSTAT MD, B. Braun Biotech Inc. ‘), a digital measurement and control unit (DMCU), a machine vision sensing system (MVSS), and a neural network-based supervisory unit (NNSU). Fig. 1 shows how these elements were integrated in this system. 2.1. Bioreactor control unit
221
and
associated
C) Cl Fig. 2. Architecture of the neural network simulator with real input pattern and qualitative output pattern.
measurement /
The laboratory bioreactor had a working volume of 2.0 liters. An agitator, driven by a dc motor, was installed in the center of the vessel. The dissolved oxygen level in the medium was measured by a p0, probe, and controlled by adjusting the agitating speed of the stirrer. The process temperature was monitored by a thermocouple, and regulated by changing the temperature of water through a glass jacket. The pH of the medium was detected by a pH probe, and adjusted by supplying either acid or alkali through two peristaltic pumps. The foam level inside the vessel was checked by a foam sensor, and controlled by pumping antifoam solution into the vessel as needed. The digital measurement and control unit (DMCU) was a stand-alone control system. A software interface offered all functions necessary for automation including: data acquisition, PID control loops, and alarm monitoring. The data acquisition system was capable of acquiring and handling process variables through traditional
1 Trade names used in this publication are solely for the purpose of providing specific information. Mention of a trade name, proprietary product, or specific equipment does not constitute a guarantee or warranty by the University of Illinois and does not imply the approval of the named product to the exclusion of other products that may be suitable.
sensors such as a p0, probe, a pH probe, and a thermocouple. The PID controllers were used to regulate the process by adjusting stirrer rate, jacket water temperature, acid/alkali supply, and antifoam solution feeding based on corresponding control algorithms. The DMCU was capable of adapting set-points of the PID controllers either from local human-machine interface (key board) or from the supervisory control unit. All the set-points and related measured values could be displayed on a monitor. 2.2. Neural network supervisory unit The neural network supervisory unit (NNSU), hosted in a Sun SPARCstation 1 + + workstation, consisted of a neural network simulator and a supervisory controller. This unit supervised the DMCU which regulated the bioreactor operation according to an initial schedule at start up. After the start up, the NNSU requested operational information from a human operator, the DMCU, and the MVSS. The NNSU performed process simulation and adjusted set-points for the standby PID controllers in the DMCU at one hour intervals. Based on the results obtained from previous studies (Zhang et al., 19941, the neural network simulator consisted of a four-layer backpropagation neural network (Fig. 2) with six inputs of process information, and nine outputs of pre-
222
Q. Zhang et 01. /Jourrml
o/Biotechmlogy
dieted levels of ceil counts. The process information inputs included: (1) whether the cultivation inoculum was vegetation cells or spores (the only binary-valued process input), (2) accumulated process time (up to 10 h), (3) temperature (ranged from 26 to 32”C), (4) pH (ranged from 6.0 to 8.01, (5) current cell count per image from the medium sample (ranged from 0 to 26 cells per image), and (6) the cell count increase since last sampling. A total of 28 sets of historic data were used as the training set. The outputs of the neural network were the weights of each predicted grade of image cell counts. These outputs contained some degree of uncertainty. As a mathematical tool for handling vague information, fuzzy logic (Zadeh, 1976) can be used to interpret these uncertain outputs from the network. The predicted cell counts for the next sampling time were therefore presented in the form of fuzzy qualitative values of Ll to L9. The domains for those fuzzy values were defined so that Ll contained one or less cells in each image, L2 contained zero to two cells, L3 contained one to five cells, L4 contained two to eight cells, L.5 contained five to 13 cells, L6 contained eight to 18 cells, L7 contained 13 to 24 cells, L8 contained 18 to 30 cells, and L9 contained more than 24 cells. The degree of overlap for domain was indicated by fuzzy memberships (Zhang et al., 1994). The neural network was trained off line, using 18 sets of historic process data as the training set. The simulator was used to perform an optimization, searching for the optimal setpoints for the PID controller at l-h intervals. The supervisory controller (Fig. 3) coordinated the operation of all subsystems: (1) informing the operator to take samples from the bioreactor, (2) starting the MVSS to check the cell counts from the sample and sending the information to the neural network simulator, (3) driving the DMCU to measure other process variables and to deliver the information to the neural network simulator, (4) prompting the simulator to search for optimal set-points for the PID controllers, and (5) modifying the set-points for the PID controllers so those controllers could regulate the process at optimal conditions. The communications between NNSU and DMCU were through an RS-232 serial line.
38 (1995) 219-22X
~pi!F
0
NeWal-net
optimization
initialization
Fig. 3. Primary functions performed
by the supervisory control
unit.
The MVSS was connected to the NNSU directly through a VME bus extension connector. A graphical interface on the NNSU provided information about the status of the NNSU and of the other subsystems. 2.3. Machine u&ion sensing system The machine vision sensing system (MVSS) consisted of (1) a microscope, (2) a monochrome video camera, and (3) an image processing sys-
pzT
& Vision system initialization
Fig.
4. The
operations
sensing system.
performed
by the
machine
vision
Q. Zhang et al. /Journal of Biotechnology 38 (1995) 219-228
tern. The microscope (OLYMPUS BHZRFCA, with an adapter for mounting a video camera) was equipped for phase contrast microscopy, which provided the greatest amount of contrast between objects and background for inspecting cellular information in the medium. A Model
223
CCD-72 monochrome video camera (DAGEMTI, Inc.) was used for image acquisition. The video signal generated by the camera was sent to the image processing system and pre-conditioned by a video control box which optimized the contrast, either automatically or manually. The spa-
Fig. 5. Images acquired from the sample medium collected from the B. thuringiensis cultivation at hour 6.
Q. Zhang et al. /Journal ofBiotechnology 38 (I 9951 219-228
224
tial resolution of the camera system was 512 x 480 picture elements (pixels) and the gray level resolution was eight bits (256 gray levels). The image processing system was selected for its fast image acquisition rate and processing speed. Modules in the system were DIGIMAX, MAX-GRAPH, MAX-MUX, MAX-SP, ROISTORE and APA-512. DIGIMAX is an A/D and D/A conversion module which accepts standard video signal, digitizes and displays images in real-time. MAX-GRAPG is a graphics generator and display module which enables various graphics functions. MAX-MUX is a multiplexer with an onboard look-up table which allows programmable control of the source and destination of input and output data. MAX-SP is a signal processing module which allows real-time application of temporal and spatial filters, image merging, image subtraction and addition, and maximum/minimum processing. ROI-STORE is a 16-bit deep image buffer module which provides two megabytes of region-of-interest (ROD video memory. APAis a preprocessing module (Vision Systems International, Ltd.) which provides real-time feature analysis on a full image. All MaxVideo modules operated at the speed of 10 MHz, and the image scanning rate was up to 33 ms per frame. This system (Inovision, Inc.)
was installed in a separate box and linked to the Sun SPARCstation with a VME bus extension connector (Ren, 1992). The machine vision system was used to detect cellular properties from the medium during the cultivation. The operation of the system followed the sequence: (1) acquire cell images from medium sample, (2) execute appropriate vision algorithms for performing image feature enhancement, (31 perform image feature extraction and classification, and (4) feed information about the medium to the neural simulator in the supervisory unit (Fig. 4). The vision system software was developed to threshold the image to create binary images of the objects. Feature extraction and classification algorithms were used to categorize objects as (a) vegetable cells, (b) spores, (cl crystals, or (d) debris, based on the shape and the brightness of the object. The number of vegetable cell per image was related to the cell count and the number of spores and crystals per image were related to the yield of the process.
3. Analysis of image information The analysis of image information consisted of (1) the determination of the number of images
Table 1 The cell count of the sequential images and the mean cell count over the number of images at selected sampling time Sequent of the image
1 2 3 4 5 6 7 8 9 10 11 12 I3 14 15 I6
Sample of hour 6
Sample of hour 4 cell count
mean value
standard deviation
cell count
mean value
standard deviation
2 10 4 Y Y 13 8 14
2.0 6.0 5.3 6.3 6.X 7.8 7.9 8.6 7.8 7.9 8.0 7.8 7.0 7.6 7.7 7.6
_ 4.0 3.4 3.3 3.2 3.1 3.4 3.8 4.3 4.1 3.‘) 3.9 3.8 3.X 3.6 3.6
19 33 20 18 22 17 31 21 I3 23 20 14 24 15 26 21
19.0 26.0 24.0 22.5 22.4 21.5 22.9 22.6 21.6 21.7 21.5 20.9 21.2 20.7 21.1 21.1
_ 7.0 6.4 6.1 5.5 5.4 6.0
1 9 Y s IO 4 9 6
5.6
6.1 5.8 5.6 5.7 5.6 5.6 5.6
5.4
Q. Zhang et al. /Journal
of Biotechnology 38 (1995) 219-228
required at each sample time to obtain reliable cell counts and (2) the comparison of cell counts and optical density of the medium for use as process information in the supervisory controller. 3.1. Image number and cell count stability To acquire cellular growth information from the sample, several images of the sample were taken. Only those fully separable cells in the image were counted. The distribution of vegetative cells in the images was not always uniform, which caused difficulties in acquiring reliable cell concentration information. Fig. 5 shows four images from a sample medium collected from the reactor vessel at hour 6. As shown in these images, the number of cells per image varied widely even when they were acquired from the same sample. To obtain a reliable estimation of cell count from a sample, it was necessary to use the average number of cells of several images from that particular sample. Table 1 lists individual cell counts, average values, and standard deviation over 16 sequential images from samples collected at hours 4 and 6. As listed, cell counts per image randomly varied between 1 and 14 from the sample of hour 4 and between 14 and 33 from the sample of hour 6. The average values stabilized as the number of images analyzed increased. The standard deviations were quite stable around 3.6 for the sample of hour 4 and around 5.6 for the sample of hour 6 when at least five images were acquired. Similar results were obtained each time samples were collected. 3.2. Comparison of cell counts and optical density To train the neural network simulator, a large set of historic process data is required. On actual practice, it may be difficult to collect enough training data for neural network simulator training from prototype processes. Optical density is a traditional, indirect means of measuring cell concentration in bioreaction medium. Converting optical density data into cell count data for initial neural network simulator training purpose could
“n
-2
0 0
12
I
I
C&vat&
,
&nap
I
,
7
,
8
/
9
0
h
Fig. 6. Cell counts and optical density of samples collected during a B. thuringiensis cultivation.
reduce the cost and time of system development significantly. To determine the relationship between optical density and cell counts, the same samples were analyzed by using both means. The optical density of the sample was measured using a Spectronic 20 (Bausch and Lomb, USA) at 600 nm. The average cell count was obtained from 16 images of the same sample. The resulting relationship was almost linear in the lag and exponential phases. Fig. 6 shows optical density and cell counts vs. cultivation time in the lag and exponential phases of one cultivation of Bacillus thuringiensis. Linear regression showed that the average cell count and optical density of samples collected during the lag and exponential phases were linearly related W2 of 0.98). Cell counts = 0.3 + 2.08 X optical density
4. Simulation
and implementation
(1)
of the control
system
Simulation and implementation of the supervisory control system consisted of (1) on-line simulation of the microbial cultivation using the neural network-based simulator and (2) testing of the supervisory control system with the laboratory bioreactor. Part of the training data were the converted cell counts from optical densities according to Eq. 1. Thus, the implementation was limited to the lag and exponential phases.
Q. Zhang et al. /Journul
726
4.1. Simulation
of the
microbial
of Biotechnology38
cultication
The input information to perform the process simulation included: (1) the type of inoculum, (2) the accumulated process time, (3) the possible set-point range for temperature, (4) the possible set-point range for pH, (51 the cell count in the medium at last sample time, and (6) the increase in cell count since the previous sample. The outputs of the simulation were the optimal set-points for temperature and pH and the predicted cell count at the next sample time. The neural network simulator was used to select conditions to optimize the process, but some initial process conditions had to be provided to the simulator. For example, the initial process conditions for one of the control experiments included (1) cultivation inocula of B. thuringiensis spores, (2) accumulated process time of zero hours, (3) medium temperature of 30°C (4) medium pH of 7.0, and (5) initial cell concentration of 0.2 cells per image (mean value of ten images). The neural simulator predicted the cell count for the next sample time based on current process information and all possible future set-points. The set-points which predicted the highest cell count increase were selected as the optimal setpoints for that moment. After the process was begun, the neural network simulator used the actual acquired cell counts as input data. 4.2. Implementation tem
of the supenisory
(1995) 219-228
procedures 3 and 4 each hour until the end of the exponential phase of growth. Spores of B. thuringiensis were used as inocula in both experiments. The growth medium used was described by Chang (1993). Solutions of 2 N NaOH and 2 M H,PO, were used as the acid and alkali source for adjusting the pH. The dissolved oxygen concentration of the medium was controlled by a cascade control loop through adjusting the agitation rate while the air supply was constant at 1 wm. The initial set-points for this experiment were pH 7.0 and temperature 30°C. The dissolved oxygen concentration was kept at 50% throughout the experiments. The temperature, pH, dissolved oxygen level, and agitation rate were measured and controlled by the stand-by digital measurement and control unit (DMCU), which was supervised by the neural network supervisory unit (NNSU). To acquire cell count information, samples of the growth medium were collected from the reactor vessel hourly starting at hour zero. After collection, the sample was placed on a slide and shielded with a cover slip. Ten images from each sample were acquired and the mean number of vegetative cells was used as an indication of cell count. The neural simulator searched for the optimal set-points for temperature and pH at hour zero. The supervisory unit would adjust the PID controllers according to the optimal set-points, so the
control sys-
Two experiments were conducted. The purposes were to test the performance of the supervisory control system and to inspect the accuracy of the neural network simulator in the supervisory unit. The experimental procedures were to (1) fill and autoclave the growth medium in the bioreactor vessel, (2) inoculate and start up the control loop managed by the supervisory unit, (3) acquire cell count information from medium samples and perform on-line neural optimization to search for optimal set-points, (4) adjust set-points for temperature and pH controllers according to the predicted optimal set-points, and (5) repeat
Fig. 7. Predicted work supervisory
and observed cell counts control experiments.
from
neural
net-
Q. Zhang et al. /Journal
of Biotechnology 38 (1995) 219-228
227
were held constant. The cell counts from the PID controlled tests were converted from the measured optical density based on the method discussed in the previous section. The supervisory controller achieved higher peak cell counts in a shorter time than the PID controller in both experiments (Fig. 8). This demonstrated the advantage of the supervisory controller in that it was capable of adjusting the process to its optimal conditions throughout the operation. D’
0
1
2
3
4
,
e
7
CultivationTime, h
Fig. 8. Comparison of cell counts from experiments neural network supervisory control and fixed set-points control.
using PID
PID controllers regulated operating temperature and pH to the new levels. The predicted cell counts matched the observed ones well in the course of both experiments (Fig. 7). The first experiment started at controller set-points of 3O.O”C for temperature and 7.0 for pH. The neural network-based supervisory unit adjusted the set-points to 32°C and 6.0 at hour 1. As the number of cells increased, the temperature set-point was decreased to 30, 28, and 26°C at hours 2, 3, and 4, respectively. Meanwhile, the pH set-point was maintained at 6.0. After hour 4, during the exponential phase, the neural supervisory unit maintained the temperature set-point at 26.O”C and increased the pH set-point to 7.0, 7.5, and 8.0 at hours 5, 6, and 7, respectively, and then maintained pH 8.0 thereafter. As designed, this experiment was terminated at hour 8 when the process reached the end of the exponential phase. At this point, the cell number had reached its peak value according to the simulation result from the neural network simulator. The mean squares error between the predicted and the observed values was 1.0 cell per image with the largest error of 1.7 cell per image at hour 5. The second experiment was similar to the first, except the operation was terminated at hour 6 due to mechanical failure of the stirrer in the reactor vessel (Fig. 7). Two other PID control experiments were conducted in which temperature and pH set-points
5. Conclusions The application of a machine vision/neural network supervisory control system for microbial cultivation showed substantial potential. Using information captured by the machine vision system, the neural network simulator was capable of predicting microbial cultivation states reliably. Based on the predicted microbial cultivation behavior, the supervisory unit was capable of optimizing the set-points for the stand-by PID controllers. Experiments using this supervisory control achieved faster cell growth rates and higher product yields than experiments using conventional PID control only. So, the combination of machine vision sensing and neural network simulation are attractive for automatic control of pharmaceutical, biological, and food processes.
References Chang, SW. (1993) Studies of Growth Kinetics and Estimation of Oxygen Uptake Rate for Bacillus thuringiensis. PhD Thesis, University of Illinois at Urbana-Champaign, IL. Fordyce, A.P., Rawlings, J.B. and Edgar, T.F. (1990) Control Strategies for Fermentation Processes. In: D.R. Omstead, (Ed.), Computer Control of Fermentation Processes, CRC Press, Inc., Boca Raton, FL, pp. 165-206. Ren, J. (1992) Computer Control of Fermentation with a Vision-Based Sensing System. PhD Thesis, University of Illinois at Urbana-Champaign, IL. Rolf, M.J. and Lim, H.C. (1985) Systems for fermentation process control. In: M. Murry (Ed.), Comprehensive Biotechnology: the Principles, Applications, and Regulations of Biotechnology in Industry, Agriculture, and Machine. Pergamon Press, Oxford, UK, pp. 165-174.
228
Q. Zhang et al. /Journal
of Biotechnology 38 (1995) 219-228
Shimizu, K. and Morisue, T. (1989) Efficient control strategies for bioreactor systems. Proceedings of SICE’ 89, pp. 11591162. Williams, D. (1990) Problems of measurement and control in biotechnological processes. In: M.A. Winkler (Ed.). Chemical Engineering Problems in Biotechnology, Elsevier Applied Science, London, pp. 167-213.
Zadeh, L.A. (1976) The concept of a linguistic variable and its application to approximate reasoning - 1. Inform. Sci. 8, 199-249. Zhang, Q., Reid, J.F., Litchfield, J.B., Ren, J. and Chang, SW. (1994) A prototype neural network supervised control system for Bacillus thuringiensis fermentations. Biotechnol. Bioengin. 43, 483-489.