EJOR special issue on stochastic control theory and operational research

European Journal of Operational Research 73 (1994) 205-208 North-Holland

205

Editorial

EJOR special issue on stochastic control theory and operational research This special issue originated from an invitation by C. Bernhard Tilanus to guest-edit a special or feature issue of E J O R on stochastic control and operational research. A call for papers was released in 1990 to leading specialists in this field as well as to the O R scientific community. All papers received were evaluated using the E J O R standard review process. Earlier versions of some of the papers were also presented at a topical session at E U R O XI, the l l t h European Congress on Operational Research, in Aachen, Germany, July 16-19, 1991. The thorough review process, which involved at least two referees for each paper, was completed late in 1992. As a result, one invited review and 13 research contributions are gathered in the present special issue. The primary objective of this special issue is to publish significant research involving applications of stochastic control theory to operations research, management science and economics. 'Stochastic control theory' is understood in a rather broad sense, referring to stochastic dynamic optimization problems and qualitative properties of stochastic dynamic systems both in discrete and in continuous time (but excluding Markov decision processes with a finite state space, which have largely developed as a separate field of research). Contributions from different areas of operational research and from economics are included, as are algorithmic and theoretical papers, including purely mathematical ones which were judged to be of potential interest to operational research. Thus, the present special issue should provide an introduction to this field as well as a sample of ongoing research, thereby contributing to an improved communication between mathe-

maticians and operational researchers interested in stochastic control theory. The issue opens with a review of stochastic control theory and its applications by Charles S. Tapiero. He introduces the main elements required to formulate, analyze and apply stochastic control theory. An overview of analytical solution techniques (especially stochastic dynamic programming and stochastic maximum principles) and of numerical techniques (discretization techniques, Markov decision process formulations for stochastic control problems, perturbation and Monte Carlo techniques, among others) is provided. Hints are also given to the development of expert systems for stochastic control problems. Two examples which draw upon the author's own research illustrate the solution of actual stochastic control problems in quality production and option trading. One of the prototypes in stochastic control modelling has been the L Q G problem, i.e., the optimization of a quadratic objective function subject to a linear dynamic system disturbed by Gaussian random processes. Tamer Ba~ar and Rajesh Bansal start from the familiar continuous-time L Q G feedback controller design problem with noisy observations of the state variables, but extend it to the case where measurement is not fixed but is determined to some extent by the decision-maker. This problem is especially important if there are more decision-makers communicating through noisy information channels. Using results from information theory, Ba~ar and Bansal derive optimal solutions (control and measurement strategies) for the scalar case, where they provide also numerical examples, and for multidi-

0377-2217/94/$07.00 © 1994 - Elsevier Science B.V. All rights reserved SSDI 0 3 7 7 - 2 2 1 7 ( 9 3 ) E 0 0 9 3 - D

206

R. Neck / Editorial

mensional problems under linear measurement schemes. Linear feedback designs turn out to be optimal in the scalar but not necessarily in the vector version of their model. Piecewise deterministic control problems for continuous-time linear systems with bounded state space are the subject of the paper by A. Haurie, A. Leizarowitz and Ch. van Delft. Here the system changes its mode at random times, but remains deterministic between. These problems lead to piecewise open-loop controls. For a class of problems with an undiscounted objective function to be optimized over an infinite planning horizon, where the disturbances are modelled as jump Markov processes, they characterize boundedly optimal controls (an optimality concept that is stronger than more familiar average cost optimality) by using dynamic programming. An application of the results to a manufacturing control problem is also provided, for which a turnpike property is shown to hold. N. Christopeit considers a one-dimensional nonlinear stochastic differential equation arising as a filter equation in the control of linear diffusion processes with an unobservable additive drift term. The asymptotic behavior of this system is investigated, and the results are applied to derive adaptive feedback controls over finite and infinite planning horizons. Although this is a highly mathematical paper, the author also gives some hints on how to generalize the model used to those which allow applications to real-life problems. It has been recognized long ago that there are duality relations between control and estimation of stochastic dynamic systems. Under some special circumstances, these two tasks can be separated; similar mathematical tools can be applied to their solution, anyway. Masanao Aoki's paper concentrates on the problem of estimating economic models in state space innovation representation from time series data, using an asymptotically efficient instrumental variable estimator originally adapted from a system identification algorithm in realization theory. Forward and backward state space innovation models are used jointly to estimate the system parameters of the same state space model. This paper illustrates how ideas from dynamic systems and control theory can be used to obtain insights into problems of time-series analysis. Quite a different application of tools from

stochastic control theory is the topic of the paper by Karl Hinderer and Michael Stieglitz, namely polychotomous search problems. These are stochastic sequential search problems where the current search region is decomposed into a finite number of parts upon which the searcher obtains information where the object he looks for is hidden. Drawing on the authors' research, they show that minimal expected search costs can be computed by using dynamic programming and derive mathematical results characterizing optimal search rules. Special examples of the very general setup of this paper's model arise in optimal coding, the detection of failures in communication systems, the search for a leak in a pipeline, the time-minimal rescue of people buried in a collapsed tunnel, or the search for a mineral deposit, among others. An area where much research has been done during the last few years is the extension of stochastic control theory to include risk-sensitivity, a concept which has originally been developed in the economics literature. One of the leading contributors to this field, P. Whittle, in his paper gives an introductory survey of his own and related work on risk-sensitive optimum stochastic control using an exponential objective function. Whittle states a risk-sensitive certaintyequivalence principle for L Q G problems and a risk-sensitive maximum principle, generalizing results from the risk-neutral case. Non-LQG problems can be trated by using the approach of large-deviation theory. The paper also presents some new applications of these results to economic problems, namely monopoly with sticky prices and optimization of consumption over the consumer's lifetime. A different concept of sensitivity is explored in the paper by Ber t Rustem, namely robustness under uncertainty for nonlinear systems, where for a special structure of the model (particularly, LQ problems) a mean-variance interpretation of sensitivity is possible. Parameterized feedback control rules are applied to a static transcription of a dynamic discrete-time system, which allows the use of iterative nonlinear optimization algorithms without necessitating the computational difficulties associated with nonlinear adaptive dynamic programming. Robustness means introducing an additional penalty term in the objective function such that the sensitivity of the solution

R. Neck / Editorial

to perturbations is optimized simultaneously with the original objective function. A local approximation to the robust optimal solution is provided for a sufficiently large neighborhood. The nonlinearities are treated by introducing a bias term, which takes account of deviations from certainty equivalence and is evaluated by Monte Carlo simulations, in calculating the expected values of the approximately optimal state vector. Robustness with respect to either the objective function or the endogenous variables can be taken into account by the algorithms proposed. The paper also gives two numerical examples, one for a large econometric model of the UK economy, the other for a planning model of the West European petrochemical industry. Algorithmic questions of stochastic control are also the topic of the paper by Alfred L. Norman, focussing on a theoretical analysis of the computational complexity of solving discrete-time LQ control models used in economics. He examines four different statistical assumptions about the parameters of such models: (1) they are fixed and unknown; (2) their estimates are treated as fixed and known; (3) the parameter and covariance matrix estimates are treated as independent random coefficients; and (4) the 'active learning' approximation due to MacRae. Norman applies the information-based model of computational complexity to compare the computational costs of calculating optimal policies under the four schemes, depending on the planning horizon T and the number of states n. He shows that the computational complexity of (1) is transfinite; that of (2) is linear in T and has the same computational complexity in n as a matrix multiplication of two (n × n)-matrices (practically approximately n3T); the computational complexity of (3) is n4T; and that of (4) is n4T for each iteration. Formulations (2) and (3) are nested in (4), and (1) cannot be solved analytically. The results of this paper can be used in assessing which approximation scheme to use for calculating approximately optimal policies in a multivariable LQ model. The following papers contain examples of applications of stochastic control theory to models of operations research on the firm level. S. Lou, S.P. Sethi and Q. Zhang consider a production planning problem in a stochastic two-machine flowshop where machines can break down and need repairment. Non-negativity constraints are

207

imposed on the inventory of unfinished goods, and stochastics are introduced by assuming the machine capacities and demand processes to be finite-state Markov chains. The costs to be minimized by choosing the rate of production over an infinite planning horizon include production and inventory backlog costs. The solution to this problem is theoretically characterized using dynamic programming to arrive at optimal feedback controls. An intuitive interpretation of the optimal policies and a brief discussion of two numerical approaches to solve actual problems of this kind are provided, too. Randolph F.C. Shen in a numerical study applies the stochastic control methods of certainty equivalence, passive learning and active learning developed by Kendrick to the classical operational research problem of production planning due to Holt, Modigliani, Muth and Simon. Stochastics are present in the system equations, in the parameters and in the observations. Shen uses the computer program DUAL to derive numerical solutions which are compared between the desired values, the original solution of Holt et al., the optimal deterministic and the three stochastic solutions with respect to their state and control trajectories by means of statistical analysis. One conclusion he draws from this exercise states that passive learning gives the best performance (as measured by the objective function) among the stochastic methods. The paper by Uday S. Karmarkar and Jinsung Yoo considers a stochastic dynamic discrete-time product cycling problem, where several products are sequenced on a single machine to meet timevarying stochastic demands, with dynamic costs for inventories, backorders and production and changeover costs. The problem ist formulated as a stochastic dynamic programming problem with mixed continuous and integer variables and approximated by three alternative decompositions based on a 'restricted' Lagrangean method. One of them is used for the actual solution; some numerical results for small problems of this class are given, deriving bounds for the optimal solution. Peter M. Kort generalizes a stochastic dynamic model of the firm with bankruptcy due to Bensoussan and Lesourne to include adjustment costs. He applies dynamic programming to derive analytically optimal investment, saving, and dividend

208

R. Neck / Editorial

policies for various problems within this framework, differing with respect to the expected earnings function, the adjustment cost function and the parameters of the model. The results show that, in contrast to the original model of Bensoussan and Lesourne, mixed policies involving investment and dividend or investment and saving policies may be optimal in this extension of the model. The final paper by Reinhard Neck and Josef Matulka describes their algorithm OPTCON, which determines approximately optimal numerical solutions for multivariable stochastic control problems with a quadratic objective function and a nonlinear dynamic macroeconometric model, where there are additive and multiplicative (random parameters) stochastic disturbances. The paper reports about some applications of OPTCON as implemented in a GAUSS program to two small econometric models of the Austrian economy, where the effects of introducing different assumptions about parameter uncertainty are investigated.

We hope that the papers contained in this special issue will not only disseminate knowledge about stochastic control theory and the current research in this field, but will also motivate operations researchers to develop additional applications of this theory to problems of interest to them. Finally, we would like to thank the referees of the papers submitted to the special issue and to acknowledge gratefully financial support from the Austrian Federal Ministry of Science and Research and the Austrian Bankers' Association to the guest-editor's stay as Schumpeter Research Fellow at the John F. Kennedy School of Government, Harvard University, where most of the editorial work on this special issue has been done. Reinhard Neck

Guest Editor Department of Economics, University of BielefeM P.O. Box 10 01 31 D-4800 Bielefeld 1 Germany