- Email: [email protected]
Personal Impressions of the Dawn of Modern Control
European Journal of Control (2007)13:61–70 # 2007 EUCA DOI:10.3166/EJC.13.61–70
Personal Impressions of the Dawn of Modern Control D.Q. Mayne Department of Electrical and Electronic Engineering, Imperial College London
For many, the dawn of modern control was the dramatic appearance at the first IFAC world congress (1960) of the papers by Boltyanski, Gamkrelidze, Mischenko, and Pontryagin on the maximum principle, by Bellman on dynamic programming and feedback control and by Kalman on the general theory of control systems. These, and related papers by the same authors, triggered a revolution in our subject that continues to this day. I joined Imperial College London in 1959 and was soon swept up into the excitement of this revolution. This essay traces my personal impressions of the dawn of modern control and its evolution, in a few selected areas with which I am familiar, into a mature and comprehensive subject; these impressions are personal and restricted, inevitably, by my ability to appreciate the whole picture. They hopefully give some idea of how the new ideas were received and developed but do not provide a complete picture and certainly fall far short of constituting, as the editors have emphasized, a history of this development. They should be regarded as the impressions of one participant in this exciting revolution. Keywords: Dynamic programming; infinite dimensional systems; Kalman filtering; modern control; multivariable frequency response; nonlinear control; optimal control; predictive control; state space; stochastic control; the maximum principle
Correspondence to: E-mail: [email protected]
1. The Background in the UK 1.1.
The War Years
The background is rooted in World War II in which a large number of scientists and engineers were engaged on problems such as high-power servomechanisms for remote control of heavy guns, radar systems with automatic tracking and gun turret control for aircraft; an informative history of this activity, written by Stuart Bennett, is available. My purpose here is not to review this history, fascinating though it is, but merely to point out a few participants because of their impact on later developments. One of the major players in the UK was Arnold Tustin, who directed control design at Metropolitan Vickers. The original methodology employed by his team was to determine the differential equation of the closed-loop system and then either to employ a differential analyser, based at Manchester University, to obtain its step response or to determine the closed-loop poles; this information was used to modify, if necessary, the design. The necessity to speed up the design process, and to use frequency response data for components for which differential equation models were not available, led the group to develop an ad-hoc design procedure based on frequency response; the group was seemingly unaware of Nyquist’s paper of 1932 until it was brought to their attention in 1943 or later by Professor P. J. Daniels, a mathematics professor at Sheffield University. Tustin had wide
Received 6 December 2006; Accepted 18 January 2007. Recommended by S. Bittanti and M. Gevers
62
ranging interests in human operator models, backlash, friction and nonlinear systems. For computing solutions of linear and nonlinear systems, Tustin proposed a ‘time-series’ methodology for hand computation (a forerunner of the z-transform method); this was before the days of computers. Control of ‘massive objects’, and, in later years, control of economic systems also engaged his interest. Tustin’s interest in nonlinear systems arose from problems caused by backlash and stiction. Daniell proposed using only the first harmonic of the sinusoidal response of backlash nonlinearity; Tustin developed the idea and published in 1947 two of the earliest papers on the describing function method. After the war, he was appointed Professor and Head of the Department of Electrical Engineering, firstly at Birmingham University and, later, in 1953, at Imperial College where John Westcott, appointed a lecturer in 1951, had founded the Control Section that he subsequently led for many years. Tustin appointed me (sight unseen) as a lecturer in 1959, the fourth member of Westcott’s group. The second major activity in control engineering during the war in the UK was directed by A. L. Whiteley and his lieutenant, L. C. Ludbrook at the British Thomson Houston (BTH) Company in Rugby where Westcott worked as an apprentice in radar research and development before proceeding to a radar establishment in Malvern. Whiteley, who had joined the Industrial Engineering Department in 1930, was an experienced engineer with considerable theoretical ability, and was well known to earlier generations of control engineers for his ‘standard forms’ that gave ‘desirable’ (10% overshoot) closed-loop transfer functions of various orders for systems with zero position, velocity or acceleration error and was also the first (at least in the UK) to develop the inverse Nyquist methodology for design. He obtained the degree of D.Sc. after the war and could have taken up a professorial appointment but decided to remain at BTH as director of the Industrial Engineering Department. He employed me as a research engineer for two years in 1955–1956 where I worked under the direction of Ludbrook, who taught me my first lesson in ‘fundamental limitations’ of control systems: the area of overshoot in the step response of a zero position error servomechanism is equal to the area of undershoot (with similar results for systems with zero velocity or acceleration error). Given that the undershoot area is dictated by actuator limits, this posed a fundamental limitation in the speed of response of servomechanisms with limited overshoot, a lesson painfully learnt by the group in their endeavour to design servomechanisms with rapid response. A
D.Q. Mayne
100-page secret report by Whiteley on servomechanism design became available to Westcott while he was working with the Control Commission for Germany and encouraged him to pursue research in control at Imperial College for which Whiteley was later his external examiner and reputedly gave him a hard time. Fire control required prediction of an airplane’s future position, a problem that eventually occupied Wiener’s attention. Wiener’s report Extrapolation, Interpolation and Prediction of Stationary Time Series with Engineering Applications was immediately dubbed The Yellow Peril on account of its yellow cover and formidable mathematics. Again Daniell, who had earlier worked closely with Tustin on smoothing circuits for ad-hoc predictors, intervened by writing a digest to make the report more accessible to UK engineers; Wiener’s report was eventually published in 1949 as a book with the same title. The third UK player from the war years whom I would like to mention is John Coales. Born in 1907, he read mathematics and physics at Cambridge, graduating in 1929. He immediately joined the Admiralty Scientific Service and was soon involved in radar development. By the time war was declared, a naval radar system for gunnery control had been developed. Coales was soon leading a large team and went on to work on radar systems for the control and guidance of anti-aircraft missiles. In 1946, he left the Admiralty to become research director at Elliot Brothers which had then a very large research laboratory (500 people). With the transition to peace, Elliot’s research laboratory decreased in size and, in 1953 at the age of 46, John Coales moved to Cambridge to form a new postgraduate research group in control and systems engineering. 1.2.
1945–1956
Subsequent to the war, the ‘classical’ design methods were consolidated and described in a series of text books; the first UK book, An introduction to Servomechanisms by Porter, appeared in 1950 and was widely used in universities and technical colleges. It was followed in 1951 by An introduction to the theory of control in mechanical engineering by Macmillan of Cambridge University and, in 1953, by Servomechanisms by West of the University of Manchester. Also very influential at the time were a series of books from the US: the classical text Network analysis and design by Bode, The theory of servomechanisms by James, Nichols and Phillips, and the appealing book Automatic feedback control synthesis by Truxall. The prescient book Engineering cybernetics by Tsien (1954) provided a foretaste of the exciting
Personal Impressions of the Dawn of Modern Control
developments of the 1960s but was probably only really appreciated in retrospect. Research and teaching was consolidated in many universities, by including Belfast, Cambridge, Imperial, Manchester and Swansea, and focused on themes, such as backlash, friction, saturation, and anti-windup, initiated during the war. Research was also carried out in many government and industrial laboratories such as the National Physical Laboratory, the Royal Aircraft Establishment, the UK Atomic Energy Establishment, the Central Electricity Generating Board and the British Iron and Steel Research Association. As 1960 approached, control research was generally considered to have reached a plateau; Gordon Brown of MIT told Rosenbrock in 1958 that ‘‘we can’t find suitable research topics for our Ph.D students any more’’.
2.
The Revolution, 1956–1961
By the time I joined Imperial College as a lecturer in 1959, John Westcott, after spending a year at MIT (where he followed Wiener’s course on time series) and completing his Ph.D, had established an active research team with two academics (Peter Blackman and Adrian Roberts) and about 12 research students working in a laboratory equipped with state-of-the-art analogue computers. Over the next few years, student numbers increased to 20 or more as Westcott won research contracts from many of the research laboratories listed above. There were a few postgraduate courses given by Westcott but no formal supervision; there were, instead, regular group discussions, often very lively. Knowledge of the new results filtered into the group through the Proceedings of the First IFAC International Congress in Moscow, 1960, which Westcott attended, Bellman’s book Dynamic Programming, 1957, Kalman’s papers A new approach to linear filtering and prediction problems (1960) and On the general theory of control systems (1960) and the book The mathematical theory of optimal processes (1962) by Pontryagin et al., although it is possible that an earlier translation of one of Pontryagin’s papers was provided by Tom Fuller of Cambridge. These results were disseminated to the whole group (including at least some of the academic staff) by lectures given by the research students, notably by John Florentin whose main interest was stochastic optimal control. These lectures were a revelation for the audience, opening up a whole new world; constraints imposed by the limitations of inadequate theories (such as time-varying transforms to analyse time-varying systems) were thrown away,
63
and clear and direct methods for understanding optimal control of linear and nonlinear systems, timeinvariant or time-varying, deterministic or stochastic fell into our laps. A whole set of new tools, and new problems, was suddenly available, injecting new life into graduate schools, invigorating researchers and opening up new research areas for research students. The possibilities seemed boundless, even for those who had limited mathematical sophistication by today’s standards and made it possible for research students with little supervision to find their own niche; this would not be possible today. By 1961, Florentin, who was in contact with John Barrett, Tom Fuller and Murray Wonham at Cambridge, had published (independently of the work by Joseph and Tou and by Kushner) the first results on the linear, quadratic, Gaussian problem and the separation principle in his papers Optimal control of continuous time, stochastic systems (1961) and Partial observability and optimal control (1962). Interestingly, Florentin, perhaps influenced by the English statistician Bartlett, derived the stochastic dynamic programming equation via a differential equation for the evolution of the (conditional) probability density of the state rather than directly. The results of Kalman and Pontryagin also rapidly permeated the group and by 1961 Westcott was able to organize a symposium An Exposition of Adaptive Control (published in 1962) in which there were papers, many given by students, on stochastic processes, the optimal trajectory problem, Kalman filtering, dynamic programming and other topics. This was followed by a symposium on optimal control in 1964 in which, inter alia, Michael Arbib, one of the many distinguished visitors Westcott attracted, spoke on a rapprochement between automata and control theory. In these few years, the whole direction of research of the control group had changed. At Cambridge, John Coales reinvigorated a postgraduate group started earlier by R. H MacMillan. He introduced a postgraduate course in control engineering in 1955 and, with his strong industrial connections, persuaded industry to send some of their best staff to participate; research students were not encouraged to attend. That formidable Australian, Doug Lampard completed his Ph.D. in 1955 and perhaps persuaded John Barrett, who started, I think, in the same year and who was regarded with awe by fellow students, to work on stochastic processes; certainly Barrett and Lampard wrote a joint paper in the IRE transactions on information theory in 1955, and Barrett’s paper Applications of Kolmogorov’s equations to randomly disturbed automatic control systems in the Proceedings of the First IFAC Congress (1960) influenced Florentin at Imperial. Cambridge then, as
64
now, attracted the best British students some of whom went on to become a force in UK universities. A. R. M Noton left about this time to take up a post in Nottingham University where he wrote a book Introduction to variational methods in Control, published in 1965; he later became a professor in the University of Waterloo, Canada. Tom Fuller, who was later to play a highly important role in Cambridge and the UK, commenced his Ph.D in 1955 on optimal control of systems with saturation, completing his degree in 1959. Fuller, perhaps because of his strong interest in analytical mechanics, was probably the first in the UK to employ a state-space description of deterministic and stochastic systems (he published a paper on this topic in the Electronics Journal of Control in 1960) and the first to be aware of Pontryagin’s results; it was probably through Fuller that this knowledge permeated the Cambridge group (and more widely). In his thesis, and in many subsequent papers (1959–1964) in the Journal of Electronics and Control, Fuller worked out explicit solutions for switching surfaces for optimal control of low order deterministic and stochastic systems. He discovered the Fuller phenomenon (or the Zeno effect): an optimal control that steers an initial state to the origin in finite time may require an infinite number of switches. This surprising behaviour has recently been shown to be expected for a broad class of optimal control problems whose dynamics and cost functions are affine in the control. Fuller’s students have followed up his interests in discontinuous control with subsequent research on Universal Adaptive Control (a field in which we seek a single control strategy to stabilize each of a whole family of control systems) and Sliding Mode Control. As at Imperial, there was very little supervision. The initial guidance that Murray Wonham (who commenced his Ph.D in 1956) received was ‘‘look over these papers and come back when you have a bright idea’’ and that key words that might guide him to a fruitful research area were ‘‘optimal, bang-bang and stochastic’’. Murray writes (personal communication) that ‘‘after two years of this intolerable freedom, the future did not look bright’’. Fortunately, he then decided to investigate the smoothed random telegraph signal, and managed, with some help from Fuller, to determine the resultant distribution which he published, in a joint paper, with Fuller in the Journal of Electronics and Control, 1958. Further development provided a second paper and sufficient material for his thesis Control systems and filters with Markov step inputs that he submitted in 1960. Stimulated, I think, by these results and his fruitful collaboration with Fuller, Wonham wrote, after he left Cambridge, definitive papers on linear quadratic
D.Q. Mayne
gaussian control and the separation principle. Other research students at this time went on to take up academic posts in UK universities where they stimulated further research. By the late 1950s and early 1960s, Cambridge was well set to make further progress in optimal deterministic and stochastic control. The liberating effect of the new ideas of the late 1950s was felt at other universities in the UK; for example, Alistair MacFarlane, who joined Queen Mary College in the late 1950s, was well acquainted with the new developments and published in 1963 a paper on the eigenvector solution of the Riccati equation. However, my knowledge of these developments is limited.
3. 3.1.
1962 Onwards Introduction
Subsequent development of control theory in the UK continued to be impacted by progress elsewhere. The many books on dynamic programming, the state-space description of linear systems and optimal control all stimulated further research. Also avidly read in the early days was the journal Avtomatika i Telemekhanika because it contained many readable papers on optimal and stochastic control. Progress was aided by a few eminent researchers from Mathematics and Applied Mathematics who provided welcome support such as Professor Lighthill who was a Director of the Royal Aircraft Establishment and the first President of the Institute of Mathematics and its Applications before taking up academic posts and Professor Larry Markus who established the Control Theory Centre at the University of Warwick. The influence of academic visitors, mainly from the USA, was particularly important. In the mid 1960s, the Science Research Council of the UK, spurred on by the huge effort in the USA on space-systems and industrial automation, decided to concentrate UK research in this area in three centres of excellence at Cambridge (led by Coales), Imperial College (led by Westcott) and the University of Manchester Institute of Science and Technology (UMIST) led by Rosenbrock. Although support for these centres was soon dropped, they remained strong but with different research flavours: Cambridge, under the influence of Coales, was heavily involved in applications; research at UMIST was very strongly focused on frequency response methods for multivariable control while Imperial was largely engaged, on rather a broad front, in the development of optimal filtering and control.
Personal Impressions of the Dawn of Modern Control
Turning from the general to the personal, the biggest impact on me was initially the lively research atmosphere at Imperial which engendered a deep interest in optimal, stochastic and adaptive control, and adaptive control’s prerequisite, identification. Of course, I was far from qualified to work in all these areas, but the heady atmosphere and the sheer excitement of the main underlying ideas overrode any qualms I should perhaps have had and I worked on differential dynamic programming (a sequential optimization procedure for optimal control motivated by dynamic programming), smoothing, identification and Monte-Carlo methods for stochastic control and nonlinear filtering (the latter a forerunner of particle filtering). Much of the early work of the group was formal and published in the Electronics Journal of Control which was not widely read outside the UK. Almost as important was the impact of the many visitors Westcott supported on a large research grant he had for adaptive control for industrial processes. Hubert Halkin described his approach to the maximum principle and greatly extended our understanding of this result. I also remember Hans Witsenhausen talking about his counterexample in stochastic control, Larry Ho introducing us to Radner’s work on team theory and Kalman’s elegant lectures on partial realization. Larry Ho made it possible for several members of the control group to spend considerable time at Harvard where we absorbed recent research on differential games, information patterns necessary for linear stochastic control and the beginnings of a geometric theory for nonlinear systems; the latter presented us with an opportunity to enter geometric nonlinear control at the ground floor, an opportunity we sadly did not take up. My experience in those years left me with a lasting interest in optimal and stochastic control and in filtering even when these areas became too complex for me to contribute so I will simply trace a few developments to which UK researchers contributed.
3.2.
Optimal Control
The two main threads of research in this area were Pontryagin’s maximum principle and Bellman’s dynamic programming. Despite its conceptual advances, the initial impact of the maximum principle on the mathematics community was small; many believed that the new theory was a minor addition to the calculus of variations. However, this was the aerospace era in which open-loop optimal control problems were meaningful and research in the USA and the Soviet Union was soon aimed at distilling the
65
essential ideas underlying the maximum principle and giving them expression in settings of greater and greater generality. Many developments were made, and useful concepts such as relaxed controls introduced. A later line of research, in which my colleague Richard Vinter was actively involved, was to adapt optimality conditions such as the maximum principle to take account of more general constraints (such as ‘mixed state and control constraints’) than those allowed for in Pontryagin’s original formulation. Here, one would expect that the relevant optimality conditions would involve the maximized Hamiltonian, but this is not, in general, differentiable (in the conventional sense) and required a new concept of derivative. Control theorists responded by creating a new theory of ‘nonsmooth optimal control’ which has been successful in meeting many of these challenges. An exposition of this theory is given in Vinter’s book Optimal Control (2000). An interesting fruit of this research has been the development of an alternative approach to deriving the maximum principle. The starting point is that the maximum principle is easily derived for an optimal control problem with no terminal equality constraints. These troublesome constraints can be removed using an exact penalty function which is non-smooth and necessary conditions can then be obtained for a perturbed problem which is smooth (proximal analysis). Necessary conditions for the original problem are then obtained by passage to the limit. Motivated by aerospace applications, a host of researchers in the US and elsewhere developed algorithms to solve significant constrained optimal control problems with great success. An early problem, successfully solved, was the determination of the angle of attack profile to achieve a specified altitude (20,000 feet) and speed (Mach 1) in minimum time; the optimal path consisted of a rapid climb to (30,000 feet) followed by a period in which height was decreased and speed increased. In the main, accumulation points generated by these algorithms satisfied a weak form of the maximum principle. However, ‘strong variation’ algorithms in which the successor control differed appreciably from the nominal control in a subset of the control horizon (whose measure served as the steplength) were also developed at Imperial College through the initiative of David Jacobson, who was then my research student. I recruited the help of Lucien Polak to establish convergence of these algorithms and to show that accumulation points generated by these algorithms satisfied the maximum principle; this marked the start of a fruitful, 20-year long, international collaboration in which many algorithms for optimal control and semi-infinite and
66
nondifferentiable optimization problems (motivated by design optimization) were studied. I learnt much from Lucien Polak during this cooperation. I was also influenced by Roger Sargent who led a major activity in optimization and optimal control in the Chemical Engineering Department at Imperial. The research of his group had a major impact on the development internationally of process systems engineering; some idea of the magnitude of this influence can be gained from Sargent’s academic tree which, at the last count in 2001, contained more than 600 names! From a mathematical programming point of view, continuoustime optimal control problems are infinite dimensional. Second variation algorithms that generate a Newton step exploit the structure imposed by the differential equation describing the system and require merely integration of a vector and matrix differential equation; evaluation of the infinite dimensional Hessian is avoided. Nowadays, because of the availability of high-quality code for finite dimensional optimization, the control is often discretized and the decision variable composed of both state and control sequences; with this formulation, the Hessian (which is high-dimensional if fine discretization is employed) is banded, a structure that can be exploited by modern code. Dynamic programming probably had a greater initial impact on the engineering community than did the maximum principle providing, as it did, a powerful conceptual framework for formulating feedback control (in contrast to the maximum principle); it remains the only general methodology for stochastic optimal control. An important line of research since the early 1970s has been to put dynamic programming on a rigorous footing. The dynamic programming approach to continuous time optimal control is to express optimal feedback control in terms of the solution (the ‘value function’) of the Hamilton Jacobi Bellman equation. The difficulty here is that the solution to the Hamilton Jacobi Bellman equation for many control systems of interest is not differentiable in the conventional sense. What is required is a notion of solution in terms of which the value function is the unique solution of the Hamilton Jacobi Bellman equation. Overcoming this difficulty has required the introduction of new concepts of derivative (‘viscosity solutions’ and ‘proximal solutions’ among others). The value function provides a verification function enabling optimality of a candidate optimal control to be established. Recent research has addressed the existence of verification functions; the main result is that, under mild conditions, there always exists a Lipschitz continuous verification
D.Q. Mayne
function even when the value function (the natural candidate for a verification function) is not even continuous. A recent development that greatly interested me has been the characterization of the solution to constrained optimal control problems for linear discretetime systems; the value function has been shown to be piecewise quadratic or piecewise affine (depending on the cost function). Although optimal open-loop control is highly relevant to aerospace and to some industrial problems (for example, control of batch processes), it does not directly address the major concern of most engineers, feedback control. Model predictive control, a highly successful method for controlling constrained linear and nonlinear systems and very widely used in the process industries, replaces the complex offline problem of determining an optimal feedback control law by computing online a control action at the current state by solving an open-loop optimal control problem using the current state of the plant as the initial state. Europe is very active in this area; I moved into it as a result of a very helpful suggestion of Karl A˚stro¨m. For practical reasons, the optimal control problem must have a finite horizon, necessitating the use of a receding horizon strategy that required conditions on the optimal control problem to be imposed to ensure closedloop stability. Interestingly, the wide-spread adoption of this form of control is due to its independent development in industry rather than academia; stability analysis, using the value function of the optimal control problem as a Lyapunov function, was, however, carried out in universities. The success of this method for obtaining feedback control has rewakened interest in (open-loop) optimal control and in algorithms for solving optimal control problems.
3.3.
Controller Synthesis/Design
Dynamic programming also provides a constructive method for computing the value function and optimal control law for those problems (e.g. H2 and H1) where the value function can be simply parameterized. The solution to multivariable linear-quadratic (LQ) control problems provided in the early papers of Kalman was a revelation and was extensively exploited in the 1960s and 1970s, particularly at MIT where many applications were studied. For a time, LQ control and its stochastic extension (LQG) appeared to be a universal panacea. This assumption was challenged by the development of multivariable frequency response design procedures and, later, by H1 control to which the UK also contributed.
67
Personal Impressions of the Dawn of Modern Control
3.3.1.
Frequency Response Methods
Research at the Control System Centre at the University of Manchester led by Howard Rosenbrock took a distinctive path. Rosenbrock made a sharp distinction between design and synthesis and thought that synthesis methods based on optimization had a limited place in design: if the problem were posed in a realistic way, it could not be solved and, if simplified, the optimal solution was not the best. He developed an ‘algebraic system theory’ to provide a theoretical basis for a design method closer to the earlier frequency response methods. In his design method, the multivariable problem is reduced to a sequence of modified single-loop designs; for each loop, a ‘generalised’ Nyquist locus, consisting of the Nyquist locus of a diagonal element of the compensated (diagonally dominant) transfer function of the plant on which is superimposed Gershgorin circles bounding the effect of the off-diagonal elements, is employed to design a single input single output controller. The inverse Nyquist array version of this method was successfully applied to a wide range of problems. This research had a big impact especially in the UK where it stimulated alternative approaches. Alistair MacFarlane had a long-standing ambition to generalize Nyquist-Bode theory to the multivariable case. This led him to the concept of characteristic loci (the eigenvalues of the return-ratio matrix); he showed that performance and stability of a multivariable system could be guaranteed provided the characteristic loci satisfied certain conditions analogous to those for single input single output designs. The characteristic loci were manipulated systematically using multivariable compensators. The method had intuitive appeal and was also applied to many problems. MacFarlane and his colleagues also introduced the use of singular values for design as a generalization of gain in the Nyquist–Bode theory and demonstrated the duality between multivariable Nyquist–Bode and rootlocus descriptions.
3.3.2.
H1 Synthesis
It is an interesting fact that our fascination with the power of the new methods for dealing with linear multivariable control made us lose sight for a considerable period of design issues that were prominent in the Nyquist–Bode theory; robustness of the new designs was ignored for some time in the research literature. Fortunately, there are always individuals who criticize the status-quo; Doyle and Stein, in 1981, used singular values to show that both LQG
(the output feedback version of LQ) and frequency response designs were not necessarily robust but that the robustness properties of full-state LQ control could be recovered by appropriate tuning of the LQG controller. About the same time, in 1980, George Zames’ seminal paper on feedback and optimal sensitivity appeared and prepared the ground for a new design method, H1 control, that addressed uncertainty directly and opened a whole new area of research. Early solutions of H1 control problem, obtained through an interesting collaboration between control and operator theorists, were in an input-output setting but were not computationally practical; by the end of this decade practical state-space solutions had been obtained. Keith Glover of Cambridge, influenced by the remarkable results of the Soviet mathematicians Adamjan, Arov and Krein on approximation in the Hankel norm, contributed substantially to this development through his 1984 paper on optimal Hankel-norm approximations of linear multivariable systems and their L1 error bounds. This paper is significant, not only for its relevance to model reduction, but also because it yields a solution for a special case of the H1 problem and had considerable impact. Glover then proceeded to make other major contributions to H1 control theory and to H1 design through loop shaping. The UK also contributed to establishing the existence of lowcomplexity controllers and to the differential game formulation of H1 control (Limebeer), to risk sensitive control (Whittle) and to the gap metric (Smith and Vinnicombe).
3.4.
Distributed Parameter Control
Starting in the early 1970s, the Control Centre at the University of Warwick, founded by Larry Markus, had a significant role in stimulating interest in Control Theory in the UK. The principal research activity was in the theory of infinite dimensional control systems, referred to also as distributed parameter control systems. At the heart of this research was the question of whether the solution to the standard linear quadratic optimal control problem could be extended to problems with state-space an infinite dimensional Banach space. However, the analytic tools for achieving such an extension also made possible the investigation of important system theoretic properties, such as controllability and observability, when the state-space is infinite dimensional. Use of such tools also led to infinite dimensional analogues of the continuous-time Kalman filter. The ‘semigroup’ framework adopted in
68
D.Q. Mayne
this research focuses attention on the integrated form of the state equation, expressed in terms of the semigroup which transforms the initial state into subsequent states and which is the analogue (in this broader setting) of the fundamental matrix of finite dimensional theory. It covers systems described by partial differential equations (such as the heat and wave or the beam equations describing large space structures), and also differential delay systems which can be regarded as delay free infinite dimensional control systems describing how the state histories (over the delay period) evolve with time. There was parallel activity in infinite dimensional theory in France and North America. However, the Control Theory Centre made distinct contributions, notably in developing a framework (‘unbounded’ input operators) that allowed for the handling of boundary control in a simple and natural way and is now widely used. This was important, because the ‘distributed’ control action assumed in the early theory did not reflect the realities of control actuation.
3.5.
when the observation process is bounded, a result later extended to the case when the observation process has an energy bound. A revolution in nonlinear filtering has recently taken place with the emergence in the UK of particle filtering (a sequential Monte Carlo procedure for estimating conditional densities); this research emerged independently of the earlier research and incorporates procedures (such as resampling) that are considerably more effective for variance reduction than those proposed earlier. Although particle filtering has had a large impact, it is computationally expensive. There has been a continuing interest in nonlinear filtering algorithms, based on a Gaussian or Gaussian mixture approximation of the evolving conditional density of the state, which sacrifice the accuracy of high order particle filters but are computationally more efficient. The recent shifted Rayleigh filter (Clark and Vinter) for bearing only tracking illustrates the success that can be achieved by algorithms that exploit the structure of the nonlinear filtering problem, in this case by using analytic integration rather than Monte Carlo estimation.
Stochastic Filtering and Control 3.5.2.
3.5.1.
Stochastic Control
Filtering
The clarity and simplicity of Kalman’s state-space approach to linear filtering, and the extension of these ideas to nonlinear filtering stimulated researchers at Imperial. Early results include an analysis of the properties of the discrete-time Riccati equation, a solution of the smoothing problem and a MonteCarlo procedure for nonlinear filtering. Most effort was devoted to the nonlinear filtering problem. A representation theorem by Clark on functionals of Brownian motion provided the key underpinning of the solution to the general nonlinear filtering problem for nonlinear diffusions and also paved the way for the Malliavin calculus. It is reasonable to require any filter model to be statistically robust; Clark introduced this concept (continuity of the estimator with respect to the data) when he showed that the Ito equations of evolution of conditional probabilities for signals that are Markov chains and diffusion processes have robust versions that are described, respectively, by ordinary and partial differential equations. Following Kailath’s elegant exposition of linear filtering in terms of the innovation process, a major concern of many researchers was the determination, in the nonlinear case, of conditions for one-to-one correspondence between the observation process and its innovation. These conditions were obtained in 1969 for the case
Dynamic programming provided a general methodology for handling optimal feedback control when stochastic disturbances are present and was soon used to solve the linear quadratic gaussian (LQG) optimal control problem. However, as in the deterministic case, there was a need to place dynamic programming for general nonlinear diffusion processes on a rigorous footing; this was done in the 1973 paper of Davis and Varaiya on dynamic programming conditions for partially observed processes using advanced Martingale methods. Much of the theoretical work on stochastic control, concerned as it is with control of systems described by stochastic differential equations, does not address an important class of problems where ‘randomness’ appears in the form of point processes, such as arrival of customers and production demands. Davis introduced Piecewise-deterministic Markov process in 1984 as a framework for such (nondiffusion) stochastic processes; this framework has since been widely adopted. Other areas of UK research include the spaceship control problem (stochastic control with finite resource) and resource allocation (the ‘multi-armed bandit problem’). An interesting aspect of recent research has been the emergence of new and important application areas such as mathematical finance. UK contributions include the development of singular stochastic
Personal Impressions of the Dawn of Modern Control
control to solve problems of optimal investment with transaction costs, the first use of martingale methods in economic growth theory (for optimal saving) and use of general semimartingale theory to obtain conditions of optimality for the infinite horizon portfoliocum-savings problem. Another interesting application is the internet which is a network used by many users competing for a set of limited resources. The key to efficient operation is allocation of resources to different users during times of congestion. Frank Kelly at Cambridge was the first to view the problem as one of distributed controls to maximize user utility and to show that net user utility is a natural Lyapunov function. Each controller is subject to a different delay; stability was established using results on stability of delay differential equations. Interestingly, this research made it possible to use the multivariable Nyquist criterion as a natural tool for network controller design (Vinnicombe). The research has spawned a new generation of congestion algorithms.
4.
Conclusion
To illustrate the impact of Bellman, Kalman and Pontryagin and coworkers on our subject, I have drawn attention to some areas of research to which the UK has contributed. In doing so, I have neglected whole areas such as the geometric theory of linear and nonlinear systems, identification and adaptive control, stability, dissipative systems, switching systems and hybrid control and mathematical system theory. Our subject has matured enormously during the period surveyed and its literature has become very rich; becoming expert in even one subject, H1 control for example, requires considerable effort. This richness was acquired via many different flavours of research. Perhaps the most important, and most revolutionary, is the introduction of a framework that provides a radically different model or problem formulation. The introduction of state-space models, explicitly by Kalman and implicitly by Bellman and Pontryagin, into the control systems community provided new and simpler ways of looking at old problems and opened up new problems. Of course, the state-space formulation had existed for a long time in mathematics and physics but the control community had been constrained by a restrictive transfer function model. Old problems, such as filtering, became much more transparent, and new research areas, such as optimal control and nonlinear control, revealed. Another interesting example is George Zames seminal
69
paper on H1 control; there we find reasoning leading to the formulation of an optimal control problem more appropriate for achieving robustness than the standard LQG problem that does not address robustness with the same effectiveness as classical control techniques. This paper opened up a wide and fruitful area of research. The geometric theory of linear and nonlinear systems provided valuable new insights; particularly fascinating was the fact that so much of the linear theory could be generalized to the nonlinear case. Nonsmooth analysis widened the class of finite-dimensional and dynamic optimization problems, allowing deterministic dynamic programming to be put on a rigourous footing and permitting adaptation of the maximum principle to a larger class of problems. Piecewise deterministic Markov processes provided a new framework for stochastic control for problems in which randomness took the form of point processes (such as random arrival of customers and production demands) rather than as a diffusion process. Within a new framework (e.g. statespace models), new concepts (such as controllability, observability, input-to-state stability) were introduced to aid analysis and synthesis. A very different contribution might be termed ‘invention’, the discovery of a new controller, filter or identifier that is relatively easily implemented. Normally an optimal solution to the appropriate problem is sought, but in many cases this is not practical. A good illustration is provided by deterministic adaptive control. Obtaining an optimal solution, via dynamic programming, is impossibly difficult so that alternative solutions were sought. The first solution to the model reference adaptive control problem was obtained in 1978 (for continuous-time systems) by Feuer and Morse using a least squares estimator and complex nonlinear damping to achieve global convergence. A second, relatively simple solution for discrete-time systems, obtained by Goodwin et al. in 1980, used normalization to slow down the estimator sufficiently to ensure convergence and was a major milestone in the evolution of adaptive control. Interestingly, the original adaptive controller due to Monopoli, which does not employ normalization, has recently been shown to be semiglobally stable but not globally stable. Another important ‘invention’ is the backstepping solution to nonlinear control. A third type of contribution is ‘whistle-blowing’. It is all too easy to be carried along by exciting discoveries and to suppress our critical faculties. The LQG solution was so captivating that the issue of robustness was neglected for a considerable period; it took the 1980 paper by Doyle and Stein to alert us to this fact and to prepare us for the H1 era. A similar
70
role was played by Rohrs et al. in their demonstration of the lack of robustness of adaptive control; robust adaptive control is still a relevant area of research. Cross fertilization has also aided the development of our subject. An example of this, in an area I have not discussed, was the use by Patrick Parks in 1966 of the celebrated Kalman–Yakubovich–Popov positive real lemma to establish stability of an adaptive control system that he proposed as an alternative to the heuristic MIT rule which was then in common use. New applications also provide a considerable impetus to our field; recent examples include network congestion control, mathematical finance and multiagent formations. In retrospect, the revolutionary period might be regarded as control’s ‘big bang’; it was a unique period, not likely to be repeated, in which researchers had the opportunity to be generalists and research students the ability to find a niche with little or no help from advisers. Progress since then has been spectacular, but not uniform. There have been plateaus such
D.Q. Mayne
as the LQG era followed by significant revolutions such as H1 optimal control and the geometric theory of nonlinear systems; each time we might feel that progress is slowing to a standstill, something new emerges. What Peter Whittle says of research in linear systems is perhaps true for our whole field: ‘‘the linear model seems to have infinite depth, and yields only to reveal further mysteries . . . . its known theory becomes more extensive and definite with time, but somehow never definitive.’’
Acknowledgement I am very grateful to Mark Davis, Keith Glover, Petar Kokotovic, David Limebeer, Alistair MacFarlane, Larry Markus, Alistair Mees, Howard Rosenbrock, Roger Sargent, Malcolm Smith, Richard Vinter, John Westcott and Murray Wonham for their helpful advice. Any shortcomings are, of course, my responsibility.
Personal Impressions of the Dawn of Modern Control D.Q. Mayne Department of Electrical and Electronic Engineering, Imperial College London
For many, the dawn of modern control was the dramatic appearance at the first IFAC world congress (1960) of the papers by Boltyanski, Gamkrelidze, Mischenko, and Pontryagin on the maximum principle, by Bellman on dynamic programming and feedback control and by Kalman on the general theory of control systems. These, and related papers by the same authors, triggered a revolution in our subject that continues to this day. I joined Imperial College London in 1959 and was soon swept up into the excitement of this revolution. This essay traces my personal impressions of the dawn of modern control and its evolution, in a few selected areas with which I am familiar, into a mature and comprehensive subject; these impressions are personal and restricted, inevitably, by my ability to appreciate the whole picture. They hopefully give some idea of how the new ideas were received and developed but do not provide a complete picture and certainly fall far short of constituting, as the editors have emphasized, a history of this development. They should be regarded as the impressions of one participant in this exciting revolution. Keywords: Dynamic programming; infinite dimensional systems; Kalman filtering; modern control; multivariable frequency response; nonlinear control; optimal control; predictive control; state space; stochastic control; the maximum principle
Correspondence to: E-mail: [email protected]
1. The Background in the UK 1.1.
The War Years
The background is rooted in World War II in which a large number of scientists and engineers were engaged on problems such as high-power servomechanisms for remote control of heavy guns, radar systems with automatic tracking and gun turret control for aircraft; an informative history of this activity, written by Stuart Bennett, is available. My purpose here is not to review this history, fascinating though it is, but merely to point out a few participants because of their impact on later developments. One of the major players in the UK was Arnold Tustin, who directed control design at Metropolitan Vickers. The original methodology employed by his team was to determine the differential equation of the closed-loop system and then either to employ a differential analyser, based at Manchester University, to obtain its step response or to determine the closed-loop poles; this information was used to modify, if necessary, the design. The necessity to speed up the design process, and to use frequency response data for components for which differential equation models were not available, led the group to develop an ad-hoc design procedure based on frequency response; the group was seemingly unaware of Nyquist’s paper of 1932 until it was brought to their attention in 1943 or later by Professor P. J. Daniels, a mathematics professor at Sheffield University. Tustin had wide
Received 6 December 2006; Accepted 18 January 2007. Recommended by S. Bittanti and M. Gevers
62
ranging interests in human operator models, backlash, friction and nonlinear systems. For computing solutions of linear and nonlinear systems, Tustin proposed a ‘time-series’ methodology for hand computation (a forerunner of the z-transform method); this was before the days of computers. Control of ‘massive objects’, and, in later years, control of economic systems also engaged his interest. Tustin’s interest in nonlinear systems arose from problems caused by backlash and stiction. Daniell proposed using only the first harmonic of the sinusoidal response of backlash nonlinearity; Tustin developed the idea and published in 1947 two of the earliest papers on the describing function method. After the war, he was appointed Professor and Head of the Department of Electrical Engineering, firstly at Birmingham University and, later, in 1953, at Imperial College where John Westcott, appointed a lecturer in 1951, had founded the Control Section that he subsequently led for many years. Tustin appointed me (sight unseen) as a lecturer in 1959, the fourth member of Westcott’s group. The second major activity in control engineering during the war in the UK was directed by A. L. Whiteley and his lieutenant, L. C. Ludbrook at the British Thomson Houston (BTH) Company in Rugby where Westcott worked as an apprentice in radar research and development before proceeding to a radar establishment in Malvern. Whiteley, who had joined the Industrial Engineering Department in 1930, was an experienced engineer with considerable theoretical ability, and was well known to earlier generations of control engineers for his ‘standard forms’ that gave ‘desirable’ (10% overshoot) closed-loop transfer functions of various orders for systems with zero position, velocity or acceleration error and was also the first (at least in the UK) to develop the inverse Nyquist methodology for design. He obtained the degree of D.Sc. after the war and could have taken up a professorial appointment but decided to remain at BTH as director of the Industrial Engineering Department. He employed me as a research engineer for two years in 1955–1956 where I worked under the direction of Ludbrook, who taught me my first lesson in ‘fundamental limitations’ of control systems: the area of overshoot in the step response of a zero position error servomechanism is equal to the area of undershoot (with similar results for systems with zero velocity or acceleration error). Given that the undershoot area is dictated by actuator limits, this posed a fundamental limitation in the speed of response of servomechanisms with limited overshoot, a lesson painfully learnt by the group in their endeavour to design servomechanisms with rapid response. A
D.Q. Mayne
100-page secret report by Whiteley on servomechanism design became available to Westcott while he was working with the Control Commission for Germany and encouraged him to pursue research in control at Imperial College for which Whiteley was later his external examiner and reputedly gave him a hard time. Fire control required prediction of an airplane’s future position, a problem that eventually occupied Wiener’s attention. Wiener’s report Extrapolation, Interpolation and Prediction of Stationary Time Series with Engineering Applications was immediately dubbed The Yellow Peril on account of its yellow cover and formidable mathematics. Again Daniell, who had earlier worked closely with Tustin on smoothing circuits for ad-hoc predictors, intervened by writing a digest to make the report more accessible to UK engineers; Wiener’s report was eventually published in 1949 as a book with the same title. The third UK player from the war years whom I would like to mention is John Coales. Born in 1907, he read mathematics and physics at Cambridge, graduating in 1929. He immediately joined the Admiralty Scientific Service and was soon involved in radar development. By the time war was declared, a naval radar system for gunnery control had been developed. Coales was soon leading a large team and went on to work on radar systems for the control and guidance of anti-aircraft missiles. In 1946, he left the Admiralty to become research director at Elliot Brothers which had then a very large research laboratory (500 people). With the transition to peace, Elliot’s research laboratory decreased in size and, in 1953 at the age of 46, John Coales moved to Cambridge to form a new postgraduate research group in control and systems engineering. 1.2.
1945–1956
Subsequent to the war, the ‘classical’ design methods were consolidated and described in a series of text books; the first UK book, An introduction to Servomechanisms by Porter, appeared in 1950 and was widely used in universities and technical colleges. It was followed in 1951 by An introduction to the theory of control in mechanical engineering by Macmillan of Cambridge University and, in 1953, by Servomechanisms by West of the University of Manchester. Also very influential at the time were a series of books from the US: the classical text Network analysis and design by Bode, The theory of servomechanisms by James, Nichols and Phillips, and the appealing book Automatic feedback control synthesis by Truxall. The prescient book Engineering cybernetics by Tsien (1954) provided a foretaste of the exciting
Personal Impressions of the Dawn of Modern Control
developments of the 1960s but was probably only really appreciated in retrospect. Research and teaching was consolidated in many universities, by including Belfast, Cambridge, Imperial, Manchester and Swansea, and focused on themes, such as backlash, friction, saturation, and anti-windup, initiated during the war. Research was also carried out in many government and industrial laboratories such as the National Physical Laboratory, the Royal Aircraft Establishment, the UK Atomic Energy Establishment, the Central Electricity Generating Board and the British Iron and Steel Research Association. As 1960 approached, control research was generally considered to have reached a plateau; Gordon Brown of MIT told Rosenbrock in 1958 that ‘‘we can’t find suitable research topics for our Ph.D students any more’’.
2.
The Revolution, 1956–1961
By the time I joined Imperial College as a lecturer in 1959, John Westcott, after spending a year at MIT (where he followed Wiener’s course on time series) and completing his Ph.D, had established an active research team with two academics (Peter Blackman and Adrian Roberts) and about 12 research students working in a laboratory equipped with state-of-the-art analogue computers. Over the next few years, student numbers increased to 20 or more as Westcott won research contracts from many of the research laboratories listed above. There were a few postgraduate courses given by Westcott but no formal supervision; there were, instead, regular group discussions, often very lively. Knowledge of the new results filtered into the group through the Proceedings of the First IFAC International Congress in Moscow, 1960, which Westcott attended, Bellman’s book Dynamic Programming, 1957, Kalman’s papers A new approach to linear filtering and prediction problems (1960) and On the general theory of control systems (1960) and the book The mathematical theory of optimal processes (1962) by Pontryagin et al., although it is possible that an earlier translation of one of Pontryagin’s papers was provided by Tom Fuller of Cambridge. These results were disseminated to the whole group (including at least some of the academic staff) by lectures given by the research students, notably by John Florentin whose main interest was stochastic optimal control. These lectures were a revelation for the audience, opening up a whole new world; constraints imposed by the limitations of inadequate theories (such as time-varying transforms to analyse time-varying systems) were thrown away,
63
and clear and direct methods for understanding optimal control of linear and nonlinear systems, timeinvariant or time-varying, deterministic or stochastic fell into our laps. A whole set of new tools, and new problems, was suddenly available, injecting new life into graduate schools, invigorating researchers and opening up new research areas for research students. The possibilities seemed boundless, even for those who had limited mathematical sophistication by today’s standards and made it possible for research students with little supervision to find their own niche; this would not be possible today. By 1961, Florentin, who was in contact with John Barrett, Tom Fuller and Murray Wonham at Cambridge, had published (independently of the work by Joseph and Tou and by Kushner) the first results on the linear, quadratic, Gaussian problem and the separation principle in his papers Optimal control of continuous time, stochastic systems (1961) and Partial observability and optimal control (1962). Interestingly, Florentin, perhaps influenced by the English statistician Bartlett, derived the stochastic dynamic programming equation via a differential equation for the evolution of the (conditional) probability density of the state rather than directly. The results of Kalman and Pontryagin also rapidly permeated the group and by 1961 Westcott was able to organize a symposium An Exposition of Adaptive Control (published in 1962) in which there were papers, many given by students, on stochastic processes, the optimal trajectory problem, Kalman filtering, dynamic programming and other topics. This was followed by a symposium on optimal control in 1964 in which, inter alia, Michael Arbib, one of the many distinguished visitors Westcott attracted, spoke on a rapprochement between automata and control theory. In these few years, the whole direction of research of the control group had changed. At Cambridge, John Coales reinvigorated a postgraduate group started earlier by R. H MacMillan. He introduced a postgraduate course in control engineering in 1955 and, with his strong industrial connections, persuaded industry to send some of their best staff to participate; research students were not encouraged to attend. That formidable Australian, Doug Lampard completed his Ph.D. in 1955 and perhaps persuaded John Barrett, who started, I think, in the same year and who was regarded with awe by fellow students, to work on stochastic processes; certainly Barrett and Lampard wrote a joint paper in the IRE transactions on information theory in 1955, and Barrett’s paper Applications of Kolmogorov’s equations to randomly disturbed automatic control systems in the Proceedings of the First IFAC Congress (1960) influenced Florentin at Imperial. Cambridge then, as
64
now, attracted the best British students some of whom went on to become a force in UK universities. A. R. M Noton left about this time to take up a post in Nottingham University where he wrote a book Introduction to variational methods in Control, published in 1965; he later became a professor in the University of Waterloo, Canada. Tom Fuller, who was later to play a highly important role in Cambridge and the UK, commenced his Ph.D in 1955 on optimal control of systems with saturation, completing his degree in 1959. Fuller, perhaps because of his strong interest in analytical mechanics, was probably the first in the UK to employ a state-space description of deterministic and stochastic systems (he published a paper on this topic in the Electronics Journal of Control in 1960) and the first to be aware of Pontryagin’s results; it was probably through Fuller that this knowledge permeated the Cambridge group (and more widely). In his thesis, and in many subsequent papers (1959–1964) in the Journal of Electronics and Control, Fuller worked out explicit solutions for switching surfaces for optimal control of low order deterministic and stochastic systems. He discovered the Fuller phenomenon (or the Zeno effect): an optimal control that steers an initial state to the origin in finite time may require an infinite number of switches. This surprising behaviour has recently been shown to be expected for a broad class of optimal control problems whose dynamics and cost functions are affine in the control. Fuller’s students have followed up his interests in discontinuous control with subsequent research on Universal Adaptive Control (a field in which we seek a single control strategy to stabilize each of a whole family of control systems) and Sliding Mode Control. As at Imperial, there was very little supervision. The initial guidance that Murray Wonham (who commenced his Ph.D in 1956) received was ‘‘look over these papers and come back when you have a bright idea’’ and that key words that might guide him to a fruitful research area were ‘‘optimal, bang-bang and stochastic’’. Murray writes (personal communication) that ‘‘after two years of this intolerable freedom, the future did not look bright’’. Fortunately, he then decided to investigate the smoothed random telegraph signal, and managed, with some help from Fuller, to determine the resultant distribution which he published, in a joint paper, with Fuller in the Journal of Electronics and Control, 1958. Further development provided a second paper and sufficient material for his thesis Control systems and filters with Markov step inputs that he submitted in 1960. Stimulated, I think, by these results and his fruitful collaboration with Fuller, Wonham wrote, after he left Cambridge, definitive papers on linear quadratic
D.Q. Mayne
gaussian control and the separation principle. Other research students at this time went on to take up academic posts in UK universities where they stimulated further research. By the late 1950s and early 1960s, Cambridge was well set to make further progress in optimal deterministic and stochastic control. The liberating effect of the new ideas of the late 1950s was felt at other universities in the UK; for example, Alistair MacFarlane, who joined Queen Mary College in the late 1950s, was well acquainted with the new developments and published in 1963 a paper on the eigenvector solution of the Riccati equation. However, my knowledge of these developments is limited.
3. 3.1.
1962 Onwards Introduction
Subsequent development of control theory in the UK continued to be impacted by progress elsewhere. The many books on dynamic programming, the state-space description of linear systems and optimal control all stimulated further research. Also avidly read in the early days was the journal Avtomatika i Telemekhanika because it contained many readable papers on optimal and stochastic control. Progress was aided by a few eminent researchers from Mathematics and Applied Mathematics who provided welcome support such as Professor Lighthill who was a Director of the Royal Aircraft Establishment and the first President of the Institute of Mathematics and its Applications before taking up academic posts and Professor Larry Markus who established the Control Theory Centre at the University of Warwick. The influence of academic visitors, mainly from the USA, was particularly important. In the mid 1960s, the Science Research Council of the UK, spurred on by the huge effort in the USA on space-systems and industrial automation, decided to concentrate UK research in this area in three centres of excellence at Cambridge (led by Coales), Imperial College (led by Westcott) and the University of Manchester Institute of Science and Technology (UMIST) led by Rosenbrock. Although support for these centres was soon dropped, they remained strong but with different research flavours: Cambridge, under the influence of Coales, was heavily involved in applications; research at UMIST was very strongly focused on frequency response methods for multivariable control while Imperial was largely engaged, on rather a broad front, in the development of optimal filtering and control.
Personal Impressions of the Dawn of Modern Control
Turning from the general to the personal, the biggest impact on me was initially the lively research atmosphere at Imperial which engendered a deep interest in optimal, stochastic and adaptive control, and adaptive control’s prerequisite, identification. Of course, I was far from qualified to work in all these areas, but the heady atmosphere and the sheer excitement of the main underlying ideas overrode any qualms I should perhaps have had and I worked on differential dynamic programming (a sequential optimization procedure for optimal control motivated by dynamic programming), smoothing, identification and Monte-Carlo methods for stochastic control and nonlinear filtering (the latter a forerunner of particle filtering). Much of the early work of the group was formal and published in the Electronics Journal of Control which was not widely read outside the UK. Almost as important was the impact of the many visitors Westcott supported on a large research grant he had for adaptive control for industrial processes. Hubert Halkin described his approach to the maximum principle and greatly extended our understanding of this result. I also remember Hans Witsenhausen talking about his counterexample in stochastic control, Larry Ho introducing us to Radner’s work on team theory and Kalman’s elegant lectures on partial realization. Larry Ho made it possible for several members of the control group to spend considerable time at Harvard where we absorbed recent research on differential games, information patterns necessary for linear stochastic control and the beginnings of a geometric theory for nonlinear systems; the latter presented us with an opportunity to enter geometric nonlinear control at the ground floor, an opportunity we sadly did not take up. My experience in those years left me with a lasting interest in optimal and stochastic control and in filtering even when these areas became too complex for me to contribute so I will simply trace a few developments to which UK researchers contributed.
3.2.
Optimal Control
The two main threads of research in this area were Pontryagin’s maximum principle and Bellman’s dynamic programming. Despite its conceptual advances, the initial impact of the maximum principle on the mathematics community was small; many believed that the new theory was a minor addition to the calculus of variations. However, this was the aerospace era in which open-loop optimal control problems were meaningful and research in the USA and the Soviet Union was soon aimed at distilling the
65
essential ideas underlying the maximum principle and giving them expression in settings of greater and greater generality. Many developments were made, and useful concepts such as relaxed controls introduced. A later line of research, in which my colleague Richard Vinter was actively involved, was to adapt optimality conditions such as the maximum principle to take account of more general constraints (such as ‘mixed state and control constraints’) than those allowed for in Pontryagin’s original formulation. Here, one would expect that the relevant optimality conditions would involve the maximized Hamiltonian, but this is not, in general, differentiable (in the conventional sense) and required a new concept of derivative. Control theorists responded by creating a new theory of ‘nonsmooth optimal control’ which has been successful in meeting many of these challenges. An exposition of this theory is given in Vinter’s book Optimal Control (2000). An interesting fruit of this research has been the development of an alternative approach to deriving the maximum principle. The starting point is that the maximum principle is easily derived for an optimal control problem with no terminal equality constraints. These troublesome constraints can be removed using an exact penalty function which is non-smooth and necessary conditions can then be obtained for a perturbed problem which is smooth (proximal analysis). Necessary conditions for the original problem are then obtained by passage to the limit. Motivated by aerospace applications, a host of researchers in the US and elsewhere developed algorithms to solve significant constrained optimal control problems with great success. An early problem, successfully solved, was the determination of the angle of attack profile to achieve a specified altitude (20,000 feet) and speed (Mach 1) in minimum time; the optimal path consisted of a rapid climb to (30,000 feet) followed by a period in which height was decreased and speed increased. In the main, accumulation points generated by these algorithms satisfied a weak form of the maximum principle. However, ‘strong variation’ algorithms in which the successor control differed appreciably from the nominal control in a subset of the control horizon (whose measure served as the steplength) were also developed at Imperial College through the initiative of David Jacobson, who was then my research student. I recruited the help of Lucien Polak to establish convergence of these algorithms and to show that accumulation points generated by these algorithms satisfied the maximum principle; this marked the start of a fruitful, 20-year long, international collaboration in which many algorithms for optimal control and semi-infinite and
66
nondifferentiable optimization problems (motivated by design optimization) were studied. I learnt much from Lucien Polak during this cooperation. I was also influenced by Roger Sargent who led a major activity in optimization and optimal control in the Chemical Engineering Department at Imperial. The research of his group had a major impact on the development internationally of process systems engineering; some idea of the magnitude of this influence can be gained from Sargent’s academic tree which, at the last count in 2001, contained more than 600 names! From a mathematical programming point of view, continuoustime optimal control problems are infinite dimensional. Second variation algorithms that generate a Newton step exploit the structure imposed by the differential equation describing the system and require merely integration of a vector and matrix differential equation; evaluation of the infinite dimensional Hessian is avoided. Nowadays, because of the availability of high-quality code for finite dimensional optimization, the control is often discretized and the decision variable composed of both state and control sequences; with this formulation, the Hessian (which is high-dimensional if fine discretization is employed) is banded, a structure that can be exploited by modern code. Dynamic programming probably had a greater initial impact on the engineering community than did the maximum principle providing, as it did, a powerful conceptual framework for formulating feedback control (in contrast to the maximum principle); it remains the only general methodology for stochastic optimal control. An important line of research since the early 1970s has been to put dynamic programming on a rigorous footing. The dynamic programming approach to continuous time optimal control is to express optimal feedback control in terms of the solution (the ‘value function’) of the Hamilton Jacobi Bellman equation. The difficulty here is that the solution to the Hamilton Jacobi Bellman equation for many control systems of interest is not differentiable in the conventional sense. What is required is a notion of solution in terms of which the value function is the unique solution of the Hamilton Jacobi Bellman equation. Overcoming this difficulty has required the introduction of new concepts of derivative (‘viscosity solutions’ and ‘proximal solutions’ among others). The value function provides a verification function enabling optimality of a candidate optimal control to be established. Recent research has addressed the existence of verification functions; the main result is that, under mild conditions, there always exists a Lipschitz continuous verification
D.Q. Mayne
function even when the value function (the natural candidate for a verification function) is not even continuous. A recent development that greatly interested me has been the characterization of the solution to constrained optimal control problems for linear discretetime systems; the value function has been shown to be piecewise quadratic or piecewise affine (depending on the cost function). Although optimal open-loop control is highly relevant to aerospace and to some industrial problems (for example, control of batch processes), it does not directly address the major concern of most engineers, feedback control. Model predictive control, a highly successful method for controlling constrained linear and nonlinear systems and very widely used in the process industries, replaces the complex offline problem of determining an optimal feedback control law by computing online a control action at the current state by solving an open-loop optimal control problem using the current state of the plant as the initial state. Europe is very active in this area; I moved into it as a result of a very helpful suggestion of Karl A˚stro¨m. For practical reasons, the optimal control problem must have a finite horizon, necessitating the use of a receding horizon strategy that required conditions on the optimal control problem to be imposed to ensure closedloop stability. Interestingly, the wide-spread adoption of this form of control is due to its independent development in industry rather than academia; stability analysis, using the value function of the optimal control problem as a Lyapunov function, was, however, carried out in universities. The success of this method for obtaining feedback control has rewakened interest in (open-loop) optimal control and in algorithms for solving optimal control problems.
3.3.
Controller Synthesis/Design
Dynamic programming also provides a constructive method for computing the value function and optimal control law for those problems (e.g. H2 and H1) where the value function can be simply parameterized. The solution to multivariable linear-quadratic (LQ) control problems provided in the early papers of Kalman was a revelation and was extensively exploited in the 1960s and 1970s, particularly at MIT where many applications were studied. For a time, LQ control and its stochastic extension (LQG) appeared to be a universal panacea. This assumption was challenged by the development of multivariable frequency response design procedures and, later, by H1 control to which the UK also contributed.
67
Personal Impressions of the Dawn of Modern Control
3.3.1.
Frequency Response Methods
Research at the Control System Centre at the University of Manchester led by Howard Rosenbrock took a distinctive path. Rosenbrock made a sharp distinction between design and synthesis and thought that synthesis methods based on optimization had a limited place in design: if the problem were posed in a realistic way, it could not be solved and, if simplified, the optimal solution was not the best. He developed an ‘algebraic system theory’ to provide a theoretical basis for a design method closer to the earlier frequency response methods. In his design method, the multivariable problem is reduced to a sequence of modified single-loop designs; for each loop, a ‘generalised’ Nyquist locus, consisting of the Nyquist locus of a diagonal element of the compensated (diagonally dominant) transfer function of the plant on which is superimposed Gershgorin circles bounding the effect of the off-diagonal elements, is employed to design a single input single output controller. The inverse Nyquist array version of this method was successfully applied to a wide range of problems. This research had a big impact especially in the UK where it stimulated alternative approaches. Alistair MacFarlane had a long-standing ambition to generalize Nyquist-Bode theory to the multivariable case. This led him to the concept of characteristic loci (the eigenvalues of the return-ratio matrix); he showed that performance and stability of a multivariable system could be guaranteed provided the characteristic loci satisfied certain conditions analogous to those for single input single output designs. The characteristic loci were manipulated systematically using multivariable compensators. The method had intuitive appeal and was also applied to many problems. MacFarlane and his colleagues also introduced the use of singular values for design as a generalization of gain in the Nyquist–Bode theory and demonstrated the duality between multivariable Nyquist–Bode and rootlocus descriptions.
3.3.2.
H1 Synthesis
It is an interesting fact that our fascination with the power of the new methods for dealing with linear multivariable control made us lose sight for a considerable period of design issues that were prominent in the Nyquist–Bode theory; robustness of the new designs was ignored for some time in the research literature. Fortunately, there are always individuals who criticize the status-quo; Doyle and Stein, in 1981, used singular values to show that both LQG
(the output feedback version of LQ) and frequency response designs were not necessarily robust but that the robustness properties of full-state LQ control could be recovered by appropriate tuning of the LQG controller. About the same time, in 1980, George Zames’ seminal paper on feedback and optimal sensitivity appeared and prepared the ground for a new design method, H1 control, that addressed uncertainty directly and opened a whole new area of research. Early solutions of H1 control problem, obtained through an interesting collaboration between control and operator theorists, were in an input-output setting but were not computationally practical; by the end of this decade practical state-space solutions had been obtained. Keith Glover of Cambridge, influenced by the remarkable results of the Soviet mathematicians Adamjan, Arov and Krein on approximation in the Hankel norm, contributed substantially to this development through his 1984 paper on optimal Hankel-norm approximations of linear multivariable systems and their L1 error bounds. This paper is significant, not only for its relevance to model reduction, but also because it yields a solution for a special case of the H1 problem and had considerable impact. Glover then proceeded to make other major contributions to H1 control theory and to H1 design through loop shaping. The UK also contributed to establishing the existence of lowcomplexity controllers and to the differential game formulation of H1 control (Limebeer), to risk sensitive control (Whittle) and to the gap metric (Smith and Vinnicombe).
3.4.
Distributed Parameter Control
Starting in the early 1970s, the Control Centre at the University of Warwick, founded by Larry Markus, had a significant role in stimulating interest in Control Theory in the UK. The principal research activity was in the theory of infinite dimensional control systems, referred to also as distributed parameter control systems. At the heart of this research was the question of whether the solution to the standard linear quadratic optimal control problem could be extended to problems with state-space an infinite dimensional Banach space. However, the analytic tools for achieving such an extension also made possible the investigation of important system theoretic properties, such as controllability and observability, when the state-space is infinite dimensional. Use of such tools also led to infinite dimensional analogues of the continuous-time Kalman filter. The ‘semigroup’ framework adopted in
68
D.Q. Mayne
this research focuses attention on the integrated form of the state equation, expressed in terms of the semigroup which transforms the initial state into subsequent states and which is the analogue (in this broader setting) of the fundamental matrix of finite dimensional theory. It covers systems described by partial differential equations (such as the heat and wave or the beam equations describing large space structures), and also differential delay systems which can be regarded as delay free infinite dimensional control systems describing how the state histories (over the delay period) evolve with time. There was parallel activity in infinite dimensional theory in France and North America. However, the Control Theory Centre made distinct contributions, notably in developing a framework (‘unbounded’ input operators) that allowed for the handling of boundary control in a simple and natural way and is now widely used. This was important, because the ‘distributed’ control action assumed in the early theory did not reflect the realities of control actuation.
3.5.
when the observation process is bounded, a result later extended to the case when the observation process has an energy bound. A revolution in nonlinear filtering has recently taken place with the emergence in the UK of particle filtering (a sequential Monte Carlo procedure for estimating conditional densities); this research emerged independently of the earlier research and incorporates procedures (such as resampling) that are considerably more effective for variance reduction than those proposed earlier. Although particle filtering has had a large impact, it is computationally expensive. There has been a continuing interest in nonlinear filtering algorithms, based on a Gaussian or Gaussian mixture approximation of the evolving conditional density of the state, which sacrifice the accuracy of high order particle filters but are computationally more efficient. The recent shifted Rayleigh filter (Clark and Vinter) for bearing only tracking illustrates the success that can be achieved by algorithms that exploit the structure of the nonlinear filtering problem, in this case by using analytic integration rather than Monte Carlo estimation.
Stochastic Filtering and Control 3.5.2.
3.5.1.
Stochastic Control
Filtering
The clarity and simplicity of Kalman’s state-space approach to linear filtering, and the extension of these ideas to nonlinear filtering stimulated researchers at Imperial. Early results include an analysis of the properties of the discrete-time Riccati equation, a solution of the smoothing problem and a MonteCarlo procedure for nonlinear filtering. Most effort was devoted to the nonlinear filtering problem. A representation theorem by Clark on functionals of Brownian motion provided the key underpinning of the solution to the general nonlinear filtering problem for nonlinear diffusions and also paved the way for the Malliavin calculus. It is reasonable to require any filter model to be statistically robust; Clark introduced this concept (continuity of the estimator with respect to the data) when he showed that the Ito equations of evolution of conditional probabilities for signals that are Markov chains and diffusion processes have robust versions that are described, respectively, by ordinary and partial differential equations. Following Kailath’s elegant exposition of linear filtering in terms of the innovation process, a major concern of many researchers was the determination, in the nonlinear case, of conditions for one-to-one correspondence between the observation process and its innovation. These conditions were obtained in 1969 for the case
Dynamic programming provided a general methodology for handling optimal feedback control when stochastic disturbances are present and was soon used to solve the linear quadratic gaussian (LQG) optimal control problem. However, as in the deterministic case, there was a need to place dynamic programming for general nonlinear diffusion processes on a rigorous footing; this was done in the 1973 paper of Davis and Varaiya on dynamic programming conditions for partially observed processes using advanced Martingale methods. Much of the theoretical work on stochastic control, concerned as it is with control of systems described by stochastic differential equations, does not address an important class of problems where ‘randomness’ appears in the form of point processes, such as arrival of customers and production demands. Davis introduced Piecewise-deterministic Markov process in 1984 as a framework for such (nondiffusion) stochastic processes; this framework has since been widely adopted. Other areas of UK research include the spaceship control problem (stochastic control with finite resource) and resource allocation (the ‘multi-armed bandit problem’). An interesting aspect of recent research has been the emergence of new and important application areas such as mathematical finance. UK contributions include the development of singular stochastic
Personal Impressions of the Dawn of Modern Control
control to solve problems of optimal investment with transaction costs, the first use of martingale methods in economic growth theory (for optimal saving) and use of general semimartingale theory to obtain conditions of optimality for the infinite horizon portfoliocum-savings problem. Another interesting application is the internet which is a network used by many users competing for a set of limited resources. The key to efficient operation is allocation of resources to different users during times of congestion. Frank Kelly at Cambridge was the first to view the problem as one of distributed controls to maximize user utility and to show that net user utility is a natural Lyapunov function. Each controller is subject to a different delay; stability was established using results on stability of delay differential equations. Interestingly, this research made it possible to use the multivariable Nyquist criterion as a natural tool for network controller design (Vinnicombe). The research has spawned a new generation of congestion algorithms.
4.
Conclusion
To illustrate the impact of Bellman, Kalman and Pontryagin and coworkers on our subject, I have drawn attention to some areas of research to which the UK has contributed. In doing so, I have neglected whole areas such as the geometric theory of linear and nonlinear systems, identification and adaptive control, stability, dissipative systems, switching systems and hybrid control and mathematical system theory. Our subject has matured enormously during the period surveyed and its literature has become very rich; becoming expert in even one subject, H1 control for example, requires considerable effort. This richness was acquired via many different flavours of research. Perhaps the most important, and most revolutionary, is the introduction of a framework that provides a radically different model or problem formulation. The introduction of state-space models, explicitly by Kalman and implicitly by Bellman and Pontryagin, into the control systems community provided new and simpler ways of looking at old problems and opened up new problems. Of course, the state-space formulation had existed for a long time in mathematics and physics but the control community had been constrained by a restrictive transfer function model. Old problems, such as filtering, became much more transparent, and new research areas, such as optimal control and nonlinear control, revealed. Another interesting example is George Zames seminal
69
paper on H1 control; there we find reasoning leading to the formulation of an optimal control problem more appropriate for achieving robustness than the standard LQG problem that does not address robustness with the same effectiveness as classical control techniques. This paper opened up a wide and fruitful area of research. The geometric theory of linear and nonlinear systems provided valuable new insights; particularly fascinating was the fact that so much of the linear theory could be generalized to the nonlinear case. Nonsmooth analysis widened the class of finite-dimensional and dynamic optimization problems, allowing deterministic dynamic programming to be put on a rigourous footing and permitting adaptation of the maximum principle to a larger class of problems. Piecewise deterministic Markov processes provided a new framework for stochastic control for problems in which randomness took the form of point processes (such as random arrival of customers and production demands) rather than as a diffusion process. Within a new framework (e.g. statespace models), new concepts (such as controllability, observability, input-to-state stability) were introduced to aid analysis and synthesis. A very different contribution might be termed ‘invention’, the discovery of a new controller, filter or identifier that is relatively easily implemented. Normally an optimal solution to the appropriate problem is sought, but in many cases this is not practical. A good illustration is provided by deterministic adaptive control. Obtaining an optimal solution, via dynamic programming, is impossibly difficult so that alternative solutions were sought. The first solution to the model reference adaptive control problem was obtained in 1978 (for continuous-time systems) by Feuer and Morse using a least squares estimator and complex nonlinear damping to achieve global convergence. A second, relatively simple solution for discrete-time systems, obtained by Goodwin et al. in 1980, used normalization to slow down the estimator sufficiently to ensure convergence and was a major milestone in the evolution of adaptive control. Interestingly, the original adaptive controller due to Monopoli, which does not employ normalization, has recently been shown to be semiglobally stable but not globally stable. Another important ‘invention’ is the backstepping solution to nonlinear control. A third type of contribution is ‘whistle-blowing’. It is all too easy to be carried along by exciting discoveries and to suppress our critical faculties. The LQG solution was so captivating that the issue of robustness was neglected for a considerable period; it took the 1980 paper by Doyle and Stein to alert us to this fact and to prepare us for the H1 era. A similar
70
role was played by Rohrs et al. in their demonstration of the lack of robustness of adaptive control; robust adaptive control is still a relevant area of research. Cross fertilization has also aided the development of our subject. An example of this, in an area I have not discussed, was the use by Patrick Parks in 1966 of the celebrated Kalman–Yakubovich–Popov positive real lemma to establish stability of an adaptive control system that he proposed as an alternative to the heuristic MIT rule which was then in common use. New applications also provide a considerable impetus to our field; recent examples include network congestion control, mathematical finance and multiagent formations. In retrospect, the revolutionary period might be regarded as control’s ‘big bang’; it was a unique period, not likely to be repeated, in which researchers had the opportunity to be generalists and research students the ability to find a niche with little or no help from advisers. Progress since then has been spectacular, but not uniform. There have been plateaus such
D.Q. Mayne
as the LQG era followed by significant revolutions such as H1 optimal control and the geometric theory of nonlinear systems; each time we might feel that progress is slowing to a standstill, something new emerges. What Peter Whittle says of research in linear systems is perhaps true for our whole field: ‘‘the linear model seems to have infinite depth, and yields only to reveal further mysteries . . . . its known theory becomes more extensive and definite with time, but somehow never definitive.’’
Acknowledgement I am very grateful to Mark Davis, Keith Glover, Petar Kokotovic, David Limebeer, Alistair MacFarlane, Larry Markus, Alistair Mees, Howard Rosenbrock, Roger Sargent, Malcolm Smith, Richard Vinter, John Westcott and Murray Wonham for their helpful advice. Any shortcomings are, of course, my responsibility.