Some results on optimal control applied to epidemics

Some Results on Optimal Control Applied to Epidemics DAVID GREENHALGH Control and Management Systems Division, Engineering Mill Lane, Cambridge, CB2 IRX, England Received

Department,

19 August 1986; revised I9 August 1987

ABSTRACT This paper deals with a mathematical model for controlling an epidemic by the removal and isolation of infected people. The objective is taken to be to maximize the expected number of people removed at some terminal time. Some simple results are found for a deterministic model with a homogeneously mixing population by using the maximum principle. It is found that the optimal policy with the above objective function is to wait until a switching time and then attempt to remove as many infected people as possible. Next a stochastic model is discussed, and under certain assumptions similar results obtained. For the stochastic homogeneous mixing case the relationship between switching

1.

times, the starting

state of the epidemic,

and the terminal

are the

time is explored.

INTRODUCTION

This paper examines a simple mathematical model for an epidemic which is controlled by removing infected people and isolating them from the remainder of the population. The results which are obtained are intuitively plausible and are backed up by control engineering and optimization techniques applied to the mathematical model. Until recently most of the literature on mathematical models for the spread of disease made the assumption that the population amongst whom the disease was spreading mixes homogeneously. This assumption will not always be appropriate; often differences between individuals will make it more appropriate to assume heterogeneous mixing within the population. This heterogeneity could be due to a variety of factors such as geographical differences, genetic differences, age structure, or some other social structure in the population. Examples of this type of approach are given by Cane and McNamee [l], Rushton and Mautner [2], and Watson [3]. Watson considers a stochastic heterogeneously mixing epidemic model by using simulations and analytic approximations, whereas Rushton and Mautner consider the case of a MATHEMATICAL

BIOSCIENCES

88:125-158

OElsevier Science Publishing Co., Inc., 1988 52 Vanderbilt Ave., New York, NY 10017

125

(1988)

0025-5564/88/$03.50

126

DAVID GREENHALGH

deterministic simple epidemic in m equivalent classes. The model discussed in this paper is based on Cane and McNamee’s model, which will therefore be discussed in detail later. The assumption that the population mixes homogeneously will also not be appropriate for a disease spreading over a large area. In this case geographical differences will mean that the meeting rates will be different for different individuals in the population. This sort of model is discussed by Hethcote [4], May and Anderson [5] and Nold [6]. Another significant cause of heterogeneity in the population is differences in age, particularly as schoolchildren tend to spread many viral diseases such as measles, mumps, and rubella. A separate group of papers considers a population which is divided into several different age classes with a “contact matrix” giving the rates of spread of infection between several different age classes (Anderson and May [7-91, Dietz and Schenzle [lo], Longini et al. [ll], and Schenzle [12]). These papers are aimed more at specific diseases. Longini et al. consider a deterministic model for the spread of influenza and use computational optimization techniques to determine an optimal vaccine distribution between age classes. The first paper by Anderson and May discusses a model in which the population is divided into separate distinct age classes, but these age classes mix homogeneously. The remaining two papers by Anderson and May discuss the distribution of vaccine amongst several populations which mix heterogeneously. In one paper an age structured model is used to describe measles outbreaks: in both the optimal vaccination strategy to eradicate the disease with as small an amount of vaccine as possible is derived. However, the conclusions drawn in these papers are not directly relevant to the work discussed in this paper, for the following reasons. Anderson and May are mainly concerned with the study of endemics, and the continual birth of new susceptibles and death of immune individuals are important in their work. We are concerned with epidemics which occur over a relatively short times&c and so ignore these factors. The paper by Schenzle describes how a detailed age structured model can be fitted to data for measles in England and Wales. The joint paper by Dietz and Schenzle illustrates the practical difficulties involved in fitting data and estimating parameters, particularly for age structured models. So this is one aspect of the research which will be presented in this paper: the replacement of the usual assumption of homogeneous mixing by a more general heterogeneous mixing assumption. A second aspect of this research is the introduction of control engineering and operational research techniques to determine the optimal policy. A comprehensive survey of control theoretic methods applied to mathematical models for pests and infectious diseases up to 1977 was performed by Wickwire [13]. Unfortunately, nearly all of these models assume homogeneous mixing and thus the results are not directly applicable to the models presented here. However, the ideas behind

OPTIMAL

CONTROL

APPLIED

TO EPIDEMICS

127

the techniques, which use the maximum principle and dynamic programming, are similar. Dietz and Schenzle [lo] point out that one disadvantage of applying control theoretic techniques to gain insight into practical control problems is in the commonly made assumption that the people controlling the disease have complete knowledge of the state of the epidemic, because this assumption will not always be true. Wickwire also touches on this problem. The models discussed in this paper are used to explain why certain “obvious” policies, which are intuitively best, are optimal under certain conditions. It is not intended to be a specific model for any one disease, and unlike a simulation model, it could not be expected to give precise numerical values for the expected number of people infected, susceptible, or removed. It was necessary to balance those assumptions which would make the mathematics tractable with more realistic assumptions, and throughout the paper we shall indicate what those assumptions are, when they will be appropriate, and why they were made. This paper considers a disease which is spreading in a heterogeneously mixing community. The paper neglects the effects of the birth of new susceptibles and the death of immune individuals. So it is suitable for modeling a single outbreak of disease which is controlled over a time which is short compared with the lifetime of the population under consideration. The paper also mainly considers an epidemic which is controlled by removal and isolation of infected people, rather than immunization of susceptibles, although a model with immunization is considered briefly at the start. It is of course important to study control of the epidemic by immunization of susceptible people, but this is done elsewhere (Greenhalgh [14]). A further point to make is that in the absence of any control being applied, we study only what is usually described as a simple epidemic model (Bailey [15]). In this type of model there is no removal from circulation by death, recovery, or isolation, and so all susceptibles will ultimately become infected (in the absence of control). Bailey states that these assumptions would be approximately applicable when (1) the disease was highly infectious but not sufficiently serious for fatalities among infected cases and (2) very few infectives became clear of infection during the main course of the epidemic. Bailey suggests that such a model would be suitable for diseases of the upper respiratory tract. Thus all of the models discussed in this paper assume that there is no natural recovery from infection (in the absence of treatment). It would be useful if these results were extended to the situation where those people who were infected later became immune to the disease naturally in the absence of any treatment being applied. This would widen the scope of applicability of the results to the common childhood diseases such as measles, rubella, and mumps. However, it has not yet proved possible to do this, mainly because

DAVID GREENHALGH

128

the mathematics appears intractable. Incidentally, Cane and McNamee’s model does allow one to take account of degrees of natural immunity when a person is immune to the disease at the start (this could be due to genetic or other factors). We now briefly outline the contents of the paper. In the following section the basic model is outlined and some simple results derived for the deterministic homogeneous mixing case. We consider mainly trying to maximize the expected number of people removed at some terminal time; the justification for this is given at the start of Section 3. It is found that the optimal policy is to wait until a switching time and then apply full removal effort to the epidemic. In the next part of the paper we consider replacing the homogeneous mixing hypothesis with the hypothesis that the population mixes heterogeneously. This is based on an assumption described in a paper by Cane and McNamee [l]. Section 3 sets up a stochastic model based on Cane and McNamee’s assumption and proves some elementary results. In Section 4 it is shown that under these assumptions the optimal policy is to wait until a switching time and then apply full removal effort to the least infectious infected person in the affected population. In the last section, the relationships between this switching time, the starting state of the epidemic, and the terminal time are explored.

2.

THE DETERMINISTIC

HOMOGENEOUS

MIXING

CASE

Here we discuss control of an epidemic spreading in a homogeneously mixing population which is controlled by both immunizing susceptible people and isolating infected people and removing them from the rest of the population. Let: x(t) y(t) z(t) r(t)

= = = =

number number number number

of of of of

susceptible people at time t; infected people at time t; immune people at time t; people removed at time t.

Suppose that an amount u(t) of immunization effort and an amount p(t) of removal effort is applied to the epidemic at time t. Then the equations for the spread of the disease are taken to be k=-crxy-uZ(x>O), j= 2s

“V

-

I.4 Y >O),

uZ(x>O),

(2.1)

i=j.hZ(y>O) where a: > 0 is the constant meeting rate and Z(A) is the indicator function of the event A. These equations ensure that the values of x and y remain positive throughout the epidemic. It is more usual to replace the term

OPTIMAL

CONTROL

APPLIED TO EPIDEMICS

129

pI(y > 0) by py, both to take account of the fact that as infected people become fewer it becomes harder to find and isolate them (Bailey [15]), and to take account of the fact that for many diseases infected people naturally become immune after a certain period of time. A similar problem arises for the immunization term in that it is more usual to replace ul(x > 0) by ux to take account of the fact that as susceptible people get fewer it becomes harder to find them and treat them. However, this problem is less serious than the corresponding problem for the removal term outlined above. This model is also discussed with these modifications at the end of this section. Here we have used the expression “immunization effort” to denote any treatment effort applied to susceptible individuals. It is not possible to vaccinate against most diseases for which there is no natural immunity (although there is talk of developing a vaccine for AIDS). The model would be applicable to any situation where some treatment was applied to susceptible individuals which either reduced their chances of catching the disease or prevented them from catching it. Thus the model would be applicable to sexually transmitted diseases such as AIDS and herpes if we interpreted vaccination as taking measures such as better health education and supplying barrier contraceptives etc. We suppose first that there is a limited time T for which the doctor can control the epidemic. To specify a suitable objective function we take into account the following costs: (1) a cost a for each person infected at the end of the epidemic; (2) a benefit b for each person immunized at the end of the epidemic; (3) a cost c per unit time of applying treatment effort; (4) a cost e per unit time of applying removal effort; (5) a cost f per unit time of one person being infected (for example, this could correspond to the economic cost of this person being off work); (6) a benefit g for having removed one infected person at the end of the epidemic. A justification of this objective function is given at the start of Section 3. There is a maximum amount a0 of immunization effort available and a maximum amount CL,,of removal effort available to apply to the epidemic at any one instant. THEOREM

2.1

(a) Suppose that a > 0 orf > 0. For the aboveproblem the optimal policy is to apply full immunization effort up to a switching time t, and from then on apply no immunization effort, and to apply full removal effort up to time t, and from then on apply no removal effort. (b) In the case where a = f = 0 the optimal policy again applies full immunization effort up to a switching time t, and from then on applies no

130

DAVID

immunization

GREENHALGH

effort, but it either applies full removal effort all of the time or

applies no removal effort all of the time, according as g is greater than or less than e. Proof. We shall prove result (a). Result (b) is proved similarly. Suppose that a > 0 or f > 0. This result is proved by a straightforward application of the maximum principle. The problem can be formulated as follows: Choose u(t) and p(t) in [0, T] subject to 0 6 u(t) < u. and 0
subject to the equations function as

/[0

(2.2)

(2.1). We can use (2.1) to rewrite

the objective

r olaxy - apZ( y > 0) - buZ( x>O)+cu+ep+fy+guZ(y>O)]dT +

ay(0) - bz(0) + gr(0).

So we must minimize

/[0

’

aaxy - apZ( y > 0) - buZ( x>O)+cu+ep+fy+guZ(y>O)]dT.

Then the Hamiltonian

ff(x,y

can be written as

,z,u,P,t)

= aaxy - auZ( y > 0) - buZ( x>O)+cu+el+fy+gFZ(y>O) +P2(axy)+P*(-axy)+(P4-P,)CLZ(Y>O)+(P3-P1)uZ(x>O) =u[c-bZ(x>O)+(p,-p,)Z(x>O)]

+P[e-aZ(y>O)+gZ(y>O)(p~-p,)Z(y>0)1 +fy+(pz-pl)a(xy)+“axy, where pl, pz, pJ, and p4 are respectively and r. The adjoint equations are

the variables

adjoint

to x, y, z,

A=-ta+p,-p,)ay, A=-f-(a+p,-p,)ax, PS = 0,

(2.3)

j_Q= 0 with terminalconditionsp,(Z’)=p,(T)=p,(T)=p,(T)=O.

Thus ps and

OPTIMAL

CONTROL

APPLIED TO EPIDEMICS

131

p4 are zero throughout

the course of the epidemic, (To be rigorous, what is meant here is that these equations hold in each of the four regions x > 0, y > 0; x > 0, y < 0; x < 0, y > 0; and x < 0, y < 0. Then the definition of the adjoint variables as marginal increments of the optimal value function implies that the adjoint variables are continuous across the boundaries of these regions.) The optimal control at time t is the control which minimizes the value of the Hamiltonian at time t. It is given by

tl=

0

if

c>(b+p,)Z(x>O),

i u.

if

c<(b+p,)Z(x>O);

if

e>(a-g+p,)Z(y>O),

if

e<(a-g+p,)Z(x>O).

pc1= O i p.

(2.4)

Now

Expressing this equation we deduce that a-p,+p,=aexp

in terms of s = T - t (the time to go) and solving it,

‘a(~-y)d[ V0

1

+fi’exp

[$

-/‘cY(x-Y)&

1

dq.

a - p1 + pr > 0. Thus from the adjoint equations

Hence

(2.3) we deduce that both p1 and p2 are monotonic decreasing to the value zero [using the terminal conditions in (2.3)]. Looking at the criterion (2.4) it can be deduced that full immunization effort should be applied up to the switching time t,, which is the first time that either x = 0 or p1 < c - b, and full removal effort should be applied up to the switching time t,, which is the first time that either y = 0 orpz
The idea behind Theorem 2.1 is that to minimize the number of people infected it is more effective to apply immunization and removal effort earlier in the epidemic than later. Note that the adjoint equations (2.3) do not depend on c, the cost per unit time of immunization, or e, the cost per unit time of removal. If c and e are small enough, then the best policy is to apply full immunization effort until there are no susceptible people in the epidemic and apply full removal effort until there are no infected people in the epidemic. As c increases, the immunization switching time also decreases

132

DAVID

GREENHALGH

until eventually no immunization effort is applied at all. A similar argument applies to the removal switching time t,. These results relate to work which we shah discuss later in Section 3 and Section 4. POSSIBLE

EXTENSIONS

As noted before, it is possible (and for the removal rate it is usual) to replace the terms involving p and u in (2.1) by py and ux respectively. We now summarize the results obtained when this was done, although these are less complete than the previous results. THEOREM

2.2

(i) Suppose that it is required to control the epidemic by immunization so as to prevent people from catching the disease at minimal treatment cost. Then it is appropriate to minimize the objective function ay(T)-bz(T)+ircu(T)dT.

The optimal policy is to apply full immunization effort up to a switching time t, and then to allow the epidemic to progress by itserf. (ii) Suppose that it is desired to remove as many people as possible. Then we want to maximize

d T) The best policy is to wait until a switching time and then apply full removal effort. The above results are all proved using the arguments of Theorem 2.1, and they can be extended to more general situations. They hold if there is no limit to the time for which immunization can be applied to the epidemic (so that T = 00). They also apply if ua, the maximum amount of immunization effort, and pa, the maximum amount of removal effort, are functions of time rather than constants. These results tie in with other results in the literature. Hethcote and Waltman [16] consider a similar problem with a simpler cost function but additional constraints. They use dynamic programming methods to solve the problem numerically and deduce that the vaccination rate should be highest at the start of an epidemic. Gupta and Rink [17] consider the same basic model as Hethcote and Waltman solved by the same basic methods. However, they consider vaccination by two different types of vaccine and take the cost associated with each to be a quadratic function of the amount of vaccine used. Their conclusions depend heavily upon estimating the parameters of the model.

OPTIMAL

CONTROL

133

APPLIED TO EPIDEMICS

Morton and Wickwire [18] consider the cost of controlling an epidemic by immunization. They take the cost of an epidemic to be the integral of the sum of the number of people infected and a term proportional to the immunization rate. The model for the spread of the disease is the same as Hethcote and Waltman used but the vaccination rate (Y is restricted by 0 < OL< 1. They show that an optimal policy to control the epidemic will apply at each instant either all or no immunization effort, and that amongst all policies with only finitely many discontinuities the optimal policy has only one switch, which is from applying full treatment effort to applying none, Finally, Sanders [19] looks at the control of an eye disease spreading amongst a tribe of American Indians, using a discrete time version of the Kermack-McKendrick model. The cost is of the same form as that discussed by Morton and Wickwire discounted over successive time periods. Sanders concludes, as Morton and Wickwire do and we also do, that an optimal policy always applies either all or no removal effort and that it is better to apply removal effort at the start of an epidemic. The review paper by Wickwire [13] lists some of the above results and also one or two other results which support our conclusions for different epidemic models. 3.

THE STOCHASTIC

HETEROGENEOUS

MIXING

CASE

The heterogeneous mixing epidemic model which we are about to examine was originally formulated to study how different social habits would affect the spread of influenza and how far a meeting rate estimated for one city might reasonably be applied to another of a different social structure (Cane and McNamee [l]). In this model each person is assigned a parameter X which denotes both his innate capacity for infectiousness and his innate predisposition to infection. The population consists of n individuals Zi, Z2,. . _, Z,,, where n is large, each of whom acts independently. In a small interval of time [t, t + At),

It is assumed that when a susceptible person meets an infected person, then the susceptible person is infected instantaneously. It is convenient to think of Xi as the proportion of his time that individual Z, spends in some communal meeting place, and 1 - X, as the proportion of his time that he spends in isolation. If Z, is infected and Zj is susceptible, then Z, will infect 4 after a time which has an exponential distribution with parameter XiXj. The epidemic is supposed to be controlled by removing infected people and isolating them from the rest of the population. If an amount p of removal effort is applied to an infected individual, then he or she will be removed after a time which has an exponential distribution with parameter

134

DAVID GREENHALGH

p.

Removal effort has effect only when applied to an infected individual. However, we wish to allow the amount of removal effort applied to vary with time. Hence we suppose that if an amount p,(r) of removal effort is applied to the infected individual 1, in the small time interval [T, T + AT], then he or she will be removed in that time interval with probability

Events in successive time intervals are independent. However, in practice there will be some physical or economic constraint on the rate at which infected people can be removed from the epidemic. To take this into account we suppose that there is a maximum total rate p0 at which removal effort can be applied at each instant so that at each instant T

We shall now discuss in general terms the model which we shall be analyzing in the remainder of the paper. The first point to make is that we shall consider only controlling the epidemic so as to maximize the expected number of people removed at some terminal time. At first sight this seems a curious objective function. It is more natural to take the objective to be to minimize the expected number of people infected at some terminal time; this problem is discussed in [14]. Consider a disease which is not in itself serious and which we wish to encourage either because it is preferable for individuals to catch the disease whilst the epidemic is being treated or because having had the disease provides protection against some more serious disease. If we decide to maximize the expected number of people removed at some terminal time, then we are maximizing the protection in the community against the disease. The community is protected in two ways: first, the people who are immune will never catch the disease in the future, and second, this mass of immune people will act to slow down the future spread of the disease. The best example of this sort of application is to diseases amongst cattle and other livestock. A particular example is East Coast Fever (Purnell [20]). This is a tick borne disease which is common amongst cattle on the east coast of Africa. It is a highly prevalent disease which is invariably fatal and to which there is no natural immunity. Treatment therefore consists of deliberately infecting cattle and injecting them with a drug which cures them of East Coast Fever and renders them immune to subsequent reinfection. There are two points which must be made. First, the cattle will die naturally from the disease in the absence of any treatment. However, as the disease spreads very quickly, it is reasonable to assume that this mortality will not

OPTIMAL

CONTROL

APPLIED

TO EPIDEMICS

135

have a great effect and thus our model will be approximately applicable. Second, it is more convenient in practice to infect the herd in groups, infecting the whole of each group simultaneously, rather than letting the disease spread by itself. This aspect is not covered by our model. We now briefly mention some other examples where we wish to encourage the spread of disease so as to build up immunity in the population. German measles (rubella) is not a serious disease if caught in childhood, but if a pregnant woman catches rubella in the first three months of pregnancy it can have serious effects on her unborn child. Similarly, mumps is not a serious disease for young children, but can have complications involving sterility for postadolescent males. Other common childhood diseases, such as measles and chicken pox, are more serious if caught later in life. So for each of these diseases there is an incentive to encourage the spread of disease amongst young children. However, these diseases are nowadays mostly controlled by vaccination (although vaccines for measles and mumps are quite a recent development), and all have natural immunity, so our model would not be directly applicable to them. Another situation where we wish to encourage the spread of disease amongst livestock is coccidiosis, which is a parasitic disease amongst battery farmed chickens, caused by ingesting the parasite coccidia. The disease spreads naturally amongst the chickens very rapidly, and they eventually develop a natural immunity to it. However, unless they catch the disease in a controlled environment they will develop large parasite burdens and this will stunt their growth. It is unacceptable to vaccinate the birds, as they are ultimately for human consumption. So here a reasonable objective would be to maximize the number of birds who had experienced the disease when treatment was stopped. Our model will not be exactly applicable to this situation, as it does not take into account the fact that the birds develop natural immunity in the absence of treatment. It would be desirable to introduce this feature into the model, but it appears to make the mathematics intractable. It is usual in these sorts of problems to assume that the policy used may depend on the time and the state of the epidemic, with the latter being specified by the number of people susceptible, infected, and immune. However, in this paper it is assumed that this policy can depend only on the time, although it turns out that removal effort must be applied to the least infectious infected person in the epidemic. For the heterogeneous mixing case this raises problems in addition to the fact that the state of the epidemic is often incompletely known. In practice it would be difficult to know which person was the least infectious infected person without knowing precisely who was infected. The assumption that the removal effort which could be applied could depend only on the time would be appropriate if a team of doctors were employed to control the epidemic for a fixed time, and the

136

DAVID GREENHALGH

periods for which each doctor was employed had to be determined in advance, but the precise state of the epidemic was known at each instant. Then the removal effort applied at each instant could depend only on the time, and yet whilst the epidemic was in progress all of this removal effort could be applied to the least infectious infected person in the epidemic. A more practical justification for the work which we are about to present is that this objection does not hold for the stochastic homogeneously mixing epidemic and our results are more complete for that case. To be fair, it should be stated that the original reason why it was stipulated that the removal effort could depend only on the time was that this problem is more mathematically tractable than assuming that the removal effort can depend on both the time and the current state of the epidemic. NOTATION n denotes the total number of people amongst whom the disease is spreading; R(t), I(t), and X(t) are respectively the numbers of people removed, infected, and susceptible at time t. We denote by S a state of the epidemic and by Cp the set of all possible states. A state S is determined by knowledge of the infected, susceptible, and removed people in the epidemic, and the parameters Xi are assumed known for each individual. Suppose that Ip is any susceptible individual and Zq is any infected individual in the state S. Let SP denote the state obtained from S by IP becoming infected, and S, denote the state obtained from S by removing I,. Thus for example S: denotes the state obtained from S by IP becoming infected and I, being removed; if I, is any other susceptible individual in the state S, then Sr’ denotes the state obtained from S by both Ip and Z, being infected, and if I, is any other individual in the state S, then Sq, denotes the state obtained from S by both I4 and I, being removed. More generally, suppose that P is any set of susceptibles and Q is any set of infected people in the state S. Then Sp denotes the state obtained from S by each person in the set P becoming infected, and S, denotes the state obtained from S by each person in the set Q being removed. The obvious convention applies for multiple combinations of superscripts and subscripts; for example, Sp denotes the state obtained from S by each person in the set P being infected and I, being removed. A policy for treatment will be a function u: 0 x [O,T] + R”, where ui( S, t) is the amount of removal effort applied to I, at time t if the epidemic is then in the state S. Let E[REM(~),u: S] be the expected number of people removed from the epidemic at time t if the epidemic starts in the state S and the policy u is used. E[REM(t),u :S, r] denotes the expected number of people removed from the epidemic at time t given that it is in the state S at tune T and the policy ZJis used. u(t) denotes a typical policy, and

OPTIMAL

CONTROL

APPLIED TO EPIDEMICS

137

u*(t) the optimal policy for 0 Q t < T. P( S’, u, t: S, 7) is the probability that the epidemic is in the state S’ at time t given that it was in the state S at time r and the policy u is used. rem(S) is the number of people removed in the state S. Suppose that the epidemic starts in the state S, and the policy u is used. The policy u can be taken to consist of two parts: a function [0, T] + R” giving the amount of removal effort to apply at each instant, and a function depending on the current state of the epidemic telling whom to apply this removal effort to. First we shall consider the optimization over the second half of this function. LEMMA

3.1

Let A denote the set of infected people in the state S, and B denote the set of susceptible people in the state S. If I, is a susceptible individual in the state S, then let [t,(S) denote the total rate at which the individual Ir is infected by the infected people in the state S.

(i) Suppose that u is a policy which applies an amount ui of removal effort to individual Z, in the small time interval [0, At), where At is small and positive. Then E[REM(A~),u:S]

=rem(S)+

c ujAt+o(At) rcl

if the state S contains infected people, and E[REM(A~),u:S]

=rem(S)

if the state S contains no infected people. (ii) Suppose that the state S contains m infected people (m > 0) and u applies an amount u of removal effort to Ik, the least infectious infected individual in A. Then E[REM( At), u: S] =rem(S)+~At-(~At)m+l + jP+t(

ml.+5p(S)+~r(Sk)+~r(Sk,)+

...)

Atm+2+~(Atm+3).

Here I, is the second least infectious infected person in S. Proof. The first result is clear by considering the events which could happen in [0, At). To prove the second result we again first condition on the events which could happen in [0, At). Writing SP = tP(S) for notational

DAVID GREENHALGH

138 convenience,

+ 1-pAtWe deduce that

,u:s] = c

[pb[REM(t),u:SP]

PEA +jd[REM(t),U:&]

-(~+~~~,!E[~M(f),u:S].

(3.1)

Putting t = 0, this proves the result for m = 1. Differentiating (3.1) s - 1 times with respect to t, we can prove by a double induction on m and s that

$E[REht(t),u: s] has the required value at t = 0 for s = 1,2,. . . , m + 2 and thus prove the statement of the lemma. The next lemma says that the expected number of people to be removed is less if the epidemic starts in a state with either one more person infected or a person replaced by a more infectious one. LEMMA

3.2

Suppose that u is a policy which at each instant applies removal effort only to the least infectious infected person in the epidemic. X is a state of the epidemic reachable from S,,, and P is a set of susceptible individuals in X, I/ and Ik are infected persons in X with A, < A, and T > 0. Then (9

E[REM(T),u:x~]

>E[REM(T),u:X]

with equality if and only if u applies no removal effort in [0, T] or the set P is empty; 69

@EM(T),+‘]

2E[REM(T),u:Xk]

with equality tf and only if the set P is empty and Xj = A,, u applies no removal effort in [O,T), or X contains no susceptibiepeople I, with A, > 0. Proof. We first discuss the proof of (ii) and note that the result (i) is proved similarly. The result is obvious if X contains just one infected person. If X contains no susceptible people or the set P consists of all the

OPTIMAL

CONTROL

APPLIED TO EPIDEMICS

139

susceptible people in X, then the result is clear by conditioning on the number of removals to occur in [0, T). Thus one can assume inductively that the result is true whenever X is replaced by a state containing fewer susceptible or fewer infected people or the set P is replaced by a larger set of susceptibles. If T = AT is small and positive, then the result is true by Lemma 3.l(ii). It is clear that the conditions which are given for equality are sufficient. Hence it is necessary only to check that if these conditions are not satisfied, then the result holds with strict inequality. Define

We know that t > 0. We shall suppose that t < CQand deduce a contradiction. This is done by showing that t),u:

+EM(

x,‘]

> E[REM(

t),

u: &].

Continuity then contradicts the definition of t. Let Q denote the set of remaining susceptibles in X excluding P; Z the set of remaining infecteds excluding I, and Ik. If 1, is a susceptible individual and I, an infected individual, then let tx, denote the rate at which I, is infected by IV. So if the epidemic is in the state X,!‘, then infections happen at rates .$,, for XEPUQ, yEZU{I,}. As a trick to balance the rate at which events happen we introduce extra “infections” which occur at rate txY for x E P, y E Z U {I,} if the epidemic is in the state X,‘, and X,(X, - X,) for q E Q if the epidemic is in the state X,. These “infections” produce no effect on the state of the epidemic. Let Z, be the least infectious infected person in the state X,‘, and Z, be the least infectious infected person in the state X,. We will use the inductive hypotheses to advance the induction assumption. Note that as the result is trivial if the state X contains just one infected person, we can assume that the state X contains at least two infected people. Conditioning on the first event to occur in the epidemic when it is in the state Xjp, we deduce that E[REM( T), U: x!‘] =

c

(a)

(5,,+~,p+Eqk)E[~~(T),~:~pq,7]

qEQ

+(~pr+Spj)E[REM(T),U:XP,T] +~E[REM(T),u:~;J

x

exp[ - ( &I

+ &P

(b)

]I +

5‘Qk +

IP,

+

6P,

+

P)

71

1 dT

(4

140

DAVIDGREENHALGH

Here we have written

for convenience

of notation

tAB = COG,.,, bG B[ob,

L,

=LEAEob, and so on. Similarly conditioning on the first event to occur in the epidemic when it is in the state X,, we deduce that J+EM(T),u:&]

) +rem(X,)exP[-(~pr+leP+~ak+Epy+~pj+p)T]. Xev[- ( tQr+ Epp+tQk+L+bj+~)~]

We compare

these two expressions +EM(?-),u:X;]

dT

term by term to show that 2E[REM(T),u:&].

(a) For qEQ, E[REM(T),u:X,?,T]>E[REM(T),u:X$,7] bytheinduction hypothesis applied to state X4. Also E[REM(T),u : X,?,T] 2 E[REM(~'), u: Xj, T] by the induction hypothesis for (i) applied to the situation where the set P is replaced by the larger set P U {q }. (b) E[REM(T), u: X,?, T] > E[REM(T),u: X[, T] by the induction hypothesis applied to the state XJ’. (c) We require to show that

E[REM(T),u:x,;,T] >E[REM(T),u:X,,,T]. Thisresult follows by noting that as X, is the least infectious infected person in X,, either X, = h, or I,E I,whence X, < X, < X,. In either case the result will follow from the induction assumptions (d) Clearly rem( X,!) = rem( X,). Thus

made.

E[REM(T),u: X,?]>E[REM(T),u:X~], and the result is proved.

A crucial idea in the proof of the above result was the idea dominance. Suppose that we are given two stochastic processes, example the epidemic starting in two different states, which compare so as to m aximize some given objective. Suppose that to pair the set of all possible realizations of the two stochastic

of stochastic in the above we wish to it is possible processes so

OPTIMAL CONTROL APPLIED TO EPIDEMICS

141

that both realizations in any pair have the same probability of occurrence and for each pair the outcome of the first stochastic process is preferable to the outcome of the second. Then it is clear that the overall expected outcome of the first stochastic process is preferable to the overall expected outcome of the second. Note that to set up the situation described above it was necessary to introduce an imaginary null event into both the epidemics starting in the states X,? and X, to balance the rates at which events happen. 4.

THE OPTIMAL

POLICY

We have already stated that the policy u can be regarded as consisting of two parts: the first telling how much removal effort to apply, and the second whom to apply it to. The epidemic is being controlled so as to maximize the expected number of people removed at some terminal time. The following theorem gives the optimization over the last part. THEOREM

4.1

The optimal policy applies removal effort only to the least infectious infected person in the epidemic. Proof. The proof of this result is straightforward and will be given in outline only. Given any policy, define a new policy as follows. The new policy agrees with the old policy unless the epidemic is in the starting state. When the epidemic is in the starting state, the new policy applies the same amount of removal effort but applies it all to the least infectious infected person in the epidemic. The idea is to compare this new policy with the old policy. Proceeding to compare these policies by stochastic dominance as in Lemma 3.2 and using the results of Lemma 3.2, the result follows. The next lemma is needed to show that the optimal policy has the form “wait until a switching time and then apply full removal effort to the least infectious infected person in the epidemic.” The idea behind the lemma is that if a small amount c As of removal effort is moved to later in time by an amount At, then the policy is improved. LEMMA

4.2

Suppose that u is a policy which always applies removal effort only to the least infectious infected person in the epidemic. u can be characterized by a function [0, T] + R giving the amount of removal effort to apply at each instant. Suppose also that this function is piecewise constant on half-open intervals. Let v be the policy which applies exactly the same amount of removal effort but has moved an amount L As of removal effort forwards from time [q,q + As) to time [q+At,q+As+At) where C> 0. Then thepolicy v is at least as good as u to the first order in As and At.

DAVID GREENHALGH

142

Proof. Suppose that the epidemic starts in the state S,,. Conditioning on the state X of the epidemic at time 9, it is necessary only to compare E[REM( T), u : X, q] and E[REM(T), u : X, 111for each state X reachable from S,,. This will be done explicitly only in the case where X contains both susceptible people Z, with X, > 0 and more than one infected person, as the details for the other case are similar. Let Zk be the least infectious infected person and Z, the second least infectious infected person in the state X. For each susceptible ZP in the state X let t,, denote the rate at which ZP is infected. Suppose that u applies an amount a + E of removal effort in [q, 17+ As) and an amount a of removal effort in [ 7 + As, v+ As + At), and u applies an amount a of removal effort in [ 9, q + At) and an amount a + c of removal effort in [ 77+ At, q+ As + At). Table 1 gives the probabilities for the first event to happen to first order in As and At. By using Table 1 to condition on the events to occur in [q, q + As + At ) and noting that the policies u and u agree on [ VJ+ As + At, T) we deduce that

+rAs

1

E[REM(

T)

,u:x[,q+As+Ar]}}

p~A&Af{E[R(T),~:Xk,$+A~+At]

-E[R(T),u:X/‘,~+As+Ar]}) +o(As)o(At).

(4.2)

We adopt the convention that Zk denotes the least infectious infected person in the state being considered: in other words, in Xi, Zk denotes the least infectious infected person in the state Xp, and in X,, Z, denotes the least infectious infected person in the state X. We assert that Equation (4.2) holds to second order in As and At. This can be seen by putting the second order terms in Table 1. Equation (4.2) was derived for the case where the state X contains susceptible people Z, with X, > 0 and at least two infected people. More generally suppose that we define a function A = A( T, u, q + As + At, X) as follows: (1) If X is a state not reachable from S,, using the policy u in [0, TJ) or a state in which the disease has stopped spreading, then A = 0.

OPTIMAL

CONTROL

APPLIED

TABLE Probabilities

143

TO EPIDEMICS 1

of Events in [q, q + As + At) to First Order Probability

First Event

Policy -

Second Event

Null

Null 1-nAt-x($,+A,h,)Ar

l-(a++-&+h,h,)As P

P

I-~AI-~([,+X,h,)At

l-(a++-x([,+k,&)As

P

P

Infection

of Ip

(6, + A,&)

Null I-uAt-

As

2

(E,+X,h,+X,X,)Ar

q+P (Z,

+ +A)

I-(a+c)As-

At

c

([,+X,h,+X,h,)As

qfP

Null 1-(a

Infection

+ c) As - x(6,

+ h,X,)

As

of Ip

(6, + $,A,)

At

P

lh~Ar-~($,++~h,)Ar

I-

P

x(6,

+ $,h)As

P

Infection

of Ip

(E, + Api,) (&+X,h)At

As

Infection (6, + h,h (I, + A,&

Removal

of I, + X,X,) Ar + $,kJ As

Null

(a+c)As

I-aAt-&At P

aAt

1-(a++-&As P

Removal

Infection

(a+c)As

&At

UAl

Z,AJ

Removal (a+r)As

of Ip

Removal uAt

aAt

(a + <) As

Null

Removal

l-(a++-~($+X,h,)As

UAf P

l-aAt-&+XpXk)Ar

(n+c)As

P

Infection (.$,+$h)As (5, + &h)

of I, Al

Removal aAt (a + c) As

DAVID GREENHALGH

144 (2) If X does not satisfy (1) but contains

only one infected person, then

(3) Otherwise

Then A B 0 by Lemma 3.2 and for any state X with P( X, u, 11:S,) > 0, u: X,9]

E[REM(~‘),u:~,v]-E[REM(T),

=AAhsAt+o(As)o(At).

Suppose that we define

where the sum is taken over all states X. Then

and E[REM(T),

u: s,,] - E[REM(T),

zx&,] =BAsAr+o(As)o(At).

Moreover it is straightforward to show that B is continuous arguments. The result of Lemma 4.2 follows.

in each of its

Suppose that u(5) is the policy which agrees with u except that it has moved an amount EAs of removal effort from time [q + 5,~ + [ + As) to time [q + 5 + A[, 11 + 5 + A[ + As), and define

B(T,~(~),~,Y:S)=CP(X,U(~),~:~,)A(T,U(~),Y:X). X It is straightforward

to show that B is continuous

in each of a, B, and y.

OPTIMAL THEOREM

CONTROL

APPLIED TO EPIDEMICS

145

4.3

The optimalpolicy waits until a switching time [(T, SO) and then applies full removal effort to the least infectious infected person in the epidemic. Proof. Suppose that u is a general policy which applies removal effort only to the least infectious infected person in the epidemic. We use Lemma 4.2 to continuously deform u into a policy of the required form, increasing the expected number of people removed and keeping the total amount of removal effort applied the same. Suppose that we are given a policy u which applies an amount a + E of removal effort in [n, n + As) and an amount a of removal effort in [ 7 + As, c) for some c. We want to examine the effect of moving an amount CAS of removal effort from time [TJ,q + As) to time [n + 5 - As, 17+ [). Let this effect be G(t; As). It is straightforward to show that c(&As)=--JrB(T,u(B-As),~+B-As,~+8,&,)d~As AS

and the integrand is an integrable function of 8. Next we want to extend this result. Given a policy u as above, we want to consider the effect of moving an extra amount es of removal effort from [TJ,n + s) to [q + 6,~ + 5 + s). Let u(a) be the policy which has moved an amount EU of removaleffortfrom[n+s-a,q+s) to[q+[+s-a,n+[ + s). Let G(n, u, E) be the effect of moving an extra amount cu of removal effort from [q+s--x,q+s) to [n+[+s-u,n+[+s). Then arguing as above, we deduce that

which is negative. So the effect of moving an amount cs of removal effort from [TJ,7 + s) to [n + 5,~ + 4 + s) is to increase the total expected number of people removed from the epidemic. Now suppose that we are given a policy u which is piecewise constant and at some stage decreases the amount of removal effort which it applies. u is characterized by a sequence of blocks of removal effort which at some stage decreases. By switching successive pairs of these blocks around we obtain a policy which applies the same amount of removal effort as the original and is monotonic increasing in the amount of removal effort it applies. Next we consider the effect of moving some of this removal effort to a later time. Suppose that u is a policy which applies an amount a + c of removal effort in [n, 17+ AZ) and an amount b of removal effort in [TJ+ AZ, n +2 AZ). We want to compare this policy with the policy v which applies an amount a of removal effort in [q, 7 + AZ) and an amount b + c

DAVID GREENHALGH

146

of removal effort in [r] + AZ, 9 + 2 AZ). We can perform a similar analysis to that of Lemma 4.2 to compare E[REM(T), u: So] with E[REM(T), u : S,,]. Consider the probabilities of the transitions in the time interval [q, q + 2 A z) to first order in AZ. For all the separate possibilities for the state X,

Multiplying by P( X, u, q : S,,) and summing deduce that

over all possible

E[REM(T),u:S,]-E[REM(T),U:S,]

states X, we

=0(h).

Now we can commence the next stage of the argument. Suppose that and we wish to compare a policy u which applies an amount a of removal effort in [ 7, [] with a policy u which is the same as u except that in [ q,t] it applies an amount 2a - p of removal effort in [ 7, (5 + 9)/2) and an amount ~1 of removal effort in [([ + 17)/2,[). For notational convenience write x = (i + q)/2 and let u(a) denote the intermediate policy which has moved an amount (II - a)a of removal effort from [x - u, x) to [ 5 - u, t). Thus u(u) applies an amount 2a - p of removal effort in [x - u, x) and an amount p of removal effort in [[ - u, 4) for 0 G u G q. Let G(u) denote the effect of moving this removal effort on the total expected number of people removed. A straightforward modification of the previous argument thinking of this removal effort as moving forwards in three stages:a > p/2,

(1) from [x - u - Au, x - a) to [x - Au, x), (2) from [x - Au, x) to [x, x + Aa), (3) from [x, x + Au) to [[ - u - Au, [ - a), -shows

that

which is negative, and so v is a better policy than u. Now suppose that a < p/2. A similar argument shows that if u applies an amount a of removal effort in [v, 0, then a better policy is one which applies no removal effort in [ q,( 5 + q)/2) and applies an amount 2a of removal effort in [( .$+ TJ)/~, .$), but is the same as u outside the interval 1179(1. We can now complete the proof of Theorem 4.3. Suppose that we are given a policy u which is characterized by a piecewise constant function [0, T] + [0, pO]. Then u can be thought of as a sequence in time of blocks of removal effort, and we know that these blocks can be arranged in ascending order to obtain a better policy. Then we can perform one of the above

OPTIMAL

CONTROL

APPLIED TO EPIDEMICS

147

operations on each of the blocks of removal effort in the new policy. If this block applies an amount of removal effort greater than p/2, then we transform it to one applying an amount /J half of the time, and if this block applies an amount of removal effort less than k/2, then we transform it to one applying no removal effort half of the time. This gives a new policy u1 which has a greater expected number of people removed, applies the same total amount of removal effort as the original policy, and applies either all or no removal effort at least half of the time. Repeating this process, we get a sequence of policies U, such that E[REM(T), u, : So] is a monotonic increasing function of n, each policy applies the same total amount of removal effort as the original, and the length of time where the policy u, differs from the proposed optimal policy is at most T/2”. It is straightforward to show that E[REM(T), u: So] is monotonic increasing to E[RJzM(T), u* : So] as n tends to infinity. Thus E[REM(T),u:S,,]

~E[REM(T),u*:&]

with equality if and only if S, is a state in which the epidemic is not still spreading or the policies u and u* are identical. This completes the proof of Theorem 4.3. This result was proved only in the case where the policy u was characterized by a piecewise constant function [0, T] + [0, ~~1. It is straightforward to extend the result to the case where u is characterized by a piecewise continuous function. COROLLARY

4.4

Suppose that it costs an amount cper unit time to have a doctor available to treat the epidemic. There is a benefit d associated with each person removed from the epidemic, and the epidemic is to be controlled so as to minimize the expected total cost. Then the optimal policy waits until a switching time [(T, S,) and then applies full removal effort to the least infectious infected person

in the epidemic.

This result relates to the results for the deterministic homogeneously mixing epidemic model which was discussed earlier in the paper. It would be very desirable to extend this work to the situation where the people amongst whom the disease was spreading also recovered from the disease naturally in the absence of any control effort being applied. It would then be possible to apply the results to common childhood diseases such as measles and mumps. However, it has not yet proved possible to do this.

5.

RELATIONSHIPS

BETWEEN

SWITCHING

TIMES

Let u(t) denote the policy which waits until time 5 and then applies full removal effort to the least infectious infected person in the epidemic. If we

148

DAVID GREENHALGH

define G(t) then it is straightforward

=-+M(T),@):$,],

to show that

G’(~)=I~CP(X,~,,E:S,)[E[REM(T),~,:X,~] X -E[R@T),u,:&,t]],

(5.1)

where u,, is the policy which applies no removal effort to the epidemic and ui is the policy which applies full removal effort to the least infectious infected person in the epidemic. Recall that we use the notation that the individual Ik is the least infectious infected person in the state X and X, denotes the state X with I, removed. The sum is again taken over all states X. Consider the version of Cane and McNamee’s model which corresponds to homogeneous mixing. This version has all the A’s equal: X, = h, = X, = . . . = A,,. We require the following condition for discussion: CONDITION

5.1

Suppose that u is a policy which applies full removal effort if the time to go lies between T - At and T for At sufficiently small: (i) If Z, and I4 are susceptible

people in the state S, then

E[REM(T),u:S]+E[REM(T),U:SP~] >E[REM(T),u:SP]+E[~M(T),u:S4]. (ii)If Zp is a susceptible

and I, is an infected person in the state S, then

E[REM(T),u:SP]+E[REM(T),U:~]
E[REM(T),u:S~,]

+E[REM(T),u:S]


We restrict our attention to the homogeneous mixing case where X, = X, = . . . = A,. The first point to note is that Condition 5.1 is a genuine condition and is not true in general, as the following counterexample shows: Counterexample 5.2. Consider first Condition 5.1(i). Assume first that the state S of the epidemic contains no infected people and r susceptible people besides Ip ad Iq. Suppose also that the orders of magnitude of h, p,

OPTIMAL

CONTROL

APPLIED TO EPIDEMICS

149

and T are given by h = r, p = 1, and T = l/e’, where E is small and positive. Then it is straightforward to show that if r is large enough and z is small enough,

In fact a similar argument shows that all three inequalities fail under these conditions even if the state S contains more than one infected person. The more infected people the state S contains, the more likely this result is to be true. The next obvious question to ask is: under what circumstances is Condition 5.1 likely to be true? Suppose that Condition 5.1(i)-(E) are true whenever the state S contains no infected people. It is possible to show that they will then be true in general. This is done along the lines of Lemma 3.2. It is possible to use this fact to argue heuristically as follows: It is a standard result (Whittle [21], Bailey [15]) that for the stochastic homogeneous mixing case which we are considering, there is a threshold density of susceptibles, p. If at the start the number of susceptibles is above this threshold value, then a major epidemic outbreak is likely to occur however many susceptible people there are. If the number of susceptible people at the start is less than p, then a major epidemic outbreak will not occur. If the time for which the epidemic is treated is relatively long, then everyone who becomes infected will eventually be removed. Suppose that n, the number of susceptibles at the start of the epidemic, is larger than the threshold value p. Then an epidemic affecting a large proportion of the population is very likely to occur even if there are only one or two infected people at the start of the epidemic. Thus we would expect Condition 5.1 to be true under these circumstances. These results can be quantified by using Whittle’s stochastic threshold theorem [21] to bound r( (Y),the probability of an epidemic outbreak infecting no more than a fraction (r of the people who are susceptible at the start. Suppose that there are a infected people at the start of the epidemic. The expected number of people ever to catch the disease can be bounded above by rr((~). cm + [l - n( ol)]n and bounded below by [l- n(a)]an. By Whittle’s stochastic threshold theorem we can bound rr((Y).an+[l-a(a)]n above by [

.(l’*)]“~~++~~la)

provided that p < n(l- a). Taking (Y= 0, the expected ever infected can be bounded above by

number

of people

150

DAVID GRQENHALGH

Suppose that there are no infected people and r susceptible people in the state S. Applying the above result to the epidemic starting in the state Sf’q, we deduce that
E[REM(T),u:SPq]

{

l-

KTi.

Applying the lower bound given by [l - TT(cy)]an to the states SP and S‘J, we deduce that E[REM(T),u:SP]+E[REM(T),U:S~]>~~L(~+~) The right hand side is maximized

(r+IiI-a)

over LTE [0,1 - (p/{ P \i r+l

cu,=l-

1’

r + l})] at

.

Its value is then 2crz( r + 1). Hence the first part of Condition the state S contains no infected people and

5.1 will hold if

(5.2) Similarly the second part of Condition only one infected person and 2&>(1-[5]‘)

5.1 will hold if the state S contains

and

$4,

(5.3)

and if this condition holds, then the third part of Condition 5.1 holds. Note that (5.3) implies (5.2). By the remarks made previously, then, if (5.3) holds, then Condition 5.1 will be true however many people the state S contains. Condition 5.1 allows us to derive relationships between the switching times for different starting states of the homogeneous mixing epidemic. First we need some preliminary lemmas: LEMMA 5.3 are monotonic increasing functions of x

Suppose that fi (x), f2 (x), . . . , f, (x) and differentiable with respect to x. Let f(t)

=J:[alf~(t-7)+a2f2(t-T)+

xexp[-(ar,+a,+ +foexp[-(&,+a,+

*.* +%fX-T>I

...

+a,,)~] . ..+a.)t],

dT (5.4)

wheref,(O)=f,(O)=... =f,(O)=f, anda,,...,a,>O. Thenf(t) ismonotonic increasing in t, differentiable with respect to t, and such that f (0) = fO.

OPTIMAL

CONTROL

APPLIED

151

TO EPIDEMICS

Proof. The last two statements are obvious using the definition of f(t). The first statement follows by differentiating (5.4) with respect to t. This derivative is positive, as each f, is monotonic increasing, and so f,‘( t - T) a 0. Hence f is monotonic increasing. Moreover, if for some i one has (Y~> 0 and f, is strictly monotonic increasing, then f is strictly monotonic increasing. LEMMA

5.4

For the homogeneous mixing epidemic

is a monotonic increasing function of T for any state X that contains at least one infected person. Proof. This lemma is proved by induction on the hierarchy of states in a similar manner to Lemma 3.2. The result is immediate if the state X contains just one infected person. If the state X contains m infected people (m 2 2) and no susceptible people, then (conditioning on the next event to occur in the epidemic) the result follows from Lemma 5.3. We assert that if P is any set of people in a general state X, then E[REM(T),u:

x’]

-E[REM(T),u:

Xk]

(5.5)

is a monotonic increasing function of T. The proof proceeds along the lines of the proof of Lemma 3.2, using stochastic dominance and Lemma 5.3 to advance the induction hypothesis first for this case and then for the general case. Note. epidemic,

In fact Lemma 5.4 is also true for the heterogeneous although the proof is more complicated.

Now we derive connections starting states of the epidemic. LEMMA

between

the switching

mixing

times for different

5.5

For the homogeneous mixing epidemic with the assumptions of Condition 5.1, G’( 6) is a monotonic decreasing function of 5. Thus the equation G’( .$‘)= 0 has at most one root. Proof Let 0 d ii < t2 Q T. Recall that G(t) = E[REM(T), u(5) : S,,] and that from (5.1)

G’(h) =~cp(X,uo,&: S,,) X

152

DAVID GREENHALGH

Now for each state X reachable

from S,, using the policy u0 we can write

Here each R is of the form XQ, where Q is a set of susceptibles X. Then it is straightforwards to show from Condition 5.1 that

in the state

But by the previous lemma

Hence

=cLCCP(R,U~,EZ:X,~~)P(X,U~,E~:S,) XR

X{E[REM(T),u,:X,5,]--E[REM(T),U1:Xk,~1]} (as the epidemic must be in some state R at time [,) >~CCP(R,~,,~~:,X,~~)P(X,U~,~~:S,) R

X

~{E[REM(T),u,:R,~,]--E[REM(T),u~:R~,~~]}

=

pCP(R,uo,

52:

So){

[uW(5.7)]

E[=f(T),u,:

R&]

R -E[=f(%u,:&,t,]}

(by the law of total probability)

= G’(c;,> Hence

G’(E,) > G’([2) as required, completing

(by definition). the proof of Lemma 5.5.

It is a corollary of this lemma that G(c) is a concave function of 5 and achieves its maximum over [0, T] at the unique switching time to = .&( T, S,). THEOREM

5.6

Suppose that Condition 5.1 holdr for the homogeneous mixing epidemic, S is a state of the epidemic, IP is a susceptible individual in S, and Ik is the least

OPTIMAL

CONTROL

APPLIED

153

TO EPIDEMICS

infectious infected individual in S. Then (i) &,(T, Sp) Q &,(T, S) with equality if and only if both are zero; (ii) &(T, S) < &,(T, Sk) with equality if and only if both are zero. This theorem expresses the dependence of the switching state of the epidemic if the terminal time T is fixed.

time [ on the

Proof. (i): Consider the epidemic starting in the state SP and using the policy u,, in [0, q), so that the disease proceeds by itself. Label all of the people initially infected in the state S in some way; say we color them red. Suppose that these people start an epidemic, which we call the epidemic of red infected people. Label all of the people who are initially infected in the state SP in some way; say we color them blue. These people start an epidemic, which we call the epidemic of blue infected people. The idea is that we can write P( X, I+,, 7 : So), the probability that the red epidemic is in the state X at time q if we use u0 in [0, n), as

where the sum is taken over all states R reachable from X using uO. Here PX( R , uo, 9 : S) is the probability that the red epidemic is in the state X and the blue epidemic is in the state R at time q. Thus CPX(R,u,,ll:S)=P(R,u,,~:SP). x

(54

Hence if we define G(

6) = E[=(T),

~(6) : S]

and

H(t)

=E[REM(T),u([):SP]

[using the notation defined earlier that u(t) is the policy which waits until time [ and then applies full removal effort to the least infectious infected person in the epidemic], then

G’(5) =&P(X,u&S){ X

J+=(T)+,:

X,6]

-E[@Th:Xd]} Now if 5 E [0, T) then

G’(5) =~~~&(R,uo,t:S){

E[@Th:X,5.]

XR -E[=(T),u1:&,6])

[by(5.1)].

DAVID GREENHALGH

154 (by the law of total probability, R at time .$)

as the blue epidemic must be in some state

>CCP,(R,u,,~:S){E[REM(T),ul:R,~] R X

-E[REM(T),u,: =pCP(R,u,, R

t:S’){

R,,

61}

[using

(5.7)]

E[@T),vR,t]

-E[REM(T),u~:ZQ,[]}

[using(5.8)]

= H’(t). Recall that &( T, S) and &,( T, SP) are the unique zeros of G’( 5) and H’( 5) respectively. We deduce that &,( T, SP) > tO( T, S) with equality if and only if both are zero. (ii): This result is proved similarly. This theorem shows how Condition 5.1 for the homogeneous mixing epidemic determines the dependence of the unique switching time [(T, S,) on the starting state S, of the epidemic. The next theorem shows how this depends on the terminal time T. This theorem says that if the time for which the epidemic can be treated increases, then the optimal switching time also increases, but not by as much as the time for which the epidemic is treated. THEOREM

5.7

For the homogeneous mixing epidemic suppose that TI < T2 and Condition 5.1 holds. Then (i) either E(T2, SO) < 5(TI, S,) or 6 (T,, SO) = 6(T2, SO) = 0; (3 5(T~,s,)-(T,-T,)<5(r,,s,). Proof. Let G,(t) = E[REM(T~), u(5) ple removed at time T,, and Gz( 5) = number of people removed at time T2 epidemic starts in the state S,. Then G{(t) = 0, and [( T2, S,,) is the unique that Git-9

’ G;(5)

: S,], the expected number of peoE[REM( T2), u(5) : So], the expected if the policy u(5) is used and the ,$‘(TI, S,,) is the unique root of root of Gi( 6) = 0. We shall prove

for

EE[O,T~),

which proves the first assertion, and

G;(t)>G;(5‘+T,-T,) which proves the second.

for

6’~ f&T,),

OPTIMAL

CONTROL

APPLIED

155

TO EPIDEMICS

First suppose that [ E [0, T,). Then

G;(t) =&P(R ,Ug,5‘:So){E[REM(T,),u,:X,61 X

-E[dC),u1:Xwt]} >ccCP(X,~,,~:S~){E[REM(T,),~,:X,~]

by (5.1)I

X

- E[-(T&J

i : X,

,t]

} (by Lemma 5.5)

=G;(t> as required.

Secondly

G;(5)

=~cP(X,uo,&

So){ E[n=.@,),u,:

X,(1

X =EL~

c X

P(R,u,,

E[=(Tz),q:

&,i]}

~+T,-T,:x,5)p(x,u,,5:so)

R(X)

(5.9)

~{E[REM(T*),u~:X,~.]--E[REM(T~),U~:X~,~]}

by the law of total probability, where the second sum is taken over all states R(X) reachable from X using the policy z+,. Now

using the time homogeneity of the policy Substituting into (5.9), we deduce that

This completes 6.

SUMMARY

ui and the inequality

(5.6).

the proof of the theorem. AND CONCLUSIONS

In this paper we have explored the optimal control policies for various epidemic models. The population was divided into categories of susceptible people, infectious people, and removed people, and it was supposed that once a person was removed he was then cured and permanently immune to the disease. In the absence of removal effort, infected people were assumed to stay infected forever. For most of the paper it was supposed that the epidemic was being controlled so as to maximize the expected number of people who were removed at some terminal time T.

156

DAVID GREENHALGH

It should be emphasized that the simplified models presented in this paper neglect some features which would be important in many situations, such as the renewal of the susceptible population and the depletion of the immune population by births and deaths. Perhaps a more important factor which is neglected is that the model assumes that in the absence of treatment infected individuals stay infected permanently and do not recover naturally. It would be important if natural recovery could be introduced into the model, but it has not yet proved possible to do this. It is desirable to include this feature as then the results would be applicable to common childhood diseases such as measles. The simple model discussed at the start provides general results and principles concerning the control of diseases. For this simple model with a general objective function we found that the optimal policy was to apply full immunization effort up to a switching time and from then on apply no immunization effort, and to apply full removal effort up to a different switching time and from then on to apply no removal effort. For the model where the objective function is to maximize the expected number of people removed at some terminal time, the best policy is to wait until a switching time and then apply full removal effort. This is true for both the deterministic and the stochastic models. For the stochastic homogeneous mixing case the relationships between different switching times were explored. Under the assumptions outlined in Condition 5.1 it was found that if the time for which the disease is controlled increases, then the switching time also increases, but by a smaller amount. Also if the epidemic were to start in a state obtained from a given state by removing an extra person, then the switching time would increase, and if the epidemic were to start in a state obtained from a given state by one of the extra susceptible people becoming infected, then the switching time would decrease. These results are what one would intuitively expect. For the stochastic epidemic spreading in a heterogeneously mixing population it was found that it was most effective to apply removal effort to the least infectious infected person in the epidemic. This is intuitively plausible, as with the objective function we have specified it is desirable to keep the disease spreading in the early stages so as to have people available for removal and isolation, and also desirable to remove infected people. One point to note, and this point is raised by Dietz and Schenzle [lo], is that it would be difficult in a practical situation to know who the infected people were, because this information would not necessarily be immediately available to the decision makers controlling the epidemic. Similarly it might be difficult to determine the precise values of the parameters Ai. However, this paper does indicate the people on whom attention should ideally be focused. A more serious practical problem is that it was supposed that the control policy which was being used could depend only on the current time and yet

OPTIMAL

CONTROL

APPLIED

157

TO EPIDEMICS

it was assumed that removal effort could be applied to the least infectious infected person. In most practical applications we would need to know precisely who was infected in order to apply removal effort to the least infectious infected person in the heterogeneous mixing case. Thus the conclusions of this paper have more practical relevance for the homogeneous mixing case than for the heterogeneous mixing model of Cane and McNamee. Finally we wish to make two points. First, it could be asked why we did not allow the control policy to depend on the current state of the epidemic. Apart from the practical considerations referred to above, when this is done there are considerable theoretical complications and the optimal policy appears very complex, even if there are only a small number of individuals in the population. Second, it could be asked why we did not take the objective function to be to minimize the expected number of people infected at some terminal time T. This is dealt with by Greenhalgh [14]. The intuitively obvious policy of always applying full removal effort to the most infectious infected person appears to be optimal. I am grateful to Eddie Anderson and Richard Weber for much discussion and criticism of this work. This work was supported by an S.E.R.C. whilst the author was in the Control and Management Cambridge

University Engineering

studentship

Systems Division of

Department.

REFERENCES 1

V. R. Cane and R. McNamee,

J. Appl. Probub. 19A:173-184 and A. J. Mautner,

The spread of infection in a heterogeneous population, (1982). The deterministic model for a simple epidemic in more

2

S. Rushton

3

than one community, Eiometrika 42:126-132 R. K. Watson, On an epidemic in a stratified

(1955). population,

J. Appl. Probab. 9:659-666

(1972). 4

H. W. Hethcote,

An immunization model for a heterogeneous population, Theoret. (1978). R. M. May and R. M. Anderson, Spatial heterogeneity and the design of vaccination programs, Math. Biosci. 72:83-111 (1984). A. Nold, Heterogeneity in disease transmission modelhng, Math. Biosci. 52:227-240

PopulationBiol. 14:338-349 5 6 7 8

9

(1980). R. M. Anderson tive investigations R. M. Anderson populations and Biol. 1~233-266 R. M. Anderson sion: Implications (1985).

and R. M. May, Vaccination against rubella and measles: Quantitaof different policies, J. Hyg. Camb. 90:259-325 (1983). and R. M. May, Spatial, temporal and genetic heterogeneity in host the design of immunization programs, IMA J. Math. Appl. Med. (1984). and R. M. May, Age-related changes in the rate of disease transmisfor the design of vaccination programs, J. Hyg. Camb. 94:365-436

158 10

11 12 13 14 15 16

DAVID

GREENHALGH

K. Dietz and D. Schenzle, Mathematical models for infectious disease statistics, in A Celebration of Sfuristics, IS1 centenary volume (A. C. Atkinson and S. E. Fienberg, Eds.), Springer, New York, 1985, pp. 167-204. I. M. Longini, Jr., E. Ackermann, and L. R. Elveback, An optimization model for influenza A epidemics, Math. Biosci. 38:141-157 (1978). D. Schenzle, An age structured model of pre- and post-vaccination measles transmission, IMA J. Math. Appl. Med. Biol. 1:233-266 (1984). K. H. Wickwire, Mathematical models for the control of pests and infectious diseases: A survey, Theoret. Population Biol. 11:182-238 (1977). D. Greenhalgh, Control of an epidemic spreading in a heterogeneously population, M&h. Biosci. 80:23-45 (1986). N. T. .I. Bailey, The mathematical 2nd ed., Griffin, London, 1975. H. W. Hethcote and P. Waltman, epidemic

model,

Math.

theory

of infectious

Optimal

Biosci. 18:365-381

vaccination

schedules

and its applications, in a deterministic

(1973).

17

N. K. Gupta (1973).

18

R. Morton and K. H. Wickwire, On the optimal J. Appl. Probab. 61622-635 (1974).

19

J. L. Sanders, Quantitative guidelines Biometrics 27:883-893 (1971).

20

R. E. Pumell, East Coast fever: Some recent research 15:83-133 (1977).

21

P. Whittle, The outcome of a stochastic metrika 42:116-122 (1955).

and R. E. Rink, Optimum

diseases

mixing

control

of epidemics, control

for communicable

epidemic-a

Marh. Biosci. 18:383-396

of a deterministic disease

control

in East Africa,

epidemic, programs,

Adu. Par&to/.

note on Bailey’s

paper,

Bio-