Modern control theory and optimal drug regimens. II: Combination therapy

MATHEMATICAL

Modern

Control

Combination J.

67

BIOSCIENCES

Theory

and Optimal

Drug

Regimens.

I I:

Therapy

BUELL

Department

of Electrical

Los Angeles,

Engineering,

University

of Southern

California

Calijbrnia

R. JELLIFFE Department

of Medicine,

Los Angeles,

University

of Southern

California

California

R. KALABA Department

of Electrical

Los Angeles,

Engineering,

University

of Southern

California

California

AND

R. SRIDHAR California

Institute

of Technology,

Pasadena,

Califbrnia

ABSTRACT In this article we show how well-known concepts from control theory are directly applicable to the problem of determining methods for administering a combination of drugs to achieve a desired level for the equivalent amount of any one of the drugs under consideration. The problem of drug administration is shown to be equivalent to a discrete-time optimal control problem. Methods for solving the discrete optimal control problem via dynamic programming are shown.

1.

INTRODUCTION

Considerable attention has been focused in recent years on the mathematical formulation of the drug distribution problem in the human system [l-5]. These problems of the type general theme of this from control theory regimens.

problems often lead to identification and control that are standard in modern control theory. The series of investigations is the utilization of concepts to treat various aspects of optimal drug dosage

Mathematical

Copyright

Biosciences

6 (1970), 67-74

@ 1970 by American Elsevier Publishing Company, Inc.

68

J. BUELL,

R. JELLIFFE,

R. KALABA,

AND

R. SRIDHAR

In this second article in the series, we use a three-compartment of the human

system and consider

how to administer

modet.

a combination

of

drugs to achieve a desired level over time for the equivalent amount of any one of the drugs under consideration. The problem of drug administration

is shown

to be equivalent

to a discrete-time

optimal

control

problem [6] in which the availability of various drug sizes is naturally introduced as a constraint. Methods for solving the discrete optimal control problem using dynamic programming are next pointed out. In this paper, the drug administration problem is formulated as a Future papers will be concerned deterministic optimization problem. with stochastic and adaptive aspects of the problem that naturally occur when more realism is brought into the drug administration problem. It is evident that the papers of E. Kruger-Thiemer on pharmacokinetics have decisively influenced our thinking.

2.

FORMULATION

It is evident from the Appendix of a given drug can be represented

The equivalent is given by

amount

that the distribution and degradation by the following difference equations.

yk+l

=

ayk

+

auk;

Zkfl

=

C_yk

+

bz,

(1) +

CUk.

of the drug in the apparent

(2)

volume

lJk = Zk + Wk where IV, is a known

monotone

decreasing

of distribution (3)

sequence.

It is required that the amount of drug taken orally at time k, k = 0, N - 1 be chosen in a suitable manner such that the equivalent 1,2,..., amount

of drug in the apparent

volume

uk, k = 1, 2, . . . , N be close

(in some reasonable sense) to a therapeutically desirable value t(. An intuitively reasonable criterion for “closeness” is the sum of the squared error. Thus, a cost function of the form

I(%, . . . UN-l) = 1

$(u,- a)”

k=l

(4)

may be required to achieve a minimum by proper choice of the “control sequence” uO, ur, . . . , u~~_~. Furthermore, because of the form in which the administered drug is Mathematical

Biosciences

6 (1970),

67-74

COMBINATION

available,

69

THERAPY

it may be necessary

to require

that

where U is a suitable subset of the nonnegative real line. Thus, if the drug is available only in standard size tablets,

U consists

of certain distinct points on the real line. It is assumed that zO and y,, are known. Substituting for vk from Eq. (3) in Eq. (4), the optimal drug administration problem may be formulated as follows. Given the difference equations (1) and (2) and the values for zO and y,, determine the sequence zlk E U such that we minimize

. . . 9UN-I>= E [Zk- (a - w312.

Z(u,,

(5)

1=1

3.

SOLUTION

USING

DYNAMIC

Define the biological fn(c1, %) =

PROGRAMMING

[7, 81

cost functionf,

WJY UN-_n,u~-_n-,,..., 1 (6) Min

2 1% - (a [ k=N-(n-1)

UN-iEU,i=l,P,...,n

for a process that obeys the difference equations (1) and (2) withy+ and zN_,, = c2. The principle of optimality leads to the Bellman equation fn(e1, CZ) =

= c,

Min {[CC, + bc, + CuN+ - (a - WN--(n__1))]2 *A-“EU +f,-,(ac, + au&n, CC1+ bc, + CU&,,)}

with the initial

condition

fi(ClV c2) =

It is evident

(7)

Min

[cc1 -I-

bc, $ cuN_l - (ct -

wN)]~.

un-_1MJ

that Eq. (8) can be obtained

from Eq. (7) by defining

(8)

a

new initial condition

fo(%c2> = 4.

0.

(9)

RESULTS FOR A SPECIAL CASE

Consider the special case when U is the positive real line and zO + wO Q a. This corresponds to the case where there is no restriction on the amount of drug that can be administered and the equivalent amount of drug in Mathematical Biosciences 6 (1970), 67-74

70

J. BUELL,

the apparent

volume

R. JELLIFFE,

of distribution

R. KALABA,

AND

at the initiation

R. SRIDHAR

of therapy

is no

greater than the desired equivalent amount cc. It is evident from physical considerations that if no drug is administered orally,

the total equivalent

amount

of drug uk in the distribution

space will decrease monotonically and will always be less than CL Hence, to keep Q “close” to tl will require uk > 0. Thus, the minimization indicated in the right-hand sides of Eqs. (7) and (8) will require suitable partial derivatives to be zero. Performing the minimization in Eq. (8) yields for the optimal drug dosage at time N -

1 denoted

by u_:,_,

-

U&l =

be,) -

(10)

Cl

with (11)

fi(c1, 6) = 0. Similarly, optimal

performing

the minimization

in Eq.

drug dosage at time N - n denoted

ug-,= A(u -

WNqn_l)

C

(7) yields

for

the

by u,$_,

-

bc,) -

(12)

Cl

with L&1,

5. ANOTHER

(13)

c2) = 0.

SPECIAL CASE

Consider now the special case when U is the positive real line and z,, + w,, > u. In this case it is straightforward to show that the optimal dose is u;_,

= Rect

1(, c

-

Wit-(n-1) -

w

-

1

Cl

(14)

where Rect[x]

=

0,

x < 0,

x,

x > 0.

(15)

This solution corresponds to administering no drugs until the equivalent amount of the drug of interest in the apparent volume of distribution does not exceed CL Beyond this point, the drug is administered in the manner specified by the previous special case. Mathematical

Biosciences

6 (1970), 67-74

COMBINATION

71

THERAPY

6. DISCUSSION The drug administration optimization

problem.

problem

is formulated

For the cost function

here as a deterministic

defined here, it is shown that

closed-form solutions for the dynamic programming equations can be This is a direct consequence of obtained for several practical situations. the linear equations and quadratic cost in the optimal control problem. It should be pointed out that even if the equations are nonlinear (e.g., due to protein binding effects) and the cost is nonquadratic, the optimal drug dosage can be determined numerically. The deterministic formulation assumes that the amount of each drug in the gastrointestinal tract and the apparent volume of distribution is known at each instant of time. In reality, these quantities are known only approximately. The approximate knowledge of these quantities can be introduced in the formulation by treating y, and zO in Eqs. (1) and (2) as random variables with appropriate distributions, and the sequence zk The expected value of a cost similar in Eq. (3) as a stochastic sequence. to Eq. (4) is now minimized. The solution of this stochastic problem will require only minor modifications in the development in this article and will be pointed out in a forthcoming article. Another interesting realistic feature that has to be introduced is that the quantities k; in Eqs. (A2) and (A3), which are a measure of the kidney function, are not really constants but can change from day to day. The current values of these parameters have to be estimated based on drug concentrations in the urine, and these estimated values have to be used to determine the current drug dose. This leads to problems in adaptive control [6-91, the formulation and solution of which will be pointed out in a forthcoming article. ACKNOWLEDGMENT This work was supported in part by the National Institutes of Health under grants no. GM 14633-02, GM 16197-01, HE 09428, and HE 11475. REFERENCES 1 E. Kruger-Thiemer, Formal theory of drug dosage regimens, I, J. Theoret. Biol. 13(1966), 212-235. 2 E. Kruger-Thiemer, Formal theory of drug dosage regimens, II: The exact plateau effect, J. Theoret. Biol., to appear. 3 R. Jelliffe, A mathematical analysis of digitalis kinetics in patients with normal and reduced renal function, Math. Biosci. 1(1967), 305-325. Mathematical

Biosciences

6 (1970),

67-74

72

J. BUELL,

R. JELLIFFE,

R. KALABA,

AND R. SRIDHAR

4 R. Bellman, J. Jacquez, R. Kalaba, and B. Kotkin, A mathematical model of dmg distribution in the body: Implications for cancer chemotherapy, Proceedings 3rd George Thieme, Stuttgart, 1964. international congress of chemotherapy. 5 J. Buell, R. Jelliffe, R. Kalaba, and R. Sridhar, Modem control theory and optima1 drug regimens, I: The plateau effect, Math. Biosci. 5(1969), 285-296. 6 R. Bellman, Adaptive control processes-A guided tour. Princeton Univ. Press, Princeton, New Jersey, 1961. 7 R. Bellman, Dynamic programming. Princeton Univ. Press, Princeton, New Jersey, 1957. 8 R. Bellman and R. Kalaba, Dynamic programming and modern control theory. Academic Press, New York, 1965. 9 R. Bellman and R. Kalaba, Quasilinearization and nonlinear boundary-value problems. American Elsevier, New York, 1965. APPENDIX Difference

Equation Representation

for a Three-Compartment

Model for Drug Accumu-

lation and Disappearance

Letp drugs,p = 1,2,. . . , P, be used singly or in combination for therapy. The following assumptions are usually made to determine a mathematical model. (i) The drugs do not interact with each other. (ii) At each instant of time, the combination of drugs present can be expressed as an equivalent amount of any one of the drugs. The three-compartment model for drug p is shown schematically below.

m:(t)

= amount of drugp

in the gastrointestinal

tract at time t,

m:(t) = amount of drugp

in the apparent volume of distribution

m:(t) = amount of drugp

eliminated at time t,

at time t,

k:(t) = rate constant for absorption into the apparent volume of distribution drug p, k:(t) = rate constant for elimination for drugp,

for

g’(t) = rate at which drugp is taken orally at time t. The differential equations corresponding

s_ dt

Mathematical

Biosciences

to the schematic diagram are --kgrni

+ g”(t)

(Al)

drny = kirnz - k:rny. dt

(A21

dm: = k’m9 dt 11’

fA3)

6 (1970), 67-74

COMRINATION

73

THERAPY

The equivalent amount of drug n in the apparent denoted by m&(t) is given by

volume of distribution

at time t

P

m&(t) = z 6zm;(t)

(A4)

%I=1

with s;=

1.

It will be assumed that the therapy will be initiated at time t = 0 when m:(O) and my(O) are known. It will also be assumed that the drug corresponding top = n will be administered for t > 0 and the effect of all the drugs present will be lumped by considering the equivalent amount of drug n. Hence g”(t) E 0,

t>

0,pZn.

(A3

From Eqs. (Al), (A2), and (A5), it is evident that forp # n, we have a system of homogeneous differential equations with known initial conditions. They can be solved either numerically or analytically. Hence, it is reasonable to assume that ml(t), m:(t), t > 0, p # n are now known functions of time. From the structure of Eqs. (Al) and (A2) it is evident that m:(t), p # n will be monotone decreasing functions of time. We now consider the case when p = n. Let us assume that the drug n is administered orally at time t = 0 and at fixed instants of time t = kT, k = 1,2, , . . , N thereafter. Henceg(t) takes the form g"(t) =

f u&t

- /CT)

W)

kc0

where uk is the drug amount administered at time kTand d(t) is the Dirac delta function. The solutions of Eqs. (Al) and (A2) at the instants t = kT when the input is given by Eq. (A6) can now be directly written as ml;(k i- IT) = 1% + mz(kT)] exp(-k,T),

(-47)

m:(k + 1 T) = exp(-k,T)m:(kT) $1

?-

--Y

exp(--klT){exp[-_(y

- VT1 - 111~~+ mz(kT)]

(A8)

where y=-. For ease of notation,

k,

(A9

kl

define

c = y l--Y

yk = mXkT),

(AlO)

Z~ = m;(kT),

(All)

a = exp(-k,T),

(AW

b = exp(-k,T),

(A13)

exp(-k,T){exp[-(y uk = al.

- l)T] - l},

(A14) (AW

Mathematical Biosciences 6 (1970), 67-74

74

J. BUELL,

R. JELLIFFE,

R. KALABA,

AND R. SRI;)HAR

II

Then, Eqs. (A7) and (AS) can be rewritten as

yk+l = ay, + auk,

6416)

ZL?,l= C,Vk + bZk+ cut. In vector matrix notation,

(Al7)

Eqs. (A16) and (A17) can be rewritten as x’s+1= Axk + bu,

(Al@

where Xk =

wyk,

4.

b = co&?, c), a

0

(A19)

1 .

A= [c b

Equations (A16) and (A17) or equivalently Eq. (A18) will be used as the difference equations governing the relationship between the amounts of drug n in the gastrointestinal tract and apparent volume of distribution at time t = (k + 1)Tand the same amounts at t = kTand the amount of drug ingested orally at time t = kT. From Eq. (A4), the equivalent amount of drug n at time t = kT is given by m&(kT) = m:(kT) -i- 2 aimT(kT).

(A20)

9=1 Pf72

Define wK =DfQn;(kT),

(.421)

v&n uk = m$(kT).

‘322)

From Eqs. (A20), (A21), (A22), and (Al 1) follows Ok= Zk + I”*. Note that wk is a known sequence that is monotone

Mathematical Biosciences 6 (1970), 67-74

(A23) decreasing;

that is, w~+~< wli.