- Email: [email protected]
Blood platelet production: Optimization by dynamic programming and simulation
Computers & Operations Research 34 (2007) 760 – 779 www.elsevier.com/locate/cor
Blood platelet production: Optimization by dynamic programming and simulation René Haijemaa,∗ , Jan van der Wala, b , Nico M. van Dijka a Department of Quantitative Economics, Faculty of Economics and Econometrics, University of Amsterdam, The Netherlands b Department of Mathematics and Computing Science, Eindhoven University of Technology, Eindhoven, The Netherlands
Available online 25 May 2005
Abstract Blood platelets are precious, as voluntarily supplied by donors, and highly perishable, with limited lifetimes of 5–7 days. Demand is highly variable and uncertain. A practical production and inventory rule is strived for that minimizes shortages and spill. The demand and production are periodic, as varying over the seven days of the week. Demand for ‘young’ platelets (oncology and hematology) and demand for platelets of ‘any’ age up to the maximal shelf life (traumatology and general surgery) are distinguished. A combined Markov dynamic programming (MDP) and simulation approach is presented and applied to a real life case of a Dutch blood bank. By down-sizing the dimension and applying this combined approach it is shown that order-up-to type replenishment rules that perform quite well can be found. Particularly, a double-level order-up-to rule, so-called 2D rule, is derived, with one level corresponding to ‘young’ platelets and one to the total inventory. This rule is easy to implement and is shown to be ‘nearly optimal’. This approach and the double order-up-to rule seem to be new. The results are most suitable for sensitivity analyses such as with respect to shortages and production costs for blood platelet inventory management. 䉷 2005 Elsevier Ltd. All rights reserved.
1. Introduction This paper will focus on the production and inventory management of platelets at a blood bank, responsible for the supply of platelets to a large number of hospitals. Platelets rapidly deteriorate in quality and can only be used up to an age of 5–7 days. Platelets should thus be regarded as highly ∗ Corresponding author.
E-mail addresses: [email protected] (R. Haijema), [email protected] (J. van der Wal), [email protected] (N.M. van Dijk). 0305-0548/$ - see front matter 䉷 2005 Elsevier Ltd. All rights reserved. doi:10.1016/j.cor.2005.03.023
R. Haijema et al. / Computers & Operations Research 34 (2007) 760 – 779
761
perishable. As platelets are also expensive (the blood bank calculates a selling price of more than 450 euro per patient per treatment), while shortages may involve high extra expenses, if not put lives at risk, production volumes have to be set carefully. A practical and (nearly) optimal trade-off is thus to be found between spill and shortages. Before stepping further into the details of this optimization, let us elaborate on the underlying aspects and our particular motivation. 1.1. Background and motivation The management of blood and blood products is a problem of general interest with a number of complicating aspects. Roughly a distinction can be made in supply, demand and costs. 1.1.1. Supply At the supply side, the necessity of natural human blood remains to be dominant, despite substantial developments for technologically produced substitutes of specific blood products. Therefore, one still relies mainly on the availability of volunteering donors. In most less developed countries (70% of the entire world population), the donors are the major concern, particularly in combination with the high risk of infections and contamination, such as HIV and hepatitis. In developed countries, in contrast, due to the efforts in ‘donor management’, the availability of donors is not a major problem. The supply also depends on aspects ranging from population densities, the infrastructure of transport, the set-up of regional blood banks, the number of hospitals, up to the capacities for processing blood at blood banks and for obtaining blood from donors with limited transportation time frames and capacity constraints (e.g. not at weekends). 1.1.2. Demand At the demand side, first of all, there is the uncertainty in the quantities required during any day, week or month. Even with all detailed information about patients requiring regular transfusions and planned surgeries, in the order of 50% of the demand during a week remains to be uncertain. Secondly, there is the complexity of the various blood types and percentages of exceptions that is involved in forecasting and determining the demand. Finally, restricted lifetimes for usability (perishability) for the various blood products are to be taken into account. Whereas red blood cells and plasma can be kept for months up to over a year, platelets have a restricted shelf life of at most one week. This paper therefore will merely focus on the production and inventory problem of platelets. Furthermore, for platelet demand a distinction can be made between two types of patients. For the first type of patients, mainly oncology and hematology, ‘young’ platelets are highly preferred. These patients are not able to produce enough platelets themselves. For the other type, mainly traumatology and general surgery, platelets of any age up to the maximal shelf life, can be used, since these patients quickly resume their own platelet production. The first type of demand is to a certain extent predictable. Also, these patients are treated on a regular basis, more often on Monday, Wednesday and Friday and seldom in the weekend. The second type of demand is quite unpredictable. 1.1.3. Costs The supply of blood involves a number of different and substantial costs. Among them are general costs for technical development (research) and instruments, for safety reasons and protection against contamination, for transport and infrastructure and for personnel capacities. These costs will not be
762
R. Haijema et al. / Computers & Operations Research 34 (2007) 760 – 779
considered. In this paper, we will only consider costs that are directly related to the production and inventory of platelets, per unit. We will distinguish inventory costs, costs for ‘outdating’ and costs for shortages. While the first component is rather concrete, the second and third are less so. The second also reflects the esthetic value that blood is precious and supplied at voluntary basis for humanity purposes. There should therefore be no ‘unnecessary’ spill. On the other hand, shortages will at best lead to substantial additional costs to obtain blood from another blood bank or by special actions. This trade-off between ‘spill’ and ‘shortages’, even though not in concrete quantifiable numbers, is in fact of crucial importance in the field of blood transfusion medicine. An optimal (or nearly optimal) and practical production policy for platelets is to be developed that takes these ‘costs’ into account.
1.2. Literature and objective The research on the inventory management of perishables in general and blood products in particular started in the 1960s with van Zyl [1] and boosted in the seventies and early eighties, see [2–5]. Nahmias’s review in 1982 [6] on perishable inventory contains nearly 100 references and pays considerable attention to blood bank applications. In 1984 Prastacos [7] gave an overview of the theory and practice of blood inventory management. Platelets have a lifetime of only a few days so that outdating is a real problem. Quite a few papers consider the platelet inventory problem, see e.g. [8–11]. Platelet studies usually deal with blood banks. An exception is Hesse et al. [12] who investigate keeping platelet inventories at hospitals in the Chicago area. Most studies report order-up-to policies evaluated by simulation or by a Markov chain under very restrictive assumptions, such as zero lead time, no periodicity, order-up-to policies only, etc. Some related work is Chazan and Gal [13] and Cohen [14] (considering the age distribution under a base-stock replenishment rule), Lian and Liu [15] (dealing with numerical results for the class of (s, S) production policies), Schmidt and Nahmias [16] and Perry and Posner [17] (who look at (S −1, S) policies with a non-zero leadtime) and Goyal and Giri [18] (describing trends in the modelling of deteriorating inventory). More than one type of demand and different hospital supply rules, as we consider, are rare in literature. Some results are reported by Pierskalla and Roach [19] and Goh et al. [20]. The structure of the problem suggests using a Markov Decision Process (MDP) approach, but the size of the problem makes this impossible for realistic problem instances. Although, with today’s increased computer speed the possibilities are, as we will see, far greater than in the seventies and eighties. Some early MDP formulations can be found in [1,21–24]. A far more recent study is Blake [25] on the platelet production in Nova Scotia, Canada. As an extension of the references mentioned, our platelet problem formulation includes the following four aspects: • the periodicity due to different demand and production characteristics for different days of the week, • two types of competing demands: demand for ‘young’ platelets and demand for platelets of ‘any’ age up to the maximal shelf life, • alternative supply rules for the two types of demand, • the production lead time of 1 day.
R. Haijema et al. / Computers & Operations Research 34 (2007) 760 – 779
763
More precisely, the objective is to extend the type of results from the literature in a number of ways: • to present a dynamic programming formulation for the blood platelet problem which takes these complicating factors of the two types of periodic demand, of alternative supply rules and of the production lead time into account, • to investigate the numerical computation and the structure of an optimal policy for down-sized, but still sufficiently realistic, situations, • to show the ‘near-optimality’ of single and double-level order-up-to policies, • to provide a simulation-based search algorithm to find these order-up-to policies, • to verify that ignoring the different blood types is justified. 1.3. Approach A combined MDP and simulation approach will therefore be followed. This approach consists of the following steps: 1. First, a MDP-formulation is provided for ‘the’ blood platelet problem. 2. Next, for realistic data the state space and the demands are down-sized such that the optimal value(s) can still be computed by standard successive approximation. 3. As a third step, for this down-sized situation it is shown that: (i) the optimal policy can be quite complicated, and (ii) a double-level replenishment rule is nearly optimal. 4. A heuristic simulation-based search procedure is then applied to find the best single (1D) and double (2D) level order-up-to policy. 5. Finally, the quality of these order-up-to rules is evaluated using the MDP. This approach seems to be new in the literature. 1.4. Results Executing these steps for realistic data of a Dutch blood bank led to a number of observations that seem to be of general interest: (1) (2) (3) (4)
The ‘optimal’ production rule for platelet production is complicated and not practical to implement. ‘Nearly optimal’ rules, however, can be found in the class of simple order-up-to rules. A simulation table can provide useful insights. Single level (1D) order-up-to rules may already perform quite well. Double level (2D) order-up-to rules provide a further improvement and can be proven to be ‘nearly optimal’ when distinguishing demand for ‘fresh’ and ‘any’ platelets. (5) The effect of expanding the shelf life from 4 to 6 days reduces costs substantially. (6) If less than 50% of all donor whole blood donations is needed for platelet production, the total production numbers may be determined without paying attention to the different blood types in the inventory. (7) Using sensible cost functions, the number of shortages will be below 1 promille and outdating, about 1%.
764
R. Haijema et al. / Computers & Operations Research 34 (2007) 760 – 779
1.5. Outline The outline of the paper is as follows: First, in Section 2.1, the detailed problem description is given along with a description and illustration of the supply rules. The dynamic programming formulation and computation are presented in Section 2.2, while a discussion on its complexity and its down-sizing can be found in Section 2.3. After a presentation of the realistic test problem in Section 3.1, steps 2–5 of Section 1.3 are executed in Sections 3.2–3.4. Most notably, a comparison of the optimal policy and the simple order-up-to rules is provided in Section 3.4. Finally, in Section 3.5, results for these simple rules are also provided for the original full-sized problem. Section 4 applies the approach and addresses a number of what–if questions of practical interest, concerning the extension of the shelf life time, the sensitivity to cost parameters and the distinction of blood types. An evaluation concludes the paper. 2. The MDP model The problem we are facing is the platelet production inventory problem at the blood bank, with the possibility of production during weekdays but not in weekends. The production becomes available after one day. Furthermore, there are two types of demand: one for fairly young platelets, the other for platelets of any age up to the maximal shelf life. In line with literature, we can formulate this problem, with the extensions of periodicity and two types of demand, quite well in terms of a finite state MDP. 2.1. MDP description and supply rules This MDP model is characterized by its decision epochs, decisions, states, transitions from one state to another and costs. 2.1.1. Decision epochs At daily basis decisions need to be taken on the production volume. During the day a number of things happen. First the inventory is inspected and a production decision is taken. Then demand occurs and is satisfied according to some fixed ‘supply’ rule, as will be specified below. The production of that day is available early in the morning of the next day. We will monitor the system 7 days a week. During the weekends there is no production. Note that as a result of this, the MDP will be periodic with period 7. 2.1.2. Decisions Two decision elements are to be distinguished: (1) a decision about the platelet production size for that day. (2) a decision that follows from a supply rule that prescribes which pools should be taken from stock in oder to fulfill the demand. The first decision element, of how much to produce during a day, concerns the actual decision that is going to be optimized depending on the ‘state’ of the MDP. The amount produced is expressed in ‘platelet pools’, or patient units. A platelet pool consists of the platelets from 5 whole blood donations, thus platelets from 5 different donors. The second decision element, the supply rules, will be considered as different scenarios and will thus not be part of the MDP itself. The supply rule will just appear as an index R (for Rule). This means that for
R. Haijema et al. / Computers & Operations Research 34 (2007) 760 – 779
765
each problem that we consider, we will formulate an a priori ‘supply’ rule for the two types of demand. (The supply rules will be discussed later.) In practice, there are no restrictions with respect to the production capacity nor the storage capacity, except that there is no production on Saturday and Sunday. 2.1.3. States A state description is required so that once the production decision has been taken, the state transitions and the costs during the next period are uniquely determined in stochastic sense. As production is possible only at weekdays and demand varies with the day of the week, the day of the week has to be included in the state description. Furthermore, as the platelets have a finite lifetime, one also has to take into account the residual lifetimes (or ages) of the platelet pools in inventory. A natural instant to describe the platelet inventory is the moment just after the production of the previous day has arrived and just before the production decision for the coming day is taken. With m the maximal residual shelf life of the platelets (between 4 and 6 in our case), the state of the system can thus be characterized by (d, x) = (d, x1 , x2 , . . . , xm ) with d:
the day of the week (d = 1, . . . , 7)
and x = (x1 , . . . , xm ) :
the inventory state, where xr is the number of pools with a residual shelf life of r days.
Note that, since there is no production in the weekend, some of these xr have to be 0 (there can be no platelets with an age that corresponds to production on Saturday or Sunday). Thus the sets of possible inventory states x for a specific day d, denoted by Xd differ. Note that there is no prior information about the demand in the near future other than statistical information in the form of the demand distributions. Hence there is no need to incorporate additional information into the state description. 2.1.4. Transitions The transitions are determined by three elements: the production decision, the demand from hospitals and the rule by which the platelets are supplied. We will separately discuss each of these elements and step by step illustrate the transition mechanism by numerical examples. Assume that we are in state (d, x) = (d, x1 , . . . , xm ) and that the production decision is k. When discussing the transition from one state to the next, we distinguish the following elements: • (No demand) First assume that there is no demand. As the residual shelf life of all inventory decreases by one day, with part of the inventory expiring, the transition from day d to day d + 1 then becomes (d, x1 , x2 , . . . , xm ) → (d + 1, x2 , . . . , xm , k). • (Demand) Demand may occur for two types of platelets: of type ‘any’ and of type ‘young’. Denote by i the demand for ‘any’ and by j the demand for ‘young’. The corresponding demand probabilities
766
R. Haijema et al. / Computers & Operations Research 34 (2007) 760 – 779
at day d are denoted by pd,a (i) : for demand of type any, pd,y (j ) : for demand of type young. • (Supply rules) As there are two types of interfering demands, a supply rule has to specify in which way these demands are satisfied from the available inventory levels. • (Rule for ‘any’) For ‘any’ demand we only consider the natural FIFO rule : oldest platelets first. For example, with m = 6 (the maximal shelf life counted from the moment the pools are added to stock), demand (i, j ) = (7, 0), for inventory state (2, 0, 3, 0, 0, 3) and production k = 2, first the x1 = 2 (oldest platelets) are used, next x3 = 3 and finally 2 are taken from x6 = 3. As 2 pools are produced and all remaining platelets age one day, the state changes from (d, 2, 0, 3, 0, 0, 3) → (d + 1, 0, 0, 0, 0, 1, 2). • (Rules for ‘young’ demand) For ‘young’ demand we consider a family of supply rules; parameterized by a value r and is denoted by FIFOR(r), with r : the minimal residual shelf life required. According to this rule demand for ‘young’ is satisfied by first taking pools with a residual shelf life r, then r + 1 etc., until all ‘young’ demand is satisfied. If the number of pools with a residual shelf life of at least r is not sufficient, we will use pools with residual shelf life r − 1, r − 2, etc. This will be called mismatch. If a demand cannot be satisfied at all, it is called a shortage. (Mismatches and shortages have to be taken into account as they lead to special costs.) In other words, a FIFOR(r) rule first works as a FIFO rule from level r and up, and next, in case the demand cannot be fully met in this manner, as a LIFO rule from level r − 1 downwards. As special cases, this family of FIFOR(r) rules includes the FIFO rule as well as the LIFO rule (youngest platelets first) by FIFOR(m) = LIFO, FIFOR(1) = FIFO. • (Rule for which first) Further, it is assumed that demand for young platelets has priority over ‘any’. (This might seem illogical given the emergency character of ‘any’, but demand for ‘young’ is usually known at the beginning of the day, whereas emergency demand arrives later.) As a consequence, the demand for young platelets is dealt with first. Next from the remaining inventories the demand for ‘any’ is satisfied. • (Supply rule) A supply rule for all demand can thus be specified by a combination of first a FIFOR(r) rule for ‘young’ demand, r = 1, . . . , m, and next a FIFO rule for ‘any’ demand. • (General notation) In general, for inventory state x, and given demand (i, j ), production decision k and supply rule R and at an arbitrary day d, denote by tR (x, k, i, j ): the new inventory state at day d + 1.
R. Haijema et al. / Computers & Operations Research 34 (2007) 760 – 779
767
Table 1 (d, 2, 0, 3, 0, 0, 3) → (d + 1, 0, 2, 0, 0, 0, 5)
(Example) As an example, for m = 6, r = 4, k = 5 and (i, j ) = (1, 4), we obtain the state transition in state (d, x) = (d, 2, 0, 3, 0, 0, 3) shown in Table 1 . 2.1.5. Costs We can distinguish five types of costs: production costs, outdating costs, inventory costs, shortage costs and mismatch costs. Production costs are mainly material and laboratory costs. Therefore the production costs are assumed to be linear in the production size. At present, the hospitals pay more than 450 euro per pool delivered at the hospital. The variable production costs are significantly less, say 150 euro. Outdating costs are the costs incurred when a pool has to be thrown away when the maximal shelf life has passed. If donors are scarce, these costs will be high. If there is ample supply of donors, as in the Netherlands, these costs will be merely the variable production costs. So we take the outdating costs equal to the production costs. In the sequel we will not consider the variable production costs, only the outdating costs, i.e. costs for overproduction. Inventory costs are the costs made to keep the pools under perfect conditions so that they can reach their maximal shelf life. We will ignore the fixed costs and only consider variable costs, linear in the total inventory. Although these costs are not quantified formally, it seems reasonable to estimate them as being between 0.005 and 0.01 times the production costs per unit per day. Shortage costs reflect the fact that when the ordinary supply fails precious time is lost and costly solutions have to be found. This could mean short-term production or getting the platelets from another blood bank. Shortage costs are taken to be linear and are high compared to the other costs. It seems reasonable to estimate them as being roughly 2 to 10 times higher than the ordinary production costs. Mismatch costs, finally, are incurred when there is demand for ‘young’ but there are not enough young platelet pools so that older pools have to be used. As mismatch is accepted by the hospitals the costs should not be estimated as being very high. On the other hand, if this happens frequently the patients will definitely suffer from it. We will quantify them as being somewhat higher than the outdating costs. (Notation) For any inventory state x, production level k and demand (i, j ), denote by cR (x, k, i, j ) := the costs during a day under supply rule R. The expected costs for day d are then obtained as pd,a (i)pd,y (j )cR (x, k, i, j ). cR (d, x, k) := i,j
768
R. Haijema et al. / Computers & Operations Research 34 (2007) 760 – 779
2.2. DP-computation Our focus is to minimize average costs, say, at daily, weekly or yearly level. To this end, and to use standard dynamic programming, define, under supply rule R VnR (d, x) := minimal expected cost over a period of n days starting in state (d, x). Then VnR (d, x) can be computed recursively, starting with V0R (·) = 0, from ⎧ ⎨
VnR (d, x) = min cR (d, x, k) + k ⎩
R pd,a (i)pd,y (j )Vn−1
i,j
⎫ ⎬ d + 1, tR (x, k, i, j ) . ⎭
(1)
The above relation deals with the increase in costs over one day. However, as the problem is periodic with a period of one week, it is more natural and convenient to consider the average costs on a weekly basis. Let us define C R∗ as the minimal average weekly costs under supply rule R:
R C R∗ := lim Vn+7 (d, x) − VnR (d, x) . n→∞
Note that this limit is independent of the initial state as the underlying MDP is ergodic. An approximation of the optimal value C R∗ and a corresponding (nearly) optimal stationary policy R can then be found by applying the so-called Odoni bounds (cf. [26]) and its periodic extension (see Su and Deininger [27]) by determining
R R L(n) := min Vn+7 (d, x) − VnR (d, x) C R∗ max Vn+7 (d, x) − VnR (d, x) =: U (n). (2) x∈Xd
x∈Xd
The lower and upper bound L(n) and U (n) of this inequality are known to converge (exponentially fast) to C R∗ for n → ∞. Thus C R∗ can be approximated arbitrarily closely for sufficiently large n. More precisely that is, up to an accuracy by continuing the successive computation (1) until U (n) − L(n) . Furthermore, an optimal or nearly optimal stationary policy R R R R = (1), (2), . . . , (7) n n n n can be obtained by optimizing the right-hand side of (1) for d = 1, 2, . . . , 7. That is, R n represents a decision rule that prescribes an optimizing decision k for the right-hand side of (1) for any state (d, x). Then, an -optimal stationary policy is guaranteed as for R C R n := the expected costs under the stationary policy n we have C R∗ C R n + when U (n) − L(n) .
(3)
R. Haijema et al. / Computers & Operations Research 34 (2007) 760 – 779
769
Problem: However, for practical purposes and implementation the DP-procedure sketched above has two major complications: (a) the computational complexity and limitations involved for realistic data sizes, mainly due to the number of states, (b) the structure of the (nearly) optimal policy which may not be suitable for practical implementation. In what follows these complications will be dealt with by (a) a ‘down-sizing’ of the size of the problem, (b) a search for ‘simple’ nearly optimal policies. 2.3. Computational complexity and down-sizing For each dynamic programming step (1) one has to consider each day of the week d, each possible inventory state x and all possible production levels k. For each combination (d, x, k) we thus have to compute R cR (d, x, k) and d + 1, t R (x, k, i, j ) . pd,a (i)pd,y (j )Vn−1 i,j
The main complexity problem concerns the inventory vector x. When m = 6, the vector x may have 4 or 5 components that can be nonzero, depending on the day of the week. If each of these components has values between 0 and the number of production levels Mp , then, including the day of the week, the number of states becomes 2Mp5 + 5Mp4 . For a realistic situation with Mp = 50 the number of states is already more than 6 × 108 . Even if we take into account that the total inventory is limited, to 150 say, the number of states is tremendous. Per iteration step one also has to consider all relevant production levels Mp . Furthermore, for each combination of the state and decision, the number of possible transitions is determined by the possible number of combinations of demands (i, j ), Ma My , with Ma the number of values of ‘any’ demand and My the number of values for ‘young’ demand. In our case Ma and My are smaller than Mp , say Ma My = 1000. The overall complexity for a one week iteration is thus of order O (2Mp6 + 5Mp5 )Ma My , in our case
already 3 × 1013 . For cases of a realistic size, we thus have to find other ways than the standard dynamic programming approach above to obtain good policies. The dynamic programming approach is thus limited due to its computational complexity. As a consequence, we can only use the DP-approach for down-sized versions of the original problem. Nevertheless, the down-sized problem will give insight in • the complexity of the optimal policy, • and the near-optimality of simple order-up-to policies. In order to preserve the characteristics of the original problem, we scale all cost factors and we transform the demand figures to distributions which have the down-sized mean and the original coefficient of variation. To this end, we will use a fitting procedure suggested by Adan et al. [28].
770
R. Haijema et al. / Computers & Operations Research 34 (2007) 760 – 779
3. Numerical results, MDP We first consider the down-sized problem, for which the MDP approach is still computationally feasible. From the results, we will learn what the structure of the optimal policy is and whether simple order-upto rules are indeed nearly optimal. We conclude this section with the results for the original full-sized problem. First we describe our base test case. 3.1. Base test case The Netherlands, with 16 million inhabitants, has 4 regional blood banks. Our data are obtained from one of them (Sanquin blood bank, region North-East). The average weekly demand is about 180 platelet pools. Roughly 70% of the demand is for ‘young’ platelets, and 30% is for ‘any’. Demand for ‘any’ is 7 days a week and more or less evenly spread over the week. For the blood bank North-East the mean demand per day for ‘any’ is 8 units. Demand for ‘young’ platelets occurs mainly from Monday till Friday, with means (24, 16, 32, 16, 24, 0, 8). Hence, there are peaks on Monday, Wednesday and Friday. (To simplify the presentation and the down-sizing, the real data, as obtained from Hinloopen [29] are rounded to multiples of 4.) For ‘any’, the assumption of purely random, hence √ Poisson demand, is realistic, which gives a coefficient of variation (cv) for the daily demand of 1/ 8√= 0.35. For ‘young’ the demand is more regular than Poisson. For Poisson the cv would be around 1/ 24 = 0.20. As this demand is less variable than Poisson, we take the cv for demand ‘young’ equal to 0.15, irrespective of the day of the week. From these real data, we derived small test case as shown in Table 2 by down-sizing the instance, as if platelets are issued by units of 4 pools. Then we get an average demand for ‘any’ of 2 per day (i.e. 8 pools) and for ‘young’ platelets (6, 4, 8, 4, 6, 0, 2) for Monday to Sunday. As stated before, we use the fitting procedure suggested by Adan et al. [28] to transform these figures into discrete demand distributions. 3.2. The optimal policy When executing the DP approach for the base test case, we obtain an optimal (may be nearly optimal) policy with a fairly complicated structure. As an illustration, consider the optimal decisions in a number of states at Tuesday morning, all having a total inventory of 12. The optimal policy prescribes to produce • 8 units in the states (0,0,4,0,0,8) and (0,0,5,0,0,7), • 9 units in (0,0,6,0,0,6) and (0,1,5,0,0,6), and • 10 units in (0,0,7,0,0,5), (0,1,6,0,0,5), (0,2,5,0,0,5), (0,3,4,0,0,5) and (0,4,3,0,0,5). Apparently, the optimal decisions are based not only on the total inventory, but also on the age distribution of the inventory. This illustrates the well-known fact that the optimal policy will in general not be a simple order-up-to rule. To investigate this a little further, let us group the states according to their total inventory. 3.2.1. Total inventory procedure Each inventory state is aggregated into an expected total end-of-the-day inventory level by first subtracting the two rounded mean demands and then removing the inventory that has become outdated.
R. Haijema et al. / Computers & Operations Research 34 (2007) 760 – 779
771
Table 2 Base test case Attribute
Reality
Base test case
Average ‘any’ cv(‘any’) Distribution ‘any’ Average ‘young’ cv(‘young’) Distribution ‘young’ Maximum shelf life Policy ‘any’/‘young’ Holding costs Outdating cost Shortage costs Mismatch costs
8 per day 0.35 Poisson (24, 16, 32, 16, 24, 0, 8) 0.15 Fit first 2 moments 6 days FIFO-FIFOR(4) 1 per pool per day 150 per pool 750 per pool 200 per pool
2 per day 0.35 Fit first 2 moments (6, 4, 8, 4, 6, 0, 1) 0.15 Fit first 2 moments 6 days FIFO-FIFOR(4) 4 per unit per day 600 per unit 3000 per unit 800 per unit
Table 3 Order-up-to frequencies for the base case Predicted Up-to 21 20 19 18 17 16 15 14 13 : 0 Total
2
3
4
5
6
7
8
9
10
11
12 1
1 1 12 71 267 3473 5926 90249 0 : 0
1
100000
1 12
13101
26854
30896
3952 15140
2119 1462 3820
13101
26854
30896
19092
7401
1221 449 438
2108
261 125 63
449
71 6 8
85
12
1
Total
For example, consider the case with m = 6 and supply rule FIFO-FIFOR(4). Then, at Tuesday, for inventory state x=(0, 1, 5, 0, 0, 6) mean demands of 2 for ‘any’ and 4 for ‘young’ result in (0, 0, 4, 0, 0, 2) at the end of the day. As there is no outdated inventory, the ‘predicted’ inventory state at Wednesday morning before the replenishment (from Tuesday) arrives becomes (0, 4, 0, 0, 2, 0) with a total inventory of 6. (In a somewhat different setting Pinson et al. [2] investigated the use of weight factors when aggregating the stock, but concluded that this gave no significant improvement.) To visualize the optimal policy Table 3 is created by simulating the optimal policy for the base test case, as obtained from the MDP, for 100000 weeks. It represents all predicted ‘total inventory’ and replenishment level combinations for Tuesday. Each inventory state that occurs during the simulation is mapped into a total end-of-the-day inventory. Also it is registered how many is produced in that state. As a result, we get frequencies for combinations of the two: inventory and production. For example,
772
R. Haijema et al. / Computers & Operations Research 34 (2007) 760 – 779
in Table 3, the number 3820 in row ‘up to 14’ and column ‘6’ means that during the simulation 3820 times a state was visited that corresponds to a predicted end-of-the-day inventory of 6 in which the optimal production volume was 14 − 6 = 8. To be precise, 3820 is the sum of 1962 visits to state (0,4,0,0,0,8) and 1858 visits to (0,5,0,0,0,7). So, in this case, 14 is the target inventory level that will be reached at Wednesday morning if the demands are equal to the expected demand (there will be no outdating). The sum of the Tuesday morning total inventory and the replenishment order, however, is 22! The table shows that the replenishment order-up-to level of 14 for Tuesday fits in 90.2% of the states. But the table also shows some higher order-up-to levels for higher inventories. As inventories that are higher than usual must be the result of lower than average demand, the inventory will also be a little older than average. This indicates that there is some additional need for ‘young’. 3.2.2. Conclusion The optimal policy resembles an order-up-to policy, but it is likely that a more sophisticated order-up-to policy that looks at both the total inventory and the ‘young’ inventory will be even better. This means that hereafter we will be looking for, what we will call, double order-up-to policies with one-level aiming at total inventory and one looking at ‘young’ inventory only. 3.3. 1D and 2D order-up-to policies There are several ways in which we can obtain order-up-to policies. First, we describe how we derive them from the simulation tables directly, but clearly this is possible only for ‘small’ problems for which the dynamic programming approach is still feasible. Then we will introduce the search algorithm that will also be applied to the full-sized problem. We will consider single order-up-to policies called 1D policies and double order-up-to policies denoted as 2D. A 1D policy specifies for each of the 5 production days the order-up-to level and is described in the form of a 5-dimensional row vector 1D policy: S (T) := S1(T) , S2(T) , . . . , S5(T) . The 2D policies are characterized by two 5-dimensional vectors, one for the total inventory level and one for ‘young’ only. (T) (T) (T) (Y) (Y) (Y) 2D policy: S (T) , S (Y) := S1 , S2 , . . . , S5 , S1 , S2 , . . . , S5 . From the simulation tables (cf. Table 3), we see which 1D level fits best, in the sense of being the most frequent. Another way to obtain a 1D level is a quadratic fit procedure that minimizes the quadratic difference between the decision prescribed in the MDP and the proposed level. Similarly, one may use a quadratic fit procedure to obtain a 2D policy. Per day one looks for the combination of levels for ‘total’ and ‘young’ for which the mean quadratic difference between the decision obtained from the two levels and the decision prescribed by the MDP analysis is minimized. (Deriving the most-frequent 2D rule is less trivial.) If we once again look at the inventory state x = (0, 1, 5, 0, 0, 6) at Tuesday morning, then we predict a total inventory x (T) = 12 − 6 = 6 and a ‘young’ inventory x (Y) = 6 − 4 = 2 for Wednesday morning. Then the 1D rule S (T) = (10, 14, 10, 12, 17) in Table 4 suggests to produce
(T) max 0, S2 − x (T) = max{0, 14 − 6} = 8.
R. Haijema et al. / Computers & Operations Research 34 (2007) 760 – 779
773
Table 4 Results base test case Rule
Total
Young
Cost
Difference
MDP Optimum 1D Most frequent 1D Quadratic fit 1D Local search 2D Quadratic fit 2D Local search
— (10, 14, 10, 12, 17) (10, 14, 10, 12, 17) (11, 15, 10, 12, 17) (10, 14, 10, 12, 17) (10, 14, 10, 12, 17)
— — — — (5, 11, 7, 12, 17) (5, 11, 7, 12, 17)
514 529 529 523 515 515
— 2.98 2.98 1.76 0.22 0.22
The 2D quadratic fit rule in Table 4 prescribes to produce
(T) (Y) max 0, S2 − x (T) , S2 − x (Y) = max{0, 14 − 6, 11 − 2} = 9, which, as we have seen, is just the optimal decision in (0, 1, 5, 0, 0, 6). Apart from these approaches, we also propose a local search procedure based on simulation. Local search will also be used here for the smaller problem that is analyzed by dynamic programming, but is especially meant for the larger, real-size problems for which the dynamic programming approach fails. There is a huge literature on highly sophisticated combinations of simulation and search, see e.g., [30,31]. Here the local search method can be quite simple. The structure of the problem is such that typically, decisions are taken day by day, even though today’s decision directly affects tomorrow’s decision. In the early stages of the search, the duration of the simulation runs for 25 years. For the final selection, runs of 1600 years are used. Also note that the final solution is a ‘local optimum’, according to the simulation. (We think that the structure of the problem is such that there are no ‘real’ sub-optimal local optima.) Further, (i) the quality of the data used is never perfect, and (ii) around the optimum small changes of the order-up-to levels will result in very small changes in expected costs. (Notation) Let ed denote the 5-dimensional row vector with all components equal to 0 except for component d which is 1, e.g., e2 = (0, 1, 0, 0, 0). 3.3.1. 1D local search • Initialize. Define S (T) = (0, 0, 0, 0, 0). • Step 1. Replace S (T) by the best improvement S (T) + ed until no further improvement. • Step 2. Replace S (T) by the best improvement S (T) − ed until no further improvement. 3.3.2. 2D local search • Initialize. First execute the 1D local search procedure. Let S ∗ be the best policy found. Define S (T) := S ∗ and S (Y) := S ∗ . • Step 1. This overestimates the levels for ‘young’, thus replace S (Y) by the best improvement S (Y) −ed until no further improvement. • Step 2. Replace the pair S (T) by the best S (T) − ed for d = 1, . . . , 5 until no further improvement. • Step 3. Replace the pair S (Y) by the best S (Y) + ed until no further improvement.
774
R. Haijema et al. / Computers & Operations Research 34 (2007) 760 – 779
3.3.3. Computation times1 Solving the MDP takes about 4 min for the down-sized test case. The 2D local search takes about 1 min. For the original full-sized problem, the MDP approach is not feasible, whereas the 2D local search takes about 5 min, where most of the time is spent in the final selection runs. 3.4. Results The results of these approaches are compared to the results for the optimal policy as displayed in Table 4. For the down-sized base case (at grid 4) the table presents the minimal weekly cost as computed for the MDP via dynamic programming, together with the results for the most frequent 1D rule, the 1D quadratic fit, the 2D quadratic fit and the 1D and 2D rules obtained by local search. (The latter costs are computed by evaluating the corresponding Markov chain.) 3.4.1. Conclusion What appears from the results is that the 1D policies perform quite well but that the 2D policy is even better and very close to the optimum. That, together with the fact that the structure of the 2D policy is on the one hand still quite simple and on the other hand pays attention to the age distribution as well, makes the 2D policy very attractive for practical implementation. The fact that the local search policies do so well is also promising for the large problem instances to be considered next. 3.5. The full-sized problem For the original problem we can no longer compare the results with the exact results from the MDP approach. We can only compare the results obtained via the 1D and 2D local search approaches. Table 5 presents the average weekly costs and the 1D and 2D policies for shelf lives 4 and 6 and supply rules FIFOR and LIFO for ‘young’. We see that the difference in performance between the 1D and 2D local search optima is limited. Looking at the costs per week, we see that the difference between shelf life 4 and 6 is about 40 000 euro annually. Also note that on Thursday and Friday for the best 2D policy in the FIFO-FIFOR situation the orderup-to levels for ‘total’ and ‘young’ are (almost) identical. This is the result of the fact that on these days there is hardly any ‘old’ inventory left, because there has been no production in the previous weekend. Apparently the optimal policy aims at having only ‘young’ at the end of the week! Remark. Note that the straightforward translation of the 1D and 2D policies for the down-sized base case to the original problem by just blowing up the order-up-to levels gets already fairly close to the nearly optimal policy. (Recall that the demand distributions in the down-sized problem and the original problem have the same coefficient of variation.) For instance, blowing up the replenishment levels of the 1D rule for the base case would result in replenishment levels (44, 60, 40, 48, 68) compared to the (45, 62, 45, 49, 70) rule obtained via local search. 1 Processor: CPU Intel Pentium (R) 4 AT/AT Compatible, 261 Mb RAM.
R. Haijema et al. / Computers & Operations Research 34 (2007) 760 – 779
775
Table 5 Results local-search for full-sized problem Shelf life
R rule
4 days
FIFO-FIFOR(2)
FIFO-LIFO
6 days
FIFO-FIFOR(4)
FIFO-LIFO
1D rule
2D rule
% Difference (2D vs 1D)
Cost ‘total’ ‘young’ Cost ‘total’ ‘young’
1288 (41, 60, 44, 49, 65) — 1374 (39, 55, 41, 45, 66) —
1288 (41, 60, 44, 49, 65) (21, 59, 44, 49, 65) 1363 (37, 55, 41, 45, 65) (29, 55, 41, 28, 57)
0.02
Cost ‘total’ ‘young’ Cost ‘total’ ‘young’
560 (45, 62, 45, 49, 70) — 563 (44, 61, 44, 49, 70) —
556 (41, 61, 44, 49, 70) (22, 45, 27, 49, 69) 558 (41, 60, 44, 49, 70) (22, 46, 26, 33, 43)
0.68
0.85
0.89
4. Sensitivity analyses In this section, for the down-sized problem, the sensitivity of the best 1D and 2D policies, and thus the local search methods, is investigated by varying the shelf lives, the supply rules and the costs. Further it is verified that ignoring the 8 blood types is justified. 4.1. Shelf lives and supply rules An interesting discussion in practice is about the importance of extending the maximal shelf life of platelets and about the supply rule for ‘young’. Table 6 presents the results for the shelf lives varying from 4 to 6 days in combination with two supply rules: FIFO-FIFOR(r) with r such that the maximal age that can be used for young (without mismatch costs) is the same in all cases considered and FIFO-LIFO, where the 1D and 2D rules are obtained via local search (LS). We see that the difference between FIFO-FIFOR(r) and FIFO-LIFO is small and decreases with the maximal shelf life. From a practical point of view, the difference is already negligible for a maximal shelf life of 5 days. Again, we see that the 2D-policies as obtained via local search perform excellently. We also conclude that the relative difference between the best 1D and 2D rule increases if the shelf life is increased. Also the LIFO-LIFO combination has been investigated. It appeared that irrespective of the maximal shelf life, this rule leads to an enormous increase in outdating. The FIFO rule for ‘any’ is needed to reduce the outdating! Another interesting aspect is, how are the costs split over the four categories ‘inventory’, ‘outdating’, ‘shortage’ and ‘mismatch’. Table 7 shows these results. Not surprisingly, the split is nearly identical for the MDP and 2D LS policies. The 1D LS policy shows, as expected, higher mismatch and inventory costs that are partially compensated for by the outdating and shortage costs.
776
R. Haijema et al. / Computers & Operations Research 34 (2007) 760 – 779
Table 6 Varying shelf lives and different supply rules for ‘young’; MDP versus Local Search Shelf life
R rule
MDP
1D LS
2D LS
4 days
FIFO-FIFOR(2) FIFO-LIFO FIFO-FIFOR(3) FIFO-LIFO FIFO-FIFOR(4) FIFO-LIFO
1181 1222 719 724 514 515
0.23 2.04 1.08 1.39 1.76 2.14
0.19 1.15 0.13 0.50 0.22 0.27
5 days 6 days
Table 7 Cost split
Average weekly cost Inventory cost Outdating cost Shortage cost Mismatch cost
MDP
1D LS
2D LS
514
523
515
69.28% 12.20% 18.48% 0.04%
69.63% 11.20% 17.79% 1.39%
69.34% 12.22% 18.44% 0.00%
4 days
5 days
6 days
1181
719
514
28.95% 40.55% 30.50% 0.00%
49.47% 37.33% 13.14% 0.06%
69.28% 12.20% 18.48% 0.04%
Table 8 Shelf life and cost split
Average weekly cost Inventory cost Outdating cost Shortage cost Mismatch cost
The cost distribution does however strongly depend on the shelf life as is illustrated in Table 8. The shorter the period platelets used, the higher the costs for outdating. 4.2. Costs We are also interested in the sensitivity of the optimal solution with respect to changes in the cost parameters. For the base test case, with a weekly demand of 44 units of 4 pools, 30 for ‘young’ and 14 for ‘any’, Table 9 reports the average numbers of outdates, shortages and mismatches per week (all numbers are in units of 4 pools). First of all, note that the number of shortages is extremely small, about 0.03 units of 4 pools per week, so about 0.13 (4 × 0.033) pools per week. That is, about 6 times per year. Increasing the shortage cost to
R. Haijema et al. / Computers & Operations Research 34 (2007) 760 – 779
777
Table 9 Cost parameter sensitivity; exact results from MDP with mean demand of 44 # Outdates
# Shortages
# Mismatches
Base case
0.1039
0.0322
0.0003
Holding cost 2 instead of 4 Holding cost 8 instead of 4 Mismatch cost 400 instead of 800 Mismatch cost 1200 instead of 800 Shortage cost 5000 instead of 3000
0.1041 0.1035 0.0914 0.1041 0.2363
0.0304 0.0335 0.0319 0.0322 0.0063
0.0000 0.0008 0.0153 0.0000 0.0008
5000 reduces the number of shortages to once a year (at the cost of additional inventory and outdating). Note that all fractions for shortages are clearly below 1 pro mille (0.044). The table also shows that halving or doubling the holding cost has hardly any effect. An increase of the mismatch cost also has practically no effect. A decrease of the mismatch cost so that it gets below the outdating cost does have an effect. The optimal production volume is in some states relatively low, so mismatches occur in order to get rid of the inventory before it gets outdated. 4.3. The different blood types So far the fact that there is not just one type of platelet pool has been ignored. In reality, the blood types O, A, B and AB, and the Rhesus D factor are also important. Fortunately, rarely an exact donor–patient matching is required. Compatibility is the standard, which means that O can serve all four blood types, A can serve A and AB, etc. Similarly, Rhesus− can help both Rhesus− and Rhesus+. The main group of Rhesus− patients that should not receive Rhesus+ are women in the fertile age, between 15 and 45. So, only about 25% of all Rhesus− patients really need Rhesus−. Using detailed simulation we have verified that in our context the restriction to one blood type is allowed, see Haijema et al. [32]. In this study, the donors are assumed to show up according to 8 different Poisson processes with given means. If more donors show up than needed for production, then the blood types to be used are selected in the order O−, O+, A−, etc. Further, it is assumed that demand is according to 16 Poisson processes (distinguishing ‘any’, ‘young’ and blood type). Using compatibility, it could be concluded that in case only 50% of all whole blood donations are needed for the production of platelet pools, the increase in costs is less than 1%.
5. Evaluation In this paper the periodic production and inventory problem of a blood bank is studied, with two types of demand, one of which requires fairly ‘young’ platelets. The MDP approach is applied to a down-sized, but still quite realistic version of the problem. The results show that the optimal policy is quite complicated, and also that simple 1D and 2D order-up-to policies are nearly optimal. A fairly straightforward local search method is developed to find these order-up-to policies. Particularly, the best 2D policy is within 1% of the optimum. The local search method also works quite well for the original full-sized problem.
778
R. Haijema et al. / Computers & Operations Research 34 (2007) 760 – 779
As an important application it is shown that an increase of the shelf life of platelets from 4 to 6 days substantially reduces outdating and shortages. The results have been obtained without distinction of blood types. It has been shown however, that this is fully justified in the Dutch situation where less than 50% of all whole blood donations is needed for the production of platelets. If, however, the limited availability of whole blood donations leads to restrictions on the production capacity for platelets, this (random) availability has to be (and can easily be) incorporated in the model. Also in that situation there will exist a nearly optimal 2D policy. At present at the blood bank outdating is 20%. Other studies have shown that outdating need not be more than 3%. For the situation in the Netherlands, outdating can be about 1%. A project concerning the implementation of these results at the blood bank will start soon. Acknowledgements The authors are most grateful to professor Dr. Cees Th. Smit Sibinga, Director of Sanquin Consulting Services and advisor of the WHO Collaborating Center for Blood Transfusion and the WFH International Hemophilia Training Centre, and to Ing. Boudewijn Hinloopen (Sanquin blood bank, region North-East) for the fruitful and pleasant discussions and for providing the necessary data. The third author is particularly indebted to Dr. Martin Beun (†) for initially addressing the platelet problem in 1996, as a practicing physician and as a friend. His scientific curiosity stimulated the first steps of this research. He will be kept in memory with deep respect. He should not have died in the spring of 2004. References [1] van Zyl GJJ. Inventory control for perishable commodities. Unpublished PhD, dissertation, University of North Carolina, 1964. [2] Pinson SD, Pierskalla WP, Schaefer B. A computer simulation analysis of blood bank inventory policies. Technical Report, Department of Industrial Engineering and Management Sciences, Northwestern University, Evanston, Illinois, 1972. [3] Jennings JB. Blood bank inventory control. Management Science 1973;19:637–45. [4] Cohen MA, Pierskalla WP. Perishable inventory theory and its application to blood bank management. 1974. Working Paper, Department of Industrial Engineering and Management Sciences, Northwestern University, Evanston, Illinois. [5] Cohen MA, Pierskalla WP. Management policies for a regional blood bank. Transfusion 1975;15:58–69. [6] Nahmias S. Perishable inventory theory: a review. Operations Research 1982;30:680–708. [7] Prastacos GP. Blood inventory management: an overview of theory and practice. Management Science 1984;30:777–800. [8] McCullough J. Platelet production and inventory management. Platelet physiology and transfusion, technical workshop. New Orleans, Louisiana: American Association of Blood Banks; 1978. p. 17–38. [9] Katz AJ, Carter CW, Saxton P, Blutt J, Kakaya RM. Simulation analysis of platelets production and inventory management. Vox Sang 1983;44:31–6. [10] Ledman RE, Groh N. Platelet production planning to ensure availability while minimizing outdating. Transfusion 1984;24:532–3. [11] Sirelson V, Brodheim E. A computer planning model for blood platelet production and distribution. Computer Methods and Programs in Biomedicine 1991;35:279–91. [12] Hesse SM, Coullard C, Daskin MS, Hurter AP. A case study in platelet inventory management. Proceedings of the sixth annual industrial engineering research conference. Miami Beach, Florida: 1997. [13] Chazan D, Gal S. A Markovian model for a perishable product inventory. Management Science 1977;23:512–21.
R. Haijema et al. / Computers & Operations Research 34 (2007) 760 – 779
779
[14] Cohen MA. Analysis of single critical number ordering policies for perishable inventories. Operations Research 1976;24:726–41. [15] Lian Z, Liu L. A discrete-time model for perishable inventory systems. Annual of Operations Research 1999;87:103–16. [16] Schmidt CP, Nahmias S. (S-1, S) policies for perishable inventory. Management Science 1985;31:719–28. [17] Perry D, Posner MJM. An (S-1, S) inventory system with fixed shelf life and constant leadtimes. Operations Research 1998;46:565–71. [18] Goyal SK, Giri BC. Recent trends in modeling of deteriorating inventory. European Journal of Operational Research 2001;134:1–16. [19] Pierskalla WP, Roach CD. Optimal issuing policies for perishable inventory. Management Science 1972;18:603–14. [20] Goh CH, Greenberg BS, Matsuo H. Two-stage perishable inventory models. Management Science 1993;39:633–49. [21] Nahmias S, Pierskalla WP. Optimal ordering policies for product that perishes in two periods subject to stochastic demand. Naval Research Logistics Quarterly 1973;20:207–29. [22] Nahmias S. Optimal ordering policies for perishable inventory—II. Operations Research 1975;23:735–49. [23] Nahmias S. Higher order approximations for the perishable inventory problem. Operations Research 1977;25:630–40. [24] Fries B. Optimal ordering policy for a perishable commodity with fixed lifetime. Operations Research 1975;23:46–61. [25] Blake JT. Using dynamic programming to optimize the platelet supply chain in Nova Scotia. In: Proceedings of the 29th meeting of the European working group on operational research applied to health services. Dlouhý M, editor. Prague, Czech Republic: ORAHS; 2003, to appear. [26] Odoni A. On finding the maximal gain for Markov decision processes. Operations Research 1969;17:857–60. [27] Su S, Deininger R. Generalization of White’s method of successive approximations to periodic Markovian decision processes. Operations Research 1972;20:318–26. [28] Adan IJBF, van Eenige MJA, Resing JAC. Fitting discrete distributions on the first two moments. Probability in the Engineering and Informational Sciences 1995;9:623–32. [29] Hinloopen B. Personal communication. [30] Bertsekas DP, Tsisiklis J. Neuro dynamic programming. Athena Scientific; 1996. [31] Swisher JR, Heyden PD, Jacobson SH, Schruben LW. A survey of simulation optimization techniques and procedures. In: Joines JA, Barton RR, Kang K, Fishwick PA, editors. Proceedings of the 2000 Winter Simulation Conference. 2000. [32] Haijema R, van der Wal J, van Dijk NM. Blood platelet production: a multi-type perishable inventory problem. Research Memorandum, FEE, University of Amsterdam, 2004.
Blood platelet production: Optimization by dynamic programming and simulation René Haijemaa,∗ , Jan van der Wala, b , Nico M. van Dijka a Department of Quantitative Economics, Faculty of Economics and Econometrics, University of Amsterdam, The Netherlands b Department of Mathematics and Computing Science, Eindhoven University of Technology, Eindhoven, The Netherlands
Available online 25 May 2005
Abstract Blood platelets are precious, as voluntarily supplied by donors, and highly perishable, with limited lifetimes of 5–7 days. Demand is highly variable and uncertain. A practical production and inventory rule is strived for that minimizes shortages and spill. The demand and production are periodic, as varying over the seven days of the week. Demand for ‘young’ platelets (oncology and hematology) and demand for platelets of ‘any’ age up to the maximal shelf life (traumatology and general surgery) are distinguished. A combined Markov dynamic programming (MDP) and simulation approach is presented and applied to a real life case of a Dutch blood bank. By down-sizing the dimension and applying this combined approach it is shown that order-up-to type replenishment rules that perform quite well can be found. Particularly, a double-level order-up-to rule, so-called 2D rule, is derived, with one level corresponding to ‘young’ platelets and one to the total inventory. This rule is easy to implement and is shown to be ‘nearly optimal’. This approach and the double order-up-to rule seem to be new. The results are most suitable for sensitivity analyses such as with respect to shortages and production costs for blood platelet inventory management. 䉷 2005 Elsevier Ltd. All rights reserved.
1. Introduction This paper will focus on the production and inventory management of platelets at a blood bank, responsible for the supply of platelets to a large number of hospitals. Platelets rapidly deteriorate in quality and can only be used up to an age of 5–7 days. Platelets should thus be regarded as highly ∗ Corresponding author.
E-mail addresses: [email protected] (R. Haijema), [email protected] (J. van der Wal), [email protected] (N.M. van Dijk). 0305-0548/$ - see front matter 䉷 2005 Elsevier Ltd. All rights reserved. doi:10.1016/j.cor.2005.03.023
R. Haijema et al. / Computers & Operations Research 34 (2007) 760 – 779
761
perishable. As platelets are also expensive (the blood bank calculates a selling price of more than 450 euro per patient per treatment), while shortages may involve high extra expenses, if not put lives at risk, production volumes have to be set carefully. A practical and (nearly) optimal trade-off is thus to be found between spill and shortages. Before stepping further into the details of this optimization, let us elaborate on the underlying aspects and our particular motivation. 1.1. Background and motivation The management of blood and blood products is a problem of general interest with a number of complicating aspects. Roughly a distinction can be made in supply, demand and costs. 1.1.1. Supply At the supply side, the necessity of natural human blood remains to be dominant, despite substantial developments for technologically produced substitutes of specific blood products. Therefore, one still relies mainly on the availability of volunteering donors. In most less developed countries (70% of the entire world population), the donors are the major concern, particularly in combination with the high risk of infections and contamination, such as HIV and hepatitis. In developed countries, in contrast, due to the efforts in ‘donor management’, the availability of donors is not a major problem. The supply also depends on aspects ranging from population densities, the infrastructure of transport, the set-up of regional blood banks, the number of hospitals, up to the capacities for processing blood at blood banks and for obtaining blood from donors with limited transportation time frames and capacity constraints (e.g. not at weekends). 1.1.2. Demand At the demand side, first of all, there is the uncertainty in the quantities required during any day, week or month. Even with all detailed information about patients requiring regular transfusions and planned surgeries, in the order of 50% of the demand during a week remains to be uncertain. Secondly, there is the complexity of the various blood types and percentages of exceptions that is involved in forecasting and determining the demand. Finally, restricted lifetimes for usability (perishability) for the various blood products are to be taken into account. Whereas red blood cells and plasma can be kept for months up to over a year, platelets have a restricted shelf life of at most one week. This paper therefore will merely focus on the production and inventory problem of platelets. Furthermore, for platelet demand a distinction can be made between two types of patients. For the first type of patients, mainly oncology and hematology, ‘young’ platelets are highly preferred. These patients are not able to produce enough platelets themselves. For the other type, mainly traumatology and general surgery, platelets of any age up to the maximal shelf life, can be used, since these patients quickly resume their own platelet production. The first type of demand is to a certain extent predictable. Also, these patients are treated on a regular basis, more often on Monday, Wednesday and Friday and seldom in the weekend. The second type of demand is quite unpredictable. 1.1.3. Costs The supply of blood involves a number of different and substantial costs. Among them are general costs for technical development (research) and instruments, for safety reasons and protection against contamination, for transport and infrastructure and for personnel capacities. These costs will not be
762
R. Haijema et al. / Computers & Operations Research 34 (2007) 760 – 779
considered. In this paper, we will only consider costs that are directly related to the production and inventory of platelets, per unit. We will distinguish inventory costs, costs for ‘outdating’ and costs for shortages. While the first component is rather concrete, the second and third are less so. The second also reflects the esthetic value that blood is precious and supplied at voluntary basis for humanity purposes. There should therefore be no ‘unnecessary’ spill. On the other hand, shortages will at best lead to substantial additional costs to obtain blood from another blood bank or by special actions. This trade-off between ‘spill’ and ‘shortages’, even though not in concrete quantifiable numbers, is in fact of crucial importance in the field of blood transfusion medicine. An optimal (or nearly optimal) and practical production policy for platelets is to be developed that takes these ‘costs’ into account.
1.2. Literature and objective The research on the inventory management of perishables in general and blood products in particular started in the 1960s with van Zyl [1] and boosted in the seventies and early eighties, see [2–5]. Nahmias’s review in 1982 [6] on perishable inventory contains nearly 100 references and pays considerable attention to blood bank applications. In 1984 Prastacos [7] gave an overview of the theory and practice of blood inventory management. Platelets have a lifetime of only a few days so that outdating is a real problem. Quite a few papers consider the platelet inventory problem, see e.g. [8–11]. Platelet studies usually deal with blood banks. An exception is Hesse et al. [12] who investigate keeping platelet inventories at hospitals in the Chicago area. Most studies report order-up-to policies evaluated by simulation or by a Markov chain under very restrictive assumptions, such as zero lead time, no periodicity, order-up-to policies only, etc. Some related work is Chazan and Gal [13] and Cohen [14] (considering the age distribution under a base-stock replenishment rule), Lian and Liu [15] (dealing with numerical results for the class of (s, S) production policies), Schmidt and Nahmias [16] and Perry and Posner [17] (who look at (S −1, S) policies with a non-zero leadtime) and Goyal and Giri [18] (describing trends in the modelling of deteriorating inventory). More than one type of demand and different hospital supply rules, as we consider, are rare in literature. Some results are reported by Pierskalla and Roach [19] and Goh et al. [20]. The structure of the problem suggests using a Markov Decision Process (MDP) approach, but the size of the problem makes this impossible for realistic problem instances. Although, with today’s increased computer speed the possibilities are, as we will see, far greater than in the seventies and eighties. Some early MDP formulations can be found in [1,21–24]. A far more recent study is Blake [25] on the platelet production in Nova Scotia, Canada. As an extension of the references mentioned, our platelet problem formulation includes the following four aspects: • the periodicity due to different demand and production characteristics for different days of the week, • two types of competing demands: demand for ‘young’ platelets and demand for platelets of ‘any’ age up to the maximal shelf life, • alternative supply rules for the two types of demand, • the production lead time of 1 day.
R. Haijema et al. / Computers & Operations Research 34 (2007) 760 – 779
763
More precisely, the objective is to extend the type of results from the literature in a number of ways: • to present a dynamic programming formulation for the blood platelet problem which takes these complicating factors of the two types of periodic demand, of alternative supply rules and of the production lead time into account, • to investigate the numerical computation and the structure of an optimal policy for down-sized, but still sufficiently realistic, situations, • to show the ‘near-optimality’ of single and double-level order-up-to policies, • to provide a simulation-based search algorithm to find these order-up-to policies, • to verify that ignoring the different blood types is justified. 1.3. Approach A combined MDP and simulation approach will therefore be followed. This approach consists of the following steps: 1. First, a MDP-formulation is provided for ‘the’ blood platelet problem. 2. Next, for realistic data the state space and the demands are down-sized such that the optimal value(s) can still be computed by standard successive approximation. 3. As a third step, for this down-sized situation it is shown that: (i) the optimal policy can be quite complicated, and (ii) a double-level replenishment rule is nearly optimal. 4. A heuristic simulation-based search procedure is then applied to find the best single (1D) and double (2D) level order-up-to policy. 5. Finally, the quality of these order-up-to rules is evaluated using the MDP. This approach seems to be new in the literature. 1.4. Results Executing these steps for realistic data of a Dutch blood bank led to a number of observations that seem to be of general interest: (1) (2) (3) (4)
The ‘optimal’ production rule for platelet production is complicated and not practical to implement. ‘Nearly optimal’ rules, however, can be found in the class of simple order-up-to rules. A simulation table can provide useful insights. Single level (1D) order-up-to rules may already perform quite well. Double level (2D) order-up-to rules provide a further improvement and can be proven to be ‘nearly optimal’ when distinguishing demand for ‘fresh’ and ‘any’ platelets. (5) The effect of expanding the shelf life from 4 to 6 days reduces costs substantially. (6) If less than 50% of all donor whole blood donations is needed for platelet production, the total production numbers may be determined without paying attention to the different blood types in the inventory. (7) Using sensible cost functions, the number of shortages will be below 1 promille and outdating, about 1%.
764
R. Haijema et al. / Computers & Operations Research 34 (2007) 760 – 779
1.5. Outline The outline of the paper is as follows: First, in Section 2.1, the detailed problem description is given along with a description and illustration of the supply rules. The dynamic programming formulation and computation are presented in Section 2.2, while a discussion on its complexity and its down-sizing can be found in Section 2.3. After a presentation of the realistic test problem in Section 3.1, steps 2–5 of Section 1.3 are executed in Sections 3.2–3.4. Most notably, a comparison of the optimal policy and the simple order-up-to rules is provided in Section 3.4. Finally, in Section 3.5, results for these simple rules are also provided for the original full-sized problem. Section 4 applies the approach and addresses a number of what–if questions of practical interest, concerning the extension of the shelf life time, the sensitivity to cost parameters and the distinction of blood types. An evaluation concludes the paper. 2. The MDP model The problem we are facing is the platelet production inventory problem at the blood bank, with the possibility of production during weekdays but not in weekends. The production becomes available after one day. Furthermore, there are two types of demand: one for fairly young platelets, the other for platelets of any age up to the maximal shelf life. In line with literature, we can formulate this problem, with the extensions of periodicity and two types of demand, quite well in terms of a finite state MDP. 2.1. MDP description and supply rules This MDP model is characterized by its decision epochs, decisions, states, transitions from one state to another and costs. 2.1.1. Decision epochs At daily basis decisions need to be taken on the production volume. During the day a number of things happen. First the inventory is inspected and a production decision is taken. Then demand occurs and is satisfied according to some fixed ‘supply’ rule, as will be specified below. The production of that day is available early in the morning of the next day. We will monitor the system 7 days a week. During the weekends there is no production. Note that as a result of this, the MDP will be periodic with period 7. 2.1.2. Decisions Two decision elements are to be distinguished: (1) a decision about the platelet production size for that day. (2) a decision that follows from a supply rule that prescribes which pools should be taken from stock in oder to fulfill the demand. The first decision element, of how much to produce during a day, concerns the actual decision that is going to be optimized depending on the ‘state’ of the MDP. The amount produced is expressed in ‘platelet pools’, or patient units. A platelet pool consists of the platelets from 5 whole blood donations, thus platelets from 5 different donors. The second decision element, the supply rules, will be considered as different scenarios and will thus not be part of the MDP itself. The supply rule will just appear as an index R (for Rule). This means that for
R. Haijema et al. / Computers & Operations Research 34 (2007) 760 – 779
765
each problem that we consider, we will formulate an a priori ‘supply’ rule for the two types of demand. (The supply rules will be discussed later.) In practice, there are no restrictions with respect to the production capacity nor the storage capacity, except that there is no production on Saturday and Sunday. 2.1.3. States A state description is required so that once the production decision has been taken, the state transitions and the costs during the next period are uniquely determined in stochastic sense. As production is possible only at weekdays and demand varies with the day of the week, the day of the week has to be included in the state description. Furthermore, as the platelets have a finite lifetime, one also has to take into account the residual lifetimes (or ages) of the platelet pools in inventory. A natural instant to describe the platelet inventory is the moment just after the production of the previous day has arrived and just before the production decision for the coming day is taken. With m the maximal residual shelf life of the platelets (between 4 and 6 in our case), the state of the system can thus be characterized by (d, x) = (d, x1 , x2 , . . . , xm ) with d:
the day of the week (d = 1, . . . , 7)
and x = (x1 , . . . , xm ) :
the inventory state, where xr is the number of pools with a residual shelf life of r days.
Note that, since there is no production in the weekend, some of these xr have to be 0 (there can be no platelets with an age that corresponds to production on Saturday or Sunday). Thus the sets of possible inventory states x for a specific day d, denoted by Xd differ. Note that there is no prior information about the demand in the near future other than statistical information in the form of the demand distributions. Hence there is no need to incorporate additional information into the state description. 2.1.4. Transitions The transitions are determined by three elements: the production decision, the demand from hospitals and the rule by which the platelets are supplied. We will separately discuss each of these elements and step by step illustrate the transition mechanism by numerical examples. Assume that we are in state (d, x) = (d, x1 , . . . , xm ) and that the production decision is k. When discussing the transition from one state to the next, we distinguish the following elements: • (No demand) First assume that there is no demand. As the residual shelf life of all inventory decreases by one day, with part of the inventory expiring, the transition from day d to day d + 1 then becomes (d, x1 , x2 , . . . , xm ) → (d + 1, x2 , . . . , xm , k). • (Demand) Demand may occur for two types of platelets: of type ‘any’ and of type ‘young’. Denote by i the demand for ‘any’ and by j the demand for ‘young’. The corresponding demand probabilities
766
R. Haijema et al. / Computers & Operations Research 34 (2007) 760 – 779
at day d are denoted by pd,a (i) : for demand of type any, pd,y (j ) : for demand of type young. • (Supply rules) As there are two types of interfering demands, a supply rule has to specify in which way these demands are satisfied from the available inventory levels. • (Rule for ‘any’) For ‘any’ demand we only consider the natural FIFO rule : oldest platelets first. For example, with m = 6 (the maximal shelf life counted from the moment the pools are added to stock), demand (i, j ) = (7, 0), for inventory state (2, 0, 3, 0, 0, 3) and production k = 2, first the x1 = 2 (oldest platelets) are used, next x3 = 3 and finally 2 are taken from x6 = 3. As 2 pools are produced and all remaining platelets age one day, the state changes from (d, 2, 0, 3, 0, 0, 3) → (d + 1, 0, 0, 0, 0, 1, 2). • (Rules for ‘young’ demand) For ‘young’ demand we consider a family of supply rules; parameterized by a value r and is denoted by FIFOR(r), with r : the minimal residual shelf life required. According to this rule demand for ‘young’ is satisfied by first taking pools with a residual shelf life r, then r + 1 etc., until all ‘young’ demand is satisfied. If the number of pools with a residual shelf life of at least r is not sufficient, we will use pools with residual shelf life r − 1, r − 2, etc. This will be called mismatch. If a demand cannot be satisfied at all, it is called a shortage. (Mismatches and shortages have to be taken into account as they lead to special costs.) In other words, a FIFOR(r) rule first works as a FIFO rule from level r and up, and next, in case the demand cannot be fully met in this manner, as a LIFO rule from level r − 1 downwards. As special cases, this family of FIFOR(r) rules includes the FIFO rule as well as the LIFO rule (youngest platelets first) by FIFOR(m) = LIFO, FIFOR(1) = FIFO. • (Rule for which first) Further, it is assumed that demand for young platelets has priority over ‘any’. (This might seem illogical given the emergency character of ‘any’, but demand for ‘young’ is usually known at the beginning of the day, whereas emergency demand arrives later.) As a consequence, the demand for young platelets is dealt with first. Next from the remaining inventories the demand for ‘any’ is satisfied. • (Supply rule) A supply rule for all demand can thus be specified by a combination of first a FIFOR(r) rule for ‘young’ demand, r = 1, . . . , m, and next a FIFO rule for ‘any’ demand. • (General notation) In general, for inventory state x, and given demand (i, j ), production decision k and supply rule R and at an arbitrary day d, denote by tR (x, k, i, j ): the new inventory state at day d + 1.
R. Haijema et al. / Computers & Operations Research 34 (2007) 760 – 779
767
Table 1 (d, 2, 0, 3, 0, 0, 3) → (d + 1, 0, 2, 0, 0, 0, 5)
(Example) As an example, for m = 6, r = 4, k = 5 and (i, j ) = (1, 4), we obtain the state transition in state (d, x) = (d, 2, 0, 3, 0, 0, 3) shown in Table 1 . 2.1.5. Costs We can distinguish five types of costs: production costs, outdating costs, inventory costs, shortage costs and mismatch costs. Production costs are mainly material and laboratory costs. Therefore the production costs are assumed to be linear in the production size. At present, the hospitals pay more than 450 euro per pool delivered at the hospital. The variable production costs are significantly less, say 150 euro. Outdating costs are the costs incurred when a pool has to be thrown away when the maximal shelf life has passed. If donors are scarce, these costs will be high. If there is ample supply of donors, as in the Netherlands, these costs will be merely the variable production costs. So we take the outdating costs equal to the production costs. In the sequel we will not consider the variable production costs, only the outdating costs, i.e. costs for overproduction. Inventory costs are the costs made to keep the pools under perfect conditions so that they can reach their maximal shelf life. We will ignore the fixed costs and only consider variable costs, linear in the total inventory. Although these costs are not quantified formally, it seems reasonable to estimate them as being between 0.005 and 0.01 times the production costs per unit per day. Shortage costs reflect the fact that when the ordinary supply fails precious time is lost and costly solutions have to be found. This could mean short-term production or getting the platelets from another blood bank. Shortage costs are taken to be linear and are high compared to the other costs. It seems reasonable to estimate them as being roughly 2 to 10 times higher than the ordinary production costs. Mismatch costs, finally, are incurred when there is demand for ‘young’ but there are not enough young platelet pools so that older pools have to be used. As mismatch is accepted by the hospitals the costs should not be estimated as being very high. On the other hand, if this happens frequently the patients will definitely suffer from it. We will quantify them as being somewhat higher than the outdating costs. (Notation) For any inventory state x, production level k and demand (i, j ), denote by cR (x, k, i, j ) := the costs during a day under supply rule R. The expected costs for day d are then obtained as pd,a (i)pd,y (j )cR (x, k, i, j ). cR (d, x, k) := i,j
768
R. Haijema et al. / Computers & Operations Research 34 (2007) 760 – 779
2.2. DP-computation Our focus is to minimize average costs, say, at daily, weekly or yearly level. To this end, and to use standard dynamic programming, define, under supply rule R VnR (d, x) := minimal expected cost over a period of n days starting in state (d, x). Then VnR (d, x) can be computed recursively, starting with V0R (·) = 0, from ⎧ ⎨
VnR (d, x) = min cR (d, x, k) + k ⎩
R pd,a (i)pd,y (j )Vn−1
i,j
⎫ ⎬ d + 1, tR (x, k, i, j ) . ⎭
(1)
The above relation deals with the increase in costs over one day. However, as the problem is periodic with a period of one week, it is more natural and convenient to consider the average costs on a weekly basis. Let us define C R∗ as the minimal average weekly costs under supply rule R:
R C R∗ := lim Vn+7 (d, x) − VnR (d, x) . n→∞
Note that this limit is independent of the initial state as the underlying MDP is ergodic. An approximation of the optimal value C R∗ and a corresponding (nearly) optimal stationary policy R can then be found by applying the so-called Odoni bounds (cf. [26]) and its periodic extension (see Su and Deininger [27]) by determining
R R L(n) := min Vn+7 (d, x) − VnR (d, x) C R∗ max Vn+7 (d, x) − VnR (d, x) =: U (n). (2) x∈Xd
x∈Xd
The lower and upper bound L(n) and U (n) of this inequality are known to converge (exponentially fast) to C R∗ for n → ∞. Thus C R∗ can be approximated arbitrarily closely for sufficiently large n. More precisely that is, up to an accuracy by continuing the successive computation (1) until U (n) − L(n) . Furthermore, an optimal or nearly optimal stationary policy R R R R = (1), (2), . . . , (7) n n n n can be obtained by optimizing the right-hand side of (1) for d = 1, 2, . . . , 7. That is, R n represents a decision rule that prescribes an optimizing decision k for the right-hand side of (1) for any state (d, x). Then, an -optimal stationary policy is guaranteed as for R C R n := the expected costs under the stationary policy n we have C R∗ C R n + when U (n) − L(n) .
(3)
R. Haijema et al. / Computers & Operations Research 34 (2007) 760 – 779
769
Problem: However, for practical purposes and implementation the DP-procedure sketched above has two major complications: (a) the computational complexity and limitations involved for realistic data sizes, mainly due to the number of states, (b) the structure of the (nearly) optimal policy which may not be suitable for practical implementation. In what follows these complications will be dealt with by (a) a ‘down-sizing’ of the size of the problem, (b) a search for ‘simple’ nearly optimal policies. 2.3. Computational complexity and down-sizing For each dynamic programming step (1) one has to consider each day of the week d, each possible inventory state x and all possible production levels k. For each combination (d, x, k) we thus have to compute R cR (d, x, k) and d + 1, t R (x, k, i, j ) . pd,a (i)pd,y (j )Vn−1 i,j
The main complexity problem concerns the inventory vector x. When m = 6, the vector x may have 4 or 5 components that can be nonzero, depending on the day of the week. If each of these components has values between 0 and the number of production levels Mp , then, including the day of the week, the number of states becomes 2Mp5 + 5Mp4 . For a realistic situation with Mp = 50 the number of states is already more than 6 × 108 . Even if we take into account that the total inventory is limited, to 150 say, the number of states is tremendous. Per iteration step one also has to consider all relevant production levels Mp . Furthermore, for each combination of the state and decision, the number of possible transitions is determined by the possible number of combinations of demands (i, j ), Ma My , with Ma the number of values of ‘any’ demand and My the number of values for ‘young’ demand. In our case Ma and My are smaller than Mp , say Ma My = 1000. The overall complexity for a one week iteration is thus of order O (2Mp6 + 5Mp5 )Ma My , in our case
already 3 × 1013 . For cases of a realistic size, we thus have to find other ways than the standard dynamic programming approach above to obtain good policies. The dynamic programming approach is thus limited due to its computational complexity. As a consequence, we can only use the DP-approach for down-sized versions of the original problem. Nevertheless, the down-sized problem will give insight in • the complexity of the optimal policy, • and the near-optimality of simple order-up-to policies. In order to preserve the characteristics of the original problem, we scale all cost factors and we transform the demand figures to distributions which have the down-sized mean and the original coefficient of variation. To this end, we will use a fitting procedure suggested by Adan et al. [28].
770
R. Haijema et al. / Computers & Operations Research 34 (2007) 760 – 779
3. Numerical results, MDP We first consider the down-sized problem, for which the MDP approach is still computationally feasible. From the results, we will learn what the structure of the optimal policy is and whether simple order-upto rules are indeed nearly optimal. We conclude this section with the results for the original full-sized problem. First we describe our base test case. 3.1. Base test case The Netherlands, with 16 million inhabitants, has 4 regional blood banks. Our data are obtained from one of them (Sanquin blood bank, region North-East). The average weekly demand is about 180 platelet pools. Roughly 70% of the demand is for ‘young’ platelets, and 30% is for ‘any’. Demand for ‘any’ is 7 days a week and more or less evenly spread over the week. For the blood bank North-East the mean demand per day for ‘any’ is 8 units. Demand for ‘young’ platelets occurs mainly from Monday till Friday, with means (24, 16, 32, 16, 24, 0, 8). Hence, there are peaks on Monday, Wednesday and Friday. (To simplify the presentation and the down-sizing, the real data, as obtained from Hinloopen [29] are rounded to multiples of 4.) For ‘any’, the assumption of purely random, hence √ Poisson demand, is realistic, which gives a coefficient of variation (cv) for the daily demand of 1/ 8√= 0.35. For ‘young’ the demand is more regular than Poisson. For Poisson the cv would be around 1/ 24 = 0.20. As this demand is less variable than Poisson, we take the cv for demand ‘young’ equal to 0.15, irrespective of the day of the week. From these real data, we derived small test case as shown in Table 2 by down-sizing the instance, as if platelets are issued by units of 4 pools. Then we get an average demand for ‘any’ of 2 per day (i.e. 8 pools) and for ‘young’ platelets (6, 4, 8, 4, 6, 0, 2) for Monday to Sunday. As stated before, we use the fitting procedure suggested by Adan et al. [28] to transform these figures into discrete demand distributions. 3.2. The optimal policy When executing the DP approach for the base test case, we obtain an optimal (may be nearly optimal) policy with a fairly complicated structure. As an illustration, consider the optimal decisions in a number of states at Tuesday morning, all having a total inventory of 12. The optimal policy prescribes to produce • 8 units in the states (0,0,4,0,0,8) and (0,0,5,0,0,7), • 9 units in (0,0,6,0,0,6) and (0,1,5,0,0,6), and • 10 units in (0,0,7,0,0,5), (0,1,6,0,0,5), (0,2,5,0,0,5), (0,3,4,0,0,5) and (0,4,3,0,0,5). Apparently, the optimal decisions are based not only on the total inventory, but also on the age distribution of the inventory. This illustrates the well-known fact that the optimal policy will in general not be a simple order-up-to rule. To investigate this a little further, let us group the states according to their total inventory. 3.2.1. Total inventory procedure Each inventory state is aggregated into an expected total end-of-the-day inventory level by first subtracting the two rounded mean demands and then removing the inventory that has become outdated.
R. Haijema et al. / Computers & Operations Research 34 (2007) 760 – 779
771
Table 2 Base test case Attribute
Reality
Base test case
Average ‘any’ cv(‘any’) Distribution ‘any’ Average ‘young’ cv(‘young’) Distribution ‘young’ Maximum shelf life Policy ‘any’/‘young’ Holding costs Outdating cost Shortage costs Mismatch costs
8 per day 0.35 Poisson (24, 16, 32, 16, 24, 0, 8) 0.15 Fit first 2 moments 6 days FIFO-FIFOR(4) 1 per pool per day 150 per pool 750 per pool 200 per pool
2 per day 0.35 Fit first 2 moments (6, 4, 8, 4, 6, 0, 1) 0.15 Fit first 2 moments 6 days FIFO-FIFOR(4) 4 per unit per day 600 per unit 3000 per unit 800 per unit
Table 3 Order-up-to frequencies for the base case Predicted Up-to 21 20 19 18 17 16 15 14 13 : 0 Total
2
3
4
5
6
7
8
9
10
11
12 1
1 1 12 71 267 3473 5926 90249 0 : 0
1
100000
1 12
13101
26854
30896
3952 15140
2119 1462 3820
13101
26854
30896
19092
7401
1221 449 438
2108
261 125 63
449
71 6 8
85
12
1
Total
For example, consider the case with m = 6 and supply rule FIFO-FIFOR(4). Then, at Tuesday, for inventory state x=(0, 1, 5, 0, 0, 6) mean demands of 2 for ‘any’ and 4 for ‘young’ result in (0, 0, 4, 0, 0, 2) at the end of the day. As there is no outdated inventory, the ‘predicted’ inventory state at Wednesday morning before the replenishment (from Tuesday) arrives becomes (0, 4, 0, 0, 2, 0) with a total inventory of 6. (In a somewhat different setting Pinson et al. [2] investigated the use of weight factors when aggregating the stock, but concluded that this gave no significant improvement.) To visualize the optimal policy Table 3 is created by simulating the optimal policy for the base test case, as obtained from the MDP, for 100000 weeks. It represents all predicted ‘total inventory’ and replenishment level combinations for Tuesday. Each inventory state that occurs during the simulation is mapped into a total end-of-the-day inventory. Also it is registered how many is produced in that state. As a result, we get frequencies for combinations of the two: inventory and production. For example,
772
R. Haijema et al. / Computers & Operations Research 34 (2007) 760 – 779
in Table 3, the number 3820 in row ‘up to 14’ and column ‘6’ means that during the simulation 3820 times a state was visited that corresponds to a predicted end-of-the-day inventory of 6 in which the optimal production volume was 14 − 6 = 8. To be precise, 3820 is the sum of 1962 visits to state (0,4,0,0,0,8) and 1858 visits to (0,5,0,0,0,7). So, in this case, 14 is the target inventory level that will be reached at Wednesday morning if the demands are equal to the expected demand (there will be no outdating). The sum of the Tuesday morning total inventory and the replenishment order, however, is 22! The table shows that the replenishment order-up-to level of 14 for Tuesday fits in 90.2% of the states. But the table also shows some higher order-up-to levels for higher inventories. As inventories that are higher than usual must be the result of lower than average demand, the inventory will also be a little older than average. This indicates that there is some additional need for ‘young’. 3.2.2. Conclusion The optimal policy resembles an order-up-to policy, but it is likely that a more sophisticated order-up-to policy that looks at both the total inventory and the ‘young’ inventory will be even better. This means that hereafter we will be looking for, what we will call, double order-up-to policies with one-level aiming at total inventory and one looking at ‘young’ inventory only. 3.3. 1D and 2D order-up-to policies There are several ways in which we can obtain order-up-to policies. First, we describe how we derive them from the simulation tables directly, but clearly this is possible only for ‘small’ problems for which the dynamic programming approach is still feasible. Then we will introduce the search algorithm that will also be applied to the full-sized problem. We will consider single order-up-to policies called 1D policies and double order-up-to policies denoted as 2D. A 1D policy specifies for each of the 5 production days the order-up-to level and is described in the form of a 5-dimensional row vector 1D policy: S (T) := S1(T) , S2(T) , . . . , S5(T) . The 2D policies are characterized by two 5-dimensional vectors, one for the total inventory level and one for ‘young’ only. (T) (T) (T) (Y) (Y) (Y) 2D policy: S (T) , S (Y) := S1 , S2 , . . . , S5 , S1 , S2 , . . . , S5 . From the simulation tables (cf. Table 3), we see which 1D level fits best, in the sense of being the most frequent. Another way to obtain a 1D level is a quadratic fit procedure that minimizes the quadratic difference between the decision prescribed in the MDP and the proposed level. Similarly, one may use a quadratic fit procedure to obtain a 2D policy. Per day one looks for the combination of levels for ‘total’ and ‘young’ for which the mean quadratic difference between the decision obtained from the two levels and the decision prescribed by the MDP analysis is minimized. (Deriving the most-frequent 2D rule is less trivial.) If we once again look at the inventory state x = (0, 1, 5, 0, 0, 6) at Tuesday morning, then we predict a total inventory x (T) = 12 − 6 = 6 and a ‘young’ inventory x (Y) = 6 − 4 = 2 for Wednesday morning. Then the 1D rule S (T) = (10, 14, 10, 12, 17) in Table 4 suggests to produce
(T) max 0, S2 − x (T) = max{0, 14 − 6} = 8.
R. Haijema et al. / Computers & Operations Research 34 (2007) 760 – 779
773
Table 4 Results base test case Rule
Total
Young
Cost
Difference
MDP Optimum 1D Most frequent 1D Quadratic fit 1D Local search 2D Quadratic fit 2D Local search
— (10, 14, 10, 12, 17) (10, 14, 10, 12, 17) (11, 15, 10, 12, 17) (10, 14, 10, 12, 17) (10, 14, 10, 12, 17)
— — — — (5, 11, 7, 12, 17) (5, 11, 7, 12, 17)
514 529 529 523 515 515
— 2.98 2.98 1.76 0.22 0.22
The 2D quadratic fit rule in Table 4 prescribes to produce
(T) (Y) max 0, S2 − x (T) , S2 − x (Y) = max{0, 14 − 6, 11 − 2} = 9, which, as we have seen, is just the optimal decision in (0, 1, 5, 0, 0, 6). Apart from these approaches, we also propose a local search procedure based on simulation. Local search will also be used here for the smaller problem that is analyzed by dynamic programming, but is especially meant for the larger, real-size problems for which the dynamic programming approach fails. There is a huge literature on highly sophisticated combinations of simulation and search, see e.g., [30,31]. Here the local search method can be quite simple. The structure of the problem is such that typically, decisions are taken day by day, even though today’s decision directly affects tomorrow’s decision. In the early stages of the search, the duration of the simulation runs for 25 years. For the final selection, runs of 1600 years are used. Also note that the final solution is a ‘local optimum’, according to the simulation. (We think that the structure of the problem is such that there are no ‘real’ sub-optimal local optima.) Further, (i) the quality of the data used is never perfect, and (ii) around the optimum small changes of the order-up-to levels will result in very small changes in expected costs. (Notation) Let ed denote the 5-dimensional row vector with all components equal to 0 except for component d which is 1, e.g., e2 = (0, 1, 0, 0, 0). 3.3.1. 1D local search • Initialize. Define S (T) = (0, 0, 0, 0, 0). • Step 1. Replace S (T) by the best improvement S (T) + ed until no further improvement. • Step 2. Replace S (T) by the best improvement S (T) − ed until no further improvement. 3.3.2. 2D local search • Initialize. First execute the 1D local search procedure. Let S ∗ be the best policy found. Define S (T) := S ∗ and S (Y) := S ∗ . • Step 1. This overestimates the levels for ‘young’, thus replace S (Y) by the best improvement S (Y) −ed until no further improvement. • Step 2. Replace the pair S (T) by the best S (T) − ed for d = 1, . . . , 5 until no further improvement. • Step 3. Replace the pair S (Y) by the best S (Y) + ed until no further improvement.
774
R. Haijema et al. / Computers & Operations Research 34 (2007) 760 – 779
3.3.3. Computation times1 Solving the MDP takes about 4 min for the down-sized test case. The 2D local search takes about 1 min. For the original full-sized problem, the MDP approach is not feasible, whereas the 2D local search takes about 5 min, where most of the time is spent in the final selection runs. 3.4. Results The results of these approaches are compared to the results for the optimal policy as displayed in Table 4. For the down-sized base case (at grid 4) the table presents the minimal weekly cost as computed for the MDP via dynamic programming, together with the results for the most frequent 1D rule, the 1D quadratic fit, the 2D quadratic fit and the 1D and 2D rules obtained by local search. (The latter costs are computed by evaluating the corresponding Markov chain.) 3.4.1. Conclusion What appears from the results is that the 1D policies perform quite well but that the 2D policy is even better and very close to the optimum. That, together with the fact that the structure of the 2D policy is on the one hand still quite simple and on the other hand pays attention to the age distribution as well, makes the 2D policy very attractive for practical implementation. The fact that the local search policies do so well is also promising for the large problem instances to be considered next. 3.5. The full-sized problem For the original problem we can no longer compare the results with the exact results from the MDP approach. We can only compare the results obtained via the 1D and 2D local search approaches. Table 5 presents the average weekly costs and the 1D and 2D policies for shelf lives 4 and 6 and supply rules FIFOR and LIFO for ‘young’. We see that the difference in performance between the 1D and 2D local search optima is limited. Looking at the costs per week, we see that the difference between shelf life 4 and 6 is about 40 000 euro annually. Also note that on Thursday and Friday for the best 2D policy in the FIFO-FIFOR situation the orderup-to levels for ‘total’ and ‘young’ are (almost) identical. This is the result of the fact that on these days there is hardly any ‘old’ inventory left, because there has been no production in the previous weekend. Apparently the optimal policy aims at having only ‘young’ at the end of the week! Remark. Note that the straightforward translation of the 1D and 2D policies for the down-sized base case to the original problem by just blowing up the order-up-to levels gets already fairly close to the nearly optimal policy. (Recall that the demand distributions in the down-sized problem and the original problem have the same coefficient of variation.) For instance, blowing up the replenishment levels of the 1D rule for the base case would result in replenishment levels (44, 60, 40, 48, 68) compared to the (45, 62, 45, 49, 70) rule obtained via local search. 1 Processor: CPU Intel Pentium (R) 4 AT/AT Compatible, 261 Mb RAM.
R. Haijema et al. / Computers & Operations Research 34 (2007) 760 – 779
775
Table 5 Results local-search for full-sized problem Shelf life
R rule
4 days
FIFO-FIFOR(2)
FIFO-LIFO
6 days
FIFO-FIFOR(4)
FIFO-LIFO
1D rule
2D rule
% Difference (2D vs 1D)
Cost ‘total’ ‘young’ Cost ‘total’ ‘young’
1288 (41, 60, 44, 49, 65) — 1374 (39, 55, 41, 45, 66) —
1288 (41, 60, 44, 49, 65) (21, 59, 44, 49, 65) 1363 (37, 55, 41, 45, 65) (29, 55, 41, 28, 57)
0.02
Cost ‘total’ ‘young’ Cost ‘total’ ‘young’
560 (45, 62, 45, 49, 70) — 563 (44, 61, 44, 49, 70) —
556 (41, 61, 44, 49, 70) (22, 45, 27, 49, 69) 558 (41, 60, 44, 49, 70) (22, 46, 26, 33, 43)
0.68
0.85
0.89
4. Sensitivity analyses In this section, for the down-sized problem, the sensitivity of the best 1D and 2D policies, and thus the local search methods, is investigated by varying the shelf lives, the supply rules and the costs. Further it is verified that ignoring the 8 blood types is justified. 4.1. Shelf lives and supply rules An interesting discussion in practice is about the importance of extending the maximal shelf life of platelets and about the supply rule for ‘young’. Table 6 presents the results for the shelf lives varying from 4 to 6 days in combination with two supply rules: FIFO-FIFOR(r) with r such that the maximal age that can be used for young (without mismatch costs) is the same in all cases considered and FIFO-LIFO, where the 1D and 2D rules are obtained via local search (LS). We see that the difference between FIFO-FIFOR(r) and FIFO-LIFO is small and decreases with the maximal shelf life. From a practical point of view, the difference is already negligible for a maximal shelf life of 5 days. Again, we see that the 2D-policies as obtained via local search perform excellently. We also conclude that the relative difference between the best 1D and 2D rule increases if the shelf life is increased. Also the LIFO-LIFO combination has been investigated. It appeared that irrespective of the maximal shelf life, this rule leads to an enormous increase in outdating. The FIFO rule for ‘any’ is needed to reduce the outdating! Another interesting aspect is, how are the costs split over the four categories ‘inventory’, ‘outdating’, ‘shortage’ and ‘mismatch’. Table 7 shows these results. Not surprisingly, the split is nearly identical for the MDP and 2D LS policies. The 1D LS policy shows, as expected, higher mismatch and inventory costs that are partially compensated for by the outdating and shortage costs.
776
R. Haijema et al. / Computers & Operations Research 34 (2007) 760 – 779
Table 6 Varying shelf lives and different supply rules for ‘young’; MDP versus Local Search Shelf life
R rule
MDP
1D LS
2D LS
4 days
FIFO-FIFOR(2) FIFO-LIFO FIFO-FIFOR(3) FIFO-LIFO FIFO-FIFOR(4) FIFO-LIFO
1181 1222 719 724 514 515
0.23 2.04 1.08 1.39 1.76 2.14
0.19 1.15 0.13 0.50 0.22 0.27
5 days 6 days
Table 7 Cost split
Average weekly cost Inventory cost Outdating cost Shortage cost Mismatch cost
MDP
1D LS
2D LS
514
523
515
69.28% 12.20% 18.48% 0.04%
69.63% 11.20% 17.79% 1.39%
69.34% 12.22% 18.44% 0.00%
4 days
5 days
6 days
1181
719
514
28.95% 40.55% 30.50% 0.00%
49.47% 37.33% 13.14% 0.06%
69.28% 12.20% 18.48% 0.04%
Table 8 Shelf life and cost split
Average weekly cost Inventory cost Outdating cost Shortage cost Mismatch cost
The cost distribution does however strongly depend on the shelf life as is illustrated in Table 8. The shorter the period platelets used, the higher the costs for outdating. 4.2. Costs We are also interested in the sensitivity of the optimal solution with respect to changes in the cost parameters. For the base test case, with a weekly demand of 44 units of 4 pools, 30 for ‘young’ and 14 for ‘any’, Table 9 reports the average numbers of outdates, shortages and mismatches per week (all numbers are in units of 4 pools). First of all, note that the number of shortages is extremely small, about 0.03 units of 4 pools per week, so about 0.13 (4 × 0.033) pools per week. That is, about 6 times per year. Increasing the shortage cost to
R. Haijema et al. / Computers & Operations Research 34 (2007) 760 – 779
777
Table 9 Cost parameter sensitivity; exact results from MDP with mean demand of 44 # Outdates
# Shortages
# Mismatches
Base case
0.1039
0.0322
0.0003
Holding cost 2 instead of 4 Holding cost 8 instead of 4 Mismatch cost 400 instead of 800 Mismatch cost 1200 instead of 800 Shortage cost 5000 instead of 3000
0.1041 0.1035 0.0914 0.1041 0.2363
0.0304 0.0335 0.0319 0.0322 0.0063
0.0000 0.0008 0.0153 0.0000 0.0008
5000 reduces the number of shortages to once a year (at the cost of additional inventory and outdating). Note that all fractions for shortages are clearly below 1 pro mille (0.044). The table also shows that halving or doubling the holding cost has hardly any effect. An increase of the mismatch cost also has practically no effect. A decrease of the mismatch cost so that it gets below the outdating cost does have an effect. The optimal production volume is in some states relatively low, so mismatches occur in order to get rid of the inventory before it gets outdated. 4.3. The different blood types So far the fact that there is not just one type of platelet pool has been ignored. In reality, the blood types O, A, B and AB, and the Rhesus D factor are also important. Fortunately, rarely an exact donor–patient matching is required. Compatibility is the standard, which means that O can serve all four blood types, A can serve A and AB, etc. Similarly, Rhesus− can help both Rhesus− and Rhesus+. The main group of Rhesus− patients that should not receive Rhesus+ are women in the fertile age, between 15 and 45. So, only about 25% of all Rhesus− patients really need Rhesus−. Using detailed simulation we have verified that in our context the restriction to one blood type is allowed, see Haijema et al. [32]. In this study, the donors are assumed to show up according to 8 different Poisson processes with given means. If more donors show up than needed for production, then the blood types to be used are selected in the order O−, O+, A−, etc. Further, it is assumed that demand is according to 16 Poisson processes (distinguishing ‘any’, ‘young’ and blood type). Using compatibility, it could be concluded that in case only 50% of all whole blood donations are needed for the production of platelet pools, the increase in costs is less than 1%.
5. Evaluation In this paper the periodic production and inventory problem of a blood bank is studied, with two types of demand, one of which requires fairly ‘young’ platelets. The MDP approach is applied to a down-sized, but still quite realistic version of the problem. The results show that the optimal policy is quite complicated, and also that simple 1D and 2D order-up-to policies are nearly optimal. A fairly straightforward local search method is developed to find these order-up-to policies. Particularly, the best 2D policy is within 1% of the optimum. The local search method also works quite well for the original full-sized problem.
778
R. Haijema et al. / Computers & Operations Research 34 (2007) 760 – 779
As an important application it is shown that an increase of the shelf life of platelets from 4 to 6 days substantially reduces outdating and shortages. The results have been obtained without distinction of blood types. It has been shown however, that this is fully justified in the Dutch situation where less than 50% of all whole blood donations is needed for the production of platelets. If, however, the limited availability of whole blood donations leads to restrictions on the production capacity for platelets, this (random) availability has to be (and can easily be) incorporated in the model. Also in that situation there will exist a nearly optimal 2D policy. At present at the blood bank outdating is 20%. Other studies have shown that outdating need not be more than 3%. For the situation in the Netherlands, outdating can be about 1%. A project concerning the implementation of these results at the blood bank will start soon. Acknowledgements The authors are most grateful to professor Dr. Cees Th. Smit Sibinga, Director of Sanquin Consulting Services and advisor of the WHO Collaborating Center for Blood Transfusion and the WFH International Hemophilia Training Centre, and to Ing. Boudewijn Hinloopen (Sanquin blood bank, region North-East) for the fruitful and pleasant discussions and for providing the necessary data. The third author is particularly indebted to Dr. Martin Beun (†) for initially addressing the platelet problem in 1996, as a practicing physician and as a friend. His scientific curiosity stimulated the first steps of this research. He will be kept in memory with deep respect. He should not have died in the spring of 2004. References [1] van Zyl GJJ. Inventory control for perishable commodities. Unpublished PhD, dissertation, University of North Carolina, 1964. [2] Pinson SD, Pierskalla WP, Schaefer B. A computer simulation analysis of blood bank inventory policies. Technical Report, Department of Industrial Engineering and Management Sciences, Northwestern University, Evanston, Illinois, 1972. [3] Jennings JB. Blood bank inventory control. Management Science 1973;19:637–45. [4] Cohen MA, Pierskalla WP. Perishable inventory theory and its application to blood bank management. 1974. Working Paper, Department of Industrial Engineering and Management Sciences, Northwestern University, Evanston, Illinois. [5] Cohen MA, Pierskalla WP. Management policies for a regional blood bank. Transfusion 1975;15:58–69. [6] Nahmias S. Perishable inventory theory: a review. Operations Research 1982;30:680–708. [7] Prastacos GP. Blood inventory management: an overview of theory and practice. Management Science 1984;30:777–800. [8] McCullough J. Platelet production and inventory management. Platelet physiology and transfusion, technical workshop. New Orleans, Louisiana: American Association of Blood Banks; 1978. p. 17–38. [9] Katz AJ, Carter CW, Saxton P, Blutt J, Kakaya RM. Simulation analysis of platelets production and inventory management. Vox Sang 1983;44:31–6. [10] Ledman RE, Groh N. Platelet production planning to ensure availability while minimizing outdating. Transfusion 1984;24:532–3. [11] Sirelson V, Brodheim E. A computer planning model for blood platelet production and distribution. Computer Methods and Programs in Biomedicine 1991;35:279–91. [12] Hesse SM, Coullard C, Daskin MS, Hurter AP. A case study in platelet inventory management. Proceedings of the sixth annual industrial engineering research conference. Miami Beach, Florida: 1997. [13] Chazan D, Gal S. A Markovian model for a perishable product inventory. Management Science 1977;23:512–21.
R. Haijema et al. / Computers & Operations Research 34 (2007) 760 – 779
779
[14] Cohen MA. Analysis of single critical number ordering policies for perishable inventories. Operations Research 1976;24:726–41. [15] Lian Z, Liu L. A discrete-time model for perishable inventory systems. Annual of Operations Research 1999;87:103–16. [16] Schmidt CP, Nahmias S. (S-1, S) policies for perishable inventory. Management Science 1985;31:719–28. [17] Perry D, Posner MJM. An (S-1, S) inventory system with fixed shelf life and constant leadtimes. Operations Research 1998;46:565–71. [18] Goyal SK, Giri BC. Recent trends in modeling of deteriorating inventory. European Journal of Operational Research 2001;134:1–16. [19] Pierskalla WP, Roach CD. Optimal issuing policies for perishable inventory. Management Science 1972;18:603–14. [20] Goh CH, Greenberg BS, Matsuo H. Two-stage perishable inventory models. Management Science 1993;39:633–49. [21] Nahmias S, Pierskalla WP. Optimal ordering policies for product that perishes in two periods subject to stochastic demand. Naval Research Logistics Quarterly 1973;20:207–29. [22] Nahmias S. Optimal ordering policies for perishable inventory—II. Operations Research 1975;23:735–49. [23] Nahmias S. Higher order approximations for the perishable inventory problem. Operations Research 1977;25:630–40. [24] Fries B. Optimal ordering policy for a perishable commodity with fixed lifetime. Operations Research 1975;23:46–61. [25] Blake JT. Using dynamic programming to optimize the platelet supply chain in Nova Scotia. In: Proceedings of the 29th meeting of the European working group on operational research applied to health services. Dlouhý M, editor. Prague, Czech Republic: ORAHS; 2003, to appear. [26] Odoni A. On finding the maximal gain for Markov decision processes. Operations Research 1969;17:857–60. [27] Su S, Deininger R. Generalization of White’s method of successive approximations to periodic Markovian decision processes. Operations Research 1972;20:318–26. [28] Adan IJBF, van Eenige MJA, Resing JAC. Fitting discrete distributions on the first two moments. Probability in the Engineering and Informational Sciences 1995;9:623–32. [29] Hinloopen B. Personal communication. [30] Bertsekas DP, Tsisiklis J. Neuro dynamic programming. Athena Scientific; 1996. [31] Swisher JR, Heyden PD, Jacobson SH, Schruben LW. A survey of simulation optimization techniques and procedures. In: Joines JA, Barton RR, Kang K, Fishwick PA, editors. Proceedings of the 2000 Winter Simulation Conference. 2000. [32] Haijema R, van der Wal J, van Dijk NM. Blood platelet production: a multi-type perishable inventory problem. Research Memorandum, FEE, University of Amsterdam, 2004.