Valuing data in aircraft maintenance through big data analytics: A probabilistic approach for capacity planning using Bayesian networks

Accepted Manuscript Valuing Data in Aircraft Maintenance through Big Data Analytics: A Probabilistic Approach for Capacity Planning using Bayesian Networks Duarte Dinis, Ana Barbosa-Póvoa, Ângelo Palos Teixeira PII: DOI: Reference:

S0360-8352(18)30485-6 https://doi.org/10.1016/j.cie.2018.10.015 CAIE 5454

To appear in:

Computers & Industrial Engineering

Please cite this article as: Dinis, D., Barbosa-Póvoa, A., Palos Teixeira, A., Valuing Data in Aircraft Maintenance through Big Data Analytics: A Probabilistic Approach for Capacity Planning using Bayesian Networks, Computers & Industrial Engineering (2018), doi: https://doi.org/10.1016/j.cie.2018.10.015

This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.

Valuing Data in Aircraft Maintenance through Big Data Analytics: A Probabilistic Approach for Capacity Planning using Bayesian Networks Duarte Dinisa,*,[email protected], Ana Barbosa-Póvoaa, Ângelo Palos Teixeirab aCenter

for Management Studies, Instituto Superior Técnico, Universidade de Lisboa, 1049-001 Lisboa, Portugal bCenter

for Marine Technology and Ocean Engineering, Instituto Superior Técnico, Universidade de Lisboa, 1049-001 Lisboa, Portugal

*Corresponding

author.

Highlights The capacity planning problem faced by aircraft MRO companies is described. Bayesian networks to address the capacity planning problem are developed. A validation process for Bayesian networks is proposed. Examples of the applicability of the developed Bayesian networks are presented.

Abstract Capacity planning is an important problem faced by aircraft Maintenance, Repair and Overhaul (MRO) organizations given the uncertainty of maintenance workloads. Despite the considerable amount of data generated and stored during the planning process, these have yet to provide a decisive competitive advantage to aircraft MROs. This paper addresses this problem by exploring Bayesian networks (BNs) as a big data and predictive analytics (BDPA) tool to cope with the uncertainty on both scheduled and unscheduled maintenance workloads and to improve the MROs capacity planning decision-making process based on incomplete information. The BNs were developed 1

from a real industrial dataset referring to 372 aircraft maintenance projects of a Portuguese MRO and comprise information variables representing typical information collected during the planning process and hypothesis variables representing the workloads required to be estimated. The benefits of applying BNs as a BDPA tool in aircraft maintenance are demonstrated through examples referring to capacity planning, but also sales planning, using real maintenance data. The BDPA tool based on BNs is generic and can be applied to the maintenance capacity planning process of any MRO, allowing accurate estimations and more informed decisions to be made when compared to current practices, which are based on descriptive statistics of past maintenance workloads. Keywords Maintenance; capacity planning; Bayesian networks; big data analytics; decision support systems.

1.

Introduction

For aircraft Maintenance, Repair and Overhaul (MRO) organizations, maintenance can be regarded as a production process that needs to be planned (Budai et al. 2008). The purpose of maintenance capacity planning is to establish the required resources to face expected maintenance workloads and its critical aspect is to determine the exact number of technicians and their skills given the uncertainty of the maintenance work (Haroun and Duffuaa 2009). In other words, maintenance capacity planning is concerned with the balance between available capacity, i.e. manpower, and required capacity, i.e. workload (Al-Fares and Duffuaa 2009). On the one hand, if the available capacity is higher than the required capacity, underutilization of resources and financial 2

inefficiency occur. On the other hand, if the available capacity is lower than the required capacity, delays will surely happen with potential financial penalties and damages to the reputation of the maintenance organization. Aircraft maintenance comprises scheduled maintenance, which ‘consists of all the individual maintenance tasks performed according to the maintenance time limitations’ (FAA 2012), and unscheduled maintenance, which results from discrepancies identified during scheduled maintenance tasks, other unscheduled maintenance tasks, pilot reports, unforeseen events occurred during normal operation, or data analysis (ATA 2007; FAA 2012). In terms of workload, scheduled maintenance is essentially deterministic since it consists in prespecified inspection tasks carried out at predetermined intervals, while unscheduled maintenance is inherently stochastic since it depends on the probabilistic nature of failures1 (Al-Fares and Duffuaa 2009). The uncertainty of the capacity planning process stems, therefore, primarily from the unscheduled maintenance. This problem has been identified in the literature as a problem that affects several activities in aircraft maintenance such as capacity planning, budgeting, materials and spare parts management, and tasks scheduling and resources allocation (Samaranayake and Kiridena 2012). Despite the considerable amount of data generated and stored during the aircraft maintenance planning process, these have yet to be effectively used to improve decision-making in capacity planning and to provide a decisive competitive advantage to aircraft MROs. To the best of the authors knowledge, capacity planning approaches currently used in aircraft maintenance do not explore the value of stored data in its full potential, being based on average workloads and costs calculated from past maintenance

1

According to the MSG-3 document (ATA 2007), failure is ‘the inability of an item to perform within previously specified limits’. 3

events. MRO organizations rely fundamentally on the decision-maker experience in estimating ‘safety buffers’ for manpower and budgeting, typically overestimating workloads and costs. This practice is known as overplanning (Murthy and Ma 1991; Mula et al. 2006). By doing so, MROs disregard, or at least poorly address, the uncertainty of the decision-making process and assume a considerable risk regarding the ability to meet the actual workload and to perform according to the contracted service price. The problem has been hardly studied in the literature and few works address the uncertainty of the MRO capacity planning decision-making process. The works of Eickemeyer et al. (2013, 2014) are exceptions to this. The authors state that in cases of ‘imprecise’ or ‘fuzzy’ information, such as the case of aircraft MRO capacity planning, Bayesian networks (BNs) should be used to estimate maintenance workload. Eickemeyer et al. (2013) identify BNs as the most advantageous artificial intelligence technique to address the uncertainty of maintenance capacity planning. BNs are considered to be adaptable to new problem conditions, to produce high quality workload forecasts with scarce input data, to require only a fair amount of work to be elaborated, to be understandable in industrial practice, and to be compatible with legacy data (Eickemeyer et al. 2013). To the best of the authors knowledge, the works of Eickemeyer et al. (2013, 2014) are the only applying BNs to maintenance capacity planning. A related work is that of de Melo and Sanchez (2008), in which the authors use a BN to predict software maintenance project delays based on maintenance risk, service complexity, and service criticality. BNs are more commonly applied within the scope of maintenance for failure and degradation prediction such as in the work of Jones et al. (2010), where BNs are used to predict equipment failure in a manufacturing industry, or in the work of Ferreiro et al. (2012), where they are used to estimate brake

4

wear in aeronautics. A particularly interesting work is that of Neil and Marquez (2012). The authors use a hybrid BN to model corrective maintenance, logistics delays, and scheduled maintenance time distributions to estimate the renewal time of repairable systems. Weber et al. (2012) present a comprehensive literature review on the application of BNs to dependability, risk, and maintenance. An area in which BNs have been profusely used is safety and risk analysis. In Khakzad et al. (2011), the authors discussed the advantages of BNs over Fault Trees (FTs), which have been a conventional technique in this particular area. According to the authors, BNs are able to explicitly represent dependencies between events, updating of probabilities, coping with uncertainties, and including expert opinion. Examples of the application of BNs in safety and risk analysis include: modelling domino effects, i.e. chains of accidents, regarding propagation patterns and probability estimation in chemical and process plants (Khakzad et al. 2013b, 2016, 2018); modelling risk factors such as ice, severe operating conditions, and climatic changes in marine transportation (Afenyo et al. 2017; Baksh et al. 2018; Khan et al. 2018); modelling the influence of human and organizational factors in ship accidents (Sotiralis et al. 2016); modelling operational conditions in offshore drilling (Khakzad et al. 2013a; Abimbola et al. 2015; Abimbola and Khan 2016); or modelling deterioration processes of subsea pipelines (Arzaghi et al. 2017; Li et al. 2017; Yang et al. 2017). Particularly in Arzaghi et al. (2017), the authors developed a model capable of supporting decision-making about repairs and maintenance methods of subsea pipelines given the observation of a given damage state. Not being directly related with capacity planning, these examples demonstrate the versatility and applicability of BNs, particularly in maintenance problems.

5

Under this context, BNs are considered a suitable big data and predictive analytics (BDPA) (Gandomi and Haider 2015; Hazen et al. 2016) tool to be used within aircraft MROs, providing meaning to the large amount of data generated and stored during the planning process, and transforming such data into valuable information. The BNs developed in this paper address the uncertainty of workloads estimation in aircraft maintenance, considering variables that are typically disregarded from the capacity planning process such as the aircraft operator, the aircraft version, and the aircraft utilization. The benefits of applying BNs as a BDPA tool in aircraft maintenance are demonstrated through two practical examples in which accurate estimations and informed decisions regarding maintenance workloads are made possible, and the improvement of such estimations and decisions is expected as new information becomes available. One example refers to capacity planning, in which workloads are inferred by instantiating variables with different information. A second example refers to sales planning, in which operators, maintenance services, and aircraft versions, are inferred by instantiating workloads representing available manpower. Real maintenance data is used, collected from a Portuguese independent aircraft MRO whose experts were involved in the definition of objectives, selection of variables, and establishment of the BNs structure. This allowed the development of a generic BDPA tool based on BNs, applicable to any aircraft MRO, or even to other MRO industries with due adaptations. The remainder of the paper is organized as follows. In Section 2, the aircraft maintenance planning process is described and the problem resulting from its uncertainty is explained. In Section 3, basic concepts related to BNs are presented. In Section 4, the dataset used in the development of the BNs is characterized. In Section 5, the steps followed in the construction and validation of the BNs are explained. In Section 6, two examples on the application of the developed BDPA tool are presented.

6

Finally, in Section 7, conclusions are drawn on the performed work and future research opportunities are identified.

2.

Aircraft Maintenance Planning Process

The maintenance planning process of aircraft MROs is presented in

7

Fig. 1. After receiving a request for quotation for a given maintenance intervention, the MRO gathers data about the aircraft that is to be intervened and about the maintenance intervention that is to be performed. Gathered data includes the aircraft operator, the aircraft family, and the type of maintenance service that is to be contracted. The MRO then develops a tactical plan and elaborates a proposal. The proposal contains workload and cost estimations, divided between scheduled and unscheduled maintenance work, and defines a delivery date. Estimations for the scheduled and unscheduled maintenance workloads are commonly based on average values of past workloads for the type of maintenance intervention being planned, and on the experience of the MRO in performing similar interventions. In order to address the uncertainty of the estimations, MRO organizations rely fundamentally on the decision-maker experience in overplanning workloads and costs. By doing so, MROs assume a considerable risk regarding the ability to meet the actual workload and to perform according to the contracted service price. Despite all uncertainty, a commitment must be made between the MRO and the client regarding the delivery date and the service price. If the client accepts the proposal, a project is formalized, containing a list of scheduled maintenance tasks and a rough workload estimation for the unscheduled maintenance work. The scheduled tasks include workloads, materials, and spare parts, while the unscheduled work is not completely known until the execution of the maintenance intervention, more specifically until the end of the aircraft inspection. A considerable amount of data results from the planning process. During the tactical phase, mainly unstructured data (Gandomi and Haider 2015) about the maintenance service is generated such as email correspondence, websites about operators and aircraft, images about known damages to the aircraft, and digitized formal documents (e.g. proposals and contracts). On the contrary, during the operational phase,

8

mainly structured data (Gandomi and Haider 2015) are used, including data about maintenance tasks, materials, spare parts, tools, schedules, costs, etc., which are stored in the form of databases. Nonetheless, such data are yet to be effectively used to assess the uncertainty of capacity planning and to improve the MROs decision-making process.

9

3.

A Review on Bayesian Networks

3.1

Basic Concepts

Bayesian networks (Pearl 1988) represent and process knowledge probabilistically, making them an ideal tool for reasoning and decision-making under uncertainty (Kjærulff and Madsen 2008). They consist in a qualitative part, a directed acyclic graph (DAG), and a quantitative part, a set of conditional probability tables (CPTs) (Jensen and Nielsen 2007). The DAG contains nodes representing random variables and directed arcs representing dependencies or causal relationships between variables (Weber et al. 2012). A joint probability distribution is defined over the variables according to the directed arcs. More specifically, the joint probability distribution, 𝑃(𝑋) , over the set of variables 𝑋 is given by equation (1): 𝒏

𝑷(𝑿) = ∏𝒊 = 𝟏𝑷(𝑿𝒊|𝑿𝒑𝒂(𝒊))

(1)

where 𝑋𝑝𝑎(𝑖) is the set of the parent variables of 𝑋𝑖 for each node 𝑖 = 1,…, 𝑛. A parent variable 𝑋𝑝𝑎(𝑖) is such that it has an outgoing arc directed at a child variable 𝑋𝑖. If no arc exists between two variables, a conditional independence statement is specified between those two variables. Dependence and independence assumptions allow to perform 10

inference with BNs through conditional probabilities of the form of equation (2): 𝑷(𝑿𝒊 = 𝒙𝒊 | 𝑿𝒑𝒂(𝒊) = 𝒙𝒑𝒂(𝒊)) = 𝒛

(2)

where 𝑥𝑖 and 𝑥𝑝𝑎(𝑖) are values assigned to the variables 𝑋𝑖 and 𝑋𝑝𝑎(𝑖), respectively, and 𝑧 is the probability of 𝑋𝑖 = 𝑥𝑖 given that 𝑋𝑝𝑎(𝑖) = 𝑥𝑝𝑎(𝑖). In practice, this allows the probabilities associated to the values (or states) of any variable to be computed given the value of one or several variables in the network. Fig. 2 presents a BN example with variables 𝑋 = (𝑋1, 𝑋2, 𝑋3, 𝑋4, 𝑋5). The probabilities to specify are 𝑃(𝑋1), 𝑃(𝑋2), 𝑃(𝑋3 | 𝑋1, 𝑋2), 𝑃(𝑋4 | 𝑋3), and 𝑃(𝑋5 | 𝑋3). The joint probability distribution is given by 𝑃

(𝑋1, 𝑋2, 𝑋3, 𝑋4, 𝑋5) = 𝑃(𝑋1).𝑃(𝑋2).𝑃(𝑋3 | 𝑋1, 𝑋2). 𝑃(𝑋4 | 𝑋3). 𝑃(𝑋5 | 𝑋3). The concept of d-separation is an important concept to understand reasoning under the use of BNs: two variables, 𝑋1 and 𝑋3, are said to be d-separated if between them there is a third variable, 𝑋2, such that either the connection between the variables is serial or diverging and 𝑋2 is instantiated, i.e. its value is known through evidence, or the connection is converging and the value of 𝑋2 is unknown (Jensen and Nielsen 2007). Fig. 3 presents the three types of connections that exist in BNs. In a serial connection (a), if the value of 𝑋2 is known, any knowledge about 𝑋1 is irrelevant for the knowledge about 𝑋3. The same can be said about a diverging connection (b). On the contrary, in a converging connection (c), only if the value of 𝑋2 is known, any knowledge about 𝑋1 can influence the knowledge about 𝑋3. This is called the explaining away effect.

11

The explaining away effect is what allows inter-causal reasoning to be performed with BNs: if the consequences are known, knowledge about one of two or more competing hypotheses will influence the knowledge about the remainder. Other possible types of reasoning with BNs are causal reasoning (also called deductive or predictive reasoning), which follows the direction of the arcs between variables, and diagnostic reasoning (also called abductive reasoning), which follows the opposite direction of the arcs (Kjærulff and Madsen 2008).

3.2

Building BNs

The qualitative and quantitative parts of BNs can be established through ‘manual’ or, otherwise, ‘automatic’ means. With the word ‘manual’ one means the definition by a domain expert of variables and arcs (and corresponding values and directions) for the qualitative part, and of the resulting conditional probabilities for the quantitative part. On the contrary, the word ‘automatic’ refers to the computational methods capable of defining the structure of a BN and its CPTs from data. The structures of the BNs presented in this work were manually defined, while the conditional probabilities were automatically established, thus emphasis is given on these approaches. Adapted from Eickemeyer et al. (2013), a five-step process for the development of BNs was followed: (1) Formulation of network objectives; (2) Definition of variables; (3) Definition of network structure; (4) Definition of network CPTs; and (5) Validation. These steps are thoroughly explained in Section 5.

4.

Dataset

As mentioned in previous Sections, the work presented in this paper is based on real maintenance data from a Portuguese aircraft MRO. The studied dataset refers to 372 maintenance projects of a given aircraft family, occurred between September 2002 and 12

December 2015. Two types of data were collected: (1) data referring to the identification and characterization of each of the 372 maintenance projects (Table 1); and (2) data referring to the maintenance tasks performed in each maintenance project (Table 2). The collected projects referred to a significant number of different maintenance interventions, or maintenance event types (Table 1, No.4), which were classified according to the classification system presented in Fig. 4. The considered classification system, which was validated by experts of the host MRO, contains a total of 43 different maintenance events divided between Standard Maintenance Checks, Non-Standard Maintenance Checks, and Other Services. The 372 collected projects varied significantly regarding the amount of maintenance tasks, with projects referring to complaints, material supply, missions, refueling, among others, with only one task, and others referring to heavy checks2 involving up to around 2000 maintenance tasks. As for the development of the BNs presented in Section 5, referring to a particular aircraft family as mentioned before, the following data requirements had to be met: (1) each maintenance project (Table 1, No.1) was characterized by the aircraft version (Table 1, No.2), the aircraft operator (Table 1, No.3), the maintenance event type (Table 1, No.4), and the aircraft utilization (Table 1, No.7); and (2) each maintenance task of each project was characterized by the work type (Table 2, No.3), the work phase (Table 2, No.5), the work skill (Table 2, No.6), and the workload in man-hours (Table 2, No.7). These variables are further detailed in Section 5.2.

2

In aircraft maintenance a check is a periodic maintenance intervention composed mainly by scheduled tasks with equal time intervals and resulting unscheduled tasks. 13

The collected data allowed a quantitative assessment to be performed on the actual workloads for the scheduled and unscheduled maintenance. The assessment is presented in Fig. 5. As mentioned by Samaranayake and Kiridena (2012), the uncertainty of the maintenance work, particularly of the unscheduled maintenance, is an important problem to be addressed by MRO organizations given the additional costs and delays that may occur. Fig. 5 shows the relative proportion of average workloads for scheduled and unscheduled maintenance of ‘C’ checks, having the average scheduled maintenance workload of the 5000 flight hours (FH) ‘C’ check as reference (100%). According to the linear regressions obtained through the least squares method, both scheduled and unscheduled maintenance workloads are expected to increase by 18% and 46%, respectively, at each ‘C’ check, from the 5000 FH ‘C’ check to the 40000 FH ‘4C’ check. As explained in Section 1 and considering these results, the uncertainty of the maintenance workload, particularly of unscheduled maintenance, represents a serious threat to the capacity planning process.

5.

Building BNs for the Aircraft Capacity Planning Problem

This Section describes the steps followed in the development of the BNs (Fig. 6). Section 5.1 explains the objective of the developed BNs, i.e. their purpose. Section 5.2 describes the defined variables, divided between information variables and hypothesis variables. Section 5.3 presents the structure of the BNs, i.e. the relationships between variables. Section 5.4 explains how the CPTs were estimated. Finally, Section 5.5 presents the validation process of the BNs.

5.1

Formulation of network objectives

The BNs presented in this paper are intended to be used as a BDPA tool to improve the output of the maintenance capacity planning process within aircraft MROs. More 14

precisely, they are intended to improve the accuracy of the workload estimations and to provide MROs with a means to contemplate the uncertainty of the decision-making process. Instead of average values calculated from past maintenance events, estimated workloads should be based on probability distributions conditioned to the existing information at the time of the planning process, which may be incomplete (Pearl 1988).

5.2

Definition of variables

There are two main types of variables3 in BNs: (1) information variables, referring to the events that are observable and that reveal something about the hypothesis variables; and (2) hypothesis variables, referring to the events that are not directly observable and to which posterior probabilities are to be calculated (Jensen and Nielsen 2007).

5.2.1

Information Variables

The information variables of the developed BNs refer to the information that MROs typically receive in the tactical phase of the capacity planning process such as the aircraft operator, the maintenance intervention to be performed, the aircraft version, and the aircraft utilization type, which may be used to infer the expected workloads of maintenance interventions. In other words, dependence and independence relationships may be established between these information variables and the hypothesis variables.

3

Only chance variables, i.e. random variables, are considered in this paper. Other decision models, such as influence diagrams, contain also decision variables and utility functions (Jensen and Nielsen 2007; Kjærulff and Madsen 2008). 15

5.2.1.1

Operator

The operator variable (‘Operator’) identifies the operator of the aircraft at the time of the maintenance intervention. Given the difficulty in obtaining other relevant information for the capacity planning process, one can consider that the ‘Operator’ summarizes a set of factors which impact the aircraft wear and subsequent expected maintenance workload. Such factors can be differentiated between environmental and operational. Environmental factors refer to regional weather conditions, corrosion severity indexes (Cessna 2013), runways condition (ICAO), etc., while operational factors refer to crew training and experience, operating policies, etc. A total of 35 different operators were identified in the collected data.

5.2.1.2

Event Type

The event type variable (‘EventType’) refers to the maintenance interventions applicable to the studied aircraft family. Each maintenance project was classified by a maintenance event according to the classification system presented in Fig. 4.

5.2.1.3

Aircraft Version

The aircraft version variable (‘AircraftVersion’) refers to the aircraft variant within its family. The studied aircraft family has a total of 11 commercial versions, from which 8 16

were found in the collected data. Maintenance requirements differ according to the aircraft version, i.e. maintenance tasks may be specific to a given aircraft variant, being fundamental to establish a variable for this information.

5.2.1.4

Utilization

The utilization variable (‘Utilization’) refers to the type of utilization an operator performs with its aircraft. The manufacturer of the studied aircraft family differentiates between ‘normal utilization’ and ‘low utilization’ aircraft and establishes specific maintenance requirements for each. ‘Normal utilization’ aircraft are those that have average annual utilizations of 2000 to 2700 FH, e.g. airlines, and ‘low utilization’ aircraft are those that operate less than 1500 FH a year, e.g. private or governmental operators.

5.2.2

Hypothesis Variables

In the developed BNs, the hypothesis variables refer to the workloads that are to be estimated. These variables are defined through continuous domains, however, in order to use CPTs to encode the uncertainty of the models, such variables have to be discretized in a finite set of intervals (Jensen and Nielsen 2007). According to the inputs provided by the host MRO experts, four types of workloads were defined in order to fully characterize the maintenance work for capacity planning: (1) total workload; (2) workload per work type; (3) workload per work phase; and (4) workload per work skill.

5.2.2.1

Total Workload

17

The total workload is calculated by summing, within each maintenance project p, the workload of each maintenance task t according to equation (3): 𝑊𝐿𝑝 = ∑𝑡𝑈𝑝𝑤𝑙𝑡 ∀ 𝑝

(3)

where 𝑊𝐿𝑝 is the total workload of project p (in man-hours) and 𝑤𝑙𝑡 is the workload of maintenance task t (in man-hours). The total workload, measured in man-hours, would be naturally represented by a continuous variable with (theoretical) domain [0, +∞[. However, in order to use CPTs to infer the posterior probabilities of workloads, these were discretized in a finite number of intervals (Jensen and Nielsen 2007). In the case of the total workload variable (‘TotalWL’), in addition to an interval of [0, 0], referring to a workload of 0 man-hours, i.e. the inexistence of work, intervals of 500 man-hours were established, from 0 manhours to the interval that contained the maximum workload value observed in the collected data. This discretization was established in accordance with the inputs provided by the host MRO experts, which considered them adequate to be used in the capacity planning process.

5.2.2.2

Workload per Work Type

The work types are related to the classification of the maintenance tasks as defined by the host MRO. For example, scheduled tasks are differentiated between reception inspection, final test, etc., while unscheduled tasks are differentiated between log book anomaly, inspection anomaly, etc. In addition, tasks referring to maintenance actions enforced, or otherwise recommended, by regulators, are classified as Airworthiness Directives (ADs), or Service Bulletins (SBs), respectively. A total of 14 classifications were found in the data, grouped in 6 work types to be used in the BNs. The considered work types, referring each to a hypothesis variable, are presented in 18

Table 3.

The workload of each work type is calculated by summing, within each maintenance project p, the workload of each maintenance task t referring to each work type i according to equation (4): 𝑊𝑇𝑖𝑝 = ∑𝑡𝑈𝑝𝑤𝑡𝑖𝑡 ∀ 𝑖, 𝑝

(4)

where 𝑊𝑇𝑖𝑝 is the workload of work type i in project p (in man-hours) and 𝑤𝑡𝑖𝑡 is the workload of maintenance task t referring to work type i (in man-hours). Similarly to the total workload variable, each of the six variables referring to the workload per work type were also discretized. Besides the interval of [0, 0], intervals of 100 man-hours were established, from 0 man-hours to the interval that contained the maximum workload value observed in the collected data for the considered variable.

5.2.2.3

Workload per Work Phase

Virtually all maintenance interventions unfold according to a chronological order that can be used to establish work phases. These phases can be used to define timeframes in the project for the accomplishment of the maintenance tasks. The considered work phases, referring each to a hypothesis variable, are presented in Table 4. The workload of each work phase is calculated by summing, within each maintenance project p, the workload of each maintenance task t referring to each work phase j according to equation (5): 19

𝑊𝑃𝑗𝑝 = ∑𝑡𝑈𝑝𝑤𝑝𝑗𝑡 ∀ 𝑗, 𝑝

(5)

where 𝑊𝑃𝑗𝑝 is the workload of work phase j in project p (in man-hours) and 𝑤𝑝𝑗𝑡 is the workload of maintenance task t referring to work phase j (in man-hours). The variables referring to the workload per work phase were also discretized in 100 man-hours intervals, from 0 man-hours to the interval that contained the maximum workload value observed in the collected data for the considered variable, including the interval of [0, 0] man-hours.

5.2.2.4

Workload per Work Skill

Each maintenance technician is licensed by the competent aviation authority to perform only a certain type of tasks in a given aircraft family, according to the work skills that she/he possesses (Dijkstra et al. 1991). A total of 58 skills designations were found in the collected data, some referring to the same technical skills. After removing redundant designations, a total of 35 unique maintenance skills were obtained. From these, 5 skills accounted for around 90 % of the performed work. The remainder 30 were included in ‘Other’. The considered maintenance skills, referring each to a hypothesis variable, are presented in Table 5. The workload of each work skill is calculated by summing, within each maintenance project p, the workload of each maintenance task t referring to each work skill k according to equation (6): 𝑊𝑆𝑘𝑝 = ∑𝑡𝑈𝑝𝑤𝑠𝑘𝑡 ∀ 𝑘, 𝑝

(6)

where 𝑊𝑆𝑘𝑝 is the workload of work skill k in project p (in man-hours) and 𝑤𝑠𝑘𝑡 is the workload of maintenance task t referring to the work skill k (in man-hours).

20

The variables referring to the workload per work skill were also discretized in 100 man-hours intervals, from 0 man-hours to the interval that contained the maximum workload value observed in the collected data, including the interval of [0, 0] manhours. It is important to note that the four summations presented before are equal according to equation (7): 𝑊𝐿𝑝 = 𝑊𝑇𝑖𝑝 = 𝑊𝑃𝑗𝑝 = 𝑊𝑆𝑘𝑝 ∀ 𝑖, 𝑗, 𝑘, 𝑝 5.3

(7)

Definition of network structure

After the definition of the variables to be used in the BNs, it was necessary to establish their dependencies by means of directed arcs. Although the present BN structures have been ‘manually’ defined, computational methods based on learning algorithms exist to ‘automatically’ estimate the structure of a BN from data. Examples of such algorithms include: (1) Bayesian Search (Cooper and Herskovits 1992; Heckerman and Shachter 1995); (2) PC (Spirtes et al. 1993); (3) Essential Graph Search (Dash and Druzdzel 1999); (4) Greedy Thick Thinning (Cheng et al. 1997); (5) Tree Augmented Naive Bayes (Friedman et al. 1997); and (6) Augmented Naive Bayes (Friedman et al. 1997). All the aforementioned learning algorithms were tested for the construction of the BNs prior to their manual definition, but, nonetheless, unsatisfactory results were obtained with the present dataset. The most important observation was that a different network was obtained with each algorithm, differing each network in the number and directions of the arcs, and none corresponded to the expectations that the experts had about the model structure. Most importantly, the algorithms did not establish arcs consistently between those that were defined as information variables and those that were defined as hypothesis variables, being it in the existence/inexistence of the arcs themselves, or being it in their directions, which were often established from the 21

hypothesis variables to the information variables, i.e. in the opposite direction of what was intuitively considered the dependence relationships between variables. For example, all networks obtained with the learning algorithms established one or more arcs from different hypothesis variables to the ‘EventType’ variable. This is not a reasonable assumption to make, as the workloads are dependent on the type of the maintenance interventions and not the contrary. The fact that all networks were not in accordance with the expectations of the experts regarding the model structure is not sufficient to justify the dismissal of the learning algorithms per se, as unknown or hidden relationships between variables may exist, or the assumptions of the experts may be wrong altogether. However, as mentioned before, each network was different from the remainder and by not presenting consistent results regarding the arcs between variables and their directions, it would be unreasonable to assume that any network was preferable to the remainder, i.e. that the arcs of a given network better described the dependence and independence relationships between variables. Thus, although the software used in the development of the BNs, the GeNIe software (Druzdzel 1999), allows structural learning from data using the algorithms presented above, the decision was made to manually define the structures of the BNs, i.e. to establish the dependencies between variables in the form of directed arcs according to the knowledge of the host MRO experts, and to validate the structures afterwards as explained in Section 5.5. Instead of building a single BN considering all the hypothesis variables defined in Section 5.2, i.e. total workload, workload per work type, workload per work phase, and workload per work skill, which, besides being intractable for parameter learning, would be illegible for decision-making, three BNs were built: (1) considering the total

22

workload and workload per work type; (2) considering the total workload and workload per work phase; and (3) considering the total workload and workload per work skill. The first BN characterizes the total workload in terms of work types, i.e. it provides estimates about workloads referring to scheduled maintenance, unscheduled maintenance, and others, through the variables defined in

23

Table 3. The second BN characterizes the total workload in terms of work phases, i.e. it provides estimates about workloads referring to different time periods of the maintenance event through the variables defined in Table 4. Finally, the third BN characterizes the total workload in terms of work skills, i.e. it provides estimates about workloads referring to the different technical skills through the variables defined in Table 5. The combined use of the developed BNs in the same BDPA tool ensures the representativeness of the capacity planning problem and allows a comprehensive understanding of the expected maintenance work early in the planning process.

5.3.1

Relationships between Information Variables

The main purpose of using the developed BNs as a BDPA tool is to help MROs determine what workloads are the most likely to occur provided a set of evidences. Such reasoning is of the causal type, as explained in Section 3.1. However, BNs are intended to be used in situations with incomplete information, thus being advisable to perform inter-causal reasoning for the information variables. For example, by knowing the operator, the utilization type, and the maintenance event of a given aircraft, it should be possible to infer the aircraft version in the case of not knowing it. Similar reasoning is possible for other situations of incomplete information. Fig. 7 presents the connections between information variables that were established and that are common to the three developed BNs.

5.3.2

Relationships between Hypothesis Variables

For the three BNs built, hypothesis variables are considered conditionally independent between them, i.e. no arcs exist connecting those variables. In practice, this means that by instantiating a hypothesis variable, the probabilities associated with the other hypothesis variables in the BN will not change. Such assumption is a simplification 24

since relationships exist between hypothesis variables. For example, part of the unscheduled (inspections),

maintenance hence

arcs

(repairs) could

results be

from

established

the

scheduled

from

maintenance

‘ScheduledWL’

to

‘UnscheduledWL’ in the BN referring to the work types, or from ‘P3_InspectionWL’ to ‘P4_RepairsWL’ in the BN referring to the work phases. However, considering such relationships turned the BNs intractable and minimal benefits were obtained in the results. The only exception to the conditional independence assumptions between hypothesis variables was the establishment of arcs from ‘TotalWL’ to all the remainder hypothesis variables, within each of the three developed BNs. The purpose of such connections is to provide an indication on how the remainder hypothesis variables vary after instantiating the ‘TotalWL’ variable. From a decision-making perspective, it is important to know what is the most likely workload distribution per work type, per work phase, and per work skill, after a value for the ‘TotalWL’ variable is selected. Fig. 8 presents the connections between hypothesis variables for the work phases BN.

5.3.3

Relationships between Information and Hypothesis Variables

In the proposed BNs, all the hypothesis variables were assumed to be conditionally dependent on all the information variables, i.e. arcs were established from all the information variables to all the hypothesis variables, within each of the three proposed BNs. Fig. 9 presents the established relationships between information and hypothesis variables for the work phases BN. With these assumptions, workloads may be inferred in cases of incomplete information, in which evidence is provided to only some of the information variables.

25

5.4

Definition of network CPTs

The quantitative part of a BN refers to the CPTs established after its structure, which are populated with parameters, i.e. conditional probabilities. Depending on the complexity of the BN, specifying these parameters ‘manually’ can be a demanding task, if not an impossible one. An alternative to the ‘manual’ process of eliciting probabilities is to use computationally implemented algorithms capable of learning parameters from data. A widely used algorithm for parameter estimation is the Expectation-Maximization (EM) algorithm (Dempster et al. 1977; Lauritzen 1995). The EM algorithm calculates maximum likelihood estimates for the parameters from datasets that may contain missing values. The expectation step (or E-step) consists in the calculation of expectations for the missing values using the estimates of existing parameters, while in the maximization step (or M-step) new maximum likelihood estimates are calculated using the original dataset plus the expected missing values found in the expectation step. The algorithm then runs iteratively for a predetermined number of iterations or until it reaches convergence (Jensen and Nielsen 2007). As the used dataset is considered complete and without missing values (as a legal requirement MROs have to collect and store data about the performed maintenance interventions which is periodically audited), the exact maximum likelihood estimates are possible to be calculated by simply counting frequencies in the database (Jensen and Nielsen 2007). As such, one may consider that the expectation step is not as fundamental as the maximization step for the calculation of the CPTs in this particular case. Nonetheless, the fact that maintenance databases may be considered complete in aviation contexts, constitute a particular case in industrial practice.

26

The software used for the development of the BNs is able to estimate parameters from data with the EM algorithm. After the manual definition of the BNs structure, parameters were automatically calculated with the GeNIe software.

5.5

Validation

After the establishment of a BN, it is necessary to evaluate if its output is in accordance with the knowledge one possesses about the problem being modelled. In practice, this is achieved by evaluating if the probabilities referring to the values of the hypothesis variables behave as expected after instantiating the information variables for wellknown scenarios (Langseth and Portinale 2007). If unsatisfactory results are obtained, the DAG should be iteratively improved until no substantial benefit is gained by further modifying the model. Following the process presented in Fig. 6, the developed BNs were iteratively modified until a significant amount of confidence was gained by the MRO experts in using the BNs. First, variables considered redundant were removed from the BNs, and dependencies between information variables were defined to allow inter-causal reasoning and probabilities to be estimated in situations of incomplete information. Second, the discretization of the hypothesis variables changed until adequate intervals for the decision-making process were found. Third, parameters were estimated from data using the EM algorithm. Several tests are proposed in the literature for the validation of BNs and the type of test applied to a particular model may depend on the availability of data. On one extreme, BNs are learned and validated through automatic methods from complete datasets. On the other extreme, BNs are completely established from expert knowledge and may be validated through conceptual approaches such as that proposed by Pitchforth and Mengersen (2013). In this paper, given the availability of data and the 27

fact that the DAGs were established from expert knowledge, an alternative validation method is proposed. The validation process of the developed BNs is represented in Fig. 10 and explained in the following Sections.

28

5.5.1

BN/Test BN/Dataset

In order to assess the performance of the proposed BNs and their associated uncertainty, a second set of BNs was established. Marcot (2012) defines performance as how well a BN predicts an outcome, i.e. the accuracy of the results, and uncertainty, as the dispersion of posterior probabilities among different outcome states, i.e. the spread of alternative predictions. The new BNs, henceforth called ‘test BNs’, are in all identical to the proposed BNs, considering the same variables, values, and conditional dependencies, except for the information variables that are now considered conditionally independent between themselves, i.e. no arcs are established between them. This allows the performance and the uncertainty of the proposed BNs to be assessed by comparison with equivalent, but simpler test BNs. The dataset is used for the calculation of the conditional relative frequencies of the hypothesis variables states, i.e. the workload intervals, used as a reference measure for the results obtained with the BNs.

5.5.2

Information Variable Instantiation/Event Definition

After the definition of the test BNs, the same information variable is instantiated in the proposed BN and in the equivalent test BN. Regarding the dataset, an equivalent event 29

is defined to calculate the relative frequencies of the hypothesis variables states conditioned to that event. For example, instantiating the proposed and test BNs with the evidence ‘Operator 9’ is equivalent to filter out all data from the dataset not referring to ‘Operator 9’ and to calculate frequencies from this data sample.

5.5.3

Posterior Probabilities/Conditional Relative Frequencies

For both the proposed and test BNs, after the instantiation of a given information variable, the posterior probabilities of all states of all the hypothesis variables are inferred. Results for the proposed BNs (considering the relationships between information variables) are denoted as 𝐵p𝑖, and results for the test BNs (not considering the relationships between information variables) are denoted as 𝐵q𝑖. Regarding the dataset, relative frequencies are calculated for all states of all the hypothesis variables, conditioned to an equivalent event as explained before. These conditional relative frequencies are denoted as 𝐹𝑖. As an example, Table 6 and Fig. 11 present the probabilities for the ‘TotalWL’ variable after instantiating the ‘Operator’ variable with ‘Operator09’ and the ‘EventType’ variable with ‘4C’. In this example, the probability distribution estimated by the test BNs, 𝐵q𝑖, presents a higher uncertainty, i.e. larger dispersion, than that presented by the proposed BNs, 𝐵p𝑖, or that presented by the conditional relative frequencies, 𝐹𝑖. In fact, these two later distributions, 𝐵p𝑖 and 𝐹𝑖, practically overlap. Similar results were obtained for almost all other hypothesis variables, after a different number of evidences have been provided to the information variables.

30

5.5.4

Mean Absolute Deviation (MAD) Calculation

The results obtained with the two sets of BNs were then compared with the conditional relative frequencies calculated from data using the mean absolute deviation (MAD), an error measure commonly used to evaluate forecasting methods (Regattieri et al. 2005). First, errors were calculated for all states, in all hypothesis variables, between the proposed BNs and the relative frequencies calculated from data according to equation (8), and between the test BNs and the relative frequencies according to equation (9):

|𝐸p𝑖| = |𝐹𝑖 ‒ 𝐵p𝑖|

(8)

|𝐸q𝑖| = |𝐹𝑖 ‒ 𝐵q𝑖|

(9)

where: 𝐸p𝑖 – error of interval i of hypothesis variable h 𝐹𝑖 – conditional relative frequency of interval i of hypothesis variable h, calculated from data 𝐵p𝑖 – conditional probability of interval i of hypothesis variable h, estimated by the proposed BN p 𝐸q𝑖 – error of interval i of hypothesis variable h 𝐵q𝑖 – conditional probability of interval i of hypothesis variable h, estimated by the test BN q Second, after the calculation of the errors, the MAD of each hypothesis variable h was calculated according to equation (10) for the proposed BNs, and according to equation (11) for the test BNs: 1

𝑛 𝑀𝐴𝐷pℎ = 𝑛∑𝑖 = 1|𝐸p𝑖|

1

𝑛 𝑀𝐴𝐷qℎ = 𝑛∑𝑖 = 1|𝐸q𝑖|

(10)

(11) 31

where: 𝑀𝐴𝐷pℎ – mean absolute deviation of hypothesis variable h of the proposed BN p (considering the relationships between information variables) 𝑀𝐴𝐷qℎ – mean absolute deviation of hypothesis variable h of the test BN q (not considering the relationships between information variables) 𝑛 – number of discrete intervals of hypothesis variable h MADs were calculated each time a new information variable was instantiated in the BNs and an equivalent event was defined in the dataset. This process repeated itself until all information variables were instantiated.

5.5.5

MAD Comparison

After the instantiation of all information variables and the calculation of the corresponding MADs, the results of the proposed BNs were compared with those of the test BNs. Table 7 presents the evolution of the MADs in the two sets of BNs and having the workload per work phase BN as example. The results presented in Table 7 confirm the relevance of the structure in the performance and uncertainty of BNs. They also confirm the superiority of the proposed BNs when compared to BNs not considering conditional dependencies between the defined information variables. The proposed BNs present overall lower mean absolute deviations (MADp) than those presented by the test BNs (MADq). The test BNs only present lower MADs, but nonetheless comparable to those obtained with the proposed BNs, in the case when one evidence is provided (Evidence 1 - ‘Operator’). To better highlight the differences between the results obtained with the proposed and the test BNs, Table 8 presents the maximum error values for each of the hypothesis variables in the workload per work phase BN. For example, although the MADq of the ‘TotalWL’

32

variable with two evidences provided is 4,06% (Table 7), the maximum absolute error, Max |Eq|, is 20,70% (Table 8). The MAD is therefore proposed to be used for the validation of BNs in cases where the estimations from two sets of BNs, the proposed BNs and the test BNs, can be compared with conditional relative frequencies calculated from data. Differences in the MAD between both BNs can be used to assess which one is better modelling the existing data.

6. Application Examples In order to demonstrate the benefits of applying BNs as a BDPA tool in aircraft maintenance, two examples using real maintenance data are hereby presented, one regarding capacity planning in Section 6.1, and one regarding sales planning in Section 6.2. Section 6.3 presents a discussion on the obtained results.

6.1

Example 1 – Capacity Planning

This example focuses on the application of the developed BNs for capacity planning. Total and work phases workloads are estimated through the instantiation of the ‘Operator’, ‘Event Type’, and ‘Aircraft Version’ information variables. Two cases are presented in order to highlight the relevance of information for the accuracy of the workload estimations and the results obtained with the developed BNs are compared with the following: (1) a model simulating the maintenance estimation method currently used by the host MRO; (2) a probabilistic model; and (3) actual workloads of real maintenance checks of the same type as the ones being predicted.

6.1.1

‘Operator09’, ‘6C’ check, ‘B1’ aircraft version

The host MRO receives a quotation request from ‘Operator09’ to perform a 33

maintenance intervention not yet specified regarding its type. As presented in Fig. 12, by instantiating the ‘Operator’ variable with the ‘Operator09’ state, under ‘EventType’ the most probable maintenance interventions to be performed are standard maintenance checks for normal utilization (Fig. 4); under ‘AircraftVersion’ the most probable aircraft versions to be intervened are the ‘A1’ and ‘B1’; and under ‘Utilization’ the operator is identified as being a ‘normal utilization’ operator, i.e. an airline. After further inquiring the client during the planning process, the maintenance intervention is identified as a ‘6C’ check and the aircraft version is determined to be ‘B1’. This information is included in the BN under ‘EventType’ and ‘AircraftVersion’, respectively. By providing the three evidences to the BN – ‘Operator09’, ‘6C’ check, and ‘B1’ version – the results presented in Table 9 and Fig. 13 are obtained for the ‘TotalWL’ variable.

According to the host MRO experts attitude towards risk, a policy of planning a workload interval with 75% of minimum cumulative probability for the ‘TotalWL’ variable is established. The ]5000, 5500] workload interval (Table 9) is therefore selected. The results presented in

34

Table 10 and Fig. 14 are obtained for the remainder hypothesis variables after instantiating the ‘TotalWL’ with the ]5000, 5500] interval, and maintaining the same 75% of minimum cumulative probability policy. Besides the workload intervals obtained with the developed BNs (‘BN’),

35

Table 10 also presents the workloads obtained with a model simulating the estimation method currently used by the host MRO in capacity planning (‘Current Method’), workload estimations obtained with a probabilistic model (‘Probabilistic Model’), and actual workloads of a ‘6C’ check from ‘Operator09’ and ‘B1’ aircraft version (‘Actual Workload’), performed after the definition of the models and not considered in the dataset. The model simulating the estimation method currently used is based on average workloads of past checks and considers 10% of overplanning. The probabilistic model assumes the lognormal distribution as a descriptor of the total workload4, with mean and standard deviation calculated from past data, and assigns past ratios for the remainder hypothesis variables. For example, if a given hypothesis variable had an average 10% ratio relatively to the total workload in a given check type, this ratio would be assigned to the same variable considering the total workload obtained from the lognormal distribution. In addition, the total workload referring to the 75% quantile was considered, maintaining the criterion explained before. According to the obtained results, by planning the maximum workload of each interval obtained with the proposed BN, the MRO would plan 492 man-hours over the actual, observed, total workload, against 1055 man-hours less than the actual workload with the current method, and 588 man-hours less by using a probabilistic model based on the lognormal distribution.

4

The lognormal distribution is considered in this paper as an adequate approximation of maintenance workloads as it is strictly positive and right-skewed (e.g. Baker and Trietsch 2009). In the particular case of corrective maintenance, Kline (1984) concluded that the lognormal is a suitable descriptor of repair times. 36

6.1.2

‘Operator31’, ‘6C’ check, ‘B1’ aircraft version

If the MRO receives a quotation request for the same ‘6C’ check and for the same ‘B1’ version aircraft, but for ‘Operator31’ instead of ‘Operator09’, the results presented in Table 11 are obtained for the ‘TotalWL’ variable. Considering the same policy of selecting an interval with 75% of minimum cumulative probability for the ‘TotalWL’ variable, the MRO would select the ]1000, 1500] workload interval (Table 11). The results presented in Table 12 are obtained for the remainder hypothesis variables in the BN after instantiating the ‘TotalWL’ variable with the ]1000, 1500] interval, maintaining the same policy of selecting intervals with 75% of minimum cumulative probability. According to the obtained results, the MRO would plan 120 man-hours over the actual total workload with the proposed BN, against 2573 manhours over the actual workload with the current estimation method, and 3040 man-hours more with the probabilistic model.

6.2

Example 2 – Sales Planning

This example explores the possibility of performing sales planning with the developed BNs. Instantiating the ‘TotalWL’ variable allows to infer what operator, maintenance intervention, and aircraft version should be targeted from a sales planning perspective to fulfil available manpower, represented by the instantiated interval.

37

Considering that the MRO has 2000 man-hours available during a given time period, instantiating the ‘TotalWL’ variable with the ]1500, 2000] workload interval (Fig. 15) indicates that the ‘2C’, ‘3C’, ‘4C’, and ‘5C’ checks, from ‘Operator09’, ‘Operator28’, and ‘Operator31’, for the ‘B1’ aircraft version, are the states that most probably fulfil the available manpower. With these results the MRO is able to direct its sales effort to these maintenance interventions, operators, and aircraft version.

6.3

Discussion

When comparing the proposed BNs with the maintenance estimation method currently used by the host MRO and with the considered probabilistic model, the first difference that can be identified is that by using data about the aircraft operator, the aircraft version, and the aircraft utilization type, more accurate workload estimations are obtained with the proposed BNs. The same information could be used with the current estimation method and with the considered probabilistic model to improve their accuracy, but this would require statistical parameters to be calculated in the former, and lognormal distributions in the latter, for all combinations of variables. On the contrary, after the definition of the BN structures, their parameters can be automatically calculated as explained in Section 5.4. The second difference is that MROs can contemplate the uncertainty of the decision-making process with the proposed BNs. For the case presented in Section 6.1.1, the MRO gains the knowledge that the ]3500, 4000] workload interval under the ‘TotalWL’ variable is the interval with the highest probability of occurrence (34,86%) for a ‘6C’ check from ‘Operator09’ and ‘B1’ version aircraft. However, the cumulative probability of such interval is only of 48,04%, meaning that there is still 51,96% probability of the workload to be higher than this value (Table 9). If the MRO establishes the policy of planning intervals with higher cumulative probabilities, such as 38

the minimum of 75% in the example, it may opt to plan the ]5000, 5500] interval instead. This type of reasoning is not possible with current approaches based on average values calculated from past data. A common heuristic used by MROs to overcome the uncertainty of the estimation is to ‘overplan’ the total workload, however, such ‘safety buffers’ may not be enough to face the actual workload, as in the case of Section 6.1.1 or, on the contrary, may tie up important manpower and other maintenance resources that could be used in planning other aircraft, as in the case of Section 6.1.2. A similar quantitative reasoning is possible with the probabilistic model, however, the graphical nature of the BNs remains a distinctive advantage over this method. The third difference is that with the proposed BNs, whichever the workload interval is selected for the ‘TotalWL’, a probability distribution is associated to each of the remainder hypothesis variables, allowing a detailed characterization of the total workload across the latter. For the case presented in Section 6.1.1, by selecting the ]3500, 4000] interval, a given set of probabilities will be obtained for the workload intervals of the work phase variables. If the ]5000, 5500] workload interval is selected instead, a different set of probabilities is obtained. Finally, as exemplified in Section 6.2, the properties of BNs allow the decisionmaker to perform sales planning in addition to capacity planning with the developed BNs. Instead of inferring what workloads are the most probable to occur given a set of evidences provided to the information variables, it is possible to determine what values of the information variables are the most probable to fulfil a given amount of available manpower, by providing evidence to a given workload interval in the ‘TotalWL’ hypothesis variable.

39

7.

Conclusion and Future Research

The objective of this paper is to explore the applicability of BNs as a BDPA tool to address the aircraft maintenance capacity planning problem, transforming data into valuable information within MROs. Given their probabilistic nature, BNs prove to be a suitable technique to address the inherent uncertainty of maintenance workload estimations and to improve the MRO capacity planning decision-making process. When compared to a maintenance estimation method currently in use based on average workloads of past maintenance checks, and to a probabilistic model based on the lognormal distribution, the use of the developed BNs as a BDPA tool presents the following advantages: (1) the use of data typically collected during the planning process such as the operator, the aircraft version, the maintenance event type, and the utilization type to increase the accuracy of the workload estimations; (2) the contemplation of the uncertainty of the decision-making process, allowing MROs to make more informed decisions about the manpower to plan given the workload estimations; (3) the detailed characterization of the estimated total workload across other workload variables such as work types, work phases, and work skills; and (4) the possibility of performing sales planning in addition to capacity planning by inferring which operators, aircraft versions, maintenance event types, and utilization types are the most likely to fulfil a given amount of available manpower. Based on real maintenance data from a Portuguese MRO, BNs were developed comprising information variables referring to typical data collected during the planning process and hypothesis variables referring to the workloads required to be estimated. Besides the selection of variables, experts from the host MRO were also involved in the definition of relationships between them, manually establishing the BNs structure. Regarding the conditional probabilities, these were automatically calculated from data. 40

Three BNs were developed to comprise the BDPA tool for capacity planning, one BN for each group of hypothesis variables: (1) considering the total workload and workload per work type; (2) considering the total workload and workload per work phase; and (3) considering the total workload and workload per work skill. The decision to develop three BNs was taken since a single BN considering all the hypothesis variables would be intractable for parameter learning and illegible for decision-making. The combined use of the established BNs in the same BDPA tool ensures the representativeness of the capacity planning problem and allows a comprehensive understanding of the expected maintenance work early in the planning process. The developed BNs were validated through the use of MAD, an error measure commonly used to evaluate forecasting methods. It is demonstrated that the MAD can be used for the validation of BNs in cases where posterior probabilities can be compared with conditional relative frequencies calculated from data. The superiority of the proposed BNs was proven when compared to BNs not considering conditional dependencies between the defined information variables. Furthermore, two examples using real maintenance data are presented in order to demonstrate the applicability of BNs as a BDPA tool and their superiority against current approaches to capacity planning. As for future research, three potential extensions to the BNs developed in this paper have been identified. First, similar models to the ones presented in this paper can be developed for costs estimation and material forecasting. Such models would allow a more comprehensive characterization of aircraft maintenance checks. Second, the BNs were developed based solely on historical data, an important limitation in planning future and unprecedented maintenance events. By only considering historical data, the proposed BNs are limited to infer maintenance events that already occurred in the past.

41

If workloads are to be estimated for unprecedented maintenance interventions, the only option left for MROs is to select the most similar maintenance event in the BNs and plan the capacity accordingly. This may lead to bias decisions since the expected increase of maintenance work over the service life of aircraft is not taken into account. In order to overcome this limitation, historical data can be used to forecast future maintenance interventions. Trends in the workloads of past events should be analyzed and workloads of future ones should be predicted. Forecasting and simulation techniques are potential tools to address this issue and such work will greatly extend the capabilities of the BDPA tool presented in this paper. Finally, Petri nets (PNs) should be explored as a potential method to model aircraft maintenance checks as dynamic systems. Similarly to BNs, PNs are directed graphs. However, instead of nodes representing random variables as in BNs, PNs feature ‘places’, which represent possible states for the elements of the system, ‘transitions’, which model how the system can change between states, and ‘tokens’, which represent a given status of the system. Through simulation tokens can be transferred throughout the network and by analyzing the duration or the number of times tokens are in a given place it is possible to determine how the system behaves. This method can be used to analyze how the execution of maintenance checks evolves by modelling, for example, the work phases as places and adding transitions between them. Such PN models could then be integrated with BNs to combine the features of both methods and, ultimately, to improve the decision-making process of MROs. These constitute important research opportunities to further enhance the applicability of BNs as a BDPA tool in capacity planning. Acknowledgments This work was supported by the Portuguese National Science Foundation (FCT) under Grant PD/BD/52345/2013. 42

References Abimbola M, Khan F (2016) Development of an integrated tool for risk analysis of drilling operations. Process Saf Environ Prot 102:421–430. doi: 10.1016/j.psep.2016.04.012 Abimbola M, Khan F, Khakzad N, Butt S (2015) Safety and risk analysis of managed pressure drilling operation using Bayesian network. Saf Sci 76:133–144. doi: 10.1016/j.ssci.2015.01.010 Afenyo M, Khan F, Veitch B, Yang M (2017) Arctic shipping accident scenario analysis using Bayesian Network approach. Ocean Eng 133:224–230. doi: 10.1016/j.oceaneng.2017.02.002 Al-Fares HK, Duffuaa SO (2009) Maintenance Forecasting and Capacity Planning. In: Ben-Daya M, Duffuaa SO, Raouf A, et al. (eds) Handbook of Maintenance Management and Engineering. Springer London, pp 157–190 Arzaghi E, Abaei MM, Abbassi R, et al (2017) Risk-based maintenance planning of subsea pipelines through fatigue crack growth monitoring. Eng Fail Anal 79:928– 939. doi: 10.1016/j.engfailanal.2017.06.003 ATA (2007) ATA MSG-3 Operator/Manufacturer Scheduled Maintenance Development, Revision 2007.1. Air Transport Association of America Inc., Washington DC, Baker KR, Trietsch D (2009) Principles of Sequencing and Scheduling. John Wiley & Sons Baksh AA, Abbassi R, Garaniya V, Khan F (2018) Marine transportation risk assessment using Bayesian Network: Application to Arctic waters. Ocean Eng 159:422–436. doi: 10.1016/j.oceaneng.2018.04.024 Budai G, Dekker R, Nicolai RP (2008) Maintenance and Production: A Review of Planning Models. In: Kobbacy KAH, Murthy DNP (eds) Complex System Maintenance Handbook. Springer London, pp 321–344 Cessna (2013) CIL-99-03 Service Letter, Revision 1. Cheng J, Cheng J, Bell DA, Liu W (1997) An Algorithm for Bayesian Belief Network Construction from Data. Proc AI STAT’97 83--90. Cooper GF, Herskovits E (1992) A Bayesian method for the induction of probabilistic networks from data. Mach Learn 9:309–347. doi: 10.1007/BF00994110 Dash D, Druzdzel MJ (1999) A Hybrid Anytime Algorithm for the Construction of Causal Models from Sparse Data. In: Proceedings of the Fifteenth Conference on Uncertainty in Artificial Intelligence. Morgan Kaufmann Publishers Inc., San Francisco, CA, USA, pp 142–149 de Melo AC V, Sanchez AJ (2008) Software maintenance project delays prediction using Bayesian Networks. Expert Syst Appl 34:908–919. doi: 10.1016/j.eswa.2006.10.040 Dempster AP, Laird NM, Rubin DB (1977) Maximum Likelihood from Incomplete Data via the EM Algorithm. J R Stat Soc Ser B 39:1–38. Dijkstra MC, Kroon LG, van Nunen JAEE, Salomon M (1991) A DSS for capacity planning of aircraft maintenance personnel. Int J Prod Econ 23:69–78. doi: 10.1016/0925-5273(91)90049-Y 43

Druzdzel MJ (1999) SMILE : Structural Modeling, Inference, and Learning Engine and GeNIe: A Development Environment for Graphical Decision-Theoretic Models. Proc Sixt Natl Conf Artif Intell 342–343. Eickemeyer SC, Borcherding T, Schäfer S, Nyhuis P (2013) Validation of data fusion as a method for forecasting the regeneration workload for complex capital goods. Prod Eng 7:131–139. doi: 10.1007/s11740-013-0444-8 Eickemeyer SC, Herde F, Irudayaraj P, Nyhuis P (2014) Decision models for capacity planning in a regeneration environment. Int J Prod Res 52:7007–7026. doi: 10.1080/00207543.2014.923122 FAA (2012) Advisory Circular 120-16F. Federal Aviation Administration, U.S. Department of Transportation Ferreiro S, Arnaiz A, Sierra B, Irigoien I (2012) Application of Bayesian networks in prognostics for a new Integrated Vehicle Health Management concept. Expert Syst Appl 39:6402–6418. doi: 10.1016/j.eswa.2011.12.027 Friedman N, Geiger D, Goldszmidt M (1997) Bayesian Network Classifiers. Mach Learn 29:131–163. doi: 10.1023/A:1007465528199 Gandomi A, Haider M (2015) Beyond the hype : Big data concepts , methods , and analytics. Int J Inf Manage 35:137–144. Haroun AE, Duffuaa SO (2009) Maintenance Organization. In: Ben-Daya M, Duffuaa SO, Raouf A, et al. (eds) Handbook of Maintenance Management and Engineering. Springer London, pp 3–15 Hazen BT, Skipper JB, Ezell JD, Boone CA (2016) Big data and predictive analytics for supply chain sustainability: A theory-driven research agenda. Comput Ind Eng 101:592–598. doi: 10.1016/j.cie.2016.06.030 Heckerman D, Shachter R (1995) Decision-Theoretic Foundations for Causal Reasoning. J Artif Intell Res 3:405–430. ICAO ICAO Circular 329 - Runway Surface Condition Assessment, Measurement and Reporting, Draft. Jensen F V., Nielsen TD (2007) Bayesian networks and decision graphs. Springer Jones B, Jenkinson I, Yang Z, Wang J (2010) The use of Bayesian network modelling for maintenance planning in a manufacturing industry. Reliab Eng Syst Saf 95:267–277. doi: 10.1016/j.ress.2009.10.007 Khakzad N, Khan F, Amyotte P (2011) Safety analysis in process facilities: Comparison of fault tree and Bayesian network approaches. Reliab Eng Syst Saf 96:925–932. doi: 10.1016/j.ress.2011.03.012 Khakzad N, Khan F, Amyotte P (2013a) Quantitative risk analysis of offshore drilling operations: A Bayesian approach. Saf Sci 57:108–117. doi: 10.1016/j.ssci.2013.01.022 Khakzad N, Khan F, Amyotte P, Cozzani V (2013b) Domino Effect Analysis Using Bayesian Networks. Risk Anal 33:292–306. doi: 10.1111/j.15396924.2012.01854.x Khakzad N, Landucci G, Cozzani V, et al (2018) Cost-effective fire protection of chemical plants against domino effects. Reliab Eng Syst Saf 169:412–421. doi: 10.1016/j.ress.2017.09.007 Khakzad N, Reniers G, Abbassi R, Khan F (2016) Vulnerability analysis of process plants subject to domino effects. Reliab Eng Syst Saf 154:127–136. doi: 10.1016/j.ress.2016.06.004 Khan B, Khan F, Veitch B, Yang M (2018) An operational risk analysis tool to analyze marine transportation in Arctic waters. Reliab Eng Syst Saf 169:485–502. doi: 10.1016/j.ress.2017.09.014 44

Kjærulff UB, Madsen AL (2008) Bayesian Networks and Influence Diagrams. Springer New York, New York, NY Kline MB (1984) Suitability of the lognormal distribution for corrective maintenance repair times. Reliab Eng 9:65–80. doi: 10.1016/0143-8174(84)90041-6 Langseth H, Portinale L (2007) Bayesian networks in reliability. Reliab Eng Syst Saf 92:92–108. doi: 10.1016/j.ress.2005.11.037 Lauritzen SL (1995) The EM algorithm for graphical association models with missing data. Comput Stat Data Anal 19:191–201. doi: 10.1016/0167-9473(93)E0056-A Li X, Zhu H, Chen G, Zhang R (2017) Optimal maintenance strategy for corroded subsea pipelines. J Loss Prev Process Ind 49:145–154. doi: 10.1016/j.jlp.2017.06.019 Marcot BG (2012) Metrics for evaluating performance and uncertainty of Bayesian network models. Ecol Modell 230:50–62. doi: 10.1016/j.ecolmodel.2012.01.013 Mula J, Poler R, García-Sabater GS, Lario FC (2006) Models for production planning under uncertainty: A review. Int J Prod Econ 103:271–285. doi: 10.1016/j.ijpe.2005.09.001 Murthy DNP, Ma L (1991) MRP with uncertainty: a review and some extensions. Int J Prod Econ 25:51–64. doi: 10.1016/0925-5273(91)90130-L Neil M, Marquez D (2012) Availability modelling of repairable systems using Bayesian networks. Eng Appl Artif Intell 25:698–704. doi: 10.1016/j.engappai.2010.06.003 Pearl J (1988) Probabilistic reasoning in intelligent systems : networks of plausible inference. Morgan Kaufmann Publishers Pitchforth J, Mengersen K (2013) A proposed validation framework for expert elicited Bayesian Networks. Expert Syst Appl 40:162–167. doi: 10.1016/j.eswa.2012.07.026 Regattieri A, Gamberi M, Gamberini R, Manzini R (2005) Managing lumpy demand for aircraft spare parts. J Air Transp Manag 11:426–431. doi: 10.1016/j.jairtraman.2005.06.003 Samaranayake P, Kiridena S (2012) Aircraft maintenance planning and scheduling: an integrated framework. J Qual Maint Eng 18:432–453. doi: 10.1108/13552511211281598 Sotiralis P, Ventikos NP, Hamann R, et al (2016) Incorporation of human factors into ship collision risk models focusing on human centred design aspects. Reliab Eng Syst Saf 156:210–227. doi: 10.1016/j.ress.2016.08.007 Spirtes P, Glymour C, Scheines R (1993) Causation, Prediction, and Search. Springer New York, New York, NY Weber P, Medina-Oliva G, Simon C, Iung B (2012) Overview on Bayesian networks applications for dependability, risk analysis and maintenance areas. Eng Appl Artif Intell 25:671–682. doi: 10.1016/j.engappai.2010.06.002 Yang Y, Khan F, Thodi P, Abbassi R (2017) Corrosion induced failure analysis of subsea pipelines. Reliab Eng Syst Saf 159:214–222. doi: 10.1016/j.ress.2016.11.014

45

Fig. 1. MRO maintenance planning process.

Fig. 2. BN example (Langseth and Portinale 2007).

Fig. 3. The three types of connections in BNs: (a) serial; (b) diverging; and (c) converging - adapted from Langseth and Portinale (2007). Fig. 4. Maintenance

events classification system.

Fig. 5. Average maintenance workloads for ‘C’ checks.

Fig. 6. BN modelling process.

Fig. 7.

Fig. 8.

Relationships between information variables

Relationships between hypothesis variables (work phases BN)

Fig. 9. Relationships between information and hypothesis variables (work phases BN)

Fig. 10. BN validation process.

Fig. 11. Results

for the ‘TotalWL’ variable after the evidences ‘Operator09’ and ‘4C’ are provided.

46

Fig. 12.

Inference of the ‘EventType’, ‘AircraftVersion’, and ‘Utilization’ variables given evidence to the ‘Operator’ variable.

Fig. 13.

‘TotalWL’ probabilities given evidences to ‘Operator’, ‘EventType’, and ‘AircraftVersion’ variables.

Fig. 14.

Workload per work phase probabilities given evidence to ‘TotalWL’ variable.

Fig. 15. Inference of the ‘Operator’, ‘EventType’, ‘AircraftVersion’, and ‘Utilization’

variables given evidence to the ‘TotalWL’ variable.

Table 1. Collected data regarding the identification of the maintenance projects. No. 1 2 3 4 5 6 7

Heading Project Number Version Operator Event Type Start Date End Date Utilization

Description The project identification number within the MRO. The aircraft version. The aircraft operator identification. The performed maintenance intervention. The project start date. The project end date. The aircraft utilization type.

Table 2. Collected data regarding the maintenance tasks performed in the projects. No. 1 2 3 4 5 6 7

Heading Project Number Task Task Type Work Zone Work Phase Main Resource Total Hours

Description The project identification number within the MRO. The maintenance task identification. The task work type. The aircraft zone in which the task was performed. The work phase in which the task was performed. The technical skill which performed the task. The total time, in man-hours, spent in accomplishing the task.

47

Table 3. Considered work types. Work Type ADs/SBs Scheduled Unscheduled Nonconformity Other Scheduled Other Unscheduled

Description Tasks referring to ADs and SBs. Tasks referring to scheduled maintenance actions, e.g. inspections, specified by the manufacturer and enforced by the maintenance program. Tasks referring to unscheduled maintenance actions, e.g. repairs, resulting from the scheduled tasks. Tasks referring to quality issues resulting from the MRO work. Tasks referring to maintenance actions specified by the manufacturer, but not enforced by the maintenance program. Tasks referring to unscheduled maintenance actions not resulting from the scheduled tasks.

Variable ADsSBsWL ScheduledWL UnscheduledWL NonconformityWL OtherScheduledWL OtherUnscheduledWL

Table 4. Considered work phases. Work Phase Reception Accesses Opening/ Disassembly Inspection/ Evaluation Repairs/ Modifications Assembly/ Accesses Closing Certification and Delivery

Description Tasks referring to the aircraft reception such as log book analysis, initial testing, fuel removal, exterior washing, etc. Tasks referring to the opening of panels and equipment removal.

Variable P1_ReceptionWL

Tasks referring to the inspection of aircraft engines, systems, and components. Tasks referring to the correction of failures found during the maintenance intervention. Tasks referring to the reinstallation of removed equipment and closing of panels.

P3_InspectionWL

Tasks referring to the operational and functional testing of aircraft engines, systems, and components, and subsequent delivery to the client.

P6_DeliveryWL

P2_AccOpeningWL

P4_RepairsWL P5_AccClosingWL

Table 5. Considered work skills Skill Systems Structures Avionics Painting Generic Other

Description Tasks referring to the aircraft mechanical systems. Tasks referring to the aircraft structures. Tasks referring to the aircraft avionics. Tasks referring to the aircraft stripping and painting. Tasks referring to generic work, e.g. panels opening. Tasks referring to skills other than the above.

Variable SystemsWL StructuresWL AvionicsWL PaintingWL GenericWL OtherWL

48

Table 6. Results for the ‘TotalWL’ variable after evidences ‘Operator09’ and ‘4C’ check are provided. TotalWL

Fi

Bpi

Bqi

[0, 0]

0,00%

0,01%

0,05%

]0, 500]

0,00%

1,39%

5,89%

]500, 1000]

0,00%

0,36%

1,51%

]1000, 1500]

0,00%

1,04%

4,43%

]1500, 2000]

5,88%

6,75%

8,54%

]2000, 2500]

17,65%

16,88%

19,57%

]2500, 3000]

58,82%

54,38%

36,35%

]3000, 3500]

11,76%

11,30%

9,01%

]3500, 4000]

0,00%

1,39%

5,89%

]4000, 4500]

5,88%

5,07%

2,71%

]4500, 5000]

0,00%

0,36%

1,51%

]5000, 5500]

0,00%

0,87%

3,70%

]5500, 6000]

0,00%

0,18%

0,78%

]6000, 6500]

0,00%

0,01%

0,05%

Table 7. MAD evolution (workload per work phase BN).

Variable

Evidence 1 – ‘Operator’

Evidence 2 – ‘EventType’

Evidence 3 – ‘AircraftVersion’

Evidence 4 – ‘Utilization’

MADp

MADq

MADp

MADq

MADp

MADq

MADp

MADq

Total WL

0,56%

0,19%

0,93%

4,06%

0,66%

2,22%

0,66%

0,66%

P1_ReceptionWL

0,30%

0,23%

0,65%

1,01%

0,84%

0,70%

0,84%

0,84%

P2_AccOpeningWL

1,13%

0,11%

6,18%

12,77%

3,66%

6,61%

3,66%

3,66%

P3_InspectionWL

0,54%

0,12%

2,44%

4,67%

1,95%

2,99%

1,95%

1,95%

P4_RepairsWL

0,28%

0,09%

0,87%

1,68%

0,80%

1,30%

0,80%

0,80%

P5_AccClosingWL

0,60%

0,24%

2,32%

4,88%

2,15%

3,50%

2,15%

2,15%

P6_DeliveryWL

0,65%

0,26%

1,57%

3,12%

1,17%

1,90%

1,17%

1,17%

Table 8. Maximum error values (workload per work phase BN).

Variable

Evidence 1 – ‘Operator’ Max Max |Ep| |Eq|

Evidence 2 – ‘EventType’ Max Max |Ep| |Eq|

Evidence 3 – ‘AircraftVersion’ Max Max |Ep| |Eq|

Evidence 4 – ‘Utilization’ Max Max |Ep| |Eq|

Total WL

0,37%

0,66%

0,02%

20,70%

0,00%

11,33%

0,00%

0,00%

P1_ReceptionWL

0,49%

2,12%

0,20%

2,20%

1,32%

0,88%

1,32%

0,70%

49

P2_AccOpeningWL

0,15%

2,32%

4,95%

32,37%

2,58%

12,26%

2,58%

1,12%

P3_InspectionWL

0,61%

4,17%

5,08%

19,54%

4,66%

7,61%

4,66%

2,30%

P4_RepairsWL

0,83%

3,05%

1,28%

6,65%

2,31%

4,52%

2,31%

1,17%

P5_AccClosingWL

2,03%

11,97%

2,88%

24,00%

3,41%

16,25%

3,41%

1,47%

P6_DeliveryWL

0,48%

1,47%

0,49%

2,79%

0,00%

1,09%

0,00%

0,00%

Table 9. Results for the ‘TotalWL’ variable after evidences ‘Operator09’, ‘6C’ check, and ‘B1’ aircraft version are provided. TotalWL

Probability

Cumulative Probability

[0, 0]

0,01%

0,01%

]0, 500]

1,53%

1,54%

]500, 1000]

0,39%

1,94%

]1000, 1500]

1,15%

3,09%

]1500, 2000]

1,53%

4,62%

]2000, 2500]

3,61%

8,23%

]2500, 3000]

3,61%

11,84%

]3000, 3500]

1,34%

13,18%

]3500, 4000]

34,86%

48,04%

]4000, 4500]

0,39%

48,44%

]4500, 5000]

17,06%

65,49%

]5000, 5500]

17,63%

83,12%

]5500, 6000]

16,87%

99,99%

]6000, 6500]

0,01%

100,00%

50

Table 10. Comparison between the BN inferred workload intervals, current estimation method, a probabilistic model, and actual workloads for a ‘6C’ check from ‘Operator09’ and ‘B1’ aircraft version.

Current Probabilistic Actual Method Model Workload (man-hours) (man-hours) (man-hours)

BN Δ (manhours)

Current Probabilistic Method Δ Model Δ (man-hours) (man-hours)

Variable

BN (manhours)

TotalWL

]5000, 5500]

3953

4420

5008

492

-1055

-588

P1_ReceptionWL

]0, 100]

41

46

40

60

1

6

P2_AccOpeningWL

]300, 400]

285

319

302

98

-17

17

P3_InspectionWL

]1100, 1200]

857

958

987

213

-130

-29

P4_RepairsWL

]3100, 3200]

2352

2629

3219

-19

-867

-590

P5_AccClosingWL

]400, 500]

387

433

437

63

-50

-4

P6_DeliveryWL

]0, 100]

31

35

23

77

8

12

Table 11. Results for the ‘TotalWL’ variable after the evidences ‘Operator31’, ‘6C’check, and ‘B1’ aircraft version are provided. TotalWL

Probability

Cumulative Probability

[0, 0]

0,03%

0,03%

]0, 500]

6,63%

6,65%

]500, 1000]

1,19%

7,85%

]1000, 1500]

68,62%

76,46%

]1500, 2000]

20,57%

97,03%

]2000, 2500]

0,80%

97,84%

]2500, 3000]

1,58%

99,42%

]3000, 3500]

0,42%

99,83%

]3500, 4000]

0,03%

99,86%

]4000, 4500]

0,03%

99,89%

]4500, 5000]

0,03%

99,92%

]5000, 5500]

0,03%

99,94%

]5500, 6000]

0,03%

99,97%

]6000, 6500]

0,03%

100,00%

Table 12. Comparison between the BN inferred workload intervals, current estimation method, a probabilistic model, and actual workloads for a ‘6C’ check from ‘Operator31’ and ‘B1’ aircraft version.

51

Current Probabilistic Actual Method Model Workload (man-hours) (man-hours) (man-hours)

BN Δ (manhours)

Current Probabilistic Method Δ Model Δ (man-hours) (man-hours)

Variable

BN (manhours)

TotalWL

]1000, 1500]

3953

4420

1380

120

2573

3040

P1_ReceptionWL

]0, 100]

41

46

31

69

10

15

P2_AccOpeningWL

]100, 200]

285

319

181

19

104

138

P3_InspectionWL

]200, 300]

857

958

189

111

668

769

P4_RepairsWL

]700, 800]

2352

2629

748

52

1604

1881

P5_AccClosingWL

]100, 200]

387

433

192

8

195

241

P6_DeliveryWL

]0, 100]

31

35

39

61

-8

-4

52