Factors that affect the improvement of demand forecast accuracy through point-of-sale reporting

Accepted Manuscript

Factors that Affect the Improvement of Demand Forecast Accuracy through Point-of-Sale Reporting Charles A. Wood , Kathy Hartzel PII: DOI: Reference:

S0377-2217(16)30972-9 10.1016/j.ejor.2016.11.047 EOR 14126

To appear in:

European Journal of Operational Research

Received date: Revised date: Accepted date:

4 September 2014 1 November 2016 30 November 2016

Please cite this article as: Charles A. Wood , Kathy Hartzel , Factors that Affect the Improvement of Demand Forecast Accuracy through Point-of-Sale Reporting, European Journal of Operational Research (2016), doi: 10.1016/j.ejor.2016.11.047

This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.

ACCEPTED MANUSCRIPT

HIGHLIGHTS We compare point-of-sale and order history forecasts from a major food manufacturer. We examine 60,651 orders using hierarchical linear modeling and multinomial logit. We find that point-of-sale forecast improve most with low frequency items. We find that point-of-sale forecast improve most with low order variance. We find that point-of-sale forecast improve most with moderate order quantities.

AC

CE

PT

ED

M

AN US

CR IP T

    

ACCEPTED MANUSCRIPT

Factors that Affect the Improvement of Demand Forecast Accuracy through Point-of-Sale Reporting

AN US

CR IP T

ABSTRACT. Recent research has examined the use of real-time, shared point-of-sale (POS) data in forecasting. Although initial research posited that POS data can improve forecast accuracy, recent research has called some of these findings into question. We identify item order quantity, item order variability, and item order frequency of orders as specific factors that can affect the degree of improvement in POS demand forecasting accuracy when compared to order history-based techniques. Using a hierarchical linear model, we examine 60,651 orders for hundreds of items across 25 different distribution centers. We find a 11.2% overall improvement in using real-time, shared POS data in demand forecasting over order history-based forecasting. However, we find a curvilinear relationship between these improvements and both the order quantity and the item order variability. Additionally, we find that POS based forecasting improvements are greatest (1) when items are not frequently ordered, (2) when there is low variance in the number of distribution center ordering an item each week, and (3) when order quantities are neither relatively high, nor relatively low.

AC

CE

PT

ED

Contact Author: Name: Charles A. Wood 600 Forbes Ave Pittsburgh, PA 15282 Duquesne University Office: 412-396-1775 Fax: 412-396-4764 Email: [email protected]

M

KEYWORDS. Supply Chain Management, Shared Information, Demand Forecasting, Point-of-Sale Data, Order History

ACCEPTED MANUSCRIPT

1. INTRODUCTION Inter-organizational information sharing contributes to the evolution of Supply Chain Management’s (SCM) role from passive cost control and operational efficiency to a mechanism for achieving sustainable competitive advantage (Tracey, Lim, & Vonderembse, 2005).

Information

technology (IT) is vital in improving supply chain performance, both within the organization and across the supply chain, in that IT continues to improve cost and service performance. IT-supported

CR IP T

information sharing options range from loosely coupled channels where data is simply transmitted from the boundary of one organization through the boundary of another to highly integrated supply chain management systems. Many researchers have described how electronic information transfer (EIT) regulates the exchange of information between buyers and suppliers. EIT has been shown to increase the performance of the supply chain by decreasing delivery time and increasing service levels,

AN US

thus increasing profits (e.g., Gal-Or, Geylani, & Dukes, 2008; Rai, Patnayakuni, & Seth, 2006).

Borrowing from previous research (e.g., Lyu, Ding & Chen, 2010) and from current observations, a typical supply chain model handles information flows tier by tier, with the information being aggregated or disaggregated as the information flows up or downstream, as shown in Figure 1. Figure 2 illustrates a supply chain using POS data to communicate consumer demand where information from

M

individual retail sites is transmitted directly to the manufacturer in disaggregated form rather than aggregated at the retail chain’s distribution centers.1 Essentially, real-time POS data can be visible to

ED

the manufacturer from each retail location, thus providing pre-order demand signals. The availability of the real-time information allows more time for the manufacturer to improve production scheduling. It should be noted that the demand signal comes from the retail stores and the observed orders are

PT

released by the distribution centers.

Prior to technological advances that enable the sharing of POS data, a manufacturer’s order

CE

forecast was, by necessity, based on order history and expert market analysis. Now, many firms, such as Wal-Mart (Fulcher, 2009), Del Monte, Unilever, and Procter & Gamble (Folinas & Rabi, 2012)

AC

have implemented advanced forecasting methods including POS data. Their insight illustrates how POS signals real-time demand, but our interest resides in how this information impacts the accuracy of the manufacturer’s forecast of demand by replacing or supplementing forecast based on order history.

1

Figures 1 and 2 follow the line notation from Lyu, et al. (2010), where a dotted line represents the flow of information and the solid line represents the flow of physical goods

CR IP T

ACCEPTED MANUSCRIPT

Figure 1. Order History Model where Information Is Captured and Rebroadcasted

Figure 2. POS Model where Information Is Directly Transmitted

AN US

Folinas and Rabi (2012) describe how increased forecast accuracy can benefit the firm in the two critical areas of in-stock availability and working capital: 

In-Stock Availability is a critical success factor for many manufacturers and retailers as it measures how much product is available at any given time to ship to retailers. Poor forecasting can cause orders to go unfulfilled, losing sales as customers try alternative brands, and losing

M

market share as customers start to rely on alternative products or stores. Safety stock, where companies purposely stock more than the forecasted demand, acts to ensure in-stock 

ED

availability, but at the expense of consistently higher inventory costs. By contrast, it is vital to consider Working Capital when planning the efficient operations of a

PT

firm, especially when meeting current expenses. Working capital (cash, liquid assets, and inventory) is required for the firm to function and cannot be used for investments or other long-

CE

term expenses. Baker (2007) describes that, at any given time, working capital can account for greater than 24% of total logistics cost; this capital could be used for competitive advantage, but is instead tied up in overstocked inventory.

AC

In-stock availability and working capital objectives compete against each other. If a demand

forecasting methodology tends to over-predict demand leading to an increase in inventory levels, then there is an increased probability of in-stock situations, thus allowing every interested customer to purchase the item. However, over-prediction requires more working capital to be tied up in inventory. Conversely, if a demand forecasting methodology tends to under predict demand leading to a decrease in inventory levels, then less of the firm’s capital is tied up in inventory. This frees the firm to profit from investments using the additional funds, but also increases the probability of an out-of-stock situation which can result in lost sales and lost customers.

ACCEPTED MANUSCRIPT

An information system (IS) that integrates data and processes between supply chain partners can provide better quality information as well as reduced communication and coordination expenses in the production and distribution of consumer goods. Well-executed supply chain practices (planning, JIT production and delivery) and information sharing are both necessary to attain first-rate supply chain performance (Zhou & Benton, 2007). However, there has been little or no coverage in existing research of empirical studies that examines item-specific characteristics that allow demand-sensing

CR IP T

technologies to over-perform or under-perform against order-history methods. This includes how these technologies not only impact overall demand forecasts, but also impact the accuracy of predictions in both over-ordering and under-ordering within the supply chain. As implementation of POS demand forecasting can be expensive, it behoves us to examine the types of products that would benefit most from POS demand forecasting. We pose the following research questions:

What item-specific order patterns improve the likelihood of maintaining inventory levels when

AN US



using POS forecasting? 

What item-specific order patterns improve the accuracy of POS forecasting improvements when compared to order-history forecasting techniques?

Recent research questions the efficacy of exclusively using POS data to predict demand. Some

M

results have shown no significant improvement when using POS and other studies have found POS forecasts under-performing order-history forecasts Researchers have been pointing out the need for

ED

more investigation in this area (e.g., Williams, Waller, Ahire & Ferrier, 2014; Williams & Waller, 2011). This paper contributes to this ongoing investigation through an empirical examination of a large dataset obtained from a large international food seller that includes 60,651 orders for hundreds of items

PT

across 25 different distribution centers. Our findings include that (1) POS did improve the accuracy of order forecasts for this manufacturer, (2) the benefits of using the POS forecasting are greater when the

CE

forecasted item is not a big seller, (3) POS improvement is reduced when there is higher variability in order quantity, and (4) the benefit of using POS forecasting is greater when the forecast is an

AC

overestimate of the observed order quantity than when forecast is an underestimate. These findings are important because they may help guide future research directions, and also could aid managers in improving their own forecasts.

2. LITERATURE REVIEW This work builds upon insights found in both SCM and IS research, specifically how IS can facilitate information sharing.

ACCEPTED MANUSCRIPT

2.1. Information Sharing Implementing

inter-organizational

information

systems

that

provide

CRP

(continuous

replenishment processes) functionality, which provide the opportunity for real-time sharing of supply and demand information, can enhance supply chain performance (Simatupang & Sridharan, 2005). In supply chains, shared information, management and strategy have been operationalized through programs such as quick response (QR) (Setaputra, Yue & Yao, 2010), vendor managed inventory

CR IP T

(VMI) (Lee & Whang, 2000b), collaborative performance systems (CPS) (Hammer, 2001), and collaborative planning, forecasting, and replenishment (CPFR) (Aviv, 2001).

Sharing data in the supply chain refers to sharing private information. Private data creates information asymmetry. Simatugpang and Sridharan (2002) described this concept in this way:

AN US

‘Asymmetric information refers to different parties having different states of private information about demand conditions, products, and the chain operations. The problem of asymmetric information arises because participating firms generally lack the knowledge required about each other's plans and intentions to adequately harmonize their services and activities. Supply chain members often do not wish to share their private information completely and faithfully with all other members due to the economic value of that information (actual or perceived). As a result, the supply chain suffers from suboptimal decisions and opportunistic behavior’ (p. 17).

Supply chain management systems reduce information asymmetry by providing earlier, shared

M

signals of changing demand, thus creating an integrated view of supply, demand, and delivery information through shared, inter-organizational systems (Sahin & Robinson, 2002). Supply chain

ED

integration across firms must be preceded by the integration of IT infrastructures. Integrated IT infrastructures allow:

PT

‘…firms to unbundle information flows from physical flows, and to share information with their supply chain partners to create information-based approaches for superior demand planning, for the staging and movement of physical products, and for streamlining voluminous and complex financial work processes.’ (Rai, et.al., 2006, p. 225)

CE

Integrated systems support alternative communication patterns among supply chain members. Lyu, et al. (2010) present a number of collaborative replenishment mechanism models, including two that The first model is referred to as the As-Is Model of

AC

are particularly relevant to this study.

manufacturer retailer interaction where information is intercepted and the rebroadcasted at every node in the supply chain. Thus, in this As-Is Model, the individual retail stores communicate their demand to a centralized retail unit which aggregates the demand and places an order with the manufacturer. The order is then sent to the retailer’s distribution center who disaggregates the order for shipment to the individual stores. The disaggregated order is the order data for use in order-history forecasting. The second model is referred to as the To-Be Model which is a more collaborative replenishment model where the retail store demand is communicated directly to the manufacturer who aggregates the

ACCEPTED MANUSCRIPT

demand in anticipation of the order or uses the demand to create an order in a vendor managed inventory arrangement. That order is then sent to the manufacturer’s or retailer’s distribution center, disaggregated, and forwarded to the individual retail stores. This second model provides the individual store-level POS data used in POS forecasting. The expectation of reduced supplier inventory and cost savings associated with using interorganizational information systems have been modeled analytically (Lee, So & Tang, 2000a; Wu &

CR IP T

Cheng, 2008). These analytical models assume the manufacturer bares the cost of ensuring order fulfillment, and thus the retailer has little incentive to share information. Inter-firm partnerships create value through both knowledge sharing and processes coupling where the processes in both firms work together to increase performance (Saraf, Langdon & Gosain, 2007). Relationship-specific investment by suppliers provides sharable cost improvements and data accuracy while increasing the switching

AN US

costs of the buyer which enhances the likelihood of attaining a preferred supplier status. In another survey, 110 supply chain and logistics managers reported that supply chain process integration lead to a competitive advantage in operations by reducing delays and process improvements (Rai, et al., 2006). Thus, the value recognized by suppliers when investing in inter-firm systems is both strategic and operational (Sanders, 2007). more volatile.

M

Investment in coordination technology has the potential for greater benefit when demand volume is Technology’s provision of more and timelier information reduces uncertainty in

ED

demand estimates. A survey of Korean buyers, which were responsibility for supplier relationships, assessed how demand uncertainty was correlated with the degree of information transfer. The type of shared information they observed included production schedules, delivery schedules, and finished

PT

goods inventory levels. The data indicated that demand uncertainty was positively associated with

CE

extent of information electronically transferred between supply chain partners (Kim, et al., 2005-6).

2.2. Bullwhip Effect

AC

Demand uncertainty is a major contributor to the bullwhip effect phenomenon. The bullwhip effect describes how forecasting errors are magnified as information is passed upstream in the supply chain. The problems resulting from the bullwhip effect include difficulty in production planning, missed delivery dates, costs associated with expediting and returning orders, compromised customer service, and loss of goodwill (Lee, Padmanabhan & Whang, 1997). Given small downstream errors in predicting retail demand can result in large errors in production planning, adopting a better method for predicting downstream demand could provide great value. However, the degree of value that POS-

ACCEPTED MANUSCRIPT

based forecasting provides in mitigating demand forecast error is circumstantial. For example, in Williams and Waller’s (2010) rfocused on the level of bullwhip, they found that: ‘… as the level of bullwhip increases, the probability that POS data will be a better predictor than order history of future orders decreases. Further, we found that as the level of bullwhip increases the magnitude of POS improvement decreases’ (Williams & Waller, 2010, p. 246).

Thus, POS-based forecasts should provide the greatest benefits when the level of bullwhip is attenuated. Cachon, Randall, and Schmidt (2007) found that the bullwhip effect varies in magnitude

production smoothing which mitigates the bullwhip effect.

CR IP T

by industry; where demand variation is predictable. Their research indicates that there tends to be

In this study, we extend this research stream by focusing on other factors that may have an impact on the performance of POS versus order-history demand forecasts, specifically how the size of order

forecasts over order-history demand forecasts.

2.3. Order-History and POS Forecasting

AN US

quantity, variability in order size, and the frequency of orders affect the improvement of POS demand

Empirical work validating the benefits of sharing POS data is limited. From a theoretical stance, the literature suggests that providing access to POS data could improve forecast accuracy (Chen &

M

Lee, 2009; Kiely, 1999, Lee et al., 2000a; Lee, Padmanabhan and Whang, 2004). The limited number of studies on the effect of using POS data, rather than order-history data, has presented mixed results.

ED

Using order data from the grocery industry, Williams and Waller developed two sets of order forecasts comparing the relative accuracy of competing techniques. One set of forecasts used point-ofsale (POS) data and the other set used order-history data. They found that POS data was more

PT

frequently a better predictor of order quantity, but not always. Factors positively associated with POSbased forecasts were higher sales rates and longer forecast horizons. Factors positively associated with

CE

order-based forecasts were high bullwhip and high non-turn volume. Williams and Waller (2011) found that the level of aggregation of the forecast mediated the positive impact of using POS data.

AC

When estimating the shipping quantity, using POS data increased accuracy. However, POS data did not help with estimates of overall retailer demand, and at times it underperformed using order-history data. Trapero, Kourentzes, and Fildes (2012) also examined data from the grocery industry, and found an 8% reduction in the mean absolute percentage error (MAPE) when the POS data was included. Raghunathan (2001) compared simulated order-history and POS forecast data and found as the number of periods of order-history data increased in his model, the value of using POS data to forecast decreased until it became negligible. Using an experimental simulation, Steckel, Gupta and Banerji (2004) found that POS data’s impact on forecast accuracy depended on the demand pattern. When the

ACCEPTED MANUSCRIPT

demand increased at a constant rate, POS forecasts often showed more accuracy over traditional forecast methods. However when the demand pattern was S-shaped, POS data often led to less accurate forecasts. It is interesting to note that the Raghunathan (2001) and Steckel, et al. (2004) papers both presented mathematical models that simulated multiple period forecasts under various conditions. It is the modeling studies that framed POS forecasting in the most negative light. The studies that both used

CR IP T

real manufacturer and retailer data and also real manufacturer forecasts were the ones that found the most value in using POS data in forecasting. The work by Williams and Waller (2010, 2011) was based on grocery data, and the item-level order characteristics may have be more heterogeneous than the data found in the simulations. More research is necessary to understand how various combinations of item-level data affect the various forecasting models.

AN US

3. DATA, EMPIRICAL MODELS AND HYPOTHESES

Two compelling reasons to implement POS demand forecasting are that better forecasts can (a) improve the probability that forecasts are sufficient to meet or exceed the eventual order, and (b) reduce inventory-related working-capital. We use a hierarchical linear model to compare the accuracy of POS forecasts when compared to forecasts made using order-history.

We also examine the

M

probability that a forecast maintains or increases inventory levels by employing a multinomial probit model to compare POS forecasting to order-history forecasts. Figure 3 shows a conceptual model that

ED

clearly delineates the proposed effects. While related, the two dependent variables, order accuracy (with higher accuracy leading to less working capital investment) and the probability that a POS order

PT

maintains or increases inventory levels, are not identical in that it is possible that overestimated demand with order-history estimation will cause a low forecast accuracy, but will result in a higher

CE

probability of a forecast meeting or exceeding the order. The remainder of this section examines the data provided for this study by a multi-billion dollar food retailer, and then describes the empirical

AC

models and hypotheses we use to examine the effect of POS forecasting on forecast accuracy.

ACCEPTED MANUSCRIPT

Dependent Variables

Order Quantity

Item Order Variability

Control Variables Random and Fixed Effects Item Effects

Increase in Order Accuracy when POS Forecasting Is Compared to Order History Forecasting

AN US

Distribution Center Effects

CR IP T

Increase in Probability of Forecast Meeting or Exceeding Observed Order when POS Forecasting Is Compared to Order History Forecasting

Item Order Frequency

Figure 3. Conceptual model Describing Hypothesized Relationships

3.1. Data

Our data consists of 63,605 observations of orders at a prepackaged food provider whose annual

M

revenues exceed billions of dollars. The orders in this data set involve 494 items delivered to 25 distribution centers from March 31, 2010 until January 31, 2011, categorized into 36 weekly segments.

ED

The orders were placed by multiple distribution centers that service different regions of the country. For each order, we record the distribution center, the date of the order, the item that was ordered,

PT

the forecast made by using order-history data utilizing an exponential smoothing methodology, the forecast made by using POS data utilizing a neural network methodology to forecast demand, and the

CE

observed quantity ordered by the retailer. In this research, we are most interested in which factors are related to the improvement of the POS forecast over the order-history forecast.

AC

3.1.1. Forecasting Methods Examined in This Research For the order-history forecasts, the manufacturer utilized a popular third-party software and

technique to determine the best forecasting measure at the item level, using either single exponential smoothing or double exponential smoothing that adjusts for trends (Holt 1957; Winters 1960). Table 1

ACCEPTED MANUSCRIPT

shows the explanatory variables used in this research, which corresponded to the order-based SES forecasts.2 Table 1. Description of Explanatory Variables

Description

Capacitywid

ItemOrderFrequencyi ItemOrderVariabilityid

AvgOrderSized WeeklyActivityw

This is the log quantity ordered by the customer’s distribution center d for item i in week w. This variable is scaled to thousands of items ordered, and is centered to facilitate the examination of non-linear relationships. This is the log of the quantity promised by the manufacturer to the retailer as a percent of the observed order, used as a control variable to indicate the capability of the manufacturer to meet the demand. This is how many orders that item i receives throughout the period of this study across all distribution centers. This is the log of the variability in the amount of weekly orders made by distribution center d for item i. This variable is centered to facilitate the examination of non-linear relationships. This is the average number of orders processed by the distribution center on a per order basis, used as a control variable. This is the quantity ordered in week w across all items and distribution centers. This variable is scaled to thousands of items ordered.

CR IP T

OrderQuantitywid

AN US

Variable

To establish a baseline to assess the company’s forecasts, we calculate a random walk forecast, where future week’s order is best predicted using the prior week’s orders:

M

RandomWalkForecast ( w1)id  Orderwid  id  ewid

i and distribution center d.

ED

Where id is an arbitrary best-fitting drift coefficient and ewid is the random error for week w for item

For another benchmark, we also computed a single exponential smoothing (SES) forecast

PT

where expected future orders are set to a smoothed value which is calculated using a geometrically weighted moving-average filter that incorporates a smoothing factor ():

SESForecas t ( w1)id   Orderwid   1    S0

CE

W

Following the methodology of the company in this study, we compute optimal starting values (S0)

AC

using an average of the three previous weeks’ order quantities, and we calculate the optimal smoothing parameter () for each item/distribution center combination using two out-of-sample observations for a

2

Trapero, Kourentzes, and Fildes (2012) do an excellent job in clearly describing single exponential smoothing. They also describe how neural network forecasting techniques involve a multilayer perceptron, which is a specific type of feedforward neural network model that maps sets of input data onto a set of appropriate outputs and assigned weights within each layer to various observed factors to optimally forecast future orders, and they compare these techniques to other demand forecasting methodologies. For the interested reader, they provide formal models of these techniques.

ACCEPTED MANUSCRIPT

rolling origin calculation that adjusts for the inflated standard error of the forecasts, as is defined in the literature (Fildes and Petropoulos 2015, Tashman 2000). In addition to the explanatory variables shown in Table 1, we also include a dependent variable that measures the improvement of POS forecasts compared to order-history forecasts: specifically the ratio of the error of POS forecasting compared to the error of order-history forecasts, with the mean calculated by taking the log-average of the ratio:3

SESForecas t wid  .0001

 ActualDema nd wid  .0001

RandomWalkErrorwid 

1

RandomWalkForecast wid  .0001 1  ActualDema nd wid  .0001

M

SESErrorwid 

OrderHistoryForecast wid  .0001 1  ActualDema nd wid  .0001

AN US

OrderHistoryErrorwid 

(1)

CR IP T

 OrderHistoryErrorwid  .0001   % POSI mprovementwid  ln  POSErrorwid  .0001   where: POSForecast wid  .0001 POSErrorwid  1  ActualDema nd wid  .0001

ED

3.1.2. Handling of Influential Observations In this research, we used the logarithm of all ratios for our calculations so that the weight of Also, it is possible for several demand

PT

influential observations is reduced (Wooldridge 2012).

forecasting algorithms to return an estimate for a negative forecast, so records with a negative order-

CE

history forecast or a negative POS forecast were removed from the analysis. In addition, Davydenko and Fildes (2013) describe how outliers in forecast errors should not be ignored when examining forecast accuracy because the estimates will be biased toward the influential points, and that forecast

AC

accuracy estimates can be significantly distorted by just an influential observation. In keeping with their research, we first removed observations whose POSErrorwid or OrderHistoryErrorwid, when

3

The minimum and maximum of %POSImprovement is misleading, in that it is a ratio where the denominator can be zero. As such, .0001 has been added to the numerator and denominator of all variables in the calculation. Thus, as a factor of zero, the minimum can be very small and the maximum can be very large. The log-average and log-standard deviation of %POSImprovement is used for the descriptive statistics in Table 2. Also, please note that the closer an error is to zero, the better that measurement. So the %POSImprovement measurement is the log measurement of how much closer POSErrorwid is to zero in comparison to OrderHistoryErrorwid.

ACCEPTED MANUSCRIPT

graphed, appeared to be far away from most observations. We then trimmed the data so that any observation whose accuracy fell more than two standard deviations away from the mean accuracy was removed from the data. This trimming resulted in a dataset containing 60,651 observations.4 In order to reduce the bias of comparing the company’s inventory forecasts to established forecasting benchmarks, we calculated demand forecasts using SES and random walk Forecasting, and we repeated this process to remove outliers separately for SESErrorwid and for RandomWalkErrorwid.

CR IP T

Because we were forced to use some of our data to perform the calculation for these two forecast measurements, rather than relying on the company to provide the observation with the forecasts, our trimmed dataset was somewhat smaller, resulting in 56,930 and 56,811 observations for these forecasting methodologies, respectively. Table 2 shows the descriptive statistics for the trimmed dataset.

Mean 11.2% 70.5% 77.6% 139.7% 76.2% 1006 99.8% 6833 201 2,276 2,126

Standard Deviation 1.393 2.083 2.277 3.431 7.791 3,389 0.009 210,144 49 4,952 133

Minimum -12.267 0 0 0 0 1 0% 0 8 63 1,903

Maximum 11.017 94.428 86.443 197.68 740.92 144,458 850% 9,015,612 350 70,142 2,321

CE

PT

ED

%POSImprovementwid POSErrorwid OrderHistoryErrorwid SESErrorwid RandomWalkErrorwid7 OrderQuantitywid Capacitywid ItemOrderVariabilityid ItemOrderFrequencyi AvgOrderSized WeeklyActivityw

M

Sample Size 60,651 60,651 60,651 61,356 60,531 60,651 60,651 60,651 60,651 60,651 60,651

AN US

Table 2. Descriptive Statistics for Trimmed Dataset Used in This Study5, 6

Data is at the observation level, and not at the sku level. The non-trimmed dataset performs statistically similar as the trimmed dataset. We believe that the trimmed dataset defines a more central tendency, and with thank an EJOR reviewer for the suggestion.

5

The SESAccuracy and the RandomWalkAccuracy were calculated to provide benchmark descriptive statistics for OrderHistoryAccuracy and POSAccuracy. As can be seen, in aggregate, both OrderHistoryAccuracy and POSAccuracy outperform SESAccuracy and RandomWalkAccuracy. This indicates that the forecasting methods used by the company outperform simple forecasting, as expected.

6

The %POSImprovement variable is a natural log variable, so raising e to its power effectively transforms the descriptive statistics to a nominal measure. The natural log weights the outliers so as to reduce their influence. The 11.2% mean value for %POSImprovement reported in Table 2 results from an average log value of 0.106 which is converted to a

AC

4

nominal percentage by taking 7

e % PosI mprovement - 1 where e is the natural log base.

Note that while the mean forecasting error for all forecasting methodologies are similar, the standard deviation of the forecasting error measures the average error of each estimate, and is a better measure of forecasting accuracy.

ACCEPTED MANUSCRIPT

For each order, then, we calculate how much better (or worse) the accuracy of POS forecasts compare to the accuracy of order-history forecasts. Calculations for POS forecasting error (POSErrorwid) and order-history forecasting (OrderHistoryErrorwid) both describe the percentage away from the amount ordered, so the optimal target is 0%, indicating that the forecast and the observed demand are equal with any deviation from zero indicating increased error. Reliability (measured by standard deviation) should be considered of more importance than mean error; the less deviation, the

CR IP T

more reliable the forecasting methodology. Of the standard deviations of error measurements, the POSErrorwid shows the tightest grouping close to the OrderQuantitywid, followed by OrderHistoryErrorwid, SESErrorwid, and finally RandomWalkErrorwid.

For each item, we calculate the ItemOrderFrequencyi by taking the total number of orders placed for a specific item by all distribution centers during the time period in this study. WeeklyActivityw is

AN US

calculated by taking the number of orders processed in a two-week period by the manufacturer, and is intended to capture overall demand fluctuations for all products. The descriptive statistics of the data shown in Table 2 reports that the error of order-history forecasting 11.2% higher than that of POS forecasting, using a log mean analysis.

The large swings in %POSImprovementwid indicate the need for some type of logarithmic

M

transformation whenever these variables are used in calculations, as described in Chellappa, Sin and Siddarth (2011). Following their insight, a log is taken of variables in Table 2 in order to capture the

ED

declining effect of explanatory variables on the dependent variable, and to scale all variables for outliers.

The average forecast error of all 4 methods is similar, which could be expected to be close to zero

PT

for most forecasting techniques, but the standard error within each forecast is where accuracy can be best determined. With the simplified assumption of a normal distribution of both POSErrorwid and

CE

OrderHistoryErrorwid, the reduced error of the forecasts made from POS information results in a

narrower distribution when compared to the forecasts made from order history. This means that, on

AC

the whole, forecasts made from POS information are closer to the observed demand than forecasts made by order history, and both outperform SES and random walk forecasts. The descriptive statistics in Table 2 also show that, in this data set, demands made by order history lean toward over-ordering rather than under-ordering at a greater degree than forecasts made through POS information. 8

8

There were several econometric challenges that had to be addressed in the formation of the models used in this research, specifically in the areas of nonlinearity, multicollinearity, heteroscedasticity, and a dependent variable that is not normally distributed. The interested reader can read about how these were handled in Appendix A.

ACCEPTED MANUSCRIPT

3.2. Empirical Model 1: A Nested Hierarchical Linear Model to Examine Factors that Increase in Demand Forecasting When Using POS Data For sporadic, sparsely ordered items, improvement of POS forecasting accuracy over order-history forecasting should be high as POS forecasting can sense these sporadic orders that are missed by orderhistory forecasting methodologies (Fay, 2010; Byrne, 2012;). At the other end of the spectrum, high order quantities allow for greater volatility in the quantity demanded per item.

Thus we see a

CR IP T

relationship between order quantity and improvements in forecast accuracy that is more complicated than can be described by a linear relationship: H1:

Higher Order Quantity will have a positive second order (parabolic) effect on the POS forecasting’s improvement in forecasting accuracy when predicting demand for that item when compared to order-history forecasting methods.

One of the promises of demand forecasts using POS data is that manufacturers can sense demand

AN US

volatility with less data. As such, the improvement in demand forecasting can be seen with a lower frequency in orders, and these improvements will decrease as more data becomes available and orderhistory techniques can finally start approaching the accuracy of POS forecasting: Higher Item Order Frequency for an item in a distribution center will be negatively correlated with the POS forecasting’s improvement in accuracy when predicting demand for that item when compared to order-history forecasting methods.

M

H2:

Byrne (2012) points out that another premise of POS forecasting is that when using POS As such, it is a natural

ED

forecasting, manufacturers can sense demand volatility more quickly.

hypothesis to predict that at extremely low levels of order quantity variability, order-history methods of forecasting will approach the accuracy of POS forecasting. However, as the variability of the order

techniques.

PT

quantity for an item increases, we should see improvements in POS forecasting over order-history While overall variability in item orders can be more accurately measured using more

CE

advanced estimation techniques and more rich data sources, excessive idiosyncratic variability in specific items may challenge the accuracy of the pattern-matching algorithms found within POS

AC

forecasting software. Thus, we have a situation where POS forecasting may perform better than orderhistory forecasting methods at lower levels of item order variability, but that improvements may decrease as variability becomes more pronounced, leading to the following hypothesis: H3:

Higher Item Order Variability for an item in a distribution center will have a negative second order (parabolic) effect on the POS forecasting’s improvement in forecasting accuracy when predicting demand for that item when compared to order-history forecasting methods.

We also propose that there will be observable or latent characteristics at the customer’s distribution center level that can affect POS forecasting at the manufacturer’s production level. Order quantities

ACCEPTED MANUSCRIPT

can change due to characteristics at the item level (e.g., Item Order Frequency, existing DC stock of the item) or at the DC-level (managerial expertise, average order size of the DC). As such, we propose that any analysis that examines orders for differences across distribution centers needs to take itemspecific and distribution-specific differences into consideration, thus leading to the following hypotheses: There will be a relationship between factors and POS forecasting improvements among observations within the same distribution center (i.e., there will be either significant random or fixed effects or both, at the item level nested within the DC level).

H5:

There will be a relationship between factors and POS forecasting improvements among observations within the same firm (i.e., there will either significant random or fixed effects or both at the DC level).

3.3. Hierarchical Linear Model (HLM) Development

CR IP T

H4:

AN US

We begin our analysis with an examination of how the explanatory variables impact the dependent variable (%POSImprovementwid). In particular, we hypothesize a curvilinear relationship between two explanatory variables, OrderQuantitywid and ItemOrderFrequencyi, and %POSImprovementwid. Such nonlinearities may be “washed out” as more regression terms are added to a regression model that can help explain variability in the dependent variable. It follows that non-linear relationships must be

M

considered when examining the effect that order quantity and item order frequency have on the POS improvement.

ED

In addition, there may be a relationship among individual observations within each distribution center (DC) due to similar practices and ordering procedures that are distinct to each DC. As such,

PT

ordinary least squares (OLS) analysis would violate the required assumption of independent observations (Kennedy, 1998). Chellappa, Sin, and Siddarth (2011) point out how grouped inventory

CE

data may be correlated (within groups) and heteroskedastic (between groups), thus violating two critical assumptions of OLS and also allowing noise to enter a model that may make stable unbiased estimates of coefficients and standard errors impossible using simple OLS.

AC

A hierarchical linear model (HLM) is particularly well suited for datasets where there is inherent

nesting in the data (Levin & Cross, 2004).

HLM relaxes the OLS assumption of independent

observations, and can be used to overcome the problems of grouped data faced by OLS models. This is done by applying both fixed effects and random effects to the regression, and by separating residuals into estimates at each group level, thus adjusting for any relationship of observations within groups and allowing for the possibility that observations within the same group may be more similar than observations from different groups (Hoffman, 1997; Raudenbush & Bryk, 2002).

ACCEPTED MANUSCRIPT

To illustrate, the following linear model similar to those often used in an OLS regression describes various observations (each denoted by i) made from various groups (denoted by j): yij = j0+j1xij1+2jxij2+…+kjxijk + ij Figure 4 (based heavily on insights found in Lee. et al., 2000a) describes how fixed and random effects can be applied to this linear the model in an HLM. The HLM moderates coefficients to adjust for group-level interdependencies among observations. In addition, the HLM moderates residuals that

 jk =

k0+k1W1+k2W2+…+kpWp+

The kth beta Coefficient for group j

jk

The unique random effect associated with Group j

AN US

The group-level effects that modify the kj estimate for within-group relationships

CR IP T

need to be moderated for group level random effects.

Figure 4. Explanation of Hierarchical Linear Modeling

HLM models are sometimes referred to as “mixed models” because they employ both fixed and random effects. The fixed effects allow us to investigate how variables at different levels of analysis can affect the explanatory variable. The random effects allow us to account for unobservable latent

M

variables that affect our analysis at different levels of analysis.

To begin our HLM, we start with the lowest level, which is at the purchase order level, defining (at

residuals normally distributed:

ED

the beginning) a standard linear model in Equation (2) for DC d ordering item i in week w with the

Level 1: Purchase Order Level:

PT

% POSI mprovementwid  

  id 1OrderQuantity wid   id 2 OrderQuantity wid   id 3 Capacity wid 2

2

(2)

CE

  id 4 ItemOrderVariability id   id 5 ItemOrderVariability id

  id 6 ItemOrderFrequency i   id 7 AvgOrderSized   id 8WeeklyActi vity w   wid

AC

 wid ~ N 0,  2 

Our data has two nested groups (in that purchase orders are nested within items, and items are

nested within DC). Equation (2) describes direct effects that can affect forecast improvement from POS demand forecasting taken from all three levels: DC-level variables are subscripted with a d); item-level variables nested within that DC are subscripted with an i (across all DCs) or an id; and weekly order-level variables nested within that item are subscripted with a w (across all items and DCs) or a wid. Non-linear relationships are captured by the squared terms shown in Equation (2)

ACCEPTED MANUSCRIPT

Because we have three levels of data nested within (the weekly purchase order, the item, and the DC levels), we are required to nest our HLM, so that the DC-level coefficients affect the item-level coefficients which in turn affect the purchase-order level coefficients. The coefficient adjustments are shown in Equations (3) and (4). Level 2: Item Level:

Level 3: Distribution Center Level  dm  m 0  m1 DCActivity d  m 2 DCSized   dm v dm ~ N 0, 

(3)

CR IP T

 idn   dn   d 1 ItemOrderFrequency id   d 2 ItemOrderVariability id   idn  idn ~ N 0, 

(4)

Note that our model in Equation (2) describes the direct effect of DC factors and item factors on

AN US

the accuracy of a prediction of quantity ordered on a purchase order. Also, Equations (2), (3), and (4) describe the HLM model that can be estimated using restricted maximum likelihood (REML) estimation to obtain unbiased estimates of the factors that affect the improvement of POS forecasting estimation over order-history forecasting methodologies.9

M

4. EMPIRICAL ANALYSIS RESULTS

This section examines relationships in the data with the HLM model described in Equations (2),

ED

(3), and (4), followed by a MNL model examination described in Equation (6).

4.1. HLM Model Examining Demand Sensing Forecast Accuracy

PT

Our data consists of 24 DCs that order from the manufacturer, and 2,285 item/DC combinations. Table 3 shows the results of our HLM analysis, conducted using the Stata 12.0 statistical package,

CE

which converged in eight iterations.

A Wald test of the overall model fit returned a 2 = 76.91, which was significant at p-value <

AC

0.0001, thus showing a statistical fit. Furthermore, when the HLM was tested against OLS regression, a Likelihood Ratio test returned a 2 = 1,154, showing an improvement of the HLM over OLS regression.

9

REML is a particular form of maximum likelihood estimation that uses a likelihood function calculated from a transformed set of data. REML delivers the most accurate estimators of the variance components when using HLM when compared to standard maximum likelihood estimations (Verbeke & Molenberghs, 2009).

ACCEPTED MANUSCRIPT

Our results show support for most of our hypotheses tested by this model. We find support for H1, where we predict a nonlinear effect of order quantities upon POS forecasting. POS forecasts show the greatest improvement when order quantities are relatively low or relatively high. At low order quantities, we propose that POS forecasting performs best when little order data is available. At high order quantities, we propose that POS forecasting better able handle a high variability of demand that is often present at higher order quantities.

CR IP T

Our results support H2, where POS forecasting is shown to have larger improvements in items that have lower levels of order frequency. We posit that as the order frequency of an item increases, the increased order history information makes any POS improvement less pronounced. This supports Raghunathan’s (2001) contention that sharing POS data is of negligible value when order-history data

AC

CE

PT

ED

M

AN US

is extensive. 10

10

Note that extremely small coefficients may be the result of large magnitude items (like ItemOrderVariabilityid) affecting dependent variables that vary only slightly (such as %POSImprovementwid). It would be a mistake in these cases to assume that a small coefficient means a small overall effect.

ACCEPTED MANUSCRIPT

Table 3. Hierarchical Linear Model Regression using Restricted Maximum Likelihood Estimation Estimating Factors that Affect the Percent of Improvement of POS Forecasts over Order History Forecasts

ItemOrderFrequencyi ItemOrderVariabilityid Random Effects (id)

H4

2.50e-05 0.0812 0.2914

9.90e-06 0.0180 0.0105

CR IP T

(Constant) OrderQuantitywid OrderQuantity2wid Capacitywid ItemOrderFrequencyi ItemOrderVariabilityid ItemOrderVariability2id AvgOrderSized WeeklyActivityw

Hyp. Coeff. Std. Err. Z-stat {95% Confidence} Level 1 –Weekly Observation (Direct Effects) 0.2013 0.1609 1.25*** {-0.114, 0.517} *** 0.0311 0.0082 3.79 {0.015, 0.047} H1 *** 0.0151 0.0031 4.86 {0.009, 0.021} 0.0382 0.0171 2.24*** {0.005, 0.072} H2 -0.0985 0.0260 -3.78*** {-0.150, -0.047} -0.0279 0.0109 -2.57*** {-0.049, -0.007} H3 *** -0.0106 0.0039 -2.75 {-0.018, -0.003} -0.0026 0.0029 -0.91*** {-0.008, 0.003} 0.1921 0.0421 4.57*** {0.110, 0.275} Level 2 – Item Level (Fixed and Random Effects)

AN US

Variable

{1.15e-05, 5.43e-05} {0.053, 0.125} {0.272, 0.313}

Level 3 – DC Level (Fixed and Random Effects) AvgOrderSized Random Effects (d)

H5

7.16e-09 0.110

3.74e-08 0.029

Residual 1.357 0.004

wid

{1.349, 1.365}

p-value < .05; **p-value < .01; ***p-value < .001; Number of observations = 60,651 Variables with squared terms are centered to mitigate collinearity with the first and second order terms. Dependent Variable is from Equation (2): %POSImprovementwid

ED

M

*

{2.56E-13, 2.00E-04} {0.066, 0.184}

Our results also support H3, in that the relationship between item order variability and the improvements of POS forecasting follow a negatively parabolic path. We find that improvements of

PT

POS forecasting are greatest when items are at a relatively moderate level of variability. We posit that, at the low end of item order variability, order-history forecasting accuracy approaches that of POS

CE

forecasting since the lower levels of variability allow order-history forecasting to predict more accurately. At high levels, we point out that additional information used by POS forecasting has a

AC

smoothing effect on the demand calculations, and while POS forecasting is still shown to have an improvement in forecast accuracy at these levels, the improvements are not as pronounced in this case. This finding is in-line with Williams and Waller’s (2010) conclusion that non-linear trends can distort the interpretation of POS data. The significant second order terms in this analysis might call for a type of ABC/XYZ SKU analysis, which examines relationships using product group classifications based on demand volume (the ABC classification) paired with product group classifications based on demand variance (the XYZ classification) to find relationships that vary within each group (Schomer 1965; Wildemann 1990).

ACCEPTED MANUSCRIPT

A limitation of random effects models, such as HLM, is that these models do not allow for negative within-group correlations in the error terms (Mason 1995). To address this weakness, it is common practice to show HLM calculations with a 95% confidence interval. Standard error calculations can still be examined against coefficient magnitudes to see the impact of each level 2 and level 3 HLM effect on the model shown in level 1. Group level random effects had significantly higher coefficient magnitudes when compared to the standard error of the estimates at both the DC-level and the item-

CR IP T

level, indicating that there are unobservable latent effects (e.g., DC management style, item level competition) that are affecting orders at both the DC level and the item level, thus supporting H4 and H5. While the item-level variables (ItemOrderVariabilityid and ItemOrderFrequencyi) also showed relatively low levels of standard error as a proportion to the coefficient magnitude, indicating group level effects of these variables, the DC-level variable we examined in level 3, AvgOrderSized, showed

AN US

a higher standard error than the coefficient estimated, indicating that level 1 coefficients are not impacted much at the DC level (level 3) by the average order sizes made by that DC. There are many managerial implications from the results shown in Table 3. While POS forecasting strictly outperforms order-history forecasting techniques, in our dataset, our results show that POS forecasting works at relatively low or relatively high levels of order quantity, so in order to justify the

M

cost of a movement to POS forecasting, managers can examine cost and profit factors for items at these levels. Our results also suggest that POS forecasting outperforms order-history forecasting when the

ED

order frequency is low. For companies that are expanding into new product lines where new products become available for sale on a frequent basis, POS forecasting can adjust more quickly than orderhistory forecasting. Finally, Table 3 shows that POS forecasting outperforms order-history forecasting

PT

at a moderate level of item order variability, although the improvements of POS forecasting decrease at

CE

extremely high levels of item order variability.

5. FURTHER EXPLORATORY ANALYSIS USING MULTINOMIAL LOGIT Because underestimated forecasts can result in lost sales, some firms consider an overestimated

AC

demand more preferable to an underestimated demand.11 Langer, Forman, Kekre and Scheller-Wolf (2007) address this issue and estimate the probability of an under-order (thus reducing the safety stock inventory or resulting in an out-of-stock order) when radio frequency ID (RFID) is implemented within the supply chain, by examining the marginal increase in the probability that a forecast over-estimates

11

Conversations with the manufacturer confirm that, in this case, any out-of-stock situation resulting from an underestimated demand could result in lost sales and they maintain a safety stock that, in combination with new manufacturing, allows over 99% of orders to be filled.

ACCEPTED MANUSCRIPT

an order. We follow their logic and conduct further analysis to explore if a methodology has factors that lead to an over-order or under-order using a multinomial logit (MNL) model. Table 4 describes the four exhaustive possible binary outcomes used in a MNL analysis. Table 4. Four Possible Forecasting Comparison Options Discussed in This Research12

Observations 21,644

6,679

Pr(ISAPOS)

6,794

Pr(ISABoth)

25,534

Total

60,651

AN US

Pr(ISAORDHIST)

Description This describes the probability that neither POS forecasts nor order-history forecasts meet or exceed the observed order level; thus both forecasts decrease the inventory/stock availability. This describes the probability that order-history forecasts meet or exceed the observed order level while POS forecasts are below the observed order level. This describes the probability that POS forecasts meet or exceed the observed order level while order-history forecasts are below the observed order level. This describes the probability of both POS forecasting and order-history forecasts meet or exceed the observed order level`, thus both forecasts maintain or increase the inventory/stock availability.

CR IP T

Probability Pr(ISANone)

The goal of this MNL model is to estimate the effects of independent factors on the marginal increase in the probability that a forecast meets or exceeds observed orders when comparing POS forecasting to order-history forecasting. Equation (5) shows the XNx9 matrix of explanatory variables used in our MNL analysis, which are identical to the explanatory variables used in our HLM analysis

ED

2  OrderQuantity wid  ItemOrderFrequency i  ItemOrderVariability id2   WeeklyActi vity w 

(5)

PT

 Nx9

1 OrderQuantity wid  Capacity wid   ItemOrderVariability id   AvgOrderSized

M

where N is the number of observations and there are eight explanatory variables and the constant.

It is reasonable to use the same independent variables for both empirical analyses. The HLM analysis measures how accurate the POS forecast is compared to the order-history forecasts while the

CE

MNL analysis measures how likely the probability that the POS forecast is to exceed the demand when compared to order-history forecast. As such, independent variables that inform on one analysis should

AC

also inform on the other. We are interested in the examination of factors that affect the functioning of POS forecasting: specifically factors that result in maintaining or increasing inventory levels. Particularly, this research examines the factors that affect the improvement of Pr(ISAPOS) when compared to Pr(ISAOrdHist). Logit models allow the probability of a binary outcome to be transformed

12

Organizations can keep an inventory of safety stock to avoid out of stock situations, thus sacrificing working capital tied up in excess inventory in order to ensure that orders are met. More accurate forecasts, then, can reduce the required quantity of safety stock, thus providing more working capital to the organization.

ACCEPTED MANUSCRIPT

using a logistic transformation that resembles a cumulative normal distribution (Kennedy, 1998). Following the technique described by Kennedy (1998) and by Jain and Kini (1999), we derive Equation (6), which shows a MNL model where a log-odds ratio formula estimates the coefficients (in BMNL, a 9x1 column vector of coefficient and constant estimates) of the observed factors that lead to an improvement of POS forecasting (ISAPOS) over order-history forecasting (ISAOrdHist): (6)

CR IP T

 PrISAPOS    PrISAPOS    PrISAOrdHist     ln    ln     POS  OrdHist    MNL ln   PrISAOrdHist    PrISABoth    PrISABoth  

Through our application of the MNL model exhausting all possible alternatives, Equation (6) allows us to estimate factors that cause a marginal increase in probability that a forecast meets or exceeds actual orders when comparing POS to order-history forecasts.

Because even irrelevant

alternatives were added to the model to reach an exhaustive state, the normal weakness of logit models

AN US

to the effects of irrelevant alternatives cannot apply in this case, and our estimates are unbiased. This MNL analysis shown in Table 5 uses robust standard error estimates that are clustered by Item and DC, and was conducted using the Stata 12.0 statistical package, which converged in five iterations. Table 5. Robust Multinomial Logit Model Results Examining Factors that Improve the Probability That Forecasts Meet or Exceed Observed Order Quantity when Comparing POS Forecasts to Order-History Forecast

CE

M

-0.0012 0.0007 0.0067 -0.0003 0.0114 -0.0001 -0.0023 0.0212

ED

Coeff. 0.4641 -0.0408 0.0184 0.0818 0.0288 0.1800 -0.0088 -0.0139 -0.2708

PT

Variable (Constant) OrderQuantitywid OrderQuantity2wid Capacitywid ItemOrderFrequencyi ItemOrderVariabilityid ItemOrderVariability2id AvgOrderSized WeeklyActivityw

Marginal Effects

Robust Standard Error 0.7112 0.0495 0.0157 0.0676 0.1091 0.0531 0.0098 0.0087 0.1907

Z-stat 0.65*** -0.83*** 1.17*** 1.21*** 0.26*** 3.39*** -0.90*** -1.60*** -1.42***

{95% Confidence Interval} {-0.930, 1.858} {-0.138, 0.056} {-0.012, 0.049} {-0.051, 0.214} {-0.185, 0.243} {0.076, 0.284} {-0.028, 0.010} {-0.031, 0.003} {-0.644, 0.103}

**

AC

p-value < .01; Number of observations = 60,651 Robust Standard Errors are Clustered by Item and DC Variables with Second Order Terms (OrderQuantitywid and OrderQuantitywid) are centered to mitigate collinearity Dependent variable is the ratio of forecasts that meet or exceed observed orders when POS forecasts are compared to order history forecasts, from Equation (6): ln  PrISAPOS    PrISA  OrdHist   

For qualitative methodologies like MLM, there is not a universally accepted goodness of fit measure analogous to R2 in OLS regression (Siverson & King, 1980) although many statistics packages report a pseudo-R2 measure for MNL models. We report a McFadden's adjusted R2=12.3%, which is a conservative test that is indicative of a reasonable explicative capability. A more accepted

ACCEPTED MANUSCRIPT

evaluator used for MLMs is a Wald test using the 2 statistic, which shows a good fit ( 2 = 2,094; p < 0.001). The results in Table 5 show that ItemOrderVariabilityid is positively related to an increase in the probability that a POS forecast meets or exceeds the observed order quantity compared to the probability that an order history forecast meets or exceeds the observed order quantity in this case. Taken with the results in Table 3, the results in Table 5 present an interesting picture. While POS

CR IP T

forecasts are superior to order history forecasts in this study’s data, the accuracy of POS forecasts over order history forecasts decreases as ItemOrderVariabilityid increases. In addition, POS forecasts tend to overstate forecasts as ItemOrderVariabilityid increases. This gives support to a contention that a slight downward adjustment of POS forecasts is appropriate as ItemOrderVariabilityid increases. However, more research is required in this area before any definitive conclusions can be made.

AN US

6. FURTHER DISCUSSION AND MANAGERIAL IMPLICATIONS

The overall results from our hypotheses can be used to further examine the relationships between item, order, and DC characteristics and the overall improvement of POS forecasting. Using a bar chart for various percentile measures of OrderQuantitywid, Figure 5 graphically depicts how OrderQuantitywid

M

relates to %POSImprovementwid, with the best fitting second order trend line added to the bar chart.

12% 10% 8%

ED

14%

PT

Average %POSImprovement

16%

6%

CE

4% 2%

AC

0%

0%-20%

20%-40%

40%-60%

60%-80%

80%-100%

OrderQuantity Percentile

Figure 5. Average %POSImprovement per OrderQuantity Percentile with the Best Fitting Second Order Trend Line Added

The graph in Figure 5 depicts the movement of %POSImprovementwid per OrderQuantitywid percentile without accounting for any group effects or variability absorbed by our HLM empirical model. Despite that, the curvilinear relationship between OrderQuantitywid and %POSImprovementwid seems to be readily apparent, as shown by the best fitting second-order trend line. Notice the bar chart

ACCEPTED MANUSCRIPT

shows that, at every quartile, POS forecasting outperforms order-history forecasting with %POSImprovementwid ranging from around 6% at the 20%-40% quartile to over 14% at the 80%-100% quartile. Notice also that %POSImprovementwid starts at around 11% at the lowest quintile, and immediately drops in the second quintile, but then rises as the order quantity increases. This is why the second order term in Table 3 was significant. Figure 6 shows the relationship between ItemOrderFrequencyi and %POSImprovementwid. Notice

CR IP T

that the second order condition that defines a statistically significant parabolic curve, yet the curve is very slight, and the overall curve appears to show an approximate linear relationship. 20%

16% 14%

12%

AN US

Average %POSImprovement

18%

10% 8% 6% 4%

2% 0%

25%-50%

M

0%-25%

50%-75%

75%-100%

ED

ItemOrderFrequency Percentile

Figure 6. Average %POSImprovement per ItemOrderFrequency Percentile with the Best Fitting Second Order Trend Line Added

average,

PT

Figure 6 shows how the improvements in POS forecasting over order-history forecasting are, on consistently

positive

at

every

quartile.

While

always

positive,

however,

CE

%POSImprovementwid decreases at every quartile, indicating how POS forecasting shows the largest improvements (over 18%) over order-history forecasting when at low order frequencies and when little

AC

information is known about the item. This improvement decreases as more information becomes available, allowing order-history forecast methodologies to approach (though not quite achieve) the accuracy of POS forecasting at high levels of item order frequency. Figure 7 shows the relationship between ItemOrderVariabilityid and %POSImprovementwid. Once

more, the best fitting trend line is indicative of a strong second order (parabolic) effect at different levels of variability in a DC’s orders for an item.

ACCEPTED MANUSCRIPT

Figure 7 shows that, at extremely high levels of variability (80%-100%), the improvement in average accuracy drops to just above 6%, as pattern matching techniques are known to underperform in the presence of high variability (Russell, et al., 1986; Malhotra & Malhotra, 2003).

16% 14% 12% 10%

8% 6% 4% 2% 0%

0%-20%

20%-40%

40%-60%

60%-80%

80%-100%

AN US

ItemOrderVariability Percentile

CR IP T

Average %POSImprovement

18%

Figure 7. Average %POSImprovement per ItemOrderVariability Percentile with the Best Fitting Second Order Trend Line Added

M

Taken as a whole, Figures 5, 6, and 7 paint a more complete picture of how POS demand forecasting compares to order-history demand forecasting. Note that while order-history demand

ED

forecasting can outperform POS demand forecasting on individual items, on average, POS forecasting tends to outperform order-history forecasting in every percentile aggregation shown in these figures. Taken together, our observations show that POS forecasting outperforms order-history forecasting

PT

overall, but that different factors can increase or decrease that improvement. Managers can benefit from this analysis and these techniques to examine how their own product

CE

forecasting techniques are affected by item order variability, item order frequency, and order quantities in order to use all available information to make more accurate forecasts. Using the insights from this

AC

research, other researchers and consultants can build models and tools that incorporate this information to improve forecasts. Therefore, they can increase sales through increased item order variability and reduce costs by reducing the safety stock that is kept on hand to adjust for errors in forecasting.

7. CONCLUSION The strength of this research is in tackling the interesting problem of demand forecasting using complex data from a major manufacturer. In this research, we continue the empirical investigation of demand forecasting, comparing order-history forecasting with POS forecasting. Our analysis shows that, in our case, POS forecasting outperforms order-history forecasting, but that these improvements

ACCEPTED MANUSCRIPT

can increase or decrease depending upon specific order and item characteristics, and can tend to be curvilinear.

As such, it is no surprise that some empirical models and organizations see little

significant improvement when implementing POS forecasting. We hope that the results from this research will guide future research when developing empirical models and forecasting tools. Our dataset contains 60,651 observations of orders given to a manufacturer for various items from various retailer distribution centers. Using a hierarchical linear model, we show group-level specific

CR IP T

and latent effects for both specific items and specific distribution centers. After these effects have been accounted for, we hypothesize and find a positive curvilinear relationship between the order quantity and the percent of improvement of POS forecasting when compared to order-history forecasting. This indicates that improvements found when using POS forecasting are best (over order-history forecasting) when order quantities are relatively low (and therefore, there is little prior information to

AN US

predict orders) or when order quantities are relatively high and there is significant room for DC managers to adjust their order.

Conversely, our results illustrate a negative curvilinear relationship between item order variability and the percent of improvement of POS forecasting when compared to order-history forecasting. This indicates, in this case, that POS forecasting improvements are highest with items whose order

M

variability is near the median. We attribute this to improvements in forecasting when variability is higher, but when there are extremely high levels of order variability, longer-term information found in

ED

order-history data is necessary to determine a pattern and predict demand. Finally, in this case, POS forecasting improves most with low order frequency, and because of order-history forecasting to overestimate demand, the probability that a POS forecast meets or exceeds observed order quantities

PT

increases with higher order frequency when compared to order history forecasts. There are some limitations of this study. Primarily, the software and techniques used by this

CE

manufacturer for both order history forecasts and POS forecasts were provided to the manufacturer by third-party software vendors. As such, the mathematical formulas and processes used by the third

AC

party vendors are considered proprietary and were only described to us and to the manufacturer at a general level with no specificity. Thus, while our statistical analysis delivers unique insights, we are limited with this dataset to make significant analytical contributions that could add to the theory in this area. This research should be supplemented with analytical research that can shed more light on the analytical models. In addition, we take our data from a single manufacturer. Deeper insights can be gleaned from future research that spans multiple manufacturers who have implemented POS forecasting, and further empirical research can also be conducted to test this newly developed theory

ACCEPTED MANUSCRIPT

and to examine further boundaries regarding the accuracy of demand forecasts using different methodologies.

8. REFERENCES Aviv, Y. (2001). The effect of collaborative forecasting on supply chain performance. Management Science, 47(10), 1326-1343.

CR IP T

Baker, P. (2007). An exploratory framework of the role of inventory and warehousing in international supply chains. International Journal of Logistics Management, 18(1), 64-80. Byrne, R. F. (2012). Beyond Traditional Time-Series: Using Demand Sensing to Improve Forecasts in Volatile Times. The Journal of Business Forecasting, 31(2), 13-19. Cachon, G. P., Randall, T., & Schmidt, G. M. (2007). In search of the bullwhip effect. Manufacturing & Service Operations Management, 9(4), 457-479.

AN US

Chellappa, R. K., Sin, R. G., & Siddarth, S. (2011). Price formats as a source of price dispersion: A study of online and offline prices in the domestic U.S. airline markets. Information Systems Research, 22 (1), 83-98 Chen, L., & Lee, H. L. (2009). Information sharing and order variability control under a generalized demand model. Management Science, 55(5), 781-797.

M

Cho, D. W., & Lee, Y. H. (2013). The value of information sharing in a supply chain with a seasonal demand process. Computers & Industrial Engineering, 65(1), 97-108.

ED

Davydenko, A., & Fildes, R. (2013). Measuring forecasting accuracy: The case of judgmental adjustments to SKU-level demand forecasts. International Journal of Forecasting, 29(3), 510-522. Fay, G. (2010). Demand sensing in volatile times. The Journal of Business Forecasting, 29(3), 31.

PT

Fildes, R. (1989). Evaluation of aggregate and individual forecast method selection rules. Management Science, 35(9), 1056-1065.

CE

Fildes, R., & Petropoulos, F. (2015). Simple versus complex selection rules for forecasting many time series. Journal of Business Research, 68(8), 1692-1701.

AC

Folinas, D., & Rabi, S. (2012). Estimating benefits of Demand Sensing for consumer goods organisations. Journal of Database Marketing & Customer Strategy Management, 19(4), 245-261. Fulcher, J. (2009). Unilever betting on ‘demand sensing’ to weather volatile market conditions. Manufacturing Business Technology, 27(5), 7. Gal-Or, E., T. Geylani, and A. Dukes. (2008). Information sharing in a channel with partially informed retailers. Marketing Science, 27(4), 642-658. Gardner, E. S. (1985). Exponential smoothing: The state of the art. Journal of Forecasting, 4(1), 1-28. Hammer, M. (2001). The superefficient company. Harvard Business Review, 79(8), 82-93.

ACCEPTED MANUSCRIPT

Holt, C.C. (1957). Forecasting seasonals and trends by exponentially weighted moving averages. Office of Naval Research Memorandum (Vol. 52), Pittsburgh, PA: Carnegie Institute of Technology. Hoffman, D. (1997). An overview of the logic and rationale of hierarchical linear models. Journal of Management, 23(6) 723-744. Jain, B. A., & Kini, O. (1999). The life cycle of initial public offering firms. Journal of Business Finance & Accounting, 26(9‐10), 1281-1307.

CR IP T

Kennedy, P. (1998). A Guide to Econometrics, 4th Edition. Cambridge, MA: MIT Press. Kiely, D. A. (1999). Synchronizing supply chain operations with consumer demand using customer data. Journal of Business Forecasting Methods and Systems, 17, 3-9. Kim, K. K., Umanath, N. S., & Kim, B. H. (2005). An assessment of electronic information transfer in B2B supply-channel relationships. Journal of Management Information Systems, 22(3), 294-320.

AN US

Langer, N., Forman, C., Kekre, S., & Scheller-Wolf, A. (2007). Assessing the impact of RFID on return center logistics. Interfaces, 37(6), 501-514. LaViola, J. J. (2003, May). Double exponential smoothing: an alternative to Kalman filter-based predictive tracking. In the Proceedings of the workshop on virtual environments 2003 (pp. 199-206). Lee, H. L., Padmanabhan, V., & Whang, S. (2004). Information distortion in a supply chain: the bullwhip effect. Management Science, 50 (12_supplement), 1875-1886.

M

Lee, H. L., Padmanabhan, V., & Whang, S. (1997). The bullwhip effect in supply chains1. Sloan management review, 38(3), 93-102.

ED

Lee, H. L., So, K. C., & Tang, C. S. (2000a). The value of information sharing in a two-level supply chain. Management Science, 46(5), 626-643.

PT

Lee, H. L., & Whang, S. (2000b). Information sharing in a supply chain. International Journal of Manufacturing Technology and Management, 1(1), 79-93.

CE

Levin, D. Z., & Cross, R. (2004). The strength of weak ties you can trust: The mediating role of trust in effective knowledge transfer. Management Science, 50(11), 1477-1490.

AC

Lyu, J., Ding, J. H., & Chen, P. S. (2010). Coordinating replenishment mechanisms in supply chain: From the collaborative supplier and store-level retailer perspective. International Journal of Production Economics, 123(1), 221-234. Makridakis, S. (1990). Sliding simulation: a new approach to time series forecasting. Management Science, 36(4), 505-512. Mason, W.M. (1995) Hierarchical Linear Models: Problems and Prospects. Journal of Educational and Behavioral Statistics, 20(2), 221-227. McClain, J. O. (1974). Dynamics of exponential smoothing with trend and seasonal terms. Management Science, 20(9), 1300-1304.

ACCEPTED MANUSCRIPT

Raghunathan, S. (2001). Information sharing in a supply chain: A note on its value when demand is nonstationary. Management Science, 47(4), 605-610. Rai, A., Patnayakuni, R., & Seth, N. (2006). Firm performance impacts of digitally enabled supply chain integration capabilities. MIS quarterly, 30(2), 225-246. Raudenbush, S., & Bryk, A. (2002). Hierarchical linear models: Applications and data analysis methods (2nd ed.). Thousand Oaks, CA: Sage.

CR IP T

Sahin, F., & Robinson, E. P. (2002). Flow coordination and information sharing in supply chains: review, implications, and directions for future research. Decision sciences, 33(4), 505-536. Sanders, N. R. (2007). The benefits of using E‐business technology: the supplier perspective. Journal of Business Logistics, 28(2), 177-207. Saraf, N., Langdon, C. S., & Gosain, S. (2007). IS application capabilities and relational value in interfirm partnerships. Information Systems Research, 18(3), 320-339.

AN US

Schomer, A. J. (1965). Management controls & information: An approach to inventory management. Journal of Accountancy, 120 (2), 75-77. Setaputra, R., Yue, X., & Yao, D. (2010). Impact of Information Systems on Quick Response Programs. In Innovative Quick Response Programs in Logistics and Supply Chain Management (pp. 23-36). Springer Berlin Heidelberg.

M

Simatupang, T. M., & Sridharan, R. (2002). The collaborative supply chain. The International Journal of Logistics Management, 13(1), 15-30.

ED

Simatupang, T. M., & Sridharan, R. (2005). An integrative framework for supply chain collaboration. The International Journal of Logistics Management, 16(2), 257-274.

PT

Siverson, R.M., J. King. (1980). Attributes of national alliance membership and war participation: 1815-1965. American Journal of Political Science, 24(1), 1-15. StataCorp, Stata Time-series: Reference Manual. StataCorp LP, 2007.

CE

Steckel, J. H., Gupta, S., & Banerji, A. (2004). Supply chain decision making: Will shorter cycle times and shared point-of-sale information necessarily help? Management Science, 50(4), 458-464.

AC

Taylor, J. W. (2010). Triple seasonal methods for short-term electricity demand forecasting. European Journal of Operational Research, 204(1), 139-152. Trapero, J. R., Kourentzes, N., & Fildes, R. (2012). Impact of information exchange on supplier forecasting performance. Omega, 40(6), 738-747. Tracey, M., Lim, J. S., & Vonderembse, M. A. (2005). The impact of supply-chain management capabilities on business performance. Supply Chain Management: An International Journal 10(3), 179-191. Verbeke, G., & Molenberghs, G. (2009). Linear mixed models for longitudinal data. New York: Springer Science & Business Media.

ACCEPTED MANUSCRIPT

Williams, B. D., & Waller, M. A. (2010). Creating order forecasts: point‐of‐sale or order history? Journal of Business Logistics, 31(2), 231-251. Williams, B. D., & Waller, M. A. (2011). Top‐Down Versus Bottom‐Up Demand Forecasts: The Value of Shared Point‐of‐Sale Data in the Retail Supply Chain. Journal of Business Logistics, 32(1), 17-26.

CR IP T

Williams, B. D., Waller, M. A., Ahire, S., & Ferrier, G. D. (2014). Predicting retailer orders with POS and order data: The inventory balance effect. European Journal of Operational Research, 232(3), 593600. Wildemann, H. (1990). Das Just-in-Time-Konzept: Produktion und Zulieferung auf Abruf. Grin Verlag Gmbh. Winters, P. R. (1960). Forecasting sales by exponentially weighted moving averages. Management Science, 6(3), 324-342.

AN US

Wooldridge, J. (2012). Introductory econometrics: A modern approach. Cengage Learning.

Wu, Y. N., & Cheng, T. E. (2008). The impact of information sharing in a multiple-echelon supply chain. International Journal of Production Economics, 115(1), 1-11.

AC

CE

PT

ED

M

Zhou, H., & Benton, W. C. (2007). Supply chain practice and information sharing. Journal of Operations Management, 25(6), 1348-1365.