Decision support for university enrollment management: Implementation and experience

Decision Support Systems 44 (2007) 106 – 123 www.elsevier.com/locate/dss

Decision support for university enrollment management: Implementation and experience Elliot N. Maltz a , Kenneth E. Murphy b,⁎, Michael L. Hand c a

Professor of Marketing, Atkinson Graduate School of Management, Willamette University, 900 State Street, Salem, Oregon 97301, United States b Associate Professor of Information Systems, Atkinson Graduate School of Management, Willamette University, 900 State Street, Salem, Oregon 97301, United States c Professor of Applied Statistics and Information Systems, Atkinson Graduate School of Management, Willamette University, 900 State Street, Salem, Oregon 97301, United States Received 2 April 2006; received in revised form 5 March 2007; accepted 18 March 2007 Available online 30 March 2007

Abstract Enrollment management is a process critical to many universities that rely on tuition for a significant portion of their operating budgets. This study describes how the development and implementation of a system to support decisions in the enrollment process allowed for increased responsiveness and real-time management as well as substantially increased institutional knowledge of the process itself. This, in turn, led to dramatic improvements in both operational performance and in the attainment of strategic admission objectives. © 2007 Elsevier B.V. All rights reserved. Keywords: Decision support system; Enrollment management; Data mining; Organizational learning

1. Introduction Most private colleges, unless they have developed a very large endowment, base their revenue primarily on tuition income. Consider, as an example, a moderatesized undergraduate liberal arts program with a budget of $50 million. The college would require an endowment of $500 million to cover half of their budget under standard 5% annual growth assumptions. Since few private liberal arts colleges have an endowment of that magnitude, a systematic approach to enrollment management is critical to ensuring stability in fiscal planning. Schools approach the technical challenges associated with en⁎ Corresponding author. E-mail address: [email protected] (K.E. Murphy). 0167-9236/$ - see front matter © 2007 Elsevier B.V. All rights reserved. doi:10.1016/j.dss.2007.03.008

rollment management in a variety of ways, often relying on offices of institutional research to perform this function or by staffing the admission office with statistical specialists [5]. However, many smaller schools lack the resources or the technical expertise to address these problems internally. In these cases, outside consultants are often hired to assist in determining which students to admit and how much financial aid to offer in order to recruit a desirable incoming class. This approach can result in suboptimal performance, additional costs and may curtail the opportunity for institutional learning with respect to managing the admissions process. This manuscript presents the design and implementation of a successful decision support system (DSS) for enrollment management at a small liberal arts college. The DSS, an integral component of the admissions

E.N. Maltz et al. / Decision Support Systems 44 (2007) 106–123

process consists of two components—a predictive model and a user-friendly interface which allows the school to dispense with the services of outside consultants while at the same time making significant operational gains. The interaction between the DSS and the admissions process can be thought of as the enrollment work system [2]. The two-year, two-phase implementation project improved the enrollment work system by enhancing understanding of the admissions process overall through the conversion of tacit process knowledge to explicit and by its impact on financial measures of performance. A variety of data mining techniques and associated methodologies were used to assist in developing the predictive model. As will be demonstrated, the methodologies employed to develop the DSS as well as the DSS itself contributed to both the operational success of the system and the organizational learning achieved during the design and implementation phases. The enrollment management tool was implemented in an environment that was based on principles that have been recognized as important by the decision support system literature [3,11,29,32]. As such the insights provided in this paper contribute to both the enrollment management and decision support system implementation literatures. The balance of the manuscript is organized as follows. The following section provides a brief literature review of decision support and expert systems in the admissions setting as well as relevant observations on system implementation from the DSS literature. The legacy admission process and its associated challenges at the institution where the decision support system was implemented are then described, followed by a description of the data mining methodology used for constructing the system. The manuscript then reviews the operational and learning outcomes of the implementation. This section provides guidance for the development of DSS systems for enrollment management. The paper concludes with a broader discussion of the insights for successful implementation of DSS systems. 2. Decision support systems in the admissions process Applications of management science techniques in academic administration go back forty or more years. In early implementations, the issues addressed were planning, budgeting or resource allocation problems including the forecasting of enrollment levels as well as facilities requirement planning, course scheduling and staffing to support estimated enrollments (e.g., see [28] and [30]). A survey of 146 articles identified 104 that

107

employed management science (optimization) techniques while only 6 of 146 articles featured DSSs to tackle academic management problems. In this survey, there were no examples of DSS deployed directly for enrollment management [34]. In general, research on enrollment management has centered on two areas: developing forecasting models for predicting overall enrollment levels and on tools for identifying which individual applicants to admit. Many of the institutional level forecasting studies build models to identify the traits of students that choose the focal institution over others (see, e.g., [13] and [27]). Multiple linear, logistic and probit regression models were observed as the most commonly employed techniques for forecasting at this level [26]. Logistic regression has been compared to neural networks for classifying which students will and will not enroll in a university, based on a variety of applicant attributes, and neural networks were found to outperform logistic regression for the correct classification of admitted applicants who ultimately will and will not enroll [33]. With respect to the current problem, these results provide insight, but unfortunately, none of these studies incorporate the amount of financial aid awarded to applicants, a significant factor in the enrollment decision [6]. Beginning in the late 1980s several researchers discussed the use of expert systems for determining which students to admit into a variety of academic programs in Great Britain and elsewhere [10–12, 19,23,24]. While several of these papers implicitly consider the probability of enrollment in the analysis, this body of work does not explicitly consider the financial impact of enrollment decisions on the institution, a fundamental concern for many institutions. As such, from an operational perspective, our work builds on previous studies by considering both the probability of enrolling and the amount of financial aid awarded. The quality of the predictive model incorporated into the DSS is an essential element in improving the performance of its associated work system [2]. However, DSS systems should also provide for the systematic acquisition and sharing of tacit and explicit knowledge to improve effectiveness and control [1] as well getting the right information to the right people at the right time [25]. In our context, the DSS system should incorporate knowledge acquired from experience and historical data on enrollment probability and provide a mechanism to share this information explicitly with the admissions decision makers. With this in mind, this paper describes how the DSS' interface was crucial to make this information available effectively and expediently.

108

E.N. Maltz et al. / Decision Support Systems 44 (2007) 106–123

Hartono et al., [17] provide a valuable review of factors that lead to DSS success, and in particular the research demonstrates that the relative importance of various implementation factors depends on how success is defined. In this setting success for the DSS is defined as “organizational impacts” [8], that is improving operational performance and increasing explicit knowledge and knowledge sharing. Antecedents of success on organizational impacts include management support, organizational support and attitude, user participation and system characteristics [3,15,17,29,32,35]. In particular Teo [32] suggests four categories of critical success factors in a knowledge management DSS implementation: people and culture, implementation method, content management and technology. Experience with respect to these factors over the two year cycle of the DSS design and implementation is described towards the end of the paper. However, as one might expect, the specific aspects of these factors that are important differ in each setting. As such, previous work is useful in providing a basis towards a better understanding of how particular critical success factors transferred to this

setting. The next section proceeds with a discussion of the enrollment management process prior to the implementation of the DSS. 3. The traditional enrollment management process The Willamette University College of Liberal Arts (CLA) in Salem, Oregon is typical of small schools that have traditionally relied upon outside consultants for technical guidance. Each year, from a pool of more than 3000 applications, admission is offered to approximately 1800 applicants, to achieve a target entering class of approximately 500. In fact, the actual percentage of admitted students who will enroll is not known in advance, and hence a critical task for any admissions office is to accurately predict this percentage. The percentage of admitted applicants who ultimately enroll is referred to as the enrollment yield. If the yield is overestimated, fewer students than expected will enroll and revenue to the university will be reduced. If the yield is underestimated, a higher than expected number of students will enroll, possibly exceeding the fixed capacity

Fig. 1. Traditional enrollment management process.

E.N. Maltz et al. / Decision Support Systems 44 (2007) 106–123

of the school and resulting in significant incremental costs for additional housing, faculty, and other resources. In the worst case, over-enrollment could compromise the quality of instruction, as classrooms become overcrowded and student to faculty ratios exceed levels conducive to optimal learning. Therefore an accurate estimate of the enrollment yield is essential to effective fiscal planning. A second critical decision for the CLA admission staff is the allocation of financial aid to admitted students. All universities offer financial aid to a large proportion of their incoming students, both as a means of meeting students' financial need and as a recruiting tool. Financial aid allocations provide a powerful lever for admissions, but these decisions have major fiscal implications as well. Once the admission office has identified a set of students to admit, the admissions staff conducts an assessment of financial need and merit to determine how much aid to offer each admitted student. Prior to sending out the final admission letters and financial aid packages the admission office must estimate the discount rate, defined as the percentage of the total tuition which is offered to the enrolled class in the form of financial aid. From a fiscal point of view, operational performance of the admission office is assessed based on the accuracy of both the predicted enrollment yield for the admitted class and the discount rate associated with the admitted applicants who actually enroll. 3.1. The traditional process for estimating yield and discount rate The enrollment management process traditionally begins by establishing targets for enrollment and the discount rate for the incoming class (see Fig. 1). Consultations begin in the summer prior to the year when the admit decisions must be made to establish enrollment and discount rate targets. For instance targets are established in summer of 2005, for admission decisions to be made in the spring of 2006 that result in an entering class in the fall of 2006. These discussions between the Dean's office, the admission office, the President and the VP of Finance attempt to balance the long-term strategic goals of the college (e.g., academic quality, geographic and ethnic diversity of the student body) with the fiscal implications of attempting to achieve those goals. Once enrollment and discount rate targets are set, the information is sent to an outside consultant who returns a suggested financial aid allocation strategy for achieving these goals. This strategy is embodied in a

109

Table 1 Example of financial need-academic quality grid a Academic rank

Need rank

1 2 3 4 5 6 7

1

2

3

4

5

$0 $0 $0 $1000 $2000 $5000 $10,000

$0 $0 $1000 $2000 $5000 $8000 $12,000

$0 $1000 $2000 $5000 $8000 $10,000 $15,000

$1000 $2000 $5000 $8000 $10,000 $12,000 $18,000

$2000 $5000 $8000 $10,000 $12,000 $18,000 $22,000

a Data in this table are for illustrative purposes only. Actual allocations vary from year to year and are proprietary.

grid corresponding to seven levels of financial need and five levels of academic quality. The grid in Table 1 provides an example of a recommended financial aid figure for each level of need and academic quality, which is typically received from the consultant no later than the middle of October.1 By February 1, all applications have been received. The more than 3000 applicants are reviewed to make decisions on the obvious candidates for admission or denial. Preliminary decisions are based primarily on academic credentials (e.g., GPA, classes taken, test scores). At this initial stage of selection, other factors, including student background, interests and activities play a lesser role. Approximately 20% of the applicants who have good but not outstanding credentials are then subject to a subsequent review. In this subsequent stage of selection, the admissions staff confers to make final decisions as to which students will be admitted, denied admittance or entered onto a wait-list. Once the final admit list is determined, admitted applicants are classified by need and academic rank, into the squares of the Need-Academic Quality grid and counts are determined. The grid, along with the data on admitted students and the total financial aid budget is then sent to the consultant who uses their proprietary models to estimate the enrollment yield and discount rate for the admit pool. The results of the consultant's analysis are returned to the admissions office and, if the estimates of yield and discount rate do not meet preselected targets, the consultants offer advice on how the dollar values in the grid might be altered to improve the results. Based on this advice, the CLA admission office makes final financial aid allocations for each cell of the grid. Admitted applicants are then sent offers of 1 In November, the first of the applications are received. These applications, identified as early-decision applications are for prospective students who have identified Willamette as their first choice.

110

E.N. Maltz et al. / Decision Support Systems 44 (2007) 106–123

admission, including financial aid awards determined by their position in the grid. Admission office personnel wait for students to either accept or decline admission which is indicated through personal contact or receipt of a deposit from the student. As acceptances and declines arrive at the admission office, progress is evaluated based on historical trends to determine whether developing yields appear to be on target. If deposits arrive at a rate lower than anticipated, the admission staff may turn to the wait-list to admit additional students. If acceptances are higher than

anticipated, the university faces the prospect of being substantially above targets for enrollment, discount rate, and total financial aid budget. 3.2. Drawbacks of the traditional admissions process Several shortcomings are apparent in the traditional CLA enrollment management process. First because the consultant's work utilizes proprietary models, admissions staff gains limited explicit knowledge into what factors are most important in influencing enrollment.

Fig. 2. The CRISP Methodology [5].

E.N. Maltz et al. / Decision Support Systems 44 (2007) 106–123

Second, the consultant's model is based upon assumptions that are used to build models for a variety of clients. Thus, forecasts may not adequately account for idiosyncratic differences of the CLA. The model used is built based simply on the previous year's model and modified to incorporate, to a limited degree, any shift in strategic goals. Finally and perhaps most significantly, because the analysis is performed by an outside agency, admission personnel are limited in their decision making by the timing and scope of the information provided by the consultant. The admission process has become much more fluid and unpredictable due to the sophistication of applicants who are likely to research and apply to multiple institutions over varying time periods [14,18]. This behavior leads to significant shifts in the make-up of the applications pool even within a specific year. As new applications are received the admission office must make intermediate decisions with respect to the admitted pool, and changes in the admit pool requiring ondemand model adjustments. While the consultant would typically provide updates on request, the timeliness of these updates is dependent on the consultant's workload at the time of the request. Since the consultant has multiple clients all with similar admission decision calendars, the CLA admission office often did not get the information when it was needed to make timely decisions. The combination of the limited incorporation of the CLA specific factors and lack of timely updates led to a loss of control of the enrollment management process at CLA resulting in poor operational performance. This, in turn, led to the need for and development of an in-house DSS. As will be described in the sections that follow, the

111

resulting system resulted in dramatically improved operational performance and increased institutional knowledge. 4. Building the predictive enrollment model for CLA Data mining is the process of discovering trends and usable patterns in data. The objective of this process is to sort through large quantities of data to extract new information [16]. Data mining models are built guided by key outcomes desired by the users of the model (e.g., accurate yield prediction,) and what the data suggest are the key factors relating to the outcome. A model deployed via data mining may often rely upon a mixture of traditional statistical techniques (e.g., logistic regression,) in combination with standard data mining techniques discussed below. The system developers, in this case professor and graduate students, followed the CRISP paradigm for data mining projects [7]. The paradigm suggests six steps to developing successful data mining models. The developers involved closely followed the steps from Institutional understanding, Data Understanding, Data Preparation, Modeling to Evaluation and Deployment. Fig. 2 presents a broader description of these steps and Fig. 3 shows the two-year development process actually used at the CLA. 4.1. System development — 2002–2003 Work began on the system in 2002 as part of a graduate data mining course being developed at the university. The developers agreed to, over a three-year period, build a DSS consisting of: (1) a predictive model

Fig. 3. The CRISP Process Applied to the Admissions Management DSS Project.

112

E.N. Maltz et al. / Decision Support Systems 44 (2007) 106–123

to project, for individual applicants, the likelihood of enrollment and, in aggregate, the enrollment yield and discount rate; and (2) a managerial tool with a userfriendly interface that could be used to provide timely and effective guidance for policies and decisions surrounding admissions and financial aid. In the fall of 2002, the following business goals were identified: annually enroll approximately 500 students with the highest academic quality possible; achieve diversity goals; maintain a discount rate at or below the target level. Following the CRISP methodology model, development began with interviews with enrollment managers to gain a better understanding of the institutional setting (i.e., develop business and data understanding). This also provided opportunities for institutional knowledge creation as admissions personnel became more familiar with data that would be used to drive the models. Once data preparation was complete and the initial analysis database was constructed, the model building stage commenced. The initial analysis data set consisted of applicant records for thousands of applicants from three preceding admissions cycles, 2000–2002, and comprised over 60 variables, 40 of which were suggested by enrollment managers. The remaining variables were chosen from among those available but not initially considered important by these same managers. The data set was split equally into training and validation sets. All model development was conducted using training data. As is typical of data mining initiatives, meta-level modeling was employed using a multiplicity of approaches – neural networks, decision trees, and logistic regression models – to arrive at the ultimate predictive model and tool. Because neural networks have the ability to accurately predict outcomes in complex problems ([9] p. 64), and because neural network models were found in a previous study to outperform other techniques in correctly classifying admitted applicant who will ultimately enroll or not enroll [33] modeling began with neural networks. All available predictors were included in a neural network model to: lend insight as to the most influential variables and to set initial benchmarks for the predictive accuracy that might reasonably be attainable from the available data. In this way, the goal of correctly predicting the ultimate enroll/decline decision for admitted students 71% of the time was established for any predictive model. Using the relative importance of inputs data reported the original collection of more than 60 candidate predictor variables included in the neural network model was narrowed to 30 as input for the logistic regression model building process. Because it is

difficult to determine the exact relationships being modeled in neural network approaches and because this limited model transparency for the managers, the decision tree and logistic regression approaches were employed to determine if similar predictive accuracy could be achieved. In contrast to regression-based approaches, decision trees offer potentially attractive modeling alternatives as they do not rely upon assumptions about the linearity relationship between the response and selected predictor variables, nor does their interpretation suffer from correlation among the predictor variables. Thus, in theory, decision trees potentially promise greater predictive accuracy and simpler interpretation. In the course of the modeling process, decision tree models based on C5.0 and C and R Tree algorithms2 were developed, both as additional benchmarks of predictive accuracy and to lend additional modeling insights to further modeling development. The best of the tree-based models exhibited an overall predictive accuracy of 70.3%, comparable to that attained via neural networks. However, from the standpoint of the enrollment management application, decision tree models have a significant drawback, producing only discrete breakpoints to describe the influence of financial aid on applicant propensity to enroll. CLA managers, instead, required a tool that would allow them to assess the impact of relatively small adjustments to financial aid policies and individual award packages. Moreover, the decision tree models could not be easily deployed in a software package available to the managers. With these factors in mind a logistic regression was considered to determine if a model with similar predictive accuracy could be developed. Logistic regression offered the potential to provide insights offered into the relative strength and effect of individual predictors and in particular the ability to smoothly assess the impact of financial aid allocations. It was also easily deployed in the form of a Microsoft Access or Excel-based decision support tool. These factors together maximized the ease of understanding and implementation by enrollment managers. The modeling process began by exploring main effects that would ultimately provide a parsimonious description with a reasonable degree of fit. A model consisting of eight variables – including applicant characteristics and financial aid allocations – was ultimately selected to predict the likelihood of enrollment. It accurately predicted enroll/decline decisions for a little over 70% of the applicants in the training set. To test the predictive 2 See David Hand, Heikki Mannila and Padhraic Smyth, Principles of Data Mining, MIT Press: Cambridge 2001, pages 327–367 for a broader discussion of these models.

E.N. Maltz et al. / Decision Support Systems 44 (2007) 106–123 Table 2 Enrollee classification matrix (Overall 70%) Predicted 2003 Actual 2003 Enroll Enroll Decline

Decline

25% (Goal is to maximize) 12% (Goal is to minimize) 75% (Goal is to minimize) 88% (Goal is to maximize)

accuracy, the model was then scored using the validation data set, and accurately predicted enroll/decline decisions 68% of the time, only marginally lower than training set results. The model was further evaluated over the summer of 2003 based on the actual enrollee data from 2003, and again, the model performed well, accurately predicting 70% of the time. Consultation between managers and analysts revealed one principal concern. While the overall predictive accuracy of the model was acceptable the model did a much better job of predicting who would not enroll than those who would enroll (see Table 2). This finding is not surprising because the model is designed to maximize the probability of correctly classifying any individual and approximately 70% of those admitted decline to enroll. However, for CLA, it is more important to predict which applicants will enroll than those who won't. Thus, it was agreed that future models would place a greater emphasis on predicting those who would actually attend. One of the key pitfalls of data mining models is that they often may be too complex for managers to interpret and use. To address this issue, during summer of 2003, a user-friendly deployment tool was developed to allow the admission personnel to make use of the predictive model with minimal understanding of the underlying model. This interface was developed in the Microsoft Access database environment with the focus on maximizing ease of use for the managers. Functionally, the interface allowed managers to predict total yield and discount rate at the aggregate level. It also allowed managers to assess the likelihood that any individual admitted applicant would attend. This enhanced the transparency of the process for CLA managers and afforded them the opportunity to react in real time to shifts in strategy and/or the enrollment environment. However, because of only limited understanding of the Microsoft Access database environment among enrollment managers, this interface did not allow the managers to easily view components of the underlying models, limiting transparency to some degree. In the fall of 2003, the CLA admission office began using the initial DSS to guide enrollment management. Because of the instant access to model results, the admission office was able to generate timely initial estimates of yield and discount rate based on the inputs

113

to the earlier mentioned financial aid matrix. Combined with the ability to predict enrollment on an individual basis, this provided the opportunity to better manage the discount rate by allowing the admission office to fine tune financial aid allocations to the individual level and optimize yield while still maintaining desired academic and diversity profiles. Thus, as will be discussed in the next section, operational results improved significantly. Further, the initial DSS development process yielded significant enhancements to institutional knowledge. The inclusion of certain variables in the yield model suggested others that should be considered for inclusion in subsequent analysis. In addition, improved data understanding suggested that a more comprehensive predictive model incorporating interactions between variables might be useful in increasing accuracy. In terms of the interface, while predicting at an individual level was thought to be useful for managing enrollment yield and discount rates, it was found to be prohibitively time consuming. To address this, managers suggested that the interface be modified to provide a vehicle to break out projected enrollments by geographic region to support strategic CLA geographic diversity initiatives. Finally, the manager of institutional research suggested that the interface be provided in a format that would allow easy modifications to the underlying model in order to support additional analyses as will be described below.

Fig. 4. Decision trees for interaction detection. A: interaction between geography and aid award. B: interaction between geography and campus visit.

114

E.N. Maltz et al. / Decision Support Systems 44 (2007) 106–123

4.2. System refinement 2003–2004 The second year of DSS development began by incorporating a number of new variables into the database and reformatting others to make the model development process more efficient. The total time devoted to understanding and setting up the data for the second stage of modeling was reduced from eight months to four. The modeling phase for the second year began with the investigation of interactions between predictors suggested by the 2002–03 model development process. Four years of data (2000–2003 admits) were used to identify highly correlated variables (via the use of plots and correlation matrices) and highly correlated cases (through the use of cluster analysis). In terms of the highly correlated cases, the team searched for large groups of applicants with similar characteristics that appeared to have an unusually high or unusually low propensity to attend. When these groups were identified, indicator variables were created to capture group membership information. Following data exploration, the full data set was again split randomly into two equal parts and the training set was used to develop a revised logistic regression model for deployment. Potential extensions of the 2002–03 model were investigated including using variable interaction terms determined via additional meta-modeling. The refinement work began by executing several decision tree modeling routines on the training data set to identify potential interaction terms for inclusion. The process of interaction detection was as follows: a decision tree model with the outcome variable ▸ First, enroll/decline was created. This model included:

▸

○ All the variables included in the 2002–03 model. ○ Any variables suggested by managers in working with the model in 2002–03. ○ Flag variables to identify the effects of being a member of 2 groups suggested in the exploration phase described above. The output of the decision tree was examined to determine if any of the most important variables suggested by the tree differed at lower levels based on additional variable included in the model. For example:3 ○ In Fig. 4A admitted students from Oregon were somewhat more likely to enroll (55% likelihood of enrollment). However, if they were promised

3 Examples include actual significant interactions. However, the magnitude and influence of these interactions, as measured by the actual estimated regression coefficients, is proprietary.

high levels of financial aid (in this case more than $10,000) they were much more likely to enroll (75% likelihood of enrollment). However, outof-state applicants did not show a similar increase in propensity to enroll at similar levels of financial aid. This might suggest an interaction between Oregon residence and aid provided. ○ In Fig. 4B, admitted students from outside Oregon were not very likely to enroll (22% chance of enrollment). However, out of state applicants who also visited campus were much more likely to enroll (55% chance of enrollment). For Oregon applicants, the likelihood of enrollment was relatively high whether they visited campus or not (55%). This suggests an interaction between non-residents and campus visits. The main effects (variables from year 2002–03 model, new variables suggested by managers based on their learning from the first year, new flag variables suggested in the exploration stage) and interaction effects (suggested by the decision trees) were included in a preliminary logistic regression model. The new model initially consisted of 18 variables (see Table 3 for the set of predictors used in the model.) The predictors of enrollment probability included entrance scores, high school grade point average, geographic origin, the Table 3 Main effect predictors in the model Predictor Hsgpa Othersch

Description

High School GPA Number of other schools to which the applicant applied based on FAFSA information applied Need grant Amount of aid award based on need Merit grant Additional award based on merit Workstudy Promised amount of work dollars SATSOFT A measure of SAT and/or imputed SAT based on ACT score Sex Gender of the applicant Apptype Early admit or Regular admit Alum Were the applicant's parents alumni? Appfacstaff Were the applicant's parents faculty or staff? Visit Had the applicant visited campus? High school Public or private type Comp1 Had the applicant applied to an identified competitor based on FAFSA information? Comp2 Had the applicant applied to an identified competitor based on FAFSA information? Comp3 Had the applicant applied to an identified competitor based on FAFSA information? Territory What part of the country was the applicant from? Need rank Need rank on the grid Merit rank Merit rank on the grid

E.N. Maltz et al. / Decision Support Systems 44 (2007) 106–123 Table 4 Enrollee classification matrix-training data set (Overall 69.2%) Predicted 2000–2003 Actual 2000–2003

Enroll

Decline

Enroll

65.7% (Goal is to maximize) 34.3% (Goal is to minimize)

27.3% (Goal is to minimize) 72.7% (Goal is to maximize)

Decline

financial need of applicants as well as grant, scholarship and loan amounts in the financial aid package.4 Prior to commencing the final model selection process, the training data set was balanced to reduce the model's tendency to over-predict the decliners. Many successive iterations of the logistic regression model were investigated before arriving at a final model with a prediction accuracy of 69.2% detailed in the Table 4 below. Examining the table reveals that by including selected interactions and balancing the data, the ability to successfully predict those admitted applicants who would actually enroll, as per management requirements, increased substantially from 25% to 65.7%. The 2003–04 predictive model was submitted to test–retest validation, and while the results of this test were somewhat lower than those observed in the test set (see Table 5 below), they were deemed consistent with respect to the training data set. This conclusion is further supported by the observation that the same variables were statistically significant in both data sets and that the predictive validity was similar. The DSS interface was also altered substantially from the first year to enhance functionality and ease of use. Specific changes included:

▸ ▸

▸

The DSS was re-deployed in Microsoft Excel. This approach permitted the institutional research manager to easily develop auxiliary tools to better inform the enrollment management process. The revised DSS incorporated a new screen where the managers could modify financial aid allocations for each cell in the grid, to assess the effect on yield within particular cells and overall. For example, in Fig. 5A, the amount of financial aid given a student deemed a “3” on the academic quality scale and a “4” on the financial need scale is $3000, yielding 25%. Fig. 5B shows changing the amount of financial aid in the cell to $5000 increases the projected enrollment yield to 32.7%. The revised DSS includes a screen which breaks out the expected yield by in-state and out-of-state

▸

admits. This feature was introduced to support CLA's strategic goal of geographic diversity. In addition to providing cell based results, the screen also provides aggregate level predictions of total financial aid outlays and the discount rate.

The DSS continued to include a screen that allowed managers to assess individual applicant level probabilities of enrollment. Users simply enter actual data values for the required variables and the interface displays the predicted enrollment probability for an individual applicant. If desired, the manager can then experiment with alternative financial aid awards to increase the probability of an applicant enrolling (See Fig. 6A and B). 5. Organizational impacts on the admissions work system realized with the DSS The implementation of the DSS resulted in superior operational performance, and perhaps even more importantly, the modeling and system development activity provided a number of learning opportunities for CLA admissions office and the internal developers. The most significant indicator of the impact of the new DSS and its associated implementation activities was that it substantially and beneficially altered the process by which admissions activities are carried out. Using the knowledge acquired through the project and the resulting system, the admission staff altered the sequence, effectiveness and expediency of enrollment decision making (See Fig. 7.) 5.1. An improved enrollment management process Through direct participation in the model development, the CLA admissions staff gained new insight into their own process, insight that would never have emerged under the traditional approaches employed by outside consultants. For example, admissions staff was now able to create an initial financial aid allocation grid, demonstrating the admissions decisions makers' explicit understanding of what may formerly have been only tacit knowledge. Managers now understand how Table 5 Enrollee classification matrix-validation set (Overall 66.8%) Predicted 2000–2003 Actual 2000–2003

Enroll

Decline

Enroll

61.0% (Goal is to maximize) 39.0% (Goal is to minimize)

30.4% (Goal is to minimize) 69.6% (Goal is to maximize)

Decline 4

The actual model is proprietary.

115

116

E.N. Maltz et al. / Decision Support Systems 44 (2007) 106–123

financial aid choices affect discount rate and can capitalize on previous experience as well as an improved understanding of process goals, resulting in more precise estimates of enrollment yield and discount rate. The DSS reduces the time spent by admissions staff on the grid construction and assessment, increasing the time available to manage individual cases. Under the old process, almost all evaluation of financial aid allocations was conducted at the grid level and individual applicant level modifications were limited. Under the new process, up to 50% of the individuals admitted receive applicant-specific adjustments to their financial aid package, and the estimated impact of these adjustments is immediately assessed. The number of adjustments to the financial aid grid and the level of attention to directed towards individual financial aid packages would not have been possible without the analytical support of the DSS tools or the in-depth knowledge gained through the implementation process.

In addition more precise estimates of admission yield, enrollment predictions at the individual level support better estimates of incoming class characteristics – expected revenue and class profile with respect to academic quality, geographic and ethnic diversity. For individual applicants with given academic quality indicators, geographic origin and financial need, sensitivity curves can developed for varying levels of financial aid. As grants and scholarships increase while, correspondingly, loans decrease as a proportion of the total aid package, the probability of enrollment increases. Analyzing groups of admitted applicants that are homogeneous with respect to academic quality and financial need, financial aid sensitivity curves can also be obtained, allowing for the identification of aid thresholds above which the probability of enrollment moves a group into the “likely to enroll” range. This information is used to advise CLA admission policy and to inform the distribution of financial aid.

Fig. 5. A: financial need-academic quality grid. B: revised financial aid example in financial need-academic quality grid.

E.N. Maltz et al. / Decision Support Systems 44 (2007) 106–123

117

Fig. 5 (continued ).

5.2. Real-time management of enrollment during the acceptance period Once offers of admission are sent to applicants, these potential students have approximately one month to accept or decline. Admissions staff can use the system to rank students based on the model's estimate of their probability of enrolling. Once enrollment accept/declines begin coming in, the managers can use the system's interface to track how well the model predicted actual enrollment decisions. If a trend is detected where enrollment acceptance rates are coming in below projections, admissions managers can move quickly to start making offers from the wait-list. In addition, they can observe acceptance trends in financial aid, and gain insight into how much additional financial aid incentives might be available to influence admitted students who have not yet responded. In this way, the model and system interface support real-time enrollment management as the early results unfold.

5.3. Shifts in responsibilities of admissions managers The introduction of the interface resulted in several new responsibilities for the managers that were formerly performed by the outside consultant. Shifts in responsibilities are summarized in Figs. 1 and 7, depicting the traditional and revised enrollment processes. These include: ➢ The initial financial aid grid, which serves as the foundation for the overall strategy, was previously developed by the outside consultant, largely on the basis of a single year of data. The revised process relies upon CLA top management using knowledge that they have accumulated through the in-house development process. ➢ The new interface allows managers to systematically analyze individual applicants, resulting in more individual adjustments to aid awards. Thus, the DSS and supporting interface is driving a migration from grid level analysis to individual level analysis.

118

E.N. Maltz et al. / Decision Support Systems 44 (2007) 106–123

Fig. 6. A: individual prediction 7of probability of enrolling. B: individual prediction7 of probability of enrolling with increase in aid.7

➢ The lead time required when working with the outside consultant was significantly reduced. Previously, it was difficult to determine in a timely fashion, if enrollment projections for enrollment developing according to plan. The in-house DSS interface allows for more rapid adjustment to unexpected market conditions, permitting enrollment decision makers to manage their wait list more strategically. 5.4. Operational results The operational results attributable to the revised data-driven process for enrollment management (see Table 6) are quite impressive, especially relative to those realized in the two years preceding implementation. The first row of Table 6 illustrates the percentage variance between the targeted and actual enrollment. In the preceding years, the actual enrollment was 17–21%

above or below desired enrollment.5 In 2004–05, using the new system, the variance was less than 5%. Variance from the target discount rate was reduced from 2–3.5% under the consultant to less than 1%.6 Moreover, a 1% reduction in the discount rate can yield hundreds of thousands of dollars in additional revenue to the 5

Note that while, on the surface, it may seem beneficial to enroll more students than planned because it will result in more revenue. However, over-enrollment is problematic on two dimensions. First, when a school enrolls significantly more students than anticipated, the costs of housing and other ancillary costs increase disproportionately as the school is forced to look outside its fixed set of assets, resulting in significantly higher cost per student. Second, because the preceding year's enrollment had been so far below expectations, the admission office increased the financial aid awards substantially in 2003. The higher discount rate resulted in substantially lower revenue per student. These two factors combined to produce significant adverse impact on university cash flows. 6 Actual targets for enrollment and discount rate are proprietary. 7 Note that variable names are concealed here for proprietary reasons, but are transparent in the actual interface.

E.N. Maltz et al. / Decision Support Systems 44 (2007) 106–123

119

Fig. 6 (continued ).

university. From 2003 to 2005, using the new DSS, the CLA was able to reduce the discount rate by over 10%. These results were achieved without any decline in academic quality (as indicated by SAT scores in Table 6.) There was a precipitous drop in ethnic diversity in 2004 and hence, ethnic diversity was identified as a point of emphasis in developing the underlying enrollment forecasting for the subsequent year. In year four, 2005, ethnic diversity rebounded along with significant gains in geographic diversity, with 4.6% more students from outside Oregon and 5.8% more students from outside the northwest region, while median SAT declined. 5.5. Costs of DSS implementation The CLA paid the University's school of management a fixed yearly fee to support the purchase of specialized software, and a faculty member's time for project oversight. During the development period, total monetary outlays

were less than $50,000. As noted above, a 1% reduction in the discount rate yields hundreds of thousands of dollars in additional revenue to the university and the discount rate decreased 10% over the three-year implementation period. Thus, from a purely monetary cost-benefit perspective the implementation was hugely successful. Other costs that are more difficult to quantify relate to the time that CLA management spent in: 1) educating project participants (graduate students) in enrollment business procedures; 2) providing input on interface development to achieve both maximum functionality and ease of use; 3) learning how to assimilate DSS tools into business processes. Initial time costs of education and development were substantial in the first year of the project but declined dramatically in years two and three. However development costs also yielded intangible benefits equally difficult to quantify. The activities associated with DSS development and the use of the DSS itself resulted in immeasurable gains in process knowledge

120

E.N. Maltz et al. / Decision Support Systems 44 (2007) 106–123

Fig. 7. The revised CLA admission process.

and significant process improvements. The DSS implementation project assists enrollment managers in making more precise financial projections and allows for significant gains in real-time response to shifts in the enrollment environment. All of these factors, led to improved enrollment management as well as major strides in class quality, diversity, and financial performance. Table 6 Enrollment outcomes for new freshmen Outcome

2002

2003

2004

2005

Enrollment variance (actualtarget)/target Tuition discount variance (actual-target)/target SAT median Ethnic minority representation Non-Oregon representation Non-northwest representation (students from outside, Washington, Montana, Idaho, or Wyoming)

− 16.9% +21.1% +2.5% − 4.4% − 3.5%

+2.0%

− 0.5% − 0.4%

1230 20.8% 59.8% 32.5%

1250 19.2% 59.7% 34.2%

1260 15.4% 61.4% 34.4%

1230 18.0% 66.0% 40.2%

6. Insights on the implementation of DSS systems Project outcomes detailed in the preceding section provide guidance for managers developing and implementing DSS systems in enrollment work systems. This section considers the general contributions of the project to the DSS implementation literature. In terms of the frameworks provided in [17] and [32], implementation insights from the enrollment management DSS support the necessity of top management and organizational support, in particular a culture accepting of knowledge sharing. Both the Vice President for Enrollment and the University President, spurred by the lackluster performance of the traditional process, were strong advocates for process change. This permitted internal developers initial access to data and personnel that would not otherwise have been available. Over time, the internal developers were able to gain the deep enterprise and data understanding that are crucial to any successful DSS development project. It was also noted in [3,11,17,29] that top management support was a key to

E.N. Maltz et al. / Decision Support Systems 44 (2007) 106–123

success in the development of an admissions DSS with respect to “opening doors”. The internal developers spent a great deal of time educating the admissions staff to the potential of various solutions alternatives. The decision to use a logistic regression model as the fundamental predictive tool was one result of this interaction. Over the duration of the project, the internal developers and the admissions staff built up trust, resulting in the reciprocal knowledge sharing culture that has been suggested as a key success factor [3,15,17,32]. Further, the transparency of the DSS interface further supported the knowledge sharing culture, especially during the second year when it was redeployed in Microsoft Excel. By providing a mechanism whereby non-technical users and technical users could collaborate and share ideas, the interface promoted a free flow of knowledge that led to significant enhancements. Project leadership and organization (during implementation) are other “people” factors generally believed to be critical to the success of a DSS implementation project [3,15,17,32]. Given that the number of project stakeholders was relatively small, fewer than ten, project leadership and team composition were perhaps less important than observed in larger project settings [11,32]. Moreover, given the environmental factors (poor historical performance) and the clearly articulated goals, individuals on both sides (admissions and internal developers) were highly motivated to succeed. As is typical of many DSS and data mining projects, the admissions DSS was implemented in two phases and continues to be enhanced annually. Similar to the examples in [11] and [32] components were added over time, with more sophisticated functionality added in the later phases of the project. The key observation here is that learning by DSS implementers must continue to occur even after initial implementation. For example, user feedback from the 2002–03 year helped drive interface movement from MS Access to MS Excel, improving system functionality and transparency. Overall, the gradual implementation of new complex functionality is preferred to all at once rollout [11]. System characteristics, and in particular content management, was central to the successful implementation of a knowledge management DSS for Singapore's housing department [32]. In the enrollment management project, content is not as directly relevant to success as is the management of the predictive model's quality. The model is currently reevaluated and updated annually by the internal developers with ongoing performance evaluation. In this sense, the project requires continual attention from both the internal developers and admissions staff. As observed elsewhere, incentives are a requirement for

121

ongoing participation [15,17,32]. In this case admissions managers provide their insight in order to improve performance gains with respect to their activities. The internal DSS developers continue to view the system as a learning opportunity and real-life experience, both for the professors and participating graduate students. At the heart of many DSSs is the technological platform upon which the system is built. The enrollment management model was developed using Clementine, the SPSS data mining product, but the end-user sees only the final Microsoft Excel deployment. Deployment migration from Microsoft Access to Microsoft Excel was based on the admissions staff desire for instantaneous feedback on the impact of grid adjustments that was not so immediately achievable in the initial database application. This also supports the contention of many [11,29,32] that choosing technology with minimum training and/or providing effective training for the technology chosen can enhance information flows in the organization. 7. Conclusion and further research DSS design and implementation is, at its core, an iterative activity. Data mining procedures lend themselves well to data intensive DSS implementations as additional data and new modeling approaches can be readily incorporated. In any case, properly built DSSs require regular user interaction and trust, both from enduser and designer perspectives. Effective implementation processes promote knowledge sharing through business and data understanding phases and favors deployment mechanisms that provide for the capture and free flow of tacit knowledge between managerial and technical project personnel. Enrollment management essentially requires decisions on which students to admit and what price to charge for each available slot in the university; in order to maximize student quality; with constraints on capacity, discount rate, and target demographic composition of the admitted class. Optimization solutions for seemingly related yield or revenue management problems in the airline and hospitality industry have been broadly investigated [4,20–22,31]. However, rather than optimizing on a financial objective, enrollment managers seek to maximize quality, long-term customer value, subject to financial constraints, capacity and discount rate. Continuing research will seek to more systematically address optimization objectives and incorporate optimization tools into a further improved DSS. This paper presents an example of successful DSS design and implementation to improve the enrollment

122

E.N. Maltz et al. / Decision Support Systems 44 (2007) 106–123

work system at a small liberal arts college. Comprised of two interrelated components, a predictive model and a user friendly deployment tool, the DSS and the associated implementation have significantly improved financial performance in enrollment. More importantly, the two-year implementation yielded dramatically enhanced understanding of the enrollment work system and serves as a vehicle for the conversion of tacit process knowledge into readily deployable explicit understanding. References [1] M. Alavi, D.E. Leidner, Review: knowledge management and knowledge management systems: conceptual foundations and research issues, MIS Quarterly 25 (1) (1999) 107–136. [2] S. Alter, A work system view of DSS in its fourth decade, Decision Support Systems 38 (2004) 319–327. [3] D.S. Bajwa, A. Rai, I. Brennen, Key antecedents of executive information systems success: a path analytic approach, Decision Support Systems 22 (1) (1998) 31–43. [4] P.P. Belobaba, Airline yield management: an overview of inventory seat control, Transportation Science 21 (2) (1987) 63–73. [5] B. Bontrager, Strategic enrollment management: core strategies and best practices, College and University Journal 79 (4) (2004) 9–16. [6] A. Braunstein, M. McGrath, D. Pescatrice, Measuring the impact of income and financial aid offers on college enrollment decisions, Research in Higher Education 40 (3) (1999) 247–259. [7] P. Chapman, J. Clinton, R. Kerber, T. Khabaza, T. Reinartz, C. Shearer, R. Wirth, CRISP_ DM 1.0, CRISP-DM Consortium, 2000. [8] W.H. Delone, E.R. MacLean, Information system success: the quest for the dependent variable, Information Systems Research 3 (1) (1992) 60–95. [9] M. Dunham, Data Mining: Introductory and Advanced Topics, Pearson Education Inc, Upper Saddle River, New Jersey, 1993. [10] J.S. Edwards, J.L. Bader, Expert systems for university admissions, Journal of the Operational Research Society 39 (1) (1988) 33–40. [11] A.A. Elimam, A decision support system for university admission policies, European Journal of Operational Research 50 (2) (1991) 140–156. [12] P.N. Finlay, M. King, Experiences in developing an expert system for MBA admissions, Journal of the Operational Research Society 40 (7) (1989) 625–635. [13] G.H. Gaither, F.O. Dukes, J.R. Swanson, Enrollment forecasting: use of a multiple-method model for planning and budgeting, Decisions Sciences 12 (2) (1981) 217–230. [14] K.M. Galoti, M.C. Mark, How do high school students structure an important life decision? A short-term longitudinal study of the college decision making process, Research in Higher Education 17 (1994) 589–607. [15] M.J. Ginzberg, Early diagnosis of MIS implementation failure: promising results and unanswered questions, Management Science 27 (4) (1981) 459–478. [16] R. Groth, Data Mining: Building Competitive Advantage, Prentice Hall, New Jersey, 1999. [17] E. Hartono, R. Santhanam, C.W. Holsapple, Factors that contribute to management support systems success: an analysis of field studies, Decision Support Systems 43 (1) (2007) 256–268.

[18] D. Hossler, J. Schmit, N. Vesper, Going to college: how social, economic and educational factors influence the decisions students make, The Johns Hopkins University Press, Baltimore, 1999. [19] J.V. Iyengar, An expert system for MBA admissions, Journal of Computer Information Systems 36 (1995). [20] V. Liberman, U. Yechiali, On the hotel overbooking problem, Management Science 24 (11) (1978) 1117–1126. [21] K. Littlewood, Forecasting and control of passenger bookings, AGIFORS Proceedings, vol. 12, Nathanya, Israel, 1972. [22] J.I. McGill, G.J. Van Rizyn, Revenue management: research overview and prospects, Transportation Science 33 (2) (1999) 233–256. [23] C.A. Molinero, M. Qing, Decision support systems for university undergraduate admissions, Journal of the Operational Research Society 41 (3) (1990) 219–228. [24] J.S. Moore, An expert system approach to graduate school admissions decisions and academic performance prediction, Omega International Journal of Management Science 26 (5) (1998) 659–670. [25] C. O'Dell, S. Elliot, C. Hubert, Achieving knowledge management outcomes, in: C. Holsapple (Ed.), Handbook on Knowledge Management: Knowledge Directions, Springer, Heidelberg, 2003. [26] M.B. Paulsen, College choice: understanding student enrollment behavior, ASHE-ERIC Higher Education Report No. 6, George Washington University, School of Education and Human Development, Washington, DC, 1990. [27] J.A. Pope, J.P Evans, A forecasting system for college admissions, College and University Journal 60 (1985) 113–131. [28] G. Rath, Management science in university operation, Management Science 14 (6) (1968) B373–B384. [29] R. Santhanam, T. Guimaraes, J. George, An empirical investigation of ODSS impact on individuals and organizations, Decision Support Systems 30 (1) (2000) 51–72. [30] R. Schroeder, A survey of management science in university operations, Management Science 19 (3) (1973) 895–906. [31] B.C. Smith, J.F. Leimkuhler, R.M. Darrow, Yield management at american airlines, Interfaces 22 (1) (1992) 8–31. [32] T.S.H. Teo, Meeting the challenges of knowledge management at the housing and development board, Decision Support Systems 41 (1) (2005) 147–159. [33] S. Walczak, T. Sincich, A comparative analysis of regression and neural networks for university admissions, Information Sciences 119 (1/2) (1999) 1–20. [34] G.P. White, A survey of recent management science applications in higher education, Interfaces 17 (2) (1987) 97–108. [35] Y. Yoon, T. Guimares, Q. O'Neal, Exploring the factors associated with expert system success, MIS Quarterly 19 (1) (1995) 83–106. Elliot Maltz received his Ph.D. in Marketing from the University of Texas at Austin. Dr. Maltz's current research focuses on effectively transmitting and combining market information to facilitate new product development and respond to changes in market conditions. He has published in the Harvard Business Review, Journal of Marketing, Journal of Marketing Research, The Journal of the Academy of Marketing Science, and Sloan Management Review.

E.N. Maltz et al. / Decision Support Systems 44 (2007) 106–123 Kenneth Murphy holds a Ph.D. in Operations Research from Carnegie Mellon University. Dr. Murphy's work on integrated systems has followed several threads including the financial justification of large-scale systems using both tangible and intangible factors, and investigating the tools and methods for successful system implementation. He has published in Operations Research, Communications of the ACM, and the Information Systems Journal among others.

123

Michael L. Hand is a Professor of Applied Statistics and Information Systems. Dr. Hand has been widely recognized for his distinguished teaching and is a two-time recipient of Willamette University's highest teaching award. Professor Hand is the coauthor of a leading college statistics textbook and is the author or coauthor of a number of scholarly articles in statistics and statistical computing, both basic and applied. He is an experienced management consultant, with clients including — Hewlett Packard, Safeco Corporation, the State of Arizona, and most major agencies of the State of Oregon.