- Email: [email protected]
A grid-based approach for enterprise-scale data mining
Future Generation Computer Systems 23 (2007) 48–54 www.elsevier.com/locate/fgcs
A grid-based approach for enterprise-scale data mining Ramesh Natarajan a,∗ , Radu Sion b , Thomas Phan c a IBM Thomas J. Watson Research Center, P.O. Box 218, Yorktown Heights, NY 10598, United States b Department of Computer Science, State University of New York, Stonybrook, NY 11794, United States c IBM Almaden Research Center, 650 Harry Road, San Jose, CA 95120, United States
Available online 6 June 2006
Abstract We describe a grid-based approach for enterprise-scale data mining, which is based on leveraging parallel database technology for data storage, and on-demand compute servers for parallelism in the statistical computations. This approach is targeted towards the use of data mining in highly-automated vertical business applications, where the data is stored on one or more relational database systems, and an independent set of high-performance compute servers or a network of low-cost, commodity processors is used to improve the application performance and overall workload management. The goal of this paper is to describe an algorithmic decomposition of data mining kernels between the data storage and compute grids, which makes it possible to exploit the parallelism on the respective grids in a simple way, while minimizing the data transfer between these grids. This approach is compatible with existing standards for data mining task specification and results reporting, so that larger applications using these data mining algorithms do not have to be modified to benefit from this grid-based approach. c 2006 Elsevier B.V. All rights reserved.
Keywords: Data mining; Grid computing; Predictive modeling; Parallel databases
1. Introduction Data mining technologies that automate the generation and application of data-based statistical models are of interest in a variety of applications, including for example, customer relationship management (Retail, Banking and Finance, Telecom), fraud detection (Banking and Finance, Telecom), lead generation for marketing and sales (Insurance, Retail), clinical data analysis (Health Care), risk modeling and management (Banking and Finance, Insurance), process modeling and quality control (Manufacturing), genomic data and micro-array analysis (Life Sciences), yield management and logistics (Travel and Transportation), text classification and categorization (cross-Industry), among others (specific casestudies for some of these can be found in [1]). In general, the statistical analysis techniques of predictive modeling, forecasting, optimization, and exploratory data analysis underlying these applications are very data and computation inten-
∗ Corresponding author. Tel.: +1 914 945 2766.
E-mail address: [email protected] (R. Natarajan). c 2006 Elsevier B.V. All rights reserved. 0167-739X/$ - see front matter doi:10.1016/j.future.2006.04.003
sive. The goal of this paper is to describe a grid-based approach for the requirements of enterprise-scale data mining solutions, which include the tight integration of the overall workflow of a vertical business application, and the storage of the relevant mining data on highly-available, secure, commercial relational database systems. These aspects differentiate the present work from other data-intensive computing problems studied in the data grid and scientific computing literature (e.g., [2,3]). The outline of the paper is as follows. Section 1 gives the rationale for evolving database-integrated data mining kernels towards a grid-based architecture. Section 2 describes the proposed algorithmic decomposition of data mining kernels between the data and compute grids. Section 3 illustrates a class of segmentation-based data mining algorithms for which this proposed decomposition provides significant performance benefits. Section 4 provides a schematic of the various components of the grid architecture, including the scheduling interface between the data and compute grids. Section 5 provides the summary and conclusions.
R. Natarajan et al. / Future Generation Computer Systems 23 (2007) 48–54
2. Overview Data mining applications consist of two steps, referred to as modeling and scoring respectively. The modeling step is used to construct probabilistic models from a training data set, for applications such as predictive modeling, exploratory data analysis or data summarization. Once a suitable model has been obtained and validated, the subsequent scoring step uses these models for analyzing new data as required. In this paper, we are primarily concerned with the modeling step, although many of the considerations will apply to the highly data-parallel scoring step as well. From the business application perspective, the modeling and scoring steps are ultimately used to trigger business actions that optimize the relevant business objectives. This approach may be illustrated by an example. An airline company designs a new loyalty program by using purchase and behavioral data on its customers to build a response model for a new promotion. This response model is used in conjunction with a profitability model to score and rank customers. The ranking is then used to identify a group of preferred customers, and to decide on the specific details of the new promotion. In this application, the modeling and scoring steps would be typically performed in batch mode (e.g., once a year if the promotion is offered annually), although evolving business objectives, competitive pressures and technological capabilities may change these requirements. For example, the modeling step may be performed more frequently to accommodate new data or new data features as they become available, particularly if the current model rankings and predictions are significantly affected by the changes in the input data distributions or the modeling assumptions. In addition, the scoring step may be performed interactively (e.g., the customer may need to be rescored in response to a transaction event that leads to a profile change, which could lead to a immediate loyalty program offer at the customer point-of-contact). From the data storage perspective, the business may use a central data warehouse to store all the relevant data and schema in a form suitable for mining, and a parallel database system is typically used to obtain scalable storage and query performance for the large data tables that are encountered. For example, many parallel databases support multi-threaded, shared-memory or distributed, shared-nothing modes of parallelism or both (e.g., [4], where these two modes of parallelism are termed as INTRA and INTER PARALLEL respectively). However, in evolving scenarios, the relevant data may also be distributed in multiple, multi-vendor data warehouses across various organizational dimensions, departments and geographies, and across supplier, process and customer databases. In addition, external databases containing frequently-changing industry or economic data, market intelligence, demographics, and psychographics may also be incorporated into the training data for data mining in specific applications. Finally, we consider the scenario where independent entities may collaborate to “virtually” share their data for modeling purposes, without explicitly exporting or exchanging raw data across their organizational boundaries
49
(e.g., a set of hospitals may pool their radiology data to improve the robustness of diagnostic modeling algorithms). For these applications, the use of federated and data grid technologies, such as [5], which hide the complexity and access permission details of these multiple, multi-vendor databases from the application developer, and provide query optimizations that can minimize excessive data movement and distributed processing overheads, are also important for data mining. Finally, from the computational perspective, many statistical modeling techniques for forecasting and optimization are unsuitable for massive data sets, and typically only use a smaller sampled fraction of the data, which leads to inaccurate models with high variance in the model parameter estimates. In addition, these techniques rely on a variety of computational heuristics which affect the quality of the model search and optimization. A further limitation is that many data mining algorithms are implemented as standalone or client applications that extract database-resident data into their own memory workspace or disk area for the computational processing, which leads to high data transfer and storage costs for large data sets. Even for small or sampled data sets, this approach raises issues of managing multiple data copies and schemas in the client applications that cannot be easily synchronized to the changing data specifications or content on the database servers. In addition, such external data mining client processes, each with its own proprietary API’s and programming requirements, cannot be easily integrated into the SQL-based, data-centric framework of business applications. The implications for the evolution of these data mining architectures are summarized in Fig. 1 (where we assume that the result of the data mining modeling step is an exportable model). In (a), the data mining kernel is implemented as a client application, which uses data that is extracted from the database into its own workspace. In (b), the kernel is implemented as a database-extender consisting of stored procedures and user defined functions installed on the database server. Finally, (c) shows the proposed partitioned data mining kernel with external computational servers implementing high-level mining constructs. The client-based approach in (a) is useful for carrying out exploratory data mining studies, for developing new algorithms, or for testing parallel or high-performance implementations of various data mining kernels, while the approach in (b) is based on having the model generation and scoring algorithms for a set of robust, well-tested data mining kernels implemented as database extenders, and all major database vendors now support integrated mining capabilities on their platforms. The use of de facto standards such as SQL/MM, which is an SQLbased API for task and data specification [6], and PMML, which is an XML-based format for results reporting and model exchange [7], enables these integrated mining kernels to be easily incorporated into the production workflow of data-centric business applications. Furthermore, (b) has the advantage over (a) that the data modeling can be triggered based on the state of internal events recorded in the database. The relevance of the grid-based data mining architecture in (c) relative to (b) for enterprise-scale applications is considered
50
R. Natarajan et al. / Future Generation Computer Systems 23 (2007) 48–54
Fig. 1. Evolution of data mining architectures.
below. First, most stored procedure implementations of the common mining kernels are straightforward adaptations of existing client-based programs, and although the cost of the data transfer to the external clients is reduced, there are further significant performance gains obtained by reducing the data traffic on the database subsystem network itself (for partitioned databases), or by reducing thread synchronization and serialization during the database I/O operations (for multi-threaded databases). In addition, the memory and CPU requirements of these stored procedure adaptations, particularly for long-running data mining tasks, can also negatively impact the performance of a multi-purpose, operational database server. As data mining applications grow in importance, they will have to compete for CPU cycles and memory on the database server with the more traditional transaction processing, decision support and database maintenance workloads. Here, depending on the service-level requirements for the individual components in this workload, it may be necessary to offload data mining calculations in an efficient way to other computational servers for peak workload management. Therefore, if the associated distributed computing overheads can be kept small, the outsourcing of the data mining workloads to an external compute grid provides workload management flexibility, as well as vastly greater compute resources for more extensive model search and optimization. 3. Algorithmic formulation 3.1. Functional decomposition of mining kernel The grid-mining architecture described in this paper is based on reformulating the data mining algorithm into two independent functional phases, namely, a parallel sufficient statistics collection phase implemented on the data grid, and a parallel model selection and parameter estimation phase implemented on a compute grid, with successive iterations of these two phases being used for model refinement and convergence. This functional decomposition is illustrated schematically in Fig. 2, where the data grid may be a parallel or federated database, and the compute grid may be high-performance compute-server or a collection of low-cost, commodity processors.
Fig. 2. Functional decomposition schematic.
The decoupling of the model parameter estimation from the sufficient statistics computation is a consequence of the Neyman–Fisher factorization criterion [8], which states that under the assumption that the data consists of an i.i.d. sample X 1 , X 2 , . . . , X n , drawn from a probability distribution f (x|θ ), where x is a multivariate random variable and θ is a vector of parameters, then the set of functions S1 (X 1 , X 2 , . . . , X n ), . . . , Sk (X 1 , X 2 , . . . , X n ) of the data are sufficient statistics for θ if and only if the likelihood function defined as L(X 1 , X 2 , . . . , X n ) = f (X 1 |θ ) f (X 2 |θ ) . . . f (X n |θ ) can be factorized in the form L(X 1 , X 2 , . . . , X n ) = g1 (X 1 , X 2 , . . . , X n )g2 (S1 , . . . , Sk , θ), where g1 is independent of θ, and g2 depends on the data only through the sufficient statistics. A similar argument holds for conditional probability distributions f (y|x, θ ), where (x, y) are joint multi-variate random variables (the conditional probability formulation is required for classification and regression applications with y denoting the response variable). The most interesting cases are those for which the Neyman–Fisher factorization criterion holds with small values of k, in which case the sufficient statistics S1, S2 , . . . , Sk provide a compressed representation of the information in the data needed to estimate the optimal model parameters θ using maximum likelihood,
R. Natarajan et al. / Future Generation Computer Systems 23 (2007) 48–54
as well as to compute a model likelihood score for a (holdout) data set for given specific values of the model parameters θ (note that the function g1 is a multiplicative constant for a given data set, and can hence be ignored for comparing model scores). This means that steps of model parameter estimation and model validation can be performed economically without having to refer to the original training and validation data. There is a large class of probability distributions for which the Neyman–Pearson factorization criterion holds with a compact set of sufficient statistics (for example, these include many of the distributions in the exponential family such as Normal, Poisson, Log-Normal, Gamma, etc.). The functional decomposition of the data mining kernel in Fig. 2 is interesting because the data transfer between the data and compute grids is just the set of sufficient statistics, which is a significant reduction over the entire data set. Furthermore, the computation of the sufficient statistics can be performed using data-parallel algorithms with minimal communication overheads on the data-grid subsystem, without requiring any specialized parallel libraries (e.g., message passing or synchronization libraries) on the data grid. Finally, the model parameter estimation and model search can also be implemented using straightforward, non-interacting parallel tasks on the compute grid. 3.2. Algorithm mapping and applicability The proposed algorithmic decomposition described here can accommodate many of the data mining formulations in the literature as special cases. In the trivial case, the entire data set is a sufficient statistic for any modeling algorithm (although not a very useful one from the data compression point of view). Another example is obtained when individual partitions of a row-partitioned database table to compute nodes on a one-to-one basis, and the individual models computed from each separate data partition are combined using weighted ensemble averaging to get the final model (e.g., [9,10]). Other algorithm formulations such as bagging [11] and competitive algorithms [12] also fit into the present framework, and efficient implementations of these would use the relevant sufficient statistics instead of the actual data. The implementation of well-known mining algorithms such as association rules, K-means clustering and decision trees with database-resident data can be structured so that the database queries return only the relevant sufficient statistics, rather than the full data (e.g., [13,14] describe such algorithms for decision tree expansion). The pre-computation and caching of the sufficient statistics for eventual data mining applications is described in [15] for compactly storing all possible contingency tables in the record-oriented data set for log-linear response modeling. The idea of using approximate rather than exact sufficient statistics to reduce the computational and data access costs has been considered in [16,17], which is useful in modeling algorithms such as logistic regression, Na¨ıve Bayes with feature selection, and Bayesian network structure identification for which there is no compact set of sufficient statistics from the data.
51
4. Segmentation-based modeling 4.1. Motivation In commercial applications of data mining, the primary interest is often in extending, automating and scaling up the existing and traditional predictive modeling methodology. The frequent requirement is the need to deal with heterogeneous data populations comprising of segments, each of which exhibits the same general model characteristics but with different values for the model parameters. A general class of methods that is very useful in this context is segmentationbased modeling [18]. Here the space of the explanatory variables in the training data is partitioned into mutuallyexclusive, non-overlapping segments, and individual predictive models are constructed for each segment using multivariate probability models that are standard practice in the relevant application domain. The overall model naturally takes the form of “if–then” rules, where the “if ” part defines the condition for segment membership, and the “then” part defines the corresponding segment predictive model. The segment definitions are Boolean combinations of univariate tests on each explanatory variable, including range membership tests for continuous variables, and subset membership tests for nominal variables (note that these segment definitions can be easily translated into the where-clause of an SQL query for retrieving all the data records in the corresponding segment from the database). The determination of the appropriate segments and the estimation of the model parameters in the corresponding segment models can be carried out by jointly optimizing the likelihood function of the overall model for the training data, with validation or hold-out data being used to prevent model overfitting. This is a complex optimization problem involving search and numerical computation, and a variety of heuristics including top-down segment partitioning, bottom-up segment agglomeration, and combinations of these two approaches are used in order to determine the best segmentation/segmentmodel combination. The segment models that have been studied include a bi-variate Poisson–Lognormal model for insurance risk modeling [19], and multivariate linear and logistic regression models for retail response modeling [20]. These segmentation-based predictive data modeling problems are ideally suited for the proposed grid architecture, making good use of parallelism in the data access and sufficient statistics computations on the data grid, as well as in the model computation and search on the compute grid. As seen below, for typical data sets these modeling runs are estimated to require several hours of computational time running serially on standard data mining platforms. 4.2. Performance analysis The potential benefits of the proposed formulation for segmentation-based modeling can be examined using the following architectural model. The data grid and compute grids are assumed to consist of P1 and P2 processors respectively,
52
R. Natarajan et al. / Future Generation Computer Systems 23 (2007) 48–54
with the corresponding time for one floating point operation (flop) on each grid node denoted by α1 and α2 respectively, and the time for the data access of a single field denoted by β1 . Similarly, the cost of invoking a remote method on a compute grid node is denoted by γ1 + γ2 w, where γ1 is the latency for remote method invocation, γ2 is the cost per word for moving data over the network, and w is the size (in words) of the data parameters that are transmitted. Finally, the database table used for the segmentation-based modeling is assumed to have n records and m columns, and these records are partitioned so that each data grid node has n/P1 records (for simplicity, n is assumed to be perfectly divisible by P1 ). Using this model, we consider one pass of a segmented predictive model evaluation, in which a linear regression model with feature selection is computed in each segment (e.g., using the algorithms described for evaluating the sufficient statistics as described in [20]). Since several potential segmentations can be evaluated in parallel, we assume that there are N segments, which may be non-overlapping or overlapping (with N P1 , P2 in general). The sufficient statistics for each potential segment are a pair of covariance matrices (training + evaluation) for the data in each segment, which can evaluated for all N segments in a single parallel scan over the data table. The overall time TD for this aggregation is given by (β1 nm + 0.5α1 knm 2 )/P1 + α1 N P1 m 2 , where k < N is an overlap factor which denotes the average number of segments that individual data records contribute to, with k = 1 in the case of non-overlapping segments. The three terms in TD are respectively the time for reading the data from the database, the time for updating the covariance matrices locally, and the time for aggregating the local covariance matrix contributions for each segment at the end of a data scan. These aggregates are then dispatched to a compute node, for which the scheduling time TS can be estimated as γ1 + γ2 N m 2 /P2 (where we assume that these independent scheduling tasks can be performed in parallel using a multi-threaded scheduler). On the compute nodes, the algorithm used to perform stepwise feature selection proceeds by successively adding features to the regression model, based on first computing the regression model parameters from the training data sufficient statistics, and then examining the degree-of-fit of these models using the evaluation data sufficient statistics. The time TC for the parameter computation and model selection step can be 1 estimated as 12 α2 N m 4 /P2 to leading order for large m, where we note that the usual normal equations algorithm in each segment is O(m 3 ) (e.g., see [20]), but incorporating feature selection algorithm based on the degree-of-fit with the evaluation data as proposed above, increases the complexity to O(m 4 ). Finally, the total time is given by T = TD + TS + TC . For data sets that are representative of some proprietary retail customer applications, we take the nominal values n = 105 , m = 500 for the data size, k = 15, N = 500 for the algorithm parameters, and α1 = α2 = 2 × 10−8 s, β1 = 5 × 10−6 s/word, γ1 = 4 × 10−2 s, γ2 = 2 × 105 s/word for the architectural parameters respectively. Then in the case P1 = 1, P2 = 0 when all the computation must be performed on the data server itself (this is then equivalent to the case (b) described in Fig. 1), the
Fig. 3. Grid data mining architecture schematic.
overall execution time is T = 7.8 h (we note that in this case there is no scheduling cost as all the computation is performed on the data server itself). For the case P1 = 1, P2 = 1, the execution time increases to T = 8.91 h (which consists of TD = 0.57 h, TS = 1.11 h and TC = 7.23 h), which is an overhead of 14% over the serial case, although with 80% of the overall time being off-loaded from the data server. However, by increasing the number of processors on the data and compute grids to P1 = 16, P2 = 128 the execution time comes down to T = 0.106 h (which consists of TD = 0.041 h, TS = 0.009 h and TC = 0.057 h), and this represents a speedup of about 74 over the base case when all the computation is performed on the data server itself. In practice, many retail data sets can have millions of customer records, and the number of features can increase dramatically if interaction terms involving the primary features that are incorporated in the segment models (for example, some segmentation methods use fewer overall segments, but with more complicated nonlinear models within each segment), and the performance analysis indicates that the use of a separate compute grid is even more compelling for these applications. It is much less compelling when the overhead TS is large due to large scheduling latencies or high inter-grid data transfer overheads. 5. Grid architecture schematic 5.1. Overall schematic The overall schematic for grid-based data mining is shown in Fig. 3. It consists of a parallel or federated database, a web service engine for task scheduling and monitoring, and a compute grid. A functional schematic describing the various components is shown in Fig. 4. 5.2. Detailed description of individual components Our description for the data grid layer will refer to the DB2 family of products [4], although the details are quite generic and can be ported to other relational databases as well. The data grid layer implements the SQL/MM interface for data mining task specification and submission. A stored procedure performs various control-flow and book-keeping tasks, such as, for example, issuing parallel queries for sufficient statistics collection, invoking the web service scheduler for parallel task assignment to the compute grid, aggregating and processing the results from the compute grid, managing the control flow for model refinement, and exporting the final model. Many parallel databases provide built-in parallel column functions like MAX, MIN, AVG, SUM and other common
R. Natarajan et al. / Future Generation Computer Systems 23 (2007) 48–54
Fig. 4. Functional schematic of data mining components.
associative–commutative functions, but do not yet provide an API for application programmers to implement general-purpose multi-column parallel aggregation operators. Nevertheless, these parallel data aggregation operators for accumulating the sufficient statistics can be implemented using scratchpad user defined functions, which on parallel databases leverage the parallelism in the SQL query processor (for both INTRA and INTER PARALLEL modes) by using independent scratchpads for each partition or thread. The results in the scratchpads are then aggregated at the end of the data scan using the shared memory or shared disk regions for communication (e.g., [21, 22]). For federated databases, these data aggregation operators would be based on the federated data view, but would leverage the technologies developed for the underlying federated query processor and its cost model in order to optimize the tradeoffs between function shipping, data copying, materialization of intermediate views, and work scheduling and synchronization on the components of the federated view to compute the sufficient statistics in the most efficient way (e.g., [23–25]). The task scheduler, which is implemented as a web service for full generality, can be invoked via SQL queries issued from the database stored procedure (in the case of DB2, using the SOAP messaging capabilities provided by a set of user defined functions for invoking remote web services with database objects as parameters, as provided in the XML extender [26]). This invocation of the scheduler is asynchronous, and the parameters that are passed to the scheduler include the task metadata and the relevant task data aggregate. It also includes a runtime estimate for the task, parameterized by CPU speed and memory requirements. In the special case when the compute grid is co-located within the same administrative domain as the data grid, rather than passing the data aggregate as an in-line task parameter, a connection reference to this data is instead passed to the scheduler. This connection reference is used by the corresponding remote task on the compute grid to retrieve the relevant data aggregate, thereby avoiding the small but serial overhead of processing a large in-line parameter in the invocation of the scheduler. The task scheduler, which shields the data layer from the details of the compute grid, also has modules for automatic discovery of compute resources with the relevant compute task libraries, built-in scheduling algorithms for load balancing, task-to-hardware matching based on the
53
processor and memory requirements, polling mechanisms for monitoring task processes, and task life-cycle management including mechanisms for resubmitting incomplete tasks. The parameterized runtime estimates for the tasks are combined with the server performance statistics for task matching, load balancing and task cycle management (which includes the diagnosis of task failures or improper execution on the compute grid). The scheduler can also implement various back-end optimizations to reduce task dispatching overheads on trusted compute-grid resources. Our experiments on a four-node Linux cluster located on the same LAN as the data server shows an average task scheduling overhead of about 40 ms through the web-service scheduler, suggesting that the granularity of the scheduled tasks must be somewhat larger in order for the linear speedup regime to apply on the compute grid. The compute grid layer contains the code base for high-level mining services including numerically-robust algorithms for parameter estimation and feature selection from the input data or from the sufficient statistics of the data where applicable. The compute grid nodes also contain the resource discovery and resource monitoring components of the task scheduler, which are used for task matching by the scheduler as described above. Further implementation details and experimental performance results on a computational grid consisting of 70 1.2 GHz Linux nodes can be found in [27]. 6. Summary As business applications of data mining become more widespread, the performance and quality of data mining kernels must be improved and the impact of this workload on other aspects of the operational data processing must be minimized. The evolutionary approach described in this paper is based on a functional decomposition of the data mining kernel, which is capable of exploiting the parallelism on data and compute grids with minimal use of explicit parallel programming software constructs and inter-grid communication overheads. It leverages existing or evolving data mining standards such as SQL/MM for job specification and submission, and PMML for model specification and export, but can provide better scalable performance or improved data mining model quality when compared to existing implementations of databaseintegrated mining kernels. Finally, the present approach can lead to greater flexibility in the deployment of a data mining application, leading to better overall workload management on enterprise data servers that support a complex mix of transactional, decision support, data management and data mining applications. References [1] C. Apte, B. Liu, E.P.D. Pednault, P. Smyth, Business applications of data mining, Communications of the ACM 45 (8) (2002) 49–53. [2] W.E. Johnston, Computational and data grids in large-scale science and engineering, Future Generation Computer Systems 8 (2002) 1085–1100. [3] D. Arnold, S. Vadhiyar, J. Dongarra, On the convergence of computational and data grids, Parallel Processing Letters 11 (2001) 187–202. [4] The IBM DB2 Universal Database V8.1, http://www.ibm.com/software/ data/db2, 2004.
54
R. Natarajan et al. / Future Generation Computer Systems 23 (2007) 48–54
[5] The IBM DB2 Information Integrator, http://www.ibm.com/software/ integration, 2004. [6] ISO/IEC 13249 Final Committee Draft, Information Technology – Database Languages – SQL Multimedia and Application Packages, http:// www.iso.org, 2002. [7] Predictive Modeling Markup Language, http://www.dmg.org, 2002. [8] M.H. DeGroot, M.J. Schervish, Probability and Statistics, third ed., Addison Wesley, 2002. [9] A. Prodromides, P. Chan, S. Stolfo, Meta learning in distributed data systems — issues and approaches, in: H. Kargupta, P. Chan (Eds.), Advances in Distributed Data Mining, AAAI Press, 2000. [10] M. Cannataro, D. Talia, P. Trunfio, Distributed data mining on the grid, Future Generation Computer Systems 18 (2002) 1101–1112. [11] L. Breiman, Bagging predictors, Machine Learning 24 (2) (1996) 123–140. [12] P. Giudici, R. Castelo, Improving Markov chain Monte Carlo model search for data mining, Machine Learning 50 (2003) 127–158. [13] G. Graefe, U. Fayyad, S. Chaudhuri, On the efficient gathering of sufficient statistics for classification from large SQL databases, in: Proceedings Fourth International Conference on Knowledge Discovery and Data Mining, AAAI Press, Menlo Park, 1998, pp. 204–208. [14] D. Caragea, A. Silvescu, V. Honavar, A framework for learning from distributed data using sufficient statistics and its application to learning decision trees, International Journal of Hybrid Intelligent Systems 1 (2004) 80–89. [15] A. Moore, M.S. Lee, Cached sufficient statistics for efficient machine learning with massive datasets, Journal of Artificial Intelligence Research 8 (1998) 67–91. [16] R. Natarajan, E.P.D. Pednault, Using simulated pseudo data to speed up statistical predictive modeling from large data sets, in: Proceedings of the 1st SIAM Conference on Data Mining, Chicago IL, 2000. [17] N. Friedman, L. Getoor, Efficient learning using constrained sufficient statistics, in: Proceedings of the 7th International Workshop on Artificial Intelligence and Statistics, 1999. [18] C. Apte, R. Natarajan, E. Pednault, F. Tipu, A probabilistic framework for predictive modeling analytics, IBM Systems Journal 41 (3) (2002). [19] C. Apte, E. Grossman, E. Pednault, B. Rosen, F. Tipu, B. White, Probabilistic estimation based data mining for discovering insurance risks, IEEE Intelligent Systems 14 (6) (1999). [20] R. Natarajan, E.P.D. Pednault, Segmented regression estimators for massive data sets, in: Proceedings of the 2nd SIAM Conference on Data Mining, Crystal City, VA, 2002. [21] A. Dorneich, E. Pednault, R. Natarajan, F. Tipu, Embedded predictive modeling in a parallel relational database, in: Proceedings of the 21st ACM Symposium on Applied Computing, Dijon, France, 2006. [22] M. Jaedicke, B. Mitschang, On parallel processing of aggregate and scalar function in object-relational DBMS, in: Proceedings of the ACM SIGMOD Int. Conf. on Management of Data, Seattle, WA, 1998. [23] M. Atkinson, A.L. Chervenak, P. Kunszt, I. Narang, N.W. Paton, D. Pearson, A. Shoshani, P. Wilson, Data access, integration and management, in: I. Foster, C. Kesselman (Eds.), The Grid: Blueprint for a New Computing Infrastructure, second ed., Morgan Kaufman, 2003 (Chapter 22). [24] M. Rodriguez-Martinez, N. Roussopoulos, MOCHA: A self-extensible database middleware system for distributed data sources, in: Proceedings
of the ACM SIGMOD International Conference for Distributed Data Sources, Dallas, TX, 2000, pp. 213–224. [25] D. Kossmann, M.J. Franklin, G. Drasch, Cache investment: Integrating query optimization and distributed data placement, ACM Transactions on Database Systems 25 (2000) 517–558. [26] The IBM DB2 XML Extender, http://www.ibm.com/software/data/db2/ extender/xmlext, 2004. [27] R. Sion, R. Natarajan, I. Narang, W. Li, T. Phan, XG: A datadriven computational grid for enterprise-scale mining, in: Proceedings of the 16th International Conference on Database and Expert Systems Applications, Copenhagen, Denmark, 2005. Dr. Ramesh Natarajan has been a member of the Data Abstraction group at the IBM T. J. Watson Research Center since 1998. His research interests are in the areas of statistical data mining and databases. Prior to 1998, he was in the Parallel Applications group at IBM Watson, where he worked in the areas of parallel scientific computing, performance benchmarking and supercomputing applications. He has authored numerous papers in computational fluid dynamics, applied mathematics, parallel computing and statistical data mining, and is the recipient of two IBM research division awards for his contributions to the IBM SP-2 parallel computer project. He received his B. Tech. (I.I.T. Madras, 1978), Ph.D. (UT Austin, 1984) and was Post-doctoral associate (M.I.T., 1984–1986) all in Chemical Engineering. From 1986 to 1987 he held a joint appointment as Supercomputing Specialist at Harvard University, and Senior Consultant at M.I.T. in their Information Systems department. Radu Sion is an Assistant Professor of Computer Sciences at Stony Brook University. His research interests are in the areas of Data Security and Privacy in Distributed Networked Environments. He is excited by inter-connected entities that access data and need to do so with assurances of security, privacy, and functionality, preferably efficiently. Applications include: wireless security, secure data outsourcing, queries over encrypted data, secure reputation systems, digital rights protection, integrity proofs in sensor networks, secure storage in peer to peer and ad hoc environments, data privacy and bounds on illicit inference over multiple data sources, security and policy management in computation/data grids, etc. He received his Ph.D. (2004) in Computer Sciences from Purdue University. While at Purdue, Radu was affiliated with the Center of Education and Research in Information Assurance and with the Indiana Center of Database Systems. In 2004 Radu visited with the IBM Almaden Research Center, while on leave from Stony Brook. Thomas Phan is a postdoctoral researcher at IBM Almaden Research Center. He received his Ph.D. in Computer Science from UCLA. His research interests include distributed computing, middleware, and mobile services.
A grid-based approach for enterprise-scale data mining Ramesh Natarajan a,∗ , Radu Sion b , Thomas Phan c a IBM Thomas J. Watson Research Center, P.O. Box 218, Yorktown Heights, NY 10598, United States b Department of Computer Science, State University of New York, Stonybrook, NY 11794, United States c IBM Almaden Research Center, 650 Harry Road, San Jose, CA 95120, United States
Available online 6 June 2006
Abstract We describe a grid-based approach for enterprise-scale data mining, which is based on leveraging parallel database technology for data storage, and on-demand compute servers for parallelism in the statistical computations. This approach is targeted towards the use of data mining in highly-automated vertical business applications, where the data is stored on one or more relational database systems, and an independent set of high-performance compute servers or a network of low-cost, commodity processors is used to improve the application performance and overall workload management. The goal of this paper is to describe an algorithmic decomposition of data mining kernels between the data storage and compute grids, which makes it possible to exploit the parallelism on the respective grids in a simple way, while minimizing the data transfer between these grids. This approach is compatible with existing standards for data mining task specification and results reporting, so that larger applications using these data mining algorithms do not have to be modified to benefit from this grid-based approach. c 2006 Elsevier B.V. All rights reserved.
Keywords: Data mining; Grid computing; Predictive modeling; Parallel databases
1. Introduction Data mining technologies that automate the generation and application of data-based statistical models are of interest in a variety of applications, including for example, customer relationship management (Retail, Banking and Finance, Telecom), fraud detection (Banking and Finance, Telecom), lead generation for marketing and sales (Insurance, Retail), clinical data analysis (Health Care), risk modeling and management (Banking and Finance, Insurance), process modeling and quality control (Manufacturing), genomic data and micro-array analysis (Life Sciences), yield management and logistics (Travel and Transportation), text classification and categorization (cross-Industry), among others (specific casestudies for some of these can be found in [1]). In general, the statistical analysis techniques of predictive modeling, forecasting, optimization, and exploratory data analysis underlying these applications are very data and computation inten-
∗ Corresponding author. Tel.: +1 914 945 2766.
E-mail address: [email protected] (R. Natarajan). c 2006 Elsevier B.V. All rights reserved. 0167-739X/$ - see front matter doi:10.1016/j.future.2006.04.003
sive. The goal of this paper is to describe a grid-based approach for the requirements of enterprise-scale data mining solutions, which include the tight integration of the overall workflow of a vertical business application, and the storage of the relevant mining data on highly-available, secure, commercial relational database systems. These aspects differentiate the present work from other data-intensive computing problems studied in the data grid and scientific computing literature (e.g., [2,3]). The outline of the paper is as follows. Section 1 gives the rationale for evolving database-integrated data mining kernels towards a grid-based architecture. Section 2 describes the proposed algorithmic decomposition of data mining kernels between the data and compute grids. Section 3 illustrates a class of segmentation-based data mining algorithms for which this proposed decomposition provides significant performance benefits. Section 4 provides a schematic of the various components of the grid architecture, including the scheduling interface between the data and compute grids. Section 5 provides the summary and conclusions.
R. Natarajan et al. / Future Generation Computer Systems 23 (2007) 48–54
2. Overview Data mining applications consist of two steps, referred to as modeling and scoring respectively. The modeling step is used to construct probabilistic models from a training data set, for applications such as predictive modeling, exploratory data analysis or data summarization. Once a suitable model has been obtained and validated, the subsequent scoring step uses these models for analyzing new data as required. In this paper, we are primarily concerned with the modeling step, although many of the considerations will apply to the highly data-parallel scoring step as well. From the business application perspective, the modeling and scoring steps are ultimately used to trigger business actions that optimize the relevant business objectives. This approach may be illustrated by an example. An airline company designs a new loyalty program by using purchase and behavioral data on its customers to build a response model for a new promotion. This response model is used in conjunction with a profitability model to score and rank customers. The ranking is then used to identify a group of preferred customers, and to decide on the specific details of the new promotion. In this application, the modeling and scoring steps would be typically performed in batch mode (e.g., once a year if the promotion is offered annually), although evolving business objectives, competitive pressures and technological capabilities may change these requirements. For example, the modeling step may be performed more frequently to accommodate new data or new data features as they become available, particularly if the current model rankings and predictions are significantly affected by the changes in the input data distributions or the modeling assumptions. In addition, the scoring step may be performed interactively (e.g., the customer may need to be rescored in response to a transaction event that leads to a profile change, which could lead to a immediate loyalty program offer at the customer point-of-contact). From the data storage perspective, the business may use a central data warehouse to store all the relevant data and schema in a form suitable for mining, and a parallel database system is typically used to obtain scalable storage and query performance for the large data tables that are encountered. For example, many parallel databases support multi-threaded, shared-memory or distributed, shared-nothing modes of parallelism or both (e.g., [4], where these two modes of parallelism are termed as INTRA and INTER PARALLEL respectively). However, in evolving scenarios, the relevant data may also be distributed in multiple, multi-vendor data warehouses across various organizational dimensions, departments and geographies, and across supplier, process and customer databases. In addition, external databases containing frequently-changing industry or economic data, market intelligence, demographics, and psychographics may also be incorporated into the training data for data mining in specific applications. Finally, we consider the scenario where independent entities may collaborate to “virtually” share their data for modeling purposes, without explicitly exporting or exchanging raw data across their organizational boundaries
49
(e.g., a set of hospitals may pool their radiology data to improve the robustness of diagnostic modeling algorithms). For these applications, the use of federated and data grid technologies, such as [5], which hide the complexity and access permission details of these multiple, multi-vendor databases from the application developer, and provide query optimizations that can minimize excessive data movement and distributed processing overheads, are also important for data mining. Finally, from the computational perspective, many statistical modeling techniques for forecasting and optimization are unsuitable for massive data sets, and typically only use a smaller sampled fraction of the data, which leads to inaccurate models with high variance in the model parameter estimates. In addition, these techniques rely on a variety of computational heuristics which affect the quality of the model search and optimization. A further limitation is that many data mining algorithms are implemented as standalone or client applications that extract database-resident data into their own memory workspace or disk area for the computational processing, which leads to high data transfer and storage costs for large data sets. Even for small or sampled data sets, this approach raises issues of managing multiple data copies and schemas in the client applications that cannot be easily synchronized to the changing data specifications or content on the database servers. In addition, such external data mining client processes, each with its own proprietary API’s and programming requirements, cannot be easily integrated into the SQL-based, data-centric framework of business applications. The implications for the evolution of these data mining architectures are summarized in Fig. 1 (where we assume that the result of the data mining modeling step is an exportable model). In (a), the data mining kernel is implemented as a client application, which uses data that is extracted from the database into its own workspace. In (b), the kernel is implemented as a database-extender consisting of stored procedures and user defined functions installed on the database server. Finally, (c) shows the proposed partitioned data mining kernel with external computational servers implementing high-level mining constructs. The client-based approach in (a) is useful for carrying out exploratory data mining studies, for developing new algorithms, or for testing parallel or high-performance implementations of various data mining kernels, while the approach in (b) is based on having the model generation and scoring algorithms for a set of robust, well-tested data mining kernels implemented as database extenders, and all major database vendors now support integrated mining capabilities on their platforms. The use of de facto standards such as SQL/MM, which is an SQLbased API for task and data specification [6], and PMML, which is an XML-based format for results reporting and model exchange [7], enables these integrated mining kernels to be easily incorporated into the production workflow of data-centric business applications. Furthermore, (b) has the advantage over (a) that the data modeling can be triggered based on the state of internal events recorded in the database. The relevance of the grid-based data mining architecture in (c) relative to (b) for enterprise-scale applications is considered
50
R. Natarajan et al. / Future Generation Computer Systems 23 (2007) 48–54
Fig. 1. Evolution of data mining architectures.
below. First, most stored procedure implementations of the common mining kernels are straightforward adaptations of existing client-based programs, and although the cost of the data transfer to the external clients is reduced, there are further significant performance gains obtained by reducing the data traffic on the database subsystem network itself (for partitioned databases), or by reducing thread synchronization and serialization during the database I/O operations (for multi-threaded databases). In addition, the memory and CPU requirements of these stored procedure adaptations, particularly for long-running data mining tasks, can also negatively impact the performance of a multi-purpose, operational database server. As data mining applications grow in importance, they will have to compete for CPU cycles and memory on the database server with the more traditional transaction processing, decision support and database maintenance workloads. Here, depending on the service-level requirements for the individual components in this workload, it may be necessary to offload data mining calculations in an efficient way to other computational servers for peak workload management. Therefore, if the associated distributed computing overheads can be kept small, the outsourcing of the data mining workloads to an external compute grid provides workload management flexibility, as well as vastly greater compute resources for more extensive model search and optimization. 3. Algorithmic formulation 3.1. Functional decomposition of mining kernel The grid-mining architecture described in this paper is based on reformulating the data mining algorithm into two independent functional phases, namely, a parallel sufficient statistics collection phase implemented on the data grid, and a parallel model selection and parameter estimation phase implemented on a compute grid, with successive iterations of these two phases being used for model refinement and convergence. This functional decomposition is illustrated schematically in Fig. 2, where the data grid may be a parallel or federated database, and the compute grid may be high-performance compute-server or a collection of low-cost, commodity processors.
Fig. 2. Functional decomposition schematic.
The decoupling of the model parameter estimation from the sufficient statistics computation is a consequence of the Neyman–Fisher factorization criterion [8], which states that under the assumption that the data consists of an i.i.d. sample X 1 , X 2 , . . . , X n , drawn from a probability distribution f (x|θ ), where x is a multivariate random variable and θ is a vector of parameters, then the set of functions S1 (X 1 , X 2 , . . . , X n ), . . . , Sk (X 1 , X 2 , . . . , X n ) of the data are sufficient statistics for θ if and only if the likelihood function defined as L(X 1 , X 2 , . . . , X n ) = f (X 1 |θ ) f (X 2 |θ ) . . . f (X n |θ ) can be factorized in the form L(X 1 , X 2 , . . . , X n ) = g1 (X 1 , X 2 , . . . , X n )g2 (S1 , . . . , Sk , θ), where g1 is independent of θ, and g2 depends on the data only through the sufficient statistics. A similar argument holds for conditional probability distributions f (y|x, θ ), where (x, y) are joint multi-variate random variables (the conditional probability formulation is required for classification and regression applications with y denoting the response variable). The most interesting cases are those for which the Neyman–Fisher factorization criterion holds with small values of k, in which case the sufficient statistics S1, S2 , . . . , Sk provide a compressed representation of the information in the data needed to estimate the optimal model parameters θ using maximum likelihood,
R. Natarajan et al. / Future Generation Computer Systems 23 (2007) 48–54
as well as to compute a model likelihood score for a (holdout) data set for given specific values of the model parameters θ (note that the function g1 is a multiplicative constant for a given data set, and can hence be ignored for comparing model scores). This means that steps of model parameter estimation and model validation can be performed economically without having to refer to the original training and validation data. There is a large class of probability distributions for which the Neyman–Pearson factorization criterion holds with a compact set of sufficient statistics (for example, these include many of the distributions in the exponential family such as Normal, Poisson, Log-Normal, Gamma, etc.). The functional decomposition of the data mining kernel in Fig. 2 is interesting because the data transfer between the data and compute grids is just the set of sufficient statistics, which is a significant reduction over the entire data set. Furthermore, the computation of the sufficient statistics can be performed using data-parallel algorithms with minimal communication overheads on the data-grid subsystem, without requiring any specialized parallel libraries (e.g., message passing or synchronization libraries) on the data grid. Finally, the model parameter estimation and model search can also be implemented using straightforward, non-interacting parallel tasks on the compute grid. 3.2. Algorithm mapping and applicability The proposed algorithmic decomposition described here can accommodate many of the data mining formulations in the literature as special cases. In the trivial case, the entire data set is a sufficient statistic for any modeling algorithm (although not a very useful one from the data compression point of view). Another example is obtained when individual partitions of a row-partitioned database table to compute nodes on a one-to-one basis, and the individual models computed from each separate data partition are combined using weighted ensemble averaging to get the final model (e.g., [9,10]). Other algorithm formulations such as bagging [11] and competitive algorithms [12] also fit into the present framework, and efficient implementations of these would use the relevant sufficient statistics instead of the actual data. The implementation of well-known mining algorithms such as association rules, K-means clustering and decision trees with database-resident data can be structured so that the database queries return only the relevant sufficient statistics, rather than the full data (e.g., [13,14] describe such algorithms for decision tree expansion). The pre-computation and caching of the sufficient statistics for eventual data mining applications is described in [15] for compactly storing all possible contingency tables in the record-oriented data set for log-linear response modeling. The idea of using approximate rather than exact sufficient statistics to reduce the computational and data access costs has been considered in [16,17], which is useful in modeling algorithms such as logistic regression, Na¨ıve Bayes with feature selection, and Bayesian network structure identification for which there is no compact set of sufficient statistics from the data.
51
4. Segmentation-based modeling 4.1. Motivation In commercial applications of data mining, the primary interest is often in extending, automating and scaling up the existing and traditional predictive modeling methodology. The frequent requirement is the need to deal with heterogeneous data populations comprising of segments, each of which exhibits the same general model characteristics but with different values for the model parameters. A general class of methods that is very useful in this context is segmentationbased modeling [18]. Here the space of the explanatory variables in the training data is partitioned into mutuallyexclusive, non-overlapping segments, and individual predictive models are constructed for each segment using multivariate probability models that are standard practice in the relevant application domain. The overall model naturally takes the form of “if–then” rules, where the “if ” part defines the condition for segment membership, and the “then” part defines the corresponding segment predictive model. The segment definitions are Boolean combinations of univariate tests on each explanatory variable, including range membership tests for continuous variables, and subset membership tests for nominal variables (note that these segment definitions can be easily translated into the where-clause of an SQL query for retrieving all the data records in the corresponding segment from the database). The determination of the appropriate segments and the estimation of the model parameters in the corresponding segment models can be carried out by jointly optimizing the likelihood function of the overall model for the training data, with validation or hold-out data being used to prevent model overfitting. This is a complex optimization problem involving search and numerical computation, and a variety of heuristics including top-down segment partitioning, bottom-up segment agglomeration, and combinations of these two approaches are used in order to determine the best segmentation/segmentmodel combination. The segment models that have been studied include a bi-variate Poisson–Lognormal model for insurance risk modeling [19], and multivariate linear and logistic regression models for retail response modeling [20]. These segmentation-based predictive data modeling problems are ideally suited for the proposed grid architecture, making good use of parallelism in the data access and sufficient statistics computations on the data grid, as well as in the model computation and search on the compute grid. As seen below, for typical data sets these modeling runs are estimated to require several hours of computational time running serially on standard data mining platforms. 4.2. Performance analysis The potential benefits of the proposed formulation for segmentation-based modeling can be examined using the following architectural model. The data grid and compute grids are assumed to consist of P1 and P2 processors respectively,
52
R. Natarajan et al. / Future Generation Computer Systems 23 (2007) 48–54
with the corresponding time for one floating point operation (flop) on each grid node denoted by α1 and α2 respectively, and the time for the data access of a single field denoted by β1 . Similarly, the cost of invoking a remote method on a compute grid node is denoted by γ1 + γ2 w, where γ1 is the latency for remote method invocation, γ2 is the cost per word for moving data over the network, and w is the size (in words) of the data parameters that are transmitted. Finally, the database table used for the segmentation-based modeling is assumed to have n records and m columns, and these records are partitioned so that each data grid node has n/P1 records (for simplicity, n is assumed to be perfectly divisible by P1 ). Using this model, we consider one pass of a segmented predictive model evaluation, in which a linear regression model with feature selection is computed in each segment (e.g., using the algorithms described for evaluating the sufficient statistics as described in [20]). Since several potential segmentations can be evaluated in parallel, we assume that there are N segments, which may be non-overlapping or overlapping (with N P1 , P2 in general). The sufficient statistics for each potential segment are a pair of covariance matrices (training + evaluation) for the data in each segment, which can evaluated for all N segments in a single parallel scan over the data table. The overall time TD for this aggregation is given by (β1 nm + 0.5α1 knm 2 )/P1 + α1 N P1 m 2 , where k < N is an overlap factor which denotes the average number of segments that individual data records contribute to, with k = 1 in the case of non-overlapping segments. The three terms in TD are respectively the time for reading the data from the database, the time for updating the covariance matrices locally, and the time for aggregating the local covariance matrix contributions for each segment at the end of a data scan. These aggregates are then dispatched to a compute node, for which the scheduling time TS can be estimated as γ1 + γ2 N m 2 /P2 (where we assume that these independent scheduling tasks can be performed in parallel using a multi-threaded scheduler). On the compute nodes, the algorithm used to perform stepwise feature selection proceeds by successively adding features to the regression model, based on first computing the regression model parameters from the training data sufficient statistics, and then examining the degree-of-fit of these models using the evaluation data sufficient statistics. The time TC for the parameter computation and model selection step can be 1 estimated as 12 α2 N m 4 /P2 to leading order for large m, where we note that the usual normal equations algorithm in each segment is O(m 3 ) (e.g., see [20]), but incorporating feature selection algorithm based on the degree-of-fit with the evaluation data as proposed above, increases the complexity to O(m 4 ). Finally, the total time is given by T = TD + TS + TC . For data sets that are representative of some proprietary retail customer applications, we take the nominal values n = 105 , m = 500 for the data size, k = 15, N = 500 for the algorithm parameters, and α1 = α2 = 2 × 10−8 s, β1 = 5 × 10−6 s/word, γ1 = 4 × 10−2 s, γ2 = 2 × 105 s/word for the architectural parameters respectively. Then in the case P1 = 1, P2 = 0 when all the computation must be performed on the data server itself (this is then equivalent to the case (b) described in Fig. 1), the
Fig. 3. Grid data mining architecture schematic.
overall execution time is T = 7.8 h (we note that in this case there is no scheduling cost as all the computation is performed on the data server itself). For the case P1 = 1, P2 = 1, the execution time increases to T = 8.91 h (which consists of TD = 0.57 h, TS = 1.11 h and TC = 7.23 h), which is an overhead of 14% over the serial case, although with 80% of the overall time being off-loaded from the data server. However, by increasing the number of processors on the data and compute grids to P1 = 16, P2 = 128 the execution time comes down to T = 0.106 h (which consists of TD = 0.041 h, TS = 0.009 h and TC = 0.057 h), and this represents a speedup of about 74 over the base case when all the computation is performed on the data server itself. In practice, many retail data sets can have millions of customer records, and the number of features can increase dramatically if interaction terms involving the primary features that are incorporated in the segment models (for example, some segmentation methods use fewer overall segments, but with more complicated nonlinear models within each segment), and the performance analysis indicates that the use of a separate compute grid is even more compelling for these applications. It is much less compelling when the overhead TS is large due to large scheduling latencies or high inter-grid data transfer overheads. 5. Grid architecture schematic 5.1. Overall schematic The overall schematic for grid-based data mining is shown in Fig. 3. It consists of a parallel or federated database, a web service engine for task scheduling and monitoring, and a compute grid. A functional schematic describing the various components is shown in Fig. 4. 5.2. Detailed description of individual components Our description for the data grid layer will refer to the DB2 family of products [4], although the details are quite generic and can be ported to other relational databases as well. The data grid layer implements the SQL/MM interface for data mining task specification and submission. A stored procedure performs various control-flow and book-keeping tasks, such as, for example, issuing parallel queries for sufficient statistics collection, invoking the web service scheduler for parallel task assignment to the compute grid, aggregating and processing the results from the compute grid, managing the control flow for model refinement, and exporting the final model. Many parallel databases provide built-in parallel column functions like MAX, MIN, AVG, SUM and other common
R. Natarajan et al. / Future Generation Computer Systems 23 (2007) 48–54
Fig. 4. Functional schematic of data mining components.
associative–commutative functions, but do not yet provide an API for application programmers to implement general-purpose multi-column parallel aggregation operators. Nevertheless, these parallel data aggregation operators for accumulating the sufficient statistics can be implemented using scratchpad user defined functions, which on parallel databases leverage the parallelism in the SQL query processor (for both INTRA and INTER PARALLEL modes) by using independent scratchpads for each partition or thread. The results in the scratchpads are then aggregated at the end of the data scan using the shared memory or shared disk regions for communication (e.g., [21, 22]). For federated databases, these data aggregation operators would be based on the federated data view, but would leverage the technologies developed for the underlying federated query processor and its cost model in order to optimize the tradeoffs between function shipping, data copying, materialization of intermediate views, and work scheduling and synchronization on the components of the federated view to compute the sufficient statistics in the most efficient way (e.g., [23–25]). The task scheduler, which is implemented as a web service for full generality, can be invoked via SQL queries issued from the database stored procedure (in the case of DB2, using the SOAP messaging capabilities provided by a set of user defined functions for invoking remote web services with database objects as parameters, as provided in the XML extender [26]). This invocation of the scheduler is asynchronous, and the parameters that are passed to the scheduler include the task metadata and the relevant task data aggregate. It also includes a runtime estimate for the task, parameterized by CPU speed and memory requirements. In the special case when the compute grid is co-located within the same administrative domain as the data grid, rather than passing the data aggregate as an in-line task parameter, a connection reference to this data is instead passed to the scheduler. This connection reference is used by the corresponding remote task on the compute grid to retrieve the relevant data aggregate, thereby avoiding the small but serial overhead of processing a large in-line parameter in the invocation of the scheduler. The task scheduler, which shields the data layer from the details of the compute grid, also has modules for automatic discovery of compute resources with the relevant compute task libraries, built-in scheduling algorithms for load balancing, task-to-hardware matching based on the
53
processor and memory requirements, polling mechanisms for monitoring task processes, and task life-cycle management including mechanisms for resubmitting incomplete tasks. The parameterized runtime estimates for the tasks are combined with the server performance statistics for task matching, load balancing and task cycle management (which includes the diagnosis of task failures or improper execution on the compute grid). The scheduler can also implement various back-end optimizations to reduce task dispatching overheads on trusted compute-grid resources. Our experiments on a four-node Linux cluster located on the same LAN as the data server shows an average task scheduling overhead of about 40 ms through the web-service scheduler, suggesting that the granularity of the scheduled tasks must be somewhat larger in order for the linear speedup regime to apply on the compute grid. The compute grid layer contains the code base for high-level mining services including numerically-robust algorithms for parameter estimation and feature selection from the input data or from the sufficient statistics of the data where applicable. The compute grid nodes also contain the resource discovery and resource monitoring components of the task scheduler, which are used for task matching by the scheduler as described above. Further implementation details and experimental performance results on a computational grid consisting of 70 1.2 GHz Linux nodes can be found in [27]. 6. Summary As business applications of data mining become more widespread, the performance and quality of data mining kernels must be improved and the impact of this workload on other aspects of the operational data processing must be minimized. The evolutionary approach described in this paper is based on a functional decomposition of the data mining kernel, which is capable of exploiting the parallelism on data and compute grids with minimal use of explicit parallel programming software constructs and inter-grid communication overheads. It leverages existing or evolving data mining standards such as SQL/MM for job specification and submission, and PMML for model specification and export, but can provide better scalable performance or improved data mining model quality when compared to existing implementations of databaseintegrated mining kernels. Finally, the present approach can lead to greater flexibility in the deployment of a data mining application, leading to better overall workload management on enterprise data servers that support a complex mix of transactional, decision support, data management and data mining applications. References [1] C. Apte, B. Liu, E.P.D. Pednault, P. Smyth, Business applications of data mining, Communications of the ACM 45 (8) (2002) 49–53. [2] W.E. Johnston, Computational and data grids in large-scale science and engineering, Future Generation Computer Systems 8 (2002) 1085–1100. [3] D. Arnold, S. Vadhiyar, J. Dongarra, On the convergence of computational and data grids, Parallel Processing Letters 11 (2001) 187–202. [4] The IBM DB2 Universal Database V8.1, http://www.ibm.com/software/ data/db2, 2004.
54
R. Natarajan et al. / Future Generation Computer Systems 23 (2007) 48–54
[5] The IBM DB2 Information Integrator, http://www.ibm.com/software/ integration, 2004. [6] ISO/IEC 13249 Final Committee Draft, Information Technology – Database Languages – SQL Multimedia and Application Packages, http:// www.iso.org, 2002. [7] Predictive Modeling Markup Language, http://www.dmg.org, 2002. [8] M.H. DeGroot, M.J. Schervish, Probability and Statistics, third ed., Addison Wesley, 2002. [9] A. Prodromides, P. Chan, S. Stolfo, Meta learning in distributed data systems — issues and approaches, in: H. Kargupta, P. Chan (Eds.), Advances in Distributed Data Mining, AAAI Press, 2000. [10] M. Cannataro, D. Talia, P. Trunfio, Distributed data mining on the grid, Future Generation Computer Systems 18 (2002) 1101–1112. [11] L. Breiman, Bagging predictors, Machine Learning 24 (2) (1996) 123–140. [12] P. Giudici, R. Castelo, Improving Markov chain Monte Carlo model search for data mining, Machine Learning 50 (2003) 127–158. [13] G. Graefe, U. Fayyad, S. Chaudhuri, On the efficient gathering of sufficient statistics for classification from large SQL databases, in: Proceedings Fourth International Conference on Knowledge Discovery and Data Mining, AAAI Press, Menlo Park, 1998, pp. 204–208. [14] D. Caragea, A. Silvescu, V. Honavar, A framework for learning from distributed data using sufficient statistics and its application to learning decision trees, International Journal of Hybrid Intelligent Systems 1 (2004) 80–89. [15] A. Moore, M.S. Lee, Cached sufficient statistics for efficient machine learning with massive datasets, Journal of Artificial Intelligence Research 8 (1998) 67–91. [16] R. Natarajan, E.P.D. Pednault, Using simulated pseudo data to speed up statistical predictive modeling from large data sets, in: Proceedings of the 1st SIAM Conference on Data Mining, Chicago IL, 2000. [17] N. Friedman, L. Getoor, Efficient learning using constrained sufficient statistics, in: Proceedings of the 7th International Workshop on Artificial Intelligence and Statistics, 1999. [18] C. Apte, R. Natarajan, E. Pednault, F. Tipu, A probabilistic framework for predictive modeling analytics, IBM Systems Journal 41 (3) (2002). [19] C. Apte, E. Grossman, E. Pednault, B. Rosen, F. Tipu, B. White, Probabilistic estimation based data mining for discovering insurance risks, IEEE Intelligent Systems 14 (6) (1999). [20] R. Natarajan, E.P.D. Pednault, Segmented regression estimators for massive data sets, in: Proceedings of the 2nd SIAM Conference on Data Mining, Crystal City, VA, 2002. [21] A. Dorneich, E. Pednault, R. Natarajan, F. Tipu, Embedded predictive modeling in a parallel relational database, in: Proceedings of the 21st ACM Symposium on Applied Computing, Dijon, France, 2006. [22] M. Jaedicke, B. Mitschang, On parallel processing of aggregate and scalar function in object-relational DBMS, in: Proceedings of the ACM SIGMOD Int. Conf. on Management of Data, Seattle, WA, 1998. [23] M. Atkinson, A.L. Chervenak, P. Kunszt, I. Narang, N.W. Paton, D. Pearson, A. Shoshani, P. Wilson, Data access, integration and management, in: I. Foster, C. Kesselman (Eds.), The Grid: Blueprint for a New Computing Infrastructure, second ed., Morgan Kaufman, 2003 (Chapter 22). [24] M. Rodriguez-Martinez, N. Roussopoulos, MOCHA: A self-extensible database middleware system for distributed data sources, in: Proceedings
of the ACM SIGMOD International Conference for Distributed Data Sources, Dallas, TX, 2000, pp. 213–224. [25] D. Kossmann, M.J. Franklin, G. Drasch, Cache investment: Integrating query optimization and distributed data placement, ACM Transactions on Database Systems 25 (2000) 517–558. [26] The IBM DB2 XML Extender, http://www.ibm.com/software/data/db2/ extender/xmlext, 2004. [27] R. Sion, R. Natarajan, I. Narang, W. Li, T. Phan, XG: A datadriven computational grid for enterprise-scale mining, in: Proceedings of the 16th International Conference on Database and Expert Systems Applications, Copenhagen, Denmark, 2005. Dr. Ramesh Natarajan has been a member of the Data Abstraction group at the IBM T. J. Watson Research Center since 1998. His research interests are in the areas of statistical data mining and databases. Prior to 1998, he was in the Parallel Applications group at IBM Watson, where he worked in the areas of parallel scientific computing, performance benchmarking and supercomputing applications. He has authored numerous papers in computational fluid dynamics, applied mathematics, parallel computing and statistical data mining, and is the recipient of two IBM research division awards for his contributions to the IBM SP-2 parallel computer project. He received his B. Tech. (I.I.T. Madras, 1978), Ph.D. (UT Austin, 1984) and was Post-doctoral associate (M.I.T., 1984–1986) all in Chemical Engineering. From 1986 to 1987 he held a joint appointment as Supercomputing Specialist at Harvard University, and Senior Consultant at M.I.T. in their Information Systems department. Radu Sion is an Assistant Professor of Computer Sciences at Stony Brook University. His research interests are in the areas of Data Security and Privacy in Distributed Networked Environments. He is excited by inter-connected entities that access data and need to do so with assurances of security, privacy, and functionality, preferably efficiently. Applications include: wireless security, secure data outsourcing, queries over encrypted data, secure reputation systems, digital rights protection, integrity proofs in sensor networks, secure storage in peer to peer and ad hoc environments, data privacy and bounds on illicit inference over multiple data sources, security and policy management in computation/data grids, etc. He received his Ph.D. (2004) in Computer Sciences from Purdue University. While at Purdue, Radu was affiliated with the Center of Education and Research in Information Assurance and with the Indiana Center of Database Systems. In 2004 Radu visited with the IBM Almaden Research Center, while on leave from Stony Brook. Thomas Phan is a postdoctoral researcher at IBM Almaden Research Center. He received his Ph.D. in Computer Science from UCLA. His research interests include distributed computing, middleware, and mobile services.