- Email: [email protected]
Macroeconomics based Grid resource allocation
Future Generation Computer Systems 24 (2008) 694–700
Contents lists available at ScienceDirect
Future Generation Computer Systems journal homepage: www.elsevier.com/locate/fgcs
Macroeconomics based Grid resource allocation Peijie Huang a,b,∗ , Hong Peng a , Piyuan Lin b , Xuezhen Li c a College of Computer Science and Engineering, South China University of Technology, Guangzhou 510640, PR China b College of Informatics, South China Agricultural University, Guangzhou 510642, PR China c Department of Computer and Information Engineering, Guangdong Technical College of Water Resources and Electric Engineering, Guangzhou 510635, PR China
article
info
Article history: Received 7 April 2007 Received in revised form 16 March 2008 Accepted 24 March 2008 Available online 30 March 2008 Keywords: Grid computing Resource allocation Macroeconomics Hierarchical market
a b s t r a c t Resource allocation is the key technology in Grid computing. While most of the existing literature of economics-based Grid resource allocation relies on the basic microeconomic principle, which concerns the behavior of individual agents in the Grid, this paper provides a novel approach based on macroeconomics, which concerns large aggregate behavior instead of individual actions. First, we propose a macroeconomics-based market framework that is well suited for a service-oriented Grid. Then, some macroeconomics-based resource allocation strategies, which can effectively improve the performance of the whole Grid market, are given. Simulation results prove good performance of the proposed method. © 2008 Elsevier B.V. All rights reserved.
1. Introduction With the proliferation of the Internet comes the possibility of aggregating vast collections of computers into large-scale service platforms. A new network paradigm known as the Grid [1] articulates a vision of distributed computing in which applications “plug” into a “power Grid” of service resources including computational ability, information, knowledge, etc., when they execute, dynamically drawing what they need from the global supply [2]. A great deal of existing research concerning software mechanism that will be necessary to bring Grid to fruition is underway [3–6]. Thereby, we are able to move away from implementation details and focus on how to effectively allocate Grid resources. Because of the dynamic, heterogeneous and autonomous nature of the Grid, resource allocation methods of traditional parallel and distributed computing cannot work. How to effectively allocate Grid resources remains a challenge. Since the core issue in economics is how to effectively allocate scarce resources of the real-world society [7], economics has recently become a research hotspot of Grid resource allocation. Buyya [2] proposed and developed a distributed computational economy-based framework, called Grid Architecture for Computational Economy (GRACE), for resource allocation and to regulate supply and demand of available
∗ Corresponding author at: College of Computer Science and Engineering, South China University of Technology, Guangzhou 510640, PR China. Tel.: +86 20 31664182. E-mail address: [email protected] (P. Huang). 0167-739X/$ – see front matter © 2008 Elsevier B.V. All rights reserved. doi:10.1016/j.future.2008.03.003
resources. The proposed economics-based resource scheduling involves some optimization strategies of resource allocation. Wolski et al. [8] investigated G-commerce computational economies for controlling resource allocation in Computational Grid settings. They measured the efficiency of resource allocation under two different market conditions: commodity markets and auctions. Stuer et al. [9] continued the study based on the Wolski et al.’s [8] model and extended it to allow for trading and pricing of substitutable goods, which more closely modeled Grid markets. They also introduced improvements to the optimization algorithms used to compute equilibrium prices. Their contribution focused on commodity markets. In [10], Subramoniam et al. came up with a new algorithm for determining the price for a commodity market. The proposed algorithm for the tâtonnement process gives an efficient performance for different combinations of resources. Li et al. provided a price-directed proportional resource allocation algorithm for solving the grid task agent resource allocation problem in [11]. Huang et al. explored a market-based grid resource trading system from a cognitive computing perspective in [12]. They proposed a hybrid grid resource trading simulation framework and demonstrated a special trading case whereby user agents reserved resources by participating in sequential ascending auctions. The above works make Grid resource allocation more effective. However, they all only consider the basic microeconomic principle. Economic theory is usually divided into microeconomics and macroeconomics [7]. Microeconomics, also known as price theory, concerns the behavior of individual agents and their interaction in the market, while macroeconomics concerns large aggregate behavior (group of agents), instead of individual actions (a single
P. Huang et al. / Future Generation Computer Systems 24 (2008) 694–700
agent). Most of the existing literature of economics-based Grid resource allocation only relies on the basic microeconomic principle. However, coverage of the Grid service includes end users in different fields; therefore it may be regarded as a social problem. Due to the similarity of resource allocation in the Grid market and the real-world society, merely considering the microeconomic principle is not enough. Macroeconomic guidance and adjustment should also be introduced to maintain the efficiency and fair trading environment of the whole Grid market. The focus of this paper is on macroeconomics-based resource allocation for a service-oriented Grid. Our solution is novel in the sense that we introduce a hierarchical Grid market model, which maintains the autonomy of Grid end users, but incorporates macro information analysis and guidance of the Grid Information Center (GIC) into resource allocation. Moreover, some realizations of resource allocation strategies driven by the macroeconomic principle are proposed, which can effectively improve the performance of the whole Grid market. The rest of this paper is organized as follows. In the next section, we describe the design of macroeconomics-based Grid market architecture. Some realizations of macroeconomics-based resource allocation are given in Section 3. In Section 4, we present and discuss simulation experiments. Finally, Section 5 lists some conclusions and discusses some areas of future research. 2. Macroeconomics based grid market 2.1. Motivation and grid market architecture How to effectively match Grid tasks with available Grid resources is a challenge for a Grid computing system because of the dynamic, heterogeneous and autonomous nature of the Grid. Grid does not own local resources and therefore does not have control over them. Furthermore, the Grid does not have control over the set of tasks submitted to it. The lack of ownership and control results in embarrassment for a traditional central-controlled Grid resource scheduler [13]. This calls for a decentralized architecture, in which Grid Service Providers (GSPs) and Grid Resource Consumers (GRCs) can trade with each other autonomously. Second, both GSPs and GRCs are geographically distributed in the Grid. Visiting Grid resources located far away will cause a waste of resource and network bandwidth use. Thus, for tasks submitted to the Grid system, local resources should be searched with higher priority. One common solution is to design the hierarchical Grid market. This model looks like a hybrid architecture of a central model and a decentralized model. Our resource management architecture follows this model. Although a number of Grid models and mechanisms employ hierarchical organization [3,4,6], to the best of the authors’ knowledge, few of them has dealt with resource allocation based on hierarchical architecture. In the Grid market, GSPs contributing their resources to the Grid and GRCs using the Grid to achieve goals trade with each other autonomously. Both of them have their own expectations and strategies for being part of the Grid [2]. GIC links Grid resource users and service providers. Traditionally, most of the Grid information service, such as Metacomputing Directory Service (MDS) [14] in Globus and Grid Market Directory (GMD) [15] in Gridbus, only act as a service publication directory. Each GSP or GRC only keeps limited information. This may cause a nonoptimal resource allocation for the whole Grid market. In order to overcome this problem, we extend the capability of the GIC, and propose a macroeconomics-based Grid market architecture. The Grid system is a hierarchical architecture with several different levels of Grid market. An example of a two-level hierarchical Grid market is shown in Fig. 1. The following are the three key players of our hierarchical Grid market:
695
Fig. 1. The hierarchical grid market.
• Grid Resource Consumer (GRC): The GRCs adopt the strategies of solving their problems within their QoS constraint, such as a required timeframe and budget. They choose the service providers that best meet their requirements in the Grid market. • Grid Service Provider (GSP): GSPs can be single resource owners or they can be Grid service agents gathering resources from several resource owners. They adopt the strategy of obtaining the best possible return on their investment. GSPs may associate with the global market, but considering their spend on managing and processing the resources collected, they prioritize gathering resources of their local markets. In this paper, we take GSP as a single “virtual service provider”. How GSPs gather resources in the local market can be found in related work proposed by Kwok et al. [16]. • Grid Information Center (GIC): This service can act as an information publisher. GRCs can identify GSPs for their static and dynamic properties. We extend the capability of the GIC. It also provides information analysis and guidance. The lowest level Grid market covers a certain local network area. In local Grid markets, GSPs and GRCs trade with each other autonomously. Each local Grid market has a local GIC. The local GIC links GSPs and GRCs in its local market. The network environment covering larger organization or administrative domain forms an upper Grid market, which consists of several lower-level Grid markets. Each upper Grid market has an upper GIC. Upper GIC links local GICs within its upper market. Moreover, local GIC stores and makes some intelligent analysis of historical Grid trade data of its local Grid market, and thus can give some guidance information to GSPs. An upper GIC can make macro adjustments to the lowerlevel Grid markets within it by analyzing the aggregate supplies and demands of those lower-level Grid markets. In the Grid market, we make two assumptions: First, we assume that there is a currency called “G-money” (G$ as its unit) in Grid trade. GSPs can use it to publish the price of different kinds of resources that they provide. Second, we assume that GSPs do not cooperate, mainly due to high messaging and processing overheads associated with cooperative allocating. Instead, they act noncooperatively with the objective of maximizing their individual profits. 2.2. Grid Information Center (GIC) The architecture of the information center is shown in Fig. 2. GIC consists of two key components: the Grid Market Directory and the Intelligent Information Analyzer.
• Grid Market Directory: It provides resource registration services and maintains a list of resources available in the Grid market. It allows GSPs to publish their services in order to attract
696
P. Huang et al. / Future Generation Computer Systems 24 (2008) 694–700
Fig. 2. Grid Information Center architecture.
GRCs to inquire for their tasks. Whenever a GSP in the Grid decides to sell its resources, change its pricing structure, or update available capacity, it will connect to the local GIC and update the advertised information. • Intelligent Information Analyzer: Different types of GIC have different information analyzers. For a local GIC, the intelligent information analyzer focuses on Grid end users. By analyzing historical Grid trade data of its local Grid market, a local GIC can provide macro information guidance for GSPs to choose proper resource combinations, which may increase the profits gained by GSPs and reduce the failure rate of the Grid service requests for GRCs. For an upper GIC, the intelligent information analyzer focuses on the lower-level Grid markets within it. By analyzing the aggregate demands and supplies of lower-level Grid markets, an upper GIC can give macro adjustments in its market, improving the performance of the whole upper Grid market. The realizations of some resource allocation strategies, which have been incorporated into the GIC, are discussed in Section 3. 3. Macroeconomics based resource allocation method
In the Grid, such a virtual computing environment, although the service demand changes every day due to the dynamic nature of GRCs, we can also assume that in local Grid markets, consumer groups are relatively stable and demand behavior of individual GRCs has periodical similarity, which support us to use the historical Grid trade data to predict the future demand. In addition, although the most obvious source of economic information is from accidental observation, individual observation may suffer from limited cognition and the uncertainty of information and environment, so only economic statistics can serve as a systematic and objective source of information [18]. After a statistical period, such as a week or a month, by analyzing historical Grid trade data, local GICs can mine the important associations between different kinds of Grid resources. Data mining such as Frequent Pattern Mining in Data Streams can be used to mine frequent demand patterns, which are proper for GSPs to choose as their resource combinations [19], and statistical methods such as Principal Component Analysis can be used to analyze the optimal ratio of each resource combination [20]. GSPs can inquire of the GIC about such guiding information and use it as a target ratio, denoted as TR, for resource combination adjustment. Here we briefly introduce the resource combination adjusting strategy of GSPs according to TR. More details can be found in our early work for optimizing resource combination [20]. While a GSP may decrease the resources whose relative size in the combination is too large, for brevity, we only consider that it will increase those resources with a smaller relative size. Given a certain GSP A, suppose that there are k kinds of resources included in its resource combination, with sizes respectively as r1 , r2 , . . . , rk . According to the TR getting from local GIC, GSP A can compute the fitness degree, which is denoted as FD, of each kind of resource included as follows: , ( ) FDi =
There are many different methods of macroeconomics, which can improve the overall benefit of the economics group. In Grid markets, resource consumers are concerned with inquiry efficiency and the failure rate of Grid service requests, and service providers are concerned with their service profits, while as to the whole Grid market, good load balancing can avoid congestion of some local markets within it. In this section, some realizations of resource allocation strategies are proposed to improve the performance of the whole Grid market. In Section 3.1, we present a method to help GSPs to choose proper resource combinations. Section 3.2 presents another algorithm for task scheduling in a hierarchical Grid market. 3.1. Macro information guidance for optimizing grid resource combination In a Grid market, resources can be grouped into classes such as CPU cycles, disk space, memory space, network bandwidth, specialized processing power, etc. All GSPs should choose a certain combination between different kinds of resources before they start to provide Grid services. There are two issues GSPs should address to maximize their profits. (1) how to choose the optimal resource combination; (2) how to adjust the resource price under a certain resource combination. Most of the existing Grid literature only considers the second problem [2,8–11]. In the short run, resource combinations are fixed. Increases in (shortrun) profit result from a more effective resource pricing strategy. But in the long run, resource combinations can be varied. Getting statistical information from local GICs and then adjusting resource combinations properly can also maximize profits to GSPs [7]. In the real-world market, the future is similar to the past, which is an important assumption of time series prediction [17].
ri
maxj=1,2,...,k
tri
rj
trj
i = 1, 2, . . . , k
(1)
where tri is the component of the vector TR. Suppose that the investment to adjust the resource combination is Adj_ cos t, the adjustment of resource i is ∆ri =
(1 − FDi ) · tri k P j=1
((1 − FDj ) · trj )
Adj_ cos t
i = 1, 2, . . . , k .
(2)
The adjustment according to Eq. (2) makes fitness degree of different kinds of resources approach 1 at the same speed. In addition, we suppose that the Adj_ cos t is moderate. Once the Adj_ cos t is big enough to increase the fitness degree of all kinds of resources to 1, the excess will be used to increase all kinds of resources simply based on the TR. 3.2. Macro market adjustment for grid task scheduling The aim of the Grid market is to provide an effective service environment for Grid end users. Thus task scheduling becomes a key technology of Grid resource allocation. Typical task scheduling heuristic include on-line models, e.g., MCT and MET, and batch models, e.g., Min–Min, Max–Min, Greedy, Max–Int and Sufferage [21]. Much existing Grid task scheduling literature based on the typical heuristic has been proposed and achieved better performance in a single Grid market [2,8,22,23]. However, the above works only consider the single market. Given the hierarchical market model, how to effectively schedule Grid tasks remains an important issue. Because of the geographically distributed and dynamic nature of the Grid, different local markets may have different resource supplies and demands, which leads to an imbalance of supply and
P. Huang et al. / Future Generation Computer Systems 24 (2008) 694–700
demand among them, especially during a peak demand period. By analyzing supplies and demands of local Grid markets within it, an upper GIC can give macro adjustments in its market [24]. Suppose that there are n different kinds of Grid resources, we can denote supply and demand of each local market as nvectors. A local GIC can estimate the supply and demand by the available resources published by GSPs and the tasks submitted by GRCs within the local market. During statistical period t, suppose that there are s GSPs in the local market m, and p tasks submitted within the local market m during the peak demand period. Inspired by the fact that as time advances, old data no longer have the same importance as current data, we consider the following iterative procedure of classical time series prediction, exponentially weighted moving average (EWMA in short) [25], for GIC m to obtain new estimates of supply and demand for period t + 1, (Sm )t+1 and (Dm )t+1 :
(smi )t+1 = α · T ·
s X
rji + (1 − α) · (smi )t
i = 1, 2, . . . , n
(3)
697
3.3. Other macroeconomics factors Except for the macroeconomics-based resource allocation methods described in the above subsections, there are more macroeconomic factors that also play an important role in resource allocation in Grid markets. For instance, when new GSPs enter the market, in order to attract GRCs from the existing GSPs, they are likely to offer minimal price by under-valuing resources. This may lead to price wars, which will disturb market order. Proper market standards and codes can lead the Grid market to become efficient and lead to standardization. Moreover, supply and demand is the most common one but one also has to take into consideration different pricing policies within different countries such as taxation, consumer price index, inflation, etc. These factors are not dealt with in this paper, but implementations of a realworld Grid may need to consider them. 4. Simulation experiments
j=1
(dmi )t+1 = β ·
p X
(uji · tji ) + (1 − β) · (dmi )t i = 1, 2, . . . , n
(4)
j=1
where (smi )t+1 and (dmi )t+1 are respectively the component of vectors (Sm )t+1 and (Dm )t+1 . rji is the published resource capacity of resource i of GSPj, and T is the duration of the peak demand period. uji is the demand of task j for resource i, with an occupy time tji . (Sm )t and (Dm )t are estimates of supply and demand for period t. α and β are the factors between 0 and 1 to control the impact of the past estimates. After statistical period t, an upper GIC collects new estimates of supplies and demands from those lower-level Grid markets within it and analyzes the new estimate of excess supply capability for each resource combination of each local market. The new estimate of excess supply of local market m is denoted as an n-vector (Zm )t+1 , that is
(Zm )t+1 = (Sm )t+1 − (Dm )t+1 .
(5)
For a certain resource combination c, we denote the demand ratio of this combination of the whole Grid market as an n-vector DRc , where the 0 components of the vector indicates that those kinds of resource are not included in resource combination c. New estimates of excess supply capability for resource combination c of c local market m, (Cm )t+1 , can be computed as follow:
(Cmc )t+1 = max{0, minRi ∈c ((zmi )t+1 /drci )}
(6)
where (zmi )t+1 and drci are respectively the component of vectors (Zm )t+1 and DRc . (Cmc )t+1 is 0 indicates that local market m lacks excess supply of certain kinds of resources in combination c. Then, during period t + 1, in the hierarchical Grid market, local GIC m searches within its local market for the submitted tasks with priority. According to the excess supply capability for each resource combination of each local market getting from the upper GIC, when there is no available resource for a certain submitted task with resource combination c, the probability that local market i is chosen to inquire is, X ρci = (Cic )t+1 / (Cjc )t+1 (7) c j∈CMm
c CMm
is the candidate local market set of the local GIC m, where including those local markets that have excess supply capabilities for resource combination c and have not been searched. The local market with a larger excess supply capability has a larger probability to be chosen by GICs of other local markets to run a secondary inquiry.
In this section, the effect of the proposed method, namely macroeconomics-based Grid resource allocation (ME-Based in short), is investigated using simulation. Section 4.1 describes the general simulation setup, including the simulated Grid environment and the metrices for evaluating the usefulness of our method. Section 4.2 shows the experiment results and gives brief analysis. 4.1. Simulation setup 4.1.1. Simulated grid environment Designing a grid system is very complex system engineering, which should consider many issues brought about by sharing of global resources, such as dynamicity, heterogeneity, security, effective resource management and scheduling, network performance, fault tolerance, scalability and adaptability. The emergence of the Grid simulator provides a designer with a great convenience in evaluating system performance and verifying the design. In the simulated Grid environment, researchers can do different kinds of research, such as feasibility and performance. By configuring different parameters, a variety of actual applications can be simulated with more authentic simulation results. By analyzing simulation results, researchers can continue to improve Grid design. We have developed a simulated Grid environment based on GridSim [26] to evaluate the newly proposed ME-Based Grid resource allocation method. We have redesigned the entities of the simulator, especially the information service entity. We designed an intelligent Grid Information Center entity and incorporated macroeconomics guidance and adjustment into the Grid resource allocation. In simulation experiments, a simulated day is defined as 1440 time units, each of which stands for 1 min time. The simulated Grid system consists of 20 local markets. Each local market has 10 GSPs and random number of GRCs, uniformly distributed between 200 and 1000, simulating different service demands among different local markets. There are in total 4 kinds of Grid resource in this simulated Grid system. Each GSP owns a certain resource combination with a random resource capacity. The pricing strategy of GSPs follows the one in GridSim, that is to say resources used in the experiment are priced differently at peak hours and off-peak hours; both prices of peak hours and off-peak hours are fixed. Each Grid job requires a certain resource combination, with a random amount of units of each kind of resource included. Similar to GridSim, deadline and budget constrained scheduling are supported, i.e., GRCs adopt the strategy of solving their problems at a low cost within a required time constraint. The duration of a task is a Gaussian distribution with a
698
P. Huang et al. / Future Generation Computer Systems 24 (2008) 694–700
standard deviation of 0.8 centered about 2 h. In addition, working hours usually last from 8:00 am to 6:00 pm. So in our simulation, GRCs submit their tasks within working hours at a random time using a Poisson distribution with a mean equal to the time 2 h after the beginning of the working hours, which is 10:00 am. This results in a peak demand period during 9:00 am to 11:00 am. All experiment results are averaged over 10 independently runs of the experiments. 4.1.2. Experiment evaluations The goal of this paper is to improve the system performance of the hierarchical Grid market. Experimental evaluations include the total profit of GSPs, the inquiry efficiency and the failure rate of the Grid service request of GRCs and the load balancing of the whole Grid market. The total profit of GSPs is measured by the G$s gained by all of the GSPs. The failure rate of the Grid service request is defined as the percentage of Grid service requests that cannot be served. A low failure rate of Grid service requests is important to ensure the application feasibility of the Grid. In this paper, we observe the failure rate of Grid service requests during the peak demand period. The inquiry efficiency is measured by the average number of local markets in which Grid tasks that need secondary inquiries inquire. A smaller number of local markets in which a task needs to inquire means more effective inquiry efficiency. Suppose that there are M local markets. load balancing of the whole Grid market is measured by the balance degree during the peak service period, which is denoted as BD, that is v , u M u1 X (Oi − O)2 O BD = 1 − t (8) M i=1 where Oi is the resource occupancy rate of local market i, which is defined as the percentage of resources occupied during the peak service period. The load balancing of the whole Grid market becomes better when BD approach to 1. 4.2. Experiment results and analysis In this subsection, we evaluate our ME-Based resource allocation method by comparing it with some contrasting methods. First, we observe the total profit of all of the GSPs and the failure rate of Grid service requests of GRCs during the peak demand period, which can show the performance gain from the macro information guidance provided by local GICs. We compare the performance of our method with the following method: PR-Based: In this method, without the guiding information coming from local GICs, GSPs can only adjust their resource combinations by observing occupation of their resources. Under this limitation, the target ratio that GSPs use for resource combination adjustment is the occupation ratio of resources of the current resource combination during the peak demand period. In our experiment, investments to adjust the resource combination of all the GSPs are set to 10% of their former resource capacity. Simulation results of the total profit of all of the GSPs and the failure rate of Grid service requests are shown in Figs. 3 and 4. The method labeled as “Former” is corresponding to the case where GSPs used the former resource configuration. As we can see from Fig. 3, comparing with PR-Based, our method helps all of the GSPs gain more 277 thousand G$s. But relative to the total daily profit, there does not seem to be a great improvement. This is mainly because except at the peak demand period, during 9:00 am to 11:00 am in the experiment, the former resource configuration can already satisfy user demand. So the additional investment cannot gain more profit
Fig. 3. Total profit of all of the GSPs.
Fig. 4. The failure rate of Grid service requests during peak demand period.
Fig. 5. The average number of local markets in which Grid tasks that need secondary inquiries inquire.
during the off-peak demand period. Thus we also examine the failure rate of Grid service requests during the peak demand period. As we can see from Fig. 4, our method, effectively reducing the failure rate from 11.54% to 1.96%, gets a better performance than the contrasted one, which still maintains 5.93% of the failure rate. We then observe the inquiry efficiency of GRCs and load balancing of the whole Grid market, which can show the performance gained from the macro market adjustment provided by an upper GIC. We compare the performance of our method with two contrastive methods: Random Selection: In this method, without macro information guidance, when local GICs cannot find an appropriate GSP in their own local markets, they choose another local market randomly with uniform probability to run a secondary inquiry. Least loaded: In the Least Loaded method, local GICs choose the least loaded local market to run a secondary inquiry. Note that this method assumes that the upper GIC has up to date information about the current utilization level of lower-level markets. This can be done by keeping global knowledge with the current load-level of each market. However, in real environments the information is usually not up to date. We use a parameter p so that once a task is submitted to a local market, the load of that local market is updated only with probability p (for the results presented in this paper we use p = 0.5). Simulation results of inquiry efficiency and load balancing of the whole Grid market are shown in Figs. 5 and 6. As we can see from Fig. 5, in inquiry efficiency, comparing to Least Loaded, the performance gain of our method is about 10.4%. This could be caused by the probable failure to update load levels after each task submission while using Least Loaded. When compared with Random Selection, the performance of our method gain reaches as high as 28.8%. We then observe the load balancing of the whole Grid market, as we can see from Fig. 6, our method guides most
P. Huang et al. / Future Generation Computer Systems 24 (2008) 694–700
Fig. 6. The balance degree of the whole Grid market.
of the tasks to search in local markets with a large excess supply capability when they cannot be served in their own local markets, and thus gets better load balancing than the contrastive ones. The improvement of our method comparing to Random Selection and Least Loaded are respectively 3.1% and 0.6%. Least Loaded also outperforms Random Selection. But it should pay for the large amount of information exchanged between the upper GIC and the lower-level GICs. 5. Conclusions While most of the existing literature of economics-based Grid resource allocation only relies on the microeconomic principle, which concerns the behavior of individual agents in the Grid, this paper provides a novel approach based on macroeconomics, which concerns the overall benefit of the whole Grid market. In this paper, the macroeconomics-based market framework that is well suited for service-oriented Grid is proposed. Then, some realizations of resource allocation strategies driven by macroeconomic principle are given in the proposed Grid market. As we can see from simulation results, the proposed methods increase the total profit of all of the GSPs, and reduce the failure rate of Grid service requests and the inquiry time for resource consumers, while getting good load balancing of the whole Grid market. This study is a first attempt to incorporate the macroeconomics method into resource allocation in the Grid market. Many issues remain open. Because the commercial model of the Grid is still unclear, using a macroeconomics method for the analysis of Grid resource allocation can only be preliminary. But it still can provide necessary preparations for establishing the theoretical Grid service model. Future work includes further research of our newly proposed macroeconomics-based resource allocation method and investigation of how to combine our approach with existing microeconomics-based resource allocation methods. Acknowledgements This work is supported by National Natural Science Foundation of China (30230350) and Science and Technology Planning Project of Guangdong Province, China (A10202001, 2005B10101033, 2007A020300010).
699
[8] R. Wolski, J.S. Plank, J. Brevik, et al., Analyzing market-based resource allocation strategies for the computational Grid, International Journal of High Performance Computing Applications 15 (3) (2001) 258–281. [9] G. Stuer, K. Vanmechelena, J. Broeckhovea, A commodity market algorithm for pricing substitutable Grid resources, Future Generation Computer Systems 23 (5) (2007) 688–701. [10] K. Subramoniam, M. Maheswaran, M. Toulouse, Towards a micro-economic model for resource allocation in Grid computing systems, in: Proceedings of the 2002 IEEE Canadian Conference on Electrical & Computer Engineering, Manitoba, Canada, 2002. [11] C.L. Li, L.Y. Li, Competitive proportional resource allocation policy for computational grid, Future Generation Computer Systems 20 (6) (2004) 1041–1054. [12] Z.X. Huang, Y.H. Qiu, Resource trading using cognitive agents: A hybrid perspective and its simulation, Future Generation Computer Systems 23 (7) (2007) 837–845. [13] Q.F. Fu, S.B. Yang, M.S. Li, et al., Decentralized computational market model for Grid resource management, in: Proceedings of the Second International Workshop on Grid and Cooperative Computing, GCC 2003, Shanghai, China, in: Lecture Notes in Computer Science, vol. 3033, Springer-Verlag, 2004, pp. 227–230. [14] S. Fitzgerald, I. Foster, C. Kesselman, et al., A directory service for configuring high-performance distributed computations, 6th IEEE Symposium on High Performance Distributed Computing, HPDC’97, Portland, OR, USA, 1997. [15] J. Yu, S. Venugopal, R. Buyya, A market-oriented Grid directory service for publication and discovery of Grid service providers and their services, The Journal of Supercomputing 36 (1) (2006) 17–31. [16] Y.K. Kwok, S. Song, K. Hwang, Selfish grid computing: Game-theoretic modeling and NAS performance results, in: Proceedings of the 2005 IEEE International Symposium on Cluster Computing and the Grid, CCGrid 2005, 2005. [17] J.H. Stock, M.W. Watson, Introduction to Econometrics, Addison Wesley, Boston, 2003. [18] N.G. Mankiw, Macroeconomics, 4th ed., Worth Publisher, New York, 2000. [19] P.J. Huang, H. Peng, Q.L. Zheng, X.Z. Li, Optimizing grid resource combination using frequent pattern mining, in: Proceedings of the 2nd International Conference on Computer Intelligence and Security, November 2006. [20] P.J. Huang, H. Peng, Q.L. Zheng, Mining optimal resource combination in computational Grid, in: Proceedings of the 5th International Conference on Machine Learning and Cybernetics, August 2006. [21] M. Maheswaran, S. Ali, H.J. Siegel, et al., Dynamic mapping of a class of independent tasks onto heterogeneous computing systems, in: the 8th IEEE Heterogeneous Computing Workshop, HCW ’99, San Juan, Puerto Rico, April 1999, pp. 30–44. [22] H. Casanova, A. Legrand, D. Zagorodnov, F. Berman, Heuristics for scheduling parameter sweep applications in Grid environments, in: Proc. of the 9th Heterogeneous Computing Workshop, HCW’ 2000, Cancun, Mexico, 2000, pp. 349–363. [23] C.L. Li, L.Y. Li, QoS based resource scheduling by computational economy in computational grid, Information Processing Letters 98 (3) (2006) 119–126. [24] P.J. Huang, H. Peng, X.Z. Li, Macro adjustment based task scheduling in hierarchical grid market, in: Proceedings of the 7th International Conference on Computational Science, ICCS 2007, Beijing, China, in: Lecture Notes in Computer Science, vol. 4487, Springer-Verlag, 2007, pp. 430–433. [25] J.S. Hunter, The exponentially weighted moving average, Journal of Quality Technology 18 (4) (1986) 203–209. [26] R. Buyya, M. Murshed, GridSim: A toolkit for the modeling and simulation of distributed resource management and scheduling for Grid computing, The Journal of Concurrency and Computation: Practice and Experience 14 (13–15) (2002) 1175–1220.
Peijie Huang received the B.S. degree in computer science from South China University of Technology. He is currently a Ph.D. candidate in College of Computer Science and Engineering, South China University of Technology. His research interests include Grid computing and intelligent computing.
References [1] I. Foster, C. Kesselman, The Grid: Blueprint for a New Computing Infrastructure, Morgan Kaufmann Publishers, Inc., 1998. [2] R. Buyya, Economic-based distributed resource management and scheduling for Grid computing, Ph.D. Thesis, Melbourne, Monash University, 2002. [3] I. Foster, C. Kesselman, Globus: A meta-computing infrastructure toolkit, International Journal of Supercomputer Applications (1997). [4] S. Chapin, D. Katramatos, J. Karpovich, et al., Resource management in Legion, Future Generation Computer Systems 15 (5–6) (1999) 583–594. [5] T. Tannenbaum, M. Litzkow, The Condor distributed processing system, Dr. Dobbs Journal (1995). [6] D. Abramson, J. Giddy, I. Foster, L. Kotler, High performance parametric modeling with Nimrod/G: Killer application for the global Grid?, in: Proceedings of the International Parallel and Distributed Processing Symposium, May 2000. [7] B.R. Schiller, The Economy Today, 8th ed., McGraw-Hill Irwin, New York, 2000.
Hong Peng received the Ph.D. degree in computer science from Xi’an Jiaotong University and has done postdoctoral research at Zhejiang University. He is currently a Professor and Ph.D. tutor of computer science in South China University of Technology. He has presided over and participated in 15 projects supported by National Natural Science Foundation, 863 Program and Provincial Hightech Program of Guangdong etc. His research interests are in the areas of intelligent network computing, intelligent business and data mining. Professor Peng has published over 40 research papers.
700
P. Huang et al. / Future Generation Computer Systems 24 (2008) 694–700
Piyuan Lin received the B.S. and M.S degrees both in Computer Science from University of Electronic Science and Technology of China (UESTC) respectively in 1984 and 1989 respectively. He is currently a senior member of China Computer Federation (CCF), a professor and the vice-dean of College of Informatics, South China Agricultural University (SCAU). He has presided over and participated in 21 projects supported by National Natural Science Foundation, 863 Program, Provincial High-tech Program of Guangdong, and some companies. His research interests include bioinformatics, data mining, and network & information security. Professor Lin has published more than 40 research papers and 22 books.
Xuezhen Li received the B.S. degree in computer science from South China Normal University and the M.E. degree in computer science from South China University of Technology. She is currently a Ph.D. candidate in College of Computer and Information Engineering, HoHai University. She is now a lecturer of computer science in Guangdong Technical College of Water Resources and Electric Engineering. Her research interests include intelligent computing and hydro-information technology.
Contents lists available at ScienceDirect
Future Generation Computer Systems journal homepage: www.elsevier.com/locate/fgcs
Macroeconomics based Grid resource allocation Peijie Huang a,b,∗ , Hong Peng a , Piyuan Lin b , Xuezhen Li c a College of Computer Science and Engineering, South China University of Technology, Guangzhou 510640, PR China b College of Informatics, South China Agricultural University, Guangzhou 510642, PR China c Department of Computer and Information Engineering, Guangdong Technical College of Water Resources and Electric Engineering, Guangzhou 510635, PR China
article
info
Article history: Received 7 April 2007 Received in revised form 16 March 2008 Accepted 24 March 2008 Available online 30 March 2008 Keywords: Grid computing Resource allocation Macroeconomics Hierarchical market
a b s t r a c t Resource allocation is the key technology in Grid computing. While most of the existing literature of economics-based Grid resource allocation relies on the basic microeconomic principle, which concerns the behavior of individual agents in the Grid, this paper provides a novel approach based on macroeconomics, which concerns large aggregate behavior instead of individual actions. First, we propose a macroeconomics-based market framework that is well suited for a service-oriented Grid. Then, some macroeconomics-based resource allocation strategies, which can effectively improve the performance of the whole Grid market, are given. Simulation results prove good performance of the proposed method. © 2008 Elsevier B.V. All rights reserved.
1. Introduction With the proliferation of the Internet comes the possibility of aggregating vast collections of computers into large-scale service platforms. A new network paradigm known as the Grid [1] articulates a vision of distributed computing in which applications “plug” into a “power Grid” of service resources including computational ability, information, knowledge, etc., when they execute, dynamically drawing what they need from the global supply [2]. A great deal of existing research concerning software mechanism that will be necessary to bring Grid to fruition is underway [3–6]. Thereby, we are able to move away from implementation details and focus on how to effectively allocate Grid resources. Because of the dynamic, heterogeneous and autonomous nature of the Grid, resource allocation methods of traditional parallel and distributed computing cannot work. How to effectively allocate Grid resources remains a challenge. Since the core issue in economics is how to effectively allocate scarce resources of the real-world society [7], economics has recently become a research hotspot of Grid resource allocation. Buyya [2] proposed and developed a distributed computational economy-based framework, called Grid Architecture for Computational Economy (GRACE), for resource allocation and to regulate supply and demand of available
∗ Corresponding author at: College of Computer Science and Engineering, South China University of Technology, Guangzhou 510640, PR China. Tel.: +86 20 31664182. E-mail address: [email protected] (P. Huang). 0167-739X/$ – see front matter © 2008 Elsevier B.V. All rights reserved. doi:10.1016/j.future.2008.03.003
resources. The proposed economics-based resource scheduling involves some optimization strategies of resource allocation. Wolski et al. [8] investigated G-commerce computational economies for controlling resource allocation in Computational Grid settings. They measured the efficiency of resource allocation under two different market conditions: commodity markets and auctions. Stuer et al. [9] continued the study based on the Wolski et al.’s [8] model and extended it to allow for trading and pricing of substitutable goods, which more closely modeled Grid markets. They also introduced improvements to the optimization algorithms used to compute equilibrium prices. Their contribution focused on commodity markets. In [10], Subramoniam et al. came up with a new algorithm for determining the price for a commodity market. The proposed algorithm for the tâtonnement process gives an efficient performance for different combinations of resources. Li et al. provided a price-directed proportional resource allocation algorithm for solving the grid task agent resource allocation problem in [11]. Huang et al. explored a market-based grid resource trading system from a cognitive computing perspective in [12]. They proposed a hybrid grid resource trading simulation framework and demonstrated a special trading case whereby user agents reserved resources by participating in sequential ascending auctions. The above works make Grid resource allocation more effective. However, they all only consider the basic microeconomic principle. Economic theory is usually divided into microeconomics and macroeconomics [7]. Microeconomics, also known as price theory, concerns the behavior of individual agents and their interaction in the market, while macroeconomics concerns large aggregate behavior (group of agents), instead of individual actions (a single
P. Huang et al. / Future Generation Computer Systems 24 (2008) 694–700
agent). Most of the existing literature of economics-based Grid resource allocation only relies on the basic microeconomic principle. However, coverage of the Grid service includes end users in different fields; therefore it may be regarded as a social problem. Due to the similarity of resource allocation in the Grid market and the real-world society, merely considering the microeconomic principle is not enough. Macroeconomic guidance and adjustment should also be introduced to maintain the efficiency and fair trading environment of the whole Grid market. The focus of this paper is on macroeconomics-based resource allocation for a service-oriented Grid. Our solution is novel in the sense that we introduce a hierarchical Grid market model, which maintains the autonomy of Grid end users, but incorporates macro information analysis and guidance of the Grid Information Center (GIC) into resource allocation. Moreover, some realizations of resource allocation strategies driven by the macroeconomic principle are proposed, which can effectively improve the performance of the whole Grid market. The rest of this paper is organized as follows. In the next section, we describe the design of macroeconomics-based Grid market architecture. Some realizations of macroeconomics-based resource allocation are given in Section 3. In Section 4, we present and discuss simulation experiments. Finally, Section 5 lists some conclusions and discusses some areas of future research. 2. Macroeconomics based grid market 2.1. Motivation and grid market architecture How to effectively match Grid tasks with available Grid resources is a challenge for a Grid computing system because of the dynamic, heterogeneous and autonomous nature of the Grid. Grid does not own local resources and therefore does not have control over them. Furthermore, the Grid does not have control over the set of tasks submitted to it. The lack of ownership and control results in embarrassment for a traditional central-controlled Grid resource scheduler [13]. This calls for a decentralized architecture, in which Grid Service Providers (GSPs) and Grid Resource Consumers (GRCs) can trade with each other autonomously. Second, both GSPs and GRCs are geographically distributed in the Grid. Visiting Grid resources located far away will cause a waste of resource and network bandwidth use. Thus, for tasks submitted to the Grid system, local resources should be searched with higher priority. One common solution is to design the hierarchical Grid market. This model looks like a hybrid architecture of a central model and a decentralized model. Our resource management architecture follows this model. Although a number of Grid models and mechanisms employ hierarchical organization [3,4,6], to the best of the authors’ knowledge, few of them has dealt with resource allocation based on hierarchical architecture. In the Grid market, GSPs contributing their resources to the Grid and GRCs using the Grid to achieve goals trade with each other autonomously. Both of them have their own expectations and strategies for being part of the Grid [2]. GIC links Grid resource users and service providers. Traditionally, most of the Grid information service, such as Metacomputing Directory Service (MDS) [14] in Globus and Grid Market Directory (GMD) [15] in Gridbus, only act as a service publication directory. Each GSP or GRC only keeps limited information. This may cause a nonoptimal resource allocation for the whole Grid market. In order to overcome this problem, we extend the capability of the GIC, and propose a macroeconomics-based Grid market architecture. The Grid system is a hierarchical architecture with several different levels of Grid market. An example of a two-level hierarchical Grid market is shown in Fig. 1. The following are the three key players of our hierarchical Grid market:
695
Fig. 1. The hierarchical grid market.
• Grid Resource Consumer (GRC): The GRCs adopt the strategies of solving their problems within their QoS constraint, such as a required timeframe and budget. They choose the service providers that best meet their requirements in the Grid market. • Grid Service Provider (GSP): GSPs can be single resource owners or they can be Grid service agents gathering resources from several resource owners. They adopt the strategy of obtaining the best possible return on their investment. GSPs may associate with the global market, but considering their spend on managing and processing the resources collected, they prioritize gathering resources of their local markets. In this paper, we take GSP as a single “virtual service provider”. How GSPs gather resources in the local market can be found in related work proposed by Kwok et al. [16]. • Grid Information Center (GIC): This service can act as an information publisher. GRCs can identify GSPs for their static and dynamic properties. We extend the capability of the GIC. It also provides information analysis and guidance. The lowest level Grid market covers a certain local network area. In local Grid markets, GSPs and GRCs trade with each other autonomously. Each local Grid market has a local GIC. The local GIC links GSPs and GRCs in its local market. The network environment covering larger organization or administrative domain forms an upper Grid market, which consists of several lower-level Grid markets. Each upper Grid market has an upper GIC. Upper GIC links local GICs within its upper market. Moreover, local GIC stores and makes some intelligent analysis of historical Grid trade data of its local Grid market, and thus can give some guidance information to GSPs. An upper GIC can make macro adjustments to the lowerlevel Grid markets within it by analyzing the aggregate supplies and demands of those lower-level Grid markets. In the Grid market, we make two assumptions: First, we assume that there is a currency called “G-money” (G$ as its unit) in Grid trade. GSPs can use it to publish the price of different kinds of resources that they provide. Second, we assume that GSPs do not cooperate, mainly due to high messaging and processing overheads associated with cooperative allocating. Instead, they act noncooperatively with the objective of maximizing their individual profits. 2.2. Grid Information Center (GIC) The architecture of the information center is shown in Fig. 2. GIC consists of two key components: the Grid Market Directory and the Intelligent Information Analyzer.
• Grid Market Directory: It provides resource registration services and maintains a list of resources available in the Grid market. It allows GSPs to publish their services in order to attract
696
P. Huang et al. / Future Generation Computer Systems 24 (2008) 694–700
Fig. 2. Grid Information Center architecture.
GRCs to inquire for their tasks. Whenever a GSP in the Grid decides to sell its resources, change its pricing structure, or update available capacity, it will connect to the local GIC and update the advertised information. • Intelligent Information Analyzer: Different types of GIC have different information analyzers. For a local GIC, the intelligent information analyzer focuses on Grid end users. By analyzing historical Grid trade data of its local Grid market, a local GIC can provide macro information guidance for GSPs to choose proper resource combinations, which may increase the profits gained by GSPs and reduce the failure rate of the Grid service requests for GRCs. For an upper GIC, the intelligent information analyzer focuses on the lower-level Grid markets within it. By analyzing the aggregate demands and supplies of lower-level Grid markets, an upper GIC can give macro adjustments in its market, improving the performance of the whole upper Grid market. The realizations of some resource allocation strategies, which have been incorporated into the GIC, are discussed in Section 3. 3. Macroeconomics based resource allocation method
In the Grid, such a virtual computing environment, although the service demand changes every day due to the dynamic nature of GRCs, we can also assume that in local Grid markets, consumer groups are relatively stable and demand behavior of individual GRCs has periodical similarity, which support us to use the historical Grid trade data to predict the future demand. In addition, although the most obvious source of economic information is from accidental observation, individual observation may suffer from limited cognition and the uncertainty of information and environment, so only economic statistics can serve as a systematic and objective source of information [18]. After a statistical period, such as a week or a month, by analyzing historical Grid trade data, local GICs can mine the important associations between different kinds of Grid resources. Data mining such as Frequent Pattern Mining in Data Streams can be used to mine frequent demand patterns, which are proper for GSPs to choose as their resource combinations [19], and statistical methods such as Principal Component Analysis can be used to analyze the optimal ratio of each resource combination [20]. GSPs can inquire of the GIC about such guiding information and use it as a target ratio, denoted as TR, for resource combination adjustment. Here we briefly introduce the resource combination adjusting strategy of GSPs according to TR. More details can be found in our early work for optimizing resource combination [20]. While a GSP may decrease the resources whose relative size in the combination is too large, for brevity, we only consider that it will increase those resources with a smaller relative size. Given a certain GSP A, suppose that there are k kinds of resources included in its resource combination, with sizes respectively as r1 , r2 , . . . , rk . According to the TR getting from local GIC, GSP A can compute the fitness degree, which is denoted as FD, of each kind of resource included as follows: , ( ) FDi =
There are many different methods of macroeconomics, which can improve the overall benefit of the economics group. In Grid markets, resource consumers are concerned with inquiry efficiency and the failure rate of Grid service requests, and service providers are concerned with their service profits, while as to the whole Grid market, good load balancing can avoid congestion of some local markets within it. In this section, some realizations of resource allocation strategies are proposed to improve the performance of the whole Grid market. In Section 3.1, we present a method to help GSPs to choose proper resource combinations. Section 3.2 presents another algorithm for task scheduling in a hierarchical Grid market. 3.1. Macro information guidance for optimizing grid resource combination In a Grid market, resources can be grouped into classes such as CPU cycles, disk space, memory space, network bandwidth, specialized processing power, etc. All GSPs should choose a certain combination between different kinds of resources before they start to provide Grid services. There are two issues GSPs should address to maximize their profits. (1) how to choose the optimal resource combination; (2) how to adjust the resource price under a certain resource combination. Most of the existing Grid literature only considers the second problem [2,8–11]. In the short run, resource combinations are fixed. Increases in (shortrun) profit result from a more effective resource pricing strategy. But in the long run, resource combinations can be varied. Getting statistical information from local GICs and then adjusting resource combinations properly can also maximize profits to GSPs [7]. In the real-world market, the future is similar to the past, which is an important assumption of time series prediction [17].
ri
maxj=1,2,...,k
tri
rj
trj
i = 1, 2, . . . , k
(1)
where tri is the component of the vector TR. Suppose that the investment to adjust the resource combination is Adj_ cos t, the adjustment of resource i is ∆ri =
(1 − FDi ) · tri k P j=1
((1 − FDj ) · trj )
Adj_ cos t
i = 1, 2, . . . , k .
(2)
The adjustment according to Eq. (2) makes fitness degree of different kinds of resources approach 1 at the same speed. In addition, we suppose that the Adj_ cos t is moderate. Once the Adj_ cos t is big enough to increase the fitness degree of all kinds of resources to 1, the excess will be used to increase all kinds of resources simply based on the TR. 3.2. Macro market adjustment for grid task scheduling The aim of the Grid market is to provide an effective service environment for Grid end users. Thus task scheduling becomes a key technology of Grid resource allocation. Typical task scheduling heuristic include on-line models, e.g., MCT and MET, and batch models, e.g., Min–Min, Max–Min, Greedy, Max–Int and Sufferage [21]. Much existing Grid task scheduling literature based on the typical heuristic has been proposed and achieved better performance in a single Grid market [2,8,22,23]. However, the above works only consider the single market. Given the hierarchical market model, how to effectively schedule Grid tasks remains an important issue. Because of the geographically distributed and dynamic nature of the Grid, different local markets may have different resource supplies and demands, which leads to an imbalance of supply and
P. Huang et al. / Future Generation Computer Systems 24 (2008) 694–700
demand among them, especially during a peak demand period. By analyzing supplies and demands of local Grid markets within it, an upper GIC can give macro adjustments in its market [24]. Suppose that there are n different kinds of Grid resources, we can denote supply and demand of each local market as nvectors. A local GIC can estimate the supply and demand by the available resources published by GSPs and the tasks submitted by GRCs within the local market. During statistical period t, suppose that there are s GSPs in the local market m, and p tasks submitted within the local market m during the peak demand period. Inspired by the fact that as time advances, old data no longer have the same importance as current data, we consider the following iterative procedure of classical time series prediction, exponentially weighted moving average (EWMA in short) [25], for GIC m to obtain new estimates of supply and demand for period t + 1, (Sm )t+1 and (Dm )t+1 :
(smi )t+1 = α · T ·
s X
rji + (1 − α) · (smi )t
i = 1, 2, . . . , n
(3)
697
3.3. Other macroeconomics factors Except for the macroeconomics-based resource allocation methods described in the above subsections, there are more macroeconomic factors that also play an important role in resource allocation in Grid markets. For instance, when new GSPs enter the market, in order to attract GRCs from the existing GSPs, they are likely to offer minimal price by under-valuing resources. This may lead to price wars, which will disturb market order. Proper market standards and codes can lead the Grid market to become efficient and lead to standardization. Moreover, supply and demand is the most common one but one also has to take into consideration different pricing policies within different countries such as taxation, consumer price index, inflation, etc. These factors are not dealt with in this paper, but implementations of a realworld Grid may need to consider them. 4. Simulation experiments
j=1
(dmi )t+1 = β ·
p X
(uji · tji ) + (1 − β) · (dmi )t i = 1, 2, . . . , n
(4)
j=1
where (smi )t+1 and (dmi )t+1 are respectively the component of vectors (Sm )t+1 and (Dm )t+1 . rji is the published resource capacity of resource i of GSPj, and T is the duration of the peak demand period. uji is the demand of task j for resource i, with an occupy time tji . (Sm )t and (Dm )t are estimates of supply and demand for period t. α and β are the factors between 0 and 1 to control the impact of the past estimates. After statistical period t, an upper GIC collects new estimates of supplies and demands from those lower-level Grid markets within it and analyzes the new estimate of excess supply capability for each resource combination of each local market. The new estimate of excess supply of local market m is denoted as an n-vector (Zm )t+1 , that is
(Zm )t+1 = (Sm )t+1 − (Dm )t+1 .
(5)
For a certain resource combination c, we denote the demand ratio of this combination of the whole Grid market as an n-vector DRc , where the 0 components of the vector indicates that those kinds of resource are not included in resource combination c. New estimates of excess supply capability for resource combination c of c local market m, (Cm )t+1 , can be computed as follow:
(Cmc )t+1 = max{0, minRi ∈c ((zmi )t+1 /drci )}
(6)
where (zmi )t+1 and drci are respectively the component of vectors (Zm )t+1 and DRc . (Cmc )t+1 is 0 indicates that local market m lacks excess supply of certain kinds of resources in combination c. Then, during period t + 1, in the hierarchical Grid market, local GIC m searches within its local market for the submitted tasks with priority. According to the excess supply capability for each resource combination of each local market getting from the upper GIC, when there is no available resource for a certain submitted task with resource combination c, the probability that local market i is chosen to inquire is, X ρci = (Cic )t+1 / (Cjc )t+1 (7) c j∈CMm
c CMm
is the candidate local market set of the local GIC m, where including those local markets that have excess supply capabilities for resource combination c and have not been searched. The local market with a larger excess supply capability has a larger probability to be chosen by GICs of other local markets to run a secondary inquiry.
In this section, the effect of the proposed method, namely macroeconomics-based Grid resource allocation (ME-Based in short), is investigated using simulation. Section 4.1 describes the general simulation setup, including the simulated Grid environment and the metrices for evaluating the usefulness of our method. Section 4.2 shows the experiment results and gives brief analysis. 4.1. Simulation setup 4.1.1. Simulated grid environment Designing a grid system is very complex system engineering, which should consider many issues brought about by sharing of global resources, such as dynamicity, heterogeneity, security, effective resource management and scheduling, network performance, fault tolerance, scalability and adaptability. The emergence of the Grid simulator provides a designer with a great convenience in evaluating system performance and verifying the design. In the simulated Grid environment, researchers can do different kinds of research, such as feasibility and performance. By configuring different parameters, a variety of actual applications can be simulated with more authentic simulation results. By analyzing simulation results, researchers can continue to improve Grid design. We have developed a simulated Grid environment based on GridSim [26] to evaluate the newly proposed ME-Based Grid resource allocation method. We have redesigned the entities of the simulator, especially the information service entity. We designed an intelligent Grid Information Center entity and incorporated macroeconomics guidance and adjustment into the Grid resource allocation. In simulation experiments, a simulated day is defined as 1440 time units, each of which stands for 1 min time. The simulated Grid system consists of 20 local markets. Each local market has 10 GSPs and random number of GRCs, uniformly distributed between 200 and 1000, simulating different service demands among different local markets. There are in total 4 kinds of Grid resource in this simulated Grid system. Each GSP owns a certain resource combination with a random resource capacity. The pricing strategy of GSPs follows the one in GridSim, that is to say resources used in the experiment are priced differently at peak hours and off-peak hours; both prices of peak hours and off-peak hours are fixed. Each Grid job requires a certain resource combination, with a random amount of units of each kind of resource included. Similar to GridSim, deadline and budget constrained scheduling are supported, i.e., GRCs adopt the strategy of solving their problems at a low cost within a required time constraint. The duration of a task is a Gaussian distribution with a
698
P. Huang et al. / Future Generation Computer Systems 24 (2008) 694–700
standard deviation of 0.8 centered about 2 h. In addition, working hours usually last from 8:00 am to 6:00 pm. So in our simulation, GRCs submit their tasks within working hours at a random time using a Poisson distribution with a mean equal to the time 2 h after the beginning of the working hours, which is 10:00 am. This results in a peak demand period during 9:00 am to 11:00 am. All experiment results are averaged over 10 independently runs of the experiments. 4.1.2. Experiment evaluations The goal of this paper is to improve the system performance of the hierarchical Grid market. Experimental evaluations include the total profit of GSPs, the inquiry efficiency and the failure rate of the Grid service request of GRCs and the load balancing of the whole Grid market. The total profit of GSPs is measured by the G$s gained by all of the GSPs. The failure rate of the Grid service request is defined as the percentage of Grid service requests that cannot be served. A low failure rate of Grid service requests is important to ensure the application feasibility of the Grid. In this paper, we observe the failure rate of Grid service requests during the peak demand period. The inquiry efficiency is measured by the average number of local markets in which Grid tasks that need secondary inquiries inquire. A smaller number of local markets in which a task needs to inquire means more effective inquiry efficiency. Suppose that there are M local markets. load balancing of the whole Grid market is measured by the balance degree during the peak service period, which is denoted as BD, that is v , u M u1 X (Oi − O)2 O BD = 1 − t (8) M i=1 where Oi is the resource occupancy rate of local market i, which is defined as the percentage of resources occupied during the peak service period. The load balancing of the whole Grid market becomes better when BD approach to 1. 4.2. Experiment results and analysis In this subsection, we evaluate our ME-Based resource allocation method by comparing it with some contrasting methods. First, we observe the total profit of all of the GSPs and the failure rate of Grid service requests of GRCs during the peak demand period, which can show the performance gain from the macro information guidance provided by local GICs. We compare the performance of our method with the following method: PR-Based: In this method, without the guiding information coming from local GICs, GSPs can only adjust their resource combinations by observing occupation of their resources. Under this limitation, the target ratio that GSPs use for resource combination adjustment is the occupation ratio of resources of the current resource combination during the peak demand period. In our experiment, investments to adjust the resource combination of all the GSPs are set to 10% of their former resource capacity. Simulation results of the total profit of all of the GSPs and the failure rate of Grid service requests are shown in Figs. 3 and 4. The method labeled as “Former” is corresponding to the case where GSPs used the former resource configuration. As we can see from Fig. 3, comparing with PR-Based, our method helps all of the GSPs gain more 277 thousand G$s. But relative to the total daily profit, there does not seem to be a great improvement. This is mainly because except at the peak demand period, during 9:00 am to 11:00 am in the experiment, the former resource configuration can already satisfy user demand. So the additional investment cannot gain more profit
Fig. 3. Total profit of all of the GSPs.
Fig. 4. The failure rate of Grid service requests during peak demand period.
Fig. 5. The average number of local markets in which Grid tasks that need secondary inquiries inquire.
during the off-peak demand period. Thus we also examine the failure rate of Grid service requests during the peak demand period. As we can see from Fig. 4, our method, effectively reducing the failure rate from 11.54% to 1.96%, gets a better performance than the contrasted one, which still maintains 5.93% of the failure rate. We then observe the inquiry efficiency of GRCs and load balancing of the whole Grid market, which can show the performance gained from the macro market adjustment provided by an upper GIC. We compare the performance of our method with two contrastive methods: Random Selection: In this method, without macro information guidance, when local GICs cannot find an appropriate GSP in their own local markets, they choose another local market randomly with uniform probability to run a secondary inquiry. Least loaded: In the Least Loaded method, local GICs choose the least loaded local market to run a secondary inquiry. Note that this method assumes that the upper GIC has up to date information about the current utilization level of lower-level markets. This can be done by keeping global knowledge with the current load-level of each market. However, in real environments the information is usually not up to date. We use a parameter p so that once a task is submitted to a local market, the load of that local market is updated only with probability p (for the results presented in this paper we use p = 0.5). Simulation results of inquiry efficiency and load balancing of the whole Grid market are shown in Figs. 5 and 6. As we can see from Fig. 5, in inquiry efficiency, comparing to Least Loaded, the performance gain of our method is about 10.4%. This could be caused by the probable failure to update load levels after each task submission while using Least Loaded. When compared with Random Selection, the performance of our method gain reaches as high as 28.8%. We then observe the load balancing of the whole Grid market, as we can see from Fig. 6, our method guides most
P. Huang et al. / Future Generation Computer Systems 24 (2008) 694–700
Fig. 6. The balance degree of the whole Grid market.
of the tasks to search in local markets with a large excess supply capability when they cannot be served in their own local markets, and thus gets better load balancing than the contrastive ones. The improvement of our method comparing to Random Selection and Least Loaded are respectively 3.1% and 0.6%. Least Loaded also outperforms Random Selection. But it should pay for the large amount of information exchanged between the upper GIC and the lower-level GICs. 5. Conclusions While most of the existing literature of economics-based Grid resource allocation only relies on the microeconomic principle, which concerns the behavior of individual agents in the Grid, this paper provides a novel approach based on macroeconomics, which concerns the overall benefit of the whole Grid market. In this paper, the macroeconomics-based market framework that is well suited for service-oriented Grid is proposed. Then, some realizations of resource allocation strategies driven by macroeconomic principle are given in the proposed Grid market. As we can see from simulation results, the proposed methods increase the total profit of all of the GSPs, and reduce the failure rate of Grid service requests and the inquiry time for resource consumers, while getting good load balancing of the whole Grid market. This study is a first attempt to incorporate the macroeconomics method into resource allocation in the Grid market. Many issues remain open. Because the commercial model of the Grid is still unclear, using a macroeconomics method for the analysis of Grid resource allocation can only be preliminary. But it still can provide necessary preparations for establishing the theoretical Grid service model. Future work includes further research of our newly proposed macroeconomics-based resource allocation method and investigation of how to combine our approach with existing microeconomics-based resource allocation methods. Acknowledgements This work is supported by National Natural Science Foundation of China (30230350) and Science and Technology Planning Project of Guangdong Province, China (A10202001, 2005B10101033, 2007A020300010).
699
[8] R. Wolski, J.S. Plank, J. Brevik, et al., Analyzing market-based resource allocation strategies for the computational Grid, International Journal of High Performance Computing Applications 15 (3) (2001) 258–281. [9] G. Stuer, K. Vanmechelena, J. Broeckhovea, A commodity market algorithm for pricing substitutable Grid resources, Future Generation Computer Systems 23 (5) (2007) 688–701. [10] K. Subramoniam, M. Maheswaran, M. Toulouse, Towards a micro-economic model for resource allocation in Grid computing systems, in: Proceedings of the 2002 IEEE Canadian Conference on Electrical & Computer Engineering, Manitoba, Canada, 2002. [11] C.L. Li, L.Y. Li, Competitive proportional resource allocation policy for computational grid, Future Generation Computer Systems 20 (6) (2004) 1041–1054. [12] Z.X. Huang, Y.H. Qiu, Resource trading using cognitive agents: A hybrid perspective and its simulation, Future Generation Computer Systems 23 (7) (2007) 837–845. [13] Q.F. Fu, S.B. Yang, M.S. Li, et al., Decentralized computational market model for Grid resource management, in: Proceedings of the Second International Workshop on Grid and Cooperative Computing, GCC 2003, Shanghai, China, in: Lecture Notes in Computer Science, vol. 3033, Springer-Verlag, 2004, pp. 227–230. [14] S. Fitzgerald, I. Foster, C. Kesselman, et al., A directory service for configuring high-performance distributed computations, 6th IEEE Symposium on High Performance Distributed Computing, HPDC’97, Portland, OR, USA, 1997. [15] J. Yu, S. Venugopal, R. Buyya, A market-oriented Grid directory service for publication and discovery of Grid service providers and their services, The Journal of Supercomputing 36 (1) (2006) 17–31. [16] Y.K. Kwok, S. Song, K. Hwang, Selfish grid computing: Game-theoretic modeling and NAS performance results, in: Proceedings of the 2005 IEEE International Symposium on Cluster Computing and the Grid, CCGrid 2005, 2005. [17] J.H. Stock, M.W. Watson, Introduction to Econometrics, Addison Wesley, Boston, 2003. [18] N.G. Mankiw, Macroeconomics, 4th ed., Worth Publisher, New York, 2000. [19] P.J. Huang, H. Peng, Q.L. Zheng, X.Z. Li, Optimizing grid resource combination using frequent pattern mining, in: Proceedings of the 2nd International Conference on Computer Intelligence and Security, November 2006. [20] P.J. Huang, H. Peng, Q.L. Zheng, Mining optimal resource combination in computational Grid, in: Proceedings of the 5th International Conference on Machine Learning and Cybernetics, August 2006. [21] M. Maheswaran, S. Ali, H.J. Siegel, et al., Dynamic mapping of a class of independent tasks onto heterogeneous computing systems, in: the 8th IEEE Heterogeneous Computing Workshop, HCW ’99, San Juan, Puerto Rico, April 1999, pp. 30–44. [22] H. Casanova, A. Legrand, D. Zagorodnov, F. Berman, Heuristics for scheduling parameter sweep applications in Grid environments, in: Proc. of the 9th Heterogeneous Computing Workshop, HCW’ 2000, Cancun, Mexico, 2000, pp. 349–363. [23] C.L. Li, L.Y. Li, QoS based resource scheduling by computational economy in computational grid, Information Processing Letters 98 (3) (2006) 119–126. [24] P.J. Huang, H. Peng, X.Z. Li, Macro adjustment based task scheduling in hierarchical grid market, in: Proceedings of the 7th International Conference on Computational Science, ICCS 2007, Beijing, China, in: Lecture Notes in Computer Science, vol. 4487, Springer-Verlag, 2007, pp. 430–433. [25] J.S. Hunter, The exponentially weighted moving average, Journal of Quality Technology 18 (4) (1986) 203–209. [26] R. Buyya, M. Murshed, GridSim: A toolkit for the modeling and simulation of distributed resource management and scheduling for Grid computing, The Journal of Concurrency and Computation: Practice and Experience 14 (13–15) (2002) 1175–1220.
Peijie Huang received the B.S. degree in computer science from South China University of Technology. He is currently a Ph.D. candidate in College of Computer Science and Engineering, South China University of Technology. His research interests include Grid computing and intelligent computing.
References [1] I. Foster, C. Kesselman, The Grid: Blueprint for a New Computing Infrastructure, Morgan Kaufmann Publishers, Inc., 1998. [2] R. Buyya, Economic-based distributed resource management and scheduling for Grid computing, Ph.D. Thesis, Melbourne, Monash University, 2002. [3] I. Foster, C. Kesselman, Globus: A meta-computing infrastructure toolkit, International Journal of Supercomputer Applications (1997). [4] S. Chapin, D. Katramatos, J. Karpovich, et al., Resource management in Legion, Future Generation Computer Systems 15 (5–6) (1999) 583–594. [5] T. Tannenbaum, M. Litzkow, The Condor distributed processing system, Dr. Dobbs Journal (1995). [6] D. Abramson, J. Giddy, I. Foster, L. Kotler, High performance parametric modeling with Nimrod/G: Killer application for the global Grid?, in: Proceedings of the International Parallel and Distributed Processing Symposium, May 2000. [7] B.R. Schiller, The Economy Today, 8th ed., McGraw-Hill Irwin, New York, 2000.
Hong Peng received the Ph.D. degree in computer science from Xi’an Jiaotong University and has done postdoctoral research at Zhejiang University. He is currently a Professor and Ph.D. tutor of computer science in South China University of Technology. He has presided over and participated in 15 projects supported by National Natural Science Foundation, 863 Program and Provincial Hightech Program of Guangdong etc. His research interests are in the areas of intelligent network computing, intelligent business and data mining. Professor Peng has published over 40 research papers.
700
P. Huang et al. / Future Generation Computer Systems 24 (2008) 694–700
Piyuan Lin received the B.S. and M.S degrees both in Computer Science from University of Electronic Science and Technology of China (UESTC) respectively in 1984 and 1989 respectively. He is currently a senior member of China Computer Federation (CCF), a professor and the vice-dean of College of Informatics, South China Agricultural University (SCAU). He has presided over and participated in 21 projects supported by National Natural Science Foundation, 863 Program, Provincial High-tech Program of Guangdong, and some companies. His research interests include bioinformatics, data mining, and network & information security. Professor Lin has published more than 40 research papers and 22 books.
Xuezhen Li received the B.S. degree in computer science from South China Normal University and the M.E. degree in computer science from South China University of Technology. She is currently a Ph.D. candidate in College of Computer and Information Engineering, HoHai University. She is now a lecturer of computer science in Guangdong Technical College of Water Resources and Electric Engineering. Her research interests include intelligent computing and hydro-information technology.