Q-learning based collaborative cache allocation in mobile edge computing

Future Generation Computer Systems 102 (2020) 603–610

Contents lists available at ScienceDirect

Future Generation Computer Systems journal homepage: www.elsevier.com/locate/fgcs

Q-learning based collaborative cache allocation in mobile edge computing Wei-Che Chien, Hung-Yen Weng, Chin-Feng Lai

∗

Department of Engineering Science, National Cheng Kung University, Tainan, Taiwan, ROC

article

info

Article history: Received 27 February 2019 Received in revised form 13 August 2019 Accepted 29 August 2019 Available online 3 September 2019 Keywords: Software defined networking (SDN) Artificial intelligence (AI) Collaborative cache problem Q-learning

a b s t r a c t The rapid development of Augmented Reality (AR), Virtual Reality(VR), Internet of Things (IoT), and high-definition video has caught the attention of low latency and high bandwidth network requirements. For huge data transmission, cache technology has been regarded as an effective solution for reducing transmission time from users to remote clouds. However, the increasing variety of cached data has caused the challenge of cache performance. Based on high flexibility, scalability, and deployability of the characteristics of Software-Defined Networking (SDN), this study proposes a collaborative cache mechanism in multiple Remote Radio Heads (RRHs) to multiple Baseband Units (BBUs). In addition, the traditional rule-based and metaheuristics methods are difficult to consider all environmental factors. To reduce the traffic load of backhaul and transmission latency from the remote cloud, we use Q-learning to design the cache mechanism and propose an action selection strategy for the cache problem. Through reinforcement learning to find the appropriate cache state. The simulation results show that the proposed method can effectively improve the cache performance. © 2019 Elsevier B.V. All rights reserved.

1. Introduction The Internet has been developed for many years. People have evolved from using personal computers to mobile phones and laptops. The number of mobile devices is increasing every year. Under the demand of multimedia and video, auto-driving and other applications, the transmission speed and low-latency demand of mobile networks are constantly increasing. Currently, 3GPP is developing 5G standards and is expected to be completed in 2020. The standard followed three development directions, including enhanced Mobile Broadband (eMBB) related scenarios, massive Machine Type Communications (mMTC) and Ultra-Reliable and Low Latency Communications (URLLC). For a large number of requirements for network devices, although cloud computing [1,2] can effectively improve the scalability and convenience of resources, it is easy for users who are far away from the database to have more latency. Moreover, repeated transmission of the same data will generate a lot of unnecessary traffic. In order to meet low latency, high data rate, and large capacity. Mobile Edge Computing (MEC) [3,4] is a potential solution. When users request content from the database through the network, there is a certain degree of waiting time from request to response. The concept of MEC is like memory ∗ Corresponding author. E-mail addresses: [email protected] (W.-C. Chien), [email protected] (H.-Y. Weng), [email protected] (C.-F. Lai). https://doi.org/10.1016/j.future.2019.08.032 0167-739X/© 2019 Elsevier B.V. All rights reserved.

and caching on the computer. MEC will cache data on the edge server that closer to the user to reduce the latency of users accessing network information and service [5]. However, in traditional network architecture, packet forwarding is limited by hardware devices. In order to support the rapid development of IoT devices [6], high scalability, low latency, high throughput, and reliable data transmission are required. Therefore, SDN [7,8] and Network Function Virtualization (NFV) [9,10] are indispensable technologies for implementing MEC [11]. The advantages of SDN are manageability, flexibility and low cost. It was redesigned as a new internet architecture to replace the traditional network architecture which is hard to optimize. The most different between SDN architecture and the traditional network architecture is that the control layer and data layer are separated in SDN. Manager can control all network node through the Open Flow protocol [12]. In addition, in order to improve the scalability and flexibility of network functions, NFV similar to the SDN concept was proposed. In NFV, some network features are removed from the hardware device and converted to software applications. All services can be controlled and managed by using NFV. NFV and SDN are often used together, called service function chain (SFC). SFC can concatenate all service functions and use the appropriate algorithm to design the forwarding path to improve network performance [13]. Although SDN has well control capabilities, in the case of high complexity and huge information, suitable algorithms are needed to enhance the management effect of SDN. There are many scholars adopt heuristic algorithms to design resource allocation and

604

W.-C. Chien, H.-Y. Weng and C.-F. Lai / Future Generation Computer Systems 102 (2020) 603–610

load balancing methods but heuristic algorithms are hard to avoid falling into local optimum and consider the overall situation. Deep learning can generate models by training. These models continuously adjust the parameters to minimize loss. Through training have the opportunity to find the most appropriate model. Therefore, the development of artificial intelligence (AI) has become an indispensable technology for SDN potential. [14–16]. Deep learning has been developed many years, it was first published in the concept of an artificial neural network (ANN) [17]. However, since the computing power of the device at the time is insufficient, there is not enough performance to support the huge computational complexity of the neural network. Moreover, it is easy has vanishing gradient problem in the calculation of BackproPagation (BP) in the multi-layer neural network. Simpler methods such as support vector machine (SVM) [18] and logistic regression (LR) [19]have become more popular solutions. Although these methods can process a variety of statistical work and analytical work, it cannot handle more complicated problems so that it has been hindered in the application of network issues. In recent years, since the improvement of computing devices and the implementation of fog computing, Parallel Computing and General Purpose GPU (GPGPU) [20], deep learning has enough model complexity and ability to be applied in the mobile network. Deep learning usually consists of multiple levels of architecture. The characteristics of which are multi-layered nonlinear processing methods and the gradient can be backpropagated through the chain rule. The nonlinear activation function is used to solve the vanishing gradient problem, which causes the neural network to be taken seriously again [21]. In order to reduce the latency when UE access cloud server to request data, we propose a network architecture that integrates MEC and Cloud of Radio Access Network(C-RAN). C-RAN divides the base station functions into RRH and BBU pools to provide more flexible resource allocation and management. Therefore, the transmission latency can be reduced by the caching mechanism. As the limited capacity of the MEC, in order to maximize the cache hit rate and maximize the use of the cache capacity, we adopt reinforcement learning [22] to solve cache problem in multiple RRHs and multiple BBU pools that based on SDN and MEC architecture. The remainder of this article is organized as follows. Section 2 introduces the related work of the cache methods. The network architecture, system model and problem definition are shown in Section 3. Section 4 introduces the Q-learning-based Collaborative cache algorithm. Followed by performance evaluation in Section 5. Finally, we conclude this work in Section 6.

and computation in a multi-user cache-assisted MEC. They formulate the problem as a mixed discrete–continuous optimization problem and adopt a dual method to obtain an optimal solution. [27] proposed a two-fold solution for caching in MEC. They split cache into two logical parts, the first part of the cache is used to store full videos, and the second part of the cache is used to store initial segments of the video. Then, the cache consolidation and splitting are combined to reduce average access delay and external traffic load. [28] build a cooperative caching plan for serving popular video clips by exploring the similarity between users and using a spectral clustering algorithm to categorize users. [29] proposed a MEC intelligent cache analysis platform and proposed an improved genetic algorithm to decide a file list that needs to be cached in advance of the MEC server. In addition, edge caching is also applied to different network architectures. [30] work on Cache and Multi-layer MEC enabled Cloud of Radio Access Network (CMM-CRAN) to minimize latency and energy cost. They formulate cache problem into two parts. First part is a many-to-many matching problem with peer effects. The second part is Multi-Dimension Multiple-Choice Knapsack (MMCK) problem. Then, the Joint Cache content placement and task Offloading Solution (JCOS), utilizing Proportional Fairness (PF) as the user scheduling policy is proposed. The Gale– Shapley (GS) is applied to design cache content placement, and a Population Evolution (PE) game theory and Analytic Hierarchy Process(AHP) are used to calculate the dynamic user task offloading. [31] proposed radio network-aware edge caching framework to improve users’ QoE in 5G communication network. They use Radio Network Information Service (RNIS) and MEC to capture the information about RAN conditions that can be used to cache and update algorithm. In addition, they proposed a Network-aware cache updating algorithm that focuses on the cache replacement and selects which files to save. [32] proposed a mobility-aware coded probabilistic caching scheme for MEC-enabled small cell networks (SCNs). Different from other mobility caching schemes is that they consider user mobility and distributed storage to maximize throughput. Although MEC caching technology has been recognized as an effective solution to address the backhaul constraint, there are many limitations because of the rapidly changing network architecture and various network requirements. Therefore, we use the reinforcement learning to let the machine self-train the appropriate cache list, while considering the global cache and local cache to achieve the best cache performance in C-RAN of multiple RRHs to multiple BBU pools [33,34]. 3. Problem formulation

2. Related works 5G applications such as automatic control, autonomous vehicle, healthcare system [23], etc. have low latency network requirements. The traditional architecture of cloud computing cannot support the 5G application. In order to solve this problem, the Mobile Edge Computing (MEC) is used to complement the shortcomings of cloud computing and SDN is used to enhance network flexibility. Therefore, the data transmission delay between users and remote clouds can be reduced by caching data. Since the current base station is not enough to process 4C, including pushing Computing, Caching, Communication, and Control, [24] proposed collaborative cache allocation and computation offloading in MEC. They use Mobile Network Operator to allocate resources based on weighted proportional to maximizing the resource utilization. [25] propose the MEC enhanced adaptive bitrate (ABR) video delivery scheme, combining content caching and ABR streaming technology together to enhance the mobile video service providing the ability of the RAN. [26] consider communication, caching

In this section, we first introduce a novel network architecture that combined C-RAN and MEC, and based on the SDN and AI. Then we illustrate the system model and problem definition. 3.1. Network architecture 3GPP is discussing the 5G standard. The report RP-160750 indicates that 5G will consider the architecture of C-RAN. Therefore, in order to meet the low-latency and high-traffic network services, this study proposes a network architecture that combined C-RAN and MEC. The traditional base station will be split into RRU and BBU in C-RAN. The functions of the RRU are power amplifiers and filters. The BBU is used to provide external interfaces and complete functions of system resource management, operation, maintenance, and environmental monitoring. Multiple BBUs forms a BBU pool, providing greater resource scheduling flexibility. Since the concept of cloud concentrates resources on a database, it is easy to cause a lot of repeated data to occupy

W.-C. Chien, H.-Y. Weng and C.-F. Lai / Future Generation Computer Systems 102 (2020) 603–610

605

Table 1 Definition of important symbols. Variable

Definition

VUE = {ue1 , ue2 , . . . , uek1 } VRRH = {RRH1 , RRH2 , . . . , RRHk2 } VIBP = {IBP1 , IBP2 , . . . , IBPk3 } VCore = {Core1 , Core2 , . . . , Corek4 } VCloud = {Cloud1 , Cloud2 , . . . , Cloudk5 } |VUE | = k1 |VRRH | = k2 |VIBP | = k3 |VCore | = k4 |VCloud | = k5 ERadio EFoundhaul EBackhaul EIBP Ecore Ecore D = {d1 , d2 , . . . , dz } | D| = z ST S R C (S) Tc Tr IBPC HEi,j (R, S ) UCCHR(R, S) Lrb Lbb

The set of UEs The set of RRHs The set of IBPs The set of core network nodes. The set of cloud servers Number of UEs Number of RRHs Number of IPBs Number of core network nodes. Number of cloud servers. The transmission time between VUE and VRRH . The transmission time between VRRH and VIBP The transmission time between VIBP and VCore The transmission time between within VIBP The transmission time within VCore The transmission time between VCore and VCloud The set of data size Number of data The total number of cached data The cached status of the ith data in jth IBP The status of requested data of RRH The total cached size in all IBPs The threshold of cache capacity The threshold of updated reward The capacity of each IBP The hit efficiency The unit cache capacity hit efficiency The connection status between RRHs and IBPs The connection status between IBPs. The learning rate The UCCHE ratio

β δ

Fig. 1. Network architecture.

bandwidth and generate latency. Therefore, we combine the BBU pool with the MEC. It not only can provide basic BBU functions but also cache data to reduce the load in backhaul and core network. In order to effectively manage data, the combination of SDN and AI is a potential solution. The MEC and BBU pool are integrated as Intelligent BBU Pool (IBP) in this paper. IBP has some computation resource to collect the original data, preprocess data, and convert data into the correct format for applied by AI application. Based on SDN technology, the SDN controller connects each

network component and controls the forwarding path of the data. The load balancing of the network components can be achieved through a suitable algorithm. In addition, in order to optimize the cache mechanism, SDN needs to consider the type and quantity of data requirements of each BBU pool. Therefore, IBP coordination and cache control is a very important function. IBPs can be connected to each other, but based on distance constraints, not all IBPs can be interconnected. The detail network architecture is shown in Fig. 1. With this architecture, the UE can directly obtain the data from IBP to reduce the transmission delay. When many

606

W.-C. Chien, H.-Y. Weng and C.-F. Lai / Future Generation Computer Systems 102 (2020) 603–610

Lrb represented the connection states between RRHs and IBPs. rb Where lrb h,j is a Boolean value, lh,j = 1 means that the hth RRH can connect to jth IBP and vice versa.

⎡ Lbb = ⎣

lbb 1 ,1

⎢

UEs send the same data request, since it is not necessary to access the cloud server to obtain data, the transmission of the same data in Backhaul can be greatly reduced. 3.2. System model In our scenario, the data is from the cloud server. IBP will cache the data from the cloud server to reduce the transmission time and latency within the backhaul. The system model is as follows. Given an undirected graph G = {V , E }, where V is the set of vertices, including UEs (VUE ), RRH(VRRH ), IBPs (VIBP ), Core network nodes (VCore ), Cloud server (VCloud ). Then, VUE ∩ VRRH ∩ VIBP ∩ VCore ∩ VCloud = ∅. Assume that VUE = {ue1 , ue2 , . . . , uek1 } with |VUE | = k1 , VRRH = {RRH1 , RRH2 , . . . , RRHk2 } with |VRRH | = k2 , VIBP = {IBP1 , IBP2 , . . . , IBPk3 } with |VIBP | = k3, VCore = {Core1 , Core2 , . . . , Corek4 } with |VCore | = k4 , and VCloud = {Cloud1 , Cloud2 , . . . , Cloudk5 } with |VCloud | = k5. The nodes in VCore are the equipment in core networks, which can be MME, SGW, PGW, HHS. The E is the set of edges, including ERadio , EFoundhaul , EBackhaul , EIBP , ECore , and ECloud . E means the transmission time within different vertices. The link between VUE and VRRH is called ERadio ; the link between VRRH and VIBP is called EFoundhaul ; the link between VIBP and VCore is called EBackhaul ; The link between Vcore and VCloud is called ECloud . Then, the link within VIBP is called EIBP ; the link within VCore is called Ecore . The detail schematic diagram of system model is shown in Fig. 2. Moreover, the set of data size is defined as D = {d1 , d2 , . . . , dz } with |D| = z. We used Eq. (1) to represent the cached status of the ith data in jth IBP. s1,1

.. .

S=⎣

⎢

sz ,1

··· .. . ···

s1,k3

.. .

⎤ ⎥ ⎦,

(1)

z k3 ∑ ∑

si,j · di ,

(2)

i=1 j=1

Eq. (3) is defined as data request from each RRH.

⎡

r1,1

R=⎣

⎢

.. .

rz ,1

··· .. . ···

r1,k2

.. .

⎤ ⎥ ⎦,

(3)

rz ,k2

The R is the status of requested data of RRH. The ri,h is a Boolean value, ri,h = 1 means that the hth RRH request ith data and vice versa.

⎡ ⎢ Lrb = ⎣

lrb 1,1

.. .

lrb k2,1

··· .. . ···

lbb k3,1

lrb 1,k3

.. .

lrb k2,k3

⎤ ⎥ ⎦,

.. .

⎥ ⎦,

lbb k2,k3

(5)

Tcase1 (R, S ) = ERadio + EFoundhaul + EBackhaul + EIBP + Ecore .

(4)

(6)

There are three transmission cases as follows. Case 1 is that the data is cached in the cloud server. Compared with the other two cases, it will spend additional transmission time from the core networks to clouds. The case 2 is that the data is cached in IBPs. The transmission time in case 2 can be modified as Eq. (7). Tcase2 (R, S ) = ERadio + EFoundhaul ,

(7)

Case 3 is that data is cached by vicinal IBP. Therefore, it will spend transmission time from different IBPs. The transmission time in case 3 can be modified as Eq. (8). Tcase3 (R, S ) = Tcase2 (R, S ) + EIBP ,

(8)

To evaluate catch performance, the hit efficiency is defined as follows. Tcase2 (R, S ) HEi,j (R, S ) = , (9) (Tcase2 (S , R) + EBackhaul + EIBP + Ecore ) HEi,j (R, S ) means the hit efficiency of ith data in jth IBP. Hit efficiency is used to evaluate cache performance. For example, in case 2, the data is cached at the IBP that is the closest to the RRH. Therefore, the EIBP , EBackhaul , and Ecore are equal to 0 and the hit efficiency in case 2 is 100%. In case 3, it needs to spend transmission time of EIBP , thence, hit efficiency will decrease. In addition, we proposed a metric to evaluate hit efficiency of the unit cache capacity called Unit Cache Capacity Hit Efficiency (UCCHE). UCCHE can be calculated by Eq. (10). Where IBPC means the capacity of each IBP.

∑z

∑k2

(

C (S ) k3×IBPC

i=1

UCCHE(R, S) =

sz ,k3

The S is the cached status of the ith data in jth IBP. The si,j is a Boolean value, si,j = 1 means that the ith data is cached by jth IBP and vice versa. Therefore, the total cached size in all IBPs can be calculated by Eq. (2). C (S) =

.. .

⎤

Lbb represented the connection states between IBPs. lbb 1,k3 = 1 means that 1th IBP can connect with k3th IBP. The cached status will affect the latency. Therefore, the transmission time can be calculated by Eq. (6) to Eq. (8).

Fig. 2. Schematic diagram of system model.

⎡

lbb 1,k3

··· .. . ···

j=1

)

HEi,j (R, S )

× 100%

,

(10)

3.3. Problem definition Based on the proposed network architecture in this paper, IBP can cache data to reduce the transmission time in the core network and clouds but the cache capacity in IBPs is limited. Therefore, the main goal of this study is to maximize unit cache capacity hit rate. In other words, the transmission time and cached capacity can be minimized. For better readability, we adopted linear programming to formulate the caching problem in IBP. Maximize UCCHE(R, S) Subject to. si,j ∈ {0, 1} ,

for ∀i ∈ D and j ∈ VIBP .

(11)

ri,h ∈ 0, 1N ,

for ∀i ∈ D and h ∈ VRRH .

(12)

HEi,j (S , R) ≤ 1,

for ∀i ∈ D and j ∈ VIBP .

C (S) > 0,

(13) (14)

k3

∑ j=1

lrb h,j ≥ 1,

for ∀h ∈ VRRH and j ∈ VIBP .

(15)

W.-C. Chien, H.-Y. Weng and C.-F. Lai / Future Generation Computer Systems 102 (2020) 603–610

607

changes rapidly, the rule-based method is difficult to provide superior performance. Therefore, we proposed the Q-Learningbased Collaborative Cache Algorithm (Q-LCCA) to solve the IBP cache problem. In Q-LCCA, the action weight can be calculated by Eq. (10). The action in this study means that the cache status of data. Then, we proposed a selection policy for action. There are three actions for caching data. When popular data is cached by IBPs, there will be a better UCCHE. Therefore, action 1 is to cache the data with the most popular data in local IBP, called local cache. In order to avoid repeated caches in neighbor IBPs, action 2 is to cache the most popular data in global IBP, called global cache. The concept of action 2 is based on the network architecture proposed in this paper. IBPs can be coordinated through SDN technology, so it can get data from neighbor IBP. In other words, when global popular data is cached in the core IBP, it can achieve the highest UCCHE with the least cache capacity. Core IBP means that it has the shortest transmission time with other neighboring IBPs. In order to avoid to fall into local optimum quickly, action 3 is random cache. The probability of caching each data can be calculated by Eq. (17) δ

P(a|S) = ∑

Fig. 3. The reward update flow chart.

k3 ∑

lbb j1,j2 ≥ 1,

for ∀j1 and j2 ∈ VIBP .

(16)

j2=1

Some restrictions are defined as follows. IBP will cache the data from the cloud server to reduce the transmission time and latency within the backhaul. The Eq. (11) restricts that the data only can be cached in IBPs and not by RRHs. The Eq. (12) restricts that the requested data are different in each RRH. Eq. (13) limits when maximum hit efficiency is equal to 1, the data is cached at the nearest IBP to the RRH. Eq. (14) represents that IBPs will cache at least one type of data. Therefore, C(S) must be greater than zero. Eq. (15) limits that RRH can link at least one IBP. Eq. (16) restrict at least one link between IBPs. The definition of important symbols is shown in Table 1. 4. Q-learning-based collaborative Cache algorithm We use reinforcement learning to solve IBP cache problem. Since the capacity of IBP is not infinite, it is very important to be able to increase UCCHE in a limited capacity. The direct method is to cache the most popular data. However, for multiple IBP caches problem, if neighboring IBPs have the same requests, this method will cause redundant cache cost. In addition, as the data request

eQ (S ,a)∗(UCCHE(R,S))

eQ (S ,a )∗(UCCHE(R,S )) ′

a′ ̸ =a

′ δ

,

(17)

Where a is one of action (Cache one of the data types), a′ denotes to other actions. S is current state, Q (S , a) is the Q-value of action a at state S, δ is a parameter called UCCHE ratio, which determines the relative importance of Q (S , a) versus UCCHE. In this way, the search space can be extended. The choice of these three actions is very important for QLCCA. If Q-LCCA only takes Action 1, it will cause a lot of cache capacity waste because the popular data will be stored repeatedly in neighbor IBPs. If action 2 is only used, although Q-LCCA can reduce the cache capacity, the unit cache capacity hit rate will not be the best. Because some of the popular data may not be cached. Although Action 3 can increase the search space, it may cause a lot of computation time. Therefore, in order to effectively select the action, we propose the Roulette-based Action Policy (RAP). According to the value of the weight, the probability that each action is selected can be calculated by Eq. (18). We defined that A1 is the weight of action 1 and rap1 is the probability of choosing action 1; A2 is the weight of action 2 and rap2 is the probability of choosing action 3; A3 is the weight of action 3 and rap3 is the probability of choosing action 3. In addition, the RAP must be executed under the condition that A1 , A2 , and A3 are not equal to 0. The reason is that if the value of A1 , A2 , or A3 is equal to 0, the action will not be chosen. When the conditions do not match, the 3 actions are randomly selected.

⎧ ⎨rap1 = A1 /(A1 + A2 + A3 ) RAP(A1 , A2 , A3 ) = rap2 = A2 /(A1 + A2 + A3 ) ⎩ rap3 = A2 /(A1 + A2 + A3 )

,

(18)

In Q-LCCA, the Q table (Q (S , A)) is initialized first. Then, the connection states (Lrb , Lbb ) and the state of the requested data are updated, and Tr is calculated. Where Tr is the threshold of reward. The Q table can store the weight values for cache state and action that is shown in line 1 to line 3 of Algorithm 1.

608

W.-C. Chien, H.-Y. Weng and C.-F. Lai / Future Generation Computer Systems 102 (2020) 603–610 Table 2 Parameter setting. Parameter

Value

Number of data Type of data Number of RRHs Number of IBPs Data size Capacity of IBP Reword

1000 10–100 100 20 1G–500G 10T 1

Fig. 4. The connection status of IBPs.

Fig. 6. Comparison of average UCCHE and number of data types. (For interpretation of the references to color in this figure legend, the reader is referred to the web version of this article.)

Fig. 5. The distribution of RRHs and IBPs.

we add a strategy for updating the reward in the Q-LCCA. The reward update flow chart is shown in Fig. 3. First, we will have a threshold Tr and Tc. The Tr is calculated as follows. Each IBP caches data from most popular to unpopular in sequence until the limit of the cache capacity, where the limit of the cache capacity is Tc. Then, calculate the UCCHE for the cached state of all IBPs (S) by Eq. (10). If UCCHR of the new state is better than threshold Tr, all actions of S is rewarded(Q (S , A) and weight of RAP) and Tr is updated. Where β is the learning rate, 0 < β ≤ 1. The value of β determines how fast learning occurs that is shown in line 12 to line 16 of the algorithm 1. Then, repeat the iteration until the termination condition is met. According to Algorithm 1, the greater the learning rate β , the less the effect of pre-retention training. If the IBP (has performed ) A’ in the past S’ state and got a good reward, Q S ′ , A′ can let it know the message early. Therefore, when the S ′ state occurs again, the Q-LCCA can choose the correct action to continue to get a good reward. 5. Performance evaluation 5.1. Simulation setup

Then, each IBP will select action through the RAP. According to the selected action, we can get the current cache state, action, and the next state until the cache table is created for this IBP. When cache tables are created for all IBPs, we calculate UCCHR for the current state that is shown in line 4 to line 12 of the Algorithm 1. The design of reward will affect the probability of action selection. Since the cache problem cannot directly provide a reward,

We adopt Matlab R2017a as the simulation tool. The number of requested data of each RRH is set as 1000. The requested data will be randomly generated in different RRHs. The positions of RRH and IBP are randomly generated. Each RRH and IBP have a connection limit according to the distance. In other words, not all BBUs can connect to each other. The number of data types is set 10 to 100. There are 100 RRHs and 20 IBPs in our scenario. The size of each data is between 1G and 500G, and the capacity of IBP is 10T. The detailed parameter settings are shown in Table 2.

W.-C. Chien, H.-Y. Weng and C.-F. Lai / Future Generation Computer Systems 102 (2020) 603–610

609

5.2. Simulation results

Fig. 7. Comparison of average cache size usage rate and number of data types.

Fig. 8. Comparison of average number of cached data types and number of cached data types.

Fig. 4 shows the distribution of RRHs and IBPs, with the blue circle as IBPs and the red crosses as RRHs. All RRHs are randomly distributed, and IBPs are randomly distributed according to the location of the RRHs. Fig. 5 shows the connection status of each IBP. Due to distance limitations, each IBP can be connected to at least 3 IBPs. Fig. 6 shows the comparison of the average number of cached data types for each IBP and the number of cached data types. The blue line shows the result of Q-LCCA with RAP; the red line shows the result of the global cache; the green line shows the result of the local cache. In our scenario, each IBP can cache up to around 40 types of data. Therefore, when the number of data types is less than 40, all data will be cached by the IBP. When the number of data types exceeds 40, the number of cached data types begins to slightly change. Fig. 7 shows the comparison of average IBP cache size and number of data types. When the number of data types exceeds 40, the average cache size used by IBP with these three methods is higher than 90% (almost fully loaded). The cache sizes used by these three methods are slightly different due to the different data types of the cache. Fig. 8 shows the comparison of average hit efficiency and the number of data types with different action policy. When the number of data types is less than 40, the average hit efficiency of the three methods is the same because almost all data types will be cached. When the IBPs are fully loaded and the number of data types are increasing, the hit efficiency of the three methods is significantly decreased because RRH needs to obtain data via the cloud server or the remote IBP. Comparing with these three methods, Q-LCCA with RAP considers global cache and local cache at the same time so that it has better performance. The concept of global cache method is to reduce duplicate cache data with neighbor IBPs by obtaining data from neighboring IBPs, so it needs to spend a lot of communication costs between IBPs. Conversely, the local cache method mainly caches local popular data. When the IBP cannot provide service for the neighboring RRHs, it needs to spend more transmission time to obtaining data from other IBPs or cloud server. Fig. 9 shows the comparison of average UCCHE and number of data types. Before the IBP is not fully loaded, as the number of data types increases, the cache size used by IBPs increases. Therefore, the UCCHE of the three methods is rapidly reduced under the condition that the hit efficiency is unchanged. When IBPs are fully loaded, the hit efficiency will significantly affect the UCCHE. Q-LCCA with RAP has the best performance than other methods because it can find the most appropriate cache list between global cache and local catch in a limited cache capacity. 6. Conclusion

Fig. 9. Comparison of average hit efficiency and number of data types.

Based on the network architecture of multiple RRHs to multiple BBUs, we propose Q-LCCA to solve the collaborative cache problem and define the UCCHE to measure the cache performance. Besides, we consider the global cache and local cache and propose an action selection policy called RAP to maximize UCCHE. Integrating the BBU pool and MEC into IBP not only provide signal process but also preprocess data to improve the performance of AI applications. The simulation results show that Q-LCCA can avoid duplicate caches through the collaborative cache to improve UCCHE. However, Q-LCCA has a limitation that each IBP must be connected to at least one IBP for the collaborative cache. In future work, we will combine different deep learning algorithms to enhance cache performance.

610

W.-C. Chien, H.-Y. Weng and C.-F. Lai / Future Generation Computer Systems 102 (2020) 603–610

Declaration of competing interest The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper. Acknowledgments This work was supported in part by the Ministry of Science and Technology of Taiwan, R.O.C., under Contracts MOST 107-2622-S006-001-CC3 and 107-2221-E259-005-MY3. References [1] C. Stergiou, K.E. Psannis, B.G. Kim, B. Gupta, Secure integration of IoT and cloud computing, Future Gener. Comput. Syst. 78 (2018) 964–975. [2] B. Varghese, R. Buyya, Next generation cloud computing: New trends and research directions, Future Gener. Comput. Syst. 79 (2018) 849–861. [3] N. Abbas, Y. Zhang, A. Taherkordi, T. Skeie, Mobile edge computing: A survey, IEEE Internet Things J. 5 (1) (2018) 450–465. [4] M. Chen, W. Li, G. Fortino, Y. Hao, L. Hu, I. Humar, A dynamic service migration mechanism in edge cognitive computing, ACM Trans. Internet Technol. 19 (2) (2019) 30. [5] M. Chen, Y. Hao, L. Hu, M.S. Hossain, A. Ghoneim, Edge-CoCaCo: Toward joint optimization of computation, caching, and communication on edge cloud, IEEE Trans. Wirel. Commun. 25 (3) (2018) 21–27. [6] M. Chen, Y. Miao, Y. Hao, K. Hwang, Narrow band internet of things, IEEE Access 5 (2017) 20557–20577. [7] W.C. Chien, C.F. Lai, H.H. Cho, H.C. Chao, A SDN-sfc-based service-oriented load balancing for the IoT applications, J. Netw. Comput. Appl. 114 (2018) 88–97. [8] K.M. Modieginyane, B.B. Letswamotse, R. Malekian, A.M. Abu-Mahfouz, Software defined wireless sensor networks application opportunities for efficient network management: A survey, Comput. Electr. Eng. 66 (2018) 274–287. [9] A. Basta, W. Kellerer, M. Hoffmann, H.J. Morper, K. Hoffmann, Applying NFV and SDN to LTE mobile core gateways, the functions placement problem, in: Proceedings of the 4th Workshop on All Things Cellular: Operations, Applications, & Challenges, 2014, 33–38. [10] Bo Han, Vijay Gopalakrishnan, Lusheng Ji, Seungjoon Lee, Bo han vijay gopalakrishnan lusheng ji seungjoon lee network function virtualization: Challenges and opportunities for innovations, IEEE Commun. Mag. 53 (2) (2015) 90–97. [11] M. Chen, Y. Hao, Task offloading for mobile edge computing in software defined ultra-dense network, IEEE J. Sel. Areas Commun. 36 (3) (2018) 587–597. [12] A. Shalimov, D. Zuikov, D. Zimarina, V. Pashkov, R. Smeliansky, Advanced study of SDN/OpenFlow controllers, in: Proceedings of the 9th Central & Eastern European Software Engineering Conference in Russia, 2013, p. 1. [13] W.C. Chien, H.Y. Weng, C.F. Lai, Z. Fan, H.C. Chao, Y. Hu, A SFC-based access point switching mechanism for software-defined wireless network in IoV, Future Gener. Comput. Syst. 98 (2019) 577–585. [14] M. Latah, L. Toker, Application of artificial intelligence to software defined networking: A survey, Indian J. Sci. Technol. 9 (44) (2016) 1–7. [15] S. Yan, et al., Field trial of machine-learning-assisted and SDN-based optical network planning with network-scale monitoring database, in: Proceedings of 2017 European Conference on Optical Communication, ECOC, 2017, pp. 1–3. [16] M. Chen, Y. Hao, K. Lin, Z. Yuan, L. Hu, Label-less learning for traffic control in an edge network, IEEE Netw. 32 (6) (2018) 8–14. [17] X. Yao, Evolving artificial neural networks, IEEE 87 (9) (1999) 1423–1447. [18] S. Tong, D. Koller, Support vector machine active learning with applications to text classification, J. Mach. Learn. Res. 2 (2001) 45–66. [19] G. King, L. Zeng, Logistic regression in rare events data, Political Anal. 9 (2) (2001) 137–163. [20] T. Suzuki, T. Mani, K. Tsuchida, M. Yamaguchi, GPGPU-based high performance parallel computation method for valve body failure mode, SAE Int. J. Passeng. Cars-Mech. Syst. 9 (2016) 301–309. [21] M. Chen, J. Zhou, G. Tao, J. Yang, L. Hu, Wearable affective robot, IEEE Access 6 (2018) 64766–64776. [22] L.P. Kaelbling, M.L. Littman, A.W. Moore, Reinforcement learning: A survey, J. Artif. Intell. Res. 4 (1996) 237–285. [23] M. Chen, W. Li, Y. Hao, Y. Qian, I. Humar, Edge cognitive computing based smart healthcare system, Future Gener. Comput. Syst. 86 (2018) 403–411.

[24] A. Ndikumana, S. Ullah, T. LeAnh, N.H. Tran, C.S. Hong, Collaborative cache allocation and computation offloading in mobile edge computing, in: Proceeding of 2017 19th Asia-Pacific Network Operations and Management Symposium, APNOMS, 2017, pp. 366–369. [25] X. Xu, J. Liu, X. Tao, Mobile edge computing enhanced adaptive bitrate video delivery with joint cache and radio resource allocation, IEEE Access 5 (2017) 16406–16415. [26] Y. Cui, W. He, C. Ni, C. Guo, Z. Liu, Energy-efficient resource allocation for cache-assisted mobile edge computing, in: Proceeding of 2017 IEEE 42nd Conference on Local Computer Networks, LCN, 2017, pp. 640–648. [27] S. Kumar, A.A. Franklin, Consolidated caching with cache splitting and trans-rating in mobile edge computing networks, in: Proceeding of 2017 IEEE International Conference on Advanced Networks and Telecommunications Systems, ANTS, 2017, pp. 1–6. [28] Y.T. Chen, C.C. Yen, Y.T. Lin, J.S. Wang, Cooperative Caching Plan of Popular Videos for Mobile Users by Grouping Preferences, in: Proceeding of 2018 IEEE 16th Intl Conf on Dependable, Autonomic and Secure Computing, 16th Intl Conf on Pervasive Intelligence and Computing, 4th Intl Conf on Big Data Intelligence and Computing and Cyber Science and Technology Congress, DASC/PiCom/DataCom/CyberSciTech, 2018, pp. 762–769. [29] W. He, Y. Su, L. Huang, Y. Zhao, Research on Streaming Media Cache Optimization Based on Mobile Edge Computing, in: Proceeding of 2018 13th International Conference on Computer Science & Education, ICCSE, 2018, pp. 1–6. [30] H. Mei, K. Wang, K. Yang, Joint cache content placement and task offloading in C-RAN enabled by multi-layer MEC, Sensors 18 (6) (2018) 1826. [31] Y. Tan, C. Han, M. Luo, X. Zhou, X. Zhang, Radio network-aware edge caching for video delivery in mec-enabled cellular networks, in: Proceeding of 2018 IEEE Wireless Communications and Networking Conference Workshops, WCNCW, 2018, pp. 179–184. [32] X. Liu, J. Zhang, X. Zhang, W. Wang, Mobility-aware coded probabilistic caching scheme for mec-enabled small cell networks, IEEE Access 5 (2017) 17824–17833. [33] W.C. Chien, C.F. Lai, H.C. Chao, Dynamic resource prediction and allocation in c-ran with edge artificial intelligence, IEEE Trans. Ind. Inf. (2019) http: //dx.doi.org/10.1109/TII.2019.2913169. [34] L. Velasco, A. Castro, A. Asensio, M. Ruiz, G. Liu, C. Qin, R. Proietti, S.J.B. Yoo, Meeting the requirements to deploy cloud RAN over optical networks, IEEE J. Opt. Commun. Netw. 9 (3) (2017) B22–B32.

Wei-Che Chien received the B.S. and M.S degree in Computer Science and Information Engineering from National I-Lan University, Taiwan in 2013 and 2016. He is currently pursuing the Ph.D. degree at the Engineering Science, National Cheng Kung University. His research interests include wireless rechargeable sensor networks, next generation network, IoT application, and cloud computing.

Hung-Yen Weng received the master degree in Engineering Science from National Cheng-Kung University in 2009. He had been working in HTC for 7 years since 2010 and then joined Google in 2018. He works mainly on system software of baseband OS on mobile phones. He has contributed to many flagship smart phones, such as HTC M7/M8/M9/10/U11 and Google Pixel/Pixel2. He is a PhD student in Engineering Science from National Cheng-Kung University.

Chin-Feng Lai received the PhD degree from the Department of Engineering Science, National Cheng Kung University, Taiwan, in 2008. He is an associate professor in the Department of Engineering Science, National Cheng Kung University since 2016. He received Best Paper Award from IEEE 17th CCSE, 2014 International Conference on Cloud Computing, IEEE 10th EUC, IEEE 12th CIT. He has more than 100 paper publications. He is an associate editor-in-chief of the Journal of Internet Technology. His research focuses on Internet of Things, body sensor networks, E-healthcare, mobile cloud computing, cloud-assisted multimedia network, embedded systems, etc. He is a senior member of the IEEE since 2014.