Recommendation generation using personalized weight of meta-paths in heterogeneous information networks

Journal Pre-proof

Recommendation generation using personalized weight of meta-paths in heterogeneous information networks Mukul Gupta , Pradeep Kumar PII: DOI: Reference:

S0377-2217(20)30031-X https://doi.org/10.1016/j.ejor.2020.01.010 EOR 16264

To appear in:

European Journal of Operational Research

Received date: Accepted date:

17 October 2018 6 January 2020

Please cite this article as: Mukul Gupta , Pradeep Kumar , Recommendation generation using personalized weight of meta-paths in heterogeneous information networks, European Journal of Operational Research (2020), doi: https://doi.org/10.1016/j.ejor.2020.01.010

This is a PDF file of an article that has undergone enhancements after acceptance, such as the addition of a cover page and metadata, and formatting for readability, but it is not yet the definitive version of record. This version will undergo additional copyediting, typesetting and review before it is published in its final form, but we are providing this version to give early visibility of the article. Please note that, during the production process, errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain. © 2020 Published by Elsevier B.V.

Highlights 

Personalized recommendations are generated using noisy binary implicit feedback



Popularity of items and interest of users are simultaneously leveraged



Heterogeneous information network is formed using meta-information of items



A method is proposed to perform the personalized weight learning for meta-paths



Semantics of various meta-paths are integrated for personalized recommendations

1

Recommendation generation using personalized weight of meta-paths in heterogeneous information networks Mukul Guptaa,*, Pradeep Kumarb a

Indian Institute of Management Indore, Information Systems, India b

Indian Institute of Management Lucknow, IT & Systems, India

*

Corresponding Author

Email Addresses: [email protected] (Mukul Gupta), [email protected] (Pradeep Kumar) Abstract In today’s era of electronic markets with information overload, generating personalized recommendations for e-commerce users is a challenging and interesting problem. Recommending topN items of interest to e-commerce users is more challenging using binary implicit feedback. The training data is usually highly sparse and has binary values capturing a user’s action or inaction. Due to the sparseness of data and lack of explicit user preferences, neighborhood-based and model-based approaches may not be effective to generate accurate recommendations. Of late, network-based item recommendation methods, which utilize item related meta-information, have started getting attention. In this work, we propose a heterogeneous information network-based recommendation model called HeteroPRS for personalized top-

recommendations using binary implicit feedback. To utilize the

potential of meta-information related to items, we use the concept of meta-path. To improve the effectiveness of the recommendations, the popularity of items and interest of users are leveraged simultaneously. Personalized weight learning of various meta-paths in the network is performed to determine the intrinsic interests of users from the binary implicit feedback. The proposed model is experimentally evaluated and compared with various recommendation techniques for implicit feedback using real-world datasets, and the results show the effectiveness of the proposed model.

Keywords: E-commerce, Networks, Recommendation system, Binary implicit feedback, Decision Support

2

1. Introduction With the advancement in the technology for generating and capturing data, information filtering for generating recommendations has become very important (Scholz et al., 2017; Bobadilla et al., 2013; Papamichail & Papamichail, 2007). Users of e-commerce are overloaded with a multitude of choices. Providing effective recommendations to users has become a major challenge in ecommerce. In the last decade, the research on approaches for generating accurate personalized recommendations have gained the attention of researchers and practitioners (Scholz et al., 2017; Berkovsky & Freyne, 2015; Adomavicius & Tuzhilin, 2005). For recommendation generation, we can utilize either explicit or implicit feedback from users (Geuens et al., 2018; Aggarwal, 2016; Ricci et al., 2011; Jannach et al., 2010). In the explicit feedback, user preferences are usually mentioned in terms of ratings/stars specified on a scale for an item (Scholz et al., 2017; Aggarwal, 2016). However, in implicit feedback, only the interactions between users and items are mentioned. The implicit feedback could be quantitative or binary. In quantitative implicit feedback, the interaction between a user and item is represented in terms of frequency (Ricci et al., 2011; Jannach et al., 2010; Bauer & Nanopoulos, 2014; Hu et al., 2008). However, in binary implicit feedback, values are from the set {1, 0} where a 1 represents a user’s interaction with the item and a 0 represents a lack of interaction (Yu et al., 2014; Rendle et al., 2009; Pan et al., 2008). Implicit feedback is considered highly noisy as a 0 in the data is due to either negative feedback or unobserved interaction (Rendle et al., 2009; Pan et al., 2008). In this paper, we utilize binary implicit feedback for generating top-

recommendations. The

recommendation task using the binary implicit feedback is more challenging due to the absence of explicit signals of interest of users (Aggarwal, 2016; Ricci et al., 2011; Jannach et al., 2010; Demiriz, 2004). Collaborative filtering (neighborhood-based and model-based) techniques have been utilized for generating recommendations using binary implicit feedback. Due to the lack of explicit negative signals of interest of users and sparseness of the training data, the accuracy of recommender system may not be high (Yu et al., 2014; McFee et al., 2012). The content-based approach forms the feature vectors for items to be recommended using the attributes of the items (Aggarwal, 2016; Bobadilla et al., 2013). In real-world scenarios, these feature vectors are usually high dimensional and highly sparse that adversely affect the performance and accuracy of the content-based approach (Aggarwal, 2016; Jannach et al., 2010). Recently, researchers have tried to utilize the meta-information related to items to deal with the problem of the sparseness of the feedback data (Gupta et al., 2017c; Shi et al., 2015; Yu et al., 2014; Luo et al., 2014). They formed the heterogeneous information network (HIN) using the meta-information related to items. For example, in the case of an online book store, there would be different meta-information like the category of book, discipline, subject, course for which book is suitable, author(s) of the book and so on. This meta-information can be utilized to form the HIN for books in the context of an online store. Shi et al. (2015) proposed an approach using 3

heterogeneous information network but they utilized explicit feedback data for recommendation generation. Luo et al. (2014), in their work, utilized explicit ratings in the interval [0, 1]. Yu et al. (2014) used binary implicit feedback and they proposed a methodology based on information diffusion

for

generating

personalized

recommendations.

For

generating

personalized

recommendations, they performed the clustering of users to deal with the problem of data sparsity (Yu et al., 2014). However, as shown in Sarwar et al. (2002), the accuracy of clustering-based personalized recommender systems may be suffered due to clustering. In this paper, we propose a meta-path based non-clustering recommendation model named as HeteroPRS (Heterogeneous information network-based Personalized Recommendation System) for generating top-

recommendations. HeteroPRS incorporates the meta-information related to items to

form a heterogeneous information network and deals with the problem of data sparsity for binary implicit feedback without performing the clustering of users. The proposed approach is a link-based (network-based) approach and different from the content-based approach. In HeteroPRS, the network is formed using the meta-information of the items and the similarity of items is measured in the network by considering the link structure of the network. The proposed model HeteroPRS is unique in terms of simultaneously utilizing the popularity of items and interest of users for generating personalized recommendations. It is a non-clustering based method and computes the personalized weights for various meta-paths taken into consideration in the network for personalized recommendations. It utilizes the potential of the information network completely by considering the arbitrary meta-paths from the network. The personalized weight learning methodology of the proposed model HeteroPRS is effective in finding the intrinsic interest of the users in the network by following the meta-paths in the network. The proposed personalized weight learning methodology is non-iterative and efficient. This makes the proposed recommendation model HeteroPRS more suitable in a real-world situation. Also, in the proposed model there is no need to generate the meta-paths manually. The problem of data sparsity can be dealt with heterogeneous information network for items, but to improve the accuracy of recommendations we need to know the intrinsic interest of users. Using personalized weight learning for various meta-paths in the network, we can determine the intrinsic interest of users. The semantic associated with a higher weighted meta-path in the network would represent the interest of a user more accurately. To improve the accuracy of the personalized recommendations, we simultaneously leverage the popularity of items and interest of users. The popularity of items is an important factor for recommendation generation as in the study by Barman & Dabeer (2012), authors have shown that the popular items in a group of similar users would have good chance to be purchased/experienced. Also, the study by Bobadilla et al. (2013) has shown that users would like to purchase/experience those items first which are as per their interest. The proposed approach has combined both popularity and intrinsic interest of users. The popularity of items is a global information and intrinsic interest of a user is local information. The combination of both may 4

give improved results. Therefore, we have utilized both information. Other information of items i.e. the meta-information of items is utilized to form the heterogeneous information network which is the underlying framework for HeteroPRS. The weight learning process of the proposed model HeteroPRS assigns higher weights to those meta-paths that represent the interest of users effectively as compared to other meta-paths. The effectiveness of the personalized weight learning in the proposed methodology helps in improving the accuracy of the recommendations by HeteroPRS. The expected contribution of this study is a meta-path based recommendation model named as HeteroPRS for recommending top-

items to users. Recommendations are generated using the

personalized weights of various meta-paths to represents the intrinsic interest of users and the popularity of items simultaneously. HeteroPRS can utilize arbitrary meta-paths for recommendation generation and performs personalized weight learning for those meta-paths. The proposed personalized weight learning in HeteroPRS is effective to find the intrinsic interest of users. The meta-paths are generated using the concept of network exploration, so there is no need to manually find out the meta-paths from the network schema. HeteroPRS can utilize the symmetric as well as asymmetric meta-paths for recommendation generation to fully utilize the information present in the network, and hence the accuracy of recommendations would be high. The proposed model does not require the clustering of either users or items, so the recommendations are truly personalized without hampering the accuracy of recommendations. The remaining of the paper is organized as follows. In Section 2, the related work is presented. In Section 3, the background and problem definition for the recommendation generation is presented. The proposed model for recommending top-

items is presented in Section 4.

Experimental setup and results are discussed in Section 5 and Section 6 respectively. Finally, we conclude the paper with future research directions in Section 7.

2. Related Work In this work, we focus on recommendation generation using implicit feedback. In this section, we discuss the related work, which has utilized the implicit feedback for recommendation generation. We also discuss the work that incorporates the meta-data related to items to form the heterogeneous information network and utilize that for recommendation generation using implicit feedback. In Table 1, we have summarized the related work which can utilize implicit feedback data for recommendation generation.

5

Table 1. Summary of related work Work/Study

HIN-based approach

Technique/methodology

Limitation(s)

PopularityBased

No

Recommend the most popular item to the active user

Non-personalized method of recommendation

Association Rule-Based Model (Demiriz, 2004)

No

Develops, first, a set of association rules, and then apply those rules while generating recommendations

Accuracy of recommendations would not be good as training data would be noisy

Hu et al. (2008)

No

Works for quantitative implicit feedback data and based on pointwise preference assumption

Not optimized for ranking

Pan et al. (2008)

No

Works on pointwise preference assumption

Not optimized for ranking

BPRMF (Rendle et al., 2009)

No

Works on pairwise preference assumption

Accuracy of recommendations would be adversely affected due to data sparsity

IBCF (Sarwar et al., 2001)

No

Neighborhood-based method

Data sparsity would lead to poor accuracy of recommendations

HeteRec

Yes

Utilizes the HIN (Heterogeneous Information Network) based preference diffusion methodology; utilized metainformation related to items and performed clustering of users for personalization

Clustering-based personalization; weight determination of meta-paths is not effective; can utilize only symmetric meta-paths

(Yu et al., 2014)

A trivial method to generate recommendations using implicit feedback data is the Popularitybased method. This method recommends the most popular items. However, this method is not personalized and the recommendations generated may not be accurate. Using implicit feedback data, recommendations can be generated by Association rule-based method (Demiriz, 2004). This method generates the association rules from the training data and recommendations are generated using those rules. The accuracy of recommendation may be low when the data is noisy. Matrix factorization, for recommendation generation using explicit feedback, is a popular model-based technique and has been studied extensively (Felfernig et al., 2013; Koren et al., 2009; Adomavicius & Tuzhilin, 2005). Hu et al. (2008) and Pan et al. (2008) have proposed matrix factorization based filtering techniques for implicit feedback. However, their approaches work on the pointwise preference assumption and solve an optimization problem to predict whether an item will be selected by a user. For the task of generating top-

recommendations, Rendle et al. (2009) have

proposed a Bayesian Personalized Ranking based Matrix Factorization (BPRMF) methodology. Their 6

methodology is based on the pairwise-preference assumption and optimized for the ranking of items (Rendle et al., 2009). Of late, it has been shown by McFee et al. (2012) using million song dataset that a neighborhood-based filtering technique like item-based collaborative filtering (IBCF) can give more accurate recommendations than BPRMF for binary implicit feedback. However, for highly sparse data, the effectiveness of recommendations generated by IBCF is affected adversely (Bobadilla et al., 2013). To deal with explicit user feedback, Salakhutdinov et al. (2007) also proposed a collaborative filtering technique called Restricted Boltzmann Machines (RBM) which can utilize missing ratings. However, their technique is not applicable to implicit feedback as it utilizes the combination of missing ratings and explicit ratings given by users (Salakhutdinov et al., 2007). For improving the effectiveness of recommendations and dealing with the problem of the sparseness of binary implicit feedback, researchers have incorporated the different types of information related to items (Felfernig et al., 2013; Li et al., 2013; Basilico & Hofmann, 2004). However, recently, researchers are more interested in extracting the knowledge from the heterogeneous linked information related to items rather than the content information (Gan, 2016; Shi et al., 2015; Yu et al., 2014; Luo et al., 2014; Bellogin et al., 2013). Recently, the information-rich mining results using the heterogeneous information networks (Gupta et al., 2017a; Gupta et al., 2017b; Sun & Han, 2013) has attracted the attention of researchers for exploring their applicability for recommendation generation (Yu et al., 2014; Shi et al., 2015). In this paper, the focus is on the applicability of the heterogeneous information network for information filtering to generate accurate top-

personalized recommendations. Also, using the

heterogeneous information network, the problem of data sparsity is handled for binary implicit feedback. Recently, Gupta et al. (2017c), Yu et al. (2014), Luo et al. (2014), and Shi et al. (2015) formed the heterogeneous information network using the meta-information related to items and performed the information filtering for recommendation generation. Gupta et al. (2017c) proposed to utilize the graph regularization framework for recommendation generation. Yu et al. (2014), in their work, utilized binary implicit feedback for recommendation generation; however, Shi et al. (2015) utilized explicit feedback in their work for recommendation generation. Luo et al. (2014), in their work, used PathSim (Sun et al., 2011) to compute the relatedness in heterogeneous information network between users and items, and utilized those values as proxy for explicit ratings in the interval [0, 1] given by users to items to predict whether an item will be selected by a user. For recommendation generation, the user preference diffusion-based methodology was utilized by Yu et al. (2014) in the HeteRec recommendation model. For the training of their proposed model, they utilized a Bayesian ranking-based optimization methodology (Rendle et al., 2009). For personalization of the recommendations and to deal with the sparsity of the data, they performed clustering of users. However, their work has some limitations. First, for preference diffusion along various paths in the network, they utilize PathSim which can utilize only symmetric meta-paths in a network (Sun et al., 2011). Therefore, their model cannot utilize the full potential of the linked 7

information present in the heterogeneous information network. Further, for personalization of the recommendations and to deal with the sparseness of the data, their model requires clustering of users. However, finding the appropriate number of clusters for a dataset is a challenging and difficult problem and the effectiveness of the recommendations from their model significantly depends on the quality of the clusters created. This makes their recommendation model difficult to apply for realworld applications. Considering the limitations of the aforementioned approaches, we form a heterogeneous information network for items and fully utilize the potential of the linked information present in the network by considering symmetric and asymmetric meta-paths. Using this network, personalized recommendations are generated. In this paper, we have proposed a recommendation model with a novel weight learning methodology and demonstrated its effectiveness by comparing it with a number of state-of-the-art techniques using three real-world datasets. In this paper, the proposed novel personalized weight learning methodology is non-iterative and efficient. This makes the proposed approach more suitable in a real-world situation. Also, we have incorporated the meta-path generation methodology which alleviates the need for manual meta-path exploration.

3. Background In this section, the background, preliminaries and important definitions are presented. Some important and frequently utilized notations are listed in Table 2. Table 2. Important notations Notation

Description Heterogeneous information network (HIN) for objects and network schema Set of all objects and set of all edges/links Set of object types and set of relations in HIN Set of meta-paths from HIN Relatedness between object and in HIN following meta-path Set of all users Set of all items Binary Implicit feedback matrix for training Popularity of item in the training data Personalized weight matrix for meta-paths Personalized recommendation score matrix

To understand information network and network schema, they are formally defined in Definition 1 similar to the definition by Sun et al. (2011). Definition 1 (Information Network and Network Schema). An information network is defined as a directed graph

with an object type mapping function

and a link type mapping 8

function link

, where each object belongs to a particular relation

, is the meta-level representation of

belongs to a particular object type

, and each

. However, the network schema denoted as and it is a directed graph over object types

and relations . In an information network, if the number of relations

or types of objects

, then

the network is termed as a heterogeneous information network (Sun et al., 2011). In this paper, we form a heterogeneous information network using meta-information related to items. In Fig. 1, the snippet of a heterogeneous information network for songs is shown.

Fig. 1. A heterogeneous information network snippet for songs The network schema of that information network is shown in Fig. 2. The network schema for songs has four types of nodes i.e. song, artist, album, and genre. Since users have listened to the songs, therefore, we have also included the users in this network schema. Hence, there are five different types of nodes and four different types of bidirectional relations in the network schema for songs. A path on the network schema is called a meta-path (Gupta et al., 2017b; Sun et al., 2011).

Fig. 2. Network Schema of the song information network 9

For recommendation generation, using the meta-information related to items, first, a heterogeneous information network is formed. Then, relatedness between items and also between users and items is computed following different meta-paths. For relatedness calculation, we first give the Definition 2 of the meta-path similar to the definition by Sun et al. (2011).

Definition 2 (Meta-path). A meta-path denoted in the form of

→

→

length between source object type

is a path defined on a network schema

and

that defines a composite relation and target object type

of

using composition operator о on

relations. A meta-path

can be represented using only object types as

when the

two object types are unambiguously related to each other. For example, in Fig. 2, several meta-paths →

exist like →

→ →

(SHS),

→

→

→

(SAS), and

(USHS). By following a meta-path, we can measure the

relatedness between objects in the network and can incorporate the semantic associated with that meta-path. For example, meta-path SHS links those songs that were sung by the same artist. Using this meta-path we can calculate the relatedness between songs on the basis of the artist. Similarly, the semantic meaning of other meta-paths can be explained, and they can be utilized for relevance measurement. In this work, we have utilized binary implicit feedback to generate recommendations. The binary implicit feedback matrix is defined in Definition 3.

Definition 3 (Binary Implicit feedback matrix). Binary Implicit feedback matrix users

and {

(

items

for

is defined as

)

(1)

In the binary implicit feedback matrix, the value 1 represents that the user has interacted with the item. For example, in the case of songs, song . However,

implies that the user

has listened to the

implies no interaction. The value 0 represents either the unobserved

interaction between a user and an item (due to the unawareness of the user about the item) or the user is not interested in that item (user dislikes the item). To measure the relatedness between items, and between users and items to generate the recommendations, the heterogeneous information network for items is utilized instead of the sparse binary implicit feedback. For example, a snippet of a heterogeneous information network for songs is shown in Fig. 3. This network incorporates the meta-information related to songs i.e. artists, genres, 10

and albums. In this network, users have also been incorporated as they have listened to the songs and this relationship is shown using dashed lines. Other relationships between songs and various metainformation related to songs have been shown using solid lines. For measuring the relatedness between objects and for generating recommendations, we can follow various paths to incorporate the corresponding semantics. To understand the significance of heterogeneous information networks for recommendation generation, consider a toy network shown in Fig. 3 for users ( meta-information (albums (

) and artists (

) and songs (

) with

)) of songs.

Fig. 3. Interest diffusion in a toy heterogeneous information network for recommendation generation Consider the following two meta-paths which connect users to songs in the above network:

If the above two paths are followed in the network for the diffusion of implicit interest (shown with the bold arrows in the network for user recommended songs

and

), we can see the users

and

may be

respectively depending on the recommendation score for those two

songs. If the heterogeneous information network is not formed, then, in that case, it won’t be possible to recommend any song to the user

. So, by forming a heterogeneous information network, the

accurate recommendations can be generated by diffusion of interest.

4. HeteroPRS: Model for Personalized Recommendations In this section, we describe the proposed model HeteroPRS for recommendation generation using binary implicit feedback. HeteroPRS utilizes the meta-path based framework for recommendation generation. 4.1. Model Description

11

The proposed recommendation model is shown in Fig. 4. The two inputs to HeteroPRS are a set of meta-paths generated from the network schema and training feedback data. Both inputs are fed to the offline module. However, the online module requires the feedback data and the learned model generated from the offline module. In the offline module, personalized weight learning and personalized model learning are performed. After personalized weight learning, the learned weights of meta-paths are utilized as inputs by personalized model learning. The online module of the framework generates personalized top-

recommendations for an active user and takes as input the learned

personalized model for that user and feedback data from the user log. The offline module of the framework is updated after regular time-intervals using feedback data from the user log. Immediate retraining of the offline module is not performed as it is time-intensive. The retraining after a regular time interval would save time with a little reduction in the accuracy of recommendations (Linden et al., 2003).

Fig. 4. Recommendation Model of HeteroPRS

4.2. Generation of Meta-paths For generation of meta-paths, we use the network exploration algorithm proposed by Gupta et al. (2017a). The network schema of the heterogeneous information network is explored to generate all meta-paths up to a specified length. The algorithm for network exploration takes as input, source and target object types which are the start and end object types of the meta-paths respectively, the schema of the network, and the maximum length of the meta-paths. For example, using the network schema shown in Fig. 2, this algorithm, as shown in Fig. 5, generates meta-paths by exploring the current node under consideration and if any neighboring node is the target type node ( generates the meta-path by augmenting the meta-path from the source object type (

), then it ) to the target

object type.

12

Fig. 5. An example of network exploration for generation of meta-paths

4.3. Personalized Weight Learning (PWL) PWL for meta-paths is required to determine the intrinsic interest of users. Each meta-path in the network has a semantic meaning and that may reflect the interest of users. The weight learning process assigns higher weights to those meta-paths that represent the interest of users effectively as compared to other meta-paths. For example, if a user has listened most of the songs sung by a particular artist then it means that the user is a fan of that artist and the meta-path SHS (song – artist – song) would represent the interest of that user effectively. Therefore, the meta-path SHS would be assigned higher weight as compared to other meta-paths. Hence, weight learning is required to determine the intrinsic interest of users from binary implicit feedback data. For a user, the effective recommendation of items can be generated after computing the personalized weight for different meta-paths. For a user, the importance of a meta-path depends on Preference Diffusion Accuracy (PDA). The PDA of a meta-path for a user indicates that how accurately the meta-path, if followed, would diffuse the user interest only among those items which are as per the interest of that user and highly related to the previously experienced items by that user. PDA can be used to exploit the training data for a user to find the most similar items to the previously experienced items. Formally, PDA is explained in Definition 4. Definition 4 (Preference Diffusion Accuracy). For a user, the preference diffusion accuracy of a meta-path

→

→

is the ability of that path for diffusing user preference among those

items that are as per the interest of users in the heterogeneous item information network through the items that have been experienced by the user and is defined as ∑

where user and

(2)

is the training set for the user consisting of those items that have been experienced by that is the cardinality of the set

. The denominator in Eq. (2) normalizes the value of PDA.

13

The

computes the relatedness between items

and

following meta-path

using

DPRel (Gupta et al., 2017b) that is defined in Eq. (3).

(

|

Where

( )

)

∑

is the degree of item

items

and

)

( )

∑

in the network and

(3) ) is the number of paths connecting

in the network. DPRel can compute the relatedness between objects following

arbitrary meta-path, therefore, DPRel reduces the information loss from the network (Gupta et al., 2017b). Using the PDA values for various meta-paths in the network, we can compute the personalized weights for these meta-paths for a user as given below in Eq. (4). (4)

∑

The Algorithm-1 for PWL is given below. In this algorithm, for each user, first, the PDA is computed for each meta-path (line-2 to Line-4) then the personalized weights of these meta-paths are computed (Line-5 to line7). Algorithm-1 Input: : meta-paths from in format of : training data : relatedness between items and following meta-path Output: : personalized weights of meta-paths Begin 1. foreach user do 2. foreach meta-path do ∑

3. 4. 5. 6.

end for for

to K do ∑

7. end for 8. end for End

4.4. Personalized Model Learning (PML) The algorithm for PML generates personalized recommendation scores of items for users. The output of this algorithm is the trained model and has personalized recommendation scores for inexperienced items in the sorted (descending) order for each user. The output of PWL is supplied as input to PML. The Algorithm-2 for PML is given below. 14

Algorithm-2 Input: : meta-paths in format of : relatedness between users and items following meta-path : training data : personalized weights of meta-paths Output: : recommendation score matrix Begin 1. for all and do 2.

// initialization

3.

for

to

do ( )

4. 5. end for 6. end for 7. for all do 8. 9. 10. end for End

Algorithm-2 for PML leverages the popularity of items and interest of users. The popularity of the item

∑

( ) Where

is defined in Eq. (5).

[

(5) ]

if user

has experienced the item , otherwise, it is zero.

To find those items which are related to previously experienced items for a user, the relevance measurement is performed in the item information network between experienced and inexperienced items for a user following different meta-paths. For relevance measurement, we have utilized DPRel (Gupta et al., 2017b). The relevance score for those inexperienced items would be high that are closely related to the experienced items for that user. For a user-item pair, the recommendation score is computed by taking the summation of the weighted product of item popularity and relatedness score following various meta-paths. Finally, the personalized recommendation model is computed by storing the recommendation scores of all inexperienced items in sorted (descending) order. This recommendation model is used as input in the online module of the framework and personalized toprecommendations are generated using this learned model.

4.5. Personalized Top-N Recommendations (PTR) The Algorithm-3 for PTR is given below. For an active user, the PTR in the online module of the proposed framework generates top-N recommendations.

15

Algorithm-3 Input: : implicit feedback data : active user : recommendation score of items for active user Output: : top-N recommended items for active user Begin 1. for user do 2. 3. 4. // counter for experienced items for active user 5. // position of items experienced by the active user in the list 6. for to do 7. if // and is the set of top-N items for active user 8. 9. 10. end if 11. end for 12. // recommending top-N unexperienced items 13. end for End

The learned model from PML and feedback data from the user log is utilized as input for PTR. Typically, in real-world scenarios, 5 to 10 items are recommended to users so the value of would be 5 to 10. Thus, the online module of the proposed framework generates the top-N recommendations in constant time as the loop in this algorithm executes only

number of times. If a

recommended item is experienced/purchased by a user, then the list of recommended items is updated immediately to accommodate this new information.

5. Experimental Setup In this section, we discuss the experimental setup to perform a set of experiments on datasets from different domains. All experiments were performed on a system using R version 3.2.2. 5.1. Datasets For experiments, we have utilized three real-world datasets namely MovieLens-100K (http://files.grouplens.org/datasets/movielens/ml-100k.zip), Yahoo! Music (R2 - Yahoo! Music user ratings

of

songs

with

artist,

album,

and

genre

https://webscope.sandbox.yahoo.com/catalog.php?datatype=r)

and

meta-information,

v.

1.0,

HetRec2011-MovieLens-2K

(http://files.grouplens.org/datasets/hetrec2011/hetrec2011-movielens-2k-v2.zip). We, first, utilized MovieLens-100K dataset for experiments. This dataset consists of 100,000 ratings (on the scale of 1 – 5) from 943 users on 1,682 movies. In this dataset, each user has rated at 16

least 20 movies. The density of this dataset is about 6.3%. To make the dataset binary, we consider only rating 5 given by users which unambiguously represents a like by the user. The final dataset has 20,805 ratings given by 779 users to 1,169 movies such that each user has rated at least 5 movies and each movie is rated by at least one user. The density of this new dataset is only about 2.3%. The metainformation related to movies like directors, genres, and actors from IMDB (http://www.imdb.com/) website are utilized to form the heterogeneous movie information network. The top-5 actors, according to their credits, for each movie are collected. We term this dataset as Dataset-1, and the description of this dataset is given below in Table 3.

Table 3. Description of Dataset-1 # Items # Users # Ratings # Actors # Directors # Genres # Links # Entities Density (Movies) 1,169 779 20,805 3,499 728 19 29,905 6,194 2.3% Fig. 6(a) shows the network schema of Dataset-1. For experiments, the real ratings are converted to binary to form the binary implicit feedback matrix

using Eq. (1). Fig. 6(b) shows the

distribution of the user feedback in the dataset.

(a)

(b)

Fig. 6. Network schema and number of feedbacks for Dataset-1

The second dataset considered for experiments is Yahoo! Music dataset. This dataset is available with the meta-information related to songs. For our experiments, we take a subset of this dataset. Similar to Gao et al. (2009) and Salakhutdinov et al. (2007), we randomly sample the dataset for 2,000 songs such that each user has rated at least 20 songs and each song is rated by at least one user. The number of users in the sampled dataset is 12,923. The density of this dataset is about 1.9%. We term this dataset as Dataset-2, and the description of this dataset is given below in Table 4. 17

Table 4. Description of Dataset-2 # Items # Users # Ratings # Artists # Albums # Genres # Links # Entities Density (Songs) 2,000 12,923 515,869 1,312 1,838 28 521,869 18,101 1.9% The network schema of this heterogeneous network is shown below in Fig. 7(a). For empirical study in this work, the real ratings are considered to form a corresponding implicit feedback matrix

as defined in Eq. (1). The distribution of user feedback in this dataset is shown in Fig. 7(b).

(a)

(b)

Fig. 7. Network schema and number of feedbacks for Dataset-2 The third dataset considered for experiments is HetRec2011-MovieLens-2K which is an extension of MovieLens-10M dataset published by GroupLens research group (https://grouplens.org/). The dataset contains 2,113 users and 10,197 movies with the meta-information related to movies i.e. directors, actors, and genres. The density of this dataset is around 3.9%. To make the dataset binary, we consider only rating 5 given by users. The final dataset has 68,378 ratings (all ratings are 5) given by 1,428 users to 5,282 movies such that each user has rated at least 10 movies and each movie is rated by at least one user. The density of this new dataset is only 0.9%. The meta-information related to movies provided with the dataset like directors, genres, and lead actor for movies are utilized to form the heterogeneous movie information network for this dataset. We term this dataset as Dataset-3, and the description of this dataset is given below in Table 5.

Table 5. Description of Dataset-3 # Items # Users # Ratings # Actors # Directors # Genres # Links # Entities Density (Movies) 5,282 1,428 68,378 3,662 2,264 19 92,014 7,373 0.9%

18

The network schema of the heterogeneous network for Dataset-3 is shown below in Fig. 8(a). For empirical study in this work, the real ratings are considered to form a corresponding implicit feedback matrix

as defined in Eq. (1). The distribution of user feedback in this dataset is shown in

Fig. 8(b).

(a)

(b)

Fig. 8. Network schema and number of feedbacks for Dataset-3

To evaluate a recommendation technique, the binary implicit feedback matrix

for each dataset

is divided into training and test matrices (Bobadilla et al., 2013). Following 80% - 20% division of five train (

) and corresponding test (

) matrices are formed for each

dataset. A recommendation model is first trained using a train matrix ( testing is performed using the corresponding test matrix (

,

) for a dataset and then

). This training - testing process is

performed for all five train and test pairs of matrices for each dataset.

5.2. Meta-paths In our experiments, we have utilized meta-paths that are listed in Table 6 for both Dataset-1 and Dataset-3 as the network schema is same for both datasets. For Dataset-2, meta-paths are listed in Table 7. These meta-paths are generated considering only the item and its meta-information part in the network schema. For personalized model learning, we take the meta-paths in the format of . However, the meta-paths in the format of

are taken for

personalized weight learning.

19

Table 6. Meta-paths for Dataset-1 and Dataset-3 Notation

Symmetric

movie – director – movie

MDM

Yes

movie – actor – movie

MAM

Yes

movie – genre – movie

MGM

Yes

movie – director – movie – actor – movie

MDMAM

No

movie – genre – movie – actor – movie

MGMAM

No

movie – director – movie – genre – movie

MDMGM

No

movie – actor – movie – director – movie

MAMDM

No

movie – actor – movie – genre – movie

MAMGM

No

movie – genre – movie – director – movie

MGMDM

No

Meta-path ( )

Table 7. Meta-paths for Dataset-2 Meta-path ( )

Notation

Symmetric

song – album – song

SAS

Yes

song – artist – song

SHS

Yes

song – genre – song

SGS

Yes

song – album – song – artist – song

SASHS

No

song – genre – song – artist – song

SGSHS

No

song – album – song – genre – song

SASGS

No

song – artist – song – album – song

SHSAS

No

song – artist – song – genre – song

SHSGS

No

song – genre – song – album – song

SGSAS

No

In our experiments, we have considered those meta-paths that have length not more than four. For example, meta-path SASHS has length four. The longer meta-paths in the network would not be 20

semantically meaningful and the following those meta-paths would result in an increased number of noisy relations (Yu et al., 2014). Also, those meta-paths that have identical sub-paths are not considered in our experiments as this kind of meta-paths would be highly noisy (Kong et al., 2012). For example, the meta-path SHSHS has two identical sub-paths i.e. SHS and would be highly noisy.

5.3. Algorithms for Comparison To show the effectiveness, the performance of HeteroPRS is compared with the following recommendation techniques applicable to binary implicit feedback: 

Popularity: The Popularity based recommendation model recommends the most popular items to the users (Bobadilla et al., 2013).



Neighborhood Model: For comparison, IBCF is utilized in this work as it is scalable and has experimentally shown better performance in many cases as compared to user-based collaborative filtering (Deshpande & Karypis, 2004).



Association Rule Model: This method generates recommendations on the basis of a set of association rules computed using the training data (Demiriz, 2004).



Bayesian Personalized Ranking based Matrix Factorization (BPRMF): This model is optimized for personalized item ranking task for implicit feedback (Rendle et al., 2009).



FunkSVD: It is a stochastic gradient descent optimization technique which loops through all feedbacks in the training set (Koren et al., 2009).



HeteRec: It is a preference diffusion based methodology and in this work, both the global and personalized versions of HeteRec are utilized for comparison (Yu et al., 2014). In this work, the “recommenderlab” package in R (Hahsler, 2011) is utilized for Popularity,

Neighborhood Model, and Association Rule Model. For BPRMF and FunkSVD, “rrecsys” package in R (Çoba & Zanker, 2016) is utilized. Meta-paths listed in Table 6 are utilized for Dataset-1 and Dataset-3, while meta-paths listed in Table 7 are utilized for Dataset-2 when comparing the performance of HeteroPRS1 with the aforementioned techniques. For HeteRec2, only symmetric metapaths are utilized as HeteRec cannot utilize asymmetric meta-paths. In the experiments, for all three datasets, the best parameter values are selected for different algorithms by repeated random sub-sampling cross-validation using training data (Yu et al., 2014; Sarwar et al., 2001; Rendle et al., 2009; Bauer & Nanopoulos, 2014). For IBCF, the neighborhood size is tuned from {10, 20, 30, …, 80} (Sarwar et al., 2001). The number of factors for BPRMF are tuned from {10, 20, 30, …, 200} with a threshold of 1e-05 for delta error (Rendle et al., 2009). The three regularization terms in BPRMF i.e. for user features, positive and negative updates for the item features are all set to 0.0025 and learning rate is set to 0.05 (Rendle et al., 2009). For FunkSVD, the

1 2

https://github.com/mukulg17/HeteroPRS https://github.com/mukulg17/HeteRec 21

number of factors is tuned from {10, 20, 30, …, 200} with a threshold of 1e-05 for delta error, and learning rate and regularization parameter are set to 0.001 and 0.01 respectively (Çoba & Zanker, 2016; Bauer & Nanopoulos, 20014). For HeteRec, the number of user clusters are tuned from {2, 10, 20, 40, 100, 200, 400} and the sampling rate is set to

(Yu et al., 2014). The regularization

parameter is set to 0.1 and the dimensionality of the low-rank representation is tuned from {5, 10, 20, 40, 60, 100, 150} (Yu et al., 2014). 5.4. Performance Evaluation The trust of users on the recommendations depends on the accuracy of the recommended items (Bobadilla et al., 2013; Bellogin et al., 2013). In this work, top-N items are recommended to users; therefore, relevance-based evaluation measure would be appropriate. For evaluation, those items which belong to the test set of a user (have 1 in the test set of that user) are relevant items (Bellogin et al., 2013). However, the position of relevant items in the recommended ranked list of items matters significantly as in practical situation, users would see only a small number of topranked items. Therefore, we utilized generalized Average Reciprocal Hit Rate (ARHR) for evaluation (Aggarwal, 2016; Deshpande & Karypis, 2004). For a recommendation list of size N for a user u, the ARHR is defined as follows: ∑

(6)

where

The global ARHR value can be computed by averaging the ARHR value for all m users as given below: ∑

(7)

We utilized another evaluation measure called Mean Average Precision (MAP) for the performance comparison of different recommendation approaches (Aggarwal, 2016). For a ranked list of length N of items, the average precision at the rank of each relevant item is computed for a user u as given below: ∑

(8)

where,

22

By taking the mean of AP for all m users, we compute the MAP as given below: ∑

(9)

To evaluate the performance of different recommendation approaches, we also use Recall measure (Aggarwal, 2016). For a ranked list of length N of items, the Average Recall is computed as given below: ∑

(10)

where,

6. Experimental Results and Discussion In this section, we present the experimental results. 6.1 Results for Dataset-1 Tables 8-10 show the performance of different recommendation algorithms. The mean values from all five test sets are reported in these tables. From the values, we can see that the proposed method outperforms other methods taken for comparison. In Table 8, the FunkSVD method gives slightly better Average Recall as compared to HeteroPRS after

. For the rest of the evaluation

measures, the HeteroPRS performs better and gives stable results.

Table 8. Average Recall values of different algorithms for Dataset-1 AR@N(↑)

1

2

3

4

5

6

7

8

9

10

POPULAR

0.0271 0.004

0.0455 0.003

0.059 0.003

0.0723 0.005

0.0841 0.006

0.0943 0.006

0.106 0.007

0.1146 0.005

0.1248 0.003

0.1322 0.004

IBCF

0.0209 0.003

0.0377 0.002

0.0533 0.004

0.0665 0.004

0.0792 0.007

0.091 0.01

0.1016 0.008

0.1137 0.007

0.1247 0.005

0.1358 0.004

AR

0.0173 0.003

0.028 0.003

0.034 0.006

0.0378 0.005

0.0427 0.005

0.047 0.004

0.0501 0.004

0.053 0.002

0.0533 0.002

0.0533 0.002

BPRMF

0.0263 0.004

0.0416 0.005

0.056 0.002

0.069 0.005

0.087 0.002

0.0979 0.001

0.1075 0.002

0.1215 0.005

0.1341 0.006

0.1444 0.006

FunkSVD

0.0184 0.006

0.0375 0.006

0.0517 0.005

0.0664 0.005

0.0809 0.004

0.0966 0.006

0.1112 0.005

0.1271 0.004

0.1391 0.003

0.1499 0.004

HeteRec

0.001 0.001

0.003 0.001

0.0049 0.002

0.0067 0.002

0.0084 0.003

0.0101 0.003

0.0118 0.003

0.014 0.005

0.0153 0.005

0.0164 0.006

HeteroPRS

0.0314 0.002

0.0517 0.004

0.067 0.005

0.0803 0.005

0.0925 0.006

0.1037 0.004

0.1147 0.005

0.1249 0.005

0.1359 0.005

0.1456 0.005

23

Table 9. MAP values of different algorithms for Dataset-1 AR@N(↑)

1

2

3

4

5

6

7

8

9

10

POPULAR

0.1089 0.009

0.1443 0.008

0.1595 0.008

0.1678 0.008

0.1727 0.009

0.1746 0.009

0.1762 0.009

0.1765 0.008

0.1768 0.008

0.1769 0.008

IBCF

0.0555 0.01

0.0792 0.008

0.0932 0.01

0.1011 0.008

0.1059 0.008

0.1105 0.007

0.1136 0.008

0.1158 0.007

0.1182 0.008

0.1196 0.007

AR

0.0642 0.008

0.0882 0.01

0.095 0.012

0.0988 0.011

0.1017 0.011

0.1043 0.01

0.1061 0.01

0.1074 0.009

0.1075 0.01

0.1075 0.009

BPRMF

0.1186 0.009

0.1579 0.007

0.1764 0.008

0.1861 0.009

0.1916 0.01

0.1953 0.009

0.1966 0.008

0.1972 0.008

0.1978 0.009

0.1968 0.008

FunkSVD

0.0752 0.021

0.1014 0.021

0.116 0.02

0.1253 0.019

0.132 0.017

0.1378 0.017

0.1408 0.016

0.1439 0.016

0.1455 0.016

0.1463 0.016

HeteRec

0.0082 0.003

0.0144 0.004

0.0184 0.006

0.0213 0.006

0.023 0.006

0.0243 0.006

0.0249 0.006

0.0255 0.007

0.0261 0.007

0.0263 0.007

HeteroPRS

0.1389 0.012

0.1769 0.012

0.1903 0.013

0.1979 0.012

0.2009 0.011

0.2023 0.01

0.2029 0.01

0.2026 0.01

0.2014 0.009

0.2008 0.009

10

Table 10. ARHR values of different algorithms for Dataset-1 AR@N(↑)

1

2

3

4

5

6

7

8

9

POPULAR

0.1089 0.009

0.1519 0.008

0.1751 0.008

0.1921 0.009

0.2043 0.009

0.2134 0.01

0.2211 0.011

0.2263 0.011

0.2312 0.01

0.235 0.01

IBCF

0.0555 0.01

0.0814 0.008

0.097 0.01

0.107 0.008

0.1148 0.008

0.1213 0.007

0.1266 0.008

0.1316 0.007

0.1366 0.007

0.141 0.007

AR

0.0642 0.008

0.0915 0.01

0.1012 0.012

0.1066 0.01

0.1103 0.01

0.114 0.01

0.1165 0.01

0.1186 0.009

0.1189 0.009

0.1189 0.009

BPRMF

0.1186 0.009

0.1674 0.008

0.1955 0.009

0.214 0.01

0.2267 0.01

0.2378 0.01

0.2465 0.01

0.2532 0.01

0.2586 0.01

0.2638 0.01

FunkSVD

0.0752 0.021

0.1076 0.022

0.1273 0.021

0.1421 0.02

0.1536 0.019

0.1637 0.02

0.1721 0.019

0.1796 0.02

0.1859 0.019

0.191 0.019

HeteRec

0.0082 0.003

0.0149 0.005

0.0193 0.006

0.0223 0.006

0.0243 0.006

0.0258 0.006

0.0269 0.006

0.0279 0.007

0.0286 0.008

0.0292 0.008

HeteroPRS

0.1389 0.012

0.1899 0.013

0.213 0.012

0.2297 0.012

0.2424 0.011

0.2515 0.011

0.2598 0.012

0.2665 0.012

0.272 0.012

0.2769 0.012

6.2. Results for Dataset-2 Tables 11-13 show the performance of different recommendation algorithms. The mean values from all five test sets are reported in these tables. From the values, we can see that the proposed method outperforms other methods taken for comparison. 24

Table 11. Average Recall values of different algorithms for Dataset-2 AR@N(↑) POPULAR IBCF AR BPRMF FunkSVD HeteRec HeteroPRS

1

2

3

4

5

6

7

8

9

10

0.021 0 0.0167 0 0.0156 0.001 0.0287 0.001 0.0119 0.001 0.0008 0 0.0352 0

0.0396 0.001 0.0318 0 0.0283 0.001 0.052 0.001 0.0224 0.002 0.0013 0 0.0598 0.001

0.0562 0.001 0.0468 0.001 0.0394 0.001 0.0727 0.001 0.0307 0.002 0.0017 0 0.0815 0.001

0.0709 0.001 0.0613 0.001 0.0497 0.001 0.0912 0.001 0.0374 0.002 0.0021 0 0.1011 0.001

0.0838 0.001 0.0752 0.002 0.0591 0.001 0.108 0.002 0.0436 0.003 0.0026 0 0.1186 0.001

0.0955 0.001 0.0888 0.002 0.068 0.001 0.1235 0.001 0.05 0.003 0.003 0 0.1342 0.001

0.1067 0.001 0.1022 0.002 0.0772 0.001 0.1374 0.001 0.0562 0.004 0.0036 0 0.1482 0.001

0.1163 0.001 0.1151 0.002 0.0853 0.001 0.1509 0.001 0.062 0.004 0.0041 0 0.161 0.001

0.1249 0.001 0.128 0.001 0.0923 0.001 0.1632 0.001 0.0678 0.005 0.0046 0 0.173 0

0.133 0.001 0.1403 0.001 0.0984 0.001 0.1753 0.001 0.0734 0.005 0.005 0 0.1839 0.001

Table 12. MAP values of different algorithms for Dataset-2 AR@N(↑) POPULAR IBCF AR BPRMF FunkSVD HeteRec HeteroPRS

1

2

3

4

5

6

7

8

9

10

0.1611 0.002 0.1214 0.002 0.114 0.004 0.222 0.007 0.0894 0.01 0.009 0 0.2571 0.004

0.2168 0.003 0.1611 0.003 0.154 0.005 0.2855 0.007 0.1241 0.012 0.0106 0 0.318 0.004

0.2423 0.003 0.1835 0.002 0.1746 0.005 0.3115 0.007 0.1398 0.013 0.0116 0.001 0.3443 0.004

0.2544 0.003 0.1973 0.002 0.1861 0.005 0.3232 0.006 0.1481 0.013 0.0122 0.001 0.3563 0.003

0.2596 0.003 0.2062 0.003 0.1929 0.005 0.3282 0.006 0.1532 0.012 0.0129 0 0.3602 0.003

0.2618 0.003 0.2122 0.003 0.1969 0.004 0.3293 0.006 0.1569 0.012 0.0135 0.001 0.3606 0.003

0.2627 0.003 0.2164 0.002 0.1994 0.004 0.3289 0.006 0.1594 0.012 0.0142 0.001 0.3596 0.003

0.2622 0.003 0.219 0.002 0.2008 0.004 0.3272 0.005 0.1611 0.012 0.0146 0.001 0.3573 0.003

0.2611 0.002 0.2205 0.002 0.2012 0.004 0.3248 0.005 0.1621 0.011 0.015 0.001 0.3541 0.003

0.2596 0.002 0.2216 0.002 0.2012 0.004 0.322 0.005 0.1626 0.011 0.0152 0.001 0.3511 0.003

Table 13. ARHR values of different algorithms for Dataset-2 AR@N(↑) POPULAR IBCF AR BPRMF FunkSVD HeteRec HeteroPRS

1

2

3

4

5

6

7

8

9

10

0.1611 0.002 0.1214 0.002 0.114 0.004 0.222 0.007 0.0894 0.01 0.009 0 0.2571 0.004

0.2345 0.004 0.1768 0.003 0.1609 0.006 0.3138 0.009 0.1286 0.014 0.0122 0 0.3496 0.004

0.2793 0.003 0.2134 0.004 0.1891 0.005 0.3682 0.009 0.149 0.015 0.0139 0 0.405 0.004

0.3088 0.003 0.2403 0.004 0.2082 0.005 0.4052 0.009 0.1615 0.016 0.0149 0 0.443 0.004

0.3295 0.003 0.2613 0.005 0.2226 0.005 0.4322 0.009 0.1709 0.017 0.0159 0 0.4704 0.003

0.3453 0.003 0.2784 0.005 0.2343 0.005 0.4531 0.009 0.179 0.017 0.0168 0.001 0.4908 0.003

0.3578 0.004 0.2931 0.005 0.2442 0.004 0.4695 0.008 0.1857 0.018 0.0176 0.001 0.5066 0.003

0.3673 0.003 0.3056 0.005 0.2521 0.005 0.4832 0.008 0.1914 0.019 0.0183 0.001 0.5192 0.003

0.3753 0.003 0.3167 0.004 0.2582 0.005 0.4945 0.008 0.1963 0.019 0.0189 0.001 0.5298 0.003

0.3819 0.003 0.3264 0.004 0.2626 0.005 0.5044 0.009 0.2006 0.019 0.0194 0.001 0.5384 0.003

25

6.3. Results for Dataset-3 This dataset is sparser than Dataset-1 and Dataset-2. Tables 14-16 show the performance of different recommendation algorithms. The mean values from all five test sets are reported in these tables. From the values, we can see that the proposed method outperforms other methods taken for comparison. Table 14. Average Recall values of different algorithms for Dataset-3 AR@N(↑) POPULAR IBCF AR BPRMF FunkSVD HeteRec HeteroPRS

1

2

3

4

5

6

7

8

9

10

0.014 0.001 0.0198 0.002 0.0229 0.003 0.0137 0.002 0.0142 0.001 0.0004 0 0.0307 0.001

0.0287 0.002 0.0328 0.002 0.0375 0.003 0.0287 0.001 0.0285 0.002 0.0006 0 0.0498 0.002

0.0428 0.004 0.043 0.002 0.0487 0.003 0.0417 0.002 0.0431 0.003 0.0008 0 0.0652 0.002

0.0551 0.003 0.0519 0.003 0.0599 0.003 0.0532 0.003 0.0554 0.003 0.0012 0 0.0771 0.002

0.0675 0.003 0.0612 0.003 0.0711 0.002 0.0652 0.003 0.0681 0.003 0.0014 0 0.0899 0.002

0.0783 0.002 0.0686 0.003 0.0798 0.003 0.0746 0.003 0.0783 0.002 0.0017 0 0.1013 0.002

0.0857 0.003 0.0757 0.003 0.0871 0.003 0.082 0.003 0.0857 0.003 0.0019 0 0.1098 0.002

0.0926 0.004 0.0831 0.002 0.0938 0.003 0.0907 0.003 0.0926 0.004 0.0023 0 0.1188 0.003

0.1001 0.004 0.0896 0.002 0.0995 0.002 0.0987 0.003 0.1002 0.003 0.0029 0.001 0.1269 0.004

0.1065 0.003 0.0963 0.002 0.1047 0.002 0.107 0.005 0.1063 0.003 0.0031 0.001 0.1339 0.004

Table 15. MAP values of different algorithms for Dataset-3 AR@N(↑) POPULAR IBCF AR BPRMF FunkSVD HeteRec HeteroPRS

1

2

3

4

5

6

7

8

9

10

0.1011 0.008 0.0908 0.004 0.1373 0.015 0.1022 0.009 0.1018 0.008 0.0032 0.002 0.1873 0.006

0.1429 0.008 0.1151 0.003 0.1751 0.013 0.1409 0.009 0.1429 0.008 0.0041 0.002 0.2338 0.009

0.1645 0.011 0.1278 0.003 0.1909 0.012 0.1621 0.008 0.1648 0.011 0.0052 0.003 0.2518 0.009

0.1755 0.01 0.1355 0.003 0.2008 0.011 0.1728 0.009 0.1758 0.01 0.0061 0.002 0.2602 0.009

0.1829 0.01 0.1407 0.003 0.2061 0.011 0.1789 0.008 0.1833 0.009 0.0068 0.002 0.2648 0.009

0.1879 0.009 0.1441 0.003 0.209 0.01 0.1824 0.008 0.188 0.009 0.0072 0.002 0.2662 0.008

0.1896 0.008 0.1467 0.003 0.2113 0.01 0.1848 0.008 0.1898 0.008 0.0077 0.002 0.2659 0.007

0.1906 0.008 0.1481 0.003 0.2116 0.01 0.186 0.008 0.1906 0.008 0.008 0.002 0.2657 0.007

0.1911 0.008 0.149 0.002 0.2115 0.009 0.1857 0.008 0.1907 0.008 0.0084 0.002 0.265 0.008

0.1902 0.008 0.1496 0.003 0.2115 0.01 0.1849 0.007 0.1902 0.008 0.0085 0.002 0.263 0.007

26

Table 16. ARHR values of different algorithms for Dataset-3 AR@N(↑) POPULAR IBCF AR BPRMF FunkSVD HeteRec HeteroPRS

1

2

3

4

5

6

7

8

9

10

0.1011 0.008 0.0908 0.004 0.1373 0.015 0.1022 0.009 0.1018 0.008 0.0032 0.002 0.1873 0.006

0.1521 0.008 0.1236 0.003 0.1856 0.014 0.1499 0.008 0.1518 0.008 0.0041 0.002 0.254 0.011

0.1852 0.012 0.1419 0.004 0.2109 0.014 0.1795 0.007 0.1853 0.012 0.0052 0.002 0.2897 0.012

0.2074 0.011 0.154 0.004 0.2293 0.014 0.1989 0.009 0.2078 0.011 0.0061 0.002 0.3125 0.01

0.2237 0.012 0.1637 0.004 0.2426 0.013 0.2148 0.009 0.2241 0.011 0.0069 0.002 0.3304 0.01

0.236 0.011 0.1706 0.004 0.2522 0.014 0.2263 0.009 0.2364 0.011 0.0074 0.002 0.3444 0.01

0.2448 0.011 0.1763 0.005 0.2601 0.014 0.2357 0.009 0.245 0.011 0.0079 0.002 0.3544 0.01

0.252 0.01 0.1816 0.005 0.2661 0.013 0.2428 0.009 0.2522 0.01 0.0082 0.002 0.3636 0.009

0.2584 0.01 0.1858 0.004 0.2711 0.013 0.2488 0.009 0.2588 0.01 0.0087 0.002 0.371 0.009

0.2639 0.01 0.1899 0.004 0.2751 0.013 0.2551 0.008 0.2641 0.01 0.0088 0.002 0.3767 0.009

From the results, we can see that the performance of HeteroPRS is significantly better than the other algorithms. The performance of HeteRec is not effective and that may be due to the high level of sparsity of the feedback data. As HeteRec can utilize only symmetric meta-paths for recommendation generation and depends on the clustering of users (Yu et al., 2014), the accuracy of the recommendation by HeteRec is low. 6.4 Statistical comparison of performance differences We use the non-parametric Friedman test (Demšar, 2006; Bogaert et al., 2019; Fitzpatrick & Mues, 2016) to determine whether performance of POPULAR, IBCF, AR, BPRMF, FunkSVD, HeteRec and HeteroPRS differ significantly (

) or not (

). To perform the Friedman test, we

consider the average ranks of the algorithms based on all three datasets (Demšar, 2006). The results of the test are shown in Table 17. Table 17: Related samples Friedman’s two-way analysis of variance by ranks Friedman Test Evaluation measure

Value (

)

-value

AR@N

50.735

< 0.001

6

MAP@N

59.555

< 0.001

6

ARHR@N

59.674

< 0.001

6

Note: For each dataset and evaluation measure (5% level of significance). From the results of Friedman test ( and -value), the null hypothesis ( ) stating that there is no significant difference between performance of POPULAR, IBCF, AR, BPRMF, FunkSVD, HeteRec and HeteroPRS, is rejected for all three datasets using three evaluation measures.

27

Based on the results, we are able to conclude with a 95 percent confidence that the performance differ significantly. Since the null hypothesis (

) is rejected, we perform the

Bonferroni-Dunn post-hoc test to compare the best performing algorithm i.e. HeteroPRS with other algorithms (Demšar, 2006; Bogaert et al., 2019). For pairwise statistical comparison of HeteroPRS with other algorithms, we use Wilcoxon signed-rank test (Debaere et al., 2018; Lin et al., 2015). The results of the post-hoc test are shown in Table 18. From the results, we can see that for all three datasets, the performance of HeteroPRS is significantly better than the other algorithms for the three evaluation measures. These results show the effectiveness of HeteroPRS for recommendation generation. Table 18: Bonferroni-Dunn post-hoc test for Algorithm compared with HeteroPRS

Test Statistics

AR@N

MAP@N

ARHR@N

-2.821 -2.913 -2.97 POPULAR -value 0.005 0.004 0.003 Adjusted -value 0.03 0.024 0.018 -value -2.816 -2.919 -2.919 IBCF -value 0.005 0.004 0.004 Adjusted -value 0.03 0.024 0.024 -value -2.911 -2.919 -2.919 AR -value 0.004 0.004 0.004 Adjusted -value 0.024 0.024 0.024 -value -2.831 -3.051 -3.051 BPRMF -value 0.005 0.002 0.002 Adjusted -value 0.03 0.012 0.012 -value -2.805 -2.919 -2.972 FunkSVD -value 0.005 0.004 0.003 Adjusted -value 0.03 0.024 0.018 -value -2.919 -3.162 -3.162 HeteRec -value 0.004 0.002 0.002 Adjusted -value 0.024 0.012 0.012 Note: values are based on negative ranks. For pairwise comparison of HeteroPRS with other algorithms i.e. POPULAR, IBCF, AR, BPRMF, FunkSVD and HeteRec, we use the nonparametric Wilcoxon signed-rank test. The -values are adjusted using the Bonferroni-Dunn method. -value

7. Conclusion and Future Work In this paper, the effectiveness of recommendation systems have been studied for sparse and noisy binary implicit feedback. From the literature, it is found that the model based IBCF is a popular technique and utilized by many e-commerce websites like Amazon.com. However, IBCF is not effective for highly sparse binary implicit feedback. Also, matrix factorization based techniques like BPRMF and FunkSVD are not effective for sparse and noisy binary implicit feedback. In this paper, we have utilized meta-information related to items to form heterogeneous item information network to address the problem of sparseness and noisiness of binary implicit feedback. To improve the 28

effectiveness of recommendations, the popularity of items and the intrinsic interest of users are leveraged simultaneously. We have proposed a novel personalized weight learning methodology for different meta-paths in the network to determine the intrinsic interest of users. The effectiveness of the proposed recommendation framework is demonstrated using real-world datasets. The performance of the proposed framework is compared with a number of recommendation algorithms including HeteRec that also uses the meta-information related to the items. The performance of HeteRec is not effective as it cannot utilize arbitrary meta-paths and it uses the clustering of users for recommendation generation. Also, the parameter learning methodology of HeteRec is not effective that also reduces the accuracy of the recommendations generated by HeteRec. The proposed framework for recommendation generation utilizes the popularity of items, therefore, if a new item is introduced or if there is an item which has never been experienced then that item will not be recommended using the proposed framework. Similarly, the recommendations will not be generated for a user as that user has not experienced any item. These problems are known as user and item cold start respectively and the proposed framework will not be able to deal with these problems. As future research directions, we would like to modify the proposed framework to deal with the user and item cold start problems. Also, it would be interesting to incorporate the demographic of users for improving the effectiveness of recommendations. Another interesting research direction would be to make the model flexible to incorporate the users’ explicit choice of meta-paths and also the incorporation of graph-regularization to improve the accuracy.

29

References Adomavicius, G., & Tuzhilin, A. (2005). Toward the next generation of recommender systems: A survey of the state-of-the-art and possible extensions. IEEE transactions on knowledge and data engineering, 17(6), 734-749. Aggarwal, C. C. (2016). Recommender Systems: The Textbook. Springer. Barman, K., & Dabeer, O. (2012). Analysis of a collaborative filter based on popularity amongst neighbors. IEEE Transactions on Information Theory, 58(12), 7110-7134. Basilico, J., & Hofmann, T. (2004). Unifying collaborative and content-based filtering In Proceedings of the twenty-first international conference on Machine learning (p. 9). ACM. Bauer, J., & Nanopoulos, A. (2014). Recommender systems based on quantitative implicit customer feedback. Decision Support Systems, 68, 77-88. Bellogin, A., Cantador, I., & Castells, P. (2013). A comparative study of heterogeneous item recommendations in social systems. Information Sciences, 221, 142-169. Berkovsky, S., & Freyne, J. (2015). Web Personalization and Recommender Systems. In Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (pp. 2307-2308). ACM. Bobadilla, J., Ortega, F., Hernando, A., & Gutiérrez, A. (2013). Recommender systems survey. Knowledge-Based Systems, 46, 109-132. Bogaert, M., Lootens, J., Van den Poel, D., & Ballings, M. (2019). Evaluating Multi-Label Classifiers and Recommender Systems in the Financial Service Sector. European Journal of Operational Research, 279(2), 620-634. Çoba, L. & Zanker, M. project.org/web/packages/rrecsys/rrecsys.pdf

(2016).

Package

‘rrecsys’.

https://cran.r

Debaere, S., Coussement, K., & De Ruyck, T. (2018). Multi-label classification of member participation in online innovation communities. European Journal of Operational Research, 270(2), 761-774. Demiriz, A. (2004). Enhancing product recommender systems on sparse binary data. Data Mining and Knowledge Discovery, 9(2), 147-170. Demšar, J. (2006). Statistical comparisons of classifiers over multiple data sets. Journal of Machine learning research, 7(Jan), 1-30. Deshpande, M., & Karypis, G. (2004). Item-based top-n recommendation algorithms. ACM Transactions on Information Systems (TOIS), 22(1), 143-177. Fitzpatrick, T., & Mues, C. (2016). An empirical comparison of classification algorithms for mortgage default prediction: evidence from a distressed mortgage market. European Journal of Operational Research, 249(2), 427-439. Felfernig, A., Jeran, M., Ninaus, G., Reinfrank, F., & Reiterer, S. (2013). Toward the next generation of recommender systems: applications and research challenges. In Multimedia services in intelligent environments (pp. 81-98). Springer International Publishing Gan, M. (2016). COUSIN: A network-based regression model for personalized recommendations. Decision Support Systems, 82, 58-68. Gao, J., Fan, W., Sun, Y., & Han, J. (2009). Heterogeneous source consensus learning via decision propagation and negotiation. In Proceedings of the 15th ACM SIGKDD international conference on Knowledge discovery and data mining (pp. 339-348). ACM.

30

Geuens, S., Coussement, K., & De Bock, K. W. (2018). A framework for configuring collaborative filtering-based recommendations derived from purchase data. European Journal of Operational Research, 265(1), 208-218. Gupta, M., Kumar, P., & Bhasker, B. (2017a). HeteClass: A Meta-path based framework for transductive classification of objects in heterogeneous information networks. Expert Systems with Applications, 68, 106-122. Gupta, M., Kumar, P., & Bhasker, B. (2017b). DPRel: A Meta-Path Based Relevance Measure for Mining Heterogeneous Networks. Information Systems Frontiers, 1-17. Gupta, M., Kumar, P., & Mishra, R. (2017c). Interest Diffusion in Heterogeneous Information Network for Personalized Item Ranking. In Proceedings of the 2017 ACM on Conference on Information and Knowledge Management (pp. 2087-2090). ACM. Hahsler, M. (2011). recommenderlab: A Framework for Developing and Testing Recommendation Algorithms. Nov. Hu, Y., Koren, Y., & Volinsky, C. (2008). Collaborative filtering for implicit feedback datasets. In 2008 Eighth IEEE International Conference on Data Mining (pp. 263-272). IEEE. Jannach, D., Zanker, M., Felfernig, A., & Friedrich, G. (2010). Recommender systems: an introduction. Cambridge University Press. Kong, X., Yu, P. S., Ding, Y., & Wild, D. J. (2012). Meta path-based collective classification in heterogeneous information networks. In Proceedings of the 21st ACM international conference on Information and knowledge management (pp. 1567-1571). ACM. Koren, Y., Bell, R., & Volinsky, C. (2009). Matrix factorization techniques for recommender systems. Computer, 42(8), 30-37. Linden, G., Smith, B., & York, J. (2003). Amazon.com recommendations: Item-to-item collaborative filtering. IEEE Internet computing, 7(1), 76-80. Lin, W. T., Chen, Y. H., & Shao, B. B. (2015). Assessing the business values of information technology and e-commerce independently and jointly. European Journal of Operational Research, 245(3), 815-827. Li, Y. M., Wu, C. T., & Lai, C. Y. (2013). A social recommender mechanism for e-commerce: Combining similarity, trust, and relationship. Decision Support Systems, 55(3), 740-752. Luo, C., Pang, W., Wang, Z., & Lin, C. (2014). Hete-CF: Social-based collaborative filtering recommendation using heterogeneous relations. In Data Mining (ICDM), IEEE International Conference on (pp. 917-922). IEEE. McFee, B., Bertin-Mahieux, T., Ellis, D. P., & Lanckriet, G. R. (2012). The million song dataset challenge. In Proceedings of the 21st International Conference on World Wide Web (pp. 909-916). ACM. Pan, R., Zhou, Y., Cao, B., Liu, N. N., Lukose, R., Scholz, M., & Yang, Q. (2008, December). One class collaborative filtering. In 2008 Eighth IEEE International Conference on Data Mining (pp. 502511). IEEE. Papamichail, G. P., & Papamichail, D. P. (2007). The k-means range algorithm for personalized data clustering in e-commerce. European Journal of Operational Research, 177(3), 1400-1408. Rendle, S., Freudenthaler, C., Gantner, Z., & Schmidt-Thieme, L. (2009). BPR: Bayesian personalized ranking from implicit feedback. In Proceedings of the twenty-fifth conference on uncertainty in artificial intelligence (pp. 452-461). AUAI Press. Ricci, F., Rokach, L., & Shapira, B. (2011). Introduction to recommender systems handbook (pp. 135). Springer US.

31

Salakhutdinov, R., Mnih, A., & Hinton, G. (2007). Restricted Boltzmann machines for collaborative filtering. In Proceedings of the 24th international conference on Machine learning (pp. 791-798). ACM. Sarwar, B. M., Karypis, G., Konstan, J., & Riedl, J. (2002). Recommender systems for large-scale ecommerce: Scalable neighborhood formation using clustering. In Proceedings of the fifth international conference on computer and information technology. Sarwar, B. M., Karypis, G., Konstan, J., & Riedl, J. (2001). Item-based collaborative filtering recommendation algorithms. In Proceedings of the 10th international conference on World Wide Web (pp. 285-295). ACM. Scholz, M., Dorner, V., Schryen, G., & Benlian, A. (2017). A configuration-based recommender system for supporting e-commerce decisions. European Journal of Operational Research, 259(1), 205-215. Shi, C., Zhang, Z., Luo, P., Yu, P. S., Yue, Y., & Wu, B. (2015). Semantic path based personalized recommendation on weighted heterogeneous information networks. In Proceedings of the 24th ACM International on Conference on Information and Knowledge Management (pp. 453-462). ACM. Sun, Y., & Han, J. (2013). Mining heterogeneous information networks: a structural analysis approach. ACM SIGKDD Explorations Newsletter, 14(2), 20-28. Sun, Y., Han, J., Yan, X., Yu, P. S., & Wu, T. (2011). PathSim: Meta path-based top-k similarity search in heterogeneous information networks. Proceedings of the VLDB Endowment, 4(11), 9921003. Yu, X., Ren, X., Sun, Y., Gu, Q., Sturt, B., Khandelwal, U., Norick B., & Han, J. (2014). Personalized entity recommendation: A heterogeneous information network approach. In Proceedings of the 7th ACM international conference on Web search and data mining (pp. 283-292). ACM.

32