Methods Supporting Supplier Selection Processes – Knowledge-based Approach

Available online at www.sciencedirect.com

ScienceDirect ScienceDirect

Procediaonline Computer 00 (2019) 000–000 Available at Science www.sciencedirect.com Procedia Computer Science 00 (2019) 000–000

ScienceDirect

www.elsevier.com/locate/procedia www.elsevier.com/locate/procedia

Procedia Computer Science 159 (2019) 1629–1641

23rd International Conference on Knowledge-Based and Intelligent Information & Engineering 23rd International Conference on Knowledge-Based Systems and Intelligent Information & Engineering Systems

Methods Supporting Supplier Selection Processes – KnowledgeMethods Supporting Supplier Selection Processes – Knowledgebased Approach based Approach Agnieszka Konys* Agnieszka Konys*

West-Pomeranian University of Technology in Szczecin, Faculty of Computer Science and Information Technology, Żołnierska 49, 71-210 Szczecin, Poland Science and Information Technology, Żołnierska 49, 71-210 West-Pomeranian University of Technology in Szczecin, Faculty of Computer Szczecin, Poland

Abstract Abstract Nowadays, companies have to improve their practices in the management of sustainable supply chain with increased awareness of environmental, Nowadays, companies economic have toand improve social their issues practices globally. in the Selecting management the optimum of sustainable sustainable supply supplier chain with is crucial increased for sustainable awareness of environmental, economic which and social globally. Selecting the optimum supplier isperspective, crucial for sustainable supply chain management, is a issues challenging multi-dimensional problem.sustainable From a systematic supply management, which is a challenging multi-dimensional problem. From This a systematic perspective, sustainable supplierchain selection problem can be separated into two parts, including criteria and methods. paper concentrates on the method supplier be separated intoexamples. two parts,Thus, including criteria and concentrates on the method selectionselection aspects, problem referring can to their application the objectives ofmethods. this paperThis are paper twofold: one is to summarize the selection referring to their application examples. Thus, this paper are twofold: oneapproach is to summarize the literature aspects, on supplier selection issues from related articles, thenthe to objectives present anofattempt to ontology-based to handling literature supplier selectionsupplier issues selection from related articles, present an attempt to ontology-based approach to handling knowledgeonabout sustainable methods. Tothen meettothis aim, ontology-based model is proposed to synthetize the knowledge of about sustainable supplier selection methods. Tothe meet this aim, ontology-based model is proposed tobysynthetize the analyzed methods and related papers. Finally, proposed ontology-based model is demonstrated an empirical knowledge analyzed methods and related papers.itsFinally, the proposed ontology-based model is demonstrated by an empirical example of of using competency questions to confirm correctness and effectiveness. example of using competency questions to confirm its correctness and effectiveness. © 2019 The Author(s). Published by Elsevier B.V. © 2019 The Authors. Published by Elsevier B.V. This © 2019 is an The open Author(s). access article Published under bythe Elsevier CC BY-NC-ND B.V. license (https://creativecommons.org/licenses/by-nc-nd/4.0/) This is an open access article under the CC BY-NC-ND license (https://creativecommons.org/licenses/by-nc-nd/4.0/) Peer-review This is an open under access responsibility article under of the International. BY-NC-ND license (https://creativecommons.org/licenses/by-nc-nd/4.0/) Peer-review under responsibility of KES KESCC International. Peer-review under responsibility of KES International. Keywords: sustainable supplier selection, supplier selection methods, ontology-based model Keywords: sustainable supplier selection, supplier selection methods, ontology-based model

1. Introduction 1. Introduction Supplier selection is combined multi-dimensional problem which includes considering both qualitative and Supplier factors selection is combined multi-dimensional problem whichinfluences includes on considering both qualitative and quantitative [1,2]. Rapid globalization of conducting business business competition, changing quantitative factors [1,2]. Rapid globalization of conducting business influences on business competition, changing

* Corresponding author. Tel.: +48-91-449-5662; fax: +48-91-449-5662. E-mail address:author. [email protected] * Corresponding Tel.: +48-91-449-5662; fax: +48-91-449-5662. E-mail address: [email protected] 1877-0509 © 2019 The Author(s). Published by Elsevier B.V. This is an open access underPublished the CC BY-NC-ND 1877-0509 © 2019 The article Author(s). by Elsevier license B.V. (https://creativecommons.org/licenses/by-nc-nd/4.0/) Peer-review under responsibility of KES International. This is an open access article under the CC BY-NC-ND license (https://creativecommons.org/licenses/by-nc-nd/4.0/) Peer-review under responsibility of KES International. 1877-0509 © 2019 The Authors. Published by Elsevier B.V. This is an open access article under the CC BY-NC-ND license (https://creativecommons.org/licenses/by-nc-nd/4.0/) Peer-review under responsibility of KES International. 10.1016/j.procs.2019.09.333

1630 2

Agnieszka Konys / Procedia Computer Science 159 (2019) 1629–1641 Author name / Procedia Computer Science 00 (2019) 000–000

the model from a ‘company versus company’ into a ‘supply chain versus supply chain’ model [1,3]. Thus, selecting a good set of suppliers to collaborate with is crucial to a company’s success. Over the years, the significance of supplier selection has been long recognized and emphasized. Adding sustainability aspects to the supplier selection process highlights the existing trends in environmental, economic and social issues related to business management and processes. Apart from that, sustainability development allows integrating environmental, economic and social thinking into conventional supplier selection [4]. From a systematic perspective, study on sustainable supplier selection problem can be separated into two parts, including criteria and methods [1,3,5]. Undoubtedly, the sustainable supplier selection problem and suitable criteria selection are very important for operation of all companies [57,61]. The literature provides a set of various methods exploiting different aspects and using single or mixed approaches, as well as the examples of selection criteria [6-8]. The new trends force using new methods or adapting existing ones, and alternatively considering new factors [2]. Currently, this transformation brings new approaches to supplier selection [65] and helps in achieving sustainable development goals through considering at least four issues: moving from an environmental focus to social and economic dimensions, sustainability and innovation [3], ethical supply, and measurement indicators [6]. Meanwhile, a number of scientific papers treating about sustainable supplier selection have increased significantly [5,7]. That means we need to found a new general way to systematically collect and handle the knowledge about existing selection and evaluation methods, by offering a useful supplement tool to the existing research. As response to these issues, author presents an attempt to ontology-based approach to handling knowledge about sustainable supplier selection methods. The main advantage of the ontology-based approach is to share information about existing literature sources and applied methods for supplier selection for enabling its semantics-driven knowledge processing, although the proposed ontology needs to be constantly improved in the future practice and enriched by new knowledge. The objectives of this paper are twofold: one is to summarize the literature on supplier selection issues from selected articles published during 2005 to 2018, particularly on supplier selection methods. Second, based on that, the ontology-based approach synthetizing the analyzed methods and related papers are proposed. The proposed ontology-based model is demonstrated by an empirical example of using competency questions to confirm its correctness and effectiveness. The paper is structured as follows: Section 2 presents sustainable selection problems, whereas Section 3 highlights sustainable selection methods, detailing an analysis based on literature review as well as a classification of them. In Section 4 accessing and leveraging methods’ knowledge about sustainable suppliers’ selection is described. Section 5 provides an ontology-based model for supplier classification methods and its formalization. Moreover, the offered model is verified using defined competency questions. The outcomes of this work and future research are detailed with a conclusion for this review. 2. Sustainable supplier selection problem Suppliers play a key role in supply chain management which involves evaluation for supplier selection problem, as well as other multifaceted issues that organizations should consider [9-11]. Supplier selection problem is one of the most important competitive challenges used by modern enterprises, although one problem is which criteria should be considered during the selection problem, the second one is which method should be used [2,6,12]. Nowadays, companies have to enhance the effectiveness of their sustainable supply chain management activities to survive in the global marketplace. Due to increasing global sustainable awareness, supply chains that consider environmental, economic and social protection tend to be favored by sustainable-minded customers and organizations [4]. Thus, research on sustainable supplier evaluation and selection has attracted more and more attention both from academic and industrial sector [9]. In searching supplier selection methods various techniques are used to evaluate and to select suppliers. Supplier selection process is a complex and multi-dimensional problem [13,14] because there may be conflicts among qualitative, quantitative criteria and also sustainable criteria [2,6,15]. In this regard, numerous organizations have been invested in sustainable supplier development programs to enhance their sustainable performance with respect to the supply chain. Implementing the idea of TBL (Triple Bottom Line) in sustainable supply chain management increases awareness of environmental protection, economic issues and social aspects [5,10] and it allows companies to conduct sustainable supply chain management practices to minimize or eliminate the negative environmental, social and economic effects of their business operations [5,16]. Within



Agnieszka Konys / Procedia Computer Science 159 (2019) 1629–1641 Author name / Procedia Computer Science 00 (2019) 000–000

1631 3

sustainable supply chain management, organizations and collaborators are entailed to measure the sustainable performance of their suppliers and select the most appropriate one in various stages of product life cycle [3,17]. In the competitive environment, sustainable supplier selection is a challenging multi-dimensional issue, requiring seeking appropriate methods for sustainable supplier selection problems and documented examples of their usage [1,11,18-20,64,67]. Moreover, supplier evaluation and selection is critical for sustainable organizations since its performance determines the quality of the procurement, and ultimately affect the organization’s operations performance and market competitiveness [2,19,21,65,66]. The process of selecting appropriate suppliers starts from performing sustainable supplier evaluation, which is crucial to the performance of supply chain operations [8,13,17]. The sustainable supplier selection criteria can be acquired from the comprehensive literature review according to the triple bottom line principle. However, methodological approaches are needed for further evaluation. Undoubtedly, a systematic approach for managing suppliers is required as it can help to build the closeness and long-term relationship between clients and suppliers. The main problem is how to select the proper method and where we can find any information about it. 3. Sustainable supplier selection methods 3.1. Analysis of selected methods based on literature review The classification of the supplier quantitative approaches to supplier selection proposed by Weber in 1991 [4,19,22-25] contained three main categories: mathematical programming models, linear weighting models, and statistical / probabilistic approaches [25-33]. Generally, linear weighting models are based on a situation where a weight is assigned to each criterion (typically subjectively determined) to reach a total score for each supplier by summing up his performance on the criteria multiplied by these weights (e.g. AHP, most frequently used in linear weighting models [5,14,21], however there are other techniques such as Technique for Order Performance by Similarity to Ideal Solution (TOPSIS) [26,37], or PROMETHEE. Mathematical programming models encompass linear programming [21], multi objective programming [19,39], and goal programming [26,31,34]. Mathematical optimization method aim is to select several suppliers in order to maximize an objective function subject to supplier (or buyer) constraints [21]. The objective function can receive single criterion value (classical optimization models) or multiple criteria values (goal programming or multi-objective programming) [26,31,34,39] .Data envelop analysis (DEA) is a mathematical programming method for assessing the comparative efficiencies of decision-making units. In general, DEA is used to rank efficient and inefficient suppliers, whereas AHP to obtain the weights of criteria [35]. Some works exploiting this method were presented by [5,8]. Further, there are mixed approaches including DEA and Multi-Objective Programming [22]. Statistical approaches refer to methods such as cluster analysis and stochastic economic order quantity (EOQ) model [36-38,40,44]. Apart from that, there are some methods employed in supplier selection problem solving, for example cost based method, total cost of ownership [6] simulations [28] and heuristics [25]. To be more detailed, literature reviews point out that the main single and combined approaches used to solve this problem are mathematics methods and artificial intelligence approaches, especially including Analytic Hierarchy Process [26,33,41], Linear Programming [41,51], Multi-Objective Programming [2,16,18,24,41], Total Cost Ownership [3,5,6], Goal Programming [11,16,26,31,34,39,43], Data Envelopment Analysis [5,46,48], Simulation [8,28,34,37], Heuristics [25,29,53], Statistical [44], Cluster Analysis [38,42], Multiple Regression [27,31,55], Discriminant Analysis [35], Conjoint Analysis, Principal Component Analysis [38,49], Neural Networks [22,35,45,48,50], Software Agent [50,55], Case-Based Reasoning [1,30,36], Expert System [47,54], and Fuzzy Set Theory [13-19,21-24] as well as combinations of selected pairs. For example, fuzzy set theory has been integrated with multiple criteria techniques, as AHP or TOPSIS [31-33,40-41,43]. These methods are ranged from classic mathematical approaches and artificial intelligence solutions to more sophisticated semantic-matching-based ones using ontologies [10,42,56,58,63]. These classical approaches are widely adapted to sustainable supplier selection problems. However, to the best of author’s knowledge there is a lack of complex approach which deals with the general problem of supplier selection, which can provide the company the information about several approaches and simultaneously collect the knowledge

Agnieszka Konys / Procedia Computer Science 159 (2019) 1629–1641 Author name / Procedia Computer Science 00 (2019) 000–000

1632 4

about various methods for supplier selection in one place offering a public available solution. In order to overcome this limitation, an ontology-based approach to sustainable supplier selection methods is elaborated. 3.2. Classification of selected methods Based on the revised literature, a general schema of most frequently used methods was particularized. Selective decision models comprehended the linear weighting, total cost of ownership [3,5,6], mathematical programming [2,16,18,24,41], and statistical [44] and artificial intelligence-based models [1,22,35,45,48,50,55] were considered. The existing analytical methods schema for supplier selection presented by Chen [15] was enriched by adding single models in the form of the statistical approaches, and combined models containing TOPSIS methods integrated fuzzy set theory [26,31,37,39] and also semantic matching and ontologies [10,42]. The proposed schema can be developed by adding additional methods or modified existing ones. Based on the proposed schema, the further taxonomy elaboration took place. The final set contained 2 main criteria (single model and combined models) with assigned sub-criteria in the form of class hierarchy. Criterion single model covers mathematics approaches (Analytic Hierarchy Process [26,33,41], Linear Programming [41,51], Multi-Objective Programming [2,16,18,24,41], Total Cost Ownership [3,5,6], Goal Programming [11,16,26,31,34,39,43], Data Envelopment Analysis [5,46,48], Simulation [8,28,34,37], Heuristics [25,29,53], and Statistical [44]), single model (Cluster Analysis [38,42], Multiple Regression [27,31,55], Discriminant Analysis [35], Conjoint Analysis, Principal Component Analysis [38,49]) and artificial intelligence (Neural Networks [22,35,45,48,50], Software Agent [50,55], Case-Based Reasoning [1,30,36], Expert System [47,54], Fuzzy Set Theory [13-19,21-24]). Combined models are a collection of mixed methods such as Analytic Hierarchy Process and Goal Programming [26,43], Analytic Hierarchy Process and Linear Programming [41,51], Analytic Hierarchy Process and Fuzzy Set Theory [14,15,26,33], Data Envelopment Analysis and MultiObjective Programming [8,46,48], Technique for Order Performance by Similarity to Ideal Solution and Goal Programming, Technique for Order Performance by Similarity to Ideal Solution and Fuzzy Set Theory [26,31], and also Semantic Matching and Ontologies [10,42]. Table 1 presents the selected supplier selection methods assigned to relevant categories. Table 1. Supplier selection methods. Supplier selection methods Single Model

Abbreviation Mathematics

Analytic Hierarchy Process

AHP

Linear Programming

LP

Multi-Objective Programming

MOP

Total Cost Ownership

TCO

Goal Programming

GP

Data Envelopment Analysis

DEA

Simulation Heuristics Statistical Single model

Cluster Analysis Multiple Regression Discriminant Analysis Conjoint Analysis Principal Component Analysis

Artificial Intelligence

Neural Networks

NN

Software Agent

SA

Case-Based Reasoning

CBR



Agnieszka Konys / Procedia Computer Science 159 (2019) 1629–1641 Author name / Procedia Computer Science 00 (2019) 000–000

Combined Models

Expert System

ES

Fuzzy Set Theory

FST

Analytic Hierarchy Process and Goal Programming

AHP + GP

Analytic Hierarchy Process and Linear Programming

AHP + LP

Analytic Hierarchy Process and Fuzzy Set Theory

AHP + FST

Data Envelopment Analysis and Multi-Objective Programming

DEA + MOP

Technique for Order Performance by Similarity to Ideal Solution and Goal Programming

TOPSIS + GP

Technique for Order Performance by Similarity to Ideal Solution and Fuzzy Set Theory

TOPSIS + FST

1633 5

Semantic Matching and Ontologies

4. Accessing and leveraging methods’ knowledge about sustainable suppliers’ selection The methods mentioned above (in section 3) present multifariousness of used approaches from methodological point of view. The variety of applied methods beginning from single models by mathematics and artificial intelligence highlights the complexity of sustainable supplier selection in this domain [7,12,17]. In-depth literature studies bring still novel practical applications of used methods. In view of the potential benefits brought about to the current knowledge about used methods through the practice and adoption of knowledge shared by suppliers, some of the knowledge required is not available internally and hence have to be sourced externally [32,40]. A number of research papers have investigated the selection of suppliers and various methods used, however having current access to the applicable knowledge and its integration and encapsulation into one system can improve the overall access between suppliers and collaborators [14,18,22,44]. Thus, by using ontology-based model for supplier classification methods, the simple access to domain knowledge will be improved by providing more complete set of knowledge captured from the methods and their applications [59,62]. Furthermore, the ability to contribute useful knowledge to improve existing applied projects and methods can be incorporated as additional element for developing the proposed ontology-based model. This process comprises capturing knowledge of the sustainable supplier selection methods, validating the knowledge captured, and disseminating knowledge captured for reuse. The developed ontology-based model comprises a semantic database and module for various methods related to sustainable supplier selection. All the users (scientific researchers, suppliers, organizations) can view the set of methods and their practical applications and access to the data and share new knowledge. The access to gathered knowledge is public i full available. However, it is needed to verify and to rate the contributed knowledge by experts, and convey/transfer acquired knowledge to ontology-based model. As knowledge may become obsolete over time [17,32,60], it has to be constantly reviewed, corrected, updated and even removed as appropriate. To build effective ontology-based model for supplier classification methods, the processes related to updates and customization to suit sustainable organization's requirements are key elements for enrichment of the contents for better decision making. 5. Ontology-based model for supplier classification methods The procedure of construction of ontology-based system requires a definite set of actions to perform. At the beginning, based on the elaborated table 1 containing the set of selected methods, the exemplary research papers referred to these methods were reviewed [1-55]. To investigate further the practical implications of presented methods in a given paper, the form and achieved results were analysed. In the aftermath of this, the representative set of distinctive methods for supplier selection was selected. Each of method has assigned detailed information about authorship, title and place of publication, and it is represented by Digital Object Identifier (DOI). In a few cases, where the identification of Digital Object Identifier was impossible, the International Standard Serial Number (ISSN) identifier was used. There are totally 50 representatives from 26 methods presented both in a single and combined forms.

1634 6

Agnieszka Konys / Procedia Computer Science 159 (2019) 1629–1641 Author name / Procedia Computer Science 00 (2019) 000–000

The procedure of ontology-based model construction imposes 7 particular steps to be fulfilled [12,17]. The analysis of identified set of criteria (1) is a basis for a taxonomy construction as a next step (2), followed by ontology implementation (3). In the aftermath of this, the set of descriptive classes is constructed with assigned relations between them as well as necessary constraints. The formalization of constructed ontology-based model was also included (4). To ensure description logic-based reasoning, the set of defined classes was elaborated and implemented (5). This step is treated as a validation stage and consistency checking (6) of elaborated ontologybased model. At the end the final set of results is provided (7). In practice, the feature analysis allowed defining a class hierarchy which is treated as a taxonomic representation of knowledge. Finally, the class hierarchy is represented by main class Supplier selection methods, which contains 2 classes called Single Model and Combined Model. To these main classes, sets of sub-classes are added. And so the class Single Model contains the 3 sub-classes such as: Mathematics, Single model, Artificial Intelligence. Each of them is preceded by prefix SM. These classes are featured by additional sub-classes that they contain inside. Mathematics encapsulates 9 individuals: Analytic Hierarchy Process, Linear Programming, Objective Programming, Total Cost Ownership, Goal Programming, Data Envelopment Analysis, Simulation, Heuristics, and Statistical. Similarly, the Single model has 5 individuals: Cluster Analysis, Multiple Regression, Discriminant Analysis, Conjoint Analysis, and Principal Component Analysis. The sub-class Artificial Intelligence is represented by the following 5 individuals: Neural Networks, Software Agent, Case-Based Reasoning, Expert System, and Fuzzy Set Theory. Going back to the main hierarchy, class Combined Model is distinguished by 7 individuals: Analytic Hierarchy Process and Goal Programming, Analytic Hierarchy Process and Linear Programming, Analytic Hierarchy Process and Fuzzy Set Theory, Data Envelopment Analysis and Multi-Objective Programming, Technique for Order Performance by Similarity to Ideal Solution and Goal Programming, Technique for Order Performance by Similarity to Ideal Solution and Fuzzy Set Theory, and also Semantic Matching and Ontologies. Likewise, they are also preceded by prefix CM. As it was mentioned in the section xxx, the combinations of two methods from Single Model set are allocated in this class. The class Applied methods is covered by the 50 individual references of DOI [1-55], encapsulated in DOI subclass (10.1016/j.eswa.2012.01.149, 10.1108/13598540210436586, 10.1016/j.jclepro.2010.03.020, 10.1016/j.eswa.2012.05.051, 10.1016/j.engappai.2011.10.012, 10.1016/j.eswa.2014.07.057, 10.1016/j.jclepro.2013.02.010, 10.1016/j.eswa.2011.05.008, 10.1080/00207543.2015.1054452, 10.1016/j.eswa.2010.04.022, 10.1016/j.ijpe.2014.11.007, 10.1016/j.ejor.2009.01.026, 10.1108/09576050310503367, 10.1016/j.cie.2009.12.004., 10.1080/00207540600787200, 10.1108/BIJ-11-2015-0110, ISSN 0976–6359, 10.1016/j.ijpe.2006.11.022, 10.1016/j.jocs.2014.11.002, 10.1108/13598540910970144, 10.1080/0951192X.2013.874594, 10.1051/matecconf/20166, 10.1108/13598540710759772, 10.1111/j.19375956.2002.tb00474.x, 10.1016/j.ijpe.2005.03.009., 10.1007/s00521-015-1890-3, 10.1016/j.eswa.2006.12.008., 10.1016/j.eswa.2010.12.055, 10.1080/0951192X.2013.834467, 0.1504/IJISM.2016.074418, 10.1080/0951192052000288161, 10.1016/j.eswa.2008.12.039, 10.1108/01443579610151788, 10.1007/s11135-0099223-1, 10.1016/j.cor.2012.11.005.,.1016/j.autcon.2010.02.007, 10.1016/S0957-4174(03)00042-3, 10.1111/itor.12155, 10.1016/S0925-5273(97)00009-1, 10.1108/17410390610708526, 10.1016/j.eswa.2008.01.063, 10.1016/j.eswa.2012.01.034, 10.1016/j.pursup.2010.06.004, 10.1016/j.ejor.2007.01.019, 10.1016/j.omega.2005.08.004, 10.1080/09537280600940655 10.1108/09600039310038161, 10.1007/s10845-0120640-y, 10.1111/j.1745-493X.2000.tb00078.x, ISSN 2302-4009). To establish relations between classes and object properties, a set of relations was constituted as follows: has Criterion and its inversion: is Criterion of. Each of relations is characterized by domain and range, has Criterion has defined Applied methods as a domain and Supplier selection methods as a range, whereas is Criterion of as defined Supplier selection methods as a domain and Applied methods as a range. These object properties should be disjoint. The ontology was built using the Protégé application. The applied technology standard is OWL (Ontology Web Language). Overall, 2 main criteria and 29 sub-criteria were implemented with reference to the 50 applied methods dedicated to sustainable supplier selection, represented in the form of digital object. The final class hierarchy is depicted on figure 1 using OntoGraf and on figure 2 using OWLViz.



Agnieszka Konys / Procedia Computer Science 159 (2019) 1629–1641 Author name / Procedia Computer Science 00 (2019) 000–000

Fig. 1. A class hierarchy of implemented criteria and sub-criteria presented using OntoGraf tool.

Fig. 2. A class hierarchy of implemented criteria and sub-criteria presented using OWLViz tool.

1635 7

Agnieszka Konys / Procedia Computer Science 159 (2019) 1629–1641 Author name / Procedia Computer Science 00 (2019) 000–000

1636 8

5.1. Formalization The formalization of the proposed ontology-based model can be performed twofold: using Description Logic (DL) and using the set theory to ensure the detailed mathematical depiction [23,32]. For the purpose of this paper, the set theory is applied to present the mathematical background of constructed ontology-based model. In the domain and range of a relation, if R is a relation from set Ss and A, then the set of all taxons (all of the first components of the ordered pairs) belonging to R is called the domain of R. Thus, Dom is defined as follows: (𝑅𝑅) = {𝑠𝑠𝑠𝑠 ∈ 𝑆𝑆𝑠𝑠: (𝑠𝑠𝑠𝑠, 𝑎𝑎) ∈ 𝑅𝑅 𝑓𝑓𝑓𝑓𝑓𝑓 𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠 𝑎𝑎 ∈ 𝐴𝐴}

(1)

𝑅𝑅 = {𝑝𝑝 ∈ 𝑃𝑃: (𝑠𝑠𝑠𝑠, 𝑝𝑝) ∈ 𝑅𝑅 𝑓𝑓𝑓𝑓𝑓𝑓 𝑠𝑠𝑠𝑠𝑠𝑠𝑒𝑒 𝑠𝑠𝑠𝑠 ∈ 𝑇𝑇}

(2)

𝑆𝑆𝑆𝑆 𝑥𝑥 𝐴𝐴 = {(𝑠𝑠𝑠𝑠, 𝑎𝑎): 𝑠𝑠𝑠𝑠 ∈ 𝑆𝑆𝑠𝑠, 𝑎𝑎 ∈ 𝐴𝐴}

(3)

The set of all second components of the ordered pairs (the set of all taxons) belonging to R is called the range of R. Thus, the range of R is defined as follows:

If supplier selection methods Ss and applied methods A are two non-empty sets, then the Cartesian product T of Ss and A, denoted Ss x A, is the set of all ordered pairs (ss, a) such that ss ∈ Ss and a ∈ A:

Applied methods A contain the finite set of taxons, defined as follows: 𝐴𝐴 = {𝑑𝑑𝑑𝑑𝑑𝑑1, 𝑑𝑑𝑑𝑑𝑑𝑑15, 𝑑𝑑𝑑𝑑𝑑𝑑16, 𝑑𝑑𝑑𝑑𝑑𝑑29, 𝑑𝑑𝑑𝑑𝑑𝑑30, 𝑑𝑑𝑑𝑑𝑑𝑑42, 𝑑𝑑𝑑𝑑𝑑𝑑43,

𝑑𝑑𝑑𝑑𝑑𝑑2, 𝑑𝑑𝑑𝑑𝑑𝑑3, 𝑑𝑑𝑑𝑑𝑑𝑑4, 𝑑𝑑𝑑𝑑𝑑𝑑5, 𝑑𝑑𝑑𝑑𝑑𝑑6, 𝑑𝑑𝑑𝑑𝑑𝑑7, 𝑑𝑑𝑑𝑑𝑑𝑑17, 𝑑𝑑𝑑𝑑𝑑𝑑18, 𝑑𝑑𝑑𝑑𝑑𝑑19, 𝑑𝑑𝑑𝑑𝑑𝑑20, 𝑑𝑑𝑑𝑑𝑑𝑑21, 𝑑𝑑𝑑𝑑𝑑𝑑31, 𝑑𝑑𝑑𝑑𝑑𝑑32, 𝑑𝑑𝑑𝑑𝑑𝑑33, 𝑑𝑑𝑑𝑑𝑑𝑑34, 𝑑𝑑𝑑𝑑𝑑𝑑35, 𝑑𝑑𝑑𝑑𝑑𝑑44, 𝑑𝑑𝑑𝑑𝑑𝑑45, 𝑑𝑑𝑑𝑑𝑑𝑑46, 𝑑𝑑𝑑𝑑𝑑𝑑47, 𝑑𝑑𝑑𝑑𝑑𝑑48,

𝑑𝑑𝑑𝑑𝑑𝑑8, 𝑑𝑑𝑑𝑑𝑑𝑑9, 𝑑𝑑𝑑𝑑𝑑𝑑10, 𝑑𝑑𝑑𝑑𝑑𝑑11, 𝑑𝑑𝑑𝑑𝑑𝑑12, 𝑑𝑑𝑑𝑑𝑑𝑑13, 𝑑𝑑𝑑𝑑𝑑𝑑14, 𝑑𝑑𝑑𝑑𝑑𝑑22, 𝑑𝑑𝑑𝑑𝑑𝑑23, 𝑑𝑑𝑑𝑑𝑑𝑑24, 𝑑𝑑𝑑𝑑𝑑𝑑25, 𝑑𝑑𝑑𝑑𝑑𝑑26, 𝑑𝑑𝑑𝑑𝑑𝑑27, 𝑑𝑑𝑑𝑑𝑑𝑑28, 𝑑𝑑𝑑𝑑𝑑𝑑36, 𝑑𝑑𝑑𝑑𝑑𝑑37, 𝑑𝑑𝑑𝑑𝑑𝑑38, 𝑑𝑑𝑑𝑑𝑑𝑑39, 𝑑𝑑𝑑𝑑𝑑𝑑40, 𝑑𝑑𝑑𝑑𝑑𝑑41, 𝑑𝑑𝑑𝑑𝑑𝑑42, 𝑑𝑑𝑑𝑑𝑑𝑑49, 𝑑𝑑𝑑𝑑𝑑𝑑50} (4)

where the taxons numbered from doi1 to doi50 represent subsequently the authors of the papers. Supplier selection methods Ss contains the finite set of taxons, which are subsets: 𝑆𝑆𝑆𝑆 = {𝑆𝑆𝑆𝑆, 𝐶𝐶𝐶𝐶}

(5)

𝑆𝑆𝑆𝑆 = {𝑀𝑀, 𝑆𝑆𝑆𝑆𝑆𝑆, 𝐴𝐴𝐴𝐴}

(6)

𝑀𝑀 = {𝐴𝐴ℎ𝑝𝑝, 𝐿𝐿𝐿𝐿, 𝑀𝑀𝑀𝑀𝑀𝑀, 𝑇𝑇𝑇𝑇𝑇𝑇, 𝐺𝐺𝐺𝐺, 𝐷𝐷𝐷𝐷𝐷𝐷, 𝑆𝑆𝑖𝑖, 𝐻𝐻, 𝑆𝑆𝑆𝑆}

(7)

𝑆𝑆𝑆𝑆𝑆𝑆 = {𝐶𝐶𝐶𝐶, 𝑀𝑀𝑀𝑀, 𝐷𝐷𝐷𝐷, 𝐶𝐶𝐶𝐶𝐶𝐶, 𝑃𝑃𝑃𝑃𝑃𝑃}

(8)

where Sm refers to single model and Cm presents combined models. Single model Sm contains the finite set of 3 taxons as follows:

where M refers to mathematics, Smd presents single model, and Ai depicts artificial intelligence. Mathematics M contains the finite set of taxons, represented as follows:

where Ahp refers to Analytic Hierarchy Process, Lp means Linear Programming, Mop presents Multi-Objective Programming, Tco depicts Total Cost Ownership, Gp refers to Goal Programming, Dea shows Data Envelopment Analysis, Si means Simulation, H depicts Heuristics, and St presents Statistical. Single models Smd contain the finite set of taxons, represented as follows:

where Ca presents Cluster Analysis, Mr shows Multiple Regression, Da depicts Discriminant Analysis, Cja refers to Conjoint Analysis, and Pca means Principal Component Analysis.



Agnieszka Konys / Procedia Computer Science 159 (2019) 1629–1641 Author name / Procedia Computer Science 00 (2019) 000–000

1637 9

Artificial intelligence Ai contains the finite set of taxons, represented as follows: (9)

𝐴𝐴𝐴𝐴 = {𝑁𝑁𝑁𝑁, 𝑆𝑆𝑎𝑎, 𝐶𝐶𝐶𝐶𝐶𝐶, 𝐸𝐸𝐸𝐸, 𝐹𝐹𝐹𝐹𝐹𝐹}

where Nn depicts Neural Networks, Sa shows Software Agent, Cbr means Case-Based Reasoning, Es presents Expert System, and Fst refers to Fuzzy Set Theory. Combined models Cm represent the finite set of taxons, defined as follows: 𝐶𝐶𝐶𝐶 = {𝐴𝐴ℎ𝑝𝑝𝑝𝑝𝑝𝑝, 𝐴𝐴ℎ𝑝𝑝𝑝𝑝𝑝𝑝, 𝐴𝐴ℎ𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝, 𝐷𝐷𝐷𝐷𝐷𝐷𝐷𝐷𝐷𝐷𝐷𝐷, 𝑇𝑇𝑇𝑇𝑇𝑇𝑇𝑇𝑇𝑇𝑇𝑇𝑇𝑇𝑇𝑇, 𝑇𝑇𝑇𝑇𝑇𝑇𝑇𝑇𝑇𝑇𝑇𝑇𝑇𝑇𝑇𝑇𝑇𝑇, 𝑆𝑆𝑆𝑆𝑆𝑆}

(10)

where AhpGp refers to Analytic Hierarchy Process and Goal Programming, AhpLp means Analytic Hierarchy Process and Linear Programming, AhpFst depicts Analytic Hierarchy Process and Fuzzy Set Theory, DeaMop shows Data Envelopment Analysis and Multi-Objective Programming, TopsisGp refers to Technique for Order Performance by Similarity to Ideal Solution and Goal Programming, TopsisFst presents Technique for Order Performance by Similarity to Ideal Solution and Fuzzy Set Theory, and SmO means Semantic Matching and Ontologies. 5.2. Competency questions To validate the proposed ontology-based model for supplier classification methods, the set of competency questions was defined. Firstly, to ensure description logic-based reasoning, the set of defined classes was elaborated and implemented, according to the procedure presented in section 5. For this reason, this step addresses a validation stage and consistency checking of elaborated ontology-based model. The sample competence question demonstrates the working procedure of implemented ontology-based model. The correctness of this process informs of welldefined ontology, providing at the end the final set of results. The exemplary competence question contains the followed constraints to be fulfilled by applied methods: exploiting single models, especially mathematics such as Analytic Hierarchy Process or Goal Programming, or using combined models of both of them. The typed description of the criteria was constituted and formulated in the form of defined class. It is worth to mention that this definition should be fulfilled partially to belong to the final ranking. The competency question was implemented manually to Description Logic query mechanism. Visualizing the final ranking of the solutions is presented on figure 3 and figure 4. From 50 applied methods, 17 fulfil this definition. The detailed information assigned to DOI is also available.

Fig. 3. Results presented using OntoGraf tool.

10 1638

Author name /Konys Procedia Computer Science Science 00 (2019) Agnieszka / Procedia Computer 159000–000 (2019) 1629–1641

Fig. 4. Results presented using OWLViz tool.

Additional experimental queries have been posed to find the relevant applied methods only by considering single models from artificial intelligence such as Neural Networks and Software Agents. Similarly to the first competence question, the definition was formulated and implemented using Description Logic query mechanism. After reasoning process the final set of results was provided as presented on figure 5.

Fig. 5. Results presented using OntoGraf tool



Agnieszka Konys / Procedia Computer Science 159 (2019) 1629–1641 Author name / Procedia Computer Science 00 (2019) 000–000

1639 11

6. Conclusions The supplier selection problem has gained more attention emphasizing the role of the efficient discovery and selection of capable suppliers. Through the phenomenon of sustainable development, the sustainable practices and increased awareness of environmental, economic and social issues have direct influence on building a long-term supply chain collaboration. Selecting the optimum sustainable supplier is crucial for sustainable supply chain management, which is a challenging multi-dimensional problem. To overcome these issues, this paper presented an attempt to ontology-based approach to handling knowledge about sustainable supplier selection methods. The ontology-based model was built on basis of previously elaborated literature review, identifying the set of groups of various methods supporting supplier selection. To each of selected group, a number of examples of research papers were investigated. Based on that, the ontology-based approach synthetizing the analysed methods and related papers were proposed. The main perquisite of the ontology-based approach is to share information about existing literature sources and applied methods for supplier selection for enabling its semantics-driven knowledge processing, what has been successfully employed. The practical application of the proposed model was checked using working competency questions, where the correctness of this process informed of well-defined ontology. However, this model has some limitations. In the future, it is needed to constantly improve and enrich by new knowledge the proposed ontology-based model. References [1] Özder, Emir, Hüseyin, and Eren, Tamer. (2015) “Supplier selection with topsis and goal programming methods: a case study.” Journal of Trends in the Development of Machinery and Associated Technology 19 (1): 109-112. [2] Weber, Charles A., and Ellram, Lisa M. (1993) “Supplier Selection Using Multi‐objective Programming: A Decision Support System Approach". International Journal of Physical Distribution & Logistics Management 23 (2): 3-14. [3] Bhutta, Khurrum S., and Huq, Faizul. (2002) "Supplier selection problem: a comparison of the total cost of ownership and analytic hierarchy process approaches". Supply Chain Management: An International Journal 7 (3): 126-135. [4] Peeyush, Pandey, Bhavin J., Shah, Hasmukh, Gajjar. (2017) "A fuzzy goal programming approach for selecting sustainable suppliers". Benchmarking: An International Journal 24 (5): 1138-1165. [5] Ramakrishnan, Ramanathan. (2007) "Supplier selection problem: integrating DEA with the approaches of total cost of ownership and AHP". Supply Chain Management: An International Journal 12 (4): 258-261. [6] Reza, Mohammady Garfamy. (2006) "A data envelopment analysis approach based on total cost of ownership for supplier selection". Journal of Enterprise Information Management 19 (6): 662-678. [7] Ordoobadi, Sharon M. (2009) "Development of a supplier selection model using fuzzy logic". Supply Chain Management: An International Journal 14 (4): 314-327. [8] Azadeh, A., Alem, S.M. (2010) “A flexible deterministic, stochastic and fuzzy Data Envelopment Analysis approach for supply chain risk and vendor selection problem: Simulation analysis”. Expert Systems with Applications 37 (12): 7438-7448. [9] Wątróbski, Jarosław, Jankowski, Jarosław, Ziemba, Paweł, Karczmarczyk, Artur, and Zioło, Magdalena. (2019). Generalised framework for multi-criteria method selection. Omega 86: 107-124. [10] Farhad, Ameri, and McArthur, Christian. (2014) "Semantic rule modelling for intelligent supplier discovery." International Journal of Computer Integrated Manufacturing 27 (6): 570-590. [11] Singh, Amol. (2016) “A goal programming approach for supplier evaluation and demand allocation among suppliers”. Int. J. Integrated Supply Management 10 (1). [12] Wątróbski, Jarosław. (2016) “Outline of Multicriteria Decision-making in Green Logistics.” Transportation Research Procedia 16: 537-552. [13] Chan, Felix TS, and Niraj, Kumar. (2007) "Global supplier development considering risk factors using fuzzy extended AHP-based approach." Omega 35 (4): 417-431. [14] Chan, Felix TS, et al. (2008) "Global supplier selection: a fuzzy-AHP approach." International Journal of Production Research 46 (14): 3825-3857. [15] Chen-Tung, Chen, Ching-Torng, Lin, Sue-Fn, Huang. (2006) “A fuzzy approach for supplier evaluation and selection in supply chain management.” International Journal of Production Economics 102 (2): 289-301. [16] Chin-Nung, Liao, Hsing-Pei, Kao. (2010) “Supplier selection model using Taguchi loss function, analytical hierarchy process and multichoice goal programming.” Computers & Industrial Engineering 58 (4): 571-577. [17] Konys, Agnieszka (2018). “Towards Knowledge Handling in Ontology-Based Information Extraction Systems.” Procedia Computer Science 126: 2208-2218. [18] Wu, Desheng Dash, Zhang, Yidong, Wu, Dexiang, Olson, David L.(2010) “Fuzzy multi-objective programming for supplier selection and risk modeling: A possibility approach.” European Journal of Operational Research 200 (3): 774-787.

1640 12

Agnieszka Konys / Procedia Computer Science 159 (2019) 1629–1641 Author name / Procedia Computer Science 00 (2019) 000–000

[19] Kannan, Devika et al. (2013) “Integrated fuzzy multi criteria decision making method and multi-objective programming approach for supplier selection and order allocation in a green supply chain.” Journal of Cleaner Production 47: 355-367. [20] Dotoli, Mariagrazia, et al. (2016) "A stochastic cross‐efficiency data envelopment analysis approach for supplier selection under uncertainty." International Transactions in Operational Research 23 (4): 725-748. [21] Farhad, Faez, Ghodsypour, S. H., and O’Brien, Christopher. (2009) "Vendor selection and order allocation using an integrated fuzzy casebased reasoning and mathematical programming model." International Journal of Production Economics, Vol. 121, No. 2 pp. 395-408. [22] Fallahpour, Alireza, et al. (2016) "An integrated model for green supplier selection under fuzzy environment: application of data envelopment analysis and genetic programming approach." Neural Computing and Applications 27 (3): 707-725. [23] Konys, Agnieszka. (2018) “An Ontology-Based Knowledge Modelling for a Sustainability Assessment Domain.” Sustainability 10 (300). [24] Feyzan, Arikan. (2013) “A fuzzy solution approach for multi objective supplier selection.” Expert Systems with Applications 40 (3): 947-952. [25] Burke, Gerard J., Carrillo, Janice, Vakharia, Asoo J. (2008) “Heuristics for sourcing from multiple suppliers with alternative quantity discounts.” European Journal of Operational Research 186 (1): 317-329. [26] Ha Thi, Xuan, Chia, Do Hoang, Ngoc, Trinh. (2016) “Supplier selection by Using AHP-TOPSIS and Goal Programming - A case Study in Casumina Rubber Company – Vietnam.” MATEC Web of Conferences 6 (2). [27] Ho, Li-Hsing, et al. (2012) "Using modified IPA to evaluate supplier’s performance: Multiple regression analysis and DEMATEL approach." Expert Systems with Applications 39 (8): 7102-7109. [28] Ding, Hongwei, Benyoucef, Lyès, and Xie, Xiaolan. (2005) “A simulation optimization methodology for supplier selection problem.” International Journal of Computer Integrated Manufacturing 18 (2-3): 210-224. [29] Badri, Hossein, Bashiri, Mahdi, Hejazi, Taha Hossein. (2013) “Integrated strategic and tactical planning in a supply chain network design with a heuristic solution method.” Computers & Operations Research 40 (4): 1143-1154. [30] Humphreys, Paul, McIvor, Ronan, and Chan, Felix. (2003) "Using case-based reasoning to evaluate supplier environmental management performance." Expert systems with Applications 25 (2): 141-153. [31] Jolai, Fariborz, et al. (2011)"Integrating fuzzy TOPSIS and multi-period goal programming for purchasing multiple products from multiple suppliers." Journal of purchasing and Supply Management 17 (1): 42-53. [32] Jankowski, Jarosław, Hamari Juho, and Wątróbski, Jarosław. (2019) “A gradual approach for maximising user conversion without compromising experience with high visual intensity website elements.” Internet Research 29 (1): 194-217. [33] Kahraman, Cengiz, Ufuk Cebeci, and Ziya Ulukan. (2003)" Multi-criteria supplier selection using fuzzy AHP." Logistics information management 16 (6): 382-394. [34] Kamran S., Moghaddam. (2015) “Supplier selection and order allocation in closed-loop supply chain systems using hybrid Monte Carlo simulation and goal programming.” International Journal of Production Research 53 (20): 6320-6338. [35] Kar, Arpan Kumar. (2015) "A hybrid group decision support system for supplier selection using analytic hierarchy process, fuzzy set theory and neural network." Journal of Computational Science 6: 23-33. [36] Zhao, Kai, and Xin Yu. (2011) "A case based reasoning approach on supplier selection in petroleum enterprises." Expert Systems with Applications 38 (6): 6839-6847. [37] Akram, Zouggari, and Benyoucef, Lyes. (2012) "Simulation based fuzzy TOPSIS approach for group multi-criteria supplier selection problem." Engineering Applications of Artificial Intelligence 25 (3): 507-519. [38] Petroni, Alberto, and Marcello, Braglia. (2000) "Vendor selection using principal component analysis." Journal of supply chain management 36 (1): 63-69. [39] Sharma, Sanjay, and Srinivasan, Balan. (2013) "An integrative supplier selection model using Taguchi loss function, TOPSIS and multi criteria goal programming." Journal of Intelligent Manufacturing 4 (6): 1123-1130. [40] Karczmarczyk, Artur, Jankowski, Jarosław, and Wątróbski, Jarosław. (2018). „Multi-criteria decision support for planning and evaluation of performance of viral marketing campaigns in social networks.” PloS one 13 (12): e0209372 [41] Shaw, Krishnendu, Shankar, Ravi, Yadav, Surendra S., Thakur, Lakshman S. (2012) “Supplier selection using fuzzy AHP and fuzzy multiobjective linear programming for developing low carbon supply chain.” Expert Systems with Applications 39 (9): 8182-8192. [42] Lee, Jaehun, et al. (2015) "Semantic web-based supplier discovery system for building a long-term supply chain." International Journal of Computer Integrated Manufacturing 28 (2): 155-169. [43] Ku, CY., Chang, CT. and Ho, HP. (2010) “Global supplier selection using fuzzy analytic hierarchy process and fuzzy goal programming.” Qual Quant 44 (623). [44] Sarkis, Joseph, and Dileep G. Dhavale. (2015) "Supplier selection for sustainable operations: A triple-bottom-line approach using a Bayesian framework." International Journal of Production Economics 166: 177-191. [45] Kuo, Renjieh J., Yu C. Wang, and Fangchih C. Tien. (2010) "Integration of artificial neural network and MADA methods for green supplier selection." Journal of Cleaner Production 18 (12): 1161-1170. [46] Toloo, Mehdi, and Soroosh Nalchigar. (2011) "A new DEA method for supplier selection in presence of both cardinal and ordinal data." Expert Systems with Applications 38 (12): 14726-14731. [47] Yigin, I. H., et al. (2007) "Supplier selection: an expert system approach." Production Planning and Control 18 (1): 16-24. [48] Wu, Desheng. (2009) "Supplier selection: A hybrid model using DEA, decision tree and neural network." Expert Systems with Applications 36 (5): 9105-9112.



Agnieszka Konys / Procedia Computer Science 159 (2019) 1629–1641 Author name / Procedia Computer Science 00 (2019) 000–000

1641 13

[49] Ka-Chi, Lam, Tao, Ran, and Lam, Mike, Chun-Kit. (2010) "A material supplier selection model for property developers using fuzzy principal component analysis." Automation in Construction 19 (5): 608-618. [50] Chun Ching, Lee, and Ou-Yang, C. (2009) "A neural networks approach for forecasting the supplier’s bid prices in supplier selection negotiation process." Expert Systems with Applications 36 (2): 2961-2970. [51] Ghodsypour, S.H., O'Brien, C. (1998) “A decision support system for supplier selection using an integrated analytic hierarchy process and linear programming.” International Journal of Production Economics 56–57: 199-212. [52] Sung, Ho Ha, Ramayya, Krishnan. (2008) “A hybrid approach to supplier selection for the maintenance of a competitive supply chain.” Expert Systems with Applications 34 (2): 1303-1311. [53] Tempelmeier, H. (2002) “A Simple Heuristic for Dynamic Order Sizing And Supplier Selection With Time‐Varying Data.” Production and Operations Management 11: 499-515. [54] Vokurka, Robert J., Joobin Choobineh, and Lakshmi Vadi. (1996) "A prototype expert system for the evaluation and selection of potential suppliers." International Journal of Operations & Production Management 16 (12): 106-127. [55] Chunxia, Yu, and Wong, T. N. (2015) "An agent-based negotiation model for supplier selection of multiple products with synergy effect." Expert Systems with Applications 42 (1): 223-237. [56] Sałabun, Wojciech, and Piegat, Andrzej. (2017). “Comparative analysis of MCDM methods for the assessment of mortality in patients with acute coronary syndrome." Artificial Intelligence Review, 48(4): 557-571. [57] Wątróbski, Jarosław, Jankowski, Jarosław, and Piotrowski, Zbigniew. (2014) „The selection of multicriteria method based on unstructured decision problem description.” In: Hwang D., Jung J.J., Nguyen NT. (eds) Computational Collective Intelligence. Technolo gies and Applications. Lecture Notes in Computer Science, Springer, Cham, 8733: 454-465. [58] Nermend, Kesra, and Piwowarski, Mateusz. (2018) “Neuroscience Techniques in Supporting Decision Making and the Analysis of Social Campaign.” In: Proceedings book International Conference on Accounting, Business, Economics and Politics (ICABEP-2018), Erbil, Iraq, 112. [59] Konys, Agnieszka. (2017) “Ontology-Based Approaches to Big Data Analytics.” In: Kobayashi S., Piegat A., Pejaś J., El Fray I., Kacprzyk J. (eds) Hard and Soft Computing for Artificial Intelligence, Multimedia and Security. Advances in Intelligent Systems and Computing, Springer, Cham, 534: 355-365. [60] Sałabun, Wojciech. (2014). “Reduction in the number of comparisons required to create matrix of expert judgment in the comet method.” Management and Production Engineering Review, 5(3): 62-69. [61] Piwowarski, Mateusz. et al. (2018) “TOPSIS and VIKOR methods in study of sustainable development in the EU countries, Procedia Computer Science, 126: 1683-1692. [62] Konys, Agnieszka. (2018). “Konys Agnieszka. (2018) “Knowledge systematization for ontology learning methods.” Procedia Computer Science, 126: 2194-2207. [63] Sałabun, Wojciech. (2015). “The Characteristic Objects Method: A New Distance‐based Approach to Multicriteria Decision‐making Problems.” Journal of Multi‐Criteria Decision Analysis, 22(1-2): 37-50.

[64] Wątróbski, Jarosław, Ziemba, Ewa, Karczmarczyk, Artur, and Jankowski, Jarosław. (2018). „An index to measure the sustainable information society: the Polish households case.” Sustainability, 10(9): 3223. [65] Piwowarski, Mateusz, et al. (2018) „Application of the Vector Measure Construction Method and Technique for Order Preference by Similarity Ideal Solution for the Analysis of the Dynamics of Changes in the Poverty Levels in the European Union Countries.” Sustainability, 10: 2858.

[66] Jankowski, Jarosław, Kolomvatsos, Kostas, Kazienko, Przemysław, and Watróbski, Jarosław. (2016) „Fuzzy Modeling of User Behaviors and Virtual Goods Purchases in Social Networking Platforms.” J. UCS, 22(3): 416-437. [67] Wątróbski, Jarosław, Ziemba, Paweł, and Wolski, Waldemar. (2015) Methodological aspects of decision support system for the location of renewable energy sources. In 2015 Federated Conference on Computer Science and Information Systems (FedCSIS), IEEE, 1451-1459.