BI4BI: A continuous evaluation system for Business Intelligence systems

Expert Systems With Applications 76 (2017) 97–112

Contents lists available at ScienceDirect

Expert Systems With Applications journal homepage: www.elsevier.com/locate/eswa

BI4BI: A continuous evaluation system for Business Intelligence systems Manel Brichni a,b,c,∗, Sophie Dupuy-Chessa a, Lilia Gzara b, Nadine Mandran a, Corinne Jeannet c a b c

Univ. Grenoble Alpes, LIG, F-38000 Grenoble, France CNRS, LIG, F-38000 Grenoble, France Univ. Grenoble Alpes, G-SCOP, F-38000 Grenoble, France CNRS, G-SCOP, F-38000 Grenoble, France STMicroelectronics, 850 rue Jean Monnet, Grenoble, France

a r t i c l e

i n f o

Article history: Received 3 July 2016 Revised 2 January 2017 Accepted 22 January 2017 Available online 29 January 2017 Keywords: Business Intelligence Design Science Evaluation Indicators Evolution

a b s t r a c t Business Intelligence (BI) allows supporting decision making by providing methods and tools to easily access and manage business information. Indeed, it is crucial to maintain BI systems and make them evolve in order to fit business and users’ needs. For this purpose, applying BI for BI is suggested. It helps to analyze the current BI environment and to make decisions about its use and evolution. Then, the main contribution of this paper consists in developing a novel system called BI4BI which is based on both BI systems’ data and BI users’ feedbacks. As a result, based on a user centered approach, an automated system for BI evaluation is developed helping its continuous evolution. To this end, system-based and user-based solutions provide two complementary data sources, required to effectively gather insights on the use of BI systems and the way they should evolve. A case study in a semiconductor manufacturer company demonstrates the impact of our solution on the evaluation and evolution of BI systems. Hence, thanks to the involvement of the company’s BI users during the solution design, a set of evaluation criteria responding to their objectives for evaluating the BI system are identified and validated. Furthermore, the results of this case study show that our proposed solution is better than existing ones, in terms of level of focus of its BI evaluation criteria, its level of automation, and its continuous processing. © 2017 Elsevier Ltd. All rights reserved.

1. Introduction Business Intelligence (BI) aims to support business users to make decisions by providing methods and tools to easily access and manage their information (Horkoff et al., 2012; McBride, 2014; Moro, Cortez, & Rita, 2015). This is performed by means of data collection, storage, distribution and exploitation (Jooste, Van Biljon, & Mentz, 2014). The evolution of business requirements and the availability of many BI tools motivate many companies to systematically measure their performance over years and to align BI tools with its

∗ Corresponding author at: Laboratoire LIG Btiment IMAG 700 avenue Centrale, Domaine Universitaire de Saint-Martin-d’Hres CS 40700 - 38058 Grenoble cedex 9 France. E-mail addresses: [email protected], [email protected], [email protected] (M. Brichni), [email protected] (S. Dupuy-Chessa), [email protected] (L. Gzara), [email protected] (N. Mandran), [email protected] (C. Jeannet).

http://dx.doi.org/10.1016/j.eswa.2017.01.018 0957-4174/© 2017 Elsevier Ltd. All rights reserved.

long-term strategic objectives (Barone, Topaloglou, & Mylopoulos, 2012). Actually, BI acts on many business processes, uses different resources, involves different users’ profiles and continuously generates information to make business decisions. However, in many organizations, the rapid growth of business needs promotes a fast changing BI activity (Cook & Nagy, 2014; Foshay & Kuziemsky, 2014; McBride, 2014; Moro et al., 2015). Therefore, it becomes crucial to evaluate and make improvement actions on the BI system in order to ensure its continuous evolution (Isik, Jones, & Sidorova, 2013; Lahrmann, Marx, Winter, & Wortmann, 2011) to fit business needs. That is why, we think that it should be continuously evaluated while taking into account its use, and its evolution over time. Based on these statements, our main research question is: How to continuously evaluate a BI system in order to align it with business needs? The evaluation of BI applications is a topic widely discussed in literature (Dell’Aquila, Di Tria, Lefons, & Tangorra, 2008; Isik et al., 2013; Jooste et al., 2014; Lahrmann ´ et al., 2011; Polanska & Zyznarski, 2009; Rouhani, Ghazanfari, & Jafari, 2012) for example, in magazine distribution

98

M. Brichni et al. / Expert Systems With Applications 76 (2017) 97–112

(McBride, 2014), in banking (Moro et al., 2015) or in medical (Cook & Nagy, 2014; Foshay & Kuziemsky, 2014). However, most studies limited this topic to a statical evaluation problem where most of them suggested evaluation criteria with almost no particularities of BI systems. Such solutions do not allow to continuously analyze the BI system, to make decisions about its use and to ensure its evolution. In order to overcome these limits, in this paper, the development of a BI system is presented. It is able to evaluate and to help analyzing and making decisions about the existing BI applications. This contribution is called BI for BI (BI4BI). Our BI4BI system includes two complementary solutions. First, a system-based solution includes the development of a BI system to evaluate the existing BI application based on data such as BI objects’ uses. Second, a user-based solution includes evaluating BI users’ points of view on the BI applications. Our proposal is applied to the semiconductor manufacturer company, STMicroelectronics, currently using BI applications for its production activities. The remainder of the paper is organized as follows. We present, firstly, our research methodology (Section 2), followed by a literature review (Section 3) and the context of STMicroelectronics (Section 4). Next, our proposal for the development of a BI for BI system is presented in Section 5. To validate our proposal, its codesign is detailed in Section 6 and its main implications, advantages and limitations are discussed in Section 7. We end this paper with a conclusion and some perspectives (Section 8).

Consequently, DSR would be the most appropriate methodology to respond our research challenges. DSR seeks to create innovations in terms of ideas, practices, technical capabilities, and products. The purpose of the Design Science Research approach is at the intersection of people, organizations, and technology when designing an artifact that should impact and be impacted by people and their needs (Hevner & March, 2004). All of these characteristics lead us to adopt such a methodology. DSR is based on three cycles: Relevance, Design and Rigor: •

•

2. Research methodology Our main research question is “How to continuously evaluate a BI system in order to align it with business needs?”. To answer this question, we aim to develop an evaluation solution to help analyzing and making decisions about the BI activity and its evolution, particularly in the STMicroelectronics’s context. In our research, both gaining theoretical insights and improving practical processes within an organization through the use of Information Technology (IT) are our challenges. Being the same challenges of Information System (IS) research (Jrad, Ahmed, & Sundaram, 2014), we need an appropriate IS research methodology. Over the last few decades, two IS research disciplines, namely Action Research (AR) and Design Science Research (DSR), have ˚ raised interests among IS scholars (Agerfalk & Goldkuhl, 2013; Hevner & March, 2004). In AR, the focus of interest is the organizational context and the active search for solutions in an actual and immediate problematic situation (Peffers, Tuunanen, Rothenberger, & Chatterjee, 2007). As a result, one important limitation is that external researchers are not significantly impacted with AR findings and knowledge, so they are not expected to directly benefit from solving the participating organization’s problem (Jrad et al., 2014). To overcome this limit, AR have been evolved to Insider Action Research (IAR) suggesting that researchers can gain even more knowledge if they are also employed by the researched organization. However, switching from employee to researcher and back is a challenging mission and would generate conflicts and lack of objectivity in conducting the research. This gap between “research” and “action” would be overcome with DSR, since it combines at the same time needs and knowledge from academia (rigor) and business world (relevance). The main strengths of DSR compared to other research methodologies consist of its rigorous process in motivating, developing, designing, demonstrating, evaluating, and communicating the artifact. All these steps are required and are complementary to each other for conducting an IS research. They were illustrated in Hevner and March (2004) with three exemplar articles for analysis from three different IS journals. That demonstrates the applicability and contributions of such a methodology in the IS field.

•

Rigor: the Design Science Research relies upon the application of rigorous methods in both the construction and evaluation of the design artifact. To proceed, a literature review allows us to identify and define BI evaluation criteria and their integration in the system. In addition, DSR is assessed as it contributes to the content of the knowledge base with the creative development of novel, appropriately evaluated constructs, models, methods, or instantiations to extend and improve the existing foundations. To this end, communicating the processes by which the artifact was constructed and evaluated is crucial for their understanding. This could be achieved for example through publishing academic researches or discussing the solution’s advantages, limitations and significance. Relevance: the definition of business needs allows us to study the business environment, to identify business problems, opportunities and goals. Therefore, at STMicroelectronics, we explored with users (BI experts, end users and business experts) their expectations. As a result, they contribute to the design of our solution, for example, to define the most appropriate indicators responding to their need for evaluating the BI system. In addition, business needs must be applied in an appropriate environment and findings of the research must be useful for this environment. To this end, DSR requires that designed artifacts must respond to people’s needs, be integrated into the organization culture and strategies and effectively implemented within the existing technologies and systems. Design: designing the artifact is the core of the Design Science Research. The result of this stage is an artifact responding to business needs and using or inspiring from findings in the literature. Here, our contribution is to propose a BI for BI system.

In remainder of this paper, the presentation of our work is structured according to the DSR cycles. Then, firstly, a literature review is presented (part of rigor cycle), followed by a description of the context of work (part of relevance cycle) and finally the proposal development (part of design cycle). We end this paper by presenting the co-construction with users at STMicroelectronics, as part of the relevance cycle as well as a discussion of its main implications, advantages and limitations as part of the rigor cycle. 3. Literature review ´ Many studies (Isik et al., 2013; Polanska & Zyznarski, 2009; Popovicˇ , Hackney, Simões Coelho, & Jaklicˇ , 2012) aim to identify success factors for BI evaluation. In this section, existing BI evaluation criteria are discussed. To proceed, several analysis criteria are considered to compare existing approaches in the literature. First, regarding the BI evaluation criteria, the sources of their identification is important. They can be reused from literature and/or standards and/or actors, etc. The way BI evaluation criteria are identified can determine their significance, accuracy and focus. Second, the level of focus of the BI evaluation criteria on BI systems is considered. This means how they evaluate BI from different points of view. BI evaluation criteria can consider generally the system like any tool or consider as well its content while focusing on its particularities (tools, sources, outputs, users, etc.). More the

M. Brichni et al. / Expert Systems With Applications 76 (2017) 97–112

evaluation criteria are focused more the considered system is better evaluated. Third, the integration of the identified criteria in a global approach is important. It helps their reuse and their evolution in different contexts. Reusability helps to adapt the identified criteria to evaluate other systems than BI. It also allows to make them evolve according to the context while reducing their identification time. Fourth, it is crucial to discuss the ability to continuously evaluate BI systems in order to ensure its evolution. Fifth, to be easier to use and to adopt, an evaluation solution should be rather automated and easily integrated in the considered environment. Therefore, the level of automation is our last criteria. In the following, we investigate the literature on the BI evaluation criteria and the way they are used to evaluate BI systems. 3.1. Standards The ISO 250 0 0 family is one of the most known models dedicated for the development of software products. It specifies quality requirements and evaluation quality characteristics. It provides a quality model allowing to decide which quality characteristics will be taken into account when evaluating the properties of a software product (ISO, 2014). The product quality model defined in ISO 250 0 0 comprises eight quality characteristics and their subcharacteristics. The idea to base on formal specifications described in international standards is very interesting. It allows to integrate criteria in a global approach for evaluation systems. This is the reason why many researchers in the literature reuse standards for BI ´ systems’ evaluation (Dell’Aquila et al., 2008; Polanska & Zyznarski, 2009). ´ In his work, Polanska and Zyznarski (2009) aim to elaborate a method for comparison of existing BI systems that are supporting data mining. The method consists of a quality model, built according to the ISO 250 0 0 characteristics and sub characteristics, and a set of steps which should be taken to choose a BI system. While ´ integrated in a global approach for BI evaluation, Polanska and Zyznarski (2009) proposed therefore criteria according to the ISO 250 0 0: the functional suitability, learnability, ease of use, operability, security, maintainability, portability, etc. In order to compare selected BI systems based on the identified BI evaluation criteria, ´ some tests have been conducted. To this end, Polanska and Zyznarski (2009) selected the same set of data that should be analyzed with all of the selected BI systems. Their results should be evaluated and compared according to the selected ISO 250 0 0 criteria and sub criteria. An evaluator in this case study performed evaluation in order to compare existing BI tools for purchasing one of those products. This person has little experience with data mining, but has developed few data mining models. Such a method lacks of automation and prevents therefore a continuous evaluation, since it requires regular human involvement. In the cases of using standards for BI evaluation, we note that authors’ efforts were focused on evaluating the BI system like any other tool. This means that we did not note particularities for BI systems. For example, they studied its organizational and technical environment as well as its quality and quality in use. Actually, the aim of BI is to offer users solutions to effectively make business decisions. Therefore, analyzing the system business content itself should be considered in the identified evaluation criteria. For example, analyzing the use of business indicators, dimensions, reports, etc. Despite its importance, this point of view is not discussed in literature. Moreover, even though literature on using ISO 250 0 0 for BI ´ evaluation is very limited (Polanska & Zyznarski, 2009), many other researchers have based on it to assess other kinds of software in different contexts. Their case studies can be helpful for evaluating its effectiveness. For example, Borim, Mukai, Lucas Emanuel, Lopes, and Cabral Moro (2015) aim to identify the main aspects

99

used to evaluate health information systems. One of the main methods to proceed has been the ISO 250 0 0. However, in such a context, alone, it does not fulfill the evaluation objectives. Hence, authors in Borim et al. (2015) suggest to complement it with sociotechnical and human factors, such as, efficacy and effectiveness, showing whether the technology will bring the requested benefits or not. This point of view regarding the readjustment of the ` and Tyrychtr ISO model is also shared with Ulman, Vostrovsky, (2013). Authors propose CABAG (Communication between Agricultural Businesses and Government) method enabling to evaluate quality of agricultural electronic services by means of ISO 250 0 0 model. Since in the agriculture domain, the level of notion and understanding of users of electronic services is lower than in several other industries, the list of evaluation criteria derived from ISO norm are adapted for agricultural users of e-services. The results of this readjustment need to be tested in practice by using real users’ data in order to be validated. What we learn from these case studies is that, despite its effectiveness, due to its general evaluation criteria, the ISO 250 0 0 model needs to be completed with other kinds of evaluation methods to meet specific business needs. To this end, more investigation on the considered business domain needs to be conducted. To summarize, using ISO 250 0 0 for BI evaluation presents advantages and limitations. The most important advantage is the ability to integrate it in a global approach allowing its reuse. However, its lack of focus and automation preventing the continuous evaluation are its main limitations.

3.2. Fuzzy TOPSIS technique Fuzzy TOPSIS (Technique for Order Preference by Similarity to Ideal Situation) is a method that can be applied to multiple criteria decision-making to classify preference by similarity. Applying this method to BI aims to help the decision-makers to select the enterprise system which has suitable intelligence to support managers’ decisional tasks. This is the case of Rouhani et al. (2012) who proposed a model to assess enterprise systems in BI aspects. Literature review was done on BI specifications and criteria that a system should have to cover BI definitions. 34 qualitative and quantitative criteria were identified, such as reliability, accuracy of analysis, alarms and warning, environmental awareness, stakeholders’ satisfaction, etc. These criteria are presented with a fuzzy evaluation model that is applied to evaluate and rank candidate BI systems. Such a method is therefore integrated in a global approach based on a reusable model. This model is applied to evaluate and rank candidate enterprise systems like Enterprise Recourse Planning (ERP), Supply Chain Management (SCM), Customer Relationship Management (CRM), and Accounting and Office Automation system. However, like the ISO 250 0 0 criteria, those used in Rouhani et al. (2012) remain general. They do not provide particularities for BI systems and they can be rather used to evaluate any system. To proceed with the evaluation approach, Rouhani et al. (2012) suggested a model that exploits Fuzzy TOPSIS techniques for the evaluation of BI systems. On this basis, organizations will be able to select, assess and purchase BI systems which make possible better decision support environment in their work systems. However, the limitation identified at this stage is the fact to base on a technique and a model to conduct the evaluation process. This limits the level of automation of the solution. Without providing an evaluation system, the solution requires advanced expertise in the domain of Fuzzy TOPSIS techniques to be effectively reused. It prevents also the continuous BI evaluation since the evaluation criteria are not integrated within a system preventing their continuous reuse.

100

M. Brichni et al. / Expert Systems With Applications 76 (2017) 97–112

Table 1 Overview of existing BI evaluation criteria and solutions. BI evaluation case studies

ISO 250 0 0 (Borim et al., 2015; ´ Dell’Aquila et al., 2008; Polanska & Zyznarski, 2009; Ulman et al., 2013)

Fuzzy TOPSIS (Lahrmann et al., 2011; Rouhani et al., 2012)

Sources of the evaluation criteria identification The level of focus of the evaluation criteria The integration of the identified criteria in a global approach The level of automation of the proposed solution

ISO 250 0 0 characteristics and sub characteristics general criteria

literature review: 34 qualitative and quantitative criteria general criteria

´ yes: a BI quality model (Polanska & Zyznarski, 2009) lack of automation: regular human ´ evaluation (Polanska & Zyznarski, 2009) no: it requires regular human involvement

yes: a fuzzy evaluation model (Rouhani et al., 2012) lack of automation: only a model is build based on Fuzzy TOPSIS Technique (Rouhani et al., 2012) no: the evaluation criteria are not integrated within a system

A continuous evaluation to ensure the BI evolution

3.3. Backpropagation (BP) neural network A BP neural network is a multi-hierarchic feedback structure, which aims at adjusting the network weights through backpropagation algorithm, including input layer, hidden layer and output layer (Yan, Wang, & Liu, 2012). The objective is, therefore, to propose an optimal BP neural score evaluating systems. An example of application of BP neural to BI was given by Yan et al. (2012). He established a comprehensive evaluation index system according to the construction principles of BI systems. At this stage, developing an evaluation system reduces the complexity of the evaluation procedure for non technical users and allows automating the process. To proceed, in Yan et al. (2012), a set of criteria, sub-criteria and indicators were identified, to meet the construction principles based on a literature review. 18 evaluation criteria were identified (qualitative and quantitative). They are classified into four groups: system construction, users’ satisfaction, internal influence and external influence. In that way, they are integrated in a global approach and can be easily evolved, particularly when the proposed evaluation index system is used to evaluate as long as the evaluation criteria are given. This allows therefore to continuously evaluate BI systems. However, some limitations are identified, such as large subjective, arbitrary and low accuracy to determine evaluation criteria. As a result, they do not provide particularities for BI systems and they can be used to evaluate any system. More generally, although the algorithm of BP neural network is successful, in practice, it has some inadequates. As mentioned by the authors of Jin, Li, Wei, and Zhen (20 0 0) in their investigation on BP neural network limitations, because of adopting its gradient method, the problems including a slow learning convergent velocity and easily converging to local minimum cannot be avoided. In addition, the selection of the learning factor and inertial factor affects the convergence of the BP neural network, which are usually determined by experience. Therefore, the effective application of the BP neural network is limited. These limitations are also highlighted by Hu and Zhao (2010), with multiple iterations and the poorness of its numerical stability. To summarize, in addition to the limitations related to our business needs for BI, some procedural problems have been identified with BP neural network method in the literature. Those ones are as important as they concern the stability, the results and the slowness of its execution. Consequently, such problems could affect the effectiveness of its continuous evaluation capability. 3.4. Discussion Literature suggested different solutions for BI systems’ evaluation. Most of them use processes or techniques

Backpropagation (BP) Neural Network (Hu & Zhao, 2010; Jin et al., 20 0 0; Raber, Wortmann, & Winter, 2013; Yan et al., 2012) literature review: 18 qualitative and quantitative criteria general criteria yes: a comprehensive evaluation index system (Yan et al., 2012) an evaluation index system based on Backpropagation (BP) Neural Network (Yan et al., 2012) no: procedural problems could affect the effectiveness of its continuous evaluation capability

´ (Polanska & Zyznarski, 2009; Rouhani et al., 2012; Yan et al., 2012). Each solution is based on a different technique. Previously, three categories of solutions were detailed: fuzzy TOPSIS technique (Rouhani et al., 2012), BP Neural Network technique (Yan et al., ´ 2012) and ISO 250 0 0 based solution (Polanska & Zyznarski, 2009). Table 1 details findings and limitations of some proposed BI evaluation criteria and solutions. They are organized according to their compliance with our comparison criteria. To summarize, in the literature review, two main limitations have been identified: first, about the identified evaluation crite´ ria and second about the proposed evaluation system (Polanska & Zyznarski, 2009; Rouhani et al., 2012; Yan et al., 2012). On the one hand, we consider that the focus of the considered BI evaluation criteria should not be limited to the evaluation of BI as a tool, as proposed in literature, but it should be also extended to the evaluation of its content. On the other hand, the presented solutions are limited to a statical contribution ´ (Polanska & Zyznarski, 2009; Rouhani et al., 2012; Yan et al., 2012). More specifically, proposals in literature are able to evaluate BI systems only to statically measure their qualities without considering their evolution. Actually, what BI systems need is a solution that continuously evaluates and helps to analyze its use while allowing making decisions about its evolution. In addition, many solutions do not provide an evaluation system but base rather on techniques ´ (Polanska & Zyznarski, 2009; Rouhani et al., 2012). This prevents the automation of the evaluation process and makes it more difficult its integration within organizations. In order to respond to these objectives, before presenting our proposal, the context of work at STMicroelectronics is detailed.

4. Context of work 4.1. STMicroelectronics and its reporting team STMicroelectronics is a global leader in semiconductors. It is the largest European company in its field. It existed since 1987. Our research is focused only at Crolles300 site in France, particularly the Manufacturing Solutions Group, having almost 75 employees and around 60 subcontractors. The main missions of this group are to support, maintain and adapt over time Crolles300 information systems to the level required by a world class 300mm_Fabrication and R&D plant (Research and Development). Our proposal is applied to the “Reporting team” of the Business Intelligence activity. The Reporting activity consists in describing the organization activities through reports concerning one or more areas for a given period. For example, by providing an overview of the time spent by one activity, a report allows users to know which part of the manufacturing process is difficult to manage. In the following, we will

M. Brichni et al. / Expert Systems With Applications 76 (2017) 97–112

101

Fig. 1. Business Intelligence architecture at STMicroelectronics.

describe the BI system employed by the Reporting team at STMicroelectronics. 4.2. Introduction to Business Intelligence at STMicroelectronics 4.2.1. STMicroelectronics’ BI solution To achieve end-to-end visibility into business critical functions, STMicroelectronics needs to access volumes of data and information every day to make better decisions, for example, about the daily manufacturing activity. To this end, as presented in Fig. 1, STMicroelectronics uses a traditional BI architecture including several tools: •

•

•

•

To access to data sources, for example, the MES (Manufacturing execution Systems) database, in charge of storing production data. Each source is different from the other in terms of technology, storing process and data uses. To Extract, Transform and Load data, with an ETL (Panos, 2009) process, into a dimensional databases called a data warehouse. At STMicroelectronics, the data warehouse is called TGV (Tool for Global Visibility). To use decision making tools, for example, analysis, mining or reporting tools. At STMicroelectronics, Business Objects (BO)1 (Hilgefort, 2011) is used for querying, reporting and monitoring data about the production activity.

At STMicroelectronics, BI is mainly used for reporting the manufacturing activities, using appropriate tools. It allows having a periodically view on the activity evolution and efficiency, equipment maintenance and productivity. One of the most used report at STMicroelectronics concerns products’ uses, requiring to continuously supervise successful produced wafers and lots as well as rejected ones. In addition, the BI system at STMicroelectronics is completed by management tools, as follows: Safir Reporting Portal is the platform for reports sharing. It allows Reporting engineers at STMicroelectronics to publish created reports and to make them available for users. It integrates some functionalities, such as managing favourite reports, sharing them with colleagues or accessing to their documentation in Stiki, presented in the following. Stiki is a STMicroelectronics’ wiki, designed and implemented in 2009 to cover the support, technical, business and project documentation, mainly within the IT department. Currently, in the Reporting team, it is used as the main tool for knowledge sharing in the BI system as shown in our previous work (Brichni, Mandran, Gzara, Dupuy-Chessa, & Rozier, 2014).

1

www.sap.com.

Fig. 2. BI actors at STMicroelectronics.

Blog Crolles300 is the notification platform used by the Reporting team to notify users by email for each new creation or update of a report. The email redirects them to a brief description of the report and its links to Safir and Stiki. To summarize, we consider that a BI system should promote the right exploitation of obtained results. Generally, results of BI are presented in the form of reports of different natures. To promote the right use and made decisions, such reports should be stored, shared and documented in order to be effectively reused. This is the reason why the BI system at STMicroelectronics is composed of not only BI technical tools but also of knowledge management tools, where each one has its own functions and objectives, while interacting between each other, as shown in Fig. 1. 4.2.2. BI actors At STMicroelectronics, three different profiles, with different needs and skills, are concerned by the BI system. As shown in figure 2, profiles are: •

•

•

An end user: he/she is the client asking for the creation of a report for his/her business needs and he/she is its final user. A business expert: he/she provides the work methodology when a BI need is identified. He/she ensures consistency, alignment and relevance of users’ needs. He/she also ensures good communication and use of the reports. A BI expert: he/she creates reports and maintains the BI system. Thanks to a collaborative work with both end users and business experts, he/she aligns users’ needs with defined specifications in order to provide the most appropriate solution.

As depicted in Fig. 2, to create a report, for example about the last month production activity, an end user transmits his/her needs and the main objective of the required report to a business expert (1). This one treats the request and transmits it to a BI expert (2). For example, he/she eliminates some machines, selects the involved products and determine the calculation formula. According to these requirements, the BI expert selects the most appropriate production data and creates the report. Then he/she shares it in Safir portal, documents it in Stiki and notifies interested users via the Blog Crolles300 (3). Finally, the end user and the business expert retrieve it (4). Having different users’ profiles increases difficulties in responding to each profile’s needs. For example, the BI expert who is daily

102

M. Brichni et al. / Expert Systems With Applications 76 (2017) 97–112

Fig. 3. BI for BI system.

faced to use technical tools, needs to know how an existing report was created (techniques behind, queries used, tables addressed, etc.). An end user needs to search for existing reports responding to his/her objective. The business expert needs explanations about the selected indicators and their calculation formulas. These needs are different but also changing according to users’ needs evolution. Consequently, these facts require ensuring a continuous evolution of the BI system. To summarize, at STMicroelectronics, a BI solution allows monitoring the manufacturing activity. It includes both technical and management tools where each one has its own functions and objectives. They help therefore to meet users’ needs according to their profiles and competencies. However, the BI system should as well be aligned with the evolution of business needs. Consequently, a solution for its continuous evolution is required. After discussing the way literature addresses the problem of BI evaluation in addition to our context at STMicroelectronics, in the next section, our proposal is detailed. 5. Our proposal: Business Intelligence for Business Intelligence 5.1. Overview of our proposal: BI4BI As we discussed previously, our objective is to develop a solution for continuously evaluating to help analyzing and making decisions about the BI activity and its evolution. Since this is itself the principle of BI, we aim to develop a BI system for evaluating the BI system itself. We call it a BI4BI system. Its main goal is to allow to: •

•

• • •

access to the BI system content and to collect the required BI data for analysis analyze users’ answers to a questionnaire evaluating the BI system to provide indicators and measures according to dimensions analyze the considered BI systems and therefore their uses use decision tools to help making decisions on BI systems evolution

To respond to this goal, a BI4BI system is developed (Fig. 3). It consists in using BI elements and techniques, including the development of indicators, dimensions and a data warehouse accessible for decision making tools. Following a basic approach to design and develop a BI system, our BI4BI system was created as follows: • • •

• • •

defining the evaluation criteria for assessing the BI system identifying data sources defining indicators and dimensions, based on the identified evaluation criteria defining an architecture of the BI for BI system modeling and developing the dimensional data warehouse using the BI4BI tool for reporting

As depicted in Fig. 3, in organizations, a BI system allows monitoring a business activity in order to create knowledge and to help making business decisions. To this end, it is based on business data of the considered activity. This is the case at STMicroelectronics, where a BI system uses production data to measure, to evaluate and to help to analyze the production activity. At STMicroelectronics, the BI system is composed of BI tools as well as communication tools (Stiki, Safir and the Blog, presented in Section 4.2.1). As part of our proposal, a BI4BI system is developed. It will be, in turn, used to measure, to evaluate and to help to analyze the current BI system’s behaviour in order to make decisions about its evolution. It will be able to evaluate the use of the different BI tools at STMicroelectronics (BO, Stiki, Safir and the blog). As a result, the BI4BI system allows to maintain the BI activity and involved systems according to the evolution of business needs. Consequently, its monitoring will ensure making the right business decisions. The technical architecture of the BI4BI system is presented in the following. 5.2. Technical architecture As depicted in Fig. 4, we choose a basic BI architecture based on the relational OLAP paradigm to develop the BI4BI system. Its components are presented in the following.

M. Brichni et al. / Expert Systems With Applications 76 (2017) 97–112

103

Fig. 4. BI4BI technical architecture.

5.2.1. Data sources The technical architecture of our BI4BI system is composed of two complementary solutions: system-based and user-based. First, a system-based solution uses indicators based on produced data about the BI activity. Second, a user-based solution provides measures based on users’ opinions. Both sources provide relevant measures for calculating indicators depending on dimensions. The system architecture includes the data sources. In our research example, our industrial context corresponds to Safir Reporting portal, as part of the system-based solution, and the results of a questionnaire on the BI system as part of the user-based solution. Sources could be as well BO, Stiki and the Blog databases as a part of the system-based solution (Fig. 4). Such a solution helps to analyze and make decisions on the evolution of BI based on both data and users’ opinions. For instance, during migration projects, it will help to make decisions about the BI objects that should be migrated, deleted, merged, etc. 5.2.2. ETL -Extract, Transform and LoadBy ETL processes, data from the databases of the four tools (Safir, BO, Stiki and the Blog) can be extracted, transformed and loaded into the data warehouse, where data are stored in a suitable format for their decisional analysis. In the current version of the BI4BI prototype, only the Safir database is exploited. APF and Oracle Data Integrater (ODI) constitute our ETL solution that are known to be relevant systems according to STMicroelectronics’s experience. Briefly, APF is a software system allowing to retrieve production data in real-time, insert it into a repository and exploit it throughout the creation of reports on this repository. Its objective is to discharge the source database (usually an operational production database) from analysis and querying, while offering a high level of responsiveness thanks to its real-time functions. In addition to APF, ODI is a data integration platform. It covers moving extracted data in real time with the possibility to make advanced transformations that could not be done before by APF. We note for example, filtering, conversions, inserts, etc. In the currently developed BI4BI system, in addition to the Safir database, the extraction, transformation and loading are also applied to the questionnaire results initially stored in a text file. 5.2.3. Data warehouse The target data warehouse is based on Oracle database as our object-relational database management system. Typically, OLAP data are stored in star schema or snow flake schema. In our case, we use the snow flake schema to represent data in the data warehouse.

5.2.4. Data exploitation Once data are loaded into the data warehouse, it is ready to be analyzed. In order to effectively exploit these data, different BI tools can be used for reporting, analysis or data mining. BusinessObjects (BO) is our solution for reporting the BI activity, since it is used and known by the organization’s users for its effectiveness. Before presenting both system-based and user-based solutions, we detail, in the following, evaluation criteria and measures identification. These findings are crucial for the development of our BI4BI solution. 5.3. Evaluation criteria and measures identification Since we are interested in evaluating the BI system, we aim, in a first stage, to identify systems’ evaluation criteria. Therefore, we based on a literature review to identify evaluation criteria, as a part of the rigor cycle of the design science methodology. As we ´ discussed in the state of the art (Section 3), Polanska and Zyznarski (20 09) based on ISO 250 0 0 family to evaluate his BI system from three points of view: quality in use, external quality and internal quality. To base on formal specifications described in international standards is very interesting. ISO 250 0 0 is one of the most recognized models having the goal of creating a framework for the evaluation of systems quality. This justifies our choice of the ISO 250 0 0 model to select the evaluation criteria. To construct STMicroelectronics’ BI4BI system, among identified ISO characteristics, we first focus on the functional suitability and its sub characteristics in order to identify their associated indicators. Actually, despite the importance of the rest of characteristics, we consider that the functional suitability is the characteristic that distinguishes the most the systems from each other. It considers the degree to which the system provides specific functions that meet implied needs. The functional suitability is composed of the following sub characteristics: •

•

•

Functional completeness: degree to which the set of functions covers all the specified tasks and user objectives Functional correctness: degree to which a product or system provides the correct results with the needed degree of precision Functional appropriateness: degree to which the functions facilitate the accomplishment of specified tasks and objectives.

To proceed, we use the definition of characteristics and sub characteristics provided by ISO in order to define measures. It is a property on which aggregations (e.g., sum, count, average, etc.) can be made to calculate indicators. Co-designing with users helped to enrich identifying measures from their points of view (Section 6),

104

M. Brichni et al. / Expert Systems With Applications 76 (2017) 97–112

for example, the number of BI objects, the number of reports’ users and the objects organization. As a result, evaluation criteria and measures are defined for each BI tool (BO, Stiki, Safir and the blog) for the functional suitability, in our case. As depicted in Appendix A (at the end of this paper), two types of measures were identified: objective and subjective. This is the reason why, two complementary solutions are proposed where the system-based solution allows integrating the system-based measures and the user-based solution allows integrating the subjective ones. Both allow evaluating the BI system from two complementary points of view: systems and users. 5.4. System-based-solution In the system-based solution, we are interested in developing the objective measures required to evaluate the BI system. Those can be defined by accessing the appropriate data stored in systems’ databases. To this end, measures involve the use of the BI tools. In the ST context, they concern: •

•

•

•

•

Involved BI users: for example, “the number of subscribed users” on the Blog to evaluate “its notification support” indicator Used BI objects: for example, “the number of duplicated or similar objects” in BO to measure “the objects coverage” indicator Used BI resources: for example, “the number of distributed licences” of BO to measure “the resources uses” indicator Content BI uses: for example, “the number of contributions per Stiki page” to measure “the content uses” indicator Organization in the BI system: for example, “the ease of access to a report” measure to evaluate “the access to information” indicator

To summarize, the particularity of our measures is that they are oriented BI content, for example, used BI objects (indicators, dimensions, reports, etc.). In addition, they correspond to the different BI tools’ databases (BO, Stiki, Safir and the blog). This allows to evaluate the whole BI system, with its involved tools and their content, which makes the difference with existing solutions for BI evaluation. In addition to system-based solution, the subjective measures can be integrated to our BI4BI system thanks to BI users’ points of view. In the following, we present our user-based solution and how both proposals would ensure the correctness of the evaluation. 5.5. User-based solution: questionnaire The user-based solution involves the integration of subjective measures. They represent BI users’ points of view and help to enrich our BI4BI evaluation solution. To proceed, they are integrated in a questionnaire and are presented in this section. They are about, content relevance, content correctness, content manipulation, ease of access and use, adequacy to business needs and tools functionalities (detailed in Section 5.3). One part of the proposed questionnaire is presented in Appendix B. It contains 40 questions about the BI system and its different tools. Questions are almost containing the same number and types of answers (Very easy/satisfied, Fairly easy/satisfied, Neither easy/satisfied nor not easy/satisfied, Not very easy/satisfied, Very difficult/Not at all satisfied). Opting for the same type of answers has been made for two reasons. First, it makes it easy to answer by users. Second, it allows providing the same structure of questions and answers which facilitates their integration into the BI4BI data warehouse. This step is detailed in the next section. To summarize, being complementary to each other, the systembased and user-based solutions consider together two different types of sources including objective and subjective data. These two

types of data should not be separated but should be mutually complementary entities, to take advantage of their insights and to help making decisions. This complementarity would ensure the relevance of the evaluation criteria and thus the correctness of the created report. As a result, since the BI4BI is itself a BI system, the created reports help users making decisions, in our case, on the evolution of the BI system. 5.6. BI data warehouse modeling and reporting A data warehouse modeling considers two key concepts: indicator and dimension. Indicators are composed of measures previously identified based on both sources systems and users. A dimension is an element constituting the context of an indicator and required in the data warehousing modeling. In the following, indicators and then dimensions’ identification are detailed. 5.6.1. Indicators identification In BI, an indicator corresponds to the aggregation of measures. We note that we are searching for indicators to evaluate the BI activity. To this end, with BI users, we identified the most important ones according to their points of view (detailed in Section 6). The particularity of our indicators is that they are applied to the BI activity and its objects, for example, to the business indicators or dimensions. They may even reuse some existing ones, for example, a business indicator monitoring the production activity may be reused to monitor the reporting activity (Brichni, Dupuy-Chessa, Gzara, Mandran, & Jeannet, 2015). For example, to evaluate the functional completeness subcharacteristics for BO, we suggest to measure its objects coverage as well as its activity evolution indicators. To this end, a set of measures is proposed for each one. To measure BI objects coverage indicator, with users, we think that it is important to measure the number of BI objects uses, the number of covered domains, the number of duplicated BI objects, etc. To measure BI objects relevance indicator, users suggest considering their adequacy to their needs as well as the number of made changes and the number of objects providing the same result (Appendix A). 5.6.2. Dimensions identification In BI, dimensions are grouped into meaningful sets for users and decision makers. In BI, dimensions represent business concepts and we often talk about a hierarchy of dimensions that could be geographical (cleanroom, town, etc.), temporal (year, month, day, etc.) or of products (Brichni et al., 2015). For example, at STMicroelectronics, in the BI system we have as product dimensions: equipment, operation, lot, step, etc. In our proposal, we do not talk about business dimensions but about dimensions to apply to identified indicators analyzing the BI activity (described in the previous section). To this end, with users, for each measure about the BI system, we select and define the appropriate dimensions (detailed in Section 6). As a result, we identified four hierarchies of dimensions. •

•

Date dimension hierarchy: one or more time dimensions are often required. In BI, since we use objects to create reports and make decisions, the time dimensions could vary between the year to the day level. User dimension hierarchy: this hierarchy is composed of three levels: Domain dimension: BI system could be used for different domains within the organization. This is why, the domain is one dimension for some identified indicators. For example, at STMicroelectronics, domains could be: IT, finance, communication, etc. Team dimension: in a domain, the BI system could be used in different teams within the organization. This is why, the team

M. Brichni et al. / Expert Systems With Applications 76 (2017) 97–112

•

•

is one dimension for some identified indicators. Each team belongs to one domain. This is why team is a sub dimension of domain. For example, in IT at STMicroelectronics, teams could be: local architecture, manufacturing execution systems, process control and automation, etc. User dimension: different users use the BI system in the organization. This is why several indicators are measured according to users. For example, a report can be used by only one user per team. System dimension: a system dimension could be any of STMicroelectronics’ BI tools (BO, Safir Reporting Portal, Stiki or the Blog). System content dimension: for each system a content dimension is identified. For example, for BO, we identified its objects including Indicators, dimensions, universes and reports. The content of Safir concerns published reports or their categories.

Above, indicators and dimensions required for the BI4BI system are presented. Those ones can be obviously evolved and enriched with new ones if needed. Next, we will detail how they are integrated and used for the data warehouse modeling that is considered as the core of BI systems. 5.6.3. Data warehouse modeling To design the data warehouse, we should base on the identified indicators, measures and dimensions previously presented. In order to demonstrate how the BI can be applied to the BI system activity, in this section, we suggest an example of a BI data warehouse modeling. For simplicity reasons, we focus on the evaluation of Safir as a tool of the BI system. The following objectives are identified: • • • •

To To To To

track the reports’ uses in Safir track the shared reports uses and users in Safir track the documented reports uses and users gather users’ opinions on the shared reports in Safir

As described in these objectives, we need to combine systembased with user-based results. Therefore, both objective and subjective data are considered. To proceed, indicators should be based on both types of data. As shown in the formula below, we can define indicators measuring reports’ uses in Safir, in addition to indicators measuring users’ opinions on the shared reports in Safir. In that way, one new potential indicator can combine both results of the last ones, for example, on the usefulness of shared reports in Safir. For reasons of confidentiality, these results are given for information purposes only. Usefulness of Safir’s reports= Average (Reports’ uses, Users’ opinions) = Average (0.65 + 0.90) = 0.77 Indeed, this new indicator can be measured according to one or more dimensions among those identified with users (users, date, categories, etc.). As a result, the combination of system-based with user-based data and the set of dimensions can be effectively integrated in the BI4BI system thanks to its data warehouse modeling detailed in the following. To explain this process, we present the dimensional data warehouse modeling using the Snow Flake schema. In this example, we made the choice of using a Snow Flake schema because it is more detailed and explicit than the star schema. In the snow flake schema, presented in Fig. 5, two types of tables are identified, fact tables (BI_REPORTACCDATE and BI_QUESTRESPONSES) connected to dimension tables (ST_CALENDAR, ODI_BIREPORT, BI_HIERARCHY, BI_QUESTIONMODEL). Fact tables hold the measures to be analyzed. These measures can directly represent an indicator or be

105

aggregated to identify new indicators. For instance, to measure Safir’s reports obsolescence indicator, we base on the number of unused reports and the number of unused categories measures. Dimension tables store data about the ways in which the measures in the fact table can be analyzed. First, we model the fact table “BI_REPORTACCDATE” used to track Safir reports’ uses with a set of measures “NumberOfUsers”, “NumberOfUses”, “ReportLikes”. At this level, “BI_REPORTACCDATE” table’s content corresponds to the system-based solution where measures are extracted from the Safir database. The second fact table “BI_QUESTRESPONSES” aims to evaluate the level of information, structuring and relevance of the shared reports in Safir. Measures in the “BI_QUESTRESPONSES” table are based on the questionnaire results, as a part of the system-based solution. For example, as a result of the development of the “BI_REPORTACCDATE” table, we are able to report: •

Safir reports obsolescence indicator based on “the number of used/unused/liked/etc.” measures per “year” dimension. With the same manner, as a result of the development of the “BI_QUESTRESPONSES” table, we are able to report:

•

The relevance of Safir reports indicator based on “the level of information and structuring” measures.

Such findings will help engineers in making decisions about the usefulness of these BI objects. 5.6.4. Reporting Once the BI4BI data warehouse is developed, the Reporting tool (Business Objects in the STMicroelectronics’ case) is used. An example is shown in Appendix C that presents the level of information in Safir reports according to the results of the questionnaire. The “Level Of Information” measure is based on the answers to the question asked to users in the questionnaire: “considering the last 5 used Safir reports, are they informative?”. Finally, based on the results obtained from the B4BI system applied to Safir (questionnaire results), we obtain the following findings: Safir reports relevance indicator = Average (Level Of Information, Level Of Structuring) = Average (0.88 + 0.84) = 0.86 This result can be eventually completed with an indicator on “Safir Reports Uses”. A result combining both Safir content relevance (Safir database results) and Safir content uses (questionnaire results) will provide an overview that helps to assess and make decisions on the evolution of Safir content. Thanks to the involvement of its users, our BI4BI system is able to respond to their objectives for evaluating the current BI system. In the following, we detail how users were involved throughout our design process to respond to their needs. 6. Co-design and application of the BI4BI system As a part of the relevance cycle of the design science methodology, BI users are involved in the co-design of the BI4BI system thanks to a user-centered approach. This step allows gathering their opinions to respond to their needs. In fact, among the identified objectives, described in Section 5.1, users are mainly involved in the identification of the evaluation criteria, indicators, measures and dimensions, required for the development of the BI4BI system. Two main steps are performed: qualitative and quantitative (Brichni et al., 2014). First, face to face qualitative interviews aim at identifying in depth a maximum of BI indicators, measures and dimensions. Second, in order to generalize findings of the first step to a large population, a quantitative on-line questionnaire is suggested. These co-designed steps are presented in this following.

106

M. Brichni et al. / Expert Systems With Applications 76 (2017) 97–112

Fig. 5. Snow flake schema. Table 2 Experiments description. Methods

Participants profiles

Number of participants

Strengths

Face to face qualitative interviews, one hour per interviewee

BI experts, business users and end users

6 users with 2 of each profile

Quantitative on-line questionnaire to measure the level of agreement

BI experts, business users and end users

20 BI experts, business users and end users

To gather in depth and detailed 27 indicators, 43 measures and information and feedbacks 8 dimensions have been about BI indicators, identified measures and dimensions To quantify and assess the More than half of users have relevance of the proposed agreed with 40 measures indicators, measures and dimensions

Findings

6.1. Face to face qualitative interviews

6.2. On-line questionnaire

As presented in Table 2, in a first stage, we relied on qualitative methods. Face-to-face interviews are conducted and a list of participants is defined based on their profiles and activity on the BI system. Interviews were conducted with the three different profiles of users presented in Fig. 2, BI experts, business experts and end users of the BI system. Two users of each category participated. The objective was to gather users’ experiences and expertise in BI domain throughout our design process. This helps us to collect their opinions and enrich our findings from different users’ points of view. To proceed, the ISO 2500 standard is considered. Among its characteristics, we focused, in a first stage, on the functional suitability characteristic because we consider that it distinguishes the most a system from an other. Then, the procedure, during interviews, consisted in presenting to users the definition of ISO functional suitability characteristic and its sub characteristics. They suggested therefore a set of indicators, measures and dimensions for each sub characteristic to evaluate the used BI system. Each interviewee validated and enriched proposals of the other interviewees while justifying his/her opinion. Gathering BI users’ suggestions helped to effectively collect indicators, measures and dimensions for the evaluation of the BI activity.

In a second stage, a questionnaire based on interviews results was proposed. It contains questions of two natures (open questions and leading questions). The questionnaire was mainly designed for BI users, except the interviewees. They evaluated interviews results, particularly about the validity of identified indicators, measures and dimensions. In the questionnaire, for each identified indicator, the answerer is asked to evaluate corresponding measures and dimension according to rating scale, as well as to enrich the set of proposed measures. The questionnaire allows us to analyze the answers to check if users are generally satisfied with the proposed indicators, measures and dimensions, so we are more able to validate our proposal. A summary of these experiments are presented in Table 2. In total, we identified 29 quantitative indicators, 10 for BO, 9 for Safir, 6 for Stiki and 4 for the BlogCrolles300. In addition to quantitative indicators, we identified several user-based ones (with a subjective nature). Those cannot be objectively evaluated. That is why our proposal is composed of two complementary solutions: system-based and user-based.

M. Brichni et al. / Expert Systems With Applications 76 (2017) 97–112

6.3. Results of the co-design As a result, our proposal has practical added values to the considered environment at STMicroelectronics. one clear contribution of our research is the development of the BI4BI system for a continuous BI systems evolution. We demonstrated the feasibility and applicability of our system through different application examples on the existing BI system. At STMicroelectronics, it provides interests for people, technology and organization. First, the developed BI4BI system helps BI users evaluate, assess and make decisions on their own produced knowledge evolution based on their own uses, which makes them feel more comfortable with obtained results. Second, we note that, in the case of STMicroelectronics, the BI4BI system is actually based on a yet used technology in the organization. Therefore, its integration was adequate and suitable to the existing technological environment. Third, our proposal fits with the strategy of the organization to ensure a continuous BI evolution. These results are shown throughout the evolution in the use of the BI4BI system, as well as the intention of users to expand it to evaluate other tools, such as, the scheduler system.

107

facts and opinions. Even though evaluation remains a crucial step to make a system and its use evolve, we believe that alone it is not sufficient. Consequently, we believe that the evaluation step should be completed to put its results into action. Second, our proposal is mainly focused on a BI system evaluation at STMicroelectronics. Even though this focus is important to target real business needs, it should be easily expandable to allow evaluate other systems in other contexts. However, re-applying the current approach for each system and context risks to take a lot of time to suit each company business needs, which can be problematic in most cases. As a result, a more generalized approach should be proposed. Third, even though two important sources of data are considered, they can be insufficient, especially in other contexts. Actually, in the context of STMicroelectronics, systems’ and users’ data are the main available sources to evaluate an internal BI system. However, for evaluating other types of products, such as an open source software or a new device, taking into account social media users’ opinions and/or sensors data, becomes crucial. In such cases, BI systems analytics capabilities become limited and need to be evolved to consider new data sources.

7. Advantages and limitations of our proposal 8. Conclusion and perspectives As part of the rigor cycle, “DSR must provide clear contributions in the areas of the design construction knowledge (i.e., foundations), and/or design evaluation knowledge (i.e., methodologies)” (Hevner & March, 2004). In this section, we present our contributions to this knowledge base. This involves the communication of the advantages and limitations of our findings compared to existing ones. 7.1. Advantages In terms of foundations, in our research, a continuous BI evaluation solution has been proposed. The design process, specification and principle of the BI4BI approach constitute creative novel methods for evaluating BI knowledge compared to findings in the literature. Indeed, it considers two complementary data sources, bases on focused BI evaluation criteria and allows a continuous evaluation process. Actually, most of existing solutions present limitations related either to the incompatibility with our business needs or to their limited execution process and results. That is the case of the ´ ISO 250 0 0 model, BP neural network and fuzzy TOPSIS (Polanska & Zyznarski, 2009; Rouhani et al., 2012; Yan et al., 2012) (Section 3). As a result, this research addresses these limitations with a BI4BI system that can be adapted to both practical and research needs. While combining system and user based approaches, it offers an automated solution allowing a continuous evaluation. In terms of methodologies, we note that one main strength of our proposal is the use of a user-centered approach to conduct our research. As a result, with users, a set of evaluation criteria, indicators and measures to assess a BI system is identified. Unlike those defined in the literature (Table 1), based on users’ experiences and expertise, our identified evaluation criteria and indicators are focused on the BI domain knowledge, are of several categories and natures, and could be generalized to make them applicable not only for assessing a BI system but also information systems in general and in different contexts. These findings confirm the interest of using a user-centered approach for systems’ evaluation. 7.2. Limitations Despite the care we took with our BI evaluation proposal, some limits have been identified. First, currently, our solution is limited to the evaluation of a BI system. It includes targeting business needs and analyzing current

In this paper, our continuous BI evaluation proposal is presented. To this end, a BI4BI system is developed. It aims at evaluating to help analyzing and making decisions about the BI activity and its evolution. Its design process, specification and implementation are detailed. It includes two complementary solutions. A system-based solution relies on the BI system’s tools’ databases. A user-based solution relies on the results of a questionnaire evaluating the BI system from its users’ points of view. Together, the system-based and user-based solutions help to evaluate the BI system’s activity in order to ensure its continuous evolution. Our BI4BI system was designed and developed in an industrial context at STMicroelectronics. In terms of perspective, we aim to expand our continuous evaluation solution to a continuous improvement cycle. It includes not only the evaluation step but also the analysis, the decision making and the improvement application. In addition, our aim is to apply our proposal not only for Business Intelligence systems but also for information systems in general. To this end, first, identified indicators and measures for BI system evaluation will evolve to make them system independent, which is not discussed in this work. Actually, in our current work, indicators have been identified for each tool of the BI system (BO, Stiki, Safir and the Blog). As a result, our identified indicators are system-oriented. For example, the way ”content uses” is measured for BO is not the same for Safir. Therefore, our idea is to propose a more suitable indicators for all considered systems. Such a generalized solution makes it applicable not only for monitoring a BI system but also information systems in general and in different contexts. Second, in the context of STMicroelectronics, the BI domain has been considered for its evaluation. However, it could also be interesting to generalize our solution to be applied to different domains and contexts. Nowadays, due to the proliferation of systems that continuously produce data (social networks, mobile applications, electronic sensors, etc.), the volume of data and information produced every day is exponentially growing and data sources are various and very different (CXP, 2016). Complementary to BI, Big Data is seeking to navigate this avalanche of data in order to identify in real time relevant and useful information. This is the reason why, we believe that our vision fits in well with the Big Data field. It would be one way to improve making decisions for systems and better target areas of progress.

108

M. Brichni et al. / Expert Systems With Applications 76 (2017) 97–112

Appendix A. BO indicators, measures and dimensions

M. Brichni et al. / Expert Systems With Applications 76 (2017) 97–112

Appendix B. Questionnaire for user-base BI evolution (part1)

109

110

M. Brichni et al. / Expert Systems With Applications 76 (2017) 97–112

Appendix C. Example of a report about the level of information in Safir reports

References ˚ Agerfalk, P. J., & Goldkuhl, G. (2013). Action research & design research integrations. In AIS SIG Prag & Pre-ICIS workshop (pp. 1–4). Barone, D., Topaloglou, T., & Mylopoulos, J. (2012). Business intelligence modeling in action: A hospital case study. In Proceedings of the 24th international conference of advanced information systems engineering, Gdansk, Poland. In VL ’7328 (pp. 502–517). Springer Berlin Heidelberg. doi:10.1007/978- 3- 642- 31095- 9_33. Borim, H. C., Mukai, L. M., Lucas Emanuel, S. E. O., Lopes, V. J., & Cabral Moro, C. M. (2015). Integrating methods to evaluate health information systems. Studies in Health Technology and Informatics, 218, 100–104. Brichni, M., Dupuy-Chessa, S., Gzara, L., Mandran, N., & Jeannet, C. (2015). Business intelligence for business intelligence: A case study at STMicroelectronics. In Proceedings of the ninth international conference on research challenges in information science, Athens, Greece (pp. 239–249). IEEE Computer Society. doi:10.1109/RCIS.2015.7128885. Brichni, M., Mandran, N., Gzara, L., Dupuy-Chessa, S., & Rozier, D. (2014). Wiki for knowledge sharing, a user-centred evaluation approach: A case study at STMicroelectronics. Journal of Knowledge Management, 18, 1217–1232. doi:10.1108/ JKM- 04- 2014- 0123. Cook, T. S., & Nagy, P. (2014). Business intelligence for the radiologist: Making your data work for you. Journal of the American College of Radiology, 11, 1238–1240. doi:10.1016/j.jacr.2014.09.008.

CXP (2016). Big data pour doper la BI traditionnelle. Accessed 02.04.2016 http:// www.cxp.fr/content/forum- news/big- data- pour- doper- la- bi- traditionnelle. Dell’Aquila, C., Di Tria, F., Lefons, E., & Tangorra, F. (2008). Evaluating business intelligence platforms: A case study. In Proceedings of the 7th WSEAS international conference on artificial intelligence, knowledge engineering and data bases, Cambridge, UK. In AIKED’ 08 (pp. 558–564). World Scientific and Engineering Academy and Society (WSEAS). Foshay, N., & Kuziemsky, C. (2014). Towards an implementation framework for business intelligence in healthcare. International Journal of Information Management, 34, 20–27. doi:10.1016/j.ijinfomgt.2013.09.003. Hevner, A. R., & March, S. T. (2004). Design science in information systems research. MIS Quarterly, 28, 75–105. Hilgefort, I. (2011). Selecting the right SAP business objects BI client product based on your business requirements for SAP BW customers. Accessed 03.03.2016 http://www.scn.sap.com. Horkoff, J., Borgida, A., Mylopoulos, J., Barone, D., Jiang, L., Yu, E., & Amyot, D. (2012). Making data meaningful: The business intelligence model and its formal semantics in description logics. In Proceedings of on the move to meaningful internet systems: OTM, Rome, Italy. In VL’ 7566 (pp. 700–717). Springer Berlin Heidelberg. doi:10.1007/978- 3- 642- 33615- 7_17. Hu, C., & Zhao, F. (2010). Improved methods of BP neural network algorithm and its limitation. International Forum on Information Technology and Applications, 1, 11–14.

M. Brichni et al. / Expert Systems With Applications 76 (2017) 97–112 Isik, O., Jones, M. C., & Sidorova, A. (2013). Business intelligence success: The roles of BI capabilities and decision environments. Information & Management Journal, 50, 13–23. doi:10.1016/j.im.2012.12.001. ISO (2014). Iso 250 0 0 software product quality. Accessed 20.03.2016 http://www. iso250 0 0.com. Jrad, B. N., Ahmed, D., & Sundaram, D. (2014). Insider action design research a multi-methodological information systems research approach. In IEEE eighth international conference on research challenges in information science (RCIS) (pp. 1–12). Jin, W., Li, Z. J., Wei, L. S., & Zhen, H. (20 0 0). The improvements of BP neural network learning algorithm. In WCC 2000 - ICSP 2000. 2000 5th international conference on signal processing proceedings. 16th world computer congress 2000: 3 (pp. 1647–1649vol.3). Jooste, C., Van Biljon, J., & Mentz, J. (2014). Usability evaluation for business intelligence applications: A user support perspective. South African Computer Journal: SAICSIT 2013, 53, 32–44. doi:10.18489/sacj.v53i0.198. Lahrmann, G., Marx, F., Winter, R., & Wortmann, F. (2011). Business intelligence maturity: Development and evaluation of a theoretical model. In Proceedings of the 44th Hawaii international conference on system sciences, Kauai, Hawaii (pp. 1–10). System Sciences (HICSS). doi:10.1109/HICSS.2011.90. McBride, N. (2014). Business intelligence in magazine distribution. International Journal of Information Management, 34, 58–62. doi:10.1016/j.ijinfomgt.2013.09.006. Moro, S., Cortez, P., & Rita, P. (2015). Business intelligence in banking: A literature analysis from 2002 to 2013 using text mining and latent Dirichlet allocation. Expert Systems with Applications, 42, 1314–1324. doi:10.1016/j.eswa.2014.09.024.

111

Panos, V. (2009). A survey of extract-transform-load technology. International Journal of Data Warehousing and Mining, 5, 1–27. doi:10.4018/978- 1- 60960- 537- 7. ch008. Peffers, K., Tuunanen, T., Rothenberger, M., & Chatterjee, S. (2007). A design science research methodology for information systems research. Journal of Management Information Systems, 24, 45–77. ´ Polanska, J., & Zyznarski, M. (2009). Elaboration of a method for comparison of business intelligence systems which support data mining process. School of Engineering Blekinge Institute of Technology Ph.D. thesis.. Popovicˇ , A., Hackney, R., Simões Coelho, P., & Jaklicˇ , J. (2012). Towards business intelligence systems success: Effects of maturity and culture on analytical decision making. Decision Support Systems, 54, 729–739. doi:10.1016/j.dss.2012.08.017. Raber, D., Wortmann, F., & Winter, R. (2013). Towards the measurement of business intelligence maturity. In Proceedings of the 21st European conference on information systems completed research. In Paper 95 (pp. 1–12). AIS Electronic Library. Rouhani, S., Ghazanfari, M., & Jafari, M. (2012). Evaluation model of business intelligence for enterprise systems using fuzzy TOPSIS. Expert Systems with Applications, 39, 3764–3771. doi:10.1016/j.eswa.2011.09.074. ` V., & Tyrychtr, J. (2013). Agricultural e-government: Design Ulman, M., Vostrovsky, of quality evaluation method based on ISO square quality model. AGRIS On-Line Papers in Economics and Informatics, 5(4), 211. Yan, S.-L., Wang, Y., & Liu, J.-C. (2012). Research on the comprehensive evaluation of business intelligence system based on BP neural network. Systems Engineering Procedia, 4, 275–281. doi:10.1016/j.sepro.2011.11.076.

112

M. Brichni et al. / Expert Systems With Applications 76 (2017) 97–112

Manel Brichni has a Ph.D. degree in the domain of software engineering. During her Ph.D., she worked at STMicroelectronics, LIG laboratory and G-SCOP laboratory. Her thesis subject is about: Toward a continuous improvement cycle for knowledge capitalization at STMicroelectronics. Currently, she is a temporary teacher and researcher in computer science at Grenoble University (ATER). Her research interest concerns complex information systems management and Business Intelligence. Sophie Dupuy-Chessa is Professor at University of Grenoble. She got her Ph.D. thesis in 20 0 0 in the domain of software engineering, more precisely in software modeling. Then she has two post-doctoral positions: she was lecturer at University of Geneva and research scientist at Research Center Europe. She holds an authorization to steer researches (HDR) in Computer Science from the University of Grenoble in 2011. Currently, her research interest concerns model-driven engineering for human-computer interaction and information system design. Lilia Gzara is Associate Professor at Grenoble Institute of Technology. She is also member of the G-SCOP lab. She received her Ph.D. from the Grenoble Institute of Technology (20 0 0) and she occupied since then research/teaching positions at Polytechnic Montreal, Nancy University and Grenoble Institute of Technology. She co-directed many Ph.D. and master thesis and participated in five French national research projects. She is member of the design society and the French chapter of the International Council on Systems Engineering. Her research interests include information systems engineering, BPM, PLM and knowledge management in collaborative design context. Nadine Mandran is an engineer in Humanities and Social Sciences at the LIG laboratory in Grenoble, France. Her research interest concerns the production and analysis of data. Its mission is to provide methodological support to design experiments in user based research. She helps doctoral students of the laboratory in the implementation of these experiments. Corinne Jeannet is a team leader IT at STMicroelectronics. She has nearly 20 years of experience in the design and software development, the last 12 years mainly in project management and team management. Her main missions concern user interfaces adapted to the operational constraints (e.g. manufacturing), real-time reporting tools and data warehousing, production management systems and knowledge management. STMicroelectronics is one of the leaders in the semiconductor domain. STMicroelectronics is a French-Italian fusion between the microelectronic branch of Thomson and the SGS Microelectronica in 1987.