An Evaluation of Requirements Specification Capability Index

Available online at www.sciencedirect.com Available online at www.sciencedirect.com

ScienceDirect ScienceDirect

Procedia Computer Science 00 (2016) 000–000 Available online at www.sciencedirect.com Procedia Computer Science 00 (2016) 000–000

ScienceDirect

www.elsevier.com/locate/procedia www.elsevier.com/locate/procedia

Procedia Computer Science 112 (2017) 998–1006

20th International Conference on Knowledge Based and Intelligent Information and Engineering 20th International Conference on Knowledge and Intelligent Information and Engineering Systems,Based KES2017 Systems, KES2017

An Evaluation of Requirements Specification Capability Index An Evaluation of Requirements Specification Capability Index a a

Shuichiro Yamamotoa* Shuichiro Yamamotoa*

NAGOYA UNIVERSITY. Furo-cho, Chikusa-ku, Nagoya Aichi 460-8601 Japan NAGOYA UNIVERSITY. Furo-cho, Chikusa-ku, Nagoya Aichi 460-8601 Japan

Abstract Abstract It is well known that requirements specification is the critical success factor for developing information systems. However, the It is well known requirements specification specification is the critical capability success factor developing information systems. However, the quantitative indexthat to evaluate the requirements of theforsystem development organizations has not been quantitative evaluate the requirements specification capability the system development organizations not been known. Thisindex paperto proposes a requirements capability ofindex to evaluate information system has development known. This The paperrequirements proposes aspecification requirements specification capability indexontotheevaluate information development organizations. capability index is defined based 6 dimensions, that aresystem requirements vision organizations. The requirements specificationrequirements capability index is defined on the 6 process dimensions, thatrequirements are requirements vision development, requirements communication, product design,based requirements design, investment, development, requirements communication, requirements product design,index requirements process design, requirements investment, and requirements human resource development. The proposed capability has also been applied to several Japanese software and requirements human resource development. Thethe proposed capability index has also been appliedtotovisualize several Japanese software development organizations. The result shows that capability index can effectively be applied the requirements development organizations. result shows capability index can effectively be applied to visualize the requirements specification activities of theThe organizations fromthat thethe 6 dimensions. specification activities of the organizations from the 6 dimensions. © 2016 The Authors. Published by Elsevier B.V. © 2017 The Authors. Published by Elsevier B.V. © 2016 The Authors. Published by Elsevier B.V. Peer-review Peer-review under under responsibility responsibility of of KES KES International. International Peer-review under responsibility of KES International. Keywords: Requirements Specification, Capability Index, Quantitative evaluation Keywords: Requirements Specification, Capability Index, Quantitative evaluation

1. Introduction 1. Introduction It is well noticed that requirements specification is the most important success factor3,4, 10-21. There were many 3,4, 10-21 It is well methods, noticed that specification the most important success . There wereproject many approaches, and requirements guidelines so far. However, is organizational capability index factor of software development approaches, methods, and guidelines so far. However, organizational capability index of software development project has rarely been attracted. Requirements process and artifact definition is insufficient to describe appropriate has rarely been Requirements process artifact definition insufficient to describeengineers appropriate requirements. It isattracted. also difficult to encourage good and communication betweenisclients and requirements by requirements. It is also difficult to encourage good communication between clients and requirements engineers by only teaching requirements template for describing specification. only teaching requirements template for describing specification. * Corresponding author. * Corresponding E-mail address:author. [email protected] E-mail address: [email protected] 1877-0509 © 2016 The Authors. Published by Elsevier B.V. 1877-0509 2016responsibility The Authors. of Published by Elsevier B.V. Peer-review©under KES International. Peer-review under responsibility of KES International.

1877-0509 © 2017 The Authors. Published by Elsevier B.V. Peer-review under responsibility of KES International 10.1016/j.procs.2017.08.080



Shuichiro Yamamoto / Procedia Computer Science 112 (2017) 998–1006 Author name / Procedia Computer Science 00 (2016) 000–000

999

It is not enough to provide requirements specification guidelines for enterprises. The reason is that effective organizational capability is necessary to describe good requirements specifications. Therefore, it is necessary to evaluate organizational capability to develop requirements specifications. For example, it is difficult to achieve requirements specification activities based on pre-defined requirements process and artifact configuration without requirements vison. The rest of the paper is as follows. Section 2 describes related work. The requirements specification capability index is proposed in section 3. Evaluation of the proposed capability index is described in section 4. Discussions are shown in section 5. Section 6 summarizes this paper. 2. Related work Arthur1 proposed the quality metric on the development document maintainability. SW-CMM2 showed Requirements management as the Key Process Area (KPA) in software process capability aspects. CMMI-SE/SW adds Requirements Development as the KPA. Dautovic5 developed a tool to automatically evaluate software development document described by word processor based on the practical review rules. Wiegers3 described 20 questions to diagnose requirements practices2. Each question shows four choices to select the current levels of organizational practice. The answers of the questions can be effectively used to improve requirements techniques of organizations. These questions are bounded to technical aspects of requirements. It was not designed to measure organizational capability of requirements specification. Pohl4 describes a capability model to validate requirements with three levels. The three levels of requirements validation are minimal, standard, and advanced validation. These levels are defined through five validation aspects of activities. The aspects are context consideration, artefacts-context dimension, artefacts-documentation dimension, artefact-agreement dimension, and activity execution. Each aspect has sub activities. The number of sub activities of five aspects are 3, 8, 4, 4, and 5 respectively. The minimal validation level defines activities that each project should consider. The standard validation level defines activities that projects should achieve. The advanced validation level defines activities that quality critical project should achieve. Yamamoto6 proposed an Assurance Case Capability Index. The Assurance Case Capability Index (ACCI) defines evaluation indexes to introduce assurance case into enterprises from the points of technical and management. The index was used to clarify practical needs of assurance case introduction by trial applications. The ACCI consists of 7 aspects. The aspects are assurance case construction, assurance case vison development, assurance case communication, product design, process design, assurance case investment optimization, and system assurance human resource development. The ACCI has been developed to support O-DA standard7. The Open Group Real Time & Embedded Systems Forum focuses on standards for high assurance, secure dependable and complete systems7. At the heart of this O-DA (Open Dependability through Assuredness) standard, there is the concept of modeling dependencies, building assurance cases, and achieving agreement on accountability in the event of actual or potential failures. Assurance cases are necessary to assure architectures of dependable systems. The O-DA is based on TOGAF8. Yamamoto9 has studied the application of O-DA. 3. Design of requirements specification capability index To design capability index, evaluation aspects of activities are defined and then sub activities are elicited for each aspect. The evaluation aspects are designed based on the Assurance Case Capability Index by adding necessity aspects for specifying requirements and deleting weakly related aspects. Specifically, vison construction, communication, process design, product design, investment optimization, and human resource design aspects are adopted to the requirements specification capability index, although Assurance case building and Risk analysis aspects are deleted. The names of aspects are also changed to fit the requirements specification capability index. Finally, the aspects of requirements specification capability are defined as Requirements vison construction, Requirements communication, Requirement product design, Requirement process design, Requirement investment optimization, and Requirement

Shuichiro Yamamoto / Procedia Computer Science 112 (2017) 998–1006 Author name / Procedia Computer Science 00 (2016) 000–000

1000

Human Resource Development. Table 1 shows the Requirements Specification Capability Index including 30 activities. Each activity is evaluated based on the criteria shown in Table 2.

Table 1

Requirements Specification Capability Index 30

Aspects Requirements vison construction (6)

Requirements communication (5) Requirement product design (4) Requirement process design (4) Requirement investment optimization (6)

Requirement Human Resource Development (5)

Table 2 Level 0 1 2 3 4

Evaluation activity index 1)Roles of Strategic objectives and requirements definition are clearly defined 2)Organization to support that requirements definition achieves roles is built 3)Requirements investment is highly weighted 4)Utilization policy of requirements in development is clarified 5)Responsibility of requirement team is clearly defined 6)Responsibility of development team is clearly defined based on requirements 1)Employees share roles of requirements definition 2)Development team understands objectives of requirements 3)Problem solving process by requirements is transversally defined over divisions 4)Enterprise wide method to share requirements definition cases is defined 5)Management, requirements, and development divisions share return on investment of requirements 1)Quality goal for requirements artifact is defined 2)Utilization policy of requirements artifact is standardized 3)Requirements artifact is standardized from the points of inside and outside collaboration of enterprise 4)Requirements artifact is realized with no redundant description items 1)Requirements engineering process is defined 2)Utilization policy of requirements in development process is standardized 3)Requirements collaboration process inside as well as outside is standardized 4)Requirements engineering process is realized with no redundancy 1)Development budget of requirements asset is allocated 2)Independence of requirements team is considered 3)ROI of requirements definition is verified beforehand 4)Compatibility to enterprise optimization is considered in requirements definition 5)Utilization and effectiveness of requirements are measured after requirements definition 6)Issues of requirements utilization are resolved in the course of requirements engineering 1)Human resource is developed to propose improved requirements engineering process 2)Human resource who are familiar with requirements as well as development is assigned to business management level 3)Opportunity to learn business management knowledge is provided for requirement engineers 4)Opportunity to understand development process is provided to requirements engineers 5)Requirements utilization skill seminar is provided to requirements engineers

Levels of evaluation index

Documentation None Oral order Informal memo Department standard Enterprise standard

Explanation Activity is not carried out Activity is ordered orally without prescriptions Activity is ordered by memo Activity is carried out based on department standard manual Activity is carried out based on department standard manual

The Capability Index value CI(x) is defined as the mean value of (level ai)/4 for all ai of aspect x. For example, Requirements vison aspect include 6 activities. Suppose the levels of six activities are 1, 2, 1, 0, 3, and 1. Then CI (Requirements vison) is calculated as (1/4+2/4+1/4+0/4+3/4)/6 = 8/4/6 = 2/6 = 0.333. The meaning of the value 0.333 is more than oral order and less than informal memo, because 0.333 x 4 = 1.332. If the value is greater than 2, there is the evidence more than memo on the activity.



Shuichiro Yamamoto / Procedia Computer Science 112 (2017) 998–1006 Author name / Procedia Computer Science 00 (2016) 000–000

1001

4. Evaluation of the requirements specification capability index The evaluation of the proposed RSCI (Requirements Specification Capability Index) has been conducted by investigating Japanese embedded system development client and engineering companies. 4.1. Subjects The RSCI questionnaire form was provided to 14 engineers describing embedded system requirements. The RSCI questionnaire consists of 30 questions corresponding to RSCI activities. There were 9 client company engineers and 5 vendor engineers. 4.2. Evaluation result The result of the questionnaire for 14 subjects is shown in Table 3 by discriminating six aspects of RSCI. The value is defined as the summation of values for each activity. Table 3. Application result of Requirements Specification Capability Index

Client Vendor Mean

Requirement vison 0.315 0.267 0.291

Requirement communication 0.347 0.224 0.285

Requirement product 0.322 0.38 0.351

Requirement process 0.289 0.24 0.264

Requirement investment 0.215 0.167 0.191

Requirement HRD 0.276 0.312 0.294

The mean value of client is 0.294, whereas those value of vendor is 0.265. The RSCI value of client companies is higher than those of vendor companies for Requirement vison, Requirement communication, Requirement process, and Requirement investment aspects. The RSCI value of vendor companies is higher those of client companies for Requirement product and Requirement HRD aspects. Figure 1 shoes this result. The mean value of RSCI is approximately 0.279. The mean RSCI value of client companies is higher than those of vendor companies. The correlation between RSCI is shown in Table 4. The number with under-lines shows relatively high correlation. Figure 2, 3, 4 show the Requirements vision vs process, Requirements communication vs investment, and Requirements investment vs HRD, respectively.

Shuichiro Yamamoto / Procedia Computer Science 112 (2017) 998–1006 Author name / Procedia Computer Science 00 (2016) 000–000

1002

Fig. 1

Table 4

means of requirement specification capability index

Correlation between Requirements Specification Capability Index

Requirement vison Requirement communication Requirement product Requirement process Requirement investment

Requirement communication

Requirement product

Requirement process

Requirement investment

Requirement HRD

0.530

0.397

0.842

0.584

0.444

0.225

0.498

0.668

0.420

0.543

0.212

0.450

0.487

0.479

0.692

Author name / Procedia Computer Science 00 (2016) 000–000



Shuichiro Yamamoto / Procedia Computer Science 112 (2017) 998–1006

Fig. 2 Requirement vision vs requirement process

Fig. 3 Requirement communication vs requirement investment

1003

Author name / Procedia Computer Science 00 (2016) 000–000

1004

Shuichiro Yamamoto / Procedia Computer Science 112 (2017) 998–1006

Fig. 4

Requirement investment vs requirement HRD

5. Discussion 5.1. Effectiveness As mentioned above, RSCI (Requirements Specification Capability Index) is easily applicable to detect quantitatively week points of organizations and improve requirements specification capability by taking measures for the detected week points. The result also showed that only the knowledge on requirements artifact structure and requirement process is insufficient for utilizing requirements specification. The knowledge on requirements vison, communication, investment and HRD is also necessary. The mean RSCI of requirements product and process is 0.35 and 0.26 respectively. This showed that Requirements engineering methods have still not been well established for practical field engineers. In other words, the proposed RSCI is useful to show the limitation of current requirements engineering approaches quantitatively. This also disclosed that new practical requirements engineering method is necessary to fit the field engineers. These arguments showed the effectiveness of RSCI to explain the necessity of the new approaches to resolve issues on organizational requirements engineering activities quantitatively. Moreover, RSCI can quantitatively visualize requirements specification capability inter organizations. Industry wide requirements capability can also be visualized. For example, requirements specification capability of each industry will be clarified. And then, each company can evaluate the capability according to the industry capability criteria. For example, standard values of RSCI can be provided for each industry. Organizations can objectively recognize the requirements specification capability according to the industry standard RSCI. If the organization capability is lower than the standard value, capability improvement shall be necessary. Client companies can rationally select vendors that have appropriate requirements specification capability by providing necessary levels of RSCI. Traditional discipline just qualitatively describes that it is necessary to build consensus on requirements specification between client and vendor companies. If vendors have insufficient requirements specification capability, it is impossible to build consensus on requirements specification. The RSCI can be applied to support for building consensus by choosing appropriate level of capability. The RSCI is the quantitative method to achieve consensus building.



Shuichiro Yamamoto / Procedia Computer Science 112 (2017) 998–1006 Author name / Procedia Computer Science 00 (2016) 000–000

1005

5.2. Application method of RSCI RSCI will help elicit improvement points by visualizing issues of requirements specification activities, because RSCI defines evaluation index on requirements specification activities. (1) RSCI is useful to know if requirements specification is able to success by identifying necessary capability of requirements specification and investigating the corresponding capability index value of the organization. (2) RSCI is useful for identifying activities necessary to improve by investigating requirements specification capability. For example, cross-sectional evaluation of requirements specification capability can be achieved by company hearing based on RSCI 5.3. Relationship with traditional requirements engineering As we mentioned before, there are two types of modelling for requirements engineering methods. These modelling types are product and process modelling. RSCI can be used to combine with these types of models for identifying issues and improvement of models. If the relationship between RSCI and the quality of requirements process and product will be clarified, RSCI evaluation result will help requirement organization improve requirements engineering approaches. RSCI can be used to improve specific capabilities by choosing closely related aspects of RSCI. In case of the VC value of the specific aspect is not sufficient, activities of the aspect shall be revised to improve. 5.4. Extensibility In this paper, six aspects are used to define RSCI. New aspects and sub activities can be added to RSCI. It is also easy to add sub activities for the current aspects. The extension of RSCI shall be considered based on the use case scenario of RSCI. The RSCI itself has been developed by reusing assurance case capability index. This fact shows the extensible structure of the capability index. 5.5. Limitation The purpose of this paper is to reveal that requirements specification capability can quantitatively be evaluated through RSCI. Therefore, the completeness of RSCI has not been confirmed. The more complete RSCI will be developed by more number of case studies. For example, it is necessary to apply the RSCI for enterprise information system engineers. 6. Summary In this paper, the requirements specification capability index, RSCI, for evaluating organizational capability is proposed. The application result of RSCI has also been clarified for Japanese embedded system development companies. Traditional introduction of requirements engineering approaches tried to improve quality of system requirements specification individually by educating requirements specification methods for clients and engineers. However, it is more efficient to specify appropriate requirements specification methods based on the result by evaluating requirements specification capability of each organization. If the organization did not have sufficient requirements specification capability, it is useless to apply methods and principles which require sufficient capability of requirements specification. The requirements specification approaches shall be fit to the organizational capability of requirements specification. RSCI can visualize quantitatively the organizational capability of requirements specification. Acknowledgements This work has been conducted as a part of "Research Initiative on Advanced Software Engineering in 2015"

1006

Shuichiro Yamamoto / Procedia Computer Science 112 (2017) 998–1006 Author name / Procedia Computer Science 00 (2016) 000–000

supported by Software Reliability Enhancement Center (SEC), Information Technology Promotion Agency Japan (IPA). References 1. Arthur, J. and Stevens, K., Assessing the Adequacy of Documentation Through Document Quality Indicators, Proceedings of the International Conference on Software Maintenance, 1989. 2. CMU SEI, CMMI for Systems Engineering/ Software Engineering, Technical Report CMU/SEI-2000-TR-018, 2000. 3. Wiegers, K., Software Requirements, Practical techniques for gathering and managing requirements throughout the product development cycle, Microsoft, 2003. 4. Pohl, K., Requirements Engineering – Fundamentals, Principles, and Techniques, Springer, 2010. 5. Dautovic, A., Plösch, V, Saft, M., Automatic Checking of Quality Best Practices in Software Development Documents, 2011 11th International Conference on Quality Software, pp.208-217, 2011. 6. Yamamoto, S., A Proposal on the Assurance Case Capability Index, SIG-KBSE, KBSE2015-63, pp.85-90, 2016(in Japanese) 7. Open Group Standard, Real-Time and Embedded Systems, Dependability through Assuredness™ (O-DA) Framework, 2013 8. TOGAF®, an Open Group Standard; www.opengroup.org/togaf 9. Yamamoto, S. and Morisaki, S., A case study on architecture quality assurance service using O-DA, CENTERIS 2016 10. IIBA, A Guide to the Business Analysis Body of Knowledge, Lightning Source Inc., ISBN-13: 978-1927584026 11. The home of Requirements Engineering, http://www.ireb.org/ 12. Sommerville, I. and Sawyer, P., Requirements Engineerign– A Good Practice Guide, John Wiley & Sons, 1997 13. Leffingwel, D. and Widrig, D., Managing Software Requirements- A Unified Approach, Addison-Wesley Professional, 2000 14. Kotonya, G., and Sonnmerville, I., Requirements Engineering, John Wiley & Sons, 2002 15. Hull, E., Jackson K., and Dick, J., Requirements Engineering, Springer-Verlag, 2002 16. Aurum, A., Wohlin, C., Engineering and Managing Software Requirements, Springer, 2005 17. Davis, A., Software Requirements: Objects, Functions and States, Prentice-Hall, 1993 18. Davis, A., Just Enough Requirements Management – Where Software Development Meets Marketing -, DORSET HOUSE PUBLISHING, 2005 19. Berenbach, B., Paulish, D., Kazmeier, J., and Dudorfeer, A., Software & Systems Requirements Engineering In Practice, McGraw Hill, 2009. 20. IEEE Std. 830-1998, IEEE Recommended Practice for Software Requirements Specification 21. ISO/IEC/IEEE 29148, Systems and software engineering —Life cycle processes — Requirements engineering, 2011