- Email: [email protected]
Software re-engineering and reuse from a Japanese point of view
Software re-engineering and reuse from a Japanese point of view N Mii and T Takeshita*
It is generally recognized that software re-engineering and reuse technologies are vitally important and a highly effective means of alleviating the so-called 'software crisis" The paper discusses various methodologies and tools in a review of related research and development efforts as well as commercialized products, mainly in Japan, and places them in a consolidated perspective, giving some examples of those items considered as particularly interesting from a Japanese point of view. In Japan, the concept of treating reusable pieces of software as parts has been very effectively employed, and tree-structured graphical charts are widely used in both forward and reverse software engineering. An increasing emphasis is on the use of domain knowledge and object-oriented methodologies. software maintenance, software restructuring, software reverse engineering, software re-engineering, software reuse, software skeletons, software parts, repository, knowledge bases, domain knowledge
Research and development projects on software development technologies and CASE tools conducted in Japan have been introduced to English-speaking readers by several papers 1-5 and books 6'7. According to Professor Ohno 5, the current main subjects in Japanese software engineering are CASE, object-oriented methodology, repositories, user interface management systems, software processes, and groupware. For a long time the primary author of this paper, N. Mii, has been seriously interested in and strongly advocated the reduction of both the hardware and software development cycles. The co-author, T. Takeshita, has surveyed CASE technologies in the United States, Europe and Japan, and in 1990 published a book on the subject s. Following this, he pursued studies of software maintenance/re-engineering and reuse, which led to the publication of another book 9 two years later. As it has been fully recognized that the two most effective means of shortening the software development and maintenance cycle are re-engineering and reuse of software, Mii and Takeshita decided to work together to explore currently used technologies as well as the state of the arts in these two areas. This paper focuses on the situation
Vice President Office,LAB-007,APTO, IBM Japan, 1623-14,Shimotsuruma, Yamato City, Kanagawa Pref. 242, Japan *Collegeof Business Administration and Information Science,Chubu University 1200,Matsumoto-cho,Kasugai City, Aichi Pref. 487, Japan Voi 35 No I January 1993
in Japan, thus minimizing any overlap with the contents of a recent paper by Hall 1°. Before proceeding to the subject matter, listed below are those software engineering technologies the authors consider to have made great progress in Japan in the past two decades: • • • • • •
Development processes and management. Quantitative measurement and control. Graphics (tree-structured charts). Testing methods and tools. Software reusability and reuse. Integrating tools and interfaces.
In general, Japan has devoted considerable effort to implementation-phase (lower CASE) technologies for software development 7. The authors feel that the combination of re-engineering and reuse technologies is a powerful answer to the question of how to improve significantly both the quality and productivity of software production and maintenance. According to a survey made in 1986 by Capers Jones, a US software consultant, the ratios of common functions contained in application, tele-communication, and systems programs are 75%, 70%, and 50% respectively. Similar statistics have been reported in Japan. For example, according to a paper presented by Fujitsu in March 1984, 10 common paradigms were used in 80% of code in 100 electronic switching service modules. Dynamically changing business environments, including the need for higher usability and quality, diversification and complication of consumers' demands, globalization of business operations, and so forth, as well as the advance of hardware and communication technologies, are resulting in smaller and heterogeneous network computing, and making business re-engineering (re-design of business processes) an essential and frequently repeated task. Unlike in the USA, the lifetime of a majority of application systems is considered to be less than five years in Japan. Software re-engineering technologies can play an effective role in facilitating this re-engineering. The proportion of costs in enterprises for software development and maintenance is now much greater than that for hardware. There are astronomical quantities of software assets (estimated at well over 100 billion lines of code in the USA alone). The life of these assets needs to be prolonged, and their components should be reused or recycled as much as possible.
0950-5849/93/010045-09 O 1993 Butterworth-Heinemann Ltd
45
Software re-engineering and reuse from a Japanese point of view The vital importance of re-engineering and reuse has been well recognized since the late 1970s, and related technologies have been steadily developed and put to practical use in Japan. Most of the original ideas came from the USA and Europe but have evolved and been refined in pragmatic ways by Japanese software engineers and practitioners. The paper includes reviews of major R&D efforts along with marketed products and will, we hope, serve to shed light on recent trends and directions in these software engineering fields in Japan.
SOFTWARE RE-ENGINEERING TECHNOLOGIES The term 'CASE' began to be used toward the end of 1986, and various CASE tools and environments have been developed and marketed for several years, but their utilization and effect on productivity improvement do not seem to be as impressive as expected. Why should this be? So far, CASE tools have been designed primarily for the development of new (application) s o f t w a r e - - t h a t is, in forward engineering. However, in large enterprises a greater part of DP resources (80% or even more) is spent on the maintenance or re-engineering of existing code, so the improvement of new application development does not contribute much to the reduction of the total workload or cost in the DP department. According to a survey conducted by Nikkei Computer (a technical magazine published in Tokyo) in the middle of 1990, 47.3% of the DP cost in 396 Japanese companies that responded to the questionnaire was spent on maintenance of software. For this reason, people have currently begun to pay much closer attention to software re-engineering technologies, as well as to software reuse, as these technologies not only reduce the maintenance workload and cost, but also facilitate business re-engineering (re-design of business processes) by adapting the processes to new and changing business environments and customer needs. Moreover, it has been admitted by experts that, even with best-of-the-breed tools in a well-integrated CASE environment, it is difficult to improve productivity by an order of magnitude without extensive reuse. Software re-engineering technologies, if used for re-engineering of business processes, help rejuvenate some or all of the components of existing applications-source code, file and database structures, screen and report definitions, and so on. This eliminates or reduces the need to develop new software from scratch. Therefore, the authors believe that reverse engineering and reuse technologies help enterprises both to derive maximum benefit from previous efforts and to reduce the amount of entirely new efforts, thus considerably shortening the software development and maintenance cycle and greatly improving its quality. 46
Re-engineering at the source code level If a design specification is represented in a structured programming style using a pseudo-natural language--in Japan a mixture of Kanji (Chinese characters) and Kana (phonetic symbols)--then its contents are more easily understood than otherwise, and source statements can be changed and/or added with much less effort than for unstructured code. Japanese computer manufacturers and Nippon Telephone and Telegraph Corporation (NTT) have made enormous efforts in developing graphical notations for constructs in structured programming. The graphical representations are often called 'tree-structured charts '2"''. The following different kinds of notation have been developed in Japan: • Hitachi: PAD (Problem Analysis Diagram). • NEC: SPD (Structured Programming Diagram). • NTT: HCP (Hierarchical and ComPact description chart). • Toshiba: T F P / T F D (Technical description For Procedure/Data design). • Fujitsu: YAC II (Yet Another Control chart II). Though a similar notation called 'Greenprint' was created at IBM's T J Watson Research Center toward the end of 1970s, and Action Diagrams are currently supported by a few CASE environments developed in the USA, Japan is further advanced in utilizing treestructured graphical representations in software development and maintenance than the rest of the world. Tree-structured graphic notations were developed in Japan from the late 1970s to the beginning of the 1980s. Once a detailed design has been completed, using one of these representations, source code in a 3GL is automatically generated. Reverse conversion from source code to a graphical representation is also automated, which eliminates both the need to keep detailed design specifications and also the inconsistency between the source code and design specifications. It is noted that the tree-structured representations are also used for design review, code inspection, and debugging by animation. Until the debut of powerful workstations with large, high-resolution screens with GUI in 1987, many people were sceptical about the practicality of tree-structured charts, as it is cumbersome for a human to create and maintain them, and they consume too much expensive space on display screens. These problems were resolved with the development of high-resolution, colour graphics on PCs and workstations. This kind of visual programming, though at a low level, allows better and faster design and modification, as well as enforcing design and programming standards. CASE tools are provided to convert source c o d e - - m o s t l y in Cobol, Fortran, or C - - i n t o tree-structured charts containing labels and the names of functions and procedures as well as variables in Japanese. For example, Fujitsu's YPS (YAC II Programming System) converts source code into YAC II charts, Hitachi's SEWB2 (Software Engineering Workbench 2) converts it into PAD, and so forth. IBM's Information and Software Technology
N MI1 AND T TAKESHITA
Tokyo Research Laboratory has developed a research prototype called 'RE/Cycle' which includes the capability to convert Cobol and PL/I source code into Action Diagrams.
Extraction of higher-level design information The design information contained in a tree-structured chart is almost at the source code level. Several research efforts have been made to obtain higher-level design information. An early example is an NEC tool called SYSDES (System Design Data Generation), which generates system design information from source code, a set of Job Control Language statements, and a set of parameters for menus. A paper on this was presented in October 1986. It is a component of NEC's SEA/I (Software Engineering Architecture/l), the initial version of which was released in 1983. In 1991 Fujitsu released a product called YPS ReEngineering. It generates from Cobol source code not only YAC II charts, but also hierarchical charts showing the relationships of routines. RE/Cycle converts spaghetti code in Cobol or PL/I into structured code, using algorithms based on the graph theory, and produces a structure chart containing not only who-call-whom relationships of routines or procedures, but also information on the dataflow between a parent and child boxes (routines or procedures). It also has an extensive impact analysis capability. Fujitsu Laboratories is reportedly developing a tool to generate detailed specifications from Cobol source code by referring to a data dictionary and program pattern library. It is currently applied to financial applications, whose programs often have common patterns.
Re-engineering of data (files and databases) Although programs developed in Japan in the past decade or so are mostly structured, conforming to some kind of installation standard, software maintainers still suffer from insufficient or out-of-date documentation on data. One problem is the existence of homonyms and synonyms used as names of data items. Tools are needed to detect these and help the user to resolve them by changing the names so that they conform to a suitable rule. A good naming convention should be established, and names used in old programs should be changed to follow it when programs containing them are maintained or rejuvenated. Another and possibly much bigger problem is the difficulty of understanding the way in which existing files and databases are organized. It is highly desirable that documentation of their conceptual and logical design should be automatically generated from their existing physical definitions and that they should be semi-automatically converted to accommodate new requirements, with subsequent changes, if necessary, to programs that have access to them. N T T Software Laboratory has developed a database design support tool called DBprompt '2, which consists of Vol 35 No 1 January 1993
the Information Resource Dictionary System and tools to support data name standardization (NAME), database conceptual design (ANLZ), logical design (DSGN), physical design (GNTL), and design information retrieval (RTRV). D B p r o m p t / N A M E checks the names of data items in data bases, eliminates synonyms and homonyms, and standardizes the names, using a naming rule and a terminology dictionary (containing 1296 words). In most cases, the following rule holds: A data item name = a qualifying word + a main word + a subdivision word The database design information is extracted from the database design sheets (of the existing databases) by RTRV. The extracted data names are standardized by using NAME. The forward engineering (logical and physical design) is performed by using D S G N and GNTL.
Re-engineering at the application system level Although the understanding of an individual component or module of each application system is essential for its maintenance and redesign, it has been found that all the components of an application system must be reviewed, and syntactical and semantic analysis must then be performed, so that the relations and dependencies among files, transaction messages, screen panels, reports, source programs and so on can be grasped and placed into the right contexts. The following items of information are required: • File and database organizations. • Record layouts and formats; transaction message formats. • Report formats; screen formats and screen transitions. • Relations and dependencies among modules, procedures, and subroutines/functions. • Relations between files/databases and program modules. • Relations between screens and program modules. • Potential impact of changes and of change history. In general, these are neither documented nor stored systematically, but are very often vaguely stored in the memory of designers and programmers. Some limited CASE capabilities addressing this need have recently been provided by Japanese computer manufacturers. A comprehensive CASE environment called RESCUE (Reverse Engineering System for Comfortable User's Environment), with a strong focus on software maintenance, has been developed by a Tokyo software house, CASE Technology '3. It consists of three categories of function: (l) static analysis (entry into repository and retrieval for maintenance), (2) dynamic analysis (acquisition and retrieval of tracing information), and (3) quality assurance (generation and comparison of test data). Not only programs, but also other application components such as JCL statements,
47
Software re-engineering and reuse from a Japanese point of view
and databases, screen panels, COPY and system definitions are analysed, and the extracted information is stored in master files by control category. Business terms in Japanese are attached to the objects to be looked at, enabling the user to make references without knowing the mnemonic names used in an implementation. Additional information, such as relations between file names and data set names and between panel names and program names, can be added by the user. Various cross-reference tables are generated.
SOFTWARE REUSE TECHNOLOGIES The reuse of software has been realized in different ways and at different levels '4"'5. The most common and takenfor-granted approaches are the use of subroutines, functions, macro routines, and generators. The smaller the size of software pieces, the greater the chances of their repeated use. However, because of their high granularity and the relatively large overhead of linking them to the invoking routine, programming productivity cannot be greatly improved in general by the existing approach. What is required for a large gain is much less extra work on the part of programmers in invoking many short sequences of statements in a target language, or the reuse of software of proven quality in a module or at the component level, especially the reuse of design specifications rather than source code. In Japan, the concept of treating reusable pieces of code as software parts began to be implemented as a key ingredient of integrated software development environments toward the end of the 1970s, and the number of papers on this topic reached a peak in the middle of the last decade. Although software parts are designed primarily for professional programmers of commercial and process control applications, 4GLs and their generators are provided to enable non-DP professionals or novice programmers to write commercial applications in much less time. We first take a brief look at the 4GLs that have been developed in Japan.
4GLs and generators A number of R&D and product development efforts have been devoted to 4GLs and their generators. It is particularly noteworthy that many of them have precoded standard units that can be selected and incorporated into generated code. The following are some of the 4GL products marketed by Japanese computer manufacturers. • Fujitsu: Hyper Cobol, CASET, BAGLES II, STRACT. • Hitachi: CORAL, EAGLE/4GL. • NEC: IDL II. Hyper Cobol is a typical case of enabling the reuse of relatively large software parts, as it has I i precoded standard units: Edit, Report, Check, Sort, Merge, 48
Summarize, Break, Separate, Chain, Match, and Control. It supports a dataflow concept, and a source program generally consists of record definitions similar to Cobol record descriptions, Stream Areas (I/O definitions), and some of the standard units listed above. Cobol source code can be added. CASET (Computer Aided Software Engineering Tools) is also a Cobol generator. It consists of FORM (screen and report definition functions with prototyping capability), FILE (for file definition), CAPE (Customer Application Program Editor), which is used for editing a dataflow-oriented language, and 13 categories of standard patterns (such as billing entry, record selection, separation, summary report printing, and merging) to be used for generating Cobol source code. BAGLES II (Business Application Generator's Library for Extensive Support II) generates Cobol code from a table-format business specification. BAGLES was originally designed for banking applications. STRACT (STRategic interACTive information system) is an end-user-oriented language that has access to relational databases. CORAL (Customer-Oriented Application Development System) allows the use of a pseudo-Japanese language for design specifications, and provides flexible decision tables, dialogue procedures and menus, a dictionary, macros, and source level debugging. EAGLE (Effective Approach to achieve hiGh-LEvel software productivity)/4GL is a Cobol generator for small general-purpose systems. It provides three kinds of process pattern (file creation/update, file matching, and interactive editing), and allows the use of Japanese words in source statements and definitions of data. It can now be used in a distributed environment '6. IDL II (Integrated Data-oriented Language II) receives design information created by SPECDESSIN (a system analysis and high-level design tool), which supports the dataflow diagrams. It then generates Cobol code including commonly used patterns.
Reuse at the source code level Bread-and-butter business applications, such as payroll management, inventory control, and sales accounting, contain common process patterns such as sorting, merging, and updating of files. The names of files and data items, and the internal processes (computation algorithms) may vary from application to application. For the repeated use of common patterns, these names and the processing logics are left blank, to be filled in for each new application. Each blank field needs an associated indication of what should be entered. Such incomplete source code modules or routines can be provided with proper explanations and supported by retrieval, customization, and synthesis/fabrication mechanisms in a CASE environment t4'7''8. Kanno '4 contains detailed discussion of 'module design technologies for reuse', 'standard platform and parts', and 'reuse-oriented software development environments and software production control technologies'. Information and Software Technology
N MII AND T TAKESHITA
The patterns are customized to create application skeletons, into which common software parts and new coding pieces are to be inserted by a synthesizer or fabricator. A customization aid helps the user to make changes to fit his particular need. An integration mechanism receives as input completed modules, and checks the consistency of data names and attributes and the completeness of other interface information before combining them into a single integrated module or a set of modules• A module constructed to allow the user to look into its inside and make changes to it is called a 'white box'. This approach makes it possible to use existing pieces of code whose granularity is larger than that of library functions or subroutines, which are mostly of a blackbox type. With a minor risk of degradation of quality, it is an easy-to-use approach in some application domains such as routine business application systems. To facilitate understanding of the internal functions and algorithms/logics, it is desirable that the module should be displayed in a graphical (for example, treestructured) representation. Such attempts have been made by Japanese computer manufacturers and NTT. As early as 1981, Fujitsu made available a product called P A R A D I G M , which is a set of standard patterns (in which input checking, matching, updating, and report generation are the primary paradigms) to be used for commercial application development. It also provides facilities that allow the customer to create his own software parts. NEC's SEA/1 was designed for the development of commercial applications in Cobol, initially on office systems, and its first version began to be used for production in 1983. It handles the three types of software parts: • Data definition parts: portions of the Cobol Data Division that define the formats of output reports• • Process parts: units referenced by P E R F O R M or C A L L statements (in the Procedure Division). • Skeleton parts: program control structures and information on parts to be used. Each SEA/I part consists of a header and body, and is combined with other parts by a program synthesizer. As components of SEA/l, NEC developed a pattern customization control language (PCL) for creating patterns of high generality, and a parts description language (PDL). According to a paper presented in September 1984, standard parts and their numbers initially provided by NEC were standard file I/O (10), table handling (13), item checking (3), item editing (15), data processing (17), time processing (3), code conversion (11), calculation (4), unit conversion (5), and code conversion, sorting, and so on (8). The number of different parts in each category is shown in parentheses. In 1987 NEC announced that 'SEA/I' now meant a combination of a methodology (STEPS) and supporting tools (SOFPIA, IDL II, TQF II), and in the following year it made available SOFPIA (SOFtware Productivity Improvement Aid) and PWB (Personal Work Bench). Vol 35 No 1 January 1993
The former includes tools for data definition, program synthesis, testing, system installation, and product management, while the latter supports data (screen layout) definition and program generation on a workstation. Hitachi's EAGLE (released in March 1984) and EAGLE2 (September 1985) also support reuse of software parts, initially with 22 standard patterns and 16 parts for batch processing, and 9 patterns and 60 parts for on-line processing. According to the results of a survey on the use of EAGLE by Hitachi customers over a period of more than two years from March 1984 '9, standard patterns were applied to 80% of batch applications and 70% of on-line applications, and the ratio of patterns and parts to the generated program in size (LOC) was 60%, the average productivity improvement was by a factor of two and the number of errors was reduced by two-thirds• Similar data have been reported by other manufacturers, indicating the effectiveness of this software reuse approach. The reuse of software parts for Cobol applications is also supported by Toshiba's MYSTAR (released in 1985). It has four types of skeleton (white box) library: individual user, project, global, and auxiliary. It includes various tools for functions such as listing, storing, retrieving, and deleting. One way to increase the reusability of source code is to limit the size of a module to one page with its functions self-contained, minimizing its interface complexity for easier and simpler linkage. An example of this approach is 50SM (50 Steps/Module, meaning 50 lines of code per module), which was initiated as part of the IMAP (Integrated software Management And Production support system) project at Toshiba 2°. It is a parts-oriented design/coding technology, on which a paper was first presented in March 1985.50SM supports development activities from module design through system integration, and includes T F F (Technical description Formula for Fifty SM design) and TFP/TFD. As an early example of the effectiveness of TFF, Toshiba reported in September 1985 that more than 30% reuse in new software had been achieved by a group of 10 developers. The SIGMA project also produced tools for software parts such as a pattern customizer, a skeleton generator (to create skeletons from tree-structured charts), a parts compiler, and a program synthesizer2''22 Fujitsu's YPS/APG (YPS-based Application Program Generation system), which was released in 1989, includes an editor for YPS (YAC II Programming System), a decision table, an editing table, generation of YPS, referencing, and copying; the replacement by specified character strings at the time of synthesis; and the reuse of screen and report layouts and on-line screen trans,t,ons-. According to a paper presented in March 199124, the advantages of parts reuse include the reduction of statements for unique processes (not using software parts) to between 20% and 30% of the total program, the narrowed scope and increased efficiency of 49
Software re-engineering and reuse from a Japanese point of view
design review, and the elimination of unit tests, resulting in productivity improvement by a factor of 1.3 to 1.5. Judging from these experiences, the extension of this kind of source code reuse technology will double the total productivity of application development and will increase the maintenance productivity by several times in many cases.
Reuse of the design specification level In general, the reuse of specifications at the functional or system level can be realized in some application subdomains where application systems consisting of similar inputs, outputs, and processes are developed. Programming elements that vary from system to system or from user to user must be identifiable. Supposing that such subdomains exist, development processes as well as design representation must be standardized to enable the reuse of design specifications. In some cases, customizable packages are developed and provided to different customers who tailor them to their needs by adding their unique characteristics. This can be regarded as one way of reusing design specifications. If design specifications are represented in a very high-level textual language or in a table format in which those items that can be changed or inserted are left blank with some indications, then it is easier to reuse them. Another way of reusing design specifications is to provide a method and supporting tool to help the user to understand the requirement definitions and to search for a design specification that best satisfies the given requirement. In Japan, papers on reuse of specifications began to appear in 1984. In a paper presented in September 1985, Hitachi discussed the creation and reuse of specifications, including data flow diagrams on SEWB, which is connected to EAGLE on the host system. CADRIS (Computer-Aided Design and Reuse with Intelligent Support system), developed at Hitachi Omika Works provides an FIF (Fill-in-the-Form) method using table-format worksheets and a guidance method to facilitate the entry of design specifications, and supports the retrieval of design specifications from a database, which is based on the E-R model25.
USE OF APPLICATION DOMAIN KNOWLEDGE Papers on the use of knowledge bases or expert systems for software development and maintenance began to appear in Japan in 1983, when Hitachi presented a paper entitled 'Two-level method for finding software parts using knowledge'. NEC presented a paper on 'Automatic program generation system using knowledge' in 1985. The annual number of papers on the use of knowledge bases in software engineering then gradually 50
increased, reaching 10 in 1988 and peaking at 20 in the following year. Without application domain knowledge, it is not possible to represent extracted design information by using terminology and expressions familiar to non-DP professionals in such a way that they can readily understand the processing logics and the business rules embedded in the source code as well as the meanings of the screen panels, file and database structures, and report formats. If frequently used process patterns (represented in a pseudo-natural language) that perform business rules or their refinements and their corresponding process patterns (in incomplete source code) are stored in a knowledge base, and ifa mechanism that interactively supports searching for and retrieving the patterns is provided, then this knowledge base can be used for both designing in forward engineering and design recovery in reverse engineering. Such a knowledge base should contain business terms and corresponding mnemonic names and processing logics to be used in source code. According to a paper printed in the Hitachi Journal in 1987, Hitachi established a basic technology for applying knowledge engineering to software reuse 26. Requirement definitions are entered in Japanese. Keywords for retrieving specifications are automatically extracted. Words not needed for software development are removed. Required data and functions are identified by inference, resulting in the addition of related keywords. By using the obtained keywords, a specification is retrieved and modified for new, different features and converted into source code by using SEWB (Software Engineering Work Bench) and EAGLE2. Mitsubishi Electric has developed a research prototype, SAGE (System for Automatic program GEneration) which infers a process pattern from a requirement specification by using domain knowledge, and generates a batch file processing program. NEC's PGEN (Program Generator), which was commissioned by the Institute for New Generation Computer Technology (ICOT), generates source code by understanding the meaning and context of detailed specifications prepared by a system analyst. This system is based on natural language understanding and knowledge engineering, and deals with detailed semantic analysis, heuristics (essentially rules of thumb) for understanding dependency relationships, and multiplicities of meanings. Initially PGEN worked only with detailed specifications. It is hoped that the system will be extended so that it can work with parts of programs such as common subroutines and automatically adapt them to a new application, rather than simply doing lineby-line coding. SoftDA (Software Design Automation), developed by NTT, is a CASE environment with knowledge database 27. It converts Japanese names used in pseudo-natural language statements into predetermined, standard mnemonic names by referring to a vocabulary in a domain knowledge library. Conversely, it can generate detailed design specifications in pseudo-Japanese from Information and Software Technology
N MI1 AND T TAKESHITA
source code, with the mnemonic names replaced by business terms. The Domain Knowledge Database contains data names, their semantic information, their mnemonic names, process patterns, and corresponding source code patterns. In 1991 Toshiba presented J O K E R (JapaneseOriented Knowledge Engineering/Environment for software Reuse), which accepts a requirement specification in Japanese, represents it in a formal DFD, and verifies and evaluates it 28. It was expanded into J O K E R '91, which acquires knowledge about the terms used in a requirement specification written in Japanese, and organizes, stores, and represents the knowledge on the basis of an object model, increasing the reuse of application domain know-how and enabling this to be linked to object-oriented analysis and design 29. Hitachi is working on an experimental software reuse development system based on co-operative problem solving using a knowledge base. This system was also commissioned by ICOT. As a unique product, Hitachi is marketing an intelligent business document recognition and application generator called 'BELIEVE' for an office system 3°. Formats and character sizes are recognized from an existing document (which may have been created on a word processor) by an image reader and displayed on the screen in such a way that modifications can be made. A generated program is in the same format as one created by using ETOILE (Effective Object-Oriented language for End Users on Office Processor), and so can be easily modified. BELIEVE uses a knowledge dictionary in which recognition and program generation rules are stored. It contains word and phrase dictionaries, each of which consists of the attribute/property area and the rule (range and calculation) area.
OBJECT-ORIENTED
TECHNOLOGIES
In Japan, several research and development projects on OOPL and OODB have been conducted. Although Object-Oriented ( 0 0 ) analysis and design methods have not yet become well established or standardized, some attempts to use them and their results have been reported. First, we will look at R&D on user interface builders making use of O 0 technology, and then at the CASE environments using or supporting this technology.
User interface builders NEC has developed Canae, a high-level software platform for user interface development using editor parts 3'. The system runs as a client process under X-Windows and consists of two kinds of parts: • Interactive parts: buttons, menus, scrolling bars, iconic menus, and so forth. • Editor parts for editing Japanese text, figures, image, graph structures, hierarchical structures, and tables. Vol 35 No 1 January 1993
An OO approach has been adopted to implement the editors based on a modified version of the MVC (ModelView-Controller) model designed for Smalltalk-80. Mitsubishi has developed a prototype of a C + + library called GhostHouse 32. It is designed for the XWindows System, using Xt (for resource management and event control) and the widget of OSF/Motif (for standard parts). Hitachi System Development Lab has developed an interactive presentation application development platform using an OO approach 33. It has been built by using the OSF/Motif toolkit on the X-server under HI-UX (Hitachi's UNIX). This has reduced by four-fifths the time required to create presentation packages on car sales. Hitachi Central Laboratory and its two subsidiary companies have developed an application builder called O D E T T E as a design support tool. This allows the entry, manipulation, and observation of objects through easyto-understand figures on the basis of the Common Lisp Objects System 34. CASE
environments
In 1991, Hitachi Omika Works developed Hyproof (Hyper-Programming system by OO Formula), a 4GL system for control system software development 35. Its objectives are (1) to accumulate and reuse functional and program specifications, and (2) to allow development of software without knowing how it will be implemented. Its basic approaches are modelling of the process of functional specification by domain, extraction of reusable program parts from application programs, and employment of a domain-oriented software reuse process. NEC C&C System Research Lab and C&C System Engineering Division have developed a prototype of an office application development environment using an OODB (Odin) 36. A business application system consists of three types of processing: user interface, database access, and business logic. Data structures are defined as instance variables, and the display methods and item relationships with the business database are defined as class variables of the form objects. Data structures and procedures can be inherited. This system can be considered as an OODB/4GL. Form views are implemented on OSF/Motif, but the system can be applied to other GUIs by changing the form view. CONCLUSION Software re-engineering and reuse, if properly designed, planned, and used, help enterprises to derive maximum benefit from past development efforts and/or to prolong the life of software artefacts, reduce the software development and maintenance cycle and, most importantly, improve productivity and quality in software production and maintenance by an order of magnitude. In Japan, lower CASE technologies (which can be considered as product engineering technologies) have 51
Software re-engineering and reuse from a Japanese point of view
advanced with the visualization of detailed design, the utilization of the concept of software parts, and so on. Technologies for re-engineering and reuse of detailed design and source code appear to be reaching maturity. The remaining challenges are to increase the number of ready-made standard parts, improve the techniques for selecting and retrieving parts, and develop standard procedures to be applied. Looking further ahead, the need for knowledge of particular application subdomains has been recognized. As the first modest steps, a dictionary is referred to in order to convert data item names into daily use business terms in reverse engineering, and a knowledge base is used to understand requirement definitions and to retrieve appropriate patterns of functional designs in forward engineering. The phrase 'object-oriented' has come into fashion, and a large number of R&D efforts are currently being made to take advantage of this approach, particularly for software reuse--initially for building user interfaces, but increasingly for developing industrial and business applications. It is predicted that OO 4GLs with class libraries will be developed for some application areas in the near future. Furthermore, with the increase in heterogeneous distributed environments, technologies for invoking and sharing software pieces located on different machines, and the management of remotely scattered development work products, will be expanded and enhanced further. The authors feel that continued efforts to extend conventional technologies and to utilize new o n e s - - p a r ticularly for visualization, knowledge engineering, and object-oriented development--will bring significant advancement in sharing and recycling of expensive software assets, partially solving the software crisis problem in Japan in the next 10 years. The blending of Western and Japanese approaches in effective ways will continue to bear fruit.
REFERENCES 1 Tajima, D and Matsubara, T 'Inside the Japanese software industry' IEEE Computer Vol 17 No 3 (March 1984) pp 34-43 2 Aoyama, Met al. 'Design specification in Japan: tree-structured charts' IEEE Software Vol 6 No 3 (March 1989) pp 31-43 3 Arima, N and Ooi, F 'Industrializing software development: a Japanese approach' IEEE Software Vol 6 No 3 (March 1989) pp 13-21 4 Cusumano, M A 'The software factory: a historical interpretation' IEEE Software Vol 6 No 3 (March 1989) 5 0 h n o , Y 'Current status of software engineering in Japan' J. Inform. Proc. Vol 14 No 3 (December 1991) pp 232-236 6 Matsumoto, Y and Ohno, Y A Japanese perspective in software engineering Addison-Wesley (1988) 7 Cusumano, M A Japan's software factories: a challenge to US management Oxford University Press (1991) 8 Takeshita, T CASE: computer-aided software engineering (in Japanese) Kyoritsu Shuppan, Tokyo (1990) 52
9 Takeshita, T Software maintenance~re-engineering and reuse (in Japanese), Kyoritsu Shuppan, Tokyo (1992) 10 Hall, P A V 'Overview of reverse engineering and reuse research' Inf. Soft. Technol. Vol 34 No 4 (April 1992) pp 239-249 I1 Hanada, S Program design diagramming methods (in Japanese), Kikaku Center, Tokyo (1983) 12 Okubo, M 'Architecture of database design support tool, DBprompt' (in Japanese) Proc. 44th Nat. Conf. IPSJ (March 1992) pp4.237-238 13 Honmura, T 'Development of RESCUE: a CASE for maintenance' (in Japanese) Nikkei Computer No 268 (November 1991) pp 169-186 14 Kanno F Software production methodologies (in Japanese) Nikka-giren Shuppan-sha, Tokyo (1987) 15 Kataoka, M Software reuse technologies (in Japanese), Nikka-giren Shuppan-sha, Tokyo (1989) 16 Yoshino, M, Tanigachi, K and Nishiyama, I 'Distributed 4th generation language system EAGLE/4GL' (in Japanese), Proc. 42nd Nat. Conf. IPSJ (March 1991) pp 5.345-346 17 Komiya, S and Harada, M 'Automatic programming by fabrication of reusable program components' (in Japanese) J. Inform. Proc. IPSJ, Vol 28 No 10 (October 1987) pp 1329-1345 18 Kataoka, M Software modelling (in Japanese), Nikka-giren Shuppan-sha, Tokyo (1988) 19 Ando, J, Tsuda, M and Hogaki, H 'EAGLE: effective approach to achieving high level software productivity' (in Japanese), Proc. 33rd Nat. IPSJ (October 1986) pp 619-620 20 Kaneko, Set al. 'Technology for development and reuse of software parts' (in Japanese) Toshiba Review Vol 41 No 8 (August 1986) pp 677-680 21 Nakata, N, Nishijima, A, Konishi, Y and Kubo, T 'Sigma tool implementation and integration' Proc. Info. Japan '90 IPSJ (October 1990) pp 449-456 22 Akima, N 'SIGMA (software industrialized generator and maintenance aids), project' J. Inform. Proc. Vol 14 No 3 (December 1991) pp 267-271 23 Kato, H Introduction to CASE: examples using S D A S (in Japanese), Kyoritsu Shuppan, Tokyo (1990) 24 Nabeshima, I 'Productivity improvement by reuse of parts' (in Japanese), Proc. 42nd Nat. Conf. IPSJ (March 1991) pp 5.281-282 25 Suda, M, Kawamata, Y, Yanagi, K and Takahashi, Y 'Design support environment for realtime software system CADRIS--a method of storing software specification' (in Japanese), Proc. Nat. Conf. IPSJ (March 1992) pp 5.205-206 26 Chigira, E et al. 'Software development by reuse of system specifications (ICAS-REUSE)' (in Japanese) Hitachi Hyoron Vol 69 No 3 (March 1987) pp 249-254 27 Kuroki, H and Isoda, S 'Functionality of integrated CASE SoftDA and its evaluation' (in Japanese), SE SIG of IPSJ 83-3 (February 1992) pp 17-24 28 Yasuda, T et al. 'Requirement analysis support method incorporating/based on reuse of terminology knowledge' (in Japanese), Proc. 43rd Nat. Conf. IPSJ (March 1991) pp 5.177-178 29 Inoue, H et aL 'JOKER '91: requirement analysis support system' (in Japanese), Proc. 44th Nat. Conf. IPSJ (March 1992) pp 5.333-334 30 Noguehi, K et ai. 'Basic technology for the operating system (MIOS 7/AS)' (in Japanese), Hitachi Hyoron Voi 71 No I 1 (November 1989) pp 57-64 31 Rekimoto, J e t ai. "Canae: a high-level software platform for user-interface development using editor parts' (in Japanese), J. Inform. Proc. IPSJ Vol 31 No 5 (May 1990) pp 602-6 ! i 32 Kitamura, M e t al. 'GhostHouse: a C + + GUI toolkit' (in Japanese), Proc. 44th Nat. Conf. IPSJ (March 1992) pp 5.27-28 Information and Software Technology
N MII A N D T T A K E S H I T A
33 Yanagi K et aL 'System architecture of an interactive presentation platform' (in Japanese), Proc. 44th Nat. Conf. IPSJ (March 1992) pp 6.269-270 34 Yuura, K et al. 'ODETTE: an OO application builder' (in Japanese), Proc. 44th Nat. Conf. IPSJ (March 1992) pp 5.37-38
Vol 35 No I January 1993
35 Owaki, T et aL 'Hyper programming by object-oriented formula (Hyproof)' (in Japanese), Proc. 43rd Nat. Conf. IPSJ (October 1991) pp 5.431-432 36 Yamazaki, K et aL 'Office application development environment--OO form management" (in Japanese), Proc. 44th Nat. Conf. IPSJ (March 1992) pp 5.195-196
53
It is generally recognized that software re-engineering and reuse technologies are vitally important and a highly effective means of alleviating the so-called 'software crisis" The paper discusses various methodologies and tools in a review of related research and development efforts as well as commercialized products, mainly in Japan, and places them in a consolidated perspective, giving some examples of those items considered as particularly interesting from a Japanese point of view. In Japan, the concept of treating reusable pieces of software as parts has been very effectively employed, and tree-structured graphical charts are widely used in both forward and reverse software engineering. An increasing emphasis is on the use of domain knowledge and object-oriented methodologies. software maintenance, software restructuring, software reverse engineering, software re-engineering, software reuse, software skeletons, software parts, repository, knowledge bases, domain knowledge
Research and development projects on software development technologies and CASE tools conducted in Japan have been introduced to English-speaking readers by several papers 1-5 and books 6'7. According to Professor Ohno 5, the current main subjects in Japanese software engineering are CASE, object-oriented methodology, repositories, user interface management systems, software processes, and groupware. For a long time the primary author of this paper, N. Mii, has been seriously interested in and strongly advocated the reduction of both the hardware and software development cycles. The co-author, T. Takeshita, has surveyed CASE technologies in the United States, Europe and Japan, and in 1990 published a book on the subject s. Following this, he pursued studies of software maintenance/re-engineering and reuse, which led to the publication of another book 9 two years later. As it has been fully recognized that the two most effective means of shortening the software development and maintenance cycle are re-engineering and reuse of software, Mii and Takeshita decided to work together to explore currently used technologies as well as the state of the arts in these two areas. This paper focuses on the situation
Vice President Office,LAB-007,APTO, IBM Japan, 1623-14,Shimotsuruma, Yamato City, Kanagawa Pref. 242, Japan *Collegeof Business Administration and Information Science,Chubu University 1200,Matsumoto-cho,Kasugai City, Aichi Pref. 487, Japan Voi 35 No I January 1993
in Japan, thus minimizing any overlap with the contents of a recent paper by Hall 1°. Before proceeding to the subject matter, listed below are those software engineering technologies the authors consider to have made great progress in Japan in the past two decades: • • • • • •
Development processes and management. Quantitative measurement and control. Graphics (tree-structured charts). Testing methods and tools. Software reusability and reuse. Integrating tools and interfaces.
In general, Japan has devoted considerable effort to implementation-phase (lower CASE) technologies for software development 7. The authors feel that the combination of re-engineering and reuse technologies is a powerful answer to the question of how to improve significantly both the quality and productivity of software production and maintenance. According to a survey made in 1986 by Capers Jones, a US software consultant, the ratios of common functions contained in application, tele-communication, and systems programs are 75%, 70%, and 50% respectively. Similar statistics have been reported in Japan. For example, according to a paper presented by Fujitsu in March 1984, 10 common paradigms were used in 80% of code in 100 electronic switching service modules. Dynamically changing business environments, including the need for higher usability and quality, diversification and complication of consumers' demands, globalization of business operations, and so forth, as well as the advance of hardware and communication technologies, are resulting in smaller and heterogeneous network computing, and making business re-engineering (re-design of business processes) an essential and frequently repeated task. Unlike in the USA, the lifetime of a majority of application systems is considered to be less than five years in Japan. Software re-engineering technologies can play an effective role in facilitating this re-engineering. The proportion of costs in enterprises for software development and maintenance is now much greater than that for hardware. There are astronomical quantities of software assets (estimated at well over 100 billion lines of code in the USA alone). The life of these assets needs to be prolonged, and their components should be reused or recycled as much as possible.
0950-5849/93/010045-09 O 1993 Butterworth-Heinemann Ltd
45
Software re-engineering and reuse from a Japanese point of view The vital importance of re-engineering and reuse has been well recognized since the late 1970s, and related technologies have been steadily developed and put to practical use in Japan. Most of the original ideas came from the USA and Europe but have evolved and been refined in pragmatic ways by Japanese software engineers and practitioners. The paper includes reviews of major R&D efforts along with marketed products and will, we hope, serve to shed light on recent trends and directions in these software engineering fields in Japan.
SOFTWARE RE-ENGINEERING TECHNOLOGIES The term 'CASE' began to be used toward the end of 1986, and various CASE tools and environments have been developed and marketed for several years, but their utilization and effect on productivity improvement do not seem to be as impressive as expected. Why should this be? So far, CASE tools have been designed primarily for the development of new (application) s o f t w a r e - - t h a t is, in forward engineering. However, in large enterprises a greater part of DP resources (80% or even more) is spent on the maintenance or re-engineering of existing code, so the improvement of new application development does not contribute much to the reduction of the total workload or cost in the DP department. According to a survey conducted by Nikkei Computer (a technical magazine published in Tokyo) in the middle of 1990, 47.3% of the DP cost in 396 Japanese companies that responded to the questionnaire was spent on maintenance of software. For this reason, people have currently begun to pay much closer attention to software re-engineering technologies, as well as to software reuse, as these technologies not only reduce the maintenance workload and cost, but also facilitate business re-engineering (re-design of business processes) by adapting the processes to new and changing business environments and customer needs. Moreover, it has been admitted by experts that, even with best-of-the-breed tools in a well-integrated CASE environment, it is difficult to improve productivity by an order of magnitude without extensive reuse. Software re-engineering technologies, if used for re-engineering of business processes, help rejuvenate some or all of the components of existing applications-source code, file and database structures, screen and report definitions, and so on. This eliminates or reduces the need to develop new software from scratch. Therefore, the authors believe that reverse engineering and reuse technologies help enterprises both to derive maximum benefit from previous efforts and to reduce the amount of entirely new efforts, thus considerably shortening the software development and maintenance cycle and greatly improving its quality. 46
Re-engineering at the source code level If a design specification is represented in a structured programming style using a pseudo-natural language--in Japan a mixture of Kanji (Chinese characters) and Kana (phonetic symbols)--then its contents are more easily understood than otherwise, and source statements can be changed and/or added with much less effort than for unstructured code. Japanese computer manufacturers and Nippon Telephone and Telegraph Corporation (NTT) have made enormous efforts in developing graphical notations for constructs in structured programming. The graphical representations are often called 'tree-structured charts '2"''. The following different kinds of notation have been developed in Japan: • Hitachi: PAD (Problem Analysis Diagram). • NEC: SPD (Structured Programming Diagram). • NTT: HCP (Hierarchical and ComPact description chart). • Toshiba: T F P / T F D (Technical description For Procedure/Data design). • Fujitsu: YAC II (Yet Another Control chart II). Though a similar notation called 'Greenprint' was created at IBM's T J Watson Research Center toward the end of 1970s, and Action Diagrams are currently supported by a few CASE environments developed in the USA, Japan is further advanced in utilizing treestructured graphical representations in software development and maintenance than the rest of the world. Tree-structured graphic notations were developed in Japan from the late 1970s to the beginning of the 1980s. Once a detailed design has been completed, using one of these representations, source code in a 3GL is automatically generated. Reverse conversion from source code to a graphical representation is also automated, which eliminates both the need to keep detailed design specifications and also the inconsistency between the source code and design specifications. It is noted that the tree-structured representations are also used for design review, code inspection, and debugging by animation. Until the debut of powerful workstations with large, high-resolution screens with GUI in 1987, many people were sceptical about the practicality of tree-structured charts, as it is cumbersome for a human to create and maintain them, and they consume too much expensive space on display screens. These problems were resolved with the development of high-resolution, colour graphics on PCs and workstations. This kind of visual programming, though at a low level, allows better and faster design and modification, as well as enforcing design and programming standards. CASE tools are provided to convert source c o d e - - m o s t l y in Cobol, Fortran, or C - - i n t o tree-structured charts containing labels and the names of functions and procedures as well as variables in Japanese. For example, Fujitsu's YPS (YAC II Programming System) converts source code into YAC II charts, Hitachi's SEWB2 (Software Engineering Workbench 2) converts it into PAD, and so forth. IBM's Information and Software Technology
N MI1 AND T TAKESHITA
Tokyo Research Laboratory has developed a research prototype called 'RE/Cycle' which includes the capability to convert Cobol and PL/I source code into Action Diagrams.
Extraction of higher-level design information The design information contained in a tree-structured chart is almost at the source code level. Several research efforts have been made to obtain higher-level design information. An early example is an NEC tool called SYSDES (System Design Data Generation), which generates system design information from source code, a set of Job Control Language statements, and a set of parameters for menus. A paper on this was presented in October 1986. It is a component of NEC's SEA/I (Software Engineering Architecture/l), the initial version of which was released in 1983. In 1991 Fujitsu released a product called YPS ReEngineering. It generates from Cobol source code not only YAC II charts, but also hierarchical charts showing the relationships of routines. RE/Cycle converts spaghetti code in Cobol or PL/I into structured code, using algorithms based on the graph theory, and produces a structure chart containing not only who-call-whom relationships of routines or procedures, but also information on the dataflow between a parent and child boxes (routines or procedures). It also has an extensive impact analysis capability. Fujitsu Laboratories is reportedly developing a tool to generate detailed specifications from Cobol source code by referring to a data dictionary and program pattern library. It is currently applied to financial applications, whose programs often have common patterns.
Re-engineering of data (files and databases) Although programs developed in Japan in the past decade or so are mostly structured, conforming to some kind of installation standard, software maintainers still suffer from insufficient or out-of-date documentation on data. One problem is the existence of homonyms and synonyms used as names of data items. Tools are needed to detect these and help the user to resolve them by changing the names so that they conform to a suitable rule. A good naming convention should be established, and names used in old programs should be changed to follow it when programs containing them are maintained or rejuvenated. Another and possibly much bigger problem is the difficulty of understanding the way in which existing files and databases are organized. It is highly desirable that documentation of their conceptual and logical design should be automatically generated from their existing physical definitions and that they should be semi-automatically converted to accommodate new requirements, with subsequent changes, if necessary, to programs that have access to them. N T T Software Laboratory has developed a database design support tool called DBprompt '2, which consists of Vol 35 No 1 January 1993
the Information Resource Dictionary System and tools to support data name standardization (NAME), database conceptual design (ANLZ), logical design (DSGN), physical design (GNTL), and design information retrieval (RTRV). D B p r o m p t / N A M E checks the names of data items in data bases, eliminates synonyms and homonyms, and standardizes the names, using a naming rule and a terminology dictionary (containing 1296 words). In most cases, the following rule holds: A data item name = a qualifying word + a main word + a subdivision word The database design information is extracted from the database design sheets (of the existing databases) by RTRV. The extracted data names are standardized by using NAME. The forward engineering (logical and physical design) is performed by using D S G N and GNTL.
Re-engineering at the application system level Although the understanding of an individual component or module of each application system is essential for its maintenance and redesign, it has been found that all the components of an application system must be reviewed, and syntactical and semantic analysis must then be performed, so that the relations and dependencies among files, transaction messages, screen panels, reports, source programs and so on can be grasped and placed into the right contexts. The following items of information are required: • File and database organizations. • Record layouts and formats; transaction message formats. • Report formats; screen formats and screen transitions. • Relations and dependencies among modules, procedures, and subroutines/functions. • Relations between files/databases and program modules. • Relations between screens and program modules. • Potential impact of changes and of change history. In general, these are neither documented nor stored systematically, but are very often vaguely stored in the memory of designers and programmers. Some limited CASE capabilities addressing this need have recently been provided by Japanese computer manufacturers. A comprehensive CASE environment called RESCUE (Reverse Engineering System for Comfortable User's Environment), with a strong focus on software maintenance, has been developed by a Tokyo software house, CASE Technology '3. It consists of three categories of function: (l) static analysis (entry into repository and retrieval for maintenance), (2) dynamic analysis (acquisition and retrieval of tracing information), and (3) quality assurance (generation and comparison of test data). Not only programs, but also other application components such as JCL statements,
47
Software re-engineering and reuse from a Japanese point of view
and databases, screen panels, COPY and system definitions are analysed, and the extracted information is stored in master files by control category. Business terms in Japanese are attached to the objects to be looked at, enabling the user to make references without knowing the mnemonic names used in an implementation. Additional information, such as relations between file names and data set names and between panel names and program names, can be added by the user. Various cross-reference tables are generated.
SOFTWARE REUSE TECHNOLOGIES The reuse of software has been realized in different ways and at different levels '4"'5. The most common and takenfor-granted approaches are the use of subroutines, functions, macro routines, and generators. The smaller the size of software pieces, the greater the chances of their repeated use. However, because of their high granularity and the relatively large overhead of linking them to the invoking routine, programming productivity cannot be greatly improved in general by the existing approach. What is required for a large gain is much less extra work on the part of programmers in invoking many short sequences of statements in a target language, or the reuse of software of proven quality in a module or at the component level, especially the reuse of design specifications rather than source code. In Japan, the concept of treating reusable pieces of code as software parts began to be implemented as a key ingredient of integrated software development environments toward the end of the 1970s, and the number of papers on this topic reached a peak in the middle of the last decade. Although software parts are designed primarily for professional programmers of commercial and process control applications, 4GLs and their generators are provided to enable non-DP professionals or novice programmers to write commercial applications in much less time. We first take a brief look at the 4GLs that have been developed in Japan.
4GLs and generators A number of R&D and product development efforts have been devoted to 4GLs and their generators. It is particularly noteworthy that many of them have precoded standard units that can be selected and incorporated into generated code. The following are some of the 4GL products marketed by Japanese computer manufacturers. • Fujitsu: Hyper Cobol, CASET, BAGLES II, STRACT. • Hitachi: CORAL, EAGLE/4GL. • NEC: IDL II. Hyper Cobol is a typical case of enabling the reuse of relatively large software parts, as it has I i precoded standard units: Edit, Report, Check, Sort, Merge, 48
Summarize, Break, Separate, Chain, Match, and Control. It supports a dataflow concept, and a source program generally consists of record definitions similar to Cobol record descriptions, Stream Areas (I/O definitions), and some of the standard units listed above. Cobol source code can be added. CASET (Computer Aided Software Engineering Tools) is also a Cobol generator. It consists of FORM (screen and report definition functions with prototyping capability), FILE (for file definition), CAPE (Customer Application Program Editor), which is used for editing a dataflow-oriented language, and 13 categories of standard patterns (such as billing entry, record selection, separation, summary report printing, and merging) to be used for generating Cobol source code. BAGLES II (Business Application Generator's Library for Extensive Support II) generates Cobol code from a table-format business specification. BAGLES was originally designed for banking applications. STRACT (STRategic interACTive information system) is an end-user-oriented language that has access to relational databases. CORAL (Customer-Oriented Application Development System) allows the use of a pseudo-Japanese language for design specifications, and provides flexible decision tables, dialogue procedures and menus, a dictionary, macros, and source level debugging. EAGLE (Effective Approach to achieve hiGh-LEvel software productivity)/4GL is a Cobol generator for small general-purpose systems. It provides three kinds of process pattern (file creation/update, file matching, and interactive editing), and allows the use of Japanese words in source statements and definitions of data. It can now be used in a distributed environment '6. IDL II (Integrated Data-oriented Language II) receives design information created by SPECDESSIN (a system analysis and high-level design tool), which supports the dataflow diagrams. It then generates Cobol code including commonly used patterns.
Reuse at the source code level Bread-and-butter business applications, such as payroll management, inventory control, and sales accounting, contain common process patterns such as sorting, merging, and updating of files. The names of files and data items, and the internal processes (computation algorithms) may vary from application to application. For the repeated use of common patterns, these names and the processing logics are left blank, to be filled in for each new application. Each blank field needs an associated indication of what should be entered. Such incomplete source code modules or routines can be provided with proper explanations and supported by retrieval, customization, and synthesis/fabrication mechanisms in a CASE environment t4'7''8. Kanno '4 contains detailed discussion of 'module design technologies for reuse', 'standard platform and parts', and 'reuse-oriented software development environments and software production control technologies'. Information and Software Technology
N MII AND T TAKESHITA
The patterns are customized to create application skeletons, into which common software parts and new coding pieces are to be inserted by a synthesizer or fabricator. A customization aid helps the user to make changes to fit his particular need. An integration mechanism receives as input completed modules, and checks the consistency of data names and attributes and the completeness of other interface information before combining them into a single integrated module or a set of modules• A module constructed to allow the user to look into its inside and make changes to it is called a 'white box'. This approach makes it possible to use existing pieces of code whose granularity is larger than that of library functions or subroutines, which are mostly of a blackbox type. With a minor risk of degradation of quality, it is an easy-to-use approach in some application domains such as routine business application systems. To facilitate understanding of the internal functions and algorithms/logics, it is desirable that the module should be displayed in a graphical (for example, treestructured) representation. Such attempts have been made by Japanese computer manufacturers and NTT. As early as 1981, Fujitsu made available a product called P A R A D I G M , which is a set of standard patterns (in which input checking, matching, updating, and report generation are the primary paradigms) to be used for commercial application development. It also provides facilities that allow the customer to create his own software parts. NEC's SEA/1 was designed for the development of commercial applications in Cobol, initially on office systems, and its first version began to be used for production in 1983. It handles the three types of software parts: • Data definition parts: portions of the Cobol Data Division that define the formats of output reports• • Process parts: units referenced by P E R F O R M or C A L L statements (in the Procedure Division). • Skeleton parts: program control structures and information on parts to be used. Each SEA/I part consists of a header and body, and is combined with other parts by a program synthesizer. As components of SEA/l, NEC developed a pattern customization control language (PCL) for creating patterns of high generality, and a parts description language (PDL). According to a paper presented in September 1984, standard parts and their numbers initially provided by NEC were standard file I/O (10), table handling (13), item checking (3), item editing (15), data processing (17), time processing (3), code conversion (11), calculation (4), unit conversion (5), and code conversion, sorting, and so on (8). The number of different parts in each category is shown in parentheses. In 1987 NEC announced that 'SEA/I' now meant a combination of a methodology (STEPS) and supporting tools (SOFPIA, IDL II, TQF II), and in the following year it made available SOFPIA (SOFtware Productivity Improvement Aid) and PWB (Personal Work Bench). Vol 35 No 1 January 1993
The former includes tools for data definition, program synthesis, testing, system installation, and product management, while the latter supports data (screen layout) definition and program generation on a workstation. Hitachi's EAGLE (released in March 1984) and EAGLE2 (September 1985) also support reuse of software parts, initially with 22 standard patterns and 16 parts for batch processing, and 9 patterns and 60 parts for on-line processing. According to the results of a survey on the use of EAGLE by Hitachi customers over a period of more than two years from March 1984 '9, standard patterns were applied to 80% of batch applications and 70% of on-line applications, and the ratio of patterns and parts to the generated program in size (LOC) was 60%, the average productivity improvement was by a factor of two and the number of errors was reduced by two-thirds• Similar data have been reported by other manufacturers, indicating the effectiveness of this software reuse approach. The reuse of software parts for Cobol applications is also supported by Toshiba's MYSTAR (released in 1985). It has four types of skeleton (white box) library: individual user, project, global, and auxiliary. It includes various tools for functions such as listing, storing, retrieving, and deleting. One way to increase the reusability of source code is to limit the size of a module to one page with its functions self-contained, minimizing its interface complexity for easier and simpler linkage. An example of this approach is 50SM (50 Steps/Module, meaning 50 lines of code per module), which was initiated as part of the IMAP (Integrated software Management And Production support system) project at Toshiba 2°. It is a parts-oriented design/coding technology, on which a paper was first presented in March 1985.50SM supports development activities from module design through system integration, and includes T F F (Technical description Formula for Fifty SM design) and TFP/TFD. As an early example of the effectiveness of TFF, Toshiba reported in September 1985 that more than 30% reuse in new software had been achieved by a group of 10 developers. The SIGMA project also produced tools for software parts such as a pattern customizer, a skeleton generator (to create skeletons from tree-structured charts), a parts compiler, and a program synthesizer2''22 Fujitsu's YPS/APG (YPS-based Application Program Generation system), which was released in 1989, includes an editor for YPS (YAC II Programming System), a decision table, an editing table, generation of YPS, referencing, and copying; the replacement by specified character strings at the time of synthesis; and the reuse of screen and report layouts and on-line screen trans,t,ons-. According to a paper presented in March 199124, the advantages of parts reuse include the reduction of statements for unique processes (not using software parts) to between 20% and 30% of the total program, the narrowed scope and increased efficiency of 49
Software re-engineering and reuse from a Japanese point of view
design review, and the elimination of unit tests, resulting in productivity improvement by a factor of 1.3 to 1.5. Judging from these experiences, the extension of this kind of source code reuse technology will double the total productivity of application development and will increase the maintenance productivity by several times in many cases.
Reuse of the design specification level In general, the reuse of specifications at the functional or system level can be realized in some application subdomains where application systems consisting of similar inputs, outputs, and processes are developed. Programming elements that vary from system to system or from user to user must be identifiable. Supposing that such subdomains exist, development processes as well as design representation must be standardized to enable the reuse of design specifications. In some cases, customizable packages are developed and provided to different customers who tailor them to their needs by adding their unique characteristics. This can be regarded as one way of reusing design specifications. If design specifications are represented in a very high-level textual language or in a table format in which those items that can be changed or inserted are left blank with some indications, then it is easier to reuse them. Another way of reusing design specifications is to provide a method and supporting tool to help the user to understand the requirement definitions and to search for a design specification that best satisfies the given requirement. In Japan, papers on reuse of specifications began to appear in 1984. In a paper presented in September 1985, Hitachi discussed the creation and reuse of specifications, including data flow diagrams on SEWB, which is connected to EAGLE on the host system. CADRIS (Computer-Aided Design and Reuse with Intelligent Support system), developed at Hitachi Omika Works provides an FIF (Fill-in-the-Form) method using table-format worksheets and a guidance method to facilitate the entry of design specifications, and supports the retrieval of design specifications from a database, which is based on the E-R model25.
USE OF APPLICATION DOMAIN KNOWLEDGE Papers on the use of knowledge bases or expert systems for software development and maintenance began to appear in Japan in 1983, when Hitachi presented a paper entitled 'Two-level method for finding software parts using knowledge'. NEC presented a paper on 'Automatic program generation system using knowledge' in 1985. The annual number of papers on the use of knowledge bases in software engineering then gradually 50
increased, reaching 10 in 1988 and peaking at 20 in the following year. Without application domain knowledge, it is not possible to represent extracted design information by using terminology and expressions familiar to non-DP professionals in such a way that they can readily understand the processing logics and the business rules embedded in the source code as well as the meanings of the screen panels, file and database structures, and report formats. If frequently used process patterns (represented in a pseudo-natural language) that perform business rules or their refinements and their corresponding process patterns (in incomplete source code) are stored in a knowledge base, and ifa mechanism that interactively supports searching for and retrieving the patterns is provided, then this knowledge base can be used for both designing in forward engineering and design recovery in reverse engineering. Such a knowledge base should contain business terms and corresponding mnemonic names and processing logics to be used in source code. According to a paper printed in the Hitachi Journal in 1987, Hitachi established a basic technology for applying knowledge engineering to software reuse 26. Requirement definitions are entered in Japanese. Keywords for retrieving specifications are automatically extracted. Words not needed for software development are removed. Required data and functions are identified by inference, resulting in the addition of related keywords. By using the obtained keywords, a specification is retrieved and modified for new, different features and converted into source code by using SEWB (Software Engineering Work Bench) and EAGLE2. Mitsubishi Electric has developed a research prototype, SAGE (System for Automatic program GEneration) which infers a process pattern from a requirement specification by using domain knowledge, and generates a batch file processing program. NEC's PGEN (Program Generator), which was commissioned by the Institute for New Generation Computer Technology (ICOT), generates source code by understanding the meaning and context of detailed specifications prepared by a system analyst. This system is based on natural language understanding and knowledge engineering, and deals with detailed semantic analysis, heuristics (essentially rules of thumb) for understanding dependency relationships, and multiplicities of meanings. Initially PGEN worked only with detailed specifications. It is hoped that the system will be extended so that it can work with parts of programs such as common subroutines and automatically adapt them to a new application, rather than simply doing lineby-line coding. SoftDA (Software Design Automation), developed by NTT, is a CASE environment with knowledge database 27. It converts Japanese names used in pseudo-natural language statements into predetermined, standard mnemonic names by referring to a vocabulary in a domain knowledge library. Conversely, it can generate detailed design specifications in pseudo-Japanese from Information and Software Technology
N MI1 AND T TAKESHITA
source code, with the mnemonic names replaced by business terms. The Domain Knowledge Database contains data names, their semantic information, their mnemonic names, process patterns, and corresponding source code patterns. In 1991 Toshiba presented J O K E R (JapaneseOriented Knowledge Engineering/Environment for software Reuse), which accepts a requirement specification in Japanese, represents it in a formal DFD, and verifies and evaluates it 28. It was expanded into J O K E R '91, which acquires knowledge about the terms used in a requirement specification written in Japanese, and organizes, stores, and represents the knowledge on the basis of an object model, increasing the reuse of application domain know-how and enabling this to be linked to object-oriented analysis and design 29. Hitachi is working on an experimental software reuse development system based on co-operative problem solving using a knowledge base. This system was also commissioned by ICOT. As a unique product, Hitachi is marketing an intelligent business document recognition and application generator called 'BELIEVE' for an office system 3°. Formats and character sizes are recognized from an existing document (which may have been created on a word processor) by an image reader and displayed on the screen in such a way that modifications can be made. A generated program is in the same format as one created by using ETOILE (Effective Object-Oriented language for End Users on Office Processor), and so can be easily modified. BELIEVE uses a knowledge dictionary in which recognition and program generation rules are stored. It contains word and phrase dictionaries, each of which consists of the attribute/property area and the rule (range and calculation) area.
OBJECT-ORIENTED
TECHNOLOGIES
In Japan, several research and development projects on OOPL and OODB have been conducted. Although Object-Oriented ( 0 0 ) analysis and design methods have not yet become well established or standardized, some attempts to use them and their results have been reported. First, we will look at R&D on user interface builders making use of O 0 technology, and then at the CASE environments using or supporting this technology.
User interface builders NEC has developed Canae, a high-level software platform for user interface development using editor parts 3'. The system runs as a client process under X-Windows and consists of two kinds of parts: • Interactive parts: buttons, menus, scrolling bars, iconic menus, and so forth. • Editor parts for editing Japanese text, figures, image, graph structures, hierarchical structures, and tables. Vol 35 No 1 January 1993
An OO approach has been adopted to implement the editors based on a modified version of the MVC (ModelView-Controller) model designed for Smalltalk-80. Mitsubishi has developed a prototype of a C + + library called GhostHouse 32. It is designed for the XWindows System, using Xt (for resource management and event control) and the widget of OSF/Motif (for standard parts). Hitachi System Development Lab has developed an interactive presentation application development platform using an OO approach 33. It has been built by using the OSF/Motif toolkit on the X-server under HI-UX (Hitachi's UNIX). This has reduced by four-fifths the time required to create presentation packages on car sales. Hitachi Central Laboratory and its two subsidiary companies have developed an application builder called O D E T T E as a design support tool. This allows the entry, manipulation, and observation of objects through easyto-understand figures on the basis of the Common Lisp Objects System 34. CASE
environments
In 1991, Hitachi Omika Works developed Hyproof (Hyper-Programming system by OO Formula), a 4GL system for control system software development 35. Its objectives are (1) to accumulate and reuse functional and program specifications, and (2) to allow development of software without knowing how it will be implemented. Its basic approaches are modelling of the process of functional specification by domain, extraction of reusable program parts from application programs, and employment of a domain-oriented software reuse process. NEC C&C System Research Lab and C&C System Engineering Division have developed a prototype of an office application development environment using an OODB (Odin) 36. A business application system consists of three types of processing: user interface, database access, and business logic. Data structures are defined as instance variables, and the display methods and item relationships with the business database are defined as class variables of the form objects. Data structures and procedures can be inherited. This system can be considered as an OODB/4GL. Form views are implemented on OSF/Motif, but the system can be applied to other GUIs by changing the form view. CONCLUSION Software re-engineering and reuse, if properly designed, planned, and used, help enterprises to derive maximum benefit from past development efforts and/or to prolong the life of software artefacts, reduce the software development and maintenance cycle and, most importantly, improve productivity and quality in software production and maintenance by an order of magnitude. In Japan, lower CASE technologies (which can be considered as product engineering technologies) have 51
Software re-engineering and reuse from a Japanese point of view
advanced with the visualization of detailed design, the utilization of the concept of software parts, and so on. Technologies for re-engineering and reuse of detailed design and source code appear to be reaching maturity. The remaining challenges are to increase the number of ready-made standard parts, improve the techniques for selecting and retrieving parts, and develop standard procedures to be applied. Looking further ahead, the need for knowledge of particular application subdomains has been recognized. As the first modest steps, a dictionary is referred to in order to convert data item names into daily use business terms in reverse engineering, and a knowledge base is used to understand requirement definitions and to retrieve appropriate patterns of functional designs in forward engineering. The phrase 'object-oriented' has come into fashion, and a large number of R&D efforts are currently being made to take advantage of this approach, particularly for software reuse--initially for building user interfaces, but increasingly for developing industrial and business applications. It is predicted that OO 4GLs with class libraries will be developed for some application areas in the near future. Furthermore, with the increase in heterogeneous distributed environments, technologies for invoking and sharing software pieces located on different machines, and the management of remotely scattered development work products, will be expanded and enhanced further. The authors feel that continued efforts to extend conventional technologies and to utilize new o n e s - - p a r ticularly for visualization, knowledge engineering, and object-oriented development--will bring significant advancement in sharing and recycling of expensive software assets, partially solving the software crisis problem in Japan in the next 10 years. The blending of Western and Japanese approaches in effective ways will continue to bear fruit.
REFERENCES 1 Tajima, D and Matsubara, T 'Inside the Japanese software industry' IEEE Computer Vol 17 No 3 (March 1984) pp 34-43 2 Aoyama, Met al. 'Design specification in Japan: tree-structured charts' IEEE Software Vol 6 No 3 (March 1989) pp 31-43 3 Arima, N and Ooi, F 'Industrializing software development: a Japanese approach' IEEE Software Vol 6 No 3 (March 1989) pp 13-21 4 Cusumano, M A 'The software factory: a historical interpretation' IEEE Software Vol 6 No 3 (March 1989) 5 0 h n o , Y 'Current status of software engineering in Japan' J. Inform. Proc. Vol 14 No 3 (December 1991) pp 232-236 6 Matsumoto, Y and Ohno, Y A Japanese perspective in software engineering Addison-Wesley (1988) 7 Cusumano, M A Japan's software factories: a challenge to US management Oxford University Press (1991) 8 Takeshita, T CASE: computer-aided software engineering (in Japanese) Kyoritsu Shuppan, Tokyo (1990) 52
9 Takeshita, T Software maintenance~re-engineering and reuse (in Japanese), Kyoritsu Shuppan, Tokyo (1992) 10 Hall, P A V 'Overview of reverse engineering and reuse research' Inf. Soft. Technol. Vol 34 No 4 (April 1992) pp 239-249 I1 Hanada, S Program design diagramming methods (in Japanese), Kikaku Center, Tokyo (1983) 12 Okubo, M 'Architecture of database design support tool, DBprompt' (in Japanese) Proc. 44th Nat. Conf. IPSJ (March 1992) pp4.237-238 13 Honmura, T 'Development of RESCUE: a CASE for maintenance' (in Japanese) Nikkei Computer No 268 (November 1991) pp 169-186 14 Kanno F Software production methodologies (in Japanese) Nikka-giren Shuppan-sha, Tokyo (1987) 15 Kataoka, M Software reuse technologies (in Japanese), Nikka-giren Shuppan-sha, Tokyo (1989) 16 Yoshino, M, Tanigachi, K and Nishiyama, I 'Distributed 4th generation language system EAGLE/4GL' (in Japanese), Proc. 42nd Nat. Conf. IPSJ (March 1991) pp 5.345-346 17 Komiya, S and Harada, M 'Automatic programming by fabrication of reusable program components' (in Japanese) J. Inform. Proc. IPSJ, Vol 28 No 10 (October 1987) pp 1329-1345 18 Kataoka, M Software modelling (in Japanese), Nikka-giren Shuppan-sha, Tokyo (1988) 19 Ando, J, Tsuda, M and Hogaki, H 'EAGLE: effective approach to achieving high level software productivity' (in Japanese), Proc. 33rd Nat. IPSJ (October 1986) pp 619-620 20 Kaneko, Set al. 'Technology for development and reuse of software parts' (in Japanese) Toshiba Review Vol 41 No 8 (August 1986) pp 677-680 21 Nakata, N, Nishijima, A, Konishi, Y and Kubo, T 'Sigma tool implementation and integration' Proc. Info. Japan '90 IPSJ (October 1990) pp 449-456 22 Akima, N 'SIGMA (software industrialized generator and maintenance aids), project' J. Inform. Proc. Vol 14 No 3 (December 1991) pp 267-271 23 Kato, H Introduction to CASE: examples using S D A S (in Japanese), Kyoritsu Shuppan, Tokyo (1990) 24 Nabeshima, I 'Productivity improvement by reuse of parts' (in Japanese), Proc. 42nd Nat. Conf. IPSJ (March 1991) pp 5.281-282 25 Suda, M, Kawamata, Y, Yanagi, K and Takahashi, Y 'Design support environment for realtime software system CADRIS--a method of storing software specification' (in Japanese), Proc. Nat. Conf. IPSJ (March 1992) pp 5.205-206 26 Chigira, E et al. 'Software development by reuse of system specifications (ICAS-REUSE)' (in Japanese) Hitachi Hyoron Vol 69 No 3 (March 1987) pp 249-254 27 Kuroki, H and Isoda, S 'Functionality of integrated CASE SoftDA and its evaluation' (in Japanese), SE SIG of IPSJ 83-3 (February 1992) pp 17-24 28 Yasuda, T et al. 'Requirement analysis support method incorporating/based on reuse of terminology knowledge' (in Japanese), Proc. 43rd Nat. Conf. IPSJ (March 1991) pp 5.177-178 29 Inoue, H et aL 'JOKER '91: requirement analysis support system' (in Japanese), Proc. 44th Nat. Conf. IPSJ (March 1992) pp 5.333-334 30 Noguehi, K et ai. 'Basic technology for the operating system (MIOS 7/AS)' (in Japanese), Hitachi Hyoron Voi 71 No I 1 (November 1989) pp 57-64 31 Rekimoto, J e t ai. "Canae: a high-level software platform for user-interface development using editor parts' (in Japanese), J. Inform. Proc. IPSJ Vol 31 No 5 (May 1990) pp 602-6 ! i 32 Kitamura, M e t al. 'GhostHouse: a C + + GUI toolkit' (in Japanese), Proc. 44th Nat. Conf. IPSJ (March 1992) pp 5.27-28 Information and Software Technology
N MII A N D T T A K E S H I T A
33 Yanagi K et aL 'System architecture of an interactive presentation platform' (in Japanese), Proc. 44th Nat. Conf. IPSJ (March 1992) pp 6.269-270 34 Yuura, K et al. 'ODETTE: an OO application builder' (in Japanese), Proc. 44th Nat. Conf. IPSJ (March 1992) pp 5.37-38
Vol 35 No I January 1993
35 Owaki, T et aL 'Hyper programming by object-oriented formula (Hyproof)' (in Japanese), Proc. 43rd Nat. Conf. IPSJ (October 1991) pp 5.431-432 36 Yamazaki, K et aL 'Office application development environment--OO form management" (in Japanese), Proc. 44th Nat. Conf. IPSJ (March 1992) pp 5.195-196
53