- Email: [email protected]
Emergent media environment for idea creation support
Knowledge-Based ~VSTEIYIff--ELSEVIER
Knowledge-Based Systems 10 (1997) 51-58
Emergent media environment for idea creation support Kozo Sugiyama a'*, Kazuo Misue a, Isamu Watanabe a, Kiyoshi Nitta a, Yuji Takada b ~Multimedia Systems Laboratories, Fujitsu Laboratories Limited, 140 Miyamoto, Numazu, Shizuoka 410-03. Japan bpersonal Systems lz~boratories, Fujitsu Laboratories" Limited, 2-2-1 Momochihama. Sawaru-ku, Fukuoku 814. Japan
Abstract
An interactive computer system for supporting the emergent stages (or upper streams) in human intellectual activities has been developed. In the system it is intended to integrate facilities for supporting generation, collection, organisation, and presentation of ideas and advising on the divergence and convergence of the ideas. This paper explains the facilities of the system and how to use it, and describes some examples of their use. © 1997 Elsevier Science B.V.
I. Introduction Our major concern of this paper is to develop a highly interactive computer system for supporting early, emergent and usually creative stages in human intellectual activities such as research planning, conceptual design, software requirement analysis, knowledge acquisition, problem solving, decision making, counselling, motivation and so on. We can often observe that the following steps are generally common to the emergent stages of the abovementioned activities: (i) encountering or finding a problem, (ii) generating segments of ideas and collecting piece-wise information relating to the problem, (iii) organizing them into a structured form to understand or solve the problem, and (iv) presenting the structured form as a document or a speech to an audience. The process consisting of these steps is called an emergent process in general or an idea creation process more specifically where the term idea means an understanding, insight or solutions of a problem. As an emergent process proceeds, information treated in each step is gradually elaborated from vague, informal, nonstructured (or piece-wise), personal and small-sized information to more precise, formal, structured (or wellorganised), public and large-sized information. We call such information methods to express ideas in an emergent process emergent media. We can identify the three chunks of emergent media as follows. Cards are suitable for expressing piece-wise information (memos, sketches, messages, etc.); they are mainly used in the generation and/or collection step. Diagrams are suitable for organising
a set of piece-wise information into structured forms (lists, clusters, trees, various types of graphs, etc.); they are mainly used in the organisation step. Documents are suitable for expressing more precise, formal and large-sized information (concepts, ideas, etc.); they are mainly used in the presentation step. Fig. 1 illustrates steps of an emergent process and chunks of emergent media. An emergent media environment (EME) is defined as a set of emergent media-based facilities to support the emergent process effectively on the basis of a more basic environment such as hardware, basic software, multimedia, networks and so on. To perform an idea creation process, we should specify a methodology which explains how to utilise the facilities provided by EME. In developing an experimental version of EME, we refer to the KJ method [ 11 (see Appendix) as a methodological basis. There are some other related works in this area: Marshall et al. [2] classified current spatial hypertext systems into four categories: permissive, emergent, descriptive and prescriptive. Our work is relates to the emergent spatial hypertext. Our term emergent media relates to the term knowledge media proposed by Stefik [3]. Haake et al. [4] developed a hypermedia prototype where seamless idea transformation from the informal to the formal is considered. In this paper facilities that the current EME provides are outlined. The reminder of this paper is organised as follows: Section 2 presents an overview of EME implemented and Sections 3 - 5 explain each facility in more detail. Section 6 describes concluding remarks and plans for future research.
2. Overview of current emergent media environment *
Corresponding author.
0950-7051/97/$17.00 © 1997 Elsevier Science B.V. All rights reserved PII S0950-7051 ( 9 7 ) 0 0 0 1 3 - 0
In developing the experimental EME our efforts have
K. Sugiyama et al./Knowledge-Based Systems 10 (1997) 51-58
52
DIAG~
con~ton
Organization
~UMENT
Prec,,marion
Fig. 1. Chunks of emergent media.
been focused on various dimensions. In order to attain integrated support from the generation and/or collection step to the presentation step, we have developed user-friendly interfaces for handling information in each chunk of emergent media and for flexibly converting information from one chunk to another. To concentrate on thinking itself, we developed facilities for avoiding chores and unpleasant jobs when arranging cards, drawing diagrams, etc. Computer-aided techniques have been applied for idea divergence advising in the generation/collection step and idea convergence advising in the organisation step. Networks have been employed for idea collection and methods of handling multimedia information have been implemented for idea representation in each step. Both personal use and group use should be supported. The current version is mainly for personal use, but distributed brainstorming by a group is partly supported for idea generation. The current EME has been implemented on UNIX/X-Window. Fig. 2 shows on overview of a process, tools, and facilities of the current EME [5].
3. The generation/collection step 3.1. Instant idea memorisation and transfer Ideas often come to mind suddenly and unexpectedly, for example, when we are in bed, walking on a street, driving a car, or at the desk. Unless we memorise or otherwise record the ideas instantly, they tend to disappear within a minute. Therefore an instant idea-capturing or memorising facility
Fig. 3. Instant idea memorisation and transfer.
anywhere and anytime is most important for the generation/ collection step. Further, we need to be able to easily and smoothly transfer memorised ideas to the organisation step. For this purpose we use a portable and pen-based computer called ZAURUS (commercial product by Sharp Corporation) and connect it via an infrared-light link to an idea-organising tool called D-ABDUCTOR running on a PC or workstation (see Fig. 3). Fig. 4 is an example of DABDUCTOR screens displaying idea segments written on cards. Fig. 4 shows a collection of cards with text and a photo taken with a digital camera.
3.2. Distributed asynchronous brainstorming Group works for the idea generation are classified into face-to-face meetings, distributed synchronous meetings, and distributed asynchronous meetings. The benefits of group idea generation through such meetings include an increase in the amount of available information/knowledge and viewpoints, and synergy between participants. We are
_1
Fig. 2. An overview of the process, tools and facilities of the current EME.
.............
Fig. 4. Example of cards (collected idea segments) with texts and images•
K. Sugiyama et al./Knowledge-Based Systems 10 (1997) 51 58
developing a brainstorming tool called EBICS for a distributed asynchronous situation using multi-agent programming technologies. EBICS consists of servers, clients (or participants to the meeting) and other related tools such as a database management system. Clients and servers are implemented as agents in a multi-agent programming language, and data are managed by the agent-oriented distributed database engine.
3.3. Information collection via network Retrieving or collecting information through the network (e-mail, e-news, WWW, etc.) is another useful method in this step. We have used our EME to, for example, collect and summarise users' comments on D-ABDUCTOR and responses to a questionnaire for market research. Network searchers, intelligent agents, software robots, WWWWorms, and methods for data mining and knowledge discovery will be used much more for this purpose in the near future. N u m b e r of articles Total n u m b e r of words Size o f dictionary Time for reirieval
- 200 000 - 1 5 0 000 -~300 MB 1 s - 2 rain
- 15 0 0 0 - 1 l0 0 0 0 - - 2 0 MB 1-10 s
3.4. Idea divergence advising We have developed an idea divergence advising tool
53
called Keyword Associator [6,7]. There exist two major facilities provided by Keyword Associator: automatic construction of a collection of associative dictionaries, and a user-friendly interface for associative retrieval of keywords from the associative dictionaries.
3.4.1. Automatic construction of associative dictionaries We have constructed an associative dictionary by using NetNews articles. NetNews is a suitable choice for constructing a dictionary because: (i) the articles are very large, (ii) it covers a wide range of subjects (computer
Fig. 5. User interface of K e y w o r d Associator.
54
K. Sugiyama et al./Knowledge-Based Systems 10 (1997) 51 58
decreasing order of the relevance degree, shown with slider bars. Several combinations of input words such as " a n d " , " o r " and " n o t " are usable. The left window shows an idea editing canvas for expanding our images. We can place words anywhere on the canvas for editing our images and make links between them. The words can be moved between the right window and the canvas with drag-and-drop operations. During the associative retrieval we can control the search region by a filtering facility which restricts searching within specified fields and word classes.
4. The organisation step 4.1. Idea organisation We often exploit diagrams when we think. One reason of using diagrams is that they are a good medium to represent and organise personal ideas, and to share ideas among persons on group works. We developed a diagram-based idea organiser called D-ABDUCTOR [8-10]. The organiser integrates diagrammatic techniques with a menu, direct manipulation and an animation environment, and has pro-
vided communication facilities for group works and multimedia facilities for pictures. In particular, D-ABDUCTOR can deal with diagrams of a generic class of diagrams called compound graphs [1 1] which represent both adjacency and inclusion relationships among nodes (i.e., cards and groups of cards). The compound graphs are used in the KJ method. D-ABDUCTOR provides the following facilities: (i) fast automatic layout of various classes of graphs [11-131, (ii) various operations for graph editing in a menu and direct manipulation environment, (iii) collapse and expand operations of groups of nodes (abstraction, information hiding), (iv) fisheye viewer to overcome the limited size of the computer screen (Diagram Dressing) [14], (v) display with animation so that the user's mental map is preserved, (vi) communication with other D-ABDUCTORs and other applications, and (vii) interpreter language Simple to control D-ABDUCTOR or record editing histories, etc. We now explain how to use D-ABDUCTOR using a simple example (see Figs. 4, 6 and 7). The task in this example is to organise idea segments relating to the idea creation process into an integrated conceptual diagram. These idea segments were memorised using a portable computer. We first transferred the idea segments to
lo0-t
F°-! i~l!llr ;111 f
Fig. 6. Intermediatestage of idea organization using D-ABDUCTOR.
U i i
i
K. Sugiyama et al./Knowled<~e-Based Systems 10 (1997) 51-58
55
Fig. 7. Final result of idea organization using D-ABDUCTOR.
D-ABDUCTOR on a PC or workstation and spread them out on cards on the screen (see Fig. 4). Here, instead of the idea segments being memorised on a portable computer we could also use another kind of data which has been generated through brainstorming or collected via networks and then stored on a disk (see Fig. 2). We manipulate the cards (or edit a diagram) for grouping, title-making, and linking through trial and error (see Fig. 6). D-ABDUCTOR greatly facilitates the elaboration of diagrams. Finally we obtain a conceptually well-organised diagram (see Fig. 7) which expresses the idea creation process. We have evaluated D-ABDUCTOR in terms of operability [15]. We are also distributing D-ABDUCTOR and collecting users' reports for further evaluation. Some early reports indicate that: (i) D-ABDUCTOR is applicable to several practical tasks such as consideration of document structures, design of program structures, and design of database schema, (ii) D-ABDUCTOR can increase work efficiency, and (iii) D-ABDUCTOR's ability to change the layout stimulates the user's thinking. The uses of DABDUCTOR other than idea organisation include layout of documentation, outline processing, software engineering, WWW browsing, CAI design, database schema design, and network monitoring [16i.
4.2. Idea convergence advising By using the facilities provided by the Keyword Associator described in Section 3.4, an emergent media environment can provide the user with various kinds of advising facilities for organising idea cards into a well-structured diagram in D-ABDUCTOR. Given a list of the text content of cards, Keyword Associator can calculate such keywords and values of various relevance among cards as (i) similarities of cards, (ii) groups of cards by clustering, (iii) specific keywords characterising each group, and (iv) similarities among groups and cards. In this calculation we employ the same method utilised in Section 3.4, in which News Groups are replaced with cards. These calculated results are displayed as a diagram or a list of keywords on D-ABDUCTOR for aiding the user in various thinking phases of the idea organisation, tbr example, (i) initial configuration, (ii) grouping of cards, (iii) titlemaking of groups of cards, and (iv) grouping and linking of cards (see Fig. 8). These advising facilities suggest the possibility to automate the organisation step. They might be especially effective when the number of cards is large.
56
K. Sugiyama et al./Knowledge-Based Systems I0 (1997) 51-58
We should therefore determine the description order among cards and groups in a diagram; (ii) when we write a document to more precisely explain the information in a diagram, we often need to see the whole structure of the diagram and change the description order. To solve these difficulties we have developed a new type of outline processor called GOL [17] that uses both graphical and textual modes. An example screen of GOL is shown in Fig. 9. The method of using GOL is described below. A diagram produced in the previous step is converted into a new type of diagram. For example, the diagram shown in Fig. 7 is converted into the diagram shown in the window on the upper right corner of Fig. 9. Layout conventions of the new diagram are: (i) nodes in the depths are arranged alternately horizontally then vertically, and (ii) links between nodes are drawn as elliptic arcs whose directions correspond to the clockwise directions of the ellipses. This diagram can be easily edited (or a description order can be changed) using a direct manipulation tool such as D-ABDUCTOR. This is called Graphical Outline Mode. We use an outline processor in EMACS. This is called Textual Outline Mode. Both modes are connected in both
Fig. 8. Convergence advising facilities provided using Keyword Associator.
5. The presentation step Since the final results of idea creation are presented as documents, we need a tool for converting diagrams to documents. The difficulties in this conversion are: (i) a diagram has a net structure while a document has a linear structure.
Graphical Outline Mode
Textual Outline Mode
or~i~tion s t e p since they are on] idea. For these require~tmts ~e u~ typed computer f o r the i n s t a n t ~ a m i n f r a r e d - r a y transmission to the fat personal c o . r a t e r or a ~orksLation an i l l u s t r a t i o n o f irrsta~-~t i ~ maorized on a PC or ~
I)-ABDtJCTI~,~ on the scree i n f r ~ r a ~ and i~es. ]
.i
m
tr~.omissi
m
a J ~ r s atte illustration of i cards with 1+ memoriz~l are transfem ~ * * I n s t a n t ~n a PC or Workstation. ZN~RUS )-k_~UCT~ ~ ¢ r e cards ===~ zaurusthe screens. In Fil~ **** IrmpiPa and im~es. Fibre 6 pt ~ > **** On ~ t ~ r s attended a ~ect ~ > **** On :aeds with hand~ritte~ ~ > **** On ~ut+,,su~+'~ **** TE i n s p i r a qxsgi l+icc~t eri **** On I ~ \epsfi le {fi le~+aurtm-i¢ freeha~ **** On Trai freelmt+ **** On a C~ file {fi le=i nspi rati ==~ f r + k:enterl
!
**** T~)~I'I IA.I.
TEX Code
+ +
lt,',+,+l.+.,'~Ir
III,I++~ +,,it+toO
a~,+++m,,®+ + + + + ++,~+++m++.+++.+~. +,+~++~ + . + + ,+ + + ,.+,1,.~+ + . t + . + + + + - + + + ~ + ++,~++ ++.+ ~ + + . +++ ++ ~+ ~ + , + + ~ + , + < + ++ ++++ + + ~ + m + ~ + m ~ . + * + +~++,+++ + + ++++ , ~ ~ . + + ++ + + + + + . + ~ + , + + + + +
/ , "~(
• TEX Viewer
Fig. 9. An example screen of GOL.
K. Sugiyamu et al./Knowledge-Based Systems I0 (1997~ 51-58
ways. In other words, changes in a description order and outlines in one mode always affect the changes in the other mode consistently, and vice versa. By using both outline modes we can convert a conceptual diagram into a document. GOL has a facility to automatically convert texts in the Textual Outline into LaTeX code. For example, the window in the lower right of Fig. 9 shows the final document (TeX Viewer) which precisely describes the diagram in the upper right. It should be noted that this conversion is possible even if a diagram includes cards with images. Our experience with GOL indicates that: (i) the graph view is useful for finding and jumping to topics related to the current one, and for checking forward and backward references; these references are drawn using a different colour: (ii) it is always possible to check the whole document and the cur-. rent position, which increases the user's confidence, and (iii) the frequency of consulting the graph view decreases when the user begins to manipulate the low-level details of a document, for example, details in LaTeX.
6. Concluding remarks We have described the facilities of our current emergent media environment and have shown some examples of the idea creation process. The facilities are not fully integrated in the current version, and the system has not been sufficiently evaluated. We need to further integrate and evaluate the system by applying it to many practical problems. To do this, the following studies are envisaged for future research: (i) employing multimedia portable equipment for instant capture of photos, video, sound, etc., (ii) employing recognition equipment for hand-written texts and diagrams, (iii) investigating how to use a big screen for idea organisation, and (iv) conducting various experiments with methods other than the KJ method in the emergent media environment.
Appendix A KJ method The KJ method[l] is very well known in Japan as an effective label-based method for organising ideas and solving problems without computer support. The terms label and team are used in the K J-method instead of card and group). The main part of the KJ method contains four basic steps as follows:
I. Label making. We start with a supply of labels on which ideas or information (text or image, etc.) relevant to our problem are written. We collect and record ideas until we feel we have exhausted all information necessary to solve the problem. 2. Label grouping and title making. The labels are shuffled well and spread on a large sheet. Then all the labels are read several times. If there are labels that seem to belong
57
together, we make a team of the labels. This process is repeated. After about two-thirds of all the labels are arranged in teams, making titles for the teams is started. The titles should clearly describe the essence of all labels in the team. Once a title is made for a team, we put all the labels together in a pile with the title clipped on its top. Next, we arrange the teams in larger teams in the same manner. This iterative process of grouping labels may be repeated as many times as necessary. Usually it is terminated when the number of the teams is reduced to less than 10. 3. Spatial arrangement and chart making. We carefully find the arrangement of the final groups in which a consistent understanding of all the groups can be obtained. Then we proceed to arrange all sub-teams or elements in the same manner. After completing this spatial arrangement, we draw a chart in our own handwriting by showing the relationships using various symbols and signs. 4. Verbal or written explanation. To explain the chart clearly, we try to describe the chart verbally or in writing. As a general rule our explanation should proceed to a team adjacent to where it started. The cumulative effect of idea generation will continue to increase as our explanation advances.
References [I] J. Kawakita, The KJ Method, Kawakita Research Institute, Tokyo, 1975,
[21 C. Marshall, F. Shipman Ill, J. Coombs, VIKI: spatial hypertext supporting emergent structure, Proceedings of ECHT'94, Edinburgh, September 1994, pp. 13-23. [3] M. Stefik, The next knowledge medium, AI Magazine 7 (1) (1986) 34-46. [4] J. Haake, C. Neuwirth, N. Streitz, Coexistence and transformation of informal and formal structures: requirements for more flexible hypermedia systems, Proceedings of ECHT'94, Fdinburgh+ September 1994, pp. 1 12. [5] K. Sugiyama, K. Misue, I. Watanabe+ K, Nitta, Y. Takada, Integration of idea creation tools: emergent media environment. FUJITSU Scientific and Technical Journal 32 (2) (19961 154-169. [61 I. Watanabe. Divergent thinking support system: keyword Associamr, Proceedings of SICE Joint Symposium, 199 I, pp. 411-418 (in Japanese). [7] I. Watanabe, Divergent thinking support system: keyword Associator, Proceedings of SICE 15th System Engineering Workshop. 1994, pp. 9-16 (in Japanese). 18] K. Misue. K. Sugiyama, How does D-ABDUCTOR support human thinking processes?, in: M. Gigante, T.L. Kunii (Eds.), Proceedings of the Computer Graphics International (CGl'94), Melbourne, June-July 1994. World Scientific, 1996. [91 K. Misue, D-ABDUCTOR 2.0 user manual Research Report llASRR-93-9E, Fujitsu Laboratories, 1993, [10] K. Sugiyama, K. Misue, A generic compound graph visualiser/manipulator: D-ABDUCTOR, in: F.J. Brandenburg (Ed.), Proceedings of Symposium on Graph Drawing (GD'95), Passau, September 1995+ Springer LNCS-1027, 1996. pp. 500-503. [111 K. Sugiyama, S. Tagawa, M. Toda, Methods for visual understanding of hierarchical system structures, 1EEE Transactions on Systems, Man, and Cybernetics SMC-I I (2) (1981) 109-125.
58
K. Sugiyama et aL/Knowledge-Based Systems 10 (I997) 51 58
[12] K. Sugiyama, K. Misue, Visualisation of structural information: automatic drawing of compound digraphs, IEEE Transactions on Systems, Man, and Cybernetics 21 (4) (1991) 876-893. [13] K. Sugiyama, K. Misue, Graph drawing by the magnetic spring model, Journal of Visual Languages and Computing 6 (3) (1995) 217-231. [14] K. Misue, P. Eades, W. Lai, K. Sugiyama, Layout adjustment and the mental map, Journal of Visual Languages and Computing 6 (2) (1995) 183-210. [15] K. Misue, K. Sugiyama, Evaluation of a thinking support system from
operational points of view, Proceedings of HCI International '95, Yokohama, July 1995, pp. 679-702. [ 16] K. Misue, K. Nitta, K. Sugiyama, T. Koshiba, R. lnder, Techniques for DUI platforms: developing graph drawing applications on D-ABDUCTOR, Research Report ISIS-RR-96-7E, Fujitsu Laboratories. 1996. [17] K, Nitta, R. lnder, K. Misue, K. Sugiyama, GOL: graphical outline processor simultaneously using a text view and a graph view, Proceedings of the First Asian Pacific Conference on Computer Human Interaction (APCHI'96), Singapore, June 1996, pp. 469-478.
Knowledge-Based Systems 10 (1997) 51-58
Emergent media environment for idea creation support Kozo Sugiyama a'*, Kazuo Misue a, Isamu Watanabe a, Kiyoshi Nitta a, Yuji Takada b ~Multimedia Systems Laboratories, Fujitsu Laboratories Limited, 140 Miyamoto, Numazu, Shizuoka 410-03. Japan bpersonal Systems lz~boratories, Fujitsu Laboratories" Limited, 2-2-1 Momochihama. Sawaru-ku, Fukuoku 814. Japan
Abstract
An interactive computer system for supporting the emergent stages (or upper streams) in human intellectual activities has been developed. In the system it is intended to integrate facilities for supporting generation, collection, organisation, and presentation of ideas and advising on the divergence and convergence of the ideas. This paper explains the facilities of the system and how to use it, and describes some examples of their use. © 1997 Elsevier Science B.V.
I. Introduction Our major concern of this paper is to develop a highly interactive computer system for supporting early, emergent and usually creative stages in human intellectual activities such as research planning, conceptual design, software requirement analysis, knowledge acquisition, problem solving, decision making, counselling, motivation and so on. We can often observe that the following steps are generally common to the emergent stages of the abovementioned activities: (i) encountering or finding a problem, (ii) generating segments of ideas and collecting piece-wise information relating to the problem, (iii) organizing them into a structured form to understand or solve the problem, and (iv) presenting the structured form as a document or a speech to an audience. The process consisting of these steps is called an emergent process in general or an idea creation process more specifically where the term idea means an understanding, insight or solutions of a problem. As an emergent process proceeds, information treated in each step is gradually elaborated from vague, informal, nonstructured (or piece-wise), personal and small-sized information to more precise, formal, structured (or wellorganised), public and large-sized information. We call such information methods to express ideas in an emergent process emergent media. We can identify the three chunks of emergent media as follows. Cards are suitable for expressing piece-wise information (memos, sketches, messages, etc.); they are mainly used in the generation and/or collection step. Diagrams are suitable for organising
a set of piece-wise information into structured forms (lists, clusters, trees, various types of graphs, etc.); they are mainly used in the organisation step. Documents are suitable for expressing more precise, formal and large-sized information (concepts, ideas, etc.); they are mainly used in the presentation step. Fig. 1 illustrates steps of an emergent process and chunks of emergent media. An emergent media environment (EME) is defined as a set of emergent media-based facilities to support the emergent process effectively on the basis of a more basic environment such as hardware, basic software, multimedia, networks and so on. To perform an idea creation process, we should specify a methodology which explains how to utilise the facilities provided by EME. In developing an experimental version of EME, we refer to the KJ method [ 11 (see Appendix) as a methodological basis. There are some other related works in this area: Marshall et al. [2] classified current spatial hypertext systems into four categories: permissive, emergent, descriptive and prescriptive. Our work is relates to the emergent spatial hypertext. Our term emergent media relates to the term knowledge media proposed by Stefik [3]. Haake et al. [4] developed a hypermedia prototype where seamless idea transformation from the informal to the formal is considered. In this paper facilities that the current EME provides are outlined. The reminder of this paper is organised as follows: Section 2 presents an overview of EME implemented and Sections 3 - 5 explain each facility in more detail. Section 6 describes concluding remarks and plans for future research.
2. Overview of current emergent media environment *
Corresponding author.
0950-7051/97/$17.00 © 1997 Elsevier Science B.V. All rights reserved PII S0950-7051 ( 9 7 ) 0 0 0 1 3 - 0
In developing the experimental EME our efforts have
K. Sugiyama et al./Knowledge-Based Systems 10 (1997) 51-58
52
DIAG~
con~ton
Organization
~UMENT
Prec,,marion
Fig. 1. Chunks of emergent media.
been focused on various dimensions. In order to attain integrated support from the generation and/or collection step to the presentation step, we have developed user-friendly interfaces for handling information in each chunk of emergent media and for flexibly converting information from one chunk to another. To concentrate on thinking itself, we developed facilities for avoiding chores and unpleasant jobs when arranging cards, drawing diagrams, etc. Computer-aided techniques have been applied for idea divergence advising in the generation/collection step and idea convergence advising in the organisation step. Networks have been employed for idea collection and methods of handling multimedia information have been implemented for idea representation in each step. Both personal use and group use should be supported. The current version is mainly for personal use, but distributed brainstorming by a group is partly supported for idea generation. The current EME has been implemented on UNIX/X-Window. Fig. 2 shows on overview of a process, tools, and facilities of the current EME [5].
3. The generation/collection step 3.1. Instant idea memorisation and transfer Ideas often come to mind suddenly and unexpectedly, for example, when we are in bed, walking on a street, driving a car, or at the desk. Unless we memorise or otherwise record the ideas instantly, they tend to disappear within a minute. Therefore an instant idea-capturing or memorising facility
Fig. 3. Instant idea memorisation and transfer.
anywhere and anytime is most important for the generation/ collection step. Further, we need to be able to easily and smoothly transfer memorised ideas to the organisation step. For this purpose we use a portable and pen-based computer called ZAURUS (commercial product by Sharp Corporation) and connect it via an infrared-light link to an idea-organising tool called D-ABDUCTOR running on a PC or workstation (see Fig. 3). Fig. 4 is an example of DABDUCTOR screens displaying idea segments written on cards. Fig. 4 shows a collection of cards with text and a photo taken with a digital camera.
3.2. Distributed asynchronous brainstorming Group works for the idea generation are classified into face-to-face meetings, distributed synchronous meetings, and distributed asynchronous meetings. The benefits of group idea generation through such meetings include an increase in the amount of available information/knowledge and viewpoints, and synergy between participants. We are
_1
Fig. 2. An overview of the process, tools and facilities of the current EME.
.............
Fig. 4. Example of cards (collected idea segments) with texts and images•
K. Sugiyama et al./Knowledge-Based Systems 10 (1997) 51 58
developing a brainstorming tool called EBICS for a distributed asynchronous situation using multi-agent programming technologies. EBICS consists of servers, clients (or participants to the meeting) and other related tools such as a database management system. Clients and servers are implemented as agents in a multi-agent programming language, and data are managed by the agent-oriented distributed database engine.
3.3. Information collection via network Retrieving or collecting information through the network (e-mail, e-news, WWW, etc.) is another useful method in this step. We have used our EME to, for example, collect and summarise users' comments on D-ABDUCTOR and responses to a questionnaire for market research. Network searchers, intelligent agents, software robots, WWWWorms, and methods for data mining and knowledge discovery will be used much more for this purpose in the near future. N u m b e r of articles Total n u m b e r of words Size o f dictionary Time for reirieval
- 200 000 - 1 5 0 000 -~300 MB 1 s - 2 rain
- 15 0 0 0 - 1 l0 0 0 0 - - 2 0 MB 1-10 s
3.4. Idea divergence advising We have developed an idea divergence advising tool
53
called Keyword Associator [6,7]. There exist two major facilities provided by Keyword Associator: automatic construction of a collection of associative dictionaries, and a user-friendly interface for associative retrieval of keywords from the associative dictionaries.
3.4.1. Automatic construction of associative dictionaries We have constructed an associative dictionary by using NetNews articles. NetNews is a suitable choice for constructing a dictionary because: (i) the articles are very large, (ii) it covers a wide range of subjects (computer
Fig. 5. User interface of K e y w o r d Associator.
54
K. Sugiyama et al./Knowledge-Based Systems 10 (1997) 51 58
decreasing order of the relevance degree, shown with slider bars. Several combinations of input words such as " a n d " , " o r " and " n o t " are usable. The left window shows an idea editing canvas for expanding our images. We can place words anywhere on the canvas for editing our images and make links between them. The words can be moved between the right window and the canvas with drag-and-drop operations. During the associative retrieval we can control the search region by a filtering facility which restricts searching within specified fields and word classes.
4. The organisation step 4.1. Idea organisation We often exploit diagrams when we think. One reason of using diagrams is that they are a good medium to represent and organise personal ideas, and to share ideas among persons on group works. We developed a diagram-based idea organiser called D-ABDUCTOR [8-10]. The organiser integrates diagrammatic techniques with a menu, direct manipulation and an animation environment, and has pro-
vided communication facilities for group works and multimedia facilities for pictures. In particular, D-ABDUCTOR can deal with diagrams of a generic class of diagrams called compound graphs [1 1] which represent both adjacency and inclusion relationships among nodes (i.e., cards and groups of cards). The compound graphs are used in the KJ method. D-ABDUCTOR provides the following facilities: (i) fast automatic layout of various classes of graphs [11-131, (ii) various operations for graph editing in a menu and direct manipulation environment, (iii) collapse and expand operations of groups of nodes (abstraction, information hiding), (iv) fisheye viewer to overcome the limited size of the computer screen (Diagram Dressing) [14], (v) display with animation so that the user's mental map is preserved, (vi) communication with other D-ABDUCTORs and other applications, and (vii) interpreter language Simple to control D-ABDUCTOR or record editing histories, etc. We now explain how to use D-ABDUCTOR using a simple example (see Figs. 4, 6 and 7). The task in this example is to organise idea segments relating to the idea creation process into an integrated conceptual diagram. These idea segments were memorised using a portable computer. We first transferred the idea segments to
lo0-t
F°-! i~l!llr ;111 f
Fig. 6. Intermediatestage of idea organization using D-ABDUCTOR.
U i i
i
K. Sugiyama et al./Knowled<~e-Based Systems 10 (1997) 51-58
55
Fig. 7. Final result of idea organization using D-ABDUCTOR.
D-ABDUCTOR on a PC or workstation and spread them out on cards on the screen (see Fig. 4). Here, instead of the idea segments being memorised on a portable computer we could also use another kind of data which has been generated through brainstorming or collected via networks and then stored on a disk (see Fig. 2). We manipulate the cards (or edit a diagram) for grouping, title-making, and linking through trial and error (see Fig. 6). D-ABDUCTOR greatly facilitates the elaboration of diagrams. Finally we obtain a conceptually well-organised diagram (see Fig. 7) which expresses the idea creation process. We have evaluated D-ABDUCTOR in terms of operability [15]. We are also distributing D-ABDUCTOR and collecting users' reports for further evaluation. Some early reports indicate that: (i) D-ABDUCTOR is applicable to several practical tasks such as consideration of document structures, design of program structures, and design of database schema, (ii) D-ABDUCTOR can increase work efficiency, and (iii) D-ABDUCTOR's ability to change the layout stimulates the user's thinking. The uses of DABDUCTOR other than idea organisation include layout of documentation, outline processing, software engineering, WWW browsing, CAI design, database schema design, and network monitoring [16i.
4.2. Idea convergence advising By using the facilities provided by the Keyword Associator described in Section 3.4, an emergent media environment can provide the user with various kinds of advising facilities for organising idea cards into a well-structured diagram in D-ABDUCTOR. Given a list of the text content of cards, Keyword Associator can calculate such keywords and values of various relevance among cards as (i) similarities of cards, (ii) groups of cards by clustering, (iii) specific keywords characterising each group, and (iv) similarities among groups and cards. In this calculation we employ the same method utilised in Section 3.4, in which News Groups are replaced with cards. These calculated results are displayed as a diagram or a list of keywords on D-ABDUCTOR for aiding the user in various thinking phases of the idea organisation, tbr example, (i) initial configuration, (ii) grouping of cards, (iii) titlemaking of groups of cards, and (iv) grouping and linking of cards (see Fig. 8). These advising facilities suggest the possibility to automate the organisation step. They might be especially effective when the number of cards is large.
56
K. Sugiyama et al./Knowledge-Based Systems I0 (1997) 51-58
We should therefore determine the description order among cards and groups in a diagram; (ii) when we write a document to more precisely explain the information in a diagram, we often need to see the whole structure of the diagram and change the description order. To solve these difficulties we have developed a new type of outline processor called GOL [17] that uses both graphical and textual modes. An example screen of GOL is shown in Fig. 9. The method of using GOL is described below. A diagram produced in the previous step is converted into a new type of diagram. For example, the diagram shown in Fig. 7 is converted into the diagram shown in the window on the upper right corner of Fig. 9. Layout conventions of the new diagram are: (i) nodes in the depths are arranged alternately horizontally then vertically, and (ii) links between nodes are drawn as elliptic arcs whose directions correspond to the clockwise directions of the ellipses. This diagram can be easily edited (or a description order can be changed) using a direct manipulation tool such as D-ABDUCTOR. This is called Graphical Outline Mode. We use an outline processor in EMACS. This is called Textual Outline Mode. Both modes are connected in both
Fig. 8. Convergence advising facilities provided using Keyword Associator.
5. The presentation step Since the final results of idea creation are presented as documents, we need a tool for converting diagrams to documents. The difficulties in this conversion are: (i) a diagram has a net structure while a document has a linear structure.
Graphical Outline Mode
Textual Outline Mode
or~i~tion s t e p since they are on] idea. For these require~tmts ~e u~ typed computer f o r the i n s t a n t ~ a m i n f r a r e d - r a y transmission to the fat personal c o . r a t e r or a ~orksLation an i l l u s t r a t i o n o f irrsta~-~t i ~ maorized on a PC or ~
I)-ABDtJCTI~,~ on the scree i n f r ~ r a ~ and i~es. ]
.i
m
tr~.omissi
m
a J ~ r s atte illustration of i cards with 1+ memoriz~l are transfem ~ * * I n s t a n t ~n a PC or Workstation. ZN~RUS )-k_~UCT~ ~ ¢ r e cards ===~ zaurusthe screens. In Fil~ **** IrmpiPa and im~es. Fibre 6 pt ~ > **** On ~ t ~ r s attended a ~ect ~ > **** On :aeds with hand~ritte~ ~ > **** On ~ut+,,su~+'~ **** TE i n s p i r a qxsgi l+icc~t eri **** On I ~ \epsfi le {fi le~+aurtm-i¢ freeha~ **** On Trai freelmt+ **** On a C~ file {fi le=i nspi rati ==~ f r + k:enterl
!
**** T~)~I'I IA.I.
TEX Code
+ +
lt,',+,+l.+.,'~Ir
III,I++~ +,,it+toO
a~,+++m,,®+ + + + + ++,~+++m++.+++.+~. +,+~++~ + . + + ,+ + + ,.+,1,.~+ + . t + . + + + + - + + + ~ + ++,~++ ++.+ ~ + + . +++ ++ ~+ ~ + , + + ~ + , + < + ++ ++++ + + ~ + m + ~ + m ~ . + * + +~++,+++ + + ++++ , ~ ~ . + + ++ + + + + + . + ~ + , + + + + +
/ , "~(
• TEX Viewer
Fig. 9. An example screen of GOL.
K. Sugiyamu et al./Knowledge-Based Systems I0 (1997~ 51-58
ways. In other words, changes in a description order and outlines in one mode always affect the changes in the other mode consistently, and vice versa. By using both outline modes we can convert a conceptual diagram into a document. GOL has a facility to automatically convert texts in the Textual Outline into LaTeX code. For example, the window in the lower right of Fig. 9 shows the final document (TeX Viewer) which precisely describes the diagram in the upper right. It should be noted that this conversion is possible even if a diagram includes cards with images. Our experience with GOL indicates that: (i) the graph view is useful for finding and jumping to topics related to the current one, and for checking forward and backward references; these references are drawn using a different colour: (ii) it is always possible to check the whole document and the cur-. rent position, which increases the user's confidence, and (iii) the frequency of consulting the graph view decreases when the user begins to manipulate the low-level details of a document, for example, details in LaTeX.
6. Concluding remarks We have described the facilities of our current emergent media environment and have shown some examples of the idea creation process. The facilities are not fully integrated in the current version, and the system has not been sufficiently evaluated. We need to further integrate and evaluate the system by applying it to many practical problems. To do this, the following studies are envisaged for future research: (i) employing multimedia portable equipment for instant capture of photos, video, sound, etc., (ii) employing recognition equipment for hand-written texts and diagrams, (iii) investigating how to use a big screen for idea organisation, and (iv) conducting various experiments with methods other than the KJ method in the emergent media environment.
Appendix A KJ method The KJ method[l] is very well known in Japan as an effective label-based method for organising ideas and solving problems without computer support. The terms label and team are used in the K J-method instead of card and group). The main part of the KJ method contains four basic steps as follows:
I. Label making. We start with a supply of labels on which ideas or information (text or image, etc.) relevant to our problem are written. We collect and record ideas until we feel we have exhausted all information necessary to solve the problem. 2. Label grouping and title making. The labels are shuffled well and spread on a large sheet. Then all the labels are read several times. If there are labels that seem to belong
57
together, we make a team of the labels. This process is repeated. After about two-thirds of all the labels are arranged in teams, making titles for the teams is started. The titles should clearly describe the essence of all labels in the team. Once a title is made for a team, we put all the labels together in a pile with the title clipped on its top. Next, we arrange the teams in larger teams in the same manner. This iterative process of grouping labels may be repeated as many times as necessary. Usually it is terminated when the number of the teams is reduced to less than 10. 3. Spatial arrangement and chart making. We carefully find the arrangement of the final groups in which a consistent understanding of all the groups can be obtained. Then we proceed to arrange all sub-teams or elements in the same manner. After completing this spatial arrangement, we draw a chart in our own handwriting by showing the relationships using various symbols and signs. 4. Verbal or written explanation. To explain the chart clearly, we try to describe the chart verbally or in writing. As a general rule our explanation should proceed to a team adjacent to where it started. The cumulative effect of idea generation will continue to increase as our explanation advances.
References [I] J. Kawakita, The KJ Method, Kawakita Research Institute, Tokyo, 1975,
[21 C. Marshall, F. Shipman Ill, J. Coombs, VIKI: spatial hypertext supporting emergent structure, Proceedings of ECHT'94, Edinburgh, September 1994, pp. 13-23. [3] M. Stefik, The next knowledge medium, AI Magazine 7 (1) (1986) 34-46. [4] J. Haake, C. Neuwirth, N. Streitz, Coexistence and transformation of informal and formal structures: requirements for more flexible hypermedia systems, Proceedings of ECHT'94, Fdinburgh+ September 1994, pp. 1 12. [5] K. Sugiyama, K. Misue, I. Watanabe+ K, Nitta, Y. Takada, Integration of idea creation tools: emergent media environment. FUJITSU Scientific and Technical Journal 32 (2) (19961 154-169. [61 I. Watanabe. Divergent thinking support system: keyword Associamr, Proceedings of SICE Joint Symposium, 199 I, pp. 411-418 (in Japanese). [7] I. Watanabe, Divergent thinking support system: keyword Associator, Proceedings of SICE 15th System Engineering Workshop. 1994, pp. 9-16 (in Japanese). 18] K. Misue. K. Sugiyama, How does D-ABDUCTOR support human thinking processes?, in: M. Gigante, T.L. Kunii (Eds.), Proceedings of the Computer Graphics International (CGl'94), Melbourne, June-July 1994. World Scientific, 1996. [91 K. Misue, D-ABDUCTOR 2.0 user manual Research Report llASRR-93-9E, Fujitsu Laboratories, 1993, [10] K. Sugiyama, K. Misue, A generic compound graph visualiser/manipulator: D-ABDUCTOR, in: F.J. Brandenburg (Ed.), Proceedings of Symposium on Graph Drawing (GD'95), Passau, September 1995+ Springer LNCS-1027, 1996. pp. 500-503. [111 K. Sugiyama, S. Tagawa, M. Toda, Methods for visual understanding of hierarchical system structures, 1EEE Transactions on Systems, Man, and Cybernetics SMC-I I (2) (1981) 109-125.
58
K. Sugiyama et aL/Knowledge-Based Systems 10 (I997) 51 58
[12] K. Sugiyama, K. Misue, Visualisation of structural information: automatic drawing of compound digraphs, IEEE Transactions on Systems, Man, and Cybernetics 21 (4) (1991) 876-893. [13] K. Sugiyama, K. Misue, Graph drawing by the magnetic spring model, Journal of Visual Languages and Computing 6 (3) (1995) 217-231. [14] K. Misue, P. Eades, W. Lai, K. Sugiyama, Layout adjustment and the mental map, Journal of Visual Languages and Computing 6 (2) (1995) 183-210. [15] K. Misue, K. Sugiyama, Evaluation of a thinking support system from
operational points of view, Proceedings of HCI International '95, Yokohama, July 1995, pp. 679-702. [ 16] K. Misue, K. Nitta, K. Sugiyama, T. Koshiba, R. lnder, Techniques for DUI platforms: developing graph drawing applications on D-ABDUCTOR, Research Report ISIS-RR-96-7E, Fujitsu Laboratories. 1996. [17] K, Nitta, R. lnder, K. Misue, K. Sugiyama, GOL: graphical outline processor simultaneously using a text view and a graph view, Proceedings of the First Asian Pacific Conference on Computer Human Interaction (APCHI'96), Singapore, June 1996, pp. 469-478.