- Email: [email protected]
PDA-based graphical interchange for field service and repair workers
Comput.
Pergamon
& Graphics, Vol. 20, No. 5, pp. 641-649, 1996 Copyright 0 1996 Elsevk Science Ltd Printed in Great Britain. All rights reserved 0097-8493/96 $15.00+0.00
PIIt soo97-8493(~ooo38-6 Mobile Computing
PDA-BASED GRAPHICAL INTERCHANGE FOR FIELD SERVICE AND REPAIR WORKERS WAYNE V. CITRIN’+ and MARK
D. GROSS’
‘Deparhnent
of Electrical and Computer Engineering, University of Colorado, Boulder, CO 80309-0425, U.S.A. e-ma2 [email protected] ‘School of Environmental Design, University of Colorado, Boulder, CO, U.S.A.
Abstract-We present an ongoing project to develop a system to provide field service.workers with timely and accurate service information. The system will allow workers to download diagrams or photographs from a host computer’s central database onto a PDA. The workers will be able to annotate the diagrams to reflect work performed, and later upload the annotations to the host computer, where they will be integrated into an updated database. Diagram recognition functionality is distributed between the PDA (which performs low-level shape and handwriting recognition) and the host computer (which performs high-level domain-based diagram recognition). Distributing the functionality offers a number of advantages: it allows the relatively resource-poor PDA to be part of a Powerful diagram recognition environment, it allows the use of standardized hardware-based recognition facilities in a domain-based recognition system, and it allows off-line drawing recognition and storage of diagrams, thereby avoiding excessive use of slow or expensive communications channels. Copyright 0 1996 Elsevier Science Ltd
1. INTRODUCTION
through
use
of
personal
tions back to the central
digital
assistants
database
for use by the
next worker. PDAs (the Apple Newton MessagePad, for example) are an appropriate technology for this task, since they are small and lightweight, support display of graphic information, allow penbased input (desirable for graphical data entry and also desirable when use of a keyboard is inappropriate), and support both wireless and wired communication.
connectors, splices, and equipment. They carry microfiche maps in their trucks, but these are maintained and produced by a central office and are likely to be out of date. (Some service locations maintain their own map sets for this reason.) To understand the on-site configuration, a good field worker often makes a diagram on a scrap of paper, drawing first the “as is” and then the “change to” configurations. After the job is finished, the worker discards the paper and proceeds to the next job. The updates never make it back to the central repository, and the process must proceed all over again when the next repair or maintenance activity is performed. We are addressing this and similar problems
t Author for correspondence..
the
(PDAs) to deliver graphical information to field workers from the central office, allow the workers to record their modifications, and upload the modifica-
As infrastructure systems (such as telephone and computer networks, electrical power grids, and building support systems) become more complex, and as more workers are involved in their support and maintenance, it becomes even more important to provide field service workers with timely, accurate, and complete information. Delivery of such information can provide savings to industries and customers, and may even affect public safety. The situation of US WEST, the local telephone company in Colorado, represents a case in point. Field service workers are dispatched from a service location to service communications equipment in the field. When they arrive at the job site, they find a confusing and often undocumented configuration of
The system employs diagram recognition, which is distributed between the PDA (which performs lowlevel shape and handwriting recognition) and the host computer (which performs high-level domainbased diagram recognition). Distributing the functionality in this way offers a number of advantages: it allows the relatively resource-poor PDA to be part of a powerful diagram recognition environment (something that would probably not be possible on the PDA alone), it allows the use of standardized hardware-based recognition facilities in a domainbased recognition system (desirable since the lowlevel shape and handwriting recognition functions do not change drastically from one domain to the next), and it allows off-line shape and handwriting recognition and storage of diagrams, thereby avoiding excessiveuse of slow or expensive communicati&s channels. This paper begins by motivating the design of our system through three scenarios which illustrate the uses to which such a system can be put. It then
641
642
W. V. Citrin and M. D. Gross
describes the underlying architecture for such a system, both at a high level and in detail, discusses technical challenges that remain to be solved, reviews some related projects we are conducting that employ the same underlying distributed architecture for diagram recognition, and reviews related literature.
2. MOTIVATING
SCENARIOS
The use of the PDA-based graphical interchange system can best be explained by the following three scenarios. Subsequent sections will explain the system’s organization and illustrate how the system supports the various scenarios. The first scenario indicates a way in which PDAbased graphical interchange can assist field workers in understanding complex hardware configurations, and can provide them with easy ways to update the information to reflect the changes they make, thus keeping the information up to date. Figure l(a) shows a tangle of hardware encountered by a field service worker on a call. With out-of-date information, it may be extremely time consuming for the worker to understand the hardware or the modifications that have already been made. However, if the worker has downloaded information from the central database onto a PDA, he or she can consult the notes of the previous worker who serviced the hardware [Fig. l(b)]. The worker can then make the required changes and update the photograph of the hardware with additional annotations. These additional annotations are entered through the PDA’s pen-based interface, and are recorded on a new virtual ‘layer’, which is later uploaded to the central database. In this way, future workers can consult the layers with the notes of individual workers, or can group several layers together to see the effects of a sequence of service calls.
The second scenario illustrates the way in which diagram recognition can be incorporated into a graphical interchange system. In this case, the worker sketches a schematic diagram indicating the current configuration or work that was performed. Using the graphical interchange system, this diagram can be sketched on the PDA rather than on paper. This offers a number of advantages. First, the diagram is now in a digital form that can be easily uploaded into the central database. If the diagram was sketched on paper, there is more possibility that the diagram will be lost, or that the worker will forego the added effort of later putting the diagram through a scanner. Second, the diagram is not just an image but rather a collection of shapes grouped hierarchically into components. As such, the diagram can be further edited, either by the original worker on site, or by subsequent workers. Finally, since the diagram is such a collection of shapes and text, further diagram recognition operations can be performed on it, in order to assign meaning to the diagram and to determine the diagrammed system’s correctness, estimate the performance of the system, etc. This further recognition need not be performed on site by the PDA (in fact, it is unlikely that it will, since the PDA probably lacks sufficient computing power for this task), but rather will be performed on some back end host to which the diagram is uploaded. The next section describes the relation between the front-end PDA and the back-end host when diagram recognition is performed. Figure 2(a) shows a possible diagram as sketched by a worker on the PDA. Figure 2(b) shows a possible result of applying diagram recognition to the sketch. The third scenario is actually an extension of the first, and illustrates the way in which a worker may enter new images into the system. In this case, the worker visits a site for which detailed records or
Fig. 1. (a) Worker finds this tangle of equipment on site. (b) Systemdeliversthis photo annotated by previousworker.
PDA-basedgraphicalinterchange
643 To Higher Floors of EWD
BuiIding
(a) Fig. 2. (a) Telecommunicationsworker makesthis diagram on site. (b) Host convertsand storesit for future use.
The Newton also contains support through communications, either through the built-in serial and infrared ports, or through optional telephone and wireless modems. Finally, the Newton contains onboard memory, expandable to several megabytes through PCMCIA cards. The back end host is currently a Macintosh computer running a diagram recognition program called the Electronic Cocktail Napkin [l]. This system accepts shapes and text recognized by the PDA and groups them into higher-level domainspecific aggregates. The system can be configured so that the Cocktail Napkin informs the PDA of the recognition results, which are displayed. The Electronic Cocktail Napkin is described in more detail in 3. SYSTEM OVERVIEW Section 4.2. The graphical interchange system is composed of Figure 3 shows the high-level organization of the two components: a PDA-based front end, and a host- graphical interchange system, and the distribution of based back end. functionality between the components. The PDA The front end is carried into the field by service captures strokes entered by the user and converts workers, who can download data from the host them into shapes through the built-in recognition before leaving their base, or who can download data facilities. These recognized items are either stored in in the field over telephone lines through an attachable the PDA for later transmission, or, if the PDA is modem. In the future, we plan to employ wireless connected to the host, are uploaded as they are communications for the exchange of data. The PDA entered and recognized. The back end host receives (an Apple Newton MessagePad) provides an LCD the shapes and text and, depending on the spatial screen on which the user may draw with a stylus. The relationships between the elements, recognizes highpen strokes entered by the user are captured by the er-level aggregates, depending on the domain for Newton and redisplayed as ‘ink’. The PDA contains which the host is configured. If the system is so built-in shape recognition facilities, which can configured, and the PDA is connected while recognirecognize simple, domain-independent shapesinclud- tion is going on, the host will inform the PDA of the ing circles, triangles, rectangles, closed and open results of the recognition, which are then displayed. curves, lines, and closed and open polygons, as well Figure 4 shows the operation of the current as a trainable handwriting recognition facility and prototype implementation of the system on a simple the ability to edit recognized shapes and text. The set of glyphs. In Fig. 4(a), the user enters a circle on Newton is programmable, and the aforementioned the PDA. In Fig. 4(b), the PDA recognizes it as a facilities can be easily incorporated into the program. circle, and the back end is notified (assuming the
images do not yet exist. In this case, the user may wish to generate an image and annotate it for future reference. In such a case, the worker will photograph the site with a digital camera, download the image into the PDA through the PDA’s serial port, and annotate it using the pen-based interface as described in the first scenario. As in the first two scenarios, this offers the advantages of allowing the user to easily make annotations to digital images rather than paper, preserves the structure (image and multiple annotation layers) for storage in a database, and does not require the worker to go to the extra effort of preserving his or her paper notes and later entering them into the central system through a scanner.
W. V. Chin PDA front end
recognized
shapes
and M. D. Gross
Host computer
arid
Annotations indicating compound objects
High-level diagram re&nitiin, diagram management and storage
Basic shape and handwriting recognition editing functions
/
Fig. 3. Organization
of distributed recognition system
back end is connected). In Fig. 4(c), a line is drawn, and in Fig. 4(d), the PDA recognizes it as a line, and it again notifies the back end of what it recognized and the location of the recognized item. The back end has been configured to recognize the circle over the line as the ‘sunset’ glyph; it notifies the PDA of the recognition and the result is displayed [Fig. 4(e)]. Figure 5 shows the exchange of messagesbetween the PDA and the host. Newton PDAs were chosen as the front end of the system for a number of reasons. Newtons are fairly small and light. Their pen-based input allows them to be used in the field where use of a keyboard would be awkward. Their built-in stroke capture and recognition facilities let us avoid the non-trivial task of reimplementing such facilities. Most importantly, Newtons are much less expensive than laptop computers, making it much more likely that companies will adopt PDA technology than laptops. It is true that laptops with pen-based entry capability currently exist that have more processing power and better resolution screens than current PDAs. How-
(a)
(b)
back end
ever, their larger size and greater expense led us Eo explore more economical solutions. In addition, future PDA models should address the problems of processing power and screen resolution (see Section 5). 4. SYsTEkI
DETAILS
4.1. PDA subsystem We have developed a program called Smart&d that runs on the Newton as a basic front end. SmartPad supports stroke and shape recognition, and captures and saves digital ink, as well as the recognized shapes. It supports basic gesture-based editing, including erasing, copying, moving, and changing the shapes of elements. Finally, it supports text entry, recognition, and editing, either through handwriting or through a soft keyboard. All of these facilities make use of hardware-based recog&tion, editing, and storage servicesprovided by the Newton. SmartPad can be co&wed to transmit shapes to the back end over the seriai line as soon as they are recognized, or to save them while the PDA is
(cl Fig. 4. Recognition of a simple diagram.
03
fd
PDA-basedgraphicalinterchange Host
PDA Drawing New Shape(circle,
1)
%
Display
Drawing New Shape(line, 2) c
t
Display t
Associated text([l,2],“sunset”)
Fig. 5. Exchange of messages betweenPDA and host.
operating remotely and then download them all at once when it is reconnected. SmartPad also organizes diagrams by combining them into pages, which the user can page through and edit at a later time. The SmartPad front end also allows the Newton to function as an output device by accepting information from the back end and displaying text, shapes, or other graphical information. Because it has both input and output functions, two Newtons running SmartPad can be connected together and will function as a cooperative work environment, in which any drawing done on one PDA is displayed on the other. We describe this cooperative drawing application later in the paper. SmartPad also receivesinformation from the back end, and displays that information on the Newton’s screen. Asynchronously, and concurrently with its other functions, SmartPad listens on the Newton’s serial port for input from the back end. This information may include shape descriptors, text descriptors, and editing operations. For example, the high-level recognizers in the back end may identify configurations of elements in the drawing. The back end may also be connected to databases, simulations, or other applications and may deliver information from these applications back to the SmartPad front end. The back end program may also transmit editing commands, either from editing operations performed by a user on the Macintosh (host) side or by another PDA client. When a command is read from the port and decoded, SmartPad performs the appropriate operation, which may include updating the graphics and/or the internal representation of the diagram. 4.2. Back end subsystem The program running on the host computer back end is the Electronic Cocktail Napkin program, a trainable diagram recognition system. The system takes strokes and shapes uploaded by the PDA,
645
combines them into aggregates based on their spatial relations. Since SmartPad also transmits the raw digital ink to the back end as a point stream, the Cocktail Napkin has the option of re-recognizing the input shapes based on other information that may not be available to the front end, including domain information and information based on the shape’s context. The spatial relations recognized by the Electronic Cocktail Napkin may be specified through example, from a database, or by direct entry through a relation editor. When a group of shapes and the spatial relations among them are recognized by the Cocktail Napkin, it identifies the group, assigns the group a name, and downloads that information to the front end, where SmartPad displays it in association with the corresponding shapes. Shapes may also be displayed, entered, and edited directly into the Cocktail Napkin, either with a mouse or with a conventional digitizing tablet, in which case they are downloaded for display onto the Newton. The Cocktail Napkin has been used as front end for diagram based query for visual and other databases, for constraint based diagram editing [2], and for support of conceptual and creative design [3], and has been linked to visual databases and case based design tools, interactive design simulations, and Netscape. Although the Cocktail napkin is capable of performing its own stroke, shape, and handwriting recognition, when the Newton is used as the input device, it does not, but rather makes use of the higher-level messages transmitted by the Newton. When the user draws on the Newton PDA, the Cocktail Napkin receivesthe recognized shapes, and higher level recognizers parse the drawing (that is, the set of shapes, or glyphs, received so far) for configurations of glyphs in certain spatial relations. The Cocktail Napkin monitors the serial line and detects when the Newton begins to send data according to the protocol described below. The Napkin interprets the protocol and creates its own representation of the recognized diagram. Since the communication protocol also contains the raw stroke (that is, point stream) information, the Cocktail Napkin, if so configured, may employ that information to “second guess” the Newton, and make a new attempt at recognition. The system can be configured to suppress the stroke information, which leads to more efficient communication, but no longer allows the Cocktail Napkin the possibility of doing its own secondary low-level recognition. When the Cocktail Napkin recognizes a high-level construct, or rerecognizes a shape, it reports it to the Newton, which updates its own internal representations of the diagram and displays the updated diagram, as in Fig. 4. 5. TJixlmxcAL CHALLENGES A number of technical challenges remain to be solved in implementing the graphical interchange system. Some of the challenges will no doubt be
646
W. V. Chin and M. D. Gross
solved through the introduction of a new generation of Newton PDAs. Others must be solved in the current implementation. One issue that will ultimately be solved in future generations of Newton technology is the issue of the small screen size and low screen resolution of current Apple Newton MessagePad. In current models, the display is 3” x4-1/8”. However, the Newton represents a classof platforms, not a single one, and future Newton-based devices are under development with 8” x 11” display, which should be adequate for most problems. Similarly, we anticipate devices whose displays have resolution approaching those of modern laptop computers. Currently, we plan to address the problems of screen size and resolution by displaying graphics at a size providing maximum detail and including a scrolling facility to navigate graphics larger than the Newton screen. A second problem is the slow speed of the current Newton processor, a 25MHz ARM 610 chip. In current units, recognition, display, and memory access are noticeably slow. However, a 100MHz chip known as the StrongARM has been jointly developed by Digital Equipment Corporation and Advanced RISC Machines, Ltd., which may be included in future generations of Newtons, in which case we expect the speed problem to go away. A third problem with the current generation of Newtons is the speed of their communications. Our experience indicates that the 9600 baud serial connection is too slow for many of the activities we anticipate in the graphical interchange system. As mentioned earlier, we plan to investigate higherbandwidth wireless communication, infrared communication, and AppleTalk. In addition, future versions of the Newton operating system will support a full TCP/IP stack, allowing the use of highbandwidth network communication between the front and back ends. We intend to exploit that technology when it becomes available. Among the challenges that must be addressed in the graphical interchange system while employing the current generation of technology is the dynamic delivery of graphics. Most of the Newton applications we have identified are text-only. When they do employ graphics, the graphics are compiled into the application. However, graphics can be received and manipulated by the Newton in the form of Macintosh PICT resources, and we plan to employ this technique to receive and render graphics. The current version of the system will exchange uncompressed graphics; future versions will employ data compression to increase bandwidth. Currently, the Newton MessagePad provides a strictly black-and-white display, without intermediate shades of gray (Fig. 6). While future generations may provide gray scale or even color, we feel that it is desirable to provide gray scaleor something like it, in order to support scenarios like those illustrated in. One possibility is to synthesize a half-tone rendering of an enlarged version of the graphics, but this may
Fig. 6. Black-and-whitedisplayon the current Newton
lead to unacceptably low resolution. Another possibility is to dither the image, but the display or the processor may be too slow for such a strategy. In such a case, we plan to concentrate on black-andwhite diagrams such as those in Fig. 2. Finally, there are a number of issues concerning the synchronization of the PDA front end and the host-based back end that must be addressed. In order to transmit shape and text editing operations between devices, each shape is assigned a unique identifier when it is created. Subsequent editings operations refer to the identifier of the shape on which the operation is performed, and all devices (which may include multiple PDAs as we11 as the back end) receiving the messageperform the operation on their copy of the shape. Since the elementsof the system may all operate in a disconnected mode (for example, a serviceworker may use a PDA in the field, where it is not connected to the back end), to guarantee that shape identifiers are always unique, we must assign a unique identifier to each device and host running the system, and incorporate the device identifier of the device on which a given shape originated into the shape identifier. Since all devices in a single graphical interchange system will be used within a single organization (the local telephone company, for example), it is not unreasonable for some -centraladministrator to assign such iden~rs when the system is first configured, or when new devices are added to the system. A related problem concerns synchronization of page numbers between devices. Again, we can incorporate the device identifier of the 0~~~~~~ into~page identifiers, and associate those identtfiers with each copy of the page across the system.
PDA-basedgraphicalinterchange 6. RELATED
PROJECTS
In addition to the graphical interchange system, we are working on a number of other projects that employ the basic architecture described in this paper. The tlrst project is a PDA-based collaborative drawing environment. The protocol used by the PDAs and the host reflects changes to the current state of the diagram shared by all devices on the system. Thus, multiple PDAs may be connected and images drawn or edited on one PDA will appear on all the others. Figure 7 illustrates a scenario similar to that of Figs 4 and 5, except that each picture element is drawn on a different PDA. Such a system will be useful for designers located at widely separated locations. A number of such systems have already been designed and developed [4], but our PDA-based system has the advantages of using inexpensive equipment and allowing the users to do work in the field and later combine their diagrams. The system can be enhanced by hooking the PDAs to hosts acting as network gateways, thereby allowing the PDAs to communicate over long distance networks, and by allowing a host running the Electronic Cocktail Napkin to eavesdrop on the network and act as a mediator in the conversation, supplying additional information as parts of the diagram being drawn are recognized. The second related project uses a PDA to add a pen-based interface to a mouse-and-palette-based graphical editor. Employing a pen-based interface in a graphical editor leads to improved user performance, fewer errors, and greater satisfaction [5, 61. Rather than attempt to implement our own point capture and shape recognition routines, or to incorporate existing libraries, we reuse the shape recognition facilities of the Newton. We have modified a graphical editor for the VIPR visual programming language [?‘I. The editor has a conventional mouse-and-palette interface, but also offers keyboard-based shortcuts that support the same
repertoire of editing operation operations relative to a current graphical selection. We have modified the editor to monitor the serial port of the machine on which it is running. We have developed an application for the Newton that allows the user to draw VIPR programs. The Newton recognizes the graphical elements, determines the desired editor operation and current selection, and translates that into a message packet which is sent to the graphical editor over a serial connection. The editor receives the message and performs the operation. The modifications are much simpler than the alternative approaches mentioned above, and again have the additional advantage of allowing disconnected use with subsequent uploading of editing operations as a group. Figure 8 shows an example of the use of the modified VIPR editor with the PDA-based interface. 7. RELATED
* New Shape
(circle) Display
Display
4
WORR
We do not know of other efforts to distribute low and high level recognition of hand drawn input across a PDA and a back end host computer. However, we discusshere three categories of related work: pen-based drawing and recognition, computer supported collaborative work (CSCW) and distributed drawing environments, and other projects using PDA’s. Low-level (i.e. single stroke, single glyph) recognizers for hand drawn diagrams similar to the Newton’s built in recognizer have been built, for example by Rubine [8], Apte and Kimura [S], and Zhao [9]. Any of these algorithms could be used for the low level PDA recognition, but the Newton’s built in recognizer is adequate and it is convenient. We pass strokes the Newton cannot recognize to the Napkin’s trainable recognizer, which can be customized interactively to recognize domain specific marks and gestures. Higher level recognition and parsing [lo, Ill, which in our distributed architecture takes place
PDA
PDA
Drawing
641
b Drawing 4
Display 4
New shape
(line)
4 Display +
Fig. 7. Communication betweentwo PDAs in a collaborativedrawing environment. (Shapeidentifiers have been omitted.)
W. V. Citrin and M. D. Gross
648
td)
Cc)
Fig. 8. Use of graphicaleditor with Newton front end. (a) Initial Newton conf@ration. (b) Initial editor display. (c) Input of new element before recognition. (d) Result of editing operation.
on the back end host computer, features in diagram editors for visual programming, but these parsers typically do not work on hand drawn input. Landay and Myers’ SILK program [12] does parse handdrawn diagrams of user interface designs. Distributed drawing environments such as ClearBoard [13] and Commune [14] enable two or more users to share a virtual drawing surface, but these systems do not attempt recognition, nor do they distribute different levels of functionality across connected hardware. PDAs are often used in conjunction with other hardware devices, which leads to interesting multidevice user interface design problems. For example, Robertson et al. describe a PDA and TV-display interface for real estate agents [IS]. In their system the user interacts only with the PDA, which controls the display of still images and video on the TV. Most PDA systems in commercial use do not use the PDA for entering diagrams, but for text and digital ink signatures (as in the system that United Parcel Service drivers use for logging deliveries), or with an even more limited button-menu interface. 8. CURRENT
STATUS
AND
CONCLUSIONS
We have described a system that supports the interchange of graphical information for fiefd service
workers. The system incorporates a PDA (in our case, an Apple Newton ) as the mobile front end. The PDA performs low-level point capture and stroke shape, and handwriting recognition using built-in fa&ties, and can either store the results for later use, or can upload them to a more sophisticated host-based recognizer for further processing. The system will give field serviceworkers accessto- timely and accurate graphical information, and will allow them to make annotations and updates to the information in a convenient and transparent way. The graphical interchange system builds on a more general two-stage diagram recognition architecture using a PDA as the front end and a more powerful host as the back end. We are currently building two applications on top of this architecture: a colhtborative drawing environment and a pen-based front end to a graphical editor. There are a number of advantages to our approach. While the PDA has limited computing power to perform domain-based diagram recognition, it does support built-in shape and handwriting recognition, thereby allowing these functions to be performed in the unit, where sufTioient computing power to perform the tasks does exist. In addition, performing these frequent operations in the unit avoids communications delays and the expense of constantly keeping the communication channel open. The fact that the diagrams can also be stored in the unit means that annotated diagrams usually need not be uploaded immediately, but may be stored until the worker returns to the home base, in which case they may be uploaded through a fast connection or over a slower telephone line in a situation where speed of response is not as important. We currently anticipate that the graphical interchange system described here will be completed in the Fall of 1997. authors wish to thank Ellen Do, Kyle Kuczun, and Adrian Warmaek of me University of Colorado Schoolof EnvironmentalDesign for their work on the ElectronicCocktail Napkin and the Newton-Napkin interface,and Paul Bauerof US WEST Advanced Teelmologiesfor his commentsand encouragement.This work was supportedin part by the NEC Corporation. Acknowledgements-The
RF#FERENcEs
1. Gross, M. D., The Electronic
Cocktail
Napkin-
computer support for working with diagrams.D&xn St&es, 1996,-17(l), 53-69.
-
2. Gross. M. D.. Stretch-A-Sketch:a dvnamic diaarammer. in IEEE Symlposium on Vkuai Languages- (YL ‘94), St. Louis, 1994, pp. 232-238. 3. Gross, M. D. and Do, E. Y.-L., Shape-baaed reminding in creative design. In Compater-Ai&d Arc&fectural DesignFutures (CAAD Futures ‘92). Singapore,1995, pp. l-11. 4. Greenberca, S.,Hayne,S. and Rada, R., eds., Groupware for Real-The Drawing: A Designer’s Gut&. McGrawHill, London, 1995.
5. Apte, A. and Kimura, T. D., A comparisanstudy of the pen and the mouse in editing graphic diagrams. In IEEE Symposium on VisuaJ Lmwges Bergen, Norway, 1993, pp. 352-357.
(VL
W),
PDA-based graphical interchange 6. Citrin, W. V., Requirements for graphical front ends for visual languages. In IEEE Symposium on Visual Lunguages (VL ‘fi), Bergen, Norway, 1993, pp. 142-151. 7. Citrin. W.. Santiaeo. C. and Zom. B.. Scalable interfaces to suppo;progmm comprehension. In IEEE Workshop on Program Comprehension, Berlin, 1996, pp. 123-132. 8. Rubine, D., Specifying gestures by example. Computer Graphics, 1991,X!, 329-337. 9. Zhao, R., Incremental recognition in gesture-based and syntax-directed diagram editors. In ACM SIGCHZ Conference on Human Factors in Computing (INTERCHI ‘93), Amsterdam, 1993, pp. 95-100. 10. Lakin, F., Visual grammars for visual languages. In National Conference on Art#icial InteNigence (AAAI ‘87), Seattle, 1987, pp. 683-688. 11. Helm, R., Marriott,- K. and Odersky, M., Building visual parsers. In ACM SIGCHI Conference on Human Factors in Computing Systems (CHI PI), New Orleans, 1991, pp. 105-112.
649
12. Landay, J. A. and Myers, B. A., Interactive sketching for the early stages of interface design. In ACM SIGCHI Conference on Human Factors in Computing Systems (CHI ‘95), Denver, 1995, pp. 43-50. 13. Ishii, H. and Kobayashi, M., ClearBoard: a seamless medium for shared drawing and conversation with eye contact. In ACM SIGCHI Conference on Human Factors in Computing Systems (CHI ‘92), Monterey, CA, 1992, pp. 525-532. 14. Minneman, S. L. and Bly, S. A., Managing a trois: a study of a multi-user drawing tool in distributed design work. In ACM SIGCHI Conference on Human Factors in Computing Systems (CHI PI), New Orleans, 1991, pp. 217-224. 15. Robertson, S., Wharton, C., Ashworth, C. et al.. Dual device interface design: PDAs and interactive television. In ACM SIGCHI Conference on Human Factors in Computing Conference (CHI ‘96), Vancouver, BC, 1996, pp. 79-86.
Pergamon
& Graphics, Vol. 20, No. 5, pp. 641-649, 1996 Copyright 0 1996 Elsevk Science Ltd Printed in Great Britain. All rights reserved 0097-8493/96 $15.00+0.00
PIIt soo97-8493(~ooo38-6 Mobile Computing
PDA-BASED GRAPHICAL INTERCHANGE FOR FIELD SERVICE AND REPAIR WORKERS WAYNE V. CITRIN’+ and MARK
D. GROSS’
‘Deparhnent
of Electrical and Computer Engineering, University of Colorado, Boulder, CO 80309-0425, U.S.A. e-ma2 [email protected] ‘School of Environmental Design, University of Colorado, Boulder, CO, U.S.A.
Abstract-We present an ongoing project to develop a system to provide field service.workers with timely and accurate service information. The system will allow workers to download diagrams or photographs from a host computer’s central database onto a PDA. The workers will be able to annotate the diagrams to reflect work performed, and later upload the annotations to the host computer, where they will be integrated into an updated database. Diagram recognition functionality is distributed between the PDA (which performs low-level shape and handwriting recognition) and the host computer (which performs high-level domain-based diagram recognition). Distributing the functionality offers a number of advantages: it allows the relatively resource-poor PDA to be part of a Powerful diagram recognition environment, it allows the use of standardized hardware-based recognition facilities in a domain-based recognition system, and it allows off-line drawing recognition and storage of diagrams, thereby avoiding excessive use of slow or expensive communications channels. Copyright 0 1996 Elsevier Science Ltd
1. INTRODUCTION
through
use
of
personal
tions back to the central
digital
assistants
database
for use by the
next worker. PDAs (the Apple Newton MessagePad, for example) are an appropriate technology for this task, since they are small and lightweight, support display of graphic information, allow penbased input (desirable for graphical data entry and also desirable when use of a keyboard is inappropriate), and support both wireless and wired communication.
connectors, splices, and equipment. They carry microfiche maps in their trucks, but these are maintained and produced by a central office and are likely to be out of date. (Some service locations maintain their own map sets for this reason.) To understand the on-site configuration, a good field worker often makes a diagram on a scrap of paper, drawing first the “as is” and then the “change to” configurations. After the job is finished, the worker discards the paper and proceeds to the next job. The updates never make it back to the central repository, and the process must proceed all over again when the next repair or maintenance activity is performed. We are addressing this and similar problems
t Author for correspondence..
the
(PDAs) to deliver graphical information to field workers from the central office, allow the workers to record their modifications, and upload the modifica-
As infrastructure systems (such as telephone and computer networks, electrical power grids, and building support systems) become more complex, and as more workers are involved in their support and maintenance, it becomes even more important to provide field service workers with timely, accurate, and complete information. Delivery of such information can provide savings to industries and customers, and may even affect public safety. The situation of US WEST, the local telephone company in Colorado, represents a case in point. Field service workers are dispatched from a service location to service communications equipment in the field. When they arrive at the job site, they find a confusing and often undocumented configuration of
The system employs diagram recognition, which is distributed between the PDA (which performs lowlevel shape and handwriting recognition) and the host computer (which performs high-level domainbased diagram recognition). Distributing the functionality in this way offers a number of advantages: it allows the relatively resource-poor PDA to be part of a powerful diagram recognition environment (something that would probably not be possible on the PDA alone), it allows the use of standardized hardware-based recognition facilities in a domainbased recognition system (desirable since the lowlevel shape and handwriting recognition functions do not change drastically from one domain to the next), and it allows off-line shape and handwriting recognition and storage of diagrams, thereby avoiding excessiveuse of slow or expensive communicati&s channels. This paper begins by motivating the design of our system through three scenarios which illustrate the uses to which such a system can be put. It then
641
642
W. V. Citrin and M. D. Gross
describes the underlying architecture for such a system, both at a high level and in detail, discusses technical challenges that remain to be solved, reviews some related projects we are conducting that employ the same underlying distributed architecture for diagram recognition, and reviews related literature.
2. MOTIVATING
SCENARIOS
The use of the PDA-based graphical interchange system can best be explained by the following three scenarios. Subsequent sections will explain the system’s organization and illustrate how the system supports the various scenarios. The first scenario indicates a way in which PDAbased graphical interchange can assist field workers in understanding complex hardware configurations, and can provide them with easy ways to update the information to reflect the changes they make, thus keeping the information up to date. Figure l(a) shows a tangle of hardware encountered by a field service worker on a call. With out-of-date information, it may be extremely time consuming for the worker to understand the hardware or the modifications that have already been made. However, if the worker has downloaded information from the central database onto a PDA, he or she can consult the notes of the previous worker who serviced the hardware [Fig. l(b)]. The worker can then make the required changes and update the photograph of the hardware with additional annotations. These additional annotations are entered through the PDA’s pen-based interface, and are recorded on a new virtual ‘layer’, which is later uploaded to the central database. In this way, future workers can consult the layers with the notes of individual workers, or can group several layers together to see the effects of a sequence of service calls.
The second scenario illustrates the way in which diagram recognition can be incorporated into a graphical interchange system. In this case, the worker sketches a schematic diagram indicating the current configuration or work that was performed. Using the graphical interchange system, this diagram can be sketched on the PDA rather than on paper. This offers a number of advantages. First, the diagram is now in a digital form that can be easily uploaded into the central database. If the diagram was sketched on paper, there is more possibility that the diagram will be lost, or that the worker will forego the added effort of later putting the diagram through a scanner. Second, the diagram is not just an image but rather a collection of shapes grouped hierarchically into components. As such, the diagram can be further edited, either by the original worker on site, or by subsequent workers. Finally, since the diagram is such a collection of shapes and text, further diagram recognition operations can be performed on it, in order to assign meaning to the diagram and to determine the diagrammed system’s correctness, estimate the performance of the system, etc. This further recognition need not be performed on site by the PDA (in fact, it is unlikely that it will, since the PDA probably lacks sufficient computing power for this task), but rather will be performed on some back end host to which the diagram is uploaded. The next section describes the relation between the front-end PDA and the back-end host when diagram recognition is performed. Figure 2(a) shows a possible diagram as sketched by a worker on the PDA. Figure 2(b) shows a possible result of applying diagram recognition to the sketch. The third scenario is actually an extension of the first, and illustrates the way in which a worker may enter new images into the system. In this case, the worker visits a site for which detailed records or
Fig. 1. (a) Worker finds this tangle of equipment on site. (b) Systemdeliversthis photo annotated by previousworker.
PDA-basedgraphicalinterchange
643 To Higher Floors of EWD
BuiIding
(a) Fig. 2. (a) Telecommunicationsworker makesthis diagram on site. (b) Host convertsand storesit for future use.
The Newton also contains support through communications, either through the built-in serial and infrared ports, or through optional telephone and wireless modems. Finally, the Newton contains onboard memory, expandable to several megabytes through PCMCIA cards. The back end host is currently a Macintosh computer running a diagram recognition program called the Electronic Cocktail Napkin [l]. This system accepts shapes and text recognized by the PDA and groups them into higher-level domainspecific aggregates. The system can be configured so that the Cocktail Napkin informs the PDA of the recognition results, which are displayed. The Electronic Cocktail Napkin is described in more detail in 3. SYSTEM OVERVIEW Section 4.2. The graphical interchange system is composed of Figure 3 shows the high-level organization of the two components: a PDA-based front end, and a host- graphical interchange system, and the distribution of based back end. functionality between the components. The PDA The front end is carried into the field by service captures strokes entered by the user and converts workers, who can download data from the host them into shapes through the built-in recognition before leaving their base, or who can download data facilities. These recognized items are either stored in in the field over telephone lines through an attachable the PDA for later transmission, or, if the PDA is modem. In the future, we plan to employ wireless connected to the host, are uploaded as they are communications for the exchange of data. The PDA entered and recognized. The back end host receives (an Apple Newton MessagePad) provides an LCD the shapes and text and, depending on the spatial screen on which the user may draw with a stylus. The relationships between the elements, recognizes highpen strokes entered by the user are captured by the er-level aggregates, depending on the domain for Newton and redisplayed as ‘ink’. The PDA contains which the host is configured. If the system is so built-in shape recognition facilities, which can configured, and the PDA is connected while recognirecognize simple, domain-independent shapesinclud- tion is going on, the host will inform the PDA of the ing circles, triangles, rectangles, closed and open results of the recognition, which are then displayed. curves, lines, and closed and open polygons, as well Figure 4 shows the operation of the current as a trainable handwriting recognition facility and prototype implementation of the system on a simple the ability to edit recognized shapes and text. The set of glyphs. In Fig. 4(a), the user enters a circle on Newton is programmable, and the aforementioned the PDA. In Fig. 4(b), the PDA recognizes it as a facilities can be easily incorporated into the program. circle, and the back end is notified (assuming the
images do not yet exist. In this case, the user may wish to generate an image and annotate it for future reference. In such a case, the worker will photograph the site with a digital camera, download the image into the PDA through the PDA’s serial port, and annotate it using the pen-based interface as described in the first scenario. As in the first two scenarios, this offers the advantages of allowing the user to easily make annotations to digital images rather than paper, preserves the structure (image and multiple annotation layers) for storage in a database, and does not require the worker to go to the extra effort of preserving his or her paper notes and later entering them into the central system through a scanner.
W. V. Chin PDA front end
recognized
shapes
and M. D. Gross
Host computer
arid
Annotations indicating compound objects
High-level diagram re&nitiin, diagram management and storage
Basic shape and handwriting recognition editing functions
/
Fig. 3. Organization
of distributed recognition system
back end is connected). In Fig. 4(c), a line is drawn, and in Fig. 4(d), the PDA recognizes it as a line, and it again notifies the back end of what it recognized and the location of the recognized item. The back end has been configured to recognize the circle over the line as the ‘sunset’ glyph; it notifies the PDA of the recognition and the result is displayed [Fig. 4(e)]. Figure 5 shows the exchange of messagesbetween the PDA and the host. Newton PDAs were chosen as the front end of the system for a number of reasons. Newtons are fairly small and light. Their pen-based input allows them to be used in the field where use of a keyboard would be awkward. Their built-in stroke capture and recognition facilities let us avoid the non-trivial task of reimplementing such facilities. Most importantly, Newtons are much less expensive than laptop computers, making it much more likely that companies will adopt PDA technology than laptops. It is true that laptops with pen-based entry capability currently exist that have more processing power and better resolution screens than current PDAs. How-
(a)
(b)
back end
ever, their larger size and greater expense led us Eo explore more economical solutions. In addition, future PDA models should address the problems of processing power and screen resolution (see Section 5). 4. SYsTEkI
DETAILS
4.1. PDA subsystem We have developed a program called Smart&d that runs on the Newton as a basic front end. SmartPad supports stroke and shape recognition, and captures and saves digital ink, as well as the recognized shapes. It supports basic gesture-based editing, including erasing, copying, moving, and changing the shapes of elements. Finally, it supports text entry, recognition, and editing, either through handwriting or through a soft keyboard. All of these facilities make use of hardware-based recog&tion, editing, and storage servicesprovided by the Newton. SmartPad can be co&wed to transmit shapes to the back end over the seriai line as soon as they are recognized, or to save them while the PDA is
(cl Fig. 4. Recognition of a simple diagram.
03
fd
PDA-basedgraphicalinterchange Host
PDA Drawing New Shape(circle,
1)
%
Display
Drawing New Shape(line, 2) c
t
Display t
Associated text([l,2],“sunset”)
Fig. 5. Exchange of messages betweenPDA and host.
operating remotely and then download them all at once when it is reconnected. SmartPad also organizes diagrams by combining them into pages, which the user can page through and edit at a later time. The SmartPad front end also allows the Newton to function as an output device by accepting information from the back end and displaying text, shapes, or other graphical information. Because it has both input and output functions, two Newtons running SmartPad can be connected together and will function as a cooperative work environment, in which any drawing done on one PDA is displayed on the other. We describe this cooperative drawing application later in the paper. SmartPad also receivesinformation from the back end, and displays that information on the Newton’s screen. Asynchronously, and concurrently with its other functions, SmartPad listens on the Newton’s serial port for input from the back end. This information may include shape descriptors, text descriptors, and editing operations. For example, the high-level recognizers in the back end may identify configurations of elements in the drawing. The back end may also be connected to databases, simulations, or other applications and may deliver information from these applications back to the SmartPad front end. The back end program may also transmit editing commands, either from editing operations performed by a user on the Macintosh (host) side or by another PDA client. When a command is read from the port and decoded, SmartPad performs the appropriate operation, which may include updating the graphics and/or the internal representation of the diagram. 4.2. Back end subsystem The program running on the host computer back end is the Electronic Cocktail Napkin program, a trainable diagram recognition system. The system takes strokes and shapes uploaded by the PDA,
645
combines them into aggregates based on their spatial relations. Since SmartPad also transmits the raw digital ink to the back end as a point stream, the Cocktail Napkin has the option of re-recognizing the input shapes based on other information that may not be available to the front end, including domain information and information based on the shape’s context. The spatial relations recognized by the Electronic Cocktail Napkin may be specified through example, from a database, or by direct entry through a relation editor. When a group of shapes and the spatial relations among them are recognized by the Cocktail Napkin, it identifies the group, assigns the group a name, and downloads that information to the front end, where SmartPad displays it in association with the corresponding shapes. Shapes may also be displayed, entered, and edited directly into the Cocktail Napkin, either with a mouse or with a conventional digitizing tablet, in which case they are downloaded for display onto the Newton. The Cocktail Napkin has been used as front end for diagram based query for visual and other databases, for constraint based diagram editing [2], and for support of conceptual and creative design [3], and has been linked to visual databases and case based design tools, interactive design simulations, and Netscape. Although the Cocktail napkin is capable of performing its own stroke, shape, and handwriting recognition, when the Newton is used as the input device, it does not, but rather makes use of the higher-level messages transmitted by the Newton. When the user draws on the Newton PDA, the Cocktail Napkin receivesthe recognized shapes, and higher level recognizers parse the drawing (that is, the set of shapes, or glyphs, received so far) for configurations of glyphs in certain spatial relations. The Cocktail Napkin monitors the serial line and detects when the Newton begins to send data according to the protocol described below. The Napkin interprets the protocol and creates its own representation of the recognized diagram. Since the communication protocol also contains the raw stroke (that is, point stream) information, the Cocktail Napkin, if so configured, may employ that information to “second guess” the Newton, and make a new attempt at recognition. The system can be configured to suppress the stroke information, which leads to more efficient communication, but no longer allows the Cocktail Napkin the possibility of doing its own secondary low-level recognition. When the Cocktail Napkin recognizes a high-level construct, or rerecognizes a shape, it reports it to the Newton, which updates its own internal representations of the diagram and displays the updated diagram, as in Fig. 4. 5. TJixlmxcAL CHALLENGES A number of technical challenges remain to be solved in implementing the graphical interchange system. Some of the challenges will no doubt be
646
W. V. Chin and M. D. Gross
solved through the introduction of a new generation of Newton PDAs. Others must be solved in the current implementation. One issue that will ultimately be solved in future generations of Newton technology is the issue of the small screen size and low screen resolution of current Apple Newton MessagePad. In current models, the display is 3” x4-1/8”. However, the Newton represents a classof platforms, not a single one, and future Newton-based devices are under development with 8” x 11” display, which should be adequate for most problems. Similarly, we anticipate devices whose displays have resolution approaching those of modern laptop computers. Currently, we plan to address the problems of screen size and resolution by displaying graphics at a size providing maximum detail and including a scrolling facility to navigate graphics larger than the Newton screen. A second problem is the slow speed of the current Newton processor, a 25MHz ARM 610 chip. In current units, recognition, display, and memory access are noticeably slow. However, a 100MHz chip known as the StrongARM has been jointly developed by Digital Equipment Corporation and Advanced RISC Machines, Ltd., which may be included in future generations of Newtons, in which case we expect the speed problem to go away. A third problem with the current generation of Newtons is the speed of their communications. Our experience indicates that the 9600 baud serial connection is too slow for many of the activities we anticipate in the graphical interchange system. As mentioned earlier, we plan to investigate higherbandwidth wireless communication, infrared communication, and AppleTalk. In addition, future versions of the Newton operating system will support a full TCP/IP stack, allowing the use of highbandwidth network communication between the front and back ends. We intend to exploit that technology when it becomes available. Among the challenges that must be addressed in the graphical interchange system while employing the current generation of technology is the dynamic delivery of graphics. Most of the Newton applications we have identified are text-only. When they do employ graphics, the graphics are compiled into the application. However, graphics can be received and manipulated by the Newton in the form of Macintosh PICT resources, and we plan to employ this technique to receive and render graphics. The current version of the system will exchange uncompressed graphics; future versions will employ data compression to increase bandwidth. Currently, the Newton MessagePad provides a strictly black-and-white display, without intermediate shades of gray (Fig. 6). While future generations may provide gray scale or even color, we feel that it is desirable to provide gray scaleor something like it, in order to support scenarios like those illustrated in. One possibility is to synthesize a half-tone rendering of an enlarged version of the graphics, but this may
Fig. 6. Black-and-whitedisplayon the current Newton
lead to unacceptably low resolution. Another possibility is to dither the image, but the display or the processor may be too slow for such a strategy. In such a case, we plan to concentrate on black-andwhite diagrams such as those in Fig. 2. Finally, there are a number of issues concerning the synchronization of the PDA front end and the host-based back end that must be addressed. In order to transmit shape and text editing operations between devices, each shape is assigned a unique identifier when it is created. Subsequent editings operations refer to the identifier of the shape on which the operation is performed, and all devices (which may include multiple PDAs as we11 as the back end) receiving the messageperform the operation on their copy of the shape. Since the elementsof the system may all operate in a disconnected mode (for example, a serviceworker may use a PDA in the field, where it is not connected to the back end), to guarantee that shape identifiers are always unique, we must assign a unique identifier to each device and host running the system, and incorporate the device identifier of the device on which a given shape originated into the shape identifier. Since all devices in a single graphical interchange system will be used within a single organization (the local telephone company, for example), it is not unreasonable for some -centraladministrator to assign such iden~rs when the system is first configured, or when new devices are added to the system. A related problem concerns synchronization of page numbers between devices. Again, we can incorporate the device identifier of the 0~~~~~~ into~page identifiers, and associate those identtfiers with each copy of the page across the system.
PDA-basedgraphicalinterchange 6. RELATED
PROJECTS
In addition to the graphical interchange system, we are working on a number of other projects that employ the basic architecture described in this paper. The tlrst project is a PDA-based collaborative drawing environment. The protocol used by the PDAs and the host reflects changes to the current state of the diagram shared by all devices on the system. Thus, multiple PDAs may be connected and images drawn or edited on one PDA will appear on all the others. Figure 7 illustrates a scenario similar to that of Figs 4 and 5, except that each picture element is drawn on a different PDA. Such a system will be useful for designers located at widely separated locations. A number of such systems have already been designed and developed [4], but our PDA-based system has the advantages of using inexpensive equipment and allowing the users to do work in the field and later combine their diagrams. The system can be enhanced by hooking the PDAs to hosts acting as network gateways, thereby allowing the PDAs to communicate over long distance networks, and by allowing a host running the Electronic Cocktail Napkin to eavesdrop on the network and act as a mediator in the conversation, supplying additional information as parts of the diagram being drawn are recognized. The second related project uses a PDA to add a pen-based interface to a mouse-and-palette-based graphical editor. Employing a pen-based interface in a graphical editor leads to improved user performance, fewer errors, and greater satisfaction [5, 61. Rather than attempt to implement our own point capture and shape recognition routines, or to incorporate existing libraries, we reuse the shape recognition facilities of the Newton. We have modified a graphical editor for the VIPR visual programming language [?‘I. The editor has a conventional mouse-and-palette interface, but also offers keyboard-based shortcuts that support the same
repertoire of editing operation operations relative to a current graphical selection. We have modified the editor to monitor the serial port of the machine on which it is running. We have developed an application for the Newton that allows the user to draw VIPR programs. The Newton recognizes the graphical elements, determines the desired editor operation and current selection, and translates that into a message packet which is sent to the graphical editor over a serial connection. The editor receives the message and performs the operation. The modifications are much simpler than the alternative approaches mentioned above, and again have the additional advantage of allowing disconnected use with subsequent uploading of editing operations as a group. Figure 8 shows an example of the use of the modified VIPR editor with the PDA-based interface. 7. RELATED
* New Shape
(circle) Display
Display
4
WORR
We do not know of other efforts to distribute low and high level recognition of hand drawn input across a PDA and a back end host computer. However, we discusshere three categories of related work: pen-based drawing and recognition, computer supported collaborative work (CSCW) and distributed drawing environments, and other projects using PDA’s. Low-level (i.e. single stroke, single glyph) recognizers for hand drawn diagrams similar to the Newton’s built in recognizer have been built, for example by Rubine [8], Apte and Kimura [S], and Zhao [9]. Any of these algorithms could be used for the low level PDA recognition, but the Newton’s built in recognizer is adequate and it is convenient. We pass strokes the Newton cannot recognize to the Napkin’s trainable recognizer, which can be customized interactively to recognize domain specific marks and gestures. Higher level recognition and parsing [lo, Ill, which in our distributed architecture takes place
PDA
PDA
Drawing
641
b Drawing 4
Display 4
New shape
(line)
4 Display +
Fig. 7. Communication betweentwo PDAs in a collaborativedrawing environment. (Shapeidentifiers have been omitted.)
W. V. Citrin and M. D. Gross
648
td)
Cc)
Fig. 8. Use of graphicaleditor with Newton front end. (a) Initial Newton conf@ration. (b) Initial editor display. (c) Input of new element before recognition. (d) Result of editing operation.
on the back end host computer, features in diagram editors for visual programming, but these parsers typically do not work on hand drawn input. Landay and Myers’ SILK program [12] does parse handdrawn diagrams of user interface designs. Distributed drawing environments such as ClearBoard [13] and Commune [14] enable two or more users to share a virtual drawing surface, but these systems do not attempt recognition, nor do they distribute different levels of functionality across connected hardware. PDAs are often used in conjunction with other hardware devices, which leads to interesting multidevice user interface design problems. For example, Robertson et al. describe a PDA and TV-display interface for real estate agents [IS]. In their system the user interacts only with the PDA, which controls the display of still images and video on the TV. Most PDA systems in commercial use do not use the PDA for entering diagrams, but for text and digital ink signatures (as in the system that United Parcel Service drivers use for logging deliveries), or with an even more limited button-menu interface. 8. CURRENT
STATUS
AND
CONCLUSIONS
We have described a system that supports the interchange of graphical information for fiefd service
workers. The system incorporates a PDA (in our case, an Apple Newton ) as the mobile front end. The PDA performs low-level point capture and stroke shape, and handwriting recognition using built-in fa&ties, and can either store the results for later use, or can upload them to a more sophisticated host-based recognizer for further processing. The system will give field serviceworkers accessto- timely and accurate graphical information, and will allow them to make annotations and updates to the information in a convenient and transparent way. The graphical interchange system builds on a more general two-stage diagram recognition architecture using a PDA as the front end and a more powerful host as the back end. We are currently building two applications on top of this architecture: a colhtborative drawing environment and a pen-based front end to a graphical editor. There are a number of advantages to our approach. While the PDA has limited computing power to perform domain-based diagram recognition, it does support built-in shape and handwriting recognition, thereby allowing these functions to be performed in the unit, where sufTioient computing power to perform the tasks does exist. In addition, performing these frequent operations in the unit avoids communications delays and the expense of constantly keeping the communication channel open. The fact that the diagrams can also be stored in the unit means that annotated diagrams usually need not be uploaded immediately, but may be stored until the worker returns to the home base, in which case they may be uploaded through a fast connection or over a slower telephone line in a situation where speed of response is not as important. We currently anticipate that the graphical interchange system described here will be completed in the Fall of 1997. authors wish to thank Ellen Do, Kyle Kuczun, and Adrian Warmaek of me University of Colorado Schoolof EnvironmentalDesign for their work on the ElectronicCocktail Napkin and the Newton-Napkin interface,and Paul Bauerof US WEST Advanced Teelmologiesfor his commentsand encouragement.This work was supportedin part by the NEC Corporation. Acknowledgements-The
RF#FERENcEs
1. Gross, M. D., The Electronic
Cocktail
Napkin-
computer support for working with diagrams.D&xn St&es, 1996,-17(l), 53-69.
-
2. Gross. M. D.. Stretch-A-Sketch:a dvnamic diaarammer. in IEEE Symlposium on Vkuai Languages- (YL ‘94), St. Louis, 1994, pp. 232-238. 3. Gross, M. D. and Do, E. Y.-L., Shape-baaed reminding in creative design. In Compater-Ai&d Arc&fectural DesignFutures (CAAD Futures ‘92). Singapore,1995, pp. l-11. 4. Greenberca, S.,Hayne,S. and Rada, R., eds., Groupware for Real-The Drawing: A Designer’s Gut&. McGrawHill, London, 1995.
5. Apte, A. and Kimura, T. D., A comparisanstudy of the pen and the mouse in editing graphic diagrams. In IEEE Symposium on VisuaJ Lmwges Bergen, Norway, 1993, pp. 352-357.
(VL
W),
PDA-based graphical interchange 6. Citrin, W. V., Requirements for graphical front ends for visual languages. In IEEE Symposium on Visual Lunguages (VL ‘fi), Bergen, Norway, 1993, pp. 142-151. 7. Citrin. W.. Santiaeo. C. and Zom. B.. Scalable interfaces to suppo;progmm comprehension. In IEEE Workshop on Program Comprehension, Berlin, 1996, pp. 123-132. 8. Rubine, D., Specifying gestures by example. Computer Graphics, 1991,X!, 329-337. 9. Zhao, R., Incremental recognition in gesture-based and syntax-directed diagram editors. In ACM SIGCHZ Conference on Human Factors in Computing (INTERCHI ‘93), Amsterdam, 1993, pp. 95-100. 10. Lakin, F., Visual grammars for visual languages. In National Conference on Art#icial InteNigence (AAAI ‘87), Seattle, 1987, pp. 683-688. 11. Helm, R., Marriott,- K. and Odersky, M., Building visual parsers. In ACM SIGCHI Conference on Human Factors in Computing Systems (CHI PI), New Orleans, 1991, pp. 105-112.
649
12. Landay, J. A. and Myers, B. A., Interactive sketching for the early stages of interface design. In ACM SIGCHI Conference on Human Factors in Computing Systems (CHI ‘95), Denver, 1995, pp. 43-50. 13. Ishii, H. and Kobayashi, M., ClearBoard: a seamless medium for shared drawing and conversation with eye contact. In ACM SIGCHI Conference on Human Factors in Computing Systems (CHI ‘92), Monterey, CA, 1992, pp. 525-532. 14. Minneman, S. L. and Bly, S. A., Managing a trois: a study of a multi-user drawing tool in distributed design work. In ACM SIGCHI Conference on Human Factors in Computing Systems (CHI PI), New Orleans, 1991, pp. 217-224. 15. Robertson, S., Wharton, C., Ashworth, C. et al.. Dual device interface design: PDAs and interactive television. In ACM SIGCHI Conference on Human Factors in Computing Conference (CHI ‘96), Vancouver, BC, 1996, pp. 79-86.