CollaboRoid: Mobile platform support for collaborative applications

Pervasive and Mobile Computing 55 (2019) 13–31

Contents lists available at ScienceDirect

Pervasive and Mobile Computing journal homepage: www.elsevier.com/locate/pmc

CollaboRoid: Mobile platform support for collaborative applications Hochul Lee a , Jaehun Lee a , Young Choon Lee b , Sooyong Kang a , a b

∗

Department of Computer Science, Hanyang University, 222 Wangsimni-ro, Seongdong-gu, Seoul 04763, Republic of Korea Department of Computing, Macquarie University, Balaclava Road, North Ryde, NSW, 2109, Australia

article

info

Article history: Received 14 May 2018 Received in revised form 31 January 2019 Accepted 17 February 2019 Available online 28 February 2019 Keywords: Mobile platform Mobile collaboration Resource sharing Mobile multimedia

a b s t r a c t Development of mobile collaborative applications requires non-trivial amounts of efforts in enabling multi-device resource sharing. The current practice of mobile collaborative application development remains largely at the application level. In this paper, we present CollaboRoid, a platform-level solution that provides a set of system services in the Android stack. It abstracts the sharing of not only hardware resources, but also software resources and multimedia contents between multiple mobile devices. We demonstrate the efficacy of CollaboRoid first with an example mobile collaborative application developed using CollaboRoid followed by extensive experiments with several collaboration scenarios on Nexus 5 and 7 devices. Our experimental results confirm that (1)the latency of remote resource access remains sufficiently low for collaboration, and (2)the additional energy consumption for collaboration is insignificant considering the benefit of energy sharing effect. © 2019 Elsevier B.V. All rights reserved.

1. Introduction Mobile devices are becoming increasingly powerful with new hardware resources and expanded computing and storage capacities. These devices are prevalent, and ever-connected via fast communication network (e.g., LTE and upcoming 5G) and widely deployed Wi-Fi networks. As a consequence, there emerged a trend to collectively and collaboratively use mobile devices, i.e., mobile collaboration. Example applications are multiplayer mobile games, collaborative document editing and multimedia contents sharing. The easy access to collaborative use of hardware and software resources of multiple mobile devices opens a new application development avenue promoting mobile collaboration. It is, however, not an average task for one mobile device to use resources of others’. The current model of mobile collaboration is largely application-driven. In particular, a mobile collaboration is enabled by an application specifically developed for that. In this paper, we propose a platform-driven collaboration model, in which mobile platform takes the entire roles of the physical collaboration among devices while application takes control of the platform in accordance with the semantic of the collaboration. To this end, we develop CollaboRoid as a platform-level solution for mobile collaboration. CollaboRoid enables multiple mobile devices to share their hardware and software resources as well as multimedia contents via its platform-level services. In particular, it provides core collaboration functionalities as system services in the Android stack1 ; ∗ Corresponding author. E-mail addresses: [email protected] (H. Lee), [email protected] (J. Lee), [email protected] (Y.C. Lee), [email protected] (S. Kang). 1 The current implementation of CollaboRoid in Android is due primarily to the open nature of Android and its representativeness as a mobile platform; however, CollaboRoid is not limited to a specific platform. Source code and demo videos are available from [1]. https://doi.org/10.1016/j.pmcj.2019.02.006 1574-1192/© 2019 Elsevier B.V. All rights reserved.

14

H. Lee, J. Lee, Y.C. Lee et al. / Pervasive and Mobile Computing 55 (2019) 13–31

Fig. 1. Platform-supported collaborative work: (a) A tennis game application is being executed (only) in a smart TV and each user is playing the game using his/her own device as a tennis racket. (b) The device with low battery energy (left) can make use of the power-hungry GPS resource of a remote device (right).

they can be simply accessed via API calls. Such functionalities are implemented in two main system services, device management service and remote access service with slight modifications of the native Android. There are several merits in the platform-driven model: (1) applications do not need to implement core collaboration functionalities, (2) dedicated server is not needed for collaboration purpose, and (3) collaboration is possible without application being installed in all participating devices. Directly supported by the mobile platform, mobile applications can easily evolve into collaborative ones, thereby creating new collaborative work scenarios and opening new avenues for mobile collaboration. Fig. 1 shows good examples of platform-driven mobile collaboration. Platform support for mobile collaboration provides a new possibility of the emergence of more active, more productive and more interesting collaborative applications with reduced development cost. If a mobile platform supports hardware resource sharing among multiple devices, a multi-user game application in a device can use sensors in remote devices so that multiple users can play the game together by using their device as a game controller (or input device), as shown in Fig. 1(a). If a mobile platform supports software sharing, a computation-intensive application in a device can assign parts of its input data to remote software (i.e., pre-installed software in other devices) for computation to get better performance. Furthermore, using both hardware and software sharing capabilities, a video content in a device can be decoded, rendered and displayed in both local and remote devices almost at the same time, that is real-time content sharing. This is why we believe that platform-level support for collaboration can open new avenues for mobile collaboration. We have conducted experiments with multiple resource sharing scenarios on Nexus 5 and 7 smartphones. Our experimental results show the feasibility of the platform-level collaboration using CollaboRoid in terms of latency and energy consumption. For example, the latency of remote hardware access is smaller than 100 ms regardless of hardware type and the number of remote devices. And the low battery device can effectively save its power by exploiting remote hardware (e.g., display or GPS), which we call the energy sharing effect in this paper. This paper is an extended version of our prior work [2]. The details of the enhancements are as follows: (1) we implement a GPU-based transrating functionality in CollaboRoid to effectively support heterogeneous devices with different display sizes, (2) we design and implement a multiqueue-based fair remote events delivery technique to cope with skewed response times across devices in CollaboRoid, (3) we present how to develop collaborative applications on CollaboRoid by showing an example application code, and (4) we show additional experimental results that support the feasibility of mobile collaboration using CollaboRoid. The rest of this paper is organized as follows. Section 2 provides related work. In Sections 3 and 4, we present CollaboRoid in terms of design and implementation, respectively. Section 5 discusses implementation issues and our resolutions. Section 6 provides methods for developing collaborative applications by showing an example application code. Section 7 presents our evaluation results in various scenarios. And Section 8 presents lessons learned in this work, followed by the conclusion in Section 9. 2. Related work Harnessing the great amount of devices pervasive in our environment for collaboration purposes has been studied in several ways, such as sharing I/O resources of remote devices, remote device control, offloading tasks to remote devices or clouds, and sharing multimedia contents among devices. Sharing I/O resources in a remote device has been elaborated for a long time. Examples include IP Webcam [3], Wi-Fi Speaker [4] and Screen Sharing [5], which are application-level solutions for remote I/O sharing. Their capacity is often limited to sharing/accessing a particular I/O resource in remote device since applications used are specifically designed for such sharing purpose. Recently, a more generic approach for remote I/O sharing for mobile systems, Rio, has been proposed [6]. It adopts kernel-level approach. For a pair of local and remote devices, Rio maps the device file for an I/O resource in the remote device to a virtual device file for a corresponding I/O resource in the local device. While Rio’s kernel-level approach is

H. Lee, J. Lee, Y.C. Lee et al. / Pervasive and Mobile Computing 55 (2019) 13–31

15

characterized by its generic design for remote ‘‘I/O’’ sharing, CollaboRoid’s platform-level approach is characterized by its general usability for mobile collaboration. More specifically, CollaboRoid differentiates its design from Rio’s in the following four ways: (1) the source device of I/O’s is easily identifiable, (2) applications can control whether or not to exploit remote resources, (3) various types of resources not only I/O resources, but also, software resources and even multimedia contents can be shared, and (4) more broader hardware heterogeneity support, such as coordinate translation for devices with different display resolutions, is provided. These are also main reasons why we adopt a platform-level approach. Similarly to I/O resource sharing, controlling remote devices has been explored in several different ways, RemoteUI [7] and AirPlay [8]. RemoteUI controls remote devices using a web server and web sockets. It allows users to control buttons in a remote device and to share media files. AirPlay’s intuitive design, of split user interface for shared panel and control panel, much improves user experience. These works primarily aim to share media files; and thus, hardware resources they can share are limited to, for example, speakers and display. Offloading computation tasks in smart devices to servers in clouds or nearby mobile devices is another well studied approach as some form of collaboration. Works with the use of cloud servers for such offloading include MAUI [9], CloneCloud [10], f_Mantis [11], Cuckoo [12], ThinkAir [13] and Cloudlet [14]. In [15], they used a dedicated proxy system, instead of cloud servers, to offload computationally heavy JavaScript code for mobile client. With recent advances in mobile devices in terms of capabilities and capacities, these devices are directly used for computation offloading, e.g., Hyrax [16], Serendipity [17], HyMobi [18], LWMR [19] and others [20]. These types of mobile computing approaches essentially fall into cloud-dependent offloading or mobile distributed processing, rather than collaboration. Recently, a mobile collaboration framework, SAMD, has been proposed [21]. SAMD enables parts of a mobile application code to be offloaded to remote mobile devices for execution. To that end, it decomposes application into a set of controllers, selectively transfers controllers to other devices and then adapts received controllers for execution. SAMD does not provide any functionality for resource sharing among devices. An application cannot either access remote hardware nor execute software code installed in other device, without implementing those functionalities in itself and offloading their code to the remote devices. However, CollaboRoid enables an application to directly use remote hardware/software resources in almost the same manner as local ones. There have been several efforts on multimedia contents sharing among mobile devices, and their approaches can be categorized into: (1) central or cloud server based contents sharing [22–25], and (2) device-to-device contents sharing based on opportunistic P2P network [26–29]. While they use different approaches and technical schemes, they are common in that they are not directly supported by the mobile platform for their objectives. 3. CollaboRoid CollaboRoid consists of two main system services: Device Management Service (DMS) and Remote Access Service (RAS) (Fig. 2). They provide core collaboration functionalities in conjunction with two auxiliary components of CollaboRoid’s, Data Injection Library (DIL) and Proxy Application (PA). Devices in mobile collaboration are classified into the initiator and the participant(s). As we describe CollaboRoid from the initiator’s perspective, we refer the initiator as the local device and the participant as the remote device. 3.1. Device Management Service (DMS) As the collaboration environment we concern is with mobile devices that can be connected without a server, DMS provides the information on mobile devices available for collaboration. In particular, DMS collects and maintains several pieces of information, including collaboration activation state and available resources, from mobile devices. DMS consists of three sub-components: Broadcaster, Information Sender and Device Directory. Broadcaster sends a broadcast signal to devices connected to the same Wi-Fi access point (AP) with the local device requesting necessary information for collaboration. The broadcast signal is then handled by Information Sender of the remote device to respond to the local device with what is asked, such as availability for collaboration and the states of resources. Device Directory maintains the list of remote devices in the current collaboration. DMS is capable of dealing with devices connected to the same Wi-Fi AP as well as those in the contact list. The latter set of devices may not share the same AP, but are connectable via mobile communication networks. This functionality of DMS allows the Settings application in a device to configure a number of settings, such as the set of remote devices to be participated in the collaboration and the type of collaboration. For instance, the collaboration initiator can configure the collaboration with two devices detected, one being shared its sensor resource and the other allowed for API calls. 3.2. Remote Access Service (RAS) RAS is responsible for the seamless resource sharing between mobile devices participating in collaboration. The key functionality is device-to-device communication. RAS consists of four sub-components: Listener, Sender, Receiver and Handler. Listener is the entity for establishing the communication channel between devices, either for control (from local to remote devices) or for data transmission (from remote to local devices). Once Listener gets a request to establish a session, an instance of Receiver is created and it is associated with the communication channel (socket). Sender and

16

H. Lee, J. Lee, Y.C. Lee et al. / Pervasive and Mobile Computing 55 (2019) 13–31

Fig. 2. CollaboRoid system architecture.

Receiver are paired up for control and data communication. When the local device issues a request to access resources of a remote device, Sender in the local device forwards that request to Receiver in the remote device. The result of that request (e.g., sensor data) is then returned back by the remote device’s Sender to the Receiver instance specifically created, for such communication, by the local device’s Listener. This returning data is a result of refining, by Data Refinery, the original result of that request with the removal of device-dependent data. The R-to-D (resource-to-devices) map in the Sender contains resource sharing information that describe which resource is being shared with which devices. These pieces of information are used by the Sender either when sending remote resource access requests to remote devices or when sending the results of remote requests. Handler processes the request/data Receiver gets according to the type of resource. In particular, Handler in the remote device deals with the request received from the local device. Interpreter identifies an appropriate system service to call, and Executor actually makes the call. In the meantime, Handler in the local device forwards data, as the result of a remote request, to an appropriate service in the local device based on the type of resource the data is associated with. The service then delivers data to the application which initiated the request. 3.3. Data Injection Library (DIL) DIL enables the transparent process of events received from remote devices by feeding them to Android’s native library to be treated in exactly the same way as local events. In Android, system services use native library to obtain local hardware-generated events from the event queue. Data Injection Library enables uniform data acquisition by ‘‘injecting’’ remote events directly to the event queue. System services do not need to differentiate how to obtain local and remote hardware events. Mobile platforms maintain event queues for each type of event in which hardware events are temporarily stored before being delivered to applications. As stated above, CollaboRoid injects received remote events to the corresponding event queue in the local device (i.e., remote touch events are enqueued to the touch event queue in the local device, for example). Hence, when collaborating users generate a burst of events simultaneously, the aggregated number of events can instantly enlarge the queue length, and then response times of late events become large. Its representative case is when streams of events are generated by users through their ‘touch & drag’ or ‘rotate device’ actions. What really matters is the skewed response times across devices. Since event streams rapidly enlarge the queue length, a slightly late user action makes all of her action events enqueued far behind the front of the queue, and so processed far later than the time the action has been performed. Then the user experiences unfair responsiveness to her action. This skewed response times can hinder mobile collaboration. For example, consider a multi-user car racing game where each user’s car is controlled by device rotation.

H. Lee, J. Lee, Y.C. Lee et al. / Pervasive and Mobile Computing 55 (2019) 13–31

17

Fig. 3. Multiqueue-based fair event delivery in CollaboRoid.

To mitigate this problem, CollaboRoid implements a multiqueue-based fair event delivery technique (Fig. 3) for streamed events. It creates individual event queue (i.e., per-device event queue) for each remote device when the number of collaborating devices is determined by the DMS, and enqueues remote events to their respective per-device event queue. Then the event dispatcher delivers events to the application in a Round-Robin manner across per-device queues. CollaboRoid’s fair event delivery technique mitigates aforementioned skewed response times. Since events are delivered to an application in a round-robin manner across per-device queues, slightly late user action is fairly reflected to the application from the time when its first event arrives to the local device. One may have concerns about the forceful reordering of event delivery because the order of events itself can be an important factor for certain applications. In an ordinary networked system, however, the non-uniform network latency inevitably changes the arrival order of events generated from different devices, so arrival times already does not exactly preserve their order of generation. Furthermore, sporadic events such as touch (without drag) events do not seriously enlarge the queue length and so their response times are sufficiently short not to make users experience unfairness due to the out-of-order event delivery. 3.4. Proxy Application (PA) PA is a user-transparent system application designed to run for mobile collaboration. It performs a specific role depending on the type of mobile collaboration. The three roles in the current design of CollaboRoid are Event Absorber, Video Player and Image Viewer, each of which is implemented as an activity in the PA. Event Absorber enables a remote device user to control his/her device even when the device is being used as a remote input device for an application in a local device. Video Player enables video streaming by decoding sample data chunks of a video file. Image Viewer displays the received image on the screen. 4. Remote resource access In this section, we detail RAS implementation of remote resource access of three types: hardware, software and multimedia contents. 4.1. Hardware resource access Applications issue a service call to access hardware resource. CollaboRoid provides applications a set of remote hardware types, such as TYPE_REMOTE_ORIENTATION for remote orientation sensor and REMOTE_GPS_PROVIDER for remote GPS. If a service call is for remote device, i.e., when a remote hardware type is used, the corresponding service sends a remote request to RAS, which eventually will be forwarded to Receivers in collaborating devices listed in the R-to-D map. Otherwise, it accesses local hardware. In this way, CollaboRoid provides uniform access interface for both local and remote resources. Upon receiving the remote request, Receiver in a remote device forwards the request to Handler. Interpreter and Executor in Handler identifies the type of service and makes an appropriate service call for the request, respectively. The service called for the request gets and sends the event from the hardware resource to RAS (Steps 5–2 and 6 in Fig. 4). Sender in the remote device sends a message to Listener in the local device; this message is for the creation of a Receiver instance to establish a session that is used for returning the event. Once Receiver is ready for communication,

18

H. Lee, J. Lee, Y.C. Lee et al. / Pervasive and Mobile Computing 55 (2019) 13–31

Fig. 4. Procedure of remote hardware/software resource access. Note that after the creation of Receiver in the local device (Step 8), only data flow steps (Steps 5–2, 6, 9, 10, 11 and 12) are repeated upon subsequent hardware events until remote device user disables the remote sharing capability.

Sender in the remote device sends the event data to the Receiver. At this time, instead of sending the event itself, Sender extracts only key attributes from the event using Data Refinery and sends them to the local device, not only to minimize the communication overhead but also to remove device-dependent data in the event. Data Refinery also transforms the extracted data, if necessary, to a normalized form so that the data receiving device can adjust them to its hardware specification. For example, the key attribute of touch events is the set of x and y coordinates. In case of two collaborating devices with different display resolutions (a form of resource heterogeneity), coordinates conversion must be done to resolve any discrepancy. Data Refinery converts absolute coordinates into relative ones before sending them. Handler in the local device rebuilds the remote event using the received data and local device-dependent data. Interpreter of Handler, if necessary, adjusts the received data to overcome the hardware heterogeneity, e.g., converting relative coordinates to absolute ones. During rebuilding the event, Handler embeds the source device ID to the event so that applications can identify the source device of each event. For this purpose, when a remote device connects to the local device, CollaboRoid in the local device assigns a new ID to the remote device. It is worth note that by assigning one Receiver instance to each remote device, applications in the local device can use hardware resources in multiple remote devices, simultaneously. In this way, CollaboRoid permits more than one remote devices to participate in collaboration, which enables developing multi-user applications exploiting multiple devices. Handler then passes the rebuilt remote event to the local service using Data Injection Library. 4.2. Software resource access Access to software resources we concern in this work is assumed to be made possible via API calls. Therefore, applications should be able to identify the destination device API call is for. CollaboRoid adopts function overloading to make a remote API call. In particular, we add a flag to the original API that can be set to either local call, remote call or both. The process of remote API call is similar to that of accessing hardware resources shown in Fig. 4. We envision that this function overloading for remote API calls would be meaningful for distributed computing with computationally intensive tasks. In particular, custom-designed compute-intensive library functions can be installed to multiple devices for distributed computing. This use case scenario illustrates the possible use of CollaboRoid as a base layer of distributed computing across mobile devices. CollaboRoid provides several APIs, such as minimax value calculation, to enable application programmers to use distributed processing capabilities. When the distributed processing API is called, Sender in the local device distributes input data to remote devices along with the name of API, and Handler in the remote device calls a requested API with the received data and returns the result to the local device. After receiving results from all remote devices, Handler in the local device collects the results and returns them to the application. When using CollaboRoid for distributed processing, the participating devices have very different performance characteristics from one another. Thus, it is necessary to address such performance heterogeneity to make CollaboRoid’s distributed processing functionality feasible. To that end, CollaboRoid equips with a simple relative speed-based load balancing algorithm. 4.3. Multimedia contents sharing Multimedia contents sharing is probably one of most common use cases of mobile collaboration. However, the current mobile platforms do not provide content sharing functions by default. The common practice for content sharing largely remains the use of simple file copy, which may have copyright violation issues. To avoid ‘file copy’ when sharing multimedia content with users dispersed across various physical locations, we adopt streaming. Fig. 5(b) illustrates the process of video and audio sharing in CollaboRoid. For video sharing, we implemented streaming functionality to the

H. Lee, J. Lee, Y.C. Lee et al. / Pervasive and Mobile Computing 55 (2019) 13–31

19

Fig. 5. Multimedia contents sharing process in CollaboRoid.

media framework in Android. Media framework in local device streams video data in a sample data chunk (i.e., frame) unit to the Proxy Application (PA) in the remote device. PA with the Video Player activity is launched by the Handler upon receiving a remote request (with VIDEO tag) from the local device. PA creates a decoder and plays the streamed data using the MediaCodec in the media framework. Mobile devices sharing video contents may have different display resolutions each other. Streaming a high resolution video to devices with lower display resolution not only wastes network bandwidth but also increases the possibility of playback hiccup when a large number of devices are sharing the video. To resolve this problem, CollaboRoid implements a real time resolution transcoding (or transrating) that changes the video resolution in accordance with the target device display specification. Real time transrating is most effective when network is a bottleneck in contents sharing, possibly in WAN environment, in which case decreasing the amount of data traffic through transrating becomes pivotal. However, since real time transrating is a computation-intensive job, it inevitably accompanies increased playback latency. Also, if multiple lowend devices with different display resolutions each other participate in the video sharing, simultaneous transrating for each device is required. While we used OpenGL library that exploits GPU power to minimize the transrating overhead, the limited GPU throughput still necessitates a proper control of the number of simultaneous transrating for the stable video playback in the remote devices. To that end, CollaboRoid provides a user interface to set the maximum number of simultaneous transrating, and if the number of devices with different display resolution is larger than the maximum number then CollaboRoid selects target devices for transrating according to the number of devices that have the same display resolution. To each device that was not selected, CollaboRoid serves a video stream with the most similar resolution larger than its display resolution. CollaboRoid also provides a user interface to manually set the target transrating resolution for each remote device. This interface is specifically provided to transrate a video into a lower resolution than the remote device’s display resolution to decrease both the network traffic and latency when the quality of video does not matter. The media framework in the local device directly streams video data to PA in the remote device through socket connection, bypassing RAS. This is because video sharing with long and heavy data transfer imposes significant overhead on RAS. Our design of the direct connection much eliminates such overhead. Another advantage is the avoidance of unnecessary data transfer. In particular, when the user of a remote device terminates PA to stop watching video in the middle of streaming/playing, the network socket connected with the local device is closed. As soon as the media framework in the local device detects the socket disconnection, it stops streaming. Audio sharing is handled differently from video sharing in that it does not require the control of the display. The major difference of audio sharing from video sharing is the use of media framework directly by Handler, not by PA. Media framework in the local device streams audio data to the Handler in the remote device via Sender (local) and Receiver (remote). Handler creates decoder in the media framework and plays the streamed audio data. CollaboRoid also provides image sharing in a way similar to how video content is shared. When the user selects an image, Sender in the local device sends a remote request containing IMAGE tag and the selected image file. Then the Handler in the remote device launches PA with the Image Viewer activity and then PA displays the received image on the screen. 5. Implementation issues and resolutions In this section, we give a few important issues we encountered and describe how they are resolved in our implementation.

20

H. Lee, J. Lee, Y.C. Lee et al. / Pervasive and Mobile Computing 55 (2019) 13–31

5.1. Identifying touch & drag actions from multiple devices Collaboration-oblivious design of Android imposes additional effort on CollaboRoid to identify each user’s touch & drag action. A single touch & drag action generates an event sequence, ACTION_DOWN → ACTION_MOVE, . . ., ACTION_MOVE → ACTION_UP. If another ACTION_DOWN event arrives before ACTION_UP event arrived, Android considers the ACTION_DOWN and its subsequent ACTION_MOVE events as either illegal events that should be ignored (if their point#, or finger number, is the same as that of the ongoing action events) or multi-touch events that should be delivered to applications (otherwise). In CollaboRoid, during a collaboration, multiple users may perform touch & drag actions simultaneously, and their corresponding sequence of events are transferred to the local device. At that time, since all of their events have the same point#, their events except the first arrived action events are ignored as illegal ones until any ACTION_UP event arrives. In Section 8.2, we suggest a solution not only to this problem but also to other hurdles encountered while designing CollaboRoid. In the current version of CollaboRoid, as a temporary measure, we avoid this problem by adding 10 × deviceID to the point# in each event so that Android treat ACTION_DOWN and its subsequent events as multi-touch events. Applications can still differentiate the two types of touch (i.e., single- or multi-touch) by referring both the point# and device ID. 5.2. Device control vs. remote input Even when the collaboration is enabled, the remote device owner should be able to control his/her device by ‘touching’ display or pressing keys in the device. Those actions by the owner should be interpreted as local commands rather than remote inputs to be transferred to the other device. However, when a device is being used for remote motion (i.e., touch) input, it is extremely difficult to identify whether a motion event is for control of the device or for remote input. To resolve this problem, CollaboRoid essentially uses bidirectional event forwarding meaning that every hardware event in the remote device is forwarded to both the local and remote devices, so that a remote device user can control his/her device even when the device is being used as a remote input device. However, in that case, a motion (touch) event in a remote device intended to be forwarded to the local device as a remote input can unintentionally control the remote device itself. Proxy application (PA) solves this problem. When an application in the local device wants to exploit a remote device as an input device, it can first request to the remote device to launch a PA with Event Absorber activity which merely displays a black screen and ignores any motion event forwarded to itself. Then any motion event generated from the remote device does not control the device but is forwarded, as a remote input, to the application in the local device. Remote device user can terminate the PA at any time by pressing (or touching) ‘back button’, and then can control the device again. 5.3. Supporting device heterogeneity CollaboRoid currently works only among CollaboRoid-enabled Android devices. However, it supports hardware specification heterogeneity, in terms of display resolution and sensor availability, and Android version heterogeneity. Supporting hardware specification heterogeneity: Mobile devices come with varying types and models in terms of hardware specifications. As a result, displays and sensors equipped with mobile devices can be different, each other. CollaboRoid supports the display resolution heterogeneity by using coordinates conversion between absolute and relative coordinates, for touch events. Also, ColloboRoid enables applications to use even those sensors which are not equipped in the local device by exploiting those in the remote devices. To that end, it provides additional sensor types for remote sensors, such as TYPE_REMOTE_ORIENTATION and TYPE_REMOTE_PROXIMITY, in the Hardware Abstraction Layer (HAL). Then, applications can use remote sensor events, generated by remote devices and injected by Data Injection Library, in the same manner as local events. Supporting Android version heterogeneity: All components of CollaboRoid, shown in Fig. 2, are add-on modules and we slightly modified the native Android so that its system services invoke appropriate CollaboRoid’s services or library functions, if CollaboRoid is enabled by user. For example, the remote event handling, which is the basis of all CollaboRoid’s functionalities, is designed to inject received remote events to the native event queue of Android. To implement it, we slightly modified an Android native service (i.e., input dispatcher) so that it calls CollaboRoid’s Data Injection Library function that takes the role. At present, the only necessary condition for supporting version heterogeneity is the existence of unused variable (or bits) in event class of the native Android. Whenever an event is received from a remote device, CollaboRoid sends the event to the application with the remote device ID. Handler of the RAS takes the role of combining device ID with the data received from the remote device to make an event that will be eventually forwarded to the application in charge. To add device ID to an event object, we need an additional member variable in the event class, which may involve numerous modifications to large portions of the Android implementation due to the widespread use of event objects in many Android components. We therefore make use of unused variable (or bits) in event class to embed device ID. To extract embedded device ID from the existing variable, we added a new member method, getSourceDevice(), in the event class. This is forward-compatible as long as the variable (or bits) remain unused in future versions of Android.

H. Lee, J. Lee, Y.C. Lee et al. / Pervasive and Mobile Computing 55 (2019) 13–31

21

Table 1 List of APIs provided by CollaboRoid. Name

Implemented in

Function

getSourceDevice() getDeviceInfo() launchProxyApp() killProxyApp()

Event classes DMS RAS RAS

Return device ID in an event object Deliver device information Launch PA in the remote device Kill PA in the remote device

Fig. 6. Screenshot of the tug-of-war: The game is executed in a third-party device and each participant uses his/her own device to pull the rope.

6. Application development In this section, we show the usability of CollaboRoid in terms of collaborative application development. We first show an example application developed using CollaboRoid and then compare it with its hand-coded version without using CollaboRoid. Finally, we present the results of user studies in terms of both application development and usage. Table 1 lists CollaboRoid’s core APIs used in our example application development. 6.1. Example application: tug-of-war Application developers can use getSourceDevice() method to obtain device ID from each event object. Using the device ID, applications can distinguish events coming from multiple remote devices. getDeviceInfo(deviceID) method delivers device information such as user name, sensor availability, CPU load and memory utilization, to applications. launchProxyApp(deviceID, ROLE) and killProxyApp(deviceID) methods sends a request to the remote device to launch or kill a proxy application. We implemented launchProxyApp() method such that we can specify the role of PA in the ROLE argument, for future extension. For example, if we set ‘EVENT_ABSORBER’ as its role, PA executes its Event Absorber activity. PA can be extended by adding activities for other roles, such as launching keyboard for remote key input in the remote device. Using only these four simple APIs, application developers can easily extend their legacy collaboration-oblivious applications to collaborative ones. To show the ease of collaborative application development on CollaboRoid, we have implemented a simple multi-user game, tug-of-war (Fig. 6). In this game, two groups of two are formed as teams. To pull the rope, user repeatedly touches the screen of his/her own device. The motion event from user devices are sent to the third-party device running the game application simultaneously and then delivered to the application by CollaboRoid. Program 1 shows a part of program code for the tug-of-war. In line 5, a local device launches a PA with an Event Absorber role in each remote device. And it terminates remote PAs when the game finishes (line 11). While the constructor of the GameView initializes the UI, it obtains user names using getDeviceInfo() (line 19) and shows them on the screen. In the OnTouchEvent() method where touch events are handled, upon receiving an event, the remote device is identified and then the rope is moved to the event owner’s direction (lines 24 and 25). Program 1: Example Code (Tug-of-War) 1 2 3 4 5 6 7 8 9 10 11 12 13

public class BeginActivity extends Activity { public void onCreate(Bundle savedInstanceState) { ... startButton.setOnClickListener(...) { for( int i=1; i<=player_number; i++) mRemoteService.launchProxyApp(i, EVENT_ABSORBER); startActivity (GameActivity.class); } } public void onStop() { ... for( int i=1; i<=player_number; i++) mRemoteService.killProxyApp(i); } ...

22 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29

H. Lee, J. Lee, Y.C. Lee et al. / Pervasive and Mobile Computing 55 (2019) 13–31 } public class GameView extends View { public void GameView(Context context, ...) { ... // initialize graphic for( int i=1; i<=player_number; i++) setPlayerName(i,DMS.getDeviceInfo(i).getName()); ... } public boolean OnTouchEvent(MotionEvent ev) { if (ev.getAction()== MotionEvent.ACTION_DOWN) { if (ev.getSourceDevice() == 1 || ev.getSourceDevice() == 3) pullTheRopeToLeft(); // team 1 else if (ev.getSourceDevice() == 2 || ev.getSourceDevice() == 4) pullTheRopeToRight(); // team 2 } } ... }

We first implemented a single-device version of tug-of-war for vanilla (i.e., unmodified) Android, in which touching the left half (or the right half) side of the display represents input for team 1 (or team 2). We then turn this single-device game into collaborative one by adding three lines (lines 5, 11 and 19) and by modifying two lines (lines 24 and 25). This shows how CollaboRoid facilitates developing collaborative applications. 6.2. Comparison with hand-coded tug-of-war To investigate the usability of CollaboRoid quantitatively, we implemented hand-coded version of multi-user, multidevice tug-of-war on Android, and compare it with its CollaboRoid version. Four users individually install and launch the hand-coded tug-of-war in their own devices and then communicate each other. A touch event generated by a user is delivered to the local application for processing and then local application transfers the touch information (i.e., coordinates) to the same applications running in other devices. A notable difference between two versions of tug-ofwar is that, in hand-coded version, users watch their own device to see the current state of the game, while in CollaboRoid version they share the same device (i.e., local device). Table 2 compares the two versions of tug-of-war. The lines of code (excluding image files) of hand-coded version is 811 while CollaboRoid has only 368 lines. Hand-coded version includes code for device detection and communication which are not necessary in CollaboRoid version. Hence, it has far more lines of code, larger numbers of functions and classes than CollaboRoid version. In addition, the average and standard deviation of touch events latencies in hand-coded version are slightly larger than those in CollaboRoid. It is due to the larger number of communicated events in hand-coded version. Note that while CollaboRoid version sends events only to the local device, the hand-coded version sends them to all other devices. Each device in hand-coded version executes its own tug-of-war application, while each device in CollaboRoid version executes the Proxy application with the event absorber activity. Since the proxy application with event absorber activity merely displays a black screen, it consumes far less energy than the tug-of-war application. Hence, the total energy consumption in hand-coded version is larger than that in CollaboRoid version in spite that the CollaboRoid version has one more device, i.e., local device. 6.3. User study We investigated the practical usability of CollaboRoid in terms of two metrics – application development and usage – and conducted user studies using NASA Task Load Index (NASA-TLX) [30] and verified the results using Mann–Whitney U test [31]. Total 24 people participated in the user experience (UX) assessment. We divided participants into two groups (12 people in each group), G1 and G2 , each of which plays Hand-coded and CollaboRoid versions of tug-of-war, respectively, in the LAN environment. For hand-coded version assessment (G1 ), we incorporated the app installation process in their experience, which is not included in the CollaboRoid version assessment (G2 ). We examined six different dimensions of the NASA-TLX between the two groups. Fig. 7(a) shows the results. Among the six dimensions, Mental Demand and Effort are the dimensions that show the significant difference (MD: Mann–Whitney U = 19, n1 = n2 = 12, p < 0.05, two-tailed, E: Mann–Whitney U = 28, n1 = n2 = 12, p < 0.05, two-tailed). Their scores in G2 (CollaboRoid version) were lower than those in G1 (Hand-coded version) owing mainly to the unnecessity of app installation at which most participants expressed their satisfaction. There was no statistically meaningful difference in other dimensions, as we anticipated. Again, we assessed the usability of CollaboRoid in terms of app development. Two groups of people2 are requested to develop the hand-coded and CollaboRoid versions of tug-of-war, respectively, providing all image files and APIs used in the app. Fig. 7(b) shows the results. The overall scores of two groups differ significantly (Mann–Whitney U = 22, n1 = n2 = 12, p < 0.05, two-tailed), which confirms that CollaboRoid meaningfully helps developing collaborative applications. In detail, among the six dimensions, Mental Demand (Mann–Whitney U = 23, n1 = n2 = 12, p < 0.05, two-tailed) and Effort (Mann–Whitney U = 5, n1 = n2 = 12, p < 0.05, two-tailed) are the dimensions that show the significant difference and represent the difficulty in implementing the tug-of-war game. From the results of our user studies, we can draw a conclusion that CollaboRoid can be effectively used in developing collaborative applications and there is obvious convenience in enjoying applications based on CollaboRoid. 2 All of the 24 participants have more than six months of Android app development careers.

H. Lee, J. Lee, Y.C. Lee et al. / Pervasive and Mobile Computing 55 (2019) 13–31

23

Table 2 Comparison between two versions of tug-of-war: Latency in CollaboRoid version is between remote and local devices, while in hand-coded version it is between each pair of devices. Latencies of 50 touches per device are averaged. Lines of code # of functions # of classes Average latency Total energy consumption

CollaboRoid

Hand-Coded

368 9 3 40 (±11) ms 288 J

811 14 7 42 (±15) ms 343 J

Fig. 7. Subjective mental workload scores from the NASA-TLX (MD: Mental Demand, PD: Physical Demand, TD: Temporal Demand, P: Performance, E: Effort, F: Frustration). Lower is better.

7. Performance evaluation To verify the feasibility of the platform-level collaboration using CollaboRoid, we have measured both the latency and energy consumption for various types of mobile collaboration. Mobile collaborations can be categorized into two forms: interactive collaborations such as multiplayer games and non-interactive collaborations represented by multimedia contents sharing. The interactive collaboration model we assume is that collaborative applications are installed and launched only in one device (i.e., local device) and other (remote) devices are used as simple input devices. Since remote devices neither execute nor install the application, there is no case where remote devices communicate with each other. Also, in multimedia contents sharing, only one device (local) has multimedia contents to be shared with multiple remote devices. Hence, we use one local device and multiple remote devices throughout the evaluation. We used one Nexus 7 and eight Nexus 5 devices for experiments. 7.1. Latency We define the latency of remote resource access as the elapsed time from the occurrence of an event in the remote device to its arrival at an application in the local device. In the case of multimedia content sharing, the latency is defined as the elapsed time from when the local device user issues the content playback command (by touching the ‘Play’ button) to the time playback starts in the remote device. For accurate clock synchronization in all devices, we used Network Time Protocol (NTP) [32] with millisecond-level synchronization error correction. We measured latencies in both LAN and WAN environments. In the LAN environment, all devices are connected to the same Wi-Fi AP that supports 802.11n (maximum 300 Mbps speed). We implemented two simple test applications for latency measurement: one for vibrator services and another for other services. Both applications are executed in the local device. The vibrator application issues remote vibration requests to the vibrator service. It logs timestamp the very moment after calling vibrator service. We modified the vibrator service in the remote device so that it logs timestamp the very after issuing a command to the vibrator hardware. The difference between timestamps in both local and remote devices is the latency of the remote vibrator access, which corresponds to the time for Steps 2 ∼ 5–1 in Fig. 3. The other application is for other hardware peripherals (i.e., touch, key, sensor and GPS) from which received data is delivered to the application in the local device. It logs timestamp whenever receiving an event from the remote device. In the remote device, each system service for hardware access is modified to log timestamp whenever it receives an event from the hardware. Hence, for each event, two timestamps (local and remote) are generated, whose difference is the latency of the event transfer from remote system service to local application and corresponds to the time for Steps 6 and 9 ∼ 12 in Fig. 3. Table 3 shows actions to generate remote hardware events and remote vibration requests. Actions for all hardware sensors except for vibrator were performed in the remote device.

24

H. Lee, J. Lee, Y.C. Lee et al. / Pervasive and Mobile Computing 55 (2019) 13–31 Table 3 Actions for generating events and requests. Hardware

Actions

Repetition

Touch Key Orientation Proximity Location Vibrator

Touch display 100 times Random keyboard input Repeatedly change orientation for 5 s Move a finger closer to the sensor and further away Generate random coordinates Issue random vibration

10 times 100 times N/A 30 times 100 times 100 times

Fig. 8. Average remote hardware access latency with standard deviation in LAN environment.

Fig. 9. Remote hardware access latency when accessing four different remote resources, simultaneously. The horizontal line on each bar represents the average latency when accessing the same hardware resource from all remote devices in 1 : 4 or 1 : 8 case (shown in Fig. 8).

7.1.1. Latency in LAN environment For the latency measurement in LAN environment, we used Nexus 7 as a local device and up to eight Nexus 5 devices as remote devices. The first collaboration scenario we assume is that a collaborative application running in a local device processes a single type of user input from remote devices, like in ‘tug-of-war’. Fig. 8 shows the average latency of each remote hardware access. In the figure, ‘1 : N’ denotes the case where a local device accesses N remote devices, simultaneously. In all cases, the average latency is smaller than 100 ms (in particular, smaller than 50 ms in latency-sensitive touch hardware) and it does not increase noticeably as the number of remote devices increases, which means that the latency is not a serious obstacle for CollaboRoid applications exploiting (at least up to eight) remote devices, provided that all devices are connected to the same AP. In a certain collaboration scenario, participating users may take different roles each other. Then the collaborative application running in a local device accesses different remote hardware resources depending on each user’s role. For that scenario, we conducted an experiment that measures the remote hardware access latency when four different hardware resources (touch, key, sensor and GPS) are accessed from four or eight remote devices, concurrently. Fig. 9 shows the results. In case of 1 : 8, four pairs of remote devices are providing different resources, each other. For this experiment, we added ‘drag’ action to Touch action so as to expose the effect of ‘Touch & Drag’ action to the latency of Key event. Overall, except for Key, the latency when concurrently accessing different hardware resources is similar to that of single resource access. Since system services for each hardware access exploit their own channels to deliver hardware events to applications, latency does not change greatly even when receiving different hardware events from remote devices at the same time. However, in Android, Key and Touch events are managed by the same system service (specifically, input service), and so delivered via the same channel. Also, Key events are significantly less than Touch & Drag events in their frequency. As a result, each Key event is delivered after delivering a bunch of preceding Touch (exactly, ACTION_MOVE) events, which increases average latency. Nevertheless, considering Key event is less delay-sensitive than other hardware events, we believe that its increased latency still remains in the reasonable boundary.

H. Lee, J. Lee, Y.C. Lee et al. / Pervasive and Mobile Computing 55 (2019) 13–31

25

Fig. 10. The effect of the fair event delivery technique: average response times of remote devices over 10 repetitive experiments are averaged.

Fig. 11. Average latency of 10 experiments with standard deviation in multimedia content sharing.

Next, we evaluate the effect of the fair event delivery technique which is designed to mitigate the skewed response times across remote devices when multiple streams of events are received from remote devices in a short time. In this experiment, we performed ‘Touch & Drag’ actions in up to eight devices almost simultaneously and measured the response time of the first event (i.e., ACTION_DOWN) from each remote device by measuring the difference between the time the event is generated in a remote device and the time the processing is done in the local device. For this experiment, we implemented a simple application that records a timestamp every time it processes an event arrived. Since the effect of every touch event is shown in the display, applications process touch events at most as fast as the display refresh rate: in general 60 Hz. Hence, our test application is implemented based on the assumption that it processes an event every 1/60 s. Fig. 10 clearly shows the effect of our fair event delivery technique. Without this technique, the average response time and its standard deviation rapidly grow as the number of remote devices increases. In particular, the standard deviation revealing the unfairness of the response times among devices is very high (223 ms in 1 : 8 case), which confirms the existence of the skewed response time problem. With our technique, both the average response time and its standard deviation are effectively suppressed, providing fast and fair response times to all devices. The next collaboration scenario is that a group of users enjoy a multimedia content that is stored in a single device, using her own device. In this scenario, a user device that stores the content is a local device and other user devices are remote devices. For the multimedia content playback, we used native media playback applications in Nexus devices: ‘Gallery’ and ‘Music’ for video and audio playback, respectively. Since both applications call MediaPlayer service as soon as the ‘play’ button is touched by user, we modified the ‘start()’ function in the MediaPlayer service to log timestamp at the beginning of the function, in the local device. In the remote device, we modified the MediaCodec class so that it logs timestamp as soon as decoding the first media sample data. The difference between local and remote timestamps corresponds to the time for Steps 2 ∼ 10 for video playback and Steps 2 ∼ 9 for audio playback in Figs. 5(a) and 5(b), respectively, and becomes the playback latency in the remote device. We used two media files whose sizes are 5.4 MiB (audio) and 1.2 GiB (video), respectively. The Nexus 7 device was used as a local device and up to eight Nexus 5 devices were used as remote devices for simultaneous content sharing. Fig. 11 shows the average latency in multimedia content sharing, varying the number of remote devices. ‘Local’, in the figure, denotes the local media file playback latency in Nexus 7 with the vanilla Android, which was measured in exactly the same way as the sharing case. As expected, the latencies when content is shared with remote devices are larger than the local playback latency because of the communication overheads between local and remote devices. Nevertheless, the latency increases are smaller than 110 ms and 600 ms in audio and video sharing even with eight remote devices. Based on the results, we can say that CollaboRoid provides a very feasible and convenient way for multimedia content sharing among multiple mobile devices, provided that all devices are connected to the same AP.

26

H. Lee, J. Lee, Y.C. Lee et al. / Pervasive and Mobile Computing 55 (2019) 13–31

Fig. 12. Video sharing latency in each remote device.

Fig. 13. The effect of transrating in 1 : 4 case: ‘tr’ denotes the number of transrating. When tr = 1, transrating is performed once and the transrated video is streamed to four remote devices. Hence, their transrating latencies are the same while there are slight differences in network latencies. ‘no tr.’ denotes the case where the original video is streamed to all remote devices without transrating, and latencies in four remote devices are averaged.

In real time streaming-based multimedia content sharing, it is also important to synchronize multiple content streams to each device, which enables participating users to watch or listen to the same scene/sound of the content. Then users are able to have offline talk on a particular scene while everyone is watching the same scene on his/her own device. We can determine whether or not CollaboRoid provides synchronized content streams by comparing the latencies in all devices. Fig. 12(a) shows the individual latency in each remote device in the 1 : 8 video sharing case. The maximum latency difference among eight devices is 166 ms, which means that CollaboRoid offers almost synchronized content sharing, provided that all devices are connected to the same AP. While video playback applications use buffers to absorb network delay jitter, excessive jitter may hinder QoS of the streaming video playback. To investigate the delay jitter, we measured the time difference between each I-frame is played in local and remote devices. Fig. 12(b) shows the time differences while playing a video. Although the I-frame latency varies during the playback, their standard deviation were as small as 10.4 ms, 13.4 ms, 12.4 ms and 11.1 ms, in 1 : 1, 1 : 2, 1 : 4 and 1 : 8 cases, respectively. And the gap between the maximum and minimum latencies were 35 ms, 45 ms, 43 ms and 40 ms, respectively. Based on those numbers, for 4 Mbps (with 1920 × 1080 resolution) bitrate video, only 23.6 KB of buffer space is enough to absorb the delay jitter in the LAN environment, and so we could not observe any playback hiccup during our experiments. Remote devices sharing a video content may have different display resolutions with the original video resolution. In that case, CollaboRoid transrates the original video in real time and streams the transrated video. Fig. 13 illustrates the latency and aggregate network bandwidth varying the number of simultaneous transrating in 1 : 4 case. In this experiment, we only used downscale transrating from a HD quality (1280 × 720, 1378 Kbps) original video. When only one transrating is performed (i.e., tr = 1 case), four remote devices are assumed to have the same display resolution (320 × 240, 242 Kbps). In tr = 2 case, two devices (R1 and R2) have 320 × 240 resolution and the other two devices (R3 and R4) have 640 × 360 (525 Kbps) resolution. In tr = 3 case, two devices (R1 and R2) have 320 × 240 resolution, and R3 and R4 have 640 × 360 and 854 × 480 (1155 Kbps) resolutions, respectively. In case of four transrating, each device has 320 × 240, 640 × 360, 854 × 480 and 960 × 540 (1236 Kbps) resolutions. To see the effect of transrating in detail, we divided the latency into network latency and transrating latency.

H. Lee, J. Lee, Y.C. Lee et al. / Pervasive and Mobile Computing 55 (2019) 13–31

27

Fig. 14. Average latencies in WAN environment with 1 : 4 case.

It is obvious that transrating decreases the network bandwidth consumption. As shown in the figure, the aggregate bandwidth for streaming to four remote devices noticeably small when participating devices have lower display resolutions. The downscaled resolution by transrating has positive effect to the latency. However, the overhead of transrating has negative effect to the latency. As the number of transrating increases, its latency grows due to the increased GPU load. For example, the transrating latencies for R1 in tr = 1 and tr = 4 cases greatly differ, although they are served the same resolution of videos. Hence, an excessive number of simultaneous transrating can significantly increase the overall latency, which necessitates a proper control of the simultaneous transrating as presented in Section 4.3. On the other hand, the transrating latency decreases as target video quality becomes lower, as we can clearly see from tr = 4 case. This means that if participating devices for video sharing have low display resolutions, we can benefit from transrating in terms of both latency and network bandwidth. The benefit is expected to be larger as smart devices equip more powerful processing units, CPU and GPU. Considering the above observations, we can effectively use CollaboRoid’s transrating functionality when the local device is connected to a narrowband network (ex., ADSL) or costly mobile network (ex., 3G or LTE) to reduce the total amount of outbound traffic from the local device, or when all remote devices have the same low resolution displays where a single transrating is enough. We can also use the functionality for remote devices even with high resolution displays if the quality of the video does not matter (for example, a multimedia content in which audio is the main content with subsidiary still image video) to decrease both network traffic and latency. 7.1.2. Latency in WAN environment To measure the latencies in WAN environment under the same scenarios with those in the LAN environment, we used five Nexus 5 devices where one (L) is for a local device and the others (R1, R2, R3, R4) for remote devices (i.e., 1 : 4 sharing case). Each device is connected to the subscriber line through Wi-Fi AP at home. The remote devices are about 18.5 km, 22.6 km, 10.3 km and 9.5 km away from local device, as the crow flies. L, R1, R2, R3 and R4 are connected to the 100 Mbps, 320 Mbps, 100 Mbps, 100 Mbps and 100 Mbps subscriber lines, respectively. For this experiment, we modified DMS in CollaboRoid so that devices can communicate each other through manual setting of IP address. Fig. 14 shows the latencies in WAN environment. We measured latencies in the same manner as in LAN environment. The most notable difference between results in LAN and WAN environments is that the latency differences among remote devices in WAN environment are far larger than those in LAN environment, which is natural considering their different network environment. However, interestingly, the latency itself was not as large as we anticipated. The largest hardware access latency in this experiment was 252 ms for remote Key input from R2, and the latencies for remote touch input, which is the most latency-sensitive case, from all remote devices were less than 120 ms. In video sharing case, while the average latency in WAN environment (3088 ms) increased by about 240 ms than in LAN environment (2851 ms), the largest latency difference among all devices was 560 ms (local: 2575 ms, R2: 3135 ms) which was similar to that in the LAN environment (570 ms). Based on the results, we can say that CollaboRoid can be effectively used for resource sharing among mobile devices in the WAN environment, too. 7.2. Energy consumption Remote resource access and content sharing incur additional energy consumption because devices communicate to each other via network. On the other hand, by exploiting remote resources instead of local ones, a local device with low battery can save its energy, which we call the energy sharing effect in this paper. In this section, we measure both the additional energy consumption and energy sharing effect in CollaboRoid. For the measurement, we used Trepn Power Profiler [33] developed by Qualcomm. For the fair comparison, we only used Nexus 5 devices in experiments when measuring energy consumption. We selected two collaboration scenarios, remote GPS access and multimedia sharing, for experiments, since they are expected to consume higher energy than other scenarios [34]. All experiments were conducted in the LAN environment.

28

H. Lee, J. Lee, Y.C. Lee et al. / Pervasive and Mobile Computing 55 (2019) 13–31

Fig. 15. Average per-device energy consumption when exploiting remote GPS for one minute.

Fig. 16. Energy consumption during the sharing of multimedia content.

7.2.1. Remote GPS access We first measure the energy consumption assuming the following scenario: A user is waiting for other users at an appointed place watching her device which is executing a map application that shows other users’ current location on the map. In this scenario, a device running a map applications is a local device and others are remote devices that transfer their location information to the local device. Fig. 15 shows the consumed energy in each device when location data is received from up to three remote devices simultaneously. ‘L’ and ‘R’ denote the energy consumption in local and remote devices, respectively. ‘Local’ denotes the energy consumption when local GPS is used on the vanilla Android. In the remote GPS access experiments, we turned off the GPS in the local device. CollaboRoid enables us to access remote GPS regardless whether the local GPS is turned on or off. Since each remote device performs exactly the same operations, they consume the same amount of energy regardless of the number of remote devices. The energy consumption in the local device slightly increases as the number of remote devices increases, since it receives location data from all remote devices. The difference between ‘L’ and ‘R’ in 1 : 1 case (18.73 J) presents the energy consumed by GPS hardware. And the difference between ‘Local’ and ‘R’ (2.68 J) is the energy consumed for location information transfer in a remote device. It is notable that the local energy consumption (L) in the 1 : 1 case is far smaller than the ‘Local’ case. It means that, by exploiting remote GPS instead of local one, we can effectively save the local device energy (i.e., 16.05 J/min), which demonstrates the energy sharing effect. Hence, for a device with low battery, we can effectively use the remote hardware access functionality in CollaboRoid for energy saving. 7.2.2. Multimedia content sharing We measured the energy consumption in each device when a local device shares its content with up to three remote devices, simultaneously. We used audio and video files whose sizes and playback times are 5.4 MiB/215 s and 80 MiB/139 s, respectively. We measured the total energy consumption during the playback. Fig. 16 shows the energy consumption in each device. In this experiment, both local and remote devices play the shared content. For comparison, we also measured the consumed energy when legacy file transfer-based content sharing is used. File transfer-based content sharing consists of file copy via network from local device to remote device using a file sharing application, and playback of the content in both devices. We implemented a simple network file sharing

H. Lee, J. Lee, Y.C. Lee et al. / Pervasive and Mobile Computing 55 (2019) 13–31

29

Fig. 17. Comparison of energy consumption between local and remote playback: Android and CollaboRoid denote local and remote playbacks, respectively.

application of which the only functionality is network file transfer, to remove any subsidiary energy consumption by other functionalities. Overall, the energy consumption in CollaboRoid is higher than that in the legacy file sharing, in all cases due to the data streaming. In the legacy file sharing, network file copy finishes in a short time (audio: 0.87 s, video: 12.19 s), which consumes far less energy for communication than a long lasting network streaming in CollaboRoid. It is notable that the local energy consumption (L) in CollaboRoid does not increase noticeably as the number of remote devices increases. It means that, using multimedia streaming in the Wi-Fi network, the playback time of the media content, rather than the amount of streamed data, is the dominant factor in terms of the energy consumption. Another scenario we consider is the case when a user enjoys a multimedia content stored in her device using remote device to either save energy of her device or enjoy the content in a larger display. In this scenario, local device does not play the content but streams it to a remote device for playback (i.e., 1 : 1 content sharing without local playback). To investigate the energy sharing effect by CollaboRoid, we compare the energy consumption between local and remote playbacks. For local playback, we used original Android instead of CollaboRoid, and measured its energy consumption during the playback. For remote playback, we used CollaboRoid and measured the energy consumption in local device. If the remote playback using CollaboRoid consumes less energy than the local playback using Android, we can say that the content sharing functionality of CollaboRoid can also be used for energy sharing effect. Fig. 17 shows the results. When an audio content is played in the remote device, the local device cannot save its energy because data streaming via Wi-Fi network consumes more energy than the local playback. When a video content is played in the remote device, the local device can save its energy since the display consumes large amount of energy. While remote playback consumes similar amount of energy with local playback when the display brightness is set to the minimum level (‘Dark’), it consumes far less energy than local playback with the maximum brightness level (‘Bright’). Hence, the content sharing functionality of CollaboRoid can be effectively exploited when a user wants to enjoy a video content in her smartphone using a large screen tablet as a remote playback device, while obtaining the benefit of energy sharing effect. 8. Discussion In this section, we discuss about the usability of CollaboRoid including its limitations. Also, we present lessons learned performing this work. 8.1. Usability of CollaboRoid To the best of our knowledge, CollaboRoid is the first attempt to support development of collaborative applications from the mobile platform. There have been a great number of legacy ‘multi-user’ mobile applications, for example, multi-user games. In principle, they are standalone, single device applications which are independently installed and executed by each user in her device. Collaboration between users is enabled by communications (mostly via a dedicated server) between applications that are being executed in each user’s device. CollaboRoid enables development of a true collaborative application that makes an ad-hoc collaboration group spontaneously, (implicitly) assigns an appropriate role to each participating device (user), collects each user’s actions (inputs) for collaboration and shows the aggregated results. CollaboRoid relieves users of the burden of application installation for collaboration. Application is required to be installed and executed in only one device, which greatly facilitates instant and ad-hoc collaboration. CollaboRoid provides APIs for collaborative application development. Using APIs, application developers can be free from implementing the following functionalities/programs in developing a true collaborative application: detecting, connecting to and gathering information about devices to collaborate with, collecting remote input generated by collaborating users, implementing a separate server program that orchestrates communication among collaborating devices, among others. While CollaboRoid exhibits reasonable latencies in remote hardware accesses as shown in Section 7, it has a few drawbacks that hinder us in claiming its general usability. First, there can be some applications in which the order of

30

H. Lee, J. Lee, Y.C. Lee et al. / Pervasive and Mobile Computing 55 (2019) 13–31

remote users’ actions is critical (e.g., a multi-user game in which a user who touches earlier than all others gets a credit). In those applications, CollaboRoid cannot guarantee the preservation of an accurate order of users’ actions due to the different latencies in delivering remote events to local device. Second, as shown in Section 7.2, contents sharing on CollaboRoid consumes larger energy than legacy file copy-based sharing. We recommend to use CollaboRoid for contents sharing only in those environment where file copy is practically restricted. 8.2. Lessons learned Mobile device is rapidly evolving from personal computing device to social networking device. Through the continued evolution, in a near future, we expect that mobile device will also be used as a group computing device that assists collaboration among group members such as sharing multimedia contents and enjoying multi-user, multi-device applications. After performing this work, we concluded that the core of the collaboration support is in the efficient processing of remote events. And we found that the Remote Device Abstraction can be a good design approach for mobile collaboration in future mobile platforms that not only support but also drive mobile device evolution. In mobile collaboration, from a local device’s point of view, a remote device can be thought of as ‘a peripheral input device’ that is connected through wireless network and generates various types of data spontaneously. Each type of data generated by the remote device peripheral is encapsulated in its own event object, so that applications can easily identify its type. Hence, it is possible to abstract a remote device as a kind of logical device that spontaneously generates events in the local device. In Android, remote devices can be abstracted in the hardware abstraction layer (HAL), and they should be individually registered in HAL. Remote device abstraction greatly facilitates remote events processing for mobile collaboration as follows: Simple device identification: Since an individual event queue is assigned to each abstracted remote device like other hardware devices, we do not need to associated each event with device ID, as CollaboRoid does using unused variable (or bits) in the event class. Applications can identify the remote event owner in the same manner as identifying each hardware event in local device. Also, the problem of ignoring events for simultaneous touch & drag actions, presented in Section 6.1, is naturally prevented since each sequence of events for the actions is processed individually in each per-device queue. Homogeneous event delivery: Since remote events are directly enqueued to the local event queue assigned to each abstracted remote device, both local and remote events are delivered to applications in exactly the same way. Hence, remote event injection is not necessary as CollaboRoid does using its Data Injection Library. Application-driven event delivery: Since remote events are distributed across multiple event queues depending on their source device, applications can schedule the order of event delivery from each event queue. For example, an application in which providing fair response times to collaborating users is important, it can use RR schedule. Hence, the fair event delivery technique in CollaboRoid is unnecessary. Flexible device join and leave: In CollaboRoid, neither a collaborating device can leave from the collaboration nor a new device can join an existing collaboration since application cannot control the delivery of remote events. On the other hand, since applications can dynamically determine whether or not to use a particular hardware, remote device abstraction gives more flexibility in device participation in collaboration. 9. Conclusion and future work Mobile platform support for inter-device collaboration provides a new possibility for applications to evolve into collaborative ones. In this paper, we have presented a novel mobile platform-level solution, CollaboRoid, which provides inter-device collaboration functionalities including remote hardware/software resource access and multimedia content sharing. Using only a few additional or modified programming interfaces, application developers can easily develop collaborative applications on CollaboRoid, in comparison with the case for the development with legacy mobile platforms. Our experimental results show that CollaboRoid brings the collaboration functionalities to applications with acceptable overhead in terms of both latency and power consumption. CollaboRoid also enables users to save energy in their low battery devices by exploiting remote hardware instead of local ones. While CollaboRoid provides functional usability to application developers, it has its own limitations. First, since latency is one of most critical factors that determine the feasibility of mobile collaboration, system program-level effort to minimize the remote resource access latency is desired. Second, our claim that CollaboRoid makes it easy and convenient to develop collaborative applications needs to be supported by an extensive user study with various applications. Finally, showing the general usability of CollaboRoid is required by applying CollaboRoid either to other types of devices such as wearable devices and smart TVs that are operated by Android-based software platforms, or to other mobile platforms such as Tizen. These are our scheduled future work. Acknowledgment This work was supported by the National Research Foundation of Korea (NRF), grants funded by the Korean Government (No. 2019R1A2C1011009).

H. Lee, J. Lee, Y.C. Lee et al. / Pervasive and Mobile Computing 55 (2019) 13–31

31

References [1] CollaboRoid Homepage, http://dcslab.hanyang.ac.kr/collaboroid/. [2] J. Lee, H. Lee, Y.C. Lee, H. Han, S. Kang, Platform support for mobile edge computing, in: Proceedings of the IEEE International Conference on Cloud Computing, CLOUD ’17, 2017. [3] P. Khlebovich, IP Webcam, 2015, https://play.google.com/store/apps/details?id=com.pas.webcam&hl=en. [4] W. Morrison, WiFi Speaker, 2013, https://play.google.com/store/apps/details?id=pixelface.android.audio&hl=en. [5] Wifi display (miracast), 2015, https://play.google.com/store/apps/details?id=com.wikimediacom.wifidisplayhelper&hl=en. [6] A. Amiri Sani, K. Boos, M.H. Yun, L. Zhong, Rio: A system solution for sharing I/O between mobile systems, in: Proceedings of the 12th Annual International Conference on Mobile Systems, Applications, and Services, MobiSys ’14, 2014. [7] D. Thommes, Q. Wang, A. Gerlicher, C. Grecos, RemoteUI: A high-performance remote user interface system for mobile consumer electronic devices, in: Proceedings of the 2012 IEEE International Conference on Consumer Electronics, ICCE ’15, 2012. [8] Apple, AirPlay, 2015, https://developer.apple.com/airplay. [9] E. Cuervo, A. Balasubramanian, D.-k. Cho, A. Wolman, S. Saroiu, R. Chandra, P. Bahl, MAUI: Making smartphones last longer with code offload, in: Proceedings of the 8th International Conference on Mobile Systems, Applications, and Services, MobiSys ’10, 2010. [10] B.-G. Chun, S. Ihm, P. Maniatis, M. Naik, A. Patti, CloneCloud: Elastic execution between mobile device and cloud, in: Proceedings of the European Conference on Computer Systems, 2011. [11] Y. Kwon, H. Yi, D. Kwon, S. Yang, Y. Cho, Y. Paek, Precise execution offloading for applications with dynamic behavior in mobile cloud computing, Pervasive Mob. Comput. 27 (2016). [12] R. Kemp, N. Palmer, T. Kielmann, H. Bal, Cuckoo: A computation offloading framework for smartphones, in: Proceedings of the International Conference on Mobile Computing, Applications, and Services, MobiCASE 2010, 2010. [13] S. Kosta, A. Aucinas, P. Hui, R. Mortier, X. Zhang, ThinkAir: Dynamic resource allocation and parallel execution in the cloud for mobile code offloading, in: Proceedings of the 31st IEEE International Conference on Computer Communications, INFOCOM ’12, 2012. [14] M. Satyanarayanan, P. Bahl, R. Caceres, N. Davies, The case for vm-based cloudlets in mobile computing, IEEE Pervasive Comput. 8 (4) (2009). [15] S. Park, Q. Chen, H. Han, H.Y. Yeom, Design and evaluation of mobile offloading system for web-centric devices, J. Netw. Comput. Appl. 40 (2014) 105–115. [16] E.E. Marinelli, Hyrax: Cloud Computing on Mobile Devices using MapReduce (Masters thesis), Carnegie Mellon University, 2009. [17] C. Shi, V. Lakafosis, M.H. Ammar, E.W. Zegura, Serendipity: Enabling remote computing among intermittently connected mobile devices, in: Proceedings of the Thirteenth ACM International Symposium on Mobile Ad Hoc Networking and Computing, MobiHoc ’12, 2012. [18] H. Flores, R. Sharma, D. Ferreira, V. Kostakos, J. Manner, S. Tarkoma, P. Hui, Y. Li, Social-aware hybrid mobile offloading, Pervasive Mob. Comput. 36 (2017) 25–43. [19] D. Diaz-Sanchez, A. Lopez, F. Almenares, R. Sanchez, P. Arias, Flexible Computing for personal electronic devices, in: Proceedings of the 2013 IEEE International Conference on Consumer Electronics, ICCE ’13, 2013. [20] M. Hirsch, J.M. Rodriguez, A. Zunino, C. Mateos, Battery-aware centralized schedulers for CPU-bound jobs in mobile grids, Pervasive Mob. Comput. 29 (2016) 73–94. [21] J. Lee, H. Lee, B. Seo, Y.C. Lee, H. Han, S. Kang, SAMD: Fine-grained application sharing for mobile collaboration, in: Proceedings of the IEEE International Conference on Pervasive Computing and Communications, PerCom, 2018. [22] X. Wang, M. Chen, T. Kwon, L. Yang, V. Leung, AMES-Cloud: A framework of adaptive mobile video streaming and efficient social video sharing in the clouds, IEEE Trans. Multimed. 15 (4) (2013) 811–820. [23] K. Yadav, V. Kumar, S. Jairath, R. Khurana, V. Naik, A. Singh, Poster: Cloud-enabled content search and sharing system for mobile phones (MobiShare), in: Proceedings of the 10th International Conference on Mobile Systems, Applications, and Services, MobiSys ’12, 2012. [24] X. Wang, T. Kwon, Y. Choi, H. Wang, J. Liu, Cloud-assisted adaptive video streaming and social-aware video prefetching for mobile users, IEEE Wirel. Commun. [25] C.-L. Hu, C.-A. Cho, User-provided multimedia content distribution architecture in mobile and ubiquitous communication networks, J. Netw. Comput. Appl. 34 (1). [26] E. Toledano, D. Sawada, A. Lippman, H. Holtzman, F. Casalegno, CoCam: A collaborative content sharing framework based on opportunistic P2P networking, in: Proceedings of the 2013 IEEE Consumer Communications and Networking Conference, CCNC ’13, 2013. [27] S. Ramasubramanian, S. Chung, S. Ryu, L. Ding, Secure and smart media sharing based on a novel mobile device-to-device communication framework with security and procedures, in: Proceedings of the 4th Annual ACM Conference on Research in Information Technology, 2015. [28] X. Jiang, Y. Dong, D. Xu, B. Bhargava, GnuStream: a P2P media streaming system prototype, in: Proceedings of 2003 IEEE International Conference on Multimedia & Expo, ICME ’03, 2003. [29] A. Qureshi, D. Megías, H. Rifà-Pous, Psum: peer-to-peer multimedia content distribution using collusion-resistant fingerprinting, J. Netw. Comput. Appl. 66 (2016). [30] S. Hart, NASA-task load index (NASA-TLX): 20 years later, in: Proceedings of the Human Factors and Ergonomics Society 50th Annual Meeting, 2006. [31] H.B. Mann, D.R. Whitney, On a test of whether one of two random variables is stochastically larger than the other, Ann. Math. Stat. 18 (1) (1947) 50–60. [32] D. Mills, J. Martin, J. Burbank, W. Kasch, Network Time Protocol Version 4: Protocol and Algorithms Specification, Tech. Rep., 2010. [33] Qualcomm Technologies, Trepn power profiler, 2015, https://developer.qualcomm.com/software/trepn-power-profiler. [34] N. Vallina-Rodriguez, J. Crowcroft, ErdOS: Achieving energy savings in mobile OS, in: Proceedings of the Sixth International Workshop on MobiArch, MobiArch ’11, 2011.