- Email: [email protected]
A web-based sensor network system with distributed data processing approach via web application
Computer Standards & Interfaces 33 (2011) 565–573
Contents lists available at ScienceDirect
Computer Standards & Interfaces j o u r n a l h o m e p a g e : w w w. e l s ev i e r. c o m / l o c a t e / c s i
A web-based sensor network system with distributed data processing approach via web application Tokihiro Fukatsu ⁎, Takuji Kiura, Masayuki Hirafuji National Agricultural Research Center, National Agriculture and Food Research Organization, 3-1-1 Kannondai, Tsukuba, Ibaraki, 305-8666, Japan
a r t i c l e
i n f o
Article history: Received 3 December 2010 Accepted 16 March 2011 Available online 30 March 2011 Keywords: Sensor network Image monitoring Agriculture Web application Distributed data processing
a b s t r a c t We have proposed a Web-based sensor network constructed of Web-based sensor nodes and a remote management system. The Web-based sensor nodes consist of communication units and measurement devices with Web servers. The management system has intelligent processing and rule-based function to manage them flexibly via the Internet and performs various image analyses easily with Web application services. By distributing the image analyses to Web application services, our proposed system provides versatile and scalable data processing. We demonstrated that it can realize the desired image analyses effectively and perform complicated management by changing its operations depending on the results of analysis. © 2011 Elsevier B.V. All rights reserved.
1. Introduction In modern agriculture, it is important to monitor crop growth, the field environment and farm operations to increase agricultural productivity and produce high quality products. A key technology for monitoring them in open fields is a sensor network [1,2], but it is difficult to apply it to farmers' needs without modification. Few sensor networks have camera devices because they are designed to reduce data transmission for power saving, although image data provides useful and helpful information for agricultural users to check crop conditions, estimate the influence of diseases, detect vermin and record farming operations [3,4]. On the other hand, it has recently become possible to obtain image data at field sites by using Field Servers [5] and Internet protocol (IP) cameras. The Field Server, a kind of sensor networks for agricultural use, has various kinds of sensors including image sensors, a wireless LAN to provide high-speed transmission, which enables the use of high-resolution image data, and a measurement unit with a Web server to perform easy management at a remote site via the Internet. Thanks to the development of information technology, we can also use high-performance IP cameras with a Web server at a reasonable price. Because of their ease of use in agricultural fields, these Web-based monitoring devices have come into use, and monitoring systems with these devices have become increasingly important [6,7]. However, there are some obstacles to realizing Web-based monitoring for practical purposes. There are numerous kinds of ⁎ Corresponding author. Tel.: + 81 29 838 7177; fax: + 81 29 838 8551. E-mail addresses: [email protected] (T. Fukatsu), [email protected] (T. Kiura), [email protected] (M. Hirafuji). 0920-5489/$ – see front matter © 2011 Elsevier B.V. All rights reserved. doi:10.1016/j.csi.2011.03.002
Field Servers and IP cameras [8], and each device has different controls and poor functions. Therefore, it is difficult and troublesome for nonexpert users to apply them to a monitoring system for long periods. In the case of agriculture, the desired system must be able not only to collect image data periodically but also to allow for changes in the monitoring procedure and device settings according to the plant condition and the environmental situation. For example, users want to obtain finer-detail images when the risk of pest occurrence becomes higher. In addition, the function of analyzing the collected image data according to users' requests is needed in order to provide useful information for decision support. The study of image analysis has been promoted [9,10], and some approaches for agricultural applications with image analysis such as pest and insect detection have been proposed [11,12]. However, special techniques and great effort are required to construct a monitoring system with an image analysis function that must be prepared according to each target and request. A flexible and smart monitoring system that performs complicated operations for various kinds of devices simply and provides suitable image analysis without great effort is desired. To meet such needs, we have proposed a Web-based sensor network (Fig. 1) in which Web-based devices such as Field Servers and IP cameras can provide useful information via the Internet as smart sensor nodes, and developed an autonomous management system, called an “Agent System”, to realize the Web-based sensor network. The Agent System with intelligent processing and flexible data analysis function manages various kinds of Web-based devices effectively at a remote site. Therefore, Web-based devices don't have intelligent processing on the inside like typical sensor nodes, but allow for smart monitoring by the Agent System. Moreover, by accessing not only Web-based devices but also Web application services that perform data processing, the Agent System also
566
T. Fukatsu et al. / Computer Standards & Interfaces 33 (2011) 565–573
Fig. 1. Web-based sensor network consisting of Field Servers and the Agent System. In the Web-based sensor network, the Agent System collects data from Field Servers at a remote site and analyzes them using Web applications to provide meaningful monitoring results in real-time.
provides various image analyses easily with a distributed data processing approach. In this paper, we describe the concept of the Web-based sensor network, the features and architecture of the Agent System, and the approach of distributed data processing by Web application services. To demonstrate the scalability and the effectiveness of our proposed system, we constructed and evaluated an experimental system in which the connection speed, the number of analysis Web application services and image data can be changed. We also constructed and evaluated a monitoring system which automatically detected the events of a farm operation from collected image data using Web application services in order to demonstrate the system's functionality. By conducting these experiments and field experiments, we could evaluate the effectiveness and practical potential of our proposed system. 2. Approaches 2.1. Web-based sensor network A typical sensor network is constructed of many sensor nodes, which are small sensor units with radio data links that communicate with each other to send monitoring data. A sensor network enables large-scale environmental monitoring with high scalability and robustness, but it is difficult to perform complicated operations and to treat a large amount of data because of the performance
deficiencies in the hardware. In order to realize a useful sensor network in agriculture, we have proposed a Web-based sensor network constructed of Web-based sensor nodes and the Agent System (Fig. 2). The Web-based sensor node is composed of several modules with Web servers connected with each other via the Ethernet. Such a sensor node has several advantageous features. First, we can use various kinds of low-cost commercial devices with the sensor node such as a wireless LAN and IP cameras, so monitoring image data can be performed effectively. Second, all sensor nodes are provided their data when their Web pages are accessed. That is, our proposed system has a “pull type [13]” architecture that functions interactively and flexibly compared with a “push type [14]” typical sensor network, so it is easy to change their scheduled actions depending on the situation after deploying them. The pull type architecture is also effective for long-term field monitoring under harsh conditions. It enables us to check the condition of each sensor node by accessing it at a remote site. Therefore, it is easy and effective to perform troubleshooting and to determine exactly what kinds of problems have occurred, especially important for sensor nodes exposed to severe conditions. Third, we can construct a smart monitoring system by separating intelligent function from sensor nodes. The sensor nodes themselves do not have highly programmable firmware on the inside, but we can control them easily via the Internet with a simple HTTP protocol. Therefore, by using a remote intelligent management system such as the Agent System, our system performs complicated monitoring with various
Fig. 2. Typical sensor network vs. Web-based sensor network. The Web-based sensor network is constructed of Web-based sensor nodes composed of several Web device modules and the Agent System that controls them with intelligent processing.
T. Fukatsu et al. / Computer Standards & Interfaces 33 (2011) 565–573
567
types of data analysis, and users need not care about onerous management of the system at the field site except when deploying the Web-based sensor nodes and connecting them to the Internet. In order to operate the Web-based sensor network effectively, the Agent System which serves as the management part of the sensor nodes must be constructed with several functions. The Agent System must fetch the data from each Web-based sensor node, so a function to access Web servers and store the data automatically is needed as a user collects data manually by accessing it with a Web browser. If the system were only used to collect data regularly on the Web, simple software such as a Web crawler [15] that automatically collects Web pages and stores data in databases could be applied, but the Agent System additionally requires the function of being able to change the monitoring procedure and the device setting flexibly. By implementing an autonomous management function with intelligent processing based on the configuration files, the Agent System can manage the sensor nodes with complicated operations according to the situation. In our proposed system, various kinds of data analysis are also needed, but it is difficult to gather all data analysis that users may want into the Agent System in advance. Instead, we arranged that the Agent System uses Web application services with analysis function as external processing modules by using the function of accessing Web servers. It can realize a versatile and expansible analysis function without changing or rebooting the main program, and this distributed data processing approach makes it possible to distribute calculation tasks.
and outputting the results. In some cases, extracting a desired value such as sensor data and control parameters from raw HTML data is required. By adding some parameters in the Profiles, the main program can search a target value from collected Web pages, and additionally provide a meaningful value calculated with the extracted raw values such as voltage output of the sensors and certain expressions such as calibration formulae. These results are stored in a database based on a certain template file which can be specified with various formats such as HTML, XML and CSV. By using these functions, the main program also acts as a middleware which outputs any data of Web contents with the same format so that it can avoid data format problems. In order to realize intelligent control, the main program implements a rule-based algorithm based on a number of IF–THEN rules [16]. This function constantly evaluates the given IF–THEN rules in the Profiles based on monitoring data and status values in the program, and then judges whether corresponding action commands should be executed normally, skipped until conditions change, or shifted to other action commands. A lot of IF–THEN rules act as autonomous decision functions, so the program can deal with sensor nodes depending on the situations in a given environment. The Agent System is managed via the Web interface that facilitates its control and its sensor nodes. Thanks to the Web interfaces, the Agent System can also manage other Agent Systems as just one of the sensor nodes in our system, so we can construct the Agent System as a high-scalable and distributed system.
2.2. Agent System architecture
2.3. Web application services for data processing
In order to realize the Agent System with the needed functions, we designed and developed it with the main program written in Java (JDK5.0), program configuration files named Profiles describing parameters of sensor nodes and request operations in XML format, and Web interfaces for easy management of the Agent System (Fig. 3). When the Profiles are set, the main program automatically executes their contents with a multi-thread. Naturally, the main program implements a function to access Web pages with HTTP. In this paper, it can access Web pages with basic authentication and handle “cookie” services, but HTTPS and ActiveX are not supported yet. In this program, some action commands that emulates user operations on a Web browser such as Post, Get, Search, Copy, Save, and so on are also set in order to provide flexible operations. By describing operation steps with a combination of these commands in the Profiles, the main program can manage various kinds of sensor nodes and execute the complicated operations the user desires. The system's has useful functions not only in accessing Web pages but also in processing the collected data with arithmetic operations
The Agent System can access not only sensor nodes but also various kinds of Web pages including Web applications which provide data processing services. Therefore, this system itself does not have data processing functions but it can provide them fully. This distributed data processing approach with Web application services has various advantages. When a certain data processing is needed, we only have to prepare or find a suitable Web application service, and it can be shared efficiently. Some researchers have developed data processing services on the Internet, and we can benefit from them using our proposed system. For example, an image analysis service with optical-flow, which generally requires a large amount of calculation, is provided as a Web application services on the Internet [17]. Yu et al. [18] developed the Remote Image Analysis Service (RIAS) which provides some image analysis using the Java RMI technique. An efficient image exchanging system that allows for cooperation with the RIAS was also developed [19]. Using the function of the Agent System, we can also realize complicated data processing with multiple Web application services
Fig. 3. System architecture of the Agent System consisting of the main program, Profiles, and the Web interface. The main program, which has an HTTP access function, a data processing function, a data output function, and an agent control function, executes its operation based on the Profiles describing the users' requests.
568
T. Fukatsu et al. / Computer Standards & Interfaces 33 (2011) 565–573
in combination and dynamic data processing depending on the situation without changing or rebooting the main program. Since the same Web application services are operated on many servers, processing tasks can be distributed according to calculation load and frequency of processing; hence, this system has scalability and effectiveness to handle a large amount of tasks and to reduce processing time. Meanwhile, we have to consider under what conditions this approach should be used. This approach requires frequent exchange of data on the Internet and exerts either a small or large influence on the network. We used an experimental evaluation under varying conditions of connection speed, volume of tasks and data processing type, to determine whether this approach would work well to reduce the total time of processing data including data transfer time. In this paper, we prepared three Web application services that perform basic image analysis to clip, subtract, and binarize inputted image data for the experimental evaluation of the distributed data processing approach. One Web application service outputs the result of clipped image data according to a specified range which is assigned by a given mask image, and it is usually used to cut outside of the target area from the raw image data. Another Web application service outputs the difference between two given image data by calculating the subtraction of each pixel after converting to 8-bit grayscale based on the NTSC coefficients method. Yet another Web application service outputs binarized image data in which each pixel of a given image is converted to black or white based on specified threshold values given by HSL or grayscale, and also outputs the total number of pixels that meet the threshold level. These Web application services are implemented by a Java Servlet, and the Agent System can use them with the POST method of HTTP. 3. Experimental evaluation 3.1. Distributed processing We evaluated the possibility of an Agent System that provides image analysis using Web application services and examined the scalability and effectiveness of the distributed data processing approach under various conditions. In this experiment, the Agent System performed binarization image analysis of four prepared test images and we measured the total time from sending the image data to receiving all results of the binarized image data and calculated values (Fig. 4). This image analysis is a common approach to calculating the Leaf Area Index for growth diagnosis in agriculture.
We prepared five small computers (CPU: PowerPC 266 MHz, RAM: 128 MB) by installing Vine Linux 3.1 for the Agent System and Web application services. One computer with the Agent System called the FSAB and the other four computers providing the Web application services (Web servers) were connected to each other by wireless LAN. For reference, the FSAB was equipped with the same function of the Web servers so that it could also perform image analysis as a standalone system. Two series of experiments were conducted with this experimental system, changing the communication speed, the size of the test image data and the number of Web servers (Fig. 5). In the first experiment, the communication speed and the size of image data are fixed, and we measured the total time when the number of Web servers was set at one, two or four. The test image data were in 24-bit color BMP format and the image size was 640 × 480 pixels (0.88 MB). The average communication speed between the FSAB and Web servers was 9.5 Mbps (SD = 1.1 Mbps); values were measured five times per Web server (totally twenty times). When four Web servers are active, the Agent System distributes the image data to each Web server. When only two Web servers are active, it distributes data from each of two images to each Web server. We also measured the average image analysis time, from receiving the image data to outputting the results on each Web servers. The total time and the image analysis time were measured ten times for each condition and averaged. Fig. 6 shows the results for the total time in this experiment. When the Agent System processed the image analysis without the Web server, as a stand-alone system, the total time (i.e. almost the image analysis time) was 6.2 s. When the Agent System processed it with the distributed data processing approach, the total processing times when using one, two and four Web servers were 8.7, 6.7 and 4.7 s. The greater the number of Web servers, the lesser the total time. In the case of using one Web server, the image analysis time was 6.0 s, almost the same as that in a stand-alone system, and the time it takes to transfer all image data (totally 3.5 MB) at this communication speed (9.5 Mbps) was also the same. The transfer time tends to decrease as the number of Web servers increase, though the decrease is not proportional because it depends on the processing ability of the sending and receiving devices, the condition of the wireless LAN and so on. The image analysis time also tends to decrease as the number of Web servers increase, but this effect depends on the processing ability of the Web servers, the program cording, the type of data processing, and so on. In this experiment, our proposed system could reduce the total time compared with a stand-alone system when the task was distributed to four Web servers.
Fig. 4. Web page of binarization image analysis (left) and an example of the result output (right). The test image data is uploaded with HSL parameters to obtain the LAI by the Agent System, and binarized image data and the number of white pixels are received.
T. Fukatsu et al. / Computer Standards & Interfaces 33 (2011) 565–573
569
Fig. 5. Experimental system for the distributed data processing approach. The Agent System accesses one or more Web servers to perform image analysis of four test images. The conditions of communication speed, size of image data and number of Web servers can be changed.
In the second experiment, we measured the total time by changing the communication speed and the size of image data with four active Web servers to examine their influence on the result. We prepared five types of BMP image data with sizes of 160 × 120, 320 × 240, 640 × 480, 800 × 600 and 1024 × 768 pixels (0.05, 0.22, 0.88, 1.37 and 2.25 MB). The communication speeds were set to 2.9, 5.8, 9.5 and 19.0 Mbps by changing the location of the wireless LAN. Fig. 7 shows the result of the total time in this experiment. The total time has a trend to increase approximately in proportion to the size of image data in every condition of the communication speed. The reason is that the computational load of this binarization image analysis is in proportion to the size of the image data and the transfer time is also in proportion to the size of image data. The total times in the conditions of 19.0 and 9.5 Mbps are shorter than in a stand-alone system, and the total times of 5.8 and 2.9 Mbps are almost the same as or longer than that. As the communication speed becomes slow, the total time becomes long because the transfer time is increased though the image analysis time remains the same. 3.2. Event detection application To evaluate the possibility and the effectiveness of an Agent System that seamlessly obtains image data, analyzes it and changes its own operation depending on the situation, we conducted an experiment to record the events of farm operation in detail by detecting them based on
Fig. 6. Result of total time with distributed data processing approach by changing the number of Web servers. Each total time and image analysis time is the average of ten separate trials. The more the number of Web servers is increased, the lesser the total time and the image analysis time are shortened.
the results of image analysis and IF–THEN rules. In this experiment, we deployed two Field Servers (FS-II, elab experience Inc.) in a series with 640 × 480-pixel image sensors as sensor nodes in an agricultural field. These Field Servers were managed by the Agent System via the Internet, and the communication speed between the Agent System and the terminal Field Server was about 12 Mbps. For image analysis, we set up two Web servers (running on the same computers described in the subsection about distributed processing) that can be accessed via the Internet with 48.0 Mbps. The Agent System performs image analysis of simple event detection based on a background differencing technique [20] using three Web application services to clip, subtract, and binarize image data provided on the two Web Servers. Fig. 8 shows the flow of the image analysis to detect events. After collecting image data, the Agent System accesses the clipping Web application service with the data and a mask image to lessen the impact of disturbance from raw image data. The mask image is set to pick up the target agricultural field, which accounts for 43.1% of the total image data. Then, the result is sent to the subtracting Web application service with reference image data to calculate the differences of the target field. Along the way, the reference image data is updated periodically to the latest image data that has few differences. Finally, the Agent System sends the subtracted result to the binarizing Web application services with a threshold value and judges whether or not a certain object is detected in the agricultural field based on the number of calculated white pixels.
Fig. 7. Result of total time with distributed data processing approach by changing the communication speed and the size of the image data. The total time increases depending on the size of the image data. The faster the communication speed, the lesser the total time is shortened.
570
T. Fukatsu et al. / Computer Standards & Interfaces 33 (2011) 565–573
Based on the preliminary experimental data, the threshold value for binarization is set to 64 out of an 8-bit grayscale, and the threshold value for judging object detection is set at 200 pixels. This experiment was conducted for 30 days in a barley field. The Agent System collected the image data and analyzed them at two-minute intervals to detect events such as harvesting, chopping, fertilizing and plowing operations. When the Agent System judged that an event of farm operation had occurred, it also collected additional image data at twenty-second intervals by changing its own operation. During the experiment, the Agent System collected 99.9% image data from Field Servers and analyzed it without a hitch using Web application services. The main reason for data loss was that the Agent System automatically skipped collecting image data when it had no response from the Field Servers for over three seconds. It usually took about ten seconds to do a series of procedures; with one second for collecting image data, four for image analysis, three for updating reference image data, and two for preparing a Web site of results. By checking all image data manually, the events of the farm operation were captured in 230 image data, of which the Agent System detected 197 (85.7%). The main reason for the missed detection was that some of the workers were hidden in crops, the color of the workers' uniforms was similar to that of the field, and workers and machines stopped for rest once in a while. On the other hand, the Agent System correctly judged 96.8% of no event image data. The main reason for the false detection was that image data had effects of wind, rain, car light in nighttime and unforeseen objects such as walkers on weekends. Fig. 9 shows part of the results of the number of white pixels calculated by the image analysis. On July 4, the number of white pixels surged because water droplets were on the camera. Chopping and plowing operations were detected on July 5 and July 9, respectively. In this experiment, we also evaluate the function by which the Agent System changes its operation depending on the situation. In order to collect more information when detecting a certain event of farm operation, we set several IF–THEN rules in combination with the results of image analysis and the given information that farm operations are only performed during daytime (08:00–18:00) on weekdays without rainy weather (i.e., rainfall of more than one millimeter per hour). Under the rule base, the Agent System found
261 events of farm operations and executed multi-thread operations to collect additional image data at twenty-second intervals. One hundred ninety-seven actual events of farm operations were detected in this system, and the aspects of the farm operations could be recorded in detail (Fig. 10). We also set another set of IF–THEN rules to change the Web server when the primary Web server had no response for more than three seconds, so that the system could analyze collected image data stably. The primary Web server was shut down ten times during the experiment, and the system worked well without problems by automatically switching to the other Web server. 3.3. Field experiments To evaluate the Agent System with regard to managing various Web-based sensor nodes effectively and stably, we deployed many sensor nodes in different locations all over the world and managed them with this system via the Internet for a long time. As of March 2009, totally 124 Field Servers assembled out of six types of Web monitoring units and seven types of IP cameras were deployed in twenty-eight sites from nine countries such as Japan, China, and the United States. The Agent Systems were executed at a remote site with VPN connections and performed monitoring and analyzing operations based on their Profiles, which differed for each parameter of device setting, operation procedure, data processing, and display contents according to the users' requests. This system has performed reliably for more than seven years since February 2002, and collected an enormous amount of data equaling about 2.2 TB. Some collected image data was used to measure the Leaf Area Index for growth diagnosis using the Web application services of binarization image analysis. Other image data were used to monitor insect occurrences on pheromone traps by detecting target shapes from it with Web application services. Others, processed by filtering out unnecessary image data, were used to pick out some events of farm operations for keeping a farm diary. These results were outputted on the viewer Web pages which were created by the Agent System according to the output templates described in the Profiles. The standard output template displays the thumbnailed or analyzed image data with a cutoff
Fig. 8. Flow of image analysis to detect events of farm operations. Using three different Web application services, the Agent System clips the target area of agricultural field from the collected image data, subtracts it with reference to the image data, and binarizes the subtraction result to judge whether a certain object is detected.
T. Fukatsu et al. / Computer Standards & Interfaces 33 (2011) 565–573
Fig. 9. Results of image analysis to detect events of farm operation. On July 4, the number of white pixels calculated by the image analysis surged because of water droplets on the camera. Chopping and plowing operations were detected on July 5 and July 9, respectively.
animation that is linked to the original image data so that users can inspect image data quickly and view original images by clicking on it. The sensor data and analyzed results are also displayed in synchronization with the image data on the viewer Web pages. Fig. 11 shows an example of archived data displayed on the viewer Web page, and the image data of harvesting operations at a grape farm are shown automatically with neighboring weather information. By using the Agent System, users can obtain useful information easily and effectively.
4. Discussion and conclusion In order to realize a smart image monitoring system with expansible data processing, we have proposed a Web-based sensor network in which an Agent System manages various kinds of Web-based sensor nodes and performs desired image analysis with Web application services. Our proposed system is designed 1), to provide flexible monitoring and operations according to the users' requests by separating intelligent function from sensor nodes and 2), to provide versatile and scalable image analysis by distributing data processing to Web servers. We demonstrated that this distributed approach enabled us to process a lot of data effectively with less time in certain conditions and that this system could perform complicated management satisfactorily such as farm operation monitoring that automatically changes the protocol depending on the results of its analysis of collected image data.
571
This distributed data processing approach is relatively ineffective in reducing the total time at a slow communication speed because it requires extra time to transfer data besides the analysis time even though the analysis time can be shortened by this approach. Conversely, this approach works efficiently when the ratio of the transfer time in the total time is low. Therefore, it is more effective for image-processing functions requiring heavy calculation load such as pattern recognition than for other simpler functions. As the communication speed becomes faster with more advanced network environments in the future, our proposed approach can be used with advantage in many situations because the ratio of the transfer time will become progressively smaller. In the case of handling many tasks to perform image analysis, this distributed data processing approach has the potential of scalability by increasing the number of Web application services. In order to make the most of the features of this approach, we should consider how optimally the Agent System distributes all tasks to Web servers depending on the situation. It is important to divide all tasks to Web servers evenly based on the processing capacity of each Web server, the communication speed to each Web server, the data size, the types of data processing, and so on. For example, it is better to use Web servers with high processing capacity when the data processing requires a large amount of calculation, and it is better to use the Web servers under high-speed communication when the size of data is large. In order to maintain the performance of this distributed approach when processing a large amount of data, it is also important to consider the network and management load of the Agent System. By using the developed Multi-Agent System [21] in which many Agent Systems manage their tasks in a coordinated manner, this objective can be achieved. In the experiment of the event detection, we demonstrated that the proposed system could realize the requested operations adequately using the rule-base function and various Web application services in combination that perform image analysis using background differentiation techniques. This time, we simply established a few IF–THEN rules and used a simple image analysis algorithm. By preparing more sensitive IF–THEN rules and detailed operation steps, this system can work more intelligently and flexibly according to the situation. Our proposed system can perform various image analyses simply by preparing a suitable Web application service, so it is also easy to equip it with advanced image analysis algorithms for various applications. In some field experiments, we not only examined the possibility and the stability of this system but also tried to construct some applications to provide useful information to
Fig. 10. Example of collecting additional image data after detecting events of farm operations. When the Agent System detects an event of plowing operation according to the described rule base, it executes another monitoring protocol to obtain high-frequency image data with a multi-thread.
572
T. Fukatsu et al. / Computer Standards & Interfaces 33 (2011) 565–573
Fig. 11. Example of archived data displayed on viewer Web page. By clicking a target icon of a Field Server on the main Web page (left figure), we can see a Web page (right figure) showing selected image data based on the results of image analysis and the sensor data.
farmers by using the function of the system. This system facilitates the introduction of some applications that have been useful in the laboratory stage, such as maturity evaluation and growth diagnosis, [22,23] to the field stage because it enables us to construct a field monitoring system without great effort and to incorporate these applications simply by preparing the contents on the Web. Thus, it is expected to help provide some useful support systems for practical use. If the Web-based control of equipment such as irrigation machines and environmental control units can be prepared, this system would also be able to control them automatically based on the monitoring results. This proposed system has the possibility to be applied to various agricultural situations. The proposed system has the potential to perform advanced monitoring with Web-based sensor nodes and to be utilized for various purposes. By using the system's functionality, collected data can be provided in a consistent format. Therefore, it can be used as middleware that enables users to treat various sensor nodes without regard to the differences and enables applications to avoid data format problems. By preparing Web interfaces in which users can intelligibly control some objects on the Web such as sensor nodes, this system can also provide a wrapper function to perform a series of actions and provide desired results automatically according to users' request on the interface. In the same manner, it helps non-expert users to control sensor nodes without complicated steps. This system can provide users who lack specialized technical experience useful information as a mashup that combines with many kinds of Web application services and database into a single representation and executes a series of requested services seamlessly. The more Web-based monitoring devices, Web application services, and the communication environment are developed, the more our proposed system is expected to be used as an effective tool. To improve the system, several tasks need to be accomplished, such as simplifying the operating interfaces and achieving effective management with intelligent algorithms. Such an improved system would have more potential to perform many applications in a wide variety of situations. References [1] J.M. Kahn, R.H. Katz, K.S.J. Pister, Next century challenges: mobile networking for “smart dust”, Proceedings of the 5th Annual ACM/IEEE International Conference on Mobile Computing and Networking, Seattle, WA, USA, August 1999, pp. 271–278. [2] I.F. Akyildiz, W. Su, Y. Sankarasubramaniam, E. Cayirci, Wireless sensor networks: a survey, Computer Networks 38 (2002) 393–422. [3] R.D. Tillett, Image analysis for agricultural processes: a review of potential opportunities, Journal of Agricultural Engineering Research 50 (1991) 247–258. [4] N. Shimoda, T. Kataoka, H. Okamoto, M. Terawaki, S. Hata, Automatic pest counting system using image processing technique (in Japanese with English
[5] [6]
[7]
[8]
[9]
[10] [11] [12]
[13]
[14]
[15]
[16]
[17] [18]
[19]
[20]
[21]
[22] [23]
summary), Journal of the Japanese Society of Agricultural Machinery 68 (2006) 59–64. T. Fukatsu, M. Hirafuji, Field monitoring using sensor-nodes with a Web server, Journal of Robotics and Mechatronics 17 (2005) 164–172. S. Ninomiya, K. Kunio, Internet remote camera system and its application to agriculture, in: G. Schiefer, R. Helbig, U. Rickert (Eds.), Perspectives of Modern Information and Communication Systems in Agriculture, Food Production and Environmental Control, University Bonn-ILB, Germany, 1999, pp. 75–82. Y. Motonaga, N. Iguchi, N. Ohtake, T. Kameoka, Image monitoring of paddy field using BIX image broker system, Proceedings of SICE Annual Conference 2007, Kagawa, Japan, September 2007, pp. 224–229. T. Fukatsu, M. Hirafuji, T. Kiura, Construction of support tools to assist an Agent System, Proceedings of IAALD-AFITA-WCCA 2008, Tokyo, Japan, August 2008, pp. 221–227. K. Toyama, J. Krumm, B. Brumitt, B. Meyers, Wallflower: principles and practice of background maintenance, Proceedings of Seventh International Conference on Computer Vision, Kerkyra, Greece, September 1999, pp. 255–261. R.J. Radke, Image change detection algorithms: a systematic survey, IEEE Trans. Image Process. 14 (2005) 294–307. C. Bauchi, T. Rath, Prototype of a vision based system for measurements of white fly infestation, ISHS Acta Horticulturae 691 (2005) 773–780. P. Boissard, V. Martin, S. Moisan, A cognitive vision approach to early pest detection in greenhouse crops, Computers and Electronics in Agriculture 62 (2008) 81–93. Z. Duan, Y. Dong, K. Gopalan, Push vs. pull: implications of protocol design on controlling unwanted traffic, Proceedings of Steps to Reducing Unwanted Traffic on the Internet Workshop, Cambridge, MA, USA, July 2005, pp. 25–30. M. Hauswirth, M. Jazayeri, A component and communication model for push systems, Proceedings of 7th European software engineering conference, Toulouse, France, September 1999, pp. 20–38. B. Pinkerton, Finding what people want: experiences with the Webcrawler, Proceedings of the 2nd International World Wide Web conference, Chicago, IL, USA, October 1994, pp. 821–829. L. Brownston, R. Farrell, E. Kant, N. Martin, Programming Expert Systems in OPS5: an Introduction to Rule-Based Programming, Addison-Weskey Longman Publishing Co., Inc., Boston, MA, USA, 1985. C. Sun, Fast optical flow using 3D shortest path techniques, Image and Vision Computing 20 (2002) 981–991. X. Yu, M. Laurenson, T. Kiura, S. Ninomiya, RIAS: an approach to provide internetaccessible image analysis service, Agricultural Information Research 16 (2007) 212–218. N. Iguchi, Y. Motonaga, F. Uchio, S. Ninomiya, T. Kameoka, Development of a gateway-P2P-node for enhancing the function of BIX image broker system (in Japanese with English summary), Agricultural Information Research 15 (2006) 281–292. A.M. Elgammal, D. David, L.S. Davis, Non-parametric model for background subtraction, in: D. Vernon (Ed.), Lecture Notes in Computer Science 1843, Springer, New York, NY, USA, 2000, pp. 751–767. T. Fukatsu, T. Kiura, T. Tanaka, M. Hirafuji, Hierarchical agent system for Webbased sensor network, Proceedings of 2007 International Symposium on Applications and the Internet Workshops, Hiroshima, Japan, January 2007, p. 77. K. Choi, G. Lee, Y.J. Han, J.M. Bun, Tomato maturity evaluation using color image analysis, Transactions of the ASABE 38 (1995) 171–176. K. Iwaya, H. Yamamoto, Diagnosis of optimal harvesting time of rice using digital imaging, Journal of Agricultural Meteorology 60 (2005) 981–984.
T. Fukatsu et al. / Computer Standards & Interfaces 33 (2011) 565–573 Tokihiro Fukatsu Name: Affiliation: National Agricultural Research Center, National Agriculture and Food Research Organization Address: 3-1-1 Kannondai, Tsukuba, Ibaraki, 305-8666 Japan Brief biographical history: 1998 Received M. Eng from Tokyo Institute of Technology 1998 — FUJITSU LIMITED 2001 — National Agricultural Research Center 2007 Received Ph. D (Ag) from Tsukuba University
Name: Takuji Kiura Affiliation: National Agricultural Research Center, National Agriculture and Food Research Organization Address: 3-1-1 Kannondai, Tsukuba, Ibaraki, 305-8666 Japan Brief biographical history: 1989 — Researcher, Tropical Agriculture Research Center, Japan 1995 — Researcher, National Agriculture Research Center, Japan
573
Name: Masayuki Hirafuji Affiliation: National Agricultural Research Center, National Agriculture and Food Research Organization Address: 3-1-1 Kannondai, Tsukuba, Ibaraki, 305-8666 Japan Brief biographical history: 1983 Received M. Ag from University of Tokyo 1983 — National Agriculture Research Center 1994 — Research coordinator, MAFF (Ministry of Agriculture, Forestry and Fisheries) 1995 Received Ph D.(Ag) from University of Tokyo 1996 — National Agricultural Research Center
Contents lists available at ScienceDirect
Computer Standards & Interfaces j o u r n a l h o m e p a g e : w w w. e l s ev i e r. c o m / l o c a t e / c s i
A web-based sensor network system with distributed data processing approach via web application Tokihiro Fukatsu ⁎, Takuji Kiura, Masayuki Hirafuji National Agricultural Research Center, National Agriculture and Food Research Organization, 3-1-1 Kannondai, Tsukuba, Ibaraki, 305-8666, Japan
a r t i c l e
i n f o
Article history: Received 3 December 2010 Accepted 16 March 2011 Available online 30 March 2011 Keywords: Sensor network Image monitoring Agriculture Web application Distributed data processing
a b s t r a c t We have proposed a Web-based sensor network constructed of Web-based sensor nodes and a remote management system. The Web-based sensor nodes consist of communication units and measurement devices with Web servers. The management system has intelligent processing and rule-based function to manage them flexibly via the Internet and performs various image analyses easily with Web application services. By distributing the image analyses to Web application services, our proposed system provides versatile and scalable data processing. We demonstrated that it can realize the desired image analyses effectively and perform complicated management by changing its operations depending on the results of analysis. © 2011 Elsevier B.V. All rights reserved.
1. Introduction In modern agriculture, it is important to monitor crop growth, the field environment and farm operations to increase agricultural productivity and produce high quality products. A key technology for monitoring them in open fields is a sensor network [1,2], but it is difficult to apply it to farmers' needs without modification. Few sensor networks have camera devices because they are designed to reduce data transmission for power saving, although image data provides useful and helpful information for agricultural users to check crop conditions, estimate the influence of diseases, detect vermin and record farming operations [3,4]. On the other hand, it has recently become possible to obtain image data at field sites by using Field Servers [5] and Internet protocol (IP) cameras. The Field Server, a kind of sensor networks for agricultural use, has various kinds of sensors including image sensors, a wireless LAN to provide high-speed transmission, which enables the use of high-resolution image data, and a measurement unit with a Web server to perform easy management at a remote site via the Internet. Thanks to the development of information technology, we can also use high-performance IP cameras with a Web server at a reasonable price. Because of their ease of use in agricultural fields, these Web-based monitoring devices have come into use, and monitoring systems with these devices have become increasingly important [6,7]. However, there are some obstacles to realizing Web-based monitoring for practical purposes. There are numerous kinds of ⁎ Corresponding author. Tel.: + 81 29 838 7177; fax: + 81 29 838 8551. E-mail addresses: [email protected] (T. Fukatsu), [email protected] (T. Kiura), [email protected] (M. Hirafuji). 0920-5489/$ – see front matter © 2011 Elsevier B.V. All rights reserved. doi:10.1016/j.csi.2011.03.002
Field Servers and IP cameras [8], and each device has different controls and poor functions. Therefore, it is difficult and troublesome for nonexpert users to apply them to a monitoring system for long periods. In the case of agriculture, the desired system must be able not only to collect image data periodically but also to allow for changes in the monitoring procedure and device settings according to the plant condition and the environmental situation. For example, users want to obtain finer-detail images when the risk of pest occurrence becomes higher. In addition, the function of analyzing the collected image data according to users' requests is needed in order to provide useful information for decision support. The study of image analysis has been promoted [9,10], and some approaches for agricultural applications with image analysis such as pest and insect detection have been proposed [11,12]. However, special techniques and great effort are required to construct a monitoring system with an image analysis function that must be prepared according to each target and request. A flexible and smart monitoring system that performs complicated operations for various kinds of devices simply and provides suitable image analysis without great effort is desired. To meet such needs, we have proposed a Web-based sensor network (Fig. 1) in which Web-based devices such as Field Servers and IP cameras can provide useful information via the Internet as smart sensor nodes, and developed an autonomous management system, called an “Agent System”, to realize the Web-based sensor network. The Agent System with intelligent processing and flexible data analysis function manages various kinds of Web-based devices effectively at a remote site. Therefore, Web-based devices don't have intelligent processing on the inside like typical sensor nodes, but allow for smart monitoring by the Agent System. Moreover, by accessing not only Web-based devices but also Web application services that perform data processing, the Agent System also
566
T. Fukatsu et al. / Computer Standards & Interfaces 33 (2011) 565–573
Fig. 1. Web-based sensor network consisting of Field Servers and the Agent System. In the Web-based sensor network, the Agent System collects data from Field Servers at a remote site and analyzes them using Web applications to provide meaningful monitoring results in real-time.
provides various image analyses easily with a distributed data processing approach. In this paper, we describe the concept of the Web-based sensor network, the features and architecture of the Agent System, and the approach of distributed data processing by Web application services. To demonstrate the scalability and the effectiveness of our proposed system, we constructed and evaluated an experimental system in which the connection speed, the number of analysis Web application services and image data can be changed. We also constructed and evaluated a monitoring system which automatically detected the events of a farm operation from collected image data using Web application services in order to demonstrate the system's functionality. By conducting these experiments and field experiments, we could evaluate the effectiveness and practical potential of our proposed system. 2. Approaches 2.1. Web-based sensor network A typical sensor network is constructed of many sensor nodes, which are small sensor units with radio data links that communicate with each other to send monitoring data. A sensor network enables large-scale environmental monitoring with high scalability and robustness, but it is difficult to perform complicated operations and to treat a large amount of data because of the performance
deficiencies in the hardware. In order to realize a useful sensor network in agriculture, we have proposed a Web-based sensor network constructed of Web-based sensor nodes and the Agent System (Fig. 2). The Web-based sensor node is composed of several modules with Web servers connected with each other via the Ethernet. Such a sensor node has several advantageous features. First, we can use various kinds of low-cost commercial devices with the sensor node such as a wireless LAN and IP cameras, so monitoring image data can be performed effectively. Second, all sensor nodes are provided their data when their Web pages are accessed. That is, our proposed system has a “pull type [13]” architecture that functions interactively and flexibly compared with a “push type [14]” typical sensor network, so it is easy to change their scheduled actions depending on the situation after deploying them. The pull type architecture is also effective for long-term field monitoring under harsh conditions. It enables us to check the condition of each sensor node by accessing it at a remote site. Therefore, it is easy and effective to perform troubleshooting and to determine exactly what kinds of problems have occurred, especially important for sensor nodes exposed to severe conditions. Third, we can construct a smart monitoring system by separating intelligent function from sensor nodes. The sensor nodes themselves do not have highly programmable firmware on the inside, but we can control them easily via the Internet with a simple HTTP protocol. Therefore, by using a remote intelligent management system such as the Agent System, our system performs complicated monitoring with various
Fig. 2. Typical sensor network vs. Web-based sensor network. The Web-based sensor network is constructed of Web-based sensor nodes composed of several Web device modules and the Agent System that controls them with intelligent processing.
T. Fukatsu et al. / Computer Standards & Interfaces 33 (2011) 565–573
567
types of data analysis, and users need not care about onerous management of the system at the field site except when deploying the Web-based sensor nodes and connecting them to the Internet. In order to operate the Web-based sensor network effectively, the Agent System which serves as the management part of the sensor nodes must be constructed with several functions. The Agent System must fetch the data from each Web-based sensor node, so a function to access Web servers and store the data automatically is needed as a user collects data manually by accessing it with a Web browser. If the system were only used to collect data regularly on the Web, simple software such as a Web crawler [15] that automatically collects Web pages and stores data in databases could be applied, but the Agent System additionally requires the function of being able to change the monitoring procedure and the device setting flexibly. By implementing an autonomous management function with intelligent processing based on the configuration files, the Agent System can manage the sensor nodes with complicated operations according to the situation. In our proposed system, various kinds of data analysis are also needed, but it is difficult to gather all data analysis that users may want into the Agent System in advance. Instead, we arranged that the Agent System uses Web application services with analysis function as external processing modules by using the function of accessing Web servers. It can realize a versatile and expansible analysis function without changing or rebooting the main program, and this distributed data processing approach makes it possible to distribute calculation tasks.
and outputting the results. In some cases, extracting a desired value such as sensor data and control parameters from raw HTML data is required. By adding some parameters in the Profiles, the main program can search a target value from collected Web pages, and additionally provide a meaningful value calculated with the extracted raw values such as voltage output of the sensors and certain expressions such as calibration formulae. These results are stored in a database based on a certain template file which can be specified with various formats such as HTML, XML and CSV. By using these functions, the main program also acts as a middleware which outputs any data of Web contents with the same format so that it can avoid data format problems. In order to realize intelligent control, the main program implements a rule-based algorithm based on a number of IF–THEN rules [16]. This function constantly evaluates the given IF–THEN rules in the Profiles based on monitoring data and status values in the program, and then judges whether corresponding action commands should be executed normally, skipped until conditions change, or shifted to other action commands. A lot of IF–THEN rules act as autonomous decision functions, so the program can deal with sensor nodes depending on the situations in a given environment. The Agent System is managed via the Web interface that facilitates its control and its sensor nodes. Thanks to the Web interfaces, the Agent System can also manage other Agent Systems as just one of the sensor nodes in our system, so we can construct the Agent System as a high-scalable and distributed system.
2.2. Agent System architecture
2.3. Web application services for data processing
In order to realize the Agent System with the needed functions, we designed and developed it with the main program written in Java (JDK5.0), program configuration files named Profiles describing parameters of sensor nodes and request operations in XML format, and Web interfaces for easy management of the Agent System (Fig. 3). When the Profiles are set, the main program automatically executes their contents with a multi-thread. Naturally, the main program implements a function to access Web pages with HTTP. In this paper, it can access Web pages with basic authentication and handle “cookie” services, but HTTPS and ActiveX are not supported yet. In this program, some action commands that emulates user operations on a Web browser such as Post, Get, Search, Copy, Save, and so on are also set in order to provide flexible operations. By describing operation steps with a combination of these commands in the Profiles, the main program can manage various kinds of sensor nodes and execute the complicated operations the user desires. The system's has useful functions not only in accessing Web pages but also in processing the collected data with arithmetic operations
The Agent System can access not only sensor nodes but also various kinds of Web pages including Web applications which provide data processing services. Therefore, this system itself does not have data processing functions but it can provide them fully. This distributed data processing approach with Web application services has various advantages. When a certain data processing is needed, we only have to prepare or find a suitable Web application service, and it can be shared efficiently. Some researchers have developed data processing services on the Internet, and we can benefit from them using our proposed system. For example, an image analysis service with optical-flow, which generally requires a large amount of calculation, is provided as a Web application services on the Internet [17]. Yu et al. [18] developed the Remote Image Analysis Service (RIAS) which provides some image analysis using the Java RMI technique. An efficient image exchanging system that allows for cooperation with the RIAS was also developed [19]. Using the function of the Agent System, we can also realize complicated data processing with multiple Web application services
Fig. 3. System architecture of the Agent System consisting of the main program, Profiles, and the Web interface. The main program, which has an HTTP access function, a data processing function, a data output function, and an agent control function, executes its operation based on the Profiles describing the users' requests.
568
T. Fukatsu et al. / Computer Standards & Interfaces 33 (2011) 565–573
in combination and dynamic data processing depending on the situation without changing or rebooting the main program. Since the same Web application services are operated on many servers, processing tasks can be distributed according to calculation load and frequency of processing; hence, this system has scalability and effectiveness to handle a large amount of tasks and to reduce processing time. Meanwhile, we have to consider under what conditions this approach should be used. This approach requires frequent exchange of data on the Internet and exerts either a small or large influence on the network. We used an experimental evaluation under varying conditions of connection speed, volume of tasks and data processing type, to determine whether this approach would work well to reduce the total time of processing data including data transfer time. In this paper, we prepared three Web application services that perform basic image analysis to clip, subtract, and binarize inputted image data for the experimental evaluation of the distributed data processing approach. One Web application service outputs the result of clipped image data according to a specified range which is assigned by a given mask image, and it is usually used to cut outside of the target area from the raw image data. Another Web application service outputs the difference between two given image data by calculating the subtraction of each pixel after converting to 8-bit grayscale based on the NTSC coefficients method. Yet another Web application service outputs binarized image data in which each pixel of a given image is converted to black or white based on specified threshold values given by HSL or grayscale, and also outputs the total number of pixels that meet the threshold level. These Web application services are implemented by a Java Servlet, and the Agent System can use them with the POST method of HTTP. 3. Experimental evaluation 3.1. Distributed processing We evaluated the possibility of an Agent System that provides image analysis using Web application services and examined the scalability and effectiveness of the distributed data processing approach under various conditions. In this experiment, the Agent System performed binarization image analysis of four prepared test images and we measured the total time from sending the image data to receiving all results of the binarized image data and calculated values (Fig. 4). This image analysis is a common approach to calculating the Leaf Area Index for growth diagnosis in agriculture.
We prepared five small computers (CPU: PowerPC 266 MHz, RAM: 128 MB) by installing Vine Linux 3.1 for the Agent System and Web application services. One computer with the Agent System called the FSAB and the other four computers providing the Web application services (Web servers) were connected to each other by wireless LAN. For reference, the FSAB was equipped with the same function of the Web servers so that it could also perform image analysis as a standalone system. Two series of experiments were conducted with this experimental system, changing the communication speed, the size of the test image data and the number of Web servers (Fig. 5). In the first experiment, the communication speed and the size of image data are fixed, and we measured the total time when the number of Web servers was set at one, two or four. The test image data were in 24-bit color BMP format and the image size was 640 × 480 pixels (0.88 MB). The average communication speed between the FSAB and Web servers was 9.5 Mbps (SD = 1.1 Mbps); values were measured five times per Web server (totally twenty times). When four Web servers are active, the Agent System distributes the image data to each Web server. When only two Web servers are active, it distributes data from each of two images to each Web server. We also measured the average image analysis time, from receiving the image data to outputting the results on each Web servers. The total time and the image analysis time were measured ten times for each condition and averaged. Fig. 6 shows the results for the total time in this experiment. When the Agent System processed the image analysis without the Web server, as a stand-alone system, the total time (i.e. almost the image analysis time) was 6.2 s. When the Agent System processed it with the distributed data processing approach, the total processing times when using one, two and four Web servers were 8.7, 6.7 and 4.7 s. The greater the number of Web servers, the lesser the total time. In the case of using one Web server, the image analysis time was 6.0 s, almost the same as that in a stand-alone system, and the time it takes to transfer all image data (totally 3.5 MB) at this communication speed (9.5 Mbps) was also the same. The transfer time tends to decrease as the number of Web servers increase, though the decrease is not proportional because it depends on the processing ability of the sending and receiving devices, the condition of the wireless LAN and so on. The image analysis time also tends to decrease as the number of Web servers increase, but this effect depends on the processing ability of the Web servers, the program cording, the type of data processing, and so on. In this experiment, our proposed system could reduce the total time compared with a stand-alone system when the task was distributed to four Web servers.
Fig. 4. Web page of binarization image analysis (left) and an example of the result output (right). The test image data is uploaded with HSL parameters to obtain the LAI by the Agent System, and binarized image data and the number of white pixels are received.
T. Fukatsu et al. / Computer Standards & Interfaces 33 (2011) 565–573
569
Fig. 5. Experimental system for the distributed data processing approach. The Agent System accesses one or more Web servers to perform image analysis of four test images. The conditions of communication speed, size of image data and number of Web servers can be changed.
In the second experiment, we measured the total time by changing the communication speed and the size of image data with four active Web servers to examine their influence on the result. We prepared five types of BMP image data with sizes of 160 × 120, 320 × 240, 640 × 480, 800 × 600 and 1024 × 768 pixels (0.05, 0.22, 0.88, 1.37 and 2.25 MB). The communication speeds were set to 2.9, 5.8, 9.5 and 19.0 Mbps by changing the location of the wireless LAN. Fig. 7 shows the result of the total time in this experiment. The total time has a trend to increase approximately in proportion to the size of image data in every condition of the communication speed. The reason is that the computational load of this binarization image analysis is in proportion to the size of the image data and the transfer time is also in proportion to the size of image data. The total times in the conditions of 19.0 and 9.5 Mbps are shorter than in a stand-alone system, and the total times of 5.8 and 2.9 Mbps are almost the same as or longer than that. As the communication speed becomes slow, the total time becomes long because the transfer time is increased though the image analysis time remains the same. 3.2. Event detection application To evaluate the possibility and the effectiveness of an Agent System that seamlessly obtains image data, analyzes it and changes its own operation depending on the situation, we conducted an experiment to record the events of farm operation in detail by detecting them based on
Fig. 6. Result of total time with distributed data processing approach by changing the number of Web servers. Each total time and image analysis time is the average of ten separate trials. The more the number of Web servers is increased, the lesser the total time and the image analysis time are shortened.
the results of image analysis and IF–THEN rules. In this experiment, we deployed two Field Servers (FS-II, elab experience Inc.) in a series with 640 × 480-pixel image sensors as sensor nodes in an agricultural field. These Field Servers were managed by the Agent System via the Internet, and the communication speed between the Agent System and the terminal Field Server was about 12 Mbps. For image analysis, we set up two Web servers (running on the same computers described in the subsection about distributed processing) that can be accessed via the Internet with 48.0 Mbps. The Agent System performs image analysis of simple event detection based on a background differencing technique [20] using three Web application services to clip, subtract, and binarize image data provided on the two Web Servers. Fig. 8 shows the flow of the image analysis to detect events. After collecting image data, the Agent System accesses the clipping Web application service with the data and a mask image to lessen the impact of disturbance from raw image data. The mask image is set to pick up the target agricultural field, which accounts for 43.1% of the total image data. Then, the result is sent to the subtracting Web application service with reference image data to calculate the differences of the target field. Along the way, the reference image data is updated periodically to the latest image data that has few differences. Finally, the Agent System sends the subtracted result to the binarizing Web application services with a threshold value and judges whether or not a certain object is detected in the agricultural field based on the number of calculated white pixels.
Fig. 7. Result of total time with distributed data processing approach by changing the communication speed and the size of the image data. The total time increases depending on the size of the image data. The faster the communication speed, the lesser the total time is shortened.
570
T. Fukatsu et al. / Computer Standards & Interfaces 33 (2011) 565–573
Based on the preliminary experimental data, the threshold value for binarization is set to 64 out of an 8-bit grayscale, and the threshold value for judging object detection is set at 200 pixels. This experiment was conducted for 30 days in a barley field. The Agent System collected the image data and analyzed them at two-minute intervals to detect events such as harvesting, chopping, fertilizing and plowing operations. When the Agent System judged that an event of farm operation had occurred, it also collected additional image data at twenty-second intervals by changing its own operation. During the experiment, the Agent System collected 99.9% image data from Field Servers and analyzed it without a hitch using Web application services. The main reason for data loss was that the Agent System automatically skipped collecting image data when it had no response from the Field Servers for over three seconds. It usually took about ten seconds to do a series of procedures; with one second for collecting image data, four for image analysis, three for updating reference image data, and two for preparing a Web site of results. By checking all image data manually, the events of the farm operation were captured in 230 image data, of which the Agent System detected 197 (85.7%). The main reason for the missed detection was that some of the workers were hidden in crops, the color of the workers' uniforms was similar to that of the field, and workers and machines stopped for rest once in a while. On the other hand, the Agent System correctly judged 96.8% of no event image data. The main reason for the false detection was that image data had effects of wind, rain, car light in nighttime and unforeseen objects such as walkers on weekends. Fig. 9 shows part of the results of the number of white pixels calculated by the image analysis. On July 4, the number of white pixels surged because water droplets were on the camera. Chopping and plowing operations were detected on July 5 and July 9, respectively. In this experiment, we also evaluate the function by which the Agent System changes its operation depending on the situation. In order to collect more information when detecting a certain event of farm operation, we set several IF–THEN rules in combination with the results of image analysis and the given information that farm operations are only performed during daytime (08:00–18:00) on weekdays without rainy weather (i.e., rainfall of more than one millimeter per hour). Under the rule base, the Agent System found
261 events of farm operations and executed multi-thread operations to collect additional image data at twenty-second intervals. One hundred ninety-seven actual events of farm operations were detected in this system, and the aspects of the farm operations could be recorded in detail (Fig. 10). We also set another set of IF–THEN rules to change the Web server when the primary Web server had no response for more than three seconds, so that the system could analyze collected image data stably. The primary Web server was shut down ten times during the experiment, and the system worked well without problems by automatically switching to the other Web server. 3.3. Field experiments To evaluate the Agent System with regard to managing various Web-based sensor nodes effectively and stably, we deployed many sensor nodes in different locations all over the world and managed them with this system via the Internet for a long time. As of March 2009, totally 124 Field Servers assembled out of six types of Web monitoring units and seven types of IP cameras were deployed in twenty-eight sites from nine countries such as Japan, China, and the United States. The Agent Systems were executed at a remote site with VPN connections and performed monitoring and analyzing operations based on their Profiles, which differed for each parameter of device setting, operation procedure, data processing, and display contents according to the users' requests. This system has performed reliably for more than seven years since February 2002, and collected an enormous amount of data equaling about 2.2 TB. Some collected image data was used to measure the Leaf Area Index for growth diagnosis using the Web application services of binarization image analysis. Other image data were used to monitor insect occurrences on pheromone traps by detecting target shapes from it with Web application services. Others, processed by filtering out unnecessary image data, were used to pick out some events of farm operations for keeping a farm diary. These results were outputted on the viewer Web pages which were created by the Agent System according to the output templates described in the Profiles. The standard output template displays the thumbnailed or analyzed image data with a cutoff
Fig. 8. Flow of image analysis to detect events of farm operations. Using three different Web application services, the Agent System clips the target area of agricultural field from the collected image data, subtracts it with reference to the image data, and binarizes the subtraction result to judge whether a certain object is detected.
T. Fukatsu et al. / Computer Standards & Interfaces 33 (2011) 565–573
Fig. 9. Results of image analysis to detect events of farm operation. On July 4, the number of white pixels calculated by the image analysis surged because of water droplets on the camera. Chopping and plowing operations were detected on July 5 and July 9, respectively.
animation that is linked to the original image data so that users can inspect image data quickly and view original images by clicking on it. The sensor data and analyzed results are also displayed in synchronization with the image data on the viewer Web pages. Fig. 11 shows an example of archived data displayed on the viewer Web page, and the image data of harvesting operations at a grape farm are shown automatically with neighboring weather information. By using the Agent System, users can obtain useful information easily and effectively.
4. Discussion and conclusion In order to realize a smart image monitoring system with expansible data processing, we have proposed a Web-based sensor network in which an Agent System manages various kinds of Web-based sensor nodes and performs desired image analysis with Web application services. Our proposed system is designed 1), to provide flexible monitoring and operations according to the users' requests by separating intelligent function from sensor nodes and 2), to provide versatile and scalable image analysis by distributing data processing to Web servers. We demonstrated that this distributed approach enabled us to process a lot of data effectively with less time in certain conditions and that this system could perform complicated management satisfactorily such as farm operation monitoring that automatically changes the protocol depending on the results of its analysis of collected image data.
571
This distributed data processing approach is relatively ineffective in reducing the total time at a slow communication speed because it requires extra time to transfer data besides the analysis time even though the analysis time can be shortened by this approach. Conversely, this approach works efficiently when the ratio of the transfer time in the total time is low. Therefore, it is more effective for image-processing functions requiring heavy calculation load such as pattern recognition than for other simpler functions. As the communication speed becomes faster with more advanced network environments in the future, our proposed approach can be used with advantage in many situations because the ratio of the transfer time will become progressively smaller. In the case of handling many tasks to perform image analysis, this distributed data processing approach has the potential of scalability by increasing the number of Web application services. In order to make the most of the features of this approach, we should consider how optimally the Agent System distributes all tasks to Web servers depending on the situation. It is important to divide all tasks to Web servers evenly based on the processing capacity of each Web server, the communication speed to each Web server, the data size, the types of data processing, and so on. For example, it is better to use Web servers with high processing capacity when the data processing requires a large amount of calculation, and it is better to use the Web servers under high-speed communication when the size of data is large. In order to maintain the performance of this distributed approach when processing a large amount of data, it is also important to consider the network and management load of the Agent System. By using the developed Multi-Agent System [21] in which many Agent Systems manage their tasks in a coordinated manner, this objective can be achieved. In the experiment of the event detection, we demonstrated that the proposed system could realize the requested operations adequately using the rule-base function and various Web application services in combination that perform image analysis using background differentiation techniques. This time, we simply established a few IF–THEN rules and used a simple image analysis algorithm. By preparing more sensitive IF–THEN rules and detailed operation steps, this system can work more intelligently and flexibly according to the situation. Our proposed system can perform various image analyses simply by preparing a suitable Web application service, so it is also easy to equip it with advanced image analysis algorithms for various applications. In some field experiments, we not only examined the possibility and the stability of this system but also tried to construct some applications to provide useful information to
Fig. 10. Example of collecting additional image data after detecting events of farm operations. When the Agent System detects an event of plowing operation according to the described rule base, it executes another monitoring protocol to obtain high-frequency image data with a multi-thread.
572
T. Fukatsu et al. / Computer Standards & Interfaces 33 (2011) 565–573
Fig. 11. Example of archived data displayed on viewer Web page. By clicking a target icon of a Field Server on the main Web page (left figure), we can see a Web page (right figure) showing selected image data based on the results of image analysis and the sensor data.
farmers by using the function of the system. This system facilitates the introduction of some applications that have been useful in the laboratory stage, such as maturity evaluation and growth diagnosis, [22,23] to the field stage because it enables us to construct a field monitoring system without great effort and to incorporate these applications simply by preparing the contents on the Web. Thus, it is expected to help provide some useful support systems for practical use. If the Web-based control of equipment such as irrigation machines and environmental control units can be prepared, this system would also be able to control them automatically based on the monitoring results. This proposed system has the possibility to be applied to various agricultural situations. The proposed system has the potential to perform advanced monitoring with Web-based sensor nodes and to be utilized for various purposes. By using the system's functionality, collected data can be provided in a consistent format. Therefore, it can be used as middleware that enables users to treat various sensor nodes without regard to the differences and enables applications to avoid data format problems. By preparing Web interfaces in which users can intelligibly control some objects on the Web such as sensor nodes, this system can also provide a wrapper function to perform a series of actions and provide desired results automatically according to users' request on the interface. In the same manner, it helps non-expert users to control sensor nodes without complicated steps. This system can provide users who lack specialized technical experience useful information as a mashup that combines with many kinds of Web application services and database into a single representation and executes a series of requested services seamlessly. The more Web-based monitoring devices, Web application services, and the communication environment are developed, the more our proposed system is expected to be used as an effective tool. To improve the system, several tasks need to be accomplished, such as simplifying the operating interfaces and achieving effective management with intelligent algorithms. Such an improved system would have more potential to perform many applications in a wide variety of situations. References [1] J.M. Kahn, R.H. Katz, K.S.J. Pister, Next century challenges: mobile networking for “smart dust”, Proceedings of the 5th Annual ACM/IEEE International Conference on Mobile Computing and Networking, Seattle, WA, USA, August 1999, pp. 271–278. [2] I.F. Akyildiz, W. Su, Y. Sankarasubramaniam, E. Cayirci, Wireless sensor networks: a survey, Computer Networks 38 (2002) 393–422. [3] R.D. Tillett, Image analysis for agricultural processes: a review of potential opportunities, Journal of Agricultural Engineering Research 50 (1991) 247–258. [4] N. Shimoda, T. Kataoka, H. Okamoto, M. Terawaki, S. Hata, Automatic pest counting system using image processing technique (in Japanese with English
[5] [6]
[7]
[8]
[9]
[10] [11] [12]
[13]
[14]
[15]
[16]
[17] [18]
[19]
[20]
[21]
[22] [23]
summary), Journal of the Japanese Society of Agricultural Machinery 68 (2006) 59–64. T. Fukatsu, M. Hirafuji, Field monitoring using sensor-nodes with a Web server, Journal of Robotics and Mechatronics 17 (2005) 164–172. S. Ninomiya, K. Kunio, Internet remote camera system and its application to agriculture, in: G. Schiefer, R. Helbig, U. Rickert (Eds.), Perspectives of Modern Information and Communication Systems in Agriculture, Food Production and Environmental Control, University Bonn-ILB, Germany, 1999, pp. 75–82. Y. Motonaga, N. Iguchi, N. Ohtake, T. Kameoka, Image monitoring of paddy field using BIX image broker system, Proceedings of SICE Annual Conference 2007, Kagawa, Japan, September 2007, pp. 224–229. T. Fukatsu, M. Hirafuji, T. Kiura, Construction of support tools to assist an Agent System, Proceedings of IAALD-AFITA-WCCA 2008, Tokyo, Japan, August 2008, pp. 221–227. K. Toyama, J. Krumm, B. Brumitt, B. Meyers, Wallflower: principles and practice of background maintenance, Proceedings of Seventh International Conference on Computer Vision, Kerkyra, Greece, September 1999, pp. 255–261. R.J. Radke, Image change detection algorithms: a systematic survey, IEEE Trans. Image Process. 14 (2005) 294–307. C. Bauchi, T. Rath, Prototype of a vision based system for measurements of white fly infestation, ISHS Acta Horticulturae 691 (2005) 773–780. P. Boissard, V. Martin, S. Moisan, A cognitive vision approach to early pest detection in greenhouse crops, Computers and Electronics in Agriculture 62 (2008) 81–93. Z. Duan, Y. Dong, K. Gopalan, Push vs. pull: implications of protocol design on controlling unwanted traffic, Proceedings of Steps to Reducing Unwanted Traffic on the Internet Workshop, Cambridge, MA, USA, July 2005, pp. 25–30. M. Hauswirth, M. Jazayeri, A component and communication model for push systems, Proceedings of 7th European software engineering conference, Toulouse, France, September 1999, pp. 20–38. B. Pinkerton, Finding what people want: experiences with the Webcrawler, Proceedings of the 2nd International World Wide Web conference, Chicago, IL, USA, October 1994, pp. 821–829. L. Brownston, R. Farrell, E. Kant, N. Martin, Programming Expert Systems in OPS5: an Introduction to Rule-Based Programming, Addison-Weskey Longman Publishing Co., Inc., Boston, MA, USA, 1985. C. Sun, Fast optical flow using 3D shortest path techniques, Image and Vision Computing 20 (2002) 981–991. X. Yu, M. Laurenson, T. Kiura, S. Ninomiya, RIAS: an approach to provide internetaccessible image analysis service, Agricultural Information Research 16 (2007) 212–218. N. Iguchi, Y. Motonaga, F. Uchio, S. Ninomiya, T. Kameoka, Development of a gateway-P2P-node for enhancing the function of BIX image broker system (in Japanese with English summary), Agricultural Information Research 15 (2006) 281–292. A.M. Elgammal, D. David, L.S. Davis, Non-parametric model for background subtraction, in: D. Vernon (Ed.), Lecture Notes in Computer Science 1843, Springer, New York, NY, USA, 2000, pp. 751–767. T. Fukatsu, T. Kiura, T. Tanaka, M. Hirafuji, Hierarchical agent system for Webbased sensor network, Proceedings of 2007 International Symposium on Applications and the Internet Workshops, Hiroshima, Japan, January 2007, p. 77. K. Choi, G. Lee, Y.J. Han, J.M. Bun, Tomato maturity evaluation using color image analysis, Transactions of the ASABE 38 (1995) 171–176. K. Iwaya, H. Yamamoto, Diagnosis of optimal harvesting time of rice using digital imaging, Journal of Agricultural Meteorology 60 (2005) 981–984.
T. Fukatsu et al. / Computer Standards & Interfaces 33 (2011) 565–573 Tokihiro Fukatsu Name: Affiliation: National Agricultural Research Center, National Agriculture and Food Research Organization Address: 3-1-1 Kannondai, Tsukuba, Ibaraki, 305-8666 Japan Brief biographical history: 1998 Received M. Eng from Tokyo Institute of Technology 1998 — FUJITSU LIMITED 2001 — National Agricultural Research Center 2007 Received Ph. D (Ag) from Tsukuba University
Name: Takuji Kiura Affiliation: National Agricultural Research Center, National Agriculture and Food Research Organization Address: 3-1-1 Kannondai, Tsukuba, Ibaraki, 305-8666 Japan Brief biographical history: 1989 — Researcher, Tropical Agriculture Research Center, Japan 1995 — Researcher, National Agriculture Research Center, Japan
573
Name: Masayuki Hirafuji Affiliation: National Agricultural Research Center, National Agriculture and Food Research Organization Address: 3-1-1 Kannondai, Tsukuba, Ibaraki, 305-8666 Japan Brief biographical history: 1983 Received M. Ag from University of Tokyo 1983 — National Agriculture Research Center 1994 — Research coordinator, MAFF (Ministry of Agriculture, Forestry and Fisheries) 1995 Received Ph D.(Ag) from University of Tokyo 1996 — National Agricultural Research Center