- Email: [email protected]
Development of Automatic Badminton Playing Robot with Distance Image Sensor
10th IFAC Symposium on Intelligent Autonomous Vehicles 10th IFACPoland, Symposium on Intelligent Autonomous Vehicles Gdansk, July 3-5, 2019 10th IFACPoland, Symposium on Intelligent Autonomous Vehicles Gdansk, July 3-5, 2019 Available online at www.sciencedirect.com 10th IFAC Symposium on Intelligent Autonomous Vehicles Gdansk, July 3-5, 2019 10th IFACPoland, Symposium on Intelligent Autonomous Vehicles Gdansk, Poland, July 3-5, 2019 Gdansk, Poland, July 3-5, 2019
ScienceDirect
IFAC PapersOnLine 52-8 (2019) 67–72
Development of Automatic Badminton Playing Robot with Distance Image Sensor Development of Automatic Badminton Playing Robot with Distance Image Sensor Development of Automatic Badminton Playing Robot with Distance Image Sensor Development of Automatic Badminton Robot with Distance Image Sensor Naoki Mizuno*, Takuya Makishima*, Playing Kenta Tsuge*, Sho Kondo*, Tomoaki Nonome*, Development of Automatic Badminton Playing Robot with Distance Image Sensor Naoki Mizuno*, Takuya Makishima*, Kenta Tsuge*, Sho Kondo*, Tomoaki Nonome*, Hideaki Kurebayashi*, Shota Otake*, Daichi Shibata* and Satoko Yamakawa** Naoki Mizuno*, Takuya Makishima*, Kenta Tsuge*, Sho Kondo*, Tomoaki Nonome*, Hideaki Kurebayashi*, Shota Otake*, Shibata* and Satoko Yamakawa** Daichi Naoki Mizuno*, Takuya Makishima*, Kenta Tsuge*, Sho Kondo*, Tomoaki Nonome*, Naoki Mizuno*, Takuya Makishima*, Kenta Tsuge*, Sho Kondo*, Tomoaki Nonome*, Hideaki Kurebayashi*, Shota Otake*, Daichi Shibata* and Satoko Yamakawa** Daichi Shibata* and Satoko Yamakawa** Hideaki Shota Otake*, Otake*, *Graduate School of Engineering, Nagoya Institute of Technology, Nagoya, 466-8555 Hideaki Kurebayashi*, Kurebayashi*, Shota Shibata* and Satoko Yamakawa** Daichi Institute of Technology, Nagoya, 466-8555 *GraduateJAPAN School (Tel: of Engineering, Nagoya e-mail: nmizuno@ nitech.ac.jp). +81-52-735-5339; *Graduate School of Engineering, Nagoya Institute of Technology, Nagoya, 466-8555 (Tel: +81-52-735-5339; e-mail: nmizuno@ nitech.ac.jp). *Graduate School of Nagoya Institute of Technology, Nagoya, 466-8555 **Faculty of Science andJAPAN Engineering, Toyo University, JAPAN (e-mail:[email protected]) *Graduate School of Engineering, Engineering, NagoyaKawagoe, Institutenmizuno@ of350-8585 Technology, Nagoya, 466-8555 JAPAN (Tel: +81-52-735-5339; e-mail: nitech.ac.jp). **Faculty of Science andJAPAN Engineering, Toyo University, Kawagoe, 350-8585nitech.ac.jp). JAPAN (e-mail:[email protected]) (Tel: +81-52-735-5339; e-mail: nmizuno@ (Tel: +81-52-735-5339; e-mail: nmizuno@ **Faculty of Science andJAPAN Engineering, Toyo University, Kawagoe, 350-8585nitech.ac.jp). JAPAN (e-mail:[email protected]) **Faculty **Faculty of of Science Science and and Engineering, Engineering, Toyo Toyo University, University, Kawagoe, Kawagoe, 350-8585 350-8585 JAPAN JAPAN (e-mail:[email protected]) (e-mail:[email protected])
Abstract: We developed a full automatic badminton playing robot "MRO Brothers". The developed Abstract: We developed a full automatic badminton as playing "MRO player. Brothers". Thesystem, developed robot can receive and serve in the same environment humanrobot badminton In this the Abstract: We developed a full automatic badminton as playing robot "MRO player. Brothers". Thesystem, developed robot can receive and serve in the same environment human badminton In this the Abstract: We developed a full automatic badminton playing robot "MRO Brothers". The developed shuttle is detected by Kinect sensor and its orbit is estimated based on the distance image of the shuttle. Abstract: We developed a full automatic badminton as playing robot "MRO player. Brothers". Thesystem, developed robot can receive and serve in the same human badminton In this the shuttle is detected bycan Kinect sensor and itsenvironment orbit estimated based on will the distance image of system, the robot can receive serve in environment as human badminton player. In the Moreover, the robotand automatically move to theis spot where a shuttle fall and hit back by shuttle. normal robot can receive and serve in the the same same environment as human badminton player. Init this this system, the shuttle is detected by Kinect sensor and its orbit is estimated based on the distance image of the shuttle. Moreover, the robotbycan automatically move to theis spot wherebased a shuttle will fall and hit it back by shuttle. normal shuttle is detected Kinect sensor and its orbit estimated on the distance image of the badminton racket. shuttle is detected bycan Kinect sensor and its orbit is spot estimated based on will the distance image of the shuttle. Moreover, the robot automatically move to the where a shuttle fall and hit it back by normal badminton the racket. Moreover, robot move to the where aa shuttle will fall and hit it back by Moreover, the robot can can automatically automatically move to the spot spot where shuttle will fallLtd. and All hitKalman it backreserved. by normal normal badminton racket. © 2019, IFAC (International Federation Automatic Control) Hosting by Elsevier rights Keywords: Autonomous robot, mobile of robot, visual sensing, badminton playing robot, filter. badminton racket. badminton Keywords: racket. Autonomous robot, mobile robot, visual sensing, badminton playing robot, Kalman filter. Keywords: Autonomous robot, robot, mobile robot, robot, visual sensing, sensing, badminton playing playing robot, Kalman Kalman filter. filter. Keywords: Keywords: Autonomous Autonomous robot, mobile mobile robot, visual visual sensing, badminton badminton playing robot, robot, of Kalman filter. robots Table.1 Specifications Badminton 1. INTRODUCTION Table.1 Specifications of Badminton robots Dimension [mm] W675×D890×H900 1. INTRODUCTION Table.1 Specifications of Badminton robots Dimension [mm] W675×D890×H900 Weight [kg] 20robots Table.1 Specifications of Badminton 1. INTRODUCTION Every year, the "NHK Student Robocon," one of a Japanese Table.1 Specifications of Badminton robots 1. INTRODUCTION Dimension [mm] W675×D890×H900 20 1. INTRODUCTION NumberDimension ofWeight [kg] DC motor 7 [mm] W675×D890×H900 Every year, theis"NHK oneleading of a Japanese Robot contest held Student to selectRobocon," a Japanese student Dimension [mm] W675×D890×H900 Weight [kg] 20 Every year, the "NHK Student Robocon," one of a Japanese Number of DC motor 7 actuatorsWeight Servo motor 4 [kg] 20 Robot for contest is (Asia-Pacific held Student to selectRobocon," a Japanese leading student Every year, the one of [kg] 20 Number ofWeight Servo DC motor 74 team ABU Broadcasting Union) Robot actuators Every year, theis"NHK "NHK oneleading of aa Japanese Japanese motor Robot contest held Student to selectRobocon," a Japanese student [unit] of Air-cylinder Number DC motor 717 team for ABU (Asia-Pacific Broadcasting Union) Robot Number of DC motor Robot contest is held to select a Japanese leading student contest. actuators Servo motor 41 [unit] Air-cylinder Robot contest held to selectBroadcasting a Japanese leading team for ABUis (Asia-Pacific Union) student Robot actuators Servo motor 44 contest. [unit] actuators Servo motor team for ABU (Asia-Pacific Broadcasting Union) Robot Air-cylinder 1 team for ABU (Asia-Pacific Broadcasting Union) Robot [unit] Air-cylinder 11 contest. In this ABU Robot contest, the theme of the competition is [unit] Air-cylinder contest. contest. In this ABU Robot contest, the theme of the competition is determined by the host country. In this ABU contest, the theme of the competition is determined byRobot the host country. In this ABU Robot contest, the theme of the competition is In this ABU Robot contest, the theme of the competition is determined by the host country. In 2015, thebyABU Robot contest was held in Indonesia and determined the host country. determined by the host country. In 2015, the ABU Robot contest was held in Indonesia and the theme of competition was Badminton. In ABU Robot contest was held in Indonesia and the2015, themethe of competition was Badminton. In 2015, the ABU Robot contest was held in Indonesia and In 2015, the ABU Robot contest was held in Indonesia and the theme of competition was Badminton. To cope with this competition theme, we have developed the theme of competition was Badminton. the theme of competition was Badminton. To copeBrothers," with thisa competition theme, we have badminton developed "MRO pair of robot that performs To copeBrothers," with thisa competition theme, we have badminton developed "MRO pair of robot that performs To cope with this competition theme, we have developed play fully automatically. To copeBrothers," with thisa competition theme, we have badminton developed "MRO pair of robot that performs Fig.1 Overview of the Badminton Playing Robot play fully automatically. "MRO Brothers," a pair of robot that performs badminton "MRO Brothers," a pair of robot that performs badminton play fully automatically. Fig.1 Overview of the Badminton Playing Robot The developed all in one robots recognize the shuttle using play fully automatically. Fig.1 Overview of the Badminton Robot In designing the badminton robot, the Playing following functions play fully automatically. The developed all in one robots recognize the shuttle using Fig.1 Overview of the Badminton Playing Robot Kinect sensor, predict trajectory of shuttle, automatically In designing the badminton robot, the following functions Fig.1 Overview of the Badminton Playing Robot The developed all in one robots recognize the shuttle using were aimed to be realized. Kinect sensor, predict trajectory of shuttle, automatically The all one robots recognize the shuttle using movedeveloped to a point ofin and perform a receiving motion, designing badminton robot, the following functions were aimed tothe be realized. The developed all in fall, onetrajectory robots recognize the automatically shuttle using In In designing the badminton robot, the following functions Kinect sensor, predict of shuttle, move tosensor, a pointpredict of fall,trajectory and perform a receiving designing badminton robot, the following functions were aimed tothe be realized. Kinect automatically thereby enabling badmintons to of be shuttle, played in themotion, same In 1) The shuttle during flight is recognized, and the vehicle unit Kinect sensor, predict trajectory of shuttle, automatically were aimed to be realized. move to aa point of fall, and perform aa receiving motion, thereby enabling badmintons to be played in the same 1) The shuttle during flight is recognized, and the vehicle unit were aimed to be realized. move to point of fall, and perform receiving motion, environments as human players. is accelerated and decelerated by the wheel mechanism to move to a point of fall, and perform a receiving motion, thereby enabling badmintons be played in the same 1) The shuttle during flight is recognized, and the vehicle unit environments as human players. to is accelerated and decelerated by the wheel mechanism to thereby enabling badmintons to be played in the same 1) The shuttle during flight is recognized, and the vehicle unit the shuttle vicinityduring of theflight predicted fall position, thereby moving thereby enabling badmintons to be played in the same 1) The is recognized, and the vehicle unit environments as human players. is by the wheel mechanism to In recent years, there are several reports and patents about theaccelerated vicinity ofand the decelerated predicted fall position, thereby moving environments as human players. is accelerated and decelerated by the wheel mechanism to at high speed. environments as human players. is accelerated and decelerated byposition, the wheelthereby mechanism to In recent years, thereplaying are several and patents about the vicinity of the predicted fall moving automatic badminton robots.reports at high speed. the vicinity of the predicted fall position, thereby moving 2) In the vicinity of the predicted fall position, fine In recent years, there are several reports and patents about the vicinity of the predicted fall position, thereby moving at high speed. automatic badminton playing robots.reports and patents about 2) In the vicinity of the predicted fall position, fine In recent there are several at high speed. In recent years, years, thereplaying are reports and patents about 2) adjustment of the ofposition of the shuttle hitting unitfine is automatic badminton robots. at However, most reports are several the studies on part of badminton In high the speed. vicinity the predicted fall position, automatic badminton playing robots. adjustment of the the of position of themounted shuttle hitting unitfine is 2) In the vicinity the predicted fall position, automatic badminton playing robots. performed by XY stage unit on the vehicle However, most reports are the studies on part of badminton In the vicinity ofposition the predicted fall position, fine play (optimal control of servicing (Liu et al. 2013), shuttle 2) adjustment of the of the shuttle hitting unit is performed by the XY stage unit mounted on the vehicle However, most reports are the studies on part of badminton adjustment of the position of the shuttle hitting unit unit. play control of servicing (Liu et al.(2011), 2013), shuttle However, most reports the studies on of adjustment by of the the XY position of themounted shuttle on hitting unit is is hitting(optimal back mechanism (China Patent shuttle performed stage unit the vehicle However, mostcontrol reportsofare are the studies onet part part of badminton badminton unit. play (optimal servicing (Liu al. 2013), shuttle performed by the XY stage unit mounted on the vehicle 3) A shuttle hitting unit mounted on the XY stage unit hits hitting back mechanism (China Patent (2011), shuttle play (optimal control of servicing (Liu et al. 2013), performed by the XY stage unit mounted on the vehicle recognition (Raina et al. 2015), etc.). unit. play (optimal control of servicing (Liu et al.(2011), 2013), shuttle shuttle 3) Aback shuttle hitting inunit mounted on the XY stage unit hits hitting back mechanism (China unit. the shuttle a timely manner. recognition (Raina et al. 2015), etc.). Patent hitting back mechanism (China Patent (2011), shuttle unit. 3) A shuttle hitting unit mounted on the XY stage unit hits hitting back mechanism (China Patent (2011), shuttle back the shuttle in a timely manner. recognition (Raina et al. 2015), etc.). 3) A shuttle hitting unit mounted on the The robothitting servicing the gameon without helpunit of hits the There are very few reports ofetc.). integrated robots that can 4) recognition (Raina et al. 2015), 3) A shuttle unit mounted the XY XYthestage stage unit hits back the shuttle in a timely manner. recognition (Raina et al. 2015), etc.). 4) The robot servicing the game without the help of the There are very few reports of integrated robots that can back the shuttle in a timely manner. human operator. operate independently. back the shuttle in a timely manner. The robot servicing the game without the help of the There very few reports of integrated robots that can 4) human operator. operateare independently. robot servicing There are very 4) The The robot servicing the the game game without without the the help help of of the the There very few few reports reports of of integrated integrated robots robots that that can can 4) human operator. operateare independently. 2.1 human The vehicle unit operator. operate independently. 2. HARDWARE CONFIGURATION human operator. operate independently. 2.1 The vehicle unit 2. HARDWARE CONFIGURATION 2.1 The vehicle unit 2. HARDWARE CONFIGURATION 2.1 The vehicle 2. The specificationsCONFIGURATION of the developed badminton playing robot In 2.1 the Thebadminton vehicle unit unitgame, the falling position of the shuttle 2. HARDWARE HARDWARE CONFIGURATION In theonbadminton The specifications of 1. the developed badminton playing robot move wide area.game, the falling position of the shuttle are shown in Table The overview of one the robot is In the the falling position of the shuttle The specifications of 1. the developed badminton robot move onbadminton wide area.game, the badminton game, the falling of shuttle are shown in Table The overview of oneplaying the robot is In The specifications of the badminton playing robot shown in Fig.1. In thefalling badminton game, theestimated falling position position of the the shuttle move on wide area. The specifications of 1. the developed developed badminton playing robot If the position will be in the coat, the shuttle are shown in Table The overview of one the robot is move on wide area. shown in Fig.1. are shown in Table 1. The overview of one the robot is move on wide area. the falling position willopposite be estimated inthe thenet. coat, the shuttle are shown in Table 1. The overview of one the robot is If should be hit back to the side of shown in Fig.1. If the falling willopposite be estimated thenet. coat, the shuttle shown should be hit position back to the side ofin shown in in Fig.1. Fig.1. If the position will be inthe the coat, If the falling falling willopposite be estimated estimated thenet. coat, the the shuttle shuttle should be hit position back to the side ofinthe should be hit back to the opposite side of the net. Copyright © 2019 IFAC 2405-8963 © 2019, IFAC (International Federation of Automatic Control) Hosting by Elsevier All opposite rights reserved. should be hit backLtd. to the side of the net. Copyright 2019 responsibility IFAC Peer review©under of International Federation of Automatic Control. Copyright © 2019 IFAC 10.1016/j.ifacol.2019.08.050 Copyright © 2019 IFAC Copyright © 2019 IFAC
2019 IFAC IAV 68 Gdansk, Poland, July 3-5, 2019
Naoki Mizuno et al. / IFAC PapersOnLine 52-8 (2019) 67–72
The shuttle is hit by rotating the plate cam (Fig. 5) via the timing belt with a DC motor to stretch the spring and release its stored energy. The cam shape is a logarithmic spiral shape, and the spring is stretched as the cam rotates, and at the moment when the cam exceeds one rotation, the energy is released and the shuttle is hit back.
To achieve this function, the badminton playing robot should move quickly to the position near the estimated falling position of the shuttle. For the quick movement of the robot, "Swerve Drive" unit was adopted, which can move in all directions and can move at high speed. In this system, we have develop the original steering-driving units as shown in Fig. 2 and arranged at the four corners of the robot pedestal.
2.4 Servicing auxiliary unit In order to perform badminton fully automatically, it is necessary that the robot has not only the passive shuttle hit back function but also the active shuttle servicing function. For this purpose, a shuttle servicing assisting unit using an air cylinder was mounted (Fig. 6 and Fig. 7).
Fig. 2 Swerve Drive Unit
Fig. 3 XY Staging Unit
In this drive unit, since a polyurethane wheel having a high friction coefficient is used as the wheel, the slip of the wheel hardly occurs even at the time of rapid acceleration, and stable running is possible.
Fig.6 Shuttle Servicing Auxiliary Unit
For the steering of the wheels, Dynamixel's high-speed servomotor RX-24F is used. The direct driven design of the "Swerve Drive Unit" makes the unit slim and the backlash of the steering minimum. The driving of the wheels is performed by transmitting the power of the DC motor by a timing belt, and the rotational speed of the tire is measured by the rotary encoder to perform the feedback speed control.
Since it is necessary to set the servicing auxiliary unit outside the moving range of the XY stage unit, the servicing auxiliary unit is designed that the arm of the shuttle grasper can be extended to the front of the robot by the air cylinder. By using the air cylinder, the arm can be contracted at high speed so that the racquet does not collide during servicing.
2.2 XY stage unit
When the air cylinder contracts from the extended state, the shuttle grasper opens and the shuttle drops as shown in Fig. 7. Servicing is done by hitting the shuttle with the racket at that moment.
After the robot was moved by the vehicle unit to the vicinity where the shuttle was predicted to fall, the position of the shuttle hitting unit mounted with an XY stage unit was finely adjusted (Fig. 3). A shuttle hitting unit, which will be described later, is mounted on the unit, and the shuttle hitting unit can be moved by driving a timing belt with a DC motor. The position is measured by a rotary encoder and a potentiometer, and feedback control is performed. This unit has a stroke length of about 500 mm in both the X-axis and the Y-axis, and makes it possible to hit back a shuttle falling within a range of 500 mm×500 mm in front of the robot.
(a) Closed state (b) Open Fig. 7 Open and close stages of the shuttle grasper 3. CONTROL CIRCUIT CONFIGURATION 3.1 Overall circuit configuration
2.3 Shuttle hitting unit
Fig. 8 shows the overall configuration of the control circuit. There is a main circuit board for controlling the entire movement of robots, and each motor is controlled by transmitting commands to motor drivers of respective mechanisms using CANs (Controller Area Network).
The shuttle hitting unit plays a role of returning the dropped shuttle to the opposite side of the central net (Fig. 4).
The values of the various sensors are also acquired using CAN.
Fig. 4 Shuttle hitting unit
Fig. 5 Periphery of the cam 68
2019 IFAC IAV Gdansk, Poland, July 3-5, 2019
Naoki Mizuno et al. / IFAC PapersOnLine 52-8 (2019) 67–72
69
In this controller board, by converting the UART signal for writing and debugging programs to microcomputers into Bluetooth signal, all developing procedure can be executed without connecting cables to the robot.
Main CB Self-position estimation Ground encoder Gyro sensor
In addition, the main control board has a microSD card slot, which can be read/written from/to the microcomputer, and the robot's operation can be changed without rewriting programs by holding parameters of the robot in the microSD card.
Mobile mechanism Motor driver Rotation speed encoder Steering servomotor
Moreover, the main control board includes a user interface such as a tact switch, a rotary encoder, a DIP switch, and a liquid crystal display, as well as a USB interface corresponding to the DualShock3 and a Bluetooth other than writing, and the robots can be controlled using the game controller.
Main PCB control XY-Stage Motor driver Rotary encoder
3.4 Motor driver A gate driver IC (IRS2184S) is used to drive the FETs, all of which are composed of H-bridges made of IXTP180N10T made by Nch FET (IXYS, and the FETs operate by PWM signals from a control microcomputer that receives commands via CANs.
Shuttle grasper control Pneumatic electro-magnetic valve Cam rotation control
A magnetic digital isolator (ADUM1400ARW) is connected between the control microcomputer and the gate driver, so that noises from the motor do not affect the control microcomputer.
Motor driver Rotary encoder Estimation of shuttle orbit
In the H-bridge section, the common mode choke coil, the X/Y capacitor, and the ferrite bead are arranged at appropriate positions to prevent noise from entering the weak electric system from the driver circuit.
Box type computer-Kinect v2 Fig. 8 Overall Circuit Configuration Diagram 3.2 CAN
3.5 Sensors
CAN using Ethernet Cat5 cables is used for communication between control circuit board.
3.5.1 Rotary encoder Incremental rotary encoders (E6A2-C manufactured by Omron, etc.) are used to detect the position and velocity of robotic mechanisms. The standard cable was replaced with a shielded cable because the standard cables had low noise tolerance and caused malfunction.
Since CAN is characterized as the differential communication, it is resistant to external noise generated by motors etc. The reason of using Ethernet cables is also to prevent noise.
3.5.2 Gyro-sensor This robot is equipped with a uniaxial gyro-sensor (R1350N manufactured by MicroInfinity) for self-position estimation.
The twisted pair cables in the Ethernet cable increases the resistance to external noise. In addition, since the CAN is a bus type network, in a robot on which many independent circuit units are mounted, the amount of wiring can be saved, the weight of cables can be reduced, and at the same time, operation maintenance cost can be reduced.
Inside the R1350N, the Kalman filtering of signals is performed, so that highly accurate angular data can be obtained. 3.5.3 Kinect v2
The communication rate was 500 kbps, which is commonly used in in-vehicle systems.
Table 2 Specification of Kinect v2 Depth retrieval method Time of Flight Depth resolution 512px × 424px Depth frame rate 30fps Depth awareness 0.5m ~ 8.0m Depth angle of view 70 degrees horizontal and 60 degrees vertical
3.3 Main control board The Renesas Electronics RX631 is used as a microcontroller for the main control board (Main CB).
69
2019 IFAC IAV 70 Gdansk, Poland, July 3-5, 2019
Naoki Mizuno et al. / IFAC PapersOnLine 52-8 (2019) 67–72
By utilizing such a property, the shuttle is recognized in real time.
To detect the flying shuttle, the Kinect v2 made by Microsoft, is used. Table 2 shows the specifications of Kinect v2.
4.1.1 Labeling processing
3.6 Orbit prediction calculation PC
In order to detect shuttle candidates from the image, the distance image is subjected to a laveling process.
To acquire range data from Kinect v2 and perform trajectory estimation calculation, the “Brix Pro” PC is used.
In the labelling process, the same label is assigned when the difference in brightness between the target pixel and the neighboring 8 pixels is less than the threshold value.
Table 3 shows the specifications of “Brix Pro”. Table 3.2 Brix Pro specifications
CPU Memory Storage
Fig. 10 shows the result of color coding for each label applied to Fig. 9.
Core i7-4770R (Haswell/4 Core/8 Thread/3.2GHz/6MB Cash) DDR3L-1600 4GB × 2 TS128GSSD370
The PC is connected to a USB-CAN convert unit via a USB isolator. The USB-CAN convert unit includes a USB-toUART converter chip and a microcomputer for CAN communication. The PC transmits data by transmitting binary data to the COM port.
Fig. 10 Results of the labeling process By the laveling process, it can be confirmed that the region is divided into a shuttle region, a net region, and a region without anything.
4. SOFTWARE CONFIGURATION The badminton playing robot needs to perform the following operations in real time.
4.1.2 Extraction of shuttle images
1) Recognize the shuttle in flight. 2) Predicting the fall position of the shuttle (Shishido H. et al. 2012). 3) The robot is moved to the predicted fall position, and the shuttle is hit back with good timing using the XY stage unit and the hitting unit.
This section describes how to identify whether a region is a shuttle. The size of the region is used for identification. The size of the region S is defined in Eq. (1) using the number of pixels p in the region, the average value of the distance image of the region d, and the focal length of the sensor f.
4.1 Shuttle recognition (1) Kinect v2, a visual sensor for gaming machines, was used to recognize shuttle in flight.
If the size of the region is within the threshold, the region is considered as a shuttle. However, since the distance image contains a lot of noise, the noise may be erroneously recognized as a shuttle. To avoid the recognition error due to noise, the past several frames are referenced and whether the recognition result of the current frame is appropriate as a shuttle trajectory is considered.
When a shuttle in flight is shot using this sensor, a distance image is obtained as shown in Fig. 9.
To extract the trajectory of the shuttle, the RANSAC algorithms were used. The shuttle trajectory was approximated by a quadratic function with respect to time, and the one with the smallest error between the approximation function and the measured data was defined as the shuttle trajectory.
Fig. 9 Example of distance image In the figure, objects at distances far from the viewpoint are displayed in black, and objects at close distances are displayed in white. In this case, the shuttle image is indicated by a red circle.
4.2 Prediction of shuttle falling position An extended Kalman filter is used to predict the shuttle falling position. The algorithm is an extension of a Kalman filter, which is a technique for obtaining an optimum estimation value from data including noise, so as to be able to cope with a nonlinear system.
From this distance image, it can be seen that the difference in luminance between the shuttle and the periphery of the shuttle is large, that is, a difference in depth occurs. 70
2019 IFAC IAV Gdansk, Poland, July 3-5, 2019
Naoki Mizuno et al. / IFAC PapersOnLine 52-8 (2019) 67–72
4.2.1 Extended Kalman Filter (Shuku T. 2014)
71
In the above algorithms, the Predict step estimates the status of the system, and the Correct step modifies the system estimate based on the observations. This processing enables state prediction of the nonlinear transition system.
In this case, it is assumed that a nonlinear system including noise and an observation signal are expressed by the following equation with time as.t
4.2.3 Extended Kalman Filter Trajectory Prediction
(2) (3)
The motion state of the shuttle is defined by the following three-dimensional position, velocity and acceleration.
Where,
Where, the z-axis represents the vertical position, and the upword is positive. (5) In addition, it is assumed that are normal distribution white noises having a mean value of 0 and the variances , respectively. First, the function is linearized around the states to obtain the following Jacobian matrix :
(6)
(7)
(4)
The transition of the state is a parabolic motion of the object in which the air resistance is proportional to the square of the velocity as follows.
In addition, defining : a priori and a posteriori estimates of the state : covariance matrix of a priori and a posteriori estimate error : Kalman gain
(8) (9) (10)
The states of the systems can be estimated in two steps of Correct and Predict as follows. Where the covariance matrix is known. Give an initial value While(true){ If (observe ) Correct( )//Correct steps Predict()//Predict steps
(11)
Where k is the air resistance coefficient and gravitational acceleration coefficient. In this case, the Jacobian matrix is as follows.
} Correct( ){
} Predict(){
is the
(12)
Where, dt is the infinitesimal time. One Prediction step and one Correction step were carried out based on the observed position data of the shuttles, and after that, only the Predict step was repeated until the predicted
}
71
2019 IFAC IAV 72 Gdansk, Poland, July 3-5, 2019
Naoki Mizuno et al. / IFAC PapersOnLine 52-8 (2019) 67–72
height z of the shuttle match the position of the racquet, and the position at that time was used as the predicted fall point x,y. 4.3 Robot movement and Shuttle hit back
Fig. 12 MRO Brothers Fig. 11 Schematic diagram of movement mechanism Fig. 11 shows the schematic diagram of movement mechanism of the robot.
Highly accurate recognition of shuttles and trajectory prediction were realized by labeling and extended Kalman filtering of distance images obtained by Kinect.
Each steering unit is oriented at the center of rotation and needs to provide a speed proportional to the distance from the center of rotation.
These functions have made it possible to hit back shuttle with high success rate. This robot participated in the Student Robocon 2015 and received the Idea Award.
Assuming that the target speed V at the center of the robot, the target angular velocity , and the direction of travel of and velocity of the steering unit the target, the angle located at the coordinates of the pivot center(X,Y) and the coordinates satisfy the following equations (13) to (16).
Acknowledgement We would like to express our heartfelt gratitude to many people, including the Nagoya Institute of Technology's Tomoekai, for your great support for this research. We would also like to express our particular gratitude to Tatsuya Murao for his great cooperation.
(13)
REFERENCES
(14)
ABU Robocon: http://www.abu.org.my /ABU_Robocon-@About_ABU_Robocon.aspx
(15)
China Pataent (2011): googleapis. com/ CN102363073B.pdf
(16)
https: //patentimages. storage. 80/31/e7 /932bd228ff677f/
By applying the above equation to the four steering units and performing the control, it is possible to move to the predicted fall position of the shuttle at high speed.
NHK Student Robocon 2015 Rulebook. http://www.officialrobocon.com/ jp/ daigaku/ daigaku2015/ entry/ 2015_RULE_ROBOMINTON_0910ver.pdf
The speed control is realized by the PID control and the feed forward control using the target values of the acceleration of the robot, the angular acceleration, the speed of each wheel, and the acceleration.
Liu M. et al. (2013): Model-free and model-based timeoptimal control of a badminton robot, Proceedings of 9th Asian Control Conference (ASCC). Rina A. et al. (2015): Shuttlecock tracking and trajectory estimation using Microsoft Kinect sensor, International Journal of Application or Innovation in Engineering & Management (IJAIEM), Volume 4, Issue 10, pp.80-87.
When the time required for the shuttle to reach the ground becomes equal to or less than the threshold value, the XY stage is moved and the shuttle is hit in accordance with the predicted fall position.
Shishido H. et al. (2012): Shuttle trajectory estimation from Badminton images using particle filter and Kalman filter, Workshop on Practical Application of Motion Picture Processing DIA2012 in Japanese.
5. CONCLUSIONS Two fully automated Badminton playing robots were developed (Fig. 12) that employs "Swerve Drive" in the steering-driving unit and enables high-speed movement in all directions.
Shuku T. (2014): Extended Kalman Filter, Website of the INVERSE PROBLEM SUBCOMMITTEE, Japan Society of Civil Engineers 'Applied Dynamics Committee.
The movie of our robots playing badminton each other was uploaded to YouTube.
https://www.youtube.com/watch?v=4FyqSeGt6NQ 72
ScienceDirect
IFAC PapersOnLine 52-8 (2019) 67–72
Development of Automatic Badminton Playing Robot with Distance Image Sensor Development of Automatic Badminton Playing Robot with Distance Image Sensor Development of Automatic Badminton Playing Robot with Distance Image Sensor Development of Automatic Badminton Robot with Distance Image Sensor Naoki Mizuno*, Takuya Makishima*, Playing Kenta Tsuge*, Sho Kondo*, Tomoaki Nonome*, Development of Automatic Badminton Playing Robot with Distance Image Sensor Naoki Mizuno*, Takuya Makishima*, Kenta Tsuge*, Sho Kondo*, Tomoaki Nonome*, Hideaki Kurebayashi*, Shota Otake*, Daichi Shibata* and Satoko Yamakawa** Naoki Mizuno*, Takuya Makishima*, Kenta Tsuge*, Sho Kondo*, Tomoaki Nonome*, Hideaki Kurebayashi*, Shota Otake*, Shibata* and Satoko Yamakawa** Daichi Naoki Mizuno*, Takuya Makishima*, Kenta Tsuge*, Sho Kondo*, Tomoaki Nonome*, Naoki Mizuno*, Takuya Makishima*, Kenta Tsuge*, Sho Kondo*, Tomoaki Nonome*, Hideaki Kurebayashi*, Shota Otake*, Daichi Shibata* and Satoko Yamakawa** Daichi Shibata* and Satoko Yamakawa** Hideaki Shota Otake*, Otake*, *Graduate School of Engineering, Nagoya Institute of Technology, Nagoya, 466-8555 Hideaki Kurebayashi*, Kurebayashi*, Shota Shibata* and Satoko Yamakawa** Daichi Institute of Technology, Nagoya, 466-8555 *GraduateJAPAN School (Tel: of Engineering, Nagoya e-mail: nmizuno@ nitech.ac.jp). +81-52-735-5339; *Graduate School of Engineering, Nagoya Institute of Technology, Nagoya, 466-8555 (Tel: +81-52-735-5339; e-mail: nmizuno@ nitech.ac.jp). *Graduate School of Nagoya Institute of Technology, Nagoya, 466-8555 **Faculty of Science andJAPAN Engineering, Toyo University, JAPAN (e-mail:[email protected]) *Graduate School of Engineering, Engineering, NagoyaKawagoe, Institutenmizuno@ of350-8585 Technology, Nagoya, 466-8555 JAPAN (Tel: +81-52-735-5339; e-mail: nitech.ac.jp). **Faculty of Science andJAPAN Engineering, Toyo University, Kawagoe, 350-8585nitech.ac.jp). JAPAN (e-mail:[email protected]) (Tel: +81-52-735-5339; e-mail: nmizuno@ (Tel: +81-52-735-5339; e-mail: nmizuno@ **Faculty of Science andJAPAN Engineering, Toyo University, Kawagoe, 350-8585nitech.ac.jp). JAPAN (e-mail:[email protected]) **Faculty **Faculty of of Science Science and and Engineering, Engineering, Toyo Toyo University, University, Kawagoe, Kawagoe, 350-8585 350-8585 JAPAN JAPAN (e-mail:[email protected]) (e-mail:[email protected])
Abstract: We developed a full automatic badminton playing robot "MRO Brothers". The developed Abstract: We developed a full automatic badminton as playing "MRO player. Brothers". Thesystem, developed robot can receive and serve in the same environment humanrobot badminton In this the Abstract: We developed a full automatic badminton as playing robot "MRO player. Brothers". Thesystem, developed robot can receive and serve in the same environment human badminton In this the Abstract: We developed a full automatic badminton playing robot "MRO Brothers". The developed shuttle is detected by Kinect sensor and its orbit is estimated based on the distance image of the shuttle. Abstract: We developed a full automatic badminton as playing robot "MRO player. Brothers". Thesystem, developed robot can receive and serve in the same human badminton In this the shuttle is detected bycan Kinect sensor and itsenvironment orbit estimated based on will the distance image of system, the robot can receive serve in environment as human badminton player. In the Moreover, the robotand automatically move to theis spot where a shuttle fall and hit back by shuttle. normal robot can receive and serve in the the same same environment as human badminton player. Init this this system, the shuttle is detected by Kinect sensor and its orbit is estimated based on the distance image of the shuttle. Moreover, the robotbycan automatically move to theis spot wherebased a shuttle will fall and hit it back by shuttle. normal shuttle is detected Kinect sensor and its orbit estimated on the distance image of the badminton racket. shuttle is detected bycan Kinect sensor and its orbit is spot estimated based on will the distance image of the shuttle. Moreover, the robot automatically move to the where a shuttle fall and hit it back by normal badminton the racket. Moreover, robot move to the where aa shuttle will fall and hit it back by Moreover, the robot can can automatically automatically move to the spot spot where shuttle will fallLtd. and All hitKalman it backreserved. by normal normal badminton racket. © 2019, IFAC (International Federation Automatic Control) Hosting by Elsevier rights Keywords: Autonomous robot, mobile of robot, visual sensing, badminton playing robot, filter. badminton racket. badminton Keywords: racket. Autonomous robot, mobile robot, visual sensing, badminton playing robot, Kalman filter. Keywords: Autonomous robot, robot, mobile robot, robot, visual sensing, sensing, badminton playing playing robot, Kalman Kalman filter. filter. Keywords: Keywords: Autonomous Autonomous robot, mobile mobile robot, visual visual sensing, badminton badminton playing robot, robot, of Kalman filter. robots Table.1 Specifications Badminton 1. INTRODUCTION Table.1 Specifications of Badminton robots Dimension [mm] W675×D890×H900 1. INTRODUCTION Table.1 Specifications of Badminton robots Dimension [mm] W675×D890×H900 Weight [kg] 20robots Table.1 Specifications of Badminton 1. INTRODUCTION Every year, the "NHK Student Robocon," one of a Japanese Table.1 Specifications of Badminton robots 1. INTRODUCTION Dimension [mm] W675×D890×H900 20 1. INTRODUCTION NumberDimension ofWeight [kg] DC motor 7 [mm] W675×D890×H900 Every year, theis"NHK oneleading of a Japanese Robot contest held Student to selectRobocon," a Japanese student Dimension [mm] W675×D890×H900 Weight [kg] 20 Every year, the "NHK Student Robocon," one of a Japanese Number of DC motor 7 actuatorsWeight Servo motor 4 [kg] 20 Robot for contest is (Asia-Pacific held Student to selectRobocon," a Japanese leading student Every year, the one of [kg] 20 Number ofWeight Servo DC motor 74 team ABU Broadcasting Union) Robot actuators Every year, theis"NHK "NHK oneleading of aa Japanese Japanese motor Robot contest held Student to selectRobocon," a Japanese student [unit] of Air-cylinder Number DC motor 717 team for ABU (Asia-Pacific Broadcasting Union) Robot Number of DC motor Robot contest is held to select a Japanese leading student contest. actuators Servo motor 41 [unit] Air-cylinder Robot contest held to selectBroadcasting a Japanese leading team for ABUis (Asia-Pacific Union) student Robot actuators Servo motor 44 contest. [unit] actuators Servo motor team for ABU (Asia-Pacific Broadcasting Union) Robot Air-cylinder 1 team for ABU (Asia-Pacific Broadcasting Union) Robot [unit] Air-cylinder 11 contest. In this ABU Robot contest, the theme of the competition is [unit] Air-cylinder contest. contest. In this ABU Robot contest, the theme of the competition is determined by the host country. In this ABU contest, the theme of the competition is determined byRobot the host country. In this ABU Robot contest, the theme of the competition is In this ABU Robot contest, the theme of the competition is determined by the host country. In 2015, thebyABU Robot contest was held in Indonesia and determined the host country. determined by the host country. In 2015, the ABU Robot contest was held in Indonesia and the theme of competition was Badminton. In ABU Robot contest was held in Indonesia and the2015, themethe of competition was Badminton. In 2015, the ABU Robot contest was held in Indonesia and In 2015, the ABU Robot contest was held in Indonesia and the theme of competition was Badminton. To cope with this competition theme, we have developed the theme of competition was Badminton. the theme of competition was Badminton. To copeBrothers," with thisa competition theme, we have badminton developed "MRO pair of robot that performs To copeBrothers," with thisa competition theme, we have badminton developed "MRO pair of robot that performs To cope with this competition theme, we have developed play fully automatically. To copeBrothers," with thisa competition theme, we have badminton developed "MRO pair of robot that performs Fig.1 Overview of the Badminton Playing Robot play fully automatically. "MRO Brothers," a pair of robot that performs badminton "MRO Brothers," a pair of robot that performs badminton play fully automatically. Fig.1 Overview of the Badminton Playing Robot The developed all in one robots recognize the shuttle using play fully automatically. Fig.1 Overview of the Badminton Robot In designing the badminton robot, the Playing following functions play fully automatically. The developed all in one robots recognize the shuttle using Fig.1 Overview of the Badminton Playing Robot Kinect sensor, predict trajectory of shuttle, automatically In designing the badminton robot, the following functions Fig.1 Overview of the Badminton Playing Robot The developed all in one robots recognize the shuttle using were aimed to be realized. Kinect sensor, predict trajectory of shuttle, automatically The all one robots recognize the shuttle using movedeveloped to a point ofin and perform a receiving motion, designing badminton robot, the following functions were aimed tothe be realized. The developed all in fall, onetrajectory robots recognize the automatically shuttle using In In designing the badminton robot, the following functions Kinect sensor, predict of shuttle, move tosensor, a pointpredict of fall,trajectory and perform a receiving designing badminton robot, the following functions were aimed tothe be realized. Kinect automatically thereby enabling badmintons to of be shuttle, played in themotion, same In 1) The shuttle during flight is recognized, and the vehicle unit Kinect sensor, predict trajectory of shuttle, automatically were aimed to be realized. move to aa point of fall, and perform aa receiving motion, thereby enabling badmintons to be played in the same 1) The shuttle during flight is recognized, and the vehicle unit were aimed to be realized. move to point of fall, and perform receiving motion, environments as human players. is accelerated and decelerated by the wheel mechanism to move to a point of fall, and perform a receiving motion, thereby enabling badmintons be played in the same 1) The shuttle during flight is recognized, and the vehicle unit environments as human players. to is accelerated and decelerated by the wheel mechanism to thereby enabling badmintons to be played in the same 1) The shuttle during flight is recognized, and the vehicle unit the shuttle vicinityduring of theflight predicted fall position, thereby moving thereby enabling badmintons to be played in the same 1) The is recognized, and the vehicle unit environments as human players. is by the wheel mechanism to In recent years, there are several reports and patents about theaccelerated vicinity ofand the decelerated predicted fall position, thereby moving environments as human players. is accelerated and decelerated by the wheel mechanism to at high speed. environments as human players. is accelerated and decelerated byposition, the wheelthereby mechanism to In recent years, thereplaying are several and patents about the vicinity of the predicted fall moving automatic badminton robots.reports at high speed. the vicinity of the predicted fall position, thereby moving 2) In the vicinity of the predicted fall position, fine In recent years, there are several reports and patents about the vicinity of the predicted fall position, thereby moving at high speed. automatic badminton playing robots.reports and patents about 2) In the vicinity of the predicted fall position, fine In recent there are several at high speed. In recent years, years, thereplaying are reports and patents about 2) adjustment of the ofposition of the shuttle hitting unitfine is automatic badminton robots. at However, most reports are several the studies on part of badminton In high the speed. vicinity the predicted fall position, automatic badminton playing robots. adjustment of the the of position of themounted shuttle hitting unitfine is 2) In the vicinity the predicted fall position, automatic badminton playing robots. performed by XY stage unit on the vehicle However, most reports are the studies on part of badminton In the vicinity ofposition the predicted fall position, fine play (optimal control of servicing (Liu et al. 2013), shuttle 2) adjustment of the of the shuttle hitting unit is performed by the XY stage unit mounted on the vehicle However, most reports are the studies on part of badminton adjustment of the position of the shuttle hitting unit unit. play control of servicing (Liu et al.(2011), 2013), shuttle However, most reports the studies on of adjustment by of the the XY position of themounted shuttle on hitting unit is is hitting(optimal back mechanism (China Patent shuttle performed stage unit the vehicle However, mostcontrol reportsofare are the studies onet part part of badminton badminton unit. play (optimal servicing (Liu al. 2013), shuttle performed by the XY stage unit mounted on the vehicle 3) A shuttle hitting unit mounted on the XY stage unit hits hitting back mechanism (China Patent (2011), shuttle play (optimal control of servicing (Liu et al. 2013), performed by the XY stage unit mounted on the vehicle recognition (Raina et al. 2015), etc.). unit. play (optimal control of servicing (Liu et al.(2011), 2013), shuttle shuttle 3) Aback shuttle hitting inunit mounted on the XY stage unit hits hitting back mechanism (China unit. the shuttle a timely manner. recognition (Raina et al. 2015), etc.). Patent hitting back mechanism (China Patent (2011), shuttle unit. 3) A shuttle hitting unit mounted on the XY stage unit hits hitting back mechanism (China Patent (2011), shuttle back the shuttle in a timely manner. recognition (Raina et al. 2015), etc.). 3) A shuttle hitting unit mounted on the The robothitting servicing the gameon without helpunit of hits the There are very few reports ofetc.). integrated robots that can 4) recognition (Raina et al. 2015), 3) A shuttle unit mounted the XY XYthestage stage unit hits back the shuttle in a timely manner. recognition (Raina et al. 2015), etc.). 4) The robot servicing the game without the help of the There are very few reports of integrated robots that can back the shuttle in a timely manner. human operator. operate independently. back the shuttle in a timely manner. The robot servicing the game without the help of the There very few reports of integrated robots that can 4) human operator. operateare independently. robot servicing There are very 4) The The robot servicing the the game game without without the the help help of of the the There very few few reports reports of of integrated integrated robots robots that that can can 4) human operator. operateare independently. 2.1 human The vehicle unit operator. operate independently. 2. HARDWARE CONFIGURATION human operator. operate independently. 2.1 The vehicle unit 2. HARDWARE CONFIGURATION 2.1 The vehicle unit 2. HARDWARE CONFIGURATION 2.1 The vehicle 2. The specificationsCONFIGURATION of the developed badminton playing robot In 2.1 the Thebadminton vehicle unit unitgame, the falling position of the shuttle 2. HARDWARE HARDWARE CONFIGURATION In theonbadminton The specifications of 1. the developed badminton playing robot move wide area.game, the falling position of the shuttle are shown in Table The overview of one the robot is In the the falling position of the shuttle The specifications of 1. the developed badminton robot move onbadminton wide area.game, the badminton game, the falling of shuttle are shown in Table The overview of oneplaying the robot is In The specifications of the badminton playing robot shown in Fig.1. In thefalling badminton game, theestimated falling position position of the the shuttle move on wide area. The specifications of 1. the developed developed badminton playing robot If the position will be in the coat, the shuttle are shown in Table The overview of one the robot is move on wide area. shown in Fig.1. are shown in Table 1. The overview of one the robot is move on wide area. the falling position willopposite be estimated inthe thenet. coat, the shuttle are shown in Table 1. The overview of one the robot is If should be hit back to the side of shown in Fig.1. If the falling willopposite be estimated thenet. coat, the shuttle shown should be hit position back to the side ofin shown in in Fig.1. Fig.1. If the position will be inthe the coat, If the falling falling willopposite be estimated estimated thenet. coat, the the shuttle shuttle should be hit position back to the side ofinthe should be hit back to the opposite side of the net. Copyright © 2019 IFAC 2405-8963 © 2019, IFAC (International Federation of Automatic Control) Hosting by Elsevier All opposite rights reserved. should be hit backLtd. to the side of the net. Copyright 2019 responsibility IFAC Peer review©under of International Federation of Automatic Control. Copyright © 2019 IFAC 10.1016/j.ifacol.2019.08.050 Copyright © 2019 IFAC Copyright © 2019 IFAC
2019 IFAC IAV 68 Gdansk, Poland, July 3-5, 2019
Naoki Mizuno et al. / IFAC PapersOnLine 52-8 (2019) 67–72
The shuttle is hit by rotating the plate cam (Fig. 5) via the timing belt with a DC motor to stretch the spring and release its stored energy. The cam shape is a logarithmic spiral shape, and the spring is stretched as the cam rotates, and at the moment when the cam exceeds one rotation, the energy is released and the shuttle is hit back.
To achieve this function, the badminton playing robot should move quickly to the position near the estimated falling position of the shuttle. For the quick movement of the robot, "Swerve Drive" unit was adopted, which can move in all directions and can move at high speed. In this system, we have develop the original steering-driving units as shown in Fig. 2 and arranged at the four corners of the robot pedestal.
2.4 Servicing auxiliary unit In order to perform badminton fully automatically, it is necessary that the robot has not only the passive shuttle hit back function but also the active shuttle servicing function. For this purpose, a shuttle servicing assisting unit using an air cylinder was mounted (Fig. 6 and Fig. 7).
Fig. 2 Swerve Drive Unit
Fig. 3 XY Staging Unit
In this drive unit, since a polyurethane wheel having a high friction coefficient is used as the wheel, the slip of the wheel hardly occurs even at the time of rapid acceleration, and stable running is possible.
Fig.6 Shuttle Servicing Auxiliary Unit
For the steering of the wheels, Dynamixel's high-speed servomotor RX-24F is used. The direct driven design of the "Swerve Drive Unit" makes the unit slim and the backlash of the steering minimum. The driving of the wheels is performed by transmitting the power of the DC motor by a timing belt, and the rotational speed of the tire is measured by the rotary encoder to perform the feedback speed control.
Since it is necessary to set the servicing auxiliary unit outside the moving range of the XY stage unit, the servicing auxiliary unit is designed that the arm of the shuttle grasper can be extended to the front of the robot by the air cylinder. By using the air cylinder, the arm can be contracted at high speed so that the racquet does not collide during servicing.
2.2 XY stage unit
When the air cylinder contracts from the extended state, the shuttle grasper opens and the shuttle drops as shown in Fig. 7. Servicing is done by hitting the shuttle with the racket at that moment.
After the robot was moved by the vehicle unit to the vicinity where the shuttle was predicted to fall, the position of the shuttle hitting unit mounted with an XY stage unit was finely adjusted (Fig. 3). A shuttle hitting unit, which will be described later, is mounted on the unit, and the shuttle hitting unit can be moved by driving a timing belt with a DC motor. The position is measured by a rotary encoder and a potentiometer, and feedback control is performed. This unit has a stroke length of about 500 mm in both the X-axis and the Y-axis, and makes it possible to hit back a shuttle falling within a range of 500 mm×500 mm in front of the robot.
(a) Closed state (b) Open Fig. 7 Open and close stages of the shuttle grasper 3. CONTROL CIRCUIT CONFIGURATION 3.1 Overall circuit configuration
2.3 Shuttle hitting unit
Fig. 8 shows the overall configuration of the control circuit. There is a main circuit board for controlling the entire movement of robots, and each motor is controlled by transmitting commands to motor drivers of respective mechanisms using CANs (Controller Area Network).
The shuttle hitting unit plays a role of returning the dropped shuttle to the opposite side of the central net (Fig. 4).
The values of the various sensors are also acquired using CAN.
Fig. 4 Shuttle hitting unit
Fig. 5 Periphery of the cam 68
2019 IFAC IAV Gdansk, Poland, July 3-5, 2019
Naoki Mizuno et al. / IFAC PapersOnLine 52-8 (2019) 67–72
69
In this controller board, by converting the UART signal for writing and debugging programs to microcomputers into Bluetooth signal, all developing procedure can be executed without connecting cables to the robot.
Main CB Self-position estimation Ground encoder Gyro sensor
In addition, the main control board has a microSD card slot, which can be read/written from/to the microcomputer, and the robot's operation can be changed without rewriting programs by holding parameters of the robot in the microSD card.
Mobile mechanism Motor driver Rotation speed encoder Steering servomotor
Moreover, the main control board includes a user interface such as a tact switch, a rotary encoder, a DIP switch, and a liquid crystal display, as well as a USB interface corresponding to the DualShock3 and a Bluetooth other than writing, and the robots can be controlled using the game controller.
Main PCB control XY-Stage Motor driver Rotary encoder
3.4 Motor driver A gate driver IC (IRS2184S) is used to drive the FETs, all of which are composed of H-bridges made of IXTP180N10T made by Nch FET (IXYS, and the FETs operate by PWM signals from a control microcomputer that receives commands via CANs.
Shuttle grasper control Pneumatic electro-magnetic valve Cam rotation control
A magnetic digital isolator (ADUM1400ARW) is connected between the control microcomputer and the gate driver, so that noises from the motor do not affect the control microcomputer.
Motor driver Rotary encoder Estimation of shuttle orbit
In the H-bridge section, the common mode choke coil, the X/Y capacitor, and the ferrite bead are arranged at appropriate positions to prevent noise from entering the weak electric system from the driver circuit.
Box type computer-Kinect v2 Fig. 8 Overall Circuit Configuration Diagram 3.2 CAN
3.5 Sensors
CAN using Ethernet Cat5 cables is used for communication between control circuit board.
3.5.1 Rotary encoder Incremental rotary encoders (E6A2-C manufactured by Omron, etc.) are used to detect the position and velocity of robotic mechanisms. The standard cable was replaced with a shielded cable because the standard cables had low noise tolerance and caused malfunction.
Since CAN is characterized as the differential communication, it is resistant to external noise generated by motors etc. The reason of using Ethernet cables is also to prevent noise.
3.5.2 Gyro-sensor This robot is equipped with a uniaxial gyro-sensor (R1350N manufactured by MicroInfinity) for self-position estimation.
The twisted pair cables in the Ethernet cable increases the resistance to external noise. In addition, since the CAN is a bus type network, in a robot on which many independent circuit units are mounted, the amount of wiring can be saved, the weight of cables can be reduced, and at the same time, operation maintenance cost can be reduced.
Inside the R1350N, the Kalman filtering of signals is performed, so that highly accurate angular data can be obtained. 3.5.3 Kinect v2
The communication rate was 500 kbps, which is commonly used in in-vehicle systems.
Table 2 Specification of Kinect v2 Depth retrieval method Time of Flight Depth resolution 512px × 424px Depth frame rate 30fps Depth awareness 0.5m ~ 8.0m Depth angle of view 70 degrees horizontal and 60 degrees vertical
3.3 Main control board The Renesas Electronics RX631 is used as a microcontroller for the main control board (Main CB).
69
2019 IFAC IAV 70 Gdansk, Poland, July 3-5, 2019
Naoki Mizuno et al. / IFAC PapersOnLine 52-8 (2019) 67–72
By utilizing such a property, the shuttle is recognized in real time.
To detect the flying shuttle, the Kinect v2 made by Microsoft, is used. Table 2 shows the specifications of Kinect v2.
4.1.1 Labeling processing
3.6 Orbit prediction calculation PC
In order to detect shuttle candidates from the image, the distance image is subjected to a laveling process.
To acquire range data from Kinect v2 and perform trajectory estimation calculation, the “Brix Pro” PC is used.
In the labelling process, the same label is assigned when the difference in brightness between the target pixel and the neighboring 8 pixels is less than the threshold value.
Table 3 shows the specifications of “Brix Pro”. Table 3.2 Brix Pro specifications
CPU Memory Storage
Fig. 10 shows the result of color coding for each label applied to Fig. 9.
Core i7-4770R (Haswell/4 Core/8 Thread/3.2GHz/6MB Cash) DDR3L-1600 4GB × 2 TS128GSSD370
The PC is connected to a USB-CAN convert unit via a USB isolator. The USB-CAN convert unit includes a USB-toUART converter chip and a microcomputer for CAN communication. The PC transmits data by transmitting binary data to the COM port.
Fig. 10 Results of the labeling process By the laveling process, it can be confirmed that the region is divided into a shuttle region, a net region, and a region without anything.
4. SOFTWARE CONFIGURATION The badminton playing robot needs to perform the following operations in real time.
4.1.2 Extraction of shuttle images
1) Recognize the shuttle in flight. 2) Predicting the fall position of the shuttle (Shishido H. et al. 2012). 3) The robot is moved to the predicted fall position, and the shuttle is hit back with good timing using the XY stage unit and the hitting unit.
This section describes how to identify whether a region is a shuttle. The size of the region is used for identification. The size of the region S is defined in Eq. (1) using the number of pixels p in the region, the average value of the distance image of the region d, and the focal length of the sensor f.
4.1 Shuttle recognition (1) Kinect v2, a visual sensor for gaming machines, was used to recognize shuttle in flight.
If the size of the region is within the threshold, the region is considered as a shuttle. However, since the distance image contains a lot of noise, the noise may be erroneously recognized as a shuttle. To avoid the recognition error due to noise, the past several frames are referenced and whether the recognition result of the current frame is appropriate as a shuttle trajectory is considered.
When a shuttle in flight is shot using this sensor, a distance image is obtained as shown in Fig. 9.
To extract the trajectory of the shuttle, the RANSAC algorithms were used. The shuttle trajectory was approximated by a quadratic function with respect to time, and the one with the smallest error between the approximation function and the measured data was defined as the shuttle trajectory.
Fig. 9 Example of distance image In the figure, objects at distances far from the viewpoint are displayed in black, and objects at close distances are displayed in white. In this case, the shuttle image is indicated by a red circle.
4.2 Prediction of shuttle falling position An extended Kalman filter is used to predict the shuttle falling position. The algorithm is an extension of a Kalman filter, which is a technique for obtaining an optimum estimation value from data including noise, so as to be able to cope with a nonlinear system.
From this distance image, it can be seen that the difference in luminance between the shuttle and the periphery of the shuttle is large, that is, a difference in depth occurs. 70
2019 IFAC IAV Gdansk, Poland, July 3-5, 2019
Naoki Mizuno et al. / IFAC PapersOnLine 52-8 (2019) 67–72
4.2.1 Extended Kalman Filter (Shuku T. 2014)
71
In the above algorithms, the Predict step estimates the status of the system, and the Correct step modifies the system estimate based on the observations. This processing enables state prediction of the nonlinear transition system.
In this case, it is assumed that a nonlinear system including noise and an observation signal are expressed by the following equation with time as.t
4.2.3 Extended Kalman Filter Trajectory Prediction
(2) (3)
The motion state of the shuttle is defined by the following three-dimensional position, velocity and acceleration.
Where,
Where, the z-axis represents the vertical position, and the upword is positive. (5) In addition, it is assumed that are normal distribution white noises having a mean value of 0 and the variances , respectively. First, the function is linearized around the states to obtain the following Jacobian matrix :
(6)
(7)
(4)
The transition of the state is a parabolic motion of the object in which the air resistance is proportional to the square of the velocity as follows.
In addition, defining : a priori and a posteriori estimates of the state : covariance matrix of a priori and a posteriori estimate error : Kalman gain
(8) (9) (10)
The states of the systems can be estimated in two steps of Correct and Predict as follows. Where the covariance matrix is known. Give an initial value While(true){ If (observe ) Correct( )//Correct steps Predict()//Predict steps
(11)
Where k is the air resistance coefficient and gravitational acceleration coefficient. In this case, the Jacobian matrix is as follows.
} Correct( ){
} Predict(){
is the
(12)
Where, dt is the infinitesimal time. One Prediction step and one Correction step were carried out based on the observed position data of the shuttles, and after that, only the Predict step was repeated until the predicted
}
71
2019 IFAC IAV 72 Gdansk, Poland, July 3-5, 2019
Naoki Mizuno et al. / IFAC PapersOnLine 52-8 (2019) 67–72
height z of the shuttle match the position of the racquet, and the position at that time was used as the predicted fall point x,y. 4.3 Robot movement and Shuttle hit back
Fig. 12 MRO Brothers Fig. 11 Schematic diagram of movement mechanism Fig. 11 shows the schematic diagram of movement mechanism of the robot.
Highly accurate recognition of shuttles and trajectory prediction were realized by labeling and extended Kalman filtering of distance images obtained by Kinect.
Each steering unit is oriented at the center of rotation and needs to provide a speed proportional to the distance from the center of rotation.
These functions have made it possible to hit back shuttle with high success rate. This robot participated in the Student Robocon 2015 and received the Idea Award.
Assuming that the target speed V at the center of the robot, the target angular velocity , and the direction of travel of and velocity of the steering unit the target, the angle located at the coordinates of the pivot center(X,Y) and the coordinates satisfy the following equations (13) to (16).
Acknowledgement We would like to express our heartfelt gratitude to many people, including the Nagoya Institute of Technology's Tomoekai, for your great support for this research. We would also like to express our particular gratitude to Tatsuya Murao for his great cooperation.
(13)
REFERENCES
(14)
ABU Robocon: http://www.abu.org.my /ABU_Robocon-@About_ABU_Robocon.aspx
(15)
China Pataent (2011): googleapis. com/ CN102363073B.pdf
(16)
https: //patentimages. storage. 80/31/e7 /932bd228ff677f/
By applying the above equation to the four steering units and performing the control, it is possible to move to the predicted fall position of the shuttle at high speed.
NHK Student Robocon 2015 Rulebook. http://www.officialrobocon.com/ jp/ daigaku/ daigaku2015/ entry/ 2015_RULE_ROBOMINTON_0910ver.pdf
The speed control is realized by the PID control and the feed forward control using the target values of the acceleration of the robot, the angular acceleration, the speed of each wheel, and the acceleration.
Liu M. et al. (2013): Model-free and model-based timeoptimal control of a badminton robot, Proceedings of 9th Asian Control Conference (ASCC). Rina A. et al. (2015): Shuttlecock tracking and trajectory estimation using Microsoft Kinect sensor, International Journal of Application or Innovation in Engineering & Management (IJAIEM), Volume 4, Issue 10, pp.80-87.
When the time required for the shuttle to reach the ground becomes equal to or less than the threshold value, the XY stage is moved and the shuttle is hit in accordance with the predicted fall position.
Shishido H. et al. (2012): Shuttle trajectory estimation from Badminton images using particle filter and Kalman filter, Workshop on Practical Application of Motion Picture Processing DIA2012 in Japanese.
5. CONCLUSIONS Two fully automated Badminton playing robots were developed (Fig. 12) that employs "Swerve Drive" in the steering-driving unit and enables high-speed movement in all directions.
Shuku T. (2014): Extended Kalman Filter, Website of the INVERSE PROBLEM SUBCOMMITTEE, Japan Society of Civil Engineers 'Applied Dynamics Committee.
The movie of our robots playing badminton each other was uploaded to YouTube.
https://www.youtube.com/watch?v=4FyqSeGt6NQ 72