Sinkage definition and visual detection for planetary rovers wheels on rough terrain based on wheel–soil interaction boundary

Accepted Manuscript Sinkage definition and visual detection for planetary rovers wheels on rough terrain based on wheel–soil interaction boundary Haibo Gao, Fengtian Lv, Baofeng Yuan, Nan Li, Liang Ding, Ningxi Li, Guangjun Liu, Zongquan Deng

PII: DOI: Reference:

S0921-8890(16)30782-5 https://doi.org/10.1016/j.robot.2017.09.011 ROBOT 2913

To appear in:

Robotics and Autonomous Systems

Received date : 12 December 2016 Revised date : 8 September 2017 Accepted date : 20 September 2017 Please cite this article as: H. Gao, F. Lv, B. Yuan, N. Li, L. Ding, N. Li, G. Liu, Z. Deng, Sinkage definition and visual detection for planetary rovers wheels on rough terrain based on wheel–soil interaction boundary, Robotics and Autonomous Systems (2017), https://doi.org/10.1016/j.robot.2017.09.011 This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.

Sinkage Definition and Visual Detection for Planetary Rovers wheels on Rough Terrain Based on Wheel–soil Interaction Boundary Haibo Gaoa, Fengtian Lva, Baofeng Yuanb, Nan Li*a, Liang Ding*a, Ningxi Li a, Guangjun Liuc, Zongquan Denga a State Key Laboratory of Robotics and System, Harbin Institute of Technology, Harbin 150001, China b China Academy of Space Technology, Beinjing 100094, China c Department of Aerospace Engineering, Ryerson University, 350 Victoria Street, Toronto M5B 2K3, ON, Canada Highlights Novel wheel sinkage definition is proposed for rover moving on rough terrain. A vision-based detecting method of the new sinkage definition is described. Sinkage detection method is verified through wheel sinkage detection experiments.

1

Abstract: Wheel sinkage detection is of tremendous significance for planetary rover mobility optimization control and prevention of serious wheel sinking. Wheel sinkage is regarded as the distance between the lowest point of the wheel in the soil and the horizontal flat terrain on relative research. For rovers moving on rough terrains, the above definition is not reasonable because the horizontal flat terrain cannot be found. New wheel sinkage definition and detection method are proposed, based on vision. The sinkage definition rationality is analyzed for various terrain conditions. The discrete mathematical wheel sinkage calculation model, for which the input is acquired through a visual method, is built. Saturation of wheel–soil interaction image is adjusted by a dynamic piecewise nonlinear adjustment method. The Haibo Gao e-mail: [email protected] Fengtian Lv e-mail: [email protected] Baofeng Yuan e-mail:[email protected] Corresponding author :Nan Li e-mail: [email protected] Corresponding author: Liang Ding e-mail: [email protected] Ningxi Li e-mail: [email protected] Guangjun Liu e-mail: [email protected] Zongquan Deng e-mail: [email protected]

image is processed into a binary image based on HSI (Hue, Saturation, Intensity) color space. The wheel–soil boundary is extracted from the corrected wheel region outline according to its morphological features. The wheel sinkage and equivalent terrain interface angles are calculated through the discrete mathematical wheel sinkage calculation model. Sinkage definition rationality and detection method applicability are experimentally validated in the four typical terrain conditions (flat, bulgy, sunken, and uneven terrains). Particularly, the experimental results prove the sinkage detection method has high accuracy for flat terrain, for which, the deviation of the sinkage modulus is less than 2% of the wheel radius. The sinkage detection method is proved to have fairly reasonable adaptability to complex illumination conditions, based on the experimental results for different illumination conditions. Keywords: sinkage definition ； sinkage detection ； wheel–soil boundary； planetary rover； color image enhancement

1. Introduction Rovers have become a major tool for planetary exploration. They need to exhibit challenging mobility on rough terrain during the planetary exploration missions. A planetary surface is typically covered with regolith, which contains fine soil grains. The wheel–soil interaction has a significant influence on the rovers’ mobility [1-2]. In general, a hard terrain produces greater traction than a soft terrain and thus, results in more effective mobility. When moving on a soft terrain, a rover’s wheels may slip and/or sink into the soil and experience traction loss and consequently, diverge from the expected path. Substantial wheel sinkage may result in total traction loss and immobilize the rover [3]. For example, the Spirit and

Op pportunity rovvers had sunkk into soil many m times [44]. A maajor wheel ssinkage in the loose sandy soil off the Op pportunity M Mars rover, between b Apriil to June 2 005, cau used it to be nnearly immobbilized for seeveral weeks, and theereafter, it experienced similar isssues on sevveral occcasions. An ability to measure m wheeel sinkage coould hav ve potentiallyy enabled vehicle controll schemes, wh which cou uld have mitiigated these problems. p At A present, the study of terramech hanics of roover mo obility on a deeformable annd rough terraain is challengging. Wh heel sinkagee is a key variable for estimating and preedicting whheel-soil inteeraction [5– –8], and iss a sig gnificant paraameter for caalculating thee rover dynam mics of motion on a deformablee terrain [9]. Thus, definiition d detection of wheel sinkage are of tremenddous and imp portance for achieving rovver control when w it is movving on soft and rouggh terrains. On O detection of o wheel sinkkage, b modulatedd to thee wheel torqque of a roover could be imp prove tractioon, or its mootion plan is revised to avvoid pottentially hazzardous highlly-deformable terrain. W When wh heel sinkage is substantiaal, a rover requires r a laarger driiving torque to traverse thhrough the area. a Howeveer, if this driving torqque is beyonnd the capabillity of the mootor, ger of becom ming theen the rover will be willl be in dang unu usable. Wheeel sinkage iss also an imp portant inputt for soiil mechaniccs parameter estimation n [10], terrrain ideentification, aand classificattion algorithm ms [11-12]. In a previoously reportedd research stu udy, sinkage of a rov ver’s wheel has been measured m relaative to anoother wh heel [13]. Hoowever, this method doees not yield the abssolute sinkaage relative to the terrrain. Traditiional meethods applyy the wheell–soil interaaction modell to esttimate wheell sinkage [114–15]. The machine viision meethod involvinng non-contaact measurem ment, can direectly dettect wheel siinkage. In reecent years, some s researcchers hav ve proposedd estimation of sinkagee by wheel––soil interaction imaage processing [16–24]. Most of thhese meethods are baased on gray image processing, whichh can ideentify the whheel–soil inteeraction boun ndary and/orr the wh heel–terrain innterface poinnts (WTIP), through t the ggray vallue variation between thee wheel area and terrain aarea, to detect wheell sinkage. Foor example, Reina R et al. [16] o identify itss rim attaached a speccial pattern too the wheel to and d plane, andd defined a target t retriev val area. Conntact poiints between the wheel annd terrain weere then deteected by computing tthe intensity difference along a each raadial r of l° in line on the wheel plane withh an angular resolution mage. The deppth of sinkag ge was calcullated thee captured im usiing the entraance and deeparture anglles. Though this meethod can dettect wheel sinnkage faster, it is sensitivve to thee illuminatioon intensity and shado ow. Hegde has pro oposed a new w method to detect d wheel sinkage s basedd on collor image proocessing [21––22]. In the sttudy, a few arreas, wh hich were cconsidered irrelevant i to sinkage, w were rem moved. Thee complete wheel–soil boundary was ideentified by paartitioning a wheel–soil image i into a soil reg gion and a non-soil reegion. Both the wheel––soil

inteeraction bou undary and WTIPs were acquired to calcculate the terrain interactiion angles (T TIA) and sinkkage dep pth. Spiteri et al. presentss a monocullar vision baased app proach that caan detect and estimate robu ustly the sinkkage of a hybrid legged wheel inn real time, under changging opeerational con nditions [23––24]. The prroposed methhod invo olves color-space segmenntation that id dentifies the leg con ntour and consequently deppth of wheel sinkage into the rego olith. Wheel W sinkag ge is produceed by wheel– –soil interactiion. Thee wheel sinkaage z0, whichh is depicted d in Figure 1, 1 is regaarded as the distance betw ween the low west point of the wheeel in the soil and the horrizontal flat teerrain [1]. It can be calculated c using the equat ation z0 = r(1-cosθ1), wheree θ1 is th he entrance angle. A planeetary surface has vast areas of rough terrain with soft soil, aas shown in Figure F 2. In case c n, the aboveementioned proposed p whheel of rough terrain kage definittion neglectts the imp pact of terrrain sink und dulations on wheel w sinkagee. wh eel

y x

O

θ1

θ2

horizon ntal terrain

z0

Fig.. 1 Earlier sinkaage definition, w where z0 is the wheel w sinkage, θ1 is th he entrance anglle and θ2 is the departure anglee

Fig.. 2 Rough terraiin with soft soill on the surface of Mars

For F rovers moving m on roough planetarry surfaces, this defi finition is un nreasonable because of the absence of horizontal flat terrains. Foor example, intuitively, for rovers moving on bulgy terrrain (a terraain conditionn of bum mpy terrains) shown in Fiigure 3, the wheel sinkagge z willl be greater than z0. Thus,, if z0 = r(1-cosθ1) is takenn as the wheel sinkaage, then it w will not pred dict an accurrate wheeel sinkage and providee enough waarning of whheel sink king in advaance. With reegard to terramechanics, the

normal stress calculated from the Bekker model will be much smaller than its actual value, and it will cause the driving resistance force to be smaller than actual value calculated using the Wong–Reece model. The draw pull calculated by the Wong–Reece model will also be smaller than the actual value. For efficient movement of a rover in bulgy terrain, correct evaluation of the traction and wheel driving torque are required. If these are calculated with z0 (the previous definition of wheel sinkage in Figure 1) as the sinkage in the relative calculation equation, then the values obtained will be smaller and the rover will be unable to predict accurately the mobility of the terrain, which will be a drawback for rover to maintain control. The wheel sinkage z is between the maximum wheel penetration in the terrain H and the traditional definition of sinkage, h = z0. Thus, the earlier definition of wheel sinkage is not perfectly suitable for application to rough terrain and another definition is required that will be more apt.

morphological features. Experimental results show that the algorithm is accurate and robust to variation in terrain and lighting conditions. This paper is organized as follows. In Section 2, we propose a new sinkage definition and analyze its rationality. In Section 3, we describe the color image enhancement and image binarization processing. In Section 4, we present the wheel–soil extraction method. In Section 5, we analyze the experiment results. The paper is concluded in Section 6.

2. Wheel Sinkage Definition and Mathematical Calculation for Rovers Moving on Rough Terrain 2.1 Wheel Sinkage Definition for Rovers Moving on Rough Terrain The area of the wheel lying hidden in the soil in rough terrain is different from the area covered by the soil when the wheel has the same entrance angle on a flat terrain. This is illustrated in Figure 3. The actual wheel sinkage is greater than h (h = r(1-cosθ1)) and less than H. According to the previous definition of wheel sinkage, the direction of sinkage is perpendicular to the flat terrain. As shown in Figure 4, the distance between the lowest point of the wheel and the plane of the flat slope which the rover is climbing, is defined as the sinkage in slope control and terramechanics research [25]. Consequently, the sinkage direction is perpendicular to the slope plane. Thus, the sinkage may be considered as a vector that is perpendicular to the plane of the flat terrain.

Fig. 3 Wheel sinkage in a bulgy terrain. H is the distance between the lowest wheel point in the soil and the highest point of the terrain. The variables z0, θ1 and θ2 are defined in Figure 1.

In this paper, we present wheel sinkage as a vector in our new definition based on the wheel–soil boundary. Our definition is validated to be more reasonable than the abovementioned definitions via analysis of its application to a variety of terrain conditions. A new sinkage detection method, based on vision, is proposed. We present a new strategy to adjust the color image saturation for enhancing the color difference between the wheel and terrain regions. We believe that this method could result in more accurate wheel region binarization and wheel–soil boundary extraction. Because the image binarization processing is based on the color difference between the wheel and terrain regions, the wheel region outline extracted from the binary image is corrected to ensure the wheel rim is nummular. It is essential to describe mathematically the wheel–soil boundary extraction based on its

Fig. 4 Wheel sinkage (z0) in a flat slope

An assumption is that the rover’s wheels are closed wheels. Such types of wheels can avoid wheel–soil

boundary dislocation caused by the soil flowing into the wheel, so the wheel–soil boundary can be extracted through specific methods. Figure 5 illustrates our wheel sinkage definition and detection model. In the figure, O is the wheel’s center; r is the wheel’s radius. The origin of the coordinates ΣO-xy is located at the center of the wheel, the x-axis is parallel to the horizontal direction, and y-axis is negative of the gravity direction. Each point P of the wheel–soil   boundary can be expressed as P ( x )  xi  f ( x ) j , where 



f(x) is the smooth terrain function. i and j are unit vectors along the x-axis and y-axis, respectively.

θ1

θ2

 p

B

 z

A

 rz

  xB  (2) p   ( xi  f ( x ) j ) dx ( xB  x A ) xA  The vector rz (vector modulus is r) shown in Figure 5  is in the same direction of the vector p . The straight line  segment AB  is perpendicular to the vector p and  passes its terminal point. The starting point of vector p is at the origin O. The endpoints A and B are on the wheel rim. The vector Vdown is aligned with the gravity direction. The angle 1 is the angle between OA and  the vector p , 2 is the angle between OB  and the  vector p . The equivalent entrance angle and equivalent

departure angle are 1 and 2 , respectively. The equivalent entrance angle between vector OA can be calculated by the following equation:     1  arccos (OA  p ) ( OA p )    The equivalent departure angle between vector OB  can be obtained by the following equation:      2  arccos (OB  p) ( OB p )   

 p and

(3)  p and

(4)

2.2 Analysis of Wheel Sinkage Definition Rationality for Wheels Moving on Typical Terrain Conditions Rationality of the wheel sinkage definition is analyzed by calculating it in several standard terrain conditions. For a flat horizontal terrain shown in Figure 6, the terrain function is y = f(x) = b (b < 0).   P ( x)  xi  bj (5)

Fig. 5 Wheel sinkage definition and detection model, where O is the wheel’s center, r is the wheel’s radius, 1 is the angle between



O A and the vector p and  2 is the angle between OB and the   vector p . The vector rz (vector modulus is r) red line is in the  same direction of the vector p .

Sinkage can represent extent of wheel sinking, i.e., sinkage can reflect the position of the terrain in the coordinate system as O-xy. The straight line of sinkage passes through the wheel center. Sinkage is defined as the  difference between the mean vector p of the wheel–soil  boundary vectors and the radius vector rz , which is in the same direction of the mean vector. Mean vector of the wheel–soil boundary vectors is equal to the quotient of the vector function integral in the [xA, xB] interval divided by the difference of xA and xB. Thus, the wheel sinkage can be mathematically expressed as: xB   z  rz   P ( x) dx ( xB  x A ) xA (1)   xB   rz   ( xi  f ( x ) j ) dx ( xB  x A ) xA  Vector p is expressed as:

θ1

θ2  p

A

B

Fig. 6 Wheel sinkage in a flat horizontal terrain

A and B coordinates are represented as (xA, yA) and (xB,

yB), respectively.  x A   xB   y A  yB  b



B

A

  P ( x)  ( xB 2  xA2 )i 2  b( xB  xA ) j

(6)

 z  z  r  b  r  b

 z  z  r  b  axA2 3

(16)

H  r b

(17)

(7)

     xB z  rz   ( xi  f ( x ) j ) dx ( xB  x A )  (  r  b ) j (8) xA

   xB 1  z  rz   ( xi  f ( x) j )dx ( xB  xA )  (r  b  ax A2 ) j (15) xA 3

(9)

The sinkage z0 can then be expressed as z0 =r + b.  The modulus z of vector z is equal to z0 and its direction is Vdown. For a flat sloped terrain depicted in Figure 7, the terrain function can be expressed as y = f(x) = kx + b (k≠ 0, b<0). The following equation is obtained from the integral calculation of Equation (2):   xB  p   ( xi  f ( x) j )dx ( xB  x A ) xA (10)    ( xA  xB )i 2  ( y A  yB ) j 2  So vector p is located at the center of the wheel–soil  boundary and γ is the angle between p and Vdown.   arctan k (11)    2 p  b cos  sin  i  b cos  j (12)   xB     z  rz   ( xi  f ( x) j )dx ( xB  xA )  rz  p xA (13)    (r  b cos  )sin  i  (r  b cos  ) cos  j  z  z  r  b cos  (14)

h  r  b  ax A

2

(18)

 In this case, the modulus z of vector z is larger than

h but smaller than H. Thus, this sinkage definition is more suitable than the earlier definition (see Figure 1).  Direction of vector z is along the wheel–soil boundary symmetry-axis (y-axis).

θ1

θ2

 p

A

B



Here, the modulus z of vector z is equal to z0 and its direction is perpendicular to the terrain. Fig 8 Wheel sinkage in a bulgy terrain

For a sunken terrain displayed in Figure 9, the wheel– soil boundary function is assumed as y=f(x)=ax2+b (a > 0) in the wheel size range. The sunken terrain is symmetric about the y-axis (Vdown) in this condition.   xB   z  rz   ( xi  f ( x) j )dx ( xB  x A ) xA

 1  ( r  b  ax A 2 ) j 3

θ1

θ2  p B

A

Fig 7 Wheel sinkage in a flat sloped terrain

For a bulgy terrain illustrated in Figure 8, the wheel– soil boundary function is assumed to be y = f(x) = -ax2 + b (a > 0) in the wheel size range. The bulgy terrain is symmetric about the y-axis (Vdown) in this condition.

(19)

 z  z  r  b  axA2 3

(20)

H  r b

(21)

h  r  b  axA 2 (22)  The modulus z of vector z is smaller than h but greater than H. Thus, this sinkage definition is more reasonable than the earlier definition (see Figure 1).  Direction of vector z is along the wheel–soil boundary symmetry-axis (y-axis). For an uneven terrain displayed in Figure 10, the wheel–soil boundary function is assumed as a cosine function y = f(x) = A0 cos(2 ( x  x0 ) T )  b (A0 > 0) in the wheel size range. The uneven terrain is symmetric about the x = x0 axis in this condition.

  xB  p   ( xi  f ( x) j )dx ( xB  xA ) xA    ( xA  xB )i 2  b  A0Tc 2 ( xB  xA ) j

(23)

c  sin(2 ( xB  x0 ) T )  sin(2 ( xA  x0 ) T )

(24)

    z  rz  p   r sin   ( xB  x A ) 2 i

   r cos   b  A0Tc 2 ( xB  x A ) j



    2 b( xB  xA )  A0Tc  2   z z r p

  arctan 

 ( xB 2  xA2 )

(25)

(26)

θ1

(27)

If xB - xA is an integer multiple of the cosine wave cycle, then the area of the wheel under the line y = b is equal to the sinking area. If the cosine wave of the wheel–soil boundary is symmetric about the y-axis (i.e., x0 = 0 and xA = -xB), the sinkage is   xB   z  rz   ( xi  ( A0 cos(2 x T )  b) j )dx ( xB  xA ) xA (28)   (r  b  A0T  ( xB  xA )sin(2 xB T )) j  z  z  r  b  ( A0T 2 xB )sin(2 xB T ) (29) When the x coordinate of point B is in the range nT  ，(2n  1)T 2 ( n  N ), the number of troughs under the line y = b is less than the number of crests above this line in the wheel region. Modulus of sinkage should be in the range [r + b, r + b + A0], but if there are more waves of the wheel–soil boundary, then it is closer to r + b. Thus, the following inequality is tenable. A0T A 2 xB (30) 0 sin( ) 0 2n  ( xB  x A ) T Thus, it can be inferred that z is among the range [r + b, r + b + A0], such that when the value of n is greater (i.e., the wheel–soil boundary consists more waves), z is closer to r + b. So. As sinkage depth, z is more reasonable  than z0, and the direction of vector z is along the wheel–soil boundary symmetry-axis (y-axis). When the x coordinate of point B is in the range ((2n  1)T 2， (n  1)T ) ( n  N ), the number of troughs under the line y = b is greater than the number of crests above this line in the wheel region. The modulus of sinkage should be in the range [r + b – A0, r + b], but when there are more waves of the wheel–soil boundary, it is closer to r + b. Thus, the following inequality is tenable. A0T  A0 2 xB (31) 0 sin( ) 2n  ( xB  x A ) T Therefore, it can be said that z is in the range [r + b – A0, r + b], but when the value of n is greater (i.e., the wheel–soil boundary consists more waves), z is closer to r + b. Thus, as sinkage depth, z is more reasonable than  z0, and the direction of vector z is along the wheel–soil boundary symmetry-axis (y-axis).

θ2

 p B

A

Fig 9 Wheel sinkage in a sunken terrain

y  A cos(

θ2

θ1  p A

2 ( x  x0 ) )b T

B

Fig 10 Wheel sinkage in an uneven terrain. The black terrain line is universal rough terrain. The blue lines are a specific symmetric  uneven terrain and the vector p in this condition

Summarizing, for flat terrain, both the definitions yield the same depth of sinkage. For rough terrain, the new sinkage definition is different from the previous sinkage definition. New sinkage is regarded as a vector. Knowledge of the sinkage direction is advantageous for terrain slope recognition and reflects the slope effect of wheel–soil interaction in wheel size range for rough terrain.

2.3 Wheel Sinkage Discretized Mathematical Calculation Method The random unobtainable wheel–soil boundary is required for the wheel sinkage calculation using Equation (1). At present, the wheel–soil boundary can be extracted

thrrough visual m methods [21--22]. The extrracted wheel––soil bou undary is disccrete pixels and a not a con ntinuous functtion, so Equation (1) needs to be discretized in nto Equation (32) n of whheel– forr sinkage calcculation, wheere np is the number soiil boundary ppixels, and (xxt, yt) is the coordinate c off t-th pix xel. Equatioons (32) annd (1) can be considdered equ uivalent whenn np is large enough. e np     z  rz   ( xt i  yt j ) np (32) t 1

np

   p   ( xt i  yt j ) np

(33)

t 1

The T straightt line segm ment AB callculated by ccombining thhe vector rim m function. Thhus, the coorrdinates of acq quired to callculate the equivalent e

function f cann be  p and the w wheel A and B can an be en ntrance anglee 1

on (37) definned in Section n 3.2 to magnnify function Equatio nce between the wheel an nd soil region.. the color differen As A shown in Figure 12 (aa), the area of wheel reggion (WR R) is about a half of the whhole image, th he area of terrrain regiion (TR) is ab bout a quarterr of the wholle image, and the areaa of backgrou und region (B BR) is about a quarter off the who ole image. From F Figure 12(b), it caan be seen that t satu uration of wh heel region iss the lowest comparing c to the otheer two regio ons, and satuuration of the soil regionn is high her than wheeel region andd lower than the backgrouund. Thee total numbeer of pixels oof the wholee image is N.. NS sho own in Figuree 12 (c) is thhe pixel numb ber of saturattion valu ue S (0 ≤S ≤ 1). 1 BR B

BR

WR R

and d departure anngle 2 via Equations (3 3) and (4).

(a)

3. Wheel––soil Interaction Imaage Prreprocessing Based on HSI Coolor Sp pace

Ada aptive threshold d calculation fo or saturation ad djustment

Wheel-soil inte eraction image e

(b)

4 2 x 10 1.6

e number u be NS Pixel

Image I preproocessing, whhich is condu ucted for whheel– soiil boundary extraction, consists of o color im mage enh hancement aand binarizaation processing. Figuree 11 sho ows the steps of image preprocessing p g. The saturaation thrresholds of thhree regions (wheel regio on, terrain reggion and d backgroundd region) obtaained in the ad daptive threshhold callculation forr saturation adjustment step serve for satturation adjusstment. Saturation of the image i is adjuusted viaa a dynamic piecewise nonlinear n metthod. The im mage bin narization whhich will be described in n Section 3..3 is bassed on wheeel region coolor. The pu urpose of im mage bin narization is tto segment thhe wheel regiion and obtainn an image in which wheel regionn is white and other regionss are blaack. The wheel region outline can be ex xtracted from m the image and then the wheel-sooil boundary can c be extractted.

TR

1.2 0.8 0.4 0 0

0.2

0..4

0.6

0.8

Saturaation value S

1

(cc) Fig 12 Wheel–soil interaction images: (a) original image, (b) uration image an nd (c) histogram m of saturation satu

Saturation is conductedd in the wheeel region [0, Swt) occcupying 0.5 N pixels, soill region [Swt, Stb) occupyying 0.25 5 N pixels, and a backgrouund region [S Stb, 1] occupyying 0.25 5 N pixels. The param meters Swt and a Stb are the indeependent varriable threshoolds of piecewise nonlinnear function Equation (37). Thee values of Swt and Stb are ollows: deteermined as fo ki 1

ki

( !ki  K ,  N S  0.5 N   N S  0.5N  S wt  ki (34) S 0

S 0

k j 1

kk j

S 0

SS  0

!k j  K ,  NS  0.75N   N S  0.75N  Stb  k j (35) ( K   g 255 | g  Z & 0  g  255

Bin nary im mage

Im mage bina arization

S Saturation dyn namic piecewise e nonlinearity y adjustment

Fiig. 11 Flow chaart of wheel–soiil interaction im mage preprocesssing

3.1 1

Adaptivve

Thresshold

Callculation

for

Sa aturation A Adjustment Figure 12 iillustrates thaat the image can c be segmennted into three reggions (wheel region, so oil region, and bacckground reggion) based onn image saturration. Saturaation of the three reggions is adjussted by a piecewise nonliinear

( (36)

3.2 2 Saturatio on Dynamiic Piecewiise Nonlineear Ad djustment The T region seg gmentation inn Section 3.1 is cursory, baased on the threshold ds of three reegions calculaated, and it may m nerate inaccurrate wheel reegion segmen nt and mistaaken gen wheeel-soil boun ndary extracction. The dependability d of wheeel region seg gment based oon Swt and Stb is low. Thus, the satu uration needss to be adjussted. The pu urpose of im mage enh hancement is to enlarge tthe color diffference betw ween wheeel region an nd terrain reggion, so the wheel w region has

mo ore pixels whoose value is “1” and terrain n has fewer piixels wh hose value iss “1” in binnary image acquired a throough image binarizattion processinng which wiill be descrieed in Secction 3.3. A new dynam mic piecewisee nonlinear ad djustment is uused to enhance imaage saturatioon for a better enhancem ment s is adjusted byy an efffect. The whheel region saturation exp ponential funnction of indeex of 2, to low wer its value and fassten on low vvalue closer too 0. The soil region saturaation is adjusted via a sine functtion to enhan nce its value and more exttend its intervval range. Thhe adjusted saaturation is m eveen in the raange. The difference d off the saturattions bettween the w wheel and soil regions iss larger thann the oriiginal imagee. The backgground regio on saturationn is adjjusted using aan exponentiial function of o index of 0..5 to sm moothen it andd average its value. v The dy ynamic piecew wise non nlinear adjusttment equatioon is as follow ws: 2   S   S wt      S wt    S  S wt    ( S tb  S wt  )  sin  S    S wt  S tb  S wt    S  S tb     S tb  (1  S ttb )   1  S  tb   

0  S  S wt  2  (37)    S wt  S  S tb 3    S tb  S  1

In I Equation (37), the parameters Swt and Stb are obttained from Equations (34)-(36). S is the origginal satturation valuee and S  is the adjusted saturation vaalue.  and Stb are the adjusted uppper Th he parameters S wt lim mitation of saaturation valuue of wheel reegion and terrrain reg gion respectivvely. Figures 13 (a) and (b) ( showcasee the collor image ssaturation dyynamic pieccewise nonliinear adjjustment resuults. Figure 13 1 (c) demon nstrates the efffect of terrain color removal.

(a)

(b)

( (c) Fig g 13 Image enhaancement resultts: (a) color imaage enhancemen ent resu ult, (b) saturatioon adjustment result, r and (c) im mage filtering rresult

Clearly, C the tterrain color is enhanced, while the w wheel collor remains uunchanged, so s the differeence betweenn the collors of the tterrain and wheel w is morre apparent. The wh heel region inn the enhanceed image hass noise, whicch is duee to light refl flection from the terrain. The T proportioon of noiise section iss about a quaarter of wheeel region. Soo the pro oportion of nooise section in wheel regio on is about 1//8 of thee whole imagge. The imagge is filtered according too the

or in the terrrain connecteed region pro oportion in orrder colo to remove r it from m the wheel region. When the proporttion is large enough (the proporttion occupiess at least 1/66 of who ole image), th he color remaains invarian nt; otherwise, the colo or is replaced d by the wheeel color.

3.3 3 Image Bin narization P Processing for the Coolor En nhanced Wh heel–soil Innteraction Image I The cameraa’s position rrelative to thee wheel is fixxed, so that t the wheell confidence ssection (see Figure F 14) cann be diviided uniformly. A color im mage can be expressed ass an mH×n × l×3 matrix. Thus, colorr image can be b disassembbled into o three components, such as H, S, I. Each E of whichh is an mH×nl matrix x. The value oof k-th (k=1, 2, 3) componnent of each e pixel is expressed ass f(u, v, k), where w u and v are the pixel coord dinates of tthe k-th imaage component. e E[ f (u , v, k )] is the staandard deviation of each com mponent of the t wheel coonfidence seection. The k-th k com mponent value of image raange [f(u, v, k) k a, f(u, v, k)d] of the wheel region n can be calcuulated by the equations: f (u, v, k )a  Pak E[ f (u, v, k )] k  1,2,3 (38) (

f (u, v, k )d  Pdk E[ f (u, v, k )] k  1,2,3

( (39) wheere Pak and d Pdk are thhe upper an nd lower liimit perccentages resp pectively. Thee image binarization is baased on the value raange of eachh component.. For the whhole osen image coomponents value v f(u, v, k) k at imaage, if all cho poin nt (u, v) are in n [f(u, v, k)a, ff(u, v, k)d], th he value of pooint (u, v) is “1” in th he binary imaage, otherwisse, it is “0”. The T osen image co omponent in ggray space iss the image gray, cho the chosen imag ge componennts in RGB (red-green-bllue) mage componeents space are R, G, and B, and thhe chosen im H space aree H and S. Thhe results are shown in Figg.15 in HSI

Fig 14 Wheel confi fidence section

White W pixels formed an exxtensive conn nected regionn, in the soil region that is undeer the wheell bottom, in the binaary image acquired from tthe gray spacce. However, the whiite pixels in the wheel re region did no ot fill the enntire wheeel region, an nd were spreead all over the t wheel. If the gray y value rang ge is enlargeed in the imaage binarizattion processing to sattisfy the whitte pixel spreaading all over the w wheeel, then the white wheell region and soil region will com mbine together, so that tthe wheel reegion cannot be segmented properly. As the R R, G, and B components are corrrelated to brightness, the bbinary imagee acquired in the RG GB color spacce shown inn Figure 15(b b) has the saame chaaracteristics as Figure 15(aa), which is the t binary im mage acq quired in the gray g space. H However, the binary imagee in the HSI space haas a better deescription of the t wheel reggion use the H and S components are segment becau w indeependent of I (i.e., inteensity). Thus, in the work

rep ported herein, the image processing p is conducted inn the HS SI space to redduce the influuence of brigh htness.

(a))

described as [26]:

u

(b b)

Image I binarization is caarried out by b binary im mage mo orphological ooperations, suuch as expansion and erossion, to convert thee whole whheel region into i white. The M aree shown in Figgure opeerations impllemented in MATLAB 16. The result iss shown in Fiigure 17. Imclose

Bwareao open

Imfill holes

Imopen I

Fin nal binary image

Fig g 16 Flow chart of operations

y

y zw 1

T

( (40)

x OI v

u

Yc

Fig 18 Coordinate systems of the w wheel and cam mera, where O-xyyzw and Oc - Xc Yc Zc rep present the form mer and latter, respectively r

Figure F 19 dissplays the moodification of the wheel rim from m an oval to a circular shhape. The wheel rim imagge is num mmular in thee objective im mage plane, which w is paraallel to th he wheel plan ne. 1 T T ( u1 v1 1  M 1  x y zw 1 (42) Z cc1

v2 1  T

Boundaary

1 T M 2  x y zw 1 Zc 22

( (43)

Wheel plane

A y

zw O

x

4.1 1 Image Correction Based on Cam mera Im maging Prin nciple In I case the iimage plane is not paralllel to the w wheel plaane, it will caause the wheel rim in the image to apppear ovaal, which is a drawback for extracting the wheel––soil bou undary. Thuus, correctionn of the wheel w outlinee is req quired to ensuure that the wheel w rim is nu ummular. Figure F 18 illustrates the coordinate c sy ystem chosenn for thee wheel andd camera. Thhe wheel co oordinate sysstem (W WCS), whose O-xy plane coincides c witth the coordiinate sysstem of the w wheel sinkagee detection mo odel, is alongg the wh heel movemeent in the wheel plan ne. The cam mera coo ordinate systtem is exprressed as Oc-XcYcZc, whhose oriigin Oc is locaated at the opptical center of o the lens annd Zc axiis coincides eexactly with the t optical ax xis. The origiin of thee image planee coordinate system locatted at the topp left corrner pixel of the image, can be expressed as OI-uv (see Fig gure 18). Thee unit is pixell and each po oint is an inteeger. Th he relationshhip between OI-uv and O-xyzw cann be

Xc

Oc Zc

u2 Intteraction

1 M x Z cc

zw O

Fig g 17 Final binarry image

4. Wheel––soil xtraction Ex

T

 m11 m12 m13 m14   R t   ( M F T    m 21 m22 m23 m24  (41) 0 1   m  31 m32 m33 m34  c interioor parameter matrix, R is the wheere F is the camera rotaation matrix, and t is the trranslation maatrix of Oc - Xc Yc Zc relative r to O-xxyz.

((c) Fig g 15 Preliminaryy binarization processing p resullts: (a) binary im mage in gray g space, (b) bbinary image inn RGB color space, and (c) binnary imaage in HSI coloor space

Initial bin nary image

v 1 

Wheel rim OI2 u 2 Desired v 2 image a2 plane

Zc1 a1 O2

x2

y2 Zc2 Oc2 c Yc2

Xc2 amera Desired Ca positio on

x1

u1 OI1 RealO1 v1 image plane y1

Xc1 O c1

Real cam mera position

Yc1

Fig 19 Wheel rim correction c modeel

The T zw coordiinate of poinnts on wheel rim is 0, so that t each point can be b expressed aas (x, y, 0). Removing R Zc1 and Zc2 leads to thee following eequations, where (u1,v1) and (u2,v2) are the co oordinates off real image plane p and desiired a m1ij and m2ij imaage plane in Figure 19 reespectively, and are the parameteers of matrix M 1 and M2 reespectively:

1 1 1 1 1 1 (u1m31  m11 ) x  (u1m32  m12 ) y  m14  u1m34

(44)

1 34

(v1m  m ) x  (v1m  m ) y  m  v1m

(45)

2 (u2 m312  m112 ) x  (u2 m322  m122 ) y  m142  u2 m34

(46)

1 31

1 21

1 32

1 22

1 24

(v2 m  m ) x  (v2 m  m ) y  m  v2 m Combining Equations (44)-(47) yields: 2 31

2 21

2 32

2 22

2 24

2 34

(48)

v2  g2 (u1 , v1 )

(49)

4.2 Wheel–soil Boundary Geometric Feature Analysis Figure 20 shows the wheel–soil boundary geometric feature analysis model. The set W represents the wheel area, and it can be described as W = {(x, y) | x2 + y2 ≤ r2}. The soil is expressed as set Soil and the upper boundary of terrain is expressed as the set T, so T  Soil . The set Sik, which represents the wheel sinking section, is described as Sik = W ∩ Soil.



Sik  P( x, y) x2  y 2  r 2 &P( x, y)  Soil

(50)

The wheel–soil boundary, which is expressed as set Siksec, is defined as the upper boundary of terrain on the wheel sinking section. Siksec  T  W  P ( x, y ) x 2  y 2  r 2 &P( x, y )  T (51)





The outline of the wheel area set is Wol and the set Wrep is the exposed part of the wheel rim.



Wrep  P ( x, y ) x 2  y 2  r 2 &P  Soil



M

(47)

u2  g1 (u1 , v1 )



Wheel section : W

(52)

Wol  Wrep  Siksec (53) The sets T and Wol can be obtained using machine vision methods. The wheel area outline consists of the wheel outer rim line, which is above the terrain and the wheel–soil region line AB. (54)  A, B  Wrep  Siksec The distance dOM between each point M on the wheel rim and the wheel center O is r. The wheel–soil region line has nw points. The distance between the wheel–soil region line points Pi and O is dOPi. The starting point A of the AB line is P1, and its ending point B is Pnw. So dOP1 and dOPnw are r. However, the distances between other points of AB and O are less than r. This morphological feature can be described as:  P  Siksec, iff i  1  i  n, d i OPi  r   (55)  1
r O

C

B Terrain : T

A

Pi Sinking section: Sik N

Soil section : Soil Fig 20 Wheel–soil boundary geometric feature model

4.3 Wheel–soil Boundary Extraction Method Description Figure 21 shows the result of each step of wheel–soil boundary extraction. The wheel outline is extracted from the segmented binary image. The edge of binary image is corrected via Equations (48) and (49). The Hough-circle transformation is applied to the corrected edge to convert the wheel outline to points of the three-dimensional Hough space in which the wheel center pixel coordinate and wheel pixel radius rp are combined together to constitute a point (uo, vo, rp). The point appearing most often is used to derive the wheel center and radius. The wheel rim can also be obtained.

Fig 21 Wheel–soil boundary extraction flow

The points A and B (see Figure 5) can be acquired from the wheel region outline by following the steps in the flow chart shown in Figure 22. The wheel–terrain contact points A and B can be extracted using Equation (55). The points of the wheel outline between A and B constitute the wheel–soil boundary.

Start N

tp<1

Input wheel outline image I, wheel cente er pixel coordinate (u o, v o) and pixel radiu us rp

Y j=j+1 N

Calculate the e size of image : m × n

Y N

i=1; j= =1 N

I(j, i)=1 1

j=n

i=i +1, j= 1

u A=i; v A=j

Y

tp  rp  (i  uo ) 2  ( j  vo ) 2

End

Fig g 22 A WTIP exxtraction flow. (u ( A, vA) is the piixel coordinate of point A. For B exttraction, the i orriginal value is uo in the third sstep of the t flow chart.

and d the sensorrs on the pplatform are torque sennsor, disp placement sen nsor, six-axiss force / torqu ue sensor andd so on. The drive motor m drives the wheel forward f and the drag gging motorr is used to simulate th he rover’s boody mov vement. The wheel movess with different slip ratios can be controlled by y regulating the two motors at differrent rotaational speed ds. The dispplacement sen nsor is usedd to meaasure the sinkage depth. IIts base leveel is the samee as the horizontal pllane of the flaat terrain. Thee counter weiight is used u to adjustt the vertical load of the wheel, w while the actu ual load is measured m byy the six-axiss force / torrque sensor. The visiion collectionn system is used u to gain the wheeel–soil interraction imagee. The radius of the wheeel is 140 0 mm and its width is 150 mm. The 15 mm high whheel with h 28 lugs is shown s in Figuure 24. HIT-L LSS1 soil, whhich wass provided by the Shangh ghai Academy y of Spacefliight Tecchnology [8], was used in tthe experimen nts. The physical and d mechanical properties off the soil are listed l in Tablee 1. Hoisting appliance a

Counte er weight

5. Experim mental Verrification n

Dragging mottor

5.1 1 Sinkage C Calculation n Steps Ba ased on Im mage Prrocessing Herein, H we list the sinkagee calculation steps. (1) ( Acquire a wheel–soill interaction area image ffrom thee camera andd divide the wheel w confid dence area inn the image. (2) ( Transform m the image color c space frrom RGB to H HSI. Ad djust the saaturation usiing the dyn namic piecew wise non nlinear adjusstment, then resynthesize the image w with huee, adjusted saturation and a intensity.. Transform the ressynthesized iimage color space from m HSI to R RGB. Reemove the nnoise generatted by light reflection ffrom terrain in RGB color space. (3) ( Transform m the image color c space from fr RGB to HSI and d then binarizze the image. (4) ( Extract thhe outline off the wheel reegion and corrrect thee outline usinng Equations (48) and (49). ( Extractt the wh heel center and radiius by the Hough-ciircle transformation. Extract the wheel–soiil boundary via Eq quation (55). (5) ( Input thee wheel–soil boundary piixels coordinnates into the sinkagge calculationn equation (E Equation (32))) to esttimate the whheel sinkage.

5.2 2 Singlee Ex xperiments

Wheel

Sinkagee

Detecttion

5.2 2.1 Introducttion of Experrimental Equ uipment Experiments E were perfformed via the wheel––soil interaction test platform with a vision collection c sysstem as shown in Figgure 23. The three motorss on the platfform aree the drive m motor, turningg motor, and dragging mootor,

six-axis force / torque sensor

Displacem ment sensor

Came era

Fig 23 Wheel–soil interaction testt platform

Fig.. 24 Experimenttal wheel Tab ble 1 Physical and mechaniccal parameters of the soil. ρ is denssity of soil. kc is cohesive m modulus of soiil. kφ is frictional mod dulus of soil. c is cohesion off soil. φ is inteernal friction anngle of so oil [8] ρ (kg/m3)

kc (kPa/mn-1)

k φ (kPa/mn)

c (kPa)

φ(º)

1.61

15.6

2407.4

0.25

31.9

In this stud dy, images accquired from the camera were w colllected with different terrrain conditio ons and lightting con nditions, inclluding variabble wheel slip s and terrrain uneevenness. Th he lighting was varied d from norm mal unifform illuminaation to a com mplex illumination that caasts

shaarp shadows. In the data presented heere, actual vaalues forr terrain intteraction anggles (TIAs) were manuually ideentified by a hhuman analysst, based on these t images. The image is acquired at a momeent when the wheel is movving on terrain by maanually triggeering the cam mera. Thirty-eeight gro oups of experiments weree conducted. For each im mage, hun ndreds of pixxel samples are used in Equation (322) to  callculate the sinnkage z .

(a)

(b)

5.2 2.2 Wheel S Sinkage Detection Results for W Wheel Mo oving on Varrious Terrain n Conditionss Figure F 25 illlustrates the sample imaages capturedd on fou ur different teerrain conditioons (flat terraain, bulgy terrrain, sun nken terrain, and unevenn terrain). Th he wheel sinkkage dep pth is changeed via counterr weight variation and conntrol of the wheel sliip variation in i the set of experiments. For flat terrain, thhe modulus of sinkage is the distaance bettween lowestt point of thhe wheel an nd the horizoontal plaane of the terrrain, which iss detected by the displacem ment sen nsor. For otheer terrain connditions, the test bed couldd not dettect the sinkaage accuratelyy.

(aa)

(b)

(cc) (d) Fig g. 25 Terrain connditions: (a) flaat terrain, (b) bu ulgy terrain, (c)) sun nken terrain, andd (d) uneven terrrain

A. Wheel Sinnkage Detection Results fo or Wheel Movving on Various Terrains The T slip ratioo of wheel iss 0.4 and the forward veloocity of wheel is 100 mm/s. Figgure 26 disp plays the sam mple images for the wheel sinkagge detection on rough terrrain con nditions. Figgure 26(a) shows the original im mage cap ptured by caamera. The boundaries b extracted e by our meethod are shoown in Figuree 26(b). Figu ure 26(c) pressents  thee calculation result of thhe vector p . The sinkagge is sho own in Figuree 26(d). Table T 2 and F Figures 27–330 show the sinkage s detecction   ressults. The direction anglee α (α = arccos (< z ,  j >)) rep presenting thhe direction of o sinkage is defined heerein witth the anticllockwise direection taken as the posiitive dirrection of anngle value, annd z0 is the sinkage deffined preeviously in the terrameechanics stu udy. The m model pro oposed by D Ding, et al. [27] [ is used to calculatee the wh heel draw pull FDp.

(c) (d) d steps: Fig.. 26 Sample images for whheel sinkage detection orig ginal image, (b)) wheel–soil booundary extracttion results, (c))  vecttor p calculaation result, aand (d) the sinkage s vector calcculation result.

(a) the  z

For F the flat terrain shownn in Figure 27, 2 the moduulus deteection result of sinkage ass defined in th his study is very v sim milar to the preevious definitition z0. The direction d anglle α is also a very similar to 0. Thee area covereed by the soill on the right side off the y-axis iis larger than n the left sidee of the y-axis for a bulgy terrainn, as illustrated in Figure 28. Thu us, the sinkag ge should bee in the fourtth quadrant. The T  deteection sinkag ge z shownn in Figure 28 8 indeed is in the fourth quadrant. Its modulus z, which is 1.85 times ass z0, is larger l than h, h but smalleer than H. The T level of the sun nken terrain on o the right siide is higher than on the left sidee in Figure 29 9, but the breeadth of the su unken terrainn on the right side is i smaller thhan on the left side. Thhus, deteermining the area covereed by the so oil is greater on whiich side is im mpossible, annd the sinkag ge maybe in the thirrd/forth quadrrant or y-axiss. The directiion of sinkagge is Vdoown, and the resultant r sinkkage moduluss, which is 0.72 0 times as z0, is greater than H but smaller than t h. The area a cov vered by the soil on thee right side of o the y-axiss is greaater than the left side forr an uneven terrain t in Figgure 30. Thus, the siinkage shouldd be in the fourth quadrant.  ure 30, whichh is Thee detection siinkage z shhown in Figu 1.25 5 times as z0, is correctly iin the fourth quadrant. q From F Table 2, 2 it can be sseen that thee draw pull FDpn  calcculated with z has a sig ignificant diffference from the draw w pull FDpo calculated c wiith z0, exceptt for flat terrain.  And d the draw pull p FDpn calcculated with z is closerr to FDpps that was measured m wiith the six-axis force/torrque sensor. Thus, T the sin nkage definedd in this stu udy has a larrger diffference with the previou ous definition n and is more m valu uable for thee wheel–soil interaction force f calculattion wheen wheels aree moving on a rough terraiin. B. Wheel Sin nkage Detecction Resultss for a Whheel Mo oving on Flat Terrain In this paper, the slip is chaanged to mak ke wheel sinkkage larg ger and the correspondinng image iss acquired at a a mom ment. The purpose p of thhis series ex xperiments iss to valiidate the preccision of the ssinkage detecction methodd for

wh heel moving on flat teerrain. Figurre 31 plots the dissplacement ssensor measured and visually-measuured sin nkages. The x-axis is thee index of the t image bbeing anaalyzed, corressponding to one o of the 15 images colleected. Th he y-axis depicts the sinkagge modulus as a a percentagge of thee wheel radiuus in Figure 31(a). The y-axis y yieldss the   sin nkage directioon as the anggle α (α = arrcos (< z ,  j >))  bettween the sinnkage direction and  j in Figure 311(b). Deeviation of tthe visually--measured diirection and the acttual directionn (Vdown) is leess than 1° fo or the 15 imaages. Th he visually m measured sinkkage results in matchingg the

sink kage measurred by the displacemen nt sensor very v accurately. Detection deviatioon of sinkagee modulus is less l than n 2%, and an ngle deviationn of sinkage direction is less l than n 1º. Figure F 32 pllots displaceement sensorr measured and visu ually-measureed sinkages ffor a set imag ges acquired on o a con ntinuous wheeel motion wiith 1s intervaal. The slip raatio of wheel w is 0.3 and the forw ward velocity y of wheel iss 10 mm m/s. The resu ults have the same featurres with last set exp periments.

 Tab ble 2 Sinkage ddetection resultss for a wheel on rough terrainn. z0 is the prev vious sinkage (ssee Figure 1), z is the defin ned sinkage (see  Equ uation (1)), α iss the angle bettween sinkage direction definned by Equatio on (1) and  j , FDpo is FDp ccalculated with h z0, FDpn is FDp D



calcculated with z , FDps is measuured with six-ax xis force/torquee sensor Terrain conndition

z0/r

 z /r

 z /z0

α

FDpo(N)

FDpn D (N)

FDps(N N)

|FDpo- FDpss| / FDps

|FDpnn- FDps| / FDps

Flat terraain Bulgy terrrain sunken terrrain Uneven teerrain

0.15388 0.10055 0.16255 0.12799

0.1443 0.1861 0.1173 0.1599

0.94 0 1.85 0.72 0 1.25

-0.17° 2.16° 0.00° 1.61°

23.74 11.06 28.56 18.39

23.64 2 35.22 3 11.12 1 27.34 2

23.19 9 30.43 3 14.42 2 26.08 8

2.37% % 63.655% 98.066% 29.499%

1.94% 15.74% 21.13% 4.83%

 Fig. 27 Compaarison of wheel sinkage z and d z0 for a wheell on flat terrain

 Fig. 28 Compaarison of the whheel sinkage z and z0 for a whheel on a bulgy terrain

 Fig. 29 Compaarison of the whheel sinkage z and z0 for a whheel on a sunkeen terrain

 Fig. 30 Compaarison of the whheel sinkage z and z0 for a whheel on an unev ven terrain 50

4

40

3

35

2

30

1 α (°)

Sinkage (% of radius)

5

Meassured by displa acement senso or Meassured by vision n

45

25

Actual Measured d by vision

0

20

-1

15

-2

10

-3

5

-4

-5 0 2 4 6 8 10 12 1 14 16 6 6 8 10 0 12 14 16 Image N Number Im mage Numberr (a) (bb) Fig. 31 Sinkage visual-detectiion results for a wheel on a flat at terrain: (a) sin nkage modulus and as a percenntage of the wheel radius (b) the direction anngle α defined between b the sin nkage direction and Vdown

0

2

4

α (°)

(a) (b) Fig. 32 Sinkage visual-detection results for a set images acquired on a continuous wheel motion with 1s interval: (a) sinkage modulus and as a percentage of the wheel radius, (b) the direction angle α defined between the sinkage direction and Vdown

5.2.3 Wheel Sinkage Detection Results for Images Captured in Different Illumination Conditions

A. Wheel Sinkage Detection Results for Images captured with Multilevel Illumination Intensity Table 3 depicts the relationship between the image number and the illumination intensity, with which the images were captured. Figure 33 illustrates the detection result sample images for three light intensity conditions. Figure 33(a), (b), and (c) show the original images, respectively. The boundaries extracted accurately from the images shown in Figures 33(d), (e), and (f), respectively. Our wheel–soil boundary extracted method has an extensive adaptive capacity for illumination intensity, except the images having a lot of illumination supersaturated areas or extreme dark images. The deviation of the sinkage modulus detection results, shown in Figure 34(a), is less than 1% of the radius and the deviation of the angle in the visually measured direction and the actual direction (Vdown) shown in Figure 34(b) is less than 1°. Table 3 Relationship between image numbers, and the illumination intensity with which the images were captured Illumination intensity (LUX) Image number 32500 16 40625 17 48750 18 56875 19 65000 20 73125 21

B. Wheel Sinkage Detection Results for Images Captured with Non-uniform Illumination

Figures 35(a) and (b) show the original images acquired while the direction of light was changed. The boundaries extracted accurately from these images are shown in Figures 35(c) and (d), respectively. Table 4 lists the sinkage detection results for the two images. The sinkage algorithm results match the sinkage measured by the displacement sensor very accurately. Thus, the method proposed in this paper can adapt to variation of light direction, except some particular situations (such as backlighting). Table 4 Sinkage and TIAs detection results for non-uniform illumination. z0 is the previous sinkage definition (see Figure 1),  z is our sinkage definition (see Equation1), α is the angle between sinkage direction defined by Equation (1) and Vdown image

z0/r

 z /r

 z / z0

α

a b

0.0845 0.0845

0.0833 0.0916

0.99 1.08

-0.1751° -0.7895°

5.2.4 Computational requirements

Computational times of the single wheel sinkage detection experiments were obtained by using an image of 512 × 288 pixels. The sinkage detection requires 1.12 s to obtain the result using Matlab 2013 version on an Intel Core i5-4460 3.2 GHz computer. Image color enhancement involving saturation adjustment and filtration requires 0.53s per image. Image binarization processing requires 0.45s. Wheel region outline and wheel–soil boundary extraction take 0.008s and 0.012s, respectively. The MATLAB procedure is converted to .dll file, which is called to accomplish the sinkage detection procedure in Visual Studio 2013, and the sinkage detection requires 5.24 s. It takes time to call the .dll file. If the procedure is compiled with C independently, then the time cost may be significantly reduced.

(a)

(b)

(c)

(d) (e) (f) Fig. 33 Samplee images capturred with multileevel light intenssity for a wheel on a flat terrain n: (a), (b), and ((c) are the origiinal images, andd (d), (e), and (f) are the wheel––soil boundariess extraction resuults 5

50 Measured by disp M placement sensoor M Measured by visio on

3

40

2 30

1

α (°)

Sinkage (% of radius)

Actual Measured by vision

4

20

0 -1 -2

10

-3 -4

0 16

17

1 18 19 Image Number

20

21

-5 16

17

18

19

20

21

Image Number

(a) ((b) Fig. 34 Sinkage detection resuults for images captured with m multilevel illum mination intensitty: (a) sinkage m modulus detecttion results and (b) the angle beetween the sinkkage direction and Vdown

(a)

(b)

(c) (d) Fig. 35 Samplee images capturred with non-un niform illuminattion for a wheeel on flat terrain: (a) and (b) aree the original im mages, while (c) and (d) are the wheel–soil bouundaries extracttion results

5.3 Mobilee Robot Wheel W Sink kage Detecttion Experimen nts Figure 36 shows the mobile m robot, with six wheeels,

machine visiion camera is used in the experiments. A m used to monitor each of the ro robot’s wheels. The cameraa is mounted m on a supportingg arm, with three movingg deg gree of freedo oms (DOFs) aand three rottational DOFs insttalled on the robot body. A gyroscope is attached too

the camera tto detect the rotation anglle of the cam mera. The positionn of camera can be calcculated with the relation of arm and robbot body. The slip ratioo of observed wheel is 0.4 andd the forward velocity of roobot body is 17.144 mm/s. The sample image captured byy the camera is shoown in Figuree 37. aarm

5.3..2 Experimeental Resultss Obtained in Differen nt Illu umination Co onditions Illumination conditions c coontain normal illuminationn inteensity, strong illuminnation inten nsity, weakk illumination intensity, non-uuniform illum mination (seee Figures 38(a) an nd (b) acquir ired through changing thee direection of the light) l and obsscured illumiination, whichh cau uses the images to have shhadows (see Figure F 38(c))). Tab ble 6 presen nts the relaationship bettween imagee num mber and the changed illuumination co onditions withh whiich the imagees were captur ured. The T detection n sinkage moodulus is high hly similar too the depth of sink kage detectedd by manuallly showing inn Figure 39. Thuss, the sinkagee method pro oposed in this pap per has fairly y great adapptability of the complexx illumination con nditions.

lig ght source

forrward wheel camera

soil

d uneven terrain in the seet of experim ments. Table 5 and listss the sinkage detection ressults of the forward f wheeel with h the robot on n the terrain. The results have h the samee chaaracter as the single wheel experimental results.

Fig. 36 Mobilee robot in experiments

5.3..3 Computattional requirrements

Fig. 37 A sampple image captuured by the camera attached to the mobile robot

5.3.1 Experiiment Resultts for Mobilee Robot Movving on Various T Terrain Cond ditions As mentiooned earlier,, the terrain n conditions are divided into flat terrain, bulgy terrain n, sunken terrrain,

Computationa C al times of the mobile robot wheeel sink kage detectio on experiment nts were obtaiined by usingg an image of 480 0 × 640 pixeels. The sink kage detectionn requ uires 2.30s to yield the rresults using Matlab 20133 verssion on an Intel Core i5--4460 3.2 GH Hz computerr. Imaage color enhancemennt containing saturationn adju ustment and filtration takees 0.89s per image. Imagee binaarization pro ocessing takkes 1.17s. Wheel W regionn outlline and wh heel–soil bouundary extraaction requiree 0.017s and 0.188s, 0 resppectively. Th he MATLAB B procedure is con nverted to .ddll file, which h is called too accomplish the sinkage deteection proced dure in Visuaal Studio 2013, and d the sinkagee detection reequires 6.01 s. s It takes time to o call the .dlll file. If the procedure is com mpiled with C independent ntly, then the time t cost mayy be significantly s reduced. r

Table 5 Sinkagge detection ressults for a mobille robot on diffferent terrain co onditions. H and d h are the paraameters shown in i Figures 8 andd  9, z0 is the earrlier sinkage deefinition (see Figure F 1), z iis our sinkage definition (see Equation1), α is the angle between b sinkage direction defineed by Equationn (1) and Vdown T Terrain conditio ons

H/r

h/r

z0/rr

 z /r

 z /z0

α

Flat terrain Bulgy terrain n Sunken terrain n Uneven terrain n

0.4426 0.0840 -

0.1930 0.2013 -

0.1187 0.1930 0.2013 0.1930

0.1122 0.3033 0.1616 0.1921

0.95 1.57 0.80 0.995

-0.7699° 1.2611° -4.5911° -0.1277°

Table 6 Relatioonship betweenn image numberr and the illuminnation conditions with which the t images weree captured Sttrong Illumination Normaal Obscured Non-unniform Weak illuminaation illum mination illlumination conditions illumination illuminaation 1 Image numberr 22 23 24 25 266

Non-uniform N illumination 2 27

(a) (b) (cc) Fig. 38 Samplee images for com mplex illuminaation conditionss: (a) and (b) no on-uniform illum mination, and (cc) image with shadows

5

50 Ma anually-measured d anually-measure ed Vis siually -measuredd siually-measured

Actual d by vision Measured

4 3

40

1

30 α (°)

Sinkage (% of radius)

2

0 -1 1

20

-2 2 -3 3

10

-4 4 0 22

23

24

25

26

27

Image Number

-5 5 22

23

24

25

26

27

Image N Number

(a) ((b) Fig. 39 Sinkagge detection ressults for images captured withh complex illum mination: (a) sinkage moduluus detection ressults and (b) the angle between the sinkage direction and Vdow wn

6. Conclu usions an nd Discusssions A new ssinkage definnition and visual detecction method for pplanetary rovvers moving on rough terrrain was proposeed in this paaper. The con nclusions of this study are as ffollows: (1) This ppaper proposees a new deffinition of w wheel sinkage for rrough terrain. Rationality of the definiition is analyzed ffor a variety of terrain con nditions to prrove that the definnition is moree reasonable than t the prevvious definition. (2) A new w method based b on thee vision sinkkage detection m method is deescribed in this paper. The saturation iss adjusted via v the dyn namic piecew wise nonlinear addjustment to enhance the color conntrast between the wheel and sooil regions in n the images. The enhanced im mage is binarrization processed to acqquire the image in which the wheel region iss white and oother regions are bblack. Thus, the wheel reegion outline can be extractedd from the binary b image using the eedge

od. The ouutline is corrrected usingg deteection metho Equ uations (48) and (49) to represent th he wheel rim m worrking as a ciircle. The maathematical description d of the wheel–soil boundary b extrraction from wheel regionn outlline is propo osed through the analysis of the wheeel regiion outline and a wheel–sooil boundary y. Wheel–soiil bou undary is exttracted by im mage processsing based onn the mathematicaal descriptionn. The sinkagee is calculatedd by supplying as a an input the wheel–ssoil boundaryy coo ordinates in to o Equation (322). (3) Single wh heel sinkage ddetection exp periments andd mob bile robot wh heel sinkage detection exp periments aree accomplished in n four typicaal terrain con nditions. Thee exp perimental ressults prove th that the sinkaage definitionn prop posed in thiss paper is m more reasonab ble for rovers mov ving on rough and softt terrain. Parrticularly, thee sink kage detectio on method hhas a high accuracy for wheeels moving on a flat terr rrain, and thee deviation of the sinkage mo odulus is lesss than 2% of the wheeel radiius. The sinkage detectionn method is proved to havee

a great adaptability of complex illumination conditions based on the experimental results for different illumination conditions. New sinkage definition and visual detection method proposed in this paper have advantages and limitations. (1) Advantages. The primary advantages of new sinkage definition lie in reflecting the impact of terrain undulations on wheel sinkage. For rough terrain, the new sinkage definition is corresponding to our intuitive recognition. For flat terrain, both the definitions yield the same depth of sinkage. The sinkage definition proposed in this paper is an extension of the pervious definition. The direction of sinkage reflects the slope effect of the wheel–soil interaction in the wheel size range for a rough terrain. The image binarization is based on H and S components value range of wheel region, which are independent of I (intensity), so the detection method is not sensitive to light condition. The visually detection method proposed in this paper have a great adaptability of complex illumination conditions (illumination intensity variation, light direction variation, and shadow produced by obscured illumination) based on the experimental results for different illumination conditions. (2) Limitations. The variation of terrain in the lateral direction is not considered in the model proposed in this paper. The terrain is regarded as uniform in the lateral direction, and the sinkage can be observed in the inner or outer xy plane of the wheel is constant along the z direction. With the wheel moving on terrain, the soil will flow. As the flow of soil particles is three-dimensional, besides the sinkage resulting from longitudinal flow of soil, the lateral flow of it can also cause sinkage and the wheel-soil boundary changing. From the experiment results we can know the sinkage caused by lateral flow of soil is slight. The sinkage caused by lateral flow of soil is neglected in this paper. Soil compaction and wheel lugs motion need to be discussed. The discussions are as follows: (1) Soil compaction. The compaction of loose soil can influence the relative position of wheel and terrain, and the sinkage will become smaller with compact soil. New sinkage depth is concerned to the position of wheel-soil boundary in wheel coordinate system O-xy. Regardless the soil is loose or compact, the new definition is a measure of the difference between wheel radius and the distance which is between the wheel center and the terrain surface. The accuracy of sinkage detection is irrelevant to soil properties. But soil property variation needs to be considered for calculating the wheel-soil interaction force. (2) Motion of lugs. The motion of lugs has influence on wheel slip-sinkage process. On the one hand, the soil is dug by lugs when the wheel is rotating, then this soil will be discharged to the back of the wheel, so the sinkage will be bigger as a result. On the other hand, the soil acts at the top end of the lugs and the wheel is

somewhat supported by the resulted supporting force, thus the wheel sinkage can be reduced. However, the sinkage defined in this paper is through analyzing the position of wheel-soil boundary in wheel coordinate system O-xy, and the sinkage definition is irrelative to wheel slip-sinkage process. The position of wheel-soil boundary in wheel coordinate system O-xy is formed by wheel rotating, lug digging soil and supporting the wheel. Thus, the influences by motion of lugs are taken into account in the new sinkage definition.

7. Future work (1) The sinkage detection method proposed in this paper is appropriate for closed wheels. For open wheels, soil flowing into the wheel interior will cause the wheel–soil not to be coincident with the previous terrain that the wheel is not in contact with. The boundary cannot be extracted integrally for an open wheel. Thus, the equation cannot be used to calculate the wheel sinkage. These issues need to be solved for the sinkage detection method to be appropriate for open wheels. (2) The sinkage definition proposed in this paper is more reasonable than the previous definition for rovers moving on rough and soft terrains. The sinkage definition proposed in this paper is an extension of the previous definition to make the sinkage depth being more reasonable and becoming a vector. For rover moving on rough terrain, the study of terramechanics is a challenging issue. The sinkage proposed in this paper is preparation for terramechanics study on rough terrain. It may be used to reinforce the Bekker model or other improved terramechanics models [8, 28-29], so that the models can be used to calculate the wheel– soil interaction force for rover moving on rough terrain. This is a key issue of rovers’ control, which needs to be solved urgently. Acknowledgments This study was supported in part by the National Natural Science Foundation of China (Grant No. 61370033), Foundation of Chinese State Key Laboratory of Robotics and Systems (Grant No. SKLRS201501B, SKLRS20164B), National Basic Research Program of China (Grant No. 2013CB035502), the “111 Project” (Grant No. B07018) ， and Foundation for Innovative Research Groups of the Natural Science Foundation of China (Grant No. 51521003).

Reference [1] Bekker M G. Theory of land locomotion: the mechanics of vehicle mobility. The University of Michigan Press (1956). [2] Wong J Y. Theory of Ground Vehicles. Wiley (1978). [3] Ishigami G, Miwa A, Nagatani K, Yoshida K. Terramechanics-based model for steering maneuver of planetary

exploration rovers on loose soil. Journal of Field Robotics. 24(3) (2007) 233–250. [4] Petit C. Roving about the Red Planet. U.s.news & World Report (2004). [5] Iagnemma K, Kang S, Brooks C, Dubowsky S. Multi sensor terrain estimation for planetary rovers. Proc.of Intl.symposium on Artificial Intelligence Robotics & Automation in Space (2003). [6] Kang, Shinwoo. Terrain parameter estimation and traversability assessment for mobile robots. Massachusetts Institute of Technology (2003). [7] Ishigami G, Miwa A, Yoshida K. Steering trajectory analysis of planetary exploration rovers based on All-Wheel Dynamics Model[J]. Proc. of the 8th Int. Symp. on Artificial Intelligence, Robotics and Automation in Space, 603 (2005). [8] Ding L, Deng Zq, Gao Hb, Guo Jl, Zhang Dp, Iagnemma K. Experimental study and analysis of the wheels' steering mechanics for planetary exploration wheeled mobile robots moving on deformable terrain. International Journal of Robotics Research. 32(6) (2013) 712-743. [9] Yoshida K, Hamano H. Motion dynamics of a rover with slip-based traction model. Proc of the IEEE International Conference on Robotics & Automation, IEEE 2002, pp.3155 – 3160. [10] Ding L, Yoshida K, Nagatani K, Gao Hb, Deng Zq. Parameter identification for planetary soil based on a decoupled analytical wheel–soil interaction terramechanics model. The 2009 IEEE/RSJ International Conference on Intelligent Robots and Systems, 2009, pp. 4122-4127 [11] Brooks C A, Iagnemma K. Self–supervised terrain classification for planetary surface exploration rovers. Journal of Field Robotics. 29(3) (2012) 445-468. [12] Rigelsford J. Mobile robots in rough terrain: estimation, motion planning, and control with application to planetary rovers. Ppi Pulp & Paper International. 31(5) (2013) 459-459. [13] Wilcox B H. Non-geometric hazard detection for a Mars microrover. in Proc. AIAA Conf. Intelligent Robotics in Field, Factory, Service, and Space, 1994, pp. 675–684. [14] Meirion-Griffith G, Spenko M. A modified pressure-sinkage model for small, rigid wheels on deformable terrains. Journal of Terramechanics. 48(2) (2011) 149–155. [15] Ding L, Gao HB, Deng ZQ, Tao Jg. Wheel slip-sinkage and its prediction model of lunar rover. Journal of Central South University of Technology. 17(1) (2010) 129-135. [16] Reina G, Ojeda L, Milella A, Borenstein J. Wheel slippage and sinkage detection for planetary rovers. Mechatronics IEEE/ASME Transactions on. 11(2) (2006) 185-195. [17] Reina G, Milella A, Panella F W. Vision-based wheel sinkage estimation for rough-terrain mobile robots. Mechatronics & Machine Vision in Practice. m2vip. International Conference on, 2008, pp. 75 – 80. [18] Brooks C A, Iagnemma K and Dubowsky S. Visual wheel sinkage measurement for planetary rover mobility characterization. Autonomous Robotics. 21 (1) (2006) 55-64. [19] Liu Bing, Wang Liang, Cui Pingyuan. Visual wheel sinkage measurement for a lunar rover. Computer Measurement & Control. 16(12) (2008) 1809-1810. [20] Wang L, Dai X B, He-Hua J U. Homography-based visual measurement of wheel sinkage for a lunar rover. Journal of Astronautics. 32(8) (2011) 3543-3548. [21] Hegde G P, Robinson C J, Ye C, Stroupe A, Tunstel E. Computer vision based wheel sinkage detection for robotic lunar exploration tasks. Mechatronics & Automation International Conference on, 2010, pp. 1777 – 1782 [22] Hegde G M, Ye C, Robinson C A, Stroupe A, Tunstel E. Computer-vision-based wheel sinkage estimation for robot navigation on lunar terrain. IEEE/ASME Transactions on

Mechatronics, 18(4) (2013) 1346-1356. [23] Spiteri C, Al-Milli S, Gao Y, León ASD. Real-time visual sinkage detection for planetary rovers. Robotics & Autonomous Systems, 72(C) (2015) 307-317. [24] Al-Milli S, Spiteri C, Comin F, Gao Y. Real-time vision based dynamic sinkage detection for exploration rovers. International Conference on Intelligent Robots and System. 2013, pp. 4675-4680. [25] Ding L, Gao Hb, Deng Zq, Guo Jl, Liu Gj. Longitudinal slip versus skid of planetary rovers' wheels traversing on deformable slopes. 2013 IEEE/RSJ International Conference on Intelligent Robots and Systems, 2013, pp. 2842-2848 [26] Zhang Z Y, Deriehe R,Faugeras O. A robust technique for matching two uncalibrated images thorugh the recovery of the unknown epipolar geometry. Artificial Intelligence. 78(1-2) (1995) 87-119. [27] Ding L, Deng Zq, Gao Hb, Tao Jg, Iagnemma K D, Liu Gj. Interaction mechanics model for rigid driving wheels of planetary rovers moving on sandy terrain considering multiple effects. Journal of Field Robotics, (2013) 1-56. [28] Azimi A, Kövecses J, Angeles J. Wheel–soil interaction model for rover simulation and analysis using elastoplasticity theory. IEEE Transactions on Robotics, 29(5) (2013) 1271-1288. [29] Skonieczny K, Moreland S J, Asnani V M, Creager C M, Inotsume H, Wettergreen D S. Visualizing and analyzing machine-soil interactions using computer vision. Journal of Field Robotics, 31(5) (2014) 820–836.

ntrol. mecchanism and con Haaibo Gao, bornn in 1970, is cu urrently a profe fessor annd a PhD candidate supervisorr at Harbin Insttitute off Technology, China. C He receiv ved his PhD deegree m Harbin Instituute of in mechanical enngineering from Teechnology in 20003. His researrch interests incclude aerospace mechhanism and conntrol. i a student inn the Feengtian Lv, boorn in 1990, is Scchool of Mechaatronics Engineeering of the Haarbin Insstitute of Technnology. He received his B.S. deegree froom the Harbin Institute of Teechnology in 22013, annd his M.S. deggree from the Harbin Instituute of Technology inn 2015. His research interests include rrover machine visionn, terrain classsification, and their applicatioon to improve rover mobility. Baaofeng Yuan, born b in 1979, is i currently a seenior enngineer at Chinna Academy off Space Technoology, Chhina. His researrch interests incclude space struucture meechanism and sppace intelligentt machine.

Naan li, born in 1986, is currently a PhD annd an enngineer at harbiin Institute of Technology, C China. Hee received his M.S. M degree in pattern recognnition annd intelligent syystem from Haarbin Universitty of Sccience and Teechnology, Chiina, in 2012. His research interessts include roveer machine visio on, rover moveement state estimationn.

urrently a profe fessor Liiang Ding, bornn in 1980, is cu annd a PhD candiidate supervisorr at harbin Insttitute off Technology, China. C He receiv ved his PhD deegree m Harbin Instituute of in mechanical enngineering from Teechnology, Chinna, in 2010. Hiis research inteerests include mechannics, control annd simulation off mobile robots.. Niingxi Li, born in 1995, is currrently a B.S. inn the Scchool of Mechaatronics Engineeering of the Haarbin Insstitute of Tecchnology. Her research inteerests incclude terramechhanics.

uangjun Liu, is currently a Professor andd the Gu Caanada Researchh Chair in Co ontrol Systemss and Roobotics in thhe Departmen nt of Aerosspace Enngineering, Ryyerson Univerrsity, Toronto. He recceived the Ph.D D. degree from m the Universitty of Tooronto, Torontoo, ON, Canaada, in 1996. His research interrests include control systeems and robootics, particularly inn modular and reconfigurab ble robots, m mobile manipulators, aand aircraft systtems. Zoongquan Dengg, born in 19 956, is currenttly a proofessor and a PhD P candidate supervisor s at Haarbin Insstitute of Technology, China. His reseearch intterests include special robottics and aerosspace