- Email: [email protected]
Posture prediction and physics-based human motion simulation
Chapter 31
Posture prediction and physics-based human motion simulation Rajan Bhatt, Kimberly Farrell, Karim Abdel-Malek, Jasbir Arora and Chris Murphy The University of Iowa, Iowa City, IA, United States
1. Introduction There have been many advances in the field of digital human modeling and simulation. Many tools are now available for modeling digital humans within virtual environments to facilitate designing products, evaluating capabilities, training workers and warfighters, predicting injuries, and simulating real-world scenarios. In this chapter, we introduce methods that allow a digital human to interact with objects and the environment in a realistic, scenario-specific manner. Human behavior changes in response to changes in the objects and environment that define the scenario. For instance, our walk is different when we carry a heavy backpack on our back than when we carry the same backpack in two hands in front of us. The same walk will be different if the backpack weight is changed or the speed at which we walk is changed. In addition, there is a limit, for backpack weight and for speed, beyond which we cannot walk while carrying the backpack. To realistically simulate these scenarios, we need physics-based models that account for human limitations, external loads, and environmental factors. Many approaches have attempted to model physics in humans with the aim of realistically simulating humans performing various tasks and activities. Most approaches rely heavily on motion capture technology. A limiting factor of motion capture technology when used to simulate human motion is that it fails to account for individual variation; in reality, motion varies depending on individual body characteristics (body segment lengths, bulk because of muscle or fat, strength, etc.) (Borno et al., 2018). Another issue is the need to capture additional motion as the conditions of the scenario change, which can be time-consuming, costly, and limited to scenarios that can be replicated in the motion capture environment. For instance, the motion capture of the task of sitting in a chair will not work for another chair with different properties such as seat height, seat depth, presence of back support, and armrest width and height (Andrews et al., 2016). Not only must a new motion capture experiment be performed, but also a new chair must be built with the desired properties. While these approaches are well-suited for specific applications, they cannot be employed to simulate many scenarios. Santos, a human modeling and simulation environment developed at the University of Iowa’s Virtual Soldier Research (VSR) program, uses an optimization-based approach to simulate human motion. This approach allows for predictive capabilities based on changes in internal parameters, such as anthropometry, and external parameters, such as load carriage and obstacles in the environment. Posture prediction (PP) uses optimization to determine the most natural human pose for a given static scenario, where the motion to and from the pose is not considered. For instance, PP can be used to predict the pose an individual would use to hold a handbrake in a vehicle while sitting with one foot on the accelerator and the other resting on the floor. It can also be applied to predict the pose in that same scenario for a human with very different anthropometry. By calculating physics within the PP problem, we can also estimate the maximum force that can be generated in that pose by a human with certain strength characteristics. A similar predictive optimization-based approach can be used to simulate human motion over time. This approach to human motion prediction is referred to as predictive dynamics (PD). For instance, if we are interested in learning about the forces required throughout the handbrake engagement process and the corresponding joint torques necessary for the human to generate these forces, the problem can be solved with PD. The next section focuses on our high-fidelity digital human model and the mathematical structure that enables the optimization-based framework for both PP and PD.
DHM and Posturography. https://doi.org/10.1016/B978-0-12-816713-7.00031-3 Copyright © 2019 Elsevier Inc. All rights reserved.
425
426 PART | VI Activities of daily living
2. Digital human model Methods for simulating human motion are largely dependent on the underlying model used to represent the human skeletal system. For the approach presented in this chapter, the human skeleton is modeled as a series of rigid links connected by kinematic joints to represent the human skeletal system. The kinematic representation of the digital human, modeled as a three-dimensional, 215 degree- of- freedom, rigid-link, articulated mechanical structure is based on the Denavite Hartenberg (DH) parameterization (Denavit & Hartenberg, 1955). The DH parameterization is a matrix transformation method to systematically describe the translational and rotational relationship between adjacent reference frames in an articulated chain. Kinematically, the digital human is a branched structure with seven branches. The first branch contains the six global degrees of freedom (DOFs) (three translational and three rotational) that reference the location of the hip point on the digital human with respect to an inertial reference frame located on the ground. The other six branches correspond to the right leg, left leg, spine, right hand, left hand, and head. One benefit of the DH method is that it can efficiently represent the transformation from one DOF frame to the next in the kinematic chain using four parameters (Farrell, 2005) instead of the usual six parameters. This reduced parameterization is achieved by imposing certain restrictions on how the successive 3D frames are arranged. Another benefit of DH parameterization is its ability to be implemented iteratively. Thus, for large and computationally intensive systems, such as humans, it is relatively easy to develop a model that is suitable for computer implementation. Denavit-Hartenberg notation requires a single DOF between two consecutive links. Human joints with multiple DOFs, such as spines and hips, are modeled as collocated DH joints to adhere to the DH notation. Hence, for any joint in the human body that has more than one DOF, one or more virtual links with zero length are inserted between two consecutive joints. For instance, one virtual link is inserted between the two joints in the ankle, and two virtual links are inserted between the three collocated joints in the hips, as shown in Fig. 31.1. The mass and inertia properties of each body segment are estimated based on the Air Force Research Lab’s Articulated Total Body software for any avatar with given height and weight (Cheng et al., 1996). The default human model is a 50th percentile male with a body weight of about 770 N and a height of about 1.7 m (Penmatsa et al., 2009).
3. Recursive kinematics and dynamics An efficient calculation of kinematics and dynamics is vital to generating physically accurate digital human simulations. Equations of motion (EOMs) are developed using computationally efficient recursive Lagrangian dynamics. The recursive kinematics calculations lead to a simpler form for the transformation matrix Ai: Ai ¼ T1 T2 T3 :::Ti1 Ti ¼ Ai1 Ti th
th
(31.1)
st
where Ti is the i link transformation that relates the i and i-1 local reference frames and is expressed as 3 2 cos qi cos ai sin qi sin ai sin qi ai cos qi 6 sin q cos ai cos qi sin ai cos qi ai sin qi 7 i 7 6 Ti ¼ 6 7 4 0 sin ai cos ai di 5 0 0 0 1
(31.2)
Any point of interest in the ith frame, ix, can be expressed in the global reference frame, 0x, using these parameters as 0
x ¼ Ai i x
(31.3)
The above equation can be used to calculate the global position of a point on the body for any given posture, such as a contact point or a body-based obstacle sphere defined with respect to a frame in the kinematic human model. These calculations can be used to define the objective functions and constraints in the optimization-based PP problem, as explained later in the chapter. The DH parameters qi, di, ai, and ai that define the transformation matrix Ti are explained in Fig. 31.2. For each frame, the ith z-axis controls the motion for the (i þ 1)th DOF. Farrell (2005) describes the four parameters of the method that are used to obtain the position and orientation of frame i with respect to frame i e 1, as follows: Angle qi between the (i-1)th and ith x-axis about the (i-1)th z-axis. Distance di from the (i-1)th to the ith x-axis along the (i-1)th z-axis. Angle ai between the (i-1)th and ith z-axis about the ith x-axis. Distance ai from the (i-1)th to the ith x-axis along the ith x-axis.
Posture prediction and physics-based human motion simulation Chapter | 31
427
FIGURE 31.1 Rigid-link mechanical structure of a digital human with collocated multiple joints. The local reference frames are attached to all the degrees of freedom of the model based on DenaviteHartenberg parameterization.
FIGURE 31.2
DenaviteHartenberg parameters used in modeling a kinematic representation of a digital human.
428 PART | VI Activities of daily living
Time derivatives of the transformation matrix Ai are necessary to calculate the dynamics and can also be obtained in the recursive form as vTi Bi ¼ A_ i ¼ Bi1 Ti þ Ai1 q_i vqi
(31.4)
vTi v2 Ti vTi Ci ¼ B_ i ¼ Ci1 Ti þ 2Bi1 q_i þ Ai1 2 q_2i þ Ai1 q€i vqi vqi vqi
(31.5)
where qi is the generalized coordinate for transformation Ti,A0 ¼ I, and B0¼C0¼0. The Lagrange’s equation is given as d vL vL si ¼ dt vq_i vqi
(31.6)
where L ¼ K e V (kinetic energy e potential energy), q is the generalized coordinate vector (joint angles), and si is the joint torque vector. Given the mass and inertia properties of each body segment and the external force f Tk ¼ ½ fx fy fz 0 and moment hTk ¼ ½ hx hy hz 0 for the link k (1 k n) defined in the global coordinate system, the joint actuation torques si for each of the joints can be computed using recursive backward dynamics as follows: vAi vAi vAi D i gT Ei f Tk Fi GTi Ai1 z0 (31.7) si ¼ tr vqi vqi vqi where Di ¼ Ji CTi þ Tiþ1 Diþ1
(31.7a)
Ei ¼ mi i r i þ Tiþ1 Eiþ1
(31.7b)
F i ¼ k rf dik þ Tiþ1 Fiþ1
(31.7c)
G i ¼ hk dik þ Giþ1
(31.7d)
Dnþ1 ¼ Enþ1 ¼ Fnþ1 ¼ 0 i
(31.7e) k
where g is the gravity vector, ri is the location of the center of mass in the ith local frame, rf is the location of the external force acting in the kth frame, and dik is the Kronecker delta. The gradients of the EOMs are also required for faster implementation of gradient-based optimization methods. These gradients with respect to joint angles, joint angle velocities, and joint angle accelerations can also be analytically calculated (Xiang et al., 2009).
4. Optimization The EOMs must be solved to determine the torques required to perform any given task. Many different approaches are available in the literature, each with its own advantages and disadvantages (Otten, 2003). However, these traditional approaches are not suitable for use with human posture and motion simulation. While these approaches work well with open-chain systems with relatively fewer DOFs, solving the EOMs of a high-fidelity human model as described in this chapter is a very time-consuming and computationally intensive task. In addition, most tasks would require the human model to alternate between closed-loop and open-chain systems. Traditional EOM solvers are also not well suited for applications where there are intermediate constraints on the state of the variables. For instance, to simulate the walking task, the initial and final foot placement is typically constrained to be on the floor. In addition, there might be intermediate constraints on the hands to simulate an activity being performed by the human while walking, such as carrying a box. Hence, we employ an optimization-based strategy to estimate the solution of a large, interdependent system of EOMs as described in this section. This approach also allows us to expand the modeling capabilities by adding more components to the objective function and constraints as the research matures. A typical optimization problem has three main components: design variables, an objective function, and constraints (Arora, 2016). Design variables are the parameters of the problem that can change and the values of which need to be determined. The objective function, a function of design variables, is a measure that must be minimized. For the problem of human modeling and simulation, we use one or many performance measures as the objective function. The constraints are
Posture prediction and physics-based human motion simulation Chapter | 31
429
the bounds on the design variables or the bounds on some function of the design variables that must not be exceeded. Below, we describe these components for a typical human posture and motion simulation problem.
5. Design variables Joint angles, qi, are the unknowns that must be determined in most human posture and simulation problems. For PP problems, the joint angles are the design variables. When the problem involves a time component, as in PD, joint angle profiles qi(t) are approximated as linear combinations of cubic B-spline basis functions. In this case, the control points of these cubic B-spline functions serve as the design variables. Corresponding joint angle, velocity, and acceleration values are calculated at each iteration from these control point values.
6. Performance measure The goal of the optimization process is to minimize/maximize some performance criteria. The performance criteria typically represent real-world driving factors for human posture and motion. For example, human posture can be driven in part to minimize discomfort in a given pose, or a human might choose a posture that makes it possible to acquire a visual target. In optimization-based human simulation, these performance criteria are modeled as mathematical functions of the design variables that are minimized to guide the solution. Some common performance measures used in PP are described below.
7. Joint displacement In PP, joint displacement is used to represent the difference between a final posture and a neutral posture as shown in Fig. 31.3 (Farrell, 2005). The neutral pose qN represents the joint angles in a relatively comfortable position. For example, a typical neutral pose is standing straight with the arms at the side (N-pose). The joint displacement performance measure is defined to minimize the distance between the final joint angle and the corresponding neutral angle (Eq. 31.8). To avoid nondifferentiability, the squared difference is used in place of absolute difference. In addition, a weighted sum is used to emphasize the stronger influence of certain joints over others (Abdel-Malek & Arora, 2013). min
f ðqÞ ¼
n X
2 wi qi qNi
(31.8)
i¼1
where qi is the ith joint angle in the final posture, qNi is the ith joint angle in the neutral posture, wi is the weighting factor for the ith joint, and n is the number of joint angles.
FIGURE 31.3 Joint displacement represents the difference between the final posture and the neutral posture.
430 PART | VI Activities of daily living
8. Joint discomfort While joint displacement focuses directly on minimizing the difference from a comfortable posture, joint discomfort expands that concept to include penalty terms designed to mathematically represent the additional factors in musculoskeletal discomfort. As described by Abdel-Malek and Arora (2013), these factors are incorporated into the joint discomfort performance measure: (1) the tendency to move different segments of the body sequentially, (2) the tendency to gravitate to a reasonably comfortable neutral position, and (3) the discomfort associated with moving when joints are near their respective limits. The second factor (i.e., gravitating to a neutral position) is accounted for in the joint displacement formulation (Eq. 31.8). To accommodate the tendency to move body segments sequentially, the difference in joint angle is normalized by the total range of the joint, and the weighting factors are substantially increased to discourage movement in those body parts that should move last (e.g., the clavicles). To represent discomfort near the joint limits, penalty terms are added to the performance measure to spike the function value as the limit is approached (Marler et al., 2005). Mathematically, the resulting function is defined as n 1 X qi qNi min f ðqÞ ¼ þ G QL wi U þ G QU (31.9) i i G i¼1 qi qLi U 100 5:0 qi qi þ 1:571 þ 1 QUi ¼ 0:5sin qUi qLi 100 5:0 qi qLi QLi ¼ 0:5sin þ 1:571 þ 1 qUi qLi
(31.10)
(31.11)
where GQUi is the penalty term for joint angles that near their upper limits and GQLi is the penalty term for joint angles that near their lower limits.
9. Vision A common driving factor for static posture is one’s ability to see a given target. For example, when reaching to engage the handbrake in a vehicle, a human would typically use a pose in which the handbrake is visible. To simulate this behavior in PP, vision is modeled mathematically as a performance measure for the optimization problem. Specifically, visual acuity, or the capacity to discriminate the details of an object in the field of view, is formulated into a continuous objective function that behaves well in optimization (Eq. 31.12) (Smith et al., 2009). To this end, the vision vector estimates the line of sight from a point between the eyes relative to the last frame of the head/neck system, while the target vector calculates the direction from the point between the eyes to the target as shown in Fig. 31.4. min
f ðqÞ ¼ e35ð1cos½qðqÞÞ þ 1
(31.12)
where q(q) is the angle of sight between the vision vector and the target vector, dependent on the current posture q. While this performance measure closely represents the visual acuity as described in the literature, it is not well behaved in the gradient-based optimization problem in practice, especially when combined with other performance measures. This is because of a tendency of the curve to flatten (i.e., near-zero gradient) for many potential values of the angle of sight q(q) as shown in Fig. 31.5.
FIGURE 31.4 Head vector representing the line of sight relative to the target.
Posture prediction and physics-based human motion simulation Chapter | 31
FIGURE 31.5
431
The visual acuity performance measure (Eq. 31.12) has significant portions with near-zero gradient.
FIGURE 31.6 The visual displacement performance measure (Eq. 31.13).
FIGURE 31.7 Predicted postures on Santos using (A) visual acuity, (B) visual acuity combined with joint displacement, (C) visual displacement, and (D) visual displacement combined with joint displacement.
Smith et al. (2009) proposed a more intuitive visual displacement performance measure for driving posture with vision that relies directly on minimizing the angle of sight (Eq. 31.13). Visual displacement drives the posture to see the target and still combines well with other performance measures because of its curve, as shown in Fig. 31.6. qðqÞ min f ðqÞ ¼ 10 1 cos (31.13) 2 These vision-based performance measures are compared on Santos in Fig. 31.7. They are used both as the only objective function and in combination with the joint displacement performance measure described above. Further development of vision-based human performance measures, including eye angles, has been formulated by Knake et al. (2010).
432 PART | VI Activities of daily living
10. Joint torque The most common performance measure for PD is the dynamic effort at each joint. The performance measure (or objective function) is, therefore, to minimize the sum of the torques squared for all joints over the simulation time. min f ðqÞ ¼
ZT X n t¼0
s2i ðqÞdt
(31.14)
i¼1
where n is the total number of joints of the human model, si is the actuator torque of the ith joint, and T is the total simulation time. Joint actuation torques are calculated using the recursive EulereLagrangian formulation as a function of joint angles, velocities, and accelerations.
11. Constraints Several physics-based, task-based, and environment-based constraints must be employed to predict the postures and motions. The physics-based constraints typically limit the capabilities of the human model to avoid unrealistic or impossible situations such as lifting a heavy table with one finger. Joint angle limits, joint torque limits, self-avoidance, and stability criteria are examples of these physics-based constraints. Task-based constraints are dependent on the task being simulated. For instance, it is necessary for the hand to touch the handbrake in a specific location at a particular angle. The environment-based constraints depend on the task and the environment surrounding the human in which the task must be simulated. For instance, the feet must never go below the floor, and body segments cannot go into obstacles or walls. A few common constraints for the predictive algorithms are discussed in the following sections.
12. Distance The most common constraint requires the distance between a point on the body and a global position to be within a sufficiently small tolerance. For example, this can be used to constrain a hand to a grip or an elbow to an armrest. The formulation for the distance constraint is xðqÞlocal xglobal ε (31.15) where x(q)local is a local position in the ith frame whose global position is determined by the current posture q using Eq. (31.3). This distance constraint can be used for multiple points on the body (Farrell, 2005) and in tandem with other constraints as shown in Fig. 31.8.
13. Vision In certain scenarios using static postures, acquiring a particular visual target might be required rather than desired. For example, analyzing a seating posture in a vehicle might require a posture in which a human can see the rear-view mirror or a target in front of the cab, as shown in Fig. 31.9. In this case, the visual displacement performance measure in Eq. (31.13) can be reformulated and used as a constraint (Eq. 31.16). Furthermore, the visual displacement constraint can be combined with obstacle avoidance constraints to avoid obstructions to vision by obstacles in the line of sight (Knake et al., 2010). qðqÞ 10 1 cos ε (31.16) 2
FIGURE 31.8 Result from on Santos using multiple distance constraints on the body.
Posture prediction and physics-based human motion simulation Chapter | 31
433
FIGURE 31.9 Predicted posture using visual displacement as a constraint on Santos.
14. Self-avoidance Using the digital human model described in Fig. 31.1, there is no mathematical reason for the predictive solution to avoid colliding with its own body. To ensure a realistic simulation of posture and motion, constraints must be added to prevent self-intersection. To this end, the body of the digital human is filled with spheres, which are added to the kinematic chain and grouped according to segment to distinguish between the torso and limbs, as shown in Fig. 31.10 (Degenhardt et al., 2014; Farrell et al., 2016). The spheres in one group are algorithmically checked against the spheres in all other groups (Johnson et al., 2010). The use of spheres allows a computationally efficient approach that simply requires the distance between checked spheres to be greater than the sum of their radii. To ensure that the constraints are differentiable, the constraint effectively models the same behavior by restricting the distance squared to be greater than the squared sum of the radii (Eq. 31.17). 2
ðri þ rj Þ kxðqÞ i xðqÞ j k 0 2
s
s
FIGURE 31.10 Human model filled with spheres grouped by segment, shown here in a different color for each group.
(31.17)
434 PART | VI Activities of daily living
Note that this method adds one constraint to the optimization problem for each pair of spheres that need to be checked. Methods to address computational efficiency by reducing the number of active constraints are addressed in the literature (Johnson et al., 2009, Johnson et al., 2010). Constraints allow flexibility in optimization-based human simulation. Additional constraints have been and can be modeled to expand the current capabilities. The next sections discuss in more detail the benefits of using these optimization-based methods to simulate human motion through PP and PD.
15. Posture prediction Postures are predicted by solving an optimization problem to minimize an objective function, or more specifically one of a combination of human performance measures, as described above. Depending on the task, one or more of the performance measures are employed to predict the desired posture. In addition, many constraints can be added and combined to represent the real-world restrictions in a given scenario, such as contact with the environment or objects. This flexibility makes it possible to simulate a specific scenario across varying anthropometry. For example, PP can be used to analyze differences in posture and reach on a motorcycle for both a taller male and a shorter female while ensuring that contact with the handlebars is maintained (see Fig. 31.11). The capabilities of PP can be increased as the need arises. For instance, if there are any obstacles to be avoided in the environment while trying to reach a point in space, an obstacle avoidance constraint can be added (Johnson et al., 2009) (see Fig. 31.12). Additional constraints can also be used to account for obstacles that obstruct the line of sight (Knake et al., 2010) and obstacles that represent equipment attached to the body (Farrell et al., 2016) (see Fig. 31.13).
FIGURE 31.11 Posture prediction used to analyze different anthropometries (A) male and (B) female on a motorcycle with hand and foot contact constraints.
FIGURE 31.12 Using posture prediction to simulate a reach around heavy machinery with (A) no obstacle awareness and (B) obstacle avoidance using constraints.
Posture prediction and physics-based human motion simulation Chapter | 31
435
FIGURE 31.13 Equipment attached to the body as obstacles (A) before and (B) after additional constraints in posture prediction.
16. External forces In some situations, external forces are present as the human performs the task. These external forces can also be accounted for in PP to generate realistic postures (Liu et al., 2009). The predictions are purely static in nature and assume there is no velocity or acceleration of movement. This can be used, for example, to predict postures that exert a push or pull, as shown in Fig. 31.14, or postures that require holding tools and applying a force while performing the task.
17. Motion capture processing Furthermore, PP can be extended to automatically generate digital human motion from motion capture in real time by solving the optimization problem for a sequence of frames (Seydel et al., 2017). To this end, virtual markers are placed on the digital human model corresponding to the motion capture markers as shown in Fig. 31.15. A distance performance measure derived from the distance constraint formulation in Eq. (31.15) is then used to minimize the distance between a virtual marker and its corresponding motion capture marker for the current frame. Processing motion capture using optimization-based PP has the additional benefit that it is compatible with any of the performance measures and constraints described earlier. For example, it can be used with vision constraints, obstacle avoidance, or force calculations, providing a wealth of analytical information as shown in Fig. 31.16.
FIGURE 31.14 Posture prediction with forces applied to Santos to analyze (A) hold, (B) push, and (C) pull.
436 PART | VI Activities of daily living
FIGURE 31.15 Motion capture markers on the human subject and the digital human model: (A) motion capture markers, (B) virtual markers, and (C) minimized distances.
FIGURE 31.16 Motion capture using posture prediction on Santos with force calculations and associated joint torques.
18. Predictive dynamics While PP predicts realistic postures of humans performing various tasks in the presence of external forces, obstacles, and other physical and environmental limitations, PD predicts the full motion of a task within the bounds of physical and environmental limitations. Posture prediction can predict and analyze a task or a posture while accounting for static forces, but there are many tasks where a dynamic motion analysis becomes necessary. For instance, a vertical jumping task with and without carrying loads or a box lifting and placement task requires a full dynamic analysis to learn more about the requirements of the task on the human body. As discussed in the optimization section, the goal of PD is essentially finding the optimal joint angle profiles for all the joints in the human body to achieve the task objective while complying with physical and environmental limitations. In a single joint angle profile curve, there are infinite points, and thus the problem defined above becomes infinite and cannot be solved using optimization. The problem needs to be parameterized, and there are multiple ways to go about it. For example, if we discretize the problem at every fixed time interval, e.g., every 0.1 s, and for every joint, the solution is a vector of 10 values per second, and the continuous motion is generated by interpolating these values. Another option is to parameterize a curve using B-splines so the control points for these B-splines become the design variables. Joint angle values, velocities, and accelerations can be derived from these B-splines. Santos uses cubic B-splines parameterization rather than time discretization or other methods. The use of B-spline interpolation inherently constrains each joint profile to be continuous and differentiable, which is what typical human motion looks like.
Posture prediction and physics-based human motion simulation Chapter | 31
437
The complexity and length of the task to be predicted typically dictates the number of control points. As a rule of thumb, the number of control points, and thus the number of design variables for the optimization problem, increases as the task being simulated gets longer. For instance, a step simulation of a task would require fewer control points than a stride simulation of the same task. However, the computational requirements of the optimization problem, and thus the time necessary to solve the problem, also increase substantially as the number of control points increases. Hence, for every task, an optimal balance of the number of control points must be chosen that allows for the variations of the task to be predicted while minimizing the computational time requirements. Many tasks have been developed using PD, including walking (Xiang, Chung et al., 2010), walking backward (Kwon et al., 2014), running (Chung et al., 2015), jumping (Hariri et al., 2013), stair-ascending (Bhatt et al., 2008), box lifting (Xiang et al., 2010), and ingress-egress (Kwon, 2011). Predictive dynamics uses a gradient-based optimization algorithm to solve the human motion prediction problem (Xiang et al., 2009). The solver time reduces drastically because the analytical constraints for each objective function and constraint function are calculated analytically. Predictive Dynamics also leverages motion capture data for a specific task to predict natural task motions using fewer constraints (Xiang et al., 2012). In the following paragraphs, we will discuss the vertical jump task and the insights gained using PD. The vertical jump task is defined as predicting the maximum height that the tip of the right hand can reach from the floor. A Santos avatar 1.82 m tall and weighing 174 lbs was used to predict the maximum height that the avatar can reach while being loaded under different conditions. Santos was able to achieve an unrealistic maximum height as high as 3.2 m when only the isokinetic strength limits were applied. The predictions presented in this chapter were accomplished using dynamic strength limits (Marler et al., 2012), where the peak torque a human can produce at any given angle and velocity is derived from a strength surface equation. The strength surface is derived by using isokinetic (constant velocity) and isometric (constant length) strength testing for all major joints of the human body. In addition to predicting the realistic maximum vertical jump height under various loadings for Santos (see Fig. 31.17), PD also provides a lot of insight on the jump mechanism and how the motion changes because of the dynamic strength constraints.
Additional load (in lbs) 9.83
13.44
20.49
42.62
93.64
2.31
2.13
1.99
Maximum height (in meters) 2.45
2.4
FIGURE 31.17 Maximum height predictions using a predictive dynamics vertical jump task with different loading conditions for the same avatar.
438 PART | VI Activities of daily living
FIGURE 31.18 Plot of right knee torque profile for the Santos vertical jump task with static and dynamic strength limits.
The jump height of 2.45 m is easily achievable when static strength limits are used. However, when dynamic strength limits are applied, the right knee torque becomes the limiting factor, as shown in Fig. 31.18. Also, the motion predicted for the same height of 2.45 m when static strength limits are used is different than the motion predicted when dynamic strength limits are used. Assuming the dynamic strength limits are only used for postprediction analysis, the motion would seem infeasible because the torque curve would breach the dynamic strength limits curve. However, when the motion is predicted by applying dynamic strength limits as a constraint, the algorithm predicts a different motion with different joint angle values and velocities so that it can achieve the 2.45 m height without breaching the constraints. Note that, as the motion changes, the corresponding joint angle and joint velocity valuesdand thus the dynamic strength limitsdalso change. Such analysis capabilities result only when the simulation task is predictive and full dynamic strength implementation is realized.
19. Future research Although the capabilities of PD have increased throughout the last decade with the addition of performance measures, the use of motion capture as a seed motion for a hybrid approach, and the addition of strength surfaces as an analysis tool and as constraints, a lot more can be done to achieve even better results in less time. The most important development could be a decrease in the time spent on the development of PD tasks and on solving a prediction problem once the task is developed. We are currently working on a modular approach to solving the optimization PD problem so that new tasks can be developed using an automated implementation. In addition, a few steps have been taken to increase the speed of solving each PD problem, such as using neural networks to identify a good starting point for the optimization solver based on task input parameters, performing a few computations in parallel by leveraging multicore processors, and optimizing the tolerances for the optimization problem. However, additional research is necessary in both automated task development and improved computation time to make PD occur in real time or at least near real time.
20. Conclusion The predictive capabilities of the human simulation software Santos have been in development since 2003 at the University of Iowa’s VSR program. Substantial effort also went to validating these predictive capabilities (Rahmatalla et al., 2011). These capabilities have been used extensively to answer questions in military and nonmilitary applications. Previously, human motion simulation relied heavily on motion capture experiments, and it was time-consuming and expensive to perform a new analysis when parameters such as weight or speed changed. Using the predictive tools in the
Posture prediction and physics-based human motion simulation Chapter | 31
439
Santos environment, PP and PD, it is easier to perform such trade-off analysis. Predictive capabilities provide a very good approximation of each case; however, testing of the final design or a few designs may be appropriate, depending on the importance of the task or its associated risk. While such predictive simulation capabilities do not eliminate the need for experimentation, they can drastically reduce the number of iterations required, which in turn reduces time and cost. This chapter describes the predictive capabilities of Santos while focusing on the unique insights that these predictive tools can provide to the end user.
References Abdel-Malek, K., & Arora, J. S. (2013). Human motion simulation: Predictive dynamics. Academic Press. Andrews, S., Huerta, I., Komura, T., Sigal, L., & Mitchell, K. (December 2016). Real-time physics-based motion capture with sparse sensors. In Proceedings of the 13th European conference on visual media production (pp. 1e10). Arora, J. S. (2016). Introduction to optimum design. Academic Press. Bhatt, R., Xiang, Y., Kim, J. H., Mathai, A., Penmatsa, R., Chung, H.-J., … Obuseck, J. (2008). Dynamic optimization of human stairs climbing motion. In Proceedings of SAE digital human modeling conference. Pittsburg, PA. Borno, M. A., Righetti, L., Black, M. J., Delp, S. L., Fiume, E., & Romero, J. (July 2018). Robust physics-based motion retargeting with realistic body shapes. ACM SIGGRAPH/Eurographics Symposium on Computer Animation 2018, 37(6), 1e12. Cheng, H., Obergefell, L., & Rizer, A. (1996). The development of the GEBOD program. In 1996 fifteenth southern biomedical engineering conference. Dayton, OH, USA: IEEE. Chung, H.-J., Xiang, Y., Arora, J., & Abdel-Malek, K. (2015). Dynamic optimization of human running with analytical gradients. Journal of Computational and Nonlinear Dynamics, 10(2), 1e18. Degenhardt, R. K., Farrell, K. D., Bhatt, R. M., & Abdel-Malek, K. (May 2014). Body-based obstacle avoidance for motion simulation. In Digital human modeling symposium. Tokyo, Japan. Denavit, J., & Hartenberg, R. S. (1955). A kinematic notation for lower-pair mechanisms based on matrices. Journal of Applied Mechanics, 77, 215e221. Farrell, K. (2005). Kinematic human modeling and simulation using optimization-based posture prediction. Iowa City: University of Iowa. M. S. thesis. Farrell, K., Degenhardt, R., Bhatt, R., & Abdel-Malek, K. (2016). Body-based obstacle avoidance for motion simulation using optimization-based posture prediction. In 4th international digital human modeling symposium. Montreal, Canada. Hariri, M., Xiang, Y., Chung, H.-J., Bhatt, R., Arora, J. S., & Abdel-Malek, K. (2013). Simulation and prediction of the motion of a human in a vertical jumping task. In ASME 2013 dynamic systems and control conference. Palo Alto, CA. Johnson, R., Fruehan, C., Schikore, M., & Marler, T. (2010). New developments with collision avoidance for posture prediction. In V. Duffy (Ed.), Advances in applied digital human modeling (pp. 336e346). CRC Press. Johnson, R., Smith, B. L., Penmatsa, R., Marler, T., & Abdel-Malek, K. (2009). Real-time obstacle avoidance for posture prediction. In SAE digital human modeling conference. Goteborg, Sweden. Knake, L., Mathai, A., Marler, T., Farrell, K., Johnson, R., & Abdel-Malek, K. (2010). New capabilities for vision-based posture prediction. In V. Duffy (Ed.), Advances in applied digital human modeling (pp. 628e638). CRC Press. Kwon, H. J. (2011). Simulating ingress and egress motion for heavy earthmoving machines. Iowa City: University of Iowa. Ph.D. thesis. Kwon, H.-J., Xiang, Y., Bhatt, R., Rahmatalla, S., Arora, J., & Abdel-Malek, K. (July 2014). Backward walking simulation of humans using optimization. Structural and Multidisciplinary Optimization, 50(1), 169e179. Liu, Q., Marler, T., Yang, J., & Kim, J. H. (2009). Posture prediction with external loads e a pilot study. SAE International Journal of Passenger Cars, 2(1), 1014e1023. Marler, T., Frey-Law, L., Mathai, A., Spurrier, K., Avin, K., & Cole, E. (2012). Integration of strength models with optimization-based posture predition. In V. G. Duffy (Ed.), Advances in applied human modeling and simulation (pp. 327e336). CRC Press. Marler, R. T., Rahmatalla, S., Shanahan, M., & Abdel-Malek, K. (2005). A new discomfort function for optimization-based posture prediction. In SAE human modeling for design and engineering conference, Iowa City, IA. Otten, E. (September 29, 2003). Inverse and forward dynamics: Models of multi-body systems. Philosophical Transactions of the Royal Society B: Biological Sciences, 358(1437), 1493e1500. Penmatsa, R., Bhatt, R., Farrell, K., Rochambeau, B., Fruehan, C., Verma, U., … Abdel-Malek, K. (2009). Estimation of mass and inertia properties of human body segments for physics-based human modeling and simulation applications. SAE International Journal of Passenger Cars, 2(1), 1626e1632. Rahmatalla, S., Xiang, Y., Li, J., Smith, R., Meusch, J., & Bhatt, R. (2011). A validation framework for predictive human models. International Journal of Human Factors Modelling and Simulation, 2(1e2), 67e84. Seydel, A. L., Farrell, K., Johnson, R., Marler, T., Bhatt, R. M., Rahmatalla, S., & Abdel-Malek, K. (2017). Improved motion capture processing for high fidelity human models using optimization-based prediction of posture and anthropometry. In International conference on applied human factors and ergonomics. Los Angeles, CA. Smith, B. L., Marler, T., & Abdel-Malek, K. (2009). Studying visibility as a constraint and as an objective for posture prediction. SAE International Journal of Passenger Cars - Mechanical Systems, 1(1), 1118e1124.
440 PART | VI Activities of daily living
Xiang, Y., Arora, J. S., & Abdel-Malek, K. (2009). Optimization-based motion prediction of mechanical systems: Sensitivity analysis. Structural and Multidisciplinary Optimization, 37(6), 595e608. Xiang, Y., Arora, J. S., & Abdel-Malek, K. (September 2012). Hybrid predictive dynamics: A new approach to simulate human motion. Multibody System Dynamics, 28(3), 199e224. Xiang, Y., Arora, J. S., Rahmatalla, S., Marler, T., Bhatt, R., & Abdel-Malek, K. (April 2010). Human lifting simulation using a multi-objective optimization approach. Multibody System Dynamics, 23(4), 431e451. Xiang, Y., Chung, H.-J., Kim, J. H., Bhatt, R. M., Rahmatalla, S., Yang, J., … Abdel-Malek, K. (April 2010). Predictive dynamics: An optimization-based novel approach for human motion simulation. Structural and Multidisciplinary Optimization, 41(3), 465e479.
Posture prediction and physics-based human motion simulation Rajan Bhatt, Kimberly Farrell, Karim Abdel-Malek, Jasbir Arora and Chris Murphy The University of Iowa, Iowa City, IA, United States
1. Introduction There have been many advances in the field of digital human modeling and simulation. Many tools are now available for modeling digital humans within virtual environments to facilitate designing products, evaluating capabilities, training workers and warfighters, predicting injuries, and simulating real-world scenarios. In this chapter, we introduce methods that allow a digital human to interact with objects and the environment in a realistic, scenario-specific manner. Human behavior changes in response to changes in the objects and environment that define the scenario. For instance, our walk is different when we carry a heavy backpack on our back than when we carry the same backpack in two hands in front of us. The same walk will be different if the backpack weight is changed or the speed at which we walk is changed. In addition, there is a limit, for backpack weight and for speed, beyond which we cannot walk while carrying the backpack. To realistically simulate these scenarios, we need physics-based models that account for human limitations, external loads, and environmental factors. Many approaches have attempted to model physics in humans with the aim of realistically simulating humans performing various tasks and activities. Most approaches rely heavily on motion capture technology. A limiting factor of motion capture technology when used to simulate human motion is that it fails to account for individual variation; in reality, motion varies depending on individual body characteristics (body segment lengths, bulk because of muscle or fat, strength, etc.) (Borno et al., 2018). Another issue is the need to capture additional motion as the conditions of the scenario change, which can be time-consuming, costly, and limited to scenarios that can be replicated in the motion capture environment. For instance, the motion capture of the task of sitting in a chair will not work for another chair with different properties such as seat height, seat depth, presence of back support, and armrest width and height (Andrews et al., 2016). Not only must a new motion capture experiment be performed, but also a new chair must be built with the desired properties. While these approaches are well-suited for specific applications, they cannot be employed to simulate many scenarios. Santos, a human modeling and simulation environment developed at the University of Iowa’s Virtual Soldier Research (VSR) program, uses an optimization-based approach to simulate human motion. This approach allows for predictive capabilities based on changes in internal parameters, such as anthropometry, and external parameters, such as load carriage and obstacles in the environment. Posture prediction (PP) uses optimization to determine the most natural human pose for a given static scenario, where the motion to and from the pose is not considered. For instance, PP can be used to predict the pose an individual would use to hold a handbrake in a vehicle while sitting with one foot on the accelerator and the other resting on the floor. It can also be applied to predict the pose in that same scenario for a human with very different anthropometry. By calculating physics within the PP problem, we can also estimate the maximum force that can be generated in that pose by a human with certain strength characteristics. A similar predictive optimization-based approach can be used to simulate human motion over time. This approach to human motion prediction is referred to as predictive dynamics (PD). For instance, if we are interested in learning about the forces required throughout the handbrake engagement process and the corresponding joint torques necessary for the human to generate these forces, the problem can be solved with PD. The next section focuses on our high-fidelity digital human model and the mathematical structure that enables the optimization-based framework for both PP and PD.
DHM and Posturography. https://doi.org/10.1016/B978-0-12-816713-7.00031-3 Copyright © 2019 Elsevier Inc. All rights reserved.
425
426 PART | VI Activities of daily living
2. Digital human model Methods for simulating human motion are largely dependent on the underlying model used to represent the human skeletal system. For the approach presented in this chapter, the human skeleton is modeled as a series of rigid links connected by kinematic joints to represent the human skeletal system. The kinematic representation of the digital human, modeled as a three-dimensional, 215 degree- of- freedom, rigid-link, articulated mechanical structure is based on the Denavite Hartenberg (DH) parameterization (Denavit & Hartenberg, 1955). The DH parameterization is a matrix transformation method to systematically describe the translational and rotational relationship between adjacent reference frames in an articulated chain. Kinematically, the digital human is a branched structure with seven branches. The first branch contains the six global degrees of freedom (DOFs) (three translational and three rotational) that reference the location of the hip point on the digital human with respect to an inertial reference frame located on the ground. The other six branches correspond to the right leg, left leg, spine, right hand, left hand, and head. One benefit of the DH method is that it can efficiently represent the transformation from one DOF frame to the next in the kinematic chain using four parameters (Farrell, 2005) instead of the usual six parameters. This reduced parameterization is achieved by imposing certain restrictions on how the successive 3D frames are arranged. Another benefit of DH parameterization is its ability to be implemented iteratively. Thus, for large and computationally intensive systems, such as humans, it is relatively easy to develop a model that is suitable for computer implementation. Denavit-Hartenberg notation requires a single DOF between two consecutive links. Human joints with multiple DOFs, such as spines and hips, are modeled as collocated DH joints to adhere to the DH notation. Hence, for any joint in the human body that has more than one DOF, one or more virtual links with zero length are inserted between two consecutive joints. For instance, one virtual link is inserted between the two joints in the ankle, and two virtual links are inserted between the three collocated joints in the hips, as shown in Fig. 31.1. The mass and inertia properties of each body segment are estimated based on the Air Force Research Lab’s Articulated Total Body software for any avatar with given height and weight (Cheng et al., 1996). The default human model is a 50th percentile male with a body weight of about 770 N and a height of about 1.7 m (Penmatsa et al., 2009).
3. Recursive kinematics and dynamics An efficient calculation of kinematics and dynamics is vital to generating physically accurate digital human simulations. Equations of motion (EOMs) are developed using computationally efficient recursive Lagrangian dynamics. The recursive kinematics calculations lead to a simpler form for the transformation matrix Ai: Ai ¼ T1 T2 T3 :::Ti1 Ti ¼ Ai1 Ti th
th
(31.1)
st
where Ti is the i link transformation that relates the i and i-1 local reference frames and is expressed as 3 2 cos qi cos ai sin qi sin ai sin qi ai cos qi 6 sin q cos ai cos qi sin ai cos qi ai sin qi 7 i 7 6 Ti ¼ 6 7 4 0 sin ai cos ai di 5 0 0 0 1
(31.2)
Any point of interest in the ith frame, ix, can be expressed in the global reference frame, 0x, using these parameters as 0
x ¼ Ai i x
(31.3)
The above equation can be used to calculate the global position of a point on the body for any given posture, such as a contact point or a body-based obstacle sphere defined with respect to a frame in the kinematic human model. These calculations can be used to define the objective functions and constraints in the optimization-based PP problem, as explained later in the chapter. The DH parameters qi, di, ai, and ai that define the transformation matrix Ti are explained in Fig. 31.2. For each frame, the ith z-axis controls the motion for the (i þ 1)th DOF. Farrell (2005) describes the four parameters of the method that are used to obtain the position and orientation of frame i with respect to frame i e 1, as follows: Angle qi between the (i-1)th and ith x-axis about the (i-1)th z-axis. Distance di from the (i-1)th to the ith x-axis along the (i-1)th z-axis. Angle ai between the (i-1)th and ith z-axis about the ith x-axis. Distance ai from the (i-1)th to the ith x-axis along the ith x-axis.
Posture prediction and physics-based human motion simulation Chapter | 31
427
FIGURE 31.1 Rigid-link mechanical structure of a digital human with collocated multiple joints. The local reference frames are attached to all the degrees of freedom of the model based on DenaviteHartenberg parameterization.
FIGURE 31.2
DenaviteHartenberg parameters used in modeling a kinematic representation of a digital human.
428 PART | VI Activities of daily living
Time derivatives of the transformation matrix Ai are necessary to calculate the dynamics and can also be obtained in the recursive form as vTi Bi ¼ A_ i ¼ Bi1 Ti þ Ai1 q_i vqi
(31.4)
vTi v2 Ti vTi Ci ¼ B_ i ¼ Ci1 Ti þ 2Bi1 q_i þ Ai1 2 q_2i þ Ai1 q€i vqi vqi vqi
(31.5)
where qi is the generalized coordinate for transformation Ti,A0 ¼ I, and B0¼C0¼0. The Lagrange’s equation is given as d vL vL si ¼ dt vq_i vqi
(31.6)
where L ¼ K e V (kinetic energy e potential energy), q is the generalized coordinate vector (joint angles), and si is the joint torque vector. Given the mass and inertia properties of each body segment and the external force f Tk ¼ ½ fx fy fz 0 and moment hTk ¼ ½ hx hy hz 0 for the link k (1 k n) defined in the global coordinate system, the joint actuation torques si for each of the joints can be computed using recursive backward dynamics as follows: vAi vAi vAi D i gT Ei f Tk Fi GTi Ai1 z0 (31.7) si ¼ tr vqi vqi vqi where Di ¼ Ji CTi þ Tiþ1 Diþ1
(31.7a)
Ei ¼ mi i r i þ Tiþ1 Eiþ1
(31.7b)
F i ¼ k rf dik þ Tiþ1 Fiþ1
(31.7c)
G i ¼ hk dik þ Giþ1
(31.7d)
Dnþ1 ¼ Enþ1 ¼ Fnþ1 ¼ 0 i
(31.7e) k
where g is the gravity vector, ri is the location of the center of mass in the ith local frame, rf is the location of the external force acting in the kth frame, and dik is the Kronecker delta. The gradients of the EOMs are also required for faster implementation of gradient-based optimization methods. These gradients with respect to joint angles, joint angle velocities, and joint angle accelerations can also be analytically calculated (Xiang et al., 2009).
4. Optimization The EOMs must be solved to determine the torques required to perform any given task. Many different approaches are available in the literature, each with its own advantages and disadvantages (Otten, 2003). However, these traditional approaches are not suitable for use with human posture and motion simulation. While these approaches work well with open-chain systems with relatively fewer DOFs, solving the EOMs of a high-fidelity human model as described in this chapter is a very time-consuming and computationally intensive task. In addition, most tasks would require the human model to alternate between closed-loop and open-chain systems. Traditional EOM solvers are also not well suited for applications where there are intermediate constraints on the state of the variables. For instance, to simulate the walking task, the initial and final foot placement is typically constrained to be on the floor. In addition, there might be intermediate constraints on the hands to simulate an activity being performed by the human while walking, such as carrying a box. Hence, we employ an optimization-based strategy to estimate the solution of a large, interdependent system of EOMs as described in this section. This approach also allows us to expand the modeling capabilities by adding more components to the objective function and constraints as the research matures. A typical optimization problem has three main components: design variables, an objective function, and constraints (Arora, 2016). Design variables are the parameters of the problem that can change and the values of which need to be determined. The objective function, a function of design variables, is a measure that must be minimized. For the problem of human modeling and simulation, we use one or many performance measures as the objective function. The constraints are
Posture prediction and physics-based human motion simulation Chapter | 31
429
the bounds on the design variables or the bounds on some function of the design variables that must not be exceeded. Below, we describe these components for a typical human posture and motion simulation problem.
5. Design variables Joint angles, qi, are the unknowns that must be determined in most human posture and simulation problems. For PP problems, the joint angles are the design variables. When the problem involves a time component, as in PD, joint angle profiles qi(t) are approximated as linear combinations of cubic B-spline basis functions. In this case, the control points of these cubic B-spline functions serve as the design variables. Corresponding joint angle, velocity, and acceleration values are calculated at each iteration from these control point values.
6. Performance measure The goal of the optimization process is to minimize/maximize some performance criteria. The performance criteria typically represent real-world driving factors for human posture and motion. For example, human posture can be driven in part to minimize discomfort in a given pose, or a human might choose a posture that makes it possible to acquire a visual target. In optimization-based human simulation, these performance criteria are modeled as mathematical functions of the design variables that are minimized to guide the solution. Some common performance measures used in PP are described below.
7. Joint displacement In PP, joint displacement is used to represent the difference between a final posture and a neutral posture as shown in Fig. 31.3 (Farrell, 2005). The neutral pose qN represents the joint angles in a relatively comfortable position. For example, a typical neutral pose is standing straight with the arms at the side (N-pose). The joint displacement performance measure is defined to minimize the distance between the final joint angle and the corresponding neutral angle (Eq. 31.8). To avoid nondifferentiability, the squared difference is used in place of absolute difference. In addition, a weighted sum is used to emphasize the stronger influence of certain joints over others (Abdel-Malek & Arora, 2013). min
f ðqÞ ¼
n X
2 wi qi qNi
(31.8)
i¼1
where qi is the ith joint angle in the final posture, qNi is the ith joint angle in the neutral posture, wi is the weighting factor for the ith joint, and n is the number of joint angles.
FIGURE 31.3 Joint displacement represents the difference between the final posture and the neutral posture.
430 PART | VI Activities of daily living
8. Joint discomfort While joint displacement focuses directly on minimizing the difference from a comfortable posture, joint discomfort expands that concept to include penalty terms designed to mathematically represent the additional factors in musculoskeletal discomfort. As described by Abdel-Malek and Arora (2013), these factors are incorporated into the joint discomfort performance measure: (1) the tendency to move different segments of the body sequentially, (2) the tendency to gravitate to a reasonably comfortable neutral position, and (3) the discomfort associated with moving when joints are near their respective limits. The second factor (i.e., gravitating to a neutral position) is accounted for in the joint displacement formulation (Eq. 31.8). To accommodate the tendency to move body segments sequentially, the difference in joint angle is normalized by the total range of the joint, and the weighting factors are substantially increased to discourage movement in those body parts that should move last (e.g., the clavicles). To represent discomfort near the joint limits, penalty terms are added to the performance measure to spike the function value as the limit is approached (Marler et al., 2005). Mathematically, the resulting function is defined as n 1 X qi qNi min f ðqÞ ¼ þ G QL wi U þ G QU (31.9) i i G i¼1 qi qLi U 100 5:0 qi qi þ 1:571 þ 1 QUi ¼ 0:5sin qUi qLi 100 5:0 qi qLi QLi ¼ 0:5sin þ 1:571 þ 1 qUi qLi
(31.10)
(31.11)
where GQUi is the penalty term for joint angles that near their upper limits and GQLi is the penalty term for joint angles that near their lower limits.
9. Vision A common driving factor for static posture is one’s ability to see a given target. For example, when reaching to engage the handbrake in a vehicle, a human would typically use a pose in which the handbrake is visible. To simulate this behavior in PP, vision is modeled mathematically as a performance measure for the optimization problem. Specifically, visual acuity, or the capacity to discriminate the details of an object in the field of view, is formulated into a continuous objective function that behaves well in optimization (Eq. 31.12) (Smith et al., 2009). To this end, the vision vector estimates the line of sight from a point between the eyes relative to the last frame of the head/neck system, while the target vector calculates the direction from the point between the eyes to the target as shown in Fig. 31.4. min
f ðqÞ ¼ e35ð1cos½qðqÞÞ þ 1
(31.12)
where q(q) is the angle of sight between the vision vector and the target vector, dependent on the current posture q. While this performance measure closely represents the visual acuity as described in the literature, it is not well behaved in the gradient-based optimization problem in practice, especially when combined with other performance measures. This is because of a tendency of the curve to flatten (i.e., near-zero gradient) for many potential values of the angle of sight q(q) as shown in Fig. 31.5.
FIGURE 31.4 Head vector representing the line of sight relative to the target.
Posture prediction and physics-based human motion simulation Chapter | 31
FIGURE 31.5
431
The visual acuity performance measure (Eq. 31.12) has significant portions with near-zero gradient.
FIGURE 31.6 The visual displacement performance measure (Eq. 31.13).
FIGURE 31.7 Predicted postures on Santos using (A) visual acuity, (B) visual acuity combined with joint displacement, (C) visual displacement, and (D) visual displacement combined with joint displacement.
Smith et al. (2009) proposed a more intuitive visual displacement performance measure for driving posture with vision that relies directly on minimizing the angle of sight (Eq. 31.13). Visual displacement drives the posture to see the target and still combines well with other performance measures because of its curve, as shown in Fig. 31.6. qðqÞ min f ðqÞ ¼ 10 1 cos (31.13) 2 These vision-based performance measures are compared on Santos in Fig. 31.7. They are used both as the only objective function and in combination with the joint displacement performance measure described above. Further development of vision-based human performance measures, including eye angles, has been formulated by Knake et al. (2010).
432 PART | VI Activities of daily living
10. Joint torque The most common performance measure for PD is the dynamic effort at each joint. The performance measure (or objective function) is, therefore, to minimize the sum of the torques squared for all joints over the simulation time. min f ðqÞ ¼
ZT X n t¼0
s2i ðqÞdt
(31.14)
i¼1
where n is the total number of joints of the human model, si is the actuator torque of the ith joint, and T is the total simulation time. Joint actuation torques are calculated using the recursive EulereLagrangian formulation as a function of joint angles, velocities, and accelerations.
11. Constraints Several physics-based, task-based, and environment-based constraints must be employed to predict the postures and motions. The physics-based constraints typically limit the capabilities of the human model to avoid unrealistic or impossible situations such as lifting a heavy table with one finger. Joint angle limits, joint torque limits, self-avoidance, and stability criteria are examples of these physics-based constraints. Task-based constraints are dependent on the task being simulated. For instance, it is necessary for the hand to touch the handbrake in a specific location at a particular angle. The environment-based constraints depend on the task and the environment surrounding the human in which the task must be simulated. For instance, the feet must never go below the floor, and body segments cannot go into obstacles or walls. A few common constraints for the predictive algorithms are discussed in the following sections.
12. Distance The most common constraint requires the distance between a point on the body and a global position to be within a sufficiently small tolerance. For example, this can be used to constrain a hand to a grip or an elbow to an armrest. The formulation for the distance constraint is xðqÞlocal xglobal ε (31.15) where x(q)local is a local position in the ith frame whose global position is determined by the current posture q using Eq. (31.3). This distance constraint can be used for multiple points on the body (Farrell, 2005) and in tandem with other constraints as shown in Fig. 31.8.
13. Vision In certain scenarios using static postures, acquiring a particular visual target might be required rather than desired. For example, analyzing a seating posture in a vehicle might require a posture in which a human can see the rear-view mirror or a target in front of the cab, as shown in Fig. 31.9. In this case, the visual displacement performance measure in Eq. (31.13) can be reformulated and used as a constraint (Eq. 31.16). Furthermore, the visual displacement constraint can be combined with obstacle avoidance constraints to avoid obstructions to vision by obstacles in the line of sight (Knake et al., 2010). qðqÞ 10 1 cos ε (31.16) 2
FIGURE 31.8 Result from on Santos using multiple distance constraints on the body.
Posture prediction and physics-based human motion simulation Chapter | 31
433
FIGURE 31.9 Predicted posture using visual displacement as a constraint on Santos.
14. Self-avoidance Using the digital human model described in Fig. 31.1, there is no mathematical reason for the predictive solution to avoid colliding with its own body. To ensure a realistic simulation of posture and motion, constraints must be added to prevent self-intersection. To this end, the body of the digital human is filled with spheres, which are added to the kinematic chain and grouped according to segment to distinguish between the torso and limbs, as shown in Fig. 31.10 (Degenhardt et al., 2014; Farrell et al., 2016). The spheres in one group are algorithmically checked against the spheres in all other groups (Johnson et al., 2010). The use of spheres allows a computationally efficient approach that simply requires the distance between checked spheres to be greater than the sum of their radii. To ensure that the constraints are differentiable, the constraint effectively models the same behavior by restricting the distance squared to be greater than the squared sum of the radii (Eq. 31.17). 2
ðri þ rj Þ kxðqÞ i xðqÞ j k 0 2
s
s
FIGURE 31.10 Human model filled with spheres grouped by segment, shown here in a different color for each group.
(31.17)
434 PART | VI Activities of daily living
Note that this method adds one constraint to the optimization problem for each pair of spheres that need to be checked. Methods to address computational efficiency by reducing the number of active constraints are addressed in the literature (Johnson et al., 2009, Johnson et al., 2010). Constraints allow flexibility in optimization-based human simulation. Additional constraints have been and can be modeled to expand the current capabilities. The next sections discuss in more detail the benefits of using these optimization-based methods to simulate human motion through PP and PD.
15. Posture prediction Postures are predicted by solving an optimization problem to minimize an objective function, or more specifically one of a combination of human performance measures, as described above. Depending on the task, one or more of the performance measures are employed to predict the desired posture. In addition, many constraints can be added and combined to represent the real-world restrictions in a given scenario, such as contact with the environment or objects. This flexibility makes it possible to simulate a specific scenario across varying anthropometry. For example, PP can be used to analyze differences in posture and reach on a motorcycle for both a taller male and a shorter female while ensuring that contact with the handlebars is maintained (see Fig. 31.11). The capabilities of PP can be increased as the need arises. For instance, if there are any obstacles to be avoided in the environment while trying to reach a point in space, an obstacle avoidance constraint can be added (Johnson et al., 2009) (see Fig. 31.12). Additional constraints can also be used to account for obstacles that obstruct the line of sight (Knake et al., 2010) and obstacles that represent equipment attached to the body (Farrell et al., 2016) (see Fig. 31.13).
FIGURE 31.11 Posture prediction used to analyze different anthropometries (A) male and (B) female on a motorcycle with hand and foot contact constraints.
FIGURE 31.12 Using posture prediction to simulate a reach around heavy machinery with (A) no obstacle awareness and (B) obstacle avoidance using constraints.
Posture prediction and physics-based human motion simulation Chapter | 31
435
FIGURE 31.13 Equipment attached to the body as obstacles (A) before and (B) after additional constraints in posture prediction.
16. External forces In some situations, external forces are present as the human performs the task. These external forces can also be accounted for in PP to generate realistic postures (Liu et al., 2009). The predictions are purely static in nature and assume there is no velocity or acceleration of movement. This can be used, for example, to predict postures that exert a push or pull, as shown in Fig. 31.14, or postures that require holding tools and applying a force while performing the task.
17. Motion capture processing Furthermore, PP can be extended to automatically generate digital human motion from motion capture in real time by solving the optimization problem for a sequence of frames (Seydel et al., 2017). To this end, virtual markers are placed on the digital human model corresponding to the motion capture markers as shown in Fig. 31.15. A distance performance measure derived from the distance constraint formulation in Eq. (31.15) is then used to minimize the distance between a virtual marker and its corresponding motion capture marker for the current frame. Processing motion capture using optimization-based PP has the additional benefit that it is compatible with any of the performance measures and constraints described earlier. For example, it can be used with vision constraints, obstacle avoidance, or force calculations, providing a wealth of analytical information as shown in Fig. 31.16.
FIGURE 31.14 Posture prediction with forces applied to Santos to analyze (A) hold, (B) push, and (C) pull.
436 PART | VI Activities of daily living
FIGURE 31.15 Motion capture markers on the human subject and the digital human model: (A) motion capture markers, (B) virtual markers, and (C) minimized distances.
FIGURE 31.16 Motion capture using posture prediction on Santos with force calculations and associated joint torques.
18. Predictive dynamics While PP predicts realistic postures of humans performing various tasks in the presence of external forces, obstacles, and other physical and environmental limitations, PD predicts the full motion of a task within the bounds of physical and environmental limitations. Posture prediction can predict and analyze a task or a posture while accounting for static forces, but there are many tasks where a dynamic motion analysis becomes necessary. For instance, a vertical jumping task with and without carrying loads or a box lifting and placement task requires a full dynamic analysis to learn more about the requirements of the task on the human body. As discussed in the optimization section, the goal of PD is essentially finding the optimal joint angle profiles for all the joints in the human body to achieve the task objective while complying with physical and environmental limitations. In a single joint angle profile curve, there are infinite points, and thus the problem defined above becomes infinite and cannot be solved using optimization. The problem needs to be parameterized, and there are multiple ways to go about it. For example, if we discretize the problem at every fixed time interval, e.g., every 0.1 s, and for every joint, the solution is a vector of 10 values per second, and the continuous motion is generated by interpolating these values. Another option is to parameterize a curve using B-splines so the control points for these B-splines become the design variables. Joint angle values, velocities, and accelerations can be derived from these B-splines. Santos uses cubic B-splines parameterization rather than time discretization or other methods. The use of B-spline interpolation inherently constrains each joint profile to be continuous and differentiable, which is what typical human motion looks like.
Posture prediction and physics-based human motion simulation Chapter | 31
437
The complexity and length of the task to be predicted typically dictates the number of control points. As a rule of thumb, the number of control points, and thus the number of design variables for the optimization problem, increases as the task being simulated gets longer. For instance, a step simulation of a task would require fewer control points than a stride simulation of the same task. However, the computational requirements of the optimization problem, and thus the time necessary to solve the problem, also increase substantially as the number of control points increases. Hence, for every task, an optimal balance of the number of control points must be chosen that allows for the variations of the task to be predicted while minimizing the computational time requirements. Many tasks have been developed using PD, including walking (Xiang, Chung et al., 2010), walking backward (Kwon et al., 2014), running (Chung et al., 2015), jumping (Hariri et al., 2013), stair-ascending (Bhatt et al., 2008), box lifting (Xiang et al., 2010), and ingress-egress (Kwon, 2011). Predictive dynamics uses a gradient-based optimization algorithm to solve the human motion prediction problem (Xiang et al., 2009). The solver time reduces drastically because the analytical constraints for each objective function and constraint function are calculated analytically. Predictive Dynamics also leverages motion capture data for a specific task to predict natural task motions using fewer constraints (Xiang et al., 2012). In the following paragraphs, we will discuss the vertical jump task and the insights gained using PD. The vertical jump task is defined as predicting the maximum height that the tip of the right hand can reach from the floor. A Santos avatar 1.82 m tall and weighing 174 lbs was used to predict the maximum height that the avatar can reach while being loaded under different conditions. Santos was able to achieve an unrealistic maximum height as high as 3.2 m when only the isokinetic strength limits were applied. The predictions presented in this chapter were accomplished using dynamic strength limits (Marler et al., 2012), where the peak torque a human can produce at any given angle and velocity is derived from a strength surface equation. The strength surface is derived by using isokinetic (constant velocity) and isometric (constant length) strength testing for all major joints of the human body. In addition to predicting the realistic maximum vertical jump height under various loadings for Santos (see Fig. 31.17), PD also provides a lot of insight on the jump mechanism and how the motion changes because of the dynamic strength constraints.
Additional load (in lbs) 9.83
13.44
20.49
42.62
93.64
2.31
2.13
1.99
Maximum height (in meters) 2.45
2.4
FIGURE 31.17 Maximum height predictions using a predictive dynamics vertical jump task with different loading conditions for the same avatar.
438 PART | VI Activities of daily living
FIGURE 31.18 Plot of right knee torque profile for the Santos vertical jump task with static and dynamic strength limits.
The jump height of 2.45 m is easily achievable when static strength limits are used. However, when dynamic strength limits are applied, the right knee torque becomes the limiting factor, as shown in Fig. 31.18. Also, the motion predicted for the same height of 2.45 m when static strength limits are used is different than the motion predicted when dynamic strength limits are used. Assuming the dynamic strength limits are only used for postprediction analysis, the motion would seem infeasible because the torque curve would breach the dynamic strength limits curve. However, when the motion is predicted by applying dynamic strength limits as a constraint, the algorithm predicts a different motion with different joint angle values and velocities so that it can achieve the 2.45 m height without breaching the constraints. Note that, as the motion changes, the corresponding joint angle and joint velocity valuesdand thus the dynamic strength limitsdalso change. Such analysis capabilities result only when the simulation task is predictive and full dynamic strength implementation is realized.
19. Future research Although the capabilities of PD have increased throughout the last decade with the addition of performance measures, the use of motion capture as a seed motion for a hybrid approach, and the addition of strength surfaces as an analysis tool and as constraints, a lot more can be done to achieve even better results in less time. The most important development could be a decrease in the time spent on the development of PD tasks and on solving a prediction problem once the task is developed. We are currently working on a modular approach to solving the optimization PD problem so that new tasks can be developed using an automated implementation. In addition, a few steps have been taken to increase the speed of solving each PD problem, such as using neural networks to identify a good starting point for the optimization solver based on task input parameters, performing a few computations in parallel by leveraging multicore processors, and optimizing the tolerances for the optimization problem. However, additional research is necessary in both automated task development and improved computation time to make PD occur in real time or at least near real time.
20. Conclusion The predictive capabilities of the human simulation software Santos have been in development since 2003 at the University of Iowa’s VSR program. Substantial effort also went to validating these predictive capabilities (Rahmatalla et al., 2011). These capabilities have been used extensively to answer questions in military and nonmilitary applications. Previously, human motion simulation relied heavily on motion capture experiments, and it was time-consuming and expensive to perform a new analysis when parameters such as weight or speed changed. Using the predictive tools in the
Posture prediction and physics-based human motion simulation Chapter | 31
439
Santos environment, PP and PD, it is easier to perform such trade-off analysis. Predictive capabilities provide a very good approximation of each case; however, testing of the final design or a few designs may be appropriate, depending on the importance of the task or its associated risk. While such predictive simulation capabilities do not eliminate the need for experimentation, they can drastically reduce the number of iterations required, which in turn reduces time and cost. This chapter describes the predictive capabilities of Santos while focusing on the unique insights that these predictive tools can provide to the end user.
References Abdel-Malek, K., & Arora, J. S. (2013). Human motion simulation: Predictive dynamics. Academic Press. Andrews, S., Huerta, I., Komura, T., Sigal, L., & Mitchell, K. (December 2016). Real-time physics-based motion capture with sparse sensors. In Proceedings of the 13th European conference on visual media production (pp. 1e10). Arora, J. S. (2016). Introduction to optimum design. Academic Press. Bhatt, R., Xiang, Y., Kim, J. H., Mathai, A., Penmatsa, R., Chung, H.-J., … Obuseck, J. (2008). Dynamic optimization of human stairs climbing motion. In Proceedings of SAE digital human modeling conference. Pittsburg, PA. Borno, M. A., Righetti, L., Black, M. J., Delp, S. L., Fiume, E., & Romero, J. (July 2018). Robust physics-based motion retargeting with realistic body shapes. ACM SIGGRAPH/Eurographics Symposium on Computer Animation 2018, 37(6), 1e12. Cheng, H., Obergefell, L., & Rizer, A. (1996). The development of the GEBOD program. In 1996 fifteenth southern biomedical engineering conference. Dayton, OH, USA: IEEE. Chung, H.-J., Xiang, Y., Arora, J., & Abdel-Malek, K. (2015). Dynamic optimization of human running with analytical gradients. Journal of Computational and Nonlinear Dynamics, 10(2), 1e18. Degenhardt, R. K., Farrell, K. D., Bhatt, R. M., & Abdel-Malek, K. (May 2014). Body-based obstacle avoidance for motion simulation. In Digital human modeling symposium. Tokyo, Japan. Denavit, J., & Hartenberg, R. S. (1955). A kinematic notation for lower-pair mechanisms based on matrices. Journal of Applied Mechanics, 77, 215e221. Farrell, K. (2005). Kinematic human modeling and simulation using optimization-based posture prediction. Iowa City: University of Iowa. M. S. thesis. Farrell, K., Degenhardt, R., Bhatt, R., & Abdel-Malek, K. (2016). Body-based obstacle avoidance for motion simulation using optimization-based posture prediction. In 4th international digital human modeling symposium. Montreal, Canada. Hariri, M., Xiang, Y., Chung, H.-J., Bhatt, R., Arora, J. S., & Abdel-Malek, K. (2013). Simulation and prediction of the motion of a human in a vertical jumping task. In ASME 2013 dynamic systems and control conference. Palo Alto, CA. Johnson, R., Fruehan, C., Schikore, M., & Marler, T. (2010). New developments with collision avoidance for posture prediction. In V. Duffy (Ed.), Advances in applied digital human modeling (pp. 336e346). CRC Press. Johnson, R., Smith, B. L., Penmatsa, R., Marler, T., & Abdel-Malek, K. (2009). Real-time obstacle avoidance for posture prediction. In SAE digital human modeling conference. Goteborg, Sweden. Knake, L., Mathai, A., Marler, T., Farrell, K., Johnson, R., & Abdel-Malek, K. (2010). New capabilities for vision-based posture prediction. In V. Duffy (Ed.), Advances in applied digital human modeling (pp. 628e638). CRC Press. Kwon, H. J. (2011). Simulating ingress and egress motion for heavy earthmoving machines. Iowa City: University of Iowa. Ph.D. thesis. Kwon, H.-J., Xiang, Y., Bhatt, R., Rahmatalla, S., Arora, J., & Abdel-Malek, K. (July 2014). Backward walking simulation of humans using optimization. Structural and Multidisciplinary Optimization, 50(1), 169e179. Liu, Q., Marler, T., Yang, J., & Kim, J. H. (2009). Posture prediction with external loads e a pilot study. SAE International Journal of Passenger Cars, 2(1), 1014e1023. Marler, T., Frey-Law, L., Mathai, A., Spurrier, K., Avin, K., & Cole, E. (2012). Integration of strength models with optimization-based posture predition. In V. G. Duffy (Ed.), Advances in applied human modeling and simulation (pp. 327e336). CRC Press. Marler, R. T., Rahmatalla, S., Shanahan, M., & Abdel-Malek, K. (2005). A new discomfort function for optimization-based posture prediction. In SAE human modeling for design and engineering conference, Iowa City, IA. Otten, E. (September 29, 2003). Inverse and forward dynamics: Models of multi-body systems. Philosophical Transactions of the Royal Society B: Biological Sciences, 358(1437), 1493e1500. Penmatsa, R., Bhatt, R., Farrell, K., Rochambeau, B., Fruehan, C., Verma, U., … Abdel-Malek, K. (2009). Estimation of mass and inertia properties of human body segments for physics-based human modeling and simulation applications. SAE International Journal of Passenger Cars, 2(1), 1626e1632. Rahmatalla, S., Xiang, Y., Li, J., Smith, R., Meusch, J., & Bhatt, R. (2011). A validation framework for predictive human models. International Journal of Human Factors Modelling and Simulation, 2(1e2), 67e84. Seydel, A. L., Farrell, K., Johnson, R., Marler, T., Bhatt, R. M., Rahmatalla, S., & Abdel-Malek, K. (2017). Improved motion capture processing for high fidelity human models using optimization-based prediction of posture and anthropometry. In International conference on applied human factors and ergonomics. Los Angeles, CA. Smith, B. L., Marler, T., & Abdel-Malek, K. (2009). Studying visibility as a constraint and as an objective for posture prediction. SAE International Journal of Passenger Cars - Mechanical Systems, 1(1), 1118e1124.
440 PART | VI Activities of daily living
Xiang, Y., Arora, J. S., & Abdel-Malek, K. (2009). Optimization-based motion prediction of mechanical systems: Sensitivity analysis. Structural and Multidisciplinary Optimization, 37(6), 595e608. Xiang, Y., Arora, J. S., & Abdel-Malek, K. (September 2012). Hybrid predictive dynamics: A new approach to simulate human motion. Multibody System Dynamics, 28(3), 199e224. Xiang, Y., Arora, J. S., Rahmatalla, S., Marler, T., Bhatt, R., & Abdel-Malek, K. (April 2010). Human lifting simulation using a multi-objective optimization approach. Multibody System Dynamics, 23(4), 431e451. Xiang, Y., Chung, H.-J., Kim, J. H., Bhatt, R. M., Rahmatalla, S., Yang, J., … Abdel-Malek, K. (April 2010). Predictive dynamics: An optimization-based novel approach for human motion simulation. Structural and Multidisciplinary Optimization, 41(3), 465e479.