- Email: [email protected]
Motion Analysis System for the digitalization and assessment of manual manufacturing and assembly processes
Proceedings,16th IFAC Symposium on Proceedings,16th IFAC Symposium on Information Control Problems in Manufacturing Proceedings,16th IFAC Symposium on Information Control Problems in Manufacturing Available online at www.sciencedirect.com Proceedings,16th IFAC Symposium on Bergamo, Italy, June 11-13, 2018 Information Control Problems in Manufacturing Bergamo, Italy, June 11-13, 2018 Proceedings,16th IFAC Symposium on Information Control in Manufacturing Bergamo, Italy, JuneProblems 11-13, 2018 Information Control in Manufacturing Bergamo, Italy, JuneProblems 11-13, 2018 Bergamo, Italy, June 11-13, 2018
ScienceDirect
IFAC PapersOnLine 51-11 (2018) 411–416
Motion Motion Analysis Analysis System System for for the the digitalization digitalization and and assessment assessment of of manual manual Motion Analysismanufacturing System for the and digitalization and assessment of manual assembly processes Motion Analysismanufacturing System for the and digitalization assessment of manual assembly and processes Motion Analysismanufacturing System for the and digitalization assessment of manual assembly and processes manufacturing and assembly processes † Emilio Mauro Francesco †, Alberto Regattieri* manufacturing and assembly processes , Alberto Regattieri* Emilio Ferrari*, Ferrari*, Mauro Gamberi*, Gamberi*, Francesco Pilati* Pilati* †
Emilio Ferrari*, Mauro Gamberi*, Francesco Pilati* †, Alberto Regattieri* Emilio Ferrari*, Mauro Gamberi*, Francesco Pilati* †, Alberto Regattieri* * Department of Engineering, Viale del 2, Alberto Regattieri* Emilio Ferrari*, Mauro Gamberi*, Pilati* FrancescoViale * University University of of Bologna, Bologna, Department of Industrial Industrial Engineering, del ,Risorgimento Risorgimento 2, 40136 40136 Bologna, Bologna, Italy Italy * University of Bologna, Department of Industrial Engineering, Viale del Risorgimento 2, 40136 Bologna, Italy * University of Bologna, Department of Industrial Engineering, Viale del Risorgimento 2, 40136 Bologna, Italy †† Corresponding Author: [email protected] * University of Bologna, Department of Industrial Engineering, Viale del Risorgimento 2, 40136 Bologna, Italy Corresponding Author: [email protected] † Corresponding Author: [email protected] † Corresponding Author: [email protected] † Corresponding Author: [email protected] Abstract: Abstract: Manual Manual manufacturing manufacturing and and assembly assembly systems systems radically radically changed changed after after the the advent advent of of the the fourth fourth industrial revolution (Industry 4.0). The production paradigm shift to mass personalization involves the Abstract: Manual manufacturing and assembly systems radically changed after the advent of the fourth industrial revolution (Industry 4.0). The production paradigm shift to mass personalization involves the Abstract: Manual manufacturing andphase. assembly systems radically changed after the of advent of the fourth customers since the product design The remarkable variety and complexity these processes is industrial revolution (Industry 4.0). The production paradigm shift to mass personalization involves the Abstract: Manual manufacturing and assembly systems radically changed after the advent of the fourth customers since the product design phase. The remarkable variety and complexity of these processes is industrial revolution (Industry 4.0). The production paradigm shift to and mass personalization involves the tackled by highly skilled human operator, which perform value added non-repetitive tasks. Thus, the customers since the product design phase. The remarkable variety and complexity of these processes is industrial revolution (Industry 4.0). The production paradigm shift to mass personalization involves the tackled by since highlythe skilled human operator, which perform value added and non-repetitive tasks. Thus, is customers product design phase. The remarkable variety and complexity of these processes virtualization of manual operations represents a great opportunity to monitor, analyze and successively tackled by highly skilled human operator, which perform value added and non-repetitive tasks. Thus, the customers since the product design phase. The remarkable variety and complexity of these processes is virtualization of manual represents great opportunity to monitor, analyze and successively tackled by these highly skilled operations human operator, whichaa perform value added and non-repetitive tasks. Thus, the optimize production processes. Within this context, this research proposes an original virtualization of manual operations represents great opportunity to monitor, analyze and successively tackled by these highly skilled operations human operator, whicha perform value added and non-repetitive tasks. the optimize production processes. Within this opportunity context, this research proposes an Thus, original virtualization of manual represents great to monitor, analyze and successively hardware/software architecture called Motion Analysis System (MAS) developed to monitor and to optimize these production processes. Within this opportunity context, this research proposes an original virtualization of manual operations represents a great to monitor, analyze and successively hardware/software architecture called Motion Analysis System (MAS) developed to monitor and to optimize manual these manufacturing production processes. Within this context, this research proposes anThe original evaluate and assembly processes through motion capture technologies. MAS hardware/software architecture called Motion Analysis System (MAS) developed to monitor and to optimize these production processes. Within this context, this research proposes an original evaluate manual manufacturing and assembly processes through motion capture technologies. The MAS hardware/software architecture called Motion Analysis System (MAS) developed to monitor and to adopts a hardware architecture represented by a network of depth cameras, e.g. a marker-less optical evaluate manual manufacturing and assembly processes through motion capture technologies. The MAS hardware/software architecture and called Motion Analysis System (MAS) developed to monitor to adopts a manual hardware represented byprocesses a network of depth cameras, e.g. a marker-less optical evaluate manufacturing assembly through motion capture technologies. Theand MAS motion to digitalize the operator movements and postures at 30 Hz while he production adopts acapture, hardware architecture represented byprocesses a network of depth cameras, e.g. aperforms marker-less optical evaluate manual manufacturing and assembly through motion capture technologies. The MAS motion capture, to digitalize the operator movements and postures at 30 Hz while he performs production adopts hardware architecture represented byexploits a network ofdata depth cameras, e.g.he aperforms marker-less optical activities. A software architecture these in to workstation 3D layout motion aacapture, to digitalize the operator movements and postures atrelation 30 Hz while production adopts hardware architecture represented byexploits a network ofdata depth cameras, e.g.he aperforms marker-less activities. A customized customized software architecture these inat relation to the the workstation 3D optical layout motion capture, to digitalize the operator movements and postures 30 Hz while production to automatically and quantitatively evaluate a set of productive KPIs. The MAS represents a reliable, activities. A customized software architecture exploits these data in relation to the workstation 3D layout motion capture, to digitalize the operator movements and postures at 30 Hz while he performs production to automatically and quantitatively evaluate aexploits set of productive KPIs. ThetoMAS represents a3Dreliable, activities. A customized software architecture these data in relation the workstation layout powerful and meaningful tool for production managers and practitioners able to provide an in-depth time to automatically and quantitatively evaluate a set of productive KPIs. The MAS represents a reliable, activities. A customized software architecture exploits these data in relation to the workstation 3D layout powerful and meaningful tool for production and practitioners able MAS to provide an in-depth time to automatically and quantitatively evaluate managers aby setanofoperator, productive KPIs. The represents a reliable, and space analysis of the activities performed as the walking paths, the accessed picking powerful and meaningful tool for production managers and practitioners able to provide an in-depth time to automatically and evaluate managers aby setanofoperator, productive KPIs. The MAS represents a reliable, and spaceand analysis of quantitatively the tool activities performed as the walking paths, the accessed picking powerful meaningful for production and practitioners able to provide an in-depth time locations, the used space within the workstation and the hand movements. Finally, this contribution ends and space analysis of the activities performed by an operator, as the walking paths, the accessed picking powerful and meaningful tool forthe production managers practitioners able to provide an in-depth time locations, the used space within workstation theand hand movements. Finally, this ends and space analysis of the activities byand an operator, as the walking paths, the contribution accessed picking with aa real industrial application of the MAS which analyzes the manual assembly processes of locations, the used space within theperformed workstation and the hand movements. Finally, this contribution ends and space analysis of the activities performed by an operator, as the walking paths, the accessed picking with real industrial application of the MAS which analyzes the manual assembly processes of aa locations, the usedThe space within theof workstation and the hand movements. Finally, this contribution ends microwave oven. system setup is presented and discussed along with the case study key results and with a real industrial application the MAS which analyzes the manual assembly processes of locations, the usedThe space within theof workstation the hand movements. Finally, this contribution microwave oven. system setup is presented discussed along the case study key resultsends andaa with a real industrial application the MAS and which analyzes thewith manual assembly processes of findings. microwave oven. The system setup is presented and discussed along with the case study key results and with a real industrial application the MAS and which analyzes thewith manual assembly processes findings. microwave The system setup of is presented discussed along the case study key resultsof anda findings. oven. microwave oven. The system setup is presented and discussed along with the case study key results and © 2018, IFAC (International Federation of Automatic Control) Hosting by Elsevier Ltd. All rights reserved. findings. findings. Keywords: Motion capture, Depth camera, Control volume, Industry 4.0, Smart factory, Manufacturing, Keywords: Motion capture, Depth camera, Control volume, Industry 4.0, Smart factory, Manufacturing, Assembly, Assessment, KPI,Depth Performance, Keywords: Motion capture, camera, Digitalization. Control volume, Industry 4.0, Smart factory, Manufacturing, Assembly, Assessment, KPI,Depth Performance, Keywords: Motion capture, camera, Digitalization. Control volume, Industry 4.0, Smart factory, Manufacturing, Assembly, Assessment, KPI, Performance, Digitalization. Keywords: Motion capture, Depth camera, Control volume, Industry 4.0, Smart factory, Manufacturing, Assembly, Assessment, KPI, Performance, Digitalization. Assembly, Assessment, KPI, Performance, Digitalization. conceived conceived for for gaming gaming and and accurately accurately customized customized to to extend extend 1. 1. INTRODUCTION INTRODUCTION their applicability to the industrial environment, whereas the conceived for gaming and accurately customized to extend their applicability to the industrial environment, whereas the 1. INTRODUCTION conceived for gaming and accurately customized toproperly extend latter is an original software application The industrial environment is currently experiencing its their applicability to the industrial environment, whereas the 1. INTRODUCTION conceived for gaming and accurately customized to extend latter is an original software application properly The industrial environment is currently experiencing its their applicability to the industrial environment, whereas the 1. INTRODUCTION programmed to dynamically assess the operator productive fourth industrial revolution (Bortolini et al., 2017a; Faccio et latter is an original software application properly The industrial environment is currently experiencing its their applicability to the industrial environment, whereas the programmed to dynamically assess theapplication operator productive fourth industrial environment revolution (Bortolini et al., experiencing 2017a; Faccio its et latter is an original software properly The industrial is currently performance. The adoption of the MAS in an industrial al., 2015). The use of ubiquitous sensors in factory shop programmed to dynamically assess the operator productive fourth industrial revolution (Bortolini et al., 2017a; Faccio et latter is an original software application properly The industrial environment is currently experiencing its performance. toThe adoption of the the MAS in an productive industrial al., 2015). The revolution use of ubiquitous in factory shop dynamically assess operator fourth industrial (Bortolinisensors et al.,networks 2017a; Faccio et programmed environment to and automatically evaluate floors connected through communication enables adoption of the MAS in an productive industrial al., 2015). The revolution use of ubiquitous in factory shop programmed toThe dynamically assess operator fourth industrial (Bortolinisensors et al.,networks 2017a; Faccio et performance. environment enables enables to accurately accurately andthe automatically evaluate floors connected through communication enables performance. The adoption of the MAS in an industrial al., 2015). The use of ubiquitous sensors in factory shop the operator tasks and to quantitatively measure his the real-time communication of machines, operators, environment enables to accurately and automatically evaluate floors connected through communication networks enables performance. The adoption of the MAS in an industrial al., 2015). The use of ubiquitous sensors in factory shop the operator tasks and to quantitatively measure his the real-time communication of machines, operators, environment enables toand accurately andwith automatically evaluate floors connected through communication networks enables productive performances. These deal the time and space products, tools and costumers defining the so-called Smart the operator tasks to quantitatively measure his the real-time communication of machines, operators, environment enables to accurately and automatically evaluate floors connected through communication networks enables productive performances. These deal with the time and space products, tools and costumers defining the so-called Smart the operator tasks and to quantitatively measure his the real-time communication ofof manual machines, operators, aspects of the operator activities as the movements to perform Factory concept. The virtualization manufacturing productive performances. These deal with the time and space products, tools and costumers defining the so-called Smart the operator tasks and to quantitatively measure his the real-time communication ofof manual machines, operators, aspects of the operator activities as thewith movements toand perform Factory concept. Thecostumers virtualization manufacturing productive performances. These deal the time space products, tools and defining the so-called Smart aa task, the travelled of specific body parts and the and assembly operations (e.g. virtual reality) represents aa productive aspects of the operatordistance activities as thewith movements toand perform Factory concept. Thecostumers virtualization of manual manufacturing performances. These deal the time space products, tools and defining the so-called Smart task, the travelled distance of specific body parts and the and assembly operations (e.g. virtual reality) represents aspects of the operatordistance activities the movements to perform Factory concept. The virtualization of manual manufacturing locations in relation to 3D layout of great opportunity to monitor, analyse and successively aoccupied task, the travelled ofas body parts and and assembly operations (e.g. virtual reality) represents of the operatordistance activities asspecific the to perform Factory concept. The of manual occupied locations in relation to movements 3D layout of the the great opportunity to virtualization monitor, analyse andmanufacturing successivelyaa aspects a task, the travelled of specific body parts and and assembly operations (e.g. virtual reality) represents manufacturing or assembly area. optimize the production processes distinguished by the occupied locations in relation to 3D layout of the the great opportunity to monitor, analyse and successively a task, the travelled distance of specific body parts and the and assembly operations (e.g. virtual reality) represents a manufacturing or assembly area. optimize the production processes distinguished by the occupied locations in relation to 3D layout of the great opportunity to monitor, analyse and successively aforementioned features. Motion capture (MOCAP) manufacturing or assembly area. optimize the production processes distinguished by the occupied locations in relation to 3D layout of the great opportunity to monitor, analyse and successively aforementioned features. processes Motion distinguished capture (MOCAP) According to this purpose, this manuscript is organized as manufacturing or assembly area. optimize the production by the According to this purpose, area. this manuscript is organized as it it technologies are of strong help to target this goal. Aim of aforementioned features. Motion capture (MOCAP) manufacturing or assembly optimize the production processes distinguished by the technologies are offeatures. strong help to target this goal. Aim of According follows. Section investigates the different technologies to this 2 purpose, this manuscript is organized as it aforementioned Motion capture (MOCAP) follows. Section 2 investigates the different technologies these systems is to accurately track and record over time the technologies are of strong help to target this goal. Aim of According to this 2purpose, this manuscript istheir organized as it aforementioned Motion capture (MOCAP) these systemsare is tooffeatures. accurately track and record over time the currently available for MOCAP purpose, application follows. Section investigates the different technologies technologies strong of help target this goal. Aim of According to this 2purpose, this manuscript istheir organized as it currently available for MOCAP purpose, application postures and movements aa to human operator aa the 3D these systems is toofaccurately track and record overin time follows. Section investigates the different technologies technologies are strong help to target this goal. Aim of postures and movements of human operator in 3D and adoption in real industrial environment and the most currently available for MOCAP purpose, their application these systems is to accurately track and record over time the follows. Section 2 investigates the different technologies and adoption in realforindustrial andapplication the most environment. postures and is movements of track a human operator a the 3D currently available MOCAP environment purpose, their these systems to accurately and record overin environment. literature contributions concerning the usage of and adoption in real and the most postures and movements of a human operator intime a 3D relevant currently available forindustrial MOCAP environment purpose, their application relevant literature contributions concerning the usage of environment. and adoption in real industrial environment and the most postures and movements of a human operator in a 3D MOCAP for the assessment of the operator productive Considering the presented scenario, this paper proposes an relevant literature contributions concerning the usage of environment. and adoption in real industrial ofenvironment and the most MOCAP for the assessment the operator productive Considering the presented scenario, this paper proposes an relevant literature contributions concerning the usage of environment. performances. Section 3 presents the original MAS original Motion Analysis System (MAS) to digitalize the MOCAP for the assessment of the operator productive Considering the presented scenario, this paper proposes an relevant literature contributions concerning the productive usage of performances. Section 3 presents the original MAS original Motion Analysis System (MAS) to digitalize the MOCAP for the assessment of the operator Considering the presented scenario, this paper proposes an hardware/software architecture developed for the automatic operator activities in whatever manufacturing or assembly performances. Section 3 presents the original MAS original Motion Analysis System (MAS) to digitalize the MOCAP for the assessment of the operator productive Considering the presented scenario, this paper proposes an hardware/software architecture developed for the automatic operator activities in whatever manufacturing or assembly Section 3 presents the MAS original Motion Analysis System (MAS) to digitalize the performances. and quantitative assessment of manual manufacturing and environment and analyse the manual production architecture fororiginal the automatic operator activities in whatever manufacturing or processes. assembly performances. 3 presents original MAS original Motion System (MAS) to digitalize the hardware/software and quantitativeSection assessment ofdeveloped manualthe manufacturing and environment and Analysis analyse the manual production processes. hardware/software architecture developed for the automatic operator activities in whatever manufacturing or assembly assembly processes, along with the most relevant productive This system is based on the integration between a hardware and quantitative assessment of manual manufacturing and environment and analyse the manual production processes. hardware/software architecture developed for the automatic operator activities in whatever manufacturing or assembly assembly processes, along with the most relevant productive This system is based on the integration between a hardware and quantitative assessment of the manual manufacturing and environment and analyse theThe manual production processes. performances of the operator measurable by the MAS. and a software architecture. former is represented by a assembly processes, along with most relevant productive This system is based on the integration between a hardware and quantitative assessment of manual manufacturing and environment and analyse theThe manual production processes. performances of thealong operator measurable by the MAS. and asystem software architecture. former is represented by a assembly processes, with the most relevant productive This is based on the integration between a hardware Section 4 proposes a case study of a manual production network of commercial MOCAP devices originally performances of the operator measurable by the MAS. and a software architecture. The former is represented by a assembly processes, along with the most relevant productive This system is based on the integration between a hardware 4 proposes case studymeasurable of a manual network of commercial MOCAP originally performances of theaa operator by production the MAS. and a software architecture. The former devices is represented by a Section 4 proposes case studymeasurable of a manual network of commercial MOCAP originally performances of thea operator by production the MAS. and a software architecture. The former devices is represented by a Section Section 4 proposes case study of a manual production network of commercial MOCAP devices originally Copyright 2018commercial IFAC 411 Hosting 2405-8963 © 2018, IFAC (International Federation of Automatic Control) by Elsevier Ltd. All rights reserved. Section 4 proposes a case study of a manual production network of MOCAP devices originally Copyright © 2018 IFAC 411 Peer review©under of International Federation of Automatic Copyright 2018 responsibility IFAC 411Control. Copyright © 2018 IFAC 411 10.1016/j.ifacol.2018.08.329 Copyright © 2018 IFAC 411
IFAC INCOM 2018 412 Bergamo, Italy, June 11-13, 2018
Emilio Ferrari et al. / IFAC PapersOnLine 51-11 (2018) 411–416
process to assembly a microwave oven in an industrial assembly station, whereas Section 5 analyses and discusses the case study main results and outcomes. Finally, Section 6 provides the paper conclusions and it suggests further research opportunities.
analysed production process (Bortolini et al., 2017c; Bortolini et al., 2016). Du and Duffy (2007) represent the first literature contribution to the adoption of MOCAP technologies in the industrial environment with an active marker system aimed at the operator tracking during manual assembly activities. The limitations of this technology are overcame by Nguyen et al. (2013) which first exploit marker-less optical MOCAP in a manufacturing process. The developed system is distinguished by promising results both in term of system set up easiness and measurement accuracy. The latest contributions to this field of research are represented by Agethen et al. (2016a, 2016b) and Geiselhart et al. (2016) which adopted the aforementioned MOCAP technology to monitor the discrepancies between the real assembly activities performed by a human operator and the ones forecasted by traditional simulation models and methods. The measurement of the operator productive performances is limited to the evaluation of the travelled distance and the task duration with almost no considerations concerning the operator interaction with the assembly station 3D layout (e.g. component picking from storage locations, used portions of workbench, visited station areas, etc.).
2. LITERATURE REVIEW Aim of MOCAP technologies is the accurate digitalization of the human movements through tracking and recording the posture evolution of the different body parts over time (Oyekan et al., 2017). Three are the main technologies adopted for MOCAP purpose. Inertial MOCAP exploits proper sensors, called inertial measurement units (IMUs), displaced on the human body which measure their acceleration, rotation and magnetic field on three orthogonal axes. These data are processed through specific algorithms to offer a proper representation of the human movements and postures. However, this technology does not guarantee an accurate absolute position of the limbs due to a positional drift which compounds over the recording time. Thus, this technology is distinguished by a major drawback which limits its adoption in real industrial environment, as in manufacturing or assembly shop floor.
Considering the revised literature and as far as the Authors knowledge, this manuscript represents the first contribution concerning the adoption of MOCAP technologies for the monitoring and assessment of manual manufacturing and assembly processes which analyses the operator activities in relation to the production system 3D environment. The developed Motion Analysis System exploits a network of depth cameras to avoid any interference with the operator activities while recording its postures without any use of cumbersome suits. This technology ensures an accurate measurement of the operator absolute positons in the production 3D layout and in relation to the machines, tools and components displaced in the shop floor area providing an in-depth assessment of the human productive performances based on a time and space analysis. Indeed, the MAS automatically and quantitatively evaluates a set of Key Performance Indicators (KPIs) concerning the production process as the different paths travelled by the operators, the added value portion of the working time and the instant, frequency and duration of picking activities from each possible storage location of the shop floor.
Marker-based optical MOCAP overcome this disadvantage exploiting active or passive markers displaced in specific parts of the human body. A network of camera simultaneously detects at a constant frequency the position of each marker in their field of view. The interpolation of this information is adopted to provide the absolute positon of each marker in a 3D environment, measured for each monitored instant (Tian and Duffy, 2011). Both inertial and markerbased optical MOCAP are affected by a major limitation for their adoption in the industrial environment. The monitored operator necessarily has to wear cumbersome and uncomfortable suit where the IMUs and markers are mounted. This major disadvantage is overcame by the latest advance in the MOCAP technologies, namely marker-less optical MOCAP. This technology frees the operator to perform his movements and activities in whatever outfit without wearing any suit nor having sensors displaced on the body (Puthenveetil et al., 2015). The images resulting from the depth camera types are properly processed by computer vision algorithms aimed at distinguishing the human body movements from the background scene.
3. MOTION ANALYSIS SYSTEM MAS is an original hardware/software architecture developed to facilitate the monitoring of manual manufacturing and assembly processes and to assess their productive performances. The hardware is represented by a marker-less MOCAP system whose purpose is to digitalize the human operator movements without any interference with his actions. The developed software exploits the big volume of data which represents the operator digitalization and to match them with the 3D layout of the manufacturing or assembly system and dynamically assess the productive performances. The following Fig. 1 proposes the MAS conceptual framework presenting the required input information, their
The adoption of MOCAP technologies in the industrial environment achieved a remarkable importance with the diffusion of the Smart Factory concept. The flexibility and reconfigurability required to the involved manufacturing and assembly processes forces the human operators to focus their effort on complex and non-repetitive tasks (Bortolini et al., 2017b). Within this context, MOCAP technologies are of major help to track the operator activities and exploit the recorded motion data. Indeed, this information could be adopted to capitalize the operator knowledge and expertise as well to optimize the manual manufacturing and assembly tasks often distinguished by high uncertainty and low standardization, thus maximizing the performances of the 412
IFAC INCOM 2018 Bergamo, Italy, June 11-13, 2018
Emilio Ferrari et al. / IFAC PapersOnLine 51-11 (2018) 411–416
413
integration with the hardware and software architectures and the output obtained.
Fig. 1. MAS conceptual framework. Fig. 2. Optimal configuration of the MAS hardware architecture.
3.1 Hardware architecture The MAS adopts marker-less optical MOCAP to track the movements of a human operator in a manufacturing or assembly area. The hardware architecture is made of up to four depth cameras each connected to a standard PC through a USB port. The PCs communicate through a Wi-fi network to provide a unique representation of the operator movements in the monitored area. First, an accurate calibration process is carried out to determine the location of each camera in the 3D environment. Then, the cameras are synchronized to simultaneously capture the operator motions within a workstation. The camera calibration and synchronization enables to obtain a stream of data which represents the digitalization of the operator movements over time. The technical features of the adopted depth cameras are summarized in the following: time of flight technology; RGB sensor resolution of 1080p at 30 Hz; depth sensor resolution of 512x424 at 30 Hz; minimum/maximum tracking distance: 1.5/6.0 m; horizontal/vertical field of view: 70°/60°; tracked human body of 26 body joints simultaneously.
3.2 Software architecture An original software architecture is developed in MatlabTM to provide a detailed analysis concerning the productive performance of human operators within the monitored manufacturing or assembly area. The software exploits the operator body digitalization provided by the hardware architecture which offers the position of the body joints for each tracked frame. The file format .TRC (Meredith and Maddock, 2001) is adopted to guarantee a standard communication between the hardware and software architectures. The software architecture integrates the information related to the operator movements and postures with the workstation layout to automatically and quantitatively assess a set of KPIs which measure the production system performances. The required input data to provide to the developed software are the followings:
Human operator physical features, height in particular. Workstation 3D layout. Dimensions and positon of the workbenches, racks, shelves, boxes, pallets and machines located in the monitored area. Information of the product to be assembled or manufacture, as the bill of materials, dimensions and weight. Information concerning the product component and workstation tools (dimensions, weight, etc.) Components and tools location within the workstation. Along with the aforedescribed input data, the novel concept of Control Volume (CV) is proposed within this research to provide an in-depth analysis concerning the system productive performances. A CV is a solid geometric figure of any shape displaced in whatever workstation location. For each CV, the software user has to define its dimensions and
An experimental campaign is carried out to determine the best trade-off configurations of the MAS hardware. An almost circular monitored area of 25 m2 ensures an acceptable tracking precision of about 3-4 cm. Indeed, this is the maximum measured difference between the real location of an operator joint and its digital representation, whatever the considered body part is. Beyond this optimal configuration represented by Fig. 2, the hardware architecture can be flexibly and easily adapted to several other workstation layouts thanks to the Wi-fi connection between the depth cameras. Different manufacturing and assembly shop floors have been monitored exploiting the presented hardware architecture considering the constraints and limitations which distinguish real production processes. The MAS tracking precision of 5-6 cm is ensured for all those monitored layouts distinguished by an area within 30-35m2.
413
IFAC INCOM 2018 414 Bergamo, Italy, June 11-13, 2018
Emilio Ferrari et al. / IFAC PapersOnLine 51-11 (2018) 411–416
3D position in the station layout. A proper procedure is developed and embedded into the software architecture to evaluate the dynamic interaction of the operator with the displaced CVs. This procedure calculates the entry and exit instants of a selected body joint or the operator himself from a specific CV. A proper displacement of the CVs in the workstation area enables to automatically monitor the operator interaction with the manufacturing or assembly environment. For instance, a CV placed on a shelf helps the analyst to monitor the instant, duration and frequency of the picking of the component stored on the considered shelf. Fig. 4 and Table 2 exemplifies these features of the software architecture.
Vertical movements due to lifting and lowering activities. Operator travelled paths and hand trajectories. Workspace usage for both the station and the benches. Picking activity in-depth assessment (duration, frequency, etc.) Working time partitioning, e.g. distinction between added-value (task execution) and no added-value (walking, picking, etc.) activities. 4. CASE STUDY
To test and validate the developed MAS, an industrial manual assembly process of a commercial microwave oven is considered. The product requires 68 components to be assembled through 27 tasks with a cycle time of 6.4 min/pcs. The 25m2 assembly station layout is well known and presented in Fig. 5. The station is equipped with a workbench on which the assembly tasks are performed, a trolley to provide the necessary tools to the operator and a bunch of locations to store the product components, namely a rack for small-size components and a shelf, a box and a pallet for medium-large ones. The geometric dimensions and the 3D location within the assembly station of each of these elements are well known, as well as the storage location for each component. The depth cameras adopted to track the operator movements within the station are displaced almost in the corners of the monitored area.
Fig. 4. Assessment of component picking from shelves through the Control Volume Analysis. Table 2. Visited locations of the operator hands through the Control Volume Analysis. Initial instant [sec]
Final instant [sec]
Body joints
Accessed Control Volume
Stored component /tool
56.34
58.92
right hand
level2.shelf
carter
Fig. 5. Assembly station layout.
187.45
193.76
left hand
box1
cog
436.45
443.28
right & left hand
level1.shelf
packaging
…
…
…
…
…
The developed CV analysis enables to analyse in detail the operator interaction with specific elements of the assembly station. For the considered case study, of major interest is the picking activity of small components from the 8 bins displaced on the 2 level rack behind the workbench. The following Fig. 6 presents the 8 CVs defined for this purpose. The CV dimensions and 3D location are well known.
The software architecture adopts all the aforementioned input information along with a proper displacement of the CVs to automatically and quantitatively calculate a set of KPI to evaluate the system productive performances. The most relevant productive KPIs provided by the software architecture are the followings:
Travelled distance and velocity of the different body parts.
414
Emilio Ferrari et al. / IFAC PapersOnLine 51-11 (2018) 411–416
415
6
6 5 4 3 2 1 0
n° accesses
5 4 3 2 1 Peripheral 2
Peripheral 1
Top 2
Top 1
Lateral 2
Lateral 1
Central 2
Central 1
0
average access duration [sec]
IFAC INCOM 2018 Bergamo, Italy, June 11-13, 2018
Control Volume access frequency access duration Fig. 6. Control Volume displacement on the station rack.
Fig. 7. Control Volume Analysis for picking from rack bins.
5. RESULTS AND DISCUSSION
A further productive KPI offered by the MAS is the station area utilization by the operator. The operator spends most of the cycle time (59%) close to the workbench in a 0.8 m2 area to perform the assembly tasks or the picking activities from the rack. Concerning the storage locations used for medium and large-size components, 9.8% of the cycle time is required to handle components on the pallet or in the box, whereas just 4.9% is needed to access to the 3 levels of the shelf. The results provided by the analysis of the operator movements in the station area suggest to further investigate the different cycle time components.
The MAS is adopted to assess the manual assembly process performed by the operator in the monitored station. The MAS automatically and quantitatively evaluates a set of KPIs to measure the productive performance of the assembly system presented in the case study. Table 3 summarizes the movements of the operator and his hands. The operator walks for a total distance of 29.7 m during the cycle time at an average speed of 5.8 m/min. The alarming value of 5.6 m of vertical movements is experienced due to lifting and lowering activities for component picking from storage locations at ground level. Table 3. Assessment of the operator and upper limbs movements. Body part
Travelled distance [m]
Vertical drop [m]
Average speed [m/min]
Right hand
78.4
28.3
12.8
Left hand
73.5
26.9
12.4
Operator
29.7
5.6
5.8
The CV analysis focuses on the component picking from the rack bins (Fig. 7). The operator retrieves 11 times per cycle the components stored in the Peripheral bins of the rack, whereas the Central bins are underused (1 access only) considering their convenient location close to the workbench. Taking into account this outcome and the identical dimensions of the bins, a possible improvement could be the inversion of the Central and Peripheral bins to minimize the duration of the most performed activities. Furthermore, the CV analysis evaluates the average access duration in each monitored bin. The results proposed in Fig. 7 request a further investigation concerning the Top 1 bin, since the picking duration from this bin is more than double compared to the other bins (5.3 sec vs 2.1 sec on average). The sharp glass cover for the microwave display stored in this bin requires a particular attention by the operator during the picking which fully justifies the long duration of this activity.
Fig. 8. Utilization of the assembly station area by the operator. The MAS is able to automatically and quantitatively provide the cycle time partition between the added-value and nonadded-value activities. For the considered case study, the execution of assembly tasks on the station workbench is the only activity which adds value to the production process and it represents less than half of the cycle time (48%). Concerning the non-added-value activities, the operator walks for 22% of the cycle time, whereas he performs picking activities for the remaining 30%. Most of this time is required to retrieve components from storage locations far 415
IFAC INCOM 2018 416 Bergamo, Italy, June 11-13, 2018
Emilio Ferrari et al. / IFAC PapersOnLine 51-11 (2018) 411–416
from the workbench (e.g. box and shelf), whereas picking from the rack accounts for just 11% of the cycle time. Finally, the time required to access to the trolley to handle the tools used in the assembly process is negligible (4% of cycle time).
Walk Path Segmentation and Drift Analysis in Manual Assembly. Procedia CIRP, 52, 286-291. Agethen, P., Otto, M., Mengel, S., & Rukzio, E. (2016b). Using Marker-less Motion Capture Systems for Walk Path Analysis in Paced Assembly Flow Lines. Procedia CIRP, 54, 152-157. Bortolini, M., Faccio, M., Ferrari, E., Gamberi, M., & Pilati, F. (2017c). Time and energy optimal unit-load assignment for automatic S/R warehouses. International Journal of Production Economics, 190, 133-145. Bortolini, M., Faccio, M., Gamberi, M., & Pilati, F. (2017b). Multi-objective assembly line balancing considering component picking and ergonomic risk. Computers & Industrial Engineering, 112, 348-367. Bortolini, M., Ferrari, E., Gamberi, M., Pilati, F., & Faccio, M. (2017a) Assembly system design in the Industry 4.0 era: a general framework. IFAC-PapersOnLine, 50(1), 5700-5705. Bortolini, M., Faccio, M., Gamberi, M., & Pilati, F. (2016). Multi-objective design of multi-modal fresh food distribution networks. International Journal of Logistics Systems and Management, 24(2), 155-177. Ceseracciu, E., Sawacha, Z., & Cobelli, C. (2014). Comparison of Markerless and Marker-Based Motion Capture Technologies through Simultaneous Data Collection during Gait: Proof of Concept, PloS one, 9(3), 1–7. Du J. C. & Duffy, V. G. (2007). A methodology for assessing industrial workstations using optical motion capture integrated with digital human models. Occupational Ergonomics, 7(1), 11-25. Faccio, M., Gamberi, M., Pilati, F., & Bortolini, M. (2015). Packaging strategy definition for sales kits within an assembly system. International Journal of Production Research, 53(11), 3288-3305. Geiselhart, F., Otto, M., & Rukzio, E. (2016). On the use of Multi-Depth-Camera based Motion Tracking Systems in Production Planning Environments. Procedia CIRP, 41, 759764. Meredith, M., & Maddock, S. (2001). Motion capture file formats explained. Department of Computer Science, University of Sheffield, 211, 241-244. Nguyen, T. D., Kleinsorge, M., Postawa, A., Wolf, K., Scheumann, R., Krüger, J., & Seliger, G. (2013). Human centric automation: using marker-less motion capturing for ergonomics analysis and work assistance in manufacturing processes. In Proceedings of the 11th Global Conference on Sustainable Manufacturing (GCSM)—Innovative Solutions, Berlin, 586-592). Oyekan, J., Prabhu, V., Tiwari, A., Baskaran, V., Burgess, M., & Mcnally, R. (2017). Remote real-time collaboration through synchronous exchange of digitised human– workpiece interactions. Future Generation Computer Systems, 67, 83-93. Puthenveetil, S. C., Daphalapurkar, C. P., Zhu, W., Leu, M. C., Liu, X. F., Gilpin-Mcminn, J. K., & Snodgrass, S. D. (2015). Computer-automated ergonomic analysis based on motion capture and assembly simulation. Virtual Reality, 19(2), 119-128.
Fig. 9. Cycle time partition between the added-value and nonadded-value activities. 6. CONCLUSIONS This paper presents an original Motion Analysis System (MAS) developed for the monitoring and evaluation of manual manufacturing and assembly processes through motion capture (MOCAP) technologies. The MAS hardware is represented by a network of depth cameras, e.g. a markerless optical MOCAP technology, whose purpose is the digitalization of human movements and posture at 30 Hz with no interferences with the performed activity. A customized software architecture is developed to exploit these data in relation to the 3D layout of the workstation where the operator performs his activities. The original Control Volume (CV) Analysis proposed within this context provides an indepth assessment concerning the operator time and space interaction with the workstation elements, e.g. machines, workbench, tools, product components, racks, shelves, etc. This information is exploited by the MAS to automatically and quantitatively evaluate a set of productive KPIs to assess the monitored manual manufacturing or assembly process. A case study of a manual assembly process of a microwave oven is adopted to test and validate the MAS. The MAS performs a Control Volume Analysis for the component picking from the rack bins highlighting the picking frequency and duration from each location. Finally, the MAS provides an accurate cycle time partition between the added-value (e.g. assembly) and non-added-value activities (e.g. walking and picking). Further research should focus on the meaningful and powerful output data provided by the MAS to propose a relayout of the monitored workstation considering the weaknesses and opportunities highlighted by the MAS. REFERENCES Agethen, P., Otto, M., Gaisbauer, F., & Rukzio, E. (2016a). Presenting a Novel Motion Capture-based Approach for
416
ScienceDirect
IFAC PapersOnLine 51-11 (2018) 411–416
Motion Motion Analysis Analysis System System for for the the digitalization digitalization and and assessment assessment of of manual manual Motion Analysismanufacturing System for the and digitalization and assessment of manual assembly processes Motion Analysismanufacturing System for the and digitalization assessment of manual assembly and processes Motion Analysismanufacturing System for the and digitalization assessment of manual assembly and processes manufacturing and assembly processes † Emilio Mauro Francesco †, Alberto Regattieri* manufacturing and assembly processes , Alberto Regattieri* Emilio Ferrari*, Ferrari*, Mauro Gamberi*, Gamberi*, Francesco Pilati* Pilati* †
Emilio Ferrari*, Mauro Gamberi*, Francesco Pilati* †, Alberto Regattieri* Emilio Ferrari*, Mauro Gamberi*, Francesco Pilati* †, Alberto Regattieri* * Department of Engineering, Viale del 2, Alberto Regattieri* Emilio Ferrari*, Mauro Gamberi*, Pilati* FrancescoViale * University University of of Bologna, Bologna, Department of Industrial Industrial Engineering, del ,Risorgimento Risorgimento 2, 40136 40136 Bologna, Bologna, Italy Italy * University of Bologna, Department of Industrial Engineering, Viale del Risorgimento 2, 40136 Bologna, Italy * University of Bologna, Department of Industrial Engineering, Viale del Risorgimento 2, 40136 Bologna, Italy †† Corresponding Author: [email protected] * University of Bologna, Department of Industrial Engineering, Viale del Risorgimento 2, 40136 Bologna, Italy Corresponding Author: [email protected] † Corresponding Author: [email protected] † Corresponding Author: [email protected] † Corresponding Author: [email protected] Abstract: Abstract: Manual Manual manufacturing manufacturing and and assembly assembly systems systems radically radically changed changed after after the the advent advent of of the the fourth fourth industrial revolution (Industry 4.0). The production paradigm shift to mass personalization involves the Abstract: Manual manufacturing and assembly systems radically changed after the advent of the fourth industrial revolution (Industry 4.0). The production paradigm shift to mass personalization involves the Abstract: Manual manufacturing andphase. assembly systems radically changed after the of advent of the fourth customers since the product design The remarkable variety and complexity these processes is industrial revolution (Industry 4.0). The production paradigm shift to mass personalization involves the Abstract: Manual manufacturing and assembly systems radically changed after the advent of the fourth customers since the product design phase. The remarkable variety and complexity of these processes is industrial revolution (Industry 4.0). The production paradigm shift to and mass personalization involves the tackled by highly skilled human operator, which perform value added non-repetitive tasks. Thus, the customers since the product design phase. The remarkable variety and complexity of these processes is industrial revolution (Industry 4.0). The production paradigm shift to mass personalization involves the tackled by since highlythe skilled human operator, which perform value added and non-repetitive tasks. Thus, is customers product design phase. The remarkable variety and complexity of these processes virtualization of manual operations represents a great opportunity to monitor, analyze and successively tackled by highly skilled human operator, which perform value added and non-repetitive tasks. Thus, the customers since the product design phase. The remarkable variety and complexity of these processes is virtualization of manual represents great opportunity to monitor, analyze and successively tackled by these highly skilled operations human operator, whichaa perform value added and non-repetitive tasks. Thus, the optimize production processes. Within this context, this research proposes an original virtualization of manual operations represents great opportunity to monitor, analyze and successively tackled by these highly skilled operations human operator, whicha perform value added and non-repetitive tasks. the optimize production processes. Within this opportunity context, this research proposes an Thus, original virtualization of manual represents great to monitor, analyze and successively hardware/software architecture called Motion Analysis System (MAS) developed to monitor and to optimize these production processes. Within this opportunity context, this research proposes an original virtualization of manual operations represents a great to monitor, analyze and successively hardware/software architecture called Motion Analysis System (MAS) developed to monitor and to optimize manual these manufacturing production processes. Within this context, this research proposes anThe original evaluate and assembly processes through motion capture technologies. MAS hardware/software architecture called Motion Analysis System (MAS) developed to monitor and to optimize these production processes. Within this context, this research proposes an original evaluate manual manufacturing and assembly processes through motion capture technologies. The MAS hardware/software architecture called Motion Analysis System (MAS) developed to monitor and to adopts a hardware architecture represented by a network of depth cameras, e.g. a marker-less optical evaluate manual manufacturing and assembly processes through motion capture technologies. The MAS hardware/software architecture and called Motion Analysis System (MAS) developed to monitor to adopts a manual hardware represented byprocesses a network of depth cameras, e.g. a marker-less optical evaluate manufacturing assembly through motion capture technologies. Theand MAS motion to digitalize the operator movements and postures at 30 Hz while he production adopts acapture, hardware architecture represented byprocesses a network of depth cameras, e.g. aperforms marker-less optical evaluate manual manufacturing and assembly through motion capture technologies. The MAS motion capture, to digitalize the operator movements and postures at 30 Hz while he performs production adopts hardware architecture represented byexploits a network ofdata depth cameras, e.g.he aperforms marker-less optical activities. A software architecture these in to workstation 3D layout motion aacapture, to digitalize the operator movements and postures atrelation 30 Hz while production adopts hardware architecture represented byexploits a network ofdata depth cameras, e.g.he aperforms marker-less activities. A customized customized software architecture these inat relation to the the workstation 3D optical layout motion capture, to digitalize the operator movements and postures 30 Hz while production to automatically and quantitatively evaluate a set of productive KPIs. The MAS represents a reliable, activities. A customized software architecture exploits these data in relation to the workstation 3D layout motion capture, to digitalize the operator movements and postures at 30 Hz while he performs production to automatically and quantitatively evaluate aexploits set of productive KPIs. ThetoMAS represents a3Dreliable, activities. A customized software architecture these data in relation the workstation layout powerful and meaningful tool for production managers and practitioners able to provide an in-depth time to automatically and quantitatively evaluate a set of productive KPIs. The MAS represents a reliable, activities. A customized software architecture exploits these data in relation to the workstation 3D layout powerful and meaningful tool for production and practitioners able MAS to provide an in-depth time to automatically and quantitatively evaluate managers aby setanofoperator, productive KPIs. The represents a reliable, and space analysis of the activities performed as the walking paths, the accessed picking powerful and meaningful tool for production managers and practitioners able to provide an in-depth time to automatically and evaluate managers aby setanofoperator, productive KPIs. The MAS represents a reliable, and spaceand analysis of quantitatively the tool activities performed as the walking paths, the accessed picking powerful meaningful for production and practitioners able to provide an in-depth time locations, the used space within the workstation and the hand movements. Finally, this contribution ends and space analysis of the activities performed by an operator, as the walking paths, the accessed picking powerful and meaningful tool forthe production managers practitioners able to provide an in-depth time locations, the used space within workstation theand hand movements. Finally, this ends and space analysis of the activities byand an operator, as the walking paths, the contribution accessed picking with aa real industrial application of the MAS which analyzes the manual assembly processes of locations, the used space within theperformed workstation and the hand movements. Finally, this contribution ends and space analysis of the activities performed by an operator, as the walking paths, the accessed picking with real industrial application of the MAS which analyzes the manual assembly processes of aa locations, the usedThe space within theof workstation and the hand movements. Finally, this contribution ends microwave oven. system setup is presented and discussed along with the case study key results and with a real industrial application the MAS which analyzes the manual assembly processes of locations, the usedThe space within theof workstation the hand movements. Finally, this contribution microwave oven. system setup is presented discussed along the case study key resultsends andaa with a real industrial application the MAS and which analyzes thewith manual assembly processes of findings. microwave oven. The system setup is presented and discussed along with the case study key results and with a real industrial application the MAS and which analyzes thewith manual assembly processes findings. microwave The system setup of is presented discussed along the case study key resultsof anda findings. oven. microwave oven. The system setup is presented and discussed along with the case study key results and © 2018, IFAC (International Federation of Automatic Control) Hosting by Elsevier Ltd. All rights reserved. findings. findings. Keywords: Motion capture, Depth camera, Control volume, Industry 4.0, Smart factory, Manufacturing, Keywords: Motion capture, Depth camera, Control volume, Industry 4.0, Smart factory, Manufacturing, Assembly, Assessment, KPI,Depth Performance, Keywords: Motion capture, camera, Digitalization. Control volume, Industry 4.0, Smart factory, Manufacturing, Assembly, Assessment, KPI,Depth Performance, Keywords: Motion capture, camera, Digitalization. Control volume, Industry 4.0, Smart factory, Manufacturing, Assembly, Assessment, KPI, Performance, Digitalization. Keywords: Motion capture, Depth camera, Control volume, Industry 4.0, Smart factory, Manufacturing, Assembly, Assessment, KPI, Performance, Digitalization. Assembly, Assessment, KPI, Performance, Digitalization. conceived conceived for for gaming gaming and and accurately accurately customized customized to to extend extend 1. 1. INTRODUCTION INTRODUCTION their applicability to the industrial environment, whereas the conceived for gaming and accurately customized to extend their applicability to the industrial environment, whereas the 1. INTRODUCTION conceived for gaming and accurately customized toproperly extend latter is an original software application The industrial environment is currently experiencing its their applicability to the industrial environment, whereas the 1. INTRODUCTION conceived for gaming and accurately customized to extend latter is an original software application properly The industrial environment is currently experiencing its their applicability to the industrial environment, whereas the 1. INTRODUCTION programmed to dynamically assess the operator productive fourth industrial revolution (Bortolini et al., 2017a; Faccio et latter is an original software application properly The industrial environment is currently experiencing its their applicability to the industrial environment, whereas the programmed to dynamically assess theapplication operator productive fourth industrial environment revolution (Bortolini et al., experiencing 2017a; Faccio its et latter is an original software properly The industrial is currently performance. The adoption of the MAS in an industrial al., 2015). The use of ubiquitous sensors in factory shop programmed to dynamically assess the operator productive fourth industrial revolution (Bortolini et al., 2017a; Faccio et latter is an original software application properly The industrial environment is currently experiencing its performance. toThe adoption of the the MAS in an productive industrial al., 2015). The revolution use of ubiquitous in factory shop dynamically assess operator fourth industrial (Bortolinisensors et al.,networks 2017a; Faccio et programmed environment to and automatically evaluate floors connected through communication enables adoption of the MAS in an productive industrial al., 2015). The revolution use of ubiquitous in factory shop programmed toThe dynamically assess operator fourth industrial (Bortolinisensors et al.,networks 2017a; Faccio et performance. environment enables enables to accurately accurately andthe automatically evaluate floors connected through communication enables performance. The adoption of the MAS in an industrial al., 2015). The use of ubiquitous sensors in factory shop the operator tasks and to quantitatively measure his the real-time communication of machines, operators, environment enables to accurately and automatically evaluate floors connected through communication networks enables performance. The adoption of the MAS in an industrial al., 2015). The use of ubiquitous sensors in factory shop the operator tasks and to quantitatively measure his the real-time communication of machines, operators, environment enables toand accurately andwith automatically evaluate floors connected through communication networks enables productive performances. These deal the time and space products, tools and costumers defining the so-called Smart the operator tasks to quantitatively measure his the real-time communication of machines, operators, environment enables to accurately and automatically evaluate floors connected through communication networks enables productive performances. These deal with the time and space products, tools and costumers defining the so-called Smart the operator tasks and to quantitatively measure his the real-time communication ofof manual machines, operators, aspects of the operator activities as the movements to perform Factory concept. The virtualization manufacturing productive performances. These deal with the time and space products, tools and costumers defining the so-called Smart the operator tasks and to quantitatively measure his the real-time communication ofof manual machines, operators, aspects of the operator activities as thewith movements toand perform Factory concept. Thecostumers virtualization manufacturing productive performances. These deal the time space products, tools and defining the so-called Smart aa task, the travelled of specific body parts and the and assembly operations (e.g. virtual reality) represents aa productive aspects of the operatordistance activities as thewith movements toand perform Factory concept. Thecostumers virtualization of manual manufacturing performances. These deal the time space products, tools and defining the so-called Smart task, the travelled distance of specific body parts and the and assembly operations (e.g. virtual reality) represents aspects of the operatordistance activities the movements to perform Factory concept. The virtualization of manual manufacturing locations in relation to 3D layout of great opportunity to monitor, analyse and successively aoccupied task, the travelled ofas body parts and and assembly operations (e.g. virtual reality) represents of the operatordistance activities asspecific the to perform Factory concept. The of manual occupied locations in relation to movements 3D layout of the the great opportunity to virtualization monitor, analyse andmanufacturing successivelyaa aspects a task, the travelled of specific body parts and and assembly operations (e.g. virtual reality) represents manufacturing or assembly area. optimize the production processes distinguished by the occupied locations in relation to 3D layout of the the great opportunity to monitor, analyse and successively a task, the travelled distance of specific body parts and the and assembly operations (e.g. virtual reality) represents a manufacturing or assembly area. optimize the production processes distinguished by the occupied locations in relation to 3D layout of the great opportunity to monitor, analyse and successively aforementioned features. Motion capture (MOCAP) manufacturing or assembly area. optimize the production processes distinguished by the occupied locations in relation to 3D layout of the great opportunity to monitor, analyse and successively aforementioned features. processes Motion distinguished capture (MOCAP) According to this purpose, this manuscript is organized as manufacturing or assembly area. optimize the production by the According to this purpose, area. this manuscript is organized as it it technologies are of strong help to target this goal. Aim of aforementioned features. Motion capture (MOCAP) manufacturing or assembly optimize the production processes distinguished by the technologies are offeatures. strong help to target this goal. Aim of According follows. Section investigates the different technologies to this 2 purpose, this manuscript is organized as it aforementioned Motion capture (MOCAP) follows. Section 2 investigates the different technologies these systems is to accurately track and record over time the technologies are of strong help to target this goal. Aim of According to this 2purpose, this manuscript istheir organized as it aforementioned Motion capture (MOCAP) these systemsare is tooffeatures. accurately track and record over time the currently available for MOCAP purpose, application follows. Section investigates the different technologies technologies strong of help target this goal. Aim of According to this 2purpose, this manuscript istheir organized as it currently available for MOCAP purpose, application postures and movements aa to human operator aa the 3D these systems is toofaccurately track and record overin time follows. Section investigates the different technologies technologies are strong help to target this goal. Aim of postures and movements of human operator in 3D and adoption in real industrial environment and the most currently available for MOCAP purpose, their application these systems is to accurately track and record over time the follows. Section 2 investigates the different technologies and adoption in realforindustrial andapplication the most environment. postures and is movements of track a human operator a the 3D currently available MOCAP environment purpose, their these systems to accurately and record overin environment. literature contributions concerning the usage of and adoption in real and the most postures and movements of a human operator intime a 3D relevant currently available forindustrial MOCAP environment purpose, their application relevant literature contributions concerning the usage of environment. and adoption in real industrial environment and the most postures and movements of a human operator in a 3D MOCAP for the assessment of the operator productive Considering the presented scenario, this paper proposes an relevant literature contributions concerning the usage of environment. and adoption in real industrial ofenvironment and the most MOCAP for the assessment the operator productive Considering the presented scenario, this paper proposes an relevant literature contributions concerning the usage of environment. performances. Section 3 presents the original MAS original Motion Analysis System (MAS) to digitalize the MOCAP for the assessment of the operator productive Considering the presented scenario, this paper proposes an relevant literature contributions concerning the productive usage of performances. Section 3 presents the original MAS original Motion Analysis System (MAS) to digitalize the MOCAP for the assessment of the operator Considering the presented scenario, this paper proposes an hardware/software architecture developed for the automatic operator activities in whatever manufacturing or assembly performances. Section 3 presents the original MAS original Motion Analysis System (MAS) to digitalize the MOCAP for the assessment of the operator productive Considering the presented scenario, this paper proposes an hardware/software architecture developed for the automatic operator activities in whatever manufacturing or assembly Section 3 presents the MAS original Motion Analysis System (MAS) to digitalize the performances. and quantitative assessment of manual manufacturing and environment and analyse the manual production architecture fororiginal the automatic operator activities in whatever manufacturing or processes. assembly performances. 3 presents original MAS original Motion System (MAS) to digitalize the hardware/software and quantitativeSection assessment ofdeveloped manualthe manufacturing and environment and Analysis analyse the manual production processes. hardware/software architecture developed for the automatic operator activities in whatever manufacturing or assembly assembly processes, along with the most relevant productive This system is based on the integration between a hardware and quantitative assessment of manual manufacturing and environment and analyse the manual production processes. hardware/software architecture developed for the automatic operator activities in whatever manufacturing or assembly assembly processes, along with the most relevant productive This system is based on the integration between a hardware and quantitative assessment of the manual manufacturing and environment and analyse theThe manual production processes. performances of the operator measurable by the MAS. and a software architecture. former is represented by a assembly processes, along with most relevant productive This system is based on the integration between a hardware and quantitative assessment of manual manufacturing and environment and analyse theThe manual production processes. performances of thealong operator measurable by the MAS. and asystem software architecture. former is represented by a assembly processes, with the most relevant productive This is based on the integration between a hardware Section 4 proposes a case study of a manual production network of commercial MOCAP devices originally performances of the operator measurable by the MAS. and a software architecture. The former is represented by a assembly processes, along with the most relevant productive This system is based on the integration between a hardware 4 proposes case studymeasurable of a manual network of commercial MOCAP originally performances of theaa operator by production the MAS. and a software architecture. The former devices is represented by a Section 4 proposes case studymeasurable of a manual network of commercial MOCAP originally performances of thea operator by production the MAS. and a software architecture. The former devices is represented by a Section Section 4 proposes case study of a manual production network of commercial MOCAP devices originally Copyright 2018commercial IFAC 411 Hosting 2405-8963 © 2018, IFAC (International Federation of Automatic Control) by Elsevier Ltd. All rights reserved. Section 4 proposes a case study of a manual production network of MOCAP devices originally Copyright © 2018 IFAC 411 Peer review©under of International Federation of Automatic Copyright 2018 responsibility IFAC 411Control. Copyright © 2018 IFAC 411 10.1016/j.ifacol.2018.08.329 Copyright © 2018 IFAC 411
IFAC INCOM 2018 412 Bergamo, Italy, June 11-13, 2018
Emilio Ferrari et al. / IFAC PapersOnLine 51-11 (2018) 411–416
process to assembly a microwave oven in an industrial assembly station, whereas Section 5 analyses and discusses the case study main results and outcomes. Finally, Section 6 provides the paper conclusions and it suggests further research opportunities.
analysed production process (Bortolini et al., 2017c; Bortolini et al., 2016). Du and Duffy (2007) represent the first literature contribution to the adoption of MOCAP technologies in the industrial environment with an active marker system aimed at the operator tracking during manual assembly activities. The limitations of this technology are overcame by Nguyen et al. (2013) which first exploit marker-less optical MOCAP in a manufacturing process. The developed system is distinguished by promising results both in term of system set up easiness and measurement accuracy. The latest contributions to this field of research are represented by Agethen et al. (2016a, 2016b) and Geiselhart et al. (2016) which adopted the aforementioned MOCAP technology to monitor the discrepancies between the real assembly activities performed by a human operator and the ones forecasted by traditional simulation models and methods. The measurement of the operator productive performances is limited to the evaluation of the travelled distance and the task duration with almost no considerations concerning the operator interaction with the assembly station 3D layout (e.g. component picking from storage locations, used portions of workbench, visited station areas, etc.).
2. LITERATURE REVIEW Aim of MOCAP technologies is the accurate digitalization of the human movements through tracking and recording the posture evolution of the different body parts over time (Oyekan et al., 2017). Three are the main technologies adopted for MOCAP purpose. Inertial MOCAP exploits proper sensors, called inertial measurement units (IMUs), displaced on the human body which measure their acceleration, rotation and magnetic field on three orthogonal axes. These data are processed through specific algorithms to offer a proper representation of the human movements and postures. However, this technology does not guarantee an accurate absolute position of the limbs due to a positional drift which compounds over the recording time. Thus, this technology is distinguished by a major drawback which limits its adoption in real industrial environment, as in manufacturing or assembly shop floor.
Considering the revised literature and as far as the Authors knowledge, this manuscript represents the first contribution concerning the adoption of MOCAP technologies for the monitoring and assessment of manual manufacturing and assembly processes which analyses the operator activities in relation to the production system 3D environment. The developed Motion Analysis System exploits a network of depth cameras to avoid any interference with the operator activities while recording its postures without any use of cumbersome suits. This technology ensures an accurate measurement of the operator absolute positons in the production 3D layout and in relation to the machines, tools and components displaced in the shop floor area providing an in-depth assessment of the human productive performances based on a time and space analysis. Indeed, the MAS automatically and quantitatively evaluates a set of Key Performance Indicators (KPIs) concerning the production process as the different paths travelled by the operators, the added value portion of the working time and the instant, frequency and duration of picking activities from each possible storage location of the shop floor.
Marker-based optical MOCAP overcome this disadvantage exploiting active or passive markers displaced in specific parts of the human body. A network of camera simultaneously detects at a constant frequency the position of each marker in their field of view. The interpolation of this information is adopted to provide the absolute positon of each marker in a 3D environment, measured for each monitored instant (Tian and Duffy, 2011). Both inertial and markerbased optical MOCAP are affected by a major limitation for their adoption in the industrial environment. The monitored operator necessarily has to wear cumbersome and uncomfortable suit where the IMUs and markers are mounted. This major disadvantage is overcame by the latest advance in the MOCAP technologies, namely marker-less optical MOCAP. This technology frees the operator to perform his movements and activities in whatever outfit without wearing any suit nor having sensors displaced on the body (Puthenveetil et al., 2015). The images resulting from the depth camera types are properly processed by computer vision algorithms aimed at distinguishing the human body movements from the background scene.
3. MOTION ANALYSIS SYSTEM MAS is an original hardware/software architecture developed to facilitate the monitoring of manual manufacturing and assembly processes and to assess their productive performances. The hardware is represented by a marker-less MOCAP system whose purpose is to digitalize the human operator movements without any interference with his actions. The developed software exploits the big volume of data which represents the operator digitalization and to match them with the 3D layout of the manufacturing or assembly system and dynamically assess the productive performances. The following Fig. 1 proposes the MAS conceptual framework presenting the required input information, their
The adoption of MOCAP technologies in the industrial environment achieved a remarkable importance with the diffusion of the Smart Factory concept. The flexibility and reconfigurability required to the involved manufacturing and assembly processes forces the human operators to focus their effort on complex and non-repetitive tasks (Bortolini et al., 2017b). Within this context, MOCAP technologies are of major help to track the operator activities and exploit the recorded motion data. Indeed, this information could be adopted to capitalize the operator knowledge and expertise as well to optimize the manual manufacturing and assembly tasks often distinguished by high uncertainty and low standardization, thus maximizing the performances of the 412
IFAC INCOM 2018 Bergamo, Italy, June 11-13, 2018
Emilio Ferrari et al. / IFAC PapersOnLine 51-11 (2018) 411–416
413
integration with the hardware and software architectures and the output obtained.
Fig. 1. MAS conceptual framework. Fig. 2. Optimal configuration of the MAS hardware architecture.
3.1 Hardware architecture The MAS adopts marker-less optical MOCAP to track the movements of a human operator in a manufacturing or assembly area. The hardware architecture is made of up to four depth cameras each connected to a standard PC through a USB port. The PCs communicate through a Wi-fi network to provide a unique representation of the operator movements in the monitored area. First, an accurate calibration process is carried out to determine the location of each camera in the 3D environment. Then, the cameras are synchronized to simultaneously capture the operator motions within a workstation. The camera calibration and synchronization enables to obtain a stream of data which represents the digitalization of the operator movements over time. The technical features of the adopted depth cameras are summarized in the following: time of flight technology; RGB sensor resolution of 1080p at 30 Hz; depth sensor resolution of 512x424 at 30 Hz; minimum/maximum tracking distance: 1.5/6.0 m; horizontal/vertical field of view: 70°/60°; tracked human body of 26 body joints simultaneously.
3.2 Software architecture An original software architecture is developed in MatlabTM to provide a detailed analysis concerning the productive performance of human operators within the monitored manufacturing or assembly area. The software exploits the operator body digitalization provided by the hardware architecture which offers the position of the body joints for each tracked frame. The file format .TRC (Meredith and Maddock, 2001) is adopted to guarantee a standard communication between the hardware and software architectures. The software architecture integrates the information related to the operator movements and postures with the workstation layout to automatically and quantitatively assess a set of KPIs which measure the production system performances. The required input data to provide to the developed software are the followings:
Human operator physical features, height in particular. Workstation 3D layout. Dimensions and positon of the workbenches, racks, shelves, boxes, pallets and machines located in the monitored area. Information of the product to be assembled or manufacture, as the bill of materials, dimensions and weight. Information concerning the product component and workstation tools (dimensions, weight, etc.) Components and tools location within the workstation. Along with the aforedescribed input data, the novel concept of Control Volume (CV) is proposed within this research to provide an in-depth analysis concerning the system productive performances. A CV is a solid geometric figure of any shape displaced in whatever workstation location. For each CV, the software user has to define its dimensions and
An experimental campaign is carried out to determine the best trade-off configurations of the MAS hardware. An almost circular monitored area of 25 m2 ensures an acceptable tracking precision of about 3-4 cm. Indeed, this is the maximum measured difference between the real location of an operator joint and its digital representation, whatever the considered body part is. Beyond this optimal configuration represented by Fig. 2, the hardware architecture can be flexibly and easily adapted to several other workstation layouts thanks to the Wi-fi connection between the depth cameras. Different manufacturing and assembly shop floors have been monitored exploiting the presented hardware architecture considering the constraints and limitations which distinguish real production processes. The MAS tracking precision of 5-6 cm is ensured for all those monitored layouts distinguished by an area within 30-35m2.
413
IFAC INCOM 2018 414 Bergamo, Italy, June 11-13, 2018
Emilio Ferrari et al. / IFAC PapersOnLine 51-11 (2018) 411–416
3D position in the station layout. A proper procedure is developed and embedded into the software architecture to evaluate the dynamic interaction of the operator with the displaced CVs. This procedure calculates the entry and exit instants of a selected body joint or the operator himself from a specific CV. A proper displacement of the CVs in the workstation area enables to automatically monitor the operator interaction with the manufacturing or assembly environment. For instance, a CV placed on a shelf helps the analyst to monitor the instant, duration and frequency of the picking of the component stored on the considered shelf. Fig. 4 and Table 2 exemplifies these features of the software architecture.
Vertical movements due to lifting and lowering activities. Operator travelled paths and hand trajectories. Workspace usage for both the station and the benches. Picking activity in-depth assessment (duration, frequency, etc.) Working time partitioning, e.g. distinction between added-value (task execution) and no added-value (walking, picking, etc.) activities. 4. CASE STUDY
To test and validate the developed MAS, an industrial manual assembly process of a commercial microwave oven is considered. The product requires 68 components to be assembled through 27 tasks with a cycle time of 6.4 min/pcs. The 25m2 assembly station layout is well known and presented in Fig. 5. The station is equipped with a workbench on which the assembly tasks are performed, a trolley to provide the necessary tools to the operator and a bunch of locations to store the product components, namely a rack for small-size components and a shelf, a box and a pallet for medium-large ones. The geometric dimensions and the 3D location within the assembly station of each of these elements are well known, as well as the storage location for each component. The depth cameras adopted to track the operator movements within the station are displaced almost in the corners of the monitored area.
Fig. 4. Assessment of component picking from shelves through the Control Volume Analysis. Table 2. Visited locations of the operator hands through the Control Volume Analysis. Initial instant [sec]
Final instant [sec]
Body joints
Accessed Control Volume
Stored component /tool
56.34
58.92
right hand
level2.shelf
carter
Fig. 5. Assembly station layout.
187.45
193.76
left hand
box1
cog
436.45
443.28
right & left hand
level1.shelf
packaging
…
…
…
…
…
The developed CV analysis enables to analyse in detail the operator interaction with specific elements of the assembly station. For the considered case study, of major interest is the picking activity of small components from the 8 bins displaced on the 2 level rack behind the workbench. The following Fig. 6 presents the 8 CVs defined for this purpose. The CV dimensions and 3D location are well known.
The software architecture adopts all the aforementioned input information along with a proper displacement of the CVs to automatically and quantitatively calculate a set of KPI to evaluate the system productive performances. The most relevant productive KPIs provided by the software architecture are the followings:
Travelled distance and velocity of the different body parts.
414
Emilio Ferrari et al. / IFAC PapersOnLine 51-11 (2018) 411–416
415
6
6 5 4 3 2 1 0
n° accesses
5 4 3 2 1 Peripheral 2
Peripheral 1
Top 2
Top 1
Lateral 2
Lateral 1
Central 2
Central 1
0
average access duration [sec]
IFAC INCOM 2018 Bergamo, Italy, June 11-13, 2018
Control Volume access frequency access duration Fig. 6. Control Volume displacement on the station rack.
Fig. 7. Control Volume Analysis for picking from rack bins.
5. RESULTS AND DISCUSSION
A further productive KPI offered by the MAS is the station area utilization by the operator. The operator spends most of the cycle time (59%) close to the workbench in a 0.8 m2 area to perform the assembly tasks or the picking activities from the rack. Concerning the storage locations used for medium and large-size components, 9.8% of the cycle time is required to handle components on the pallet or in the box, whereas just 4.9% is needed to access to the 3 levels of the shelf. The results provided by the analysis of the operator movements in the station area suggest to further investigate the different cycle time components.
The MAS is adopted to assess the manual assembly process performed by the operator in the monitored station. The MAS automatically and quantitatively evaluates a set of KPIs to measure the productive performance of the assembly system presented in the case study. Table 3 summarizes the movements of the operator and his hands. The operator walks for a total distance of 29.7 m during the cycle time at an average speed of 5.8 m/min. The alarming value of 5.6 m of vertical movements is experienced due to lifting and lowering activities for component picking from storage locations at ground level. Table 3. Assessment of the operator and upper limbs movements. Body part
Travelled distance [m]
Vertical drop [m]
Average speed [m/min]
Right hand
78.4
28.3
12.8
Left hand
73.5
26.9
12.4
Operator
29.7
5.6
5.8
The CV analysis focuses on the component picking from the rack bins (Fig. 7). The operator retrieves 11 times per cycle the components stored in the Peripheral bins of the rack, whereas the Central bins are underused (1 access only) considering their convenient location close to the workbench. Taking into account this outcome and the identical dimensions of the bins, a possible improvement could be the inversion of the Central and Peripheral bins to minimize the duration of the most performed activities. Furthermore, the CV analysis evaluates the average access duration in each monitored bin. The results proposed in Fig. 7 request a further investigation concerning the Top 1 bin, since the picking duration from this bin is more than double compared to the other bins (5.3 sec vs 2.1 sec on average). The sharp glass cover for the microwave display stored in this bin requires a particular attention by the operator during the picking which fully justifies the long duration of this activity.
Fig. 8. Utilization of the assembly station area by the operator. The MAS is able to automatically and quantitatively provide the cycle time partition between the added-value and nonadded-value activities. For the considered case study, the execution of assembly tasks on the station workbench is the only activity which adds value to the production process and it represents less than half of the cycle time (48%). Concerning the non-added-value activities, the operator walks for 22% of the cycle time, whereas he performs picking activities for the remaining 30%. Most of this time is required to retrieve components from storage locations far 415
IFAC INCOM 2018 416 Bergamo, Italy, June 11-13, 2018
Emilio Ferrari et al. / IFAC PapersOnLine 51-11 (2018) 411–416
from the workbench (e.g. box and shelf), whereas picking from the rack accounts for just 11% of the cycle time. Finally, the time required to access to the trolley to handle the tools used in the assembly process is negligible (4% of cycle time).
Walk Path Segmentation and Drift Analysis in Manual Assembly. Procedia CIRP, 52, 286-291. Agethen, P., Otto, M., Mengel, S., & Rukzio, E. (2016b). Using Marker-less Motion Capture Systems for Walk Path Analysis in Paced Assembly Flow Lines. Procedia CIRP, 54, 152-157. Bortolini, M., Faccio, M., Ferrari, E., Gamberi, M., & Pilati, F. (2017c). Time and energy optimal unit-load assignment for automatic S/R warehouses. International Journal of Production Economics, 190, 133-145. Bortolini, M., Faccio, M., Gamberi, M., & Pilati, F. (2017b). Multi-objective assembly line balancing considering component picking and ergonomic risk. Computers & Industrial Engineering, 112, 348-367. Bortolini, M., Ferrari, E., Gamberi, M., Pilati, F., & Faccio, M. (2017a) Assembly system design in the Industry 4.0 era: a general framework. IFAC-PapersOnLine, 50(1), 5700-5705. Bortolini, M., Faccio, M., Gamberi, M., & Pilati, F. (2016). Multi-objective design of multi-modal fresh food distribution networks. International Journal of Logistics Systems and Management, 24(2), 155-177. Ceseracciu, E., Sawacha, Z., & Cobelli, C. (2014). Comparison of Markerless and Marker-Based Motion Capture Technologies through Simultaneous Data Collection during Gait: Proof of Concept, PloS one, 9(3), 1–7. Du J. C. & Duffy, V. G. (2007). A methodology for assessing industrial workstations using optical motion capture integrated with digital human models. Occupational Ergonomics, 7(1), 11-25. Faccio, M., Gamberi, M., Pilati, F., & Bortolini, M. (2015). Packaging strategy definition for sales kits within an assembly system. International Journal of Production Research, 53(11), 3288-3305. Geiselhart, F., Otto, M., & Rukzio, E. (2016). On the use of Multi-Depth-Camera based Motion Tracking Systems in Production Planning Environments. Procedia CIRP, 41, 759764. Meredith, M., & Maddock, S. (2001). Motion capture file formats explained. Department of Computer Science, University of Sheffield, 211, 241-244. Nguyen, T. D., Kleinsorge, M., Postawa, A., Wolf, K., Scheumann, R., Krüger, J., & Seliger, G. (2013). Human centric automation: using marker-less motion capturing for ergonomics analysis and work assistance in manufacturing processes. In Proceedings of the 11th Global Conference on Sustainable Manufacturing (GCSM)—Innovative Solutions, Berlin, 586-592). Oyekan, J., Prabhu, V., Tiwari, A., Baskaran, V., Burgess, M., & Mcnally, R. (2017). Remote real-time collaboration through synchronous exchange of digitised human– workpiece interactions. Future Generation Computer Systems, 67, 83-93. Puthenveetil, S. C., Daphalapurkar, C. P., Zhu, W., Leu, M. C., Liu, X. F., Gilpin-Mcminn, J. K., & Snodgrass, S. D. (2015). Computer-automated ergonomic analysis based on motion capture and assembly simulation. Virtual Reality, 19(2), 119-128.
Fig. 9. Cycle time partition between the added-value and nonadded-value activities. 6. CONCLUSIONS This paper presents an original Motion Analysis System (MAS) developed for the monitoring and evaluation of manual manufacturing and assembly processes through motion capture (MOCAP) technologies. The MAS hardware is represented by a network of depth cameras, e.g. a markerless optical MOCAP technology, whose purpose is the digitalization of human movements and posture at 30 Hz with no interferences with the performed activity. A customized software architecture is developed to exploit these data in relation to the 3D layout of the workstation where the operator performs his activities. The original Control Volume (CV) Analysis proposed within this context provides an indepth assessment concerning the operator time and space interaction with the workstation elements, e.g. machines, workbench, tools, product components, racks, shelves, etc. This information is exploited by the MAS to automatically and quantitatively evaluate a set of productive KPIs to assess the monitored manual manufacturing or assembly process. A case study of a manual assembly process of a microwave oven is adopted to test and validate the MAS. The MAS performs a Control Volume Analysis for the component picking from the rack bins highlighting the picking frequency and duration from each location. Finally, the MAS provides an accurate cycle time partition between the added-value (e.g. assembly) and non-added-value activities (e.g. walking and picking). Further research should focus on the meaningful and powerful output data provided by the MAS to propose a relayout of the monitored workstation considering the weaknesses and opportunities highlighted by the MAS. REFERENCES Agethen, P., Otto, M., Gaisbauer, F., & Rukzio, E. (2016a). Presenting a Novel Motion Capture-based Approach for
416