- Email: [email protected]
The Kinect Recording System for objective three- and four-dimensional breast assessment with image overlays
Journal of Plastic, Reconstructive & Aesthetic Surgery (2016) 69, e27ee34
The Kinect Recording System for objective three- and four-dimensional breast assessment with image overlays Helga Henseler a,*, Sarah Kim Bonkat a, Peter Maria Vogt a, Bodo Rosenhahn b a Medical School of Hannover (Medizinische Hochschule Hannover), Department of Plastic-, Hand- and Reconstructive Surgery, Germany b Institute for Information Processing, Department of Computer Science and Electrical Engineering, Leibniz University of Hannover, Germany
Received 3 March 2015; accepted 15 October 2015
KEYWORDS Kinect; Three-dimensional imaging; Objective method; Breast assessment; Arm positioning; Flap planning
Summary Introduction: We investigated the application of the validated portable Kinect camera for three- and four-dimensional breast assessment in female life models. Method: Breast images from six life models were captured using the Kinect camera. Capture was conducted with taking three different arm positions while standing upright: with the arms straight down, straight up to the side at 90 and straight all the way up. Images of the volunteers were superimposed on each other. Digital linear distances between sternal notch and nipple-areola complexes were obtained and compared. The views of plastic and breast surgeons to arm positions were questioned. An example for clinical application was provided. Results: Successful capture of images of the female life breast models was achieved. Digital breast measurements at the three different arm positions revealed considerable variation in linear distances measured on the images obtained with the Kinect camera. The dynamic of breast movements could be demonstrated by image overlay and the first ever fourdimensional breast assessment was demonstrated. Fourteen plastic and breast surgeons were found to have nine different opinions regarding their favoured arm positions for breast capture. Even though precision of image sharpness still needs improvement, the images were satisfactory for clinical patient use. The Kinect data were shown to be applicable to surgery planning by designing a planar flap from the 3D mesh.
* Corresponding author. Medizinische Hochschule Hannover, Klinik fu ¨r Plastische, Hand- und Wiederherstellungschirurgie, Germany. Tel.: þ49 511 532 8864, þ49 176 1532 3753 (mobile); fax: þ49 511 532 8890. E-mail addresses: [email protected], [email protected] (H. Henseler), [email protected] (S.K. Bonkat), [email protected] (P.M. Vogt), [email protected] (B. Rosenhahn). http://dx.doi.org/10.1016/j.bjps.2015.10.021 1748-6815/ª 2015 British Association of Plastic, Reconstructive and Aesthetic Surgeons. Published by Elsevier Ltd. All rights reserved.
e28
H. Henseler et al. Conclusion: The portable and low-cost Kinect camera proved to be easy to use for the first application in life models for three- and four-dimensional breast assessment. ª 2015 British Association of Plastic, Reconstructive and Aesthetic Surgeons. Published by Elsevier Ltd. All rights reserved.
Introduction Various methods have emerged in recent years for threedimensional (3D) imaging of the breast. Among these are multiple stereo photogrammetry systems with varying numbers of cameras.1e3 Laser scanning tools have been used to obtain three-dimensional images in life models and compared with alternative methods.4 Early attempts to improve surgical planning in patients resulted in the calculation of a formula by which breast resection weights for reduction was supposed to be predicted.5 Some systems have been presented to the market in the field of aesthetic plastic surgery; however, various problems including high costs, limitations in the possible applications and lack of independent validation procedures have hindered widespread use.6 The set-up of 3D capture/scanning systems has not always been the focus of the researchers investigating various clinical aspects.7 Recently, even portable systems that use iPhone Apps with a necessary online storage of images have been examined8; however data protection issues have not been solved. In previous work the Microsoft Kinect camera as a portable low-cost tool for offline 3D image creation was examined in a mannequin model in a laboratory setting for validation purposes.9 In this study, Euclidian and surface distances between landmarks were compared with linear measurements and results were judged as promising. However, because the design of this system requires two Kinect cameras connected together, calibration procedures ahead of capture were required, hindering its ease of use. While the portable three-dimensional Kinect Recording System we present here that is based on one single camera was previously validated for the first time by comparing its images to implant volumes and measurements with the Arthur Morris device,10 the examination of the Kinect camera in life models has still to be conducted.
way up (Figure 2). Capture was conducted for various breast shapes, from very small to large and ptotic. New visualization software helped the examiner to judge correct capture distances to the portable system. Three-dimensional images were built with raw data sets of textured and depth images using Matlab Software. Images were visualized from all directions in a coordinate system. The anterioreposterior view was used on a personal computer screen, and landmarks were set following conjoint decision making of a computer scientist and consultant plastic surgeon experienced in breast surgery. Linear distances from jugulum to nipple-areola complexes were calculated by computer software after landmark placement. Variations in triangle formation between the jugulum and both nipple-areola complexes at varying arm positions between the three-dimensional images were noted and variation of movement of the breasts displayed. A questionnaire was given to 14 plastic and breast surgeons to assess their views on which position they would prefer the arms to be taken for breast image capture. Furthermore, an application useful to patients by generating a planar flap design from the curved 3D breast surface data was investigated. A mass-spring model allows model deformations to be made on a polygonal mesh, based on physical forces for stiffness, gravity, stretching and bending. This model is widely used for cloth or tissue simulation and is based on Hooke’s law. In this paper, the model is used to project a 3D polygonal mesh onto a 2D plane while maintaining relative distances, thus generating a planar flap from the 3D breast surface data.
Aim We investigated the first application of the validated portable Kinect Recording System in female life models for three- and four-dimensional breast assessment. Arm positioning and flap planning were examined.
Method Six life breast models were captured using the Kinect camera10 (Figure 1). Following a standardized protocol, capture was conducted using three different arm positions. The images were taken while standing upright with the arms down, straight up to the side at 90 and straight all the
Figure 1
Set-up of Kinect Recording System.
The Kinect Recording System
e29
Figure 2
Three different arm positions for capture.
Results Successful capture of the female life breast model was achieved. Capture time was short, and neither calibration procedures nor special background aspects were required. By application of specially programmed visualization aides, reproducible and clinically satisfactory captures were achieved. Software adjustment indicated correct capture distance. The changes in the three arm positions of the six volunteers revealed considerable variation in linear distances measured on the images between volunteers; see Table 1 (Figure 3). The dynamics of breast movements with varying arm positions were demonstrated in two examples of volunteers with large and small breasts through image overlays (Figure 4). Three-dimensional images were displayed in a coordinate system. Changes in breast positions were provoked by changes in arm positions demonstrating a curve like movement of the nipple-areola complexes from a lower medial position with the arms down, to lateral with the arms to the side, to upper medial with the arms up. The Kinect Recording System served to objectively demonstrate these changes through the movements, providing the first ever four-dimensional breast assessment. The answers given by the plastic and breast surgeons revealed a variation in opinion as to the best possible arm or body positions during capture. Among the 14 doctors, nine different answers were provided in the questionnaire. The clinical example of a patient after mastectomy, after breast reconstruction with a free deep inferior epigastric artery perforator flap (DIEP) and after nipple reconstruction
Table 1 Volunteer
Recordings with Kinect Camera. arms down
arms side
Jugulum/right Mamilla distance in cm 1 18 17.4 2 16.7 18.3 3 19.5 16.6 4 20.1 19.6 5 22.6 22.1 6 25.9 25.4 Jugulum/left Mamilla distance in cm 1 20.5 18.7 2 18 17.8 3 20.5 18.9 4 20.3 18.7 5 24 22.7 6 25.4 25.2 Inter Mamilla distance in cm 1 22.4 22.7 2 17.9 19 3 20.2 20.2 4 21 21.4 5 25.2 25.7 6 25.2 28.5
arms up 15 15.8 16.3 16.8 19.1 22.3 16.8 14.6 16.7 15.6 18.2 22.3 21.3 18.2 19.8 20 24.8 28
e30
Figure 3 Linear measurements in six volunteers with varying arms positions recorded with Kinect Camera.
(Figure 5) revealed that the Kinect Recording System served well in a clinical/office type setting for standardized threedimensional patient photography. The mass-spring model was optimized to map the curved 3D surface mesh onto a plane while maintaining the relative distances (Figure 6).14 This allows a flap design to be generated for breast surgery, and produces an estimate of the optimal amount and shape of the required skin/tissue patch. We found, that generated surfaces changed dependent on the arm positions; however, the flap planning templates turned out to be not fundamentally different (Figure 7).
Discussion With the presented low-cost portable system, we have been able to successfully capture the female breast. By conducting an image overlay and displaying the variance of linear distances measurements at three different arm positions between the three-dimensional images, we conducted the first application of the Kinect Recording system in female life models. The considerable variation in linear distances of the aesthetic triangle at different arm
H. Henseler et al. positions is a finding that has not been quantified before. While clinically plastic surgeons are aware that the breast is a pendulous body part with varying degrees of the amount of skin excess and soft tissue movement this is the first study to objectively display this finding using a portable three-dimensional imaging method. Using image overlay, we present the first ever four-dimensional display of breast changes. The Kinect Recording system described here is one of the few that has undergone an independent previous validation.9,10 While the primary goal in three-dimensional imaging is an objective analysis and outcome assessment,11 some early studies of prospective simulation programs12,13 for surgical results have been presented. Nevertheless, independent assessments are lacking. The investigation presented her also found that different plastic and breast surgeons think differently about the required arm positions while conducting clinical photography. For purposes of three-dimensional breast assessments, any arm position to the side seems to be preferable because of the lack of obscurance of the sides of the breast. While overall there is a multitude of possible applications for the human body, the here presented system was built to focus on breast capture. As an example application we implemented a mass-spring model to project the reconstructed curved 3D mesh onto a plane while maintaining the relative distances. Thus, the needed skin sample can be generated when planning a flap. As a computer graphics method, the mass-spring model determines the deformation behaviour of objects based on polygonal meshes and demonstrates the changes in the shapes of objects, similar to classical multidimensional scaling.14 The application lies in the simulation of surgical procedures as presented and explained. At different arm positions, we found that the multidimensional model provided a template of roughly the same extent independent from the arm positions. It seems, therefore, that the breast tends to shift on the chest wall; however, the vertical and horizontal measurements of the breast images changed little. The reason for this might be that independent from the motion, the breast volume stays nearly constant. Therefore, we postulate that no detectable errors seem to be made when conducting 3D capture at various arm positions, which matches the results from our clinical questionnaire that the arm positions are judged quite flexibly regarding image capture. While this is the first investigation regarding arm positioning in 3D capture, further research is needed to finalize these results. Alternative methods include those that the project an optical grid on the breast15 and obtain a three-dimensional image by creating a model from two images and calculate the surface area. Laser scanning has also been extensively investigated before.4 Comparisons can also be made with other methods such as plaster cast applications,4 weighing of mastectomy specimens1 or application of water displacement and breast implants.3,10 A comparison of the three-dimensional imaging method with subjective breast assessment was previously conducted rating symmetry16 and finding the relationship highly significant. Similarly, two objective methods for the aesthetic evaluation of breast cancer conservative
The Kinect Recording System
Figure 4
Figure 5
Examples of breast image overlays, large and small breast and with 4D coordinates.
Patient after mastectomy, after breast reconstruction (DIEP flap) and after nipple reconstruction.
e31
e32
H. Henseler et al.
Figure 6
Breast mesh mapping, 3D polygonal mesh (in red) and 2D flap planning template (in blue).
treatment were compared,17 the software programs “Breast Cancer Conservative Treatment cosmetic results” (BCCT. core) and “Breast Analyzing Tool” (BAT). The inclusion of multiple parameters in image analyses such as
Figure 7
with BCCT core was found to have the potential to improve results and picture clarity was judged as important. Some critics have mentioned that clarity of the images obtained with the portable three-dimensional imaging
Breast surface changes at different arm positions and 2D flap planning templates (in blue).
The Kinect Recording System system of the Kinect camera is lacking. Naturally, a lowcost portable system will have some limitations. Nevertheless, computational developments are rapid and soon image quality should improve considerably. For our needs, we feel that the image quality is entirely sufficient. For clinical purposes, images serve as a documentation tool to display the true three-dimensional anatomy of the breast. As a no touch measurement tool for breast volume calculations, the eligibility of patients for breast surgery can be discussed with health insurance companies or hospital trusts. As shown with the multidimensional model, breast shapes can be planned in breast reconstruction by creating a curved three-dimensional model of the breast and changing it into a flattened two-dimensional template that can be printed out using an office printer and used for marking the outlines of flap extensions to be raised at the donor side. With the required software, the method serves as a visualization aid allowing patients to imagine possible postsurgery results. An increase in patient interest, appreciation and even medical referrals in our clinic was noted while conducting the studies, opening the door to the application of the method as a tool for patient information and clinic advertising. Future research however is needed to focus on the agreements of preoperative visualizations and postoperative results. Some critics have mentioned issues with ethical approval and patients’ rights to request their three-dimensional images. We did not observe these problems. First, special software is required to visualize the three-dimensional data files. All participants were informed about the research aspects of the study, received verbal and written information and provided their informed consent. In case of any future requests we are able to release raw data files, which could be viewed in two dimensions on common personal computers. Threedimensional software licenses are currently not freely available, but are commonly held by the institutions that developed or purchased or obtained them in the frame of a research project. An important benefit of the Kinect Recording System is the offline storage of images so that data protection standards can be followed. This is in contrast to smartphone applications, by which image storage is conducted online8 breaching patient confidentiality needs. The solution to this problem is outstanding. Three-dimensional imaging technology will replace the current two-dimensional practice in the future, and continuous further development of the system is expected.
Conclusion The portable, validated and low-cost Kinect camera for three-dimensional capture proved to be easy to use and reliable for the first application in life breast models. Variation in digital measurements with varying arm positions could be shown. The dynamics of breast movements could be demonstrated using image overlays in the first ever four-dimensional breast assessment, and digitized linear differences were provided. While there was considerable movement of the breast at varying arm positions as demonstrated in the 4D image overlays, there was not much
e33 variation of the 2D flap planning templates derived from the 3D polygonal mesh at different arm positions. Plastic and breast surgeons had various opinions of their favoured arm positions for breast capture. Images and depth data were satisfactory for use in flap planning. Threedimensional imaging technology will replace the current two-dimensional practice in the future, and a continuous further development of the system is expected.
Sources of funding The research study was self-funded.
Ethical approval Obtained from the local ethics committee, volunteer and patient confidentiality needs were observed, written consent in the use of the photographs was obtained.
List of products and devices Hardware: Kinect Recorder by Microsoft. Software: Self-developed by the Department of Computer Science, Leibniz University, Hannover, Germany.
Disclosures None of the authors has a financial interest in any of the products or devices mentioned in this article.
References 1. Losken A, Seify H, Denson DD, Paredes AA, Carlson GW. Validating three dimensional imaging of the breast. Ann Plast Surg 2005;54:471e6. 2. Tepper OM, Small KH, Unger JG, et al. 3D analysis of breast augmentation defines operative changes and their relationship to implant dimensions. Ann Plast Surg 2009;62:570e5. 3. Henseler H, Smith J, Bowman A, Khambay BS, Ju X, Ayoub A, Ray AK. Objective evaluation of the latissimus dorsi flap for breast reconstruction using three-dimensional Imaging. J Plast Reconstr Aesthet Surg 2012;65:1209e15. 4. Kovacs L, Eder M, Hollweck R, et al. Comparison between breast volume measurement using 3D surface imaging and classical techniques. Breast 2007;16:137e45. 5. Eder M, Grabhorn A, Waldenfels F v, et al. Prediction of resection weight in reduction mammaplasty based on 3dimensional surface imaging. Surg Innov 2013;20:356e64. 6. Weinberg S, Kolar J. Three-dimensional surface imaging: limitations and considerations from the anthropometric perspective. J Craniofac Surg 2005;16:847e51. 7. Ji K, Luan J, Liu C, et al. A prospective study of breast dynamic morphological changes after dual-plane augmentation mammoplasty with 3D scanning technique. PloS One 2014;9:e93010. 8. Koban KC, Leitsch S, Holzbach T, et al. 3D Bilderfassung und Analyse in der plastischen Chirurgie mit Smartphone und Tablet: eine Alternative zu professionellen Systemen? Handchir Mikrochir Plast Chir 2014;46:97e104. 9. Wheat JS, Choppin S, Goyal A. Development and assessment of a microsoft Kinect based system for imaging the breast in three dimensions. Med Eng Phys 2014;36:732e8.
e34 10. Henseler H, Kuznetsova A, Vogt PM, Rosenhahn B. Validation of the Kinect device as a new portable imaging system for threedimensional breast assessment. J Plast Reconstr Aesthetic Surg 2014;67:483e8. 11. Catanuto G, Spano A, Pennati A, et al. Experimental methodology for digital breast shape analysis and objective surgical outcome evaluation. J Plast Reconstr Aesth Surg 2008;61: 314e8. 12. Georgii J, Eder M, Bu ¨rger K, et al. A computational tool for preoperative breast augmentation planning in aesthetic plastic surgery. IEEE J Biomed Health Inf 2014;18:907e19. 13. Eder M, Raith S. Comparison of different material models to simulate breast deformations using finite element analysis. Ann Biomed Eng 2014;42:843e57.
H. Henseler et al. 14. Cox TF, Cox MAA. Multidimensional scaling. Chapman and Hall; 2001. 15. Thomson JG, Liu YJ, Restifo RJ, Rinker BD, Reis A. Surface area measurement of the female breast: phase I. Validation of a novel optical technique. Plast Reconstr Surg 2009;123: 1588e96. 16. Henseler H, Smith J, Bowman A, et al. Subjective versus objective assessment of breast reconstruction. J Plast Reconstr Aesthet Surg 2013;66:634e9. 17. Cardoso MJ, Cardoso JS, Wild T, et al. Comparing two objective methods for aesthetic evaluation of breast cancer conservative treatment. Breast Cancer Res Treat 2009;116:149e52.
The Kinect Recording System for objective three- and four-dimensional breast assessment with image overlays Helga Henseler a,*, Sarah Kim Bonkat a, Peter Maria Vogt a, Bodo Rosenhahn b a Medical School of Hannover (Medizinische Hochschule Hannover), Department of Plastic-, Hand- and Reconstructive Surgery, Germany b Institute for Information Processing, Department of Computer Science and Electrical Engineering, Leibniz University of Hannover, Germany
Received 3 March 2015; accepted 15 October 2015
KEYWORDS Kinect; Three-dimensional imaging; Objective method; Breast assessment; Arm positioning; Flap planning
Summary Introduction: We investigated the application of the validated portable Kinect camera for three- and four-dimensional breast assessment in female life models. Method: Breast images from six life models were captured using the Kinect camera. Capture was conducted with taking three different arm positions while standing upright: with the arms straight down, straight up to the side at 90 and straight all the way up. Images of the volunteers were superimposed on each other. Digital linear distances between sternal notch and nipple-areola complexes were obtained and compared. The views of plastic and breast surgeons to arm positions were questioned. An example for clinical application was provided. Results: Successful capture of images of the female life breast models was achieved. Digital breast measurements at the three different arm positions revealed considerable variation in linear distances measured on the images obtained with the Kinect camera. The dynamic of breast movements could be demonstrated by image overlay and the first ever fourdimensional breast assessment was demonstrated. Fourteen plastic and breast surgeons were found to have nine different opinions regarding their favoured arm positions for breast capture. Even though precision of image sharpness still needs improvement, the images were satisfactory for clinical patient use. The Kinect data were shown to be applicable to surgery planning by designing a planar flap from the 3D mesh.
* Corresponding author. Medizinische Hochschule Hannover, Klinik fu ¨r Plastische, Hand- und Wiederherstellungschirurgie, Germany. Tel.: þ49 511 532 8864, þ49 176 1532 3753 (mobile); fax: þ49 511 532 8890. E-mail addresses: [email protected], [email protected] (H. Henseler), [email protected] (S.K. Bonkat), [email protected] (P.M. Vogt), [email protected] (B. Rosenhahn). http://dx.doi.org/10.1016/j.bjps.2015.10.021 1748-6815/ª 2015 British Association of Plastic, Reconstructive and Aesthetic Surgeons. Published by Elsevier Ltd. All rights reserved.
e28
H. Henseler et al. Conclusion: The portable and low-cost Kinect camera proved to be easy to use for the first application in life models for three- and four-dimensional breast assessment. ª 2015 British Association of Plastic, Reconstructive and Aesthetic Surgeons. Published by Elsevier Ltd. All rights reserved.
Introduction Various methods have emerged in recent years for threedimensional (3D) imaging of the breast. Among these are multiple stereo photogrammetry systems with varying numbers of cameras.1e3 Laser scanning tools have been used to obtain three-dimensional images in life models and compared with alternative methods.4 Early attempts to improve surgical planning in patients resulted in the calculation of a formula by which breast resection weights for reduction was supposed to be predicted.5 Some systems have been presented to the market in the field of aesthetic plastic surgery; however, various problems including high costs, limitations in the possible applications and lack of independent validation procedures have hindered widespread use.6 The set-up of 3D capture/scanning systems has not always been the focus of the researchers investigating various clinical aspects.7 Recently, even portable systems that use iPhone Apps with a necessary online storage of images have been examined8; however data protection issues have not been solved. In previous work the Microsoft Kinect camera as a portable low-cost tool for offline 3D image creation was examined in a mannequin model in a laboratory setting for validation purposes.9 In this study, Euclidian and surface distances between landmarks were compared with linear measurements and results were judged as promising. However, because the design of this system requires two Kinect cameras connected together, calibration procedures ahead of capture were required, hindering its ease of use. While the portable three-dimensional Kinect Recording System we present here that is based on one single camera was previously validated for the first time by comparing its images to implant volumes and measurements with the Arthur Morris device,10 the examination of the Kinect camera in life models has still to be conducted.
way up (Figure 2). Capture was conducted for various breast shapes, from very small to large and ptotic. New visualization software helped the examiner to judge correct capture distances to the portable system. Three-dimensional images were built with raw data sets of textured and depth images using Matlab Software. Images were visualized from all directions in a coordinate system. The anterioreposterior view was used on a personal computer screen, and landmarks were set following conjoint decision making of a computer scientist and consultant plastic surgeon experienced in breast surgery. Linear distances from jugulum to nipple-areola complexes were calculated by computer software after landmark placement. Variations in triangle formation between the jugulum and both nipple-areola complexes at varying arm positions between the three-dimensional images were noted and variation of movement of the breasts displayed. A questionnaire was given to 14 plastic and breast surgeons to assess their views on which position they would prefer the arms to be taken for breast image capture. Furthermore, an application useful to patients by generating a planar flap design from the curved 3D breast surface data was investigated. A mass-spring model allows model deformations to be made on a polygonal mesh, based on physical forces for stiffness, gravity, stretching and bending. This model is widely used for cloth or tissue simulation and is based on Hooke’s law. In this paper, the model is used to project a 3D polygonal mesh onto a 2D plane while maintaining relative distances, thus generating a planar flap from the 3D breast surface data.
Aim We investigated the first application of the validated portable Kinect Recording System in female life models for three- and four-dimensional breast assessment. Arm positioning and flap planning were examined.
Method Six life breast models were captured using the Kinect camera10 (Figure 1). Following a standardized protocol, capture was conducted using three different arm positions. The images were taken while standing upright with the arms down, straight up to the side at 90 and straight all the
Figure 1
Set-up of Kinect Recording System.
The Kinect Recording System
e29
Figure 2
Three different arm positions for capture.
Results Successful capture of the female life breast model was achieved. Capture time was short, and neither calibration procedures nor special background aspects were required. By application of specially programmed visualization aides, reproducible and clinically satisfactory captures were achieved. Software adjustment indicated correct capture distance. The changes in the three arm positions of the six volunteers revealed considerable variation in linear distances measured on the images between volunteers; see Table 1 (Figure 3). The dynamics of breast movements with varying arm positions were demonstrated in two examples of volunteers with large and small breasts through image overlays (Figure 4). Three-dimensional images were displayed in a coordinate system. Changes in breast positions were provoked by changes in arm positions demonstrating a curve like movement of the nipple-areola complexes from a lower medial position with the arms down, to lateral with the arms to the side, to upper medial with the arms up. The Kinect Recording System served to objectively demonstrate these changes through the movements, providing the first ever four-dimensional breast assessment. The answers given by the plastic and breast surgeons revealed a variation in opinion as to the best possible arm or body positions during capture. Among the 14 doctors, nine different answers were provided in the questionnaire. The clinical example of a patient after mastectomy, after breast reconstruction with a free deep inferior epigastric artery perforator flap (DIEP) and after nipple reconstruction
Table 1 Volunteer
Recordings with Kinect Camera. arms down
arms side
Jugulum/right Mamilla distance in cm 1 18 17.4 2 16.7 18.3 3 19.5 16.6 4 20.1 19.6 5 22.6 22.1 6 25.9 25.4 Jugulum/left Mamilla distance in cm 1 20.5 18.7 2 18 17.8 3 20.5 18.9 4 20.3 18.7 5 24 22.7 6 25.4 25.2 Inter Mamilla distance in cm 1 22.4 22.7 2 17.9 19 3 20.2 20.2 4 21 21.4 5 25.2 25.7 6 25.2 28.5
arms up 15 15.8 16.3 16.8 19.1 22.3 16.8 14.6 16.7 15.6 18.2 22.3 21.3 18.2 19.8 20 24.8 28
e30
Figure 3 Linear measurements in six volunteers with varying arms positions recorded with Kinect Camera.
(Figure 5) revealed that the Kinect Recording System served well in a clinical/office type setting for standardized threedimensional patient photography. The mass-spring model was optimized to map the curved 3D surface mesh onto a plane while maintaining the relative distances (Figure 6).14 This allows a flap design to be generated for breast surgery, and produces an estimate of the optimal amount and shape of the required skin/tissue patch. We found, that generated surfaces changed dependent on the arm positions; however, the flap planning templates turned out to be not fundamentally different (Figure 7).
Discussion With the presented low-cost portable system, we have been able to successfully capture the female breast. By conducting an image overlay and displaying the variance of linear distances measurements at three different arm positions between the three-dimensional images, we conducted the first application of the Kinect Recording system in female life models. The considerable variation in linear distances of the aesthetic triangle at different arm
H. Henseler et al. positions is a finding that has not been quantified before. While clinically plastic surgeons are aware that the breast is a pendulous body part with varying degrees of the amount of skin excess and soft tissue movement this is the first study to objectively display this finding using a portable three-dimensional imaging method. Using image overlay, we present the first ever four-dimensional display of breast changes. The Kinect Recording system described here is one of the few that has undergone an independent previous validation.9,10 While the primary goal in three-dimensional imaging is an objective analysis and outcome assessment,11 some early studies of prospective simulation programs12,13 for surgical results have been presented. Nevertheless, independent assessments are lacking. The investigation presented her also found that different plastic and breast surgeons think differently about the required arm positions while conducting clinical photography. For purposes of three-dimensional breast assessments, any arm position to the side seems to be preferable because of the lack of obscurance of the sides of the breast. While overall there is a multitude of possible applications for the human body, the here presented system was built to focus on breast capture. As an example application we implemented a mass-spring model to project the reconstructed curved 3D mesh onto a plane while maintaining the relative distances. Thus, the needed skin sample can be generated when planning a flap. As a computer graphics method, the mass-spring model determines the deformation behaviour of objects based on polygonal meshes and demonstrates the changes in the shapes of objects, similar to classical multidimensional scaling.14 The application lies in the simulation of surgical procedures as presented and explained. At different arm positions, we found that the multidimensional model provided a template of roughly the same extent independent from the arm positions. It seems, therefore, that the breast tends to shift on the chest wall; however, the vertical and horizontal measurements of the breast images changed little. The reason for this might be that independent from the motion, the breast volume stays nearly constant. Therefore, we postulate that no detectable errors seem to be made when conducting 3D capture at various arm positions, which matches the results from our clinical questionnaire that the arm positions are judged quite flexibly regarding image capture. While this is the first investigation regarding arm positioning in 3D capture, further research is needed to finalize these results. Alternative methods include those that the project an optical grid on the breast15 and obtain a three-dimensional image by creating a model from two images and calculate the surface area. Laser scanning has also been extensively investigated before.4 Comparisons can also be made with other methods such as plaster cast applications,4 weighing of mastectomy specimens1 or application of water displacement and breast implants.3,10 A comparison of the three-dimensional imaging method with subjective breast assessment was previously conducted rating symmetry16 and finding the relationship highly significant. Similarly, two objective methods for the aesthetic evaluation of breast cancer conservative
The Kinect Recording System
Figure 4
Figure 5
Examples of breast image overlays, large and small breast and with 4D coordinates.
Patient after mastectomy, after breast reconstruction (DIEP flap) and after nipple reconstruction.
e31
e32
H. Henseler et al.
Figure 6
Breast mesh mapping, 3D polygonal mesh (in red) and 2D flap planning template (in blue).
treatment were compared,17 the software programs “Breast Cancer Conservative Treatment cosmetic results” (BCCT. core) and “Breast Analyzing Tool” (BAT). The inclusion of multiple parameters in image analyses such as
Figure 7
with BCCT core was found to have the potential to improve results and picture clarity was judged as important. Some critics have mentioned that clarity of the images obtained with the portable three-dimensional imaging
Breast surface changes at different arm positions and 2D flap planning templates (in blue).
The Kinect Recording System system of the Kinect camera is lacking. Naturally, a lowcost portable system will have some limitations. Nevertheless, computational developments are rapid and soon image quality should improve considerably. For our needs, we feel that the image quality is entirely sufficient. For clinical purposes, images serve as a documentation tool to display the true three-dimensional anatomy of the breast. As a no touch measurement tool for breast volume calculations, the eligibility of patients for breast surgery can be discussed with health insurance companies or hospital trusts. As shown with the multidimensional model, breast shapes can be planned in breast reconstruction by creating a curved three-dimensional model of the breast and changing it into a flattened two-dimensional template that can be printed out using an office printer and used for marking the outlines of flap extensions to be raised at the donor side. With the required software, the method serves as a visualization aid allowing patients to imagine possible postsurgery results. An increase in patient interest, appreciation and even medical referrals in our clinic was noted while conducting the studies, opening the door to the application of the method as a tool for patient information and clinic advertising. Future research however is needed to focus on the agreements of preoperative visualizations and postoperative results. Some critics have mentioned issues with ethical approval and patients’ rights to request their three-dimensional images. We did not observe these problems. First, special software is required to visualize the three-dimensional data files. All participants were informed about the research aspects of the study, received verbal and written information and provided their informed consent. In case of any future requests we are able to release raw data files, which could be viewed in two dimensions on common personal computers. Threedimensional software licenses are currently not freely available, but are commonly held by the institutions that developed or purchased or obtained them in the frame of a research project. An important benefit of the Kinect Recording System is the offline storage of images so that data protection standards can be followed. This is in contrast to smartphone applications, by which image storage is conducted online8 breaching patient confidentiality needs. The solution to this problem is outstanding. Three-dimensional imaging technology will replace the current two-dimensional practice in the future, and continuous further development of the system is expected.
Conclusion The portable, validated and low-cost Kinect camera for three-dimensional capture proved to be easy to use and reliable for the first application in life breast models. Variation in digital measurements with varying arm positions could be shown. The dynamics of breast movements could be demonstrated using image overlays in the first ever four-dimensional breast assessment, and digitized linear differences were provided. While there was considerable movement of the breast at varying arm positions as demonstrated in the 4D image overlays, there was not much
e33 variation of the 2D flap planning templates derived from the 3D polygonal mesh at different arm positions. Plastic and breast surgeons had various opinions of their favoured arm positions for breast capture. Images and depth data were satisfactory for use in flap planning. Threedimensional imaging technology will replace the current two-dimensional practice in the future, and a continuous further development of the system is expected.
Sources of funding The research study was self-funded.
Ethical approval Obtained from the local ethics committee, volunteer and patient confidentiality needs were observed, written consent in the use of the photographs was obtained.
List of products and devices Hardware: Kinect Recorder by Microsoft. Software: Self-developed by the Department of Computer Science, Leibniz University, Hannover, Germany.
Disclosures None of the authors has a financial interest in any of the products or devices mentioned in this article.
References 1. Losken A, Seify H, Denson DD, Paredes AA, Carlson GW. Validating three dimensional imaging of the breast. Ann Plast Surg 2005;54:471e6. 2. Tepper OM, Small KH, Unger JG, et al. 3D analysis of breast augmentation defines operative changes and their relationship to implant dimensions. Ann Plast Surg 2009;62:570e5. 3. Henseler H, Smith J, Bowman A, Khambay BS, Ju X, Ayoub A, Ray AK. Objective evaluation of the latissimus dorsi flap for breast reconstruction using three-dimensional Imaging. J Plast Reconstr Aesthet Surg 2012;65:1209e15. 4. Kovacs L, Eder M, Hollweck R, et al. Comparison between breast volume measurement using 3D surface imaging and classical techniques. Breast 2007;16:137e45. 5. Eder M, Grabhorn A, Waldenfels F v, et al. Prediction of resection weight in reduction mammaplasty based on 3dimensional surface imaging. Surg Innov 2013;20:356e64. 6. Weinberg S, Kolar J. Three-dimensional surface imaging: limitations and considerations from the anthropometric perspective. J Craniofac Surg 2005;16:847e51. 7. Ji K, Luan J, Liu C, et al. A prospective study of breast dynamic morphological changes after dual-plane augmentation mammoplasty with 3D scanning technique. PloS One 2014;9:e93010. 8. Koban KC, Leitsch S, Holzbach T, et al. 3D Bilderfassung und Analyse in der plastischen Chirurgie mit Smartphone und Tablet: eine Alternative zu professionellen Systemen? Handchir Mikrochir Plast Chir 2014;46:97e104. 9. Wheat JS, Choppin S, Goyal A. Development and assessment of a microsoft Kinect based system for imaging the breast in three dimensions. Med Eng Phys 2014;36:732e8.
e34 10. Henseler H, Kuznetsova A, Vogt PM, Rosenhahn B. Validation of the Kinect device as a new portable imaging system for threedimensional breast assessment. J Plast Reconstr Aesthetic Surg 2014;67:483e8. 11. Catanuto G, Spano A, Pennati A, et al. Experimental methodology for digital breast shape analysis and objective surgical outcome evaluation. J Plast Reconstr Aesth Surg 2008;61: 314e8. 12. Georgii J, Eder M, Bu ¨rger K, et al. A computational tool for preoperative breast augmentation planning in aesthetic plastic surgery. IEEE J Biomed Health Inf 2014;18:907e19. 13. Eder M, Raith S. Comparison of different material models to simulate breast deformations using finite element analysis. Ann Biomed Eng 2014;42:843e57.
H. Henseler et al. 14. Cox TF, Cox MAA. Multidimensional scaling. Chapman and Hall; 2001. 15. Thomson JG, Liu YJ, Restifo RJ, Rinker BD, Reis A. Surface area measurement of the female breast: phase I. Validation of a novel optical technique. Plast Reconstr Surg 2009;123: 1588e96. 16. Henseler H, Smith J, Bowman A, et al. Subjective versus objective assessment of breast reconstruction. J Plast Reconstr Aesthet Surg 2013;66:634e9. 17. Cardoso MJ, Cardoso JS, Wild T, et al. Comparing two objective methods for aesthetic evaluation of breast cancer conservative treatment. Breast Cancer Res Treat 2009;116:149e52.