- Email: [email protected]
The Employment of the “Sniff Test”
Perspectives Commentary on: Biomechanical Stability of a Posterior-Alone Fixation Technique After Craniovertebral Junction Realignment by Daniel et al. pp. 357-361.
Edward C. Benzel, M.D. Professor of Surgery, Cleveland Clinic Lerner College of Medicine of Case Western Reserve University Chairman, Department of Neurosurgery Co-Director, Spine Surgery Fellowship Program, Cleveland Clinic
The Employment of the “Sniff Test” Edward C. Benzel
“There are three kinds of lies—lies, damned lies, and statistics.” Variously attributed to Benjamin Disraeli, Mark Twain, and others
T
he testing of a spinal construct by mechanical means is fraught with difficulty. Oftentimes, in the case of published studies, the data acquired and presented are found to be statistically significant. Although, on the surface, the study may appear to be methodologically sound, the actual and literal conclusions may, in fact, be irrelevant to the clinical situation at hand. The study presented herein by Daniel et al. is no exception. I am not being critical of Daniel et al. in particular; nor am I singling them out. I am, conversely, casting aspersions on the majority of biomechanical studies that assess the integrity and efficacy of spinal constructs in cadavers via quasistatic flexibility testing strategies. Of the 3 C1-2 fixation techniques studied for the realignment of the craniovertebral junction by Daniel et al., the first technique involved distraction and placement of a spacer only. Each of the subsequent 2 involved the placement of additional stabilizing components. Each of these techniques was then quasistatically compared via flexibility testing using a 6-degree-of-freedom spine simulator and the application of pure moments. Standalone spacers were found to provide a significant improvement in stability. Dorsal lateral mass fixation further increased stability. Adding a third point of fixation further increased stability. The differences, however, were not significant. The authors hence concluded that distraction of the C1-C2 articular facets and direct articular joint atlantoaxial fixation is an ideal method of management of basilar invagination. I ask the question, “Would you rather sit on a 2-legged or 3-legged stool”? Let me help here. A 3-legged stool is much
Key words 䡲 Basilar invagination 䡲 Biomechanics 䡲 Craniovertebral junction realignment
more stable than a 2-legged stool. To not demonstrate a difference (2 lateral mass legs, with or without an interspinous leg) via a biomechanical study is illustrative of a fundamental methodological flaw or inadequate power of the study (“lies, damn lies, and statistics”). The results do not pass the sniff test. Again, I am not being critical of this particular study, but rather, I am being critical of the majority of such studies. An underpowered study that incorporates multiple assumptions that are each associated with errors of relevance and interpretation results in the acquisition of information that is portrayed as truth, but in reality is very flawed. Let us consider the assumptions made in such a study. A quasistatic biomechanical study, in this case, assesses range of motion and, indirectly, stiffness. On the surface, that sounds like a noble thing to study. In reality, the range of motion and stiffness of the construct at hand is assessed in a cadaver at room temperature during the first several cycles of motion after insertion of the device. Three assumptions have been identified so far. First, the study was done at room temperature. Viscoelasticity and stiffness properties change with temperature. Second, the specimen is a cadaver. The tissue is dead, without blood supply and without the ability to repair or adapt by remodeling. More than likely, the preservation process (e.g., cooling or freezing) has further altered the properties of the tissues. As an aside, the specimens most likely do not properly represent the patient population for whom such surgical techniques may be applicable (e.g., age, gender, and bone quality differences). Perhaps most importantly regarding the variables thus far mentioned, the specimens were studied within the first several cycles of motion after implant insertion. What happens after 100,000 cycles (approximately 1 month’s worth of activity)? Does the implant loosen? Would bone fusion have ensued? If it had, at
Department of Neurosurgery, Cleveland Clinic, Cleveland, Ohio, and Lerner College of Medicine, Case Western Reserve University, Cleveland, Ohio, USA To whom correspondence should be addressed: Edward C. Benzel, M.D. [E-mail: [email protected];[email protected]] Citation: World Neurosurg. (2012) 77, 2:277-278. DOI: 10.1016/j.wneu.2011.08.012
WORLD NEUROSURGERY 77 [2]: 277-278, FEBRUARY 2012
www.WORLDNEUROSURGERY.org
277
PERSPECTIVES
which location within the construct had fusion begun—at the dorsal interspinous graft site or at the spacer/facet joint complex site? These questions cannot be answered with such a study. Unfortunately, I have only begun listing the assumptions that can lead to significant errors of interpretation. Each assumption-related error exponentially affects the others, thus resulting in final results that are essentially uninterpretable regarding clinical utility. To me, what is perhaps most distressing about such studies is that the conclusions derived can be misleading and result in misapplication of techniques and technologies. Poor studies eliminate very few variables. Good studies eliminate many. Great studies eliminate most. No study can eliminate all variables. With regard to the study by Daniel et al., a low-cycle (100,000 cycle), body-temperature study performed in a humid environment may have partially addressed the aforementioned preliminary concerns. In addition, the utilization of more specimens may have powered the study sufficiently to show a difference between a 2and a 3-legged stool.
ances cannot help but be affected by authors with vested interests. Four of the 5 authors of this study are industry employees. Their biases and rational thought are most certainly affected by such. I (Ed Benzel) have been legitimately criticized for authoring papers with colleagues from industry. I am, thus, guilty, as I have charged. I accept such criticism when guilty, as I have delivered such here. With all similar studies, we should “sniff harder,” strive to eliminate variables and assumptions, scrutinize results and conclusions more carefully, and demand that methodological assumptions be aggressively kept to a minimum. We can all do better. We can conceive and carry out better, more appropriately designed studies. We can assess the literature more carefully by “sniffing” more often and harder. Finally, we can and should guide our clinical decision making processes by distilling from the literature what is meaningful and applicable to our patients. I apologize to Daniel et al. for singling them out when in fact I and many, many, many others are similarly guilty. Regardless, whether my criticisms are appropriate or irrational, they should provide food for thought.
Finally, Daniel et al. concluded in their study presented here that “distraction of the C1-C2 articular facets and direct articular joint atlantoaxial fixation would be an ideal method of management of basilar invagination.” I do not understand how such a conclusion could be derived with the data as presented and considering the assumptions as made. If one considers only the data presented, such a conclusion cannot be derived.
In the end, which of the 3 constructs do I think is mechanically most sound? After reviewing the data and analysis presented, I cannot say with certainty. My nose, however, tells me that the 3-legged stool is the most mechanically sound construct. It just “smells” better.
When studies presented in the literature do not pass the “sniff test” (e.g., something about the study just does not “smell” or seem right), it is likely that it is not. It is also likely that multiple methodological assumptions have perverted the results. Such may be reflective of naïveté on the part of the researchers, bias on the part of the researchers, inadequate available resources, etc. Such vari-
Citation: World Neurosurg. (2012) 77, 2:277-278. DOI: 10.1016/j.wneu.2011.08.012
278
www.SCIENCEDIRECT.com
Journal homepage: www.WORLDNEUROSURGERY.org Available online: www.sciencedirect.com 1878-8750/$ - see front matter © 2012 Elsevier Inc. All rights reserved.
WORLD NEUROSURGERY, DOI:10.1016/j.wneu.2011.08.012
Edward C. Benzel, M.D. Professor of Surgery, Cleveland Clinic Lerner College of Medicine of Case Western Reserve University Chairman, Department of Neurosurgery Co-Director, Spine Surgery Fellowship Program, Cleveland Clinic
The Employment of the “Sniff Test” Edward C. Benzel
“There are three kinds of lies—lies, damned lies, and statistics.” Variously attributed to Benjamin Disraeli, Mark Twain, and others
T
he testing of a spinal construct by mechanical means is fraught with difficulty. Oftentimes, in the case of published studies, the data acquired and presented are found to be statistically significant. Although, on the surface, the study may appear to be methodologically sound, the actual and literal conclusions may, in fact, be irrelevant to the clinical situation at hand. The study presented herein by Daniel et al. is no exception. I am not being critical of Daniel et al. in particular; nor am I singling them out. I am, conversely, casting aspersions on the majority of biomechanical studies that assess the integrity and efficacy of spinal constructs in cadavers via quasistatic flexibility testing strategies. Of the 3 C1-2 fixation techniques studied for the realignment of the craniovertebral junction by Daniel et al., the first technique involved distraction and placement of a spacer only. Each of the subsequent 2 involved the placement of additional stabilizing components. Each of these techniques was then quasistatically compared via flexibility testing using a 6-degree-of-freedom spine simulator and the application of pure moments. Standalone spacers were found to provide a significant improvement in stability. Dorsal lateral mass fixation further increased stability. Adding a third point of fixation further increased stability. The differences, however, were not significant. The authors hence concluded that distraction of the C1-C2 articular facets and direct articular joint atlantoaxial fixation is an ideal method of management of basilar invagination. I ask the question, “Would you rather sit on a 2-legged or 3-legged stool”? Let me help here. A 3-legged stool is much
Key words 䡲 Basilar invagination 䡲 Biomechanics 䡲 Craniovertebral junction realignment
more stable than a 2-legged stool. To not demonstrate a difference (2 lateral mass legs, with or without an interspinous leg) via a biomechanical study is illustrative of a fundamental methodological flaw or inadequate power of the study (“lies, damn lies, and statistics”). The results do not pass the sniff test. Again, I am not being critical of this particular study, but rather, I am being critical of the majority of such studies. An underpowered study that incorporates multiple assumptions that are each associated with errors of relevance and interpretation results in the acquisition of information that is portrayed as truth, but in reality is very flawed. Let us consider the assumptions made in such a study. A quasistatic biomechanical study, in this case, assesses range of motion and, indirectly, stiffness. On the surface, that sounds like a noble thing to study. In reality, the range of motion and stiffness of the construct at hand is assessed in a cadaver at room temperature during the first several cycles of motion after insertion of the device. Three assumptions have been identified so far. First, the study was done at room temperature. Viscoelasticity and stiffness properties change with temperature. Second, the specimen is a cadaver. The tissue is dead, without blood supply and without the ability to repair or adapt by remodeling. More than likely, the preservation process (e.g., cooling or freezing) has further altered the properties of the tissues. As an aside, the specimens most likely do not properly represent the patient population for whom such surgical techniques may be applicable (e.g., age, gender, and bone quality differences). Perhaps most importantly regarding the variables thus far mentioned, the specimens were studied within the first several cycles of motion after implant insertion. What happens after 100,000 cycles (approximately 1 month’s worth of activity)? Does the implant loosen? Would bone fusion have ensued? If it had, at
Department of Neurosurgery, Cleveland Clinic, Cleveland, Ohio, and Lerner College of Medicine, Case Western Reserve University, Cleveland, Ohio, USA To whom correspondence should be addressed: Edward C. Benzel, M.D. [E-mail: [email protected];[email protected]] Citation: World Neurosurg. (2012) 77, 2:277-278. DOI: 10.1016/j.wneu.2011.08.012
WORLD NEUROSURGERY 77 [2]: 277-278, FEBRUARY 2012
www.WORLDNEUROSURGERY.org
277
PERSPECTIVES
which location within the construct had fusion begun—at the dorsal interspinous graft site or at the spacer/facet joint complex site? These questions cannot be answered with such a study. Unfortunately, I have only begun listing the assumptions that can lead to significant errors of interpretation. Each assumption-related error exponentially affects the others, thus resulting in final results that are essentially uninterpretable regarding clinical utility. To me, what is perhaps most distressing about such studies is that the conclusions derived can be misleading and result in misapplication of techniques and technologies. Poor studies eliminate very few variables. Good studies eliminate many. Great studies eliminate most. No study can eliminate all variables. With regard to the study by Daniel et al., a low-cycle (100,000 cycle), body-temperature study performed in a humid environment may have partially addressed the aforementioned preliminary concerns. In addition, the utilization of more specimens may have powered the study sufficiently to show a difference between a 2and a 3-legged stool.
ances cannot help but be affected by authors with vested interests. Four of the 5 authors of this study are industry employees. Their biases and rational thought are most certainly affected by such. I (Ed Benzel) have been legitimately criticized for authoring papers with colleagues from industry. I am, thus, guilty, as I have charged. I accept such criticism when guilty, as I have delivered such here. With all similar studies, we should “sniff harder,” strive to eliminate variables and assumptions, scrutinize results and conclusions more carefully, and demand that methodological assumptions be aggressively kept to a minimum. We can all do better. We can conceive and carry out better, more appropriately designed studies. We can assess the literature more carefully by “sniffing” more often and harder. Finally, we can and should guide our clinical decision making processes by distilling from the literature what is meaningful and applicable to our patients. I apologize to Daniel et al. for singling them out when in fact I and many, many, many others are similarly guilty. Regardless, whether my criticisms are appropriate or irrational, they should provide food for thought.
Finally, Daniel et al. concluded in their study presented here that “distraction of the C1-C2 articular facets and direct articular joint atlantoaxial fixation would be an ideal method of management of basilar invagination.” I do not understand how such a conclusion could be derived with the data as presented and considering the assumptions as made. If one considers only the data presented, such a conclusion cannot be derived.
In the end, which of the 3 constructs do I think is mechanically most sound? After reviewing the data and analysis presented, I cannot say with certainty. My nose, however, tells me that the 3-legged stool is the most mechanically sound construct. It just “smells” better.
When studies presented in the literature do not pass the “sniff test” (e.g., something about the study just does not “smell” or seem right), it is likely that it is not. It is also likely that multiple methodological assumptions have perverted the results. Such may be reflective of naïveté on the part of the researchers, bias on the part of the researchers, inadequate available resources, etc. Such vari-
Citation: World Neurosurg. (2012) 77, 2:277-278. DOI: 10.1016/j.wneu.2011.08.012
278
www.SCIENCEDIRECT.com
Journal homepage: www.WORLDNEUROSURGERY.org Available online: www.sciencedirect.com 1878-8750/$ - see front matter © 2012 Elsevier Inc. All rights reserved.
WORLD NEUROSURGERY, DOI:10.1016/j.wneu.2011.08.012