- Email: [email protected]
STATISTICAL EXPRESSION OF RESULTS IN MEDICAL RESEARCH : TWO COMMON PITFALLS.
207
CORRESPONDENCE would mean that in a series of 1000 cases, the errors with Test A would be 7 per cent. of 100, plus 1 per
STATISTICAL EXPRESSION OF RESULTS IN MEDICAL RESEARCH : TWO COMMON PITFALLS.
of THE LANCET. SIR,-The statistics required for the convenient recording of results in medical research work are not of a complicated kind, but it not infrequently happens that the interpretation of these statistics is rendered none the easier by an inappropriate method of calculating percentages. Let us take, as an example from laboratory work, the investigation of the accuracy of a particular To the Editor
B I
cent. of 900, that is, 16 cases, or 1-6 per cent. Those with Test B would work out at 2 per cent. of 100, plus 2 per cent. of 900, that is, 20 cases, or 2 per cent. This shows that in actual practice Test A would yield, not more, but 7ess errors than Test B. All this is elementary enough, and one would hesitate to mention these pitfalls were it not for the hope that such mention might assist towards a greater uniformity in recording results. It may be argued that where detailed figures are given it is an easy matter to work out the required facts for oneself ; but in these days of increased and rapid output of research, anyone who has to work through many references in a short while would be considerably assisted were the statistical results invariably given in a form that would express the true state of affairs, instead of one that is not only of little value but which may, to the unwary, prove misleading.
test. How often are the results couched in the form, " In so many cases, so many positives were obtained of which so many were incorrect." Or this may be abbreviated into, So many per cent. of the positives were false positives." If left in that form, what do these results convey1 Very little. For if, for we are told that " 5 per cent. of the posiexample, I am, Sir, yours faithfully, tives were incorrect," it subtly suggests a necessary J. ERNEST NICOLE. relation between the number of false positives and Winwick Mental Hospital, Jan. 17th, 1932. the total number of all positives. But any relation between them is really no more than an accidental one. The true relation to be expressed is the one THE CEREBRO-SPINAL FLUID PRESSURE IN between the number of cases giving false positivesCASES OF HEAD INJURY. i.e., cases failing to give correct negatives-and the To the Editor of THE LANCET. total number of cases that should have given negative results. A reductio ad absurdum will make the Sm,-At a discussion at the Royal Society matter clear. Suppose in the investigation of a test of Medicine on the Diagnosis and Treatment of Head for syphilis that 100 non-syphilitic cases-and only in Injuries reported THE LANCET of Dec. 12th, 1931, these cases-were tested, and suppose further that it seemed to me that quite an unwarranted degree of five positive results (therefore false positives) were was attached by most of the speakers to importance obtained, would we return our results as " 100 per the pressure of the cerebro-spinal fluid. The matter is cent. of the positives were incorrect " ?‘? The absurdity raised again by Dr. C. M. Hinds-Howell in the paper is obvious. What we really should say is 5 per cent. on Trauma and the Nervous System which appeared of the non-syphilitic cases were incorrectly returned in THE LANCET last week. as positive." At the discussion one of the speakers said The second source of misunderstanding lies in the that he decided not to operateopening on a case in which difference between statistical material obtained from he suspected intracranial haemorrhage, because the experimental series and the material likely to occur cerebro-spinal fluid pressure was not markedly raised. in actual practice. Again let a hypothetical example As long, however, as there is no interference with the illustrate the difficulty. circulation of the cerebro-spinal fluid, mere increase Two tests for syphilis, Test A and Test B, are in the volume of the other intracranial contents does under investigation, and the number of correct and not produce any significant rise in the pressure. Thus, incorrect results are expressed as percentages in the a large haematoma may be present the though manner suggested above. The actual criterion of fluid pressure may remain within cerebro-spinal accuracy, and the means of deciding which cases are normal limits. Another closely related fact is that syphilitic and which not, need not concern us here ; localcerebral compression (by which haemorrhage often it may be based on clinical material, or the histories causes its symptoms) is independent of the general of the cases, or the collateral use of other tests as pressure. The following example from my controls. Let us suppose the results worked out as own illustrates this point. In a fatal case experience follows :a large subdural haematoma compressed the underlying cerebral hemisphere, causing gross deformity ; yet the general intracranial pressure was only 80 mm. Aq. In a case in which subarachnoid haemorrhage resulted Let us not run away With the idea that Test A will from a fall and produced a clinical picture of local yield more errors than Test B. For Test A will only cerebral compression, the cerebro-spinal fluid pressure yield more errors than Test B as long as the number of i was under 180 mm. Aq. In another case of subdural syphilitics and non-syphilitics are equal. But will haematoma, in which depression of the cerebral they ever be equal in actual practice ? Most likely hemisphere was seen at operation, the pressure was not. And if these tests are investigated for the ’, 150 mm. Aq. In cases of head injury the pressure is often raised purpose of applying them as a routine to all such cases as are found in ordinary practice, or in a general when there is no haemorrhage and it is apparent from hospital, or in a mental hospital (to mention a few what I have said that it may be within normal limits possibilities), it is more likely that the material will and even low when haemorrhage is present. The include, not 50 per cent. of syphilitics, but something pressure reading has therefore little diagnostic value like 10 per cent. in the very cases in which, because of the necessity On that assumption of 10 per cent. syphilitics, it for operation, help in diagnosis is most required. "
I
intracranial
!I
CORRESPONDENCE would mean that in a series of 1000 cases, the errors with Test A would be 7 per cent. of 100, plus 1 per
STATISTICAL EXPRESSION OF RESULTS IN MEDICAL RESEARCH : TWO COMMON PITFALLS.
of THE LANCET. SIR,-The statistics required for the convenient recording of results in medical research work are not of a complicated kind, but it not infrequently happens that the interpretation of these statistics is rendered none the easier by an inappropriate method of calculating percentages. Let us take, as an example from laboratory work, the investigation of the accuracy of a particular To the Editor
B I
cent. of 900, that is, 16 cases, or 1-6 per cent. Those with Test B would work out at 2 per cent. of 100, plus 2 per cent. of 900, that is, 20 cases, or 2 per cent. This shows that in actual practice Test A would yield, not more, but 7ess errors than Test B. All this is elementary enough, and one would hesitate to mention these pitfalls were it not for the hope that such mention might assist towards a greater uniformity in recording results. It may be argued that where detailed figures are given it is an easy matter to work out the required facts for oneself ; but in these days of increased and rapid output of research, anyone who has to work through many references in a short while would be considerably assisted were the statistical results invariably given in a form that would express the true state of affairs, instead of one that is not only of little value but which may, to the unwary, prove misleading.
test. How often are the results couched in the form, " In so many cases, so many positives were obtained of which so many were incorrect." Or this may be abbreviated into, So many per cent. of the positives were false positives." If left in that form, what do these results convey1 Very little. For if, for we are told that " 5 per cent. of the posiexample, I am, Sir, yours faithfully, tives were incorrect," it subtly suggests a necessary J. ERNEST NICOLE. relation between the number of false positives and Winwick Mental Hospital, Jan. 17th, 1932. the total number of all positives. But any relation between them is really no more than an accidental one. The true relation to be expressed is the one THE CEREBRO-SPINAL FLUID PRESSURE IN between the number of cases giving false positivesCASES OF HEAD INJURY. i.e., cases failing to give correct negatives-and the To the Editor of THE LANCET. total number of cases that should have given negative results. A reductio ad absurdum will make the Sm,-At a discussion at the Royal Society matter clear. Suppose in the investigation of a test of Medicine on the Diagnosis and Treatment of Head for syphilis that 100 non-syphilitic cases-and only in Injuries reported THE LANCET of Dec. 12th, 1931, these cases-were tested, and suppose further that it seemed to me that quite an unwarranted degree of five positive results (therefore false positives) were was attached by most of the speakers to importance obtained, would we return our results as " 100 per the pressure of the cerebro-spinal fluid. The matter is cent. of the positives were incorrect " ?‘? The absurdity raised again by Dr. C. M. Hinds-Howell in the paper is obvious. What we really should say is 5 per cent. on Trauma and the Nervous System which appeared of the non-syphilitic cases were incorrectly returned in THE LANCET last week. as positive." At the discussion one of the speakers said The second source of misunderstanding lies in the that he decided not to operateopening on a case in which difference between statistical material obtained from he suspected intracranial haemorrhage, because the experimental series and the material likely to occur cerebro-spinal fluid pressure was not markedly raised. in actual practice. Again let a hypothetical example As long, however, as there is no interference with the illustrate the difficulty. circulation of the cerebro-spinal fluid, mere increase Two tests for syphilis, Test A and Test B, are in the volume of the other intracranial contents does under investigation, and the number of correct and not produce any significant rise in the pressure. Thus, incorrect results are expressed as percentages in the a large haematoma may be present the though manner suggested above. The actual criterion of fluid pressure may remain within cerebro-spinal accuracy, and the means of deciding which cases are normal limits. Another closely related fact is that syphilitic and which not, need not concern us here ; localcerebral compression (by which haemorrhage often it may be based on clinical material, or the histories causes its symptoms) is independent of the general of the cases, or the collateral use of other tests as pressure. The following example from my controls. Let us suppose the results worked out as own illustrates this point. In a fatal case experience follows :a large subdural haematoma compressed the underlying cerebral hemisphere, causing gross deformity ; yet the general intracranial pressure was only 80 mm. Aq. In a case in which subarachnoid haemorrhage resulted Let us not run away With the idea that Test A will from a fall and produced a clinical picture of local yield more errors than Test B. For Test A will only cerebral compression, the cerebro-spinal fluid pressure yield more errors than Test B as long as the number of i was under 180 mm. Aq. In another case of subdural syphilitics and non-syphilitics are equal. But will haematoma, in which depression of the cerebral they ever be equal in actual practice ? Most likely hemisphere was seen at operation, the pressure was not. And if these tests are investigated for the ’, 150 mm. Aq. In cases of head injury the pressure is often raised purpose of applying them as a routine to all such cases as are found in ordinary practice, or in a general when there is no haemorrhage and it is apparent from hospital, or in a mental hospital (to mention a few what I have said that it may be within normal limits possibilities), it is more likely that the material will and even low when haemorrhage is present. The include, not 50 per cent. of syphilitics, but something pressure reading has therefore little diagnostic value like 10 per cent. in the very cases in which, because of the necessity On that assumption of 10 per cent. syphilitics, it for operation, help in diagnosis is most required. "
I
intracranial
!I