- Email: [email protected]
The Vitamin and Mineral Mystery
STATISTICAL
CASE
Joseph M. Strayhorn, M.D.
PUZZLE
The Vitamin and Mineral Mystery "Is it possible that vitamin B6 and magnesium would help my autistic child behave better?" a parent asks Dr. Seeker. "I have a 1985 article here that reports positive results with that combination, and since that time I've found no more research on it," says Dr. Seeker. "As things are particularly bad for your child at this time, let's give it a try." So in a very nonrigorous trial, the vitamin and mineral are given. The parents report 2 weeks later that things have gotten a little better. The child's aggression is down, his attention span is better, he seems more cooperative, and he's a little happier, according to the parents. But none of these differences is very large. Dr. Seeker pays a visit to the cave of Dr. Wiseoldman to get some advice. Dr. Seeker says, "I think the vitamin and mineral might have really helped. Should I send this in as a case report? I may see another two or three children like this in the next 6 months, but I don't see enough to do a controlled study with random assignment of children l' IT ," to groups. How can I know we ' re reaylseeing an errect: Dr. Wiseoldman says, "I'm sorry, I can't give you as full an answer as you need right now. I'm due at a dance competition in 20 minutes. But let me pose for you two questions, the answers to which will let you answer your own question. "First: what are some of the 'rival hypotheses' that are threats to causal inferences that people make? Round up the usual suspects, and think about how they apply to the study as you've done it so far. "Second: what are two designs that can be used with only one or three cases, that can overcome many of these threats to your inference? Here are some books to look at (Campbell and Stanley, 1963; Cook and Campbell, 1979; Huck and Sandler, 1979; Kazdin, 1980). I'm sorry, that's all for now, goodbye. Stay as long as you want and lock the door when you leave. Write me a note about what you decide." And with that Dr. Wiseoldman takes off. Dr. Seeker reads Dr. Wiseoldman's statistics and research design books for a long time. What do you think he should say in his note to Dr. Wiseoldman?
Accepted March 23, 1993. Dr. Strayhorn is with the Medical College ofPennsylvania, Pittsburgh, PA. 0890-8567/94/3309-1346$03.00/0©1994 by the American Academy of Child and Adolescent Psychiatry.
1346
ANSWER: THE VITAMIN AND MINERAL MYSTERY
Dear Dr. Wiseoldman: One of the first things I learned was that the whole subject of research design is set up so as to make causal inferences. In our case, we want to infer that a treatment caused an improvement. In doing that, we are like a detective: before we settle on our main suspect hypothesis, we have to consider several rival hypotheses that can account for your findings unless your design rules them out. Campbell and Stanley's (1963) classic monograph systematized these rival hypotheses; Huck and Sandler (1979) extended the list; Cook and Campbell (1979) revised the list. Here are a few of the common rival hypotheses. Problem 1: History. This term refers to the events occurring between the first and second measurement other than the experimental variable. Example: In this case, if the child's parents started doing something better to control his behavior, or a substitute teacher arrived at school, or his sister went to camp, these events conceivably could have changed his behavior rather than the vitamin and mineral. Problem 2: Maturation. Here we refer to processes within the patient or subject, not related to external events, such as growing older, growing hungrier, growing more tired, etc. It is possible that in this child's uneven development, he may have taken a leap forward during the 2 weeks of treatment, one that had nothing to do with the treatment. Problem 3: Differential selection ofsubjects fOr groups. If I were to advertise for parents willing to try an experimental treatment with their child and to use as a comparison group a set of children at an institution, perhaps the group answering the ad has more motivated parents and children destined to improve more, even without the treatment. Problem 4: Statistical regression toward the mean. As explained in a previous statistical mystery, there is some random fluctuation in almost every variable we want to measure. If we select for extreme values for our first measurement, random fluctuation will tend to make those values closer to the overall average values for the second measurement. In the case of my patient, he was doing particularly poorly when we tried the treatment; it could be that during the next 2 weeks the improvement we saw was his simply returning to his average status quo. If we're talking about group studies, the problems I've mentioned so far are overcome by randomly assigning subjects to experimental or control groups. You can still get
]. AM. ACAD. CHILD ADOLESC. PSYCHIATRY. 33:9, NOVEMBERIDECEMBER 1994
STATISTICAL CASE PUZZLE
one group that has a different history, or that matures faster, or that was destined to improve more, or that regresses differently, but if the p found in the study is low enough, these rival hypotheses are highly improbable. However, two other problems can occur even in studies with random assignment, as follows: Problem 5: Instrumentation, in which differencesin measurement rather than differences in the real phenomenon account for the differences observed. Example: The measuring device in this case is the parents' informal observation and verbal rating. It could be that the hope induced by the new treatment caused them to pay more attention to the positive things and disregard negative things and to report improvement when, in fact, there was none. This problem is one of the main reasons for "blind" studies. Problem 6: Experimental mortality or attrition, where one group loses more subjects to follow-up than does the other group. This is a special case of selection. For example, suppose an experimental group has to attend many sessions of therapy; a control group has to do almost nothing. Therefore the therapy group selects for the most motivated subjects, who might have improved more no matter what the intervention was. If you are doing a group study, you can get around this by following up on all subjects, including those who dropped out of treatment as well as those who stayed on. Of course, the dropouts tend to reduce your average effect size and to lower the power of your study. Now back to my case. How can I rule out some of these rival hypotheses, with only one to three cases? With this case, I can use an "intrasubject replication design" (Kazdin, 1980), otherwise known as single-case design. One type is an ABAB study. I can give the treatment, take it away, give it back, and take it away again. If I see the patient's condition going up and down correspondingly, it's less probable that maturation or history or regression to the mean would account for the findings. An even more rigorous extension of this is the type of study where the giving and taking away of the variable is randomly assigned over time, such as studies where giving of stimulant medication is randomly assigned to days. If I can get together three patients, I can use a multiple baseline study. Here I would simultaneously conduct repeated measurements on the three subjects and enact the
treatment first with one, then with the second, then the third. If! can display the effect's taking place in the subjects in the order in which I enact the causal variable, the rival hypotheses of history, maturation, differential selection, and regression to the mean are weakened. If I use placebo and start the treatment at times to which the parents and children are blind, I can reduce the probability of instrumentation's accounting for the effects. Thanks so much for lending me the books that answered my question. I hope you enjoyed your dance competition. Sincerely, Dr. Seeker EPILOGUE
Three months later, Dr. Seeker writes again to Dr. Wiseoldman. Dear Dr. Wiseoldman, After using the principles of single case design that you guided me to, it appears that the vitamin and mineral don't seem to do much for this boy. It turns out that the most relevant variable looks to be a rival hypothesis that was in the category of "history." The patient gets better whenever his grandmother visits. She is an extremely joyous and patient and nice woman, and she's great with this kid. I'm now planning a study on the effects of joyous, patient, and nice adults on children. I'm expecting a huge effect size.
Sincerely, Dr. Seeker
REFERENCES Campbell OT, Stanley]C (1963), Experimental and Quasi-Experimental Designs for Research. Chicago: Rand, McNally College Publishing Cook TO, Campbell OT (1979), Quasi-Experimentation: Design and Analysis Issues for Field Settings. Boston: Houghton Mifflin Huck SW, Sandler HM (1979), Rival Hypotheses: AlternativeInterpretations of Data-Based Conclusions. New York: Harper & Row Kazdin AE (1980), Research Design in Clinical Psychology. New York: Harper & Row
]. AM. ACAD. CHILD ADOLESC. PSYCHIATRY. 33:9 NOVEMBERIDECEMBER 1994
1347
CASE
Joseph M. Strayhorn, M.D.
PUZZLE
The Vitamin and Mineral Mystery "Is it possible that vitamin B6 and magnesium would help my autistic child behave better?" a parent asks Dr. Seeker. "I have a 1985 article here that reports positive results with that combination, and since that time I've found no more research on it," says Dr. Seeker. "As things are particularly bad for your child at this time, let's give it a try." So in a very nonrigorous trial, the vitamin and mineral are given. The parents report 2 weeks later that things have gotten a little better. The child's aggression is down, his attention span is better, he seems more cooperative, and he's a little happier, according to the parents. But none of these differences is very large. Dr. Seeker pays a visit to the cave of Dr. Wiseoldman to get some advice. Dr. Seeker says, "I think the vitamin and mineral might have really helped. Should I send this in as a case report? I may see another two or three children like this in the next 6 months, but I don't see enough to do a controlled study with random assignment of children l' IT ," to groups. How can I know we ' re reaylseeing an errect: Dr. Wiseoldman says, "I'm sorry, I can't give you as full an answer as you need right now. I'm due at a dance competition in 20 minutes. But let me pose for you two questions, the answers to which will let you answer your own question. "First: what are some of the 'rival hypotheses' that are threats to causal inferences that people make? Round up the usual suspects, and think about how they apply to the study as you've done it so far. "Second: what are two designs that can be used with only one or three cases, that can overcome many of these threats to your inference? Here are some books to look at (Campbell and Stanley, 1963; Cook and Campbell, 1979; Huck and Sandler, 1979; Kazdin, 1980). I'm sorry, that's all for now, goodbye. Stay as long as you want and lock the door when you leave. Write me a note about what you decide." And with that Dr. Wiseoldman takes off. Dr. Seeker reads Dr. Wiseoldman's statistics and research design books for a long time. What do you think he should say in his note to Dr. Wiseoldman?
Accepted March 23, 1993. Dr. Strayhorn is with the Medical College ofPennsylvania, Pittsburgh, PA. 0890-8567/94/3309-1346$03.00/0©1994 by the American Academy of Child and Adolescent Psychiatry.
1346
ANSWER: THE VITAMIN AND MINERAL MYSTERY
Dear Dr. Wiseoldman: One of the first things I learned was that the whole subject of research design is set up so as to make causal inferences. In our case, we want to infer that a treatment caused an improvement. In doing that, we are like a detective: before we settle on our main suspect hypothesis, we have to consider several rival hypotheses that can account for your findings unless your design rules them out. Campbell and Stanley's (1963) classic monograph systematized these rival hypotheses; Huck and Sandler (1979) extended the list; Cook and Campbell (1979) revised the list. Here are a few of the common rival hypotheses. Problem 1: History. This term refers to the events occurring between the first and second measurement other than the experimental variable. Example: In this case, if the child's parents started doing something better to control his behavior, or a substitute teacher arrived at school, or his sister went to camp, these events conceivably could have changed his behavior rather than the vitamin and mineral. Problem 2: Maturation. Here we refer to processes within the patient or subject, not related to external events, such as growing older, growing hungrier, growing more tired, etc. It is possible that in this child's uneven development, he may have taken a leap forward during the 2 weeks of treatment, one that had nothing to do with the treatment. Problem 3: Differential selection ofsubjects fOr groups. If I were to advertise for parents willing to try an experimental treatment with their child and to use as a comparison group a set of children at an institution, perhaps the group answering the ad has more motivated parents and children destined to improve more, even without the treatment. Problem 4: Statistical regression toward the mean. As explained in a previous statistical mystery, there is some random fluctuation in almost every variable we want to measure. If we select for extreme values for our first measurement, random fluctuation will tend to make those values closer to the overall average values for the second measurement. In the case of my patient, he was doing particularly poorly when we tried the treatment; it could be that during the next 2 weeks the improvement we saw was his simply returning to his average status quo. If we're talking about group studies, the problems I've mentioned so far are overcome by randomly assigning subjects to experimental or control groups. You can still get
]. AM. ACAD. CHILD ADOLESC. PSYCHIATRY. 33:9, NOVEMBERIDECEMBER 1994
STATISTICAL CASE PUZZLE
one group that has a different history, or that matures faster, or that was destined to improve more, or that regresses differently, but if the p found in the study is low enough, these rival hypotheses are highly improbable. However, two other problems can occur even in studies with random assignment, as follows: Problem 5: Instrumentation, in which differencesin measurement rather than differences in the real phenomenon account for the differences observed. Example: The measuring device in this case is the parents' informal observation and verbal rating. It could be that the hope induced by the new treatment caused them to pay more attention to the positive things and disregard negative things and to report improvement when, in fact, there was none. This problem is one of the main reasons for "blind" studies. Problem 6: Experimental mortality or attrition, where one group loses more subjects to follow-up than does the other group. This is a special case of selection. For example, suppose an experimental group has to attend many sessions of therapy; a control group has to do almost nothing. Therefore the therapy group selects for the most motivated subjects, who might have improved more no matter what the intervention was. If you are doing a group study, you can get around this by following up on all subjects, including those who dropped out of treatment as well as those who stayed on. Of course, the dropouts tend to reduce your average effect size and to lower the power of your study. Now back to my case. How can I rule out some of these rival hypotheses, with only one to three cases? With this case, I can use an "intrasubject replication design" (Kazdin, 1980), otherwise known as single-case design. One type is an ABAB study. I can give the treatment, take it away, give it back, and take it away again. If I see the patient's condition going up and down correspondingly, it's less probable that maturation or history or regression to the mean would account for the findings. An even more rigorous extension of this is the type of study where the giving and taking away of the variable is randomly assigned over time, such as studies where giving of stimulant medication is randomly assigned to days. If I can get together three patients, I can use a multiple baseline study. Here I would simultaneously conduct repeated measurements on the three subjects and enact the
treatment first with one, then with the second, then the third. If! can display the effect's taking place in the subjects in the order in which I enact the causal variable, the rival hypotheses of history, maturation, differential selection, and regression to the mean are weakened. If I use placebo and start the treatment at times to which the parents and children are blind, I can reduce the probability of instrumentation's accounting for the effects. Thanks so much for lending me the books that answered my question. I hope you enjoyed your dance competition. Sincerely, Dr. Seeker EPILOGUE
Three months later, Dr. Seeker writes again to Dr. Wiseoldman. Dear Dr. Wiseoldman, After using the principles of single case design that you guided me to, it appears that the vitamin and mineral don't seem to do much for this boy. It turns out that the most relevant variable looks to be a rival hypothesis that was in the category of "history." The patient gets better whenever his grandmother visits. She is an extremely joyous and patient and nice woman, and she's great with this kid. I'm now planning a study on the effects of joyous, patient, and nice adults on children. I'm expecting a huge effect size.
Sincerely, Dr. Seeker
REFERENCES Campbell OT, Stanley]C (1963), Experimental and Quasi-Experimental Designs for Research. Chicago: Rand, McNally College Publishing Cook TO, Campbell OT (1979), Quasi-Experimentation: Design and Analysis Issues for Field Settings. Boston: Houghton Mifflin Huck SW, Sandler HM (1979), Rival Hypotheses: AlternativeInterpretations of Data-Based Conclusions. New York: Harper & Row Kazdin AE (1980), Research Design in Clinical Psychology. New York: Harper & Row
]. AM. ACAD. CHILD ADOLESC. PSYCHIATRY. 33:9 NOVEMBERIDECEMBER 1994
1347