- Email: [email protected]
Dealing with skewed data: an example using asthma-related costs of medicaid clients
CLINICAL THERAPEUTICSVVOL.
23, NO. 3,2001
Dealing with Skewed Data: An Example Using Asthma-Related Costs of Medicaid Clients Karen L. Rascati, PhD,’ Michael J. Smith, MS,‘* and Tor Neilands, PhD2* ‘University of Texas College of Pharmacy, Austin, Texas, and 2Center for AIDS Prevention Studies, University of California, San Francisco, California
ABSTRACT Background: Cost data often are nonnormally distributed due to a few very high cost values that may not necessarily be dismissed as outliers. Researchers have not reached agreement on how to appropriately deal with skewed cost data. Objectives: This study presents an example of skewed cost data that were collected retrospectively from the Texas Medicaid database. Common methods of dealing with skewed cost distributions are discussed. Data were analyzed using various methods, and the statistical results of each test were compared. Methods: Prescription and medical claims data extracted from the Texas Medicaid database were analyzed using the Mann-Whitney U test and t tests of untransformed, logtransformed, and bootstrapped data. Results: All distributions of the untransformed cost data were nonnormally distributed, and comparison groups had unequal variances. The Mann-Whitney U test negated the effect of the high-cost patients and gave a significant result for overall cost differences between groups, but in the opposite direction of the mean. The t tests on raw data and logtransformed data may not have been optimal because distributions of both raw costs and log-costs were nonnormal. Conclusions: The bootstrap method does not need to meet the assumptions of normality and equal variances. In analyses of small sample sizes with skewed cost data, the bootstrap method may offer an alternative to the more traditional nonparametric or logtransformation techniques. Key words: statistics, bootstrap, costs. (Clin Ther. 2001;23:481-498) Preliminary results of this study were presented at the annual meeting of the International Society for Pharmacoeconomics and Outcomes Research, Arlington, Virginia, May 2 l-24, 2000. *Michael .I. Smith was an American Foundation for Pharmaceutical Education Fellow and Tor Neilands was a senior systems analyst at the University of Texas at the time this manuscript was written. Accepted
for publication
February
1, 2001.
Printed in the USA. Reproduction in whole or part is not permitted.
0149.2918/01/$19.00
481
CLINICAL
INTRODUCTION Pharmacoeconomics researchers often work with cost data that are nonnormally distributed. Skewed cost distributions occur because a large number of costs cluster around a certain range of values, whereas a few high cost values are present in the tail. In addition, the distributions of 2 comparator groups may have unequal variances beyond a certain difference in the degree of skewness between the groups. Depending on the objectives of the study, as well as the perspective of the economic analysis, a researcher may be unable to dismiss the high cost values as outliers. Analyzing such data for statistical differences between groups can be problematic for the investigator. Researchers have not reached agreement on how to deal with this statistical problem. When comparing the mean cost differences between 2 groups, the independentgroup’s t test is a commonly used statistic. However, the t test for mean equality can be adversely affected by nonnormality, especially when the variances between groups are unequal and group sizes are sharply unequal. * The standard deviation is greatly influenced by extreme cost values, which also can affect the standard error of the mean. As a result, the power to detect differences between groups can be reduced.’ This especially becomes an issue for the t test when a researcher is dealing with small to moderate sample sizes. Hays* considered homogeneity of variance a more important assumption of the t test than the assumption of normality, although the tests of equality of variances are less reliable when most needed (ie, under small sample conditions). However, Hays noted that even when large variance differences occur between 2 groups, there
482
THERAPEUTICS”
tends to be little effect on the t statistic when the group sizes are equal or nearly equal.2 When group sizes are unequal and the sample sizes are small, the homogeneityof-variances assumption of the t test becomes important. Stevens3 has asserted that skewness tends to have little effect on inference from models of analysis of variance (of which the t test is a special case), although platykurtosis curtails power somewhat. Stevens noted that research in the statistical literature has shown that if the ratio of the largest group to the smallest group does not exceed 1.5: 1, the F test also is robust against violations of the equalityof-variances assumption3 When that ratio of largest to smallest group is exceeded, the F test becomes too liberal (ie, rejects the null hypothesis too often) if the larger variance is associated with the smaller group; conversely, if the larger variance is associated with the larger group, F becomes too conservative and does not reject the null hypothesis often enough. Alternative techniques have been proposed to analyze skewed data sets. For example, a pharmacoeconomics study comparing treatments for patients with chronic obstructive pulmonary disease and asthma addressed the issue of analyzing skewed data.4 The authors used a simple logarithmic transformation of the data to create a symmetric cost distribution. Under this method, statistical testing usually is performed on the log-transformed data, and the resulting P values are reported for the null hypothesis based on untransformed data. According to Zhou and colleagues5 the problem with such a practice is that the null hypothesis based on logtransformed data may not be equivalent to the null hypothesis based on the original outcomes. Using a case example, these
K.L. RASCATI
ET AL.
authors illustrated that when variances are unequal, the type I error rate of the t test based on log-transformed data increases as the sample size or the difference in skewness increases. Log-transformed data also must be assessed for normality and variability.6*7 Parametric testing (eg, the f test) is appropriate when the log-data are approximately normally distributed and the variances of log-data are equal between comparator groups. Violating these assumptions may produce inaccurate results, especially when there is a larger degree of skewness in 1 of the comparator group distributions. Coyle6 has suggested that other analyses may be necessary if the log-transformed data fail to approximate a normal distribution. Nonparametric or distribution-free tests are statistical options when data are nonnormally distributed. The advantage of these tests is that there are no restricting distributions that limit their use. However, nonparametric methods require that the shape and variance of the distributions of the 2 comparator groups be the same. The Mann-Whitney U test is considered the nonparametric alternative to the independent-groups t test. Statistical significance is determined by analyzing the difference in mean ranks of the data, as opposed to the actual data values. Coyle believes nonparametric tests are not ideal for pharmacoeconomic analyses, because rank data remove the relative value of each cost data point6 As a result, Coyle notes, patients who consume a disproportionate amount of resources are not accurately considered or accounted for in nonparametric testing.6 The aforementioned statistical tests are common approaches for conducting pharmacoeconomic analyses when one is in-
terested in whether a significant difference in mean costs exists between 2 therapeutic options. Given the nature of cost data, however, the assumptions of parametric tests limit their utility in economic studies. Furthermore, a nonparametric approach may be inappropriate for analyzing cost data due to the inherent testing procedures of these types of statistics. Coyle argues that the main concern of pharmacoeconomic analyses is a comparison of mean values between treatment options and recommends that methods of analysis that compare mean values should be used even when cost data are nonnormally distributed.6 Recently, the bootstrap technique has been used as an alternative method for analyzing skewed pharmacoeconomic data.8-” Efron and colleagues’2~13 first used the term bootstrap and developed the statistical theory behind the technique. The bootstrap method analyzes samples that are randomly selected from an empirical distribution of observed data. The original data set is considered to be the parent population. A simulation of repeated random sampling is conducted on the original data set, with each sampled item replaced after each random draw. The statistic of interest is calculated for each sample, and a subsequent sampling distribution of the statistic is formed. The sampling distribution’s variability is then used to calculate standard errors, confidence intervals, and P values. Resampling theory differs from traditional parametric theory in that the observed distribution is considered to be the population, unlike the hypothetical or unobserved population used in traditional parametric theory. The advantage of the bootstrap resampling technique is that “... no theoretical models or mathe-
483
CLINICAL THERAPEUTICS@
matical assumptions are needed ....” I4 The only assumption of the bootstrap is that the original data set that serves as the parent population is representative of the larger population to which the investigator wishes to draw inferences (an assumption that is implicitly part of any parametric or nonparametric statistical analysis). Therefore, the distribution does not have to behave in any particular manner when a bootstrap technique is used.15 Statistical values are calculated directly from the resamplings of the empirically observed data. There are limitations to the bootstrap technique, however. It is a more complicated manipulation and may require sophisticated software to compute the resampling statistics. Although the data do not have to be normally distributed, the researcher must assume that the distribution of the sample approximates the distribution of the populations. Moreover, although the bootstrap can be used for inferential statistics, it is not as useful for point estimates of parameters.i6 In 1998, Desgagne et al” presented an example of the use of the bootstrap on a skewed pharmacoeconomic data set. The authors presented the results of different statistical tests conducted on the data set and provided a discussion of the bootstrap procedure. The authors recommend the use of the nonparametric bootstrap test when conducting pharmacoeconomic analyses on small to moderate samples of skewed data in which the difference in mean costs between treatments is being determined.” Most of the literature discussed earlier focuses on the collection of economic data parallel to a randomized clinical trial. The purpose of this article is to present an example of skewed cost data that were collected retrospectively from a large claims
484
database. Results of various statistical tests conducted on the data are reported and discussed.
METHODS The data used for these analyses were taken from a previous study that investigated the cost differences of adding inhaled corticosteroids to medication regimens for patients with persistent asthma in the Texas Medicaid system.i8 Asthmarelated medical and prescription claims in the Texas Medicaid database were used to form a retrospective, matched cohort. Twelve months of data were analyzed in which the cost difference between the 6 months before introduction of treatment and the 6 months after introduction of treatment was compared between a group of asthma patients introduced to inhaled steroids (intervention group) and a group of asthma patients who were treated with any asthma medications other than inhaled steroids (comparison group). Information generated from this study was analyzed using 4 methods: MannWhitney U nonparametric test, t test on untransformed data, t test on logarithmictransformed data, and t test on bootstrapped data. For each of the statistical methods conducted, 3 tests were compared between groups. One test compared differences in Medicaid prescription costs, 1 test compared differences in Medicaid medical utilization costs, and 1 test compared differences in the combined prescription and medical costs (overall costs). The perspective of the study was costs to the Texas Medicaid program. An a level of 0.05 was used to test for significance. Analyses were conducted using SPSS@ version 8.0 for Windows (SPSS, Inc, Chicago, Illinois) and SAS@ Language and Procedures ver-
K.L. RASCATI
ET AL.
Table I. Summary of Medicaid prescription, tion and comparison groups.
medical, and overall costs for the interven-
Mean Cost Difference (SD)
Intervention Group* Type of Cost Prescription Medical Overall
(n = 99) $342.29 ($336.52) -$3 17.84 ($1301.24) $24.45 ($1380.42)
Comparison Group+ (n = 99) $23.12 ($2 14.94) $81.93 ($2207.08) $105.04 ($2222.78)
Cost Difference Between Groups $319.17 -$399.77 -$80.59
*Six-month costs after initiation of inhaled corticosteroid therapy minus 6-month costs before initiation of therapy. %x-month costs after index date minus 6-month costs before index date.
sion 8.1 (SAS Institute, Inc, Cary, North Carolina) software. The data were examined for outliers via visual inspection by the researchers. In addition, histograms, normal Q-Q plots,‘9 and the KolmogorovSmirnov (K-S)19 test for normality were used to assess the distribution of data. To be included in the intervention group, patients had to have 4 corticosteroid inhalers filled within 6 months after beginning this therapy (index date). With this strict criterion, 99 patients were included in the intervention group. Thousands of Medicaid patients met the criteria for the comparison group (ie, continuous enrollment, diagnosis of asthma, and 24 asthmarelated prescription claims for each 6-month period). Therefore, one-to-one matching was conducted based on index date, age, sex, diagnosis, and preindex-date B-month prescription and medical costs.
RESULTS The summary statistics of the asthmarelated costs for the intervention and comparison groups are shown in Table I.
Normality and Variunce Tests: Untransformed Data Prescription
Costs
To determine whether the change in Medicaid costs before and after treatment introduction for the intervention and comparison groups were normally distributed, visual inspections of graphics and statistical tests for normality were conducted. Figure 1A shows the frequency distribution of the difference in untransformed Medicaid prescription costs before and after treatment introduction for the intervention group. Figure 2A shows the distribution of untransformed data for the comparison group. The distribution of prescription cost differences for the intervention group shows that these data are positively skewed (see Figure 1A). The K-S test shows that the distribution is nonnorma1 (K-S = 0.113, df = 99, P = 0.003). The distribution for the comparison group (see Figure 2A) shows high prescription cost differences in both tails, and the K-S test shows nonnormality (K-S = 0.101, df = 99, P = 0.015). The equality-of-
485
CLINICAL THERAPEUTICS@
variances test showed that variances between the groups were unequal (F = 2.450, df, = 98, df2 = 98, P < 0.001). Medical Costs
Figure 3A shows the untransformed data frequency distribution of the difference in Medicaid medical costs before and after treatment introduction for the intervention group. Figure 4A shows the untransformed data distribution for the comparison group. The distributions of untransformed data are similar in both groups in that a large number of cost differences cluster around the mode, whereas a few high cost-difference values lie in the tails (see Figures 3A and 4A). The K-S tests show that the untransformed distributions for the intervention group (K-S = 0.369, df = 99, P < 0.001) and comparison group (K-S = 0.353, df = 99, P < 0.001) are nonnormal. The equality-ofvariances test showed that the variances between groups were unequal (F = 2.880, df, = 98, df2 = 98, P < 0.001). Overall Costs
Figures 5A and 6A show the frequency distributions of the difference in untransformed Medicaid overall costs before and after treatment introduction for the intervention and comparison groups, respectively. The distributions of untransformed data are similar in both groups in that a large number of cost differences cluster around a certain range of values, whereas a few high cost-difference values lie in the tails. The K-S tests show that the untransformed distributions for the intervention group (K-S = 0.297, df = 99, P < 0.001) and comparison group (K-S = 0.299, df = 99, P < 0.001) are nonnormal. There was a significant difference in the variances between the groups (F = 2.590, df, = 98, df2 = 98, P < 0.001).
486
Normality and Variance Tests: Log Data Prescription
Costs
The prescription, medical, and overall data sets for both treatment groups also were transformed logarithmically. Figure 1B shows the frequency distribution of the log of the difference in Medicaid prescription costs before and after treatment introduction for the intervention group. Figure 2B shows the distribution of log data for the comparison group. The K-S test showed that only the log of the difference in prescription costs among the intervention group was normally distributed (K-S = 0.080, df = 99, P = 0.127). The shape of the distributions and the standard deviations between the 2 groups are quite different. The variances between groups were significantly different (F = 4.780, df, = 98, df, = 98, P = 0.001). Medical Costs
Figure 3B shows the log-data frequency distribution of the difference in Medicaid medical costs before and after treatment introduction for the intervention group. Figure 4B shows the log-data distribution for the comparison group. The K-S test shows that both distributions are nonnorma1 (intervention group: K-S = 0.432, df = 99, P < 0.001; comparison group: K-S = 0.326, df = 99, P < 0.001). The equalityof-variances test showed that the variances between groups were unequal (F = 16.860, df, = 98, df, = 98, P = 0.001). Overall Costs
Figures 5B and 6B show the frequency distributions of the log of the difference in Medicaid overall costs before and after treatment introduction for the intervention and comparison groups, respectively. The K-S test shows that the transformed
K.L. RASCATI ET AL.
AwanbaJ
a
j
AwanbaJ
j
487
1
0
10
40
Dollars ($)
214.94 Mean = 23.12 N = 99
B
20
60
SD = 0.78 Mean = 6.35 N = 99
Figure 2. Distribution of the difference in prescription costs before and after treatment (B), and bootstrap (C) data for the comparison group.
A
introduction
in untransformed
Dollars ($)
SD = 23.82 Mean = 22.36 N=150
(A), log
E@
5 8 $ 3 z
0
20
60
100
SD = 1301.24 Mean = -317.84 N=99
B SD = 0.92 Mean = 8.71 N=99
Figure 3. Distribution of the difference in medical costs before and after treatment and bootstrap (C) data for the intervention group.
A
introduction
C
in untransformed
Dollars ($)
SD = 126.46 Mean = -315.77 N=150
(A), log (B),
1
100
Dollars ($)
SD = 2207.08 Mean = 81.93 N = 99
B
100
SD = 0.22 Mean = 8.85 N = 99
Lo9 6)
Figure 4. Distribution of the difference in medical costs before and after treatment and bootstrap (C) data for the comparison group.
A
introduction
C
in untransformed
Dollars ($)
SD = 205.28 Mean = 101.38 N= 150
(A), log (B),
E0
2
2 E %
F
K.L. RASCATI
ET AL.
data sets of the intervention and comparison groups were nonnormally distributed (K-S = 0.423, df = 99, P < 0.001; K-S = 0.299, df = 99, P < 0.001, respectively). The variances between groups were significantly different (F = 18.230, df, = 98, df2 = 98, P = 0.001).
Distributions of Untransformed Bootstrap Data
Versus
The untransformed prescription, medical, and overall data sets for both treatment groups also were analyzed using the bootstrap technique. In this study, 150 bootstrap samples were selected in which each sample contained 99 random item draws with replacement. Therefore, the mean and standard deviations produced by the bootstrap methods in this study were calculated on a sample of 150 data points, whereas the descriptive statistics for the untransformed and log data sets were based on the sample of 99 empirical observations. Figures 1C and 2C show the frequency distributions of the difference in Medicaid prescription costs for the intervention and comparison groups after bootstrapping. Figure 3C shows the frequency distribution of the difference in Medicaid medical costs for the intervention group after bootstrapping, and Figure 4C shows the distribution of bootstrap medical data for the comparison group. Figures 5C and 6C show the frequency distributions of the difference in Medicaid overall costs for the intervention and comparison groups, respectively, after bootstrapping. The calculated mean difference in medical and overall Medicaid costs for the comparison group is slightly larger in the bootstrap distributions than in the corresponding untransformed distributions
(see Figure 4C vs Figure 4A, and Figure 6C vs Figure 6A). However, the mean values produced by the other bootstrap distributions are similar to those produced from the untransformed data. As expected, in all instances the variation in the bootstrap distributions is much lower than that in the corresponding untransformed distributions. Comparisons
of Statistical
Tests
Prescription Costs The results of the different statistical tests comparing differences in Medicaid prescription, medical, and overall costs between groups are shown in Tables II through IV. All of the statistical tests show that Medicaid prescription costs among patients with asthma treated with inhaled steroids were significantly higher than those for patients who were treated with therapies other than inhaled steroids (see Table II) Medical Costs When statistical differences were compared between groups with respect to medical costs, the Mann-Whitney U method was the only test to report a significant difference (Table III). The MannWhitney U test showed that patients with asthma treated with inhaled steroids had a lower median ranking of medical costs than those who did not receive an inhaled steroid. Overall Costs When statistical differences were compared between groups with respect to overall costs, the Mann-Whitney U test again was the only test to report a significant difference (Table IV). However, the direction of significance was oppo-
491
CLINICAL THERAPEUTICS”
m
492
K.L. RASCATI
ET AL.
m
a
kJuanbaJj
493
CLINICAL THERAPEUTICS”
Table II. Results of the different statistical tests comparing scription costs between groups. Method
differences
in Medicaid pre-
Test Statistic
t test (log data)* t test (bootstrap data)
Z = -7.720 t = 7.953 t=5.412 t = 7.960
IG = intervention group; CG = comparison *Test reported for unequal group variances.
Mean Mean Mean Mean
rank cost cost cost
of IG > CG of IG > CG of IG > CG of IG > CG
group.
Table III. Results of the different statistical tests comparing differences in Medicaid medical costs between groups. Method Mann-Whitney
U test
t test (untransformed data)* t test (log data)* t test (bootstrap data)
Test Statistic
P
Direction of Significance
Z = -2.004 t = -1.550 t = -1.379 t = -1.620
0.045 0.122 0.171 0.106
Mean rank of IG < CG NS NS NS
IG = intervention group; CG = comparison *Test reported for unequal group variances.
group.
site to the mean dollar value. The mean difference in overall Medicaid costs for the intervention group ($24.25) was lower than for the comparison group ($105.04), yet the Mann-Whitney U test indicated that the intervention group had a higher median ranking than the comparison group.
Sample Size Variation Various factors may play a role in the need for alternate statistical methods, including the degree of skewness and nonnormality, the overall sample size, and the comparability of the sample sizes between groups. The effects of 2 of these factors-
494
the equality of sample sizes and the total sample size-were addressed through a reanalysis of medical costs, the most highly skewed type of cost data in our original analyses. Table V shows how variations in the overall sample size and the equality of sample sizes affect the results. To conduct this reanalysis, we matched each original patient in the intervention group with 2 more comparison patients. Random samples of the intervention group were then chosen and matched to the corresponding comparison patients (in a 1: 1 or 1:3 ratio). It is noteworthy that even when the comparison group has 3 times as many observations as the intervention group, the standard deviation of the com-
K.L. RASCATI
ET AL.
Table IV. Results of the different statistical tests comparing all costs between groups. Method
t test (untransformed data)* t test (log data)* t test (bootstrap data)
differences
in Medicaid over-
Test Statistic Z = -3.216 t =-0.310 t = -0.894 t = -0.370
IG = intervention group; CG = comparison *Test reported for unequal group variances.
0.001 0.760 0.373 0.710
Mean rank of IG > CG NS NS NS
group.
parison group is still higher and the inequality of variances is still an issue. In comparing a sample of 50 intervention patients to a sample of 150 comparison patients, although the total sample size (N = 200) is similar to the original total sample (N = 198), the differences between the untransformed t test and the bootstrap t test results are important. Assuming the P value was set a priori at 0.05, a different conclusion would be reached using the bootstrap method.
DISCUSSION The nature of the frequency distributions discussed in this article are typical of those in pharmacoeconomic analyses. The Mann-Whitney U test found statistically significant cost differences between groups for each type of cost. However, in this study, as in the study by Desgagne and coworkers,” the Mann-Whitney U test produced an unexpected result. For example, the test result of the overall costs analysis would lead a researcher to conclude that the intervention group had significantly higher overall asthma-related costs than the comparison group. The direction of significance is contrary to the
mean cost differences listed in Table I. This is probably due to the fact that the Mann-Whitney U nonparametric test replaces actual observations with rank order, thereby reducing the magnitude of certain high cost values. Often, a treatment is specifically aimed at minimizing the chance of these few, but high-cost, events. All of the methods used to conduct t tests in the original study sample produced similar t values. However, all but one of the log-transformed data sets were nonnormally distributed. In addition, the variance tests showed that variances were significantly different between groups for all types of costs. Under such circumstances, the assumptions of log-transformed data are not met, and the t test may produce unreliable results. A t test also was conducted on the untransformed data. Because the initial test was conducted on samples of equal size (n = 99), the assumption of homogeneity of variances probably does not affect the test results. Furthermore, the t test allows the adjustment of unequal variances. However, the nonnormal distributions of the original data sets analyzed in this study violate the assumption of the test. Although the t test is robust to nonnormality, some-
495
$67.67 ($1952.46) $146.17 ($1623.42)
-$405.06 ($1353.63)
-$317.84
($1301.24)
50: 150
99:297 -2.88
-1.89
-1.63
-1.19
-1.55
t
0.06
0.11
0.24
0.12
P
._.
for CGt
Difference
($126.46)
-$315.77
($95.86)
$154.92
$62.03 ($153.27)
($202.75)
($205.26)
$101.38
($595.16)
$274.50
($205.26)
$101.38
(SD)
Mean
-$416.54
($285.87)
-$515.81
($285.87)
-$515.81
($126.46)
-$315.77
(SD) for IG”
Mean Difference
Bootstrap Data
IG = intervention group; CG = comparison group. *Six-month medical costs after initiation of inhaled corticosteroid therapy minus 6-month medical costs before initiation of therapy. %ix-month medical costs after index date minus 6-month medical costs before index date. *Original sample.
$81.93 ($2207.08)
$311.70 ($3528.46)
-$481.33
($1514.29)
$81.93 ($2207.08)
-$3 17.84
(SD) for CG’
Mean Difference
($1301.24)
(SD) for IG*
Mean Difference
-$481.33
Size (1G:CG)
Untransformed Data
on medical costs results.
($1514.29)
33:99
33:33
99:99*
Sample
Table V. Effect of sample size variations
-2.90
-1.94
-1.76
-1.21
-1.62
t
0.05
0.08
0.23
0.11
P
K.L. RASCATI ET AL.
times nonnormality can be severe. The researcher should use discretion in interpreting the results of a t test when economic data from small to moderate sample sizes are analyzed. Interestingly, the t tests reported in this study for the untransformed and bootstrapped data were similar for each type of cost (see Tables II through IV). A reanalysis of the medical costs using different sample sizes was conducted. As expected, analyses using smaller sample sizes were less likely to detect a statistical difference between the means. In 1 case, when sample sizes were unequal between groups, the bootstrap technique and the untransformed t tests produced different results. The bootstrap technique may be a more appealing method to analyze the data in this study because of its relative freedom from restricting assumptions and because not all research studies have equal group sizes. The bootstrap method can be generalized to most statistical tests and can even be used to test hypotheses for which there currently is not a normal-theory standard error. When pharmacoeconomic data grossly violate the assumption of normality, the bootstrap method is a way to ensure that the researcher is using a correct underlying distribution for hypothesistesting purposes. Although the unequalvariances t test is a viable option to control for assumption violations, the bootstrap is a more general solution for inequality of variances and nonnormality. Although the bootstrap technique is not without limitations (eg, complex programming and the assumption that the distribution of the sample approximates the true distribution), it can be used to check the validity of the results of untransformed t tests when working with small, skewed samples.
CONCLUSIONS When comparing cost differences between treatment options, pharmacoeconomics researchers often analyze data that are skewed and nonnormally distributed. The bootstrap method does not need to meet the assumptions of normality or equal variances, or the shape of the distributions. In addition, the bootstrap method considers the impact of high cost values, unlike nonparametric statistics, which remove their influence. When working with small sample sizes of skewed cost data, the bootstrap method may be an alternative to nonparametric tests or log transformations.
ACKNOWLEDGMENTS The authors would like to thank and acknowledge James P. Wilson, PhD, and Kenneth Lawson, PhD, University of Texas College of Pharmacy, for their guidance and collaboration on this study.
REFERENCES Wilcox RR, Keselman HJ, Kowalchuk RK. Can tests for treatment group equality be improved? The bootstrap and trimmed means conjecture. Br J Math Stat Psychol. 1998;51:123-134. Hays WL. Statistics. New York, NY Holt, Winston, Rinehart; 1988:303. Stevens J. Intermediate Statistics: A Modern Approach. Hillsdale, NJ: Lawrence
Erlbaum and Associates: 1986:42. Rutten-van MPMH, Molken van Doorslaer EKA, van Vliet RCJA. Statistical analysis of cost outcomes in a randomized controlled clinical trial. Health Econ. 1994;3:333-345.
497
CLINICAL THERAPEUTICS”
5. Zhou XH, Gao S, Hui SL. Methods for comparing the means of two independent log-normal samples. Biometrics. 1997;53: 1129-1135.
13. Efron B, Tibshirani R. An Introduction to the Bootstrap. New York, NY: Chapman & Hall; 1993. 14. Fiellin DA, Feinstein AR. Bootstraps and jackknives: New, computer-intensive statistical tools that require no mathematical theories. J Invest Med. 1998;46:22-26.
6. Coyle D. Statistical
analysis in pharmacoeconomic studies. Pharmacoeconomics. 1996;9:506-5 16.
15. O’Brien BJ, Drummond MF, Labelle RJ, Willan A. In search of power and significance: Issues in the design and analysis of stochastic cost-effectiveness studies in health care. Med Care. 1994;32:15&163.
7. Zhou XH, Melfi CA, Hui SL. Methods for comparison of cost data. Ann Intern Med. 1997; 127:752-756.
8. Obenchain
RL. Resampling and multiplicity in cost-effectiveness inference. J Biophamz Stat. 1999;9:563-582.
16. Mooney CZ, Duval RD. Bootstrapping: A Nonparametric Approach to Statistical Inference. Sage University Paper Series on Quantitative Applications in the Social Sciences, 07-095. Newbury Park, CA: Sage Publications; 1993.
9. Jones J, Wilson A, Parker H, et al. Economic evaluation of hospital at home versus hospital care: Cost minimisation analysis of data from randomised controlled trial. Br Med J. 1999;3 19: 1547-1550.
17. Desgagne A, Castilloux AM, Angers JF, LeLorier J. The use of the bootstrap statistical method for the pharmacoeconomic cost analysis of skewed data. Pharmacoeconomics. 1998; 13:487-497.
10. Mehta SS, Suzuki S, Glick HA, Schulman KA. Determining an episode of care using claims data. Diabetic foot ulcer. Diabetes Care. 1999;22:1110-1115.
18. Smith MJ, Rascati KL, Johnsrud MT. Use of inhaled corticosteroids to treat Texas Medicaid clients with asthma: Assessment of prescriptions and medical service payments. J Am Pharm Assoc. 2000;40:307. Abstract.
11. Briggs AH, Wonderling DE, Mooney CZ. Pulling cost-effectiveness analysis up by its bootstraps: A non-parametric approach to confidence interval estimation. Health Econ. 1997;6:327-340.
19. SPSS Inc. SPSS@ 7.5 Guide to Data Analysis, 1997. Chicago, III: Prentice-Hall; 1997: 223-225.
12 Efron B. Bootstrap methods: Another look at the jackknife. Ann Stat. 1979;7: l-26.
Address correspondence to: Karen L. Rascati, of Pharmacy Practice [email protected]
498
and Administration,
PhD, The University Mail Code: A1930, Austin,
of Texas, Division TX 78712. E-mail:
23, NO. 3,2001
Dealing with Skewed Data: An Example Using Asthma-Related Costs of Medicaid Clients Karen L. Rascati, PhD,’ Michael J. Smith, MS,‘* and Tor Neilands, PhD2* ‘University of Texas College of Pharmacy, Austin, Texas, and 2Center for AIDS Prevention Studies, University of California, San Francisco, California
ABSTRACT Background: Cost data often are nonnormally distributed due to a few very high cost values that may not necessarily be dismissed as outliers. Researchers have not reached agreement on how to appropriately deal with skewed cost data. Objectives: This study presents an example of skewed cost data that were collected retrospectively from the Texas Medicaid database. Common methods of dealing with skewed cost distributions are discussed. Data were analyzed using various methods, and the statistical results of each test were compared. Methods: Prescription and medical claims data extracted from the Texas Medicaid database were analyzed using the Mann-Whitney U test and t tests of untransformed, logtransformed, and bootstrapped data. Results: All distributions of the untransformed cost data were nonnormally distributed, and comparison groups had unequal variances. The Mann-Whitney U test negated the effect of the high-cost patients and gave a significant result for overall cost differences between groups, but in the opposite direction of the mean. The t tests on raw data and logtransformed data may not have been optimal because distributions of both raw costs and log-costs were nonnormal. Conclusions: The bootstrap method does not need to meet the assumptions of normality and equal variances. In analyses of small sample sizes with skewed cost data, the bootstrap method may offer an alternative to the more traditional nonparametric or logtransformation techniques. Key words: statistics, bootstrap, costs. (Clin Ther. 2001;23:481-498) Preliminary results of this study were presented at the annual meeting of the International Society for Pharmacoeconomics and Outcomes Research, Arlington, Virginia, May 2 l-24, 2000. *Michael .I. Smith was an American Foundation for Pharmaceutical Education Fellow and Tor Neilands was a senior systems analyst at the University of Texas at the time this manuscript was written. Accepted
for publication
February
1, 2001.
Printed in the USA. Reproduction in whole or part is not permitted.
0149.2918/01/$19.00
481
CLINICAL
INTRODUCTION Pharmacoeconomics researchers often work with cost data that are nonnormally distributed. Skewed cost distributions occur because a large number of costs cluster around a certain range of values, whereas a few high cost values are present in the tail. In addition, the distributions of 2 comparator groups may have unequal variances beyond a certain difference in the degree of skewness between the groups. Depending on the objectives of the study, as well as the perspective of the economic analysis, a researcher may be unable to dismiss the high cost values as outliers. Analyzing such data for statistical differences between groups can be problematic for the investigator. Researchers have not reached agreement on how to deal with this statistical problem. When comparing the mean cost differences between 2 groups, the independentgroup’s t test is a commonly used statistic. However, the t test for mean equality can be adversely affected by nonnormality, especially when the variances between groups are unequal and group sizes are sharply unequal. * The standard deviation is greatly influenced by extreme cost values, which also can affect the standard error of the mean. As a result, the power to detect differences between groups can be reduced.’ This especially becomes an issue for the t test when a researcher is dealing with small to moderate sample sizes. Hays* considered homogeneity of variance a more important assumption of the t test than the assumption of normality, although the tests of equality of variances are less reliable when most needed (ie, under small sample conditions). However, Hays noted that even when large variance differences occur between 2 groups, there
482
THERAPEUTICS”
tends to be little effect on the t statistic when the group sizes are equal or nearly equal.2 When group sizes are unequal and the sample sizes are small, the homogeneityof-variances assumption of the t test becomes important. Stevens3 has asserted that skewness tends to have little effect on inference from models of analysis of variance (of which the t test is a special case), although platykurtosis curtails power somewhat. Stevens noted that research in the statistical literature has shown that if the ratio of the largest group to the smallest group does not exceed 1.5: 1, the F test also is robust against violations of the equalityof-variances assumption3 When that ratio of largest to smallest group is exceeded, the F test becomes too liberal (ie, rejects the null hypothesis too often) if the larger variance is associated with the smaller group; conversely, if the larger variance is associated with the larger group, F becomes too conservative and does not reject the null hypothesis often enough. Alternative techniques have been proposed to analyze skewed data sets. For example, a pharmacoeconomics study comparing treatments for patients with chronic obstructive pulmonary disease and asthma addressed the issue of analyzing skewed data.4 The authors used a simple logarithmic transformation of the data to create a symmetric cost distribution. Under this method, statistical testing usually is performed on the log-transformed data, and the resulting P values are reported for the null hypothesis based on untransformed data. According to Zhou and colleagues5 the problem with such a practice is that the null hypothesis based on logtransformed data may not be equivalent to the null hypothesis based on the original outcomes. Using a case example, these
K.L. RASCATI
ET AL.
authors illustrated that when variances are unequal, the type I error rate of the t test based on log-transformed data increases as the sample size or the difference in skewness increases. Log-transformed data also must be assessed for normality and variability.6*7 Parametric testing (eg, the f test) is appropriate when the log-data are approximately normally distributed and the variances of log-data are equal between comparator groups. Violating these assumptions may produce inaccurate results, especially when there is a larger degree of skewness in 1 of the comparator group distributions. Coyle6 has suggested that other analyses may be necessary if the log-transformed data fail to approximate a normal distribution. Nonparametric or distribution-free tests are statistical options when data are nonnormally distributed. The advantage of these tests is that there are no restricting distributions that limit their use. However, nonparametric methods require that the shape and variance of the distributions of the 2 comparator groups be the same. The Mann-Whitney U test is considered the nonparametric alternative to the independent-groups t test. Statistical significance is determined by analyzing the difference in mean ranks of the data, as opposed to the actual data values. Coyle believes nonparametric tests are not ideal for pharmacoeconomic analyses, because rank data remove the relative value of each cost data point6 As a result, Coyle notes, patients who consume a disproportionate amount of resources are not accurately considered or accounted for in nonparametric testing.6 The aforementioned statistical tests are common approaches for conducting pharmacoeconomic analyses when one is in-
terested in whether a significant difference in mean costs exists between 2 therapeutic options. Given the nature of cost data, however, the assumptions of parametric tests limit their utility in economic studies. Furthermore, a nonparametric approach may be inappropriate for analyzing cost data due to the inherent testing procedures of these types of statistics. Coyle argues that the main concern of pharmacoeconomic analyses is a comparison of mean values between treatment options and recommends that methods of analysis that compare mean values should be used even when cost data are nonnormally distributed.6 Recently, the bootstrap technique has been used as an alternative method for analyzing skewed pharmacoeconomic data.8-” Efron and colleagues’2~13 first used the term bootstrap and developed the statistical theory behind the technique. The bootstrap method analyzes samples that are randomly selected from an empirical distribution of observed data. The original data set is considered to be the parent population. A simulation of repeated random sampling is conducted on the original data set, with each sampled item replaced after each random draw. The statistic of interest is calculated for each sample, and a subsequent sampling distribution of the statistic is formed. The sampling distribution’s variability is then used to calculate standard errors, confidence intervals, and P values. Resampling theory differs from traditional parametric theory in that the observed distribution is considered to be the population, unlike the hypothetical or unobserved population used in traditional parametric theory. The advantage of the bootstrap resampling technique is that “... no theoretical models or mathe-
483
CLINICAL THERAPEUTICS@
matical assumptions are needed ....” I4 The only assumption of the bootstrap is that the original data set that serves as the parent population is representative of the larger population to which the investigator wishes to draw inferences (an assumption that is implicitly part of any parametric or nonparametric statistical analysis). Therefore, the distribution does not have to behave in any particular manner when a bootstrap technique is used.15 Statistical values are calculated directly from the resamplings of the empirically observed data. There are limitations to the bootstrap technique, however. It is a more complicated manipulation and may require sophisticated software to compute the resampling statistics. Although the data do not have to be normally distributed, the researcher must assume that the distribution of the sample approximates the distribution of the populations. Moreover, although the bootstrap can be used for inferential statistics, it is not as useful for point estimates of parameters.i6 In 1998, Desgagne et al” presented an example of the use of the bootstrap on a skewed pharmacoeconomic data set. The authors presented the results of different statistical tests conducted on the data set and provided a discussion of the bootstrap procedure. The authors recommend the use of the nonparametric bootstrap test when conducting pharmacoeconomic analyses on small to moderate samples of skewed data in which the difference in mean costs between treatments is being determined.” Most of the literature discussed earlier focuses on the collection of economic data parallel to a randomized clinical trial. The purpose of this article is to present an example of skewed cost data that were collected retrospectively from a large claims
484
database. Results of various statistical tests conducted on the data are reported and discussed.
METHODS The data used for these analyses were taken from a previous study that investigated the cost differences of adding inhaled corticosteroids to medication regimens for patients with persistent asthma in the Texas Medicaid system.i8 Asthmarelated medical and prescription claims in the Texas Medicaid database were used to form a retrospective, matched cohort. Twelve months of data were analyzed in which the cost difference between the 6 months before introduction of treatment and the 6 months after introduction of treatment was compared between a group of asthma patients introduced to inhaled steroids (intervention group) and a group of asthma patients who were treated with any asthma medications other than inhaled steroids (comparison group). Information generated from this study was analyzed using 4 methods: MannWhitney U nonparametric test, t test on untransformed data, t test on logarithmictransformed data, and t test on bootstrapped data. For each of the statistical methods conducted, 3 tests were compared between groups. One test compared differences in Medicaid prescription costs, 1 test compared differences in Medicaid medical utilization costs, and 1 test compared differences in the combined prescription and medical costs (overall costs). The perspective of the study was costs to the Texas Medicaid program. An a level of 0.05 was used to test for significance. Analyses were conducted using SPSS@ version 8.0 for Windows (SPSS, Inc, Chicago, Illinois) and SAS@ Language and Procedures ver-
K.L. RASCATI
ET AL.
Table I. Summary of Medicaid prescription, tion and comparison groups.
medical, and overall costs for the interven-
Mean Cost Difference (SD)
Intervention Group* Type of Cost Prescription Medical Overall
(n = 99) $342.29 ($336.52) -$3 17.84 ($1301.24) $24.45 ($1380.42)
Comparison Group+ (n = 99) $23.12 ($2 14.94) $81.93 ($2207.08) $105.04 ($2222.78)
Cost Difference Between Groups $319.17 -$399.77 -$80.59
*Six-month costs after initiation of inhaled corticosteroid therapy minus 6-month costs before initiation of therapy. %x-month costs after index date minus 6-month costs before index date.
sion 8.1 (SAS Institute, Inc, Cary, North Carolina) software. The data were examined for outliers via visual inspection by the researchers. In addition, histograms, normal Q-Q plots,‘9 and the KolmogorovSmirnov (K-S)19 test for normality were used to assess the distribution of data. To be included in the intervention group, patients had to have 4 corticosteroid inhalers filled within 6 months after beginning this therapy (index date). With this strict criterion, 99 patients were included in the intervention group. Thousands of Medicaid patients met the criteria for the comparison group (ie, continuous enrollment, diagnosis of asthma, and 24 asthmarelated prescription claims for each 6-month period). Therefore, one-to-one matching was conducted based on index date, age, sex, diagnosis, and preindex-date B-month prescription and medical costs.
RESULTS The summary statistics of the asthmarelated costs for the intervention and comparison groups are shown in Table I.
Normality and Variunce Tests: Untransformed Data Prescription
Costs
To determine whether the change in Medicaid costs before and after treatment introduction for the intervention and comparison groups were normally distributed, visual inspections of graphics and statistical tests for normality were conducted. Figure 1A shows the frequency distribution of the difference in untransformed Medicaid prescription costs before and after treatment introduction for the intervention group. Figure 2A shows the distribution of untransformed data for the comparison group. The distribution of prescription cost differences for the intervention group shows that these data are positively skewed (see Figure 1A). The K-S test shows that the distribution is nonnorma1 (K-S = 0.113, df = 99, P = 0.003). The distribution for the comparison group (see Figure 2A) shows high prescription cost differences in both tails, and the K-S test shows nonnormality (K-S = 0.101, df = 99, P = 0.015). The equality-of-
485
CLINICAL THERAPEUTICS@
variances test showed that variances between the groups were unequal (F = 2.450, df, = 98, df2 = 98, P < 0.001). Medical Costs
Figure 3A shows the untransformed data frequency distribution of the difference in Medicaid medical costs before and after treatment introduction for the intervention group. Figure 4A shows the untransformed data distribution for the comparison group. The distributions of untransformed data are similar in both groups in that a large number of cost differences cluster around the mode, whereas a few high cost-difference values lie in the tails (see Figures 3A and 4A). The K-S tests show that the untransformed distributions for the intervention group (K-S = 0.369, df = 99, P < 0.001) and comparison group (K-S = 0.353, df = 99, P < 0.001) are nonnormal. The equality-ofvariances test showed that the variances between groups were unequal (F = 2.880, df, = 98, df2 = 98, P < 0.001). Overall Costs
Figures 5A and 6A show the frequency distributions of the difference in untransformed Medicaid overall costs before and after treatment introduction for the intervention and comparison groups, respectively. The distributions of untransformed data are similar in both groups in that a large number of cost differences cluster around a certain range of values, whereas a few high cost-difference values lie in the tails. The K-S tests show that the untransformed distributions for the intervention group (K-S = 0.297, df = 99, P < 0.001) and comparison group (K-S = 0.299, df = 99, P < 0.001) are nonnormal. There was a significant difference in the variances between the groups (F = 2.590, df, = 98, df2 = 98, P < 0.001).
486
Normality and Variance Tests: Log Data Prescription
Costs
The prescription, medical, and overall data sets for both treatment groups also were transformed logarithmically. Figure 1B shows the frequency distribution of the log of the difference in Medicaid prescription costs before and after treatment introduction for the intervention group. Figure 2B shows the distribution of log data for the comparison group. The K-S test showed that only the log of the difference in prescription costs among the intervention group was normally distributed (K-S = 0.080, df = 99, P = 0.127). The shape of the distributions and the standard deviations between the 2 groups are quite different. The variances between groups were significantly different (F = 4.780, df, = 98, df, = 98, P = 0.001). Medical Costs
Figure 3B shows the log-data frequency distribution of the difference in Medicaid medical costs before and after treatment introduction for the intervention group. Figure 4B shows the log-data distribution for the comparison group. The K-S test shows that both distributions are nonnorma1 (intervention group: K-S = 0.432, df = 99, P < 0.001; comparison group: K-S = 0.326, df = 99, P < 0.001). The equalityof-variances test showed that the variances between groups were unequal (F = 16.860, df, = 98, df, = 98, P = 0.001). Overall Costs
Figures 5B and 6B show the frequency distributions of the log of the difference in Medicaid overall costs before and after treatment introduction for the intervention and comparison groups, respectively. The K-S test shows that the transformed
K.L. RASCATI ET AL.
AwanbaJ
a
j
AwanbaJ
j
487
1
0
10
40
Dollars ($)
214.94 Mean = 23.12 N = 99
B
20
60
SD = 0.78 Mean = 6.35 N = 99
Figure 2. Distribution of the difference in prescription costs before and after treatment (B), and bootstrap (C) data for the comparison group.
A
introduction
in untransformed
Dollars ($)
SD = 23.82 Mean = 22.36 N=150
(A), log
E@
5 8 $ 3 z
0
20
60
100
SD = 1301.24 Mean = -317.84 N=99
B SD = 0.92 Mean = 8.71 N=99
Figure 3. Distribution of the difference in medical costs before and after treatment and bootstrap (C) data for the intervention group.
A
introduction
C
in untransformed
Dollars ($)
SD = 126.46 Mean = -315.77 N=150
(A), log (B),
1
100
Dollars ($)
SD = 2207.08 Mean = 81.93 N = 99
B
100
SD = 0.22 Mean = 8.85 N = 99
Lo9 6)
Figure 4. Distribution of the difference in medical costs before and after treatment and bootstrap (C) data for the comparison group.
A
introduction
C
in untransformed
Dollars ($)
SD = 205.28 Mean = 101.38 N= 150
(A), log (B),
E0
2
2 E %
F
K.L. RASCATI
ET AL.
data sets of the intervention and comparison groups were nonnormally distributed (K-S = 0.423, df = 99, P < 0.001; K-S = 0.299, df = 99, P < 0.001, respectively). The variances between groups were significantly different (F = 18.230, df, = 98, df2 = 98, P = 0.001).
Distributions of Untransformed Bootstrap Data
Versus
The untransformed prescription, medical, and overall data sets for both treatment groups also were analyzed using the bootstrap technique. In this study, 150 bootstrap samples were selected in which each sample contained 99 random item draws with replacement. Therefore, the mean and standard deviations produced by the bootstrap methods in this study were calculated on a sample of 150 data points, whereas the descriptive statistics for the untransformed and log data sets were based on the sample of 99 empirical observations. Figures 1C and 2C show the frequency distributions of the difference in Medicaid prescription costs for the intervention and comparison groups after bootstrapping. Figure 3C shows the frequency distribution of the difference in Medicaid medical costs for the intervention group after bootstrapping, and Figure 4C shows the distribution of bootstrap medical data for the comparison group. Figures 5C and 6C show the frequency distributions of the difference in Medicaid overall costs for the intervention and comparison groups, respectively, after bootstrapping. The calculated mean difference in medical and overall Medicaid costs for the comparison group is slightly larger in the bootstrap distributions than in the corresponding untransformed distributions
(see Figure 4C vs Figure 4A, and Figure 6C vs Figure 6A). However, the mean values produced by the other bootstrap distributions are similar to those produced from the untransformed data. As expected, in all instances the variation in the bootstrap distributions is much lower than that in the corresponding untransformed distributions. Comparisons
of Statistical
Tests
Prescription Costs The results of the different statistical tests comparing differences in Medicaid prescription, medical, and overall costs between groups are shown in Tables II through IV. All of the statistical tests show that Medicaid prescription costs among patients with asthma treated with inhaled steroids were significantly higher than those for patients who were treated with therapies other than inhaled steroids (see Table II) Medical Costs When statistical differences were compared between groups with respect to medical costs, the Mann-Whitney U method was the only test to report a significant difference (Table III). The MannWhitney U test showed that patients with asthma treated with inhaled steroids had a lower median ranking of medical costs than those who did not receive an inhaled steroid. Overall Costs When statistical differences were compared between groups with respect to overall costs, the Mann-Whitney U test again was the only test to report a significant difference (Table IV). However, the direction of significance was oppo-
491
CLINICAL THERAPEUTICS”
m
492
K.L. RASCATI
ET AL.
m
a
kJuanbaJj
493
CLINICAL THERAPEUTICS”
Table II. Results of the different statistical tests comparing scription costs between groups. Method
differences
in Medicaid pre-
Test Statistic
t test (log data)* t test (bootstrap data)
Z = -7.720 t = 7.953 t=5.412 t = 7.960
IG = intervention group; CG = comparison *Test reported for unequal group variances.
Mean Mean Mean Mean
rank cost cost cost
of IG > CG of IG > CG of IG > CG of IG > CG
group.
Table III. Results of the different statistical tests comparing differences in Medicaid medical costs between groups. Method Mann-Whitney
U test
t test (untransformed data)* t test (log data)* t test (bootstrap data)
Test Statistic
P
Direction of Significance
Z = -2.004 t = -1.550 t = -1.379 t = -1.620
0.045 0.122 0.171 0.106
Mean rank of IG < CG NS NS NS
IG = intervention group; CG = comparison *Test reported for unequal group variances.
group.
site to the mean dollar value. The mean difference in overall Medicaid costs for the intervention group ($24.25) was lower than for the comparison group ($105.04), yet the Mann-Whitney U test indicated that the intervention group had a higher median ranking than the comparison group.
Sample Size Variation Various factors may play a role in the need for alternate statistical methods, including the degree of skewness and nonnormality, the overall sample size, and the comparability of the sample sizes between groups. The effects of 2 of these factors-
494
the equality of sample sizes and the total sample size-were addressed through a reanalysis of medical costs, the most highly skewed type of cost data in our original analyses. Table V shows how variations in the overall sample size and the equality of sample sizes affect the results. To conduct this reanalysis, we matched each original patient in the intervention group with 2 more comparison patients. Random samples of the intervention group were then chosen and matched to the corresponding comparison patients (in a 1: 1 or 1:3 ratio). It is noteworthy that even when the comparison group has 3 times as many observations as the intervention group, the standard deviation of the com-
K.L. RASCATI
ET AL.
Table IV. Results of the different statistical tests comparing all costs between groups. Method
t test (untransformed data)* t test (log data)* t test (bootstrap data)
differences
in Medicaid over-
Test Statistic Z = -3.216 t =-0.310 t = -0.894 t = -0.370
IG = intervention group; CG = comparison *Test reported for unequal group variances.
0.001 0.760 0.373 0.710
Mean rank of IG > CG NS NS NS
group.
parison group is still higher and the inequality of variances is still an issue. In comparing a sample of 50 intervention patients to a sample of 150 comparison patients, although the total sample size (N = 200) is similar to the original total sample (N = 198), the differences between the untransformed t test and the bootstrap t test results are important. Assuming the P value was set a priori at 0.05, a different conclusion would be reached using the bootstrap method.
DISCUSSION The nature of the frequency distributions discussed in this article are typical of those in pharmacoeconomic analyses. The Mann-Whitney U test found statistically significant cost differences between groups for each type of cost. However, in this study, as in the study by Desgagne and coworkers,” the Mann-Whitney U test produced an unexpected result. For example, the test result of the overall costs analysis would lead a researcher to conclude that the intervention group had significantly higher overall asthma-related costs than the comparison group. The direction of significance is contrary to the
mean cost differences listed in Table I. This is probably due to the fact that the Mann-Whitney U nonparametric test replaces actual observations with rank order, thereby reducing the magnitude of certain high cost values. Often, a treatment is specifically aimed at minimizing the chance of these few, but high-cost, events. All of the methods used to conduct t tests in the original study sample produced similar t values. However, all but one of the log-transformed data sets were nonnormally distributed. In addition, the variance tests showed that variances were significantly different between groups for all types of costs. Under such circumstances, the assumptions of log-transformed data are not met, and the t test may produce unreliable results. A t test also was conducted on the untransformed data. Because the initial test was conducted on samples of equal size (n = 99), the assumption of homogeneity of variances probably does not affect the test results. Furthermore, the t test allows the adjustment of unequal variances. However, the nonnormal distributions of the original data sets analyzed in this study violate the assumption of the test. Although the t test is robust to nonnormality, some-
495
$67.67 ($1952.46) $146.17 ($1623.42)
-$405.06 ($1353.63)
-$317.84
($1301.24)
50: 150
99:297 -2.88
-1.89
-1.63
-1.19
-1.55
t
0.06
0.11
0.24
0.12
P
._.
for CGt
Difference
($126.46)
-$315.77
($95.86)
$154.92
$62.03 ($153.27)
($202.75)
($205.26)
$101.38
($595.16)
$274.50
($205.26)
$101.38
(SD)
Mean
-$416.54
($285.87)
-$515.81
($285.87)
-$515.81
($126.46)
-$315.77
(SD) for IG”
Mean Difference
Bootstrap Data
IG = intervention group; CG = comparison group. *Six-month medical costs after initiation of inhaled corticosteroid therapy minus 6-month medical costs before initiation of therapy. %ix-month medical costs after index date minus 6-month medical costs before index date. *Original sample.
$81.93 ($2207.08)
$311.70 ($3528.46)
-$481.33
($1514.29)
$81.93 ($2207.08)
-$3 17.84
(SD) for CG’
Mean Difference
($1301.24)
(SD) for IG*
Mean Difference
-$481.33
Size (1G:CG)
Untransformed Data
on medical costs results.
($1514.29)
33:99
33:33
99:99*
Sample
Table V. Effect of sample size variations
-2.90
-1.94
-1.76
-1.21
-1.62
t
0.05
0.08
0.23
0.11
P
K.L. RASCATI ET AL.
times nonnormality can be severe. The researcher should use discretion in interpreting the results of a t test when economic data from small to moderate sample sizes are analyzed. Interestingly, the t tests reported in this study for the untransformed and bootstrapped data were similar for each type of cost (see Tables II through IV). A reanalysis of the medical costs using different sample sizes was conducted. As expected, analyses using smaller sample sizes were less likely to detect a statistical difference between the means. In 1 case, when sample sizes were unequal between groups, the bootstrap technique and the untransformed t tests produced different results. The bootstrap technique may be a more appealing method to analyze the data in this study because of its relative freedom from restricting assumptions and because not all research studies have equal group sizes. The bootstrap method can be generalized to most statistical tests and can even be used to test hypotheses for which there currently is not a normal-theory standard error. When pharmacoeconomic data grossly violate the assumption of normality, the bootstrap method is a way to ensure that the researcher is using a correct underlying distribution for hypothesistesting purposes. Although the unequalvariances t test is a viable option to control for assumption violations, the bootstrap is a more general solution for inequality of variances and nonnormality. Although the bootstrap technique is not without limitations (eg, complex programming and the assumption that the distribution of the sample approximates the true distribution), it can be used to check the validity of the results of untransformed t tests when working with small, skewed samples.
CONCLUSIONS When comparing cost differences between treatment options, pharmacoeconomics researchers often analyze data that are skewed and nonnormally distributed. The bootstrap method does not need to meet the assumptions of normality or equal variances, or the shape of the distributions. In addition, the bootstrap method considers the impact of high cost values, unlike nonparametric statistics, which remove their influence. When working with small sample sizes of skewed cost data, the bootstrap method may be an alternative to nonparametric tests or log transformations.
ACKNOWLEDGMENTS The authors would like to thank and acknowledge James P. Wilson, PhD, and Kenneth Lawson, PhD, University of Texas College of Pharmacy, for their guidance and collaboration on this study.
REFERENCES Wilcox RR, Keselman HJ, Kowalchuk RK. Can tests for treatment group equality be improved? The bootstrap and trimmed means conjecture. Br J Math Stat Psychol. 1998;51:123-134. Hays WL. Statistics. New York, NY Holt, Winston, Rinehart; 1988:303. Stevens J. Intermediate Statistics: A Modern Approach. Hillsdale, NJ: Lawrence
Erlbaum and Associates: 1986:42. Rutten-van MPMH, Molken van Doorslaer EKA, van Vliet RCJA. Statistical analysis of cost outcomes in a randomized controlled clinical trial. Health Econ. 1994;3:333-345.
497
CLINICAL THERAPEUTICS”
5. Zhou XH, Gao S, Hui SL. Methods for comparing the means of two independent log-normal samples. Biometrics. 1997;53: 1129-1135.
13. Efron B, Tibshirani R. An Introduction to the Bootstrap. New York, NY: Chapman & Hall; 1993. 14. Fiellin DA, Feinstein AR. Bootstraps and jackknives: New, computer-intensive statistical tools that require no mathematical theories. J Invest Med. 1998;46:22-26.
6. Coyle D. Statistical
analysis in pharmacoeconomic studies. Pharmacoeconomics. 1996;9:506-5 16.
15. O’Brien BJ, Drummond MF, Labelle RJ, Willan A. In search of power and significance: Issues in the design and analysis of stochastic cost-effectiveness studies in health care. Med Care. 1994;32:15&163.
7. Zhou XH, Melfi CA, Hui SL. Methods for comparison of cost data. Ann Intern Med. 1997; 127:752-756.
8. Obenchain
RL. Resampling and multiplicity in cost-effectiveness inference. J Biophamz Stat. 1999;9:563-582.
16. Mooney CZ, Duval RD. Bootstrapping: A Nonparametric Approach to Statistical Inference. Sage University Paper Series on Quantitative Applications in the Social Sciences, 07-095. Newbury Park, CA: Sage Publications; 1993.
9. Jones J, Wilson A, Parker H, et al. Economic evaluation of hospital at home versus hospital care: Cost minimisation analysis of data from randomised controlled trial. Br Med J. 1999;3 19: 1547-1550.
17. Desgagne A, Castilloux AM, Angers JF, LeLorier J. The use of the bootstrap statistical method for the pharmacoeconomic cost analysis of skewed data. Pharmacoeconomics. 1998; 13:487-497.
10. Mehta SS, Suzuki S, Glick HA, Schulman KA. Determining an episode of care using claims data. Diabetic foot ulcer. Diabetes Care. 1999;22:1110-1115.
18. Smith MJ, Rascati KL, Johnsrud MT. Use of inhaled corticosteroids to treat Texas Medicaid clients with asthma: Assessment of prescriptions and medical service payments. J Am Pharm Assoc. 2000;40:307. Abstract.
11. Briggs AH, Wonderling DE, Mooney CZ. Pulling cost-effectiveness analysis up by its bootstraps: A non-parametric approach to confidence interval estimation. Health Econ. 1997;6:327-340.
19. SPSS Inc. SPSS@ 7.5 Guide to Data Analysis, 1997. Chicago, III: Prentice-Hall; 1997: 223-225.
12 Efron B. Bootstrap methods: Another look at the jackknife. Ann Stat. 1979;7: l-26.
Address correspondence to: Karen L. Rascati, of Pharmacy Practice [email protected]
498
and Administration,
PhD, The University Mail Code: A1930, Austin,
of Texas, Division TX 78712. E-mail: