- Email: [email protected]
Case–non-case studies: Principle, methods, bias and interpretation
Therapie (2019) 74, 225—232
Available online at
ScienceDirect www.sciencedirect.com
PHARMACOEPIDEMIOLOGY
Case—non-case studies: Principle, methods, bias and interpretation夽 Jean-Luc Faillie a,b,∗ a
Regional Pharmacovigilance Centre, Department of Medical Pharmacology and Toxicology, CHU de Montpellier, 34295 Montpellier, France b Laboratory of Biostatistics, Epidemiology and Public Health, EA2415, Faculty of Medicine, University of Montpellier, University Institute of Clinical Research, 34295 Montpellier, France Received 15 January 2019; accepted 22 January 2019 Available online 31 January 2019
KEYWORDS Case—non-case studies; Disproportionality analysis; Reporting odds ratio; Pharmacovigilance; Pharmacoepidemiology; Bias
Summary Case—non-case studies are among the methods used to assess drug safety by analyzing the disproportionality of adverse drug reaction reports in pharmacovigilance databases. First used in the 1980s, the last few decades have seen a significant increase in the use of this study design. The principle of case—non-case studies is to compare the drug exposure of cases of an adverse reaction of interest with that of cases with other reported reactions and called ‘‘non-cases’’. Their results are presented in the form of reporting odds ratio (ROR), the interpretation of which makes it possible to highlight pharmacovigilance signals. This article describes the principle of case—non-case studies, the calculation method of the ROR and its confidence interval, the different analytical modalities and how to interpret its results with regard to the advantages and limitations of this type of study. © 2019 Soci´ et´ e franc ¸aise de pharmacologie et de th´ erapeutique. Published by Elsevier Masson SAS. All rights reserved.
Abbreviations
夽
This manuscript is an updated translation of an article published in French by the same author in the journal Thérapie in 2018 (Therapie 2018;73:247—55). ∗ Corresponding author. Département de pharmacologie médicale et toxicologie, CHU de Montpellier, 371, avenue du Doyen-GastonGiraud, 34295 Montpellier, France. E-mail address: [email protected]
AERS adverse event reporting system ATC anatomical therapeutic chemical BCPNN Bayesian confidence propagation neural network EMA European Medicines Agency FDA: Food and Drug AdministrationHERG human ether-a-gogo-related gene HRT hormone replacement therapy MedDRA medical dictionary for regulatory activities
https://doi.org/10.1016/j.therap.2019.01.006 0040-5957/© 2019 Soci´ et´ e franc ¸aise de pharmacologie et de th´ erapeutique. Published by Elsevier Masson SAS. All rights reserved.
226 MGPS PRR PTs ROR SMQ WHO
J.-L. Faillie mutli-item gamma Poisson Shrinker proportional reporting ratio preferred terms reporting odds ratio standardized MedDRA queries World Health Organization
Introduction Case—non-case studies are among the disproportionality studies that specifically concern the analysis of pharmacovigilance databases. These databases are composed of adverse reaction cases spontaneously reported to pharmacovigilance systems. Different databases exists: at the national level (such as the French national pharmacovigilance database or the adverse event reporting system [AERS] in the United States) or at the international level (such as the European Medicines Agency [EMA] database, EudraVigilance or the World Health Organization [WHO] database, VigiBase). The objective of studying the disproportionality of spontaneous reports is to generate pharmacovigilance signals concerning unknown or underestimated adverse drug reactions, as early as possible after marketing. In this context, a signal refers to a higher than expected number of adverse reaction reports with a specific drug, generating a ‘‘disproportionate’’ rate of reporting compared to other reactions recorded in the pharmacovigilance database. Disproportionality studies are used either for automatic signal detection in a database or for investigating a hypothesis based on pharmacological analysis or clinical signs of drug risk. Disproportionality studies have been increasingly used since the 1990s. Indeed, the development and availability of pharmacovigilance databases have made it possible, especially since it does not require significant logistical resources. At the same time, the frequent health scandals concerning the safety of medicines and the lack of any real improvement in risk assessment before medicines are marketed highlight the need for early detection of signals once the medicine is on the market. The different methods for studying the disproportionality of adverse drug reaction reports include frequentist methods such as the case—non-case study or the proportional reporting ratio (PRR) study and Bayesian methods such as the multi-item gamma Poisson Shrinker (MGPS) method [1] or the Bayesian confidence propagation neural network (BCPNN) method [2].
Principle of the case—non-case In order to highlight a link between the reporting of an adverse reaction of interest and a drug of interest, the case—non-case study is based on the analysis of a 2 × 2 contingency table describing the number of reports of the adverse reaction of interest in the database and the mention in the report of the exposure to the drug of interest (Table 1). Reports of the adverse reaction of interest are called ‘‘cases’’ and other reports are called ‘‘non-cases’’. Despite its similarities to the case-control method, the term ‘‘noncases’’ is used instead of ‘‘controls’’ because this group
Table 1 Contingency table for the case—non-case analysis.
Drug of interest Other drugs
ADR of interest ‘‘Cases’’
Other ADRs ‘‘Non-cases’’
a c
b d
ADR: adverse drug reaction.
refers to patients who have all been exposed to at least one drug and have all experienced at least one reaction other than the adverse reaction of interest. Typically, the number of reports in each of the cells of the contingency table is represented with the letters a, b, c and d, corresponding to cases exposed to the drug of interest (a), non-cases exposed to the drug of interest (b), cases exposed to other drugs (c) and non-cases exposed to other drugs (d). In the absence of a signal, the distribution of cases and non-cases should be independent of the exposure to the drug of interest, i.e. similar between reports exposed to the drug of interest and those exposed to other drugs. A signal is identified when a difference in this distribution (a ‘‘disproportionality’’) is statistically demonstrated. The case—non-case method was used for the first time in 1982 in a study exploring the association between the use of valproic acid during pregnancy and occurrence of spina bifida, based on a register of fetal malformations in a region around Lyon in France [3]. Among the 146 mothers of children with spina bifida, 9 were exposed to valproic acid in the first trimester of pregnancy and among the 6616 mothers of children with other malformations, 21 were exposed, resulting in an odds ratio of 20.6 (P < 0.001). The authors used a case-control design in which the controls were cases of fetal malformations other than spina bifida, i.e. ‘‘noncases’’. This study was thus retrospectively considered as the first case—non-case study [4]. The demonstration of disproportionality is based on the calculation of a reporting odds ratio (ROR) [5,6] The term and calculation of the ROR was first used by Stricker and Tijssen in 1992, when they studied the link between serum sickness and the use of cefaclor (compared to other antibiotics) in an international pharmacovigilance database [7]. Based on the calculation of the odds ratio typically used in case-control study, the calculation of the ROR is quite simple. For an exposure group, the reporting odd is equal to the reporting frequency (f) divided by (1−f). Thus, for the group exposed to the drug of interest, the reporting frequency corresponds to a/(a + b) and the reporting odd is equal to [a/(a + b)]/[1 − a/(a + b)] which simplifies to a/b. Similarly, for the group exposed to other drugs, the reporting odd is c/d. The ROR can be calculated if there is no cell in the contingency table with a null value [6]. It corresponds to the ratio of the reporting odds between groups exposed and not exposed to the drug of interest, and is therefore [a/b]/[c/d], or ad/bc [8]. The ROR measures the strength of disproportionality; if the ROR is 1, there is no signal, meaning that the adverse reaction of interest is similarly reported with the drug of interest as with other drugs (if the ROR is less than 1, there is no signal either, the adverse
Case—non-case studies reaction being less reported with the drug of interest than with other drugs). On the contrary, if the ROR is greater than 1, cases are more reported with the drug of interest than with other drugs: there is a signal and the higher the value of the ROR, the greater the disproportionality. The ROR being a statistical estimate, it should always be presented and interpreted with its 95% confidence interval (95% CI). The calculation of the 95% CI of the ROR involves the variance of the ROR natural logarithm, which is represented by the following simplified formula: 1/a + 1/b + 1/c + 1/d. Thus, the 95% CI of the ROR can be calculated as follows:
ROR ∗ e
±1.96
1 1 1 1 a+b+c+d
If the lower bound of the 95% CI is greater than 1, then the disproportionality signal is statistically significant and the ROR is generally interpreted in these terms: the adverse reaction of interest is ROR times more reported with the drug of interest than with other drugs. Conversely, if the lower bound of the 95% CI is lower than 1, there is no signal. In the study presented by Stricker and Tijssen, the ROR for serum sickness reports with cefaclor compared to amoxicillin was 12.4 (95% CI: 8.2—18.7), showing a largely statistically significant signal: serum sickness reports were more than 12 times more reported with cefaclor than with amoxicillin [7].
Comparison with other methods for studying disproportionality Among other disproportionality measures, the proportional reporting ratio (PRR) is also calculated from the contingency table and is [a/(a + b)]/[c/(c + d)]. A signal is generally considered to be present when PPR is greater than 2, Chisquared statistic value is greater than 4 and number of cases is greater than 3 [5]. The PPR value is generally similar to the ROR. In Bayesian methods, for instance the Bayesian confidence propagation neural network (BCPNN) method, disproportionality is expressed with the information component (IC), which is interpreted in reference to zero. Bayesian methods are considered more robust than frequentist methods when the number of exposed cases is low. However, in a study comparing different disproportionality measures on 10 drug/adverse reaction combinations in the Food and Drug Administration (FDA) reporting database, Chen et al. concluded that the ROR would perform better than other techniques in terms of signal precocity [9]. Another advantage of the ROR is that it allows the use of multivariate logistic regression and thus the consideration of confounding and interaction effects [6].
The different approaches to the disproportionality analysis The study of disproportionality can be carried out according to various methodological approaches. First, various choices can be made regarding the temporality of notifications. Depending on the risk investigated, the analysis could only
227 concern a specific period (for example, the total marketing period of the drug of interest, or only a period preceding a safety alert). The RORs can also be calculated and presented as a function of time (cumulative ROR or per unit of time) to explore time-related biases (see section on temporal biases). Second, the analysis could focus on subgroups or selected reports such as serious cases, cases reported by health professionals, cases for which the imputability of the drug of interest is highest (drug considered ‘‘suspect’’), adverse reactions corresponding to a specific medical domain (e.g. dermatology, psychiatry, etc.), or adverse reactions concerning drugs of a specific therapeutic class (e.g. antidiabetics, antibiotics, etc.). Implementing exclusion criteria to the selection of reports could also allow to explore competition biases (see section on competition biases). The disproportionality analysis made for a specific drug/adverse reaction combination could also be supplemented by the analysis of other adverse reactions related to the reaction of interest or by the analysis of drugs belonging to the same pharmacological or therapeutic class as the drug of interest. All these modalities can be the subject of secondary or sensitivity analyses exploring the robustness of the main results.
Strengths One of the main advantages of disproportionality studies is that that they allow the study of rare adverse reactions, given that only a few reports of exposed cases are needed to carry out the analysis. Also, compared to clinical trials, disproportionality studies theoretically concern an exhaustive population of real users (including specific subgroups: the elderly, children, patients with comorbidities, etc.) and therefore are representative of drug use in real population and real conditions. In addition, the interest of disproportionality studies in drug risk monitoring is their ability to provide early (and possibly automated) detection of signals that will justify the conduct of confirmatory studies. On many occasions, disproportionality estimates have been shown to be well correlated with the strength of the risk association later confirmed by controlled studies [10]. For example, a case—non-case study of bladder cancer reports associated with pioglitazone in the FDA adverse events database between 2004 and 2009 revealed that a signal of disproportionality was present as early as 2004 (ROR 4.8, 95% CI: 1.3—15.9) [11], i.e. 7 years before the confirmation of the risk and implementation of regulatory measures taken [12]. However, confirmatory studies are not always feasible and, when they are, do not always lead to firm conclusions. For example, the signal of acute pancreatitis associated with incretin-based drugs suggested by several case—non-cases studies in various pharmacovigilance databases [13—15] has not yet been confirmed by pharmacoepidemiological studies, leaving the risk as a signal. Finally, the logistics of case—non-case studies are relatively simple, they are not expensive, and, thanks to development of international standards, pharmacovigilance databases are available to study adverse drug reactions at the international level. The case—non-case method also allows pharmacodynamic data to be used to explore mechanistic hypotheses of iatrogeny. For example, a study investigating the
228 pharmacological inhibition of the human ether-a-go-gorelated gene (HERG) potassium channels used a case—noncase method to show that the anti-HERG activity of drugs was directly associated with more frequent reports of severe ventricular arrhythmias and sudden death in the WHO database. The authors thus recommended the measurement of anti-HERG activity in preclinical drug tests to predict their pro-arrhythmic effect [16]. Another example is a case—non-case study combining pharmacodynamic data on inhibition of various monoamine receptors by antipsychotics with adverse reaction reports related to antipsychotics. As a result, the inhibition of serotonin 5-HT2c and histamine H1 receptors was shown to be associated with increased reporting of diabetes mellitus (ROR 2.1; 95% CI: 1.7—2.6 and 95% CI: 1.4—2.6, respectively) [17]. Disproportionality analysis can also be applied to other data sources such as epidemiologic surveys. For example, in the field of addictovigilance, the abuse potential of psychoactive drugs has been studied by comparing the disproportionality estimates of criteria such as abuse and dependence, illegal acquisition, diverted route of administration and concomitant alcohol consumption based on data from annual cross-sectional surveys in drug addicts [18].
Limitations and biases Under-reporting of adverse drug reactions is an inherent limitation to the interpretation of studies conducted on pharmacovigilance databases. Pharmacovigilance systems cannot aim to collect the exhaustivity of adverse reactions but focus on their relevance for alerting about a risk. It has been estimated in France that no more than 5% of serious adverse drug reactions are actually reported [19]. Underreporting hinders the measurement of the actual incidence of adverse drug reaction and limits the sensitivity of signal detection by disproportionality methods. In addition, under-reporting is variable (referred to as ‘‘selective reporting’’), being influenced by many parameters depending on the reporter, the drugs, the reaction itself or time. Thus, a disproportionality signal may be the result of either an actual increase in the incidence of the adverse reaction or an artifactual increase in reports not correlated to an increase in incidence thus impacting signal detection specificity. Factors of variation in reporting include experience and type of reporter, local usage and reporting patterns, severity of the adverse reaction (serious adverse reactions are usually more reported), specificity of the adverse reaction to the drug, unexpectedness in relation to the known safety profile of the drug (unexpected adverse reactions are more commonly reported) [20], media coverage of an adverse reaction (notoriety effect) or time since market authorization (e.g. Weber effect). These factors are part of the main biases limiting the interpretation of disproportionality studies. A summary of biases and limitations of the case—non-case studies are presented in Table 2, with their consequences and the methods for measuring or avoiding them.
Temporal biases The Weber effect is defined as a variation in reporting over time that results in an increase in the number of reports
J.-L. Faillie immediately after a drug is marketed (due to the incomplete safety profile and increasing exposure). Although, it is not necessarily observed for all types of drugs, it mainly occurs in the first two years, with a high proportion of non-serious reactions followed by a decline (decreased enthusiasm for reporting adverse reactions becoming better known) [21,22]. The Weber effect can also be observed for an older drug with a new indication or a change in dosage. The impact of the Weber effect on the ROR estimate is variable and depends on its intensity on the adverse reaction of interest (increase in number [a]) compared to other adverse reaction (number [b]). In any case, disproportionality analyses are less effective in detecting long-term effects because they are remote from marketing. For the study of serious reactions, the Weber effect including a high proportion of non-serious reactions, it may constitute a masking effect showing a larger increase in the number of nonserious non-cases (number [b]) potentially responsible for an underestimation of the ROR. This point therefore justifies restricting the case—non-case analysis to serious adverse reactions only. The notoriety effect results in increased reports following media coverage of a specific adverse drug reaction (publication of a scientific study, safety alerts from regulatory agencies addressed to health professionals or to the public, articles or discussions in public media or social networks) [23]. An increase in the number of reports is observed for incident cases but also for retrospective cases (which occurred before the alert). In all cases, this effect leads to a bias that is responsible for an increase in the ROR for the drug in question. To avoid it, it is necessary, if possible, to ensure that the signal is present before the media event. Pariente et al. provided several illustrations of this bias. For example, in the French national pharmacovigilance database, the ROR for stroke that occurred with atypical antipsychotics were 0.1 (95% CI: 0.02—1.0) and 2.0 (95% CI: 1.2—3.4) respectively before and after the publication of a press release from the French medicine agency mentioning an increased risk in a clinical trial in an elderly population with dementia [23]. An alert concerning a drug adverse reaction may also lead to a ‘‘ripple effect’’ resulting in an increase in reports of the adverse reaction of interest with other drugs belonging to the same pharmacological or therapeutic class as the drug initially suspected in the alert [23]. For example, in the FDA pharmacovigilance database, an increase of nearly 40% in reports of pancreatitis was observed with sitagliptin after an FDA alert regarding the risk of pancreatitis with exenatide [14].
Information biases Missing data are frequent in pharmacovigilance databases, however, they generally do not concern the identification of the medicinal product or the diagnosis of the effect and therefore do not prevent the disproportionality analysis itself. Nevertheless, missing information regarding clinical data, indication, age, sex, comedications, evolution, type of report and reporter, country, specific doses or treatment dates limit complementary analyses and confounding assessment. Among all the reports available in VigiBase, sex and age were missing in 6.0% and 25.6%, respectively [24]. A report completeness score, called VigiGrade, has been
Case—non-case studies Table 2
229
Main biases and limitations of case—non-case studies, consequences and possible solutions.
Bias/limitation
Consequences
Under-reporting
Lack of measurement of actual incidence and risk Lack of statistical power Decrease in signal detection sensitivity Overestimated or underestimated ROR
Selective reporting
Weber effect
Notoriety bias and ripple effect Missing data
Duplicates Information bias on clinical and exposure data
Variability of the ROR over time Overestimated or underestimated ROR (if masking effect by numerous non-serious non-cases) Overestimated ROR
Limitation for additional analyses and management of confounders Incorrect ROR Incorrect ROR
Selection bias/confounding
Incorrect ROR
Drug competition bias Event competition bias All biases
Underestimated ROR Underestimated ROR Incorrect ROR
Solutions
Restriction or adjustment on factors influencing the reporting (severity, type of reporter, region. . .) Use of positive and negative controls to explore the reporting of the studied effect ROR temporal analysis Restriction of the analysis to serious cases only (if masking effect)
ROR temporal analysis Restriction of the analysis to time period preceding media coverage Comparison of the analysis restricted to non-missing data with that using a ‘‘missing data’’ category Exploration of imputation possibilities Manual or algorithmic ‘‘deduplication’’ Restriction to medically confirmed reports Appropriate definitions using adverse reaction and drug classifications Restriction to cases where the drug is considered ‘‘suspect’’ Reference group consisting of drugs of the same pharmacological or therapeutic class than the studied drug Subgrouping/stratification/adjustment for confounders (factors influencing report and risk factors) Exclusion of reports concerning drugs frequently associated with the adverse reaction of interest Exclusion of reports of adverse reactions known to be frequently associated with the drug of interest Exploring biases by performing sensitivity analyses Search for consistent results for positive and negative controls
ROR: reporting odds ratio.
developed by the Uppsala center managing VigiBase. Its calculation is carried out by assigning penalties according to the availability of information and its clinical relevance [25]. A report is considered well documented when the VigiGrade score is higher than 0.8. In 2014, only 13% of the reports present in VigiBase met this criterion and this rate showed great variability according to the country (e.g. it is generally better in Europe than in the United States), the type of notification (24% for doctors vs. 4% for non-health professionals) [25] but also according to the clinical context (for example, there are more missing data for serious cases in diabetic patient), or the origin of the report (missing data are more frequent in cases reported by the industry) [26]. In 2016, France was among the 10 most contributing countries to VigiBase [27] with a median completeness score of 0.80 [28]. There is no ideal solution to take missing data into account. It may be useful to explore this bias by comparing analyses restricted to reports with all the data provided and
those with a ‘‘missing data’’ category. Imputation methods could also be experimented. Multiple reports of the same adverse reaction are a significant source of error, therefore, ‘‘deduplication’’ is the first step in any analysis using pharmacovigilance databases. In VigiBase, an algorithm called VigiMatch is used to identify and remove duplicates while keeping the most informative reports [29]. Similarly, due to the anonymization of reports and the lack of a unique patient identifier, it is often difficult to perform an analysis with the patient as the statistical unit: in this case, multiple events for the same patient (e.g. a recurrent effect) would count as several independent observations. Errors in reporting diagnosis, exposure, or imputability are often dependent on the type of reporter. In order to limit this information bias, it is possible to select observations by restricting the analysis to observations for which the risk of error is lower such as adverse reactions reported by health
230 professionals or reports in which the drug is considered ‘‘suspect’’ or has a suggestive chronology. To avoid exposure errors, particular attention should be paid to the definition of exposure using drug classifications. For example, failure to include the anatomical therapeutic chemical (ATC) codes for the combined forms of the study drug may result in information bias. Regarding the definition of the adverse reaction, it is crucial to assess the clinical relevance of the list of selected terms from adverse reaction classifications such as MedDRA (medical dictionary for regulatory activities). Although it is recommended to use MedDRA preferred terms (PTs) to conduct signal detection [10], standardised MedDRA queries (SMQ) may also be helpful by providing selections of adverse reaction terms with narrow (more specific) or broad (more sensitive) definitions. Also, the search for other clinical data (indication, medical history) from proxies based on drug exposures can also be a source of bias. Temporal analyses may also be limited by the time lag between the occurrence of the reaction and the recording of reports in pharmacovigilance databases, particularly multinational databases. For example, in VigiBase, reports are recorded monthly for some countries, but quarterly or twice a year for other countries.
J.-L. Faillie In disproportionality analyses, confounders include variables involved in the association between exposure and adverse reaction reporting (potentially all the factors responsible for selective reporting presented above) but also include the variables involved in the association between exposure and the effect itself, such as the indication of the drug. For example, a signal concerning the risk of hypoglycemia associated with angiotensin-converting enzyme (ACE) inhibitors was identified in the French national pharmacovigilance database (RORs ranging from 3 to 4). In fact, diabetes was a confounder: ACE inhibitors are more favorably prescribed to diabetic patients who are also treated with antihyperglycemic agents, this increases the risk of episodes of hypoglycemia and therefore their reporting. This is referred to as indication bias or co-prescription bias. When stratifying the analysis on the presence or absence of antihyperglycemic agents, the signal disappeared (ROR equal to 1 in each stratum) [32]. Solutions for confounding generally include subgrouping, stratification or adjustment. It is suggested that subgroup analyses (in which a signal is identified if it is present in any of the subgroup strata) tend to perform better than stratified/adjusted analyses [10].
Competition biases Selection bias/confounding In disproportionality analyses, as in pharmacoepidemiology studies in general, comparisons may be biased by the fact that patients exposed to the drug of interest may be more (or less) at risk for the adverse reaction than those exposed to other drugs: this can refer to indication bias, severity bias, channeling bias or healthy user effect. In the exposed group, an increase risk due to other factors would result in an ‘‘over-reporting’’ and therefore a false signal. To limit these effects, the choice of the reference group is crucial. Choosing all ‘‘other drugs’’ as the reference group can be problematic in the presence of indication bias: the signal would reflect the risk associated with the indication for the drug and not the exposure to drug itself. For example, for the study of pancreatitis reports associated with exenatide (indicated for the treatment of type 2 diabetes), if the reference group includes all other drugs and not only diabetes drugs, the observed signal may be explained by the effect of the indication (diabetic patient being at higher risk of pancreatitis). The reference group should therefore be defined by the exposure to other drugs belonging to the same therapeutic group (in this example, diabetes drugs). A study conducted in the European database EudraVigilance compared the results of disproportionality analyses with and without restricting the reference group to the therapeutic class of the studied drugs (drugs for prostate disorders and drugs for type 2 diabetes). The authors demonstrated that, by using this type of restriction, the performance of the disproportionality estimator was improved with a decrease in false positives (confounded by the disease) and a better ability to detect true positives [30]. Also, in a pilot study evaluating a signal prioritization score from the analysis of pharmacovigilance databases, Salvo et al. recommended to take into account pharmacological criteria such as the ROR calculated from the therapeutic class of the drug to increase signal specificity [31].
Competition biases tend to mask signal detection. A bias due to competition between drugs may occur when the adverse reaction studied is also frequently reported with one or more drugs other than the drug of interest [33]. Thus, the rate of unexposed cases (number [c] of the contingency table) is high, leading to an underestimation of the ROR, i.e. a signal that goes unnoticed. This bias increases false negatives and therefore reduces the sensitivity of signal detection. A solution to this bias is to remove from the analysis the reports concerning drugs frequently associated with the adverse reaction of interest: this reduces the number of exposed cases needed to show a signal. In the example of strokes with atypical antipsychotics, there is a competitive bias between antipsychotics and anticoagulants, the latter being known to increase the risk of stroke and thus their report. The exclusion of anticoagulant reports revealed a previously undetected signal for stroke with atypical antipsychotics: RORs equal to 1.1 (95% CI: 0.7—1.9) and 2.0 (95% CI: 1.2—3.4) respectively before and after exclusion [33]. A bias due to competition between events can occur when the drug of interest is associated with a significant number of reports for adverse reactions other than the adverse reaction of interest [34]. In this case, the rate of exposed non-cases (number [b] in the contingency table) is high, also leading to an underestimation of the ROR and a decreased sensitivity. The solution is similar to the previous situation: removing the adverse reaction reports known to be frequently associated with the drug of interest. For example, in a statin disproportionality study, the withdrawal of muscle disease reports, a known adverse reaction to statins, reveals previously undetected signals such as libido disorder, alopecia, cholestasis, dermatitis, or digestive bleeding [34]. This bias also justifies conducting the disproportionality analysis restricted to reports belonging to the same system-organ-class than the adverse reaction of interest (e.g. dermatological, psychiatric, hematologic, etc.).
Case—non-case studies
231
Other potential biases
Funding
Other potential biases often studied in pharmacoepidemiology may also limit the interpretation of disproportionality studies. One example is the protopathic bias that occurs when a drug is initiated in response to a symptom or an early stage of the suspected effect. For example, the use of analgesics in response to pain caused by an undiagnosed cancer, may lead, when the cancer is diagnosed, to reports suggesting that the analgesic caused the cancer while the causality is reversed. In this situation, the true chronology must be restored by introducing a latency period following the exposure and during which the cases that have been diagnosed are considered unexposed. Surveillance bias or detection bias is another example to keep in mind. It occurs when the detection of the event is more likely in an exposure group due to an increased surveillance, disease screening or an associated symptom. For example, reports suggesting a risk of endometrial cancer with hormone replacement therapy (HRT) could theoretically be the consequence of better detection of these cancers in women treated with HRT. Indeed, HRT are known to cause uterine bleeding, which is a symptom that motivates screening for endometrial cancer.
This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors.
The use of positive and negative controls to assess the presence of bias The presence of an undetected bias can be investigated by studying ‘‘control’’ drugs whose association or absence of association with the studied effect is expected from the literature. This involves measuring the disproportionality of (i) drugs known to cause the adverse reaction (positive controls) and (ii) drugs that are not related to the adverse reaction (negative controls) and verifying that there is a signal for positive controls and no signal for negative controls. The consistency between expected and obtained results for controls would suggest that the study is not compromised by major biases. For example, the disproportionality study of the association between antihypertensive drugs and hypoglycemia by Gregoire et al. used cibenzoline and disopyramide as positive controls and diazepam as negative controls. As expected, a signal suggesting a higher risk of hypoglycemia was found for cibenzoline (ROR 107; 95% CI: 78—148) and disopyramide (ROR 17; 95% CI: 10—29) while no signal was found for diazepam (ROR 0.3; 95% CI: 0.1—2) [32].
Conclusions Case—non-case studies, like other disproportionality studies, analyze real-life pharmacovigilance data and, despite significant limitations and many biases, allow for the early detection of pharmacovigilance signals. They are sometimes the only tools available, particularly for the study of rare events or uncommonly used drugs. Signals from case—noncase studies, coupled with pharmacological and clinical analysis, remain crucial elements of the body of evidence needed for the management of drug safety [35].
Disclosure of interest The author declares that he has no competing interest.
References [1] DuMouchel W. Bayesian data mining in large frequency tables, with an application to the FDA spontaneous reporting system. Am Stat 1999;53:177—90. [2] Bate A, Lindquist M, Edwards IR, Olsson S, Orre R, Lansner A, et al. A Bayesian neural network method for adverse drug reaction signal generation. Eur J Clin Pharmacol 1998;54:315—21. [3] Bjerkedal T, Czeizel A, Goujard J, Kallen B, Mastroiacova P, Nevin N, et al. Valproic acid and spina bifida. Lancet 1982;2(8307):1096. [4] Montastruc JL, Sommet A, Bagheri H, Lapeyre-Mestre M. Benefits and strengths of the disproportionality analysis for identification of adverse drug reactions in a pharmacovigilance database. Br J Clin Pharmacol 2011;72:905—8. [5] Moore N, Thiessard F, Begaud B. The history of disproportionality measures (reporting odds ratio, proportional reporting rates) in spontaneous reporting of adverse drug reactions. Pharmacoepidemiol Drug Saf 2005;14:285—6. [6] van Puijenbroek EP, Bate A, Leufkens HGM, Lindquist M, Orre R, Egberts ACG. A comparison of measures of disproportionality for signal detection in spontaneous reporting systems for adverse drug reactions. Pharmacoepidemiol Drug Saf 2002;11:3—10. [7] Stricker BH, Tijssen JG. Serum sickness-like reactions to cefaclor. J Clin Epidemiol 1992;45:1177—84. [8] Moore N, Kreft-Jais C, Haramburu F, Noblet C, Andrejak M, Ollagnier M, et al. Reports of hypoglycaemia associated with the use of ACE inhibitors and other drugs: a case/non-case study in the French pharmacovigilance system database. Br J Clin Pharmacol 1997;44:513—8. [9] Chen Y, Guo JJ, Steinbuch M, Lin X, Buncher CR, Patel NC. Comparison of sensitivity and timing of early signal detection of four frequently used signal detection methods: an empirical study based on the US FDA adverse event reporting system database. Pharm Med 2008;22:359—65. [10] Wisniewski AFZ, Bate A, Bousquet C, Brueckner A, Candore G, Juhlin K, et al. Good signal detection practices: evidence from IMI PROTECT. Drug Saf 2016;39:469—90, http://dx.doi.org/10.1007/s40264-016-0405-1. [11] Piccinni C, Motola D, Marchesini G, Poluzzi E. Assessing the association of pioglitazone use and bladder cancer through drug adverse event reporting. Diabetes Care 2011;34:1369—71. [12] Faillie JL, Petit P, Montastruc JL, Hillaire-Buys D. Scientific evidence and controversies about pioglitazone and bladder cancer: which lessons can be drawn? Drug Saf 2013;36:693—707. [13] Elashoff M, Matveyenko AV, Gier B, Elashoff R, Butler PC. Pancreatitis, pancreatic, and thyroid cancer with glucagon-like peptide-1-based therapies. Gastroenterology 2011;141:150—6. [14] Raschi E, Piccinni C, Poluzzi E, Marchesini G, De Ponti F. The association of pancreatitis with antidiabetic drug use: gaining insight through the FDA pharmacovigilance database. Acta Diabetol 2013;50:569—77. [15] Faillie JL, Babai S, Crepin S, Bres V, Laroche ML, Le Louet H, et al. Pancreatitis associated with the use of GLP-1 analogs
232
[16]
[17]
[18]
[19]
[20]
[21]
[22]
[23]
[24]
[25]
J.-L. Faillie and DPP-4 inhibitors: a case/non-case study from the French Pharmacovigilance Database. Acta Diabetol 2014;51:491—7. De Bruin ML, Pettersson M, Meyboom RHB, Hoes AW, Leufkens HGM. Anti-HERG activity and the risk of drug-induced arrhythmias and sudden death. Eur Heart J 2005;26:590—7. Montastruc F, Palmaro A, Bagheri H, Schmitt L, Montastruc JL, Lapeyre-Mestre M. Role of serotonin 5-HT2C and histamine H1 receptors in antipsychotic-induced a pharmacoepidemiological-pharmacodynamic diabetes: study in VigiBase. Eur Neuropsychopharmacol 2015;25: 1556—65. Pauly V, Lapeyre-Mestre M, Braunstein D, Rueter M, Thirion X, Jouanjus E, et al. Detection of signals of abuse and dependence applying disproportionality analysis. Eur J Clin Pharmacol 2015;71:229—36. Bégaud B, Martin K, Haramburu F, Moore N. Rates of spontaneous reporting of adverse drug reactions in France. JAMA 2002;288:1588. Martin RM, Kapoor KV, Wilton LV, Mann RD. Underreporting of suspected adverse drug reactions to newly marketed (‘‘black triangle’’) drugs in general practice: observational study. BMJ 1998;317:119—20. Wallenstein EJ, Fife D. Temporal patterns of NSAID spontaneous adverse event reports: the Weber effect revisited. Drug Saf 2001;24:233—7. Hartnell NR, Wilson JP. Replication of the Weber effect using postmarketing adverse event reports voluntarily submitted to the United States Food and Drug Administration. Pharmacotherapy 2004;24:743—9. Pariente A, Gregoire F, Fourrier-Reglat A, Haramburu F, Moore N. Impact of safety alerts on measures of disproportionality in spontaneous reporting databases: the notoriety bias. Drug Saf 2007;30:891—8. Upssala Monitoring Center VigiLyze. Search and analysis tool for VigiBaseTM the WHO global ICSR (Individual Case Safety Report) database; 2019, https://www.vigilyze.who-umc.org/ [Accessed 25 January 2019]. Bergvall T, Norén GN, Lindquist M. VigiGrade: a tool to identify well-documented individual case reports and highlight systematic data quality issues. Drug Saf 2014;37:65—77.
[26] Faillie JL, Robin P, Bres V, Pinzani V, Bos-Thompson MA, Hillaire-Buys D. The ATHE score: a new quality score for spontaneous adverse drug reaction reports. Fundam Clin Pharmacol 2013;27:106—7. [27] Uppsala Monitoring Center (UMC). Annual Report 2015—2016; 2016, https://www.who-umc.org/media/3081/umc-annualreport-final-version small.pdf [Accessed 25 January 2019 (24 pp.)]. [28] VigiLyze. Search and analysis tool for VigiBaseTM the WHO global ICSR (Individual Case Safety Report) database. Data from France in 2016; 2019, https://www.vigilyze.who-umc.org/#/ [Accessed 25 January 2019]. [29] Norén GN, Orre R, Bate A, Edwards IR. Duplicate detection in adverse drug reaction surveillance. Data Min Knowl Disc 2007;14:305—28, http://dx.doi.org/10.1007/s10618006-0052-8. [30] Grundmark B, Holmberg L, Garmo H, Zethelius B. Reducing the noise in signal detection of adverse drug reactions by standardizing the background: a pilot study on analyses of proportional reporting ratios-by-therapeutic area. Eur J Clin Pharmacol 2014;70:627—35. [31] Salvo F, Raschi E, Moretti U, Chiarolanza A, Fourrier-Réglat A, Moore N, et al. Pharmacological prioritisation of signals of disproportionate reporting: proposal of an algorithm and pilot evaluation. Eur J Clin Pharmacol 2014;70:617—25. [32] Grégoire F, Pariente A, Fourrier-Reglat A, Haramburu F, B?gaud B, Moore N. A signal of increased risk of hypoglycaemia with angiotensin receptor blockers caused by confounding. Br J Clin Pharmacol 2008;66:142—5. [33] Pariente A, Didailler M, Avillach P, Miremont-Salamé G, Fourrier-Reglat A, Haramburu F, et al. A potential competition bias in the detection of safety signals from spontaneous reporting databases. Pharmacoepidemiol Drug Saf 2010;19:1166—71. [34] Salvo F, Leborgne F, Thiessard F, Moore N, Bégaud B, Pariente A. A potential event-competition bias in safety signal detection: results from a spontaneous reporting research database in France. Drug Saf 2013;36:565—72. [35] Faillie JL, Montastruc F, Montastruc JL, Pariente A. Pharmacoepidemiology and its input to pharmacovigilance. Therapie 2016;71:211—6.
Available online at
ScienceDirect www.sciencedirect.com
PHARMACOEPIDEMIOLOGY
Case—non-case studies: Principle, methods, bias and interpretation夽 Jean-Luc Faillie a,b,∗ a
Regional Pharmacovigilance Centre, Department of Medical Pharmacology and Toxicology, CHU de Montpellier, 34295 Montpellier, France b Laboratory of Biostatistics, Epidemiology and Public Health, EA2415, Faculty of Medicine, University of Montpellier, University Institute of Clinical Research, 34295 Montpellier, France Received 15 January 2019; accepted 22 January 2019 Available online 31 January 2019
KEYWORDS Case—non-case studies; Disproportionality analysis; Reporting odds ratio; Pharmacovigilance; Pharmacoepidemiology; Bias
Summary Case—non-case studies are among the methods used to assess drug safety by analyzing the disproportionality of adverse drug reaction reports in pharmacovigilance databases. First used in the 1980s, the last few decades have seen a significant increase in the use of this study design. The principle of case—non-case studies is to compare the drug exposure of cases of an adverse reaction of interest with that of cases with other reported reactions and called ‘‘non-cases’’. Their results are presented in the form of reporting odds ratio (ROR), the interpretation of which makes it possible to highlight pharmacovigilance signals. This article describes the principle of case—non-case studies, the calculation method of the ROR and its confidence interval, the different analytical modalities and how to interpret its results with regard to the advantages and limitations of this type of study. © 2019 Soci´ et´ e franc ¸aise de pharmacologie et de th´ erapeutique. Published by Elsevier Masson SAS. All rights reserved.
Abbreviations
夽
This manuscript is an updated translation of an article published in French by the same author in the journal Thérapie in 2018 (Therapie 2018;73:247—55). ∗ Corresponding author. Département de pharmacologie médicale et toxicologie, CHU de Montpellier, 371, avenue du Doyen-GastonGiraud, 34295 Montpellier, France. E-mail address: [email protected]
AERS adverse event reporting system ATC anatomical therapeutic chemical BCPNN Bayesian confidence propagation neural network EMA European Medicines Agency FDA: Food and Drug AdministrationHERG human ether-a-gogo-related gene HRT hormone replacement therapy MedDRA medical dictionary for regulatory activities
https://doi.org/10.1016/j.therap.2019.01.006 0040-5957/© 2019 Soci´ et´ e franc ¸aise de pharmacologie et de th´ erapeutique. Published by Elsevier Masson SAS. All rights reserved.
226 MGPS PRR PTs ROR SMQ WHO
J.-L. Faillie mutli-item gamma Poisson Shrinker proportional reporting ratio preferred terms reporting odds ratio standardized MedDRA queries World Health Organization
Introduction Case—non-case studies are among the disproportionality studies that specifically concern the analysis of pharmacovigilance databases. These databases are composed of adverse reaction cases spontaneously reported to pharmacovigilance systems. Different databases exists: at the national level (such as the French national pharmacovigilance database or the adverse event reporting system [AERS] in the United States) or at the international level (such as the European Medicines Agency [EMA] database, EudraVigilance or the World Health Organization [WHO] database, VigiBase). The objective of studying the disproportionality of spontaneous reports is to generate pharmacovigilance signals concerning unknown or underestimated adverse drug reactions, as early as possible after marketing. In this context, a signal refers to a higher than expected number of adverse reaction reports with a specific drug, generating a ‘‘disproportionate’’ rate of reporting compared to other reactions recorded in the pharmacovigilance database. Disproportionality studies are used either for automatic signal detection in a database or for investigating a hypothesis based on pharmacological analysis or clinical signs of drug risk. Disproportionality studies have been increasingly used since the 1990s. Indeed, the development and availability of pharmacovigilance databases have made it possible, especially since it does not require significant logistical resources. At the same time, the frequent health scandals concerning the safety of medicines and the lack of any real improvement in risk assessment before medicines are marketed highlight the need for early detection of signals once the medicine is on the market. The different methods for studying the disproportionality of adverse drug reaction reports include frequentist methods such as the case—non-case study or the proportional reporting ratio (PRR) study and Bayesian methods such as the multi-item gamma Poisson Shrinker (MGPS) method [1] or the Bayesian confidence propagation neural network (BCPNN) method [2].
Principle of the case—non-case In order to highlight a link between the reporting of an adverse reaction of interest and a drug of interest, the case—non-case study is based on the analysis of a 2 × 2 contingency table describing the number of reports of the adverse reaction of interest in the database and the mention in the report of the exposure to the drug of interest (Table 1). Reports of the adverse reaction of interest are called ‘‘cases’’ and other reports are called ‘‘non-cases’’. Despite its similarities to the case-control method, the term ‘‘noncases’’ is used instead of ‘‘controls’’ because this group
Table 1 Contingency table for the case—non-case analysis.
Drug of interest Other drugs
ADR of interest ‘‘Cases’’
Other ADRs ‘‘Non-cases’’
a c
b d
ADR: adverse drug reaction.
refers to patients who have all been exposed to at least one drug and have all experienced at least one reaction other than the adverse reaction of interest. Typically, the number of reports in each of the cells of the contingency table is represented with the letters a, b, c and d, corresponding to cases exposed to the drug of interest (a), non-cases exposed to the drug of interest (b), cases exposed to other drugs (c) and non-cases exposed to other drugs (d). In the absence of a signal, the distribution of cases and non-cases should be independent of the exposure to the drug of interest, i.e. similar between reports exposed to the drug of interest and those exposed to other drugs. A signal is identified when a difference in this distribution (a ‘‘disproportionality’’) is statistically demonstrated. The case—non-case method was used for the first time in 1982 in a study exploring the association between the use of valproic acid during pregnancy and occurrence of spina bifida, based on a register of fetal malformations in a region around Lyon in France [3]. Among the 146 mothers of children with spina bifida, 9 were exposed to valproic acid in the first trimester of pregnancy and among the 6616 mothers of children with other malformations, 21 were exposed, resulting in an odds ratio of 20.6 (P < 0.001). The authors used a case-control design in which the controls were cases of fetal malformations other than spina bifida, i.e. ‘‘noncases’’. This study was thus retrospectively considered as the first case—non-case study [4]. The demonstration of disproportionality is based on the calculation of a reporting odds ratio (ROR) [5,6] The term and calculation of the ROR was first used by Stricker and Tijssen in 1992, when they studied the link between serum sickness and the use of cefaclor (compared to other antibiotics) in an international pharmacovigilance database [7]. Based on the calculation of the odds ratio typically used in case-control study, the calculation of the ROR is quite simple. For an exposure group, the reporting odd is equal to the reporting frequency (f) divided by (1−f). Thus, for the group exposed to the drug of interest, the reporting frequency corresponds to a/(a + b) and the reporting odd is equal to [a/(a + b)]/[1 − a/(a + b)] which simplifies to a/b. Similarly, for the group exposed to other drugs, the reporting odd is c/d. The ROR can be calculated if there is no cell in the contingency table with a null value [6]. It corresponds to the ratio of the reporting odds between groups exposed and not exposed to the drug of interest, and is therefore [a/b]/[c/d], or ad/bc [8]. The ROR measures the strength of disproportionality; if the ROR is 1, there is no signal, meaning that the adverse reaction of interest is similarly reported with the drug of interest as with other drugs (if the ROR is less than 1, there is no signal either, the adverse
Case—non-case studies reaction being less reported with the drug of interest than with other drugs). On the contrary, if the ROR is greater than 1, cases are more reported with the drug of interest than with other drugs: there is a signal and the higher the value of the ROR, the greater the disproportionality. The ROR being a statistical estimate, it should always be presented and interpreted with its 95% confidence interval (95% CI). The calculation of the 95% CI of the ROR involves the variance of the ROR natural logarithm, which is represented by the following simplified formula: 1/a + 1/b + 1/c + 1/d. Thus, the 95% CI of the ROR can be calculated as follows:
ROR ∗ e
±1.96
1 1 1 1 a+b+c+d
If the lower bound of the 95% CI is greater than 1, then the disproportionality signal is statistically significant and the ROR is generally interpreted in these terms: the adverse reaction of interest is ROR times more reported with the drug of interest than with other drugs. Conversely, if the lower bound of the 95% CI is lower than 1, there is no signal. In the study presented by Stricker and Tijssen, the ROR for serum sickness reports with cefaclor compared to amoxicillin was 12.4 (95% CI: 8.2—18.7), showing a largely statistically significant signal: serum sickness reports were more than 12 times more reported with cefaclor than with amoxicillin [7].
Comparison with other methods for studying disproportionality Among other disproportionality measures, the proportional reporting ratio (PRR) is also calculated from the contingency table and is [a/(a + b)]/[c/(c + d)]. A signal is generally considered to be present when PPR is greater than 2, Chisquared statistic value is greater than 4 and number of cases is greater than 3 [5]. The PPR value is generally similar to the ROR. In Bayesian methods, for instance the Bayesian confidence propagation neural network (BCPNN) method, disproportionality is expressed with the information component (IC), which is interpreted in reference to zero. Bayesian methods are considered more robust than frequentist methods when the number of exposed cases is low. However, in a study comparing different disproportionality measures on 10 drug/adverse reaction combinations in the Food and Drug Administration (FDA) reporting database, Chen et al. concluded that the ROR would perform better than other techniques in terms of signal precocity [9]. Another advantage of the ROR is that it allows the use of multivariate logistic regression and thus the consideration of confounding and interaction effects [6].
The different approaches to the disproportionality analysis The study of disproportionality can be carried out according to various methodological approaches. First, various choices can be made regarding the temporality of notifications. Depending on the risk investigated, the analysis could only
227 concern a specific period (for example, the total marketing period of the drug of interest, or only a period preceding a safety alert). The RORs can also be calculated and presented as a function of time (cumulative ROR or per unit of time) to explore time-related biases (see section on temporal biases). Second, the analysis could focus on subgroups or selected reports such as serious cases, cases reported by health professionals, cases for which the imputability of the drug of interest is highest (drug considered ‘‘suspect’’), adverse reactions corresponding to a specific medical domain (e.g. dermatology, psychiatry, etc.), or adverse reactions concerning drugs of a specific therapeutic class (e.g. antidiabetics, antibiotics, etc.). Implementing exclusion criteria to the selection of reports could also allow to explore competition biases (see section on competition biases). The disproportionality analysis made for a specific drug/adverse reaction combination could also be supplemented by the analysis of other adverse reactions related to the reaction of interest or by the analysis of drugs belonging to the same pharmacological or therapeutic class as the drug of interest. All these modalities can be the subject of secondary or sensitivity analyses exploring the robustness of the main results.
Strengths One of the main advantages of disproportionality studies is that that they allow the study of rare adverse reactions, given that only a few reports of exposed cases are needed to carry out the analysis. Also, compared to clinical trials, disproportionality studies theoretically concern an exhaustive population of real users (including specific subgroups: the elderly, children, patients with comorbidities, etc.) and therefore are representative of drug use in real population and real conditions. In addition, the interest of disproportionality studies in drug risk monitoring is their ability to provide early (and possibly automated) detection of signals that will justify the conduct of confirmatory studies. On many occasions, disproportionality estimates have been shown to be well correlated with the strength of the risk association later confirmed by controlled studies [10]. For example, a case—non-case study of bladder cancer reports associated with pioglitazone in the FDA adverse events database between 2004 and 2009 revealed that a signal of disproportionality was present as early as 2004 (ROR 4.8, 95% CI: 1.3—15.9) [11], i.e. 7 years before the confirmation of the risk and implementation of regulatory measures taken [12]. However, confirmatory studies are not always feasible and, when they are, do not always lead to firm conclusions. For example, the signal of acute pancreatitis associated with incretin-based drugs suggested by several case—non-cases studies in various pharmacovigilance databases [13—15] has not yet been confirmed by pharmacoepidemiological studies, leaving the risk as a signal. Finally, the logistics of case—non-case studies are relatively simple, they are not expensive, and, thanks to development of international standards, pharmacovigilance databases are available to study adverse drug reactions at the international level. The case—non-case method also allows pharmacodynamic data to be used to explore mechanistic hypotheses of iatrogeny. For example, a study investigating the
228 pharmacological inhibition of the human ether-a-go-gorelated gene (HERG) potassium channels used a case—noncase method to show that the anti-HERG activity of drugs was directly associated with more frequent reports of severe ventricular arrhythmias and sudden death in the WHO database. The authors thus recommended the measurement of anti-HERG activity in preclinical drug tests to predict their pro-arrhythmic effect [16]. Another example is a case—non-case study combining pharmacodynamic data on inhibition of various monoamine receptors by antipsychotics with adverse reaction reports related to antipsychotics. As a result, the inhibition of serotonin 5-HT2c and histamine H1 receptors was shown to be associated with increased reporting of diabetes mellitus (ROR 2.1; 95% CI: 1.7—2.6 and 95% CI: 1.4—2.6, respectively) [17]. Disproportionality analysis can also be applied to other data sources such as epidemiologic surveys. For example, in the field of addictovigilance, the abuse potential of psychoactive drugs has been studied by comparing the disproportionality estimates of criteria such as abuse and dependence, illegal acquisition, diverted route of administration and concomitant alcohol consumption based on data from annual cross-sectional surveys in drug addicts [18].
Limitations and biases Under-reporting of adverse drug reactions is an inherent limitation to the interpretation of studies conducted on pharmacovigilance databases. Pharmacovigilance systems cannot aim to collect the exhaustivity of adverse reactions but focus on their relevance for alerting about a risk. It has been estimated in France that no more than 5% of serious adverse drug reactions are actually reported [19]. Underreporting hinders the measurement of the actual incidence of adverse drug reaction and limits the sensitivity of signal detection by disproportionality methods. In addition, under-reporting is variable (referred to as ‘‘selective reporting’’), being influenced by many parameters depending on the reporter, the drugs, the reaction itself or time. Thus, a disproportionality signal may be the result of either an actual increase in the incidence of the adverse reaction or an artifactual increase in reports not correlated to an increase in incidence thus impacting signal detection specificity. Factors of variation in reporting include experience and type of reporter, local usage and reporting patterns, severity of the adverse reaction (serious adverse reactions are usually more reported), specificity of the adverse reaction to the drug, unexpectedness in relation to the known safety profile of the drug (unexpected adverse reactions are more commonly reported) [20], media coverage of an adverse reaction (notoriety effect) or time since market authorization (e.g. Weber effect). These factors are part of the main biases limiting the interpretation of disproportionality studies. A summary of biases and limitations of the case—non-case studies are presented in Table 2, with their consequences and the methods for measuring or avoiding them.
Temporal biases The Weber effect is defined as a variation in reporting over time that results in an increase in the number of reports
J.-L. Faillie immediately after a drug is marketed (due to the incomplete safety profile and increasing exposure). Although, it is not necessarily observed for all types of drugs, it mainly occurs in the first two years, with a high proportion of non-serious reactions followed by a decline (decreased enthusiasm for reporting adverse reactions becoming better known) [21,22]. The Weber effect can also be observed for an older drug with a new indication or a change in dosage. The impact of the Weber effect on the ROR estimate is variable and depends on its intensity on the adverse reaction of interest (increase in number [a]) compared to other adverse reaction (number [b]). In any case, disproportionality analyses are less effective in detecting long-term effects because they are remote from marketing. For the study of serious reactions, the Weber effect including a high proportion of non-serious reactions, it may constitute a masking effect showing a larger increase in the number of nonserious non-cases (number [b]) potentially responsible for an underestimation of the ROR. This point therefore justifies restricting the case—non-case analysis to serious adverse reactions only. The notoriety effect results in increased reports following media coverage of a specific adverse drug reaction (publication of a scientific study, safety alerts from regulatory agencies addressed to health professionals or to the public, articles or discussions in public media or social networks) [23]. An increase in the number of reports is observed for incident cases but also for retrospective cases (which occurred before the alert). In all cases, this effect leads to a bias that is responsible for an increase in the ROR for the drug in question. To avoid it, it is necessary, if possible, to ensure that the signal is present before the media event. Pariente et al. provided several illustrations of this bias. For example, in the French national pharmacovigilance database, the ROR for stroke that occurred with atypical antipsychotics were 0.1 (95% CI: 0.02—1.0) and 2.0 (95% CI: 1.2—3.4) respectively before and after the publication of a press release from the French medicine agency mentioning an increased risk in a clinical trial in an elderly population with dementia [23]. An alert concerning a drug adverse reaction may also lead to a ‘‘ripple effect’’ resulting in an increase in reports of the adverse reaction of interest with other drugs belonging to the same pharmacological or therapeutic class as the drug initially suspected in the alert [23]. For example, in the FDA pharmacovigilance database, an increase of nearly 40% in reports of pancreatitis was observed with sitagliptin after an FDA alert regarding the risk of pancreatitis with exenatide [14].
Information biases Missing data are frequent in pharmacovigilance databases, however, they generally do not concern the identification of the medicinal product or the diagnosis of the effect and therefore do not prevent the disproportionality analysis itself. Nevertheless, missing information regarding clinical data, indication, age, sex, comedications, evolution, type of report and reporter, country, specific doses or treatment dates limit complementary analyses and confounding assessment. Among all the reports available in VigiBase, sex and age were missing in 6.0% and 25.6%, respectively [24]. A report completeness score, called VigiGrade, has been
Case—non-case studies Table 2
229
Main biases and limitations of case—non-case studies, consequences and possible solutions.
Bias/limitation
Consequences
Under-reporting
Lack of measurement of actual incidence and risk Lack of statistical power Decrease in signal detection sensitivity Overestimated or underestimated ROR
Selective reporting
Weber effect
Notoriety bias and ripple effect Missing data
Duplicates Information bias on clinical and exposure data
Variability of the ROR over time Overestimated or underestimated ROR (if masking effect by numerous non-serious non-cases) Overestimated ROR
Limitation for additional analyses and management of confounders Incorrect ROR Incorrect ROR
Selection bias/confounding
Incorrect ROR
Drug competition bias Event competition bias All biases
Underestimated ROR Underestimated ROR Incorrect ROR
Solutions
Restriction or adjustment on factors influencing the reporting (severity, type of reporter, region. . .) Use of positive and negative controls to explore the reporting of the studied effect ROR temporal analysis Restriction of the analysis to serious cases only (if masking effect)
ROR temporal analysis Restriction of the analysis to time period preceding media coverage Comparison of the analysis restricted to non-missing data with that using a ‘‘missing data’’ category Exploration of imputation possibilities Manual or algorithmic ‘‘deduplication’’ Restriction to medically confirmed reports Appropriate definitions using adverse reaction and drug classifications Restriction to cases where the drug is considered ‘‘suspect’’ Reference group consisting of drugs of the same pharmacological or therapeutic class than the studied drug Subgrouping/stratification/adjustment for confounders (factors influencing report and risk factors) Exclusion of reports concerning drugs frequently associated with the adverse reaction of interest Exclusion of reports of adverse reactions known to be frequently associated with the drug of interest Exploring biases by performing sensitivity analyses Search for consistent results for positive and negative controls
ROR: reporting odds ratio.
developed by the Uppsala center managing VigiBase. Its calculation is carried out by assigning penalties according to the availability of information and its clinical relevance [25]. A report is considered well documented when the VigiGrade score is higher than 0.8. In 2014, only 13% of the reports present in VigiBase met this criterion and this rate showed great variability according to the country (e.g. it is generally better in Europe than in the United States), the type of notification (24% for doctors vs. 4% for non-health professionals) [25] but also according to the clinical context (for example, there are more missing data for serious cases in diabetic patient), or the origin of the report (missing data are more frequent in cases reported by the industry) [26]. In 2016, France was among the 10 most contributing countries to VigiBase [27] with a median completeness score of 0.80 [28]. There is no ideal solution to take missing data into account. It may be useful to explore this bias by comparing analyses restricted to reports with all the data provided and
those with a ‘‘missing data’’ category. Imputation methods could also be experimented. Multiple reports of the same adverse reaction are a significant source of error, therefore, ‘‘deduplication’’ is the first step in any analysis using pharmacovigilance databases. In VigiBase, an algorithm called VigiMatch is used to identify and remove duplicates while keeping the most informative reports [29]. Similarly, due to the anonymization of reports and the lack of a unique patient identifier, it is often difficult to perform an analysis with the patient as the statistical unit: in this case, multiple events for the same patient (e.g. a recurrent effect) would count as several independent observations. Errors in reporting diagnosis, exposure, or imputability are often dependent on the type of reporter. In order to limit this information bias, it is possible to select observations by restricting the analysis to observations for which the risk of error is lower such as adverse reactions reported by health
230 professionals or reports in which the drug is considered ‘‘suspect’’ or has a suggestive chronology. To avoid exposure errors, particular attention should be paid to the definition of exposure using drug classifications. For example, failure to include the anatomical therapeutic chemical (ATC) codes for the combined forms of the study drug may result in information bias. Regarding the definition of the adverse reaction, it is crucial to assess the clinical relevance of the list of selected terms from adverse reaction classifications such as MedDRA (medical dictionary for regulatory activities). Although it is recommended to use MedDRA preferred terms (PTs) to conduct signal detection [10], standardised MedDRA queries (SMQ) may also be helpful by providing selections of adverse reaction terms with narrow (more specific) or broad (more sensitive) definitions. Also, the search for other clinical data (indication, medical history) from proxies based on drug exposures can also be a source of bias. Temporal analyses may also be limited by the time lag between the occurrence of the reaction and the recording of reports in pharmacovigilance databases, particularly multinational databases. For example, in VigiBase, reports are recorded monthly for some countries, but quarterly or twice a year for other countries.
J.-L. Faillie In disproportionality analyses, confounders include variables involved in the association between exposure and adverse reaction reporting (potentially all the factors responsible for selective reporting presented above) but also include the variables involved in the association between exposure and the effect itself, such as the indication of the drug. For example, a signal concerning the risk of hypoglycemia associated with angiotensin-converting enzyme (ACE) inhibitors was identified in the French national pharmacovigilance database (RORs ranging from 3 to 4). In fact, diabetes was a confounder: ACE inhibitors are more favorably prescribed to diabetic patients who are also treated with antihyperglycemic agents, this increases the risk of episodes of hypoglycemia and therefore their reporting. This is referred to as indication bias or co-prescription bias. When stratifying the analysis on the presence or absence of antihyperglycemic agents, the signal disappeared (ROR equal to 1 in each stratum) [32]. Solutions for confounding generally include subgrouping, stratification or adjustment. It is suggested that subgroup analyses (in which a signal is identified if it is present in any of the subgroup strata) tend to perform better than stratified/adjusted analyses [10].
Competition biases Selection bias/confounding In disproportionality analyses, as in pharmacoepidemiology studies in general, comparisons may be biased by the fact that patients exposed to the drug of interest may be more (or less) at risk for the adverse reaction than those exposed to other drugs: this can refer to indication bias, severity bias, channeling bias or healthy user effect. In the exposed group, an increase risk due to other factors would result in an ‘‘over-reporting’’ and therefore a false signal. To limit these effects, the choice of the reference group is crucial. Choosing all ‘‘other drugs’’ as the reference group can be problematic in the presence of indication bias: the signal would reflect the risk associated with the indication for the drug and not the exposure to drug itself. For example, for the study of pancreatitis reports associated with exenatide (indicated for the treatment of type 2 diabetes), if the reference group includes all other drugs and not only diabetes drugs, the observed signal may be explained by the effect of the indication (diabetic patient being at higher risk of pancreatitis). The reference group should therefore be defined by the exposure to other drugs belonging to the same therapeutic group (in this example, diabetes drugs). A study conducted in the European database EudraVigilance compared the results of disproportionality analyses with and without restricting the reference group to the therapeutic class of the studied drugs (drugs for prostate disorders and drugs for type 2 diabetes). The authors demonstrated that, by using this type of restriction, the performance of the disproportionality estimator was improved with a decrease in false positives (confounded by the disease) and a better ability to detect true positives [30]. Also, in a pilot study evaluating a signal prioritization score from the analysis of pharmacovigilance databases, Salvo et al. recommended to take into account pharmacological criteria such as the ROR calculated from the therapeutic class of the drug to increase signal specificity [31].
Competition biases tend to mask signal detection. A bias due to competition between drugs may occur when the adverse reaction studied is also frequently reported with one or more drugs other than the drug of interest [33]. Thus, the rate of unexposed cases (number [c] of the contingency table) is high, leading to an underestimation of the ROR, i.e. a signal that goes unnoticed. This bias increases false negatives and therefore reduces the sensitivity of signal detection. A solution to this bias is to remove from the analysis the reports concerning drugs frequently associated with the adverse reaction of interest: this reduces the number of exposed cases needed to show a signal. In the example of strokes with atypical antipsychotics, there is a competitive bias between antipsychotics and anticoagulants, the latter being known to increase the risk of stroke and thus their report. The exclusion of anticoagulant reports revealed a previously undetected signal for stroke with atypical antipsychotics: RORs equal to 1.1 (95% CI: 0.7—1.9) and 2.0 (95% CI: 1.2—3.4) respectively before and after exclusion [33]. A bias due to competition between events can occur when the drug of interest is associated with a significant number of reports for adverse reactions other than the adverse reaction of interest [34]. In this case, the rate of exposed non-cases (number [b] in the contingency table) is high, also leading to an underestimation of the ROR and a decreased sensitivity. The solution is similar to the previous situation: removing the adverse reaction reports known to be frequently associated with the drug of interest. For example, in a statin disproportionality study, the withdrawal of muscle disease reports, a known adverse reaction to statins, reveals previously undetected signals such as libido disorder, alopecia, cholestasis, dermatitis, or digestive bleeding [34]. This bias also justifies conducting the disproportionality analysis restricted to reports belonging to the same system-organ-class than the adverse reaction of interest (e.g. dermatological, psychiatric, hematologic, etc.).
Case—non-case studies
231
Other potential biases
Funding
Other potential biases often studied in pharmacoepidemiology may also limit the interpretation of disproportionality studies. One example is the protopathic bias that occurs when a drug is initiated in response to a symptom or an early stage of the suspected effect. For example, the use of analgesics in response to pain caused by an undiagnosed cancer, may lead, when the cancer is diagnosed, to reports suggesting that the analgesic caused the cancer while the causality is reversed. In this situation, the true chronology must be restored by introducing a latency period following the exposure and during which the cases that have been diagnosed are considered unexposed. Surveillance bias or detection bias is another example to keep in mind. It occurs when the detection of the event is more likely in an exposure group due to an increased surveillance, disease screening or an associated symptom. For example, reports suggesting a risk of endometrial cancer with hormone replacement therapy (HRT) could theoretically be the consequence of better detection of these cancers in women treated with HRT. Indeed, HRT are known to cause uterine bleeding, which is a symptom that motivates screening for endometrial cancer.
This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors.
The use of positive and negative controls to assess the presence of bias The presence of an undetected bias can be investigated by studying ‘‘control’’ drugs whose association or absence of association with the studied effect is expected from the literature. This involves measuring the disproportionality of (i) drugs known to cause the adverse reaction (positive controls) and (ii) drugs that are not related to the adverse reaction (negative controls) and verifying that there is a signal for positive controls and no signal for negative controls. The consistency between expected and obtained results for controls would suggest that the study is not compromised by major biases. For example, the disproportionality study of the association between antihypertensive drugs and hypoglycemia by Gregoire et al. used cibenzoline and disopyramide as positive controls and diazepam as negative controls. As expected, a signal suggesting a higher risk of hypoglycemia was found for cibenzoline (ROR 107; 95% CI: 78—148) and disopyramide (ROR 17; 95% CI: 10—29) while no signal was found for diazepam (ROR 0.3; 95% CI: 0.1—2) [32].
Conclusions Case—non-case studies, like other disproportionality studies, analyze real-life pharmacovigilance data and, despite significant limitations and many biases, allow for the early detection of pharmacovigilance signals. They are sometimes the only tools available, particularly for the study of rare events or uncommonly used drugs. Signals from case—noncase studies, coupled with pharmacological and clinical analysis, remain crucial elements of the body of evidence needed for the management of drug safety [35].
Disclosure of interest The author declares that he has no competing interest.
References [1] DuMouchel W. Bayesian data mining in large frequency tables, with an application to the FDA spontaneous reporting system. Am Stat 1999;53:177—90. [2] Bate A, Lindquist M, Edwards IR, Olsson S, Orre R, Lansner A, et al. A Bayesian neural network method for adverse drug reaction signal generation. Eur J Clin Pharmacol 1998;54:315—21. [3] Bjerkedal T, Czeizel A, Goujard J, Kallen B, Mastroiacova P, Nevin N, et al. Valproic acid and spina bifida. Lancet 1982;2(8307):1096. [4] Montastruc JL, Sommet A, Bagheri H, Lapeyre-Mestre M. Benefits and strengths of the disproportionality analysis for identification of adverse drug reactions in a pharmacovigilance database. Br J Clin Pharmacol 2011;72:905—8. [5] Moore N, Thiessard F, Begaud B. The history of disproportionality measures (reporting odds ratio, proportional reporting rates) in spontaneous reporting of adverse drug reactions. Pharmacoepidemiol Drug Saf 2005;14:285—6. [6] van Puijenbroek EP, Bate A, Leufkens HGM, Lindquist M, Orre R, Egberts ACG. A comparison of measures of disproportionality for signal detection in spontaneous reporting systems for adverse drug reactions. Pharmacoepidemiol Drug Saf 2002;11:3—10. [7] Stricker BH, Tijssen JG. Serum sickness-like reactions to cefaclor. J Clin Epidemiol 1992;45:1177—84. [8] Moore N, Kreft-Jais C, Haramburu F, Noblet C, Andrejak M, Ollagnier M, et al. Reports of hypoglycaemia associated with the use of ACE inhibitors and other drugs: a case/non-case study in the French pharmacovigilance system database. Br J Clin Pharmacol 1997;44:513—8. [9] Chen Y, Guo JJ, Steinbuch M, Lin X, Buncher CR, Patel NC. Comparison of sensitivity and timing of early signal detection of four frequently used signal detection methods: an empirical study based on the US FDA adverse event reporting system database. Pharm Med 2008;22:359—65. [10] Wisniewski AFZ, Bate A, Bousquet C, Brueckner A, Candore G, Juhlin K, et al. Good signal detection practices: evidence from IMI PROTECT. Drug Saf 2016;39:469—90, http://dx.doi.org/10.1007/s40264-016-0405-1. [11] Piccinni C, Motola D, Marchesini G, Poluzzi E. Assessing the association of pioglitazone use and bladder cancer through drug adverse event reporting. Diabetes Care 2011;34:1369—71. [12] Faillie JL, Petit P, Montastruc JL, Hillaire-Buys D. Scientific evidence and controversies about pioglitazone and bladder cancer: which lessons can be drawn? Drug Saf 2013;36:693—707. [13] Elashoff M, Matveyenko AV, Gier B, Elashoff R, Butler PC. Pancreatitis, pancreatic, and thyroid cancer with glucagon-like peptide-1-based therapies. Gastroenterology 2011;141:150—6. [14] Raschi E, Piccinni C, Poluzzi E, Marchesini G, De Ponti F. The association of pancreatitis with antidiabetic drug use: gaining insight through the FDA pharmacovigilance database. Acta Diabetol 2013;50:569—77. [15] Faillie JL, Babai S, Crepin S, Bres V, Laroche ML, Le Louet H, et al. Pancreatitis associated with the use of GLP-1 analogs
232
[16]
[17]
[18]
[19]
[20]
[21]
[22]
[23]
[24]
[25]
J.-L. Faillie and DPP-4 inhibitors: a case/non-case study from the French Pharmacovigilance Database. Acta Diabetol 2014;51:491—7. De Bruin ML, Pettersson M, Meyboom RHB, Hoes AW, Leufkens HGM. Anti-HERG activity and the risk of drug-induced arrhythmias and sudden death. Eur Heart J 2005;26:590—7. Montastruc F, Palmaro A, Bagheri H, Schmitt L, Montastruc JL, Lapeyre-Mestre M. Role of serotonin 5-HT2C and histamine H1 receptors in antipsychotic-induced a pharmacoepidemiological-pharmacodynamic diabetes: study in VigiBase. Eur Neuropsychopharmacol 2015;25: 1556—65. Pauly V, Lapeyre-Mestre M, Braunstein D, Rueter M, Thirion X, Jouanjus E, et al. Detection of signals of abuse and dependence applying disproportionality analysis. Eur J Clin Pharmacol 2015;71:229—36. Bégaud B, Martin K, Haramburu F, Moore N. Rates of spontaneous reporting of adverse drug reactions in France. JAMA 2002;288:1588. Martin RM, Kapoor KV, Wilton LV, Mann RD. Underreporting of suspected adverse drug reactions to newly marketed (‘‘black triangle’’) drugs in general practice: observational study. BMJ 1998;317:119—20. Wallenstein EJ, Fife D. Temporal patterns of NSAID spontaneous adverse event reports: the Weber effect revisited. Drug Saf 2001;24:233—7. Hartnell NR, Wilson JP. Replication of the Weber effect using postmarketing adverse event reports voluntarily submitted to the United States Food and Drug Administration. Pharmacotherapy 2004;24:743—9. Pariente A, Gregoire F, Fourrier-Reglat A, Haramburu F, Moore N. Impact of safety alerts on measures of disproportionality in spontaneous reporting databases: the notoriety bias. Drug Saf 2007;30:891—8. Upssala Monitoring Center VigiLyze. Search and analysis tool for VigiBaseTM the WHO global ICSR (Individual Case Safety Report) database; 2019, https://www.vigilyze.who-umc.org/ [Accessed 25 January 2019]. Bergvall T, Norén GN, Lindquist M. VigiGrade: a tool to identify well-documented individual case reports and highlight systematic data quality issues. Drug Saf 2014;37:65—77.
[26] Faillie JL, Robin P, Bres V, Pinzani V, Bos-Thompson MA, Hillaire-Buys D. The ATHE score: a new quality score for spontaneous adverse drug reaction reports. Fundam Clin Pharmacol 2013;27:106—7. [27] Uppsala Monitoring Center (UMC). Annual Report 2015—2016; 2016, https://www.who-umc.org/media/3081/umc-annualreport-final-version small.pdf [Accessed 25 January 2019 (24 pp.)]. [28] VigiLyze. Search and analysis tool for VigiBaseTM the WHO global ICSR (Individual Case Safety Report) database. Data from France in 2016; 2019, https://www.vigilyze.who-umc.org/#/ [Accessed 25 January 2019]. [29] Norén GN, Orre R, Bate A, Edwards IR. Duplicate detection in adverse drug reaction surveillance. Data Min Knowl Disc 2007;14:305—28, http://dx.doi.org/10.1007/s10618006-0052-8. [30] Grundmark B, Holmberg L, Garmo H, Zethelius B. Reducing the noise in signal detection of adverse drug reactions by standardizing the background: a pilot study on analyses of proportional reporting ratios-by-therapeutic area. Eur J Clin Pharmacol 2014;70:627—35. [31] Salvo F, Raschi E, Moretti U, Chiarolanza A, Fourrier-Réglat A, Moore N, et al. Pharmacological prioritisation of signals of disproportionate reporting: proposal of an algorithm and pilot evaluation. Eur J Clin Pharmacol 2014;70:617—25. [32] Grégoire F, Pariente A, Fourrier-Reglat A, Haramburu F, B?gaud B, Moore N. A signal of increased risk of hypoglycaemia with angiotensin receptor blockers caused by confounding. Br J Clin Pharmacol 2008;66:142—5. [33] Pariente A, Didailler M, Avillach P, Miremont-Salamé G, Fourrier-Reglat A, Haramburu F, et al. A potential competition bias in the detection of safety signals from spontaneous reporting databases. Pharmacoepidemiol Drug Saf 2010;19:1166—71. [34] Salvo F, Leborgne F, Thiessard F, Moore N, Bégaud B, Pariente A. A potential event-competition bias in safety signal detection: results from a spontaneous reporting research database in France. Drug Saf 2013;36:565—72. [35] Faillie JL, Montastruc F, Montastruc JL, Pariente A. Pharmacoepidemiology and its input to pharmacovigilance. Therapie 2016;71:211—6.