- Email: [email protected]
Quality issues in clinical research and the implications on health policy (QICRHP)
Quality Issues in Clinical Research and the Implications on Health Policy (QICRHP) VERONICA P. S. NJIE, MSN, RN, CS,* AND ANNE C. THOMAS, PHD, RN, CS, ANP†
Scientific (research) misconduct relating to fraudulent behaviors, plagiarism, and violations in research protocol is extant in the literature. These behaviors result in inadvertent infringement on subjects’ rights, safety, and confidentiality, and can negatively impact the implementation of health policies. Funding agencies, legislators, regulators, clinicians, and scientists continue to subject researchers’ work to scrutiny and in-depth analyses to evaluate its quality because of increased reports of scientific misconduct. Therefore, clinical researchers are motivated to restore and maintain credibility throughout clinical trials to regain public trust because of adverse publicity. Quality in clinical research addresses the study’s integrity and merit, which has important practice and policy implications. Study integrity and merit encompass both the scientific and ethical components of clinical research studies. Researchers suggest using structured evaluation processes, educating, and mentoring aspiring scientists on scientific integrity to prevent scientific misconduct. In addressing these issues, this article explicates and provides a synthesized critique of quality issues in clinical research, analyzes its implications on health policy and clinical practice, and proposes a systematic framework to evaluate clinical research studies. (Index words: Quality, Scientific misconduct, Scientific integrity, Health policy, Clinical research/practice) J Prof Nurs 17:233-242, 2001. Copyright © 2001 by W.B. Saunders Company
Q
UALITY ASSURANCE/IMPROVEMENT is a mandatory and vital process in health care. Quality related to clinical research specifically addresses the integrity and merit of the research study,
*Assistant Professor, Baltimore City Community College, Baltimore, MD. †Clinical Director, Division of Intramural Research, Chief, Health Promotion Laboratory, National Institute of Nursing Research, National Institutes of Health, Bethesda, MD. Address correspondence and reprint requests to Ms. Njie: Baltimore City Community College, 2901 Liberty Heights Ave, Baltimore, MD 21215. E-mail: [email protected] This is a US government work. There are no restrictions on its use. 8755-7223/01/1705-0008$35.00/0 doi:10.1053/jpnu.2001.26308
which has important implications in health policy and clinical practice. This, in turn, has resulted in scrutiny and in-depth analyses of clinical research studies by funding agencies, legislators, regulators, clinicians, and scientists (Joseph, 1998; Kefalides, 2000; Rankin & Esteves, 1997; Ryan, 1999). Prompting this course of action are reports of scientific misconduct relating to fraudulent behaviors, plagiarism, and advances in research and technology. These issues can potentially and inadvertently cause infringement on subjects’ rights, safety, and confidentiality, and negatively impact the implementation of health policies. Clinical researchers struggle to regain public trust from the adverse publicity and to maintain and ensure credibility. The purposes of this article are to: (1) explicate and provide a synthesized critique of quality issues in clinical research; (2) analyze its implications on health policy and clinical practice; and (3) propose a systematic framework to evaluate clinical research studies. Health policy regulators greatly depend on research evidence to make informed decisions that affect clinical practice (Lohr, Eleazer, & Mauskopf, 1998; Prashker, 1996). At the National Institutes of Health (NIH), each Institute has the responsibility to ensure quality data and information in research studies. This strategy of individualized standards by each NIH Institute invokes autonomy and motivates creativity (Lindquist, Tracy, & Treat-Jacobson, 1995; Rudy & Kerr, 2000); however, there is potential for confusion and difficulties in reinforcing the standards adopted by each Institute (Rudy & Kerr). In addition, individualized quality standards promote stringent policies that are instituted to ascertain the safety and protection of research subjects. With the advent of increasingly complex scientific and technological advances, participating subjects are potentially placed at greater risk. The importance of establishing quality research standards was reinforced by the announcement from Secretary Shalala to “further strengthen protection of human subjects in clinical trials.” The Secretary worked closely with the NIH and the Office of Protection from Research Risks (OPRR) to update and ex-
Journal of Professional Nursing, Vol 17, No 5 (September–October), 2001: pp 233-242
233
234
pand the Human Protection Guidelines in clinical research that were initially instituted. The announcement addressed the following issues of education and training for investigators and members of the research team: Informed consent with the expectation that research institutions will “audit records for evidence of compliance with informed consent requirements”; conflict of interest; and civil monetary penalties for “violations of informed consent and other important research practices” (United States Department of Health and Human Services [U.S. DHHS], 2000a, 2000b). Before this announcement, Miller, Rosenstein, and DeRenzo (1998) also suggested educating principal investigators and participating researchers in conducting clinical research studies. Miller et al believe that scientific integrity is achieved through training and experience. They advocate for medical students to be exposed to scientific literature through training and to be allowed to assist researchers in clinical trials. Other researchers suggest providing nurse scientists with mentors to train them on the processes and importance of scientific integrity (Ketefian & Lenz, 1995; Lenz & Ketefian, 1995; Macrina & Munro, 1995). In this way, aspiring researchers will learn to respect the importance of achieving integrity and merit in clinical research. These issues of scientific integrity and merit and the protection of human subjects in clinical research, motivated the initiation of specific clinical standards by the NIH/Intramural Research Program (IRP) to ascertain that participating investigators “fulfill their responsibilities” and maintain “quality of the research.” This standard mandates that clinical trials be “monitored adequately and centrally” and that the Institute is responsible for determining the “appropriate extent and nature of monitoring” (NIH/IRP, 2000 p. 3). Quality assurance/improvement has a myriad of definitions and its theoretical foundations are embedded in management and organizational theories (Hyde, 1998). In this article, quality in clinical research is the process of developing and implementing guidelines to ensure the inclusion of all pertinent aspects of the research process, ensure accountability of research team members, adherence to protocol guidelines, and maintenance of study integrity and merit. Limited scientific studies have been conducted to determine whether established guidelines to provide evaluation of clinical research are effective in achieving quality. However, clinical researchers involved in reviewing or auditing clinical research suggest that errors
NJIE AND THOMAS
were identified and modifications made before submitting the study for external or institutional reviews (Rudy & Kerr, 2000; Ryan, 1999). In some circumstances, fraudulent behaviors were identified during the peer review process, prompting regulators to establish scientific review offices (Becker, 1998; Blancett, 1996; Ryan). Other studies identified infringement on subjects’ rights (Karlawish, Hougham, Stocking, & Sachs, 1999; Rankin & Esteves, 1997).
Literature Review QUALITY ISSUES
Quality assurance/improvement is a process used to monitor and maintain the integrity and validity of practice standards in any system. Quality assurance (QA) is now referred to as quality improvement (QI) in the health care industry. When the term QA was originally used, the intent was to ensure that organizational system standards were strictly adhered to by comparing actual practice with stipulated standards. Most health care practitioners consider QA a passive, reactive, and unidirectional process because changes are not made unless problems are identified. On the other hand, some believe that quality improvement is a dynamic, proactive, and continuous process and is initiated before problems arise. Notwithstanding the stage or time that the problems are identified in both processes, the main objectives are to solve identified problems and to promote and maintain quality standards. In addition to QI and QA, other quality processes extant in the scientific literature are continuous quality improvement (CQI) and total quality management (TQM) (Hyde, 1998; McNelly, 1997; Volgelsang, 1999). The concept of quality remains the same depending on the context in which it is used; however, the differences are embedded in the modalities used to implement the processes. An elaborate discussion of the various contexts in which quality is used is beyond the scope of this article. Quality processes focus on issues of infrastructure, process, and outcome (Lohr et al., 1998) as they relate to practice standards. In clinical research, the process dimension addresses all aspects of the research process; that is, accountability of research participants, maintenance of the integrity and credibility of the data, and protocol adherence. Over the years, scientific researchers monitored studies conducted by their peers because of increasing incidents of “scientific misconduct and fraud” reported in various agencies and institutions.
235
QUALITY IN CLINICAL RESEARCH
These incidents ranged from plagiarism to actual fabrication of scientific results (Blancett, 1996; Macrina, 1995; Miller et al., 1998; Pallikkathayil, Crighton, & Aaronson, 1998a; Rankin & Esteves, 1997; Rudy & Kerr, 2000; Ryan, 1999). Researchers succumbed to these behaviors, perhaps, because of personal and professional reasons such as pressure from colleagues to advance academically, to secure funds for the study, to publish results, and to be promoted. Funding agencies and regulators are therefore motivated to address these issues because the public continues to show mistrust in scientific results. Multiple mechanisms are adopted by researchers to implement quality processes in clinical research. Among them are peer reviews, scientific evaluations, research audits, and research monitors. A research audit conducted by Christian McCabe, Korn, Abrams, Kaplan, and Friedman (1995) at the NIH established the adequacy of data generated from a study protocol regarding lumpectomy and breast irradiation research. The audit was prompted by reports from the National Surgical Adjuvant Breast and Bowel Protocol investigators alleging fraudulent data and conclusions. The audit, which involved a replication of the study at the participating institutions, confirmed the accuracy and “adequacy of the data” and the results. The investigators’ intent was to verify the data and results of the initial study rather than evaluate specific components of the research process. In the replication study, 37 of the original 89 institutions were used. The investigators failed to clarify informed consent issues albeit the absence of a number of undocumented informed consent forms, which further validates their intent to verify data and results and not evaluate the research process. A qualitative study conducted by Douglas, Jarvis, and Bloore (1998) to assess the effectiveness of a monitoring tool used by an ethics committee reported some discrepancies identified by the auditing investigators that were overlooked by the ethics committee. The discrepancies included a breach of confidentiality when clients’ names and other data were given to the student investigator to contact subjects. Both of these examples indicate the need and importance of pre- and postreviews to evaluate quality methodology, which could identify discrepancies throughout the research process. The important link among clinical research, clinical practice, and health policy amplifies the importance of scientific rigor and integrity in research. Quality in clinical research serves to validate and verify the research process, to ensure accountability by participants in
the study, and to determine congruency between the funding contract and the completed proposal/protocol. Scientists must take the moral and ethical responsibility to establish data accuracy and veracity, which enhances the integration of the results in clinical practice. QUALITY ISSUES AND CLINICAL RESEARCH
Interventional clinical research studies have the potential to improve clinical practice (Straus & Sackett, 1998). It is clear then that clinical research studies conducted should be of the utmost integrity and merit to promote this process. The importance of ensuring that the research would benefit the subjects studied is an area of intense debate (Noble-Adams, 1999b; Thomas, 1999). This encompasses and crystallizes the concept of evidence-based practice, which advocates the clinical implementation of research results into practice. To implement and use research findings in clinical practice, the study has to be applicable to the subjects. Scientific evaluation is a mechanism that this important step of the research process can be ensured. A structured review of research publications by Karlawish et al. (1999) to determine the quality of research ethics in studies conducted on nursing home residents identified a myriad of incomplete data with regard to informed consent. The issue of justifying the use of nursing home residents for the study was explained clearly on the consent form (100 per cent). Only 40 per cent (n ⫽ 45) of the publications checked reported that an Institutional Review Board (IRB) reviewed the research, and 80 per cent (n ⫽ 45) reported that informed consent was obtained or that it was waivered. More alarming results were related to the consent form and consent process. Only 29 per cent (n ⫽ 35) indicated documentation of consent; none of the studies reported whether potential benefits were disclosed; 46 per cent (n ⫽ 35) reported the decisional capacity of the subjects; and only 69 per cent (n ⫽ 35) indicated whether those incapacitated were excluded from or included in the study. Nursing home residents are vulnerable to violations of human rights because of numerous physical or psychologic challenges such as hearing and visual loss, physical disabilities, and dementia. If consent is not possible, assent may be indicated and should be reported by the investigators. An assent suggests that the subject can verbally agree to join the study but cannot provide written consent. The researcher must obtain the informed consent from a legal guardian or respon-
236
NJIE AND THOMAS
sible adult before the study is completed. In addition to maintaining the integrity and validity of the study, it is also crucial to maintain the protection of human rights and safety throughout the research process. The subjects’ rights were clearly violated in these studies. Emanuel, Wendler, and Grady (2000) posit that “valuable” clinical research is one that can be implemented in clinical practice for the purpose of optimizing the health of the research participants. Research valuability is a construct shared by most researchers because it relates to clinical relevancy. It is important that clinical researchers tailor their studies to address the needs of the subjects in lieu of their professional advancement. The plethora of clinical research studies with no implications or relevance to practice continues to infiltrate and dilute clinical research with the potential to minimize the importance and benefits of research. With the availability of commercial and nonprofit IRBs, it can be a challenge for the scientific research community to ensure integrity and merit of clinical research studies. The commercial IRBs are financially motivated and may produce a turnaround time of approximately 10 working days in contrast to an average of 1 month for nonprofit IRBs (Kefalides, 2000). The potential errors and oversight inherent in assessing integrity and merit by commercial IRBs in the research studies are compounded by the accelerated process time coupled with reviewers who may have minimal expertise in the research process. When some ethics/ review committees have too many studies to evaluate, it propagates the potential for oversight and errors. An internal scientific evaluation process before submitting the study for review is essential to ensure that all components of the research process and the various indicators of quality are included.
QUALITY ISSUES AND SCIENTIFIC INTEGRITY AND MERIT
The literature contains extensive information about scientific (research) integrity and merit. Integrity and merit encompass both the scientific and ethical components of clinical research studies. It is assumed that integrity is an inherent trait in clinical researchers; however, recent incidents have dispelled this assumption. The notion that clinical researchers are perhaps consumed with the need to advance academically or professionally has resulted in reports suggesting that research results have been falsified, fabricated, plagiarized, or manipulated. Also, reports indicate that inves-
tigators have failed to adhere to established protocol (Becker, 1998; Fain, 1997; Macrina, 1995; Miller et al., 1998; Pallikkathayil et al., 1998; Rankin & Esteves, 1997; Rudy & Kerr, 2000; Ryan, 1999). These have caused negative publicity for the research community and resulted in an erosion of public trust. Scientific misconduct occurs when researchers infringe on the integrity and merit of clinical studies. Reviewers of clinical research still debate about whether some of these actions should be considered “sloppy science” caused by inexperience or “scientific misconduct” as a result of intent to deceive (Macrina, 1995, p. 2). They argue that in some cases researchers may have innocently failed to include data because of lack of knowledge or poor judgment (Macrina, 1995; Rudy & Kerr, 2000). In either case, negative effects will occur if discrepancies are not identified and corrected early. In addition, notwithstanding, it does not preclude that the information is inaccurate and may in some cases cause tragic consequences that can affect clinical practice or negatively impact the implementation of health policies. Rankin and Esteves (1997) surveyed nurse research coordinators, directors, and deans from master’s and doctoral nursing programs to determine the perceived frequency (incidence) of scientific misconduct at the participants’ institutions. Of the 88 nurses surveyed, 49.7 per cent of respondents reported an occasional occurrence of plagiarism, 27.2 per cent on cheating on data collection, 33.2 per cent on misinterpretation of the findings, and a total of 65.8 per cent on protocol violations relating to the subjects and research site. With regard to fraudulent behavior, Ryan (1999) provided incidents of scientific misconduct found in some research studies. In one of the cases, false data were reported by a graduate student working with a well-known and respected researcher at the NIH. During the review process, the researcher was openly questioned about the fraudulent behavior. The senior researcher was unaware of the student’s behavior but later personally investigated the incident. He found “extensive fraud in the study,” which the student confirmed. The researcher sent letters to at least “100 scientists” (p. 94) to apprise them of the problem and then rescinded the published information. In institutions and agencies such as the NIH in which protocols are used, one of the biggest challenges is strict adherence to protocol. Participating researchers are challenged with the importance of recognizing when to appropriately intervene because of clinical consequences experienced by subjects during the study
QUALITY IN CLINICAL RESEARCH
and when to just observe and not do anything. Not adhering to protocol is also considered scientific misconduct; but it has been suggested that poor judgement related to lack of knowledge may be a confounding factor. Researchers should be adequately trained to prepare them for their role in scientific studies and provided with closer supervision by senior research members. On the issue of data falsification, Rudy and Kerr (2000) reported media coverage about claims of scientific misconduct on a breast cancer study at the University of Pittsburgh. The misconduct involved one of the investigators who “falsified data on the eligibility of subjects” (p. 117–118). The study was also conducted at other sites. An evaluation process confirmed the accusations. The subjects recruited by the investigator were removed from the analyses of the study and the data were reanalyzed. Most of the clinical research studies that reported scientific misconduct both involved multiple investigators and institutions and are usually important studies that have the propensity for media coverage and public attention. As a result of these reports, regulators and agencies established offices such as the Office of Research Integrity (ORI), the Recombinant DNA Advisory Committee (RAC), and the OPRR to facilitate implementation of quality procedures in clinical trials. Most of the reported behaviors were identified during a review/evaluation process. It is fair to state that it is usually difficult to predict future challenges, especially when it relates to the intricacies of the clinical research process. However, the challenges, when identified during the experience/practice process, must be addressed immediately by implementing appropriate strategies. Scientific misconduct has been identified in clinical research. With the tendency for the media to exaggerate incidents for public consumption and regulators to overreact from public pressure, it is incumbent on the scientific community to be proactive in the process and establish internal structured processes that would stymie misconduct. By implementing quality evaluation processes, it is possible to detect scientific misconduct. It is the responsibility of the principal investigators and participating researchers to ensure that the scientific integrity and merit of the study are maintained.
QUALITY ISSUES, CLINICAL RESEARCH, AND HEALTH POLICY
Health policy decisions that affect clinical practice are embedded in evidence established through
237
research. Quality in clinical research is closely linked to cost containment and may determine appropriate allocation of grants for research. Although funding institutions and agencies require that specific criteria in grant applications be addressed, many funding agencies fail to provide follow-up on the results after the proposals have been deemed adequate or acceptable. The literature is clear on issues of infringement of the research process after money has been disbursed; and, therefore, researchers strongly advocate a continuous evaluation process instead of a onetime strategy (Arnot, Jepsen, & Kilama, 2000; Kefalides, 2000; Macrina, 1995; Noble-Adams, 1999a). This has become more important as the competition for research grants escalates and financial resources diminish (Volgelsang, 1999). The current trend by legislators is to provide more funds for clinical research, which suggests that an increase in competition for research grants by researchers seems to grow faster than the rate of the increase in research funds. The more researchers are motivated to conduct pragmatically applicable clinical research, the more regulators will prioritize funding based on societal/public needs. Funding agencies and regulators continue to rely on scientific research evidence to initiate health policy decisions; hence, the need to ensure that quality clinical research is conducted (Lohr et al., 1998; Prashker, 1996). Regulators also assess consistencies between research results and clinical recommendations to make informed decisions. Winther and Hole (1997) conducted a study to determine the scientific quality of 40 randomly selected clinical research projects. The indicators used to determine quality were: (1) Does the project contribute to new knowledge or can it be used as a control for results of previous studies? and (2) Is the protocol of good (study can be used and published without substantial alteration), inferior (study can be used and published after revisions are made), or unfit/unsatisfactory (the study should not be started nor published) quality? The results indicated that 80 per cent (n ⫽ 40) would contribute to new knowledge, 37.5 per cent (n ⫽ 40) were of good quality, 37.5 per cent (n ⫽ 40) of inferior quality, and 10 were unfit for use. Of the 40, 9 could be accepted for publication, 10 were in need of revision, and 7 were unsatisfactory/unfit for publication. In one of the studies, the investigators made some changes in the research process between the time the proposal was accepted and when the results were reported. The
238
NJIE AND THOMAS
investigators of this study identified the importance of an evaluation process before submission to the review board, and a postevaluation procedure to ensure that the same study proposal accepted was the same one maintained at the completion of the study. The study also determined that quality clinical research studies have the propensity to be cost effective in the interest of funding agencies and can also ensure that the ethical dimension of the research process is maintained. Even though the investigators indicated that pharmacology/pharmacotherapy trials were more likely to conduct clinical research with humans, a larger sample size and the inclusion of other types of clinical research projects would have lent the results more generalizability. It is also important to note that the investigators failed to operationally define “substantial alteration” on the evaluation guidelines and the other aspects of the research study that were evaluated. Operationally defining the quality indicators would have provided a more thorough and objective perspective of the evaluation process.
Implications for Clinical Practice and Health Policy
Interventional and quality clinical research studies have the propensity to positively impact on health policies. Regulators use the results as a basis for making decisions and its subsequent impact on clinical practice. The quality assurance process in clinical research is a catalyst for health policy changes. In an international study conducted by Grant, Cottrell, and Cluzeau (2000), a structured peer-review process was used to evaluate the impact of clinical research on practice. Data were collected from clinical guidelines based on diverse areas of disease management. The National Health Service (NHS) that determines the quality of clinical research studies in Great Britain usually evaluates these guidelines. One of the criteria for selection for review was citation of the article in clinical guidelines because the NHS deemed them quality research; hence, the reason these 2,501 articles were selected for review in which the guidelines were cited. The indicators used were the time the article was published to the time it was cited in a clinical guideline; the country of authorship; and the extent of clinical research cited in clinical guidelines. The indicator of interest here is the extent the research was cited in a guideline. Of the 2,501 articles evaluated, 2,043
(82 per cent) were found in research journals and 3% were used in clinical practice. It is important to note that the practice guidelines were “developed in and for the United Kingdom.” However, the implications for practice and health policy are obvious. Structured evaluations of clinical research studies identify and filter studies that are not practice oriented and have the potential to incur expenses that do not benefit the public. With stringent policies by regulators and funding agencies, clinical researchers must ensure quality studies through an evaluation process. This trend is becoming more attractive as financial resources for funding continue to be competitive. It also reinforces the need for researchers to be accountable for the responsibilities bestowed them (Gaughwin, 1999). One of the most important variables in health policy is the improvement of public health in a cost-effective manner. Regulators depend on quality clinical research data to direct them to appropriate and effective interventions for the public’s benefit. An objective, structured, scientific evaluation process must be instituted in research centers, universities, and other researchoriented agencies to promote facilitation of scientific integrity and ethical veracity. The ethical component has direct implications on health policy with regard to infringement on the rights and safety of the subjects, and the misuse of financial resources (Winther & Hole, 1997). Therefore, the potential risks to human subjects cannot be overemphasized. The ANA’s position on ethics and human rights contends to Promulgate in collaboration. . .both theoretical and practical. . . to address issues in ethics and human rights; develop and disseminate information about and advocate for public policy to assure that ethics and human rights are addressed; assure that short and longrange objectives regarding human rights will be addressed. . .(www.ana.org/ethics, p. 1).
The quality research audit undertaken by Karlawish et al. (1999) has the potential to mitigate the chances of ethical infringements. Secretary Shalala’s announcement will also help to ensure protection of subject participants in any research study. However, there is a dire need to implement internal scientific evaluation process in the various agencies and institutions to curb these mistakes. This will facilitate the early identification of discrepancies and the revisions made before external review. The evaluation process will serve as a verificational and validation process for quality clinical research studies.
239
QUALITY IN CLINICAL RESEARCH
Proposed Methodology
A fixed, prescriptive, perfect protocol to conduct a scientific evaluation process is not possible; however, a generic framework will assist individuals or groups in various institutions and agencies to evaluate their research studies before submitting them for external review. Although some reviews are conducted informally, it is important to institute structured reviews that would propagate objectivity and abate potential biases (Lindquist et al., 1995). An independent evaluation process reinforces accountability of the research participants (Emanuel et al., 2000) before it gets to the external reviewers. The author does not suggest that the proposed framework is new, but encourages researchers to conduct internal structured evaluation processes by using the general principles of the research process while incorporating recent Federal requirements. The extent and format for the evaluation process is left to the discretion of the research institutions and agencies. This article proposes a generic scientific evaluation framework as a mechanism to maintain quality in clinical research. There are three areas of foci: infrastructure, process, and outcome dimensions. The quality indicators on this evaluation framework will focus on the infrastructure and process dimensions. The outcome dimension is not addressed in evaluating clinical research because it is not applicable until after the results have been clinically implemented. The proposed method facilitates an unbiased and objective manner to address essential aspects of the research study. Kitson, Harvey, Hyndman, and Yerrell (1994) suggest specific characteristics to provide a guide to develop “good” quality criteria indicators (p. 157). These characteristics include statements that are relevant, understandable, measurable, stated in behavioral terms, and achievable; hence, the acronym RUMBA (Kitson et al, 1994). These characteristics are used as a guide to develop the scientific evaluation framework proposed here.
At the NIH, the pre-IRB scientific review committee members usually include the “Institute’s Scientific and Clinical Directors, and a staff member from the Institute’s extramural program” (NIH, 2000, p. 28). If other research institutions or agencies wish to adopt this scientific evaluation framework, the committee should include scientific and academic researchers, clinician(s), and ethics expert(s). The scientific and academic researchers evaluate the scientific integrity and merit of the study, the clinician(s) evaluate the study’s applicability to clinical practice, and the ethics experts evaluate the ethical component of the study. The exact number of team members to conduct the evaluation is left to the discretion of each institute and agency. This comprehensive evaluation process ascertains the scientific integrity, merit, and the clinical practice applicability of clinical research studies. Successful completion of the review process does not necessarily mean that the study will be successful nor that integrity and merit will be maintained. Evaluating the study before submitting the research proposal for external review (pre) and after the study has been conducted (post) may ensure quality through the whole research process. A point system scale method of scoring is used for this evaluation (see table 1). Table 2 shows the scientific evaluation framework for clinical research studies to be used by institutions and agencies to evaluate their clinical research studies. As Table 2 shows, the quality indicators are consistent with the general principles of the research process. Specific Federal requirements such as the mandate for signed consent/assent forms and training participating investigators are incorporated for evaluation. Each criterion is worth 1 point and the strengths and/or weaknesses can be explained in the “comments” section. The “point” section has two columns for the pre- and postevaluation. The total number of points is computed after each evaluation session (pre and post), and the recommendations are submitted as shown on the “Evaluation Scoring Chart” (table 1).
TABLE 1. Evaluation Scoring Chart Quality Evaluation
Description
Score Range Pre-evaluation
Score Range Postevaluation
Good quality Fair quality Poor quality/unacceptable
Minimum to no revisions necessary Some revisions necessary Major revisions and rewrite necessary
20–23 16–19 ⬍16
30–32 25–29 ⬍25
240
NJIE AND THOMAS
TABLE 2. Criteria to Evaluate Clinical Research Studies Point Content
Pre
Post
Comments
Evaluation of Infrastructure Investigators/resources/presentation Reports expertise of principal investigators and the other research members Documents congruency between the investigator’s expertise and the research objectives Clearly documents the role of each participating investigator Documents that participating investigators received training on the “Protection of Human Subjects” States the availability of human, material, and equipment resources Documents budget that is consistent with the resources cited Presents a clear, concise, and logical study Evaluation of Process Hypotheses/objectives/background Clearly states the aims/objectives/purposes/hypotheses of the study Reports relevancy of the research objectives to clinical practice Clearly reports the implications and significance of the study to nursing science and nursing (clinical) practice Critically evaluates research studies in the literature Shows originality of the study Methodology Uses appropriate instrument(s)/intervention(s) Reports validity and reliability of the instruments Uses an appropriate selection criteria for the subjects Provides clear steps for the data collection procedure or protocol Uses statistical analyses methods consistent with the research questions Human subjects Explains the study in simple terms on the consent (assent) form States rationale for choosing research subjects Explains clearly the potential risks and benefits of the study Indicates on the consent (assent) form that subjects can voluntarily withdraw without penalty Indicates that subjects’ confidentiality will be maintained Includes contact number(s) on the form for subjects to call for information Results/discussion* Documents evidence of signed and dated consent forms Shows consistency between the number of subjects and the number of signed and dated consent forms Presents the results accurately Shows consistency between the information collected and the information reported Shows strict adherence to all steps of the research procedure/protocol Analyzes the study’s applicability to nursing (clinical) practice Shows that the discussion of the study is based on the results Deduces/infers important points from the results Provides recommendations for future research studies *To be evaluated after the research study is completed. This is in addition to all the previously mentioned indicators.
Conclusion
Quality assurance/improvement is a vital process in clinical research and health policy. A quality clinical research study suggests that the research process is adhered to, participating researchers take accountability for their role in the research process, and the integrity and merit of the study is maintained. Instituting guidelines for clinical practice depends on accurate and appropriately interpreted data. Health policy regulators rely on the data produced by quality clinical research studies to make decisions; in turn, this validates the importance of quality in clinical research. To promote quality in clinical research, the mechanisms used in-
clude peer review, research audit, scientific evaluation, and research monitoring. A structured, scientific evaluation process has been proposed. Researchers are encouraged to use this process as a guide to ascertain quality clinical research, thereby maintaining the integrity and merit of the study. The framework contains quality indicators that focus on the infrastructure and process dimensions of the quality process. The team members of the evaluation process must be diverse to include scientific and academic researchers, clinicians, and ethics experts. Quality clinical research studies must be maintained to provide health policy regulators valid and credible
241
QUALITY IN CLINICAL RESEARCH
information on which to base their decisions that affect clinical practice. Acknowledgements The authors would like to extend their heartfelt thanks to Mr. Daniel O’Neal, Director of the Office of Scientific Policy and Public Liaison at the National Institutes of Health, for his suggestions on the overall manuscript and especially on the health
policy component. Also, to Mr. Mark Waldo at the National Institutes of Health for agreeing to read the paper; and to Dr. Abdoulaye Saine, Miami University in Oxford, Ohio, for reviewing the manuscript. This manuscript was written in partial fulfillment of a Summer Intramural Research Training Award under the preceptorship of Dr. Anne C. Thomas, Clinical Director of the Division of Intramural Research and Chief of the Health Promotion Laboratory at the National Institute of Nursing Research, National Institutes of Health.
References Arnot, D. E., Jepsen, S., & Kilama, W. (2000). Health research ethics in Africa. Parasitology Today, 16, 136-137. Becker, P. T. (1998). Scientific integrity and the journal review process. Research in Nursing and Health, 21, 283-284. Blancett, S. S. (Ed.) (1996). Integrity and misconduct in research. Journal of Nursing Administration, 26, 7-8. Christian, M. C., McCabe, M. S., Korn, E. L., Abrams, J. S., Kaplan, R. S., & Friedman, M. A. (1995). The National Cancer Institute audit of the national surgical adjuvant breast and bowel project protocol B-06. New England Journal of Medicine, 333, 1469-1474. Douglas, A. J., Jarvis, A., & Bloore, S. (1998). Monitoring of health research by ethics committees. New Zealand Medical Journal, 111, 79-81. Emanuel, E. J., Wendler, D., & Grady, C. (2000). What makes clinical research ethical. Journal of the American Medical Association, 283, 2701-2711. Fain, J. A. (1997). Maintaining scientific integrity in publications. The Diabetes Educator, 23, 232. Gaughwin, M. D. (Ed.) (1999). Health policy—the process and the politics. Medical Journal of Australia, 171, 56. Grant, J., Cottrell, R., & Cluzeau, F. (2000). Evaluating payback on biomedical research from papers cited in clinical guidelines: Applied biblometric study. British Medical Journal, 320, 1107-1111. The Center for Ethics and Human Rights: ANA [Online]. Available: http://www.ana.org/ethics (7/7/2000). Gallin, J. & Gottesman, M. Standards for clinical research within the NIH research program [On-line]. Available: http:// www.cc.nih.gov/ccc/clinicalresearch/ standards.html (6/12/2000). Hyde, J. C. (1998). Theoretical foundations of total quality management and continuous quality improvement: Or, does quality really have its own theory? Topics in Health Information Management, 18, 60-67. Joseph, K. S. (1998). Ethics in clinical research: Searching for absolutes. Canadian Medical Association Journal, 158, 1303-1305. Karlawish, J. H., Hougham, M. A., Stocking, C. B., & Sachs, G. A. (1999). What is the quality of the reporting of research ethics in publications of nursing home research? Research Ethics, 47, 76-81. Kefalides, P. T. (2000). Research on humans faces scrutiny; new policies adopted. Annals of Internal Medicine, 132, 513-516. Ketefian, S., & Lenz, E. R. (1995). Promoting scientific
integrity in nursing research, Part II: Strategies. Journal of Professional Nursing, 11, 263-269. Kitson, A., Harvey, G., Hyndman, S., & Yerrell, P. (1994). Criteria formulation and application: An evaluation framework. International Journal of Nursing Standard, 31, 155-167. Lenz, E. R., & Ketefian, S. (1995). Promoting scientific integrity in nursing research. Part I: Current approaches in doctoral programs. Journal of Professional Nursing, 11, 213219. Lindquist, R. D., Tracy, M. F., & Treat-Jacobson, D. (1995). Peer review of nursing research proposals. American Journal of Critical Care, 4, 59-65. Lohr, K. N., Eleazer, K., & Mauskopf, J. (1998). Health policy issues and applications for evidence-based medicine and clinical practice guidelines. Health Policy, 46, 1-19. Macrina, F. L. (1995). Scientific integrity: An introductory text with cases (pp. 1-14; 69-95). Washington, DC: American Society for Microbiology. Library of Congress Cataloging in Publication Data. Macrina, F. L., & Munro, C. L. (1995). The case study approach to teaching scientific integrity in nursing and the biomedical sciences. Journal of Professional Nursing, 11, 4044. McNelly, J. A. (1997). Taking the mystery out of research: Comparative analysis of research, quality improvement, and quality assurance. Orthopedic Nursing, 16, 68-69. Miller, F. G., Rosenstein, D. L., & DeRenzo, E. G. (1998). Professional integrity in clinical research. Journal of the American Medical Association, 280, 1449-1454. National Institutes of Health. (2000, January). Protomechanics: A guide to preparing and conducting a clinical research study. Bethesda, MD: The Clinical Center. Noble-Adams, R. (1999a). Ethics and nursing research 1: Development, theories and principles. British Journal of Nursing, 8, 888-892. Noble-Adams, R. (1999b). Ethics and nursing research 2: Examination of the research process. British Journal of Nursing, 8, 956-960. Pallikkathayil, L., Crighton, F., & Aaronson, L. S. (1998a). Issues in clinical nursing research: Balancing ethical quandaries with scientific rigor: Part I. Western Journal of Nursing Research, 20, 388-393. Pallikkathayil, L., Crighton, F., & Aaronson, L. S. (1998b). Issues in clinical nursing research: Balancing ethical quandaries with scientific rigor: Part II. Western Journal of Nursing Research, 20, 501-507.
242
Prashker, M. J. (1996). Health services research and policy studies. Current Opinion in Rheumatology, 8, 106-109. Rankin, M., & Esteves, M. D. (1997). Perceptions of scientific misconduct in nursing. Nursing Research, 46, 270-276. Rudy, E. B., & Kerr, M. E. (2000). Auditing research studies. Nursing Research, 49, 117-120. Ryan, K. J. (1999). Research misconduct in clinical research: The American experience and research. Acta Oncologica, 38, 93-97. Straus, S. E., & Sackett, D. L. (1998). Using research findings in clinical practice. British Medical Journal, 317, 339-342. Thomas, B. (1999). Research and audit in effective health services. Nursing Standard, 13, 40-42.
NJIE AND THOMAS
U.S. Department of Health and Human Services. (May 23, 2000a). Secretary Shalala bolsters protections for human research subjects. H.H.S News. U.S. Department of Health and Human Services. (May 23, 2000b). Protecting research subjects. H.H.S. Fact Sheet. Vogelsang, J. (1999). Quantitative research versus quality assurance, quality improvement, total quality management, and continuous quality improvement. Journal of Perianesthesia, 14, 78-81. Winther, F. O., & Hole, O. P. (1997). Scientific quality of clinical research. European Journal of Clinical Pharmacology, 51, 351-354.
Scientific (research) misconduct relating to fraudulent behaviors, plagiarism, and violations in research protocol is extant in the literature. These behaviors result in inadvertent infringement on subjects’ rights, safety, and confidentiality, and can negatively impact the implementation of health policies. Funding agencies, legislators, regulators, clinicians, and scientists continue to subject researchers’ work to scrutiny and in-depth analyses to evaluate its quality because of increased reports of scientific misconduct. Therefore, clinical researchers are motivated to restore and maintain credibility throughout clinical trials to regain public trust because of adverse publicity. Quality in clinical research addresses the study’s integrity and merit, which has important practice and policy implications. Study integrity and merit encompass both the scientific and ethical components of clinical research studies. Researchers suggest using structured evaluation processes, educating, and mentoring aspiring scientists on scientific integrity to prevent scientific misconduct. In addressing these issues, this article explicates and provides a synthesized critique of quality issues in clinical research, analyzes its implications on health policy and clinical practice, and proposes a systematic framework to evaluate clinical research studies. (Index words: Quality, Scientific misconduct, Scientific integrity, Health policy, Clinical research/practice) J Prof Nurs 17:233-242, 2001. Copyright © 2001 by W.B. Saunders Company
Q
UALITY ASSURANCE/IMPROVEMENT is a mandatory and vital process in health care. Quality related to clinical research specifically addresses the integrity and merit of the research study,
*Assistant Professor, Baltimore City Community College, Baltimore, MD. †Clinical Director, Division of Intramural Research, Chief, Health Promotion Laboratory, National Institute of Nursing Research, National Institutes of Health, Bethesda, MD. Address correspondence and reprint requests to Ms. Njie: Baltimore City Community College, 2901 Liberty Heights Ave, Baltimore, MD 21215. E-mail: [email protected] This is a US government work. There are no restrictions on its use. 8755-7223/01/1705-0008$35.00/0 doi:10.1053/jpnu.2001.26308
which has important implications in health policy and clinical practice. This, in turn, has resulted in scrutiny and in-depth analyses of clinical research studies by funding agencies, legislators, regulators, clinicians, and scientists (Joseph, 1998; Kefalides, 2000; Rankin & Esteves, 1997; Ryan, 1999). Prompting this course of action are reports of scientific misconduct relating to fraudulent behaviors, plagiarism, and advances in research and technology. These issues can potentially and inadvertently cause infringement on subjects’ rights, safety, and confidentiality, and negatively impact the implementation of health policies. Clinical researchers struggle to regain public trust from the adverse publicity and to maintain and ensure credibility. The purposes of this article are to: (1) explicate and provide a synthesized critique of quality issues in clinical research; (2) analyze its implications on health policy and clinical practice; and (3) propose a systematic framework to evaluate clinical research studies. Health policy regulators greatly depend on research evidence to make informed decisions that affect clinical practice (Lohr, Eleazer, & Mauskopf, 1998; Prashker, 1996). At the National Institutes of Health (NIH), each Institute has the responsibility to ensure quality data and information in research studies. This strategy of individualized standards by each NIH Institute invokes autonomy and motivates creativity (Lindquist, Tracy, & Treat-Jacobson, 1995; Rudy & Kerr, 2000); however, there is potential for confusion and difficulties in reinforcing the standards adopted by each Institute (Rudy & Kerr). In addition, individualized quality standards promote stringent policies that are instituted to ascertain the safety and protection of research subjects. With the advent of increasingly complex scientific and technological advances, participating subjects are potentially placed at greater risk. The importance of establishing quality research standards was reinforced by the announcement from Secretary Shalala to “further strengthen protection of human subjects in clinical trials.” The Secretary worked closely with the NIH and the Office of Protection from Research Risks (OPRR) to update and ex-
Journal of Professional Nursing, Vol 17, No 5 (September–October), 2001: pp 233-242
233
234
pand the Human Protection Guidelines in clinical research that were initially instituted. The announcement addressed the following issues of education and training for investigators and members of the research team: Informed consent with the expectation that research institutions will “audit records for evidence of compliance with informed consent requirements”; conflict of interest; and civil monetary penalties for “violations of informed consent and other important research practices” (United States Department of Health and Human Services [U.S. DHHS], 2000a, 2000b). Before this announcement, Miller, Rosenstein, and DeRenzo (1998) also suggested educating principal investigators and participating researchers in conducting clinical research studies. Miller et al believe that scientific integrity is achieved through training and experience. They advocate for medical students to be exposed to scientific literature through training and to be allowed to assist researchers in clinical trials. Other researchers suggest providing nurse scientists with mentors to train them on the processes and importance of scientific integrity (Ketefian & Lenz, 1995; Lenz & Ketefian, 1995; Macrina & Munro, 1995). In this way, aspiring researchers will learn to respect the importance of achieving integrity and merit in clinical research. These issues of scientific integrity and merit and the protection of human subjects in clinical research, motivated the initiation of specific clinical standards by the NIH/Intramural Research Program (IRP) to ascertain that participating investigators “fulfill their responsibilities” and maintain “quality of the research.” This standard mandates that clinical trials be “monitored adequately and centrally” and that the Institute is responsible for determining the “appropriate extent and nature of monitoring” (NIH/IRP, 2000 p. 3). Quality assurance/improvement has a myriad of definitions and its theoretical foundations are embedded in management and organizational theories (Hyde, 1998). In this article, quality in clinical research is the process of developing and implementing guidelines to ensure the inclusion of all pertinent aspects of the research process, ensure accountability of research team members, adherence to protocol guidelines, and maintenance of study integrity and merit. Limited scientific studies have been conducted to determine whether established guidelines to provide evaluation of clinical research are effective in achieving quality. However, clinical researchers involved in reviewing or auditing clinical research suggest that errors
NJIE AND THOMAS
were identified and modifications made before submitting the study for external or institutional reviews (Rudy & Kerr, 2000; Ryan, 1999). In some circumstances, fraudulent behaviors were identified during the peer review process, prompting regulators to establish scientific review offices (Becker, 1998; Blancett, 1996; Ryan). Other studies identified infringement on subjects’ rights (Karlawish, Hougham, Stocking, & Sachs, 1999; Rankin & Esteves, 1997).
Literature Review QUALITY ISSUES
Quality assurance/improvement is a process used to monitor and maintain the integrity and validity of practice standards in any system. Quality assurance (QA) is now referred to as quality improvement (QI) in the health care industry. When the term QA was originally used, the intent was to ensure that organizational system standards were strictly adhered to by comparing actual practice with stipulated standards. Most health care practitioners consider QA a passive, reactive, and unidirectional process because changes are not made unless problems are identified. On the other hand, some believe that quality improvement is a dynamic, proactive, and continuous process and is initiated before problems arise. Notwithstanding the stage or time that the problems are identified in both processes, the main objectives are to solve identified problems and to promote and maintain quality standards. In addition to QI and QA, other quality processes extant in the scientific literature are continuous quality improvement (CQI) and total quality management (TQM) (Hyde, 1998; McNelly, 1997; Volgelsang, 1999). The concept of quality remains the same depending on the context in which it is used; however, the differences are embedded in the modalities used to implement the processes. An elaborate discussion of the various contexts in which quality is used is beyond the scope of this article. Quality processes focus on issues of infrastructure, process, and outcome (Lohr et al., 1998) as they relate to practice standards. In clinical research, the process dimension addresses all aspects of the research process; that is, accountability of research participants, maintenance of the integrity and credibility of the data, and protocol adherence. Over the years, scientific researchers monitored studies conducted by their peers because of increasing incidents of “scientific misconduct and fraud” reported in various agencies and institutions.
235
QUALITY IN CLINICAL RESEARCH
These incidents ranged from plagiarism to actual fabrication of scientific results (Blancett, 1996; Macrina, 1995; Miller et al., 1998; Pallikkathayil, Crighton, & Aaronson, 1998a; Rankin & Esteves, 1997; Rudy & Kerr, 2000; Ryan, 1999). Researchers succumbed to these behaviors, perhaps, because of personal and professional reasons such as pressure from colleagues to advance academically, to secure funds for the study, to publish results, and to be promoted. Funding agencies and regulators are therefore motivated to address these issues because the public continues to show mistrust in scientific results. Multiple mechanisms are adopted by researchers to implement quality processes in clinical research. Among them are peer reviews, scientific evaluations, research audits, and research monitors. A research audit conducted by Christian McCabe, Korn, Abrams, Kaplan, and Friedman (1995) at the NIH established the adequacy of data generated from a study protocol regarding lumpectomy and breast irradiation research. The audit was prompted by reports from the National Surgical Adjuvant Breast and Bowel Protocol investigators alleging fraudulent data and conclusions. The audit, which involved a replication of the study at the participating institutions, confirmed the accuracy and “adequacy of the data” and the results. The investigators’ intent was to verify the data and results of the initial study rather than evaluate specific components of the research process. In the replication study, 37 of the original 89 institutions were used. The investigators failed to clarify informed consent issues albeit the absence of a number of undocumented informed consent forms, which further validates their intent to verify data and results and not evaluate the research process. A qualitative study conducted by Douglas, Jarvis, and Bloore (1998) to assess the effectiveness of a monitoring tool used by an ethics committee reported some discrepancies identified by the auditing investigators that were overlooked by the ethics committee. The discrepancies included a breach of confidentiality when clients’ names and other data were given to the student investigator to contact subjects. Both of these examples indicate the need and importance of pre- and postreviews to evaluate quality methodology, which could identify discrepancies throughout the research process. The important link among clinical research, clinical practice, and health policy amplifies the importance of scientific rigor and integrity in research. Quality in clinical research serves to validate and verify the research process, to ensure accountability by participants in
the study, and to determine congruency between the funding contract and the completed proposal/protocol. Scientists must take the moral and ethical responsibility to establish data accuracy and veracity, which enhances the integration of the results in clinical practice. QUALITY ISSUES AND CLINICAL RESEARCH
Interventional clinical research studies have the potential to improve clinical practice (Straus & Sackett, 1998). It is clear then that clinical research studies conducted should be of the utmost integrity and merit to promote this process. The importance of ensuring that the research would benefit the subjects studied is an area of intense debate (Noble-Adams, 1999b; Thomas, 1999). This encompasses and crystallizes the concept of evidence-based practice, which advocates the clinical implementation of research results into practice. To implement and use research findings in clinical practice, the study has to be applicable to the subjects. Scientific evaluation is a mechanism that this important step of the research process can be ensured. A structured review of research publications by Karlawish et al. (1999) to determine the quality of research ethics in studies conducted on nursing home residents identified a myriad of incomplete data with regard to informed consent. The issue of justifying the use of nursing home residents for the study was explained clearly on the consent form (100 per cent). Only 40 per cent (n ⫽ 45) of the publications checked reported that an Institutional Review Board (IRB) reviewed the research, and 80 per cent (n ⫽ 45) reported that informed consent was obtained or that it was waivered. More alarming results were related to the consent form and consent process. Only 29 per cent (n ⫽ 35) indicated documentation of consent; none of the studies reported whether potential benefits were disclosed; 46 per cent (n ⫽ 35) reported the decisional capacity of the subjects; and only 69 per cent (n ⫽ 35) indicated whether those incapacitated were excluded from or included in the study. Nursing home residents are vulnerable to violations of human rights because of numerous physical or psychologic challenges such as hearing and visual loss, physical disabilities, and dementia. If consent is not possible, assent may be indicated and should be reported by the investigators. An assent suggests that the subject can verbally agree to join the study but cannot provide written consent. The researcher must obtain the informed consent from a legal guardian or respon-
236
NJIE AND THOMAS
sible adult before the study is completed. In addition to maintaining the integrity and validity of the study, it is also crucial to maintain the protection of human rights and safety throughout the research process. The subjects’ rights were clearly violated in these studies. Emanuel, Wendler, and Grady (2000) posit that “valuable” clinical research is one that can be implemented in clinical practice for the purpose of optimizing the health of the research participants. Research valuability is a construct shared by most researchers because it relates to clinical relevancy. It is important that clinical researchers tailor their studies to address the needs of the subjects in lieu of their professional advancement. The plethora of clinical research studies with no implications or relevance to practice continues to infiltrate and dilute clinical research with the potential to minimize the importance and benefits of research. With the availability of commercial and nonprofit IRBs, it can be a challenge for the scientific research community to ensure integrity and merit of clinical research studies. The commercial IRBs are financially motivated and may produce a turnaround time of approximately 10 working days in contrast to an average of 1 month for nonprofit IRBs (Kefalides, 2000). The potential errors and oversight inherent in assessing integrity and merit by commercial IRBs in the research studies are compounded by the accelerated process time coupled with reviewers who may have minimal expertise in the research process. When some ethics/ review committees have too many studies to evaluate, it propagates the potential for oversight and errors. An internal scientific evaluation process before submitting the study for review is essential to ensure that all components of the research process and the various indicators of quality are included.
QUALITY ISSUES AND SCIENTIFIC INTEGRITY AND MERIT
The literature contains extensive information about scientific (research) integrity and merit. Integrity and merit encompass both the scientific and ethical components of clinical research studies. It is assumed that integrity is an inherent trait in clinical researchers; however, recent incidents have dispelled this assumption. The notion that clinical researchers are perhaps consumed with the need to advance academically or professionally has resulted in reports suggesting that research results have been falsified, fabricated, plagiarized, or manipulated. Also, reports indicate that inves-
tigators have failed to adhere to established protocol (Becker, 1998; Fain, 1997; Macrina, 1995; Miller et al., 1998; Pallikkathayil et al., 1998; Rankin & Esteves, 1997; Rudy & Kerr, 2000; Ryan, 1999). These have caused negative publicity for the research community and resulted in an erosion of public trust. Scientific misconduct occurs when researchers infringe on the integrity and merit of clinical studies. Reviewers of clinical research still debate about whether some of these actions should be considered “sloppy science” caused by inexperience or “scientific misconduct” as a result of intent to deceive (Macrina, 1995, p. 2). They argue that in some cases researchers may have innocently failed to include data because of lack of knowledge or poor judgment (Macrina, 1995; Rudy & Kerr, 2000). In either case, negative effects will occur if discrepancies are not identified and corrected early. In addition, notwithstanding, it does not preclude that the information is inaccurate and may in some cases cause tragic consequences that can affect clinical practice or negatively impact the implementation of health policies. Rankin and Esteves (1997) surveyed nurse research coordinators, directors, and deans from master’s and doctoral nursing programs to determine the perceived frequency (incidence) of scientific misconduct at the participants’ institutions. Of the 88 nurses surveyed, 49.7 per cent of respondents reported an occasional occurrence of plagiarism, 27.2 per cent on cheating on data collection, 33.2 per cent on misinterpretation of the findings, and a total of 65.8 per cent on protocol violations relating to the subjects and research site. With regard to fraudulent behavior, Ryan (1999) provided incidents of scientific misconduct found in some research studies. In one of the cases, false data were reported by a graduate student working with a well-known and respected researcher at the NIH. During the review process, the researcher was openly questioned about the fraudulent behavior. The senior researcher was unaware of the student’s behavior but later personally investigated the incident. He found “extensive fraud in the study,” which the student confirmed. The researcher sent letters to at least “100 scientists” (p. 94) to apprise them of the problem and then rescinded the published information. In institutions and agencies such as the NIH in which protocols are used, one of the biggest challenges is strict adherence to protocol. Participating researchers are challenged with the importance of recognizing when to appropriately intervene because of clinical consequences experienced by subjects during the study
QUALITY IN CLINICAL RESEARCH
and when to just observe and not do anything. Not adhering to protocol is also considered scientific misconduct; but it has been suggested that poor judgement related to lack of knowledge may be a confounding factor. Researchers should be adequately trained to prepare them for their role in scientific studies and provided with closer supervision by senior research members. On the issue of data falsification, Rudy and Kerr (2000) reported media coverage about claims of scientific misconduct on a breast cancer study at the University of Pittsburgh. The misconduct involved one of the investigators who “falsified data on the eligibility of subjects” (p. 117–118). The study was also conducted at other sites. An evaluation process confirmed the accusations. The subjects recruited by the investigator were removed from the analyses of the study and the data were reanalyzed. Most of the clinical research studies that reported scientific misconduct both involved multiple investigators and institutions and are usually important studies that have the propensity for media coverage and public attention. As a result of these reports, regulators and agencies established offices such as the Office of Research Integrity (ORI), the Recombinant DNA Advisory Committee (RAC), and the OPRR to facilitate implementation of quality procedures in clinical trials. Most of the reported behaviors were identified during a review/evaluation process. It is fair to state that it is usually difficult to predict future challenges, especially when it relates to the intricacies of the clinical research process. However, the challenges, when identified during the experience/practice process, must be addressed immediately by implementing appropriate strategies. Scientific misconduct has been identified in clinical research. With the tendency for the media to exaggerate incidents for public consumption and regulators to overreact from public pressure, it is incumbent on the scientific community to be proactive in the process and establish internal structured processes that would stymie misconduct. By implementing quality evaluation processes, it is possible to detect scientific misconduct. It is the responsibility of the principal investigators and participating researchers to ensure that the scientific integrity and merit of the study are maintained.
QUALITY ISSUES, CLINICAL RESEARCH, AND HEALTH POLICY
Health policy decisions that affect clinical practice are embedded in evidence established through
237
research. Quality in clinical research is closely linked to cost containment and may determine appropriate allocation of grants for research. Although funding institutions and agencies require that specific criteria in grant applications be addressed, many funding agencies fail to provide follow-up on the results after the proposals have been deemed adequate or acceptable. The literature is clear on issues of infringement of the research process after money has been disbursed; and, therefore, researchers strongly advocate a continuous evaluation process instead of a onetime strategy (Arnot, Jepsen, & Kilama, 2000; Kefalides, 2000; Macrina, 1995; Noble-Adams, 1999a). This has become more important as the competition for research grants escalates and financial resources diminish (Volgelsang, 1999). The current trend by legislators is to provide more funds for clinical research, which suggests that an increase in competition for research grants by researchers seems to grow faster than the rate of the increase in research funds. The more researchers are motivated to conduct pragmatically applicable clinical research, the more regulators will prioritize funding based on societal/public needs. Funding agencies and regulators continue to rely on scientific research evidence to initiate health policy decisions; hence, the need to ensure that quality clinical research is conducted (Lohr et al., 1998; Prashker, 1996). Regulators also assess consistencies between research results and clinical recommendations to make informed decisions. Winther and Hole (1997) conducted a study to determine the scientific quality of 40 randomly selected clinical research projects. The indicators used to determine quality were: (1) Does the project contribute to new knowledge or can it be used as a control for results of previous studies? and (2) Is the protocol of good (study can be used and published without substantial alteration), inferior (study can be used and published after revisions are made), or unfit/unsatisfactory (the study should not be started nor published) quality? The results indicated that 80 per cent (n ⫽ 40) would contribute to new knowledge, 37.5 per cent (n ⫽ 40) were of good quality, 37.5 per cent (n ⫽ 40) of inferior quality, and 10 were unfit for use. Of the 40, 9 could be accepted for publication, 10 were in need of revision, and 7 were unsatisfactory/unfit for publication. In one of the studies, the investigators made some changes in the research process between the time the proposal was accepted and when the results were reported. The
238
NJIE AND THOMAS
investigators of this study identified the importance of an evaluation process before submission to the review board, and a postevaluation procedure to ensure that the same study proposal accepted was the same one maintained at the completion of the study. The study also determined that quality clinical research studies have the propensity to be cost effective in the interest of funding agencies and can also ensure that the ethical dimension of the research process is maintained. Even though the investigators indicated that pharmacology/pharmacotherapy trials were more likely to conduct clinical research with humans, a larger sample size and the inclusion of other types of clinical research projects would have lent the results more generalizability. It is also important to note that the investigators failed to operationally define “substantial alteration” on the evaluation guidelines and the other aspects of the research study that were evaluated. Operationally defining the quality indicators would have provided a more thorough and objective perspective of the evaluation process.
Implications for Clinical Practice and Health Policy
Interventional and quality clinical research studies have the propensity to positively impact on health policies. Regulators use the results as a basis for making decisions and its subsequent impact on clinical practice. The quality assurance process in clinical research is a catalyst for health policy changes. In an international study conducted by Grant, Cottrell, and Cluzeau (2000), a structured peer-review process was used to evaluate the impact of clinical research on practice. Data were collected from clinical guidelines based on diverse areas of disease management. The National Health Service (NHS) that determines the quality of clinical research studies in Great Britain usually evaluates these guidelines. One of the criteria for selection for review was citation of the article in clinical guidelines because the NHS deemed them quality research; hence, the reason these 2,501 articles were selected for review in which the guidelines were cited. The indicators used were the time the article was published to the time it was cited in a clinical guideline; the country of authorship; and the extent of clinical research cited in clinical guidelines. The indicator of interest here is the extent the research was cited in a guideline. Of the 2,501 articles evaluated, 2,043
(82 per cent) were found in research journals and 3% were used in clinical practice. It is important to note that the practice guidelines were “developed in and for the United Kingdom.” However, the implications for practice and health policy are obvious. Structured evaluations of clinical research studies identify and filter studies that are not practice oriented and have the potential to incur expenses that do not benefit the public. With stringent policies by regulators and funding agencies, clinical researchers must ensure quality studies through an evaluation process. This trend is becoming more attractive as financial resources for funding continue to be competitive. It also reinforces the need for researchers to be accountable for the responsibilities bestowed them (Gaughwin, 1999). One of the most important variables in health policy is the improvement of public health in a cost-effective manner. Regulators depend on quality clinical research data to direct them to appropriate and effective interventions for the public’s benefit. An objective, structured, scientific evaluation process must be instituted in research centers, universities, and other researchoriented agencies to promote facilitation of scientific integrity and ethical veracity. The ethical component has direct implications on health policy with regard to infringement on the rights and safety of the subjects, and the misuse of financial resources (Winther & Hole, 1997). Therefore, the potential risks to human subjects cannot be overemphasized. The ANA’s position on ethics and human rights contends to Promulgate in collaboration. . .both theoretical and practical. . . to address issues in ethics and human rights; develop and disseminate information about and advocate for public policy to assure that ethics and human rights are addressed; assure that short and longrange objectives regarding human rights will be addressed. . .(www.ana.org/ethics, p. 1).
The quality research audit undertaken by Karlawish et al. (1999) has the potential to mitigate the chances of ethical infringements. Secretary Shalala’s announcement will also help to ensure protection of subject participants in any research study. However, there is a dire need to implement internal scientific evaluation process in the various agencies and institutions to curb these mistakes. This will facilitate the early identification of discrepancies and the revisions made before external review. The evaluation process will serve as a verificational and validation process for quality clinical research studies.
239
QUALITY IN CLINICAL RESEARCH
Proposed Methodology
A fixed, prescriptive, perfect protocol to conduct a scientific evaluation process is not possible; however, a generic framework will assist individuals or groups in various institutions and agencies to evaluate their research studies before submitting them for external review. Although some reviews are conducted informally, it is important to institute structured reviews that would propagate objectivity and abate potential biases (Lindquist et al., 1995). An independent evaluation process reinforces accountability of the research participants (Emanuel et al., 2000) before it gets to the external reviewers. The author does not suggest that the proposed framework is new, but encourages researchers to conduct internal structured evaluation processes by using the general principles of the research process while incorporating recent Federal requirements. The extent and format for the evaluation process is left to the discretion of the research institutions and agencies. This article proposes a generic scientific evaluation framework as a mechanism to maintain quality in clinical research. There are three areas of foci: infrastructure, process, and outcome dimensions. The quality indicators on this evaluation framework will focus on the infrastructure and process dimensions. The outcome dimension is not addressed in evaluating clinical research because it is not applicable until after the results have been clinically implemented. The proposed method facilitates an unbiased and objective manner to address essential aspects of the research study. Kitson, Harvey, Hyndman, and Yerrell (1994) suggest specific characteristics to provide a guide to develop “good” quality criteria indicators (p. 157). These characteristics include statements that are relevant, understandable, measurable, stated in behavioral terms, and achievable; hence, the acronym RUMBA (Kitson et al, 1994). These characteristics are used as a guide to develop the scientific evaluation framework proposed here.
At the NIH, the pre-IRB scientific review committee members usually include the “Institute’s Scientific and Clinical Directors, and a staff member from the Institute’s extramural program” (NIH, 2000, p. 28). If other research institutions or agencies wish to adopt this scientific evaluation framework, the committee should include scientific and academic researchers, clinician(s), and ethics expert(s). The scientific and academic researchers evaluate the scientific integrity and merit of the study, the clinician(s) evaluate the study’s applicability to clinical practice, and the ethics experts evaluate the ethical component of the study. The exact number of team members to conduct the evaluation is left to the discretion of each institute and agency. This comprehensive evaluation process ascertains the scientific integrity, merit, and the clinical practice applicability of clinical research studies. Successful completion of the review process does not necessarily mean that the study will be successful nor that integrity and merit will be maintained. Evaluating the study before submitting the research proposal for external review (pre) and after the study has been conducted (post) may ensure quality through the whole research process. A point system scale method of scoring is used for this evaluation (see table 1). Table 2 shows the scientific evaluation framework for clinical research studies to be used by institutions and agencies to evaluate their clinical research studies. As Table 2 shows, the quality indicators are consistent with the general principles of the research process. Specific Federal requirements such as the mandate for signed consent/assent forms and training participating investigators are incorporated for evaluation. Each criterion is worth 1 point and the strengths and/or weaknesses can be explained in the “comments” section. The “point” section has two columns for the pre- and postevaluation. The total number of points is computed after each evaluation session (pre and post), and the recommendations are submitted as shown on the “Evaluation Scoring Chart” (table 1).
TABLE 1. Evaluation Scoring Chart Quality Evaluation
Description
Score Range Pre-evaluation
Score Range Postevaluation
Good quality Fair quality Poor quality/unacceptable
Minimum to no revisions necessary Some revisions necessary Major revisions and rewrite necessary
20–23 16–19 ⬍16
30–32 25–29 ⬍25
240
NJIE AND THOMAS
TABLE 2. Criteria to Evaluate Clinical Research Studies Point Content
Pre
Post
Comments
Evaluation of Infrastructure Investigators/resources/presentation Reports expertise of principal investigators and the other research members Documents congruency between the investigator’s expertise and the research objectives Clearly documents the role of each participating investigator Documents that participating investigators received training on the “Protection of Human Subjects” States the availability of human, material, and equipment resources Documents budget that is consistent with the resources cited Presents a clear, concise, and logical study Evaluation of Process Hypotheses/objectives/background Clearly states the aims/objectives/purposes/hypotheses of the study Reports relevancy of the research objectives to clinical practice Clearly reports the implications and significance of the study to nursing science and nursing (clinical) practice Critically evaluates research studies in the literature Shows originality of the study Methodology Uses appropriate instrument(s)/intervention(s) Reports validity and reliability of the instruments Uses an appropriate selection criteria for the subjects Provides clear steps for the data collection procedure or protocol Uses statistical analyses methods consistent with the research questions Human subjects Explains the study in simple terms on the consent (assent) form States rationale for choosing research subjects Explains clearly the potential risks and benefits of the study Indicates on the consent (assent) form that subjects can voluntarily withdraw without penalty Indicates that subjects’ confidentiality will be maintained Includes contact number(s) on the form for subjects to call for information Results/discussion* Documents evidence of signed and dated consent forms Shows consistency between the number of subjects and the number of signed and dated consent forms Presents the results accurately Shows consistency between the information collected and the information reported Shows strict adherence to all steps of the research procedure/protocol Analyzes the study’s applicability to nursing (clinical) practice Shows that the discussion of the study is based on the results Deduces/infers important points from the results Provides recommendations for future research studies *To be evaluated after the research study is completed. This is in addition to all the previously mentioned indicators.
Conclusion
Quality assurance/improvement is a vital process in clinical research and health policy. A quality clinical research study suggests that the research process is adhered to, participating researchers take accountability for their role in the research process, and the integrity and merit of the study is maintained. Instituting guidelines for clinical practice depends on accurate and appropriately interpreted data. Health policy regulators rely on the data produced by quality clinical research studies to make decisions; in turn, this validates the importance of quality in clinical research. To promote quality in clinical research, the mechanisms used in-
clude peer review, research audit, scientific evaluation, and research monitoring. A structured, scientific evaluation process has been proposed. Researchers are encouraged to use this process as a guide to ascertain quality clinical research, thereby maintaining the integrity and merit of the study. The framework contains quality indicators that focus on the infrastructure and process dimensions of the quality process. The team members of the evaluation process must be diverse to include scientific and academic researchers, clinicians, and ethics experts. Quality clinical research studies must be maintained to provide health policy regulators valid and credible
241
QUALITY IN CLINICAL RESEARCH
information on which to base their decisions that affect clinical practice. Acknowledgements The authors would like to extend their heartfelt thanks to Mr. Daniel O’Neal, Director of the Office of Scientific Policy and Public Liaison at the National Institutes of Health, for his suggestions on the overall manuscript and especially on the health
policy component. Also, to Mr. Mark Waldo at the National Institutes of Health for agreeing to read the paper; and to Dr. Abdoulaye Saine, Miami University in Oxford, Ohio, for reviewing the manuscript. This manuscript was written in partial fulfillment of a Summer Intramural Research Training Award under the preceptorship of Dr. Anne C. Thomas, Clinical Director of the Division of Intramural Research and Chief of the Health Promotion Laboratory at the National Institute of Nursing Research, National Institutes of Health.
References Arnot, D. E., Jepsen, S., & Kilama, W. (2000). Health research ethics in Africa. Parasitology Today, 16, 136-137. Becker, P. T. (1998). Scientific integrity and the journal review process. Research in Nursing and Health, 21, 283-284. Blancett, S. S. (Ed.) (1996). Integrity and misconduct in research. Journal of Nursing Administration, 26, 7-8. Christian, M. C., McCabe, M. S., Korn, E. L., Abrams, J. S., Kaplan, R. S., & Friedman, M. A. (1995). The National Cancer Institute audit of the national surgical adjuvant breast and bowel project protocol B-06. New England Journal of Medicine, 333, 1469-1474. Douglas, A. J., Jarvis, A., & Bloore, S. (1998). Monitoring of health research by ethics committees. New Zealand Medical Journal, 111, 79-81. Emanuel, E. J., Wendler, D., & Grady, C. (2000). What makes clinical research ethical. Journal of the American Medical Association, 283, 2701-2711. Fain, J. A. (1997). Maintaining scientific integrity in publications. The Diabetes Educator, 23, 232. Gaughwin, M. D. (Ed.) (1999). Health policy—the process and the politics. Medical Journal of Australia, 171, 56. Grant, J., Cottrell, R., & Cluzeau, F. (2000). Evaluating payback on biomedical research from papers cited in clinical guidelines: Applied biblometric study. British Medical Journal, 320, 1107-1111. The Center for Ethics and Human Rights: ANA [Online]. Available: http://www.ana.org/ethics (7/7/2000). Gallin, J. & Gottesman, M. Standards for clinical research within the NIH research program [On-line]. Available: http:// www.cc.nih.gov/ccc/clinicalresearch/ standards.html (6/12/2000). Hyde, J. C. (1998). Theoretical foundations of total quality management and continuous quality improvement: Or, does quality really have its own theory? Topics in Health Information Management, 18, 60-67. Joseph, K. S. (1998). Ethics in clinical research: Searching for absolutes. Canadian Medical Association Journal, 158, 1303-1305. Karlawish, J. H., Hougham, M. A., Stocking, C. B., & Sachs, G. A. (1999). What is the quality of the reporting of research ethics in publications of nursing home research? Research Ethics, 47, 76-81. Kefalides, P. T. (2000). Research on humans faces scrutiny; new policies adopted. Annals of Internal Medicine, 132, 513-516. Ketefian, S., & Lenz, E. R. (1995). Promoting scientific
integrity in nursing research, Part II: Strategies. Journal of Professional Nursing, 11, 263-269. Kitson, A., Harvey, G., Hyndman, S., & Yerrell, P. (1994). Criteria formulation and application: An evaluation framework. International Journal of Nursing Standard, 31, 155-167. Lenz, E. R., & Ketefian, S. (1995). Promoting scientific integrity in nursing research. Part I: Current approaches in doctoral programs. Journal of Professional Nursing, 11, 213219. Lindquist, R. D., Tracy, M. F., & Treat-Jacobson, D. (1995). Peer review of nursing research proposals. American Journal of Critical Care, 4, 59-65. Lohr, K. N., Eleazer, K., & Mauskopf, J. (1998). Health policy issues and applications for evidence-based medicine and clinical practice guidelines. Health Policy, 46, 1-19. Macrina, F. L. (1995). Scientific integrity: An introductory text with cases (pp. 1-14; 69-95). Washington, DC: American Society for Microbiology. Library of Congress Cataloging in Publication Data. Macrina, F. L., & Munro, C. L. (1995). The case study approach to teaching scientific integrity in nursing and the biomedical sciences. Journal of Professional Nursing, 11, 4044. McNelly, J. A. (1997). Taking the mystery out of research: Comparative analysis of research, quality improvement, and quality assurance. Orthopedic Nursing, 16, 68-69. Miller, F. G., Rosenstein, D. L., & DeRenzo, E. G. (1998). Professional integrity in clinical research. Journal of the American Medical Association, 280, 1449-1454. National Institutes of Health. (2000, January). Protomechanics: A guide to preparing and conducting a clinical research study. Bethesda, MD: The Clinical Center. Noble-Adams, R. (1999a). Ethics and nursing research 1: Development, theories and principles. British Journal of Nursing, 8, 888-892. Noble-Adams, R. (1999b). Ethics and nursing research 2: Examination of the research process. British Journal of Nursing, 8, 956-960. Pallikkathayil, L., Crighton, F., & Aaronson, L. S. (1998a). Issues in clinical nursing research: Balancing ethical quandaries with scientific rigor: Part I. Western Journal of Nursing Research, 20, 388-393. Pallikkathayil, L., Crighton, F., & Aaronson, L. S. (1998b). Issues in clinical nursing research: Balancing ethical quandaries with scientific rigor: Part II. Western Journal of Nursing Research, 20, 501-507.
242
Prashker, M. J. (1996). Health services research and policy studies. Current Opinion in Rheumatology, 8, 106-109. Rankin, M., & Esteves, M. D. (1997). Perceptions of scientific misconduct in nursing. Nursing Research, 46, 270-276. Rudy, E. B., & Kerr, M. E. (2000). Auditing research studies. Nursing Research, 49, 117-120. Ryan, K. J. (1999). Research misconduct in clinical research: The American experience and research. Acta Oncologica, 38, 93-97. Straus, S. E., & Sackett, D. L. (1998). Using research findings in clinical practice. British Medical Journal, 317, 339-342. Thomas, B. (1999). Research and audit in effective health services. Nursing Standard, 13, 40-42.
NJIE AND THOMAS
U.S. Department of Health and Human Services. (May 23, 2000a). Secretary Shalala bolsters protections for human research subjects. H.H.S News. U.S. Department of Health and Human Services. (May 23, 2000b). Protecting research subjects. H.H.S. Fact Sheet. Vogelsang, J. (1999). Quantitative research versus quality assurance, quality improvement, total quality management, and continuous quality improvement. Journal of Perianesthesia, 14, 78-81. Winther, F. O., & Hole, O. P. (1997). Scientific quality of clinical research. European Journal of Clinical Pharmacology, 51, 351-354.