- Email: [email protected]
Systematic Review Part 2. Conducting a Systematic Review
MacDonald-Jankowski, Dozier Asian J Oral Maxillofac Surg 2003;15:231-237. SPECIAL CONTRIBUTION
Systematic Review Part 2. Conducting a Systematic Review. David MacDonald-Jankowski,1 Marshall Dozier2 Department of Oral Biological and Medical Sciences, Faculty of Dentistry, University of British Columbia, Vancouver, Canada, and 2Erskine Medical Library, University of Edinburgh, Edinburgh, Scotland
1
Abstract In Part 1 of this series, published in the previous issue of Asian Journal of Oral and Maxillofacial Surgery, systematic review was introduced to the readers of this Journal. Part 2 describes the procedure, in terms of the stages of systematic review, causes of bias, and database search. Key words: Bias, Bibliographic databases, Embase, Grey literature, Medline, MeSH, Research question, Selection criteria, Systematic Review
Introduction Much of the detail of the early part of the systematic review (SR) process, has already been addressed in the maxillofacial literature.1 This paper covers the stages of SR, the most important causes of bias and how they may be minimised, the databases, and literature searching.
Essential Features of a Systematic Review SR must include a Materials and Methods section, which defines the strategies used to avoid bias. SR follows a sequence of steps, as follows: • posing the research question • selection criteria • literature search • appraisal of identified literature by selection criteria • analysis. This is described in greater detail in Table 1 (see Stroup et al for a more detailed checklist).2 The reviewer should document each of these steps in the published review, providing all the details required to reproduce the review.
Step 1. Research Question The principles of SR can be applied to many types of research question, but this paper will focus on Correspondence: David MacDonald-Jankowski, Department of Oral Biological and Medical Sciences, Faculty of Dentistry, University of British Columbia, 2199 Wesbrook Mall, Vancouver, BC V6T 1Z3, Canada. Asian J Oral Maxillofac Surg Vol 15, No 4, 2003
questions of therapeutic effectiveness. The research question, or questions, on which the search strategy is then based should have 4 components. They are frequently not fully expressed but readily implied from the text. The research questions for 3 different examples from the field of oral and maxillofacial surgery are analysed in Table 2.3-5 It is normal to use only a single ‘gold standard’ in SR. Creugars et al were unable to use the abovementioned 4 components in their SR on ‘single tooth implants’ because of the complexity of this subject.6 In such a situation, it is better to reduce the number of variables so that the above-mentioned 4 components can be used. This will lead to a marked reduction in the papers available for SR. Blanas et al’s 2290 ‘hits’ yielded 73 reports.3 These 73 reports were further reduced to 15 for their SR by applying selection or eligibility criteria. The overall effect of care in designing the research question and appointing those selection or eligibility criteria appropriate to it is to reduce heterogeneity.
Step 2. Appoint Selection or Eligibility Criteria Once the research questions have been defined, it is easier to determine the selection, or eligibility, criteria by which the body of relevant literature will be sifted.7 These criteria could be ‘inclusion’ or ‘exclusion’ (see Arrivé et al 8 who give a more extensive 231
Conducting a Systematic Review
subclassification of selection criteria). It is important to define the width of these criteria, since narrow criteria could exclude relevant studies and wide criteria Steps Sequence 1
Form the research question a. 4 components (i) Type of investigation/feature/prevalence (ii) Type of outcome (iii) Type of person/lesion (iv) Type of control/gold standard b. Reviewer’s knowledge of topic for SR
2
Appoint selection or eligibility criteria a. Inclusion or exclusion b. Width (i) Recall = ‘sensitivity’ (ii) Precision = ‘specificity’ c. Bias — consideration of different forms of BIAS
3
Formulate a search plan (i) Reviewer has little control over a. Keywords Publication bias b. Steps 4 to 7 Database bias c. Resources (ii) Reviewer has much control over Personnel — hand Language bias searching Inclusion bias Financial — translation Competing interests Reviewer bias
4
Plan for literature search for systematic review a. Databases (i) Electronic databases Set searching Medline PreMedline Old Medline Embase Non-set searching Science Citation Index Lilacs (ii) Elements of electronic search (in particular set-searching) Controlled subject headings MeSH Emtrees Free-text searching Boolean operator terms b. Reference or citation lists c. Hand searching of key journals d. Grey literature (i) Conference reports and theses (ii) Specialist, company, and government reports (iii) Canvas colleagues for unpublished studies
5.
Appraise the identified literature by selection criteria a. Apply predetermined selection criteria
6.
The analysis
7.
Formulation of recommendations
Table 1. Outline of the 7 steps for a systematic review.
could include studies using quite different methods, thereby increasing the risk of heterogeneity. The characteristics of a database search are ‘recall’ (or comprehensiveness) and ‘precision’, which are the counterparts of ‘sensitivity’ and ‘specificity’ of the diagnostic test. Therefore, a search with a high recall and low precision may produce an unmanageable number of studies, many of them irrelevant to the subject of the systematic review.9 Nevertheless, it is generally preferable to move more towards recall (sensitivity) because irrelevant reports can be discarded,10 whereas overemphasis on specificity may exclude relevant material. Bias Although a major aim of SR is to minimise bias, it is, like any other area of research, subject to it. There are many sources of bias in SR. The reviewer has little control over some of these such as: • ‘publication bias’ — positive results are more likely to be published than negative results. • ‘database bias’ — bias of the index compilers to English language journals, but against journals originating from the developed world. Although almost all of the 1861 Indian journals are published in English, only 30 are indexed.11 Those over which the reviewer does have control include the following: • ‘reviewer bias’ — inclusion criteria selected depend upon the reviewer’s knowledge or lack of it, for example, Gøtzsche and Olsen are epidemiologists, not radiologists, commenting on mammography.12 • ‘inclusion bias’ — a reviewer uses his/her knowledge of the literature to manipulate the criteria so as to exclude reports reaching conclusions with which he/she does not agree. This affects the SR’s validity.9 One way to avoid this is not to determine selection criteria before the literature search begins. • ‘language bias’/‘English only’ bias — this offends the fundamental principle of SR to systematically scrutinise and synthesise the whole world literature • ‘competing interests’ — if not expressly declared, these may be inferred by the author’s affiliation and
Reference
Type of feature or prevalence
Type of outcome
3
Does resection or nucleation
have a lower recurrence rate
4
Does use of the lingual flap
reduce damage to
5
Does hypermobility
increase
Table 2. The ‘research questions’ used in recent systematic reviews of oral and maxillofacial surgical topics.
232
Asian J Oral Maxillofac Surg Vol 15, No 4, 2003
MacDonald-Jankowski, Dozier
funding. A randomised trial suggests that ‘declaration of competing interests may have a significant effect on readers’ perceptions of the scientific credibility of published medical research’.13 Pichler and Beirne used 2 stages of inclusion criteria.4 The first stage covered all reports concerned with trigeminal nerve damage. The second stage was composed of 2 principle criteria — primary lingual nerve damage during third molar surgery and a follow-up of at least 6 months. This was then reduced further by 4 exclusion criteria — absence of results for each surgical method, absence of clinical objective sensory testing, fewer than 10 patients, and studies duplicating study populations. Pichler and Beirne also considered and excluded 3 reports that duplicated included study populations, which would have skewed the results if they had been admitted.4
Step 3. Formulating a Search Plan A literature search should be based on a written ‘search plan’ in the ‘Materials and Methods’ section so that the reader can determine whether the plan was appropriate and whether it is reproducible. This should cover steps 4 to 7 set out in Table 1. The important points to consider are as follows : • what the key words will be and whether they are ‘controlled subject headings’ or ‘free-text’. • which Boolean operators will be used and how • the database/s to be searched and, if relevant, the interface to be used • whether the search will be restricted to only 1 or more languages; Blanas et al considered only English language.3 However, because this could affect the SR’s results it should be clearly expressed. • the category and subject areas of journals to be hand-searched and what areas of the ‘grey literature’ are to be considered. Planning must also take account of the reviewers’ resources, since hand searching in particular is labour-intensive and time-consuming and therefore expensive.
Step 4. Literature Search The primary search will be based on electronic medical databases. The most widely used electronic databases have already been discussed elsewhere and we will generally focus on Medline and its PubMed interface as access to it has been made free by the USA government. Electronic database interfaces can be divided into those that allow ‘set searching’ and those that currently do not. ‘Set searching’ is a valuable facility for clear organisation of a search strategy. ‘Medline’ (Index Medicus on-line; National Library of Medicine14) is the most established, with approximately 9 million records from 3,900 journals published since 1960; 80% of the indexed articles are in English.15 Embase (Excerpta Medica on-line; Elsevier Science)16 has more than 5 million records from more than 4000 journals since 1974, of which the Asian Journal of Oral and Maxillofacial Surgery is one. Approximately 1000 journals are included in Embase, but not in Medline. Embase is notable for its coverage of pharmacology with very good indexing of European language journals.17 Embase has a shorter indexing delay (4 to 8 weeks) than Medline (3 to 6 months). Nevertheless, this has now been partly addressed by the inclusion of ‘in process’ records for articles, even before publication, based on information sent by publishers.17,18 The National Library of Medicine has also begun to address the incompleteness of Medline relative to printed Index Medicus which goes back to the 19th century by producing OldMedline. 18 Furthermore, it also provides free training materials for PubMed and other databases.19 Elements of Electronic Search The literature search is broadly similar to a routine literature search for clinical information (see Greenhalgh 17), except that it is systematic and rigorous. Literature search can use controlled subject headings (such as MeSH or Emtree of Medline and
Type of lesion, procedure, or structure
Type of control
of keratocysts compared with
enucleation alone with adjunctive therapy?
the lingual nerve during 3rd molar surgery compared with
the use of none retractor surgery?
degenerative change in the temporomandibular joint compared with
those with normal movement?
Asian J Oral Maxillofac Surg Vol 15, No 4, 2003
233
Conducting a Systematic Review
Embase, respectively), and free-text terms, combined using Boolean logical operaters (‘or’, ‘and’, or ‘not’). The search terms used are for both the subject(s) and the methods used in trials. Clinical Queries ‘methodology filters’ were developed by McMaster University for use in conjunction with PubMed;20 these combine MeSH, free-text and publication type searches to achieve optimum results. The underlying searches are complex, but fortunately the user is easily guided through the process.15 Controlled Subject Headings Subject headings are listed in a ‘thesaurus’ or ‘controlled vocabulary’ and are used by indexers to describe the content of articles in a consistent manner, so that the articles are subsequently easier to find. The indexers only use the listed subject headings and do not make up new ones in an ad hoc manner, which is why they are called ‘controlled’. MeSH (medical subject heading) terms amount to 14,000 specific terms and 180,000 synonyms. A MeSH term is assigned to each identified topic.21 It should be noted that the original purpose of MeSH for the print version ‘Index Medicus’ is to assist indexing and cataloguing, but it is integrated with search interfaces to facilitate searching Medline.22 The Emtree (Embase subject headings, using the metaphor of the tree to describe the hierarchical, broadto-narrow arrangement of subjects) thesaurus is built on more than 42,000 headings and 180,000 synonyms.16 Subject headings in Medline and Embase are organised in a browseable hierarchy. It is possible to navigate the index hierarchy to see where a given subject heading is placed, to see what broader subjects are above it, and what narrower subjects below. The indexers use the most appropriate subject headings, but these are not always the most relevant, appropriate, or precise for dentistry. As a result, some editors have developed their own keywords and thesaurus. In dentistry, it is therefore necessary to use broader subject headings than would at first seem appropriate. Some Medline interfaces, including PubMed, clearly indicate which terms are MeSH terms and also provide a ‘scope note’ that gives definitions as used by the indexers, year of introduction, and previous terms by which they were known. This means that the reviewer can determine the usefulness of his or her subject headings. 234
‘Exploding’ and ‘Free-text Searching’ When a subject heading and all narrower headings below it are relevant, it is easiest to ‘explode’ that broader heading. Rosenfield advises that MeSH searches should be supplemented by a ‘textword’ (a word that appears in titles and abstracts) search.23 This free-text search (a search using a ‘text-word’) does not account for the context in which the term appears, so may result in irrelevant hits, but the number of irrelevant hits will diminish as the subjects sets are combined with the Boolean operator ‘and’. In addition to performing a ‘free-text’ search for each subject heading, all variants, alternative spellings, or synonyms should be included — this can be facilitated by using a ‘wildcard’. The subject heading indices are revised regularly to reflect developments, but nonetheless, sometimes there is no appropriate heading, in particular for new or unusual terms, particularly in the smaller research fields such as dentistry and radiology. In these instances ‘free-text search’ is the only option. Ironically, this also applies to ‘systematic review’, whereas its subset ‘meta-analysis’ has been a MeSH term since 1989. Hand searching and Reference or Citation Lists. Although correct cataloguing is absolutely central to indexing of the databases, it is claimed that up to 50% of all Medline records are miscatalogued.17 A way to counter this error is to hand search all the journals relevant to the research question and carefully review the references of all papers identified in the databases.7,24 In addition, it is fruitful to develop the search strategy by examining the subject headings, titles, and abstracts of relevant hits for additional subject headings and other terms not used in the initial search strategy. This is the ’gold standard’ for systematic review.24 Clearly the value of bibliographic databases would improve if they become more complete and accurate. This could enhanced if authors ensured that MeSH terms appear not only in the ‘Keywords’, but also in the title or abstract.24 An otherwise SR-eligible report25 had been omitted from Pichler and Beirne’s4 SR because it did not include the 2 search terms they had used for their electronic database search. Asian J Oral Maxillofac Surg Vol 15, No 4, 2003
MacDonald-Jankowski, Dozier
Grey Literature The ‘grey literature’ refers to literature not published in commercially produced journals or books. Although dissertations and reports published by scientific committees will have been peer-reviewed, much of the rest, such as conference reports will not. Grey literature can also be identified on the SIGLE (System for Info on the Grey Literature in Europe) database.26 This is a multidisciplinary database covering the grey literature from 1976 to the present, but only 12% of its content is biomedical. There are various databases for theses and dissertations. ‘Dissertation Abstracts’ covers theses for doctorates and masters accepted by North American and European universities since 1861; there is a notable weighting towards the former.27 The ‘Index to Theses’ covers theses accepted by British and Irish universities since 1970.28
report is rejected, it could be useful to approach the report’s authors for clarification of uncertainties prior to making the final decision. It is important to record that this step has been considered and taken and to identify these reports. This was done by Pichler and Beirne, and they recorded the authors they had successfully contacted in an acknowledgements section.4 Pichler and Beirne found 542 papers in the database search and a further 197 concerned with trigeminal nerve damage when they reviewed the reference lists of the papers identified by the former.4 Of these 739 papers, only 85 were concerned with nerve damage occurring during third molar surgery, of which 51 were concerned with lingual nerve damage. Of these 51 papers, only 8 complied sufficiently with the exclusion criteria to be included in the SR.
Step 6. Analysis Saraswat has outlined the historical background on the role of the ‘internet in health care’ with some emphasis on the region served by the Asian Journal of Oral Maxillofacial Surgery’s readership, which supports 2 medical databases, ‘Indian Medlar Centre’ 30 and the Japanese UMIN (University Hospital Medical Information Network). 31 The former, among others, indexes 75 Indian medical journals and the latter lists more than 40 university hospitals and more than 700 medical and dental associations and institutions. 29
Another way of finding grey literature is to approach colleagues to determine whether they have any relevant unpublished studies that could be included in the SR. A problem with the grey literature is the increased risk of duplicating the same reports as those identified by the database search, as 50% proceed to full publication.7
Step 5. Appraise the Identified Literature by Selection Criteria Once the relevant literature has been identified the next step is to apply the predetermined selection criteria to include or exclude papers from the systematic review. Blind and independent assessment of the data has been advocated to avoid ‘reviewer bias’, which influences the acceptance or rejection of borderline reports. Before an otherwise acceptable Asian J Oral Maxillofac Surg Vol 15, No 4, 2003
The statistical techniques applied to an SR are dependent on the nature of the SR. If it is based on randomised controlled trials, a meta-analysis using a ‘forest plot’ is appropriate. Unfortunately, occasionally the number of reports is so limited and the differences of the study design are so great that statistical testing for bias and heterogeneity becomes impossible. Such an outcome should not by itself deter the authors of an SR from submitting it for consideration for publication as it can inspire others to research this topic focusing on rectifying such shortcomings. This in turn may result in a more useful subsequent SR. Therefore, as mentioned at the beginning of part 1, an SR reveals more accurately the true situation of the literature, even in an area that the clinician has considered to be already voluminous. This is clearly the case in Dijkstra et al’s SR which could only identify 3 reports that satisfied their selection criteria and provided about 100 cases and controls.5 From this small number, these authors were unable to provide firm conclusions and stated that ‘more rigorous studies are needed’.
Step 7. Formulation of Recommendations “SR…could help decision makers cope with information overload”.32 The conclusion of the SR, 235
Conducting a Systematic Review
such as that by Pichler and Beirne, may, on occasion, be controversial in some quarters, whereas Blanas et al’s conclusion is less so. Pichler and Beirne concluded that “the current literature does not support any significant advantage for the use of a lingual flap retractor to protect the lingual nerve during third molar removals; in fact it reveals an increased tendency towards temporary injury…”.4 Blanas et al concluded “that resection or enucleation with adjunctive therapy of keratocysts [our rubric] is associated with recurrence rates that are lower than those associated with enucleation”.3
Conclusion As all systematic reviews are retrospective, they are subject to systemic and random errors. Therefore, their quality depends on the extent to which error, particularly bias, has been minimised. SR is increasingly important in our current evidence-based culture as a means of identifying good practices and, perhaps as importantly, as a means of identifying areas requiring further, rigorous, research.
References 1. MacDonald-Jankowski DS, Dozier MF. Systematic review in diagnostic radiology. Dentomaxillofac Radiol 2001;30:78-83. 2. Stroup DF, Berlin JA, Morton SC, Olkin I, Williamson GD, Rennie D, Moher D, Becker BJ, Sipe TA, Thacker SB. Meta-analysis of observational studies in epidemiology: a proposal for reporting. Meta-analysis Of Observational Studies in Epidemiology (MOOSE) group. JAMA 2000; 283:2008-12. 3. Blanas N, Freund B, Schwartz M, Furst IM. Systematic review of the treatment and prognosis of the odontogenic keratocyst. Oral Surg Oral Med Oral Pathol Oral Radiol Endod 2000;90:553-558. 4. Pichler JW, Beirne OR. Lingual flap retraction and prevention of lingual nerve damage associated with third molar surgery: a sytematic review of the literature. Oral Med Oral Surg Oral Pathol Oral Radiol Endod 2001;91:395-401. 5. Dijkstra PU, Kropmans TJB, Stegenga B. The association between generalized joint hypermobility and temporomandibular joint disorders: s systematic review. J Dent Res 2002;81:158-163. 6. Creugers NHJ, Kreulen CM, Snoek PA, de Kanter 236
RJAM. A systematic review of single-tooth restorations supported by implants. J Dent 2000;28: 209-217. 7. Counsell C. Formulating questions and locating primary studies for inclusion in systematic review. In: Mulrow C, Cook D, editors. Systematic reviews: synthesis of best evidence for health care decisions. Philadelphia: American College of Physicians, 1998:67-79. 8. Arrivé L, Renard R, Carrat F, Belkacem A, Dahan H, Le Hir P, Monnier-Cholley L, Tubiana JM. A scale of methodological quality for clinical studies of radiologic examinations. Radiology 2000;217: 69-74. 9. Decks J, Glanville J, Sheldon T. Undertaking systematic reviews of research on effectiveness. CRD guidelines for those carrying out or commissioning reviews. Report 4. York: NHS Centre for Reviews and Dissemination; 1996. 10. Goodman C. Literature searching and evidence interpretation for assessing health care practice. SBU report No. 119. Stockholm: SBU — Swedish Council on Technology Assessment in Health Care, 1993. [Full-text online from http://www. sbu.se/admin/index.asp] 11. Egger M, Smith GD. Meta-analysis. Bias in location and selection of studies. BMJ 1997;316:61-66. 12. de Konig HJ. Assessment of nationwide cancerscreening programmes: commentary. Lancet 2000; 355:80-81. Comment on: Gøtzsche PC, Olsen O. Is screening for breast cancer with mammography justifiable. Lancet 2000;355:129-134. 13. Chaudhry S, Schroter S, Smith R, Morris J. Does declaration of competing interests affect readers preceptions? A randomised trial. BMJ 2002;325: 1391-1392. 14. Medline. United States National Library of Medicine database. http://www.ncbi.nlm.nih.gov/ PubMed/ 15. Allison JJ, Kiefe CI, Weissman NW, Carter J, Centor RM. The art and science of searching Medline to answer clinical questions. Finding the right number of articles. Int J Technol Assess Health Care 1999;15:281-296. 16. Embase. Amsterdam: Elsevier. http://www. elsevier.nl 17. Greenhalgh T. How to read a paper. The Medline database. BMJ 1997;315:180-183. Asian J Oral Maxillofac Surg Vol 15, No 4, 2003
MacDonald-Jankowski, Dozier
18. Information on OldMedline and PreMedline http://www.nlm.nih.gov/databases/databases. html 19. National Traininng Center and Clearinghouse http://nnlm.gov/train/ 20. The Health Information Research Unit at McMaster University: http://hiru.mcmaster.ca 21. Mulrow C, Cook D. Systematic reviews:synthesis of best evidence for health care decisions. Philadelphia: American College of Physicians; 1998. 22. Medical subject heading annotated alphabetic list. http://www.nlm.nih.gov/mesh/ 23. Rosenfeld RM. How to systematically review the medical literature. Otolaryngol Head Neck Surg 1994;115:53-63. 24. Dickerson K, Scherer R, Lefebvre C. Identifying relevant studies for systematic reviews. BMJ 1994; 309:1286-1291.
Asian J Oral Maxillofac Surg Vol 15, No 4, 2003
25. Ucok C. Lingual nerve injury. Oral Surg Oral Med Oral Pathol Oral Radiol Endod 2002;93: 2-3. 26. SIGLE (System for Information on Grey Literature in Europe). http://www.kb.nl/infolev/eagle content.htm 27. Dissertation Abstracts. http://www.umi.com/hp/ Support/DServices/products/da.htm 28. Index to Theses. http://www.theses.com/ 29. Saraswat A. Internet and health care. http:// indmed.nic.in/imcwebel02.html 30. Indian Medlars Centre. http://indmed.nic.in/ 31. UMIN (University Hospital Medical Information Network): http://www.umin.ac.jp/eng/ 32. Jadad AR, Moher M, Browman GP, Booker L, Sigouin C, Fuentes M, Stevens R. Systematic reviews and meta-analyses on treatment of asthma: critical evaluation. BMJ 2000;320:537-40.
237
Systematic Review Part 2. Conducting a Systematic Review. David MacDonald-Jankowski,1 Marshall Dozier2 Department of Oral Biological and Medical Sciences, Faculty of Dentistry, University of British Columbia, Vancouver, Canada, and 2Erskine Medical Library, University of Edinburgh, Edinburgh, Scotland
1
Abstract In Part 1 of this series, published in the previous issue of Asian Journal of Oral and Maxillofacial Surgery, systematic review was introduced to the readers of this Journal. Part 2 describes the procedure, in terms of the stages of systematic review, causes of bias, and database search. Key words: Bias, Bibliographic databases, Embase, Grey literature, Medline, MeSH, Research question, Selection criteria, Systematic Review
Introduction Much of the detail of the early part of the systematic review (SR) process, has already been addressed in the maxillofacial literature.1 This paper covers the stages of SR, the most important causes of bias and how they may be minimised, the databases, and literature searching.
Essential Features of a Systematic Review SR must include a Materials and Methods section, which defines the strategies used to avoid bias. SR follows a sequence of steps, as follows: • posing the research question • selection criteria • literature search • appraisal of identified literature by selection criteria • analysis. This is described in greater detail in Table 1 (see Stroup et al for a more detailed checklist).2 The reviewer should document each of these steps in the published review, providing all the details required to reproduce the review.
Step 1. Research Question The principles of SR can be applied to many types of research question, but this paper will focus on Correspondence: David MacDonald-Jankowski, Department of Oral Biological and Medical Sciences, Faculty of Dentistry, University of British Columbia, 2199 Wesbrook Mall, Vancouver, BC V6T 1Z3, Canada. Asian J Oral Maxillofac Surg Vol 15, No 4, 2003
questions of therapeutic effectiveness. The research question, or questions, on which the search strategy is then based should have 4 components. They are frequently not fully expressed but readily implied from the text. The research questions for 3 different examples from the field of oral and maxillofacial surgery are analysed in Table 2.3-5 It is normal to use only a single ‘gold standard’ in SR. Creugars et al were unable to use the abovementioned 4 components in their SR on ‘single tooth implants’ because of the complexity of this subject.6 In such a situation, it is better to reduce the number of variables so that the above-mentioned 4 components can be used. This will lead to a marked reduction in the papers available for SR. Blanas et al’s 2290 ‘hits’ yielded 73 reports.3 These 73 reports were further reduced to 15 for their SR by applying selection or eligibility criteria. The overall effect of care in designing the research question and appointing those selection or eligibility criteria appropriate to it is to reduce heterogeneity.
Step 2. Appoint Selection or Eligibility Criteria Once the research questions have been defined, it is easier to determine the selection, or eligibility, criteria by which the body of relevant literature will be sifted.7 These criteria could be ‘inclusion’ or ‘exclusion’ (see Arrivé et al 8 who give a more extensive 231
Conducting a Systematic Review
subclassification of selection criteria). It is important to define the width of these criteria, since narrow criteria could exclude relevant studies and wide criteria Steps Sequence 1
Form the research question a. 4 components (i) Type of investigation/feature/prevalence (ii) Type of outcome (iii) Type of person/lesion (iv) Type of control/gold standard b. Reviewer’s knowledge of topic for SR
2
Appoint selection or eligibility criteria a. Inclusion or exclusion b. Width (i) Recall = ‘sensitivity’ (ii) Precision = ‘specificity’ c. Bias — consideration of different forms of BIAS
3
Formulate a search plan (i) Reviewer has little control over a. Keywords Publication bias b. Steps 4 to 7 Database bias c. Resources (ii) Reviewer has much control over Personnel — hand Language bias searching Inclusion bias Financial — translation Competing interests Reviewer bias
4
Plan for literature search for systematic review a. Databases (i) Electronic databases Set searching Medline PreMedline Old Medline Embase Non-set searching Science Citation Index Lilacs (ii) Elements of electronic search (in particular set-searching) Controlled subject headings MeSH Emtrees Free-text searching Boolean operator terms b. Reference or citation lists c. Hand searching of key journals d. Grey literature (i) Conference reports and theses (ii) Specialist, company, and government reports (iii) Canvas colleagues for unpublished studies
5.
Appraise the identified literature by selection criteria a. Apply predetermined selection criteria
6.
The analysis
7.
Formulation of recommendations
Table 1. Outline of the 7 steps for a systematic review.
could include studies using quite different methods, thereby increasing the risk of heterogeneity. The characteristics of a database search are ‘recall’ (or comprehensiveness) and ‘precision’, which are the counterparts of ‘sensitivity’ and ‘specificity’ of the diagnostic test. Therefore, a search with a high recall and low precision may produce an unmanageable number of studies, many of them irrelevant to the subject of the systematic review.9 Nevertheless, it is generally preferable to move more towards recall (sensitivity) because irrelevant reports can be discarded,10 whereas overemphasis on specificity may exclude relevant material. Bias Although a major aim of SR is to minimise bias, it is, like any other area of research, subject to it. There are many sources of bias in SR. The reviewer has little control over some of these such as: • ‘publication bias’ — positive results are more likely to be published than negative results. • ‘database bias’ — bias of the index compilers to English language journals, but against journals originating from the developed world. Although almost all of the 1861 Indian journals are published in English, only 30 are indexed.11 Those over which the reviewer does have control include the following: • ‘reviewer bias’ — inclusion criteria selected depend upon the reviewer’s knowledge or lack of it, for example, Gøtzsche and Olsen are epidemiologists, not radiologists, commenting on mammography.12 • ‘inclusion bias’ — a reviewer uses his/her knowledge of the literature to manipulate the criteria so as to exclude reports reaching conclusions with which he/she does not agree. This affects the SR’s validity.9 One way to avoid this is not to determine selection criteria before the literature search begins. • ‘language bias’/‘English only’ bias — this offends the fundamental principle of SR to systematically scrutinise and synthesise the whole world literature • ‘competing interests’ — if not expressly declared, these may be inferred by the author’s affiliation and
Reference
Type of feature or prevalence
Type of outcome
3
Does resection or nucleation
have a lower recurrence rate
4
Does use of the lingual flap
reduce damage to
5
Does hypermobility
increase
Table 2. The ‘research questions’ used in recent systematic reviews of oral and maxillofacial surgical topics.
232
Asian J Oral Maxillofac Surg Vol 15, No 4, 2003
MacDonald-Jankowski, Dozier
funding. A randomised trial suggests that ‘declaration of competing interests may have a significant effect on readers’ perceptions of the scientific credibility of published medical research’.13 Pichler and Beirne used 2 stages of inclusion criteria.4 The first stage covered all reports concerned with trigeminal nerve damage. The second stage was composed of 2 principle criteria — primary lingual nerve damage during third molar surgery and a follow-up of at least 6 months. This was then reduced further by 4 exclusion criteria — absence of results for each surgical method, absence of clinical objective sensory testing, fewer than 10 patients, and studies duplicating study populations. Pichler and Beirne also considered and excluded 3 reports that duplicated included study populations, which would have skewed the results if they had been admitted.4
Step 3. Formulating a Search Plan A literature search should be based on a written ‘search plan’ in the ‘Materials and Methods’ section so that the reader can determine whether the plan was appropriate and whether it is reproducible. This should cover steps 4 to 7 set out in Table 1. The important points to consider are as follows : • what the key words will be and whether they are ‘controlled subject headings’ or ‘free-text’. • which Boolean operators will be used and how • the database/s to be searched and, if relevant, the interface to be used • whether the search will be restricted to only 1 or more languages; Blanas et al considered only English language.3 However, because this could affect the SR’s results it should be clearly expressed. • the category and subject areas of journals to be hand-searched and what areas of the ‘grey literature’ are to be considered. Planning must also take account of the reviewers’ resources, since hand searching in particular is labour-intensive and time-consuming and therefore expensive.
Step 4. Literature Search The primary search will be based on electronic medical databases. The most widely used electronic databases have already been discussed elsewhere and we will generally focus on Medline and its PubMed interface as access to it has been made free by the USA government. Electronic database interfaces can be divided into those that allow ‘set searching’ and those that currently do not. ‘Set searching’ is a valuable facility for clear organisation of a search strategy. ‘Medline’ (Index Medicus on-line; National Library of Medicine14) is the most established, with approximately 9 million records from 3,900 journals published since 1960; 80% of the indexed articles are in English.15 Embase (Excerpta Medica on-line; Elsevier Science)16 has more than 5 million records from more than 4000 journals since 1974, of which the Asian Journal of Oral and Maxillofacial Surgery is one. Approximately 1000 journals are included in Embase, but not in Medline. Embase is notable for its coverage of pharmacology with very good indexing of European language journals.17 Embase has a shorter indexing delay (4 to 8 weeks) than Medline (3 to 6 months). Nevertheless, this has now been partly addressed by the inclusion of ‘in process’ records for articles, even before publication, based on information sent by publishers.17,18 The National Library of Medicine has also begun to address the incompleteness of Medline relative to printed Index Medicus which goes back to the 19th century by producing OldMedline. 18 Furthermore, it also provides free training materials for PubMed and other databases.19 Elements of Electronic Search The literature search is broadly similar to a routine literature search for clinical information (see Greenhalgh 17), except that it is systematic and rigorous. Literature search can use controlled subject headings (such as MeSH or Emtree of Medline and
Type of lesion, procedure, or structure
Type of control
of keratocysts compared with
enucleation alone with adjunctive therapy?
the lingual nerve during 3rd molar surgery compared with
the use of none retractor surgery?
degenerative change in the temporomandibular joint compared with
those with normal movement?
Asian J Oral Maxillofac Surg Vol 15, No 4, 2003
233
Conducting a Systematic Review
Embase, respectively), and free-text terms, combined using Boolean logical operaters (‘or’, ‘and’, or ‘not’). The search terms used are for both the subject(s) and the methods used in trials. Clinical Queries ‘methodology filters’ were developed by McMaster University for use in conjunction with PubMed;20 these combine MeSH, free-text and publication type searches to achieve optimum results. The underlying searches are complex, but fortunately the user is easily guided through the process.15 Controlled Subject Headings Subject headings are listed in a ‘thesaurus’ or ‘controlled vocabulary’ and are used by indexers to describe the content of articles in a consistent manner, so that the articles are subsequently easier to find. The indexers only use the listed subject headings and do not make up new ones in an ad hoc manner, which is why they are called ‘controlled’. MeSH (medical subject heading) terms amount to 14,000 specific terms and 180,000 synonyms. A MeSH term is assigned to each identified topic.21 It should be noted that the original purpose of MeSH for the print version ‘Index Medicus’ is to assist indexing and cataloguing, but it is integrated with search interfaces to facilitate searching Medline.22 The Emtree (Embase subject headings, using the metaphor of the tree to describe the hierarchical, broadto-narrow arrangement of subjects) thesaurus is built on more than 42,000 headings and 180,000 synonyms.16 Subject headings in Medline and Embase are organised in a browseable hierarchy. It is possible to navigate the index hierarchy to see where a given subject heading is placed, to see what broader subjects are above it, and what narrower subjects below. The indexers use the most appropriate subject headings, but these are not always the most relevant, appropriate, or precise for dentistry. As a result, some editors have developed their own keywords and thesaurus. In dentistry, it is therefore necessary to use broader subject headings than would at first seem appropriate. Some Medline interfaces, including PubMed, clearly indicate which terms are MeSH terms and also provide a ‘scope note’ that gives definitions as used by the indexers, year of introduction, and previous terms by which they were known. This means that the reviewer can determine the usefulness of his or her subject headings. 234
‘Exploding’ and ‘Free-text Searching’ When a subject heading and all narrower headings below it are relevant, it is easiest to ‘explode’ that broader heading. Rosenfield advises that MeSH searches should be supplemented by a ‘textword’ (a word that appears in titles and abstracts) search.23 This free-text search (a search using a ‘text-word’) does not account for the context in which the term appears, so may result in irrelevant hits, but the number of irrelevant hits will diminish as the subjects sets are combined with the Boolean operator ‘and’. In addition to performing a ‘free-text’ search for each subject heading, all variants, alternative spellings, or synonyms should be included — this can be facilitated by using a ‘wildcard’. The subject heading indices are revised regularly to reflect developments, but nonetheless, sometimes there is no appropriate heading, in particular for new or unusual terms, particularly in the smaller research fields such as dentistry and radiology. In these instances ‘free-text search’ is the only option. Ironically, this also applies to ‘systematic review’, whereas its subset ‘meta-analysis’ has been a MeSH term since 1989. Hand searching and Reference or Citation Lists. Although correct cataloguing is absolutely central to indexing of the databases, it is claimed that up to 50% of all Medline records are miscatalogued.17 A way to counter this error is to hand search all the journals relevant to the research question and carefully review the references of all papers identified in the databases.7,24 In addition, it is fruitful to develop the search strategy by examining the subject headings, titles, and abstracts of relevant hits for additional subject headings and other terms not used in the initial search strategy. This is the ’gold standard’ for systematic review.24 Clearly the value of bibliographic databases would improve if they become more complete and accurate. This could enhanced if authors ensured that MeSH terms appear not only in the ‘Keywords’, but also in the title or abstract.24 An otherwise SR-eligible report25 had been omitted from Pichler and Beirne’s4 SR because it did not include the 2 search terms they had used for their electronic database search. Asian J Oral Maxillofac Surg Vol 15, No 4, 2003
MacDonald-Jankowski, Dozier
Grey Literature The ‘grey literature’ refers to literature not published in commercially produced journals or books. Although dissertations and reports published by scientific committees will have been peer-reviewed, much of the rest, such as conference reports will not. Grey literature can also be identified on the SIGLE (System for Info on the Grey Literature in Europe) database.26 This is a multidisciplinary database covering the grey literature from 1976 to the present, but only 12% of its content is biomedical. There are various databases for theses and dissertations. ‘Dissertation Abstracts’ covers theses for doctorates and masters accepted by North American and European universities since 1861; there is a notable weighting towards the former.27 The ‘Index to Theses’ covers theses accepted by British and Irish universities since 1970.28
report is rejected, it could be useful to approach the report’s authors for clarification of uncertainties prior to making the final decision. It is important to record that this step has been considered and taken and to identify these reports. This was done by Pichler and Beirne, and they recorded the authors they had successfully contacted in an acknowledgements section.4 Pichler and Beirne found 542 papers in the database search and a further 197 concerned with trigeminal nerve damage when they reviewed the reference lists of the papers identified by the former.4 Of these 739 papers, only 85 were concerned with nerve damage occurring during third molar surgery, of which 51 were concerned with lingual nerve damage. Of these 51 papers, only 8 complied sufficiently with the exclusion criteria to be included in the SR.
Step 6. Analysis Saraswat has outlined the historical background on the role of the ‘internet in health care’ with some emphasis on the region served by the Asian Journal of Oral Maxillofacial Surgery’s readership, which supports 2 medical databases, ‘Indian Medlar Centre’ 30 and the Japanese UMIN (University Hospital Medical Information Network). 31 The former, among others, indexes 75 Indian medical journals and the latter lists more than 40 university hospitals and more than 700 medical and dental associations and institutions. 29
Another way of finding grey literature is to approach colleagues to determine whether they have any relevant unpublished studies that could be included in the SR. A problem with the grey literature is the increased risk of duplicating the same reports as those identified by the database search, as 50% proceed to full publication.7
Step 5. Appraise the Identified Literature by Selection Criteria Once the relevant literature has been identified the next step is to apply the predetermined selection criteria to include or exclude papers from the systematic review. Blind and independent assessment of the data has been advocated to avoid ‘reviewer bias’, which influences the acceptance or rejection of borderline reports. Before an otherwise acceptable Asian J Oral Maxillofac Surg Vol 15, No 4, 2003
The statistical techniques applied to an SR are dependent on the nature of the SR. If it is based on randomised controlled trials, a meta-analysis using a ‘forest plot’ is appropriate. Unfortunately, occasionally the number of reports is so limited and the differences of the study design are so great that statistical testing for bias and heterogeneity becomes impossible. Such an outcome should not by itself deter the authors of an SR from submitting it for consideration for publication as it can inspire others to research this topic focusing on rectifying such shortcomings. This in turn may result in a more useful subsequent SR. Therefore, as mentioned at the beginning of part 1, an SR reveals more accurately the true situation of the literature, even in an area that the clinician has considered to be already voluminous. This is clearly the case in Dijkstra et al’s SR which could only identify 3 reports that satisfied their selection criteria and provided about 100 cases and controls.5 From this small number, these authors were unable to provide firm conclusions and stated that ‘more rigorous studies are needed’.
Step 7. Formulation of Recommendations “SR…could help decision makers cope with information overload”.32 The conclusion of the SR, 235
Conducting a Systematic Review
such as that by Pichler and Beirne, may, on occasion, be controversial in some quarters, whereas Blanas et al’s conclusion is less so. Pichler and Beirne concluded that “the current literature does not support any significant advantage for the use of a lingual flap retractor to protect the lingual nerve during third molar removals; in fact it reveals an increased tendency towards temporary injury…”.4 Blanas et al concluded “that resection or enucleation with adjunctive therapy of keratocysts [our rubric] is associated with recurrence rates that are lower than those associated with enucleation”.3
Conclusion As all systematic reviews are retrospective, they are subject to systemic and random errors. Therefore, their quality depends on the extent to which error, particularly bias, has been minimised. SR is increasingly important in our current evidence-based culture as a means of identifying good practices and, perhaps as importantly, as a means of identifying areas requiring further, rigorous, research.
References 1. MacDonald-Jankowski DS, Dozier MF. Systematic review in diagnostic radiology. Dentomaxillofac Radiol 2001;30:78-83. 2. Stroup DF, Berlin JA, Morton SC, Olkin I, Williamson GD, Rennie D, Moher D, Becker BJ, Sipe TA, Thacker SB. Meta-analysis of observational studies in epidemiology: a proposal for reporting. Meta-analysis Of Observational Studies in Epidemiology (MOOSE) group. JAMA 2000; 283:2008-12. 3. Blanas N, Freund B, Schwartz M, Furst IM. Systematic review of the treatment and prognosis of the odontogenic keratocyst. Oral Surg Oral Med Oral Pathol Oral Radiol Endod 2000;90:553-558. 4. Pichler JW, Beirne OR. Lingual flap retraction and prevention of lingual nerve damage associated with third molar surgery: a sytematic review of the literature. Oral Med Oral Surg Oral Pathol Oral Radiol Endod 2001;91:395-401. 5. Dijkstra PU, Kropmans TJB, Stegenga B. The association between generalized joint hypermobility and temporomandibular joint disorders: s systematic review. J Dent Res 2002;81:158-163. 6. Creugers NHJ, Kreulen CM, Snoek PA, de Kanter 236
RJAM. A systematic review of single-tooth restorations supported by implants. J Dent 2000;28: 209-217. 7. Counsell C. Formulating questions and locating primary studies for inclusion in systematic review. In: Mulrow C, Cook D, editors. Systematic reviews: synthesis of best evidence for health care decisions. Philadelphia: American College of Physicians, 1998:67-79. 8. Arrivé L, Renard R, Carrat F, Belkacem A, Dahan H, Le Hir P, Monnier-Cholley L, Tubiana JM. A scale of methodological quality for clinical studies of radiologic examinations. Radiology 2000;217: 69-74. 9. Decks J, Glanville J, Sheldon T. Undertaking systematic reviews of research on effectiveness. CRD guidelines for those carrying out or commissioning reviews. Report 4. York: NHS Centre for Reviews and Dissemination; 1996. 10. Goodman C. Literature searching and evidence interpretation for assessing health care practice. SBU report No. 119. Stockholm: SBU — Swedish Council on Technology Assessment in Health Care, 1993. [Full-text online from http://www. sbu.se/admin/index.asp] 11. Egger M, Smith GD. Meta-analysis. Bias in location and selection of studies. BMJ 1997;316:61-66. 12. de Konig HJ. Assessment of nationwide cancerscreening programmes: commentary. Lancet 2000; 355:80-81. Comment on: Gøtzsche PC, Olsen O. Is screening for breast cancer with mammography justifiable. Lancet 2000;355:129-134. 13. Chaudhry S, Schroter S, Smith R, Morris J. Does declaration of competing interests affect readers preceptions? A randomised trial. BMJ 2002;325: 1391-1392. 14. Medline. United States National Library of Medicine database. http://www.ncbi.nlm.nih.gov/ PubMed/ 15. Allison JJ, Kiefe CI, Weissman NW, Carter J, Centor RM. The art and science of searching Medline to answer clinical questions. Finding the right number of articles. Int J Technol Assess Health Care 1999;15:281-296. 16. Embase. Amsterdam: Elsevier. http://www. elsevier.nl 17. Greenhalgh T. How to read a paper. The Medline database. BMJ 1997;315:180-183. Asian J Oral Maxillofac Surg Vol 15, No 4, 2003
MacDonald-Jankowski, Dozier
18. Information on OldMedline and PreMedline http://www.nlm.nih.gov/databases/databases. html 19. National Traininng Center and Clearinghouse http://nnlm.gov/train/ 20. The Health Information Research Unit at McMaster University: http://hiru.mcmaster.ca 21. Mulrow C, Cook D. Systematic reviews:synthesis of best evidence for health care decisions. Philadelphia: American College of Physicians; 1998. 22. Medical subject heading annotated alphabetic list. http://www.nlm.nih.gov/mesh/ 23. Rosenfeld RM. How to systematically review the medical literature. Otolaryngol Head Neck Surg 1994;115:53-63. 24. Dickerson K, Scherer R, Lefebvre C. Identifying relevant studies for systematic reviews. BMJ 1994; 309:1286-1291.
Asian J Oral Maxillofac Surg Vol 15, No 4, 2003
25. Ucok C. Lingual nerve injury. Oral Surg Oral Med Oral Pathol Oral Radiol Endod 2002;93: 2-3. 26. SIGLE (System for Information on Grey Literature in Europe). http://www.kb.nl/infolev/eagle content.htm 27. Dissertation Abstracts. http://www.umi.com/hp/ Support/DServices/products/da.htm 28. Index to Theses. http://www.theses.com/ 29. Saraswat A. Internet and health care. http:// indmed.nic.in/imcwebel02.html 30. Indian Medlars Centre. http://indmed.nic.in/ 31. UMIN (University Hospital Medical Information Network): http://www.umin.ac.jp/eng/ 32. Jadad AR, Moher M, Browman GP, Booker L, Sigouin C, Fuentes M, Stevens R. Systematic reviews and meta-analyses on treatment of asthma: critical evaluation. BMJ 2000;320:537-40.
237