- Email: [email protected]
Googling endometriosis: a systematic review of information available on the Internet
Accepted Manuscript Googling Endometriosis: A Systematic Review of Information Available on the Internet Martin Hirsch, BM, Shivani Aggarwal, BSc, Claire Barker, BSc MSc, Colin J. Davis, Frcog MD, James M.N. Duffy, MBChB MRes PII:
S0002-9378(16)31987-1
DOI:
10.1016/j.ajog.2016.11.1007
Reference:
YMOB 11397
To appear in:
American Journal of Obstetrics and Gynecology
Received Date: 3 September 2016 Revised Date:
31 October 2016
Accepted Date: 3 November 2016
Please cite this article as: Hirsch M, Aggarwal S, Barker C, Davis CJ, Duffy JMN, Googling Endometriosis: A Systematic Review of Information Available on the Internet, American Journal of Obstetrics and Gynecology (2016), doi: 10.1016/j.ajog.2016.11.1007. This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.
Hirsch
ACCEPTED MANUSCRIPT 1 2 3 4 5 6 7
Googling Endometriosis: A Systematic Review of Information Available on the Internet.
8
Dentistry, Yvonne Carter Building, 58 Turner Street, Whitechapel, London, E1 2AB,
9
United Kingdom.
Martin HIRSCH1 BM, Shivani AGGARWAL 1BSc, Claire BARKER2,3BSc MSc, Colin J. DAVIS1 FRCOG MD, James M. N. DUFFY2,4 MBChB MRes Women’s Health Research Unit, Barts and the London School of Medicine and
RI PT
1
10
2
11
Quarter, University of Oxford, Oxford, OX1 2JD, United Kingdom.
12
3
13
Oxford, Oxford, OX1 2JD, United Kingdom
14
4
Nuffield Department of Primary Care Health Sciences, Radcliffe Observatory
SC
Radcliffe Women’s Health Patient and Public Involvement Group, University of
M AN U
Balliol College, University of Oxford, Oxford, OX1 3BJ, United Kingdom.
15 16
International Prospective Register of Systematic Reviews (PROSPERO):
17
CRD42016036134.
18
The authors report no conflicts of interest
20
This study received no financial support
21
TE D
19
Corresponding author
23
Dr James M N Duffy MBChB MRes BSc (Hons) PG HCL
24
Nuffield Department of Primary Care Health Sciences, Radcliffe Observatory
25
Quarter, University of Oxford, Oxford, OX1 2JD, United Kingdom.
26
[email protected]
27
+44 (0)1865 289298
28
@jamesmnduffy
AC C
29
EP
22
30
Abstract word count: 314
31
Main text word count: 3342
32 33 34 35
1
Hirsch
ACCEPTED MANUSCRIPT 36 37 38 39
Short title: Online Endometriosis Information
40 Condensation:
42
Online endometriosis information for patients is of poor quality.
RI PT
41
43 44
SC
45 46 47
M AN U
48 49 50 51 52
56 57 58 59 60 61 62 63 64 65 66 67 68 69
EP
55
AC C
54
TE D
53
2
Hirsch
ACCEPTED MANUSCRIPT 70 71 Objective
73
We aim to evaluate the credibility, quality, readability, and accuracy of online patient
74
information concerning endometriosis.
75
Data sources
76
We searched the five popular internet search engines: [1] aol.com; [2] ask.com; [3]
77
bing.com; [4] google.com; and [5] yahoo.com. We developed a search strategy in
78
consultation with patients with endometriosis, to identify relevant websites.
79
Website eligibility
80
Websites containing information related to endometriosis for women with
81
endometriosis or the public.
82
Website appraisal and synthesis methods
83
Two independent authors screened the search results. Websites were evaluated
84
using validated instruments across four domains, including assessments of: [1]
85
credibility (White Paper instrument; range 0-10); [2] quality (DISCERN instrument;
86
range 0-85); and [3] readability (Flesch-Kincaid instrument; range 0-100). Accuracy
87
was assessed by a prioritized criteria developed in consultation with healthcare
88
professionals, researchers, and women with endometriosis based upon the
89
European Society of Human Reproduction of Endometriosis guidelines (range 0 –
90
30). We summarized these data in diagrams, tables, and narratively.
91
Results
92
We identified 750 websites, of which 54 were included. Over a third of websites did
93
not attribute authorship and almost half the included websites did not report the
94
sources of information or academic references. No websites provided information
95
assessed as being written in plain English. A minority of websites were assessed as
96
high quality. A single website provided accurate information, evidentlycochrane.net.
97
Available information was, in general, skewed towards the diagnosis of
98
endometriosis. There were 16 credible websites, however the content limitations
99
were infrequently discussed. No website scored highly across all four domains.
AC C
EP
TE D
M AN U
SC
RI PT
72
100
Comment
101
In the unlikely event that a website reports high quality, accurate, and credible health
102
information it is typically challenging for a lay audience to comprehend. Healthcare
103
professionals, and the wider community, should inform women with endometriosis of
3
Hirsch
ACCEPTED MANUSCRIPT 104
the risk of outdated, inaccurate, or even dangerous information online. The
105
implementation of an Information Standard will incentivize providers of online
106
information to establish and adhere to codes of conduct.
107 International Prospective Register of Systematic Reviews (PROSPERO):
109
CRD42016036134.
111
Keywords
112
[1] Endometriosis
[5] Quality
113
[2] Patients
[6] Readability
114
[3] Accuracy
[7] Online information
115
[4] Credibility
[8] Systematic review
M AN U
116 117 118 119 120
125 126 127 128 129 130 131 132 133 134 135 136 137
EP
124
AC C
123
TE D
121 122
SC
110
RI PT
108
4
Hirsch
ACCEPTED MANUSCRIPT 138
Introduction
140
Endometriosis is benign gynecological disease which affects one in ten women of
141
reproductive age. It is characterized by pain and subfertility with associated reduced
142
quality of life [1]. The economic burden of endometriosis is of a similar magnitude to
143
other chronic diseases such as diabetes [2]. There is a paucity of high quality
144
research to guide clinical practice, this leads to unwarranted and unjustified
145
variations in patient care [3].
SC
RI PT
139
146
The internet is the source of health information as patients can access health
148
information quickly, conveniently and privately. There are an estimated 6.75 million
149
health searches daily in Google representing 4.5% of all searches performed [4].
150
There has been a rapid growth in the number of websites providing health
151
information with little or no governance [5]. Seven in ten adults regularly search for
152
an explanation and information on a new diagnosis or treatment [6-8]. Information
153
provided is commonly written at a high literacy level compounding the difficulties for
154
patients untrained in establishing whether the information is accurate. Exposure to
155
complex, ungoverned, unfounded health information which lacks expert editorial
156
supervision, could negatively affect patient understanding, compliance, and decision
157
making. This could lead to poorer health outcomes, including harm [9-13]. There are
158
no systematic reviews assessing the quality of online patient information pertaining
159
to endometriosis.
160
AC C
EP
TE D
M AN U
147
5
Hirsch
ACCEPTED MANUSCRIPT 161
We systematically assessed the accuracy, quality, readability, and credibility of
162
websites providing women with endometriosis and the public information regarding
163
the diagnosis and management of endometriosis.
164
RI PT
165
Materials and Methods
167
Sources
168
A protocol with explicitly defined objectives, criteria for website selection, and
169
approaches assessing outcome selection was developed and registered with the
170
International Prospective Register of Systematic Reviews (PROSPERO),
171
Identification number: CRD42016036134. This review was performed in accordance
172
with the Preferred Reporting Items for Systematic Reviews and Meta-analyses
173
(PRISMA) statement [14].
TE D
174
M AN U
SC
166
Website Selection
176
We developed a comprehensive search strategy in consultation with healthcare
177
professionals, researchers, and women with endometriosis. We used a keyword
178
analytic instrument, SEMrush (www.semrush.com), to inform our selection of search
179
terms. SEMrush provides analytical information related to search terms. We are
180
confident we identified and selected all search terms commonly used by women with
181
endometriosis. We used the following search terms: 1) endometriosis, 4,560,000
182
searches per annum; 2) endometriosis symptoms, 325,200 searches per annum; 3)
183
endometriosis treatment, 64,800 searches per annum; 4) endometriosis pain, 19,200
184
searches per annum; and 5) endometriosis diagnosis, 15,600 searches per annum.
185
We searched the most popular search engines including: 1) aol.com; 2) ask.com; 3)
AC C
EP
175
6
Hirsch
ACCEPTED MANUSCRIPT bing.com; 4) google.com; and 5) yahoo.com, during March 2016.
187
Individuals rarely examine more than the first three pages of a search [11]. We
188
therefore extracted the websites from the first three pages for each search term
189
within each search engine. Location services were disabled to eliminate
190
geographical bias.
RI PT
186
191
We organized the extracted websites and removed duplicates. Two reviewers (M.H.
193
and S.A.) independently screened the full content of websites to assess eligibility. All
194
data extraction was performed using piloted data extraction instruments. We pilot
195
tested each instrument using a representative sample of the websites to be
196
reviewed. This testing helped identify data that are missing from the form, or likely to
197
be superfluous. This allows authors trialing the form to provide feedback that certain
198
coding instructions are confusing or incomplete (e.g. a list of options may not cover
199
all situations). Any discrepancies between the reviewers were resolved by discussion
200
with a consensus required before the form is modified to avoid any
201
misunderstandings or later disagreements. We repeated the pilot testing on a new
202
set of websites where no major changes were needed after the first pilot testing [15].
M AN U
TE D
EP
AC C
203
SC
192
204
We included websites providing health information about endometriosis greater than
205
300 words in length. We excluded websites for the following reasons; 1) non-English
206
language; 2) inaccessible, for example password restricted; 3) aimed at a
207
professional audience, for example scientific publication; 4) excessive commercial
208
advertising (two or more commercial advertisements); and 5) content related solely
209
to the lived experience of endometriosis, for example a patient’s diary or blog.
210
7
Hirsch
ACCEPTED MANUSCRIPT 211
Those websites which met the criteria for inclusion were saved as a portable
212
document format for evaluation and data extraction by two independent authors (M.H
213
& S.A). M.H. and J.D. did not assess any websites they had previously contributed
214
too.
RI PT
215 Website Characteristics
217
Two reviewers (M.H. and S.A.) extracted the website characteristics independently
218
using a piloted data extraction sheet. Information extracted from each website
219
included country of origin, disease specific information, listed authors, and privacy
220
statements. Two reviewers (M.H. and S.A.) independently assessed each website
221
using validated instruments including assessments of 1) credibility assessed using
222
the White instrument [16] anchored between 0 (poor) and 10 (excellent) 2) quality
223
assessed using the DISCERN [17] instrument anchored between 0 (poor) and 85
224
(excellent), and 3) readability assessed using the Flesch-Kincaid [18] instrument
225
anchored between 0 (poor) and 100 (excellent). Discrepancies were resolved by
226
discussion.
M AN U
TE D
EP
227
SC
216
Quality Assessment
229
Two reviewers (MH and SA) underwent training in the use of the quality assessment
230
instruments. We assessed accuracy using a prioritized list of recommendations
231
included within the European Society of Human Reproduction and Embryology
232
endometriosis guidelines (ESHRE) [19]. The ESHRE guideline was selected for
233
comparison as this was objectively assessed to represent the highest quality
234
endometriosis guideline [20]. All recommendations were extracted by two authors
235
independently. Discrepancies were resolved by discussion. In consultation with
AC C
228
8
Hirsch
ACCEPTED MANUSCRIPT healthcare professionals, researchers and women with endometriosis, the
237
recommendations were scored as 1) critical for decision making, 2) important but not
238
critical for decision making and 3) not critical and not important for decision making.
239
Fifteen guideline recommendations were selected as statements critical for decision
240
making (appendix 1). The assessment of accuracy was standardized against
241
selected guideline recommendations. This approach has been utilized in similar
242
research studies [21].
244
Two reviewers (M.H and S.A) independently reviewed each website and using a
245
piloted standardized proforma assessing the accuracy of information. Each
246
recommendation was scored: 0 (if absent or incorrectly described), one (present and
247
incompletely described), or two (present and completely described). Accuracy
248
assessment was anchored between zero and 30. Discrepancies were resolved by
249
discussion. We classified websites with a score greater than or equal to 20 as
250
accurate.
M AN U
TE D
251
SC
243
RI PT
236
The website’s credibility was assessed by two reviewers independently using a
253
validated instrument, White [16]. This instrument, designed for consumers of health
254
information, provides a set of criteria that can be used to accurately and reliably
255
assess the quality of health information on the Internet. Credibility was assessed
256
using a ten point criteria: 1) source; 2) context; 3) currency; 4) utility; 5) editorial
257
review process; 6) hierarchy of evidence; 7) statement of original source; 8)
258
disclaimer, which included ownership, sponsorship, funding and advertising; 9)
259
omissions; and 10) feedback. Each criterion was scored 0 (absent) or one (present)
260
giving a score anchored between 0 to 10 [22]. Discrepancies were resolved by
AC C
EP
252
9
Hirsch
ACCEPTED MANUSCRIPT 261
discussion. We classified those websites with a score greater than or equal to seven
262
as credible.
263 The website’s quality was assessed by two reviewers independently using a
265
validated instrument, DISCERN [17], a validated instrument designed to assess the
266
quality of written information on treatment choices which can be applied to any
267
disease [7,17]. The DISCERN instrument offers a framework for the production,
268
evaluation, and screening of written consumer health information. This includes 16
269
questions assessed using a Likert scale anchored between one (do not agree) and
270
five (agree) [17]. Discrepancies were resolved by discussion. We classified those
271
websites as high (>53), moderate (27-52), and low (<27) quality.
M AN U
SC
RI PT
264
272
The website’s readability was assessed using the Flesch-Kincaid reading-ease test
274
[18]. This formula presents a score as a U.S. grade level, making it easier for teachers,
275
parents, librarians, and consumers of health information to judge the readability level
276
of various texts. The Flesch Kincaid score is generated from the following equation:
277
206.835 – 1.015 (total words / total sentences) – 84.6 (total syllables / total words)
278
(www.readability-score.com) [18]. The scores were anchored between 0 (complex
279
language) and 100 (simple language) and can be categorized by reading age or
280
educational status: 1) 90-100 (5th grade); 2) 80-90 (6th grade); 3) 70-80 (7th grade);
281
4) 60-70 (8th and 9th grade); 5) 50-60 (10th, 11th and 12th grade); 6) 30-50
282
(college); 7) 0-30 (college graduate). Discrepancies were resolved by discussion.
AC C
EP
TE D
273
283 284
A large-scale national assessment of the average reading level among Americans
285
performed by the National Center for Education Statistics found that the typical
10
Hirsch
ACCEPTED MANUSCRIPT American reads between a 7th and 8th grade level [23]. It is recommended that
287
online health information should not exceed the level of American 7th grade writing
288
and reading [24]. We therefore expected websites to have a readability score at or
289
below the level of American education 7th Grade (>70) to be deemed appropriate for
290
a patient and public audience.
RI PT
286
291 Analysis
293
The website characteristics and assessments were summarized in tabular form and
294
presented with descriptive statistics within summary tables and diagrams.
SC
292
M AN U
295 296 297
Results
299
The search strategy identified 750 websites which were assessed for eligibility. We
300
screened 211 websites following the exclusion of 539 duplicate websites. Two
301
authors independently applied an inclusion and exclusion criteria when screening the
302
websites. We included 54 websites in our final assessment (figure 1, table 1).
EP
AC C
303
TE D
298
304
Website characteristics
305
Twenty-one (39%) websites did not report the authors and 25 (46%) of websites did
306
not reporting sources of information or academic references. The majority of
307
included websites were published in the United Kingdom (25 websites; 46%). All
308
websites presented structured content. Almost two thirds of the websites reported a
309
privacy statement (38 websites; 70%) (table 1).
310
11
Hirsch
ACCEPTED MANUSCRIPT Accuracy
312
A single website provided accurate information, evidentlycochrane.net. The median
313
accuracy of included websites was 5 (Interquartile range [IQR] 4 – 7). Included
314
websites contained limited information (Table 1), skewed towards the diagnosis of
315
endometriosis. Information pertaining to the medical or surgical management of pain
316
or infertility associated with endometriosis were poorly represented. The most
317
commonly reported recommendation, “Clinicians should consider the diagnosis of
318
endometriosis in the presence of gynecological symptoms such as: dysmenorrhea,
319
non-cyclical pelvic pain, deep dyspareunia, infertility, fatigue in the presence of any
320
of the above”, was described by four fifths of included websites (43 websites, 80%).
321
The least frequently described recommendations, described by a small minority of
322
included websites (3 websites; 6%) were: 1) “In infertile women with endometriosis,
323
clinicians may offer treatment with assisted reproductive technologies after surgery,
324
since cumulative endometriosis recurrence rates are not increased after controlled
325
ovarian stimulation for IVF/ICSI.” 2) “Clinicians [should] inform women with
326
endometriosis requesting information on their risk of developing cancer that 1) there
327
is no evidence that endometriosis causes cancer, 2) there is no increase in overall
328
incidence of cancer in women with endometriosis, and 3) some cancers (ovarian
329
cancer and non-Hodgkin’s lymphoma) are slightly more common in women with
330
endometriosis”. The delivery of inaccurate, outdated or dangerous information
331
remains prevalent in websites. Inaccuracies include: 1) “Your specialist may also
332
suggest flushing out your blocked fallopian tubes. This procedure is an alternative to
333
surgery and is usually successful” website ID 1. Routine tubal flushing is used in
334
diagnostic evaluation of tubal patency and it is not recommended therapeutic
335
approach [25]. 2) “The only reliable way to confirm the presence of the disease is by
AC C
EP
TE D
M AN U
SC
RI PT
311
12
Hirsch
ACCEPTED MANUSCRIPT visually inspecting the abdominal organs by a procedure called a laparoscopy”
337
Website ID 20. There are many difficulties associated with visually confirming
338
endometriosis. The most reliable way to diagnose endometriosis is laparoscopy,
339
biopsy and histopathological examination. Visual diagnosis is no longer
340
recommended [19]. 3)”It is suspected that between 10-20% of reproductive aged
341
women have the disease.”
342
Website ID 20. The estimated prevalence within the general population is up to 10%
343
[19].
SC
RI PT
336
344 Credibility
346
Credibility was defined as a score equal to or greater than seven. Sixteen websites
347
(29%) were assessed as credible. The median credibility of included websites was 5
348
(IQR 2 - 8.8). The highest scoring criteria included context relevant to the disease
349
and originality with all websites fulfilling these criteria. The least frequently described
350
area of credibility was the discussion of content limitations which was reported by
351
one website (table 1).
TE D
EP
352
M AN U
345
Quality assessment
354
Thirteen websites (24%) were assessed to be high quality, 40 (74%) websites were
355
assessed to be of moderate quality, and a single website (2%) was assessed as low
356
quality. The highest scoring criteria included describing aims (median = 5; IQR 3-4)
357
and being unbiased (median 5; IQR 4-5). Websites typically did not describe the
358
consequences of no treatment (median 1; IQR 1-1).
AC C
353
359 360
Readability
13
Hirsch
ACCEPTED MANUSCRIPT
All included websites were assessed as fairly difficult to read (10th, 11th, and 12th
362
grade), difficult to read (college), or very difficult to read (college graduate). The
363
median readability score was 38.2 (IQR 30.7 – 48.0), indicating an average
364
educational status of a college student would be required to understand the written
365
content (table 1 and 2). Forty-five websites (83%) presented written information at a
366
level at or above college standard.
RI PT
361
367
There were no substantial discrepancies between authors in the data extraction of
369
quantitative parameters and we observed very high interrater agreement.
M AN U
SC
368
370 371
Comment
373
Summary
374
There are no websites which provide high quality, accurate, and credible health
375
information pertaining to endometriosis. Currently, websites contain limited amounts
376
of information which are skewed towards the diagnosis of endometriosis. In the
377
unlikely event that a website reports high quality, accurate, and credible health
378
information, it is typically written in language that is challenging for a lay audience to
379
comprehend.
380
AC C
EP
TE D
372
381
Strengths and Weaknesses
382
To our knowledge, this is the first study to examine the quality, credibility, accuracy,
383
and readability of patient focused online information pertaining to the diagnosis and
384
management of endometriosis. We followed a robust, prospective systematic review
14
Hirsch
ACCEPTED MANUSCRIPT method with validated instruments to assess the information presented. We
386
evaluated individual websites using four validated instruments in a systematic
387
process, independently performing all assessments in duplicate. We involved women
388
with endometriosis, to inform the research question, design and delivery of the
389
research study, and its dissemination. All reviewers underwent recommended
390
training prior to commencing the study.
391
RI PT
385
This study is not without limitations. Limiting the search to the first three pages may
393
have resulted in the exclusion of potentially eligible websites, however only 2.6% of
394
people search past Googles’ third page (www.protofuse.com). Included websites
395
were only written in English language, limiting the generalizability of our findings.
396
The search was conducted while computer location services were disabled, however
397
there may have been regional differences in search results, out of the authors
398
control, which account for the predominance of British websites. We designed and
399
registered this systematic review prospectively with a pre-defined inclusion criteria
400
and analysis plan. There are few scientific publications which evaluate online
401
information for patients allowing limited precedent to guide our methods. We
402
observed diminishing returns, however this was not quantified. All websites were
403
designed and managed within high resource countries. This limits the applicability of
404
this research to inform low resource settings. We did not calculate weighted kappa to
405
explore agreement between authors as the statistical level of agreement required in
406
health research is unclear [26]. This evaluation is not currently recommended by the
407
Cochrane Collaboration [15]. We could have conducted in-depth qualitative
408
interviews of women with endometriosis to explore their satisfaction with reading
AC C
EP
TE D
M AN U
SC
392
15
Hirsch
ACCEPTED MANUSCRIPT 409
individual websites and evaluate the correlation with accuracy, credibility, quality and
410
readability.
411 412
Interpretation
RI PT
413
As clinicians we must be aware that patients are increasingly seeking unregulated
415
health information online which shapes opinions and treatment choices. The
416
essence of modern clinical consultations is changing from a reliance on face-to-face
417
interaction to information gathering online prior to seeking professional opinion. In
418
the United States of America, there are over 400,000 endometriosis searches per
419
month in Google alone. We have demonstrated that individual websites are
420
frequently incomplete, inaccurate, and poorly written. This is a barrier to patient
421
education and results in those vulnerable patients who seek reliable information
422
being misinformed. This is of greater importance to non-expert patients (majority)
423
who may be less able to evaluate the reliability of online information and be
424
susceptible to the bias and inaccuracies contained within. These forays into online
425
information gathering can lead to a breakdown in doctor patient relationships.
426
Inaccurate online health information can lead to clinicians advocating guideline-
427
supported recommendations different from those read on “reputable” online sources.
428
This mismatch of information can lead to a breakdown in trust in the clinician-patient
429
relationship.
AC C
EP
TE D
M AN U
SC
414
430 431
A review conducted by the United States Office of Disease Prevention and Health
432
Promotion (ODPHP) concluded that the potential for harm from inaccurate online
433
information is significant [27]. Harm can be: 1) physical, from inappropriate
16
Hirsch
ACCEPTED MANUSCRIPT
treatments, adverse effects, or untreated disease; 2) emotional, from anxiety or false
435
hope arising from inaccurate diagnostic, prognostic, or therapeutic information; 3)
436
financial, costs incurred from unnecessary purchase of ineffective health services or
437
products [27]. The ODPHP concluded that the Internet is critical to disease
438
prevention, health promotion, and health care because of the increasing amount of
439
information and services available via the internet. This included a key objective to
440
increase the quality of online health information [28].
SC
441
RI PT
434
The readability of a website is an essential facet of online information. Information
443
presented at a standard above patients’ comprehension will limit its ability to inform
444
the patient. Healthcare professionals should be aware that there is very limited
445
information available to women with endometriosis with basic levels of literacy
446
(indicates skills necessary to perform simple and everyday literacy activities), and
447
therefore directing them to online information is of limited value in informing decision
448
making.
TE D EP
449
M AN U
442
Many online information rating systems use proxy markers for quality that do not
451
consider the needs and opinions of patients and the public. Meric and colleagues
452
[29], determined website popularity did not correlate well with traditional standards of
453
website quality. Quality of online information is crucial as patients want to know
454
about the risks, benefits, and uncertainty associated with diagnostic and therapeutic
455
options. This information must be accurate to ensure that patients seeking
456
information are gaining correct and complete information about the disease from up
457
to date scientific evidence. Without access to good quality information, patients are
458
unable to make informed choices about their treatment.
AC C
450
17
Hirsch
ACCEPTED MANUSCRIPT 459 Recommendation(s)
461
Healthcare professionals and the wider medical community are increasingly quizzed
462
by patients regarding health information found online. It is essential that healthcare
463
professionals acknowledge their position of responsibility and proactively inform
464
women with endometriosis about the risk of outdated, inaccurate, or even dangerous
465
information online. Interactive consultations using online clinical practice guidelines
466
such as those produced by the American College of Obstetrics and Gynecology [30]
467
or the Society of Obstetricians and Gynaecologists of Canada [31] can provide the
468
basis for clear, concise, evidence based management discussions. Following
469
consultations, patients should be sign posted towards higher quality and more
470
reliable sources of online information to answer questions they may have forgotten to
471
ask during their limited consultation time.
M AN U
SC
RI PT
460
TE D
472
While it may sound unrealistic to regulate health information on the internet, codes of
474
conduct have been developed and implemented. The Health on the Net Foundation,
475
based in the United States, provides accreditation to websites, which meet pre-
476
defined standards related to readability, accessibility, and accuracy [18]. The
477
Information Standard, based in the United Kingdom, assesses online health
478
information to ensure the information is clear, accurate, balanced, evidence-based,
479
and up-to-date. Information produced by the Royal College of Obstetricians and
480
Gynaecologists is accredited by this information standard
481
(https://www.england.nhs.uk/tis/).
AC C
EP
473
482 483
We acknowledge that regulating health information on the internet has inherent
18
Hirsch
ACCEPTED MANUSCRIPT
difficulties as online authors are not bound by the same codes of practice as licensed
485
healthcare professionals. The implementation of a robust Information Standard
486
internationally will incentivize providers of online information to establish and adhere
487
to codes of conduct ensuring an improvement in the quality of online information.
488
Healthcare professionals and professional bodies should direct women with
489
endometriosis towards higher quality, more reliable sources of online information. In
490
general, websites who comply with an Information Standard, should be prioritized.
RI PT
484
SC
491
The internet will continue to increase its role as a provider of online health
493
information. The media by which health information is transferred from source to
494
patient should not compromise the fundamental features of accuracy, credibility,
495
quality and readability. It would not be tolerated if a healthcare professional were
496
delivering sub-standard information in a face-to-face consultation. A strategy is
497
required to improve the standard of online information for women with endometriosis
498
with evident need for the development of patient focused online information with a
499
robust evidence base. The translation of research from trials or systematic reviews
500
into online sources has a direct pathway currently being delivered by Cochrane in
501
the form of Evidently Cochrane summaries. These webpages summarize Cochrane
502
systematic reviews into patient focused bite size pieces of information [32].
504
TE D
EP
AC C
503
M AN U
492
505
Conclusion
506
In the unlikely event that a website reports high quality, accurate, and credible health
507
information it is typically challenging for a lay audience to comprehend. Healthcare
508
professionals, and the wider community, should inform women with endometriosis of
19
Hirsch
ACCEPTED MANUSCRIPT 509
the risk of outdated, inaccurate, or even dangerous information online. Providers of
510
online information should engage with established codes of conduct, such as the
511
Information Standard.
512
RI PT
513 514 515
Acknowledgements
516
The authors would to acknowledge the contribution of the International Collaboration
518
to Harmonise Outcomes and Measures for Endometriosis. Steering Group: Dr
519
James M N Duffy (Balliol College, University of Oxford, Oxford, UK); Cindy Farquhar
520
(Department of Obstetrics & Gynecology, University of Auckland, Auckland, New
521
Zealand); Dr Martin Hirsch (Women’s Health Research Unit, Queen Mary University
522
of London, London, UK); Neil Johnson (University of Adelaide, Adelaide, Australia);
523
and Khalid Khan (Women’s Health Research Unit, Queen Mary University of
524
London, London, UK).
M AN U
SC
517
526 527 528
References
TE D
525
[1] Duffy JMN, Arambage K, Correa FJS, Olive D, Farquhar C, Garry R.
530
Laparoscopic surgery for endometriosis. Cochrane Database of Systematic Reviews
531
2014, Issue 4. Art. No.: CD011031.
AC C
532
EP
529
533
[2] Simoens S, Dunselman G, Dirksen C, Hummelshoj L, Bokor A, Brandes I, et al.
534
The burden of endometriosis: costs and quality of life of women with endometriosis
535
and treated in referral centres. Hum Reprod. 2012 May;27(5):1292-9.
536 537
[3] Hirsch M, Duffy JM, Kusznir JO, Davis CJ, Plana MN, Khan KS. Variation in
538
outcome reporting in endometriosis trials: a systematic review. Am J Obstet Gynecol.
539
2016 Apr;214(4):452-64.
540
20
Hirsch
ACCEPTED MANUSCRIPT 541
[4] Eysenbach G, Kohler C. What is the prevalence of health-related searches on the
542
World Wide Web? Qualitative and quantitative analysis of search engine queries on
543
the internet. AMIA Annu Symp Proc 2003;225–229.
544 [5] Mackey TK, Liang BA, Kohler JC, Attaran A. Health domains for sale: the need
546
for global health Internet governance.J Med Internet Res. 2014 Mar 5;16(3):e62.
RI PT
545
547
[6] Handfield B, Turnbull S, Bell RJ. What do obstetricians think about media
549
influences on their patients? Aust N Z J Obstet Gynaecol 2006;46:379–83.
550
SC
548
[7] Kaicker J, Debono VB, Dang W, Buckley N, Thabane L. Assessment of the
552
quality and variability of health information on chronic pain websites using the
553
DISCERN instrument. BMC Med 2010;8:59.
M AN U
551
554
[8] Hesse BW, Nelson DE, Kreps GL, Croyle RT, Arora NK, Rimer BK, et al. Trust
556
and sources of health information: the impact of the Internet and its implications for
557
health care providers: findings from the first Health Information National Trends
558
Survey. Arch Intern Med 2005;165:2618–24.
TE D
555
559 560
[9] Cline RJ, Haynes KM. Consumer health information seeking on the Internet: the
561
state of the art. Health Educ Res 2001;16:671–92.
EP
562
[10] Berkman ND, Sheridan SL, Donahue KE, Halpern DJ, Crotty K. Low health
564
literacy and health outcomes: an updated systematic review. Ann Intern Med
565
2011;155:97–107.
566
AC C
563
567
[11] Sacchetti P, Zvara P, Plante MK. The Internet and patient education– resources
568
and their reliability: focus on a select urologic topic. Urology 1999;53:1117–20.
569 570
[12] McMullan M. Patients using the internet to obtain health information: how this
571
affects the patient-health professional relationship. Patient Educ Couns 2006;63(1–
572
2):24–28
573 574
[13] Benigeri M, Pluye P. Shortcomings of health information on the internet. Health
21
Hirsch
ACCEPTED MANUSCRIPT 575
Promot Int 2003;18(4):381–386
576 577
[14] Moher D, Liberati A, Tetzlaff J, Altman DG, The PRISMA Group. Preferred
578
Reporting Items for Systematic Reviews and Meta-Analyses: The PRISMA
579
Statement. BMJ 2009;339:b2535.
RI PT
580 581
[15] Higgins JPT, Green S (editors). Cochrane Handbook for Systematic Reviews of
582
Interventions Version 5.1.0 [updated March 2011]. The Cochrane Collaboration, 2011.
583
Available from www.cochrane-handbook.org.
SC
584 585
[16] Ambre J, Guard R, Perveiler FM, Renner J, Rippen H (1997) White paper:
586
Criteria for assessing the quality of health information on the internet.
M AN U
587 588
[17] Charnock D, Shepperd S, Needham G, Gann R. DISCERN: an instrument for
589
judging the quality of written consumer health information on treatment choices. J
590
Epidemiol Community Health 1999;53:105–111.
591
[18] Kincaid JP, Fishburne RP Jr, Rogers RL, Chissom BS. Derivation of new
593
readability formulas (Automated Readability Index, Fog Count and Flesch Reading
594
Ease Formula) for Navy enlisted personnel. Memphis, TN: Naval Air Station; 1975.
595
TE D
592
[19] Dunselman GA, Vermeulen N, Becker C, et al. ESHRE guideline: management
597
of women with endometriosis. European Society of Human Reproduction and
598
Embryology. Hum Reprod. 2014 Mar;29(3):400-12.
AC C
599
EP
596
600
[20] Hirsch M, Begum MR, Paniz É, Barker C, Davis CJ, Duffy JMN. Diagnosis and
601
management of endometriosis: a systematic review of international and national
602
guidelines. Submitted for publication to Obstetrics and Gynecology, October 2016.
603 604
[21] Fisher JH, O'Connor D, Flexman AM, Shapera S, Ryerson CJ. Accuracy and
605
Reliability of Internet Resources for Information on Idiopathic Pulmonary Fibrosis.
606
Am J Respir Crit Care Med. 2016 Jul 15;194(2):218-25.
607 608
[22] Tirlapur SA, Leiu C, Khan KS. Quality of information on the internet related to
22
Hirsch
ACCEPTED MANUSCRIPT
bladder pain syndrome: a systematic review of the evidence. Int Urogynecol J. 2013
610
Aug; 24(8):1257–62.
611
[23] National Center for Education Statistics. National Assessment of Adult Literacy
612
(NAAL). 2003. http://nces.ed.gov/naal. Accessed August 6, 2008.
613
[24] Bernstam EV, Shelton DM, Walji M, Meric-Bernstam F. Instruments to assess
614
the quality of health information on the world wide web: what can our patients
615
actually use? Int J Med Inform 2005;74(1):13–19
RI PT
609
616
[25] Duffy JMN, Johnson N, Ahmad G, Watson A. Postoperative procedures for
618
improving fertility following pelvic reproductive surgery. Cochrane Database of
619
Systematic Reviews 2009, Issue 2. Art. No.: CD001897.
620
M AN U
SC
617
[26] McHugh ML. Interrater reliability: the kappa statistic. Biochem Med (Zagreb).
622
2012 Oct; 22(3): 276–282
623
[27] U.S. Department of Health and Human Services. Healthy People 2010:
624
Understanding and Improving Health. 2nd ed. Washington, DC: U.S. Government
625
Printing Office, November 2000.
626
[28] Crocco AG, Villasis-Keever M, Jadad AR. Analysis of cases of harm associated
627
with use of health information on the internet. JAMA. 2002 Jun 5;287(21):2869-71.
628
[29] Meric F, Bernstam EV, Mirza NQ, Hunt KK, Ames FC, Ross MI, et al. Breast
629
cancer on the world wide web: cross sectional survey of quality of information and
630
popularity of websites. BMJ. 2002 Mar 9;324(7337):577-81.
EP
AC C
631
TE D
621
632
[30] Practice bulletin no. 114: management of endometriosis. Obstet Gynecol. 2010
633
Jul;116(1):223-36.
634 635
[31] http://www.cngof.fr/pratiques-
636
cliniques/guidelines/apercu?path=RPC_endometriosis_en.pdf&i=227
637 638
[32] http://www.evidentlycochrane.net/hormone-therapy-menopause-endometriosis-
23
Hirsch
ACCEPTED MANUSCRIPT 639
surgery-friend-foe/
640 641 642
RI PT
643 644 645
SC
646 647 Table Legends
M AN U
648 649
Table 1. Included websites characteristics and summary statistics of quality,
651
accuracy, credibility, and readability of included websites.
652
Table 2. Readability presented by US reading age.
655 656 657 658 659 660 661 662 663 664 665 666
EP
654
AC C
653
TE D
650
24
Hirsch
ACCEPTED MANUSCRIPT
668 669 670 671 672 673 674 675 676
AC C
667
EP
TE D
M AN U
SC
RI PT
Table 1. Web site characteristics and a summary of quality, accuracy, credibility, and readability assessment
25
Hirsch
Web Domain
Listed Authors
Privacy Statement
1.
endocenter.org
USA
No
Yes
2.
endometriosis.org
Global
No
Yes
3.
endometriosis.org
Global
No
Yes
4.
endometriosis.org
Global
No
Yes
5.
endometriosis.org
Global
Yes
Yes
6.
endometriosis.org
Global
No
7.
home.bt.com
UK
Yes
8.
lifestyle.one
UK
Yes
9.
medical-dictionary.thefreedictionary.com
USA
10.
metro.co.uk
11.
pain.about.com
12.
patient.info
13.
shetrust.org.uk
14.
sogc.org
15.
womenshealth.about.com
16.
activebeat.com
17.
babycentre.co.uk
18.
channel4embarrassingillnesses.com
19.
cwhn.ca/node/40781
20.
26
Credibility
Readability
10
4
23.5
RI PT
a
c
Accuracy
46
b
Quality
d
I.D
Country
ACCEPTED MANUSCRIPT
12
7
39
50
1
4
34.8
50
10
8
55.4
37
3
3
38.1
SC
62
42
4
5
46.3
Yes
46
10
5
49.1
No
48
3
4
38.3
No
Yes
62
10
9
38.3
UK
Yes
No
37
5
2
49.3
USA
Yes
Yes
61
7
6
38.5
UK
Yes
No
69
6
7
26.8
UK
No
No
35
11
8
23.9
Canada
No
Yes
42
6
3
34.9
USA
Yes
Yes
42
2
8
24.8
Canada
No
Yes
28
2
7
40.4
Global
No
Yes
40
6
4
34.2
UK
No
Yes
32
2
5
32
Canada
No
No
43
3
4
51.7
endo-resolved.com
UK
No
No
35
1
5
44.9
21.
endo-resolved.com
UK
No
No
37
4
4
55.1
22.
endo-resolved.com
UK
No
No
54
10
9
48.1
23.
endometriosis.ie
Ireland
No
Yes
39
6
3
56.8
24.
endometriosisaustralia.org
Australia
No
No
58
10
8
24.3
25.
endometriosisinstitute.com
USA
No
No
50
2
5
49.8
26.
endometriosisinstitute.com
USA
No
No
51
9
4
33.7
27.
evidentlycochrane.net
UK
Yes
No
45
7
8
40.6
28.
evidentlycochrane.net
UK
Yes
No
56
13
6
45.9
AC C
EP
TE D
M AN U
Yes
Hirsch
ACCEPTED MANUSCRIPT Yes
Yes
62
1
5
47.6
30.
hellomagazine.com
UK
No
Yes
38
5
4
53.4
31.
independent.co.uk
UK
Yes
Yes
32
0
5
51.6
32.
livescience.com
Global
Yes
Yes
47
2
4
32.2
33.
medicalnewstoday.com
UK
Yes
Yes
45
8
5
23
34.
netmums.com
UK
No
No
45
8
5
21.3
35.
nytimes.com
USA
Yes
Yes
51
2
6
32.6
36.
nzendo.org.nz
New Zealand
No
No
40
11
8
21.5
37.
pelvicpain.org.uk
UK
No
Yes
57
2
3
61
38.
pelvicpain.org.uk
UK
No
Yes
38
13
9
33.8
39.
prevention.com
USA
Yes
Yes
35
8
7
30.8
40.
students4bestevidence.net
UK
Yes
Yes
47
5
4
47.3
41.
theguardian.com
42.
theguardian.com
43.
uptodate.com
44.
womens-health.co.uk
45.
womens-health.co.uk
46.
youngwomenshealth.org
47.
en.wikipedia.org
48.
health.facty.com
49.
betterhealth.vic.gov.au
50.
RI PT
USA
SC
healthline.com
M AN U
29.
UK
Yes
Yes
40
6
5
33.6
UK
Yes
Yes
31
4
7
29.6
UK
Yes
Yes
64
10
9
30.7
No
Yes
22
28
7
5
No
Yes
35
2
5
31
USA
Yes
No
61
5
5
38.2
Global
No
Yes
50
4
3
38.5
Canada
Yes
Yes
32
2
5
48.3
Australia
No
Yes
61
4
5
52.3
endometriosis-uk.org
UK
No
Yes
42
5
4
28.1
51.
endometriosis-uk.org
UK
No
Yes
40
3
4
32.8
52.
endometriosis-uk.org
UK
No
Yes
41
5
8
57
53.
endometriosis-uk.org
UK
No
Yes
53
2
4
23
54.
endometriosis-uk.org
UK
No
Yes
33
1
5
24.8
44 (37–51)
5 (4-7)
5 (2–9)
38.2 (30–48)
AC C
EP
TE D
New Zealand New Zealand
Median IQR
a
DISCERN tool to assess quality of information (range 16 - 80) Accuracy assessed using selected criteria from 2013 ESHRE guidelines (range 0 - 30) c Credibility based on ten criteria (range 0 - 10) d Readability assessed using the Flesch Reading Ease tool (range 0 - 100) b
Ease of reading
677
27
Table 2. Readability
USA educational level
Webpages (n)
Hirsch
ACCEPTED MANUSCRIPT th
Very easy to read (score 90-100) Easy to read (score 80-90) Fairly easy to read (score 70-80) Plain English (score 60-70)
5 Grade th 6 Grade th 7 Grade th th 8 -9 Grade
Fairly difficult to read (score 50-60)
10 -12 Grade
8
Difficult to read (score 30-50)
College
32
Very difficult to read (score 0-30)
College Graduate
13
th
679 680
SC
681 682
M AN U
683 684 685 686
TE D
687 688 689
EP
690 691
694 695
AC C
692 693
696 697 698 699 700
th
RI PT
678
0 0 0 1
Figure Legend
28
Hirsch
ACCEPTED MANUSCRIPT
EP
TE D
M AN U
SC
RI PT
Figure 1. Flow of included websites.
AC C
701
29
ACCEPTED MANUSCRIPT
RI PT
Websites retrieved n=750 AOL 150 Ask 150 Bing 150 Google 150 Yahoo 150
SC
Identification
Figure 1. Flow of included websites.
EP
M AN U
TE D
Potentially relevant n=211 AOL 44 Ask 37 Bing 44 Google 48 Yahoo 38
AC C
Included
Eligibility
Screening
Duplicates excluded n=539
Included websites n=54 AOL 15 Ask 10 Bing 12 Google 9 Yahoo 8
Excluded Websites n=155 Institution 54 Other search website 29 Advertising: 26 Insufficient information 16 Not specific 10 Individual experience 9 Scientific 8 Requires login 2 Inaccessible 2 Other 1
ACCEPTED MANUSCRIPT
Appendix 1. Summary of ESHRE Guidelines for Accuracy Assessment
1. The GDG recommends that clinicians should consider the diagnosis of endometriosis in the presence of gynecological symptoms such as: dysmenorrhea, non-cyclical pelvic pain, deep dyspareunia, infertility, fatigue in the presence of any of the above. 2. The GDG recommends that clinicians confirm a positive laparoscopy by histology, since positive histology confirms the diagnosis of endometriosis, even though negative histology does not exclude it.
RI PT
3. Clinicians are recommended to perform transvaginal sonography to diagnose or to exclude an ovarian endometrioma. 4. Clinicians are recommended not to use immunological biomarkers, including CA-125, in plasma, urine or serum to diagnose endometriosis.
SC
5. The GDG recommends clinicians to counsel women with symptoms presumed to be due to endometriosis thoroughly, and to empirically treat them with adequate analgesia, combined hormonal contraceptives or progestagens.
M AN U
6. Clinicians are recommended to prescribe hormonal treatment [hormonal contraceptives (level B), progestagens (level A), anti-progestagens (level A), or GnRH agonists (level A)] as one of the options, as it reduces endometriosis-associated pain. 7. When endometriosis is identified at laparoscopy, clinicians are recommended to surgically treat endometriosis, as this is effective for reducing endometriosis-associated pain i.e. ‘see and treat’. 8. When performing surgery in women with ovarian endometrioma, clinicians should perform cystectomy instead of drainage and coagulation, as cystectomy reduces endometriosis-associated pain.
TE D
9. The GDG recommends that clinicians refer women with suspected or diagnosed deep endometriosis to a centre of expertise that offers all available treatments in a multidisciplinary context.
EP
10. In infertile women with AFS/ASRM stage I/II endometriosis, clinicians should perform operative laparoscopy (excision or ablation of the endometriosis lesions) including adhesiolysis, rather than performing diagnostic laparoscopy only, to increase ongoing pregnancy rates.
AC C
11. In infertile women with ovarian endometrioma undergoing surgery, clinicians should perform excision of the endometrioma capsule, instead of drainage and electrocoagulation of the endometrioma wall, to increase spontaneous pregnancy rates. 12. The GDG recommends that clinicians counsel women with endometrioma regarding the risks of reduced ovarian function after surgery and the possible loss of the ovary. The decision to proceed with surgery should be considered carefully if the woman has had previous ovarian surgery. 13. Clinicians can prescribe GnRH agonists for a period of 3 to 6 months prior to treatment with assisted reproductive technologies to improve clinical pregnancy rates in infertile women with endometriosis. 14. In infertile women with endometriosis, clinicians may offer treatment with assisted reproductive technologies after surgery, since cumulative endometriosis recurrence rates are not increased after controlled ovarian stimulation for IVF/ICSI. 15. The GDG recommends that clinicians inform women with endometriosis requesting information on their risk of developing cancer that 1) there is no evidence that endometriosis causes cancer, 2) there is no increase in overall incidence of cancer in women with endometriosis, and 3) some cancers (ovarian cancer and non-Hodgkin’s lymphoma) are slightly more common in women with endometriosis.
S0002-9378(16)31987-1
DOI:
10.1016/j.ajog.2016.11.1007
Reference:
YMOB 11397
To appear in:
American Journal of Obstetrics and Gynecology
Received Date: 3 September 2016 Revised Date:
31 October 2016
Accepted Date: 3 November 2016
Please cite this article as: Hirsch M, Aggarwal S, Barker C, Davis CJ, Duffy JMN, Googling Endometriosis: A Systematic Review of Information Available on the Internet, American Journal of Obstetrics and Gynecology (2016), doi: 10.1016/j.ajog.2016.11.1007. This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.
Hirsch
ACCEPTED MANUSCRIPT 1 2 3 4 5 6 7
Googling Endometriosis: A Systematic Review of Information Available on the Internet.
8
Dentistry, Yvonne Carter Building, 58 Turner Street, Whitechapel, London, E1 2AB,
9
United Kingdom.
Martin HIRSCH1 BM, Shivani AGGARWAL 1BSc, Claire BARKER2,3BSc MSc, Colin J. DAVIS1 FRCOG MD, James M. N. DUFFY2,4 MBChB MRes Women’s Health Research Unit, Barts and the London School of Medicine and
RI PT
1
10
2
11
Quarter, University of Oxford, Oxford, OX1 2JD, United Kingdom.
12
3
13
Oxford, Oxford, OX1 2JD, United Kingdom
14
4
Nuffield Department of Primary Care Health Sciences, Radcliffe Observatory
SC
Radcliffe Women’s Health Patient and Public Involvement Group, University of
M AN U
Balliol College, University of Oxford, Oxford, OX1 3BJ, United Kingdom.
15 16
International Prospective Register of Systematic Reviews (PROSPERO):
17
CRD42016036134.
18
The authors report no conflicts of interest
20
This study received no financial support
21
TE D
19
Corresponding author
23
Dr James M N Duffy MBChB MRes BSc (Hons) PG HCL
24
Nuffield Department of Primary Care Health Sciences, Radcliffe Observatory
25
Quarter, University of Oxford, Oxford, OX1 2JD, United Kingdom.
26
[email protected]
27
+44 (0)1865 289298
28
@jamesmnduffy
AC C
29
EP
22
30
Abstract word count: 314
31
Main text word count: 3342
32 33 34 35
1
Hirsch
ACCEPTED MANUSCRIPT 36 37 38 39
Short title: Online Endometriosis Information
40 Condensation:
42
Online endometriosis information for patients is of poor quality.
RI PT
41
43 44
SC
45 46 47
M AN U
48 49 50 51 52
56 57 58 59 60 61 62 63 64 65 66 67 68 69
EP
55
AC C
54
TE D
53
2
Hirsch
ACCEPTED MANUSCRIPT 70 71 Objective
73
We aim to evaluate the credibility, quality, readability, and accuracy of online patient
74
information concerning endometriosis.
75
Data sources
76
We searched the five popular internet search engines: [1] aol.com; [2] ask.com; [3]
77
bing.com; [4] google.com; and [5] yahoo.com. We developed a search strategy in
78
consultation with patients with endometriosis, to identify relevant websites.
79
Website eligibility
80
Websites containing information related to endometriosis for women with
81
endometriosis or the public.
82
Website appraisal and synthesis methods
83
Two independent authors screened the search results. Websites were evaluated
84
using validated instruments across four domains, including assessments of: [1]
85
credibility (White Paper instrument; range 0-10); [2] quality (DISCERN instrument;
86
range 0-85); and [3] readability (Flesch-Kincaid instrument; range 0-100). Accuracy
87
was assessed by a prioritized criteria developed in consultation with healthcare
88
professionals, researchers, and women with endometriosis based upon the
89
European Society of Human Reproduction of Endometriosis guidelines (range 0 –
90
30). We summarized these data in diagrams, tables, and narratively.
91
Results
92
We identified 750 websites, of which 54 were included. Over a third of websites did
93
not attribute authorship and almost half the included websites did not report the
94
sources of information or academic references. No websites provided information
95
assessed as being written in plain English. A minority of websites were assessed as
96
high quality. A single website provided accurate information, evidentlycochrane.net.
97
Available information was, in general, skewed towards the diagnosis of
98
endometriosis. There were 16 credible websites, however the content limitations
99
were infrequently discussed. No website scored highly across all four domains.
AC C
EP
TE D
M AN U
SC
RI PT
72
100
Comment
101
In the unlikely event that a website reports high quality, accurate, and credible health
102
information it is typically challenging for a lay audience to comprehend. Healthcare
103
professionals, and the wider community, should inform women with endometriosis of
3
Hirsch
ACCEPTED MANUSCRIPT 104
the risk of outdated, inaccurate, or even dangerous information online. The
105
implementation of an Information Standard will incentivize providers of online
106
information to establish and adhere to codes of conduct.
107 International Prospective Register of Systematic Reviews (PROSPERO):
109
CRD42016036134.
111
Keywords
112
[1] Endometriosis
[5] Quality
113
[2] Patients
[6] Readability
114
[3] Accuracy
[7] Online information
115
[4] Credibility
[8] Systematic review
M AN U
116 117 118 119 120
125 126 127 128 129 130 131 132 133 134 135 136 137
EP
124
AC C
123
TE D
121 122
SC
110
RI PT
108
4
Hirsch
ACCEPTED MANUSCRIPT 138
Introduction
140
Endometriosis is benign gynecological disease which affects one in ten women of
141
reproductive age. It is characterized by pain and subfertility with associated reduced
142
quality of life [1]. The economic burden of endometriosis is of a similar magnitude to
143
other chronic diseases such as diabetes [2]. There is a paucity of high quality
144
research to guide clinical practice, this leads to unwarranted and unjustified
145
variations in patient care [3].
SC
RI PT
139
146
The internet is the source of health information as patients can access health
148
information quickly, conveniently and privately. There are an estimated 6.75 million
149
health searches daily in Google representing 4.5% of all searches performed [4].
150
There has been a rapid growth in the number of websites providing health
151
information with little or no governance [5]. Seven in ten adults regularly search for
152
an explanation and information on a new diagnosis or treatment [6-8]. Information
153
provided is commonly written at a high literacy level compounding the difficulties for
154
patients untrained in establishing whether the information is accurate. Exposure to
155
complex, ungoverned, unfounded health information which lacks expert editorial
156
supervision, could negatively affect patient understanding, compliance, and decision
157
making. This could lead to poorer health outcomes, including harm [9-13]. There are
158
no systematic reviews assessing the quality of online patient information pertaining
159
to endometriosis.
160
AC C
EP
TE D
M AN U
147
5
Hirsch
ACCEPTED MANUSCRIPT 161
We systematically assessed the accuracy, quality, readability, and credibility of
162
websites providing women with endometriosis and the public information regarding
163
the diagnosis and management of endometriosis.
164
RI PT
165
Materials and Methods
167
Sources
168
A protocol with explicitly defined objectives, criteria for website selection, and
169
approaches assessing outcome selection was developed and registered with the
170
International Prospective Register of Systematic Reviews (PROSPERO),
171
Identification number: CRD42016036134. This review was performed in accordance
172
with the Preferred Reporting Items for Systematic Reviews and Meta-analyses
173
(PRISMA) statement [14].
TE D
174
M AN U
SC
166
Website Selection
176
We developed a comprehensive search strategy in consultation with healthcare
177
professionals, researchers, and women with endometriosis. We used a keyword
178
analytic instrument, SEMrush (www.semrush.com), to inform our selection of search
179
terms. SEMrush provides analytical information related to search terms. We are
180
confident we identified and selected all search terms commonly used by women with
181
endometriosis. We used the following search terms: 1) endometriosis, 4,560,000
182
searches per annum; 2) endometriosis symptoms, 325,200 searches per annum; 3)
183
endometriosis treatment, 64,800 searches per annum; 4) endometriosis pain, 19,200
184
searches per annum; and 5) endometriosis diagnosis, 15,600 searches per annum.
185
We searched the most popular search engines including: 1) aol.com; 2) ask.com; 3)
AC C
EP
175
6
Hirsch
ACCEPTED MANUSCRIPT bing.com; 4) google.com; and 5) yahoo.com, during March 2016.
187
Individuals rarely examine more than the first three pages of a search [11]. We
188
therefore extracted the websites from the first three pages for each search term
189
within each search engine. Location services were disabled to eliminate
190
geographical bias.
RI PT
186
191
We organized the extracted websites and removed duplicates. Two reviewers (M.H.
193
and S.A.) independently screened the full content of websites to assess eligibility. All
194
data extraction was performed using piloted data extraction instruments. We pilot
195
tested each instrument using a representative sample of the websites to be
196
reviewed. This testing helped identify data that are missing from the form, or likely to
197
be superfluous. This allows authors trialing the form to provide feedback that certain
198
coding instructions are confusing or incomplete (e.g. a list of options may not cover
199
all situations). Any discrepancies between the reviewers were resolved by discussion
200
with a consensus required before the form is modified to avoid any
201
misunderstandings or later disagreements. We repeated the pilot testing on a new
202
set of websites where no major changes were needed after the first pilot testing [15].
M AN U
TE D
EP
AC C
203
SC
192
204
We included websites providing health information about endometriosis greater than
205
300 words in length. We excluded websites for the following reasons; 1) non-English
206
language; 2) inaccessible, for example password restricted; 3) aimed at a
207
professional audience, for example scientific publication; 4) excessive commercial
208
advertising (two or more commercial advertisements); and 5) content related solely
209
to the lived experience of endometriosis, for example a patient’s diary or blog.
210
7
Hirsch
ACCEPTED MANUSCRIPT 211
Those websites which met the criteria for inclusion were saved as a portable
212
document format for evaluation and data extraction by two independent authors (M.H
213
& S.A). M.H. and J.D. did not assess any websites they had previously contributed
214
too.
RI PT
215 Website Characteristics
217
Two reviewers (M.H. and S.A.) extracted the website characteristics independently
218
using a piloted data extraction sheet. Information extracted from each website
219
included country of origin, disease specific information, listed authors, and privacy
220
statements. Two reviewers (M.H. and S.A.) independently assessed each website
221
using validated instruments including assessments of 1) credibility assessed using
222
the White instrument [16] anchored between 0 (poor) and 10 (excellent) 2) quality
223
assessed using the DISCERN [17] instrument anchored between 0 (poor) and 85
224
(excellent), and 3) readability assessed using the Flesch-Kincaid [18] instrument
225
anchored between 0 (poor) and 100 (excellent). Discrepancies were resolved by
226
discussion.
M AN U
TE D
EP
227
SC
216
Quality Assessment
229
Two reviewers (MH and SA) underwent training in the use of the quality assessment
230
instruments. We assessed accuracy using a prioritized list of recommendations
231
included within the European Society of Human Reproduction and Embryology
232
endometriosis guidelines (ESHRE) [19]. The ESHRE guideline was selected for
233
comparison as this was objectively assessed to represent the highest quality
234
endometriosis guideline [20]. All recommendations were extracted by two authors
235
independently. Discrepancies were resolved by discussion. In consultation with
AC C
228
8
Hirsch
ACCEPTED MANUSCRIPT healthcare professionals, researchers and women with endometriosis, the
237
recommendations were scored as 1) critical for decision making, 2) important but not
238
critical for decision making and 3) not critical and not important for decision making.
239
Fifteen guideline recommendations were selected as statements critical for decision
240
making (appendix 1). The assessment of accuracy was standardized against
241
selected guideline recommendations. This approach has been utilized in similar
242
research studies [21].
244
Two reviewers (M.H and S.A) independently reviewed each website and using a
245
piloted standardized proforma assessing the accuracy of information. Each
246
recommendation was scored: 0 (if absent or incorrectly described), one (present and
247
incompletely described), or two (present and completely described). Accuracy
248
assessment was anchored between zero and 30. Discrepancies were resolved by
249
discussion. We classified websites with a score greater than or equal to 20 as
250
accurate.
M AN U
TE D
251
SC
243
RI PT
236
The website’s credibility was assessed by two reviewers independently using a
253
validated instrument, White [16]. This instrument, designed for consumers of health
254
information, provides a set of criteria that can be used to accurately and reliably
255
assess the quality of health information on the Internet. Credibility was assessed
256
using a ten point criteria: 1) source; 2) context; 3) currency; 4) utility; 5) editorial
257
review process; 6) hierarchy of evidence; 7) statement of original source; 8)
258
disclaimer, which included ownership, sponsorship, funding and advertising; 9)
259
omissions; and 10) feedback. Each criterion was scored 0 (absent) or one (present)
260
giving a score anchored between 0 to 10 [22]. Discrepancies were resolved by
AC C
EP
252
9
Hirsch
ACCEPTED MANUSCRIPT 261
discussion. We classified those websites with a score greater than or equal to seven
262
as credible.
263 The website’s quality was assessed by two reviewers independently using a
265
validated instrument, DISCERN [17], a validated instrument designed to assess the
266
quality of written information on treatment choices which can be applied to any
267
disease [7,17]. The DISCERN instrument offers a framework for the production,
268
evaluation, and screening of written consumer health information. This includes 16
269
questions assessed using a Likert scale anchored between one (do not agree) and
270
five (agree) [17]. Discrepancies were resolved by discussion. We classified those
271
websites as high (>53), moderate (27-52), and low (<27) quality.
M AN U
SC
RI PT
264
272
The website’s readability was assessed using the Flesch-Kincaid reading-ease test
274
[18]. This formula presents a score as a U.S. grade level, making it easier for teachers,
275
parents, librarians, and consumers of health information to judge the readability level
276
of various texts. The Flesch Kincaid score is generated from the following equation:
277
206.835 – 1.015 (total words / total sentences) – 84.6 (total syllables / total words)
278
(www.readability-score.com) [18]. The scores were anchored between 0 (complex
279
language) and 100 (simple language) and can be categorized by reading age or
280
educational status: 1) 90-100 (5th grade); 2) 80-90 (6th grade); 3) 70-80 (7th grade);
281
4) 60-70 (8th and 9th grade); 5) 50-60 (10th, 11th and 12th grade); 6) 30-50
282
(college); 7) 0-30 (college graduate). Discrepancies were resolved by discussion.
AC C
EP
TE D
273
283 284
A large-scale national assessment of the average reading level among Americans
285
performed by the National Center for Education Statistics found that the typical
10
Hirsch
ACCEPTED MANUSCRIPT American reads between a 7th and 8th grade level [23]. It is recommended that
287
online health information should not exceed the level of American 7th grade writing
288
and reading [24]. We therefore expected websites to have a readability score at or
289
below the level of American education 7th Grade (>70) to be deemed appropriate for
290
a patient and public audience.
RI PT
286
291 Analysis
293
The website characteristics and assessments were summarized in tabular form and
294
presented with descriptive statistics within summary tables and diagrams.
SC
292
M AN U
295 296 297
Results
299
The search strategy identified 750 websites which were assessed for eligibility. We
300
screened 211 websites following the exclusion of 539 duplicate websites. Two
301
authors independently applied an inclusion and exclusion criteria when screening the
302
websites. We included 54 websites in our final assessment (figure 1, table 1).
EP
AC C
303
TE D
298
304
Website characteristics
305
Twenty-one (39%) websites did not report the authors and 25 (46%) of websites did
306
not reporting sources of information or academic references. The majority of
307
included websites were published in the United Kingdom (25 websites; 46%). All
308
websites presented structured content. Almost two thirds of the websites reported a
309
privacy statement (38 websites; 70%) (table 1).
310
11
Hirsch
ACCEPTED MANUSCRIPT Accuracy
312
A single website provided accurate information, evidentlycochrane.net. The median
313
accuracy of included websites was 5 (Interquartile range [IQR] 4 – 7). Included
314
websites contained limited information (Table 1), skewed towards the diagnosis of
315
endometriosis. Information pertaining to the medical or surgical management of pain
316
or infertility associated with endometriosis were poorly represented. The most
317
commonly reported recommendation, “Clinicians should consider the diagnosis of
318
endometriosis in the presence of gynecological symptoms such as: dysmenorrhea,
319
non-cyclical pelvic pain, deep dyspareunia, infertility, fatigue in the presence of any
320
of the above”, was described by four fifths of included websites (43 websites, 80%).
321
The least frequently described recommendations, described by a small minority of
322
included websites (3 websites; 6%) were: 1) “In infertile women with endometriosis,
323
clinicians may offer treatment with assisted reproductive technologies after surgery,
324
since cumulative endometriosis recurrence rates are not increased after controlled
325
ovarian stimulation for IVF/ICSI.” 2) “Clinicians [should] inform women with
326
endometriosis requesting information on their risk of developing cancer that 1) there
327
is no evidence that endometriosis causes cancer, 2) there is no increase in overall
328
incidence of cancer in women with endometriosis, and 3) some cancers (ovarian
329
cancer and non-Hodgkin’s lymphoma) are slightly more common in women with
330
endometriosis”. The delivery of inaccurate, outdated or dangerous information
331
remains prevalent in websites. Inaccuracies include: 1) “Your specialist may also
332
suggest flushing out your blocked fallopian tubes. This procedure is an alternative to
333
surgery and is usually successful” website ID 1. Routine tubal flushing is used in
334
diagnostic evaluation of tubal patency and it is not recommended therapeutic
335
approach [25]. 2) “The only reliable way to confirm the presence of the disease is by
AC C
EP
TE D
M AN U
SC
RI PT
311
12
Hirsch
ACCEPTED MANUSCRIPT visually inspecting the abdominal organs by a procedure called a laparoscopy”
337
Website ID 20. There are many difficulties associated with visually confirming
338
endometriosis. The most reliable way to diagnose endometriosis is laparoscopy,
339
biopsy and histopathological examination. Visual diagnosis is no longer
340
recommended [19]. 3)”It is suspected that between 10-20% of reproductive aged
341
women have the disease.”
342
Website ID 20. The estimated prevalence within the general population is up to 10%
343
[19].
SC
RI PT
336
344 Credibility
346
Credibility was defined as a score equal to or greater than seven. Sixteen websites
347
(29%) were assessed as credible. The median credibility of included websites was 5
348
(IQR 2 - 8.8). The highest scoring criteria included context relevant to the disease
349
and originality with all websites fulfilling these criteria. The least frequently described
350
area of credibility was the discussion of content limitations which was reported by
351
one website (table 1).
TE D
EP
352
M AN U
345
Quality assessment
354
Thirteen websites (24%) were assessed to be high quality, 40 (74%) websites were
355
assessed to be of moderate quality, and a single website (2%) was assessed as low
356
quality. The highest scoring criteria included describing aims (median = 5; IQR 3-4)
357
and being unbiased (median 5; IQR 4-5). Websites typically did not describe the
358
consequences of no treatment (median 1; IQR 1-1).
AC C
353
359 360
Readability
13
Hirsch
ACCEPTED MANUSCRIPT
All included websites were assessed as fairly difficult to read (10th, 11th, and 12th
362
grade), difficult to read (college), or very difficult to read (college graduate). The
363
median readability score was 38.2 (IQR 30.7 – 48.0), indicating an average
364
educational status of a college student would be required to understand the written
365
content (table 1 and 2). Forty-five websites (83%) presented written information at a
366
level at or above college standard.
RI PT
361
367
There were no substantial discrepancies between authors in the data extraction of
369
quantitative parameters and we observed very high interrater agreement.
M AN U
SC
368
370 371
Comment
373
Summary
374
There are no websites which provide high quality, accurate, and credible health
375
information pertaining to endometriosis. Currently, websites contain limited amounts
376
of information which are skewed towards the diagnosis of endometriosis. In the
377
unlikely event that a website reports high quality, accurate, and credible health
378
information, it is typically written in language that is challenging for a lay audience to
379
comprehend.
380
AC C
EP
TE D
372
381
Strengths and Weaknesses
382
To our knowledge, this is the first study to examine the quality, credibility, accuracy,
383
and readability of patient focused online information pertaining to the diagnosis and
384
management of endometriosis. We followed a robust, prospective systematic review
14
Hirsch
ACCEPTED MANUSCRIPT method with validated instruments to assess the information presented. We
386
evaluated individual websites using four validated instruments in a systematic
387
process, independently performing all assessments in duplicate. We involved women
388
with endometriosis, to inform the research question, design and delivery of the
389
research study, and its dissemination. All reviewers underwent recommended
390
training prior to commencing the study.
391
RI PT
385
This study is not without limitations. Limiting the search to the first three pages may
393
have resulted in the exclusion of potentially eligible websites, however only 2.6% of
394
people search past Googles’ third page (www.protofuse.com). Included websites
395
were only written in English language, limiting the generalizability of our findings.
396
The search was conducted while computer location services were disabled, however
397
there may have been regional differences in search results, out of the authors
398
control, which account for the predominance of British websites. We designed and
399
registered this systematic review prospectively with a pre-defined inclusion criteria
400
and analysis plan. There are few scientific publications which evaluate online
401
information for patients allowing limited precedent to guide our methods. We
402
observed diminishing returns, however this was not quantified. All websites were
403
designed and managed within high resource countries. This limits the applicability of
404
this research to inform low resource settings. We did not calculate weighted kappa to
405
explore agreement between authors as the statistical level of agreement required in
406
health research is unclear [26]. This evaluation is not currently recommended by the
407
Cochrane Collaboration [15]. We could have conducted in-depth qualitative
408
interviews of women with endometriosis to explore their satisfaction with reading
AC C
EP
TE D
M AN U
SC
392
15
Hirsch
ACCEPTED MANUSCRIPT 409
individual websites and evaluate the correlation with accuracy, credibility, quality and
410
readability.
411 412
Interpretation
RI PT
413
As clinicians we must be aware that patients are increasingly seeking unregulated
415
health information online which shapes opinions and treatment choices. The
416
essence of modern clinical consultations is changing from a reliance on face-to-face
417
interaction to information gathering online prior to seeking professional opinion. In
418
the United States of America, there are over 400,000 endometriosis searches per
419
month in Google alone. We have demonstrated that individual websites are
420
frequently incomplete, inaccurate, and poorly written. This is a barrier to patient
421
education and results in those vulnerable patients who seek reliable information
422
being misinformed. This is of greater importance to non-expert patients (majority)
423
who may be less able to evaluate the reliability of online information and be
424
susceptible to the bias and inaccuracies contained within. These forays into online
425
information gathering can lead to a breakdown in doctor patient relationships.
426
Inaccurate online health information can lead to clinicians advocating guideline-
427
supported recommendations different from those read on “reputable” online sources.
428
This mismatch of information can lead to a breakdown in trust in the clinician-patient
429
relationship.
AC C
EP
TE D
M AN U
SC
414
430 431
A review conducted by the United States Office of Disease Prevention and Health
432
Promotion (ODPHP) concluded that the potential for harm from inaccurate online
433
information is significant [27]. Harm can be: 1) physical, from inappropriate
16
Hirsch
ACCEPTED MANUSCRIPT
treatments, adverse effects, or untreated disease; 2) emotional, from anxiety or false
435
hope arising from inaccurate diagnostic, prognostic, or therapeutic information; 3)
436
financial, costs incurred from unnecessary purchase of ineffective health services or
437
products [27]. The ODPHP concluded that the Internet is critical to disease
438
prevention, health promotion, and health care because of the increasing amount of
439
information and services available via the internet. This included a key objective to
440
increase the quality of online health information [28].
SC
441
RI PT
434
The readability of a website is an essential facet of online information. Information
443
presented at a standard above patients’ comprehension will limit its ability to inform
444
the patient. Healthcare professionals should be aware that there is very limited
445
information available to women with endometriosis with basic levels of literacy
446
(indicates skills necessary to perform simple and everyday literacy activities), and
447
therefore directing them to online information is of limited value in informing decision
448
making.
TE D EP
449
M AN U
442
Many online information rating systems use proxy markers for quality that do not
451
consider the needs and opinions of patients and the public. Meric and colleagues
452
[29], determined website popularity did not correlate well with traditional standards of
453
website quality. Quality of online information is crucial as patients want to know
454
about the risks, benefits, and uncertainty associated with diagnostic and therapeutic
455
options. This information must be accurate to ensure that patients seeking
456
information are gaining correct and complete information about the disease from up
457
to date scientific evidence. Without access to good quality information, patients are
458
unable to make informed choices about their treatment.
AC C
450
17
Hirsch
ACCEPTED MANUSCRIPT 459 Recommendation(s)
461
Healthcare professionals and the wider medical community are increasingly quizzed
462
by patients regarding health information found online. It is essential that healthcare
463
professionals acknowledge their position of responsibility and proactively inform
464
women with endometriosis about the risk of outdated, inaccurate, or even dangerous
465
information online. Interactive consultations using online clinical practice guidelines
466
such as those produced by the American College of Obstetrics and Gynecology [30]
467
or the Society of Obstetricians and Gynaecologists of Canada [31] can provide the
468
basis for clear, concise, evidence based management discussions. Following
469
consultations, patients should be sign posted towards higher quality and more
470
reliable sources of online information to answer questions they may have forgotten to
471
ask during their limited consultation time.
M AN U
SC
RI PT
460
TE D
472
While it may sound unrealistic to regulate health information on the internet, codes of
474
conduct have been developed and implemented. The Health on the Net Foundation,
475
based in the United States, provides accreditation to websites, which meet pre-
476
defined standards related to readability, accessibility, and accuracy [18]. The
477
Information Standard, based in the United Kingdom, assesses online health
478
information to ensure the information is clear, accurate, balanced, evidence-based,
479
and up-to-date. Information produced by the Royal College of Obstetricians and
480
Gynaecologists is accredited by this information standard
481
(https://www.england.nhs.uk/tis/).
AC C
EP
473
482 483
We acknowledge that regulating health information on the internet has inherent
18
Hirsch
ACCEPTED MANUSCRIPT
difficulties as online authors are not bound by the same codes of practice as licensed
485
healthcare professionals. The implementation of a robust Information Standard
486
internationally will incentivize providers of online information to establish and adhere
487
to codes of conduct ensuring an improvement in the quality of online information.
488
Healthcare professionals and professional bodies should direct women with
489
endometriosis towards higher quality, more reliable sources of online information. In
490
general, websites who comply with an Information Standard, should be prioritized.
RI PT
484
SC
491
The internet will continue to increase its role as a provider of online health
493
information. The media by which health information is transferred from source to
494
patient should not compromise the fundamental features of accuracy, credibility,
495
quality and readability. It would not be tolerated if a healthcare professional were
496
delivering sub-standard information in a face-to-face consultation. A strategy is
497
required to improve the standard of online information for women with endometriosis
498
with evident need for the development of patient focused online information with a
499
robust evidence base. The translation of research from trials or systematic reviews
500
into online sources has a direct pathway currently being delivered by Cochrane in
501
the form of Evidently Cochrane summaries. These webpages summarize Cochrane
502
systematic reviews into patient focused bite size pieces of information [32].
504
TE D
EP
AC C
503
M AN U
492
505
Conclusion
506
In the unlikely event that a website reports high quality, accurate, and credible health
507
information it is typically challenging for a lay audience to comprehend. Healthcare
508
professionals, and the wider community, should inform women with endometriosis of
19
Hirsch
ACCEPTED MANUSCRIPT 509
the risk of outdated, inaccurate, or even dangerous information online. Providers of
510
online information should engage with established codes of conduct, such as the
511
Information Standard.
512
RI PT
513 514 515
Acknowledgements
516
The authors would to acknowledge the contribution of the International Collaboration
518
to Harmonise Outcomes and Measures for Endometriosis. Steering Group: Dr
519
James M N Duffy (Balliol College, University of Oxford, Oxford, UK); Cindy Farquhar
520
(Department of Obstetrics & Gynecology, University of Auckland, Auckland, New
521
Zealand); Dr Martin Hirsch (Women’s Health Research Unit, Queen Mary University
522
of London, London, UK); Neil Johnson (University of Adelaide, Adelaide, Australia);
523
and Khalid Khan (Women’s Health Research Unit, Queen Mary University of
524
London, London, UK).
M AN U
SC
517
526 527 528
References
TE D
525
[1] Duffy JMN, Arambage K, Correa FJS, Olive D, Farquhar C, Garry R.
530
Laparoscopic surgery for endometriosis. Cochrane Database of Systematic Reviews
531
2014, Issue 4. Art. No.: CD011031.
AC C
532
EP
529
533
[2] Simoens S, Dunselman G, Dirksen C, Hummelshoj L, Bokor A, Brandes I, et al.
534
The burden of endometriosis: costs and quality of life of women with endometriosis
535
and treated in referral centres. Hum Reprod. 2012 May;27(5):1292-9.
536 537
[3] Hirsch M, Duffy JM, Kusznir JO, Davis CJ, Plana MN, Khan KS. Variation in
538
outcome reporting in endometriosis trials: a systematic review. Am J Obstet Gynecol.
539
2016 Apr;214(4):452-64.
540
20
Hirsch
ACCEPTED MANUSCRIPT 541
[4] Eysenbach G, Kohler C. What is the prevalence of health-related searches on the
542
World Wide Web? Qualitative and quantitative analysis of search engine queries on
543
the internet. AMIA Annu Symp Proc 2003;225–229.
544 [5] Mackey TK, Liang BA, Kohler JC, Attaran A. Health domains for sale: the need
546
for global health Internet governance.J Med Internet Res. 2014 Mar 5;16(3):e62.
RI PT
545
547
[6] Handfield B, Turnbull S, Bell RJ. What do obstetricians think about media
549
influences on their patients? Aust N Z J Obstet Gynaecol 2006;46:379–83.
550
SC
548
[7] Kaicker J, Debono VB, Dang W, Buckley N, Thabane L. Assessment of the
552
quality and variability of health information on chronic pain websites using the
553
DISCERN instrument. BMC Med 2010;8:59.
M AN U
551
554
[8] Hesse BW, Nelson DE, Kreps GL, Croyle RT, Arora NK, Rimer BK, et al. Trust
556
and sources of health information: the impact of the Internet and its implications for
557
health care providers: findings from the first Health Information National Trends
558
Survey. Arch Intern Med 2005;165:2618–24.
TE D
555
559 560
[9] Cline RJ, Haynes KM. Consumer health information seeking on the Internet: the
561
state of the art. Health Educ Res 2001;16:671–92.
EP
562
[10] Berkman ND, Sheridan SL, Donahue KE, Halpern DJ, Crotty K. Low health
564
literacy and health outcomes: an updated systematic review. Ann Intern Med
565
2011;155:97–107.
566
AC C
563
567
[11] Sacchetti P, Zvara P, Plante MK. The Internet and patient education– resources
568
and their reliability: focus on a select urologic topic. Urology 1999;53:1117–20.
569 570
[12] McMullan M. Patients using the internet to obtain health information: how this
571
affects the patient-health professional relationship. Patient Educ Couns 2006;63(1–
572
2):24–28
573 574
[13] Benigeri M, Pluye P. Shortcomings of health information on the internet. Health
21
Hirsch
ACCEPTED MANUSCRIPT 575
Promot Int 2003;18(4):381–386
576 577
[14] Moher D, Liberati A, Tetzlaff J, Altman DG, The PRISMA Group. Preferred
578
Reporting Items for Systematic Reviews and Meta-Analyses: The PRISMA
579
Statement. BMJ 2009;339:b2535.
RI PT
580 581
[15] Higgins JPT, Green S (editors). Cochrane Handbook for Systematic Reviews of
582
Interventions Version 5.1.0 [updated March 2011]. The Cochrane Collaboration, 2011.
583
Available from www.cochrane-handbook.org.
SC
584 585
[16] Ambre J, Guard R, Perveiler FM, Renner J, Rippen H (1997) White paper:
586
Criteria for assessing the quality of health information on the internet.
M AN U
587 588
[17] Charnock D, Shepperd S, Needham G, Gann R. DISCERN: an instrument for
589
judging the quality of written consumer health information on treatment choices. J
590
Epidemiol Community Health 1999;53:105–111.
591
[18] Kincaid JP, Fishburne RP Jr, Rogers RL, Chissom BS. Derivation of new
593
readability formulas (Automated Readability Index, Fog Count and Flesch Reading
594
Ease Formula) for Navy enlisted personnel. Memphis, TN: Naval Air Station; 1975.
595
TE D
592
[19] Dunselman GA, Vermeulen N, Becker C, et al. ESHRE guideline: management
597
of women with endometriosis. European Society of Human Reproduction and
598
Embryology. Hum Reprod. 2014 Mar;29(3):400-12.
AC C
599
EP
596
600
[20] Hirsch M, Begum MR, Paniz É, Barker C, Davis CJ, Duffy JMN. Diagnosis and
601
management of endometriosis: a systematic review of international and national
602
guidelines. Submitted for publication to Obstetrics and Gynecology, October 2016.
603 604
[21] Fisher JH, O'Connor D, Flexman AM, Shapera S, Ryerson CJ. Accuracy and
605
Reliability of Internet Resources for Information on Idiopathic Pulmonary Fibrosis.
606
Am J Respir Crit Care Med. 2016 Jul 15;194(2):218-25.
607 608
[22] Tirlapur SA, Leiu C, Khan KS. Quality of information on the internet related to
22
Hirsch
ACCEPTED MANUSCRIPT
bladder pain syndrome: a systematic review of the evidence. Int Urogynecol J. 2013
610
Aug; 24(8):1257–62.
611
[23] National Center for Education Statistics. National Assessment of Adult Literacy
612
(NAAL). 2003. http://nces.ed.gov/naal. Accessed August 6, 2008.
613
[24] Bernstam EV, Shelton DM, Walji M, Meric-Bernstam F. Instruments to assess
614
the quality of health information on the world wide web: what can our patients
615
actually use? Int J Med Inform 2005;74(1):13–19
RI PT
609
616
[25] Duffy JMN, Johnson N, Ahmad G, Watson A. Postoperative procedures for
618
improving fertility following pelvic reproductive surgery. Cochrane Database of
619
Systematic Reviews 2009, Issue 2. Art. No.: CD001897.
620
M AN U
SC
617
[26] McHugh ML. Interrater reliability: the kappa statistic. Biochem Med (Zagreb).
622
2012 Oct; 22(3): 276–282
623
[27] U.S. Department of Health and Human Services. Healthy People 2010:
624
Understanding and Improving Health. 2nd ed. Washington, DC: U.S. Government
625
Printing Office, November 2000.
626
[28] Crocco AG, Villasis-Keever M, Jadad AR. Analysis of cases of harm associated
627
with use of health information on the internet. JAMA. 2002 Jun 5;287(21):2869-71.
628
[29] Meric F, Bernstam EV, Mirza NQ, Hunt KK, Ames FC, Ross MI, et al. Breast
629
cancer on the world wide web: cross sectional survey of quality of information and
630
popularity of websites. BMJ. 2002 Mar 9;324(7337):577-81.
EP
AC C
631
TE D
621
632
[30] Practice bulletin no. 114: management of endometriosis. Obstet Gynecol. 2010
633
Jul;116(1):223-36.
634 635
[31] http://www.cngof.fr/pratiques-
636
cliniques/guidelines/apercu?path=RPC_endometriosis_en.pdf&i=227
637 638
[32] http://www.evidentlycochrane.net/hormone-therapy-menopause-endometriosis-
23
Hirsch
ACCEPTED MANUSCRIPT 639
surgery-friend-foe/
640 641 642
RI PT
643 644 645
SC
646 647 Table Legends
M AN U
648 649
Table 1. Included websites characteristics and summary statistics of quality,
651
accuracy, credibility, and readability of included websites.
652
Table 2. Readability presented by US reading age.
655 656 657 658 659 660 661 662 663 664 665 666
EP
654
AC C
653
TE D
650
24
Hirsch
ACCEPTED MANUSCRIPT
668 669 670 671 672 673 674 675 676
AC C
667
EP
TE D
M AN U
SC
RI PT
Table 1. Web site characteristics and a summary of quality, accuracy, credibility, and readability assessment
25
Hirsch
Web Domain
Listed Authors
Privacy Statement
1.
endocenter.org
USA
No
Yes
2.
endometriosis.org
Global
No
Yes
3.
endometriosis.org
Global
No
Yes
4.
endometriosis.org
Global
No
Yes
5.
endometriosis.org
Global
Yes
Yes
6.
endometriosis.org
Global
No
7.
home.bt.com
UK
Yes
8.
lifestyle.one
UK
Yes
9.
medical-dictionary.thefreedictionary.com
USA
10.
metro.co.uk
11.
pain.about.com
12.
patient.info
13.
shetrust.org.uk
14.
sogc.org
15.
womenshealth.about.com
16.
activebeat.com
17.
babycentre.co.uk
18.
channel4embarrassingillnesses.com
19.
cwhn.ca/node/40781
20.
26
Credibility
Readability
10
4
23.5
RI PT
a
c
Accuracy
46
b
Quality
d
I.D
Country
ACCEPTED MANUSCRIPT
12
7
39
50
1
4
34.8
50
10
8
55.4
37
3
3
38.1
SC
62
42
4
5
46.3
Yes
46
10
5
49.1
No
48
3
4
38.3
No
Yes
62
10
9
38.3
UK
Yes
No
37
5
2
49.3
USA
Yes
Yes
61
7
6
38.5
UK
Yes
No
69
6
7
26.8
UK
No
No
35
11
8
23.9
Canada
No
Yes
42
6
3
34.9
USA
Yes
Yes
42
2
8
24.8
Canada
No
Yes
28
2
7
40.4
Global
No
Yes
40
6
4
34.2
UK
No
Yes
32
2
5
32
Canada
No
No
43
3
4
51.7
endo-resolved.com
UK
No
No
35
1
5
44.9
21.
endo-resolved.com
UK
No
No
37
4
4
55.1
22.
endo-resolved.com
UK
No
No
54
10
9
48.1
23.
endometriosis.ie
Ireland
No
Yes
39
6
3
56.8
24.
endometriosisaustralia.org
Australia
No
No
58
10
8
24.3
25.
endometriosisinstitute.com
USA
No
No
50
2
5
49.8
26.
endometriosisinstitute.com
USA
No
No
51
9
4
33.7
27.
evidentlycochrane.net
UK
Yes
No
45
7
8
40.6
28.
evidentlycochrane.net
UK
Yes
No
56
13
6
45.9
AC C
EP
TE D
M AN U
Yes
Hirsch
ACCEPTED MANUSCRIPT Yes
Yes
62
1
5
47.6
30.
hellomagazine.com
UK
No
Yes
38
5
4
53.4
31.
independent.co.uk
UK
Yes
Yes
32
0
5
51.6
32.
livescience.com
Global
Yes
Yes
47
2
4
32.2
33.
medicalnewstoday.com
UK
Yes
Yes
45
8
5
23
34.
netmums.com
UK
No
No
45
8
5
21.3
35.
nytimes.com
USA
Yes
Yes
51
2
6
32.6
36.
nzendo.org.nz
New Zealand
No
No
40
11
8
21.5
37.
pelvicpain.org.uk
UK
No
Yes
57
2
3
61
38.
pelvicpain.org.uk
UK
No
Yes
38
13
9
33.8
39.
prevention.com
USA
Yes
Yes
35
8
7
30.8
40.
students4bestevidence.net
UK
Yes
Yes
47
5
4
47.3
41.
theguardian.com
42.
theguardian.com
43.
uptodate.com
44.
womens-health.co.uk
45.
womens-health.co.uk
46.
youngwomenshealth.org
47.
en.wikipedia.org
48.
health.facty.com
49.
betterhealth.vic.gov.au
50.
RI PT
USA
SC
healthline.com
M AN U
29.
UK
Yes
Yes
40
6
5
33.6
UK
Yes
Yes
31
4
7
29.6
UK
Yes
Yes
64
10
9
30.7
No
Yes
22
28
7
5
No
Yes
35
2
5
31
USA
Yes
No
61
5
5
38.2
Global
No
Yes
50
4
3
38.5
Canada
Yes
Yes
32
2
5
48.3
Australia
No
Yes
61
4
5
52.3
endometriosis-uk.org
UK
No
Yes
42
5
4
28.1
51.
endometriosis-uk.org
UK
No
Yes
40
3
4
32.8
52.
endometriosis-uk.org
UK
No
Yes
41
5
8
57
53.
endometriosis-uk.org
UK
No
Yes
53
2
4
23
54.
endometriosis-uk.org
UK
No
Yes
33
1
5
24.8
44 (37–51)
5 (4-7)
5 (2–9)
38.2 (30–48)
AC C
EP
TE D
New Zealand New Zealand
Median IQR
a
DISCERN tool to assess quality of information (range 16 - 80) Accuracy assessed using selected criteria from 2013 ESHRE guidelines (range 0 - 30) c Credibility based on ten criteria (range 0 - 10) d Readability assessed using the Flesch Reading Ease tool (range 0 - 100) b
Ease of reading
677
27
Table 2. Readability
USA educational level
Webpages (n)
Hirsch
ACCEPTED MANUSCRIPT th
Very easy to read (score 90-100) Easy to read (score 80-90) Fairly easy to read (score 70-80) Plain English (score 60-70)
5 Grade th 6 Grade th 7 Grade th th 8 -9 Grade
Fairly difficult to read (score 50-60)
10 -12 Grade
8
Difficult to read (score 30-50)
College
32
Very difficult to read (score 0-30)
College Graduate
13
th
679 680
SC
681 682
M AN U
683 684 685 686
TE D
687 688 689
EP
690 691
694 695
AC C
692 693
696 697 698 699 700
th
RI PT
678
0 0 0 1
Figure Legend
28
Hirsch
ACCEPTED MANUSCRIPT
EP
TE D
M AN U
SC
RI PT
Figure 1. Flow of included websites.
AC C
701
29
ACCEPTED MANUSCRIPT
RI PT
Websites retrieved n=750 AOL 150 Ask 150 Bing 150 Google 150 Yahoo 150
SC
Identification
Figure 1. Flow of included websites.
EP
M AN U
TE D
Potentially relevant n=211 AOL 44 Ask 37 Bing 44 Google 48 Yahoo 38
AC C
Included
Eligibility
Screening
Duplicates excluded n=539
Included websites n=54 AOL 15 Ask 10 Bing 12 Google 9 Yahoo 8
Excluded Websites n=155 Institution 54 Other search website 29 Advertising: 26 Insufficient information 16 Not specific 10 Individual experience 9 Scientific 8 Requires login 2 Inaccessible 2 Other 1
ACCEPTED MANUSCRIPT
Appendix 1. Summary of ESHRE Guidelines for Accuracy Assessment
1. The GDG recommends that clinicians should consider the diagnosis of endometriosis in the presence of gynecological symptoms such as: dysmenorrhea, non-cyclical pelvic pain, deep dyspareunia, infertility, fatigue in the presence of any of the above. 2. The GDG recommends that clinicians confirm a positive laparoscopy by histology, since positive histology confirms the diagnosis of endometriosis, even though negative histology does not exclude it.
RI PT
3. Clinicians are recommended to perform transvaginal sonography to diagnose or to exclude an ovarian endometrioma. 4. Clinicians are recommended not to use immunological biomarkers, including CA-125, in plasma, urine or serum to diagnose endometriosis.
SC
5. The GDG recommends clinicians to counsel women with symptoms presumed to be due to endometriosis thoroughly, and to empirically treat them with adequate analgesia, combined hormonal contraceptives or progestagens.
M AN U
6. Clinicians are recommended to prescribe hormonal treatment [hormonal contraceptives (level B), progestagens (level A), anti-progestagens (level A), or GnRH agonists (level A)] as one of the options, as it reduces endometriosis-associated pain. 7. When endometriosis is identified at laparoscopy, clinicians are recommended to surgically treat endometriosis, as this is effective for reducing endometriosis-associated pain i.e. ‘see and treat’. 8. When performing surgery in women with ovarian endometrioma, clinicians should perform cystectomy instead of drainage and coagulation, as cystectomy reduces endometriosis-associated pain.
TE D
9. The GDG recommends that clinicians refer women with suspected or diagnosed deep endometriosis to a centre of expertise that offers all available treatments in a multidisciplinary context.
EP
10. In infertile women with AFS/ASRM stage I/II endometriosis, clinicians should perform operative laparoscopy (excision or ablation of the endometriosis lesions) including adhesiolysis, rather than performing diagnostic laparoscopy only, to increase ongoing pregnancy rates.
AC C
11. In infertile women with ovarian endometrioma undergoing surgery, clinicians should perform excision of the endometrioma capsule, instead of drainage and electrocoagulation of the endometrioma wall, to increase spontaneous pregnancy rates. 12. The GDG recommends that clinicians counsel women with endometrioma regarding the risks of reduced ovarian function after surgery and the possible loss of the ovary. The decision to proceed with surgery should be considered carefully if the woman has had previous ovarian surgery. 13. Clinicians can prescribe GnRH agonists for a period of 3 to 6 months prior to treatment with assisted reproductive technologies to improve clinical pregnancy rates in infertile women with endometriosis. 14. In infertile women with endometriosis, clinicians may offer treatment with assisted reproductive technologies after surgery, since cumulative endometriosis recurrence rates are not increased after controlled ovarian stimulation for IVF/ICSI. 15. The GDG recommends that clinicians inform women with endometriosis requesting information on their risk of developing cancer that 1) there is no evidence that endometriosis causes cancer, 2) there is no increase in overall incidence of cancer in women with endometriosis, and 3) some cancers (ovarian cancer and non-Hodgkin’s lymphoma) are slightly more common in women with endometriosis.