- Email: [email protected]
What parents are reading about laryngomalacia: Quality and readability of internet resources on laryngomalacia
Accepted Manuscript What parents are reading about laryngomalacia: Quality and readability of internet resources on laryngomalacia Erica Corredera, Kara S. Davis, Jeffrey P. Simons, Noel Jabbour PII:
S0165-5876(18)30109-5
DOI:
10.1016/j.ijporl.2018.02.036
Reference:
PEDOT 8911
To appear in:
International Journal of Pediatric Otorhinolaryngology
Received Date: 17 December 2017 Revised Date:
20 February 2018
Accepted Date: 22 February 2018
Please cite this article as: E. Corredera, K.S. Davis, J.P. Simons, N. Jabbour, What parents are reading about laryngomalacia: Quality and readability of internet resources on laryngomalacia, International Journal of Pediatric Otorhinolaryngology (2018), doi: 10.1016/j.ijporl.2018.02.036. This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.
ACCEPTED MANUSCRIPT
What parents are reading about laryngomalacia: Quality and readability of Internet resources on laryngomalacia.
M AN U
b. Children's Hospital of Pittsburgh of UPMC Division of Otolaryngology 4401 Penn Avenue Faculty Pavilion, 7th Floor Pittsburgh, PA 15224 Admin Office: 412-692-5466 Fax: 412-692-6074
SC
Affiliations: a. University of Pittsburgh School of Medicine M240 Scaife Hall, 3550 Terrace St Pittsburgh, PA 15213 Admin Office: 412-648-9891
RI PT
Erica Correderaa, Kara S. Davisb, Jeffrey P. Simonsb, Noel Jabbourb
Corresponding Author:
TE D
Noel Jabbour, MD
EP
Children's Hospital of Pittsburgh of UPMC Division of Otolaryngology 4401 Penn Avenue Faculty Pavilion, 7th Floor Pittsburgh, PA 15224 Admin Office: 412-692-5466 Fax: 412-692-6074
AC C
[email protected]
Keywords: laryngomalacia; stridor; website; patient information; internet; readability
The authors have no conflicts of interest. Word Count: 2539 words
ACCEPTED MANUSCRIPT
Abstract:
RI PT
Objective: The goal of this study is to measure the quality and readability of websites related to laryngomalacia, and to compare the quality and readability scores for the sites accessed through the most popular search engines.
SC
Introduction: Laryngomalacia is a common diagnosis in children but is often difficult for parents to comprehend. As information available on the internet is unregulated, the quality and readability of this information may vary.
M AN U
Methods: An advanced search on Google, Yahoo, and Bing was conducted using the terms “laryngomalacia” OR “soft larynx” OR “floppy voice box.” The first ten websites meeting inclusion and exclusion criteria were evaluated, for each search engine. Quality and readability were assessed using the DISCERN criteria and the Flesch reading ease scoring (FRES) and Flesch-Kincaid grade level (FKGL) tests, respectively. Results: The top 10 hits on each search engine yielded 15 unique web pages. The median DISCERN score (out of a possible high-score of 80) was 48.5 (SD 12.6). The median USA grade-level estimated by the FKGL was 11.3 (SD 1.4). Only one website (6.7%), had a readability score in the optimal range of 6th to 8th grade reading level. DISCERN scores did not correlate with FKGL scores (r=0.10).
AC C
EP
TE D
Conclusion: Online information discussing laryngomalacia often varies in quality and may not be easily comprehensible to the public. It is important for healthcare professionals to understand the quality of health information accessible to patients as it may influence medical decision-making by patient families.
ACCEPTED MANUSCRIPT
1. Introduction
The utilization of the internet as a source of medical information has increased immensely over
RI PT
the last twenty years. Many patients now rely on the internet as part of their medical decisionmaking [1, 2]. Certain populations are more likely to use the internet as a source of advice on the medical decision-making, due to its immediate availability and free content. The obtainment of
SC
online medical information should naturally progress to understanding and application of the information, which requires some degree of health literacy. A particular population that must be
children that require medical treatment.
M AN U
considered in regards to the effect of health literacy on the healthcare process is parents of
Laryngomalacia is a leading cause of pediatric stridor and is one of the most common pediatric diagnoses. Breathing and airway symptoms often generate high stress levels amongst family
TE D
members and thus lead to information-seeking behavior [3, 4, 5, 6]. The purpose of this study was to evaluate the quality and ease of readability for websites related to laryngomalacia. Decreased levels of parent health literacy have been associated with higher rates of pediatric
EP
emergency department visits, non-adherence to treatment plans, and difficulty understanding healthcare-related tools such as growth charts [7, 8]. Evidence has shown that high-quality
11].
AC C
websites with easily read content are lacking and there is wide variability in the content [9, 10,
The online information that parents gather about laryngomalacia most likely has an effect on the decision-making process regarding their child’s care. This study aimed to (1) measure the quality and readability of websites related to laryngomalacia, and (2) to compare the quality and readability scores for the most popular websites on each of the three most widely used search
ACCEPTED MANUSCRIPT
engines. Quality and readability are assessed using the DISCERN criteria and the Flesch reading ease scoring (FRES) and Flesch-Kincaid grade level (FKGL) tests, respectively. The DISCERN instrument is a validated rating tool and the first standardized quality index of consumer health
RI PT
information that was created and made available for content producers, health professionals, and patients to appraise written information on treatment options [12]. The Flesch-Kincaid readability tests consist of two tests— Flesch Reading Ease Score (FRES) and Flesch-Kincaid
SC
Grade Level (FKGL)— that indicate the level of difficulty for a reading passage. The hypothesis of this study was that an increase in the DISCERN score, as a metric of quality, correlated with
M AN U
decreased readability scores, represented by a low ease of reading score (FRES) and a high reading level (FKGL).
2. Methods
TE D
2.1 Strategy
An advanced search was conducted on the Google, Yahoo, and Bing search engines using the “OR” operator in order to browse the following string term on a single date, July 26, 2016:
EP
“laryngomalacia” OR “soft larynx” OR “floppy voice box”. The online query was conducted in Pittsburgh, Pennsylvania. The first 10 websites meeting inclusion and exclusion criteria on each
AC C
search engine were evaluated. Websites must have been documented in the English language and must have been functional at the time of access for inclusion. Exclusion criteria included newspaper articles, sponsored websites, information consisting of less than 30 sentences, imagesonly websites, and online dictionaries.
Quality and readability were assessed using the DISCERN criteria and the Flesch reading ease scoring (FRES) and Flesch-Kincaid grade level (FKGL) tests, respectively. The written content
ACCEPTED MANUSCRIPT
was copied from each website into a word processor. Identifying information, such as hospital/institution affiliations, were removed to minimize bias. All included websites were independently reviewed by two attending pediatric otolaryngologists (NJ, JPS) for quality using
RI PT
the DISCERN criteria questions [12]. When the average DISCERN score for each publication was within 1 point difference (out of 5) between the raters, the scores for that publication were averaged without further consensus review. For all other scores with greater than 1 point
SC
discrepancy between the raters, a consensus DISCERN score was made by simultaneous
M AN U
repeated review by both reviewers.
The written content was then evaluated in Microsoft Word for Mac Version 15.15 (Microsoft Corp., Redmond, WA) for the FRES and FKGL by a single evaluator (EC). The mean DISCERN score (out of a possible high-score of 80), mean rating of publication, mean ease of readability score, and mean reading level was independently calculated for each search engine and also
TE D
averaged to obtain a single value for each measure of quality and readability.
2.2 Measures DISCERN
AC C
2.2.1
EP
Date of last update and the HONcode accreditation were recorded as of July 26, 2016.
While using the DISCERN validated rating tool, websites are given a total rating out of 80 points that is based on 16 key questions assessing the site’s content. Questions 1-8 emphasize the reliability of the publication and encourage one to consider its dependability as a source of medical information. Questions 9-15 address specifics of the treatment options. Question 16 represents the overall rating of publication. Each question is rated on a 5-point scale ranging
ACCEPTED MANUSCRIPT
from “No” to “Yes”, with a 1 defined as a definitive “No”, a 2 representing serious/extensive shortcomings, a 3 meaning the content is potentially important with no serious shortcomings, a 4 affirming minimal shortcomings, and a 5 characterizing a definitive “Yes” and complete
Flesch-Kincaid readability
SC
2.2.2
RI PT
fulfillment of criteria.
The formula to calculate the FRES is as follows: 206.835 - (1.015 x average sentence length) -
M AN U
(84.6 x average number of syllables per word). Average sentence length is measured as total words per total sentences. A score is determined from 0 to 100, with higher scores indicating greater ease of reading [13].
The formula to calculate the FKGL is as follows: 0.39*(average number of words per sentence)
TE D
+ 11.8*(average number of syllables per word) - 15.59. The FKGL score represents a U.S. grade level or the number of years of education generally required to understand the text under evaluation. For example, a score of 8.0 would indicate that the text is likely understandable by an
2.2.3
AC C
years [13].
EP
average student in the 8th grade of the U.S. education system, which is often between ages 13-15
Fry Readability
The Fry readability is a readability metric that determines the grade reading level of a passage of written text. The intersection between average number of sentences and syllables per 100 words on a fry graph determines the grade reading level.
ACCEPTED MANUSCRIPT
2.2.4
HONcode
The Health On the Net Foundation (HON) is a non-profit, non-governmental organization,
RI PT
accredited to the Economic and Social Council of the United Nations. The HONcode certification is an ethical standard aimed at offering quality health information. The HONcode certification symbol is identifiable through a downloadable toolbar extension, and will become
SC
active if a website is HONcode certified [14]. For each website, the HONcode seal was checked on a single date (July 26, 2016).
Statistics
M AN U
2.2.5
Descriptive statistics were calculated from all data, using the mean, range, and standard deviation, when applicable, for reporting. D’Agostino and Pearson omnibus normality test was
TE D
run on DISCERN, FKGL and FRES data to determine normality of the data sets. A correlation study was then performed using the Pearson’s 1-tailed test. Inter-rater agreement was tested
3. Results
EP
using Spearman's correlation.
AC C
3.1 Website characteristics
The websites retrieved from Google, Yahoo, and Bing are listed in Table 1. A significant number of the websites evaluated were owned by academic institutions (n=5, 33.3% of a total of 15 unique websites). Of the websites recorded from all three search engines with a listed ‘last date of update’ retrieved from the page source, 5 (33.3%) of the unique websites had been updated in the previous 12 months, 33% (3/9) from the Google search, 50% (4/8) from the Yahoo search, 37.5% (3/8) from the Bing search, as noted in Table 1.
ACCEPTED MANUSCRIPT
3.2 Measures 3.2.1
DISCERN
RI PT
The mean DISCERN score was 46.1. The median DISCERN score (out of a possible high-score of 80) was 48.5 (SD 12.6). The independent mean DISCERN score for the Google, Yahoo, and Bing websites was 48.5, 45.3, and 44.6, respectively. The highest scoring website was the
SC
Children’s Hospital of Philadelphia laryngomalacia page, with a DISCERN score of 60.5/80. The lowest scoring website was the Right Diagnosis laryngomalacia page, with a DISCERN
M AN U
score of 16.5/80. The mean rating of publication based on question-16 in the DISCERN score for websites recorded was 3.1, 2.9, and 2.8 from the Google, Yahoo, and Bing search, respectively, each falling into the ‘partially valid source of information’ category on a 1-5 scale.
3.2.2
Flesch-Kincaid readability
TE D
The ease of readability score estimated by the FRES test was 40.7 before removal of source identifiers and 41.8 after removal of source identifiers. The independent ease of readability score
EP
estimated by the Flesch reading ease scoring test (FRES) was 41.9 for the Google websites, 40.8 for the Yahoo websites, and 39.4 for the Bing websites before removal of source identifiers; 43.5
AC C
for the Google websites, 42.0 for the Yahoo websites, and 39.9 for the Bing websites after the removal of source identifiers.
The mean reading level estimated by the FKGL test was 11.3 before removal of source identifiers and 10.9 after removal of source identifiers. The median USA grade-level estimated by the FKGL was 11.3 (SD 1.4). The independent mean reading level estimated by the FleschKincaid grade level (FKGL) test was 11.1 for the Google websites, 11.3 for the Yahoo websites,
ACCEPTED MANUSCRIPT
and 11.4 for the Bing websites before the removal of source identifiers; 10.7 for the Google websites, 10.9 for the Yahoo websites, and 11.2 for the Bing websites after the removal of source
3.2.3
Correlation with DISCERN and readability
RI PT
identifiers.
SC
To examine the hypothesis that a greater DISCERN score, as a metric of quality, would correlate with decreased readability scores as represented by a low score on FRES and a high grade level
M AN U
(FKGL), a correlation study between DISCERN and the readability metrics was performed. DISCERN, FRES, and FKGL data demonstrated a normal distribution through the D’Agostino and Pearson omnibus normality test, indicating suitability of using the Pearson’s 1-tailed correlation test. There was a positive correlation between DISCERN score and FRES when a
TE D
Pearson correlation test was conducted on the entirety of the websites used in this study (r=0.41, N=29, p=0.01), excluding one outlier, as depicted in Figure 1. The correlation between total
EP
DISCERN score and FKGL was not statistically significant (r=0.14, N=30, p=0.22).
Isolated correlation studies were also conducted comparing quality and readability for the sites
AC C
recorded from each search engine. The correlation between total DISCERN score and FRES was not statistically significant for Google (r=-0.11, N=10, p=0.75), Yahoo (r=0.53, N=10, p=0.11) or Bing (r=0.42, N=10, p=0.22). The correlation between total DISCERN score and FKGL was not statistically significant for Google (r=0.47, N=10, p=0.17), Yahoo (r=0.12, N=10, p=0.74), and Bing (r=-0.02, N=10, p=0.94).
3.2.4
Fry readability
ACCEPTED MANUSCRIPT
Five unique online sources scored less than a 12th grade reading level. Two unique online sources scored in the college reading level range. Eight unique online sources scored at a college
One
HONcode Accreditation
website
from
the
Google
search
was
HONcode
SC
3.2.5
RI PT
level or above, and outside of the range of the Fry graph (Figure 2).
accredited:
http://patient.info/doctor/congenital-throat-problems. None of the websites associated with
M AN U
academic institutions were HONcode certified.
4. Discussion
The goals of this study were to (1) measure the quality and readability of websites related to laryngomalacia, and (2) to compare the quality and readability scores for the most popular
TE D
websites on each of the three most widely used search engines in the world. Overall, laryngomalacia related websites were found to be of fair quality by DISCERN criteria, with wide
EP
variability. There were 9 Google, 10 Yahoo, and 10 Bing websites with a readability score less than the suggested readability score of 65 for public health information; only one Google website
AC C
achieved the suggested readability score of at least 65. The FKGL estimated average was the 10th grade reading level, which is above the recommended reading level of grade 6-8 for public health information.
Online resources suggest that medical information especially related to surgical specialties is often difficult to read and that authors should aim for FRES scores of at least 30 in order to maintain a higher degree of quality [15]. Previous studies have shown a negative correlation
ACCEPTED MANUSCRIPT
between ease of readability and quality of information. Studies evaluating online information regarding other pediatric otolaryngologic diagnoses such as glue ear, microtia, and tympanostomy tubes have also reported generally low-quality, difficult to read websites [8, 9,
RI PT
10]. In the results published on aural atresia and microtia, the websites with the most reliable content had the highest reading levels, potentially reducing their usefulness as a healthcare tool to parents and adolescents with access to the internet. In this study, quality as measured by the
SC
DISCERN score showed a positive correlation with FRES (Figure 1), thus rejecting the notion
M AN U
that higher quality translates to increased reading difficulty.
Only one website in this study (3.33% overall), taken from the Google search (10% of Google sources), was HONcode accredited; statistical analyses comparing websites with and without HONcode accreditation was infeasible. Anecdotally, the HONcode accredited website had an average FRES of 28.95 and was estimated to be in the 12th grade reading level, qualifying as
TE D
fairly difficult. Like the DISCERN tool, HONcode certification does not address the accuracy of the information nor does it account for omitted information, an important limitation to consider
EP
when counseling patients and family members on entities such as HONcode. The value of HONcode certification is that it does address some common issues with quality by ensuring that
AC C
websites define their intended audience, provide an author or editors name, list the date that the material was updated, disclose funding sources, and avoid advertisements, among other reasonable requirements. With increased use, consumers could become accustomed to look for this certification when searching the internet for healthcare related questions.
ACCEPTED MANUSCRIPT
Additionally, the number of online sources studied was limited to 30 and required that the content be written in the English language. The subset of keywords chosen for the search (“laryngomalacia” OR “soft larynx” OR floppy voice box”) introduce a congruence bias due to
RI PT
the assumption that the general population will most likely search the same terms. Furthermore, the information presented in this study is based on a search conducted on a single date, July 26, 2016; websites may be added and content from existing websites may be altered at any time.
SC
Lastly, location-based query is a specialization of web search that considers the user’s
M AN U
geographic location; all online queries for this study were conducted in Pittsburgh, Pennsylvania.
In contrast to previous studies on readability of health information, this study shows that higher quality of online information regarding surgically managed diseases does not always translate to a decreased ease of readability. Such a trend is promising for physicians seeking to provide patients with online resources, and suggests that high quality medical information can be
EP
5. Conclusion
TE D
presented in such a way that does not sacrifice ease of readability.
The online information that patients and parents gather about laryngomalacia may have an effect
AC C
on the decision-making process regarding their child’s care. Websites relating information about laryngomalacia may not be comprehensible sources nor easily understandable to the general public. Healthcare providers should be aware of the quality of such information, as well as how the quality correlates with ease of readability, as this information may influence parental decision-making and consequently lead to bias towards treatment options.
ACCEPTED MANUSCRIPT
6. References 1) Baker, T.H. Wagner, S. Singer. Use of internet and email for health care information: Results from a national survey. JAMA (2003) 2400- 2406.
International Journal of Medical Informatics (2006) 29-34.
RI PT
2) M.L. Ybarra, M. Suman. Help seeking behavior and the internet: A national survey.
3) L. D. Holinger. Etiology of stridor in the neonate, infant and child. Ann Otol Rhinol
SC
Laryngol, 89 (1980) 397- 400.
4) P. Narcy, S. Bobin, P. Contencin, et al. Laryngeal anomalies in newborn infants. Apropos
M AN U
of 687 cases. Ann Otolaryngol Chir Cervicofac 101 (1984( 363- 373.
5) E. M. Friedman, A. P. Vastola, T. J. McGill, et al. Chronic pediatric stridor: etiology and outcome. Laryngoscope, 100 (1990) 277- 280.
6) R. Zoumalan, J. Maddalozzo, L. D. Holinger. Etiology of stridor in infants. Ann Otol
TE D
Rhinol Laryngol, 116 (2007) 329- 334.
7) D. A. DeWalt, A. Hink. Health literacy and child health outcomes: a systematic review of the literature. Pediatrics. 124 (2009) S265-74.
EP
8) H. S. Yin, R. S. Gupta, S. Tomopoulos, M. S. Wolf, A. L. Mendelsohn, L. Antler, D. C. Sanchez, C. H. Lau, B. P. Dreyer. Readability, suitability, and characteristics of asthma
AC C
action plans: examination of factors that may impair understanding. Pediatrics. 131(1) (2013) e116-26.
9) U. Alamoudi, P. Hong.
Readability and quality assessment of websites related to
microtia and aural atresia. Int J Pediatr Otorhinolaryngol. 79(2) (2015) 151-6.
10) L. Ritchie, C. Tornari, P. M. Patel, R. Lakhani. Glue ear: how good is the information on the World Wide Web? J Laryngol Otol. 130(2) (2016) 157-61.
ACCEPTED MANUSCRIPT
11) T. C. McKearney, R. M. McKearney. The quality and accuracy of internet information on the subject of ear tubes. Int J Pediatr Otorhinolaryngol. 77(6) (2013) 894-7. 12) D. Charnock, S. Shepperd, G. Needham, R. Gann. DISCERN: an instrument for judging
Community Health. 53(2) (1999) 105-11.
RI PT
the quality of written consumer health information on treatment choices. J Epidemiol
13) R. Flesch. "How to Write Plain English". University of Canterbury. Retrieved from:
SC
http://www.mang.canterbury.ac.nz/writing_guide/writing/flesch.shtml 14) Health On the Net Foundation. www.hon.ch (retrieved 07/26/2016)
M AN U
15) J. C. Hall. The readability of original articles in medical journals. ANZ Journal of
AC C
EP
TE D
Surgery. 76: 68–70 (2006). doi:10.1111/j.1445-2197.2006.03651.x.
ACCEPTED MANUSCRIPT
Table 1. List of websites retrieved from the following search engines on Tuesday, July 26, 2016 Advanced Search: “laryngomalacia” OR “soft larynx” OR “floppy voice box” Google [date of last update retrieved from page source] ++
1. 2. 3. 4. 5. 6.
http://emedicine.medscape.com/article/1002527-overview [6/18/2015] https://en.wikipedia.org/wiki/Laryngomalacia** [5/27/2016] ++ https://www.cincinnatichildrens.org/health/l/laryngomalacia-infantile [12/2013] ++ http://www.chop.edu/conditions-diseases/laryngomalacia/about#.V5fK-ZODGko [9/17/2014] ++ http://www.childrenshospital.vanderbilt.org/services.php?mid=9993** [7/20/2016] ++ http://www.hopkinsmedicine.org/otolaryngology/specialty_areas/pediatric_otolaryngology/pdf/l aryngomalacia_handout.pdf [2/24/2014] http://www.fairview.org/healthlibrary/Article/89835 [8/15/2011] http://www.copingwithlm.org/#!what-is-lm/c2n2 [Not reported] http://voicefoundation.org/health-science/voice-disorders/voice-disorders/pediatric-voicedisorders/understanding-pediatric-voice-disorders/** [6/2016] http://patient.info/doctor/congenital-throat-problems* [10/16/2014]
10.
M AN U
7. 8. 9.
SC
RI PT
++
Yahoo
http://emedicine.medscape.com/article/1002527-overview [6/18/2015] https://en.wikipedia.org/wiki/Laryngomalacia** [5/27/2016] ++ http://www.chop.edu/conditions-diseases/laryngomalacia/about#.V5fK-ZODGko [9/17/2014] ++ http://www.hopkinsmedicine.org/otolaryngology/specialty_areas/pediatric_otolaryngology/pdf/l aryngomalacia_handout.pdf [2/24/2014] ++ https://www.cincinnatichildrens.org/health/l/laryngomalacia-infantile [12/2013] + http://www.rightdiagnosis.com/l/laryngomalacia/intro.htm** [8/13/2015] ++ http://www.childrenshospital.vanderbilt.org/services.php?mid=9993** [7/20/2016] + http://www.notjustaboutwee.com/laryngomalacia/ [Not reported] + http://www.kidsgrowth.com/resources/articledetail.cfm?id=935 [Not reported] http://www.childrenshospital.org/conditions-and-treatments/conditions/laryngomalaciacongenital-laryngeal-stridor** [1/11/2016] ++
Bing ++
http://emedicine.medscape.com/article/1002527-overview [6/18/2015] https://www.cincinnatichildrens.org/health/l/laryngomalacia-infantile [12/2013] ++ http://www.chop.edu/conditions-diseases/laryngomalacia/about#.V5fK-ZODGko [9/17/2014] ++ https://en.wikipedia.org/wiki/Laryngomalacia** [5/27/2016] ++ http://www.hopkinsmedicine.org/otolaryngology/specialty_areas/pediatric_otolaryngology/pdf/l aryngomalacia_handout.pdf [2/24/2014] 26. +http://www.rightdiagnosis.com/l/laryngomalacia/intro.htm** [8/13/2015] 27. ++http://www.childrenshospital.vanderbilt.org/services.php?mid=9993** [7/20/2016] 28. +http://www.notjustaboutwee.com/laryngomalacia/ [Not reported] 29. +http://www.kidsgrowth.com/resources/articledetail.cfm?id=935 [Not reported] 30. https://rarediseases.info.nih.gov/gard/6865/laryngomalacia/resources/1 [10/22/2012] + Duplicated from one other search engine under study ++ Duplicated from two other search engines under study *HONcode accredited **Updated within the last 12 months ++
AC C
21. 22. 23. 24. 25.
TE D
15. 16. 17. 18. 19. 20.
++
EP
11. 12. 13. 14.
ACCEPTED MANUSCRIPT
RI PT
Figure 1: Correlation between total DISCERN score and FRES A positive correlation between total DISCERN score and FRES is shown (r=0.41, N=29, p=0.01), excluding one outlier. This depicts that an increase in quality as measured by the total
Figure 2: Fry readability graph
M AN U
SC
DISCERN score correlates with an increase in the reading ease of text.
Five unique online sources scored less than a 12th grade reading level. Eight unique online sources scored at a college level or above, and outside the range of the Fry graph. Two unique
AC C
EP
TE D
online sources scored in the college reading level range.
AC C
EP
TE D
M AN U
SC
RI PT
ACCEPTED MANUSCRIPT
AC C
EP
TE D
M AN U
SC
RI PT
ACCEPTED MANUSCRIPT
S0165-5876(18)30109-5
DOI:
10.1016/j.ijporl.2018.02.036
Reference:
PEDOT 8911
To appear in:
International Journal of Pediatric Otorhinolaryngology
Received Date: 17 December 2017 Revised Date:
20 February 2018
Accepted Date: 22 February 2018
Please cite this article as: E. Corredera, K.S. Davis, J.P. Simons, N. Jabbour, What parents are reading about laryngomalacia: Quality and readability of internet resources on laryngomalacia, International Journal of Pediatric Otorhinolaryngology (2018), doi: 10.1016/j.ijporl.2018.02.036. This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.
ACCEPTED MANUSCRIPT
What parents are reading about laryngomalacia: Quality and readability of Internet resources on laryngomalacia.
M AN U
b. Children's Hospital of Pittsburgh of UPMC Division of Otolaryngology 4401 Penn Avenue Faculty Pavilion, 7th Floor Pittsburgh, PA 15224 Admin Office: 412-692-5466 Fax: 412-692-6074
SC
Affiliations: a. University of Pittsburgh School of Medicine M240 Scaife Hall, 3550 Terrace St Pittsburgh, PA 15213 Admin Office: 412-648-9891
RI PT
Erica Correderaa, Kara S. Davisb, Jeffrey P. Simonsb, Noel Jabbourb
Corresponding Author:
TE D
Noel Jabbour, MD
EP
Children's Hospital of Pittsburgh of UPMC Division of Otolaryngology 4401 Penn Avenue Faculty Pavilion, 7th Floor Pittsburgh, PA 15224 Admin Office: 412-692-5466 Fax: 412-692-6074
AC C
[email protected]
Keywords: laryngomalacia; stridor; website; patient information; internet; readability
The authors have no conflicts of interest. Word Count: 2539 words
ACCEPTED MANUSCRIPT
Abstract:
RI PT
Objective: The goal of this study is to measure the quality and readability of websites related to laryngomalacia, and to compare the quality and readability scores for the sites accessed through the most popular search engines.
SC
Introduction: Laryngomalacia is a common diagnosis in children but is often difficult for parents to comprehend. As information available on the internet is unregulated, the quality and readability of this information may vary.
M AN U
Methods: An advanced search on Google, Yahoo, and Bing was conducted using the terms “laryngomalacia” OR “soft larynx” OR “floppy voice box.” The first ten websites meeting inclusion and exclusion criteria were evaluated, for each search engine. Quality and readability were assessed using the DISCERN criteria and the Flesch reading ease scoring (FRES) and Flesch-Kincaid grade level (FKGL) tests, respectively. Results: The top 10 hits on each search engine yielded 15 unique web pages. The median DISCERN score (out of a possible high-score of 80) was 48.5 (SD 12.6). The median USA grade-level estimated by the FKGL was 11.3 (SD 1.4). Only one website (6.7%), had a readability score in the optimal range of 6th to 8th grade reading level. DISCERN scores did not correlate with FKGL scores (r=0.10).
AC C
EP
TE D
Conclusion: Online information discussing laryngomalacia often varies in quality and may not be easily comprehensible to the public. It is important for healthcare professionals to understand the quality of health information accessible to patients as it may influence medical decision-making by patient families.
ACCEPTED MANUSCRIPT
1. Introduction
The utilization of the internet as a source of medical information has increased immensely over
RI PT
the last twenty years. Many patients now rely on the internet as part of their medical decisionmaking [1, 2]. Certain populations are more likely to use the internet as a source of advice on the medical decision-making, due to its immediate availability and free content. The obtainment of
SC
online medical information should naturally progress to understanding and application of the information, which requires some degree of health literacy. A particular population that must be
children that require medical treatment.
M AN U
considered in regards to the effect of health literacy on the healthcare process is parents of
Laryngomalacia is a leading cause of pediatric stridor and is one of the most common pediatric diagnoses. Breathing and airway symptoms often generate high stress levels amongst family
TE D
members and thus lead to information-seeking behavior [3, 4, 5, 6]. The purpose of this study was to evaluate the quality and ease of readability for websites related to laryngomalacia. Decreased levels of parent health literacy have been associated with higher rates of pediatric
EP
emergency department visits, non-adherence to treatment plans, and difficulty understanding healthcare-related tools such as growth charts [7, 8]. Evidence has shown that high-quality
11].
AC C
websites with easily read content are lacking and there is wide variability in the content [9, 10,
The online information that parents gather about laryngomalacia most likely has an effect on the decision-making process regarding their child’s care. This study aimed to (1) measure the quality and readability of websites related to laryngomalacia, and (2) to compare the quality and readability scores for the most popular websites on each of the three most widely used search
ACCEPTED MANUSCRIPT
engines. Quality and readability are assessed using the DISCERN criteria and the Flesch reading ease scoring (FRES) and Flesch-Kincaid grade level (FKGL) tests, respectively. The DISCERN instrument is a validated rating tool and the first standardized quality index of consumer health
RI PT
information that was created and made available for content producers, health professionals, and patients to appraise written information on treatment options [12]. The Flesch-Kincaid readability tests consist of two tests— Flesch Reading Ease Score (FRES) and Flesch-Kincaid
SC
Grade Level (FKGL)— that indicate the level of difficulty for a reading passage. The hypothesis of this study was that an increase in the DISCERN score, as a metric of quality, correlated with
M AN U
decreased readability scores, represented by a low ease of reading score (FRES) and a high reading level (FKGL).
2. Methods
TE D
2.1 Strategy
An advanced search was conducted on the Google, Yahoo, and Bing search engines using the “OR” operator in order to browse the following string term on a single date, July 26, 2016:
EP
“laryngomalacia” OR “soft larynx” OR “floppy voice box”. The online query was conducted in Pittsburgh, Pennsylvania. The first 10 websites meeting inclusion and exclusion criteria on each
AC C
search engine were evaluated. Websites must have been documented in the English language and must have been functional at the time of access for inclusion. Exclusion criteria included newspaper articles, sponsored websites, information consisting of less than 30 sentences, imagesonly websites, and online dictionaries.
Quality and readability were assessed using the DISCERN criteria and the Flesch reading ease scoring (FRES) and Flesch-Kincaid grade level (FKGL) tests, respectively. The written content
ACCEPTED MANUSCRIPT
was copied from each website into a word processor. Identifying information, such as hospital/institution affiliations, were removed to minimize bias. All included websites were independently reviewed by two attending pediatric otolaryngologists (NJ, JPS) for quality using
RI PT
the DISCERN criteria questions [12]. When the average DISCERN score for each publication was within 1 point difference (out of 5) between the raters, the scores for that publication were averaged without further consensus review. For all other scores with greater than 1 point
SC
discrepancy between the raters, a consensus DISCERN score was made by simultaneous
M AN U
repeated review by both reviewers.
The written content was then evaluated in Microsoft Word for Mac Version 15.15 (Microsoft Corp., Redmond, WA) for the FRES and FKGL by a single evaluator (EC). The mean DISCERN score (out of a possible high-score of 80), mean rating of publication, mean ease of readability score, and mean reading level was independently calculated for each search engine and also
TE D
averaged to obtain a single value for each measure of quality and readability.
2.2 Measures DISCERN
AC C
2.2.1
EP
Date of last update and the HONcode accreditation were recorded as of July 26, 2016.
While using the DISCERN validated rating tool, websites are given a total rating out of 80 points that is based on 16 key questions assessing the site’s content. Questions 1-8 emphasize the reliability of the publication and encourage one to consider its dependability as a source of medical information. Questions 9-15 address specifics of the treatment options. Question 16 represents the overall rating of publication. Each question is rated on a 5-point scale ranging
ACCEPTED MANUSCRIPT
from “No” to “Yes”, with a 1 defined as a definitive “No”, a 2 representing serious/extensive shortcomings, a 3 meaning the content is potentially important with no serious shortcomings, a 4 affirming minimal shortcomings, and a 5 characterizing a definitive “Yes” and complete
Flesch-Kincaid readability
SC
2.2.2
RI PT
fulfillment of criteria.
The formula to calculate the FRES is as follows: 206.835 - (1.015 x average sentence length) -
M AN U
(84.6 x average number of syllables per word). Average sentence length is measured as total words per total sentences. A score is determined from 0 to 100, with higher scores indicating greater ease of reading [13].
The formula to calculate the FKGL is as follows: 0.39*(average number of words per sentence)
TE D
+ 11.8*(average number of syllables per word) - 15.59. The FKGL score represents a U.S. grade level or the number of years of education generally required to understand the text under evaluation. For example, a score of 8.0 would indicate that the text is likely understandable by an
2.2.3
AC C
years [13].
EP
average student in the 8th grade of the U.S. education system, which is often between ages 13-15
Fry Readability
The Fry readability is a readability metric that determines the grade reading level of a passage of written text. The intersection between average number of sentences and syllables per 100 words on a fry graph determines the grade reading level.
ACCEPTED MANUSCRIPT
2.2.4
HONcode
The Health On the Net Foundation (HON) is a non-profit, non-governmental organization,
RI PT
accredited to the Economic and Social Council of the United Nations. The HONcode certification is an ethical standard aimed at offering quality health information. The HONcode certification symbol is identifiable through a downloadable toolbar extension, and will become
SC
active if a website is HONcode certified [14]. For each website, the HONcode seal was checked on a single date (July 26, 2016).
Statistics
M AN U
2.2.5
Descriptive statistics were calculated from all data, using the mean, range, and standard deviation, when applicable, for reporting. D’Agostino and Pearson omnibus normality test was
TE D
run on DISCERN, FKGL and FRES data to determine normality of the data sets. A correlation study was then performed using the Pearson’s 1-tailed test. Inter-rater agreement was tested
3. Results
EP
using Spearman's correlation.
AC C
3.1 Website characteristics
The websites retrieved from Google, Yahoo, and Bing are listed in Table 1. A significant number of the websites evaluated were owned by academic institutions (n=5, 33.3% of a total of 15 unique websites). Of the websites recorded from all three search engines with a listed ‘last date of update’ retrieved from the page source, 5 (33.3%) of the unique websites had been updated in the previous 12 months, 33% (3/9) from the Google search, 50% (4/8) from the Yahoo search, 37.5% (3/8) from the Bing search, as noted in Table 1.
ACCEPTED MANUSCRIPT
3.2 Measures 3.2.1
DISCERN
RI PT
The mean DISCERN score was 46.1. The median DISCERN score (out of a possible high-score of 80) was 48.5 (SD 12.6). The independent mean DISCERN score for the Google, Yahoo, and Bing websites was 48.5, 45.3, and 44.6, respectively. The highest scoring website was the
SC
Children’s Hospital of Philadelphia laryngomalacia page, with a DISCERN score of 60.5/80. The lowest scoring website was the Right Diagnosis laryngomalacia page, with a DISCERN
M AN U
score of 16.5/80. The mean rating of publication based on question-16 in the DISCERN score for websites recorded was 3.1, 2.9, and 2.8 from the Google, Yahoo, and Bing search, respectively, each falling into the ‘partially valid source of information’ category on a 1-5 scale.
3.2.2
Flesch-Kincaid readability
TE D
The ease of readability score estimated by the FRES test was 40.7 before removal of source identifiers and 41.8 after removal of source identifiers. The independent ease of readability score
EP
estimated by the Flesch reading ease scoring test (FRES) was 41.9 for the Google websites, 40.8 for the Yahoo websites, and 39.4 for the Bing websites before removal of source identifiers; 43.5
AC C
for the Google websites, 42.0 for the Yahoo websites, and 39.9 for the Bing websites after the removal of source identifiers.
The mean reading level estimated by the FKGL test was 11.3 before removal of source identifiers and 10.9 after removal of source identifiers. The median USA grade-level estimated by the FKGL was 11.3 (SD 1.4). The independent mean reading level estimated by the FleschKincaid grade level (FKGL) test was 11.1 for the Google websites, 11.3 for the Yahoo websites,
ACCEPTED MANUSCRIPT
and 11.4 for the Bing websites before the removal of source identifiers; 10.7 for the Google websites, 10.9 for the Yahoo websites, and 11.2 for the Bing websites after the removal of source
3.2.3
Correlation with DISCERN and readability
RI PT
identifiers.
SC
To examine the hypothesis that a greater DISCERN score, as a metric of quality, would correlate with decreased readability scores as represented by a low score on FRES and a high grade level
M AN U
(FKGL), a correlation study between DISCERN and the readability metrics was performed. DISCERN, FRES, and FKGL data demonstrated a normal distribution through the D’Agostino and Pearson omnibus normality test, indicating suitability of using the Pearson’s 1-tailed correlation test. There was a positive correlation between DISCERN score and FRES when a
TE D
Pearson correlation test was conducted on the entirety of the websites used in this study (r=0.41, N=29, p=0.01), excluding one outlier, as depicted in Figure 1. The correlation between total
EP
DISCERN score and FKGL was not statistically significant (r=0.14, N=30, p=0.22).
Isolated correlation studies were also conducted comparing quality and readability for the sites
AC C
recorded from each search engine. The correlation between total DISCERN score and FRES was not statistically significant for Google (r=-0.11, N=10, p=0.75), Yahoo (r=0.53, N=10, p=0.11) or Bing (r=0.42, N=10, p=0.22). The correlation between total DISCERN score and FKGL was not statistically significant for Google (r=0.47, N=10, p=0.17), Yahoo (r=0.12, N=10, p=0.74), and Bing (r=-0.02, N=10, p=0.94).
3.2.4
Fry readability
ACCEPTED MANUSCRIPT
Five unique online sources scored less than a 12th grade reading level. Two unique online sources scored in the college reading level range. Eight unique online sources scored at a college
One
HONcode Accreditation
website
from
the
search
was
HONcode
SC
3.2.5
RI PT
level or above, and outside of the range of the Fry graph (Figure 2).
accredited:
http://patient.info/doctor/congenital-throat-problems. None of the websites associated with
M AN U
academic institutions were HONcode certified.
4. Discussion
The goals of this study were to (1) measure the quality and readability of websites related to laryngomalacia, and (2) to compare the quality and readability scores for the most popular
TE D
websites on each of the three most widely used search engines in the world. Overall, laryngomalacia related websites were found to be of fair quality by DISCERN criteria, with wide
EP
variability. There were 9 Google, 10 Yahoo, and 10 Bing websites with a readability score less than the suggested readability score of 65 for public health information; only one Google website
AC C
achieved the suggested readability score of at least 65. The FKGL estimated average was the 10th grade reading level, which is above the recommended reading level of grade 6-8 for public health information.
Online resources suggest that medical information especially related to surgical specialties is often difficult to read and that authors should aim for FRES scores of at least 30 in order to maintain a higher degree of quality [15]. Previous studies have shown a negative correlation
ACCEPTED MANUSCRIPT
between ease of readability and quality of information. Studies evaluating online information regarding other pediatric otolaryngologic diagnoses such as glue ear, microtia, and tympanostomy tubes have also reported generally low-quality, difficult to read websites [8, 9,
RI PT
10]. In the results published on aural atresia and microtia, the websites with the most reliable content had the highest reading levels, potentially reducing their usefulness as a healthcare tool to parents and adolescents with access to the internet. In this study, quality as measured by the
SC
DISCERN score showed a positive correlation with FRES (Figure 1), thus rejecting the notion
M AN U
that higher quality translates to increased reading difficulty.
Only one website in this study (3.33% overall), taken from the Google search (10% of Google sources), was HONcode accredited; statistical analyses comparing websites with and without HONcode accreditation was infeasible. Anecdotally, the HONcode accredited website had an average FRES of 28.95 and was estimated to be in the 12th grade reading level, qualifying as
TE D
fairly difficult. Like the DISCERN tool, HONcode certification does not address the accuracy of the information nor does it account for omitted information, an important limitation to consider
EP
when counseling patients and family members on entities such as HONcode. The value of HONcode certification is that it does address some common issues with quality by ensuring that
AC C
websites define their intended audience, provide an author or editors name, list the date that the material was updated, disclose funding sources, and avoid advertisements, among other reasonable requirements. With increased use, consumers could become accustomed to look for this certification when searching the internet for healthcare related questions.
ACCEPTED MANUSCRIPT
Additionally, the number of online sources studied was limited to 30 and required that the content be written in the English language. The subset of keywords chosen for the search (“laryngomalacia” OR “soft larynx” OR floppy voice box”) introduce a congruence bias due to
RI PT
the assumption that the general population will most likely search the same terms. Furthermore, the information presented in this study is based on a search conducted on a single date, July 26, 2016; websites may be added and content from existing websites may be altered at any time.
SC
Lastly, location-based query is a specialization of web search that considers the user’s
M AN U
geographic location; all online queries for this study were conducted in Pittsburgh, Pennsylvania.
In contrast to previous studies on readability of health information, this study shows that higher quality of online information regarding surgically managed diseases does not always translate to a decreased ease of readability. Such a trend is promising for physicians seeking to provide patients with online resources, and suggests that high quality medical information can be
EP
5. Conclusion
TE D
presented in such a way that does not sacrifice ease of readability.
The online information that patients and parents gather about laryngomalacia may have an effect
AC C
on the decision-making process regarding their child’s care. Websites relating information about laryngomalacia may not be comprehensible sources nor easily understandable to the general public. Healthcare providers should be aware of the quality of such information, as well as how the quality correlates with ease of readability, as this information may influence parental decision-making and consequently lead to bias towards treatment options.
ACCEPTED MANUSCRIPT
6. References 1) Baker, T.H. Wagner, S. Singer. Use of internet and email for health care information: Results from a national survey. JAMA (2003) 2400- 2406.
International Journal of Medical Informatics (2006) 29-34.
RI PT
2) M.L. Ybarra, M. Suman. Help seeking behavior and the internet: A national survey.
3) L. D. Holinger. Etiology of stridor in the neonate, infant and child. Ann Otol Rhinol
SC
Laryngol, 89 (1980) 397- 400.
4) P. Narcy, S. Bobin, P. Contencin, et al. Laryngeal anomalies in newborn infants. Apropos
M AN U
of 687 cases. Ann Otolaryngol Chir Cervicofac 101 (1984( 363- 373.
5) E. M. Friedman, A. P. Vastola, T. J. McGill, et al. Chronic pediatric stridor: etiology and outcome. Laryngoscope, 100 (1990) 277- 280.
6) R. Zoumalan, J. Maddalozzo, L. D. Holinger. Etiology of stridor in infants. Ann Otol
TE D
Rhinol Laryngol, 116 (2007) 329- 334.
7) D. A. DeWalt, A. Hink. Health literacy and child health outcomes: a systematic review of the literature. Pediatrics. 124 (2009) S265-74.
EP
8) H. S. Yin, R. S. Gupta, S. Tomopoulos, M. S. Wolf, A. L. Mendelsohn, L. Antler, D. C. Sanchez, C. H. Lau, B. P. Dreyer. Readability, suitability, and characteristics of asthma
AC C
action plans: examination of factors that may impair understanding. Pediatrics. 131(1) (2013) e116-26.
9) U. Alamoudi, P. Hong.
Readability and quality assessment of websites related to
microtia and aural atresia. Int J Pediatr Otorhinolaryngol. 79(2) (2015) 151-6.
10) L. Ritchie, C. Tornari, P. M. Patel, R. Lakhani. Glue ear: how good is the information on the World Wide Web? J Laryngol Otol. 130(2) (2016) 157-61.
ACCEPTED MANUSCRIPT
11) T. C. McKearney, R. M. McKearney. The quality and accuracy of internet information on the subject of ear tubes. Int J Pediatr Otorhinolaryngol. 77(6) (2013) 894-7. 12) D. Charnock, S. Shepperd, G. Needham, R. Gann. DISCERN: an instrument for judging
Community Health. 53(2) (1999) 105-11.
RI PT
the quality of written consumer health information on treatment choices. J Epidemiol
13) R. Flesch. "How to Write Plain English". University of Canterbury. Retrieved from:
SC
http://www.mang.canterbury.ac.nz/writing_guide/writing/flesch.shtml 14) Health On the Net Foundation. www.hon.ch (retrieved 07/26/2016)
M AN U
15) J. C. Hall. The readability of original articles in medical journals. ANZ Journal of
AC C
EP
TE D
Surgery. 76: 68–70 (2006). doi:10.1111/j.1445-2197.2006.03651.x.
ACCEPTED MANUSCRIPT
Table 1. List of websites retrieved from the following search engines on Tuesday, July 26, 2016 Advanced Search: “laryngomalacia” OR “soft larynx” OR “floppy voice box” Google [date of last update retrieved from page source] ++
1. 2. 3. 4. 5. 6.
http://emedicine.medscape.com/article/1002527-overview [6/18/2015] https://en.wikipedia.org/wiki/Laryngomalacia** [5/27/2016] ++ https://www.cincinnatichildrens.org/health/l/laryngomalacia-infantile [12/2013] ++ http://www.chop.edu/conditions-diseases/laryngomalacia/about#.V5fK-ZODGko [9/17/2014] ++ http://www.childrenshospital.vanderbilt.org/services.php?mid=9993** [7/20/2016] ++ http://www.hopkinsmedicine.org/otolaryngology/specialty_areas/pediatric_otolaryngology/pdf/l aryngomalacia_handout.pdf [2/24/2014] http://www.fairview.org/healthlibrary/Article/89835 [8/15/2011] http://www.copingwithlm.org/#!what-is-lm/c2n2 [Not reported] http://voicefoundation.org/health-science/voice-disorders/voice-disorders/pediatric-voicedisorders/understanding-pediatric-voice-disorders/** [6/2016] http://patient.info/doctor/congenital-throat-problems* [10/16/2014]
10.
M AN U
7. 8. 9.
SC
RI PT
++
Yahoo
http://emedicine.medscape.com/article/1002527-overview [6/18/2015] https://en.wikipedia.org/wiki/Laryngomalacia** [5/27/2016] ++ http://www.chop.edu/conditions-diseases/laryngomalacia/about#.V5fK-ZODGko [9/17/2014] ++ http://www.hopkinsmedicine.org/otolaryngology/specialty_areas/pediatric_otolaryngology/pdf/l aryngomalacia_handout.pdf [2/24/2014] ++ https://www.cincinnatichildrens.org/health/l/laryngomalacia-infantile [12/2013] + http://www.rightdiagnosis.com/l/laryngomalacia/intro.htm** [8/13/2015] ++ http://www.childrenshospital.vanderbilt.org/services.php?mid=9993** [7/20/2016] + http://www.notjustaboutwee.com/laryngomalacia/ [Not reported] + http://www.kidsgrowth.com/resources/articledetail.cfm?id=935 [Not reported] http://www.childrenshospital.org/conditions-and-treatments/conditions/laryngomalaciacongenital-laryngeal-stridor** [1/11/2016] ++
Bing ++
http://emedicine.medscape.com/article/1002527-overview [6/18/2015] https://www.cincinnatichildrens.org/health/l/laryngomalacia-infantile [12/2013] ++ http://www.chop.edu/conditions-diseases/laryngomalacia/about#.V5fK-ZODGko [9/17/2014] ++ https://en.wikipedia.org/wiki/Laryngomalacia** [5/27/2016] ++ http://www.hopkinsmedicine.org/otolaryngology/specialty_areas/pediatric_otolaryngology/pdf/l aryngomalacia_handout.pdf [2/24/2014] 26. +http://www.rightdiagnosis.com/l/laryngomalacia/intro.htm** [8/13/2015] 27. ++http://www.childrenshospital.vanderbilt.org/services.php?mid=9993** [7/20/2016] 28. +http://www.notjustaboutwee.com/laryngomalacia/ [Not reported] 29. +http://www.kidsgrowth.com/resources/articledetail.cfm?id=935 [Not reported] 30. https://rarediseases.info.nih.gov/gard/6865/laryngomalacia/resources/1 [10/22/2012] + Duplicated from one other search engine under study ++ Duplicated from two other search engines under study *HONcode accredited **Updated within the last 12 months ++
AC C
21. 22. 23. 24. 25.
TE D
15. 16. 17. 18. 19. 20.
++
EP
11. 12. 13. 14.
ACCEPTED MANUSCRIPT
RI PT
Figure 1: Correlation between total DISCERN score and FRES A positive correlation between total DISCERN score and FRES is shown (r=0.41, N=29, p=0.01), excluding one outlier. This depicts that an increase in quality as measured by the total
Figure 2: Fry readability graph
M AN U
SC
DISCERN score correlates with an increase in the reading ease of text.
Five unique online sources scored less than a 12th grade reading level. Eight unique online sources scored at a college level or above, and outside the range of the Fry graph. Two unique
AC C
EP
TE D
online sources scored in the college reading level range.
AC C
EP
TE D
M AN U
SC
RI PT
ACCEPTED MANUSCRIPT
AC C
EP
TE D
M AN U
SC
RI PT
ACCEPTED MANUSCRIPT