- Email: [email protected]
The Disparity Between Public Utilization and Surgeon Awareness of the STS Patient Education Website
Journal Pre-proof The disparity between public utilization and surgeon awareness of the STS patient education website Robbin G. Cohen, MD, MMM, S. Ram Kumar, MD PhD, Jules Lin, MD, Rishindra M. Reddy, MD, Lauren Kane, MD, Jennifer Bagley, MA, Armando Juarez, BA, Fernando Fleischman, MD, Emily A. Farkas, MD, Amy E. Hackmann, MD, Kendra J. Grubb, MD, Seenu Reddy, MD, Loretta Erhunmwunsee, MD, Nestor R. Villamizar, MD, Muhammad F. Masood, MD, Melissa Griffin, PA-C, Natalie Boden, MBA PII:
S0003-4975(19)31700-X
DOI:
https://doi.org/10.1016/j.athoracsur.2019.09.074
Reference:
ATS 33218
To appear in:
The Annals of Thoracic Surgery
Received Date: 9 March 2019 Revised Date:
27 July 2019
Accepted Date: 23 September 2019
Please cite this article as: Cohen RG, Kumar SR, Lin J, Reddy RM, Kane L, Bagley J, Juarez A, Fleischman F, Farkas EA, Hackmann AE, Grubb KJ, Reddy S, Erhunmwunsee L, Villamizar NR, Masood MF, Griffin M, Boden N, The disparity between public utilization and surgeon awareness of the STS patient education website, The Annals of Thoracic Surgery (2019), doi: https://doi.org/10.1016/ j.athoracsur.2019.09.074. This is a PDF file of an article that has undergone enhancements after acceptance, such as the addition of a cover page and metadata, and formatting for readability, but it is not yet the definitive version of record. This version will undergo additional copyediting, typesetting and review before it is published in its final form, but we are providing this version to give early visibility of the article. Please note that, during the production process, errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain. © 2019 by The Society of Thoracic Surgeons
1 The disparity between public utilization and surgeon awareness of the STS patient education website Running head: STS patient education website data
Robbin G. Cohen, MD, MMM2, S. Ram Kumar MD PhD2, Jules Lin MD4, Rishindra M. Reddy MD4, Lauren Kane MD3, Jennifer Bagley, MA1, Armando Juarez, BA1, Fernando Fleischman MD2, Emily A. Farkas MD6, Amy E. Hackmann MD2, Kendra J. Grubb MD7, Seenu Reddy MD8, Loretta Erhunmwunsee MD9, Nestor R. Villamizar10 MD, Muhammad F. Masood MD11, Melissa Griffin PA-C5, and Natalie Boden, MBA1
1
Society for Thoracic Surgeons Media Office, Chicago, Illinois, USA
2
Department of Surgery, Keck School of Medicine of USC, University of Southern California, Los
Angeles, CA, USA 3
The Heart Center at Arnold Palmer Hospital for Children, University of Central Florida, Orlando,
FL, USA 4
Department of Surgery, University of Michigan, Ann Arbor, MI, USA
5
Huntington Hospital, Los Angeles, CA, USA
6
ThedaCare Appleton Heart Institute, Appleton, WI, USA
7
Department of Surgery, Emory School of Medicine, Atlanta, GA, USA
8
TriStar Cardiovascular Surgery, Nashville, TN, USA
9
Department of Surgery, City of Hope Comprehensive Cancer Center, Los Angeles, CA, USA
10
Sylvester Comprehensive Cancer Center, University of Miami Health System, Miami, FL, USA
11
Department of Surgery, Washington University School of Medicine in St. Louis, MO, USA
Title Character Count: 99 Short Title Character Count: 34
2 Abstract Word Count: 250 Manuscript Word Count: 3530
Corresponding Author:
Robbin G. Cohen, MD MMM Associate Professor of Cardiothoracic Surgery Department of Surgery Keck School of Medicine of USC University of Southern California Los Angeles, CA, USA [email protected]
3 Abstract Background: Many online resources currently provide healthcare information to the public. In 2015, The Society of Thoracic Surgeons (STS) created a multimedia web portal (ctsurgerypatients.org) to educate the public regarding cardiothoracic surgery and to provide an informative tool to which cardiothoracic surgeons could refer patients.
Methods: A patient education task force was created, and disease-specific content created for 25 pathological conditions. After launching the website online, a marketing campaign was initiated to make STS members aware of its availability. Website visits were monitored, and an online survey for public users created. An email survey was sent to STS members to evaluate awareness and content. Surveys were analyzed for effectiveness and utilization by both public users and STS member surgeons.
Results: From 2016-2018, the website had more than 1 million visits, with visits increasing yearly. Surveyed user ratings of the website were positive regarding quality and utility of the information provided. STS member response was poor (379 responses/6347 emails) and 78.3% of responders were unaware of the website. Surgeon responders were positive about the content, though many still refrain from referring patients.
Conclusions: Online education for cardiothoracic surgery is seeing increased public use, with high ratings for content and utility. Despite aggressive marketing to STS members, most remain unaware of this website’s existence. Those who are aware approve of its content, but adoption of referring patients to it has been slow. Improved strategies are necessary to make surgeons aware of this STS-provided service and to increase patient referrals to it.
4 A growing number of patients consult online resources for medical information, not only to learn about their own health issues and therapeutic options, but also to evaluate the background and reputation of their physicians. These resources are vast in number and vary from physiciandriven marketing tools to information resources developed by major disease-specific organizations, such as the American Heart Association or American Cancer Society. In 2015, the Workforce on Media Relations and Communications of The Society of Thoracic Surgeons (STS) created a Patient Information Task Force for the purpose of overhauling its patient information website into a comprehensive, multimedia, bilingual online education resource (ctsurgerypatients.org) for cardiothoracic surgical patients and their families. The website was also designed to be used as a reference tool to which STS member surgeons could refer their patients. The methodology and structure of the website are described, as well as an evaluation of its success using survey results both from patients utilizing the website and STS members.
Material and Methods Content Development: An eight-member Patient Education Task Force was created with two surgeon members representing each specialty category: adult cardiac surgery, adult thoracic surgery, congenital heart surgery, and heart and lung transplantation. Each specialty category was then divided into the most common surgical diseases treated by STS members (Table 1). In addition, a pre- and postoperative care section was created and similarly divided into specialty categories. Over a 1-year period, content for each section was developed by the STS Marketing & Communications Department, in conjunction with surgeon task force members. Content was organized by disease (as opposed to procedure) in order to maximize the potential for our website to appear on Google searches. Graphics on the website required STS ownership and were limited to previously published graphics from The Annals of Thoracic Surgery, web content for which STS received usage permission, and original graphics created by the STS Marketing & Communications Department. Topic-specific videos featuring expert
5 STS member surgeons, as well as two- and three-dimensional video animations depicting surgical anatomy and disease states, were also created internally (Figure 1). Final content was reviewed during weekly phone conferences prior to placement on the website. All text was also translated from English into Spanish.
In addition to didactic content and graphics, a news section was created containing current topics in cardiothoracic surgery, as well as press releases highlighting current manuscripts from The Annals of Thoracic Surgery. A biweekly blog was also created, covering a broad range of cardiothoracic surgical topics and written by STS members and patients.
Marketing: Once content was completed, reviewed, and posted on the internet, an extensive marketing strategy was implemented to raise awareness of the website utilizing colorful advertisements in The Annals of Thoracic Surgery, STS News, STS e-newsletters, and at the two subsequent STS Annual Meetings. In 2016, this included 8 full page advertisements in The Annals of Thoracic Surgery, 7 posts in the STS News, multiple bag inserts and fliers at the Annual Meetings of the STS, AATS, ESTS, and EACTS, 14 electronic messages to attendees of those meetings, and 28 emails to all STS members or data managers regarding our existence, new content, or blogs.
Evaluation: Numbers of visits to the website as well as navigations to specific pages were monitored over a 3-year period from 2016-2018. Users: A seven-question survey was created as a pop-up window requesting that website users evaluate our patient education portal. In addition, an eight question survey was emailed to 6347 STS surgeon member email addresses regarding their awareness of the web portal, their evaluation of the content, and whether or not they currently referred patients to
6 ctsurgerypatients.org (Table 2). Follow-up reminder emails from the STS President and in the STS weekly email were sent to the same addresses 9 and 13 days later respectively. Survey results were analyzed to determine awareness of our website, the quality and utility of our content, suggestions for content improvement, and whether or not STS surgeon members were referring patients to our portal. A net promoter score was determined for the question “Was the content useful”, using a scale of 1-10. The website was also analyzed regarding numbers of hits and most popular components.
Results Online visits to ctsurgerypatients.org for 2016 through 2018 are depicted in Figure 2. Visits doubled from 2016-2017 and increased another 23% from 2017-2018. The most commonly visited pages by content and disease are seen in Figure 3. The “What is a cardiothoracic surgeon” page had always been the most visited page when it was on the original STS website and continued to be the most visited page on the new patient education portal through 2017. As of 2018, it has been surpassed by the Spanish versions of “After Heart Surgery” and “End-Stage Lung Disease”. In fact, the number of visitors from predominantly Spanish speaking countries to the top 10 most accessed pages during 2018 exceeded those from English speaking countries (186,438 vs. 158,412).
There were 1038 responses to the patient user survey. Seventy-five percent of responders found the site via a search engine, whereas only 10% were referred by surgeons. The net promoter score for the usefulness of content is depicted in Figure 4, with more than 70% of respondents scoring 7 or greater on a scale of 1-10. Ninety three percent of responders felt that the website was easy to navigate, and almost 90% would recommend the site to a friend. Only 36.5% of respondents used the website to make a surgical decision.
7 There were 379 responses to the surgeon member survey for a response rate of 6%. Of those responding, 78.3% said that they had not previously heard of ctsurgerypatients.org. However, almost 90% of responders felt that the information provided on the website was relevant to their practices, and 83% felt that the information was presented in a useful way. When asked why they didn’t refer patients to the website, responses included that they didn’t know about it, didn’t think about it, provided their own written materials, or preferred to explain things to their patients themselves. Forty-five percent responded that they would be willing to contribute to the website.
Comment Over the past 2 decades, the internet has become populated with an abundance of online educational materials targeting the public with respect to prevalence, diagnosis, and treatment of a plethora of health-related conditions. These include a wide range of pharmaceutical marketing materials, as well as promotional sites for physicians, medical groups, universities and payers. They also include informational sites developed by condition-specific, non-profit health-related organizations, whose purpose is to bring attention to their causes and serve as an accurate, unbiased educational resource for patients and their families who may be affected by these conditions. The American Heart Association and The Marfan Foundation are excellent examples of this online educational format, which is both promotional and educational. Our goal was to have our patient education portal falls into this category.
In 2014, the STS committed to upgrading its online patient education portal to a more independent and comprehensive website. The goal was to provide easily understood, multimedia content to the public regarding the diagnosis and surgical therapy of a broad range of adult and pediatric cardiothoracic conditions. The content needed to be readily accessible and searchable to consumers on the internet. It also needed to be comprehensive but unbiased. For instance, we were careful not to create promotional material that would serve one surgeon
8 over another, such as recommending a “minimally invasive” incision over sternotomy for cardiac valve surgery. In addition, we wanted to create an online tool to which STS surgeon members would refer patients and their families, to further educate them and make them comfortable with their surgical decisions.
There were a number of challenges inherent to this process, many of which have been previously identified.1 The first was to provide content that was understandable and useful to the general public. Health literacy, defined as the “capacity to obtain, interpret, and understand basic health information and services, and the competence to use such information and services to enhance one’s own health,” is adequate in only 12% of the US adult population.2 Of the hundreds of medical and surgical informational websites currently available online, most fail to comply with the standards set by the National Institutes of Health (NIH), which calls for a Readability Grade Level (RGL) of between 6th and 7th grade. The average American adult reads at approximately the 8th grade level.3
Despite our best intentions, it was virtually impossible for our surgeon content developers to consistently write at or below the NIH prescribed 7th grade level. In fact, much of our content contains medical vocabulary and sophisticated graphics depicting anatomic and physiologic details. Despite this, more than 90% of user respondents said that our content was easy to navigate, 80% stated that it was useful, and more than 90% said they would recommend our website to a friend. Success of our user experience was also reflected in the Net Promotor Score (NPS) of 37. The NPS is a management tool used by business entities to gauge customer experience and loyalty. Customers who respond with a score of 9-10 are deemed “Promotors” and are more likely to remain customers and serve as a referral base to others. A score of greater than zero is deemed “good”, with a score of 50 considered “excellent”4. These indicators led us to believe that our multimedia platform was accomplishing our goals. Others
9 have reported better-than-expected patient-reported comprehension of online surgical material.4 One reason may be that consumers with the sophistication to search for healthcare related information online have a higher level of comprehension than those called for by the NIH. Another might be that the addition of animated graphics and videos may enhance overall comprehension of the material, despite the sophistication of the text. Finally, as our site is unlikely to be the sole source of information to patients, the additional information or reiteration of previously obtained content may be better comprehended even if presented in a more comprehensive format. Regarding utilization of our material, it is not surprising that only slightly more than one-third of respondents used our content to actually make a surgical decision, as most patients have probably chosen the recommendation of their surgeon before leaving their surgeon’s office. Still, we are confident that our online information was useful for clarifying and reiterating what the patients and their families have already heard from their surgeon.
Data regarding website and page visits demonstrate consistently increasing us of the website by the public. It is particularly interesting that 3 of the 4 most visited pages in 2018 are the Spanish versions. This undersocres the diversity of patients in search for information regarding cardiothoracic surgery, as well as the international preponderance of information on the internet. Given the complexities of translating and updating multilingual websites, expanding ctsurgerypatients.org into additional languages will require significantly more resources than currently exist at the STS. Updating the website with new or more effective content is a continuous project. During a complete review and update of the pages that were initiated in 2017 and completed in December 2018, new disease topics were identified and have subsequently been added and expanded content is being developed for procedures that cardiothoracic surgeons perform.
The STS Patient Education website was recreated under the assumption that the public,
10 especially patients and their families with cardiothoracic surgical disorders, would benefit from a current, factual, unbiased, usable website that explains their clinical entities and treatment options. We also assumed that the STS member surgeons would approve of the information and refer patients to it. Results from our member surgeon survey suggest that despite patient satisfaction with our online content, surgeons either don’t know about it or are aware of it but are not utilizing it. This was despite a sizable advertising campaign and strong agreement from surgeon respondents who have viewed the material that the content is both relevant and presented in a useful way. The low level of awareness of the website was surprising, given the large-scale effort to advertise the website at the STS Annual Meeting, in STS news, and in The Annals of Thoracic Surgery. The STS Marketing and Communications Department continues to explore new and novel ways to raise awareness and increase utilization of member related services such as this one. Potential options include even more aggressive and novel advertising, more demonstrations of website content in between scientific sessions and the Annual Meeting, as well as providing surgeons’ offices with written materials aimed at guiding patients to the website.
Regarding the response to our online survey, physicians are notorious for poor response rates.6 In fact, the expected response rate for an STS-generated physician survey is 15-30%.7 Though incentives, such as financial rewards for completing surveys, are known to increase physician response rates, this is not possible for a nonprofit specialty society such as the STS.8 Regardless, the information that we received was useful, especially when it came to determining the rationale for why surgeons who were aware of the patient education website were not utilizing it. Many surgeons feel that their own explanations to patients and their families are sufficient. This has clearly been shown to be untrue.9 Patient comprehension and retention after preoperative visits is notoriously low, emphasizing the need for an easy access online portal to reinforce and enhance the information that has been provided during the preoperative visit.
11 Whereas many surgeons provide their own written material, we feel that the online, multimedia approach is more comprehensive and current, as well as frequently updatable. We agree with surgeons who cautioned against allowing our content to serve as marketing materials for surgeons utilizing new or unproven techniques and have been careful to avoid that in our content. Clearly, more needs to be done to further elucidate member surgeons’ needs when it comes to the use of an online information portal to which they can refer patients, in order to improve utilization of what we feel is an important resource.
In conclusion, ctsurgerypatients.org is a comprehensive online multimedia patient education tool that is increasingly utilized by the public for information pertaining to cardiothoracic surgery. Whereas the content and format are highly rated by users, cardiothoracic surgeons remain largely unaware of its existence despite an aggressive marketing campaign. For those aware of this resource, patient referral rates are presently low. Improved strategies to increase surgeon awareness and utilization are necessary.
12 References (1) Kapoor K, Praven G, Evans M, Miller W, Liu S. Health Literacy: Readability of ACC/AHA Online Patient Education Material. Cardiology 2017; 138:36-40. (2) Friedman DB, Hoffman-Goetz L; A systematic review of readability and comprehension instruments used for print and web-based cancer information. Health Educ Behav 2006; 33;352-373. (3) National Assessment of Adult Literacy (NAAL). Available at https://nces.ed.gov (accessed October 2018). (4) Amaresan S. “What Is a Good Promotor Score?”. HubSpot. Retrieved July 2019 (5) Roberts HJ, Zhang D, Earp BE, Blazar P, Dyer GSM. Patient self-reported utility of hand surgery online patient education materials. Musculoskeletal Care. 2018:1-5. (6) Cunningham CT, Quan H, Hemmelgarn B, Noseworthy T, Beck CA, Dixon E, Samuel S, Ghali WA, Sykes LL, Jette N. Exploring physician specialist response rates to webbased surveys. BMC Medical Research Methodology. 15:32 2015 (7) Personal communication. STS marketing department. October 2018 (8) James KM, Ziegenfuss JY, Tilburt JC, Harris AM, Beebe TJ. Getting Physicians to Respond: The impact of incentive type and timing on physician survey response rates. Health Serv Res. 2011 Feb 46: 232-242 (9) Stewart M. (1995) Patient Recall and Comprehension After the Medical Visit. In: Lipkin M., Putnam S.M., Lazare A., Carroll J.G., Frankel R.M. (eds) The Medical Interview. Frontiers of Primary Care. Springer, New York, NY
13 Table 1. Cardiothoracic disease categories included in ctsurgerypatients.org 1. Adult Heart Disease a. Aortic Valve Disease b. Coronary Artery Disease c. Mitral Valve Disease d. Tricuspid Valve Disease e. Cardiac Rhythm Disturbances f. Thoracic Aortic Aneurysm g. Aortic Dissection h. Left Ventricular Assist Device i. Atrial Fibrillation j. VA ECMO k. Heart Failure 2. Pediatric and Congenital Heart Disease a. Atrial Septal Defect b. Ventricular Septal Defect c. Tetralogy of Fallot d. Atrioventricular Canal Defect e. Transposition of the Great Arteries f. Coarctation of the Aorta g. Truncus Arteriosus h. Single Ventricle Defects i. Patent Ductus Arteriosus 3. Lung, Esophageal, and Other Chest Diseases a. Lung Cancer b. Esophageal Cancer c. Gastroesophageal Reflux Disease d. Barrett’s Esophagus e. Chest Wall Tumors f. Mediastinal Tumors g. Achalasia and Esophageal Motility Disorders h. Pleural Diseases i. Mesothelioma j. VV ECMO k. End-Stage Lung Disease 4. Heart and Lung Transplantation a. Heart Failure b. Adult Heart Transplantation c. Pediatric Heart Transplantation d. End-Stage Lung Disease e. Adult Lung Transplantation f. Pediatric Lung Transplantation 5. Before, During, and After Surgery a. Before Heart Surgery b. Day of Heart Surgery c. After Heart Surgery d. Before Congenital Heart Surgery e. Day of Congenital Heart Surgery f. After Congenital Heart Surgery
14 Table 2. Public user and surgeon member survey questions. Public User Survey 1. Is the content useful? 2. Is the content easy to navigate? 3. What kind of content would you like to see more of? 4. How Did you find us? 5. Did you use us to make a decision regarding surgery? 6. Would you recommend us to a friend? 7. Were their other websites that you found useful? Surgeon Member Survey 1. Were you previously aware of ctsurgerypatients.org? 2. How did you find out about ctsurgerypatients.org? 3. Have you referred a patient to ctsurgerypatients.org? 4. Is the information of ctsurgerypatients.org relevant to your practice? 5. Is the information presented in a useful way? 6. What topics should be added to the website? 7. How could the website be improved? 8. Would you be willing to contribute to the website?
15 Figure Legends
Figure 1. An example of original graphics created by the STS Media Relations Department depicting an Atrial Septal defect video with left to right shunt in animated video form.
Figure 2. Website and page visits from 2016-2018.
Figure 3. Most commonly visited page topics in 2018.
Figure 4. Net Promoter Score for ctsurgerypatients.org content.
S0003-4975(19)31700-X
DOI:
https://doi.org/10.1016/j.athoracsur.2019.09.074
Reference:
ATS 33218
To appear in:
The Annals of Thoracic Surgery
Received Date: 9 March 2019 Revised Date:
27 July 2019
Accepted Date: 23 September 2019
Please cite this article as: Cohen RG, Kumar SR, Lin J, Reddy RM, Kane L, Bagley J, Juarez A, Fleischman F, Farkas EA, Hackmann AE, Grubb KJ, Reddy S, Erhunmwunsee L, Villamizar NR, Masood MF, Griffin M, Boden N, The disparity between public utilization and surgeon awareness of the STS patient education website, The Annals of Thoracic Surgery (2019), doi: https://doi.org/10.1016/ j.athoracsur.2019.09.074. This is a PDF file of an article that has undergone enhancements after acceptance, such as the addition of a cover page and metadata, and formatting for readability, but it is not yet the definitive version of record. This version will undergo additional copyediting, typesetting and review before it is published in its final form, but we are providing this version to give early visibility of the article. Please note that, during the production process, errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain. © 2019 by The Society of Thoracic Surgeons
1 The disparity between public utilization and surgeon awareness of the STS patient education website Running head: STS patient education website data
Robbin G. Cohen, MD, MMM2, S. Ram Kumar MD PhD2, Jules Lin MD4, Rishindra M. Reddy MD4, Lauren Kane MD3, Jennifer Bagley, MA1, Armando Juarez, BA1, Fernando Fleischman MD2, Emily A. Farkas MD6, Amy E. Hackmann MD2, Kendra J. Grubb MD7, Seenu Reddy MD8, Loretta Erhunmwunsee MD9, Nestor R. Villamizar10 MD, Muhammad F. Masood MD11, Melissa Griffin PA-C5, and Natalie Boden, MBA1
1
Society for Thoracic Surgeons Media Office, Chicago, Illinois, USA
2
Department of Surgery, Keck School of Medicine of USC, University of Southern California, Los
Angeles, CA, USA 3
The Heart Center at Arnold Palmer Hospital for Children, University of Central Florida, Orlando,
FL, USA 4
Department of Surgery, University of Michigan, Ann Arbor, MI, USA
5
Huntington Hospital, Los Angeles, CA, USA
6
ThedaCare Appleton Heart Institute, Appleton, WI, USA
7
Department of Surgery, Emory School of Medicine, Atlanta, GA, USA
8
TriStar Cardiovascular Surgery, Nashville, TN, USA
9
Department of Surgery, City of Hope Comprehensive Cancer Center, Los Angeles, CA, USA
10
Sylvester Comprehensive Cancer Center, University of Miami Health System, Miami, FL, USA
11
Department of Surgery, Washington University School of Medicine in St. Louis, MO, USA
Title Character Count: 99 Short Title Character Count: 34
2 Abstract Word Count: 250 Manuscript Word Count: 3530
Corresponding Author:
Robbin G. Cohen, MD MMM Associate Professor of Cardiothoracic Surgery Department of Surgery Keck School of Medicine of USC University of Southern California Los Angeles, CA, USA [email protected]
3 Abstract Background: Many online resources currently provide healthcare information to the public. In 2015, The Society of Thoracic Surgeons (STS) created a multimedia web portal (ctsurgerypatients.org) to educate the public regarding cardiothoracic surgery and to provide an informative tool to which cardiothoracic surgeons could refer patients.
Methods: A patient education task force was created, and disease-specific content created for 25 pathological conditions. After launching the website online, a marketing campaign was initiated to make STS members aware of its availability. Website visits were monitored, and an online survey for public users created. An email survey was sent to STS members to evaluate awareness and content. Surveys were analyzed for effectiveness and utilization by both public users and STS member surgeons.
Results: From 2016-2018, the website had more than 1 million visits, with visits increasing yearly. Surveyed user ratings of the website were positive regarding quality and utility of the information provided. STS member response was poor (379 responses/6347 emails) and 78.3% of responders were unaware of the website. Surgeon responders were positive about the content, though many still refrain from referring patients.
Conclusions: Online education for cardiothoracic surgery is seeing increased public use, with high ratings for content and utility. Despite aggressive marketing to STS members, most remain unaware of this website’s existence. Those who are aware approve of its content, but adoption of referring patients to it has been slow. Improved strategies are necessary to make surgeons aware of this STS-provided service and to increase patient referrals to it.
4 A growing number of patients consult online resources for medical information, not only to learn about their own health issues and therapeutic options, but also to evaluate the background and reputation of their physicians. These resources are vast in number and vary from physiciandriven marketing tools to information resources developed by major disease-specific organizations, such as the American Heart Association or American Cancer Society. In 2015, the Workforce on Media Relations and Communications of The Society of Thoracic Surgeons (STS) created a Patient Information Task Force for the purpose of overhauling its patient information website into a comprehensive, multimedia, bilingual online education resource (ctsurgerypatients.org) for cardiothoracic surgical patients and their families. The website was also designed to be used as a reference tool to which STS member surgeons could refer their patients. The methodology and structure of the website are described, as well as an evaluation of its success using survey results both from patients utilizing the website and STS members.
Material and Methods Content Development: An eight-member Patient Education Task Force was created with two surgeon members representing each specialty category: adult cardiac surgery, adult thoracic surgery, congenital heart surgery, and heart and lung transplantation. Each specialty category was then divided into the most common surgical diseases treated by STS members (Table 1). In addition, a pre- and postoperative care section was created and similarly divided into specialty categories. Over a 1-year period, content for each section was developed by the STS Marketing & Communications Department, in conjunction with surgeon task force members. Content was organized by disease (as opposed to procedure) in order to maximize the potential for our website to appear on Google searches. Graphics on the website required STS ownership and were limited to previously published graphics from The Annals of Thoracic Surgery, web content for which STS received usage permission, and original graphics created by the STS Marketing & Communications Department. Topic-specific videos featuring expert
5 STS member surgeons, as well as two- and three-dimensional video animations depicting surgical anatomy and disease states, were also created internally (Figure 1). Final content was reviewed during weekly phone conferences prior to placement on the website. All text was also translated from English into Spanish.
In addition to didactic content and graphics, a news section was created containing current topics in cardiothoracic surgery, as well as press releases highlighting current manuscripts from The Annals of Thoracic Surgery. A biweekly blog was also created, covering a broad range of cardiothoracic surgical topics and written by STS members and patients.
Marketing: Once content was completed, reviewed, and posted on the internet, an extensive marketing strategy was implemented to raise awareness of the website utilizing colorful advertisements in The Annals of Thoracic Surgery, STS News, STS e-newsletters, and at the two subsequent STS Annual Meetings. In 2016, this included 8 full page advertisements in The Annals of Thoracic Surgery, 7 posts in the STS News, multiple bag inserts and fliers at the Annual Meetings of the STS, AATS, ESTS, and EACTS, 14 electronic messages to attendees of those meetings, and 28 emails to all STS members or data managers regarding our existence, new content, or blogs.
Evaluation: Numbers of visits to the website as well as navigations to specific pages were monitored over a 3-year period from 2016-2018. Users: A seven-question survey was created as a pop-up window requesting that website users evaluate our patient education portal. In addition, an eight question survey was emailed to 6347 STS surgeon member email addresses regarding their awareness of the web portal, their evaluation of the content, and whether or not they currently referred patients to
6 ctsurgerypatients.org (Table 2). Follow-up reminder emails from the STS President and in the STS weekly email were sent to the same addresses 9 and 13 days later respectively. Survey results were analyzed to determine awareness of our website, the quality and utility of our content, suggestions for content improvement, and whether or not STS surgeon members were referring patients to our portal. A net promoter score was determined for the question “Was the content useful”, using a scale of 1-10. The website was also analyzed regarding numbers of hits and most popular components.
Results Online visits to ctsurgerypatients.org for 2016 through 2018 are depicted in Figure 2. Visits doubled from 2016-2017 and increased another 23% from 2017-2018. The most commonly visited pages by content and disease are seen in Figure 3. The “What is a cardiothoracic surgeon” page had always been the most visited page when it was on the original STS website and continued to be the most visited page on the new patient education portal through 2017. As of 2018, it has been surpassed by the Spanish versions of “After Heart Surgery” and “End-Stage Lung Disease”. In fact, the number of visitors from predominantly Spanish speaking countries to the top 10 most accessed pages during 2018 exceeded those from English speaking countries (186,438 vs. 158,412).
There were 1038 responses to the patient user survey. Seventy-five percent of responders found the site via a search engine, whereas only 10% were referred by surgeons. The net promoter score for the usefulness of content is depicted in Figure 4, with more than 70% of respondents scoring 7 or greater on a scale of 1-10. Ninety three percent of responders felt that the website was easy to navigate, and almost 90% would recommend the site to a friend. Only 36.5% of respondents used the website to make a surgical decision.
7 There were 379 responses to the surgeon member survey for a response rate of 6%. Of those responding, 78.3% said that they had not previously heard of ctsurgerypatients.org. However, almost 90% of responders felt that the information provided on the website was relevant to their practices, and 83% felt that the information was presented in a useful way. When asked why they didn’t refer patients to the website, responses included that they didn’t know about it, didn’t think about it, provided their own written materials, or preferred to explain things to their patients themselves. Forty-five percent responded that they would be willing to contribute to the website.
Comment Over the past 2 decades, the internet has become populated with an abundance of online educational materials targeting the public with respect to prevalence, diagnosis, and treatment of a plethora of health-related conditions. These include a wide range of pharmaceutical marketing materials, as well as promotional sites for physicians, medical groups, universities and payers. They also include informational sites developed by condition-specific, non-profit health-related organizations, whose purpose is to bring attention to their causes and serve as an accurate, unbiased educational resource for patients and their families who may be affected by these conditions. The American Heart Association and The Marfan Foundation are excellent examples of this online educational format, which is both promotional and educational. Our goal was to have our patient education portal falls into this category.
In 2014, the STS committed to upgrading its online patient education portal to a more independent and comprehensive website. The goal was to provide easily understood, multimedia content to the public regarding the diagnosis and surgical therapy of a broad range of adult and pediatric cardiothoracic conditions. The content needed to be readily accessible and searchable to consumers on the internet. It also needed to be comprehensive but unbiased. For instance, we were careful not to create promotional material that would serve one surgeon
8 over another, such as recommending a “minimally invasive” incision over sternotomy for cardiac valve surgery. In addition, we wanted to create an online tool to which STS surgeon members would refer patients and their families, to further educate them and make them comfortable with their surgical decisions.
There were a number of challenges inherent to this process, many of which have been previously identified.1 The first was to provide content that was understandable and useful to the general public. Health literacy, defined as the “capacity to obtain, interpret, and understand basic health information and services, and the competence to use such information and services to enhance one’s own health,” is adequate in only 12% of the US adult population.2 Of the hundreds of medical and surgical informational websites currently available online, most fail to comply with the standards set by the National Institutes of Health (NIH), which calls for a Readability Grade Level (RGL) of between 6th and 7th grade. The average American adult reads at approximately the 8th grade level.3
Despite our best intentions, it was virtually impossible for our surgeon content developers to consistently write at or below the NIH prescribed 7th grade level. In fact, much of our content contains medical vocabulary and sophisticated graphics depicting anatomic and physiologic details. Despite this, more than 90% of user respondents said that our content was easy to navigate, 80% stated that it was useful, and more than 90% said they would recommend our website to a friend. Success of our user experience was also reflected in the Net Promotor Score (NPS) of 37. The NPS is a management tool used by business entities to gauge customer experience and loyalty. Customers who respond with a score of 9-10 are deemed “Promotors” and are more likely to remain customers and serve as a referral base to others. A score of greater than zero is deemed “good”, with a score of 50 considered “excellent”4. These indicators led us to believe that our multimedia platform was accomplishing our goals. Others
9 have reported better-than-expected patient-reported comprehension of online surgical material.4 One reason may be that consumers with the sophistication to search for healthcare related information online have a higher level of comprehension than those called for by the NIH. Another might be that the addition of animated graphics and videos may enhance overall comprehension of the material, despite the sophistication of the text. Finally, as our site is unlikely to be the sole source of information to patients, the additional information or reiteration of previously obtained content may be better comprehended even if presented in a more comprehensive format. Regarding utilization of our material, it is not surprising that only slightly more than one-third of respondents used our content to actually make a surgical decision, as most patients have probably chosen the recommendation of their surgeon before leaving their surgeon’s office. Still, we are confident that our online information was useful for clarifying and reiterating what the patients and their families have already heard from their surgeon.
Data regarding website and page visits demonstrate consistently increasing us of the website by the public. It is particularly interesting that 3 of the 4 most visited pages in 2018 are the Spanish versions. This undersocres the diversity of patients in search for information regarding cardiothoracic surgery, as well as the international preponderance of information on the internet. Given the complexities of translating and updating multilingual websites, expanding ctsurgerypatients.org into additional languages will require significantly more resources than currently exist at the STS. Updating the website with new or more effective content is a continuous project. During a complete review and update of the pages that were initiated in 2017 and completed in December 2018, new disease topics were identified and have subsequently been added and expanded content is being developed for procedures that cardiothoracic surgeons perform.
The STS Patient Education website was recreated under the assumption that the public,
10 especially patients and their families with cardiothoracic surgical disorders, would benefit from a current, factual, unbiased, usable website that explains their clinical entities and treatment options. We also assumed that the STS member surgeons would approve of the information and refer patients to it. Results from our member surgeon survey suggest that despite patient satisfaction with our online content, surgeons either don’t know about it or are aware of it but are not utilizing it. This was despite a sizable advertising campaign and strong agreement from surgeon respondents who have viewed the material that the content is both relevant and presented in a useful way. The low level of awareness of the website was surprising, given the large-scale effort to advertise the website at the STS Annual Meeting, in STS news, and in The Annals of Thoracic Surgery. The STS Marketing and Communications Department continues to explore new and novel ways to raise awareness and increase utilization of member related services such as this one. Potential options include even more aggressive and novel advertising, more demonstrations of website content in between scientific sessions and the Annual Meeting, as well as providing surgeons’ offices with written materials aimed at guiding patients to the website.
Regarding the response to our online survey, physicians are notorious for poor response rates.6 In fact, the expected response rate for an STS-generated physician survey is 15-30%.7 Though incentives, such as financial rewards for completing surveys, are known to increase physician response rates, this is not possible for a nonprofit specialty society such as the STS.8 Regardless, the information that we received was useful, especially when it came to determining the rationale for why surgeons who were aware of the patient education website were not utilizing it. Many surgeons feel that their own explanations to patients and their families are sufficient. This has clearly been shown to be untrue.9 Patient comprehension and retention after preoperative visits is notoriously low, emphasizing the need for an easy access online portal to reinforce and enhance the information that has been provided during the preoperative visit.
11 Whereas many surgeons provide their own written material, we feel that the online, multimedia approach is more comprehensive and current, as well as frequently updatable. We agree with surgeons who cautioned against allowing our content to serve as marketing materials for surgeons utilizing new or unproven techniques and have been careful to avoid that in our content. Clearly, more needs to be done to further elucidate member surgeons’ needs when it comes to the use of an online information portal to which they can refer patients, in order to improve utilization of what we feel is an important resource.
In conclusion, ctsurgerypatients.org is a comprehensive online multimedia patient education tool that is increasingly utilized by the public for information pertaining to cardiothoracic surgery. Whereas the content and format are highly rated by users, cardiothoracic surgeons remain largely unaware of its existence despite an aggressive marketing campaign. For those aware of this resource, patient referral rates are presently low. Improved strategies to increase surgeon awareness and utilization are necessary.
12 References (1) Kapoor K, Praven G, Evans M, Miller W, Liu S. Health Literacy: Readability of ACC/AHA Online Patient Education Material. Cardiology 2017; 138:36-40. (2) Friedman DB, Hoffman-Goetz L; A systematic review of readability and comprehension instruments used for print and web-based cancer information. Health Educ Behav 2006; 33;352-373. (3) National Assessment of Adult Literacy (NAAL). Available at https://nces.ed.gov (accessed October 2018). (4) Amaresan S. “What Is a Good Promotor Score?”. HubSpot. Retrieved July 2019 (5) Roberts HJ, Zhang D, Earp BE, Blazar P, Dyer GSM. Patient self-reported utility of hand surgery online patient education materials. Musculoskeletal Care. 2018:1-5. (6) Cunningham CT, Quan H, Hemmelgarn B, Noseworthy T, Beck CA, Dixon E, Samuel S, Ghali WA, Sykes LL, Jette N. Exploring physician specialist response rates to webbased surveys. BMC Medical Research Methodology. 15:32 2015 (7) Personal communication. STS marketing department. October 2018 (8) James KM, Ziegenfuss JY, Tilburt JC, Harris AM, Beebe TJ. Getting Physicians to Respond: The impact of incentive type and timing on physician survey response rates. Health Serv Res. 2011 Feb 46: 232-242 (9) Stewart M. (1995) Patient Recall and Comprehension After the Medical Visit. In: Lipkin M., Putnam S.M., Lazare A., Carroll J.G., Frankel R.M. (eds) The Medical Interview. Frontiers of Primary Care. Springer, New York, NY
13 Table 1. Cardiothoracic disease categories included in ctsurgerypatients.org 1. Adult Heart Disease a. Aortic Valve Disease b. Coronary Artery Disease c. Mitral Valve Disease d. Tricuspid Valve Disease e. Cardiac Rhythm Disturbances f. Thoracic Aortic Aneurysm g. Aortic Dissection h. Left Ventricular Assist Device i. Atrial Fibrillation j. VA ECMO k. Heart Failure 2. Pediatric and Congenital Heart Disease a. Atrial Septal Defect b. Ventricular Septal Defect c. Tetralogy of Fallot d. Atrioventricular Canal Defect e. Transposition of the Great Arteries f. Coarctation of the Aorta g. Truncus Arteriosus h. Single Ventricle Defects i. Patent Ductus Arteriosus 3. Lung, Esophageal, and Other Chest Diseases a. Lung Cancer b. Esophageal Cancer c. Gastroesophageal Reflux Disease d. Barrett’s Esophagus e. Chest Wall Tumors f. Mediastinal Tumors g. Achalasia and Esophageal Motility Disorders h. Pleural Diseases i. Mesothelioma j. VV ECMO k. End-Stage Lung Disease 4. Heart and Lung Transplantation a. Heart Failure b. Adult Heart Transplantation c. Pediatric Heart Transplantation d. End-Stage Lung Disease e. Adult Lung Transplantation f. Pediatric Lung Transplantation 5. Before, During, and After Surgery a. Before Heart Surgery b. Day of Heart Surgery c. After Heart Surgery d. Before Congenital Heart Surgery e. Day of Congenital Heart Surgery f. After Congenital Heart Surgery
14 Table 2. Public user and surgeon member survey questions. Public User Survey 1. Is the content useful? 2. Is the content easy to navigate? 3. What kind of content would you like to see more of? 4. How Did you find us? 5. Did you use us to make a decision regarding surgery? 6. Would you recommend us to a friend? 7. Were their other websites that you found useful? Surgeon Member Survey 1. Were you previously aware of ctsurgerypatients.org? 2. How did you find out about ctsurgerypatients.org? 3. Have you referred a patient to ctsurgerypatients.org? 4. Is the information of ctsurgerypatients.org relevant to your practice? 5. Is the information presented in a useful way? 6. What topics should be added to the website? 7. How could the website be improved? 8. Would you be willing to contribute to the website?
15 Figure Legends
Figure 1. An example of original graphics created by the STS Media Relations Department depicting an Atrial Septal defect video with left to right shunt in animated video form.
Figure 2. Website and page visits from 2016-2018.
Figure 3. Most commonly visited page topics in 2018.
Figure 4. Net Promoter Score for ctsurgerypatients.org content.