- Email: [email protected]
Biomarker data visualisation for decision making in clinical trials
International Journal of Medical Informatics 132 (2019) 104008
Contents lists available at ScienceDirect
International Journal of Medical Informatics journal homepage: www.elsevier.com/locate/ijmedinf
Biomarker data visualisation for decision making in clinical trials a,⁎
b
c
b
T b
Alan Davies , Marisa Cunha , Kamilla Kopec-Harding , Paul Metcalfe , James Weatherall , Caroline Jayc a b c
School of Health Sciences, University of Manchester, Manchester, UK Advanced Analytics Centre, Global Medicines Development, AstraZeneca, Cambridge, UK School of Computer Science, University of Manchester, Manchester, UK
A R T I C LE I N FO
A B S T R A C T
Keywords: Visualizations Biomarkers Clinical trials Interviews Thematic analysis Trust Provenance
Objective: To understand how visualization of biomarker data is used for decision making in clinical trials, and identify problems with and suggest improvements to this process. Methods: We carried out semi-structured interviews with 18 professionals involved in various aspects of developing or using visualizations of biomarker data for decision making in clinical trials. We used an inductive thematic analysis to identify implicit and explicit ideas within the data captured from the interviews. Results: We identified 6 primary themes, including: how visualizations were used in clinical trials; the importance of having a clear understanding of the underlying data; the purpose or use of the visualization, and the properties of the visualizations themselves. The results show that participants’ ‘trust’ in the visualization depends on access to the underlying data, and that there is currently no standard or straightforward way to support this access. Conclusions: Incorporating information about data provenance into biomarker-related visualizations used for decision making in clinical trials may increase users’ trust, and therefore facilitate the decision making process.
1. Background and significance Biomarkers – substances that are measured in a sample or biological system as an indicator of exposure, effect, susceptibility, or clinical disease – are an important tool in drug development. Based on their application, biomarkers can be classified as predisposition, diagnostic, prognostic, and predictive. Biomarker data may hold tremendous amounts of value, with the potential to reduce clinical trial costs, time, and failure rate. To realise this value, insights gleaned from the data must be translated into actions. Effective knowledge acquisition depends on technologies that support the translation of biomarker data into readily understandable information. This process involves the interaction of access to data, its visualization, and its interpretation [1]. The visualization of data can have a profound influence on both the speed of information interpretation, and how that information is interpreted [2,3]. Scientific data visualization aims to support discovery and analysis via the mapping of abstract data structure attributes to visual attributes [4]. When dealing with complex data, both visual search and analysis are utilized depending on the task and the complexity of underlying data [5]. Visualizations that allow for effective interpretation of
⁎
underlying data rely on the use of pre-attentive graphical features that can be processed by the human visual system with limited effort [4]. For more complex visualizations, additional dimensions are often mapped to other visual attributes, such as colour and size, or via use of dynamic filtering [4]. A review of data visualizations used in oncology research by Chia et al. indicated that data visualization in the field of oncology had evolved to represent complex datasets [6]. This includes novel representations of the underlying tumour biology and exploring visualizations that could be used to communicate with both non-specialist audiences and stakeholders [6]. Nevertheless, at present, there are still limitations in the visualisation of large data sets, including possible occlusion of data and potential misinterpretation [7]. An example of this can be seen in probabilistic reasoning, where it was found that different visualization formats resulted in different reasoning strategies [8]. In a study examining the visual representation of statistical information for improving diagnostic inferences involving over 162 participants (81 doctors, 81 patients), it was found that both doctors and patients were able to make more accurate inferences when presented with visual aids representative of numerical information, compared with the numerical information alone [9]. The study also found that the use of visual aids not only improved accuracy and
Corresponding author at: Room 1.001, Vaughan House, Portsmouth St, Manchester, M13 9GB, UK. E-mail address: [email protected] (A. Davies).
https://doi.org/10.1016/j.ijmedinf.2019.104008 Received 27 June 2019; Received in revised form 3 October 2019; Accepted 14 October 2019 1386-5056/ © 2019 Published by Elsevier B.V.
International Journal of Medical Informatics 132 (2019) 104008
A. Davies, et al.
data visualizations for decision making in clinical trials.
participants’ perception of the information's usefulness, but also decreased the perceived difficulty of the task [9]. Different studies highlight the need for the continual evaluation of biomarkers to ensure their validity. Evaluation and discovery of new biomarkers can be supported by data visualization, which is useful for hypothesis generation and gaining insights into the underlying data, as visualization enables patterns and signals to be more rapidly identified by humans compared with data displayed in tabular or numeric forms [10]. Currently, pharmaceutical development primarily relies on listings and tables, often under-utilizing graphics despite their ability to improve the efficiency of data analysis [3]. The aim of our study is to gain a perspective on the current working practices and challenges involved in creating and using biomarker data visualizations for decision making in clinical trials.
2.3. Procedure We conducted both face-to-face and online (Skype) semi-structured interviews. The same interviewer (AD) asked participants a series of questions and audio recorded the interviews. The interview questions were used as prompts for the discussions, which varied in their details according to the role of the interviewee. The broad interview questions were: 1 When reviewing the results of a structured analysis looking at all the biomarkers and patient-level covariates in an early study, how do you currently identify patient sub-groups for subsequent analysis/ investment in these situations? 2 Please describe your current workflow/system 3 What are the advantages of this system? 4 Are there any problems or disadvantages with this system? 5 If you currently use visualizations to identify biomarkers for this or similar purposes: a What visualizations do you use? b How do you use these visualizations? c What are the advantages of these visualizations? d What are the disadvantages of these visualizations? e How could they be improved? 6 What sort of data would be useful for this purpose? 7 How do you think these data/results should be displayed? 8 What role do you think visualization can play in the decision making process? 9 Do you think using visualizations could improve how you make decisions in this regard? a Why/why not? 10 How important or useful do you think visualizations are in this context? 11 What features do you think a visualization for this purpose should include? a Why?
2. Methods We carried out open ended, semi-structured interviews with participants involved in producing or using biomarker data visualizations to make decisions regarding clinical trials. We transcribed the interviews and analysed them using inductive thematic analysis (TA) by following the methods described in [12]. We followed the Standards for Reporting Qualitative Research (SRQR) guidelines [13]. The study received ethical approval from the University of Manchester Research Ethics Committee (4551). All participants provided informed consent prior to taking part in the study. A second coder (KKH) examined a subset of interview quotations (n = 300) and matched them to the primary themes with a percentage agreement of 70.5% and an unweighted Cohen’s Kappa showing substantial agreement (κ = 0.63). 2.1. Researcher characteristics AD has a background in Computer/Data Science and Nursing Science (interventional cardiology). AD has produced routine and bespoke visualisations in most commonly used data science/statistical software for research projects. AD did not personally know any of the interviewees that took part in the study. AD’s research paradigm lies between postpositivism and interpretivist depending on the nature of the research being undertaken. KKH (the second coder) has a background in data management, data science, computational chemistry and pharmacology. KKH has produced standard visualisations for research purposes using a range of statistical tools and generated bespoke interactive and dynamic visualisations for public engagement events.
3. Results Over 8 h of interviews (516 min) with an average duration of 28.66 min (SD = 10.36) were transcribed. Participants reported using a number of tools for visualisation, including: REACT [14], TIBCO Spotfire®, Microsoft Excel and R [15]. The most frequently mentioned were Spotfire® and AstraZeneca's REACT (Fig. 1). Fig. 2 shows how often each named plot was mentioned, with OncoPrints (a type of heatmap), waterfall plots, heatmaps and line plots
2.2. Participants We recruited 18 participants (Table 1) as a sample of convenience from three different organizations using snowball sampling, and ended recruitment on reaching saturation. A participant was identified as someone having a background in either creating or using biomarker Table 1 Participants’ professional role, gender and organization. Role
n
Bioinformatician (BI) Clinical consultant/Medical Scientist (MD/MS) Research software engineer (SE) Statistician/Data Scientist (ST/DS) Gender Male Female Organisation Pharmaceutical company Medical charity University Total
7 5 3 3 13 5 13 4 1 18
Fig. 1. Software tools used to generate visualizations by word frequency. 2
International Journal of Medical Informatics 132 (2019) 104008
A. Davies, et al.
achieve a world where once we have the data, we can kind of push a button, and produce all the analysis and recording. Because of that, the workflow for dynamic visualization is a bit poorly characterized.” [P2, BI] Whilst static visualizations were viewed as more appropriate for regulatory reporting, academic publishers in the field were keener to explore different ways of visualizing the data. “Increasingly journals have been asking for some sort of a simple web interface to explore these images which have been included in a publication, and I think it is just going to become a standard going down the line where every journal will say, right every image you have within your main [inaudible 29:16] script has got a supplementary; we want to see all of these in an interactive dynamic way on a web interface. And again this doesn't have to be a full-fledged web interface or a website like CBioPortal; it can be a simple collection of these figures the readers can play with… It's not yet a standard but it will make life so much easier for the people reading as well as people published.” [P18, BI] The usability of visualization tools (n = 7, [13]) was also highlighted by participants as an issue.
Fig. 2. Plot showing the most mentioned visualization types.
“One of the things that we are really trying to push in-house now is usability and user experience, so that's something that we're trying to pull in earlier in any of the projects that we do.” [P3, ST/DS]
mentioned the most often. Results of the thematic analysis highlighted 6 primary themes and a number of related sub-themes (Table 2 and Fig. 3). When a theme is discussed, the number of participants mentioning the theme (n) is shown followed by the number of references made to the theme in square brackets.
Reproducibility was raised by the majority of participants (n = 16, [61]). Certain plots are used as standard in trials, and as such their production can be automated in a pipeline, which helps to improve reproducibility.
3.1. Application to clinical trials
“The pre-planned visualization is well characterized, so you think about the total graphs you'd want, so it could be means over time or you know just dot and, line plots, forest plots, scatter plots, all those things we can pre-plan, and there's kind of a basic set that you know you want to have to describe the data. Then all along, while you're producing…doing the automated bit, we do have access to dynamic visualization where we'll drill down, mainly to try to explain things we don't quite understand, and there you are sort of letting the data guide you, and there visualization is helpful, but again, how the results of those explorations get translated into a report, is kind of poorly characterized and variable.” [P2, BI]
The various ways visualizations may be used within clinical trials was mentioned 264 times by 18 participants. A key application was trial documentation (n = 12, [35]) where a set of standardized visualizations were often used for reporting of trials. In this context visualisations tended to be traditional static plots (bar, line, etc.). Dynamic visualizations were used less, as they did not currently fit into the workflow. “We try to be very quick in reporting our studies… the core workflow is to automate and pre-plan as much as possible, to try to
Table 2 Interview response themes and sub-themes showing the number of participants in each theme/sub-theme and how many times each theme was mentioned. Theme
Participants
References
Sub-theme
Participants
References
1.0 Application to clinical trials
18
264
2.0 Underlying data
18
110
3.0 Understanding
17
127
4.0 Visualization properties
18
114
5.0 Purpose or use of visualisation
18
174
6.0 Type of visualisation
18
168
1.1 1.2 1.3 1.4 1.5 1.6 1.7 2.1 2.2 2.3 3.1 3.2 4.1 4.2 5.1 5.2 5.3 5.4 6.1 6.2 6.3 6.4 6.5
12 7 16 16 11 10 14 13 17 7 17 7 14 17 8 16 16 7 15 14 10 5 8
35 13 61 34 39 41 41 25 70 13 111 16 43 71 17 61 65 31 42 46 50 17 13
Documentation Evaluation Reproducibility Required properties Drug efficacy Patient risk Working practice Access to data Contextualising data Trusting the data Interpretation Training Labelling/annotation Representation of dimensionality or granularity Barriers to use Data exploration Decision making Synthesising results Bespoke or novel Dynamic Named plot/chart/graph Static Traditional
3
International Journal of Medical Informatics 132 (2019) 104008
A. Davies, et al.
Fig. 3. Treemap visualising the themes and sub-themes by number of participants.
“[…] they'll give us as much data as possible and obviously the more people who have access to it the more likely it is to get out into different arenas or be changed or the message gets changed. You do have to be a bit more cautious about that.” [P4, MD/MS]
It appears that this is less well defined and more complex in the case of dynamic visualizations: “[…] so it lends itself well to if you're all sitting around a computer together, and you're exploring the data, it doesn't lend itself as well to documentation, you know, yes can do screen shots, but then you perhaps have to document how you got to that graph, if for example you did a lot of filtering or rotating or pointing and clicking, how do you document that, and show it to someone else, so that they could reproduce it or understand what you got.” [P2, BI]
Interviewer: “So you think it might be good that for some users those features are not available unless you're a certain type of user or in a certain role and then you can lock…” Participant 10 (MD/MS): “Yes, definitely.” The context of the data was an important consideration. Providing background information about statistics was considered essential. This could be in the form of additional information accompanying the visualization. Effect size and measures of variability were also considered important. It was highlighted that not all the software used to generate visualizations included this information by default, and that additional thought and effort was required to include it.
Most participants (n = 16, [34]) also discussed the features or properties that they thought visualizations should possess. This was expressed as either higher level requirements/standards or in terms of ‘would like to have’ in an ideal or future situation. “…some are. png, some are. jpegs, some are gifs. I think standardizing the output is important to maintain the quality of the visualization, because more often than not it's generated to put on a PowerPoint slide or a publication, and in order to maintain the quality of it I think all of these should have a standard output within a certain quality.” [P18, BI] “Yes, some history or if it's easier to make some kind of bookmarks that can capture both the filter settings and the visualization, corresponding with that, that's a good way to work.” [P1, BI]
“Another point around you know displaying some sort of information about variability, I know that it's kind of a big pet peeve of my boss, is to show a graph and you don't show, you know the sample size or some sort of information about variability, so, yes for example you can have an overall null effect and then do some sub groups, and you find one sub group that looks great, but if that's based on you know just one percent of the data, it's less compelling that if it's, you know, a much more substantial subset, and a lot of software won't include a sample size or variability for free, you've got to add that on.” [P2, BI]
3.2. Underlying data
Having trust in the data related to being able to ‘get back’ to the original data after a number of transformations had been applied prior to the visualization(s).
Several issues pertaining to the data used to generate the visualizations were presented by participants (n = 18, [110]). Issues included who should have access to the data (n = 13, [25]), the context of the data (n = 17, [70]) and the trustworthiness of the data (n = 7, [13]). There was disagreement about who should access data, with participants believing in either complete access:
“Well you just rely on what people tell you and people will tell you what they want to tell you, there's always more to the story kind of thing. So they can present the data in a way that means that…you can present data in any way you want can't you really, it's not…you can present the same data two different ways, but you can only get to the bottom of that if you actually have access to the raw data.” [P4, MD/MS]
“So, this is a personal view, but I think it should be accessible to everybody, because I think, we worry about whether somebody misinterprets it, but equally if they don't have access to it they can't even interpret it the right way. I would say, more eyes on the data than not.” [P13, BI]
The importance of focusing on the data, rather than storytelling, was also raised:
And others stating that access should only be granted to people with a certain role or training:
“You know, if it's a senior person who has now reviewed the data, 4
International Journal of Medical Informatics 132 (2019) 104008
A. Davies, et al.
and they've got a nice set of slides, and the narrative they're telling them is that this drug seems to be working, we just need to add a few more patients, it's all on track. There is a drive at, certainly [name removed for anonymity], and [name removed for anonymity] had a research there about really using the data and evidence-based science, so he drove really hard on it, he doesn't like storytelling, data, data, data […]” [P13, BI]
“The colour scale we use is quite important. So, there is that. It's in [name removed for anonymity] visualization course where she shows you like a rainbow, it's a rainbow colour scale and like a constant…it's based on…basically you see things in a rainbow scale which aren't there because you get quite quick changes in…how much is it, a hue I think it is, compared to lightness or something. But basically, you get these effects which don't really exist if you like.” [P17, SE]
3.3. Visualization properties
It was also mentioned by one participant that there was no official standard for the colours used with OncoPrints, and that this could lead to different people representing different genetic information (i.e. mutations) using different colours. An important aspect of interpretation discussed was training in the use of visualizations (n = 7, [16]).
Labelling and annotation was discussed by 14 participants 43 times. This was often added to visualizations created with one tool using another subsequent tool. Examples of this included using software like PowerPoint to add labels and other annotations to visualizations created with tools such as R. There is, however, a risk of introducing errors in this process. “[…] it can be quite complex, and it can be difficult to give all the information there [in the visualization]. One thing we often do, I think is to get the visualization to PowerPoint and there you can add on this information.” [P1, BI] A further participant mentions that it can be difficult adding this information depending on the software tool used to generate the visualization. They highlight that this is neither efficient nor reliable, but that this practice is the status quo. Another important property of visualizations that was discussed was their ability to represent different dimensions of the data and show the data at different levels of granularity (n = 17, [71]). Being able to go from a population level view of the data to specific data points was considered very important, especially for clinicians.
“CBioPortal is essentially an omics system, so it allows us to generate OncoPrints so we can see genomic heat maps of our tumour biopsies, of our circulating tumour DNA and so on and so forth. We're looking at bringing in kind of immune bio markers into that. Now that really requires a trained translational scientist to be able to do that. That's not trivial and we would not recommend that would be used by someone inexperienced. So access to that is something that we would consider restricting, or not restricting but putting a proviso that any interpretation of the data or those data within the cBioPortal should be done by an expert or someone who's an expert on the system.” [P14, MD/MS] The focus of training was mainly on how to use tools correctly and on cautioning people about the pitfalls and dangers of making conclusions about data based on misinterpretation.
“what that means is essentially a physician or a medical scientist will look at the data, they'll look at that on a patient level which is an individual patient set of data, or they'll look at an aggregate level, so a population view. And an example of what might happen is they'll review the adverse events and any associated lab data to support those adverse events, looking for trends, looking for kinds of things that need immediate attention.” [P3, ST/DS]
“We think all the visualizations on [name removed for anonymity] are simple-ish. With all the filters it becomes complex and too complex for people, but we do require them all to do training first before they can use it for that reason. Because you don't want people just going in, not knowing what they're doing, not set the filters correctly, think they've got something that represents what they want and then go and take away the grand conclusion.” [P10, MD/ MS]
There were also occasions where it was considered more beneficial to use multiple plots with lower dimensions than try to include higher dimensions in visualisations, as highlighted by one participant:
Another participant pointed out the issue of medical liability when interpreting data, especially when communicating results to patients themselves.
“Three dimensional plots are lovely to look at and you can move them up and down, left and right but they're actually very, very difficult to interpret, very, very difficult. Not only are they difficult to interpret they actually can be quite dangerous to interpret because you can miss things on them. […]It's actually sometimes better just to have two separate plots with a common axis.” [P14, MD/MS]
“I think there's a lot of thinking about medical liability in their minds. So, if they have too much data, or data that they're not quite sure what it means, and they don't act on it, then they're very uncomfortable. If they know that there's four genes and their associated variance, if they know what they mean, they're happy about that. Because, the science is advancing so quickly, if you say, hey, you might want to take a look at this, there's a paper just been published on it, you might want to look at it, unless they're an experimental geneticist really deep in it, they don't really want to be in that space.” [P13, BI]
3.4. Understanding The ability to interpret the visualization was connected to the properties of the visualization, such as the labelling/annotation, which in turn were dependent on the contextualization of the underlying data.
3.5. Visualization purpose or use
“Quite often we see people using percentages but then when you say, well what's that, you know, it sounds like an impressive percentage but then it's based upon 20 samples or whatever, you know, it's quite a small number. So it can give quite a false message unless you've got that information about your sample size or the error bars.” [P16, BI]
Three primary reasons were given by participants for using visualizations. These included: data exploration (n = 16, [61]), decision making (n = 16, [65]) and synthesizing results (n = 7, [31]). Using visualizations for data exploration was particularly important in early stage clinical trials, where users focus on signal detection (i.e. find a pattern in the data that may lead to further investigation) and explore hypotheses.
Interpretation was discussed by 17 participants 111 times. The colours used in some visualizations, such as heatmaps and OncoPrints were highlighted as factors that influenced interpretation of visualizations.
“This is hypothesis generating rather than hypothesis confirming. We want to have the ability to explore the results/the data and to try to understand the different perspectives…” [P11, MD/MS] 5
International Journal of Medical Informatics 132 (2019) 104008
A. Davies, et al.
In late stage clinical trials, visualization was used more for confirmatory purposes. Visualization was considered to be important or essential in the clinical trial go/no-go decision-making process.
“…there are certain times when an end user will come to you and say, look, I really need this visualization, and you've a choice. You've got this generic tool in front of you. So other people use Excel and you say, well, Excel will only get me this far, I probably need to look at maybe Spotfire to do more complex interrogation of data but it's going to take me longer and you're kind of in your head you're trying to balance, which one do I do, because once you're committed to one you kind of have to keep going with it. It's kind of similar for [name removed for anonymity] is that what we've tried to do is the more commonly used one so it's the 80/20 rule. There's about 20 per cent of visualizations that are commonly used and we try to sort of address those most commonly used visualizations. Then there are those other ones that are more specific.” [P14, MD/MS]
“A lot of times we'll do kind of…standardized effect plots where you do some sort of adjustment for the number of different sub groups or biomarkers you're presenting, and in that case, the graphic becomes very important, and fundamental in decision making around, you know, should we run a new trial to understand is there an effect in a sub group, or do we think it's just a chance finding. So, the visualization is much more helpful than say, you know looking at lots of different tables, or you know, having P values, or some sort of numerical measure of strength of effect.” [P2, BI] When it came to the synthesis of results, visualizations were often included in presentations to communicate findings and assist understanding.
Fourteen participants discussed novel visualizations and all agreed that there was a place for new ways of visualizing their data. They did, however, caution against the potential to misinterpret such visualizations due to users' unfamiliarity with them: “There have been instances where we've kind of developed more novel visualizations, and there's absolutely a place for, you know, the… like the starburst graphs, the network diagrams and all the other different types of visuals that you can do. The thing that we have to be mindful of is a lot of the users, you know, they're obviously more familiar with what a bar chart is and you don't have to tell them how to interpret it. So we tend to use the more traditional ones, partly for that reason but also because the data lends itself to that.” [P3, ST/DS] This participant went on to say: Participant 3 (ST/DS): “Now one of the things I personally am quite keen on, but also within IT we're quite keen on, is identifying what's the best way of displaying data. So not what's the easy way, i.e. the bar chart, but actually is there something out there that would give you a much better presentation of that data or allow you to show more dimensions? So, you know, not just shape, colour, X and Y axes, but are there some other things that we could be looking at? We did have an example actually which was really cool of a, like a dynamic, like a true dynamic visualization where it was like a video almost. I don't know if you've seen that famous one? It's completely gone off the top of my head, but it's to do with…it's something to do with poverty or some kind of financial index and all the different
“Oh yes, certainly, if you're doing a presentation, yes, I would definitely convert it into visual formats, because it's asking someone to interpret data very quickly, I think you have to.” [P15, MD/MS] Participants also identified problems with using visualizations (n = 8, [17]), including greater numbers of people requesting more and different ways of viewing things, certain visualizations not being validated, and the difficulty of accurate interpretation: “If you don't use the right visualization then you go home with a very wrong message of what your data is telling you. On how comprehensive is the visualization […] often things like OncoPrint it's very easy to filter out samples without a single variant. If there is no alteration defined or found in a sample, it's very common to filter out samples because it's easier to see.” [P18, BI]
3.6. Types of visualization All participants (n = 18, [168]) mentioned using specific plots in their work, such as waterfall plots, forrest plots and OncoPrints (Fig. 4), as well as more traditional line and bar charts/plots. Participants also discussed bespoke plots (n = 15, [42]).
Fig. 4. An example of an OncoPrint. 6
International Journal of Medical Informatics 132 (2019) 104008
A. Davies, et al.
context has on decision making when we present the same information in different contextual situations. As well as context, trust in the underlying data also appears to be a factor in the extent that people would rely on the data visualization for subsequent decision making. In this case trust is related to the accuracy of the data, what it represents and knowledge of any transformations that it has undergone. Participants’ subsequent trust in visualization appears to be mitigated by their access to the underlying data leading them to trust the visualization more if they have access to the underlying data. A lack of time within their job role, a lack of knowledge about visualization construction, and the lack of tools to build visualizations efficiently, may be why people often use visualizations constructed by others [19]. Increased access to tools that make it possible to quickly and easily generate visualizations may remove the potential “bottle neck” of relying on one or more persons to generate the visualization for another to interpret or use. It is conceivable that desiring access to the “raw” or original data may reflect a perceived need for access to the data that in reality would not be of practical use. However, given the comments of participants who were in favour of increased access and “more eyes on the data”, supporting this may be beneficial, particularly given the high stakes involved in running clinical trials, both financially and from a patient safety perspective. These findings highlight the importance of maintaining and providing access to data provenance in clinical trial decision making, which can be defined as a record trail accounting for the origins of data in a storage medium, such as a database or other document [20]. The provenance of data is also critical for its subsequent sharing and reuse [21]. As highlighted in the interviews, data visualization can facilitate understanding of the data and reduce “information overload” [5], or alternatively obfuscate it [3]. Complex data requires transformation prior to it being queried, grouped or compared [5]. Potential errors and the different choices made along the way can affect the end outcome. Recording and availability of information concerning data lineage and provenance can aid interpretation and reproducibility. The requirements for reproducibility related to decision support have been identified by Curcin et al. [22] as: transparency of the system; auditability and reproducibility of recommendations; validation of readiness; tractability of evidence; responsibility; privacy/security; usability; and scalability [22]. Dynamic visualization offers a way of mitigating information overload [5], and this was mentioned by several participants. This allows for highly complex data, or data with high dimensionality, to be reduced, as the visualization is dependent on both perception and the cognitive manipulation of objects in space [23]. This form of visualization, although useful for data exploration, is not as suitable for documentation, for which static printable images are preferred. Dynamic visualization also suffers from reduced reproducibility, due to the filtering and other manipulations applied to the data. This is also related to the issue of hypothesis vs. data-driven research, which in the former case relies on the human ability to formulate meaningful questions and determine how these can be addressed using data, and in the latter case on machine learning/pattern mining techniques that automatically extract patterns from data [7]. By their nature, biomarker data are often multivariate, including measurements taken at different points in time [10]. Due to the superior ability of humans to process visual information compared with tabular or numeric information [10], visualizations can aid humans working with such data. Visualizations, if constructed and used well, can help humans to detect signals and patterns in large amounts of data. This supports a method of combining the best of human expertise with machine learning algorithms within drug development. Substantive work has been carried out in the visualization field examining how humans interact with visualization (see, [24] [25], and [26] for examples). Despite this, further work is required to generate foundational theories for validation and understanding of design work in the InfoVis community, a community that hosts resources for information visualization [27].
countries…” Interviewer: “Oh.” Participant 3 (ST/DS): “…and how it changes over the years.” Interviewer: “Yeah, the Hans Rosling thing…” Participant 3 (ST/DS): “Yes.” Interviewer: “…the Gapminder one, yeah.” Participant 3 (ST/DS): “Yes, that's the one, that's the one. So it was kind of a similar thing to that where you could press play and then watch all the bits move, and then pause at a certain point so it, you know, having that extra dimension. Then it gives you additional information that you can interpret.” Dynamic visualization (n = 14, [46]) was also discussed. In the context of the interviews, this related to the ability to interact with the visualization i.e. to zoom, rotate, highlight properties etc. It appears that these sorts of visualizations were used often earlier on in a given trial for the exploration phase, rather than later in the trials where more regulated and confirmatory visualizations were used. “…but getting round the fact, the problem that dynamic visualizations can lead to very subtly different, but perhaps very different interpretations, with the application of the slightly different backgrounds. That's the thing that we did quite a lot of work on with [name removed for anonymity]. It works well.” [P10, MD/MS] Static (n = 5, [17]) and traditional (n = 8, [13]) visualizations were mentioned. The majority of plots used in practice tended to be of a more static and traditional type (box plots, bar and line etc.) than dynamic or bespoke visualizations. “So I would say that the majority of the visualization tools that we have and that we use the more traditional graphics, so as you say, bar charts, line charts, et cetera.” [P3, ST/DS] 3.7. Conceptual model Based on the findings of the interviews, it appears that the interpretation of the visualization is dependent on several mediating factors. The interpreter's ability to interpret the graphic derives from their current knowledge and understanding of the data, and its chosen form of representation. Understanding and trust appears to be influenced by having access to the initial data from which the visualization was derived, which also increases trust in the visualization output and therefore the likelihood of using it for subsequent decision making. The effectiveness in conveying the intended message is also dependent on the skill and ability of the visualization creator to accurately communicate the underlying data. This is influenced by the purpose of the visualization and the available tools used to create it. This in turn mediates the type of visualization created and its properties (labelling, colours used, additional information included) (Fig. 5). 4. Discussion With our semi-structured interview, we were able to gather information about the working practices and challenges faced by 18 professionals involved in developing or using visualizations for decision making in clinical trials. We grouped this information into different themes and sub-themes to facilitate analysis and identified that the most referred information was related to contextualising data, representation of data dimensionality or granularity, and data interpretation. These are all known to be critical steps for decision making. It is well accepted that the context in which an option is presented can influence the evaluation of the option and some studies are evaluating the context effect in decision making [16] and policy formation [17], as well as for making risky choices [18]. Being context-aware is important, as it has been demonstrated that people do not always evaluate options independently. The set of choice options can influence the evaluation of single options [18]. This highlights the importance of the impact that 7
International Journal of Medical Informatics 132 (2019) 104008
A. Davies, et al.
Fig. 5. Conceptual model of data visualization process for clinical trial decision making.
in this area.
Building complex visualizations requires additional skills in areas such as software engineering. Programming is being used increasingly by decision analysts [28], with software such as R [15], Python [29] and MATLAB [30] being widely used for data science due to the combined features of reproducibility, documentation, automation and advanced visualization tools [28]. The skills required to use such tools effectively differ from those required for more traditional spreadsheet style tools like Excel. Due to the time, effort and skill involved in learning and utilizing such tools it is not surprising that many bespoke visualizations are requested by users from those with the relevant skills, rather than the requester opting to generate such visualizations themselves. Production of tools that allow for the rapid creation of visualization and include default information on variability and data context could help to overcome this barrier. Alternatively if visualizations continue to be provided by others, they should include information on data provenance and lineage by default, in order to increase trust in subsequent decisions.
6. Conclusions Given the complexity of the data analysis from which biomarkers are derived and the ever expanding quantity of available data, the importance of visualization in reducing complexity and mental overload becomes increasingly relevant. This challenge can be met by the development of visualization tools that allow the rapid creation of visualizations to represent these data for data exploration and confirmatory purposes. Our study also indicates that given the nature of clinical trials, data provenance and lineage are important considerations and need to be communicated to the user of the visualization to increase trust in the visualization, interpretability and subsequent confidence in decisions made based upon such visualizations. To ensure good decisions, visualizations need to provide the right context. Therefore, we recommend building feedback on data provenance into future visualization tools and adding more context by default to visualisations.
5. Limitations The primary limitation of this study is that the majority of participants were from the same pharmaceutical company. This may in turn have had an impact on the frequency of times they mentioned the REACT tool, which was developed internally by the same company, as well as some of the other software tools that were commonly used by participants, as these may also be standard in this setting. It is likely however those tools are used to carry out this type of work more broadly and thus reflects industry standards. Additionally due to the regulations surrounding the later phases of clinical trials, the type of visualisations used and their use for decision making are limited by regulatory process, in contrast to the earlier data exploration phases of trial development. Finally the study faces limitations inherent to all interview studies and qualitative research in general, such as increased subjectivity and reduced generalizability when compared to quantitative methods. These methods do however offer us a starting point from which to explore the issues covered in this study and focus future work
Authorship statement All persons who meet authorship criteria are listed as authors, and all authors certify that they have participated sufficiently in the work to take public responsibility for the content, including participation in the concept, design, analysis, writing, or revision of the manuscript. Furthermore, each author certifies that this material or similar material has not been and will not be submitted to or published in any other publication before its appearance in the International Journal of Medical Informatics. Specifically: A.D. carried out the interviews, analysis and writing of the manuscript. C.J., and M.C. contributed to the design of the study. K.KH carried out the second coding of the interview transcripts. All authors contributed to discussions about the direction of the work and read and made edits to the manuscript. 8
International Journal of Medical Informatics 132 (2019) 104008
A. Davies, et al.
Declaration of Competing Interest
[13] B.C. O’Brien, I.B. Harris, T.J. Beckman, D.A. Reed, D.A. Cook, Standards for reporting qualitative research: a synthesis of recommendations, Acad. Med. 89 (9) (2014) 1245–1251. [14] AstraZeneca, REal Time Analytics for Clinical Trials (REACT) [Internet], Available from: (2019) https://www.astrazeneca.com/what-science-can-do/our-technology/ real-time-analytics-for-clinical-trials-react.html. [15] R Core Team, R: A Language and Environment for Statistical Computing [Internet]. Vienna: R Foundation for Statistical Computing, Available from: (2014) http:// www.r-project.org/. [16] D. Dowding, J.A. Merrill, N. Onorato, Y. Barrón, R.J. Rosati, D. Russell, The impact of home care nurses’ numeracy and graph literacy on comprehension of visual display information: implications for dashboard design, J. Am. Med. Inform. Assoc. 25 (2) (2018) 175–182. [17] P. Monsivais, O. Francis, R. Lovelace, M. Chang, E. Strachan, T. Burgoine, Data visualisation to support obesity policy: case studies of data tools for planning and transport policy in the UK, Int. J. Obes. [Internet] 42 (12) (2018) 1977–1986, https://doi.org/10.1038/s41366-018-0243-6 Available from:. [18] P.N.C. Mohr, H.R. Heekeren, J. Rieskamp, Attraction effect in risky choice can Be explained by subjective distance between choice alternatives, Sci. Rep. 7 (1) (2017) 1–10. [19] Bradstreet T. Grables, Visual displays that combine the Best attributes of graphs and tables, in: A. Krause, M. O’Connell (Eds.), A Picture is Worth a Thousand Tables: Graphics in Life Sciences, Springer, London, 2012, p. 429. [20] A. Gupta, Data provenance, in: M. LING LIU, T. ÖZSU (Eds.), Encyclopedia of Database Systems, Springer, Boston, 2009. [21] I. Suriarachchi, Q. Zhou, Plale B. Komadu, A capture and visualization system for scientific data provenance, J. Open Res. Softw. 3 (1) (2015) 1–7. [22] V. Curcin, E. Fairweather, R. Danger, D. Corrigan, Templates as a method for implementing data provenance in decision support systems, J. Biomed. Inform. (65) (2017) 1–21. [23] J.K. Gilbert, M. Reiner, M. Nakhleh, Visualization: Theory and Practice in Science Education, Springer, 2008, p. 326. [24] E.R. Tufte, The visual display of quantitative information [Internet], Technometrics 2nd (2001) 197. Available from: http://www.amazon.co.uk/dp/0961392142. [25] S.K. Card, J. Mackinlay, B. Shneiderman, Readings in Information Visualization: Using Vision to Think, Morgan Kaufmann Publishers, San Francisco, 1999. [26] Ware C. Information Visualization: Perception for Design, 3rd ed., Morgan Kaufmann, Massachusetts, 2012, p. 536. [27] Z. Liu, J. Stasko, Theories in information visualization: what, why and how, Inf. Vis. (2010). [28] H. Jalal, P. Pechlivanoglou, E. Krijkamp, F. Alarid-Escudero, E. Enns, M.G. Myriam Hunink, An Overview of R in health decision sciences, Med. Decis. Mak. 37 (7) (2017) 735–746. [29] Python Software Foundation, Python [Internet], [cited 2019 Feb 10]. Available from: (2019) https://www.python.org/. [30] MathWorks, MATLAB 9.6, MathWorks, Massachusetts, 2019.
This work was funded by AstraZeneca. No other competing interests exist. Acknowledgements We would like to thank the Advanced Analytics Centre at AstraZeneca for funding this work. References [1] M. Gaylord, J. Calley, H. Qiang, E.W. Su, B. Liao, A flexible integration and visualisation system for biomarker discovery, Appl. Bioinformatics 5 (4) (2006) 219–223. [2] B. Reeder, H.J. Thompson, G. Demiris, Health providers’ perceptions of novel approaches to visualizing integrated health information, Methods Inf. Med. 52 (3) (2013) 250–258. [3] A. Krause, Concepts and principles of clinical data graphics, in: A. Krause, M. O’Connell (Eds.), A Picture is Worth a Thousand Tables: Graphics in Life Sciences, Springer, London, 2012, p. 429. [4] J. Fekete, C. Plaisant, Interactive information visualization of a million items proceedings of IEEE symposium on information visualization, The Craft of Information Visualization: Readings and Reflections Vol. 2002 Elsevier Science, USA), 2002, pp. 279–286. [5] J. Dill, R. Earnshaw, D. Kasik, J. Vince, P.C. Wong, Expanding the Frontiers of Visual Analytics and Visualization, Springer, London, 2012, p. 531. [6] P.L. Chia, C. Gedye, P.C. Boutros, P. Wheatley-Price, T. John, Current and evolving methods to visualize biological data in cancer research, J. Natl. Cancer Inst. 108 (8) (2016) 1–11. [7] B. Shneiderman, Inventing discovery tools: combining information visualization with data mining, Inf. Vis. 1 (1) (2002) 5–12. [8] M. Reani, N. Peek, C. Jay, How different visualizations affect human reasoning about uncertainty: zn analysis of visual behaviour, Comput. Human Behav. 92 (October) (2019) 55–64. [9] R. Garcia-retamero, U. Hoffrage, Visual representation of statistical information improves diagnostic inferences in doctors and their patients, Soc. Sci. Med. 83 (2013) 27–33. [10] M. Merz, Interactive) Graphics for biomarker assessment, in: A. Krause, M. O’Connell (Eds.), A Picture is Worth a Thousand Tables: Graphics in Life Sciences, Springer, London, 2012, p. 429. [12] V. Braun, V. Clarke, Using thematic analysis in psychology using thematic analysis in psychology, Qual. Res. Psychol. [Internet] 3 (October 2012) (2008) 37–41, https://doi.org/10.1191/1478088706qp063oa Available from:.
9
Contents lists available at ScienceDirect
International Journal of Medical Informatics journal homepage: www.elsevier.com/locate/ijmedinf
Biomarker data visualisation for decision making in clinical trials a,⁎
b
c
b
T b
Alan Davies , Marisa Cunha , Kamilla Kopec-Harding , Paul Metcalfe , James Weatherall , Caroline Jayc a b c
School of Health Sciences, University of Manchester, Manchester, UK Advanced Analytics Centre, Global Medicines Development, AstraZeneca, Cambridge, UK School of Computer Science, University of Manchester, Manchester, UK
A R T I C LE I N FO
A B S T R A C T
Keywords: Visualizations Biomarkers Clinical trials Interviews Thematic analysis Trust Provenance
Objective: To understand how visualization of biomarker data is used for decision making in clinical trials, and identify problems with and suggest improvements to this process. Methods: We carried out semi-structured interviews with 18 professionals involved in various aspects of developing or using visualizations of biomarker data for decision making in clinical trials. We used an inductive thematic analysis to identify implicit and explicit ideas within the data captured from the interviews. Results: We identified 6 primary themes, including: how visualizations were used in clinical trials; the importance of having a clear understanding of the underlying data; the purpose or use of the visualization, and the properties of the visualizations themselves. The results show that participants’ ‘trust’ in the visualization depends on access to the underlying data, and that there is currently no standard or straightforward way to support this access. Conclusions: Incorporating information about data provenance into biomarker-related visualizations used for decision making in clinical trials may increase users’ trust, and therefore facilitate the decision making process.
1. Background and significance Biomarkers – substances that are measured in a sample or biological system as an indicator of exposure, effect, susceptibility, or clinical disease – are an important tool in drug development. Based on their application, biomarkers can be classified as predisposition, diagnostic, prognostic, and predictive. Biomarker data may hold tremendous amounts of value, with the potential to reduce clinical trial costs, time, and failure rate. To realise this value, insights gleaned from the data must be translated into actions. Effective knowledge acquisition depends on technologies that support the translation of biomarker data into readily understandable information. This process involves the interaction of access to data, its visualization, and its interpretation [1]. The visualization of data can have a profound influence on both the speed of information interpretation, and how that information is interpreted [2,3]. Scientific data visualization aims to support discovery and analysis via the mapping of abstract data structure attributes to visual attributes [4]. When dealing with complex data, both visual search and analysis are utilized depending on the task and the complexity of underlying data [5]. Visualizations that allow for effective interpretation of
⁎
underlying data rely on the use of pre-attentive graphical features that can be processed by the human visual system with limited effort [4]. For more complex visualizations, additional dimensions are often mapped to other visual attributes, such as colour and size, or via use of dynamic filtering [4]. A review of data visualizations used in oncology research by Chia et al. indicated that data visualization in the field of oncology had evolved to represent complex datasets [6]. This includes novel representations of the underlying tumour biology and exploring visualizations that could be used to communicate with both non-specialist audiences and stakeholders [6]. Nevertheless, at present, there are still limitations in the visualisation of large data sets, including possible occlusion of data and potential misinterpretation [7]. An example of this can be seen in probabilistic reasoning, where it was found that different visualization formats resulted in different reasoning strategies [8]. In a study examining the visual representation of statistical information for improving diagnostic inferences involving over 162 participants (81 doctors, 81 patients), it was found that both doctors and patients were able to make more accurate inferences when presented with visual aids representative of numerical information, compared with the numerical information alone [9]. The study also found that the use of visual aids not only improved accuracy and
Corresponding author at: Room 1.001, Vaughan House, Portsmouth St, Manchester, M13 9GB, UK. E-mail address: [email protected] (A. Davies).
https://doi.org/10.1016/j.ijmedinf.2019.104008 Received 27 June 2019; Received in revised form 3 October 2019; Accepted 14 October 2019 1386-5056/ © 2019 Published by Elsevier B.V.
International Journal of Medical Informatics 132 (2019) 104008
A. Davies, et al.
data visualizations for decision making in clinical trials.
participants’ perception of the information's usefulness, but also decreased the perceived difficulty of the task [9]. Different studies highlight the need for the continual evaluation of biomarkers to ensure their validity. Evaluation and discovery of new biomarkers can be supported by data visualization, which is useful for hypothesis generation and gaining insights into the underlying data, as visualization enables patterns and signals to be more rapidly identified by humans compared with data displayed in tabular or numeric forms [10]. Currently, pharmaceutical development primarily relies on listings and tables, often under-utilizing graphics despite their ability to improve the efficiency of data analysis [3]. The aim of our study is to gain a perspective on the current working practices and challenges involved in creating and using biomarker data visualizations for decision making in clinical trials.
2.3. Procedure We conducted both face-to-face and online (Skype) semi-structured interviews. The same interviewer (AD) asked participants a series of questions and audio recorded the interviews. The interview questions were used as prompts for the discussions, which varied in their details according to the role of the interviewee. The broad interview questions were: 1 When reviewing the results of a structured analysis looking at all the biomarkers and patient-level covariates in an early study, how do you currently identify patient sub-groups for subsequent analysis/ investment in these situations? 2 Please describe your current workflow/system 3 What are the advantages of this system? 4 Are there any problems or disadvantages with this system? 5 If you currently use visualizations to identify biomarkers for this or similar purposes: a What visualizations do you use? b How do you use these visualizations? c What are the advantages of these visualizations? d What are the disadvantages of these visualizations? e How could they be improved? 6 What sort of data would be useful for this purpose? 7 How do you think these data/results should be displayed? 8 What role do you think visualization can play in the decision making process? 9 Do you think using visualizations could improve how you make decisions in this regard? a Why/why not? 10 How important or useful do you think visualizations are in this context? 11 What features do you think a visualization for this purpose should include? a Why?
2. Methods We carried out open ended, semi-structured interviews with participants involved in producing or using biomarker data visualizations to make decisions regarding clinical trials. We transcribed the interviews and analysed them using inductive thematic analysis (TA) by following the methods described in [12]. We followed the Standards for Reporting Qualitative Research (SRQR) guidelines [13]. The study received ethical approval from the University of Manchester Research Ethics Committee (4551). All participants provided informed consent prior to taking part in the study. A second coder (KKH) examined a subset of interview quotations (n = 300) and matched them to the primary themes with a percentage agreement of 70.5% and an unweighted Cohen’s Kappa showing substantial agreement (κ = 0.63). 2.1. Researcher characteristics AD has a background in Computer/Data Science and Nursing Science (interventional cardiology). AD has produced routine and bespoke visualisations in most commonly used data science/statistical software for research projects. AD did not personally know any of the interviewees that took part in the study. AD’s research paradigm lies between postpositivism and interpretivist depending on the nature of the research being undertaken. KKH (the second coder) has a background in data management, data science, computational chemistry and pharmacology. KKH has produced standard visualisations for research purposes using a range of statistical tools and generated bespoke interactive and dynamic visualisations for public engagement events.
3. Results Over 8 h of interviews (516 min) with an average duration of 28.66 min (SD = 10.36) were transcribed. Participants reported using a number of tools for visualisation, including: REACT [14], TIBCO Spotfire®, Microsoft Excel and R [15]. The most frequently mentioned were Spotfire® and AstraZeneca's REACT (Fig. 1). Fig. 2 shows how often each named plot was mentioned, with OncoPrints (a type of heatmap), waterfall plots, heatmaps and line plots
2.2. Participants We recruited 18 participants (Table 1) as a sample of convenience from three different organizations using snowball sampling, and ended recruitment on reaching saturation. A participant was identified as someone having a background in either creating or using biomarker Table 1 Participants’ professional role, gender and organization. Role
n
Bioinformatician (BI) Clinical consultant/Medical Scientist (MD/MS) Research software engineer (SE) Statistician/Data Scientist (ST/DS) Gender Male Female Organisation Pharmaceutical company Medical charity University Total
7 5 3 3 13 5 13 4 1 18
Fig. 1. Software tools used to generate visualizations by word frequency. 2
International Journal of Medical Informatics 132 (2019) 104008
A. Davies, et al.
achieve a world where once we have the data, we can kind of push a button, and produce all the analysis and recording. Because of that, the workflow for dynamic visualization is a bit poorly characterized.” [P2, BI] Whilst static visualizations were viewed as more appropriate for regulatory reporting, academic publishers in the field were keener to explore different ways of visualizing the data. “Increasingly journals have been asking for some sort of a simple web interface to explore these images which have been included in a publication, and I think it is just going to become a standard going down the line where every journal will say, right every image you have within your main [inaudible 29:16] script has got a supplementary; we want to see all of these in an interactive dynamic way on a web interface. And again this doesn't have to be a full-fledged web interface or a website like CBioPortal; it can be a simple collection of these figures the readers can play with… It's not yet a standard but it will make life so much easier for the people reading as well as people published.” [P18, BI] The usability of visualization tools (n = 7, [13]) was also highlighted by participants as an issue.
Fig. 2. Plot showing the most mentioned visualization types.
“One of the things that we are really trying to push in-house now is usability and user experience, so that's something that we're trying to pull in earlier in any of the projects that we do.” [P3, ST/DS]
mentioned the most often. Results of the thematic analysis highlighted 6 primary themes and a number of related sub-themes (Table 2 and Fig. 3). When a theme is discussed, the number of participants mentioning the theme (n) is shown followed by the number of references made to the theme in square brackets.
Reproducibility was raised by the majority of participants (n = 16, [61]). Certain plots are used as standard in trials, and as such their production can be automated in a pipeline, which helps to improve reproducibility.
3.1. Application to clinical trials
“The pre-planned visualization is well characterized, so you think about the total graphs you'd want, so it could be means over time or you know just dot and, line plots, forest plots, scatter plots, all those things we can pre-plan, and there's kind of a basic set that you know you want to have to describe the data. Then all along, while you're producing…doing the automated bit, we do have access to dynamic visualization where we'll drill down, mainly to try to explain things we don't quite understand, and there you are sort of letting the data guide you, and there visualization is helpful, but again, how the results of those explorations get translated into a report, is kind of poorly characterized and variable.” [P2, BI]
The various ways visualizations may be used within clinical trials was mentioned 264 times by 18 participants. A key application was trial documentation (n = 12, [35]) where a set of standardized visualizations were often used for reporting of trials. In this context visualisations tended to be traditional static plots (bar, line, etc.). Dynamic visualizations were used less, as they did not currently fit into the workflow. “We try to be very quick in reporting our studies… the core workflow is to automate and pre-plan as much as possible, to try to
Table 2 Interview response themes and sub-themes showing the number of participants in each theme/sub-theme and how many times each theme was mentioned. Theme
Participants
References
Sub-theme
Participants
References
1.0 Application to clinical trials
18
264
2.0 Underlying data
18
110
3.0 Understanding
17
127
4.0 Visualization properties
18
114
5.0 Purpose or use of visualisation
18
174
6.0 Type of visualisation
18
168
1.1 1.2 1.3 1.4 1.5 1.6 1.7 2.1 2.2 2.3 3.1 3.2 4.1 4.2 5.1 5.2 5.3 5.4 6.1 6.2 6.3 6.4 6.5
12 7 16 16 11 10 14 13 17 7 17 7 14 17 8 16 16 7 15 14 10 5 8
35 13 61 34 39 41 41 25 70 13 111 16 43 71 17 61 65 31 42 46 50 17 13
Documentation Evaluation Reproducibility Required properties Drug efficacy Patient risk Working practice Access to data Contextualising data Trusting the data Interpretation Training Labelling/annotation Representation of dimensionality or granularity Barriers to use Data exploration Decision making Synthesising results Bespoke or novel Dynamic Named plot/chart/graph Static Traditional
3
International Journal of Medical Informatics 132 (2019) 104008
A. Davies, et al.
Fig. 3. Treemap visualising the themes and sub-themes by number of participants.
“[…] they'll give us as much data as possible and obviously the more people who have access to it the more likely it is to get out into different arenas or be changed or the message gets changed. You do have to be a bit more cautious about that.” [P4, MD/MS]
It appears that this is less well defined and more complex in the case of dynamic visualizations: “[…] so it lends itself well to if you're all sitting around a computer together, and you're exploring the data, it doesn't lend itself as well to documentation, you know, yes can do screen shots, but then you perhaps have to document how you got to that graph, if for example you did a lot of filtering or rotating or pointing and clicking, how do you document that, and show it to someone else, so that they could reproduce it or understand what you got.” [P2, BI]
Interviewer: “So you think it might be good that for some users those features are not available unless you're a certain type of user or in a certain role and then you can lock…” Participant 10 (MD/MS): “Yes, definitely.” The context of the data was an important consideration. Providing background information about statistics was considered essential. This could be in the form of additional information accompanying the visualization. Effect size and measures of variability were also considered important. It was highlighted that not all the software used to generate visualizations included this information by default, and that additional thought and effort was required to include it.
Most participants (n = 16, [34]) also discussed the features or properties that they thought visualizations should possess. This was expressed as either higher level requirements/standards or in terms of ‘would like to have’ in an ideal or future situation. “…some are. png, some are. jpegs, some are gifs. I think standardizing the output is important to maintain the quality of the visualization, because more often than not it's generated to put on a PowerPoint slide or a publication, and in order to maintain the quality of it I think all of these should have a standard output within a certain quality.” [P18, BI] “Yes, some history or if it's easier to make some kind of bookmarks that can capture both the filter settings and the visualization, corresponding with that, that's a good way to work.” [P1, BI]
“Another point around you know displaying some sort of information about variability, I know that it's kind of a big pet peeve of my boss, is to show a graph and you don't show, you know the sample size or some sort of information about variability, so, yes for example you can have an overall null effect and then do some sub groups, and you find one sub group that looks great, but if that's based on you know just one percent of the data, it's less compelling that if it's, you know, a much more substantial subset, and a lot of software won't include a sample size or variability for free, you've got to add that on.” [P2, BI]
3.2. Underlying data
Having trust in the data related to being able to ‘get back’ to the original data after a number of transformations had been applied prior to the visualization(s).
Several issues pertaining to the data used to generate the visualizations were presented by participants (n = 18, [110]). Issues included who should have access to the data (n = 13, [25]), the context of the data (n = 17, [70]) and the trustworthiness of the data (n = 7, [13]). There was disagreement about who should access data, with participants believing in either complete access:
“Well you just rely on what people tell you and people will tell you what they want to tell you, there's always more to the story kind of thing. So they can present the data in a way that means that…you can present data in any way you want can't you really, it's not…you can present the same data two different ways, but you can only get to the bottom of that if you actually have access to the raw data.” [P4, MD/MS]
“So, this is a personal view, but I think it should be accessible to everybody, because I think, we worry about whether somebody misinterprets it, but equally if they don't have access to it they can't even interpret it the right way. I would say, more eyes on the data than not.” [P13, BI]
The importance of focusing on the data, rather than storytelling, was also raised:
And others stating that access should only be granted to people with a certain role or training:
“You know, if it's a senior person who has now reviewed the data, 4
International Journal of Medical Informatics 132 (2019) 104008
A. Davies, et al.
and they've got a nice set of slides, and the narrative they're telling them is that this drug seems to be working, we just need to add a few more patients, it's all on track. There is a drive at, certainly [name removed for anonymity], and [name removed for anonymity] had a research there about really using the data and evidence-based science, so he drove really hard on it, he doesn't like storytelling, data, data, data […]” [P13, BI]
“The colour scale we use is quite important. So, there is that. It's in [name removed for anonymity] visualization course where she shows you like a rainbow, it's a rainbow colour scale and like a constant…it's based on…basically you see things in a rainbow scale which aren't there because you get quite quick changes in…how much is it, a hue I think it is, compared to lightness or something. But basically, you get these effects which don't really exist if you like.” [P17, SE]
3.3. Visualization properties
It was also mentioned by one participant that there was no official standard for the colours used with OncoPrints, and that this could lead to different people representing different genetic information (i.e. mutations) using different colours. An important aspect of interpretation discussed was training in the use of visualizations (n = 7, [16]).
Labelling and annotation was discussed by 14 participants 43 times. This was often added to visualizations created with one tool using another subsequent tool. Examples of this included using software like PowerPoint to add labels and other annotations to visualizations created with tools such as R. There is, however, a risk of introducing errors in this process. “[…] it can be quite complex, and it can be difficult to give all the information there [in the visualization]. One thing we often do, I think is to get the visualization to PowerPoint and there you can add on this information.” [P1, BI] A further participant mentions that it can be difficult adding this information depending on the software tool used to generate the visualization. They highlight that this is neither efficient nor reliable, but that this practice is the status quo. Another important property of visualizations that was discussed was their ability to represent different dimensions of the data and show the data at different levels of granularity (n = 17, [71]). Being able to go from a population level view of the data to specific data points was considered very important, especially for clinicians.
“CBioPortal is essentially an omics system, so it allows us to generate OncoPrints so we can see genomic heat maps of our tumour biopsies, of our circulating tumour DNA and so on and so forth. We're looking at bringing in kind of immune bio markers into that. Now that really requires a trained translational scientist to be able to do that. That's not trivial and we would not recommend that would be used by someone inexperienced. So access to that is something that we would consider restricting, or not restricting but putting a proviso that any interpretation of the data or those data within the cBioPortal should be done by an expert or someone who's an expert on the system.” [P14, MD/MS] The focus of training was mainly on how to use tools correctly and on cautioning people about the pitfalls and dangers of making conclusions about data based on misinterpretation.
“what that means is essentially a physician or a medical scientist will look at the data, they'll look at that on a patient level which is an individual patient set of data, or they'll look at an aggregate level, so a population view. And an example of what might happen is they'll review the adverse events and any associated lab data to support those adverse events, looking for trends, looking for kinds of things that need immediate attention.” [P3, ST/DS]
“We think all the visualizations on [name removed for anonymity] are simple-ish. With all the filters it becomes complex and too complex for people, but we do require them all to do training first before they can use it for that reason. Because you don't want people just going in, not knowing what they're doing, not set the filters correctly, think they've got something that represents what they want and then go and take away the grand conclusion.” [P10, MD/ MS]
There were also occasions where it was considered more beneficial to use multiple plots with lower dimensions than try to include higher dimensions in visualisations, as highlighted by one participant:
Another participant pointed out the issue of medical liability when interpreting data, especially when communicating results to patients themselves.
“Three dimensional plots are lovely to look at and you can move them up and down, left and right but they're actually very, very difficult to interpret, very, very difficult. Not only are they difficult to interpret they actually can be quite dangerous to interpret because you can miss things on them. […]It's actually sometimes better just to have two separate plots with a common axis.” [P14, MD/MS]
“I think there's a lot of thinking about medical liability in their minds. So, if they have too much data, or data that they're not quite sure what it means, and they don't act on it, then they're very uncomfortable. If they know that there's four genes and their associated variance, if they know what they mean, they're happy about that. Because, the science is advancing so quickly, if you say, hey, you might want to take a look at this, there's a paper just been published on it, you might want to look at it, unless they're an experimental geneticist really deep in it, they don't really want to be in that space.” [P13, BI]
3.4. Understanding The ability to interpret the visualization was connected to the properties of the visualization, such as the labelling/annotation, which in turn were dependent on the contextualization of the underlying data.
3.5. Visualization purpose or use
“Quite often we see people using percentages but then when you say, well what's that, you know, it sounds like an impressive percentage but then it's based upon 20 samples or whatever, you know, it's quite a small number. So it can give quite a false message unless you've got that information about your sample size or the error bars.” [P16, BI]
Three primary reasons were given by participants for using visualizations. These included: data exploration (n = 16, [61]), decision making (n = 16, [65]) and synthesizing results (n = 7, [31]). Using visualizations for data exploration was particularly important in early stage clinical trials, where users focus on signal detection (i.e. find a pattern in the data that may lead to further investigation) and explore hypotheses.
Interpretation was discussed by 17 participants 111 times. The colours used in some visualizations, such as heatmaps and OncoPrints were highlighted as factors that influenced interpretation of visualizations.
“This is hypothesis generating rather than hypothesis confirming. We want to have the ability to explore the results/the data and to try to understand the different perspectives…” [P11, MD/MS] 5
International Journal of Medical Informatics 132 (2019) 104008
A. Davies, et al.
In late stage clinical trials, visualization was used more for confirmatory purposes. Visualization was considered to be important or essential in the clinical trial go/no-go decision-making process.
“…there are certain times when an end user will come to you and say, look, I really need this visualization, and you've a choice. You've got this generic tool in front of you. So other people use Excel and you say, well, Excel will only get me this far, I probably need to look at maybe Spotfire to do more complex interrogation of data but it's going to take me longer and you're kind of in your head you're trying to balance, which one do I do, because once you're committed to one you kind of have to keep going with it. It's kind of similar for [name removed for anonymity] is that what we've tried to do is the more commonly used one so it's the 80/20 rule. There's about 20 per cent of visualizations that are commonly used and we try to sort of address those most commonly used visualizations. Then there are those other ones that are more specific.” [P14, MD/MS]
“A lot of times we'll do kind of…standardized effect plots where you do some sort of adjustment for the number of different sub groups or biomarkers you're presenting, and in that case, the graphic becomes very important, and fundamental in decision making around, you know, should we run a new trial to understand is there an effect in a sub group, or do we think it's just a chance finding. So, the visualization is much more helpful than say, you know looking at lots of different tables, or you know, having P values, or some sort of numerical measure of strength of effect.” [P2, BI] When it came to the synthesis of results, visualizations were often included in presentations to communicate findings and assist understanding.
Fourteen participants discussed novel visualizations and all agreed that there was a place for new ways of visualizing their data. They did, however, caution against the potential to misinterpret such visualizations due to users' unfamiliarity with them: “There have been instances where we've kind of developed more novel visualizations, and there's absolutely a place for, you know, the… like the starburst graphs, the network diagrams and all the other different types of visuals that you can do. The thing that we have to be mindful of is a lot of the users, you know, they're obviously more familiar with what a bar chart is and you don't have to tell them how to interpret it. So we tend to use the more traditional ones, partly for that reason but also because the data lends itself to that.” [P3, ST/DS] This participant went on to say: Participant 3 (ST/DS): “Now one of the things I personally am quite keen on, but also within IT we're quite keen on, is identifying what's the best way of displaying data. So not what's the easy way, i.e. the bar chart, but actually is there something out there that would give you a much better presentation of that data or allow you to show more dimensions? So, you know, not just shape, colour, X and Y axes, but are there some other things that we could be looking at? We did have an example actually which was really cool of a, like a dynamic, like a true dynamic visualization where it was like a video almost. I don't know if you've seen that famous one? It's completely gone off the top of my head, but it's to do with…it's something to do with poverty or some kind of financial index and all the different
“Oh yes, certainly, if you're doing a presentation, yes, I would definitely convert it into visual formats, because it's asking someone to interpret data very quickly, I think you have to.” [P15, MD/MS] Participants also identified problems with using visualizations (n = 8, [17]), including greater numbers of people requesting more and different ways of viewing things, certain visualizations not being validated, and the difficulty of accurate interpretation: “If you don't use the right visualization then you go home with a very wrong message of what your data is telling you. On how comprehensive is the visualization […] often things like OncoPrint it's very easy to filter out samples without a single variant. If there is no alteration defined or found in a sample, it's very common to filter out samples because it's easier to see.” [P18, BI]
3.6. Types of visualization All participants (n = 18, [168]) mentioned using specific plots in their work, such as waterfall plots, forrest plots and OncoPrints (Fig. 4), as well as more traditional line and bar charts/plots. Participants also discussed bespoke plots (n = 15, [42]).
Fig. 4. An example of an OncoPrint. 6
International Journal of Medical Informatics 132 (2019) 104008
A. Davies, et al.
context has on decision making when we present the same information in different contextual situations. As well as context, trust in the underlying data also appears to be a factor in the extent that people would rely on the data visualization for subsequent decision making. In this case trust is related to the accuracy of the data, what it represents and knowledge of any transformations that it has undergone. Participants’ subsequent trust in visualization appears to be mitigated by their access to the underlying data leading them to trust the visualization more if they have access to the underlying data. A lack of time within their job role, a lack of knowledge about visualization construction, and the lack of tools to build visualizations efficiently, may be why people often use visualizations constructed by others [19]. Increased access to tools that make it possible to quickly and easily generate visualizations may remove the potential “bottle neck” of relying on one or more persons to generate the visualization for another to interpret or use. It is conceivable that desiring access to the “raw” or original data may reflect a perceived need for access to the data that in reality would not be of practical use. However, given the comments of participants who were in favour of increased access and “more eyes on the data”, supporting this may be beneficial, particularly given the high stakes involved in running clinical trials, both financially and from a patient safety perspective. These findings highlight the importance of maintaining and providing access to data provenance in clinical trial decision making, which can be defined as a record trail accounting for the origins of data in a storage medium, such as a database or other document [20]. The provenance of data is also critical for its subsequent sharing and reuse [21]. As highlighted in the interviews, data visualization can facilitate understanding of the data and reduce “information overload” [5], or alternatively obfuscate it [3]. Complex data requires transformation prior to it being queried, grouped or compared [5]. Potential errors and the different choices made along the way can affect the end outcome. Recording and availability of information concerning data lineage and provenance can aid interpretation and reproducibility. The requirements for reproducibility related to decision support have been identified by Curcin et al. [22] as: transparency of the system; auditability and reproducibility of recommendations; validation of readiness; tractability of evidence; responsibility; privacy/security; usability; and scalability [22]. Dynamic visualization offers a way of mitigating information overload [5], and this was mentioned by several participants. This allows for highly complex data, or data with high dimensionality, to be reduced, as the visualization is dependent on both perception and the cognitive manipulation of objects in space [23]. This form of visualization, although useful for data exploration, is not as suitable for documentation, for which static printable images are preferred. Dynamic visualization also suffers from reduced reproducibility, due to the filtering and other manipulations applied to the data. This is also related to the issue of hypothesis vs. data-driven research, which in the former case relies on the human ability to formulate meaningful questions and determine how these can be addressed using data, and in the latter case on machine learning/pattern mining techniques that automatically extract patterns from data [7]. By their nature, biomarker data are often multivariate, including measurements taken at different points in time [10]. Due to the superior ability of humans to process visual information compared with tabular or numeric information [10], visualizations can aid humans working with such data. Visualizations, if constructed and used well, can help humans to detect signals and patterns in large amounts of data. This supports a method of combining the best of human expertise with machine learning algorithms within drug development. Substantive work has been carried out in the visualization field examining how humans interact with visualization (see, [24] [25], and [26] for examples). Despite this, further work is required to generate foundational theories for validation and understanding of design work in the InfoVis community, a community that hosts resources for information visualization [27].
countries…” Interviewer: “Oh.” Participant 3 (ST/DS): “…and how it changes over the years.” Interviewer: “Yeah, the Hans Rosling thing…” Participant 3 (ST/DS): “Yes.” Interviewer: “…the Gapminder one, yeah.” Participant 3 (ST/DS): “Yes, that's the one, that's the one. So it was kind of a similar thing to that where you could press play and then watch all the bits move, and then pause at a certain point so it, you know, having that extra dimension. Then it gives you additional information that you can interpret.” Dynamic visualization (n = 14, [46]) was also discussed. In the context of the interviews, this related to the ability to interact with the visualization i.e. to zoom, rotate, highlight properties etc. It appears that these sorts of visualizations were used often earlier on in a given trial for the exploration phase, rather than later in the trials where more regulated and confirmatory visualizations were used. “…but getting round the fact, the problem that dynamic visualizations can lead to very subtly different, but perhaps very different interpretations, with the application of the slightly different backgrounds. That's the thing that we did quite a lot of work on with [name removed for anonymity]. It works well.” [P10, MD/MS] Static (n = 5, [17]) and traditional (n = 8, [13]) visualizations were mentioned. The majority of plots used in practice tended to be of a more static and traditional type (box plots, bar and line etc.) than dynamic or bespoke visualizations. “So I would say that the majority of the visualization tools that we have and that we use the more traditional graphics, so as you say, bar charts, line charts, et cetera.” [P3, ST/DS] 3.7. Conceptual model Based on the findings of the interviews, it appears that the interpretation of the visualization is dependent on several mediating factors. The interpreter's ability to interpret the graphic derives from their current knowledge and understanding of the data, and its chosen form of representation. Understanding and trust appears to be influenced by having access to the initial data from which the visualization was derived, which also increases trust in the visualization output and therefore the likelihood of using it for subsequent decision making. The effectiveness in conveying the intended message is also dependent on the skill and ability of the visualization creator to accurately communicate the underlying data. This is influenced by the purpose of the visualization and the available tools used to create it. This in turn mediates the type of visualization created and its properties (labelling, colours used, additional information included) (Fig. 5). 4. Discussion With our semi-structured interview, we were able to gather information about the working practices and challenges faced by 18 professionals involved in developing or using visualizations for decision making in clinical trials. We grouped this information into different themes and sub-themes to facilitate analysis and identified that the most referred information was related to contextualising data, representation of data dimensionality or granularity, and data interpretation. These are all known to be critical steps for decision making. It is well accepted that the context in which an option is presented can influence the evaluation of the option and some studies are evaluating the context effect in decision making [16] and policy formation [17], as well as for making risky choices [18]. Being context-aware is important, as it has been demonstrated that people do not always evaluate options independently. The set of choice options can influence the evaluation of single options [18]. This highlights the importance of the impact that 7
International Journal of Medical Informatics 132 (2019) 104008
A. Davies, et al.
Fig. 5. Conceptual model of data visualization process for clinical trial decision making.
in this area.
Building complex visualizations requires additional skills in areas such as software engineering. Programming is being used increasingly by decision analysts [28], with software such as R [15], Python [29] and MATLAB [30] being widely used for data science due to the combined features of reproducibility, documentation, automation and advanced visualization tools [28]. The skills required to use such tools effectively differ from those required for more traditional spreadsheet style tools like Excel. Due to the time, effort and skill involved in learning and utilizing such tools it is not surprising that many bespoke visualizations are requested by users from those with the relevant skills, rather than the requester opting to generate such visualizations themselves. Production of tools that allow for the rapid creation of visualization and include default information on variability and data context could help to overcome this barrier. Alternatively if visualizations continue to be provided by others, they should include information on data provenance and lineage by default, in order to increase trust in subsequent decisions.
6. Conclusions Given the complexity of the data analysis from which biomarkers are derived and the ever expanding quantity of available data, the importance of visualization in reducing complexity and mental overload becomes increasingly relevant. This challenge can be met by the development of visualization tools that allow the rapid creation of visualizations to represent these data for data exploration and confirmatory purposes. Our study also indicates that given the nature of clinical trials, data provenance and lineage are important considerations and need to be communicated to the user of the visualization to increase trust in the visualization, interpretability and subsequent confidence in decisions made based upon such visualizations. To ensure good decisions, visualizations need to provide the right context. Therefore, we recommend building feedback on data provenance into future visualization tools and adding more context by default to visualisations.
5. Limitations The primary limitation of this study is that the majority of participants were from the same pharmaceutical company. This may in turn have had an impact on the frequency of times they mentioned the REACT tool, which was developed internally by the same company, as well as some of the other software tools that were commonly used by participants, as these may also be standard in this setting. It is likely however those tools are used to carry out this type of work more broadly and thus reflects industry standards. Additionally due to the regulations surrounding the later phases of clinical trials, the type of visualisations used and their use for decision making are limited by regulatory process, in contrast to the earlier data exploration phases of trial development. Finally the study faces limitations inherent to all interview studies and qualitative research in general, such as increased subjectivity and reduced generalizability when compared to quantitative methods. These methods do however offer us a starting point from which to explore the issues covered in this study and focus future work
Authorship statement All persons who meet authorship criteria are listed as authors, and all authors certify that they have participated sufficiently in the work to take public responsibility for the content, including participation in the concept, design, analysis, writing, or revision of the manuscript. Furthermore, each author certifies that this material or similar material has not been and will not be submitted to or published in any other publication before its appearance in the International Journal of Medical Informatics. Specifically: A.D. carried out the interviews, analysis and writing of the manuscript. C.J., and M.C. contributed to the design of the study. K.KH carried out the second coding of the interview transcripts. All authors contributed to discussions about the direction of the work and read and made edits to the manuscript. 8
International Journal of Medical Informatics 132 (2019) 104008
A. Davies, et al.
Declaration of Competing Interest
[13] B.C. O’Brien, I.B. Harris, T.J. Beckman, D.A. Reed, D.A. Cook, Standards for reporting qualitative research: a synthesis of recommendations, Acad. Med. 89 (9) (2014) 1245–1251. [14] AstraZeneca, REal Time Analytics for Clinical Trials (REACT) [Internet], Available from: (2019) https://www.astrazeneca.com/what-science-can-do/our-technology/ real-time-analytics-for-clinical-trials-react.html. [15] R Core Team, R: A Language and Environment for Statistical Computing [Internet]. Vienna: R Foundation for Statistical Computing, Available from: (2014) http:// www.r-project.org/. [16] D. Dowding, J.A. Merrill, N. Onorato, Y. Barrón, R.J. Rosati, D. Russell, The impact of home care nurses’ numeracy and graph literacy on comprehension of visual display information: implications for dashboard design, J. Am. Med. Inform. Assoc. 25 (2) (2018) 175–182. [17] P. Monsivais, O. Francis, R. Lovelace, M. Chang, E. Strachan, T. Burgoine, Data visualisation to support obesity policy: case studies of data tools for planning and transport policy in the UK, Int. J. Obes. [Internet] 42 (12) (2018) 1977–1986, https://doi.org/10.1038/s41366-018-0243-6 Available from:. [18] P.N.C. Mohr, H.R. Heekeren, J. Rieskamp, Attraction effect in risky choice can Be explained by subjective distance between choice alternatives, Sci. Rep. 7 (1) (2017) 1–10. [19] Bradstreet T. Grables, Visual displays that combine the Best attributes of graphs and tables, in: A. Krause, M. O’Connell (Eds.), A Picture is Worth a Thousand Tables: Graphics in Life Sciences, Springer, London, 2012, p. 429. [20] A. Gupta, Data provenance, in: M. LING LIU, T. ÖZSU (Eds.), Encyclopedia of Database Systems, Springer, Boston, 2009. [21] I. Suriarachchi, Q. Zhou, Plale B. Komadu, A capture and visualization system for scientific data provenance, J. Open Res. Softw. 3 (1) (2015) 1–7. [22] V. Curcin, E. Fairweather, R. Danger, D. Corrigan, Templates as a method for implementing data provenance in decision support systems, J. Biomed. Inform. (65) (2017) 1–21. [23] J.K. Gilbert, M. Reiner, M. Nakhleh, Visualization: Theory and Practice in Science Education, Springer, 2008, p. 326. [24] E.R. Tufte, The visual display of quantitative information [Internet], Technometrics 2nd (2001) 197. Available from: http://www.amazon.co.uk/dp/0961392142. [25] S.K. Card, J. Mackinlay, B. Shneiderman, Readings in Information Visualization: Using Vision to Think, Morgan Kaufmann Publishers, San Francisco, 1999. [26] Ware C. Information Visualization: Perception for Design, 3rd ed., Morgan Kaufmann, Massachusetts, 2012, p. 536. [27] Z. Liu, J. Stasko, Theories in information visualization: what, why and how, Inf. Vis. (2010). [28] H. Jalal, P. Pechlivanoglou, E. Krijkamp, F. Alarid-Escudero, E. Enns, M.G. Myriam Hunink, An Overview of R in health decision sciences, Med. Decis. Mak. 37 (7) (2017) 735–746. [29] Python Software Foundation, Python [Internet], [cited 2019 Feb 10]. Available from: (2019) https://www.python.org/. [30] MathWorks, MATLAB 9.6, MathWorks, Massachusetts, 2019.
This work was funded by AstraZeneca. No other competing interests exist. Acknowledgements We would like to thank the Advanced Analytics Centre at AstraZeneca for funding this work. References [1] M. Gaylord, J. Calley, H. Qiang, E.W. Su, B. Liao, A flexible integration and visualisation system for biomarker discovery, Appl. Bioinformatics 5 (4) (2006) 219–223. [2] B. Reeder, H.J. Thompson, G. Demiris, Health providers’ perceptions of novel approaches to visualizing integrated health information, Methods Inf. Med. 52 (3) (2013) 250–258. [3] A. Krause, Concepts and principles of clinical data graphics, in: A. Krause, M. O’Connell (Eds.), A Picture is Worth a Thousand Tables: Graphics in Life Sciences, Springer, London, 2012, p. 429. [4] J. Fekete, C. Plaisant, Interactive information visualization of a million items proceedings of IEEE symposium on information visualization, The Craft of Information Visualization: Readings and Reflections Vol. 2002 Elsevier Science, USA), 2002, pp. 279–286. [5] J. Dill, R. Earnshaw, D. Kasik, J. Vince, P.C. Wong, Expanding the Frontiers of Visual Analytics and Visualization, Springer, London, 2012, p. 531. [6] P.L. Chia, C. Gedye, P.C. Boutros, P. Wheatley-Price, T. John, Current and evolving methods to visualize biological data in cancer research, J. Natl. Cancer Inst. 108 (8) (2016) 1–11. [7] B. Shneiderman, Inventing discovery tools: combining information visualization with data mining, Inf. Vis. 1 (1) (2002) 5–12. [8] M. Reani, N. Peek, C. Jay, How different visualizations affect human reasoning about uncertainty: zn analysis of visual behaviour, Comput. Human Behav. 92 (October) (2019) 55–64. [9] R. Garcia-retamero, U. Hoffrage, Visual representation of statistical information improves diagnostic inferences in doctors and their patients, Soc. Sci. Med. 83 (2013) 27–33. [10] M. Merz, Interactive) Graphics for biomarker assessment, in: A. Krause, M. O’Connell (Eds.), A Picture is Worth a Thousand Tables: Graphics in Life Sciences, Springer, London, 2012, p. 429. [12] V. Braun, V. Clarke, Using thematic analysis in psychology using thematic analysis in psychology, Qual. Res. Psychol. [Internet] 3 (October 2012) (2008) 37–41, https://doi.org/10.1191/1478088706qp063oa Available from:.
9