Liver lesion detection in CT using deep learning techniques

CHAPTER 3

Liver lesion detection in CT using deep learning techniques Avi Ben-Cohen, Hayit Greenspan Tel Aviv University, Faculty of Engineering, Department of Biomedical Engineering, Medical Image Processing Laboratory, Tel Aviv, Israel

Contents 3.1. Introduction 3.1.1 Prior work: segmentation vs. detection 3.1.2 FCN for pixel-to-pixel transformations 3.2. Fully convolutional network for liver lesion detection in CT examinations 3.2.1 Lesion candidate detection via a fully convolutional network architecture 3.2.1.1 FCN candidate generation results 3.2.2 Superpixel sparse-based classification for false-positives reduction 3.2.3 Experiments and results 3.2.3.1 Data 3.2.3.2 Comparative system performance 3.3. Fully convolutional network for CT to PET synthesis to augment malignant liver lesion detection 3.3.1 Related work 3.3.2 Deep learning-based virtual-PET generation 3.3.2.1 Training data preparation 3.3.2.2 The networks 3.3.2.3 SUV-adapted loss function 3.3.3 Experiments and results 3.3.3.1 Dataset 3.3.3.2 Experimental setting 3.3.3.3 Liver lesion detection using the virtual-PET 3.4. Discussion and conclusions Acknowledgments References

65 66 67 68 68 70 71 73 73 74 77 78 79 79 80 81 82 82 82 83 84 87 87

3.1. Introduction In this chapter we focus on liver analysis using deep learning tools. Specifically we focus on liver lesion detection. CT is the most important imaging tool for diagnosis and follow-up of oncological patients. For example, in 2007, over 70 million CT scans were performed in the United States [1]. CT images are widely used by clinicians for detection, diagnosis, and monitoring of liver lesions [2]. Images are acquired before and after intravenous injection of a contrast agent. Radiologists usually detect and diagnose liver Handbook of Medical Image Computing and Computer Assisted Intervention https://doi.org/10.1016/B978-0-12-816176-0.00008-9

Copyright © 2020 Elsevier Inc. All rights reserved.

65

66

Handbook of Medical Image Computing and Computer Assisted Intervention

lesions based on the different density of the lesions which are expressed in Hounsfield units (HU) with different scan timing and an optimal detection in the portal phase (60 seconds post injection) [3]. A variety of lesions, both benign and malignant, can be found in the liver. Among them, metastasis are the major cause of death from cancer [4]. The liver is frequently affected by metastatic disease, arising commonly from primary sites in the colon, breast, lung, pancreas, and stomach [5]. There is a large variability in the number, size, and general appearance of metastatic liver lesions, making computerized metastatic liver lesion detection a challenging task. Fig. 3.1 presents a few liver examples with metastasis. Current radiological practice is to manually analyze the liver. This is a timeconsuming task requiring the radiologist to search through a 3D CT scan which may include hundreds of slices and multiple lesions. For early detection purposes, it is of particular importance to detect what are known as “too small to characterize” lesions. This requires additional radiologist time and focus, and is considered a difficult challenge. Given the advances in artificial intelligence capabilities in recent years and more specifically deep learning methods, the integration of these tools in clinical practice can lead to a much more efficient and accurate diagnosis and treatment. Thus, developing computational tools for CT interpretations has a high clinical value, as well as economic potential.

3.1.1 Prior work: segmentation vs. detection Automated computerized analysis tools are starting to emerge for the detection, segmentation, and characterization of liver lesions. One important distinction to note is the difference in definitions and in the objectives for a segmentation vs. detection task. In the detection task the objective is to find all lesions. This includes very small lesions, as finding those can support the radiologist’s diagnosis to an even greater extent than detecting large obvious lesions. Therefore, in evaluating automated detection system performance, we need to use an evaluation measure that penalizes a miss of a small and large lesion equally. This often implies increasing the sensitivity – so as to catch all possible lesion candidates. The challenge is then to achieve the high sensitivity goal without increasing the detection of false-positive candidates. It is common practice to aim for a false-positive rate of between 1–3 lesions per case, in order to provide a tool that will be useful in actual clinical practice. The objective of the segmentation task is to output a lesion segmentation mask similar to the one manually circumscribed by experts. A segmentation result can miss one or two small lesions and still get a high segmentation score if the larger lesions were correctly segmented. Fig. 3.1 demonstrated the difference in the output provided by a detection versus segmentation mask. The liver lesion segmentation task has attracted a great deal of attention in recent years, including the MICCAI 2008 Grand Challenge [6], ISBI 2017, and MICCAI 2017 LiTS challenge [7].

Liver lesion detection in CT using deep learning techniques

Figure 3.1 Liver lesion examples: (A)–(C) include a cropped CT image with lesions marked by a blue rectangle. Detection and segmentation examples: (D) cropped CT including two liver lesions; (E) lesion detection; (F) lesion segmentation masks. The segmentation mask includes every voxel inside the lesion (in blue). The detection includes a marked point inside the lesion.

In this chapter we focus on the liver lesion detection task. Prior work in the detection task involves region-based approaches (e.g., [8]) as well as patch-based approaches (e.g., [9]). In the work presented herein, we focused on a global analysis scheme via a fully-convolutional network (FCN), along with a localized refining stage. We also use cross-modality analysis to augment the system performance further. The use of the FCN as a tool for global image analysis will be introduced next. Section 3.2 presents our FCN-based combined global and local scheme for the lesion detection task. In Sect. 3.3 we describe the combined PET and CT solution we developed.

3.1.2 FCN for pixel-to-pixel transformations The Fully Convolutional Network (FCN) [10] has been increasingly used in different medical image segmentation problems. In recent years we also see its use in liver tumor segmentation and detection tasks [11–14]. The FCN was introduced in the image segmentation domain, as an alternative to using image patches. Using FCN, the image could be analyzed globally instead of using localized patches. Thus, there is no need to select representative patches, eliminate redundant calculations where patches overlap, and scale up more efficiently with image resolution. Moreover, there can be a fusion of different scales by adding links that combine the final prediction layer with lower layers with finer strides. Since all the fully connected layers are replaced by convolution layers, the FCN can take inputs of arbitrary sizes and produce correspondingly-sized outputs with efficient inference and learning. Unlike patch based methods, the loss function

67

68

Handbook of Medical Image Computing and Computer Assisted Intervention

using this architecture is computed over the entire image segmentation result. The most common FCN used in medical imaging applications is the U-Net [15]. The U-Net architecture consists of a contracting path to capture context and a symmetric expanding path that enables precise localization. In general, the FCN architectures can be used for tasks that involve a pixel-wise loss function such as reconstruction, synthesis, and other image-to-image tasks.

3.2. Fully convolutional network for liver lesion detection in CT examinations We describe next a fully automated system that can analyze an input CT examination, and detect liver metastases with high accuracy, specifically by exploring detection performance of small lesions. The system comprises two main stages: In the first stage, lesion candidates are generated from the input CT volume. The focus of this stage is on high sensitivity – so as not to miss lesions. In the second stage, each lesion candidate is further reviewed, to validate its categorization as a lesion. The goal of this stage is to fine-tune the original detection by removing false-positives from the system. In most of this chapter, we will assume that the liver has been segmented, and we focus on detecting lesions within it. Experimental results for a complete system that automatically segments the liver as one of its tasks, will be provided as well. The described system is based on several of our published works [16,14,17].

3.2.1 Lesion candidate detection via a fully convolutional network architecture In their reading routine, radiologists scan axial slices of the liver. Following that routine, the input to the system consists of axial slices from the CT volume. In a classical network-based solution, these slices are input to the network sequentially, with the slice of interest copied three times in order to utilize the standard networks that are commonly pretrained on color images and expect three channels as input. When considering the context of lesions, we note that the lesions are present in several sequential slices. In the regular screening routine, radiologists scan across the slices in order to detect and verify findings. Following this, it was suggested in [18] to provide three slices to the network simultaneously: the target slice in the center and two adjacent slices above and below. Using an FCN-based analysis, this module learns to output a lesion probability map. In the following we provide details about the FCN architecture used: The proposed network architecture is based on that proposed by Long et al. [10]. It uses the VGG-16 architecture [19] by discarding the final classifier layer and converting all fully connected layers to convolutions. We appended a 1 × 1 convolution with channel dimension 2 to predict scores for lesion or liver at each of the coarse output locations, followed by a

Liver lesion detection in CT using deep learning techniques

Figure 3.2 FCN-4s network architecture. Each convolution layer is illustrated by a straight line with the receptive field size and number of channels denoted above. The ReLU activation function and drop-out are not shown for brevity.

deconvolution layer to upsample the coarse outputs to pixel-dense outputs. The upsampling was performed in-network for end-to-end learning by backpropagation from the pixelwise loss. However, since small lesions require higher resolution for detection, we used an additional lower level linking, which make use of higher resolution for the final output. This is done by linking the second max-pool layer in a similar way to the linking of the third and fourth max-pool layers. The final FCN network architecture, termed FCN-4s net, is presented in Fig. 3.2. The fusion steps within the FCN architecture were shown to support the analysis of different lesion sizes. The output of the network is a lesion probability map. Additional network implementation details are provided next: To achieve fixed inslice resolution across all patient data, as well as fixed slice spacing in the network input, we resized (using linear interpolation) to obtain a fixed pixel spacing of 0.71 mm and a fixed slice spacing between adjacent slices of 1 mm. In this work, we focus on liver metastases that vary considerably in size. To reflect this in the training data, we applied

69

70

Handbook of Medical Image Computing and Computer Assisted Intervention

different scale transformations to the available training images in each batch. For each input image in each batch, two numerical values between 0.9 to 1.1 were randomly selected (uniform distribution). Two corresponding scale transformations were applied to the image (3 adjacent slices), using nearest-neighbor for resampling, to generate two training images. Training of the network was conducted as follows: Input images along with the corresponding liver and lesion segmentation masks provided by the human expert were used to train the network. Examples of masks are shown in Fig. 3.4. We used the stochastic gradient descent with GPU acceleration (Nvidia GTX 1080). Regions surrounding the liver, including the different organs and tissues, were ignored during training. The softmax log-loss function was computed pixel-wise with different weights for each class of pixels to overcome class imbalance: Most of the pixels in each image belonged to the liver, and a much smaller number belonged to the lesions. The learning process was therefore balanced using fixed weights that were inversely proportional to the population ratios. The learning rate was chosen to be 0.0005 for the first 30 epochs and 0.0001 for the last 40 epochs (a total of 70 epochs). The weight decay was chosen to be 0.0005 and the momentum parameter was 0.9.

3.2.1.1 FCN candidate generation results Experiments were conducted to evaluate the FCN across several architecture variations. We start with the well-known FCN architecture, termed FCN-8s [10], as the baseline architecture. To that we add the proposed usage of adjacent slices as input as well as the additional lower level linking. In this initial experiment, we used CT scans from the Sheba medical center, taken during the period from 2009 to 2015. Different CT scanners were used with 0.71–1.17 mm pixel spacing and 1.25–5 mm slice thickness. Markings of the entire liver and hepatic metastases’ boundaries were conducted by a board certified radiologist (E.K. with eight years of experience) and a radiology resident (A.K. with one year of experience), in consensus. Each CT image was resized to obtain a fixed pixel spacing of 0.71 mm. They included 20 patients with 1–3 CT examinations per patient and overall 68 lesion segmentation masks and 43 liver segmentation masks. The data included a variety of liver metastatic lesions derived from different primary cancers. Each lesion was defined based on the radiologist’s annotations. In case of overlapping lesions they were considered as one. In assessing the automated system performance, certain definitions and measures are needed: We defined a lesion as detected when a certain connected component (CC) in the system output overlapped with the radiologist’s marked lesion mask. Two detection measures were used here: • •

TPR – number of correctly detected lesions divided by the total number of lesions. FPC – defined as the total number of false detections divided by the number of patients.

Liver lesion detection in CT using deep learning techniques

Table 3.1 Testing the FCN using additional adjacent slices (3 slices) and linking lower level pooling layer (FCN-4s). Method TPR FPC

FCN-4s 3 slices FCN-8s 3 slices FCN-8s

0.88 0.86 0.85

0.74 1.1 1.1

A 3-fold cross validation (CV) was used (each group containing different patients) using the lesion detection dataset. Results are presented in Table 3.1. We note that the FCN-4s with the addition of neighbor slices performed better than the other methods, with better TPR and better FPC. Intuitively, we believe that in our task of lesion detection, the FCN-4s is more appropriate than the FCN-8s due to the fact that lesions are inherently smaller in size than general objects in natural scenery or other real-world domains.

3.2.2 Superpixel sparse-based classification for false-positives reduction We next propose to combine the global FCN-based system, with local patch level analysis for false-positive reduction. The proposed liver lesion detection framework is comprised of two main modules, as shown in Fig. 3.3. The first module is an FCN. Based on the high-probability candidate lesion regions from the first module, the second module follows with localized patch level analysis using superpixel sparse based classification. This module’s objective is to classify each localized superpixel as a lesion or not. Thus it provides a fine-tuning step, with the objective of increasing sensitivity to lesions while removing false-positives (FPs). This work was published in [17]. The combination of global and local analysis for false-positive reduction has been shown to be a promising approach in a range of medical applications [20–22]. In this particular work, we represent the local patch information using superpixels to extract homogeneous localized regions and a sparse representation to represent the content of those localized regions. Superpixels provide an efficient way to select patches of interest and to avoid the redundancy of densely extracted patches. We divided each liver in each CT examination into a set of homogeneous superpixels. Each image was clustered into N superpixels using the well-known SLIC algorithm [23], where N is chosen so that each superpixel will be approximately of size 25 mm2 . The SLIC algorithm is known to preserve strong edges. This ensures that the superpixels align to preserve the overall liver boundary and the lesion boundaries. As in our network training, non-liver superpixels were not included in the classification. For each candidate region a superpixel map is generated, and for each superpixel, a representative feature vector is extracted. Using a pre-learned dictionary, a sparse-code representation is computed and classified. The rationale for using superpixels as well as

71

72

Handbook of Medical Image Computing and Computer Assisted Intervention

Figure 3.3 The proposed lesion detection framework.

the generation of the superpixel representation are described in detail in [16,17]. The LC-KSVD method was used here [24]. This method aims to leverage the supervised information of input signals or feature vectors to learn a reconstructive and discriminative dictionary. The dictionary items were chosen so that each one represented a subset of the training feature vectors ideally from a single class. Hence, each dictionary item could be associated with a single label. Thus there was an explicit correspondence between dictionary items and the appropriate labels. This connection was created using a label consistency regularization term and a joint classification error in the objective function for learning a dictionary with more balanced reconstructive and discriminative power. The learned classifier was used to classify superpixel representation as lesion or non-lesion. An additional fine-tuning was conducted next, using the FCN output to enable dictionary and classifier personalization. A selection of superpixels was made, from each given test case, based on the corresponding FCN probability map. This was done as follows: Superpixels with an average probability above a high-value threshold (0.98) were labeled “lesion” and those with an average probability below a low-threshold (0.2) of were labeled “non-lesion”. Using this set of selected superpixels (in cases where the number of lesion superpixels was at least half the dictionary size), the sparsity based dictionary and the linear classifier parameters were fine-tuned. Following the fine-tuning stage, all candidate superpixels from the first module were classified as lesion or non-lesion using the jointly learned linear classifier, to provide a

Liver lesion detection in CT using deep learning techniques

Figure 3.4 Experts’ segmentation masks: (A) axial view; (B) coronal view; (C) 3D segmentation mask; liver mask in red and lesion mask in green.

binary detection map for the input CT scan. Next we used a connected-component labeling algorithm to transform the binary detection map to a lesion detection map, where each connected-component represented a lesion detected by our system.

3.2.3 Experiments and results To evaluate the performance of the system, we conducted several sets of experiments. These included evaluations of a development set followed by the results of an independent testing set. We focused on the lesion detection within the liver to evaluate the detection performance independent of automated liver segmentation capabilities. Thus, the majority of the experiments and evaluations were constrained to the manually circumscribed liver area. As a final experiment, we included our own liver segmentation algorithm in a fully automated liver segmentation and lesion detection framework.

3.2.3.1 Data The data used in the current work included CT scans from the Sheba Medical Center. Algorithms were developed using the set described in Sect. 3.2.1.1 as a development dataset (20 patients and 68 lesions) and were tested on an additional testing set that included CT examinations from 14 patients with a total of 55 lesions, out of which 35% were considered small lesions. In the development set, we obtained 1–3 marked slices per patient. In the testing set, we obtained marked slices in the entire CT volume (multislice). Fig. 3.4 shows examples of the segmentation masks provided by the radiologists. We defined a lesion as a small lesion if its longest diameter (LD) was less than 1.5 cm [25]. In our testing set we had 19 small lesions out of the 55 lesions (35%). The mean LD of all lesions in the development set was 39.2 ± 17.3 mm (7.1–55.3 mm) and 39.3 ± 28.4 mm (9.3–91.2 mm) for the lesions in the testing set. The data included various liver metastatic lesions derived from different primary cancers.

73

74

Handbook of Medical Image Computing and Computer Assisted Intervention

3.2.3.2 Comparative system performance We used the FPC and TPR detection measurements as described in Sect. 3.2.1. Note that due to the data available in our study, we extracted a 2D CC in the development set whereas for the testing set, the CC and corresponding overlap measures were in 3D. We compared the proposed system to five alternative analysis schemes: 1. A superpixel sparse dictionary learning and classification – sparsity based scheme. In this scheme, the CT input is divided into superpixels which are directly input to the superpixel dictionary learning, and classified using superpixel sparse-based classification [16]. This is a local patch based analysis method that uses the smart selection of patches based on super pixels. 2. Patch-based CNN based on Li et al. [26], which used patches of 17 × 17 pixels which are fed to the CNN for classification into lesion/normal area. A CNN model with seven hidden layers was used, which included three convolutional layers, two max-pooling layers, a fully connected layer, ReLU, and a softmax classifier. This is also a local patch based analysis with no special selection of the patches but using a CNN architecture for the final classification. 3. The U-Net scheme. Here, the proposed FCN architecture is replaced by the wellknown U-Net architecture [15] with no use of localized analysis. The U-Net includes several skip connections with additional convolution filters in the expansion path compared to the FCN-4s used in this study. This makes the U-Net more sensitive to different changes in the image in low and high scales. 4. FCN-4s. The first module described in this work with no use of localized analysis. 5. FCN-4s + Sparsity based scheme. Our proposed system, combining the FCN global analysis with a sparsity-based superpixel analysis. A 3-fold CV was used (each group containing different patients) for both the development set and the testing set using the experts’ defined masks as the gold standard. Each test fold included approximately 7 patients in the development set and 5 patients in the testing set. Average results are presented for all patients (20 in the development set and 14 in the testing set) using CV. Table 3.2 presents the results for the development and testing set. Using the development set, the sparsity-based superpixel analysis scheme on its own generated the lowest TPR, at a reasonable FPC. Comparison of the two global analysis schemes that we compare (FCN-4s and U-Net) indicated that the FCN-4s had a higher TPR with a lower FPC. Combining the global FCN-4s with the sparsity-based superpixel analysis maintained a high detection performance of more than 88%, with a reduction in the FPC. Using the testing set, a trend similar to performance on the development set was observed. The FCN-4s with the sparsity based false-positive reduction had the highest detection performance compared to other methods with a TPR of 94.6% and a

Liver lesion detection in CT using deep learning techniques

Table 3.2 Performance evaluation on the development and testing sets. Development Testing Method TPR [%] FPC TPR [%] FPC

Patch-based CNN Sparsity based U-Net FCN-4s FCN-4s + sparsity based

85.3 82.4 83.8 88.2 88.2

1.9 1.1 3.0 0.74 0.53

78.2 83.6 92.7 92.7 94.6

9.7 9.2 7.6 4.2 2.9

Table 3.3 Detection performance for small and normal sized lesions on the testing set. Method TPR [%] FPC TPR [%] TPR [%] All All Normal Small

Patch-based CNN Sparsity based U-Net FCN-4s FCN-4s + sparsity

78.2 83.6 92.7 92.7 94.6

9.7 9.2 7.6 4.2 2.9

88.9 91.7 91.7 91.7 94.4

63.2 68.4 94.7 94.7 94.7

significant improvement (P-value < 0.001) in the number of false-positives with a 2.9 FPC. Fig. 3.5 shows sample results using our method: (A) the CT input slices, (B) the output results, and (C) the results using patch based CNN, sparsity based classification, and U-Net. The results show the per-pixel labels as true-positive (TP), false-positive (FP), and false-negative (FN). Using this visualization, blobs including blue regions (true-positive) represent correctly detected lesions; blobs that only include a red region are defined as false-positives, and blobs that only include a green region are defined as false-negatives. Table 3.3 depicts the results for small sized lesions with a lesion-diameter of less than 1.5 cm and normal sized lesions with a lesion-diameter above 1.5 cm. When adding the small candidate detection, the best results were obtained using the FCN-4s with the sparsity based false-positives reduction which yielded 2.9 FPC and 94.6% TPR. A TPR of 94.4% was obtained for the normal sized lesions and 94.7% for the small lesions. The sparsity based false-positives reduction increased the TPR, which occurred in cases where two correctly detected candidates were connected by a false classification of some voxels. In these cases we obtained one connected component instead of two. The reduction of the connecting voxels added another correct detection of a lesion and increased the TPR. Analyzing the average runtime per-slice of each method, we get the following results: FCN-4s, 0.178 s; FCN-4s + sparsity, 0.489 s; U-Net, 0.578 s; Sparsity based, 2.023 s; Patch-based CNN, 5.321 s.

75

76

Handbook of Medical Image Computing and Computer Assisted Intervention

Figure 3.5 Lesion detection sample results: (A) input-CT slices (cropped); (B) detection candidates using our method; (C) detection candidates using different methods from left to right: patch-based CNN, sparsity based, and U-Net method. For each connected component the following conventions were used: false-positive pixels are in red, false-negative pixels are in green, and true-positive pixels are in blue.

In the final experiment, we tested a clinically viable scenario, in which the automated system needs to first localize the liver, following which it can detect the lesions within it. For this experiment, we used our previously developed liver segmentation network which is based on the FCN-8s architecture (for details see [14]). Following the liver segmentation output, we used our lesion detection framework to automatically detect the lesions in the liver segmentation results. We achieved a TPR of 90.9% and an FPC of 3.0 using the fully automatic algorithm. These results are comparable to the detection results achieved using the manual segmentation of the liver with a lower TPR and slightly higher FPC.

Liver lesion detection in CT using deep learning techniques

Figure 3.6 An axial CT slice (left) with its corresponding PET slice (right). Dark regions in the PET image indicate high FDG uptake.

3.3. Fully convolutional network for CT to PET synthesis to augment malignant liver lesion detection CT scans provide anatomical information. This information includes liver lesions, which need to be detected by the human expert as well as by automated systems. However, in some cases malignant tumors can have low contrast with the liver parenchyma or are small in size – and this introduces a detection challenge for the expert radiologist as well as for the automated system. Additional modalities exist which acquire information on the functional state of the body. One such modality is the Positron Emission Tomography (PET). PET images provide metabolic information; thus they show images with augmented response in a malignant tumor region. The combination of PET and CT scanners has become a standard component of diagnosis and staging in oncology [27,28]. An increased accumulation of fluoro-D-glucose (FDG) in PET relative to normal tissue is a useful marker for many cancers and can help in detection and localization of malignant lesions [28]. Additionally, PET/CT imaging is becoming an important evaluation tool for new drug therapies [29]. An example of an axial slice taken from a CT scan and its corresponding PET slice is shown in Fig. 3.6. It can be seen that the PET image resolution is lower than the CT image resolution showing less anatomical details. However, there is a malignant liver lesion that is less visible in the CT image and can be easily detected in the PET image as a large dark blob. Although PET imaging has many advantages and its use is steadily increasing, it has a few disadvantages. PET/CT entails added radiation exposure in comparison to CT-only scans. Moreover, PET/CT is relatively expensive compared to CT. Hence, it is still not offered in a large proportion of medical centers in the world. The clinical importance of PET in the management of cancer patients and, on the other hand, the difficulty in providing PET imaging as part of standard imaging raises a potential need for an alternative, less expensive, fast, and easy to use PET-like imaging. In this section we show our work on using deep learning capabilities for image synthesis to enable the generation of PET-like images, and the ability to use these images similar to the clinical PET input, to mask-out false candidates. We have explored several

77

78

Handbook of Medical Image Computing and Computer Assisted Intervention

variations on the FCN-based network, for the image synthesis [30,31]. Here, we will present a system that combines advantages of FCN and Generative Adversarial Network (GAN) based networks for the generation of PET-like images from CT volumes. The strengths of both methods are used to create realistic looking virtual PET images. We focus our attention to hepatic malignant lesions.

3.3.1 Related work Cross modality synthesis is the process in which one image modality source enables the generation of another, possibly less available modality. Several studies in the medical imaging field have shown cross-modality synthesis: Roy et al. [32] proposed an example based approach relying on sparse reconstruction from image patches for Magnetic Resonance (MR) image contrast synthesis. Similar work was proposed in [33]. Bahrami et al. [34] proposed a multilevel Canonical Correlation Analysis (CCA) method and group sparsity as a hierarchical framework to reconstruct 7T-like from 3T MR imaging (MRI). Van Nguyen et al. proposed [35] a location-sensitive deep network by integrating intensity feature from image voxels and spatial information for synthesizing MR images across domains. The deep learning field has enabled a new approach to this task, as seen by many new works that have recently come out. In [36–38] different CNN based methods and architectures were explored to learn an end-to-end nonlinear mapping from MR images to CT images. For the case of unpaired data, a CycleGAN model was used to synthesize brain CT images from brain MR by Wolterink et al. [39]. Chartsias et al. [40] demonstrated a similar concept for synthesizing cardiac MR images from CT images. In the case of PET/CT pairs, the PET study can be used to highlight malignant lesions and improve the detection compared to the use of CT data alone. In a study conducted in parallel to ours, Bi et al. [41] used a multichannel generative adversarial network to synthesize PET images from CT images with manually annotated lung tumors. Their model learns the integration from both CT and a given annotated label, to synthesize the high uptake and the anatomical background. They demonstrated that using the synthesized PET images a comparable detection performance to that achieved using the original PET data. We note that manual labeling of the tumors is needed in their work. In the current work our objective is to use information from CT data to estimate PET-like images with an emphasis on malignant lesions in the liver. The suggested system is fully automated, with no manual labeling needed. Similar to the radiologists, who have an easier time identifying malignant liver lesions in a PET/CT scan (vs only a CT scan) we want to make use of the estimated PET-like images to improve the detection of malignant lesions using an automated lesion detection software. The overview of the advanced detection methodology is depicted in Fig. 3.7, and will be described in detail below.

Liver lesion detection in CT using deep learning techniques

Figure 3.7 Overview of proposed system: combining synthesized PET image with a lesion detection software. The output of the detection software includes true-positives (in green) along with falsepositives (in red). Thresholding over the synthesized PET image to extract high-response regions (in blue) can help reduce the false-positives by intersecting the detection mask with the thresholding mask.

3.3.2 Deep learning-based virtual-PET generation The proposed framework includes two main modules: a training module which includes data preparation, and a testing module which accepts CT images as input and predicts synthesized PET images. We use an FCN to generate initial PET-like images given the input CT images. We next use a conditional GAN (cGAN) to improve and refine the FCN output. Fig. 3.8 shows a diagram of our general framework. High SUV uptake values are shown as dark regions in the output images, similar to the PET scans that are used by radiologists today. In the following subsections we describe the training data preparation, the FCN and cGAN architectures, and the proposed loss functions.

3.3.2.1 Training data preparation In the training phase a source, CT image, and a target, PET image, are input to the network. The images need to be similar in size in order to be simultaneously introduced. Hence, a preprocessing step is needed to align the lower-resolution PET images with the CT scans. The alignment step uses the given offset provided for each scan and the voxel size (in mm) ratio between both scans. CT and PET studies include a large value range. In CT imagery, Hounsfield units (HUs) are used. In PET, the standardized uptake value (SUV) is commonly used as a relative measure of FDG uptake [42] defined as SUV =

r a /w

,

(3.1)

where r is the radioactivity concentration [kBq/ml] measured by the PET scanner within a region of interest (ROI), a is the decay-corrected amount of injected radiolabeled FDG [kBq], and w is the weight of the patient [g], which is used as a surrogate for a distribution volume of tracer. It was found experimentally that contrast adjustment was a useful step in translating between the SUV and CT HU values: contrast adjustment was done by clipping extreme values and by scaling, to adjust the PET images into

79

80

Handbook of Medical Image Computing and Computer Assisted Intervention

Figure 3.8 The proposed virtual PET system.

the SUV range of 0 to 20. This range includes most of the interesting SUV values of malignant lesions. Similarly, CT image values were adjusted to be within −160 to 240 HU, which is the standard HU windowing used by the radiologists when evaluating the liver parenchyma.

3.3.2.2 The networks I. Fully convolutional network architecture The FCN architecture used is similar to that shown in Fig. 3.2. Here, the CT image serves as input and the estimated PET image as output. In this case only one CT slice was used as input, we empirically found that using adjacent slices did not improve the quality of the results. II. Conditional-GAN architecture Generative Adversarial Networks (GANs) have been highly popular in the computer vision community ever since they were introduced by Goodfellow et al. [43]. Different variations of GANs were recently proposed to generate high quality realistic natural images [44–47]. Interesting applications of GAN include generating images of one style from another style (image-to-image translation) [48] and image inpainting using GAN [49].

Liver lesion detection in CT using deep learning techniques

GANs are generative models that learn a mapping from random noise vector z to output image y. cGANs [48] learn a mapping from observed image x and random noise vector z to y. The generator G is trained to produce outputs that cannot be distinguished from “real” images by an adversely trained discriminator, D, which is trained to detect real vs. fake images. Recently, several medical imaging applications have applied the GAN framework [50–55,30,36–38]. Most studies have employed the image-to-image GAN technique to create label-to-segmentation translation, segmentation-to-image translation or medical cross modality translations, and recently GANs were used for data augmentation purposes [56]. III. Combined network The output from the FCN was found experimentally to have a satisfactory response in regions with high SUV but to be less accurate and blurry in regions with low contrast. We therefore introduced the cGAN network to refine the FCN’s output. The input to the cGAN includes two channels, one with the CT image and the second with the corresponding FCN output (simple concatenation). We adapt a similar cGAN architecture as in [48] with a few modifications. In the original cGAN the objective can be expressed as LcGAN (G, D) = Ect,pet [log D(ct, pet)] + Ect,z [log(1 − D(ct, G(ct, z))],

(3.2)

where G tries to minimize this objective against an adversarial D that tries to maximize it, ct is the CT input slice, pet is the corresponding PET slice, and z is random noise. In our objective we embed the FCN’s output (fcn) as in LModified−cGAN (G, D) = Efcn,ct,pet [log D(fcn, ct, pet)] +

Efcn,ct,z [log(1 − D(fcn, ct, G(fcn, ct, z))].

(3.3)

We tried both L1 and L2 distance measures for the generator G. No noticeable difference was observed using the different distance measures, and we chose to use the L2 distance in our experiments. The final optimization process is as follows: G∗ = arg minG maxD LModified−cGAN (G, D) + λEfcn,ct,z,pet pet − G(fcn, ct, z)2 ,

(3.4)

where G∗ is the optimal setting and λ balances the contribution of the two terms. We used a “U-Net” [15] based generator (encoder and decoder) and a 5-layer discriminator (architecture specifications can be found in [31]).

3.3.2.3 SUV-adapted loss function Malignant lesions in PET scans are usually observed with high SUV values (> 2.5) [57]. We note that most of the SUV values in PET scans are low and only a minority include

81

82

Handbook of Medical Image Computing and Computer Assisted Intervention

high SUVs. Hence, we used the SUV value in each pixel as a weight for the pixel-wise loss function, as shown in Eq. (3.5) (N is the number of samples). With this additional term in the loss function, the network focuses on high SUV valued pixels, even though most pixels include lower values: L=

N 1  IPET (i)(SynPET (i) − IPET (i))2 . N i=1

(3.5)

While this approach helped the FCN learn the malignant tumor appearance and provide a better response in the synthesized PET images, it did not help when training the cGAN. Hence, we modified the loss function by computing the weighted average reconstruction loss (Eq. (3.5)) for high SUVs (> 2.5) and for low SUVs (≤ 2.5) as in LcGAN = LlowSUV + LhighSUV .

(3.6)

This way the cGAN training was able to achieve better response in regions with high SUV while not substantially reducing the quality of reconstruction in other regions.

3.3.3 Experiments and results To evaluate the performance of the system, we conducted several sets of experiments. A development set was used for training and validation. The final results are presented for an independent testing set. We demonstrate the applicability of the system to augment performance of a liver lesion detection system.

3.3.3.1 Dataset The data used in this work includes PET/CT scans (pairs of contrast-enhanced portal phase CT scans with their corresponding PET scans) from the Sheba Medical Center, obtained from 2014 to 2015. An institutional review board (IRB) approval was granted for this retrospective study. The dataset contains 60 CT (with 0.97 mm pixel spacing and 4 mm slice thickness) and PET (with 3 mm pixel spacing and 4 mm slice thickness) pairs, from 60 different patients. We constrained the analysis to slices in the region of the liver for our study. Not all PET/CT scans in our dataset included liver lesions. The data was collected in two phases. In the first phase the collected data included PET/CT scans that were used for the development and validation of the explored methods. In the second phase new data was collected for testing with no additional modifications of the algorithms. The training set included 23 PET/CT pairs (6 with malignant liver lesions) and the testing was performed on 37 pairs (9 with malignant liver lesions).

3.3.3.2 Experimental setting The networks were implemented and trained using Keras framework [58] on a PC with a single Nvidia 1080 Ti GPU. The following hyperparameters were used for all

Liver lesion detection in CT using deep learning techniques

Figure 3.9 Sample results of the predicted PET using our method compared to the real PET with the corresponding CT images.

networks: the learning rate of 0.00001 with a batch size of 4. Adam optimizer [59] was used with β = 0.5. For the cGAN, we used λ = 20 in the optimization process presented in Eq. (3.4). To make the network model more robust to variability in location and scaling, online data transformations were performed with uniform sampling of scale [−0.9, 1.1] and translations [−25, 25] in each epoch. We randomly chose 20% of the training images for validation which were used to optimize the training process. The proposed method reconstruction performance was compared with several method variations and was demonstrated to most-strongly emphasize malignant lesions for the detection task [31]. Qualitative results are shown in Fig. 3.9. It compares the method’s virtual PET images with the original PET study images. The virtual PET provided a very similar response to the real PET in the presented cases. The left column includes one malignant lesion, the second column includes three liver metastases (malignant lesions), and the right column includes two cysts (benign lesions).

3.3.3.3 Liver lesion detection using the virtual-PET In the following experiment, we used the synthesized (virtual) PET images as an additional false-positive reduction layer for an existing lesion detection software. Since high SUV values can be indicative of malignant lesions, thresholding the PET using a high SUV threshold (th = 2.5) can reduce irrelevant regions for lesion detection. We suggest the following scheme for improving a given lesion detection software: Given an input CT scan, the detection software outputs lesion candidates as a binary mask. This mask may include false detections. Give the same CT scan, a virtual PET image is generated. It is next thresholded, as described above. The two generated binary masks are next intersected. This step results in the removal of false detections. A block

83

84

Handbook of Medical Image Computing and Computer Assisted Intervention

Table 3.4 Detection measurements using the FCN and sparsity based detection software presented in Sect. 3.2 with and without SUV thresholding on the synthesized PET. In bold we show the results obtained using the proposed method including the FCN and cGAN. Method TPR [%] Average FPC Detection soft. 94.6 2.9 ± 2.1 Detection soft. & virtual PET (FCN) 90.9 2.2 ± 1.7 Detection soft. & virtual PET (FCN+cGAN) 94.6 2.1 ± 1.7

Figure 3.10 Sample results using the FCN and sparsity based detection software presented in Sect. 3.2: correctly detected lesions (green); false-positives (red) that were removed by combining the virtual PET (FCN+cGAN) using SUV thresholding.

diagram of the proposed framework is shown in Fig. 3.7. This, rather naïve approach, shows the clinical relevance of using the virtual PET to improve existing software. We use the same lesion detection system that was presented in Sect. 3.2 [17]. The test set includes 14 CT scans with 55 lesions, as presented in Sect. 3.2.3.1. Two evaluation measurements were computed, the TPR and the FPC for each case, as in Sect. 3.2.1. Table 3.4 shows the performance with and without the proposed false-positive reduction layer. In this scenario, results were constrained to the manually annotated liver. Using the proposed method, the average FPC decreased from 2.9 to 2.1; an improvement of 28% (with P-value < 0.05) with a similar TPR. Fig. 3.10 shows examples of cases with false-positives that were removed (in red) by combining the proposed virtual PET method with the detection software presented in Sect. 3.2. As a final experiment, we tested a fully automatic framework with an automatic liver segmentation scheme, as in [17]. Thus, we use an automatic liver segmentation instead of the manually circumscribed liver. Using our method the average FPC decreased from 3.0 to 2.3; an improvement of 23% (with P-value < 0.05) with a slight, not significant decrease of the TPR from 90.9% to 89.1%.

3.4. Discussion and conclusions In this chapter we introduced the liver lesion detection task and presented two deep learning based methods as a clinical relevant solution. In Sect. 3.2 we presented a

Liver lesion detection in CT using deep learning techniques

computer-aided detection system to detect liver lesions in CT images using an FCN with a sparsity based online fine-tuning for false-positive reduction. The FCN-4s architecture was shown to successfully detect lesions in both the development as well as the testing sets, as shown in Table 3.2. The sparsity based classification scheme was shown to significantly reduce the FPC. We note a difference in the range of the results in the two sets. This is due to the difference in the data settings: The data we obtained in the testing set had multi-slice lesion segmentation masks with only 1–2 slices marked in the development set. Given the additional data per patient, most lesions appeared on several slices in the test set, and thus had a higher likelihood of being detected. We therefore expected an overall increase in the TPR as well as in the FPC. Both these trends were present in the results. The experimental results listed in Table 3.3 demonstrate that the FCN-4s method with the sparsity based fine-tuning is robust to lesion size, detecting 94.7% of the small lesions. Fig. 3.5 shows the detection results for several patients with different lesion sizes. Although all the lesions were detected, in some cases the overlap with the experts’ segmentation mask was smaller than in others. However, as we discuss shortly, we believe that the output of the detection system can serve as an initial lesion segmentation mask. We compared our FCN architecture to the U-Net architecture in Table 3.2. In both the development and testing sets the U-Net had a higher FPC than the FCN-4s. Thus, whereas the U-Net architecture exhibited superior segmentation performance in several medical applications, in the detection task the FCN-4s showed superiority. Most of the experiments in our study were designed for detection performance evaluation, and concentrated on a manually circumscribed liver region. As a final experiment, we evaluated the proposed method using an automated liver segmentation tool. In our initial experiment, we achieved clinically relevant results of 90.9% TPR with 3.0 FPC. In this work we focused on lesion detection. To further enhance the system to provide accurate segmentation, these detection results can be used as an initial mask for the lesions, following which additional segmentation algorithms can be implemented for expansion and refinement. Note that no significant pre- or postprocessing was implemented in the suggested method. Adding these steps could increase lesion detection accuracy and enable more accurate segmentation as well. The FCN-4s in our system uses 3 adjacent slices as input. In examinations with large slice spacing, the images were interpolated to have a fixed slice spacing of 1 mm. For very small lesions, this means that a lesion will be present on a single slice alone which results in a lower TPR. It is clear that higher resolution in acquisitions will increase the overall detection rate. Moreover, using the automatic liver segmentation, a slight decrease in TPR was observed. A more accurate liver segmentation could improve the results of our fully automatic algorithm. The data augmentation in our training only included scale transformations. Additional augmentation techniques such as gray level transformations and deformations could make the system more robust to these changes.

85

86

Handbook of Medical Image Computing and Computer Assisted Intervention

In Sect. 3.3, a novel system for PET synthesis using only CT scans was presented. The system includes an FCN model and a cGAN model that refines the synthesized output extracted from the FCN. The proposed methodology is an extension to an earlier work [30], in which we used a pyramid based image blending step to combine the advantages of an FCN and a cGAN network. The proposed framework was shown to provide realistic PET estimation with special attention to malignant lesions using a custom loss function for each model. Not using an image blending step provided savings in time and reduced the need for manually defining a threshold for the blending mask, while improving the system performance. We wanted to see if the proposed method can be used to improve the proposed automatic liver lesion detection software. Our system was easily integrated into the lesion detection software. Using a pathological SUV threshold of 2.5 we achieved a decrease in false-positive from an average of 2.9 per case to 2.1 (28% improvement). This experiment shows the benefit of using the proposed system to improve a given liver lesion analysis software. Since this method was trained on a dataset that was not seen by the existing detection software it improved its results. However, no manual labeling was conducted in this experiment since our method uses only PET/CT pairs for training. One possible application could be to use the virtual PET to improve lesion segmentation. However, the PET images are quite blurry and so is the virtual PET, making it hard to assess the segmentation process. Hence, we believe that detection approaches are more relevant for this method. We used a rather naïve approach by thresholding the virtual PET to reduce the amount of false-positives per case in an existing lesion detection software. However, the proposed system can be easily integrated into the training process of different networks for different tasks such as detection (as shown here) and classification. Our proposed framework with the FCN–cGAN combination and the custom loss function has shown promising results in terms of reconstruction measures as well as detection measures by integrating it with the lesion detection software presented in Sect. 3.2. A major strength of this framework is that no manual labeling was used to train the system to synthesize PET images. As we well know, the task of manually labeling and annotating medical data is hard, and contributes to the usually small data sets that are used in todays’ medical imaging research. Each year millions of PET/CT studies are conducted worldwide, and utilizing the current method, the CT and PET pairing can be used as “free” labeled and annotated data, with potential for big data, approaching millions of studies. This work can be expanded by obtaining a larger dataset with vast experiments using the entire CT and not just the liver region as well as integrating it into the training process of deep learning based detection and classification networks. The presented system can be used for many applications in which PET examination is needed such as evaluation of drug therapies and detection of malignant lesions.

Liver lesion detection in CT using deep learning techniques

There are additional possible directions for future work. In order to transition the work to a clinically-viable tool, large scale validation should be conducted in a real clinical environment. Preliminary steps in this direction have already been initiated by deploying stand-alone executables for each system presented in this chapter. The integration of our automated analysis tools with user interface applications for radiologists’ decision support systems will enable us to conduct clinical studies at the Sheba Medical Center, get feedback, and further improve the current systems. To conclude, the proposed methods aim to provide a solution to several key areas of concern in healthcare today: (i) Data overload: Advances in technology have resulted in the availability of higher resolution scanners that generate large amounts of imagery which are now available to the physician. However, no tools have been developed to support the analysis of these data, and thus much of the data available is not used efficiently or correctly; (ii) Subjective medical analysis and cross-physician variability: The analysis of image data is highly subjective and based on the experience of the expert. There is a general consensus that two physicians can differ considerably in their diagnosis; (iii) Low-resource countries lack basic healthcare: there is a lack of advanced imaging systems as well as experts. The tools presented in this chapter can contribute to the ongoing improvement of healthcare by responding to the issues mentioned above, improving physicians’ workflow and decision making process while increasing objectivity in the process, enhancing accuracy and saving time.

Acknowledgments We thank Dr. Eyal Klang, Prof. Michal M. Amitai, Prof. Simona Ben-Haim, Prof. Eli Konen, and Dr. Stephen P. Raskin from the Sheba Medical Center for providing the clinical knowledge and guidance as well as relevant data used in this study. This research was supported by the Israel Science Foundation (grant No. 1918/16).

References [1] A. Sarma, M.E. Heilbrun, K.E. Conner, S.M. Stevens, S.C. Woller, C.G. Elliott, Radiation and chest ct scan examinations: what do we know? Chest 142 (3) (2012) 750–760. [2] H.M. Taylor, P.R. Ros, Hepatic imaging: an overview, Radiologic Clinics 36 (2) (1998) 237–245. [3] K.D. Hopper, K. Singapuri, A. Finkel, Body CT and oncologic imaging 1, Radiology 215 (1) (2000) 27–40. [4] W.H. Organization, Fact sheet n 297, February 2006, available online at: http://www.who.int/cancer/ en/index.html. [5] G.T. Sica, H. Ji, P.R. Ros, CT and MR imaging of hepatic metastases, American Journal of Roentgenology 174 (3) (2000) 691–698. [6] X. Deng, G. Du, Editorial: 3D segmentation in the clinic: a grand challenge II-liver tumor segmentation, in: International Conference on Medical Image Computing and Computer-Assisted Intervention 2008, Springer, 2008.

87

88

Handbook of Medical Image Computing and Computer Assisted Intervention

[7] P.F. Christ, LiTS – Liver Tumor Segmentation Challenge, URL https://competitions.codalab.org/ competitions/15595, 2017. [8] L. Ruskó, Á. Perényi, Automated liver lesion detection in CT images based on multi-level geometric features, International Journal of Computer Assisted Radiology and Surgery 9 (4) (2014) 577–593, https://doi.org/10.1007/s11548-013-0949-9. [9] M. Frid-Adar, I. Diamant, E. Klang, M. Amitai, J. Goldberger, H. Greenspan, Modeling the intra-class variability for liver lesion detection using a multi-class patch-based CNN, in: International Workshop on Patch-Based Techniques in Medical Imaging, Springer, 2017, pp. 129–137. [10] J. Long, E. Shelhamer, T. Darrell, Fully convolutional networks for semantic segmentation, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2015, pp. 3431–3440. [11] P.F. Christ, M.E.A. Elshaer, F. Ettlinger, S. Tatavarty, et al., Automatic liver and lesion segmentation in ct using cascaded fully convolutional neural networks and 3D conditional random fields, in: International Conference on Medical Image Computing and Computer-Assisted Intervention, Springer, 2016, pp. 415–423. [12] Q. Dou, H. Chen, Y. Jin, L. Yu, J. Qin, P.-A. Heng, 3D deeply supervised network for automatic liver segmentation from CT volumes, in: International Conference on Medical Image Computing and Computer-Assisted Intervention, Springer, 2016, pp. 149–157. [13] X. Li, H. Chen, X. Qi, Q. Dou, C.-W. Fu, P.A. Heng, H-DenseUNet: hybrid densely connected UNet for liver and liver tumor segmentation from CT volumes, preprint, arXiv:1709.07330. [14] A. Ben-Cohen, I. Diamant, E. Klang, M. Amitai, H. Greenspan, Fully convolutional network for liver segmentation and lesions detection, in: Deep Learning and Data Labeling for Medical Applications, Springer, 2016, pp. 77–85. [15] O. Ronneberger, P. Fischer, T. Brox, U-Net: convolutional networks for biomedical image segmentation, in: International Conference on Medical Image Computing and Computer-Assisted Intervention, Springer, 2015, pp. 234–241. [16] A. Ben-Cohen, E. Klang, M. Amitai, H. Greenspan, Sparsity-based liver metastases detection using learned dictionaries, in: 2016 IEEE 13th International Symposium on Biomedical Imaging (ISBI), IEEE, 2016, pp. 1195–1198. [17] A. Ben-Cohen, E. Klang, A. Kerpel, E. Konen, M.M. Amitai, H. Greenspan, Fully convolutional network and sparsity-based dictionary learning for liver lesion detection in CT examinations, Neurocomputing 275 (2017) 1585–1594. [18] A. Ben-Cohen, I. Diamant, E. Klang, M. Amitai, H. Greenspan, Fully convolutional network for liver segmentation and lesions detection, in: International Workshop on Large-Scale Annotation of Biomedical Data and Expert Label Synthesis, Springer, 2016, pp. 77–85. [19] K. Simonyan, A. Zisserman, Very deep convolutional networks for large-scale image recognition, preprint, arXiv:1409.1556. [20] L. Lu, P. Devarakota, S. Vikal, D. Wu, Y. Zheng, M. Wolf, Computer aided diagnosis using multilevel image features on large-scale evaluation, in: International MICCAI Workshop on Medical Computer Vision, Springer, 2013, pp. 161–174. [21] H.R. Roth, L. Lu, J. Liu, J. Yao, A. Seff, K. Cherry, R.M. Summers, Improving computer-aided detection using convolutional neural networks and random view aggregation, IEEE Transactions on Medical Imaging 35 (5) (2016) 1170–1181. [22] A.A.A. Setio, F. Ciompi, G. Litjens, P. Gerke, C. Jacobs, S.J. van Riel, M.M.W. Wille, M. Naqibullah, C.I. Sánchez, B. van Ginneken, Pulmonary nodule detection in CT images: false positive reduction using multi-view convolutional networks, IEEE Transactions on Medical Imaging 35 (5) (2016) 1160–1169. [23] R. Achanta, A. Shaji, K. Smith, A. Lucchi, P. Fua, S. Süsstrunk, SLIC superpixels compared to state-of-the-art superpixel methods, IEEE Transactions on Pattern Analysis and Machine Intelligence 34 (11) (2012) 2274–2282, https://doi.org/10.1109/TPAMI.2012.120.

Liver lesion detection in CT using deep learning techniques

[24] Z. Jiang, Z. Lin, L.S. Davis, Learning a discriminative dictionary for sparse coding via label consistent k-SVD, in: 2011 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), IEEE, 2011, pp. 1697–1704. [25] H.I. Khalil, S.A. Patterson, D.M. Panicek, Hepatic lesions deemed too small to characterize at CT: prevalence and importance in women with breast cancer, Radiology 235 (3) (2005) 872–878, https:// doi.org/10.1148/radiol.2353041099. [26] W. Li, F. Jia, Q. Hu, Automatic segmentation of liver tumor in CT images with deep convolutional neural networks, Journal of Computer and Communications 3 (11) (2015) 146, https://doi.org/10. 4236/jcc.2015.311023. [27] W.A. Weber, Assessing tumor response to therapy, Journal of Nuclear Medicine 50 (Suppl. 1) (2009) 1S–10S. [28] G.J. Kelloff, J.M. Hoffman, B. Johnson, H.I. Scher, B.A. Siegel, E.Y. Cheng, B.D. Cheson, J. O’Shaughnessy, K.Z. Guyton, D.A. Mankoff, et al., Progress and promise of FDG-PET imaging for cancer patient management and oncologic drug development, Clinical Cancer Research 11 (8) (2005) 2785–2808. [29] W.A. Weber, A.L. Grosu, J. Czernin, Technology insight: advances in molecular imaging and an appraisal of PET/CT scanning, Nature Reviews. Clinical Oncology 5 (3) (2008) 160. [30] A. Ben-Cohen, E. Klang, S.P. Raskin, M.M. Amitai, H. Greenspan, Virtual PET images from CT data using deep convolutional networks: initial results, in: International Workshop on Simulation and Synthesis in Medical Imaging, Springer, 2017, pp. 49–57. [31] A. Ben-Cohen, E. Klang, S.P. Raskin, S. Soffer, S. Ben-Haim, E. Konen, M.M. Amitai, H. Greenspan, Cross-modality synthesis from CT to PET using FCN and GAN networks for improved automated lesion detection, Engineering Applications of Artificial Intelligence 78 (2019) 186–194. [32] S. Roy, A. Carass, J.L. Prince, Magnetic resonance image example-based contrast synthesis, IEEE Transactions on Medical Imaging 32 (12) (2013) 2348–2363. [33] J.E. Iglesias, E. Konukoglu, D. Zikic, B. Glocker, K. Van Leemput, B. Fischl, Is synthesizing MRI contrast useful for inter-modality analysis? in: International Conference on Medical Image Computing and Computer-Assisted Intervention, Springer, 2013, pp. 631–638. [34] K. Bahrami, F. Shi, X. Zong, H.W. Shin, H. An, D. Shen, Hierarchical reconstruction of 7T-like images from 3T MRI using multi-level CCA and group sparsity, in: International Conference on Medical Image Computing and Computer-Assisted Intervention, Springer, 2015, pp. 659–666. [35] H. Van Nguyen, K. Zhou, R. Vemulapalli, Cross-domain synthesis of medical images using efficient location-sensitive deep network, in: International Conference on Medical Image Computing and Computer-Assisted Intervention, Springer, 2015, pp. 677–684. [36] D. Nie, X. Cao, Y. Gao, L. Wang, D. Shen, Estimating CT image from MRI data using 3D fully convolutional networks, in: Deep Learning and Data Labeling for Medical Applications, Springer, 2016, pp. 170–178. [37] X. Han, MR-based synthetic CT generation using a deep convolutional neural network method, Medical Physics 44 (4) (2017) 1408–1419. [38] L. Xiang, Q. Wang, D. Nie, Y. Qiao, D. Shen, Deep embedding convolutional neural network for synthesizing CT image from T1-weighted MR image, preprint, arXiv:1709.02073. [39] J.M. Wolterink, A.M. Dinkla, M.H. Savenije, P.R. Seevinck, C.A. van den Berg, I. Išgum, Deep MR to CT synthesis using unpaired data, in: International Workshop on Simulation and Synthesis in Medical Imaging, Springer, 2017, pp. 14–23. [40] A. Chartsias, T. Joyce, R. Dharmakumar, S.A. Tsaftaris, Adversarial image synthesis for unpaired multi-modal cardiac data, in: International Workshop on Simulation and Synthesis in Medical Imaging, Springer, 2017, pp. 3–13. [41] L. Bi, J. Kim, A. Kumar, D. Feng, M. Fulham, Synthesis of positron emission tomography (PET) images via multi-channel generative adversarial networks (GANs), in: Molecular Imaging, Reconstruction and Analysis of Moving Body Organs, and Stroke Imaging and Treatment, Springer, 2017, pp. 43–51.

89

90

Handbook of Medical Image Computing and Computer Assisted Intervention

[42] K. Higashi, A.C. Clavo, R.L. Wahl, Does FDG uptake measure proliferative activity of human cancer cells? In vitro comparison with DNA flow cytometry and tritiated thymidine uptake, Journal of Nuclear Medicine 34 (3) (1993) 414–419. [43] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, Y. Bengio, Generative adversarial nets, in: Advances in Neural Information Processing Systems, 2014, pp. 2672–2680. [44] A. Radford, L. Metz, S. Chintala, Unsupervised representation learning with deep convolutional generative adversarial networks, preprint, arXiv:1511.06434. [45] E.L. Denton, S. Chintala, R. Fergus, et al., Deep generative image models using a Laplacian pyramid of adversarial networks, in: Advances in Neural Information Processing Systems, 2015, pp. 1486–1494. [46] M. Mirza, S. Osindero, Conditional generative adversarial nets, preprint, arXiv:1411.1784. [47] A. Odena, C. Olah, J. Shlens, Conditional image synthesis with auxiliary classifier GANs, preprint, arXiv:1610.09585. [48] P. Isola, J.-Y. Zhu, T. Zhou, A.A. Efros, Image-to-image translation with conditional adversarial networks, arXiv preprint. [49] R. Yeh, C. Chen, T.Y. Lim, M. Hasegawa-Johnson, M.N. Do, Semantic image inpainting with perceptual and contextual losses, preprint, arXiv:1607.07539. [50] P. Costa, A. Galdran, M.I. Meyer, M.D. Abràmoff, M. Niemeijer, A.M. Mendonça, A. Campilho, Towards adversarial retinal image synthesis, preprint, arXiv:1701.08974. [51] W. Dai, J. Doyle, X. Liang, H. Zhang, N. Dong, Y. Li, E.P. Xing, SCAN: structure correcting adversarial network for chest X-rays organ segmentation, preprint, arXiv:1703.08770. [52] Y. Xue, T. Xu, H. Zhang, R. Long, X. Huang Segan, Adversarial network with multi-scale l1 loss for medical image segmentation, preprint, arXiv:1706.01805. [53] D. Nie, R. Trullo, C. Petitjean, S. Ruan, D. Shen, Medical image synthesis with context-aware generative adversarial networks, preprint, arXiv:1612.05362. [54] T. Schlegl, P. Seeböck, S.M. Waldstein, U. Schmidt-Erfurth, G. Langs, Unsupervised anomaly detection with generative adversarial networks to guide marker discovery, in: International Conference on Information Processing in Medical Imaging, Springer, 2017, pp. 146–157. [55] V. Alex, M.S. KP, S.S. Chennamsetty, G. Krishnamurthi, Generative adversarial networks for brain lesion detection, in: SPIE Medical Imaging, International Society for Optics and Photonics, 2017, p. 101330G. [56] M. Frid-Adar, E. Klang, M. Amitai, J. Goldberger, H. Greenspan, Synthetic data augmentation using GAN for improved liver lesion classification, preprint, arXiv:1801.02385. [57] L. Kostakoglu, H. Agress Jr, S.J. Goldsmith, Clinical role of FDG pet in evaluation of cancer patients, Radiographics 23 (2) (2003) 315–340. [58] F. Chollet, et al., Keras, 2015. [59] D.P. Kingma, J. Ba Adam, A method for stochastic optimization, preprint, arXiv:1412.6980.