- Email: [email protected]
A method to improve HEVC lossless coding of volumetric medical images
Signal Processing: Image Communication (xxxx) xxxx–xxxx
Contents lists available at ScienceDirect
Signal Processing: Image Communication journal homepage: www.elsevier.com/locate/image
A method to improve HEVC lossless coding of volumetric medical images ⁎
André F.R. Guardaa,b, , João M. Santosa,b, Luís A. da Silva Cruza,c, Pedro A.A. Assunçãoa,b,1, Nuno M.M. Rodriguesa,b, Sérgio M.M. de Fariaa,b a b c
Instituto de Telecomunicações, Portugal Instituto Politécnico de Leiria, ESTG, Portugal Universidade de Coimbra, DEEC, Portugal
A R T I C L E I N F O
A B S T R A C T
Keywords: HEVC LSP Lossless compression Medical imaging
Medical imaging technology and applications are continuously evolving, dealing with images of increasing spatial and temporal resolutions, which enable more accurate processing, feature analysis and medical diagnosis. However, the increase in resolution also requires a growing amount of data to be stored, processed and exchanged or transmitted through networks. Despite the high coding efficiency achieved by the most recent image and video coding standards in lossy compression, they are not well suited for quality-critical medical image compression where either near-lossless or lossless coding is required. In this paper we propose a method to improve lossless coding of volumetric medical images, such as Magnetic Resonance and Computed Tomography, and medical sequences such as X-ray Angiography images, using the latest standard High Efficiency Video Encoder (HEVC). New pixel-wise prediction techniques are proposed to extend the current HEVC lossless tools, based on Least-Squares Prediction (LSP). Experimental results show a bitrate reduction of over 44%, when compared to DICOM recommended encoders, and 13.8% when compared to standard lossless HEVC, for 8 bpp volumetric images, and over 8% and 4.6%, respectively, for volumetric images using more than 8 bpp.
1. Introduction
compression techniques in this type of applications. Nevertheless, some regulatory bodies, such as in the UK, USA and Canada, allow the use of lossy compression, with this choice being the responsibility of the institutions and radiologists [3]. The preference for lossless techniques is acknowledged by the Digital Imaging and Communications in Medicine (DICOM) [4], which recommends the use of lossless encoders such as JPEG in lossless mode, RLE (run-length encoding), TIFF PackBits, JPEG-LS [5], lossless JPEG2000 [6], H.264/AVC [7], as well as lossy JPEG and JPEG2000 with near-lossless compression ratios for this purpose. JPEG2000 is a wavelet transform-based image encoder, which allows quality and spatial scalability, followed by the entropy encoder Embedded Block Coding with Optimized Truncation (EBCOT), while JPEG-LS uses the median edge detector (MED) predictor, followed by context modelling of the prediction error and entropy encoding, using Golomb codes. MED is a simple edge detector that uses three raster-scan order causal pixels to determine if a vertical or horizontal edge exists, and then selects one of three predictors accordingly. Other state-of-theart lossless image encoders are context-based adaptive lossless image codec (CALIC) [8] and minimum rate predictors (MRP) [9]. CALIC uses a gradient adaptive predictor (GAP), in which six causal pixels are used
Recent advances in digital imaging and video, as well as rapid evolution of acquisition and processing systems using increasingly higher resolutions, require efficient compression formats for file exchange, storage and visual communications over networks. High Efficiency Video Coding (HEVC) is the most recent standard to fulfil the challenging requirements imposed by new services and applications [1], but in the case of medical imaging systems, with the growing use of telemedicine and information sharing among the medical community, efficient image compression assumes a different meaning due to more challenging requirements [2]. Medical imaging systems are following the same trend of higher spatial and temporal resolutions and bit depths. Therefore, a simple medical examination, such as Magnetic Resonance (MR) or Computed Tomography (CT), generates a very large amount of data. Typically these data must be stored for several years, with resulting high costs in creating and maintaining large databases of medical images. Since the use of lossy compression algorithms can result in the loss of important details in medical images, which can lead to more difficult analysis or even wrong diagnosis, it is recommended to use lossless or near-lossless
⁎
Corresponding author at: Instituto de Telecomunicações, Portugal. E-mail addresses: [email protected] (A.F.R. Guarda), [email protected] (J.M. Santos), [email protected] (L.A. da Silva Cruz), [email protected] (P.A.A. Assunção), [email protected] (N.M.M. Rodrigues), [email protected] (S.M.M. de Faria). http://dx.doi.org/10.1016/j.image.2017.02.002 Received 31 January 2016; Received in revised form 1 February 2017; Accepted 2 February 2017 0923-5965/ © 2017 Elsevier B.V. All rights reserved.
Please cite this article as: Guarda, A.F.R., Signal Processing: Image Communication (2017), http://dx.doi.org/10.1016/j.image.2017.02.002
Signal Processing: Image Communication (xxxx) xxxx–xxxx
A.F.R. Guarda et al.
to estimate the local gradients and determine the prediction value, and an adaptive arithmetic encoder for the prediction residue. As for MRP, it uses several linear predictors adapted to each image, by dividing the image in blocks of variable size and sorting them into classes, with each class using different coefficients. Multiple optimizations are performed in order to minimize the cost function that represents the prediction error data, which is encoded with a Huffman or a range coder. Volumetric medical images, such as MRI or CT scans, comprise a set of static images, each one representing a slice of a relevant volume for medical purposes. Although each slice can be independently encoded using a standard image encoder, such option does not exploit the interslice redundancy present in volumetric image sets. In order to address this issue, an extension of the JPEG2000 encoder was developed for volumetric images, known as JP3D [10], which uses 3D blocks, a 3D discrete wavelet transform and also a 3D EBCOT, instead of their 2D counterparts. Another common technique to encode volumetric medical images is to arrange the slices as a pseudo-video sequence, where each slice corresponds to a video frame. Such data structure allows the use of state-of-the-art video encoders like HEVC [1], in order to take advantage of the inter-frame prediction, by exploiting inter-slice correlation. An alternative is to use the extensions of CALIC, proposed for compression of multi-spectral and hyper-spectral data, 3D CALIC [11] and M-CALIC [12], respectively, considering each slice of the volume as a different band. In these extensions, an inter-band prediction technique is introduced, in which a linear combination of pixels in the reference band is used to predict a pixel in the current band, based on the gradients. 3D CALIC switches between intra-band prediction with GAP and inter-band prediction, according to the correlation between bands in the current region, while M-CALIC always uses a modified inter-band prediction with the previous two bands being used as references. In this paper, we propose a method to enhance the HEVC lossless coding efficiency of volumetric medical images, by improving the LeastSquares Prediction (LSP) method introduced in [13]. LSP has been successfully used in previous works for single and stereo image compression, due to its ability to efficiently representing edges orientations [14–16]. Experimental results show that the proposed HEVC prediction mode results in higher lossless compression efficiency for encoding volumetric medical images, achieving an average bitrate reduction of approximately 21% over standard HEVC and up to 49% over DICOM recommended coding methods. This paper is organized as follows: Section 2 introduces relevant background information about medical imaging technologies and data sets used in this work. Section 3 presents an overview of lossless encoding with HEVC, Section 4 describes the LSP method, and Section 5 details the proposed HEVC lossless coding mode based on LSP. In Section 6 the experimental results are presented and discussed while Section 7 concludes the paper.
Fig. 1. Sagittal slices of the brain by different imaging modalities. From top-left to downright: Magnetic Resonance Imaging, Computed Tomography, Positron Emission Tomography and Ultrasound [19].
modalities are also dependent on the constraints of each specific application domain and also on the associated medical tasks. For instance, X-rays are more adequate to represent bone structures, while Magnetic Resonance provide higher definition and works better for softer tissues. Four image datasets were used in this work to evaluate the performance of the proposed coding scheme. The first one is composed of eight volumetric medical images: four CT and four MR scans, all available from [20], the image and video repository of the Center for Image Processing (CIPR) of the Rensselaer Polytechnic Institute. The volumetric data comprise sets of spatially adjacent slices. Since these slices represent cuts through anatomical organs which extend into the three dimensions, typically, neighbouring pixels in adjacent slices are highly correlated. The second set consists of seven X-ray Angiography images with multiple slices or frames taken over time, which were provided by [21]. Similarly to video frames, these pseudo-video sequences also present high correlation between slices. However, most of the encoding methods currently recommended by DICOM do not exploit this correlation to achieve higher compression ratios. Therefore it is quite opportune to investigate new coding techniques, capable of taking advantage of the inter-slice correlation characteristics of these images, to increase the compression efficiency. The third and fourth datasets present two more CT scans, which are available at the National Biomedical Imaging Archive (NBIA) [22], and two more MR scans from [23], made available by the Center for In Vivo Microscopy (CIVM), respectively, with different spatial resolutions and bit depths. The spatial resolution, bit depth and number of slices of each volumetric set (or pseudo-video sequence) are presented in Table 1. Figs. 2, 3 and 4 show the midpoint slice of each volume (or frame) of all datasets.
2. Medical imaging technologies Recent advances in understanding and exploiting several physical phenomena, such as X-ray, γ-ray, ultrasound waves and positron emission production, propagation and recording are strongly linked to the progress in different fields of medical imaging technologies. The availability of digital acquisition technologies and powerful computational resources are also driving new developments and useful solutions e.g., tomography reconstruction both static and time-varying. There are several types of medical imaging modalities, for example Computed Tomography (CT) [17], Magnetic Resonance Imaging (MRI) [18], ultrasound imaging, Positron Emission Tomography, Elastography, each with its own characteristics [19], that depend on the associated physical processes and acquisition methods. Fig. 1 shows four sagittal slices of the brain acquired with different technologies, where the different characteristics of the various imaging modalities are clearly observable. The characteristics of specific medical imaging
3. Lossless encoding using HEVC HEVC is the current state-of-the-art compression standard for lossy video compression, which also includes a lossless encoding profile. The lossless encoding process differs from the lossy one in several ways. In the lossless encoding mode the transform, quantization and in-loop filters are skipped, and the resulting residue from intra and inter prediction is entropically encoded. Since the crucial steps of residue transformation and quantization are not used, the compression gain is small and the efficiency of prediction methods becomes of utmost importance for the overall encoding performance. 2
Signal Processing: Image Communication (xxxx) xxxx–xxxx
A.F.R. Guarda et al.
outside the current prediction unit (PU), i.e., without excluding those in the current PU itself. This is equivalent to perform Differential PulseCode Modulation (DPCM) in these directions, so the resulting residue of SAP is similar to that obtained from DPCM. To provide insight about the relative performance between the DICOM recommended encoders such as JPEG2000, JPEG-LS and H.264/AVC (using the Intra profile and IPPP coding structure [26]), and HEVC (using Intra, Random Access and Random Access RExt profiles), Table 2 shows the average lossless coding efficiency obtained for the first set of volumetric medical images described in Section 2. It can be seen that JPEG2000 and JPEG-LS present better results than the Intra profiles of H.264/AVC and HEVC. However, when using interframe prediction to exploit the redundancy between slices, greater compression gain is achieved, as expected. The RExt profile presents the best results due to the use of SAP, with an average bitrate reduction of 8.9% over HEVC Random Access. It is clear that HEVC compression performance surpasses that of JPEG2000, JPEG-LS and H.264/AVC. This advantage can be explained by the use of HEVC coding tools which exploit the inter-slice redundancy in these type of images, whereas such tools are not available in JPEG2000 and JPEG-LS. This issue is discussed with more detail in Section 6. Several other techniques for lossless prediction have been proposed to replace or improve the SAP predictor of HEVC, most of them aiming at screen content coding. The Sample-based Weighted Prediction (SWP) proposed in [27] performs a weighted average of neighbouring pixels surrounding the current pixel to form its prediction. An alternative to SWP is the Directional Template Matching (DTM) [28], which instead of averaging the neighbouring pixels, uses the one with the highest similarity, achieving better reconstruction of sharp edges. The authors also found that using a combination of SWP and DTM provides better prediction performance. Another method recently proposed is the Sample-based Angular intra-Prediction with Gradient-based edge prediction (SAP-G) [29], which uses the surrounding pixels to compute the gradients in four directions (0°, 45°, 90° and 135°), and then selects the adjacent neighbouring pixel in the direction of the smallest gradient as the prediction for the current pixel. In [30], the Sample-based Angular intra-Prediction with Median and Edge (SAP-ME) was proposed, in which the MED predictor of JPEG-LS is applied in order to improve the prediction of sharp edges, as well as a median predictor to improve prediction in smooth regions.
Table 1 Medical datasets' information. Sequence
Resolution
Bit Depth
No. Slices
CT_Aperts CT_carotid CT_skull CT_wrist MR_liver_t1 MR_liver_t2e1 MR_ped_chest MR_sag_head
256x256 256x256 256x256 256x256 256x256 256x256 256x256 256x256
8 8 8 8 8 8 8 8
97 74 203 183 58 58 77 58
Xray_30584065 Xray_39068112 Xray_39468655 Xray_46939122 Xray_46948506 Xray_51166511 Xray_51741424
1024x1024 1024x1024 1024x1024 1024x1024 1024x1024 1024x1024 1024x1024
12 12 12 12 12 12 12
5 4 6 5 5 5 4
CT_noncon_chest CT_sag_chest
512x512 512x512
12 12
199 109
MR_kidney_t1 MR_kidney_t2
368x272 368x272
16 16
163 163
The directional intra prediction used in HEVC is based on a palette of 33 angular modes plus a planar and a DC mode. This type of prediction is based on previously decoded neighbouring samples found along the directions defined by the angular modes. Due to the use of a wide range of intra prediction directions, combined with its very efficient inter prediction scheme, HEVC is able to achieve good performance in lossless compression of volumetric medical images, such as CT and MR, as well as X-ray Angiography images. After completion of the first edition of the HEVC standard, several additional coding tools were added as part of the Range Extension (RExt) [24]. These extensions include coding tools to support content with higher bit depths and higher chroma-resolution, and also lossless encoding through a technique known as Sample Adaptive intraPrediction (SAP) [25]. SAP improves the horizontal and vertical intraprediction modes by computing pixel predictions from the original values of their neighbours, regardless whether they are located inside or
Fig. 2. Middle slice of each medical volume from CIPR [20].
3
Signal Processing: Image Communication (xxxx) xxxx–xxxx
A.F.R. Guarda et al.
Fig. 3. Middle frame of each X-ray Angiography from [21].
Fig. 4. Middle slice of the medical volumes from (a) and (b) [22], and (c) and (d) [23].
Table 2 Lossless performance of HEVC and DICOM recommended encoders. JPEG 2000
JPEG LS
H.264 Intra
H.264 IPPP
HEVC Intra
HEVC R.A.
HEVC RExt
2.473
2.290
2.545
1.914
2.620
1.614
1.470
where N is the filter order and ai are the optimum filter coefficients. The optimal filter coefficients ai are estimated using a previously encoded region, commonly referred to as training window, of size M = 2 T (T + 1) pixels (with N ⪡M ), as shown in Fig. 5. By arranging all pixels in the training window as an M × 1 column vector → y = [t0, …, tM )]T , one can write an equation similar to Eq. (1) for every pixel ti of the training window, where the filter coefficients ai are the unknowns. The optimal filter coefficients are obtained by minimising the prediction error for all M predictions of pixels ti, given by Eq. (2):
All the previously described techniques show that pixel-wise prediction is efficient for lossless compression. Following a similar approach, we propose to extend current methods by using LSP, in addition to the existing SAP technique defined in the Range Extension of HEVC, as described in the following sections.
4. Least-Squares Prediction The LSP method aims to find the optimum prediction, x0, s , of a current pixel, x0, by using a linear adaptive filter applied to a selected set of N neighbouring pixels. The filter support s is defined relatively to the position of pixel x0 and corresponds to a vector Xs = [x1, s, …, xN , s] of previously encoded image pixels. The predicted value of a current pixel x0 is given by: N
x0, s =
∑ ai xi,s i =1
s = 1, …, M (1)
Fig. 5. A filter support Xs and training window.
4
Signal Processing: Image Communication (xxxx) xxxx–xxxx
A.F.R. Guarda et al.
→ → ∥2 }, a = arg min{∥ → y − Ca (2) → where the column vector a = [a1, …, aN ]T are the filter coefficients and matrix CM × N is composed by M filter supports of size N, as shown in Eq. (3): ⎡ x1,1 ⋯ xN ,1 ⎤ C = [X1, X2, …, XM ]T = ⎢ ⋮ ⋱ ⋮ ⎥ . ⎢⎣ x ⎥ 1, M ⋯ xN , M ⎦
•
(3)
A closed form solution to this problem [13] is given by Eq. (4):
→ a = (C T C )−1 (C T→ y ).
(4)
After computing the optimum filter coefficients ai and the best predictions for the M pixels, the training windows is shifted and the optimisation process is repeated for a new training window, centred around a new pixel to be predicted. This is repeated along the whole block until finding the best predictions for all pixels. Least Squares-based prediction techniques have been successfully used in different approaches [13–16]. By using pixels in the training window, LSP can accurately predict the pixels in the current PU, even in cases where traditional prediction methods fail, as shown in [13]. For lossless compression of natural images, due to its ability to correctly predict arbitrarily oriented edges, the LSP produces better results than state-of-the-art predictors, such as MED and GAP, used by JPEG-LS and CALIC, respectively. In some situations (e. g . when the pixel to be predicted belongs to the right extreme of the current PU or of the image), part of the filter support may belong to a block not yet encoded (see pixels 4, 7, and 9 in Fig. 5). In those cases, an alternative filter support can be used instead, as shown in Fig. 6. This adaptation of the training window for the current pixel position, proposed by Danillo et al. [14], resulted in better prediction coefficients, improving the LSP prediction capability for block prediction. LSP was also used for stereo image encoding, by extending the filter support to both views, after disparity compensation with template matching [15]. In this case, the pixels located below and to the right of the current pixel in the other view, which has already been encoded, can be used for the filter support. In order to improve the temporal prediction, LSP was proposed for video encoding in [16], to replace the commonly used motion estimation and compensation based on block-matching. Due to its edge prediction ability, LSP can also be used to obtain the prediction of the current pixel along time, by extending the filter support to the previous frame. By using an adaptive filter support, different types of motion can be compensated more effective, without the need for any signalling overhead.
spread-out, thus applying DPCM may ensue in a higher encoding rate. As such, we propose the use of DPCM only for medical volumes with 8 bit depth. The second stage implements a new lossless mode specifically devised for HEVC Range Extension (main-RExt) profile. This lossless mode is based on LSP, but uses some improvements for both intra and inter prediction, which are described in the following subsections. In order to minimize the modifications to the HEVC algorithm, the LSP mode replaces one of the less used intra-prediction modes. Fig. 7 shows the percentage of usage of each Intra mode in HEVC for the test sequences, where it can be seen that modes 28, 11, 25 and 27 are the least used, with the Intra direction mode 27 presenting the overall minimum usage. In [32] a detailed statistical study about the usage of each intra mode in HEVC is presented for video signals, confirming that mode 27 is generally one of the least frequently used modes. Based on this evidence, Intra direction mode 27 was selected to be replaced by the proposed LSP.
Although the LSP mode is applied to the whole PU, Least Squares optimization is performed on a pixel-by-pixel basis, that is, the predictor coefficients are estimated for each pixel of the PU, individually. Since this operation uses an implicit method, which relies only on the causal pixels, it can be performed in both the encoder and decoder without the need of transmitting any additional overhead information, as mentioned before. 5.1. Intra improvements to LSP Since LSP is applied for block prediction, the causal filter support and training window adaptation shown in Fig. 6 were implemented as proposed in [14]. In lossy applications, the decoder only has an approximation of the original pixel values, thus, when the filter support includes causal pixels of the same PU, their previously determined prediction values are used to predict the current pixel. However, a significant difference between the original and predicted value may be propagated to the following pixels, leading to a high prediction error. An advantage of lossless coding comes from the fact that the original pixel values (non-modified) are always available at both the encoder and decoder. Thus by using these values for prediction, LSP achieves better efficiency due to lower prediction error. In order to improve the stability of the method by giving preference to solutions with smaller norms, Tikhonov regularization is used [33,34]. For this purpose, a regularization term is added to the minimization problem, as follows:
→ → ∥2 + ∥ Γa → ∥2 }, a = arg min{∥ → y − Ca
5. New prediction mode for lossless HEVC
(5)
and the regularized solution is computed according to
→ a = (C T C + Γ T Γ )−1 (C T→ y ),
In order to efficiently compress the volumetric medical images, a two-stage prediction method is proposed.
(6)
where Γ is the Tikhonov matrix, defined as Γ = αI , with I being the
• The
first stage is a preprocessing stage, with the purpose of performing inter-slice decorrelation, and consists in a slice-wise difference, also referred to as DPCM, as applied in [31]. Since images with higher bit depth inherently present more noise, the histogram of the residue images resulting from DPCM is more
Fig. 6. Alternative filter support and training window as proposed in [14].
Fig. 7. Usage of HEVC Intra modes in descending order.
5
Signal Processing: Image Communication (xxxx) xxxx–xxxx
A.F.R. Guarda et al.
identity matrix and α the regularization parameter. A set of experimental tests were performed in order to determine this parameter. These tests show that small variations of α do not have a significant impact in the results, with 0.1 ≤ α ≤ 5 producing the best results.
CALIC which allows encoding of images with bit depth up to 16, as well as the H.264/AVC and a standard version of HEVC with Range Extensions. The compression results are presented in Table 3, in bitsper-pixel (bpp). The first four columns show the performance of the lossless image encoders: JPEG2000, JPEG-LS, CALIC and MRP. JPEG2000 has the worst performance, which can be explained by the use of a wavelet transform instead of a pixel-wise predictor, as in the other three encoders. It can also be seen that the more advanced the prediction method, the better the results obtained, with MRP outperforming both CALIC (GAP) and JPEG-LS (MED). The next three columns present the results for a different implementation of 2D CALIC as well as its extensions 3D CALIC and M-CALIC. Comparing to the original CALIC, this implementation of 2D CALIC shows slightly worse results, while the 3D CALIC shows a great improvement due to its ability to perform interslice prediction, and as for M-CALIC, a much smaller gain is achieved. The next two columns present the results for JP3D, which are similar to those of 3D CALIC, and for H.264/AVC with IPPP coding structure, which compared to MRP presents a worse compression ratio, despite using inter-frame prediction, thanks to the effectiveness of MRP prediction technique. The performance of HEVC with Range Extension can also be compared in Table 3. It can be seen that the Range Extension profile in HEVC for lossless coding provides the best performance, with a significant advantage over the remaining encoders. The last column shows that the proposed LSP mode achieves a significant reduction of bitrate, from 1.470 bpp to 1.267 bpp (i.e., 13.8% on average), in comparison to the RExt profile of HEVC. This major improvement of the encoding efficiency is mainly due to the inter-slice prediction efficiency of the proposed method, since it better exploits the redundancy of the volumetric medical images. When compared to DICOM recommended encoders, such as JPEG-LS, HEVC with the proposed method presents a much higher encoding efficiency, reducing the bitrate from 2.290 bpp to 1.267 bpp (i.e., 44.7%). To better evaluate the impact of each of the techniques used in the proposed LSP mode, we performed some tests by enabling them incrementally. The results of these experiments, presented in Table 4, show that when the LSP is introduced in HEVC, using a straightforward implementation in HEVC with RExt profile, only a marginal gain is obtained (0.1%). However, when adapting LSP for lossless coding using the original pixel values for the filter support this gain becomes much more significant (5.0%). The coding efficiency is further enhanced after extending the LSP filter support to a previous and a following slices (bitrate reduction of 13.1%). Finally, using Tikhonov regularization increases the bitrate savings to 13.8%, when comparing to the unmodified HEVC with RExt profile.
5.2. Inter slice improvements to LSP As pointed out before, due to the characteristics of volumetric images such as MR and CT, and sequences such as X-ray Angiography images, in which human organs, tissues and bones are represented, adjacent slices are highly correlated. To take advantage of this similarity, the LSP support was expanded to include pixels from the previous and the following slices, allowing the prediction accuracy to be improved. This extended support is used only in non-Intra slices, with the previous slice being the first one from Reference_List_0, and the following slice being the first one from Reference_List_1 of HEVC. This approach adapts LSP to volumetric datasets by exploiting the reference frame structure of HEVC, leading to an increased overall coding efficiency. 6. Experimental results The proposed lossless LSP mode was implemented into the HEVC reference software and its performance was evaluated using the previously described medical image datasets. The latest release of the HEVC reference software HM 16.9 was used, with the Intra Main profile, and the Random Access Main and RExt profiles. To guarantee lossless coding, QP was set to 0, and both TransquantBypassEnableFlag and CUTransquantBypassFlagForce were set to 1. For the RExt profile, the parameter CostMode was set to lossless, while the default values were used for the remaining configuration parameters. For all the other encoders, the default configurations for lossless encoding were used. The proposed LSP mode uses a filter support which was empirically determined after testing several geometries for various images. The best results were achieved for a triangular template with 12 pixels in the current slice, and a diamond shaped template with five pixels centred in the pixel co-located pixel of the previous and next slices, as depicted in Fig. 8 by the dashed line. For simplicity, the index s is also omitted in this Figure. Using more than five pixels does not improve the prediction performance and significantly increases the computational complexity, therefore the five pixel region was adopted as the best choice. The training window size was set to T=7. 6.1. 8 bit depth images A first test was performed for images with eight bits per pixel (CIPR dataset [20]), where the encoding performance of the second stage of the proposed method is compared to DICOM recommended encoders, JPEG2000 and JPEG-LS, the extension for volumetric images JP3D, the state-of-the-art lossless image encoders MRP and CALIC, extensions for multi-spectral images 3D CALIC and M-CALIC and also a version of 2D
6.2. 12/16 bit depth images The performance of the second stage of the proposed method was also evaluated and compared with the other encoders for images with
Fig. 8. Extended LSP filter support for volumetric images: each additional slice provides the 5 pixels shown within the dashed line.
6
Signal Processing: Image Communication (xxxx) xxxx–xxxx
A.F.R. Guarda et al.
Table 3 Lossless coding performance (bpp) using the 8 bpp dataset [20]. Sequence
JPEG 2000
JPEG LS
CALIC
MRP
2D CALIC
3D CALIC
M CALIC
JP3D
H.264 IPPP
HEVC RExt
Proposed LSP
CT_Aperts CT_carotid CT_skull CT_wrist MR_liver_t1 MR_liver_t2e1 MR_ped_chest MR_sag_head
1.261 2.019 2.991 1.757 3.256 2.572 3.021 2.905
1.058 1.778 2.761 1.627 3.160 2.418 2.937 2.582
0.998 1.684 2.628 1.550 3.022 2.269 2.789 2.519
0.775 1.374 2.329 1.173 2.582 1.722 2.337 2.279
1.137 1.728 2.755 1.697 3.086 2.306 2.868 2.626
0.994 1.517 2.153 1.127 2.323 1.828 1.860 2.329
1.205 1.800 2.585 1.435 2.832 2.179 2.213 2.605
0.941 1.547 2.088 1.238 2.356 1.745 2.071 2.160
0.892 1.644 2.207 1.421 2.813 2.155 1.982 2.196
0.728 1.425 1.766 1.002 2.052 1.509 1.534 1.748
0.619 1.201 1.558 0.826 1.762 1.192 1.364 1.616
Average
2.473
2.290
2.183
1.821
2.275
1.766
2.107
1.768
1.914
1.470
1.267
6.3. Inter-slice decorrelation
Table 4 Breakdown of the improvement achieved by the proposed LSP mode. HEVC RExt
LSP
w/ Original Pixels
Extended Support
Tikhonov Regularization
1.470
1.468 (-0.1%)
1.396 (-5.0%)
1.277 (-13.1%)
1.267 (-13.8%)
Since JPEG2000 and JPEG-LS do not exploit the inherent inter-slice redundancy, it is expected that their performance falls short of other encoders such as HEVC. However, Part 2 of JPEG2000 includes multicomponent transformations, designed for multi-spectral and volumetric images, and is also recommended by DICOM [4]. The included multicomponent transformations are 5/3 Reversible Wavelet Transform (RWT), Reversible Haar Transform (RHAAR), Reversible Karhunen Loève Transform (RKLT) and DPCM, which corresponds to the first stage of the proposed method. The executable files and scripts to perform these transformations were provided by [21]. The transformations were applied to all test datasets as a preprocessing step, and the resulting images were compressed with all previous encoders which allow 16 bit depth, even for encoders that already present inter-slice prediction such as HEVC and the proposed LSP method. The compression results for each transformation and the original sequences are presented in Table 6, with the average results for each dataset shown in bpp. For each encoder and each dataset, the best transformation results are underlined, and the best overall results are also in bold. It can be seen that the use of multi-component transformations greatly improves the performance of JPEG2000, JPEG-LS and 2D CALIC, for the first two datasets, with a small advantage for 2D CALIC, with DPCM achieving the best results for 8 bit depth images and RKLT for X-ray Angiography images. The performance of 3D CALIC and MCALIC, which already exploit inter-slice redundancy, is severely deteriorated for all transformations. As for HEVC and the proposed LSP technique, a significant gain is obtained for 8 bit depth images using the first stage DPCM, while for the remaining datasets, the best ones are still obtained when encoding the original sequences. Comparing the proposed LSP with the remaining encoders, it can be seen that it still presents the best results for all datasets.
higher bit depth, i.e., 12 or 16, using the datasets available from [21,22] and [22]. The following encoders support images with bit depths up to 16: JPEG2000, JPEG-LS, the new implementations of 2D CALIC and extensions, 3D CALIC and M-CALIC, and also a standard version of HEVC with and without Range Extension. For this dataset, some additional configuration parameters of HEVC were changed, besides the previously presented ones. Namely, the chroma format was set to 4:0:0, the bit depth was set to 12, and the “monochrome16” profile was used. Also, to allow encoding data of over 10 bit depth, the reference software was compiled with the macro “RExt__HIGH_BIT_DEPTH_SUPPORT” set to 1. The compression results are presented in Table 5, in bits-per-pixel (bpp). It can be seen that JPEG2000 and JPEG-LS present similar results, with 2D CALIC achieving a slightly better compression ratio. However, the extensions to 2D CALIC did not improve the results, even though they exploit the redundancy between slices. As for HEVC, the use of Range Extension allowed a significant gain, and the proposed LSP mode improved HEVC RExt even further, with a bitrate reduction from 5.954 bpp to 5.848 bpp (i.e., 1.8%) for X-ray sequences, and from 5.036 bpp to 4.803 bpp (i.e., 4.6%) for CT and MR scans, presenting the best overall performance. These results verify the same tendency as those obtained for the sequences with eight bit depth, from Table 3, however, with smaller gains.
Table 5 Lossless performance comparison of the proposed LSP mode with HEVC and other state-of-the-art encoders for the 12/16 bpp datasets (in bpp). Sequence
JPEG 2000
JPEG LS
2D CALIC
3D CALIC
M CALIC
HEVC
HEVC RExt
Proposed LSP
Xray_30584065 Xray_39068112 Xray_39468655 Xray_46939122 Xray_46948506 Xray_51166511 Xray_51741424
6.027 6.055 6.479 6.243 6.109 6.409 5.962
5.943 6.011 6.462 6.256 6.121 6.350 5.973
5.868 5.909 6.336 6.135 6.014 6.236 5.858
5.846 5.913 6.352 6.216 6.007 6.179 5.948
5.915 5.901 6.334 6.186 5.973 6.157 5.954
6.019 6.257 6.690 6.397 6.144 6.542 6.201
5.649 5.929 6.193 6.050 5.837 6.133 5.885
5.530 5.769 6.103 6.005 5.767 6.011 5.749
Average
6.183
6.159
6.051
6.066
6.060
6.322
5.954
5.848
CT_noncon_chest CT_sag_chest MR_kidney_t1 MR_kidney_t2
5.301 3.530 6.145 6.518
5.373 3.935 5.602 5.959
5.193 3.647 5.729 6.091
5.027 4.073 5.830 6.173
5.430 4.438 5.917 6.215
5.303 4.559 8.242 9.053
4.907 3.761 5.555 5.921
4.475 3.281 5.547 5.909
Average
5.374
5.217
5.165
5.276
5.500
6.789
5.036
4.803
7
Signal Processing: Image Communication (xxxx) xxxx–xxxx
A.F.R. Guarda et al.
Table 6 Lossless performance comparison of the proposed method with HEVC and other state-of-the-art encoders when using multi-component transformations (in bpp). JPEG2000
JPEG LS
2D CALIC
3D CALIC
M CALIC
HEVC
HEVC RExt
Proposed LSP
Original
CT/MR (8 bpp) X-ray (12 bpp) CT/MR (12/16 bpp)
2.473 6.183 5.374
2.290 6.159 5.217
2.275 6.051 5.165
1.766 6.066 5.276
2.107 6.060 5.500
1.614 6.322 6.789
1.470 5.954 5.036
1.267 5.848 4.803
DPCM
CT/MR (8 bpp) X-ray (12 bpp) CT/MR (12/16 bpp)
1.729 6.206 5.372
1.767 6.227 5.264
1.689 6.104 5.186
2.250 6.983 5.605
7.298 8.557 6.901
1.329 6.505 7.061
1.277 6.146 5.218
1.155 5.961 4.935
RWT
CT/MR (8 bpp) X-ray (12 bpp) CT/MR (12/16 bpp)
1.751 6.011 5.515
1.791 6.032 5.425
1.734 5.915 5.409
2.525 7.042 5.876
6.911 8.068 7.393
1.665 6.317 7.296
1.589 6.008 5.419
1.466 5.916 5.132
RHAAR
CT/MR (8 bpp) X-ray (12 bpp) CT/MR (12/16 bpp)
1.971 6.008 5.401
2.031 6.026 5.289
1.925 5.919 5.259
2.942 7.088 5.783
7.722 8.112 7.298
1.897 6.298 7.119
1.785 5.965 5.297
1.590 5.876 5.013
RKLT
CT/MR (8 bpp) X-ray (12 bpp) CT/MR (12/16 bpp)
2.289 5.946 7.074
2.359 5.982 6.948
2.303 5.879 7.025
3.034 6.891 7.367
8.734 8.676 8.724
2.173 6.260 9.534
2.099 5.958 6.983
1.979 5.887 6.649
LSP mode. We have improved LSP for lossless encoding by using the original causal pixels, instead of the previously determined prediction values, to predict the following ones. This method has also been extended to exploit inter-slice redundancy using the nearest previously encoded images for LSP prediction, thus increasing its efficiency for the compression of medical images such as CT, MR and X-ray Angiography images. To increase the robustness of the linear filter parameters estimation, a regularization has been included in the algorithm. The experimental results show that the proposed method achieves a considerable improvement of the compression gain, with bitrate reductions of approximately 13.8% for 8 bpp CT and MR images, 1.8% for 12 bpp X-ray Angiography images, and 4.6% for 12/16 bpp CT and MR images, when comparing to the HEVC RExt profile. When comparing to DICOM recommended encoders, a gain of over 44% for 8 bpp CT and MR images, 5% for 12 bpp X-ray Angiography images, and 8% for 12/16 bpp CT and MR images is obtained.
In [31], the MRP was optimised for compression of volumetric medical images, by exploiting inter-slice redundancy and also including DPCM preprocessing stage. The results obtained in [31] are a little better than those of the proposed LSP method based on HEVC. This can be justified by the fact that HEVC was developed mainly for lossy video encoding, requiring more overhead bits than MRP, which was specifically developed for lossless coding of still images. Nevertheless, the slightly higher efficiency of MRP is obtained with around three times higher computational time than the current proposal. 6.4. Computational complexity As can be easily inferred from the description of the LSP-based method presented in Sections 4 and 5, the computational complexity of standard HEVC encoders and decoders is significantly increased by the LSP algorithm. Although in typical medical image compression applications real-time encoding is not always required and so the encoding complexity may not be a very serious constraint, the decoding complexity can be a problem due to the response times expected from image rendering devices. To evaluate the approximate order of magnitude of the complexity increase, a set of experiments were performed on a PC with an Intel Xeon CPU E3-1240 V2 @ 3.40 GHz and 24 GB of RAM, running Ubuntu 16.04. The average processing time per slice of the proposed LSP method (in seconds) was compared with that of the HEVC reference software with the RExt profile, for encoding and decoding all datasets. For the encoder, the proposed method consumes around 30 times more than the original HEVC, while for the decoder the processing time can rise up to few hundred times higher. Since none of the implementations was optimized for speed, it is reasonable to assume that such results may be brought down to acceptable levels, particularly taking into account that the complexity of HEVC encoding can be easily reduced about 70% [35]. Therefore, there is plenty of room for algorithmic optimization and improvement of the proposed method. This will be addressed in future research efforts, namely by investigating new techniques to reduce the complexity of the decoder through the use of side information or implicit coding modes, capable of avoiding exhaustive training of the LSP coefficients in the decoder side.
Acknowledgements This work is a part of the UDICMI project funded by CENTRO-07ST24-FEDER-002022 of QREN and project Medicomp in the scope of R & D Unit 50008, Grant no. PEst-OE/EEI/LA0008/2013, financed by the applicable financial framework (FCT/MEC through national funds and when applicable co-funded by FEDER-PT2020 partnership agreement). The authors would like to thank Martin Briano for providing his implementation [36] of 2D CALIC, 3D CALIC and M-CALIC. References [1] G. Sullivan, J. Ohm, W.-J. Han, T. Wiegand, Overview of the high efficiency video coding (HEVC) standard, IEEE Trans. Circuits Syst. Video Technol. 22 (12) (2012) 1649–1668. [2] J. Taquet, C. Labit, Hierarchical oriented predictions for resolution scalable lossless and near-lossless compression of CT and MRI biomedical images, IEEE Trans. Image Process. 21 (5) (2012) 2641–2652. [3] R. C. of Radiologists, The adoption of lossy image data compression for the purpose of clinical interpretation, version 1.1, April 2008 (Jan 2017). URL 〈https://www. rcr.ac.uk/it-guidance-adoption-lossy-image-data-compression-purpose-clinicalinterpretation〉. [4] DICOM, Digital Image and Communications in Medicine, Strategic document, 2014. [5] M. Weinberger, G. Seroussi, G. Sapiro, The LOCO-I lossless image compression algorithm: principles and standardization into JPEG-LS, IEEE Trans. Image Process. 9 (8) (2000) 1309–1324. [6] Information technology: JPEG 2000 image coding system, part 1: Core coding system, document 15444-1, ISO/IEC, 2000. [7] G. J. Sullivan, P. N. Topiwala, A. Luthra, The H.264/AVC advanced video coding standard: overview and introduction to the fidelity range extensions, in:
7. Conclusion In this paper we propose to improve the HEVC lossless compression tools for volumetric medical images and sequences, by using a modified 8
Signal Processing: Image Communication (xxxx) xxxx–xxxx
A.F.R. Guarda et al.
[8] [9]
[10]
[11] [12]
[13] [14]
[15]
[16] [17]
[18]
[19]
[20] [21]
[22] N. B. I. Archive, Subject GSM714044, Jan 2017. URL 〈https://imaging.nci.nih.gov/ ncia/login.jsf〉. [23] L. Xie, M.A. Sparks, W. Li, Y. Qi, C. Liu, T.M. Coffman, G.A. Johnson, Quantitative susceptibility mapping of kidney inflammation and fibrosis in type 1 angiotensin receptor-deficient mice, NMR Biomed. 26 (12) (2013) 1853–1863. URL 〈http:// www.civm.duhs.duke.edu/lx201204/〉. [24] G. Sullivan, J. Boyce, Y. Chen, J.-R. Ohm, C. Segall, A. Vetro, Standardized extensions of high efficiency video coding (HEVC), IEEE J. Select. Top. Signal Process. 7 (6) (2013) 1001–1016. [25] M. Zhou, W. Gao, M. Jiang, H. Yu, HEVC lossless coding and improvements, IEEE Trans. Circuits Syst. Video Technol. 22 (12) (2012) 1839–1843. [26] V. Sanchez, P. Nasiopoulos, R. Abugharbieh, Efficient lossless compression of 4-d medical images based on the advanced video coding scheme, IEEE Trans. Inform. Technol. Biomed. 12 (4) (2008) 442–446. [27] E. Wige, G. Yammine, P. Amon, A. Hutter, A. Kaup, Pixel-based averaging predictor for HEVC lossless coding, in: Proceedings of IEEE International Conference on Image Processing (ICIP), 2013, pp. 1806–1810. [28] E. Wige, G. Yammine, P. Amon, A. Hutter, A. Kaup, Sample-based weighted prediction with directional template matching for HEVC lossless coding, Picture Coding Symposium (PCS), 2013, pp. 305–308. [29] V. Sanchez, Sample-based edge prediction based on gradients for lossless screen content coding in HEVC, in: Proceedings of Picture Coding Symposium (PCS), 2015, pp. 134–138. [30] V. Sanchez, Lossless screen content coding in HEVC based on sample-wise median and edge prediction, in: Proceedings of IEEE International Conference on Image Processing (ICIP), 2015, pp. 4604–4608. [31] J. Santos, A. Guarda, N. Rodrigues, S. de Faria, Contributions to lossless coding of medical images using minimum rate predictors, in: Proceedings of IEEE International Conference on Image Processing (ICIP), 2015, pp. 2935–2939. [32] R.I. Chernyak, Analysis of the intra predictions in H.265/HEVC, Appl. Math. Sci. 8 (148) (2014) 7389–7408. [33] A.-I. N. Tikhonov, V. Y. Arsenin, Solutions of Ill Posed Problems (Scripta series in mathematics), Vh Winston, 1977. [34] I. Rish, G. Grabarnik, Sparse Modeling: Theory, Algorithms, and Applications, CRC Press, Inc., Boca Raton, FL, USA, 2015. [35] G. Correa, P. Assuncao, L. Agostini, L.A. da Silva Cruz, Complexity-Aware High Efficiency Video Coding, Springer International Publishing, Switzerland, 2016. [36] M. Briano, CALIC, 3D CALIC and M-CALIC customizable software implementation, July 2014. URL 〈https://prezi.com/j2ichhip4ls7/calic-implementation/〉.
Proceedings of SPIE Conference on Applications of Digital Image Processing XXVII, 2004, pp. 454–474. X. Wu, N. Memon, Context-based, adaptive, lossless image coding, IEEE Trans. Commun. 45 (4) (1997) 437–444. I. Matsuda, Y. Umezu, N. Ozaki, J. Maeda, S. Itoh, A lossless coding scheme using adaptive predictors and arithmetic code optimized for each image, Syst. Comput. Jpn. 38 (4) (2007) 1–11. P. Schelkens, A. Munteanu, A. Tzannes, C. Brislawn, JPEG2000 – Part 10. Volumetric data encoding, in: Proceedings of IEEE International Symposium on Circuits and Systems, ISCAS Proceedings, 2006, pp. 3877–3880. X. Wu, N. Memon, Context-based lossless interband compression-extending CALIC, IEEE Trans. Image Process. 9 (6) (2000) 994–1001. E. Magli, G. Olmo, E. Quacchio, Optimized onboard lossless and near-lossless compression of hyperspectral data using CALIC, IEEE Geosci. Remote Sens. Lett. 1 (1) (2004) 21–25. X. Li, M. Orchard, Edge-directed prediction for lossless compression of natural images, IEEE Trans. Image Process. 10 (6) (2001) 813–817. D. Graziosi, N. Rodrigues, E. da Silva, S. de Faria, M. de Carvalho, Improving multiscale recurrent pattern image coding with least-squares prediction mode, in: Proceedings of IEEE International Conference on Image Processing (ICIP), 2009, pp. 2813–2816. L. Lucas, N. Rodrigues, E. da Silva, S. de Faria, Adaptive least squares prediction for stereo image coding, in: Proceedings of the IEEE International Conference on Image Processing (ICIP), 2011, pp. 2013–2016. X. Li, Least-square prediction for backward adaptive video coding, EURASIP J. Appl. Signal Process. 2006 (2006) 126. SiemensHealthcare, Computed tomography, Jan 2017. URL 〈http://www. medicalradiation.com/types-of-medical-imaging/imaging-using-x-rays/computedtomography-ct/〉. SiemensHealthcare, Magnetic resonance imaging, Jan 2017. URL 〈http://www. medicalradiation.com/types-of-medical-imaging/other-types-of-medical-imaging/ magnetic-resonance-imaging/〉. C. Cavaro-Menard L. Zhang P. le Callet. Diagnostic quality assessment of medical images: challenges and trends, in: Proceedings of European Workshop on Visual Information Processing (EUVIP), 2010, pp. 277–284. C. for Image Processing Research, Medical images datasets, Jan 2017. URL 〈http:// www.cipr.rpi.edu/resource/sequences/sequence01.html〉. Z. Xu, J. Bartrina-Rapesta, I. Blanes, V. Sanchez, J. Serra-Sagrist, M. Garca-Bach, J. F. Muoz, Diagnostically lossless coding of X-ray angiography images based on background suppression, Computers & Electrical Engineering, 2016.
9
Contents lists available at ScienceDirect
Signal Processing: Image Communication journal homepage: www.elsevier.com/locate/image
A method to improve HEVC lossless coding of volumetric medical images ⁎
André F.R. Guardaa,b, , João M. Santosa,b, Luís A. da Silva Cruza,c, Pedro A.A. Assunçãoa,b,1, Nuno M.M. Rodriguesa,b, Sérgio M.M. de Fariaa,b a b c
Instituto de Telecomunicações, Portugal Instituto Politécnico de Leiria, ESTG, Portugal Universidade de Coimbra, DEEC, Portugal
A R T I C L E I N F O
A B S T R A C T
Keywords: HEVC LSP Lossless compression Medical imaging
Medical imaging technology and applications are continuously evolving, dealing with images of increasing spatial and temporal resolutions, which enable more accurate processing, feature analysis and medical diagnosis. However, the increase in resolution also requires a growing amount of data to be stored, processed and exchanged or transmitted through networks. Despite the high coding efficiency achieved by the most recent image and video coding standards in lossy compression, they are not well suited for quality-critical medical image compression where either near-lossless or lossless coding is required. In this paper we propose a method to improve lossless coding of volumetric medical images, such as Magnetic Resonance and Computed Tomography, and medical sequences such as X-ray Angiography images, using the latest standard High Efficiency Video Encoder (HEVC). New pixel-wise prediction techniques are proposed to extend the current HEVC lossless tools, based on Least-Squares Prediction (LSP). Experimental results show a bitrate reduction of over 44%, when compared to DICOM recommended encoders, and 13.8% when compared to standard lossless HEVC, for 8 bpp volumetric images, and over 8% and 4.6%, respectively, for volumetric images using more than 8 bpp.
1. Introduction
compression techniques in this type of applications. Nevertheless, some regulatory bodies, such as in the UK, USA and Canada, allow the use of lossy compression, with this choice being the responsibility of the institutions and radiologists [3]. The preference for lossless techniques is acknowledged by the Digital Imaging and Communications in Medicine (DICOM) [4], which recommends the use of lossless encoders such as JPEG in lossless mode, RLE (run-length encoding), TIFF PackBits, JPEG-LS [5], lossless JPEG2000 [6], H.264/AVC [7], as well as lossy JPEG and JPEG2000 with near-lossless compression ratios for this purpose. JPEG2000 is a wavelet transform-based image encoder, which allows quality and spatial scalability, followed by the entropy encoder Embedded Block Coding with Optimized Truncation (EBCOT), while JPEG-LS uses the median edge detector (MED) predictor, followed by context modelling of the prediction error and entropy encoding, using Golomb codes. MED is a simple edge detector that uses three raster-scan order causal pixels to determine if a vertical or horizontal edge exists, and then selects one of three predictors accordingly. Other state-of-theart lossless image encoders are context-based adaptive lossless image codec (CALIC) [8] and minimum rate predictors (MRP) [9]. CALIC uses a gradient adaptive predictor (GAP), in which six causal pixels are used
Recent advances in digital imaging and video, as well as rapid evolution of acquisition and processing systems using increasingly higher resolutions, require efficient compression formats for file exchange, storage and visual communications over networks. High Efficiency Video Coding (HEVC) is the most recent standard to fulfil the challenging requirements imposed by new services and applications [1], but in the case of medical imaging systems, with the growing use of telemedicine and information sharing among the medical community, efficient image compression assumes a different meaning due to more challenging requirements [2]. Medical imaging systems are following the same trend of higher spatial and temporal resolutions and bit depths. Therefore, a simple medical examination, such as Magnetic Resonance (MR) or Computed Tomography (CT), generates a very large amount of data. Typically these data must be stored for several years, with resulting high costs in creating and maintaining large databases of medical images. Since the use of lossy compression algorithms can result in the loss of important details in medical images, which can lead to more difficult analysis or even wrong diagnosis, it is recommended to use lossless or near-lossless
⁎
Corresponding author at: Instituto de Telecomunicações, Portugal. E-mail addresses: [email protected] (A.F.R. Guarda), [email protected] (J.M. Santos), [email protected] (L.A. da Silva Cruz), [email protected] (P.A.A. Assunção), [email protected] (N.M.M. Rodrigues), [email protected] (S.M.M. de Faria). http://dx.doi.org/10.1016/j.image.2017.02.002 Received 31 January 2016; Received in revised form 1 February 2017; Accepted 2 February 2017 0923-5965/ © 2017 Elsevier B.V. All rights reserved.
Please cite this article as: Guarda, A.F.R., Signal Processing: Image Communication (2017), http://dx.doi.org/10.1016/j.image.2017.02.002
Signal Processing: Image Communication (xxxx) xxxx–xxxx
A.F.R. Guarda et al.
to estimate the local gradients and determine the prediction value, and an adaptive arithmetic encoder for the prediction residue. As for MRP, it uses several linear predictors adapted to each image, by dividing the image in blocks of variable size and sorting them into classes, with each class using different coefficients. Multiple optimizations are performed in order to minimize the cost function that represents the prediction error data, which is encoded with a Huffman or a range coder. Volumetric medical images, such as MRI or CT scans, comprise a set of static images, each one representing a slice of a relevant volume for medical purposes. Although each slice can be independently encoded using a standard image encoder, such option does not exploit the interslice redundancy present in volumetric image sets. In order to address this issue, an extension of the JPEG2000 encoder was developed for volumetric images, known as JP3D [10], which uses 3D blocks, a 3D discrete wavelet transform and also a 3D EBCOT, instead of their 2D counterparts. Another common technique to encode volumetric medical images is to arrange the slices as a pseudo-video sequence, where each slice corresponds to a video frame. Such data structure allows the use of state-of-the-art video encoders like HEVC [1], in order to take advantage of the inter-frame prediction, by exploiting inter-slice correlation. An alternative is to use the extensions of CALIC, proposed for compression of multi-spectral and hyper-spectral data, 3D CALIC [11] and M-CALIC [12], respectively, considering each slice of the volume as a different band. In these extensions, an inter-band prediction technique is introduced, in which a linear combination of pixels in the reference band is used to predict a pixel in the current band, based on the gradients. 3D CALIC switches between intra-band prediction with GAP and inter-band prediction, according to the correlation between bands in the current region, while M-CALIC always uses a modified inter-band prediction with the previous two bands being used as references. In this paper, we propose a method to enhance the HEVC lossless coding efficiency of volumetric medical images, by improving the LeastSquares Prediction (LSP) method introduced in [13]. LSP has been successfully used in previous works for single and stereo image compression, due to its ability to efficiently representing edges orientations [14–16]. Experimental results show that the proposed HEVC prediction mode results in higher lossless compression efficiency for encoding volumetric medical images, achieving an average bitrate reduction of approximately 21% over standard HEVC and up to 49% over DICOM recommended coding methods. This paper is organized as follows: Section 2 introduces relevant background information about medical imaging technologies and data sets used in this work. Section 3 presents an overview of lossless encoding with HEVC, Section 4 describes the LSP method, and Section 5 details the proposed HEVC lossless coding mode based on LSP. In Section 6 the experimental results are presented and discussed while Section 7 concludes the paper.
Fig. 1. Sagittal slices of the brain by different imaging modalities. From top-left to downright: Magnetic Resonance Imaging, Computed Tomography, Positron Emission Tomography and Ultrasound [19].
modalities are also dependent on the constraints of each specific application domain and also on the associated medical tasks. For instance, X-rays are more adequate to represent bone structures, while Magnetic Resonance provide higher definition and works better for softer tissues. Four image datasets were used in this work to evaluate the performance of the proposed coding scheme. The first one is composed of eight volumetric medical images: four CT and four MR scans, all available from [20], the image and video repository of the Center for Image Processing (CIPR) of the Rensselaer Polytechnic Institute. The volumetric data comprise sets of spatially adjacent slices. Since these slices represent cuts through anatomical organs which extend into the three dimensions, typically, neighbouring pixels in adjacent slices are highly correlated. The second set consists of seven X-ray Angiography images with multiple slices or frames taken over time, which were provided by [21]. Similarly to video frames, these pseudo-video sequences also present high correlation between slices. However, most of the encoding methods currently recommended by DICOM do not exploit this correlation to achieve higher compression ratios. Therefore it is quite opportune to investigate new coding techniques, capable of taking advantage of the inter-slice correlation characteristics of these images, to increase the compression efficiency. The third and fourth datasets present two more CT scans, which are available at the National Biomedical Imaging Archive (NBIA) [22], and two more MR scans from [23], made available by the Center for In Vivo Microscopy (CIVM), respectively, with different spatial resolutions and bit depths. The spatial resolution, bit depth and number of slices of each volumetric set (or pseudo-video sequence) are presented in Table 1. Figs. 2, 3 and 4 show the midpoint slice of each volume (or frame) of all datasets.
2. Medical imaging technologies Recent advances in understanding and exploiting several physical phenomena, such as X-ray, γ-ray, ultrasound waves and positron emission production, propagation and recording are strongly linked to the progress in different fields of medical imaging technologies. The availability of digital acquisition technologies and powerful computational resources are also driving new developments and useful solutions e.g., tomography reconstruction both static and time-varying. There are several types of medical imaging modalities, for example Computed Tomography (CT) [17], Magnetic Resonance Imaging (MRI) [18], ultrasound imaging, Positron Emission Tomography, Elastography, each with its own characteristics [19], that depend on the associated physical processes and acquisition methods. Fig. 1 shows four sagittal slices of the brain acquired with different technologies, where the different characteristics of the various imaging modalities are clearly observable. The characteristics of specific medical imaging
3. Lossless encoding using HEVC HEVC is the current state-of-the-art compression standard for lossy video compression, which also includes a lossless encoding profile. The lossless encoding process differs from the lossy one in several ways. In the lossless encoding mode the transform, quantization and in-loop filters are skipped, and the resulting residue from intra and inter prediction is entropically encoded. Since the crucial steps of residue transformation and quantization are not used, the compression gain is small and the efficiency of prediction methods becomes of utmost importance for the overall encoding performance. 2
Signal Processing: Image Communication (xxxx) xxxx–xxxx
A.F.R. Guarda et al.
outside the current prediction unit (PU), i.e., without excluding those in the current PU itself. This is equivalent to perform Differential PulseCode Modulation (DPCM) in these directions, so the resulting residue of SAP is similar to that obtained from DPCM. To provide insight about the relative performance between the DICOM recommended encoders such as JPEG2000, JPEG-LS and H.264/AVC (using the Intra profile and IPPP coding structure [26]), and HEVC (using Intra, Random Access and Random Access RExt profiles), Table 2 shows the average lossless coding efficiency obtained for the first set of volumetric medical images described in Section 2. It can be seen that JPEG2000 and JPEG-LS present better results than the Intra profiles of H.264/AVC and HEVC. However, when using interframe prediction to exploit the redundancy between slices, greater compression gain is achieved, as expected. The RExt profile presents the best results due to the use of SAP, with an average bitrate reduction of 8.9% over HEVC Random Access. It is clear that HEVC compression performance surpasses that of JPEG2000, JPEG-LS and H.264/AVC. This advantage can be explained by the use of HEVC coding tools which exploit the inter-slice redundancy in these type of images, whereas such tools are not available in JPEG2000 and JPEG-LS. This issue is discussed with more detail in Section 6. Several other techniques for lossless prediction have been proposed to replace or improve the SAP predictor of HEVC, most of them aiming at screen content coding. The Sample-based Weighted Prediction (SWP) proposed in [27] performs a weighted average of neighbouring pixels surrounding the current pixel to form its prediction. An alternative to SWP is the Directional Template Matching (DTM) [28], which instead of averaging the neighbouring pixels, uses the one with the highest similarity, achieving better reconstruction of sharp edges. The authors also found that using a combination of SWP and DTM provides better prediction performance. Another method recently proposed is the Sample-based Angular intra-Prediction with Gradient-based edge prediction (SAP-G) [29], which uses the surrounding pixels to compute the gradients in four directions (0°, 45°, 90° and 135°), and then selects the adjacent neighbouring pixel in the direction of the smallest gradient as the prediction for the current pixel. In [30], the Sample-based Angular intra-Prediction with Median and Edge (SAP-ME) was proposed, in which the MED predictor of JPEG-LS is applied in order to improve the prediction of sharp edges, as well as a median predictor to improve prediction in smooth regions.
Table 1 Medical datasets' information. Sequence
Resolution
Bit Depth
No. Slices
CT_Aperts CT_carotid CT_skull CT_wrist MR_liver_t1 MR_liver_t2e1 MR_ped_chest MR_sag_head
256x256 256x256 256x256 256x256 256x256 256x256 256x256 256x256
8 8 8 8 8 8 8 8
97 74 203 183 58 58 77 58
Xray_30584065 Xray_39068112 Xray_39468655 Xray_46939122 Xray_46948506 Xray_51166511 Xray_51741424
1024x1024 1024x1024 1024x1024 1024x1024 1024x1024 1024x1024 1024x1024
12 12 12 12 12 12 12
5 4 6 5 5 5 4
CT_noncon_chest CT_sag_chest
512x512 512x512
12 12
199 109
MR_kidney_t1 MR_kidney_t2
368x272 368x272
16 16
163 163
The directional intra prediction used in HEVC is based on a palette of 33 angular modes plus a planar and a DC mode. This type of prediction is based on previously decoded neighbouring samples found along the directions defined by the angular modes. Due to the use of a wide range of intra prediction directions, combined with its very efficient inter prediction scheme, HEVC is able to achieve good performance in lossless compression of volumetric medical images, such as CT and MR, as well as X-ray Angiography images. After completion of the first edition of the HEVC standard, several additional coding tools were added as part of the Range Extension (RExt) [24]. These extensions include coding tools to support content with higher bit depths and higher chroma-resolution, and also lossless encoding through a technique known as Sample Adaptive intraPrediction (SAP) [25]. SAP improves the horizontal and vertical intraprediction modes by computing pixel predictions from the original values of their neighbours, regardless whether they are located inside or
Fig. 2. Middle slice of each medical volume from CIPR [20].
3
Signal Processing: Image Communication (xxxx) xxxx–xxxx
A.F.R. Guarda et al.
Fig. 3. Middle frame of each X-ray Angiography from [21].
Fig. 4. Middle slice of the medical volumes from (a) and (b) [22], and (c) and (d) [23].
Table 2 Lossless performance of HEVC and DICOM recommended encoders. JPEG 2000
JPEG LS
H.264 Intra
H.264 IPPP
HEVC Intra
HEVC R.A.
HEVC RExt
2.473
2.290
2.545
1.914
2.620
1.614
1.470
where N is the filter order and ai are the optimum filter coefficients. The optimal filter coefficients ai are estimated using a previously encoded region, commonly referred to as training window, of size M = 2 T (T + 1) pixels (with N ⪡M ), as shown in Fig. 5. By arranging all pixels in the training window as an M × 1 column vector → y = [t0, …, tM )]T , one can write an equation similar to Eq. (1) for every pixel ti of the training window, where the filter coefficients ai are the unknowns. The optimal filter coefficients are obtained by minimising the prediction error for all M predictions of pixels ti, given by Eq. (2):
All the previously described techniques show that pixel-wise prediction is efficient for lossless compression. Following a similar approach, we propose to extend current methods by using LSP, in addition to the existing SAP technique defined in the Range Extension of HEVC, as described in the following sections.
4. Least-Squares Prediction The LSP method aims to find the optimum prediction, x0, s , of a current pixel, x0, by using a linear adaptive filter applied to a selected set of N neighbouring pixels. The filter support s is defined relatively to the position of pixel x0 and corresponds to a vector Xs = [x1, s, …, xN , s] of previously encoded image pixels. The predicted value of a current pixel x0 is given by: N
x0, s =
∑ ai xi,s i =1
s = 1, …, M (1)
Fig. 5. A filter support Xs and training window.
4
Signal Processing: Image Communication (xxxx) xxxx–xxxx
A.F.R. Guarda et al.
→ → ∥2 }, a = arg min{∥ → y − Ca (2) → where the column vector a = [a1, …, aN ]T are the filter coefficients and matrix CM × N is composed by M filter supports of size N, as shown in Eq. (3): ⎡ x1,1 ⋯ xN ,1 ⎤ C = [X1, X2, …, XM ]T = ⎢ ⋮ ⋱ ⋮ ⎥ . ⎢⎣ x ⎥ 1, M ⋯ xN , M ⎦
•
(3)
A closed form solution to this problem [13] is given by Eq. (4):
→ a = (C T C )−1 (C T→ y ).
(4)
After computing the optimum filter coefficients ai and the best predictions for the M pixels, the training windows is shifted and the optimisation process is repeated for a new training window, centred around a new pixel to be predicted. This is repeated along the whole block until finding the best predictions for all pixels. Least Squares-based prediction techniques have been successfully used in different approaches [13–16]. By using pixels in the training window, LSP can accurately predict the pixels in the current PU, even in cases where traditional prediction methods fail, as shown in [13]. For lossless compression of natural images, due to its ability to correctly predict arbitrarily oriented edges, the LSP produces better results than state-of-the-art predictors, such as MED and GAP, used by JPEG-LS and CALIC, respectively. In some situations (e. g . when the pixel to be predicted belongs to the right extreme of the current PU or of the image), part of the filter support may belong to a block not yet encoded (see pixels 4, 7, and 9 in Fig. 5). In those cases, an alternative filter support can be used instead, as shown in Fig. 6. This adaptation of the training window for the current pixel position, proposed by Danillo et al. [14], resulted in better prediction coefficients, improving the LSP prediction capability for block prediction. LSP was also used for stereo image encoding, by extending the filter support to both views, after disparity compensation with template matching [15]. In this case, the pixels located below and to the right of the current pixel in the other view, which has already been encoded, can be used for the filter support. In order to improve the temporal prediction, LSP was proposed for video encoding in [16], to replace the commonly used motion estimation and compensation based on block-matching. Due to its edge prediction ability, LSP can also be used to obtain the prediction of the current pixel along time, by extending the filter support to the previous frame. By using an adaptive filter support, different types of motion can be compensated more effective, without the need for any signalling overhead.
spread-out, thus applying DPCM may ensue in a higher encoding rate. As such, we propose the use of DPCM only for medical volumes with 8 bit depth. The second stage implements a new lossless mode specifically devised for HEVC Range Extension (main-RExt) profile. This lossless mode is based on LSP, but uses some improvements for both intra and inter prediction, which are described in the following subsections. In order to minimize the modifications to the HEVC algorithm, the LSP mode replaces one of the less used intra-prediction modes. Fig. 7 shows the percentage of usage of each Intra mode in HEVC for the test sequences, where it can be seen that modes 28, 11, 25 and 27 are the least used, with the Intra direction mode 27 presenting the overall minimum usage. In [32] a detailed statistical study about the usage of each intra mode in HEVC is presented for video signals, confirming that mode 27 is generally one of the least frequently used modes. Based on this evidence, Intra direction mode 27 was selected to be replaced by the proposed LSP.
Although the LSP mode is applied to the whole PU, Least Squares optimization is performed on a pixel-by-pixel basis, that is, the predictor coefficients are estimated for each pixel of the PU, individually. Since this operation uses an implicit method, which relies only on the causal pixels, it can be performed in both the encoder and decoder without the need of transmitting any additional overhead information, as mentioned before. 5.1. Intra improvements to LSP Since LSP is applied for block prediction, the causal filter support and training window adaptation shown in Fig. 6 were implemented as proposed in [14]. In lossy applications, the decoder only has an approximation of the original pixel values, thus, when the filter support includes causal pixels of the same PU, their previously determined prediction values are used to predict the current pixel. However, a significant difference between the original and predicted value may be propagated to the following pixels, leading to a high prediction error. An advantage of lossless coding comes from the fact that the original pixel values (non-modified) are always available at both the encoder and decoder. Thus by using these values for prediction, LSP achieves better efficiency due to lower prediction error. In order to improve the stability of the method by giving preference to solutions with smaller norms, Tikhonov regularization is used [33,34]. For this purpose, a regularization term is added to the minimization problem, as follows:
→ → ∥2 + ∥ Γa → ∥2 }, a = arg min{∥ → y − Ca
5. New prediction mode for lossless HEVC
(5)
and the regularized solution is computed according to
→ a = (C T C + Γ T Γ )−1 (C T→ y ),
In order to efficiently compress the volumetric medical images, a two-stage prediction method is proposed.
(6)
where Γ is the Tikhonov matrix, defined as Γ = αI , with I being the
• The
first stage is a preprocessing stage, with the purpose of performing inter-slice decorrelation, and consists in a slice-wise difference, also referred to as DPCM, as applied in [31]. Since images with higher bit depth inherently present more noise, the histogram of the residue images resulting from DPCM is more
Fig. 6. Alternative filter support and training window as proposed in [14].
Fig. 7. Usage of HEVC Intra modes in descending order.
5
Signal Processing: Image Communication (xxxx) xxxx–xxxx
A.F.R. Guarda et al.
identity matrix and α the regularization parameter. A set of experimental tests were performed in order to determine this parameter. These tests show that small variations of α do not have a significant impact in the results, with 0.1 ≤ α ≤ 5 producing the best results.
CALIC which allows encoding of images with bit depth up to 16, as well as the H.264/AVC and a standard version of HEVC with Range Extensions. The compression results are presented in Table 3, in bitsper-pixel (bpp). The first four columns show the performance of the lossless image encoders: JPEG2000, JPEG-LS, CALIC and MRP. JPEG2000 has the worst performance, which can be explained by the use of a wavelet transform instead of a pixel-wise predictor, as in the other three encoders. It can also be seen that the more advanced the prediction method, the better the results obtained, with MRP outperforming both CALIC (GAP) and JPEG-LS (MED). The next three columns present the results for a different implementation of 2D CALIC as well as its extensions 3D CALIC and M-CALIC. Comparing to the original CALIC, this implementation of 2D CALIC shows slightly worse results, while the 3D CALIC shows a great improvement due to its ability to perform interslice prediction, and as for M-CALIC, a much smaller gain is achieved. The next two columns present the results for JP3D, which are similar to those of 3D CALIC, and for H.264/AVC with IPPP coding structure, which compared to MRP presents a worse compression ratio, despite using inter-frame prediction, thanks to the effectiveness of MRP prediction technique. The performance of HEVC with Range Extension can also be compared in Table 3. It can be seen that the Range Extension profile in HEVC for lossless coding provides the best performance, with a significant advantage over the remaining encoders. The last column shows that the proposed LSP mode achieves a significant reduction of bitrate, from 1.470 bpp to 1.267 bpp (i.e., 13.8% on average), in comparison to the RExt profile of HEVC. This major improvement of the encoding efficiency is mainly due to the inter-slice prediction efficiency of the proposed method, since it better exploits the redundancy of the volumetric medical images. When compared to DICOM recommended encoders, such as JPEG-LS, HEVC with the proposed method presents a much higher encoding efficiency, reducing the bitrate from 2.290 bpp to 1.267 bpp (i.e., 44.7%). To better evaluate the impact of each of the techniques used in the proposed LSP mode, we performed some tests by enabling them incrementally. The results of these experiments, presented in Table 4, show that when the LSP is introduced in HEVC, using a straightforward implementation in HEVC with RExt profile, only a marginal gain is obtained (0.1%). However, when adapting LSP for lossless coding using the original pixel values for the filter support this gain becomes much more significant (5.0%). The coding efficiency is further enhanced after extending the LSP filter support to a previous and a following slices (bitrate reduction of 13.1%). Finally, using Tikhonov regularization increases the bitrate savings to 13.8%, when comparing to the unmodified HEVC with RExt profile.
5.2. Inter slice improvements to LSP As pointed out before, due to the characteristics of volumetric images such as MR and CT, and sequences such as X-ray Angiography images, in which human organs, tissues and bones are represented, adjacent slices are highly correlated. To take advantage of this similarity, the LSP support was expanded to include pixels from the previous and the following slices, allowing the prediction accuracy to be improved. This extended support is used only in non-Intra slices, with the previous slice being the first one from Reference_List_0, and the following slice being the first one from Reference_List_1 of HEVC. This approach adapts LSP to volumetric datasets by exploiting the reference frame structure of HEVC, leading to an increased overall coding efficiency. 6. Experimental results The proposed lossless LSP mode was implemented into the HEVC reference software and its performance was evaluated using the previously described medical image datasets. The latest release of the HEVC reference software HM 16.9 was used, with the Intra Main profile, and the Random Access Main and RExt profiles. To guarantee lossless coding, QP was set to 0, and both TransquantBypassEnableFlag and CUTransquantBypassFlagForce were set to 1. For the RExt profile, the parameter CostMode was set to lossless, while the default values were used for the remaining configuration parameters. For all the other encoders, the default configurations for lossless encoding were used. The proposed LSP mode uses a filter support which was empirically determined after testing several geometries for various images. The best results were achieved for a triangular template with 12 pixels in the current slice, and a diamond shaped template with five pixels centred in the pixel co-located pixel of the previous and next slices, as depicted in Fig. 8 by the dashed line. For simplicity, the index s is also omitted in this Figure. Using more than five pixels does not improve the prediction performance and significantly increases the computational complexity, therefore the five pixel region was adopted as the best choice. The training window size was set to T=7. 6.1. 8 bit depth images A first test was performed for images with eight bits per pixel (CIPR dataset [20]), where the encoding performance of the second stage of the proposed method is compared to DICOM recommended encoders, JPEG2000 and JPEG-LS, the extension for volumetric images JP3D, the state-of-the-art lossless image encoders MRP and CALIC, extensions for multi-spectral images 3D CALIC and M-CALIC and also a version of 2D
6.2. 12/16 bit depth images The performance of the second stage of the proposed method was also evaluated and compared with the other encoders for images with
Fig. 8. Extended LSP filter support for volumetric images: each additional slice provides the 5 pixels shown within the dashed line.
6
Signal Processing: Image Communication (xxxx) xxxx–xxxx
A.F.R. Guarda et al.
Table 3 Lossless coding performance (bpp) using the 8 bpp dataset [20]. Sequence
JPEG 2000
JPEG LS
CALIC
MRP
2D CALIC
3D CALIC
M CALIC
JP3D
H.264 IPPP
HEVC RExt
Proposed LSP
CT_Aperts CT_carotid CT_skull CT_wrist MR_liver_t1 MR_liver_t2e1 MR_ped_chest MR_sag_head
1.261 2.019 2.991 1.757 3.256 2.572 3.021 2.905
1.058 1.778 2.761 1.627 3.160 2.418 2.937 2.582
0.998 1.684 2.628 1.550 3.022 2.269 2.789 2.519
0.775 1.374 2.329 1.173 2.582 1.722 2.337 2.279
1.137 1.728 2.755 1.697 3.086 2.306 2.868 2.626
0.994 1.517 2.153 1.127 2.323 1.828 1.860 2.329
1.205 1.800 2.585 1.435 2.832 2.179 2.213 2.605
0.941 1.547 2.088 1.238 2.356 1.745 2.071 2.160
0.892 1.644 2.207 1.421 2.813 2.155 1.982 2.196
0.728 1.425 1.766 1.002 2.052 1.509 1.534 1.748
0.619 1.201 1.558 0.826 1.762 1.192 1.364 1.616
Average
2.473
2.290
2.183
1.821
2.275
1.766
2.107
1.768
1.914
1.470
1.267
6.3. Inter-slice decorrelation
Table 4 Breakdown of the improvement achieved by the proposed LSP mode. HEVC RExt
LSP
w/ Original Pixels
Extended Support
Tikhonov Regularization
1.470
1.468 (-0.1%)
1.396 (-5.0%)
1.277 (-13.1%)
1.267 (-13.8%)
Since JPEG2000 and JPEG-LS do not exploit the inherent inter-slice redundancy, it is expected that their performance falls short of other encoders such as HEVC. However, Part 2 of JPEG2000 includes multicomponent transformations, designed for multi-spectral and volumetric images, and is also recommended by DICOM [4]. The included multicomponent transformations are 5/3 Reversible Wavelet Transform (RWT), Reversible Haar Transform (RHAAR), Reversible Karhunen Loève Transform (RKLT) and DPCM, which corresponds to the first stage of the proposed method. The executable files and scripts to perform these transformations were provided by [21]. The transformations were applied to all test datasets as a preprocessing step, and the resulting images were compressed with all previous encoders which allow 16 bit depth, even for encoders that already present inter-slice prediction such as HEVC and the proposed LSP method. The compression results for each transformation and the original sequences are presented in Table 6, with the average results for each dataset shown in bpp. For each encoder and each dataset, the best transformation results are underlined, and the best overall results are also in bold. It can be seen that the use of multi-component transformations greatly improves the performance of JPEG2000, JPEG-LS and 2D CALIC, for the first two datasets, with a small advantage for 2D CALIC, with DPCM achieving the best results for 8 bit depth images and RKLT for X-ray Angiography images. The performance of 3D CALIC and MCALIC, which already exploit inter-slice redundancy, is severely deteriorated for all transformations. As for HEVC and the proposed LSP technique, a significant gain is obtained for 8 bit depth images using the first stage DPCM, while for the remaining datasets, the best ones are still obtained when encoding the original sequences. Comparing the proposed LSP with the remaining encoders, it can be seen that it still presents the best results for all datasets.
higher bit depth, i.e., 12 or 16, using the datasets available from [21,22] and [22]. The following encoders support images with bit depths up to 16: JPEG2000, JPEG-LS, the new implementations of 2D CALIC and extensions, 3D CALIC and M-CALIC, and also a standard version of HEVC with and without Range Extension. For this dataset, some additional configuration parameters of HEVC were changed, besides the previously presented ones. Namely, the chroma format was set to 4:0:0, the bit depth was set to 12, and the “monochrome16” profile was used. Also, to allow encoding data of over 10 bit depth, the reference software was compiled with the macro “RExt__HIGH_BIT_DEPTH_SUPPORT” set to 1. The compression results are presented in Table 5, in bits-per-pixel (bpp). It can be seen that JPEG2000 and JPEG-LS present similar results, with 2D CALIC achieving a slightly better compression ratio. However, the extensions to 2D CALIC did not improve the results, even though they exploit the redundancy between slices. As for HEVC, the use of Range Extension allowed a significant gain, and the proposed LSP mode improved HEVC RExt even further, with a bitrate reduction from 5.954 bpp to 5.848 bpp (i.e., 1.8%) for X-ray sequences, and from 5.036 bpp to 4.803 bpp (i.e., 4.6%) for CT and MR scans, presenting the best overall performance. These results verify the same tendency as those obtained for the sequences with eight bit depth, from Table 3, however, with smaller gains.
Table 5 Lossless performance comparison of the proposed LSP mode with HEVC and other state-of-the-art encoders for the 12/16 bpp datasets (in bpp). Sequence
JPEG 2000
JPEG LS
2D CALIC
3D CALIC
M CALIC
HEVC
HEVC RExt
Proposed LSP
Xray_30584065 Xray_39068112 Xray_39468655 Xray_46939122 Xray_46948506 Xray_51166511 Xray_51741424
6.027 6.055 6.479 6.243 6.109 6.409 5.962
5.943 6.011 6.462 6.256 6.121 6.350 5.973
5.868 5.909 6.336 6.135 6.014 6.236 5.858
5.846 5.913 6.352 6.216 6.007 6.179 5.948
5.915 5.901 6.334 6.186 5.973 6.157 5.954
6.019 6.257 6.690 6.397 6.144 6.542 6.201
5.649 5.929 6.193 6.050 5.837 6.133 5.885
5.530 5.769 6.103 6.005 5.767 6.011 5.749
Average
6.183
6.159
6.051
6.066
6.060
6.322
5.954
5.848
CT_noncon_chest CT_sag_chest MR_kidney_t1 MR_kidney_t2
5.301 3.530 6.145 6.518
5.373 3.935 5.602 5.959
5.193 3.647 5.729 6.091
5.027 4.073 5.830 6.173
5.430 4.438 5.917 6.215
5.303 4.559 8.242 9.053
4.907 3.761 5.555 5.921
4.475 3.281 5.547 5.909
Average
5.374
5.217
5.165
5.276
5.500
6.789
5.036
4.803
7
Signal Processing: Image Communication (xxxx) xxxx–xxxx
A.F.R. Guarda et al.
Table 6 Lossless performance comparison of the proposed method with HEVC and other state-of-the-art encoders when using multi-component transformations (in bpp). JPEG2000
JPEG LS
2D CALIC
3D CALIC
M CALIC
HEVC
HEVC RExt
Proposed LSP
Original
CT/MR (8 bpp) X-ray (12 bpp) CT/MR (12/16 bpp)
2.473 6.183 5.374
2.290 6.159 5.217
2.275 6.051 5.165
1.766 6.066 5.276
2.107 6.060 5.500
1.614 6.322 6.789
1.470 5.954 5.036
1.267 5.848 4.803
DPCM
CT/MR (8 bpp) X-ray (12 bpp) CT/MR (12/16 bpp)
1.729 6.206 5.372
1.767 6.227 5.264
1.689 6.104 5.186
2.250 6.983 5.605
7.298 8.557 6.901
1.329 6.505 7.061
1.277 6.146 5.218
1.155 5.961 4.935
RWT
CT/MR (8 bpp) X-ray (12 bpp) CT/MR (12/16 bpp)
1.751 6.011 5.515
1.791 6.032 5.425
1.734 5.915 5.409
2.525 7.042 5.876
6.911 8.068 7.393
1.665 6.317 7.296
1.589 6.008 5.419
1.466 5.916 5.132
RHAAR
CT/MR (8 bpp) X-ray (12 bpp) CT/MR (12/16 bpp)
1.971 6.008 5.401
2.031 6.026 5.289
1.925 5.919 5.259
2.942 7.088 5.783
7.722 8.112 7.298
1.897 6.298 7.119
1.785 5.965 5.297
1.590 5.876 5.013
RKLT
CT/MR (8 bpp) X-ray (12 bpp) CT/MR (12/16 bpp)
2.289 5.946 7.074
2.359 5.982 6.948
2.303 5.879 7.025
3.034 6.891 7.367
8.734 8.676 8.724
2.173 6.260 9.534
2.099 5.958 6.983
1.979 5.887 6.649
LSP mode. We have improved LSP for lossless encoding by using the original causal pixels, instead of the previously determined prediction values, to predict the following ones. This method has also been extended to exploit inter-slice redundancy using the nearest previously encoded images for LSP prediction, thus increasing its efficiency for the compression of medical images such as CT, MR and X-ray Angiography images. To increase the robustness of the linear filter parameters estimation, a regularization has been included in the algorithm. The experimental results show that the proposed method achieves a considerable improvement of the compression gain, with bitrate reductions of approximately 13.8% for 8 bpp CT and MR images, 1.8% for 12 bpp X-ray Angiography images, and 4.6% for 12/16 bpp CT and MR images, when comparing to the HEVC RExt profile. When comparing to DICOM recommended encoders, a gain of over 44% for 8 bpp CT and MR images, 5% for 12 bpp X-ray Angiography images, and 8% for 12/16 bpp CT and MR images is obtained.
In [31], the MRP was optimised for compression of volumetric medical images, by exploiting inter-slice redundancy and also including DPCM preprocessing stage. The results obtained in [31] are a little better than those of the proposed LSP method based on HEVC. This can be justified by the fact that HEVC was developed mainly for lossy video encoding, requiring more overhead bits than MRP, which was specifically developed for lossless coding of still images. Nevertheless, the slightly higher efficiency of MRP is obtained with around three times higher computational time than the current proposal. 6.4. Computational complexity As can be easily inferred from the description of the LSP-based method presented in Sections 4 and 5, the computational complexity of standard HEVC encoders and decoders is significantly increased by the LSP algorithm. Although in typical medical image compression applications real-time encoding is not always required and so the encoding complexity may not be a very serious constraint, the decoding complexity can be a problem due to the response times expected from image rendering devices. To evaluate the approximate order of magnitude of the complexity increase, a set of experiments were performed on a PC with an Intel Xeon CPU E3-1240 V2 @ 3.40 GHz and 24 GB of RAM, running Ubuntu 16.04. The average processing time per slice of the proposed LSP method (in seconds) was compared with that of the HEVC reference software with the RExt profile, for encoding and decoding all datasets. For the encoder, the proposed method consumes around 30 times more than the original HEVC, while for the decoder the processing time can rise up to few hundred times higher. Since none of the implementations was optimized for speed, it is reasonable to assume that such results may be brought down to acceptable levels, particularly taking into account that the complexity of HEVC encoding can be easily reduced about 70% [35]. Therefore, there is plenty of room for algorithmic optimization and improvement of the proposed method. This will be addressed in future research efforts, namely by investigating new techniques to reduce the complexity of the decoder through the use of side information or implicit coding modes, capable of avoiding exhaustive training of the LSP coefficients in the decoder side.
Acknowledgements This work is a part of the UDICMI project funded by CENTRO-07ST24-FEDER-002022 of QREN and project Medicomp in the scope of R & D Unit 50008, Grant no. PEst-OE/EEI/LA0008/2013, financed by the applicable financial framework (FCT/MEC through national funds and when applicable co-funded by FEDER-PT2020 partnership agreement). The authors would like to thank Martin Briano for providing his implementation [36] of 2D CALIC, 3D CALIC and M-CALIC. References [1] G. Sullivan, J. Ohm, W.-J. Han, T. Wiegand, Overview of the high efficiency video coding (HEVC) standard, IEEE Trans. Circuits Syst. Video Technol. 22 (12) (2012) 1649–1668. [2] J. Taquet, C. Labit, Hierarchical oriented predictions for resolution scalable lossless and near-lossless compression of CT and MRI biomedical images, IEEE Trans. Image Process. 21 (5) (2012) 2641–2652. [3] R. C. of Radiologists, The adoption of lossy image data compression for the purpose of clinical interpretation, version 1.1, April 2008 (Jan 2017). URL 〈https://www. rcr.ac.uk/it-guidance-adoption-lossy-image-data-compression-purpose-clinicalinterpretation〉. [4] DICOM, Digital Image and Communications in Medicine, Strategic document, 2014. [5] M. Weinberger, G. Seroussi, G. Sapiro, The LOCO-I lossless image compression algorithm: principles and standardization into JPEG-LS, IEEE Trans. Image Process. 9 (8) (2000) 1309–1324. [6] Information technology: JPEG 2000 image coding system, part 1: Core coding system, document 15444-1, ISO/IEC, 2000. [7] G. J. Sullivan, P. N. Topiwala, A. Luthra, The H.264/AVC advanced video coding standard: overview and introduction to the fidelity range extensions, in:
7. Conclusion In this paper we propose to improve the HEVC lossless compression tools for volumetric medical images and sequences, by using a modified 8
Signal Processing: Image Communication (xxxx) xxxx–xxxx
A.F.R. Guarda et al.
[8] [9]
[10]
[11] [12]
[13] [14]
[15]
[16] [17]
[18]
[19]
[20] [21]
[22] N. B. I. Archive, Subject GSM714044, Jan 2017. URL 〈https://imaging.nci.nih.gov/ ncia/login.jsf〉. [23] L. Xie, M.A. Sparks, W. Li, Y. Qi, C. Liu, T.M. Coffman, G.A. Johnson, Quantitative susceptibility mapping of kidney inflammation and fibrosis in type 1 angiotensin receptor-deficient mice, NMR Biomed. 26 (12) (2013) 1853–1863. URL 〈http:// www.civm.duhs.duke.edu/lx201204/〉. [24] G. Sullivan, J. Boyce, Y. Chen, J.-R. Ohm, C. Segall, A. Vetro, Standardized extensions of high efficiency video coding (HEVC), IEEE J. Select. Top. Signal Process. 7 (6) (2013) 1001–1016. [25] M. Zhou, W. Gao, M. Jiang, H. Yu, HEVC lossless coding and improvements, IEEE Trans. Circuits Syst. Video Technol. 22 (12) (2012) 1839–1843. [26] V. Sanchez, P. Nasiopoulos, R. Abugharbieh, Efficient lossless compression of 4-d medical images based on the advanced video coding scheme, IEEE Trans. Inform. Technol. Biomed. 12 (4) (2008) 442–446. [27] E. Wige, G. Yammine, P. Amon, A. Hutter, A. Kaup, Pixel-based averaging predictor for HEVC lossless coding, in: Proceedings of IEEE International Conference on Image Processing (ICIP), 2013, pp. 1806–1810. [28] E. Wige, G. Yammine, P. Amon, A. Hutter, A. Kaup, Sample-based weighted prediction with directional template matching for HEVC lossless coding, Picture Coding Symposium (PCS), 2013, pp. 305–308. [29] V. Sanchez, Sample-based edge prediction based on gradients for lossless screen content coding in HEVC, in: Proceedings of Picture Coding Symposium (PCS), 2015, pp. 134–138. [30] V. Sanchez, Lossless screen content coding in HEVC based on sample-wise median and edge prediction, in: Proceedings of IEEE International Conference on Image Processing (ICIP), 2015, pp. 4604–4608. [31] J. Santos, A. Guarda, N. Rodrigues, S. de Faria, Contributions to lossless coding of medical images using minimum rate predictors, in: Proceedings of IEEE International Conference on Image Processing (ICIP), 2015, pp. 2935–2939. [32] R.I. Chernyak, Analysis of the intra predictions in H.265/HEVC, Appl. Math. Sci. 8 (148) (2014) 7389–7408. [33] A.-I. N. Tikhonov, V. Y. Arsenin, Solutions of Ill Posed Problems (Scripta series in mathematics), Vh Winston, 1977. [34] I. Rish, G. Grabarnik, Sparse Modeling: Theory, Algorithms, and Applications, CRC Press, Inc., Boca Raton, FL, USA, 2015. [35] G. Correa, P. Assuncao, L. Agostini, L.A. da Silva Cruz, Complexity-Aware High Efficiency Video Coding, Springer International Publishing, Switzerland, 2016. [36] M. Briano, CALIC, 3D CALIC and M-CALIC customizable software implementation, July 2014. URL 〈https://prezi.com/j2ichhip4ls7/calic-implementation/〉.
Proceedings of SPIE Conference on Applications of Digital Image Processing XXVII, 2004, pp. 454–474. X. Wu, N. Memon, Context-based, adaptive, lossless image coding, IEEE Trans. Commun. 45 (4) (1997) 437–444. I. Matsuda, Y. Umezu, N. Ozaki, J. Maeda, S. Itoh, A lossless coding scheme using adaptive predictors and arithmetic code optimized for each image, Syst. Comput. Jpn. 38 (4) (2007) 1–11. P. Schelkens, A. Munteanu, A. Tzannes, C. Brislawn, JPEG2000 – Part 10. Volumetric data encoding, in: Proceedings of IEEE International Symposium on Circuits and Systems, ISCAS Proceedings, 2006, pp. 3877–3880. X. Wu, N. Memon, Context-based lossless interband compression-extending CALIC, IEEE Trans. Image Process. 9 (6) (2000) 994–1001. E. Magli, G. Olmo, E. Quacchio, Optimized onboard lossless and near-lossless compression of hyperspectral data using CALIC, IEEE Geosci. Remote Sens. Lett. 1 (1) (2004) 21–25. X. Li, M. Orchard, Edge-directed prediction for lossless compression of natural images, IEEE Trans. Image Process. 10 (6) (2001) 813–817. D. Graziosi, N. Rodrigues, E. da Silva, S. de Faria, M. de Carvalho, Improving multiscale recurrent pattern image coding with least-squares prediction mode, in: Proceedings of IEEE International Conference on Image Processing (ICIP), 2009, pp. 2813–2816. L. Lucas, N. Rodrigues, E. da Silva, S. de Faria, Adaptive least squares prediction for stereo image coding, in: Proceedings of the IEEE International Conference on Image Processing (ICIP), 2011, pp. 2013–2016. X. Li, Least-square prediction for backward adaptive video coding, EURASIP J. Appl. Signal Process. 2006 (2006) 126. SiemensHealthcare, Computed tomography, Jan 2017. URL 〈http://www. medicalradiation.com/types-of-medical-imaging/imaging-using-x-rays/computedtomography-ct/〉. SiemensHealthcare, Magnetic resonance imaging, Jan 2017. URL 〈http://www. medicalradiation.com/types-of-medical-imaging/other-types-of-medical-imaging/ magnetic-resonance-imaging/〉. C. Cavaro-Menard L. Zhang P. le Callet. Diagnostic quality assessment of medical images: challenges and trends, in: Proceedings of European Workshop on Visual Information Processing (EUVIP), 2010, pp. 277–284. C. for Image Processing Research, Medical images datasets, Jan 2017. URL 〈http:// www.cipr.rpi.edu/resource/sequences/sequence01.html〉. Z. Xu, J. Bartrina-Rapesta, I. Blanes, V. Sanchez, J. Serra-Sagrist, M. Garca-Bach, J. F. Muoz, Diagnostically lossless coding of X-ray angiography images based on background suppression, Computers & Electrical Engineering, 2016.
9