Inverse color to black-and-white halftone conversion via dictionary learning and color mapping

Information Sciences 299 (2015) 1–19

Contents lists available at ScienceDirect

Information Sciences journal homepage: www.elsevier.com/locate/ins

Inverse color to black-and-white halftone conversion via dictionary learning and color mapping Chang-Hwan Son ⇑, KangWoo Lee, Hyunseung Choo College of Information and Communication Engineering, Sungkyunkwan University, 300 Chunchundong, Jangangu, Suwon 440-746, South Korea

a r t i c l e

i n f o

Article history: Received 9 August 2014 Received in revised form 27 November 2014 Accepted 2 December 2014 Available online 9 December 2014 Keywords: Reversible color to gray mapping Halftoning Dictionary learning Color embedding

a b s t r a c t This paper challenges the problem of estimating the original red–green–blue (RGB) image from a black-and-white (B&W) halftone image with homogeneously distributed dot patterns. To achieve this goal, training RGB images are converted into color-embedded gray images using the conventional reversible color to gray conversion method, and then converted into halftone images using error diffusion in order to produce the corresponding B&W halftone images. The proposed method is composed of two processing steps: (1) restoring the color-embedded gray image from an input B&W halftone image using a sparse linear representation between the image patch pairs obtained from the images and (2) restoring the original colors from the color-embedded gray image using the reversible color to gray conversion and linear color mapping methods. The proposed method successfully demonstrates the recovery of colors similar to the originals. The experimental results indicate that the proposed method outperforms the conventional methods. It is suggested that our method is not only successfully applied for the color recovery of the B&W halftone image, but that it can also be extended to various applications including color restoration of printed image, hardcopy data hiding, and halftone color compression. Ó 2014 Elsevier Inc. All rights reserved.

1. Introduction Color is useful as a means of enhancing and emphasizing the visual information in digital images including pictures, illustrations, documents, presentation slides, etc. It is also an important factor influencing the visual attraction of such images. Colors are usually designed with graphics or office software programs, or they can be captured by digital imaging devices such as cameras or scanners. In most cases, each color is expressed as a three-dimensional (3D) vector whose elements contain 256 discrete levels, especially in the RGB color space [25]; however, color often needs to be converted to a gray scalar quantity for many purposes including black-and-white printing, data transmission via a mono fax, design of an algorithm in image processing and computer vision, and artistic rendering [42]. Since this color to gray conversion reduces the dimensionality, various colors can be mapped to the same gray; hence, the reverse color to gray conversion that estimates the original colors from the gray color in an image is an ill-posed problem. Even though a possible effective solution is the use of a new color transferring method [51] or colorization method [24,50] that assigns realistic pseudo colors to the grays according to the structure and object similarity between a reference color image and an input gray image, such methods are limited in recovering the unknown original colors due to unstable spatial

⇑ Corresponding author. Tel.: +82 31 299 4642; fax: +82 31 299 4134. E-mail address: [email protected] (C.-H. Son). http://dx.doi.org/10.1016/j.ins.2014.12.002 0020-0255/Ó 2014 Elsevier Inc. All rights reserved.

2

C.-H. Son et al. / Information Sciences 299 (2015) 1–19

and luminance matching between the two images. In addition, the same objects or species in an image may have different colors, and thus no reference color image that is perfectly matched to a unique gray image would be found. However, recently developed reversible color to gray conversion methods [9,29] have been successful in recovering the original colors from the gray image. These methods are based on color extraction from the embedded chrominance images stored in the high-frequency wavelet sub-bands. A relatively large amount of information (e.g., two resized chrominance images) can be encoded as pleasant and less visible texture patterns that are highly correlated with image contents. These reversible color to gray conversion methods have provided a breakthrough in original color estimation from a gray image. Nevertheless, the more challenging problem remains in estimating the original colors from a B&W halftone image with homogeneous dot patterns [5], a type of image widely used in digital printers, copiers, and fax machines. This problem is generalized to finding the 3D vector with 256 discrete levels from a scalar quantity with two discrete levels per pixel position, and thus it is a highly ill-posed problem. In some applications, it is more important to recover the original colors from the B&W halftone image than from the gray image with 256 discrete levels. In B&W printing, digital halftoning [16,27] is conducted after the color to gray conversion to produce a B&W halftone image. The B&W halftone image is used to determine whether or not the black toner particles will touch the surface of the photoconductor drum in laser printers, whereas inkjet printers use it to determine the spatial position of the black ink drops on the paper. Regardless of the type of printed materials, such as printouts, newspapers or books, the printed images are binary images composed of homogeneous dot patterns. In the binary images, the printed areas with black toner or ink represent the logical value 1, and the unprinted areas represent the logical value 0. Even when a color image is converted into a B&W halftoned image in order to reduce consumption of expensive color inks or toners, recovery of the original colors is often required. In addition, the B&W halftone image transmitted via a monochromatic fax can be stored in the memory of the client’s fax machine or multifunctional printer (MFP). In this case, it is very useful to directly recover the original colors from the B&W halftone image. This inverse color to B&W halftone conversion is widely applicable for practical purposes, including hardcopy data hiding and halftone image compression.

2. Background and motivation Conventional color to gray conversion methods [3,18,19] have attempted to preserve salient chromatic structures, color edges, and contrast, thereby offering salient information. However, reversible processing has not been considered in previous studies. Reversible color to gray conversion algorithms have been developed in recent years [6,8,23,26,28,36]. A basic algorithm was introduced by de Queiroz and Braun [9] and involves smoothly mapping colors to high-frequency textures. The procedure of embedding the color information into a gray image is illustrated in Fig. 1. First, the RGB image is converted into the YCbCr image, and the discrete wavelet transform (DWT) is then applied to the Y image. To embed the color information (Cb and Cr images) in the gray image (Y image), the down-sampled Cb and Cr images are copied into the HL and LH subbands, respectively. The color-embedded gray image is finally obtained through the inverse discrete wavelet transform (IDWT). To verify how the embedded colors in the subbands are encoded in the gray image, the region marked with a red rectangle is magnified in the next row. Compared to the original gray patch extracted from the Y image, the embedded colors are encoded as less visible textures. This is possible due to the characteristics of the human visual system (HVS) that are insensitive to high frequencies [5,37]. However, since de Queiroz’s method embeds the color information into HL and LH subbands, the textures tend to be vertical or horizontal lines. To reduce the texture visibility, Ko et al. [29] used a discrete wavelet packet transform (DWPT), providing more decomposed subbands. In their method, the color information is inserted into two subbands with a minimum amount of information, i.e., the horizontal subband of a vertical subband and the vertical subband of a horizontal subband. As shown in Fig. 1, the texture visibility can be significantly reduced by Ko’s method. Despite the existence of other types of color-embedding methods [8,23,36], the goal of this paper is not to evaluate the performance of the methods and select the best one. Thus, one color-embedding method is sufficient to verify the effectiveness of the proposed method. The procedure for the color decoding method for extracting the original Cb and Cr images from the color-embedded gray image is in the inverse order of the color embedding method. A more detailed algorithm can be found in [9,29]. As mentioned earlier, the estimation of the original colors from the B&W halftone image, rather than the gray image, is required in some applications. In Fig. 1, if the color-embedded gray image is halftoned through digital halftoning, most textures indicating the color information, i.e., Cb and Cr images, will be destroyed, and a new original color estimation will therefore be needed from the B&W halftone image where color information loss has already occurred. Three B&W halftone images corresponding to the original gray image and two color-embedded gray images generated by de Queiroz’s and Ko’s methods are given in the third row of Fig. 1. The B&W halftone patterns corresponding to the color-embedded gray image with de Queiroz’s method are not homogeneous due to the encoded vertical or horizontal textures. This contradicts the halftone design principle [5,16]. In addition, the horizontal and vertical line textures can be seen on the restored color images, which can degrade the visual quality. On the other hand, the B&W halftone patterns of the color-embedded gray image generated by Ko’s method are similar to those of the original gray image due to the less visibly encoded textures. Therefore, it is preferable to use the homogeneously distributed halftone patterns for original color estimation. As shown in the third row of Fig. 1, the B&W halftone images are binary images with black and white dots, and thus estimating the original RGB image (3D vector) from the B&W halftone image (scalar quantity) is an ill-posed problem.

C.-H. Son et al. / Information Sciences 299 (2015) 1–19

3

Fig. 1. Illustration of reversible color to gray algorithm and its related results. (For interpretation of the references to color in this figure legend, the reader is referred to the web version of this article.)

In this paper, the inverse color to B&W halftone conversion is composed of two steps. The first step involves recovering the color-embedded gray image from an input B&W halftone image, and the second step involves estimating the original colors from the recovered color-embedded gray image. The first step is similar to inverse halftoning [52] because a gray image is reconstructed from its halftone version. However, conventional inverse halftoning methods do not deal with the issue at hand, i.e., original color estimation from a B&W halftone image. To verify this idea, state-of-the-art conventional inverse halftoning methods [35,38,43,48] are used to conduct the first step, and the conventional reversible color to gray conversion method is used for the second step, thereby producing the recovered color images. Fig. 2 shows that the recovered colors from the ‘flower’ images are significantly desaturated and inaccurate compared to the original colors shown in Fig. 1. This is because the conventional inverse halftoning methods focus on removing the noisy halftone patterns on flat regions and enhancing the edge expression for high-quality image reconstruction. Therefore, a different approach that is more targeted to the estimation of the original colors from a B&W halftone image should be considered. Specifically, the relation between the color-embedded textures and the corresponding halftone patterns needs to be established. In addition, a color enhancement algorithm needs to be developed to correct the initially recovered colors. In this paper, the relation will be modeled via the dictionary learning method, while the color mapping method based on linear regression will be introduced for the recovered color correction. Other processing, such as inverse halftoning and median filtering, will be used to reduce unwanted artifacts. It should be noted that our approach focuses on the direct recovery of the original colors from the B&W halftone patterns. In contrast, the reversible color to gray conversion methods cannot directly recover the original colors from the halftone patterns without additional information and resolution conversion, which will be discussed later in terms of our experimental results. 3. Proposed method Fig. 3 shows the block-diagram of the proposed algorithm. The initial RGB color image is recovered from an input B&W halftone image via dictionary-based color-embedded gray reconstruction and color recovery with reversible color to gray

4

C.-H. Son et al. / Information Sciences 299 (2015) 1–19

Fig. 2. Recovered ‘flower’ images with conventional inverse halftoning methods; (a) IIH-TED [43], (b) WInHD [38], (c) MAP [48], and (d) LUT [35].

Fig. 3. Block-diagram of the proposed algorithm.

conversion; the initially recovered color image is then converted into the L⁄a⁄b⁄ opponent color space. For the a⁄ and b⁄ images, median filtering is applied to remove the undesirable color noises, and the L⁄ image is directly replaced by the inversely halftoned image from the input B&W halftone image to increase the overall sharpness. Finally, the color mapping method based on the linear regression model fitted to the training image sets is applied to the corrected L⁄a⁄b⁄ images to adjust their colors, and the backward L⁄a⁄b⁄ to RGB color conversion is then conducted to obtain the final RGB image. 3.1. Color estimation The proposed original color estimation is carried out in two steps, as previously mentioned. In the first step, the relation between the color-embedded textures and the corresponding B&W halftone patterns is established in the sparse representation. For the generation of the halftone patterns, error diffusion halftoning [30], which is widely used in digital printers, fax

5

C.-H. Son et al. / Information Sciences 299 (2015) 1–19

machines, and even plasma display panel (PDP) TVs, is adapted. Even though this error diffusion is a non-linear model (because of the use of binary quantization), it can be considered a linear approximation model with a noise term [38]. Depending on the color embedding method, the textures have different patterns such as vertical or horizontal lines (for de Queiroz’s method) and some regular shapes (for Ko’s method). Moreover, it is well-known that natural images can be sparsely represented [1]. Since the color-embedded textures can be regarded as natural images with added textures, it is assumed that the color-embedded textures can be sparsely represented. Based on the observations of the linear error diffusion model and on the sparsity of the natural images, the relation between the color-embedded textures and the corresponding B&W halftone patterns is modeled via a dictionary learning method in this paper. 3.1.1. Dictionary learning Dictionaries, known as textons or visual codewords, are the redundant basis vectors used for sparse linear representation, which has been extensively applied to image denoising [11], facial recognition [53], texture classification [33], scene categorization [41], and image fusion [47]. The basis functions of the discrete cosine transform (DCT) or DWT have been directly used for the dictionaries; however, it has recently been increasingly recommended that the many natural patches extracted from training image sets be learned for improved dictionary generation [1,11]. In this paper, the color-embedded texture patch and the corresponding B&W halftone patch are sparsely represented via the learned dictionaries, as follows.

yc ¼ xc;1  d

c;1

þ xc;2  d þ; . . . ; þxc;K  d

b;1

þ xb;2  d þ; . . . ; þxb;K  d

yb ¼ xb;1  d

c;2

n1

c

b;2

c;K

subject to #ðijxc;i – 0Þ < TH

b;K

ð1Þ

subject to #ðijxb;i – 0Þ < TH

ð2Þ

n1

b

where y 2 R and y 2 R indicate the lexicographically ordered column vectors that include the patches’ pixel data extracted from the color-embedded gray image and the corresponding B&W halftone image, respectively. For your reference, n oK n oK c;i b;i mathematical expressions used in [1,11] have been adopted in this paper. d 2 RnK and d 2 RnK denote the i¼1

i¼1

dictionaries used to represent the color-embedded textures and the B&W halftone patterns, respectively. Certainly, the pffiffiffi pffiffiffi dimension (n) of the dictionary is equal to the patch size of n  n. This means that the dictionary learning is conducted  c;i K  b;i K based on the patch unit. x i¼1 and x i¼1 are the representation coefficients, and # denotes the number of non-zero representation coefficients. TH is a threshold value controlling the sparsity, and it is less than the dimension (n) of the dictionary. To induce the shared representation coefficients, i.e., xi = xc,i = xb,i, (1) and (2) can be modified, as follows.

 y¼

yc yb



" ¼ x1 

d d

c;1

b;1

#

" þ x2 

d

c;2

b;2

d

# þ; . . . ; þxK 

"

d d

c;K

b;K

#

 ¼ Dx ¼

 subject to kxk0 < TH b x

Dc D

ð3Þ

where D 2 R2nK is the dictionary matrix, which can be decomposed into [Dc,T, Db,T]T; T is the transpose operator and n oK c;i Dc ¼ d 2 RnK includes the dictionary vectors to represent the color-embedded texture patch, whereas i¼1 b;i K

K

Db ¼ fd gi¼1 2 RnK contains the dictionary vectors used for the halftone patch representation. xT ¼ fxi gi¼1 2 R1K is the representation coefficient vector. In (3), it is noteworthy that two dictionary submatrices, Dc and Db, have a shared representation coefficient vector, x, and this differs from (1) and (2). This means that the column vector of the dictionary matrix D, that is, [dc,i,Tdb,i,T]T, is treated as a new dictionary vector for dictionary learning. To obtain the [yc,Tyb,T]T measurement data in (3), many natural images with different colors are downloaded from websites to ensure the various color-embedded textures, and Ko’s reversible color to gray conversion method [29] is then implemented to create the color-embedded gray images. The color-embedded texture patches are extracted from the color-embedded gray images using uniform sampling or feature detection, and the corresponding B&W halftone patches are then created using Floyd–Steinberg error diffusion [30]. Each color-embedded texture patch and the corresponding B&W halftone patch are reshaped as lexicographically ordered column vectors and then inserted into yc and yb, respectively, thus forming a matrix Y = {y1, y2, .. , ym}, where m is the extracted total patch number. The ith column vector yi of the matrix consists of [yc,i,T, yb,i,T]T. Given the matrix Y 2 R2nm composed of the measured data, i.e., the color-embedded texture patches and the corresponding B&W halftone patches, the dictionary matrix D 2 R2nK can be trained by minimizing the following cost function.

minkY  DXk22 D;X

subject to 8i;

kxi k0 6 TH

ð4Þ

where X ¼ fx1 ; x2 ; . . . ; xm g 2 R;Km contains the representation coefficient vectors corresponding to each column vector of Y. (4) is the ‘0 -minimization problem and thus can be solved using the well-known K-SVD learning algorithm [1]. In the learning algorithm, the initial dictionary submatrices, Db and Dc, are filled with randomly generated binary data and DCT basis functions, respectively. Error diffusion halftoning can be represented by the linear model with a noise term, as discussed in [38]. Thus, similar to image denoising via sparse representation [11], a sparse regularization term is needed to suppress the noisy dots of the reconstructed color-embedded images. As discussed in [11], the threshold value TH leads to a trade-off between reconstruction accuracy and noise suppression. That is, a small TH value reduces the reconstruction accuracy but suppresses the noise. On the contrary, a large TH value increases the reconstruction accuracy but has little effect on noise

6

C.-H. Son et al. / Information Sciences 299 (2015) 1–19

suppression. Similar effects can be applied to the reconstructed color-embedded gray images. In general, the value of TH should be less than half of the patch dimension [1,11], i.e., the column vector of Y. In this paper, a TH value of 8 is used heuristically. More details about TH can be found in [1,11]. Fig. 4 shows an example of the learned dictionaries. The used total patch number, m, is about 160,000, the extracted patch size is 6  6, and the number of used dictionaries, K, is 1024. To visualize the dictionaries, each column vector of the learned dictionary submatrices, Db and Dc, is reshaped as a block and then normalized to [0–255]. In Fig. 4, each 6  6-sized patch is a visualized dictionary; for example, black patches are positioned at the top-left corners to represent a constant patch. Fig. 4(a) and (b) shows the learned dictionaries for Db and Dc, respectively, which are responsible for the representation of the B&W halftone patterns and color-embedded textures. Thus, it can be observed that the image in Fig. 4(b) has texture, whereas that in Fig. 4(a) features gray patterns similar to the B&W halftone patterns. The yellow circles in Fig. 4(a) and (b) are examples of the gray patterns and textures. Fig. 4(c) shows examples of the training color images containing a wide variety of colors, downloaded from various websites including photo sharing sites such as Flickr. The total number of training color images is 200. From these color images, training patch pairs consisting of color-embedded patches and corresponding B&W halftoned patches were generated. 3.1.2. Color-embedded gray reconstruction Given the learned dictionary matrix D and the input B&W halftone image H, the unknown representation coefficient vectors corresponding to the extracted patches from H can be estimated by minimizing the cost function, defined as

8i minkxi k0 subject to kRi H  Db xi k22 6 e2 xi

ð5Þ

where Ri is a matrix [11] that extracts the ith patch from an input image. In (5), it is assumed that the input B&W halftone image is reshaped as the column vector and then inserted into H; thus, RiH is actually the column vector including the extracted halftoned patch. If the size of the input halftoned image is N  N, the matrix dimensions of H and R are N2  1

Fig. 4. Example of the learned dictionaries; (a) dictionaries for B&W halftoned patches, (b) dictionaries for color-embedded gray patches, and (c) examples of the training color images. (For interpretation of the references to color in this figure legend, the reader is referred to the web version of this article.)

7

C.-H. Son et al. / Information Sciences 299 (2015) 1–19

and n  N2, respectively. Eq. (5) shows that the extracted ith halftone patch, RiH, can be approximated by the linear combination of the learned dictionary submatrix, Db, and the unknown representation coefficient vector, xi, with the bounded representation error, e. In (3), the ith halftone patch, RiH, is repeatedly extracted from the input halftoned image at location i along a raster scanning order, and thus some overlapping areas occur between the extracted patches, RiH. To solve (5), the matching pursuit (MP) algorithm [1,11] is used in this paper. The estimated representation coefficient vector, xi, can be shared with another learned dictionary submatrix, Dc, as shown in (3), and thus the unknown color-embedded gray image can be directly calculated, as follows.

C ¼ minkRi C  Dc xi k22 ¼ C

X i;T i R R

!1

i

X i;T c R D xi

! ð6Þ

i

where C and C⁄ denote the unknown and estimated color-embedded gray images, respectively. The above equation shows that the unknown color-embedded patch, RiC, can be represented by Dcxi, i.e., the linear combination of the dictionary submatrix, Dc, and the representation coefficient vector, xi. The dictionary submatrix Dc is already learned, as shown in Fig. 4(b), and the representation coefficient vector, xi, can be estimated using (5). Thus, Dc xi 2 Rn1 is actually the estimated colorembedded patch; however, RT 2 RN i;T

c

2

n

is needed in order to place Dcxi into the patch region of C⁄ at location i. In other words,

2

2

has a matrix size equal to C 2 RN 1 ; however, only the ith patch region is reconstructed. As already menR D x 2R tioned, RiH is repeatedly extracted along a raster scanning order, and thus some overlapping areas occur between the extracted patches. To complete the color-embedded gray image, the estimated color-embedded patches, Ri,TDcxi, are aggreP gated with overlapping areas, and this is expressed as Ri,TDcxi; the aggregated color-embedded patches are then divided by P the number of aggregated patches, which is described as the diagonal matrix, Ri,TRi [11]. In other words, this indicates that the estimated color-embedded patches are averaged with some overlapping areas. Since simple patch averaging can cause a desaturation effect by reducing the color-embedded textures, the weighing values that are proportional to the variances of the estimated color-embedded patches can be combined with (6). The related equation can be found in another patch-based image restoration method [14]. In a similar way, the reconstructed B&W halftone image can be obtained as follows. i

N 1

H ¼ minkRi H  Db xi k22 ¼ H

X i;T i R R i

!1

X i;T b R D xi

! ð7Þ

i

This equation shows that the reconstructed B&W halftone image, H⁄, can be calculated based on the linear combination of the learned dictionary submatrix, Db, and the shared representation coefficient vector, xi. This is possible because the representation coefficient vector, xi, is shared between two sub-matrices, Db and Dc. This means that the color-embedded gray image and the B&W halftone image can be simultaneously reconstructed with the estimated representation coefficient vector. The halftone patterns of the input B&W halftone image, H, can be compared with those of the reconstructed B&W halftone image, H⁄, as shown in Fig. 5(c) and (d). Algorithm 1 summarizes the proposed color-embedded gray reconstruction from the input B&W halftone image. As shown in Algorithm 1, the proposed color-embedded gray reconstruction is divided into two phases: the training and testing phases. In the training phase, the dictionary pair, Dc and Db, is learned from the pairs of extracted color-embedded patches and the corresponding B&W halftoned patches. In the testing phase, the representation coefficient vector (xi) of the extracted halftone patch (RiH) at the ith pixel location is first estimated by solving (6), and then the color-embedded gray patch is reconstructed via the matrix–vector product, Dcxi, and patch averaging is conducted via (6) to complete the whole image. Algorithm 1. Proposed color-embedded gray reconstruction 1. Training phase  Collect the training images with different colors and then apply the reversible color-to-gray conversion [29] to the training color images to create color-embedded gray images.  Apply the error diffusion [30] to the color-embedded gray images to obtain the corresponding B&W halftoned images.  Extract the color-embedded and B&W halftoned patches from the color-embedded gray images and the corresponding B&W halftoned images at the same pixel positions, and then reshape the color-embedded and B&W halftoned patches as column vectors, respectively, which are stacked and then inserted into the column of the matrix, Y, in (4).  Solve (4) using the K-SVD algorithm [1] to learn the dictionary, D = [Dc,T, Db,T]T 2. Testing phase  Extract the ith halftone patch (RiH) from the input halftone image along the raster scanning order, and then estimate the representation coefficient vector, xi, by solving (5)  Reconstruct the ith color-embedded gray patch via the matrix-vector product, Dcxi, and then conduct patch averaging via (6) to produce the whole image

8

C.-H. Son et al. / Information Sciences 299 (2015) 1–19

Fig. 5. Results of the proposed color estimation for ‘flower’ image; (a) original color image, (b) original color-embedded gray image, (c) original halftone image, (d) recovered halftone image, (e) recovered color-embedded gray image, (f) recovered color image (before applying median and inverse halftoning), (g) recovered color image (after applying median and inverse halftoning), (h) recovered color image (after applying the chroma mapping), and (i) recovered color image (after applying the color mapping). (For interpretation of the references to color in this figure legend, the reader is referred to the web version of this article.)

3.1.3. Original color recovery The use of the two learned dictionaries, Db and Dc, enables the first step to be completed; that is, the color-embedded gray image can be recovered from the input B&W halftone image. The second step involves generating the original color image from the recovered color-embedded gray image. This is simply performed by applying the decoding process of Ko’s reversible color to gray conversion [29]. Fig. 5 shows the results of the proposed color estimation. Fig. 5(a), (b), and (c) shows the original color image, the color-embedded gray image with Ko’s method, and the input B&W halftone image, respectively. Given only the input B&W halftone image, as shown in Fig. 5(c), the representation coefficient vectors can be first estimated via (5), and the reconstructed B&W halftone image, as shown in Fig. 5(d), can then be obtained via (7). In the magnified region at the top-left side, the halftone patterns are similar to those in Fig. 5(c). However, the histogram in Fig. 5(d) shows that the reconstructed B&W halftone image is not a perfect binary image. Nevertheless, a perfect B&W halftone image can be obtained with simple thresholding because its histogram bins are distributed around 0 or 255. Fig. 5(e) shows the recovered color-embedded gray image via (6). Compared to the original textures magnified at the top-left of Fig. 5(b), the recovered textures of Fig. 5(e) indicating the embedded color information are not the same as the original textures. This is the

9

C.-H. Son et al. / Information Sciences 299 (2015) 1–19

limitation of the dictionary learning algorithm used. However, the recovered colors, shown in Fig. 5(f), are significantly improved compared to the recovered colors with the conventional methods, as shown in Fig. 2, where the violet colors cannot be recovered. There is still some room for visual quality improvement. In Fig. 5(f), the recovered colors are desaturated, and their hues need to be adjusted. In addition, in the magnified region, there are inaccurately estimated colors seen as color noise, and the overall sharpness is low. In the proposed method, the approach of decomposing the color estimation into two steps, that is, color-embedded gray reconstruction and original color recovery, is adopted. However, it is possible that the unknown original colors can be directly estimated from an input B&W halftone image. To verify this, the direct estimation is conducted with the newly learned dictionaries that are responsible for the representations of the RGB patch and the corresponding halftone patch. To obtain the learned dictionaries, the same procedure, as in the previous experiment, is conducted, except that each column vector of Y, as shown in (4), should be filled with the values of the RGB patches and the corresponding halftone patches. However, the direct estimation with the newly learned dictionaries cannot recover the original colors because the estimated colors are extremely desaturated. Since the relation between the RGB patch and the corresponding halftone patch is manyto-one mapping, many RGB patches can be mapped into the same halftone patch. This means that the halftone patch can be inversely mapped to the averaged RGB patch, which can be shown as a grayish RGB patch. It seems that the used dictionary learning algorithm needs to be modified to achieve direct estimation. For example, patch clustering according to the RGB colors of training patches may be an effective solution to reduce the desaturation of the estimated colors. Compared to the direct estimation, the proposed two-step approach can reduce the extent of the many-to-one mapping relation. In addition, in the proposed two-step approach, the color-embedded textures can be retained to some degree in the halftone patterns, as shown in Fig. 1. The relation between the color embedded textures and the corresponding halftone patterns can be further established through the two-step approach. 3.2. Color enhancement The raised problems regarding overall sharpness and color noises can be simply solved using the conventional median filter [17] and inverse halftoning method [43]. As shown in Fig. 3, the initially recovered color image is first separated into L⁄, a⁄, and b⁄ images [12], and median filtering is then applied to the a⁄ and b⁄ images to remove undesirable color noises. For sharpness enhancement, the L⁄ image of the initially recovered color image is substituted with the inversely halftoned image [43]. As shown in Fig. 5(g), the color noises are reduced and the overall sharpness is increased compared to the image in Fig. 5(f). This indicates that conventional median filtering and inverse halftoning of the separated L⁄, a⁄, and b⁄ images are sufficiently effective to remedy the problems. However, the recovered colors are still desaturated, and their hues need to be adjusted. The simplest method of overcoming this desaturation is to reconvert the L⁄, a⁄, and b⁄ images to L⁄, C⁄, and H⁄ images [12], and then increase the chroma values of C⁄ while maintaining the L⁄ and H⁄ images.

C;out ¼ s  C;in ¼ s 

rffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi  2 2 ;in ða;in Þ þ b

ð8Þ

where the input chroma image, C⁄,in, is multiplied by a positive scale factor, s, for chroma enhancement. This is well known as the simplest linear chroma mapping algorithm [32]. A more sophisticated method is to simultaneously change the L⁄ and C⁄ images [31]; however, the inaccurate hue values cannot be corrected. In some sense, our problem can be regarded as a color matching problem between two images. Color matching problems between cross-media devices have been studied for several decades [45]. To match the colors between cross-media devices, e.g., cameras and printers, the relation between the L⁄a⁄b⁄ values and the digital RGB values is first established for each device, which is called device characterization. For the characterization of input imaging devices, a regression model is used and the look-up table is designed for output device characterization. After conducting the device characterization, the matched colors between two devices can be obtained with a gamut mapping algorithm [31] to overcome gamut mismatch. The standard RGB (sRGB) international color consortium (ICC) profile [2] is used for device characterization. That is, the L⁄a⁄b⁄ values of the two images, i.e., the recovered image and its original image, are calculated using the same sRGB ICC profile. Moreover, the gamut mapping algorithm is unnecessary in this case because there is no gamut mismatch between the two images. The linear regression model is used to find the color mapping relation between the two images, as follows.

P ¼ aV 2

L;o 1

6 P ¼ 4 a;o 1

;o b1

ð9Þ L;o 2 a;o 2 ;o

b2

L;o N

3

2

3

2

11

12

1N

3

a0 a1 a2 a3 7 6 L;r L;r L;r 6 1 6 4 7 N 7 2 ;o 7 . . . aN 5; a ¼ 4 a a5 a6 a7 5; V ¼ 6 ;r ;r ;r 7 4 a1 a2 . . . aN 5 ;o bN a8 a9 a10 a11 ;r ;r ;r b1

b2

ð10Þ

bN

where the superscripts r and o indicate the recovered and original images, respectively. In (10), the L⁄a⁄b⁄ values of the original color images are reshaped as column vectors and then inserted into the matrix P. Similarly, the L⁄a⁄b⁄ values of the recovered color images are inserted into the matrix V, in which N is the total number of L⁄a⁄b⁄ values. The relation between

10

C.-H. Son et al. / Information Sciences 299 (2015) 1–19

Fig. 6. Reconstructed color images (top row) vs. original color images (bottom row). (For interpretation of the references to color in this figure legend, the reader is referred to the web version of this article.)

the two matrices, P and V, in other words, the color mapping relation between the original and recovered color images, is established based on linear transformation with the coefficient matrix a. To estimate a, six training image pairs consisting of the recovered and the original color images are used. Fig. 6 shows the training color image pairs, which consist of the reconstructed color images and the original color images. Certainly, any test images shown in the experimental result should not be included in the training image pairs. Given the training image pairs, the L⁄a⁄b⁄ values of the paired original and recovered color images are inserted into the P and V matrices at the same column position, respectively, and the pseudo-inverse transformation is then applied to (9) to obtain the a matrix.



a ¼ PVT VVT

1

ð11Þ

If the a matrix is given, the L⁄a⁄b⁄ values of the recovered color image can be mapped to be closer to the original L⁄a⁄b⁄ values based on the linear transformation, aV. Fig. 5(h) and (i) shows the recovered color images after applying the chroma mapping and color mapping methods, respectively. The value of the scale factor s is set to 1.7. It can be observed that the vividness in Fig. 5(h) is more improved than that in Fig. 5(g) due to the application of chroma mapping; however, hue differences remain between the images in Fig. 5(a) and (h) because chroma mapping only controls the strength of vividness. In contrast, the proposed color mapping method enables the hues of the recovered image to be adjusted, and thus the hues of Fig. 5(i) become closer to those of Fig. 5(a). Even though some color errors remain in the image in Fig. 5(i) compared to that in Fig. 5(a), the proposed algorithm demonstrates the possibility of estimating the original colors from the B&W halftone image. 4. Experimental results 4.1. Comparison of the proposed method with conventional color estimation methods using inverse halftoning To compare the performance of the proposed method, the first step of recovering the color embedded gray image from the B&W halftone image is implemented using state-of-the-art conventional inverse halftoning methods including MAP [48], WInHD [38], and LUT [35]. Since the proposed approach decomposes the color estimation problem into two steps, the conventional inverse halftoning methods can be used to reconstruct the color-embedded gray images. As shown in Fig. 2, the performance of inverse halftoning is not proportional to that of color reconstruction from the B&W halftoned patterns. Therefore, in this paper, state-of-the-art inverse halftoning methods [35,38,48] that provide relatively good color reconstruction were chosen. To generate the original color image from the recovered color-embedded gray image, the second step was conducted with Ko’s color decoding method [29]. Inverse halftoning [43] and median filtering [17] were also applied to the initially recovered color images using conventional color estimations to ensure a fair comparison. The test image size is 512  512, and the CPU used is i3-2012 3.3 GHz with high computing power. Fig. 7 shows the experimental results for the ‘parrot’ test image produced by the different methods. Fig. 7(a) shows the B&W halftone image with homogeneously distributed dot patterns. To analyze the halftoned dots, the red square region at the top-right corner is magnified and then placed in the bottom-right corner. From the B&W halftoned image, Fig. 7(b) is restored using the WInHD method. The small color areas presented in Fig. 7(b) imply that the WInHD method can create the inversely halftoned image with good image quality [38], but it is not appropriate for our goal, i.e., the original color estimation, because the method tends to remove the color-embedded textures. On the other hand, the MAP and LUT methods can recover some degree of color information, as shown in Fig. 7(c) and (d), respectively. For inverse halftoning, the performance of the MAP method is inferior to that of the WInHD; however, in the original color estimation, the opposite result is obtained. This is because the WInHD method tries to reduce the noisy dot patterns on flat regions and thus removes the tiny color-embedded textures. This indicates that the goal of conventional inverse halftoning differs from the goal of the original

C.-H. Son et al. / Information Sciences 299 (2015) 1–19

11

Fig. 7. Experimental results for ‘parrot’ image; (a) B&W halftone image, (b) color image recovered with WInHD, (c) color image recovered with MAP, (d) recovered color image with LUT, (e) color image recovered with proposed color estimation, (f) color image recovered with proposed color estimation and chroma mapping, (g) color image recovered with proposed color estimation and color mapping, and (h) original color image. (For interpretation of the references to color in this figure legend, the reader is referred to the web version of this article.)

color estimation. In other words, it is important to recover the color-embedded textures for improved color estimation. For this reason, the LUT method can provide better results because the color-embedded B&W halftone patterns and the corresponding averaged gray levels are stored in the LUT. In [35], the B&W halftone patterns and the corresponding averaged gray levels were originally used to construct the LUT; however, the color-embedded B&W halftone patterns and the corresponding averaged gray levels should be used for original color estimation. Fig. 7(e) shows the recovered color image with the proposed color estimation. The colors of the background and parrot can be more vividly reproduced than those of the images in Fig. 7(b) and (d). This is due to the learned dictionaries, which can associate the B&W halftone patterns with the color-embedded textures. Fig. 7(f) shows the resulting image after applying the chroma mapping algorithm, and Fig. 7(g) shows the resulting image after applying the color mapping method. The vividness of the image in Fig. 7(f) is more improved than that of Fig. 7(e); however, the recovered colors differ from the original colors in Fig. 7(h). The difference could be dramatically reduced by applying the color mapping method based on linear regression fitted to the training image sets consisting of pairs of the original and recovered color images. It should be noted that the comparison of the recovered color-embedded textures generated by the conventional and proposed methods is excluded in this paper because the recovered embedded textures differ from the original textures.

12

C.-H. Son et al. / Information Sciences 299 (2015) 1–19

Similar results were obtained for the ‘powder’ and ‘street tree’ images, as shown in Figs. 8 and 9. The proposed color estimation method produces more saturated and accurate colors than the conventional methods, and the recovered colors are adjusted to be close to the original colors through the proposed color mapping method. However, incorrect colors remain on the finally restored color images due to a limitation of the proposed algorithm. A more elegant dictionary learning algorithm needs to be developed for accurate texture recovery. This issue remains for future work. Nevertheless, the proposed method can provide a good solution for estimating the original colors from a B&W halftone image. A quantitative evaluation of the three experiments conducted in Sections 4.1–4.3 is summarized in Table 1. Experiment 4.1 show in Table 1 gives a quantitative evaluation of the proposed method and the conventional color estimation based on inverse halftoning. The overall image quality of the recovered color images is measured using the color peak signal-to-noise ratio (CPSNR) [22]. The CPSNR is the averaged PSNR value of the R, G, and B color channels. In addition, the CIEa⁄b⁄ difference qffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi ffi ;r

2 ;r P ;o 2 measure, Da⁄b⁄, which is defined as N1 Ni þ b  b ai  a;o [12], where N indicates the total pixel number and i i i i indicates the pixel index, is also used to measure the accuracies of the recovered colors. Since the L⁄ plane is restored with the same inverse halftoning method [43] in all the proposed and conventional methods, Da⁄b⁄ rather than DL⁄a⁄b⁄ is used. In experiment 4.1 of Table 1, the proposed color method shows the best performance (that is, the highest CPSNR values and the

Fig. 8. Experimental results for ‘Powder’ image; (a) B&W halftone image, (b) color image recovered with WInHD, (c) color image recovered with MAP, (d) color image recovered with LUT, (e) color image recovered with proposed color estimation, (f) color image recovered with proposed color estimation and chroma mapping, (g) color image recovered with proposed color estimation and color mapping, and (h) original color image. (For interpretation of the references to color in this figure legend, the reader is referred to the web version of this article.)

13

C.-H. Son et al. / Information Sciences 299 (2015) 1–19

Fig. 9. Experimental results for ‘Street tree’ image; (a) B&W halftone image, (b) color image recovered with WInHD, (c) color image recovered with MAP, (d) color image recovered with LUT, (e) color image recovered with proposed color estimation, (f) color image recovered with proposed color estimation and chroma mapping, (g) color image recovered with proposed color estimation and color mapping, and (h) original color image. (For interpretation of the references to color in this figure legend, the reader is referred to the web version of this article.)

Table 1 Quantitative evaluation. Experiments

Test images

‘Parrot’ image

Methods

CPSNR

Da⁄b⁄

CPSNR

Da⁄b⁄

CPSNR

Da⁄b⁄

CPSNR

Da⁄b⁄

Section 4.1

WInHD [38] MAP [48] LUT [35] Proposed color estimation Proposed color estimation with chroma mapping Proposed color estimation with color mapping

17.089 17.373 17.453 18.191 19.094 21.526

42.314 40.108 39.915 35.666 31.451 20.343

15.426 15.771 15.742 16.072 16.514 17.993

42.866 38.996 39.284 36.482 32.939 24.597

14.011 13.933 13.983 14.436 14.941 15.925

54.426 53.236 53.180 49.250 45.678 41.982

16.04 16.335 16.402 16.902 17.618 19.576

34.638 32.584 32.332 30.085 27.401 21.085

Section 4.2

Reversible color to gray conversion method using resolution conversion technique [29] Proposed color estimation using learned dictionaries and resolution conversion technique

22.385

23.538

20.750

24.650

18.012

33.124

21.166

19.806

26.338

16.325

22.470

16.886

19.336

25.906

23.556

14.529

DHED hardcopy watermarking method [15]

21.269

12.285

17.599

18.991

15.791

35.438

19.350

14.041

Section 4.3

‘Street tree’ image

‘Flower’ image

‘Powder’ image

14

C.-H. Son et al. / Information Sciences 299 (2015) 1–19

smallest Da⁄b⁄ values) compared to the conventional methods. The performance differences between the proposed method and the conventional methods are more clearly shown in the Da⁄b⁄ measure than the CPSNR measure. This is because the CPSNR measure is related to overall image quality evaluation, whereas the Da⁄b⁄ measure is related to color fidelity; in other words, the chrominance data are only used to evaluate Da⁄b⁄. Moreover, Da⁄b⁄ is calculated on the perceived uniform color space, and thus it is more appropriate for color estimation. The Da⁄b⁄ results show that the accuracies of the recovered colors are more improved with the proposed method, and that the application of chroma mapping or color mapping to the recovered images enables the recovery of more accurate colors. 4.2. Comparison of the proposed method and reversible color to gray conversion method using resolution conversion The reversible color to gray conversion methods cannot directly recover the original colors from the B&W halftoned image. However, the reversible color to gray conversion method using resolution conversion enables the recovery of the original colors from the B&W halftoned image [8]. The resolution conversion has been mainly used for printing and scanning. Fig. 10 illustrates the print-scan path consisting of resolution conversion for printing, digital halftoning, and resolution conversion for scanning. This print-scan path can be easily found in MFPs that include both printing and scanning. In [8], de Queiroz uses the print-scan path including resolution conversion to recover the color-embedded gray image. The color-embedded gray image can be recovered from the magnified B&W halftoned image with resolution conversion. In [8], the resolution conversion for printing is implemented using interpolation, and the resolution conversion for scanning is implemented using Gaussian low-pass filtering and interpolation. Gaussian low-pass filtering is used to reflect the spreading and dot shape of the real printed dots. In this paper, cubic interpolation is used for printing and scanning resolution conversions. The variance of the Gaussian low-pass filter is set to 1.5. In the resolution conversion for printing, the original colorembedded gray image is magnified by four times. For halftone pattern generation, error diffusion is used and noise addition is excluded. In the resolution conversion for scanning, the size of the low-pass-filtered halftone image is reduced by four times in order to match the size of the original color-embedded gray image. In short, the original color-embedded gray image is first generated with Ko’s color embedding method [29], and the print-scan path including resolution conversion is applied to provide the recovered color-embedded gray image. Ko’s color decoding method is conducted to recover the original color image. The recovered color-embedded gray image is presented in Fig. 11(a), and the magnified region of the red rectangle marked in Fig. 11(a) is presented in Fig. 11(b). As shown in Fig. 11(b), the color-embedded textures are accurately reconstructed. Since the textures are well preserved through the print-scan path, the original colors are accurately estimated, as shown in Fig. 11(c). This result shows that the color-embedded textures are preserved with resolution conversion. Thus, using the resolution conversion enables the color-embedded gray image to be recovered from the magnified B&W halftone image. However, recent digital cameras can provide high resolution imaging, and thus resolution conversion for printing does not need to be conducted. At this time, the resolution conversion technique cannot be used to reconstruct a color-embedded gray image. Without resolution conversion, it is impossible for the conventional reversible color to gray conversion methods to recover the original colors from a B&W halftone image. In addition, the resolutions used for printing and scanning can be set differently by users. If the resolution used for printing differs from the resolution used for scanning, the original colors cannot be recovered with the reversible color to gray conversion method. The resulting image is as shown in Fig. 11(d) if the resolutions used for printing and scanning are not the same. Even if the resolutions are known, the B&W halftone image should be saved as the magnified version, which increases the size of the image file. However, the proposed method enables the recovery of the original colors from the B&W halftone image without any magnification or any additional information about the used resolution. This is the main benefit that differentiates the proposed method from the conventional reversible color to gray conversion methods using resolution conversion. Moreover, a better quality of restored color can be obtained if the proposed method is combined with the resolution conversion technique. The textures of the recovered color-embedded gray image via the print-scan path can be re-mapped to more accurate original textures through dictionary learning, where the recovered color embedded gray patches and the corresponding original color embedded gray patches are inserted into Y in (4). The learned dictionaries for the representations of the recovered color embedded textures and original color embedded textures are shown in Fig. 11(e) and (f), respectively. The dictionaries in the yellow rectangles are magnified and then placed in the lower right area of the figures. In the magnified regions, the contrast of the textures in Fig. 11(f) is clearer than that of the textures in Fig. 11(e). Fig. 11(g) shows the recovered color-embedded gray image with the learned dictionaries shown in Fig. 11(e) and (f). Fig. 11(i) shows the corresponding recovered color image. No enhancement algorithms such as color mapping or chroma mapping were applied to the image in Fig. 11(i). The magnified textures, as shown in Fig. 11(h), are more clearly

Fig. 10. Color-embedded gray reconstruction from B&W halftone image via resolution conversion. (For interpretation of the references to color in this figure legend, the reader is referred to the web version of this article.)

C.-H. Son et al. / Information Sciences 299 (2015) 1–19

15

Fig. 11. Experimental results for ‘Parrot’ image; (a) recovered color-embedded gray image via Ko’s color embedding and print–scan path where the same resolution was used for printing and scanning, (b) magnified region of the red rectangle marked in (a), (c) recovered color image from (a) via Ko’s color decoding method, (d) recovered color image with different resolution settings for printing and scanning, (e) learned dictionaries for the recovered colorembedded textures, (f) learned dictionaries for the original color-embedded textures, (g) recovered color-embedded gray image via learned dictionaries, (h) magnified region of the red rectangle marked in (g), and (i) recovered color image from (g). (For interpretation of the references to color in this figure legend, the reader is referred to the web version of this article.)

reconstructed in comparison to those in Fig. 11(b). Therefore, the recovered colors in Fig. 11(i) are more accurately estimated than those in Fig. 11(c), especially in the background region. The CPSNR and Da⁄b⁄ values of the restored color images using resolution conversion in the print-scan path are presented in experiment 4.2 of Table 1. Assuming that the resolution information is known, the CPSNR and Da⁄b⁄ evaluations of the proposed method using learned dictionaries are better than those of Ko’s reversible color to gray conversion method using resolution conversion. 4.3. Comparison of the proposed method with a conventional hardcopy watermarking method Conventional watermarking methods [4,7,13,15,20,21,39,44] can embed the original color information into the B&W halftone image. Since Cox’s method [7] embeds the watermark information into the entire transformed domain, the encoded information cannot correlate to the image contents. To overcome this problem, HVS is exploited to reduce the watermark visibility by inserting the watermark into the high frequency regions [39]. Piva’s method [39] is very similar to the conventional reversible color to gray conversion methods [9,29]. In Piva’s method, it is assumed that original watermarks such as

16

C.-H. Son et al. / Information Sciences 299 (2015) 1–19

logos or messages created for copyright protection can be provided for watermark detection. Thus, the purpose of Piva’s method is to determine whether or not the original watermark is embedded in a given image; it is not necessary to recover the original watermark. In contrast, the reversible color to gray conversion methods focus more on recovering the embedded original watermark, i.e., the Cb and Cr images. Thus, Piva’s encoding method is better suited than Ko’s method to realize the proposed color estimation algorithm. Another approach known as hardcopy watermarking can be used to directly embed a watermark into a B&W halftone image. Even if hardcopy watermarking is not intended for development of the original color estimation, this method can be used to embed the Cb and Cr images into the B&W halftoned image. The well-known hardcopy watermarking method using error diffusion halftoning is known as data hiding error diffusion (DHED) [15]. This method is the same as the conventional error diffusion method, except that the values of the binary sequence to be embedded into the predefined pixel locations with a pseudo-random number generator are not changed during binary quantization. Since the errors occurring at the embedded pixel locations are diffused into neighborhood pixels, the DHED method can provide good quality halftone images. However, in this method, the embedded pixel locations need to be determined in order to extract the embedded bit sequence from the B&W halftone image. Whenever resolution conversion or image resizing can change the embedded pixel locations, it is annoying to memorize the locations of the embedded pixels. Moreover, image cropping removes some of the embedded pixel locations, which makes the original color estimation difficult. In contrast to the DHED method, the proposed method tries to estimate the original colors from the B&W halftone image without additional information, i.e., the embedded pixel locations. This is the main difference between DHED and the proposed method. There are other types of hardcopy watermarking methods [4,20,21,44]; however, these methods do not estimate the original colors from the B&W halftone images. The hardcopy watermarking methods [4,21,44] address dithered halftoned images, as opposed to the error-diffused halftoned images on which this paper focuses. Though the LMS-based method [20] considers error-diffused halftoned images, it requires additional information, such as embedded pixel locations, similar to the DHED method. The comparison of the proposed method to other hardcopy watermarking methods is out of the scope of this paper. The DHED method is simulated to determine original color estimation. The DHED method can only embed one bit per pixel location. Given the Cb and Cr images with a size of N  N, the actual bit number to be embedded into the B&W halftone image is N  N  2  8. Since this actual bit number is larger than the maximum bit number (that is, N  N bits) that can be embedded into the B&W halftone image, the DHED method cannot directly embed the original Cb and Cr images into the B&W halftone image. An alternative solution is to reduce the size of the Cb and Cr images. Fig. 12(a) and (b) shows the B&W halftone images generated with the DHED method where the Cb and Cr images (resized by 1/8 and 1/16 times, respectively) are embedded. The halftone patterns of Fig. 12(a) are very noisy because the Cb and Cr images are forcibly embedded into the predefined pixel locations, whereas the homogeneity of the halftone patterns shown in Fig. 12(b) is improved due to the reduced bit number. The visual quality of Fig. 12(b) can be considered acceptable; however, it has a lower quality level than that of the B&W halftone image generated with Ko’s color embedding and error diffusion, as shown in Fig. 7(a). It is noteworthy that the original size of the ‘parrot’ image is 512  512, and thus the bit number embedded into Fig. 12(b) is 512/16  512/16  8  2 = 16,384 bits. Fig. 12(c) shows the recovered color image from Fig. 12(b). For this, the same inverse halftoning method [43] is used to reconstruct the L⁄ plane. In Fig. 12(c), some noise is generated due to the noisy halftone dots in Fig. 12(b). Also, color contours occur around the parrot’s boundary due to the resizing of the Cb and Cr images. The CPSNR values and Da⁄b⁄ are presented in experiment 4.3 of Table 1. The CPSNR values of the DHED method are slightly lower than those of the proposed color estimation with color mapping due to the generated noisy dots. However, the Da⁄b⁄ values of the DHED method are better than those of the proposed method because additional information (such as the embedded pixel locations) is utilized in the DHED method to extract the Cb and Cr images, whereas the proposed method estimates the original colors from the B&W halftone image without any additional information. To improve the color estimation accuracy of the proposed method, the Cb and Cr images resized by 1/8 and 1/16 can be repeatedly copied in the high-frequency region of the wavelet subband regions. This indicates that the capacity of the DHED method is lower than that of Ko’s color embedding and error diffusion method. 4.4. Discussion 4.4.1. Geometric distortion issue in the real print-scan path The proposed method is not limited to color reconstruction of a transmitted B&W halftoned image. It can also be applied to scanned B&W halftoned images. The scanned version of the printed image is not a perfect binary image due to the dot gain of the printed inks/toners [25]. However, a simple thresholding technique can convert the scanned image into a binary image. In this paper, it is assumed that the scanned images have already been binarized, though geometric distortion problems [4,10] can arise in a real print-scan path. The corner points [34] or projective transformation [40] can be used for the geometric distortion correction. However, as discussed in [4], local geometric distortion is a challenging problem. The goal of this paper is to show that original color images can be directly reconstructed from B&W halftoned images. Thus, the color reconstruction from the real printed images, which is beyond the scope of this study, is an issue for future research. However, in an ideal case in which there is no geometric distortion, the print-scan path is simulated, as shown in Section 4.2. The difference between the reversible color-to-gray conversion method and the proposed method is discussed. Also, the proposed method is compared with the DHED method, as shown in Section 4.3.

C.-H. Son et al. / Information Sciences 299 (2015) 1–19

17

Fig. 12. Experimental results for ‘Parrot’ image; (a) B&W halftone image with DHED where the Cb and Cr images resized 1/8 times are embedded, (b) B&W halftone image with DHED where Cb and Cr images are resized 1/16 times are embedded, and (c) recovered color image from (b). (For interpretation of the references to color in this figure legend, the reader is referred to the web version of this article.)

4.4.2. Visual comparison between color inverse halftoning and the proposed methods Since the proposed method considers the scanned B&W halftoned images, its goal differs from that of the color inverse halftoning method. The color inverse halftoning method is used to remove moire artifacts [49] from the scanned images, while the proposed method is used to recover the original colors of the scanned images, thereby reducing the consumption and cost of expensive color inks or toners. The color inverse halftoning method can provide more accurate colors than the proposed method because it reconstructs the original colors from the color halftoned image. Therefore, it is unnecessary to compare the images produced via the color inverse halftoning method with those produced via our proposed method. In addition, the color restoration quality of the proposed method can be enhanced using Cb and Cr redundancies. The Cb and Cr images exhibit a level of sparseness in the gradient domain [46], which can be compressed according to the JPEG (Joint Photographic Experts Group) standard and then repeatedly inserted into the wavelet sub-bands of the color-embedded gray image, improving the color restoration quality of the proposed method. 4.4.3. Computational cost issue In this paper, inverse halftoning must be performed twice. However, the inverse halftoning used to reconstruct the luminance plain (L⁄), as shown in Fig. 3, can be implemented via a look-up table (LUT) [35]. As already discussed in [43], LUTbased inverse halftoning is very fast, with an average computation time of 0.045 s under the same experimental conditions that controls for image size and CPU speed. Also, good visual quality can be ensured with post-processing and median filtering, which can be rapidly implemented [54]. The IIH-TED method [43] is used in this paper, though the LUT-based method, which is considered state-of-the-art, can be used without any loss of color reconstruction quality. The computational time of the color mapping method is 1.386 s; therefore, the bulk of the computation time is spent in color-embedded gray reconstruction via the learned dictionaries. This is due to non-convex minimization of (5). Color-embedded gray reconstruction requires approximately 239 s to minimize (5) and (6) for a 512  512 image. To reduce the computation time, the minimization of (5) can be conducted in parallel way. This is possible because the current color-embedded gray reconstruction is processed in a repetitive way in which the halftoned patches extracted from input halftoned image are sequentially estimated. Another solution is to reduce the number of column vectors in the dictionary matrix. In this paper, there are 1024 column vectors. In other papers [1,11], 256 column vectors have been used to solve the restoration problem. Thus, it is possible to reduce the computational time by using a resized dictionary matrix. 5. Conclusion A method of directly recovering the original RGB image from an input B&W halftone image without any magnification or additional information such as the used resolution setting is shown via dictionary learning, reversible color to gray conversion, and the color mapping method. The experimental results show that the use of learned dictionaries is more helpful to recover the color-embedded textures from an input B&W halftone pattern via a linear sparse representation. Furthermore, the color mapping method based on the linear regression model that fits the training image sets enables the recovered colors to be adjusted closer to the original colors. In addition, the proposed color estimation method can recover more vivid and accurate colors than the conventional methods. Acknowledgments This research was supported by Basic Science Research Program through the National Research Foundation of Korea (NRF) funded by the Ministry of Education (NRF-2013R1A1A2061165), (NRF-2012009055), and (NRF-2010-0020210).

18

C.-H. Son et al. / Information Sciences 299 (2015) 1–19

References [1] M. Aharon, M. Elad, A. Bruckstein, K-SVD: an algorithm for designing overcomplete dictionaries for sparse representation, IEEE Trans. Signal Process. 54 (2006) 4311–4322. [2] M. Anderson, R. Motta, S. Chandrasekar, M. Stokes, Proposal for a standard default color space for the internet – sRGB, in: Proc. 4th Color Imaging Conf., 1996, pp. 238–246. [3] R. Bala, R. Eschbach, Spatial color-to-grayscale transform preserving chrominance edge information, in: Proc. 12th Color Imaging Conf., 2004, pp. 82– 86. [4] O. Bulan, G. Sharma, V. Monga, Orientation modulation for data hiding in clustered-dot halftone prints, IEEE Trans. Image Process. 19 (2010) 2070– 2084. [5] A. Chatterjee, B. Tudu, K. Paul, Towards optimized binary pattern generation for grayscale digital halftoning: a binary particle swarm optimization (BPSO) approach, J. Vis. Commun. Image Represent. 23 (2012) 1245–1259. [6] M. Chaumont, W. Puech, Protecting the color information by hiding it, in: Recent Advances in Signal Processing, vol. 22, In-TECH, 2009. [7] I.J. Cox, J. Kilian, F.T. Leighton, T. Shamoon, Secure spread spectrum watermarking for multimedia, IEEE Trans. Image Process. 6 (1997) 1673–1687. [8] R.L. de Queiroz, Reversible color-to-gray mapping using subband domain texturization, Pattern Recogn. Lett. 31 (2010) 269–276. [9] R.L. de Queiroz, K.M. Braun, Color to gray and back: color embedding into textured gray images, IEEE Trans. Image Process. 15 (2006) 1464–1470. [10] J.-L. Dugelay, S. Roche, C. Rey, G. Doerr, Still-image watermarking robust to local geometric distortions, IEEE Trans. Image Process. 15 (2006) 2831– 2842. [11] M. Elad, M. Aharon, Image denoising via sparse and redundant representation over learned dictionaries, IEEE Trans. Image Process. 15 (2006) 3735– 3736. [12] M. Fairchild, Color Appearance Model, Wiley, 2005. [13] L. Feng, D. Cong, H. Shu, B. Liu, Adaptive halftone watermarking algorithm based on particle swarm optimization, J. Multimedia 8 (2013) 183–190. [14] A. Foi, V. Katkovnik, K. Egiazarian, Point shape-adaptive DCT for high-quality denoising and deblocking of grayscale and color images, IEEE Trans. Image Process. 16 (2006) 1395–1411. [15] M. Fu, O. Au, Data hiding watermarking for halftone images, IEEE Trans. Image Process. 11 (2002) 477–484. [16] Y.-H. Fung, Y.-H. Chan, Optimizing the error diffusion filter for blue noise halftoning with multiscale error diffusion, IEEE Trans. Image Process. 22 (2013) 413–417. [17] R. Gonzales, R. Woods, Digital Image Processing, Prentice Hall, 2002. [18] A.A. Gooch, S.C. Olsen, J. Tumblin, B. Gooch, Color2gray: salience-preserving color removal, in: Proc. ACM SIGGRAPH, 2005, pp. 634–639. [19] M. Grundland, N.A. Dodgson, Decolorize: fast, contrast enhancing, color to grayscale conversion, Pattern Recogn. 40 (2007) 2891–2896. [20] J.-M. Guo, A new model-based digital halftoning and data hiding designed with LMS optimization, IEEE Trans. Multimedia 9 (2007) 687–700. [21] J.-M. Guo, Watermarking in dithered halftone images with embeddable cells selection and inverse halftoning, Signal Process. 88 (2008) 1496–1510. [22] A. Hore, D. Ziou, Image quality metrics: PSNR vs. SSIM, in: Proc. Int. Conf. Pattern Recognition, 2010, pp. 23–26. [23] T. Horiuchi, F. Nohara, S. Tominaga, Accurate reversible color-to-gray mapping algorithm without distortion conditions, Pattern Recogn. Lett. 31 (2010) 2405–2414. [24] R. Irony, D. Cohen-Or, D. Lischinski, Colorization by example, in: Proc. Eurographics Symposium on Rendering, 2005, pp. 201–210. [25] H.R. Kang, Color Technology for Electronic Imaging Devices, SPIE Press, 1997. [26] H.B. Kekre, S.D. Thepade, R.N. Chaturvedi, Color to gray and back’ using normalization of color components with Cosine, Haar and Walsh Wavelet, IOSR J. Comput. Eng. 10 (2013) 95–104. [27] J.-G. Kim, G. Kim, Design of optimal filters for error-feedback quantization of monochrome pictures, Inform. Sci. 39 (1986) 285–298. [28] K.-W. Ko, D.-C. Kim, W.-J. Kyung, Y.-H. Ha, Color embedding and recovery using wavelet packet transform with pseudorandomized saturation code, J. Imag. Sci. Technol. 55 (2011) 030501-1–030501-10. [29] K.-W. Ko, O.-S. Kwon, C.-H. Son, Y.-H. Ha, Color embedding and recovery based on wavelet packet transform, J. Imag. Sci. Technol. 52 (2008) 010501-1– 010501-10. [30] J.-H. Kwon, C.-H. Son, Y.-H. Cho, Y.-H. Ha, Text-enhanced error diffusion using multiplicative parameters and error scaling factor, J. Imag. Sci. Technol. 50 (2006) 437–447. [31] C.-S. Lee, Y.-W. Park, S.-J. Cho, Y.-H. Ha, Gamut mapping algorithm using lightness mapping and multiple anchor points for linear tone and maximum chroma reproduction, J. Imag. Sci. Technol. 45 (2001) 209–223. [32] M.-Y. Lee, C.-H. Son, J.-M. Kim, C.-H. Lee, Y.-H. Ha, Illumination-level adaptive color reproduction method with lightness adaptation and flare compensation for mobile display, J. Imag. Sci. Technol. 51 (2007) 44–52. [33] T. Leung, J. Malik, Recognizing surfaces using three-dimensional textons, in: Proc. Int. Conf. Comput. Vis., 1999, pp. 1010–1017. [34] J. Lin, T. Horiuchi, K. Hirai, S. Tominaga, Color image recovery system from printed gray image, in: Proc. IEEE Southwest Symposium on Image Analysis and Interpretation, 2014, pp. 41–44. [35] M. Mese, P. Vaidyanathan, Look-up table (LUT) method for inverse halftoning, IEEE Trans. Image Process. 10 (2001) 1566–1578. [36] Y. Miyashita, M. Hasegawa, S. Kato, Y. Tanaka, Improved adaptive color embedding and recovery using DC level shifting, in: Proc. IEEE International Conference on Acoustics, Speech and Signal Processing, 2013, pp. 1588–1592. [37] R. Nasanen, Visibility of halftone dot textures, IEEE Trans. Syst., Man, Cybernet. smc-14 (1984) 920–924. [38] R. Neelamani, R. Nowak, R. Baraniuk, Model-based inverse halftoning with wavelet–vaguelette deconvolution, in: Proc. IEEE Int. Conf. Image Processing, 2000, pp. 973–976. [39] A. Piva, M. Barni, F. Bartolini, V. Cappellini, DCT-based watermark recovering without resorting to the uncorrupted original image, in: Proc. IEEE Int. Conf. Image Processing, 1997, pp. 520–523. [40] S. Pollard, R. Ulichney, M. Gaubatz, Recovering planar projections of printed clustered-dot halftones, in: Proc. IEEE Int. Conf. Image Processing, 2014, pp. 4226–4230. [41] J. Qin, N. Yung, Scene categorization via contextual visual words, Pattern Recogn. 43 (2010) 1874–1888. [42] M.A. Ruzon, S. Cohen, G. Wilensky, M. Newell, Fast and adaptive color-to-grayscale conversion, in: Proc. ACM SIGGRAPH Sketches, 2006, pp. 124–125. [43] C.-H. Son, H. Choo, Iterative inverse halftoning based on texture-enhancing deconvolution and error-compensating feedback, Signal Process. 93 (2013) 1126–1140. [44] C.-H. Son, H. Choo, Watermark detection from clustered halftone dots via learned dictionaries, Signal Process. 102 (2014) 77–84. [45] C.-H. Son, C.-H. Lee, K.-H. Park, Y.-H. Ha, Real-time color matching between camera and LCD based on 16-bit lookup table design in mobile phone, J. Imag. Sci. Technol. 51 (2007) 348–359. [46] C.-H. Son, H.-M. Park, A pair of noisy/blurry patches-based PSF estimation and channel-dependent deblurring, IEEE Trans. Consum. Electron 57 (2011) 1791–1799. [47] H. Song, B. Huang, K. Zhang, H. Zhang, Spatio-spectral fusion of satellite images based on dictionary-pair learning, Inform. Fusion 18 (2014) 148–160. [48] R. Stevenson, Inverse halftoning via MAP estimation, IEEE Trans. Image Process. 6 (1997) 574–583. [49] B. Sun, S. Li, J. Sun, Scanned image descreening with image redundancy and adaptive filtering, IEEE Trans. Image Process. 23 (2014) 3698–3710. [50] K. Uruma, K. Konishi, T. Takahashi, T. Furukawa, An image colorization algorithm using sparse optimization, in: Proc. IEEE International Conference on Acoustics, Speech and Signal Processing, 2013, pp. 1588–1592. [51] T. Welsh, M. Ashikhmin, K. Mueller, Transferring color to greyscale images, in: Proc. ACM SIGGRAPH, 2002, pp. 277–280.

C.-H. Son et al. / Information Sciences 299 (2015) 1–19

19

[52] Z.-Q. Wen, Y.-L. Lu, Z.-G. Zeng, W.-Q. Zhu, J.-H. Ai, Optimizing template for lookup-table inverse halftoning using elitist genetic algorithm, IEEE Signal Process. Lett. 22 (2015) 71–75. [53] J. Wright, A.Y. Yang, A. Ganesh, S. Sastry, Y. Ma, Robust face recognition via sparse representation, IEEE Trans. Pattern Anal. Mach. Intell. 31 (2009) 210– 227. [54] Q. Zhang, J. Jia, 100+ times faster weighted median filter (WMF), in: Proc. IEEE Conf. Computer Vision and Pattern Recognition, 2014, pp. 2830–2837.