video retexturing using GPU parallelism

Computers & Graphics 36 (2012) 1048–1059

Contents lists available at SciVerse ScienceDirect

Computers & Graphics journal homepage: www.elsevier.com/locate/cag

VEA 2012

Interactive image/video retexturing using GPU parallelism Ping Li a,n, Hanqiu Sun a, Chen Huang a, Jianbing Shen b, Yongwei Nie c a b c

The Chinese University of Hong Kong, China Beijing Institute of Technology, China Wuhan University, China

a r t i c l e i n f o

a b s t r a c t

Article history: Received 30 March 2012 Received in revised form 8 July 2012 Accepted 27 July 2012 Available online 4 August 2012

This paper presents an interactive retexturing approach that preserves similar underlying texture distortion between the original and retextured images/videos. The system offers real-time feedback interaction for easy control of target objects definition, texture selection with size adjusting, and overall lighting tuning using latest GPU parallelism. Existing retexturing and synthesis methods deal with texture distortion by inter-pixel distances manipulation, and the underlying texture distortion of the original images is always destroyed due to limitations like improper distortion caused by human mesh stretching, or unavoidable texture splitting through synthesis. The long processing time due to timeconsuming filtering is also unacceptable. We propose to utilize SIFT corner features to naturally discover the underlying texture distortion. Gradient depth recovery and wrinkle energy optimization are applied to accomplish the distortion process. We facilitate the interactive retexturing upon needs of users via real-time bilateral grid and feature-guided texture distortion optimization using CUDA parallelism, and video retexturing is accomplished by a keyframe-based texture transferring using realtime TV-L1 optical flow with patch-based block motion techniques. Our interactive retexturing using feature-guided gradient optimization provides realistic retexturing while preserving elite texture distortion in cornered area. In the experiments, our method consistently demonstrates high-quality image/video retexturing with real-time feedback interaction. & 2012 Elsevier Ltd. All rights reserved.

Keywords: GPU Retexturing Real-time processing Bilateral grid

1. Introduction Fashion is one of the most popular daily topics in nowaday lives of ordinary people. In real life, one model of sculpture, artwork or clothing is mostly used for building multi-style commodities with very different looking surface appearance. Before the invention of computer-aided design tools for fashion production, the most realistic way for testing whether one style of goods is pleased looking is to make a real sample of the style. However, these tasks are very unproductive, even a waste of time and labor. Most important of all, it is really a waste of money when the constructed style turns out to be ugly or unwelcome. Recent advances in virtual rendering have demonstrated that different appearance styles of certain objects can be generated virtually according to users’ needs [13,33,34,37]. The ability to virtually produce surface and texture appearance for more effective generation of scenes, ideas and objects would allow media creators for films and video games to attain a new level of creativity. In many cases, e.g. fashion design, image-based retexturing and rendering directly in the image space have been

n

Corresponding author. Tel.: þ852 51686571. E-mail address: [email protected] (P. Li).

0097-8493/$ - see front matter & 2012 Elsevier Ltd. All rights reserved. http://dx.doi.org/10.1016/j.cag.2012.07.004

proved to be more effective in reproducing the realism of different costume styles than real sample producing. Image retexturing and re-rendering algorithms are designed to update the visual properties of images, for objects of interest in images, with variable appearances upon the needs of users. The growing interests in image retexturing are due to the expressive power for replacing textures and changing surface appearance as well as for artwork making and dress designing. Recently, there are more and more researchers focusing on the hot topics of image rerendering through introducing new computer graphic techniques to simulate the effects of traditional real sample producing using ways of texture synthesis [7,23] and retexturing [11,18,36,39]. One essential process in retexturing is to replace existing texture in the region of interest by other user-preferred textures, while preserving the original shading effects and maintaining similar texture distortion [10,14]. However, the texture distortion generated using the nowaday image retexturing and synthesis methods is based on inter-pixel distances manipulation in image space, which is not faithfully belong to the original image due to limitations like improper texture distortion caused by human mesh stretching, or unavoidable texture splitting caused by texture synthesis, which may influence users’ understanding of visual image and fashion design greatly. The long processing time due to frequent time-consuming bilateral filtering is also unacceptable.

P. Li et al. / Computers & Graphics 36 (2012) 1048–1059

Therefore, introduce new approaches to deal with retexturing efficiently, while maintaining similar underlying texture distortion of the original images is very important and a must. In this paper, we present an interactive approach to retexture images/videos with various user-preferred textures while preserving similar texture distortion between the original and the retextured photos/videos to make an effort through exploring SIFT corner features to naturally model the underlying texture distortion of the original images, especially in the wrinkled area where corners and distortion are usual, with only very simple user inputs for different textures selection and scaling. Compared with [21], this paper makes progress in video processing, the easy control and the design for interactive retexturing with real-time feedback using GPU parallelism. Gradient-based depth map recovery and feature-guided texture distortion optimization are then applied to accomplish the distortion process. Retexturing is achieved through edge-aware bilateral grid processing with feature-guided optimization using real-time GPU-CUDA, and video retexturing is further accomplished by a keyframe-based texture transferring method using TV-L1 optical flow with patchbased block motion techniques. Fig. 1 shows one example of interactive video retexturing with gradient wrinkle stress optimization using GPU parallelism, our image/video retexturing in gradient space can produce user-customized visually pleasing results with real-time control feedback. We apply robust TV-L1 optical flow to track the motion of the clothes, and the new clothes textures are transferred well in the video clip, giving out real-time realistic retexturing using parallel GPU-CUDA acceleration, and the fine underlying texture distortion is preserved well, also in much less computing time compared with [11] due to latest GPU-CUDA acceleration. And we apply regular circle texture to retexture the child’s clothes here for easy viewing of the distortion effects, especially in the wrinkled region. To the best of our knowledge, there are few methods specifically designed to retexture images/videos with feature guidance in real-time. On the contrary, feature-guided retexturing, as a more delicate form of image/ video retexturing, allows the viewers to enjoy the content of the retextured image/video more realistically and intuitively when they are organized as utilities like fashion styles. This also facilitates many photo analysis tasks. In the following, Section 2 briefly reviews the related techniques in texture replacement, video processing and GPU acceleration. Section 3 describes our real-time parallel retexturing, including feature-guided wrinkle energy optimization,

1049

efficient video retexturing, and retexturing with GPU parallelism. Section 4 presents the experimental results of our approach. Finally, we summarize our work in Section 5. 2. Related work Texture Replacement: Image retexturing is an image processing technique which takes a single image as input, and produces different texture appearances to the objects of interest in the image upon the needs of users. This paper is made possible by the inspiration of previous work. In general, two basic forms of retexturing exist, the texture synthesis method and image retexturing. The former achieves the retexturing by constructing parametric models to synthesize the texture appearance of objects in images, while the latter applies new textures onto the objects of interest to replace the existing textures. Oh et al. [28] presented an approach to modify the shape, color and illumination of objects specified in images. Pe´rez et al. [32] introduced novel tools for seamless editing of images, using generic interpolation based on solving Poisson equations. Liu et al. [23] introduced a user-assisted texture synthesis adjustment on the regular grid of real textures, and obtained a bijective mapping between the regular grid of the texture and the deformed grid of the surface image. These approaches give out good opening for image retexturing and related processing, however, they are slow mainly due to elaborate user interactions. Using high dynamic range (HDR) images as input [6,15], Khan et al. [18] proposed an image-based material editing approach, which focuses on changing objects’ micro structure that would usually be modeled with functions like bi-directional reflectance distribution functions (BRDFs). Shen et al. [35] presented an image-based processing for tone mapping and retexturing of both HDR and LDR images. These two methods provide right direction of single image retexturing using HDR images, which have much broader luminance range and can generate more accurate recovered gradient depth map for retexturing. However, these approaches are still slow mainly due to time-consuming bilateral or trilateral filtering in gradient domain and software coding only. One essential issue of retexturing is to replace existing textures in the region of interest in images by new ones, while preserving the original shading and similar underlying texture distortion. Fang and Hart [7,8] proposed an object retexturing technique using texture synthesis method based on the assumption that

Fig. 1. Our interactive video retexturing using parallel GPU acceleration, showing the original and retextured frames of 1st, 10th, 20th and 30th, with frame 30 as the keyframe for the whole video clip because the clothes of the mother and child both have maximized visibility. (a) The original video, (b) retexturing of mother and child’s clothes, 81.90 frames/s. Fine texture distortion is preserved well between the original and retextured clothes. We apply robust TV-L1 optical flow to track the motion of the clothes, and new textures are transferred well in the video, giving out real-time realistic retexturing with parallel bilateral grid processing on GPU-CUDA platform.

1050

P. Li et al. / Computers & Graphics 36 (2012) 1048–1059

lighting satisfies the Lambertian reflectance model, and the object’s macro structure can be altered. They deal with texture distortion by inter-pixel distances manipulation in image space. Guo et al. [10,11] presented a mesh-guided retexturing that maintained the shading without knowing the underlying surface and lighting conditions, and they utilize mesh stretching to process the inter-pixel distances to preserve texture distortion. These methods provide efficient solutions to the image retexturing and make efforts in starting to consider the distortion effects, however, when performing retexturing, the underlying texture distortion of the original images is always destroyed due to limitations like unavoidable texture splitting caused by texture synthesis or improper texture distortion caused by inaccurate human mesh stretching, which is actually essential information for better understanding the material of retextured images. For instance, fashion and artwork design, where nice texture distortion shows the effects of material fitting on objects not the material itself. Besides, these algorithms use LDR images as input, which may not be accurate enough for shape from shading technique and cannot deal with single image well. The long processing time due to time-consuming filtering is also unacceptable. Li et al. [21] proposed a texture replacement method, which enabled efficient realistic retexturing using parallel bilateral grids. Therefore, it’s important to develop the real-time interactive retexturing for images/videos using feature guidance for elite texture distortion effects with GPU parallelism, and new progress in video processing, easy control and the design for interactive retexturing with real-time feedback on GPU-CUDA is addressed in the paper. Video Processing: Video processing offers us great ability of generating various videos for daily use and professional multimedia manipulation without actually shooting them with cameras. Recently many researches are carried out in video processing to synthesize creative videos for different media use by applying techniques proposed by computer graphics scientists. Bennett and McMillan [2] introduced a per pixel virtual exposures approach to enhance underexposed, low dynamic range videos for after effect processing, the virtual exposure they invented is a dynamic function that consider for each pixel both the spatial and temporal neighborhood. Barnes et al. [1] presented a video tapestry method to summarize videos in a form of multi-scale images, which are continuous not only in the spatial domain but also across scale dimension. Their video tapestries offer advantages like visual clarity of DVD chapter menus with video information density and a multi-scale representation of video editing timeline. Correa and Ma [5] proposed an approach to produce dynamic narratives from videos. Their narratives are characterized with features of compactness, coherence and interactivity, which are inspired by the spirit of sequential art, and their narratives present the motion of actors over time with dynamic visualization. The system also offers a compact view for video browsing and highlights motion sequences in movies, which can be applied to generate interactive dynamic posters from video datasets. Liu et al. [22] presented an algorithm for video stabilization which gives out high-quality camera motion for many videos. They put their focus on the tasks of transforming a set of 2D motion trajectories to make sure they are both smooth and visually plausible. Then they achieve the stabilization by assembling the tracked video features into a trajectory matrix to perform filtering or curve fitting in a low dimensional space. Wang et al. [38] proposed a method for video resizing which offers a possible answer to the scalability problem raised in resizing without compromising the resized video quality. They view the videos as spatial and temporal components, and resize each frame independently to maintain the shape of the salient regions in videos. Xu et al. [41] presented an approach to generate visually pleasing video clips of realistic human

performance according to user-defined body motions based on the initial input database, which enables creating video sequences of dangerous stunt acting without placing the actors in a vulnerable way. Inspired by creative video generating methods like these, we find it’s important to present our interactivity image/ video retexturing approach with real-time processing for userdefined texture needs. GPU Acceleration: Graphic Processor Unit (GPU) is a highly parallel, multi-threaded, many-core processor with fantastic computing power and high memory bandwidth. The stream GPU processing allows applications and data manipulation to be easily exploited and investigated in the form of parallel acceleration. Many researches are carried onto utilize GPUs to investigate the image processing domain. Bertalmio et al. [3] presented an anisotropic diffusion partial differential equation to be applied ¨ to images allows real-time rendering. Winnemoller et al. [40] proposed a real-time abstraction framework that abstracts imagery by modifying the contrast of visually important features. The framework is designed in parallel, and allows for a GPU-based real-time processing. Kazhdan and Hoppe [17] proposed a novel streaming multi-grid GPU solver to solve large linear systems arising from image processing in gradient domain. Orzan et al. [29] presented a GPU-based real-time system for rendering images defined by a set of diffusion curves, which is new vector-based primitive for creating smooth-shaded images. McCann and Pollard [26] proposed a GPU-based image editing program that allows artists to paint in the gradient domain with real-time ¨ ¨ et al. [19] presented a feedback on megapixel images. Krahenb uhl novel integrated system for content-aware retargeting on the GPU CUDA platform using an iterative multi-grid solver for the feature estimation, energy minimization and OpenGL for the EWA image synthesis. Hachisuka and Jensen [12] utilized GPU to carry the parallel progressive photon mapping with more accurate global illumination rendering. Fatahalian et al. [9] proposed a method for enhancing GPU pipeline to combine rasterized fragments from adjacent triangles in meshes to reduce redundant shading when surfaces are organized in small triangles. Parker et al. [31] introduced a general purpose GPU ray tracing engine OptiX, which enables most ray tracing methods be processed by small sets of programmable operations. Mavridis and Papaioannou [25] introduced a high quality texture filtering on GPU based on the elliptical weighted average filter. Their method uses the underlying anisotropic filtering hardware to construct a filter that closely matches the shape and the properties of the EWA filter. Nehab et al. [27] presented a GPU-efficient recursive filtering by dividing images into 2D blocks together with a small band of additional data buffered along the block perimeter, which shows notable speedup in GPU processing. For our work in featureguided retexturing, wrinkle energy optimization is needed to preserving the underlying texture distortion well while performing bilateral filtering in real-time. As known to all, optimization processing is really time-consuming, especially processed with bilateral filtering manipulation together. These latest work on GPUs give us good inspirations of performing our feature-guided retexturing with optimization using GPU-CUDA acceleration for real-time processing.

3. Image/video retexturing on GPU We propose to utilize SIFT corner features [24] to naturally discover the underlying texture distortion existed in the original images automatically. Table 1 shows the pipeline of our image/ video retexturing. Gradient depth map recovery and featureguided wrinkle energy optimization are applied to accomplish the distortion process. Our goal is to retexture the objects in

P. Li et al. / Computers & Graphics 36 (2012) 1048–1059

images/videos by user preferred visual textures while preserving the elaborate texture distortion of important human perception, to facilitate the feature-guided retexturing for artistic and commercial products design. Fig. 2 shows the interactive control flow of our feature-guided image/video retexturing. Our system offers users with simple and real-time feedback interaction of easy control for target objects definition, texture selection from texture library, retexturing texture size adjusting, and overall lighting tuning on a latest parallel GPU-CUDA platform. For video retexturing, keyframes are first extracted and retextured. We select the keyframes in which the visibility of concerned objects is maximized. The new textures are further transferred from the retextured keyframes to the neighboring frames using the latest GPU-based accurate TV-L1 optical flow tracking [42]. In our approach, the image/keyframe retexturing is performed directly in the gradient space, without the need of knowing the geometry/ mesh details. Using the gradient space information, our system computes the initial depth map, bilateral grid manipulation, SIFT feature, texture/image gradients, and retextured image reconstruction through solving Poisson equation. We develop the feature-guided image retexturing using GPU-CUDA acceleration.

Table 1 Image/video Retexturing Pipeline. 1. 2. 3. 4. 5. 6. 7. 8.

Image/keyframe selection maximizing objects’ visibility; Bilateral grids construction; Gradient depth recovery using bilateral grids; SIFT corner strength function computing; Texture indices optimization by wrinkle energy; New color gradient derivation from image and texture gradient; Retextured keyframe reconstruction by new color gradient; Keyframe texture propagation via optical flow among video clip.

1051

With the feature-guided texture distortion optimization in the wrinkled area, our approach preserves similar texture distortion and shading well. Our interactive system provides the users with easy control of image/video retexturing with real-time feedback, e.g. users can easily selecting the input textures according to needs and tuning the overall lighting in real-time. Using spectral matting strategy [20], objects of interest can be automatically identified from the background through minimizing a quadratic energy function. High quality retexturing can then be achieved on the concerned objects in images/videos with nice texture distortion effects. In general, bilateral grid is applied to get the smoothed depth map. First, we construct a bilateral grid from the initial depth map image according to the luminance distribution of the objects of interest. Second, we perform processing inside the bilateral grid. Finally, we slice the bilateral grid to build the final output depth map. Note that grid construction and slicing are symmetric processes that convert between image and grid space. After the bilateral grid manipulation, the smoothed depth map Isd is applied to describe the locally consistent gradient depth map rIg ðx,yÞ using terms of neighboring gradient values as ðrIg x ðx,yÞ, rIgy ðx,yÞÞ. According to the principle of human vision and depth map technique using shape from shading [16], human vision system treats depth inversely related to intensity, farther parts of the objects will have lower intensity values. Fig. 3 shows the gradient depth maps reconstructed from the input image using our real-time bilateral grid processing on GPU with feature-guided retexturing result. The gradient depth maps generated here may not be fully accurate for general use. However, this will not affect the retexturing task which cares more for visual effects and puts its focus in generating user pleasing results. Asymmetric texture is applied in Fig. 3 for checking the realistic retexturing of costume parts like the right bent arm and the connection part of the arm and the shoulder,

Fig. 2. Interactive retexturing control flow. Our retexturing system takes images/videos with user desired textures as input, computes the gradient depth maps, optimizes texture distortion in wrinkled area with feature guidance, and finally outputs the retextured images/videos. The system offers users with simple and real-time feedback interaction for target objects definition, texture selection from library, retexturing texture size adjusting, and overall lighting tuning on a latest parallel GPU-CUDA platform.

Fig. 3. Reconstructed gradient depth map and retexturing for a HDR image using bilateral grid processing, 62.83 frames/s. (a) Image for clothing retexturing, (b) gradient depth map for the costume recovered using parallel bilateral grid, (c) SIFT feature points of the costume, mainly on wrinkled area, (d) our retextured result of the costume, we also perform retexturing for the skirt. Asymmetric texture is applied for checking the realistic retexturing of parts like the right bent arm and the connection between the arm and the shoulder, where there is global change in texture direction. And the underlying texture distortion is maintained well.

1052

P. Li et al. / Computers & Graphics 36 (2012) 1048–1059

where global change of texture direction exists. The underlying texture distortion is preserved well here, especially in the wrinkled area where feature points are usual. We will discuss the wrinkle energy optimization using SIFT corner strength function in detail in Section 3.1.

3.1. Wrinkle energy optimization Gradient vale shows the slope of the objects’ depth at a point, larger gradient value describes that the objects’ depth is steeper at that point. Therefore the objects’ gradient depth map rIg ðx,yÞ is in principle sufficient to estimate the warping of a new arbitrary texture T around the surface of the objects, and can be used to retexture the objects of interest. Larger gradient valued points on the original image will generate more warping for the new texture. We introduce ðux ,uy Þ to lookup the input texture gradient value to perform the texture warping around the objects as rT g ðux ,uy Þ. However, the gradient values of the original image are usually small, thus we introduce two linear scale factors vx and vy to control the strength of the gradient for texture warping around the object. Then we use the linear combination of gradient value and 2D spatial coordinates to compute the texture coordinates (ux, uy) for estimating the texture warping (

ux ¼ x þ vx r I g x uy ¼ yþ vy rIg y

ð1Þ

However, the control of the strength of the gradient is tedious task for generating visually pleasing retexturing at different region of the object. Therefore we need to propose an optimization method to automate the step for generating proper strength of the gradient at different points of the object to be retextured. Besides, this texture coordinates computing method do not consider much for underlying texture distortion in the original images. In our method, SIFT feature [24] is extracted to guide the retexturing process to preserving similar underlying texture distortion effects, especially in the wrinkled area where SIFT corner features are normal. Precisely, we propose to compute the SIFT corner strength function s(H) at different points of the objects to estimate the strength of the gradient, where large value of s(H) denotes obvious feature points. Corner strength function can be computed everywhere around the objects, especially in the wrinkled area where corners are normal and distortion naturally appear. Hence, we can accentuate the wrinkle stress in retexturing automatically as shown in Fig. 3. For each point p in the image, consider a window of pixels around that point, we compute the P Harris matrix H for point p, which is defined as H ¼ p wp T rIp ðrIp Þ . Where H is a 2  2 matrix, and wp is chosen as circularly symmetric Gaussian mask for rotation invariance. To find the interest points and enhance the wrinkle stress in the objects of interest, we compute the corner strength function s(H)

for the objects as jH j sðHÞ ¼ trðHÞ 8 < jHj ¼ rI2gx rI2gy ðrIg x rIgy Þ2

ð2Þ

: trðHÞ ¼ rI2g þ rI2g x y where 9H9 is the determinant of the Harris matrix H, tr(H) is the trace of H, and large value of the corner strength function s(H) denotes obvious feature points. After computing s(H) for every point of the objects in images, we choose points where s(H) is above a threshold for further optimization use, here we set the threshold as 0.02 experimentally. Of course we can set the threshold according to the number of feature points we need for certain objects in images. We also need to make sure s(H) is a local maximum in a 3  3 neighborhood. This SIFT corner strength function will be further utilized as description of the strength of the gradient to enhance the wrinkle stress for the objects to be retextured. Within the wrinkled region c, we apply the energy function ESIFT(  ) with SIFT feature guidance to compute again for optimizing the texture coordinates (ux, uy) to warp the input texture properly and preserve the wrinkle stress for maintaining underlying texture distortion in the original image automatically, our objective is to minimize the difference between the texture coordinates (ux, uy) at the position (x, y) and gradient image coordinate components ½x þsðHÞvx rIg x , ½y þsðHÞvy rIg y  at the feature neighboring pixels: 8 R < ESIFT ðux Þ ¼ c :ux ½x þ sðHÞvx rIg x :2 ð3Þ R : ESIFT ðuy Þ ¼ :uy ½y þsðHÞvy rIg :2 y c Now we can utilize the new texture coordinates (ux, uy) computed with feature-guided gradient optimization to perform image retexturing to accentuate the wrinkle stress. This texture coordinates considering the featured area with wrinkle stress in the objects will provide us with similar underlying texture distortion as in the original image. A careful inspection of the texture regions in Fig. 4(b) (e.g. the enlarged area in the red square) reveals that the retexturing technique without optimization destroys the original underlying texture distortion pattern. Fig. 4(c) shows the automatic texture distortion effects using our feature-guided approach in the gradient space. As can be clearly seen, this image reproduces the correct texture distortion, and at the same time it is faithful to the original texture look. Note that human-based mesh stretching [11] is tedious task and unavoidably causes improper texture distortion, i.e. too large or too small distortion. It can be seen from Fig. 4 that the automatic texture distortion effects generated using our approach provide nice texture distortion solution for aesthetical retexturing tasks. Considering a pixel at (x, y) belonging to the object with a RGB color gradient rIg ðx,yÞ, we derive the new retextured color gradient rInew g ðx,yÞ from its original color gradient rI g ðx,yÞ and

Fig. 4. Our feature-guided cloth retexturing showing texture distortion effects on GPU. (a) The input cloth image, (b) retextured image without texture distortion using CPU bilateral filtering, 0.1 frame/s, and (c) our result using parallel bilateral grids on GPU, 79.25 frames/s. Note that our result faithfully preserves the underlying texture distortion in the image, i.e. the enlarged details show the fine distortion effects of our method. (For interpretation of the references to color in this figure legend, the reader is referred to the web version of this article.)

P. Li et al. / Computers & Graphics 36 (2012) 1048–1059

1053

Fig. 5. Feature-guided keyframe retexturing of a vase video using our approach, 93.05 frames/s. (a) The original image, (b) our feature-guided deformable brick texture retexturing using real-time bilateral grid, and (c) our feature-guided deformable circle texture retexturing result. The results produced by our method are very efficient in real-time interaction with fine automatic distortion.

the input texture gradient rT g ðux ,uy Þ using the matting equation

rInew g ðx,yÞ ¼ sr Ig ðx,yÞ þð1sÞr T g ðux ,uy Þ

ð4Þ

where s is a scalar parameter which linearly interpolates data between the original object’s color gradient rIg ðx,yÞ and the texture color gradient rT g ðux ,uy Þ. After we computed the retextured gradient map rInew g ðx,yÞ, we then get the final retextured image Inew by solving a Poisson equation. Fig. 5 shows the feature-guided image/keyframe retexturing of a vase video using parallel bilateral grid optimization with regular textures on GPU, two kinds of regular textures, namely brick texture and circle texture, proved the robustness of our method. And our retexturing can preserve the underlying texture distortion well, especially in the vase’s wrinkled area where distortion is usual. We can see from Fig. 5 that our result preserves the fine distortion due to feature-guided wrinkle energy optimization in the gradient space, which can discover and maintain the underlying distortion automatically. 3.2. Efficient video retexturing After the process of wrinkle energy optimization, it’s now possible to change the texture appearance of objects of interest in single images. Using our feature-guided approach, with just simple interaction for texture inputs and scaling, our method can provide variable texture appearances for different single photograph inputs upon the needs of users with real-time control feedback. However, a more desirable requirement for retexturing is to alter the texture appearance of objects depicted in video clips, which is even more interesting and attractive. The ability to generate new texture appearances for a video sequence offers an alterable way for video animation production, which usually takes time-consuming and money-wasting special after effects processing, or even more tedious work of shooting the video many times with different appearance models setting in the scenes again and again, which is very uncomfortable and ordinary especially in science fiction or fairy tale film shooting. Thus surface texture appearances should be able to apply to objects of interest in video clips directly, e.g. simply and most automatically replacing the texture look of objects or clothes depicted in videos virtually without actually building different types of physical models of the texture material and reshooting the video sequence many times, which is very expensive and hard to control random shooting failure for time consideration. Video retexturing therefore is especially useful but challenging problem because we want mostly automatic processing of generating retextured videos from single view uncalibrated videos with just simple user interaction for inputting different textures, and we need an amazing video retexturing tool specially for making the personal video creation task easy and interesting with real-time control feedback. For our video retexturing, a keyframe-based method is applied to generating the retextured videos, which means keyframes in a

video clip are first extracted and retextured, and we select the keyframe in which the visibility of the concerned objects is maximized. For evaluating frames, we need to fuse together a set of intrinsic image features. We mainly take into account the visual features which represent the frame contents of the concerned objects. The features we used are luminance/color histogram feature, edge feature, gradient histogram feature, wavelet feature, and motion vector for objects with adjacent frames in the original unretextured video clips. Other features such as frequency domain information can be also integrated into our approach. Without losing generality, for a frame, suppose that overall m feature vectors are extracted for the concerned objects. We expand each feature Fi into a one-column vector as [fi,1 fi,2 y fi,n]T, note that different features have different number of elements. We denote the feature sets Mfeatures as [F1 F2 y Fm], and F1 F2 y Fm in Mfeatures are normalized for future processing. Note that here we store the feature sets in Mfeatures just for easy storage and processing, we do not carry out computing among different kind of features, one kind of feature is only compared with its own kind. The feature sets contain useful information of concerned objects content in frames. Besides, they are most probably contaminated by noises. Therefore, we need to take the noise influence away from the useful information of the feature sets. We make assumption that different features have uncorrelated noise characteristics, and noise information is reduced through performing Laplacian pyramid (Difference of Gaussians) filtering to the objects in the original unretextured video clips. Then the clear feature sets are used to compute the keyframes in each video clips. Note that here within each video clips, frames have very high similarity with each other. We need to select the keyframe in which the visibility of concerned objects is maximized. We extract the most representative salient keyframe K from a video clip C, and formulate the problem as an optimization process of finding K, which differs least from C as K ¼ arg min DðK i ,CÞ 1rirn

ð5Þ

where n is the number of frames in C, and D is the dissimilarity measure. Semantic similarity of objects in video frames is implicitly contained in the objects’ feature value difference, and small value of feature value difference tells the two objects are more similar. Here, we select the keyframe of each video clip based on the objects’ feature sets difference with all the frames in a video clip. The dissimilarity measure D is computed as DðK i ,CÞ ¼

1 n

n  X  1Gð:dvisual ðK i ,K j Þ:Þ j¼1

8 2 x > > > GðxÞ ¼ e 2s2 > > > m X < dvisual ðK i ,K j Þ ¼ ws VisSims ðK i ,K j Þ > s¼1 > > >   > > : VisSims ðK i ,K j Þ ¼ C1 F s ðK i ÞF s ðK j Þ2 s

ð6Þ

1054

P. Li et al. / Computers & Graphics 36 (2012) 1048–1059

where, D(Ki,C) is the dissimilarity between potential keyframe Ki and the video clip C; dvisual is the visual distance between potential keyframe Ki and frame Kj; G(d) is the Gaussian distribution function, to estimate the probability of p(dvisual), a common idea is to approximate it with the normal distribution, which is usually modeled using Gaussian functions; VisSims(Ki,Kj) is the visual similarity of feature s between frame Ki and Kj, and ws is the related weight; Fs(d) is the value of feature s, and Cs is the related normalizing coefficient. Note that the values of D(Ki,C) are between 0 and 1, which is good property for dissimilarity measure functions and easy for processing. Actually, people can choose to compute the dissimilarity measure D for different feature set independently and set different goal for different feature to carry out a multi-objective optimization for selecting the keyframes. However, here we choose to experimentally extract ws to combine different feature set into one dissimilarity measure. Although there will be more testing to go for choosing proper ws, yet once done we will be possible to transfer a multiobjective optimization into a single objective optimization and save a lot of computation, which is suggested by researchers from mathematical modeling. After the keyframe selection process and retexturing of objects of interest in keyframes using our feature-guided approach by bilateral grids, we then utilized the latest GPU-based accurate TV-L1 optical flow tracking [42] to transfer the new textures from the retextured keyframes to the neighboring frames. Video retexturing is much more complicated than retexturing a single image, for retexturing video, simply applying the image retexturing approach frame by frame will cause texture drifting phenomenon, which will make the generated texture appearance of the concerned objects in the retextured video look very unstable and inconsistent. The underlying reason for this drifting phenomenon is that the replaced texture on the objects in each frame is too different to be consistent with each other. After the wrinkle energy optimization for texture distortion, together with the objects’ motion in many directions, the position for the textures in each frame differs a lot and each retextured pixel of the objects of interest mostly changes over the adjacent frames. If played, the frame-by-frame retextured video clips will display the phenomenon of texture flickering, and affect the users’ understanding of the video animation a lot. Here, we propose a keyframe-based method to transfer the pixels of the retextured objects in the keframes to the other frames through the accurate TV-L1 optical flow tracking. To increase the accuracy for the tracking of the

concerned objects in videos, block motion techniques are employed to extract the TV-L1 optical flow. Therefore, tiny unobvious motions of extraessential pixels will be ignored, which is usually inaccurate, however, major motions of small pixel blocks (5  5 here) will be tracked accurately along the timeline in video clips. The bock motion techniques also reduce the computational complexity of the optical flow computing process, thus make our video retexturing more efficient besides the GPU acceleration. The concerned objects in the selected keyframes usually have maximal visibility, the correspondences between the objects in the original unretextured keyframes and the neighboring frames are established. The objects in original keyframes are then tracked forward and backward throughout the video clips using robust TV-L1 optical flow with block motion techniques. The image retexturing results of the objects in keyframs are finally propagated to other frames one by one repeatedly using more accurate block patches transfer instead of pixel level processing. To increase accuracy for the robust TV-L1 optical flow, we add a parallel pre-smoothing to the concerned objects in videos using GPU-CUDA before the tracking and texture transferring process, which is very efficient in reducing noise information and makes the flow computing process to be more accurate and robust.

3.3. Interactive parallel retexturing We have developed the retexturing with simple user control interface for human-computer interaction. There are mainly two problems need to be addressed in the interactive retexturing system, namely, easy control and real-time feedback. For the task of easy control, the most important thing is to know our users. Our retexturing work is designed both for assisting users in artistic fashion design and for people who are interested in daily photo editing and sharing. Therefore, our interactive retexturing is designed not only for professional artists but also amateur lovers, and these people are all intuitive based users. This characteristic determines that we should not offer intricate parameters for them to tune as usually a system for scientists and engineers will do. We gained suggestions from both novice users and texture design experts, and finally designed the retexturing system, which can easily map the users’ intentions and aims through our retexturing interfaces. The input and output of the system are very intuitive. Users can easily interpret and evaluate the outcome that displayed to them by our interactive

Fig. 6. Interactive parallel retexturing control interface. Our system provide users with simple and real-time feedback for interesting objects definition, texture selection from library, texture size adjusting, overall texture distortion strength tuning, overall lighting and transparency controlling on a latest hardware platform with GPU parallelism.

P. Li et al. / Computers & Graphics 36 (2012) 1048–1059

retexturing interface. Fig. 6 shows the control interface of our interactive parallel retexturing. Instead of offering a fully automatic parallel retexturing with GPU acceleration, we provide the users with a real-time feedback system that can replace the existing texture of objects with user preferred textures while preserving the original shading and similar texture distortions. Our retexturing system takes images/videos with user desired textures as input, recovers the gradient depth maps, optimizes texture distortion in wrinkled area with feature guidance, and finally outputs the visually pleasing retextured results. The system offers the users with simple and real-time feedback interaction for target objects definition using alpha map, different texture loading from texture library, texture size adjusting on a latest parallel GPU-CUDA platform. Users also can adjust the overall lighting by controlling the brightness scroll bar, transparency is realized by linearly interpolating between the original and the retextured objects. In our system, the strength of the gradient at different point is computed automatically using SIFT corner strength function, however, if the users were still not fully satisfied with the effects of new texture warping around the objects, we offer them with a global control of texture distortion strength tuning. Global control is easy and sufficient enough because the local strength of the gradient is naturally discovered by SIFT corner strength function. With easy control and real-time feedback, the retexturing results displayed in the after window of the interface will be refreshed very efficiently in real-time with each processing step made, which gives out intuitive control for both professional and amateur users. Real-time feedback is very essential in interactive system because low efficient processing will greatly affect the users’ experience in graphical and virtual arts design. To enhance the users’ immersive feeling for the retexturing system, we apply latest GPU-CUDA version 4.0 to carry out the real-time parallel retexturing with immediate display feedback. The latest GPUs are actually highly parallel, multi-thread, and many-core stream processors with amazing computing ability. With the GPGPU ability emerged in CUDA, we can explore general purpose programming model with shared memory and streaming processing ability of multiple block and thread features to accelerate the originally time-consuming retexturing. Bilateral grid is a primary data structure that enables a variety of edge-aware operations. With one more axis to represent the pixel intensity, bilateral grid is able to achieve much faster image/video processing. The main feature of the bilateral grid is utilizing 3D structure to represent 2D data, e.g. 3D representation of 2D images. Thus axis x and y

1055

correspond to pixel position in 2D images, axis z corresponds to the pixel intensity. The Euclidean distance in bilateral grid accounts for edges, with space distance (x, y) and intensity distance (z). The data structure is first introduced by Paris and Durand [30]. Chen et al. [4] treated the data structure differently, and expanded the bilateral grid as a primary data structure which enables a variety of edge-aware operations. Hence they take more advantage of the extra data dimension created with the original 2D images. In this paper, we utilize the novel 3D data structure, the GPU-based edge-aware bilateral grid, to manipulate the gradient domain, to convert the image gradient space information into approximate geometry details for retexturing. By lifting image retexturing into a higher dimensional space, we are able to develop retexturing algorithm that naturally preserving strong edges in images. Our feature-guided retexturing using bilateral grid maps well onto GPUs and enables real-time interactive retexturing of high resolution images. Fig. 7 shows our featureguided real-time retexturing using parallel bilateral grid processing in the gradient space on GPU-CUDA platform, all the data are processed in parallel with wrinkle stress enhancement optimization. It can be seen from Fig. 7(b) that the retexturing method without winkle energy minimization destroys the original underlying texture distortion pattern. However, Fig. 7(c) using our approach gives out nice brick texture retexturing result while reproduce the correct texture distortion at the same time. For GPU CUDA coding we set the total number of blocks to be 128 and total number of threads within a block to be 256, thus totally 128  256 threads are available in our program for computing the bilateral grid according to the objects’ coordinates, which is sufficient enough for the bilateral grid manipulation and retexturing processing. The pseudo code for computing bilateral grid manipulation kernel on GPU-CUDA is outlined as follows: __global__ static void GPU_grid (float **objectsIndex, float *inputIntensity, float *outputIntensity) { tix¼ threadIdx.x; //Get thread index tiy ¼threadIdx.y; x ¼blockIdx.x * blockDim.x þtix; //Get global index y¼blockIdx.y * blockDim.yþtiy; //Bilateral grid construction BG(i, j, k) ¼(0, 0); // Initialize all bilateral grid nodes if (x and y within objectsIndex) { //Accumulate the input depth intensity //and the number of pixels into each grid cell       BGð x=s , y=s , Id ðx,yÞ=i Þ þ ¼ ðId ðx,yÞ, 1Þ; g

Fig. 7. Our real-time interactive stone lion retexturing using parallel bilateral grid processing with brick texture. (a) The original stone lion image, (b) retexturing without texture distortion using CPU bilateral filtering, 0.09 frame/s, and (c) our feature-guided retexturing preserving texture distortion effects using parallel bilateral grid processing on GPU, 75.31 frames/s. Note that, in the wrinkled area, the new applied brick texture is distorted well around the winkle in (c), which can be easily seen from the enlarged window.

1056

P. Li et al. / Computers & Graphics 36 (2012) 1048–1059

//Obtain new bilateral grid by Gaussian convolution BGnew ¼fGaussian(BG); //Coefficient ss and sr //Depth map extraction by slicing if (x and y within objectsIndex) { //Extract the output smoothed depth intensity //by accessing the grid at (x/s, y/s, Id(x, y)/i) Isd ¼ slicing Id ðBGÞ; gg The global qualifier declares the function GPU_grid() as being a kernel, the function is executed on GPU, and callable from the CPU only. Once called, the bilateral grid kernel will run efficiently in parallel on multiple GPU cores. There are grid.x  grid.y blocks run synchronously in the GPU grid, and thread.x  thread.y threads also run synchronously within each block. Totally, there are (grid.x  grid.y)  (thread.x  thread.y) threads run synchronously to perform the fast bilateral grid processing kernel GPU_grid() to compute for the bilateral filtering for all the pixels in parallel on GPU-CUDA platform, to achieve real-time interactive image/video retexturing preserving underlying texture distortion.

4. Experimental results In the experiments, we used Cþþ with GPU CUDA version 4.0 to develop our feature-guided real-time interactive retexturing. The example images/videos have been tested on an Intel(R) Core(TM)2 Duo CPU 2.3 GHz PC with a NVIDIA GeForce 8800 GPU and 2 GB RAM. We have tested many examples for feature-guided retexturing using bilateral grid processing with gradient optimization. Fig. 8 shows our feature-guided retexturing results of a sculpture image using parallel bilateral grid optimization with random and regular textures on GPU-CUDA. Two kinds of regular brick textures in different direction and color with distortion effects proved the robustness of our energy optimization in the sculpture’s wrinkled area, where corner and distortion normally appear. We can see from Fig. 8 that our result preserves the fine texture distortion due to feature-guided

wrinkle stress optimization in the gradient space, which discovers the underlying texture distortion of objects in the input images automatically. Besides, compared to the synthesis approach [7], our result produces less chapped brick texture appearance. Also, compared to the existing retexturing and synthesis approach, our results produce realistic retexturing with much less computing time due to real-time GPU parallelism avoiding tedious human mesh stretching for deforming the texture conforming to the underlying texture distortion as is needed in [11], which will unavoidably cause improper texture distortion, too large or too small. Fig. 9 shows the feature-guided real-time girl dress retexturing with asymmetric textures, where in the input image, the girl’s both arms are bent which tells that there are global changes in texture direction, but still our interactive retexturing with energy optimization produces realistic retexturing even for bent parts where there are global texture direction changes. Fig. 10 shows the vase video retexturing with bilateral grid processing using GPU parallelism. We apply robust TV-L1 optical flow to track the motion of the vase, and the new regular texture is transferred well in the video clip, giving out real-time realistic retexturing using parallel GPU-CUDA acceleration. Regular circle texture is applied for easy checking of nice texture distortion effects produced along the whole vase video. Fig. 11 shows the feature-guided girl video retexturing using edge-aware parallel bilateral grid processing, our GPU-based retexturing of girl’s costume using bilateral grid processing with feature guidance and wrinkle stress optimization preserves the underlying texture distortion well, also in the costume’s wrinkled area. Clothing retexturing also facilitates some fashion design tasks, real models may not be needed for trying costume, and we can try new textured cloth virtually on certain style dress instead of making real samples to see whether the new textured cloth fit the dress or not. This advantage saves money and time for costume design, and retexturing also can be used for testing whether certain kinds of appearance design fit for the wall of a building as shown by clothes retexturing in Figs. 1, 3, 9 and 11.

Fig. 8. Feature-guided sculpture image retexturing using our approach, 76.35 frames/s. (a) The original image, (b) retexturing—random stone texture, (c) retexturing— regular level brick texture, and (d) retexturing—regular acclivitous brick texture.

Fig. 9. Our feature-guided real-time girl dress retexturing using parallel bilateral grids in the gradient domain, 68.11 frames/s. (a) The input image, (b) recovered gradient depth map of the dress using bilateral grid manipulation, (c) and (d) retexturing with asymmetric textures, which is distorted well using energy optimization. In the input image, the girl’s both arms are bent which means there are global changes in texture direction, and our approach with feature energy optimization can reproduce realistic retexturing even for parts like bent arms, the underlying texture distortions is preserved well.

P. Li et al. / Computers & Graphics 36 (2012) 1048–1059

1057

Fig. 10. Video retexturing using parallel bilateral grid processing, 93.05 frames/s, showing the original and retextured frames of 1st, 68th, 128th and 188th, with frame 1 as the only keyframe for the whole video clip because the original vase have maximized visibility. (a) The original input video, (b) circle texture retexturing with texture distortion effects. We apply robust TV-L1 optical flow to track the motion of the vase, and the new textures with distortion effects are transferred well along the video clip, giving out real-time realistic retexturing using lates GPU-CUDA parallelism.

Fig. 11. Feature-guided girl video retexturing using parallel bilateral grid processing, 80.06 frames/s, showing the original and retextured frames of 1st, 25th, 45th and 60th, with frame 1 as the keyframe for the whole video shot since the girl’s clothes has maximum visibility. (a) The original input video, (b) our feature-guided retexturing of the girl’s costume preserving texture distortion effects using parallel bilateral grid processing, and propagation the clothes texture using latest GPU-accelerated accurate TV-L1 optical flow. (Recreated from source: Yung-Yu Chuang, used with permission.)

Table 2 Timing performance (frame/s) for retexturing using bilateral filtering (CPU only) and GPU accelerated methods. Example

Fig. 1

Fig. 3

Fig. 4

Fig. 7

Fig. 8

Fig. 9

Fig. 10

Fig. 11

Resolution CPU only GPU bilateral filtering GPU Without distortion bilateral grid With distortion

360  240 0.76 21.15 85.61 81.90

1131  1200 0.03 13.02 69.55 62.83

550  355 0.10 18.53 81.36 79.25

768  994 0.09 16.87 79.28 75.31

750  811 0.06 14.59 78.93 76.35

1200  1200 0.08 15.16 73.29 68.11

480  320 0.83 23.91 97.68 93.05

640  480 0.32 19.79 84.17 80.06

Timing performances (frame/s) for interactive retexturing using CPU-bilateral filtering and GPU-accelerated methods are tested and shown in Table 2, interest object identification step is not counted in, all the timing performance is tested under an automatic retexturing scheme, which means all the user interaction values are set to default value. Fig. 12 shows retexturing using CPU bilateral filtering (CPU), GPU bilateral filtering (GPU1), GPU bilateral grid without texture distortion (GPU2), and our feature-guided interactive retexturing using parallel bilateral grids with texture distortion (GPU3). Here, we can see that our feature-guided interactive image/video retexturing with bilateral grid processing using GPU parallelism runs very efficiently in

real-time, and much faster than the traditional CPU-based realization and the GPU-based bilateral filtering realization, and our retexturing with feature guidance produces more realistic retexturing preserving fine underlying texture distortion, shown by the examples. Our image/video retexturing using bilateral grid processing with feature guidance can run automatically or in an interactive manner, providing the users with easy control of image/video retexturing with real-time feedback using latest GPU parallelism. In [11], user needs to perform large amount of mesh stretching to deform the new texture conforming to the underlying image geometry, and human-based mesh stretching is unavoidably improper, either too large or too small. We perform

1058

P. Li et al. / Computers & Graphics 36 (2012) 1048–1059

Fig. 12. Timing performance for retexturing using CPU only and GPU-accelerated methods. Our feature-guided interactive retexturing using GPU parallelism (GPU3) gives out realistic retexturing with much high performance in terms of frame/s.

the retext in gradient space with feature-guided wrinkle energy optimization, and the texture distortion effects are generated automatically based on the wrinkle energy computed. Besides, our approach is able to replace the existing texture with user preferred textures while preserving the original shading and similar texture distortions well and robustly. The images/videos we tested have shown the high-quality effects and real-time interactive performance of our feature-guided retexturing using latest GPU parallelism.

5. Summary and future work Existing retexturing and texture synthesis approaches deal with texture distortion by inaccurate inter-pixel distances manipulation in image space, and the underlying texture distortion of the original images is always destroyed while retexturing. The long processing time due to time-consuming filtering and software coding only is also unacceptable. In this paper, we present to apply SIFT corner features to naturally discover the underlying texture distortion of the objects within the target images/videos. Gradientbased depth map recovery and feature-guided optimization are utilized to accomplish the distortion process. We facilitate the interactive image/video retexturing via real-time bilateral grid on GPU, while preserving similar underlying texture distortion with feature guidance. Our interactive retexturing system offers users with real-time feedback for easy control of target objects definition, texture selection with size adjusting, and overall lighting tuning on latest GPU platform. We perform the interactive retexturing directly in the gradient space with feature optimization guidance. Video retexturing is further achieved by a keyframe-based texture transferring process using real-time TV-L1 optical flow with patch-based block motion tracking techniques. Compared with other retexturing approaches, our method focuses on real-time image/video processing, which gives out a solution for efficient easy control feedback for retexturing interaction. Multiple experiments have shown the high-quality realistic retexturing of interactive images/videos with real-time feedback using our approach with latest GPU parallelism. Our future work includes investigating different robust optical flows and extending the interactive retexturing to more complicated video applications with large motion and occluded scene avoiding textures flickering. Specially, we will put our focus in addressing the consistency between keyframes when there are more than one keyframe required in a video for retexturing. One simple solution is that we retexture the keyframes independently with similar texture. Suppose two keyframes’ new textures are transferred forward and backward throughout the video clip

using robust optical flow, we perform a novel transition when the new retextured frames source from two keyframes meet. The transition algorithms can be a solution. Another possible solution we will investigate is that we know the inner reason why more than one keyframe is needed for retexturing, mainly due to occluded regions happen within a certain frame range. However, the occluded regions may be found again after certain amount of frames passed in a video. Then it will be possible to apply a main keyframe with assistant keyframes approach for video retexturing. The retextured main keyframe will transfer texture to the assistant keyframes, of course there will be texture missing in some parts of the assistant keyframes due to occlusion happens, but still we can again perform a retexturing or texture synthesis in the texture missing regions of the assistant keyframes, then it will be possible to retexture the whole video consistently with these consistent keyframes. However, using this method, two problems need to be addressed, namely, texture break caused by texture synthesis on the texture missing regions of the assistant keyframes, and cranny caused by retexturing on the texture missing regions of the assistant keyframes, which can be possibly addressed using method like Poisson image blending/Editing [32]. We will also investigate new approaches to make video retexturing more stable by inspiration from research fruit of video stabilization or other latest video processing techniques.

Acknowledgments The authors would like to thank the anonymous reviewers for their helpful comments and suggestions, which helped improve the paper. We thank Jiawen Chen, Yanwen Guo, Yung-Yu Chuang, Hui Fang and John C. Hart for providing test images and helpful discussion. This work was supported by RGC research Grant (Ref. 416007) and UGC direct Grant for research (No. 2050454, and 2050485). References [1] Barnes C, Goldman D, Shechtman E, Finkelstein A. Video tapestries with continuous temporal zoom. In: Proceedings of ACM SIGGRAPH; 2010. [2] Bennett E, McMillan L. Video enhancement using per-pixel virtual exposures. In: Proceedings of ACM SIGGRAPH; 2005. p. 845–52. [3] Bertalmio M, Fort P, Sanchez-Crespo D. Real-time, accurate depth of field using anisotropic diffusion and programmable graphics cards. In: Proceedings of 3DPVT; 2004. p. 767–73. [4] Chen J, Paris S, Durand F. Real-time edge-aware image processing with the bilateral grid. In: Proceedings of ACM SIGGRAPH; 2007. [5] Correa C, Ma K. Dynamic video narratives. In: Proceedings of ACM SIGGRAPH; 2010. [6] Debevec P, Malik J. Recovering high dynamic range radiance maps from photographs. In: Proceedings of ACM SIGGRAPH; 1997. p. 369–78. [7] Fang H, Hart J. Textureshop: texture synthesis as a photograph editing tool. In: Proceedings of ACM SIGGRAPH; 2004. p. 354–9. [8] Fang H, Hart J. Rototexture: automated tools for texturing raw video. IEEE Trans Visual Comput Graphics 2006;12(6):1580–9. [9] Fatahalian K, Boulos S, Hegarty J, Akeley K, Mark W, Moreton H, et al. Reducing shading on GPUs using quad-fragment merging. In: Proceedings of ACM SIGGRAPH; 2010. [10] Guo Y, Wang J, Zeng X, Xie Z, Sun H, Peng Q. Image and video retexturing. Comput Anim Virtual Worlds 2005;16:451–61. [11] Guo Y, Sun H, Peng Q, Jiang Z. Mesh-guided optimized retexturing for image and video. IEEE Trans Visual Comput Graphics 2008;14(2):426–39. [12] Hachisuka T, Jensen H. Parallel progressive photon mapping on GPUs. In: Proceedings of SIGGRAPH Asia sketches; 2010. [13] Hilsmann A, Schneider D, Eisert P. Realistic cloth augmentation in single view video under occlusions. Comput Graphics 2010;34(5):567–74. [14] Hilsmann A, Schneider D, Eisert P. Image-based retexturing of deformed surfaces from a single image. In: Proceedings of eurographics; 2011. [15] Hoefflinger B. High-dynamic-range (HDR) vision. Berlin: Springer; 2007. [16] Horn B, Brooks M. Shape from shading. Mass: MIT press Cambridge; 1989. [17] Kazhdan M, Hoppe H. Streaming multigrid for gradient-domain operations on large images. In: Proceedings of ACM SIGGRAPH; 2008. [18] Khan E, Reinhard E, Fleming R, Bulthoff H. Image-based material editing. In: Proceedings of ACM SIGGRAPH; 2006. p. 654–63.

P. Li et al. / Computers & Graphics 36 (2012) 1048–1059

¨ ¨ [19] Krahenb uhl P, Lang M, Hornung A, Gross M. A system for retargeting of streaming video. In: Proceedings of ACM SIGGRAPH Asia; 2009. [20] Levin A, Rav-Acha A, Lischinski D. Spectral matting. IEEE Trans Pattern Anal Mach Intell 2008;30(10):1699–712. [21] Li P, Sun H, Huang C, Shen J, Nie Y. Efficient image/video retexturing using parallel bilateral grids. In: Proceedings of ACM SIGGRAPH VRCAI; 2011. p. 131–40. [22] Liu F, Gleicher M, Wang J, Jin H, Agarwala A. Subspace video stabilization. ACM Trans Graph 2011;30(1). [23] Liu Y, Lin W, Hays J. Near-regular texture analysis and manipulation. In: Proceedings of ACM SIGGRAPH; 2004. p. 368–76. [24] Lowe D. Distinctive image features from scale-invariant keypoints. Int J Comput Vision 2004;60(2):91–110. [25] Mavridis P, Papaioannou G. High quality elliptical texture filtering on GPU. In: Proceedings of ACM SIGGRAPH symposium on interactive 3D graphics and games; 2011. p. 23–30. [26] McCann J, Pollard N. Real-time gradient-domain painting. In: Proceedings of ACM SIGGRAPH; 2008. [27] Nehab D, Maximo A, Lima R, Hoppe H. GPU-efficient recursive filtering and summed-area tables. In: Proceedings of SIGGRAPH Asia; 2011. [28] Oh B, Chen M, Dorsey J, Durand F. Image-based modeling and photo editing. In: Proceedings of ACM SIGGRAPH; 2001. p. 433–42. ¨ [29] Orzan A, Bousseau A, Winnemoller H, Barla P, Thollot J, Salesin D. Diffusion curves: a vector representation for smooth-shaded images. In: Proceedings of ACM SIGGRAPH; 2008. [30] Paris S, Durand F. A fast approximation of the bilateral filter using a signal processing approach. In: Proceedings of the european conference on computer vision; 2006.

1059

[31] Parker S, Bigler J, Dietrich A, Friedrich H, Hoberock J, Luebke D, et al. OptiX: a general purpose ray tracing engine. In: Proceedings of ACM SIGGRAPH; 2010. [32] Pe´rez P, Gangnet M, Blake A. Poisson image editing. ACM Trans Graph 2003;22(3):313–8. [33] Pilet J, Lepetit V, Fua P. Retexturing in the presence of complex illumination and occlusions. In: Proceedings of ISMAR; 2007. p. 249–58. [34] Scholz V, Magnor M. Texture replacement of garments in monocular video. In: Proceedings of EGSR. Rendering Techniques; 2006. p. 305–12. [35] Shen J, Jin X, Sun H. High dynamic range image tone mapping and retexturing using fast trilateral filtering. Visual Comput 2007;23(9):641–50. [36] Tiilikainen N, Bartoli A, Olsen S. Contour-based registration and retexturing of cartoon-like videos. In: Proceedings of the British machine vision conference; 2008. [37] Tsin Y, Liu Y, Ramesh V. Texture replacement in real images. In: Proceedings of IEEE CVPR 2; 2001. p. 539–44. [38] Wang Y, Hsiao J, Sorkine O, Lee T. Scalable and coherent video resizing with per-frame optimization. In: Proceedings of ACM SIGGRAPH; 2011. [39] White R, Forsyth D. Retexturing single views using texture and shading. In: Proceedings of the european conference on computer vision; 2006. p. 70–81. ¨ [40] Winnemoller H, Olsen S, Gooch B. Real-time video abstraction. In: Proceedings of ACM SIGGRAPH; 2006. [41] Xu F, Liu Y, Stoll C, Tompkin J, Bharaj G, Dai Q, et al. Video-based characters: creating new human performances from a multi-view video database. In: Proceedings of ACM SIGGRAPH; 2011. p.1221–6. [42] Zach C, Pock T, Bischof H. A duality based approach for realtime TV-L1 optical flow. In: Proceedings of the 29th DAGM conference on Pattern recognition; 2007. p. 214–23.