- Email: [email protected]
Obtaining virtual lighting condition based on images using NNs
Image and Vision Computing 25 (2007) 1249–1254 www.elsevier.com/locate/imavis
Obtaining virtual lighting condition based on images using NNs Zhanwei Li
a,* a
Guolin Duan a, Jizhou Sun b, Xinran Lv
a
Hebei University of Technology, Tianjin 300130, China b Tianjin University, Tianjin 300072, China
Received 20 April 2005; received in revised form 11 January 2006; accepted 24 July 2006
Abstract Using Neural Networks, this paper present a method of generating virtual images under virtual lighting conditions. The relationship between the lighting conditions and the color of image is non-linear, several neural networks technologies are introduced to make curve fitting. As a result, virtual images of same scene under different lighting conditions can be deduced from known real scene image, and the inherent relations of image color and outside lighting conditions are discovered. Combining techniques of IBMR or panoramic image, the model of scene is recovered. Further, combining the dynamic virtual images obtained by our method, time-sequence dynamic virtual scene can be reconstructed and revisited in virtual reality. Ó 2006 Elsevier B.V. All rights reserved. Keywords: Image-based rendering (IBR); Neural networks (NNs); Virtual images; Lighting conditions; Dynamic reconstruction; Virtual reality
1. Introduction One of the important aims of the computer graphics is to render realistic image. Image-Based Rendering techniques are powerful alternatives to traditional geometrybased techniques for image synthesis. IBR techniques render novel viewpoint images from input images. Acquiring geometric models of scene has been the subject of research in interactive Image-Based Modeling (IBM) techniques, and is now becoming practical to perform with techniques such as interactive photogrammetry. Acquiring the corresponding appearance information (under given lighting conditions) is easy to performed with a digital camera. Image-based rendering can have many interesting applications. Two scenarios, large environments and dynamic environments, in particular, are worth pursuing [1]. Until now, most of image-based rendering systems have been focused on static environments.
*
Corresponding author. Tel.: +86 022 60202273. E-mail addresses: [email protected] (Z. Li), [email protected] (G. Duan), [email protected] (J. Sun). 0262-8856/$ - see front matter Ó 2006 Elsevier B.V. All rights reserved. doi:10.1016/j.imavis.2006.07.029
With the development of panoramic video systems [2,3], it is conceivable that image-based rendering can be applied to dynamic environments as well, but for panoramic video the lighting conditions are static, although scenery is dynamic. Debevec [4] put forward projecting real images from dawn to dusk in sun, clouds and even fog weather condition to recovered model, the scene of different lighting conditions is recurred. In his method scene images are real images and are obtained in advance, the geometry model of the scene is computed using IBM techniques. Image-Based Lighting (IBL) techniques [5,6] recover the reflectance properties of all surfaces in a real scene. But geometric model of the scene and the position of lighting are needed in IBL, and its arithmetic is very complicated. In this paper, we propose to reconstruct virtual images at appointed lighting condition. At different lighting condition the same scene is differently lighted. We can infer the virtual images in virtual outside lighting conditions by finding the rule of image color changing with outside conditions. Using IBM techniques, the model of scene is recovered. Further, combining the dynamic virtual images obtained by our method, time-sequence dynamic virtual
1250
Z. Li et al. / Image and Vision Computing 25 (2007) 1249–1254
scene can be reconstructed. It can be applied in virtual reality to recur virtual scene. Over the past decade, neural network technologies have found their way into numerous competitive industries. Neural Networks are touted for their predictive accuracy compared with more common linear modeling methods. This paper adopts several neural networks technologies to deduce the relationship between lighting conditions and color of images. 2. Analysis of image colors changing with lighting conditions At different time of day or different weather conditions such as noon or night, sunshine or fog, the same scene are different distinctly. Eyes can distinguish the fine variations of colors. So we can accurately recognize the outside lighting conditions of scene. Seeing an image we can approximately know the lighting conditions, time, weather and so on when the image is generated. The same object with different lighting conditions will present different colors in our eyes. If the change rules can be found by analysing the changes of colors, then we can obtain virtual images under different lighting conditions from a real image. But the relation between image color
and outside lighting conditions are so complicated, even little change will influence the lighting condition, further influence the color of image. Therefore it is difficult to show using linear modeling. Neural network is a good tool to solve the perplexing problem, but how to construct neural network models and define the samples are important. In order to study the change rules between color and lighting condition, we choose 28 colors as samples and shot these colors at different time of day (see Fig. 1(a and b)). The 28 colors are symmetrically allocated in 24 bit colors. The camera is KODAK DC240. Flash lamp was shut down in order to get lighting information accurately. It is obvious from Figs. 1(a) and (b) that the effect of lighting and shades is different from the shooting time. The 28 colors in image shot at 10:10 AM are the input samples, and that at 17:30 PM are the output samples, which are both in Table 1. The input variable (image) is image c in Fig. 1. The output will be a virtual image at lighting condition of 17:30 PM, and it can be compared with the real image shot at 17:30 PM (Fig. 1(d)). This paper adopt image color curve (x-axis: x-coordinate of corresponding image, y-axis: average of pixel color) and image pixels statistics curve (x-axis: pixel energy value, y-axis: sta-
a
b
Color samples at 10:10AM
Color samples at 17:30PM
c
d
Real images shot at 10:10AM
Real images shot at 17:30PM
e 240
f 10:10AM
220
500 400 300 200
200
100 0 0
180
17:30PM
160
50
100
150
200
250 300
350
400 450
50
100
150
200
250 300
350
400 450
500 400 300
140
200 100
120 0
20 40 60 80 100 120 140 160 180 200
Color curve of image
0 0
Pixels statistics curve of image
Fig. 1. (a and b) are color maps shot at different time of a day. It is obvious that the image colors change with the lighting conditions. (c and d) are picture images shot at different time. (e and f) are, respectively, color curve and pixels statistics curve of image (c and d).
Z. Li et al. / Image and Vision Computing 25 (2007) 1249–1254
1251
Table 1 Input and output samples Output samples (17:30)
152, 52, 51 70, 110, 125 90, 39, 39 37, 77, 82 90, 86, 57 25, 75, 118 21, 44, 82
84, 114, 72 181, 162, 51 51, 84, 59 109, 100, 48 177, 166, 96 90, 124, 133 147, 35, 59
57, 90, 132 194, 190, 186 50, 70, 95 112, 106, 100 50, 78, 75 42, 77, 111 107, 66, 46
168, 47, 89 47, 45, 45 139, 66, 97 156, 151, 140 87, 127, 99 83, 108, 148 181, 93, 62
Input samples (10:10)
146, 63, 69 72, 118, 141 85, 50, 50 42, 81, 90 80, 85, 62 31, 73, 113 26, 45, 80
74, 107, 76 160, 151, 55 49, 81, 61 93, 93, 53 157, 155, 96 76, 116, 130 119, 35, 61
55, 92, 139 185, 189, 197 41, 65, 96 96, 100, 104 42, 72, 75 37, 72, 109 88, 61, 50
154, 55, 101 49, 50, 53 120, 59, 103 143, 148, 149 80, 124, 102 77, 108, 155 164, 87, 64
tistic amount) to analyze and compare images. From color curve (Fig. 1(e)) the two real images have a gap in color. From pixels statistics curve (Fig. 1(f)) the energy of the two images is different. 3. Create virtual lighting condition based on images using NNs
in Fig. 3. The third curve is from virtual images at 17:30 PM adopted Linear Neural Networks. The first and second curves are respectively real image at 17:30 PM and 10:10 AM. The third curve tends from second curve to the first, but still different distinctly. It shows that Linear Neural Networks is not fit to the problem. 3.2. Fixed kernel width of GRNNs
Several neural networks will be try below to check the result of forecasting image under virtual lighting condition. If we could find out the changing rules of colors with lighting conditions, we would be able to deduce virtual images under virtual outside conditions from a real image. 3.1. Linear neural networks Linear Neural Networks are often used in function approximation. The computing virtual image is show in Fig. 2(a) using linear NN. Its pixels statistics curve is show
a
General Regression Neural Networks (GRNNs) is presented by D.L.Specht [7] in 1991. GRNNs is effective with only a few samples and even with sparse data in multidimensional space. GRNN is radial basis function networks as well as probabilistic neural networks. It can be used in function approximation, and hence image interpolation. The kernel width r of GRNNs can adjust the perceptive region of nerve cells. Small value for r means that the perceptive region of nerve cells is small, and the output contribution is mainly from the nearby nerve cells, so computing
c
b
Linear NN
Fixed Kernel Width of GRNNs
Holdout method
e
d
dynamic adjustment of kernel width of GRNNs
Increment Addition
Fig. 2. Virtual images at 17:30 PM from five methods.
1252
Z. Li et al. / Image and Vision Computing 25 (2007) 1249–1254
4
3 500
500
real image 00 400
50
100
150
200
250
300
350
400
0 0 400
50
100
150
200
250
300
350
400
450
50
100
150
200
250
300
350
400
450
400
450
400
450
200
200
input image
00 400
50
100
150
200
250
300
350
400
450
0 0 500
1
200 00 400
50
100
150
200
250
300
350
3 400
450
400
450
0 0 500
50
100
150
200
250
300
350
50
100
150
200
250
300
350
2
200 0
450
0
50
100
150
200
250
300
350
4 0
0
Fig. 3. Pixels statistics curve of virtual images adopted linear NN (1) and holdout method (2). Fig. 4. Pixels statistics curve of virtual images adopted dynamic adjustment of kernel width (3) and fixed kernel width of GRNN (4).
result is influenced by fewer nerve cells and interpolation is relatively accurate, but serious steep. Large value for r means that the perceptive region of nerve cells is large, the output is from both the nearby nerve cells and farther ones, so computing result is influenced by more nerve cells and interpolation is smooth but noisy and inaccurate. Sometime there is no sample in the perceptive region of input vectors, so the computing is failure for dividing by zero. The situation happens especially when samples are not symmetrically allocated. If we choose a small r then perceptive region is small. Because the samples are sparse, there are serious color abruptness, bad smoothness and scarce colors in the image. If we choose a large r, there is wide perceptive region, good color smoothness and abundant colors but serious color distortion. The computing virtual image obtained from an optimizing r is show in Fig. 2(b) using fixed kernel width of GRNNs. The fourth curve in Fig. 4 is its pixels statistics curve. The first and second curve are respectively from real image at 17:30 PM, 10:10 AM. The curve of virtual image is still a wide distance to the curve of real image. In conclusion constant kernel width in GRNNs can not well adapt to input vectors, so method of dynamic kernel width below is adopted.
3.4. Dynamic adjustment of kernel width of GRNNs [8,9] Holdout method still can not well adapt every input vector of different distance with samples. So we present to adjust r dynamically. r is chosen in Formula 3.1, pffiffiffiffi T a ri ¼ C ½ðX X i Þ ðX X i Þ ð3:1Þ where a is a number between 0 and 1, C is a constant. Xi are inputs samples, X is input vector. When the distance between samples and input vector is large, there is no sample near the input vector, so in order to search for samples in wider regions, we should choose a larger r. When the distance between samples and input vector is short, there are samples near the input vector. Hence it is not necessary to search for samples in large regions. As a result, choosing smaller r is OK. Thus such images are colorful and not distorted. Optimizing a and C is chosen using Levenberg–Marquardt [10]. The virtual image computing from an optimizing a and C is show in Fig. 2(d) using dynamic adjustment of kernel width of GRNNs. Fig. 4 shows pixels statistics curve of virtual images adopted Dynamic Adjustment of Kernel Width. The third curve tends from second curve to first better than the fourth one.
3.3. Holdout method: Diverse kernel width of GRNNs [7] 3.5. Increment addition Holdout method is a useful method of selecting kernel width. First, removing one sample at a time and constructing a network based on all of other samples. Then the network is used to estimate output for the removed sample. By repeating this process for each sample and storing each estimate, then mean-squared error can be measured between the actual sample value and the estimates. The value of kernel width giving the smallest error should be used in final network. The computing virtual image is show in Fig. 2(c) using holdout method. Fig. 3 shows pixels statistics curve of virtual images adopted holdout method. The curve of virtual image is still a wide distance to the curve of real image.
When a larger r is chosen, the image contains the lighting information of samples. The images generating by the method of dynamic adjustment of kernel width of GRNNs is just the one, which smoothly balances and shows up the lighting changing. Let Y ðX Þ ¼ X þ kf ðX ; r; X 1 ; X 2 ; . . . . . . X n ; Y 1 ; Y 2 ; . . . . . . Y n Þ ð3:2Þ where resulting Y (X) is virtual image. X is the input image. f () is the computing image by the method of dynamic adjustment of kernel width of GRNNs. k is coefficients. The flow chart is shown as follows
Z. Li et al. / Image and Vision Computing 25 (2007) 1249–1254
5
6
240 220
real image 50
100
150
200
250
300
350
50
100
150
200
250
300
350
400
50
100
150
200
250
300
350
400
450
50
100
150
200
250
300
350
400
450
200
180
5
0 0 400
input image
real image 0 0 500
140
0
20
40
60
80
100 120 140 160 180 200
0
0
400
450
input image 450
5
200
160
120
500
0 0 400
200
1253
Fig. 5. Image color curve of virtual images adopted increment addition (5). Fig. 6. Pixels statistics curve of virtual images adopted increment addition (5).
Samples σ
GRNN
Input
Model Output1 k Output2
Optimizing k, a and C is chosen using Levenberg–Marquardt [10], we get the image in Fig. 2(e) using the method of increment addition. Comparing Figs. 2(e) and 1(d), their colors are closed, which is we need. Figs. 5 and 6 shows color curve and pixels statistics curve of virtual images adopted Increment Addition. Note that the output images by method of increment addition are almost similar to the real one. 4. Application and conclusions Methods of generating virtual images of the same scene at different lighting conditions are presented, and the inherent relations of color and outside lighting conditions are discovered by the use of neural networks in this paper. As a result, novel images of same scene under different lighting conditions can be deduced from known scene image. Using our method, we can obtain the information about the change of lighting in the scene without any space geometry information. The samples in this paper are 28
colors. As we know, the more the samples there are the more accurate the result will be. However, our samples are less but the images obtained by using our method are close to real image. It shows that the method presented here is effective to obtain virtual image with virtual lighting conditions. More examples with different training sets, for example, from 10:00 to 19:00, can give a larger variation on the color response. Therefore more experimentation should be processed. The questions below need to be studied farther: 1. How to choose proper value of a and r for the methods presented in 3.4, so that the images we obtained are closer to real images? 2. Number of samples and distribution of samples will both influence the result images. Is there better method to choose samples? 3. Is there other neural networks which suit better? 4. For various scene, such as outside, inside, architecture, etc, how to construct the model of neural networks? 5. What is the relationship between material and the response? For example, objects have same color but make of different materials may give different response to light variation? The methods presented in this paper can generate virtual scene images dynamically and lengthways. We can reconstruct a series of virtual images, which is changed with time (Fig. 7, a example using the method in this paper, the first one is a real image, the others are virtual images). We can also reconstruct images virtually generated at appointed time. Using IBM techniques, the model of scene is recovered.
Fig. 7. The first image is the real image. The others are virtual images.
1254
Z. Li et al. / Image and Vision Computing 25 (2007) 1249–1254
Further, combining the dynamic virtual images obtained by our method, dynamic virtual scene can be reconstructed. References [1] Heung-Yeung Shum, Sing Bing Kang, A review of image-based rendering techniques, in: IEEE/SPIE Visual Communications and Image Processing (VCIP) 2000, Perth, June 2000, pp. 2–13. [2] H.Y. Shum, L.W. He, Rendering with Concentric Mosaics, SIGGRAPH 99. [3] R. Szeliski, H.-Y. Shum, Creating full view panoramic image mosaics and texture-mapped models, Computer Graphics (SIGGRAPH’97) (1997) 251–258. [4] Paul Ernest Debevec, Modeling and Rendering Architecture from Photographs, PhD Thesis, University of California at Berkeley, 1996, pp. 109–119.
[5] Yizhou Yu, Paul Debevec, Jitendra Malik, Tim Hawkins, Inverse Global Illumination: Recovering Reflectance Models of Real Scenes from Photographs’’, SIGGRAPH99. [6] Ko Nishino, Zhengyou Zhang, Katsushi Ikeuchi, Determining reflectance parameters and illumination distribution from a sparse set of images for view-dependent image synthesis. in: Proc. of ICCV 2001. [7] D.F. Specht, A general regression neural network, IEEE Transactions on Neural Networks 2 (1991) 568–576. [8] Zhanwei Li, Jizhou Sun, Jiawan Zhang, Time-Sequence Dynamic Virtual Images, the Computer Graphics International (CGI 2003), Japan, July 9–11, 2003. [9] Zhanwei Li, Jizhou Sun, Jiawan Zhang, Zunce Wei, Computing Models Based on GRNNs, CCECE2003, Canada. [10] Lagarias, Jeffrey C., James A. Reeds, Convergence properties of the Nelder-Mead simplex algorithm in low dimensions, Journal of Optimization, 1997.
Obtaining virtual lighting condition based on images using NNs Zhanwei Li
a,* a
Guolin Duan a, Jizhou Sun b, Xinran Lv
a
Hebei University of Technology, Tianjin 300130, China b Tianjin University, Tianjin 300072, China
Received 20 April 2005; received in revised form 11 January 2006; accepted 24 July 2006
Abstract Using Neural Networks, this paper present a method of generating virtual images under virtual lighting conditions. The relationship between the lighting conditions and the color of image is non-linear, several neural networks technologies are introduced to make curve fitting. As a result, virtual images of same scene under different lighting conditions can be deduced from known real scene image, and the inherent relations of image color and outside lighting conditions are discovered. Combining techniques of IBMR or panoramic image, the model of scene is recovered. Further, combining the dynamic virtual images obtained by our method, time-sequence dynamic virtual scene can be reconstructed and revisited in virtual reality. Ó 2006 Elsevier B.V. All rights reserved. Keywords: Image-based rendering (IBR); Neural networks (NNs); Virtual images; Lighting conditions; Dynamic reconstruction; Virtual reality
1. Introduction One of the important aims of the computer graphics is to render realistic image. Image-Based Rendering techniques are powerful alternatives to traditional geometrybased techniques for image synthesis. IBR techniques render novel viewpoint images from input images. Acquiring geometric models of scene has been the subject of research in interactive Image-Based Modeling (IBM) techniques, and is now becoming practical to perform with techniques such as interactive photogrammetry. Acquiring the corresponding appearance information (under given lighting conditions) is easy to performed with a digital camera. Image-based rendering can have many interesting applications. Two scenarios, large environments and dynamic environments, in particular, are worth pursuing [1]. Until now, most of image-based rendering systems have been focused on static environments.
*
Corresponding author. Tel.: +86 022 60202273. E-mail addresses: [email protected] (Z. Li), [email protected] (G. Duan), [email protected] (J. Sun). 0262-8856/$ - see front matter Ó 2006 Elsevier B.V. All rights reserved. doi:10.1016/j.imavis.2006.07.029
With the development of panoramic video systems [2,3], it is conceivable that image-based rendering can be applied to dynamic environments as well, but for panoramic video the lighting conditions are static, although scenery is dynamic. Debevec [4] put forward projecting real images from dawn to dusk in sun, clouds and even fog weather condition to recovered model, the scene of different lighting conditions is recurred. In his method scene images are real images and are obtained in advance, the geometry model of the scene is computed using IBM techniques. Image-Based Lighting (IBL) techniques [5,6] recover the reflectance properties of all surfaces in a real scene. But geometric model of the scene and the position of lighting are needed in IBL, and its arithmetic is very complicated. In this paper, we propose to reconstruct virtual images at appointed lighting condition. At different lighting condition the same scene is differently lighted. We can infer the virtual images in virtual outside lighting conditions by finding the rule of image color changing with outside conditions. Using IBM techniques, the model of scene is recovered. Further, combining the dynamic virtual images obtained by our method, time-sequence dynamic virtual
1250
Z. Li et al. / Image and Vision Computing 25 (2007) 1249–1254
scene can be reconstructed. It can be applied in virtual reality to recur virtual scene. Over the past decade, neural network technologies have found their way into numerous competitive industries. Neural Networks are touted for their predictive accuracy compared with more common linear modeling methods. This paper adopts several neural networks technologies to deduce the relationship between lighting conditions and color of images. 2. Analysis of image colors changing with lighting conditions At different time of day or different weather conditions such as noon or night, sunshine or fog, the same scene are different distinctly. Eyes can distinguish the fine variations of colors. So we can accurately recognize the outside lighting conditions of scene. Seeing an image we can approximately know the lighting conditions, time, weather and so on when the image is generated. The same object with different lighting conditions will present different colors in our eyes. If the change rules can be found by analysing the changes of colors, then we can obtain virtual images under different lighting conditions from a real image. But the relation between image color
and outside lighting conditions are so complicated, even little change will influence the lighting condition, further influence the color of image. Therefore it is difficult to show using linear modeling. Neural network is a good tool to solve the perplexing problem, but how to construct neural network models and define the samples are important. In order to study the change rules between color and lighting condition, we choose 28 colors as samples and shot these colors at different time of day (see Fig. 1(a and b)). The 28 colors are symmetrically allocated in 24 bit colors. The camera is KODAK DC240. Flash lamp was shut down in order to get lighting information accurately. It is obvious from Figs. 1(a) and (b) that the effect of lighting and shades is different from the shooting time. The 28 colors in image shot at 10:10 AM are the input samples, and that at 17:30 PM are the output samples, which are both in Table 1. The input variable (image) is image c in Fig. 1. The output will be a virtual image at lighting condition of 17:30 PM, and it can be compared with the real image shot at 17:30 PM (Fig. 1(d)). This paper adopt image color curve (x-axis: x-coordinate of corresponding image, y-axis: average of pixel color) and image pixels statistics curve (x-axis: pixel energy value, y-axis: sta-
a
b
Color samples at 10:10AM
Color samples at 17:30PM
c
d
Real images shot at 10:10AM
Real images shot at 17:30PM
e 240
f 10:10AM
220
500 400 300 200
200
100 0 0
180
17:30PM
160
50
100
150
200
250 300
350
400 450
50
100
150
200
250 300
350
400 450
500 400 300
140
200 100
120 0
20 40 60 80 100 120 140 160 180 200
Color curve of image
0 0
Pixels statistics curve of image
Fig. 1. (a and b) are color maps shot at different time of a day. It is obvious that the image colors change with the lighting conditions. (c and d) are picture images shot at different time. (e and f) are, respectively, color curve and pixels statistics curve of image (c and d).
Z. Li et al. / Image and Vision Computing 25 (2007) 1249–1254
1251
Table 1 Input and output samples Output samples (17:30)
152, 52, 51 70, 110, 125 90, 39, 39 37, 77, 82 90, 86, 57 25, 75, 118 21, 44, 82
84, 114, 72 181, 162, 51 51, 84, 59 109, 100, 48 177, 166, 96 90, 124, 133 147, 35, 59
57, 90, 132 194, 190, 186 50, 70, 95 112, 106, 100 50, 78, 75 42, 77, 111 107, 66, 46
168, 47, 89 47, 45, 45 139, 66, 97 156, 151, 140 87, 127, 99 83, 108, 148 181, 93, 62
Input samples (10:10)
146, 63, 69 72, 118, 141 85, 50, 50 42, 81, 90 80, 85, 62 31, 73, 113 26, 45, 80
74, 107, 76 160, 151, 55 49, 81, 61 93, 93, 53 157, 155, 96 76, 116, 130 119, 35, 61
55, 92, 139 185, 189, 197 41, 65, 96 96, 100, 104 42, 72, 75 37, 72, 109 88, 61, 50
154, 55, 101 49, 50, 53 120, 59, 103 143, 148, 149 80, 124, 102 77, 108, 155 164, 87, 64
tistic amount) to analyze and compare images. From color curve (Fig. 1(e)) the two real images have a gap in color. From pixels statistics curve (Fig. 1(f)) the energy of the two images is different. 3. Create virtual lighting condition based on images using NNs
in Fig. 3. The third curve is from virtual images at 17:30 PM adopted Linear Neural Networks. The first and second curves are respectively real image at 17:30 PM and 10:10 AM. The third curve tends from second curve to the first, but still different distinctly. It shows that Linear Neural Networks is not fit to the problem. 3.2. Fixed kernel width of GRNNs
Several neural networks will be try below to check the result of forecasting image under virtual lighting condition. If we could find out the changing rules of colors with lighting conditions, we would be able to deduce virtual images under virtual outside conditions from a real image. 3.1. Linear neural networks Linear Neural Networks are often used in function approximation. The computing virtual image is show in Fig. 2(a) using linear NN. Its pixels statistics curve is show
a
General Regression Neural Networks (GRNNs) is presented by D.L.Specht [7] in 1991. GRNNs is effective with only a few samples and even with sparse data in multidimensional space. GRNN is radial basis function networks as well as probabilistic neural networks. It can be used in function approximation, and hence image interpolation. The kernel width r of GRNNs can adjust the perceptive region of nerve cells. Small value for r means that the perceptive region of nerve cells is small, and the output contribution is mainly from the nearby nerve cells, so computing
c
b
Linear NN
Fixed Kernel Width of GRNNs
Holdout method
e
d
dynamic adjustment of kernel width of GRNNs
Increment Addition
Fig. 2. Virtual images at 17:30 PM from five methods.
1252
Z. Li et al. / Image and Vision Computing 25 (2007) 1249–1254
4
3 500
500
real image 00 400
50
100
150
200
250
300
350
400
0 0 400
50
100
150
200
250
300
350
400
450
50
100
150
200
250
300
350
400
450
400
450
400
450
200
200
input image
00 400
50
100
150
200
250
300
350
400
450
0 0 500
1
200 00 400
50
100
150
200
250
300
350
3 400
450
400
450
0 0 500
50
100
150
200
250
300
350
50
100
150
200
250
300
350
2
200 0
450
0
50
100
150
200
250
300
350
4 0
0
Fig. 3. Pixels statistics curve of virtual images adopted linear NN (1) and holdout method (2). Fig. 4. Pixels statistics curve of virtual images adopted dynamic adjustment of kernel width (3) and fixed kernel width of GRNN (4).
result is influenced by fewer nerve cells and interpolation is relatively accurate, but serious steep. Large value for r means that the perceptive region of nerve cells is large, the output is from both the nearby nerve cells and farther ones, so computing result is influenced by more nerve cells and interpolation is smooth but noisy and inaccurate. Sometime there is no sample in the perceptive region of input vectors, so the computing is failure for dividing by zero. The situation happens especially when samples are not symmetrically allocated. If we choose a small r then perceptive region is small. Because the samples are sparse, there are serious color abruptness, bad smoothness and scarce colors in the image. If we choose a large r, there is wide perceptive region, good color smoothness and abundant colors but serious color distortion. The computing virtual image obtained from an optimizing r is show in Fig. 2(b) using fixed kernel width of GRNNs. The fourth curve in Fig. 4 is its pixels statistics curve. The first and second curve are respectively from real image at 17:30 PM, 10:10 AM. The curve of virtual image is still a wide distance to the curve of real image. In conclusion constant kernel width in GRNNs can not well adapt to input vectors, so method of dynamic kernel width below is adopted.
3.4. Dynamic adjustment of kernel width of GRNNs [8,9] Holdout method still can not well adapt every input vector of different distance with samples. So we present to adjust r dynamically. r is chosen in Formula 3.1, pffiffiffiffi T a ri ¼ C ½ðX X i Þ ðX X i Þ ð3:1Þ where a is a number between 0 and 1, C is a constant. Xi are inputs samples, X is input vector. When the distance between samples and input vector is large, there is no sample near the input vector, so in order to search for samples in wider regions, we should choose a larger r. When the distance between samples and input vector is short, there are samples near the input vector. Hence it is not necessary to search for samples in large regions. As a result, choosing smaller r is OK. Thus such images are colorful and not distorted. Optimizing a and C is chosen using Levenberg–Marquardt [10]. The virtual image computing from an optimizing a and C is show in Fig. 2(d) using dynamic adjustment of kernel width of GRNNs. Fig. 4 shows pixels statistics curve of virtual images adopted Dynamic Adjustment of Kernel Width. The third curve tends from second curve to first better than the fourth one.
3.3. Holdout method: Diverse kernel width of GRNNs [7] 3.5. Increment addition Holdout method is a useful method of selecting kernel width. First, removing one sample at a time and constructing a network based on all of other samples. Then the network is used to estimate output for the removed sample. By repeating this process for each sample and storing each estimate, then mean-squared error can be measured between the actual sample value and the estimates. The value of kernel width giving the smallest error should be used in final network. The computing virtual image is show in Fig. 2(c) using holdout method. Fig. 3 shows pixels statistics curve of virtual images adopted holdout method. The curve of virtual image is still a wide distance to the curve of real image.
When a larger r is chosen, the image contains the lighting information of samples. The images generating by the method of dynamic adjustment of kernel width of GRNNs is just the one, which smoothly balances and shows up the lighting changing. Let Y ðX Þ ¼ X þ kf ðX ; r; X 1 ; X 2 ; . . . . . . X n ; Y 1 ; Y 2 ; . . . . . . Y n Þ ð3:2Þ where resulting Y (X) is virtual image. X is the input image. f () is the computing image by the method of dynamic adjustment of kernel width of GRNNs. k is coefficients. The flow chart is shown as follows
Z. Li et al. / Image and Vision Computing 25 (2007) 1249–1254
5
6
240 220
real image 50
100
150
200
250
300
350
50
100
150
200
250
300
350
400
50
100
150
200
250
300
350
400
450
50
100
150
200
250
300
350
400
450
200
180
5
0 0 400
input image
real image 0 0 500
140
0
20
40
60
80
100 120 140 160 180 200
0
0
400
450
input image 450
5
200
160
120
500
0 0 400
200
1253
Fig. 5. Image color curve of virtual images adopted increment addition (5). Fig. 6. Pixels statistics curve of virtual images adopted increment addition (5).
Samples σ
GRNN
Input
Model Output1 k Output2
Optimizing k, a and C is chosen using Levenberg–Marquardt [10], we get the image in Fig. 2(e) using the method of increment addition. Comparing Figs. 2(e) and 1(d), their colors are closed, which is we need. Figs. 5 and 6 shows color curve and pixels statistics curve of virtual images adopted Increment Addition. Note that the output images by method of increment addition are almost similar to the real one. 4. Application and conclusions Methods of generating virtual images of the same scene at different lighting conditions are presented, and the inherent relations of color and outside lighting conditions are discovered by the use of neural networks in this paper. As a result, novel images of same scene under different lighting conditions can be deduced from known scene image. Using our method, we can obtain the information about the change of lighting in the scene without any space geometry information. The samples in this paper are 28
colors. As we know, the more the samples there are the more accurate the result will be. However, our samples are less but the images obtained by using our method are close to real image. It shows that the method presented here is effective to obtain virtual image with virtual lighting conditions. More examples with different training sets, for example, from 10:00 to 19:00, can give a larger variation on the color response. Therefore more experimentation should be processed. The questions below need to be studied farther: 1. How to choose proper value of a and r for the methods presented in 3.4, so that the images we obtained are closer to real images? 2. Number of samples and distribution of samples will both influence the result images. Is there better method to choose samples? 3. Is there other neural networks which suit better? 4. For various scene, such as outside, inside, architecture, etc, how to construct the model of neural networks? 5. What is the relationship between material and the response? For example, objects have same color but make of different materials may give different response to light variation? The methods presented in this paper can generate virtual scene images dynamically and lengthways. We can reconstruct a series of virtual images, which is changed with time (Fig. 7, a example using the method in this paper, the first one is a real image, the others are virtual images). We can also reconstruct images virtually generated at appointed time. Using IBM techniques, the model of scene is recovered.
Fig. 7. The first image is the real image. The others are virtual images.
1254
Z. Li et al. / Image and Vision Computing 25 (2007) 1249–1254
Further, combining the dynamic virtual images obtained by our method, dynamic virtual scene can be reconstructed. References [1] Heung-Yeung Shum, Sing Bing Kang, A review of image-based rendering techniques, in: IEEE/SPIE Visual Communications and Image Processing (VCIP) 2000, Perth, June 2000, pp. 2–13. [2] H.Y. Shum, L.W. He, Rendering with Concentric Mosaics, SIGGRAPH 99. [3] R. Szeliski, H.-Y. Shum, Creating full view panoramic image mosaics and texture-mapped models, Computer Graphics (SIGGRAPH’97) (1997) 251–258. [4] Paul Ernest Debevec, Modeling and Rendering Architecture from Photographs, PhD Thesis, University of California at Berkeley, 1996, pp. 109–119.
[5] Yizhou Yu, Paul Debevec, Jitendra Malik, Tim Hawkins, Inverse Global Illumination: Recovering Reflectance Models of Real Scenes from Photographs’’, SIGGRAPH99. [6] Ko Nishino, Zhengyou Zhang, Katsushi Ikeuchi, Determining reflectance parameters and illumination distribution from a sparse set of images for view-dependent image synthesis. in: Proc. of ICCV 2001. [7] D.F. Specht, A general regression neural network, IEEE Transactions on Neural Networks 2 (1991) 568–576. [8] Zhanwei Li, Jizhou Sun, Jiawan Zhang, Time-Sequence Dynamic Virtual Images, the Computer Graphics International (CGI 2003), Japan, July 9–11, 2003. [9] Zhanwei Li, Jizhou Sun, Jiawan Zhang, Zunce Wei, Computing Models Based on GRNNs, CCECE2003, Canada. [10] Lagarias, Jeffrey C., James A. Reeds, Convergence properties of the Nelder-Mead simplex algorithm in low dimensions, Journal of Optimization, 1997.