- Email: [email protected]
TV-optical iterative picture restoration: Experimental results
Volume 38, number 5,6
OPTICS COMMUNICATIONS
1 September 1981
TV-OPTICAL ITERATIVE PICTURE RESTORATION: EXPERIMENTAL RESULTS Gert FERRANO Physikalisches Institut, Universitd't Erlangen/Nffrnberg, 8520 Ergangen, Fed. Rep. Germany and Henri MAITRE Ecole Nationale Supdrieure des T~l~communications, 75013 Paris, France Received 13 February 1981 Revised manuscript received 19 May 1981 We present experimental results of picture correction using a hybrid TV-optical system. The method relies on an iterative restoration, identical to Jacobi method for solving linear systems and is performed with a video loop.
1. Introduction The domain of picture deblurring is an open field for present time workers, because many problems are still unsolved at different levels: theoretically the ways seem well known, but indeed, space variance, singularities, or ill-conditioning are actually not mastered [1 ]. Psychovisually, no correct quality criterion is actually available. Practically, computational complexity, computational delay, and memory requirements are often limiting factors when industrial, cheap, or real time applications are concerned. In this paper, we present an alternative to the classical digital or optical methods of picture restoration. It relies on the ability of a closed TV loop to perform iterative corrections in quasi-real time, with a very affordable set up. This publication is essentially devoted to the presentation of experimental results.
di/= di+k, f+k whatever i, j, k so that i, j, i + k and j + k are within the range [1, mn]. In any case (space variant or space invariant) the system is described by the relation:
g=df
The theoretical bases as well as the historical developments of this method were presented in a separate paper [2]. It takes its background from the field of numerical analysis, and especially from solving large systems. Let us denote b y f a n d g the developped vec-
(1)
and an iterative solution will be to calculate a series o f approximations f ( k ) , converging towards a limit f = f. The most known solution was proposed by Jacobi and has been used very often since this time under various names (Van Cittert method, simultaneous displacement method, etc.), it obeys the rule: ;(k+l) =f(k) +g_df(k).
2. Theoretical premisses
336
torial representations of the original and the blurred picture [3]. Suppose the picture has n lines o f m pixels, l a n d g have nm components, d denotes the defect matrix, with size n m × rim. If the defect is space invariant, d is a Toeplitz matrix so that:
(2)
Many other solutions provide an iterative resolution of eq. (1) but it has been shown [2] that for TV optical implementation the Jacobi method is probably the most suitable. The hybrid optical set-up is presented on fig. 1 in the general case of a 2-X D defect. It is worthwhile to
0 0 3 0 - 4 0 1 8 / 8 1 / 0 0 0 0 - 0 0 0 0 / $ 02.50 © North-Holland Publishing Company
Volume 38, number 5,6
OPTICS COMMUNICATIONS
1 September 1981 4
'2
I Fig. 1. TV-optical set-up proposed in [4], for iterative restoration of blurred pictures. Camera C3 provides the blurred signal, the backward loop is made of one camera C2 looking into its own monitor Mt through the defect D. The loop gain j3 must be high. compare it with the set-up proposed in [4], to restore a picture using a TV-optical operational amplifier (fig. 2). Indeed the complexity is greater, but the solution is a better approximation o f the " t r u e " solution o f eq. (1):
f=d-lg, and stability conditions seem simpler to handle. The theoretical advantages of this m e t h o d when compared with direct resolution o f eq. (1) are clear: we have no need for an inversion o f the matrix d, the speed o f convergence will be great (25 iterations per second), no m e m o r y is required, and we can easily introduce non-linear constraints, whose influence on the restoration quality is essential [5]. For instance, positivity and saturation constraints are ensured b y hardclipping the video signal before displaying it on the
Fig. 2. TV-optical set-up used to implement Jacobi iterative restoration. A new branch is needed: the camera C1 provides an unblurred image of the current estimate. No additional gain is necessary. screen. This will give a method similar to the one proposed b y Jansson [6].
3. Experimental results In a first step we have electronically created the blur on the video signal b y use of a low pass filter degrading the output o f the camera C 3 . The result is a horizontal blur smearing the original signal fig. 3a into fig. 3b. The defect being one-dimensional we have no need of the camera C 2 in the reconstruction stage. Just an identical low pass filter will be introduced on the output o f C 1 . The reconstruction set up is shown o f fig. 4, and the corrected picture on fig. 5. In a second step, the defect was optically introduced on the object when taking the picture g. The unblurred
Fig. 3. (a) Original picture. (b) Blurred picture degraded by a low pass •ter in the video link. 337
Volume 38, number 5,6
OPTICS COMMUNICATIONS
1 September 1981
,11
l
23
Fig. 4. Simplified set-up in case of electronical degradation due to a low pass filter in the output of camera C 3. Camera C3 is suppressed. The low pass filter is obtained using a capacitor 7 and the internal resistance of the TV cable (75S2).
image f w a s moved during the exposure time using a step-motor whose programmation simulated a constant acceleration. Here also the defect is one-dimensional and we have taken advantage o f the similarity o f the defect impulse response with the one of the previous low pass filter (fig. 6) to correct it using the set up o f fig. 4. The blurred pictures are presented on figs. 7 and 8 for various values o f accelerations, along with the corresponding corrections.
Fig. 6. Impulse response of a constant acceleration blur (solid curve), approximation using a low pass filter (dotted curve). experimental evidences show that the only feasible solution is to have two matched identical cameras. When a pair o f such cameras is not at our disposal, different possibilities can be adopted: we can use the 1 × D trick proposed in section 3, even if the defect is 2 × D, because we know that the increase o f quality is often good enough when approximating a 2 × D filter b y its 1 × D projection [7] ; - we can take advantage of the perfect storage capability of a picture m e m o r y to suppress one of these cameras. This way is actually under investigation by one of the authors (G.F.) and will be soon presented in another paper. we can directly convert the signals into numerical data and process it purely numerically as, for instance, in [81; in some cases, if the defect can be written under the form d = 1 + d ' where d ' is easily done optically, then we suppress one branch o f the backward loop writting eq. (2) under the form: -
-
4. Discussions
-
A major difficulty of the practical realization of the bidimensional set up (fig. 2) is a perfect optical alignment o f the two cameras C 1 and C 2. Furthermore, in order to have a true subtraction o f f ( k ) and d f (k), the two target transfer functions must be equal. Thus,
f(k+l) =g_
d'f(k);
this simplification was done in [9], to suppress ghost images. It has been proved, in [2], that the camera C 1 must
Fig. 5. Corrected picture corresponding to 3b. 338
Fig. 7. Example of correction: (a) blurred picture, (b) correct ed picture.
Volume 38, number 5, 6
OPTICS COMMUNICATIONS
1 September 1981
system, the obtained limit will be stable, but often poorer on edges and high frequencies.
5. Conclusions
Fig. 8. Example of correction: (a) blurred picture, (b) corrected picture. have a fast target to give a proper convergence towards the true inverse d-lg. In our experiment we made use of a plumbicon camera. The equation system (eq. (1)) is sometimes singular and often ill-conditioned. Thus, in case o f noise, the solution (eq. (2)) will not converge, being either divergent or periodical. This instability will appear as ripples propagating from the edges inside the picture. A way to control this unstability is to introduce a relaxation factor:
f(k+l)=f(k)+ot(g-df(k)),
O
Then, due to the positivity and saturation constraints, and also to the dynamical behaviour o f the
We have verified the feasibility of TV optical iterarive picture restoration using Jacobi method. The increase of quality due to restoration is noticeable, the speed o f convergence is very high (less than one second), the price o f set-up reasonably low. We have pointed out the experimental difficulties of 2 X D restoration and proposed some ways to circumvent them References [1] H.C. Andrews and B.R. Hunt, Digital image restoration (Prentice Hall, New York, 1977). [2] H. Maitre, Comp. Graph. Im. Proc., to be published, March 1981. [3] W.K. Pratt, Comp. Graph. Im. Proc. 4 (1975) 1. [4] J. Goetz, G. H/iusler and R. Sesselmann, Appl. Optics 18 (1979) 2754. [5] B.R. Frieden, in: Picture processing and digital filtering, ed. T.S. Huang (Soringer Verlag, Berlin, 1979) p. 177. [6] P.A. Jansson, J. Opt. Soc. Am. 60 (1970) 184. [7] G. Hi/usler, Optics Comm. 6 (1972) 38. [8] S. Kawata and Y. Ichioka, J. Opt. Soc. Am. 70 (1980) 762. [9] T. Sato, K. Sasaki and R. Yamamoto, Appl. Optics 17 (1978) 717.
339
OPTICS COMMUNICATIONS
1 September 1981
TV-OPTICAL ITERATIVE PICTURE RESTORATION: EXPERIMENTAL RESULTS Gert FERRANO Physikalisches Institut, Universitd't Erlangen/Nffrnberg, 8520 Ergangen, Fed. Rep. Germany and Henri MAITRE Ecole Nationale Supdrieure des T~l~communications, 75013 Paris, France Received 13 February 1981 Revised manuscript received 19 May 1981 We present experimental results of picture correction using a hybrid TV-optical system. The method relies on an iterative restoration, identical to Jacobi method for solving linear systems and is performed with a video loop.
1. Introduction The domain of picture deblurring is an open field for present time workers, because many problems are still unsolved at different levels: theoretically the ways seem well known, but indeed, space variance, singularities, or ill-conditioning are actually not mastered [1 ]. Psychovisually, no correct quality criterion is actually available. Practically, computational complexity, computational delay, and memory requirements are often limiting factors when industrial, cheap, or real time applications are concerned. In this paper, we present an alternative to the classical digital or optical methods of picture restoration. It relies on the ability of a closed TV loop to perform iterative corrections in quasi-real time, with a very affordable set up. This publication is essentially devoted to the presentation of experimental results.
di/= di+k, f+k whatever i, j, k so that i, j, i + k and j + k are within the range [1, mn]. In any case (space variant or space invariant) the system is described by the relation:
g=df
The theoretical bases as well as the historical developments of this method were presented in a separate paper [2]. It takes its background from the field of numerical analysis, and especially from solving large systems. Let us denote b y f a n d g the developped vec-
(1)
and an iterative solution will be to calculate a series o f approximations f ( k ) , converging towards a limit f = f. The most known solution was proposed by Jacobi and has been used very often since this time under various names (Van Cittert method, simultaneous displacement method, etc.), it obeys the rule: ;(k+l) =f(k) +g_df(k).
2. Theoretical premisses
336
torial representations of the original and the blurred picture [3]. Suppose the picture has n lines o f m pixels, l a n d g have nm components, d denotes the defect matrix, with size n m × rim. If the defect is space invariant, d is a Toeplitz matrix so that:
(2)
Many other solutions provide an iterative resolution of eq. (1) but it has been shown [2] that for TV optical implementation the Jacobi method is probably the most suitable. The hybrid optical set-up is presented on fig. 1 in the general case of a 2-X D defect. It is worthwhile to
0 0 3 0 - 4 0 1 8 / 8 1 / 0 0 0 0 - 0 0 0 0 / $ 02.50 © North-Holland Publishing Company
Volume 38, number 5,6
OPTICS COMMUNICATIONS
1 September 1981 4
'2
I Fig. 1. TV-optical set-up proposed in [4], for iterative restoration of blurred pictures. Camera C3 provides the blurred signal, the backward loop is made of one camera C2 looking into its own monitor Mt through the defect D. The loop gain j3 must be high. compare it with the set-up proposed in [4], to restore a picture using a TV-optical operational amplifier (fig. 2). Indeed the complexity is greater, but the solution is a better approximation o f the " t r u e " solution o f eq. (1):
f=d-lg, and stability conditions seem simpler to handle. The theoretical advantages of this m e t h o d when compared with direct resolution o f eq. (1) are clear: we have no need for an inversion o f the matrix d, the speed o f convergence will be great (25 iterations per second), no m e m o r y is required, and we can easily introduce non-linear constraints, whose influence on the restoration quality is essential [5]. For instance, positivity and saturation constraints are ensured b y hardclipping the video signal before displaying it on the
Fig. 2. TV-optical set-up used to implement Jacobi iterative restoration. A new branch is needed: the camera C1 provides an unblurred image of the current estimate. No additional gain is necessary. screen. This will give a method similar to the one proposed b y Jansson [6].
3. Experimental results In a first step we have electronically created the blur on the video signal b y use of a low pass filter degrading the output o f the camera C 3 . The result is a horizontal blur smearing the original signal fig. 3a into fig. 3b. The defect being one-dimensional we have no need of the camera C 2 in the reconstruction stage. Just an identical low pass filter will be introduced on the output o f C 1 . The reconstruction set up is shown o f fig. 4, and the corrected picture on fig. 5. In a second step, the defect was optically introduced on the object when taking the picture g. The unblurred
Fig. 3. (a) Original picture. (b) Blurred picture degraded by a low pass •ter in the video link. 337
Volume 38, number 5,6
OPTICS COMMUNICATIONS
1 September 1981
,11
l
23
Fig. 4. Simplified set-up in case of electronical degradation due to a low pass filter in the output of camera C 3. Camera C3 is suppressed. The low pass filter is obtained using a capacitor 7 and the internal resistance of the TV cable (75S2).
image f w a s moved during the exposure time using a step-motor whose programmation simulated a constant acceleration. Here also the defect is one-dimensional and we have taken advantage o f the similarity o f the defect impulse response with the one of the previous low pass filter (fig. 6) to correct it using the set up o f fig. 4. The blurred pictures are presented on figs. 7 and 8 for various values o f accelerations, along with the corresponding corrections.
Fig. 6. Impulse response of a constant acceleration blur (solid curve), approximation using a low pass filter (dotted curve). experimental evidences show that the only feasible solution is to have two matched identical cameras. When a pair o f such cameras is not at our disposal, different possibilities can be adopted: we can use the 1 × D trick proposed in section 3, even if the defect is 2 × D, because we know that the increase o f quality is often good enough when approximating a 2 × D filter b y its 1 × D projection [7] ; - we can take advantage of the perfect storage capability of a picture m e m o r y to suppress one of these cameras. This way is actually under investigation by one of the authors (G.F.) and will be soon presented in another paper. we can directly convert the signals into numerical data and process it purely numerically as, for instance, in [81; in some cases, if the defect can be written under the form d = 1 + d ' where d ' is easily done optically, then we suppress one branch o f the backward loop writting eq. (2) under the form: -
-
4. Discussions
-
A major difficulty of the practical realization of the bidimensional set up (fig. 2) is a perfect optical alignment o f the two cameras C 1 and C 2. Furthermore, in order to have a true subtraction o f f ( k ) and d f (k), the two target transfer functions must be equal. Thus,
f(k+l) =g_
d'f(k);
this simplification was done in [9], to suppress ghost images. It has been proved, in [2], that the camera C 1 must
Fig. 5. Corrected picture corresponding to 3b. 338
Fig. 7. Example of correction: (a) blurred picture, (b) correct ed picture.
Volume 38, number 5, 6
OPTICS COMMUNICATIONS
1 September 1981
system, the obtained limit will be stable, but often poorer on edges and high frequencies.
5. Conclusions
Fig. 8. Example of correction: (a) blurred picture, (b) corrected picture. have a fast target to give a proper convergence towards the true inverse d-lg. In our experiment we made use of a plumbicon camera. The equation system (eq. (1)) is sometimes singular and often ill-conditioned. Thus, in case o f noise, the solution (eq. (2)) will not converge, being either divergent or periodical. This instability will appear as ripples propagating from the edges inside the picture. A way to control this unstability is to introduce a relaxation factor:
f(k+l)=f(k)+ot(g-df(k)),
O
Then, due to the positivity and saturation constraints, and also to the dynamical behaviour o f the
We have verified the feasibility of TV optical iterarive picture restoration using Jacobi method. The increase of quality due to restoration is noticeable, the speed o f convergence is very high (less than one second), the price o f set-up reasonably low. We have pointed out the experimental difficulties of 2 X D restoration and proposed some ways to circumvent them References [1] H.C. Andrews and B.R. Hunt, Digital image restoration (Prentice Hall, New York, 1977). [2] H. Maitre, Comp. Graph. Im. Proc., to be published, March 1981. [3] W.K. Pratt, Comp. Graph. Im. Proc. 4 (1975) 1. [4] J. Goetz, G. H/iusler and R. Sesselmann, Appl. Optics 18 (1979) 2754. [5] B.R. Frieden, in: Picture processing and digital filtering, ed. T.S. Huang (Soringer Verlag, Berlin, 1979) p. 177. [6] P.A. Jansson, J. Opt. Soc. Am. 60 (1970) 184. [7] G. Hi/usler, Optics Comm. 6 (1972) 38. [8] S. Kawata and Y. Ichioka, J. Opt. Soc. Am. 70 (1980) 762. [9] T. Sato, K. Sasaki and R. Yamamoto, Appl. Optics 17 (1978) 717.
339