Multichannel approach to adaptive image restoration using the 2-D least squares algorithm

INTEGRATION OF STEREO-VISION AND OPTICAL FLOW USING MARKOV RANDOM FIELDS: Sandra P. Clifford and Nasser M. Nasrabadi r Computer Vision Research Group, Worcester Polytechnic Institute, Electrical Engineering Department, Worcester, MA 01609 USA. The research presented describes a stereo-matching algorithm particularly well suited for a neural network implementation. The stereomatching problem is an ill-posed problem solved by regularization. In contrast to standard regalarization methods, we represent the apriori knowledge in terms of probability distributions. The approach of our stereo-matching algorithm development is to find an estimate of the "optimal" disparity solution for two stereo images. Optimal is defined as the maximum aposteriori probability (MAP) solution based on the assumption that our solution can be modeled by a Markov Random Field (MRF). A stochastic relaxation algorithm (Simulated Annealing) is used to minimize an energy function which yields an estimate of the optimal MAP stexeo-matched solution. Geman and Geman (1984) presented a thorough derivation of this estimator as applied to the image segmentation problem. The energy function describes the constraints on the solution using local characteristics. Because of the locality of the algorithm, it can be efficiently implemented with parallel processing architectures. More intelligent energy functions may possess discontinuities. It was shown in Koch eL al. (1986) that neural networks can effectively solve these problems as well. In order to obtain the MAP solution, we minimize the energy of the solution by achieving an equilibrium state at low temperature. The method we use is called Simulated Annealing (SA) whose name is borrowed from the physical process of annealing where materials are brought to low energy states by gradually lowering their temperature. As in chemical annealing, if the temperature is dropped too quickly the solution doesn't have sufficient time to reach equilibrium states, resulting in configurations which correspond to local minima of the energy function and will contain errors compared with the true solution. If the ideal temperature schedule is followed (shown in Geman and Geman (1984)) by the SA algorithm, the MAP solution is guaranteed. However, due to the slowness of this schedule, a non-ideal schedule is used yielding satisfactory results. Our research extends an algorithm described by Barnard (1986) where the stereo-matching energy function is based on the smoothness of the solution and the intensity match. Here we investigate the addition of an optical flow estimate to our set of known quantities. We incorporate this knowledge in the form of an occlusion indicator which results in a positive step toward solving the occlusion problem for the case of moving objects or cameras. A corresponding site which has been occluded by another object is flagged as such by thresholding the difference in opdcal flow (9) between the left and right image points as shown by Equation (1).

~o(i,j)=

f 1, if [ PL(i,j)-- 9R(i+dij,j) ]2 >_To O, otherwise

(1)

When occluded areas are identified, the solution is filled in by a different process which does not use the measure of similarity of the corresponding pixels as they will not match for the correct disparity value. The non-zero cost (energy) associated with the occlusion process inhibits non-occluded areas from settling into a falsely-detected occluded state. The energy function incorporating the occlusion indicator is given in Equation (2). U 0 = Dij + k" Sij

(2)

Dij = (1 - ~Pii) " [ ci [ lz.(i,j) - IR(i+dii,j)] 2 + co[ ptc(i,j) - pL(i+dii,j) ]2] + ~Oj] " CB

Sij = ~ [ d ( i , j ) -

d(iN,jN) ]2. ~So0(iN,jN);

iN,jN c N l ( i , j )

Nt(i,j) is the neighborhood for the intensity process and contains the eight nearest neighbors around site (i,j). X is a fixed, weighting factor for the data and smoothness terms. The figures below show the result of applying this new energy function to an image pair exhibiting 6 pixels of occlusion. The solution is much improved over the solution obtained from an energy equation without the occlusion indicator and yields the correct disparity even at the area where there is no visible match. Both the tall and short box solutions show an error of only 1 disparity-unit over their surfaces; this improves with the number of iterations. REFERENCES : Geman S., and Geman D. (1984), "Stochastic Relaxation, Gibbs Distributions, and the Bayesian Restoration of Images", IEEE T-PAMI, vol 6 no 6. Koch C., Marroquin J., and Yuille A. (1986), "Analog 'Neuronal' Networks in Early Vision", Proc. Nat'l. Acad. Sci., vol 83, pp 4263-7. Barnard S. T. (1986), "A Stochastic Approach to Stereo Vision", Proc. of Fifth Nat'l. Conf. on Artificial Intelligence, pp 676-80.

Disparity solution resulting from intensity matching incorporating optical flow data.

Stereo matching error in occluded negions.

486