- Email: [email protected]
Image processing for primitive image features recognition in the case of discontinuous line images of printed characters
Image processing for primitive image features recognition in the case of discontinuous line images of printed characters Anne M Landraud
The basis of the algorithm is a robust line detector which first finds out line precursors from the pixels by scanning the grey-level image in both horizontal and vertical directions and by applying an appropriate set of rules. Quanta are then identified and lines are readily detected in regions where quanta overlap, in accordance with the above definition of a line. In practice, in the regions where only line precursors have been found but where quanta do not overlap, overlapping is still assumed. The final stage of the algorithm consists of withdrawing lines and quanta (after having removed possible ambiguities) in order to save only the primitive features of interest. Such primitive features are the end of a line, a change of direction, a line crossing or a fork. They do not alter with changes in scale of the pixel Keywords: image processing, line images, projections, values or with changes in size (in some ranges), with periodicities displacements in the observed plane or with topological deformations. Using this method, the first level in the identification process is overcome by using operators which give robust results with low time and space The general problem we are concerned with is to complexities. The basic principle is to perform two identify an object from some image made of lines. In successive and complementary general operations: a the case of ‘continuous’ lines, an efficient algorithm has detection of regular regions (lines and background) been formulated and implemented in the Laforia followed by subtractions leading to feature identificaLaboratory, University Pierre et Marie Curie, Paris, tion. France as reported, for example, by J C Simorrm4. In However, these operators intended for line detection Simon’s papers, a line is defined as a ‘monodimensional cannot be used when input line images include ‘lines’ list of quanta partially covering each other’. The term made of dots or dashes due to a masking effect. In this quantum of information (or ‘blob’) is used here for case a line is no longer a regular region made of referring to the ‘shortest black pulse’ made of at least characteristic continuously overlapping quanta. On the three non-white pixels in any direction. The various contrary, each dot or dash, which is a piece of the true ‘blobs’ which form a line contain approximately the line, is taken by the line detector as an individual same number of non-white pixels. isolated line with two ends in the case of a dash, and a curved boundary if it is a dot. Insignificant gaps are Laboratoire LAFORIA-Groupe de Reconnaissance des Formes et Traitement d’Tmages, UniversitC Pierre et Marie Curie, tour 45-46. found between those short ‘lines’. 4. place Jussieu. 75230 Paris Cedex 05, France A method for solving such a problem in the context 0262~8856/89/030225-OS $03.00 @ 1989 Butterworth & Co. (Publishers) Ltd vol 7 no 3 august 1989 225 An efficient algorithm for analysing and recognizing line images has been formulated. It works well when applied to ‘continuous’ line images (a continuous line is defined here as a monodimensional list of quanta or ‘blobs’ partially covering each other). If the lines, which represent special types of images like characters, are broken up by .some spurious effect during image acquisition, the detecting operators of invariant features cannot be used. In this paper, a method for filling in the gaps between known pieces of images of characters obtained from a dot-matrix printer is presented so that the existing algorithm may be applied to these ‘restored’ continuous line images.
of printed character recognition has been formulated and tested. In the following section, a real example is sketched out which enables a better understanding of the problem to be treated. Some important results of the theory of projections onto convex sets are then summarized. This theory provides tools which take into account all available a priori information which can be mathematically inserted in equations describing an image reconstruction process.
EXAMPLE OF NON-REGULAR LINE IMAGES TO BE PROCESSED Applied to character recognition, the line detector aigorithm discussed above works well only with ‘continuous’ characters which have been drawn in a regular manner and do not show discontinuities. This is the case with characters from a typewriter, laser writer,
Figure I. Examples of ink blot shapes which form character printed with a needle printer
Figure 3. Two examples of ‘d~sconti~uo~’
226
a
Figure 2. Schema needle printer
of
needle matrix of print head of a
images of characters ‘e’ and ‘t’ digitized by a VICOM image and vision computing
Therefore the variation and inaccuracy of distances must be taken into account to determine the best approach to the solution. Second, the print head is an array of 7 x 9 needles. Along the columns (including nine needles each) the distance between two successive needles is smaller than it is along the rows; this explains why the blots meet in the vertical direction. Third, the background noise is not a problem because it does not reach a high level and it can be eliminated by means of a suitable threshold. Noise is also due to non-unifo~ity, not discernible to the naked eye, of the white colour of the paper where the printing is carried out. Finally, because of light diffraction, a perfect binary image, mathematically represented by a rectangular function, will be recorded after digitization and reprinting as a bell-shaped curve as illustrated in Figure 4. This phenomenon explains why the very close blots meet in the printed image. Figure 5a illustrates an example of a
plotting table and so on. When irregularities, such as white regions or gaps occur in some lines of a character, that character cannot be recognized by the algorithm. Among these non-regular characters, those printed with a needle printer or dot-matrix printer have been considered. A character obtained from a needle printer appears as a set of successive blots of ink which describe the letter. These blots should be identical black circles but, in reality, two identical blots cannot be found and their shapes are a priori undefined. Figure 1 shows some examples of these blots put under a magnifying glass. Our aim was to work out a procedure for making all successive blots meet, in order to form a continuous line which is as regular as possible and which should have to cover the whole character represented by that set of blots. The print head of a needle printer is essentially made up of a needle matrix whose shape can be seen in Figure 2. The size, number and distance between two needles depend on the printer model. A needle is in home position when no signal is passing. Whenever a signal passes through a needle, that needle moves forward and prints an ink blot at the required location. Whenever the needle printer receives a letter code, a signal is sent to corresponding needles and an image is printed out looking like the desired letter. Characters produced by a needle printer of the CICRP have been processed. The images to be treated were digitized by a VICOM with 256 grey levels expressed in hexadecimal code. Each character with 37 grey-levels in an array made of 64 rows and 64 columns has been printed out by overprinting 37 well-chosen characters as suggested by Theo Pavlidis’.Figure 3 shows a real example of the letters ‘e’ and ‘t’ which were digitized in hexadecimal by a VICOM, transcoded in decimal code and then overprinted. Considering these images, the following remarks can be made. First, although the needles are regularly spaced, the distance between two successive blots is not constant. Nevertheless, that distance remains within limits.
a
diffraction t t
Figure 4. Diffraction of light changes a uniform pulse into image represented by a bell-shaped curve
D
Figure 5. a, Printed character ‘e’ simulated with five grey-levels; be reconstructed vol 7 no 3 august 1980
S~MHHHHHHHHWHNXWHHWNNS
sssssssssssssssssssss
b, corresponding
‘continuous’ image which has to
227
discontinuous character, simulated with five grey-levels (from zero corresponding to white to four corresponding to black), stored in some file which was overprinted with five grey-levels, each level being simulated by a particular appropriate letter. Discontinuities within a line may be considered as a result of a periodic masking effect. Figure 5b shows a corresponding type of continuous image which in principle has to be reconstructed by eliminating the effect of the periodic mask. These images each lie inside a 64 x 64 window. The problem was first treated in a simple manner using a classical method of local averaging and thresholding (see reference 6, for example). This approach is rapid, leads to a good smoothing and reduces the noise. But, it rounds off the angles and makes the blobs which are too small look like noise. A method was then developed which took into account some pertinent available a priori information that can be considered as ‘projections’ onto closed convex sets. This method is presented in the following sections.
IMAGE PROCESSING BY A METHOD OF ‘PROJECTIONS’ ONTO CLOSED CONVEX SETS The method of projections onto convex sets is based upon a mathematical theory of functional operators. The theorem of aIternating projections onto linear subspaces of a Hilbert space, stated by von Neuman’ and extended to all convex sets by Gubin et al.‘, was first applied to image restoration by Youla”. The theory enables images to be reconstructed from a priori information such as projections. The image functions to be reconstructed are all supposed to belong to the Hilbert space H which consists of the squareintegrable Fourier-transfo~able functions f, g, etc., provided with the inner product: (f, g) = / +=I += f(x, y) g*(x, y) dx dy --r --z
(1)
and with the norm: //f/j = fff, f)]“”
(2)
In that space H, two functions f and g are considered equal if Ilf - gll = 0. A projection of f onto a closed subset S of H is a nearest neighbour of f in S, which is represented by a function g = Psf, where Ps is the projection operator onto S of the functions belonging to H, so that: ~lg-f~~=min~~y-f~~
foreveryyES
i=l,...,m
(5)
where Ii are relaxation parameters such that 0 < =; < 2, can be used sequentially or eventually as compositions to restore missing information in the image. If hi = 1, then Ti = Pi is a pure projection operator. The method of projections was extended to signal or image recovery using generalized projections onto non-necessarily convex sets by Levi and Stark14. In the problem given here, each discontinuous letter Xdisc is considered as the projection of a continuous letter x,,,~ onto a discontinuous periodic space, as shown in Figure 6. Let a be the distance between the centres of needles in the horizontal direction, b the
0
0
0
0
(4)
where &= I-Ps is the projection operator onto i. S, I being the identity operator. Every element f belonging to a convex set C is a fixed point for the operator P,. Von Neumann’s theorem states the following: 228
T, = 1 + h;(P; - l),
(3)
If J_S is the orthogonal complement of S, every function f of H can be uniquely written as f = g + h, i.e.: f=P,f+Q,f
Let T = P,P2. . . P,, m > 1 and C,, be the intersection of the m corresponding linear subspaces Cj of fri, then the sequence {Tkf} converges in the norm to the projection f onto Cc. In the case of linear projections, Youla has proved that an image fcan be uniquely determined by a single projection g = PJ onto a known closed subspace S, if one has a priori information concerning the mathematical properties of the unknown function f, that is if one knows that flies in some other closed subspace &_ If P,, Q,, Ph and Qb are the projection operators onto S then Youla’s iS*, Sh and J-S,, respectively, t&orem is as follows. The necessary and sufficient condition for uniquely determining f from g is that the intersection of Sh and IS, is empty. Then, for reconstruction, an algorithm exists that only uses projection operations onto Sh and is,. Consider, for example, the intersection of the closed subspace Sb of all signals band-limited to v < b and the closed subspace S, of H consisting of all signals with a limited support [--Xa, +X,1. The elements of is, are functions that vanish over 1x /
Figure 6. General shape of 20 mask put on the image image and vision computing
distance between the centres of needles in the vertical direction and r the needle radius. That plane of projection can be considered as a convolution of a 2D rectangular Poisson distribution (‘bed of nails’), bn{a,h), by a circle with radius r. Let us represent a convolution operator by the symbol ‘*‘, a multiplication in the real space by ‘.‘, the letter to be treated by ‘x’ and the circle with radius r by ‘circle(r)‘. The following can be written: Xdiac
=
xccmt
[bn(a, h)*circle(r)]
By taking the Fourier transform equation (6). one obtains: Ff&&
= FT%,,)
* IFf[bn(a,
(6) of both sides of
b)l .m[cirWr)l>. (7)
The Fourier transform of bn(a, b) is BN(l/a, l/b). The Fourier transform of circle(r) is an Airy function Airy[(u” + v’)r”] wh ere u and v are spatial frequencies. With the notations XdiScfor FT(Xdisc) and X,,,, for FI’(x,,,~), one has: xdisc
=
.. {BN(lla, xcont
l/b). Airy[(u* + v*)“~]}
(8)
The Airy function is defined as Airy(x) = J(x)/x where J(x) is a first order Bessel function: .Figure‘ 7 schematizes. in a longitudinal section. the shaoe of the Airv function, the ‘product of the kiry fun&ion by a B6l function and the convolution of that product by X,,,,. The aim is to eliminate the periodicity of this configuration so that only the central part which represents the continuous letter of interest is kept. The following a priori information exists: first, the
luminances of the input image and of the image to be reconstructed are non-negative. Second, the character to be recognized is continuous: it must finally appear without the periodicity of the input image. Finally, the luminance of the final image must lie between two nonnegative bounds a and b. The principle of the method is that if there is m il priori information of a convex type, the unknown image function f belongs to the closed convex intersection Cc of m well-defined closed convex sets C,, . . . , C,,,. When the sets Cj and the corresponding nonnecessarily linear projection operators are given, the problem is to find some point of C,. Every point of C0 is a fixed point for each operator Pi if and only if all Ci are convex sets. If Ca is not empty and if each Pi(i = 1, . . .t m) is actually realizable, every fixed point of the product P, . . . P, belongs to Cn and the problem can be recursively solved by the algorithm described by the following equation: fkc, =PmP,-,
. . . Plfk,
f. being arbitrary
(9)
With the three a priori pieces of information, the following simpler algorithm converges to the required image: f!S+i= PZPlfk
(10)
where the projection operators PI and PZ are such that: PI ]f(x, Y)] = Ff-*IC,,,k Cm& = t * 10,
.F(K v)]
if (u, v) @mask if (u, v) /mask.
(11) (12)
and:
P2lWT
Y)l=
f(x, Y) CY P
if (YS f(x, y) Sp if f(x, Y> < 4) if f(x, y) > ho.
(13)
It is easy to show that Pi and PZ are convex but nonlinear. For example, for P, consider f(x, Y) = a+0 and g(x, y) = b,>O. Then, there is: f(x, y) + g(x, y) > b0 and. following the definition of P?: P,(f+g)=P and P,(f) + P*(g) =a -t/3. Thus P2 is not linear. For proving the convexity of P2, consider both functions f(x, y) and g(x, y) such that: atr
Figure 7. Illustration of process that leads to undesirable periodicities in the spectrum of the image to be reconstrutted. a, Fourier transform of one circle (Airy function) of the mask; b, product if an Airy function by the Poisson distribution BN(lIu, lib); c, convolution of the spectrum of the image of interest by d~tribution (b) vu1 7 no 3 uugust 1~~~
Figure 8. 1 D simple example; a, two separated parts of the same ‘continuous’ line; b, continuous line obtained at the end of the processing 229
removal of discontinuities. In that case, the projection operators Pi and Pz are such that:
and
P, [f(x)] = FT’
(14) The method was first applied to a simple 1D case which might be, for example, some section in the simulated discontinuous letter ‘e’ in Figure 5a. The shape of the 1D luminance function is sketched in Figure 8a. Figure 8b shows the look of the continuous line obtained after
[Rect
(15)
where the mask is: 1
if 1c4I< u.
0
elsewhere
(16)
and: if OSf(x)dl if f(x) > 1 if f(x) d a0
f(x)
Pm)1 = 1 0
(17)
The parameter a0 was found experimentally. It corresponds to the amplitude of the first lobe of the The parameter a0 was found experimentally. It corresponds to the amplitude of the first lobe of the undulations due to spectrum truncation. Figure 9 illustrates the use of projection operators Pi and P2 applied to some arbitrary functions f(x) or g(x).
:_.::z
....
..A..=“L ..--_ ... .. . .-
:;z: :‘_:
.-
-
..-..
:z
._”
.:I
.:‘-“I zzzZ.-.-. .-...-.-_‘i=_
:ZZ .
,&
.:;:
,::;
I.
-
‘Z
‘ZY
_I.
‘r:
G _
zz:
::‘-: ..._. ::::5:
g; .. ..
2:::
._. ‘-I :=: :=.._ ,,
a
., ,,
...
-’
“:
:=:
b
... . ..----.-z.
...-. .-. .-.. ..-.
Figure 9. Illustration of use of projection operators Pt and P2 applied to an arbitrary 1D function f(x) ,_,...........
.-. -, :=; ..,-. =...~......
“_“,
.-. ..-:
A.-.“.=. :-_ c
s = -...
,a. ‘_q
,_“,.I -, -. -. .-. .__-I’ -*.
..-. ,-. ‘I
;zzi :zEZ
zzc=i ,,__.“__“_. .
d-
Figure 10. Three different theoretical shapes of low-pass filter used in the experiments
-,,..-.
,..,_. ..=.
..‘.‘-.
“I”
..-.-.
e
.m
Figure 12. Successive stages of reconstruction of simulated letter ‘e’. a, Input image without noise; b, after the first iteration; c to e, second to fourth iterations; t’, result obtained after fifth iteration
a Figure Il. Actual shapes of low-pass filter, taking into account properties of the FFT algorithm
230
image and vision computing
So, the filter to be used is a low-pass one. As shown above, P, and P2 are convex nonlinear operators and the algorithm converges in the norm towards the solution. That result was obtained rapidly with only one iteration. In the case of an image, which is a 2D repartition of luminance, there are several possibilities for choosing the filter function. Three elementary geometrical forms have been tried, their sizes depended on fequencies u and v which were to be masked (see Figure 10). These sizes are not necessarily fixed: they can be modified by taking into account the number of the current iteration. Al1 Fourier transformations were performed using a FI?I’ algorithm. The actual shapes af the masks in the Fourier space are schematized in Figure 1. The method
has been tested on several real images of characters and their restoration can be seen in Figures 12 to 15 (as examples). These results were satisfactory and were obtained with five iterations at the most.
CONCLUSIQN Complementary methads are preferred for recognizing discontinuous line images of characters rather than searching for direct feature detectors as it has been proved that this method of processing is more efficient. The existent operators used in this laboratory are robust, view lines as regular regions and then identify primitive features. Considering this and the fact that a
.,?1
. .
b
Figure 13. Reconstruction
of letter ‘t? in a real case. a, Image to be prucessei, _ -~
I..“..
c digitized by a VICOM; b, image
231
broken out character cannot be recognized by those operators, a preliminary stage has been added for transforming the non-regular line image into an equivalent regular one, provided that the true nature of the character was not changed when filling up the gaps. The method of projection is slower than classical methods of neighbourhood averaging and thresholding but gives better results especially in the case of letters which are difficult to treat such ‘x’ and ‘m’.
5 6
7 8
ACKNOWLEDGEMENT We are grateful discussions.
to Professor
J C Simon for many useful
9
10
REFERENCES Simon, J C ‘Invariance in pattern recognition. Application to line images’ IBM seminar, Paris, France (April 1985) pp 1-32 Simon, J C ‘Invariance en reconnaissance des formes’ Prac. Cognitiva ‘8.5 Paris, France (June 1985) pp l-11 Simon, J C ‘Errors and uncertainties in feature recognition’ Proc. Conf. on Mathematics and its Applications in Remote Sensing Danbury, Essex, UK (May 1986) pp l-9 Simon, J C ‘Invariance in pattern recognition. ADDlication to line images’ Proc. 8th Int. Conf. on
232
11 12
13
14
Pattern Recognition Paris, France (October 1986) pp 11-24 Pavlidis, T Graphics and image processing Springer Verlag, Berlin-Heidelberg, FRG (1982) Gonzalez, R C ‘Image enhancement and restoration’ Handbook of pattern recognition and image processing Academic Press, Inc., NY, USA (1986) p 200 Von Neumann, J ‘Functional operators’ Ann. Math. Vol II (1950) p 55 Gubin, L G, Polyak, B T and Raik, E V ‘The method of projections for finding the common set of convex sets’ USSR Computational Math. Phys. Vol 7 No 6 (1967) pp l-24 Youla, D C ‘Generalized image restoration by the method of alternating projections’ IEEE Trans. Circuits Syst. Vol CAS-25 (September 1978) Arsac, J Transformation de Fourier et theorie des distributions Dunod, Paris, France (1961) Simon, J C La reconnaissance des formes par algorithmes Masson, Paris, France (1984) Stark, H, Cabana, D and Webb, H ‘Restoration of arbitrary finite-energy optical objects from limited spatial and spectral information’ J. Opt. Sot. Am. Vol71 No 6 (June 1981) pp 635-642 Sezan, M I and Stark, H ‘Image restoration by the method of convex projections: Part 2 - Applications and numerical results’ IEEE Trans. Medical Imaging Vol MI-l No 2 (October 1982) pp 95-101 Levi, A and Stark, H ‘Image restoration by the method of generalized projections with applications to restoration from magnitude’ J. Opt. Sot. Am. (September 1984) pp 932-943
image and vision computing
The basis of the algorithm is a robust line detector which first finds out line precursors from the pixels by scanning the grey-level image in both horizontal and vertical directions and by applying an appropriate set of rules. Quanta are then identified and lines are readily detected in regions where quanta overlap, in accordance with the above definition of a line. In practice, in the regions where only line precursors have been found but where quanta do not overlap, overlapping is still assumed. The final stage of the algorithm consists of withdrawing lines and quanta (after having removed possible ambiguities) in order to save only the primitive features of interest. Such primitive features are the end of a line, a change of direction, a line crossing or a fork. They do not alter with changes in scale of the pixel Keywords: image processing, line images, projections, values or with changes in size (in some ranges), with periodicities displacements in the observed plane or with topological deformations. Using this method, the first level in the identification process is overcome by using operators which give robust results with low time and space The general problem we are concerned with is to complexities. The basic principle is to perform two identify an object from some image made of lines. In successive and complementary general operations: a the case of ‘continuous’ lines, an efficient algorithm has detection of regular regions (lines and background) been formulated and implemented in the Laforia followed by subtractions leading to feature identificaLaboratory, University Pierre et Marie Curie, Paris, tion. France as reported, for example, by J C Simorrm4. In However, these operators intended for line detection Simon’s papers, a line is defined as a ‘monodimensional cannot be used when input line images include ‘lines’ list of quanta partially covering each other’. The term made of dots or dashes due to a masking effect. In this quantum of information (or ‘blob’) is used here for case a line is no longer a regular region made of referring to the ‘shortest black pulse’ made of at least characteristic continuously overlapping quanta. On the three non-white pixels in any direction. The various contrary, each dot or dash, which is a piece of the true ‘blobs’ which form a line contain approximately the line, is taken by the line detector as an individual same number of non-white pixels. isolated line with two ends in the case of a dash, and a curved boundary if it is a dot. Insignificant gaps are Laboratoire LAFORIA-Groupe de Reconnaissance des Formes et Traitement d’Tmages, UniversitC Pierre et Marie Curie, tour 45-46. found between those short ‘lines’. 4. place Jussieu. 75230 Paris Cedex 05, France A method for solving such a problem in the context 0262~8856/89/030225-OS $03.00 @ 1989 Butterworth & Co. (Publishers) Ltd vol 7 no 3 august 1989 225 An efficient algorithm for analysing and recognizing line images has been formulated. It works well when applied to ‘continuous’ line images (a continuous line is defined here as a monodimensional list of quanta or ‘blobs’ partially covering each other). If the lines, which represent special types of images like characters, are broken up by .some spurious effect during image acquisition, the detecting operators of invariant features cannot be used. In this paper, a method for filling in the gaps between known pieces of images of characters obtained from a dot-matrix printer is presented so that the existing algorithm may be applied to these ‘restored’ continuous line images.
of printed character recognition has been formulated and tested. In the following section, a real example is sketched out which enables a better understanding of the problem to be treated. Some important results of the theory of projections onto convex sets are then summarized. This theory provides tools which take into account all available a priori information which can be mathematically inserted in equations describing an image reconstruction process.
EXAMPLE OF NON-REGULAR LINE IMAGES TO BE PROCESSED Applied to character recognition, the line detector aigorithm discussed above works well only with ‘continuous’ characters which have been drawn in a regular manner and do not show discontinuities. This is the case with characters from a typewriter, laser writer,
Figure I. Examples of ink blot shapes which form character printed with a needle printer
Figure 3. Two examples of ‘d~sconti~uo~’
226
a
Figure 2. Schema needle printer
of
needle matrix of print head of a
images of characters ‘e’ and ‘t’ digitized by a VICOM image and vision computing
Therefore the variation and inaccuracy of distances must be taken into account to determine the best approach to the solution. Second, the print head is an array of 7 x 9 needles. Along the columns (including nine needles each) the distance between two successive needles is smaller than it is along the rows; this explains why the blots meet in the vertical direction. Third, the background noise is not a problem because it does not reach a high level and it can be eliminated by means of a suitable threshold. Noise is also due to non-unifo~ity, not discernible to the naked eye, of the white colour of the paper where the printing is carried out. Finally, because of light diffraction, a perfect binary image, mathematically represented by a rectangular function, will be recorded after digitization and reprinting as a bell-shaped curve as illustrated in Figure 4. This phenomenon explains why the very close blots meet in the printed image. Figure 5a illustrates an example of a
plotting table and so on. When irregularities, such as white regions or gaps occur in some lines of a character, that character cannot be recognized by the algorithm. Among these non-regular characters, those printed with a needle printer or dot-matrix printer have been considered. A character obtained from a needle printer appears as a set of successive blots of ink which describe the letter. These blots should be identical black circles but, in reality, two identical blots cannot be found and their shapes are a priori undefined. Figure 1 shows some examples of these blots put under a magnifying glass. Our aim was to work out a procedure for making all successive blots meet, in order to form a continuous line which is as regular as possible and which should have to cover the whole character represented by that set of blots. The print head of a needle printer is essentially made up of a needle matrix whose shape can be seen in Figure 2. The size, number and distance between two needles depend on the printer model. A needle is in home position when no signal is passing. Whenever a signal passes through a needle, that needle moves forward and prints an ink blot at the required location. Whenever the needle printer receives a letter code, a signal is sent to corresponding needles and an image is printed out looking like the desired letter. Characters produced by a needle printer of the CICRP have been processed. The images to be treated were digitized by a VICOM with 256 grey levels expressed in hexadecimal code. Each character with 37 grey-levels in an array made of 64 rows and 64 columns has been printed out by overprinting 37 well-chosen characters as suggested by Theo Pavlidis’.Figure 3 shows a real example of the letters ‘e’ and ‘t’ which were digitized in hexadecimal by a VICOM, transcoded in decimal code and then overprinted. Considering these images, the following remarks can be made. First, although the needles are regularly spaced, the distance between two successive blots is not constant. Nevertheless, that distance remains within limits.
a
diffraction t t
Figure 4. Diffraction of light changes a uniform pulse into image represented by a bell-shaped curve
D
Figure 5. a, Printed character ‘e’ simulated with five grey-levels; be reconstructed vol 7 no 3 august 1980
S~MHHHHHHHHWHNXWHHWNNS
sssssssssssssssssssss
b, corresponding
‘continuous’ image which has to
227
discontinuous character, simulated with five grey-levels (from zero corresponding to white to four corresponding to black), stored in some file which was overprinted with five grey-levels, each level being simulated by a particular appropriate letter. Discontinuities within a line may be considered as a result of a periodic masking effect. Figure 5b shows a corresponding type of continuous image which in principle has to be reconstructed by eliminating the effect of the periodic mask. These images each lie inside a 64 x 64 window. The problem was first treated in a simple manner using a classical method of local averaging and thresholding (see reference 6, for example). This approach is rapid, leads to a good smoothing and reduces the noise. But, it rounds off the angles and makes the blobs which are too small look like noise. A method was then developed which took into account some pertinent available a priori information that can be considered as ‘projections’ onto closed convex sets. This method is presented in the following sections.
IMAGE PROCESSING BY A METHOD OF ‘PROJECTIONS’ ONTO CLOSED CONVEX SETS The method of projections onto convex sets is based upon a mathematical theory of functional operators. The theorem of aIternating projections onto linear subspaces of a Hilbert space, stated by von Neuman’ and extended to all convex sets by Gubin et al.‘, was first applied to image restoration by Youla”. The theory enables images to be reconstructed from a priori information such as projections. The image functions to be reconstructed are all supposed to belong to the Hilbert space H which consists of the squareintegrable Fourier-transfo~able functions f, g, etc., provided with the inner product: (f, g) = / +=I += f(x, y) g*(x, y) dx dy --r --z
(1)
and with the norm: //f/j = fff, f)]“”
(2)
In that space H, two functions f and g are considered equal if Ilf - gll = 0. A projection of f onto a closed subset S of H is a nearest neighbour of f in S, which is represented by a function g = Psf, where Ps is the projection operator onto S of the functions belonging to H, so that: ~lg-f~~=min~~y-f~~
foreveryyES
i=l,...,m
(5)
where Ii are relaxation parameters such that 0 < =; < 2, can be used sequentially or eventually as compositions to restore missing information in the image. If hi = 1, then Ti = Pi is a pure projection operator. The method of projections was extended to signal or image recovery using generalized projections onto non-necessarily convex sets by Levi and Stark14. In the problem given here, each discontinuous letter Xdisc is considered as the projection of a continuous letter x,,,~ onto a discontinuous periodic space, as shown in Figure 6. Let a be the distance between the centres of needles in the horizontal direction, b the
0
0
0
0
(4)
where &= I-Ps is the projection operator onto i. S, I being the identity operator. Every element f belonging to a convex set C is a fixed point for the operator P,. Von Neumann’s theorem states the following: 228
T, = 1 + h;(P; - l),
(3)
If J_S is the orthogonal complement of S, every function f of H can be uniquely written as f = g + h, i.e.: f=P,f+Q,f
Let T = P,P2. . . P,, m > 1 and C,, be the intersection of the m corresponding linear subspaces Cj of fri, then the sequence {Tkf} converges in the norm to the projection f onto Cc. In the case of linear projections, Youla has proved that an image fcan be uniquely determined by a single projection g = PJ onto a known closed subspace S, if one has a priori information concerning the mathematical properties of the unknown function f, that is if one knows that flies in some other closed subspace &_ If P,, Q,, Ph and Qb are the projection operators onto S then Youla’s iS*, Sh and J-S,, respectively, t&orem is as follows. The necessary and sufficient condition for uniquely determining f from g is that the intersection of Sh and IS, is empty. Then, for reconstruction, an algorithm exists that only uses projection operations onto Sh and is,. Consider, for example, the intersection of the closed subspace Sb of all signals band-limited to v < b and the closed subspace S, of H consisting of all signals with a limited support [--Xa, +X,1. The elements of is, are functions that vanish over 1x /
Figure 6. General shape of 20 mask put on the image image and vision computing
distance between the centres of needles in the vertical direction and r the needle radius. That plane of projection can be considered as a convolution of a 2D rectangular Poisson distribution (‘bed of nails’), bn{a,h), by a circle with radius r. Let us represent a convolution operator by the symbol ‘*‘, a multiplication in the real space by ‘.‘, the letter to be treated by ‘x’ and the circle with radius r by ‘circle(r)‘. The following can be written: Xdiac
=
xccmt
[bn(a, h)*circle(r)]
By taking the Fourier transform equation (6). one obtains: Ff&&
= FT%,,)
* IFf[bn(a,
(6) of both sides of
b)l .m[cirWr)l>. (7)
The Fourier transform of bn(a, b) is BN(l/a, l/b). The Fourier transform of circle(r) is an Airy function Airy[(u” + v’)r”] wh ere u and v are spatial frequencies. With the notations XdiScfor FT(Xdisc) and X,,,, for FI’(x,,,~), one has: xdisc
=
.. {BN(lla, xcont
l/b). Airy[(u* + v*)“~]}
(8)
The Airy function is defined as Airy(x) = J(x)/x where J(x) is a first order Bessel function: .Figure‘ 7 schematizes. in a longitudinal section. the shaoe of the Airv function, the ‘product of the kiry fun&ion by a B6l function and the convolution of that product by X,,,,. The aim is to eliminate the periodicity of this configuration so that only the central part which represents the continuous letter of interest is kept. The following a priori information exists: first, the
luminances of the input image and of the image to be reconstructed are non-negative. Second, the character to be recognized is continuous: it must finally appear without the periodicity of the input image. Finally, the luminance of the final image must lie between two nonnegative bounds a and b. The principle of the method is that if there is m il priori information of a convex type, the unknown image function f belongs to the closed convex intersection Cc of m well-defined closed convex sets C,, . . . , C,,,. When the sets Cj and the corresponding nonnecessarily linear projection operators are given, the problem is to find some point of C,. Every point of C0 is a fixed point for each operator Pi if and only if all Ci are convex sets. If Ca is not empty and if each Pi(i = 1, . . .t m) is actually realizable, every fixed point of the product P, . . . P, belongs to Cn and the problem can be recursively solved by the algorithm described by the following equation: fkc, =PmP,-,
. . . Plfk,
f. being arbitrary
(9)
With the three a priori pieces of information, the following simpler algorithm converges to the required image: f!S+i= PZPlfk
(10)
where the projection operators PI and PZ are such that: PI ]f(x, Y)] = Ff-*IC,,,k Cm& = t * 10,
.F(K v)]
if (u, v) @mask if (u, v) /mask.
(11) (12)
and:
P2lWT
Y)l=
f(x, Y) CY P
if (YS f(x, y) Sp if f(x, Y> < 4) if f(x, y) > ho.
(13)
It is easy to show that Pi and PZ are convex but nonlinear. For example, for P, consider f(x, Y) = a+0 and g(x, y) = b,>O. Then, there is: f(x, y) + g(x, y) > b0 and. following the definition of P?: P,(f+g)=P and P,(f) + P*(g) =a -t/3. Thus P2 is not linear. For proving the convexity of P2, consider both functions f(x, y) and g(x, y) such that: atr
Figure 7. Illustration of process that leads to undesirable periodicities in the spectrum of the image to be reconstrutted. a, Fourier transform of one circle (Airy function) of the mask; b, product if an Airy function by the Poisson distribution BN(lIu, lib); c, convolution of the spectrum of the image of interest by d~tribution (b) vu1 7 no 3 uugust 1~~~
Figure 8. 1 D simple example; a, two separated parts of the same ‘continuous’ line; b, continuous line obtained at the end of the processing 229
removal of discontinuities. In that case, the projection operators Pi and Pz are such that:
and
P, [f(x)] = FT’
(14) The method was first applied to a simple 1D case which might be, for example, some section in the simulated discontinuous letter ‘e’ in Figure 5a. The shape of the 1D luminance function is sketched in Figure 8a. Figure 8b shows the look of the continuous line obtained after
[Rect
(15)
where the mask is: 1
if 1c4I< u.
0
elsewhere
(16)
and: if OSf(x)dl if f(x) > 1 if f(x) d a0
f(x)
Pm)1 = 1 0
(17)
The parameter a0 was found experimentally. It corresponds to the amplitude of the first lobe of the The parameter a0 was found experimentally. It corresponds to the amplitude of the first lobe of the undulations due to spectrum truncation. Figure 9 illustrates the use of projection operators Pi and P2 applied to some arbitrary functions f(x) or g(x).
:_.::z
....
..A..=“L ..--_ ... .. . .-
:;z: :‘_:
.-
-
..-..
:z
._”
.:I
.:‘-“I zzzZ.-.-. .-...-.-_‘i=_
:ZZ .
,&
.:;:
,::;
I.
-
‘Z
‘ZY
_I.
‘r:
G _
zz:
::‘-: ..._. ::::5:
g; .. ..
2:::
._. ‘-I :=: :=.._ ,,
a
., ,,
...
-’
“:
:=:
b
... . ..----.-z.
...-. .-. .-.. ..-.
Figure 9. Illustration of use of projection operators Pt and P2 applied to an arbitrary 1D function f(x) ,_,...........
.-. -, :=; ..,-. =...~......
“_“,
.-. ..-:
A.-.“.=. :-_ c
s = -...
,a. ‘_q
,_“,.I -, -. -. .-. .__-I’ -*.
..-. ,-. ‘I
;zzi :zEZ
zzc=i ,,__.“__“_. .
d-
Figure 10. Three different theoretical shapes of low-pass filter used in the experiments
-,,..-.
,..,_. ..=.
..‘.‘-.
“I”
..-.-.
e
.m
Figure 12. Successive stages of reconstruction of simulated letter ‘e’. a, Input image without noise; b, after the first iteration; c to e, second to fourth iterations; t’, result obtained after fifth iteration
a Figure Il. Actual shapes of low-pass filter, taking into account properties of the FFT algorithm
230
image and vision computing
So, the filter to be used is a low-pass one. As shown above, P, and P2 are convex nonlinear operators and the algorithm converges in the norm towards the solution. That result was obtained rapidly with only one iteration. In the case of an image, which is a 2D repartition of luminance, there are several possibilities for choosing the filter function. Three elementary geometrical forms have been tried, their sizes depended on fequencies u and v which were to be masked (see Figure 10). These sizes are not necessarily fixed: they can be modified by taking into account the number of the current iteration. Al1 Fourier transformations were performed using a FI?I’ algorithm. The actual shapes af the masks in the Fourier space are schematized in Figure 1. The method
has been tested on several real images of characters and their restoration can be seen in Figures 12 to 15 (as examples). These results were satisfactory and were obtained with five iterations at the most.
CONCLUSIQN Complementary methads are preferred for recognizing discontinuous line images of characters rather than searching for direct feature detectors as it has been proved that this method of processing is more efficient. The existent operators used in this laboratory are robust, view lines as regular regions and then identify primitive features. Considering this and the fact that a
.,?1
. .
b
Figure 13. Reconstruction
of letter ‘t? in a real case. a, Image to be prucessei, _ -~
I..“..
c digitized by a VICOM; b, image
231
broken out character cannot be recognized by those operators, a preliminary stage has been added for transforming the non-regular line image into an equivalent regular one, provided that the true nature of the character was not changed when filling up the gaps. The method of projection is slower than classical methods of neighbourhood averaging and thresholding but gives better results especially in the case of letters which are difficult to treat such ‘x’ and ‘m’.
5 6
7 8
ACKNOWLEDGEMENT We are grateful discussions.
to Professor
J C Simon for many useful
9
10
REFERENCES Simon, J C ‘Invariance in pattern recognition. Application to line images’ IBM seminar, Paris, France (April 1985) pp 1-32 Simon, J C ‘Invariance en reconnaissance des formes’ Prac. Cognitiva ‘8.5 Paris, France (June 1985) pp l-11 Simon, J C ‘Errors and uncertainties in feature recognition’ Proc. Conf. on Mathematics and its Applications in Remote Sensing Danbury, Essex, UK (May 1986) pp l-9 Simon, J C ‘Invariance in pattern recognition. ADDlication to line images’ Proc. 8th Int. Conf. on
232
11 12
13
14
Pattern Recognition Paris, France (October 1986) pp 11-24 Pavlidis, T Graphics and image processing Springer Verlag, Berlin-Heidelberg, FRG (1982) Gonzalez, R C ‘Image enhancement and restoration’ Handbook of pattern recognition and image processing Academic Press, Inc., NY, USA (1986) p 200 Von Neumann, J ‘Functional operators’ Ann. Math. Vol II (1950) p 55 Gubin, L G, Polyak, B T and Raik, E V ‘The method of projections for finding the common set of convex sets’ USSR Computational Math. Phys. Vol 7 No 6 (1967) pp l-24 Youla, D C ‘Generalized image restoration by the method of alternating projections’ IEEE Trans. Circuits Syst. Vol CAS-25 (September 1978) Arsac, J Transformation de Fourier et theorie des distributions Dunod, Paris, France (1961) Simon, J C La reconnaissance des formes par algorithmes Masson, Paris, France (1984) Stark, H, Cabana, D and Webb, H ‘Restoration of arbitrary finite-energy optical objects from limited spatial and spectral information’ J. Opt. Sot. Am. Vol71 No 6 (June 1981) pp 635-642 Sezan, M I and Stark, H ‘Image restoration by the method of convex projections: Part 2 - Applications and numerical results’ IEEE Trans. Medical Imaging Vol MI-l No 2 (October 1982) pp 95-101 Levi, A and Stark, H ‘Image restoration by the method of generalized projections with applications to restoration from magnitude’ J. Opt. Sot. Am. (September 1984) pp 932-943
image and vision computing