- Email: [email protected]
Computer Vision Application to Automatic Number Plate Recognition
Copyright © IFAC Transportation Systems, Tianjin- PRC. 1994
COMPUTER VISION APPLICATION TO AUTOMATIC NUMBER PLATE RECOGNITION Maged M. M. FAHMY TORG, University of Newcastle Upon Tyne, NEI 7RU, UK. Abstract. Automatic number plate recognition, using image processing techniques, is indicating new ways of capturing traffic data. Such systems could play an important role in automatic toll collection. The requirement is to develop a practical system able to work under a variety of number plate geometries, sizes, and different fonts. The target of this research is to use neural network techniques to recognize characters in a number plate, so as to achieve an acceptable level of accuracy. In order to depend on such systems to replace the conventional traffic data collection methods, their accuracy level must match conventional methods. An advantage of using automatic systems is that it does not depend on human factors and it can work under difficult circumstances.
Key Words. Image processing; image thresholding; feature analysis; connected component; skeletonization; neural networks; self organizing model; character recognition.
1. INTRODUCTION
recognition.
Current research area in traffic toll collection tends to use automatic, i.e. nonstop, methods. These methods have the advantages of not disrupting the traffic flow. With the appearance of automatic vehicle identification (AVI) technology allowing non-stop electronic toll collection, it is important that a more efficient method of enforcement is found to register the non-paying vehicles. Automatic number-plate recognition technology is seen as a promising method of A VI without disrupting the traffic flow. At the toll plazas, license-plate recognition can identify vehicles that pass without paying their toll and automatically alert the authorities. Computer vision for automatic number-plate recognition has three parts; number-plate image capturing, image analysis and character
169
2. NUMBER-PLATE IMAGE CAPTURING The number-plate image capturing process consists of a configuration where a video camera is mounted at a suitable angle, height and distance so as to capture a scene of the front ,or rear, part of the incoming vehicle. The video camera should have high shutter speed to cope with speedy incoming vehicles. The captured image is being grabbed using an image grabber card mounted in a host computer (see Fig. I). The grabbed image is then stored in its grey-level form within the image grabber or the dynamic memory of the computer to be processed later.
number plate image noise (Gonzalez and Wintz, 1987). The automatic numberplate reading system executes an adaptive thresholding algorithm on the entire digitized scene. This process helps properly segment the vehicle numberplate and its characters from the rest of the scene.
Tr.n •• t light
5. NUMBER-PLATE IMAGE PROCESSING
Fig.I. Number-plate image capturing.
3. THE NATURE OF ACQUIRED NUMBER-PLATE IMAGES The captured number-plate image is expected to have noise. There are many sources of the noise. Electrical devices used to obtain and store the image all produce some form of electrical noise. The weather conditions under which the image is captured may add different forms of noise to the image. Dirt and dust are other sources of noise. It is also noted that image processing techniques are affected in some way by changes in overall light intensity. Consequently, changes in weather conditions could have a marked effect on the appearance of number plate characters within the scene, and so, affect the performance of the recogniser. These noises require preprocessing of the number-plate image before recognition.
4. NUMBER-PLATE IMAGE PREPROCESSING Noise can be reduced in a digital image by the application of a filter. A filter is a function which operates on a small neighbourhood of pixels. A convolution mask of 3 x 3 pixels is used to reduce
Once the image is adaptively thresholded, a number-plate location task is performed by a spatial scanning procedure which searches for closed contours that have appropriate size and aspect to represent a number-plate. If a number of these closed contours are found, then it is assumed that their position is in the area of the number plate. After the number-plate location is identified, a scaling process is carried out to scale the number-plate area, both vertically and horizontally, to a fixed size. The next step in the processing is to identify the location of number-plate characters. These characters are not expected to be mono-spaced (all letters allocated the same amount of space regardless of letter shape) or proportionately spaced (space allocated according to letter width). A labelling technique to find the location of each character in the number plate image has been developed. The result of this labelling is the coordinates of each character included. A thinning algorithm is then performed to obtain a linedrawing for the characters. Number plate image processing is reviewed in more detail in (Fahmy, 1992, 1993, 1994).
6. CHARACTER RECOGNITION PROCESS Once 170
the
number-plate
lD1age
IS
processed, and the location of each character is identified, the target is to "recognize" the character pixel matrix. The recognition of characters included in a number-plate is different from what is called optical character recognition. The characters can be in any font, and each font may have distinct patterns for many of the characters. Two adjacent characters can be kerned (overlapped without touching each other). Characters can be variously sized, so the computer must recognize not only the shapes of the characters but also the shape at different sizes. The character recognition process is achieved using neural network techniques. These have the potential to recognize the character if deformed, distorted in shape, or shifted in position. In the last stage of this research, the recognition process using back-propagation neural network was successfully performed (Fahmy, 1994). Another neural network method suitable for character recognition is self-organizing neural network. The remainder of the paper corresponds to this form of neural network. 6.1. The Self-Organizing Neural Model
The self-organization neural network model has many applications in the
pattern recognition field. It consists of two layers: an input layer and an output layer. Each neuron in the input layer is connected to each neuron in the output layer by a connection (synapse) with an associated weight. The common way to represent each layer is as a two dimensional array of neurons (see Fig.2). Most often they are arranged in either rectangular or hexagonal arrays. The distinctive property of self-organizing neural network is training without supervision. Input patterns are presented only to the input layer, and the network organizes (trains) itself (Kohonen, 1988). The main accomplishment of the selforganization model is to cluster data into classes, reducing its dimensionality. The training process consists of presenting pattern vectors from the training set to the self-organization network, one at a time. The output neuron which gives the highest indication for the input pattern introduced is called the winning neuron. The winning neuron is selected by making a series of calculations after each pattern presentation. Weight adjustment is made by using a neuron neighbourhood that shrinks over time (see Fig.3) and a learning coefficient 11 that also decreases with time. It takes different values during the iterations; it usually decreases with increasing iterations (time), and it varies with the distance from the winning neuron. When the network has self-organized, and training is complete, neurons that are topologically near each other react similarly to similar input patterns. The main concept of the training is not to tell the network what the "correct" answer is. Simply the patterns are presented to the initialized network, over and over, while varying the neighbourhood and learning parameters, and the network trains itself by adjusting its weights.
Fig.2. Self-organizing neural model 171
The "neighbourhood" is the portion of the output neurons within a specified topological radius of a given winning neuron. The initial size of the neighbourhood must be defined at first. It is a fraction of the output layer size. All neurons in the neighbourhood of the winning neuron have their weights adjusted. The weights inside the neighbourhood should be adapted to strength the winning neuron but the weights outside the neighbourhood should not be changed. An example of topological neighborhood Ne of neuron Uc> showing shrinking with training iteration ni' is shown in Fig.3.
••••••••••• ••••••••••• •••••••••••
•••••••••• •• ••••••• •••• ••
.............. ••••••••••• ••••••••••• •••••••••••" .
dip(t)-
L
[i j (t)-WpCt)]2
(1)
j-I
where d1p(t) is the distance to neuron j for the t th iteration for pattern p. Each weight in the neighbourhood of the winning neuron is then adjusted according to Equation 2. Equations 1 and 2 are calculated for each iteration of each pattern presented to the
self-organization network during training. Iterations continue until the corrections in Equation 2 become acceptably small .
7. CONCLUSION
• • • • • • Q, • • • •
• • • • • • >:• • • • •
11
Since the fonts used in the manufacture of number-plates are widely different, using neural network techniques in the character reading process has the potential of overcoming the different font problem .A self-organizing neural network is designed and trained to recognize the characters embedded in a number-plate image. The characters are english letters A to Z and digits 0 to 9. The network is trained not only to read a definite character shape, but also distorted character shapes. The distorted characters were obtained from a database of nearly a hundred real number-plates, each with many defects. These numberplates included joined characters, characters deformed in shape, and characters shifted along the horizontal and vertical axes. The number of samples per character was around twenty. The network is able to recognize character patterns which it has learned to recognize. Also, it is capable of recogruzmg many of variations of character shape which were not introduced to it during its training.
"1<"2<"3< .. .
WInning
neuron
Nc(nS) Ne(n2) Ne(n1)
Fig.3. Neighbourhood shrinking. The wmnmg neuron is picked by calculating the Euclidean distance between the input pattern vector and the weight vector associated with each output neuron. The output neuron with the minimum Euclidean distance is the winner. The Euclidean distance is the square root of the sum of the squares of the differences between each input vector component and its associated weight vector component. To conserve computing time, square root calculations are not usually done. The resulting distance calculation is defined as 172
A sort of performance metric is being used to compare network performance with specifications. 8. REFERENCES Ayland, N. and Dxzavis, P. (1989). Automatic Vehicle Identification for Vehicle Monitoring. lEE 2nd Int. Con! on Road Traffic Monitoring, lEE, London, 299, pp 152-155. Fahmy, M. M. M. (1992). Toward Low Cost Traffic Data Collection. lEE Toll Colloquium, 2 October 1992. lEE Savoy Place, London WC2R OBL. Fahmy, M. M. M. (1993). Automatic Number Plate Recognition. 26th ISATA Conference, 13-17 September 1993. Achen, Germany. Fahmy, M. M. M. (1994). An Application of Computer Vision to Automatic Vehicle Identification. Proc. of 3rd Conference on Information Technology and its Applications, 10-13 April 1994, Cardiff, Walse. Gonzalez, R. C. and Wintz, P. (1987). Digital Image Processing. 2nd edition, Addison-Wesely Publishing Company. Lotufo, R. A.; Morgan, A.D. and Johnson, A. S.(1990). Automatic NumberPlate Recognition. Faculty of Engineering, University of Bristol, Bristol BS8 1TR. Rumelhart D.E and Mc Clelland J.L. (1986). Parallel Distributed Processing: Exploration in the Microstructure of Cognition. MIT Press, Cambridge. Salter, D.R. (1984). The potential of Automatic Vehicle Identification. lEE Con! on Road Traffic Data Collection, lEE, London, 242, pp 79-82.
173
COMPUTER VISION APPLICATION TO AUTOMATIC NUMBER PLATE RECOGNITION Maged M. M. FAHMY TORG, University of Newcastle Upon Tyne, NEI 7RU, UK. Abstract. Automatic number plate recognition, using image processing techniques, is indicating new ways of capturing traffic data. Such systems could play an important role in automatic toll collection. The requirement is to develop a practical system able to work under a variety of number plate geometries, sizes, and different fonts. The target of this research is to use neural network techniques to recognize characters in a number plate, so as to achieve an acceptable level of accuracy. In order to depend on such systems to replace the conventional traffic data collection methods, their accuracy level must match conventional methods. An advantage of using automatic systems is that it does not depend on human factors and it can work under difficult circumstances.
Key Words. Image processing; image thresholding; feature analysis; connected component; skeletonization; neural networks; self organizing model; character recognition.
1. INTRODUCTION
recognition.
Current research area in traffic toll collection tends to use automatic, i.e. nonstop, methods. These methods have the advantages of not disrupting the traffic flow. With the appearance of automatic vehicle identification (AVI) technology allowing non-stop electronic toll collection, it is important that a more efficient method of enforcement is found to register the non-paying vehicles. Automatic number-plate recognition technology is seen as a promising method of A VI without disrupting the traffic flow. At the toll plazas, license-plate recognition can identify vehicles that pass without paying their toll and automatically alert the authorities. Computer vision for automatic number-plate recognition has three parts; number-plate image capturing, image analysis and character
169
2. NUMBER-PLATE IMAGE CAPTURING The number-plate image capturing process consists of a configuration where a video camera is mounted at a suitable angle, height and distance so as to capture a scene of the front ,or rear, part of the incoming vehicle. The video camera should have high shutter speed to cope with speedy incoming vehicles. The captured image is being grabbed using an image grabber card mounted in a host computer (see Fig. I). The grabbed image is then stored in its grey-level form within the image grabber or the dynamic memory of the computer to be processed later.
number plate image noise (Gonzalez and Wintz, 1987). The automatic numberplate reading system executes an adaptive thresholding algorithm on the entire digitized scene. This process helps properly segment the vehicle numberplate and its characters from the rest of the scene.
Tr.n •• t light
5. NUMBER-PLATE IMAGE PROCESSING
Fig.I. Number-plate image capturing.
3. THE NATURE OF ACQUIRED NUMBER-PLATE IMAGES The captured number-plate image is expected to have noise. There are many sources of the noise. Electrical devices used to obtain and store the image all produce some form of electrical noise. The weather conditions under which the image is captured may add different forms of noise to the image. Dirt and dust are other sources of noise. It is also noted that image processing techniques are affected in some way by changes in overall light intensity. Consequently, changes in weather conditions could have a marked effect on the appearance of number plate characters within the scene, and so, affect the performance of the recogniser. These noises require preprocessing of the number-plate image before recognition.
4. NUMBER-PLATE IMAGE PREPROCESSING Noise can be reduced in a digital image by the application of a filter. A filter is a function which operates on a small neighbourhood of pixels. A convolution mask of 3 x 3 pixels is used to reduce
Once the image is adaptively thresholded, a number-plate location task is performed by a spatial scanning procedure which searches for closed contours that have appropriate size and aspect to represent a number-plate. If a number of these closed contours are found, then it is assumed that their position is in the area of the number plate. After the number-plate location is identified, a scaling process is carried out to scale the number-plate area, both vertically and horizontally, to a fixed size. The next step in the processing is to identify the location of number-plate characters. These characters are not expected to be mono-spaced (all letters allocated the same amount of space regardless of letter shape) or proportionately spaced (space allocated according to letter width). A labelling technique to find the location of each character in the number plate image has been developed. The result of this labelling is the coordinates of each character included. A thinning algorithm is then performed to obtain a linedrawing for the characters. Number plate image processing is reviewed in more detail in (Fahmy, 1992, 1993, 1994).
6. CHARACTER RECOGNITION PROCESS Once 170
the
number-plate
lD1age
IS
processed, and the location of each character is identified, the target is to "recognize" the character pixel matrix. The recognition of characters included in a number-plate is different from what is called optical character recognition. The characters can be in any font, and each font may have distinct patterns for many of the characters. Two adjacent characters can be kerned (overlapped without touching each other). Characters can be variously sized, so the computer must recognize not only the shapes of the characters but also the shape at different sizes. The character recognition process is achieved using neural network techniques. These have the potential to recognize the character if deformed, distorted in shape, or shifted in position. In the last stage of this research, the recognition process using back-propagation neural network was successfully performed (Fahmy, 1994). Another neural network method suitable for character recognition is self-organizing neural network. The remainder of the paper corresponds to this form of neural network. 6.1. The Self-Organizing Neural Model
The self-organization neural network model has many applications in the
pattern recognition field. It consists of two layers: an input layer and an output layer. Each neuron in the input layer is connected to each neuron in the output layer by a connection (synapse) with an associated weight. The common way to represent each layer is as a two dimensional array of neurons (see Fig.2). Most often they are arranged in either rectangular or hexagonal arrays. The distinctive property of self-organizing neural network is training without supervision. Input patterns are presented only to the input layer, and the network organizes (trains) itself (Kohonen, 1988). The main accomplishment of the selforganization model is to cluster data into classes, reducing its dimensionality. The training process consists of presenting pattern vectors from the training set to the self-organization network, one at a time. The output neuron which gives the highest indication for the input pattern introduced is called the winning neuron. The winning neuron is selected by making a series of calculations after each pattern presentation. Weight adjustment is made by using a neuron neighbourhood that shrinks over time (see Fig.3) and a learning coefficient 11 that also decreases with time. It takes different values during the iterations; it usually decreases with increasing iterations (time), and it varies with the distance from the winning neuron. When the network has self-organized, and training is complete, neurons that are topologically near each other react similarly to similar input patterns. The main concept of the training is not to tell the network what the "correct" answer is. Simply the patterns are presented to the initialized network, over and over, while varying the neighbourhood and learning parameters, and the network trains itself by adjusting its weights.
Fig.2. Self-organizing neural model 171
The "neighbourhood" is the portion of the output neurons within a specified topological radius of a given winning neuron. The initial size of the neighbourhood must be defined at first. It is a fraction of the output layer size. All neurons in the neighbourhood of the winning neuron have their weights adjusted. The weights inside the neighbourhood should be adapted to strength the winning neuron but the weights outside the neighbourhood should not be changed. An example of topological neighborhood Ne of neuron Uc> showing shrinking with training iteration ni' is shown in Fig.3.
••••••••••• ••••••••••• •••••••••••
•••••••••• •• ••••••• •••• ••
.............. ••••••••••• ••••••••••• •••••••••••" .
dip(t)-
L
[i j (t)-WpCt)]2
(1)
j-I
where d1p(t) is the distance to neuron j for the t th iteration for pattern p. Each weight in the neighbourhood of the winning neuron is then adjusted according to Equation 2. Equations 1 and 2 are calculated for each iteration of each pattern presented to the
self-organization network during training. Iterations continue until the corrections in Equation 2 become acceptably small .
7. CONCLUSION
• • • • • • Q, • • • •
• • • • • • >:• • • • •
11
Since the fonts used in the manufacture of number-plates are widely different, using neural network techniques in the character reading process has the potential of overcoming the different font problem .A self-organizing neural network is designed and trained to recognize the characters embedded in a number-plate image. The characters are english letters A to Z and digits 0 to 9. The network is trained not only to read a definite character shape, but also distorted character shapes. The distorted characters were obtained from a database of nearly a hundred real number-plates, each with many defects. These numberplates included joined characters, characters deformed in shape, and characters shifted along the horizontal and vertical axes. The number of samples per character was around twenty. The network is able to recognize character patterns which it has learned to recognize. Also, it is capable of recogruzmg many of variations of character shape which were not introduced to it during its training.
"1<"2<"3< .. .
WInning
neuron
Nc(nS) Ne(n2) Ne(n1)
Fig.3. Neighbourhood shrinking. The wmnmg neuron is picked by calculating the Euclidean distance between the input pattern vector and the weight vector associated with each output neuron. The output neuron with the minimum Euclidean distance is the winner. The Euclidean distance is the square root of the sum of the squares of the differences between each input vector component and its associated weight vector component. To conserve computing time, square root calculations are not usually done. The resulting distance calculation is defined as 172
A sort of performance metric is being used to compare network performance with specifications. 8. REFERENCES Ayland, N. and Dxzavis, P. (1989). Automatic Vehicle Identification for Vehicle Monitoring. lEE 2nd Int. Con! on Road Traffic Monitoring, lEE, London, 299, pp 152-155. Fahmy, M. M. M. (1992). Toward Low Cost Traffic Data Collection. lEE Toll Colloquium, 2 October 1992. lEE Savoy Place, London WC2R OBL. Fahmy, M. M. M. (1993). Automatic Number Plate Recognition. 26th ISATA Conference, 13-17 September 1993. Achen, Germany. Fahmy, M. M. M. (1994). An Application of Computer Vision to Automatic Vehicle Identification. Proc. of 3rd Conference on Information Technology and its Applications, 10-13 April 1994, Cardiff, Walse. Gonzalez, R. C. and Wintz, P. (1987). Digital Image Processing. 2nd edition, Addison-Wesely Publishing Company. Lotufo, R. A.; Morgan, A.D. and Johnson, A. S.(1990). Automatic NumberPlate Recognition. Faculty of Engineering, University of Bristol, Bristol BS8 1TR. Rumelhart D.E and Mc Clelland J.L. (1986). Parallel Distributed Processing: Exploration in the Microstructure of Cognition. MIT Press, Cambridge. Salter, D.R. (1984). The potential of Automatic Vehicle Identification. lEE Con! on Road Traffic Data Collection, lEE, London, 242, pp 79-82.
173