- Email: [email protected]
Endocardial boundary detection using a neural network
Pattern Reco~4nition,Vol. 26, No. 7, pp. 1057 1068. 1993
0031 3203~93 $6.00+.00 Pergamon Press Lid ;~ 1993 Pattern Recognition Society
Printed in Great Britain
ENDOCARDIAL B O U N D A R Y DETECTION USING A NEURAL NETWORK CHING-TSORNGTSAI,+ YUNG-NIENSUN,~§ PAu-CHoO CHUNG~ and J1ANN-SHu LEEr + Department of Electrical Engineering, National Cheng Kung University, Tainan, Taiwan, 70101, RO.C. Institute of Information Engineering, National Cheng Kung University, Tainan, Taiwan 70101, R.O.C. (Received 20 May 1992; in revised.]brm 25 November 1992; receivedfor publication 5 January 1993) Abstract--Echocardiography has been widely used as a real-time non-invasive clinical tool to diagnose cardiac functions. Due to the poor quality and inherent ambiguity in echocardiograms, it is difficultto detect the myocardial boundaries of the left ventricle.Many existingmethods are semi-automatic and detect cardial boundaries by serial computation which is too slow to be practical in real applications. In this paper, a new method for detecting the endocardial boundary by using a Hopfield neural network is proposed. Taking advantage of parallel computation and energy convergencecapability in the Hopfield network, this method is faster and more stable for the detection of the endocardial border. Moreover, neither manual operations nor a priori assumptions are needed in this method. Experiments on several LV echocardiograms and clinical validation have shown the effectivenessof our method in these patient studies. Neural network Hopfieldnetwork Automaticendocardial border segmentation Parallel border detection Ultrasonic image
I. INTRODUCTION Echocardiography has been used as a major clinical tool to diagnose cardial function for many years. Because echocardiography is noninvasive and secure, it is readily accepted by specialists and patients. Therefore, it has been widely used as a real-time tool to evaluate the left ventricular function. In order to obtain indices of cardiac performance such as ejection fraction and ventricular volume, it is necessary to obtain the contour lines corresponding to the endocardial wall of the heart chamber/1) Manual definition of cardial contour lines is time consuming, tedious, and substantially subjective, therefore, it limits the clinical usefulness of the quantitative analysis. Computer aided processes for defining cardial boundary have been used to prevent the problems caused by manual tracing. Unfortunately, the echocardiogram usually has low signal-to-noise ratio (SNR), low spatial resolution, and occasional loss of echo signal. The echocardiogram also shows some connective tissues such as the papillary muscle or the mitral valve. Hence, the detection of cardial borders has been one of the most difficult image processing problems/z~ Recently, several studies have been devoted to the detection of the heart wall contour from twodimensional (2D) echocardiograms. In general, they can be classified into three main categories, which are edge-based, 13'4) region-based,15-12) and knowledgebased (rule-based) methods. I13 16) The edge-based methods ~3'4~apply an edge operator to the image and select a proper threshold to determine
§ Author to whom all correspondence should be addressed.
Echocardiogram
the edge points. However, the requirement of a good threshold is one of the major drawbacks of this method. Without a suitable threshold, this method could cause spurious edge points. Hence, this approach is not effective in finding the boundary of a complex echocardiogram due to its high noise and signal dropout. ~5) The region-based methods ~5-12) search the boundary points based on the estimated reference center or an initial contour of the heart chamber. Some previous works have shown that these methods are promising, but most of them are semi-automatic, because either a priori assumptions or human interventions are needed. In these methods, some additional conditions such as the ventricular center, starting contour tracing point, radially searching limits, and linear or average rate of wall motion from end-diastole to end-systole, have to be imposed in the detection procedures. However, these conditions are usually hard to define a priori, thus the capability of these methods in practical applications is limited. The knowledge-based (rule-based) methods 113 16) integrate the knowledge of heart anatomy and the characteristics of 2D echocardiograms to construct a knowledge rule which is aimed to detect the cardial borders. These methods make the detection of cardial borders more reliable and can be implemented in AI machines. However, the slow response time of present AI systems is an inevitable limitation for real-time applications. From the reasons mentioned above, it is desirable to develop a new approach that can achieve both fully automatic segmentation and fast computation for cardial border detection. In this paper, we propose a system for extracting the endocardial borders from 2D echocardiograms based on a Hopfield network.
1057
1058
C.-T. TSAIet al.
The system consists of four modules: pyramid-linked enhancement, initial endocardial border estimation, endocardial border detection by a Hopfield network, and contour interpolation. (I 7) In the implementation of Hopfield neural networks, ~18.~9) the finding ofendocardial boundaries is regarded as a constraint satisfaction problem. The constraints consist of the continuity, curvature, and edge magnitudes on the endocardial border. An initial contour, which is placed near the ventricular boundary, is first transformed into an initial state of the Hopfield network. After undergoing a parallel and dynamic process of neuron updating, a final stabilized state of each neuron satisfying the above given endocardial constraints represents the accurate endocardial boundary. In other words, a 2D binary Hopfield network is implemented to detect the endocardial boundary. The endocardial border detection is based on the network energy minimization. A condition for assuring the network convergence is derived. If this condition is not satisfied, a deterministic decision rule for neuron updating is used. Experimental studies have been performed on several serial LV echocardiograms obtained from the University Hospital at National Cheng Kung University. We utilize the detected contour on the present frame to detect the border on the next frame throughout the whole cardial cycle. Good results have been obtained for clinical validation. Compared with conventional endocardial border detection methods, our approach provides a fully automatic segmentation of the left ventricle and a solution more suitable for parallel implementation. In the next section, an overview of the proposed system will be given. Then, the Hopfield network for endocardial boundary detection will be presented in Section 3. Section 4 addresses the results of experimental studies with computer simulation for clinical validation. Finally, the conclusions are given in Section 5. The mathematical derivations for the Hopfield network are given in the appendices.
Linked-pyramid enhancement ]
Initial contour estimation i
Endocardial borders detection [ based on Hopfield N.No 1 Contour interpolation
]
Endocardial contour output Fig. 1. The system diagram.
2.1. Image enhancement Poor image quality causes difficulties in echo image processing. It is also the major obstacle in the segmentation of myocardial borders. In our proposed method, a node-linking pyramid ~2~ 23~ is applied to improve the image quality. Like other pyramids, there is a layered arrangement of square arrays in which each array is half in length and width as the next array below it. A father-son relationship is defined between nodes in adjacent layers and established according to the nearest father selection rule. t21) Image enhancement is implemented as a process which selects a single legitimate father for each node from its candidate fathers in an iterative manner. It can smooth the intensity fluctuations by using the link relationship between each level in the pyramid. The main advantage of applying a node-linking pyramid is that it can preserve the endocardial boundary information and reduce the irrelevant local perturbation as well. Figure 2(b) shows the result after applying this operation to the image in Fig. 2(a).
2. A SYSTEM FOR THE D E T E C T I O N OF ENDOCARDIAI, BOUNDARY
2.2. Initial endocardial boundary estimation for the first image
An automatic system using a Hopfield network for detecting the left ventricular boundaries from sequences of 2D echocardiograms is described in this section. Our system consists of three major modules: (1) image enhancement for the whole sequence of images, (2) initial contour model estimation for the first image, and (3) accurate endocardial boundary detection using a Hopfield network. After the boundary pixels are detected, the endocardial border can be computed by interpolation. The Catnull Rom spline {2°) is used in interpolating these endocardial boundary points. Finally, we utilize the determined edge contour in the present frame to detect the ventricular border for the subsequent frame throughout the whole cardiac cycle. The flow diagram of the system is shown in Fig. 1.
Morphological techniques" 1.24) are applied to estimate the initial endocardial boundary for the first image. The process consists of a grey-scale opening with a flat-topped hexagonal structuring element which has a radius of 11 pixels (H11). Hills, which cannot accommodate the hexagon are removed from the image surface C during the filtering operation. Thus, a background image with slowly varying illumination is obtained after applying these morphological operations (see Fig. 2(c)). The resulting image D can be expressed as
D = C-((CGH
Jl)~Hll).
(1)
To reduce the impact of echo dropout, a closing operation is used to fill the holes generated by echo dropout
Segmentation of echocardiograms
1059
Fig. 2(a~. An original echocardiogram.
Fig. 2(c). The resulting image after opening and closing.
Fig. 2(b). The resulting image after pyramid enhancement.
Fig. 2(d). The initial contour after threshold and sobel operation.
on image D: that is
D = ( D G H 9 ) Q H 9.
(2)
After the process, a binarization operation is used to obtain the ventricular area and a sobel operation is utilized to produce the initial endocardial boundary (see Fig. 2(d)). 2.3. Endocardial boundary detection The search for endocardial boundaries is one of the most difficult tasks in 2D echo image processing. In this subsection, we regard this task as a problem of constraints satisfaction. A modified Hopfield neural network is used to solve this constraint satisfaction problem. Initially, the rough boundary available from the previous morphological segmentation process is used as the initial contour. Then, uniformly sampled points on the initial contour are transformed into the states of neurons and their interconnection values. After undergoing a parallel and dynamic process of the neuron updating, the stabilized state of each neuron in the Hopfield network approaches a resulting contour point satisfying the given endocardial constraints. The
details of this new cardial boundary detection algorithm based on the Hopfield network will be given in the following section.
3. ENDOCARDIAL BORDER DETECTION USING A HOPF1ELD NETWORK
The application ofa Hopfield network for the detection of endocardial border is presented in this section. In this method, the location of the rough boundary available after the morphological segmentation process is taken as an initial contour. Under the assumption that the actual endocardial boundary yields a higher sum of gradient and contrast magnitude than any other trajectories around the boundary, the output of this method results in a new curve which is chosen as an accurate estimate of the endocardial boundary. Continuity and curvature of the endocardial boundary are also considered in our method. A cost function is defined to be the sum of gradient, contrast, continuity, and curvature of the ventricular boundary. The proposed method locates the cardial boundary point by searching in the neighborhood of the sampled point
C.-T, TSA]et al.
1060
on the initial border. The Hopfield network approaches the accurate cardial boundary by minimizing the cost function at each neural in a parallel manner (see Fig. 3). The cost function in our method is similar to the one used in the active contour model. (25) Therefore, the endocardial border detection is regarded as a minimization of the following equation:
border is allowed to stretch or bend. The constant ;, reflects its relative importance in defining the cardial boundary. Let X(Vi) and Y(Vi) denote the x-coordinate and the y-coordinate of the point Vi, respectively. Approximating the derivatives with finite differences, equation (3) can be further expanded as N
E = ~. - ~ [ ( X ( V i ) - x ( v ~
N
E= ~ -~rrv;ll"-l.~,llV,/ij2-.;etvi)
(3)
i--I
-fl[(X(V,
where V~and F'i' are the first- and second-order derivatives of the ith chosen ventricular boundary point Vi, respectively; N the size of the contour, and P(Vi)= llV/(Vi) II, in which VI(Vi)is the image intensity gradient and contrast magnitude of the point I/u The values of c~and/:t determine the extent to which the endocardial
,)F
+(Y(V~)- Y(v~_,)F]
i=1
,) - 2 X ( V ~ ) + X(Vi+ ,))2
+(Y(Vi_,)-2Y(V~)+ Y(VI+,)) 2] - 7P(Vi).
In this method, the rough boundary obtained from morphological segmentation is first uniformly sampled into N points. Each sampled point is associated with its 2D neighboring points. These
L2
-
-
1 neighboring
(a) ~o•. • •
\ 0@00 0 000
" 0 0
0 0
0000
2
rhood
/v,:. \
C0 0 2 ,..-10 0
0
°o
•
(b)
O0
0000
• 0 0
0©00
0
20 0
0©00
0
Q
0
•
10 1
0 2
0@00 •
•
•
i-1
i i+1
0 i+2
(4)
•
•
:
N
O
a firing neuron
: a resting neuron
Fig. 3. The relationship between the model of endocardial border and a Hopfield neural network. (a) The initial border points and their corresponding neighborhood. (b) The Hopfield network with initial states of neurons corresponds to the initial cardial boundary.
Segmentation of echocardiograms points and the sampled point itself are orderly indexed from 1 to L2. Figure 3(a) also gives an example showing the positions and indices of the sampled point 1/i and its L2 - 1 neighboring points. These L 2 points centered by the ith sampled point will be mentioned as points (i, k)s with I _< k < L2, subsequently in this paper. Thus, as i varies from 1 to N, the points on the endocardial border are indicated by 2D indices (i, k)s, where 1 _< i £ N and 1 _< k _< L 2. The N sampled points are taken as the initial input boundary to the Hopfield network (see Fig. 3). Through network evolutions, the optimal (or suboptimal) cardial boundary points are chosen from the initial boundary points and their neighborhood. The choice is imposed such that one and only one point is selected from points (i, 1) to (i, L2), for each i. Thus, a 2D binary Hopfield network is used to detect the endocardial borders by minimizing the total energy of the network. The network is constructed as an N* Lz array of neurons with neuron (i,k) representing the point (i, k) of the endocardial border. The state of each neuron (firing or resting) in the network represents the possible choice of the corresponding point as an endocardial boundary point. Let V~.k denote the binary state of the (i, k)th neuron (1 for firing and 0 for resting) and T~,k;ja the interconnection strength between neuron (i,k) and neuron (j,I). A neuron (i,k) in this network receives weighted inputs, ~ T,. k:ja V;a,from all the neuj.I
tons and a bias input, I~, k, from outside. Therefore, the total input to neuron (i, k) is computed as N
L2
(5)
~i.~ = Y. Y~ r,,j.,vj., + t,.~. j
ll=l
-}-
~
~
~
(7)
~ Ti.k:j,lUi.klPj.I -
2i=1
k=l
j=l
/=1
L~ E
i=1
k=l
k=l
-2
k=l
~
Yi.kUi.k-~
k=l
--7'
gi.kl)i.k k
E
Yi+l,kl~i+l.k
k=l
+ 0"
Ii.kUi,k"
k=l
(8) The network achieves a stable state when the energy of the Lyapunov function is minimized. Let X~,k and Y~,k be the x-coordinate and the y-coordinate of the point (i, k), respectively, and Yi.k the gradient strength of point (i, k). According to equation (4), the endocardial border detection can be characterized as the minimization of the following energy function: - ~ i=1
Xi,kli.k -k
1
Xi k=l
l,kUi
1,k
E
tli.k
.
(9)
k=l
The first and the second terms in equation (9) are the first- and second-order derivative terms in equation (4). The third term represents the gradient strengths. The fourth term is included for enforcing the uniqueness constraint where one and only one neuron is active in each column; that is, only one point is selected from points (i, 1) to (i, L2) for each i. This uniqueness constraint is also implied in the representations of the other terms in equation (9), where ~ Xi.kVi.k is used to k
represent {he x-coordinate of the selected boundary point, EYi.kt!i,k t h e y-coordinate, a n d Egi.kUi.k the k
k
gradient. Equation (9) can be simplified and fitted into the Lyapunov function, i.e. equation (8), of a Hopfield network. From there, the values of interconnection Ti,k;j. l and bias input li. k c a n be obtained, as shown in Appendix A, as [(4~t + 12fl)~i. j - (2~ + 8fl)ai+ 1.j - ( 2 ~ + 8fl)fi~
21:~6~. :,j]
' 4 + 2fl8~ + 2.j +
* [Xi,kXj.l -{- Yi.kYja] --
[20.t~i.j]
(1 O) (l l)
respectively, where (~i.jis the Kronecker delta function defined as 6i.j = 1 if i = j and 0 otherwise. From equation (10), one can see that the network is not fully interconnected. Neurons in a specific column receive inputs from neurons in the same, the previous two, and the posterior two columns. Furthermore, the interconnection strength between neuron (i, k) and neuron (j, l) depends only on the coordinates of the point (i, k) and point (j, l) along with the assigned constants ~t,,q, and 0". From equation (ll), the bias input to neuron (i, k) is determined by the gradient magnitude Y~.R,which is obtained from the original image. It should be noted that the interconnection strength Ti,k:;a depends on the origin of the coordinate system. This is because the continuity and curvature terms in equation (4) are decomposed into arithmetic expressions with the coordinates x~. k and Yi,k.The same situations also occur for example, E X i . k l ? i . k , EYi.kUi.k , and k
E =
1 --
1
li.k = ~'.('Ji,k -}- 20"
The Lyapunov function of a 2D Hopfield network is given by 1~
,.~-- 2 y . Xi.kt'i, k
1
and
where ,q(x) is the hard-limiting function defined as
E=-
x,_ ,.kVi k
(6)
if x < 0"
~'i- l.k i l.k k=l
-fl
Ti,k:j. I =
y(x) =
Yi,kUi,k k=l
During network evolution, a randomly selected neuron, say neuron (i, k), is updated according to Vi.k = g(Ui.k)
1061
Xi.kV,k -- ~ x i - l.kVi 1.k k
k
in equation (9). How-
1062
C.-T. TSAIet al.
ever, if the uniqueness constraint that exactly one neuron fires in each column is satisfied, the energy has the same value wherever the origin of the coordinate system is located. This uniqueness constraint is imposed in the fourth term of equation (9). In the implementation of the Hopfield network, multiple or no neurons firing in each column may occur. Therefore, a normalization of these interconnection strengths is needed to depreciate the influence of this situation. One possible normalization process is described as follows. Let C,,ax = m a x ( X l , l , X 1 , 2 . . . . . XN,L2, Y l , 1 , . . . , y s . L 2 ) , Cmin = min(xl,l,xl,2 . . . . . X N , L ~ , Y l , 1 . . . . . YN,L2), gm,x = max(g1.1, gl,2, • •., gNX2), and gmi, = min (g1,1, gl,2,-.., gN.L2), where max (.) and min (.) are the maximum and the minimum functions, respectively. The normalized x}, k and Y'i,k are expressed as
X'i,k=Xi.k--Cmin,
l
and l < k < L
2
(12)
l
and l < k < L 2
(13)
and
tion (9) in a parallel manner. The algorithm for endocardial border detection is summarized as follows. Input: A set of neurons arranged in a 2D array with
initial values set to reflect the initial boundary contour. Output: A set of stabilized neuron states representing the accurate endocardial boundary points in the sense that the cost function of equation (9) is minimized. Method:
(1) Transform those initial sampled points and their corresponding neighborhood into the neural network model. (2) Set the initial states of the neurons according to the initial contour. (3) Randomly pick up a node from the network for updating, and calculate its total input: N
Y'i,k=Y~,k--Cm~.,
u,.~ = E j=l
respectively. Obviously, x'g.k and Yl,k are independent of the origin. The weighting values ~, fl, and ? reflect the relative importance of the continuity, curvature, and gradient terms, respectively, and the sum of these three values is equal to 1, given by ~+fl+?=
I.
(14)
Also, the values of the coordinates and the gradients are normalized into the closed interval [0, 1] to unify their relative importance. They are expressed as
x','~ -
X'i, k
(15)
Cma x - - Cmi n
Y~'.k --
Y'i,k
(16)
Cma x - - C m i n
and g~,k Y~',k= " gmax -- groin
(17)
where gl.k = g i , k - gmi.- After the normalization of coordinate and gradient values, the interconnection strengths, T~,k:),l,and the bias input, Ii. k, are modified as
L2
E r , . ~ j . , v , + I,.~. /=1
(4) Update the state of this selected neuron according to equation (6). (5) Repeat steps 3 and 4 for a certain number of times and count the number of neurons having their states changed. If the number is less than a given value, then go to step 6. Otherwise, go to step 3 and repeat the process. (6) Output the final states of neurons which represent the actual endocardial boundary points. The evolution of the network is characterized as a dynamic updating process which achieves a stable state when the energy function of equation (9) is minimized. In Appendix B, the Hopfield network used for our application is proven to converge to a stable state subject to the condition that (4ct + 12fl)((xT,k)2 + (y~'k) 2) -- 2a > 0. When the given constants ~t,fl, and tr do not satisfy the condition, a deterministic decision rule is utilized. The deterministic rule is to take a new state of neuron (i, k) only when the energy change due to state change is less than zero. If the energy change is greater than zero, the neuron remains unaffected. 4. S I M U L A T I O N R E S U L T S A N D C L I N I C A L S T U D I E S
Ti,k;j, l = [(4ct + 12fl)(~i, j - (2ct + 8fl)6i+ ~,j - (2c~ + 8fl)fi~_ 1.j + 2fl6~ + 2,j -~- 2fl6i_
* [x}',kX~,, + YT,kY~,,] -- [2Crf,,j]
2,j] (18)
and li,k = Ygi',k
+ 2O"
(19)
respectively. As mentioned above, the Hopfield network searches for the best-fit endocardial border output through the process of dynamic adjustments from a rough initial border obtained from the preprocessing step. The endocardial border tries to satisfy all constraints in equa-
Ultrasonic images were acquired from an ultrasonic system (77020A Hewlett Packard) at the University Hospital of N C K U and stored by a video cassette recorder. The image data was then captured by a near real-time visual interface board, AFG, and digitized into 512.512 pixels with 256 grey-levels. First, the preprocessing processes, including the pyramid-based enhancement, the morphological operations, and the sobel operation, are implemented on the AFG image board. Figure 2 shows the intermediate results after preprocessing which include the pyramidbased enhancement image, the result of morphological segmentation, and the initial model of the ventricular contour.
1063
Fig. 4(a). The initial border superimposed on Fig. 2(a).
Fig. 4(c). The manual tracking border by a physician.
Fig. 4(b). The resulting endocardial border superimposed on Fig. 2(a).
The proposed network was simulated on an intel486 personal computer. From an initial contour, boundary points are sampled with equal intervals of 25 pixels. There are sually 25-35 sampled points in a closed endocardial border. The neighborhood is defined around each sampled point by a 15,15 window. The values ~, fl, 7, and a were determined experimentally. They are ~ = 0 . 1 , fl=0.3, 1,=0.6, and a = 2 . 0 . The simulated Hopfield network takes nearly 34,000 loops corresponding to approximately 30 C P U min to converge. Here, one loop updates the state of each neuron once. The computation is slow because the simulation is pro-
Fig. 5. The detected endocardial borders of a whole cardial cycle.
1064
C.-T. Tsm et al. pixelnumber
i0000
7500
5000
2500
1
3
5
7
9
11
13
15
Fig. 6. The graph of ventricular area with respect to frame number: ........ , by computer; - physician. cessed sequentially by a micro-computer. However, the Hopfield network can be implemented with a parallel architecture and the endocardial border detection has a great potential to be accomplished by real-time computation. In the physical implementation, after the Hopfield network is stabilized, it may encounter problems such as multiple or no active neurons in a column. In this situation, a decision rule is used to determine the best point. The decision rule takes the average of those active neurons when multiple neurons are active, or the original neuron state when no neuron is active as the final endocardial point. Figure 4(a) shows the initial contour superimposed on the original image in Fig. 2(a). The resulting endocardial border by the proposed Hopfieid network is shown in Fig. 4(b). For comparison, the ventricular border traced by an experienced physician is shown in Fig. 4(c). Results indicate that the boundary obtained by our method matches that obtained by an experienced physician closely. Five patients' data, each containing ! 6 frames from a cardiac cycle, have been processed. Validation of the proposed method is carried out throughout the whole cardiac cycle (1/30sper frame). Figure 5 shows the segmentation results from the sequence of the second patient's echo images NM01-NM16. The resulting area curve calculated from the segmentation results is plotted in Fig. 6. The area curve obtained by a physician is also plotted in Fig. 6 for comparison. The experimental studies have shown that the results obtained by the two methods are quite close to each other. It is also very robust when the image quality is good.
5. CONCLUSIONS
Due to the physical limitations of ultrasonic imaging, it is difficult to develop a parallel and fully automatic myocardial border detection system. Traditional methods for myocardial border analysis are not intelligent enough to handle the task properly. In this
, by the
paper, a Hopfield neural network is designed to achieve these impelling purposes. This method is superior to other methods in three ways. Firstly, the neural approach provides a parallel implementation for the detection of endocardial boundaries in real-time applications. Secondly, it is automatic. Thirdly, it is capable of handling the inherent problems of low contrast and high dropout in echocardiograms. The three advantages make our system more desirable for the detection of the endocardial border. The well-detected endocardial borders from several cardial cycles provide good clinical validation for our system. Acknowledgement--This research work was supported by a grant from the National ScienceCouncil, R.O.C., under contract NSC-81-0420-E-006-007.
REFERENCES
1. L. W. Eaton, W. L. Maughan, A. A. Shoukas and J. L. Weiss, Accurate volume determination in the isolated ejecting canine left ventricle by two-dimensional echocardiography, Circulation 60, 320-326 (1979). 2. A. Herment, J. P. Guolielmi, P. Dumee, P. Peronneau and P. Delouche, Limitations of ultrasound imaging and image restoration; Ultrasonics 25, 267-273 (1987). 3. D.J. Skorton, C. A. McMary, J. S. Child, F. C. Newton and P.M. Shahn, Digital image processing of twodimensional echocardiograms: identification of the endocardium, Am. J. Cardiol. 48, 479-486 (1981). 4. W. Zwehl, R. Levy, E. Garcia, R. V. Haendchen,W. Child, R. Cordav, S. Meerbau and E. Cordav, Validation of a computerized edge detection algorithm for quantitative two-dimensional echocardiography: comparison with anatomic data, Am. J. Cardiol. 53, 1380-1387 (1984). 5. C. H. Chu, E. J. Delp and A. J. Buda, Detecting left ventricular endocardial and epicardial boundaries by digital two-dimensional echocardiography, IEEE Trans. Med. Imag. MI-7, 81-90 (1988). 6. P. R. Detmer, G. Bashein and R. W. Martin, Matched filter identificationofleft-ventricular endocardial borders in transesophageal echocardiograms, I EEE Trans. Med. lmag. 9(4), 396-404 (1990). 7. D. Adam, O. Hareuveni and S. Sideman, Semiautomated border tracking of cine echocardiographic ventricular imag¢s, IEEE Trans. Med. Imag. MI-6, 266-271 (1987).
Segmentation of echocardiograms
8. L. Zhang and E. A. Geiser, An effective algorithm for extracting serial endocardial borders from 2-dimensional echocardiograms, IEEE Trans. Biomed. Engng BME31(6), 441-446 (1984). 9. L. Oliver, E. Geiser, L. Zhang, J. Gardin, A. Parisi, N. Reichek, A. Weyman, J. Werner, R. Kerber and R. Conti, Evaluation of an automated border detection algorithm in 2-D echo images of varying quality: observer versus computer variability, J. Am. Coll. Cardiol. 3(2), 506-513 (1984). 10. S. Tamura, S. Nakano, M. Matsumoto, T. Shimazu, M.T. Matsuyama and P. Hanrath, Three-dimensional reconstruction of echocardiograms based on orthogonal sections, Pattern Recognition 18, 115-124 (1985). 11. J. W. Klingler, C. L. Vaughan, T. D. Fraker and L. T. Andrews, Segmentation of echocardiographic images using mathematical morphology, IEEE Trans. Biomed. Engng BME-35, 925-934 (1988). 12. W.S. Chou, C. M. Wu, Y.C. Chen and K.S. Hsieh, Detecting myocardial boundaries of left ventricle from a single frame 2DE image, Pattern Recognition 23, 799-806 (1990). 13. C. Y. Han, K. N. Lin, W. G. Wee, R. M. Mintz and D. T. Porembka, Knowledge-based image analysis for automated boundary extraction of transesophageal echocardiographic left-ventricular images, IEEE Trans. Med. Imag. 10, 602 610(1991). 14. J. Feng, W. C. Lin and C. T. Chen, Epicardial boundary detection using fuzzy reasoning, IEEE Trans. Med. Imag. 10(2), 187-199 (1991). 15. N. Friedland and D. Adam, Automatic ventricular cavity boundary detection from sequential ultrasound images using simulated annealing, IEEE Trans. Biomed. Engng 8, 344-353 (1989). 16. S. Tamura, K. Yata, M. Matsumot, T. Matsuyama, T.
17. 18. 19. 20. 21. 22.
23. 24. 25. 26. 27.
1065
Shimazu and M. Inoue, Plan-based boundary extraction and 3-D reconstruction for orthogonal 2-D echocardiography, Pattern Recognition 20, 155-162 (1987). C. T. Tsai and Y. N. Sun, Segmentation of echocardiograms using a Hopfield network, Int. Conf. of Euromicro, Paris, pp. 791-797, September (1992). J.J. Hopfield and D. W. Tank, Neural computation of decisions in optimization problems, Biol. Cybern. 52, 141-152 (1985). J. J. Hopfield, Neural networks and physical systems with emergent collective computational abilities, Pro('. Natn. Acad. Sci. USA 79, 2554-2558 (1982). J. Foley, A. V. Dam, S. Feiner and J. Hughes, Computer Graphics Principles and Practice, 2nd Edn. Addison-Wesley, Reading, Massachusetts (1990). R.A. Rosenfeld, Multiresolution Image Processing and Analysis. Springer, Berlin (1984). P.J. Burt, T. H. Hong and A. Rosenfeld, Segmentation and estimation of image region properties through cooperative hierarchical computation, IEEE Trans. Syst. Man. Cybern. SMC-II(12), 802-810 (1981). Y. N. Sun, H. T. Chiu and X. Z. Lin, A computer system for the analysis of liver cirrhosis from ultrasonic images, Chinese J. Med. Biol. Engng 11, 119-135 (1991). R. M. Haralick, S. R. Sternberg and X. Zhuang, Image analysis using mathematical morphology, IEEE Trans. Pattern Analysis Mach. lntell. PAMI-9(4), 925 934(1987). M. Kass, A. Witkin and D. Terzopoulox, Snakes: active contour models, Int. J. Comput. Vision 1, 321-331 (1987). Y.T. Zhou, R. Chellappa, A. Vaid and B. K. Jenkins, Image restoration using a neural network, IEEE Trans. Acoust. Speech Signal Process. 36(7), 1141-1151 (1988). K. F. Cheung, L. E. Atlas and R. J. Marks, Synchronous vs asynchronous behavior of Hopfield's CAM neural network, Appl. Optics 26(22), 4808-4813 (1987).
APPENDIX A The purpose of this appendix is to rearrange the cost function represented by equation (9) to have the same form as the Lyapunov function of equation (8) of a Hopfield network. From there, the values of interconnections, Tck#a, and bias inputs, l~.t, are also obtained. From equation (9), we have
{[(5.,,:,,
i.
.
+
.
xi+,.kv/+,..,,
)a
.
+
k=l
/
k=l
,.kvi ,.k--2 Y', Vi.,vi.,+ ~, yi+L, vi+,.t k
k=l
•
'
h\k=l
--7 /
k=l
d
k=l
gi.,v,.,
+a
1-- 2 v,.,
Lk=l
.
k=l
(A1)
After expanding and rearranging all the square terms in equation (A1), we have
E =
- ~ k
i=1
4-
~ Xi&Xi.ll3i.kUi.I ll=l
--
~J~ X i - l.kXi - ldUi - 1.kUi - 1.1 k
Xi.kXi- l.lti.kUi- 1.l k
II
Yi.kYi.'t'i.kl3i,l
-[-
11=1
k
+
~
Yi
1.kYi
1.lVi
1.kUi-l.l
-2
L= L_
l ' k X i + l ' l U i - 1 .k/)i + 1 d
,.,
,., +
-2
L=
11
k
11
11
x,+
--
/
Xi_l,kXi_l,#~i_l,kt:i_,.l
--fl
l.kXi. 1 Ui- 1&Did
Yi&Yi- 1.1Ui.kt:i - 1.1 k
k
1_- ~ 1 Xi
Xi
~. k=ll=
--
1/=
\k=ll=l
+ \k=l
--
1
--2
1
V~
~ , Yi
Xi
l.kXidt'i
k=ll=
+4
k=l~
,., -2
z,=
1 Xi.kXi-I.IUi.kl)i-ld
,.,,,,+
=
1.kYidUi
\kell=l
' I Xi.kXi.lt, "i,kUi.
,+ ,.,,.,,,,+
..ktri.l
1.kl~id
1066
C.-T. TSAI et al.
~-
~=1Yi
-2
1.kYi+l.lt:i-l.tb'i+l.t
k
-- 2
~ k
Yi.kY'i+ l.lUi.kUi+ 1.1
+
)]
Yi+l.kYi+l.lVi+l.kVi+l.I =
:
k
Yi+ 1.kYi- 1.lt~i+ =
+4
I /=
-4-
I /=1
)(,f,f
~-1Yi.kYil.lt)i.kl)i-l.I
\k=ll=
1.kVi -
-2
l.I
Yi+ l&Yi.lVi+ l.kl)i.t
l=
=
][
~.qi,kv~,~ +o
--7
=
1--2
~. k=l
)
Yi.kYi.ll)i.kl)i.¿
1 l=
vit~+ "
I=
~. k= 1
t,i.kVi.i =
]}
(A2)
.
Since the endocardial border is assumed to be a closed contour, the first sampled point V~ succeeds the last sampled point VN. Therefore, the ordering of the sampled points satisfies the circular indexing property. We define x0. ~ = xN. ~ and x N+ ~.k = X~,k. The index arithmetics used in these equations can be defined by a module N operation. Thus, the following equation is obtained:
N
I}
13
~ 2 Xi i=lk=ll=l L2 L2 =
1.1l'i
1.kl'i
1,1
~ k
:
1, kXi
~ Xo.kXO. 1 llO.kt'O. 1 + Xl.kXl.lt!l.kt~l.t 11 1 L2 L2
~
+
"'" +
XN-
I.kXN - I.tUN - l.kl~N-
I.t
~ XN,k.%e.tVN.kVN.t4-.~:l.kXl,tvl,kt'l.l+'''+xN-I,kxN 1.tVN-LkvN l.t
k-I
/=1
=
1}
L2
2
2
(A3)
Xi,kXi,lt'i'kUi,l"
i=lk=l/=l
Similarly, by applying the circular indexing property to other terms, we obtain
E x,.*x,.,v,M',.,= E E S XI+'.kX'+'.'Vi+'.kV'+'.t i-lk-ll
I
i-lk-ll
Z i-I
k=l
i=1
k=l
x,.,x,_,.,,,,.,,,,
,.,
=
I-1
Y x,+,.,x,.,,,,+,;:,.,
2 i=l
k=l
i=1
k=l
E ~ .,£_,.,xi.,vi ,.kr,.,= I=1
(A4)
=1
(AS)
1-1
2 2 Xi.kX,+,.'V'.kV'+'.'
(A6)
1-1
L2 L2 ~ "21"2 E E Xi-l,l~Xi+1,ll-~i 1'kU/+lJ= S S X'.kX'+2.'V'.d"÷2,' i-1
k=l
I~1
i=1
k=l
Z %. y,.,y,.,",.~,',., = i-lk-II=l ~.
L2
2 i lk
Lz
N
(A7)
I-I
E y,-,.,y,-,.,",-,.,",-,., =
L2
Y Z y,+,.,y,+,.,,',+,.,",+,.,
L2
(A9)
2 Z Y,.kY,-mvi.kvi ,., = 2 2 2 Yi+,.D',.,vi+,M',., i-lk-ll=l L2 I. 2
i
lk-ll-1
~
E E ~',-,.~y,.,,', i-lk II L2 L 2
1"2 L 2
Y E y,.~v,+,.?,.~,,,+,.,
,.~,~,.,=
1
lk
(Am)
i=lk-ll 1 ~ L2 L 2
~'. Z Yi ,.k.Vi+,.#i ,.~v,+,.,= i
(AS)
i=lk-II-I
11-1
(All)
E Z Yi.D'i+z.?iM'i+2.v i=lk-ll-I
11~1
Substituting equations (A3)-(A11) into equation (A2), the energy function can be rewritten as
{
E=
)
-(2oc+6.B)
i=l
k
-fl
X,.kX,.,V~.~V,.,+(~+4.B) 11
II
k=ll
-9 (~ + 4fl)
)'i,kYik
l.lUi.kUi-
11=1
E Y'.*Y'-z.'v'.'v'-z.' k
kE'
1
1/~1
)
~ ~ X,.,X, ,.,V,.kV, ,., +(~+4,B)
1
~_ ~ x,.kx,+2.,vi.,v,+;., - f l k
(L ,2
l=l
-]-
(~ +
% k
II
k
)
Yi,kYl+ l,lVi,tt'i+ l,l I
I=
-
Yi.tYi+z.tt!i,kt!i+ k
k=l
k=II~l
2.1
ll=
+(~ I - 2 E v,.,+ E Z t,,.,t,,., .
7
x,.,x,+,.,v,.,v,+ m
yi.tyi.,vl.kvi., =
4fl)
~
k~ll=
~1
XI.kX,_z.,Vl.kV,_:., --(2~+6fl)
=
(L2
J)
(A121
Segmentation of echocardiograms
1067
By using the Kronecker delta function, we then have 1. 2
E=
~
N
L2
~
~ {[-(2~+6fl)~i.~+(~+4fl)3,+l.j+(~+4fl)3,
i=1 k=l j-1
Lj--flai+2.j--133,-2.j]*[xi.kXj.,+Yl.kY~i.,]
/=1
N
+aai,,lVi,RV~,,-- E
L2
(A13)
Y~ {Y#i.k + 2a}vi.k + a N "
i = 1 ,k=l
By comparing the terms in equation (Al 3) to the corresponding terms in equation (8) and ignoring the constant term crN, the interconnection strengths and bias inputs can be expressed as
T,.g:i.j = [(4a + 12fl)3~o
(2ct + 8fl)fi~+ ~.j - (2~ + 8fl)6~ ~.j + 2fl3~+2.j + 2fl6~ 2.j]* [x~.kx~.~ + Y~.kY~a] -- [2aft.j]
(A 14)
and (A15)
ILk = )',qi,k + 20-
respectively.
From equations (5) to (7), the following equations are obtained:
APPENDIX B
The proposed network model does not guarantee that the energy function E always decreases monotonically with a state change. In this appendix, a condition of sufficiency for assuring the energy decreasing during the iterations of state changes is given as follows. First, let Avi. k denote the state change of neuron (i, k), i.e. Avi. k = ti.k, new __ vo.~d. k From reference (19), the energy change, AE, due to a state change Avi, k is given by
if
Ui.new k = I=:-AVi,k>0
lli,k~O
(B3)
or i f ui. k < 0 ~ ' v n t k ~
(B4)
= 0 : : : z -A V i , k __<0.
Therefore Ul.k *Ari, k > O.
AE = -
~
r,.~:j.,~,;y + I,.~
....
av,.~. (B1)
i=lk=l
The energy change is not always negative when T~k.~k < 0 . ( 2 6 ' 2 7 ) For instance, when the network input u,.k = Y , Z Tik:J,vT!~ j
- 1 and Avi. k - 1.
Taking Tik.i k > O, we get AE=-
[Ui.k + 12Ti.k;i,k(Avl.k)lAVi.k
t
+ l~g > 0 and t#i,kd = 0, according to the Hopfield updating rule in equation (7), we have v,.".~ ~
(B5)
(B2)
But, if the network inputs u~.k are such that Tik,ik < --2ui, k in equation (B1), then AE > 0. In this situation, the condition for energy decreasing in each neuron updating is T~kak> -- 2Ui.k" However, the network input u~.k, which depends on the weighted inputs ~, Tik:jlt.rjl and the bias input Ick, is unpredictable. Even j,I
so, the energy change AE in equation (B1) is always negative subject to Tik,i k ~ 0 . (27) This condition is illustrated below.
= -
Ui.kAVi.k +
<_0.
(B6)
From the above derivations, we have shown that the energy change of the neural network is always negative with a state change when the value of self-feedback, i.e. T~k.lk, is greater than or equal to zero. From equation (18) with i = k and j = 1, we have Tik,i k = (4~t + 12fl)((X'ilk) 2 + (y'i'.k)2) -- 2a. Hence. the parameters :t, fl, and o are chosen to satisfy the condition of sufficiency that (4ct + 12fl)((X'i'k) 2 + (y'i'k)2) -- 2a > 0. In this case, the convergence of the modified Hopfield network is guaranteed.
About the Author--CHING-TSORNG TSAI was born in Chiay, Taiwan, on 14 March 1966. He received the B.S. degree in information science from Tunghai University, Taiwan, and M.S. degree in information engineering from National Cheng Kung University, Taiwan, in 1988 and 1990, respectively. Currently he is working for his Ph.D. degree at the Institute of Electrical Engineering, National Cheng Kung University. His research interests include computer vision, medical image processing and artificial neural networks. He is a student member of the Chinese Association of Image Processing and Pattern Recognition.
About the Author--YuNG-NtEN SUN is an associate professor at the Institute of Information Engineering,
National Cheng Kung University, Taiwan, which he joined in February 1989. From 1987 to 1989, he was an assistant at the Brookhaven National Laboratory, Long Island, New York. He received the B.E. degree in controlling engineering from National Chiao Tung University, Taiwan, in 1978, and the Ph.D. and M.S. degrees both in electrical engineering from the University of Pittsburgh, Pennsylvania, in 1987 and 1983, respectively. His current research interest is in computer vision, medical image processing and neural networks. Dr Sun is a member of IEEE, Sigma-Xi, the Chinese Association of Image Processing and Pattern Recognition, and the Chinese Association of Biomedical Engineering.
1068
C.-T. TSAI et al. About the Author--PAu-CHoo CHtJy6 received the B.S. and M.S. degrees in electrical engineering from National Cheng Kung University, Taiwan, in 1981 and 1983, respectively, and the Ph.D. degree in electrical engineering from Texas Tech University in 1991. From 1983 to 1986 she worked in Chung Shan Institute of Science and Technology, Taiwan. She is currently an associate professor in the Department of Electrical Engineering at National Cheng Kung University, Taiwan. Her recent research interests include artificial neural networks, neural models, and computer vision. Dr Chung is a member of IEEE and INNS.
About the Author--JIANN-SHu LEE was born in Tainan, on 4 May 1966. He received the B.S. and M.S. degrees in electrical engineering from National Cheng Kung University, Taiwan, in 1988 and 1990, respectively. Now he is a Ph.D. candidate at the Institute of Electrical Engineering, National Cheng Kung University. His current research interests are image processing, pattern recognition, data communication and digital signal processing.
0031 3203~93 $6.00+.00 Pergamon Press Lid ;~ 1993 Pattern Recognition Society
Printed in Great Britain
ENDOCARDIAL B O U N D A R Y DETECTION USING A NEURAL NETWORK CHING-TSORNGTSAI,+ YUNG-NIENSUN,~§ PAu-CHoO CHUNG~ and J1ANN-SHu LEEr + Department of Electrical Engineering, National Cheng Kung University, Tainan, Taiwan, 70101, RO.C. Institute of Information Engineering, National Cheng Kung University, Tainan, Taiwan 70101, R.O.C. (Received 20 May 1992; in revised.]brm 25 November 1992; receivedfor publication 5 January 1993) Abstract--Echocardiography has been widely used as a real-time non-invasive clinical tool to diagnose cardiac functions. Due to the poor quality and inherent ambiguity in echocardiograms, it is difficultto detect the myocardial boundaries of the left ventricle.Many existingmethods are semi-automatic and detect cardial boundaries by serial computation which is too slow to be practical in real applications. In this paper, a new method for detecting the endocardial boundary by using a Hopfield neural network is proposed. Taking advantage of parallel computation and energy convergencecapability in the Hopfield network, this method is faster and more stable for the detection of the endocardial border. Moreover, neither manual operations nor a priori assumptions are needed in this method. Experiments on several LV echocardiograms and clinical validation have shown the effectivenessof our method in these patient studies. Neural network Hopfieldnetwork Automaticendocardial border segmentation Parallel border detection Ultrasonic image
I. INTRODUCTION Echocardiography has been used as a major clinical tool to diagnose cardial function for many years. Because echocardiography is noninvasive and secure, it is readily accepted by specialists and patients. Therefore, it has been widely used as a real-time tool to evaluate the left ventricular function. In order to obtain indices of cardiac performance such as ejection fraction and ventricular volume, it is necessary to obtain the contour lines corresponding to the endocardial wall of the heart chamber/1) Manual definition of cardial contour lines is time consuming, tedious, and substantially subjective, therefore, it limits the clinical usefulness of the quantitative analysis. Computer aided processes for defining cardial boundary have been used to prevent the problems caused by manual tracing. Unfortunately, the echocardiogram usually has low signal-to-noise ratio (SNR), low spatial resolution, and occasional loss of echo signal. The echocardiogram also shows some connective tissues such as the papillary muscle or the mitral valve. Hence, the detection of cardial borders has been one of the most difficult image processing problems/z~ Recently, several studies have been devoted to the detection of the heart wall contour from twodimensional (2D) echocardiograms. In general, they can be classified into three main categories, which are edge-based, 13'4) region-based,15-12) and knowledgebased (rule-based) methods. I13 16) The edge-based methods ~3'4~apply an edge operator to the image and select a proper threshold to determine
§ Author to whom all correspondence should be addressed.
Echocardiogram
the edge points. However, the requirement of a good threshold is one of the major drawbacks of this method. Without a suitable threshold, this method could cause spurious edge points. Hence, this approach is not effective in finding the boundary of a complex echocardiogram due to its high noise and signal dropout. ~5) The region-based methods ~5-12) search the boundary points based on the estimated reference center or an initial contour of the heart chamber. Some previous works have shown that these methods are promising, but most of them are semi-automatic, because either a priori assumptions or human interventions are needed. In these methods, some additional conditions such as the ventricular center, starting contour tracing point, radially searching limits, and linear or average rate of wall motion from end-diastole to end-systole, have to be imposed in the detection procedures. However, these conditions are usually hard to define a priori, thus the capability of these methods in practical applications is limited. The knowledge-based (rule-based) methods 113 16) integrate the knowledge of heart anatomy and the characteristics of 2D echocardiograms to construct a knowledge rule which is aimed to detect the cardial borders. These methods make the detection of cardial borders more reliable and can be implemented in AI machines. However, the slow response time of present AI systems is an inevitable limitation for real-time applications. From the reasons mentioned above, it is desirable to develop a new approach that can achieve both fully automatic segmentation and fast computation for cardial border detection. In this paper, we propose a system for extracting the endocardial borders from 2D echocardiograms based on a Hopfield network.
1057
1058
C.-T. TSAIet al.
The system consists of four modules: pyramid-linked enhancement, initial endocardial border estimation, endocardial border detection by a Hopfield network, and contour interpolation. (I 7) In the implementation of Hopfield neural networks, ~18.~9) the finding ofendocardial boundaries is regarded as a constraint satisfaction problem. The constraints consist of the continuity, curvature, and edge magnitudes on the endocardial border. An initial contour, which is placed near the ventricular boundary, is first transformed into an initial state of the Hopfield network. After undergoing a parallel and dynamic process of neuron updating, a final stabilized state of each neuron satisfying the above given endocardial constraints represents the accurate endocardial boundary. In other words, a 2D binary Hopfield network is implemented to detect the endocardial boundary. The endocardial border detection is based on the network energy minimization. A condition for assuring the network convergence is derived. If this condition is not satisfied, a deterministic decision rule for neuron updating is used. Experimental studies have been performed on several serial LV echocardiograms obtained from the University Hospital at National Cheng Kung University. We utilize the detected contour on the present frame to detect the border on the next frame throughout the whole cardial cycle. Good results have been obtained for clinical validation. Compared with conventional endocardial border detection methods, our approach provides a fully automatic segmentation of the left ventricle and a solution more suitable for parallel implementation. In the next section, an overview of the proposed system will be given. Then, the Hopfield network for endocardial boundary detection will be presented in Section 3. Section 4 addresses the results of experimental studies with computer simulation for clinical validation. Finally, the conclusions are given in Section 5. The mathematical derivations for the Hopfield network are given in the appendices.
Linked-pyramid enhancement ]
Initial contour estimation i
Endocardial borders detection [ based on Hopfield N.No 1 Contour interpolation
]
Endocardial contour output Fig. 1. The system diagram.
2.1. Image enhancement Poor image quality causes difficulties in echo image processing. It is also the major obstacle in the segmentation of myocardial borders. In our proposed method, a node-linking pyramid ~2~ 23~ is applied to improve the image quality. Like other pyramids, there is a layered arrangement of square arrays in which each array is half in length and width as the next array below it. A father-son relationship is defined between nodes in adjacent layers and established according to the nearest father selection rule. t21) Image enhancement is implemented as a process which selects a single legitimate father for each node from its candidate fathers in an iterative manner. It can smooth the intensity fluctuations by using the link relationship between each level in the pyramid. The main advantage of applying a node-linking pyramid is that it can preserve the endocardial boundary information and reduce the irrelevant local perturbation as well. Figure 2(b) shows the result after applying this operation to the image in Fig. 2(a).
2. A SYSTEM FOR THE D E T E C T I O N OF ENDOCARDIAI, BOUNDARY
2.2. Initial endocardial boundary estimation for the first image
An automatic system using a Hopfield network for detecting the left ventricular boundaries from sequences of 2D echocardiograms is described in this section. Our system consists of three major modules: (1) image enhancement for the whole sequence of images, (2) initial contour model estimation for the first image, and (3) accurate endocardial boundary detection using a Hopfield network. After the boundary pixels are detected, the endocardial border can be computed by interpolation. The Catnull Rom spline {2°) is used in interpolating these endocardial boundary points. Finally, we utilize the determined edge contour in the present frame to detect the ventricular border for the subsequent frame throughout the whole cardiac cycle. The flow diagram of the system is shown in Fig. 1.
Morphological techniques" 1.24) are applied to estimate the initial endocardial boundary for the first image. The process consists of a grey-scale opening with a flat-topped hexagonal structuring element which has a radius of 11 pixels (H11). Hills, which cannot accommodate the hexagon are removed from the image surface C during the filtering operation. Thus, a background image with slowly varying illumination is obtained after applying these morphological operations (see Fig. 2(c)). The resulting image D can be expressed as
D = C-((CGH
Jl)~Hll).
(1)
To reduce the impact of echo dropout, a closing operation is used to fill the holes generated by echo dropout
Segmentation of echocardiograms
1059
Fig. 2(a~. An original echocardiogram.
Fig. 2(c). The resulting image after opening and closing.
Fig. 2(b). The resulting image after pyramid enhancement.
Fig. 2(d). The initial contour after threshold and sobel operation.
on image D: that is
D = ( D G H 9 ) Q H 9.
(2)
After the process, a binarization operation is used to obtain the ventricular area and a sobel operation is utilized to produce the initial endocardial boundary (see Fig. 2(d)). 2.3. Endocardial boundary detection The search for endocardial boundaries is one of the most difficult tasks in 2D echo image processing. In this subsection, we regard this task as a problem of constraints satisfaction. A modified Hopfield neural network is used to solve this constraint satisfaction problem. Initially, the rough boundary available from the previous morphological segmentation process is used as the initial contour. Then, uniformly sampled points on the initial contour are transformed into the states of neurons and their interconnection values. After undergoing a parallel and dynamic process of the neuron updating, the stabilized state of each neuron in the Hopfield network approaches a resulting contour point satisfying the given endocardial constraints. The
details of this new cardial boundary detection algorithm based on the Hopfield network will be given in the following section.
3. ENDOCARDIAL BORDER DETECTION USING A HOPF1ELD NETWORK
The application ofa Hopfield network for the detection of endocardial border is presented in this section. In this method, the location of the rough boundary available after the morphological segmentation process is taken as an initial contour. Under the assumption that the actual endocardial boundary yields a higher sum of gradient and contrast magnitude than any other trajectories around the boundary, the output of this method results in a new curve which is chosen as an accurate estimate of the endocardial boundary. Continuity and curvature of the endocardial boundary are also considered in our method. A cost function is defined to be the sum of gradient, contrast, continuity, and curvature of the ventricular boundary. The proposed method locates the cardial boundary point by searching in the neighborhood of the sampled point
C.-T, TSA]et al.
1060
on the initial border. The Hopfield network approaches the accurate cardial boundary by minimizing the cost function at each neural in a parallel manner (see Fig. 3). The cost function in our method is similar to the one used in the active contour model. (25) Therefore, the endocardial border detection is regarded as a minimization of the following equation:
border is allowed to stretch or bend. The constant ;, reflects its relative importance in defining the cardial boundary. Let X(Vi) and Y(Vi) denote the x-coordinate and the y-coordinate of the point Vi, respectively. Approximating the derivatives with finite differences, equation (3) can be further expanded as N
E = ~. - ~ [ ( X ( V i ) - x ( v ~
N
E= ~ -~rrv;ll"-l.~,llV,/ij2-.;etvi)
(3)
i--I
-fl[(X(V,
where V~and F'i' are the first- and second-order derivatives of the ith chosen ventricular boundary point Vi, respectively; N the size of the contour, and P(Vi)= llV/(Vi) II, in which VI(Vi)is the image intensity gradient and contrast magnitude of the point I/u The values of c~and/:t determine the extent to which the endocardial
,)F
+(Y(V~)- Y(v~_,)F]
i=1
,) - 2 X ( V ~ ) + X(Vi+ ,))2
+(Y(Vi_,)-2Y(V~)+ Y(VI+,)) 2] - 7P(Vi).
In this method, the rough boundary obtained from morphological segmentation is first uniformly sampled into N points. Each sampled point is associated with its 2D neighboring points. These
L2
-
-
1 neighboring
(a) ~o•. • •
\ 0@00 0 000
" 0 0
0 0
0000
2
rhood
/v,:. \
C0 0 2 ,..-10 0
0
°o
•
(b)
O0
0000
• 0 0
0©00
0
20 0
0©00
0
Q
0
•
10 1
0 2
0@00 •
•
•
i-1
i i+1
0 i+2
(4)
•
•
:
N
O
a firing neuron
: a resting neuron
Fig. 3. The relationship between the model of endocardial border and a Hopfield neural network. (a) The initial border points and their corresponding neighborhood. (b) The Hopfield network with initial states of neurons corresponds to the initial cardial boundary.
Segmentation of echocardiograms points and the sampled point itself are orderly indexed from 1 to L2. Figure 3(a) also gives an example showing the positions and indices of the sampled point 1/i and its L2 - 1 neighboring points. These L 2 points centered by the ith sampled point will be mentioned as points (i, k)s with I _< k < L2, subsequently in this paper. Thus, as i varies from 1 to N, the points on the endocardial border are indicated by 2D indices (i, k)s, where 1 _< i £ N and 1 _< k _< L 2. The N sampled points are taken as the initial input boundary to the Hopfield network (see Fig. 3). Through network evolutions, the optimal (or suboptimal) cardial boundary points are chosen from the initial boundary points and their neighborhood. The choice is imposed such that one and only one point is selected from points (i, 1) to (i, L2), for each i. Thus, a 2D binary Hopfield network is used to detect the endocardial borders by minimizing the total energy of the network. The network is constructed as an N* Lz array of neurons with neuron (i,k) representing the point (i, k) of the endocardial border. The state of each neuron (firing or resting) in the network represents the possible choice of the corresponding point as an endocardial boundary point. Let V~.k denote the binary state of the (i, k)th neuron (1 for firing and 0 for resting) and T~,k;ja the interconnection strength between neuron (i,k) and neuron (j,I). A neuron (i,k) in this network receives weighted inputs, ~ T,. k:ja V;a,from all the neuj.I
tons and a bias input, I~, k, from outside. Therefore, the total input to neuron (i, k) is computed as N
L2
(5)
~i.~ = Y. Y~ r,,j.,vj., + t,.~. j
ll=l
-}-
~
~
~
(7)
~ Ti.k:j,lUi.klPj.I -
2i=1
k=l
j=l
/=1
L~ E
i=1
k=l
k=l
-2
k=l
~
Yi.kUi.k-~
k=l
--7'
gi.kl)i.k k
E
Yi+l,kl~i+l.k
k=l
+ 0"
Ii.kUi,k"
k=l
(8) The network achieves a stable state when the energy of the Lyapunov function is minimized. Let X~,k and Y~,k be the x-coordinate and the y-coordinate of the point (i, k), respectively, and Yi.k the gradient strength of point (i, k). According to equation (4), the endocardial border detection can be characterized as the minimization of the following energy function: - ~ i=1
Xi,kli.k -k
1
Xi k=l
l,kUi
1,k
E
tli.k
.
(9)
k=l
The first and the second terms in equation (9) are the first- and second-order derivative terms in equation (4). The third term represents the gradient strengths. The fourth term is included for enforcing the uniqueness constraint where one and only one neuron is active in each column; that is, only one point is selected from points (i, 1) to (i, L2) for each i. This uniqueness constraint is also implied in the representations of the other terms in equation (9), where ~ Xi.kVi.k is used to k
represent {he x-coordinate of the selected boundary point, EYi.kt!i,k t h e y-coordinate, a n d Egi.kUi.k the k
k
gradient. Equation (9) can be simplified and fitted into the Lyapunov function, i.e. equation (8), of a Hopfield network. From there, the values of interconnection Ti,k;j. l and bias input li. k c a n be obtained, as shown in Appendix A, as [(4~t + 12fl)~i. j - (2~ + 8fl)ai+ 1.j - ( 2 ~ + 8fl)fi~
21:~6~. :,j]
' 4 + 2fl8~ + 2.j +
* [Xi,kXj.l -{- Yi.kYja] --
[20.t~i.j]
(1 O) (l l)
respectively, where (~i.jis the Kronecker delta function defined as 6i.j = 1 if i = j and 0 otherwise. From equation (10), one can see that the network is not fully interconnected. Neurons in a specific column receive inputs from neurons in the same, the previous two, and the posterior two columns. Furthermore, the interconnection strength between neuron (i, k) and neuron (j, l) depends only on the coordinates of the point (i, k) and point (j, l) along with the assigned constants ~t,,q, and 0". From equation (ll), the bias input to neuron (i, k) is determined by the gradient magnitude Y~.R,which is obtained from the original image. It should be noted that the interconnection strength Ti,k:;a depends on the origin of the coordinate system. This is because the continuity and curvature terms in equation (4) are decomposed into arithmetic expressions with the coordinates x~. k and Yi,k.The same situations also occur for example, E X i . k l ? i . k , EYi.kUi.k , and k
E =
1 --
1
li.k = ~'.('Ji,k -}- 20"
The Lyapunov function of a 2D Hopfield network is given by 1~
,.~-- 2 y . Xi.kt'i, k
1
and
where ,q(x) is the hard-limiting function defined as
E=-
x,_ ,.kVi k
(6)
if x < 0"
~'i- l.k i l.k k=l
-fl
Ti,k:j. I =
y(x) =
Yi,kUi,k k=l
During network evolution, a randomly selected neuron, say neuron (i, k), is updated according to Vi.k = g(Ui.k)
1061
Xi.kV,k -- ~ x i - l.kVi 1.k k
k
in equation (9). How-
1062
C.-T. TSAIet al.
ever, if the uniqueness constraint that exactly one neuron fires in each column is satisfied, the energy has the same value wherever the origin of the coordinate system is located. This uniqueness constraint is imposed in the fourth term of equation (9). In the implementation of the Hopfield network, multiple or no neurons firing in each column may occur. Therefore, a normalization of these interconnection strengths is needed to depreciate the influence of this situation. One possible normalization process is described as follows. Let C,,ax = m a x ( X l , l , X 1 , 2 . . . . . XN,L2, Y l , 1 , . . . , y s . L 2 ) , Cmin = min(xl,l,xl,2 . . . . . X N , L ~ , Y l , 1 . . . . . YN,L2), gm,x = max(g1.1, gl,2, • •., gNX2), and gmi, = min (g1,1, gl,2,-.., gN.L2), where max (.) and min (.) are the maximum and the minimum functions, respectively. The normalized x}, k and Y'i,k are expressed as
X'i,k=Xi.k--Cmin,
l
and l < k < L
2
(12)
l
and l < k < L 2
(13)
and
tion (9) in a parallel manner. The algorithm for endocardial border detection is summarized as follows. Input: A set of neurons arranged in a 2D array with
initial values set to reflect the initial boundary contour. Output: A set of stabilized neuron states representing the accurate endocardial boundary points in the sense that the cost function of equation (9) is minimized. Method:
(1) Transform those initial sampled points and their corresponding neighborhood into the neural network model. (2) Set the initial states of the neurons according to the initial contour. (3) Randomly pick up a node from the network for updating, and calculate its total input: N
Y'i,k=Y~,k--Cm~.,
u,.~ = E j=l
respectively. Obviously, x'g.k and Yl,k are independent of the origin. The weighting values ~, fl, and ? reflect the relative importance of the continuity, curvature, and gradient terms, respectively, and the sum of these three values is equal to 1, given by ~+fl+?=
I.
(14)
Also, the values of the coordinates and the gradients are normalized into the closed interval [0, 1] to unify their relative importance. They are expressed as
x','~ -
X'i, k
(15)
Cma x - - Cmi n
Y~'.k --
Y'i,k
(16)
Cma x - - C m i n
and g~,k Y~',k= " gmax -- groin
(17)
where gl.k = g i , k - gmi.- After the normalization of coordinate and gradient values, the interconnection strengths, T~,k:),l,and the bias input, Ii. k, are modified as
L2
E r , . ~ j . , v , + I,.~. /=1
(4) Update the state of this selected neuron according to equation (6). (5) Repeat steps 3 and 4 for a certain number of times and count the number of neurons having their states changed. If the number is less than a given value, then go to step 6. Otherwise, go to step 3 and repeat the process. (6) Output the final states of neurons which represent the actual endocardial boundary points. The evolution of the network is characterized as a dynamic updating process which achieves a stable state when the energy function of equation (9) is minimized. In Appendix B, the Hopfield network used for our application is proven to converge to a stable state subject to the condition that (4ct + 12fl)((xT,k)2 + (y~'k) 2) -- 2a > 0. When the given constants ~t,fl, and tr do not satisfy the condition, a deterministic decision rule is utilized. The deterministic rule is to take a new state of neuron (i, k) only when the energy change due to state change is less than zero. If the energy change is greater than zero, the neuron remains unaffected. 4. S I M U L A T I O N R E S U L T S A N D C L I N I C A L S T U D I E S
Ti,k;j, l = [(4ct + 12fl)(~i, j - (2ct + 8fl)6i+ ~,j - (2c~ + 8fl)fi~_ 1.j + 2fl6~ + 2,j -~- 2fl6i_
* [x}',kX~,, + YT,kY~,,] -- [2Crf,,j]
2,j] (18)
and li,k = Ygi',k
+ 2O"
(19)
respectively. As mentioned above, the Hopfield network searches for the best-fit endocardial border output through the process of dynamic adjustments from a rough initial border obtained from the preprocessing step. The endocardial border tries to satisfy all constraints in equa-
Ultrasonic images were acquired from an ultrasonic system (77020A Hewlett Packard) at the University Hospital of N C K U and stored by a video cassette recorder. The image data was then captured by a near real-time visual interface board, AFG, and digitized into 512.512 pixels with 256 grey-levels. First, the preprocessing processes, including the pyramid-based enhancement, the morphological operations, and the sobel operation, are implemented on the AFG image board. Figure 2 shows the intermediate results after preprocessing which include the pyramidbased enhancement image, the result of morphological segmentation, and the initial model of the ventricular contour.
1063
Fig. 4(a). The initial border superimposed on Fig. 2(a).
Fig. 4(c). The manual tracking border by a physician.
Fig. 4(b). The resulting endocardial border superimposed on Fig. 2(a).
The proposed network was simulated on an intel486 personal computer. From an initial contour, boundary points are sampled with equal intervals of 25 pixels. There are sually 25-35 sampled points in a closed endocardial border. The neighborhood is defined around each sampled point by a 15,15 window. The values ~, fl, 7, and a were determined experimentally. They are ~ = 0 . 1 , fl=0.3, 1,=0.6, and a = 2 . 0 . The simulated Hopfield network takes nearly 34,000 loops corresponding to approximately 30 C P U min to converge. Here, one loop updates the state of each neuron once. The computation is slow because the simulation is pro-
Fig. 5. The detected endocardial borders of a whole cardial cycle.
1064
C.-T. Tsm et al. pixelnumber
i0000
7500
5000
2500
1
3
5
7
9
11
13
15
Fig. 6. The graph of ventricular area with respect to frame number: ........ , by computer; - physician. cessed sequentially by a micro-computer. However, the Hopfield network can be implemented with a parallel architecture and the endocardial border detection has a great potential to be accomplished by real-time computation. In the physical implementation, after the Hopfield network is stabilized, it may encounter problems such as multiple or no active neurons in a column. In this situation, a decision rule is used to determine the best point. The decision rule takes the average of those active neurons when multiple neurons are active, or the original neuron state when no neuron is active as the final endocardial point. Figure 4(a) shows the initial contour superimposed on the original image in Fig. 2(a). The resulting endocardial border by the proposed Hopfieid network is shown in Fig. 4(b). For comparison, the ventricular border traced by an experienced physician is shown in Fig. 4(c). Results indicate that the boundary obtained by our method matches that obtained by an experienced physician closely. Five patients' data, each containing ! 6 frames from a cardiac cycle, have been processed. Validation of the proposed method is carried out throughout the whole cardiac cycle (1/30sper frame). Figure 5 shows the segmentation results from the sequence of the second patient's echo images NM01-NM16. The resulting area curve calculated from the segmentation results is plotted in Fig. 6. The area curve obtained by a physician is also plotted in Fig. 6 for comparison. The experimental studies have shown that the results obtained by the two methods are quite close to each other. It is also very robust when the image quality is good.
5. CONCLUSIONS
Due to the physical limitations of ultrasonic imaging, it is difficult to develop a parallel and fully automatic myocardial border detection system. Traditional methods for myocardial border analysis are not intelligent enough to handle the task properly. In this
, by the
paper, a Hopfield neural network is designed to achieve these impelling purposes. This method is superior to other methods in three ways. Firstly, the neural approach provides a parallel implementation for the detection of endocardial boundaries in real-time applications. Secondly, it is automatic. Thirdly, it is capable of handling the inherent problems of low contrast and high dropout in echocardiograms. The three advantages make our system more desirable for the detection of the endocardial border. The well-detected endocardial borders from several cardial cycles provide good clinical validation for our system. Acknowledgement--This research work was supported by a grant from the National ScienceCouncil, R.O.C., under contract NSC-81-0420-E-006-007.
REFERENCES
1. L. W. Eaton, W. L. Maughan, A. A. Shoukas and J. L. Weiss, Accurate volume determination in the isolated ejecting canine left ventricle by two-dimensional echocardiography, Circulation 60, 320-326 (1979). 2. A. Herment, J. P. Guolielmi, P. Dumee, P. Peronneau and P. Delouche, Limitations of ultrasound imaging and image restoration; Ultrasonics 25, 267-273 (1987). 3. D.J. Skorton, C. A. McMary, J. S. Child, F. C. Newton and P.M. Shahn, Digital image processing of twodimensional echocardiograms: identification of the endocardium, Am. J. Cardiol. 48, 479-486 (1981). 4. W. Zwehl, R. Levy, E. Garcia, R. V. Haendchen,W. Child, R. Cordav, S. Meerbau and E. Cordav, Validation of a computerized edge detection algorithm for quantitative two-dimensional echocardiography: comparison with anatomic data, Am. J. Cardiol. 53, 1380-1387 (1984). 5. C. H. Chu, E. J. Delp and A. J. Buda, Detecting left ventricular endocardial and epicardial boundaries by digital two-dimensional echocardiography, IEEE Trans. Med. Imag. MI-7, 81-90 (1988). 6. P. R. Detmer, G. Bashein and R. W. Martin, Matched filter identificationofleft-ventricular endocardial borders in transesophageal echocardiograms, I EEE Trans. Med. lmag. 9(4), 396-404 (1990). 7. D. Adam, O. Hareuveni and S. Sideman, Semiautomated border tracking of cine echocardiographic ventricular imag¢s, IEEE Trans. Med. Imag. MI-6, 266-271 (1987).
Segmentation of echocardiograms
8. L. Zhang and E. A. Geiser, An effective algorithm for extracting serial endocardial borders from 2-dimensional echocardiograms, IEEE Trans. Biomed. Engng BME31(6), 441-446 (1984). 9. L. Oliver, E. Geiser, L. Zhang, J. Gardin, A. Parisi, N. Reichek, A. Weyman, J. Werner, R. Kerber and R. Conti, Evaluation of an automated border detection algorithm in 2-D echo images of varying quality: observer versus computer variability, J. Am. Coll. Cardiol. 3(2), 506-513 (1984). 10. S. Tamura, S. Nakano, M. Matsumoto, T. Shimazu, M.T. Matsuyama and P. Hanrath, Three-dimensional reconstruction of echocardiograms based on orthogonal sections, Pattern Recognition 18, 115-124 (1985). 11. J. W. Klingler, C. L. Vaughan, T. D. Fraker and L. T. Andrews, Segmentation of echocardiographic images using mathematical morphology, IEEE Trans. Biomed. Engng BME-35, 925-934 (1988). 12. W.S. Chou, C. M. Wu, Y.C. Chen and K.S. Hsieh, Detecting myocardial boundaries of left ventricle from a single frame 2DE image, Pattern Recognition 23, 799-806 (1990). 13. C. Y. Han, K. N. Lin, W. G. Wee, R. M. Mintz and D. T. Porembka, Knowledge-based image analysis for automated boundary extraction of transesophageal echocardiographic left-ventricular images, IEEE Trans. Med. Imag. 10, 602 610(1991). 14. J. Feng, W. C. Lin and C. T. Chen, Epicardial boundary detection using fuzzy reasoning, IEEE Trans. Med. Imag. 10(2), 187-199 (1991). 15. N. Friedland and D. Adam, Automatic ventricular cavity boundary detection from sequential ultrasound images using simulated annealing, IEEE Trans. Biomed. Engng 8, 344-353 (1989). 16. S. Tamura, K. Yata, M. Matsumot, T. Matsuyama, T.
17. 18. 19. 20. 21. 22.
23. 24. 25. 26. 27.
1065
Shimazu and M. Inoue, Plan-based boundary extraction and 3-D reconstruction for orthogonal 2-D echocardiography, Pattern Recognition 20, 155-162 (1987). C. T. Tsai and Y. N. Sun, Segmentation of echocardiograms using a Hopfield network, Int. Conf. of Euromicro, Paris, pp. 791-797, September (1992). J.J. Hopfield and D. W. Tank, Neural computation of decisions in optimization problems, Biol. Cybern. 52, 141-152 (1985). J. J. Hopfield, Neural networks and physical systems with emergent collective computational abilities, Pro('. Natn. Acad. Sci. USA 79, 2554-2558 (1982). J. Foley, A. V. Dam, S. Feiner and J. Hughes, Computer Graphics Principles and Practice, 2nd Edn. Addison-Wesley, Reading, Massachusetts (1990). R.A. Rosenfeld, Multiresolution Image Processing and Analysis. Springer, Berlin (1984). P.J. Burt, T. H. Hong and A. Rosenfeld, Segmentation and estimation of image region properties through cooperative hierarchical computation, IEEE Trans. Syst. Man. Cybern. SMC-II(12), 802-810 (1981). Y. N. Sun, H. T. Chiu and X. Z. Lin, A computer system for the analysis of liver cirrhosis from ultrasonic images, Chinese J. Med. Biol. Engng 11, 119-135 (1991). R. M. Haralick, S. R. Sternberg and X. Zhuang, Image analysis using mathematical morphology, IEEE Trans. Pattern Analysis Mach. lntell. PAMI-9(4), 925 934(1987). M. Kass, A. Witkin and D. Terzopoulox, Snakes: active contour models, Int. J. Comput. Vision 1, 321-331 (1987). Y.T. Zhou, R. Chellappa, A. Vaid and B. K. Jenkins, Image restoration using a neural network, IEEE Trans. Acoust. Speech Signal Process. 36(7), 1141-1151 (1988). K. F. Cheung, L. E. Atlas and R. J. Marks, Synchronous vs asynchronous behavior of Hopfield's CAM neural network, Appl. Optics 26(22), 4808-4813 (1987).
APPENDIX A The purpose of this appendix is to rearrange the cost function represented by equation (9) to have the same form as the Lyapunov function of equation (8) of a Hopfield network. From there, the values of interconnections, Tck#a, and bias inputs, l~.t, are also obtained. From equation (9), we have
{[(5.,,:,,
i.
.
+
.
xi+,.kv/+,..,,
)a
.
+
k=l
/
k=l
,.kvi ,.k--2 Y', Vi.,vi.,+ ~, yi+L, vi+,.t k
k=l
•
'
h\k=l
--7 /
k=l
d
k=l
gi.,v,.,
+a
1-- 2 v,.,
Lk=l
.
k=l
(A1)
After expanding and rearranging all the square terms in equation (A1), we have
E =
- ~ k
i=1
4-
~ Xi&Xi.ll3i.kUi.I ll=l
--
~J~ X i - l.kXi - ldUi - 1.kUi - 1.1 k
Xi.kXi- l.lti.kUi- 1.l k
II
Yi.kYi.'t'i.kl3i,l
-[-
11=1
k
+
~
Yi
1.kYi
1.lVi
1.kUi-l.l
-2
L= L_
l ' k X i + l ' l U i - 1 .k/)i + 1 d
,.,
,., +
-2
L=
11
k
11
11
x,+
--
/
Xi_l,kXi_l,#~i_l,kt:i_,.l
--fl
l.kXi. 1 Ui- 1&Did
Yi&Yi- 1.1Ui.kt:i - 1.1 k
k
1_- ~ 1 Xi
Xi
~. k=ll=
--
1/=
\k=ll=l
+ \k=l
--
1
--2
1
V~
~ , Yi
Xi
l.kXidt'i
k=ll=
+4
k=l~
,., -2
z,=
1 Xi.kXi-I.IUi.kl)i-ld
,.,,,,+
=
1.kYidUi
\kell=l
' I Xi.kXi.lt, "i,kUi.
,+ ,.,,.,,,,+
..ktri.l
1.kl~id
1066
C.-T. TSAI et al.
~-
~=1Yi
-2
1.kYi+l.lt:i-l.tb'i+l.t
k
-- 2
~ k
Yi.kY'i+ l.lUi.kUi+ 1.1
+
)]
Yi+l.kYi+l.lVi+l.kVi+l.I =
:
k
Yi+ 1.kYi- 1.lt~i+ =
+4
I /=
-4-
I /=1
)(,f,f
~-1Yi.kYil.lt)i.kl)i-l.I
\k=ll=
1.kVi -
-2
l.I
Yi+ l&Yi.lVi+ l.kl)i.t
l=
=
][
~.qi,kv~,~ +o
--7
=
1--2
~. k=l
)
Yi.kYi.ll)i.kl)i.¿
1 l=
vit~+ "
I=
~. k= 1
t,i.kVi.i =
]}
(A2)
.
Since the endocardial border is assumed to be a closed contour, the first sampled point V~ succeeds the last sampled point VN. Therefore, the ordering of the sampled points satisfies the circular indexing property. We define x0. ~ = xN. ~ and x N+ ~.k = X~,k. The index arithmetics used in these equations can be defined by a module N operation. Thus, the following equation is obtained:
N
I}
13
~ 2 Xi i=lk=ll=l L2 L2 =
1.1l'i
1.kl'i
1,1
~ k
:
1, kXi
~ Xo.kXO. 1 llO.kt'O. 1 + Xl.kXl.lt!l.kt~l.t 11 1 L2 L2
~
+
"'" +
XN-
I.kXN - I.tUN - l.kl~N-
I.t
~ XN,k.%e.tVN.kVN.t4-.~:l.kXl,tvl,kt'l.l+'''+xN-I,kxN 1.tVN-LkvN l.t
k-I
/=1
=
1}
L2
2
2
(A3)
Xi,kXi,lt'i'kUi,l"
i=lk=l/=l
Similarly, by applying the circular indexing property to other terms, we obtain
E x,.*x,.,v,M',.,= E E S XI+'.kX'+'.'Vi+'.kV'+'.t i-lk-ll
I
i-lk-ll
Z i-I
k=l
i=1
k=l
x,.,x,_,.,,,,.,,,,
,.,
=
I-1
Y x,+,.,x,.,,,,+,;:,.,
2 i=l
k=l
i=1
k=l
E ~ .,£_,.,xi.,vi ,.kr,.,= I=1
(A4)
=1
(AS)
1-1
2 2 Xi.kX,+,.'V'.kV'+'.'
(A6)
1-1
L2 L2 ~ "21"2 E E Xi-l,l~Xi+1,ll-~i 1'kU/+lJ= S S X'.kX'+2.'V'.d"÷2,' i-1
k=l
I~1
i=1
k=l
Z %. y,.,y,.,",.~,',., = i-lk-II=l ~.
L2
2 i lk
Lz
N
(A7)
I-I
E y,-,.,y,-,.,",-,.,",-,., =
L2
Y Z y,+,.,y,+,.,,',+,.,",+,.,
L2
(A9)
2 Z Y,.kY,-mvi.kvi ,., = 2 2 2 Yi+,.D',.,vi+,M',., i-lk-ll=l L2 I. 2
i
lk-ll-1
~
E E ~',-,.~y,.,,', i-lk II L2 L 2
1"2 L 2
Y E y,.~v,+,.?,.~,,,+,.,
,.~,~,.,=
1
lk
(Am)
i=lk-ll 1 ~ L2 L 2
~'. Z Yi ,.k.Vi+,.#i ,.~v,+,.,= i
(AS)
i=lk-II-I
11-1
(All)
E Z Yi.D'i+z.?iM'i+2.v i=lk-ll-I
11~1
Substituting equations (A3)-(A11) into equation (A2), the energy function can be rewritten as
{
E=
)
-(2oc+6.B)
i=l
k
-fl
X,.kX,.,V~.~V,.,+(~+4.B) 11
II
k=ll
-9 (~ + 4fl)
)'i,kYik
l.lUi.kUi-
11=1
E Y'.*Y'-z.'v'.'v'-z.' k
kE'
1
1/~1
)
~ ~ X,.,X, ,.,V,.kV, ,., +(~+4,B)
1
~_ ~ x,.kx,+2.,vi.,v,+;., - f l k
(L ,2
l=l
-]-
(~ +
% k
II
k
)
Yi,kYl+ l,lVi,tt'i+ l,l I
I=
-
Yi.tYi+z.tt!i,kt!i+ k
k=l
k=II~l
2.1
ll=
+(~ I - 2 E v,.,+ E Z t,,.,t,,., .
7
x,.,x,+,.,v,.,v,+ m
yi.tyi.,vl.kvi., =
4fl)
~
k~ll=
~1
XI.kX,_z.,Vl.kV,_:., --(2~+6fl)
=
(L2
J)
(A121
Segmentation of echocardiograms
1067
By using the Kronecker delta function, we then have 1. 2
E=
~
N
L2
~
~ {[-(2~+6fl)~i.~+(~+4fl)3,+l.j+(~+4fl)3,
i=1 k=l j-1
Lj--flai+2.j--133,-2.j]*[xi.kXj.,+Yl.kY~i.,]
/=1
N
+aai,,lVi,RV~,,-- E
L2
(A13)
Y~ {Y#i.k + 2a}vi.k + a N "
i = 1 ,k=l
By comparing the terms in equation (Al 3) to the corresponding terms in equation (8) and ignoring the constant term crN, the interconnection strengths and bias inputs can be expressed as
T,.g:i.j = [(4a + 12fl)3~o
(2ct + 8fl)fi~+ ~.j - (2~ + 8fl)6~ ~.j + 2fl3~+2.j + 2fl6~ 2.j]* [x~.kx~.~ + Y~.kY~a] -- [2aft.j]
(A 14)
and (A15)
ILk = )',qi,k + 20-
respectively.
From equations (5) to (7), the following equations are obtained:
APPENDIX B
The proposed network model does not guarantee that the energy function E always decreases monotonically with a state change. In this appendix, a condition of sufficiency for assuring the energy decreasing during the iterations of state changes is given as follows. First, let Avi. k denote the state change of neuron (i, k), i.e. Avi. k = ti.k, new __ vo.~d. k From reference (19), the energy change, AE, due to a state change Avi, k is given by
if
Ui.new k = I=:-AVi,k>0
lli,k~O
(B3)
or i f ui. k < 0 ~ ' v n t k ~
(B4)
= 0 : : : z -A V i , k __<0.
Therefore Ul.k *Ari, k > O.
AE = -
~
r,.~:j.,~,;y + I,.~
....
av,.~. (B1)
i=lk=l
The energy change is not always negative when T~k.~k < 0 . ( 2 6 ' 2 7 ) For instance, when the network input u,.k = Y , Z Tik:J,vT!~ j
- 1 and Avi. k - 1.
Taking Tik.i k > O, we get AE=-
[Ui.k + 12Ti.k;i,k(Avl.k)lAVi.k
t
+ l~g > 0 and t#i,kd = 0, according to the Hopfield updating rule in equation (7), we have v,.".~ ~
(B5)
(B2)
But, if the network inputs u~.k are such that Tik,ik < --2ui, k in equation (B1), then AE > 0. In this situation, the condition for energy decreasing in each neuron updating is T~kak> -- 2Ui.k" However, the network input u~.k, which depends on the weighted inputs ~, Tik:jlt.rjl and the bias input Ick, is unpredictable. Even j,I
so, the energy change AE in equation (B1) is always negative subject to Tik,i k ~ 0 . (27) This condition is illustrated below.
= -
Ui.kAVi.k +
<_0.
(B6)
From the above derivations, we have shown that the energy change of the neural network is always negative with a state change when the value of self-feedback, i.e. T~k.lk, is greater than or equal to zero. From equation (18) with i = k and j = 1, we have Tik,i k = (4~t + 12fl)((X'ilk) 2 + (y'i'.k)2) -- 2a. Hence. the parameters :t, fl, and o are chosen to satisfy the condition of sufficiency that (4ct + 12fl)((X'i'k) 2 + (y'i'k)2) -- 2a > 0. In this case, the convergence of the modified Hopfield network is guaranteed.
About the Author--CHING-TSORNG TSAI was born in Chiay, Taiwan, on 14 March 1966. He received the B.S. degree in information science from Tunghai University, Taiwan, and M.S. degree in information engineering from National Cheng Kung University, Taiwan, in 1988 and 1990, respectively. Currently he is working for his Ph.D. degree at the Institute of Electrical Engineering, National Cheng Kung University. His research interests include computer vision, medical image processing and artificial neural networks. He is a student member of the Chinese Association of Image Processing and Pattern Recognition.
About the Author--YuNG-NtEN SUN is an associate professor at the Institute of Information Engineering,
National Cheng Kung University, Taiwan, which he joined in February 1989. From 1987 to 1989, he was an assistant at the Brookhaven National Laboratory, Long Island, New York. He received the B.E. degree in controlling engineering from National Chiao Tung University, Taiwan, in 1978, and the Ph.D. and M.S. degrees both in electrical engineering from the University of Pittsburgh, Pennsylvania, in 1987 and 1983, respectively. His current research interest is in computer vision, medical image processing and neural networks. Dr Sun is a member of IEEE, Sigma-Xi, the Chinese Association of Image Processing and Pattern Recognition, and the Chinese Association of Biomedical Engineering.
1068
C.-T. TSAI et al. About the Author--PAu-CHoo CHtJy6 received the B.S. and M.S. degrees in electrical engineering from National Cheng Kung University, Taiwan, in 1981 and 1983, respectively, and the Ph.D. degree in electrical engineering from Texas Tech University in 1991. From 1983 to 1986 she worked in Chung Shan Institute of Science and Technology, Taiwan. She is currently an associate professor in the Department of Electrical Engineering at National Cheng Kung University, Taiwan. Her recent research interests include artificial neural networks, neural models, and computer vision. Dr Chung is a member of IEEE and INNS.
About the Author--JIANN-SHu LEE was born in Tainan, on 4 May 1966. He received the B.S. and M.S. degrees in electrical engineering from National Cheng Kung University, Taiwan, in 1988 and 1990, respectively. Now he is a Ph.D. candidate at the Institute of Electrical Engineering, National Cheng Kung University. His current research interests are image processing, pattern recognition, data communication and digital signal processing.