neu images with fuzzy decision tree and Takagi–Sugeno reasoning

Computers in Biology and Medicine 49 (2014) 19–29

Contents lists available at ScienceDirect

Computers in Biology and Medicine journal homepage: www.elsevier.com/locate/cbm

Segmentation of histopathology HER2/neu images with fuzzy decision tree and Takagi–Sugeno reasoning Martin Tabakov n, Pawel Kozak Institute of Informatics, Wroclaw University of Technology, Poland

art ic l e i nf o

a b s t r a c t

Article history: Received 26 July 2012 Accepted 5 March 2014

The Human Epidermal Growth Factor Receptor 2 (HER2/neu) is a biomarker, recognized as a valuable prognostic and predictive factor for breast cancer. In approximately 20% of primary breast cancers, the HER2/neu protein is over-expressed. By recent clinical research, a treatment procedure, with corresponding monoclonal antibodies specifically designed to target the HER2/neu receptor, was confirmed. Therefore, in modern breast cancer diagnostics, it is critical to provide accurate recognition of the HER2/neu positive breast cancer. This can be done by segmentation of the membranes of cancer cells that are visualized as HER2/neu over-expressed on images acquired from corresponding histopathology preparations. In our research, we propose an accurate segmentation process of these structures using an appropriately defined fuzzy decision tree. Moreover, we introduce a new reasoning concept based on the Takagi–Sugeno inference model. & 2014 Elsevier Ltd. All rights reserved.

Keywords: Fuzzy decision tree Takagi–Sugeno inference Image segmentation Histopathology image processing HER2/neu breast cancer

1. Introduction Breast cancer is recognized to be one of the women’s most common cancer, which is the leading cause of morbidity in women aged 20–59 years. In clinical practice four predictive factors have been identified and are assessed in breast cancers. These markers include estrogen receptor (ER), progesterone receptor (PR), Human Epidermal Growth Factor Receptor 2 (HER2/neu) and the Ki-67 antigen, which are determined using immunohistochemical technique in paraffin sections of breast cancer [9]. In this paper the HER2/neu breast cancer recognition problem is considered. HER2/neu biomarker is recognized as a valuable prognostic and predictive factor for breast cancer. In approximately 20% of the analyzed breast cancer cases, an over-expression of HER2/neu is diagnosed [23]. The effect of HER2/neu overexpression is an increase in receptor mediated intracellular signaling, directing the cancer cells to proliferate uncontrollably, which results in an aggressive form of breast cancer. This has significant impact on the patient's prognosis [20,23]. The HER2/neu is a membrane bound receptor, which activity may be blocked by utilizing a monoclonal antibody – trastuzumab, which was found effective in clinical trials (Herceptin, Genentech, CA) [9,20,23].

n

Corresponding author. Tel.: +48713203670. E-mail addresses: [email protected] (M. Tabakov), [email protected] (P. Kozak). http://dx.doi.org/10.1016/j.compbiomed.2014.03.001 0010-4825/& 2014 Elsevier Ltd. All rights reserved.

Patients qualification for treatment with this potent anticancer agent is based on a established procedure, which identifies potential responders to trastuzumab therapy [28]. The first step is based on a semi-quantitative examination of membranous cell staining in tumor cells in immunostained paraffin sections of breast cancer. The results have been categorized in accordance with the following four – grade scale: 0 (no staining), 1þ (incomplete, weak membrane staining regardless of the proportion of tumor cells stained), 2þ (non-uniform complete membrane staining or staining with obvious circumferential distribution in at least 10% of the tumor cells, or intense, complete membrane staining r30% of the invasive tumor cells), 3þ (intense membrane staining in 430% of the invasive tumor cells) [28]. Cases scored 3þ utilizing this method have been qualified for the trastuzumab based therapy, whereas cases classified as 0 and 1þ are not. However, the 2þ results, the so called unequivocal results, require further testing utilizing a costly fluorescent in situ hybridization technique (FISH), which ultimately determines the HER2/neu amplification status [7,28]. Therefore, it is necessary to introduce a less complicated and expensive diagnostic process for correct recognition of the corresponding HER2/neu classes. For solving the above problem, computer image analysis systems have been recently proposed. A review and comparison of subjective and digital image analyses can be found in [10,22]. There are different concepts of image processing and analysis used in the area considered – based on feature extraction and analysis, data clustering or other techniques [1,5,11,13,19]. In general, the HER2/neu classification process can be transformed, in terms of digital image processing, into a problem of cell membrane staining and cell membrane connectivity/completeness recognition. If the cell

20

M. Tabakov, P. Kozak / Computers in Biology and Medicine 49 (2014) 19–29

membrane is strong stained and enough compact, then it is very likely to be a 3þ HER2/neu class. Here occurs the major image analysis problem, which is that these cell membrane characteristics are rather fuzzy term. This has a direct impact on the complexity of the segmentation process, which is a fundamental stage of the interpretation process of cell membrane staining and compactness. Generally, the HER2/neu histopathology image classification consists of the two following main processes: the first stage, due to the interpretation of cell membrane staining and compactness degree, is to segment the corresponding HER2/neu over-expressed cell membranes. Next, shape connectivity analysis must be performed in purpose to classify the corresponding histopathology preparations. The effectiveness of the classification process, should be verified by investigating correlation with the FISH test examination [14]. Nevertheless, there is still lack of standardized HER2/neu computed classification procedure. In this research, we consider the very important problem of accurate recognition of cell membranes with HER2/neu over-expression. In our proposition of cell membrane detection, we use fuzzy decision tree with appropriately defined reasoning method, based on the Takagi–Sugeno inference model. The corresponding image analysis is performed for all pixels of histopathology images and so, a segmentation decision is defined. The article is organized as follows: in Section 2 the problem formulation is given, in Section 3 some theoretical background, considering the methods used in the presented research, is described, in Section 4 the proposed segmentation process is introduced, in Section 5 some experimental results are provided and finally, in Sections 6 and 7, discussion of the results and conclusions are presented. 2. Problem formulation In terms of digital image processing, the HER2/neu classification process is a problem of cell membrane staining and the cell membrane connectivity/closure recognition. Over-expressed cell membrane stain connectivity and closure are analyzed and classified by pathologists and are key elements of differentiating the breast cancer cases between scores þ2 and þ3. This step highly depends on the experience of pathologists and may cause the observed variability of HER2/neu scoring between observers. Accurate classification of cell membrane to 2þ class causes a big problem and these cases are qualified for FISH test examination, other classes are easier for pathologists as well as for automatic classification. Above difficulty arises from fuzzy nature of the cell membrane. Moreover, the HER2/ neu sections classified by the pathologists as þ2 are mostly characterized by the ratio of amplification close to 2 (treatment threshold decision) in the FISH examination [28]. Generally, the HER2/neu histopathology image classification consists of the two following main processes [14,4]:

 segmentation of the corresponding HER2/neu over-expressed cell membranes,

 classification of the histopathology preparations on the basis of shape and connectivity analyses by investigating correlation with the FISH test examination. Therefore, in our research, we consider the very important problem of accurate recognition of cell membranes with HER2/neu overexpression. This is done by analyses of color image features, with respect to appropriately defined fuzzy decision tree and reasoning process. 3. Background In this section, the preliminaries of fuzzy sets [29], fuzzy decision trees [27] and fuzzy control systems of Takagi–Sugeno type [24] are briefly described.

3.1. Fuzzy sets Let X ¼ df.{x1, x2, …, xn}D R be some finite set of elements (domain), then we shall call A the fuzzy subset of X, if and only if: A¼ df.{(x, μA(x))|xA X}, where μA is a function that maps X onto the real unit interval [0,1], i.e. μA: X-[0,1]. The function μA is also known as the membership function of the fuzzy set A, as its values represents the grade of membership of the elements of X to the fuzzy set A. Here the idea is that we can use membership functions, as characteristic functions (any crisp set can be defined by its characteristic function) for fuzzy, imprecisely described sets. Detailed description of the basic operations, defined for fuzzy sets, can be found in the literature [29]. 3.2. Fuzzy decision trees: fuzzy ID3 algorithm Knowledge acquisition from data is very important in knowledge engineering – example, developing of decision support systems, control systems and other. There are some knowledge acquisition methods, but one of the most popular is ID3 algorithm proposed by Quinlan [18,17], which develop a decision tree for classification from symbolic data. The decision tree consists of nodes for testing attributes, edges for branching by values of symbols and leaves representing class names to be classified. Basing on concepts of the information theory, ID3 algorithm can be applied for solving a classification problems over a set of data, and generates a decision tree which minimizes the number of tests for classifying the data. Corresponding adjusted algorithms have been next proposed, which basically assume partitioning of a numerical range of attribute into intervals. Also, some algorithms have been proposed to fuzzify the last intervals. And hence, more flexible and effective decision trees can be obtained (referring to practical problems). A fuzzy extension of the classical ID3 algorithm [27] is used in this research. The obtained algorithm, called fuzzy ID3 algorithm, uses fuzzy sets of data (data with membership grades) and generates a fuzzy decision tree using fuzzy sets (defined by a user or using some data analysis concept) for all attributes. A fuzzy decision tree consists of nodes for testing attributes, edges for branching by test values of fuzzy sets used and leaves with class names and certainties. The fuzzy ID3 algorithm is very similar to ID3, but unlike the ID3, selects the test attributes using membership values for data, not using the information gain which is computed by the probability of data. The formal specification of the fuzzy ID3 algorithm is omitted here. A comprehensive study of the algorithm and corresponding calculation examples are provided in [27]. Below, new method of fuzzy tree reasoning is introduced. The classical fuzzy ID3 algorithm start reasoning from the top node (root) of the fuzzy decision tree. Repeat testing the attribute at the node, branching an edge by its value of the membership function and multiplying these values until the leaf node is reached. Next, the result is multiplied with the proportions of the classes in the leaf node and get the certainties of the classes at this leaf node. The above procedure is repeated until all the leaf nodes are reached and all the certainties are calculated. The major difference in comparison with the classical ID3 approach [27] is that we use the certainties of classes of every tree node for reasoning (not only of the leaf node). And hence, the class certainties of every tree node are used to calculate the parameters of linear functions, required in the Takagi–Sugeno reasoning process. Therefore, each tree branch is interpreted as fuzzy If–then rule, which gives the possibility to aggregate the whole information, generated by the fuzzy tree, with respect to the Takagi–Sugeno fuzzy model.

M. Tabakov, P. Kozak / Computers in Biology and Medicine 49 (2014) 19–29

21

3.3. Fuzzy control of Takagi–Sugeno type

defined by experts for Xi, i.e. fuzzy subsets of Xi, where i¼ ¼ 1, 2, …, n. Then a fuzzy IF–THEN rule antecedent takes the form: (x1 is

We propose to use fuzzy control concept, as a decision mechanism in our method. For any fuzzy controller, all input information are fuzzified and then processed with respect to the assumed knowledge base, inference method and the corresponding defuzzification method. The basics of the fuzzy control systems are shown in Fig. 1. A fuzzy controller is composed of the following four elements:

nÞ 1Þ 2Þ iÞ ) ○ (x2 is V ðX ) ○ ⋯○ (xn is V ðX ), where ( xi is V ðX ) means the V ðX k1 k2 kn ki

 A rule-base (a set of IF–THEN rules), which contains a fuzzy   

logic quantification of the expert’s linguistic description of how to achieve good control. An inference mechanism, which emulates the expert’s decision making in interpreting and applying knowledge. A fuzzification interface, which converts controller inputs into information that the inference mechanism can easily use to activate and apply rules. A defuzzification interface, which converts the conclusions of the inference mechanism into actual inputs for the process.

Next, we will focus on the concept of Takagi–Sugeno type fuzzy rules generation. 3.3.1. Fuzzy rules Let {X1, X2, …, Xn} be a family of finite sets, which defines the corresponding states of primary inputs for a fuzzy control system. ðX i Þ ðX i Þ iÞ Let V ðX i Þ ¼ df.{V ðX 1 , V 2 , …, V k } be a set of all linguistic variables i

External data flow

Fuzzification interface

Device/Application

Inference mechanism

iÞ A V ðX i Þ ), i.e. degree of membership of xi to: V jðX i Þ (xi A Xi; V ðX j

μV ðiÞ ðxi Þ A ½0; 1 (i¼1, 2, …, n; j ¼1, 2, …, ki) and ○A {  ,  }, where j

 is a binary operation over [0,1] (  : [0, 1]2--[0, 1]) which is commutative, associative, monotonic, and has 1 as unit element. Any such operation is called to be a t-norm. The t-norm operator provides the characterization of the AND operator. The dual t-conorm  (called also: s-norm), characterizing the OR operator, is defined in a similar way having 0 as unit element [3]. Without loss of generality, Zadeh’s t-norm (corresponding to the minimum operation) is used below. The conclusion of any fuzzy rule of Takagi–Sugeno type are linear or constant functions, defined with respect to the input values used in the rule antecedent – a typical (simplified) rule of Takagi–Sugeno type has the form: IF ðx is AÞ THEN ðconclusion ¼ f ðxÞ ¼ ax þ bÞ

The complete form of a classical fuzzy Takagi–Sugeno IF–THEN rule, is defined as follows: nÞ 1Þ 2Þ r ¼ df: IF ðx1 is V ðX Þ AND ðx2 is V ðX Þ AND⋯AND ðxn is V ðX Þ THEN k1 k2 kn

n

n

n

i¼1

i¼1

i¼1

Conclusion ¼ f ðx1 ; x2 ; …; xn Þ ¼ ∑ ðai xi þ bi Þ ¼ ∑ ai xi þ ∑ bi ð2Þ The set of all fuzzy rules, defines the knowledge base (denoted below as KB) of the fuzzy system.

Control process

Defuzzification interface

Rule-base Controller

Fig. 1. Conceptual diagram of a fuzzy control system.

3.3.2. Inference mechanism/defuzzification method The final system output value (the decision value, in short FSO), using the Takagi–Sugeno inference model, is defined as follows: FSO ¼ df :

∑r A KB wr f r ∑r A KB wr

ð3Þ

where wr is the antecedent value of the rth rule and fr is function, interpreted as conclusion of the rth rule.

FT = 2.33

FT = 3.47

FT = 2.81

FT = 1.08

FT = 1.28

FT = 0.95

FT = 1.05

FT = 1.45

FT = 4.88

ð1Þ

Fig. 2. Example cell membranes with different staining and compactness.

22

M. Tabakov, P. Kozak / Computers in Biology and Medicine 49 (2014) 19–29

The above fuzzy control concept can be used for image segmentation, by testing the FSO value for every image pixel, with respect to a priori given threshold value. In our work, this concept is used in order to reduce color image to binary image and so, to define segmentation of the HER2/neu over-expressed cancer cell membranes for histopathology images.

4. The proposed segmentation method The aim of our research is to develop segmentation method for histopathology images that extracts HER2/neu over-expressed cancer cell membranes. Only having a sufficiently large number of such cells on a histopathology preparation, justifies the use of therapy with monoclonal antibodies. Therefore, we have a classification problem of histopathology preparations, with respect to cancer cell membrane characteristics. The HER2/neu classification process can be transformed, in terms of digital image processing, into a problem of cell membrane staining (dark/brown colors) and cell membrane compactness/completeness recognition. Some examples of HER2/neu images are given in Fig. 2, as well as the corresponding FISH test (FT) examination values. The FT value is an objective factor, which gives a clear decision if a histopathology preparation represents suitable case for treatment with monoclonal antibodies. If (FTZ2) then a patient should be given the treatment else not (prevent treatment with monoclonal antibodies is not taking into consideration, because of its high cost). To provide accurate histopathology recognition, first we need to segment the appropriate cell membranes. This segmentation is realized using a fuzzy decision tree. Moreover, we propose a new fuzzy decision tree reasoning process, based on the Takagi–Sugeno inference model. Next, we precisely explain the construction of the used fuzzy decision tree. 4.1. Learning set In purpose to build any decision tree, first a learning set (in this case – appropriately constructed decision table) is needed. We assumed in our research that every image pixel is processed. To define the learning set, we generated three image pixel sets: Positive – set of pixels located on the target cell membrane (HER2/neu over-expressed), Negative – set of background pixels (negative examples – not located on the target cell membrane) and Random – randomly selected pixels of any type (positive or negative). The random set was used to introduce common fuzzification over the considered image data. In purpose to fuzzify, generally it is sufficient to introduce three basic linguistic variables: low, medium and high. Each of these fuzzy sets has been defined over corresponding domains which represent color and edge characteristics of image pixels (pixel features). It have been used a 'rich feature' set as a descriptor for any considered image pixel (but limited to color features, originated from the UCI Machine Learning Repository, University of Massachusetts). A detailed description of these image features can be found on the UCI Machine Learning Repository web page1, or in papers that cite this repository [25,15,6]. In addition, we also included image edge information, based on Sobel edge detector and discrete Laplace operator [8]. To define the corresponding membership functions of the introduced fuzzy sets, we assumed Gaussian data distribution for each considered image feature. The Gaussian probability density function was used as membership function of the fuzzy set 1

http://archive.ics.uci.edu/ml/datasets/Image+Segmentation

medium: 2

μmedium ðxÞ ¼ df: e  ðx  x0 Þ

=2s2

;

ð4Þ

where x0 is the expected value and s is the standard deviation. Next, using the μmedium membership function, we can define μlow, μhigh as well: ( 2 2 1  e  ðx  x0 Þ =2s : x o x0 μlow ðxÞ ¼ df : ; ð5Þ 0 : x Z x0 ( μhigh ðxÞ ¼ df :

0 : x r x0 2

1  e  ðx  x0 Þ

=2s2

: x Z x0

:

ð6Þ

As an example, the generated membership functions over the domain of the Sobel operator values are shown below (Fig. 3). Referring to the above fuzzification proposition and pixel sets construction (positive/negative pixel sets), we can define an appropriate decision table. Every row of this table corresponds to a pixel gathered from the positive or the negative pixel sets. Therefore, the value of the decision attribute is also known. A generic table is given below (see Table 1). The columns represent the set of all used linguistic variables defined over the considered image feature domains. The last column gives the value of the decision attribute, i.e. if a certain pixel is located on the HER2/neu over-expressed membrane or not. Using a decision table as constructed above, we are able to define the corresponding fuzzy decision tree through fuzzy ID3 algorithm. 4.2. Fuzzy decision tree reasoning mechanism We use the Takagi–Sugeno inference mechanism as a reasoning method for the generated fuzzy decision tree. In a natural way, each branch of the tree is interpreted as a fuzzy rule. And so, it is trivial to define the fuzzy rules antecedents. But the basic problem here is the determination of appropriate rule conclusions, i.e. corresponding parameters of the used linear functions. For this purpose, the use of the proportions of the classes calculated on every tree node, is proposed below. The value of the positive class of a tree node is used as the slope of the corresponding linear function and the value of the negative class is used as the shift of this function in the negative direction of the functions co domain. Choosing the function parameters in such a way, we use the degree of function slope as a strengthening factor for the final fuzzy system output decision and the degree of function shifting as a weakening factor, respectively. Also, in order not to 'favor' any of the considered image features, we assume that an image feature can occur only once in the corresponding fuzzy decision tree. The following example is helpful in understanding of the proposed reasoning process. Example: Let consider a sample fuzzy decision tree (Fig. 4) assuming that class C1 determines positive decision and class C2 determines negative decision, respectively. Therefore, we have three fuzzy rules of Takagi–Sugeno type, defined as follows: Rule1 ¼ df. IF (x is Feature1_FuzzySubset1) THEN f1(x)¼0.27x 0.73, where xA{pixel values, with respect to image Feature1}, Rule2 ¼ df. IF (x1 is Feature1_FuzzySubset1) AND (x2 is Feature2_ FuzzySubset1) THEN f2(x1, x2)¼0.55  x1  0.45 þ0.85  x2  0.15 ¼ 0.55  x1 þ0.85  x2  0.6, where x1 A {pixel values, with respect to image Feature1} and x2 A {pixel values, with respect to image Feature2}, Rule3 ¼ df. IF (x1 is Feature1_FuzzySubset2) AND (x2 is Feature2_ FuzzySubset2) THEN f2(x1, x2)¼0.55  x1  0.45 þ0.99  x2  0.01 ¼0.55  x1 þ 0.99  x2  0.46, where x1 A {pixel values, with

M. Tabakov, P. Kozak / Computers in Biology and Medicine 49 (2014) 19–29

23

Fig. 3. The generated fuzzification of the EdgeSobel feature. Table 1 A sample row of the proposed decision table. Image-feature1 … Rowk(pixelk) μlow(pixelk), μmedium(pixelk), μhigh(pixelk)

Image-feature2

… Image-featuren

Decision

μlow(pixelk), μmedium(pixelk), μhigh(pixelk)

… μlow(pixelk), μmedium(pixelk), μhigh(pixelk)

{Positive, negative} (depending if the considered pixel is, or is not on the target cell membrane)

…

…

Feature1 Feature1_ FuzzySubset1 C1= 0.27 C2= 0.73 Feature2_ FuzzySubset1 C1= 0.85 C2= 0.15

Feature1_ FuzzySubset2

Feature2 0.55

C1=0.55 C2=0.45

Feature2_ FuzzySubset2 C1= 0.99 C2= 0.01

Fig. 4. Sample fuzzy decision tree

respect to image Feature1} and x2 A {pixel values, with respect to image Feature2}. As an illustration, in Fig. 5, we show a sub tree of the real generated fuzzy decision tree addressing the considered HER2/neu membrane cell segmentation problem. The final cell membrane segmentation decision, i.e. if a pixel represents HER2/neu over-expressed cell membrane or not, is taken under the calculated FSO value (see Eq. (3)) with respect to a given threshold-system parameter.

5. Results The objective of this study was to verify the proposed overexpressed cell membranes segmentation process, on real clinical histopathology data of invasive ductal breast cancer HER2/neu sections scored as 2 þ . These sections pose the main clinical issue in patients selection for trastuzumab administration. What more,

experienced pathologists manually selected two groups of cancer cells – HER2/neu positive (over-expressed) and negative (detailed information on the analyzed data set are given in Table 2). These image data were used for testing of the quality of the proposed process, by corresponding statistical analysis (see point 5.3). As follows from Table 2, the most of the data used (about 88% of the considered HER2/neu images) are derived from histopathology preparations, with FISH test examination values A[1.0, 3.0]. This range is considered as highly complex for histopathology recognition, if the recognition is based on digital image information only. The corresponding histopathology preparations were performed using the Pathways HER2 (4B5) Kit. 5.1. Estimation of the segmentation threshold value The final cell membrane segmentation decision, is taken under the calculated FSO value with respect to a priori given threshold value T. Therefore, the output of the proposed image segmentation process is associated with the decision, if a certain pixel represents HER2/neu over-expressed cell membrane or not, defined by the following rule (for every image pixel p): IF ( FSO(p) ZT) THEN (pixel p is recognized as 'HER2/neu overexpressed' and should be segmented) ELSE (pixel p is not recognized as 'HER2/neu over-expressed' and it is interpreted as a background). It can be noticed, that the automation of the proposed image segmentation of HER2/neu over-expressed cancer cell membranes, depends explicitly from the value of the threshold T. Therefore, we proposed estimation of the threshold used, by the following procedure: (a) First, we selected a reference set of HER2/neu images – images scored by the FISH test examination values in the interval [1.5, 2.5] (about 33% of the all considered histopathology images). This is considered as the 'worst case' in sense of HER2/neu

24

M. Tabakov, P. Kozak / Computers in Biology and Medicine 49 (2014) 19–29

Fig. 5. A sample fragment of the fuzzy decision tree applied in the proposed cell membrane segmentation process.

Table 2 The histopathology data used. Number of considered HER2/neu histopathology images

147

Number of selected cancer cells recognized as HER2/neu over-expressed (on average, 7 cells selected for each considered HER2/neu image) Number of selected cancer cells recognized as not HER2/neu over-expressed (on average, 10 cells selected for each considered HER2/neu image) Range of FISH test examination values of the considered HER2/neu images (i.e. all HER2/neu images were tested with the FISH test) Number of HER2/neu images, with FISH test examination valuesA [1.5, 2.5] Number of selected cancer cells recognized as HER2/neu over-expressed from images with FISH test examination values A [1.5, 2.5] Number of selected cancer cells recognized as not HER2/neu over-expressed from images with FISH test examination values A [1.5, 2.5] Number of HER2/neu images, with FISH test examination valuesA [1.0, 3.0] Number of selected cancer cells recognized as HER2/neu over-expressed from images with FISH test examination values A [1.0, 3.0] Number of selected cancer cells recognized as not HER2/neu over-expressed from images with FISH test examination values A [1.0, 3.0] Number of HER2/neu images, with FISH test examination valuesA [0.84, 1.0) [(3.0, 6.97] Number of selected cancer cells recognized as HER2/neu over-expressed from images with FISH test examination values A [0.84, 1.0) [ (3.0, 6.97] Number of selected cancer cells recognized as not HER2/neu over-expressed from images with FISH test examination values A [0.84, 1.0) [ (3.0, 6.97]

1031 1463 [0.84, 6.97] 48 312 496 129 803 1296 18 228 167

recognition analysis, i.e. if not examined by FISH test, the corresponding HER2/neu preparations are almost indistinguishable for pathologists, (b) From the referenced image set, domain experts manually selected two classes of cancer cells – HER2/neu positive (denoted below as P, |P| ¼312) and HER2/neu negative (denoted below as N, |N| ¼496), (c) Naturally, the optimal segmentation threshold value maximizes the distance between the above two classes of cancer cells, with respect to a given shape coefficient. Higher connectivity/compactness of cell membranes, which can be investigated by membrane shape analysis, indicates higher probability of HER2/neu over-expression. In our preliminary study, we used the box-counting dimension [21] (marked as BCD) and a connectivity coefficient (marked as CC1) as shape recognition coefficients. The box-counting dimension is a way of determining the fractal dimension of a set, defined in certain metric space. In practice, assuming two dimensional

space, it can be used to provide precise numerical characterization of any structure with highly irregular shape. The CC1 is an often used connectivity coefficient, defined as follows: CC1 ¼ df :

L2 4πS

ð7Þ

where L is the considered object circuit and S denotes the object area. In purpose to define the optimal segmentation threshold value, we tested all possible threshold values, assuming normalization into interval [0,1] and quantization of the threshold values with ΔT¼ 0.05 (i.e. the set of all considered threshold values is: T¼ 0, 0.05, 0.1, …, 0.95, 1). Next, for each value, we investigated the difference between the average values of the elements of P and N, with respect to the assumed shape coefficients.

M. Tabakov, P. Kozak / Computers in Biology and Medicine 49 (2014) 19–29

HER2/neu Image fragment (a)

HER2/neu over-expressed cell membranes of (a)

HER2/neu Image fragment (b)

HER2/neu over-expressed cell membranes of (b)

HER2/neu Image fragment (c)

HER2/neu over-expressed cell membranes of (c)

HER2/neu Image fragment (d)

HER2/neu over-expressed cell membranes of (d)

HER2/neu Image fragment (e)

HER2/neu over-expressed cell membranes of (e)

Fig. 6. HER2/neu image fragments and corresponding segmentation of the over-expressed cell membranes, with segmentation threshold used ¼ 0.5.

25

26

M. Tabakov, P. Kozak / Computers in Biology and Medicine 49 (2014) 19–29

More formally, if we consider the relation between the threshold values T and the above mentioned difference value measured for sets P and N, as the function f: T-R, then we define the optimal threshold segmentation value (Toptimal, with respect to any shape coefficient G) as follows: T optimalG ¼ df : f

1



  ∑ Gðpi Þ ∑j Gðnj Þ   max  i ; T jPj jNj 

ð8Þ

where pi A P, i¼ 1, …, |P|, nj A N, j ¼1, …, |N| and G A {CC1, BCD}. For the both shape coefficient used, we reached similar results – optimal threshold valueE0.5 ðT optimal CC1 ¼ 0:5; T optima BCD ¼ 0:55Þ. Therefore, the optimal segmentation threshold value, derived over the assumed set of reference HER2/neu images, was set to 0.5.

5.2. Results visualization Some experimental results concerning the HER2/neu overexpressed cell membrane recognition of cancer cells are shown in Fig. 6. Having regard to the limitations of the size of the histopathology images, only chosen HER2/neu image fragments (cancer cell conglomerations) are showed. The segmentation process was implemented using the Java 7 platform. Fig. 6 presents an example of utilizing the image analysis in selecting the most appropriate cell membranes characterized by an over-expression of HER2/neu, with respect to the proposed segmentation method.

5.3. Results verification

Table 3 Results evaluation with F-score measure and Jaccard–Tanimoto similarity coefficient, concerning the CC1 shape coefficient. T

F-score CC1 (assuming quantization ¼ 0.15)

Jaccard– Tanimoto coefficient

TP

0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5

0.15 0.3 0.45 0.6 0.75 0.9 1.05 1.2 1.35 1.5 1.65 1.8 1.95 2.1 2.25 2.4 2.55 2.7 2.85 3 3.15 3.3 3.45 3.6

0.459 0.465 0.472 0.479 0.479 0.484 0.488 0.496 0.501 0.511 0.51 0.51 0.512 0.52 0.521 0.524 0.524 0.523 0.529 0.533 0.532 0.534 0.537 0.54

1004 1157 306 27 987 1090 373 44 968 1021 442 63 958 967 496 73 939 930 533 92 927 885 578 104 912 839 624 119 903 789 674 128 898 760 703 133 891 713 750 140 871 676 787 160 858 651 812 173 845 618 845 186 834 574 889 197 818 538 925 213 804 504 959 227 792 480 983 239 784 467 996 247 776 437 1026 255 770 413 1050 261 755 387 1076 276 747 367 1096 284 736 339 1124 295 728 316 1147 303

0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5

3.75 3.9 4.05 4.2 4.35 4.5 4.65 4.8 4.95 5.1 5.25 5.4 5.55 5.7 5.85

0.629 0.635 0.641 0.648 0.648 0.652 0.656 0.663 0.668 0.676 0.676 0.676 0.678 0.684 0.685 0.687 0.688 0.687 0.692 0.696 0.695 0.697 0.699 0.702 (best result) 0.697 0.698 0.691 0.689 0.684 0.67 0.662 0.658 0.65 0.641 0.627 0.614 0.599 0.573 0.559

0.535 0.536 0.528 0.526 0.52 0.504 0.495 0.491 0.481 0.472 0.457 0.443 0.428 0.402 0.388

709 699 678 665 649 622 605 590 572 551 526 507 485 451 431

FP

295 274 253 234 217 204 191 171 158 136 120 113 103 91 80

TN

1168 1189 1210 1229 1246 1259 1272 1292 1305 1327 1343 1350 1360 1372 1383

FN

322 332 353 366 382 409 426 441 459 480 505 524 546 580 600

Here, TP is the true positive: the number of HER2/neu over-expressed cancer cells, correctly recognized as HER2/neu over-expressed. FP is the false positive: the number of not HER2/neu over-expressed cancer cells, incorrectly recognized as HER2/neu over-expressed. TN is the true negative: the number of not HER2/neu over-expressed cancer cells, correctly recognized as not HER2/neu over-expressed. FN is the false negative: the number of HER2/neu over-expressed cancer cells, incorrectly recognized as not HER2/neu over-expressed. The number of all input HER2/neu over-expressed cancer cells ¼ TPþ FN ¼ 1031. The number of all input not HER2/neu over-expressed cancer cells ¼ TNþ FP¼ 1463.

If the achieved results should justify the proposed HER2/neu image processing concept, it should be showed that using the above segmentation results, it is possible to propose a numerical coefficient that could be considered as for replacement of the FISH test value. The segmentation process gives an initial recognition of the HER2/neu over-expressed cancer cells. Next, to perform a classification related to the FISH test rating, we propose to use the introduced shape coefficients CC1 and BCD, which affect the cancer cells connectivity.

Table 4 Results evaluation with F-score measure and Jaccard–Tanimoto similarity coefficient, concerning the BCD shape coefficient. T

BCD (assuming F-score quantization ¼ 0.05)

Jaccard– Tanimoto coefficient

TP

FP

0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5

0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4 0.45 0.5 0.55 0.6 0.65 0.7 0.75 0.8 0.85 0.9 0.95 1 1.05 1.1

0.456 0.459 0.461 0.46 0.46 0.466 0.468 0.471 0.473 0.478 0.48 0.483 0.487 0.487 0.493 0.498 0.506 0.513 0.512 0.511 0.517 0.522

1009 1008 1007 1002 1000 995 995 990 987 979 973 968 959 946 938 928 915 901 877 850 838 810

1181 1167 1154 1149 1141 1105 1094 1069 1055 1018 997 973 937 911 873 834 778 727 682 633 589 522

0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5

1.15 1.2 1.25 1.3 1.35 1.4 1.45 1.5 1.55 1.6 1.65 1.7 1.75 1.8

0.519 0.515 0.497 0.465 0.418 0.364 0.302 0.234 0.172 0.122 0.086 0.038 0.011 0.004

786 750 694 626 537 448 355 265 190 132 91 40 11 4

482 425 364 314 253 201 144 101 74 50 30 18 10 4

0.627 0.629 0.631 0.63 0.631 0.636 0.638 0.641 0.642 0.647 0.648 0.651 0.655 0.655 0.66 0.665 0.672 0.678 0.677 0.676 0.682 0.686 (best result) 0.684 0.68 0.664 0.635 0.59 0.533 0.464 0.379 0.293 0.218 0.158 0.073 0.021 0.008

TN

282 296 309 314 322 358 369 394 408 445 466 490 526 552 590 629 685 736 781 830 874 941

FN

22 23 24 29 31 36 36 41 44 52 58 63 72 85 93 103 116 130 154 181 193 221

981 245 1038 281 1099 337 1149 405 1210 494 1262 583 1319 676 1362 766 1389 841 1413 899 1433 940 1445 991 1453 1020 1459 1027

M. Tabakov, P. Kozak / Computers in Biology and Medicine 49 (2014) 19–29

27

Fig. 7. Illustration of the dependence between the values of the CC1 and the F-score measure as well as the Jaccard–Tanimoto coefficient. The achieved system accuracy best result is approximately 70%, with respect to the F-score measure.

Fig. 8. Illustration of the dependence between the values of the BCD and the F-score measure as well as the Jaccard–Tanimoto coefficient. The achieved system accuracy best result is approximately 69%, with respect to the F-score measure.

To verify the stated hypothesis that it is possible to replace the FISH test with a connectivity coefficient, derived over the cell membrane segmentation results, we performed classification test over the whole set of selected cancer cells with fixed segmentation threshold value T¼ 0.5. Next, we verified the test accuracy with the F-score measure [16] (it considers both the precision and the recall of the test to compute the score) and with the Jaccard–Tanimoto

similarity coefficient [12], testing the classification quality with respect to the recognition of HER2/neu over-expressed cancer cells. The corresponding results are presented in Tables 3 and 4, for CC1 and BCD shape coefficients, respectively. The best result was achieved for CC1 ¼3.6, as it is shown in Fig. 7. The best result was achieved for BCD ¼1.1, as it is shown in Fig. 8.

28

M. Tabakov, P. Kozak / Computers in Biology and Medicine 49 (2014) 19–29

6. Discussion The replacement of the expensive FISH test examination procedure with corresponding image analysis, which will be possible only with good recognition of the HER2/neu over-expressed cell membranes, is very important step in the considered research field. The first issue that should be discussed is that in the considered research area, the major problem is the lack of appropriately prepared benchmark HER2/neu image data, which makes hard any comparison analysis of different algorithms introduced into this research field (comparison analysis may give false conclusions). The lack of widely approved benchmark HER2/neu data is a fundamental problem which affects all research work in the field. The reasons might be different, for example: some researchers don't work with images processed by FISH test examination, but use (often as a test data set) the common classification values: 0, 1þ, 2þ and 3þ delivered by physicians. This common classification often is used for defining of training sets used for development of histopathology support systems [26]. But it is very subjective, examples can be shown that such approach is not proper – the main difficulty of the HER2/neu recognition process is that physicians cannot provide absolutely good prepared data. Many images designated as 2þ actually, if apply FISH test examination, will become false negative. The subjectivity of analysis is the second big problem. Actually, the problem of subjectivity, occurs in many different ways – for example physicians use dedicated software for histopathology image processing and analysis [2], but often the program packages require set of parameters. Also there are subjective procedures in the process of image acquisition – images are gathered from large (in sense of digital image information) histopathology preparations. Next, the corresponding preparations are analyzed subjectively (searching for conglomerations of cells or representative cells), which may also provide false recognition – even high incompatibility, comparing to diagnosis based on the FISH test examination. On other hand, processing HER2/neu images with a priori given FISH test examination values, occurs the problem what range of values should be used for research? For example, it is much more difficult to work only with images gathered from histopathology preparations with recognized FISH values close to the value of 2, i.e. A(1, 3) than outside this range. Also, the number of images taken from different ranges, with respect to the corresponding Fish test examination values, is very important. The above mentioned issues are the reason why in our research we aimed to introduce automatic cell membrane segmentation method and to apply it on a large set of HER2/neu images with range, in terms of FISH test examination, close to the value of 2. Considering the performed experiments, better results were obtained using the CC1 shape coefficient (system accuracy best result, with respect to the F-score measure, is approximately 70%) rather than BCD. Also it is clear, that the achieved quality of the results is very dependent from the following two major factors: the preparation of the training set of pixels, required for the process of the fuzzy decision tree generation and the shape descriptors used. Concerning the shape descriptors problem, as further work we intend to investigate more sophisticated shape descriptors in purpose to increase the achieved accuracy. For example, analyzing the provided experiments results, we noticed that the both shape coefficient used, are too sensitive to discontinuity in cell membranes. This disadvantage was confirmed by performing overlap measures, using the Jaccard–Tanimoto coefficient, as it is very sensitive even to small dissimilarities. The providing of better cell membrane shape analyses will be important step in our research, concerning the possibility of replacement of the FISH test examination value with corresponding shape coefficient, applied over HER2/neu image segmentation results. Also, we are going to automate the cell recognition

process, required for membrane connectivity analysis by shape descriptors.

7. Conclusions In this article we propose a histopathology image segmentation method of HER2/neu over-expressed cancer cell membranes. We performed our research on highly complex HER2/neu histopathology image data, in terms of FISH test examination values range. The segmentation process is realized by fuzzy decision tree with appropriately defined reasoning method. The proposed image analysis is critical, concerning the HER2/neu recognition process, as the segmentation results can be used for further classification of the HER2/neu positive instances, by shape coefficients. This may give the possibility to replace the expensive FISH test examination procedure and so facilitate and accelerate the HER2/neu diagnosis process. Nevertheless, there is still the need of introducing appropriate shape descriptors that can improve the cell membrane connectivity analysis. Also, the data analysis process was performed on manually selected cancer cells, as the main research problem concerns the introduction of automatic HER2/neu overexpressed cell membrane segmentation. To complete the process of recognition, as a further work, we will also automate the process of cell extraction. Thus, our current research is concentrated on the developing of an automatic computer system which will take as an input histopathology image itself and as an output gives accurate diagnostic decision – apply trastuzumab treatment or not.

Conflict of interest statement None declared.

Acknowledgments This work was partially supported by Polish Ministry of Science and Higher Education Grant no. N N518 290540, 2011–2014. We would also like to thank the staff of the Department of Histology and Embryology of the Wroclaw Medical University, for the substantive support and for the preparation of the considered data set. References [1] M. Avoni, Image Analysis Methods for Determining HER2neu Status of Breast Cancers (BME M.Sc.), DTU – Technical University of Denmark, Copenhagen, Denmark, in collaboration with Visiopharm A/S, Hørsholm, MATLAB User Conference, Denmark, 2008. [2] M. Braun, R. Kirsten, N.J. Rupp, H. Moch, F. Fend, N. Wernert, G. Kristiansen, S. Perner, Quantification of protein expression in cells and cellular subcompartments on immunohistochemical sections using a computer supported image analysis system, Histol. Histopathol. 28 (5) (2013) 605–610. [3] I.N. Bronstein, K.A. Semendjajew, G. Musiol, H. Mühlig, Taschenbuch der Mathematik, Verlag Harri Deutsch, 2001, p.1258. [4] A. Brügmann, et al., Digital image analysis of membrane connectivity is a robust measure of HER2 immunostains, Breast Cancer Res. Treat. 132 (1) (2012) 41–49. [5] S. Doyle, S.h. Agner, A. Madabhushi, M. Feldman, J. Tomaszewski, Automated grading of breast cancer histopathology using spectral clustering with textural and architectural image features, in: IEEE International Symposium on Biomedical Imaging, Paris, France, 2008. [6] X.Z. Fern, C. Brodley, Cluster ensembles for high dimensional clustering: an empirical study, J. Mach. Learn. Res. (2004). [7] M.A. Gavrielides, H. Masmoudi, N. Petrick, K.J. Myers, S.M. Hewitt, Automated evaluation of HER-2/neu immunohistochemical expression in breast cancer using digital microscopy, in: 5th IEEE International Symposium on Biomedical Imaging From Nano to Macro, 2008, pp. 808–811. [8] R. Gonzalez, R. Woods, Digital Image Processing, Addison Wesley, 2008.

M. Tabakov, P. Kozak / Computers in Biology and Medicine 49 (2014) 19–29

[9] J.N. Goldhirsch, R.D. Ingle, A.S. Gelber, B. Coates, H. Thürlimann, J. SennPanel Members, Thresholds for therapies: highlights of the St Gallen International Expert Consensus on the primary therapy of early breast cancer 2009, Ann. Oncol. 20 (August (8)) (2009) 1319–1329. [10] N.M. Gurcan, E. Laura, A.C. Boucheron, A. Madabhushi, N.M. Rajpoot, Y. Bulent, Histopathological image analysis – a review, IEEE Rev. Biomed. Eng. 2 (2009). [11] A.S. Joshi, G.M. Sharangpani, K. Porter, S. Keyhani, C. Morrison, A.S. Basu, G. A. Gholap, A.S. Gholap, S.H. Barsky, Semi-automated imaging system to quantitate Her-2/neu membrane receptor immunoreactivity in human, Breast Cancer Cytometry A 71 (5) (2007) 273–285. [12] P. Jaccard, The distribution of flora in the alpine zone, New Phytol. 11 (1912) 37–50. [13] Y.-L. Kuo, Ch-Ch Ko, J-Y Lai, Automated assessment in HER-2/neu immunohistochemical expression of breast cancer, in: International Symposium on Computer, Communication, Control and Automation, 3CA, 2010, pp. 585–588. [14] A. Laurinaviciene, D. Dasevicius, V. Ostapenko, S. Jarmalaite, J. Lazutka, A. Laurinavicius, Membrane connectivity estimated by digital image analysis of HER2 immunohistochemistry is concordant with visual scoring and fluorescence in situ hybridization results: algorithm evaluation on breast cancer tissue microarrays, Diagn. Pathol. 6 (2011) 87. [15] A. Likas, N.A. Vlassis, J.J. Verbeek, The global k-means clustering algorithm, Pattern Recognition 36 (2003). [16] D.M.W. Powers, Evaluation: from precision, recall and F-factor to ROC, informedness, markedness & correlation, J. Mach. Learn. Technol. 2 (1) (2011) 37–63. [17] J.R. Quinlan, Discovering rules by induction from large collections of examples, in: D. Michie (Ed.), Expert Systems in the Micro Electronics Age, Edinburgh University Press, 1979. [18] J.R. Quinlan, Induction of decision trees, Mach. Learn. 1 (1986) 81–106. [19] E. Rexhepaj, J.D. Brennan, P. Holloway, W.E. Kay, H.A. McCann, G. Landberg, J. M. MDuffy, K. Jirstrom, M.W. Gallagher, Novel image analysis approach for

[20] [21] [22]

[23]

[24] [25] [26]

[27]

[28]

[29]

29

quantifying expression of nuclear proteins assessed by immunohistochemistry – application to measurement of oestrogen and progesterone receptor levels in breast cancer, Breast Cancer Res. 10 (2008). J.S. Ross, J.A. Fletcher, The HER-2/neu oncogene in breast cancer: prognostic factor, predictive factor, and target for therapy, Stem Cells 16 (1998) 413–428. M. Schroeder, Fractals, Chaos, Power Laws: Minutes from an Infinite Paradise, W.H. Freeman, New York (1991) 41–45. I. Skaland, I. Øvestad, M.A.E. Janssen, J. Klos, H.K. Kjellevold, T. Helliesen, A.P. J. Baak, C.J. Pathol, Comparing subjective and digital image analysis HER2neu expression scores with conventional and modified FISH scores in breast cancer, J. Clin. Pathol. 61 (1) (2008). D.J. Slamon, G.M. Clark, et al., Human breast cancer: correlation of relapse and survival with amplification of the HER2/neu oncogene, Science 235 (1987) 177–182. T. Takagi, M. Sugeno, Fuzzy identification of systems and its applications to modelling and control, IEEE Trans. Syst. Man Cybern. 15 (1) (1985) 116–132. A.K.H. Tung, X. Xu, B.Ch.Ooi. Curler, Finding and visualizing nonlinear correlated clusters, in: SIGMOD Conference, 2005. V.J. Tuominen, T.T. Tolonen, J. Isola, ImmunoMembrane: a publicly available web application for digital image analysis of HER2 immunohistochemistry, Histopathology 60 (2012) 758–767. M. Umanol, H. Okamoto, I. Hatono, H. Tamura, F. Kawachi, S. Umedzu, J. Kinoshita, Fuzzy decision trees by fuzzy ID3 algorithm and its application to diagnosis systems, in: IEEE World Congress on Computational Intelligence. Proceedings of the Third IEEE Conference on Fuzzy Systems, vol. 3, 1994, pp. 2113–2118. A.C. Wolff, M.E.H. Hammond, et al., American Society of Clinical Oncology/ College of American Pathologists guideline recommendations for human epidermal growth factor receptor 2 testing in breast cancer, Arch. Pathol. Lab. Med. 131 (2007) 18–42. L. Zadeh, Fuzzy sets, Inf. Control 8 (3) (1965) 338–353.