- Email: [email protected]
Skew t mixture of experts
Communicated by Prof. Zidong Wang
Accepted Manuscript
Robust mixture of experts modeling using the skew t distribution F. Chamroukhi PII: DOI: Reference:
S0925-2312(17)30894-9 10.1016/j.neucom.2017.05.044 NEUCOM 18462
To appear in:
Neurocomputing
Received date: Revised date: Accepted date:
5 November 2015 30 November 2016 19 May 2017
Please cite this article as: F. Chamroukhi, Robust mixture of experts modeling using the skew t distribution, Neurocomputing (2017), doi: 10.1016/j.neucom.2017.05.044
This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.
ACCEPTED MANUSCRIPT
Robust mixture of experts modeling using the skew t distribution F. Chamroukhia,∗ Normandie Universit´e, UNICAEN, CNRS, Laboratoire de Math´ematiques Nicolas Oresme - LMNO, 14000 Caen, France
CR IP T
a
Abstract
PT
ED
M
AN US
Mixture of Experts (MoE) is a popular framework in the fields of statistics and machine learning for modeling heterogeneity in data for regression, classification and clustering. MoE for continuous data are usually based on the normal distribution. However, it is known that for data with asymmetric behavior, heavy tails and atypical observations, the use of the normal distribution is unsuitable. We introduce a new robust non-normal mixture of experts modeling using the skew t distribution. The proposed skew t mixture of experts, named STMoE, handles these issues of the normal mixtures experts regarding possibly skewed, heavy-tailed and noisy data. We develop a dedicated expectation conditional maximization (ECM) algorithm to estimate the model parameters by monotonically maximizing the observed data log-likelihood. We describe how the presented model can be used in prediction and in model-based clustering of regression data. Numerical experiments carried out on simulated data show the effectiveness and the robustness of the proposed model in fitting non-linear regression functions as well as in model-based clustering. Then, the proposed model is applied to the real-world data of tone perception for musical data analysis, and the one of temperature anomalies for the analysis of climate change data. The obtained results confirm the usefulness of the model for practical data analysis applications.
CE
Keywords: mixture of experts, skew t distribution; EM algorithm; ECM algorithm; non-linear regression; model-based clustering. 1. Introduction
AC
Mixture of Experts (MoE) (Jacobs et al., 1991) is a popular framework in the statistics and machine learning fields for modeling heterogeneity in data for regression, classification and clustering. They consist in a fully conditional mixture model where both ∗
Corresponding author: Faicel Chamroukhi Universit´e de Caen-Normandie, LMNO, UMR CNRS 6139 Campus 2, Bvd Mar´echal Juin, 14032 Caen Cedex, France Tel: +33(0) 2 31 56 73 67 Fax: +33(0) 2 31 56 73 20 Email address: [email protected] (F. Chamroukhi ) Preprint submitted to Neurocomputing - Elsevier
July 4, 2017
ACCEPTED MANUSCRIPT
AC
CE
PT
ED
M
AN US
CR IP T
the mixing proportions, known as the gating functions, and the component densities, known as the experts, are conditional on some input covariates. MoE have been investigated, in their simple form, as well as in their hierarchical form Jordan and Jacobs (1994) (e.g Section 5.12 of McLachlan and Peel. (2000)) for regression and model-based cluster and discriminant analyses and in different application domains. MoE Have also been investigated for rank data Gormley and Murphy (2008) and network data Gormley and Murphy (2010) with social science applications. A survey on the topic can be found in Gormley and Murphy (2011). A complete review of the MoE models can be found in Yuksel et al. (2012). MoE for continuous data are usually based on the normal distribution. Along this paper, we will call the MoE using the normal distribution the normal mixture of experts, abbreviated as NMoE. However, it is well-known that the normal distribution is sensitive to outliers. Moreover, for a set of data containing a group or groups of observations with heavy tails or asymmetric behavior, the use of normal experts may be unsuitable and can unduly affect the fit of the MoE model. In this paper, we attempt to overcome these limitations in MoE by proposing a more adapted and robust mixture of experts model which can deal with possibly skewed, heavy-tailed data and with outliers. Recently, the problem of sensitivity of NMoE to outliers have been considered by Nguyen and McLachlan (2016) where the authors proposed a Laplace mixture of linear experts (LMoLE) for a robust modeling of non-linear regression data. The model parameters are estimated by maximizing the observed-data likelihood via a minorizationmaximization (MM) algorithm. Here, we propose an alternative MoE model, by relaying on other non-normal distribution that generalizes the normal distribution, that is, the skew-t distribution introduced quite recently by Azzalini and Capitanio (2003). We call the proposed MoE model the skew-t mixture of experts (STMoE). One may use the t distribution, as in the t mixture of experts (TMoE) proposed by Chamroukhi (2015, 2016) which provides a natural robust extension of the normal distribution to model data with more heavy tails and to deal with possible outliers. The robustness of the t distribution may however be not sufficient in the presence of asymmetric observations. In mixture modeling, to deal with this issue regarding skewed data, Lin et al. (2007a) proposed the univariate skew-t mixture model which allows for accommodation of both skewness and thick tails in the data, by relying on the skew-t distribution Azzalini and Capitanio (2003). For the general multivariate case using skew-t mixtures, one can refer to Pyne et al. (2009), (Lin, 2010), Lee and McLachlan (2013a), Lee and McLachlan (2013b), Lee and McLachlan (2014), and recently, the unifying framework for previous restricted and unrestricted skew-t mixtures, using the CFUST distribution Lee and McLachlan (2015). The inference in the previously described approaches is performed by maximum likelihood estimation via the expectation-maximization (EM) algorithm or its extensions (Dempster et al., 1977; McLachlan and Krishnan, 2008), in particular the expectation conditional maximization (ECM) algorithm (Meng and Rubin, 1993). Fr¨ uhwirthSchnatter and Pyne (2010) have also considered the Bayesian inference framework for namely the skew-t mixtures. For the regression context, the robust modeling of regression data has been studied 2
ACCEPTED MANUSCRIPT
AC
CE
PT
ED
M
AN US
CR IP T
namely by Bai et al. (2012); Wei (2012); Ingrassia et al. (2012) who considered a mixture of linear regressions using the t distribution. In the same context of regression, Song et al. (2014) proposed the mixture of Laplace regressions, which has been then extended by Nguyen and McLachlan (2016) to the case of mixture of experts, by introducing the Laplace mixture of linear experts (LMoLE). Recently, Zeller et al. (2015) introduced the scale mixtures of skew-normal distributions for robust mixture regressions. However, unlike our proposed STMoE model, the regression mixture models of Wei (2012), Bai et al. (2012), Ingrassia et al. (2012), Song et al. (2014), Zeller et al. (2015) do not consider conditional mixing proportions, that is, mixing proportions depending on some input variables, as in the case of mixture of experts, which we investigate here. In addition, the approaches of Wei (2012), Bai et al. (2012), Ingrassia et al. (2012) and Song et al. (2014) do not consider both the problem of robustness to outliers together with the one of dealing with possibly asymmetric data. Here we consider the mixture of experts framework for non-linear regression problems and model-based clustering of regression data, and we attempt to overcome the limitations of the NMoE model for dealing with asymmetric, heavy-tailed data and which may contain outliers. We investigate the use of the skew t distribution for the experts, rather than the commonly used normal distribution. We propose the skew-t mixture of experts (STMoE) model which allows for accommodation of both skewness and heavy tails in the data and which is obust to outliers. This model corresponds to an extension of the unconditional skew t mixture model (Lin et al., 2007a), to the mixture of experts (MoE) framework, where the mixture means are regression functions and the mixing proportions are also covariate-varying. For the model inference, we develop a dedicated expectation conditional maximization (ECM) algorithm to estimate the model parameters by monotonically maximizing the observed data log-likelihood. The expectation-maximization algorithm and its extensions (Dempster et al., 1977; McLachlan and Krishnan, 2008) are indeed very popular and successful estimation algorithms for mixture models in general and for mixture of experts in particular. Moreover, the EM algorithm for MoE has been shown by Ng and McLachlan (2004) to be monotonically maximizing the MoE likelihood. The authors have showed that the EM (with Iteratively Reweighted Least Squares (IRLS) in this case) algorithm has stable convergence and the log-likelihood is monotonically increasing when a learning rate smaller than one is adopted for the IRLS procedure within the M-step of the EM algorithm. They have further proposed an expectation conditional maximization (ECM) algorithm to train MoE, which also has desirable numerical properties. The MoE has also been considered in the Bayesian framework, for example one can cite the Bayesian MoE Waterhouse et al. (1996); Waterhouse (1997) and the Bayesian hierarchical MoE Bishop and Svens´en (2003). Beyond the Bayesian parametric framework, the MoE models have also been investigated within the Bayesian non-parametric framework. We cite for example the Bayesian non-parametric MoE model (Rasmussen and Ghahramani, 2001) and the Bayesian non-parametric hierarchical MoE approach of J. Q. Shi and Titterington (2005) using Gaussian Processes experts for regression. For further models on mixture of experts for regression, the reader can be referred to for 3
ACCEPTED MANUSCRIPT
2. Mixture of experts for continuous data
CR IP T
example the book of Shi and Choi (2011). In this paper, we investigate semi-parametric models under the maximum likelihood estimation framework. The remainder of this paper is organized as follows. In Section 2 we briefly recall the normal MoE framework. In Section Then, in Section 3, we present the STMoE model and in Section 4 the parameter estimation technique using the ECM algorithm. We then investigate in Section 5 the use of the proposed model for non-linear regression and for prediction. We also show in Section 6 how the model can be used in a modelbased clustering prospective. In Section 7, we discuss the model selection. Section 8 is dedicated to the experimental study to assess the proposed model. Finally, in Section 9, conclusions are drawn and we open a future work.
AC
CE
PT
ED
M
AN US
Mixtures of experts (Jacobs et al., 1991; Jordan and Jacobs, 1994) are used in a variety of contexts including regression, classification and clustering. Here we consider the MoE framework for fitting (non-linear) regression functions and clustering of univariate continuous data . The aim of regression is to explore the relationship of an observed random variable Y given a covariate vector X ∈ Rp via conditional density functions for Y |X = x of the form f (y|x), rather than only exploring the unconditional distribution of Y . Thanks to their great flexibility, mixture models (McLachlan and Peel., 2000) has took much attention for non-linear regression problems and we distinguish in particular the classical mixture of regressions model Quandt (1972); Quandt and Ramsey (1978); Veaux (1989); Jones and McLachlan (1992); Gaffney and Smyth (1999); Viele and Tong (2002); Faria and Soromenho (2010); Hunter and Young (2012) and mixture of experts for regression analysis (Jacobs et al., 1991; Jordan and Jacobs, 1994; Young and Hunter, 2010). The univariate mixture of regressions model assumes that the observed pairs of data (x, y) where y ∈ R is the response for some covariate x ∈ Rp , are generated from K regression functions and are governed by a hidden categorical random variable Z indicating from which component each observation is generated. Thus, the mixture of regressions model decomposes the nonlinear regression model density f (y|x) into a convex weighted sum of K regression component models fk (y|x) and can be defined as follows: f (y|x; Ψ ) =
K X
πk fk (y|x; Ψ k )
(1)
k=1
where the πk ’s defined by πk = P(Z = k) and represent the non-negative mixing proportions that sum to 1. The model parameter vector is given by Ψ = (π1 , . . . , πK−1 , Ψ T1 , . . . , Ψ TK )T , Ψ k being the parameter vector of the kth component density.
2.1. The mixture of experts (MoE) model Although similar, the mixture of experts (Jacobs et al., 1991) differ from regression mixture models in many aspects. One of the main differences is that the MoE model 4
ACCEPTED MANUSCRIPT
CR IP T
consists in a fully conditional mixture while in the regression mixture, only the component densities are conditional. Indeed, the mixing proportions are constant for the regression mixture, while in the MoE, they are modeled as a function of the inputs, generally modeled by logistic or a softmax function. Mixture of experts (MoE) for regression analysis (Jacobs et al., 1991; Jordan and Jacobs, 1994) extend the model (1) by modeling the mixing proportions as function of some covariates r ∈ Rq . The mixing proportions, known as the gating functions in the context of MoE, are modeled by the multinomial logistic model and are defined by: exp (αTk r) πk (r; α) = P(Z = k|r; α) = PK T `=1 exp (α` r)
(2)
AN US
where r ∈ Rq is a covariate vector, αk is the q-dimensional coefficients vector associated with r and α = (αT1 , . . . , αTK−1 )T is the parameter vector of the logistic model, with αK being the null vector. Thus, the MoE model consists in a fully conditional mixture model where both the mixing proportions (the gating functions) and the component densities (the experts) are conditional on predictors (respectively r and x).
M
2.2. The normal mixture of experts (NMoE) model and maximum likelihood estimation In the case of mixture of experts for regression, it is usually assumed that the experts are normal, that is, follow a normal distribution. A K-component normal mixture of experts (NMoE) (K > 1) has the following formulation: f (y|r, x; Ψ ) =
K X
πk (r; α)N y; µ(x; β k ), σk2
ED
k=1
(3)
AC
CE
PT
which involves, in the semi-parametric case, component means defined as parametric (non-)linear regression functions µ(x; β k ). The NMoE model parameters are estimated by maximizing the observed data loglikelihood by using the EM algorithm (Dempster et al., 1977; Jacobs et al., 1991; Jordan and Jacobs, 1994; Jordan and Xu, 1995; Ng and McLachlan, 2004; McLachlan and Krishnan, 2008). Suppose we observe an i.i.d sample of n individuals (y1 , . . . , yn ) with their respective associated covariates (x1 , . . . , xn ) and (r 1 , . . . , xr ). Then, under the NMoE model, the observed data log-likelihood for the parameter vector Ψ is given by: log L(Ψ ) =
n X i=1
log
K X k=1
πk (r i ; α)N yi ; µ(x; β k ), σk2 .
(4)
The E-Step at the mth iteration of the EM algorithm for the NMoE model requires the calculation of the following posterior probability that the individual (yi , xi , r i ) belongs to expert k, given a parameter estimation Ψ (m) : (m) (m) πk (r; α(m) )N yi ; µk (xi ; βk ), σk2 (m) . (5) τik = P(Zi = k|yi , xi , r i ; Ψ (m) ) = f (yi |r i , xi ; Ψ (m) ) 5
ACCEPTED MANUSCRIPT
Then, the M-step calculates the parameter update Ψ (m+1) by maximizing the well-known Q-function (the expected complete-data log-likelihood), that is: Ψ (m+1) = arg max Q(Ψ ; Ψ (m) )
(6)
Ψ ∈Ω
µ(x; β k ) = β Tk x,
CR IP T
where Ω is the parameter space. For example, in the case of normal mixture of linear experts (NMoLE) where each expert’s mean has the flowing linear form: (7)
where β k ∈ Rp is the vector of regression coefficients of component k, the updates for each of the expert component parameters consist in analytically solving a weighted Gaussian linear regression problem and are given by: =
n hX i=1
σk2
(m+1)
=
Pn
(m) τik xi xTi
n i−1 X
(q)
τik yi xi ,
AN US
(m+1) βk
(m)
i=1 τik
i=1
yi − β Tk Pn (m) i=1 τik
(m+1)
xi
2
·
(8)
(9)
CE
PT
ED
M
For the mixing proportions, the parameter update α(m+1) cannot however be obtained in a closed form. It is calculated by IRLS (Jacobs et al., 1991; Jordan and Jacobs, 1994; Chen et al., 1999; Green, 1984; Chamroukhi et al., 2009a). However, the normal distribution is not adapted to deal with asymmetric and heavy tailed data. It is also known that the normal distribution is sensitive to outliers. In the proposal, we address these issues regarding the skewness, heavy tails and atypical observations in the data, by proposing a robust MoE modeling by using the skewt distribution, recently introduced by Azzalini and Capitanio (2003), for the expert components rather than the usually used normal one. The proposed skew t mixture of experts (STMoE) allows for simultaneously accommodating asymmetry and heavy tails in the data and is also robust to outliers. 3. The skew t mixture of experts (STMoE) model
AC
The proposed skew t mixture of experts (STMoE) model is a MoE model in which the expert components have a skew-t density, rather than the standard normal one as in the NMoE model. The skew-t distribution Azzalini and Capitanio (2003), which is a robust generalization the skew-normal distribution (Azzalini, 1985, 1986), as well as its stochastic and hierarchical representations, which will be used to define the proposed STMoE model, are recalled in the following section. 3.1. The skew t distribution Let us denote by tν (.) and Tν (.) respectively the pdf and cdf of the standard t distribution with degrees of freedom ν. The skew t distribution, introduced by Azzalini 6
ACCEPTED MANUSCRIPT
and Capitanio (2003), can be characterized as follows. Let U be an univariate random variable with a standard skew-normal distribution U ∼ SN(0, 1, λ) (which can be shortened as U ∼ SN(λ)) with pdf given by (A.1)). The skew-normal distribution is recalled in Appendix A. Then, let W be an univariate random variable independent of U and following the Gamma distribution, that is, W ∼ Gamma( ν2 , ν2 ). A random variable Y having the following representation: (10)
CR IP T
U Y = µ + σ√ W
follows the skew t distribution ST(µ, σ 2 , λ, ν) with location parameter µ, scale parameter σ, skewness parameter λ and degrees of freedom ν, whose density is defined by: ! s 2 ν + 1 (11) f (y; µ, σ 2 , λ, ν) = tν (dy ) Tν+1 λ dy σ ν + d2y
AC
CE
PT
ED
M
AN US
where dy = y−µ . From the hierarchical distribution of the skew-normal (A.3), a further σ hierarchical representation of the stochastic representation (10) of the skew t distribution is given by: 1 − δ2 2 σ , Yi |ui , wi ∼ N µ + δ|ui |, wi σ2 Ui |wi ∼ N(0, ), (12) wi ν ν Wi ∼ Gamma , . 2 2 3.2. The skew t mixture of experts (STMoE) model The skew proposed t mixture of experts (STMoE) model extends the skew t mixture model, which was first introduced by Lin et al. (2007a), to the MoE framework. In the skew-t mixture model of Lin et al. (2007a), the mixing proportions and the components means are constant, that is, they are not predictor-depending. In the proposed STMoE, however, we consider skew-t expert components in which both the mixing proportions and the mixture component means are predictor-depending. More specifically, we use polynomial regressors for the components, as well as multinomial logistic regressors for the mixing proportions. A K-component mixture of skew t experts (STMoE) is therefore defined by: f (y|r, x; Ψ ) =
K X
πk (r; α) ST(y; µ(x; β k ), σk2 , λk , νk )·
(13)
k=1
The parameter vector of the STMoE model is Ψ = (αT1 , . . . , αTK−1 , Ψ T1 , . . . , Ψ TK )T where Ψ k = (β Tk , σk2 , λk , νk )T is the parameter vector for the kth skew t expert component whose density is defined by ! s 2 ν + 1 f y|x; µ(x; β k ), σ 2 , λ, ν = tν (dy (x)) Tν+1 λ dy (x) (14) σ ν + d2y (x) 7
ACCEPTED MANUSCRIPT
CR IP T
k) where dy (x) = y−µ(x;β · σ It can be seen that, when the robustness parameter νk → ∞ for each k, the STMoE model (13) reduces to a skew-normal mixture of experts model (SNMoE) (see Chamroukhi (2015)). On the other hand, if the skewness parameter λk = 0 for each k, the STMoE model reduces to the t mixture of experts model (TMoE) (e.g., see Chamroukhi (2015, 2016)). Moreover, when νk → ∞ and λk = 0 for each k, the STMoE approaches the standard NMoE model (3). This therefore makes the STMoE very flexible as it generalizes the previously described MoE models to accommodate situations with asymmetry, heavy tails, and outliers.
ED
M
AN US
3.3. Hierarchical representation of the STMoE model By introducing the binary latent component-indicators Zik such that Zik = 1 iff Zi = k, Zi being the hidden class label of the ith observation, a hierarchical model for the STMoE model can be derived as follows. From the hierarchical representation (12) of the skew t distribution, a hierarchical model for the proposed STMoE model (13) can be derived from its stochastic representation (B.2) given in Appendix B, and is as follows: 1 − δk2 2 σ , Yi |ui , wi , Zik = 1, xi ∼ N µ(xi ; β k ) + δk |ui |, wi k σk2 , (15) Ui |wi , Zik = 1 ∼ N 0, wi ν ν k k , Wi |Zik = 1 ∼ Gamma 2 2 Z i |r i ∼ Mult 1; π1 (r i ; α), . . . , πK (r i ; α) .
PT
The variables Ui and Wi are treated as hidden in this hierarchical representation, which facilitates the inference scheme and will be used to derive the maximum likelihood estimation of the STMoE model parameters Ψ by using the ECM algorithm.
AC
CE
3.4. Identifiability of the STMoE model Jiang and Tanner (1999) have established that ordered, initialized, and irreducible MoEs are identifiable. Ordered implies that there exist a certain ordering relationship on the experts parameters Ψ k such that (αT1 , Ψ T1 )T ≺ . . . ≺ (αTK , Ψ TK )T ; initialized implies that wK , the parameter vector of the Kth logistic proportion, is the null vector, and irreducible implies that Ψ k 6= Ψ k0 for any k 6= k0. For the proposed STMoE, ordered implies that there exist a certain ordering relationship such that (β T1 , σ12 , λ1 , ν1 )T ≺ . . . ≺ 2 (β TK , σK , λK , νK )T ; initialized implies that wK is the null vector, as assumed in the model, and finally, irreducible implies that if k 6= k0, then one of the following conditions holds: β k 6= β k0 , σk 6= σk0 , λk 6= λk0 or νk 6= νk0 . Then, we can establish the identifiability of ordered and initialized irreducible STMoE models by applying Lemma 2 of Jiang and Tanner (1999), which requires the validation of the following nondegeneracy condition. The 2 set {ST(y; µ(x; β 1 ), σ12 , λ1 , ν1 ), . . . , ST(y; µ(x; β 4K ), σ4K , λ4K , ν4K )} contains 4K linearly independent functions of y, for any 4K distinct quadruplet (µ(x; β k ), σk2 , λk , νk ) for 8
ACCEPTED MANUSCRIPT
k = 1, . . . , 4K. Thus, via Lemma 2 of Jiang and Tanner (1999) we have any ordered and initialized irreducible STMoE is identifiable. 4. Maximum likelihood estimation of the STMoE model
log L(Ψ ) =
n X
log
i=1
K X
CR IP T
The unknown parameter vector Ψ of the STMoE model is estimated by maximizing the following observed-data log-likelihood given an observed i.i.d sample of n observations, that is, the responses (y1 , . . . , yn ) and the corresponding predictors (x1 , . . . , xn ) and (r 1 , . . . , r n ): πk (r i ; α)ST(y; µ(xi ; β k ), σk2 , λk , νk )·
k=1
(16)
log Lc (Ψ ) =
n X K X i=1 k=1
AN US
We perform this iteratively by a dedicated ECM algorithm. The complete data consist of the observations as well as the latent variables (u1 , . . . , un ) and (w1 , . . . , wn ), and the latent component labels (z1 , . . . , zn ). Then, from the hierarchical representation of the STMoE (15), the complete-data log-likelihood of Ψ is given by: h Zik log (P (Zi = k|r i )) + log (f (wi |Zik = 1)) +
M
log (f (ui |wi , Zik = 1)) + log (f (yi |ui , Zik = 1, xi )) = log L1c (α) +
K X
ED
k=1
log L2c (θ k ) + log L3c (νk )
i
(17)
where θ k = (β Tk , σk2 , λk )T and
log L2c (θ k ) =
Zik log πk (r i ; α),
PT
log L1c (α) =
n X K X
i=1 k=1 n h X
CE
i=1
Zik − log(2π) − log(σk2 ) −
log L3c (νk ) =
n X
AC
i=1
i wi d2ik wi ui δk dik wi u2i 1 log(1 − δk2 ) − + − , 2 2(1 − δk2 ) (1 − δk2 )σk 2(1 − δk2 )σk2
h ν ν ν ν ν i k k k k k Zik − log Γ + log + log(wi ) − wi . 2 2 2 2 2
4.1. The ECM algorithm for the STMoE model The ECM algorithm for the STMoE model starts with an initial parameter vector (0) Ψ and alternates between the E- and CM- steps until convergence.
9
ACCEPTED MANUSCRIPT
4.2. E-Step The E-Step of the CEM algorithm for the STMoE calculates the Q-function, that is the conditional expectation of the complete-data log-likelihood (17), given the observed data {yi , xi , r i }ni=1 and a current parameter estimation Ψ (m) , m being the current iteration. From (17), the Q-function is given by: K h X k=1
i Q2 (θ k , Ψ (m) ) + Q3 (νk , Ψ (m) ) ,
where ) =
n X K X
(m)
τik log πk (r i ; α),
i=1 k=1
Q2 (θ k ; Ψ
(m)
) =
n X
(m) τik
i=1
Q3 (νk ; Ψ (m) ) =
n X
"
log(σk2 )
− log(2π) −
# (m) (m) (m) δk dik e1,ik e2,ik wik d2ik 1 2 + − , − log(1 − δk ) − 2 2(1 − δk2 ) (1 − δk2 )σk 2(1 − δk2 )σk2
AN US
Q1 (α; Ψ
(m)
(18)
CR IP T
Q(Ψ ; Ψ (m) ) = Q1 (α; Ψ (m) ) +
h ν ν ν ν ν i (m) (m) k k k k k − log Γ + log − wik + e3,ik · 2 2 2 2 2
(m)
τik
i=1
It can be seen that computing the Q-function requires the following conditional expectations: (m)
= EΨ (m) [Zik |yi , xi , r i ] ,
τik
(m) wik (m) e1,ik (m) e2,ik (m) e3,ik
M
= EΨ (m) [Wi |yi , Zik = 1, xi , r i ] ,
ED
= EΨ (m) [Wi Ui |yi , Zik = 1, xi , r i ] , = EΨ (m) Wi Ui2 |yi , Zik = 1, xi , r i ,
= EΨ (m) [log(Wi )|yi , Zik = 1, xi , r i ] ·
CE
PT
Following the expressions of these conditional expectations given namely in the case of the standard skew t mixture model (Lin et al., 2007a), the conditional expectations for the case of the proposed STMoE model can be expressed similarly as: (m)
AC
τik
(m)
(m)
wik
(m)
where Mik = λk
(m) e1,ik
=
(m) δk
=
(m) 2(m) (m) (m) πk (r; α(m) ) ST yi ; µ(xi ; β k ), σk , λk , νk
=
(m)
dik
yi − µk (xi ; β
r
(m)
f (yi |r i , xi ; Ψ (m) ) r (m) νk +3 (m) ! Tν (m) +3 Mik (m) (m) νk +1 νk + 1 k × , (m) (m) (m) νk + d2ik Tν (m) +1 Mik
,
(19)
(20)
k
(m) νk +1 (m) (m) νk +d2ik
)
(m) wik
+
,
"
q (m) 1 − δk2
πf (yi |r i , xi ; Ψ (m) )
10
(m)
(m) d2ik (m) (m) νk (1 − δk2 )
+1
!−( νk2
+1) #
(21),
ACCEPTED MANUSCRIPT
=
(m) δk2
(m)
+
(m) e3,ik
δk
=
yi − µk (xi ; β
(m)
" 2 (m) (m) (m) ) wik + 1 − δk2 σk2
q (m) yi − µk (xi ; β (m) ) 1 − δk2
(m) wik
πf (yi |r i , xi ; Ψ (m) ) (m)
− log
νk
+ d2ik 2
(m)
!
×
(m)
−
νk (m)
νk
(m)
(m) d2ik (m) (m) νk (1 − δk2 )
+1
+ d2ik
(m)
+1
!
(m)
+ψ
(m)
νk
(m) tν (m) +1 Mik d2ik −1 k · + r × 3 (m) (m) (m) (m) M T (m) ik νk +1 νk + 1 νk + d2ik (m) (m)
+1) #
, (22)
2
+1
!
(23)
AN US
λk dik
!−( νk2
CR IP T
(m) e2,ik
M
We note that, for (23), we adopted a one-step-late (OSL) approach to compute the (m) conditional expectation e3,ik as described in Lee and McLachlan (2014), by setting the integral part in the expression of the corresponding conditional expectation given in (Lin et al., 2007a) to zero, rather than using a Monte Carlo approximation. We also mention that, for the multivariate skew t mixture models, recently Lee and McLachlan (2015) presented a series-based truncation approach, which exploits an exact representation of this conditional expectation and which can also be used in place of (23).
CE
PT
ED
4.3. M-Step The M-step maximizes the Q-function (18) with respect to Ψ and provides the parameter vector update Ψ (m+1) . From (18), it can be seen that the maximization of Q can be performed by separately maximizing Q1 with respect to the parameters α of the mixing proportions, and for each expert k (k = 1, . . . , K), Q2 with respect to (β Tk , σk2 )T and λk , and Q3 with respect to νk . The maximization of Q2 and Q3 is carried out by conditional maximization (CM) steps by updating (β k , σk2 ) and then updating (λ, νk ) with the given updated parameters. This leads to the following CM steps. On the (m + 1)th iteration of the M-step, the STMoE model parameters are updated as follows.
AC
CM-Step 1. Calculate α(m+1) maximizing the function Q1 (α; Ψ (m) ): α(m+1) = arg max Q1 (α; Ψ (m) ). α
(24)
Contrarily to the case of the standard mixture model and mixture of regression models, this maximization in the case of the proposed STMoE does not exist in closed form. It is performed iteratively by Iteratively Reweighted Least Squares (IRLS).
11
ACCEPTED MANUSCRIPT
The Iteratively Reweighted Least Squares (IRLS) algorithm:. The IRLS algorithm is used to maximize Q1 (α, Ψ (m) ) with respect to the parameter vector α of the softmax function in the M step at each iteration m of the ECM algorithm. The IRLS is a Newton-Raphson algorithm, which consists in starting with a vector α(0) , and, at the l + 1 iteration, updating the estimation of α as follows: (l)
=α
(m)
(m)
2
h ∂ 2 Q (α, Ψ (m) ) i−1 ∂Q1 (α, Ψ (m) ) 1 − ∂α∂αT ∂α α=α(l) α=α(l)
(25)
CR IP T
α
(l+1)
) ) 1 (α,Ψ are respectively the Hessian matrix and the gradient and ∂Q1 (α,Ψ where ∂ Q∂α∂α T ∂α vector of Q1 (α, Ψ(m) ). At each IRLS iteration the Hessian and the gradient are evaluated at α = α(l) and are computed similarly as in Chamroukhi et al. (2009b). The parameter update α(m+1) is taken at convergence of the IRLS algorithm (25). Then, for k = 1 . . . , K, T (m+1)
(m+1)
=
n hX
(q)
i=1
(m+1)
=
n i−1 X i=1
(m)
i=1 τik
(q)
τik
(m)
(m) (m+1)
wik yi − e1,ik δk
xi ,
(26)
h 2 i (m+1) (m+1) (m) (m) (m+1) (m) xi − 2δk xi ) + e2,ik e1,ik (yi − β Tk wik y i − β Tk (· 27) (m) (m) Pn 2 1 − δk2 i=1 τik
ED
σk2
Pn
(m)
τik wik xi xTi
M
(m+1)
βk
AN US
CM-Step 2. Calculate (β k , σk2 )T by maximizing Q2 (θ k ; Ψ (m) ) w.r.t (β Tk , σk2 )T . For the skew t mixture of linear experts (STMoLE) case, where the expert means are linear regressors, that is, of the form (7), this maximization can be performed in a closed form and provides the following updates:
n X
(m) τik
+ (1 +
CE
δk (1 −
δk2 )
PT
CM-Step 3. The skewness parameters λk are updated by maximizing Q2 (θ k ; Ψ (m) ) w.r.t (m+1) (m+1) λk , with β k and σk2 fixed at the update β k and σk2 , respectively. It can be easily (m+1) shown that the maximization to obtain δk (k = 1, . . . , K) consists in solving the following equation in δk :
i=1
δk2 )
(m+1) (m) e1,ik (m) dik τik (m+1) σk i=1
n X
− δk
n X
(m) τik
i=1
h (m+1) (m) wik d2ik +
(m) e2,ik i
σk2
(m+1)
= 0·(28)
AC
CM-Step 4. Similarly, the degrees of freedom νk are updated by maximizing Q3 (νk ; Ψ (m) ) (m+1) (m+1) (m+1) w.r.t νk with β k and σk2 fixed at β k and σk2 , respectively. An update νk is calculated as solution of the following equation in νk : Pn (m) (m) (m) ν ν e3,ik − wik i=1 τik k k −ψ = 0. (29) + log +1+ Pn (m) 2 2 τik i=1
The two scalar non-linear equations (28) and (29) can be solved similarly as in the TMoE model, that is with a root finding algorithm, such as Brent’s method (Brent, 1973). 12
ACCEPTED MANUSCRIPT
M
5. Prediction using the STMoE
AN US
CR IP T
As mentioned before, one can see that, when the robustness parameter νk → ∞ for all the components, the parameter updates for the STMoE model correspond to those of the SNMoE model (see Chamroukhi (2015)). On the other hand, when the skewness parameters λk = 0, the STMoE parameter updates correspond to those of the TMoE model (Chamroukhi (2015)). Finally, when both the degrees of freedom νk → ∞ and the skewness λk = 0, we obtain the parameter updates of the standard NMoE model. The STMoE therefore provides a more general framework for inferring flexible MoE models and attempts to simultaneously accommodate data with asymmetric distribution heavy tails and outliers. Here the ECM algorithm is used to infer the STMoE model parameters. We note that there is a good generalization of the EM algorithms for non-probabilistic problems which does not require to construct a complete-data likelihood nor a probabilistic interpretation of the maximization problem, that is the Minorization-Maximization (MM) algorithm (Hunter and Lange, 2004). The MM algorihm, used in the MoE framewrok namely by Gormley and Murphy (2008); Nguyen and McLachlan (2016) can also be a good alternative to the ECM algorithm used here. On the other hand, the ECM algorithm divides the space of model-parameters to perform sequentially the optimization in each sub-space. It may also be convenient to divide the space of the hidden variables and to alternate the optimisation, cyclically within each sub-space. This scheme is known as the Alternating ECM (AECM) algorithm (Meng and van Dyk, 1997).
AC
CE
PT
ED
The goal in regression is to be able to make predictions for the response variable(s) given some new value of the predictor variable(s) on the basis of a model trained on a set of training data. In regression analysis using mixture of experts, the aim is therefore to predict the response y given new values of the predictors (x, r), on the basis of a ˆ inferred from a set of training data, MoE model characterized by a parameter vector Ψ here, by maximum likelihood via EM. These predictions can be expressed in terms of the predictive distribution of y, which is obtained by substituting the maximum likelihood ˆ into (1)-(2) to give: parameter Ψ ˆ) = f (y|x, r; Ψ
K X
ˆ k ). ˆ k (y|x; Ψ πk (r; α)f
k=1
Using f , we might then predict y for a given set of x’s and r’s as the expected value under f , that is by calculating the prediction yˆ = EΨˆ (Y |r, x). We thus need to compute the expectation of the mixture of experts model. It is easy to show (see for example Section 1.2.4 in Fr¨ uhwirth-Schnatter (2006)) that the mean and the variance of a mixture
13
ACCEPTED MANUSCRIPT
of experts distribution of the form (5) are respectively given by EΨˆ (Y |r, x) =
k=1
K X k=1
ˆ n )EΨˆ (Y |Z = k, x), πk (r; α
(30)
2 2 ˆ n ) EΨˆ (Y |Z = k, x) + VΨˆ (Y |Z = k, x) − EΨˆ (Y |r, x) ,(31) πk (r; α
CR IP T
VΨˆ (Y |r, x) =
K X
where EΨˆ (Y |Z = k, x) and VΨˆ (Y |Z = k, x) are respectively the component-specific (expert) means and variances. The mean and the variance for the two MoE models described here are given as follows.
AN US
NMoE. For the NMoE model, the normal expert means and variances are respectively ˆ T x and V ˆ (Y |Z = k, x) = σ ˆk2 . Then, from (30) it follows given by EΨˆ (Y |Z = k, x) = β k Ψ that the mean of the NMoE is given by EΨˆ (Y |r, x) =
K X
ˆ T x. ˆ n )β πk (r; α k
(32)
k=1
M
STMoE. The mean and the variance for a skew t random variable, for this scalar case, can be easily computed as in Section 4.2 in Azzalini and Capitanio (2003) for a non-zero location parameter. Thus, for the STMoE model, the expert means for νˆk > 1, are given by ˆT x + σ ˆk δˆk ξ(ˆ νk ) EΨˆ (Y |Z = k, x) = β k
CE
thus given by:
ν ˆk 1 νˆk Γ 2 −2 . π Γ νˆk 2
PT
where ξ(ˆ νk ) =
q
ED
and the expert variances for νˆk > 2 are given by νˆk 2 2 ˆ − δk ξ (ˆ νk ) σ ˆk2 , VΨˆ (Y |Z = k, x) = νˆk − 2 Then, following (30), the mean of the proposed STMoE is
AC
EΨˆ (Y |r, x) =
K X k=1
T ˆ x+σ ˆ ˆ β πk (r; α) ˆ δ ξ(ˆ ν ) . k k k k
(33)
Finally, the variance for each MoE model is obtained by using (31) with the specified corresponding means and variances calculated in the above. 6. Model-based clustering using the STMoE
The MoE models can also be used for a model-based clustering perspective to provide a partition of the regression data into K clusters. Model-based clustering using the proposed STMoE consists in assuming that the observed data {xi , r i , yi }ni=1 are generated 14
ACCEPTED MANUSCRIPT
K
zˆi = arg max τˆik k=1
CR IP T
from a K component mixture of skew t experts, with parameter vector Ψ where the STMoE components are interpreted as clusters and hence associated to clusters. The problem of clustering therefore becomes the one of estimating the MoE parameters Ψ , which is performed here by using the dedicated ECM algorithm presented in section 4.1. Once the parameters are estimated, the provided posterior component memberships τik given by (19) represent a fuzzy partition of the data. A hard partition of the data can then be obtained from the posterior memberships by applying the MAP rule, that is: (34)
where zˆi represents the estimated cluster label for the ith individual. 7. Model selection for the STMoE
M
AN US
One of the issues in mixture model-based clustering is model selection. The problem of model selection for the STMoE models presented here in its general form is equivalent to the one of choosing the optimal number of experts K, the degree p of the polynomial regression and the degree q for the logistic regression. The optimal value of the triplet (K, p, q) can be computed by using some model selection criteria such as the Akaike Information Criterion (AIC) (Akaike, 1974), the Bayesian Information Criterion (BIC) (Schwarz, 1978) or the Integrated Classification Likelihood criterion (ICL) (Biernacki et al., 2000), etc. The AIC and BIC are are penalized observed log-likelihood criteria which can be defined as functions to be maximized and are respectively given by:
ED
ˆ − ηΨ , AIC(K, p, q) = log L(Ψ) ˆ ) − ηΨ log(n) . BIC(K, p, q) = log L(Ψ 2
AC
CE
PT
The ICL criterion consists in a penalized complete-data log-likelihood and can be expressed as follows: ˆ ) − ηΨ log(n) . ICL(K, p, q) = log Lc (Ψ 2 ˆ ) and log Lc (Ψ ˆ ) are respectively the incomplete (observed) data In the above, log L(Ψ log-likelihood and the complete data log-likelihood, obtained at convergence of the ECM algorithm for the corresponding MoE model and ηΨ is the number of free model parameters. The number of free parameters ηΨ is given by ηΨ = K(p + q + 3) − q − 1 for the NMoE model and ηΨ = K(p + q + 5) − q − 1 for the proposed STMoE model. Indeed, for each component, the STMoE have two additional parameters to be estimated, which are the robustness and the skewness parameters. However, note that in MoE it is common to use mixing proportions modeled as logistic transformation of linear functions of the covariates, that is the covariate vector in (2) is given by r i = (1, ri )T (corresponding to q = 2), ri being an univariate covariate variable. This is also adopted in this work. Moreover, for the case of linear experts, that is when the experts are linear regressors with parameter vector β k for which the 15
ACCEPTED MANUSCRIPT
CR IP T
corresponding covariate vector xi in (7) is given by xi = (1, xi )T (corresponding to p = 2), ri being an univariate covariate variable, the model selection reduces to choosing the number of experts K. Here we mainly consider this linear case. However, for a general use of the proposed STMoE model, even though the model selection criteria such as AIC, BIC, ICL can be easily computed, the direct model selection is difficult due to the large model space dimension νΨ . A searching strategy is then required to optimise the way of exploring the model space. 8. Experimental study
This section is dedicated to the evaluation of the proposed approach on simulated data and real-world data . We evaluated the performance of proposed ECM algorithm1 for the STMoE model in terms of modeling, robustness to outliers and clustering.
M
AN US
8.1. Initialization and stopping rules The parameters αk (k = 1, . . . , K − 1) of the mixing proportions are initialized randomly, including an initialization at the null vector for one run (corresponding to equal mixing proportions). Then, the common parameters (β k , σk2 ) (k = 1, . . . , K) are initialized from a random partition of the data into K clusters. This corresponds to fitting a normal mixture of experts where the initial values of the parameters are respectively given by (8) and (9) with the posterior memberships τik replaced by the hard assignments Zik issued from the random partition. For the STMoE model, the robustness parameters νk (k = 1, . . . , K) is initialized randomly in the range [1, 200] and the skewness parameters λk (k = 1, . . . , K) is initialized by randomly initializing the parameter δk in (−1, 1) from the relation λk = √ δk 2 . Then, the proposed ECM
ED
1−δk
PT
algorithm for each model is stopped when the relative variation of the observed-data (m+1) )−log L(Ψ (m) ) log-likelihood log L(Ψ| log L(Ψ reaches a prefixed threshold (for example = 10−6 ). (m) )| For each model, this process is repeated 10 times and the solution corresponding the highest log-likelihood is finally selected.
AC
CE
8.2. Experiments on simulation data sets In this section we perform an experimental study on simulated data sets to apply and assess the proposed model. Two sets of experiments have been performed. The first experiment aims at observing the effect of the sample size on the estimation quality and the second one aims at observing the impact of the presence of outliers in the data on the estimation quality, that is the robustness of the models. 8.2.1. Experiment 1 For this first experiment on simulated data, each simulated sample consisted of n observations with increasing values of the sample size n : 50, 100, 200, 500, 1000. The 1
The codes have been implemented in Matlab and are available upon request from the author.
16
ACCEPTED MANUSCRIPT
component 1 component 2
CR IP T
simulated data are generated from a two component mixture of linear experts, that is K = 2, p = q = 1. The covariate variables (xi , r i ) are simulated such that xi = r i = (1, xi )T where xi is simulated uniformly over the interval (−1, 1). We consider each of the two models for data generation (NMoE and STMoE), that is, given the covariates, the response yi |{xi , r i ; Ψ } is simulated according to the generative process of the models (3) and (15). For each generated sample, we fit each of the two models. Thus, the results are reported for the two models with data generated from each of them. We consider the mean square error (MSE) between each component of the true parameter vector and ˆ j ||2 . The squared errors are averaged on the estimated one, which is given by ||Ψ j − Ψ 100 trials. The used simulation parameters Ψ for each model are given in Table 1. parameters β 1 = (0, 1)T σ1 = 0.1 β 2 = (0, −1)T σ2 = 0.1
(0, 10)T
α1 = α2 = (0, 0)T
λ1 = 3 λ2 = −10
ν1 = 5 ν2 = 7
AN US
Table 1: Parameter values used in simulation.
α11
β10
525 457 247 31 8.5
5737 1815 785 565 68
0.965 0.847 0.816 0.363 0.261
β11
β20
ED
α10
2.440 0.852 0.348 0.091 0.076
PT
param. n 50 100 200 500 1000
M
8.2.2. Obtained results Table 2 shows the obtained results in terms of the MSE for the STMoE. One can observe that, for the proposed model, the parameter estimation error is decreasing as n increases, which is related the convergence property of the maximum likelihood estimator. One can also observe that the error decreases significantly for n ≥ 500, especially for the regression coefficients and the scale parameters.
4.388 0.742 0.473 0.314 0.233
β21
σ1
σ2
λ1
λ2
ν1
ν2
0.667 0.660 0.556 0.398 0.116
0.954 0.844 0.362 0.091 0.026
0.608 0.303 0.297 0.061 0.002
3115 2013 700 7.8 2.8
16095 7844 3847 1078 554
15096 5360 3135 223 49.4
4643 263 167 8.6 0.79
CE
Table 2: MSE ×103 between each component of the estimated parameter vector of the STMoE model and the actual one for a varying sample size n.
AC
In addition to the previously showed results, we plotted in Figures 1 and 2 the estimated quantities provided by applying respectively the NMoE model and the proposed STMoE model, and their true counterparts for n = 500 for the same the data set which was generated according the NMoE model. The upper-left plots show the estimated mean function, the estimated expert component mean functions, and the corresponding true ones. The upper-right plots show the estimated mean function with the estimated confidence region computed as plus and minus twice the estimated (pointwise) standard deviation of the model as presented in Section 5, and their true counterparts. The bottom-left plots show the true expert component mean functions and the true partition, and the bottom-right plots show their estimated counterparts. 17
ACCEPTED MANUSCRIPT
1
0.5
0.5
0
0
−0.5
−0.5
−1
−2 −1
−1
data True mean (NMoE) True Experts Estimated mean (NMoE) Estimated Experts −0.8
−0.6
−0.4
−0.2
−1.5
0 x
0.2
0.4
0.6
0.8
1
−0.8
−0.6
−0.4
−0.2
0 x
0.2
0.4
0.6
0.8
1
0
0.2
0.4
0.6
0.8
1
0.5
0
0
−0.5
M
−0.5
Actual expert mean 1 Actual expert mean 2 Actual cluster 1 Actual cluster 2 −0.8
−0.6
−0.4
ED
−1
−2 −1
−2 −1
1
1
0.5
−1.5
data True mean (NMoE) True conf. regions Estimated mean (NMoE) Estimated conf. regions
AN US
−1.5
CR IP T
1
y
y
One can clearly see that the estimations provided by the proposed model are very close to the true ones which correspond to those of the NMoE model in this case. This shows that the proposed algorithm performs well and provides an additional support to the fact that the corresponding proposed STMoE model is good generalization of the normal mixture of experts (NMoE), as it clearly approaches the NMoE as shown in this simulated examples.
−0.2
0
0.2
0.4
0.6
0.8
−1
−1.5
−2 −1
1
Estimated expert mean 1 Estimated expert mean 2 Estimated cluster 1 Estimated cluster 2 −0.8
−0.6
−0.4
−0.2
PT
Figure 1: Fitted NMoE model to a data set generated according to the NMoE model.
AC
CE
Figure 3 shows the true and estimated MoE mean functions and component mean functions by fitting the proposed STMoE model to a simulated data set of n = 500 observations. Each model was considered for data generation. The upper plot corresponds to the NMoE model and the bottom plot corresponds to the STMoE model. Finally, Figure 4 shows the corresponding true and estimated partitions. Again, one can clearly see that both the estimated models are precise. The fitted functions are close to the true ones. In addition, one can also see that the partitions estimated by the STMoE model are close the actual partitions. The proposed STMoE model can therefore be used as alternative to the NMoE model for both regression and model-based clustering. 8.2.3. Experiment 2 In this experiment we examine the robustness of the proposed STMoE model to outliers versus the standard NMoE one. For that, we considered each of the two models (NMoE and STMoE) for data generation. For each generated sample, each of the two models in considered for the inference. The data were generated exactly in the same 18
1
1
0.5
0.5
0
0
−0.5
−0.5
y
−1
−1.5
−2 −1
−1
data True mean (NMoE) True Experts Estimated mean (STMoE) Estimated Experts −0.8
−0.6
−0.4
−0.2
−1.5
0 x
0.2
0.4
0.6
0.8
−2 −1
1
1
0.5
0.5
0
0
−0.5
−0.5
−1
−1.5
−2 −1
−0.8
−0.6
−0.4
−0.2
0 x
AN US
1
data True mean (NMoE) True conf. regions Estimated mean (STMoE) Estimated conf. regions
CR IP T
y
ACCEPTED MANUSCRIPT
0.2
0.4
0.6
0.8
1
0.2
0.4
0.6
0.8
1
−1
−1.5
Actual expert mean 1 Actual expert mean 2 Actual cluster 1 Actual cluster 2 −0.8
−0.6
−0.4
−0.2
0
0.2
0.4
0.6
0.8
−2 −1
1
Estimated expert mean 1 Estimated expert mean 2 Estimated cluster 1 Estimated cluster 2
−0.8
−0.6
−0.4
−0.2
0
M
Figure 2: Fitted STMoE model to a data set generated according to the NMoE model.
CE
PT
ED
way as in Experiment 1, except for some observations which were generated with a probability c from a class of outliers. We considered the same class of outliers as in Nguyen and McLachlan (2016), that is the predictor x is generated uniformly over the interval (−1, 1) and the response y is set the value −2. We apply the MoE models by setting the covariate vectors as before, that is, x = r = (1, x)T . We considered varying probability of outliers c = 0%, 1%, 2%, 3%, 4%, 5% and the sample size of the generated data is n = 500. An example of simulated sample containing 5% outliers is shown in Figure 5. As a criterion of evaluation of the impact of the outliers on the quality of the results, we considered the MSE between the P true regression mean function and the estimated one. This MSE is calculated as n1 ni=1 ||EΨ (Yi |r i , xi ) − EΨˆ (Yi |r i , xi )||2 where the expectations are computed as in Section 5.
AC
8.2.4. Obtained results Table 3 shows, for each of the two models, the results in terms of mean squared error (MSE) between the true mean function and the estimated one, for an increasing number of outliers in the data. First, one can see that, when there is no outliers (c = 0%) and when the data follow a NMoE distribution, the error of fitting a NMoE is very slightly less than the one of fitting the proposed STMoE model. The STMoE then is still competitive. However, when the data do not contain outliers and follow a STMoE distribution, fitting a NMoE is restrictive since the error of this one is high compared to the one obtained by fitting a STMoE model. More importantly, it can be seen that, 19
1
1
0.5
0.5
0
0
−0.5
−0.5
y
−1
−1.5
data True mean (NMoE) True Experts Estimated mean (NMoE) Estimated Experts −0.8
−0.6
−0.4
−0.2
−1.5
0 x
0.2
0.4
0.6
0.8
−2 −1
1
1
0.5
0.5
0
0
−0.5
−0.5
y
1
−1
−1.5
−2 −1
data True mean (NMoE) True conf. regions Estimated mean (NMoE) Estimated conf. regions −0.8
−0.6
−0.4
−0.2
0 x
AN US
y
−2 −1
−1
CR IP T
y
ACCEPTED MANUSCRIPT
0.2
0.4
0.6
0.8
1
0.2
0.4
0.6
0.8
1
−1
data True mean (STMoE) True Experts Estimated mean (STMoE) Estimated Experts −0.8
−0.6
−0.4
−0.2
−1.5
0 x
0.2
0.4
0.6
0.8
−2 −1
1
data True mean (STMoE) True conf. regions Estimated mean (STMoE) Estimated conf. regions
−0.8
−0.6
−0.4
−0.2
0 x
M
Figure 3: The true and estimated mean function and expert mean functions by fitting the standard NMoE model (up) and the proposed STMoE model (bottom) to a simulated data set of n = 500 observations generating according to the corresponding model.
AC
CE
PT
ED
as expected, when there is outliers in the data, including the situations with only few atypical data points (1% and 2 %), the NMoE does not provide an adapted fit and is clearly outperformed by the proposed STMoE model. This includes the two situations, that is, including when the data are not generated according to the STMoE model. The errors of the NMoE model are high compared to those of the STMoE. This confirms that te STMoE is much more robust to outliers compared to the normal mixture of experts because the expert components in the STMoE model follow the robust skew t distribution. The NMoE is sensitive to outliers. On the other hand it can be seen that, when the number of outliers is increasing, increase in the error of the NMoE is more pronounced compared to the one of STMoE model. The error for the STMoE may indeed slightly increase, remain stable or even decrease in some situations. This provides an additional support to the expected robustness of the STMoE compared to the NMoE.
20
ACCEPTED MANUSCRIPT
1
0.5
0.5
0
0
−0.5
−0.5
−1
−1
−1.5
−2 −1
−1.5
Actual expert mean 1 Actual expert mean 2 Actual cluster 1 Actual cluster 2 −0.8
−0.6
−0.4
−0.2
0
0.2
0.4
0.6
0.8
−2 −1
1
1
0.5
0.5
0
0
−0.5
−0.5
−1
−0.8
−0.6
−0.4
−0.2
0
AN US
1
Estimated expert mean 1 Estimated expert mean 2 Estimated cluster 1 Estimated cluster 2
CR IP T
1
0.2
0.4
0.6
0.8
1
0.2
0.4
0.6
0.8
1
−1
−1.5
−2 −1
−1.5
Actual expert mean 1 Actual expert mean 2 Actual cluster 1 Actual cluster 2 −0.8
−0.6
−0.4
−0.2
0
0.2
0.4
0.6
0.8
−2 −1
1
Estimated expert mean 1 Estimated expert mean 2 Estimated cluster 1 Estimated cluster 2
−0.8
−0.6
−0.4
−0.2
0
0%
1%
2%
3%
4%
5%
0.178 0.258 0.710 0.280
1.057 0.741 0.7238 0.186
1.241 0.794 1.048 0.447
3.631 0.696 6.066 0.600
13.25 0.697 12.45 0.509
28.96 0.626 31.64 0.602
PT
ED
c Model NMoE NMoE STMoE NMoE STMoE STMoE
M
Figure 4: The true and estimated partitions by fitting the standard NMoE model (up) and the proposed STMoE model (bottom) to the simulated data sets shown in Figure 3.
CE
Table 3: MSE ×103 between the estimated mean function and the true one for each of the two models for a varying probability c of outliers for each simulation. The first column indicates the model used for generating the data and the second one indicates the model used for inference.
AC
Then, in order to highlight the robustness to noise of the TMoE and STMoE models, in addition to the previously shown numerical results, figures 5 and 6 show an example of results obtained on the same data set by, respectively, the NMoE and the STMoE. The data are generated by the NMoE model and contain c = 5% of outliers. In this example, we clearly see that the NMoE model is severely affected by the outliers. It provides a rough fit especially for the second component whose estimation is corresponds to a rough approximation due to the atypical data. However, one can see that the STMoE model clearly provides a precise fit; the estimated mean function and expert components are very close to the true ones. The STMoE is robust to outliers, in terms of estimating the true model as well as in terms of estimating the true partition 21
ACCEPTED MANUSCRIPT
1
1
data True mean (NMoE) True Experts Estimated mean (NMoE) Estimated Experts
0.5
CR IP T
of the data (as shown in the middle plots of the data). Notice that for the STMoE, the confidence regions are not shown because for this situation the estimated degrees of freedom are less than 2 (1.6097 and 1.5311) for the STMoE); Hence the variance for this model in that case is not defined (see Section 5). The STMoE model provides indeed components with small degrees of freedom corresponding to highly heavy tails, which allow to handle outliers in this noisy case. While the variance is not estimable here, resampling techniques can be used to evaluate it, such as the techniques of O’Hagan et al. (2015) for producing standard errors and confidence intervals for mixture parameters in model-based clustering.
0.5
-0.5
-0.5
-1
-1
-1.5
-1.5
-2 -1
AN US
y
0
y
0
data True mean (NMoE) True conf. regions Estimated mean (NMoE) Estimated conf. regions
-2
-0.8
-0.6
-0.4
-0.2
0
0.2
0.4
0.6
0.8
x
1
-1
-0.8
-0.6
-0.4
-0.2
0
0.2
0.4
0.6
0.8
1
0.2
0.4
0.6
0.8
1
0.2
0.4
0.6
0.8
1
x
1
1
Actual expert mean 1 Actual expert mean 2 Actual cluster 1 Actual cluster 2
0.5
M
0.5
0
0
y
-0.5
ED
y
-0.5
Estimated expert mean 1 Estimated expert mean 2 Estimated cluster 1 Estimated cluster 2
-1
-1.5
-1
PT
-2 -0.8
-0.6
-0.4
-0.2
0
0.2
0.4
0.6
-1
-1.5
-2 0.8
1
-1
-0.8
-0.6
-0.4
-0.2
x
−50
NMoE log−likelihood
0.9
CE
−100
0.8
−150
Mixing probabilities
0.7
−200
−250
AC
Observed data log−likelihood
0
x 1
−300
0.6 0.5 0.4 0.3
−350
0.2
−400
−450 0
0.1
2
4
6
8 10 EM iteration number
12
14
16
0 −1
18
−0.8
−0.6
−0.4
−0.2
0 x
Figure 5: Fitted NMoE model to a data set of n = 500 observations generated according to the NMoE model and including 5% of outliers.
22
ACCEPTED MANUSCRIPT
1
1
data True mean (NMoE) True Experts Estimated mean (STMoE) Estimated Experts
0.5
0.5
-0.5
-0.5
-1
-1
-1.5
-1.5
-2
-2
-1
-0.8
-0.6
-0.4
-0.2
0
0.2
0.4
0.6
0.8
1
-1
-0.8
-0.6
-0.4
x 1 Actual expert mean 1 Actual expert mean 2 Actual cluster 1 Actual cluster 2
0
0.2
0.4
0.6
0.8
1
0.2
0.4
0.6
0.8
1
0.2
0.4
0.6
0.8
1
Estimated expert mean 1 Estimated expert mean 2 Estimated cluster 1 Estimated cluster 2
0.5
-0.5
-0.5
-1
AN US
y
0
y
0
-1
-1.5
-1.5
-2
-2
-1
-0.8
-0.6
-0.4
-0.2
0
0.2
0.4
0.6
0.8
x
1
-1
-0.6
-0.4
-0.2
0
x
STMoE log−likelihood
0.9
100
ED
0
−100
PT
−200
10
20
30
40 50 EM iteration number
60
70
0.8 0.7
Mixing probabilities
M
200
−300 0
-0.8
1
300
Observed data log−likelihood
-0.2
x
1
0.5
CR IP T
y
0
y
0
data True mean (NMoE) True conf. regions Estimated mean (STMoE) Estimated conf. regions
0.6 0.5 0.4 0.3 0.2 0.1
80
0 −1
90
−0.8
−0.6
−0.4
−0.2
0 x
CE
Figure 6: Fitted STMoE model to a data set of n = 500 observations generated according to the NMoE model and including 5% of outliers.
AC
8.3. Application to two real-world data sets In this section, we consider an application to two real-world data sets: the tone perception data set and the temperature anomalies data set shown in Figure 7.
23
ACCEPTED MANUSCRIPT
Tone perception data
Temperature anomaly data
1
3.4 3.2 3 y: temperature anomaly
y: streach ratio
2.8 2.6 2.4 2.2 2
0.5
0
1.6
−0.5
1.4 1.4
1.6
1.8
2
2.2 x: tuned
2.4
2.6
2.8
3
1880
1900
CR IP T
1.8
1920
1940 1960 x: year
1980
2000
Figure 7: Scatter plots of the tone perception data and the temperature anomalies data.
AC
CE
PT
ED
M
AN US
8.3.1. Tone perception data set The first analyzed data set is the real tone perception data set1 which goes back to Cohen (1984). It was recently studied by (Bai et al., 2012; Ingrassia et al., 2012) and Song et al. (2014) by using robust regression mixture models based on, respectively, the t distribution and the Laplace distribution. In the tone perception experiment, a pure fundamental tone was played to a trained musician. Electronically generated overtones were added, determined by a stretching ratio (“stretch ratio” = 2) which corresponds to the harmonic pattern usually heard in traditional definite pitched instruments. The musician was asked to tune an adjustable tone to the octave above the fundamental tone and a “tuned” measurement gives the ratio of the adjusted tone to the fundamental. The obtained data consists of n = 150 pairs of “tuned” variables, considered here as predictors (x), and their corresponding “strech ratio” variables considered as responses (y). To apply the proposed MoE models, we set the response yi (i = 1, . . . , 150) as the “strech ratio” variables and the covariates xi = r i = (1, xi )T where xi is the “tuned” variable of the ith observation. We also follow the study in Bai et al. (2012) and Song et al. (2014) by using two mixture components. Model selection results are given later in Table 5. Figure 8 shows the scatter plots of the tone perception data and the linear expert components of the fitted NMoE model and the proposed STMoE model. One can observe that we obtain a good fit with the two models. The NMoE fit differs very slightly from the one of the STMoE. The two regression lines may correspond to correct tuning and tuning to the first overtone, respectively, as analyzed in Bai et al. (2012). 1
Source: http://artax.karlin.mff.cuni.cz/r-help/library/fpc/html/tonedata.html
24
ACCEPTED MANUSCRIPT
NMoE
3.2 3
Cluster 1 Cluster 2 Expert mean 1 Expert mean 2
3.4 3.2 3
2.8
2.8
2.6
2.6
2.4
2.4
y
2.2
2.2
2
2
1.8
1.8
1.6
1.6
1.4
1.4 1.4
1.6
1.8
2
2.2 x
2.4
2.6
2.8
3
1.4
1.6
1.8
CR IP T
y
STMoE
Cluster 1 Cluster 2 Expert mean 1 Expert mean 2
3.4
2
2.2 x
2.4
2.6
2.8
3
Figure 8: The fitted MoLE to the original tone data set with the NMoE model (left) and the STMoE model (right). The predictor x is the actual tone ratio and the response y is the perceived tone ratio.
NMoE
50
3
4
5 6 7 EM iteration number
100 80 60 40 20 0 −20
8
9
−40 0
10
CE
2
STMoE log−likelihood
120
Observed data log−likelihood
M
ED
100
STMoE
140
NMoE log−likelihood
PT
Observed data log−likelihood
150
0 1
AN US
Figure 9 shows the log-likelihood profiles for each of the two models. It can namely be seen that training the skew t mixture of experts may take more iterations than the normal MoE model. The STMoE has indeed more parameters to estimate (additional skewness and robustness parameters). However, in terms of computing time, the algorithm is fast and converges in only few seconds (around 10 seconds for this example) on a personal laptop with 2,9 GHz processor and 8 GB memory.
200
400
600 800 EM iteration number
1000
1200
1400
Figure 9: The log-likelihood during the EM iterations when fitting the MoLE models to the original tone data set with the NMoE model (left) and the STMoE model (right).
AC
The values of estimated parameters for the tone perception data set are given in Table 4. One can see that the regression coefficients are very similar for the two models. One can also see that the STMoE model retrieves a skewed component and with high degrees of freedom compared to the other component. This one may be seen as approaching the one of a skew-normal MoE model, while the second one in approaching a t distribution, that is the one of a t-MoE model.
25
ACCEPTED MANUSCRIPT
param. model NMoE STMoE
α10
α11
β10
β11
β20
β21
σ1
σ2
λ1
λ2
ν1
ν2
-2.690 -3.044
0.796 0.824
-0.029 -0.058
0.995 0.944
1.913 1.944
0.043 0.032
0.137 0.200
0.047 0.032
93.386
-0.011
19.070
1.461
Table 4: Values of the estimated MoE parameters for the original Tone perception data set.
NMoE AIC 6.382 134.847 137.763 148.798 176.318
ICL 1.866 107.384 76.524 94.460 123.655
M
BIC 1.866 122.805 118.193 121.703 141.696
ED
K 1 2 3 4 5
AN US
CR IP T
We also performed a model selection procedure on this data set to choose the best number of MoE components for a number of components between 1 and 5. We used BIC, AIC, and ICL. Table 5 gives the obtained values of the model selection criteria. One can see that for the NMoE model, the three criteria overestimate the number of components, but for both BIC and ICL, the solution with two components is also likely and is the most competitive to the selected one with 5 components. In deed, it can be seen that, if the number mixture of components is fixed at 4 rather than 5, both BIC and ICL would select the right number of components in that case. AIC performs poorly for the two models and overestimates the number of components. On the other hand, for the proposed STMoE model, both BIC and ICL retrieve the correct number of components. Then, one can conclude that the BIC and the ICL are the criteria that one would suggest for the analysis of this data with the proposed model. BIC 69.532 92.435 77.9753 77.7092 79.043
STMoE AIC 77.059 110.499 106.576 116.847 128.719
ICL 69.532 82.455 52.564 56.365 67.748
PT
Table 5: Choosing the number of expert components K for the original tone perception data by using the information criteria BIC, AIC, and ICL. Underlined value indicates the highest value for each criterion.
AC
CE
Now we examine the sensitivity of the MoE models to outliers based on this real data set. For this, we adopt the same scenario used in Bai et al. (2012) and Song et al. (2014) (the last and more difficult scenario) by adding 10 identical pairs (0, 4) to the original data set as outliers in the y-direction, considered as high leverage outliers. We apply the MoE models in the same way as before. The upper plots in Figure 10 clearly show that the normal mixture of experts fit is sensitive to outliers. However, note that for this situation, compared to the normal regression mixture result in Bai et al. (2012), and the Laplace regression mixture and the t regression mixture results in Song et al. (2014), the fitted NMoE model is affected less severely by the outliers. This may be attributed to the fact that the mixing proportions here are depending on the predictors, which is not the case in these regression mixture models, namely thoses of Bai et al. (2012) and Song et al. (2014). One can also see that, even the regression mean functions are affected severely by the outliers, the provided 26
ACCEPTED MANUSCRIPT
NMoE 4
STMoE
4
Cluster 1 Cluster 2 Expert mean 1 Expert mean 2
3.5
CR IP T
partitions are still reasonable and similar to those provided in the previous non-noisy case. Then, the bottom plots in Figure 10 also clearly show that the STMoE model provides a precise robust fit. For the STMoE, even if the fit differs very slightly compared to the case with outliers, the obtained fits for both situations (with and without outliers) are very reasonable. Moreover, we notice that, as showed in Song et al. (2014), for this situation with outliers, the t mixture of regressions fails; The fit is affected severely by the outliers. However, for the proposed STMoE model, the ten high leverage outliers have no significant impact on the fitted experts. This is because here the mixing proportions depend on the inputs, which is not the case for the regression mixture model described in Song et al. (2014).
3.5
y
AN US
3
y
3
Cluster 1 Cluster 2 Expert mean 1 Expert mean 2
2.5
2.5
2
2
1.5
1.5
0.5
1
1.5 x
2
2.5
3
0
0.5
1
1.5 x
2
2.5
3
M
0
ED
Figure 10: Fitting MoLE to the tone data set with ten added outliers (0, 4) with the NMoE model fit (left) and the STMoE model fit (right). The predictor x is the actual tone ratio and the response y is the perceived tone ratio.
Figure 11 shows the log-likelihood profiles for each of the two models, which show a similar behavior than the one in the case without outliers.
20 0
−20
AC
−40
60 40 20 0 −20
−60
−40
−80
−60
−100 0
2
4
6 8 EM iteration number
STMoE log−likelihood
80
Observed data log−likelihood
40
STMoE
100 NMoE log−likelihood
CE
Observed data log−likelihood
60
NMoE
PT
80
10
12
−80 0
14
100
200 300 EM iteration number
400
500
Figure 11: The log-likelihood during the EM iterations when fitting the MoLE models to the tone data set with ten added outliers (0, 4) with the NMoE model (left) and the STMoE model (right).
The values of estimated MoE parameters in this case with outliers are given in Table 6. One can see that the SNMoE model parameters are identical to those of the 27
ACCEPTED MANUSCRIPT
NMoE, with a skewness close to zero. The regression coefficients for the second expert component are very similar for the two models. For the STMoE model, it retrieves a skewed normal component while the second component is approaching a t distribution with a small degrees of freedom. α10
α11
β10
β11
β20
β21
σ1
σ2
λ1
λ2
ν1
ν2
0.811 -3.004
0.150 0.732
3.117 -0.246
-0.285 1.016
1.907 1.808
0.046 0.060
0.700 0.212
0.050 0.088
156.240
1.757
81.355
1.630
CR IP T
param. model NMoE STMoE
Table 6: Values of the estimated MoE parameters for the tone perception data set with added outliers.
AC
CE
PT
ED
M
AN US
8.3.2. Temperature anomalies data set In this experiment, we examine another real-world data set related to climate change analysis. The NASA GISS Surface Temperature (GISTEMP) analysis provides a measure of the changing global surface temperature with monthly resolution for the period since 1880, when a reasonably global distribution of meteorological stations was established. The GISS analysis is updated monthly, however the data presented here1 are updated annually as issued from the Carbon Dioxide Information Analysis Center (CDIAC), which has served as the primary climate-change data and information analysis center of the U.S. Department of Energy since 1982. The data consist of n = 135 yearly measurements of the global annual temperature anomalies (in degrees C) computed using data from land meteorological stations for the period of 1882 − 2012. These data have been analyzed earlier by Hansen et al. (1999, 2001) and recently by Nguyen and McLachlan (2016) by using the Laplace mixture of linear experts (LMoLE). To apply the proposed non-normal mixture of expert model, we consider mixtures of two experts as in Nguyen and McLachlan (2016). This number of components is also the one provided by the model selection criteria as shown later in Table 8. Indeed, as mentioned by Nguyen and McLachlan (2016), Hansen et al. (2001) found that the data could be segmented into two periods of global warming (before 1940 and after 1965), separated by a transition period where there was a slight global cooling (i.e. 1940 to 1965). Documentation of the basic analysis method is provided by Hansen et al. (1999, 2001). We set the response yi (i = 1, . . . , 135) as the temperature anomalies and the covariates xi = r i = (1, xi )T where xi is the year of the ith observation. Figures 12, 13, and 14 respectively show, for each of the two MoE models, the two fitted linear expert components, the corresponding means and confidence regions computed as plus and minus twice the estimated (pointwise) standard deviation as presented in Section 5, and the log-likelihood profiles. One can observe that the two models are successfully applied on the data set and provide very similar results. These results are also similar to those found by Nguyen and McLachlan (2016) who used a Laplace mixture of linear experts. 1
Source: Ruedy et al. (2015), http://cdiac.ornl.gov/ftp/trends/temp/hansen/gl_land.txt
28
ACCEPTED MANUSCRIPT
1
NMoE
1
Cluster 1 Cluster 2 Expert mean 1 Expert mean 2
0
0
−0.5
−0.5
1880
1900
1920
1940 x
1960
1980
2000
1880
1900
CR IP T
y
0.5
y
0.5
STMoE Cluster 1 Cluster 2 Expert mean 1 Expert mean 2
1920
1940 x
1960
1980
2000
1
AN US
Figure 12: Fitting the MoLE models to the temperature anomalies data set with the NMoE model (left) and the STMoE model (right). The predictor x is the year and the response y is the temperature anomaly.
NMoE
1
data Estimated mean (NMoE)
y
0.5
y
0.5
STMoE
data Estimated mean (STMoE)
0
−0.5
1900
1920
1940 x
1960
1980
−0.5
2000
1880
1900
1920
1940 x
1960
1980
2000
ED
1880
M
0
NMoE
CE
100
PT
Figure 13: The fitted MoLE models to the temperature anomalies data set with the NMoE model fit (left) and the STMoE model fit (right). The predictor x is the year and the response y is the temperature anomaly. The shaded region represents plus and minus twice the estimated (pointwise) standard deviation as presented in Section 5.
95
Observed data log−likelihood
85 80
AC
Observed data log−likelihood
STMoE log−likelihood
90
90
75 70 65 60
80 70 60 50 40 30 20
55 50 0
STMoE
100
NMoE log−likelihood
10
20
30 40 50 EM iteration number
60
70
10 0
80
5
10
15 20 EM iteration number
25
30
Figure 14: The log-likelihood during the EM iterations when fitting the MoLE models to the temperature anomalies data set with the NMoE model (left) and the STMoE model (right).
29
ACCEPTED MANUSCRIPT
param. model NMoE STMoE
α10
α11
β10
β11
β20
β21
946.483 931.966
-0.481 -0.474
-12.805 -12.848
0.006 0.006
-41.073 -40.876
0.020 0.020
CR IP T
The values of estimated MoE parameters for the temperature anomalies data set are given in Table 7. One can see that the parameters common for the two models are quasi-identical. It can also be seen the STMoE model provides a solution with a skewness close to zero. This may support the hypothesis of non-asymmetry for this data set. Then, the STMoE solution provides a degrees of freedom more than 17, which tends to approach a normal distribution. On the other hand, the regression coefficients are also similar to those found by Nguyen and McLachlan (2016) who used a Laplace mixture of linear experts. σ1
σ2
λ1
λ2
ν1
ν2
0.115 0.113
0.110 0.105
0.024
-0.015
41.048
17.589
AN US
Table 7: Values of the estimated MoE parameters for the temperature anomalies data set.
NMoE AIC 50.420 91.537 90.280 92.875 92.920
ED
BIC 46.062 79.916 71.396 66.727 59.510
PT
K 1 2 3 4 5
M
We also performed a model selection procedure on the temperature anomalies data set to choose the best number of MoE components from values between 1 and 5. Table 8 gives the obtained values of the used model selection criteria, that is BIC, AIC, and ICL. One can see that, except the result provided by the AIC for the NMoE model which provide a high number of components, all the others results provide evidence for two components in the data. ICL 46.062 79.624 58.487 54.752 51.242
BIC 40.971 69.638 54.126 42.308 28.037
STMoE AIC 48.234 87.069 81.726 80.0773 75.974
ICL 40.971 69.341 30.655 20.494 -8.881
CE
Table 8: Choosing the number of expert components K for the temperature anomalies data by using the information criteria BIC, AIC, and ICL. Underlined value indicates the highest value for each criterion.
9. Concluding remarks and future work
AC
In this paper we proposed a new non-normal MoE model, which generalizes the normal MoE model and attempts to simultaneously accommodate heavy tailed data with possible outliers and asymmetric distribution. The proposed STMoE is based on the flexible skew t distribution which is suggested for possibly non-symmetric, heavy tailed and noisy data. We developed a CEM algorithm for model inference and described the use of the model in non-linear regression and prediction as well as in model-based clustering. The developed model was successfully applied and validated on simulation studies and two real data sets. The results obtained on simulated data confirm the good performance of the model in terms of density estimation, non-linear regression function 30
ACCEPTED MANUSCRIPT
AN US
CR IP T
approximation and clustering. In addition, the simulation results provide evidence of the robustness of the STMoE model to outliers, compared to the standard alternative NMoE model. The proposed model was also successfully applied to two different real data sets, including situations with outliers. The model selection using information criteria tends to promote using BIC and ICL against AIC which may perform poorly in the analyzed data. The obtained results support the potential benefit of the proposed approach for practical applications. One interesting future direction is to extend the proposed model to the hierarchical mixture of experts framework (Jordan and Jacobs, 1994). Another natural future extension of this work is also to consider the case of MoE for multiple regression on multivariate responses rather than simple univariate regression. In that case, one may consider the multivariate skew-t and the multivariate Normal inverse Gaussian distribution (O’Hagan et al., 2016) which may be more stable in high-dimensional settings compared to the multivariate skew-t. Appendix A. The skew-normal distribution
M
As introduced by (Azzalini, 1985, 1986), a random variable Y follows a univariate skew-normal distribution with location parameter µ ∈ R, scale parameter σ 2 ∈ (0, ∞) and skewness parameter λ ∈ R if it has the density y−µ 2 y−µ 2 φ( )Φ λ( ) (A.1) f (y; µ, σ , λ) = σ σ σ
PT
ED
where φ(.) and Φ(.) denote, respectively, the probability density function (pdf) and the cumulative distribution function (cdf) of the standard normal distribution. It can be seen from (A.1) that when λ = 0, the skew-normal reduces to the normal distribution. As presented by Azzalini (1986); Henze (1986), if √ (A.2) Y = µ + δ|U | + 1 − δ 2 E
AC
CE
λ where δ = √1+λ 2 , U and E are independent random variables following the normal distribution N(0, σ 2 ), then Y follows the skew-normal distribution with pdf SN(µ, σ 2 , λ) given by (A.1). In the above, |U | denotes the magnitude of U . This stochastic representation of the skew-normal distribution leads to the following hierarchical representation in an incomplete data framework, as presented in Lin et al. (2007b):
Y |u U
∼ N (µ + δ|u|, (1 − δ 2 )σ 2 ) , ∼ N(0, σ 2 ).
(A.3)
Appendix B. Stochastic representation of the STMoE model The skew t mixture of experts model is characterized as follows. Suppose that conditional on a categorical variable Zi = zi ∈ {1, . . . , K} representing the hidden label of 31
ACCEPTED MANUSCRIPT
the component generating the ith observation and which, conditional on some predictor r i , follows the multinomial distribution (B.1): Zi |r i ∼ Mult(1; π1 (r i ; α), . . . , πK (r i ; α))
(B.1)
AN US
CR IP T
where each of the probabilities πzi (r i ; α) = P(Zi = zi |r i ) is given by the multinomial logistic function (2). Now suppose a random variable Yi having the following representation: Ei Yi = µ(xi ; β zi ) + σzi √ (B.2) Wi where Ei and Wi are independent univariate random variables with, respectively, a standard skew-normal distribution Ei ∼ SN(λzi ), and a Gamma distribution Wi ∼ ν ν Gamma( 2zi , 2zi ), and xi and r i are some given covariate variables. Then, the variable Yi is said to follow the skew t mixture of experts (STMoE) defined by (13). References
Akaike, H., 1974. A new look at the statistical model identification. IEEE Transactions on Automatic Control 19 (6), 716–723. Azzalini, A., 1985. A class of distributions which includes the normal ones. Scandinavian Journal of Statistics, 171–178.
M
Azzalini, A., 1986. Further results on a class of distributions which includes the normal ones. Scandinavian Journal of Statistics, 199–208.
ED
Azzalini, A., Capitanio, A., 2003. Distributions generated by perturbation of symmetry with emphasis on a multivariate skew t distribution. Journal of the Royal Statistical Society, Series B 65, 367–389.
PT
Bai, X., Yao, W., Boyer, J. E., 2012. Robust fitting of mixture regression models. Computational Statistics & Data Analysis 56 (7), 2347 – 2359.
CE
Biernacki, C., Celeux, G., Govaert, G., 2000. Assessing a mixture model for clustering with the integrated completed likelihood. IEEE Transactions on Pattern Analysis and Machine Intelligence 22 (7), 719–725.
AC
Bishop, C., Svens´en, M., 2003. Bayesian hierarchical mixtures of experts. In: In Uncertainty in Artificial Intelligence. Brent, R. P., 1973. Algorithms for minimization without derivatives. Prentice-Hall series in automatic computation. Englewood Cliffs, N.J. Prentice-Hall. Chamroukhi, F., July 2015. Non-normal mixtures of experts. arXiv:1506.06707. Chamroukhi, F., 2016. Robust mixture of experts modeling using the t distribution. Neural Networks 79, 20 – 36. 32
ACCEPTED MANUSCRIPT
Chamroukhi, F., Sam´e, A., Govaert, G., Aknin, P., June 2009a. A regression model with a hidden logistic process for feature extraction from time series. In: International Joint Conference on Neural Networks (IJCNN). Atlanta, GA, pp. 489–496. Chamroukhi, F., Sam´e, A., Govaert, G., Aknin, P., 2009b. Time series modeling by a regression approach based on a latent process. Neural Networks 22 (5-6), 593–602.
CR IP T
Chen, K., Xu, L., Chi, H., 1999. Improved learning algorithms for mixture of experts in multiclass classification. Neural Networks 12 (9), 1229–1252. Cohen, E. A., 1984. Some effects of inharmonic partials on interval perception. Music Perception 1. Dempster, A. P., Laird, N. M., Rubin, D. B., 1977. Maximum likelihood from incomplete data via the EM algorithm. Journal of The Royal Statistical Society, B 39(1), 1–38.
AN US
Faria, S., Soromenho, G., February 2010. Fitting mixtures of linear regressions. Journal of Statistical Computation and Simulation 80 (2), 201–225. Fr¨ uhwirth-Schnatter, S., 2006. Finite Mixture and Markov Switching Models (Springer Series in Statistics). Springer Verlag, New York.
M
Fr¨ uhwirth-Schnatter, S., Pyne, S., 2010. Bayesian inference for finite mixtures of univariate and multivariate skew-normal and skew-t distributions. Biostatistics 11 (2), 317–336.
ED
Gaffney, S., Smyth, P., 1999. Trajectory clustering with mixtures of regression models. In: Proceedings of the fifth ACM SIGKDD international conference on Knowledge discovery and data mining. ACM Press, pp. 63–72.
PT
Gormley, I. C., Murphy, T. B., 2008. A mixture of experts model for rank data with applications in election studies. The Annals of Applied Statistics 2 (4), 1452–1477.
CE
Gormley, I. C., Murphy, T. B., 2010. A mixture of experts latent position cluster model for social network data. Statistical methodology 7 (3), 385–405. Gormley, I. C., Murphy, T. B., 2011. Mixture of experts modelling with social science applications. Journal of Computational and Graphical Statistics 19 (2), 332–353.
AC
Green, P., 1984. Iteratively reweighted least squares for maximum likelihood estimation, and some robust and resistant alternatives. Journal of The Royal Statistical Society, B 46 (2), 149–192. Hansen, J., Ruedy, R., Glascoe, J., Sato, M., 1999. Giss analysis of surface temperature change. Journal of Geophysical Research 104, 30997–31022. Hansen, J., Ruedy, R., M., S., Imhoff, M., Lawrence, W., Easterling, D., Peterson, T., Karl, T., 2001. A closer look at united states and global surface temperature change. Journal of Geophysical Research 106, 23947–23963. 33
ACCEPTED MANUSCRIPT
Henze, N., 1986. A probabilistic representation of the skew-normal distribution. Scandinavian Journal of Statistics, 271–275. Hunter, D., Young, D., 2012. Semiparametric mixtures of regressions. Journal of Nonparametric Statistics 24 (1), 19–38.
CR IP T
Hunter, D. R., Lange, K., 2004. A tutorial on MM algorithms. The American Statistician 58, 30–37. Ingrassia, S., Minotti, S., Vittadini, G., 2012. Local statistical modeling via a clusterweighted approach with elliptical distributions. Journal of Classification 29 (3), 363– 401. J. Q. Shi, R. M.-S., Titterington, D. M., 2005. Hierarchical gaussian process mixtures for regression. Statistics and Computing 15 (1), 31–41.
AN US
Jacobs, R. A., Jordan, M. I., Nowlan, S. J., Hinton, G. E., 1991. Adaptive mixtures of local experts. Neural Computation 3 (1), 79–87. Jiang, W., Tanner, M. A., 1999. On the identifiability of mixtures-of-experts. Neural Networks 12, 197–220.
M
Jones, P. N., McLachlan, G. J., June 1992. Fitting finite mixture models in a regression context. Australian Journal of Statistics 34 (2), 233–240.
ED
Jordan, M. I., Jacobs, R. A., 1994. Hierarchical mixtures of experts and the EM algorithm. Neural Computation 6, 181–214. Jordan, M. I., Xu, L., 1995. Convergence results for the EM approach to mixtures of experts architectures. Neural Networks 8 (9), 1409–1431.
PT
Lee, S. X., McLachlan, G. J., 2013a. Model-based clustering and classification with nonnormal mixture distributions. Statistical Methods and Applications 22 (4), 427–454.
CE
Lee, S. X., McLachlan, G. J., 2013b. On mixtures of skew normal and skew tdistributions. Advances in Data Analysis and Classification 7 (3), 241–266.
AC
Lee, S. X., McLachlan, G. J., 2014. Finite mixtures of multivariate skew t-distributions: some recent and new results. Statistics and Computing 24 (2), 181–202. Lee, S. X., McLachlan, G. J., 2015. Finite mixtures of canonical fundamental skew t-distributions. Statistics and Computing (To appear). Lin, T. I., 2010. Robust mixture modeling using multivariate skew t distributions. Statistics and Computing 20 (3), 343–356. Lin, T. I., Lee, J. C., Hsieh, W. J., 2007a. Robust mixture modeling using the skew t distribution. Statistics and Computing 17 (2), 81–92. 34
ACCEPTED MANUSCRIPT
Lin, T. I., Lee, J. C., Yen, S. Y., 2007b. Finite mixture modelling using the skew normal distribution. Statistica Sinica 17, 909–927. McLachlan, G. J., Krishnan, T., 2008. The EM algorithm and extensions, 2nd Edition. New York: Wiley. McLachlan, G. J., Peel., D., 2000. Finite mixture models. New York: Wiley.
CR IP T
Meng, X. L., Rubin, D. B., 1993. Maximum likelihood estimation via the ECM algorithm: A general framework. Biometrika 80 (2), 267–278. Meng, X.-L., van Dyk, D., 1997. The EM algorithm-an old folk song sung to the fast tune (with discussion). Journal of the Royal Statistical Society, Series B 4 (59), 511 – 567.
AN US
Ng, S.-K., McLachlan, G. J., 2004. Using the em algorithm to train neural networks: misconceptions and a new algorithm for multiclass classification. IEEE Transactions on Neural Networks 15 (3), 738–749. Nguyen, H. D., McLachlan, G. J., 2016. Laplace mixture of linear experts. Computational Statistics & Data Analysis 93, 177 – 191.
M
O’Hagan, A., Murphy, T. B., Gormley, I. C., 2015. On estimation of parameter uncertainty in model-based clustering. arXiv:1510.00551.
ED
O’Hagan, A., Murphy, T. B., Gormley, I. C., McNicholas, P. D., Karlis, D., 2016. Clustering with the multivariate normal inverse gaussian distribution. Computational Statistics & Data Analysis 93, 18 – 30.
PT
Pyne, S., Hu, X., Wang, K., Rossin, E., Lin, T., Maier, L., Baecher-Allan, C., McLachlan, G., Tamayo, P., Hafler, D., De Jager, P., Mesirow, J., 2009. Automated highdimensional flow cytometric data analysis. Proceedings of the National Academy of Sciences USA.
CE
Quandt, R. E., 1972. A new approach to estimating switching regressions. Journal of the American Statistical Association 67 (338), 306–310.
AC
Quandt, R. E., Ramsey, J. B., 1978. Esimating mixtures of normal distributions and switching regressions. Journal of the American Statistical Association 73 (364), 730– 738. Rasmussen, C. E., Ghahramani, Z., 2001. Infinite mixtures of gaussian process experts. In: In Advances in Neural Information Processing Systems 14. MIT Press, pp. 881– 888. Ruedy, R., Sato, M., Lo, K., 2015. NASA GISS surface temperature (GISTEMP) analysis. DOI: 10.3334/CDIAC/cli.001, center for Climate Systems Research, NASA Goddard Institute for Space Studies 2880 Broadway, New York, NY 10025 USA. 35
ACCEPTED MANUSCRIPT
Schwarz, G., 1978. Estimating the dimension of a model. Annals of Statistics 6, 461–464. Shi, J. Q., Choi, T., 2011. Gaussian Process Regression Analysis for Functional Data. Chapman & Hall/CRC Press. Song, W., Yao, W., Xing, Y., 2014. Robust mixture regression model fitting by laplace distribution. Computational Statistics & Data Analysis 71 (0), 128 – 137.
CR IP T
Veaux, R. D. D., 1989. Mixtures of linear regressions. Computational Statistics and Data Analysis 8 (3), 227–245. Viele, K., Tong, B., 2002. Modeling with mixtures of linear regressions. Statistics and Computing 12, 315–330.
AN US
Waterhouse, S., Mackay, D., Robinson, T., 1996. Bayesian methods for mixtures of experts. In: In. MIT Press, pp. 351–357. Waterhouse, S. R., 1997. Classification and regression using mixtures of experts. Ph.D. thesis, Department of Engineering, Cambridge University. Wei, Y., 2012. Robust mixture regression models using t-distribution. Tech. rep., Master Report, Department of Statistics, Kansas State University.
M
Young, D., Hunter, D., 2010. Mixtures of regressions with predictor-dependent mixing proportions. Computational Statistics and Data Analysis 55 (10), 2253–2266.
ED
Yuksel, S. E., Wilson, J. N., Gader, P. D., 2012. Twenty years of mixture of experts. IEEE Trans. Neural Netw. Learning Syst. 23 (8), 1177–1193.
AC
CE
PT
Zeller, C. B., Lachos, V. H., Cabral, C., 2015. Robust mixture regression modelling based on scale mixtures of skew-normal distributions. Test (revision invited).
36
ACCEPTED MANUSCRIPT
Faicel Chamroukhi
AC
CE
PT
ED
M
AN US
CR IP T
Faicel Chamroukhi is since 2016 a Full Professor of Statistics and Data Science at University of Caen, department of Mathematics and Computer Science and the Lab of Mathematics Nicolas Oresme (LMNO) UMR CNRS 6139. From 2011 to 2015, he has been an Associate Professor at University of Toulon and the Information Sciences and Systems Lab. (LSIS) UMR CNRS 7296. In 2015-2016 he was awarded a CNRS research leave at the Lab of Mathematics Paul Painlevé, UMR CNRS 8524 in Lille, where he has also been an Invited Scientist at INRIA. He received his Master degree of Engineering Sciences from Pierre & Marie Curie (Paris 6) University in 2007 and his Ph.D. degree in applied mathematics and computer science, in the area of statistical leaning and data analysis from Compiègne University of Technology in 2010. In 2011, he was qualified for the position of Associate Professor in applied mathematics (CNU 26), computer science (CNU 27), and signal processing (CNU 61). In 2015, he received his Accreditation to Supervise Research (HDR) in applied mathematics and computer science, in the area of statistical learning and data science, from Toulon University. In 2016, he was qualified for the position of Professor in the area of applied mathematics, computer science, and signal processing (CNU 26, 27, 61). His multidisciplinary research is in the area of Data Science and includes statistics, machine learning and statistical signal processing, with a particular focus of the statistical methodology and inference of latent data models for complex heterogeneous highdimensional and massive data, temporal data, functional data, and their application to real-world problems including dynamical systems, acoustic/speech processing, life sciences (medicine, biology), information retrieval, social networks.
AC
CE
PT
ED
M
AN US
CR IP T
ACCEPTED MANUSCRIPT
38
Accepted Manuscript
Robust mixture of experts modeling using the skew t distribution F. Chamroukhi PII: DOI: Reference:
S0925-2312(17)30894-9 10.1016/j.neucom.2017.05.044 NEUCOM 18462
To appear in:
Neurocomputing
Received date: Revised date: Accepted date:
5 November 2015 30 November 2016 19 May 2017
Please cite this article as: F. Chamroukhi, Robust mixture of experts modeling using the skew t distribution, Neurocomputing (2017), doi: 10.1016/j.neucom.2017.05.044
This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.
ACCEPTED MANUSCRIPT
Robust mixture of experts modeling using the skew t distribution F. Chamroukhia,∗ Normandie Universit´e, UNICAEN, CNRS, Laboratoire de Math´ematiques Nicolas Oresme - LMNO, 14000 Caen, France
CR IP T
a
Abstract
PT
ED
M
AN US
Mixture of Experts (MoE) is a popular framework in the fields of statistics and machine learning for modeling heterogeneity in data for regression, classification and clustering. MoE for continuous data are usually based on the normal distribution. However, it is known that for data with asymmetric behavior, heavy tails and atypical observations, the use of the normal distribution is unsuitable. We introduce a new robust non-normal mixture of experts modeling using the skew t distribution. The proposed skew t mixture of experts, named STMoE, handles these issues of the normal mixtures experts regarding possibly skewed, heavy-tailed and noisy data. We develop a dedicated expectation conditional maximization (ECM) algorithm to estimate the model parameters by monotonically maximizing the observed data log-likelihood. We describe how the presented model can be used in prediction and in model-based clustering of regression data. Numerical experiments carried out on simulated data show the effectiveness and the robustness of the proposed model in fitting non-linear regression functions as well as in model-based clustering. Then, the proposed model is applied to the real-world data of tone perception for musical data analysis, and the one of temperature anomalies for the analysis of climate change data. The obtained results confirm the usefulness of the model for practical data analysis applications.
CE
Keywords: mixture of experts, skew t distribution; EM algorithm; ECM algorithm; non-linear regression; model-based clustering. 1. Introduction
AC
Mixture of Experts (MoE) (Jacobs et al., 1991) is a popular framework in the statistics and machine learning fields for modeling heterogeneity in data for regression, classification and clustering. They consist in a fully conditional mixture model where both ∗
Corresponding author: Faicel Chamroukhi Universit´e de Caen-Normandie, LMNO, UMR CNRS 6139 Campus 2, Bvd Mar´echal Juin, 14032 Caen Cedex, France Tel: +33(0) 2 31 56 73 67 Fax: +33(0) 2 31 56 73 20 Email address: [email protected] (F. Chamroukhi ) Preprint submitted to Neurocomputing - Elsevier
July 4, 2017
ACCEPTED MANUSCRIPT
AC
CE
PT
ED
M
AN US
CR IP T
the mixing proportions, known as the gating functions, and the component densities, known as the experts, are conditional on some input covariates. MoE have been investigated, in their simple form, as well as in their hierarchical form Jordan and Jacobs (1994) (e.g Section 5.12 of McLachlan and Peel. (2000)) for regression and model-based cluster and discriminant analyses and in different application domains. MoE Have also been investigated for rank data Gormley and Murphy (2008) and network data Gormley and Murphy (2010) with social science applications. A survey on the topic can be found in Gormley and Murphy (2011). A complete review of the MoE models can be found in Yuksel et al. (2012). MoE for continuous data are usually based on the normal distribution. Along this paper, we will call the MoE using the normal distribution the normal mixture of experts, abbreviated as NMoE. However, it is well-known that the normal distribution is sensitive to outliers. Moreover, for a set of data containing a group or groups of observations with heavy tails or asymmetric behavior, the use of normal experts may be unsuitable and can unduly affect the fit of the MoE model. In this paper, we attempt to overcome these limitations in MoE by proposing a more adapted and robust mixture of experts model which can deal with possibly skewed, heavy-tailed data and with outliers. Recently, the problem of sensitivity of NMoE to outliers have been considered by Nguyen and McLachlan (2016) where the authors proposed a Laplace mixture of linear experts (LMoLE) for a robust modeling of non-linear regression data. The model parameters are estimated by maximizing the observed-data likelihood via a minorizationmaximization (MM) algorithm. Here, we propose an alternative MoE model, by relaying on other non-normal distribution that generalizes the normal distribution, that is, the skew-t distribution introduced quite recently by Azzalini and Capitanio (2003). We call the proposed MoE model the skew-t mixture of experts (STMoE). One may use the t distribution, as in the t mixture of experts (TMoE) proposed by Chamroukhi (2015, 2016) which provides a natural robust extension of the normal distribution to model data with more heavy tails and to deal with possible outliers. The robustness of the t distribution may however be not sufficient in the presence of asymmetric observations. In mixture modeling, to deal with this issue regarding skewed data, Lin et al. (2007a) proposed the univariate skew-t mixture model which allows for accommodation of both skewness and thick tails in the data, by relying on the skew-t distribution Azzalini and Capitanio (2003). For the general multivariate case using skew-t mixtures, one can refer to Pyne et al. (2009), (Lin, 2010), Lee and McLachlan (2013a), Lee and McLachlan (2013b), Lee and McLachlan (2014), and recently, the unifying framework for previous restricted and unrestricted skew-t mixtures, using the CFUST distribution Lee and McLachlan (2015). The inference in the previously described approaches is performed by maximum likelihood estimation via the expectation-maximization (EM) algorithm or its extensions (Dempster et al., 1977; McLachlan and Krishnan, 2008), in particular the expectation conditional maximization (ECM) algorithm (Meng and Rubin, 1993). Fr¨ uhwirthSchnatter and Pyne (2010) have also considered the Bayesian inference framework for namely the skew-t mixtures. For the regression context, the robust modeling of regression data has been studied 2
ACCEPTED MANUSCRIPT
AC
CE
PT
ED
M
AN US
CR IP T
namely by Bai et al. (2012); Wei (2012); Ingrassia et al. (2012) who considered a mixture of linear regressions using the t distribution. In the same context of regression, Song et al. (2014) proposed the mixture of Laplace regressions, which has been then extended by Nguyen and McLachlan (2016) to the case of mixture of experts, by introducing the Laplace mixture of linear experts (LMoLE). Recently, Zeller et al. (2015) introduced the scale mixtures of skew-normal distributions for robust mixture regressions. However, unlike our proposed STMoE model, the regression mixture models of Wei (2012), Bai et al. (2012), Ingrassia et al. (2012), Song et al. (2014), Zeller et al. (2015) do not consider conditional mixing proportions, that is, mixing proportions depending on some input variables, as in the case of mixture of experts, which we investigate here. In addition, the approaches of Wei (2012), Bai et al. (2012), Ingrassia et al. (2012) and Song et al. (2014) do not consider both the problem of robustness to outliers together with the one of dealing with possibly asymmetric data. Here we consider the mixture of experts framework for non-linear regression problems and model-based clustering of regression data, and we attempt to overcome the limitations of the NMoE model for dealing with asymmetric, heavy-tailed data and which may contain outliers. We investigate the use of the skew t distribution for the experts, rather than the commonly used normal distribution. We propose the skew-t mixture of experts (STMoE) model which allows for accommodation of both skewness and heavy tails in the data and which is obust to outliers. This model corresponds to an extension of the unconditional skew t mixture model (Lin et al., 2007a), to the mixture of experts (MoE) framework, where the mixture means are regression functions and the mixing proportions are also covariate-varying. For the model inference, we develop a dedicated expectation conditional maximization (ECM) algorithm to estimate the model parameters by monotonically maximizing the observed data log-likelihood. The expectation-maximization algorithm and its extensions (Dempster et al., 1977; McLachlan and Krishnan, 2008) are indeed very popular and successful estimation algorithms for mixture models in general and for mixture of experts in particular. Moreover, the EM algorithm for MoE has been shown by Ng and McLachlan (2004) to be monotonically maximizing the MoE likelihood. The authors have showed that the EM (with Iteratively Reweighted Least Squares (IRLS) in this case) algorithm has stable convergence and the log-likelihood is monotonically increasing when a learning rate smaller than one is adopted for the IRLS procedure within the M-step of the EM algorithm. They have further proposed an expectation conditional maximization (ECM) algorithm to train MoE, which also has desirable numerical properties. The MoE has also been considered in the Bayesian framework, for example one can cite the Bayesian MoE Waterhouse et al. (1996); Waterhouse (1997) and the Bayesian hierarchical MoE Bishop and Svens´en (2003). Beyond the Bayesian parametric framework, the MoE models have also been investigated within the Bayesian non-parametric framework. We cite for example the Bayesian non-parametric MoE model (Rasmussen and Ghahramani, 2001) and the Bayesian non-parametric hierarchical MoE approach of J. Q. Shi and Titterington (2005) using Gaussian Processes experts for regression. For further models on mixture of experts for regression, the reader can be referred to for 3
ACCEPTED MANUSCRIPT
2. Mixture of experts for continuous data
CR IP T
example the book of Shi and Choi (2011). In this paper, we investigate semi-parametric models under the maximum likelihood estimation framework. The remainder of this paper is organized as follows. In Section 2 we briefly recall the normal MoE framework. In Section Then, in Section 3, we present the STMoE model and in Section 4 the parameter estimation technique using the ECM algorithm. We then investigate in Section 5 the use of the proposed model for non-linear regression and for prediction. We also show in Section 6 how the model can be used in a modelbased clustering prospective. In Section 7, we discuss the model selection. Section 8 is dedicated to the experimental study to assess the proposed model. Finally, in Section 9, conclusions are drawn and we open a future work.
AC
CE
PT
ED
M
AN US
Mixtures of experts (Jacobs et al., 1991; Jordan and Jacobs, 1994) are used in a variety of contexts including regression, classification and clustering. Here we consider the MoE framework for fitting (non-linear) regression functions and clustering of univariate continuous data . The aim of regression is to explore the relationship of an observed random variable Y given a covariate vector X ∈ Rp via conditional density functions for Y |X = x of the form f (y|x), rather than only exploring the unconditional distribution of Y . Thanks to their great flexibility, mixture models (McLachlan and Peel., 2000) has took much attention for non-linear regression problems and we distinguish in particular the classical mixture of regressions model Quandt (1972); Quandt and Ramsey (1978); Veaux (1989); Jones and McLachlan (1992); Gaffney and Smyth (1999); Viele and Tong (2002); Faria and Soromenho (2010); Hunter and Young (2012) and mixture of experts for regression analysis (Jacobs et al., 1991; Jordan and Jacobs, 1994; Young and Hunter, 2010). The univariate mixture of regressions model assumes that the observed pairs of data (x, y) where y ∈ R is the response for some covariate x ∈ Rp , are generated from K regression functions and are governed by a hidden categorical random variable Z indicating from which component each observation is generated. Thus, the mixture of regressions model decomposes the nonlinear regression model density f (y|x) into a convex weighted sum of K regression component models fk (y|x) and can be defined as follows: f (y|x; Ψ ) =
K X
πk fk (y|x; Ψ k )
(1)
k=1
where the πk ’s defined by πk = P(Z = k) and represent the non-negative mixing proportions that sum to 1. The model parameter vector is given by Ψ = (π1 , . . . , πK−1 , Ψ T1 , . . . , Ψ TK )T , Ψ k being the parameter vector of the kth component density.
2.1. The mixture of experts (MoE) model Although similar, the mixture of experts (Jacobs et al., 1991) differ from regression mixture models in many aspects. One of the main differences is that the MoE model 4
ACCEPTED MANUSCRIPT
CR IP T
consists in a fully conditional mixture while in the regression mixture, only the component densities are conditional. Indeed, the mixing proportions are constant for the regression mixture, while in the MoE, they are modeled as a function of the inputs, generally modeled by logistic or a softmax function. Mixture of experts (MoE) for regression analysis (Jacobs et al., 1991; Jordan and Jacobs, 1994) extend the model (1) by modeling the mixing proportions as function of some covariates r ∈ Rq . The mixing proportions, known as the gating functions in the context of MoE, are modeled by the multinomial logistic model and are defined by: exp (αTk r) πk (r; α) = P(Z = k|r; α) = PK T `=1 exp (α` r)
(2)
AN US
where r ∈ Rq is a covariate vector, αk is the q-dimensional coefficients vector associated with r and α = (αT1 , . . . , αTK−1 )T is the parameter vector of the logistic model, with αK being the null vector. Thus, the MoE model consists in a fully conditional mixture model where both the mixing proportions (the gating functions) and the component densities (the experts) are conditional on predictors (respectively r and x).
M
2.2. The normal mixture of experts (NMoE) model and maximum likelihood estimation In the case of mixture of experts for regression, it is usually assumed that the experts are normal, that is, follow a normal distribution. A K-component normal mixture of experts (NMoE) (K > 1) has the following formulation: f (y|r, x; Ψ ) =
K X
πk (r; α)N y; µ(x; β k ), σk2
ED
k=1
(3)
AC
CE
PT
which involves, in the semi-parametric case, component means defined as parametric (non-)linear regression functions µ(x; β k ). The NMoE model parameters are estimated by maximizing the observed data loglikelihood by using the EM algorithm (Dempster et al., 1977; Jacobs et al., 1991; Jordan and Jacobs, 1994; Jordan and Xu, 1995; Ng and McLachlan, 2004; McLachlan and Krishnan, 2008). Suppose we observe an i.i.d sample of n individuals (y1 , . . . , yn ) with their respective associated covariates (x1 , . . . , xn ) and (r 1 , . . . , xr ). Then, under the NMoE model, the observed data log-likelihood for the parameter vector Ψ is given by: log L(Ψ ) =
n X i=1
log
K X k=1
πk (r i ; α)N yi ; µ(x; β k ), σk2 .
(4)
The E-Step at the mth iteration of the EM algorithm for the NMoE model requires the calculation of the following posterior probability that the individual (yi , xi , r i ) belongs to expert k, given a parameter estimation Ψ (m) : (m) (m) πk (r; α(m) )N yi ; µk (xi ; βk ), σk2 (m) . (5) τik = P(Zi = k|yi , xi , r i ; Ψ (m) ) = f (yi |r i , xi ; Ψ (m) ) 5
ACCEPTED MANUSCRIPT
Then, the M-step calculates the parameter update Ψ (m+1) by maximizing the well-known Q-function (the expected complete-data log-likelihood), that is: Ψ (m+1) = arg max Q(Ψ ; Ψ (m) )
(6)
Ψ ∈Ω
µ(x; β k ) = β Tk x,
CR IP T
where Ω is the parameter space. For example, in the case of normal mixture of linear experts (NMoLE) where each expert’s mean has the flowing linear form: (7)
where β k ∈ Rp is the vector of regression coefficients of component k, the updates for each of the expert component parameters consist in analytically solving a weighted Gaussian linear regression problem and are given by: =
n hX i=1
σk2
(m+1)
=
Pn
(m) τik xi xTi
n i−1 X
(q)
τik yi xi ,
AN US
(m+1) βk
(m)
i=1 τik
i=1
yi − β Tk Pn (m) i=1 τik
(m+1)
xi
2
·
(8)
(9)
CE
PT
ED
M
For the mixing proportions, the parameter update α(m+1) cannot however be obtained in a closed form. It is calculated by IRLS (Jacobs et al., 1991; Jordan and Jacobs, 1994; Chen et al., 1999; Green, 1984; Chamroukhi et al., 2009a). However, the normal distribution is not adapted to deal with asymmetric and heavy tailed data. It is also known that the normal distribution is sensitive to outliers. In the proposal, we address these issues regarding the skewness, heavy tails and atypical observations in the data, by proposing a robust MoE modeling by using the skewt distribution, recently introduced by Azzalini and Capitanio (2003), for the expert components rather than the usually used normal one. The proposed skew t mixture of experts (STMoE) allows for simultaneously accommodating asymmetry and heavy tails in the data and is also robust to outliers. 3. The skew t mixture of experts (STMoE) model
AC
The proposed skew t mixture of experts (STMoE) model is a MoE model in which the expert components have a skew-t density, rather than the standard normal one as in the NMoE model. The skew-t distribution Azzalini and Capitanio (2003), which is a robust generalization the skew-normal distribution (Azzalini, 1985, 1986), as well as its stochastic and hierarchical representations, which will be used to define the proposed STMoE model, are recalled in the following section. 3.1. The skew t distribution Let us denote by tν (.) and Tν (.) respectively the pdf and cdf of the standard t distribution with degrees of freedom ν. The skew t distribution, introduced by Azzalini 6
ACCEPTED MANUSCRIPT
and Capitanio (2003), can be characterized as follows. Let U be an univariate random variable with a standard skew-normal distribution U ∼ SN(0, 1, λ) (which can be shortened as U ∼ SN(λ)) with pdf given by (A.1)). The skew-normal distribution is recalled in Appendix A. Then, let W be an univariate random variable independent of U and following the Gamma distribution, that is, W ∼ Gamma( ν2 , ν2 ). A random variable Y having the following representation: (10)
CR IP T
U Y = µ + σ√ W
follows the skew t distribution ST(µ, σ 2 , λ, ν) with location parameter µ, scale parameter σ, skewness parameter λ and degrees of freedom ν, whose density is defined by: ! s 2 ν + 1 (11) f (y; µ, σ 2 , λ, ν) = tν (dy ) Tν+1 λ dy σ ν + d2y
AC
CE
PT
ED
M
AN US
where dy = y−µ . From the hierarchical distribution of the skew-normal (A.3), a further σ hierarchical representation of the stochastic representation (10) of the skew t distribution is given by: 1 − δ2 2 σ , Yi |ui , wi ∼ N µ + δ|ui |, wi σ2 Ui |wi ∼ N(0, ), (12) wi ν ν Wi ∼ Gamma , . 2 2 3.2. The skew t mixture of experts (STMoE) model The skew proposed t mixture of experts (STMoE) model extends the skew t mixture model, which was first introduced by Lin et al. (2007a), to the MoE framework. In the skew-t mixture model of Lin et al. (2007a), the mixing proportions and the components means are constant, that is, they are not predictor-depending. In the proposed STMoE, however, we consider skew-t expert components in which both the mixing proportions and the mixture component means are predictor-depending. More specifically, we use polynomial regressors for the components, as well as multinomial logistic regressors for the mixing proportions. A K-component mixture of skew t experts (STMoE) is therefore defined by: f (y|r, x; Ψ ) =
K X
πk (r; α) ST(y; µ(x; β k ), σk2 , λk , νk )·
(13)
k=1
The parameter vector of the STMoE model is Ψ = (αT1 , . . . , αTK−1 , Ψ T1 , . . . , Ψ TK )T where Ψ k = (β Tk , σk2 , λk , νk )T is the parameter vector for the kth skew t expert component whose density is defined by ! s 2 ν + 1 f y|x; µ(x; β k ), σ 2 , λ, ν = tν (dy (x)) Tν+1 λ dy (x) (14) σ ν + d2y (x) 7
ACCEPTED MANUSCRIPT
CR IP T
k) where dy (x) = y−µ(x;β · σ It can be seen that, when the robustness parameter νk → ∞ for each k, the STMoE model (13) reduces to a skew-normal mixture of experts model (SNMoE) (see Chamroukhi (2015)). On the other hand, if the skewness parameter λk = 0 for each k, the STMoE model reduces to the t mixture of experts model (TMoE) (e.g., see Chamroukhi (2015, 2016)). Moreover, when νk → ∞ and λk = 0 for each k, the STMoE approaches the standard NMoE model (3). This therefore makes the STMoE very flexible as it generalizes the previously described MoE models to accommodate situations with asymmetry, heavy tails, and outliers.
ED
M
AN US
3.3. Hierarchical representation of the STMoE model By introducing the binary latent component-indicators Zik such that Zik = 1 iff Zi = k, Zi being the hidden class label of the ith observation, a hierarchical model for the STMoE model can be derived as follows. From the hierarchical representation (12) of the skew t distribution, a hierarchical model for the proposed STMoE model (13) can be derived from its stochastic representation (B.2) given in Appendix B, and is as follows: 1 − δk2 2 σ , Yi |ui , wi , Zik = 1, xi ∼ N µ(xi ; β k ) + δk |ui |, wi k σk2 , (15) Ui |wi , Zik = 1 ∼ N 0, wi ν ν k k , Wi |Zik = 1 ∼ Gamma 2 2 Z i |r i ∼ Mult 1; π1 (r i ; α), . . . , πK (r i ; α) .
PT
The variables Ui and Wi are treated as hidden in this hierarchical representation, which facilitates the inference scheme and will be used to derive the maximum likelihood estimation of the STMoE model parameters Ψ by using the ECM algorithm.
AC
CE
3.4. Identifiability of the STMoE model Jiang and Tanner (1999) have established that ordered, initialized, and irreducible MoEs are identifiable. Ordered implies that there exist a certain ordering relationship on the experts parameters Ψ k such that (αT1 , Ψ T1 )T ≺ . . . ≺ (αTK , Ψ TK )T ; initialized implies that wK , the parameter vector of the Kth logistic proportion, is the null vector, and irreducible implies that Ψ k 6= Ψ k0 for any k 6= k0. For the proposed STMoE, ordered implies that there exist a certain ordering relationship such that (β T1 , σ12 , λ1 , ν1 )T ≺ . . . ≺ 2 (β TK , σK , λK , νK )T ; initialized implies that wK is the null vector, as assumed in the model, and finally, irreducible implies that if k 6= k0, then one of the following conditions holds: β k 6= β k0 , σk 6= σk0 , λk 6= λk0 or νk 6= νk0 . Then, we can establish the identifiability of ordered and initialized irreducible STMoE models by applying Lemma 2 of Jiang and Tanner (1999), which requires the validation of the following nondegeneracy condition. The 2 set {ST(y; µ(x; β 1 ), σ12 , λ1 , ν1 ), . . . , ST(y; µ(x; β 4K ), σ4K , λ4K , ν4K )} contains 4K linearly independent functions of y, for any 4K distinct quadruplet (µ(x; β k ), σk2 , λk , νk ) for 8
ACCEPTED MANUSCRIPT
k = 1, . . . , 4K. Thus, via Lemma 2 of Jiang and Tanner (1999) we have any ordered and initialized irreducible STMoE is identifiable. 4. Maximum likelihood estimation of the STMoE model
log L(Ψ ) =
n X
log
i=1
K X
CR IP T
The unknown parameter vector Ψ of the STMoE model is estimated by maximizing the following observed-data log-likelihood given an observed i.i.d sample of n observations, that is, the responses (y1 , . . . , yn ) and the corresponding predictors (x1 , . . . , xn ) and (r 1 , . . . , r n ): πk (r i ; α)ST(y; µ(xi ; β k ), σk2 , λk , νk )·
k=1
(16)
log Lc (Ψ ) =
n X K X i=1 k=1
AN US
We perform this iteratively by a dedicated ECM algorithm. The complete data consist of the observations as well as the latent variables (u1 , . . . , un ) and (w1 , . . . , wn ), and the latent component labels (z1 , . . . , zn ). Then, from the hierarchical representation of the STMoE (15), the complete-data log-likelihood of Ψ is given by: h Zik log (P (Zi = k|r i )) + log (f (wi |Zik = 1)) +
M
log (f (ui |wi , Zik = 1)) + log (f (yi |ui , Zik = 1, xi )) = log L1c (α) +
K X
ED
k=1
log L2c (θ k ) + log L3c (νk )
i
(17)
where θ k = (β Tk , σk2 , λk )T and
log L2c (θ k ) =
Zik log πk (r i ; α),
PT
log L1c (α) =
n X K X
i=1 k=1 n h X
CE
i=1
Zik − log(2π) − log(σk2 ) −
log L3c (νk ) =
n X
AC
i=1
i wi d2ik wi ui δk dik wi u2i 1 log(1 − δk2 ) − + − , 2 2(1 − δk2 ) (1 − δk2 )σk 2(1 − δk2 )σk2
h ν ν ν ν ν i k k k k k Zik − log Γ + log + log(wi ) − wi . 2 2 2 2 2
4.1. The ECM algorithm for the STMoE model The ECM algorithm for the STMoE model starts with an initial parameter vector (0) Ψ and alternates between the E- and CM- steps until convergence.
9
ACCEPTED MANUSCRIPT
4.2. E-Step The E-Step of the CEM algorithm for the STMoE calculates the Q-function, that is the conditional expectation of the complete-data log-likelihood (17), given the observed data {yi , xi , r i }ni=1 and a current parameter estimation Ψ (m) , m being the current iteration. From (17), the Q-function is given by: K h X k=1
i Q2 (θ k , Ψ (m) ) + Q3 (νk , Ψ (m) ) ,
where ) =
n X K X
(m)
τik log πk (r i ; α),
i=1 k=1
Q2 (θ k ; Ψ
(m)
) =
n X
(m) τik
i=1
Q3 (νk ; Ψ (m) ) =
n X
"
log(σk2 )
− log(2π) −
# (m) (m) (m) δk dik e1,ik e2,ik wik d2ik 1 2 + − , − log(1 − δk ) − 2 2(1 − δk2 ) (1 − δk2 )σk 2(1 − δk2 )σk2
AN US
Q1 (α; Ψ
(m)
(18)
CR IP T
Q(Ψ ; Ψ (m) ) = Q1 (α; Ψ (m) ) +
h ν ν ν ν ν i (m) (m) k k k k k − log Γ + log − wik + e3,ik · 2 2 2 2 2
(m)
τik
i=1
It can be seen that computing the Q-function requires the following conditional expectations: (m)
= EΨ (m) [Zik |yi , xi , r i ] ,
τik
(m) wik (m) e1,ik (m) e2,ik (m) e3,ik
M
= EΨ (m) [Wi |yi , Zik = 1, xi , r i ] ,
ED
= EΨ (m) [Wi Ui |yi , Zik = 1, xi , r i ] , = EΨ (m) Wi Ui2 |yi , Zik = 1, xi , r i ,
= EΨ (m) [log(Wi )|yi , Zik = 1, xi , r i ] ·
CE
PT
Following the expressions of these conditional expectations given namely in the case of the standard skew t mixture model (Lin et al., 2007a), the conditional expectations for the case of the proposed STMoE model can be expressed similarly as: (m)
AC
τik
(m)
(m)
wik
(m)
where Mik = λk
(m) e1,ik
=
(m) δk
=
(m) 2(m) (m) (m) πk (r; α(m) ) ST yi ; µ(xi ; β k ), σk , λk , νk
=
(m)
dik
yi − µk (xi ; β
r
(m)
f (yi |r i , xi ; Ψ (m) ) r (m) νk +3 (m) ! Tν (m) +3 Mik (m) (m) νk +1 νk + 1 k × , (m) (m) (m) νk + d2ik Tν (m) +1 Mik
,
(19)
(20)
k
(m) νk +1 (m) (m) νk +d2ik
)
(m) wik
+
,
"
q (m) 1 − δk2
πf (yi |r i , xi ; Ψ (m) )
10
(m)
(m) d2ik (m) (m) νk (1 − δk2 )
+1
!−( νk2
+1) #
(21),
ACCEPTED MANUSCRIPT
=
(m) δk2
(m)
+
(m) e3,ik
δk
=
yi − µk (xi ; β
(m)
" 2 (m) (m) (m) ) wik + 1 − δk2 σk2
q (m) yi − µk (xi ; β (m) ) 1 − δk2
(m) wik
πf (yi |r i , xi ; Ψ (m) ) (m)
− log
νk
+ d2ik 2
(m)
!
×
(m)
−
νk (m)
νk
(m)
(m) d2ik (m) (m) νk (1 − δk2 )
+1
+ d2ik
(m)
+1
!
(m)
+ψ
(m)
νk
(m) tν (m) +1 Mik d2ik −1 k · + r × 3 (m) (m) (m) (m) M T (m) ik νk +1 νk + 1 νk + d2ik (m) (m)
+1) #
, (22)
2
+1
!
(23)
AN US
λk dik
!−( νk2
CR IP T
(m) e2,ik
M
We note that, for (23), we adopted a one-step-late (OSL) approach to compute the (m) conditional expectation e3,ik as described in Lee and McLachlan (2014), by setting the integral part in the expression of the corresponding conditional expectation given in (Lin et al., 2007a) to zero, rather than using a Monte Carlo approximation. We also mention that, for the multivariate skew t mixture models, recently Lee and McLachlan (2015) presented a series-based truncation approach, which exploits an exact representation of this conditional expectation and which can also be used in place of (23).
CE
PT
ED
4.3. M-Step The M-step maximizes the Q-function (18) with respect to Ψ and provides the parameter vector update Ψ (m+1) . From (18), it can be seen that the maximization of Q can be performed by separately maximizing Q1 with respect to the parameters α of the mixing proportions, and for each expert k (k = 1, . . . , K), Q2 with respect to (β Tk , σk2 )T and λk , and Q3 with respect to νk . The maximization of Q2 and Q3 is carried out by conditional maximization (CM) steps by updating (β k , σk2 ) and then updating (λ, νk ) with the given updated parameters. This leads to the following CM steps. On the (m + 1)th iteration of the M-step, the STMoE model parameters are updated as follows.
AC
CM-Step 1. Calculate α(m+1) maximizing the function Q1 (α; Ψ (m) ): α(m+1) = arg max Q1 (α; Ψ (m) ). α
(24)
Contrarily to the case of the standard mixture model and mixture of regression models, this maximization in the case of the proposed STMoE does not exist in closed form. It is performed iteratively by Iteratively Reweighted Least Squares (IRLS).
11
ACCEPTED MANUSCRIPT
The Iteratively Reweighted Least Squares (IRLS) algorithm:. The IRLS algorithm is used to maximize Q1 (α, Ψ (m) ) with respect to the parameter vector α of the softmax function in the M step at each iteration m of the ECM algorithm. The IRLS is a Newton-Raphson algorithm, which consists in starting with a vector α(0) , and, at the l + 1 iteration, updating the estimation of α as follows: (l)
=α
(m)
(m)
2
h ∂ 2 Q (α, Ψ (m) ) i−1 ∂Q1 (α, Ψ (m) ) 1 − ∂α∂αT ∂α α=α(l) α=α(l)
(25)
CR IP T
α
(l+1)
) ) 1 (α,Ψ are respectively the Hessian matrix and the gradient and ∂Q1 (α,Ψ where ∂ Q∂α∂α T ∂α vector of Q1 (α, Ψ(m) ). At each IRLS iteration the Hessian and the gradient are evaluated at α = α(l) and are computed similarly as in Chamroukhi et al. (2009b). The parameter update α(m+1) is taken at convergence of the IRLS algorithm (25). Then, for k = 1 . . . , K, T (m+1)
(m+1)
=
n hX
(q)
i=1
(m+1)
=
n i−1 X i=1
(m)
i=1 τik
(q)
τik
(m)
(m) (m+1)
wik yi − e1,ik δk
xi ,
(26)
h 2 i (m+1) (m+1) (m) (m) (m+1) (m) xi − 2δk xi ) + e2,ik e1,ik (yi − β Tk wik y i − β Tk (· 27) (m) (m) Pn 2 1 − δk2 i=1 τik
ED
σk2
Pn
(m)
τik wik xi xTi
M
(m+1)
βk
AN US
CM-Step 2. Calculate (β k , σk2 )T by maximizing Q2 (θ k ; Ψ (m) ) w.r.t (β Tk , σk2 )T . For the skew t mixture of linear experts (STMoLE) case, where the expert means are linear regressors, that is, of the form (7), this maximization can be performed in a closed form and provides the following updates:
n X
(m) τik
+ (1 +
CE
δk (1 −
δk2 )
PT
CM-Step 3. The skewness parameters λk are updated by maximizing Q2 (θ k ; Ψ (m) ) w.r.t (m+1) (m+1) λk , with β k and σk2 fixed at the update β k and σk2 , respectively. It can be easily (m+1) shown that the maximization to obtain δk (k = 1, . . . , K) consists in solving the following equation in δk :
i=1
δk2 )
(m+1) (m) e1,ik (m) dik τik (m+1) σk i=1
n X
− δk
n X
(m) τik
i=1
h (m+1) (m) wik d2ik +
(m) e2,ik i
σk2
(m+1)
= 0·(28)
AC
CM-Step 4. Similarly, the degrees of freedom νk are updated by maximizing Q3 (νk ; Ψ (m) ) (m+1) (m+1) (m+1) w.r.t νk with β k and σk2 fixed at β k and σk2 , respectively. An update νk is calculated as solution of the following equation in νk : Pn (m) (m) (m) ν ν e3,ik − wik i=1 τik k k −ψ = 0. (29) + log +1+ Pn (m) 2 2 τik i=1
The two scalar non-linear equations (28) and (29) can be solved similarly as in the TMoE model, that is with a root finding algorithm, such as Brent’s method (Brent, 1973). 12
ACCEPTED MANUSCRIPT
M
5. Prediction using the STMoE
AN US
CR IP T
As mentioned before, one can see that, when the robustness parameter νk → ∞ for all the components, the parameter updates for the STMoE model correspond to those of the SNMoE model (see Chamroukhi (2015)). On the other hand, when the skewness parameters λk = 0, the STMoE parameter updates correspond to those of the TMoE model (Chamroukhi (2015)). Finally, when both the degrees of freedom νk → ∞ and the skewness λk = 0, we obtain the parameter updates of the standard NMoE model. The STMoE therefore provides a more general framework for inferring flexible MoE models and attempts to simultaneously accommodate data with asymmetric distribution heavy tails and outliers. Here the ECM algorithm is used to infer the STMoE model parameters. We note that there is a good generalization of the EM algorithms for non-probabilistic problems which does not require to construct a complete-data likelihood nor a probabilistic interpretation of the maximization problem, that is the Minorization-Maximization (MM) algorithm (Hunter and Lange, 2004). The MM algorihm, used in the MoE framewrok namely by Gormley and Murphy (2008); Nguyen and McLachlan (2016) can also be a good alternative to the ECM algorithm used here. On the other hand, the ECM algorithm divides the space of model-parameters to perform sequentially the optimization in each sub-space. It may also be convenient to divide the space of the hidden variables and to alternate the optimisation, cyclically within each sub-space. This scheme is known as the Alternating ECM (AECM) algorithm (Meng and van Dyk, 1997).
AC
CE
PT
ED
The goal in regression is to be able to make predictions for the response variable(s) given some new value of the predictor variable(s) on the basis of a model trained on a set of training data. In regression analysis using mixture of experts, the aim is therefore to predict the response y given new values of the predictors (x, r), on the basis of a ˆ inferred from a set of training data, MoE model characterized by a parameter vector Ψ here, by maximum likelihood via EM. These predictions can be expressed in terms of the predictive distribution of y, which is obtained by substituting the maximum likelihood ˆ into (1)-(2) to give: parameter Ψ ˆ) = f (y|x, r; Ψ
K X
ˆ k ). ˆ k (y|x; Ψ πk (r; α)f
k=1
Using f , we might then predict y for a given set of x’s and r’s as the expected value under f , that is by calculating the prediction yˆ = EΨˆ (Y |r, x). We thus need to compute the expectation of the mixture of experts model. It is easy to show (see for example Section 1.2.4 in Fr¨ uhwirth-Schnatter (2006)) that the mean and the variance of a mixture
13
ACCEPTED MANUSCRIPT
of experts distribution of the form (5) are respectively given by EΨˆ (Y |r, x) =
k=1
K X k=1
ˆ n )EΨˆ (Y |Z = k, x), πk (r; α
(30)
2 2 ˆ n ) EΨˆ (Y |Z = k, x) + VΨˆ (Y |Z = k, x) − EΨˆ (Y |r, x) ,(31) πk (r; α
CR IP T
VΨˆ (Y |r, x) =
K X
where EΨˆ (Y |Z = k, x) and VΨˆ (Y |Z = k, x) are respectively the component-specific (expert) means and variances. The mean and the variance for the two MoE models described here are given as follows.
AN US
NMoE. For the NMoE model, the normal expert means and variances are respectively ˆ T x and V ˆ (Y |Z = k, x) = σ ˆk2 . Then, from (30) it follows given by EΨˆ (Y |Z = k, x) = β k Ψ that the mean of the NMoE is given by EΨˆ (Y |r, x) =
K X
ˆ T x. ˆ n )β πk (r; α k
(32)
k=1
M
STMoE. The mean and the variance for a skew t random variable, for this scalar case, can be easily computed as in Section 4.2 in Azzalini and Capitanio (2003) for a non-zero location parameter. Thus, for the STMoE model, the expert means for νˆk > 1, are given by ˆT x + σ ˆk δˆk ξ(ˆ νk ) EΨˆ (Y |Z = k, x) = β k
CE
thus given by:
ν ˆk 1 νˆk Γ 2 −2 . π Γ νˆk 2
PT
where ξ(ˆ νk ) =
q
ED
and the expert variances for νˆk > 2 are given by νˆk 2 2 ˆ − δk ξ (ˆ νk ) σ ˆk2 , VΨˆ (Y |Z = k, x) = νˆk − 2 Then, following (30), the mean of the proposed STMoE is
AC
EΨˆ (Y |r, x) =
K X k=1
T ˆ x+σ ˆ ˆ β πk (r; α) ˆ δ ξ(ˆ ν ) . k k k k
(33)
Finally, the variance for each MoE model is obtained by using (31) with the specified corresponding means and variances calculated in the above. 6. Model-based clustering using the STMoE
The MoE models can also be used for a model-based clustering perspective to provide a partition of the regression data into K clusters. Model-based clustering using the proposed STMoE consists in assuming that the observed data {xi , r i , yi }ni=1 are generated 14
ACCEPTED MANUSCRIPT
K
zˆi = arg max τˆik k=1
CR IP T
from a K component mixture of skew t experts, with parameter vector Ψ where the STMoE components are interpreted as clusters and hence associated to clusters. The problem of clustering therefore becomes the one of estimating the MoE parameters Ψ , which is performed here by using the dedicated ECM algorithm presented in section 4.1. Once the parameters are estimated, the provided posterior component memberships τik given by (19) represent a fuzzy partition of the data. A hard partition of the data can then be obtained from the posterior memberships by applying the MAP rule, that is: (34)
where zˆi represents the estimated cluster label for the ith individual. 7. Model selection for the STMoE
M
AN US
One of the issues in mixture model-based clustering is model selection. The problem of model selection for the STMoE models presented here in its general form is equivalent to the one of choosing the optimal number of experts K, the degree p of the polynomial regression and the degree q for the logistic regression. The optimal value of the triplet (K, p, q) can be computed by using some model selection criteria such as the Akaike Information Criterion (AIC) (Akaike, 1974), the Bayesian Information Criterion (BIC) (Schwarz, 1978) or the Integrated Classification Likelihood criterion (ICL) (Biernacki et al., 2000), etc. The AIC and BIC are are penalized observed log-likelihood criteria which can be defined as functions to be maximized and are respectively given by:
ED
ˆ − ηΨ , AIC(K, p, q) = log L(Ψ) ˆ ) − ηΨ log(n) . BIC(K, p, q) = log L(Ψ 2
AC
CE
PT
The ICL criterion consists in a penalized complete-data log-likelihood and can be expressed as follows: ˆ ) − ηΨ log(n) . ICL(K, p, q) = log Lc (Ψ 2 ˆ ) and log Lc (Ψ ˆ ) are respectively the incomplete (observed) data In the above, log L(Ψ log-likelihood and the complete data log-likelihood, obtained at convergence of the ECM algorithm for the corresponding MoE model and ηΨ is the number of free model parameters. The number of free parameters ηΨ is given by ηΨ = K(p + q + 3) − q − 1 for the NMoE model and ηΨ = K(p + q + 5) − q − 1 for the proposed STMoE model. Indeed, for each component, the STMoE have two additional parameters to be estimated, which are the robustness and the skewness parameters. However, note that in MoE it is common to use mixing proportions modeled as logistic transformation of linear functions of the covariates, that is the covariate vector in (2) is given by r i = (1, ri )T (corresponding to q = 2), ri being an univariate covariate variable. This is also adopted in this work. Moreover, for the case of linear experts, that is when the experts are linear regressors with parameter vector β k for which the 15
ACCEPTED MANUSCRIPT
CR IP T
corresponding covariate vector xi in (7) is given by xi = (1, xi )T (corresponding to p = 2), ri being an univariate covariate variable, the model selection reduces to choosing the number of experts K. Here we mainly consider this linear case. However, for a general use of the proposed STMoE model, even though the model selection criteria such as AIC, BIC, ICL can be easily computed, the direct model selection is difficult due to the large model space dimension νΨ . A searching strategy is then required to optimise the way of exploring the model space. 8. Experimental study
This section is dedicated to the evaluation of the proposed approach on simulated data and real-world data . We evaluated the performance of proposed ECM algorithm1 for the STMoE model in terms of modeling, robustness to outliers and clustering.
M
AN US
8.1. Initialization and stopping rules The parameters αk (k = 1, . . . , K − 1) of the mixing proportions are initialized randomly, including an initialization at the null vector for one run (corresponding to equal mixing proportions). Then, the common parameters (β k , σk2 ) (k = 1, . . . , K) are initialized from a random partition of the data into K clusters. This corresponds to fitting a normal mixture of experts where the initial values of the parameters are respectively given by (8) and (9) with the posterior memberships τik replaced by the hard assignments Zik issued from the random partition. For the STMoE model, the robustness parameters νk (k = 1, . . . , K) is initialized randomly in the range [1, 200] and the skewness parameters λk (k = 1, . . . , K) is initialized by randomly initializing the parameter δk in (−1, 1) from the relation λk = √ δk 2 . Then, the proposed ECM
ED
1−δk
PT
algorithm for each model is stopped when the relative variation of the observed-data (m+1) )−log L(Ψ (m) ) log-likelihood log L(Ψ| log L(Ψ reaches a prefixed threshold (for example = 10−6 ). (m) )| For each model, this process is repeated 10 times and the solution corresponding the highest log-likelihood is finally selected.
AC
CE
8.2. Experiments on simulation data sets In this section we perform an experimental study on simulated data sets to apply and assess the proposed model. Two sets of experiments have been performed. The first experiment aims at observing the effect of the sample size on the estimation quality and the second one aims at observing the impact of the presence of outliers in the data on the estimation quality, that is the robustness of the models. 8.2.1. Experiment 1 For this first experiment on simulated data, each simulated sample consisted of n observations with increasing values of the sample size n : 50, 100, 200, 500, 1000. The 1
The codes have been implemented in Matlab and are available upon request from the author.
16
ACCEPTED MANUSCRIPT
component 1 component 2
CR IP T
simulated data are generated from a two component mixture of linear experts, that is K = 2, p = q = 1. The covariate variables (xi , r i ) are simulated such that xi = r i = (1, xi )T where xi is simulated uniformly over the interval (−1, 1). We consider each of the two models for data generation (NMoE and STMoE), that is, given the covariates, the response yi |{xi , r i ; Ψ } is simulated according to the generative process of the models (3) and (15). For each generated sample, we fit each of the two models. Thus, the results are reported for the two models with data generated from each of them. We consider the mean square error (MSE) between each component of the true parameter vector and ˆ j ||2 . The squared errors are averaged on the estimated one, which is given by ||Ψ j − Ψ 100 trials. The used simulation parameters Ψ for each model are given in Table 1. parameters β 1 = (0, 1)T σ1 = 0.1 β 2 = (0, −1)T σ2 = 0.1
(0, 10)T
α1 = α2 = (0, 0)T
λ1 = 3 λ2 = −10
ν1 = 5 ν2 = 7
AN US
Table 1: Parameter values used in simulation.
α11
β10
525 457 247 31 8.5
5737 1815 785 565 68
0.965 0.847 0.816 0.363 0.261
β11
β20
ED
α10
2.440 0.852 0.348 0.091 0.076
PT
param. n 50 100 200 500 1000
M
8.2.2. Obtained results Table 2 shows the obtained results in terms of the MSE for the STMoE. One can observe that, for the proposed model, the parameter estimation error is decreasing as n increases, which is related the convergence property of the maximum likelihood estimator. One can also observe that the error decreases significantly for n ≥ 500, especially for the regression coefficients and the scale parameters.
4.388 0.742 0.473 0.314 0.233
β21
σ1
σ2
λ1
λ2
ν1
ν2
0.667 0.660 0.556 0.398 0.116
0.954 0.844 0.362 0.091 0.026
0.608 0.303 0.297 0.061 0.002
3115 2013 700 7.8 2.8
16095 7844 3847 1078 554
15096 5360 3135 223 49.4
4643 263 167 8.6 0.79
CE
Table 2: MSE ×103 between each component of the estimated parameter vector of the STMoE model and the actual one for a varying sample size n.
AC
In addition to the previously showed results, we plotted in Figures 1 and 2 the estimated quantities provided by applying respectively the NMoE model and the proposed STMoE model, and their true counterparts for n = 500 for the same the data set which was generated according the NMoE model. The upper-left plots show the estimated mean function, the estimated expert component mean functions, and the corresponding true ones. The upper-right plots show the estimated mean function with the estimated confidence region computed as plus and minus twice the estimated (pointwise) standard deviation of the model as presented in Section 5, and their true counterparts. The bottom-left plots show the true expert component mean functions and the true partition, and the bottom-right plots show their estimated counterparts. 17
ACCEPTED MANUSCRIPT
1
0.5
0.5
0
0
−0.5
−0.5
−1
−2 −1
−1
data True mean (NMoE) True Experts Estimated mean (NMoE) Estimated Experts −0.8
−0.6
−0.4
−0.2
−1.5
0 x
0.2
0.4
0.6
0.8
1
−0.8
−0.6
−0.4
−0.2
0 x
0.2
0.4
0.6
0.8
1
0
0.2
0.4
0.6
0.8
1
0.5
0
0
−0.5
M
−0.5
Actual expert mean 1 Actual expert mean 2 Actual cluster 1 Actual cluster 2 −0.8
−0.6
−0.4
ED
−1
−2 −1
−2 −1
1
1
0.5
−1.5
data True mean (NMoE) True conf. regions Estimated mean (NMoE) Estimated conf. regions
AN US
−1.5
CR IP T
1
y
y
One can clearly see that the estimations provided by the proposed model are very close to the true ones which correspond to those of the NMoE model in this case. This shows that the proposed algorithm performs well and provides an additional support to the fact that the corresponding proposed STMoE model is good generalization of the normal mixture of experts (NMoE), as it clearly approaches the NMoE as shown in this simulated examples.
−0.2
0
0.2
0.4
0.6
0.8
−1
−1.5
−2 −1
1
Estimated expert mean 1 Estimated expert mean 2 Estimated cluster 1 Estimated cluster 2 −0.8
−0.6
−0.4
−0.2
PT
Figure 1: Fitted NMoE model to a data set generated according to the NMoE model.
AC
CE
Figure 3 shows the true and estimated MoE mean functions and component mean functions by fitting the proposed STMoE model to a simulated data set of n = 500 observations. Each model was considered for data generation. The upper plot corresponds to the NMoE model and the bottom plot corresponds to the STMoE model. Finally, Figure 4 shows the corresponding true and estimated partitions. Again, one can clearly see that both the estimated models are precise. The fitted functions are close to the true ones. In addition, one can also see that the partitions estimated by the STMoE model are close the actual partitions. The proposed STMoE model can therefore be used as alternative to the NMoE model for both regression and model-based clustering. 8.2.3. Experiment 2 In this experiment we examine the robustness of the proposed STMoE model to outliers versus the standard NMoE one. For that, we considered each of the two models (NMoE and STMoE) for data generation. For each generated sample, each of the two models in considered for the inference. The data were generated exactly in the same 18
1
1
0.5
0.5
0
0
−0.5
−0.5
y
−1
−1.5
−2 −1
−1
data True mean (NMoE) True Experts Estimated mean (STMoE) Estimated Experts −0.8
−0.6
−0.4
−0.2
−1.5
0 x
0.2
0.4
0.6
0.8
−2 −1
1
1
0.5
0.5
0
0
−0.5
−0.5
−1
−1.5
−2 −1
−0.8
−0.6
−0.4
−0.2
0 x
AN US
1
data True mean (NMoE) True conf. regions Estimated mean (STMoE) Estimated conf. regions
CR IP T
y
ACCEPTED MANUSCRIPT
0.2
0.4
0.6
0.8
1
0.2
0.4
0.6
0.8
1
−1
−1.5
Actual expert mean 1 Actual expert mean 2 Actual cluster 1 Actual cluster 2 −0.8
−0.6
−0.4
−0.2
0
0.2
0.4
0.6
0.8
−2 −1
1
Estimated expert mean 1 Estimated expert mean 2 Estimated cluster 1 Estimated cluster 2
−0.8
−0.6
−0.4
−0.2
0
M
Figure 2: Fitted STMoE model to a data set generated according to the NMoE model.
CE
PT
ED
way as in Experiment 1, except for some observations which were generated with a probability c from a class of outliers. We considered the same class of outliers as in Nguyen and McLachlan (2016), that is the predictor x is generated uniformly over the interval (−1, 1) and the response y is set the value −2. We apply the MoE models by setting the covariate vectors as before, that is, x = r = (1, x)T . We considered varying probability of outliers c = 0%, 1%, 2%, 3%, 4%, 5% and the sample size of the generated data is n = 500. An example of simulated sample containing 5% outliers is shown in Figure 5. As a criterion of evaluation of the impact of the outliers on the quality of the results, we considered the MSE between the P true regression mean function and the estimated one. This MSE is calculated as n1 ni=1 ||EΨ (Yi |r i , xi ) − EΨˆ (Yi |r i , xi )||2 where the expectations are computed as in Section 5.
AC
8.2.4. Obtained results Table 3 shows, for each of the two models, the results in terms of mean squared error (MSE) between the true mean function and the estimated one, for an increasing number of outliers in the data. First, one can see that, when there is no outliers (c = 0%) and when the data follow a NMoE distribution, the error of fitting a NMoE is very slightly less than the one of fitting the proposed STMoE model. The STMoE then is still competitive. However, when the data do not contain outliers and follow a STMoE distribution, fitting a NMoE is restrictive since the error of this one is high compared to the one obtained by fitting a STMoE model. More importantly, it can be seen that, 19
1
1
0.5
0.5
0
0
−0.5
−0.5
y
−1
−1.5
data True mean (NMoE) True Experts Estimated mean (NMoE) Estimated Experts −0.8
−0.6
−0.4
−0.2
−1.5
0 x
0.2
0.4
0.6
0.8
−2 −1
1
1
0.5
0.5
0
0
−0.5
−0.5
y
1
−1
−1.5
−2 −1
data True mean (NMoE) True conf. regions Estimated mean (NMoE) Estimated conf. regions −0.8
−0.6
−0.4
−0.2
0 x
AN US
y
−2 −1
−1
CR IP T
y
ACCEPTED MANUSCRIPT
0.2
0.4
0.6
0.8
1
0.2
0.4
0.6
0.8
1
−1
data True mean (STMoE) True Experts Estimated mean (STMoE) Estimated Experts −0.8
−0.6
−0.4
−0.2
−1.5
0 x
0.2
0.4
0.6
0.8
−2 −1
1
data True mean (STMoE) True conf. regions Estimated mean (STMoE) Estimated conf. regions
−0.8
−0.6
−0.4
−0.2
0 x
M
Figure 3: The true and estimated mean function and expert mean functions by fitting the standard NMoE model (up) and the proposed STMoE model (bottom) to a simulated data set of n = 500 observations generating according to the corresponding model.
AC
CE
PT
ED
as expected, when there is outliers in the data, including the situations with only few atypical data points (1% and 2 %), the NMoE does not provide an adapted fit and is clearly outperformed by the proposed STMoE model. This includes the two situations, that is, including when the data are not generated according to the STMoE model. The errors of the NMoE model are high compared to those of the STMoE. This confirms that te STMoE is much more robust to outliers compared to the normal mixture of experts because the expert components in the STMoE model follow the robust skew t distribution. The NMoE is sensitive to outliers. On the other hand it can be seen that, when the number of outliers is increasing, increase in the error of the NMoE is more pronounced compared to the one of STMoE model. The error for the STMoE may indeed slightly increase, remain stable or even decrease in some situations. This provides an additional support to the expected robustness of the STMoE compared to the NMoE.
20
ACCEPTED MANUSCRIPT
1
0.5
0.5
0
0
−0.5
−0.5
−1
−1
−1.5
−2 −1
−1.5
Actual expert mean 1 Actual expert mean 2 Actual cluster 1 Actual cluster 2 −0.8
−0.6
−0.4
−0.2
0
0.2
0.4
0.6
0.8
−2 −1
1
1
0.5
0.5
0
0
−0.5
−0.5
−1
−0.8
−0.6
−0.4
−0.2
0
AN US
1
Estimated expert mean 1 Estimated expert mean 2 Estimated cluster 1 Estimated cluster 2
CR IP T
1
0.2
0.4
0.6
0.8
1
0.2
0.4
0.6
0.8
1
−1
−1.5
−2 −1
−1.5
Actual expert mean 1 Actual expert mean 2 Actual cluster 1 Actual cluster 2 −0.8
−0.6
−0.4
−0.2
0
0.2
0.4
0.6
0.8
−2 −1
1
Estimated expert mean 1 Estimated expert mean 2 Estimated cluster 1 Estimated cluster 2
−0.8
−0.6
−0.4
−0.2
0
0%
1%
2%
3%
4%
5%
0.178 0.258 0.710 0.280
1.057 0.741 0.7238 0.186
1.241 0.794 1.048 0.447
3.631 0.696 6.066 0.600
13.25 0.697 12.45 0.509
28.96 0.626 31.64 0.602
PT
ED
c Model NMoE NMoE STMoE NMoE STMoE STMoE
M
Figure 4: The true and estimated partitions by fitting the standard NMoE model (up) and the proposed STMoE model (bottom) to the simulated data sets shown in Figure 3.
CE
Table 3: MSE ×103 between the estimated mean function and the true one for each of the two models for a varying probability c of outliers for each simulation. The first column indicates the model used for generating the data and the second one indicates the model used for inference.
AC
Then, in order to highlight the robustness to noise of the TMoE and STMoE models, in addition to the previously shown numerical results, figures 5 and 6 show an example of results obtained on the same data set by, respectively, the NMoE and the STMoE. The data are generated by the NMoE model and contain c = 5% of outliers. In this example, we clearly see that the NMoE model is severely affected by the outliers. It provides a rough fit especially for the second component whose estimation is corresponds to a rough approximation due to the atypical data. However, one can see that the STMoE model clearly provides a precise fit; the estimated mean function and expert components are very close to the true ones. The STMoE is robust to outliers, in terms of estimating the true model as well as in terms of estimating the true partition 21
ACCEPTED MANUSCRIPT
1
1
data True mean (NMoE) True Experts Estimated mean (NMoE) Estimated Experts
0.5
CR IP T
of the data (as shown in the middle plots of the data). Notice that for the STMoE, the confidence regions are not shown because for this situation the estimated degrees of freedom are less than 2 (1.6097 and 1.5311) for the STMoE); Hence the variance for this model in that case is not defined (see Section 5). The STMoE model provides indeed components with small degrees of freedom corresponding to highly heavy tails, which allow to handle outliers in this noisy case. While the variance is not estimable here, resampling techniques can be used to evaluate it, such as the techniques of O’Hagan et al. (2015) for producing standard errors and confidence intervals for mixture parameters in model-based clustering.
0.5
-0.5
-0.5
-1
-1
-1.5
-1.5
-2 -1
AN US
y
0
y
0
data True mean (NMoE) True conf. regions Estimated mean (NMoE) Estimated conf. regions
-2
-0.8
-0.6
-0.4
-0.2
0
0.2
0.4
0.6
0.8
x
1
-1
-0.8
-0.6
-0.4
-0.2
0
0.2
0.4
0.6
0.8
1
0.2
0.4
0.6
0.8
1
0.2
0.4
0.6
0.8
1
x
1
1
Actual expert mean 1 Actual expert mean 2 Actual cluster 1 Actual cluster 2
0.5
M
0.5
0
0
y
-0.5
ED
y
-0.5
Estimated expert mean 1 Estimated expert mean 2 Estimated cluster 1 Estimated cluster 2
-1
-1.5
-1
PT
-2 -0.8
-0.6
-0.4
-0.2
0
0.2
0.4
0.6
-1
-1.5
-2 0.8
1
-1
-0.8
-0.6
-0.4
-0.2
x
−50
NMoE log−likelihood
0.9
CE
−100
0.8
−150
Mixing probabilities
0.7
−200
−250
AC
Observed data log−likelihood
0
x 1
−300
0.6 0.5 0.4 0.3
−350
0.2
−400
−450 0
0.1
2
4
6
8 10 EM iteration number
12
14
16
0 −1
18
−0.8
−0.6
−0.4
−0.2
0 x
Figure 5: Fitted NMoE model to a data set of n = 500 observations generated according to the NMoE model and including 5% of outliers.
22
ACCEPTED MANUSCRIPT
1
1
data True mean (NMoE) True Experts Estimated mean (STMoE) Estimated Experts
0.5
0.5
-0.5
-0.5
-1
-1
-1.5
-1.5
-2
-2
-1
-0.8
-0.6
-0.4
-0.2
0
0.2
0.4
0.6
0.8
1
-1
-0.8
-0.6
-0.4
x 1 Actual expert mean 1 Actual expert mean 2 Actual cluster 1 Actual cluster 2
0
0.2
0.4
0.6
0.8
1
0.2
0.4
0.6
0.8
1
0.2
0.4
0.6
0.8
1
Estimated expert mean 1 Estimated expert mean 2 Estimated cluster 1 Estimated cluster 2
0.5
-0.5
-0.5
-1
AN US
y
0
y
0
-1
-1.5
-1.5
-2
-2
-1
-0.8
-0.6
-0.4
-0.2
0
0.2
0.4
0.6
0.8
x
1
-1
-0.6
-0.4
-0.2
0
x
STMoE log−likelihood
0.9
100
ED
0
−100
PT
−200
10
20
30
40 50 EM iteration number
60
70
0.8 0.7
Mixing probabilities
M
200
−300 0
-0.8
1
300
Observed data log−likelihood
-0.2
x
1
0.5
CR IP T
y
0
y
0
data True mean (NMoE) True conf. regions Estimated mean (STMoE) Estimated conf. regions
0.6 0.5 0.4 0.3 0.2 0.1
80
0 −1
90
−0.8
−0.6
−0.4
−0.2
0 x
CE
Figure 6: Fitted STMoE model to a data set of n = 500 observations generated according to the NMoE model and including 5% of outliers.
AC
8.3. Application to two real-world data sets In this section, we consider an application to two real-world data sets: the tone perception data set and the temperature anomalies data set shown in Figure 7.
23
ACCEPTED MANUSCRIPT
Tone perception data
Temperature anomaly data
1
3.4 3.2 3 y: temperature anomaly
y: streach ratio
2.8 2.6 2.4 2.2 2
0.5
0
1.6
−0.5
1.4 1.4
1.6
1.8
2
2.2 x: tuned
2.4
2.6
2.8
3
1880
1900
CR IP T
1.8
1920
1940 1960 x: year
1980
2000
Figure 7: Scatter plots of the tone perception data and the temperature anomalies data.
AC
CE
PT
ED
M
AN US
8.3.1. Tone perception data set The first analyzed data set is the real tone perception data set1 which goes back to Cohen (1984). It was recently studied by (Bai et al., 2012; Ingrassia et al., 2012) and Song et al. (2014) by using robust regression mixture models based on, respectively, the t distribution and the Laplace distribution. In the tone perception experiment, a pure fundamental tone was played to a trained musician. Electronically generated overtones were added, determined by a stretching ratio (“stretch ratio” = 2) which corresponds to the harmonic pattern usually heard in traditional definite pitched instruments. The musician was asked to tune an adjustable tone to the octave above the fundamental tone and a “tuned” measurement gives the ratio of the adjusted tone to the fundamental. The obtained data consists of n = 150 pairs of “tuned” variables, considered here as predictors (x), and their corresponding “strech ratio” variables considered as responses (y). To apply the proposed MoE models, we set the response yi (i = 1, . . . , 150) as the “strech ratio” variables and the covariates xi = r i = (1, xi )T where xi is the “tuned” variable of the ith observation. We also follow the study in Bai et al. (2012) and Song et al. (2014) by using two mixture components. Model selection results are given later in Table 5. Figure 8 shows the scatter plots of the tone perception data and the linear expert components of the fitted NMoE model and the proposed STMoE model. One can observe that we obtain a good fit with the two models. The NMoE fit differs very slightly from the one of the STMoE. The two regression lines may correspond to correct tuning and tuning to the first overtone, respectively, as analyzed in Bai et al. (2012). 1
Source: http://artax.karlin.mff.cuni.cz/r-help/library/fpc/html/tonedata.html
24
ACCEPTED MANUSCRIPT
NMoE
3.2 3
Cluster 1 Cluster 2 Expert mean 1 Expert mean 2
3.4 3.2 3
2.8
2.8
2.6
2.6
2.4
2.4
y
2.2
2.2
2
2
1.8
1.8
1.6
1.6
1.4
1.4 1.4
1.6
1.8
2
2.2 x
2.4
2.6
2.8
3
1.4
1.6
1.8
CR IP T
y
STMoE
Cluster 1 Cluster 2 Expert mean 1 Expert mean 2
3.4
2
2.2 x
2.4
2.6
2.8
3
Figure 8: The fitted MoLE to the original tone data set with the NMoE model (left) and the STMoE model (right). The predictor x is the actual tone ratio and the response y is the perceived tone ratio.
NMoE
50
3
4
5 6 7 EM iteration number
100 80 60 40 20 0 −20
8
9
−40 0
10
CE
2
STMoE log−likelihood
120
Observed data log−likelihood
M
ED
100
STMoE
140
NMoE log−likelihood
PT
Observed data log−likelihood
150
0 1
AN US
Figure 9 shows the log-likelihood profiles for each of the two models. It can namely be seen that training the skew t mixture of experts may take more iterations than the normal MoE model. The STMoE has indeed more parameters to estimate (additional skewness and robustness parameters). However, in terms of computing time, the algorithm is fast and converges in only few seconds (around 10 seconds for this example) on a personal laptop with 2,9 GHz processor and 8 GB memory.
200
400
600 800 EM iteration number
1000
1200
1400
Figure 9: The log-likelihood during the EM iterations when fitting the MoLE models to the original tone data set with the NMoE model (left) and the STMoE model (right).
AC
The values of estimated parameters for the tone perception data set are given in Table 4. One can see that the regression coefficients are very similar for the two models. One can also see that the STMoE model retrieves a skewed component and with high degrees of freedom compared to the other component. This one may be seen as approaching the one of a skew-normal MoE model, while the second one in approaching a t distribution, that is the one of a t-MoE model.
25
ACCEPTED MANUSCRIPT
param. model NMoE STMoE
α10
α11
β10
β11
β20
β21
σ1
σ2
λ1
λ2
ν1
ν2
-2.690 -3.044
0.796 0.824
-0.029 -0.058
0.995 0.944
1.913 1.944
0.043 0.032
0.137 0.200
0.047 0.032
93.386
-0.011
19.070
1.461
Table 4: Values of the estimated MoE parameters for the original Tone perception data set.
NMoE AIC 6.382 134.847 137.763 148.798 176.318
ICL 1.866 107.384 76.524 94.460 123.655
M
BIC 1.866 122.805 118.193 121.703 141.696
ED
K 1 2 3 4 5
AN US
CR IP T
We also performed a model selection procedure on this data set to choose the best number of MoE components for a number of components between 1 and 5. We used BIC, AIC, and ICL. Table 5 gives the obtained values of the model selection criteria. One can see that for the NMoE model, the three criteria overestimate the number of components, but for both BIC and ICL, the solution with two components is also likely and is the most competitive to the selected one with 5 components. In deed, it can be seen that, if the number mixture of components is fixed at 4 rather than 5, both BIC and ICL would select the right number of components in that case. AIC performs poorly for the two models and overestimates the number of components. On the other hand, for the proposed STMoE model, both BIC and ICL retrieve the correct number of components. Then, one can conclude that the BIC and the ICL are the criteria that one would suggest for the analysis of this data with the proposed model. BIC 69.532 92.435 77.9753 77.7092 79.043
STMoE AIC 77.059 110.499 106.576 116.847 128.719
ICL 69.532 82.455 52.564 56.365 67.748
PT
Table 5: Choosing the number of expert components K for the original tone perception data by using the information criteria BIC, AIC, and ICL. Underlined value indicates the highest value for each criterion.
AC
CE
Now we examine the sensitivity of the MoE models to outliers based on this real data set. For this, we adopt the same scenario used in Bai et al. (2012) and Song et al. (2014) (the last and more difficult scenario) by adding 10 identical pairs (0, 4) to the original data set as outliers in the y-direction, considered as high leverage outliers. We apply the MoE models in the same way as before. The upper plots in Figure 10 clearly show that the normal mixture of experts fit is sensitive to outliers. However, note that for this situation, compared to the normal regression mixture result in Bai et al. (2012), and the Laplace regression mixture and the t regression mixture results in Song et al. (2014), the fitted NMoE model is affected less severely by the outliers. This may be attributed to the fact that the mixing proportions here are depending on the predictors, which is not the case in these regression mixture models, namely thoses of Bai et al. (2012) and Song et al. (2014). One can also see that, even the regression mean functions are affected severely by the outliers, the provided 26
ACCEPTED MANUSCRIPT
NMoE 4
STMoE
4
Cluster 1 Cluster 2 Expert mean 1 Expert mean 2
3.5
CR IP T
partitions are still reasonable and similar to those provided in the previous non-noisy case. Then, the bottom plots in Figure 10 also clearly show that the STMoE model provides a precise robust fit. For the STMoE, even if the fit differs very slightly compared to the case with outliers, the obtained fits for both situations (with and without outliers) are very reasonable. Moreover, we notice that, as showed in Song et al. (2014), for this situation with outliers, the t mixture of regressions fails; The fit is affected severely by the outliers. However, for the proposed STMoE model, the ten high leverage outliers have no significant impact on the fitted experts. This is because here the mixing proportions depend on the inputs, which is not the case for the regression mixture model described in Song et al. (2014).
3.5
y
AN US
3
y
3
Cluster 1 Cluster 2 Expert mean 1 Expert mean 2
2.5
2.5
2
2
1.5
1.5
0.5
1
1.5 x
2
2.5
3
0
0.5
1
1.5 x
2
2.5
3
M
0
ED
Figure 10: Fitting MoLE to the tone data set with ten added outliers (0, 4) with the NMoE model fit (left) and the STMoE model fit (right). The predictor x is the actual tone ratio and the response y is the perceived tone ratio.
Figure 11 shows the log-likelihood profiles for each of the two models, which show a similar behavior than the one in the case without outliers.
20 0
−20
AC
−40
60 40 20 0 −20
−60
−40
−80
−60
−100 0
2
4
6 8 EM iteration number
STMoE log−likelihood
80
Observed data log−likelihood
40
STMoE
100 NMoE log−likelihood
CE
Observed data log−likelihood
60
NMoE
PT
80
10
12
−80 0
14
100
200 300 EM iteration number
400
500
Figure 11: The log-likelihood during the EM iterations when fitting the MoLE models to the tone data set with ten added outliers (0, 4) with the NMoE model (left) and the STMoE model (right).
The values of estimated MoE parameters in this case with outliers are given in Table 6. One can see that the SNMoE model parameters are identical to those of the 27
ACCEPTED MANUSCRIPT
NMoE, with a skewness close to zero. The regression coefficients for the second expert component are very similar for the two models. For the STMoE model, it retrieves a skewed normal component while the second component is approaching a t distribution with a small degrees of freedom. α10
α11
β10
β11
β20
β21
σ1
σ2
λ1
λ2
ν1
ν2
0.811 -3.004
0.150 0.732
3.117 -0.246
-0.285 1.016
1.907 1.808
0.046 0.060
0.700 0.212
0.050 0.088
156.240
1.757
81.355
1.630
CR IP T
param. model NMoE STMoE
Table 6: Values of the estimated MoE parameters for the tone perception data set with added outliers.
AC
CE
PT
ED
M
AN US
8.3.2. Temperature anomalies data set In this experiment, we examine another real-world data set related to climate change analysis. The NASA GISS Surface Temperature (GISTEMP) analysis provides a measure of the changing global surface temperature with monthly resolution for the period since 1880, when a reasonably global distribution of meteorological stations was established. The GISS analysis is updated monthly, however the data presented here1 are updated annually as issued from the Carbon Dioxide Information Analysis Center (CDIAC), which has served as the primary climate-change data and information analysis center of the U.S. Department of Energy since 1982. The data consist of n = 135 yearly measurements of the global annual temperature anomalies (in degrees C) computed using data from land meteorological stations for the period of 1882 − 2012. These data have been analyzed earlier by Hansen et al. (1999, 2001) and recently by Nguyen and McLachlan (2016) by using the Laplace mixture of linear experts (LMoLE). To apply the proposed non-normal mixture of expert model, we consider mixtures of two experts as in Nguyen and McLachlan (2016). This number of components is also the one provided by the model selection criteria as shown later in Table 8. Indeed, as mentioned by Nguyen and McLachlan (2016), Hansen et al. (2001) found that the data could be segmented into two periods of global warming (before 1940 and after 1965), separated by a transition period where there was a slight global cooling (i.e. 1940 to 1965). Documentation of the basic analysis method is provided by Hansen et al. (1999, 2001). We set the response yi (i = 1, . . . , 135) as the temperature anomalies and the covariates xi = r i = (1, xi )T where xi is the year of the ith observation. Figures 12, 13, and 14 respectively show, for each of the two MoE models, the two fitted linear expert components, the corresponding means and confidence regions computed as plus and minus twice the estimated (pointwise) standard deviation as presented in Section 5, and the log-likelihood profiles. One can observe that the two models are successfully applied on the data set and provide very similar results. These results are also similar to those found by Nguyen and McLachlan (2016) who used a Laplace mixture of linear experts. 1
Source: Ruedy et al. (2015), http://cdiac.ornl.gov/ftp/trends/temp/hansen/gl_land.txt
28
ACCEPTED MANUSCRIPT
1
NMoE
1
Cluster 1 Cluster 2 Expert mean 1 Expert mean 2
0
0
−0.5
−0.5
1880
1900
1920
1940 x
1960
1980
2000
1880
1900
CR IP T
y
0.5
y
0.5
STMoE Cluster 1 Cluster 2 Expert mean 1 Expert mean 2
1920
1940 x
1960
1980
2000
1
AN US
Figure 12: Fitting the MoLE models to the temperature anomalies data set with the NMoE model (left) and the STMoE model (right). The predictor x is the year and the response y is the temperature anomaly.
NMoE
1
data Estimated mean (NMoE)
y
0.5
y
0.5
STMoE
data Estimated mean (STMoE)
0
−0.5
1900
1920
1940 x
1960
1980
−0.5
2000
1880
1900
1920
1940 x
1960
1980
2000
ED
1880
M
0
NMoE
CE
100
PT
Figure 13: The fitted MoLE models to the temperature anomalies data set with the NMoE model fit (left) and the STMoE model fit (right). The predictor x is the year and the response y is the temperature anomaly. The shaded region represents plus and minus twice the estimated (pointwise) standard deviation as presented in Section 5.
95
Observed data log−likelihood
85 80
AC
Observed data log−likelihood
STMoE log−likelihood
90
90
75 70 65 60
80 70 60 50 40 30 20
55 50 0
STMoE
100
NMoE log−likelihood
10
20
30 40 50 EM iteration number
60
70
10 0
80
5
10
15 20 EM iteration number
25
30
Figure 14: The log-likelihood during the EM iterations when fitting the MoLE models to the temperature anomalies data set with the NMoE model (left) and the STMoE model (right).
29
ACCEPTED MANUSCRIPT
param. model NMoE STMoE
α10
α11
β10
β11
β20
β21
946.483 931.966
-0.481 -0.474
-12.805 -12.848
0.006 0.006
-41.073 -40.876
0.020 0.020
CR IP T
The values of estimated MoE parameters for the temperature anomalies data set are given in Table 7. One can see that the parameters common for the two models are quasi-identical. It can also be seen the STMoE model provides a solution with a skewness close to zero. This may support the hypothesis of non-asymmetry for this data set. Then, the STMoE solution provides a degrees of freedom more than 17, which tends to approach a normal distribution. On the other hand, the regression coefficients are also similar to those found by Nguyen and McLachlan (2016) who used a Laplace mixture of linear experts. σ1
σ2
λ1
λ2
ν1
ν2
0.115 0.113
0.110 0.105
0.024
-0.015
41.048
17.589
AN US
Table 7: Values of the estimated MoE parameters for the temperature anomalies data set.
NMoE AIC 50.420 91.537 90.280 92.875 92.920
ED
BIC 46.062 79.916 71.396 66.727 59.510
PT
K 1 2 3 4 5
M
We also performed a model selection procedure on the temperature anomalies data set to choose the best number of MoE components from values between 1 and 5. Table 8 gives the obtained values of the used model selection criteria, that is BIC, AIC, and ICL. One can see that, except the result provided by the AIC for the NMoE model which provide a high number of components, all the others results provide evidence for two components in the data. ICL 46.062 79.624 58.487 54.752 51.242
BIC 40.971 69.638 54.126 42.308 28.037
STMoE AIC 48.234 87.069 81.726 80.0773 75.974
ICL 40.971 69.341 30.655 20.494 -8.881
CE
Table 8: Choosing the number of expert components K for the temperature anomalies data by using the information criteria BIC, AIC, and ICL. Underlined value indicates the highest value for each criterion.
9. Concluding remarks and future work
AC
In this paper we proposed a new non-normal MoE model, which generalizes the normal MoE model and attempts to simultaneously accommodate heavy tailed data with possible outliers and asymmetric distribution. The proposed STMoE is based on the flexible skew t distribution which is suggested for possibly non-symmetric, heavy tailed and noisy data. We developed a CEM algorithm for model inference and described the use of the model in non-linear regression and prediction as well as in model-based clustering. The developed model was successfully applied and validated on simulation studies and two real data sets. The results obtained on simulated data confirm the good performance of the model in terms of density estimation, non-linear regression function 30
ACCEPTED MANUSCRIPT
AN US
CR IP T
approximation and clustering. In addition, the simulation results provide evidence of the robustness of the STMoE model to outliers, compared to the standard alternative NMoE model. The proposed model was also successfully applied to two different real data sets, including situations with outliers. The model selection using information criteria tends to promote using BIC and ICL against AIC which may perform poorly in the analyzed data. The obtained results support the potential benefit of the proposed approach for practical applications. One interesting future direction is to extend the proposed model to the hierarchical mixture of experts framework (Jordan and Jacobs, 1994). Another natural future extension of this work is also to consider the case of MoE for multiple regression on multivariate responses rather than simple univariate regression. In that case, one may consider the multivariate skew-t and the multivariate Normal inverse Gaussian distribution (O’Hagan et al., 2016) which may be more stable in high-dimensional settings compared to the multivariate skew-t. Appendix A. The skew-normal distribution
M
As introduced by (Azzalini, 1985, 1986), a random variable Y follows a univariate skew-normal distribution with location parameter µ ∈ R, scale parameter σ 2 ∈ (0, ∞) and skewness parameter λ ∈ R if it has the density y−µ 2 y−µ 2 φ( )Φ λ( ) (A.1) f (y; µ, σ , λ) = σ σ σ
PT
ED
where φ(.) and Φ(.) denote, respectively, the probability density function (pdf) and the cumulative distribution function (cdf) of the standard normal distribution. It can be seen from (A.1) that when λ = 0, the skew-normal reduces to the normal distribution. As presented by Azzalini (1986); Henze (1986), if √ (A.2) Y = µ + δ|U | + 1 − δ 2 E
AC
CE
λ where δ = √1+λ 2 , U and E are independent random variables following the normal distribution N(0, σ 2 ), then Y follows the skew-normal distribution with pdf SN(µ, σ 2 , λ) given by (A.1). In the above, |U | denotes the magnitude of U . This stochastic representation of the skew-normal distribution leads to the following hierarchical representation in an incomplete data framework, as presented in Lin et al. (2007b):
Y |u U
∼ N (µ + δ|u|, (1 − δ 2 )σ 2 ) , ∼ N(0, σ 2 ).
(A.3)
Appendix B. Stochastic representation of the STMoE model The skew t mixture of experts model is characterized as follows. Suppose that conditional on a categorical variable Zi = zi ∈ {1, . . . , K} representing the hidden label of 31
ACCEPTED MANUSCRIPT
the component generating the ith observation and which, conditional on some predictor r i , follows the multinomial distribution (B.1): Zi |r i ∼ Mult(1; π1 (r i ; α), . . . , πK (r i ; α))
(B.1)
AN US
CR IP T
where each of the probabilities πzi (r i ; α) = P(Zi = zi |r i ) is given by the multinomial logistic function (2). Now suppose a random variable Yi having the following representation: Ei Yi = µ(xi ; β zi ) + σzi √ (B.2) Wi where Ei and Wi are independent univariate random variables with, respectively, a standard skew-normal distribution Ei ∼ SN(λzi ), and a Gamma distribution Wi ∼ ν ν Gamma( 2zi , 2zi ), and xi and r i are some given covariate variables. Then, the variable Yi is said to follow the skew t mixture of experts (STMoE) defined by (13). References
Akaike, H., 1974. A new look at the statistical model identification. IEEE Transactions on Automatic Control 19 (6), 716–723. Azzalini, A., 1985. A class of distributions which includes the normal ones. Scandinavian Journal of Statistics, 171–178.
M
Azzalini, A., 1986. Further results on a class of distributions which includes the normal ones. Scandinavian Journal of Statistics, 199–208.
ED
Azzalini, A., Capitanio, A., 2003. Distributions generated by perturbation of symmetry with emphasis on a multivariate skew t distribution. Journal of the Royal Statistical Society, Series B 65, 367–389.
PT
Bai, X., Yao, W., Boyer, J. E., 2012. Robust fitting of mixture regression models. Computational Statistics & Data Analysis 56 (7), 2347 – 2359.
CE
Biernacki, C., Celeux, G., Govaert, G., 2000. Assessing a mixture model for clustering with the integrated completed likelihood. IEEE Transactions on Pattern Analysis and Machine Intelligence 22 (7), 719–725.
AC
Bishop, C., Svens´en, M., 2003. Bayesian hierarchical mixtures of experts. In: In Uncertainty in Artificial Intelligence. Brent, R. P., 1973. Algorithms for minimization without derivatives. Prentice-Hall series in automatic computation. Englewood Cliffs, N.J. Prentice-Hall. Chamroukhi, F., July 2015. Non-normal mixtures of experts. arXiv:1506.06707. Chamroukhi, F., 2016. Robust mixture of experts modeling using the t distribution. Neural Networks 79, 20 – 36. 32
ACCEPTED MANUSCRIPT
Chamroukhi, F., Sam´e, A., Govaert, G., Aknin, P., June 2009a. A regression model with a hidden logistic process for feature extraction from time series. In: International Joint Conference on Neural Networks (IJCNN). Atlanta, GA, pp. 489–496. Chamroukhi, F., Sam´e, A., Govaert, G., Aknin, P., 2009b. Time series modeling by a regression approach based on a latent process. Neural Networks 22 (5-6), 593–602.
CR IP T
Chen, K., Xu, L., Chi, H., 1999. Improved learning algorithms for mixture of experts in multiclass classification. Neural Networks 12 (9), 1229–1252. Cohen, E. A., 1984. Some effects of inharmonic partials on interval perception. Music Perception 1. Dempster, A. P., Laird, N. M., Rubin, D. B., 1977. Maximum likelihood from incomplete data via the EM algorithm. Journal of The Royal Statistical Society, B 39(1), 1–38.
AN US
Faria, S., Soromenho, G., February 2010. Fitting mixtures of linear regressions. Journal of Statistical Computation and Simulation 80 (2), 201–225. Fr¨ uhwirth-Schnatter, S., 2006. Finite Mixture and Markov Switching Models (Springer Series in Statistics). Springer Verlag, New York.
M
Fr¨ uhwirth-Schnatter, S., Pyne, S., 2010. Bayesian inference for finite mixtures of univariate and multivariate skew-normal and skew-t distributions. Biostatistics 11 (2), 317–336.
ED
Gaffney, S., Smyth, P., 1999. Trajectory clustering with mixtures of regression models. In: Proceedings of the fifth ACM SIGKDD international conference on Knowledge discovery and data mining. ACM Press, pp. 63–72.
PT
Gormley, I. C., Murphy, T. B., 2008. A mixture of experts model for rank data with applications in election studies. The Annals of Applied Statistics 2 (4), 1452–1477.
CE
Gormley, I. C., Murphy, T. B., 2010. A mixture of experts latent position cluster model for social network data. Statistical methodology 7 (3), 385–405. Gormley, I. C., Murphy, T. B., 2011. Mixture of experts modelling with social science applications. Journal of Computational and Graphical Statistics 19 (2), 332–353.
AC
Green, P., 1984. Iteratively reweighted least squares for maximum likelihood estimation, and some robust and resistant alternatives. Journal of The Royal Statistical Society, B 46 (2), 149–192. Hansen, J., Ruedy, R., Glascoe, J., Sato, M., 1999. Giss analysis of surface temperature change. Journal of Geophysical Research 104, 30997–31022. Hansen, J., Ruedy, R., M., S., Imhoff, M., Lawrence, W., Easterling, D., Peterson, T., Karl, T., 2001. A closer look at united states and global surface temperature change. Journal of Geophysical Research 106, 23947–23963. 33
ACCEPTED MANUSCRIPT
Henze, N., 1986. A probabilistic representation of the skew-normal distribution. Scandinavian Journal of Statistics, 271–275. Hunter, D., Young, D., 2012. Semiparametric mixtures of regressions. Journal of Nonparametric Statistics 24 (1), 19–38.
CR IP T
Hunter, D. R., Lange, K., 2004. A tutorial on MM algorithms. The American Statistician 58, 30–37. Ingrassia, S., Minotti, S., Vittadini, G., 2012. Local statistical modeling via a clusterweighted approach with elliptical distributions. Journal of Classification 29 (3), 363– 401. J. Q. Shi, R. M.-S., Titterington, D. M., 2005. Hierarchical gaussian process mixtures for regression. Statistics and Computing 15 (1), 31–41.
AN US
Jacobs, R. A., Jordan, M. I., Nowlan, S. J., Hinton, G. E., 1991. Adaptive mixtures of local experts. Neural Computation 3 (1), 79–87. Jiang, W., Tanner, M. A., 1999. On the identifiability of mixtures-of-experts. Neural Networks 12, 197–220.
M
Jones, P. N., McLachlan, G. J., June 1992. Fitting finite mixture models in a regression context. Australian Journal of Statistics 34 (2), 233–240.
ED
Jordan, M. I., Jacobs, R. A., 1994. Hierarchical mixtures of experts and the EM algorithm. Neural Computation 6, 181–214. Jordan, M. I., Xu, L., 1995. Convergence results for the EM approach to mixtures of experts architectures. Neural Networks 8 (9), 1409–1431.
PT
Lee, S. X., McLachlan, G. J., 2013a. Model-based clustering and classification with nonnormal mixture distributions. Statistical Methods and Applications 22 (4), 427–454.
CE
Lee, S. X., McLachlan, G. J., 2013b. On mixtures of skew normal and skew tdistributions. Advances in Data Analysis and Classification 7 (3), 241–266.
AC
Lee, S. X., McLachlan, G. J., 2014. Finite mixtures of multivariate skew t-distributions: some recent and new results. Statistics and Computing 24 (2), 181–202. Lee, S. X., McLachlan, G. J., 2015. Finite mixtures of canonical fundamental skew t-distributions. Statistics and Computing (To appear). Lin, T. I., 2010. Robust mixture modeling using multivariate skew t distributions. Statistics and Computing 20 (3), 343–356. Lin, T. I., Lee, J. C., Hsieh, W. J., 2007a. Robust mixture modeling using the skew t distribution. Statistics and Computing 17 (2), 81–92. 34
ACCEPTED MANUSCRIPT
Lin, T. I., Lee, J. C., Yen, S. Y., 2007b. Finite mixture modelling using the skew normal distribution. Statistica Sinica 17, 909–927. McLachlan, G. J., Krishnan, T., 2008. The EM algorithm and extensions, 2nd Edition. New York: Wiley. McLachlan, G. J., Peel., D., 2000. Finite mixture models. New York: Wiley.
CR IP T
Meng, X. L., Rubin, D. B., 1993. Maximum likelihood estimation via the ECM algorithm: A general framework. Biometrika 80 (2), 267–278. Meng, X.-L., van Dyk, D., 1997. The EM algorithm-an old folk song sung to the fast tune (with discussion). Journal of the Royal Statistical Society, Series B 4 (59), 511 – 567.
AN US
Ng, S.-K., McLachlan, G. J., 2004. Using the em algorithm to train neural networks: misconceptions and a new algorithm for multiclass classification. IEEE Transactions on Neural Networks 15 (3), 738–749. Nguyen, H. D., McLachlan, G. J., 2016. Laplace mixture of linear experts. Computational Statistics & Data Analysis 93, 177 – 191.
M
O’Hagan, A., Murphy, T. B., Gormley, I. C., 2015. On estimation of parameter uncertainty in model-based clustering. arXiv:1510.00551.
ED
O’Hagan, A., Murphy, T. B., Gormley, I. C., McNicholas, P. D., Karlis, D., 2016. Clustering with the multivariate normal inverse gaussian distribution. Computational Statistics & Data Analysis 93, 18 – 30.
PT
Pyne, S., Hu, X., Wang, K., Rossin, E., Lin, T., Maier, L., Baecher-Allan, C., McLachlan, G., Tamayo, P., Hafler, D., De Jager, P., Mesirow, J., 2009. Automated highdimensional flow cytometric data analysis. Proceedings of the National Academy of Sciences USA.
CE
Quandt, R. E., 1972. A new approach to estimating switching regressions. Journal of the American Statistical Association 67 (338), 306–310.
AC
Quandt, R. E., Ramsey, J. B., 1978. Esimating mixtures of normal distributions and switching regressions. Journal of the American Statistical Association 73 (364), 730– 738. Rasmussen, C. E., Ghahramani, Z., 2001. Infinite mixtures of gaussian process experts. In: In Advances in Neural Information Processing Systems 14. MIT Press, pp. 881– 888. Ruedy, R., Sato, M., Lo, K., 2015. NASA GISS surface temperature (GISTEMP) analysis. DOI: 10.3334/CDIAC/cli.001, center for Climate Systems Research, NASA Goddard Institute for Space Studies 2880 Broadway, New York, NY 10025 USA. 35
ACCEPTED MANUSCRIPT
Schwarz, G., 1978. Estimating the dimension of a model. Annals of Statistics 6, 461–464. Shi, J. Q., Choi, T., 2011. Gaussian Process Regression Analysis for Functional Data. Chapman & Hall/CRC Press. Song, W., Yao, W., Xing, Y., 2014. Robust mixture regression model fitting by laplace distribution. Computational Statistics & Data Analysis 71 (0), 128 – 137.
CR IP T
Veaux, R. D. D., 1989. Mixtures of linear regressions. Computational Statistics and Data Analysis 8 (3), 227–245. Viele, K., Tong, B., 2002. Modeling with mixtures of linear regressions. Statistics and Computing 12, 315–330.
AN US
Waterhouse, S., Mackay, D., Robinson, T., 1996. Bayesian methods for mixtures of experts. In: In. MIT Press, pp. 351–357. Waterhouse, S. R., 1997. Classification and regression using mixtures of experts. Ph.D. thesis, Department of Engineering, Cambridge University. Wei, Y., 2012. Robust mixture regression models using t-distribution. Tech. rep., Master Report, Department of Statistics, Kansas State University.
M
Young, D., Hunter, D., 2010. Mixtures of regressions with predictor-dependent mixing proportions. Computational Statistics and Data Analysis 55 (10), 2253–2266.
ED
Yuksel, S. E., Wilson, J. N., Gader, P. D., 2012. Twenty years of mixture of experts. IEEE Trans. Neural Netw. Learning Syst. 23 (8), 1177–1193.
AC
CE
PT
Zeller, C. B., Lachos, V. H., Cabral, C., 2015. Robust mixture regression modelling based on scale mixtures of skew-normal distributions. Test (revision invited).
36
ACCEPTED MANUSCRIPT
Faicel Chamroukhi
AC
CE
PT
ED
M
AN US
CR IP T
Faicel Chamroukhi is since 2016 a Full Professor of Statistics and Data Science at University of Caen, department of Mathematics and Computer Science and the Lab of Mathematics Nicolas Oresme (LMNO) UMR CNRS 6139. From 2011 to 2015, he has been an Associate Professor at University of Toulon and the Information Sciences and Systems Lab. (LSIS) UMR CNRS 7296. In 2015-2016 he was awarded a CNRS research leave at the Lab of Mathematics Paul Painlevé, UMR CNRS 8524 in Lille, where he has also been an Invited Scientist at INRIA. He received his Master degree of Engineering Sciences from Pierre & Marie Curie (Paris 6) University in 2007 and his Ph.D. degree in applied mathematics and computer science, in the area of statistical leaning and data analysis from Compiègne University of Technology in 2010. In 2011, he was qualified for the position of Associate Professor in applied mathematics (CNU 26), computer science (CNU 27), and signal processing (CNU 61). In 2015, he received his Accreditation to Supervise Research (HDR) in applied mathematics and computer science, in the area of statistical learning and data science, from Toulon University. In 2016, he was qualified for the position of Professor in the area of applied mathematics, computer science, and signal processing (CNU 26, 27, 61). His multidisciplinary research is in the area of Data Science and includes statistics, machine learning and statistical signal processing, with a particular focus of the statistical methodology and inference of latent data models for complex heterogeneous highdimensional and massive data, temporal data, functional data, and their application to real-world problems including dynamical systems, acoustic/speech processing, life sciences (medicine, biology), information retrieval, social networks.
AC
CE
PT
ED
M
AN US
CR IP T
ACCEPTED MANUSCRIPT
38