- Email: [email protected]
Latent variables models
Journal
of Econometrics
41 (1989) l-3.
North-Holland
LATENT VARIABLES MODELS Editors’ Introduction
The statistical analysis of models with latent variables has a long history in statistics, econometrics, and psychometrics, and has, more recently, attracted attention in signal processing and in systems and control theory. Latent variables models are also called errors-in-variables or factor (analysis) models. Models of this kind naturally arise in many situations and they have a great practical and theoretical appeal, in particular due to the ‘symmetric’ way noise is modelled. However, in actual applications such models have been used only to a limited extent, a fact mainly caused by the genuine complications arising in their implementation, in particular in connection with the problem of identifiability. Latent variables models may be considered as consisting of two parts: The first part consists of a system relating the latent (i.e., not directly observed) variables; this system may be deterministic or stochastic. The second part describes how the observed variables are generated from the latent variables and from noise. This way of modelling may arise in a number of different situations. For instance, the latent variables as well as the system relating them may be meaningful in a certain theory and the noise may correspond to measurement errors. However, the scope of such models is much broader and the basic idea is a symmetric modelling of noise in the sense that in general all variables (i.e., input and output variables) may be contaminated by noise. This in turn allows for a symmetric system modelling, where the classification into inputs and outputs or the specification of the number of equations has not necessarily to be made a priori. The statistical problem then is to obtain information from the observations about certain system or noise characteristics or about the latent variables. The statistical analysis in general turns out to be significantly more complicated in comparison with the usual ‘regression’ type analysis, where part of the variables (in particular, the inputs) are assumed to be noise-free. The main problem is a basic ‘nonidentifiability’ in the sense that in general the parameters of the system are not uniquely determined from the (population) second moments of the observations, since the separation between the system and the noise part is not unique without imposing strong additional restrictions. For instance, in the simplest linear case (i.e., Gaussian, i.i.d. single-input, single-output), without additional assumptions on the noise, in general every slope parameter is compatible with 0304-4076/89/$3.500
1989, Elsevier Science Publishers
B.V. (North-Holland)
2
Editors’ Introduction
the given second moments of the observations. (This problem is closely related to the way the distance of a point to a line in least squares fitting is defined.) Thus some additional structure has to be imposed to obtain a certain degree of uniqueness. Examples for such additional assumptions besides the ‘natural’ ones, such as the uncorrelatedness of latent variables with noise, are the assumptions that the noise components are mutually uncorrelated or that the second moments of the latent variables provide a best approximation to those of the observed variables with a given (lower) rank, that the latent variables or the noise are restricted to certain frequencies and/or are of different orders of magnitude (cointergration can be seen this way), or, finally, assumptions on higher-order moments. The kind and the amount of additional structure imposed clearly varies with the problem setting. The two extreme philosophies are either to add as little additional structure as possible (in other words to avoid ‘prejudice’) and to describe the resulting genuine uncertainty (i.e., the uncertainty coming from observational equivalence rather than from sampling errors) or to impose additional assumptions which guarantee identifiability. The different kinds of additional structure imposed, the different assumptions concerning the system relating the latent variables (e.g., linear or nonlinear, static or dynamic) and the kind of information extracted from the observations (e.g., second moments) lead to many different aspects and facets in the operational implementation of latent variables models, a fact that is also reflected in the contributions of this issue. As has been stated already, the problems related to identifiability or, more generally, to the structure of the relation between the observations and the parameters of interest play a central role in this area. Three papers in this issue are exclusively concerned with these problems. Bekker considers a linear (static) factor model with a rather general (joint) parameterization of the factor loadings, the factor covariances, and the noise covariances, and derives rank conditions for local identifiability based on the covariances of the observations, as well as a method to compute the rank of the corresponding Jacobian matrices. The two other papers on identifiability are based on the other philosophy, namely to add as little extra structure as possible in the linear (stationary) context. Picci considers the general linear factor model with block-diagonal noise covariances and provides an analysis of the parameterization problem based on what one could call a ‘state-space formulation’ of the model, where the state-space makes inputs and outputs conditionally independent. Deistler and B.D.O. Anderson provide a survey of some recent results on linear dynamic errors-in-variables models with diagonal or block-diagonal noise spectral density. In all the other papers, emphasis has been put on inference, although problems of identifiability are also addressed.
3
Editors’ Introductton
Rational expectations models, as special latent variables models arising from theoretical concepts, are considered in Watson’s contribution. Watson derives a state-space representation for linear rational expectation models and, based on this representation, maximum likelihood estimation is considered. In T.W. Anderson’s paper a generalization of the linear factor model called the ‘multiple battery factor model’ is considered which allows for a more complicated error structure. Three criteria for goodness-of-fit are proposed and asymptotic properties of estimators and tests based on these criteria are derived. Ghosh analyzes maximum likelihood estimation in a single-input-singleoutput linear dynamic system with white nose equation and measurement errors. Thereby a state-space representation of the model is used and the maximum likelihood estimators are shown to be asymptotically efficient. Bloch deals with linear dynamic errors-in-variables systems with emphasis on minimization of total least squares on a certain Grassmannian manifold and its relation to Gaussian maximum likelihood estimation. Finally, Hsiao’s contribution is the only one dealing with nonlinear systems. Here identification and estimation for nonlinear errors-in-variables models are investigated. Emphasis is given to the asymptotic properties of a minimum distance estimator and a two-step procedure. In the early 1970’s, latent variables models enjoyed a renaissance in econometrics. Both theoretical and empirical contributions since that time have provided the basis for an optimism that, finally, models with errors-in-thevariables and models with errors-in-the-equations will be fully integrated. The cross-fertilization of econometric methodology with that of psychometrics and engineering is primarily responsible for this most welcome situation, DENNIS
J. AIGNER
University of California, Irvine
MANFRED Technical
DEISTLER
University of Vienna
of Econometrics
41 (1989) l-3.
North-Holland
LATENT VARIABLES MODELS Editors’ Introduction
The statistical analysis of models with latent variables has a long history in statistics, econometrics, and psychometrics, and has, more recently, attracted attention in signal processing and in systems and control theory. Latent variables models are also called errors-in-variables or factor (analysis) models. Models of this kind naturally arise in many situations and they have a great practical and theoretical appeal, in particular due to the ‘symmetric’ way noise is modelled. However, in actual applications such models have been used only to a limited extent, a fact mainly caused by the genuine complications arising in their implementation, in particular in connection with the problem of identifiability. Latent variables models may be considered as consisting of two parts: The first part consists of a system relating the latent (i.e., not directly observed) variables; this system may be deterministic or stochastic. The second part describes how the observed variables are generated from the latent variables and from noise. This way of modelling may arise in a number of different situations. For instance, the latent variables as well as the system relating them may be meaningful in a certain theory and the noise may correspond to measurement errors. However, the scope of such models is much broader and the basic idea is a symmetric modelling of noise in the sense that in general all variables (i.e., input and output variables) may be contaminated by noise. This in turn allows for a symmetric system modelling, where the classification into inputs and outputs or the specification of the number of equations has not necessarily to be made a priori. The statistical problem then is to obtain information from the observations about certain system or noise characteristics or about the latent variables. The statistical analysis in general turns out to be significantly more complicated in comparison with the usual ‘regression’ type analysis, where part of the variables (in particular, the inputs) are assumed to be noise-free. The main problem is a basic ‘nonidentifiability’ in the sense that in general the parameters of the system are not uniquely determined from the (population) second moments of the observations, since the separation between the system and the noise part is not unique without imposing strong additional restrictions. For instance, in the simplest linear case (i.e., Gaussian, i.i.d. single-input, single-output), without additional assumptions on the noise, in general every slope parameter is compatible with 0304-4076/89/$3.500
1989, Elsevier Science Publishers
B.V. (North-Holland)
2
Editors’ Introduction
the given second moments of the observations. (This problem is closely related to the way the distance of a point to a line in least squares fitting is defined.) Thus some additional structure has to be imposed to obtain a certain degree of uniqueness. Examples for such additional assumptions besides the ‘natural’ ones, such as the uncorrelatedness of latent variables with noise, are the assumptions that the noise components are mutually uncorrelated or that the second moments of the latent variables provide a best approximation to those of the observed variables with a given (lower) rank, that the latent variables or the noise are restricted to certain frequencies and/or are of different orders of magnitude (cointergration can be seen this way), or, finally, assumptions on higher-order moments. The kind and the amount of additional structure imposed clearly varies with the problem setting. The two extreme philosophies are either to add as little additional structure as possible (in other words to avoid ‘prejudice’) and to describe the resulting genuine uncertainty (i.e., the uncertainty coming from observational equivalence rather than from sampling errors) or to impose additional assumptions which guarantee identifiability. The different kinds of additional structure imposed, the different assumptions concerning the system relating the latent variables (e.g., linear or nonlinear, static or dynamic) and the kind of information extracted from the observations (e.g., second moments) lead to many different aspects and facets in the operational implementation of latent variables models, a fact that is also reflected in the contributions of this issue. As has been stated already, the problems related to identifiability or, more generally, to the structure of the relation between the observations and the parameters of interest play a central role in this area. Three papers in this issue are exclusively concerned with these problems. Bekker considers a linear (static) factor model with a rather general (joint) parameterization of the factor loadings, the factor covariances, and the noise covariances, and derives rank conditions for local identifiability based on the covariances of the observations, as well as a method to compute the rank of the corresponding Jacobian matrices. The two other papers on identifiability are based on the other philosophy, namely to add as little extra structure as possible in the linear (stationary) context. Picci considers the general linear factor model with block-diagonal noise covariances and provides an analysis of the parameterization problem based on what one could call a ‘state-space formulation’ of the model, where the state-space makes inputs and outputs conditionally independent. Deistler and B.D.O. Anderson provide a survey of some recent results on linear dynamic errors-in-variables models with diagonal or block-diagonal noise spectral density. In all the other papers, emphasis has been put on inference, although problems of identifiability are also addressed.
3
Editors’ Introductton
Rational expectations models, as special latent variables models arising from theoretical concepts, are considered in Watson’s contribution. Watson derives a state-space representation for linear rational expectation models and, based on this representation, maximum likelihood estimation is considered. In T.W. Anderson’s paper a generalization of the linear factor model called the ‘multiple battery factor model’ is considered which allows for a more complicated error structure. Three criteria for goodness-of-fit are proposed and asymptotic properties of estimators and tests based on these criteria are derived. Ghosh analyzes maximum likelihood estimation in a single-input-singleoutput linear dynamic system with white nose equation and measurement errors. Thereby a state-space representation of the model is used and the maximum likelihood estimators are shown to be asymptotically efficient. Bloch deals with linear dynamic errors-in-variables systems with emphasis on minimization of total least squares on a certain Grassmannian manifold and its relation to Gaussian maximum likelihood estimation. Finally, Hsiao’s contribution is the only one dealing with nonlinear systems. Here identification and estimation for nonlinear errors-in-variables models are investigated. Emphasis is given to the asymptotic properties of a minimum distance estimator and a two-step procedure. In the early 1970’s, latent variables models enjoyed a renaissance in econometrics. Both theoretical and empirical contributions since that time have provided the basis for an optimism that, finally, models with errors-in-thevariables and models with errors-in-the-equations will be fully integrated. The cross-fertilization of econometric methodology with that of psychometrics and engineering is primarily responsible for this most welcome situation, DENNIS
J. AIGNER
University of California, Irvine
MANFRED Technical
DEISTLER
University of Vienna