ELSEVIER
0905-3293(95)00030-S
ACOMPARlSONOFlMPUTATlONTECHNlQUESFOR lNTERNALPREFERENCEMAPPING,USlNG MONTECARLO SIMULATION Duncan
Hedderley
and Ian Wakeling
Institute of Food Research, Earley Gate, Whiteknights Road, Reading RG6 6BZ, UK.
(Received 26 October 1994; accepted 12 May 1995)
number
ABSTRACT
of products
columns
as vectors
space,
The usual algorithm for internal preference mapping requires a complete set of observations, meaning the technique cannot be used to analyse trials based on incomplete block designs.
such
of consumers),
and the
points
scalar
the rows and
in multidimensional
products
of
are as close to the original
the
points
data matrix
as
possible. The usual way of calculating is
to
use
requires product
for imputing missing values under various conditions. Sets of simulated preference data with dqfwent character-
jects
istics were constructed. Monte Carlo simulation was used to create missing observations in these sets; the imputation
a
Singular
these vectors
Value
and points
Decomposition,
which
a complete data matrix (e.g. scores for each from each subject). This has meant that sub
with the occasional
score which
by accident
have had to be discarded
it has also
meant
that
internal
has been
missed
from the analysis;
preference
mapping
could not be used to analyse trials which were designed
techniques were applied to the data; and the results of
to be incomplete
preference mapping based on the imputed data compared to those from the complete data set. Convergence problems were found with two techniques.
(for instance,
where
was able to assess all of the available
no one subject
products
because
of fatigue effects). As part
Analysis of variance revealed that effects on performance were dominated by the proportion of data missing, the
potential
level of noise in the data, and the size of the data set.
based
of a MAFF/industrial
LINK
strategies
this
identified:
Daffmences in performance among the three convergent imputation techniques were small; mean substitution is recommended, as it performed as well as more complex iterative techniques.
either
on
Maximisation Analysis
solving values
a technique
Algorithm
(Beale
1982);
the
to the
Kruskal’s
fit
original
of this approach
The study focused
Keywords: Preference mapping; missing values; imputation; simulation; expectation-maximisation algorithm; row-column substitution algorithm; MISTRESS algorithm;
for the number
PRINQUAL procedure; mean substitution.
approaches
were
the
missing
like
the
Expectation-
& Little,
1975;
data Demp-
form of Homogeneity
available
for handling
(similar missing
to data
1964. An example
and Zamir,
on techniques
which just
data
scaling; Kruskal,
is Gabriel
three
or an approach
suggestion
in multidimensional
project, problem
for
or on a modified
(Meulman,
maximised
1979).
for imputing
values
missing data points, because programs for a of imputation techniques were either already
available Gabriel
to
substituting
ster et al. 1977),
The results were broadly confirmed by a similar study on a genuine set of prefmence data.
in SAS or easily written, (modified
and Zamir’s
whereas
Homogeneity
weighted
the other
Analysis
least-squares)
or
were not.
A Monte Carlo simulation study was planned to compare the performance of the different imputation techniques under a range of conditions typical of
INTRODUCTION AND LITERATURE SURVEY
Preference conditions
Internal Preference Mapping is a variation on the biplot (Gabriel, 1971; Krzanowski, 1988a). Working from
adequately
a matrix
complete
preference
that
and vectors
A simulation study was carried out to compare techniques
of data (for instance,
given by a sample
its aim is to find a way of representing
scores
for a 281
Mapping studies, and to find under the techniques gave solutions which close to the results obtained data sets.
what were
from analysing
282
D. Hedderlty, I. Wakeling
MISSING _ The
VALUE
five imputation
METHODOLOGIES
techniques
chosen
were the Expectation-Maximisation & Little,
1975);
made iterative:
Krzanowski’s
Algorithm
198813);
the
values
for the study
Row-Column
(Beale
the
solution
data
until the imputed
Several rithm.
SAS PRINQUAL
(Krzanowski,
algorithm algorithm
1989, 1990); the MISTRESS (SAS Institute, (van Buuren & van Rijckevorsel, 1992); and (Bello,
to complete
repeated
problems
The
matrix.
are arbitrary,
the
is an
to predict
sufficient
statistics
of)
does this by regression; for variable known,
iterative
technique
a multivariate
if subject
data
the algorithm
set. It
S has a missing
x, but s’s values for variables
then
other,
missing values in (or at least, value
xi to X, are
would use all the records
with known values for xi and 3 to X, to form a regression equation
for x, and then use that to predict
x, for subject for each
S, given S’s values of xi to x,. This is done
missing
algorithm),
value
(the
Expectation
and then maximum
the sufficient
statistics
statistics.
The
iteration
until it converges
algorithm
iterative
updated
goes
through
(1975)
Dempster
normal
of Buck’s
eventually point,
have proved
converge
so
long
although
‘the
as rate
slow’. Outlining algorithm,
algorithm
100 iterations;
(Buck,
a matter
vector
log
(1983)
program
and Little
were
often
as an 1960))
(1975)
(cited
algorithm
likelihood
of convergence
enough,
can
in will
or a saddle-
Krzanowski
Row Column (1988b)
to implement
on sub matrices
achieved analytically; however, with multiple missing values,
if the matrix
that one of the singusign, compared
values
be updated
at each
unstable.
appears
to the To over-
comparing
to reference
the signs
left
and
right
to have changed
by -1. These
reference
iteration
the
from an SVD
sign it
matrices
to take
must
account
of the
updating of the imputed values in the data matrix. A second problem is that, if the estimation is based on a fixed number of the
lower
heavily
influenced
estimates.
of dimensions
dimensions
This
can cause
cycle,
preventing
check
on
which
reduces
the
convergence
by the
most
dependence
mates
in some cases, some
may be essentially recent
on the
the imputations convergence. rate
the number
value
recent
esti-
to get trapped
in a
To
most
overcome
of convergence
criterion
random,
missing
this,
a
was introduced,
of dimensions
used if the
is not improving.
PRINQUAL The
one
PRINQUAL
1989,199O)
the
a limit
of
study
problem
which
on qualitative ance
procedure
covers
method
perform data.
The
SAS
Minimum transforms
regression
The SAS/STAT method although
being
Generalised
Variand
the transforma-
the determinant
matrix
of the
manual
gives an example
it is accompanied
tech-
analyses
the variables,
to update
used to estimate
Institute
scaling
components
with the aim of minimising covariance
(SAS
of optimal
principal
iteratively
then uses multiple
in
a number
transformed
of the MGV
values for missing
by warnings
of
variables. data,
that the tech-
nique may get stuck in a degenerate solution where all non-missing values are merged into one category.
MISTRESS
a technique
which
the row or the column containing For a matrix with a single missing
elements
is then multiplied
the
for imputing
a missing value in a multivariate data set by reconstructing the data matrix from the result of Singular Value Decompositions
matrix
matrices;
tions,
Substitution
presents
there is a possibility the imputed
SVD. Since
vectors
this a check was introduced,
of the
be painfully
171 iterations’.
Krzanowski’s
come
is bounded;
suggest but
the algo-
of programming:
from another
will have the wrong
making
niques
they add that ‘in our simulation
10 iterations required
the
for
assumptions. Wu
in Krzanowski’s
but Beale
to a maximum
a computer
Beale
suggested
that the E-M
either
of
1987,
is
used in calcu-
distribution;
and
et al. (1977)
1991)
this process
show that it can also be seen
refinement
for
sufficient
(see Little and Rubin,
which does not make distributional Rubin,
imputations
had already been
data with a multivariate
of
from the filled-in
of the actual technique
This approach
and Little
estimates
data, which in turn give refined
for a description
of the
step). These sufficient statistics
can then be used to produce the missing
part
likelihood
are computed
data set (the Maximisation
lation).
the value of
process to a limit.
the algorithm calculates imputed values using the left singular vector from one SVD on the data matrix, and
lar vectors
Expectation-Maximisation which attempts
This
highlighted
signs within each pair of singular
Expectation-Maximisation
for each
while programming
first was purely
the right singular
1993).
is found
for the other missing
values converge
not
paper were encountered
Substitution
algorithm
Mean Substitution
an analytical
missing value, using the estimates
exclude
either
the missing value. value, this can be
to extend it to matrices the technique must be
The
MISTRESS
evorsel,
1992)
algorithm
(van Buuren
is a technique
for
& van Rijck-
imputing
missing
values in categorical data such that the internal consistency of the resulting data set is maximised. In other words, given an observation with a missing value in a set of categorical data, their technique replaces it with a category which is ‘typical’ data set.
of similar observations
in the
Comjxwison of Imputation Techniques for Internal Preference Mapping Technically, maximise
the
At each iteration, each
category
spects
algorithm
iteratively
squared
correlation
Guttman’s
missing
data
its imputed
correlation
to
coefficient.
scaled values are found
of each variable;
each
changes
optimally
attempts
the algorithm
point,
category
and
then
recovered
in-
Bello mance
the squared
values
the algorithm
treats the data as categorical, compared
to other
data.)
The
make any a priori
MISTRESS
assumptions
algorithm
about
cannot
data.
Using
and
he studied
of means
Bello
concluded
different that
(such as mean substitution) sophisticated
these from the data set itself. This means
that the data
E-M
or Krzanowski’s
high, and the proportion large.
Van
Buuren
MISTRESS
structure;
of missing
adequately
or
data not to be too
and van Rijckevorsel
performs
the
must be moderate suggest
that
with up to 10% miss-
number
better
consistency
centrate
of O-6 or higher;
for adequate Despite
consistency
for
20%
than 0.75
missing
problem,
the MISTRESS
in the study because
applicable,
and because
algo-
it was (at least
SAS code
Mean Substitution missing the
mean
Although structure
score
for
the
which
variable
preference
techniques, other
(e.g.
which
more
and
with 1993).
assumes and
as a ‘control’
sophisticated
niques were expected
Bello,
to the other
are all iterative,
It was included
them
and ignores
mapping
data, it is also quick compared mented.
to replace
this is very simple-minded
the
is in the
sumer
method,
which
time-consuming
tech-
preference
imputation
between
plete’
of data sets typical of a con-
to ‘missing’.
techniques
preference
mapping values.
preference
using a number
performed.
In the field of Multi-Dimensional
Data Sets
proposals under
for
the
different
a number
of
studies to investigate
how
recovery
data
of missing
conditions.
For
instance,
Spence and Domoney (19’74) studied whether the technique performed better when the specific comparisons which were missing systematic
pattern;
occurred
at random
and Whelehan
or followed
et al. (198’7)
studied
a
A preference
and observations run
on the
carried
of measures.
were then
from
compared
with
the complete
This process
build up a distribution
of how the different
features
set to ‘missing’,
of the data set were chosen
of the preference
and the level of noise in the data.
to
techniques
for manipula-
of subjects, of stimuli,
the dimensionality
data set,
was repeated
observations
tion: the number the number
set
obtained
50 times with different
Four
‘incom-
out on each
configurations
mappings
data
at ran-
values were saved, and a
was then The
These
(chosen
Each of the different
was then
data set; the imputed
COMPARING THE PERFORMANCE OF THE METHODS: SIMULATION TESTING
Kruskal’s
the preference
STUDY
the ‘true’ results from analysing
perform
substitu-
would be moderate
could be compared.
taken,
were changed
imputation
of imputed
Scaling,
techniques
like mean
test were constructed.
techniques
sets were then dom)
imputation easily imple-
to outperform.
have used simulation
all
to con-
mapping was done on the complete data sets to get a set of ‘true’ results against which the results from the
these
papers
under
in most situations.
OF THE
For the study, a number which has been used to deal with
values in data has been
iterative
that there
products
ones, although
the decision
techniques
levels of correlation
scores for different
OUTLINE technique
guided
moder-
techniques
superior
complex,
to simpler
were
the iterative
for it was
available.
One other
results
on investigating
to high
as
are not very
the variables
was consistently
tion, on the assumption
this potential
better (such
on data with a small
than the non-iterative
These
as opposed
is needed
results.
rithm was included vaguely)
while
greater
and where
no single technique conditions.
performed
techniques
where the variables
ately or highly intercorrelated, performed
of
non-iterative
However, for data with a higher number
of dimensions,
ing data; for 15% missing data, the algorithm only perif the data have an internal forms adequately data, internal
iterative
technique)
of dimensions
highly correlated.
the
recov-
proportions
simple,
more
internal
between
and the variance-covariance
than
of the variables
of con-
of observa-
correlations
techniques
intercorrelation
missing
how closely the techniques
after he had deleted
data.
imputing a number
numbers
it has to infer
must have a fairly strong/clear
total of the
study of the perfor-
for
instead
data values relate to each other -
of the
on the accuracy
a simulation
dimensionalities,
matrices
how the different
proportion
data sets with different
ered the vectors the
the used
techniques
in multivariate
variables,
the data as ordinal or interval scaled. which probably are safe with preference
or sensory
reports
of various
tions,
algorithms
and
data. (1993)
structed
it is at a disadvantage which treat (Assumptions
of noise
of comparisons
for
coefficient.
Because
effect
necessary
if
to improve
the
number
283
space
284
D. Hedderlq, I. Wakeling
These
were
influential
factors
et al. 1987),
performing
because
control
the
most
observations control
For instance, larger
knowledge.
most
directly
are the numbers
it is possible
the case if the proportion niques
clear.
for
for
experiequal,
to missing
whether
with certain
the
a
values
some
shapes
Row-Column
tech-
of data
Substitution
algorithm may perform better with ‘squarer’ data sets, rather than ones which are either ‘long and narrow’many subjects,
few products;
many products,
few subjects;
and
column
corresponding
when estimating matrices, matrix
discards
but are potentially
be aware of. Previous
that the underlying performance results
studied.
with higher
found
the
One
might
random
expect
more
observations
found
design
resolution structed
(confounding
IV was chosen. by randomly
co-ordinates
consumer
Normal
readjustment
noise.
Three
levels of missing
data set chance’ in
an
chosen
data)
and 65%
incomplete
block
For each
of the 2”’
but
the
being
of dimensions, performed data
the imputation
in
successfully level of
data points
pattern
When
(see,
for
1974).
However,
patterns,
this factor
in the study, and all the missing
constructing
were chosen,
these factors plus their potential be studied systematically, it was be used, data sets
the data sets, two levels of each representing
points
were
data sets and the three sets with
on the
compared
imputed
the complete
submitted mapping
data
to the results
matrix
data points/performance
indicators
3 X 24-I combinations
of technique,
The
design out,
were
under 1000
would have been
was then per-
and
the results mapping
a sample
on
of 50
for each of the 5 X level of missing
some of these data points
performed
SAS
6.07
for
h of computing
would have required
a 24 design
of
imputed
of lack of convergence.
simulations
workstation,
were
to each
of a preference
data set. This produced
are missing because
levels of
values
and the resulting
values were saved. A preference formed
missing
with
a DEC and
time.
5000 took
A full 24
twice the time, which was
although,
as a reviewer
25 data
an alternative
and allowing more effects
on
Unix,
sets per requiring
pointed
combination similar
time
to be estimated.
data
randomly.
decided that constructed data sets should although a confirmatory study using genuine was planned as well. factor
Data
by
be found
at random.
techniques,
or
To ensure that interactions could
design).
data set was then
occur
was not included
on each
Runs
felt to be impractical;
were located
by a final
were still in the range
(a level which might
Each
that recovery
the missing
Domoney,
these
(a high level for ‘missing
to be set to ‘missing’
is whether and
was followed
values were tested
5% (low), 35%
of the techniques Spence
of the
; resealing
1 to 9; and then adding
This
data and data set. In practice,
of the variety of possible
the scalar-product
by
l-9.
approximately
because
calculating
for each product
so that the scores
relative to the other factors under investigation. One factor which might influence the performance
instance,
of con-
and consumer
space;
co-ordinates)
so that they were in the range random
data
product
created.
techniques
some
sets were
choosing
product
manageable,
The
(i.e. taking
and
normal
interaction)
have found
from data with a higher
follow
and the noise
the 4way
‘true’ preference
vector-projection
50 or 20;
a random
in 2- or 4dimensional
each consumer’s
(Box,
= 1 (low) or 2 (high).
50 data
noise, but it is not clear how large this effect is
randomly,
deviation
data,
when the simulated difficulty
was either
2 or 4 dimensions,
variate with standard
is to
trends
10 or 30; the dimen-
(for a 1-9 scale) was set to be either
missing
had more than 2 dimensions. imputing
of subjects was either
factors
techniques,
numbers
that iterative
than mean substitution
the experi-
influential
on the technique
et al. (1987)
difficult
while Bello better
to depend
Whelehan
was more
of
of the data affects
of imputation
objective
space, and the
studies
dimensionality
seem
of products
The Simulation
of the preference
studies. This is a
the
which might
in the data are not within
control,
they might the
proportion
less accurate.
The dimensionality menter’s
the row
observation
matrix,
The number
was either
where
and size of major
et al. 1978).
-
With more rectangular
a greater
than it does with a square
level of noise
it ignores
to a missing
that observation.
this
make the estimation
exact
or ‘wide and shallow’ because
the presence
the number
a 2”
this would still be that
determine
mapping
in trials
To keep the time spent on the simulations
the
of data missing was kept con-
better
instance,
technique
of subjects and stimuli.
robust
strategy
sionality
data set.
It is also possible
will perform
matrix;
study,
have
that, all else being
than a smaller one, although is not
to an
factors
under
typically found in preference common
It is there-
these
appropriate
data set might be more
stant
to be
mapping
on a particular
two factors
menter’s
their
to know what effect
choosing
The
found
1993; Whelehan
a preference
or at least within
imputing
been
(Bello,
they would be under the experimenter’s
fore important when
had
studies
and were also likely to be relevant
experimenter either
which
in previous
high and low values
Performance
Measures
Two different methods of measuring the performance of each imputation technique were used. The first (following Bello, 1993) was the sum of squared differences between the imputed data and the complete data set. Because points
this sum was over a different for each
combination
number
of Number
of data
of Subjects,
Compatison of Imputation Techniques forInternal Prqmence Mapping Number
of Products
and Proportion
of Data Missing,
the sum of squares was divided by the number vations
declared
Number
missing
of Products
create
an average
value,
which
footing.
X Proportion
allow
of factors
Otherwise,
proportion
be larger,
more
terms,
imputing
rather
of Missing difference
the
results
the results
from
data,
simply
values
of Subjects
from
than
are
picture
from
‘true’
values. this only measures
values match
the original
how closely the imputed
data. The real interest
study is in how results from preference on incomplete
data
corresponding
complete
approach
of Beale
compare
al. (198’7), who chose close
their
complete
final
nique reproduced To this end,
to measure
from
difference
the
number
between
the
Similarly,
the
performance
et
by how from
the
data matrix.
be expected
tions obtained
data
set,
and
the
differences
mean divided
of consumers
between
the
based on the imputed
and
sets was calculated.
relative
positions
of the points,
such,
both
the
rather
were
to match
them
to the configurations
subjected
complete
data
the
the main
mappings and
consumer
to Procrustes
mean
is the
than their co-or-
product
configurations
before
Because
preference
derived squared
the study. This algorithm
was performing
was the proportion converged.
practical
an algorithm
&AS/STAT
Obviously, Of course,
converges criteria;
will depend
SAS Institute,
variation:
be
found.
was used
Because
to a non-optimal
(or even non-sensical)
on
that empirical
these
the
pling
for each of the data sets in the study, 50 new
assessors’
responses
were then
the one derived
analysed from
of
(complete)
distribution
mapping,
compared
the
original
the ‘true’
The 95th percentile
recent
work
crustes
Analysis
results.
on the
This
strategy
significance
configurations
Wakeling et al. 1992) Unfortunately, because
the
assessors
runs were not comparable
was not suitable
& Arents,
Configurations
varia-
were
on Pro-
1991; being
from different as they did not
This meant
for deriving
of this
is based
of Generalised (Ring
of the
configura-
was taken as the limit for ‘adequate’
tion from
to
data set. This pro-
of the variation
about
with
sets of data.
Configurations
the original
Configurations
by resam-
matrices),
using preference
Stimuli
an empirical
Stimuli
(rows
from the original
and the resulting duced
concerns,
panel-to-panel
data sets of the same size were constructed,
resampling
converge,
during
in which
this approach
a limit for an ‘adequate’
of the Assessor Configuration.
the
performance
and the most appropriate
of cases is
Reliability
of Convergence
cases,
on the set-
in some cases in the
it is possible
or will converge solution.
of the
RESULTS
algorithm
Testing compared
of
to estimate
consist of the same assessors.
chapter
1989)
will never
techniques,
could
rotations
in some
however,
that a technique
Adequacy
an
the Proc PRINQfiAL
manual,
was a
depend
all the data sets used in the simulation
from the
emerged of runs
in the vast majority
value.
ting of the convergence (see, for example,
on this
which gave a good initial impression
which does not converge whether
study
resampling
recovery
measure
of how each technique
of limited
data, in which case it would be unlikely data to match
the Assessor
were calculated.
results
might
resampled,
differences
solu-
panels of consumers.
features of the data set, such as the number of stimuli, the number of assessors and the amount of noise in the
tion due to panel variation.
difference
than the variation
that the level of variation
distribution
squared
the
are no fur-
and it was also felt that there
was calculated.
mean
is
is that
complete-data
We were not aware of any published
of dimensions)
data
Another
solution between
from different
type of variation;
These of
to the configuration
(sum of squared of points
was performed
performance’
definition and subjects)
ther from the complete-data
replacement,
data. The configuration
complete
when interpreting as
follows
and Whelehan
to the results
compared
interest dinates
based on the
(of products
X number
configurations
Having
configurations
(the
data set) to be
good
possible
two sets of co-ordinates
the
complete
This
mapping
of imputed
squared
based
than simply how well their tech-
was then
by the
were
the original
derived
sets. (1975)
a preference
matrix
the products
data
results
data, rather
on each
to those
and Little
of the
mappings
one
arises:
results
of ‘the truth’?
of ‘sufficiently
but
possibility
However,
each
a reliable
concept
that might
are
the
This
ill-defined,
matrices
the techniques
question
to the ‘true’
considered
runs with a high
further
another
close
to
they are the sum of
because
which
identified,
sufficiently
with the complete
different
data
had been
results from working
on an equal
or bigger
because
Data)
situation
is the result
X
per missing
to be compared
of missing
could
Number
mean squared
would
combinations
(i.e.
of obser-
285
different
one for a given
The
first criteria
niques
used
to compare
was in what proportion
solution to the problem The results are presented
the different
of cases
of imputing the missing in Table 1.
The simplest technique,
Mean Substitution,
in every case,
the proofs
cited
problems
programming
Row-Column
as might
by Rubin
Substitution
be expected,
(1991).
the
After
technique,
technique
a
data.
will always
find a solution, so long as at least one consumer each of the products. Expectation-Maximisation converged
tech-
they found
some
scored also given initial
Krzanowski’s also performed
D. Hedderlq, I. Wakeling
286
TABLE 1. Number of Simulation Runs which Converged (N= 50) Stimuli Assessor
Dimens
Imputation Techniques
Noise Mean
E-M
Row-Co1
MISTRESS
PRINQUAL
10
50
2
Low
5% 35% 65%
50 50 50
50 50 50
50 50 50
50 47 33
49 13 0
10
200
2
High
5% 35% 65%
50 50 50
50 50 50
50 50 49
50 48 7
31 34 0
30
50
2
High
5% 35% 65%
50 50 50
50 50 50
50 50 50
50 41 9
41 1 2
30
200
2
Low
5% 35% 65%
50 50 50
50 50 50
50 50 50
50 6 0
32 13 1
10
50
4
High
5% 35% 65%
50 50 50
50 50 50
50 50 50
50 49 27
33 24 0
10
200
4
Low
5% 35% 65%
50 50 50
50 50 50
50 50 50
50 35 3
38 20 0
30
50
4
Low
5% 35% 65%
50 50 50
50 50 50
50 50 50
50 41 15
32 0 0
30
200
4
High
5% 35% 65%
50 50 50
50 50 50
50 50 50
50 31 0
19 0 0
well, only failing on
one
MISTRESS values,
to converge
simulation
within the iteration
run.
performed
As
been
limit
expected,
well with low levels of missing
but was less successful
QUAL’s
had
performance
at higher
levels.
PRIN-
just
those
with missing
used during
the simulation
circumstances
was poor.
the manual),
these
as intended.
squared
differences
The
usual, and so is higher
data matrices
differences
and the original
of how well the imputation inal data. Because ing observation, data matrices The taking
mean
between
they are a mean the
results
squared
difference
data sets are a measure
technique
with different
duced by the imputation
recovers difference
are comparable number
differences
between
the imputed
the
per miss-
ever,
high
calculated
the
reflect
values
were retained
(if only imperfectly)
by pro-
and the entire origi-
Analysis of variance the relative influence
would
is not impor-
be the same
terms than
for
PRINQUAL result;
because
how-
they do
the fact that the solution
formance three
was performed
of the techniques.
techniques
(Mean
found
the detail of this calculation
the sum of
if the most
rithm,
is
a degenerate use;
and
the
Substitution either
solution while
to judge
on the per-
The analysis was limited Substitution,
and the Row-Column
practical
in order
of the various factors
techniques,
the results
to in
to have
unsatisfactory.
it was felt that PRINQUAL
since
scores
and questionable
nal data matrix, squaring the differences, summing them, and then dividing the result by the number of values declared missing in that matrix. For most of the tant,
this occurred,
Difference
It is a peculiar
entire data matrix
technique,
Squared
do not appear
than usual, skewing some of the
upwards.
of missing values. were
Mean
When
under some referred
involves more non-zero
the origbetween
are parameter
solutions
constraints
worked
squared
There
study; however,
(the degenerate
Mean Squared Differences Between Imputed and Actual Values mean
values.
settings which are supposed to force the procedure to leave the non-missing data unchanged, and these were
too frequently
to
algo-
technique)
did not converge,
MISTRESS
reliably, at least with small numbers
E-M
; or
to be of
converged
fairly
of missing data, initial
obvious alternative (simply comparing the true values for those observations which were declared missing with their imputed equivalents) had been used. However,
inspection performed
when creating an output data set, proc PRINQUAL reassigns the values for all the points in the data set, not
A split-plot design was used because it was felt that the experimental units fell into two strata; the results of
of the results showed that it consistently less well than the other three techniques.
Compatison
applying
the
techniques
to a given
incomplete
of Imputation Techniques for Internal Preference Mapping
of predicted
data
means or confidence
matrix were likely to depend on the pattern of the missing
tive. Instead,
data in the matrix,
model
similar
to each
matrices. being
This
treated
and so they were more
other led
than to the
as the ‘plots’
to results stratum,
as ‘subplots’.
as Wilks’
Pillai’s
Lambda,
Lawley Trace) Sphericity Test the univariate
than
matrices
and the results for and
the
different Apart
Hotelling-
explanation
(Schlich,
the analysis.
are
because
of the range
they cover range
because
from
was a risk that, if untransformed
(means 1.26
there
TABLE 2. Analysis of Variance
for Difference
Technique Technique Technique Technique Technique Technique Technique Technique Technique Technique Technique Technique Technique Technique Technique Technique
Techniques
for the
more
homogenous.
in variance, of factors
however,
the
and technique
in Table
2. Almost
all the
compared
to the
relevant
variation.
inter-replicate
the variation
the significance
those
effects
of effects
hence
the Fdistributions
have different
Imputed
in performance
Comparing
judge
Between
To assess which F ratios in order
to
can be misleading
if
degrees
of freedom
have different
and Actual
fac-
most, the
however,
degrees
(and of free-
in this case it is
Data
Wilks’
Pillai’S
H-L Trace
297.9 842.3 331.9 110281 238.0 63.8 353.2 108.8 27.9 19.3 40.5 0.6 4.9 4.6 11.7
220.1 842.3 331.9 11028.1 238.0 63.8 353.2 108.8 27.0 19.1 39.3 0.6 4.9 4.6 11.6
385.3 842.3 331.9 11028.1 238.0 63.8 353.2 108.8 28.7 19.5 41.7 0.6 4.9 4.6 11.9
561.4 354.8 70.8 104.9 617.5 48.3 34.3 250.5 139.0 33.6 12.7 55.5 0.9 7.1 6.2 16.2
561.4 281.1 70.8
_
Stratum 6, 234? 3,1173 3,1173 3,1173 3,1173 3,1173 3,1173 3,1173 6,234? 6,234? 6,234? 6,234? 6,234? 6,234? 6,234;
Per cent Missing Number of Products Number of Subjects Level of Noise Dimensionality Products X Subjects Products X Noise Products X Dimensions Per cent Missing X Products Per cent Missing X Subjects Per cent Missing X Noise Per cent Missing X Dimensions Per cent Missing X Products X Subjects Per cent Missing X Products X Noise Per cent Missing X Products X Dimensions Between
bonus
the variance
Equivalent FValues DF
Data Matrices
rather
A pragmatic
are significant
Source of Variation
Between
of the mean,
factors and interactions
dom in their denominators);
data were used, some
a of
distributed.
are presented
F ratios were inspected.
for different
to 10.95)
of factors
Normally
tors influenced
yet
made
the differences
The results
the Mean
positive;
constant.
for each combination
was roughly
1994,
of the analysis of
by definition
combinations
implies
for each combination
percentage
an absolute
from
results
were log transformed
This was done
Differences combinations
Differences
is a constant being
limits might be nega-
transformation
the variation
was that the transformation
Multivariate tests (such
Trace,
in which
factors
data
designs.)
Squared factor
data
tests were inappropriate.
The Mean Squared before
other
were considered, because Mauchly’s was highly significant, indicating that
gives a clear and simple split-plot
from
incomplete
individual techniques
likely to be
the logarithmic
287
Stratum
X X X X X X X X X X X X X X
Per cent Missing Products Subjects Noise Dimensions Products X Subjects Products X Noise Products X Dimensions Per cent Missing X Products Per cent Missing X Subjects Per cent Missing X Noise Per cent Missing X Dimensions Per cent Missing X Products X Subjects Per cent Missing X Products X Noise X Per cent Missing X Products X Dimensions
N.B. For some effects, Numerator DF vary between variation does not affect the significance level.
2346
2,1174 4,234? 2,1174 2,1174 2,1174 2,1174 2,1174 2,1174 2,1174 4,234? 4,234? 4,234? 4,234? 4,234? 4,234? 4,234? and 2350
depending
617.5 48.3 34.3 250.5 139.0 32.7 12.7 54.0 0.9 7.1 6.2 16.0
on the test; however,
561.4 70.8 104.9 617.5 48.3 34.3 250.5 139.0 34.4 12.7 57.0 0.9 7.1 6.2 16.3
with such large values,
this
288
D. Hedderky, I. Wakeling TABLE 2a. Noise
TABLE 2f. Number of Dimensions
Noise
Dimensions
Root Mean Square Error
Low (SD = 1) High (SD = 2)
1.39” 2+31b
Root Mean
TABLE 2g. Technique
Low High
Square Error 10
Interactions with Technique
Noise
TABLE 2b. Number of Products
1G39b
X Noise
Mean Subs
Root Mean Square Error Row-Co1
E-M
l-39” 2.14b
1.38” 2.46d
1.40” 2.36”
(Means with different superscripts are significantly different, as judged by a Bonferonni corrected LSD test at 5% significance on the Log transformed data.)
1.70”
30
1.74” l-85’
2 4
(Means with different superscripts are significantly different, as judged by a Bonferonni corrected LSD test at 5% significance on the Log transformed data.)
Products
Root Mean Square Error
TABLE 2c. Products X Noise Noise Level
Products
Low High Low High
10 10 30 30
TABLE 2h. Technique
Root Mean Square Error
Root Mean Square Error E-M Row-Co1
Mean Subs
1.506 2.39” 1.29” 2.24’
1.73”
1.84’
1.82b
TABLE 2i. Technique
TABLE 2d. Number of Subjects
Percentage Missing
Root Mean Squared Error
Subjects
Root Mean Squared Error Mean Subs Row-Co1
5 35 65
1.86b 1.73”
50 200
1.70b 1.72hc 1.75’
TABLE 2j. Technique Number of Products
Root Mean
E-M
1.66” 1.82” 2.06’
TABLE 2e. Percentage Data Missing Percentage
X Missing
Level of Noise
1.67” 1.716 2.1d
X Products X Noise
Root Mean Square Error MeanSubs Row-Co1 E-M
Square Error
clear that most of the effects the F ratios
are used
to judge
effect is (relative to the inter-replicate 1 is an ordered based Trace The
Lambda.
(The
and the Hotelling-Lawley Level
influencing
of Noise variations
leads to better
recovery
is the
the
Figure
Approximations
patterns
for
Pillai’s
Trace
are similar.)
most
important
in the performance of the original
and
how large
variability).
bar chart of the FRatio
on Wilks’
10 30 30
are highly significant,
simply
higher
the proportion
recovery preference
dimensional Among
spaces
factor
next most
similar Mean
are
the worse the
and higher
recovered
1.46d 2.36’ 1.34’ 2.35’
dimensional
less well
than
low
ones.) the between-technique
on low noise Substitution
the main
1.4gd 2.6Y 1.296 2.29h
of data missing,
of the data matrix;
with Noise is most influential
(lower noise
data);
1.543 2.18s 1.25” 2.1of
Low High Low High
10
1.68” 1.75b 1.96”
5 35 65
effect
data,
performs of Technique
factors, the interaction (the three techniques
but with high better
than
is almost
noise
are data,
the others); as important.
important is the Number of Products (more products lead to better recovery, though the benefit is less with high noise data), and the Number of Subjects (more
(On average, Mean Substitution performs better than E-M, which is better than Row-Column Substitution.) The Technique X Missing interaction is the next most
subjects lead to better recovery). The Proportion of Missing Data and the Dimensionality of the preference
important
space
have
a lesser
influence
on
the
results.
(The
(the iterative
Column Substitution the percentage missing
techniques
-
E-M
and Row-
were more heavily affected by than Mean Substitution; with
Compan’son of Imputation Techniques for Internal Preference Mapping
.iir, , ,
289
T’ql’ f: ; r
.ji
;:t
i
;
0
1000
500
1500
0
2
1000
500
FIG.
1. FRatios
imputed
from analysis of variance
and actual
of difference
2
1500
3
Wilks’ F Ratio
Wilks F Ratio
between
FIG.
3.
F Ratios
from
analysis
of variance
of
consumer
configurations.
Data.
Notes
on Figures:
Effects
are ordered
within
strata
by size of
F ratio.
1obo
5io
0
Letters indicate factors: 7’ = Technique (Mean Substitution, Row-Co1 or E-M) ; M = Percentage Missing Data (5%, 35% or 65%); N = Level of Random Noise (Low or High); P = Number of Products (10 or 30); S = Number of Subjects (50 or 200) ; D = Number of Dimensions (2 or 4) ; Interactions are indicated with a * (For instance T*N is the interaction of Technique and Noise).
1500
Wilks’ F Ratio
FIG.
2. FRatios
from analysis of variance
of product
configu-
rations. low
levels of missing
Mean Substitution, they performed
data they performed
whereas worse),
at moderate
followed
Products
X Noise
interaction
iterative
techniques
perform
better
by the Technique
(with better
10 products, than Mean
tution when noise is low, but Mean Substitution with high noise.
With
always performs
better.)
Tables presented
of means
30 products,
mean
X the
Substiis better
Substitution
A similar
to these
effects
are
2a-j.
split-plot
ANOVA
pared
to the inter-replicate
necessary
to chart
It is clear Level
variation,
the probability
followed
are
the
by the Dimensionality of Missing
Data,
between
ences
are small, and the major
As above,
follow Table
configurations were
(for
both
log-transformed
performed
Products
before
on them
as a precaution
predictions, and in an attempt of observations from different Again,
apart
from
for each technique
analysis
were roughly Normal.
Consumers)
of variance against
was
negative
to equalise the variances combinations of factors.
the differing under
and
the
variances,
each combination
the results of factors
between ber
Technique,
of Products.
between-Technique effects
3 (Tables
of means
of Subdiffer-
are interactions
the Level of Noise, Tables
and
factors,
Noise and the
and the Number
Mean Squared Differences Between Configurations of Stimuli and Consumer Points Based on the Complete and Imputed Data Matrices
the
influential
of the data, the Num-
an interaction
By comparison,
between
it was
of Data Missing
most
jects.
Differences
so again
levels of the effects
that the Percentage
of Noise
ber of Products,
Squared
The
(Fig. 2).
Percentage
the Mean
was performed.
results for the Stimulus Configurations are presented in Table 3. Almost all the effects are significant com-
the
corresponding
in Tables
than
or high levels,
and the Num-
for
these
effects
3a-i) .
The higher the proportion
of the data which is missing,
the less accurate the recovery of the product configuration. Similarly,
the higher
less accurate action
shows that these
nations
between
the proportion
noise in the data, the
(The Missing by Noise inter-
differences
of Noise and Percentage
difference when
the random
the recovery.
low and
hold for all combiMissing,
high
of data missing
noise
but that the data
is higher.)
is less High
290
D. Hedderlq,
I. Wakeling TABLE 3. Analysis of Variance
for Product
Configurations
Equivalent FValues
Source of Variation
DF Between
Data Matrices
Per cent Missing X Products X Subjects Per cent Missing X Products X Noise Between
X Products
Techniques
Technique Technique Technique Technique Technique Technique Technique Technique Technique Technique Technique Technique Technique
Technique Technique Technique
X X X X X X X X X X X X X X X
H-L Trace
Pihi’S
Stratum
Per cent Missing Number of Products Number of Subjects Level of Noise Dimensionality Products X Subjects Products X Noise Products X Dimensions Per cent Missing X Products Per cent Missing X Subjects Per cent Missing X Noise Per cent Missing X Dimensions
Per cent Missing
wii’
X Dimensions
7.1
342.2 258.4 94.2 1098.1 345.5 6.8 48.2 27.0 22.4 6.8 100.3 6.0 5.8 8.4 7.0
2702.2 258.4 94.2 1098.1 345.5 6.8 48.2 27.0 22.4 6.9 129.7 6.0 5.8 8.5 7.1
23.0
23.0
7g.i 26.1 80.8 17.4 4.5 55.1 38.0 24.0 3.4 26.5 4.2 7.5 8.2 7.3
7i.i 26.1 80.8 17.4 4.5 55.1 38.0 23.9 3.4 25.9 4.2 7.5 8.1 7.3
23.0 6.0 70.3 26.1 80.8 17.4 4.5 55.1 38.0 24.0 3.4 27.0 4.2 7.6 8.3 7.4
6,234? 3,1173 3,1173 3,1173 3,1173 3,1173 3,1173 3,1173 6,234? 6,234? 6,234? 6,234? 6,234? 6,234? 6,234?
1115.2 258.4 94.2 1098.1 345.5 6.8 48.2 27.0 22.4 6.9 114.8 6.0 5.8 8.4
2,1174 4,234? 2,1174 2,1174 2,1174 2,1174 2,1174 2,1174 2,1174 4,234? 4,234? 4,234? 4,234?
Stratum
Per cent Missing Products Subjects Noise Dimensions Products X Subjects Products X Noise Products X Dimensions Per cent Missing X Products Per cent Missing X Subjects Per cent Missing X Noise
Per cent Missing X Dimensions Per cent Missing X Products X Subjects Per cent Missing Per cent Missing
X Products X Products
TABLE 3a. Percentage
X Noise X Dimensions
4,234? 4,234?
4,234?
TABLE 3d. Number
of Data Missing
Percentage Missing
Root Mean Squared Error
Number of Products
0.27” 0.936 1.49’
(Means with different superscripts are significantly different, as judged by a Bonferonni corrected LSD test at 5% significance on the Log transformed data.) TABLE 3b. Level of Noise Root Mean Squared Error
Low (SD = 1) High (SD = 2)
0.53” 0.99b
Root Mean Squared Error
2 4
0.82’ 0.63”
30
TABLE 3e. Percentage Percentage
Data Missing
X Noise
Noise Level
Root Mean Squared Error
5
Low
5 35 35 65 65
High Low High Low High
0.18” 0.46’ 0.69’ 1.24d 1.28” 1.73’
TABLE 3c. Number of Dimensions Dimensions
of Products
10 5 35 65
Noise
TABLE 3f. Number of Subjects
Root Mean Squared Error 0.60” 0.866
Subjects 50 200
Root Mean Squared Error O-66” 0.78’
Compatison of Imputation Interactions with Technique TABLE Noise
dimensional
3g. Technique
product
X Level of Noise
more
Root Mean Squared Error Mean Subs E-M Row-Co1
Low High
O-51” 1.04”
0.56” 0.96’
the
3h. Technique
Products
X Number
0.86’ 0.62”
TABLE 3i. Technique Products Noise
10 10 30 30
X Number
of Products
0.70’
High Low High
l.Od 0.44” 0.87”
0.58” 1.19” 0.45” 0.90’
techniques
with a large because
X Level of Noise
Between
Techniques X X X X X X X X X X X X X X X
performs
and
better
number
it appears
X Noise
other
techniques.
tion’s
performance
of products,
(especially)
the
E-M
than Mean Substitution,
they all perform X Products
Similarly,
is more
The
is mostly sensitive
of products
Row-Column
at different
but
similarly.
interaction
Substitution
to
than the Substitu-
levels of missing
data
seems to be more sensitive to noise than the other
tech-
The
results in Table
for the Assessor
almost all the factors it was necessary to judge
configurations
4. As with the Stimulus and interactions
their relative importance
for Consumer
can be
configurations, are significant,
to chart their probability
so
levels in order
(Fig. 3).
Configurations Equivalent
F Values
Wilks’
Pillai’s
6,234? 3,1173 3,1173 3,1173 3,1173 3,1173 3,1173 3,1173 6,234? 6,234? 6,234? 6,234? 6,234? 6,234? 6,234?
1716.7 104.7 971.6 1725.6 848~ 1 9.8 113.1 75.8 21.2 8.5 149.8 2.9 6.5 13.2 22.2
376.8 104.7 971.6 1725.6 848.1 9.8 113.1 75.8 21.0 8.5 125.6 2.9 6.4 13.2 21.7
5392.6 104.7 971.6 1725.6 848.1 9.8 113.1 75.8 21.4 8.5 175.1 2.9 6.5 13.3 22.7
2,1174 4,234? 2,1174 2,1174 2,1174 2,1174 2,1174 2,1174 2,1174 4,234? 4,234? 4,234? 4,234? 4,234? 4,234? 4,234?
9.0 11.2 113.3 25.5 179.9 25.1 9.5 90.5 61.8 27.6 9.1 57.0 1.7 8.4 11.5 13.2
9.0 11.1 113.3 25.5 179.9 25.1 9.5 90.5 61.8 27.2 9.0 54.8 1.7 8.4 11.5 13.0
9.0 11.3 113.3 25.5 179.9 25.1 9.5 90.5 61.8 28.0 9.1 59.2 1.7 8.5 11.5 13.3
H-L Trace
Stratum
Stratum
Per cent Missing Products Subjects Noise Dimensions Products X Subjects Products X Noise Products X Dimensions Per cent Missing X Products Per cent Missing X Subjects Per cent Missing X Noise Per cent Missing X Dimensions Per cent Missing X Products X Subjects Per cent Missing X Products X Noise Per cent Missing X Products X Dimensions
less
slightly worse
With a small number
Row-Column
found
0.58” 1.05’ 0.45” 0.92’
Per cent Missing Number of Products Number of Subjects Level of Noise Dimensionality Products X Subjects Products X Noise Products X Dimensions Per cent Missing X Products Per cent Missing X Subjects Per cent Missing X Noise Per cent Missing X Dimnsions Per cent Missing X Products X Subjects Per cent Missing X Products X Noise Per cent Missing X Products X Dimensions Technique Technique Technique Technique Technique Technique Technique Technique Technique Technique Technique Technique Technique Technique Technique Technique
the
positions.
niques.
DF Data Matrices
Curiously, set,
with Technique,
the noise level with small number
0.78’ 0.64”
positions.
in a data
of the
leads to a
with high noise data, and Mean Substitu-
perform
Technique
recovery
products
of the product
Substitution
of Variation
Between
are
Substitution
better.
Row-Column
TABLE 4. Analysis of Variance Source
of their
there
the recovery
than expected
Root Mean Squared Error Mean Subs E-M Row-Co1
LOW
while more
recovery
subjects
tion slightly
of Products
0.83” 0.63h
to less accurate
leads
As for the interactions
Root Mean Squared Error Mean Subs Row-Co1 E-M
10 30
accurate more
that Row-Column
Means with different superscripts are significantly different, as judged by a Bonferonni corrected LSD test at 5% significance on the Log transformed data. TABLE
data
configuration,
accurate
0.51” 0.98’
291
Techniques for Internal Preference Mapping
292
D. Hedderlq,
I. Wakeling
TABLE 4a. Level of Noise
TABLE 4f. Number of Products X Level of Noise Root Mean SquaredError
Noise Low (SD = 1) High (SD = 2)
0.32” 0.50b
(Means with different superscripts are significantly different, as judged by a Bonferonni corrected LSD test at 5% significance on the Log transformed data.)
Products
Noise
Root Mean SquareError
10 10 30 30
Low High Low High
0.32” 0.46’ 0.32” 0.54’
Interactions with Technique TABLE 4b. Percentage of Data Missing Percentage
Noise
0.18” 0.50h 0.72”
Low High
5 35 65 TABLE 4c. Subjects Number of Subjects
TABLE 4g. Technique
Root Mean SquaredError
Root Mean SquaredError
Root Mean SquareError Mean Subs Row-Co1 E-M 0.326 0.47’
TABLE 4h. Technique Number of Products
TABLE 4d. Dimensions Number of Dimensions
Root Mean SquaredError
4
Percentage
Noise Level
Root Mean SquaredError
Low High Low High Low High
0.14” 0.25’ 0.40’ 0.63d 0~64~ 0.82”
5 5 35 35 65 65
Number of Products
Noise
10 10 30 30
Low High Low High
Subjects,
most
significant
the Perecentage
factors
are
between
the Percentage
Noise and the Number
Number
of
of Missing Data, the Level of
Noise in the data, the Dimensionality actions
the
of the data, inter-
Missing
of Products.
Data and the
Again,
differences
between the Techniques are less significant, the primary ones being an interactions with the Level of Noise and the Number
of Products.
Means
for these
0.38’ 0.41d
more the
effects
dimensions
Noise
and Proportion
recovery
the
and more
0.29” 0.49 0.32’ 0.5w
of the data missing The
interaction
of
is the same as with the
the more
difference
and
0.29”” 0*5og 0.32’ 0.54’
Missing
data. More products recovery,
0.37” 0.42’
Number of Products X Noise
less accurate.
configuration;
less the
X
0.33” 0.44’ 0.29b’ 0.5Zh
make
the
Number of Products
Root Mean SquaredError Mean Subs Row-Co1 F,-M
noise,
product The
X
0.38’ 0.40’
TABLE 4i. Technique
TABLE 4e. Percentage Data Missing X Level of Noise
0.29” 0.51d
Root Mean SquaredError Row-Co1 Mean Subs E-M
10 30
0.33” 0+46b
2
0.29” 0.52’
(Means with different superscripts are significantly different, as judged by a Bonferonni corrected LSD test at 5% significance on the Log transformed data.)
0.47b 0.33”
2::
Noise
X
between
data that is missing, high
and
low noise
in the data set lead to less accurate differences
between
noise data are more pronounced
low and
high
with 30 products
than
with 10. The Technique although the three
X Noise interaction techniques perform
low noise data, Mean Substitution
performs
shows that, similarly with better
than
are given in Tables 4a-j. The results for Subjects, Proportion of Data Missing, Dimensionality and Noise are as might be expected;
the others on high noise data. This is most pronounced with 65% missing data (see the Technique X Missing
with more subjects the recovery of the consumer configuration is more accurate, but a higher level of
interaction techniques
X
Noise
interaction). indicates perform
The
Technique
X
Products
that the E-M and Row-Column less well than Mean Substitution
Comparison of Imputation Techniques for Internal Preference Mapping TABLE
5. Confirmatory
Analysis:
Comparing
Imputed
and Actual
Data Matrices
FFtatio
Source
DF
ss
MS
Technique (Means, Row-Co1 or E-M) Per cent Missing (5%, 35% or 65%) %Missing X Technique Residual Total
2 2 4 441 449
0.09 1.22 0.55 3.26 5.11
0.043 0.609 0.137 0.007
293
Significance <0.05
5.79 82.33 18.58
Values quoted are based on Type III Sums of Squares (change in Sum of Squares of model when the relevant term is fitted last: SAS/S?‘A7’Munual, SAS Institute, 1989) from the analysis of the log-transformed variables.
Selected Means TABLE
5a. Percentage
Missing
TABLE Root Mean Squared Error
Per cent Missing
1.55” 1.58” 1.65’
5% 35% 65%
with high
number
X Products other
of products;
X Noise
Row-Column E-M
Per cent Missing
interaction
Substitution
techniques performing
while seems
with 10 products worse
with
30
random
and high noise, products
and
Only the three
and
set supplied
high
Analysis
studied
Switzerland).
The data matrix consisted
X 29 Stimuli, implies
that the preference TABLE
6. Confirmatory
space
Centre,
Analysis:
Comparing
converged
It Stimulus
Configurations
Source
DF
ss
Technique (Means, Row-Co1 or E-M) Per cent Missing (5%, 35% or 65%) % Missing X Technique Residual Total
2 2 4 441 449
13.42 1132.11 6.24 70.54 1222.31
data
and
the
results
the
Unfortunately,
data
set, the factors
to the proportion
different
imputation
are presented
actual
and
in Tables
imputed
data
Per cent Missing 5% 35% 65%
based on Imputed MS
and Complete
5, 6
Significance
FRatio
6.71 566.05 1.56 0.16
Data
41.95 3538.85 9.75
when the relevant
term is fitted
Missing TABLE
Root Mean Squared Error 0.16” 0.59b 1.02’
tech-
matrices
Selected Means 6a. Percentage
reli-
of the data were taken
was only a single
Values quoted are based on Type III Sums of Squares (change in Sum of Squares of model SAS/STAT Manual, SAS Institute, 1989) from the analysis of the log-transformed variables.
TABLE
E-M
(Table 5), all three effects (the imputation technique, the proprotion of missing data, and their interaction)
matrix
is 2-dimensional.
Substitution,
on them.
in the ANOVA were limited
Comparing
of 166 Assessors
and the analysis of the complete
of the level of
which
logarithms ANOVA
niques used. The and 7.
on a real data
Research
(Mean
Substitution)
natural
there
of missing
runs (50 runs at each of the (Nestle
an estimate
techniques
performing
because
by P. Leathwood
1.54” 1.54” 1.66’
ably were used, to save time. As before,
3 levels of missing values) was performed
E-M
1.52” 1.61’ 1.68’
to make
and Row-Column
before
A similar set of simulation
Error
noise in the data.
than
noise.
Confirmatory
X Technique
Mean Squared Row-Co1
1.49” 1.59b 1.61’
was not possible
to be due to worse
Root Mean Subs
5% 35% 65%
the Technique
performing
5b. Per cent Missing
Mean Subs 0.42”
6b. Technique
Root Mean Squared Row-Co1 0.51’
Error E-M 0.446
last:
294
D. Hedderley, I. Wake&g
were
significant;
data (Table
however,
5a) dominated
the proportion the effects.
(Table 5b) follows the same pattern study (Table Substitution
2h) declines
tion of missing
forms similarly with 5 or 35% of the observations ing
as found in the main
the performance the E-M
(before
mance
deteriorating
as
missing),
of Row-Column
with each increase
data, whereas
of missing
The interaction
the
in the propor-
is affected
missing data.
TABLE 7. Confirmatory Analysis: Computing Assessor Configurations
Source
DF
Technique (Means, Row-Co1 or E-M) Per cent Missing (5%, 35% or 65%) %Missing X Technique Residual Total
z 4 441 449
much
less by increases
miss-
level of perfor-
technique of mean
at
65%
substitution
in the proportion
of
based on Imputed and Complete Data MS
FRatio
1.71 368.15 0.37 0.05
33.65 7236.65 7.21
SS
3.42 736.29 1.47 22.43 763.62
Row-Column
while the performance
algorithm
per-
to a similar
Significance
Values quoted are based on Type III Sums of Squares (change in Sum of Squares of model when the relevant term is fitted last: SAS/SII’ATMunual, MS Institute, 1989) from the analysis of the log-transformed variables.
Selected Means (Log, Mean Squared Difference) TABLE 7a. Percentage
Missing TABLE
Root Mean
Per cent
7b. Technique
Squared Error
Missing
Root Mean Squared Error 5%
0.11”
35%
0.34*
65%
0.49’
TABLE 8. Adequacy
StiIUUli
Assessor
10
50
10
Dimens
Thresholds Noise
Mean Subs
Row-Co1
E-M
0.28’
0.26”
0.25”
and Numbers Missing
of Simulation
Runs Within Those
Adequacy Limit
Thresholds
Number of Runs within Threshold Mean
E-M
Row-Co1
Low
5% 35% 65%
0.33
50 26 0
50 43 5
50 46 13
200
High
5% 35% 65%
2.29
50 48 30
50 50 34
50 45 23
30
50
High
5% 35% 65%
0.62
50 46 4
50 47 10
50 43 11
30
200
Low
5% 35% 65%
0.19
50 48 0
50 36 0
50 37 0
10
50
High
5% 35% 65%
0.78
50 44 0
50 38 0
50 32 0
10
200
Low
5% 35% 65%
0.41
50 19 0
50 30 0
50 33 0
30
50
Low
5% 35% 65%
0.15
50 50 0
50 49 0
50 50 0
30
200
High
5% 35% 65%
0.72
50 48 4
50 47 0
50 44 0
Compatison of Imputation Techniques for Internal Preference Mapping Comparing
the Stimulus
again all three
effects
the proportion
of missing
the techniques, ration
more
(Table
(Table
6))
the imputation
but dominated
by
the levels chosen
data (Table
mean substitution accurately
both perform
Configurations
are significant,
than
6a). Comparing
recovers
the E-M
the configu-
algorithm,
much better than Row-Column
and
substitution
6b).
Similar rations
results were found
(Table
Adequacy
for the Assessor
Configu-
7, 7a and 7b).
apart;
do not vary as much, mance between will not
performance
derived
the data sets, and the number the three
techniques
Expectation-Maximisation, tion)
gave similar
With
the simulation
runs were within percentile
95th
differences Values,
based
on
a majority with
adequate.
65%
do occur,
were data,
differences
appear
to be any literature
sumer
data;
2-dimensional,
in
some samples has shown
and 2.24
all
Missing
the
not
threshold;
very few results
were
the techniques
in the
10 Stimuli,
data,
(on a g-point
replicates,
implying
and
the
the
Stimulus,
data,
and
the three
reliably,
10
stitution)
that it doesn’t
results
make much
one uses; performance
techniques
from
the
techniques
difference
(on all three
data missing, used).
to be in their
level of noise,
and the number
Moreover,
Dimensions
them playing a fairly minor significant tion runs).
compared
mea-
by features
the proportion
of stimuli
role (although
to the variation
of
and assessors
with interactions
seems
between
they are still
between
iterative
formance
measures,
implying The ence
effect of
ones
simula-
E-M
or
data, but
than the non-iterathe Technique
was significant its F ratio
X
for all three per-
was never
of the number
the
the
of subjects
product
very large,
number
of
products
was also unexpected.
tend
subject
Intuitively,
a larger
it;
to be further
studies with more
from
(higher
more
uncertainty
accuracy
of their
the
negative
configuration,
effect
in
upon
.judging
the the
difference)
of subjects
(say)
the product
for both products
of higher
own configuration effect
and similarly recovered
simultaneously,
in locating
and vice versa. However, the positive
the complete
total squared
when fitting a large number
the
of
product
It may be that because
are estimated
uncertainty
subjects,
instead
are less accurately
products.
two configurations
also causes
the accuracy
when one has more subjects,
configurations
higher
not
the
the
affect
at least
and
on
recovery,
or
on the differ-
configurations,
to improve
other
(like
data sets. We did not
data set might be expected
weigh
of the
(like Mean Sub-
it was never very influential.
between
points,
of the tech-
on 2-dimensional
were better
interaction
produced
which technique
the main effect of these factors
Main Effects,
conimply
performance
sures) seems to be more strongly influenced of the data set (the
to con-
which
of the main reasons
techniques
Substitution)
configurations
simulation
One
of the the data
on the dimensionality
outperformed
the iterative
effect
FURTHER oneself
for
in the study was that Bello
find this effect in this study; although
High Noise data, the
one restricts
important.
data; non-iterative
data solution
imputation
the
that the dimensionality
10
200
the
1.47
the scores
levels used in this
tive ones on higher-dimensional
worse than the
30
original
subject sidering
that the noise
50
fare worse than Mean Substitution.
that once
of between
between
space which was used to generate
configurations
verge
the data
within tasting ses-
deviations scale)
It was also surprising preference
Row-Column
Low Noise data,
Low Noise
AND
were replicated
standard
that the relative performance
35%
noticeably
while
sions
noted
Stimulus, 50 Assessor, 4dimensiona1,
It is interesting
not
convincingly
varied depending
between
performs
DISCUSSION DEVELOPMENTS
because
to produce
(1993)
Low Noise
techniques
seemed
niques
At
4dimensiona1,
iterative
Values
runs (although
200 Assessor,
Assessor,
the levels were chosen
Dimensionality
within
a pattern;
techniques;
assessors does
on the levels of noise in con-
set was not more
2-dimensional,
iterative
in perfor-
there
for including
Stimulus,
Substitution
Unfortunately,
of the
Assessor, Mean
be as great.
thresh-
resampling).
missing
to discern
the difference
data
study are reasonable.
Substitu-
it is usually at the 35% Missing level; however
it is hard
then
far
consumer
distribution
of the simulation
Where
in genuine
the adequacy
of the
an overwhelming majority) while
Substitution,
5% Missing
old
(the
(Mean
in Table 8.
and Row-Column
results.
from
of simulation
runs which passed those limits are presented In general,
may be because
‘noisy’ preference maps. However, subsequendy, analysis of several unpublished consumer preference trials
of ‘adequate’
resampling
This
were unrealistically
a panel of more or less accurate
sets they produced
of the Results
limits
used.
for the factor
if the levels of noise
where The
technique
295
and
numbers
on the
appears
to out-
accuracy
by the variance
of the ratios
of
the effects. Some
of these conclusions
simulation
process
were supported
was applied
when the
to a real data set. How-
on the
ever, because only one data set was available, it was not possible to confirm the relative importance of a num-
results might be expected, but it is surprising that it has more influence than, say, the size of the data matrix, or
ber of the factors (the size of the data set, the level of noise, and the dimensionality of the preference space).
The On
importance
the basis
of the noise
of previous
studies
factor some
was surprising. effect
296
D. Hedderlq, I. Wakeling
Given
that
each
requirements been
consumer
trial
and features,
available
structure
data sets had
centred
that they would have fitted
erence
mapping,
mean,
rather
to replace
testing of all the effects studied in the simulation. The adequacy tests imply that all the techniques
give
the values are missing.
much
the results may be questionable, while at 65% no tech‘sufficiently good results’, reliably produces nique regardless of the size of the data set or the accuracy of
Product
raised by a reviewer was that rotating
and Consumer
unreasonable,
and
a
better
would have combined fit of both A single carded
and Consumer
(which
points than the Product
for
picture
but
dis-
criteria
instance,
always contained
configuration),
have been
using
the
Product
and the Consumer the Mean
A number which
to rotate
to be applied
to both
of
bers
have equal influence
of points
in each;
Product
and
imputed
data reconstruct
Consumer
despite
the Product
a and
the different
the original
based matrix
ence scores. Given the comparatively
small differences
mance
most successful
between
the three
our recommendation tion
(replacing
num-
how well the
configurations
on the
of preferin perfor-
techniques,
would be to use mean
substitu-
missing values with the mean
score for
the corresponding
stimulus),
since it is simple
to pro-
gram, executes almost instantaneously, and performs at least as well as the more complex iterative techniques. One
refinement
take account
we would
of the assessor’s
as well as whether below average
recommend
by the group
would
average preference
the stimulus
be to score,
effects
(and
the
grand
means
scores
to
obtain
an
the data sets; however, in actual trimight be expected to have different
it may improve
performance
conducting
the
techniques
performance
original
(by reducing
an appropriate
authors
would
throughout
like
the project;
on the rotation Second
trial
sources
as in an
of random
sample size, etc.).
to thank
two anonymous
Dr Neil Gains for suggestions
problem;
Sensometrics
and several
Group
attendees
Meeting
at the
in Edinburgh
for
imputation-by-ANOVA.
This work was conducted of Agriculture,
Fisheries
as part of a U.K. Ministry and
Mathematical
Food
LINK
Methods
project
for
on
Preference
Mapping.
REFERENCES Beale E. M. L. & Little R. J. A. (1975). Missing values in multivariate analysisJ. R. StatisticalSot., Series B, 37, 129-45. Bello, A. L. (1993). Choosing among imputation techniques for incomplete multivariate data; a simulation study. Commun. Stat. Themy &Methods, 22 (3)) 853-77. Box G. E. P., Hunter W. G. & Hunter J. S. (1978). Statistics/or Exptimenters. Wiley, New York. Buck,
noticeably.
S. F. (1960).
in multivariate
A method
of estimation
data suitable
of missing
values
for use with an electronic
com-
puter.~J. R Statistical Sot. Series B, 22, 302-6. van Buuren,
S. & van Rijckevorsel,
of missing
categorical
J. L. A. (1992).
data by maximising
Imputation
internal
consis-
tency. Psychomettika, 57,567-80. Dempster
A. P., Laird
likelihood
N. & Rubin
estimation
algorithm
from
D. B. (1977).
incomplete
(with discussion)
.J
data
Maximum via the
EM
R Statistical SOL, Series B, 39,
l-38.
as a whole; for
imputed value. In the simulation study, this was not an important factor because no special effort was made to introduce different mean scores for different assessors when constructing als, where subjects
to improve
reviewers for drawing attention to more appropriate ways of analysing the data; Dr Hal MacFie for guidance
with
mean)
esti-
ACKNOWLEDGEMENTS
Gabriel
above
instance by using regression or ANOVA to calculate assessor and stimuli effects, and then summing the relevant
to the imputation
going
manner
noise, choosing
or
was scored
of assessors
be used
or ANOVA
and then calculating
or comparing
that the subject should
the
Consumer configurations, using a weighted Procrustes criterion in which the Product and Consumer configurations
appropriate
Improved
best
both
configurations, Differences;
not
as would
suggesting
subsequently:
matrix
configurations Squared
more
and so not give
suggested
rotation
the Product
single rotation,
of fit of the Con-
of the performance.
matches checking
configurations.
by the goodness
mean,
data if regression
refinements
probably
The
of
it was felt that the sum of squares would
configuration
alternative
criterion
initially,
are
was
on the goodness
was considered
have been dominated
a balanced
the
separately
performance
information
the Product criterion
because
sumer
configurations
one reviewer suggested
than the product
missing
However, are
With 35% of the values missing,
scores for each subject
of zero as the first step in pref-
mates could not be used.
results which are sufficiently close to those which would be obtained from a complete data matrix if only 5% of
the assessors. One point
to have a mean
which would have allowed the
it is unlikely
into a factorial
In fact, since the preference
has its own unique
even if more
K R. (1971).
The biplot to
application
graphical
principal
display of matrices
component
analysis.
Biometrika, 58,453-67. Gabriel
K. R. & Zamir
of matrices
S. (1979).
by least
squares
Technometrics, 21, 489-98. King B. M. & Arents P. (1991). obtained
from generalised
Lower with
rank approximation
any choice
A statistical
Procrustes
of weights.
test of consensus
ana1ysis.J.
Sens. Stud.,
6,37-48. Kruskal, J. B. (1964). goodness
l-27.
Multidimensional
of fit to a nonmetric
scaling
hypothesis.
by optimizing
Psychometrika, 29,
Comparison of Imputation Techniques for Internal Preference Mapping Krzanowski
W. J.
Oxford
University
Krzanowski
(1988a).
Principles of Multivariate
W. J. (1988b). data
matrix.
Biometrical L&t., XXV,
using
the
R. J. A. & Rubin,
value imputation
singular
value
in multi-
decomposition
of a
DSWO Rubin
J.
(1982).
D. B. (1987).
Statistical Analysis with
SAS Institute
(1989).
Users Guide Chap.
Homogeneity
Analysis of Incomplete Data. Psychometriku, 56,241-54.
‘The PRINQUAL 34.
SAS
PRINQUAL Car-y NC.
Spence
Institute
Procedure’, Inc.,
Gary
.SAS/STAT NC,
pp.
Technical Report R-108: Algorithms for the
and TRANSREG
Procedures. SAS Institute
Inc.,
sensory
I. & Domoney designs
and
designs
consumer
D. W. (1974).
for
and repeated sciences.
Food
nonmetric
Single
subject
multidimensional
incomscaling.
Psychometriku, 39,469-90. I. N., Raats M. M. & MacFie
significance Whelehan of
test
H. J. H. (1992).
in generalised
H. J. H. & Baust,
differences
recovery
Wu C. F. J. (1983). algorithm.
consensus
P., MacFie,
individual
simulated
for
A new
Procrustes
Sens. Stud., 7, 91-6. 0.
rates and error (1990).
Uses of change-over in
PreJ, 4, 223-36.
ana1ysis.J.
EM and beyond.
1265-323. SAS Institute
@al.
Wakeling
Press, Leiden
D. B. (1991).
P. (1994).
plete
31-9.
Missing Data. Wiley, New York. Meulman
Schlich
measurements
Missing
variate Little
Analysis.
Press, Oxford.
297
scaling
of structure
1evels.J.
under
for
N. G. (1987). sensory
various
Use
studies:
missing
value
Sens. Stud., 1, l-8.
On the convergence
Ann. of Statistics, 11, 95-103.
properties
of the EM