The restricted EM algorithm under inequality restrictions on the parameters

One of the most powerful algorithms for maximum likelihood estimation for many incomplete-data problems is the EM algorithm. The restricted EM algorithm for maximum likelihood estimation under linear restrictions on the parameters has been handled by Kim and Taylor (J. Amer. Statist. Assoc. 430 (1995) 708-716). This paper proposes an EM algorithm for maximum likelihood estimation under inequality restrictions A₀β?0, where β is the parameter vector in a linear model W=Xβ+ε and ε is an error variable distributed normally with mean zero and a known or unknown variance matrix Σ>0. Some convergence properties of the EM sequence are discussed. Furthermore, we consider the consistency of the restricted EM estimator and a related testing problem.     

����������������������ŷ���ģ�͵��Ʋ�ǰ���ӯ��ֲ�

本文综述混合效应模型参数估计方面的若干新进展. 平衡混合效应方差分析模型的协方差阵具有一定结构. 对这类模型, 文献[1]提出了参数估计的一种新方法, 称为谱分解法. 新方法的突出特点是, 能同时给出固定效应和方差分量的估计, 前者是线性的, 后者是二次的,且相互独立. 而后, 文献[2--9]证明了谱分解估计的进一步的统计性质, 同时给出了协方差阵对应的估计, 它不仅是正定阵, 而且可获得它的风险函数, 这些文献还研究了谱分解估计与方差分析估计, 极大似然估计, 限制极大似然估计以及最小范数二次无偏估计的关系. 本文综述这一方向的部分研究成果, 并提出一些待进一步研究的问题.     

Likelihood-Based Analysis of Linear State-Space Models Using the Cholesky Decomposition

Data associated with the linear state-space model can be assembled as a matrix whose Cholesky decomposition leads directly to a likelihood evaluation. It is possible to build several matrices for which this is true. Although the chosen matrix or assemblage can be very large, rows and columns can usually be rearranged so that sparse matrix factorization is feasible and provides an alternative to the Kalman filter. Moreover, technology for calculating derivatives of the log-likelihood using backward differentiation is available, and hence it is possible to maximize the likelihood using the Newton–Raphson approach. Emphasis is given to the estimation of dispersion parameters by both maximum likelihood and restricted maximum likelihood, and an illustration is provided for an ARMA(1,1) model.     

The Asymptotic Distribution of REML Estimators

Restricted maximum likelihood (REML) estimation is a method employed to estimate variance-covariance parameters from data that follow a Gaussian linear model. In applications, it has either been conjectured or assumed that REML estimators are asymptotically Gaussian with zero mean and variance matrix equal to the inverse of the restricted information matrix. In this article, we give conditions under which the conjecture is true and apply our results to variance-components models. An important application of variance components is to census undercount; a simulation is carried out to verify REML′s properties for a typical census undercount model.     

Relative efficiencies of estimates using patterned covariances or correlations in the multivariate normal estimation problem

Summary The relative efficiency of maximum likelihood estimates is studied when taking advantage of underlying linear patterns in the covariances or correlations when estimating covariance matrices. We compare the variances of estimates of the covariance matrix obtained under two nested patterns with the assumption that the more restricted pattern is the true state. Formulas for the asymptotic variances are given which are exact for linear covariance patterns when explicit maximum likelihood estimates exist. Several specific examples are given using complete symmetry, circular symmetry and general covariance patterns as well as an example involving a covariance matrix with a linear pattern in the correlations.     

Hamiltonicity of regular graphs and blocks of consecutive ones in symmetric matrices

We show that the Hamiltonicity of a regular graph G can be fully characterized by the numbers of blocks of consecutive ones in the binary matrix A+I, where A is the adjacency matrix of G, I is the unit matrix, and the blocks can be either linear or circular. Concretely, a k-regular graph G with girth g(G)?5 has a Hamiltonian circuit if and only if the matrix A+I can be permuted on rows such that each column has at most (or exactly) k-1 circular blocks of consecutive ones; and if the graph G is k-regular except for two (k-1)-degree vertices a and b, then there is a Hamiltonian path from a to b if and only if the matrix A+I can be permuted on rows to have at most (or exactly) k-1 linear blocks per column.Then we turn to the problem of determining whether a given matrix can have at most k blocks of consecutive ones per column by some row permutation. For this problem, Booth and Lueker gave a linear algorithm for k=1 [Proceedings of the Seventh Annual ACM Symposium on Theory of Computing, 1975, pp. 255-265]; Flammini et al. showed its NP-completeness for general k [Algorithmica 16 (1996) 549-568]; and Goldberg et al. proved the same for every fixed k?2 [J. Comput. Biol. 2 (1) (1995) 139-152]. In this paper, we strengthen their result by proving that the problem remains NP-complete for every constant k?2 even if the matrix is restricted to (1) symmetric, or (2) having at most three blocks per row.     

Differentiation of the Cholesky Algorithm

Abstract

One way to estimate variance components is by restricted maximum likelihood. The log-likelihood function is fully defined by the Cholesky factor of a matrix that is usually large and sparse. In this article forward and backward differentiation methods are developed for calculating the first and second derivatives of the Cholesky factor and its functions. These differentiation methods are general and can be applied to either a full or a sparse matrix. Moreover, these methods can be used to calculate the derivatives that are needed for restricted maximum likelihood, resulting in substantial savings in computation.     

The extent of the maximum likelihood estimator for the extreme value index

In extreme value analysis, staring from Smith (1987) [1], the maximum likelihood procedure is applied in estimating the shape parameter of tails—the extreme value index γ. For its theoretical properties, Zhou (2009) [12] proved that the maximum likelihood estimator eventually exists and is consistent for γ>−1 under the first order condition. The combination of Zhou (2009) [12] and Drees et al (2004) [11] provides the asymptotic normality under the second order condition for γ>−1/2. This paper proves the asymptotic normality for −1<γ≤−1/2 and the non-consistency for γ<−1. These results close the discussion on the theoretical properties of the maximum likelihood estimator.     

基于最优估计的数据融合理论

本文提出了一种最优加权的数据融合方法，分析了最优权值的分配原则；给出了多源信息统一的线性融合模型，使其表示不受数据类型和融合系统结构的限制，并指出在噪声协方差阵正定的前提下，线性最小方差估计融合和加权最小二乘估计融合是等价的；介绍了数据融合中的Bayes极大后验估计融合方法，给出了利用极大后验法进行传感器数据融合的一般表示公式；最后以两传感器数据融合为例，证明了利用Bayes极大后验估计进行两传感器数据融合所得到的融合状态的精度比相同条件下极大似然估计得到的精度要高，同时它们均优于任一单传感器局部估计精度。     

A Note on Random Effects Growth Curve Models

The robustness of regression coefficient estimator is a hot topic in regression analysis all the while. Since the response observations are not independent, it is extraordinarily difficult to study this problem for random effects growth curve models, especially when the design matrix is non-full of rank. The paper not only gives the necessary and sufficient conditions under which the generalized least square estimate is identical to the the best linear unbiased estimate when error covariance matrix is an arbitrary positive definite matrix, but also obtains a concise condition under which the generalized least square estimate is identical to the maximum likelihood estimate when the design matrix is full or non-full of rank respectively. In addition, by using of the obtained results, we get some corollaries for the the generalized least square estimate be equal to the maximum likelihood estimate under several common error covariance matrix assumptions. Illustrative examples for the case that the design matrix is full or non-full of rank are also given.     

On the estimation of parameters for linear stochastic differential equations

The estimation of arbitrary number of parameters in linear stochastic differential equation (SDE) is investigated. The local asymptotic normality (LAN) of families of distributions corresponding to this SDE is established and the asymptotic efficiency of the maximum likelihood estimator (MLE) is obtained for the wide class of loss functions with polynomial majorants. As an example a single-degree of freedom mechanical system is considered. The results generalize [8], where all elements of the drift matrix are estimated and the asymptotic efficiency is proved only for the bounded loss functions. Received: 12 March 1997 / Revised version: 22 June 1998     

Error bounds for linear complementarity problems of DB-matrices

Doubly B-matrices (DB-matrices), which properly contain B-matrices, are introduced by Peña (2003) [2]. In this paper we present error bounds for the linear complementarity problem when the matrix involved is a DB-matrix and a new bound for linear complementarity problem of a B-matrix. The numerical examples show that the bounds are sharp.     

A novel trace test for the mean parameters in a multivariate growth curve model

A trace test for the mean parameters of the growth curve model is proposed. It is constructed using the restricted maximum likelihood followed by an estimated likelihood ratio approach. The statistic reduces to the Lawley-Hotelling trace test for the Multivariate Analysis of Variance (MANOVA) models. Our test statistic is, therefore, a natural extension of the classical trace test to GMANOVA models. We show that the distribution of the test under the null hypothesis does not depend on the unknown covariance matrix Σ. We also show that the distributions under the null and alternative hypotheses can be represented as sums of weighted central and non-central chi-square random variables, respectively. Under the null hypothesis, the Satterthwaite approximation is used to get an approximate critical point. A novel Satterthwaite type approximation is proposed to obtain an approximate power. A simulation study is performed to evaluate the performance of our proposed test and numerical examples are provided as illustrations.     

Bootstrapping Likelihood for Model Selection with Small Samples

Abstract

Akaike's information criterion (AIC), derived from asymptotics of the maximum likelihood estimator, is widely used in model selection. However, it has a finite-sample bias that produces overfitting in linear regression. To deal with this problem, Ishiguro, Sakamoto, and Kitagawa proposed a bootstrap-based extension to AIC which they called EIC. This article compares model-selection performance of AIC, EIC, a bootstrap-smoothed likelihood cross-validation (BCV) and its modification (632CV) in small-sample linear regression, logistic regression, and Cox regression. Simulation results show that EIC largely overcomes AIC's overfitting problem and that BCV may be better than EIC. Hence, the three methods based on bootstrapping the likelihood establish themselves as important alternatives to AIC in model selection with small samples.     

Improving on the mle of a bounded location parameter for spherical distributions

For the problem of estimating under squared error loss the location parameter of a p-variate spherically symmetric distribution where the location parameter lies in a ball of radius m, a general sufficient condition for an estimator to dominate the maximum likelihood estimator is obtained. Dominance results are then made explicit for the case of a multivariate student distribution with d degrees of freedom and, in particular, we show that the Bayes estimator with respect to a uniform prior on the boundary of the parameter space dominates the maximum likelihood estimator whenever and d?p. The sufficient condition matches the one obtained by Marchand and Perron (Ann. Statist. 29 (2001) 1078) in the normal case with identity covariance matrix. Furthermore, we derive an explicit class of estimators which, for , dominate the maximum likelihood estimator simultaneously for the normal distribution with identity covariance matrix and for all multivariate student distributions with d degrees of freedom, d?p. Finally, we obtain estimators which dominate the maximum likelihood estimator simultaneously for all distributions in the subclass of scale mixtures of normals for which the scaling random variable is bounded below by some positive constant with probability one.     

The complexity of a minimum reload cost diameter problem

We consider the minimum diameter spanning tree problem under the reload cost model which has been introduced by Wirth and Steffan [H.-C. Wirth, J. Steffan, Reload cost problems: Minimum diameter spanning tree, Discrete Appl. Math. 113 (2001) 73-85]. In this model an undirected edge-coloured graph G is given, together with a nonnegative symmetrical integer matrix R specifying the costs of changing from a colour to another one. The reload cost of a path in G arises at its internal nodes, when passing from the colour of one incident edge to the colour of the other. We prove that, unless P=NP, the problem of finding a spanning tree of G having a minimum diameter with respect to reload costs, when restricted to graphs with maximum degree 4, cannot be approximated within any constant α<2 if the reload costs are unrestricted, and cannot be approximated within any constant β<5/3 if the reload costs satisfy the triangle inequality. This solves a problem left open by Wirth and Steffan [H.-C. Wirth, J. Steffan, Reload cost problems: minimum diameter spanning tree, Discrete Appl. Math. 113 (2001) 73-85].     

Some applications of optimization in matrix theory

We apply a recent characterization of optimality for the abstract convex program with a cone constraint to three matrix theory problems: (1) a generalization of Farkas's lemma; (2) paired duality in linear programming over cones; (3) a constrained matrix best approximation problem. In particular, these results are not restricted to polyhedral or closed cones.     

Estimating a model through the conditional MLE

The estimation problem of a model through the conditional maximum likelihood estimator (MLE) is explored. The estimated model is compared using the two dual Kullback-Leibler losses with that through the unconditional MLE. The former is found to be superior to the latter under familiar models. This result is applicable to the model selection problem. These suggest a novel extensive use of the conditional likelihood, since the traditional use of the conditional likelihood was restricted only on inference for the structural parameter.     

On linear forbidden submatrices

In this paper we study the extremal problem of finding how many 1 entries an n by n 0-1 matrix can have if it does not contain certain forbidden patterns as submatrices. We call the number of 1 entries of a 0-1 matrix its weight. The extremal function of a pattern is the maximum weight of an n by n 0-1 matrix that does not contain this pattern as a submatrix. We call a pattern (a 0-1 matrix) linear if its extremal function is O(n). Our main results are modest steps towards the elusive goal of characterizing linear patterns. We find novel ways to generate new linear patterns from known ones and use this to prove the linearity of some patterns. We also find the first minimal non-linear pattern of weight above 4. We also propose an infinite sequence of patterns that we conjecture to be minimal non-linear but have Ω(nlogn) as their extremal function. We prove a weaker statement only, namely that there are infinitely many minimal not quasi-linear patterns among the submatrices of these matrices. For the definition of these terms see below.     

On some methods for entropy maximization and matrix scaling

We describe and survey in this paper iterative algorithms for solving the discrete maximum entropy problem with linear equality constraints. This problem has applications e.g. in image reconstruction from projections, transportation planning, and matrix scaling. In particular we study local convergence and asymptotic rate of convergence as a function of the iteration parameter. For the trip distribution problem in transportation planning and the equivalent problem of scaling a positive matrix to achieve a priori given row and column sums, it is shown how the iteration parameters can be chosen in an optimal way. We also consider the related problem of finding a matrix X, diagonally similar to a given matrix, such that corresponding row and column norms in X are all equal. Reports of some numerical tests are given.