Interaction Model and Model Selection for Function-on-Function Regression

Regression models with interaction effects have been widely used in multivariate analysis to improve model flexibility and prediction accuracy. In functional data analysis, however, due to the challenges of estimating three-dimensional coefficient functions, interaction effects have not been considered for function-on-function linear regression. In this article, we propose function-on-function regression models with interaction and quadratic effects. For a model with specified main and interaction effects, we propose an efficient estimation method that enjoys a minimum prediction error property and has good predictive performance in practice. Moreover, converting the estimation of three-dimensional coefficient functions of the interaction effects to the estimation of two- and one-dimensional functions separately, our method is computationally efficient. We also propose adaptive penalties to account for varying magnitudes and roughness levels of coefficient functions. In practice, the forms of the models are usually unspecified. We propose a stepwise procedure for model selection based on a predictive criterion. This method is implemented in our R package FRegSigComp. Supplemental materials are available online.     

Regularities in data from factorial experiments

This article documents a meta‐analysis of 113 data sets from published factorial experiments. The study quantifies regularities observed among factor effects and multifactor interactions. Such regularities are known to be critical to efficient planning and analysis of experiments and to robust design of engineering systems. Three previously observed properties are analyzed: effect sparsity, hierarchy, and heredity. A new regularity is introduced and shown to be statistically significant. It is shown that a preponderance of active two‐factor interaction effects are synergistic, meaning that when main effects are used to increase the system response, the interaction provides an additional increase and that when main effects are used to decrease the response, the interactions generally counteract the main effects. © 2006 Wiley Periodicals, Inc. Complexity 11: 32–45, 2006     

Integrative interaction analysis using threshold gradient directed regularization

For many complex business and industry problems, high‐dimensional data collection and modeling have been conducted. It has been shown that interactions may have important implications beyond the main effects. The number of unknown parameters in an interaction analysis can be larger or much larger than the sample size. As such, results generated from analyzing a single data set are often unsatisfactory. Integrative analysis, which jointly analyzes the raw data from multiple independent studies, has been conducted in a series of recent studies and shown to outperform single–data set analysis, meta‐analysis, and other multi–data set analyses. In this study, our goal is to conduct integrative analysis in interaction analysis. For regularized estimation and selection of important interactions (and main effects), we apply a threshold gradient directed regularization approach. Advancing from the existing studies, the threshold gradient directed regularization approach is modified to respect the “main effects, interactions” hierarchy. The proposed approach has an intuitive formulation and is computationally simple and broadly applicable. Simulations and the analyses of financial early warning system data and news‐APP (application) recommendation behavior data demonstrate its satisfactory practical performance.     

Learning Interactions via Hierarchical Group-Lasso Regularization

We introduce a method for learning pairwise interactions in a linear regression or logistic regression model in a manner that satisfies strong hierarchy: whenever an interaction is estimated to be nonzero, both its associated main effects are also included in the model. We motivate our approach by modeling pairwise interactions for categorical variables with arbitrary numbers of levels, and then show how we can accommodate continuous variables as well. Our approach allows us to dispense with explicitly applying constraints on the main effects and interactions for identifiability, which results in interpretable interaction models. We compare our method with existing approaches on both simulated and real data, including a genome-wide association study, all using our R package glinternet.     

Modeling quasi-static crack growth with the embedded finite element method on multiple levels

The current work presents the multilevel approach of the embedded finite element method which is obtained by combining features of the method of domain decomposition with those of the standard embedded finite element method. The conventional requirement of fine mesh in a possible failure zone is rendered unnecessary with the new approach thereby reducing the computational expense. In addition, it is also possible to stop a propagating crack-tip in the middle of a finite element. In this approach, the finite elements at the failure-prone zone where cracks or shear bands, referred to as strong discontinuities which represent jumps in the displacement field, can form and propagate based on some failure criterion are treated as separate sub-boundary value problems which are adaptively discretized during the run time into a number of sub-elements and subjected to a kinematic constraint on their boundary. Each sub-element becomes equally capable of developing a strong discontinuity depending upon its state of stress. A linear displacement based constraint is applied initially which is modified accordingly as soon as a strong discontinuity propagates through the boundary of the main finite element. At the local equilibrium, the coupling between the quantities at two different levels of discretization is obtained by matching the virtual energies due to admissible variations of the main finite element and its constituent sub-elements. (© 2012 Wiley-VCH Verlag GmbH & Co. KGaA, Weinheim)     

半参数线性混合效应模型的联合变量选择

多数基于线性混合效应模型的变量选择方法分阶段对固定效应和随机效应进行选择,方法繁琐、易产生模型偏差,且大部分非参数和半参数的线性混合效应模型只涉及非参数部分的光滑度或者固定效应的选择,并未涉及非参变量或随机效应的选择。本文用B样条函数逼近非参数函数部分,从而把半参数线性混合效应模型转化为带逼近误差的线性混合效应模型。对随机效应的协方差矩阵采用改进的乔里斯基分解并重新参数化线性混合效应模型,接着对该模型的极大似然函数施加集群ALASSO惩罚和ALASSO惩罚两类惩罚,该法能实现非参数变量、固定效应和随机效应的联合变量选择,基于该法得出的估计量也满足相合性、稀疏性和Oracle性质。文章最后做了个数值模拟,模拟结果表明,本文提出的估计方法在变量选择的准确性、参数估计的精度两个方面均表现较好。     

Bayesian Variable Selection on Model Spaces Constrained by Heredity Conditions

This article investigates Bayesian variable selection when there is a hierarchical dependence structure on the inclusion of predictors in the model. In particular, we study the type of dependence found in polynomial response surfaces of orders two and higher, whose model spaces are required to satisfy weak or strong heredity conditions. These conditions restrict the inclusion of higher-order terms depending upon the inclusion of lower-order parent terms. We develop classes of priors on the model space, investigate their theoretical and finite sample properties, and provide a Metropolis–Hastings algorithm for searching the space of models. The tools proposed allow fast and thorough exploration of model spaces that account for hierarchical polynomial structure in the predictors and provide control of the inclusion of false positives in high posterior probability models.     

General primal-dual penalty/barrier path-following Newton methods for nonlinear programming

In the present article rather general penalty/barrier-methods are considered, that define a local continuously differentiable primal-dual path. The class of penalty/barrier terms includes most of the usual techniques like logarithmic barriers, SUMT, quadratic loss functions as well as exponential penalties, and the optimization problem which may contain inequality as well as equality constraints. The convergence of the corresponding general primal-dual path-following method is shown for local minima that satisfy strong second-order sufficiency conditions with linear independence constraint qualification (LICQ) and strict complementarity. A basic tool in the analysis of these methods is to estimate the radius of convergence of Newton's method depending on the penalty/barrier-parameter. Without using self-concordance properties convergence bounds are derived by direct estimations of the solutions of the Newton equations. Parameter selection rules are proposed which guarantee the local convergence of the considered penalty/barrier-techniques with only a finite number of Newton steps at each parameter level. Numerical examples illustrate the practical behavior of the proposed class of methods.     

Minimal Penalties for Gaussian Model Selection

This paper is mainly devoted to a precise analysis of what kind of penalties should be used in order to perform model selection via the minimization of a penalized least-squares type criterion within some general Gaussian framework including the classical ones. As compared to our previous paper on this topic (Birgé and Massart in J. Eur. Math. Soc. 3, 203–268 (2001)), more elaborate forms of the penalties are given which are shown to be, in some sense, optimal. We indeed provide more precise upper bounds for the risk of the penalized estimators and lower bounds for the penalty terms, showing that the use of smaller penalties may lead to disastrous results. These lower bounds may also be used to design a practical strategy that allows to estimate the penalty from the data when the amount of noise is unknown. We provide an illustration of the method for the problem of estimating a piecewise constant signal in Gaussian noise when neither the number, nor the location of the change points are known.     

A uniform framework for the combination of penalties in generalized structured models

Penalized estimation has become an established tool for regularization and model selection in regression models. A variety of penalties with specific features are available and effective algorithms for specific penalties have been proposed. But not much is available to fit models with a combination of different penalties. When modeling the rent data of Munich as in our application, various types of predictors call for a combination of a Ridge, a group Lasso and a Lasso-type penalty within one model. We propose to approximate penalties that are (semi-)norms of scalar linear transformations of the coefficient vector in generalized structured models—such that penalties of various kinds can be combined in one model. The approach is very general such that the Lasso, the fused Lasso, the Ridge, the smoothly clipped absolute deviation penalty, the elastic net and many more penalties are embedded. The computation is based on conventional penalized iteratively re-weighted least squares algorithms and hence, easy to implement. New penalties can be incorporated quickly. The approach is extended to penalties with vector based arguments. There are several possibilities to choose the penalty parameter(s). A software implementation is available. Some illustrative examples show promising results.     

Development of a new approach for deterministic supply chain network design

This paper proposes a mixed integer linear programming model and solution algorithm for solving supply chain network design problems in deterministic, multi-commodity, single-period contexts. The strategic level of supply chain planning and tactical level planning of supply chain are aggregated to propose an integrated model. The model integrates location and capacity choices for suppliers, plants and warehouses selection, product range assignment and production flows. The open-or-close decisions for the facilities are binary decision variables and the production and transportation flow decisions are continuous decision variables. Consequently, this problem is a binary mixed integer linear programming problem. In this paper, a modified version of Benders’ decomposition is proposed to solve the model. The most difficulty associated with the Benders’ decomposition is the solution of master problem, as in many real-life problems the model will be NP-hard and very time consuming. In the proposed procedure, the master problem will be developed using the surrogate constraints. We show that the main constraints of the master problem can be replaced by the strongest surrogate constraint. The generated problem with the strongest surrogate constraint is a valid relaxation of the main problem. Furthermore, a near-optimal initial solution is generated for a reduction in the number of iterations.     

Unified Field Theory and Principle of Representation Invariance

The main objectives of this article are to postulate a new principle of representation invariance (PRI), and to refine the unified field model of four interactions, derived using the principle of interaction dynamics (PID). Intuitively, PID takes the variation of the action functional under energy-momentum conservation constraint, and PRI requires that physical laws be independent of representations of the gauge groups. One important outcome of this unified field model is a natural duality between the interacting fields (g,A,W ^a ,S ^k ), corresponding to graviton, photon, intermediate vector bosons W ^± and Z and gluons, and the adjoint bosonic fields $(\varPhi_{\mu}, \phi^{0}, \phi^{a}_{w}, \phi^{k}_{s})$ . This duality predicts two Higgs particles of similar mass with one due to weak interaction and the other due to strong interaction. The unified field model can be naturally decoupled to study individual interactions, leading to (1) modified Einstein equations, giving rise to a unified theory for dark matter and dark energy (Ma and Wang in Discrete Contin. Dyn. Syst., Ser A. 34(2):335–366, 2014), (2) three levels of strong interaction potentials for quark, nucleon/hadron, and atom respectively (Ma and Wang in Duality theory of strong interaction, 2012), and (3) two weak interaction potentials (Ma and Wang in Duality theory of weak interaction, 2012). These potential/force formulas offer a clear mechanism for both quark confinement and asymptotic freedom—a longstanding problem in particle physics (Ma and Wang in Duality theory of strong interaction, 2012).     

Bayesian variable selection with sparse and correlation priors for high-dimensional data analysis

The main challenge in working with gene expression microarrays is that the sample size is small compared to the large number of variables (genes). In many studies, the main focus is on finding a small subset of the genes, which are the most important ones for differentiating between different types of cancer, for simpler and cheaper diagnostic arrays. In this paper, a sparse Bayesian variable selection method in probit model is proposed for gene selection and classification. We assign a sparse prior for regression parameters and perform variable selection by indexing the covariates of the model with a binary vector. The correlation prior for the binary vector assigned in this paper is able to distinguish models with the same size. The performance of the proposed method is demonstrated with one simulated data and two well known real data sets, and the results show that our method is comparable with other existing methods in variable selection and classification.     

失混合效应模型的分位回归及变量选择

纵向数据常常用正态混合效应模型进行分析.然而,违背正态性的假定往往会导致无效的推断.与传统的均值回归相比较,分位回归可以给出响应变量条件分布的完整刻画,对于非正态误差分布也可以给稳健的估计结果．本文主要考虑右删失响应下纵向混合效应模型的分位回归估计和变量选择问题．首先,逆删失概率加权方法被用来得到模型的参数估计．其次,结合逆删失概率加权和LASSO惩罚变量选择方法考虑了模型的变量选择问题.蒙特卡洛模拟显示所提方法要比直接删除删失数据的估计方法更具优势．最后,分析了一组艾滋病数据集来展示所提方法的实际应用效果．     

Objective Bayesian model selection approach to the two way analysis of variance

An objective Bayesian procedure for testing in the two way analysis of variance is proposed. In the classical methodology the main effects of the two factors and the interaction effect are formulated as linear contrasts between means of normal populations, and hypotheses of the existence of such effects are tested. In this paper, for the first time these hypotheses have been formulated as objective Bayesian model selection problems. Our development is under homoscedasticity and heteroscedasticity, providing exact solutions in both cases. Bayes factors are the key tool to choose between the models under comparison but for the usual default prior distributions they are not well defined. To avoid this difficulty Bayes factors for intrinsic priors are proposed and they are applied in this setting to test the existence of the main effects and the interaction effect. The method has been illustrated with an example and compared with the classical method. For this example, both approaches went in the same direction although the large P value for interaction (0.79) only prevents us against to reject the null, and the posterior probability of the null (0.95) was conclusive.     

中等稀疏条件下基因交互作用的两步Bayes方法

植物遗传与基因组学研究表明许多重要的农艺性状有影响的基因位点不是稀疏的,受到大量微效基因的影响,并且还存在基因交互项的影响.本文基于重要油料作物油菜的花期数据,研究中等稀疏条件下的基因选择问题,提出了一种两步Bayes模型选择方法.考虑基因间的交互作用,模型的维数急剧增长,加上数据结构特别,通常的变量选择方法效果不好....     

Variable selection via generalized SELO-penalized linear regression models

The seamless-L_0(SELO) penalty is a smooth function on [0, ∞) that very closely resembles the L_0 penalty, which has been demonstrated theoretically and practically to be effective in nonconvex penalization for variable selection. In this paper, we first generalize SELO to a class of penalties retaining good features of SELO, and then propose variable selection and estimation in linear models using the proposed generalized SELO(GSELO) penalized least squares(PLS) approach. We show that the GSELO-PLS procedure possesses the oracle property and consistently selects the true model under some regularity conditions in the presence of a diverging number of variables. The entire path of GSELO-PLS estimates can be efficiently computed through a smoothing quasi-Newton(SQN) method. A modified BIC coupled with a continuation strategy is developed to select the optimal tuning parameter. Simulation studies and analysis of a clinical data are carried out to evaluate the finite sample performance of the proposed method. In addition, numerical experiments involving simulation studies and analysis of a microarray data are also conducted for GSELO-PLS in the high-dimensional settings.     

A Stochastic Capital Rationing Model

This paper addresses the issue of optimal project selection for capital expenditures assuming uncertain budgetary allocations. A critical review of the historical development of capital budgeting models indicates two major deficiencies: (i) deterministic models ignore the uncertain nature of capital budgeting problems; and (ii) those models which do incorporate the concept of uncertainty have serious computational problems when applied to larger problems. A stochastic capital rationing model (SCRM) is proposed which makes use of recent developments in stochastic programmes with recourse. This model remains computationally tractable despite the explicit incorporation of uncertainty and the application of theoretically sound penalties for constraint violations. Two problems are introduced which illustrate the model's superiority over comparable deterministic formulations. By varying both the probability distributions of the stochastic constraints and the borrowing rates, it was possible to identify the impact these factors have on project selections.     

Global convergence of a robust filter SQP algorithm

We present a robust filter SQP algorithm for solving constrained optimization problems. This algorithm is based on the modified quadratic programming proposed by Burke to avoid the infeasibility of the quadratic programming subproblem at each iteration. Compared with other filter SQP algorithms, our algorithm does not require any restoration phase procedure which may spend a large amount of computation. The main advantage of our algorithm is that it is globally convergent without requiring strong constraint qualifications, such as Mangasarian–Fromovitz constraint qualification (MFCQ) and the constant rank constraint qualification (CRCQ). Furthermore, the feasible limit points of the sequence generated by our algorithm are proven to be the KKT points if some weaker conditions are satisfied. Numerical results are also presented to show the efficiency of the algorithm.     

Global and local convergence of a nonmonotone SQP method for constrained nonlinear optimization

In this paper, we propose a robust sequential quadratic programming (SQP) method for nonlinear programming without using any explicit penalty function and filter. The method embeds the modified QP subproblem proposed by Burke and Han (Math Program 43:277–303, 1989) for the search direction, which overcomes the common difficulty in the traditional SQP methods, namely the inconsistency of the quadratic programming subproblems. A non-monotonic technique is employed further in a framework in which the trial point is accepted whenever there is a sufficient relaxed reduction of the objective function or the constraint violation function. A forcing sequence possibly tending to zero is introduced to control the constraint violation dynamically, which is able to prevent the constraint violation from over-relaxing and plays a crucial role in global convergence and the local fast convergence as well. We prove that the method converges globally without the Mangasarian–Fromovitz constraint qualification (MFCQ). In particular, we show that any feasible limit point that satisfies the relaxed constant positive linear dependence constraint qualification is also a Karush–Kuhn–Tucker point. Under the strict MFCQ and the second order sufficient condition, furthermore, we establish the superlinear convergence. Preliminary numerical results show the efficiency of our method.