Functional fuzzy clusterwise regression analysis

We propose a functional extension of fuzzy clusterwise regression, which estimates fuzzy memberships of clusters and regression coefficient functions for each cluster simultaneously. The proposed method permits dependent and/or predictor variables to be functional, varying over time, space, and other continua. The fuzzy memberships and clusterwise regression coefficient functions are estimated by minimizing an objective function that adopts a basis function expansion approach to approximating functional data. An alternating least squares algorithm is developed to minimize the objective function. We conduct simulation studies to demonstrate the superior performance of the proposed method compared to its non-functional counterpart and to examine the performance of various cluster validity measures for selecting the optimal number of clusters. We apply the proposed method to real datasets to illustrate the empirical usefulness of the proposed method.     

Seemingly unrelated clusterwise linear regression

Advances in Data Analysis and Classification - Linear regression models based on finite Gaussian mixtures represent a flexible tool for the analysis of linear dependencies in multivariate data....     

Globally optimal clusterwise regression by mixed logical-quadratic programming

Exact global optimization of the clusterwise regression problem is challenging and there are currently no published feasible methods for performing this clustering optimally, even though it has been over thirty years since its original proposal. This work explores global optimization of the clusterwise regression problem using mathematical programming and related issues. A mixed logical-quadratic programming formulation with implication of constraints is presented and contrasted against a quadratic formulation based on the traditional big-M, which cannot guarantee optimality because the regression line coefficients, and thus errors, may be arbitrarily large. Clusterwise regression optimization times and solution optimality for two clusters are empirically tested on twenty real datasets and three series of synthetic datasets ranging from twenty to one hundred observations and from two to ten independent variables. Additionally, a few small real datasets are clustered into three lines.     

Nonsmooth nonconvex optimization approach to clusterwise linear regression problems

Clusterwise regression consists of finding a number of regression functions each approximating a subset of the data. In this paper, a new approach for solving the clusterwise linear regression problems is proposed based on a nonsmooth nonconvex formulation. We present an algorithm for minimizing this nonsmooth nonconvex function. This algorithm incrementally divides the whole data set into groups which can be easily approximated by one linear regression function. A special procedure is introduced to generate a good starting point for solving global optimization problems at each iteration of the incremental algorithm. Such an approach allows one to find global or near global solution to the problem when the data sets are sufficiently dense. The algorithm is compared with the multistart Späth algorithm on several publicly available data sets for regression analysis.     

A mathematical programming approach to clusterwise regression model and its extensions

The clusterwise regression model is used to perform cluster analysis within a regression framework. While the traditional regression model assumes the regression coefficient (β) to be identical for all subjects in the sample, the clusterwise regression model allows β to vary with subjects of different clusters. Since the cluster membership is unknown, the estimation of the clusterwise regression is a tough combinatorial optimization problem. In this research, we propose a “Generalized Clusterwise Regression Model” which is formulated as a mathematical programming (MP) problem. A nonlinear programming procedure (with linear constraints) is proposed to solve the combinatorial problem and to estimate the cluster membership and β simultaneously. Moreover, by integrating the cluster analysis with the discriminant analysis, a clusterwise discriminant model is developed to incorporate parameter heterogeneity into the traditional discriminant analysis. The cluster membership and discriminant parameters are estimated simultaneously by another nonlinear programming model.     

Inequality constrained ridge regression estimator

We carry out the idea of inequality constrained least squares (ICLS) estimation of Liew (1976) to the inequality constrained ridge regression (ICRR) estimation. We propose ICRR estimator by reducing the primal–dual relation to the fundamental problem of Dantzig and Cottle, 1967, Cottle and Dantzig, 1974 with Lemke (1962) algorithm. Furthermore, we conduct a Monte Carlo experiment.     

Regularized least square regression with dependent samples

In this paper we study the learning performance of regularized least square regression with α-mixing and ϕ-mixing inputs. The capacity independent error bounds and learning rates are derived by means of an integral operator technique. Even for independent samples our learning rates improve those in the literature. The results are sharp in the sense that when the mixing conditions are strong enough the rates are shown to be close to or the same as those for learning with independent samples. They also reveal interesting phenomena of learning with dependent samples: (i) dependent samples contain less information and lead to worse error bounds than independent samples; (ii) the influence of the dependence between samples to the learning process decreases as the smoothness of the target function increases.     

Improved ridge regression estimators for the logistic regression model

The estimation of the regression parameters for the ill-conditioned logistic regression model is considered in this paper. We proposed five ridge regression (RR) estimators, namely, unrestricted RR, restricted ridge regression, preliminary test RR, shrinkage ridge regression and positive rule RR estimators for estimating the parameters $(\beta )$ when it is suspected that the parameter $\beta $ may belong to a linear subspace defined by $H\beta =h$ . Asymptotic properties of the estimators are studied with respect to quadratic risks. The performances of the proposed estimators are compared based on the quadratic bias and risk functions under both null and alternative hypotheses, which specify certain restrictions on the regression parameters. The conditions of superiority of the proposed estimators for departure and ridge parameters are given. Some graphical representations and efficiency analysis have been presented which support the findings of the paper.     

Linear fuzzy regression

This paper presents an approach which is useful for regression analysis in the case of heterogeneity of a set of observations, for which regression is evaluated. The proposed procedure consists of two stages. First, for a set of observations, fuzzy classification is determined. Due to this, homogenous classes of observations which are of hyperellipsoidal shape, are obtained. Then for each fuzzy class, the so called linear fuzzy regression is evaluated.

In the paper the method of calculating linear fuzzy regression coefficients is given. It is a generalized version of the least squares method. The formula for the values of coefficients is given. Some properties of linear fuzzy regression are analyzed. It is proved that in one- and two-dimensional cases, the formulae are analogous to those for usual regression. A measure of goodness-of-fit and the method of determination of the number of fuzzy classes are also given.

Presented examples indicate the superiority of fuzzy regression in comparison to usual regression in the case of heterogenous observations.     

A stochastic restricted ridge regression estimator

Groß [J. Groß, Restricted ridge estimation, Statistics & Probability Letters 65 (2003) 57–64] proposed a restricted ridge regression estimator when exact restrictions are assumed to hold. When there are stochastic linear restrictions on the parameter vector, we introduce a new estimator by combining ideas underlying the mixed and the ridge regression estimators under the assumption that the errors are not independent and identically distributed. Apart from [J. Groß, Restricted ridge estimation, Statistics & Probability Letters 65 (2003) 57–64], we call this new estimator as the stochastic restricted ridge regression (SRRR) estimator. The performance of the SRRR estimator over the mixed estimator in respect of the variance and the mean square error matrices is examined. We also illustrate our findings with a numerical example. The shrinkage generalized least squares (GLS) and the stochastic restricted shrinkage GLS estimators are proposed.     

The existence theorem in general ridge regression

Hoerl and Kennard (1970) state that, like the ordinary ridge estimator, the general ridge estimator is also better than the least squares estimator relative to a mean square error. The proof of this result is given in this note.     

Robust kernel ridge regression based on M-estimation

Ridge regression (RR) and kernel ridge regression (KRR) are important tools to avoid the effects of multicollinearity. However, the predictions of RR and KRR become inappropriate for use in regression models when data are contaminated by outliers. In this paper, we propose an algorithm to obtain a nonlinear robust prediction without specifying a nonlinear model in advance. We combine M-estimation and kernel ridge regression to obtain the nonlinear prediction. Then, we compare the proposed method with some other methods.     

Least squares/ridge regression and goal programming/constrained regression alternatives

Ridge regression has been extensively reviewed in the literature of applied statistics as a method for dealing with problems of collinearity in least squares regressions. In this paper, ordinary least squares and least squares combined with ridge regressions are examined and compared with ordinary goal programming and goal programming combined with constraints on regressand values that admissible statistical estimates must satisfy. The comparisons are effected by means of an example in chemical processing that has been extensively treated in the ridge regression literature.     

Broken adaptive ridge regression for right-censored survival data

Annals of the Institute of Statistical Mathematics - Broken adaptive ridge (BAR) is a computationally scalable surrogate to $$L_0$$ -penalized regression, which involves iteratively performing...     

Intuitionistic fuzzy linear regression analysis

Linear regression analysis in an intuitionistic fuzzy environment using intuitionistic fuzzy linear models with symmetric triangular intuitionistic fuzzy number (STriIFN) coefficients is introduced. The goal of this regression is to find the coefficients of a proposed model for all given input–output data sets. The coefficients of an intuitionistic fuzzy regression (IFR) model are found by solving a linear programming problem (LPP). The objective function of the LPP is to minimize the total fuzziness of the IFR model which is related to the width of IF coefficients. An illustrative example is also presented to depict the solution procedure of the IFR problem by using STriIFNs.     

Analysis for multiple fuzzy regression

In order to resolve some regression problems that are more difficult to resolve by the common methods, we establish the relation of multiple fuzzy regression and give its analytical methods. They are not only suited to numerical value variables but can also apply to fuzzy variables. We also give a calculating example and compare these methods with the common one.     

Evaluation of fuzzy regression models by fuzzy neural network

In this paper, a novel hybrid method based on fuzzy neural network for approximate fuzzy coefficients (parameters) of fuzzy linear and nonlinear regression models with fuzzy output and crisp inputs, is presented. Here a neural network is considered as a part of a large field called neural computing or soft computing. Moreover, in order to find the approximate parameters, a simple algorithm from the cost function of the fuzzy neural network is proposed. Finally, we illustrate our approach by some numerical examples.     

An exact approach to ridge regression for big data

Ridge regression is an important approach in linear regression when explanatory variables are highly correlated. Although expressions of estimators of ridge regression parameters have been successfully obtained via matrix operation after observed data are standardized, they cannot be used to big data since it is impossible to load the entire data set to the memory of a single computer and it is hard to standardize the original observed data. To overcome these difficulties, the present article proposes new methods and algorithms. The basic idea is to compute a matrix of sufficient statistics by rows. Once the matrix is derived, it is not necessary to use the original data again. Since the entire data set is only scanned once, the proposed methods and algorithms can be extremely efficient in the computation of estimates of ridge regression parameters. It is expected that the basic knowledge gained in this article will have a great impact on statistical approaches to big data.     

An examiniation of criticisms of ridge regression simulation methodology

Page1 (1981) claimed that there is a serious design flaw in some of the recent simulation studies of ridge estimators, in particular those in Hoerl, Kennard and Baldwin (1976) and Lawless and Wang (1976). Farebrother (1983) argued that the major criticism in Page1 (1981) is unsubstantiated. In this paper we obtain a series expansion for the mean squared error of the ordinary ridge estimator, use it to prove that Pagel's claim is incorrect and reinforce Farebrother's comments.