Difficulties in Estimating the Normalizing Constant of the Posterior for a Neural Network

This article reviews a number of methods for estimating normalizing constants in the context of neural network regression. Model selection or model averaging within the Bayesian approach requires computation of the normalizing constant of the posterior. This integral can be challenging to estimate, particularly for a neural network where the posterior contours are neither unimodal nor Gaussian-shaped. Surprisingly, all of the methods discussed in this article have a large amount of difficulty with this problem.     

On regression model selection for the data with correlated errors

A class of regression model selection criteria for the data with correlated errors is proposed. The proposed class of selection criteria is an estimator of weighted prediction risk. In addition, the proposed selection criteria are the generalizations of several commonly used criteria in statistical analysis. The theoretical and asymptotic properties for the class of criteria are established. Further, in the medium-sample case, the results based on a simulation study are quite consistent with the theoretical ones. The proposed criteria perform well in the simulations. Several applications are also given for a variety of statistical models.     

Improved Model Selection Method for a Regression Function with Dependent Noise

This paper is devoted to nonparametric estimation, through the -risk, of a regression function based on observations with spherically symmetric errors, which are dependent random variables (except in the normal case). We apply a model selection approach using improved estimates. In a nonasymptotic setting, an upper bound for the risk is obtained (oracle inequality). Moreover asymptotic properties are given, such as upper and lower bounds for the risk, which provide optimal rate of convergence for penalized estimators.     

Penalized Estimation of Free-Knot Splines

Abstract

Polynomial splines are often used in statistical regression models for smooth response functions. When the number and location of the knots are optimized, the approximating power of the spline is improved and the model is nonparametric with locally determined smoothness. However, finding the optimal knot locations is an historically difficult problem. We present a new estimation approach that improves computational properties by penalizing coalescing knots. The resulting estimator is easier to compute than the unpenalized estimates of knot positions, eliminates unnecessary “corners” in the fitted curve, and in simulation studies, shows no increase in the loss. A number of GCV and AIC type criteria for choosing the number of knots are evaluated via simulation.     

A Lattice-Based Smoother for Regions With Irregular Boundaries and Holes

We consider the problem of estimating a smooth function over a spatial region that is delineated by an irregular boundary and potentially contains holes within the boundary. Methods commonly used for spatial function estimation are well-known to suffer from bias along such boundaries. The estimator we propose is a kernel regression estimator, where the kernel is an approximation to a two-dimensional diffusion process contained within the region of interest. The diffusion process is approximated by the distribution of length-k random walks originating from each observation location and constrained to stay within the domain boundaries. We propose using a cross-validation criterion to find the optimal walk length k, which controls the smoothness of the resulting estimate. Simulations show that the method outperforms the soap film smoother of Wood, Bravington, and Hedley in many realistic scenarios, when data are noisy and borders are highly irregular. We illustrate the practical use of the estimator using measurements of soil manganese concentration around Port Moller, Alaska. Supplementary materials for this article are available online.     

Using Spatial Statistics to Select Model Complexity

Testing for nonindependence among the residuals from a regression or time series model is a common approach to evaluating the adequacy of a fitted model. This idea underlies the familiar Durbin–Watson statistic, and previous works illustrate how the spatial autocorrelation among residuals can be used to test a candidate linear model. We propose here that a version of Moran's I statistic for spatial autocorrelation, applied to residuals from a fitted model, is a practical general tool for selecting model complexity under the assumption of iid additive errors. The “space” is defined by the independent variables, and the presence of significant spatial autocorrelation in residuals is evidence that a more complex model is needed to capture all of the structure in the data. An advantage of this approach is its generality, which results from the fact that no properties of the fitted model are used other than consistency. The problem of smoothing parameter selection in nonparametric regression is used to illustrate the performance of model selection based on residual spatial autocorrelation (RSA). In simulation trials comparing RSA with established selection criteria based on minimizing mean square prediction error, smooths selected by RSA exhibit fewer spurious features such as minima and maxima. In some cases, at higher noise levels, RSA smooths achieved a lower average mean square error than smooths selected by GCV. We also briefly describe a possible modification of the method for non-iid errors having short-range correlations, for example, time-series errors or spatial data. Some other potential applications are suggested, including variable selection in regression models.     

A Sparse Structured Shrinkage Estimator for Nonparametric Varying-Coefficient Model With an Application in Genomics

Many problems in genomics are related to variable selection where high-dimensional genomic data are treated as covariates. Such genomic covariates often have certain structures and can be represented as vertices of an undirected graph. Biological processes also vary as functions depending upon some biological state, such as time. High-dimensional variable selection where covariates are graph-structured and underlying model is nonparametric presents an important but largely unaddressed statistical challenge. Motivated by the problem of regression-based motif discovery, we consider the problem of variable selection for high-dimensional nonparametric varying-coefficient models and introduce a sparse structured shrinkage (SSS) estimator based on basis function expansions and a novel smoothed penalty function. We present an efficient algorithm for computing the SSS estimator. Results on model selection consistency and estimation bounds are derived. Moreover, finite-sample performances are studied via simulations, and the effects of high-dimensionality and structural information of the covariates are especially highlighted. We apply our method to motif finding problem using a yeast cell-cycle gene expression dataset and word counts in genes’ promoter sequences. Our results demonstrate that the proposed method can result in better variable selection and prediction for high-dimensional regression when the underlying model is nonparametric and covariates are structured. Supplemental materials for the article are available online.     

Image Denoising by a Local Clustering Framework

Images often contain noise due to imperfections in various image acquisition techniques. Noise should be removed from images so that the details of image objects (e.g., blood vessels, inner foldings, or tumors in the human brain) can be clearly seen, and the subsequent image analyses are reliable. With broad usage of images in many disciplines—for example, medical science—image denoising has become an important research area. In the literature, there are many different types of image denoising techniques, most of which aim to preserve image features, such as edges and edge structures, by estimating them explicitly or implicitly. Techniques based on explicit edge detection usually require certain assumptions on the smoothness of the image intensity surface and the edge curves which are often invalid especially when the image resolution is low. Methods that are based on implicit edge detection often use multiresolution smoothing, weighted local smoothing, and so forth. For such methods, the task of determining the correct image resolution or choosing a reasonable weight function is challenging. If the edge structure of an image is complicated or the image has many details, then these methods would blur such details. This article presents a novel image denoising framework based on local clustering of image intensities and adaptive smoothing. The new denoising method can preserve complicated edge structures well even if the image resolution is low. Theoretical properties and numerical studies show that it works well in various applications.     

Non-parametric adaptive estimation of the drift for a jump diffusion process

In this article, we consider a jump diffusion process _(Xt)t≥0${\left(X_t\right)}_{t\ge 0}$ observed at discrete times t=0,Δ,…,nΔ$t=0,\Delta ,\dots ,n\Delta$. The sampling interval Δ$\Delta$ tends to 0 and nΔ$n\Delta$ tends to infinity. We assume that _(Xt)t≥0${\left(X_t\right)}_{t\ge 0}$ is ergodic, strictly stationary and exponentially β$\beta$-mixing. We use a penalised least-square approach to compute two adaptive estimators of the drift function b$b$. We provide bounds for the risks of the two estimators.     

Variable Selection in Varying-Coefficient Models Using P-Splines

In this article, we consider nonparametric smoothing and variable selection in varying-coefficient models. Varying-coefficient models are commonly used for analyzing the time-dependent effects of covariates on responses measured repeatedly (such as longitudinal data). We present the P-spline estimator in this context and show its estimation consistency for a diverging number of knots (or B-spline basis functions). The combination of P-splines with nonnegative garrote (which is a variable selection method) leads to good estimation and variable selection. Moreover, we consider APSO (additive P-spline selection operator), which combines a P-spline penalty with a regularization penalty, and show its estimation and variable selection consistency. The methods are illustrated with a simulation study and real-data examples. The proofs of the theoretical results as well as one of the real-data examples are provided in the online supplementary materials.     

Plug-in method for nonparametric regression

The problem of bandwidth selection for non-parametric kernel regression is considered. We will follow the Nadaraya–Watson and local linear estimator especially. The circular design is assumed in this work to avoid the difficulties caused by boundary effects. Most of bandwidth selectors are based on the residual sum of squares (RSS). It is often observed in simulation studies that these selectors are biased toward undersmoothing. This leads to consideration of a procedure which stabilizes the RSS by modifying the periodogram of the observations. As a result of this procedure, we obtain an estimation of unknown parameters of average mean square error function (AMSE). This process is known as a plug-in method. Simulation studies suggest that the plug-in method could have preferable properties to the classical one. Supported by the MSMT: LC 06024.     

Data Adaptive Ridging in Local Polynomial Regression

Abstract

When estimating a regression function or its derivatives, local polynomials are an attractive choice due to their flexibility and asymptotic performance. Seifert and Gasser proposed ridging of local polynomials to overcome problems with variance for random design while retaining their advantages. In this article we present a data-independent rule of thumb and a data-adaptive spatial choice of the ridge parameter in local linear regression. In a framework of penalized local least squares regression, the methods are generalized to higher order polynomials, to estimation of derivatives, and to multivariate designs. The main message is that ridging is a powerful tool for improving the performance of local polynomials. A rule of thumb offers drastic improvements; data-adaptive ridging brings further but modest gains in mean square error.     

Estimating and Visualizing Conditional Densities

Abstract

We consider the kernel estimator of conditional density and derive its asymptotic bias, variance, and mean-square error. Optimal bandwidths (with respect to integrated mean-square error) are found and it is shown that the convergence rate of the density estimator is order n ^–2/3. We also note that the conditional mean function obtained from the estimator is equivalent to a kernel smoother. Given the undesirable bias properties of kernel smoothers, we seek a modified conditional density estimator that has mean equivalent to some other nonparametric regression smoother with better bias properties. It is also shown that our modified estimator has smaller mean square error than the standard estimator in some commonly occurring situations. Finally, three graphical methods for visualizing conditional density estimators are discussed and applied to a data set consisting of maximum daily temperatures in Melbourne, Australia.     

An Algorithm for Nonlinear,Nonparametric Model Choice and Prediction

We introduce an algorithm which, in the context of nonlinear regression on vector-valued explanatory variables, aims to choose those combinations of vector components that provide best prediction. The algorithm is constructed specifically so that it devotes attention to components that might be of relatively little predictive value by themselves, and so might be ignored by more conventional methodology for model choice, but which, in combination with other difficult-to-find components, can be particularly beneficial for prediction. The design of the algorithm is also motivated by a desire to choose vector components that become redundant once appropriate combinations of other, more relevant components are selected. Our theoretical arguments show these goals are met in the sense that, with probability converging to 1 as sample size increases, the algorithm correctly determines a small, fixed number of variables on which the regression mean, g say, depends, even if dimension diverges to infinity much faster than n. Moreover, the estimated regression mean based on those variables approximates g with an error that, to first order, equals the error which would arise if we were told in advance the correct variables. In this sense, the estimator achieves oracle performance. Our numerical work indicates that the algorithm is suitable for very high dimensional problems, where it keeps computational labor in check by using a novel sequential argument, and also for more conventional prediction problems, where dimension is relatively low.     

On extracting information implied in options

Options are financial instruments with a payoff depending on future states of the underlying asset. Therefore option markets contain information about expectations of the market participants about market conditions, e.g. current uncertainty on the market and corresponding risk. A standard measure of risk calculated from plain vanilla options is the implied volatility (IV). IV can be understood as an estimate of the volatility of returns in future period. Another concept based on the option markets is the state-price density (SPD) that is a density of the future states of the underlying asset. From raw data we can recover the IV function by nonparametric smoothing methods. Smoothed IV estimated by standard techniques may lead to a non-positive SPD which violates no arbitrage criteria. In this paper, we combine the IV smoothing with SPD estimation in order to correct these problems. We propose to use the local polynomial smoothing technique. The elegance of this approach is that it yields all quantities needed to calculate the corresponding SPD. Our approach operates only on the IVs—a major improvement comparing to the earlier multi-step approaches moving through the Black–Scholes formula from the prices to IVs and vice-versa.     

The Fast RODEO for Local Polynomial Regression

An open challenge in nonparametric regression is finding fast, computationally efficient approaches to estimating local bandwidths for large datasets, in particular in two or more dimensions. In the work presented here, we introduce a novel local bandwidth estimation procedure for local polynomial regression, which combines the greedy search of the regularization of the derivative expectation operator (RODEO) algorithm with linear binning. The result is a fast, computationally efficient algorithm, which we refer to as the fast RODEO. We motivate the development of our algorithm by using a novel scale-space approach to derive the RODEO. We conclude with a toy example and a real-world example using data from the Cloud-Aerosol Lidar and Infrared Pathfinder Satellite Observation (CALIPSO) satellite validation study, where we show the fast RODEO’s improvement in accuracy and computational speed over two other standard approaches.     

A Fast and Stable Updating Algorithm for Bivariate Nonparametric Curve Estimation

Abstract

An updating algorithm for bivariate local linear regression is proposed. Thereby, we assume a rectangular design and a polynomial kernel constrained to rectangular support as weight function. Results of univariate regression estimators are extended to the bivariate setting. The updates are performed in a way that most of the well-known numerical instabilities of a naive update implementation can be avoided. Some simulation results illustrate the properties of several algorithms with respect to computing time and numerical stability.     

A first–passage time random walk distribution with five transition probabilities: a generalization of the shifted inverse trinomial

In this paper a univariate discrete distribution, denoted by GIT, is proposed as a generalization of the shifted inverse trinomial distribution, and is formulated as a first-passage time distribution of a modified random walk on the half-plane with five transition probabilities. In contrast, the inverse trinomial arises as a random walk on the real line with three transition probabilities. The probability mass function (pmf) is expressible in terms of the Gauss hypergeometric function and this offers computational advantage due to its recurrence formula. The descending factorial moment is also obtained. The GIT contains twenty-two possible distributions in total. Special cases include the binomial, negative binomial, shifted negative binomial, shifted inverse binomial or, equivalently, lost-games, and shifted inverse trinomial distributions. A subclass GIT_3,1 is a particular member of Kemp’s class of convolution of pseudo-binomial variables and its properties such as reproductivity, formulation, pmf, moments, index of dispersion, and approximations are studied in detail. Compound or generalized (stopped sum) distributions provide inflated models. The inflated GIT_3,1 extends Minkova’s inflated-parameter binomial and negative binomial. A bivariate model which has the GIT as a marginal distribution is also proposed.     

The Application of Kernel Smoothing to Time Series Data

There are already a lot of models to fit a set of stationary time series, such as AR, MA, and ARMA models. For the non-stationary data, an ARIMA or seasonal ARIMA models can be used to fit the given data. Moreover, there are also many statistical softwares that can be used to build a stationary or non-stationary time series model for a given set of time series data, such as SAS, SPLUS, etc. However, some statistical softwares wouldn＇t work well for small samples with or without missing data, especially for small time series data with seasonal trend. A nonparametric smoothing technique to build a forecasting model for a given small seasonal time series data is carried out in this paper. And then, both the method provided in this paper and that in SAS package are applied to the modeling of international airline passengers data respectively, the comparisons between the two methods are done afterwards. The results of the comparison show us the method provided in this paper has superiority over SAS＇s method.