Relaxing support vectors for classification

We introduce a novel modification to standard support vector machine (SVM) formulations based on a limited amount of penalty-free slack to reduce the influence of misclassified samples or outliers. We show that free slack relaxes support vectors and pushes them towards their respective classes, hence we use the name relaxed support vector machines (RSVM) for our method. We present theoretical properties of the RSVM formulation and develop its dual formulation for nonlinear classification via kernels. We show the connection between the dual RSVM and the dual of the standard SVM formulations. We provide error bounds for RSVM and show it to be stable, universally consistent and tighter than error bounds for standard SVM. We also introduce a linear programming version of RSVM, which we call RSVMLP. We apply RSVM and RSVMLP to synthetic data and benchmark binary classification problems, and compare our results with standard SVM classification results. We show that relaxed influential support vectors may lead to better classification results. We develop a two-phase method called RSVM² for multiple instance classification (MIC) problems, where RSVM formulations are used as classifiers. We extend the two-phase method to the linear programming case and develop RSVMLP². We demonstrate the classification characteristics of RSVM² and RSVMLP², and report our classification results compared to results obtained by other SVM-based MIC methods on public benchmark datasets. We show that both RSVM² and RSVMLP² are faster and produce more accurate classification results.     

A deterministic sparse FFT algorithm for vectors with small support

In this paper we consider the special case where a signal x\({\in }\,\mathbb {C}^{N}\) is known to vanish outside a support interval of length m < N. If the support length m of x or a good bound of it is a-priori known we derive a sublinear deterministic algorithm to compute x from its discrete Fourier transform \(\widehat {\mathbf x}\,{\in }\,\mathbb {C}^{N}\). In case of exact Fourier measurements we require only \({\mathcal O}\)(m\(\log \)m) arithmetical operations. For noisy measurements, we propose a stable \({\mathcal O}\)(m\(\log \)N) algorithm.     

Sparse representation of vectors in lattices and semigroups

Mathematical Programming - We study the sparsity of the solutions to systems of linear Diophantine equations with and without non-negativity constraints. The sparsity of a solution vector is the...     

On stable refinable function vectors with arbitrary support

Refinable function vectors with arbitrary support are considered. In particular, necessary conditions for stability are given and a characterization of the symbol associated with a stable refinable function vector in terms of the transfer operator is provided: this is a generalization of Gundy’s theorem to the vector case. The proof adapts the tools provided in [S. Saliani, On stability and orthogonality of refinable functions, Appl. Comput. Harmon. Anal. 21 (2006) 254–261]. Though complications arise from noncommuting matrix products, the fundamental ideas are the same.     

Roundoff error analysis of fast DCT algorithms in fixed point arithmetic

Discrete cosine transforms (DCT) are essential tools in numerical analysis and digital signal processing. Processors in digital signal processing often use fixed point arithmetic. In this paper, we consider the numerical stability of fast DCT algorithms in fixed point arithmetic. The fast DCT algorithms are based on known factorizations of the corresponding cosine matrices into products of sparse, orthogonal matrices of simple structure. These algorithms are completely recursive, are easy to implement and use only permutations, scaling, butterfly operations, and plane rotations/rotation-reflections. In comparison with other fast DCT algorithms, these algorithms have low arithmetic costs. Using von Neumann–Goldstine’s model of fixed point arithmetic, we present a detailed roundoff error analysis for fast DCT algorithms in fixed point arithmetic. Numerical tests demonstrate the performance of our results.      

Quadratic mixed integer programming and support vectors for deleting outliers in robust regression

We consider the problem of deleting bad influential observations (outliers) in linear regression models. The problem is formulated as a Quadratic Mixed Integer Programming (QMIP) problem, where penalty costs for discarding outliers are used into the objective function. The optimum solution defines a robust regression estimator called penalized trimmed squares (PTS). Due to the high computational complexity of the resulting QMIP problem, the proposed robust procedure is computationally suitable for small sample data. The computational performance and the effectiveness of the new procedure are improved significantly by using the idea of ε-Insensitive loss function from support vectors machine regression. Small errors are ignored, and the mathematical formula gains the sparseness property. The good performance of the ε-Insensitive PTS (IPTS) estimator allows identification of multiple outliers avoiding masking or swamping effects. The computational effectiveness and successful outlier detection of the proposed method is demonstrated via simulated experiments. This research has been partially funded by the Greek Ministry of Education under the program Pythagoras II.     

Sparse finite elements for elliptic problems with stochastic loading

Summary. We formulate elliptic boundary value problems with stochastic loading in a bounded domain D ^d . We show well-posedness of the problem in stochastic Sobolev spaces and we derive a deterministic elliptic PDE in D×D for the spatial correlation of the random solution. We show well-posedness and regularity results for this PDE in a scale of weighted Sobolev spaces with mixed highest order derivatives. Discretization with sparse tensor products of any hierarchic finite element (FE) spaces in D yields optimal asymptotic rates of convergence for the spatial correlation even in the presence of singularities or for spatially completely uncorrelated data. Multilevel preconditioning in D×D allows iterative solution of the discrete equation for the correlation kernel in essentially the same complexity as the solution of the mean field equation. Mathematics Subject Classification (2000): 65N30Research performed under IHP network Breaking Complexity of the EC, contract number HPRN-CT-2002-00286, and supported in part by the Swiss Federal Office for Science and Education under grant number BBW 02.0418.     

Polybasic polyhedra: structure of polyhedra with edge vectors of support size at most 2

It has widely been recognized that submodular set functions and base polyhedra associated with them play fundamental and important roles in combinatorial optimization problems. In the present paper, we introduce a generalized concept of base polyhedron. We consider a class of pointed convex polyhedra in R^V whose edge vectors have supports of size at most 2. We call such a convex polyhedron a polybasic polyhedron. The class of polybasic polyhedra includes ordinary base polyhedra, submodular/supermodular polyhedra, generalized polymatroids, bisubmodular polyhedra, polybasic zonotopes, boundary polyhedra of flows in generalized networks, etc. We show that for a pointed polyhedron PR^V, the following three statements are equivalent:

(1) P is a polybasic polyhedron.

(2) Each face of P with a normal vector of the full support V is obtained from a base polyhedron by a reflection and scalings along axes.

(3) The support function of P is a submodular function on each orthant of R^V.

This reveals the geometric structure of polybasic polyhedra and its relation to submodularity.     

Sparse classification with paired covariates

Advances in Data Analysis and Classification - This paper introduces the paired lasso: a generalisation of the lasso for paired covariate settings. Our aim is to predict a single response from two...     

Iterative Thresholding for Sparse Approximations

Sparse signal expansions represent or approximate a signal using a small number of elements from a large collection of elementary waveforms. Finding the optimal sparse expansion is known to be NP hard in general and non-optimal strategies such as Matching Pursuit, Orthogonal Matching Pursuit, Basis Pursuit and Basis Pursuit De-noising are often called upon. These methods show good performance in practical situations, however, they do not operate on the ? ₀ penalised cost functions that are often at the heart of the problem. In this paper we study two iterative algorithms that are minimising the cost functions of interest. Furthermore, each iteration of these strategies has computational complexity similar to a Matching Pursuit iteration, making the methods applicable to many real world problems. However, the optimisation problem is non-convex and the strategies are only guaranteed to find local solutions, so good initialisation becomes paramount. We here study two approaches. The first approach uses the proposed algorithms to refine the solutions found with other methods, replacing the typically used conjugate gradient solver. The second strategy adapts the algorithms and we show on one example that this adaptation can be used to achieve results that lie between those obtained with Matching Pursuit and those found with Orthogonal Matching Pursuit, while retaining the computational complexity of the Matching Pursuit algorithm.     

Deformation Techniques for Sparse Systems

We exhibit a probabilistic symbolic algorithm for solving zero-dimensional sparse systems. Our algorithm combines a symbolic homotopy procedure, based on a flat deformation of a certain morphism of affine varieties, with the polyhedral deformation of Huber and Sturmfels. The complexity of our algorithm is cubic in the size of the combinatorial structure of the input system. This size is mainly represented by the cardinality and mixed volume of Newton polytopes of the input polynomials and an arithmetic analogue of the mixed volume associated to the deformations under consideration. Research was partially supported by the following grants: UBACyT X112 (2004–2007), UBACyT X847 (2006–2009), PIP CONICET 2461, PIP CONICET 5852/05, ANPCyT PICT 2005 17-33018, UNGS 30/3005, MTM2004-01167 (2004–2007), MTM2007-62799 and CIC 2007–2008.     

Sparse finite element methods for operator equations with stochastic data

Let A: V → V′ be a strongly elliptic operator on a d-dimensional manifold D (polyhedra or boundaries of polyhedra are also allowed). An operator equation Au = f with stochastic data f is considered. The goal of the computation is the mean field and higher moments of the solution. We discretize the mean field problem using a FEM with hierarchical basis and N degrees of freedom. We present a Monte-Carlo algorithm and a deterministic algorithm for the approximation of the moment for k⩾1. The key tool in both algorithms is a “sparse tensor product” space for the approximation of with O(N(log N) ^k−1) degrees of freedom, instead of N ^k degrees of freedom for the full tensor product FEM space. A sparse Monte-Carlo FEM with M samples (i.e., deterministic solver) is proved to yield approximations to with a work of O(M N(log N) ^k−1) operations. The solutions are shown to converge with the optimal rates with respect to the Finite Element degrees of freedom N and the number M of samples. The deterministic FEM is based on deterministic equations for in D ^k ⊂ ℝ^kd. Their Galerkin approximation using sparse tensor products of the FE spaces in D allows approximation of with O(N(log N) ^k−1) degrees of freedom converging at an optimal rate (up to logs). For nonlocal operators wavelet compression of the operators is used. The linear systems are solved iteratively with multilevel preconditioning. This yields an approximation for with at most O(N (log N) ^k+1) operations. This work was supported under IHP Network “Breaking Complexity” by the Swiss BBW under grant No. 02.0418     

Sparse potentials with fractional Hausdorff dimension

We construct non-random bounded discrete half-line Schrödinger operators which have purely singular continuous spectral measures with fractional Hausdorff dimension (in some interval of energies). To do this we use suitable sparse potentials. Our results also apply to whole line operators, as well as to certain random operators. In the latter case we prove and compute an exact dimension of the spectral measures.     

Sparse geometric graphs with small dilation

Given a set S of n points in , and an integer k such that 0k<n, we show that a geometric graph with vertex set S, at most n−1+k edges, maximum degree five, and dilation O(n/(k+1)) can be computed in time O(nlogn). For any k, we also construct planar n-point sets for which any geometric graph with n−1+k edges has dilation Ω(n/(k+1)); a slightly weaker statement holds if the points of S are required to be in convex position.     

Sparse graph certificates for mixed connectivity

We give a short proof for the Mixed Connectivity Certificate Theorem of Even, Itkis and Rajsbaum and provide an upper bound on the edge number of a certificate of local T-mixed connectivity up to k.     

Elimination for Generic Sparse Polynomial Systems

We present a new probabilistic symbolic algorithm that, given a variety defined in an n-dimensional affine space by a generic sparse system with fixed supports, computes the Zariski closure of its projection to an ?-dimensional coordinate affine space with ?<n. The complexity of the algorithm depends polynomially on some combinatorial invariants associated to the supports.     

Sparse grids for boundary integral equations

Summary. The potential of sparse grid discretizations for solving boundary integral equations is studied for the screen problem on a square in . Theoretical and numerical results on approximation rates, preconditioning, adaptivity and compression for piecewise constant and linear sparse grid spaces are obtained. Received March 17, 1998 / Revised version received September 10, 1998