Support vector machine via nonlinear rescaling method

In this paper we construct the linear support vector machine (SVM) based on the nonlinear rescaling (NR) methodology (see [Polyak in Math Program 54:177–222, 1992; Polyak in Math Program Ser A 92:197–235, 2002; Polyak and Teboulle in Math Program 76:265–284, 1997] and references therein). The formulation of the linear SVM based on the NR method leads to an algorithm which reduces the number of support vectors without compromising the classification performance compared to the linear soft-margin SVM formulation. The NR algorithm computes both the primal and the dual approximation at each step. The dual variables associated with the given data-set provide important information about each data point and play the key role in selecting the set of support vectors. Experimental results on ten benchmark classification problems show that the NR formulation is feasible. The quality of discrimination, in most instances, is comparable to the linear soft-margin SVM while the number of support vectors in several instances were substantially reduced.     

Incremental accelerated gradient methods for SVM classification: study of the constrained approach

We investigate constrained first order techniques for training support vector machines (SVM) for online classification tasks. The methods exploit the structure of the SVM training problem and combine ideas of incremental gradient technique, gradient acceleration and successive simple calculations of Lagrange multipliers. Both primal and dual formulations are studied and compared. Experiments show that the constrained incremental algorithms working in the dual space achieve the best trade-off between prediction accuracy and training time. We perform comparisons with an unconstrained large scale learning algorithm (Pegasos stochastic gradient) to emphasize that our choice can remain competitive for large scale learning due to the very special structure of the training problem.     

Nonanticipative duality,relaxations, and formulations for chance-constrained stochastic programs

We propose two new Lagrangian dual problems for chance-constrained stochastic programs based on relaxing nonanticipativity constraints. We compare the strength of the proposed dual bounds and demonstrate that they are superior to the bound obtained from the continuous relaxation of a standard mixed-integer programming (MIP) formulation. For a given dual solution, the associated Lagrangian relaxation bounds can be calculated by solving a set of single scenario subproblems and then solving a single knapsack problem. We also derive two new primal MIP formulations and demonstrate that for chance-constrained linear programs, the continuous relaxations of these formulations yield bounds equal to the proposed dual bounds. We propose a new heuristic method and two new exact algorithms based on these duals and formulations. The first exact algorithm applies to chance-constrained binary programs, and uses either of the proposed dual bounds in concert with cuts that eliminate solutions found by the subproblems. The second exact method is a branch-and-cut algorithm for solving either of the primal formulations. Our computational results indicate that the proposed dual bounds and heuristic solutions can be obtained efficiently, and the gaps between the best dual bounds and the heuristic solutions are small.     

Linear programming approaches for multicategory support vector machines

We consider linear programming approaches for support vector machines (SVM). The linear programming problems are introduced as an approximation of the quadratic programming problems commonly used in SVM. When we consider the kernel based nonlinear discriminators, the approximation can be viewed as kernel principle component analysis which generates an important subspace from the feature space characterized the kernel function. We show that any data points nonlinearly, and implicitly, projected into the feature space by kernel functions can be approximately expressed as points lying a low dimensional Euclidean space explicitly, which enables us to develop linear programming formulations for nonlinear discriminators. We also introduce linear programming formulations for multicategory classification problems. We show that the same maximal margin principle exploited in SVM can be involved into the linear programming formulations. Moreover, considering the low dimensional feature subspace extraction, we can generate nonlinear multicategory discriminators by solving linear programming problems.Numerical experiments on real world datasets are presented. We show that the fairly low dimensional feature subspace can achieve a reasonable accuracy, and that the linear programming formulations calculate discriminators efficiently. We also discuss a sampling strategy which might be crucial for huge datasets.     

Reinforced Angle-Based Multicategory Support Vector Machines

The support vector machine (SVM) is a very popular classification tool with many successful applications. It was originally designed for binary problems with desirable theoretical properties. Although there exist various multicategory SVM (MSVM) extensions in the literature, some challenges remain. In particular, most existing MSVMs make use of k classification functions for a k-class problem, and the corresponding optimization problems are typically handled by existing quadratic programming solvers. In this article, we propose a new group of MSVMs, namely, the reinforced angle-based MSVMs (RAMSVMs), using an angle-based prediction rule with k ? 1 functions directly. We prove that RAMSVMs can enjoy Fisher consistency. Moreover, we show that the RAMSVM can be implemented using the very efficient coordinate descent algorithm on its dual problem. Numerical experiments demonstrate that our method is highly competitive in terms of computational speed, as well as classification prediction performance. Supplemental materials for the article are available online.     

New smoothing SVM algorithm with tight error bound and efficient reduced techniques

The quadratically convergent algorithms for training SVM with smoothing methods are discussed in this paper. By smoothing the objective function of an SVM formulation, Lee and Mangasarian [Comput. Optim. Appl. 20(1):5-22, 2001] presented one such algorithm called SSVM and proved that the error bound between the new smooth problem and the original one was $O(\frac{1}{p})$ for large positive smoothing parameter p. We derive a new method by smoothing the optimality conditions of the SVM formulation, and we prove that the error bound is $O(\frac{1}{p^{2}})$ , which is better than Lee and Mangasarian’s result. Based on SMW identity and updating Hessian iteratively, some boosting skills are proposed to solve Newton equation with lower computational complexity for reduced smooth SVM algorithms. Many experimental results show that the proposed smoothing method has the same accuracy as SSVM, whose error bound is also tightened to $O(\frac{1}{p^{2}})$ in this paper, and the proposed boosting skills are efficient for solving large-scale problems by RSVM.     

Bounding Ramsey numbers through large deviation inequalities

We develop a new approach for proving lower bounds for various Ramsey numbers, based on using large deviation inequalities. This approach enables us to obtain the bounds for the off-diagonal Ramsey numbers R(K^r, K^k), r ≤ k, that match the best known bounds, obtained through the local lemma. We discuss also the bounds for a related Ramsey-type problem and show, for example, that there exists a K⁴-free graph G on n vertices in which every cn^3/5 log^1/2 n vertices span a copy of K³. © 1995 John Wiley & Sons, Inc.     

Adjusted support vector machines based on a new loss function

Support vector machine (SVM) has attracted considerable attentions recently due to its successful applications in various domains. However, by maximizing the margin of separation between the two classes in a binary classification problem, the SVM solutions often suffer two serious drawbacks. First, SVM separating hyperplane is usually very sensitive to training samples since it strongly depends on support vectors which are only a few points located on the wrong side of the corresponding margin boundaries. Second, the separating hyperplane is equidistant to the two classes which are considered equally important when optimizing the separating hyperplane location regardless the number of training data and their dispersions in each class. In this paper, we propose a new SVM solution, adjusted support vector machine (ASVM), based on a new loss function to adjust the SVM solution taking into account the sample sizes and dispersions of the two classes. Numerical experiments show that the ASVM outperforms conventional SVM, especially when the two classes have large differences in sample size and dispersion.     

A duality principle for groups

The duality principle for Gabor frames states that a Gabor sequence obtained by a time-frequency lattice is a frame for L²(Rd) if and only if the associated adjoint Gabor sequence is a Riesz sequence. We prove that this duality principle extends to any dual pairs of projective unitary representations of countable groups. We examine the existence problem of dual pairs and establish some connection with classification problems for II₁ factors. While in general such a pair may not exist for some groups, we show that such a dual pair always exists for every subrepresentation of the left regular unitary representation when G is an abelian infinite countable group or an amenable ICC group. For free groups with finitely many generators, the existence problem of such a dual pair is equivalent to the well-known problem about the classification of free group von Neumann algebras.     

Some properties of the exit measure for super Brownian motion

 We consider the exit measure of super Brownian motion with a stable branching mechanism of a smooth domain D of ℝ ^d . We derive lower bounds for the hitting probability of small balls for the exit measure and upper bounds in the critical dimension. This completes results given by Sheu [22] and generalizes the results of Abraham and Le Gall [2]. Because of the links between exits measure and partial differential equations, those results imply bounds on solutions of elliptic semi-linear PDE. We also give the Hausdorff dimension of the support of the exit measure and show it is totally disconnected in high dimension. Eventually we prove the exit measure is singular with respect to the surface measure on ∂D in the critical dimension. Our main tool is the subordinated Brownian snake introduced by Bertoin, Le Gall and Le Jan [4]. Received: 6 December 1999 / Revised version: 9 June 2000 / Published online: 11 December 2001     

Accuracy of suboptimal solutions to kernel principal component analysis

For Principal Component Analysis in Reproducing Kernel Hilbert Spaces (KPCA), optimization over sets containing only linear combinations of all n-tuples of kernel functions is investigated, where n is a positive integer smaller than the number of data. Upper bounds on the accuracy in approximating the optimal solution, achievable without restrictions on the number of kernel functions, are derived. The rates of decrease of the upper bounds for increasing number n of kernel functions are given by the summation of two terms, one proportional to n ^−1/2 and the other to n ⁻¹, and depend on the maximum eigenvalue of the Gram matrix of the kernel with respect to the data. Primal and dual formulations of KPCA are considered. The estimates provide insights into the effectiveness of sparse KPCA techniques, aimed at reducing the computational costs of expansions in terms of kernel units.     

An expanded mixed finite element approach via a dual–dual formulation and the minimum residual method

We apply an expanded mixed finite element method, which introduces the gradient as a third explicit unknown, to solve a linear second-order elliptic equation in divergence form. Instead of using the standard dual form, we show that the corresponding variational formulation can be written as a dual–dual operator equation. We establish existence and uniqueness of solution for the continuous and discrete formulations, and provide the corresponding error analysis by using Raviart–Thomas elements. In addition, we show that the corresponding dual–dual linear system can be efficiently solved by a preconditioned minimum residual method. Some numerical results, illustrating this fact and the rate of convergence of the mixed finite element method, are also provided.     

Relevance regression learning with support vector machines

We propose a variant of two SVM regression algorithms expressly tailored in order to exploit additional information summarizing the relevance of each data item, as a measure of its relative importance w.r.t. the remaining examples. These variants, enclosing the original formulations when all data items have the same relevance, are preliminary tested on synthetic and real-world data sets. The obtained results outperform standard SVM approaches to regression if evaluated in light of the above mentioned additional information about data quality.     

A lower bound for the minimum weight of the dual 7-ary code of a projective plane of order 49

Existing bounds on the minimum weight d ^⊥ of the dual 7-ary code of a projective plane of order 49 show that this must be in the range 76 ≤ d ^⊥ ≤ 98. We use combinatorial arguments to improve this range to 88 ≤ d ^⊥ ≤ 98, noting that the upper bound can be taken to be 91 if the plane has a Baer subplane, as in the desarguesian case. A brief survey of known results for the minimum weight of the dual codes of finite projective planes is also included. Dedicated to Dan Hughes on the occasion of his 80th birthday.     

Exploring the trade-off between generalization and empirical errors in a one-norm SVM

We propose a one-norm support vector machine (SVM) formulation as an alternative to the well-known formulation that uses parameter C in order to balance the two inherent objective functions of the problem. Our formulation is motivated by the ?-constraint approach that is used in bicriteria optimization and we propose expressing the objective of minimizing total empirical error as a constraint with a parametric right-hand-side. Using dual variables we show equivalence of this formulation to the one with the trade-off parameter. We propose an algorithm that enumerates the entire efficient frontier by systematically changing the right-hand-side parameter. We discuss the results of a detailed computational analysis that portrays the structure of the efficient frontier as well as the computational burden associated with finding it. Our results indicate that the computational effort for obtaining the efficient frontier grows linearly in problem size, and the benefit in terms of classifier performance is almost always substantial when compared to a single run of the corresponding SVM. In addition, both the run time and accuracy compare favorably to other methods that search part or all of the regularization path of SVM.     

Bounds for solid angles of lattices of rank three

We find sharp absolute constants C₁ and C₂ with the following property: every well-rounded lattice of rank 3 in a Euclidean space has a minimal basis so that the solid angle spanned by these basis vectors lies in the interval [C₁,C₂]. In fact, we show that these absolute bounds hold for a larger class of lattices than just well-rounded, and the upper bound holds for all. We state a technical condition on the lattice that may prevent it from satisfying the absolute lower bound on the solid angle, in which case we derive a lower bound in terms of the ratios of successive minima of the lattice. We use this result to show that among all spherical triangles on the unit sphere in RN with vertices on the minimal vectors of a lattice, the smallest possible area is achieved by a configuration of minimal vectors of the (normalized) face centered cubic lattice in R³. Such spherical configurations come up in connection with the kissing number problem.     

Nearness and Bound Relationships Between an Integer-Programming Problem and Its Relaxed Linear-Programming Problem

We discuss relationships between the solution to an integer-programming problem and the solution to its relaxed linear-programming problem in terms of the distance that separates them and related bounds. Assuming knowledge of a subset of extreme points, we develop bounds for associated convex combinations and show how improved bounds on the integer-programming problem's objective function and variables can be obtained.     

Improved twin support vector machine

We improve the twin support vector machine(TWSVM)to be a novel nonparallel hyperplanes classifier,termed as ITSVM(improved twin support vector machine),for binary classification.By introducing the diferent Lagrangian functions for the primal problems in the TWSVM,we get an improved dual formulation of TWSVM,then the resulted ITSVM algorithm overcomes the common drawbacks in the TWSVMs and inherits the essence of the standard SVMs.Firstly,ITSVM does not need to compute the large inverse matrices before training which is inevitable for the TWSVMs.Secondly,diferent from the TWSVMs,kernel trick can be applied directly to ITSVM for the nonlinear case,therefore nonlinear ITSVM is superior to nonlinear TWSVM theoretically.Thirdly,ITSVM can be solved efciently by the successive overrelaxation(SOR)technique or sequential minimization optimization(SMO)method,which makes it more suitable for large scale problems.We also prove that the standard SVM is the special case of ITSVM.Experimental results show the efciency of our method in both computation time and classification accuracy.     

Frames for vector spaces and affine spaces

A finite frame for a finite dimensional Hilbert space is simply a spanning sequence. We show that the linear functionals given by the dual frame vectors do not depend on the inner product, and thus it is possible to extend the frame expansion (and other elements of frame theory) to any finite spanning sequence for a vector space. The corresponding coordinate functionals generalise the dual basis (the case when the vectors are linearly independent), and are characterised by the fact that the associated Gramian matrix is an orthogonal projection. Existing generalisations of the frame expansion to Banach spaces involve an analogue of the frame bounds and frame operator.The potential applications of our results are considerable. Whenever there is a natural spanning set for a vector space, computations can be done directly with it, in an efficient and stable way. We illustrate this with a diverse range of examples, including multivariate spline spaces, generalised barycentric coordinates, and vector spaces over the rationals, such as the cyclotomic fields.     

About the length of laws for finite groups

We prove new upper bounds of the form O(n/log(n)^2?ε ) for the length of laws that hold for all groups of size at most n — improving on previous results of Bou-Rabee and Kassabov–Matucci. The methods make use of the classification of finite simple groups. Stronger bounds are proved in case the groups are assumed to be nilpotent or solvable.