Functional optimization by variable-basis approximation schemes

This is a summary of the author’s PhD thesis, supervised by Marcello Sanguineti and defended on April 2, 2009 at Università degli Studi di Genova. The thesis is written in English and a copy is available from the author upon request. Functional optimization problems arising in Operations Research are investigated. In such problems, a cost functional Φ has to be minimized over an admissible set S of d-variable functions. As, in general, closed-form solutions cannot be derived, suboptimal solutions are searched for, having the form of variable-basis functions, i.e., elements of the set span _n   G of linear combinations of at most n elements from a set G of computational units. Upper bounds on inf_{f ? S ?spann G}F(f)-inf_{f ? S}F(f){\inf_{f \in S \cap {\rm span}_n\, G}\Phi(f)-\inf_{f \in S}\Phi(f)} are obtained. Conditions are derived, under which the estimates do not exhibit the so-called “curse of dimensionality” in the number n of computational units, when the number d of variables grows. The problems considered include dynamic optimization, team optimization, and supervised learning from data.     

Approximating Networks and Extended Ritz Method for the Solution of Functional Optimization Problems

Functional optimization problems can be solved analytically only if special assumptions are verified; otherwise, approximations are needed. The approximate method that we propose is based on two steps. First, the decision functions are constrained to take on the structure of linear combinations of basis functions containing free parameters to be optimized (hence, this step can be considered as an extension to the Ritz method, for which fixed basis functions are used). Then, the functional optimization problem can be approximated by nonlinear programming problems. Linear combinations of basis functions are called approximating networks when they benefit from suitable density properties. We term such networks nonlinear (linear) approximating networks if their basis functions contain (do not contain) free parameters. For certain classes of d-variable functions to be approximated, nonlinear approximating networks may require a number of parameters increasing moderately with d, whereas linear approximating networks may be ruled out by the curse of dimensionality. Since the cost functions of the resulting nonlinear programming problems include complex averaging operations, we minimize such functions by stochastic approximation algorithms. As important special cases, we consider stochastic optimal control and estimation problems. Numerical examples show the effectiveness of the method in solving optimization problems stated in high-dimensional settings, involving for instance several tens of state variables.     

Condition measures and properties of the central trajectory of a linear program

Given a data instanced=(A, b, c) of a linear program, we show that certain properties of solutions along the central trajectory of the linear program are inherently related to the condition number C(d) of the data instanced=(A, b, c), where C(d) is a scale-invariant reciprocal of a closely-related measure ρ(d) called the “distance to ill-posedness”. (The distance to ill-posedness essentially measures how close the data instanced=(A,b,c) is to being primal or dual infeasible.) We present lower and upper bounds on sizes of optimal solutions along the central trajectory, and on rates of change of solutions along the central trajectory, as either the barrier parameter μ or the datad=(A, b, c) of the linear program is changed. These bounds are all linear or polynomial functions of certain natural parameters associated with the linear program, namely the condition number C(d), the distance to ill-posedness ρ(d), the norm of the data ‖d‖, and the dimensionsm andn.     

Management of water resource systems in the presence of uncertainties by nonlinear approximation techniques and deterministic sampling

Two methods of approximate solution are developed for T-stage stochastic optimal control (SOC) problems, aimed at obtaining finite-horizon management policies for water resource systems. The presence of uncertainties, such as river and rain inflows, is considered. Both approaches are based on the use of families of nonlinear functions, called “one-hidden-layer networks” (OHL networks), made up of linear combinations of simple basis functions containing parameters to be optimized. The first method exploits OHL networks to obtain an accurate approximation of the cost-to-go functions in the dynamic programming procedure for SOC problems. The approximation capabilities of OHL networks are combined with the properties of deterministic sampling techniques aimed at obtaining uniform samplings of high-dimensional domains. In the second method, admissible solutions to SOC problems are constrained to take on the form of OHL networks, whose parameters are determined in such a way to minimize the cost functional associated with SOC problems. Exploiting these tools, the two methods are able to cope with the so-called “curse of dimensionality,” which strongly limits the applicability of existing techniques to high-dimensional water resources management in the presence of uncertainties. The theoretical bases of the two approaches are investigated. Simulation results show that the proposed methods are effective for water resource systems of high dimension.     

Asymptotic properties of combinatorial optimization problems in <Emphasis Type="Italic">p</Emphasis>-adic space

In this article we consider two well known combinatorial optimization problems (travel-ling salesman and minimum spanning tree), when n points are randomly distributed in a unit p-adic ball of dimension d. We investigate an asymptotic behavior of their solutions at large number of n. It was earlier found that the average lengths of the optimal solutions in both problems are of order n ^1−1/d . Here we show that standard deviations of the optimal lengths are of order n ^1/2−1/d if d > 1, and prove that large number laws are valid only for special subsequences of n.     

Reduced Vertex Set Result for Interval Semidefinite Optimization Problems

In this paper we propose a reduced vertex result for the robust solution of uncertain semidefinite optimization problems subject to interval uncertainty. If the number of decision variables is m and the size of the coefficient matrices in the linear matrix inequality constraints is n×n, a direct vertex approach would require satisfaction of 2 ^{n(m+1)(n+1)/2} vertex constraints: a huge number, even for small values of n and m. The conditions derived here are instead based on the introduction of m slack variables and a subset of vertex coefficient matrices of cardinality 2 ⁿ⁻¹, thus reducing the problem to a practically manageable size, at least for small n. A similar size reduction is also obtained for a class of problems with affinely dependent interval uncertainty. This work is supported by MIUR under the FIRB project “Learning, Randomization and Guaranteed Predictive Inference for Complex Uncertain Systems,” and by CNR RSTL funds.     

Minimizing Sequences for a Family of Functional Optimal Estimation Problems

Rates of convergence are derived for approximate solutions to optimization problems associated with the design of state estimators for nonlinear dynamic systems. Such problems consist in minimizing the functional given by the worst-case ratio between the ℒ _p -norm of the estimation error and the sum of the ℒ _p -norms of the disturbances acting on the dynamic system. The state estimator depends on an innovation function, which is searched for as a minimizer of the functional over a subset of a suitably-defined functional space. In general, no closed-form solutions are available for these optimization problems. Following the approach proposed in (Optim. Theory Appl. 134:445–466, 2007), suboptimal solutions are searched for over linear combinations of basis functions containing some parameters to be optimized. The accuracies of such suboptimal solutions are estimated in terms of the number of basis functions. The estimates hold for families of approximators used in applications, such as splines of suitable orders.     

Basic Theory in Construction of Boolean Functions with Maximum Possible Annihilator Immunity

So far there is no systematic attempt to construct Boolean functions with maximum annihilator immunity. In this paper we present a construction keeping in mind the basic theory of annihilator immunity. This construction provides functions with the maximum possible annihilator immunity and the weight, nonlinearity and algebraic degree of the functions can be properly calculated under certain cases. The basic construction is that of symmetric Boolean functions and applying linear transformation on the input variables of these functions, one can get a large class of non-symmetric functions too. Moreover, we also study several other modifications on the basic symmetric functions to identify interesting non-symmetric functions with maximum annihilator immunity. In the process we also present an algorithm to compute the Walsh spectra of a symmetric Boolean function with O(n²) time and O(n) space complexity. We use the term “Annihilator Immunity” instead of “Algebraic Immunity” referred in the recent papers [3–5, 9, 18, 19]. Please see Remark 1 for the details of this notational change     

Quantum matrix algebras of the GL(m|n) type: The structure and spectral parameterization of the characteristic subalgebra

We continue the study of quantum matrix algebras of the GL(m|n) type. We find three alternative forms of the Cayley-Hamilton identity; most importantly, this identity can be represented in a factored form. The factorization allows naturally dividing the spectrum of a quantum supermatrix into subsets of “even” and “odd” eigenvalues. This division leads to a parameterization of the characteristic subalgebra (the subalgebra of spectral invariants) in terms of supersymmetric polynomials in the eigenvalues of the quantum matrix. Our construction is based on two auxiliary results, which are independently interesting. First, we derive the multiplication rule for Schur functions s _λ (M) that form a linear basis of the characteristic subalgebra of a Hecke-type quantum matrix algebra; the structure constants in this basis coincide with the Littlewood-Richardson coefficients. Second, we prove a number of bilinear relations in the graded ring Λ of symmetric functions of countably many variables. __________ Translated from Teoreticheskaya i Matematicheskaya Fizika, Vol. 147, No. 1, pp. 14–46, April, 2006.     

New cubature formulae and hyperinterpolation in three variables

A new algebraic cubature formula of degree 2n+1 for the product Chebyshev measure in the d-cube with ≈n ^d /2 ^d−1 nodes is established. The new formula is then applied to polynomial hyperinterpolation of degree n in three variables, in which coefficients of the product Chebyshev orthonormal basis are computed by a fast algorithm based on the 3-dimensional FFT. Moreover, integration of the hyperinterpolant provides a new Clenshaw-Curtis type cubature formula in the 3-cube. Work supported by the National Science Foundation under Grant DMS-0604056, by the “ex-60%” funds of the Universities of Padova and Verona, and by the INdAM-GNCS.     

Compositions of n as alternating sequences of weakly increasing and strictly decreasing partitions

Compositions and partitions of positive integers are often studied in separate frameworks where partitions are given by q-series generating functions and compositions exhibiting specific patterns are designated by generating functions for these patterns. Here, we view compositions as alternating sequences of weakly increasing and strictly decreasing partitions (i.e. alternating blocks). We obtain generating functions for the number of such partitions in terms of the size of the composition, the number of parts and the total number of “valleys” and “peaks”. From this, we find the total number of “peaks” and “valleys” in the composition of n which have the mentioned pattern. We also obtain the generating function for compositions which split into just two partition blocks. Finally, we obtain the two generating functions for compositions of n that start either with a weakly increasing partition or a strictly decreasing partition.     

Piecewise-Linear Approximations of Multidimensional Functions

We develop explicit, piecewise-linear formulations of functions f(x):ℝ ⁿ ↦ℝ, n≤3, that are defined on an orthogonal grid of vertex points. If mixed-integer linear optimization problems (MILPs) involving multidimensional piecewise-linear functions can be easily and efficiently solved to global optimality, then non-analytic functions can be used as an objective or constraint function for large optimization problems. Linear interpolation between fixed gridpoints can also be used to approximate generic, nonlinear functions, allowing us to approximately solve problems using mixed-integer linear optimization methods. Toward this end, we develop two different explicit formulations of piecewise-linear functions and discuss the consequences of integrating the formulations into an optimization problem.     

Capturing Ridge Functions in High Dimensions from Point Queries

Constructing a good approximation to a function of many variables suffers from the “curse of dimensionality”. Namely, functions on ℝ ^N with smoothness of order s can in general be captured with accuracy at most O(n ^−s/N ) using linear spaces or nonlinear manifolds of dimension n. If N is large and s is not, then n has to be chosen inordinately large for good accuracy. The large value of N often precludes reasonable numerical procedures. On the other hand, there is the common belief that real world problems in high dimensions have as their solution, functions which are more amenable to numerical recovery. This has led to the introduction of models for these functions that do not depend on smoothness alone but also involve some form of variable reduction. In these models it is assumed that, although the function depends on N variables, only a small number of them are significant. Another variant of this principle is that the function lives on a low dimensional manifold. Since the dominant variables (respectively the manifold) are unknown, this leads to new problems of how to organize point queries to capture such functions. The present paper studies where to query the values of a ridge function f(x)=g(a⋅x) when both a∈ℝ ^N and g∈C[0,1] are unknown. We establish estimates on how well f can be approximated using these point queries under the assumptions that g∈C ^s [0,1]. We also study the role of sparsity or compressibility of a in such query problems.     

The Number Of Unique-Sink Orientations of the Hypercube*

Let Q_d denote the graph of the d-dimensional cube. A unique-sink orientation (USO) is an orientation of Q_d such that every face of Q_n has exactly one sink (vertex of out degree 0); it does not have to be acyclic. USO have been studied as an abstract model for many geometric optimization problems, such as linear programming, finding the smallest enclosing ball of a given point set, certain classes of convex programming, and certain linear complementarity problems. It is shown that the number of USO is . * This research was partially supported by ETH Zürichan d done in part during the workshop “Towards the Peak” in La Claustra, Switzerland and during a visit to ETH Zürich.     

On singular perturbation in non well posed problems

Summary Let L_εu and L ₀ v be the elliptic and “backward” heat operators defined by(1.1) and(1.2), respectively. The following question is considered for a pair of “non-well posed” initial-boundary value problems for L_ε and L ₀ : if u and v are the respective solutions, under what restrictions on the classes of admissible solutions and in what sense, if any, does u converge to v as ɛ →0? This research was supported in part by the National Science Foundation Grant No. GP 5882 with Cornell University.     

N-Widths and Average Widths of Besov Classes in Sobolev Spaces

In this paper, we consider the n-widths and average widths of Besov classes in the usual Sobolev spaces. The weak asymptotic results concerning the Kolmogorov n-widths, the linear n-widths, the Gel＇fand n-widths, in the Sobolev spaces on T^d, and the infinite-dimensional widths and the average widths in the Sobolev spaces on Ra are obtained, respectively.     

Star-shaped sets in normed spaces

We prove a generalization of the Krasnosel’ski theorem on star-shaped sets. Usingd-segments inn-dimensional Minkowski spaces instead of usual segments, the notions “d-visibility” and “d-star-shapedness” are introduced. Our main aim is to give necessary and sufficient conditions ford-star-shapedness in finite-dimensional normed spaces.     

A general non-convex large deviation result with applications to stochastic equations

We prove an abstract large deviation result for a sequence of random elements of a vector space satisfying an “abstract exponential martingale condition”. The framework naturally generates non-convex rate functions. We apply the result to solutions of It? stochastic equations in R ^d driven by Brownian motion and a Poisson random measure. Received: 23 June 1999 / Revised version: 17 February 2000 / Published online: 22 November 2000     

B-spline quasi-interpolation on sparse grids

We propose a periodic B-spline quasi-interpolation for multivariate functions on sparse grids and develop a fast scheme for the evaluation of a linear combination of B-splines on sparse grids. We prove that both of these operations require only O(nlog^d−1n) number of multiplications, where n is the number of univariate B-spline basis functions used in each coordinate direction and d is the number of variables of the functions. We also establish the optimal approximation order of the periodic B-spline quasi-interpolation. Numerical examples are presented to confirm the theoretical estimates.