On smoothing,regularization, and averaging in stochastic approximation methods for stochastic variational inequality problems

Traditionally, most stochastic approximation (SA) schemes for stochastic variational inequality (SVI) problems have required the underlying mapping to be either strongly monotone or monotone and Lipschitz continuous. In contrast, we consider SVIs with merely monotone and non-Lipschitzian maps. We develop a regularized smoothed SA (RSSA) scheme wherein the stepsize, smoothing, and regularization parameters are reduced after every iteration at a prescribed rate. Under suitable assumptions on the sequences, we show that the algorithm generates iterates that converge to the least norm solution in an almost sure sense, extending the results in Koshal et al. (IEEE Trans Autom Control 58(3):594–609, 2013) to the non-Lipschitzian regime. Additionally, we provide rate estimates that relate iterates to their counterparts derived from a smoothed Tikhonov trajectory associated with a deterministic problem. To derive non-asymptotic rate statements, we develop a variant of the RSSA scheme, denoted by aRSSA\(_r\), in which we employ a weighted iterate-averaging, parameterized by a scalar r where \(r = 1\) provides us with the standard averaging scheme. The main contributions are threefold: (i) when \(r<1\) and the parameter sequences are chosen appropriately, we show that the averaged sequence converges to the least norm solution almost surely and a suitably defined gap function diminishes at an approximate rate \(\mathcal{O}({1}\slash {\root 6 \of {k}})\) after k steps; (ii) when \(r<1\), and smoothing and regularization are suppressed, the gap function admits the rate \(\mathcal{O}({1}\slash {\sqrt{k}})\), thus improving the rate \(\mathcal{O}(\ln (k)/\sqrt{k})\) under standard averaging; and (iii) we develop a window-based variant of this scheme that also displays the optimal rate for \(r < 1\). Notably, we prove the superiority of the scheme with \(r < 1\) with its counterpart with \(r=1\) in terms of the constant factor of the error bound when the size of the averaging window is sufficiently large. We present the performance of the developed schemes on a stochastic Nash–Cournot game with merely monotone and non-Lipschitzian maps.     

Intercompany Study to Evaluate the Robustness of Capillary Isoelectric Focusing Technology for the Analysis of Monoclonal Antibodies

Interlaboratory comparisons are essential to bringing emerging technologies into biopharmaceutical industry practice and regulatory acceptance. As a result, an international team including 12 laboratories from 10 independent biopharmaceutical companies in the United States and Switzerland was formed to evaluate the precision and robustness of capillary isoelectric focusing (CIEF) to assess the charge heterogeneity of monoclonal antibodies. The different laboratories determined the apparent pI and the relative distribution of the charge isoforms of a representative monoclonal antibody (rMAb) sample using the same CIEF method. Statistical evaluation of the data was performed to determine within and between-laboratory consistencies and outlying information. The apparent pI data generated for each charge variant peak showed very good precision between laboratories with percentage of RSD values of ??0.5%. Similarly, the percentage of RSD for the rMAb charge variants percent peak area values are ??4.4% across different laboratories with different analysts using different lots of ampholytes and multiple instruments. Taken together, these results validate the appropriate use of CIEF in the biopharmaceutical industry in support of regulatory submissions.     

A geometric framework for nonconvex optimization duality using augmented lagrangian functions

We provide a unifying geometric framework for the analysis of general classes of duality schemes and penalty methods for nonconvex constrained optimization problems. We present a separation result for nonconvex sets via general concave surfaces. We use this separation result to provide necessary and sufficient conditions for establishing strong duality between geometric primal and dual problems. Using the primal function of a constrained optimization problem, we apply our results both in the analysis of duality schemes constructed using augmented Lagrangian functions, and in establishing necessary and sufficient conditions for the convergence of penalty methods.     

Morphology of nanostructured platinum in mesoporous materials-effect of solvent and intrachannel surface

An intrachannel surface of host silica was functionalized through the reaction of surface silanol groups with silanes to generate a monolayer of positively charged groups, and together with the strongly adsorbed and negatively charged PtCl6(2-), resulting in nanostructured platinum-mesoporous silica composites. The highly dispersed Pt nanoparticles and nanonetworks are fabricated from (CH3O)3Si(CH2)3N(CH3)3+Cl- functionalized mesoporous silica MCM-48 with H2PtCl6 in ethanol and water solvent, and characterized by PXRD, XAS, TEM, and N2 adsorption. The solvent of H2PtCl6 solution is found to affect the mobility of Pt precursors and the resulting morphology of nanostructured metallic Pt. The effect of the intrachannel surface properties on the incorporation and the morphology of nanostructured Pt on the deposition of Pt(NH3)4Cl2 and H2PtCl6 on Al-doped or C-coated mesoporous silica MCM-41 is also studied relative to that on pure silica MCM-41.     

A Series of Collaborations Between Various Pharmaceutical Companies and Regulatory Authorities Concerning the Analysis of Biomolecules Using Capillary Electrophoresis

An international project team (including members from US, Canada and UK) has been formed from a number of interested biopharmaceutical companies and regulatory authorities to conduct a cross-organisation collaboration exercise. The results from this exercise demonstrate the robustness of CE-SDS across eight different organisations that used instruments of the same equipment model, the same reagents, and the same methodology. Data generated from the analysis of a series of molecular weight markers showed very good precision with regards to relative migration time (RMT) both within and between organisations. The apparent molecular weight of bovine serum albumin (BSA) was measured with good precision to within approximately 2% RSD across the participants. A representative IgG sample showed similar results with regards to relative migration time of its 3 main components, IgG Light Chain, IgG Non-glycosylated Heavy Chain, and IgG Heavy Chain. Fractional peak area for each peak also showed good agreement, with less than 9% RSD for all peaks. This exercise will facilitate both increased regulatory and industrial opinion of CE for biopharmaceutical analysis.CE in the Biotechnology & Pharmaceutical Industries: 7th Symposium on the Practical Applications for the Analysis of Proteins, Nucleotides and Small Molecules, Montreal, Canada, August 12–16, 2005     

The effect of deterministic noise in subgradient methods

In this paper, we study the influence of noise on subgradient methods for convex constrained optimization. The noise may be due to various sources, and is manifested in inexact computation of the subgradients and function values. Assuming that the noise is deterministic and bounded, we discuss the convergence properties for two cases: the case where the constraint set is compact, and the case where this set need not be compact but the objective function has a sharp set of minima (for example the function is polyhedral). In both cases, using several different stepsize rules, we prove convergence to the optimal value within some tolerance that is given explicitly in terms of the errors. In the first case, the tolerance is nonzero, but in the second case, the optimal value can be obtained exactly, provided the size of the error in the subgradient computation is below some threshold. We then extend these results to objective functions that are the sum of a large number of convex functions, in which case an incremental subgradient method can be used.     

Minibatch stochastic subgradient-based projection algorithms for feasibility problems with convex inequalities

In this paper we consider convex feasibility problems where the feasible set is given as the intersection of a collection of closed convex sets. We assume that each set is specified algebraically as a convex inequality, where the associated convex function is general (possibly non-differentiable). For finding a point satisfying all the convex inequalities we design and analyze random projection algorithms using special subgradient iterations and extrapolated stepsizes. Moreover, the iterate updates are performed based on parallel random observations of several constraint components. For these minibatch stochastic subgradient-based projection methods we prove sublinear convergence results and, under some linear regularity condition for the functional constraints, we prove linear convergence rates. We also derive sufficient conditions under which these rates depend explicitly on the minibatch size. To the best of our knowledge, this work is the first deriving conditions that show theoretically when minibatch stochastic subgradient-based projection updates have a better complexity than their single-sample variants when parallel computing is used to implement the minibatch. Numerical results also show a better performance of our minibatch scheme over its non-minibatch counterpart.

     

A Series of Collaborations between Various Pharmaceutical Companies and Regulatory Authorities Concerning the Analysis of Biomolecules Using Capillary Electrophoresis: Additional Instruments/Buffer

An international project team (including members from US, Canada and UK) was formed from a number of interested biopharmaceutical companies and regulatory authorities to conduct a cross-organisation collaboration exercise. The results of the first comparison with eight different organisations that used instruments of the same equipment model, the same reagents, and the same methodology has been reported previously [1]. This report represents the addition of other instruments using a different run buffer. The relative migration times were different, as expected, prohibiting a direct comparison between companies. The within-organisation variability was low for both relative migration time (<0.34% RSD% for all companies save one) and the peak area (<5% RSD% for all companies save one) when measuring the purity of a representative IgG sample. The apparent molecular weight of bovine serum albumin was measured with good precision (less than 10% RSD% across all companies) to the theoretical value when all data is utilized (67.5 kDa compared to 66.4 kDa). For a representative IgG sample, the three main components, IgG Light Chain, IgG Non-glycosylated Heavy Chain, and IgG Heavy Chain, could not be separated, specifically the IgG Non-glycosylated Heavy Chain and IgG Heavy Chain. When the IgG Non-glycosylated Heavy Chain and IgG Heavy Chain were combined for all organisations, the fractional peak area for the IgG Light Chain and IgG Non-glycosylated Heavy Chain + IgG Heavy Chain peak also showed excellent agreement, with less than 7.5 and 3.5% RSD%, respectively. The value of this exercise is in demonstrating the reliability of CE for the determination of apparent size of biopharmaceutical proteins. This underpins the appropriate use of such CE data in support of regulatory submissions.     

Random algorithms for convex minimization problems

This paper deals with iterative gradient and subgradient methods with random feasibility steps for solving constrained convex minimization problems, where the constraint set is specified as the intersection of possibly infinitely many constraint sets. Each constraint set is assumed to be given as a level set of a convex but not necessarily differentiable function. The proposed algorithms are applicable to the situation where the whole constraint set of the problem is not known in advance, but it is rather learned in time through observations. Also, the algorithms are of interest for constrained optimization problems where the constraints are known but the number of constraints is either large or not finite. We analyze the proposed algorithm for the case when the objective function is differentiable with Lipschitz gradients and the case when the objective function is not necessarily differentiable. The behavior of the algorithm is investigated both for diminishing and non-diminishing stepsize values. The almost sure convergence to an optimal solution is established for diminishing stepsize. For non-diminishing stepsize, the error bounds are established for the expected distances of the weighted averages of the iterates from the constraint set, as well as for the expected sub-optimality of the function values along the weighted averages.     

On Stochastic Mirror-prox Algorithms for Stochastic Cartesian Variational Inequalities: Randomized Block Coordinate and Optimal Averaging Schemes

Motivated by multi-user optimization problems and non-cooperative Nash games in uncertain regimes, we consider stochastic Cartesian variational inequality problems where the set is given as the Cartesian product of a collection of component sets. First, we consider the case where the number of the component sets is large and develop a randomized block stochastic mirror-prox algorithm, where at each iteration only a randomly selected block coordinate of the solution vector is updated through implementing two consecutive projection steps. We show that when the mapping is strictly pseudo-monotone, the algorithm generates a sequence of iterates that converges to the solution of the problem almost surely. When the maps are strongly pseudo-monotone, we prove that the mean-squared error diminishes at the optimal rate. Second, we consider large-scale stochastic optimization problems with convex objectives and develop a new averaging scheme for the randomized block stochastic mirror-prox algorithm. We show that by using a different set of weights than those employed in the classical stochastic mirror-prox methods, the objective values of the averaged sequence converges to the optimal value in the mean sense at an optimal rate. Third, we consider stochastic Cartesian variational inequality problems and develop a stochastic mirror-prox algorithm that employs the new weighted averaging scheme. We show that the expected value of a suitably defined gap function converges to zero at an optimal rate.