A novel method for multivariate data modelling: Piecewise Generalized EMPR

A multivariate data modelling problem consists of a number of nodes with associated function values. Increase in multivariance urges us to use divide-and-conquer algorithms in modelling process of these problems. High dimensional model representation based methods can partition a given multivariate data set into less-variate data sets and have the ability of building a model through these partitioned data sets. Generalized HDMR (GHDMR) is one of these methods and it is known that it works well for dominantly and purely additive natures. Piecewise Generalized HDMR is an alternative method and was developed to increase the efficiency of GHDMR but the performance of the method for modelling multiplicative natures is still not sufficient and acceptable. This work aims to develop a new piecewise method based on enhanced multivariance product representation which works well for representing multiplicative natures.     

Multivariate data modelling through Piecewise generalized HDMR method

This work aims to develop a new High Dimensional Model Representation (HDMR) based method which can construct an analytical structure for a given multivariate data modelling problem. Modelling multivariate data through a divide-and-conquer method stands for multivariate data partitioning process in which we deal with a number of less variate data sets instead of a single N dimensional problem. Generalized HDMR is one of these methods used to model a multivariate data set which has a number of scattered nodes with associated function values. However, Generalized HDMR includes a linear equation system with huge number of unknowns and equations to be solved. This equation sometimes has linearly dependent equations in it and this is an undesirable situation. This work offers a new method named Piecewise Generalized HDMR method which bypasses this disadvantage as well as reducing the mathematical complexity and CPU time needed to complete the algorithm of the previous method. Our new method splits the given problem domain into subdomains, applies the Generalized HDMR philosophy to each subdomain and superpositions the information coming from these subdomains. The algorithm of this new method and a number of numerical implementations are given in this paper.     

Hybrid HDMR method with an optimized hybridity parameter in multivariate function representation

High Dimensional Model Representation (HDMR) based methods are used to generate an approximation for a given multivariate function in terms of less variate functions. This paper focuses on Hybrid HDMR which is composed of Plain HDMR and Logarithmic HDMR. The Plain HDMR method works well for representing multivariate functions having additive nature. If the function under consideration has a multiplicative nature, then the Logarithmic HDMR method produces better approximation. Hybrid HDMR method aims to successfully represent a multivariate function having neither purely additive nor purely multiplicative nature under a hybridity parameter. The performance of the Hybrid HDMR method strongly depends on the value of this hybridity parameter because this parameter manages the contribution level of Plain and Logarithmic HDMR expansions. The main purpose of this work is to optimize the hybridity parameter to get the best approximations. Fluctuationlessness Approximation Theorem is used in this optimization process and in evaluating the multiple integrals appearing in HDMR based methods. A number of numerical implementations are given at the end of the paper to show the performance of our proposed method.     

Fluctuation free multivariate integration based logarithmic HDMR in multivariate function representation

This paper focuses on the Logarithmic High Dimensional Model Representation (Logarithmic HDMR) method which is a divide–and–conquer algorithm developed for multivariate function representation in terms of less-variate functions to reduce both the mathematical and the computational complexities. The main purpose of this work is to bypass the evaluation of N–tuple integrations appearing in Logarithmic HDMR by using the features of a new theorem named as Fluctuationlessness Approximation Theorem. This theorem can be used to evaluate the complicated integral structures of any scientific problem whose values can not be easily obtained analytically and it brings an approximation to the values of these integrals with the help of the matrix representation of functions. The Fluctuation Free Multivariate Integration Based Logarithmic HDMR method gives us the ability of reducing the complexity of the scientific problems of chemistry, physics, mathematics and engineering. A number of numerical implementations are also given at the end of the paper to show the performance of this new method.     

Constancy maximization based weight optimization in high dimensional model representation for multivariate functions

High Dimensional Model Representation (HDMR) method is a technique that represents a multivariate function in terms of less-variate functions. Even though the method has a finite expansion, to determine the components of this expansion is very expensive due to integration based natures of the components. Hence, the HDMR expansion is generally truncated at certain multivariance level and such approximations are produced to represent the given multivariate function approximately. The weight function selection becomes an important issue for the HDMR based applications when it is desired to give different importances to function values at different points. An appropriately chosen weight function may increase the quality of the approximation incredibly. This work aims at a multivariate weight function optimization to obtain high quality approximations through the HDMR method to represent multivariate functions. The proposed optimization considers constancy measurer maximization which produces a quadratic vector equation to be solved. Another contribution of this work is to use a recently developed method, fluctuation free integration, with HDMR, to solve this equation easily. This work is an extension of a previous work about weight optimization in HDMR for univariate functions.     

Estimation of molecular properties by high-dimensional model representation

Additivity models have been widely employed to approximate unknown molecular properties based on previously measured or calculated data for similar molecules. This paper proposes an improved formulation of additivity, which is based on high-dimensional model representation (HDMR). HDMR is a general function-mapping technique that expresses the output of a multivariate system in terms of a hierarchy of cooperative effects among its input variables. HDMR rests on the general observation that, for many physical systems, only relatively low-order input variable cooperativity is significant. A molecule is expressed as a multivariate system by defining binary-valued input variables corresponding to the presence or absence of a chemical bond, with the molecular property as the output. Conventional additivity decomposes a molecular property into contributions from nonoverlapping subcomponents of fixed size. On the other hand, HDMR decomposes a molecular property into the exact contributions from the full hierarchy of its variable-sized subcomponents and contains additivity as a special case. The complete hierarchical structure of HDMR can in many cases lead to a much more accurate estimate than conventional additivity. Also, when full group additivity is not possible, HDMR gives an expression for a lower-order approximation for the missing group additivity value, greatly expanding the scope of HDMR compared to additivity. The component terms in an HDMR approximation have well-defined physical significance. Moreover, HDMR gives an exact expression for the truncation error in any given HDMR approximation, also with a well-defined physical significance. The HDMR model is tested for the enthalpy of formation of a broad range of organic molecules, and its advantages over additivity are illustrated.     

Combined small scale high dimensional model representation

Nowadays the utilization of High Dimensional Model Representation (HDMR), which is an algorithm for approximating multivariate functions, is becoming more pervasive in the applications of approximation theory. This extensive usage motivates new works on HDMR, to get better solutions while approximating to the multivariate functions. One of them is recently developed “Combined Small Scale High Dimensional Model Representation (CSSHDMR)". This new scheme not only optimises HDMR results but also provides good approximation with less terms than HDMR does. This paper presents the theory and the numerical results of the new method and shows that it is possible to apply approximation to multivariate functions by keeping only constant term of HDMR. From this aspect CSSHDMR can be used in any scientific problem which includes multivariate functions, from chemistry to statistics.     

A novel piecewise multivariate function approximation method via universal matrix representation

Multivariance in science and engineering causes problematic situations even for continous and discrete cases. One way to overcome this situation is to decrease the multivariance level of the problem by using a divide—and—conquer based method. In this sense, Enhanced Multivariance Product Representation (EMPR) plays a part in the considered scenario and acts successfully. This method brings up a finite expansion to represent a multivariate function in terms of less-variate functions with the assistance of univariate support functions. This work aims to propose a new EMPR based algorithm which has two new features that improves the determination process of each expansion component through Fluctuation Free Integration method, which is an efficient method in evaluating multiple integrals through a universal matrix representation, and increases the approximation quality through inserting a piecewise structure into the standard EMPR algorithm. This new method is called Fluctuation Free Integration based piecewise EMPR. Some numerical implementations are also given to examine the performance of this proposed method.     

Analytical HDMR formulas for functions expressed as quadratic polynomials with a multivariate normal distribution

High Dimensional Model Representation (HDMR) is a general set of quantitative model assessment and analysis tools for systems with many variables. A general formulation for the HDMR component functions with independent and correlated variables was obtained previously. Since the HDMR component functions generally are coupled to one another and involve multi-dimensional integrals, explicit formulas for the component functions are not available for an arbitrary function with an arbitrary probability distribution amongst their variables. This paper presents analytical formulas for the HDMR component functions and the corresponding sensitivity indexes for the common case of a function expressed as a quadratic polynomial with a multivariate normal distribution over its variables. This advance is important for practical applications of HDMR with correlated variables.     

Efficient Implementation of High Dimensional Model Representations

Physical models of various phenomena are often represented by a mathematical model where the output(s) of interest have a multivariate dependence on the inputs. Frequently, the underlying laws governing this dependence are not known and one has to interpolate the mathematical model from a finite number of output samples. Multivariate approximation is normally viewed as suffering from the curse of dimensionality as the number of sample points needed to learn the function to a sufficient accuracy increases exponentially with the dimensionality of the function. However, the outputs of most physical systems are mathematically well behaved and the scarcity of the data is usually compensated for by additional assumptions on the function (i.e., imposition of smoothness conditions or confinement to a specific function space). High dimensional model representations (HDMR) are a particular family of representations where each term in the representation reflects the individual or cooperative contributions of the inputs upon the output. The main assumption of this paper is that for most well defined physical systems the output can be approximated by the sum of these hierarchical functions whose dimensionality is much smaller than the dimensionality of the output. This ansatz can dramatically reduce the sampling effort in representing the multivariate function. HDMR has a variety of applications where an efficient representation of multivariate functions arise with scarce data. The formulation of HDMR in this paper assumes that the data is randomly scattered throughout the domain of the output. Under these conditions and the assumptions underlying the HDMR it is argued that the number of samples needed for representation to a given tolerance is invariant to the dimensionality of the function, thereby providing for a very efficient means to perform high dimensional interpolation. Selected applications of HDMR's are presented from sensitivity analysis and time-series analysis.     

High dimensional model representation constructed by support vector regression. I. Independent variables with known probability distributions

High dimensional model representation (HDMR) is a general set of quantitative model assessment and analysis tools for capturing high dimensional input-output system behavior. In practice, the HDMR component functions are each approximated by an appropriate basis function expansion. This procedure often requires many input-output samples which can restrict the treatment of high dimensional systems. In order to address this problem we introduce svr-based HDMR to efficiently and effectively construct the HDMR expansion by support vector regression (SVR) for a function \(f(\mathbf{x})\). In this paper the results for independent variables sampled over known probability distributions are reported. The theoretical foundation of the new approach relies on the kernel used in SVR itself being an HDMR expansion (referred to as the HDMR kernel ), i.e., an ANOVA kernel whose component kernels are mutually orthogonal and all non-constant component kernels have zero expectation. Several HDMR kernels are constructed as illustrations. While preserving the characteristic properties of HDMR, the svr-based HDMR method enables efficient construction of high dimensional models with satisfactory prediction accuracy from a modest number of samples, which also permits accurate computation of the sensitivity indices. A genetic algorithm is employed to optimally determine all the parameters of the component HDMR kernels and in SVR. The svr-based HDMR introduces a new route to advance HDMR algorithms. Two examples are used to illustrate the capability of the method.     

A high dimensional model representation based numerical method for solving ordinary differential equations

A new numerical method for solving ordinary differential equations by using High Dimensional Model Representation (HDMR) has been developed in this work. Higher order ordinary differential equations can be reduced to a set of first order ODEs. Although HDMR is generally used for multivariate functions, univariate functions are taken into account throughout the work because of the ODEs’ natures. Not the numerical solution but its image under an appropriately chosen linear ordinary differential operator is expressed as a linear combination of the positive deviation powers of independent variable from its initial value. The linear combination of these image functions are expected to form a basis set under consideration. The unknown constants in the linear combination are found by maximizing the constancy measurer formed in terms of the HDMR components after they are evaluated. Results are compared with well-known step size based numerical methods. A semi qualitative error analysis of the proposed method is also established.     

The influence of the support functions on the quality of enhanced multivariance product representation

This paper presents recently developed Enhanced Multivariance Product Representation (EMPR) method for multivariate functions. EMPR disintegrates a multivariate function to components which are respectively constant, univariate, bivariate and so on in ascending multivariance. Although the EMPR method has the same philosophy with the High Dimensional Model Representation (HDMR) method, it has been proposed to get better quality than HDMR’s with the help of the support functions. For this purpose, we investigate the EMPR truncation qualities with respect to the selection of the support functions. The obtained results and a number of numerical implementations to show the efficiency of the method are also given in this paper.     

Automated pharmacophore identification for large chemical data sets.

The identification of three-dimensional pharmacophores from large, heterogeneous data sets is still an unsolved problem. We developed a novel program, SCAMPI (statistical classification of activities of molecules for pharmacophore identification), for this purpose by combining a fast conformation search with recursive partitioning, a data-mining technique, which can easily handle large data sets. The pharmacophore identification process is designed to run recursively, and the conformation spaces are resampled under the constraints of the evolving pharmacophore model. This program is capable of deriving pharmacophores from a data set of 1000-2000 compounds, with thousands of conformations generated for each compound and in less than 1 day of computational time. For two test data sets, the identified pharmacophores are consistent with the known results from the literature.     

Selection of orthogonal reversed-phase HPLC systems by univariate and auto-associative multivariate regression trees

In order to select chromatographic starting conditions to be optimized during further method development of the separation of a given mixture, so-called generic orthogonal chromatographic systems could be explored in parallel. In this paper the use of univariate and multivariate regression trees (MRT) was studied to define the most orthogonal subset from a given set of chromatographic systems. Two data sets were considered, which contain the retention data of 68 structurally diversive drugs on sets of 32 and 38 chromatographic systems, respectively. For both the univariate and multivariate approaches no other data but the measured retention factors are needed to build the decision trees. Since multivariate regression trees are used in an unsupervised way, they are called auto-associative multivariate regression trees (AAMRT). For all decision trees used, a variable importance list of the predictor variables can be derived. It was concluded that based on these ranked lists, both for univariate and multivariate regression trees, a selection of the most orthogonal systems from a given set of systems can be obtained in a user-friendly and fast way.     

Multicut-HDMR with an application to an ionospheric model

A new High Dimensional Model Representation (HDMR) tool, Multicut-HDMR, is introduced and applied to an ionospheric electron density model. HDMR is a general set of quantitative model assessment and analysis tools for improving the efficiency of deducing high-dimensional input-output system behavior. HDMR describes an output [f(x)] in terms of its input variables (x = [x(1), x(2), em leader, x(n)]) via a series of finite, hierarchical, correlated function expansions. Various forms of HDMR are constructed for different purposes such as modeling laboratory or field data, or reproducing a complicated mathematical model. The Cut-HDMR technique, which expresses f(x) with respect to a specified reference point x in the input space, is appropriate when the input space is sampled in an orderly fashion. However, if the desired domain of the input space is too large, the HDMR function expansion may not converge, and Cut-HDMR will be unable to accurately approximate f(x). The new Multicut-HDMR technique addresses this problem through the use of multiple reference points in the input space.     

Ensemble wavelet modelling for determination of wheat and gasoline properties by near and middle infrared spectroscopy

The wavelet transform has been shown to be a useful tool for multivariate calibration. However, the choice of wavelet transform settings (wavelet family, length and number of decomposition levels) for a given application is still an open problem. The present paper proposes an alternative approach, which consists of generating an ensemble model by combining individual models obtained with different wavelet transform settings. The advantages of the proposed method are demonstrated in two analytical problems, namely the determination of moisture and protein in wheat by near infrared spectroscopy and the determination of specific mass and three distillation temperatures (T10, T50, T90) in gasoline by middle infrared spectroscopy. In these problems, the results varied considerably among individual models, which underlines the risk associated to an inadequate choice of wavelet transform settings. In contrast, the ensemble model always provided adequate results in terms of prediction error and noise sensitivity. The proposed method can be seen as an advantageous alternative for multivariate calibration in the wavelet domain, as it frees the analyst from the need to choose a particular configuration for the wavelet transform.     

On the fundamental conjecture of HDMR: a Fourier analysis approach

Although the HDMR decomposition has become an important tool for the understanding of high dimensional functions, the fundamental conjecture underlying its practical utility is still open for theoretical analysis. In this paper, we introduce the HDMR decomposition in conjunction with the Fourier-HDMR approximation leading to the following conclusions: (1) we suggest a type of Fourier-HDMR approximation for certain classes of differentiable functions; (2) utilizing the Fourier-HDMR method, we prove the fundamental conjecture about the dominance of low order terms in the HDMR expansion under relevant conditions, and we also obtain error estimates of the truncated HDMR expansion up to order u; (3) we prove the domain decomposition approximation theorem which shows that the global Fourier-HDMR approximation is not always optimal for a given accuracy order; (4) and finally, a piecewise Fourier-HDMR approach is discussed for high dimensional modeling. These results help to further understand how to efficiently represent the high dimensional functions.     

Recursive distance partitioning algorithm for common pharmacophore identification

An improved method for exhaustively identifying common pharmacophores from a given list of 3D conformers is proposed. The method partitions feature lists into multidimensional boxes according to the distances between the pharmacophore centers. Unlike some existing techniques, each feature list is mapped into multiple boxes to ensure that good matches will never be missed due to the partitioning. To circumvent the computational complexity of the problem, a recursive distance partitioning (RDP) algorithm is introduced, in which the partitioning and the elimination of unqualified feature lists are carried out at multiple levels. The method is demonstrated to be both accurate and efficient.