Propagation effects of model-calculated probability values in Bayesian networks

Probabilistic causal interaction models have become quite popular among Bayesian-network engineers as elicitation of all probabilities required often proves the main bottleneck in building a real-world network with domain experts. The best-known interaction models are the noisy-OR model and its generalisations. These models in essence are parameterised conditional probability tables for which just a limited number of parameter probabilities are required. The models assume specific properties of intercausal interaction and cannot be applied uncritically. Given their clear engineering advantages however, they are subject to ill-considered use. This paper demonstrates that such ill-considered use can result in poorly calibrated output probabilities from a Bayesian network. By studying, in an analytical way, the propagation effects of noisy-OR calculated probability values, we identify conditions under which use of the model can be harmful for a network's performance. These conditions demonstrate that use of the noisy-OR model for mere pragmatic reasons is sometimes warranted, even when the model's underlying assumptions are not met in reality.     

Comparison of the Bayesian and Randomised Decision Tree Ensembles within an Uncertainty Envelope Technique

Multiple Classifier Systems (MCSs) allow evaluation of the uncertainty of classification outcomes that is of crucial importance for safety critical applications. The uncertainty of classification is determined by a trade-off between the amount of data available for training, the classifier diversity and the required performance. The interpretability of MCSs can also give useful information for experts responsible for making reliable classifications. For this reason Decision Trees (DTs) seem to be attractive classification models for experts. The required diversity of MCSs exploiting such classification models can be achieved by using two techniques, the Bayesian model averaging and the randomised DT ensemble. Both techniques have revealed promising results when applied to real-world problems. In this paper we experimentally compare the classification uncertainty of the Bayesian model averaging with a restarting strategy and the randomised DT ensemble on a synthetic dataset and some domain problems commonly used in the machine learning community. To make the Bayesian DT averaging feasible, we use a Markov Chain Monte Carlo technique. The classification uncertainty is evaluated within an Uncertainty Envelope technique dealing with the class posterior distribution and a given confidence probability. Exploring a full posterior distribution, this technique produces realistic estimates which can be easily interpreted in statistical terms. In our experiments we found out that the Bayesian DTs are superior to the randomised DT ensembles within the Uncertainty Envelope technique.     

Statistical merging of rating models

In this paper we introduce and discuss statistical models aimed at predicting default probabilities of Small and Medium Enterprises (SME). Such models are based on two separate sources of information: quantitative balance sheet ratios and qualitative information derived from the opinion mining process on unstructured data. We propose a novel methodology for data fusion in longitudinal and survival duration models using quantitative and qualitative variables separately in the likelihood function and then combining their scores linearly by a weight, to obtain the corresponding probability of default for each SME. With a real financial database at hand, we have compared the results achieved in terms of model performance and predictive capability using single models and our own proposal. Finally, we select the best model in terms of out-of-sample forecasts considering key performance indicators.     

Interaction Model and Model Selection for Function-on-Function Regression

Regression models with interaction effects have been widely used in multivariate analysis to improve model flexibility and prediction accuracy. In functional data analysis, however, due to the challenges of estimating three-dimensional coefficient functions, interaction effects have not been considered for function-on-function linear regression. In this article, we propose function-on-function regression models with interaction and quadratic effects. For a model with specified main and interaction effects, we propose an efficient estimation method that enjoys a minimum prediction error property and has good predictive performance in practice. Moreover, converting the estimation of three-dimensional coefficient functions of the interaction effects to the estimation of two- and one-dimensional functions separately, our method is computationally efficient. We also propose adaptive penalties to account for varying magnitudes and roughness levels of coefficient functions. In practice, the forms of the models are usually unspecified. We propose a stepwise procedure for model selection based on a predictive criterion. This method is implemented in our R package FRegSigComp. Supplemental materials are available online.     

Discrete time Markov chains with interval probabilities

The parameters of Markov chain models are often not known precisely. Instead of ignoring this problem, a better way to cope with it is to incorporate the imprecision into the models. This has become possible with the development of models of imprecise probabilities, such as the interval probability model. In this paper we discuss some modelling approaches which range from simple probability intervals to the general interval probability models and further to the models allowing completely general convex sets of probabilities. The basic idea is that precisely known initial distributions and transition matrices are replaced by imprecise ones, which effectively means that sets of possible candidates are considered. Consequently, sets of possible results are obtained and represented using similar imprecise probability models.We first set up the model and then show how to perform calculations of the distributions corresponding to the consecutive steps of a Markov chain. We present several approaches to such calculations and compare them with respect to the accuracy of the results. Next we consider a generalisation of the concept of regularity and study the convergence of regular imprecise Markov chains. We also give some numerical examples to compare different approaches to calculations of the sets of probabilities.     

Optimal inventory policy with supply uncertainty and demand cancellation

We consider a periodic review model where the firm manages its inventory under supply uncertainty and demand cancellation. We show that because of supply uncertainty, the optimal inventory policy has the structure of re-order point type. That is, we order if the initial inventory falls below this re-order point, otherwise we do not order. This is in contrast to the work of Yuan and Cheung (2003) who prove the optimality of an order up to policy in the absence of supply uncertainty. We also investigate the impact of supply uncertainty and demand cancellation on the performance of the supply chain. Using our model, we are able to quantify the importance of reducing the variance of either the distribution of yield or the distribution of demand cancellation. The single, multiple periods and the infinite horizon models are studied.     

Learning Interactions via Hierarchical Group-Lasso Regularization

We introduce a method for learning pairwise interactions in a linear regression or logistic regression model in a manner that satisfies strong hierarchy: whenever an interaction is estimated to be nonzero, both its associated main effects are also included in the model. We motivate our approach by modeling pairwise interactions for categorical variables with arbitrary numbers of levels, and then show how we can accommodate continuous variables as well. Our approach allows us to dispense with explicitly applying constraints on the main effects and interactions for identifiability, which results in interpretable interaction models. We compare our method with existing approaches on both simulated and real data, including a genome-wide association study, all using our R package glinternet.     

Random effects model for credit rating transitions

This paper proposes a random effects multinomial regression model to estimate transition probabilities of credit ratings. Unlike the previous studies on the rating transition, we applied a random effects model, which accommodates not only the environmental characteristics of the exposures of a rating but also the uncertainty not explained by such factors. The rating category specific factors such as retained earning and market equity are included in our proposed model. The random effects model provides less diagonally dominant matrix, where the transition probabilities are over-dispersed from the diagonal elements. Our study is expected to incorporate potential chances of rating transitions due to extra random variations.     

Context-specific independence in graphical log-linear models

Log-linear models are the popular workhorses of analyzing contingency tables. A log-linear parameterization of an interaction model can be more expressive than a direct parameterization based on probabilities, leading to a powerful way of defining restrictions derived from marginal, conditional and context-specific independence. However, parameter estimation is often simpler under a direct parameterization, provided that the model enjoys certain decomposability properties. Here we introduce a cyclical projection algorithm for obtaining maximum likelihood estimates of log-linear parameters under an arbitrary context-specific graphical log-linear model, which needs not satisfy criteria of decomposability. We illustrate that lifting the restriction of decomposability makes the models more expressive, such that additional context-specific independencies embedded in real data can be identified. It is also shown how a context-specific graphical model can correspond to a non-hierarchical log-linear parameterization with a concise interpretation. This observation can pave way to further development of non-hierarchical log-linear models, which have been largely neglected due to their believed lack of interpretability.     

A Bayesian network approach to making inferences in causal maps

The main goal of this paper is to describe a new graphical structure called ‘Bayesian causal maps’ to represent and analyze domain knowledge of experts. A Bayesian causal map is a causal map, i.e., a network-based representation of an expert’s cognition. It is also a Bayesian network, i.e., a graphical representation of an expert’s knowledge based on probability theory. Bayesian causal maps enhance the capabilities of causal maps in many ways. We describe how the textual analysis procedure for constructing causal maps can be modified to construct Bayesian causal maps, and we illustrate it using a causal map of a marketing expert in the context of a product development decision.     

Predicting Web Users' Next Access Based on Log Data

This article considers models that describe how people browse the Web. We restrict our attention to navigation patterns within a single site, and base our study on standard Web server access logs. Given a visitor's previous activities on the site, we propose models that predict their next page request. If the prediction is reasonably accurate, we might consider “prefetching” the page before the visitor requests it. A more conservative use for such predictions would be to simply update the freshness records in a proxy or network cache, eliminating unnecessary If-Modified-Since requests. Using data from the Web site for the Computing and Mathematical Sciences Research Division of Lucent Technologies (cm.bell-labs.com) we first evaluate the predictive performance of low-order Markov models. We next consider mixtures of first-order Markov models, achieving a kind of clustering of Web pages in the site. This approach is shown to perform well, while significantly reducing the space required to store the model. Finally, we explore a Bayesian approach using a Dirichlet prior on the collection of links available to a user at each stage in their travels through the site. We show that the posterior probabilities derived under this model are fairly close to the cross-validation estimates of the probability of success.     

Maximising entropy on the nonparametric predictive inference model for multinomial data

The combination of mathematical models and uncertainty measures can be applied in the area of data mining for diverse objectives with as final aim to support decision making. The maximum entropy function is an excellent measure of uncertainty when the information is represented by a mathematical model based on imprecise probabilities. In this paper, we present algorithms to obtain the maximum entropy value when the information available is represented by a new model based on imprecise probabilities: the nonparametric predictive inference model for multinomial data (NPI-M), which represents a type of entropy-linear program. To reduce the complexity of the model, we prove that the NPI-M lower and upper probabilities for any general event can be expressed as a combination of the lower and upper probabilities for the singleton events, and that this model can not be associated with a closed polyhedral set of probabilities. An algorithm to obtain the maximum entropy probability distribution on the set associated with NPI-M is presented. We also consider a model which uses the closed and convex set of probability distributions generated by the NPI-M singleton probabilities, a closed polyhedral set. We call this model A-NPI-M. A-NPI-M can be seen as an approximation of NPI-M, this approximation being simpler to use because it is not necessary to consider the set of constraints associated with the exact model.     

Discriminant Analysis for Regression Models with Stationary Long-Memory Disturbances

We shall consider the problems of classifying an observation from regression model with stationary long-memory or short-memory disturbances into one of two populations described by the mean functions of the model. We use the log-likelihood ratio as a discrimant statistic which is optimal in the sense of its minimizing the misclassification probabilities. Then we confirm the theoretical results by some simple polynomial regression models.     

Identifiability of intermediate variables on causal paths

We discuss the discovery of causal mechanisms and identifiability of intermediate variables on a causal path. Different from variable selection, we try to distinguish intermediate variables on the causal path from other variables. It is also different from ordinary model selection approaches which do not concern the causal relationships and do not contain unobserved variables. We propose an approach for selecting a causal mechanism depicted by a directed acyclic graph (DAG) with an unobserved variable. We consider several causal networks, and discuss their identifiability by observed data. We show that causal mechanisms of linear structural equation models are not identifiable. Furthermore, we present that causal mechanisms of nonlinear models are identifiable, and we demonstrate the identifiability of causal mechanisms of quadratic equation models. Sensitivity analysis is conducted for the identifiability.     

Operational risk modelling and organizational learning in structured finance operations: a Bayesian network approach

This paper describes the development of a tool, based on a Bayesian network model, that provides posteriori predictions of operational risk events, aggregate operational loss distributions, and Operational Value-at-Risk, for a structured finance operations unit located within one of Australia's major banks. The Bayesian network, based on a previously developed causal framework, has been designed to model the smaller and more frequent, attritional operational loss events. Given the limited availability of risk factor event information and operational loss data, we rely on the elicitation of subjective probabilities, sourced from domain experts. Parameter sensitivity analysis is performed to validate and check the model's robustness against the beliefs of risk management and operational staff. To ensure that the domain's evolving risk profile is captured through time, a formal approach to organizational learning is investigated that employs the automatic parameter adaption features of the Bayesian network model. A hypothetical case study is then described to demonstrate model adaption and the application of the tool to operational loss forecasting by a business unit risk manager.     

From behavioural to population level: Growth and competition

The aim of this work is to build models of population dynamics for growth and competition interaction by starting with detailed models at the individual level. At the individual level, we start with detailed models where the growth is described by linear terms. By considering individual interferences and by using aggregation methods, we show that the population level, different growth equation can emerge. We present an example of the emergence of logistic growth and an example of the emergence of logistic growth with Allee effect. Furthermore, in the case of two populations, we show that individual interferences can lead at the population level, to a model which has the same qualitative dynamics behaviour as the Lotka-Volterra competition model. Finally, we show that our model brings to light the effects of spatial heterogeneity on competition models. First, we find the stabilizing effects but also we show that destabilizing effects can occur.     

Bayesian Variable and Transformation Selection in Linear Regression

This article suggests a method for variable and transformation selection based on posterior probabilities. Our approach allows for consideration of all possible combinations of untransformed and transformed predictors along with transformed and untransformed versions of the response. To transform the predictors in the model, we use a change-point model, or “change-point transformation,” which can yield more interpretable models and transformations than the standard Box–Tidwell approach. We also address the problem of model uncertainty in the selection of models. By averaging over models, we account for the uncertainty inherent in inference based on a single model chosen from the set of models under consideration. We use a Markov chain Monte Carlo model composition (MC³) method which allows us to average over linear regression models when the space of models under consideration is very large. This considers the selection of variables and transformations at the same time. In an example, we show that model averaging improves predictive performance as compared with any single model that might reasonably be selected, both in terms of overall predictive score and of the coverage of prediction intervals. Software to apply the proposed methodology is available via StatLib.     

The cross-entropy method with patching for rare-event simulation of large Markov chains

There are various importance sampling schemes to estimate rare event probabilities in Markovian systems such as Markovian reliability models and Jackson networks. In this work, we present a general state-dependent importance sampling method which partitions the state space and applies the cross-entropy method to each partition. We investigate two versions of our algorithm and apply them to several examples of reliability and queueing models. In all these examples we compare our method with other importance sampling schemes. The performance of the importance sampling schemes is measured by the relative error of the estimator and by the efficiency of the algorithm. The results from experiments show considerable improvements both in running time of the algorithm and the variance of the estimator.     

Exponential family non-linear models for categorical data with errors of observation

We present a general framework for treating categorical data with errors of observation. We show how both latent class models and models for doubly sampled data can be treated as exponential family nonlinear models. These are extended generalized linear models with the link function substituted by an observationwise defined non-linear function of the model parameters. The models are formulated in terms of structural probabilities and conditional error probabilities, thus allowing natural constraints when modelling errors of observation. We use an iteratively reweighted least squares procedure for obtaining maximum likelihood estimates. This is faster than the traditionally used EM algorithm and the computations can be made in GLIM.¹ As examples we analyse three sets of categorical data with errors of observation which have been analysed before by Ashford and Sowden,² Goodman³ and Chen,⁴ respectively.     

Electroweak Nucleon Decays in a Superstrong Magnetic Field

We study the influence of a magnetic field on the electroweak processes of nucleon decay in a degenerate ideal gas of neutrons, protons, and electrons situated in an external superstrong constant and homogeneous magnetic field with effects due to the interaction of nucleon anomalous magnetic moments with the magnetic field taken into account. For different values of the chemical potentials of degenerate fermions, we obtain expressions for probabilities of electroweak processes, which are assumed to be responsible for the chemical equilibrium in the central domain of a neutron star with a frozen superstrong magnetic field. We show that the difference between the neutron decay probabilities in the presence of a magnetic field B ≪ 10¹⁷ G and without this field is completely determined by changing the phase volume of electron states. We discuss the process of proton decay into a neutron, positron, and neutrino. This process is energetically allowed only when the interaction of nucleon anomalous magnetic moments with a superstrong magnetic field is taken into account. __________ Translated from Teoreticheskaya i Matematicheskaya Fizika, Vol. 145, No. 1, pp. 108–122, October, 2005.