Information Field Theory and Artificial Intelligence

Information field theory (IFT), the information theory for fields, is a mathematical framework for signal reconstruction and non-parametric inverse problems. Artificial intelligence (AI) and machine learning (ML) aim at generating intelligent systems, including such for perception, cognition, and learning. This overlaps with IFT, which is designed to address perception, reasoning, and inference tasks. Here, the relation between concepts and tools in IFT and those in AI and ML research are discussed. In the context of IFT, fields denote physical quantities that change continuously as a function of space (and time) and information theory refers to Bayesian probabilistic logic equipped with the associated entropic information measures. Reconstructing a signal with IFT is a computational problem similar to training a generative neural network (GNN) in ML. In this paper, the process of inference in IFT is reformulated in terms of GNN training. In contrast to classical neural networks, IFT based GNNs can operate without pre-training thanks to incorporating expert knowledge into their architecture. Furthermore, the cross-fertilization of variational inference methods used in IFT and ML are discussed. These discussions suggest that IFT is well suited to address many problems in AI and ML research and application.     

Hierarchical Bayesian inference for Ill-posed problems via variational method

This paper investigates a novel approximate Bayesian inference procedure for numerically solving inverse problems. A hierarchical formulation which determines automatically the regularization parameter and the noise level together with the inverse solution is adopted. The framework is of variational type, and it can deliver the inverse solution and regularization parameter together with their uncertainties calibrated. It approximates the posteriori probability distribution by separable distributions based on Kullback–Leibler divergence. Two approximations are derived within the framework, and some theoretical properties, e.g. variance estimate and consistency, are also provided. Algorithms for their efficient numerical realization are described, and their convergence properties are also discussed. Extensions to nonquadratic regularization/nonlinear forward models are also briefly studied. Numerical results for linear and nonlinear Cauchy-type problems arising in heat conduction with both smooth and nonsmooth solutions are presented for the proposed method, and compared with that by Markov chain Monte Carlo. The results illustrate that the variational method can faithfully capture the posteriori distribution in a computationally efficient way.     

Fractional-order TV-L2 model for image denoising

This paper proposes a new fractional order total variation (TV) denoising method, which provides a much more elegant and effective way of treating problems of the algorithm implementation, ill-posed inverse, regularization parameter selection and blocky effect. Two fractional order TV-L² models are constructed for image denoising. The majorization-minimization (MM) algorithm is used to decompose these two complex fractional TV optimization problems into a set of linear optimization problems which can be solved by the conjugate gradient algorithm. The final adaptive numerical procedure is given. Finally, we report experimental results which show that the proposed methodology avoids the blocky effect and achieves state-of-the-art performance. In addition, two medical image processing experiments are presented to demonstrate the validity of the proposed methodology.     

A Quantitative Comparison between Shannon and Tsallis–Havrda–Charvat Entropies Applied to Cancer Outcome Prediction

In this paper, we propose to quantitatively compare loss functions based on parameterized Tsallis–Havrda–Charvat entropy and classical Shannon entropy for the training of a deep network in the case of small datasets which are usually encountered in medical applications. Shannon cross-entropy is widely used as a loss function for most neural networks applied to the segmentation, classification and detection of images. Shannon entropy is a particular case of Tsallis–Havrda–Charvat entropy. In this work, we compare these two entropies through a medical application for predicting recurrence in patients with head–neck and lung cancers after treatment. Based on both CT images and patient information, a multitask deep neural network is proposed to perform a recurrence prediction task using cross-entropy as a loss function and an image reconstruction task. Tsallis–Havrda–Charvat cross-entropy is a parameterized cross-entropy with the parameter α . Shannon entropy is a particular case of Tsallis–Havrda–Charvat entropy for α=1 . The influence of this parameter on the final prediction results is studied. In this paper, the experiments are conducted on two datasets including in total 580 patients, of whom 434 suffered from head–neck cancers and 146 from lung cancers. The results show that Tsallis–Havrda–Charvat entropy can achieve better performance in terms of prediction accuracy with some values of α .     

An efficient method for model refinement in diffuse optical tomography

Diffuse optical tomography (DOT) is a non-linear, ill-posed, boundary value and optimization problem which necessitates regularization. Also, Bayesian methods are suitable owing to measurements data are sparse and correlated. In such problems which are solved with iterative methods, for stabilization and better convergence, the solution space must be small. These constraints subject to extensive and overdetermined system of equations which model retrieving criteria specially total least squares (TLS) must to refine model error. Using TLS is limited to linear systems which is not achievable when applying traditional Bayesian methods. This paper presents an efficient method for model refinement using regularized total least squares (RTLS) for treating on linearized DOT problem, having maximum a posteriori (MAP) estimator and Tikhonov regulator. This is done with combination Bayesian and regularization tools as preconditioner matrices, applying them to equations and then using RTLS to the resulting linear equations. The preconditioning matrixes are guided by patient specific information as well as a priori knowledge gained from the training set. Simulation results illustrate that proposed method improves the image reconstruction performance and localize the abnormally well.     

Designing neural networks that process mean values of random variables

We develop a class of neural networks derived from probabilistic models posed in the form of Bayesian networks. Making biologically and technically plausible assumptions about the nature of the probabilistic models to be represented in the networks, we derive neural networks exhibiting standard dynamics that require no training to determine the synaptic weights, that perform accurate calculation of the mean values of the relevant random variables, that can pool multiple sources of evidence, and that deal appropriately with ambivalent, inconsistent, or contradictory evidence.     

Computation of Kullback–Leibler Divergence in Bayesian Networks

Kullback–Leibler divergence KL(p,q) is the standard measure of error when we have a true probability distribution p which is approximate with probability distribution q. Its efficient computation is essential in many tasks, as in approximate computation or as a measure of error when learning a probability. In high dimensional probabilities, as the ones associated with Bayesian networks, a direct computation can be unfeasible. This paper considers the case of efficiently computing the Kullback–Leibler divergence of two probability distributions, each one of them coming from a different Bayesian network, which might have different structures. The paper is based on an auxiliary deletion algorithm to compute the necessary marginal distributions, but using a cache of operations with potentials in order to reuse past computations whenever they are necessary. The algorithms are tested with Bayesian networks from the bnlearn repository. Computer code in Python is provided taking as basis pgmpy, a library for working with probabilistic graphical models.     

基于Tikhonov正则化特征光谱选择与最优网络参数选择的轻烷烃气体分析

特征变量选取与模型的建立是光谱定量分析的两个主要研究内容.首先讨论了Tikhonov正则化特征光谱选取算法在多组分烷烃气体分析应用中的参数确定方法,然后针对甲烷、乙烷、丙烷、异丁烷、正丁烷、异戊烷和正戊烷七种烃烷的小浓度分析,从中红外吸收光谱中提取了七组特征光谱,并用这些特征光谱作为输出,用神经网络建立了七种烃烷气体的...     

基于小波变换和非负支撑域递归逆滤波算法的盲目图像复原

提出了一种将自适应正则化方法与非负支撑域递归逆滤波(NAS-RIF)算法相结合用于小波域的盲图像复原算法.该算法先对降质图像进行小波分解,得到了图像在不同子频段的信息.在各个子频段采用NAS-RIF算法进行复原.针对各个子频段内图像的频率和方向特性,分别引入了不同的正则化约束项.在各个子频段估计出噪声方差,提出了根据噪声方差和图像局部方差来选取正则化参数.分别对两幅模糊图像进行了仿真实验,复原结果取得的信噪比分别为19.66 dB和23.86 dB.实验结果表明,复原效果相对于空间自适应正则化方法有一定的提高.     

Normalized Augmented Inverse Probability Weighting with Neural Network Predictions

The estimation of average treatment effect (ATE) as a causal parameter is carried out in two steps, where in the first step, the treatment and outcome are modeled to incorporate the potential confounders, and in the second step, the predictions are inserted into the ATE estimators such as the augmented inverse probability weighting (AIPW) estimator. Due to the concerns regarding the non-linear or unknown relationships between confounders and the treatment and outcome, there has been interest in applying non-parametric methods such as machine learning (ML) algorithms instead. Some of the literature proposes to use two separate neural networks (NNs) where there is no regularization on the network’s parameters except the stochastic gradient descent (SGD) in the NN’s optimization. Our simulations indicate that the AIPW estimator suffers extensively if no regularization is utilized. We propose the normalization of AIPW (referred to as nAIPW) which can be helpful in some scenarios. nAIPW, provably, has the same properties as AIPW, that is, the double-robustness and orthogonality properties. Further, if the first-step algorithms converge fast enough, under regulatory conditions, nAIPW will be asymptotically normal. We also compare the performance of AIPW and nAIPW in terms of the bias and variance when small to moderate L1 regularization is imposed on the NNs.     

On Architecture Selection for Linear Inverse Problems with Untrained Neural Networks

In recent years, neural network based image priors have been shown to be highly effective for linear inverse problems, often significantly outperforming conventional methods that are based on sparsity and related notions. While pre-trained generative models are perhaps the most common, it has additionally been shown that even untrained neural networks can serve as excellent priors in various imaging applications. In this paper, we seek to broaden the applicability and understanding of untrained neural network priors by investigating the interaction between architecture selection, measurement models (e.g., inpainting vs. denoising vs. compressive sensing), and signal types (e.g., smooth vs. erratic). We motivate the problem via statistical learning theory, and provide two practical algorithms for tuning architectural hyperparameters. Using experimental evaluations, we demonstrate that the optimal hyperparameters may vary significantly between tasks and can exhibit large performance gaps when tuned for the wrong task. In addition, we investigate which hyperparameters tend to be more important, and which are robust to deviations from the optimum.     

Demosaicking for multispectral images based on vectorial total variation

Multispectral images (MSIs), which consist of more color components than RGB images, can be used in the field of vegetation analysis and medical imaging. A capturing system with multispectral filter array (MSFA) technology has been researched to shorten the capturing time and reduce the cost. In this system, the mosaicked image captured by the MSFA is demosaicked to reconstruct the MSI. We propose a demosaicking method using vectorial total variation (VTV) regularization for an MSI. This process is regarded as inverse problem of the image observation model. The reconstructed image is estimated by minimizing the VTV as a regularization term under the constraint condition. In the experimental results, the reconstructed image quality obtained using the proposed method is better than that of the conventional approaches in terms of both peak signal-to-noise ratio and structural similarity.     

Sparsity-constrained SENSE reconstruction: An efficient implementation using a fast composite splitting algorithm

Parallel imaging and compressed sensing have been arguably the most successful and widely used techniques for fast magnetic resonance imaging (MRI). Recent studies have shown that the combination of these two techniques is useful for solving the inverse problem of recovering the image from highly under-sampled k-space data. In sparsity-enforced sensitivity encoding (SENSE) reconstruction, the optimization problem involves data fidelity (L2-norm) constraint and a number of L1-norm regularization terms (i.e. total variation or TV, and L1 norm). This makes the optimization problem difficult to solve due to the non-smooth nature of the regularization terms. In this paper, to effectively solve the sparsity-regularized SENSE reconstruction, we utilize a new optimization method, called fast composite splitting algorithm (FCSA), which was developed for compressed sensing MRI. By using a combination of variable splitting and operator splitting techniques, the FCSA algorithm decouples the large optimization problem into TV and L1 sub-problems, which are then, solved efficiently using existing fast methods. The operator splitting separates the smooth terms from the non-smooth terms, so that both terms are treated in an efficient manner. The final solution to the SENSE reconstruction is obtained by weighted solutions to the sub-problems through an iterative optimization procedure. The FCSA-based parallel MRI technique is tested on MR brain image reconstructions at various acceleration rates and with different sampling trajectories. The results indicate that, for sparsity-regularized SENSE reconstruction, the FCSA-based method is capable of achieving significant improvements in reconstruction accuracy when compared with the state-of-the-art reconstruction method.     

Bayesian wavelet-based analysis of functional magnetic resonance time series

Wavelet methods for image regularization offer a data-driven alternative to Gaussian smoothing in functional magnetic resonance (fMRI) analysis. Their impact has been limited by the difficulties in integrating regularization in the wavelet domain and inference in the image domain, precluding the probabilistic decision on which areas are activated by a task. Here we present an integrated framework for Bayesian estimation and regularization in wavelet space that allows the usual voxelwise hypothesis testing. This framework is flexible, being an adaptation to fMRI time series of a more general wavelet-based functional mixed-effect model. Through testing on a combination of simulated and real fMRI data, we show evidence of improved signal recovery, without compromising test accuracy in image space.     

Inverse Eigenvalue Problems for Exploring the Dynamics of Systems Biology Models

This paper describes inverse eigenvalue problems that arise in studying qualitative dynamics in systems biology models. An algorithm based on lift-and-project iterations is proposed, where the lifting step entails solving a constrained matrix inverse eigenvalue problem. In particular, prior to carrying out the iterative steps, $a$-$priori$ bounds on the entries of the Jacobian matrix are computed by relying on the reaction network structure as well as the form of the rate law expressions for the model under consideration. Numerical results on a number of models show that the proposed algorithm can be used to computationally explore the possible dynamical scenarios while identifying the important mechanisms via the use of sparsity-promoting regularization.     

Suitable features selection for monitoring thermal condition of electrical equipment using infrared thermography

Monitoring the thermal condition of electrical equipment is necessary for maintaining the reliability of electrical system. The degradation of electrical equipment can cause excessive overheating, which can lead to the eventual failure of the equipment. Additionally, failure of equipment requires a lot of maintenance cost, manpower and can also be catastrophic- causing injuries or even deaths. Therefore, the recognition processof equipment conditions as normal and defective is an essential step towards maintaining reliability and stability of the system. The study introduces infrared thermography based condition monitoring of electrical equipment. Manual analysis of thermal image for detecting defects and classifying the status of equipment take a lot of time, efforts and can also lead to incorrect diagnosis results. An intelligent system that can separate the equipment automatically could help to overcome these problems. This paper discusses an intelligent classification system for the conditions of equipment using neural networks. Three sets of features namely first order histogram based statistical, grey level co-occurrence matrix and component based intensity features are extracted by image analysis, which are used as input data for the neural networks. The multilayered perceptron networks are trained using four different training algorithms namely Resilient back propagation, Bayesian Regulazation, Levenberg–Marquardt and Scale conjugate gradient. The experimental results show that the component based intensity features perform better compared to other two sets of features. Finally, after selecting the best features, multilayered perceptron network trained using Levenberg–Marquardt algorithm achieved the best results to classify the conditions of electrical equipment.     

Unmanned aerial vehicles assisted rice seedling detection using shark smell optimization with deep learning model

Rice seedling classification using an unmanned aerial vehicle (UAV) images remains a challenging problem that needs to be addressed. It is still a difficult task because it is prone to low temporal and spatial resolution images. Recently, machine learning (ML) and deep learning (DL) models can be employed for several image preprocessing tasks such as classification, object detection, and segmentation. Therefore, this study focuses on the design of shark smell optimization with deep learning based rice seedling detection (SSODL-RSD) on UAV imagery. The presented SSODL-RSD technique recognizes the UAV images into arable land and rice seedlings. To achieve this, the SSODL-RSD technique employs the adaptive Wiener filtering (AWF) technique for the noise removal procedure. In addition, the SSODL-RSD technique exploits the NestNet feature extractor model. Moreover, the SSO algorithm is used for the hyperparameter tuning of the NestNet model. Finally, the long short term memory-recurrent neural network (LSTM-RNN) model is employed for the classification of rice seedlings. The extensive comparative study highlighted the improved outcomes of the SSODL-RSD technique over other existing models.     

基于贝叶斯准则的闪光照相图像重建

 针对闪光照相图像低信噪比的特点,研究了一种带约束的贝叶斯图像重建算法,较好地抑制了噪声对重建结果的影响,同时对细节信号具有较高的保持能力。该算法以贝叶斯重建为基础,在重建过程中考虑了闪光照相图像的受模糊影响比较严重的特点,引入了重建结果具有平滑性的先验信息进行约束,以提高重建结果的信噪比。通过在迭代过程中对重建结果进行平滑性约束以提高重建质量。数值模拟结果表明,该方法在闪光照相图像重建中具有较好的抗噪能力和边界保持能力。