On the Nature of Functional Differentiation: The Role of Self-Organization with Constraints

The focus of this article is the self-organization of neural systems under constraints. In 2016, we proposed a theory for self-organization with constraints to clarify the neural mechanism of functional differentiation. As a typical application of the theory, we developed evolutionary reservoir computers that exhibit functional differentiation of neurons. Regarding the self-organized structure of neural systems, Warren McCulloch described the neural networks of the brain as being “heterarchical”, rather than hierarchical, in structure. Unlike the fixed boundary conditions in conventional self-organization theory, where stationary phenomena are the target for study, the neural networks of the brain change their functional structure via synaptic learning and neural differentiation to exhibit specific functions, thereby adapting to nonstationary environmental changes. Thus, the neural network structure is altered dynamically among possible network structures. We refer to such changes as a dynamic heterarchy. Through the dynamic changes of the network structure under constraints, such as physical, chemical, and informational factors, which act on the whole system, neural systems realize functional differentiation or functional parcellation. Based on the computation results of our model for functional differentiation, we propose hypotheses on the neuronal mechanism of functional differentiation. Finally, using the Kolmogorov–Arnold–Sprecher superposition theorem, which can be realized by a layered deep neural network, we propose a possible scenario of functional (including cell) differentiation.     

Modélisation de séchoirs à tapis. Utilisation des réseaux de neurones

Modelling of convective layer dryers. Using neural networks. Dryer modelling is considered in this paper. A dryer scale approach is implemented in order to write the classical differential equations through parameters such as the heat transfer coefficient or drying kinetics. The behaviour of the dryers is described by a non-linear system which integrates these equations in a transfer network using the finite difference method. The finite difference method is easy to implement, but appears to be too slow for dryer designing. So, in the second part of the study, neural networks are used to model drying process in steady state. When applying neural networks method to the design of dryers, one of the main problems is to find necessary and sufficient inputs so that the neural networks can learn transfers laws. To reduce the problem, each output is defined by a single neural network and non-dimensional numbers are used. The following step deals with the determination of the number of neurones and the minimization of output error for each efficiency (change of training points). Then, neural networks are used to simulate different configurations of dryers. Results are compared with the finite difference method and an industrial application is studied in the last chapter.     

Predicting and optimising the airborne sound transmission of floor-ceiling constructions using computational intelligence

Computational intelligence (CI) techniques offer powerful alternatives for investigating acoustical issues and providing acoustical solutions to problems. This paper presents information on two CI techniques by applying them to the sound transmission performance prediction and design of floor-ceiling constructions.First a simple neural network (NN) model for predicting the airborne sound transmission of typical floor-ceiling constructions is presented and explained in detail. This model is accessible to researchers with knowledge of neural network analysis (NNA) for further sophistication, specialisation or hybridisation. The model may also be used by architects and others with no knowledge of NNA and no access to any specialised neural network software. Evolutionary algorithms (EAs) were then applied to search the multidimensional space created by the neural network model in order to optimise the airborne sound transmission of floor-ceiling constructions within the range of design parameters utilised in buildings.     

Prisoner’s Dilemma Game with Cooperation-Defection Dominance Strategies on Correlational Multilayer Networks

As multilayer networks are widely applied in modern society, numerous studies have shown the impact of a multilayer network structure and the network nature on the proportion of cooperators in the network. In this paper, we use Barabási–Albert scale-free networks (BA) and Watts and Strogatz networks (WS) to build a multilayer network structure, and we propose a new strategy-updating rule called “cooperation-defection dominance”, which can be likened to dominant and recessive traits in biogenetics. With the newly constructed multilayer network structure and the strategy-updating rules, based on the simulation results, we find that in the BA-BA network, the cooperation dominance strategy can make the networks with different rs show a cooperative trend, while the defection dominance strategy only has an obvious effect on the network cooperation with a larger r. When the BA network is connected to the WS network, we find that the effect of strategy on the proportion of cooperators in the network decreases, and the main influencing factor is the structure of the network. In the three-layer network, the cooperation dominance strategy has a greater impact on the BA network, and the proportion of the cooperators is enhanced more than under the natural evolution strategy, but the promotion effect is still smaller than that of the two-layer BA network because of the WS network. Under the defection dominance strategy, the WS layer appears different from the first two strategies, and we conclude through simulation that when the payoff parameter is at the middle level, its cooperator proportion will be suppressed, and we deduce that the proportion of cooperators and defectors, as well as the payoff, play an important role.     

Statistical physics and practical training of soft-committee machines

Equilibrium states of large layered neural networks with differentiable activation function and a single, linear output unit are investigated using the replica formalism. The quenched free energy of a student network with a very large number of hidden units learning a rule of perfectly matching complexity is calculated analytically. The system undergoes a first order phase transition from unspecialized to specialized student configurations at a critical size of the training set. Computer simulations of learning by stochastic gradient descent from a fixed training set demonstrate that the equilibrium results describe quantitatively the plateau states which occur in practical training procedures at sufficiently small but finite learning rates. Received 16 December 1998     

Multi-task learning approach for modulation and wireless signal classification for 5G and beyond: Edge deployment via model compression

Future communication networks must address the scarce spectrum to accommodate extensive growth of heterogeneous wireless devices. Efforts are underway to address spectrum coexistence, enhance spectrum awareness, and bolster authentication schemes. Wireless signal recognition is becoming increasingly more significant for spectrum monitoring, spectrum management, secure communications, among others. Consequently, comprehensive spectrum awareness on the edge has the potential to serve as a key enabler for the emerging beyond 5G (fifth generation) networks. State-of-the-art studies in this domain have (i) only focused on a single task – modulation or signal (protocol) classification – which in many cases is insufficient information for a system to act on, (ii) consider either radar or communication waveforms (homogeneous waveform category), and (iii) does not address edge deployment during neural network design phase. In this work, for the first time in the wireless communication domain, we exploit the potential of deep neural networks based multi-task learning (MTL) framework to simultaneously learn modulation and signal classification tasks while considering heterogeneous wireless signals such as radar and communication waveforms in the electromagnetic spectrum. The proposed MTL architecture benefits from the mutual relation between the two tasks in improving the classification accuracy as well as the learning efficiency with a lightweight neural network model. We additionally include experimental evaluations of the model with over-the-air collected samples and demonstrate first-hand insight on model compression along with deep learning pipeline for deployment on resource-constrained edge devices. We demonstrate significant computational, memory, and accuracy improvement of the proposed model over two reference architectures. In addition to modeling a lightweight MTL model suitable for resource-constrained embedded radio platforms, we provide a comprehensive heterogeneous wireless signals dataset for public use.     

Learning to Classify DWDM Optical Channels from Tiny and Imbalanced Data

Applying machine learning algorithms for assessing the transmission quality in optical networks is associated with substantial challenges. Datasets that could provide training instances tend to be small and heavily imbalanced. This requires applying imbalanced compensation techniques when using binary classification algorithms, but it also makes one-class classification, learning only from instances of the majority class, a noteworthy alternative. This work examines the utility of both these approaches using a real dataset from a Dense Wavelength Division Multiplexing network operator, gathered through the network control plane. The dataset is indeed of a very small size and contains very few examples of “bad” paths that do not deliver the required level of transmission quality. Two binary classification algorithms, random forest and extreme gradient boosting, are used in combination with two imbalance handling methods, instance weighting and synthetic minority class instance generation. Their predictive performance is compared with that of four one-class classification algorithms: One-class SVM, one-class naive Bayes classifier, isolation forest, and maximum entropy modeling. The one-class approach turns out to be clearly superior, particularly with respect to the level of classification precision, making it possible to obtain more practically useful models.     

Thermodynamic State Machine Network

We describe a model system—a thermodynamic state machine network—comprising a network of probabilistic, stateful automata that equilibrate according to Boltzmann statistics, exchange codes over unweighted bi-directional edges, update a state transition memory to learn transitions between network ground states, and minimize an action associated with fluctuation trajectories. The model is grounded in four postulates concerning self-organizing, open thermodynamic systems—transport-driven self-organization, scale-integration, input-functionalization, and active equilibration. After sufficient exposure to periodically changing inputs, a diffusive-to-mechanistic phase transition emerges in the network dynamics. The evolved networks show spatial and temporal structures that look much like spiking neural networks, although no such structures were incorporated into the model. Our main contribution is the articulation of the postulates, the development of a thermodynamically motivated methodology addressing them, and the resulting phase transition. As with other machine learning methods, the model is limited by its scalability, generality, and temporality. We use limitations to motivate the development of thermodynamic computers—engineered, thermodynamically self-organizing systems—and comment on efforts to realize them in the context of this work. We offer a different philosophical perspective, thermodynamicalism, addressing the limitations of the model and machine learning in general.     

Integrated Information in the Spiking–Bursting Stochastic Model

Integrated information has been recently suggested as a possible measure to identify a necessary condition for a system to display conscious features. Recently, we have shown that astrocytes contribute to the generation of integrated information through the complex behavior of neuron–astrocyte networks. Still, it remained unclear which underlying mechanisms governing the complex behavior of a neuron–astrocyte network are essential to generating positive integrated information. This study presents an analytic consideration of this question based on exact and asymptotic expressions for integrated information in terms of exactly known probability distributions for a reduced mathematical model (discrete-time, discrete-state stochastic model) reflecting the main features of the “spiking–bursting” dynamics of a neuron–astrocyte network. The analysis was performed in terms of the empirical “whole minus sum” version of integrated information in comparison to the “decoder based” version. The “whole minus sum” information may change sign, and an interpretation of this transition in terms of “net synergy” is available in the literature. This motivated our particular interest in the sign of the “whole minus sum” information in our analytical considerations. The behaviors of the “whole minus sum” and “decoder based” information measures are found to bear a lot of similarity—they have mutual asymptotic convergence as time-uncorrelated activity increases, and the sign transition of the “whole minus sum” information is associated with a rapid growth in the “decoder based” information. The study aims at creating a theoretical framework for using the spiking–bursting model as an analytically tractable reference point for applying integrated information concepts to systems exhibiting similar bursting behavior. The model can also be of interest as a new discrete-state test bench for different formulations of integrated information.     

Information Fragmentation,Encryption and Information Flow in Complex Biological Networks

Assessing where and how information is stored in biological networks (such as neuronal and genetic networks) is a central task both in neuroscience and in molecular genetics, but most available tools focus on the network’s structure as opposed to its function. Here, we introduce a new information-theoretic tool—information fragmentation analysis—that, given full phenotypic data, allows us to localize information in complex networks, determine how fragmented (across multiple nodes of the network) the information is, and assess the level of encryption of that information. Using information fragmentation matrices we can also create information flow graphs that illustrate how information propagates through these networks. We illustrate the use of this tool by analyzing how artificial brains that evolved in silico solve particular tasks, and show how information fragmentation analysis provides deeper insights into how these brains process information and “think”. The measures of information fragmentation and encryption that result from our methods also quantify complexity of information processing in these networks and how this processing complexity differs between primary exposure to sensory data (early in the lifetime) and later routine processing.     

深度复卷积递归网络模型的师生学习语声增强方法

基于深度神经网络的方法已经在语声增强领域得到了广泛的应用,然而若想取得理想的性能,一般需要规模较大且复杂度较高的模型。因此,在计算资源有限的设备或对延时要求高的环境下容易出现部署困难的问题。为了解决此问题,提出了一种基于深度复卷积递归网络的师生学习语声增强方法。在师生深度复卷积递归网络模型结构中间的复长短时记忆递归模块提取实部和虚部特征流,并分别计算帧级师生距离损失以进行知识转移。同时使用多分辨率频谱损失以进一步提升低复杂度学生模型的性能。实验在公开数据集Voice Bank Demand和DNS Challenge上进行,结果显示所提方法相对于基线学生模型在各项指标上均有明显提升。     

Multi-Class Classification of Medical Data Based on Neural Network Pruning and Information-Entropy Measures

Medical data includes clinical trials and clinical data such as patient-generated health data, laboratory results, medical imaging, and different signals coming from continuous health monitoring. Some commonly used data analysis techniques are text mining, big data analytics, and data mining. These techniques can be used for classification, clustering, and machine learning tasks. Machine learning could be described as an automatic learning process derived from concepts and knowledge without deliberate system coding. However, finding a suitable machine learning architecture for a specific task is still an open problem. In this work, we propose a machine learning model for the multi-class classification of medical data. This model is comprised of two components—a restricted Boltzmann machine and a classifier system. It uses a discriminant pruning method to select the most salient neurons in the hidden layer of the neural network, which implicitly leads to a selection of features for the input patterns that feed the classifier system. This study aims to investigate whether information-entropy measures may provide evidence for guiding discriminative pruning in a neural network for medical data processing, particularly cancer research, by using three cancer databases: Breast Cancer, Cervical Cancer, and Primary Tumour. Our proposal aimed to investigate the post-training neuronal pruning methodology using dissimilarity measures inspired by the information-entropy theory; the results obtained after pruning the neural network were favourable. Specifically, for the Breast Cancer dataset, the reported results indicate a 10.68% error rate, while our error rates range from 10% to 15%; for the Cervical Cancer dataset, the reported best error rate is 31%, while our proposal error rates are in the range of 4% to 6%; lastly, for the Primary Tumour dataset, the reported error rate is 20.35%, and our best error rate is 31%.     

Neural Network Used for the Fusion of Predictions Obtained by the K-Nearest Neighbors Algorithm Based on Independent Data Sources

The article concerns the problem of classification based on independent data sets—local decision tables. The aim of the paper is to propose a classification model for dispersed data using a modified k-nearest neighbors algorithm and a neural network. A neural network, more specifically a multilayer perceptron, is used to combine the prediction results obtained based on local tables. Prediction results are stored in the measurement level and generated using a modified k-nearest neighbors algorithm. The task of neural networks is to combine these results and provide a common prediction. In the article various structures of neural networks (different number of neurons in the hidden layer) are studied and the results are compared with the results generated by other fusion methods, such as the majority voting, the Borda count method, the sum rule, the method that is based on decision templates and the method that is based on theory of evidence. Based on the obtained results, it was found that the neural network always generates unambiguous decisions, which is a great advantage as most of the other fusion methods generate ties. Moreover, if only unambiguous results were considered, the use of a neural network gives much better results than other fusion methods. If we allow ambiguity, some fusion methods are slightly better, but it is the result of this fact that it is possible to generate few decisions for the test object.     

Multistability in bidirectional associative memory neural networks

In this Letter, the multistability issue is studied for Bidirectional Associative Memory (BAM) neural networks. Based on the existence and stability analysis of the neural networks with or without delay, it is found that the 2n-dimensional networks can have n³ equilibria and n² equilibria of them are locally exponentially stable, where each layer of the BAM network has n neurons. Furthermore, the results has been extended to (n+m)-dimensional BAM neural networks, where there are n and m neurons on the two layers respectively. Finally, two numerical examples are presented to illustrate the validity of our results.     

Feasibility of gem identification using reflectance spectra coupled with artificial intelligence

Abstract

Standard traditional gem identification requires expert supervision, while sophisticated modern methods are time-consuming and expensive. In contrast, reflectance spectroscopy coupled with artificial intelligence is economical and convenient and does not require specialist supervision. This study established an artificial neural network model that consists of standard multilayered, feed-forward, and back-propagation neural networks, and obtained reflectance spectra of a transparent gem (almandine), an opaque gem (turquoise), several almandine imitations (agate, plastic, and glass), and several treated turquoise samples (dyed, impregnated, and Zachery treated) using an Analytical Spectral Devices spectrometer. The acquired spectra were used to train and test the artificial neural network model. The results show that the model can effectively discriminate between genuine and imitation gems of different classes. However, discrimination between natural and treated gems of same class is not as effective as discrimination of gems of different classes. The results suggest that an artificial neural network based on reflectance spectroscopy could serve as a useful tool for preliminary gem identification, and the advanced identification needs further training and investigation.     

Attractor-Specific and Common Expression Values in Random Boolean Network Models (with a Preliminary Look at Single-Cell Data)

Random Boolean Networks (RBNs for short) are strongly simplified models of gene regulatory networks (GRNs), which have also been widely studied as abstract models of complex systems and have been used to simulate different phenomena. We define the “common sea” (CS) as the set of nodes that take the same value in all the attractors of a given network realization, and the “specific part” (SP) as the set of all the other nodes, and we study their properties in different ensembles, generated with different parameter values. Both the CS and of the SP can be composed of one or more weakly connected components, which are emergent intermediate-level structures. We show that the study of these sets provides very important information about the behavior of the model. The distribution of distances between attractors is also examined. Moreover, we show how the notion of a “common sea” of genes can be used to analyze data from single-cell experiments.     

Study of Nonlinear Models of Oscillatory Systems by Applying an Intelligent Computational Technique

In this paper, we have analyzed the mathematical model of various nonlinear oscillators arising in different fields of engineering. Further, approximate solutions for different variations in oscillators are studied by using feedforward neural networks (NNs) based on the backpropagated Levenberg–Marquardt algorithm (BLMA). A data set for different problem scenarios for the supervised learning of BLMA has been generated by the Runge–Kutta method of order 4 (RK-4) with the “NDSolve” package in Mathematica. The worth of the approximate solution by NN-BLMA is attained by employing the processing of testing, training, and validation of the reference data set. For each model, convergence analysis, error histograms, regression analysis, and curve fitting are considered to study the robustness and accuracy of the design scheme.     

Noninvertibility and resonance in discrete-time neural networks for time-series processing

We present a computer-assisted study emphasizing certain elements of the dynamics of artificial neural networks (ANNs) used for discrete time-series processing and nonlinear system identification. The structure of the network gives rise to the possibility of multiple inverses of a phase point backward in time; this is not possible for the continuous-time system from which the time series are obtained. Using a two-dimensional illustrative model in an oscillatory regime, we study here the interaction of attractors predicted by the discrete-time ANN model (invariant circles and periodic points locked on them) with critical curves. These curves constitute a generalization of critical points for maps of the interval (in the sense of Julia-Fatou); their interaction with the model-predicted attractors plays a crucial role in the organization of the bifurcation structure and ultimately in determining the dynamic behavior predicted by the neural network.     

水下高分辨率声图中小目标的深度网络分类方法

针对声成像数据缺少条件下的水下沉底小目标分类问题,提出一种深度网络分类算法。首先,采用高斯混合模型对声影区统计特性进行建模并提取声图阴影,在此基础上构建仿真数据集和真实数据集。将仿真数据集输入卷积神经网络进行训练,保留其特征提取部分,用于对真实数据集进行特征提取.重建网络分类部分并采用真实数据集的特征向量进行训练。结果表明,所提出的方法分类正确率可达88.24%,与6种对照方法相比平均分类正确率分别提升8.67%,20.47%,19.78%,11.59%,9.01%,11.58%。验证了所提出方法在小样本条件下具有较好对水下沉底小目标的分类能力。其学习曲线收敛到96.25%,仅比验证曲线高5.14%,说明在一定程度上缓解了过拟合问题。将改进的卷积神经网络应用于融合分类器,通过与逻辑回归分类器、支持向量机对目标进行分类并融合决策,正确率为93.33%,可进一步提高算法的正确率和稳定性.      

深度卷积网络的多品种多厂商药品近红外光谱分类

近红外光谱（NIR）分析具有分析高效、样品无损、环境无污染以及可现场检测等优点,特别适合药品的快速建模分析。但NIR存在吸收强度弱以及谱带重叠等缺点,需要建立稳健可靠的化学计量学模型对其进行分析。深度卷积神经网络是深度学习方法中一个重要分支,它通过逐层抽取数据特征并进行组合、转换,形成更高层的语义特征,具有极强的建模能力,广泛应用于计算机视觉、语音识别等领域,而在药品NIR分析方面尚未见报道。基于深度卷积网络模型,对药品NIR多分类建模进行研究。针对药品NIR数据的特点,设计若干个面向多品种、多厂商药品NIR分类的一维深度卷积网络模型。模型中卷积层和池化层交叠排列用于逐层抽取NIR数据特征,输出层连接softmax分类器,对药品NIR数据进行分类概率预测。在输出层之前采用全局最大池化层,将特征图进行整体池化,形成一个特征点,用于解决全连接层存在的限制输入维度大小,参数过多的问题。同时,在网络模型中引入批处理操作和dropout机制,以防止梯度消失和减小网络过拟合的风险。在网络模型的设计过程中,通过设计不同的卷积网络层数以及不同的卷积核尺寸大小,分析其对建模效果的影响,同时分析五种经典数据预处理方法对NIR分析的影响。以我国7个厂商生产的头孢克肟片和11个厂商生产的苯妥英钠片样本NIR为实验对象, 建立药品的多品种、多厂商分类模型,该模型在二分类、多分类实验中取得了良好的分类效果。在十八分类实验中,当训练集与测试集比例为7∶3时,分类准确率为99.37±0.45,比SVM, BP, AE和ELM算法取得更优的分类性能。同时,深度卷积神经网络模型推理速度较快,优于SVM和ELM算法,但训练速度慢于二者。大量实验结果表明,深度卷积神经网络可对多品种、多厂商药品NIR数据准确、可靠地判别分类,且模型具有良好的鲁棒性和可扩展性。该方法也可推广到烟草、石化等其他领域的NIR数据分类应用中。