Analysis for some properties of discrete time Markov decision processes

This paper investigates properties of the optimality equation and optimal policies in discrete time Markov decision processes with expected discounted total rewards under weak conditions that the model is well defined and the optimality equation is true. The optimal value function is characterized as a solution of the optimality equation and the structure of optimal policies is also given.     

Analytical modeling of Transmission Control Protocol NewReno using Generalized Stochastic Petri Nets

This paper presents a novel analytical model of Transmission Control Protocol (TCP) using a generalized stochastic Petri net (GSPN). Extensive simulation work has been done for the performance evaluation of TCP NewReno protocol. In view of the limitations of the simulation technique, we present an analytical approach using GSPN. A GSPN is a useful mathematical tool that solves continuous time Markov chains for complex systems and evaluates the stationary behavior. In this paper, we analyze the slow‐but‐steady variant of TCP NewReno. The model captures the behavioral aspects of the slow start and the congestion avoidance phase together with the fast retransmit and recovery capabilities of TCP NewReno. Performance metrics such as throughput, goodput, and task completion time of the system are obtained. The effect of variation in the model parameters on the performance is studied. The results are validated using the network simulator, and their accuracy is verified by evaluating the confidence interval. The performance of the proposed model is compared with that of TCP Reno. The performance of the proposed model is also compared with one of the previous models. The numerical illustrations and comparison of the proposed technique with simulation validates the accuracy, efficiency, and competence of the GSPN technique. While GSPN modeling for TCP is investigated in depth for the TCP NewReno and TCP Reno variant in this paper, other protocols could be also analyzed similarly. Copyright © 2013 John Wiley & Sons, Ltd.     

���з������������ǿ��������

In this paper, we mainly studied the limit properties for the countable nonhomogeneous Markov chains. We established some limit properties for the functions of the countable nonhomogeneous Markov chains with variables under the convergence in the sense, which extended the similar conclusions for the functions with two variables. At last, as a corollary, we given the similar result in the homogeneous Markov stock market.     

A Note on Convergence of the Equi-Energy Sampler

This paper studies maximum likelihood estimation for a parameterised elliptic diffusion in a manifold. The focus is on asymptotic properties of maximum likelihood estimates obtained from continuous time observation. These are well known when the underlying manifold is a Euclidean space. However, no systematic study exists in the case of a general manifold. The starting point is to write down the likelihood function and equation. This is achieved using the tools of stochastic differential geometry. Consistency, asymptotic normality and asymptotic optimality of maximum likelihood estimates are then proved, under regularity assumptions. Numerical computation of maximum likelihood estimates is briefly discussed.     

基于马尔可夫决策过程的医疗检查预约优化模型

医技部门的医疗检查如电子计算机断层扫描（CT）、核磁共振成像设备（MRI）、X射线（X－rays）常常有如下三种病人类型：门诊病人、住院病人和急诊病人。针对不同病人类型的需求特点，运用马尔可夫决策过程原理和动态规划方法，建立了医疗检查设备的预约优化模型，并证明了模型的最优性质，得出了不同病人类型的最优预约策略。数值算例的结果表明：本文的预约策略不仅易于实施，而且，通过该模型获得的最大收益比按传统先来先预约的模式所获得的收益要大。     

Mixing time of an unaligned Gibbs sampler on the square

The paper concerns a particular example of the Gibbs sampler and its mixing efficiency. Coordinates of a point are rerandomized in the unit square ${[0,1]}^2$ to approach a stationary distribution with density proportional to $exp(?A^2{(u?v)}^2)$ for $(u,v)\in {[0,1]}^2$ with some large parameter $A$.Diaconis conjectured the mixing time of this process to be $O\left(A^2\right)$ which we confirm in this paper. This improves on the currently known $O(exp\left(A^2\right))$ estimate.     

Continuum versus discrete networks,graph Laplacians,and reproducing kernel Hilbert spaces

Motivated by applications to machine learning, we construct a reversible and irreducible Markov chain whose state space is a certain collection of measurable sets of a chosen l.c.h. space $\mathfrak{X}$. We study the resulting network (connected undirected graph), including transience, Royden and Riesz decompositions, and kernel factorization. We describe a construction for Hilbert spaces of signed measures which comes equipped with a new notion of reproducing kernels and there is a unique solution to a regularized optimization problem involving the approximation of $L^2$ functions by functions of finite energy. The latter has applications to machine learning (for Markov random fields, for example).     

A human visual system-driven image segmentation algorithm

This paper presents a novel image segmentation algorithm driven by human visual system (HVS) properties. Segmentation quality metrics, based on perceptual properties of HVS with respect to segmentation, are integrated into an energy function. The energy function encodes the HVS properties from both region-based and boundary-based perspectives, where the just-noticeable difference (JND) model is employed when calculating the difference between the image contents. Extensive experiments are carried out to compare the performances of three variations of the presented algorithm and several representative segmentation and clustering algorithms available in the literature. The results show superior performance of our approach.