Optimal greedy policies for stochastic control models

We introduce the notion of a greedy policy for general stochastic control models. Sufficient conditions for the optimality of the greedy policy for finite and infinite horizon are given. Moreover, we derive error bounds if the greedy policy is not optimal. The main results are illustrated by Bayesian information models, discounted Bayesian search problems, stochastic scheduling problems, single-server queueing networks and deterministic dynamic programs.     

Estimation for nonhomogeneous Poisson processes from aggregated data

A well-known heuristic for estimating the rate function or cumulative rate function of a nonhomogeneous Poisson process assumes that the rate function is piecewise constant on a set of data-independent intervals. We investigate the asymptotic (as the amount of data grows) behavior of this estimator in the case of equal interval widths, and show that it can be transformed into a consistent estimator if the interval lengths shrink at an appropriate rate as the amount of data grows.     

MARKOV DECISION PROGRAMMING WITH CONSTRAINTS

ＭＡＲＫＯＶＤＥＣＩＳＩＯＮＰＲＯＧＲＡＭＭＩＮＧＷＩＴＨＣＯＮＳＴＲＡＩＮＴＳＬＩＵＪＩＡＮＹＯＮＧ（刘建庸）；ＬＩＵＫＥ（刘克）（ＩｎｓｔｉｔｕｔｅｏｆＡｐｐｌｉｅｄＭａｔｈｅｍａｔｉｃｓ，ｔｈｅＣｈｉｎｅｓｅＡｃａｄｅｍｙｏｆＳｃｉｅｎｃｅｓ，...     

Networks of infinite-server queues with nonstationary Poisson input

In this paper we focus on networks of infinite-server queues with nonhomogeneous Poisson arrival processes. We start by introducing a more general Poisson-arrival-location model (PALM) in which arrivals move independently through a general state space according to a location stochastic process after arriving according to a nonhomogeneous Poisson process. The usual open network of infinite-server queues, which is also known as a linear population process or a linear stochastic compartmental model, arises in the special case of a finite state space. The mathematical foundation is a Poisson-random-measure representation, which can be obtained by stochastic integration. It implies a time-dependent product-form result: For appropriate initial conditions, the queue lengths (numbers of customers in disjoint subsets of the state space) at any time are independent Poisson random variables. Even though there is no dependence among the queue lengths at each time, there is important dependence among the queue lengths at different times. We show that the joint distribution is multivariate Poisson, and calculate the covariances. A unified framework for constructing stochastic processes of interest is provided by stochastically integrating various functionals of the location process with respect to the Poisson arrival process. We use this approach to study the flows in the queueing network; e.g., we show that the aggregate arrival and departure processes at a given queue (to and from other queues as well as outside the network) are generalized Poisson processes (without necessarily having a rate or unit jumps) if and only if no customer can visit that queue more than once. We also characterize the aggregate arrival and departure processes when customers can visit the queues more frequently. In addition to obtaining structural results, we use the stochastic integrals to obtain explicit expressions for time-dependent means and covariances. We do this in two ways. First, we decompose the entire network into a superposition of independent networks with fixed deterministic routes. Second, we make Markov assumptions, initially for the evolution of the routes and finally for the entire location process. For Markov routing among the queues, the aggregate arrival rates are obtained as the solution to a system of input equations, which have a unique solution under appropriate qualifications, but not in general. Linear ordinary differential equations characterize the time-dependent means and covariances in the totally Markovian case.     

Order statistics for nonstationary time series

Order statistics has an important role in statistical inference. The main purpose of this paper is to investigate order statistics, and also explore its applications in the analysis of nonstationary time series. Our results show that linear functions of order statistics for a large class of time series are asymptotically normal. The methods of proof involve approximations of serially dependent random variables by independent ones. The problems of testing for the existence of a linear trend and the problem of testing randomness versus serial dependence are considered as applications.     

Graphical Diagnostics for Markov Models for Categorical Data

Markov models are widely used as a method for describing categorical data that exhibit stationary and nonstationary autocorrelation. However, diagnostic methods are a largely overlooked topic for Markov models. We introduce two types of residuals for this purpose: one for assessing the length of runs between state changes, and the other for assessing the frequency with which the process moves from any given state to the other states. Methods for calculating the sampling distribution of both types of residuals are presented, enabling objective interpretation through graphical summaries. The graphical summaries are formed using a modification of the probability integral transformation that is applicable for discrete data. Residuals from simulated datasets are presented to demonstrate when the model is, and is not, adequate for the data. The two types of residuals are used to highlight inadequacies of a model posed for real data on seabed fauna from the marine environment.

Supplemental materials, including an R-package RMC with functions to perform the diagnostic measures on the class of models considered in this article, are at the journal’s website. The R-package is also available at CRAN.     

A direct probabilistic approach to solve state equations for nonlinear systems under random excitation

In this paper, a direct probabilistic approach (DPA) is presented to formulate and solve moment equations for nonlinear systems excited by environmental loads that can be either a stationary or nonstationary random process. The proposed method has the advantage of obtaining the response’s moments directly from the initial conditions and statistical characteristics of the corresponding external exci-tations. First, the response’s moment equations are directly derived based on a DPA, which is completely independent of the It?/filtering approach since no specific assumptions regarding the correlation structure of excitation are made. By solving them under Gaussian closure, the response’s moments can be obtained. Subsequently, a multiscale algo-rithm for the numerical solution of moment equations is exploited to improve computational efficiency and avoid much wall-clock time. Finally, a comparison of the results with Monte Carlo (MC) simulation gives good agreement. Furthermore, the advantage of the multiscale algorithm in terms of efficiency is also demonstrated by an engineering example.