Stationary,completely mixed and symmetric optimal and equilibrium strategies in stochastic games

In this paper, we address various types of two-person stochastic games—both zero-sum and nonzero-sum, discounted and undiscounted. In particular, we address different aspects of stochastic games, namely: (1) When is a two-person stochastic game completely mixed? (2) Can we identify classes of undiscounted zero-sum stochastic games that have stationary optimal strategies? (3) When does a two-person stochastic game possess symmetric optimal/equilibrium strategies? Firstly, we provide some necessary and some sufficient conditions under which certain classes of discounted and undiscounted stochastic games are completely mixed. In particular, we show that, if a discounted zero-sum switching control stochastic game with symmetric payoff matrices has a completely mixed stationary optimal strategy, then the stochastic game is completely mixed if and only if the matrix games restricted to states are all completely mixed. Secondly, we identify certain classes of undiscounted zero-sum stochastic games that have stationary optima under specific conditions for individual payoff matrices and transition probabilities. Thirdly, we provide sufficient conditions for discounted as well as certain classes of undiscounted stochastic games to have symmetric optimal/equilibrium strategies—namely, transitions are symmetric and the payoff matrices of one player are the transpose of those of the other. We also provide a sufficient condition for the stochastic game to have a symmetric pure strategy equilibrium. We also provide examples to show the sharpness of our results.     

Zero-sum risk-sensitive stochastic games on a countable state space

Infinite horizon discounted-cost and ergodic-cost risk-sensitive zero-sum stochastic games for controlled Markov chains with countably many states are analyzed. Upper and lower values for these games are established. The existence of value and saddle-point equilibria in the class of Markov strategies is proved for the discounted-cost game. The existence of value and saddle-point equilibria in the class of stationary strategies is proved under the uniform ergodicity condition for the ergodic-cost game. The value of the ergodic-cost game happens to be the product of the inverse of the risk-sensitivity factor and the logarithm of the common Perron–Frobenius eigenvalue of the associated controlled nonlinear kernels.     

Perturbation theory for games in normal form and stochastic games

In this paper, the effect on values and optimal strategies of perturbations of game parameters (payoff function, transition probability function, and discount factor) is studied for the class of zero-sum games in normal form and for the class of stationary, discounted, two-person, zero-sum stochastic games.A main result is that, under certain conditions, the value depends on these parameters in a pointwise Lipschitz continuous way and that the sets of -optimal strategies for both players are upper semicontinuous multifunctions of the game parameters.Extensions to general-sum games and nonstationary stochastic games are also indicated.     

Two-person zero-sum stochastic games

Two-person zero-sum stochastic games with finite state and action spaces are considered. The expected average payoff criterion is introduced. In the special case of single controller games it is shown that the optimal stationary policies and the value of the game can be obtained from the optimal solutions to a pair of dual programs. For multichain structures, a decomposition algorithm is given which produces such optimal stationary policies for both players. In the case of both players controlling the transitions, a generalized game is obtained, the solution of which gives the optimal policies.     

A note on two-person zero-sum communicating stochastic games

For undiscounted two-person zero-sum communicating stochastic games with finite state and action spaces, a solution procedure is proposed that exploits the communication property, i.e., working with irreducible games over restricted strategy spaces. The proposed procedure gives the value of the communicating game with an arbitrarily small error when the value is independent of the initial state.     

Vertical linear complementarity and discounted zero-sum stochastic games with ARAT structure

In this paper we consider a two-person zero-sum discounted stochastic game with ARAT structure and formulate the problem of computing a pair of pure optimal stationary strategies and the corresponding value vector of such a game as a vertical linear complementarity problem. We show that Cottle-Dantzig’s algorithm (a generalization of Lemke’s algorithm) can solve this problem under a mild assumption. Received July 8, 1998 / Revised version received April 16, 1999? Published online September 15, 1999     

On stochastic games with additive reward and transition structure

In this paper, we introduce a new class of two-person stochastic games with nice properties. For games in this class, the payoffs as well as the transitions in each state consist of a part which depends only on the action of the first player and a part dependent only on the action of the second player.For the zero-sum games in this class, we prove that the orderfield property holds in the infinite-horizon case and that there exist optimal pure stationary strategies for the discounted as well as the undiscounted payoff criterion. For both criteria also, finite algorithms are given to solve the game. An example shows that, for nonzero sum games in this class, there are not necessarily pure stationary equilibria. But, if such a game possesses a stationary equilibrium point, then there also exists a stationary equilibrium point which uses in each state at most two pure actions for each player.     

Sensitive equilibria for ergodic stochastic games with countable state spaces

We consider stochastic games with countable state spaces and unbounded immediate payoff functions. Our assumptions on the transition structure of the game are based on a recent work by Meyn and Tweedie [19] on computable bounds for geometric convergence rates of Markov chains. The main results in this paper concern the existence of sensitive optimal strategies in some classes of zero-sum stochastic games. By sensitive optimality we mean overtaking or 1-optimality. We also provide a new Nash equilibrium theorem for a class of ergodic nonzero-sum stochastic games with denumerable state spaces.     

Stochastic Games with Average Payoff Criterion

We study two-person stochastic games on a Polish state and compact action spaces and with average payoff criterion under a certain ergodicity condition. For the zero-sum game we establish the existence of a value and stationary optimal strategies for both players. For the nonzero-sum case the existence of Nash equilibrium in stationary strategies is established under certain separability conditions. Accepted 9 January 1997     

A finite step algorithm via a bimatrix game to a single controller non-zero sum stochastic game

Given a non-zero sum discounted stochastic game with finitely many states and actions one can form a bimatrix game whose pure strategies are the pure stationary strategies of the players and whose penalty payoffs consist of the total discounted costs over all states at any pure stationary pair. It is shown that any Nash equilibrium point of this bimatrix game can be used to find a Nash equilibrium point of the stochastic game whenever the law of motion is controlled by one player. The theorem is extended to undiscounted stochastic games with irreducible transitions when the law of motion is controlled by one player. Examples are worked out to illustrate the algorithm proposed.The work of this author was supported in part by the NSF grants DMS-9024408 and DMS 8802260.     

Stochastic games with metric state space

In this paper the stochastic two-person zero-sum game of Shapley is considered, with metric state space and compact action spaces. It is proved that both players have stationary optimal strategies, under conditions which are weaker than those ofMaitra/Parthasarathy (a.o. no compactness of the state space). This is done in the following way: we show the existence of optimal strategies first for the one-period game with general terminal reward, then for then-period games (n=1,2,...); further we prove that the game over the infinite horizon has a valuev, which is the limit of then-period game values. Finally the stationary optimal strategies are found as optimal strategies in the one-period game with terminal rewardv.     

Pure strategy equilibria in symmetric two-player zero-sum games

We observe that a symmetric two-player zero-sum game has a pure strategy equilibrium if and only if it is not a generalized rock-paper-scissors matrix. Moreover, we show that every finite symmetric quasiconcave two-player zero-sum game has a pure equilibrium. Further sufficient conditions for existence are provided. Our findings extend to general two-player zero-sum games using the symmetrization of zero-sum games due to von Neumann. We point out that the class of symmetric two-player zero-sum games coincides with the class of relative payoff games associated with symmetric two-player games. This allows us to derive results on the existence of finite population evolutionary stable strategies.     

A class of stochastic games with ordered field property

It is shown that discounted general-sum stochastic games with two players, two states, and one player controlling the rewards have the ordered field property. For the zero-sum case, this result implies that, when starting with rational data, also the value is rational and that the extreme optimal stationary strategies are composed of rational components.     

Noncooperative stochastic games under a n-stage local contraction assumption

We consider a class of noncooperative stochastic games with general state and action spaces and with a state dependent discount factor. The expected time duration between any two stages of the game is not bounded away from zero, so that the usual N-stage contraction assumption, uniform over all admissible strategies, does not hold. We propose milder sufficient regularity conditions, allowing strategies that give rise with probability one to any number of simultaneous stages. We give sufficient conditions for the existence of equilibrium and ∈-equilibrium stationary strategies in the sense of Nash. In the two-player zero-sum case, when an equilibrium strategy exists, the value of the game is the unique fixed point of a specific functional operator and can be computed by dynamic programming.     

Ordered field property for stochastic games when the player who controls transitions changes from state to state

In this paper, we consider a zero-sum stochastic game with finitely many states restricted by the assumption that the probability transitions from a given state are functions of the actions of only one of the players. However, the player who thus controls the transitions in the given state will not be the same in every state. Further, we assume that all payoffs and all transition probabilities specifying the law of motion are rational numbers. We then show that the values of both a -discounted game, for rational , and of a Cesaro-average game are in the field of rational numbers. In addition, both games possess optimal stationary strategies which have only rational components. Our results and their proofs form an extension of the results and techniques which were recently developed by Parthasarathy and Raghavan (Ref. 1).The author wishes to thank Professor T. E. S. Raghavan for introducing him to this problem and for discussing stochastic games with him on many occasions. This research was supported in part by AFOSR Grant No. 78–3495B.     

Semi-markov strategies in stochastic games

For a stochastic game with countable state and action spaces we prove, that solutions in the game where all players are restricted to semi-markov strategies are solutions for the unrestricted game. In addition we show, that if all players, except for one, fix a stationary strategy, then the best the remaining player can do, is solving a markov decision problem, corresponding to the fixed stationary strategies.