On stochastic games

In this paper, we consider the stochastic games of Shapley, when the state and action spaces are all infinite. We prove that, under certain conditions, the stochastic game has a value and that both players have optimal strategies.Part of this research was supported by NSF grant. The authors are indebted to L. S. Shapley for the useful discussions on this and related topics. The authors thank the referee for pointing out an ambiguity in the formulation of Lemma 2.4 in an earlier draft of this article.     

Perturbation theory for games in normal form and stochastic games

In this paper, the effect on values and optimal strategies of perturbations of game parameters (payoff function, transition probability function, and discount factor) is studied for the class of zero-sum games in normal form and for the class of stationary, discounted, two-person, zero-sum stochastic games.A main result is that, under certain conditions, the value depends on these parameters in a pointwise Lipschitz continuous way and that the sets of -optimal strategies for both players are upper semicontinuous multifunctions of the game parameters.Extensions to general-sum games and nonstationary stochastic games are also indicated.     

Perfect equilibria in stochastic games

We examine stochastic games with finite state and action spaces. For the -discounted case, as well as for the irreducible limiting average case, we show the existence of trembling-hand perfect equilibria and give characterizations of those equilibria. In the final section, we give an example which illustrates that the existence of stationary limiting average equilibria in a nonirreducible stochastic game does not imply the existence of a perfect limiting average equilibrium.Support was provided by the Netherlands Organization for Scientific Research NWO via the Netherlands Foundation for Mathematics SMC, Project 10-64-10.     

Fictitious play in stochastic games

In this paper we examine an extension of the fictitious play process for bimatrix games to stochastic games. We show that the fictitious play process does not necessarily converge, not even in the 2 × 2 × 2 case with a unique equilibrium in stationary strategies. Here 2 × 2 × 2 stands for 2 players, 2 states, 2 actions for each player in each state.     

On stochastic games in economics

In this paper we show that many results on equilibria in stochastic games arising from economic theory can be deduced from the theorem on the existence of a correlated equilibrium due to Nowak and Raghavan. Some new classes of nonzero-sum Borel state space discounted stochastic games having stationary Nash equilibria are also presented. Three nontrivial examples of dynamic stochastic games arising from economic theory are given closed form solutions. Research partially supported by MNSW grant 1 P03A 01030.     

Average-discounted equilibria in stochastic games

In stochastic games with finite state and action spaces, we examine existence of equilibria where player 1 uses the limiting average reward and player 2 a discounted reward for the evaluations of the respective payoff sequences. By the nature of these rewards, the far future determines player 1's reward, while player 2 is rather interested in the near future. This gives rise to a natural cooperation between the players along the course of the play. First we show the existence of stationary ε-equilibria, for all ε>0, in these games. However, besides these stationary ε-equilibria, there also exist ε-equilibria, in terms of only slightly more complex ultimately stationary strategies, which are rather in the spirit of these games because, after a large stage when the discounted game is not interesting any longer, the players cooperate to guarantee the highest feasible reward to player 1. Moreover, we analyze an interesting example demonstrating that 0-equilibria do not necessarily exist in these games, not even in terms of history dependent strategies. Finally, we examine special classes of stochastic games with specific conditions on the transition and payoff structures. Several examples are given to clarify all these issues.     

Semi-markov strategies in stochastic games

For a stochastic game with countable state and action spaces we prove, that solutions in the game where all players are restricted to semi-markov strategies are solutions for the unrestricted game. In addition we show, that if all players, except for one, fix a stationary strategy, then the best the remaining player can do, is solving a markov decision problem, corresponding to the fixed stationary strategies.     

Solvable states in stochastic games

This paper deals with undiscounted stochastic games. As in Thuijsman-Vrieze [9], we consider specific states, which we call solvable. The existence of such states in every game is proved in a new way. This proof implies the existence of equilibrium payoffs in stochastic games with at most 3 states. On an example, we relate our work to the construction of Thuijsman and Vrieze.     

Small perturbations and stochastic games

The purpose of this note is to apply results from the theory of Markov chains with rare transitions to stochastic games. The results obtained here are used in the proof of existence of equilibrium payoffs in two-player stochastic games.     

Linear-quadratic stochastic pursuit-evasion games

A linear-quadratic differential game in which the system state is affected by disturbance and both players have access to different measurements is solved. The problem is first converted to an optimization problem in infinite-dimensional state space and then solved using standard techniques. For convenience, L ₂-white noise instead of Wiener process setup is used.     

Two-person zero-sum stochastic games

Two-person zero-sum stochastic games with finite state and action spaces are considered. The expected average payoff criterion is introduced. In the special case of single controller games it is shown that the optimal stationary policies and the value of the game can be obtained from the optimal solutions to a pair of dual programs. For multichain structures, a decomposition algorithm is given which produces such optimal stationary policies for both players. In the case of both players controlling the transitions, a generalized game is obtained, the solution of which gives the optimal policies.     

Non-cooperative stochastic dominance games

A non-cooperative stochastic dominance game is a non-cooperative game in which the only knowledge about the players' preferences and risk attitudes is presumed to be their preference orders on the set ofn-tuples of pure strategies. Stochastic dominance equilibria are defined in terms of mixed strategies for the players that are efficient in the stochastic dominance sense against the strategies of the other players. It is shown that the set of SD equilibria equals all Nash equilibria that can be obtained from combinations of utility functions that are consistent with the players' known preference orders. The latter part of the paper looks at antagonistic stochastic dominance games in which some combination of consistent utility functions is zero-sum over then-tuples of pure strategies.     

Cyclic Markov equilibria in stochastic games

We examine a three-person stochastic game where the only existing equilibria consist of cyclic Markov strategies. Unlike in two-person games of a similar type, stationary ε-equilibria (ε > 0) do not exist for this game. Besides we characterize the set of feasible equilibrium rewards.     

Finitely additive and measurable stochastic games

We consider two-person zero-sum stochastic games with arbitrary state and action spaces, a finitely additive law of motion and limit superior payoff function. The players use finitely additive strategies and it is shown that such a game has a value, if the payoff function is evaluated in accordance with the theory of strategic measures as developed by Dubins and Savage. Moreover, when a Borel structure is imposed on the problem, together with an equicontinuity condition on the law of motion, the value of the game is the same whether calculated in terms of countably additive strategies or finitely additive ones.Research done under the auspices of the Santa Fe Institute Economics Research Program which includes grants from Citicorp/Citibank, Research Corp., the Alex C. Walker Foundation and the Russell Sage Foundation and by grants to SFI from the John D. and Catherine T. MacArthur Foundation, the National Science Foundation (PHY-8714918), and the U.S. Department of Energy (ER-FG05-88ER25054).     

Bilinear programming and structured stochastic games

One-step algorithms are presented for two classes of structured stochastic games, namely, those with additive rewards and transitions and those which have switching controllers. Solutions to such classes of games under the average reward criterion can be derived from optimal solutions to appropriate bilinear programs. The validity of using bilinear programming as a solution method follows from two preliminary theorems, the first of which is a complete classification of undiscounted stochastic games with optimal stationary strategies. The second of these preliminary theorems relaxes the conditions of the classification theorem for certain classes of stochastic games and provides the basis for the bilinear programming results. Analogous results hold for the discounted stochastic games with the above special structures.This research was supported in part by the Air Force Office of Scientific Research and by the National Science Foundation under Grant No. ECS-850-3440.     

On stochastic games,II

In this paper, we consider positive stochastic games, when the state and action spaces are all infinite. We prove that, under certain conditions, the positive stochastic game has a value and that the maximizing player has an -optimal stationary strategy and the minimizing player has an optimal stationary strategy.The authors are grateful to Professor David Blackwell and the referee for some useful comments.     

Algorithms for discounted stochastic games

In this paper, a two-person zero-sum discounted stochastic game with a finite state space is considered. The movement of the game from state to state is jointly controlled by the two players with a finite number of alternatives available to each player in each of the states. We present two convergent algorithms for arriving at minimax strategies for the players and the value of the game. The two algorithms are compared with respect to computational efficiency. Finally, a possible extension to nonzero sum stochastic game is suggested.This research was supported in part by funds allocated to the Department of Operations Research, School of Management, Case Western Reserve University under Contract No. DAHC 19-68-C-0007 (Project Themis) with the U.S. Army Research Office, Durham, North Carolina. The authors thank the referees for their valuable suggestions.     

On noncooperative hybrid stochastic games

The present article models and analyzes a noncooperative hybrid stochastic game of two players. The main phase (prime hybrid mode) of the game is preceded by “unprovoked” hostile actions by one of the players (during antecedent hybrid mode) that at some time transforms into a large scale conflict between two players. The game lasts until one of the players gets ruined. The latter occurs when the cumulative damage to the losing player exceeds a fixed threshold. Both hybrid modes are formalized by marked point stochastic processes and the theory of fluctuations is utilized as one of the chief techniques to arrive at a closed form functional describing the status of both players at the ruin time.     

Bayesian games as stochastic processes

This paper examines stochastic processes generated by sequential games each of whose “states” includes a distribution and whose up-dating rules for state transitions may include Bayes operators. General properties of discrete-time Markov processes are applied to specific games such as iterated matrix games, including Prisoners' Dilemma, and to a new model for the Asch experiments on conflict and conformity.     

Cooperative games with stochastic payoffs

This paper introduces a new class of cooperative games arising from cooperative decision making problems in a stochastic environment. Various examples of decision making problems that fall within this new class of games are provided. For a class of games with stochastic payoffs where the preferences are of a specific type, a balancedness concept is introduced. A variant of Farkas' lemma is used to prove that the core of a game within this class is non-empty if and only if the game is balanced. Further, other types of preferences are discussed. In particular, the effects the preferences have on the core of these games are considered.