On stochastic games

In this paper, we consider the stochastic games of Shapley, when the state and action spaces are all infinite. We prove that, under certain conditions, the stochastic game has a value and that both players have optimal strategies.Part of this research was supported by NSF grant. The authors are indebted to L. S. Shapley for the useful discussions on this and related topics. The authors thank the referee for pointing out an ambiguity in the formulation of Lemma 2.4 in an earlier draft of this article.     

On noncooperative hybrid stochastic games

The present article models and analyzes a noncooperative hybrid stochastic game of two players. The main phase (prime hybrid mode) of the game is preceded by “unprovoked” hostile actions by one of the players (during antecedent hybrid mode) that at some time transforms into a large scale conflict between two players. The game lasts until one of the players gets ruined. The latter occurs when the cumulative damage to the losing player exceeds a fixed threshold. Both hybrid modes are formalized by marked point stochastic processes and the theory of fluctuations is utilized as one of the chief techniques to arrive at a closed form functional describing the status of both players at the ruin time.     

On stochastic games in economics

In this paper we show that many results on equilibria in stochastic games arising from economic theory can be deduced from the theorem on the existence of a correlated equilibrium due to Nowak and Raghavan. Some new classes of nonzero-sum Borel state space discounted stochastic games having stationary Nash equilibria are also presented. Three nontrivial examples of dynamic stochastic games arising from economic theory are given closed form solutions. Research partially supported by MNSW grant 1 P03A 01030.     

Two-player stochastic games II: The case of recursive games

This paper contains the second step in the proof of existence of equilibrium payoffs for two-player stochastic games. It deals with the case of positive absorbing recursive games     

On a problem of stochastic differential games

The process of bargaining between management and union during a strike is modelled by a nonlinear stochastic differential game. It is assumed that the two sides bargain in the mood of a cooperative game. A pair of Pareto-optimal strategies is obtained.     

On strategy improvement algorithms for simple stochastic games

The study of simple stochastic games (SSGs) was initiated by Condon for analyzing the computational power of randomized space-bounded alternating Turing machines. The game is played by two players, MAX and MIN, on a directed multigraph, and when the play terminates at a sink vertex s, MAX wins from MIN a payoff p(s)∈[0,1]. Condon proved that the problem SSG-VALUE—given a SSG, determine whether the expected payoff won by MAX is greater than 1/2 when both players use their optimal strategies—is in NP∩coNP. However, the exact complexity of this problem remains open, as it is not known whether the problem is in P or is hard for some natural complexity class. In this paper, we study the computational complexity of a strategy improvement algorithm by Hoffman and Karp for this problem. The Hoffman–Karp algorithm converges to optimal strategies of a given SSG, but no non-trivial bounds were previously known on its running time. We prove a bound of O(n²/n) on the convergence time of the Hoffman–Karp algorithm, and a bound of O(²0.78n) on a randomized variant. These are the first non-trivial upper bounds on the convergence time of these strategy improvement algorithms.     

On Nikaido-Isoda type theorems for discounted stochastic games

We present a class of countable state space stochastic games with discontinuous payoff functions satisfying some assumptions similar to the ones of Nikaido and Isoda for one-stage games. We prove that these games possess stationary equilibria. We show that after adding some concavity assumptions these equilibria are nonrandomized. Further, we present an example of input (or production) dynamic game satisfying the assumptions of our model. We give a closed-form solution for this game.     

Discounted,positive, and noncooperative stochastic games

In this paper, we consider the stochastic games ofShapley and prove under certain conditions the stochastic game has a value and both players have optimal strategies. We also prove a similar result for noncooperative stochastic games.     

On stochastic games with additive reward and transition structure

In this paper, we introduce a new class of two-person stochastic games with nice properties. For games in this class, the payoffs as well as the transitions in each state consist of a part which depends only on the action of the first player and a part dependent only on the action of the second player.For the zero-sum games in this class, we prove that the orderfield property holds in the infinite-horizon case and that there exist optimal pure stationary strategies for the discounted as well as the undiscounted payoff criterion. For both criteria also, finite algorithms are given to solve the game. An example shows that, for nonzero sum games in this class, there are not necessarily pure stationary equilibria. But, if such a game possesses a stationary equilibrium point, then there also exists a stationary equilibrium point which uses in each state at most two pure actions for each player.     

On the minimax principle and zero-sum stochastic differential games

The problem of prior and delayed commitment in zero-sum stochastic differential games is discussed. A new formulation and solution based on the delayed-commitment model is derived and its significant implications to stochastic games and control are considered.     

On linear stochastic differential games with average cost criterions

In this paper, we consider scalar linear stochastic differential games with average cost criterions. We solve the dynamic programming equations for these games and give the synthesis of saddle-point and Nash equilibrium solutions.The authors wish to thank A. Ichikawa for providing the initial impetus and helpful advice.     

Linear-quadratic stochastic pursuit-evasion games

A linear-quadratic differential game in which the system state is affected by disturbance and both players have access to different measurements is solved. The problem is first converted to an optimization problem in infinite-dimensional state space and then solved using standard techniques. For convenience, L ₂-white noise instead of Wiener process setup is used.     

Two-person zero-sum stochastic games

Two-person zero-sum stochastic games with finite state and action spaces are considered. The expected average payoff criterion is introduced. In the special case of single controller games it is shown that the optimal stationary policies and the value of the game can be obtained from the optimal solutions to a pair of dual programs. For multichain structures, a decomposition algorithm is given which produces such optimal stationary policies for both players. In the case of both players controlling the transitions, a generalized game is obtained, the solution of which gives the optimal policies.     

Non-cooperative stochastic dominance games

A non-cooperative stochastic dominance game is a non-cooperative game in which the only knowledge about the players' preferences and risk attitudes is presumed to be their preference orders on the set ofn-tuples of pure strategies. Stochastic dominance equilibria are defined in terms of mixed strategies for the players that are efficient in the stochastic dominance sense against the strategies of the other players. It is shown that the set of SD equilibria equals all Nash equilibria that can be obtained from combinations of utility functions that are consistent with the players' known preference orders. The latter part of the paper looks at antagonistic stochastic dominance games in which some combination of consistent utility functions is zero-sum over then-tuples of pure strategies.     

On extremal solutions to stochastic control problems,II

The set of attainable laws of the joint state-control process of a controlled diffusion is analyzed from a convex analytic viewpoint. Various equivalence relations depending on one-dimensional marginals thereof are defined on this set and the corresponding equivalence classes are studied.     

Perfect equilibria in stochastic games

We examine stochastic games with finite state and action spaces. For the -discounted case, as well as for the irreducible limiting average case, we show the existence of trembling-hand perfect equilibria and give characterizations of those equilibria. In the final section, we give an example which illustrates that the existence of stationary limiting average equilibria in a nonirreducible stochastic game does not imply the existence of a perfect limiting average equilibrium.Support was provided by the Netherlands Organization for Scientific Research NWO via the Netherlands Foundation for Mathematics SMC, Project 10-64-10.     

Algorithms for discounted stochastic games

In this paper, a two-person zero-sum discounted stochastic game with a finite state space is considered. The movement of the game from state to state is jointly controlled by the two players with a finite number of alternatives available to each player in each of the states. We present two convergent algorithms for arriving at minimax strategies for the players and the value of the game. The two algorithms are compared with respect to computational efficiency. Finally, a possible extension to nonzero sum stochastic game is suggested.This research was supported in part by funds allocated to the Department of Operations Research, School of Management, Case Western Reserve University under Contract No. DAHC 19-68-C-0007 (Project Themis) with the U.S. Army Research Office, Durham, North Carolina. The authors thank the referees for their valuable suggestions.     

Fictitious play in stochastic games

In this paper we examine an extension of the fictitious play process for bimatrix games to stochastic games. We show that the fictitious play process does not necessarily converge, not even in the 2 × 2 × 2 case with a unique equilibrium in stationary strategies. Here 2 × 2 × 2 stands for 2 players, 2 states, 2 actions for each player in each state.     

Cooperative games with stochastic payoffs

This paper introduces a new class of cooperative games arising from cooperative decision making problems in a stochastic environment. Various examples of decision making problems that fall within this new class of games are provided. For a class of games with stochastic payoffs where the preferences are of a specific type, a balancedness concept is introduced. A variant of Farkas' lemma is used to prove that the core of a game within this class is non-empty if and only if the game is balanced. Further, other types of preferences are discussed. In particular, the effects the preferences have on the core of these games are considered.