A Model for Multi-timescaled Sequential Decision-making Processes with Adversary

Extending the multi-timescale model proposed by the author et al. in the context of Markov decision processes, this paper proposes a simple analytical model called M timescale two-person zero-sum Markov Games (MMGs) for hierarchically structured sequential decision-making processes in two players' competitive situations where one player (the minimizer) wishes to minimize their cost that will be paid to the adversary (the maximizer). In this hierarchical model, for each player, decisions in each level in the M-level hierarchy are made in M different discrete timescales and the state space and the control space of each level in the hierarchy are non-overlapping with those of the other levels, respectively, and the hierarchy is structured in a "pyramid" sense such that a decision made at level m (slower timescale) state and/or the state will affect the evolutionary decision making process of the lower-level m+1 (faster timescale) until a new decision is made at the higher level but the lower-level decisions themselves do not affect the transition dynamics of higher levels. The performance produced by the lower-level decisions will affect the higher level decisions for each player. A hierarchical objective function for the minimizer and the maximizer is defined, and from this we define "multi-level equilibrium value function" and derive a "multi-level equilibrium equation". We also discuss how to solve hierarchical games exactly.     

Selection of a correlated equilibrium in Markov stopping games

This paper deals with an extension of the concept of correlated strategies to Markov stopping games. The Nash equilibrium approach to solving nonzero-sum stopping games may give multiple solutions. An arbitrator can suggest to each player the decision to be applied at each stage based on a joint distribution over the players’ decisions according to some optimality criterion. This is a form of equilibrium selection. Examples of correlated equilibria in nonzero-sum games related to the best choice problem are given. Several concepts of criteria for selecting a correlated equilibrium are used.     

Social systems analysis II

We argue that there are 4 basic solution concepts for 2 × 2 symmetric games and corresponding to these, 4 basic human interactions and 4 types of societies. We propose a model in which the type of interaction is predicted by independence, inequality of power, communication, and mutuality of interests. We prove theorems classifying Moulin's inessential games in the case of an interior Nash equilibrium and argue that this explains coalition formation. We discuss classification of socially significant ideologies.     

Average stochastic games for continuous-time jump processes

We consider nonzero-sum games for continuous-time jump processes with unbounded transition rates under expected average payoff criterion. The state and action spaces are Borel spaces and reward rates are unbounded. We introduce an approximating sequence of stochastic game models with extended state space, for which the uniform exponential ergodicity is obtained. Moreover, we prove the existence of a stationary almost Markov Nash equilibrium by introducing auxiliary static game models. Finally, a cash flow model is employed to illustrate the results.     

Nonzero-sum games for continuous-time Markov chains with unbounded transition and average payoff rates

This paper attempts to study two-person nonzero-sum games for denumerable continuous-time Markov chains determined by transition rates,with an expected average criterion.The transition rates are allowed to be unbounded,and the payoff functions may be unbounded from above and from below.We give suitable conditions under which the existence of a Nash equilibrium is ensured.More precisely,using the socalled "vanishing discount" approach,a Nash equilibrium for the average criterion is obtained as a limit point of a sequence of equilibrium strategies for the discounted criterion as the discount factors tend to zero.Our results are illustrated with a birth-and-death game.     

Zero-sum risk-sensitive stochastic games for continuous time Markov chains

We study infinite horizon discounted-cost and ergodic-cost risk-sensitive zero-sum stochastic games for controlled continuous time Markov chains on a countable state space. For the discounted-cost game, we prove the existence of value and saddle-point equilibrium in the class of Markov strategies under nominal conditions. For the ergodic-cost game, we prove the existence of values and saddle point equilibrium by studying the corresponding Hamilton-Jacobi-Isaacs equation under a certain Lyapunov condition.     

A maximum principle for Markov regime-switching forward–backward stochastic differential games and applications

In this paper, we present an optimal control problem for stochastic differential games under Markov regime-switching forward–backward stochastic differential equations with jumps. First, we prove a sufficient maximum principle for nonzero-sum stochastic differential games problems and obtain equilibrium point for such games. Second, we prove an equivalent maximum principle for nonzero-sum stochastic differential games. The zero-sum stochastic differential games equivalent maximum principle is then obtained as a corollary. We apply the obtained results to study a problem of robust utility maximization under a relative entropy penalty and to find optimal investment of an insurance firm under model uncertainty.     

On implementation via demand commitment games

A simple version of the Demand Commitment Game is shown to implement the Shapley value as the unique subgame perfect equilibrium outcome for any n-person characteristic function game. This improves upon previous models devoted to this implementation problem in terms of one or more of the following: a) the range of characteristic function games addressed, b) the simplicity of the underlying noncooperative game (it is a finite horizon game where individuals make demands and form coalitions rather than make comprehensive allocation proposals and c) the general acceptability of the noncooperative equilibrium concept. A complete characterization of an equilibrium strategy generating the Shapley value outcomes is provided. Furthermore, for 3 player games, it is shown that the Demand Commitment Game can implement the core for games which need not be convex but have cores with nonempty interiors. Received March 1995/Final version February 1997     

Complementarity and Diagonal Dominance in Discounted Stochastic Games

We consider discounted stochastic games characterized by monotonicity, supermodularity and diagonal dominance assumptions on the reward functions and the transition law. A thorough novel discussion of the scope and limitations of this class of games is provided. Existence of a Markov-stationary equilibrium for the infinite-horizon game, proved by Curtat (1996), is summarized. Uniqueness of Markov equilibrium and dominance solvability of the finite-horizon game are established. In both cases, the equilibrium strategies and the corresponding value functions are nondecreasing Liptschitz-continuous functions of the state vector. Some specific economic applications are discussed.     

Nonzero-sum non-stationary discounted Markov game model

The goal of this paper is provide a theory of K-person non-stationary Markov games with unbounded rewards, for a countable state space and action spaces. We investigate both the finite and infinite horizon problems. We define the concept of strong Nash equilibrium and present conditions for both problems for which strong Nash or Nash equilibrium strategies exist for all players within the Markov strategies, and show that the rewards in equilibrium satisfy the optimality equations.     

A polynomial expression for the Owen value in the maintenance cost game

The class of maintenance cost games was introduced in 2000 to deal with a cost allocation problem arising in the reorganization of the railway system in Europe. The main application of maintenance cost games regards the allocation of the maintenance costs of a facility among the agents using it. To that aim it was first proposed to utilize the Shapley value, whose computation for maintenance cost games can be made in polynomial time. In this paper, we propose to model this cost allocation problem as a maintenance cost game with a priori unions and to use the Owen value as a cost allocation rule. Although the computation of the Owen value has exponential complexity in general, we provide an expression for the Owen value of a maintenance cost game with cubic polynomial complexity. We finish the paper with an illustrative example using data taken from the literature of railways management.     

Characterization of equilibrium strategies in a class of difference games

We formulate a class of N player difference games and derive open—loop and Markov equilibria. It turns out that both types of equilibria can be characterized by a set of difference equations that describe the equilibrium dynamics. We analyze the stability properties of the difference equations that correspond to an equilibrium and find that in both the open—loop and the Markov game there is convergence towards a steady state equilibrium     

Multi-player stopping games with redistribution of payoffs and BSDEs with oblique reflection

We examine the connections between a novel class of multi-person stopping games with redistribution of payoffs and multi-dimensional reflected BSDEs in discrete- and continuous-time frameworks. Our goal is to provide an essential extension of classic results for two-player stopping games (Dynkin games) to the multi-player framework. We show the link between certain multi-period m$m$-player stopping games and a new kind of m$m$-dimensional reflected BSDEs. The existence and uniqueness of a solution to continuous-time reflected BSDEs are established. Continuous-time redistribution games are constructed with the help of reflected BSDEs and a characterization of the value of such stopping games is provided.     

Infrastructure security games

Infrastructure security against possible attacks involves making decisions under uncertainty. This paper presents game theoretic models of the interaction between an adversary and a first responder in order to study the problem of security within a transportation infrastructure. The risk measure used is based on the consequence of an attack in terms of the number of people affected or the occupancy level of a critical infrastructure, e.g. stations, trains, subway cars, escalators, bridges, etc. The objective of the adversary is to inflict the maximum damage to a transportation network by selecting a set of nodes to attack, while the first responder (emergency management center) allocates resources (emergency personnel or personnel-hours) to the sites of interest in an attempt to find the hidden adversary. This paper considers both static and dynamic, in which the first responder is mobile, games. The unique equilibrium strategy pair is given in closed form for the simple static game. For the dynamic game, the equilibrium for the first responder becomes the best patrol policy within the infrastructure. This model uses partially observable Markov decision processes (POMDPs) in which the payoff functions depend on an exogenous people flow, and thus, are time varying. A numerical example illustrating the algorithm is presented to evaluate an equilibrium strategy pair.     

Atomic weights and the combinatorial game of bipass

We define an all-small ruleset, bipass, within the framework of normal play combinatorial games. A game is played on finite strips of black and white stones. Stones of different colors are swapped provided they do not bypass one of their own kind. We find a simple surjective function from the strips to integer atomic weights (Berlekamp, Conway and Guy 1982) that measures the number of units in all-small games. This result provides explicit winning strategies for many games, and in cases where it does not, it gives narrow bounds for the canonical form game values. We find game values for some parametrized families of games, including an infinite number of strips of value ?, and we prove that the game value ?2 does not appear as a disjunctive sum of bipass. Lastly, we define the notion of atomic weight tameness, and prove that optimal misére play bipass resembles optimal normal play.     

On discounted stochastic games with incomplete information on payoffs and a security application

This paper presents a robust optimization model for n$n$-person finite state/action stochastic games with incomplete information on payoffs. For polytopic uncertainty sets, we propose an explicit mathematical programming formulation for an equilibrium calculation. It turns out that a global optimal of this mathematical program yields an equilibrium point and epsilon-equilibria can be calculated based on this result. We briefly describe an incomplete information version of a security application that can benefit from robust game theory.     

Randomized stopping games and Markov market games

We study nonzero-sum stopping games with randomized stopping strategies. The existence of Nash equilibrium and ɛ-equilibrium strategies are discussed under various assumptions on players random payoffs and utility functions dependent on the observed discrete time Markov process. Then we will present a model of a market game in which randomized stopping times are involved. The model is a mixture of a stochastic game and stopping game. Research supported by grant PBZ-KBN-016/P03/99.     

Sensitive equilibria for ergodic stochastic games with countable state spaces

We consider stochastic games with countable state spaces and unbounded immediate payoff functions. Our assumptions on the transition structure of the game are based on a recent work by Meyn and Tweedie [19] on computable bounds for geometric convergence rates of Markov chains. The main results in this paper concern the existence of sensitive optimal strategies in some classes of zero-sum stochastic games. By sensitive optimality we mean overtaking or 1-optimality. We also provide a new Nash equilibrium theorem for a class of ergodic nonzero-sum stochastic games with denumerable state spaces.     

Monitoring cooperative equilibria in a stochastic differential game

This paper deals with a class of equilibria which are based on the use of memory strategies in the context of continuous-time stochastic differential games. In order to get interpretable results, we will focus the study on a stochastic differential game model of the exploitation of one species of fish by two competing fisheries. We explore the possibility of defining a so-called cooperative equilibrium, which will implement a fishing agreement. In order to obtain that equilibrium, one defines a monitoring variable and an associated retaliation scheme. Depending on the value of the monitoring variable, which provides some evidence of a deviation from the agreement, the probability increases that the mode of a game will change from a cooperative to a punitive one. Both the monitoring variable and the parameters of this jump process are design elements of the cooperative equilibrium. A cooperative equilibrium designed in this way is a solution concept for a switching diffusion game. We solve that game using the sufficient conditions for a feedback equilibrium which are given by a set of coupled HJB equations. A numerical analysis, approximating the solution of the HJB equations through an associated Markov game, enables us to show that there exist cooperative equilibria which dominate the classical feedback Nash equilibrium of the original diffusion game model.This research was supported by FNRS-Switzerland, NSERC-Canada, FCAR-Quebec.