We consider reinforcement learning algorithms in normal form games. Using two-timescales stochastic approximation, we introduce a model-free algorithm which is asymptotically equivalent to the smooth fictitious play algorithm, in that both result in asymptotic pseudotrajectories to the flow defined by the smooth best response dynamics. Both of these algorithms are shown to converge almost surely to Nash distribution in two-player zero-sum games and $N$-player partnership games. However, there are simple games for which these, and most other adaptive processes, fail to converge--in particular, we consider the $N$-player matching pennies game and Shapley's variant of the rock--scissors--paper game. By extending stochastic approximation results to multiple timescales we can allow each player to learn at a different rate. We show that this extension will converge for two-player zero-sum games and two-player partnership games, as well as for the two special cases we consider.

Let $M_n$ be the maximum of a sample $X_1,\ldots,X_n$ from a discrete distribution and let $W_n$ be the number of $i$'s, $1\le i \le n$, such that $X_i=M_n$. We discuss the asymptotic behavior of the distribution of $W_n$ as $n\to\infty$. The probability that the maximum is unique is of interest in diverse problems, for example, in connection with an algorithm for selecting a winner, and has been studied by several authors using mainly analytic tools. We present here an approach based on the Sukhatme--Rényi representation of exponential order statistics, which gives, as we think, a new insight into the problem.

Odlyzko [Random Structures Algorithms 6 (1995) 275-295] exhibited an asymptotically optimal algorithm, with respect to the average cost, among algorithms that find the maximum of a random walk by using only probes and comparisons. We extend Odlyzko's techniques to prove that his algorithm is indeed asymptotically optimal in distribution (with respect to the stochastic order). We also characterize the limit law of its cost. Computing its moments in two ways allows us to recover a surprising identity concerning Euler sums.

We consider the process $dY_{t}=u_{t}\,dt+dW_{t},$ where $u$ is a process not necessarily adapted to $\mathcal{F}^{Y}$ (the filtration generated by the process $Y)$ and $W$ is a Brownian motion. We obtain a general representation for the likelihood ratio of the law of the $Y$ process relative to Brownian measure. This representation involves only one basic filter (expectation of $u$ conditional on observed process $Y$). This generalizes the result of Kailath and Zakai [Ann. Math. Statist. 42 (1971) 130-140] where it is assumed that the process $u$ is adapted to $\mathcal{F}^{Y}.$ In particular, we consider the model in which $u$ is a functional of $Y$ and of a random element $X$ which is independent of the Brownian motion $W.$ For example, $X$ could be a diffusion or a Markov chain. This result can be applied to the estimation of an unknown multidimensional parameter $\theta$ appearing in the dynamics of the process $u$ based on continuous observation of $Y$ on the time interval $[0,T]$. For a specific hidden diffusion financial model in which $u$ is an unobserved mean-reverting diffusion, we give an explicit form for the likelihood function of $\theta.$ For this model we also develop a computationally explicit E--M algorithm for the estimation of $\theta.$ In contrast to the likelihood ratio, the algorithm involves evaluation of a number of filtered integrals in addition to the basic filter.

Malyshev's theory of asymptotics of stationary probabilities for a random walk in a quarter-plane is extended to cover the case of join-the-shorter-queues.

A hierarchical product model seeks to model network traffic as a product of independent on/off processes. Previous studies have assumed a Markovian structure for component processes amounting to assuming that exponential distributions govern on and off periods, but this is not in good agreement with traffic measurements. However, if the number of factor processes grows and input rates are stabilized by allowing the on period distribution to change suitably, a limiting on/off process can be obtained which has exponentially distributed on periods and whose off periods are equal in distribution to the busy period of an $M/G/\infty$ queue. We give a fairly complete study of the possible limits of the product process as the number of factors grows and offer various characterizations of the approximating processes. We also study the dependence structure of the approximations.

We consider a time-homogeneous random walk $\Xi = \{\xi(t)\}$ on a two-dimensional complex. All of our results here are formulated in a constructive way. By this we mean that for any given random walk we can, with an expression using only the first and second moments of the jumps and the return probabilities for some transient one-dimensional random walks, conclude whether the process is ergodic, null-recurrent or transient. Further we can determine when $p$th moments of passage times $\tau_K$ to sets $S_K = \{x \dvtx \|x\| \leq K\}$ are finite ($p >0$, real). Our main interest is in a new critical case where we will show the long-term behavior of the random walk is very similar to that found for walks with zero mean drift inside the quadrants. Recently a partial case of a polling system model in the critical regime was investigated by Menshikov and Zuyev who give explicit results in terms of the parameters of the queueing model. This model and some others can be interpreted as random walks on two-dimensional complexes.

A fluid approximation gives the main term in the asymptotic expression of the value function for a controllable stochastic network. The policies that have the same asymptotic of their value functions as the value function of the optimal policy are called asymptotically optimal policies. We consider the problem of finding from this set of asymptotically optimal policies a best one in the sense that the next term of its asymptotic expression is minimal. The analysis of this problem is closely connected with large-deviations problems for a random walk.

In this paper, we study the stability of a fluid queue with an infinite-capacity buffer. The input and service rates are governed by a stochastic process, called the environment process, and are allowed to depend on the fluid level in the buffer. The variability of the traffic is modeled by a Brownian motion and a local variance function, which also depends on the fluid level in the buffer. The behavior of this second-order fluid flow model is described by a reflected stochastic differential equation, and, under stationarity and ergodicity assumptions on the environment process, we obtain stability conditions for this general fluid queue.

Two channel assignment policies are considered for a Kelly type loss network with an additional channel continuity requirement. It is assumed that the channels on any given link have distinct identities, and that a connection should be assigned channels with a common identity on all links of its route. Such constraints arise in circuit switched WDM optical networks and wireless cellular networks. A functional law of large numbers, which was previously developed by Hunt and Kurtz and later refined by Zachary and Ziedins, is adapted to analyze a network with two links and three connection types. Asymptotically exact fluid-type approximations for the network process are obtained and their operating points are characterized. The results lead to asymptotic call blocking rates and point out that in cases of practical interest, random channel assignment has asymptotically the same blocking performance with more sophisticated channel assignment policies.

A large financial market is described by a sequence of traditional market models with finite numbers of assets. There are various concepts in the spirit of no asymptotic arbitrage related to the contiguity of a sequence of equivalent martingale measures with respect to the sequence of historical probabilities. In this article, I show that in the case of continuous price processes, the existence of a bicontiguous sequence of martingale measures is equivalent to the property of no asymptotic free lunch with bounded risk.

Following Ann. Appl. Probab. 9 (1999) 904--950 we continue the study of the problem of expected utility maximization in incomplete markets. Our goal is to find minimal conditions on a model and a utility function for the validity of several key assertions of the theory to hold true. In the previous paper we proved that a minimal condition on the utility function alone, that is, a minimal market independent condition, is that the asymptotic elasticity of the utility function is strictly less than 1. In this paper we show that a necessary and sufficient condition on both, the utility function and the model, is that the value function of the dual problem is finite.

We consider the convexity and comparative static properties of a class of $r$-harmonic mappings for a given linear, time-homogeneous and regular diffusion process. We present a set of weak conditions under which the minimal $r$-excessive mappings for the considered diffusion are convex and under which an arbitrary nontrivial $r$-excessive mapping is convex on the regions where it is $r$-harmonic. Consequently, we are able to present a set of usually satisfied conditions under which increased volatility increases the value of $r$-harmonic mappings. We apply our results to a class of optimal stopping problems arising frequently in studies considering the pricing of perpetual American contingent claims and state a set of conditions under which the value function is convex on the continuation region and, consequently, under which increased volatility unambiguously increases the value function and expands the continuation region, thus postponing the rational exercise of the claim.

We study the asymptotic behavior of a mutation--selection genetic algorithm on the integers with finite population of size $p\ge 1$. The mutation is defined by the steps of a simple random walk and the fitness function is linear. We prove that the normalized population satisfies an invariance principle, that a large-deviations principle holds and that the relative positions converge in law. After $n$ steps, the population is asymptotically around $\sqrt{n}$ times the position at time $1$ of a Bessel process of dimension $2p-1$.

We show the existence of nontrivial quasi-stationary measures for conservative attractive particle systems on $\ZZ^d$ conditioned on avoiding an increasing local set $\A$. Moreover, we exhibit a sequence of measures $\{\nu_n\}$, whose $\omega$-limit set consists of quasi-stationary measures. For zero-range processes, with stationary measure $\nur$, we prove the existence of an $L^2(\nur)$ nonnegative eigenvector for the generator with Dirichlet boundary on $\A$, after establishing a priori bounds on the $\{\nu_n\}$.

We give bounds on the exponential decay rate of entropy in the random transposition model and the Bernoulli--Laplace model which are independent of the number of sites and the number of particles. This is then used to give a bound on the time to stationarity in the total variation norm.

Consider the standard first-passage percolation on ${\Z}^d$, $d\geq 2$. Denote by $\phi_{0,n}$ the face--face first-passage time in $[0,n]^d$. It is well known that \[ \lim_{n\rightarrow \infty} {\phi_{0,n}\over n}=\mu(F) \qquad \mbox{a.s. and in } L_1, \] where $F$ is the common distribution on each edge. In this paper we show that the upper and lower tails of $\phi_{0,n}$ are quite different when $\mu(F)>0$. More precisely, we can show that for small $\varepsilon>0$, there exist constants $\alpha(\varepsilon, F)$ and $\beta (\varepsilon, F)$ such that \[ \lim_{n\rightarrow\infty}{-1\over n} \log P \left( \phi_{0,n}\leq n(\mu -\varepsilon) \right) = \alpha (\varepsilon, F) \] and \[ \lim_{n\rightarrow\infty}{-1\over n^d} \log P \left(\phi_{0,n}\geq n(\mu +\varepsilon) \right)= \beta (\varepsilon, F). \]

We consider the large values and the mean of the uniform norms for a sequence of Gaussian processes with continuous sample paths. The convergence of the normalized uniform norm to the standard Gumbel (or double exponential) law is derived for distributions and means. The results are obtained from the Poisson convergence of the associated point process of exceedances for a general class of Gaussian processes. As an application we study the piecewise linear interpolation of Gaussian processes whose local behavior is like fractional (integrated fractional) Brownian motion (or with locally stationary increments). The overall interpolation performance for the random process is measured by the $p$th moment of the approximation error in the uniform norm. The problem of constructing the optimal sets of observation locations (or interpolation knots) is done asymptotically, namely, when the number of observations tends to infinity. The developed limit technique for a sequence of Gaussian nonstationary processes can be applied to analysis of various linear approximation methods.