Potential theory of subordinate killed Brownian motion in a domain

 Subordination of a killed Brownian motion in a bounded domain D⊂ℝ ^d via an α/2-stable subordinator gives a process Z _t whose infinitesimal generator is −(−Δ| _D )^α/2, the fractional power of the negative Dirichlet Laplacian. In this paper we study the properties of the process Z _t in a Lipschitz domain D by comparing the process with the rotationally invariant α-stable process killed upon exiting D. We show that these processes have comparable killing measures, prove the intrinsic ultracontractivity of the generator of Z _t , prove the intrinsic ultracontractivity of the semigroup of Z _t , and, in the case when D is a bounded C ^1,1 domain, obtain bounds on the Green function and the jumping kernel of Z _t . Received: 4 April 2002 / Revised version: 1 July 2002 / Published online: 19 December 2002 This work was completed while the authors were in the Research in Pairs program at the Mathematisches Forschungsinstitut Oberwolfach. We thank the Institute for the hospitality. The research of the first author is supported in part by NSF Grant DMS-9803240. The research of the second author is supported in part by MZT grant 037008 of the Republic of Croatia. Mathematics Subject Classification (2000): Primary 60J45; Secondary 60J75, 31C25 Key words or phrases: Killed Brownian motions – Stable processes – Subordination – Fractional Laplacian