Generalized 3G theorem and application to relativistic stable process on non-smooth open sets

Let G(x,y) and GD(x,y) be the Green functions of rotationally invariant symmetric α-stable process in Rd and in an open set D, respectively, where 0<α<2. The inequality GD(x,y)GD(y,z)/GD(x,z)?c(G(x,y)+G(y,z)) is a very useful tool in studying (local) Schrödinger operators. When the above inequality is true with c=c(D)∈(0,∞), then we say that the 3G theorem holds in D. In this paper, we establish a generalized version of 3G theorem when D is a bounded κ-fat open set, which includes a bounded John domain. The 3G we consider is of the form GD(x,y)GD(z,w)/GD(x,w), where y may be different from z. When y=z, we recover the usual 3G. The 3G form GD(x,y)GD(z,w)/GD(x,w) appears in non-local Schrödinger operator theory. Using our generalized 3G theorem, we give a concrete class of functions belonging to the non-local Kato class, introduced by Chen and Song, on κ-fat open sets. As an application, we discuss relativistic α-stable processes (relativistic Hamiltonian when α=1) in κ-fat open sets. We identify the Martin boundary and the minimal Martin boundary with the Euclidean boundary for relativistic α-stable processes in κ-fat open sets. Furthermore, we show that relative Fatou type theorem is true for relativistic stable processes in κ-fat open sets. The main results of this paper hold for a large class of symmetric Markov processes, as are illustrated in the last section of this paper. We also discuss the generalized 3G theorem for a large class of symmetric stable Lévy processes.