Ground state solutions for a class of fractional Hamiltonian systems

In this paper we study the following fractional Hamiltonian systems

$$begin{aligned} left{ begin{array}{lllll} -_{t}D^{alpha }_{infty }(_{-infty }D^{alpha }_{t}x(t))- L(t).x(t)+nabla W(t,x(t))=0, xin H^{alpha }(mathbb {R}, mathbb {R}^{N}), end{array} right. end{aligned}$$

where (alpha in left( {1over {2}}, 1right] , tin mathbb {R}, xin mathbb {R}^N, _{-infty }D^{alpha }_{t}) and (_{t}D^{alpha }_{infty }) are the left and right Liouville–Weyl fractional derivatives of order (alpha ) on the whole axis (mathbb {R}) respectively, (L:mathbb {R}longrightarrow mathbb {R}^{2N}) and (W: mathbb {R}times mathbb {R}^{N}longrightarrow mathbb {R}) are suitable functions. One ground state solution is obtained by applying the monotonicity trick of Jeanjean and the concentration-compactness principle in the case where the matrix L(t) is positive definite and (W in C^{1}(mathbb {R}times mathbb {R}^{N},mathbb {R})) is superquadratic but does not satisfy the usual Ambrosetti–Rabinowitz condition.