An Index Integral and Convolution Operator Related to the Kontorovich-Lebedev and Mehler-Fock Transforms

We deal with an index integral involving the product of the modified Bessel functions and associated Legendre functions. It was discovered by Ferrell (Nucl Instrum Methods Phys Res B 96:483?C485, 1995) while comparing solutions of the Laplace equation in different coordinate systems in his study of the so-called surface plasmons in various condensed matter samples. This integral is quite interesting from the pure mathematical point of view and it is absent in famous reference books for series and integrals. We give a rigorous proof of this formula and discuss its particular cases. We also construct a convolution operator associated with this integral, which is related to the classical Kontorovich-Lebedev and Mehler-Fock transforms. Mapping properties and the norm estimates in weighted L _p -spaces, 1 ?? p ?? 2 are investigated. An application to a class of convolution integral equations is considered. Necessary and sufficient conditions are found for the solvability of these equations in L ₂.