Riesz potentials and integral geometry in the space of rectangular matrices

Riesz potentials on the space of rectangular n×m matrices arise in diverse “higher rank” problems of harmonic analysis, representation theory, and integral geometry. In the rank-one case m=1 they coincide with the classical operators of Marcel Riesz. We develop new tools and obtain a number of new results for Riesz potentials of functions of matrix argument. The main topics are the Fourier transform technique, representation of Riesz potentials by convolutions with a positive measure supported by submanifolds of matrices of rank<m, the behavior on smooth and L^p functions. The results are applied to investigation of Radon transforms on the space of real rectangular matrices.