Orthonormal mode sets for the two-dimensional fractional Fourier transformation

A family of orthonormal mode sets arises when Hermite-Gauss modes propagate through lossless first-order optical systems. It is shown that the modes at the output of the system are eigenfunctions for the symmetric fractional Fourier transformation if and only if the system is described by an orthosymplectic ray transformation matrix. Essentially new orthonormal mode sets can be obtained by letting helical Laguerre-Gauss modes propagate through an antisymmetric fractional Fourier transformer. The properties of these modes and their representation on the orbital Poincaré sphere are studied.