Branching laws for minimal holomorphic representations

In this paper we study the branching law for the restriction from SU(n,m) to SO(n,m) of the minimal representation in the analytic continuation of the scalar holomorphic discrete series. We identify the group decomposition with the spectral decomposition of the action of the Casimir operator on the subspace of S(O(n)×O(m))-invariants. The Plancherel measure of the decomposition defines an L²-space of functions, for which certain continuous dual Hahn polynomials furnish an orthonormal basis. It turns out that the measure has point masses precisely when n−m>2. Under these conditions we construct an irreducible representation of SO(n,m), identify it with a parabolically induced representation, and construct a unitary embedding into the representation space for the minimal representation of SU(n,m).