Nilpotent orbits and some small unitary representations of indefinite orthogonal groups

For 2?m?l/2, let G be a simply connected Lie group with as Lie algebra, let be the complexification of the usual Cartan decomposition, let K be the analytic subgroup with Lie algebra , and let be the universal enveloping algebra of . This work examines the unitarity and K spectrum of representations in the “analytic continuation” of discrete series of G, relating these properties to orbits in the nilpotent radical of a certain parabolic subalgebra of .The roots with respect to the usual compact Cartan subalgebra are all ±e_i±e_j with 1?i<j?l. In the usual positive system of roots, the simple root e_m−e_m+1 is noncompact and the other simple roots are compact. Let be the parabolic subalgebra of for which e_m−e_m+1 contributes to and the other simple roots contribute to , let L be the analytic subgroup of G with Lie algebra , let , let be the sum of the roots contributing to , and let be the parabolic subalgebra opposite to .The members of are nilpotent members of . The group acts on with finitely many orbits, and the topological closure of each orbit is an irreducible algebraic variety. If Y is one of these varieties, let R(Y) be the dual coordinate ring of Y; this is a quotient of the algebra of symmetric tensors on that carries a fully reducible representation of .For , let . Then λ_s defines a one-dimensional module . Extend this to a module by having act by 0, and define . Let be the unique irreducible quotient of . The representations under study are and , where and Π_S is the Sth derived Bernstein functor.For s>2l−2, it is known that π_s=π_s′ and that π_s′ is in the discrete series. Enright, Parthsarathy, Wallach, and Wolf showed for m?s?2l−2 that π_s=π_s′ and that π_s′ is still unitary. The present paper shows that π_s′ is unitary for 0?s?m−1 even though π_s≠π_s′, and it relates the K spectrum of the representations π_s′ to the representation of on a suitable R(Y) with Y depending on s. Use of a branching formula of D. E. Littlewood allows one to obtain an explicit multiplicity formula for each K type in π_s′; the variety Y is indispensable in the proof. The chief tools involved are an idea of B. Gross and Wallach, a geometric interpretation of Littlewood's theorem, and some estimates of norms.It is shown further that the natural invariant Hermitian form on π_s′ does not make π_s′ unitary for s<0 and that the K spectrum of π_s′ in these cases is not related in the above way to the representation of on any R(Y).A final section of the paper treats in similar fashion the simply connected Lie group with Lie algebra , 2?m?l/2.