Spin nematic and antinematic states in a spin-3/2 isotropic non-Heisenberg magnet

The ground state phase diagram of a general isotropic spin-3/2 system with nearest-neighbor exchange is shown to contain unconventionally ordered spin nematic and antinematic states, as well as usual ferro- and antiferromagnetic phases. The two nematic phases have spontaneously broken rotational symmetry characterized by the long-range order of the nematic director u, as well as the broken time-reversal symmetry described by the pseudospin vector σ. Nematic phase differs from antinematic one by the type of ordering in σ vectors (uniform versus staggered). The ferromagnet-nematic and antiferromagnet-antinematic phase boundaries exhibit enhanced Sp(4) symmetry and correspond to the recently studied effective theory for spin-3/2 cold gases. We discuss optical properties and topological defects in the nematic phases.