Dimensionality driven spin-flop transition in layered iridates

Using resonant x-ray diffraction, we observe an easy c-axis collinear antiferromagnetic structure for the bilayer Sr_{3}Ir_{2}O_{7}, a significant contrast to the single layer Sr_{2}IrO_{4} with in-plane canted moments. Based on a microscopic model Hamiltonian, we show that the observed spin-flop transition as a function of number of IrO_{2} layers is due to strong competition among intra- and interlayer bond-directional pseudodipolar interactions of the spin-orbit entangled J_{eff}=1/2 moments. With this we unravel the origin of anisotropic exchange interactions in a Mott insulator in the strong spin-orbit coupling regime, which holds the key to the various types of unconventional magnetism proposed in 5d transition metal oxides.