Complex modes and instability of full-vectorial beam propagation methods

Full-vectorial beam propagation methods (FVBPMs) are widely used to model light waves propagating in high-index-contrast optical waveguides. We show that the paraxial FVBPM and wide-angle FVBPMs based on diagonal Padé approximants are analytically unstable for waveguides with complex modes. The instability cannot be removed by enlarging the computational domain, increasing the numerical resolution, or using perfectly matched layers, because the complex modes are highly confined around the waveguide core.