On instability and stability of three-dimensional gravity driven viscous flows in a bounded domain

We investigate the instability and stability of some steady-states of a three-dimensional nonhomogeneous incompressible viscous flow driven by gravity in a bounded domain Ω   of class C²$C^2$. When the steady density is heavier with increasing height (i.e., the Rayleigh–Taylor steady-state), we show that the steady-state is linear unstable (i.e., the linear solution grows in time in H²$H^2$) by constructing a (standard) energy functional and exploiting the modified variational method. Then, by introducing a new energy functional and using a careful bootstrap argument, we further show that the steady-state is nonlinear unstable in the sense of Hadamard. When the steady density is lighter with increasing height, we show, with the help of a restricted condition imposed on steady density, that the steady-state is linearly globally stable and nonlinearly asymptotically stable in the sense of Hadamard.