Non-separable Lattices,Gabor Orthonormal Bases and Tilings

Let \(K\subset {\mathbb {R}}^d\) be a bounded set with positive Lebesgue measure. Let \(\Lambda =M({\mathbb {Z}}^{2d})\) be a lattice in \({\mathbb {R}}^{2d}\) with density dens\((\Lambda )=1\). It is well-known that if M is a diagonal block matrix with diagonal matrices A and B, then \({\mathcal {G}}(|K|^{-1/2}\chi _K, \Lambda )\) is an orthonormal basis for \(L^2({\mathbb {R}}^d)\) if and only if K tiles both by \(A({\mathbb {Z}}^d)\) and \(B^{-t}({\mathbb {Z}}^d)\). However, there has not been any intensive study when M is not a diagonal matrix. We investigate this problem for a large class of important cases of M. In particular, if M is any lower block triangular matrix with diagonal matrices A and B, we prove that if \({\mathcal {G}}(|K|^{-1/2}\chi _K, \Lambda )\) is an orthonormal basis, then K can be written as a finite union of fundamental domains of \(A({{\mathbb {Z}}}^d)\) and at the same time, as a finite union of fundamental domains of \(B^{-t}({{\mathbb {Z}}}^d)\). If \(A^tB\) is an integer matrix, then there is only one common fundamental domain, which means K tiles by a lattice and is spectral. However, surprisingly, we will also illustrate by an example that a union of more than one fundamental domain is also possible. We also provide a constructive way for forming a Gabor window function for a given upper triangular lattice. Our study is related to a Fuglede’s type problem in Gabor setting and we give a partial answer to this problem in the case of lattices.