(1) Department of Mathematics and Statistics, York University, 4700 Keele Street, M3J 1P3, ON, Canada;(2) Department of Statistics, University of British Columbia, 33-6356 Agriculture Road, V6T 1Z2, BC, Canada
Abstract:
In this article, mixture distributions and weighted likelihoods are derived within an information-theoretic framework and shown to be closely related. This surprising relationship obtains in spite of the arithmetic form of the former and the geometric form of the latter. Mixture distributions are shown to be optima that minimize the entropy loss under certain constraints. The same framework implies the weighted likelihood when the distributions in the mixture are unknown and information from independent samples generated by them have to be used instead. Thus the likelihood weights trade bias for precision and yield inferential procedures such as estimates that can be more reliable than their classical counterparts.