A Review of Shannon and Differential Entropy Rate Estimation |
| |
Authors: | Andrew Feutrill Matthew Roughan |
| |
Institution: | 1.CSIRO/Data61, 13 Kintore Avenue, Adelaide, SA 5000, Australia;2.School of Mathematical Sciences, The University of Adelaide, Adelaide, SA 5005, Australia;3.ARC Centre of Excellence for Mathematical & Statistical Frontiers, The University of Melbourne, Parkville, VIC 3010, Australia |
| |
Abstract: | In this paper, we present a review of Shannon and differential entropy rate estimation techniques. Entropy rate, which measures the average information gain from a stochastic process, is a measure of uncertainty and complexity of a stochastic process. We discuss the estimation of entropy rate from empirical data, and review both parametric and non-parametric techniques. We look at many different assumptions on properties of the processes for parametric processes, in particular focussing on Markov and Gaussian assumptions. Non-parametric estimation relies on limit theorems which involve the entropy rate from observations, and to discuss these, we introduce some theory and the practical implementations of estimators of this type. |
| |
Keywords: | entropy rate estimation parametric non-parametric |
|
|