Discrete data, Fisher information & information geometry

INFORMATION GEOMETRY, THE EMBEDDING PRINCIPLE, AND DOCUMENT CLASSIFICATION by Guy Lebanon attempts to relate Fisher information metrics to discontinuous/discrete data.

Also from The Information Geometry of Chaos by Carlo Cafaro:

3.2   The Fisher-Rao information metric  (page 45)
In this section, a quantitative description of the notion of change is briefly reviewed (for more details see [42]).  First, change can be measured by distinguishability since the larger the change involved in going from one state to another, the easier it is to distinguish between them.  Next, using the ME method one assigns a probability distribution to each state.  This transforms the problem of distinguishing between two states into the problem of distinguishing between the corresponding probability distributions.   The extent to which one distribution can be distinguished from another is given by the distance between them as measured by the Fisher-Rao information metric [31, 32, 33].   Thus change is measured by distinguishability which is measured by distance.  Let the microstates of a physical system be labelled by x, and let m(x)dx be the number of microstates in the range dx.  We assume that a state of the system (i.e., a macrostate) is defined by the expected values θ^μ ...