# Information geometry on hierarchy of probability distributions

@article{Amari2001InformationGO, title={Information geometry on hierarchy of probability distributions}, author={Shun-ichi Amari}, journal={IEEE Trans. Inf. Theory}, year={2001}, volume={47}, pages={1701-1711} }

An exponential family or mixture family of probability distributions has a natural hierarchical structure. This paper gives an "orthogonal" decomposition of such a system based on information geometry. A typical example is the decomposition of stochastic dependency among a number of random variables. In general, they have a complex structure of dependencies. Pairwise dependency is easily represented by correlation, but it is more difficult to measure effects of pure triplewise or higher order… Expand

#### 386 Citations

Structure Analysis of a Probabilistic Network in an Information Geometric Framework

- Mathematics, Computer Science
- The 2006 IEEE International Joint Conference on Neural Network Proceedings
- 2006

By adopting a generalized definition of mutual information derived from information geometry, one can, in theory, extend Chow's method of constructing trees based on pair-wise mutual information to an arbitrary clique size. Expand

Hierarchical Quantification of Synergy in Channels

- Computer Science, Mathematics
- Front. Robot. AI
- 2016

A new point of view on the decomposition of channel information into synergies of different order is proposed, which model a multi-input channel as a Markov kernel and can be easily evaluated by an iterative scaling algorithm. Expand

Notes on information geometry and evolutionary processes

- Computer Science, Physics
- ArXiv
- 2004

In these notes I briefly review the essentials of various coordinate bases and of information geometry to give an overview and make the approaches comparable. Expand

Orthogonal decompositions of multivariate statistical dependence measures

- Mathematics, Computer Science
- 2004 IEEE International Conference on Acoustics, Speech, and Signal Processing
- 2004

We describe two multivariate statistical dependence measures which can be orthogonally decomposed to separate the effects of pairwise, triplewise, and higher order interactions between the random… Expand

Information geometry, simulation and complexity in Gaussian random fields

- Mathematics, Computer Science
- Monte Carlo Methods Appl.
- 2016

This paper proposes to quantify how changes in the spatial dependence structure affect the Riemannian metric tensor that equips the model's parametric space and defines Fisher curves, which resemble mathematical models of hysteresis in which the natural orientation is pointed by an arrow of time. Expand

Information Geometry of Multiple Spike Trains

- Mathematics
- 2010

Information geometry studies a family probability distributions by using modern geometry. Since a stochastic model of multiple spike trains is described by a family of probability distributions,… Expand

Extension of information geometry for modelling non-statistical systems

- Mathematics, Physics
- 2014

In this dissertation, an abstract formalism extending information geometry is introduced. This framework encompasses a broad range of modelling problems, including possible applications in machine… Expand

Information-theoretic inference of common ancestors

- Mathematics, Computer Science
- Entropy
- 2015

This work proves an information-theoretic inequality that allows for the inference of common ancestors of observed parts in any DAG representing some unknown larger system and shows that a large amount of dependence in terms of mutual information among the observations implies the existence of a common ancestor that distributes this information. Expand

Finite Information Geometry

- Mathematics
- 2017

This chapter investigates probability distributions on a finite sample space and takes advantage of the more elementary nature of this setting. There are two complementary ways to view a probability… Expand

Fields of Application of Information Geometry

- Mathematics
- 2017

1.
Complexity measures can be geometrically built by using the information distance (Kullback–Leibler divergence) from families with restricted statistical dependencies. The Pythagorean geometry… Expand

#### References

SHOWING 1-10 OF 39 REFERENCES

An Infinite-Dimensional Geometric Structure on the Space of all the Probability Measures Equivalent to a Given One

- Mathematics
- 1995

Let M μ be the set of all probability densities equivalent to a given reference probability measure μ. This set is thought of as the maximal regular (i.e., with strictly positive densities)… Expand

Nonnegative Entropy Measures of Multivariate Symmetric Correlations

- Computer Science, Mathematics
- Inf. Control.
- 1978

A “hierarchical structure” of probabilistic dependence relations is proposed where it is shown that any symmetric correlation associated with a nonnegative entropy is decomposed into pairwise conditional and/or nonconditional correlations. Expand

Statistical inference under multiterminal rate restrictions: A differential geometric approach

- Mathematics, Computer Science
- IEEE Trans. Inf. Theory
- 1989

It is shown that the differential geometry of the manifold of all probability distributions plays a fundamental role in this type of multiterminal problem connecting Shannon information and statistical information. Expand

Dualistic geometry of the manifold of higher-order neurons

- Mathematics, Computer Science
- Neural Networks
- 1991

An information geometrical method, which can be applied to more general neural network manifolds, is proposed and the accuracy of statistical estimation is shown in terms of the dimensionality of a model and the number of examples. Expand

Information geometry of Boltzmann machines

- Mathematics, Medicine
- IEEE Trans. Neural Networks
- 1992

Using the new theory of information geometry, a natural invariant Riemannian metric and a dual pair of affine connections on the Boltzmann neural network manifold are established and the meaning of geometrical structures is elucidated from the stochastic and the statistical point of view. Expand

The relation between information theory and the differential geometry approach to statistics

- Mathematics, Computer Science
- Inf. Sci.
- 1985

It is shown that the Riemannian metric on the probability simplex ∑xi = 1 defined by (ds) 2 = ∑(dx i ) 2 x i has an invariance property under certain probabilistically natural mappings. Expand

$I$-Divergence Geometry of Probability Distributions and Minimization Problems

- Mathematics
- 1975

JSTOR is a not-for-profit service that helps scholars, researchers, and students discover, use, and build upon a wide range of content in a trusted digital archive. We use information technology and… Expand

On the rationale of maximum-entropy methods

- Mathematics
- Proceedings of the IEEE
- 1982

We discuss the relations between maximum-entropy (MAXENT) and other methods of spectral analysis such as the Schuster, Blackman-Tukey, maximum-likelihood, Bayesian, and Autoregressive (AR, ARMA, or… Expand

Fisher information under restriction of Shannon information in multi-terminal situations

- Mathematics
- 1989

Fisher information generally decreases by summarizing observed data into encoded messages. The present paper studies the amount of Fisher information included in independently summarized messages… Expand

Information geometry of the EM and em algorithms for neural networks

- Mathematics, Computer Science
- Neural Networks
- 1995

A unified information geometrical framework for studying stochastic models of neural networks, by focusing on the EM and em algorithms, and proves a condition that guarantees their equivalence. Expand