Mutual Information Gradient Induces Contraction of Information Radius

Mori, Tsutomu

doi:10.51094/jxiv.2005

##article.authors##

Mori, Tsutomu Department of Human Lifesciences, Fukushima Medical University School of Nursing https://researchmap.jp/read0186242

DOI:

https://doi.org/10.51094/jxiv.2005

キーワード:

mutual information、 information geometry、 distance contraction、 copula representations、 influential forces、 informational dynamics

抄録

Mutual information (MI) quantifies how knowledge of one system reduces uncertainty about another. We show that when interactions between two systems follow the gradient ascent of mutual information, the information radius
r(t) = H(X_t) + H(Y_t) - 2 MI(X_t; Y_t)
decreases monotonically along the trajectory. Contraction is exact when marginal entropies remain fixed, satisfying dr(t)/dt = -2 d(MI)/dt, and remains robust in the small marginal drift regime, where changes in marginal entropies are dominated by gains in MI. The theorem applies to both discrete and continuous settings; for continuous systems, we use a copula-based formulation that preserves marginal structure and provides a projected-gradient interpretation.

Under isothermal, closed, and quasi-static conditions, the relation ΔU = -k_B T ΔMI appears, suggesting a mechanical analogy. Beyond this regime, we formalize an Attractive Influential Force (AIF) framework, describing informational interactions that ensure contraction of r(t). We also show that exponential kernels based on r, and their generalizations built from a unified information distance D_info, exhibit a syntactic invariance: all align qualitatively with the direction of the MI gradient. This framework unifies geometric, thermodynamic, and kernel-based viewpoints on informational coupling, and provides foundations for applications in learning dynamics, biological information flow, and networked multi-agent systems.

利益相反に関する開示

The authors declare no potential conflict of interests.

ダウンロード *前日までの集計結果を表示します

ダウンロード実績データは、公開の翌日以降に作成されます。

引用文献

C. E. Shannon, “A Mathematical Theory of Communication,” Bell System Technical Journal, vol. 27, pp. 379–423, 623–656, 1948. https://ieeexplore.ieee.org/abstract/document/6773024

T. M. Cover and J. A. Thomas, Elements of Information Theory, 2nd ed., Wiley, 2006. https://onlinelibrary.wiley.com/doi/book/10.1002/047174882X

R. M. Gray, Entropy and Information Theory, Springer, 2011. https://link.springer.com/book/10.1007/978-1-4419-7970-4

I. Csiszár and P. C. Shields, “Information Theory and Statistics: A Tutorial,” Foundations and Trends in Communications and Information Theory, vol. 1, 2004. https://www.nowpublishers.com/article/Details/CIT-004

T. Mori, “Mutual Information Gradient Induces Contraction of Information Distance,” Jxiv preprint, submitted 2025.

R. Landauer, “Irreversibility and Heat Generation in the Computing Process,” IBM Journal of Research and Development, 5(3), 1961. https://doi.org/10.1147/rd.53.0183

S. Amari and H. Nagaoka, Methods of Information Geometry, Amer. Math. Soc., 2000. https://www.amazon.com/Methods-Information-Geometry-Monographs-Statistics/dp/0821843028

J. M. R. Parrondo, J. M. Horowitz, and T. Sagawa, “Thermodynamics of Information,” Nature Physics, 11, 2015. https://doi.org/10.1038/nphys3230

S. Amari, Information Geometry and Its Applications, Springer, 2016. https://link.springer.com/book/10.1007/978-4-431-55978-8

T. Sagawa and M. Ueda, “Second Law of Thermodynamics with Discrete Quantum Feedback Control,” Physical Review Letters, 100:080403, 2008. https://doi.org/10.1103/PhysRevLett.100.080403

J. M. Horowitz and M. Esposito, “Thermodynamics with Continuous Information Flow,” Physical Review X, 4:031015, 2014. https://doi.org/10.1103/PhysRevX.4.031015

U. Seifert, “Stochastic thermodynamics, fluctuation theorems and molecular machines,” Reports on Progress in Physics, 75:126001, 2012. https://doi.org/10.1088/0034-4885/75/12/126001

S. Still, D. A. Sivak, A. J. Bell, and G. E. Crooks, “Thermodynamics of Prediction,” Physical Review Letters, 109:120604, 2012. https://doi.org/10.1103/PhysRevLett.109.120604

M. Deweese and W. Bialek, “Information Flow in Sensory Neurons,” Neural Computation, 18, 2006. https://link.springer.com/article/10.1007/BF02451830

T. Mori, “Influential Force: From Higgs to the Novel Immune Checkpoint KYNU,” Jxiv preprint, 2022. https://doi.org/10.51094/jxiv.156

C. Villani, Optimal Transport: Old and New, Springer, 2008. https://doi.org/10.1007/978-3-540-71050-9

F. Otto, “The Geometry of Dissipative Evolution Equations,” Communications in Partial Differential Equations, 26(1–2), 2001. https://doi.org/10.1081/PDE-100002243

R. Jordan, D. Kinderlehrer, and F. Otto, “Variational Formulation of the Fokker–Planck Equation,” SIAM Journal on Mathematical Analysis, 29(1), 1998. https://doi.org/10.1137/S0036141096303359

S. Kullback, Information Theory and Statistics, Dover, 1959. https://store.doverpublications.com/products/9780486696843

S. Amari, “Natural Gradient Works Efficiently in Learning,” Neural Computation, 10(2), 1998. https://doi.org/10.1162/089976698300017746

N. Tishby, F. Pereira, and W. Bialek, “The Information Bottleneck Method,” Proc. Allerton Conf., 1999. https://www.cs.huji.ac.il/labs/learning/Papers/allerton.pdf

R. Shwartz-Ziv and N. Tishby, “Opening the Black Box of Deep Neural Networks via Information,” arXiv:1703.00810, 2017. https://arxiv.org/abs/1703.00810

I. Csiszár, “I-Divergence Geometry and Minimization Problems,” Annals of Probability, 3(1), 1975. https://www.jstor.org/stable/2959270

A. Sklar, “Fonctions de Répartition à n Dimensions,” Publ. Inst. Statist. Univ. Paris, 1959. https://hal.science/hal-04094463/document

R. Nelsen, An Introduction to Copulas, 2nd ed., Springer, 2006. https://link.springer.com/book/10.1007/0-387-28678-0

H. Joe, Multivariate Models and Dependence Concepts, CRC Press, 1997. https://doi.org/10.1201/9780367803896

A. Kraskov, H. Stögbauer, and P. Grassberger, “Estimating Mutual Information,” Physical Review E, 69:066138, 2004. https://doi.org/10.1103/PhysRevE.69.066138

M. Belghazi et al., “Mutual Information Neural Estimation,” ICML, 2018. https://proceedings.mlr.press/v80/belghazi18a.html

J. Massey, “Causality, Feedback and Directed Information,” Proc. ISIT, 1990. https://isiweb.ee.ethz.ch/papers/arch/mass-1990-1.pdf

T. Schreiber, “Measuring Information Transfer,” Physical Review Letters, 85:461, 2000. https://doi.org/10.1103/PhysRevLett.85.461

L. Barnett, A. K. Seth, and A. B. Barrett, “Granger Causality and Transfer Entropy Are Equivalent for Gaussian Variables,” Physical Review Letters, 103:238701, 2009. https://doi.org/10.1103/PhysRevLett.103.238701

L. Kozachenko and N. Leonenko, “Sample Estimate of the Entropy of a Random Vector,” Problems of Information Transmission, 23, 1987. https://www.mathnet.ru/php/archive.phtml?wshow=paper&jrnid=ppi&paperid=797&option_lang=eng

A. Saxe, J. McClelland, and S. Ganguli, “Exact Solutions to the Nonlinear Dynamics of Deep Linear Neural Networks,” arXiv:1312.6120, 2013. https://arxiv.org/abs/1312.6120

S. Mei, A. Montanari, and P.-M. Nguyen, “Dynamics of Stochastic Gradient Descent for Two-Layer Neural Networks in the Teacher-Student Setup,” NeurIPS, 2019. https://proceedings.neurips.cc/paper_files/paper/2019/hash/cab070d53bd0d200746fb852a922064a-Abstract.html

A. Achille and S. Soatto, “Information Dropout: Learning Optimal Representations via Information Bottleneck,” IEEE Transactions on Pattern Analysis and Machine Intelligence, 2018. https://doi.org/10.1109/TPAMI.2017.2784440

M. Li, X. Chen, X. Li, B. Ma, and P. Vitányi, “The Similarity Metric,” IEEE Transactions on Information Theory, 50(12), 2004. https://doi.org/10.1109/TIT.2004.838101

M. Li and P. Vitányi, An Introduction to Kolmogorov Complexity and Its Applications, 4th ed., Springer, 2019. https://link.springer.com/book/10.1007/978-3-030-11298-1

C. W. Gardiner, Stochastic Methods: A Handbook for the Natural and Social Sciences, 4th ed., Springer, 2009. https://link.springer.com/book/9783540707127

H. Risken, The Fokker–Planck Equation: Methods of Solution and Applications, 2nd ed., Springer, 1989. https://link.springer.com/book/10.1007/978-3-642-61544-3

D. A. McQuarrie, Statistical Mechanics, University Science Books, 2000. https://mitpress.mit.edu/9781891389153/statistical-mechanics

M. Tuckerman, Statistical Mechanics: Theory and Molecular Simulation, Oxford Univ. Press, 2010. https://global.oup.com/academic/product/statistical-mechanics-theory-and-molecular-simulation-9780198825562