Journal or Publication Title:

Математические вопросы кибернетики и вычислительной техники=Կիբեռնետիկայի և հաշվողական տեխնիկայի մաթեմատիկական հարցեր=Mathematical problems of computer science

Date of publication:

1963-

Volume:

63

Additional Information:

click here to follow the link

Coverage:

25-41

Abstract:

In this survey, we explore the broad applications of Information Theory in Machine Learning, highlighting how core concepts like entropy, Mutual Information, and KLdivergence are used to enhance learning algorithms. Since its inception by Claude Shannon, Information Theory has provided mathematical tools to quantify uncertainty, optimize decision-making, and manage the trade-off between model flexibility and generalization. These principles have been integrated across various subfields of Machine Learning, including neural networks, where the Information Bottleneck offers insights into data representation, and reinforcement learning, where entropy-based methods improve exploration strategies. Additionally, measures like Mutual Information are critical in feature selection and unsupervised learning. This survey bridges foundational theory with its practical implementations in modern Machine Learning by providing both historical context and a review of contemporary research.. We also discuss open challenges and future directions, such as scalability and interpretability, highlighting the growing importance of these techniques in next-generation models.

Publisher:

Изд-во НАН РА

Format:

pdf

Identifier:

oai:arar.sci.am:405368

Location of original object:

ՀՀ ԳԱԱ Հիմնարար գիտական գրադարան