Information theory

Information theory is a branch of applied mathematics and electrical engineering involving the quantification of information. Originally introduced by Claude Shannon in 1948, the theory was primarily designed to find fundamental limits on signal processing operations such as compressing data and reliably storing and communicating data. Since its inception, it has broadened to find applications in many other areas, including statistical inference, natural language processing, cryptography, and even in genetics and neurobiology.

Overview

Information theory studies the transmission, processing, extraction, and utilization of information. Abstractly, information can be thought of as the resolution of uncertainty. In the case of communication of information over a noisy channel, this abstract concept was made concrete in 1948 by Claude Shannon in "A Mathematical Theory of Communication", in which 'information' is thought of as a set of possible messages, where the goal is to send these messages over a noisy channel, and then to have the receiver reconstruct the message with low probability of error, in spite of the channel noise.

History

Information theory was developed by Claude Shannon during World War II to improve the reliability of data transmission over noisy channels. His seminal 1948 paper, "A Mathematical Theory of Communication", laid the groundwork for virtually all modern digital communications technology, including the Internet and mobile phones.

Applications

Information theory has applications in many areas of engineering and science, including telecommunications, computer science, statistics, and genetics. It is used to quantify the amount of information in a signal, to design efficient coding schemes, and to predict the performance of communication systems.

See also

• Entropy (information theory)
• Data compression
• Channel capacity
• Coding theory
• Error detection and correction
• Shannon's source coding theorem
• Shannon's channel coding theorem
• Shannon–Hartley theorem

References

<references />

• Binary Entropy Plot
  Binary Entropy Plot
• Information Theory
  Information Theory
• Binary Symmetric Channel
  Binary Symmetric Channel
• Binary Erasure Channel
  Binary Erasure Channel