What Is Information Theory?

Information theory is a branch of mathematics that defines efficient and practical methods by which data can be exchanged and interpreted. The concept originated in a mid-twentieth-century essay by a mathematician named Claude Shannon, which set many important precedents for digital technology, including using bits as units of measurement. Shannon's work was based on several earlier attempts to define information in mathematical terms. The earliest known effort occurred in 1848 when German physicist Heinrich Hertz attempted to quantify the energy contained in electromagnetic waves. In 1923, American engineer Ralph Hartley attempted to define information as "the negative logarithm of the probability" that would receive a message would receive a letter correctly. These early attempts led to the development of an abstract mathematical theory based on the idea that each possible message has an associated probability distribution over possible symbols and that these distributions can be represented as sets of probabilities (or, more generally, sets over cardinality). These distributions from a mathematical structure called an alphabet can be used as input for any encoding scheme or output for any decoding scheme. Before Claude Shannon came along, electronic communication had many things that could have been improved. It could have been more secure, and it could have been more efficient. You see, back in the day, we used analog transmission to communicate, meaning that information was transmitted as a series of waves or pulses. While that worked okay in short distances, it didn't cut it when you wanted to go long distances. The signals would degrade and become unreliable. So Claude Shannon got to work on improving electronic communication during World War II at Bell Labs (the research and development arm of the Bell Telephone Company). He figured out how to get around those problems—and now we all have super-fast internet connections!