Course Details

Subject {L-T-P / C} : EE4404 : Information Theory and Coding {3-0-0 / 3}
Subject Nature : Theory
Coordinator : Prof. Dipti Patra


Entropy and Mutual Information: Entropy, Joint Entropy and Conditional Entropy, Relative Entropy and Mutual Information, Chain Rules, Data-Processing Inequality, Fano’s Inequality

Typical Sequences and Asymptotic Equipartition Property: Asymptotic Equipartition Property Theorem, Consequences of the AEP: Data Compression, High-Probability Sets and the Typical Set

Source Coding and Data Compression: Kraft Inequality, Huffman Codes, Optimality of Huffman Codes

Channel Capacity: Symmetric Channels, Properties of Channel Capacity, Jointly Typical Sequences, Channel Coding Theorem, Fano’s Inequality and the Converse to the Coding Theorem

Differential Entropy and Gaussian Channel: Differential Entropy, AEP for Continuous Random Variables, Properties of Differential Entropy, Relative Entropy, and Mutual Information, Coding Theorem for Gaussian Channels

Linear Binary Block Codes: Generator and Parity-Check Matrices, Repetition and Single-Parity-Check Codes, Binary Hamming Codes, Error Detection with Linear Block Codes, Weight Distribution and Minimum Hamming Distance of a Linear Block Code, Hard-decision and Soft-decision Decoding of Linear Block Codes, Cyclic Codes, Parameters of BCH and RS Codes, Interleaved and Concatenated Codes

Convolutional Codes: Encoder Realizations and Classifications, Minimal Encoders, Trellis representation, MLSD and the Viterbi Algorithm, Bit-wise MAP Decoding and the BCJR Algorithm.

Course Objectives

  1. Learn how to analyse and measure the information per symbol emitted from a source
  2. Learn how to analyse the information-carrying capacity of the communication channel
  3. Learn how to design source compression codes to improve the efficiency of information transmission.
  4. Learn the basic theory needed for data encryptions

Course Outcomes

At the end of the course, students will be able to
1. Understand and explain the basic concepts of information theory, source coding, channel and channel capacity, channel coding and relation among them.
2. Describe the real life applications based on the fundamental theory.
3. Calculate entropy, channel capacity, bit error rate, code rate, steady-state probability and so on.
4. Implement the encoder and decoder of one block code or convolutional code using any program language.

Essential Reading

  1. Thomas Cover, Joy Thomas, Elements of Information Theory, Wiley
  2. William Ryan, Shu Lin, Channel Codes: Classical and Modern, Cambridge

Supplementary Reading

  1. A. ElGamal and Y. H. Kim, Network Information Theory, Cambridge , 2011
  2. Robert Gallager, Information Theory and Reliable Communication, John Willey