Entropy and Mutual Information

About this course

Define Shannon entropy, joint and conditional entropy, mutual information, and the data processing inequality. Prove basic properties and chain rules.

• 20 lessons
• 2 h
• advanced

What you'll learn

• Explain why a quantitative measure of information is essential in communication and statistics.
• Describe the scope and structure of the course.
• Recall key probability concepts such as random variables, distributions, and expectation.
• Apply Jensen's inequality and convexity to information measures.
• Define Shannon entropy for a discrete random variable.
• Interpret entropy as average surprise or uncertainty.

Finishing the course

• 80 knowledge-check questions along the way, so you know the ideas stuck before you move on.
• Watching is free and needs no account. Sign in to save your progress. Pass each lesson's quiz (60%+) to earn the certificate. Each certificate has its own public verification link to share, marked quiz-verified.

Created by