[Book Summary - Information Theory] Information Theory: A Tutorial Intorudction, Ch 4 by James V. Stone

     1. Mutual information 

• The average surprisal of the symbols in the system 
• The entropy of a single symbol = its surprisal 
• Conditional entropy H(X|Y) = noise entropy H(Y|X) 
• Accuracy of transmission --> mutual information (overlapping part is information transmitted)

• 2. Joint probability of two systems 

• 
  

• 3. Mutual information and information rate

• For a channel with inputs from (X) and outputs from (Y), the information rate is determined by their mutual information, which is in turn related to H(X), H(Y), and their relationship  
• What's the situation with: 
• High vs. Low input H? input entropy 6; output entropy 3 
• Let's think about tossing a dice and assume that we consider either odd numbers only or even numbers only; it should be wasting information without noise cases
• High vs. Low output H? 
• High vs. Low independence of input and output?
• If totally independent, there should be no overlapping portion in a Venn diagram.
• The input/output entropy should be high, and the noise should be low   

• 4. Crosstalk 

• A case of loss information: Substitution error
• Adding "redundancy" (help compensate for information loss) might be one way to solve such loss

• 5. Redundancy is good and bad?

• Increase the rate at which information can be transmitted (Resource cost; e.g., articulatory effort)
• But increase the accuracy with which it is transmitted  

• 6. Noisy typewriter example

•  

• 7. How to add redundancy?

• Adding more values
• You can adjust text as well
• Linguistic examples
• VC[+nasal]: 
• Vowels become nasalized
• This nasalization is redundant because vowels are already nasalized
• Thus, nasality could add some noise to the vowel (F1 distinction)
• English case:  
• In the case of word 'thank,' it seems that vowel is a bit raised because nasal [n] raises the vowel height, which eventually affects perceptual space
• Just Noticeable Difference (JND):  But think about whether there is anything that is totally free of redundancy. Where is the boundary? 
• English vs. Spanish Vowels: In terms of the number of vowels, Spanish is simpler than English, meaning that assigned space per vowel is broader than that of English. English can distinguish more vowels than Spanish, but it is costly.