[Book Summary - Information Theory] Information Theory: A Tutorial Introduction, Ch 5 by James V. Stone

1. Entropy: Average surprisal 

2. Discrete variables  

• Continuous variables: divide the continuum into bins  
• Depending the number of bins, entropy changes
• If the number of bins increases, thus each bin size being smaller, entropy increases due to more options being made.

• Differential entropy
• Transforming continuous variables
• Changing the range of a discrete set of variables doesn't change the entropy
• Example: In the case of a binary dice, which features only 0 and 1, the output should be 0 or 1 even though either of them is divided by infinite numbers between 0 and 1 
• Changing the range of a continuous variable does, because the entropy is based on bin-width
• Doubling the range -> doubles the number of bins -> adds one bit, even if half of the bins aren't used
• What about adding a constant? 
• As the term "constant" implies, It doesn't change entropy because range (variance) remains the same. 
• Maximum entropy distributions
• What distribution can we engineer so that entropy is highest? 
• Fixed upper/lower bounds 
• Fixed mean, with all values >= 0 ==> exponential 
• Fixed variance (e.g., power) 
• Back to differential entropy
• Infinitely accurately…
• What in practice limits 'bin sizes'?
• Noise!
• If the amount of noise increase, it is getting harder to know what the actual signal was
• How does noise limit bin-sized connection to transmission of information in language?
• More noise, less precision due to the number of bins being smaller because of noise (i.e., log1/delta x decreases)
• Zipf's law: Infrequent words => longer, lexicon is limited        

• Bin size => Amount of signal! 

[Book Summary - Information Theory] Information Theory: A Tutorial Intorudction, Ch 4 by James V. Stone

     1. Mutual information 

• The average surprisal of the symbols in the system 
• The entropy of a single symbol = its surprisal 
• Conditional entropy H(X|Y) = noise entropy H(Y|X) 
• Accuracy of transmission --> mutual information (overlapping part is information transmitted)

• 2. Joint probability of two systems 

• 
  

• 3. Mutual information and information rate

• For a channel with inputs from (X) and outputs from (Y), the information rate is determined by their mutual information, which is in turn related to H(X), H(Y), and their relationship  
• What's the situation with: 
• High vs. Low input H? input entropy 6; output entropy 3 
• Let's think about tossing a dice and assume that we consider either odd numbers only or even numbers only; it should be wasting information without noise cases
• High vs. Low output H? 
• High vs. Low independence of input and output?
• If totally independent, there should be no overlapping portion in a Venn diagram.
• The input/output entropy should be high, and the noise should be low   

• 4. Crosstalk 

• A case of loss information: Substitution error
• Adding "redundancy" (help compensate for information loss) might be one way to solve such loss

• 5. Redundancy is good and bad?

• Increase the rate at which information can be transmitted (Resource cost; e.g., articulatory effort)
• But increase the accuracy with which it is transmitted  

• 6. Noisy typewriter example

•  

• 7. How to add redundancy?

• Adding more values
• You can adjust text as well
• Linguistic examples
• VC[+nasal]: 
• Vowels become nasalized
• This nasalization is redundant because vowels are already nasalized
• Thus, nasality could add some noise to the vowel (F1 distinction)
• English case:  
• In the case of word 'thank,' it seems that vowel is a bit raised because nasal [n] raises the vowel height, which eventually affects perceptual space
• Just Noticeable Difference (JND):  But think about whether there is anything that is totally free of redundancy. Where is the boundary? 
• English vs. Spanish Vowels: In terms of the number of vowels, Spanish is simpler than English, meaning that assigned space per vowel is broader than that of English. English can distinguish more vowels than Spanish, but it is costly.

[Speech Technology] What is Glow Text-To-Speech (TTS) model? (A brief summary)

I will upload a full paper review with codes soon :)

Text-to-Speech (TTS) is a task that speech is generated from text. TTS models, such as Tacotron 2 (Wang et al., 2017) and Deep Voice (Arik et al., 2017), are generative models that synthesize speech from text. It has been widely studied how to develop better models, especially with regard to expressiveness and speed. Tacotron 2 demonstrates excellent expressiveness – it produces a good quality of output; however, as it is an autoregressive model, the inference time increases linearly with the output. On the one hand, such models as FastSpeech (Ren et al., 2019) and ParaNet (Peng, Ping, Song, & Zhao, 2020) generate mel-spectrograms from text in parallel, which alleviates the speed issue caused by autoregressive TTS models. However, both models are dependent upon pre-trained autoregressive TTS models to extract alignments. To address these issues, Glow-TTS model was proposed.

Glow-TTS model is a flow-based parallel model. Unlike FastSpeech and ParaNet, it does not need external alignments. Rather, Glow-TTS can learn its own alignment by incorporating the properties of dynamic programming, which hidden Markov models (HMMs) and Connectionist Temporal Classification (CTC) utilizes. As with other deep learning-based models, it mainly consists of encoder and decoder. The encoder receives a text sequence and processes it using the encoder pre-net and Transformer encoder. Thereafter, the statistics of prior distribution and duration are predicted through the last projection layer and duration predictor of the encoder, respectively. The decoder receives a mel-spectrogram. It is processed through a bunch of flow blocks, each of which contains activation normalization layer, affine coupling layer, and invertible 1 * 1 convolution layer. The output is reshaped to make equal to the input size.