[Speech Technology, NLP] Differences Between Hidden Markov Models, Perceptron, and Full Neural Networks

Both HMMs and neural nets, including perceptron, have in common that they basically identify whether an item is a member of the class. However, they are essentially different from each other; HMMs are generative while neural nets are discriminative. Basically, HMMs are used to infer some output variable b given the values of an input variable or pattern a in terms of Bayes theorem, which makes it possible for HMMs to generate a language, not only judge the membership of an item. On the other hand, neural nets are used to compute the probability of an output b given an input a, thus being less model-dependent. In a nutshell, what HMMs represent is p(b|a) while neural nets represent p(a, b).

To be more specific on each model, HMMs are based on Markov chain property, which is the probability of each subsequent state depending solely on what the previous state was, viz. p(qi = a | qi...qi-1) = P(qi = a | qi-1). It is specified with a set of states, transitional probabilities, a sequence of observations, emission probabilities, and initial probability distribution. There are three tasks related to HMMs: determining the likelihood, finding the best hidden state sequence, and training. The first task is to determine the likelihood P(O|λ), given an HMM λ = (A, B) and an observation sequence O. Since the state sequence is hidden, the likelihood is calculated using the forward probability with the one-to-one mapping assumption and the aforementioned Markov assumption.

For instance, imagine that we want to determine the probability of a mask-buying observation sequence like 3 2 3 depending on weather. The computation of the forward probability for the mask-buying observation 3 2 3 from one possible hidden state sequence hot cold hot is P(3 2 3|hot cold hot) = P(3|hot) * P(2|cold) * P(3|hot), but since this is hidden, all possible weather sequences need to be considered. However, computing the total observation likelihood by a separate observation for each hidden state sequence and summation of them is inefficient; thus, the forward algorithm is adopted and the probability of it in the current time step is calculated by summing the previous forward path probability multiplied by transitional probability and the state observation likelihood from 1 to the number of hidden states. Another task is to discover the best hidden state sequence using the Viterbi algorithm, whose mechanism is to find the highest probability of the multiplication of previous Viterbi path probability, transition probability, and state observation likelihood. The other task is HMM training using the forward-backward, or Baum-Welch algorithm, by which transitional and emission probabilities are trained.

While the states of the HMMs are gone through one after another, a vector of input values are provided in a perceptron, one type of neural nets, by which output values are calculated. To be specific, a perceptron consists of input nodes, bias nodes, and output node, and an output node values is the sum of the input values multiplied by the weights of their connect to that output node and the bias values multiplied by their connection to it. For example, suppose that there is a vector of the input values, x <3, 7>, and the output value, b = 11, that the weight of the input value is w <2, -3>, that the bias value is 5, and that the activation function is 2x. The perceptron based on these values results in -8 by 2 * (11+ 3 * 2 + 7 * (-3)). However, a perceptron cannot deal with non-linear relationship, like logistic regression. In other words, the target data for it should be linearly separable. One solution is to use full neural nets. Full neural nets post more than one hidden layer between input and output nodes, and nodes in it have continuous non-linear activation functions, like sigmoid function. At that point, non-linearity can be accommodated.

In addition to the intrinsic difference between HMMs and neural nets, the learning process is also different from each other. Since states cannot be observed in HMMs and can be accessed by only state functions or probabilities, transitional and emission probabilities are learned using the forward-backward algorithm. It basically assigns probabilities in the context of indeterminacy by calculating the counts for sub-paths in one way and then changing the counts in the other way. In contrast to HMMs, in the case of neural nets, in particular the simplest neural net perceptron, weights of the input and the bias nodes are directly adjusted as outputs can be observed during learning.

[NLP - Data] A list of Korean Acoustic Corpora

The Speech Corpus of Reading-Style Standard Korean (NIKL 2005; https://github.com/homink/speech.ko)

• 120 hours (??)
• Read speech 
• 120 speakers -- gender balanced (60 males; 60 females) and the age of the speakers ranged from 19 to 71 at the time of recording in 2003.
• Region: Seoul metropolitan area -- speakers of Seoul dialect 
• Content: 19 well-known short stories and essays containing a total of 930 sentences
• Available format: Each sentence is stored as a separate wav file in the corpus. 
• 120 speakers 
• Around 88,800 audio files

The Korean Corpus of Spontaneous Speech (http://koreascience.or.kr/article/JAKO201521159149292.page)

• In order to get the material, you need to contact the authors.
• 40 hours
• 40 speakers (age and gender -- balanced)
• Interview speech: kind of a monologue: one hour per speaker -- sociolinguistic interview format
• Similar to the Buckeye corpus
• All the utterances are transcribed.

 Pansori-TEDxKR (https://github.com/yc9701/pansori-tedxkr-corpus)

• 3 hours
• Talk speech (monologue)
• 41 speakers -- gender not balanced (32 male and 9 female)
• Regions: Seoul (14), Busan (14) and Daejeon / Daedeok (13)
• Out of 11,704 fragments: ASR corpus is close to 3 hours (2 hours 48 minutes) in audio length (corresponds to 26.4% and 23.6% of the total number of fragments and audio length, respectively).

CloveCall (https://github.com/ClovaAI/ClovaCall)

• 50 hours
• Cleaned: 60,000 utterances (cleaned from a pool: from 11,000 people: each person 10 unique sentences (repeated once or twice)). 
• Read speech: 60,000 pairs of a short sentence and its corresponding spoken utterance in a restaurant reservation domain (only speakers' requests).
• 11,000 speakers (age and gender not sure) -- but only 10 utterance per speaker.

AIHub (https://aihub.or.kr/aihubdata/data/view.do?currMenu=115&topMenu=100&aihubDataSe=realm&dataSetSn=123)

• Large-scale Korean open domain dialog speech corpus from AIHub
• 610 hours: 510 hours (pre-training) , 100 hours (fine-tuning)
• Description:

1. Around 1,000 hours
2. Spontaneous speech
3. 2,000 speakers
4. Conversation between two people about various topics (e.g., weather, economics)
5. ERTI transcription rule
6. File: Segmented at the utterance level (long pause; format: 16kHz/16bits, headerless (endian) linear PCM) and transcribed (format: EUC-KR)

Zeroth (https://github.com/goodatlas/zeroth)

• 95.7 hours 
• Read speech
• 46,347 utterances, 181 speakers, 27,330 uniq. sentences

KsponSpeech (https://aihub.or.kr/aidata/105)

• 69 hours
• general open-domain dialog utterances
• 2000 native Korean speakers in a clean environment
• the dialogue of two people freely conversing on a variety of topics and manually transcribing the utterances
• a dual transcription consisting of orthography and pronunciation, and disfluency tags for spontaneity of speech, such as filler words, repeated words, and word fragments
• For preprocessing, use the script at https://github.com/sooftware/ksponspeech

[Python Code, NLP] How to install KoNLPy (a Korean NLP library) on your Mac using Anaconda?

Follow the instruction on the official page: https://konlpy.org/ko/latest/install/#

If you are using Ubuntu or Window, there might not be a problem (When I installed KoNLPy on my Window machine, I did not have any problem...), but if you are using a Mac and type just pip install konlpy on Terminal, you might see a lot of error messages. This is because most packages processing Korean are based on Java. So, what you need to do first is to install JDK (Java Development Kit) and then download Jpype1. Below is how to do it.

Prerequisite:

• Make a virtual environment using conda
• In doing so, you need to specify python version. I set it as KoNLPy does not work well with higher versions (This might have been fixed now though).

1. Install JDK: https://www.oracle.com/java/technologies/javase-jdk11-downloads.html 

2. Set JAVA environment variable: https://docs.oracle.com/cd/E19182-01/820-7851/inst_cli_jdk_javahome_t/index.html 

(My Mac OS: Mojave)

3. Install Jpype1: conda install -c conda-forge jpype1

4. Intall KoNLPy: pip install konlpy / pip3 install konlpy

5. In order to install Mecab (this is not available on Window), just type the following command in your Terminal.

bash <(curl -s https://raw.githubusercontent.com/konlpy/konlpy/master/scripts/mecab.sh) 

Now, let's test it!

1. Activate your conda virtual environment

2. Then run python3

3. Load KoNLPy, and type some sentences as below

a. Kkma

 

b. Hannanum

 

c. Mecab

 

d. Komoran