[Speech Technology, NLP] Differences Between Hidden Markov Models, Perceptron, and Full Neural Networks

Both HMMs and neural nets, including perceptron, have in common that they basically identify whether an item is a member of the class. However, they are essentially different from each other; HMMs are generative while neural nets are discriminative. Basically, HMMs are used to infer some output variable b given the values of an input variable or pattern a in terms of Bayes theorem, which makes it possible for HMMs to generate a language, not only judge the membership of an item. On the other hand, neural nets are used to compute the probability of an output b given an input a, thus being less model-dependent. In a nutshell, what HMMs represent is p(b|a) while neural nets represent p(a, b).

To be more specific on each model, HMMs are based on Markov chain property, which is the probability of each subsequent state depending solely on what the previous state was, viz. p(qi = a | qi...qi-1) = P(qi = a | qi-1). It is specified with a set of states, transitional probabilities, a sequence of observations, emission probabilities, and initial probability distribution. There are three tasks related to HMMs: determining the likelihood, finding the best hidden state sequence, and training. The first task is to determine the likelihood P(O|λ), given an HMM λ = (A, B) and an observation sequence O. Since the state sequence is hidden, the likelihood is calculated using the forward probability with the one-to-one mapping assumption and the aforementioned Markov assumption.

For instance, imagine that we want to determine the probability of a mask-buying observation sequence like 3 2 3 depending on weather. The computation of the forward probability for the mask-buying observation 3 2 3 from one possible hidden state sequence hot cold hot is P(3 2 3|hot cold hot) = P(3|hot) * P(2|cold) * P(3|hot), but since this is hidden, all possible weather sequences need to be considered. However, computing the total observation likelihood by a separate observation for each hidden state sequence and summation of them is inefficient; thus, the forward algorithm is adopted and the probability of it in the current time step is calculated by summing the previous forward path probability multiplied by transitional probability and the state observation likelihood from 1 to the number of hidden states. Another task is to discover the best hidden state sequence using the Viterbi algorithm, whose mechanism is to find the highest probability of the multiplication of previous Viterbi path probability, transition probability, and state observation likelihood. The other task is HMM training using the forward-backward, or Baum-Welch algorithm, by which transitional and emission probabilities are trained.

While the states of the HMMs are gone through one after another, a vector of input values are provided in a perceptron, one type of neural nets, by which output values are calculated. To be specific, a perceptron consists of input nodes, bias nodes, and output node, and an output node values is the sum of the input values multiplied by the weights of their connect to that output node and the bias values multiplied by their connection to it. For example, suppose that there is a vector of the input values, x <3, 7>, and the output value, b = 11, that the weight of the input value is w <2, -3>, that the bias value is 5, and that the activation function is 2x. The perceptron based on these values results in -8 by 2 * (11+ 3 * 2 + 7 * (-3)). However, a perceptron cannot deal with non-linear relationship, like logistic regression. In other words, the target data for it should be linearly separable. One solution is to use full neural nets. Full neural nets post more than one hidden layer between input and output nodes, and nodes in it have continuous non-linear activation functions, like sigmoid function. At that point, non-linearity can be accommodated.

In addition to the intrinsic difference between HMMs and neural nets, the learning process is also different from each other. Since states cannot be observed in HMMs and can be accessed by only state functions or probabilities, transitional and emission probabilities are learned using the forward-backward algorithm. It basically assigns probabilities in the context of indeterminacy by calculating the counts for sub-paths in one way and then changing the counts in the other way. In contrast to HMMs, in the case of neural nets, in particular the simplest neural net perceptron, weights of the input and the bias nodes are directly adjusted as outputs can be observed during learning.

[Speech Technology] What is Vector Quantization?

The size of training data should be large enough for reliable values to be driven for all

the parameters. However, the larger the number of feature vectors is, the more possible

values for each feature are. Not only is this memory-inefficient, but also this is problematic

because many feature vectors will not occur at all in the training data. One solution for these

problems is using Vector Quantization (VQ).

VQ is a data compression technique. It does not deal with all the feature vectors, but only

some centroids of them, which can be obtained through Euclidean distance.

As a simple example, if we want to represent 0 to 7 in one dimension. 3 bits (by 23 = 8)

are needed to do so. However, if we apply VQ to this, only 2 bits (by 22 = 4) are needed

(4 centroids in 4 clusters: 1 in 0 to 1.99; 2 in 2 to 3.99; 5 in 4 to 5.99; and 7 in 6 to 7.99).

So, if “41371512” is the target, 24 bits (by 3 * 8) are needed without VQ while 16 bits (by 2 * 8)

with VQ. However, they are represented in different ways: VQ version is

with centroids “51271512” while non-VQ version is as it is. VQ also can be applied

to more than one dimensions. For example, if there are 16 dots in a two-dimensional space

(plane), 4 bits (by 24 =16) per data value are needed, but if 16 dots are represented

with 4 centroids, only 2 bits (by 22 = 4) per data value are needed.

To be more specific on how this works, let’s take another example.

If there are 4,096 dots in a plane, 12 bits (by 212 = 4,096) per data value are needed,

but if these are represented with 16 centroids through such clustering techniques as k-means,

only 4 bits (24 = 16) per data value are needed. All these centroid values (e.g., (2, 5)) are to be

saved as vector values and will be assigned vector numbers

(i.e., 0000, 0001, 0010, … 1111 in this case). These values are further to be saved

in the codebook, with vector values as codebook values and vector numbers as codebook entries.

If a codebook is established with a training data set as such, a sequence of acoustic data

in the form of vectors can be represented with codebook entries. For instance, if (2, 5) is saved

as entry 3 in the codebook and one of the vectors in the new acoustic data is (2, 4.9), it is going

to be represented as the entry number of 3.

In short, VQ is a data compression method, by which only some representative vectors of clusters

are dealt with. As the number of feature vectors is increasing, the effect of VQ is also increasing.

[Phonetics, Speech Technology] What are Fundamental Frequency, Harmonics, and Formants?

The human vocal tract is a kind of resonator, with one fixed-end (i.e., vocal folds) and the other open-end (i.e., lips),

which is acting as a filter in terms of source-filter theory. Regarding sound source, there are largely two types of it:

glottal source and supra-glottal source. Glottal source can be periodic (i.e., voiced), aperiodic (i.e., whisper),

or mixed (i.e., breathy) while supra-glottal source is almost always aperiodic (i.e., noise).

Of the two sources, glottal source, in particular vocal folds, is pertinent to the fundamental frequency, harmonics, and formants,

because they are essentially periodic.

Among them, the fundamental frequency and harmonics are properties of the source,

exactly vocal folds while formants are those of the filter, exactly the vocal tract.

First of all, a complex wave, according to Fourier theorem, consists of simple waves, and

each simple wave is referred to as harmonic. The first harmonic is referred to as t

he fundamental frequency, which is determined by the number of the vocal folds' vibration

in one second, and the following harmonics, also called overtones, are multiple integers of it.

For instance, if the fundamental frequency, the first harmonic, is 100 Hz, the second harmonic

should be 200 Hz, the third harmonic should be 300 Hz, and so on.

When those harmonics are filtered through the vocal tract, some are going to be dampen out

while others are going to be amplified by the length and shape of the vocal tract. The harmonics

filtered through the vocal tract are called formant frequencies, essentially resonant frequencies.

The first three formant frequencies, each called the first formant (F1), the second formant (F2),

and the third formant (F3), are widely used by the hearer to categorize speech sounds,

in particular vowels.

In short, overtones are multiple integers of the fundamental frequency at the vocal cord,

harmonics are embracing the fundamental frequency and overtones, and formants are

resonant frequencies, viz. harmonics filtered through the vocal tract.

Lastly, formants are determined by the length and shape of the vocal tract and can be

calculated by Fn = (2n-1)*c/4L. 'n' represents the number of the resonance,

'c' speed of sound (mostly in air), which is around 35,000 cm/sec, and 'L' the length

of the resonator, here the length of the vocal tract. If one's vocal tract is 17.5cm long and

one is going to produce a schwa, which does not much affect the shape of the vocal tract,

the first three formants can be calculated as in the following table.

The number of the resonance | Formant frequencies

:----------:|:---------------------

1 (F1)|35,000cm/sec / 17.5cm*4 = 500 Hz

2 (F2)|35,000cm/sec / 17.5cm*4/3 = 1,500 Hz

3 (F3)|35,000cm/sec / 17.5cm*4/5 = 2,500 Hz

[Speech Technology] What is Glow Text-To-Speech (TTS) model? (A brief summary)

I will upload a full paper review with codes soon :)

Text-to-Speech (TTS) is a task that speech is generated from text. TTS models, such as Tacotron 2 (Wang et al., 2017) and Deep Voice (Arik et al., 2017), are generative models that synthesize speech from text. It has been widely studied how to develop better models, especially with regard to expressiveness and speed. Tacotron 2 demonstrates excellent expressiveness – it produces a good quality of output; however, as it is an autoregressive model, the inference time increases linearly with the output. On the one hand, such models as FastSpeech (Ren et al., 2019) and ParaNet (Peng, Ping, Song, & Zhao, 2020) generate mel-spectrograms from text in parallel, which alleviates the speed issue caused by autoregressive TTS models. However, both models are dependent upon pre-trained autoregressive TTS models to extract alignments. To address these issues, Glow-TTS model was proposed.

Glow-TTS model is a flow-based parallel model. Unlike FastSpeech and ParaNet, it does not need external alignments. Rather, Glow-TTS can learn its own alignment by incorporating the properties of dynamic programming, which hidden Markov models (HMMs) and Connectionist Temporal Classification (CTC) utilizes. As with other deep learning-based models, it mainly consists of encoder and decoder. The encoder receives a text sequence and processes it using the encoder pre-net and Transformer encoder. Thereafter, the statistics of prior distribution and duration are predicted through the last projection layer and duration predictor of the encoder, respectively. The decoder receives a mel-spectrogram. It is processed through a bunch of flow blocks, each of which contains activation normalization layer, affine coupling layer, and invertible 1 * 1 convolution layer. The output is reshaped to make equal to the input size. 

[Paper Review - Speech Technology] Automatic Voice Onset Time Estimation from Reassignment Spectra (Stouten and Van hamme, 2009)

 https://docs.google.com/presentation/d/1vjSGb4IdWlY6zVRw4Of9kGk-SE1ax07t-NmiujSvGuo/edit?usp=sharing

[Speech Technology in Korean] Bit Depth 개념

 아날로그 신호를 디지털 신호로 바꾸는 과정은 표본화(samping) -> 양자화 (quantization) -> 부호화 (coding)의 과정을 거침

표본을 수집하고 (sampling rate), 수집한 표본의 진폭을 결정한 후 (bit), 컴퓨터가 인식할 수 있도록 2진법으로 바꿔줌

그래서 PCM방식에서는 sampling rate는 시간의 가로축을 담당하고, bit depth는 진폭의 세로축을 담당

예컨대, 44.1khz 16bit: 1초에 65536개의 선택 가능한 크기에 44100번 기록했다는 말

아날로그 신호는 연속된 값이므로 정확한 기록은 불가능하여 신호가 입력되면 위의 그림과 같이 bit에 따라서 제일 가까운 값을 인식해서 그것을 기록 - 원래 파형과 최대한 비슷하게!

양자화 에러: 실제 아날로그 값과 아날로그 값을 나타내는 수치와의 차이를 뜻하는 것, 실제 아날로그 값이 양자화 단계에서 떨어질 때 발생. 이 과정에서 가장 가까운 수치를 할당 받는데, 이 에러때문에 디지털 오디오에서 잡음이 발생

SNR (Signal to Error Ratio): 에러와 신호 사이의 값

아날로그 신호에서는 아날로그 시스템의 전반적인 dynamic range를 말하고, 디지털 신호에서는 계단형식의 샘플레벨을 정확히 표현하는 것

6N + (1.8dB) (N=bit), 16bit라면 96dB, 24bit라면 144dB

Bit가 높을수록 noise floor가 낮아지고 dynamic range가 증가하게 됌?

Bit truncation: 높은 bit에서 낮은 bit로 다운 시킬 때 그게 맞는 dither를 일부러 넣어주어 노이즈를 감소

[Speech Technology in Korean] Mel-Frequency Cepstral Coefficients (MFCC) 개념

 MFCC는 입력 음성을 짧은 구간으로 나누고 이 잘개 쪼개진 음성을 프레임이라고 함

프레임 각각에 푸리에 변환을 실시해 해당 구간 음성에 담긴 주파수 정보를 추출

모든 프레임 각각에 푸리에 변환을 실시한 결과를 스펙트럼 이라고 함

스펙트럼에 사람의 말소리 인식에 민감한 주파수 영역대는 세밀하게 보고 나머지 영역대는 상대적으로 덜 촘촘히 분석하는 필터(mel filter bank)를 적용. 이를 멜 스펙트럼(mel spectrum)이라고 함. 여기에 로그를 취하면 로그 멜 스펙트럼(log-mel spectrum). mfcc는 로그 멜 스펙트럼에 역푸리에변환을 적용해 주파수 도메인의 정보를 새로운 시간 도메인으로 바꾼 것을 가리킴

mfcc는 인간의 말소리 인식에 중요한 특질들이 추출된 결과

참고자료:

https://haythamfayek.com/2016/04/21/speech-processing-for-machine-learning.html

[Bash Code - Speech Technology] A Tip to Install Sphinx for OpenSmile 3.0

Environment

My OS environment: macOS Mojave Version 10.14.6

I am not using Conda, and trying to avoid using Homebrew. 

Procedure

If OpenSmile 3.0 is successfully installed, you will see “sphinx” made under the “doc” folder. Before installing what is listed in “requirements.txt,” you might need to install “sphinx.” I first tried "sphinx_build" command, but it did not work. I thought there was something wrong with Sphinx configuration. So, after activating a virtual environment, I installed and configured “sphinx.” Below is a description of the whole processes that I took:

1. Activate your virtual environment
2. Install “sphinx” using pip install sphinx
3. Check whether it works with the following command: sphinx-build --help
4. Go to the “sphinx” folder
5. Install what is list in “requirements.txt”: pip install -r requirements.txt

IMPORTANT: You would need to change “sphinxcontrib-bibtext” to “sphinxcontrib.bibtext” before running this code. I had an error about this and then realized that that was because of this difference. 

6. Neither make clean nor make html did work for me; only make clean worked. So, I used the followings:
   1. sphinx-build -b html -d ./_build/doctrees . ./_build/html/
   2. sphinx-build -b latex -d ./_build/doctrees . ./_build/latex/

[Paper Review - Speech Technology, NLP] WaveNet: A Generative Model for Raw Audio (van den Oord et al., 2016)

Abstract

WaveNet: a generative deep neural network that generates raw audio waveform

i) fully probabilistic (modeling using random variable and probability distribution)

ii) autoregressive (the predictive distribution for each audio sample conditioned on all previous ones)

text-to-speech, a single WaveNet can capture the characteristics of many different speakers with equal fidelity, music generation, phoneme recognition, etc. 

1. Introduction

Modeling joint probabilities over pixels or words using neural architectures as products of conditional distributions yields SOTA generation.

WaveNet, an audio generative model based on the PixelCNN (van den Oord et al., 2016a;b)

Dilated causal convolution in order to deal with long-range temporal dependencies needed for raw audio generation

Contributions: 

i) natural raw speech signal in TTS

ii) a single model can be used to generate different voices

iii) strong results when tested on a small speech recognition dataset and promising when used to generate other audio modalities such as music

2. WaveNet

i) each audio sample x_t is therefore conditioned on the samples at all previous timesteps

ii) a stack of convolutional layers

iii) no pooling layers, thus the time dimensionality of output and that of input is the same as each other

iv) output: categorical value via softmax

v) optimization: update parameters to maximize log likelihood

Dilated causal convolutions

Causal convolution

i) By using causal convolutions, we make sure the model cannot violate the ordering in which we model the data.

ii) the prediction emitted by the model at timestep t only depends on the previous timesteps

iii) training: parallel, generation: sequential

iv) no recurrent connection, thus fast

v) require many layers, or large filters to increase the receptive field --> Dilated convolution can tackle this!

Dilated convolution

i) A dilated convolution is a convolution where the filter is applied over an area larger than its length by skipping input values with a certain step

ii) large receptive fields with fewer layers

iii) input shape kept (less information loss)

iv) more discriminative because of more non-linear calculations

Softmax Distributions

i) a softmax distribution tends to work better because a categorical distribution is more flexible and can more easily model arbitrary distributions because it makes no assumptions about their shape

ii) quantization: mu-law companding transformation to reduce the number of possible values

iii) nonlinear (mu-law compounding) > linear (ReLu)

Gated Activation Units: the same gated activation unit as used in the gated PixelCNN

i) Element-wise multiplication of filter and gate

ii) Filter: dilated convolution and then tanh activation / local features from a certain layer

iii) Gate: dilated convolution and then sigmoid activation / decide how much the information of the filter will be passed to the next layer

Residual and Skip Connections

i) to speed up convergence and enable training of much deeper models

ii) 1 * 1 convolution: less calculation, shaping

Conditional WaveNets

i) an addional input h to model the conditional distribution p(x|h) of the audio

ii) by conditioning the model on other input variables, we can guide WaveNet's generation to produce audio with the required characteristics

iii) conditioning the model on other inputs in 2 ways: global conditioning (a single latent representation h that influences the output distribution across all timesteps) and local conditioning (timeseries h_t; transform this series using a transposed convolutional network that maps it to a new time series y = f(h) with the same resolution as the audio signal, which is then used in the activation unit)

Context Stacks: another way to increase the receptive field

i) a complementary approach is to use a separate, smaller context stack that processes a long part of the audio signal and locally conditions a larger WaveNet that processes only a smaller part of the audio signal

ii) Shorter-range WaveNet is the main model + local conditioning

3. Experiments

Multi-speaker-speech generation

i) Dataset: English multi-speeaker corpus from CSTR voice cloning toolkit

ii) The conditioning was applied by feeding the speaker ID to the model in the form of a one-hot vector

iii) Adding speakers resulted in better validation set performance compared to training solely on a single speaker

iii) The model picked up on other characteristics in the audio apart from the voice itself

Text-To-Speech

i) Dataset: North American English and Mandarin Chinese dataset

ii) locally conditioned on linguistic features which were derived from input texts; the logarithmic fundamental frequency values in addition to the linguistic models

iii) WaveNet (both linguistic features + fundamental frequency) was the winner!

Music

i) Dataset: MargnaTagATune dataset, YouTube piano dataset

ii) conditional music models, which can generate music given a set of tags specifying e.g. genre or instruments

Speech Recognition

i) Dataset: TIMIT

ii) discrimination task

iii) For this task we added a mean-pooling layer after the dilated convolutions that aggregated the activations to coarser frames spanning 10 ms.

iv) best score

4. Conclusion

i) Deep generative model for audio data (waveform)

ii) autoregressive, dilated convolution

iii) can be conditioned on other in a global (e.g., speaker identity) or local way (e.g., linguistic features)

iv) promising results when applied to music audio modeling and speech recognition

[Book Summary - Speech Technology] Introduction to Speech Technology (Holmes & Holmes, 2001)

Ch 4. Digital coding of speech

• 3 things to consider in speech coding: data rate, speech quality, and algorithm complexity
• Human cognitive processes cannot take account of an information rate in excess of a few tens of bis per second, thus implying a ratio of information transmitted to information used of between 1,000 and 10,000
• 2 properties of speech communication: the restricted capacity of the human auditory system and the physiology of the speaking mechanism (based on the fact that the signal is known to be produced by a human talker)
• 3 coding methods: simple waveform coders, analysis/synthesis systems, and intermediate systems

Simple waveform coders

• copy the actual shape of the waveform produced by the microphone and its associated analogue circuits
• consist of a band limiting filter, a sampler, and a device for coding the samples
• types of simple waveform coders
  1. Pulse code modulation (PCM)i) used for feeding analogue signals into computers or other digital equipment for subsequent processingii) not normally used due to the high required digit rateiii) does not exploit the above two properties (i.e., speech production and/or auditory perception), except for the limited bandwidth
  2. Deltamodulation: uses its transmitted digital codes to generate a local copy of the input waveform and chooses successive digital codes

Analysis/synthesis systems (vocoders)

• analyze the speech signal in terms of parameters
• the output does not need to resemble the the original waveform in appearance, but should be perceptually similar
• based on a model of speech production
• the data are coded into frames representing speech spectra measured at 10-30 ms
• types of vocoders
  1. channel vocoders i) principle: the spectrum is represented by the response of a bank of contiguous variable-gain bandpass filters, and the control signals for the channels are derived by measuring the short-term-average power from a similar set of filters fed with the input speech signal in the transmitter, ii) limitation: need a large number of channels, iii) performance: 15-20 channels reasonable for communication purposes; operating at data rates of around 2,400 bits/s or lower
  2. sinusoidal coders i) representing the short-term spectrum of a speech signal as a sum of sinusoids specified in terms of frequency, amplitude, and phase (called sinusoidal transform coding (STC), ii) voiced speech and unvoiced speech represented by harmonically related sinusoids for the former and with random phases for the latter, iii) Multi-band excitation (MBE) coding: voiced as a combination of the relevant set of harmonic sinusoids; unvoiced using a frequency-domain method with regard to a whitenoise excitation signaliv) 2,000-4,000 bits/s range
  3. linear predictive coding (LPC) vocoders
  4. formant vocoders

Intermediate systems

• utilize both simple waveform coders and analysis/synthesis system

Ch. 8 Automatic speech recognition

General principles of pattern matching

• early methods (e.g., Hyde (1972): rule-based approaches, which was not very successful due to co-articulation and difficulty of phone identification
• pattern-matching techniques: one way is to store example acoustic patterns (called templates) for all the words

Distance metrics

• Filter-bank analysis: describes the speech as a sequence of feature vectors, which can be compared with stored templates for all the words in the vocabulary using a suitable distance metric
• Normalization
  1. adding a small consonant to the measured level before taking logarithms
  2. the square of the Euclidean distance in the multi-dimensional space
• Dynamic time warping (DTW)
  • can deal with sequences of connected words ( ⇒ can solve the end-point detection problem)
  • by matching one word on to another in a way which applies the optimum non-linear timescale distortion to achieve the best match at all points
  • Score pruning: not allowing paths from relatively badly scoring points to propagate further in the DTW calculation

Speech recognition

• Speech recognition: store the representation and choose the best matching
• 3 broad kinds of synthesis
  1. Parametric: manipulate the acoustic?
  2. Physical (Frankenstein):
  3. Concatenative (cheating): using pre-recorded units
• The synthesis stage
  1. a decoding process: the waveform of the acoustical units must be reconstructed from their coded version
  2. a concatenation process: the sequence of acoustical units must be concatenated after an appropriate modification of their intrinsic prosody
• Types of synthesis
• Concatenative synthesis: synthesize the sounds by concatenating such elementary speech units as diphones and demi-syllables→ Pitch adjustment is a problem
• Some pitch adjustment techniques
  
  1. Linear Predictive Coding (LPC): an encoding technique. Do multiple regression on the wave samples to predict any particular sample from the n samples that precede it.Example) w0, w1, w2, w3, w4, w5, w6, w7, w8, w9 (10-sample windows)c1w0 + c2w1 + c3w2 = w3c1w1 + c2w2 + c3w3 = w4...c1w6 + c2w7 + c3w8 = 29→ calculate c1, c2, and c3 by the autoregression→ can be used to reconstruct waves
  2. Pitch Synchronous Overlap and Add (PSOLA): a digital signal processing technique. The speech waveform is divided into small overlapping segments, and then they are moved to further apart or closer together, in order to change pitch.

Ways to do pattern matching

• Length
• Filter banks: take a wave, break it into time windows, make a spectrum of each window, and then quantize those spectra into a vector of machines→ But how to deal with the data with the different duration? Dynamic time warping (finding an optimal path between two given (time-dependent) sequences)

Front-end analysis

• Things to consider: i) naturalness, ii) computational efficiency, and iii) results (performance assessment)
• with regard to filter banks: i) overlapping time windows, ii) amplitude scaling, and iii) frequency scaling→ about iii) frequency scaling:
  1. Mels: the idea is that low-frequency differences are precisely than high-frequency differences to humans, so scaling is done accordingly)
  2. Cepstrum: a way of frequency decomposition for a time window—creating a spectrum via Fourier analysis, log-transform it, and do the Fourier again
  3. MEL frequency cepstral coefficients (MFCC)
  1. 3-1. 25ms overlapped time windows
  2. MEL-scaled cepstra
  3. keep the first cepstral filterbanks
  4. compute 12 additional delta values (a difference score between a coefficient and its next two horizontal neighbors)
  5. compute another 12 delta-delta values

Hidden Markov Models (HMMs)

• A generative model that models sequential data stochastically
• Markov model: represents sequence that consists of states as state transition probability matrix
• Markov assumption: the observation at the time t is dependent on the latest r observations
• The basic structure: a finite number of states, a designated start state, a finite alphabet, a finite number of arcs, a probability associated with each arc, and for each state, a probability associated with each symbol
• It is supposed that the current state is hidden and draws out it based on the shown information. The direct cause that results in it is unobservable; in other words, only what resulted from the state(s) can be seen. Keep in mind that what are hidden are not parameters, but continuous states the model goes through.
• P(W|A) = P(A|W)P(W)/P(A), A ⇒ O, W ⇒ lambda, P(A|W) ⇒ P(O|lambda)
• Forward and backward procedures: to calculate the probability given the sequence of the status and the model (lambda)
• The Viterbi algorithm: to find the optimal status sequence to explain the data most well
• Baum-Welch re-estimation procedure: to optimize the lambda (parameters of the model) for maximizing P(O|lambda)

https://blog.naver.com/jamiet1/221420317965

Some concepts

• Finite state transducers (FST): a finite state automation (FSA), which produces output as well as reading input, meaning it is useful for parsing while a bare FSA can only be used for recognizing (i.e., pattern matching)