Speech – Hank Learning

[Note] Conformer: Convolution-augmented Transformer for Speech Recognition

2022-04-14 2022-04-14 / Speech, Research

SOTA performance on Librispeech
A novel way to combine CNN + Transformer
- to model both local(CNN) and global(Self-attention) dependencies
Conformer
- 4 modules: 0.5 FFN + MHSA + CNN + 0.5 FFN
- Macaron-style half-step resudual FFN
- placing CNN after MHSA is more effective
- swish activation led to faster convergence

2022-04-11 2022-04-11 / Speech, Research

An extension of intermediate-CTC
- Inspired by HC-CTC
- Alternated use syllables(e.g. pinyin) and character as target of intermediate layer
Corpus: CSJ, AISHELL-1 (ideogram language)
Conformer-CTC

2022-03-31 2022-03-31 / Speech, Research

2022-03-21 2022-03-21 / Speech, Research

https://arxiv.org/abs/2006.11477
Self-supervised speech representation
- contrastive loss: masked continous speech input -quantized target
- quantized module: gumbel softmax(latent representation codebooks)
wav2vec2.0 Large with 10min data: 5.2/8.6 LS-clean test
Fairseq
Well explained: https://neurosys.com/wav2vec-2-0-framework

2022-03-17 2022-03-18 / Speech, Research

Motivation

CTC-based models are always weaker than AED models and requrire the assistance of external LM.
- Conditional independence assumption
- Hard to utilize contextualize information

Proposed

Transfer the knowledge of pretrained language model(BERT, GPT-2) to CTC-based ASR model. No inference speed reduction, only use CTC branch to decode.
- Two method:
  - Representation learning: use CIF or PDS(LASO) to align the number of representation.

2022-03-17 2022-03-18 / Speech, Research

wav2vec2.0 has a good robustness against the domain shift, while noise robustness is still unclear.
First analyze the noise robustness of wav2vec2.0 via expeiments
Observation:
- Wav2vec2.0 pretrained on noisy data can obtain good performance on noisy dataset, however brings performance degration on clean set.
Proposed: Enhanced wav2vec2.0
- Noisy and clean speech are fed into the same encoder,

2022-03-16 2022-03-18 / Speech, Research

Self-supervised learning model(wav2vec, HuBERT) gain great success
- Fine-tune the pretrained model is computationally expensive and does not scale well. $O(10^8)$ parameters per task.

Contributions:

Applying adapters to wav2vec2.0 model to reduce the number of parameters required for down-stream tasks.
- Adapters are small trainable modules can be applied into the layers of a frozen pre-trained nedwork for a particular task.

2021-07-19 2022-03-19 / 實作紀錄

初賽靠著隊友 Carry 進決賽，決賽的時候主要負責 MLOps 的部份，分成兩篇文章來分別描述一下初賽時我的方法以及決賽時我們怎麼處理多出來的難關 — 在 AWS 上進行 Incremental training。

本實驗的貢獻：沒有用額外的資料集，也沒有 Pre-trained 模型，只將主辦單位提供的資料做 Augmentation & Pseudo Labeling 的技巧，用 ResNet18 就獲得不錯的 Baseline 成績(Top 10%)。