Month: April 2022
[Note] Compressing Large-Scale Transformer-Based Models: A Case Study on BERT
- A extensive study on various methods for compressing BERT
- Model size, accuracy, inference speed, device…
- show different advantages and disadvantages of each methods
- gave advices and research directions to future researcher
[Note] Compressing Large-Scale Transformer-Based Models: A Case Study on BERT Read More »
[Note] Conformer: Convolution-augmented Transformer for Speech Recognition
- SOTA performance on Librispeech
- A novel way to combine CNN + Transformer
- to model both local(CNN) and global(Self-attention) dependencies
- Conformer
- 4 modules: 0.5 FFN + MHSA + CNN + 0.5 FFN
- Macaron-style half-step resudual FFN
- placing CNN after MHSA is more effective
- swish activation led to faster convergence
[Note] Conformer: Convolution-augmented Transformer for Speech Recognition Read More »
[Note] InterAug: Augmenting Noisy Intermediate Predictions for CTC-based ASR
https://arxiv.org/abs/2204.00174
- A novel training method for CTC-based ASR using augmented intermediate representations for conditioning
- a extension of self-condition CTC
- Methods: noisy conditioning
- feature space: Mask time or feature
- token space: Insert, delete, substitute token in “condition”.
[Note] InterAug: Augmenting Noisy Intermediate Predictions for CTC-based ASR Read More »
[Note] Multi-sequence Intermediate Conditioning for CTC-based ASR
https://arxiv.org/abs/2204.00175
- An extension of intermediate-CTC
- Inspired by HC-CTC
- Alternated use syllables(e.g. pinyin) and character as target of intermediate layer
- Corpus: CSJ, AISHELL-1 (ideogram language)
- Conformer-CTC
[Note] Multi-sequence Intermediate Conditioning for CTC-based ASR Read More »