BERT

[Note] Improving CTC-based speech recognition via knowledge transferring from pre-trained language models

https://arxiv.org/abs/2203.03582

Motivation

  • CTC-based models are always weaker than AED models and requrire the assistance of external LM.
    • Conditional independence assumption
    • Hard to utilize contextualize information

Proposed

  • Transfer the knowledge of pretrained language model(BERT, GPT-2) to CTC-based ASR model. No inference speed reduction, only use CTC branch to decode.
    • Two method:
      • Representation learning: use CIF or PDS(LASO) to align the number of representation.

[Note] Improving CTC-based speech recognition via knowledge transferring from pre-trained language models Read More »

Facebook Hate Speech Detection

只要有人類的地方就會有惡意言論,而 Facebook 身為全球最大的社交平台,從以往僱用審查團隊去人工檢視,近年來也開始引入 AI 系統來輔助偵測,在 NLP 領域令人振奮的 BERT 系列模型更扮演了關鍵的角色。

本文由黃偉愷Ke-Han Lu 共同完成,是「人工智慧與大數據之商業價值」這門課的期末報告,我們分成兩大方向調查了 Facebook 在惡意言論偵測的近期發展:

Facebook Hate Speech Detection Read More »