BORT: Optimal Subarchitecture Extraction for BERT
Hi guys,Wondering if anyone has read the new paper from the Alexa team regarding BERT size reduction.arXiv.orgOptimal Subarchitecture Extraction For BERTWe extract an optimal subset of architectural parameters for the BERT
architecture from Devlin et al. (2018) by applying recent breakthroughs in
algorithms for neural architecture search. This optimal subset, which we refer
to as "Bort", is...GitHubGitHub - alexa/bort: Repository for the paper "Optimal Subarchitecture...Repository for the paper "Optimal Subarchitecture Extraction for BERT" - GitHub - alexa/bort: Repository for the paper "Optimal Subarchitecture Extraction for BERT"If anyone has any thoughts on it or would like to discuss please comment here.Thanks
2020-12-05T04:16:15Z
Super interesting, thanks for sharing!! Perhaps@VictorSanhcan give us the best commentsWondering if the same technique can be efficiently used for the giant models like T5-11B and GPT-3