Incorporating my tokenizer into huggingface
Hi there,About a year ago my lab released SaGe, a tokenizer that incorporates contextual signals from corpora and thus learns tokens which are more aligned with LM objectives. The paper is here:ACL AnthologyIncorporating Context into Subword VocabulariesShaked Yehezkel, Yuval Pinter. Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics. 2023.Recently, we released a version that’s much faster than the original, better streamlining the corpus for training the vocab. The (python) implementation is here:GitHubGitHub - MeLeLBGU/SaGe: Code for SaGe subword tokenizer (EACL 2023)Code for SaGe subword tokenizer (EACL 2023). Contribute to MeLeLBGU/SaGe development by creating an account on GitHub.We were wondering if and how we can get support for porting SaGe into hf tokenizers and making it a first-class member of the codebase? Would any of your engineers be able to help? What would you need from us?Thanks,Yuvaluvp@cs.bgu.ac.il