Wav2vec2CTCTokenizer and vocab.json
I am using Wav2Vec2CTCTokenizer.from_pretrained to read in the Facebook base librispeech model:tokenizer = Wav2Vec2CTCTokenizer.from_pretrained(‘facebook/wav2vec2-base-960h’)I am seeing some behavior I am not sure I follow. It seems that if I have a vocab.json file already in the same directory from where I am running the above command, it ignores the vocab.json file in the base model and uses the one in my directory. Is this correct, and if so, where is this happening in the source code - I cannot find it.
2022-10-28T23:00:01Z
github.comhuggingface/transformers/blob/main/src/transformers/tokenization_utils_base.py#L1570`tokenizer.get_vocab()[token]` is equivalent to `tokenizer.convert_tokens_to_ids(token)` when `token` is in thevocab.Returns:`Dict[str, int]`: The vocabulary."""raise NotImplementedError()@classmethoddef from_pretrained(cls, pretrained_model_name_or_path: Union[str, os.PathLike], *init_inputs, **kwargs):r"""Instantiate a [`~tokenization_utils_base.PreTrainedTokenizerBase`] (or a derived class) from a predefinedtokenizer.Args:pretrained_model_name_or_path (`str` or `os.PathLike`):Can be either:- A string, the *model id* of a predefined tokenizer hosted inside a model repo on huggingface.co.Valid model ids can be located at the root-level, like `bert-base-uncased`, or namespaced under a
2022-10-29T19:29:40Z
I am sorry, could you elaborate more? I still don’t see why it does not default to the base model vocab file - where/how does it wind up taking something from my directory.