Very long warning when running rag-end2end-retriever
I’m trying to run rag end2end retriever as here:github.comtransformers/examples/research_projects/rag-end2end-retriever at main ·...main/examples/research_projects/rag-end2end-retriever🤗 Transformers: State-of-the-art Machine Learning for Pytorch, TensorFlow, and JAX. - transformers/examples/research_projects/rag-end2end-retriever at main · huggingface/transformersI encountered very long warning sequences with many tokenizer mismatches. Is this normal? I pasted a part of it - there are many more following this.My environment:pytorch:1.12.1-cuda11.3-cudnn8pytorch-lightning==1.6.4transformers==4.21.2@shamanezSome of the output:The tokenizer class you load from this checkpoint is not the same type as the class this function is called from. It may result in unexpected tokenization.The tokenizer class you load from this checkpoint is ‘DPRQuestionEncoderTokenizer’.The class this function is called from is ‘DPRContextEncoderTokenizerFast’.**followed by tokenizer configThe tokenizer class you load from this checkpoint is not the same type as the class this function is called from. It may result in unexpected tokenization.The tokenizer class you load from this checkpoint is ‘RagTokenizer’.The class this function is called from is ‘DPRQuestionEncoderTokenizer’.**followed by tokenizer configThe tokenizer class you load from this checkpoint is not the same type as the class this function is called from. It may result in unexpected tokenization.The tokenizer class you load from this checkpoint is ‘RagTokenizer’.The class this function is called from is ‘DPRQuestionEncoderTokenizerFast’.Could not locate the tokenizer configuration file, will try to use the model config instead.
2022-08-26T22:43:50Z
yicxu:https://github.com/huggingface/transformers/tree/main/examples/research_projects/rag-end2end-retrieveryeah these things are normal. You’re all good