Issue with sentencepiece tokenizer
Hi, I am using deepset haystack for question generation and get the following error.BlockquoteFile “/Users/kishoregarimella/Coding/workBrowDjango/env/lib/python3.10/site-packages/haystack/nodes/question_generator/question_generator.py”, line 59, ininitself.tokenizer = AutoTokenizer.from_pretrained(model_name_or_path)File “/Users/kishoregarimella/Coding/workBrowDjango/env/lib/python3.10/site-packages/transformers/models/auto/tokenization_auto.py”, line 597, in from_pretrainedreturn tokenizer_class_fast.from_pretrained(pretrained_model_name_or_path, *inputs, **kwargs)File “/Users/kishoregarimella/Coding/workBrowDjango/env/lib/python3.10/site-packages/transformers/tokenization_utils_base.py”, line 1783, in from_pretrainedreturn cls._from_pretrained(File “/Users/kishoregarimella/Coding/workBrowDjango/env/lib/python3.10/site-packages/transformers/tokenization_utils_base.py”, line 1928, in _from_pretrainedtokenizer = cls(*init_inputs, **init_kwargs)File “/Users/kishoregarimella/Coding/workBrowDjango/env/lib/python3.10/site-packages/transformers/models/t5/tokenization_t5_fast.py”, line 134, ininitsuper().init(File “/Users/kishoregarimella/Coding/workBrowDjango/env/lib/python3.10/site-packages/transformers/tokenization_utils_fast.py”, line 113, ininitfast_tokenizer = convert_slow_tokenizer(slow_tokenizer)File “/Users/kishoregarimella/Coding/workBrowDjango/env/lib/python3.10/site-packages/transformers/convert_slow_tokenizer.py”, line 1077, in convert_slow_tokenizerreturn converter_class(transformer_tokenizer).converted()File “/Users/kishoregarimella/Coding/workBrowDjango/env/lib/python3.10/site-packages/transformers/convert_slow_tokenizer.py”, line 426, ininitfrom .utils import sentencepiece_model_pb2 as model_pb2File “/Users/kishoregarimella/Coding/workBrowDjango/env/lib/python3.10/site-packages/transformers/utils/sentencepiece_model_pb2.py”, line 92, in_descriptor.EnumValueDescriptor(File “/Users/kishoregarimella/Coding/workBrowDjango/env/lib/python3.10/site-packages/google/protobuf/descriptor.py”, line 755, innew_message.Message._CheckCalledFromGeneratedFile()TypeError: Descriptors cannot not be created directly.If this call came from a _pb2.py file, your generated code is out of date and must be regenerated with protoc >= 3.19.0.If you cannot immediately regenerate your protos, some other possible workarounds are:Downgrade the protobuf package to 3.20.x or lower.Set PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python (but this will use pure-Python parsing and will be much slower).Please let me know how to resolve this.
2022-07-25T14:21:22Z
It looks like you need to install a different version ofprotobuf. Try runningpip install protobuf==3.20.*in your python environment and then rerun your code.
2022-07-25T14:28:51Z
sjrhuschlee:pip install protobuf==3.20.*Generating the questions now, but it throws the following error:[nltk_data] Error loading punkt: <urlopen error [SSL:
[nltk_data]     CERTIFICATE_VERIFY_FAILED] certificate verify failed:
[nltk_data]     unable to get local issuer certificate (_ssl.c:997)>