Import distilbert-base-uncased tokenizer to an android app along with the tflite model
I have converted the model (.h5) to tflite using:converter = tf.lite.TFLiteConverter.from_keras_model(model)
converter.target_spec.supported_ops = [tf.lite.OpsSet.TFLITE_BUILTINS, 
                                       tf.lite.OpsSet.SELECT_TF_OPS]
converter.optimizations = [tf.lite.Optimize.DEFAULT]
tflite_model = converter.convert()
open("/models/tflite_models/5th_Jan/distilbert_sms_60_5_jan.tflite", "wb").write(tflite_model)but I also need tokenizer to run the model locally on the android app (independent of internet availability).According to the articles on internet and question answered on stackoverflowHow to tokenize input text in android studio to process in NLP model?we need json file of tokenizers to tokenize words in new inputs.When I run the following code:import json

with open( 'android/word_dict.json' , 'w' ) as file:
    json.dump( tokenizer.word_index , file )The following error comes:AttributeError: 'DistilBertTokenizer' object has no attribute 'word_indexI am unable to find solution to use tokenizer of distilbert-base-uncased in android app. Any help will be appreciated. Thanks.
2021-12-28T14:45:04Z
excuse me did you solve it ?
2021-12-29T05:35:09Z
word_indexis an attribute of Keras’ own tokenizer, not Hugging Face. See docshere.
2021-12-29T23:20:23Z
excuse me . what is the equivalent manner in hugging face ?