Access Tokenizer from Sagemaker BART Endpoint
Is there a way to access the tokenizer used in the HuggingFacePredictor endpoint?I’d like to create a list of nested sentences using the same tokenizer I will use in a downstream task. How would I go about replacingnli_tokenizer()in thecreate_nested_sentences()function below given that I ran the following in my Sagemaker Notebook instance?Any advice  would be greatly appreciated!Sagemaker Endpoint:from sagemaker.huggingface import HuggingFaceModel

hub = {
  'HF_MODEL_ID':'facebook/bart-large-mnli',
  'HF_TASK':'zero-shot-classification'
}

bart = HuggingFaceModel(
    transformers_version='4.6',
    pytorch_version='1.7',
    py_version='py36',
    env=hub,
    role=role,
)

predictor = bart.deploy(initial_instance_count=1,instance_type="ml.m5.xlarge")Excerpt I’d like to replicate in SagemakerThe code below currently works in Google Colab and I’d like to replicate it using my endpoint but I am not sure how to access thenli_tokenizer()from thepredictor()above.!pip install transformers

from transformers import  AutoTokenizer
nli_tokenizer = AutoTokenizer.from_pretrained('facebook/bart-large-mnli')

import spacy
nlp = spacy.load('en_core_web_sm')

def create_nested_sentences(document:str, token_max_length = 1024):
  # Reference: https://discuss.huggingface.co/t/summarization-on-long-documents/920/7
    nested = []
    sent = []
    length = 0
    # Break up text into sentences
    tokens = nlp(document)

    for sentence in tokens.sents:
        # Use the same tokenizer as a downstream inference
        tokens_in_sentence = nli_tokenizer(str(sentence), truncation=True, padding=False)[0] 
        length += len(tokens_in_sentence)
        if length < token_max_length:
          sent.append(sentence)
        else:
          nested.append(sent)
          sent = []
          length = 0

    if sent:
        nested.append(sent)
    return nested

document = '''
The tower is 324 metres (1,063 ft) tall, about the same height as an 81-storey building, and the tallest structure in Paris. Its base is square, measuring 125 metres (410 ft) on each side. 
During its construction, the Eiffel Tower surpassed the Washington Monument to become the tallest man-made structure in the world, a title it held for 41 years until the Chrysler Building in New York City was finished in 1930.
It was the first structure to reach a height of 300 metres. Due to the addition of a broadcasting aerial at the top of the tower in 1957, it is now taller than the Chrysler Building by 5.2 metres (17 ft). 
Excluding transmitters, the Eiffel Tower is the second tallest free-standing structure in France after the Millau Viaduct.
'''
create_nested_sentences(document, token_max_length = 100)
2022-11-29T17:16:07Z
Hi@pleonova, do I understand correctly that you want to use your logic for nested sentences inside the Sagemaker endpoint?
2022-11-29T18:01:04Z
Hi@marshmellow77, I originally had intentions of using the nested sentencesoutsideof the endpoint as a prep step for my long text. However, I guess I could also use it inside a custom function inside thepredict_fn(). I was hoping to use the HuggingFace Sagemaker toolkit implementation off the shelf.My ideal order of operation:Convert long text into nested sentences based on the token length (create tokens usingAutoTokenizer.from_pretrained('facebook/bart-large-mnli'))Feed those nested sentences into aconditional generation model(BartForConditionalGeneration.from_pretrained("facebook/bart-large-cnn"))Feed the summary results from into asequence classification model(AutoModelForSequenceClassification.from_pretrained(nli_model_name))^ I’d like to do the above using Sagemaker Endpoints. I currently have this working on my hugging face apphere.
2022-11-29T20:51:11Z
I see. In that case t seems to me you can override the relevant methods in the SageMaker Hugging Face Inference Toolkit:You can load both models in themodel_fn()methodYou can overrideinput_fn()to pre-process the dataYou can chain theconditional generation modeland thesequence classification modelin thepredict_fn()methodSee thisdocumentationand thisexample. Hope this helps.CheersHeiko
2022-11-29T21:09:57Z
Thank you Heiko! I wasn’t sure if I was missing anything in terms of being able to access the tokenizer from the endpoint without chaining the models in the custompredict_fn().