Truncation of input data for Summarization pipeline
I’m usingbert-large-cnnfor a summarization task. I have been truncating the input text in order to avoid exceeding the maximum sequence length:tokenizer = BartTokenizer.from_pretrained("facebook/bart-large-cnn")
model = BartForConditionalGeneration.from_pretrained(
    "facebook/bart-large-cnn"
)

inputs = tokenizer(
    input_text, return_tensors="pt", max_length=1024, truncation=True
)

outputs = model.generate(
    inputs["input_ids"],
    max_length=300,
    min_length=100,
    length_penalty=2.0,
    num_beams=4,
    early_stopping=True,
)

summary = " ".join(
    tokenizer.decode(
        g, skip_special_tokens=True, clean_up_tokenization_spaces=False
    )
    for g in outputs
)How can I replicate this in SageMaker? I don’t see a way to pass configuration values to the tokenizer when callingpredicton an instance ofHuggingFaceModel? Here is my code so far:env = {
    "HF_MODEL_ID": "facebook/bart-large-cnn",
    "HF_TASK": "summarization",
}

huggingface_model = HuggingFaceModel(
    env=env,
    role=role,
    transformers_version="4.6",
    pytorch_version="1.7",
    py_version="py36",
)

data = {
    "inputs": input_text,
    "parameters": {
        "max_length": 300,
        "min_length": 100,
        "length_penalty": 2.0,
        "num_beams": 4,
    }
}

result = predictor.predict(data)Thank you!
2021-11-16T08:57:46Z
Hello@alistair,You can also providetokenizerskwargsasparameters.For exampledata = {
    "inputs": input_text,
    "parameters": {
        "max_length": 300,
        "min_length": 100,
        "length_penalty": 2.0,
        "num_beams": 4,
        "truncation": True,
    }
}
2021-11-16T10:00:27Z
Thanks for your suggestion@philschmid. This isn’t working for some reason. I am continuing to get the “index out of range in self” error thrown when the maximum sequence length is exceeded.
2021-11-16T15:19:39Z
I could reproduce the issue and also found the root cause of it. The issue thatAsking to truncate to max_length but no maximum length is provided and the model has no predefined maximum length. Default to no truncation.And there is currently no way to pass in themax_lengthto the inference toolkit.There are now 2 options to solve this you could either for the model into your own repository and add atokenizer_config.jsonsimilar to this onetokenizer_config.json · distilbert-base-uncased-finetuned-sst-2-english at main.or you could provide a custominference.pyasentry_pointwhen creating theHuggingFaceModel.e.g.huggingface_model = HuggingFaceModel(
    env=env,
    role=role,
    transformers_version="4.6",
    pytorch_version="1.7",
    py_version="py36",
    entry_point="inference.py",
)Theinference.pythen need to containpredict_fnand amodel_fn. Pseudo code below.def model_fn(model_dir):
    """"model_dir is the location where the model is stored"""
    tokenizer = BartTokenizer.from_pretrained("facebook/bart-large-cnn")
    model = BartForConditionalGeneration.from_pretrained("facebook/bart-large-cnn")
    return model, tokenizer 

def predict_fn(data, model):
    """model is the return of the model_fn and data is the json from the request as python dict"""
   model,tokenzier = model
    outputs = model.generate(
    inputs["input_ids"],
    max_length=300,
    min_length=100,
    length_penalty=2.0,
    num_beams=4,
    early_stopping=True,
    )

   summary = " ".join(
    tokenizer.decode(
        g, skip_special_tokens=True, clean_up_tokenization_spaces=False
    )
    for g in outputs
    )
    return {"summary":summary}You can find more documentation here:Deploy models to Amazon SageMaker
2021-11-16T16:38:39Z
Thank you@philschmidfor explaining the cause of the problem and for providing two good solutions.