Using .generate() with CodeParrot
I’m trying to run prediction usingCodeParrot. I’d like to usegenerate()becausepipelineis too high-level but__call__is too low-level:import transformers
from   transformers import AutoTokenizer, AutoModelForCausalLM, GenerationConfig
  
tokenizer = AutoTokenizer.from_pretrained("codeparrot/codeparrot")
model = AutoModelForCausalLM.from_pretrained("codeparrot/codeparrot")

config = GenerationConfig(
    max_new_tokens=50
)

inputs = tokenizer("def hello_world():", return_tensors="pt")
outputs = model.generate(
    inputs=inputs.input_ids,
    generation_config=config
)

print(tokenizer.decode(outputs[0]))but I get:ValueError: If `eos_token_id` is defined, make sure that `pad_token_id` is defined.This makes no sense to me. Isn’tgenerate()supposed to basically do the work for me? I can run inference manually with this model without having to change model details like the vocabulary… so how comegenerate()isn’t able to figure it out?
2023-04-15T17:58:47Z
Having the same problem. It seems to be related to usingGenerationConfig. Passing parameters directly togenerateworks fine.
2023-04-15T18:02:32Z
Doing this solved the problem for me, although it still feels like a bug related toGenerationConfig:config = transformers.GenerationConfig() 
config.eos_token_id = var.tokenizer.eos_token_id

output_ids = var.model.generate(
  input_ids,
  generation_config = config,
)
2023-06-30T14:30:24Z
This should fix your issuegeneration_output = model.generate(input_ids=input_ids,pad_token_id=tokenizer.eos_token_id,generation_config=generation_config