Unable to generate more than one token at a time using website API
I fine-tuned OPT 350M to create a model that extracts addresses from natural text. For example:Input:The leased property is located at 3500 S Gessner Rd Ste 200, where the tenant will have access to the premises for the duration of the lease agreement.Generated text:&& 3500 S GESSNER RD STE 200 ...I pushed my model to the HF hub, here:piazzola/address-detection-model · Hugging Face. When I use the model from the Python interpreter, I am able to generate more than one token at a time to get the output. I would like people to be able to do this using the Hosted Inference API available on the model card page linked above. However, I noticed that if I click the “Compute” button, I only see one token generated each time, and have to keep clicking to get the full expected continuation.How can I change the behavior of the Hosted Inference API available on the model card, so that it generates more than one token at a time, like using the pipeline in the Python interpreter does?
2023-11-29T16:53:44Z
Thanks for solution.