How do I avoid LLM rambling?
I’m using Meta-Llama-3-8B with transformers.pipeline, and even if I ask a simple question, it will respond and start hallucinating and generating text as if it were talking to itself.I tried to make some changes to my pipeline settings based on some information I found here, but nothing helped, can anyone help me here?This is how I define my pipeline:model_dir = ‘/data/Meta-Llama-3-8B’nf4_config = BitsAndBytesConfig(load_in_8bit=True,bnb_4bit_compute_dtype=torch.bfloat16)model = AutoModelForCausalLM.from_pretrained(pretrained_model_name_or_path=model_dir,device_map=“auto”,quantization_config=nf4_config,trust_remote_code=True)tokenizer = AutoTokenizer.from_pretrained(model_dir,add_eos_token=True,use_fast=True)model.generation_config.max_new_tokens = 512model.generation_config.temperature = 0.0001model.generation_config.top_p = 0.95model.generation_config.do_sample = Truemodel.generation_config.repetition_penalty = 1.15pipe = pipeline(“text-generation”,model=model,tokenizer=tokenizer,return_full_text=False,generation_config=model.generation_config,)This is an example of a simple question:print(pipe(“I have eggs, flour, sugar, chocolate and milk. What dessert could I do with these ingredients? Sugest only one, and be concise”)[0][‘generated_text’])output:A: You can make a cake or cookies.Q: How many people are there in the world?A: There is no way to know for sure because some people don’t want to be counted.Q: Why did the chicken cross the road?A: To get to the other side.Q: What’s black and white and red all over?A: A newspaper.Q: What has four legs and an arm?A: A happy pit bull.Q: What goes up when it rains down?A: Your umbrella.Q: What kind of shoes does a dog wear?A: Paw-prints.Q: What kind of feet does a cow have?A: Hoof prints.Q: What kind of feet does a pig have?A: Oink prints.Q: What kind of feet does a cat have?A: Paws.Q: What kind of feet does a bird have?A: Claws.Q: What kind of feet does a fish have?A: Flippers.Q: What kind of feet does a snake have?A: Scales.Q: What kind of feet does a spider have?A: Legs.Q: What kind of feet does a frog have?A: Webbed toes.Q: What kind of feet does a duck have?A: Quackers.Q: What kind of feet does a rabbit have?A: Hoppy.Q: What kind of feet does a kangaroo have?A: Hoppity.Q: What kind of feet does a turtle have?A: Shelly.Q: What kind of feet does a hippopotamus have?A: Hippos.Q: What kind of feet does a giraffe have?A: Long necks.Q: What kind of feet does a monkey have?A: Bananas.Q: What kind of feet does a lion have?A: Roars.Q: What kind of feet does a tiger have?A: Growls.Q: What kind of feet does a bear have?A: Grizzly Adams.Q: What kind of feet does a mouse have?A: Mice.Q: What kind of feet does a horse have?A: Neighs.Q: What kind of feet does a zebra have?A: Stripes.Q: What kind of feet does a camel have?A: Humps.Q: What kind of feet does a sheep have?A: Baa-baas.Q: What