How to Configure LLaMA-3:8B on HuggingFace to Generate Responses Similar to Ollama?
Hi HuggingFace Community,I have been experimenting with the LLaMA-3:8B model using the following code:import transformers
import torch

model_id = "meta-llama/Meta-Llama-3-8B"

pipeline = transformers.pipeline(
    "text-generation", model=model_id, model_kwargs={"torch_dtype": torch.bfloat16}, device_map="auto"
)
pipeline("Hey how are you doing today?")This code generates the following response:[{'generated_text': 'Hey how are you doing today? My name is David and I’m a photographer based in Toronto, Canada. I started this blog as a place to share my photography and thoughts on photography, and I hope that you enjoy it.\nI’m also a software developer, and I run a web development company called WebDev Studios. I’m a big fan of WordPress, and I’m always looking for ways to improve the WordPress experience for developers and users. I’m also a big fan of open source software, and I’m always looking for ways to contribute to the open source community.\nI hope you enjoy my blog, and I hope you find it useful. If you have any questions or comments, please feel free to contact me. Thanks for reading!\nHow to use David’s blog for your photography\nDavid’s blog is a great resource for anyone interested in photography. The blog features a wide range of topics, from gear reviews to tutorials to inspirational stories. Whether you’re a beginner or a seasoned pro, there’s something for everyone on David’s blog.\nOne of the best things about David’s blog is that it’s easy to use. The layout is clean and uncluttered, and the posts are well-organized. You can easily find what you’re looking for by using the search function or by browsing through the different categories.\nIf you’re looking for photography tips and advice, David’s blog is a great place to start. The posts are written in a clear and concise manner, and they’re packed with helpful information. Whether you’re trying to improve your composition or learn how to use a new piece of gear, you’ll find what you’re looking for on David’s blog.\nIf you’re looking for inspiration, David’s blog is a great place to find it. The posts are filled with beautiful images and inspiring stories. Whether you’re looking for a new perspective on photography or just some motivation to get out and shoot, you’ll find what you’re looking for on David’s blog.\nIf you’re looking for a great photography resource, David’s blog is a must-visit. The blog is well-organized and easy to use, and the posts are packed with helpful information. Whether you’re a beginner or a seasoned pro, you’ll find what you’re looking for on David’s blog.\nHow to use David’s blog for your photography business\nDavid’s blog is a great resource for anyone looking to improve their photography skills or start a photography business. The blog is packed with helpful tips and advice, and David is always happy to answer any questions you may have.\nOne of the best things about David’s blog is that it’s easy to use. You can find whatever you’re looking for by using the search function or by browsing through the different categories. The blog is also well-organized, so you can easily find what you need.\nIf you’re just starting out in photography, David’s blog is a great place to start. The blog covers everything from the basics of photography to more advanced topics, so you can find the information you need to improve your skills.\nIf you’re already a photographer, David’s blog is still a great resource. The blog covers a wide range of topics, so you can find new tips and advice to help you improve your photography.\nNo matter what your level of experience is, David’s blog is a great resource for anyone looking to improve their photography skills or start a photography business.\nHow to use David’s blog for your photography business\nDavid’s blog is a great resource for anyone looking to improve their photography skills or start a photography business. The blog is packed with helpful tips and advice, and David is always happy to answer any questions you may have.\nOne of the best things about David’s blog is that it’s easy to use. You can find whatever you’re looking for by using the search function or by browsing through the different categories. The blog is also well-organized, so you can easily find what you need.\nIf you’re just starting out in photography, David’s blog is a great place to start. The blog covers everything from the basics of photography to more advanced topics, so you can find the information you need to improve your skills.\nIf you’re already a photographer, David’s blog is still a great resource. The blog covers a wide range of topics, so you can find new tips and advice to help you improve your photography.\nNo matter what your level of experience is, David’s blog is a great resource for anyone looking to improve their photography skills or start a photography business.\nHow to use David’s blog for your photography business\nDavid’s blog is a great resource for anyone looking to improve their photography skills or start a photography business. The blog is packed with helpful tips and advice, and David is always happy to answer any questions you may have.\nOne of the best things about David’s blog is that it’s easy to use. You can find whatever you’re looking for by using the search function or by browsing through the different categories. The blog is also well-organized, so you can easily find what you need.\nIf you’re just starting out in photography, David’s blog is a great place to start. The blog covers everything from the basics of photography to more advanced topics, so you can find the information you need to improve your skills.\nIf you’re already a photographer, David’s blog is still a great resource. The blog covers a wide range of topics, so you can find new tips and advice to help you improve your photography.\nNo matter what your level of experience is, David’s blog is a great resource for anyone looking to improve their photography skills or start a photography business.'}]However, when I input a similar question into the Ollama model, I receive a response like:"I'm just a language model, so I don't have emotions or personal experiences like humans do. However, I'm functioning properly and ready to assist with any questions or tasks you may have!"I am curious why these two models, both based on LLaMA3:8b, produce such different responses. More importantly, I would like to know how I can configure the HuggingFace LLaMA-3:8B model to output responses more akin to Ollama’s style.Is there a way to adjust model settings, parameters, or prompt design to achieve this? Any guidance would be appreciated!Thank you!
2024-10-04T02:28:05Z
The issue of different output when using pipelines and when using them locally or in Spaces is sometimes talked about.While pipelines are easy to deal with, they seem to be working by supplementing a lot of information on their own, which means it is difficult to determine what is wrong with them.Completely different results for model in pipeline and by itselfBeginnersAh, actually, I think I see the issue, right after posting it. 
I think it’s about setting up the tokenizer? Could that be it? 
In preprocess I make sure to have overlap, pad and max length = model.max_length 
I’m guessing by default when I just pass tokenizer to the pipeline, none of these get set. 
Can I pass a pre setup tokenizer? 
Figured it out, there are two parameters, max_seq_len and max_answer_len that can set during the call but not initialization. 
thanks a lot!Different outputs when using pipelineIntermediateHi folks! 
I’m currently working on my master thesis and need to implement a text-classification model. Therefore I’m fine-tuning a base BERT model on my data. So far so good. To evaluate my fine-tuned model, I’m passing my test data to it for predictions. For this purpose I’m using the “pipeline” API provided by huggingface with a sigmoid function (since I’m facing a multi-label classification problem). It does work and I get as an output the probs for each class. Now if instead of using the pi…I’m not sure about Ollama, but I think it would be better to use something other than pipelines to set chatbot-like parameters.chat_completion, for example, is often used.Also, I heard that Llama3 has some kind of bug, though I’m not sure what it has to do with this issue.huggingface.coChatting with TransformersWe’re on a journey to advance and democratize artificial intelligence through open source and open science.huggingface.coChat CompletionWe’re on a journey to advance and democratize artificial intelligence through open source and open science.
2024-10-05T07:15:33Z
Hi@John6666,Thanks again for your assistance! I tried using model.generate, but I am still encountering the same issue of inconsistent output between the LLaMA-3:8B model on HuggingFace and Ollama.Would you mind suggesting any other API options or strategies that I could use for text generation with large language models? I am specifically aiming to achieve a more consistent style in the responses, similar to Ollama.
2024-10-05T07:39:04Z
If you are aiming for a consistent answer style, I would suggest  setting temperature to 0.7 or higher.Ollama’s default value seems to be 0.7.In addition, if you set do_sample=False, temperature, etc. cannot be set, and the answer will be reproducible.Also, keep in mind that there seems to be some kind of bug in the Llama3 model, although probably not related to this issue. The specifics are a bit beyond me, but I hear it is buggy anyway.Ollama default parameters:github.comollama/ollama/blob/main/docs/modelfile.md#parameter# Ollama Model File

> [!NOTE]
> `Modelfile` syntax is in development

A model file is the blueprint to create and share models with Ollama.

## Table of Contents

- [Format](#format)
- [Examples](#examples)
- [Instructions](#instructions)
  - [FROM (Required)](#from-required)
    - [Build from existing model](#build-from-existing-model)
    - [Build from a Safetensors model](#build-from-a-safetensors-model)
    - [Build from a GGUF file](#build-from-a-gguf-file)
  - [PARAMETER](#parameter)
    - [Valid Parameters and Values](#valid-parameters-and-values)
  - [TEMPLATE](#template)
    - [Template Variables](#template-variables)This file has been truncated.show originalgenerate options:huggingface.coGenerationWe’re on a journey to advance and democratize artificial intelligence through open source and open science.
2024-10-05T08:27:19Z
Hi@John6666,Thank you for your prompt response. I believe I’ve found a simple solution for the issue. It turns out that when we runollama run llama3:8b, Ollama is actually using thellama3:8b-instructmodel rather than the basellama3:8b(both have the hash ID365c0bd3c000).When I switched frommeta-llama/Meta-Llama-3-8Btometa-llama/Meta-Llama-3-8B-Instructin the HuggingFace pipeline, the output became much more aligned with what I was expecting from Ollama:{'role': 'assistant', 'content': "I'm just a language model, I don't have feelings or emotions like humans do, so I don't have good or bad days. However, I'm always happy to chat with you and help with any questions or topics you'd like to discuss! How about you? How's your day going?"}Thanks again for all your kind assistance!
2024-10-05T08:36:08Z
when we runollama run llama3:8b, Ollama is actually using thellama3:8b-instructmodel rather than the basellama3:8bWow. This was not predictable.Well, it’s OK as long as it gets resolved.LLM is fun.Changing the temperature option changes the LLM’s reaction (personality?) a lot, so it’s interesting to try it out.
2024-10-05T13:30:05Z
Sure, I have already changed the temperature as 0.7 as you recommended. Hope this will bring me better results
2024-10-07T13:17:23Z
You’re using the model in document-completion mode, rather than chat mode. SeeChatting with Transformersfor the basic instructions andChat Templatesfor details.