transformers endpoints_compatible region:us
# Neuronx model for [meta-llama/Llama-2-7b-chat-hf](https://huggingface.co/meta-llama/Llama-2-7b-chat-hf)

This repository contains are  [**AWS Inferentia2**](https://aws.amazon.com/ec2/instance-types/inf2/) and [`neuronx`](https://awsdocs-neuron.readthedocs-hosted.com/en/latest/) compatible checkpoint for [meta-llama/Llama-2-7b-chat-hf](https://huggingface.co/meta-llama/Llama-2-7b-chat-hf). You can find detailed information about the base model on its [Model Card](https://huggingface.co/meta-llama/Llama-2-7b-chat-hf).


## Usage with optimum-neuron

```python

from optimum.neuron import pipeline

# Load pipeline from Hugging Face repository
pipe = pipeline("text-generation", "WaelDataReply/Llama-2-7b-chat-seqlen-4096-bs-2")

# We use the tokenizer's chat template to format each message - see https://huggingface.co/docs/transformers/main/en/chat_templating
messages = [
    {"role": "user", "content": "What is the difference between AWS Step Functions and Apache Airflow?"},
]
prompt = pipe.tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
# Run generation
outputs = pipe(prompt, max_new_tokens=256, do_sample=True, temperature=0.7, top_k=50, top_p=0.95)
print(outputs[0]["generated_text"])

```

## Compilation Arguments

**compilation arguments**

```json
{
  "num_cores": 2,
  "auto_cast_type": "fp16"
}
```

**input_shapes**

```json
{
  "sequence_length": 4096,
  "batch_size": 2
}
```
