transformers pytorch llama text-generation finance en license:llama2 autotrain_compatible endpoints_compatible text-generation-inference region:us
---
license: llama2
language:
- en
tags:
- finance
---
# InvestLM
This is the repo for a new financial domain large language model, InvestLM, tuned on LLaMA-65B[1], using a carefully curated instruction dataset related to financial investment. We provide guidance on how to use InvestLM for inference.

Github Link: [InvestLM](https://github.com/AbaciNLP/InvestLM)

# About AWQ
AWQ [2] is an efficient, accurate, and blazing-fast low-bit weight quantization method, currently supporting 4-bit quantization. Compared to GPTQ, it offers faster Transformers-based inference.

It is supported by:
1. AutoAWQ: AutoAWQ is an easy-to-use package for AWQ 4-bit quantized models. [AutoAWQ Github link](https://github.com/casper-hansen/AutoAWQ)
```
pip install autoawq
```
2. vLLM: vLLM is a Python library that contains pre-compiled C++ and CUDA (11.8) binaries. We can use it to run offline inference or serve as the endpoint. It offers advanced continuous batching and a much higher (~10x) throughput. But it's more complicated. [vllm Doc](https://vllm.readthedocs.io/en/latest/getting_started/installation.html)

```
# (Optional) Create a new conda environment.
conda create -n myenv python=3.8 -y
conda activate myenv

# Install vLLM.
pip install vllm
```
3. Additional options: these are some Python libraries integrated with vLLM.
   * Fastchat: FastChat is an open platform for training, serving, and evaluating large language model based chatbots. We can use vLLM as an optimized worker implementation in FastChat. [Fastchat Github](https://github.com/lm-sys/FastChat)
   * aphrodite-engine: Aphrodite is the official backend engine for PygmalionAI. It is designed to serve as the inference endpoint for the PygmalionAI website. [Aphrodite-engine Github](https://github.com/PygmalionAI/aphrodite-engine)

# Inference
Please use the following command to log in hugging face first. 
```
huggingface-cli login

```
## Prompt template

```
PROMPT_DICT = {
    "prompt_input": (
        "Below is an instruction that describes a task, paired with further context. "
        "Write a response that appropriately completes the request.\n\n"
        "Instruction:\n{instruction}\n\n Input:\n{input}\n\n Response:"
    ),
    "prompt_no_input": (
        "Below is an instruction that describes a task. "
        "Write a response that appropriately completes the request.\n\n"
        "Instruction:\n{instruction}\n\nResponse:"
    ),
}

```

## How to use this AWQ model from Python code
```
# Please run the following command in CLI.
# pip install autoawq transformers


from transformers import AutoTokenizer
from awq import AutoAWQForCausalLM

# Inference Template 
PROMPT_DICT = {
    "prompt_input": (
        "Below is an instruction that describes a task, paired with further context. "
        "Write a response that appropriately completes the request.\n\n"
        "Instruction:\n{instruction}\n\n Input:\n{input}\n\n Response:"
    ),
    "prompt_no_input": (
        "Below is an instruction that describes a task. "
        "Write a response that appropriately completes the request.\n\n"
        "Instruction:\n{instruction}\n\nResponse:"
    ),
}

    
def generate_prompt(instruction, input=None):
    if input:
        return PROMPT_DICT["prompt_input"].format(instruction=instruction,input=input)
    else:
        return PROMPT_DICT["prompt_no_input"].format(instruction=instruction)


tokenizer =  AutoTokenizer.from_pretrained('yixuantt/InvestLM-awq', use_fast=False)
tokenizer.pad_token = tokenizer.unk_token
model = AutoAWQForCausalLM.from_quantized('yixuantt/InvestLM-awq', fuse_layers=False)

print("\n\n*** Generate:")

tokens = tokenizer(
    generate_prompt(instruction="Tell me about finance."),
    return_tensors='pt'
).input_ids.cuda()

# Generate output
generation_output = model.generate(
    tokens,
    do_sample=True,
    temperature=0.1,
    top_p=0.75,
    top_k=40,
    repetition_penalty = 1.1,
    max_new_tokens=512
)

print("Output: ", tokenizer.decode(generation_output[0]))

```
## Serving this model from vLLM

* Please ensure you are using vLLM version 0.2 or later. 
* When using vLLM as a server, pass the --quantization awq parameter.
* At the time of writing, vLLM AWQ does not support loading models in bfloat16, so to ensure compatibility with all models, also pass --dtype float16.
For example:
```
python3 python -m vllm.entrypoints.api_server --model 'yixuantt/InvestLM-awq' --quantization awq --dtype float16

```
When using vLLM from Python code, again pass the quantization=awq and dtype=float16 parameters.

```
from vllm import LLM, SamplingParams
# Inference Template 
PROMPT_DICT = {
    "prompt_input": (
        "Below is an instruction that describes a task, paired with further context. "
        "Write a response that appropriately completes the request.\n\n"
        "Instruction:\n{instruction}\n\n Input:\n{input}\n\n Response:"
    ),
    "prompt_no_input": (
        "Below is an instruction that describes a task. "
        "Write a response that appropriately completes the request.\n\n"
        "Instruction:\n{instruction}\n\nResponse:"
    ),
}

    
def generate_prompt(instruction, input=None):
    if input:
        return PROMPT_DICT["prompt_input"].format(instruction=instruction,input=input)
    else:
        return PROMPT_DICT["prompt_no_input"].format(instruction=instruction)

question = generate_prompt(instruction="Tell me about finance.")
prompts = [
   question,
]
sampling_params = SamplingParams(temperature=0.1, top_p=0.75)

llm = LLM(model="yixuantt/InvestLM-awq", quantization="awq", dtype="float16")

outputs = llm.generate(prompts, sampling_params)

# Print the outputs.
for output in outputs:
    prompt = output.prompt
    generated_text = output.outputs[0].text
    print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}")


```


# References
[1] Touvron, Hugo, et al. "Llama: Open and efficient foundation language models." arXiv preprint arXiv:2302.13971 (2023).

[2] Lin, J., Tang, J., et al. "AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration" arXiv preprint arXiv:2306.00978 (2023).

---
license: llama2
---