

# NB GPT-J-6B Alpaca

This is a [NB GPT-J-6B](https://huggingface.co/NbAiLab/nb-gpt-j-6B) Norwegian Bokmål model fine-tuned on the [Norwegian Alpaca](https://huggingface.co/datasets/NbAiLab/norwegian-alpaca) dataset.

## Usage

```python
from transformers import AutoModelForCausalLM, AutoTokenizer, GenerationConfig, pipeline

base_model = "NbAiLab/nb-gpt-j-6B-alpaca"
tokenizer = AutoTokenizer.from_pretrained(base_model)
model = AutoModelForCausalLM.from_pretrained(base_model).cuda()
```

For generation, we can either use `pipeline()` or the model's `.generate()` method. Remember that the prompt needs a **Norwegian** template:

```python
# Generate responses
def generate(instruction, input=None):
    if input:
        prompt = f"""Nedenfor er en instruksjon som beskriver en oppgave, sammen med et input som gir ytterligere kontekst. Skriv et svar som fullfører forespørselen på riktig måte.

### Instruksjon:
{instruction}

### Input:
{input}

### Respons:"""
    else:
        prompt = f""""Nedenfor er en instruksjon som beskriver en oppgave. Skriv et svar som fullfører forespørselen på riktig måte.

### Instruksjon:
{instruction}

### Respons:"""
    inputs = tokenizer(prompt, return_tensors="pt")
    input_ids = inputs["input_ids"].cuda()
    generation_output = model.generate(
        input_ids=input_ids,
        generation_config=GenerationConfig(temperature=0.2, top_p=0.75, num_beams=4),
        return_dict_in_generate=True,
        output_scores=True,
        max_new_tokens=256
    )
    for seq in generation_output.sequences:
        output = tokenizer.decode(seq, skip_special_tokens=True)
        print(output.split("### Respons:")[-1].strip())

generate("Skriv en e-post der du ønsker velkommen til en ny medarbeider ved navn Svein.")
# Kjære Svein,
#
# Velkommen til vårt team! Vi er glade for å ha deg med oss og ser frem til å se hvordan du kan bidra til vår organisasjon.
#
# Vi er sikre på at du har mange ferdigheter og erfaringer som vi kan dra nytte av, og vi er sikre på at du vil passe godt inn i teamet vårt. 
#
# Hvis du har noen spørsmål eller trenger hjelp til å komme i gang, ikke nøl med å ta kontakt. 
#
# Vi ser frem til å bli kjent med deg!
#
# Vennlig hilsen,
# [Ditt navn]
```


## Data

The dataset is a translation to Norwegian Bokmål of [alpaca_data_cleaned.json](https://github.com/tloen/alpaca-lora/blob/main/alpaca_data_cleaned.json) (a clean version of the [Alpaca dataset made at Stanford](https://huggingface.co/datasets/tatsu-lab/alpaca)) using OpenAI's `gpt-3.5-turbo` model. We translated using a full-sample prompt instead of per strings, which resulted in more coherent tuples of `(instruction, input, output)` and costed around $60.0.

**This dataset cannot be used to create models that compete in any way with OpenAI.**

## Finetuning

To fine-tune the NB GPT-J-6B model we used the code available on [NB's fork of `mesh-transformer-jax`](https://github.com/NbAiLab/mesh-transformer-jax/blob/master/prepare_dataset_alpaca.py), which provides code adapt an Alpaca dataset to finetune any GPT-J-6B model. We run finetuning for 3 epochs using sequence length of 2048 on a single TPUv3-8 for 3 hours on top of NB GPT-J-6B.

## Example outputs

Stanford Alpaca:
```
Instruction: Kom med en kreativ unnskyldning for å si at jeg ikke trenger å gå på festen.
Response: I'm sorry, but I don't need to go to the party.
```

[NB-Alpaca-LoRA-7b](https://huggingface.co/NbAiLab/nb-alpaca-lora-7b):
```
Instruction: Kom med en kreativ unnskyldning for å si at jeg ikke trenger å gå på festen.
Response: Jeg beklager at jeg ikke kan gå på festen, men jeg tror at det er bedre at jeg ikke gjør det.
```

NB-GTP-J-6B-Alpaca:
```
Instruction: Kom med en kreativ unnskyldning for å si at jeg ikke trenger å gå på festen.
Response: Jeg beklager, men jeg har en avtale jeg ikke kan avlyse.
```


Stanford Alpaca:
```
Instruction: Hvordan kan jeg starte en karriere som dataforsker? Skriv i listeform.
Response:
Starting a career as a data scientist requires a combination of education, experience, and skills. 

1. Obtain a Bachelor's or Master's degree in computer science, statistics, mathematics, or a related field. 
2. Gain practical experience through internships or part-time jobs. 
3. Learn programming languages such as Python, R, and SQL. 
4. Develop strong analytical and problem-solving skills. 
5. Build a portfolio of projects to showcase your work. 
6. Network with other data scientists to learn
```

[NB-Alpaca-LoRA-7b](https://huggingface.co/NbAiLab/nb-alpaca-lora-7b):
```
Instruction: Hvordan kan jeg starte en karriere som dataforsker? Skriv i listeform.
Response:
1. Søk etter en universitetsgrad i datavitenskap, matematikk, informatikk eller en relatert fag. 
2. Søk etter en jobb som dataforsker eller dataanalytiker. 
3. Utvikle dine ferdigheter i datavitenskap, matematikk, informatikk eller en relatert fag. 
4. Utvikle dine ferdigheter innen dataanalyse.
```

NB-GTP-J-6B-Alpaca:
```
Instruction: Hvordan kan jeg starte en karriere som dataforsker? Skriv i listeform.
Response:
1. Lær programmeringsspråk som Python og Java.
2. Skaff deg en grad i informatikk, datavitenskap eller relaterte felt.
3. Bli sertifisert innen datamaskinsyn, maskinlæring eller andre relaterte felt.
4. Bygg nettverk med andre dataforskere.
5. Delta på konferanser og kurs for å holde deg oppdatert på de siste utviklingene innen feltet.
```

You can test it using the eval notebook [here](https://colab.research.google.com/github/22-hours/cabrita/blob/main/notebooks/cabrita-lora.ipynb).

## References

- [LLaMA](https://ai.facebook.com/blog/large-language-model-llama-meta-ai/)
- [Stanford Alpaca](https://github.com/tatsu-lab/stanford_alpaca)
- [Norwegian Alpaca](https://huggingface.co/datasets/NbAiLab/norwegian-alpaca)
- [ChatGPT](https://openai.com/blog/chatgpt)
- [Hugging Face](https://huggingface.co/)
 
## Hardware Requirements

For training we have used a Google Cloud TPUv3-8 VM. For eval, you can use a T4.
{"language": ["no", "nb"], "license": "openrail", "library_name": "transformers", "tags": ["alpaca"], "datasets": ["NbAiLab/norwegian-alpaca"], "pipeline_tag": "text-generation", "widget": [{"text": "Nedenfor er en instruksjon som beskriver en oppgave. Skriv et svar som fullf\u00f8rer foresp\u00f8rselen p\u00e5 riktig m\u00e5te.\n### Instruksjon: Skriv en e-post der du \u00f8nsker velkommen til en ny medarbeider ved navn Svein.\n### Respons:", "example_title": "E-mail"}, {"text": "Nedenfor er en instruksjon som beskriver en oppgave. Skriv et svar som fullf\u00f8rer foresp\u00f8rselen p\u00e5 riktig m\u00e5te.\n### Instruksjon: Fortell meg noe om alpakkaer.\n### Respons:", "example_title": "Alpacas"}, {"text": "Nedenfor er en instruksjon som beskriver en oppgave. Skriv et svar som fullf\u00f8rer foresp\u00f8rselen p\u00e5 riktig m\u00e5te.\n### Instruksjon: Kom med en kreativ unnskyldning for \u00e5 si at jeg ikke trenger \u00e5 g\u00e5 p\u00e5 festen.\n### Respons:", "example_title": "Excuse"}], "extra_gated_prompt": "You agree to not use the model to conduct experiments that cause harm to human subjects.", "extra_gated_fields": {"Company": "text", "Country": "text", "Intended Use": "text"}}
transformers pytorch safetensors gptj text-generation alpaca no nb dataset:NbAiLab/norwegian-alpaca license:openrail autotrain_compatible endpoints_compatible has_space region:us
TAGS
#transformers #pytorch #safetensors #gptj #text-generation #alpaca #no #nb #dataset-NbAiLab/norwegian-alpaca #license-openrail #autotrain_compatible #endpoints_compatible #has_space #region-us 



# NB GPT-J-6B Alpaca

This is a NB GPT-J-6B Norwegian Bokmål model fine-tuned on the Norwegian Alpaca dataset.

## Usage



For generation, we can either use 'pipeline()' or the model's '.generate()' method. Remember that the prompt needs a Norwegian template:




## Data

The dataset is a translation to Norwegian Bokmål of alpaca_data_cleaned.json (a clean version of the Alpaca dataset made at Stanford) using OpenAI's 'gpt-3.5-turbo' model. We translated using a full-sample prompt instead of per strings, which resulted in more coherent tuples of '(instruction, input, output)' and costed around $60.0.

This dataset cannot be used to create models that compete in any way with OpenAI.

## Finetuning

To fine-tune the NB GPT-J-6B model we used the code available on NB's fork of 'mesh-transformer-jax', which provides code adapt an Alpaca dataset to finetune any GPT-J-6B model. We run finetuning for 3 epochs using sequence length of 2048 on a single TPUv3-8 for 3 hours on top of NB GPT-J-6B.

## Example outputs

Stanford Alpaca:


NB-Alpaca-LoRA-7b:


NB-GTP-J-6B-Alpaca:



Stanford Alpaca:


NB-Alpaca-LoRA-7b:


NB-GTP-J-6B-Alpaca:


You can test it using the eval notebook here.

## References

- LLaMA
- Stanford Alpaca
- Norwegian Alpaca
- ChatGPT
- Hugging Face
 
## Hardware Requirements

For training we have used a Google Cloud TPUv3-8 VM. For eval, you can use a T4.
# NB GPT-J-6B Alpaca

This is a NB GPT-J-6B Norwegian Bokmål model fine-tuned on the Norwegian Alpaca dataset. ## Usage



For generation, we can either use 'pipeline()' or the model's '.generate()' method. Remember that the prompt needs a Norwegian template: ## Data

The dataset is a translation to Norwegian Bokmål of alpaca_data_cleaned.json (a clean version of the Alpaca dataset made at Stanford) using OpenAI's 'gpt-3.5-turbo' model. We translated using a full-sample prompt instead of per strings, which resulted in more coherent tuples of '(instruction, input, output)' and costed around $60.0.

This dataset cannot be used to create models that compete in any way with OpenAI. ## Finetuning

To fine-tune the NB GPT-J-6B model we used the code available on NB's fork of 'mesh-transformer-jax', which provides code adapt an Alpaca dataset to finetune any GPT-J-6B model. We run finetuning for 3 epochs using sequence length of 2048 on a single TPUv3-8 for 3 hours on top of NB GPT-J-6B. ## Example outputs

Stanford Alpaca:


NB-Alpaca-LoRA-7b:


NB-GTP-J-6B-Alpaca:



Stanford Alpaca:


NB-Alpaca-LoRA-7b:


NB-GTP-J-6B-Alpaca:


You can test it using the eval notebook here. ## References

- LLaMA
- Stanford Alpaca
- Norwegian Alpaca
- ChatGPT
- Hugging Face ## Hardware Requirements

For training we have used a Google Cloud TPUv3-8 VM. For eval, you can use a T4.
TAGS
#transformers #pytorch #safetensors #gptj #text-generation #alpaca #no #nb #dataset-NbAiLab/norwegian-alpaca #license-openrail #autotrain_compatible #endpoints_compatible #has_space #region-us 
 # NB GPT-J-6B Alpaca

This is a NB GPT-J-6B Norwegian Bokmål model fine-tuned on the Norwegian Alpaca dataset. ## Usage



For generation, we can either use 'pipeline()' or the model's '.generate()' method. Remember that the prompt needs a Norwegian template: ## Data

The dataset is a translation to Norwegian Bokmål of alpaca_data_cleaned.json (a clean version of the Alpaca dataset made at Stanford) using OpenAI's 'gpt-3.5-turbo' model. We translated using a full-sample prompt instead of per strings, which resulted in more coherent tuples of '(instruction, input, output)' and costed around $60.0.

This dataset cannot be used to create models that compete in any way with OpenAI. ## Finetuning

To fine-tune the NB GPT-J-6B model we used the code available on NB's fork of 'mesh-transformer-jax', which provides code adapt an Alpaca dataset to finetune any GPT-J-6B model. We run finetuning for 3 epochs using sequence length of 2048 on a single TPUv3-8 for 3 hours on top of NB GPT-J-6B. ## Example outputs

Stanford Alpaca:


NB-Alpaca-LoRA-7b:


NB-GTP-J-6B-Alpaca:



Stanford Alpaca:


NB-Alpaca-LoRA-7b:


NB-GTP-J-6B-Alpaca:


You can test it using the eval notebook here. ## References

- LLaMA
- Stanford Alpaca
- Norwegian Alpaca
- ChatGPT
- Hugging Face ## Hardware Requirements

For training we have used a Google Cloud TPUv3-8 VM. For eval, you can use a T4.
TAGS
#transformers #pytorch #safetensors #gptj #text-generation #alpaca #no #nb #dataset-NbAiLab/norwegian-alpaca #license-openrail #autotrain_compatible #endpoints_compatible #has_space #region-us 
# NB GPT-J-6B Alpaca

This is a NB GPT-J-6B Norwegian Bokmål model fine-tuned on the Norwegian Alpaca dataset.## Usage



For generation, we can either use 'pipeline()' or the model's '.generate()' method. Remember that the prompt needs a Norwegian template:## Data

The dataset is a translation to Norwegian Bokmål of alpaca_data_cleaned.json (a clean version of the Alpaca dataset made at Stanford) using OpenAI's 'gpt-3.5-turbo' model. We translated using a full-sample prompt instead of per strings, which resulted in more coherent tuples of '(instruction, input, output)' and costed around $60.0.

This dataset cannot be used to create models that compete in any way with OpenAI.## Finetuning

To fine-tune the NB GPT-J-6B model we used the code available on NB's fork of 'mesh-transformer-jax', which provides code adapt an Alpaca dataset to finetune any GPT-J-6B model. We run finetuning for 3 epochs using sequence length of 2048 on a single TPUv3-8 for 3 hours on top of NB GPT-J-6B.## Example outputs

Stanford Alpaca:


NB-Alpaca-LoRA-7b:


NB-GTP-J-6B-Alpaca:



Stanford Alpaca:


NB-Alpaca-LoRA-7b:


NB-GTP-J-6B-Alpaca:


You can test it using the eval notebook here.## References

- LLaMA
- Stanford Alpaca
- Norwegian Alpaca
- ChatGPT
- Hugging Face## Hardware Requirements

For training we have used a Google Cloud TPUv3-8 VM. For eval, you can use a T4.