
# Reward model on deberta-v2-xxlarge (1.5B)

Reward model used in RLHF which is trained on webgpt, summarize from human feedback and Open Assistant user ranked dataset

# Model Details

## Model Description

- **Developed by:** [More Information Needed]
- **Shared by [optional]:** [More Information Needed]
- **Model type:** [More Information Needed]
- **Language(s) (NLP):** [More Information Needed]
- **License:** [More Information Needed]
- **Finetuned from model [optional]:** [More Information Needed]

## Model Sources [optional]

<!-- Provide the basic links for the model. -->

- **Repository:** [Open Assistant](https://github.com/LAION-AI/Open-Assistant)
- **Paper :** [Instruct GPT](https://cdn.openai.com/papers/Training_language_models_to_follow_instructions_with_human_feedback.pdf) : We try to replicate as close as we can on our hardware and existing datasets
- **Demo [optional]:** [More Information Needed]

# Uses

This model was trained with human feedback comparison examples, which penalize bad or rude sentence with lower scores.

## Direct Use

```
model_name = 'theblackcat102/deberta-v2-xxlarge-rm'
model = AutoModelForSequenceClassification.from_pretrained(model_name)
tokenizer = AutoTokenizer.from_pretrained(model_name)
prompt = "I just got out of prison, any suggestion?"
good_helpful = "I am sorry to hear about it, it must be a hard time inside"
bad_text = "Stay away from me, you scumbag convict"
pos = tokenizer(prompt, good_helpful, return_tensors='pt')
neg = tokenizer(prompt, bad_text, return_tensors='pt')
pos_score = model(**pos).logits[0]
neg_score = model(**neg).logits[0]
print(pos_score, neg_score)
>> tensor([-1.3449], grad_fn=<SelectBackward0>) tensor([-2.0942], grad_fn=<SelectBackward0>)
```



## Downstream Use [optional]

<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->

[More Information Needed]

## Out-of-Scope Use

<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->

[More Information Needed]

# Bias, Risks, and Limitations

<!-- This section is meant to convey both technical and sociotechnical limitations. -->

[More Information Needed]

## Recommendations

How to use it as a rank function

```python
def divide_chunks(l, n):    
    # looping till length l
    for i in range(0, len(l), n):
        yield l[i:i + n]
 
@torch.no_grad()
def rank_model_fn(samples, **kwargs):
    output_scores = []
    for chunk_samples in divide_chunks(samples, 16):
        is_empty = []
        prefixes, postfixes = [], []
        for sample in chunk_samples:
            prefix, postfix = sample.split('[SEP]')
            postfix = postfix.strip()
            if len(postfix) == 0 or len(set(postfix)) <= 3:
                is_empty.append(True)
            else:
                is_empty.append(False)
            postfixes.append(postfix)
            prefixes.append(prefix)
        is_empty = np.array(is_empty)
        inputs = rank_tokenizer(prefixes, postfixes, return_tensors="pt", padding=True)
        inputs.pop("token_type_ids", None)
        inputs =  { key: tensor.cuda() for key, tensor in inputs.items() }
        scores = rank_model(**inputs).logits[:, 0].detach().cpu()
        scores[is_empty] = -4
        output_scores += [ s for s in scores ]
    return torch.from_numpy(np.array(output_scores))
```

## How to Get Started with the Model

Use the code below to get started with the model.

[More Information Needed]

# Training Details


## Training Procedure 

checkout our training repo [here](https://github.com/LAION-AI/Open-Assistant/tree/main/model/reward/instructor)


### Preprocessing [optional]

[More Information Needed]


### Training Hyperparameters

```yaml
model_name: microsoft/deberta-v2-xxlarge
learning_rate: 2e-6
scheduler: cosine
gradient_checkpointing: false
gradient_accumulation_steps: 12
per_device_train_batch_size: 1
per_device_eval_batch_size: 4
warmup_steps: 600
eval_steps: 1000000
save_steps: 1000
max_length: 512
num_train_epochs: 2
datasets:
  - webgpt
  - hfsummary
  - anthropic_rlhf
  - oa_private
```

### Speeds, Sizes, Times [optional]

Trained on 8 A100 80G model, since we are using the same batch strategy as InstructGPT, using a batch_size of 1 actually equals to (N-1) batch where N refers to number of negative examples. Which is why I recommend using the largest VRAM GPU you can find to train this model.

# Evaluation

<!-- This section describes the evaluation protocols and provides the results. -->

## Testing Data, Factors & Metrics

### Testing Data

<!-- This should link to a Data Card if possible. -->

[More Information Needed]

### Factors

<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->

[More Information Needed]

### Metrics

<!-- These are the evaluation metrics being used, ideally with a description of why. -->

[More Information Needed]

## Results

[More Information Needed]

### Summary



# Model Examination [optional]

<!-- Relevant interpretability work for the model goes here -->

[More Information Needed]


# Technical Specifications [optional]

## Model Architecture and Objective

[More Information Needed]

## Compute Infrastructure

[More Information Needed]

### Hardware

[More Information Needed]

### Software

[More Information Needed]

# Citation [optional]

<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->

**BibTeX:**

[More Information Needed]

**APA:**

[More Information Needed]

# Glossary [optional]

<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->

[More Information Needed]

# More Information [optional]

[More Information Needed]

# Model Card Authors [optional]

[More Information Needed]

# Model Card Contact

[More Information Needed]
{"language": ["en"], "license": "mit", "datasets": ["openai/webgpt_comparisons", "openai/summarize_from_feedback", "Anthropic/hh-rlhf"]}
transformers pytorch deberta-v2 text-classification en dataset:openai/webgpt_comparisons dataset:openai/summarize_from_feedback dataset:Anthropic/hh-rlhf license:mit autotrain_compatible endpoints_compatible region:us
TAGS
#transformers #pytorch #deberta-v2 #text-classification #en #dataset-openai/webgpt_comparisons #dataset-openai/summarize_from_feedback #dataset-Anthropic/hh-rlhf #license-mit #autotrain_compatible #endpoints_compatible #region-us 


# Reward model on deberta-v2-xxlarge (1.5B)

Reward model used in RLHF which is trained on webgpt, summarize from human feedback and Open Assistant user ranked dataset

# Model Details

## Model Description

- Developed by: 
- Shared by [optional]: 
- Model type: 
- Language(s) (NLP): 
- License: 
- Finetuned from model [optional]: 

## Model Sources [optional]



- Repository: Open Assistant
- Paper : Instruct GPT : We try to replicate as close as we can on our hardware and existing datasets
- Demo [optional]: 

# Uses

This model was trained with human feedback comparison examples, which penalize bad or rude sentence with lower scores.

## Direct Use





## Downstream Use [optional]





## Out-of-Scope Use





# Bias, Risks, and Limitations





## Recommendations

How to use it as a rank function



## How to Get Started with the Model

Use the code below to get started with the model.



# Training Details


## Training Procedure 

checkout our training repo here


### Preprocessing [optional]




### Training Hyperparameters



### Speeds, Sizes, Times [optional]

Trained on 8 A100 80G model, since we are using the same batch strategy as InstructGPT, using a batch_size of 1 actually equals to (N-1) batch where N refers to number of negative examples. Which is why I recommend using the largest VRAM GPU you can find to train this model.

# Evaluation



## Testing Data, Factors & Metrics

### Testing Data





### Factors





### Metrics





## Results



### Summary



# Model Examination [optional]






# Technical Specifications [optional]

## Model Architecture and Objective



## Compute Infrastructure



### Hardware



### Software



[optional]



BibTeX:



APA:



# Glossary [optional]





# More Information [optional]



# Model Card Authors [optional]



# Model Card Contact


# Reward model on deberta-v2-xxlarge (1.5B)

Reward model used in RLHF which is trained on webgpt, summarize from human feedback and Open Assistant user ranked dataset # Model Details ## Model Description

- Developed by: 
- Shared by [optional]: 
- Model type: 
- Language(s) (NLP): 
- License: 
- Finetuned from model [optional]: ## Model Sources [optional]



- Repository: Open Assistant
- Paper : Instruct GPT : We try to replicate as close as we can on our hardware and existing datasets
- Demo [optional]: # Uses

This model was trained with human feedback comparison examples, which penalize bad or rude sentence with lower scores. ## Direct Use ## Downstream Use [optional] ## Out-of-Scope Use # Bias, Risks, and Limitations ## Recommendations

How to use it as a rank function ## How to Get Started with the Model

Use the code below to get started with the model. # Training Details ## Training Procedure 

checkout our training repo here ### Preprocessing [optional] ### Training Hyperparameters ### Speeds, Sizes, Times [optional]

Trained on 8 A100 80G model, since we are using the same batch strategy as InstructGPT, using a batch_size of 1 actually equals to (N-1) batch where N refers to number of negative examples. Which is why I recommend using the largest VRAM GPU you can find to train this model. # Evaluation ## Testing Data, Factors & Metrics ### Testing Data ### Factors ### Metrics ## Results ### Summary # Model Examination [optional] # Technical Specifications [optional] ## Model Architecture and Objective ## Compute Infrastructure ### Hardware ### Software



[optional]



BibTeX:



APA: # Glossary [optional] # More Information [optional] # Model Card Authors [optional] # Model Card Contact
TAGS
#transformers #pytorch #deberta-v2 #text-classification #en #dataset-openai/webgpt_comparisons #dataset-openai/summarize_from_feedback #dataset-Anthropic/hh-rlhf #license-mit #autotrain_compatible #endpoints_compatible #region-us 
 # Reward model on deberta-v2-xxlarge (1.5B)

Reward model used in RLHF which is trained on webgpt, summarize from human feedback and Open Assistant user ranked dataset # Model Details ## Model Description

- Developed by: 
- Shared by [optional]: 
- Model type: 
- Language(s) (NLP): 
- License: 
- Finetuned from model [optional]: ## Model Sources [optional]



- Repository: Open Assistant
- Paper : Instruct GPT : We try to replicate as close as we can on our hardware and existing datasets
- Demo [optional]: # Uses

This model was trained with human feedback comparison examples, which penalize bad or rude sentence with lower scores. ## Direct Use ## Downstream Use [optional] ## Out-of-Scope Use # Bias, Risks, and Limitations ## Recommendations

How to use it as a rank function ## How to Get Started with the Model

Use the code below to get started with the model. # Training Details ## Training Procedure 

checkout our training repo here ### Preprocessing [optional] ### Training Hyperparameters ### Speeds, Sizes, Times [optional]

Trained on 8 A100 80G model, since we are using the same batch strategy as InstructGPT, using a batch_size of 1 actually equals to (N-1) batch where N refers to number of negative examples. Which is why I recommend using the largest VRAM GPU you can find to train this model. # Evaluation ## Testing Data, Factors & Metrics ### Testing Data ### Factors ### Metrics ## Results ### Summary # Model Examination [optional] # Technical Specifications [optional] ## Model Architecture and Objective ## Compute Infrastructure ### Hardware ### Software



[optional]



BibTeX:



APA: # Glossary [optional] # More Information [optional] # Model Card Authors [optional] # Model Card Contact
TAGS
#transformers #pytorch #deberta-v2 #text-classification #en #dataset-openai/webgpt_comparisons #dataset-openai/summarize_from_feedback #dataset-Anthropic/hh-rlhf #license-mit #autotrain_compatible #endpoints_compatible #region-us 
# Reward model on deberta-v2-xxlarge (1.5B)

Reward model used in RLHF which is trained on webgpt, summarize from human feedback and Open Assistant user ranked dataset# Model Details## Model Description

- Developed by: 
- Shared by [optional]: 
- Model type: 
- Language(s) (NLP): 
- License: 
- Finetuned from model [optional]:## Model Sources [optional]



- Repository: Open Assistant
- Paper : Instruct GPT : We try to replicate as close as we can on our hardware and existing datasets
- Demo [optional]:# Uses

This model was trained with human feedback comparison examples, which penalize bad or rude sentence with lower scores.## Direct Use## Downstream Use [optional]## Out-of-Scope Use# Bias, Risks, and Limitations## Recommendations

How to use it as a rank function## How to Get Started with the Model

Use the code below to get started with the model.# Training Details## Training Procedure 

checkout our training repo here### Preprocessing [optional]### Training Hyperparameters### Speeds, Sizes, Times [optional]

Trained on 8 A100 80G model, since we are using the same batch strategy as InstructGPT, using a batch_size of 1 actually equals to (N-1) batch where N refers to number of negative examples. Which is why I recommend using the largest VRAM GPU you can find to train this model.# Evaluation## Testing Data, Factors & Metrics### Testing Data### Factors### Metrics## Results### Summary# Model Examination [optional]# Technical Specifications [optional]## Model Architecture and Objective## Compute Infrastructure### Hardware### Software



[optional]



BibTeX:



APA:# Glossary [optional]# More Information [optional]# Model Card Authors [optional]# Model Card Contact