

# dolly-v2-7b: **8-bit** sharded checkpoint 


<a href="https://colab.research.google.com/gist/pszemraj/8100e98caab538be32832d1208e93f65/dolly-v2-7b-8bit-inference.ipynb">
  <img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab"/>
</a>

This is a sharded checkpoint (with ~2GB shards) of the `databricks/dolly-v2-7b` model **in 8-bit precision using `bitsandbytes`**. 

Refer to the [original model](https://huggingface.co/databricks/dolly-v2-7b) for all details. For more info on loading 8bit models, refer to the [example repo](https://huggingface.co/ybelkada/bloom-1b7-8bit) and/or the `4.28.0` [release info](https://github.com/huggingface/transformers/releases/tag/v4.28.0).

- total model size is only ~7.5 GB!
- this enables low-RAM loading, i.e. Colab :)

## Basic Usage


install/upgrade `transformers`, `accelerate`, and `bitsandbytes`. For this to work **you must have** `transformers>=4.28.0` and `bitsandbytes>0.37.2`.

```bash
pip install -U -q transformers bitsandbytes accelerate
```

Load the model. As it is serialized in 8bit you don't need to do anything special:

```python
from transformers import AutoTokenizer, AutoModelForCausalLM

model_name = "ethzanalytics/dolly-v2-7b-sharded-8bit"
tokenizer = AutoTokenizer.from_pretrained(model_name)

model = AutoModelForCausalLM.from_pretrained(model_name)
```
{"language": ["en"], "license": "mit", "tags": ["dolly", "dolly-v2", "instruct", "sharded", "quantized"], "datasets": ["databricks/databricks-dolly-15k"], "pipeline_tag": "text-generation", "inference": false}
transformers pytorch gpt_neox text-generation dolly dolly-v2 instruct sharded quantized en dataset:databricks/databricks-dolly-15k license:mit autotrain_compatible text-generation-inference 8-bit region:us
TAGS
#transformers #pytorch #gpt_neox #text-generation #dolly #dolly-v2 #instruct #sharded #quantized #en #dataset-databricks/databricks-dolly-15k #license-mit #autotrain_compatible #text-generation-inference #8-bit #region-us 



# dolly-v2-7b: 8-bit sharded checkpoint 


<a href="URL
  <img src="URL alt="Open In Colab"/>
</a>

This is a sharded checkpoint (with ~2GB shards) of the 'databricks/dolly-v2-7b' model in 8-bit precision using 'bitsandbytes'. 

Refer to the original model for all details. For more info on loading 8bit models, refer to the example repo and/or the '4.28.0' release info.

- total model size is only ~7.5 GB!
- this enables low-RAM loading, i.e. Colab :)

## Basic Usage


install/upgrade 'transformers', 'accelerate', and 'bitsandbytes'. For this to work you must have 'transformers>=4.28.0' and 'bitsandbytes>0.37.2'.



Load the model. As it is serialized in 8bit you don't need to do anything special:


# dolly-v2-7b: 8-bit sharded checkpoint 


<a href="URL
  <img src="URL alt="Open In Colab"/>
</a>

This is a sharded checkpoint (with ~2GB shards) of the 'databricks/dolly-v2-7b' model in 8-bit precision using 'bitsandbytes'. 

Refer to the original model for all details. For more info on loading 8bit models, refer to the example repo and/or the '4.28.0' release info.

- total model size is only ~7.5 GB!
- this enables low-RAM loading, i.e. Colab :) ## Basic Usage


install/upgrade 'transformers', 'accelerate', and 'bitsandbytes'. For this to work you must have 'transformers>=4.28.0' and 'bitsandbytes>0.37.2'.



Load the model. As it is serialized in 8bit you don't need to do anything special:
TAGS
#transformers #pytorch #gpt_neox #text-generation #dolly #dolly-v2 #instruct #sharded #quantized #en #dataset-databricks/databricks-dolly-15k #license-mit #autotrain_compatible #text-generation-inference #8-bit #region-us 
 # dolly-v2-7b: 8-bit sharded checkpoint 


<a href="URL
  <img src="URL alt="Open In Colab"/>
</a>

This is a sharded checkpoint (with ~2GB shards) of the 'databricks/dolly-v2-7b' model in 8-bit precision using 'bitsandbytes'. 

Refer to the original model for all details. For more info on loading 8bit models, refer to the example repo and/or the '4.28.0' release info.

- total model size is only ~7.5 GB!
- this enables low-RAM loading, i.e. Colab :) ## Basic Usage


install/upgrade 'transformers', 'accelerate', and 'bitsandbytes'. For this to work you must have 'transformers>=4.28.0' and 'bitsandbytes>0.37.2'.



Load the model. As it is serialized in 8bit you don't need to do anything special:
TAGS
#transformers #pytorch #gpt_neox #text-generation #dolly #dolly-v2 #instruct #sharded #quantized #en #dataset-databricks/databricks-dolly-15k #license-mit #autotrain_compatible #text-generation-inference #8-bit #region-us 
# dolly-v2-7b: 8-bit sharded checkpoint 


<a href="URL
  <img src="URL alt="Open In Colab"/>
</a>

This is a sharded checkpoint (with ~2GB shards) of the 'databricks/dolly-v2-7b' model in 8-bit precision using 'bitsandbytes'. 

Refer to the original model for all details. For more info on loading 8bit models, refer to the example repo and/or the '4.28.0' release info.

- total model size is only ~7.5 GB!
- this enables low-RAM loading, i.e. Colab :)## Basic Usage


install/upgrade 'transformers', 'accelerate', and 'bitsandbytes'. For this to work you must have 'transformers>=4.28.0' and 'bitsandbytes>0.37.2'.



Load the model. As it is serialized in 8bit you don't need to do anything special: