
# dolly-v2-12b: sharded **8bit** checkpoint 


<a href="https://colab.research.google.com/gist/pszemraj/1bc9cea67e6c8dc450b868e0cfc18163/dolly-v2-12b-8bit-inference.ipynb">
  <img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab"/>
</a>

This is a sharded checkpoint (with ~4GB shards) of the `databricks/dolly-v2-12b` model **in `8bit` precision** using `bitsandbytes`.

Refer to the [original model](https://huggingface.co/databricks/dolly-v2-12b) for all details w.r.t. to the model. For more info on loading 8bit models, refer to the [example repo](https://huggingface.co/ybelkada/bloom-1b7-8bit) and/or the `4.28.0` [release info](https://github.com/huggingface/transformers/releases/tag/v4.28.0).

- total model size is only ~12.5 GB!
- this enables low-RAM loading, i.e. Colab :)
- **update**: generation speed can be greatly improved by setting `use_cache=True` and generating via contrastive search. [example notenook here](https://colab.research.google.com/gist/pszemraj/12c832952c88d77f6924c0718a2d257d/dolly-v2-12b-8bit-use_cache-bettertransformer.ipynb)
## Basic Usage


install/upgrade `transformers`, `accelerate`, and `bitsandbytes`. For this to work **you must have** `transformers>=4.28.0` and `bitsandbytes>0.37.2`.

```bash
pip install -U -q transformers bitsandbytes accelerate
```

Load the model. As it is serialized in 8bit you don't need to do anything special:

```python
from transformers import AutoTokenizer, AutoModelForCausalLM

model_name = "ethzanalytics/dolly-v2-12b-sharded-8bit"
tokenizer = AutoTokenizer.from_pretrained(model_name)

model = AutoModelForCausalLM.from_pretrained(model_name)
```
{"language": ["en"], "license": "mit", "tags": ["dolly", "dolly-v2", "instruct", "sharded", "8bit", "quantized"], "datasets": ["databricks/databricks-dolly-15k"], "pipeline_tag": "text-generation", "inference": false}
transformers pytorch gpt_neox text-generation dolly dolly-v2 instruct sharded 8bit quantized en dataset:databricks/databricks-dolly-15k license:mit autotrain_compatible text-generation-inference 8-bit region:us
TAGS
#transformers #pytorch #gpt_neox #text-generation #dolly #dolly-v2 #instruct #sharded #8bit #quantized #en #dataset-databricks/databricks-dolly-15k #license-mit #autotrain_compatible #text-generation-inference #8-bit #region-us 


# dolly-v2-12b: sharded 8bit checkpoint 


<a href="URL
  <img src="URL alt="Open In Colab"/>
</a>

This is a sharded checkpoint (with ~4GB shards) of the 'databricks/dolly-v2-12b' model in '8bit' precision using 'bitsandbytes'.

Refer to the original model for all details w.r.t. to the model. For more info on loading 8bit models, refer to the example repo and/or the '4.28.0' release info.

- total model size is only ~12.5 GB!
- this enables low-RAM loading, i.e. Colab :)
- update: generation speed can be greatly improved by setting 'use_cache=True' and generating via contrastive search. example notenook here
## Basic Usage


install/upgrade 'transformers', 'accelerate', and 'bitsandbytes'. For this to work you must have 'transformers>=4.28.0' and 'bitsandbytes>0.37.2'.



Load the model. As it is serialized in 8bit you don't need to do anything special:


# dolly-v2-12b: sharded 8bit checkpoint 


<a href="URL
  <img src="URL alt="Open In Colab"/>
</a>

This is a sharded checkpoint (with ~4GB shards) of the 'databricks/dolly-v2-12b' model in '8bit' precision using 'bitsandbytes'.

Refer to the original model for all details w.r.t. to the model. For more info on loading 8bit models, refer to the example repo and/or the '4.28.0' release info.

- total model size is only ~12.5 GB!
- this enables low-RAM loading, i.e. Colab :)
- update: generation speed can be greatly improved by setting 'use_cache=True' and generating via contrastive search. example notenook here ## Basic Usage


install/upgrade 'transformers', 'accelerate', and 'bitsandbytes'. For this to work you must have 'transformers>=4.28.0' and 'bitsandbytes>0.37.2'.



Load the model. As it is serialized in 8bit you don't need to do anything special:
TAGS
#transformers #pytorch #gpt_neox #text-generation #dolly #dolly-v2 #instruct #sharded #8bit #quantized #en #dataset-databricks/databricks-dolly-15k #license-mit #autotrain_compatible #text-generation-inference #8-bit #region-us 
 # dolly-v2-12b: sharded 8bit checkpoint 


<a href="URL
  <img src="URL alt="Open In Colab"/>
</a>

This is a sharded checkpoint (with ~4GB shards) of the 'databricks/dolly-v2-12b' model in '8bit' precision using 'bitsandbytes'.

Refer to the original model for all details w.r.t. to the model. For more info on loading 8bit models, refer to the example repo and/or the '4.28.0' release info.

- total model size is only ~12.5 GB!
- this enables low-RAM loading, i.e. Colab :)
- update: generation speed can be greatly improved by setting 'use_cache=True' and generating via contrastive search. example notenook here ## Basic Usage


install/upgrade 'transformers', 'accelerate', and 'bitsandbytes'. For this to work you must have 'transformers>=4.28.0' and 'bitsandbytes>0.37.2'.



Load the model. As it is serialized in 8bit you don't need to do anything special:
TAGS
#transformers #pytorch #gpt_neox #text-generation #dolly #dolly-v2 #instruct #sharded #8bit #quantized #en #dataset-databricks/databricks-dolly-15k #license-mit #autotrain_compatible #text-generation-inference #8-bit #region-us 
# dolly-v2-12b: sharded 8bit checkpoint 


<a href="URL
  <img src="URL alt="Open In Colab"/>
</a>

This is a sharded checkpoint (with ~4GB shards) of the 'databricks/dolly-v2-12b' model in '8bit' precision using 'bitsandbytes'.

Refer to the original model for all details w.r.t. to the model. For more info on loading 8bit models, refer to the example repo and/or the '4.28.0' release info.

- total model size is only ~12.5 GB!
- this enables low-RAM loading, i.e. Colab :)
- update: generation speed can be greatly improved by setting 'use_cache=True' and generating via contrastive search. example notenook here## Basic Usage


install/upgrade 'transformers', 'accelerate', and 'bitsandbytes'. For this to work you must have 'transformers>=4.28.0' and 'bitsandbytes>0.37.2'.



Load the model. As it is serialized in 8bit you don't need to do anything special: