How many GB of RAM do I need to train DBRX?
I heard it required 320 GB of RAM. However, if I was to quantize (if that is possible) would I be able to reduce it below 100 GB?In addition how would I store this model so I can keep my progress? Would I need to download it somehow to my laptop?
2024-04-11T10:29:46Z
Methods and tools for efficient training on a single GPU (huggingface.co)Quantisation will help as well. However, Exhaust the optimisations in the link above. FP16, Gradient Accumulation, Gradient Checkpointing and PEFT are very good at cutting down on VRAM usage.
2024-04-11T14:03:08Z
The 4-bit version requires about 70GB of RAM.You can run it locally on a Macbook using MLX:mlx-community/dbrx-instruct-4bit · Hugging Face.Or you could run it with Transformers by providing aquantization_configto thefrom_pretrainedmethod. The model will soon be part of the Transformers library:Support DBRX Model · Issue #29911 · huggingface/transformers · GitHub