How to perform training on CPU +GPU offloading?
I wanted to fine-tunebloom-7b1using qlora with 1x3090(24 GB)  and Nvidia Titan(12 GB) .Is there way to offload weights to CPU ?Thepeft githubhas shown offloading results . How to perform oflloading in qlora?
2023-12-19T19:09:03Z
mess around with the device_map functionmodel = AutoModelForCausalLM.from_pretrained(base_model,quantization_config=quant_config,#load_in_8bit=True,device_map={“”: Accelerator().local_process_index},)check out thishuggingface.coQuantizationWe’re on a journey to advance and democratize artificial intelligence through open source and open science.notice how they offload layers to diffrent gpus/cpus (in there case 0 is gpu_0 and 1 will be your other gpu)