Stanford Alpaca on 2x3090
So I am attempting to do the Stanford Alpaca training that they describe here:GitHubGitHub - tatsu-lab/stanford_alpaca: Code and documentation to train...Code and documentation to train Stanford's Alpaca models, and generate the data. - GitHub - tatsu-lab/stanford_alpaca: Code and documentation to train Stanford's Alpaca models, and generate...I have a workstation with 512GB Ram and 2 x 3090s with 24GB VRAM.  I have reached the point where I am trying to train it but I keep getting out of memory error.  I know they used 4x A100 with 80GB VRAM.  I tried changing the number of GPUs to 2 and reducing the batch sizes (see below for my torchrun command with args.  I have also tried using bitsandbytes to reduce down to 8bits but I am having problems getting it to run.github.comGitHub - TimDettmers/bitsandbytes at ec5fbf4cc44324829307138a4c17fd88dddd9803ec5fbf4cc44324829307138a4c17fd88dddd98038-bit CUDA functions for PyTorch. Contribute to TimDettmers/bitsandbytes development by creating an account on GitHub.Any suggestions?torchrun --nproc_per_node=2 --master_port=13833 train.py–model_name_or_path decapoda-research/llama-7b-hf–data_path ./alpaca_data.json–bf16 True–output_dir /home/dsa/stanford_alpaca–num_train_epochs 3–per_device_train_batch_size 1–per_device_eval_batch_size 1–gradient_accumulation_steps 32–evaluation_strategy “no”–save_strategy “steps”–save_steps 2000–save_total_limit 1–learning_rate 2e-5–weight_decay 0.–warmup_ratio 0.03–lr_scheduler_type “cosine”–logging_steps 1–fsdp “full_shard auto_wrap”–fsdp_transformer_layer_cls_to_wrap ‘LLaMADecoderLayer’–tf32 True