Can bloom-7b1 be fine tuned using gaudi 1?
In this github repo examplegithub.comhuggingface/optimum-habana/blob/main/examples/text-generation/run_generation.py#!/usr/bin/env python
# coding=utf-8
# Copyright 2018 Google AI, Google Brain and Carnegie Mellon University Authors and the HuggingFace Inc. team.
# Copyright (c) 2018, NVIDIA CORPORATION.  All rights reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""
Conditional text generation on Habana Gaudi/Gaudi2.
"""This file has been truncated.show originalwe can use bloom-7b1 but I don’t see an option to train (fine tune) the existing model.Can it be done?
2023-07-11T10:36:41Z
python run_generation.py \
--model_name_or_path gpt2 \
--use_hpu_graphs \
--use_kv_cache \
--max_new_tokens 100 \
--do_sample \
--prompt "Here is my prompt"this is the standard script. Can we add do_train here?
2023-07-11T10:55:09Z
If this script won’t do it, can I get any script that can?
2023-07-11T11:04:01Z
Hi@gildesh! The script you linked to enables to perform generation with a model but it does not support training.For fine-tuning BLOOM 7B, you would need to use the language modeling example:https://github.com/huggingface/optimum-habana/tree/main/examples/language-modeling#gpt-2gpt-and-causal-language-modelingHowever, not sure if Gaudi1 has enough memory to train this model. You could try using DeepSpeed ZeRO-3, it may work:Install DeepSpeed withpip install git+https://github.com/HabanaAI/DeepSpeed.git@1.10.0Then run:python ../gaudi_spawn.py \
--world_size 8 --use_deepspeed run_clm.py \
--model_name_or_path bigscience/bloom-7b1 \
--dataset_name wikitext \
--dataset_config_name wikitext-2-raw-v1 \
--per_device_train_batch_size 1 \
--per_device_eval_batch_size 1 \
--do_train \
--do_eval \
--output_dir /tmp/test-clm \
--gaudi_config_name Habana/gpt2 \
--use_habana \
--use_lazy_mode \
--use_hpu_graphs_for_inference \
--gradient_checkpointing \
--use_cache False \
--throughput_warmup_steps 3 \
--deepspeed path_to_my_deepspeed_configwith for example this DeepSpeed config:https://github.com/huggingface/optimum-habana/blob/main/examples/summarization/ds_flan_t5_z3_config_bf16.jsonIf this works, you could increase the batch size to see if you can fit bigger batches. Let me know if you manage to launch a training or if you need any help
2023-07-12T03:42:01Z
regisss:--dataset_config_name wikitext-2-raw-v1 \
--per_device_train_batch_size 1 \
--per_device_eval_batch_size 1 \
--do_train \
--do_eval \
--output_dir /tmp/test-clm \
--gaudi_config_name Habana/gpt2 \
--use_habana \
--use_lazy_mode \
--use_hpu_graphs_for_inference \
--gradient_checkpointing \
--use_cache False \
--throughput_warmup_steps 3 \
--deepspeed path_to_my_deepspeed_configThanks a lot regiss!I did run into this errorimage1838×961 237 KBUsed this deep speed config{“steps_per_print”: 64,“train_batch_size”: “auto”,“train_micro_batch_size_per_gpu”: “auto”,“gradient_accumulation_steps”: “auto”,“bf16”: {“enabled”: true},“gradient_clipping”: 1.0,“zero_optimization”: {“stage”: 2,“overlap_comm”: false,“reduce_scatter”: false,“contiguous_gradients”: false}}and this scriptpython …/gaudi_spawn.py–world_size 8 --use_deepspeed run_clm.py–model_name_or_path bigscience/bloom-7b1–dataset_name wikitext–dataset_config_name wikitext-2-raw-v1–per_device_train_batch_size 1–per_device_eval_batch_size 1–do_train–do_eval–output_dir /tmp/test-clm–gaudi_config_name Habana/gpt2–use_habana–use_lazy_mode–use_hpu_graphs_for_inference–gradient_checkpointing–use_cache False–throughput_warmup_steps 3–deepspeed deep_cnvrg.jsonIs it a simple memory issue or something else?
2023-07-12T04:01:33Z
when i ran it with bloom-560m, (below is the script for reference)python …/gaudi_spawn.py–world_size 8 --use_deepspeed run_clm.py–model_name_or_path bigscience/bloom-560m–dataset_name wikitext–dataset_config_name wikitext-2-raw-v1–per_device_train_batch_size 1–per_device_eval_batch_size 1–do_train–do_eval–output_dir /tmp/test-clm–gaudi_config_name Habana/gpt2–use_habana–use_lazy_mode–use_hpu_graphs_for_inference–gradient_checkpointing–use_cache False–throughput_warmup_steps 3–deepspeed deep_cnvrg.json–overwrite_output_dirI got this errorimage1850×952 233 KBmaybe because the gaudi_config_name is habana/gpt-2 but the model is bloom-560?
2023-07-12T06:50:33Z
And how do we know what is the amount of CPU/Memory it is using? the dl1 has 8 HPUs, 768 GB but in the script we can only know about the world_size which is the number of HPUs
2023-07-12T08:07:52Z
Weird that it fails with BLOOM-560m too. I’ll look into it in the next few days and will let you know what I find.gaudi_config_nameis mainly used to specify the operators to use in bf16 precision, but DeepSpeed manages that itself so it shouldn’t be the issue here.768 GB is the memory of the ohst (CPU) that you can monitor withtoporhtop. If you want to monitor the memory of the 8 Gaudi devices, you can runhl-smi.
2023-07-12T10:27:11Z
Thanks regiss!
2023-07-14T07:27:24Z
@gildeshThere was indeed a bug in the custom modeling of BLOOM. The fix has just been merged into the main branch of Optimum Habana, you can install the repo withpip install git+https://github.com/huggingface/optimum-habana.gitto have it.Note that BLOOM 7B is too big to fit on Gaudi1 devices even using DeepSpeed ZeRO-3 so it will fail with a memory allocation error. It works well on Gaudi2 on the other hand. For BLOOM 560m, not sure you need DeepSpeed at all since it should fit on Gaudi1 devices (unless you would like to save memory to fit bigger batches).
2023-07-24T08:06:06Z
Hello regissI tried againbut this time got another errorimage1846×957 227 KB
2024-01-09T14:43:43Z
@gildeshWe added an example showing how to fine-tune BLOOM-7b1 on Gaudi1 with DeepSpeed ZeRO-3 here:optimum-habana/examples/language-modeling at main · huggingface/optimum-habana · GitHub
2024-01-10T02:44:21Z
This topic was automatically closed 12 hours after the last reply. New replies are no longer allowed.