Notebook_launcher set num_processes=2 but it say Launching training on one GPU. in Kaggle
I am trying to test this article code with A100 x 2 GPUs. Link -Launching Multi-Node Training from a Jupyter EnvironmentBut it always gets only one GPU in Kaggle Notebook. How to solve this issue?Print -Launching training on one GPU.but it has 2 GPU.+-----------------------------------------------------------------------------+
| NVIDIA-SMI 470.82.01    Driver Version: 470.82.01    CUDA Version: 11.4     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  Tesla T4            Off  | 00000000:00:04.0 Off |                    0 |
| N/A   37C    P8     9W /  70W |      0MiB / 15109MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   1  Tesla T4            Off  | 00000000:00:05.0 Off |                    0 |
| N/A   43C    P8     9W /  70W |      0MiB / 15109MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+I think this point of code run -accelerate/launchers.py at v0.15.0 · huggingface/accelerate · GitHub
2022-12-08T13:07:57Z
Indramal:But it always gets only one GPU in Kaggle Notebook. How to solve this issue?What’s your version of Accelerate? Only the latest version (0.15.0) will launch in Kaggle successfully. What you pointed out there was the Google Colab check statement
2022-12-08T15:29:21Z
thank you very much. Kaggle has already installed that library and my installation does not work. Now I update it to 0.15.0 and it is working now.
2022-12-08T15:50:22Z
Results:Colabepoch 0: 87.67
epoch 1: 89.31
epoch 2: 93.93
epoch 3: 96.97
epoch 4: 97.55
Total execution time = 354.609 secKaggleepoch 0: 91.68
epoch 1: 89.31
epoch 2: 94.97
epoch 3: 97.46
epoch 4: 97.51
Total execution time = 427.316 sec1 GPU Colab is faster than 2 GPU.
2022-12-08T16:18:11Z
Indramal:1 GPU Colab is faster than 2 GPU.You should make sure you’re actually setting up your benchmarks right by reading our docs on it, as it’s very easy to just think that running the same script does the same thing (spoiler, it does not!)Comparing performance between different device setups
2022-12-09T03:02:50Z
I am sorry to tell you that, it is not getting fast execution on 2 GPUs. I don’t know why.This is what I use as a sample code -Launching Multi-GPU Training from a Jupyter EnvironmentThese are what I changed according toComparing performance between different device setupsSetting Seedset_seed(42)- Both are the same value in 1 GPU and 2 GPU.Batch SizesIn Colab - 128def get_dataloaders(batch_size: int = 128):
-----def training_loop(mixed_precision="fp16", seed: int = 42, batch_size: int = 128):
-----args = ("fp16", 42, 128)
notebook_launcher(training_loop, args, num_processes=1)In Kaggle - 64def get_dataloaders(batch_size: int = 64):
--def training_loop(mixed_precision="fp16", seed: int = 42, batch_size: int = 64):
---args = ("fp16", 42, 64)
notebook_launcher(training_loop, args, num_processes=2)Learning RatesBoth use same code# Intantiate the optimizer
learning_rate = 3e-2 / 25
learning_rate *= accelerator.num_processes
optimizer = torch.optim.Adam(params=model.parameters(), lr=learning_rate)Results:In Kaggle (2 GPU):Launching training on 2 GPUs.epoch 0: 89.86epoch 1: 87.50epoch 2: 94.30epoch 3: 96.78epoch 4: 97.61Total execution time = 460.321 secIn Colab (1 GPU):Launching training on one GPU.epoch 0: 87.96epoch 1: 87.36epoch 2: 94.15epoch 3: 97.16epoch 4: 97.55Total execution time = 341.572 secThis is the code I use to calculate the time:start_time = time.time()
args = ("fp16", 42, xx)
notebook_launcher(training_loop, args, num_processes=xx)
end_time = time.time()
print("Total execution time = {:.3f} sec".format(end_time - start_time))xx - it changes depending on the system.
2022-12-10T03:02:02Z
@sgugger@patrickvonplatenany suggestions?