Limiting print and log statements
Accelerate: 0.9.0Python: 3.9.7I have written a class that handles printing and logging when going between cpu and gpu training configurations. Here is the class:import logging
from accelerate import Accelerator
import wandb
from typing import (List, Tuple, Any, Union)
import torch
logging.basicConfig(level=logging.DEBUG, format='%(levelname)s: %(message)s')


class TrainRun:
    def __init__(
        self,
        tracker_project_name,
        tracker_run_name, 
        is_accelerate:bool=False,
        tracker_name:str=None) -> None:
        
        self.tracker_project_name = tracker_project_name
        self.tracker_run_name = tracker_run_name
        self.is_accelerate = is_accelerate
        self.tracker_name = tracker_name
        
        if self.is_accelerate:
            self.accelerator = Accelerator(log_with=[self.tracker_name])
            self.accelerator.init_trackers(self.tracker_project_name)
            self.accelerator.trackers[0].run.name = self.tracker_run_name
            self.device=self.accelerator.device
            
        else:
            wandb.init(project=self.tracker_project_name)
            wandb.run.name = self.tracker_run_name
            self.device = torch.device("cpu")


    def log_it(self, str_to_log: Union[str, List[str]]):
        if type(str_to_log) is str:
            if self.is_accelerate and self.accelerator.is_local_main_process:
                logging.info(str_to_log)
            else:
                logging.info(str_to_log)
        else:
            if self.is_accelerate and self.accelerator.is_local_main_process:
                for log_str in str_to_log:
                    logging.info(log_str)
            else:
                for log_str in str_to_log:
                    logging.info(log_str)


    def print_it(self, str_to_print: Union[str, List[str]]):
        if type(str_to_print) is str:
            if self.is_accelerate and self.accelerator.is_local_main_process: 
                print(str_to_print)
            else:
                print(str_to_print)
        else:
            if self.is_accelerate and self.accelerator.is_local_main_process: 
                for print_str in str_to_print:
                    print(print_str)
            else: 
                for print_str in str_to_print:
                    print(print_str)I have a machine that has 4 GPUs on it. If I’ve done things correctly, I’d expect a single print out or log statement in the terminal. Here is some sample code to try:tracker_project_name = 'my_project'
tracker_name = 'wandb'
tracker_run_name = 'full_dataset'
train_run = TrainRun(tracker_project_name, tracker_run_name, is_accelerate=True, tracker_name=tracker_name)

train_run.log_it('CREATED TrainRun OBJECT')
train_run.log_it('is_accelerate = {}'.format(train_run.is_accelerate))
train_run.log_it('device = {}'.format(train_run.device))
train_run.print_it('-'*100)However, when I runaccelerate launch tmp.pyfrom the command line, I get several repeated log and print out statements:INFO: Added key: store_based_barrier_key:1 to store for rank: 3
INFO: Added key: store_based_barrier_key:1 to store for rank: 2
INFO: Added key: store_based_barrier_key:1 to store for rank: 0
INFO: Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
INFO: Added key: store_based_barrier_key:1 to store for rank: 1
INFO: Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
INFO: Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
INFO: Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
DEBUG: git repository is invalid
DEBUG: git repository is invalid
DEBUG: git repository is invalid
DEBUG: git repository is invalid
wandb: Tracking run with wandb version 0.12.17
wandb: W&B syncing is set to `offline` in this directory.  
wandb: Run `wandb online` or set WANDB_MODE=online to enable cloud syncing.
INFO: CREATED TrainRun OBJECT
INFO: is_accelerate = True
INFO: device = cuda:1
----------------------------------------------------------------------------------------------------
wandb: Tracking run with wandb version 0.12.17
wandb: W&B syncing is set to `offline` in this directory.  
wandb: Run `wandb online` or set WANDB_MODE=online to enable cloud syncing.
wandb: Tracking run with wandb version 0.12.17
wandb: W&B syncing is set to `offline` in this directory.  
wandb: Run `wandb online` or set WANDB_MODE=online to enable cloud syncing.
INFO: Initialized WandB project my_project
INFO: Make sure to log any initial configurations with `self.store_init_configuration` before training!
INFO: CREATED TrainRun OBJECT
INFO: is_accelerate = True
INFO: device = cuda:0
----------------------------------------------------------------------------------------------------
INFO: CREATED TrainRun OBJECT
----------------------------------------------------------------------------------------------------INFO: is_accelerate = True
INFO: device = cuda:3

wandb: Tracking run with wandb version 0.12.17
wandb: W&B syncing is set to `offline` in this directory.  
wandb: Run `wandb online` or set WANDB_MODE=online to enable cloud syncing.
INFO: CREATED TrainRun OBJECT
----------------------------------------------------------------------------------------------------INFO: is_accelerate = True
INFO: device = cuda:2

wandb: Waiting for W&B process to finish... (success).
wandb: Waiting for W&B process to finish... (success).ed)
wandb:                                                                                
wandb:                                                                                
wandb: Waiting for W&B process to finish... (success).
wandb: Waiting for W&B process to finish... (success).ed)
wandb:                                                                                
wandb:                                                                                
wandb: You can sync this run to the cloud by running:
wandb: wandb sync /home/aclifton/rf_fp/wandb/offline-run-20220705_125143-3cosslzv
wandb: Find logs at: ./wandb/offline-run-20220705_125143-3cosslzv/logs
wandb: You can sync this run to the cloud by running:
wandb: wandb sync /home/aclifton/rf_fp/wandb/offline-run-20220705_125143-3x7h5hfi
wandb: Find logs at: ./wandb/offline-run-20220705_125143-3x7h5hfi/logs
wandb: You can sync this run to the cloud by running:
wandb: wandb sync /home/aclifton/rf_fp/wandb/offline-run-20220705_125143-2wkg3sg7
wandb: Find logs at: ./wandb/offline-run-20220705_125143-2wkg3sg7/logs
wandb: You can sync this run to the cloud by running:
wandb: wandb sync /home/aclifton/rf_fp/wandb/offline-run-20220705_125143-1x53nj8b
wandb: Find logs at: ./wandb/offline-run-20220705_125143-1x53nj8b/logsIs there a way to limit the number of print and log statements to 1 when usingaccelerate? For isntance, I’d like to have the following printed out once:INFO: CREATED TrainRun OBJECT
INFO: is_accelerate = True
INFO: device = cuda:1
----------------------------------------------------------------------------------------------------Thanks in advance for your help! Please let me know if there is any other information I can provide.
2022-07-06T19:20:09Z
Thanks for all these questions! It’s made us realize that there should be some better guides out there + some key discussion posts here on the forum in regards to features in AccelerateWhat you want right now is the Logging capability inside Accelerate:from accelerate.logging import get_logger

logger = get_logger(__name__)This provides the same functionality aslogging.Loggerbut will also make special care for if you want something posted on only the main process or across all processes.For example, if it should be done only on the main process you can pass that in:# Will be logged n times (where n = number of processes)
logger.info("My log", main_process_only=False)
# Will be logged once on the main process
logger.debug("My log", main_process_only=True)Can you try using that and see if you still have issues with what you’re trying to accomplish? We can work towards that further and see if our own logging API needs improvement!
2022-07-06T20:22:04Z
@muellerzrI’m immensely grateful for theAcceleratelibrary. It has made my large model and dataset experimentation much more streamlined and faster. Thank you for all your efforts!Incorporating your suggestions into mylog_itfunction works great! Here is what I have now:def log_it(self, str_to_log: Union[str, List[str]]):
        if type(str_to_log) is str:
            if self.is_accelerate:
                self.accelerate_logger.info(str_to_log, main_process_only=True)
            else:
                logging.info(str_to_log)
        else:
            if self.is_accelerate:
                for log_str in str_to_log:
                    self.accelerate_logger.info(str_to_log, main_process_only=True)
            else:
                for log_str in str_to_log:
                    logging.info(log_str)Do you have any suggestions for theprint_it()function?
2022-07-06T21:19:04Z
aclifton314:Here is what I have now:I’d actually simplify this further potentially by declaring aLogging.loggerand store that in your class. E.g.:# Somewhere in init or before your init and pass it to the class constructor
self.logger = logging.getLogger(__name__)And then when you declare youris_accelerate:if is_accelerate:
    self.logger = get_logger(__name__)(Since one isget_loggerand the other isgetLoggeryou can just importfrom accelerate import get_loggerat the top I think)This reduces yourlog_itto be:def log_it(self, str_to_log: Union[str, List[str]]):
    if isinstance(str_to_log, str):
        # Note that the accelerate logger has main_process_only=True by default
        self.logger.info(str_to_log)
    else:
        for log_str in str_to_log:
            self.logger.info(str_to_log)Forprint_itjust useaccelerator.print():huggingface.coAcceleratorWe’re on a journey to advance and democratize artificial intelligence through open source and open science.You could do something similar to what I did early by setting something like aprinterattribute in your class, e.g.:# somewhere in init
self.printer = print# when is_accelerate gets made
if is_accelerate:
    self.printer = self.accelerator.printand now ourprint_itfunction:def print_it(self, str_to_print: Union[str, List[str]]):
    if isinstance(str_to_print, str):
        self.printer(str_to_print)
    else:
        for print_str in str_to_print:
            self.printer(print_str )This will only print once per server for you
2022-07-06T21:46:01Z
@muellerzrthank you for the response and suggestion. One question. In the following:def log_it(self, str_to_log: Union[str, List[str]]):
    if isinstance(str_to_log, str):
        # Note that the accelerate logger has main_process_only=True by default
        self.logger.info(str_to_log)
    else:
        for log_str in str_to_log:
            self.logger.info(str_to_log)ifis_accelerate = Truethenself.logger.info(str_to_log)will printstr_to_logfor each process, which I can fix with the keyword argumentmain_process_only=True. But ifis_accelerate = Falsethen won’tmain_process_only=Truecause an error sinceself.logger = logging.getLogger(__name__). I don’t know for sure, but I assume the logger from the python moduleloggingdoesn’t have that keyword argument.
2022-07-06T21:47:35Z
aclifton314:rintstr_to_logfor each process, which I can fix with the keyword argumentmain_process_only=TrueIt won’t, becausemain_process_only=Trueis the default.
2022-07-06T21:57:05Z
muellerzr:hat the accelerate logger has main_process_only=True by default
        sAh yes, I see. I’ll give these two a shot and see how they work.
2022-07-06T23:37:10Z
Works great. Thank you so much!
2022-08-01T16:47:53Z
The above solutions worked when I implemented my ownprint_itandlog_itmethods. However, it seems that there are some default logging statements within the HF framework that still get printed several times. Here are some examples:INFO: Added key: store_based_barrier_key:1 to store for rank: 3
INFO: Added key: store_based_barrier_key:1 to store for rank: 2
INFO: Added key: store_based_barrier_key:1 to store for rank: 1
INFO: Added key: store_based_barrier_key:1 to store for rank: 0
INFO: Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
INFO: Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
INFO: Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
INFO: Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
DEBUG: git repository is invalid
DEBUG: git repository is invalid
DEBUG: git repository is invalid
DEBUG: git repository is invalidandDEBUG: https://s3.amazonaws.com:443 "HEAD /datasets.huggingface.co/datasets/metrics/accuracy/accuracy.py HTTP/1.1" 200 0
DEBUG: Starting new HTTPS connection (1): raw.githubusercontent.com:443
DEBUG: https://s3.amazonaws.com:443 "HEAD /datasets.huggingface.co/datasets/metrics/accuracy/accuracy.py HTTP/1.1" 200 0
DEBUG: Starting new HTTPS connection (1): raw.githubusercontent.com:443
DEBUG: https://s3.amazonaws.com:443 "HEAD /datasets.huggingface.co/datasets/metrics/accuracy/accuracy.py HTTP/1.1" 200 0
DEBUG: Starting new HTTPS connection (1): raw.githubusercontent.com:443
Flattening the indices:  20%|██████████████████████████▊                                                                                                           | 1/5 [00:00<00:01,  3.55ba/s]DEBUG: https://raw.githubusercontent.com:443 "HEAD /huggingface/datasets/2.1.0/metrics/accuracy/accuracy.py HTTP/1.1" 200 0
DEBUG: Starting new HTTPS connection (1): s3.amazonaws.com:443
DEBUG: https://raw.githubusercontent.com:443 "HEAD /huggingface/datasets/2.1.0/metrics/accuracy/accuracy.py HTTP/1.1" 200 0
DEBUG: Starting new HTTPS connection (1): s3.amazonaws.com:443
DEBUG: https://raw.githubusercontent.com:443 "HEAD /huggingface/datasets/2.1.0/metrics/accuracy/accuracy.py HTTP/1.1" 200 0
DEBUG: Starting new HTTPS connection (1): s3.amazonaws.com:443
Flattening the indices:  40%|█████████████████████████████████████████████████████▌                                                                                | 2/5 [00:00<00:00,  3.48ba/s]DEBUG: https://s3.amazonaws.com:443 "HEAD /datasets.huggingface.co/datasets/metrics/f1/f1.py HTTP/1.1" 200 0
DEBUG: Starting new HTTPS connection (1): raw.githubusercontent.com:443
DEBUG: https://s3.amazonaws.com:443 "HEAD /datasets.huggingface.co/datasets/metrics/f1/f1.py HTTP/1.1" 200 0
DEBUG: Starting new HTTPS connection (1): raw.githubusercontent.com:443
DEBUG: https://s3.amazonaws.com:443 "HEAD /datasets.huggingface.co/datasets/metrics/f1/f1.py HTTP/1.1" 200 0
DEBUG: Starting new HTTPS connection (1): raw.githubusercontent.com:443
Flattening the indices:  60%|████████████████████████████████████████████████████████████████████████████████▍                                                     | 3/5 [00:00<00:00,  3.53ba/s]DEBUG: https://raw.githubusercontent.com:443 "HEAD /huggingface/datasets/2.1.0/metrics/f1/f1.py HTTP/1.1" 200 0
DEBUG: https://raw.githubusercontent.com:443 "HEAD /huggingface/datasets/2.1.0/metrics/f1/f1.py HTTP/1.1" 200 0
DEBUG: Starting new HTTPS connection (1): s3.amazonaws.com:443
DEBUG: Starting new HTTPS connection (1): s3.amazonaws.com:443
DEBUG: https://raw.githubusercontent.com:443 "HEAD /huggingface/datasets/2.1.0/metrics/f1/f1.py HTTP/1.1" 200 0
DEBUG: Starting new HTTPS connection (1): s3.amazonaws.com:443
Flattening the indices: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 5/5 [00:01<00:00,  3.92ba/s]
DEBUG: https://s3.amazonaws.com:443 "HEAD /datasets.huggingface.co/datasets/metrics/recall/recall.py HTTP/1.1" 200 0
DEBUG: https://s3.amazonaws.com:443 "HEAD /datasets.huggingface.co/datasets/metrics/recall/recall.py HTTP/1.1" 200 0
DEBUG: Starting new HTTPS connection (1): raw.githubusercontent.com:443
DEBUG: Starting new HTTPS connection (1): raw.githubusercontent.com:443
Flattening the indices:   0%|                                                                                                                                              | 0/1 [00:00<?, ?ba/s]DEBUG: https://s3.amazonaws.com:443 "HEAD /datasets.huggingface.co/datasets/metrics/recall/recall.py HTTP/1.1" 200 0
DEBUG: Starting new HTTPS connection (1): raw.githubusercontent.com:443
Flattening the indices: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 14.17ba/s]
INFO: Saved dataset to /home/aclifton/rf_fp/proto_data/rffp_dataset
INFO: time to create rffp_data: 4.077894926071167 s
DEBUG: Starting new HTTPS connection (1): s3.amazonaws.com:443
DEBUG: https://raw.githubusercontent.com:443 "HEAD /huggingface/datasets/2.1.0/metrics/recall/recall.py HTTP/1.1" 200 0
DEBUG: Starting new HTTPS connection (1): s3.amazonaws.com:443
DEBUG: https://raw.githubusercontent.com:443 "HEAD /huggingface/datasets/2.1.0/metrics/recall/recall.py HTTP/1.1" 200 0
DEBUG: Starting new HTTPS connection (1): s3.amazonaws.com:443
DEBUG: https://raw.githubusercontent.com:443 "HEAD /huggingface/datasets/2.1.0/metrics/recall/recall.py HTTP/1.1" 200 0
DEBUG: Starting new HTTPS connection (1): s3.amazonaws.com:443
DEBUG: https://s3.amazonaws.com:443 "HEAD /datasets.huggingface.co/datasets/metrics/accuracy/accuracy.py HTTP/1.1" 200 0
DEBUG: Starting new HTTPS connection (1): raw.githubusercontent.com:443
DEBUG: https://s3.amazonaws.com:443 "HEAD /datasets.huggingface.co/datasets/metrics/precision/precision.py HTTP/1.1" 200 0
DEBUG: Starting new HTTPS connection (1): raw.githubusercontent.com:443
DEBUG: https://s3.amazonaws.com:443 "HEAD /datasets.huggingface.co/datasets/metrics/precision/precision.py HTTP/1.1" 200 0
DEBUG: Starting new HTTPS connection (1): raw.githubusercontent.com:443
DEBUG: https://raw.githubusercontent.com:443 "HEAD /huggingface/datasets/2.1.0/metrics/accuracy/accuracy.py HTTP/1.1" 200 0
DEBUG: Starting new HTTPS connection (1): s3.amazonaws.com:443
DEBUG: https://s3.amazonaws.com:443 "HEAD /datasets.huggingface.co/datasets/metrics/precision/precision.py HTTP/1.1" 200 0
DEBUG: Starting new HTTPS connection (1): raw.githubusercontent.com:443
DEBUG: https://raw.githubusercontent.com:443 "HEAD /huggingface/datasets/2.1.0/metrics/precision/precision.py HTTP/1.1" 200 0
DEBUG: https://raw.githubusercontent.com:443 "HEAD /huggingface/datasets/2.1.0/metrics/precision/precision.py HTTP/1.1" 200 0
DEBUG: https://raw.githubusercontent.com:443 "HEAD /huggingface/datasets/2.1.0/metrics/precision/precision.py HTTP/1.1" 200 0
DEBUG: https://s3.amazonaws.com:443 "HEAD /datasets.huggingface.co/datasets/metrics/f1/f1.py HTTP/1.1" 200 0
DEBUG: Starting new HTTPS connection (1): raw.githubusercontent.com:443
DEBUG: https://raw.githubusercontent.com:443 "HEAD /huggingface/datasets/2.1.0/metrics/f1/f1.py HTTP/1.1" 200 0
DEBUG: Starting new HTTPS connection (1): s3.amazonaws.com:443
DEBUG: https://s3.amazonaws.com:443 "HEAD /datasets.huggingface.co/datasets/metrics/recall/recall.py HTTP/1.1" 200 0
DEBUG: Starting new HTTPS connection (1): raw.githubusercontent.com:443
DEBUG: https://raw.githubusercontent.com:443 "HEAD /huggingface/datasets/2.1.0/metrics/recall/recall.py HTTP/1.1" 200 0
DEBUG: Starting new HTTPS connection (1): s3.amazonaws.com:443
DEBUG: https://s3.amazonaws.com:443 "HEAD /datasets.huggingface.co/datasets/metrics/precision/precision.py HTTP/1.1" 200 0
DEBUG: Starting new HTTPS connection (1): raw.githubusercontent.com:443
DEBUG: https://raw.githubusercontent.com:443 "HEAD /huggingface/datasets/2.1.0/metrics/precision/precision.py HTTP/1.1" 200 0These are not outputs that I explicitly ask to be printed out. It seems they come from the datasets library or somewhere else. Is it possible to have these statements printed out only in the local main process?
2022-08-01T16:52:27Z
Here’s another good example that occurs during a training loop:EPOCH 1/1:   0%|                                                                                                                                                     | 0/4456 [00:00<?, ?it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:   0%|▎                                                                                                                                            | 8/4456 [00:00<02:15, 32.95it/s]INFO: Reducer buckets have been rebuilt in this iteration.
INFO: Reducer buckets have been rebuilt in this iteration.
INFO: Reducer buckets have been rebuilt in this iteration.
INFO: Reducer buckets have been rebuilt in this iteration.
INFO: Successfully logged to WandB
   EPOCH 1/1:   0%|▌                                                                                                                                           | 16/4456 [00:00<01:55, 38.46it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:   1%|▊                                                                                                                                           | 24/4456 [00:00<01:58, 37.52it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:   1%|█                                                                                                                                           | 32/4456 [00:00<01:58, 37.20it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:   1%|█▎                                                                                                                                          | 40/4456 [00:01<01:59, 36.95it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:   1%|█▌                                                                                                                                          | 48/4456 [00:01<01:59, 36.81it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:   1%|█▊                                                                                                                                          | 56/4456 [00:01<01:59, 36.73it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:   1%|██                                                                                                                                          | 64/4456 [00:01<01:59, 36.70it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:   2%|██▎                                                                                                                                         | 72/4456 [00:01<01:59, 36.69it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:   2%|██▌                                                                                                                                         | 80/4456 [00:02<01:59, 36.67it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:   2%|██▊                                                                                                                                         | 88/4456 [00:02<01:59, 36.61it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:   2%|███                                                                                                                                         | 96/4456 [00:02<01:59, 36.55it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:   2%|███▏                                                                                                                                       | 104/4456 [00:02<01:59, 36.53it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:   3%|███▍                                                                                                                                       | 112/4456 [00:03<01:58, 36.54it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:   3%|███▋                                                                                                                                       | 120/4456 [00:03<01:58, 36.55it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:   3%|███▉                                                                                                                                       | 128/4456 [00:03<01:58, 36.57it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:   3%|████▏                                                                                                                                      | 136/4456 [00:03<01:58, 36.61it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:   3%|████▍                                                                                                                                      | 144/4456 [00:03<01:57, 36.59it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:   3%|████▋                                                                                                                                      | 152/4456 [00:04<01:57, 36.56it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:   4%|████▉                                                                                                                                      | 160/4456 [00:04<01:57, 36.56it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:   4%|█████▏                                                                                                                                     | 168/4456 [00:04<01:57, 36.59it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:   4%|█████▍                                                                                                                                     | 176/4456 [00:04<01:56, 36.60it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:   4%|█████▋                                                                                                                                     | 184/4456 [00:05<01:56, 36.55it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:   4%|█████▉                                                                                                                                     | 192/4456 [00:05<01:56, 36.56it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:   4%|██████▏                                                                                                                                    | 200/4456 [00:05<01:56, 36.56it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:   5%|██████▍                                                                                                                                    | 208/4456 [00:05<01:56, 36.56it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:   5%|██████▋                                                                                                                                    | 216/4456 [00:05<01:56, 36.51it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:   5%|██████▉                                                                                                                                    | 224/4456 [00:06<01:55, 36.49it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:   5%|███████▏                                                                                                                                   | 232/4456 [00:06<01:55, 36.46it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:   5%|███████▍                                                                                                                                   | 240/4456 [00:06<01:55, 36.49it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:   6%|███████▋                                                                                                                                   | 248/4456 [00:06<01:55, 36.52it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:   6%|███████▉                                                                                                                                   | 256/4456 [00:06<01:55, 36.50it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:   6%|████████▏                                                                                                                                  | 264/4456 [00:07<01:54, 36.51it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:   6%|████████▍                                                                                                                                  | 272/4456 [00:07<01:54, 36.50it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:   6%|████████▋                                                                                                                                  | 280/4456 [00:07<01:54, 36.46it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:   6%|████████▉                                                                                                                                  | 288/4456 [00:07<01:54, 36.49it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:   7%|█████████▏                                                                                                                                 | 296/4456 [00:08<01:53, 36.50it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:   7%|█████████▍                                                                                                                                 | 304/4456 [00:08<01:53, 36.50it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:   7%|█████████▋                                                                                                                                 | 312/4456 [00:08<01:53, 36.46it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:   7%|█████████▉                                                                                                                                 | 320/4456 [00:08<01:53, 36.48it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:   7%|██████████▏                                                                                                                                | 328/4456 [00:08<01:53, 36.48it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:   8%|██████████▍                                                                                                                                | 336/4456 [00:09<01:52, 36.48it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:   8%|██████████▋                                                                                                                                | 344/4456 [00:09<01:52, 36.48it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:   8%|██████████▉                                                                                                                                | 352/4456 [00:09<01:52, 36.47it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:   8%|███████████▏                                                                                                                               | 360/4456 [00:09<01:52, 36.48it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:   8%|███████████▍                                                                                                                               | 368/4456 [00:10<01:52, 36.48it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:   8%|███████████▋                                                                                                                               | 376/4456 [00:10<01:51, 36.45it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:   9%|███████████▉                                                                                                                               | 384/4456 [00:10<01:51, 36.44it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:   9%|████████████▏                                                                                                                              | 392/4456 [00:10<01:51, 36.41it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:   9%|████████████▍                                                                                                                              | 400/4456 [00:10<01:51, 36.44it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:   9%|████████████▋                                                                                                                              | 408/4456 [00:11<01:51, 36.43it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:   9%|████████████▉                                                                                                                              | 416/4456 [00:11<01:50, 36.45it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:  10%|█████████████▏                                                                                                                             | 424/4456 [00:11<01:50, 36.41it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:  10%|█████████████▍                                                                                                                             | 432/4456 [00:11<01:50, 36.36it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:  10%|█████████████▋                                                                                                                             | 440/4456 [00:12<01:50, 36.37it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:  10%|█████████████▉                                                                                                                             | 448/4456 [00:12<01:50, 36.37it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:  10%|██████████████▏                                                                                                                            | 456/4456 [00:12<01:49, 36.40it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:  10%|██████████████▍                                                                                                                            | 464/4456 [00:12<01:49, 36.43it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:  11%|██████████████▋                                                                                                                            | 472/4456 [00:12<01:49, 36.44it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:  11%|██████████████▉                                                                                                                            | 480/4456 [00:13<01:49, 36.43it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:  11%|███████████████▏                                                                                                                           | 488/4456 [00:13<01:48, 36.44it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:  11%|███████████████▍                                                                                                                           | 496/4456 [00:13<01:48, 36.39it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:  11%|███████████████▋                                                                                                                           | 504/4456 [00:13<01:48, 36.39it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:  11%|███████████████▉                                                                                                                           | 512/4456 [00:14<01:48, 36.20it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:  12%|████████████████▏                                                                                                                          | 520/4456 [00:14<01:48, 36.22it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:  12%|████████████████▍                                                                                                                          | 528/4456 [00:14<01:48, 36.28it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:  12%|████████████████▋                                                                                                                          | 536/4456 [00:14<01:47, 36.34it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:  12%|████████████████▉                                                                                                                          | 544/4456 [00:14<01:47, 36.33it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:  12%|█████████████████▏                                                                                                                         | 552/4456 [00:15<01:47, 36.34it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:  13%|█████████████████▍                                                                                                                         | 560/4456 [00:15<01:47, 36.34it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:  13%|█████████████████▋                                                                                                                         | 568/4456 [00:15<01:46, 36.34it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:  13%|█████████████████▉                                                                                                                         | 576/4456 [00:15<01:46, 36.38it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:  13%|██████████████████▏                                                                                                                        | 584/4456 [00:16<01:46, 36.37it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:  13%|██████████████████▍                                                                                                                        | 592/4456 [00:16<01:46, 36.35it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:  13%|██████████████████▋                                                                                                                        | 600/4456 [00:16<01:46, 36.32it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:  14%|██████████████████▉                                                                                                                        | 608/4456 [00:16<01:45, 36.34it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:  14%|███████████████████▏                                                                                                                       | 616/4456 [00:16<01:45, 36.34it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:  14%|███████████████████▍                                                                                                                       | 624/4456 [00:17<01:45, 36.32it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:  14%|███████████████████▋                                                                                                                       | 632/4456 [00:17<01:45, 36.32it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:  14%|███████████████████▉                                                                                                                       | 640/4456 [00:17<01:45, 36.30it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:  15%|████████████████████▏                                                                                                                      | 648/4456 [00:17<01:44, 36.29it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:  15%|████████████████████▍                                                                                                                      | 656/4456 [00:17<01:44, 36.28it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:  15%|████████████████████▋                                                                                                                      | 664/4456 [00:18<01:44, 36.25it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:  15%|████████████████████▉                                                                                                                      | 672/4456 [00:18<01:44, 36.22it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:  15%|█████████████████████▏                                                                                                                     | 680/4456 [00:18<01:44, 36.27it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:  15%|█████████████████████▍                                                                                                                     | 688/4456 [00:18<01:43, 36.27it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:  16%|█████████████████████▋                                                                                                                     | 696/4456 [00:19<01:43, 36.28it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:  16%|█████████████████████▉                                                                                                                     | 704/4456 [00:19<01:43, 36.31it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:  16%|██████████████████████▏                                                                                                                    | 712/4456 [00:19<01:43, 36.29it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:  16%|██████████████████████▍                                                                                                                    | 720/4456 [00:19<01:42, 36.31it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:  16%|██████████████████████▋                                                                                                                    | 728/4456 [00:19<01:42, 36.33it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:  17%|██████████████████████▉                                                                                                                    | 736/4456 [00:20<01:42, 36.32it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:  17%|███████████████████████▏                                                                                                                   | 744/4456 [00:20<01:42, 36.32it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:  17%|███████████████████████▍                                                                                                                   | 752/4456 [00:20<01:42, 36.28it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:  17%|███████████████████████▋                                                                                                                   | 760/4456 [00:20<01:41, 36.28it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:  17%|███████████████████████▉                                                                                                                   | 768/4456 [00:21<01:41, 36.26it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:  17%|████████████████████████▏                                                                                                                  | 776/4456 [00:21<01:41, 36.25it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:  18%|████████████████████████▍                                                                                                                  | 784/4456 [00:21<01:41, 36.26it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:  18%|████████████████████████▋                                                                                                                  | 792/4456 [00:21<01:40, 36.28it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:  18%|████████████████████████▉                                                                                                                  | 800/4456 [00:21<01:40, 36.24it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:  18%|█████████████████████████▏                                                                                                                 | 808/4456 [00:22<01:40, 36.18it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:  18%|█████████████████████████▍                                                                                                                 | 816/4456 [00:22<01:40, 36.20it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:  18%|█████████████████████████▋                                                                                                                 | 824/4456 [00:22<01:40, 36.20it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:  19%|█████████████████████████▉                                                                                                                 | 832/4456 [00:22<01:40, 36.20it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:  19%|██████████████████████████▏                                                                                                                | 840/4456 [00:23<01:39, 36.21it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:  19%|██████████████████████████▍                                                                                                                | 848/4456 [00:23<01:39, 36.19it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:  19%|██████████████████████████▋                                                                                                                | 856/4456 [00:23<01:39, 36.19it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:  19%|██████████████████████████▉                                                                                                                | 864/4456 [00:23<01:39, 36.19it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:  20%|███████████████████████████▏                                                                                                               | 872/4456 [00:23<01:38, 36.21it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:  20%|███████████████████████████▍                                                                                                               | 880/4456 [00:24<01:38, 36.20it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:  20%|███████████████████████████▋                                                                                                               | 888/4456 [00:24<01:38, 36.22it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:  20%|███████████████████████████▉                                                                                                               | 896/4456 [00:24<01:38, 36.21it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:  20%|████████████████████████████▏                                                                                                              | 904/4456 [00:24<01:38, 36.19it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:  20%|████████████████████████████▍                                                                                                              | 912/4456 [00:25<01:37, 36.22it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:  21%|████████████████████████████▋                                                                                                              | 920/4456 [00:25<01:37, 36.24it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:  21%|████████████████████████████▉                                                                                                              | 928/4456 [00:25<01:37, 36.22it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:  21%|█████████████████████████████▏                                                                                                             | 936/4456 [00:25<01:37, 36.21it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:  21%|█████████████████████████████▍                                                                                                             | 944/4456 [00:25<01:36, 36.21it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:  21%|█████████████████████████████▋                                                                                                             | 952/4456 [00:26<01:36, 36.23it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:  22%|█████████████████████████████▉                                                                                                             | 960/4456 [00:26<01:36, 36.20it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:  22%|██████████████████████████████▏                                                                                                            | 968/4456 [00:26<01:36, 36.20it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:  22%|██████████████████████████████▍                                                                                                            | 976/4456 [00:26<01:36, 36.18it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:  22%|██████████████████████████████▋                                                                                                            | 984/4456 [00:27<01:35, 36.18it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:  22%|██████████████████████████████▉                                                                                                            | 992/4456 [00:27<01:35, 36.17it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:  22%|██████████████████████████████▉                                                                                                           | 1000/4456 [00:27<01:35, 36.15it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:  23%|███████████████████████████████▏                                                                                                          | 1008/4456 [00:27<01:35, 36.13it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:  23%|███████████████████████████████▍                                                                                                          | 1016/4456 [00:27<01:35, 36.12it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:  23%|███████████████████████████████▋                                                                                                          | 1024/4456 [00:28<01:35, 36.12it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:  23%|███████████████████████████████▉                                                                                                          | 1032/4456 [00:28<01:34, 36.13it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:  23%|████████████████████████████████▏                                                                                                         | 1040/4456 [00:28<01:34, 36.14it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:  24%|████████████████████████████████▍                                                                                                         | 1048/4456 [00:28<01:34, 36.12it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:  24%|████████████████████████████████▋                                                                                                         | 1056/4456 [00:29<01:34, 36.12it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:  24%|████████████████████████████████▉                                                                                                         | 1064/4456 [00:29<01:33, 36.14it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:  24%|█████████████████████████████████▏                                                                                                        | 1072/4456 [00:29<01:33, 36.14it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:  24%|█████████████████████████████████▍                                                                                                        | 1080/4456 [00:29<01:33, 36.16it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:  24%|█████████████████████████████████▋                                                                                                        ```
2022-08-03T13:17:30Z
aclifton314:These are not outputs that I explicitly ask to be printed out. It seems they come from the datasets library or somewhere else. Is it possible to have these statements printed out only in the local main process?Yep, take a peek in the transformers examples to see how this is donehereaclifton314:Here’s another good example that occurs during a training loop:That one I’m unsure of
2022-08-03T18:06:33Z
That works! Thanks@muellerzr!!