NeoXArgs.configure_distributed_args() using world size: 64 and model-parallel size: 1 > building HFTokenizer tokenizer ... > padded vocab (size: 50277) with 27 dummy tokens (new size: 50304) > setting tensorboard ... > initializing torch distributed ... > initializing model parallel with size 1 MPU DP: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63] MPU PP: [0] MPU PP: [1] MPU PP: [2] MPU PP: [3] MPU PP: [4] MPU PP: [5] MPU PP: [6] MPU PP: [7] MPU PP: [8] MPU PP: [9] MPU PP: [10] MPU PP: [11] MPU PP: [12] MPU PP: [13] MPU PP: [14] MPU PP: [15] MPU PP: [16] MPU PP: [17] MPU PP: [18] MPU PP: [19] MPU PP: [20] MPU PP: [21] MPU PP: [22] MPU PP: [23] MPU PP: [24] MPU PP: [25] MPU PP: [26] MPU PP: [27] MPU PP: [28] MPU PP: [29] MPU PP: [30] MPU PP: [31] MPU PP: [32] MPU PP: [33] MPU PP: [34] MPU PP: [35] MPU PP: [36] MPU PP: [37] MPU PP: [38] MPU PP: [39] MPU PP: [40] MPU PP: [41] MPU PP: [42] MPU PP: [43] MPU PP: [44] MPU PP: [45] MPU PP: [46] MPU PP: [47] MPU PP: [48] MPU PP: [49] MPU PP: [50] MPU PP: [51] MPU PP: [52] MPU PP: [53] MPU PP: [54] MPU PP: [55] MPU PP: [56] MPU PP: [57] MPU PP: [58] MPU PP: [59] MPU PP: [60] MPU PP: [61] MPU PP: [62] MPU PP: [63] MPU MP: [0] MPU MP: [1] MPU MP: [2] MPU MP: [3] MPU MP: [4] MPU MP: [5] MPU MP: [6] MPU MP: [7] MPU MP: [8] MPU MP: [9] MPU MP: [10] MPU MP: [11] MPU MP: [12] MPU MP: [13] MPU MP: [14] MPU MP: [15] MPU MP: [16] MPU MP: [17] MPU MP: [18] MPU MP: [19] MPU MP: [20] MPU MP: [21] MPU MP: [22] MPU MP: [23] MPU MP: [24] MPU MP: [25] MPU MP: [26] MPU MP: [27] MPU MP: [28] MPU MP: [29] MPU MP: [30] MPU MP: [31] MPU MP: [32] MPU MP: [33] MPU MP: [34] MPU MP: [35] MPU MP: [36] MPU MP: [37] MPU MP: [38] MPU MP: [39] MPU MP: [40] MPU MP: [41] MPU MP: [42] MPU MP: [43] MPU MP: [44] MPU MP: [45] MPU MP: [46] MPU MP: [47] MPU MP: [48] MPU MP: [49] MPU MP: [50] MPU MP: [51] MPU MP: [52] MPU MP: [53] MPU MP: [54] MPU MP: [55] MPU MP: [56] MPU MP: [57] MPU MP: [58] MPU MP: [59] MPU MP: [60] MPU MP: [61] MPU MP: [62] MPU MP: [63] > setting random seeds to 1234 ... building GPT2 model ... SEED_LAYERS=False BASE_SEED=1234 SEED_FN=None Using topology: {ProcessCoord(pipe=0, data=0, model=0): 0, ProcessCoord(pipe=0, data=1, model=0): 1, ProcessCoord(pipe=0, data=2, model=0): 2, ProcessCoord(pipe=0, data=3, model=0): 3, ProcessCoord(pipe=0, data=4, model=0): 4, ProcessCoord(pipe=0, data=5, model=0): 5, ProcessCoord(pipe=0, data=6, model=0): 6, ProcessCoord(pipe=0, data=7, model=0): 7, ProcessCoord(pipe=0, data=8, model=0): 8, ProcessCoord(pipe=0, data=9, model=0): 9, ProcessCoord(pipe=0, data=10, model=0): 10, ProcessCoord(pipe=0, data=11, model=0): 11, ProcessCoord(pipe=0, data=12, model=0): 12, ProcessCoord(pipe=0, data=13, model=0): 13, ProcessCoord(pipe=0, data=14, model=0): 14, ProcessCoord(pipe=0, data=15, model=0): 15, ProcessCoord(pipe=0, data=16, model=0): 16, ProcessCoord(pipe=0, data=17, model=0): 17, ProcessCoord(pipe=0, data=18, model=0): 18, ProcessCoord(pipe=0, data=19, model=0): 19, ProcessCoord(pipe=0, data=20, model=0): 20, ProcessCoord(pipe=0, data=21, model=0): 21, ProcessCoord(pipe=0, data=22, model=0): 22, ProcessCoord(pipe=0, data=23, model=0): 23, ProcessCoord(pipe=0, data=24, model=0): 24, ProcessCoord(pipe=0, data=25, model=0): 25, ProcessCoord(pipe=0, data=26, model=0): 26, ProcessCoord(pipe=0, data=27, model=0): 27, ProcessCoord(pipe=0, data=28, model=0): 28, ProcessCoord(pipe=0, data=29, model=0): 29, ProcessCoord(pipe=0, data=30, model=0): 30, ProcessCoord(pipe=0, data=31, model=0): 31, ProcessCoord(pipe=0, data=32, model=0): 32, ProcessCoord(pipe=0, data=33, model=0): 33, ProcessCoord(pipe=0, data=34, model=0): 34, ProcessCoord(pipe=0, data=35, model=0): 35, ProcessCoord(pipe=0, data=36, model=0): 36, ProcessCoord(pipe=0, data=37, model=0): 37, ProcessCoord(pipe=0, data=38, model=0): 38, ProcessCoord(pipe=0, data=39, model=0): 39, ProcessCoord(pipe=0, data=40, model=0): 40, ProcessCoord(pipe=0, data=41, model=0): 41, ProcessCoord(pipe=0, data=42, model=0): 42, ProcessCoord(pipe=0, data=43, model=0): 43, ProcessCoord(pipe=0, data=44, model=0): 44, ProcessCoord(pipe=0, data=45, model=0): 45, ProcessCoord(pipe=0, data=46, model=0): 46, ProcessCoord(pipe=0, data=47, model=0): 47, ProcessCoord(pipe=0, data=48, model=0): 48, ProcessCoord(pipe=0, data=49, model=0): 49, ProcessCoord(pipe=0, data=50, model=0): 50, ProcessCoord(pipe=0, data=51, model=0): 51, ProcessCoord(pipe=0, data=52, model=0): 52, ProcessCoord(pipe=0, data=53, model=0): 53, ProcessCoord(pipe=0, data=54, model=0): 54, ProcessCoord(pipe=0, data=55, model=0): 55, ProcessCoord(pipe=0, data=56, model=0): 56, ProcessCoord(pipe=0, data=57, model=0): 57, ProcessCoord(pipe=0, data=58, model=0): 58, ProcessCoord(pipe=0, data=59, model=0): 59, ProcessCoord(pipe=0, data=60, model=0): 60, ProcessCoord(pipe=0, data=61, model=0): 61, ProcessCoord(pipe=0, data=62, model=0): 62, ProcessCoord(pipe=0, data=63, model=0): 63} stage=0 layers=17 0: EmbeddingPipe 1: _pre_transformer_block 2: ParallelTransformerLayerPipe 3: ParallelTransformerLayerPipe 4: ParallelTransformerLayerPipe 5: ParallelTransformerLayerPipe 6: ParallelTransformerLayerPipe 7: ParallelTransformerLayerPipe 8: ParallelTransformerLayerPipe 9: ParallelTransformerLayerPipe 10: ParallelTransformerLayerPipe 11: ParallelTransformerLayerPipe 12: ParallelTransformerLayerPipe 13: ParallelTransformerLayerPipe 14: _post_transformer_block 15: NormPipe 16: EmbeddingPipe loss: partial Configuring Optimizer type: Adam with params: {'lr': 0.0006, 'betas': [0.9, 0.95], 'eps': 1e-08} > learning rate decay style: cosine DeepSpeed is enabled. > number of parameters on model parallel rank 0: 151881216 > total params: 151,881,216 Unable to load checkpoint. Loading checkpoint and starting from iteration 0 > building train, validation, and test datasets ... reading sizes... reading pointers... reading document index... creating numpy buffer of mmap... creating memory view of numpy buffer... > dataset split: train: document indices in [0, 130154259) total of 130154259 documents validation: document indices in [130154259, 134183803) total of 4029544 documents test: document indices in [134183803, 134318121) total of 134318 documents > loading doc-idx mapping from /u/wangh/workspace_ptmp/language_dataset/pile/pile_0.87_deduped_text_document_train_indexmap_146432000ns_2048sl_1234s_doc_idx.npy > loading sample-idx mapping from /u/wangh/workspace_ptmp/language_dataset/pile/pile_0.87_deduped_text_document_train_indexmap_146432000ns_2048sl_1234s_sample_idx.npy > loading shuffle-idx mapping from /u/wangh/workspace_ptmp/language_dataset/pile/pile_0.87_deduped_text_document_train_indexmap_146432000ns_2048sl_1234s_shuffle_idx.npy loaded indexed file in 0.205 seconds total number of samples: 195915016 total number of epochs: 2 > loading doc-idx mapping from /u/wangh/workspace_ptmp/language_dataset/pile/pile_0.87_deduped_text_document_valid_indexmap_1536000ns_2048sl_1234s_doc_idx.npy > loading sample-idx mapping from /u/wangh/workspace_ptmp/language_dataset/pile/pile_0.87_deduped_text_document_valid_indexmap_1536000ns_2048sl_1234s_sample_idx.npy > loading shuffle-idx mapping from /u/wangh/workspace_ptmp/language_dataset/pile/pile_0.87_deduped_text_document_valid_indexmap_1536000ns_2048sl_1234s_shuffle_idx.npy loaded indexed file in 0.161 seconds total number of samples: 3097460 total number of epochs: 1 > loading doc-idx mapping from /u/wangh/workspace_ptmp/language_dataset/pile/pile_0.87_deduped_text_document_test_indexmap_102400ns_2048sl_1234s_doc_idx.npy > loading sample-idx mapping from /u/wangh/workspace_ptmp/language_dataset/pile/pile_0.87_deduped_text_document_test_indexmap_102400ns_2048sl_1234s_sample_idx.npy > loading shuffle-idx mapping from /u/wangh/workspace_ptmp/language_dataset/pile/pile_0.87_deduped_text_document_test_indexmap_102400ns_2048sl_1234s_shuffle_idx.npy loaded indexed file in 0.060 seconds total number of samples: 102462 total number of epochs: 1 setting training data start iteration to 0 setting validation data start iteration to 0 done with setups ... time (ms) | model and optimizer: 2453.77 | train/valid/test data iterators: 16354.41 training ... WARNING: Deleting old checkpoints: /u/wangh/workspace_ptmp/checkpoints_gptneox/flownet/160M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step0 WARNING: Deleting old checkpoints: /u/wangh/workspace_ptmp/checkpoints_gptneox/flownet/160M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step1 WARNING: Deleting old checkpoints: /u/wangh/workspace_ptmp/checkpoints_gptneox/flownet/160M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step2 WARNING: Deleting old checkpoints: /u/wangh/workspace_ptmp/checkpoints_gptneox/flownet/160M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step4 WARNING: Deleting old checkpoints: /u/wangh/workspace_ptmp/checkpoints_gptneox/flownet/160M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step8 WARNING: Deleting old checkpoints: /u/wangh/workspace_ptmp/checkpoints_gptneox/flownet/160M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step16 samples/sec: 1571.528 | iteration 500/ 143000 | elapsed time per iteration (ms): 651.6 | learning rate: 2.098E-04 | approx flops per GPU: 61.0TFLOPS | lm_loss: 6.777926E+00 | loss scale: 4096.0 | number of skipped iterations: 0 | number of nan iterations: 0 | after 500 iterations memory (MB) | allocated: 632.1875 | max allocated: 10746.48486328125 | reserved: 11840.0 | max reserved: 11840.0 time (ms) | forward: 166.94 | backward: 454.90 | backward-backward: 454.86 | backward-allreduce: 0.00 | optimizer: 16.62 | batch generator: 3.66 WARNING: Deleting old checkpoints: /u/wangh/workspace_ptmp/checkpoints_gptneox/flownet/160M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step32 samples/sec: 1615.530 | iteration 1000/ 143000 | elapsed time per iteration (ms): 633.8 | learning rate: 4.196E-04 | approx flops per GPU: 62.7TFLOPS | lm_loss: 4.169707E+00 | loss scale: 8192.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 161.72 | backward: 453.27 | backward-backward: 453.23 | backward-allreduce: 0.00 | optimizer: 16.58 | batch generator: 3.47 WARNING: Deleting old checkpoints: /u/wangh/workspace_ptmp/checkpoints_gptneox/flownet/160M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step64 samples/sec: 1615.373 | iteration 1500/ 143000 | elapsed time per iteration (ms): 633.9 | learning rate: 6.000E-04 | approx flops per GPU: 62.7TFLOPS | lm_loss: 3.411946E+00 | loss scale: 8192.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 161.75 | backward: 453.25 | backward-backward: 453.22 | backward-allreduce: 0.00 | optimizer: 16.60 | batch generator: 3.51 samples/sec: 1618.369 | iteration 2000/ 143000 | elapsed time per iteration (ms): 632.7 | learning rate: 6.000E-04 | approx flops per GPU: 62.8TFLOPS | lm_loss: 3.141428E+00 | loss scale: 16384.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 161.63 | backward: 453.46 | backward-backward: 453.42 | backward-allreduce: 0.00 | optimizer: 16.60 | batch generator: 3.27 WARNING: Deleting old checkpoints: /u/wangh/workspace_ptmp/checkpoints_gptneox/flownet/160M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step128 samples/sec: 1589.771 | iteration 2500/ 143000 | elapsed time per iteration (ms): 644.1 | learning rate: 5.999E-04 | approx flops per GPU: 61.7TFLOPS | lm_loss: 3.001465E+00 | loss scale: 16384.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 168.03 | backward: 457.00 | backward-backward: 456.96 | backward-allreduce: 0.00 | optimizer: 16.60 | batch generator: 9.63 samples/sec: 1618.286 | iteration 3000/ 143000 | elapsed time per iteration (ms): 632.8 | learning rate: 5.998E-04 | approx flops per GPU: 62.8TFLOPS | lm_loss: 2.916622E+00 | loss scale: 32768.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 162.41 | backward: 452.72 | backward-backward: 452.68 | backward-allreduce: 0.00 | optimizer: 16.60 | batch generator: 3.94 WARNING: Deleting old checkpoints: /u/wangh/workspace_ptmp/checkpoints_gptneox/flownet/160M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step256 samples/sec: 1615.109 | iteration 3500/ 143000 | elapsed time per iteration (ms): 634.0 | learning rate: 5.997E-04 | approx flops per GPU: 62.7TFLOPS | lm_loss: 2.861156E+00 | loss scale: 32768.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 163.04 | backward: 452.15 | backward-backward: 452.11 | backward-allreduce: 0.00 | optimizer: 16.51 | batch generator: 4.49 samples/sec: 1618.317 | iteration 4000/ 143000 | elapsed time per iteration (ms): 632.8 | learning rate: 5.996E-04 | approx flops per GPU: 62.8TFLOPS | lm_loss: 2.819780E+00 | loss scale: 65536.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 162.81 | backward: 452.35 | backward-backward: 452.32 | backward-allreduce: 0.00 | optimizer: 16.56 | batch generator: 4.33 WARNING: Deleting old checkpoints: /u/wangh/workspace_ptmp/checkpoints_gptneox/flownet/160M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step512 samples/sec: 1614.908 | iteration 4500/ 143000 | elapsed time per iteration (ms): 634.1 | learning rate: 5.994E-04 | approx flops per GPU: 62.7TFLOPS | lm_loss: 2.788997E+00 | loss scale: 65536.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 162.48 | backward: 452.56 | backward-backward: 452.53 | backward-allreduce: 0.00 | optimizer: 16.62 | batch generator: 3.94 samples/sec: 1618.495 | iteration 5000/ 143000 | elapsed time per iteration (ms): 632.7 | learning rate: 5.992E-04 | approx flops per GPU: 62.8TFLOPS | lm_loss: 2.763110E+00 | loss scale: 32768.0 | number of skipped iterations: 2 | number of nan iterations: 0 | time (ms) | forward: 162.06 | backward: 453.04 | backward-backward: 453.01 | backward-allreduce: 0.00 | optimizer: 16.49 | batch generator: 3.26 WARNING: Deleting old checkpoints: /u/wangh/workspace_ptmp/checkpoints_gptneox/flownet/160M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step1000 samples/sec: 1614.694 | iteration 5500/ 143000 | elapsed time per iteration (ms): 634.2 | learning rate: 5.989E-04 | approx flops per GPU: 62.7TFLOPS | lm_loss: 2.742985E+00 | loss scale: 32768.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 161.94 | backward: 453.20 | backward-backward: 453.16 | backward-allreduce: 0.00 | optimizer: 16.59 | batch generator: 3.16 samples/sec: 1618.010 | iteration 6000/ 143000 | elapsed time per iteration (ms): 632.9 | learning rate: 5.986E-04 | approx flops per GPU: 62.8TFLOPS | lm_loss: 2.724437E+00 | loss scale: 65536.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 162.37 | backward: 452.87 | backward-backward: 452.83 | backward-allreduce: 0.00 | optimizer: 16.55 | batch generator: 3.59 WARNING: Deleting old checkpoints: /u/wangh/workspace_ptmp/checkpoints_gptneox/flownet/160M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step2000 samples/sec: 1614.982 | iteration 6500/ 143000 | elapsed time per iteration (ms): 634.1 | learning rate: 5.983E-04 | approx flops per GPU: 62.7TFLOPS | lm_loss: 2.707893E+00 | loss scale: 65536.0 | number of skipped iterations: 1 | number of nan iterations: 0 | time (ms) | forward: 162.18 | backward: 452.95 | backward-backward: 452.92 | backward-allreduce: 0.00 | optimizer: 16.55 | batch generator: 3.50 samples/sec: 1617.215 | iteration 7000/ 143000 | elapsed time per iteration (ms): 633.2 | learning rate: 5.979E-04 | approx flops per GPU: 62.8TFLOPS | lm_loss: 2.695413E+00 | loss scale: 32768.0 | number of skipped iterations: 1 | number of nan iterations: 0 | time (ms) | forward: 162.16 | backward: 453.27 | backward-backward: 453.24 | backward-allreduce: 0.00 | optimizer: 16.66 | batch generator: 3.63 WARNING: Deleting old checkpoints: /u/wangh/workspace_ptmp/checkpoints_gptneox/flownet/160M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step3000 samples/sec: 1615.156 | iteration 7500/ 143000 | elapsed time per iteration (ms): 634.0 | learning rate: 5.976E-04 | approx flops per GPU: 62.7TFLOPS | lm_loss: 2.684966E+00 | loss scale: 16384.0 | number of skipped iterations: 1 | number of nan iterations: 0 | time (ms) | forward: 161.99 | backward: 453.11 | backward-backward: 453.07 | backward-allreduce: 0.00 | optimizer: 16.55 | batch generator: 3.45 samples/sec: 1618.098 | iteration 8000/ 143000 | elapsed time per iteration (ms): 632.8 | learning rate: 5.971E-04 | approx flops per GPU: 62.8TFLOPS | lm_loss: 2.674111E+00 | loss scale: 16384.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 162.17 | backward: 453.04 | backward-backward: 453.00 | backward-allreduce: 0.00 | optimizer: 16.56 | batch generator: 3.58 WARNING: Deleting old checkpoints: /u/wangh/workspace_ptmp/checkpoints_gptneox/flownet/160M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step4000 samples/sec: 1614.306 | iteration 8500/ 143000 | elapsed time per iteration (ms): 634.3 | learning rate: 5.967E-04 | approx flops per GPU: 62.7TFLOPS | lm_loss: 2.666086E+00 | loss scale: 16384.0 | number of skipped iterations: 2 | number of nan iterations: 0 | time (ms) | forward: 162.58 | backward: 452.88 | backward-backward: 452.85 | backward-allreduce: 0.00 | optimizer: 16.49 | batch generator: 3.93 samples/sec: 1617.808 | iteration 9000/ 143000 | elapsed time per iteration (ms): 633.0 | learning rate: 5.962E-04 | approx flops per GPU: 62.8TFLOPS | lm_loss: 2.656476E+00 | loss scale: 16384.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 162.43 | backward: 452.88 | backward-backward: 452.84 | backward-allreduce: 0.00 | optimizer: 16.62 | batch generator: 3.96 WARNING: Deleting old checkpoints: /u/wangh/workspace_ptmp/checkpoints_gptneox/flownet/160M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step5000 samples/sec: 1613.379 | iteration 9500/ 143000 | elapsed time per iteration (ms): 634.7 | learning rate: 5.957E-04 | approx flops per GPU: 62.6TFLOPS | lm_loss: 2.648393E+00 | loss scale: 8192.0 | number of skipped iterations: 1 | number of nan iterations: 0 | time (ms) | forward: 162.20 | backward: 453.23 | backward-backward: 453.20 | backward-allreduce: 0.00 | optimizer: 16.67 | batch generator: 3.84 samples/sec: 1617.488 | iteration 10000/ 143000 | elapsed time per iteration (ms): 633.1 | learning rate: 5.951E-04 | approx flops per GPU: 62.8TFLOPS | lm_loss: 2.642091E+00 | loss scale: 8192.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 163.00 | backward: 452.47 | backward-backward: 452.43 | backward-allreduce: 0.00 | optimizer: 16.56 | batch generator: 4.51 WARNING: Deleting old checkpoints: /u/wangh/workspace_ptmp/checkpoints_gptneox/flownet/160M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step6000 ---------------------------------------------------------------------------------------------------------- validation results at iteration 10000 | lm_loss value: 2.586534E+00 | lm_loss_ppl value: 1.328365E+01 | ---------------------------------------------------------------------------------------------------------- samples/sec: 1531.874 | iteration 10500/ 143000 | elapsed time per iteration (ms): 668.5 | learning rate: 5.946E-04 | approx flops per GPU: 59.5TFLOPS | lm_loss: 2.636124E+00 | loss scale: 4096.0 | number of skipped iterations: 1 | number of nan iterations: 0 | time (ms) | forward: 163.02 | backward: 452.52 | backward-backward: 452.49 | backward-allreduce: 0.00 | optimizer: 16.57 | batch generator: 5.08 samples/sec: 1616.647 | iteration 11000/ 143000 | elapsed time per iteration (ms): 633.4 | learning rate: 5.939E-04 | approx flops per GPU: 62.8TFLOPS | lm_loss: 2.630334E+00 | loss scale: 4096.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 162.84 | backward: 452.79 | backward-backward: 452.76 | backward-allreduce: 0.00 | optimizer: 16.65 | batch generator: 4.28 WARNING: Deleting old checkpoints: /u/wangh/workspace_ptmp/checkpoints_gptneox/flownet/160M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step7000 samples/sec: 1603.752 | iteration 11500/ 143000 | elapsed time per iteration (ms): 638.5 | learning rate: 5.933E-04 | approx flops per GPU: 62.3TFLOPS | lm_loss: 2.623638E+00 | loss scale: 8192.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 162.87 | backward: 452.87 | backward-backward: 452.83 | backward-allreduce: 0.00 | optimizer: 16.62 | batch generator: 4.24 samples/sec: 1617.216 | iteration 12000/ 143000 | elapsed time per iteration (ms): 633.2 | learning rate: 5.926E-04 | approx flops per GPU: 62.8TFLOPS | lm_loss: 2.621313E+00 | loss scale: 8192.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 162.99 | backward: 452.53 | backward-backward: 452.50 | backward-allreduce: 0.00 | optimizer: 16.58 | batch generator: 4.31 WARNING: Deleting old checkpoints: /u/wangh/workspace_ptmp/checkpoints_gptneox/flownet/160M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step8000 samples/sec: 1613.942 | iteration 12500/ 143000 | elapsed time per iteration (ms): 634.5 | learning rate: 5.919E-04 | approx flops per GPU: 62.7TFLOPS | lm_loss: 2.614554E+00 | loss scale: 16384.0 | number of skipped iterations: 1 | number of nan iterations: 0 | time (ms) | forward: 162.77 | backward: 452.82 | backward-backward: 452.78 | backward-allreduce: 0.00 | optimizer: 16.62 | batch generator: 4.15 samples/sec: 1617.407 | iteration 13000/ 143000 | elapsed time per iteration (ms): 633.1 | learning rate: 5.912E-04 | approx flops per GPU: 62.8TFLOPS | lm_loss: 2.610217E+00 | loss scale: 8192.0 | number of skipped iterations: 1 | number of nan iterations: 0 | time (ms) | forward: 162.61 | backward: 452.85 | backward-backward: 452.81 | backward-allreduce: 0.00 | optimizer: 16.58 | batch generator: 4.01 WARNING: Deleting old checkpoints: /u/wangh/workspace_ptmp/checkpoints_gptneox/flownet/160M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step9000 samples/sec: 1593.530 | iteration 13500/ 143000 | elapsed time per iteration (ms): 642.6 | learning rate: 5.904E-04 | approx flops per GPU: 61.9TFLOPS | lm_loss: 2.605885E+00 | loss scale: 8192.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 162.59 | backward: 453.01 | backward-backward: 452.98 | backward-allreduce: 0.00 | optimizer: 16.59 | batch generator: 4.10 samples/sec: 1615.424 | iteration 14000/ 143000 | elapsed time per iteration (ms): 633.9 | learning rate: 5.896E-04 | approx flops per GPU: 62.7TFLOPS | lm_loss: 2.603294E+00 | loss scale: 16384.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 162.89 | backward: 453.37 | backward-backward: 453.34 | backward-allreduce: 0.00 | optimizer: 16.59 | batch generator: 4.39 WARNING: Deleting old checkpoints: /u/wangh/workspace_ptmp/checkpoints_gptneox/flownet/160M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step10000 samples/sec: 1613.499 | iteration 14500/ 143000 | elapsed time per iteration (ms): 634.6 | learning rate: 5.887E-04 | approx flops per GPU: 62.6TFLOPS | lm_loss: 2.598590E+00 | loss scale: 16384.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 163.00 | backward: 452.51 | backward-backward: 452.47 | backward-allreduce: 0.00 | optimizer: 16.59 | batch generator: 4.40 samples/sec: 1617.322 | iteration 15000/ 143000 | elapsed time per iteration (ms): 633.1 | learning rate: 5.879E-04 | approx flops per GPU: 62.8TFLOPS | lm_loss: 2.596211E+00 | loss scale: 32768.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 162.96 | backward: 452.57 | backward-backward: 452.53 | backward-allreduce: 0.00 | optimizer: 16.58 | batch generator: 4.33 WARNING: Deleting old checkpoints: /u/wangh/workspace_ptmp/checkpoints_gptneox/flownet/160M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step11000 samples/sec: 1615.083 | iteration 15500/ 143000 | elapsed time per iteration (ms): 634.0 | learning rate: 5.870E-04 | approx flops per GPU: 62.7TFLOPS | lm_loss: 2.590233E+00 | loss scale: 32768.0 | number of skipped iterations: 1 | number of nan iterations: 0 | time (ms) | forward: 161.63 | backward: 453.52 | backward-backward: 453.48 | backward-allreduce: 0.00 | optimizer: 16.58 | batch generator: 2.99 samples/sec: 1617.934 | iteration 16000/ 143000 | elapsed time per iteration (ms): 632.9 | learning rate: 5.860E-04 | approx flops per GPU: 62.8TFLOPS | lm_loss: 2.587619E+00 | loss scale: 16384.0 | number of skipped iterations: 1 | number of nan iterations: 0 | time (ms) | forward: 161.28 | backward: 453.92 | backward-backward: 453.88 | backward-allreduce: 0.00 | optimizer: 16.61 | batch generator: 2.68 WARNING: Deleting old checkpoints: /u/wangh/workspace_ptmp/checkpoints_gptneox/flownet/160M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step12000 samples/sec: 1613.376 | iteration 16500/ 143000 | elapsed time per iteration (ms): 634.7 | learning rate: 5.851E-04 | approx flops per GPU: 62.6TFLOPS | lm_loss: 2.585794E+00 | loss scale: 16384.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 161.24 | backward: 453.99 | backward-backward: 453.95 | backward-allreduce: 0.00 | optimizer: 16.76 | batch generator: 2.73 samples/sec: 1618.230 | iteration 17000/ 143000 | elapsed time per iteration (ms): 632.8 | learning rate: 5.841E-04 | approx flops per GPU: 62.8TFLOPS | lm_loss: 2.581061E+00 | loss scale: 32768.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 161.75 | backward: 453.36 | backward-backward: 453.32 | backward-allreduce: 0.00 | optimizer: 16.63 | batch generator: 3.22 WARNING: Deleting old checkpoints: /u/wangh/workspace_ptmp/checkpoints_gptneox/flownet/160M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step13000 samples/sec: 1614.309 | iteration 17500/ 143000 | elapsed time per iteration (ms): 634.3 | learning rate: 5.830E-04 | approx flops per GPU: 62.7TFLOPS | lm_loss: 2.579639E+00 | loss scale: 32768.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 161.31 | backward: 453.92 | backward-backward: 453.88 | backward-allreduce: 0.00 | optimizer: 16.72 | batch generator: 2.81 samples/sec: 1617.718 | iteration 18000/ 143000 | elapsed time per iteration (ms): 633.0 | learning rate: 5.820E-04 | approx flops per GPU: 62.8TFLOPS | lm_loss: 2.574949E+00 | loss scale: 32768.0 | number of skipped iterations: 1 | number of nan iterations: 0 | time (ms) | forward: 161.98 | backward: 453.23 | backward-backward: 453.19 | backward-allreduce: 0.00 | optimizer: 16.70 | batch generator: 3.48 WARNING: Deleting old checkpoints: /u/wangh/workspace_ptmp/checkpoints_gptneox/flownet/160M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step14000 samples/sec: 1613.801 | iteration 18500/ 143000 | elapsed time per iteration (ms): 634.5 | learning rate: 5.809E-04 | approx flops per GPU: 62.7TFLOPS | lm_loss: 2.574907E+00 | loss scale: 16384.0 | number of skipped iterations: 1 | number of nan iterations: 0 | time (ms) | forward: 162.06 | backward: 453.23 | backward-backward: 453.19 | backward-allreduce: 0.00 | optimizer: 16.73 | batch generator: 3.40 samples/sec: 1617.342 | iteration 19000/ 143000 | elapsed time per iteration (ms): 633.1 | learning rate: 5.798E-04 | approx flops per GPU: 62.8TFLOPS | lm_loss: 2.571710E+00 | loss scale: 16384.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 161.92 | backward: 453.42 | backward-backward: 453.39 | backward-allreduce: 0.00 | optimizer: 16.68 | batch generator: 3.19 WARNING: Deleting old checkpoints: /u/wangh/workspace_ptmp/checkpoints_gptneox/flownet/160M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step15000 samples/sec: 1613.536 | iteration 19500/ 143000 | elapsed time per iteration (ms): 634.6 | learning rate: 5.786E-04 | approx flops per GPU: 62.6TFLOPS | lm_loss: 2.569687E+00 | loss scale: 32768.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 162.03 | backward: 453.41 | backward-backward: 453.38 | backward-allreduce: 0.00 | optimizer: 16.74 | batch generator: 3.44 samples/sec: 1451.789 | iteration 20000/ 143000 | elapsed time per iteration (ms): 705.3 | learning rate: 5.774E-04 | approx flops per GPU: 56.4TFLOPS | lm_loss: 2.566677E+00 | loss scale: 32768.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 183.98 | backward: 503.30 | backward-backward: 503.26 | backward-allreduce: 0.00 | optimizer: 17.00 | batch generator: 25.67 WARNING: Deleting old checkpoints: /u/wangh/workspace_ptmp/checkpoints_gptneox/flownet/160M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step16000 ---------------------------------------------------------------------------------------------------------- validation results at iteration 20000 | lm_loss value: 2.514063E+00 | lm_loss_ppl value: 1.235503E+01 | ---------------------------------------------------------------------------------------------------------- samples/sec: 1532.795 | iteration 20500/ 143000 | elapsed time per iteration (ms): 668.1 | learning rate: 5.762E-04 | approx flops per GPU: 59.5TFLOPS | lm_loss: 2.565540E+00 | loss scale: 16384.0 | number of skipped iterations: 2 | number of nan iterations: 0 | time (ms) | forward: 161.26 | backward: 453.91 | backward-backward: 453.88 | backward-allreduce: 0.00 | optimizer: 16.62 | batch generator: 3.38 samples/sec: 1617.688 | iteration 21000/ 143000 | elapsed time per iteration (ms): 633.0 | learning rate: 5.750E-04 | approx flops per GPU: 62.8TFLOPS | lm_loss: 2.560981E+00 | loss scale: 8192.0 | number of skipped iterations: 1 | number of nan iterations: 0 | time (ms) | forward: 161.40 | backward: 453.84 | backward-backward: 453.80 | backward-allreduce: 0.00 | optimizer: 16.64 | batch generator: 3.07 WARNING: Deleting old checkpoints: /u/wangh/workspace_ptmp/checkpoints_gptneox/flownet/160M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step17000 samples/sec: 1612.963 | iteration 21500/ 143000 | elapsed time per iteration (ms): 634.9 | learning rate: 5.737E-04 | approx flops per GPU: 62.6TFLOPS | lm_loss: 2.561658E+00 | loss scale: 8192.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 161.78 | backward: 453.73 | backward-backward: 453.69 | backward-allreduce: 0.00 | optimizer: 16.83 | batch generator: 3.45 samples/sec: 1617.041 | iteration 22000/ 143000 | elapsed time per iteration (ms): 633.3 | learning rate: 5.724E-04 | approx flops per GPU: 62.8TFLOPS | lm_loss: 2.559618E+00 | loss scale: 16384.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 161.73 | backward: 453.67 | backward-backward: 453.63 | backward-allreduce: 0.00 | optimizer: 16.74 | batch generator: 3.33 WARNING: Deleting old checkpoints: /u/wangh/workspace_ptmp/checkpoints_gptneox/flownet/160M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step18000 samples/sec: 1613.623 | iteration 22500/ 143000 | elapsed time per iteration (ms): 634.6 | learning rate: 5.711E-04 | approx flops per GPU: 62.6TFLOPS | lm_loss: 2.558299E+00 | loss scale: 16384.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 161.50 | backward: 453.81 | backward-backward: 453.77 | backward-allreduce: 0.00 | optimizer: 16.69 | batch generator: 3.14 samples/sec: 1618.024 | iteration 23000/ 143000 | elapsed time per iteration (ms): 632.9 | learning rate: 5.697E-04 | approx flops per GPU: 62.8TFLOPS | lm_loss: 2.555597E+00 | loss scale: 32768.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 160.80 | backward: 454.27 | backward-backward: 454.24 | backward-allreduce: 0.00 | optimizer: 16.66 | batch generator: 2.39 WARNING: Deleting old checkpoints: /u/wangh/workspace_ptmp/checkpoints_gptneox/flownet/160M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step19000 samples/sec: 1613.642 | iteration 23500/ 143000 | elapsed time per iteration (ms): 634.6 | learning rate: 5.683E-04 | approx flops per GPU: 62.6TFLOPS | lm_loss: 2.553966E+00 | loss scale: 32768.0 | number of skipped iterations: 1 | number of nan iterations: 0 | time (ms) | forward: 160.89 | backward: 454.51 | backward-backward: 454.48 | backward-allreduce: 0.00 | optimizer: 16.65 | batch generator: 2.49 samples/sec: 1617.713 | iteration 24000/ 143000 | elapsed time per iteration (ms): 633.0 | learning rate: 5.669E-04 | approx flops per GPU: 62.8TFLOPS | lm_loss: 2.554384E+00 | loss scale: 16384.0 | number of skipped iterations: 1 | number of nan iterations: 0 | time (ms) | forward: 161.56 | backward: 453.73 | backward-backward: 453.69 | backward-allreduce: 0.00 | optimizer: 16.59 | batch generator: 3.02 WARNING: Deleting old checkpoints: /u/wangh/workspace_ptmp/checkpoints_gptneox/flownet/160M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step20000 samples/sec: 1614.087 | iteration 24500/ 143000 | elapsed time per iteration (ms): 634.4 | learning rate: 5.654E-04 | approx flops per GPU: 62.7TFLOPS | lm_loss: 2.550193E+00 | loss scale: 16384.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 161.83 | backward: 453.56 | backward-backward: 453.52 | backward-allreduce: 0.00 | optimizer: 16.63 | batch generator: 3.24 samples/sec: 1616.813 | iteration 25000/ 143000 | elapsed time per iteration (ms): 633.3 | learning rate: 5.640E-04 | approx flops per GPU: 62.8TFLOPS | lm_loss: 2.549086E+00 | loss scale: 32768.0 | number of skipped iterations: 1 | number of nan iterations: 0 | time (ms) | forward: 161.46 | backward: 454.14 | backward-backward: 454.11 | backward-allreduce: 0.00 | optimizer: 16.63 | batch generator: 3.06 WARNING: Deleting old checkpoints: /u/wangh/workspace_ptmp/checkpoints_gptneox/flownet/160M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step21000 samples/sec: 1614.225 | iteration 25500/ 143000 | elapsed time per iteration (ms): 634.4 | learning rate: 5.625E-04 | approx flops per GPU: 62.7TFLOPS | lm_loss: 2.547747E+00 | loss scale: 16384.0 | number of skipped iterations: 1 | number of nan iterations: 0 | time (ms) | forward: 161.28 | backward: 454.03 | backward-backward: 453.99 | backward-allreduce: 0.00 | optimizer: 16.59 | batch generator: 2.66 samples/sec: 1617.511 | iteration 26000/ 143000 | elapsed time per iteration (ms): 633.1 | learning rate: 5.609E-04 | approx flops per GPU: 62.8TFLOPS | lm_loss: 2.547262E+00 | loss scale: 16384.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 161.09 | backward: 454.22 | backward-backward: 454.18 | backward-allreduce: 0.00 | optimizer: 16.64 | batch generator: 2.55 WARNING: Deleting old checkpoints: /u/wangh/workspace_ptmp/checkpoints_gptneox/flownet/160M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step22000 samples/sec: 1613.840 | iteration 26500/ 143000 | elapsed time per iteration (ms): 634.5 | learning rate: 5.594E-04 | approx flops per GPU: 62.7TFLOPS | lm_loss: 2.544339E+00 | loss scale: 32768.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 161.40 | backward: 454.06 | backward-backward: 454.03 | backward-allreduce: 0.00 | optimizer: 16.66 | batch generator: 2.69 samples/sec: 1617.357 | iteration 27000/ 143000 | elapsed time per iteration (ms): 633.1 | learning rate: 5.578E-04 | approx flops per GPU: 62.8TFLOPS | lm_loss: 2.543471E+00 | loss scale: 16384.0 | number of skipped iterations: 2 | number of nan iterations: 0 | time (ms) | forward: 161.64 | backward: 453.76 | backward-backward: 453.73 | backward-allreduce: 0.00 | optimizer: 16.62 | batch generator: 3.07 WARNING: Deleting old checkpoints: /u/wangh/workspace_ptmp/checkpoints_gptneox/flownet/160M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step23000 samples/sec: 1614.475 | iteration 27500/ 143000 | elapsed time per iteration (ms): 634.3 | learning rate: 5.561E-04 | approx flops per GPU: 62.7TFLOPS | lm_loss: 2.541778E+00 | loss scale: 8192.0 | number of skipped iterations: 1 | number of nan iterations: 0 | time (ms) | forward: 160.83 | backward: 454.52 | backward-backward: 454.48 | backward-allreduce: 0.00 | optimizer: 16.55 | batch generator: 2.53 samples/sec: 1617.552 | iteration 28000/ 143000 | elapsed time per iteration (ms): 633.1 | learning rate: 5.545E-04 | approx flops per GPU: 62.8TFLOPS | lm_loss: 2.540524E+00 | loss scale: 8192.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 161.00 | backward: 454.35 | backward-backward: 454.32 | backward-allreduce: 0.00 | optimizer: 16.62 | batch generator: 2.60 WARNING: Deleting old checkpoints: /u/wangh/workspace_ptmp/checkpoints_gptneox/flownet/160M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step24000 samples/sec: 1614.106 | iteration 28500/ 143000 | elapsed time per iteration (ms): 634.4 | learning rate: 5.528E-04 | approx flops per GPU: 62.7TFLOPS | lm_loss: 2.539535E+00 | loss scale: 16384.0 | number of skipped iterations: 1 | number of nan iterations: 0 | time (ms) | forward: 162.02 | backward: 453.40 | backward-backward: 453.37 | backward-allreduce: 0.00 | optimizer: 16.55 | batch generator: 3.62 samples/sec: 1617.591 | iteration 29000/ 143000 | elapsed time per iteration (ms): 633.0 | learning rate: 5.511E-04 | approx flops per GPU: 62.8TFLOPS | lm_loss: 2.537093E+00 | loss scale: 16384.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 161.86 | backward: 453.46 | backward-backward: 453.43 | backward-allreduce: 0.00 | optimizer: 16.59 | batch generator: 3.40 WARNING: Deleting old checkpoints: /u/wangh/workspace_ptmp/checkpoints_gptneox/flownet/160M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step25000 samples/sec: 1614.021 | iteration 29500/ 143000 | elapsed time per iteration (ms): 634.4 | learning rate: 5.494E-04 | approx flops per GPU: 62.7TFLOPS | lm_loss: 2.537553E+00 | loss scale: 32768.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 162.02 | backward: 453.39 | backward-backward: 453.35 | backward-allreduce: 0.00 | optimizer: 16.62 | batch generator: 3.54 samples/sec: 1617.923 | iteration 30000/ 143000 | elapsed time per iteration (ms): 632.9 | learning rate: 5.476E-04 | approx flops per GPU: 62.8TFLOPS | lm_loss: 2.534826E+00 | loss scale: 16384.0 | number of skipped iterations: 2 | number of nan iterations: 0 | time (ms) | forward: 161.11 | backward: 454.11 | backward-backward: 454.07 | backward-allreduce: 0.00 | optimizer: 16.59 | batch generator: 2.60 WARNING: Deleting old checkpoints: /u/wangh/workspace_ptmp/checkpoints_gptneox/flownet/160M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step26000 ---------------------------------------------------------------------------------------------------------- validation results at iteration 30000 | lm_loss value: 2.480956E+00 | lm_loss_ppl value: 1.195268E+01 | ---------------------------------------------------------------------------------------------------------- samples/sec: 1532.027 | iteration 30500/ 143000 | elapsed time per iteration (ms): 668.4 | learning rate: 5.458E-04 | approx flops per GPU: 59.5TFLOPS | lm_loss: 2.533670E+00 | loss scale: 16384.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 162.24 | backward: 453.18 | backward-backward: 453.14 | backward-allreduce: 0.00 | optimizer: 16.62 | batch generator: 4.52 samples/sec: 1617.721 | iteration 31000/ 143000 | elapsed time per iteration (ms): 633.0 | learning rate: 5.440E-04 | approx flops per GPU: 62.8TFLOPS | lm_loss: 2.533513E+00 | loss scale: 8192.0 | number of skipped iterations: 1 | number of nan iterations: 0 | time (ms) | forward: 161.91 | backward: 453.43 | backward-backward: 453.39 | backward-allreduce: 0.00 | optimizer: 16.55 | batch generator: 3.44 WARNING: Deleting old checkpoints: /u/wangh/workspace_ptmp/checkpoints_gptneox/flownet/160M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step27000 samples/sec: 1613.042 | iteration 31500/ 143000 | elapsed time per iteration (ms): 634.8 | learning rate: 5.422E-04 | approx flops per GPU: 62.6TFLOPS | lm_loss: 2.530172E+00 | loss scale: 8192.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 161.18 | backward: 454.18 | backward-backward: 454.15 | backward-allreduce: 0.00 | optimizer: 16.64 | batch generator: 2.69 samples/sec: 1617.744 | iteration 32000/ 143000 | elapsed time per iteration (ms): 633.0 | learning rate: 5.403E-04 | approx flops per GPU: 62.8TFLOPS | lm_loss: 2.530236E+00 | loss scale: 16384.0 | number of skipped iterations: 1 | number of nan iterations: 0 | time (ms) | forward: 160.98 | backward: 454.35 | backward-backward: 454.31 | backward-allreduce: 0.00 | optimizer: 16.58 | batch generator: 2.58 WARNING: Deleting old checkpoints: /u/wangh/workspace_ptmp/checkpoints_gptneox/flownet/160M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step28000 samples/sec: 1612.243 | iteration 32500/ 143000 | elapsed time per iteration (ms): 635.1 | learning rate: 5.384E-04 | approx flops per GPU: 62.6TFLOPS | lm_loss: 2.528818E+00 | loss scale: 16384.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 161.10 | backward: 454.28 | backward-backward: 454.24 | backward-allreduce: 0.00 | optimizer: 16.60 | batch generator: 2.66 samples/sec: 1616.711 | iteration 33000/ 143000 | elapsed time per iteration (ms): 633.4 | learning rate: 5.365E-04 | approx flops per GPU: 62.8TFLOPS | lm_loss: 2.529440E+00 | loss scale: 8192.0 | number of skipped iterations: 1 | number of nan iterations: 0 | time (ms) | forward: 161.25 | backward: 454.30 | backward-backward: 454.27 | backward-allreduce: 0.00 | optimizer: 16.77 | batch generator: 2.69 WARNING: Deleting old checkpoints: /u/wangh/workspace_ptmp/checkpoints_gptneox/flownet/160M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step29000 samples/sec: 1612.865 | iteration 33500/ 143000 | elapsed time per iteration (ms): 634.9 | learning rate: 5.346E-04 | approx flops per GPU: 62.6TFLOPS | lm_loss: 2.527193E+00 | loss scale: 4096.0 | number of skipped iterations: 1 | number of nan iterations: 0 | time (ms) | forward: 161.84 | backward: 453.78 | backward-backward: 453.75 | backward-allreduce: 0.00 | optimizer: 16.63 | batch generator: 3.32 samples/sec: 1616.456 | iteration 34000/ 143000 | elapsed time per iteration (ms): 633.5 | learning rate: 5.326E-04 | approx flops per GPU: 62.8TFLOPS | lm_loss: 2.528305E+00 | loss scale: 2048.0 | number of skipped iterations: 1 | number of nan iterations: 0 | time (ms) | forward: 161.97 | backward: 453.81 | backward-backward: 453.77 | backward-allreduce: 0.00 | optimizer: 16.67 | batch generator: 3.43 WARNING: Deleting old checkpoints: /u/wangh/workspace_ptmp/checkpoints_gptneox/flownet/160M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step30000 samples/sec: 1612.692 | iteration 34500/ 143000 | elapsed time per iteration (ms): 635.0 | learning rate: 5.306E-04 | approx flops per GPU: 62.6TFLOPS | lm_loss: 2.525521E+00 | loss scale: 1024.0 | number of skipped iterations: 1 | number of nan iterations: 0 | time (ms) | forward: 161.37 | backward: 454.48 | backward-backward: 454.44 | backward-allreduce: 0.00 | optimizer: 16.66 | batch generator: 2.90 samples/sec: 1616.571 | iteration 35000/ 143000 | elapsed time per iteration (ms): 633.4 | learning rate: 5.286E-04 | approx flops per GPU: 62.8TFLOPS | lm_loss: 2.524822E+00 | loss scale: 1024.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 161.00 | backward: 454.69 | backward-backward: 454.65 | backward-allreduce: 0.00 | optimizer: 16.67 | batch generator: 2.44 WARNING: Deleting old checkpoints: /u/wangh/workspace_ptmp/checkpoints_gptneox/flownet/160M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step31000 samples/sec: 1613.348 | iteration 35500/ 143000 | elapsed time per iteration (ms): 634.7 | learning rate: 5.266E-04 | approx flops per GPU: 62.6TFLOPS | lm_loss: 2.524562E+00 | loss scale: 2048.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 161.44 | backward: 454.16 | backward-backward: 454.13 | backward-allreduce: 0.00 | optimizer: 16.61 | batch generator: 2.92 samples/sec: 1617.107 | iteration 36000/ 143000 | elapsed time per iteration (ms): 633.2 | learning rate: 5.245E-04 | approx flops per GPU: 62.8TFLOPS | lm_loss: 2.522522E+00 | loss scale: 2048.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 161.21 | backward: 454.28 | backward-backward: 454.25 | backward-allreduce: 0.00 | optimizer: 16.66 | batch generator: 2.72 WARNING: Deleting old checkpoints: /u/wangh/workspace_ptmp/checkpoints_gptneox/flownet/160M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step32000 samples/sec: 1613.409 | iteration 36500/ 143000 | elapsed time per iteration (ms): 634.7 | learning rate: 5.224E-04 | approx flops per GPU: 62.6TFLOPS | lm_loss: 2.521075E+00 | loss scale: 4096.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 162.02 | backward: 453.55 | backward-backward: 453.51 | backward-allreduce: 0.00 | optimizer: 16.66 | batch generator: 3.59 samples/sec: 1616.513 | iteration 37000/ 143000 | elapsed time per iteration (ms): 633.5 | learning rate: 5.203E-04 | approx flops per GPU: 62.8TFLOPS | lm_loss: 2.520790E+00 | loss scale: 4096.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 163.85 | backward: 452.09 | backward-backward: 452.06 | backward-allreduce: 0.00 | optimizer: 16.61 | batch generator: 5.26 WARNING: Deleting old checkpoints: /u/wangh/workspace_ptmp/checkpoints_gptneox/flownet/160M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step33000 samples/sec: 1611.761 | iteration 37500/ 143000 | elapsed time per iteration (ms): 635.3 | learning rate: 5.182E-04 | approx flops per GPU: 62.6TFLOPS | lm_loss: 2.520272E+00 | loss scale: 4096.0 | number of skipped iterations: 1 | number of nan iterations: 0 | time (ms) | forward: 163.48 | backward: 452.44 | backward-backward: 452.41 | backward-allreduce: 0.00 | optimizer: 16.55 | batch generator: 5.06 samples/sec: 1616.550 | iteration 38000/ 143000 | elapsed time per iteration (ms): 633.4 | learning rate: 5.160E-04 | approx flops per GPU: 62.8TFLOPS | lm_loss: 2.520059E+00 | loss scale: 4096.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 163.65 | backward: 452.30 | backward-backward: 452.26 | backward-allreduce: 0.00 | optimizer: 16.59 | batch generator: 5.36 WARNING: Deleting old checkpoints: /u/wangh/workspace_ptmp/checkpoints_gptneox/flownet/160M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step34000 samples/sec: 1613.106 | iteration 38500/ 143000 | elapsed time per iteration (ms): 634.8 | learning rate: 5.138E-04 | approx flops per GPU: 62.6TFLOPS | lm_loss: 2.519216E+00 | loss scale: 8192.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 163.46 | backward: 452.43 | backward-backward: 452.40 | backward-allreduce: 0.00 | optimizer: 16.58 | batch generator: 5.18 samples/sec: 1616.672 | iteration 39000/ 143000 | elapsed time per iteration (ms): 633.4 | learning rate: 5.116E-04 | approx flops per GPU: 62.8TFLOPS | lm_loss: 2.520004E+00 | loss scale: 8192.0 | number of skipped iterations: 1 | number of nan iterations: 0 | time (ms) | forward: 163.27 | backward: 452.64 | backward-backward: 452.61 | backward-allreduce: 0.00 | optimizer: 16.56 | batch generator: 5.05 WARNING: Deleting old checkpoints: /u/wangh/workspace_ptmp/checkpoints_gptneox/flownet/160M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step35000 samples/sec: 1612.908 | iteration 39500/ 143000 | elapsed time per iteration (ms): 634.9 | learning rate: 5.094E-04 | approx flops per GPU: 62.6TFLOPS | lm_loss: 2.516927E+00 | loss scale: 8192.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 163.72 | backward: 452.10 | backward-backward: 452.07 | backward-allreduce: 0.00 | optimizer: 16.57 | batch generator: 5.40 samples/sec: 1616.686 | iteration 40000/ 143000 | elapsed time per iteration (ms): 633.4 | learning rate: 5.072E-04 | approx flops per GPU: 62.8TFLOPS | lm_loss: 2.515665E+00 | loss scale: 8192.0 | number of skipped iterations: 2 | number of nan iterations: 0 | time (ms) | forward: 163.74 | backward: 452.01 | backward-backward: 451.97 | backward-allreduce: 0.00 | optimizer: 16.58 | batch generator: 5.54 WARNING: Deleting old checkpoints: /u/wangh/workspace_ptmp/checkpoints_gptneox/flownet/160M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step36000 ---------------------------------------------------------------------------------------------------------- validation results at iteration 40000 | lm_loss value: 2.460585E+00 | lm_loss_ppl value: 1.171166E+01 | ---------------------------------------------------------------------------------------------------------- samples/sec: 1530.815 | iteration 40500/ 143000 | elapsed time per iteration (ms): 668.9 | learning rate: 5.049E-04 | approx flops per GPU: 59.4TFLOPS | lm_loss: 2.514085E+00 | loss scale: 8192.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 164.08 | backward: 451.73 | backward-backward: 451.70 | backward-allreduce: 0.00 | optimizer: 16.63 | batch generator: 6.74 samples/sec: 1616.541 | iteration 41000/ 143000 | elapsed time per iteration (ms): 633.5 | learning rate: 5.026E-04 | approx flops per GPU: 62.8TFLOPS | lm_loss: 2.514071E+00 | loss scale: 16384.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 163.73 | backward: 452.15 | backward-backward: 452.12 | backward-allreduce: 0.00 | optimizer: 16.63 | batch generator: 5.51 WARNING: Deleting old checkpoints: /u/wangh/workspace_ptmp/checkpoints_gptneox/flownet/160M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step37000 samples/sec: 1614.340 | iteration 41500/ 143000 | elapsed time per iteration (ms): 634.3 | learning rate: 5.003E-04 | approx flops per GPU: 62.7TFLOPS | lm_loss: 2.513219E+00 | loss scale: 16384.0 | number of skipped iterations: 1 | number of nan iterations: 0 | time (ms) | forward: 163.25 | backward: 452.25 | backward-backward: 452.21 | backward-allreduce: 0.00 | optimizer: 16.55 | batch generator: 5.06 samples/sec: 1616.956 | iteration 42000/ 143000 | elapsed time per iteration (ms): 633.3 | learning rate: 4.980E-04 | approx flops per GPU: 62.8TFLOPS | lm_loss: 2.511018E+00 | loss scale: 16384.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 163.24 | backward: 452.42 | backward-backward: 452.39 | backward-allreduce: 0.00 | optimizer: 16.63 | batch generator: 5.06 WARNING: Deleting old checkpoints: /u/wangh/workspace_ptmp/checkpoints_gptneox/flownet/160M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step38000 samples/sec: 1613.413 | iteration 42500/ 143000 | elapsed time per iteration (ms): 634.7 | learning rate: 4.956E-04 | approx flops per GPU: 62.6TFLOPS | lm_loss: 2.512267E+00 | loss scale: 4096.0 | number of skipped iterations: 4 | number of nan iterations: 0 | time (ms) | forward: 163.65 | backward: 452.25 | backward-backward: 452.22 | backward-allreduce: 0.00 | optimizer: 16.52 | batch generator: 5.37 samples/sec: 1616.662 | iteration 43000/ 143000 | elapsed time per iteration (ms): 633.4 | learning rate: 4.932E-04 | approx flops per GPU: 62.8TFLOPS | lm_loss: 2.510128E+00 | loss scale: 4096.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 163.44 | backward: 452.44 | backward-backward: 452.40 | backward-allreduce: 0.00 | optimizer: 16.56 | batch generator: 5.18 WARNING: Deleting old checkpoints: /u/wangh/workspace_ptmp/checkpoints_gptneox/flownet/160M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step39000 samples/sec: 1613.725 | iteration 43500/ 143000 | elapsed time per iteration (ms): 634.6 | learning rate: 4.908E-04 | approx flops per GPU: 62.7TFLOPS | lm_loss: 2.510455E+00 | loss scale: 8192.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 163.20 | backward: 452.61 | backward-backward: 452.58 | backward-allreduce: 0.00 | optimizer: 16.63 | batch generator: 4.99 samples/sec: 1617.077 | iteration 44000/ 143000 | elapsed time per iteration (ms): 633.2 | learning rate: 4.884E-04 | approx flops per GPU: 62.8TFLOPS | lm_loss: 2.507397E+00 | loss scale: 8192.0 | number of skipped iterations: 1 | number of nan iterations: 0 | time (ms) | forward: 163.38 | backward: 452.35 | backward-backward: 452.31 | backward-allreduce: 0.00 | optimizer: 16.60 | batch generator: 5.13 WARNING: Deleting old checkpoints: /u/wangh/workspace_ptmp/checkpoints_gptneox/flownet/160M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step40000 samples/sec: 1612.407 | iteration 44500/ 143000 | elapsed time per iteration (ms): 635.1 | learning rate: 4.860E-04 | approx flops per GPU: 62.6TFLOPS | lm_loss: 2.507616E+00 | loss scale: 4096.0 | number of skipped iterations: 1 | number of nan iterations: 0 | time (ms) | forward: 163.46 | backward: 452.35 | backward-backward: 452.31 | backward-allreduce: 0.00 | optimizer: 16.53 | batch generator: 5.16 samples/sec: 1617.221 | iteration 45000/ 143000 | elapsed time per iteration (ms): 633.2 | learning rate: 4.836E-04 | approx flops per GPU: 62.8TFLOPS | lm_loss: 2.507346E+00 | loss scale: 4096.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 163.29 | backward: 452.36 | backward-backward: 452.32 | backward-allreduce: 0.00 | optimizer: 16.56 | batch generator: 5.03 WARNING: Deleting old checkpoints: /u/wangh/workspace_ptmp/checkpoints_gptneox/flownet/160M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step41000 samples/sec: 1613.918 | iteration 45500/ 143000 | elapsed time per iteration (ms): 634.5 | learning rate: 4.811E-04 | approx flops per GPU: 62.7TFLOPS | lm_loss: 2.506265E+00 | loss scale: 8192.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 163.85 | backward: 451.79 | backward-backward: 451.75 | backward-allreduce: 0.00 | optimizer: 16.56 | batch generator: 5.66 samples/sec: 1616.964 | iteration 46000/ 143000 | elapsed time per iteration (ms): 633.3 | learning rate: 4.786E-04 | approx flops per GPU: 62.8TFLOPS | lm_loss: 2.505252E+00 | loss scale: 4096.0 | number of skipped iterations: 2 | number of nan iterations: 0 | time (ms) | forward: 163.30 | backward: 452.41 | backward-backward: 452.38 | backward-allreduce: 0.00 | optimizer: 16.59 | batch generator: 4.99 WARNING: Deleting old checkpoints: /u/wangh/workspace_ptmp/checkpoints_gptneox/flownet/160M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step42000 samples/sec: 1613.093 | iteration 46500/ 143000 | elapsed time per iteration (ms): 634.8 | learning rate: 4.761E-04 | approx flops per GPU: 62.6TFLOPS | lm_loss: 2.504201E+00 | loss scale: 4096.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 162.96 | backward: 452.84 | backward-backward: 452.80 | backward-allreduce: 0.00 | optimizer: 16.56 | batch generator: 4.72 samples/sec: 1616.647 | iteration 47000/ 143000 | elapsed time per iteration (ms): 633.4 | learning rate: 4.736E-04 | approx flops per GPU: 62.8TFLOPS | lm_loss: 2.502829E+00 | loss scale: 8192.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 162.64 | backward: 453.28 | backward-backward: 453.25 | backward-allreduce: 0.00 | optimizer: 16.56 | batch generator: 4.40 WARNING: Deleting old checkpoints: /u/wangh/workspace_ptmp/checkpoints_gptneox/flownet/160M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step43000 samples/sec: 1613.215 | iteration 47500/ 143000 | elapsed time per iteration (ms): 634.8 | learning rate: 4.710E-04 | approx flops per GPU: 62.6TFLOPS | lm_loss: 2.503611E+00 | loss scale: 8192.0 | number of skipped iterations: 1 | number of nan iterations: 0 | time (ms) | forward: 163.24 | backward: 452.61 | backward-backward: 452.58 | backward-allreduce: 0.00 | optimizer: 16.54 | batch generator: 4.93 samples/sec: 1616.717 | iteration 48000/ 143000 | elapsed time per iteration (ms): 633.4 | learning rate: 4.685E-04 | approx flops per GPU: 62.8TFLOPS | lm_loss: 2.502625E+00 | loss scale: 8192.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 162.93 | backward: 452.85 | backward-backward: 452.82 | backward-allreduce: 0.00 | optimizer: 16.55 | batch generator: 4.67 WARNING: Deleting old checkpoints: /u/wangh/workspace_ptmp/checkpoints_gptneox/flownet/160M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step44000 samples/sec: 1613.280 | iteration 48500/ 143000 | elapsed time per iteration (ms): 634.7 | learning rate: 4.659E-04 | approx flops per GPU: 62.6TFLOPS | lm_loss: 2.501553E+00 | loss scale: 16384.0 | number of skipped iterations: 1 | number of nan iterations: 0 | time (ms) | forward: 163.78 | backward: 452.08 | backward-backward: 452.04 | backward-allreduce: 0.00 | optimizer: 16.51 | batch generator: 5.46 samples/sec: 1616.870 | iteration 49000/ 143000 | elapsed time per iteration (ms): 633.3 | learning rate: 4.633E-04 | approx flops per GPU: 62.8TFLOPS | lm_loss: 2.501582E+00 | loss scale: 8192.0 | number of skipped iterations: 1 | number of nan iterations: 0 | time (ms) | forward: 164.04 | backward: 451.89 | backward-backward: 451.86 | backward-allreduce: 0.00 | optimizer: 16.50 | batch generator: 5.63 WARNING: Deleting old checkpoints: /u/wangh/workspace_ptmp/checkpoints_gptneox/flownet/160M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step45000 samples/sec: 1612.540 | iteration 49500/ 143000 | elapsed time per iteration (ms): 635.0 | learning rate: 4.607E-04 | approx flops per GPU: 62.6TFLOPS | lm_loss: 2.501243E+00 | loss scale: 8192.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 163.32 | backward: 452.47 | backward-backward: 452.44 | backward-allreduce: 0.00 | optimizer: 16.55 | batch generator: 4.95 samples/sec: 1616.954 | iteration 50000/ 143000 | elapsed time per iteration (ms): 633.3 | learning rate: 4.581E-04 | approx flops per GPU: 62.8TFLOPS | lm_loss: 2.497984E+00 | loss scale: 8192.0 | number of skipped iterations: 2 | number of nan iterations: 0 | time (ms) | forward: 163.03 | backward: 452.79 | backward-backward: 452.76 | backward-allreduce: 0.00 | optimizer: 16.50 | batch generator: 4.75 WARNING: Deleting old checkpoints: /u/wangh/workspace_ptmp/checkpoints_gptneox/flownet/160M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step46000 ---------------------------------------------------------------------------------------------------------- validation results at iteration 50000 | lm_loss value: 2.450358E+00 | lm_loss_ppl value: 1.159250E+01 | ---------------------------------------------------------------------------------------------------------- samples/sec: 1531.157 | iteration 50500/ 143000 | elapsed time per iteration (ms): 668.8 | learning rate: 4.554E-04 | approx flops per GPU: 59.4TFLOPS | lm_loss: 2.499007E+00 | loss scale: 4096.0 | number of skipped iterations: 1 | number of nan iterations: 0 | time (ms) | forward: 163.78 | backward: 452.15 | backward-backward: 452.12 | backward-allreduce: 0.00 | optimizer: 16.53 | batch generator: 5.96 samples/sec: 1616.370 | iteration 51000/ 143000 | elapsed time per iteration (ms): 633.5 | learning rate: 4.528E-04 | approx flops per GPU: 62.8TFLOPS | lm_loss: 2.498292E+00 | loss scale: 4096.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 164.43 | backward: 451.63 | backward-backward: 451.59 | backward-allreduce: 0.00 | optimizer: 16.55 | batch generator: 5.85 WARNING: Deleting old checkpoints: /u/wangh/workspace_ptmp/checkpoints_gptneox/flownet/160M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step47000 samples/sec: 1612.946 | iteration 51500/ 143000 | elapsed time per iteration (ms): 634.9 | learning rate: 4.501E-04 | approx flops per GPU: 62.6TFLOPS | lm_loss: 2.496488E+00 | loss scale: 8192.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 164.12 | backward: 451.89 | backward-backward: 451.85 | backward-allreduce: 0.00 | optimizer: 16.55 | batch generator: 5.74 samples/sec: 1616.777 | iteration 52000/ 143000 | elapsed time per iteration (ms): 633.4 | learning rate: 4.474E-04 | approx flops per GPU: 62.8TFLOPS | lm_loss: 2.495720E+00 | loss scale: 8192.0 | number of skipped iterations: 1 | number of nan iterations: 0 | time (ms) | forward: 163.53 | backward: 452.33 | backward-backward: 452.30 | backward-allreduce: 0.00 | optimizer: 16.52 | batch generator: 5.21 WARNING: Deleting old checkpoints: /u/wangh/workspace_ptmp/checkpoints_gptneox/flownet/160M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step48000 samples/sec: 1612.787 | iteration 52500/ 143000 | elapsed time per iteration (ms): 634.9 | learning rate: 4.447E-04 | approx flops per GPU: 62.6TFLOPS | lm_loss: 2.497996E+00 | loss scale: 4096.0 | number of skipped iterations: 1 | number of nan iterations: 0 | time (ms) | forward: 163.42 | backward: 452.29 | backward-backward: 452.25 | backward-allreduce: 0.00 | optimizer: 16.53 | batch generator: 5.13 samples/sec: 1616.612 | iteration 53000/ 143000 | elapsed time per iteration (ms): 633.4 | learning rate: 4.420E-04 | approx flops per GPU: 62.8TFLOPS | lm_loss: 2.495331E+00 | loss scale: 2048.0 | number of skipped iterations: 1 | number of nan iterations: 0 | time (ms) | forward: 163.56 | backward: 452.42 | backward-backward: 452.39 | backward-allreduce: 0.00 | optimizer: 16.53 | batch generator: 5.30 WARNING: Deleting old checkpoints: /u/wangh/workspace_ptmp/checkpoints_gptneox/flownet/160M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step49000 samples/sec: 1612.588 | iteration 53500/ 143000 | elapsed time per iteration (ms): 635.0 | learning rate: 4.392E-04 | approx flops per GPU: 62.6TFLOPS | lm_loss: 2.495067E+00 | loss scale: 2048.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 163.76 | backward: 452.33 | backward-backward: 452.30 | backward-allreduce: 0.00 | optimizer: 16.60 | batch generator: 5.46 samples/sec: 1616.432 | iteration 54000/ 143000 | elapsed time per iteration (ms): 633.5 | learning rate: 4.365E-04 | approx flops per GPU: 62.8TFLOPS | lm_loss: 2.494098E+00 | loss scale: 4096.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 163.68 | backward: 452.36 | backward-backward: 452.33 | backward-allreduce: 0.00 | optimizer: 16.57 | batch generator: 5.31 WARNING: Deleting old checkpoints: /u/wangh/workspace_ptmp/checkpoints_gptneox/flownet/160M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step50000 samples/sec: 1612.793 | iteration 54500/ 143000 | elapsed time per iteration (ms): 634.9 | learning rate: 4.337E-04 | approx flops per GPU: 62.6TFLOPS | lm_loss: 2.493621E+00 | loss scale: 4096.0 | number of skipped iterations: 1 | number of nan iterations: 0 | time (ms) | forward: 164.16 | backward: 451.79 | backward-backward: 451.76 | backward-allreduce: 0.00 | optimizer: 16.56 | batch generator: 5.79 samples/sec: 1617.026 | iteration 55000/ 143000 | elapsed time per iteration (ms): 633.3 | learning rate: 4.310E-04 | approx flops per GPU: 62.8TFLOPS | lm_loss: 2.492211E+00 | loss scale: 4096.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 164.22 | backward: 451.62 | backward-backward: 451.58 | backward-allreduce: 0.00 | optimizer: 16.54 | batch generator: 5.93 WARNING: Deleting old checkpoints: /u/wangh/workspace_ptmp/checkpoints_gptneox/flownet/160M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step51000 samples/sec: 1613.197 | iteration 55500/ 143000 | elapsed time per iteration (ms): 634.8 | learning rate: 4.282E-04 | approx flops per GPU: 62.6TFLOPS | lm_loss: 2.492571E+00 | loss scale: 8192.0 | number of skipped iterations: 1 | number of nan iterations: 0 | time (ms) | forward: 164.14 | backward: 451.77 | backward-backward: 451.74 | backward-allreduce: 0.00 | optimizer: 16.54 | batch generator: 5.86 samples/sec: 1616.619 | iteration 56000/ 143000 | elapsed time per iteration (ms): 633.4 | learning rate: 4.254E-04 | approx flops per GPU: 62.8TFLOPS | lm_loss: 2.490403E+00 | loss scale: 4096.0 | number of skipped iterations: 1 | number of nan iterations: 0 | time (ms) | forward: 162.82 | backward: 453.01 | backward-backward: 452.98 | backward-allreduce: 0.00 | optimizer: 16.56 | batch generator: 4.51 WARNING: Deleting old checkpoints: /u/wangh/workspace_ptmp/checkpoints_gptneox/flownet/160M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step52000 samples/sec: 1613.531 | iteration 56500/ 143000 | elapsed time per iteration (ms): 634.6 | learning rate: 4.226E-04 | approx flops per GPU: 62.6TFLOPS | lm_loss: 2.490166E+00 | loss scale: 4096.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 162.12 | backward: 453.60 | backward-backward: 453.57 | backward-allreduce: 0.00 | optimizer: 16.57 | batch generator: 3.80 samples/sec: 1616.846 | iteration 57000/ 143000 | elapsed time per iteration (ms): 633.3 | learning rate: 4.198E-04 | approx flops per GPU: 62.8TFLOPS | lm_loss: 2.489195E+00 | loss scale: 8192.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 163.59 | backward: 452.19 | backward-backward: 452.15 | backward-allreduce: 0.00 | optimizer: 16.54 | batch generator: 5.15 WARNING: Deleting old checkpoints: /u/wangh/workspace_ptmp/checkpoints_gptneox/flownet/160M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step53000 samples/sec: 1614.338 | iteration 57500/ 143000 | elapsed time per iteration (ms): 634.3 | learning rate: 4.169E-04 | approx flops per GPU: 62.7TFLOPS | lm_loss: 2.489191E+00 | loss scale: 8192.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 163.36 | backward: 452.19 | backward-backward: 452.15 | backward-allreduce: 0.00 | optimizer: 16.54 | batch generator: 5.00 samples/sec: 1617.007 | iteration 58000/ 143000 | elapsed time per iteration (ms): 633.3 | learning rate: 4.141E-04 | approx flops per GPU: 62.8TFLOPS | lm_loss: 2.489769E+00 | loss scale: 16384.0 | number of skipped iterations: 1 | number of nan iterations: 0 | time (ms) | forward: 164.07 | backward: 451.75 | backward-backward: 451.71 | backward-allreduce: 0.00 | optimizer: 16.51 | batch generator: 5.72 WARNING: Deleting old checkpoints: /u/wangh/workspace_ptmp/checkpoints_gptneox/flownet/160M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step54000 samples/sec: 1613.258 | iteration 58500/ 143000 | elapsed time per iteration (ms): 634.7 | learning rate: 4.113E-04 | approx flops per GPU: 62.6TFLOPS | lm_loss: 2.484501E+00 | loss scale: 8192.0 | number of skipped iterations: 1 | number of nan iterations: 0 | time (ms) | forward: 164.19 | backward: 451.78 | backward-backward: 451.74 | backward-allreduce: 0.00 | optimizer: 16.55 | batch generator: 5.85 samples/sec: 1616.476 | iteration 59000/ 143000 | elapsed time per iteration (ms): 633.5 | learning rate: 4.084E-04 | approx flops per GPU: 62.8TFLOPS | lm_loss: 2.487577E+00 | loss scale: 8192.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 164.26 | backward: 451.68 | backward-backward: 451.65 | backward-allreduce: 0.00 | optimizer: 16.60 | batch generator: 5.80 WARNING: Deleting old checkpoints: /u/wangh/workspace_ptmp/checkpoints_gptneox/flownet/160M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step55000 samples/sec: 1566.041 | iteration 59500/ 143000 | elapsed time per iteration (ms): 653.9 | learning rate: 4.055E-04 | approx flops per GPU: 60.8TFLOPS | lm_loss: 2.485723E+00 | loss scale: 16384.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 164.27 | backward: 451.61 | backward-backward: 451.58 | backward-allreduce: 0.00 | optimizer: 16.61 | batch generator: 5.95 samples/sec: 1616.940 | iteration 60000/ 143000 | elapsed time per iteration (ms): 633.3 | learning rate: 4.027E-04 | approx flops per GPU: 62.8TFLOPS | lm_loss: 2.486457E+00 | loss scale: 4096.0 | number of skipped iterations: 3 | number of nan iterations: 0 | time (ms) | forward: 164.02 | backward: 451.81 | backward-backward: 451.78 | backward-allreduce: 0.00 | optimizer: 16.51 | batch generator: 5.68 WARNING: Deleting old checkpoints: /u/wangh/workspace_ptmp/checkpoints_gptneox/flownet/160M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step56000 ---------------------------------------------------------------------------------------------------------- validation results at iteration 60000 | lm_loss value: 2.434267E+00 | lm_loss_ppl value: 1.140745E+01 | ---------------------------------------------------------------------------------------------------------- samples/sec: 1531.284 | iteration 60500/ 143000 | elapsed time per iteration (ms): 668.7 | learning rate: 3.998E-04 | approx flops per GPU: 59.5TFLOPS | lm_loss: 2.484520E+00 | loss scale: 4096.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 163.98 | backward: 451.90 | backward-backward: 451.86 | backward-allreduce: 0.00 | optimizer: 16.59 | batch generator: 6.07 samples/sec: 1616.644 | iteration 61000/ 143000 | elapsed time per iteration (ms): 633.4 | learning rate: 3.969E-04 | approx flops per GPU: 62.8TFLOPS | lm_loss: 2.483901E+00 | loss scale: 8192.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 164.12 | backward: 451.81 | backward-backward: 451.78 | backward-allreduce: 0.00 | optimizer: 16.56 | batch generator: 5.78 WARNING: Deleting old checkpoints: /u/wangh/workspace_ptmp/checkpoints_gptneox/flownet/160M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step57000 samples/sec: 1613.692 | iteration 61500/ 143000 | elapsed time per iteration (ms): 634.6 | learning rate: 3.940E-04 | approx flops per GPU: 62.7TFLOPS | lm_loss: 2.485673E+00 | loss scale: 4096.0 | number of skipped iterations: 2 | number of nan iterations: 0 | time (ms) | forward: 163.60 | backward: 452.30 | backward-backward: 452.26 | backward-allreduce: 0.00 | optimizer: 16.51 | batch generator: 5.19 samples/sec: 1616.816 | iteration 62000/ 143000 | elapsed time per iteration (ms): 633.3 | learning rate: 3.911E-04 | approx flops per GPU: 62.8TFLOPS | lm_loss: 2.480963E+00 | loss scale: 4096.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 163.41 | backward: 452.45 | backward-backward: 452.41 | backward-allreduce: 0.00 | optimizer: 16.58 | batch generator: 5.18 WARNING: Deleting old checkpoints: /u/wangh/workspace_ptmp/checkpoints_gptneox/flownet/160M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step58000 samples/sec: 1612.951 | iteration 62500/ 143000 | elapsed time per iteration (ms): 634.9 | learning rate: 3.882E-04 | approx flops per GPU: 62.6TFLOPS | lm_loss: 2.483075E+00 | loss scale: 8192.0 | number of skipped iterations: 1 | number of nan iterations: 0 | time (ms) | forward: 163.34 | backward: 452.57 | backward-backward: 452.53 | backward-allreduce: 0.00 | optimizer: 16.54 | batch generator: 5.02 samples/sec: 1617.027 | iteration 63000/ 143000 | elapsed time per iteration (ms): 633.3 | learning rate: 3.852E-04 | approx flops per GPU: 62.8TFLOPS | lm_loss: 2.481791E+00 | loss scale: 2048.0 | number of skipped iterations: 2 | number of nan iterations: 0 | time (ms) | forward: 163.32 | backward: 452.53 | backward-backward: 452.50 | backward-allreduce: 0.00 | optimizer: 16.52 | batch generator: 5.02 WARNING: Deleting old checkpoints: /u/wangh/workspace_ptmp/checkpoints_gptneox/flownet/160M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step59000 samples/sec: 1613.169 | iteration 63500/ 143000 | elapsed time per iteration (ms): 634.8 | learning rate: 3.823E-04 | approx flops per GPU: 62.6TFLOPS | lm_loss: 2.487264E+00 | loss scale: 2048.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 163.56 | backward: 452.48 | backward-backward: 452.45 | backward-allreduce: 0.00 | optimizer: 16.54 | batch generator: 5.21 samples/sec: 1616.920 | iteration 64000/ 143000 | elapsed time per iteration (ms): 633.3 | learning rate: 3.794E-04 | approx flops per GPU: 62.8TFLOPS | lm_loss: 2.489367E+00 | loss scale: 4096.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 163.46 | backward: 452.33 | backward-backward: 452.29 | backward-allreduce: 0.00 | optimizer: 16.58 | batch generator: 5.13 WARNING: Deleting old checkpoints: /u/wangh/workspace_ptmp/checkpoints_gptneox/flownet/160M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step60000 samples/sec: 1613.175 | iteration 64500/ 143000 | elapsed time per iteration (ms): 634.8 | learning rate: 3.764E-04 | approx flops per GPU: 62.6TFLOPS | lm_loss: 2.485919E+00 | loss scale: 4096.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 163.52 | backward: 452.34 | backward-backward: 452.30 | backward-allreduce: 0.00 | optimizer: 16.58 | batch generator: 5.25 samples/sec: 1616.577 | iteration 65000/ 143000 | elapsed time per iteration (ms): 633.4 | learning rate: 3.735E-04 | approx flops per GPU: 62.8TFLOPS | lm_loss: 2.487288E+00 | loss scale: 8192.0 | number of skipped iterations: 1 | number of nan iterations: 0 | time (ms) | forward: 163.39 | backward: 452.58 | backward-backward: 452.54 | backward-allreduce: 0.00 | optimizer: 16.53 | batch generator: 5.02 WARNING: Deleting old checkpoints: /u/wangh/workspace_ptmp/checkpoints_gptneox/flownet/160M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step61000 samples/sec: 1610.723 | iteration 65500/ 143000 | elapsed time per iteration (ms): 635.7 | learning rate: 3.705E-04 | approx flops per GPU: 62.5TFLOPS | lm_loss: 2.485496E+00 | loss scale: 4096.0 | number of skipped iterations: 1 | number of nan iterations: 0 | time (ms) | forward: 164.04 | backward: 451.95 | backward-backward: 451.92 | backward-allreduce: 0.00 | optimizer: 16.52 | batch generator: 5.57 samples/sec: 1616.586 | iteration 66000/ 143000 | elapsed time per iteration (ms): 633.4 | learning rate: 3.675E-04 | approx flops per GPU: 62.8TFLOPS | lm_loss: 2.485154E+00 | loss scale: 4096.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 163.31 | backward: 452.62 | backward-backward: 452.58 | backward-allreduce: 0.00 | optimizer: 16.59 | batch generator: 4.84 WARNING: Deleting old checkpoints: /u/wangh/workspace_ptmp/checkpoints_gptneox/flownet/160M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step62000 samples/sec: 1613.467 | iteration 66500/ 143000 | elapsed time per iteration (ms): 634.7 | learning rate: 3.646E-04 | approx flops per GPU: 62.6TFLOPS | lm_loss: 2.487237E+00 | loss scale: 4096.0 | number of skipped iterations: 2 | number of nan iterations: 0 | time (ms) | forward: 163.16 | backward: 452.73 | backward-backward: 452.69 | backward-allreduce: 0.00 | optimizer: 16.51 | batch generator: 4.67 samples/sec: 1616.889 | iteration 67000/ 143000 | elapsed time per iteration (ms): 633.3 | learning rate: 3.616E-04 | approx flops per GPU: 62.8TFLOPS | lm_loss: 2.485617E+00 | loss scale: 4096.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 162.85 | backward: 452.91 | backward-backward: 452.88 | backward-allreduce: 0.00 | optimizer: 16.59 | batch generator: 4.54 WARNING: Deleting old checkpoints: /u/wangh/workspace_ptmp/checkpoints_gptneox/flownet/160M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step63000 samples/sec: 1612.832 | iteration 67500/ 143000 | elapsed time per iteration (ms): 634.9 | learning rate: 3.586E-04 | approx flops per GPU: 62.6TFLOPS | lm_loss: 2.484985E+00 | loss scale: 8192.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 163.60 | backward: 452.24 | backward-backward: 452.21 | backward-allreduce: 0.00 | optimizer: 16.60 | batch generator: 5.25 samples/sec: 1616.913 | iteration 68000/ 143000 | elapsed time per iteration (ms): 633.3 | learning rate: 3.557E-04 | approx flops per GPU: 62.8TFLOPS | lm_loss: 2.479742E+00 | loss scale: 2048.0 | number of skipped iterations: 3 | number of nan iterations: 0 | time (ms) | forward: 164.10 | backward: 451.76 | backward-backward: 451.72 | backward-allreduce: 0.00 | optimizer: 16.48 | batch generator: 5.72 WARNING: Deleting old checkpoints: /u/wangh/workspace_ptmp/checkpoints_gptneox/flownet/160M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step64000 samples/sec: 1613.221 | iteration 68500/ 143000 | elapsed time per iteration (ms): 634.8 | learning rate: 3.527E-04 | approx flops per GPU: 62.6TFLOPS | lm_loss: 2.482230E+00 | loss scale: 2048.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 163.77 | backward: 452.25 | backward-backward: 452.21 | backward-allreduce: 0.00 | optimizer: 16.57 | batch generator: 5.41 samples/sec: 1616.818 | iteration 69000/ 143000 | elapsed time per iteration (ms): 633.3 | learning rate: 3.497E-04 | approx flops per GPU: 62.8TFLOPS | lm_loss: 2.480977E+00 | loss scale: 4096.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 163.26 | backward: 452.57 | backward-backward: 452.54 | backward-allreduce: 0.00 | optimizer: 16.61 | batch generator: 4.87 WARNING: Deleting old checkpoints: /u/wangh/workspace_ptmp/checkpoints_gptneox/flownet/160M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step65000 samples/sec: 1612.827 | iteration 69500/ 143000 | elapsed time per iteration (ms): 634.9 | learning rate: 3.467E-04 | approx flops per GPU: 62.6TFLOPS | lm_loss: 2.480794E+00 | loss scale: 4096.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 163.34 | backward: 452.63 | backward-backward: 452.60 | backward-allreduce: 0.00 | optimizer: 16.60 | batch generator: 4.96 samples/sec: 1616.594 | iteration 70000/ 143000 | elapsed time per iteration (ms): 633.4 | learning rate: 3.437E-04 | approx flops per GPU: 62.8TFLOPS | lm_loss: 2.481825E+00 | loss scale: 8192.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 163.28 | backward: 452.59 | backward-backward: 452.55 | backward-allreduce: 0.00 | optimizer: 16.66 | batch generator: 4.93 WARNING: Deleting old checkpoints: /u/wangh/workspace_ptmp/checkpoints_gptneox/flownet/160M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step66000 ---------------------------------------------------------------------------------------------------------- validation results at iteration 70000 | lm_loss value: 2.426140E+00 | lm_loss_ppl value: 1.131512E+01 | ---------------------------------------------------------------------------------------------------------- samples/sec: 1530.854 | iteration 70500/ 143000 | elapsed time per iteration (ms): 668.9 | learning rate: 3.407E-04 | approx flops per GPU: 59.4TFLOPS | lm_loss: 2.479565E+00 | loss scale: 8192.0 | number of skipped iterations: 1 | number of nan iterations: 0 | time (ms) | forward: 163.84 | backward: 452.00 | backward-backward: 451.96 | backward-allreduce: 0.00 | optimizer: 16.61 | batch generator: 6.06 samples/sec: 1616.916 | iteration 71000/ 143000 | elapsed time per iteration (ms): 633.3 | learning rate: 3.377E-04 | approx flops per GPU: 62.8TFLOPS | lm_loss: 2.478135E+00 | loss scale: 8192.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 163.27 | backward: 452.54 | backward-backward: 452.50 | backward-allreduce: 0.00 | optimizer: 16.58 | batch generator: 4.96 WARNING: Deleting old checkpoints: /u/wangh/workspace_ptmp/checkpoints_gptneox/flownet/160M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step67000 samples/sec: 1613.858 | iteration 71500/ 143000 | elapsed time per iteration (ms): 634.5 | learning rate: 3.348E-04 | approx flops per GPU: 62.7TFLOPS | lm_loss: 2.476369E+00 | loss scale: 8192.0 | number of skipped iterations: 2 | number of nan iterations: 0 | time (ms) | forward: 163.50 | backward: 452.39 | backward-backward: 452.36 | backward-allreduce: 0.00 | optimizer: 16.51 | batch generator: 5.13 samples/sec: 1616.975 | iteration 72000/ 143000 | elapsed time per iteration (ms): 633.3 | learning rate: 3.318E-04 | approx flops per GPU: 62.8TFLOPS | lm_loss: 2.474593E+00 | loss scale: 8192.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 163.62 | backward: 452.17 | backward-backward: 452.14 | backward-allreduce: 0.00 | optimizer: 16.58 | batch generator: 5.29 WARNING: Deleting old checkpoints: /u/wangh/workspace_ptmp/checkpoints_gptneox/flownet/160M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step68000 samples/sec: 1613.361 | iteration 72500/ 143000 | elapsed time per iteration (ms): 634.7 | learning rate: 3.288E-04 | approx flops per GPU: 62.6TFLOPS | lm_loss: 2.472859E+00 | loss scale: 16384.0 | number of skipped iterations: 1 | number of nan iterations: 0 | time (ms) | forward: 163.33 | backward: 452.55 | backward-backward: 452.52 | backward-allreduce: 0.00 | optimizer: 16.58 | batch generator: 4.90 samples/sec: 1616.934 | iteration 73000/ 143000 | elapsed time per iteration (ms): 633.3 | learning rate: 3.258E-04 | approx flops per GPU: 62.8TFLOPS | lm_loss: 2.474193E+00 | loss scale: 8192.0 | number of skipped iterations: 1 | number of nan iterations: 0 | time (ms) | forward: 163.69 | backward: 452.13 | backward-backward: 452.10 | backward-allreduce: 0.00 | optimizer: 16.55 | batch generator: 5.23 WARNING: Deleting old checkpoints: /u/wangh/workspace_ptmp/checkpoints_gptneox/flownet/160M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step69000 samples/sec: 1613.601 | iteration 73500/ 143000 | elapsed time per iteration (ms): 634.6 | learning rate: 3.228E-04 | approx flops per GPU: 62.6TFLOPS | lm_loss: 2.474405E+00 | loss scale: 4096.0 | number of skipped iterations: 1 | number of nan iterations: 0 | time (ms) | forward: 162.87 | backward: 452.93 | backward-backward: 452.90 | backward-allreduce: 0.00 | optimizer: 16.59 | batch generator: 4.56 samples/sec: 1616.341 | iteration 74000/ 143000 | elapsed time per iteration (ms): 633.5 | learning rate: 3.198E-04 | approx flops per GPU: 62.8TFLOPS | lm_loss: 2.473199E+00 | loss scale: 4096.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 163.54 | backward: 452.44 | backward-backward: 452.40 | backward-allreduce: 0.00 | optimizer: 16.65 | batch generator: 5.27 WARNING: Deleting old checkpoints: /u/wangh/workspace_ptmp/checkpoints_gptneox/flownet/160M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step70000 samples/sec: 1613.683 | iteration 74500/ 143000 | elapsed time per iteration (ms): 634.6 | learning rate: 3.168E-04 | approx flops per GPU: 62.7TFLOPS | lm_loss: 2.472323E+00 | loss scale: 8192.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 163.53 | backward: 452.36 | backward-backward: 452.33 | backward-allreduce: 0.00 | optimizer: 16.57 | batch generator: 5.20 samples/sec: 1616.924 | iteration 75000/ 143000 | elapsed time per iteration (ms): 633.3 | learning rate: 3.138E-04 | approx flops per GPU: 62.8TFLOPS | lm_loss: 2.471198E+00 | loss scale: 8192.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 163.67 | backward: 452.15 | backward-backward: 452.11 | backward-allreduce: 0.00 | optimizer: 16.56 | batch generator: 5.21 WARNING: Deleting old checkpoints: /u/wangh/workspace_ptmp/checkpoints_gptneox/flownet/160M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step71000 samples/sec: 1613.440 | iteration 75500/ 143000 | elapsed time per iteration (ms): 634.7 | learning rate: 3.108E-04 | approx flops per GPU: 62.6TFLOPS | lm_loss: 2.468674E+00 | loss scale: 8192.0 | number of skipped iterations: 1 | number of nan iterations: 0 | time (ms) | forward: 163.75 | backward: 452.15 | backward-backward: 452.12 | backward-allreduce: 0.00 | optimizer: 16.59 | batch generator: 5.47 samples/sec: 1616.474 | iteration 76000/ 143000 | elapsed time per iteration (ms): 633.5 | learning rate: 3.078E-04 | approx flops per GPU: 62.8TFLOPS | lm_loss: 2.470168E+00 | loss scale: 8192.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 163.77 | backward: 452.22 | backward-backward: 452.19 | backward-allreduce: 0.00 | optimizer: 16.56 | batch generator: 5.45 WARNING: Deleting old checkpoints: /u/wangh/workspace_ptmp/checkpoints_gptneox/flownet/160M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step72000 samples/sec: 1593.556 | iteration 76500/ 143000 | elapsed time per iteration (ms): 642.6 | learning rate: 3.049E-04 | approx flops per GPU: 61.9TFLOPS | lm_loss: 2.469558E+00 | loss scale: 16384.0 | number of skipped iterations: 1 | number of nan iterations: 0 | time (ms) | forward: 163.58 | backward: 452.49 | backward-backward: 452.46 | backward-allreduce: 0.00 | optimizer: 16.55 | batch generator: 5.19 samples/sec: 1616.293 | iteration 77000/ 143000 | elapsed time per iteration (ms): 633.5 | learning rate: 3.019E-04 | approx flops per GPU: 62.8TFLOPS | lm_loss: 2.468547E+00 | loss scale: 16384.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 163.21 | backward: 452.81 | backward-backward: 452.78 | backward-allreduce: 0.00 | optimizer: 16.58 | batch generator: 4.82 WARNING: Deleting old checkpoints: /u/wangh/workspace_ptmp/checkpoints_gptneox/flownet/160M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step73000 samples/sec: 1613.575 | iteration 77500/ 143000 | elapsed time per iteration (ms): 634.6 | learning rate: 2.989E-04 | approx flops per GPU: 62.6TFLOPS | lm_loss: 2.466623E+00 | loss scale: 8192.0 | number of skipped iterations: 1 | number of nan iterations: 0 | time (ms) | forward: 163.54 | backward: 452.35 | backward-backward: 452.32 | backward-allreduce: 0.00 | optimizer: 16.54 | batch generator: 5.23 samples/sec: 1616.615 | iteration 78000/ 143000 | elapsed time per iteration (ms): 633.4 | learning rate: 2.959E-04 | approx flops per GPU: 62.8TFLOPS | lm_loss: 2.465625E+00 | loss scale: 8192.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 163.41 | backward: 452.52 | backward-backward: 452.48 | backward-allreduce: 0.00 | optimizer: 16.58 | batch generator: 5.02 WARNING: Deleting old checkpoints: /u/wangh/workspace_ptmp/checkpoints_gptneox/flownet/160M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step74000 samples/sec: 1613.886 | iteration 78500/ 143000 | elapsed time per iteration (ms): 634.5 | learning rate: 2.930E-04 | approx flops per GPU: 62.7TFLOPS | lm_loss: 2.465195E+00 | loss scale: 8192.0 | number of skipped iterations: 2 | number of nan iterations: 0 | time (ms) | forward: 163.34 | backward: 452.47 | backward-backward: 452.44 | backward-allreduce: 0.00 | optimizer: 16.52 | batch generator: 4.98 samples/sec: 1616.100 | iteration 79000/ 143000 | elapsed time per iteration (ms): 633.6 | learning rate: 2.900E-04 | approx flops per GPU: 62.7TFLOPS | lm_loss: 2.466726E+00 | loss scale: 2048.0 | number of skipped iterations: 2 | number of nan iterations: 0 | time (ms) | forward: 163.62 | backward: 452.59 | backward-backward: 452.55 | backward-allreduce: 0.00 | optimizer: 16.51 | batch generator: 5.23 WARNING: Deleting old checkpoints: /u/wangh/workspace_ptmp/checkpoints_gptneox/flownet/160M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step75000 samples/sec: 1612.406 | iteration 79500/ 143000 | elapsed time per iteration (ms): 635.1 | learning rate: 2.871E-04 | approx flops per GPU: 62.6TFLOPS | lm_loss: 2.468871E+00 | loss scale: 2048.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 163.68 | backward: 452.53 | backward-backward: 452.49 | backward-allreduce: 0.00 | optimizer: 16.61 | batch generator: 5.23 samples/sec: 1613.041 | iteration 80000/ 143000 | elapsed time per iteration (ms): 634.8 | learning rate: 2.841E-04 | approx flops per GPU: 62.6TFLOPS | lm_loss: 2.466308E+00 | loss scale: 4096.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 163.56 | backward: 452.31 | backward-backward: 452.27 | backward-allreduce: 0.00 | optimizer: 16.57 | batch generator: 5.10 WARNING: Deleting old checkpoints: /u/wangh/workspace_ptmp/checkpoints_gptneox/flownet/160M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step76000 ---------------------------------------------------------------------------------------------------------- validation results at iteration 80000 | lm_loss value: 2.415497E+00 | lm_loss_ppl value: 1.119533E+01 | ---------------------------------------------------------------------------------------------------------- samples/sec: 1529.787 | iteration 80500/ 143000 | elapsed time per iteration (ms): 669.4 | learning rate: 2.812E-04 | approx flops per GPU: 59.4TFLOPS | lm_loss: 2.466695E+00 | loss scale: 4096.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 163.47 | backward: 452.79 | backward-backward: 452.76 | backward-allreduce: 0.00 | optimizer: 16.65 | batch generator: 5.48 samples/sec: 1616.301 | iteration 81000/ 143000 | elapsed time per iteration (ms): 633.5 | learning rate: 2.782E-04 | approx flops per GPU: 62.8TFLOPS | lm_loss: 2.464310E+00 | loss scale: 8192.0 | number of skipped iterations: 1 | number of nan iterations: 0 | time (ms) | forward: 163.48 | backward: 452.56 | backward-backward: 452.53 | backward-allreduce: 0.00 | optimizer: 16.53 | batch generator: 5.27 WARNING: Deleting old checkpoints: /u/wangh/workspace_ptmp/checkpoints_gptneox/flownet/160M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step77000 samples/sec: 1612.826 | iteration 81500/ 143000 | elapsed time per iteration (ms): 634.9 | learning rate: 2.753E-04 | approx flops per GPU: 62.6TFLOPS | lm_loss: 2.464250E+00 | loss scale: 8192.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 164.22 | backward: 451.89 | backward-backward: 451.86 | backward-allreduce: 0.00 | optimizer: 16.58 | batch generator: 5.84 samples/sec: 1616.157 | iteration 82000/ 143000 | elapsed time per iteration (ms): 633.6 | learning rate: 2.724E-04 | approx flops per GPU: 62.7TFLOPS | lm_loss: 2.463235E+00 | loss scale: 16384.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 164.01 | backward: 452.11 | backward-backward: 452.07 | backward-allreduce: 0.00 | optimizer: 16.55 | batch generator: 5.60 WARNING: Deleting old checkpoints: /u/wangh/workspace_ptmp/checkpoints_gptneox/flownet/160M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step78000 samples/sec: 1612.967 | iteration 82500/ 143000 | elapsed time per iteration (ms): 634.9 | learning rate: 2.694E-04 | approx flops per GPU: 62.6TFLOPS | lm_loss: 2.460533E+00 | loss scale: 16384.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 163.79 | backward: 452.31 | backward-backward: 452.28 | backward-allreduce: 0.00 | optimizer: 16.61 | batch generator: 5.34 samples/sec: 1600.525 | iteration 83000/ 143000 | elapsed time per iteration (ms): 639.8 | learning rate: 2.665E-04 | approx flops per GPU: 62.1TFLOPS | lm_loss: 2.460386E+00 | loss scale: 4096.0 | number of skipped iterations: 3 | number of nan iterations: 0 | time (ms) | forward: 163.61 | backward: 452.53 | backward-backward: 452.50 | backward-allreduce: 0.00 | optimizer: 16.53 | batch generator: 5.21 WARNING: Deleting old checkpoints: /u/wangh/workspace_ptmp/checkpoints_gptneox/flownet/160M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step79000 samples/sec: 1612.437 | iteration 83500/ 143000 | elapsed time per iteration (ms): 635.1 | learning rate: 2.636E-04 | approx flops per GPU: 62.6TFLOPS | lm_loss: 2.461082E+00 | loss scale: 4096.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 163.75 | backward: 452.48 | backward-backward: 452.45 | backward-allreduce: 0.00 | optimizer: 16.61 | batch generator: 5.26 samples/sec: 1615.233 | iteration 84000/ 143000 | elapsed time per iteration (ms): 634.0 | learning rate: 2.607E-04 | approx flops per GPU: 62.7TFLOPS | lm_loss: 2.460344E+00 | loss scale: 8192.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 163.62 | backward: 452.56 | backward-backward: 452.53 | backward-allreduce: 0.00 | optimizer: 16.58 | batch generator: 5.16 WARNING: Deleting old checkpoints: /u/wangh/workspace_ptmp/checkpoints_gptneox/flownet/160M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step80000 samples/sec: 1613.164 | iteration 84500/ 143000 | elapsed time per iteration (ms): 634.8 | learning rate: 2.578E-04 | approx flops per GPU: 62.6TFLOPS | lm_loss: 2.458068E+00 | loss scale: 8192.0 | number of skipped iterations: 1 | number of nan iterations: 0 | time (ms) | forward: 163.76 | backward: 452.30 | backward-backward: 452.26 | backward-allreduce: 0.00 | optimizer: 16.54 | batch generator: 5.39 samples/sec: 1615.759 | iteration 85000/ 143000 | elapsed time per iteration (ms): 633.8 | learning rate: 2.550E-04 | approx flops per GPU: 62.7TFLOPS | lm_loss: 2.458111E+00 | loss scale: 8192.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 163.67 | backward: 452.58 | backward-backward: 452.55 | backward-allreduce: 0.00 | optimizer: 16.59 | batch generator: 5.38 WARNING: Deleting old checkpoints: /u/wangh/workspace_ptmp/checkpoints_gptneox/flownet/160M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step81000 samples/sec: 1612.954 | iteration 85500/ 143000 | elapsed time per iteration (ms): 634.9 | learning rate: 2.521E-04 | approx flops per GPU: 62.6TFLOPS | lm_loss: 2.456381E+00 | loss scale: 4096.0 | number of skipped iterations: 1 | number of nan iterations: 0 | time (ms) | forward: 163.70 | backward: 452.40 | backward-backward: 452.36 | backward-allreduce: 0.00 | optimizer: 16.54 | batch generator: 5.31 samples/sec: 1615.824 | iteration 86000/ 143000 | elapsed time per iteration (ms): 633.7 | learning rate: 2.492E-04 | approx flops per GPU: 62.7TFLOPS | lm_loss: 2.457284E+00 | loss scale: 2048.0 | number of skipped iterations: 1 | number of nan iterations: 0 | time (ms) | forward: 164.25 | backward: 452.04 | backward-backward: 452.01 | backward-allreduce: 0.00 | optimizer: 16.52 | batch generator: 5.89 WARNING: Deleting old checkpoints: /u/wangh/workspace_ptmp/checkpoints_gptneox/flownet/160M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step82000 samples/sec: 1612.455 | iteration 86500/ 143000 | elapsed time per iteration (ms): 635.1 | learning rate: 2.464E-04 | approx flops per GPU: 62.6TFLOPS | lm_loss: 2.459914E+00 | loss scale: 2048.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 164.09 | backward: 452.16 | backward-backward: 452.13 | backward-allreduce: 0.00 | optimizer: 16.59 | batch generator: 5.81 samples/sec: 1527.292 | iteration 87000/ 143000 | elapsed time per iteration (ms): 670.5 | learning rate: 2.435E-04 | approx flops per GPU: 59.3TFLOPS | lm_loss: 2.457175E+00 | loss scale: 4096.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 163.42 | backward: 489.52 | backward-backward: 489.48 | backward-allreduce: 0.00 | optimizer: 16.62 | batch generator: 5.22 WARNING: Deleting old checkpoints: /u/wangh/workspace_ptmp/checkpoints_gptneox/flownet/160M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step83000 samples/sec: 1612.833 | iteration 87500/ 143000 | elapsed time per iteration (ms): 634.9 | learning rate: 2.407E-04 | approx flops per GPU: 62.6TFLOPS | lm_loss: 2.454698E+00 | loss scale: 4096.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 162.53 | backward: 453.45 | backward-backward: 453.42 | backward-allreduce: 0.00 | optimizer: 16.57 | batch generator: 4.32 samples/sec: 1615.767 | iteration 88000/ 143000 | elapsed time per iteration (ms): 633.8 | learning rate: 2.379E-04 | approx flops per GPU: 62.7TFLOPS | lm_loss: 2.455746E+00 | loss scale: 8192.0 | number of skipped iterations: 1 | number of nan iterations: 0 | time (ms) | forward: 162.92 | backward: 453.40 | backward-backward: 453.36 | backward-allreduce: 0.00 | optimizer: 16.55 | batch generator: 4.69 WARNING: Deleting old checkpoints: /u/wangh/workspace_ptmp/checkpoints_gptneox/flownet/160M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step84000 samples/sec: 1613.057 | iteration 88500/ 143000 | elapsed time per iteration (ms): 634.8 | learning rate: 2.351E-04 | approx flops per GPU: 62.6TFLOPS | lm_loss: 2.453975E+00 | loss scale: 4096.0 | number of skipped iterations: 1 | number of nan iterations: 0 | time (ms) | forward: 163.42 | backward: 452.57 | backward-backward: 452.54 | backward-allreduce: 0.00 | optimizer: 16.68 | batch generator: 5.18 samples/sec: 1616.419 | iteration 89000/ 143000 | elapsed time per iteration (ms): 633.5 | learning rate: 2.323E-04 | approx flops per GPU: 62.8TFLOPS | lm_loss: 2.455431E+00 | loss scale: 2048.0 | number of skipped iterations: 1 | number of nan iterations: 0 | time (ms) | forward: 163.09 | backward: 452.97 | backward-backward: 452.94 | backward-allreduce: 0.00 | optimizer: 16.54 | batch generator: 4.85 WARNING: Deleting old checkpoints: /u/wangh/workspace_ptmp/checkpoints_gptneox/flownet/160M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step85000 samples/sec: 1612.486 | iteration 89500/ 143000 | elapsed time per iteration (ms): 635.0 | learning rate: 2.295E-04 | approx flops per GPU: 62.6TFLOPS | lm_loss: 2.453311E+00 | loss scale: 2048.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 163.65 | backward: 452.61 | backward-backward: 452.57 | backward-allreduce: 0.00 | optimizer: 16.63 | batch generator: 5.41 samples/sec: 1616.561 | iteration 90000/ 143000 | elapsed time per iteration (ms): 633.4 | learning rate: 2.267E-04 | approx flops per GPU: 62.8TFLOPS | lm_loss: 2.452672E+00 | loss scale: 4096.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 162.95 | backward: 452.97 | backward-backward: 452.93 | backward-allreduce: 0.00 | optimizer: 16.61 | batch generator: 4.77 WARNING: Deleting old checkpoints: /u/wangh/workspace_ptmp/checkpoints_gptneox/flownet/160M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step86000 ---------------------------------------------------------------------------------------------------------- validation results at iteration 90000 | lm_loss value: 2.397496E+00 | lm_loss_ppl value: 1.099561E+01 | ---------------------------------------------------------------------------------------------------------- samples/sec: 1530.962 | iteration 90500/ 143000 | elapsed time per iteration (ms): 668.9 | learning rate: 2.240E-04 | approx flops per GPU: 59.4TFLOPS | lm_loss: 2.451771E+00 | loss scale: 4096.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 162.85 | backward: 453.05 | backward-backward: 453.01 | backward-allreduce: 0.00 | optimizer: 16.59 | batch generator: 5.20 samples/sec: 1616.472 | iteration 91000/ 143000 | elapsed time per iteration (ms): 633.5 | learning rate: 2.212E-04 | approx flops per GPU: 62.8TFLOPS | lm_loss: 2.449963E+00 | loss scale: 8192.0 | number of skipped iterations: 1 | number of nan iterations: 0 | time (ms) | forward: 163.26 | backward: 452.81 | backward-backward: 452.77 | backward-allreduce: 0.00 | optimizer: 16.54 | batch generator: 5.00 WARNING: Deleting old checkpoints: /u/wangh/workspace_ptmp/checkpoints_gptneox/flownet/160M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step87000 samples/sec: 1612.945 | iteration 91500/ 143000 | elapsed time per iteration (ms): 634.9 | learning rate: 2.185E-04 | approx flops per GPU: 62.6TFLOPS | lm_loss: 2.450699E+00 | loss scale: 4096.0 | number of skipped iterations: 1 | number of nan iterations: 0 | time (ms) | forward: 163.46 | backward: 452.62 | backward-backward: 452.59 | backward-allreduce: 0.00 | optimizer: 16.53 | batch generator: 5.29 samples/sec: 1616.143 | iteration 92000/ 143000 | elapsed time per iteration (ms): 633.6 | learning rate: 2.158E-04 | approx flops per GPU: 62.7TFLOPS | lm_loss: 2.449873E+00 | loss scale: 4096.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 163.36 | backward: 452.82 | backward-backward: 452.79 | backward-allreduce: 0.00 | optimizer: 16.54 | batch generator: 5.08 WARNING: Deleting old checkpoints: /u/wangh/workspace_ptmp/checkpoints_gptneox/flownet/160M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step88000 samples/sec: 1612.437 | iteration 92500/ 143000 | elapsed time per iteration (ms): 635.1 | learning rate: 2.131E-04 | approx flops per GPU: 62.6TFLOPS | lm_loss: 2.449048E+00 | loss scale: 8192.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 162.92 | backward: 453.23 | backward-backward: 453.19 | backward-allreduce: 0.00 | optimizer: 16.58 | batch generator: 4.68 samples/sec: 1616.169 | iteration 93000/ 143000 | elapsed time per iteration (ms): 633.6 | learning rate: 2.104E-04 | approx flops per GPU: 62.7TFLOPS | lm_loss: 2.449526E+00 | loss scale: 8192.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 163.03 | backward: 453.09 | backward-backward: 453.05 | backward-allreduce: 0.00 | optimizer: 16.56 | batch generator: 4.72 WARNING: Deleting old checkpoints: /u/wangh/workspace_ptmp/checkpoints_gptneox/flownet/160M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step89000 samples/sec: 1611.629 | iteration 93500/ 143000 | elapsed time per iteration (ms): 635.4 | learning rate: 2.077E-04 | approx flops per GPU: 62.6TFLOPS | lm_loss: 2.446281E+00 | loss scale: 4096.0 | number of skipped iterations: 2 | number of nan iterations: 0 | time (ms) | forward: 163.38 | backward: 453.05 | backward-backward: 453.01 | backward-allreduce: 0.00 | optimizer: 16.54 | batch generator: 5.00 samples/sec: 1616.000 | iteration 94000/ 143000 | elapsed time per iteration (ms): 633.7 | learning rate: 2.050E-04 | approx flops per GPU: 62.7TFLOPS | lm_loss: 2.446838E+00 | loss scale: 4096.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 163.31 | backward: 452.90 | backward-backward: 452.86 | backward-allreduce: 0.00 | optimizer: 16.57 | batch generator: 5.06 WARNING: Deleting old checkpoints: /u/wangh/workspace_ptmp/checkpoints_gptneox/flownet/160M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step90000 samples/sec: 1613.109 | iteration 94500/ 143000 | elapsed time per iteration (ms): 634.8 | learning rate: 2.024E-04 | approx flops per GPU: 62.6TFLOPS | lm_loss: 2.447458E+00 | loss scale: 8192.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 163.51 | backward: 452.66 | backward-backward: 452.62 | backward-allreduce: 0.00 | optimizer: 16.55 | batch generator: 5.11 samples/sec: 1616.479 | iteration 95000/ 143000 | elapsed time per iteration (ms): 633.5 | learning rate: 1.998E-04 | approx flops per GPU: 62.8TFLOPS | lm_loss: 2.444553E+00 | loss scale: 8192.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 163.60 | backward: 452.43 | backward-backward: 452.40 | backward-allreduce: 0.00 | optimizer: 16.57 | batch generator: 5.32 WARNING: Deleting old checkpoints: /u/wangh/workspace_ptmp/checkpoints_gptneox/flownet/160M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step91000 samples/sec: 1576.732 | iteration 95500/ 143000 | elapsed time per iteration (ms): 649.4 | learning rate: 1.972E-04 | approx flops per GPU: 61.2TFLOPS | lm_loss: 2.445119E+00 | loss scale: 8192.0 | number of skipped iterations: 2 | number of nan iterations: 0 | time (ms) | forward: 163.48 | backward: 453.59 | backward-backward: 453.56 | backward-allreduce: 0.00 | optimizer: 16.52 | batch generator: 5.12 samples/sec: 1616.366 | iteration 96000/ 143000 | elapsed time per iteration (ms): 633.5 | learning rate: 1.946E-04 | approx flops per GPU: 62.8TFLOPS | lm_loss: 2.445789E+00 | loss scale: 4096.0 | number of skipped iterations: 1 | number of nan iterations: 0 | time (ms) | forward: 163.62 | backward: 452.52 | backward-backward: 452.49 | backward-allreduce: 0.00 | optimizer: 16.51 | batch generator: 5.37 WARNING: Deleting old checkpoints: /u/wangh/workspace_ptmp/checkpoints_gptneox/flownet/160M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step92000 samples/sec: 1613.281 | iteration 96500/ 143000 | elapsed time per iteration (ms): 634.7 | learning rate: 1.920E-04 | approx flops per GPU: 62.6TFLOPS | lm_loss: 2.443626E+00 | loss scale: 4096.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 163.44 | backward: 452.49 | backward-backward: 452.45 | backward-allreduce: 0.00 | optimizer: 16.59 | batch generator: 5.24 samples/sec: 1616.822 | iteration 97000/ 143000 | elapsed time per iteration (ms): 633.3 | learning rate: 1.894E-04 | approx flops per GPU: 62.8TFLOPS | lm_loss: 2.442832E+00 | loss scale: 8192.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 163.11 | backward: 452.76 | backward-backward: 452.72 | backward-allreduce: 0.00 | optimizer: 16.59 | batch generator: 4.91 WARNING: Deleting old checkpoints: /u/wangh/workspace_ptmp/checkpoints_gptneox/flownet/160M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step93000 samples/sec: 1612.450 | iteration 97500/ 143000 | elapsed time per iteration (ms): 635.1 | learning rate: 1.869E-04 | approx flops per GPU: 62.6TFLOPS | lm_loss: 2.442605E+00 | loss scale: 4096.0 | number of skipped iterations: 2 | number of nan iterations: 0 | time (ms) | forward: 163.35 | backward: 452.85 | backward-backward: 452.82 | backward-allreduce: 0.00 | optimizer: 16.71 | batch generator: 5.12 samples/sec: 787.403 | iteration 98000/ 143000 | elapsed time per iteration (ms): 1300.5 | learning rate: 1.843E-04 | approx flops per GPU: 30.6TFLOPS | lm_loss: 2.441002E+00 | loss scale: 2048.0 | number of skipped iterations: 1 | number of nan iterations: 0 | time (ms) | forward: 164.89 | backward: 1118.02 | backward-backward: 1117.98 | backward-allreduce: 0.00 | optimizer: 16.65 | batch generator: 6.67 WARNING: Deleting old checkpoints: /u/wangh/workspace_ptmp/checkpoints_gptneox/flownet/160M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step94000 samples/sec: 496.761 | iteration 98500/ 143000 | elapsed time per iteration (ms): 2061.4 | learning rate: 1.818E-04 | approx flops per GPU: 19.3TFLOPS | lm_loss: 2.441045E+00 | loss scale: 2048.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 202.74 | backward: 1827.67 | backward-backward: 1827.63 | backward-allreduce: 0.00 | optimizer: 16.70 | batch generator: 44.71 samples/sec: 436.255 | iteration 99000/ 143000 | elapsed time per iteration (ms): 2347.3 | learning rate: 1.793E-04 | approx flops per GPU: 16.9TFLOPS | lm_loss: 2.441410E+00 | loss scale: 4096.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 168.45 | backward: 2160.92 | backward-backward: 2160.87 | backward-allreduce: 0.00 | optimizer: 16.76 | batch generator: 10.43 WARNING: Deleting old checkpoints: /u/wangh/workspace_ptmp/checkpoints_gptneox/flownet/160M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step95000 samples/sec: 249.689 | iteration 99500/ 143000 | elapsed time per iteration (ms): 4101.1 | learning rate: 1.769E-04 | approx flops per GPU: 9.7TFLOPS | lm_loss: 2.441907E+00 | loss scale: 4096.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 188.76 | backward: 3792.93 | backward-backward: 3792.88 | backward-allreduce: 0.00 | optimizer: 16.78 | batch generator: 30.63 samples/sec: 185.576 | iteration 100000/ 143000 | elapsed time per iteration (ms): 5518.0 | learning rate: 1.744E-04 | approx flops per GPU: 7.2TFLOPS | lm_loss: 2.439974E+00 | loss scale: 8192.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 179.86 | backward: 5320.45 | backward-backward: 5320.40 | backward-allreduce: 0.00 | optimizer: 16.76 | batch generator: 21.69 WARNING: Deleting old checkpoints: /u/wangh/workspace_ptmp/checkpoints_gptneox/flownet/160M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step96000 ----------------------------------------------------------------------------------------------------------- validation results at iteration 100000 | lm_loss value: 2.388441E+00 | lm_loss_ppl value: 1.089649E+01 | ----------------------------------------------------------------------------------------------------------- samples/sec: 170.564 | iteration 100500/ 143000 | elapsed time per iteration (ms): 6003.6 | learning rate: 1.720E-04 | approx flops per GPU: 6.6TFLOPS | lm_loss: 2.437224E+00 | loss scale: 8192.0 | number of skipped iterations: 1 | number of nan iterations: 0 | time (ms) | forward: 265.60 | backward: 4809.92 | backward-backward: 4809.87 | backward-allreduce: 0.00 | optimizer: 16.78 | batch generator: 674.40 samples/sec: 180.868 | iteration 101000/ 143000 | elapsed time per iteration (ms): 5661.6 | learning rate: 1.696E-04 | approx flops per GPU: 7.0TFLOPS | lm_loss: 2.436939E+00 | loss scale: 4096.0 | number of skipped iterations: 1 | number of nan iterations: 0 | time (ms) | forward: 314.00 | backward: 5329.88 | backward-backward: 5329.83 | backward-allreduce: 0.00 | optimizer: 16.72 | batch generator: 155.89 WARNING: Deleting old checkpoints: /u/wangh/workspace_ptmp/checkpoints_gptneox/flownet/160M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step97000 samples/sec: 151.808 | iteration 101500/ 143000 | elapsed time per iteration (ms): 6745.4 | learning rate: 1.672E-04 | approx flops per GPU: 5.9TFLOPS | lm_loss: 2.437546E+00 | loss scale: 4096.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 336.83 | backward: 6237.11 | backward-backward: 6237.06 | backward-allreduce: 0.00 | optimizer: 16.76 | batch generator: 178.66 samples/sec: 160.545 | iteration 102000/ 143000 | elapsed time per iteration (ms): 6378.3 | learning rate: 1.648E-04 | approx flops per GPU: 6.2TFLOPS | lm_loss: 2.435754E+00 | loss scale: 8192.0 | number of skipped iterations: 1 | number of nan iterations: 0 | time (ms) | forward: 252.64 | backward: 6108.02 | backward-backward: 6107.97 | backward-allreduce: 0.00 | optimizer: 16.72 | batch generator: 94.48 WARNING: Deleting old checkpoints: /u/wangh/workspace_ptmp/checkpoints_gptneox/flownet/160M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step98000 samples/sec: 176.469 | iteration 102500/ 143000 | elapsed time per iteration (ms): 5802.7 | learning rate: 1.624E-04 | approx flops per GPU: 6.9TFLOPS | lm_loss: 2.435586E+00 | loss scale: 4096.0 | number of skipped iterations: 1 | number of nan iterations: 0 | time (ms) | forward: 369.14 | backward: 5413.73 | backward-backward: 5413.69 | backward-allreduce: 0.00 | optimizer: 16.77 | batch generator: 211.04 samples/sec: 145.921 | iteration 103000/ 143000 | elapsed time per iteration (ms): 7017.5 | learning rate: 1.601E-04 | approx flops per GPU: 5.7TFLOPS | lm_loss: 2.435019E+00 | loss scale: 4096.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 720.19 | backward: 6279.59 | backward-backward: 6279.54 | backward-allreduce: 0.00 | optimizer: 16.79 | batch generator: 562.23 WARNING: Deleting old checkpoints: /u/wangh/workspace_ptmp/checkpoints_gptneox/flownet/160M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step99000 samples/sec: 135.926 | iteration 103500/ 143000 | elapsed time per iteration (ms): 7533.5 | learning rate: 1.578E-04 | approx flops per GPU: 5.3TFLOPS | lm_loss: 2.434794E+00 | loss scale: 8192.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 354.18 | backward: 7126.18 | backward-backward: 7126.13 | backward-allreduce: 0.00 | optimizer: 16.79 | batch generator: 196.18 samples/sec: 166.194 | iteration 104000/ 143000 | elapsed time per iteration (ms): 6161.5 | learning rate: 1.555E-04 | approx flops per GPU: 6.5TFLOPS | lm_loss: 2.434140E+00 | loss scale: 8192.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 299.54 | backward: 5844.08 | backward-backward: 5844.03 | backward-allreduce: 0.00 | optimizer: 16.83 | batch generator: 141.63 WARNING: Deleting old checkpoints: /u/wangh/workspace_ptmp/checkpoints_gptneox/flownet/160M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step100000 samples/sec: 180.398 | iteration 104500/ 143000 | elapsed time per iteration (ms): 5676.3 | learning rate: 1.532E-04 | approx flops per GPU: 7.0TFLOPS | lm_loss: 2.432863E+00 | loss scale: 4096.0 | number of skipped iterations: 3 | number of nan iterations: 0 | time (ms) | forward: 531.85 | backward: 5117.66 | backward-backward: 5117.61 | backward-allreduce: 0.00 | optimizer: 16.69 | batch generator: 373.92 samples/sec: 189.686 | iteration 105000/ 143000 | elapsed time per iteration (ms): 5398.4 | learning rate: 1.510E-04 | approx flops per GPU: 7.4TFLOPS | lm_loss: 2.434104E+00 | loss scale: 4096.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 332.84 | backward: 5047.57 | backward-backward: 5047.51 | backward-allreduce: 0.00 | optimizer: 16.79 | batch generator: 174.79 WARNING: Deleting old checkpoints: /u/wangh/workspace_ptmp/checkpoints_gptneox/flownet/160M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step101000 samples/sec: 185.821 | iteration 105500/ 143000 | elapsed time per iteration (ms): 5510.7 | learning rate: 1.487E-04 | approx flops per GPU: 7.2TFLOPS | lm_loss: 2.433261E+00 | loss scale: 8192.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 249.48 | backward: 5227.24 | backward-backward: 5227.19 | backward-allreduce: 0.00 | optimizer: 16.77 | batch generator: 91.47 samples/sec: 194.648 | iteration 106000/ 143000 | elapsed time per iteration (ms): 5260.8 | learning rate: 1.465E-04 | approx flops per GPU: 7.6TFLOPS | lm_loss: 2.430794E+00 | loss scale: 8192.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 271.78 | backward: 4971.31 | backward-backward: 4971.25 | backward-allreduce: 0.00 | optimizer: 16.79 | batch generator: 113.81 WARNING: Deleting old checkpoints: /u/wangh/workspace_ptmp/checkpoints_gptneox/flownet/160M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step102000 samples/sec: 187.317 | iteration 106500/ 143000 | elapsed time per iteration (ms): 5466.7 | learning rate: 1.443E-04 | approx flops per GPU: 7.3TFLOPS | lm_loss: 2.431040E+00 | loss scale: 16384.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 252.29 | backward: 5173.12 | backward-backward: 5173.07 | backward-allreduce: 0.00 | optimizer: 16.77 | batch generator: 94.34 samples/sec: 1619.710 | iteration 107000/ 143000 | elapsed time per iteration (ms): 632.2 | learning rate: 1.422E-04 | approx flops per GPU: 62.9TFLOPS | lm_loss: 2.431381E+00 | loss scale: 8192.0 | number of skipped iterations: 2 | number of nan iterations: 0 | time (ms) | forward: 161.41 | backward: 453.27 | backward-backward: 453.23 | backward-allreduce: 0.00 | optimizer: 16.52 | batch generator: 3.25 WARNING: Deleting old checkpoints: /u/wangh/workspace_ptmp/checkpoints_gptneox/flownet/160M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step103000 samples/sec: 1614.117 | iteration 107500/ 143000 | elapsed time per iteration (ms): 634.4 | learning rate: 1.400E-04 | approx flops per GPU: 62.7TFLOPS | lm_loss: 2.430331E+00 | loss scale: 4096.0 | number of skipped iterations: 1 | number of nan iterations: 0 | time (ms) | forward: 161.25 | backward: 453.09 | backward-backward: 453.06 | backward-allreduce: 0.00 | optimizer: 16.54 | batch generator: 3.04 samples/sec: 1620.183 | iteration 108000/ 143000 | elapsed time per iteration (ms): 632.0 | learning rate: 1.379E-04 | approx flops per GPU: 62.9TFLOPS | lm_loss: 2.428372E+00 | loss scale: 4096.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 161.51 | backward: 453.02 | backward-backward: 452.99 | backward-allreduce: 0.00 | optimizer: 16.54 | batch generator: 3.40 samples/sec: 1594.574 | iteration 108500/ 143000 | elapsed time per iteration (ms): 642.2 | learning rate: 1.358E-04 | approx flops per GPU: 61.9TFLOPS | lm_loss: 2.428006E+00 | loss scale: 8192.0 | number of skipped iterations: 0 | number of nan iterations: 0 | after 108500 iterations memory (MB) | allocated: 632.1875 | max allocated: 10746.48486328125 | reserved: 11840.0 | max reserved: 11840.0 time (ms) | forward: 170.54 | backward: 454.11 | backward-backward: 454.07 | backward-allreduce: 0.00 | optimizer: 16.54 | batch generator: 5.98 samples/sec: 1618.661 | iteration 109000/ 143000 | elapsed time per iteration (ms): 632.6 | learning rate: 1.338E-04 | approx flops per GPU: 62.8TFLOPS | lm_loss: 2.427526E+00 | loss scale: 8192.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 161.36 | backward: 453.73 | backward-backward: 453.69 | backward-allreduce: 0.00 | optimizer: 16.54 | batch generator: 3.25 WARNING: Deleting old checkpoints: /u/wangh/workspace_ptmp/checkpoints_gptneox/flownet/160M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step105000 samples/sec: 1612.861 | iteration 109500/ 143000 | elapsed time per iteration (ms): 634.9 | learning rate: 1.317E-04 | approx flops per GPU: 62.6TFLOPS | lm_loss: 2.426602E+00 | loss scale: 8192.0 | number of skipped iterations: 2 | number of nan iterations: 0 | time (ms) | forward: 161.66 | backward: 453.02 | backward-backward: 452.98 | backward-allreduce: 0.00 | optimizer: 16.47 | batch generator: 3.51 samples/sec: 1619.347 | iteration 110000/ 143000 | elapsed time per iteration (ms): 632.4 | learning rate: 1.297E-04 | approx flops per GPU: 62.9TFLOPS | lm_loss: 2.429498E+00 | loss scale: 8192.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 161.47 | backward: 453.32 | backward-backward: 453.28 | backward-allreduce: 0.00 | optimizer: 16.56 | batch generator: 3.13 WARNING: Deleting old checkpoints: /u/wangh/workspace_ptmp/checkpoints_gptneox/flownet/160M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step106000 ----------------------------------------------------------------------------------------------------------- validation results at iteration 110000 | lm_loss value: 2.376773E+00 | lm_loss_ppl value: 1.077009E+01 | ----------------------------------------------------------------------------------------------------------- samples/sec: 1534.214 | iteration 110500/ 143000 | elapsed time per iteration (ms): 667.4 | learning rate: 1.277E-04 | approx flops per GPU: 59.6TFLOPS | lm_loss: 2.426958E+00 | loss scale: 16384.0 | number of skipped iterations: 1 | number of nan iterations: 0 | time (ms) | forward: 161.40 | backward: 453.37 | backward-backward: 453.34 | backward-allreduce: 0.00 | optimizer: 16.52 | batch generator: 3.75 samples/sec: 1619.171 | iteration 111000/ 143000 | elapsed time per iteration (ms): 632.4 | learning rate: 1.257E-04 | approx flops per GPU: 62.9TFLOPS | lm_loss: 2.426144E+00 | loss scale: 8192.0 | number of skipped iterations: 1 | number of nan iterations: 0 | time (ms) | forward: 161.53 | backward: 453.38 | backward-backward: 453.34 | backward-allreduce: 0.00 | optimizer: 16.51 | batch generator: 3.14 WARNING: Deleting old checkpoints: /u/wangh/workspace_ptmp/checkpoints_gptneox/flownet/160M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step107000 samples/sec: 1614.881 | iteration 111500/ 143000 | elapsed time per iteration (ms): 634.1 | learning rate: 1.238E-04 | approx flops per GPU: 62.7TFLOPS | lm_loss: 2.424898E+00 | loss scale: 8192.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 162.72 | backward: 452.51 | backward-backward: 452.47 | backward-allreduce: 0.00 | optimizer: 16.55 | batch generator: 4.19 samples/sec: 1618.743 | iteration 112000/ 143000 | elapsed time per iteration (ms): 632.6 | learning rate: 1.219E-04 | approx flops per GPU: 62.8TFLOPS | lm_loss: 2.421898E+00 | loss scale: 16384.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 162.37 | backward: 452.70 | backward-backward: 452.66 | backward-allreduce: 0.00 | optimizer: 16.53 | batch generator: 4.17 WARNING: Deleting old checkpoints: /u/wangh/workspace_ptmp/checkpoints_gptneox/flownet/160M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step108000 samples/sec: 1615.785 | iteration 112500/ 143000 | elapsed time per iteration (ms): 633.7 | learning rate: 1.200E-04 | approx flops per GPU: 62.7TFLOPS | lm_loss: 2.423645E+00 | loss scale: 4096.0 | number of skipped iterations: 3 | number of nan iterations: 0 | time (ms) | forward: 162.09 | backward: 452.99 | backward-backward: 452.95 | backward-allreduce: 0.00 | optimizer: 16.44 | batch generator: 3.88 samples/sec: 1618.699 | iteration 113000/ 143000 | elapsed time per iteration (ms): 632.6 | learning rate: 1.181E-04 | approx flops per GPU: 62.8TFLOPS | lm_loss: 2.421002E+00 | loss scale: 4096.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 162.40 | backward: 452.72 | backward-backward: 452.69 | backward-allreduce: 0.00 | optimizer: 16.49 | batch generator: 4.16 WARNING: Deleting old checkpoints: /u/wangh/workspace_ptmp/checkpoints_gptneox/flownet/160M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step109000 samples/sec: 1615.403 | iteration 113500/ 143000 | elapsed time per iteration (ms): 633.9 | learning rate: 1.163E-04 | approx flops per GPU: 62.7TFLOPS | lm_loss: 2.423349E+00 | loss scale: 8192.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 162.19 | backward: 452.91 | backward-backward: 452.87 | backward-allreduce: 0.00 | optimizer: 16.51 | batch generator: 3.89 samples/sec: 1619.105 | iteration 114000/ 143000 | elapsed time per iteration (ms): 632.4 | learning rate: 1.145E-04 | approx flops per GPU: 62.9TFLOPS | lm_loss: 2.421496E+00 | loss scale: 4096.0 | number of skipped iterations: 2 | number of nan iterations: 0 | time (ms) | forward: 162.19 | backward: 452.81 | backward-backward: 452.77 | backward-allreduce: 0.00 | optimizer: 16.44 | batch generator: 4.00 WARNING: Deleting old checkpoints: /u/wangh/workspace_ptmp/checkpoints_gptneox/flownet/160M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step110000 samples/sec: 1614.973 | iteration 114500/ 143000 | elapsed time per iteration (ms): 634.1 | learning rate: 1.127E-04 | approx flops per GPU: 62.7TFLOPS | lm_loss: 2.421683E+00 | loss scale: 4096.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 162.12 | backward: 453.05 | backward-backward: 453.02 | backward-allreduce: 0.00 | optimizer: 16.50 | batch generator: 3.89 samples/sec: 1618.929 | iteration 115000/ 143000 | elapsed time per iteration (ms): 632.5 | learning rate: 1.109E-04 | approx flops per GPU: 62.9TFLOPS | lm_loss: 2.420394E+00 | loss scale: 8192.0 | number of skipped iterations: 1 | number of nan iterations: 0 | time (ms) | forward: 161.85 | backward: 453.20 | backward-backward: 453.16 | backward-allreduce: 0.00 | optimizer: 16.48 | batch generator: 3.57 WARNING: Deleting old checkpoints: /u/wangh/workspace_ptmp/checkpoints_gptneox/flownet/160M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step111000 samples/sec: 1610.022 | iteration 115500/ 143000 | elapsed time per iteration (ms): 636.0 | learning rate: 1.092E-04 | approx flops per GPU: 62.5TFLOPS | lm_loss: 2.419081E+00 | loss scale: 4096.0 | number of skipped iterations: 1 | number of nan iterations: 0 | time (ms) | forward: 162.67 | backward: 454.52 | backward-backward: 454.48 | backward-allreduce: 0.00 | optimizer: 16.45 | batch generator: 4.42 samples/sec: 1618.394 | iteration 116000/ 143000 | elapsed time per iteration (ms): 632.7 | learning rate: 1.075E-04 | approx flops per GPU: 62.8TFLOPS | lm_loss: 2.420936E+00 | loss scale: 2048.0 | number of skipped iterations: 1 | number of nan iterations: 0 | time (ms) | forward: 162.09 | backward: 453.16 | backward-backward: 453.13 | backward-allreduce: 0.00 | optimizer: 16.49 | batch generator: 3.87 WARNING: Deleting old checkpoints: /u/wangh/workspace_ptmp/checkpoints_gptneox/flownet/160M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step112000 samples/sec: 1614.665 | iteration 116500/ 143000 | elapsed time per iteration (ms): 634.2 | learning rate: 1.058E-04 | approx flops per GPU: 62.7TFLOPS | lm_loss: 2.421074E+00 | loss scale: 2048.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 162.58 | backward: 452.89 | backward-backward: 452.85 | backward-allreduce: 0.00 | optimizer: 16.55 | batch generator: 4.37 samples/sec: 1613.907 | iteration 117000/ 143000 | elapsed time per iteration (ms): 634.5 | learning rate: 1.041E-04 | approx flops per GPU: 62.7TFLOPS | lm_loss: 2.418272E+00 | loss scale: 4096.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 163.09 | backward: 453.86 | backward-backward: 453.82 | backward-allreduce: 0.00 | optimizer: 16.53 | batch generator: 3.96 WARNING: Deleting old checkpoints: /u/wangh/workspace_ptmp/checkpoints_gptneox/flownet/160M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step113000 samples/sec: 1613.909 | iteration 117500/ 143000 | elapsed time per iteration (ms): 634.5 | learning rate: 1.025E-04 | approx flops per GPU: 62.7TFLOPS | lm_loss: 2.419017E+00 | loss scale: 4096.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 162.20 | backward: 453.09 | backward-backward: 453.05 | backward-allreduce: 0.00 | optimizer: 16.50 | batch generator: 3.98 samples/sec: 1618.145 | iteration 118000/ 143000 | elapsed time per iteration (ms): 632.8 | learning rate: 1.009E-04 | approx flops per GPU: 62.8TFLOPS | lm_loss: 2.417656E+00 | loss scale: 8192.0 | number of skipped iterations: 1 | number of nan iterations: 0 | time (ms) | forward: 162.29 | backward: 453.05 | backward-backward: 453.02 | backward-allreduce: 0.00 | optimizer: 16.48 | batch generator: 4.08 WARNING: Deleting old checkpoints: /u/wangh/workspace_ptmp/checkpoints_gptneox/flownet/160M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step114000 samples/sec: 1615.252 | iteration 118500/ 143000 | elapsed time per iteration (ms): 634.0 | learning rate: 9.934E-05 | approx flops per GPU: 62.7TFLOPS | lm_loss: 2.417009E+00 | loss scale: 4096.0 | number of skipped iterations: 1 | number of nan iterations: 0 | time (ms) | forward: 162.63 | backward: 452.70 | backward-backward: 452.66 | backward-allreduce: 0.00 | optimizer: 16.46 | batch generator: 4.36 samples/sec: 1618.007 | iteration 119000/ 143000 | elapsed time per iteration (ms): 632.9 | learning rate: 9.780E-05 | approx flops per GPU: 62.8TFLOPS | lm_loss: 2.415432E+00 | loss scale: 4096.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 162.47 | backward: 452.89 | backward-backward: 452.86 | backward-allreduce: 0.00 | optimizer: 16.53 | batch generator: 4.27 WARNING: Deleting old checkpoints: /u/wangh/workspace_ptmp/checkpoints_gptneox/flownet/160M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step115000 samples/sec: 1614.576 | iteration 119500/ 143000 | elapsed time per iteration (ms): 634.2 | learning rate: 9.629E-05 | approx flops per GPU: 62.7TFLOPS | lm_loss: 2.416579E+00 | loss scale: 8192.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 162.49 | backward: 452.89 | backward-backward: 452.86 | backward-allreduce: 0.00 | optimizer: 16.51 | batch generator: 4.25 samples/sec: 1617.792 | iteration 120000/ 143000 | elapsed time per iteration (ms): 633.0 | learning rate: 9.480E-05 | approx flops per GPU: 62.8TFLOPS | lm_loss: 2.415278E+00 | loss scale: 8192.0 | number of skipped iterations: 1 | number of nan iterations: 0 | time (ms) | forward: 162.73 | backward: 452.77 | backward-backward: 452.73 | backward-allreduce: 0.00 | optimizer: 16.47 | batch generator: 4.48 WARNING: Deleting old checkpoints: /u/wangh/workspace_ptmp/checkpoints_gptneox/flownet/160M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step116000 ----------------------------------------------------------------------------------------------------------- validation results at iteration 120000 | lm_loss value: 2.366806E+00 | lm_loss_ppl value: 1.066327E+01 | ----------------------------------------------------------------------------------------------------------- samples/sec: 1532.723 | iteration 120500/ 143000 | elapsed time per iteration (ms): 668.1 | learning rate: 9.335E-05 | approx flops per GPU: 59.5TFLOPS | lm_loss: 2.413904E+00 | loss scale: 8192.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 162.30 | backward: 453.12 | backward-backward: 453.09 | backward-allreduce: 0.00 | optimizer: 16.56 | batch generator: 4.41 samples/sec: 1617.392 | iteration 121000/ 143000 | elapsed time per iteration (ms): 633.1 | learning rate: 9.192E-05 | approx flops per GPU: 62.8TFLOPS | lm_loss: 2.415495E+00 | loss scale: 4096.0 | number of skipped iterations: 1 | number of nan iterations: 0 | time (ms) | forward: 162.20 | backward: 453.19 | backward-backward: 453.16 | backward-allreduce: 0.00 | optimizer: 16.69 | batch generator: 3.85 WARNING: Deleting old checkpoints: /u/wangh/workspace_ptmp/checkpoints_gptneox/flownet/160M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step117000 samples/sec: 1615.220 | iteration 121500/ 143000 | elapsed time per iteration (ms): 634.0 | learning rate: 9.053E-05 | approx flops per GPU: 62.7TFLOPS | lm_loss: 2.414902E+00 | loss scale: 2048.0 | number of skipped iterations: 1 | number of nan iterations: 0 | time (ms) | forward: 162.01 | backward: 453.23 | backward-backward: 453.19 | backward-allreduce: 0.00 | optimizer: 16.50 | batch generator: 3.67 samples/sec: 1616.946 | iteration 122000/ 143000 | elapsed time per iteration (ms): 633.3 | learning rate: 8.916E-05 | approx flops per GPU: 62.8TFLOPS | lm_loss: 2.414169E+00 | loss scale: 2048.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 162.16 | backward: 453.41 | backward-backward: 453.37 | backward-allreduce: 0.00 | optimizer: 16.72 | batch generator: 3.86 WARNING: Deleting old checkpoints: /u/wangh/workspace_ptmp/checkpoints_gptneox/flownet/160M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step118000 samples/sec: 1614.642 | iteration 122500/ 143000 | elapsed time per iteration (ms): 634.2 | learning rate: 8.782E-05 | approx flops per GPU: 62.7TFLOPS | lm_loss: 2.415218E+00 | loss scale: 4096.0 | number of skipped iterations: 1 | number of nan iterations: 0 | time (ms) | forward: 162.49 | backward: 452.93 | backward-backward: 452.89 | backward-allreduce: 0.00 | optimizer: 16.48 | batch generator: 4.28 samples/sec: 1617.352 | iteration 123000/ 143000 | elapsed time per iteration (ms): 633.1 | learning rate: 8.651E-05 | approx flops per GPU: 62.8TFLOPS | lm_loss: 2.415413E+00 | loss scale: 4096.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 162.10 | backward: 453.45 | backward-backward: 453.42 | backward-allreduce: 0.00 | optimizer: 16.59 | batch generator: 3.87 WARNING: Deleting old checkpoints: /u/wangh/workspace_ptmp/checkpoints_gptneox/flownet/160M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step119000 samples/sec: 1614.248 | iteration 123500/ 143000 | elapsed time per iteration (ms): 634.4 | learning rate: 8.523E-05 | approx flops per GPU: 62.7TFLOPS | lm_loss: 2.413637E+00 | loss scale: 8192.0 | number of skipped iterations: 1 | number of nan iterations: 0 | time (ms) | forward: 162.02 | backward: 453.48 | backward-backward: 453.44 | backward-allreduce: 0.00 | optimizer: 16.58 | batch generator: 3.65 samples/sec: 1617.265 | iteration 124000/ 143000 | elapsed time per iteration (ms): 633.2 | learning rate: 8.399E-05 | approx flops per GPU: 62.8TFLOPS | lm_loss: 2.412730E+00 | loss scale: 4096.0 | number of skipped iterations: 1 | number of nan iterations: 0 | time (ms) | forward: 162.08 | backward: 453.52 | backward-backward: 453.48 | backward-allreduce: 0.00 | optimizer: 16.57 | batch generator: 3.77 WARNING: Deleting old checkpoints: /u/wangh/workspace_ptmp/checkpoints_gptneox/flownet/160M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step120000 samples/sec: 1600.094 | iteration 124500/ 143000 | elapsed time per iteration (ms): 640.0 | learning rate: 8.277E-05 | approx flops per GPU: 62.1TFLOPS | lm_loss: 2.411305E+00 | loss scale: 4096.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 162.67 | backward: 452.85 | backward-backward: 452.82 | backward-allreduce: 0.00 | optimizer: 16.58 | batch generator: 4.39 samples/sec: 1617.998 | iteration 125000/ 143000 | elapsed time per iteration (ms): 632.9 | learning rate: 8.158E-05 | approx flops per GPU: 62.8TFLOPS | lm_loss: 2.412544E+00 | loss scale: 8192.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 162.66 | backward: 452.70 | backward-backward: 452.67 | backward-allreduce: 0.00 | optimizer: 16.50 | batch generator: 4.43 WARNING: Deleting old checkpoints: /u/wangh/workspace_ptmp/checkpoints_gptneox/flownet/160M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step121000 samples/sec: 1614.465 | iteration 125500/ 143000 | elapsed time per iteration (ms): 634.3 | learning rate: 8.042E-05 | approx flops per GPU: 62.7TFLOPS | lm_loss: 2.411521E+00 | loss scale: 8192.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 162.70 | backward: 452.81 | backward-backward: 452.77 | backward-allreduce: 0.00 | optimizer: 16.54 | batch generator: 4.29 samples/sec: 1616.046 | iteration 126000/ 143000 | elapsed time per iteration (ms): 633.6 | learning rate: 7.929E-05 | approx flops per GPU: 62.7TFLOPS | lm_loss: 2.409941E+00 | loss scale: 16384.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 162.50 | backward: 453.58 | backward-backward: 453.54 | backward-allreduce: 0.00 | optimizer: 16.55 | batch generator: 4.29 WARNING: Deleting old checkpoints: /u/wangh/workspace_ptmp/checkpoints_gptneox/flownet/160M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step122000 samples/sec: 1614.145 | iteration 126500/ 143000 | elapsed time per iteration (ms): 634.4 | learning rate: 7.820E-05 | approx flops per GPU: 62.7TFLOPS | lm_loss: 2.410583E+00 | loss scale: 8192.0 | number of skipped iterations: 2 | number of nan iterations: 0 | time (ms) | forward: 162.79 | backward: 452.88 | backward-backward: 452.84 | backward-allreduce: 0.00 | optimizer: 16.49 | batch generator: 4.39 samples/sec: 1617.951 | iteration 127000/ 143000 | elapsed time per iteration (ms): 632.9 | learning rate: 7.714E-05 | approx flops per GPU: 62.8TFLOPS | lm_loss: 2.411517E+00 | loss scale: 8192.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 162.23 | backward: 453.13 | backward-backward: 453.09 | backward-allreduce: 0.00 | optimizer: 16.54 | batch generator: 3.99 WARNING: Deleting old checkpoints: /u/wangh/workspace_ptmp/checkpoints_gptneox/flownet/160M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step123000 samples/sec: 1615.005 | iteration 127500/ 143000 | elapsed time per iteration (ms): 634.1 | learning rate: 7.611E-05 | approx flops per GPU: 62.7TFLOPS | lm_loss: 2.409723E+00 | loss scale: 8192.0 | number of skipped iterations: 2 | number of nan iterations: 0 | time (ms) | forward: 162.08 | backward: 453.21 | backward-backward: 453.18 | backward-allreduce: 0.00 | optimizer: 16.44 | batch generator: 3.94 samples/sec: 1617.641 | iteration 128000/ 143000 | elapsed time per iteration (ms): 633.0 | learning rate: 7.510E-05 | approx flops per GPU: 62.8TFLOPS | lm_loss: 2.410778E+00 | loss scale: 4096.0 | number of skipped iterations: 1 | number of nan iterations: 0 | time (ms) | forward: 162.51 | backward: 453.00 | backward-backward: 452.96 | backward-allreduce: 0.00 | optimizer: 16.50 | batch generator: 4.22 WARNING: Deleting old checkpoints: /u/wangh/workspace_ptmp/checkpoints_gptneox/flownet/160M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step124000 samples/sec: 1614.462 | iteration 128500/ 143000 | elapsed time per iteration (ms): 634.3 | learning rate: 7.413E-05 | approx flops per GPU: 62.7TFLOPS | lm_loss: 2.408878E+00 | loss scale: 4096.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 162.71 | backward: 452.87 | backward-backward: 452.83 | backward-allreduce: 0.00 | optimizer: 16.55 | batch generator: 4.47 samples/sec: 1617.471 | iteration 129000/ 143000 | elapsed time per iteration (ms): 633.1 | learning rate: 7.319E-05 | approx flops per GPU: 62.8TFLOPS | lm_loss: 2.409879E+00 | loss scale: 8192.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 162.83 | backward: 452.70 | backward-backward: 452.67 | backward-allreduce: 0.00 | optimizer: 16.55 | batch generator: 4.58 WARNING: Deleting old checkpoints: /u/wangh/workspace_ptmp/checkpoints_gptneox/flownet/160M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step125000 samples/sec: 1614.740 | iteration 129500/ 143000 | elapsed time per iteration (ms): 634.2 | learning rate: 7.228E-05 | approx flops per GPU: 62.7TFLOPS | lm_loss: 2.406976E+00 | loss scale: 8192.0 | number of skipped iterations: 1 | number of nan iterations: 0 | time (ms) | forward: 162.62 | backward: 452.84 | backward-backward: 452.80 | backward-allreduce: 0.00 | optimizer: 16.52 | batch generator: 4.33 samples/sec: 1617.705 | iteration 130000/ 143000 | elapsed time per iteration (ms): 633.0 | learning rate: 7.141E-05 | approx flops per GPU: 62.8TFLOPS | lm_loss: 2.408649E+00 | loss scale: 2048.0 | number of skipped iterations: 2 | number of nan iterations: 0 | time (ms) | forward: 162.39 | backward: 453.13 | backward-backward: 453.09 | backward-allreduce: 0.00 | optimizer: 16.47 | batch generator: 4.16 WARNING: Deleting old checkpoints: /u/wangh/workspace_ptmp/checkpoints_gptneox/flownet/160M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step126000 ----------------------------------------------------------------------------------------------------------- validation results at iteration 130000 | lm_loss value: 2.357188E+00 | lm_loss_ppl value: 1.056121E+01 | ----------------------------------------------------------------------------------------------------------- samples/sec: 1530.909 | iteration 130500/ 143000 | elapsed time per iteration (ms): 668.9 | learning rate: 7.056E-05 | approx flops per GPU: 59.4TFLOPS | lm_loss: 2.409106E+00 | loss scale: 2048.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 162.38 | backward: 453.26 | backward-backward: 453.22 | backward-allreduce: 0.00 | optimizer: 16.55 | batch generator: 4.52 samples/sec: 1616.419 | iteration 131000/ 143000 | elapsed time per iteration (ms): 633.5 | learning rate: 6.975E-05 | approx flops per GPU: 62.8TFLOPS | lm_loss: 2.408563E+00 | loss scale: 4096.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 162.33 | backward: 453.36 | backward-backward: 453.32 | backward-allreduce: 0.00 | optimizer: 16.78 | batch generator: 3.98 WARNING: Deleting old checkpoints: /u/wangh/workspace_ptmp/checkpoints_gptneox/flownet/160M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step127000 samples/sec: 1614.177 | iteration 131500/ 143000 | elapsed time per iteration (ms): 634.4 | learning rate: 6.897E-05 | approx flops per GPU: 62.7TFLOPS | lm_loss: 2.406112E+00 | loss scale: 4096.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 162.49 | backward: 452.98 | backward-backward: 452.94 | backward-allreduce: 0.00 | optimizer: 16.64 | batch generator: 4.19 samples/sec: 1617.066 | iteration 132000/ 143000 | elapsed time per iteration (ms): 633.2 | learning rate: 6.822E-05 | approx flops per GPU: 62.8TFLOPS | lm_loss: 2.406906E+00 | loss scale: 8192.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 161.90 | backward: 453.70 | backward-backward: 453.67 | backward-allreduce: 0.00 | optimizer: 16.64 | batch generator: 3.72 WARNING: Deleting old checkpoints: /u/wangh/workspace_ptmp/checkpoints_gptneox/flownet/160M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step128000 samples/sec: 1614.173 | iteration 132500/ 143000 | elapsed time per iteration (ms): 634.4 | learning rate: 6.750E-05 | approx flops per GPU: 62.7TFLOPS | lm_loss: 2.407309E+00 | loss scale: 8192.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 162.24 | backward: 453.36 | backward-backward: 453.32 | backward-allreduce: 0.00 | optimizer: 16.54 | batch generator: 3.97 samples/sec: 1617.302 | iteration 133000/ 143000 | elapsed time per iteration (ms): 633.2 | learning rate: 6.682E-05 | approx flops per GPU: 62.8TFLOPS | lm_loss: 2.405259E+00 | loss scale: 8192.0 | number of skipped iterations: 2 | number of nan iterations: 0 | time (ms) | forward: 162.22 | backward: 453.43 | backward-backward: 453.39 | backward-allreduce: 0.00 | optimizer: 16.51 | batch generator: 4.02 WARNING: Deleting old checkpoints: /u/wangh/workspace_ptmp/checkpoints_gptneox/flownet/160M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step129000 samples/sec: 1614.488 | iteration 133500/ 143000 | elapsed time per iteration (ms): 634.3 | learning rate: 6.617E-05 | approx flops per GPU: 62.7TFLOPS | lm_loss: 2.405187E+00 | loss scale: 4096.0 | number of skipped iterations: 1 | number of nan iterations: 0 | time (ms) | forward: 161.33 | backward: 454.13 | backward-backward: 454.10 | backward-allreduce: 0.00 | optimizer: 16.54 | batch generator: 3.08 samples/sec: 1617.159 | iteration 134000/ 143000 | elapsed time per iteration (ms): 633.2 | learning rate: 6.555E-05 | approx flops per GPU: 62.8TFLOPS | lm_loss: 2.404214E+00 | loss scale: 4096.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 161.82 | backward: 453.80 | backward-backward: 453.76 | backward-allreduce: 0.00 | optimizer: 16.58 | batch generator: 3.62 WARNING: Deleting old checkpoints: /u/wangh/workspace_ptmp/checkpoints_gptneox/flownet/160M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step130000 samples/sec: 1614.146 | iteration 134500/ 143000 | elapsed time per iteration (ms): 634.4 | learning rate: 6.496E-05 | approx flops per GPU: 62.7TFLOPS | lm_loss: 2.404666E+00 | loss scale: 8192.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 162.42 | backward: 453.19 | backward-backward: 453.16 | backward-allreduce: 0.00 | optimizer: 16.57 | batch generator: 4.26 samples/sec: 1617.144 | iteration 135000/ 143000 | elapsed time per iteration (ms): 633.2 | learning rate: 6.440E-05 | approx flops per GPU: 62.8TFLOPS | lm_loss: 2.404946E+00 | loss scale: 8192.0 | number of skipped iterations: 1 | number of nan iterations: 0 | time (ms) | forward: 162.76 | backward: 452.95 | backward-backward: 452.91 | backward-allreduce: 0.00 | optimizer: 16.52 | batch generator: 4.57 WARNING: Deleting old checkpoints: /u/wangh/workspace_ptmp/checkpoints_gptneox/flownet/160M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step131000 samples/sec: 1603.098 | iteration 135500/ 143000 | elapsed time per iteration (ms): 638.8 | learning rate: 6.388E-05 | approx flops per GPU: 62.2TFLOPS | lm_loss: 2.403901E+00 | loss scale: 8192.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 163.03 | backward: 456.93 | backward-backward: 456.89 | backward-allreduce: 0.00 | optimizer: 16.54 | batch generator: 4.77 samples/sec: 1617.255 | iteration 136000/ 143000 | elapsed time per iteration (ms): 633.2 | learning rate: 6.339E-05 | approx flops per GPU: 62.8TFLOPS | lm_loss: 2.406157E+00 | loss scale: 4096.0 | number of skipped iterations: 1 | number of nan iterations: 0 | time (ms) | forward: 162.78 | backward: 452.86 | backward-backward: 452.83 | backward-allreduce: 0.00 | optimizer: 16.52 | batch generator: 4.43 WARNING: Deleting old checkpoints: /u/wangh/workspace_ptmp/checkpoints_gptneox/flownet/160M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step132000 samples/sec: 1609.169 | iteration 136500/ 143000 | elapsed time per iteration (ms): 636.4 | learning rate: 6.294E-05 | approx flops per GPU: 62.5TFLOPS | lm_loss: 2.403617E+00 | loss scale: 2048.0 | number of skipped iterations: 1 | number of nan iterations: 0 | time (ms) | forward: 163.19 | backward: 454.52 | backward-backward: 454.48 | backward-allreduce: 0.00 | optimizer: 16.50 | batch generator: 4.92 samples/sec: 1617.568 | iteration 137000/ 143000 | elapsed time per iteration (ms): 633.0 | learning rate: 6.251E-05 | approx flops per GPU: 62.8TFLOPS | lm_loss: 2.406209E+00 | loss scale: 2048.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 162.32 | backward: 453.20 | backward-backward: 453.16 | backward-allreduce: 0.00 | optimizer: 16.54 | batch generator: 4.06 WARNING: Deleting old checkpoints: /u/wangh/workspace_ptmp/checkpoints_gptneox/flownet/160M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step133000 samples/sec: 1614.649 | iteration 137500/ 143000 | elapsed time per iteration (ms): 634.2 | learning rate: 6.212E-05 | approx flops per GPU: 62.7TFLOPS | lm_loss: 2.403958E+00 | loss scale: 4096.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 161.86 | backward: 453.63 | backward-backward: 453.59 | backward-allreduce: 0.00 | optimizer: 16.50 | batch generator: 3.63 samples/sec: 1617.910 | iteration 138000/ 143000 | elapsed time per iteration (ms): 632.9 | learning rate: 6.176E-05 | approx flops per GPU: 62.8TFLOPS | lm_loss: 2.403215E+00 | loss scale: 4096.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 162.03 | backward: 453.39 | backward-backward: 453.35 | backward-allreduce: 0.00 | optimizer: 16.52 | batch generator: 3.85 WARNING: Deleting old checkpoints: /u/wangh/workspace_ptmp/checkpoints_gptneox/flownet/160M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step134000 samples/sec: 1614.333 | iteration 138500/ 143000 | elapsed time per iteration (ms): 634.3 | learning rate: 6.144E-05 | approx flops per GPU: 62.7TFLOPS | lm_loss: 2.405326E+00 | loss scale: 8192.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 162.61 | backward: 452.98 | backward-backward: 452.94 | backward-allreduce: 0.00 | optimizer: 16.55 | batch generator: 4.44 samples/sec: 1617.620 | iteration 139000/ 143000 | elapsed time per iteration (ms): 633.0 | learning rate: 6.115E-05 | approx flops per GPU: 62.8TFLOPS | lm_loss: 2.403896E+00 | loss scale: 8192.0 | number of skipped iterations: 1 | number of nan iterations: 0 | time (ms) | forward: 162.57 | backward: 452.85 | backward-backward: 452.82 | backward-allreduce: 0.00 | optimizer: 16.55 | batch generator: 4.30 WARNING: Deleting old checkpoints: /u/wangh/workspace_ptmp/checkpoints_gptneox/flownet/160M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step135000 samples/sec: 1614.748 | iteration 139500/ 143000 | elapsed time per iteration (ms): 634.2 | learning rate: 6.089E-05 | approx flops per GPU: 62.7TFLOPS | lm_loss: 2.402382E+00 | loss scale: 8192.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 162.36 | backward: 453.05 | backward-backward: 453.02 | backward-allreduce: 0.00 | optimizer: 16.56 | batch generator: 4.21 samples/sec: 1617.304 | iteration 140000/ 143000 | elapsed time per iteration (ms): 633.2 | learning rate: 6.066E-05 | approx flops per GPU: 62.8TFLOPS | lm_loss: 2.401437E+00 | loss scale: 16384.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 162.48 | backward: 453.09 | backward-backward: 453.06 | backward-allreduce: 0.00 | optimizer: 16.57 | batch generator: 4.16 WARNING: Deleting old checkpoints: /u/wangh/workspace_ptmp/checkpoints_gptneox/flownet/160M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step136000 ----------------------------------------------------------------------------------------------------------- validation results at iteration 140000 | lm_loss value: 2.353743E+00 | lm_loss_ppl value: 1.052489E+01 | ----------------------------------------------------------------------------------------------------------- samples/sec: 1532.005 | iteration 140500/ 143000 | elapsed time per iteration (ms): 668.4 | learning rate: 6.047E-05 | approx flops per GPU: 59.5TFLOPS | lm_loss: 2.403078E+00 | loss scale: 2048.0 | number of skipped iterations: 4 | number of nan iterations: 0 | time (ms) | forward: 161.44 | backward: 454.12 | backward-backward: 454.08 | backward-allreduce: 0.00 | optimizer: 16.50 | batch generator: 3.52 samples/sec: 1617.070 | iteration 141000/ 143000 | elapsed time per iteration (ms): 633.2 | learning rate: 6.031E-05 | approx flops per GPU: 62.8TFLOPS | lm_loss: 2.403725E+00 | loss scale: 2048.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 162.80 | backward: 452.89 | backward-backward: 452.86 | backward-allreduce: 0.00 | optimizer: 16.54 | batch generator: 4.54 WARNING: Deleting old checkpoints: /u/wangh/workspace_ptmp/checkpoints_gptneox/flownet/160M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step137000 samples/sec: 1614.153 | iteration 141500/ 143000 | elapsed time per iteration (ms): 634.4 | learning rate: 6.018E-05 | approx flops per GPU: 62.7TFLOPS | lm_loss: 2.402636E+00 | loss scale: 4096.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 162.63 | backward: 452.98 | backward-backward: 452.95 | backward-allreduce: 0.00 | optimizer: 16.57 | batch generator: 4.37 samples/sec: 1617.216 | iteration 142000/ 143000 | elapsed time per iteration (ms): 633.2 | learning rate: 6.009E-05 | approx flops per GPU: 62.8TFLOPS | lm_loss: 2.404379E+00 | loss scale: 4096.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 161.18 | backward: 454.06 | backward-backward: 454.02 | backward-allreduce: 0.00 | optimizer: 16.61 | batch generator: 3.02 WARNING: Deleting old checkpoints: /u/wangh/workspace_ptmp/checkpoints_gptneox/flownet/160M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step138000 samples/sec: 1614.922 | iteration 142500/ 143000 | elapsed time per iteration (ms): 634.1 | learning rate: 6.003E-05 | approx flops per GPU: 62.7TFLOPS | lm_loss: 2.402002E+00 | loss scale: 4096.0 | number of skipped iterations: 2 | number of nan iterations: 0 | time (ms) | forward: 161.34 | backward: 453.94 | backward-backward: 453.90 | backward-allreduce: 0.00 | optimizer: 16.51 | batch generator: 3.11 samples/sec: 1618.028 | iteration 143000/ 143000 | elapsed time per iteration (ms): 632.9 | learning rate: 6.000E-05 | approx flops per GPU: 62.8TFLOPS | lm_loss: 2.403000E+00 | loss scale: 1024.0 | number of skipped iterations: 2 | number of nan iterations: 0 | time (ms) | forward: 161.66 | backward: 453.64 | backward-backward: 453.61 | backward-allreduce: 0.00 | optimizer: 16.52 | batch generator: 3.41 --------------------------------------------------------------------------------------------------------------------------- validation results at the end of training for val data | lm_loss value: 2.351247E+00 | lm_loss_ppl value: 1.049865E+01 | --------------------------------------------------------------------------------------------------------------------------- WARNING: Deleting old checkpoints: /u/wangh/workspace_ptmp/checkpoints_gptneox/flownet/160M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step139000 ---------------------------------------------------------------------------------------------------------------------- test results at the end of training for test data | lm_loss value: 2.346909E+00 | lm_loss_ppl value: 1.045320E+01 | ----------------------------------------------------------------------------------------------------------------------