2024-09-17 12:04:10,722 INFO StreamThr :1253420 [internal.py:wandb_internal():85] W&B internal server running at pid: 1253420, started at: 2024-09-17 12:04:10.719948 2024-09-17 12:04:10,726 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: status 2024-09-17 12:04:10,740 INFO WriterThread:1253420 [datastore.py:open_for_write():87] open: /nfs/production/literature/amina-mardiyyah/NER/ewc_stabilised_no_date/wandb/run-20240917_120410-nm6gjq7r/run-nm6gjq7r.wandb 2024-09-17 12:04:10,747 DEBUG SenderThread:1253420 [sender.py:send():391] send: header 2024-09-17 12:04:10,748 DEBUG SenderThread:1253420 [sender.py:send():391] send: run 2024-09-17 12:04:11,130 INFO SenderThread:1253420 [dir_watcher.py:__init__():211] watching files in: /nfs/production/literature/amina-mardiyyah/NER/ewc_stabilised_no_date/wandb/run-20240917_120410-nm6gjq7r/files 2024-09-17 12:04:11,130 INFO SenderThread:1253420 [sender.py:_start_run_threads():1200] run started: nm6gjq7r with start time 1726571050.72261 2024-09-17 12:04:11,138 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: run_start 2024-09-17 12:04:11,244 DEBUG HandlerThread:1253420 [system_info.py:__init__():26] System info init 2024-09-17 12:04:11,244 DEBUG HandlerThread:1253420 [system_info.py:__init__():41] System info init done 2024-09-17 12:04:11,244 INFO HandlerThread:1253420 [system_monitor.py:start():194] Starting system monitor 2024-09-17 12:04:11,245 INFO SystemMonitor:1253420 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-09-17 12:04:11,245 INFO HandlerThread:1253420 [system_monitor.py:probe():214] Collecting system info 2024-09-17 12:04:11,246 INFO SystemMonitor:1253420 [interfaces.py:start():188] Started cpu monitoring 2024-09-17 12:04:11,250 INFO SystemMonitor:1253420 [interfaces.py:start():188] Started disk monitoring 2024-09-17 12:04:11,252 INFO SystemMonitor:1253420 [interfaces.py:start():188] Started gpu monitoring 2024-09-17 12:04:11,258 INFO SystemMonitor:1253420 [interfaces.py:start():188] Started memory monitoring 2024-09-17 12:04:11,261 INFO SystemMonitor:1253420 [interfaces.py:start():188] Started network monitoring 2024-09-17 12:04:11,273 DEBUG HandlerThread:1253420 [system_info.py:probe():152] Probing system 2024-09-17 12:04:11,277 DEBUG HandlerThread:1253420 [gitlib.py:_init_repo():56] git repository is invalid 2024-09-17 12:04:11,277 DEBUG HandlerThread:1253420 [system_info.py:probe():200] Probing system done 2024-09-17 12:04:11,277 DEBUG HandlerThread:1253420 [system_monitor.py:probe():223] {'os': 'Linux-4.18.0-513.24.1.el8_9.x86_64-x86_64-with-glibc2.28', 'python': '3.11.4', 'heartbeatAt': '2024-09-17T11:04:11.273502', 'startedAt': '2024-09-17T11:04:10.689415', 'docker': None, 'cuda': None, 'args': (), 'state': 'running', 'program': '', 'codePathLocal': None, 'host': 'codon-gpu-014.ebi.ac.uk', 'username': 'amrufai', 'executable': '/nfs/production/literature/amina-mardiyyah/envs/llm-prompt/bin/python', 'cpu_count': 48, 'cpu_count_logical': 48, 'cpu_freq': {'current': 2728.292645833333, 'min': 0.0, 'max': 0.0}, 'cpu_freq_per_core': [{'current': 2800.0, 'min': 0.0, 'max': 0.0}, {'current': 2800.0, 'min': 0.0, 'max': 0.0}, {'current': 2800.0, 'min': 0.0, 'max': 0.0}, {'current': 2800.0, 'min': 0.0, 'max': 0.0}, {'current': 2800.0, 'min': 0.0, 'max': 0.0}, {'current': 2800.0, 'min': 0.0, 'max': 0.0}, {'current': 1182.367, 'min': 0.0, 'max': 0.0}, {'current': 2800.0, 'min': 0.0, 'max': 0.0}, {'current': 2800.0, 'min': 0.0, 'max': 0.0}, {'current': 2800.0, 'min': 0.0, 'max': 0.0}, {'current': 2800.0, 'min': 0.0, 'max': 0.0}, {'current': 2800.0, 'min': 0.0, 'max': 0.0}, {'current': 2800.0, 'min': 0.0, 'max': 0.0}, {'current': 2800.0, 'min': 0.0, 'max': 0.0}, {'current': 2800.0, 'min': 0.0, 'max': 0.0}, {'current': 2800.0, 'min': 0.0, 'max': 0.0}, {'current': 2800.0, 'min': 0.0, 'max': 0.0}, {'current': 2800.0, 'min': 0.0, 'max': 0.0}, {'current': 2800.0, 'min': 0.0, 'max': 0.0}, {'current': 2800.0, 'min': 0.0, 'max': 0.0}, {'current': 975.68, 'min': 0.0, 'max': 0.0}, {'current': 2800.0, 'min': 0.0, 'max': 0.0}, {'current': 2800.0, 'min': 0.0, 'max': 0.0}, {'current': 2800.0, 'min': 0.0, 'max': 0.0}, {'current': 2800.0, 'min': 0.0, 'max': 0.0}, {'current': 2800.0, 'min': 0.0, 'max': 0.0}, {'current': 2800.0, 'min': 0.0, 'max': 0.0}, {'current': 2800.0, 'min': 0.0, 'max': 0.0}, {'current': 2800.0, 'min': 0.0, 'max': 0.0}, {'current': 2800.0, 'min': 0.0, 'max': 0.0}, {'current': 2800.0, 'min': 0.0, 'max': 0.0}, {'current': 2800.0, 'min': 0.0, 'max': 0.0}, {'current': 2800.0, 'min': 0.0, 'max': 0.0}, {'current': 2800.0, 'min': 0.0, 'max': 0.0}, {'current': 2800.0, 'min': 0.0, 'max': 0.0}, {'current': 2800.0, 'min': 0.0, 'max': 0.0}, {'current': 2800.0, 'min': 0.0, 'max': 0.0}, {'current': 2800.0, 'min': 0.0, 'max': 0.0}, {'current': 2800.0, 'min': 0.0, 'max': 0.0}, {'current': 2800.0, 'min': 0.0, 'max': 0.0}, {'current': 2800.0, 'min': 0.0, 'max': 0.0}, {'current': 2800.0, 'min': 0.0, 'max': 0.0}, {'current': 2800.0, 'min': 0.0, 'max': 0.0}, {'current': 2800.0, 'min': 0.0, 'max': 0.0}, {'current': 2800.0, 'min': 0.0, 'max': 0.0}, {'current': 2800.0, 'min': 0.0, 'max': 0.0}, {'current': 2800.0, 'min': 0.0, 'max': 0.0}, {'current': 2800.0, 'min': 0.0, 'max': 0.0}], 'disk': {'/': {'total': 47.760292053222656, 'used': 15.848091125488281}}, 'gpu': 'NVIDIA A100 80GB PCIe', 'gpu_count': 1, 'gpu_devices': [{'name': 'NVIDIA A100 80GB PCIe', 'memory_total': 85899345920}], 'memory': {'total': 502.83758544921875}} 2024-09-17 12:04:11,277 INFO HandlerThread:1253420 [system_monitor.py:probe():224] Finished collecting system info 2024-09-17 12:04:11,277 INFO HandlerThread:1253420 [system_monitor.py:probe():227] Publishing system info 2024-09-17 12:04:11,281 INFO HandlerThread:1253420 [system_monitor.py:probe():229] Finished publishing system info 2024-09-17 12:04:11,322 DEBUG SenderThread:1253420 [sender.py:send():391] send: files 2024-09-17 12:04:11,323 INFO SenderThread:1253420 [sender.py:_save_file():1466] saving file wandb-metadata.json with policy now 2024-09-17 12:04:11,920 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: python_packages 2024-09-17 12:04:11,921 DEBUG SenderThread:1253420 [sender.py:send_request():418] send_request: python_packages 2024-09-17 12:04:11,924 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: internal_messages 2024-09-17 12:04:11,925 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: stop_status 2024-09-17 12:04:11,926 DEBUG SenderThread:1253420 [sender.py:send():391] send: telemetry 2024-09-17 12:04:11,927 DEBUG SenderThread:1253420 [sender.py:send_request():418] send_request: stop_status 2024-09-17 12:04:11,938 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: pause 2024-09-17 12:04:11,938 INFO HandlerThread:1253420 [handler.py:handle_request_pause():728] stopping system metrics thread 2024-09-17 12:04:11,938 INFO HandlerThread:1253420 [system_monitor.py:finish():203] Stopping system monitor 2024-09-17 12:04:11,939 INFO HandlerThread:1253420 [interfaces.py:finish():200] Joined cpu monitor 2024-09-17 12:04:11,940 INFO HandlerThread:1253420 [interfaces.py:finish():200] Joined disk monitor 2024-09-17 12:04:11,942 INFO wandb-upload_0:1253420 [upload_job.py:push():130] Uploaded file /tmp/tmp39_l9k7owandb/2kmhtdvh-wandb-metadata.json 2024-09-17 12:04:11,943 DEBUG SystemMonitor:1253420 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-09-17 12:04:11,944 DEBUG SystemMonitor:1253420 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2024-09-17 12:04:11,944 DEBUG SystemMonitor:1253420 [system_monitor.py:_start():183] Publishing last batch of metrics 2024-09-17 12:04:12,004 INFO HandlerThread:1253420 [interfaces.py:finish():200] Joined gpu monitor 2024-09-17 12:04:12,005 INFO HandlerThread:1253420 [interfaces.py:finish():200] Joined memory monitor 2024-09-17 12:04:12,005 INFO HandlerThread:1253420 [interfaces.py:finish():200] Joined network monitor 2024-09-17 12:04:12,134 INFO Thread-12 :1253420 [dir_watcher.py:_on_file_created():271] file/dir created: /nfs/production/literature/amina-mardiyyah/NER/ewc_stabilised_no_date/wandb/run-20240917_120410-nm6gjq7r/files/wandb-metadata.json 2024-09-17 12:04:12,135 INFO Thread-12 :1253420 [dir_watcher.py:_on_file_created():271] file/dir created: /nfs/production/literature/amina-mardiyyah/NER/ewc_stabilised_no_date/wandb/run-20240917_120410-nm6gjq7r/files/requirements.txt 2024-09-17 12:04:12,156 DEBUG SenderThread:1253420 [sender.py:send():391] send: stats 2024-09-17 12:04:16,158 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: status_report 2024-09-17 12:04:21,923 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: status_report 2024-09-17 12:04:21,924 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: internal_messages 2024-09-17 12:04:26,921 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: stop_status 2024-09-17 12:04:26,922 DEBUG SenderThread:1253420 [sender.py:send_request():418] send_request: stop_status 2024-09-17 12:04:27,156 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: status_report 2024-09-17 12:04:31,923 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: internal_messages 2024-09-17 12:04:32,925 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: status_report 2024-09-17 12:04:38,924 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: status_report 2024-09-17 12:04:41,921 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: stop_status 2024-09-17 12:04:41,922 DEBUG SenderThread:1253420 [sender.py:send_request():418] send_request: stop_status 2024-09-17 12:04:41,962 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: internal_messages 2024-09-17 12:04:44,140 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: status_report 2024-09-17 12:04:45,171 INFO Thread-12 :1253420 [dir_watcher.py:_on_file_modified():288] file/dir modified: /nfs/production/literature/amina-mardiyyah/NER/ewc_stabilised_no_date/wandb/run-20240917_120410-nm6gjq7r/files/config.yaml 2024-09-17 12:04:47,842 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: resume 2024-09-17 12:04:47,842 INFO HandlerThread:1253420 [handler.py:handle_request_resume():719] starting system metrics thread 2024-09-17 12:04:47,842 INFO HandlerThread:1253420 [system_monitor.py:start():194] Starting system monitor 2024-09-17 12:04:47,842 INFO SystemMonitor:1253420 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-09-17 12:04:47,843 INFO SystemMonitor:1253420 [interfaces.py:start():188] Started cpu monitoring 2024-09-17 12:04:47,844 INFO SystemMonitor:1253420 [interfaces.py:start():188] Started disk monitoring 2024-09-17 12:04:47,846 INFO SystemMonitor:1253420 [interfaces.py:start():188] Started gpu monitoring 2024-09-17 12:04:47,849 INFO SystemMonitor:1253420 [interfaces.py:start():188] Started memory monitoring 2024-09-17 12:04:47,853 INFO SystemMonitor:1253420 [interfaces.py:start():188] Started network monitoring 2024-09-17 12:04:47,893 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: pause 2024-09-17 12:04:47,893 INFO HandlerThread:1253420 [handler.py:handle_request_pause():728] stopping system metrics thread 2024-09-17 12:04:47,894 INFO HandlerThread:1253420 [system_monitor.py:finish():203] Stopping system monitor 2024-09-17 12:04:47,894 DEBUG SystemMonitor:1253420 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-09-17 12:04:47,896 DEBUG SystemMonitor:1253420 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2024-09-17 12:04:47,896 INFO HandlerThread:1253420 [interfaces.py:finish():200] Joined cpu monitor 2024-09-17 12:04:47,896 DEBUG SystemMonitor:1253420 [system_monitor.py:_start():183] Publishing last batch of metrics 2024-09-17 12:04:47,897 INFO HandlerThread:1253420 [interfaces.py:finish():200] Joined disk monitor 2024-09-17 12:04:47,957 INFO HandlerThread:1253420 [interfaces.py:finish():200] Joined gpu monitor 2024-09-17 12:04:47,957 INFO HandlerThread:1253420 [interfaces.py:finish():200] Joined memory monitor 2024-09-17 12:04:47,958 INFO HandlerThread:1253420 [interfaces.py:finish():200] Joined network monitor 2024-09-17 12:04:47,959 DEBUG SenderThread:1253420 [sender.py:send():391] send: stats 2024-09-17 12:04:48,687 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: resume 2024-09-17 12:04:48,687 INFO HandlerThread:1253420 [handler.py:handle_request_resume():719] starting system metrics thread 2024-09-17 12:04:48,687 INFO HandlerThread:1253420 [system_monitor.py:start():194] Starting system monitor 2024-09-17 12:04:48,688 INFO SystemMonitor:1253420 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-09-17 12:04:48,689 INFO SystemMonitor:1253420 [interfaces.py:start():188] Started cpu monitoring 2024-09-17 12:04:48,691 INFO SystemMonitor:1253420 [interfaces.py:start():188] Started disk monitoring 2024-09-17 12:04:48,695 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: pause 2024-09-17 12:04:48,696 INFO HandlerThread:1253420 [handler.py:handle_request_pause():728] stopping system metrics thread 2024-09-17 12:04:48,696 INFO HandlerThread:1253420 [system_monitor.py:finish():203] Stopping system monitor 2024-09-17 12:04:48,697 INFO SystemMonitor:1253420 [interfaces.py:start():188] Started gpu monitoring 2024-09-17 12:04:48,697 DEBUG SystemMonitor:1253420 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-09-17 12:04:48,697 DEBUG SystemMonitor:1253420 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2024-09-17 12:04:48,697 DEBUG SystemMonitor:1253420 [system_monitor.py:_start():183] Publishing last batch of metrics 2024-09-17 12:04:48,699 INFO HandlerThread:1253420 [interfaces.py:finish():200] Joined cpu monitor 2024-09-17 12:04:48,701 INFO HandlerThread:1253420 [interfaces.py:finish():200] Joined disk monitor 2024-09-17 12:04:48,765 INFO HandlerThread:1253420 [interfaces.py:finish():200] Joined gpu monitor 2024-09-17 12:04:48,767 DEBUG SenderThread:1253420 [sender.py:send():391] send: stats 2024-09-17 12:04:49,768 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: status_report 2024-09-17 12:04:51,924 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: internal_messages 2024-09-17 12:04:54,926 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: status_report 2024-09-17 12:04:56,921 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: stop_status 2024-09-17 12:04:56,922 DEBUG SenderThread:1253420 [sender.py:send_request():418] send_request: stop_status 2024-09-17 12:05:00,082 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: status_report 2024-09-17 12:05:01,924 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: internal_messages 2024-09-17 12:05:05,926 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: status_report 2024-09-17 12:05:07,802 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: resume 2024-09-17 12:05:07,802 INFO HandlerThread:1253420 [handler.py:handle_request_resume():719] starting system metrics thread 2024-09-17 12:05:07,802 INFO HandlerThread:1253420 [system_monitor.py:start():194] Starting system monitor 2024-09-17 12:05:07,803 INFO SystemMonitor:1253420 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-09-17 12:05:07,804 INFO SystemMonitor:1253420 [interfaces.py:start():188] Started cpu monitoring 2024-09-17 12:05:07,807 INFO SystemMonitor:1253420 [interfaces.py:start():188] Started disk monitoring 2024-09-17 12:05:07,809 INFO SystemMonitor:1253420 [interfaces.py:start():188] Started gpu monitoring 2024-09-17 12:05:07,810 INFO SystemMonitor:1253420 [interfaces.py:start():188] Started memory monitoring 2024-09-17 12:05:07,811 INFO SystemMonitor:1253420 [interfaces.py:start():188] Started network monitoring 2024-09-17 12:05:07,843 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: pause 2024-09-17 12:05:07,843 INFO HandlerThread:1253420 [handler.py:handle_request_pause():728] stopping system metrics thread 2024-09-17 12:05:07,843 INFO HandlerThread:1253420 [system_monitor.py:finish():203] Stopping system monitor 2024-09-17 12:05:07,844 INFO HandlerThread:1253420 [interfaces.py:finish():200] Joined cpu monitor 2024-09-17 12:05:07,844 DEBUG SystemMonitor:1253420 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-09-17 12:05:07,845 DEBUG SystemMonitor:1253420 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2024-09-17 12:05:07,845 DEBUG SystemMonitor:1253420 [system_monitor.py:_start():183] Publishing last batch of metrics 2024-09-17 12:05:07,848 INFO HandlerThread:1253420 [interfaces.py:finish():200] Joined disk monitor 2024-09-17 12:05:07,906 INFO HandlerThread:1253420 [interfaces.py:finish():200] Joined gpu monitor 2024-09-17 12:05:07,906 INFO HandlerThread:1253420 [interfaces.py:finish():200] Joined memory monitor 2024-09-17 12:05:07,907 INFO HandlerThread:1253420 [interfaces.py:finish():200] Joined network monitor 2024-09-17 12:05:07,908 DEBUG SenderThread:1253420 [sender.py:send():391] send: stats 2024-09-17 12:05:11,910 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: status_report 2024-09-17 12:05:11,922 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: stop_status 2024-09-17 12:05:11,922 DEBUG SenderThread:1253420 [sender.py:send_request():418] send_request: stop_status 2024-09-17 12:05:11,928 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: internal_messages 2024-09-17 12:05:16,925 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: status_report 2024-09-17 12:05:21,925 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: status_report 2024-09-17 12:05:21,926 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: internal_messages 2024-09-17 12:05:26,922 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: stop_status 2024-09-17 12:05:26,923 DEBUG SenderThread:1253420 [sender.py:send_request():418] send_request: stop_status 2024-09-17 12:05:27,086 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: status_report 2024-09-17 12:05:31,926 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: internal_messages 2024-09-17 12:05:32,925 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: status_report 2024-09-17 12:05:38,926 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: status_report 2024-09-17 12:05:41,922 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: stop_status 2024-09-17 12:05:41,923 DEBUG SenderThread:1253420 [sender.py:send_request():418] send_request: stop_status 2024-09-17 12:05:41,963 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: internal_messages 2024-09-17 12:05:44,139 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: status_report 2024-09-17 12:05:49,926 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: status_report 2024-09-17 12:05:50,657 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: resume 2024-09-17 12:05:50,657 INFO HandlerThread:1253420 [handler.py:handle_request_resume():719] starting system metrics thread 2024-09-17 12:05:50,657 INFO HandlerThread:1253420 [system_monitor.py:start():194] Starting system monitor 2024-09-17 12:05:50,658 INFO SystemMonitor:1253420 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-09-17 12:05:50,659 INFO SystemMonitor:1253420 [interfaces.py:start():188] Started cpu monitoring 2024-09-17 12:05:50,661 INFO SystemMonitor:1253420 [interfaces.py:start():188] Started disk monitoring 2024-09-17 12:05:50,665 INFO SystemMonitor:1253420 [interfaces.py:start():188] Started gpu monitoring 2024-09-17 12:05:50,666 INFO SystemMonitor:1253420 [interfaces.py:start():188] Started memory monitoring 2024-09-17 12:05:50,667 INFO SystemMonitor:1253420 [interfaces.py:start():188] Started network monitoring 2024-09-17 12:05:50,698 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: pause 2024-09-17 12:05:50,698 INFO HandlerThread:1253420 [handler.py:handle_request_pause():728] stopping system metrics thread 2024-09-17 12:05:50,698 INFO HandlerThread:1253420 [system_monitor.py:finish():203] Stopping system monitor 2024-09-17 12:05:50,700 INFO HandlerThread:1253420 [interfaces.py:finish():200] Joined cpu monitor 2024-09-17 12:05:50,700 INFO HandlerThread:1253420 [interfaces.py:finish():200] Joined disk monitor 2024-09-17 12:05:50,701 DEBUG SystemMonitor:1253420 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-09-17 12:05:50,701 DEBUG SystemMonitor:1253420 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2024-09-17 12:05:50,701 DEBUG SystemMonitor:1253420 [system_monitor.py:_start():183] Publishing last batch of metrics 2024-09-17 12:05:50,760 INFO HandlerThread:1253420 [interfaces.py:finish():200] Joined gpu monitor 2024-09-17 12:05:50,761 INFO HandlerThread:1253420 [interfaces.py:finish():200] Joined memory monitor 2024-09-17 12:05:50,761 INFO HandlerThread:1253420 [interfaces.py:finish():200] Joined network monitor 2024-09-17 12:05:50,762 DEBUG SenderThread:1253420 [sender.py:send():391] send: stats 2024-09-17 12:05:51,925 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: internal_messages 2024-09-17 12:05:53,407 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: resume 2024-09-17 12:05:53,407 INFO HandlerThread:1253420 [handler.py:handle_request_resume():719] starting system metrics thread 2024-09-17 12:05:53,407 INFO HandlerThread:1253420 [system_monitor.py:start():194] Starting system monitor 2024-09-17 12:05:53,408 INFO SystemMonitor:1253420 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-09-17 12:05:53,409 INFO SystemMonitor:1253420 [interfaces.py:start():188] Started cpu monitoring 2024-09-17 12:05:53,411 INFO SystemMonitor:1253420 [interfaces.py:start():188] Started disk monitoring 2024-09-17 12:05:53,413 INFO SystemMonitor:1253420 [interfaces.py:start():188] Started gpu monitoring 2024-09-17 12:05:53,415 INFO SystemMonitor:1253420 [interfaces.py:start():188] Started memory monitoring 2024-09-17 12:05:53,417 INFO SystemMonitor:1253420 [interfaces.py:start():188] Started network monitoring 2024-09-17 12:05:53,448 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: pause 2024-09-17 12:05:53,448 INFO HandlerThread:1253420 [handler.py:handle_request_pause():728] stopping system metrics thread 2024-09-17 12:05:53,448 INFO HandlerThread:1253420 [system_monitor.py:finish():203] Stopping system monitor 2024-09-17 12:05:53,449 INFO HandlerThread:1253420 [interfaces.py:finish():200] Joined cpu monitor 2024-09-17 12:05:53,450 INFO HandlerThread:1253420 [interfaces.py:finish():200] Joined disk monitor 2024-09-17 12:05:53,451 DEBUG SystemMonitor:1253420 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-09-17 12:05:53,451 DEBUG SystemMonitor:1253420 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2024-09-17 12:05:53,451 DEBUG SystemMonitor:1253420 [system_monitor.py:_start():183] Publishing last batch of metrics 2024-09-17 12:05:53,510 INFO HandlerThread:1253420 [interfaces.py:finish():200] Joined gpu monitor 2024-09-17 12:05:53,510 INFO HandlerThread:1253420 [interfaces.py:finish():200] Joined memory monitor 2024-09-17 12:05:53,511 INFO HandlerThread:1253420 [interfaces.py:finish():200] Joined network monitor 2024-09-17 12:05:53,512 DEBUG SenderThread:1253420 [sender.py:send():391] send: stats 2024-09-17 12:05:54,529 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: resume 2024-09-17 12:05:54,529 INFO HandlerThread:1253420 [handler.py:handle_request_resume():719] starting system metrics thread 2024-09-17 12:05:54,529 INFO HandlerThread:1253420 [system_monitor.py:start():194] Starting system monitor 2024-09-17 12:05:54,530 INFO SystemMonitor:1253420 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-09-17 12:05:54,531 INFO SystemMonitor:1253420 [interfaces.py:start():188] Started cpu monitoring 2024-09-17 12:05:54,534 INFO SystemMonitor:1253420 [interfaces.py:start():188] Started disk monitoring 2024-09-17 12:05:54,535 INFO SystemMonitor:1253420 [interfaces.py:start():188] Started gpu monitoring 2024-09-17 12:05:54,537 INFO SystemMonitor:1253420 [interfaces.py:start():188] Started memory monitoring 2024-09-17 12:05:54,538 INFO SystemMonitor:1253420 [interfaces.py:start():188] Started network monitoring 2024-09-17 12:05:54,702 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: pause 2024-09-17 12:05:54,702 INFO HandlerThread:1253420 [handler.py:handle_request_pause():728] stopping system metrics thread 2024-09-17 12:05:54,702 INFO HandlerThread:1253420 [system_monitor.py:finish():203] Stopping system monitor 2024-09-17 12:05:54,703 DEBUG SystemMonitor:1253420 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-09-17 12:05:54,703 DEBUG SystemMonitor:1253420 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2024-09-17 12:05:54,703 DEBUG SystemMonitor:1253420 [system_monitor.py:_start():183] Publishing last batch of metrics 2024-09-17 12:05:54,706 INFO HandlerThread:1253420 [interfaces.py:finish():200] Joined cpu monitor 2024-09-17 12:05:54,706 INFO HandlerThread:1253420 [interfaces.py:finish():200] Joined disk monitor 2024-09-17 12:05:54,763 INFO HandlerThread:1253420 [interfaces.py:finish():200] Joined gpu monitor 2024-09-17 12:05:54,764 INFO HandlerThread:1253420 [interfaces.py:finish():200] Joined memory monitor 2024-09-17 12:05:54,764 INFO HandlerThread:1253420 [interfaces.py:finish():200] Joined network monitor 2024-09-17 12:05:54,765 DEBUG SenderThread:1253420 [sender.py:send():391] send: stats 2024-09-17 12:05:55,767 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: status_report 2024-09-17 12:05:56,922 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: stop_status 2024-09-17 12:05:56,923 DEBUG SenderThread:1253420 [sender.py:send_request():418] send_request: stop_status 2024-09-17 12:06:01,129 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: status_report 2024-09-17 12:06:01,926 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: internal_messages 2024-09-17 12:06:04,941 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: resume 2024-09-17 12:06:04,942 INFO HandlerThread:1253420 [handler.py:handle_request_resume():719] starting system metrics thread 2024-09-17 12:06:04,942 INFO HandlerThread:1253420 [system_monitor.py:start():194] Starting system monitor 2024-09-17 12:06:04,942 INFO SystemMonitor:1253420 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-09-17 12:06:04,944 INFO SystemMonitor:1253420 [interfaces.py:start():188] Started cpu monitoring 2024-09-17 12:06:04,946 INFO SystemMonitor:1253420 [interfaces.py:start():188] Started disk monitoring 2024-09-17 12:06:04,947 INFO SystemMonitor:1253420 [interfaces.py:start():188] Started gpu monitoring 2024-09-17 12:06:04,950 INFO SystemMonitor:1253420 [interfaces.py:start():188] Started memory monitoring 2024-09-17 12:06:04,952 INFO SystemMonitor:1253420 [interfaces.py:start():188] Started network monitoring 2024-09-17 12:06:05,237 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: pause 2024-09-17 12:06:05,238 INFO HandlerThread:1253420 [handler.py:handle_request_pause():728] stopping system metrics thread 2024-09-17 12:06:05,238 INFO HandlerThread:1253420 [system_monitor.py:finish():203] Stopping system monitor 2024-09-17 12:06:05,239 INFO HandlerThread:1253420 [interfaces.py:finish():200] Joined cpu monitor 2024-09-17 12:06:05,240 INFO HandlerThread:1253420 [interfaces.py:finish():200] Joined disk monitor 2024-09-17 12:06:05,241 DEBUG SystemMonitor:1253420 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-09-17 12:06:05,241 DEBUG SystemMonitor:1253420 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2024-09-17 12:06:05,241 DEBUG SystemMonitor:1253420 [system_monitor.py:_start():183] Publishing last batch of metrics 2024-09-17 12:06:05,301 INFO HandlerThread:1253420 [interfaces.py:finish():200] Joined gpu monitor 2024-09-17 12:06:05,301 INFO HandlerThread:1253420 [interfaces.py:finish():200] Joined memory monitor 2024-09-17 12:06:05,302 INFO HandlerThread:1253420 [interfaces.py:finish():200] Joined network monitor 2024-09-17 12:06:05,303 DEBUG SenderThread:1253420 [sender.py:send():391] send: stats 2024-09-17 12:06:06,304 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: status_report 2024-09-17 12:06:11,922 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: stop_status 2024-09-17 12:06:11,924 DEBUG SenderThread:1253420 [sender.py:send_request():418] send_request: stop_status 2024-09-17 12:06:11,963 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: internal_messages 2024-09-17 12:06:12,050 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: status_report 2024-09-17 12:06:13,487 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: resume 2024-09-17 12:06:13,488 INFO HandlerThread:1253420 [handler.py:handle_request_resume():719] starting system metrics thread 2024-09-17 12:06:13,488 INFO HandlerThread:1253420 [system_monitor.py:start():194] Starting system monitor 2024-09-17 12:06:13,488 INFO SystemMonitor:1253420 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-09-17 12:06:13,491 INFO SystemMonitor:1253420 [interfaces.py:start():188] Started cpu monitoring 2024-09-17 12:06:13,492 INFO SystemMonitor:1253420 [interfaces.py:start():188] Started disk monitoring 2024-09-17 12:06:13,493 INFO SystemMonitor:1253420 [interfaces.py:start():188] Started gpu monitoring 2024-09-17 12:06:13,494 INFO SystemMonitor:1253420 [interfaces.py:start():188] Started memory monitoring 2024-09-17 12:06:13,496 INFO SystemMonitor:1253420 [interfaces.py:start():188] Started network monitoring 2024-09-17 12:06:13,528 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: pause 2024-09-17 12:06:13,528 INFO HandlerThread:1253420 [handler.py:handle_request_pause():728] stopping system metrics thread 2024-09-17 12:06:13,528 INFO HandlerThread:1253420 [system_monitor.py:finish():203] Stopping system monitor 2024-09-17 12:06:13,529 INFO HandlerThread:1253420 [interfaces.py:finish():200] Joined cpu monitor 2024-09-17 12:06:13,530 DEBUG SystemMonitor:1253420 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-09-17 12:06:13,530 DEBUG SystemMonitor:1253420 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2024-09-17 12:06:13,530 DEBUG SystemMonitor:1253420 [system_monitor.py:_start():183] Publishing last batch of metrics 2024-09-17 12:06:13,532 INFO HandlerThread:1253420 [interfaces.py:finish():200] Joined disk monitor 2024-09-17 12:06:13,589 INFO HandlerThread:1253420 [interfaces.py:finish():200] Joined gpu monitor 2024-09-17 12:06:13,590 INFO HandlerThread:1253420 [interfaces.py:finish():200] Joined memory monitor 2024-09-17 12:06:13,590 INFO HandlerThread:1253420 [interfaces.py:finish():200] Joined network monitor 2024-09-17 12:06:13,591 DEBUG SenderThread:1253420 [sender.py:send():391] send: stats 2024-09-17 12:06:14,617 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: resume 2024-09-17 12:06:14,617 INFO HandlerThread:1253420 [handler.py:handle_request_resume():719] starting system metrics thread 2024-09-17 12:06:14,617 INFO HandlerThread:1253420 [system_monitor.py:start():194] Starting system monitor 2024-09-17 12:06:14,617 INFO SystemMonitor:1253420 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-09-17 12:06:14,619 INFO SystemMonitor:1253420 [interfaces.py:start():188] Started cpu monitoring 2024-09-17 12:06:14,620 INFO SystemMonitor:1253420 [interfaces.py:start():188] Started disk monitoring 2024-09-17 12:06:14,621 INFO SystemMonitor:1253420 [interfaces.py:start():188] Started gpu monitoring 2024-09-17 12:06:14,622 INFO SystemMonitor:1253420 [interfaces.py:start():188] Started memory monitoring 2024-09-17 12:06:14,624 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: pause 2024-09-17 12:06:14,624 INFO HandlerThread:1253420 [handler.py:handle_request_pause():728] stopping system metrics thread 2024-09-17 12:06:14,624 INFO HandlerThread:1253420 [system_monitor.py:finish():203] Stopping system monitor 2024-09-17 12:06:14,628 INFO SystemMonitor:1253420 [interfaces.py:start():188] Started network monitoring 2024-09-17 12:06:14,628 DEBUG SystemMonitor:1253420 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-09-17 12:06:14,628 DEBUG SystemMonitor:1253420 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2024-09-17 12:06:14,628 DEBUG SystemMonitor:1253420 [system_monitor.py:_start():183] Publishing last batch of metrics 2024-09-17 12:06:14,630 INFO HandlerThread:1253420 [interfaces.py:finish():200] Joined cpu monitor 2024-09-17 12:06:14,632 INFO HandlerThread:1253420 [interfaces.py:finish():200] Joined disk monitor 2024-09-17 12:06:14,687 INFO HandlerThread:1253420 [interfaces.py:finish():200] Joined gpu monitor 2024-09-17 12:06:14,687 INFO HandlerThread:1253420 [interfaces.py:finish():200] Joined memory monitor 2024-09-17 12:06:14,688 INFO HandlerThread:1253420 [interfaces.py:finish():200] Joined network monitor 2024-09-17 12:06:14,688 DEBUG SenderThread:1253420 [sender.py:send():391] send: stats 2024-09-17 12:06:14,930 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: resume 2024-09-17 12:06:14,931 INFO HandlerThread:1253420 [handler.py:handle_request_resume():719] starting system metrics thread 2024-09-17 12:06:14,931 INFO HandlerThread:1253420 [system_monitor.py:start():194] Starting system monitor 2024-09-17 12:06:14,931 INFO SystemMonitor:1253420 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-09-17 12:06:14,932 INFO SystemMonitor:1253420 [interfaces.py:start():188] Started cpu monitoring 2024-09-17 12:06:14,935 INFO SystemMonitor:1253420 [interfaces.py:start():188] Started disk monitoring 2024-09-17 12:06:14,938 INFO SystemMonitor:1253420 [interfaces.py:start():188] Started gpu monitoring 2024-09-17 12:06:14,941 INFO SystemMonitor:1253420 [interfaces.py:start():188] Started memory monitoring 2024-09-17 12:06:14,942 INFO SystemMonitor:1253420 [interfaces.py:start():188] Started network monitoring 2024-09-17 12:06:17,928 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: status_report 2024-09-17 12:06:21,979 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: internal_messages 2024-09-17 12:06:22,979 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: status_report 2024-09-17 12:06:24,010 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: pause 2024-09-17 12:06:24,011 INFO HandlerThread:1253420 [handler.py:handle_request_pause():728] stopping system metrics thread 2024-09-17 12:06:24,011 INFO HandlerThread:1253420 [system_monitor.py:finish():203] Stopping system monitor 2024-09-17 12:06:24,013 INFO HandlerThread:1253420 [interfaces.py:finish():200] Joined cpu monitor 2024-09-17 12:06:24,013 INFO HandlerThread:1253420 [interfaces.py:finish():200] Joined disk monitor 2024-09-17 12:06:24,014 DEBUG SystemMonitor:1253420 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-09-17 12:06:24,014 DEBUG SystemMonitor:1253420 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2024-09-17 12:06:24,014 DEBUG SystemMonitor:1253420 [system_monitor.py:_start():183] Publishing last batch of metrics 2024-09-17 12:06:24,076 INFO HandlerThread:1253420 [interfaces.py:finish():200] Joined gpu monitor 2024-09-17 12:06:24,077 INFO HandlerThread:1253420 [interfaces.py:finish():200] Joined memory monitor 2024-09-17 12:06:24,077 INFO HandlerThread:1253420 [interfaces.py:finish():200] Joined network monitor 2024-09-17 12:06:24,078 DEBUG SenderThread:1253420 [sender.py:send():391] send: stats 2024-09-17 12:06:25,314 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: resume 2024-09-17 12:06:25,314 INFO HandlerThread:1253420 [handler.py:handle_request_resume():719] starting system metrics thread 2024-09-17 12:06:25,314 INFO HandlerThread:1253420 [system_monitor.py:start():194] Starting system monitor 2024-09-17 12:06:25,314 INFO SystemMonitor:1253420 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-09-17 12:06:25,316 INFO SystemMonitor:1253420 [interfaces.py:start():188] Started cpu monitoring 2024-09-17 12:06:25,319 INFO SystemMonitor:1253420 [interfaces.py:start():188] Started disk monitoring 2024-09-17 12:06:25,320 INFO SystemMonitor:1253420 [interfaces.py:start():188] Started gpu monitoring 2024-09-17 12:06:25,323 INFO SystemMonitor:1253420 [interfaces.py:start():188] Started memory monitoring 2024-09-17 12:06:25,325 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: pause 2024-09-17 12:06:25,325 INFO HandlerThread:1253420 [handler.py:handle_request_pause():728] stopping system metrics thread 2024-09-17 12:06:25,325 INFO HandlerThread:1253420 [system_monitor.py:finish():203] Stopping system monitor 2024-09-17 12:06:25,326 INFO HandlerThread:1253420 [interfaces.py:finish():200] Joined cpu monitor 2024-09-17 12:06:25,327 INFO SystemMonitor:1253420 [interfaces.py:start():188] Started network monitoring 2024-09-17 12:06:25,328 DEBUG SystemMonitor:1253420 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-09-17 12:06:25,329 DEBUG SystemMonitor:1253420 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2024-09-17 12:06:25,329 DEBUG SystemMonitor:1253420 [system_monitor.py:_start():183] Publishing last batch of metrics 2024-09-17 12:06:25,332 INFO HandlerThread:1253420 [interfaces.py:finish():200] Joined disk monitor 2024-09-17 12:06:25,390 INFO HandlerThread:1253420 [interfaces.py:finish():200] Joined gpu monitor 2024-09-17 12:06:25,390 INFO HandlerThread:1253420 [interfaces.py:finish():200] Joined memory monitor 2024-09-17 12:06:25,391 INFO HandlerThread:1253420 [interfaces.py:finish():200] Joined network monitor 2024-09-17 12:06:25,391 DEBUG SenderThread:1253420 [sender.py:send():391] send: stats 2024-09-17 12:06:26,068 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: resume 2024-09-17 12:06:26,068 INFO HandlerThread:1253420 [handler.py:handle_request_resume():719] starting system metrics thread 2024-09-17 12:06:26,068 INFO HandlerThread:1253420 [system_monitor.py:start():194] Starting system monitor 2024-09-17 12:06:26,069 INFO SystemMonitor:1253420 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-09-17 12:06:26,070 INFO SystemMonitor:1253420 [interfaces.py:start():188] Started cpu monitoring 2024-09-17 12:06:26,071 INFO SystemMonitor:1253420 [interfaces.py:start():188] Started disk monitoring 2024-09-17 12:06:26,074 INFO SystemMonitor:1253420 [interfaces.py:start():188] Started gpu monitoring 2024-09-17 12:06:26,076 INFO SystemMonitor:1253420 [interfaces.py:start():188] Started memory monitoring 2024-09-17 12:06:26,079 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: pause 2024-09-17 12:06:26,080 INFO HandlerThread:1253420 [handler.py:handle_request_pause():728] stopping system metrics thread 2024-09-17 12:06:26,080 INFO SystemMonitor:1253420 [interfaces.py:start():188] Started network monitoring 2024-09-17 12:06:26,081 INFO HandlerThread:1253420 [system_monitor.py:finish():203] Stopping system monitor 2024-09-17 12:06:26,083 DEBUG SystemMonitor:1253420 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-09-17 12:06:26,084 DEBUG SystemMonitor:1253420 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2024-09-17 12:06:26,084 DEBUG SystemMonitor:1253420 [system_monitor.py:_start():183] Publishing last batch of metrics 2024-09-17 12:06:26,087 INFO HandlerThread:1253420 [interfaces.py:finish():200] Joined cpu monitor 2024-09-17 12:06:26,088 INFO HandlerThread:1253420 [interfaces.py:finish():200] Joined disk monitor 2024-09-17 12:06:26,145 INFO HandlerThread:1253420 [interfaces.py:finish():200] Joined gpu monitor 2024-09-17 12:06:26,145 INFO HandlerThread:1253420 [interfaces.py:finish():200] Joined memory monitor 2024-09-17 12:06:26,145 INFO HandlerThread:1253420 [interfaces.py:finish():200] Joined network monitor 2024-09-17 12:06:26,146 DEBUG SenderThread:1253420 [sender.py:send():391] send: stats 2024-09-17 12:06:26,922 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: stop_status 2024-09-17 12:06:26,923 DEBUG SenderThread:1253420 [sender.py:send_request():418] send_request: stop_status 2024-09-17 12:06:28,136 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: status_report 2024-09-17 12:06:31,944 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: internal_messages 2024-09-17 12:06:33,956 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: status_report 2024-09-17 12:06:34,983 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: resume 2024-09-17 12:06:34,984 INFO HandlerThread:1253420 [handler.py:handle_request_resume():719] starting system metrics thread 2024-09-17 12:06:34,984 INFO HandlerThread:1253420 [system_monitor.py:start():194] Starting system monitor 2024-09-17 12:06:34,984 INFO SystemMonitor:1253420 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-09-17 12:06:34,985 INFO SystemMonitor:1253420 [interfaces.py:start():188] Started cpu monitoring 2024-09-17 12:06:34,989 INFO SystemMonitor:1253420 [interfaces.py:start():188] Started disk monitoring 2024-09-17 12:06:34,991 INFO SystemMonitor:1253420 [interfaces.py:start():188] Started gpu monitoring 2024-09-17 12:06:34,993 INFO SystemMonitor:1253420 [interfaces.py:start():188] Started memory monitoring 2024-09-17 12:06:34,995 INFO SystemMonitor:1253420 [interfaces.py:start():188] Started network monitoring 2024-09-17 12:06:35,298 INFO Thread-12 :1253420 [dir_watcher.py:_on_file_created():271] file/dir created: /nfs/production/literature/amina-mardiyyah/NER/ewc_stabilised_no_date/wandb/run-20240917_120410-nm6gjq7r/files/output.log 2024-09-17 12:06:37,301 INFO Thread-12 :1253420 [dir_watcher.py:_on_file_modified():288] file/dir modified: /nfs/production/literature/amina-mardiyyah/NER/ewc_stabilised_no_date/wandb/run-20240917_120410-nm6gjq7r/files/output.log 2024-09-17 12:06:39,145 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: status_report 2024-09-17 12:06:41,651 DEBUG SenderThread:1253420 [sender.py:send():391] send: config 2024-09-17 12:06:41,652 DEBUG SenderThread:1253420 [sender.py:send():391] send: telemetry 2024-09-17 12:06:41,652 DEBUG SenderThread:1253420 [sender.py:send():391] send: metric 2024-09-17 12:06:41,652 DEBUG SenderThread:1253420 [sender.py:send():391] send: telemetry 2024-09-17 12:06:41,652 DEBUG SenderThread:1253420 [sender.py:send():391] send: metric 2024-09-17 12:06:41,652 WARNING SenderThread:1253420 [sender.py:send_metric():1417] Seen metric with glob (shouldn't happen) 2024-09-17 12:06:41,654 DEBUG SenderThread:1253420 [sender.py:send():391] send: telemetry 2024-09-17 12:06:41,654 DEBUG SenderThread:1253420 [sender.py:send():391] send: telemetry 2024-09-17 12:06:41,654 DEBUG SenderThread:1253420 [sender.py:send():391] send: config 2024-09-17 12:06:41,923 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: stop_status 2024-09-17 12:06:41,924 DEBUG SenderThread:1253420 [sender.py:send_request():418] send_request: stop_status 2024-09-17 12:06:41,944 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: internal_messages 2024-09-17 12:06:42,726 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: pause 2024-09-17 12:06:42,726 INFO HandlerThread:1253420 [handler.py:handle_request_pause():728] stopping system metrics thread 2024-09-17 12:06:42,726 INFO HandlerThread:1253420 [system_monitor.py:finish():203] Stopping system monitor 2024-09-17 12:06:42,727 INFO HandlerThread:1253420 [interfaces.py:finish():200] Joined cpu monitor 2024-09-17 12:06:42,728 INFO HandlerThread:1253420 [interfaces.py:finish():200] Joined disk monitor 2024-09-17 12:06:42,729 DEBUG SystemMonitor:1253420 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-09-17 12:06:42,729 DEBUG SystemMonitor:1253420 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2024-09-17 12:06:42,729 DEBUG SystemMonitor:1253420 [system_monitor.py:_start():183] Publishing last batch of metrics 2024-09-17 12:06:42,791 INFO HandlerThread:1253420 [interfaces.py:finish():200] Joined gpu monitor 2024-09-17 12:06:42,791 INFO HandlerThread:1253420 [interfaces.py:finish():200] Joined memory monitor 2024-09-17 12:06:42,792 INFO HandlerThread:1253420 [interfaces.py:finish():200] Joined network monitor 2024-09-17 12:06:42,793 DEBUG SenderThread:1253420 [sender.py:send():391] send: stats 2024-09-17 12:06:43,307 INFO Thread-12 :1253420 [dir_watcher.py:_on_file_modified():288] file/dir modified: /nfs/production/literature/amina-mardiyyah/NER/ewc_stabilised_no_date/wandb/run-20240917_120410-nm6gjq7r/files/output.log 2024-09-17 12:06:44,794 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: status_report 2024-09-17 12:06:49,996 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: status_report 2024-09-17 12:06:51,944 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: internal_messages 2024-09-17 12:06:56,006 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: status_report 2024-09-17 12:06:56,323 INFO Thread-12 :1253420 [dir_watcher.py:_on_file_modified():288] file/dir modified: /nfs/production/literature/amina-mardiyyah/NER/ewc_stabilised_no_date/wandb/run-20240917_120410-nm6gjq7r/files/config.yaml 2024-09-17 12:06:56,923 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: stop_status 2024-09-17 12:06:56,924 DEBUG SenderThread:1253420 [sender.py:send_request():418] send_request: stop_status 2024-09-17 12:07:01,944 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: internal_messages 2024-09-17 12:07:01,996 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: status_report 2024-09-17 12:07:06,996 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: status_report 2024-09-17 12:07:11,923 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: stop_status 2024-09-17 12:07:11,924 DEBUG SenderThread:1253420 [sender.py:send_request():418] send_request: stop_status 2024-09-17 12:07:11,964 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: internal_messages 2024-09-17 12:07:12,094 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: status_report 2024-09-17 12:07:17,997 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: status_report 2024-09-17 12:07:21,945 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: internal_messages 2024-09-17 12:07:22,997 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: status_report 2024-09-17 12:07:26,923 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: stop_status 2024-09-17 12:07:26,924 DEBUG SenderThread:1253420 [sender.py:send_request():418] send_request: stop_status 2024-09-17 12:07:28,087 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: status_report 2024-09-17 12:07:31,945 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: internal_messages 2024-09-17 12:07:33,997 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: status_report 2024-09-17 12:07:38,997 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: status_report 2024-09-17 12:07:41,923 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: stop_status 2024-09-17 12:07:41,924 DEBUG SenderThread:1253420 [sender.py:send_request():418] send_request: stop_status 2024-09-17 12:07:41,964 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: internal_messages 2024-09-17 12:07:44,112 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: status_report 2024-09-17 12:07:49,998 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: status_report 2024-09-17 12:07:51,945 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: internal_messages 2024-09-17 12:07:54,998 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: status_report 2024-09-17 12:07:56,923 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: stop_status 2024-09-17 12:07:56,924 DEBUG SenderThread:1253420 [sender.py:send_request():418] send_request: stop_status 2024-09-17 12:08:00,093 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: status_report 2024-09-17 12:08:01,945 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: internal_messages 2024-09-17 12:08:05,999 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: status_report 2024-09-17 12:08:10,999 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: status_report 2024-09-17 12:08:11,924 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: stop_status 2024-09-17 12:08:11,924 DEBUG SenderThread:1253420 [sender.py:send_request():418] send_request: stop_status 2024-09-17 12:08:11,964 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: internal_messages 2024-09-17 12:08:16,090 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: status_report 2024-09-17 12:08:21,946 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: internal_messages 2024-09-17 12:08:21,999 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: status_report 2024-09-17 12:08:26,924 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: stop_status 2024-09-17 12:08:26,925 DEBUG SenderThread:1253420 [sender.py:send_request():418] send_request: stop_status 2024-09-17 12:08:27,086 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: status_report 2024-09-17 12:08:31,946 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: internal_messages 2024-09-17 12:08:32,999 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: status_report 2024-09-17 12:08:38,000 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: status_report 2024-09-17 12:08:41,924 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: stop_status 2024-09-17 12:08:41,925 DEBUG SenderThread:1253420 [sender.py:send_request():418] send_request: stop_status 2024-09-17 12:08:41,965 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: internal_messages 2024-09-17 12:08:43,076 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: status_report 2024-09-17 12:08:49,000 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: status_report 2024-09-17 12:08:51,946 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: internal_messages 2024-09-17 12:08:54,000 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: status_report 2024-09-17 12:08:56,924 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: stop_status 2024-09-17 12:08:56,925 DEBUG SenderThread:1253420 [sender.py:send_request():418] send_request: stop_status 2024-09-17 12:08:59,063 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: status_report 2024-09-17 12:09:01,946 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: internal_messages 2024-09-17 12:09:05,001 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: status_report 2024-09-17 12:09:10,001 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: status_report 2024-09-17 12:09:11,924 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: stop_status 2024-09-17 12:09:11,925 DEBUG SenderThread:1253420 [sender.py:send_request():418] send_request: stop_status 2024-09-17 12:09:11,965 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: internal_messages 2024-09-17 12:09:15,094 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: status_report 2024-09-17 12:09:21,001 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: status_report 2024-09-17 12:09:21,947 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: internal_messages 2024-09-17 12:09:26,001 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: status_report 2024-09-17 12:09:26,924 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: stop_status 2024-09-17 12:09:26,925 DEBUG SenderThread:1253420 [sender.py:send_request():418] send_request: stop_status 2024-09-17 12:09:31,084 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: status_report 2024-09-17 12:09:31,947 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: internal_messages 2024-09-17 12:09:37,001 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: status_report 2024-09-17 12:09:41,925 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: stop_status 2024-09-17 12:09:41,925 DEBUG SenderThread:1253420 [sender.py:send_request():418] send_request: stop_status 2024-09-17 12:09:41,965 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: internal_messages 2024-09-17 12:09:42,133 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: status_report 2024-09-17 12:09:48,001 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: status_report 2024-09-17 12:09:51,947 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: internal_messages 2024-09-17 12:09:53,002 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: status_report 2024-09-17 12:09:56,925 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: stop_status 2024-09-17 12:09:56,926 DEBUG SenderThread:1253420 [sender.py:send_request():418] send_request: stop_status 2024-09-17 12:09:58,140 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: status_report 2024-09-17 12:10:01,947 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: internal_messages 2024-09-17 12:10:04,002 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: status_report 2024-09-17 12:10:09,003 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: status_report 2024-09-17 12:10:11,925 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: stop_status 2024-09-17 12:10:11,926 DEBUG SenderThread:1253420 [sender.py:send_request():418] send_request: stop_status 2024-09-17 12:10:11,966 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: internal_messages 2024-09-17 12:10:14,095 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: status_report 2024-09-17 12:10:20,003 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: status_report 2024-09-17 12:10:21,948 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: internal_messages 2024-09-17 12:10:25,003 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: status_report 2024-09-17 12:10:26,925 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: stop_status 2024-09-17 12:10:26,926 DEBUG SenderThread:1253420 [sender.py:send_request():418] send_request: stop_status 2024-09-17 12:10:30,096 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: status_report 2024-09-17 12:10:31,948 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: internal_messages 2024-09-17 12:10:36,003 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: status_report 2024-09-17 12:10:41,004 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: status_report 2024-09-17 12:10:41,925 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: stop_status 2024-09-17 12:10:41,926 DEBUG SenderThread:1253420 [sender.py:send_request():418] send_request: stop_status 2024-09-17 12:10:41,966 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: internal_messages 2024-09-17 12:10:46,112 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: status_report 2024-09-17 12:10:51,948 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: internal_messages 2024-09-17 12:10:52,003 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: status_report 2024-09-17 12:10:56,925 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: stop_status 2024-09-17 12:10:56,926 DEBUG SenderThread:1253420 [sender.py:send_request():418] send_request: stop_status 2024-09-17 12:10:57,143 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: status_report 2024-09-17 12:11:01,948 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: internal_messages 2024-09-17 12:11:03,004 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: status_report 2024-09-17 12:11:08,004 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: status_report 2024-09-17 12:11:11,926 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: stop_status 2024-09-17 12:11:11,926 DEBUG SenderThread:1253420 [sender.py:send_request():418] send_request: stop_status 2024-09-17 12:11:11,966 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: internal_messages 2024-09-17 12:11:13,079 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: status_report 2024-09-17 12:11:19,005 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: status_report 2024-09-17 12:11:21,948 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: internal_messages 2024-09-17 12:11:24,005 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: status_report 2024-09-17 12:11:26,926 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: stop_status 2024-09-17 12:11:26,927 DEBUG SenderThread:1253420 [sender.py:send_request():418] send_request: stop_status 2024-09-17 12:11:29,082 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: status_report 2024-09-17 12:11:31,949 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: internal_messages 2024-09-17 12:11:35,005 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: status_report 2024-09-17 12:11:40,005 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: status_report 2024-09-17 12:11:41,926 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: stop_status 2024-09-17 12:11:41,927 DEBUG SenderThread:1253420 [sender.py:send_request():418] send_request: stop_status 2024-09-17 12:11:41,967 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: internal_messages 2024-09-17 12:11:45,065 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: status_report 2024-09-17 12:11:51,006 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: status_report 2024-09-17 12:11:51,949 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: internal_messages 2024-09-17 12:11:56,006 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: status_report 2024-09-17 12:11:56,926 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: stop_status 2024-09-17 12:11:56,927 DEBUG SenderThread:1253420 [sender.py:send_request():418] send_request: stop_status 2024-09-17 12:11:59,448 DEBUG SenderThread:1253420 [sender.py:send():391] send: exit 2024-09-17 12:11:59,449 INFO SenderThread:1253420 [sender.py:send_exit():598] handling exit code: 0 2024-09-17 12:11:59,449 INFO SenderThread:1253420 [sender.py:send_exit():600] handling runtime: 19 2024-09-17 12:11:59,454 INFO SenderThread:1253420 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-09-17 12:11:59,455 INFO SenderThread:1253420 [sender.py:send_exit():606] send defer 2024-09-17 12:11:59,455 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: defer 2024-09-17 12:11:59,455 INFO HandlerThread:1253420 [handler.py:handle_request_defer():184] handle defer: 0 2024-09-17 12:11:59,456 DEBUG SenderThread:1253420 [sender.py:send_request():418] send_request: defer 2024-09-17 12:11:59,456 INFO SenderThread:1253420 [sender.py:send_request_defer():622] handle sender defer: 0 2024-09-17 12:11:59,456 INFO SenderThread:1253420 [sender.py:transition_state():626] send defer: 1 2024-09-17 12:11:59,457 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: defer 2024-09-17 12:11:59,457 INFO HandlerThread:1253420 [handler.py:handle_request_defer():184] handle defer: 1 2024-09-17 12:11:59,457 DEBUG SenderThread:1253420 [sender.py:send_request():418] send_request: defer 2024-09-17 12:11:59,457 INFO SenderThread:1253420 [sender.py:send_request_defer():622] handle sender defer: 1 2024-09-17 12:11:59,457 INFO SenderThread:1253420 [sender.py:transition_state():626] send defer: 2 2024-09-17 12:11:59,458 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: defer 2024-09-17 12:11:59,458 INFO HandlerThread:1253420 [handler.py:handle_request_defer():184] handle defer: 2 2024-09-17 12:11:59,459 DEBUG SenderThread:1253420 [sender.py:send_request():418] send_request: defer 2024-09-17 12:11:59,459 INFO SenderThread:1253420 [sender.py:send_request_defer():622] handle sender defer: 2 2024-09-17 12:11:59,459 INFO SenderThread:1253420 [sender.py:transition_state():626] send defer: 3 2024-09-17 12:11:59,459 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: defer 2024-09-17 12:11:59,459 INFO HandlerThread:1253420 [handler.py:handle_request_defer():184] handle defer: 3 2024-09-17 12:11:59,460 DEBUG SenderThread:1253420 [sender.py:send_request():418] send_request: defer 2024-09-17 12:11:59,460 INFO SenderThread:1253420 [sender.py:send_request_defer():622] handle sender defer: 3 2024-09-17 12:11:59,460 INFO SenderThread:1253420 [sender.py:transition_state():626] send defer: 4 2024-09-17 12:11:59,460 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: defer 2024-09-17 12:11:59,461 INFO HandlerThread:1253420 [handler.py:handle_request_defer():184] handle defer: 4 2024-09-17 12:11:59,461 DEBUG SenderThread:1253420 [sender.py:send_request():418] send_request: defer 2024-09-17 12:11:59,461 INFO SenderThread:1253420 [sender.py:send_request_defer():622] handle sender defer: 4 2024-09-17 12:11:59,461 INFO SenderThread:1253420 [sender.py:transition_state():626] send defer: 5 2024-09-17 12:11:59,462 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: defer 2024-09-17 12:11:59,462 INFO HandlerThread:1253420 [handler.py:handle_request_defer():184] handle defer: 5 2024-09-17 12:11:59,463 DEBUG SenderThread:1253420 [sender.py:send():391] send: summary 2024-09-17 12:11:59,466 INFO SenderThread:1253420 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-09-17 12:11:59,467 DEBUG SenderThread:1253420 [sender.py:send_request():418] send_request: defer 2024-09-17 12:11:59,467 INFO SenderThread:1253420 [sender.py:send_request_defer():622] handle sender defer: 5 2024-09-17 12:11:59,467 INFO SenderThread:1253420 [sender.py:transition_state():626] send defer: 6 2024-09-17 12:11:59,467 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: defer 2024-09-17 12:11:59,468 INFO HandlerThread:1253420 [handler.py:handle_request_defer():184] handle defer: 6 2024-09-17 12:11:59,468 DEBUG SenderThread:1253420 [sender.py:send_request():418] send_request: defer 2024-09-17 12:11:59,468 INFO SenderThread:1253420 [sender.py:send_request_defer():622] handle sender defer: 6 2024-09-17 12:11:59,468 INFO SenderThread:1253420 [sender.py:transition_state():626] send defer: 7 2024-09-17 12:11:59,469 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: status_report 2024-09-17 12:11:59,469 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: defer 2024-09-17 12:11:59,469 INFO HandlerThread:1253420 [handler.py:handle_request_defer():184] handle defer: 7 2024-09-17 12:11:59,470 DEBUG SenderThread:1253420 [sender.py:send_request():418] send_request: defer 2024-09-17 12:11:59,470 INFO SenderThread:1253420 [sender.py:send_request_defer():622] handle sender defer: 7 2024-09-17 12:11:59,690 INFO Thread-12 :1253420 [dir_watcher.py:_on_file_created():271] file/dir created: /nfs/production/literature/amina-mardiyyah/NER/ewc_stabilised_no_date/wandb/run-20240917_120410-nm6gjq7r/files/wandb-summary.json 2024-09-17 12:12:00,448 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: poll_exit 2024-09-17 12:12:01,273 INFO SenderThread:1253420 [sender.py:transition_state():626] send defer: 8 2024-09-17 12:12:01,274 DEBUG SenderThread:1253420 [sender.py:send_request():418] send_request: poll_exit 2024-09-17 12:12:01,274 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: defer 2024-09-17 12:12:01,275 INFO HandlerThread:1253420 [handler.py:handle_request_defer():184] handle defer: 8 2024-09-17 12:12:01,275 DEBUG SenderThread:1253420 [sender.py:send_request():418] send_request: defer 2024-09-17 12:12:01,275 INFO SenderThread:1253420 [sender.py:send_request_defer():622] handle sender defer: 8 2024-09-17 12:12:01,276 INFO SenderThread:1253420 [job_builder.py:build():443] Attempting to build job artifact 2024-09-17 12:12:01,280 INFO SenderThread:1253420 [job_builder.py:_get_source_type():583] no source found 2024-09-17 12:12:01,280 INFO SenderThread:1253420 [sender.py:transition_state():626] send defer: 9 2024-09-17 12:12:01,280 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: defer 2024-09-17 12:12:01,281 INFO HandlerThread:1253420 [handler.py:handle_request_defer():184] handle defer: 9 2024-09-17 12:12:01,281 DEBUG SenderThread:1253420 [sender.py:send_request():418] send_request: defer 2024-09-17 12:12:01,281 INFO SenderThread:1253420 [sender.py:send_request_defer():622] handle sender defer: 9 2024-09-17 12:12:01,282 INFO SenderThread:1253420 [dir_watcher.py:finish():358] shutting down directory watcher 2024-09-17 12:12:01,451 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: poll_exit 2024-09-17 12:12:01,692 INFO SenderThread:1253420 [dir_watcher.py:_on_file_modified():288] file/dir modified: /nfs/production/literature/amina-mardiyyah/NER/ewc_stabilised_no_date/wandb/run-20240917_120410-nm6gjq7r/files/output.log 2024-09-17 12:12:01,693 INFO SenderThread:1253420 [dir_watcher.py:finish():388] scan: /nfs/production/literature/amina-mardiyyah/NER/ewc_stabilised_no_date/wandb/run-20240917_120410-nm6gjq7r/files 2024-09-17 12:12:01,694 INFO SenderThread:1253420 [dir_watcher.py:finish():402] scan save: /nfs/production/literature/amina-mardiyyah/NER/ewc_stabilised_no_date/wandb/run-20240917_120410-nm6gjq7r/files/config.yaml config.yaml 2024-09-17 12:12:01,694 INFO SenderThread:1253420 [dir_watcher.py:finish():402] scan save: /nfs/production/literature/amina-mardiyyah/NER/ewc_stabilised_no_date/wandb/run-20240917_120410-nm6gjq7r/files/wandb-metadata.json wandb-metadata.json 2024-09-17 12:12:01,695 INFO SenderThread:1253420 [dir_watcher.py:finish():402] scan save: /nfs/production/literature/amina-mardiyyah/NER/ewc_stabilised_no_date/wandb/run-20240917_120410-nm6gjq7r/files/requirements.txt requirements.txt 2024-09-17 12:12:01,695 INFO SenderThread:1253420 [dir_watcher.py:finish():402] scan save: /nfs/production/literature/amina-mardiyyah/NER/ewc_stabilised_no_date/wandb/run-20240917_120410-nm6gjq7r/files/output.log output.log 2024-09-17 12:12:01,695 INFO SenderThread:1253420 [dir_watcher.py:finish():402] scan save: /nfs/production/literature/amina-mardiyyah/NER/ewc_stabilised_no_date/wandb/run-20240917_120410-nm6gjq7r/files/wandb-summary.json wandb-summary.json 2024-09-17 12:12:01,695 INFO SenderThread:1253420 [sender.py:transition_state():626] send defer: 10 2024-09-17 12:12:01,696 DEBUG SenderThread:1253420 [sender.py:send_request():418] send_request: poll_exit 2024-09-17 12:12:01,697 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: defer 2024-09-17 12:12:01,707 INFO HandlerThread:1253420 [handler.py:handle_request_defer():184] handle defer: 10 2024-09-17 12:12:01,718 DEBUG SenderThread:1253420 [sender.py:send_request():418] send_request: defer 2024-09-17 12:12:01,718 INFO SenderThread:1253420 [sender.py:send_request_defer():622] handle sender defer: 10 2024-09-17 12:12:01,719 INFO SenderThread:1253420 [file_pusher.py:finish():169] shutting down file pusher 2024-09-17 12:12:02,063 INFO wandb-upload_0:1253420 [upload_job.py:push():130] Uploaded file /nfs/production/literature/amina-mardiyyah/NER/ewc_stabilised_no_date/wandb/run-20240917_120410-nm6gjq7r/files/config.yaml 2024-09-17 12:12:02,089 INFO wandb-upload_3:1253420 [upload_job.py:push():130] Uploaded file /nfs/production/literature/amina-mardiyyah/NER/ewc_stabilised_no_date/wandb/run-20240917_120410-nm6gjq7r/files/wandb-summary.json 2024-09-17 12:12:02,094 INFO wandb-upload_1:1253420 [upload_job.py:push():130] Uploaded file /nfs/production/literature/amina-mardiyyah/NER/ewc_stabilised_no_date/wandb/run-20240917_120410-nm6gjq7r/files/requirements.txt 2024-09-17 12:12:02,107 INFO wandb-upload_2:1253420 [upload_job.py:push():130] Uploaded file /nfs/production/literature/amina-mardiyyah/NER/ewc_stabilised_no_date/wandb/run-20240917_120410-nm6gjq7r/files/output.log 2024-09-17 12:12:02,308 INFO Thread-11 (_thread_body):1253420 [sender.py:transition_state():626] send defer: 11 2024-09-17 12:12:02,309 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: defer 2024-09-17 12:12:02,309 INFO HandlerThread:1253420 [handler.py:handle_request_defer():184] handle defer: 11 2024-09-17 12:12:02,310 DEBUG SenderThread:1253420 [sender.py:send_request():418] send_request: defer 2024-09-17 12:12:02,311 INFO SenderThread:1253420 [sender.py:send_request_defer():622] handle sender defer: 11 2024-09-17 12:12:02,311 INFO SenderThread:1253420 [file_pusher.py:join():175] waiting for file pusher 2024-09-17 12:12:02,311 INFO SenderThread:1253420 [sender.py:transition_state():626] send defer: 12 2024-09-17 12:12:02,311 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: defer 2024-09-17 12:12:02,312 INFO HandlerThread:1253420 [handler.py:handle_request_defer():184] handle defer: 12 2024-09-17 12:12:02,312 DEBUG SenderThread:1253420 [sender.py:send_request():418] send_request: defer 2024-09-17 12:12:02,313 INFO SenderThread:1253420 [sender.py:send_request_defer():622] handle sender defer: 12 2024-09-17 12:12:02,313 INFO SenderThread:1253420 [file_stream.py:finish():601] file stream finish called 2024-09-17 12:12:02,452 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: poll_exit 2024-09-17 12:12:02,801 INFO SenderThread:1253420 [file_stream.py:finish():605] file stream finish is done 2024-09-17 12:12:02,801 INFO SenderThread:1253420 [sender.py:transition_state():626] send defer: 13 2024-09-17 12:12:02,801 DEBUG SenderThread:1253420 [sender.py:send_request():418] send_request: poll_exit 2024-09-17 12:12:02,802 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: defer 2024-09-17 12:12:02,802 INFO HandlerThread:1253420 [handler.py:handle_request_defer():184] handle defer: 13 2024-09-17 12:12:02,803 DEBUG SenderThread:1253420 [sender.py:send_request():418] send_request: defer 2024-09-17 12:12:02,803 INFO SenderThread:1253420 [sender.py:send_request_defer():622] handle sender defer: 13 2024-09-17 12:12:02,803 INFO SenderThread:1253420 [sender.py:transition_state():626] send defer: 14 2024-09-17 12:12:02,804 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: defer 2024-09-17 12:12:02,804 INFO HandlerThread:1253420 [handler.py:handle_request_defer():184] handle defer: 14 2024-09-17 12:12:02,805 DEBUG SenderThread:1253420 [sender.py:send():391] send: final 2024-09-17 12:12:02,805 DEBUG SenderThread:1253420 [sender.py:send():391] send: footer 2024-09-17 12:12:02,805 DEBUG SenderThread:1253420 [sender.py:send_request():418] send_request: defer 2024-09-17 12:12:02,805 INFO SenderThread:1253420 [sender.py:send_request_defer():622] handle sender defer: 14 2024-09-17 12:12:02,807 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: poll_exit 2024-09-17 12:12:02,807 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: server_info 2024-09-17 12:12:02,808 DEBUG SenderThread:1253420 [sender.py:send_request():418] send_request: poll_exit 2024-09-17 12:12:02,809 DEBUG SenderThread:1253420 [sender.py:send_request():418] send_request: server_info 2024-09-17 12:12:02,814 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: poll_exit 2024-09-17 12:12:02,815 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: get_summary 2024-09-17 12:12:02,816 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: sampled_history 2024-09-17 12:12:02,817 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: internal_messages 2024-09-17 12:12:02,921 DEBUG SenderThread:1253420 [sender.py:send_request():418] send_request: poll_exit 2024-09-17 12:12:02,922 INFO MainThread:1253420 [wandb_run.py:_footer_history_summary_info():4088] rendering history 2024-09-17 12:12:02,922 INFO MainThread:1253420 [wandb_run.py:_footer_history_summary_info():4120] rendering summary 2024-09-17 12:12:02,922 INFO MainThread:1253420 [wandb_run.py:_footer_sync_info():4047] logging synced files 2024-09-17 12:12:02,925 DEBUG HandlerThread:1253420 [handler.py:handle_request():158] handle_request: shutdown 2024-09-17 12:12:02,925 INFO HandlerThread:1253420 [handler.py:finish():884] shutting down handler 2024-09-17 12:12:03,797 WARNING StreamThr :1253420 [internal.py:is_dead():413] Internal process exiting, parent pid 1253167 disappeared 2024-09-17 12:12:03,798 ERROR StreamThr :1253420 [internal.py:wandb_internal():151] Internal process shutdown. 2024-09-17 12:12:03,815 INFO WriterThread:1253420 [datastore.py:close():296] close: /nfs/production/literature/amina-mardiyyah/NER/ewc_stabilised_no_date/wandb/run-20240917_120410-nm6gjq7r/run-nm6gjq7r.wandb 2024-09-17 12:12:03,921 INFO SenderThread:1253420 [sender.py:finish():1631] shutting down sender 2024-09-17 12:12:03,922 INFO SenderThread:1253420 [file_pusher.py:finish():169] shutting down file pusher 2024-09-17 12:12:03,922 INFO SenderThread:1253420 [file_pusher.py:join():175] waiting for file pusher