KeyError: '_data' when training on AWS
Hi all,I’ve been working through adaptingthe getting started notebookto my particular use case. I wrote out my data to s3, and kicked off .fit(), but am getting this error block:2021-04-23 04:58:40,552 sagemaker-training-toolkit ERROR    ExecuteUserScriptError:
Command "/opt/conda/bin/python3.6 train.py --epochs 1 --model_name distilbert-base-uncased --train_batch_size 32"
Traceback (most recent call last):
  File "train.py", line 42, in <module>
    train_dataset = load_from_disk(args.training_dir)
  File "/opt/conda/lib/python3.6/site-packages/datasets/load.py", line 781, in load_from_disk
    return Dataset.load_from_disk(dataset_path, fs)
  File "/opt/conda/lib/python3.6/site-packages/datasets/arrow_dataset.py", line 684, in load_from_disk
    state = {k: state[k] for k in dataset.__dict__.keys()}  # in case we add new fields
  File "/opt/conda/lib/python3.6/site-packages/datasets/arrow_dataset.py", line 684, in <dictcomp>
    state = {k: state[k] for k in dataset.__dict__.keys()}  # in case we add new fields
KeyError: '_data'What’s leaving me scratching my head is that when I reference thearrow_dataset.pyfile, I can’t find lines of this kind, making me think there’s some discrepancy in whatever AWS’ container is and that file.Regardless, does anyone have any advice/intuition on what may be going on here? I don’t know what the ‘_data’ key would refer to in this case, and am looking for help. Thanks!
2021-04-23T05:47:21Z
Hey@cccx3,Thank you for creating this topic! There was an error in the01_getting_started_pytorchwhere it installeddatasets 1.6.0, which has some changes, and in the DLC currently 1.5.0 is installed.This was already fixed in aPRyesterday.To solve this you need to installdatasets==1.5.0in your notebook and pre-process the data again.!pip install "sagemaker>=2.31.0" "transformers==4.4.2" "datasets[s3]==1.5.0" --upgrade
2021-04-26T15:32:15Z
I had the same error and this fixed it for me, thanks!Do you know if/whendatasetswill be updated to 1.6.0 in the HuggingFace DLC?
2021-04-26T15:46:38Z
We are going to include it with the nexttransformersrelease (4.6.0).If you want to use it in advance you can add arequirements.txtin yoursource_dirwithdatasets==1.6.1in it.
2021-08-19T12:47:46Z
I’m facing the same issue with following lib versions:datasets == 1.11.0sagemaker == 2.48.1UnexpectedStatusException: Error for Training job huggingface-pytorch-training-2021-08-19-12-34-40-568: Failed. Reason: AlgorithmError: ExecuteUserScriptError:Command “/opt/conda/bin/python3.6 train.py --epochs 3 --model_name bert-base-uncased --train_batch_size 16”Traceback (most recent call last):File “train.py”, line 41, intrain_dataset = load_from_disk(args.training_dir)File “/opt/conda/lib/python3.6/site-packages/datasets/load.py”, line 781, in load_from_diskreturn Dataset.load_from_disk(dataset_path, fs)File “/opt/conda/lib/python3.6/site-packages/datasets/arrow_dataset.py”, line 684, in load_from_diskstate = {k: state[k] for k in dataset.dict.keys()}  # in case we add new fieldsFile “/opt/conda/lib/python3.6/site-packages/datasets/arrow_dataset.py”, line 684, instate = {k: state[k] for k in dataset.dict.keys()}  # in case we add new fieldsKeyError: ‘_data’
2021-08-19T12:51:07Z
Is datasets version 1.5.0 strictly necessary?
2021-08-19T13:59:10Z
Hey@anikalburgi,Since the creation of this topic we released new DLC container, take a look here:Hugging Face on Amazon SageMakerYou can usetransformers_version=4.6.1in your Estimator to use the image withdatasets 1.6.2.
2021-08-19T16:51:23Z
Perfect, Thanks!This works!Although, I’m facing CUDA Out of Memory issue with batch size even as low as 8 on p3.2xlarge instance for bert-base-uncased.Usually, when trained from torch directly and outside sagemaker, the batch size of 14-16 safely works for me on both g4dn.xlarge and p3.2xlarge instances.
2021-08-19T17:03:31Z
Can you share yourHuggingFaceestimator and the script you use?
2021-08-20T13:12:55Z
Nah, It was my bad,EOD rush - added _'s instead of -'s in batch size argIt’s fixed and working smoothly!
2021-08-20T13:13:37Z
Thank you@philschmid