Data not passed to run_summarization.py
Hello!I’m in the process of trying to finetune theNbAiLab/nb-t5-basemodel on my own dataset. The training job starts fine when callinghuggingface_estimator.fit(), but gives aValueErrorafter the training image has been downloaded:ValueError: Need either a dataset name or a training/validation file.So it appears that the training data that is passed inhuggingface_estimator.fit()is notpassed correctly to therun_summarization.pyscript. Any pointers? Simplified script is below.CHECKPOINT = "NbAiLab/nb-t5-base"

train_dataset = Dataset.from_pandas(train_df)  # df with columns 'text' and 'labels'
test_dataset = Dataset.from_pandas(test_df)

tokenizer = AutoTokenizer.from_pretrained(CHECKPOINT)

def tokenize(batch):
    return tokenizer(batch['text'], padding='max_length', truncation=True)
   
train_dataset = train_dataset.map(tokenize, batched=True)
test_dataset = test_dataset.map(tokenize, batched=True)

train_dataset.set_format('torch', columns=['input_ids', 'attention_mask', 'labels'])
test_dataset.set_format('torch', columns=['input_ids', 'attention_mask', 'labels'])

s3 = S3FileSystem()  

# save train_dataset to s3
training_input_path = f's3://{sess.default_bucket()}/data/train'
train_dataset.save_to_disk(training_input_path,fs=s3)

# save preprocessed test data to S3
test_input_path = f"s3://{session.default_bucket()}/data/test"
test_dataset.save_to_disk(test_input_path, fs=s3)

hyperparameters = {
	"model_name_or_path":CHECKPOINT,
	"output_dir":"/opt/ml/model",
        "epochs": 1,
        "train_batch_size": 32
}

# git configuration to download our fine-tuning script
git_config = {'repo': 'https://github.com/huggingface/transformers.git','branch': 'v4.6.1'}

# creates Hugging Face estimator
huggingface_estimator = sagemaker.huggingface.HuggingFace(
	entry_point='run_summarization.py',
	source_dir='./examples/pytorch/summarization',
	instance_type='ml.p3.2xlarge',
	instance_count=1,
        checkpoint_s3_uri=f's3://{sess.default_bucket()}/checkpoints',
        use_spot_instances=True,
        max_wait=3600,
        max_run=1000,
	role=role,
	git_config=git_config,
	transformers_version='4.6.1',
	pytorch_version='1.7.1',
	py_version='py36',
	hyperparameters = hyperparameters
)
2022-03-04T16:21:01Z
how are you callingfit()method?
2022-03-07T08:12:30Z
So, looks like it was a typo causing the problems. I have used bothsess.default_bucket()andsessìon.default_bucket(), whilesessis never defined. My bad!For reference, I called the fit method as follows:#starting the train job
huggingface_estimator.fit(
    inputs={
        "train": training_input_path,
        "test": test_input_path
    }
)
2022-03-08T13:48:05Z
Unfortunately, the problem persists even after fixing the typo’s in the default buckets.Calling the.fit()method as follows still results in the same error. I have verified thattraining_input_pathandtest_input_pathpoint to the correct location on S3 (the location that is passed to the.save_to_disk()call from twoDatasetobjects`)huggingface_estimator.fit(
    {
        "train": training_input_path,
        "test": test_input_path
    }
)This still results in the followingValueError, any help is still appreciated!ValueError: Need either a dataset name or a training/validation file.
2022-03-08T13:57:15Z
Looking at the source code forrun_summarization.py, the error seems to be in the__post_init__call ofDataTrainingArgumentshere:if self.dataset_name is None and self.train_file is None and self.validation_file is None:
            raise ValueError("Need either a dataset name or a training/validation file.")So it looks like the data from S3 is not parsed totrain_file, looking at thetrain_fileargument it seems to expect a csv or json, rather than a location on S3train_file: Optional[str] = field(
        default=None, metadata={"help": "The input training data file (a jsonlines or csv file)."}
    )Am I right in conluding from this that therun_summarizationscript cannot be used with data from S3?
2022-03-08T14:15:26Z
If you are using run_summarization.py you need to tell it where to look. Sagemaker will load your s3 data into/opt/ml/input/data/train/whatever.csvor/opt/ml/input/data/test/whatever.csv, so pass in another parameter in the hyperparameters. iehyperparameters={
    'model_name_or_path': MODEL,
    'train_file': "/opt/ml/input/data/train/whatever_train.csv",
    'test_file': "/opt/ml/input/data/test/whatever_test.csv",
etc
2022-03-08T14:20:02Z
Thanks for the tip! I was using thesave_to_disk()method from theDatasetobject which (I believe) writes the entireDatasetobject to S3 as an.arrowfile. I’ll try it out with a simple csv file.
2022-03-08T14:35:53Z
@thuskenif you would like to use your dataset, which was saved withsave_to_diskyou would need to fork theexamples/script you use and replace theload_datasetmethod withload_from_diskmethod. [REF].
2022-03-08T14:42:02Z
Good to know@philschmid! I’ll try it out with the CSV approach first, but might give it a go with theload_from_diskmethod if that doesn’t work.