Creating Vision dataset with images on s3
Hi everyone,I’m trying to create adataset for an object detection task. The training images are stored on s3 and I would like to eventually use sagemaker and aestimator to train the model.I’m trying to build on the example from@philschmidinHuggingface Sagemaker - Vision Transformerbut with my own dataset and the model fromFine-tuning DETR on a custom dataset for object detectionby@nielsr.If I understand correctly I need to create a dataset first and then save it in the session bucket on s3 but I am not entirely sure how to do that with a dataset which is too big to pull locally first in order to create it.I have found theload_datasetfunction with the ‘imagefolder’ option which seems to do what I want for local image files but doesn’t seem to support filepaths on s3. I have also found theload_from_diskfunction which seems to do the loading fordatasets from s3 but doesn’t have an imagefolder option.What is the best way to prepare my data in this case?Thanks for the help!
2022-06-23T15:20:48Z
Hello@cotrane,you don’t need to save the dataset in the session bucket or do the pre-processing in advance. You could do everything inside sagemaker.Meaning you can either have as first the the download of your dataset from s3 to local and the useload_datasetor just provide the S3 URI when callingHuggingFace.fit()to your bucket. Those S3 URIs don’t need to be on the session bucket they could be on any bucket.huggingface_estimator.fit({'train': 'yourbucket'})sagemaker will then download it for you and save it into/opt/ml/input/train
2022-06-24T08:00:46Z
Ah ok - that makes sense! Thanks for the help. I’ll try that today!
2022-07-04T08:37:43Z
I do have a similar question@philschmid.Basically I have the same problem set up as cotrane, but I wanted to use the FastFile input mode, because of the size of my dataset. As I understand it, FastFile streams the data from S3 instead of downloading it all at once. Is there any way I can make that work together with the HF estimator/dataset approach?
2022-07-04T13:28:15Z
@johkoyes you can. The Hugging Face estimator and DLC support all known SageMaker features. Meaning you can use the File input mode for your training. Documentation can be found here:Access Training Data - Amazon SageMaker
2022-07-05T07:15:04Z
Thank you.I thought FastFile Mode was different from File Mode in terms of where and how the input data will be stored. But the figure on that page you linked makes it clearer for me
2022-07-06T09:37:41Z
Just a little addition to the path of the data, at least for me the path was/opt/ml/input/data/trainso with an additionaldatafolder in between.But I guess safest way is to use the environment variables likeSM_CHANNEL_TRAINfor the correct path
2022-08-25T15:08:57Z
Hi@philschmid. Apologies if this is answered and I just misread it, but is it possible to useload_datasetwithimagefolderfrom s3 just like I would locally?## load images from s3
import boto3
from sagemaker import get_execution_role

role = get_execution_role()
data_location = "s3/path/here"
dataset = load_dataset("imagefolder", data_dir=data_location)I get the following error in SageMaker Studio despite it working locally:FileNotFoundError: The directory at "s3/path/here" doesn't contain any data file
2022-08-25T15:48:38Z
@cgpeltier-janesthats currently not possible. But there is some efforts on improving the integration with cloud-storages:Download and prepare as Parquet for cloud storage by lhoestq · Pull Request #4724 · huggingface/datasets · GitHub
2022-09-15T11:51:00Z
@philschmidthe PR you have linked is merged, however as far as I can tell it does not contain support forimagefolder. This is a pretty important functionality, since as is I have 2 options:Download the entire dataset to SageMaker EFS, preprocess it, and save it to S3. This takes a lot of time and is inconvenient code-wise.Process data every time in HuggingFace Estimatortrain.pyscript. This is very costly and time-consuming, since e.g. in hyperparameter optimization I would have to do this every time, in every estimator, and on GPU instance.Would making a separate Github issue for this make sense in this case?I basically wantsomething like this, but without downloading everything from S3 manually.