Image classification using AutoTrain: dataset preperation authentication and column mapping
I’ve previously successfully tested using AutoTrain for image classification problems but I am currently running into issues uploading data.Authentication: most of the datasets I want to test have been uploaded to the hub but cannot be shared publicly. When I try and select these datasets, I get an error.Screenshot 2022-11-29 at 10.48.111152×292 30.9 KBI didn’t see an obvious way to pass in an auth token. Is this possible?Column selection: When adding a public dataset (in this case, using biglam/encyclopaedia_britannica_illustrated), the mapping options for image/labels differ from the underlying dataset. In this case, the original dataset exposes an ‘image’ and ‘label’ column (plus some other metadata columns). When loading in autotrain image seems to be expanded toimage.src,image.heightandimage.width.Screenshot 2022-11-29 at 10.55.342054×1388 316 KBI’m unsure if these are internal attributes or mean to be publicly exposed? Choosing what I assume would be the correct image columnimage.srcas the image column in the mapping results in an error when loading.Under the training tab an error is triggered whenformat_sourceis run:Error type: InvalidColMappingError
Details: Column mapping {'label': 'target', 'image.src': 'image'} is invalid for data with columns ['image', 'label', 'id', 'meta'].
Column 'image.src' not found in data.I assume this is because the internal loader is looking forimage.srcin the dataset and not finding it.Apologies if this has been addressed before; I dug around for other issues but didn’t see anything related.Tagging@abhishek, who might be the best person to address this.
2022-11-30T08:35:48Z
And related Discord discussionDiscord
2022-11-30T11:31:49Z
Hi@davanstrien&@osanseviero,Thank you for reporting these issues!Re: AuthenticationTraining on gated or private datasets is not supported in AutoTrain yet.Re: Column selectionThere is indeed an issue with our integration with the datasets server, which AutoTrain uses to fetch the dataset’s first rows. We are currently working on fixing this. I’ll let you know when this is fixed.
2022-11-30T11:46:56Z
Thanks,@sbrandeis!sbrandeis:Training on gated or private datasets is not supported in AutoTrain yet.Thanks for confirming – some of the datasets that I’m working on are small enough to upload via the AutoTrain interface (which allows you to keep them private), so I can work around this.sbrandeis:There is indeed an issue with our integration with the datasets server, which AutoTrain uses to fetch the dataset’s first rows. We are currently working on fixing this. I’ll let you know when this is fixed.Thanks for this. The last time I used AutoTrain for image classification, it worked very well, so looking forward to this fix
2022-12-07T15:24:26Z
Hello@davanstrienBetter late than never - we shipped a fix for the issue with image datasets today!You should be able to train on image datasets now.Let us know if you encounter any other issues!
2022-12-08T03:38:28Z
Hi, I met the similar problem in Text Classification (binary). When I select col names, it seems that all choices are from the second row . Could you please fix it?
2022-12-08T06:36:03Z
Can confirm this is happening with Binary Text-Classification.
2022-12-08T06:54:17Z
Thank you for reporting. We are working on a fix.
2022-12-08T10:28:42Z
Hi@CSAle&@VivienTang,Thanks again for reporting the issue; it should be fixed now.Apologies for the inconvenience!
2022-12-09T12:57:30Z
Thanks for working on thisI was successfully able to load thebeansdataset for image classification. I am currently trying to upload my own dataset using both the image folder upload and a dataset hosted on the hub; however, both of these seem to get stuck in the processing step for longer than I would expect for the size of these datasets.Screenshot 2022-12-09 at 12.43.251960×920 127 KBIt’s possible I’m just being impatient, but it seems like the loading might be stuck in some way.
2023-01-18T18:23:52Z
Update for people who might run across this in the future: the issue, in this case, was that the size of the images in my dataset where reasonably large. Resizing the images using theImageMagick morgifycommand to 500px i.e. fixed this, i.e something like:mogrify -resize 500x500^ top_level_data_folder/**/*.jpgWill resize all images under thetop_level_data_folder.NOTE:the morgify command does resize in place (i.e. overwrites the existing image) by default. If you don’t want this, look into the-pathoption.