HfHubHTTPError: 500 Server Error: Internal Server Error for url:
When trying to load newly uploaded datasets usingload_dataset('GBaker/lh_marked_sentence_coref')I get the following error:369         # Convert `HTTPError` into a `HfHubHTTPError` to display request information
    370         # as well (request id and/or server error message)
--> 371         raise HfHubHTTPError(str(e), response=response) from e
    372 
    373 

HfHubHTTPError: 500 Server Error: Internal Server Error for url: https://huggingface.co/api/datasets/GBaker/lh_marked_sentence_coref (Request ID: Root=1-66a3e06f-398c9a801961e61137450cbf;cf0e4b4e-6943-4f1c-8d7c-e726a62a1350)

Internal Error - We're working hard to fix this as soon as possible!```

But I am able to load the dataset by cloning the repo,
```git clone https://huggingface.co/datasets/GBaker/lh_marked_sentence_coref.git```
and then calling `load_dataset` directly on the data directory.
2024-07-26T19:05:06Z
I’m getting the same error, when I invokeload_datasetwith my dataset./usr/local/lib/python3.10/dist-packages/huggingface_hub/utils/_errors.py in REDACTED_TOKEN_for_status(response, endpoint_name)
    369         # Convert `HTTPError` into a `HfHubHTTPError` to display request information
    370         # as well (request id and/or server error message)
--> 371         raise HfHubHTTPError(str(e), response=response) from e
    372 
    373 
HfHubHTTPError: 500 Server Error: Internal Server Error for url: https://huggingface.co/api/datasets/neoneye/simon-arc-shape-v4-rev3 (Request ID: Root=1-66a3f20f-2e729b990853409076fedd4a;dee61d3a-4d7c-4a6b-b200-745b903de953)
Internal Error - We're working hard to fix this as soon as possible!BrowserWhen I access the api in the browser:https://huggingface.co/api/datasets/neoneye/simon-arc-shape-v4-rev3{"error":"Internal Error - We're working hard to fix this as soon as possible!"}Request headersAccept	text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8
Accept-Encoding	gzip, deflate, br, zstd
Accept-Language	en-US,en;q=0.5
Connection	keep-alive
Host	huggingface.co
Priority	u=1
Sec-Fetch-Dest	document
Sec-Fetch-Mode	navigate
Sec-Fetch-Site	cross-site
Upgrade-Insecure-Requests	1
User-Agent	Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:127.0) Gecko/20100101 Firefox/127.0Response headersX-Firefox-Spdy	h2
access-control-allow-origin	https://huggingface.co
access-control-expose-headers	X-Repo-Commit,X-Request-Id,X-Error-Code,X-Error-Message,X-Total-Count,ETag,Link,Accept-Ranges,Content-Range
content-length	80
content-type	application/json; charset=utf-8
cross-origin-opener-policy	same-origin
date	Fri, 26 Jul 2024 19:09:45 GMT
etag	W/"50-9qrwU+BNI4SD0Fe32p/nofkmv0c"
referrer-policy	strict-origin-when-cross-origin
vary	Origin
via	1.1 1624c79cd07e6098196697a6a7907e4a.cloudfront.net (CloudFront)
x-amz-cf-id	SP8E7n5qRaP6i9c9G83dNAiOzJBU4GXSrDRAcVNTomY895K35H0nJQ==
x-amz-cf-pop	CPH50-C1
x-cache	Error from cloudfront
x-error-message	Internal Error - We're working hard to fix this as soon as possible!
x-powered-by	huggingface-moon
x-request-id	Root=1-66a3f479-026417465ef42f49349fdca1
2024-07-26T20:10:46Z
Same here. I also ran into this error when try to access my dataset via API.
2024-07-26T21:27:29Z
Same error. Please look into it.Code# limit determines how many documents will be streamed (remove for all)
# to fetch a specific dump: hf://datasets/HuggingFaceFW/fineweb/data/CC-MAIN-2024-10
# replace "data" with "sample/100BT" to use the 100BT sample
data_reader = ParquetReader("hf://datasets/HuggingFaceFW/fineweb/sample/10BT", limit=10) 
for document in data_reader():
    # do something with document
    print(document)
2024-07-26T22:30:25Z
Same problem since this morning, cannot access any newly uploaded datasets.
2024-07-26T23:58:06Z
Same error here,. Please help
2024-07-27T01:37:47Z
same error now
2024-07-27T02:24:22Z
Same error when pullingmonology/pile-uncopyrightedHfHubHTTPError: 500 Server Error: Internal Server Error for url: https://huggingface.co/api/datasets/monology/pile-uncopyrighted (Request ID: Root=1-66a4591f-064984386142166a20871761;bf68f7b5-3022-4773-b476-a7410cf305b3)

Internal Error - We're working hard to fix this as soon as possible!
2024-07-27T02:32:09Z
I’m getting the same error. I trieddeleting the dataset and create it againuploading it to a different organisationchanging the data format fromparquettoarrowrecreating the access token that I’m using for HfApibut none of them solved the issue
2024-07-27T06:59:52Z
Me too.When I’m iterating, this happens and ruins 30 hours of training: …
2024-07-27T08:23:37Z
I have created an issue on thegithub datasets repo, that links back to this discussion.
2024-07-27T17:52:26Z
one alternative worked  for me :`!git clone yourdataset_pathdataset = load_dataset(“parquet”, data_dir=“yourdataset_local_path”)`
2024-07-27T19:06:52Z
I don’t understand what repo to clone. Lets say for ‘tasksource/icl-symbol-tuning-instruct · Datasets at Hugging Face’, what do I clone?
2024-07-27T19:10:30Z
Thanks mate !worked out for me also… seems okay when dataset is local…
2024-07-27T19:12:34Z
Hi, you had to clone the whole url :!git clonetasksource/icl-symbol-tuning-instruct · Datasets at Hugging Face
2024-07-27T19:20:02Z
Thanks for the help. This workedfrom datasets import load_dataset#!git clonetasksource/icl-symbol-tuning-instruct · Datasets at Hugging Faceds = load_dataset(‘parquet’, data_dir=‘icl-symbol-tuning-instruct’)
2024-07-27T19:37:48Z
we’re working on a fix !
2024-07-27T19:41:32Z
We fixed the issue, you can load datasets again, sorry for the inconvenience !
2024-07-27T19:45:13Z
(it was an internal bug due to a modification of our dataset tagging system, that was recently updated to detect datasets compatibility with a famous Rust-based DataFrame library - sorry again !)
2024-07-27T20:20:46Z
Hello! I am still seeing this issue on some datasets (https://huggingface.co/datasets/mlfoundations/MINT-1T-HTML).
No date available
No reply text available