LFS no longer works for my Huggingface models
I’ve been able to use LFS fine so far on a whole bunch of models on Huggingface, but since tuday I’m getting this error when trying to push data with LFS :Uploading LFS objects:   0% (0/1), 0 B | 37 KB/s, done.
LFS: Authorization error: https://s3.us-east-1.amazonaws.com/lfs.huggingface.co/repos/d7/38/d738d10b0bff88e0df117384720684e38a23e633549f988ab9a015c4f0066648/6a96896ad631726021758af7de2f6076ab54ee698300ceae28c5f1adf2450e82?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Content-Sha256=UNSIGNED-PAYLOAD&X-Amz-Credential=AKIA4N7VTDGO27GPWFUO%2F20230128%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20230128T115458Z&X-Amz-Expires=900&X-Amz-Signature=f9e4f7fefab5ffec5c565438d9c1c99876a137b7618e8ad7d175e7ae7cbf3c1f&X-Amz-SignedHeaders=host&x-amz-storage-class=INTELLIGENT_TIERING&x-id=PutObject
Check that you have proper access to the repositoryThe error message contains the following XML :<Error>
  <Code>SignatureDoesNotMatch</Code>
  <Message>The request signature we calculated does not match the signature you provided. Check your key and signing method.</Message>
  <AWSAccessKeyId>AKIA4N7VTDGO27GPWFUO</AWSAccessKeyId>
  <StringToSign>AWS4-HMAC-SHA256
20230128T114614Z
20230128/us-east-1/s3/aws4_request
ddf0c59292a0789b110f5c5749d7ae8c430cc6e281b41394f0f72b988e2e0dab</StringToSign>
  <SignatureProvided>0dce350ebe9bd071eb56a82b83781c20cbb3504b5fdd0731a31668f36dcad0f0</SignatureProvided>
  <StringToSignBytes>41 57 53 34 2d 48 4d 41 43 2d 53 48 41 32 35 36 0a 32 30 32 33 30 31 32 38 54 31 31 34 36 31 34 5a 0a 32 30 32 33 30 31 32 38 2f 75 73 2d 65 61 73 74 2d 31 2f 73 33 2f 61 77 73 34 5f 72 65 71 75 65 73 74 0a 64 64 66 30 63 35 39 32 39 32 61 30 37 38 39 62 31 31 30 66 35 63 35 37 34 39 64 37 61 65 38 63 34 33 30 63 63 36 65 32 38 31 62 34 31 33 39 34 66 30 66 37 32 62 39 38 38 65 32 65 30 64 61 62</StringToSignBytes>
  <CanonicalRequest>GET
/lfs.huggingface.co/repos/d7/38/d738d10b0bff88e0df117384720684e38a23e633549f988ab9a015c4f0066648/459ab0e6771c051601a4999452f9fa7bb67f44a893d4d4b2de9b2da806828c3b
X-Amz-Algorithm=AWS4-HMAC-SHA256&amp;X-Amz-Content-Sha256=UNSIGNED-PAYLOAD&amp;X-Amz-Credential=AKIA4N7VTDGO27GPWFUO%2F20230128%2Fus-east-1%2Fs3%2Faws4_request&amp;X-Amz-Date=20230128T114614Z&amp;X-Amz-Expires=900&amp;X-Amz-SignedHeaders=host&amp;x-amz-storage-class=INTELLIGENT_TIERING&amp;x-id=PutObject
host:s3.us-east-1.amazonaws.com host UNSIGNED-PAYLOAD</CanonicalRequest>
  <CanonicalRequestBytes>47 45 54 0a 2f 6c 66 73 2e 68 75 67 67 69 6e 67 66 61 63 65 2e 63 6f 2f 72 65 70 6f 73 2f 64 37 2f 33 38 2f 64 37 33 38 64 31 30 62 30 62 66 66 38 38 65 30 64 66 31 31 37 33 38 34 37 32 30 36 38 34 65 33 38 61 32 33 65 36 33 33 35 34 39 66 39 38 38 61 62 39 61 30 31 35 63 34 66 30 30 36 36 36 34 38 2f 34 35 39 61 62 30 65 36 37 37 31 63 30 35 31 36 30 31 61 34 39 39 39 34 35 32 66 39 66 61 37 62 62 36 37 66 34 34 61 38 39 33 64 34 64 34 62 32 64 65 39 62 32 64 61 38 30 36 38 32 38 63 33 62 0a 58 2d 41 6d 7a 2d 41 6c 67 6f 72 69 74 68 6d 3d 41 57 53 34 2d 48 4d 41 43 2d 53 48 41 32 35 36 26 58 2d 41 6d 7a 2d 43 6f 6e 74 65 6e 74 2d 53 68 61 32 35 36 3d 55 4e 53 49 47 4e 45 44 2d 50 41 59 4c 4f 41 44 26 58 2d 41 6d 7a 2d 43 72 65 64 65 6e 74 69 61 6c 3d 41 4b 49 41 34 4e 37 56 54 44 47 4f 32 37 47 50 57 46 55 4f 25 32 46 32 30 32 33 30 31 32 38 25 32 46 75 73 2d 65 61 73 74 2d 31 25 32 46 73 33 25 32 46 61 77 73 34 5f 72 65 71 75 65 73 74 26 58 2d 41 6d 7a 2d 44 61 74 65 3d 32 30 32 33 30 31 32 38 54 31 31 34 36 31 34 5a 26 58 2d 41 6d 7a 2d 45 78 70 69 72 65 73 3d 39 30 30 26 58 2d 41 6d 7a 2d 53 69 67 6e 65 64 48 65 61 64 65 72 73 3d 68 6f 73 74 26 78 2d 61 6d 7a 2d 73 74 6f 72 61 67 65 2d 63 6c 61 73 73 3d 49 4e 54 45 4c 4c 49 47 45 4e 54 5f 54 49 45 52 49 4e 47 26 78 2d 69 64 3d 50 75 74 4f 62 6a 65 63 74 0a 68 6f 73 74 3a 73 33 2e 75 73 2d 65 61 73 74 2d 31 2e 61 6d 61 7a 6f 6e 61 77 73 2e 63 6f 6d 0a 0a 68 6f 73 74 0a 55 4e 53 49 47 4e 45 44 2d 50 41 59 4c 4f 41 44</CanonicalRequestBytes>
  <RequestId>G81B83SMG39M9736</RequestId>
  <HostId>mIp/18+bzGVnGr3LupEn9Iixd3Zegf/mtmmXD/DbSItokeQt9XNft2+AH7fqqWDFVQGr16/iJw0=</HostId>
</Error>I figured I may have met some hidden data storage limit, but after deleting one of my CKPT files I kept getting the same error, even when trying to push very small files.I’m getting this for any of my models I try this with, whether hidden or public.I’m getting the same issue both when I push from my local machine or when I push from a hosted environment like Google Colab.Any idea what might be causing this or how I can fix this? I was about to officially release at least one new Stable Diffusion model in the near future. This totally blocks me both from releasing these models as well as from creating any new ones.
2023-01-28T13:50:27Z
I think it has to do with the backend of HF.Can’t even upload through the browserimage3518×1280 329 KB
2023-01-28T14:10:30Z
jed351:I think it has to do with the backend of HF.The issue definitely doesn’t seem to be specific for my account or my IP. I tried creating a new account with a new email address and move a model to my new account from a friend’s home and I got the exact same error.
2023-01-28T14:32:14Z
I just noticed the issue has been logged on Github :github.com/huggingface/huggingface_hubError when uploading any lfs file to model repoopened10:06AM - 28 Jan 23 UTCfcakyonbug### Describe the bug

When uploading a binary, huggingface-hub raises an error.…It is happening in my all attempts since this morning.

Traceback:
`error: Traceback (most recent call last):
  File "D:\miniconda3\envs\yolo\lib\site-packages\urllib3\connectionpool.py", line 699, in urlopen
    httplib_response = self._make_request(
  File "D:\miniconda3\envs\yolo\lib\site-packages\urllib3\connectionpool.py", line 394, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "D:\miniconda3\envs\yolo\lib\site-packages\urllib3\connection.py", line 234, in request
    super(HTTPConnection, self).request(method, url, body=body, headers=headers)
  File "D:\miniconda3\envs\yolo\lib\http\client.py", line 1282, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "D:\miniconda3\envs\yolo\lib\http\client.py", line 1328, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "D:\miniconda3\envs\yolo\lib\http\client.py", line 1277, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "D:\miniconda3\envs\yolo\lib\http\client.py", line 1076, in _send_output
    self.send(chunk)
  File "D:\miniconda3\envs\yolo\lib\http\client.py", line 998, in send
    self.sock.sendall(data)
  File "D:\miniconda3\envs\yolo\lib\ssl.py", line 1237, in sendall
    v = self.send(byte_view[count:])
  File "D:\miniconda3\envs\yolo\lib\ssl.py", line 1206, in send
    return self._sslobj.write(data)
ssl.SSLEOFError: EOF occurred in violation of protocol (_ssl.c:2396)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "D:\miniconda3\envs\yolo\lib\site-packages\requests\adapters.py", line 489, in send
    resp = conn.urlopen(
  File "D:\miniconda3\envs\yolo\lib\site-packages\urllib3\connectionpool.py", line 755, in urlopen
    retries = retries.increment(
  File "D:\miniconda3\envs\yolo\lib\site-packages\urllib3\util\retry.py", line 574, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='[s3.us-east-1.amazonaws.com](http://s3.us-east-1.amazonaws.com/)', port=443): Max retries exceeded with url: /[lfs.huggingface.co/repos/89/dc/89dce452e3c26d2fb61c88e246ffaf1393e7688a3acbebd3b537142e284c5b91/c1c59597678b682895ccc053e1f27c0dcb03e42a02c531e987940b06bf015d53?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Content-Sha256=UNSIGNED-PAYLOAD&X-Amz-Credential=%2F20230128%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20230128T091758Z&X-Amz-Expires=900&X-Amz-Signature=e763dfc60c61474fe8711eb1c0646121ba3fa787da371419a87ab85c9b768da3&X-Amz-SignedHeaders=host&x-amz-storage-class=INTELLIGENT_TIERING&x-id=PutObject](http://lfs.huggingface.co/repos/89/dc/89dce452e3c26d2fb61c88e246ffaf1393e7688a3acbebd3b537142e284c5b91/c1c59597678b682895ccc053e1f27c0dcb03e42a02c531e987940b06bf015d53?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Content-Sha256=UNSIGNED-PAYLOAD&X-Amz-Credential=%2F20230128%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20230128T091758Z&X-Amz-Expires=900&X-Amz-Signature=e763dfc60c61474fe8711eb1c0646121ba3fa787da371419a87ab85c9b768da3&X-Amz-SignedHeaders=host&x-amz-storage-class=INTELLIGENT_TIERING&x-id=PutObject) (Caused by SSLError(SSLEOFError(8, 'EOF occurred in violation of protocol (_ssl.c:2396)')))


During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "D:\miniconda3\envs\yolo\lib\site-packages\huggingface_hub\_commit_api.py", line 334, in _inner_upload_lfs_object
    return _upload_lfs_object(
  File "D:\miniconda3\envs\yolo\lib\site-packages\huggingface_hub\_commit_api.py", line 391, in _upload_lfs_object
    lfs_upload(
  File "D:\miniconda3\envs\yolo\lib\site-packages\huggingface_hub\lfs.py", line 273, in lfs_upload
    _upload_single_part(
  File "D:\miniconda3\envs\yolo\lib\site-packages\huggingface_hub\lfs.py", line 304, in _upload_single_part
    upload_res = http_backoff("PUT", upload_url, data=fileobj)
  File "D:\miniconda3\envs\yolo\lib\site-packages\huggingface_hub\utils\_http.py", line 129, in http_backoff
    response = requests.request(method=method, url=url, **kwargs)
  File "D:\miniconda3\envs\yolo\lib\site-packages\requests\api.py", line 59, in request
    return session.request(method=method, url=url, **kwargs)
  File "D:\miniconda3\envs\yolo\lib\site-packages\requests\sessions.py", line 587, in request
    resp = self.send(prep, **send_kwargs)
  File "D:\miniconda3\envs\yolo\lib\site-packages\requests\sessions.py", line 701, in send
    r = adapter.send(request, **kwargs)
  File "D:\miniconda3\envs\yolo\lib\site-packages\requests\adapters.py", line 563, in send
    raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host='[s3.us-east-1.amazonaws.com](http://s3.us-east-1.amazonaws.com/)', port=443): Max retries exceeded with url: /[lfs.huggingface.co/repos/89/dc/89dce452e3c26d2fb61c88e246ffaf1393e7688a3acbebd3b537142e284c5b91/c1c59597678b682895ccc053e1f27c0dcb03e42a02c531e987940b06bf015d53?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Content-Sha256=UNSIGNED-PAYLOAD&X-Amz-Credential=%2F20230128%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20230128T091758Z&X-Amz-Expires=900&X-Amz-Signature=e763dfc60c61474fe8711eb1c0646121ba3fa787da371419a87ab85c9b768da3&X-Amz-SignedHeaders=host&x-amz-storage-class=INTELLIGENT_TIERING&x-id=PutObject](http://lfs.huggingface.co/repos/89/dc/89dce452e3c26d2fb61c88e246ffaf1393e7688a3acbebd3b537142e284c5b91/c1c59597678b682895ccc053e1f27c0dcb03e42a02c531e987940b06bf015d53?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Content-Sha256=UNSIGNED-PAYLOAD&X-Amz-Credential=%2F20230128%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20230128T091758Z&X-Amz-Expires=900&X-Amz-Signature=e763dfc60c61474fe8711eb1c0646121ba3fa787da371419a87ab85c9b768da3&X-Amz-SignedHeaders=host&x-amz-storage-class=INTELLIGENT_TIERING&x-id=PutObject) (Caused by SSLError(SSLEOFError(8, 'EOF occurred in violation of protocol (_ssl.c:2396)')))`

### Reproduction

_No response_

### Logs

_No response_

### System info

```shell
windows 10, huggingfacehub 0.12.0

will post detailed info when im back home.
```
2023-01-28T14:42:05Z
Maybe the token you’re using to authenticate is a Read only token? Can you try login-cli again?
2023-01-28T14:45:54Z
I’ve been using the same token for weeks. It worked flawlessly until today.Either way, several people seem to experience the same issue. See the Github link I posted in the previous comment!
2023-01-28T16:43:05Z
Thanks for reporting it. I can reproduce the issue as well. I’ve already notified the infra team. I’ll ping you back.
2023-01-28T17:04:09Z
ok@johnslegers@jed351the error is fixed now. Sorry for the inconvenience.
2023-01-28T17:08:10Z
I noticed.I can confirm that I no longer have this issue.Thanks!