Docker Space, connection problems
Spaces…Public:SD WebUI Plus Basics - a Hugging Face Space by MiroCollasPrivate:https://huggingface.co/spaces/MiroCollas/MC_WebUI_Simple_Ahttps://huggingface.co/spaces/MiroCollas/MC_WebUI_Simple_Bhttps://huggingface.co/spaces/MiroCollas/MC_WebUI_Simple_Chttps://huggingface.co/spaces/MiroCollas/MC_WebUI_Simple_WThe problems described in these posts happen on them all. A through C have identical contents, W has small changes in UI config. The public spaces is a pared down version of the others but the scripts are basically the same.So after the issue several people had 24hrs ago with build errors, such asBuild error without logI have found spaces to be very unstable ever since. I keep getting connection errors, over and over again, and now, a script which has worked fine for weeks, to pull in models from civitai, keeps throwing 403 errors.---------------
Running script './on_start.sh' to download models ...
---------------
$ download-model --lora "BulkedUpAIR1.5.safetensors" "https://civitai.com/api/download/models/33323"


06/21 07:17:07 [ERROR] CUID#7 - Download aborted. URI=https://civitai.com/api/download/models/33323
Exception: [AbstractCommand.cc:351] errorCode=22 URI=https://civitai.com/api/download/models/33323
  -> [HttpSkipResponseCommand.cc:239] errorCode=22 The response status is not successful. status=403The space ishttps://huggingface.co/spaces/MiroCollas/MC_WebUI_Simple_Awhich was set up as Docker Blank. it uses aria2 to download.A 403 is, I believe, “forbidden”, by by whom? Civitai or HF?If there are changes which affect spaces, that users need to know about so we can adapt and edit scripts, where are these changes posted?
2023-06-21T07:58:42Z
Sod’s law… I post the above, and 5mins later, it starts working again. But for how long?
2023-06-21T08:16:40Z
LOL I knew it was too good to be true. First this…Screenshot_20230621_040221785×444 37 KBSo I rebooted … then I get this:Cloning Stable Diffusion into /app/stable-diffusion-webui/repositories/stable-diffusion-stability-ai...
Cloning Taming Transformers into /app/stable-diffusion-webui/repositories/taming-transformers...
Cloning K-diffusion into /app/stable-diffusion-webui/repositories/k-diffusion...
Cloning CodeFormer into /app/stable-diffusion-webui/repositories/CodeFormer...
Traceback (most recent call last):
  File "/app/stable-diffusion-webui/launch.py", line 38, in <module>
    main()
  File "/app/stable-diffusion-webui/launch.py", line 29, in main
    prepare_environment()
  File "/app/stable-diffusion-webui/modules/launch_utils.py", line 291, in prepare_environment
    git_clone(codeformer_repo, repo_dir('CodeFormer'), "CodeFormer", codeformer_commit_hash)
  File "/app/stable-diffusion-webui/modules/launch_utils.py", line 147, in git_clone
    run(f'"{git}" clone "{url}" "{dir}"', f"Cloning {name} into {dir}...", f"Couldn't clone {name}")
  File "/app/stable-diffusion-webui/modules/launch_utils.py", line 101, in run
    raise RuntimeError("\n".join(error_bits))
RuntimeError: Couldn't clone CodeFormer.
Command: "git" clone "https://github.com/sczhou/CodeFormer.git" "/app/stable-diffusion-webui/repositories/CodeFormer"
Error code: 128
stderr: Cloning into '/app/stable-diffusion-webui/repositories/CodeFormer'...
fatal: unable to access 'https://github.com/sczhou/CodeFormer.git/': Could not resolve host: github.com


--> ERROR: process "/bin/sh -c /opt/venv/bin/python launch.py --exit --skip-torch-cuda-test --xformers" did not complete successfully: exit code: 1I give up for tonight.
2023-06-21T19:52:35Z
I was able to successfully reboot the space a couple hours ago, but now we’re back to this:---------------
Running script './on_start.sh' to download models ...
---------------
$ download-model --lora "BulkedUpAIR1.5.safetensors" "https://civitai.com/api/download/models/33323"


06/21 19:43:45 [ERROR] CUID#7 - Download aborted. URI=https://civitai.com/api/download/models/33323
Exception: [AbstractCommand.cc:351] errorCode=22 URI=https://civitai.com/api/download/models/33323
  -> [HttpSkipResponseCommand.cc:239] errorCode=22 The response status is not successful. status=403

Download Results:
gid   |stat|avg speed  |path/URI
======+====+===========+=======================================================
126c3b|ERR |       0B/s|/app/stable-diffusion-webui/models/Lora/BulkedUpAIR1.5.safetensors

Status Legend:
(ERR):error occurred.I restarted the space again, and it worked, so it is intermittent, whatever the problem is.However on another space,SD WebUI Plus Basics - a Hugging Face Space by MiroCollaswhich is public,06/21 19:50:40 [ERROR] CUID#7 - Download aborted. URI=https://civitai.com/api/download/models/76712
Exception: [AbstractCommand.cc:351] errorCode=22 URI=https://civitai.com/api/download/models/76712
  -> [HttpSkipResponseCommand.cc:239] errorCode=22 The response status is not successful. status=500

Download Results:
gid   |stat|avg speed  |path/URI
======+====+===========+=======================================================
ff83c3|ERR |       0B/s|/app/stable-diffusion-webui/embeddings/FastNegativeEmbedding.pt

Status Legend:
(ERR):error occurred.

aria2 will resume download if the transfer is restarted.
If there are any errors, then see the log file. See '-l' option in help/man page for details.
Traceback (most recent call last):
  File "/app/stable-diffusion-webui/run.py", line 35, in <module>
    start()
  File "/app/stable-diffusion-webui/run.py", line 16, in start
    on_start()
  File "/app/stable-diffusion-webui/run.py", line 12, in on_start
    raise RuntimeError(f"Error executing ./on_start.sh [exit code: {result.returncode}]")
RuntimeError: Error executing ./on_start.sh [exit code: 22][sigh]
2023-06-21T23:57:58Z
New errors, and as far as i can tell, they are all connection-related, in one way or another.--> RUN poetry install
Installing dependencies from lock file

Package operations: 15 installs, 0 updates, 0 removals

  • Installing certifi (2022.12.7)
  • Installing charset-normalizer (3.0.1)
  • Installing idna (3.4)
  • Installing typing-extensions (4.5.0)
  • Installing urllib3 (1.26.14)
  • Installing cmake (3.25.2)
  • Installing filelock (3.9.0)
  • Installing lit (15.0.7)
  • Installing numpy (1.24.2)
  • Installing pillow (9.4.0)
  • Installing requests (2.28.2)
  • Installing torch (1.13.1+cu117 https://download.pytorch.org/whl/cu117/torch-1.13.1%2Bcu117-cp310-cp310-linux_x86_64.whl)
  • Installing numexpr (2.8.4)
  • Installing torchvision (0.14.1+cu117 https://download.pytorch.org/whl/cu117/torchvision-0.14.1%2Bcu117-cp310-cp310-linux_x86_64.whl)
  • Installing triton (2.0.0)

  RuntimeError

  Hash for triton (2.0.0) from archive triton-2.0.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl not found in known hashes (was: sha256:e95ac47e08f205714bcae4f0f2b2acd9157f6ec33fa2598080cf4bbf37e5fcc3)

  at /usr/local/poetry/venv/lib/python3.10/site-packages/poetry/installation/executor.py:818 in _validate_archive_hash
      814│         archive_hash: str = "sha256:" + get_file_hash(archive)
      815│         known_hashes = {f["hash"] for f in package.files if f["file"] == archive.name}
      816│ 
      817│         if archive_hash not in known_hashes:
    → 818│             raise RuntimeError(
      819│                 f"Hash for {package} from archive {archive.name} not found in"
      820│                 f" known hashes (was: {archive_hash})"
      821│             )
      822│ 


--> ERROR: process "/bin/sh -c poetry install" did not complete successfully: exit code: 1
2023-06-22T00:34:06Z
I decided to delete space _A (see top post, and recreate it by duplicating _B (which was running at the time). New errors…Python 3.10.6 (main, May 29 2023, 11:10:38) [GCC 11.3.0]
Version: v1.3.2
Commit hash: baf6946e06249c5af9851c60171692c44ef633e0
Installing gfpgan
Traceback (most recent call last):
  File "/app/stable-diffusion-webui/launch.py", line 38, in <module>
    main()
  File "/app/stable-diffusion-webui/launch.py", line 29, in main
    prepare_environment()
  File "/app/stable-diffusion-webui/modules/launch_utils.py", line 263, in prepare_environment
    run_pip(f"install {gfpgan_package}", "gfpgan")
  File "/app/stable-diffusion-webui/modules/launch_utils.py", line 124, in run_pip
    return run(f'"{python}" -m pip {command} --prefer-binary{index_url_line}', desc=f"Installing {desc}", errdesc=f"Couldn't install {desc}", live=live)
  File "/app/stable-diffusion-webui/modules/launch_utils.py", line 101, in run
    raise RuntimeError("\n".join(error_bits))
RuntimeError: Couldn't install gfpgan.
Command: "/opt/venv/bin/python" -m pip install https://github.com/TencentARC/GFPGAN/archive/8d2447a2d918f8eba5a4a01463fd48e45126a379.zip --prefer-binary
Error code: 1
stdout: Collecting https://github.com/TencentARC/GFPGAN/archive/8d2447a2d918f8eba5a4a01463fd48e45126a379.zip
  Downloading https://github.com/TencentARC/GFPGAN/archive/8d2447a2d918f8eba5a4a01463fd48e45126a379.zip (6.0 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 6.0/6.0 MB 61.9 MB/s eta 0:00:00
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Installing backend dependencies: started
  Installing backend dependencies: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Collecting basicsr>=1.4.2 (from gfpgan==1.3.5)
  Downloading basicsr-1.4.2.tar.gz (172 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 172.5/172.5 kB 6.6 MB/s eta 0:00:00
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Installing backend dependencies: started
  Installing backend dependencies: finished with status 'error'

stderr:   error: subprocess-exited-with-error
  
  × pip subprocess to install backend dependencies did not run successfully.
  │ exit code: 1
  ╰─> [11 lines of output]
      Collecting numpy
        Downloading numpy-1.25.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (17.6 MB)
           ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 17.6/17.6 MB 171.8 MB/s eta 0:00:00
      Collecting torch
        Downloading torch-2.0.1-cp310-cp310-manylinux1_x86_64.whl (619.9 MB)
           ╸                                      10.5/619.9 MB 144.6 MB/s eta 0:00:05
      ERROR: THESE PACKAGES DO NOT MATCH THE HASHES FROM THE REQUIREMENTS FILE. If you have updated the package versions, please update the hashes. Otherwise, examine the package contents carefully; someone may have tampered with them.
          torch from https://files.pythonhosted.org/packages/8c/4d/17e07377c9c3d1a0c4eb3fde1c7c16b5a0ce6133ddbabc08ceef6b7f2645/torch-2.0.1-cp310-cp310-manylinux1_x86_64.whl:
              Expected sha256 8ced00b3ba471856b993822508f77c98f48a458623596a4c43136158781e306a
                   Got        307e5752dbdc4dbb85d7e07196521111ece7ebee6247d9c574b5c3a4c9c6a970
      
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× pip subprocess to install backend dependencies did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.```
2023-06-22T06:11:58Z
Things have improved in the last few hours, but they’re not yet “good”. I still get “connection lost” errors while generating images. Refreshing the page generally gives me a “preparing space” page so I need to restart the space. And that has to be done several times, as I get download errors of the sort already posted above, until eventually it all fetches and I can restart the generation - from scratch.
2023-06-23T08:19:12Z
So it turns out, part of my problems were civitai barfing on download requests. However, that leaves the problem of the space often giving a “connection lost” error, totally wiping out all work in progress.Anyway, thanks for the suggestions and feedb…[looks round at the empty room, hearing the occasional cricket outside]Oh wait, never mind.
2023-06-23T18:36:33Z
hi@anon22701748I’m glad you’ve figure out the issue. Maybe send civitai feeback to returnHTTP 429 Too Many Requestsrepose code. That would be very helpful in the future.
2023-06-23T18:58:45Z
Good idea, thank you.But that is only part of the problem. As noted previously, I often get “connection lost” errors in the space, which wipes out the work in progress.[Added:] Might I also suggest something like thisstatus.secondlifegrid.netSecond Life StatusWelcome to Second Life's home for real-time and historical data on system performance.so if there are general system issues, users can be made aware? It might cut back on some of the forum posts.
2023-06-23T20:32:31Z
yes we have a status pagehttps://status.huggingface.co/
2023-06-23T21:52:52Z
Oh! Thanks! I have looked for one and not found it. Bookmarked.