How can I load a locally store model with TGI and Docker?
Hello!I’m working on project that requires me to deploy a locally store model on an air-gapped server. I’m using the TGI docker container (with podman). I’m used to Ollama, so initially attempted to load the local model from a GGUF, but looks like that’s not the case with TGI.The TGI documentation simply says the--model-idflag should use the directory where a model saved usingsave_pretrained(...)method. But what is that format?I just want to use and off-the-shelf pretrained model, in this case Mixtral 8x 7B.Thanks!
2024-06-28T04:49:09Z
Hi. I was looking for the same thing and came across your post.According to thisarticle, you can load locally saved models by mounting the volume in docker (using the argument-v).I will test the solution and update you if it worked.
2024-06-28T13:08:51Z
Yes that’s explained here (at the bottom):Non-core Model Serving
2024-07-02T13:53:58Z
Thank you for the guidance! My remaining question is: what should the localdatadirectory contain, in terms of weights format? Should it just be a copy of the hf repo for a given model distro?
2024-07-06T09:48:08Z
Yes it should just be a copy