How to Finetune and deploy LLaVA-1.6
Hi,Im new here and I’m facing the following issues:I was playing with llava-v1-6-vicuna-13b. Having looked at the github repo with info on how to fine-tune the same, I was able to finetune the model. I uploaded the model to Huggingface on a private repo and I used the deploy using interface endpoints feature but got the following error:huggingface_hub.utils._errors.RepositoryNotFoundError: 401 Client Error.Repository Not Found for url:Please make sure you specified the correctrepo_idandrepo_type.If you are trying to access a private or gated repo, make sure you are authenticated.Invalid username or password.When I was using the LlavaNextProcessor library from thislink, I was unable to use it on my fine-tuned model and there are no instructions on how to fine-tune using the Library if possible or any other ways.Any help would be highly appreciated
2024-04-15T04:44:45Z
Hi, could you please share the github repo that you used to fine-tune llava 1.6
2024-04-15T13:43:32Z
I used the following Repo:GitHubGitHub - haotian-liu/LLaVA: [NeurIPS'23 Oral] Visual Instruction Tuning...[NeurIPS'23 Oral] Visual Instruction Tuning (LLaVA) built towards GPT-4V level capabilities and beyond. - haotian-liu/LLaVA
2024-06-04T08:44:49Z
Hi@PritamSriramG, I wanted to know which data set did you use?
2024-06-04T11:33:29Z
Hi,I have used a custom data which I generated. It is of the format specified by the Repo and I was able to finetune using the github repo. The problem arises when I try to use HF libraries. I want to train using HF because, Im planning to host the model in cloud. So, it would be great if you could provide me information on how to finetune using HF or any info on how to host the finetuned model.
2024-06-04T18:07:14Z
Hi,Fine-tuning of LLaVa-Next should now work out-of-the-box as shown in the notebook here:Transformers-Tutorials/LLaVa at master · NielsRogge/Transformers-Tutorials · GitHub. Make sure to replace the processor, model and chat templates by the one of LLaVa-Next instead of LLaVa.Regarding deployment, TGI (a framework meant for deployment of LLMs and multimodal models) now supports LLaVa 1.6. See the guide here:Vision Language Model Inference in TGI. Besides that, vLLM also has added support for it:Supported Models — vLLM
2024-06-10T18:23:53Z
Hi@nielsr,Thanks for your suggestions. When I was trying to run your notebook, I got the following error:ImportError: Usingbitsandbytes8-bit quantization requires Accelerate:pip install accelerateand the latest version of bitsandbytes:pip install -iSimple indexbitsandbytesat the line:model = LlavaForConditionalGeneration.from_pretrained(even after installing the libraries, it remained the same. Can you kindly let me know how to go ahead with this issue?
2024-06-10T19:07:51Z
Hi,Are you running on a CUDA compatible device (not on CPU)? Otherwise restarting the runtime might help.