Looking for a Tiny LLM (max 1.5GB) – Need Advice
Hey everyone,I’m currently looking for a very small LLM that cannot exceed1.5GBin size. The goal is for it to handle simple Q&A tasks (nothing fact-based or overly complex, just basic interpretation of input). Additionally, the model needs to understand basic writing mistakes (typos, grammar issues) and be able to handle very primitive interactions.I understand that larger models generally perform better, but I’m really constrained by size limits here. Does anyone have experience working with models of this size or know how to achieve something like this while retaining minimal functionality?Any advice or guidance would be greatly appreciated!Thanks in advance!
2024-09-27T10:55:58Z
If you are looking for a model that is 1GB in the quantized state, you can find a 1B model GGUF and you are done, but the number of models within 1GB in the float16 state is limited.The following model meets the requirements, and the author has a lot to say about small models.huggingface.conroggendorff (Noa Roggendorff)Large language models, Illusion Diffusion, Stable Diffusion, Stable Video Diffusionhuggingface.conroggendorff/smallama-it · Hugging FaceWe’re on a journey to advance and democratize artificial intelligence through open source and open science.You can find the GGUF model of 1B at the link below.huggingface.coModels - Hugging FaceWe’re on a journey to advance and democratize artificial intelligence through open source and open science.
2024-09-28T22:46:43Z
there’s this one as well by@KingNish, really great performance considering its sizehuggingface.coKingNish/Qwen2.5-0.5b-Test-ft · Hugging FaceWe’re on a journey to advance and democratize artificial intelligence through open source and open science.
2024-09-29T00:47:11Z
Qwen 2.5 is very high performance in general. Especially the multilingual performance is much better than 2. Other 0.5B models can also be searched as follows.By the way, the automatic generation of the link from the forum to the model search is slightly buggy.One of the URLs below works fine, but I fixed it manually. The other one does not work. I don’t understand the logic.huggingface.coModels - Hugging FaceWe’re on a journey to advance and democratize artificial intelligence through open source and open science.https://huggingface.co/models?sort=modified&search=GGUF%200.5B
2024-09-29T01:20:21Z
i’ll open a separate issue in a the form and report this, thanks for sharing
2024-09-29T01:21:09Z
thank you to open issue.Broken search link when when using ggufSite FeedbackAfter searching for something using the gguf filter, the url could not be shared with others and it does not work even if I open a different tab and post the link there. 
an example of some broken urls I have foundModels - Hugging Face,Models - Hugging Faceandhttps://huggingface.co/models?library=gguf&sort=trending&search=0.5bthanks to@John6666for reporting this